# CloudBytes/dev — full content corpus

> CloudBytes/dev — code snippets, AWS Academy lessons, and tutorials for cloud, Python, Linux, and developer tooling.

Generated for AI ingestion. 97 posts, sorted newest first.

---

# Configure DynamoDB point-in-time recovery and deletion protection using AWS CDK

URL: https://cloudbytes.dev/aws-academy/configure-dynamodb-point-in-time-recovery-and-deletion-protection-using-aws-cdk
Category: AWS Academy
Published: 2026-05-31
Author: Rehan Haider
Tags: aws, cdk, python, dynamodb

> How to enable point-in-time recovery and deletion protection for a DynamoDB table using AWS CDK in Python

In the previous article, we created a basic DynamoDB table using AWS CDK. In this post, we will make the table safer by enabling
point-in-time recovery and deletion protection.

These two settings solve different problems:

1. **Point-in-time recovery** lets you restore a table to a previous point in time.
2. **Deletion protection** prevents accidental table deletion.

For a real table, especially one that stores user or business data, I would normally configure these before the table receives production
traffic.

## Prerequisites

1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md).
2. Complete the previous article: [Create a DynamoDB table using AWS CDK in Python]({filename}50004000-cdk-dynamodb-create-table.md).
3. Configure your AWS CLI profile for the account and region where you want to deploy the table.

## What we will configure

We will update the DynamoDB stack to configure:

1. Point-in-time recovery.
2. Deletion protection.
3. A retain removal policy for safer stack deletion behavior.

The project file we will edit is:

```text
cdk_app/dynamodb_stack.py
```

## Update the DynamoDB table

Open `cdk_app/dynamodb_stack.py` and update the table definition:

```python
# filename: cdk_app/dynamodb_stack.py

from aws_cdk import (
    CfnOutput,
    RemovalPolicy,
    Stack,
    aws_dynamodb as dynamodb,
)
from constructs import Construct


class DynamoDbStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        table = dynamodb.Table(
            self,
            "TodosTable",
            partition_key=dynamodb.Attribute(
                name="todo_id",
                type=dynamodb.AttributeType.STRING,
            ),
            billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
            point_in_time_recovery_specification=dynamodb.PointInTimeRecoverySpecification(
                point_in_time_recovery_enabled=True,
            ),
            deletion_protection=True,
            removal_policy=RemovalPolicy.RETAIN,
        )

        CfnOutput(self, "TableName", value=table.table_name)
        CfnOutput(self, "TableArn", value=table.table_arn)
```

In the above code:

1. `dynamodb.Table` keeps the same construct we used in the first DynamoDB article.
2. `BillingMode.PAY_PER_REQUEST` keeps the table on on-demand billing.
3. `point_in_time_recovery_specification` enables point-in-time recovery.
4. `deletion_protection=True` prevents accidental table deletion.
5. `RemovalPolicy.RETAIN` tells CloudFormation to keep the table if the stack is deleted.

AWS CDK also has an older `point_in_time_recovery=True` option, but that property is now deprecated. Use `PointInTimeRecoverySpecification`
for new code.

!!! warning
    Do not use `RemovalPolicy.DESTROY` casually for a DynamoDB table that stores real data. If the table is important, use `RemovalPolicy.RETAIN` and make data deletion a deliberate manual step.

## Why not switch to TableV2 here?

AWS CDK also has a newer `dynamodb.TableV2` construct. It is useful for global table use cases and newer per-replica configuration options.

However, do not casually replace an existing `dynamodb.Table` construct with `dynamodb.TableV2` in a deployed stack. AWS documents this in
its
[Table to TableV2 migration guidance](https://aws.amazon.com/blogs/database/zero-downtime-dynamodb-construct-migration-from-table-to-tablev2-with-cdk-orphan/):
CloudFormation can treat that kind of construct migration as a resource replacement.

For this article, we are updating the table created in the previous lesson, so we keep using `dynamodb.Table`.

The table shape stays the same:

```python
partition_key=dynamodb.Attribute(
    name="todo_id",
    type=dynamodb.AttributeType.STRING,
)
```

And the billing mode stays the same:

```python
billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST
```

## Deploy the change

Before deploying, run `cdk diff`:

```bash
cdk diff
```

You should see that the DynamoDB table configuration will be updated.

If the diff looks correct, deploy the stack:

```bash
cdk deploy
```

After deployment, the table will have point-in-time recovery and deletion protection enabled.

## Verify point-in-time recovery

Get the table name from the stack output:

```bash
table_name=$(aws cloudformation describe-stacks \
    --stack-name DynamoDbStack \
    --query "Stacks[0].Outputs[?OutputKey=='TableName'].OutputValue" \
    --output text)
```

Then run:

```bash
aws dynamodb describe-continuous-backups \
    --table-name "$table_name"
```

In the response, check the `PointInTimeRecoveryDescription` section. The `PointInTimeRecoveryStatus` value should be `ENABLED`.

## Verify deletion protection

You can verify deletion protection with `describe-table`:

```bash
aws dynamodb describe-table \
    --table-name "$table_name" \
    --query "Table.DeletionProtectionEnabled"
```

The output should be:

```text
true
```

## What happens when you destroy the stack?

Because the table uses `RemovalPolicy.RETAIN`, CloudFormation will not delete the table when you destroy the stack. This is intentional.

Run the following command only if you are done testing the stack:

```bash
cdk destroy
```

After the stack is destroyed, the DynamoDB table can remain in your AWS account. That means you may still be charged for storage or requests
against the table.

## Fully delete the table

If this is only a tutorial table and you want to delete it completely, make deletion explicit.

First, update the table settings:

```python
# filename: cdk_app/dynamodb_stack.py

table = dynamodb.Table(
    self,
    "TodosTable",
    partition_key=dynamodb.Attribute(
        name="todo_id",
        type=dynamodb.AttributeType.STRING,
    ),
    billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
    point_in_time_recovery_specification=dynamodb.PointInTimeRecoverySpecification(
        point_in_time_recovery_enabled=True,
    ),
    deletion_protection=False,
    removal_policy=RemovalPolicy.DESTROY,
)
```

Deploy the update:

```bash
cdk deploy
```

Then destroy the stack:

```bash
cdk destroy
```

!!! warning
    This deletes the DynamoDB table and the data inside it. Only do this for a tutorial table or after you have exported, backed up, or migrated the data.

## Next steps

Now that the table has basic safety settings, the next useful DynamoDB topic is to add a Global Secondary Index so we can query the same
table by a different access pattern.

---

# Create a DynamoDB table using AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/create-dynamodb-table-using-aws-cdk-in-python
Category: AWS Academy
Published: 2026-05-31
Author: Rehan Haider
Tags: aws, cdk, python, dynamodb

> How to create a DynamoDB table with AWS CDK in Python, output the table name, add a sample item, and clean up the stack

In this post, we will create a DynamoDB table using AWS CDK in Python. We will keep the table simple in this first article: one partition
key, on-demand billing, and a CloudFormation output so we can test it from the terminal.

This is the first article in the DynamoDB CDK series. Once the table is in place, we can build on it with sort keys, indexes, queries, and
backup settings in later articles.

## Prerequisites

1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md).
2. If needed, [create a new CDK application]({filename}50000020-cdk-new-app.md).
3. Configure your AWS CLI profile for the account and region where you want to deploy the table.

## What we will create

We will create the following:

1. A DynamoDB table named by CloudFormation.
2. A string partition key called `todo_id`.
3. On-demand billing so we do not need to configure read and write capacity.
4. Stack outputs for the table name and table ARN.

The project files we will edit are:

```text
app.py
cdk_app/dynamodb_stack.py
```

## Create the DynamoDB stack

Create a new file named `dynamodb_stack.py` inside the `cdk_app` directory:

```python
# filename: cdk_app/dynamodb_stack.py

from aws_cdk import (
    CfnOutput,
    RemovalPolicy,
    Stack,
    aws_dynamodb as dynamodb,
)
from constructs import Construct


class DynamoDbStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        table = dynamodb.Table(
            self,
            "TodosTable",
            partition_key=dynamodb.Attribute(
                name="todo_id",
                type=dynamodb.AttributeType.STRING,
            ),
            billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
            removal_policy=RemovalPolicy.DESTROY,
        )

        CfnOutput(self, "TableName", value=table.table_name)
        CfnOutput(self, "TableArn", value=table.table_arn)
```

In the above code:

1. `dynamodb.Table` creates a DynamoDB table using the CDK L2 construct.
2. `partition_key` defines the primary key for the table. In our case, it is a string attribute named `todo_id`.
3. `BillingMode.PAY_PER_REQUEST` enables on-demand billing, so DynamoDB charges based on requests instead of provisioned capacity.
4. `RemovalPolicy.DESTROY` lets `cdk destroy` delete the table when we clean up.
5. `CfnOutput` prints the table name and ARN after deployment.

!!! warning
    `RemovalPolicy.DESTROY` is useful for tutorials because cleanup is simple. For production tables, use `RemovalPolicy.RETAIN` unless you have a deliberate backup and deletion plan.

## Register the stack in app.py

Now update `app.py` so CDK knows about the new stack:

```python
# filename: app.py

import aws_cdk as cdk

from cdk_app.dynamodb_stack import DynamoDbStack


app = cdk.App()

DynamoDbStack(app, "DynamoDbStack")

app.synth()
```

## Deploy the stack

Before deploying, run `cdk synth` to check that CDK can generate the CloudFormation template:

```bash
cdk synth
```

If there are no errors, deploy the stack:

```bash
cdk deploy
```

After deployment, CDK will show the stack outputs. We will use the `TableName` output to write and read a test item.

## Get the table name

Run the following command to get the DynamoDB table name from the CloudFormation stack output:

```bash
table_name=$(aws cloudformation describe-stacks \
    --stack-name DynamoDbStack \
    --query "Stacks[0].Outputs[?OutputKey=='TableName'].OutputValue" \
    --output text)
```

You can confirm the value by printing it:

```bash
echo "$table_name"
```

## Add an item to the table

Now add a test item using the AWS CLI:

```bash
aws dynamodb put-item \
    --table-name "$table_name" \
    --item '{
        "todo_id": {"S": "todo-1"},
        "title": {"S": "Learn DynamoDB with CDK"},
        "status": {"S": "OPEN"}
    }'
```

In the above command:

1. `todo_id` is the partition key.
2. `title` and `status` are regular attributes.
3. `S` means the attribute value is a string.

## Read the item from the table

Read the item back using `get-item`:

```bash
aws dynamodb get-item \
    --table-name "$table_name" \
    --key '{
        "todo_id": {"S": "todo-1"}
    }'
```

You should see an `Item` object in the response with the same attributes you inserted.

## Add a sort key

Many DynamoDB tables use both a partition key and a sort key. You can add a sort key when you create the table:

```python
# filename: cdk_app/dynamodb_stack.py

table = dynamodb.Table(
    self,
    "TodosTable",
    partition_key=dynamodb.Attribute(
        name="user_id",
        type=dynamodb.AttributeType.STRING,
    ),
    sort_key=dynamodb.Attribute(
        name="todo_id",
        type=dynamodb.AttributeType.STRING,
    ),
    billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
    removal_policy=RemovalPolicy.DESTROY,
)
```

If you use a sort key, every `get-item` request must include both key values:

```bash
aws dynamodb get-item \
    --table-name "$table_name" \
    --key '{
        "user_id": {"S": "user-1"},
        "todo_id": {"S": "todo-1"}
    }'
```

!!! warning
    The table key schema is part of the table design. Changing the partition key or sort key later usually means replacing the table or creating a new one and migrating the data.

## Cleanup

When you are done testing, destroy the stack:

```bash
cdk destroy
```

Because this tutorial used `RemovalPolicy.DESTROY`, the DynamoDB table and the data inside it will be deleted with the stack.

## Next steps

Now that we can create a basic DynamoDB table, the next useful topics are:

1. [Configure point-in-time recovery and deletion protection]({filename}50004010-cdk-dynamodb-pitr-deletion-protection.md).
2. Add a Global Secondary Index.
3. Query items by partition key and sort key.
4. Use the table from a Lambda function when you are ready to connect DynamoDB to an application workflow.

---

# How to install multiple instances of Ubuntu in WSL2

URL: https://cloudbytes.dev/snippets/how-to-install-multiple-instances-of-ubuntu-in-wsl2
Category: Snippets
Published: 2026-04-13
Author: Rehan Haider
Tags: wsl, ubuntu, windows

> A guide to installing multiple fresh instances of Ubuntu in WSL2

[TOC]

**Last Updated:** 2026-04-13

Windows Subsystem for Linux 2 ([WSL2](https://docs.microsoft.com/en-us/windows/wsl/install)) is in its second iteration that uses an actual Linux Kernel, an upgrade of the previous kernel emulator which was called Windows Subsystem for Linux (WSL).

It's a great tool developers who need to Linux for developing and testings their apps. And sometimes, you just want more than one instance of Ubuntu on your machine.


## Installing multiple instances of Ubuntu in WSL2

If you are running windows 10 version 2004 or higher (Build 19041 and above), you can install the latest version of Ubuntu in WSL by running the the below command.

### Step 1: Install the latest version of Ubuntu in WSL2

```powershell
wsl --install
```
This will take care of all the steps required, i.e.

1. It will enable the optional compoenents required on windows (e.g. Windows Virtualisation Platform, etc.)
2. Enable Windows Subsystem for Linux 2 (WSL2)
3. Update the Linux kernel to the latest version
4. Install the default Linux distribution, i.e. latest Ubuntu

![Install WSL](/images/99999967-01-install-wsl.gif)

Once installed, just run `wsl` to open the WSL2 shell, on the first login you will be asked to choose username and password. 

### Step 2: Download the Ubuntu WSL tarball

You can download the Ubuntu WSL tarball from the [Ubuntu WSL2 Image for 24.04](https://cloud-images.ubuntu.com/wsl/releases/noble/current/ubuntu-noble-wsl-amd64-wsl.rootfs.tar.gz) and save it to your local machine. 

You can use your Windows Terminal / Powershell to do so, first run the following command to remove `curl` alias which the mighty intelligent 🤡 Powershell developers have built in

```powershell
Remove-Item alias:curl
```

Then, run the following command to download the Ubuntu WSL tarball. Copy paste the **entire code block** below into your Windows Terminal and run it

```powershell
curl (("https://cloud-images.ubuntu.com",
"wsl/releases/noble/current",
"ubuntu-noble-wsl-amd64-wsl.rootfs.tar.gz") -join "/") `
--output ubuntu-24.04-wsl-root-tar.gz
```


If prompted with a warning, press "Paste anyway" and then press enter to execute. This will download the Ubuntu WSL image tarball to you current directory. 

![curl-wsl-ubuntu](/images/99999967-02-curl-wsl-ubuntu.png)

!!! If you need a specific version of Ubuntu, you can find the list of available versions [here](https://cloud-images.ubuntu.com/wsl/releases/). Just replace the version number in the above command with the version you want to download.

### Step 3: Install the second instance of Ubuntu in WSL2

Just the below command and 

1. Replace the `<Distribution Name>` with the name you want to give, e.g. `ubuntu-2`, 
2. Replace `<Installation Folder>` with the folder where you want to install the second instance of Ubuntu
3. and finally replace `<Ubuntu Tarball path>` with the path of the Ubuntu WSL2 image tarball you downloaded earlier.

```powershell
wsl --import <Distribution Name> <Installation Folder> <Ubuntu WSL2 Image Tarball path>
```

![wsl-install-2](/images/99999967-03-wsl-install-2.png)

After that run, `wsl -l -v` to see the list of distributions installed.

### Step 4: Login to the second instance of Ubuntu in WSL2

To login you need to run: 

```powershell
wsl -d <Distribution Name>
```

You might see an error, ignore it and continue.

![wsl2-login](/images/99999967-04-wsl2-login.png)


### Step 5: Setup user accounts

Notice in the above image that the logged in user is a root account. So let's setup a normal user account.

First, while logged in to the second instance of Ubuntu in WSL2 as root, run the below command, replace `<USERNAME>` with the username of your choice:

```bash
NEW_USER=<USERNAME>
```

Then, run the following command to create the user account and set the password:

```bash
useradd -m -G sudo -s /bin/bash "$NEW_USER"
passwd "$NEW_USER"
```

![create-user-ubuntu](/images/99999967-05-create-user-ubuntu.png)

### Step 6: Configure default user

Next, we need to configure Ubuntu to log in as your new user by default instead of root. 

To do so, run the below command: paste the entire block of code below into your teminal and press enter.

```bash
tee /etc/wsl.conf <<_EOF
[user]
default=${NEW_USER}
_EOF
```

![set-default-user](/images/99999967-06-set-default-user.png)

### Step 7: Login as the new user

First, exit the WSL by running `logout`, then shutfown the second Ubuntu by running

```powershell
wsl --terminate <Distribution Name>
```

Finally, login to the second instance of Ubuntu again:

```powershell
wsl -d <Distribution Name>
```

![login-wsl-new](/images/99999967-07-login-wsl-new.png)

You should now be logged in as the new user.

---

# Upgrade Python to latest version (3.14) on Ubuntu Linux or WSL2

URL: https://cloudbytes.dev/snippets/upgrade-python-to-latest-version-on-ubuntu-linux
Category: Snippets
Published: 2026-04-13
Author: Rehan Haider
Tags: python, ubuntu, wsl

> A complete guide on how to upgrade Python to the latest version (Python 3.14) on Ubuntu Linux and solve associated issues. Also works on WSL2. Can also be used to upgrade to any Python version.

**Last Updated:** 2026-04-13

Ubuntu both Desktop & WSL2 Linux systems come with Python installed by default, but, they are usually not the latest. 


This is a short guide on how to upgrade your Python to the latest version (Python 3.14) on Ubuntu Linux and solve associated issues.
This guide covers two options to upgrade Python to the latest version:

1. [**Using [UV](https://docs.astral.sh/uv)**](#method-1-using-uv) - which has become the de facto standard for Python version management. This is recommended way to work with Python
2. [**Using apt**](#method-2-using-apt) - which is the default package manager for Ubuntu. This is the traditional way to work with Python.

If you want to install a different version of Python, this guide will cover that as well. 

> !!! note " This guide is tested on Ubuntu 24.04 on WSL2, but should work on any Ubuntu version."

## Prerequisites

You need the following

1. `sudo` privileges on your system
2. `curl` installed on your system


## Method 1: Using UV

UV is a Python version manager that can be used to install, manage, and switch between multiple Python versions. An extremely fast Python package and project manager, written in Rust and replaces pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and more.

This is extremely simplified and makes it easy to work with multiple Python versions and projects. Did I mention it is extremely fast? **[10-100x](https://github.com/astral-sh/uv/blob/main/BENCHMARKS.md) faster than `pip`**.

### Step 1: Install UV

For this we first need to install UV, this can be done by running the following command:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

You need to then restart you shell by running the below command:
```bash
source $HOME/.local/bin/env
```

Now, you can verify the installation by running
```bash
uv --version
```

![UV Installed](/images/99999980-01-uv-install.png)


### Step 2: Install Python 3.14

If you want to install python globally, you can do so by running the below command:
```bash
uv install python 3.14
```

This will install the latest version of Python 3.14. To view all available versions, you can run the below command:
```bash
uv python list --only-installed
```
This displays all installed Python versions, including those that are not installed by UV.

![UV Python List](/images/99999980-02-install-python-version.png)


> !!! warning " This doesn't change the system default Python version."

If you want to change the system default Python version (which I warn against as it could result in system instability), you can find the instructions [here](#method-2-using-apt-to-install-python-314).

### Step 3: [Bonus] Python 3.14 using uv

The recommendation is you use uv to create a virtual environment for your project. This is because uv is designed to work with multiple Python versions and projects.

For each project, you can create a virtual environment by running the below command:
```bash
uv venv .venv --python 3.14
```

This will create a virtual environment in the `.venv` folder. You can activate it by running

```bash
source .venv/bin/activate
```

After that you can check where you `python` command is pointing to by running
```bash
which python
```

This should point to the `.venv/bin/python` file.

Now you can run `python --version` and you should see the latest version of Python as the output.

```bash
python --version
```

![use python 3.14 in uv](/images/99999980-03-use-python-uv.png)

You can pin the version of Python in a project by running:

```bash
uv python pin 3.14
```

This create a file named `.python-version` in the root of the project with the version of Python pinned. So if in future you can track which version of Python is used in the project.

### Step 4: Installing packages in the virtual environment

The advantage of using uv is that you can isntall packages in a virtual environment without activating it or affecting the system Python version.

For example, you can install a package by running
```bash
uv pip install <package-name>
```

This will install the package in the virtual environment.

## Method 2: Using apt to install Python 3.14
Ubuntu's default repositories do not contain the latest version of Python, but an open source repository named `deadsnakes` does.


> !!! danger " If you're on Ubuntu Desktop, changing the system-wide Python will break your Gnome terminal and potentially other system tools that depend on the default python3. Using virtual environments (described above) is the better approach for Desktop users. If you still want to proceed, follow the Desktop-specific steps below carefully."

> !!! note " Since WSL doesn't have a GUI, this is largely safe on WSL, but still discouraged."

### Step 1: Prepare to install Python 3.14

First, update your system by running

```bash
sudo apt update && sudo apt upgrade -y
```

```bash
sudo add-apt-repository ppa:deadsnakes/ppa && \
sudo apt update
```

### Step 2: Check if Python 3.14 is available

Check if Python 3.14 is available by running

```bash
apt list | grep python3.14
```

This will produce the below result, if you see python3.14 it means you can install it

![apt list check if python is present](/images/99999980-03-apt_list.png)

If you see something similar to the above, it means you can install Python 3.14.

### Step 2: Install Python 3.14
Now you can install Python 3.14 by running

```bash 
sudo apt install python3.14
```

Now though Python 3.14 is installed, if you check the version of your python by running `python3 --version` you will still see an older version. This is because if you are using Ubuntu Desktop, the default Python is needed by the system and changing it will break your system.

### Step 3: Run Python 3.14

You can run Python 3.14 by running

```bash
python3.14 --version
```

**The right way to run Python 3.14 On Linux Desktops is by using a virtual environment.**

E.g. you can create a new virtual environment by running

```bash
python3.14 -m venv env
```

and activate it by running

```bash
source env/bin/activate
```

Now you can run `python --version` and you should see the latest version of Python as the output.

#### [Extra] Create an alias for Python 3.14
If you really, really, really don't want to type `python3.14` every time you want to run a file, you can create an alias. 

If you are using bash, run
```bash
echo "alias py=/usr/bin/python3.14" >> ~/.bashrc
echo "alias python=/usr/bin/python3.14" >> ~/.bashrc
```


Or, if you have [oh-my-zsh](https://ohmyz.sh/) installed, you can avoid typing out `python3` by running
```bash
echo "alias py=/usr/bin/python3.14" >> ~/.zshrc
echo "alias python=/usr/bin/python3.14" >> ~/.zshrc
```
After restarting your terminal, you can run your Python apps with `py` or `python`.


### Optional: Set Python 3.14 as system default
> !!! danger " If you're on Ubuntu Desktop, changing the system-wide Python will break your Gnome terminal and potentially other system tools that depend on the default python3. Using virtual environments (described above) is the better approach for Desktop users. If you still want to proceed, follow the Desktop-specific steps below carefully."

Changing the default alternatives for Python will break your Gnome terminal. To avoid this, you need to edit the `gnome-terminal` configuration file.

Open the terminal and run:
```bash
sudo nano /usr/bin/gnome-terminal
```
In first line, change `#!/usr/bin/python3` to `#!/usr/bin/python3.12`. Press `Ctrl +X` followed by `enter` to save and exit.

Then save and close the file.


Next, update the default Python by adding both versions to an alternatives by running the below
```bash
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.14 2
```

Now run 
```bash
sudo update-alternatives --config python3
```

Choose the selection corresponding to Python3.14 (if not selected by default). 
![Python alternatives on linux](/images/99999980-04-alternatives.png)

Now run `python3 --version` again and you should see the latest Python as the output.

#### [Extra] Fix pip and disutils errors

Installing the new version of Python will break `pip` as the `distutils` for Python3.13 is not installed yet.

#### [Extra] Fix Python3-apt errors
Running `pip` in terminal will not work, as the current pip is not compatible with Python3.13 and python3-apt will be broken, that will generate an error like
```text
Traceback (most recent call last):   
    File "/usr/lib/command-not-found", line 28, in <module>     
        from CommandNotFound import CommandNotFound   
    File "/usr/lib/python3/dist-packages/CommandNotFound/CommandNotFound.py", line 19, in <module>     
        from CommandNotFound.db.db import SqliteDatabase   
    File "/usr/lib/python3/dist-packages/CommandNotFound/db/db.py", line 5, in <module>     
        import apt_pkg ModuleNotFoundError: No module named 'apt_pkg'
```

To fix this first remove the current version of python3-apt by running
```bash
sudo apt remove --purge python3-apt
```

Then do some cleanup
```bash
sudo apt autoclean
```

!!! danger "DO NOT RUN `sudo apt autoremove` as it will remove several packages that are required. This may break your system if you're using GUI, if you're on WSL2 you can proceed."

Finally, reinstall `python3-apt` by running

```bash
sudo apt install python3-apt
```

#### [Extra] Install pip 

Running `pip`  will still throw an error `pip: command not found`. We need to install the latest version of pip compatible with Python 3.14. 

Now you can install `pip` by running

```bash
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
sudo python3.14 get-pip.py
```

> If you get an error like `bash: curl: command not found` then you need to install curl first by running `sudo apt install curl`

Now you can run `pip` and you should see the output of `pip --version`

All should be done now. It is complicated, but this is how you update Python to latest version.

## Bonus: Install a different version of Python
If you want to install a different version of Python, you can use the below command:

```bash
uv install python <version>
```

This will install the specified version of Python.

---

# Running a Web Server in a Private Subnet secured

URL: https://cloudbytes.dev/aws-academy/running-a-web-server-in-a-private-subnet-secured
Category: AWS Academy
Published: 2025-06-20
Author: Rehan Haider
Tags: aws, cdk, python

> Guide to running a Web Server in a Private Subnet with AWS CDK in Python.

The traditional way to run applications in Data Centres were to create a 3-tier architecture with:

1. **Web Tier**: That would typically be placed in a DMZ (Demilitarized Zone) that allowed the server to be exposed to internet
2. **Application Tier & Database Tier**: That would typically be placed behind firewalls that was not exposed to internet and required authentication

![Traditional 3-Tier Architecture](/images/50003100-01-traditional-dc-3-tier.png)

This was one of the weakest links in security posture of any organisation. However, with the advent of cloud, it did not need to be that way.

In this article, we will discuss a simplistic solution to this problem.





## Modern 3-Tier Architecture

We overcome the traditional limitation by creating a 4-tier architecture with all the servers behind the "firewall" in private subnets fronted by a external facing load balancer. We use the public subnet to deploy NAT Gateways to allow the servers access to external services such as updates, security patches, APIs, etc.

In the below diagram, we have a VPC with 3 private subnets and a public subnet. The public subnet is fronted by a external facing load balancer. The private subnet is where the web server is running. The web server is not exposed to the internet, it doesn't even have a public IP address.


![Modern 3-Tier Architecture](/images/50003100-02-modern-3-tier-archtiecture.png)

### Accessing the Web Server

1. **Web Traffic**: The web traffic on ports 80 and 443 are routed through the load balancer, the load balancer acting like a reverse proxy, forwards the request to the web server in the private subnet. This avoids the need for the web server to have a public IP address.
2. **SSH Access**: There are 3 ways we can have management access to the web server, in the order of security:
    - **SSH Key Pair**: This is the most secure way to access the web server. We can use the SSH key pair to access the web server.
    - **SSH Agent Forwarding**: This is a secure way to access the web server. We can use the SSH agent forwarding to access the web server.
    - **Bastion Host with SSH Forwarding**: A separate bastion host in a public subnet can be used to allow SSH access to the web server where access can be limited to specific IP addresses.

The easiest of these is to use the bastion host in the public subnet to allow SSH access to the web server.

## Scenario

To simplify the demonstration, we will limit ourselves to the web-tier only. So if we are able to access the webserver from internet while the server is in a private subnet with no public IP address, we have achieved our goal.

![Project Scenario](/images/50003100-03-project-scenario.png)

## Running the Web Server in a Private Subnet

### 1. Prerequisites

1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md). 
2. If needed [create a new CDK application]({filename}50000020-cdk-new-app.md).

### 2. Create the VPC

First we create a 3-tier VPC with 2 private subnets and 1 isolated subnet. We can modify the approach from the previous article on [how to create a 3-tier VPC with public, private and isolated subnets]({filename}50003000-cdk-vpc-3-tier.md) to create a VPC with 2 private subnets and 1 isolated subnet.

```python
# filename: cdk_app/my_stack.py
from aws_cdk import (
    Stack,
    aws_ec2 as ec2,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Define the VPC with three tiers
        vpc = ec2.Vpc(
            self,
            "MyDemoVPC",
            max_azs=2,  # Default is all AZs in the region
            subnet_configuration=[
                # 👇🏽 Public Subnets
                ec2.SubnetConfiguration(
                    name="Public",
                    subnet_type=ec2.SubnetType.PUBLIC,
                    cidr_mask=24,
                ),
                # 👇🏽 Private Subnets (Web Server)
                ec2.SubnetConfiguration(
                    name="Private",
                    subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS,
                    cidr_mask=24,
                ),
                # 👇🏽 Private Subnets (App Tier)
                ec2.SubnetConfiguration(
                    name="Private",
                    subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS,
                    cidr_mask=24,
                ),
                # 👇🏽 Isolated Subnets (Database Tier)
                ec2.SubnetConfiguration(
                    name="Isolated",
                    subnet_type=ec2.SubnetType.PRIVATE_ISOLATED,
                    cidr_mask=24,
                ),
            ],
        )
```

This will create the items in below resource map:

![Resource Map](/images/50003100-04-vpc-resource-map.png)


## 3. Create the Web Servers

We will create a web server in the private subnet. We will use the `ec2.Instance` class to create the web server.

```python
# filename: cdk_app/my_stack.py

```

---

# Compile Expo React Native application for Android

URL: https://cloudbytes.dev/snippets/compile-expo-react-native-application-for-android
Category: Snippets
Published: 2025-05-21
Author: Rehan Haider
Tags: android, react-native, javascript, typescript

> A guide to compiling a React Native application built using Expo framework for Android

[React Native](https://reactnative.dev/docs/getting-started) is one of the most popular if not the most popular tool used to build Android applications at present and developers typically use [Expo framework](https://docs.expo.dev/get-started/introduction/) to build React Native applications. 

However, [Expo documentation](https://docs.expo.dev/guides/local-app-development/), for various reasons, is surreptitiously silent about how to compile your Expo React Native application for Android. 

In this guide, we will see how to compile your Expo React Native application for Android.

You need the following to complete this guide:
1. Setup a React Native application using Expo framework.
2. Install Android SDK
3. Install Java JDK
4. Configure Android SDK and Java JDK in your system environment variables
5. Build the React Native app


## 1. Setup a React Native application using Expo framework
If you have not already setup a React Native application using Expo framework, you can do so by following the steps below:

1. Open a terminal and run the following command to create an expo project:

```bash
npx create-expo-app@latest
```

## 2. Install Android SDK

You can install Android SDK using [these instructions](https://developer.android.com/studio).

## 3. Install Java JDK

You can install Java JDK using the following command:

```bash
sudo apt install openjdk-17-jdk
```


## 4. Configure Android SDK and Java JDK in your system environment variables

Add the below lines to your `~/.bashrc` file:

```bash
export ANDROID_HOME=$HOME/Library/Android/sdk 
export PATH=$PATH:$ANDROID_HOME/emulator 
export PATH=$PATH:$ANDROID_HOME/platform-tools
```

## 5. Build the React Native app

1. First start the prebuild process by running the following command:

```bash
npx expo prebuild
```

2. Now navigate to the `android` directory:

```bash
cd android
```

3. Now run the following command to build the app:

```bash
./gradlew assembleRelease
```

4. The compiled APK file will be available in the `android/app/build/outputs/apk/release` directory.

---

# Add new users with SSH access to EC2 instance

URL: https://cloudbytes.dev/snippets/add-new-users-with-ssh-access-to-ec2-instance
Category: Snippets
Published: 2025-04-30
Author: Rehan Haider
Tags: aws, ec2

> How to new additional user with keypair for SSH access to EC2 instance

How do you deal with situations such as when you need to provide somebody else with SSH access to your EC2 instance? You can create a new user and add a new keypair for that user. This guide will show you how to do that.

## Pre-requisites

You need an EC2 instance running with SSH access. You can create a new EC2 instance through the [AWS Console]({filename}18750100-create-ec2-instance-console.md) or the [AWS CLI]({filename}18750200-create-ec2-instance-using-cli.md).


## Login to the EC2 instance
You can login to the EC2 instance using the SSH keypair you created when launching the instance. For example, if you created a keypair named `my-key-pair`, you can login to the instance using the following command:

```bash
ssh -i <my-key-pair.pem> <username>@<public-ip-address>
```

Replace `<my-key-pair.pem>` with the path to your keypair file, `<username>` with the username of the instance (e.g., `ec2-user` for Amazon Linux, or `ubuntu` for Ubuntu), and `<public-ip-address>` with the public IP address of your EC2 instance.


## Create a new user
Once you are logged in to the EC2 instance, you can create a new user using the following command:

```bash
sudo adduser <new-username>
```

Replace `<new-username>` with the desired username for the new user. For example, if you want to create a user named `john`, you would run:

```bash
sudo adduser john
```
## Create a new SSH keypair

1. First, let's create a folder for the new user to store their SSH keys. You can do this by running the following command:

```bash
sudo mkdir /home/<new-username>/.ssh
```
2. Next, set the owner of the `.ssh` directory to the new user:

```bash
sudo chown <new-username>:<new-username> /home/<new-username>/.ssh
```

3. Next, set the correct permissions for the `.ssh` directory:

```bash
sudo chmod 700 /home/<new-username>/.ssh
```

4. You can create a new SSH keypair for the new user using the following command:

```bash
sudo ssh-keygen -t rsa -b 2048 -f /home/<new-username>/.ssh/id_rsa
```
Replace `<new-username>` with the username you created in the previous step. This command will create a new SSH keypair with a 2048-bit RSA key and save it to the specified location.
The private key will be saved as `/home/<new-username>/.ssh/id_rsa` and the public key will be saved as `/home/<new-username>/.ssh/id_rsa.pub`. You can change the file name and location as needed.

## Add public key to the authorized_keys file

1. Next, you need to add the public key to the `authorized_keys` file for the new user. You can do this by running the following command:
```bash
sudo cp /home/<newuser>/.ssh/id_rsa.pub /home/<newuser>/.ssh/authorized_keys
```

2. Set the correct ownership for the `authorized_keys` file:

```bash
sudo chown <newuser>:<newuser> /home/<newuser>/.ssh/authorized_keys
```

3. Set the correct permissions for the `authorized_keys` file:

```bash
sudo chmod 600 /home/<newuser>/.ssh/authorized_keys
```

## Ensure correct permissions for the .ssh directory

1. Set the owner of the `.ssh` directory to the new user:

```bash
sudo chown -R <newuser>:<newuser> /home/<newuser>/.ssh
```

2. Set the correct permissions for the `.ssh` directory:

```bash
sudo chmod 700 /home/<newuser>/.ssh
```

## Get the private key

The private key is saved in the file `/home/<newuser>/.ssh/id_rsa`. You can download this file to your local machine using `scp` or any other file transfer method. Make sure to keep the private key secure and do not share it with anyone else.

---

# Creating a 3-Tier Network Architecture VPC with AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/creating-a-3-tier-network-architecture-vpc-with-aws-cdk-in-python
Category: AWS Academy
Published: 2025-03-24
Author: Rehan Haider
Tags: aws, cdk, python

> Guide to creating a 3-Tier Network Architecture VPC with a public, private, and isolated subnet using AWS CDK in Python.

3-tier architecture is an almost ubiquitous design patter in software development. It separates the application into three layers:

1. **Presentation Layer (Web Tier)**: This is the front-end layer that users interact with. It typically consists of web servers, load balancers, and other components that handle user requests.
2. **Application Layer (Business Logic Tier)**: This layer contains the core functionality of the application. It processes user requests, performs calculations, and interacts with the database layer.
3. **Data Layer (Database Tier)**: This layer is responsible for data storage and management.

![3-tier-architecture](/images/50003000-01-cdk-vpc-3-tier.png)


From infrastructure perspective, this architecture is replicated for better management and security. In AWS, this is typically done using a VPC with three subnets:

1. **Public Subnet**: This subnet is accessible from the internet and typically contains resources like load balancers and web servers.
2. **Private Subnet**: This subnet is not directly accessible from the internet and typically contains application servers and other resources that do not need to be publicly accessible. Typically, the servers in this subnet can access the internet through a NAT Gateway.
3. **Isolated Subnet**: This subnet is completely isolated from the internet and typically contains databases and other resources that should neither be accessible from the internet nor should have acccess to the internet.


In this article, we will create a 3-tier VPC architecture using AWS CDK in Python.

## 3 Tier VPC Architecture
On AWS, the 3-tier VPC architecture looks like this:

![3-tier-architecture](/images/50003000-02-cdk-3-tier-vpc.png)


## Creating a 3-Tier VPC with AWS CDK in Python

We can use default constructs to create a VPC with public, private, and isolated subnets. The following code creates a VPC with 3 subnets:

```python
# filename: cdk_app/my_stack.py
from aws_cdk import (
    Stack,
    aws_ec2 as ec2,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Define the VPC with three tiers
        vpc = ec2.Vpc(
            self,
            "MyDemoVPC",
            max_azs=2,  # Default is all AZs in the region
            subnet_configuration=[
                # 👇🏽 Public Subnets
                ec2.SubnetConfiguration(
                    name="Public",
                    subnet_type=ec2.SubnetType.PUBLIC,
                    cidr_mask=24,
                ),
                # 👇🏽 Private Subnets
                ec2.SubnetConfiguration(
                    name="Private",
                    subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS,
                    cidr_mask=24,
                ),
                # 👇🏽 Isolated Subnets
                ec2.SubnetConfiguration(
                    name="Isolated",
                    subnet_type=ec2.SubnetType.PRIVATE_ISOLATED,
                    cidr_mask=24,
                ),
            ],
        )
```

Each subnet type is defined using the `SubnetConfiguration` class. 

The `subnet_type` parameter specifies the type of subnet:
- `ec2.SubnetType.PUBLIC`: Public subnet that is accessible from the internet.
- `ec2.SubnetType.PRIVATE_WITH_EGRESS`: Private subnet that can access the internet through a NAT Gateway.
- `ec2.SubnetType.PRIVATE_ISOLATED`: Isolated subnet that is not accessible from the internet and does not have internet access.

The `cidr_mask` parameter specifies the size of the subnet. In this example, we are using a CIDR mask of 24, which means each subnet will have 256 IP addresses (254 usable IPs).

The `max_azs` parameter specifies the maximum number of Availability Zones to use for the VPC. By default, CDK uses all Availability Zones in the region. In this example, we are using 2 Availability Zones.

The `ec2.Vpc` construct automatically creates the necessary route tables, internet gateways, and NAT gateways for the VPC based on the subnet types specified.

## Deploying the CDK Stack
To deploy the CDK stack, run the following command in the terminal:

```bash
cdk deploy
```
This command will deploy the stack to your AWS account. You can view the resources created in the AWS Management Console under the VPC section.

## Conclusion
When you navigate to the VPC section in the AWS Management Console, you can see details of all the resources that have been created and configured in the VPC Resource Map.

![VPC resource map](/images/50003000-03-cdk-vpc-resource-map.png)


However, this is a pre-configured way to create a VPC, if you want to customise the VPC further e.g.  CIDR ranges of your own choice, lower level constructs can be used for that purpose.

---

# Running Lambda Functions in a VPC with AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/running-lambda-functions-in-a-vpc-with-aws-cdk-in-python
Category: AWS Academy
Published: 2025-03-24
Author: Rehan Haider
Tags: aws, cdk, python

> Running Lambda Functions in a VPC with AWS CDK in Python

We have created several lambda functions that are running in AWS owned default VPC. However, this means if you need to access any resources in your AWS account, the access is over the internet.

Hence, in several scenarios, you may need to run your lambda functions in a VPC, e.g.:

1. **Access to private resources such as EC2, RDS, etc**: If your Lambda function needs to access private resources inside a VPC (such as an RDS database that is not publicly accessible, an ElastiCache cluster, or an EC2 instance), you must place it inside the same VPC.
2. **Connecting to internal APIs or services**: If you have private APIs or services hosted inside the VPC, your Lambda function needs to be in the same VPC to call them.
3. **Using VPC Interface Endpoints (AWS PrivateLink)**: If your Lambda function needs to interact with AWS services like S3, DynamoDB, or SQS using VPC interface endpoints (AWS PrivateLink) for enhanced security (instead of going over the public internet).

This article will show you how to create a lambda function that is running in a VPC using AWS CDK in Python.


## Prerequisites

1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md). 
2. If needed [create a new CDK application]({filename}50000020-cdk-new-app.md).
3. We need a VPC with private subnet. You can follow this [article]({filename}50003000-cdk-vpc-3-tier.md) to create a VPC with private subnet.



## Creating a Lambda function in a VPC

Before we create a Lambda function, we need appropriate VPC and subnets. If you need to create a VPC, you can follow this [article]({filename}50003000-cdk-vpc-3-tier.md) to create a VPC with private subnet.

We will create the lambda function in the private subnet of the VPC. The following code creates a Lambda function in a VPC. 

```python
# filename: cdk_app/my_stack.py
from aws_cdk import (
    Stack,
    aws_ec2 as ec2,
    aws_lambda_python_alpha as python_lambda,
    aws_lambda as lambda_,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 Enter the appropriate values
        subnet_id = "subnet-00000000000000000"
        vpc_id = "vpc-00000000000000000"


        vpc = ec2.Vpc.from_lookup(
            self,
            "VPC",
            vpc_id=vpc_id,
        )

        # Get the specific subnet by its ID
        private_subnet = ec2.Subnet.from_subnet_id(self, "PrivateSubnet", subnet_id)

        # Define the Lambda function within the specific subnet
        my_vpc_lambda = python_lambda.PythonFunction(
            self,
            "MyVpcLambdaFunction",
            entry="cdk_app/fn",
            runtime=lambda_.Runtime.PYTHON_3_12,
            index="index.py",
            handler="handler",
            vpc=vpc,
            vpc_subnets=ec2.SubnetSelection(subnets=[private_subnet]),
        )

```


### Explanation
- **VPC**: We are using the `from_lookup` method to get the VPC by its ID. This allows us to reference an existing VPC in your AWS account.
- **Subnet**: We are using the `from_subnet_id` method to get the subnet by its ID. This allows us to reference an existing subnet in your AWS account.
- **Lambda Function**: We are using the `PythonFunction` construct to create a Lambda function. The `vpc` parameter specifies the VPC in which the Lambda function will run, and the `vpc_subnets` parameter specifies the subnets in which the Lambda function will run.
- **Entry**: The `entry` parameter specifies the directory where the Lambda function code is located. In this case, it is in the `cdk_app/fn` directory.
- **Runtime**: The `runtime` parameter specifies the runtime environment for the Lambda function. In this case, it is Python 3.12.
- **Index**: The `index` parameter specifies the name of the file that contains the Lambda function code. In this case, it is `index.py`.
- **Handler**: The `handler` parameter specifies the name of the function that will be called when the Lambda function is invoked. In this case, it is `handler`.

You can deploy the stack using the following command:

```bash 
cdk deploy
```
This will create a Lambda function in the specified VPC and subnet. You can check the AWS Management Console to verify that the lambda function is created correctly and is running in the specified VPC and subnet in the VPC section of Configuration.

![Lambda in VPC](/images/50002080-01-cdk-lambda-in-vpc.png)

---

# Using multiple environments AWS CLI and profiles with CDK

URL: https://cloudbytes.dev/aws-academy/using-multiple-environments-aws-cli-and-profiles-with-cdk
Category: AWS Academy
Published: 2025-01-12
Author: Rehan Haider
Tags: aws, cdk, python

> How to use multiple environments, AWS Accounts and profiles with AWS CDK

So far we've been using the default `cdk deploy` command that uses the default AWS CLI profile that you have configured. However, AWS CLI and AWS CDK both supports using multiple profiles and environments. This is useful when you have multiple AWS accounts and you want to deploy your CDK stacks to different accounts.


## Environments

When we intantiate a stack in CDK, we can pass an `env` parameter that specifies the environment in which the stack will be deployed. This environment contains the AWS account and region where the stack will be deployed. 

E.g., working off a [new CDK app]({filename}50000020-cdk-new-app.md), we typically have the following code in the `app.py` file:

```python
import aws_cdk as cdk

from cdk_app.my_stack import MyStack

app = cdk.App()
my_stack = MyStack(
    app,
    "MyStack",
) #👈🏽 No env define

app.synth()
```

In the above code, we are not specifying the environment in which the stack will be deployed. This means that the stack will be deployed to the default AWS CLI profile that you have configured.

To specify the environment, we can pass an `env` parameter to the `MyStack` constructor:

```python
import aws_cdk as cdk

from cdk_app.my_stack import MyStack


env = {
    "account": "123456789012",
    "region": "us-east-1",
} #👈🏽 Define the environment


app = cdk.App()
my_stack = MyStack(
    app,
    "MyStack",
    env=env,
)

app.synth()
```

In the above code, we are specifying the environment in which the stack will be deployed. The `env` parameter is a dictionary that contains the `account` and `region` keys. The `account` key specifies the AWS account where the stack will be deployed, and the `region` key specifies the AWS region where the stack will be deployed.

## Using multiple profiles

To use multiple profiles, you can utilise `--profile`` flag to configure and use a different profile.

E.g. while configuring a new profile:

```bash
aws configure --profile my_profile
```

To use this profile with CDK, you can pass the `--profile` flag to the `cdk deploy` command:

```bash
cdk deploy --profile my_profile
```

This will deploy the stack using the `my_profile` profile that you have configured.

---

# Configure log retention and removal policy for Lambda function using AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/configure-log-retention-and-removal-policy-for-lambda-function-using-aws-cdk-in-python
Category: AWS Academy
Published: 2025-01-09
Author: Rehan Haider
Tags: aws, cdk, python

> Learn how to configure CloudWatch logs retention and removal/deletion policy for a Lambda function using AWS CDK in Python

Each lambda invocation generates execution logs. By default, these logs are stored indefinitely in CloudWatch Logs. This means for a busy application you could have hundreds of thousans of log streams and log groups that are stored indefinitely. 

While in some scenarios this may be needed for certain compliances, but 9/10 times you don't need to store logs indefinitely. In this article, we will look at how to configure log retention for a Lambda function using AWS CDK in Python.

## Prerequisites
1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md). 
2. If needed [create a new CDK application]({filename}50000020-cdk-new-app.md).


## Configuring log retention for a Lambda function

To do so, we first create a simple lambda function. We will need to store the function in a variable so that we can configure the log retention for it.

```python
# filename: cdk_app/my_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
    aws_logs as logs,
    Duration,
    RemovalPolicy,
)
from constructs import Construct

class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 Create a Lambda function
        my_lambda = _lambda.Function(
            self,
            "MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="index.handler",
            code=_lambda.Code.from_inline("def handler(event, context): return 'Hello, World!'"),
            timeout=Duration.seconds(10),
        )
```

Next, we will configure the log retention for the lambda function. We will use the `LogGroup` class from the `aws_logs` module to configure the log retention. 

In the background, CDK creates another lambda function that deletes the older logs based on the retention policy. To do so it requires the name of the log group. The log group name is in the format `/aws/lambda/<function_name>`. We can get the function name using `my_lambda.function_name`.


```python
# filename: cdk_app/my_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
    aws_logs as logs,
    Duration,
    RemovalPolicy,
)
from constructs import Construct

class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 Create a Lambda function
        my_lambda = _lambda.Function(
            self,
            "MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_12,
            handler="index.handler",
            code=_lambda.Code.from_inline("def handler(event, context): return 'Hello, World!'"),
            timeout=Duration.seconds(10),
        )

        # 👇🏽 Create a log group for the Lambda function
        my_log_group = logs.LogGroup(
            self,
            "MyLogGroup",
            log_group_name=f"/aws/lambda/{my_lambda.function_name}", # 👈🏽 This is the log group name
            retention=logs.RetentionDays.ONE_WEEK,
            removal_policy=RemovalPolicy.DESTROY,
        )
```

In the above code, we create a log group for the lambda function. We set the retention policy to `ONE_WEEK`. This means that the logs will be retained for one week. After one week, the logs will be deleted. 

This also has the removal policy set to `DESTROY`. This means that when the stack is deleted, the log group will be deleted as well.

Now deploy the stack using the following commands:

```bash
cdk deploy
```

Once the stack is deployed, you can check the log group in the CloudWatch console. You will see that the log group has a retention policy of one week.

![cloudwatch log group retention policy](/images/50002100-01-cloudwatch-retention.png)

---

# Granting Lambda function permission to access DynamoDB using AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/granting-lambda-function-permission-to-access-dynamodb-using-aws-cdk-in-python
Category: AWS Academy
Published: 2025-01-03
Author: Rehan Haider
Tags: aws, cdk, python, dynamodb, lambda

> Learn how to grant Lambda permissions to access DynamoDB using AWS CDK in Python

Part of the serverless design pattern is to have a Lambda function that interacts with a DynamoDB table. The DynamoDB can be access only through APIs using appropriate credentials. In this article, we will look at how to grant a Lambda function permission to access a DynamoDB table using AWS CDK in Python.


## Prerequisites

1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md). 
2. If needed [create a new CDK application]({filename}50000020-cdk-new-app.md).


## Granting Lambda function permissions to access DynamoDB

We will need to do the following:

1. Create an DynamoDB table and insert some data.
2. Create a Lambda function.
3. Grant the Lambda function permissions to access the DynamoDB table.

### 1. Create an DynamoDB table

First, let's create an DynamoDB table in the stack.

```python
# filename: cdk_app/my_stack.py
from aws_cdk import (
    Stack,
    aws_dynamodb as dynamodb,
    RemovalPolicy,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 Create a DynamoDB table
        table = dynamodb.Table(
            self,
            "MyTable",
            partition_key={"name": "pk", "type": dynamodb.AttributeType.STRING},
            sort_key={"name": "sk", "type": dynamodb.AttributeType.STRING},
        )

```

### 2. Create a Lambda function

We can create a simple Lambda function using any of the methods we have discussed in the previous posts. For this example we will use the CDK provided `PythonFunction` feature [that allows us to specify the python dependencies in AWS Lambda with ease]({filename}50002030-cdk-fn-lambda-python-deps.md).

**Step 1**: First we install the `aws-cdk.aws-lambda-python-alpha` module:

```bash
pip install aws-cdk.aws-lambda-python-alpha
```

**Step 2**: Then we create a Lambda function. We can also pass the name of the bucket to Lambda through the environment variables.

```python
# filename: cdk_app/my_stack.py
from aws_cdk import (
    Stack,
    aws_dynamodb as dynamodb,
    aws_lambda_python_alpha as python_lambda, # 👈🏽 This is the python lambda construct
    aws_lambda as lambda_, # 👈🏽 This is needed for runtime
    RemovalPolicy,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        table = dynamodb.Table(
            self,
            "MyTable",
            partition_key={"name": "pk", "type": dynamodb.AttributeType.STRING},
            sort_key={"name": "sk", "type": dynamodb.AttributeType.STRING},
            removal_policy=RemovalPolicy.DESTROY,
        )

        # 👇🏽 Create a Lambda function
        fn = python_lambda.PythonFunction(
            self,
            "MyDynamoDBFunction",
            entry="cdk_app/fn",
            runtime=lambda_.Runtime.PYTHON_3_12,
            index="index.py",
            handler="handler",
        )
```

**Step 3**: Now we write the Lambda function code:

```python
# filename: cdk_app/fn/index.py

import requests

def handler(event, context):
    response = requests.get("https://jsonplaceholder.typicode.com/todos/1")

    return {"statusCode": 200, "body": response.json()}
```


**Step 4**: Add the Python dependencies in the `cdk_app/fn/requirements.txt` file:

```txt
requests
boto3
```


### 3. Grant the Lambda function permissions to access the S3 bucket

Finally, we need to grant the Lambda function necessary permissions to access the DynamoDB table.

```python
# filename: cdk_app/my_stack.py
from aws_cdk import (
    Stack,
    aws_dynamodb as dynamodb,
    aws_lambda_python_alpha as python_lambda,
    aws_lambda as lambda_,
    RemovalPolicy,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        table = dynamodb.Table(
            self,
            "MyTable",
            partition_key={"name": "pk", "type": dynamodb.AttributeType.STRING},
            sort_key={"name": "sk", "type": dynamodb.AttributeType.STRING},
            removal_policy=RemovalPolicy.DESTROY,
        )

        fn = python_lambda.PythonFunction(
            self,
            "MyDynamoDBFunction",
            entry="cdk_app/fn",
            runtime=lambda_.Runtime.PYTHON_3_12,
            index="index.py",
            handler="handler",
        )

        # 👇🏽 Grant the Lambda function access to the DynamoDB table
        table.grant_read_write_data(fn)
```

In the above code, we use the `grant_read_write_data` method to grant the Lambda function read and write access to the DynamoDB table.

Alternatively, you can use the `grant_read` or `grant_write` methods to grant only read or write access respectively.


## Accessing DynamoDB table using the Lambda function


Now that we have granted the Lambda function permissions to access the DynamoDB table, we can read and write data to the table. To do so, we need to use the `boto3` library to interact with the DynamoDB table.

```python
# filename: cdk_app/fn/index.py

import os
import boto3
import requests

dynamodb = boto3.resource("dynamodb")
# 👇🏽 get the table name from the environment variables
table_name = os.environ["TABLE_NAME"]
table = dynamodb.Table(table_name)

def handler(event, context):

    response = requests.get("https://jsonplaceholder.typicode.com/todos/1")

    # 👇🏽 write the response to a file in the bucket
    table.put_item(Key={"pk": "TODO", "sk": "1"}, Item={"data": response.json()})
    
    # 👇🏽 read the file and send the contexts as response
    stored_response = table.get_item(Key={"pk": "TODO", "sk": "1"})


    return {"statusCode": 200, "body": stored_response}
```

In the above code, we use the `boto3` library to interact with the DynamoDB table. We first write the response from the API to a file in the bucket and then read the file and send the contents as a response. This method can be modified based on your exact use case.


## Testing the Lambda function

To test the Lambda function we need its name. We can get the name of the lambda function from the stack output.

First list the stack outputs:

```bash
function_name=$(aws lambda list-functions --query "Functions[?contains(FunctionName, 'MyDynamoDBFunction')].[FunctionName]" --output text)
```

Then invoke the lambda function:

```bash
aws lambda invoke \
    --function-name $function_name \
    --cli-binary-format raw-in-base64-out \
    --payload '{ "key": "value" }' \
    /dev/stdout | jq
```

!!! note
    The `jq` command is used to format the JSON output. If you don't have it installed, you can install it by running `brew install jq` on macOS or `apt-get install jq` on Linux.


This will invoke the Lambda function and print the response in the terminal.

![invoke-lambda-function](/images/50002070-01-lambda-response.png)

---

# Granting S3 permissions to a Lambda function using AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/granting-s3-permissions-to-a-lambda-function-using-aws-cdk-in-python
Category: AWS Academy
Published: 2025-01-03
Author: Rehan Haider
Tags: aws, cdk, python

> Learn how to grant S3 permissions to a Lambda function and accessing files from an S3 bucket using AWS CDK in Python

One of the most common serverless use cases is to have a Lambda function that processes files stored in an S3 bucket. However, typically you would keep the contents of your S3 bucket private and limit access through appropriate means. One of those means is you can create an IAM role that Lambda assumes when it runs and grant it permissions to access the S3 bucket.

In this article, we will look at how to:
1. Grant S3 read and write permissions to a Lambda function using AWS CDK in Python.
2. Read/write files from an S3 bucket using the Lambda function.

## Prerequisites

1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md). 
2. If needed [create a new CDK application]({filename}50000020-cdk-new-app.md).


## Granting S3 permissions to a Lambda function

We will need to do the following:

1. Create an S3 bucket.
2. Create a Lambda function.
3. Grant the Lambda function permissions to access the S3 bucket.

### 1. Create an S3 bucket

First, let's create an S3 bucket in the stack.

```python
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 create a bucket with random name
        bucket = s3.Bucket(
            self,
            "MyBucket",
            removal_policy=RemovalPolicy.DESTROY,  # 👈🏽 Delete the bucket with stack
            auto_delete_objects=True,  # 👈🏽 delete all objects before deleting bucket
        )
```

### 2. Create a Lambda function

We can create a simple Lambda function using any of the methods we have discussed in the previous posts. For this example we will use the CDK provided `PythonFunction` feature [that allows us to specify the python dependencies in AWS Lambda with ease]({filename}50002030-cdk-fn-lambda-python-deps.md).

**Step 1**: First we install the `aws-cdk.aws-lambda-python-alpha` module:

```bash
pip install aws-cdk.aws-lambda-python-alpha
```

**Step 2**: Then we create a Lambda function. We can also pass the name of the bucket to Lambda through the environment variables.

```python
#filename: cdk_app/lambda_stack.py
# filename: cdk_app/my_stack.py
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    aws_lambda_python_alpha as python_lambda,
    aws_lambda as lambda_,
    RemovalPolicy,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        bucket = s3.Bucket(
            self,
            "MyBucket",
            removal_policy=RemovalPolicy.DESTROY,
        )


        # 👇🏽 Create a Lambda function
        fn = python_lambda.PythonFunction(
            self,
            "MyS3Function",
            entry="cdk_app/fn",
            runtime=lambda_.Runtime.PYTHON_3_12,
            index="index.py",
            handler="handler",
            environment={"BUCKET_NAME": bucket.bucket_name},
        )


```

**Step 3**: Now we write the Lambda function code:

```python
# filename: cdk_app/fn/index.py

import requests

def handler(event, context):
    response = requests.get("https://jsonplaceholder.typicode.com/todos/1")

    return {"statusCode": 200, "body": response.json()}
```


**Step 4**: Add the Python dependencies in the `cdk_app/fn/requirements.txt` file:

```txt
requests
boto3
```


### 3. Grant the Lambda function permissions to access the S3 bucket

Finally, we need to grant the Lambda function permissions to access the S3 bucket.

```python
# filename: cdk_app/my_stack.py
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    aws_lambda_python_alpha as python_lambda,
    aws_lambda as lambda_,
    RemovalPolicy,
)
from constructs import Construct


class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        bucket = s3.Bucket(
            self,
            "MyBucket",
            removal_policy=RemovalPolicy.DESTROY,
        )

        fn = python_lambda.PythonFunction(
            self,
            "MyS3Function",
            entry="cdk_app/fn",
            runtime=lambda_.Runtime.PYTHON_3_12,
            index="index.py",
            handler="handler",
            environment={"BUCKET_NAME": bucket.bucket_name},
        )

        # 👇🏽 Grant the Lambda function access to the S3 bucket
        bucket.grant_read_write_data(fn)

```

In the above code, we use the `grant_read_write_data` method to grant the Lambda function read and write access to the S3 bucket.

Alternatively, you can use the `grant_read` or `grant_write` methods to grant only read or write access respectively.


## Accessing files from an S3 bucket using the Lambda function


Now that we have granted the Lambda function permissions to access the S3 bucket, we can read and write files from the bucket. To do so, we need to use the `boto3` library to interact with the S3 bucket.

```python
# filename: cdk_app/fn/index.py

import os
import boto3
import requests
import json

s3 = boto3.client("s3")


def handler(event, context):
    # 👇🏽 get the bucket name from the environment variables
    bucket_name = os.environ["BUCKET_NAME"]

    response = requests.get("https://jsonplaceholder.typicode.com/todos/1")

    # 👇🏽 write the response to a file in the bucket
    # Convert the JSON response to a string before storing
    json_string = json.dumps(response.json())
    s3.put_object(Bucket=bucket_name, Key="todos-1.json", Body=json_string)

    # 👇🏽 list all the objects in the bucket
    response = s3.list_objects_v2(Bucket=bucket_name)

    # 👇🏽 read the file and parse the contents as JSON
    stored_response = s3.get_object(Bucket=bucket_name, Key="todos-1.json")
    stored_data = json.loads(stored_response["Body"].read().decode("utf-8"))

    return {"statusCode": 200, "body": json.dumps(stored_data)}
```

In the above code, we use the `boto3` library to interact with the S3 bucket. We first write the response from the API to a file in the bucket and then read the file and send the contents as a response. This method can be modified based on your exact use case.

## Testing the Lambda function

To test the Lambda function, we can use the `cdk` command to deploy the stack and then invoke the Lambda function.

```bash
cdk deploy
```

This will deploy the stack and create the Lambda function. We can get the name of the Lambda function using AWS CLI's `list-functions` command:

```bash
function_name=$(aws lambda list-functions --query "Functions[?contains(FunctionName, 'MyS3Function')].[FunctionName]" --output text)
```

Finally, invoke the Lambda function using the AWS CLI:

```bash
aws lambda invoke \
    --function-name $function_name \
    --cli-binary-format raw-in-base64-out \
    --payload '{ "key": "value" }' \
    /dev/stdout | jq
```

![invoke-lambda-function](/images/50002060-01-lambda-response.png)


!!! note
    The `jq` command is used to format the JSON output. If you don't have it installed, you can install it by running `brew install jq` on macOS or `apt-get install jq` on Linux.

---

# How to create a Lambda function in a Custom Docker image using AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/how-to-create-a-lambda-function-in-a-custom-docker-image-using-aws-cdk-in-python
Category: AWS Academy
Published: 2024-12-06
Author: Rehan Haider
Tags: aws, cdk, python, linux

> This article provides a walkthrough on how to deploy an AWS Lambda function using a Custom Ubuntu Docker image with AWS CDK in Python

In previous posts we looked at how to create using AWS CDK:

1. [Default Lambda function]({filename}50002000-cdk-fn-create-lambda.md), 
2. [Lambda function with Python dependencies using a Lambda layer]({filename}50002020-cdk-fn-lambda_layers.md)
3. [Lambda function with Python dependencies that uses AWS provided Docker image]({filename}50002030-cdk-fn-lambda-python-deps.md)
4. [Lambda function using a AWS ECR Docker image]({filename}50002040-cdk-fn-lambda-aws-docker.md)

But what if you need something more custom? We can use a similar approach to the previous post, but it requires a bit more work.


## Prerequisites

1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md). 
2. If needed [create a new CDK application]({filename}50000020-cdk-new-app.md).


## Create a Lambda function using a custom Docker image

We will need to do the following:


1. Create the `Dockerfile` 
3. Create a `requirements.txt` file to specify the Python packages to be installed.
4. Add AWS Lambda Runtime Interface Client (RIC) to the Dockerfile.
5. Write the lambda function.
6. Create the lambda stack.


### 1. Create the Dockerfile

Let's create a `lambda` directory in `cdk_app` to store the files for this function.

```Dockerfile
# filename: cdk_app/lambda/Dockerfile

FROM ubuntu:latest

ARG FUNCTION_DIR="/function"

# 👇🏽 Install Python and pip
RUN apt update -y 
RUN apt install -y python3 python3-pip

# 👇🏽 Copy the rest of the application
RUN mkdir -p ${FUNCTION_DIR}
COPY . ${FUNCTION_DIR}


# 👇🏽 Set working directory
WORKDIR ${FUNCTION_DIR}

# 👇🏽 Install dependencies
# A target directory is required since latest versions of Ubuntu have implemented PEP 668 
# which prevents pip from installing packages system-wide.
RUN python3 -m pip install --target ${FUNCTION_DIR} -r requirements.txt

# 👇🏽 Install AWS Lambda Runtime Interface Client
RUN python3 -m pip install --target ${FUNCTION_DIR} awslambdaric

# 👇🏽 Define the entrypoint
ENTRYPOINT ["python3", "-m", "awslambdaric"]

# 👇🏽 Set the handler to be used by the Lambda runtime
CMD ["index.handler"]
```

Within this image we have:

1. Used the latest Ubuntu as the base image. This could be changed to a different image if needed.
2. We ensure that Python and pip are installed.
3. Created a directory for the function code.
4. Copied the function code into function directory.
5. Installed the Python packages specified in the `requirements.txt` file in the function directory. 
6. Installed the AWS Lambda Runtime Interface Client (RIC) in the function directory.
7. Set the command to be executed when the container starts.

!!! note
    1. The RIC is required for the Lambda function to communicate with the Lambda runtime.
    2. The packages are installed in the function directory since the latest versions of Ubuntu have implemented PEP 668 which prevents pip from installing packages system-wide.
    3. The `ENTRYPOINT` and `CMD` instructions are used to specify the command to be executed when the container starts.


### 2. Create a requirements.txt file

Create a new file called `requirements.txt` in the `cdk_app/lambda` directory. Any Python packages that you may need to install can be added to this file.

```
requests
```

### 3. Create the lambda function

Within the `lambda` directory create a new file called `index.py`. This is the main Python file that will be executed by the Lambda function.

```python
# filename: cdk_app/lambda/index.py

import requests

def handler(event, context):
    response = requests.get("https://jsonplaceholder.typicode.com/todos/1")

    return {"statusCode": 200, "body": response.json()}
```


### 4. Create a lambda_stack.py file

We modify the `lambda_stack.py` file to create the CDK stack.

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)
from constructs import Construct

class LambdaStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        fn = _lambda.DockerImageFunction(
            self,
            "LambdaFunction",
            code=_lambda.DockerImageCode.from_image_asset("cdk_app/lambda"),
        )
```

In the above code we use the `DockerImageFunction` construct to create the Lambda function. It takes the following arguments:

1. `self`: The construct itself.
2. `id`: The unique identifier for the function.
3. `code`: The Docker image code. In this case we are using the `DockerImageCode.from_image_asset` method to specify the path to the Docker image.
4. `environment`: The environment variables for the function. This is not required but is useful for testing purposes.

Compared to previous examples, we don't need to identify the handler function as this is specified in the `Dockerfile`.

Now finally, let's initialise the stack by creating the `app.py` file.

```python
# filename: app.py

import aws_cdk as cdk

from cdk_app.lambda_stack import LambdaStack

app = cdk.App()
lambda_stack = LambdaStack(app, "LambdaStack")

app.synth()
```

To deploy the stack, run `cdk deploy`. 

When the lambda function is deployed, you can go to the console and test the function. It should show the below output.

![Lambda function output](/images/50002050-01-aws-lambda-output.png)

---

# How to create a Lambda function in a ECR Docker image using AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/how-to-create-a-lambda-function-in-a-ecr-docker-image-using-aws-cdk-in-python
Category: AWS Academy
Published: 2024-12-06
Author: Rehan Haider
Tags: aws, cdk, python

> This article provides a walkthrough on how to deploy an AWS Lambda function using a AWS provided ECR Docker image with AWS CDK in Python

In previous posts we looked at how to create using AWS CDK:

1. [Default Lambda function]({filename}50002000-cdk-fn-create-lambda.md), 
2. [Lambda function with Python dependencies using a Lambda layer]({filename}50002020-cdk-fn-lambda_layers.md)
3. [Lambda function with Python dependencies that uses AWS provided Docker image]({filename}50002030-cdk-fn-lambda-python-deps.md)

But in some cases, you may need to modify the Docker image that AWS Lambda uses. E.g. you may need to install additional dependencies or modify the runtime.

In this post, we'll look at how to create a Lambda function using a AWS provided Docker image.


## Prerequisites

1. Ensure that you have [AWS CDK and SAM CLI installed]({filename}00000100-cdk-installing-cdk-sam-cli.md). 
2. If needed [create a new CDK application]({filename}50000020-cdk-new-app.md).


## Create a Lambda function using a ECR Docker image

We will need to do the following:


1. Create the `Dockerfile` 
2. Create a `requirements.txt` file to specify the Python packages to be installed.
3. Write the lambda function.
4. Create the lambda stack.


### 1. Create the Dockerfile

Let's create a `lambda` directory in `cdk_app` to store the files for this function.

```Dockerfile
# filename: cdk_app/lambda/Dockerfile

FROM public.ecr.aws/lambda/python:3.12

# Copy requirements.txt
COPY requirements.txt ${LAMBDA_TASK_ROOT}

# Install the specified packages
RUN pip install -r requirements.txt

# Copy function code
COPY index.py ${LAMBDA_TASK_ROOT}

# Set the CMD to your handler
CMD [ "index.handler" ] 
```

Within this image we have:

1. Identified the image that we want to use. In this case we are using the AWS provided Python 3.12 image.
2. Copied the `requirements.txt` file to the image.
3. Installed the specified packages.
4. Copied the function code to the image.
5. Set the command to be executed when the container starts.

### 2. Create a requirements.txt file

Create a new file called `requirements.txt` in the `cdk_app/lambda` directory. Any Python packages that you may need to install can be added to this file.

```
requests
```

### 3. Create the lambda function

Within the `lambda` directory create a new file called `index.py`. This is the main Python file that will be executed by the Lambda function.

```python
# filename: cdk_app/lambda/index.py

import requests

def handler(event, context):
    response = requests.get("https://jsonplaceholder.typicode.com/todos/1")

    return {"statusCode": 200, "body": response.json()}
```


### 4. Create a lambda_stack.py file

We modify the `lambda_stack.py` file to create the CDK stack.

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)
from constructs import Construct

class LambdaStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        fn = _lambda.DockerImageFunction(
            self,
            "LambdaFunction",
            code=_lambda.DockerImageCode.from_image_asset("cdk_app/lambda"),
        )
```

In the above code we use the `DockerImageFunction` construct to create the Lambda function. It takes the following arguments:

1. `self`: The construct itself.
2. `id`: The unique identifier for the function.
3. `code`: The Docker image code. In this case we are using the `DockerImageCode.from_image_asset` method to specify the path to the Docker image.
4. `environment`: The environment variables for the function. This is not required but is useful for testing purposes.

Compared to previous examples, we don't need to identify the handler function as this is specified in the `Dockerfile`.

Now finally, let's initialise the stack by creating the `app.py` file.

```python
# filename: app.py

import aws_cdk as cdk

from cdk_app.lambda_stack import LambdaStack

app = cdk.App()
lambda_stack = LambdaStack(app, "LambdaStack")

app.synth()
```

To deploy the stack, run `cdk deploy`. 

When the lambda function is deployed, you can go to the console and test the function. It should show the below output.

![Lambda function output](/images/50002040-01-aws-lambda-output.png)

---

# Mount Amazon FSX Lustre filesystem and attack to AWS EC2 instance

URL: https://cloudbytes.dev/aws-academy/mount-amazon-fsx-lustre-filesystem-and-attack-to-aws-ec2-instance
Category: AWS Academy
Published: 2024-09-02
Author: Rehan Haider
Tags: aws, linux

> A step-by-step guide to mount Amazon FSX Lustre filesystem and attack to AWS EC2 instance

Apart from [Elastic File System (EFS)]({filename}25000000-mount-efs-ec2.md) that uses Network File System (NFS), Amazon also provides other types of distributed filesystems on AWS under the Amazon FSx service, such as:

1. **Amazon FSx for Windows File Server**: A fully managed Windows file system that is accessible from Windows and Linux instances.
2. **Amazon FSx for Lustre**: A fully managed Lustre file system that is optimized for compute-intensive workloads.
3. **Amazon FSx for NetApp ONTAP**: A fully managed NetApp ONTAP file system that is accessible from Windows and Linux instances.
4. **Amazon FSx for OpenZFS**: A fully managed OpenZFS file system that is accessible from Windows and Linux instances.

In this article, we will learn how to mount an Amazon FSx for Lustre filesystem on an Amazon EC2 instance.

## What is Amazon FSx for Lustre?

[Amazon FSx for Lustre](https://aws.amazon.com/fsx/lustre/) is a fully managed file system that is optimized for compute-intensive workloads, such as [high-performance computing (HPC)](https://en.wikipedia.org/wiki/High-performance_computing), machine learning, and media data processing workflows. 

FSx for Lustre is built on the [Lustre](https://www.lustre.org) file system, an open-source, parallel distributed file system that is designed for high-performance computing environments.

### FSx mount on EC2 - How it works?

The below architecture diagram shows how Amazon FSx for Lustre can be mounted on an Amazon EC2 instance:

![Aws fsx lustre architecture diagram](/images/27500000-01-architecture-diagram.png)

1. The VM(s) that will mount the FSx for Lustre filesystem are typically in their own subnets and security groups. In the example above, a webserver is connected to internet using port 80 or 443 and kept in a publicly accessible subnet.
2. The EFS drive could should ideally be kept in a separate subnet and security group.
3. The two security groups DMZ SG (which contains the EC2 instance) and the NFS SG (which contains the EFS drive) are connected to allow traffic between them over port 2049

## Project Lab Setup

### Objectives
**In this tutorial, we will **

1. Create an EC2 instance
2. Create an FSx for Lustre drive 
3. Mount the FSx drive on the EC2 instance
4. Download a image file onto the FSx Lustre mount
5. Access the images from another EC2 instance

### Prerequisites

You need an AWS Account and AWS CLI installed on your system. You can find the instructions on how to install AWS CLI [here]({filename}/aws/12500000-aws-cli-intro.md).

## How to Mount Amazon FSx Lustre filesystem on EC2 instance

We will mount the FSx for Lustre filesystem on Amazon EC2 instance using two methods:

- [Using AWS Management Console](#mount-amazon-fsx-for-lustre-to-ec2-using-aws-management-console)
- [Using AWS CLI](#mount-amazon-fsx-for-lustre-to-ec2-using-aws-cli)

### Mount Amazon FSx for Lustre to EC2 using AWS Management Console

#### Step 1: Create the security groups

We need to create two security groups:

1. DMZ SG: This security group will be attached to the EC2 instance and will allow inbound traffic on port 80 and 443.
2. FSX SG: This security group will be attached to the FSx for Lustre filesystem

A) Logon to AWS Management Console:

* Search for **Security Group**s in the search bar and click on it.
* Click on **Create security group** on the top right.

B) In the **Create security group** page:

* Enter `DMZ` as the name, `Security group for DMZ` as the description.
* In the **Inbound rules** section, click on **Add rule**.
* Under **Type**, select `HTTP` and under **Source** select `Anywhere-IPv4`.
* Click on **Add rule** again, and under type search for `SSH` and under **Source** select `Anywhere-IPv4`.
* Then scroll to the bottom and click on **Create security group**.

![Create DMZ security group](/images/27500000-02-dmz-security-group.png)

C) Next, go back to the **Security Groups** page and click on **Create security group** again. In the **Create security group** page:

For the FSX security, we need to create a rule that allows traffic on port 998, 1018-1023 within the security group. This will require us to first create the security group and then edit the inbound rules.

* Enter `FSX` as the name, `Security group for FSX` as the description.
* Don't add any inbound rule yet, scroll down to the bottom and click on **Create security group**.
* From the list of security groups, click on the `FSX` security group we just created.
* Click on the **Edit inbound rules** button.
* Click on **Add rule**. We need 2 rules:

| Type    | Protocol | Port range | Source | Security group |
| ------- | -------- | ---------- | ------ | -------------- |
| All TCP | TCP      | 0 - 65535  | Custom | FSX            |
| All TCP | TCP      | 0 - 65535  | Custom | DMZ            |

* Click on  **Save rules**.

![Create FSX security group](/images/27500000-03-create-fsx-security-group.png)


#### Step 2: Create the FSx for Lustre filesystem

Logon to AWS Management Console and search for **FSx** in the search bar and click on it. Then click on **Create file system**. In the **Create file system** page, select `Amazon FSx for Lustre` and click on **Next**.

Now, in the **Specify file system details** page:

* Enter a name for the file system.
* Under **Deployment and storage type**, select `Persistent, SSD`.
* Under **Throughput per unit of storage**, select `125 MB/s/TiB`.
* Under **Storage capacity**, enter `1.2` TiB.
* Under **Metadata Configuration**, select `Automatic`.
* Under **Data compression type**, select `NONE`.

Next, in the **Network & security** section:

* Select the VPC where you want to create the FSx for Lustre filesystem.
* Under **VPC security group**, select the `FSX` security group we created earlier.
* Under **Subnet**, select the subnet where you want to create the FSx for Lustre filesystem, in this case, I selected `us-east-1a`.

Leave the rest of the settings as default and click on **Next**. Then click on **Create file system**.


#### Step 3: Create the EC2 instance

Search for EC2, and click on EC2 to go to the EC2 page. In the left hand navigation panel, click on **Instances**. On the top right, click on **Launch instances**.

In the **Launch an instance** page:

- In **Name and tags** section, enter a name for the instance. I chose `myServer2`.
- In **Application and OS Images** section, click on quick start and select `Amazon Linux`. Make sure you have chose `Amazon Linux 2023 AMI` in the **Amazon Machine Image (AMI)** dropdown.
- In **Instance type** section, select `t2.micro`.
- In **Key pair** section, select an existing key pair or create a new one.
- In **Network settings**, click on **Edit** button, and change the subnet to the same subnet where the FSx for Lustre filesystem is created. E.g. in my case it was `us-east-1a`.
- In **Firewall (security groups)**, click on **Select exiting security group** and select the `DMZ` security group we created earlier.
- Leave the **Configure storage** section unchanged.
- Click on **Launch instance** to launch the instance.

#### Step 4: Mount the FSx for Lustre filesystem on the EC2 instance

Once the EC2 instance is running, connect to the instance by going to the EC2 console and click on Instances. Click on the instance you created in the previous step, then on top right, click on Connect.

A) Update the instance:

```bash
sudo yum update -y
```

B) Install the Lustre client:

```bash
sudo dnf install -y lustre-client
```

C) Create a directory to mount the FSx for Lustre filesystem:

```bash
sudo mkdir -p /mnt/fsx
```

D) Mount the FSx for Lustre filesystem:

```bash
sudo mount -t lustre -o relatime,flock <fsx-dns-name>@tcp:/<mount-name> /mnt/fsx
```

Replace the `<fsx-dns-name>` with the **DNS name**, and  `<mount-name>` with the **Mount name** of the FSx for Lustre filesystem.
You can find both in the FSx console, as shown below:

!!! warning 
    AWS documentation doesn't clarify this correctly but the `<mount-name>` is specific to the instance of FSx Lustre and needs to be specified correctly.



![FSx DNS name](/images/27500000-04-fsx-dns-name.png)


This should mount the FSx for Lustre filesystem on the EC2 instance. You can now access the filesystem by going to the `/mnt/fsx` directory. 

Verify the mount by running the `df -h` command:

```bash
df -h
```

![FSx mounted on EC2](/images/27500000-05-verify-fsx-mount.png)

#### Step 5: Test the FSx for Lustre filesystem

A) Let's donwload a few files and save them on the FSx for Lustre filesystem:

```bash
sudo curl -X GET https://cataas.com/cat -o /mnt/fsx/myFile.jpg && ls -al /mnt/fsx
```

![Download files to FSx](/images/27500000-06-download-file-to-fsx.png)


B) Now, let's create another EC2 instance and mount the FSx for Lustre filesystem on it:

1. Follow [step 3](#step-3-create-the-ec2-instance) to create an EC2 instance named myServer3.
2. Follow [step 4](#step-4-mount-the-fsx-for-lustre-filesystem-on-the-ec2-instance) to mount the FSx for Lustre filesystem on the EC2 instance.

C) Now run `ls -al /mnt/fsx` on the new EC2 instance to see the files downloaded on the FSx for Lustre filesystem.

![Access files from another EC2](/images/27500000-07-access-files-from-another-ec2.png)

### Mount Amazon FSx for Lustre to EC2 using AWS CLI

#### Step 1: Create the security groups

A) We need to create two security groups:

1. DMZ SG: This security group will be attached to the EC2 instance and will allow inbound traffic on port 80 and 443.
2. FSX SG: This security group will be attached to the FSx for Lustre filesystem. This will contain two inbound rules, one to allow traffic from the DMZ SG and another to allow traffic within the FSX SG.

```bash
DMZ_SG_ID=$(aws ec2 create-security-group --group-name "DMZ" \
  --description "Security group for DMZ" --query "GroupId" --output text) && \
  FSX_SG_ID=$(aws ec2 create-security-group --group-name "FSX" \
  --description "Security group for FSX" --query "GroupId" --output text) && \
  echo "Created security groups: DMZ: $DMZ_SG_ID and NFS: $FSX_SG_ID"
```

B) Open the inbound rules for the DMZ security group:

```bash
aws ec2 authorize-security-group-ingress --group-id $DMZ_SG_ID \
    --protocol tcp --port 80 --cidr 0.0.0.0/0 && \
    aws ec2 authorize-security-group-ingress --group-id $DMZ_SG_ID \
    --protocol tcp --port 22 --cidr 0.0.0.0/0
```

C) Create the inbound rules for the FSX security group:

```bash
aws ec2 authorize-security-group-ingress --group-id $FSX_SG_ID \
    --protocol tcp --port 0-65535 --source-group $FSX_SG_ID && \
    aws ec2 authorize-security-group-ingress --group-id $FSX_SG_ID \
    --protocol tcp --port 0-65535 --source-group $DMZ_SG_ID
```

D) Get the default subnet ID from `us-east-1a` availability zone:

```bash
SUBNET_ID=$(aws ec2 describe-subnets --query "Subnets[?AvailabilityZone=='us-east-1a'].SubnetId" --output text)
```

#### Step 2: Create the FSx for Lustre filesystem

```bash
FSX_ID=$(aws fsx create-file-system \
    --file-system-type LUSTRE \
    --storage-capacity 1200 \
    --storage-type SSD \
    --lustre-configuration DeploymentType="PERSISTENT_2",PerUnitStorageThroughput=125 \
    --subnet-id $SUBNET_ID \
    --security-group-ids $FSX_SG_ID \
    --output text \
    --query "FileSystem.FileSystemId") && \
  echo "Created FSx for Lustre filesystem: $FSX_ID"
```

#### Step 3: Create the EC2 instance

First, we need to get the latest Amazon Linux 2023 AMI ID. You can go the the AWS Console, go to **EC2 -> Instances -> Launch instances**. In the **Application and OS Images** section, select `Amazon Linux 2023 AMI`. Note the AMI ID at the bottom.

![Amazon Linux 2023 AMI ID](/images/27500000-08-amazon-linux-ami-id.png)

Alternatively, you can use the below command to get the latest Amazon Linux 2023 AMI ID:

A) Create the first EC2 instane named `myServer2` in us-east-1a, and store the IP address in a variable. 

Repalce `<key-pair>` with your key pair name and `<ami-id>` with the Amazon Linux 2023 AMI ID.

```bash
MYSERVER1_ID=$(aws ec2 run-instances \
    --image-id <ami-id> \
    --count 1 \
    --instance-type t2.micro \
    --key-name <key-pair> \
    --security-group-ids $DMZ_SG_ID \
    --subnet-id $SUBNET_ID \
    --associate-public-ip-address \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=myServer1}]" \
    --query "Instances[0].InstanceId" --output text) && \
    echo "Created EC2 instance myServer1: $MYSERVER1_ID"
```

B) Get the IP address from the instance metadata:

```bash
MYSERVER1_IP=$(aws ec2 describe-instances \
    --instance-ids $MYSERVER1_ID \
    --query "Reservations[0].Instances[0].PublicIpAddress" --output text) && \
    echo "EC2 instance myServer1 IP: $MYSERVER1_IP"
```

C) Check if the instance state is running and status checks are complete:

```bash
aws ec2 wait instance-status-ok --instance-ids $MYSERVER1_ID
```

#### Step 4: Mount the FSx for Lustre filesystem on the EC2 instance (CLI)

A) Connect to the EC2 instance:

Make sure you replace `<key-pair>` with your key pair name and that you have downloaded the key pair file.

```bash
ssh -i <key-pair>.pem ec2-user@$MYSERVER1_IP
```

B) Install the Lustre client:

```bash
sudo dnf install -y lustre-client
```

C) Create a directory to mount the FSx for Lustre filesystem:

```bash
sudo mkdir -p /mnt/fsx
```

D) Mount the FSx for Lustre filesystem:

```bash
sudo mount -t lustre -o relatime,flock <fsx-dns-name>@tcp:/<mount-name> /mnt/fsx
```

Replace the `<fsx-dns-name>` with the **DNS name**, and  `<mount-name>` with the **Mount name** of the FSx for Lustre filesystem.
You can find both in the FSx console, as shown below:

To get these values, you can use the below command on a different terminal:

```bash
FSX_DNS_NAME=$(aws fsx describe-file-systems --file-system-ids $FSX_ID --query "FileSystems[0].DNSName" --output text) && \
  FSX_MOUNT_NAME=$(aws fsx describe-file-systems --file-system-ids $FSX_ID --query "FileSystems[0].LustreConfiguration.MountName" --output text) && \
  echo "FSx DNS name: $FSX_DNS_NAME, Mount name: $FSX_MOUNT_NAME"
```

Verify the mount by running the `df -h` command.

![FSx mounted on EC2](/images/27500000-05-verify-fsx-mount.png)

#### Step 5: Test the FSx for Lustre filesystem

A) Let's donwload a few files and save them on the FSx for Lustre filesystem:

On the EC2 instance, run the below command:

```bash
sudo curl -X GET https://cataas.com/cat -o /mnt/fsx/myFile.jpg && ls -al /mnt/fsx
```

Logout from the EC2 instance by running `exit`.

B) Create another EC2 instance named `myServer3` and mount the FSx for Lustre filesystem on it:

```bash
MYSERVER2_ID=$(aws ec2 run-instances \
    --image-id <ami-id> \
    --count 1 \
    --instance-type t2.micro \
    --key-name <key-pair> \
    --security-group-ids $DMZ_SG_ID \
    --subnet-id $SUBNET_ID \
    --associate-public-ip-address \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=myServer1}]" \
    --query "Instances[0].InstanceId" --output text) && \
    echo "Created EC2 instance myServer1: $MYSERVER2_ID"
```

C) Get the IP address from the instance metadata:

```bash
MYSERVER2_IP=$(aws ec2 describe-instances \
    --instance-ids $MYSERVER2_ID \
    --query "Reservations[0].Instances[0].PublicIpAddress" --output text) && \
    echo "EC2 instance myServer1 IP: $MYSERVER2_IP"
```

D) Check if the instance state is running and status checks are complete:

```bash
aws ec2 wait instance-status-ok --instance-ids $MYSERVER2_ID
```

E) Follow the mount instructions in [step 4](#step-4-mount-the-fsx-for-lustre-filesystem-on-the-ec2-instance-cli) to mount the FSx for Lustre filesystem on the new EC2 instance.

F) Run `ls -al /mnt/fsx` on the new EC2 instance to see the files downloaded on the FSx for Lustre filesystem.

![Access files from another EC2](/images/27500000-07-access-files-from-another-ec2.png)

Log out from the EC2 instance by running `exit`.

#### Step 6: Clean up

Clean up the resources by: 

A) Deleting the EC2 instances and waiting for the instances to terminate:

```bash
aws ec2 terminate-instances --instance-ids $MYSERVER1_ID $MYSERVER2_ID && \
  aws ec2 wait instance-terminated --instance-ids $MYSERVER1_ID $MYSERVER2_ID
```

B) Delete the FSx for Lustre filesystem:

```bash
aws fsx delete-file-system --file-system-id $FSX_ID &&
```

> There is no `wait` command for FSx deletion. Hence you have to run the below command to check the status and wait till the deletion is complete.

Wait for the FSx for Lustre filesystem to be deleted. Check the status by running:

```bash
aws fsx describe-file-systems --file-system-ids $FSX_ID
```

C) Deleting the security groups:

```bash
aws ec2 delete-security-group --group-id $FSX_SG_ID && \
  aws ec2 delete-security-group --group-id $DMZ_SG_ID
```

---

# Mount Amazon EFS Drive on EC2 Ubuntu Linux using NFS Utils

URL: https://cloudbytes.dev/aws-academy/mount-amazon-efs-drive-on-ec2-ubuntu-linux-using-nfs-utils
Category: AWS Academy
Published: 2024-06-13
Author: Rehan Haider
Tags: aws, linux

> A comprehensive guide to mount Amazon Elastic File Storage (EFS) on Ubuntu Linux using NFS Utils and then use it to serve files from the EFS drive. The guided includes instructions for both AWS Console & CLI

[TOC]

The first question that you should be asking yourself is, why not use `amazon-efs-utils` to [mount the EFS drive](https://docs.aws.amazon.com/efs/latest/ug/installing-amazon-efs-utils.html)?

In short, `amazon-efs-utils` package is only available for Amazon Linux and other Linux versions require you to build it from scratch. Also the fact that EFS is supposed to act like a Network File System (NFS) and that almost all Linux versions have an already available & extensively tested NFS Utilities. 

> Even though EFS mount helper is comparatively easier to use, NFS is still preferred by most enterprises because they want to use proven and throughly tested utilities especially if they impact important services.

## What is Amazon EFS?
Amazon Elastic File Storage (EFS) is a network mountable elastic shared drive. Which means, you can attach/mount it to your Linux machine as a network drive and it starts with a capacity of almost 0 and can easily grow into Petabytes of storage. 

### EFS mount on EC2 - How it works?

The below architecture diagram explains a typical use case of how EFS is used with EC2.

![EFS EC2 Architecure diagram](/images/25000000-architecture-diagram.png)

1. The VM(s) that will mount the drive  are typically in their own subnets and security groups. In the example above, a webserver is connected to internet using port 80 or 443 and kept in a publicly accessible subnet.
2. The EFS drive could should ideally be kept in a separate subnet and security group.
3. The two security groups DMZ SG (which contains the EC2 instance) and the NFS SG (which contains the EFS drive) are connected to allow traffic between them over port 2049

## Project Lab Setup

### Objectives
**In this tutorial, we will **

1. Create an EC2 webserver with Apache installed
2. Create an EFS drive 
3. Mount the EFS drive on the EC2 webserver
4. Download a image file onto the EFS drive
5. Create a basic HTML page that will display image

### Prerequisites

You need an AWS Account and AWS CLI installed on your system. You can find the instructions on how to install AWS CLI [here]({filename}/aws/12500000-aws-cli-intro.md).


## How to Mount an Amazon EFS to EC2? 

We will mount the EFS drive on EC2 using two methods:

- [Using AWS Management Console](#mount-amazon-efs-to-ec2-ubuntu-linux-using-aws-console)
- [Using AWS CLI](#mount-amazon-efs-to-ec2-ubuntu-linux-using-aws-cli)

### Mount Amazon EFS to EC2 Ubuntu Linux using AWS Console

#### Step 1) Create the security groups

We need to create two security groups as follows:

1. **DMZ SG** - This SG is used to allow traffic from the internet to the EC2 instance, i.e. port 80 or 443.
2. **NFS SG** - This SG is used to allow traffic from the members of the DMZ SG to the EFS drive over port 2049.

A) Logon to AWS Management Console:

- Search for **Security Groups** and click on **Security Group** under features. 
- Click on **Create security group** on the top right.

B) In the **Create Security Group** dialog:

- Enter `DMZ` as the name, ``Security group for DMZ`` as the description. 
- In the **inbound rules** section, click on **Add Rule**, 
- Under **Type** select `http` and under **Source** select `Anywhere-IPv4`. 
- CLick on **Add Rule** again, and under type search for `SSH` and under **Source** select `Anywhere-IPv4`.
- Then scroll to bottom and click on **Create security group**.

![Create Security Group AWS Console](/images/25000000-create-dmz-security-console.png)

C) Next, Go back to **Security Groups** page and click on **Create security group** again. In the **Create Security Group** dialog:

- Enter `NFS` as the name, ``Security group for NFS`` as the description. 
- In the **inbound rules** section, under **Type** select `NFS` and under **Source** click on the textbox to bring up a list of CIDR & Security Group options. 
- Select `DMZ|sg-xxxxxxx`, i.e. the security group we created earlier, under security groups. Then scroll to bottom and click on **Create security group**.

![Create Security Group AWS Console](/images/25000000-create-nfs-security-console.png)


#### Step 2) Create the EFS Drive

Log on to [AWS Console](https://console.aws.amazon.com/), search for EFS and then click on EFS.

![AWS Console EFS](/images/25000000-efs-console.png)

Click on "**Create file system**", in the **Create File System** dialog:

- Choose a name for your file system, I chose **myEFS**.
- Under **Availability and durability** section, choose **One Zone** 
- Under **Availability Zone** choose `us-east-1a` and then click on **Create** button

Now you should see a File system created named **myEFS** under **File systems** section.

![EFS File system created](/images/25000000-efs-filesystem-created.png)


#### Step 3) Customise EFS & Configure Security Groups

Because we used the quick create option and didn't customize our EFS file system, it was created with several default settings such as:

1. **Automatic backups**: Enabled
2. **Lifecycle management**: EFS Intelligent tiering is enabled, and configured to transition files from Standard to Standard-Infrequent Access tier after 30 days of inactivity and to transition out on first access.
3. **Performance mode**: Only **General Purpose** is available for **One ZOne**.
4. **Throughput mode**: **Bursting** is selected by default. The alternative is **Provisioned**.
5. **Encryption**: Enabled
6. **Network access**: Default VPC, default subnet, and default security group associated with the Availability Zone is selected

While 1 - 5 are file, we need to modify the **Network access** to enable the EFS to communicate with the EC2 instance on port 2049.

A) In the left hand panel, 

- Click on **File systems**, then under **File system**, click on **myEFS**. 
- At the bottom, select the **Network** tab. 
- Then click on **Manage** button after **Mount target state** becomes **Available**.

![EFS Network Settings](/images/25000000-network-settings.png)

B) In the **Network access** page, under **Mount targets**, 

- Remove the existing default **Security Group**
- Click on the dropdown under **Security Groups** and select `NFS|sg-xxxxxxx`
- Then save the changes

![change-efs-security-group](/images/25000000-change-efs-security-group.gif)


#### Step 4) Create the EC2 Ubuntu Linux instance

Search for EC2, and click on EC2 to go to the EC2 page. In the left hand navigation panel, click on **Instances**.

Now on top right, click on **Launch instances**, then

- **Name and tags** section, provide a name. I Chose `myServer1`
- **Application and OS Images** section, select **Ubuntu**, then from the dropdown, select **Ubuntu 20.04**.
- **Instance type** section, select **t2.micro**.
- **Key pair** section, select any keypair, or create one if you don't have one.
- **Network setting**, click on **Edit** button, then change the subnet to the one in `us-east-1`. Then click on **Select existing security group** and from dropdown, select `DMZ|sg-xxxxxxx`
- **Configure storage**, leave unchanged
- Then click on **Launch instance** button.

#### Step 5) Login to the EC2 instance

Go to the EC2 console and click on **Instances**. Click on the instance you created in the previous step, then on top right, click on **Connect**. 

In **Connect to instance** dialog, click on **EC2 Instance Connect** and then at the bottom right, click on **Connect**.

Install NFS Utilities and enable it to start on startup.

```bash
sudo apt install nfs-common -y && \
    sudo systemctl status nfs-utils
```

#### Step 6) Mount the EFS Drive

Go to the EFS, click on the EFS file system you created, e.g. **myEFS**. At the bottom, click on network and note the IP address.  

Next, mount the EFS drive to the EC2 instance. Replace `<EFS IP>` with the IP address from above.

```bash
mkdir efs
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS IP>:/ efs
```

Alternatively, then click on *Attach**. In the subsequent dialog, select on **Mount via IP**. Copy the command provided and run it in the EC2 Ubuntu terminal from previous step.

![EFS mount command](/images/25000000-mount-command.png)


Next, run the below command in the EC2 terminal.

```bash
lsblk
```

You should now see the EFS drive mounted as a new partition.

![25000000-check-efs-mount-lsblk](/images/25000000-check-efs-mount-lsblk.png)

#### Step 7) Test the mounted EFS Drive

A) Now let's download a few files and save them in the directory.

```bash
sudo curl -X GET https://cataas.com/cat -o efs/myFile.jpg
```

B) List the files

```bash
ls -al efs
```

![25000000-output-server1](/images/25000000-output-server1.png)

C) Now follow steps 4, 5 and 6 again to create another EC2 instance. 

1. Follow [step 4](#step-4-create-the-ec2-ubuntu-linux-instance) to create another EC2 instance named `myServer2`
2. Follow [step 5](#step-5-login-to-the-ec2-instance) lo login to the server
3. Follow [step 6](#step-6-mount-the-efs-drive) to mount the EFS drive on the new server

D) Now run `ls -al efs` to check the contents of the mounted directory. This should be the same as that of **myServer1** above.

![25000000-efs-check-server2](/images/25000000-efs-check-server2.png)



### Mount Amazon EFS to EC2 Ubuntu Linux using AWS CLI

#### Step 1: Create Security Groups

A) We need to create two security groups as follows:

1. **DMZ SG** - This SG is used to allow traffic from the internet to the EC2 instance, i.e. port 80 or 443.
2. **NFS SG** - This SG is used to allow traffic from the members of the DMZ SG to the EFS drive over port 2049.


```bash
DMZ_SG_ID=$(aws ec2 create-security-group --group-name "DMZ" \
  --description "Security group for DMZ" --query "GroupId" --output text) && \
  NFS_SG_ID=$(aws ec2 create-security-group --group-name "NFS" \
  --description "Security group for NFS" --query "GroupId" --output text) && \
  echo "Created security groups: DMZ: $DMZ_SG_ID and NFS: $NFS_SG_ID"
```

This will create the security groups and assign the IDs to the variables.

B) Next open port 80 & 22 access from anywhere (CIDR 0.0.0.0/0) for DMZ security group


```bash
aws ec2 authorize-security-group-ingress --group-id $DMZ_SG_ID \
    --protocol tcp --port 80 --cidr 0.0.0.0/0 && \
    aws ec2 authorize-security-group-ingress --group-id $DMZ_SG_ID \
    --protocol tcp --port 22 --cidr 0.0.0.0/0
```

Next allow access from DMZ SG to NFS SG over port 2049.

```bash
aws ec2 authorize-security-group-ingress --group-id $NFS_SG_ID \
    --source-group $DMZ_SG_ID --protocol tcp --port 2049
```

#### Step 2: Create EFS File System

A) Create a new EFS file system named myEFS in `us-east-1a`.

```bash
EFS_ID=$(aws efs create-file-system \
    --availability-zone-name "us-east-1a" \
    --encrypted \
    --tags "Key=Name,Value=myEFS" \
    --query "FileSystemId" --output text) && \
    echo "Created EFS file system: $EFS_ID"
```
B) By default, AWS CLI doesn't create a **Mount target**, assign **Security group**, configure **Subnet**, or configure **Lifecycle management**. while we don't need to configure the latter, we do need to create a **Mount target**.

Creating **Mount target** also requires assigning a **Security group** and choosing the **Subnet** where the **Mount target** will be created. We know the **Security group** and its ID already, but we need to find the ID of the default **Subnet** for the `us-east-1a` availability zone.

```bash
SUBNET_ID=$(aws ec2 describe-subnets \
    --filters "Name=availability-zone,Values=us-east-1a" \
    --query "Subnets[0].SubnetId" --output text) && \
    echo "Subnet ID for us-east-1a: $SUBNET_ID"
```

C) Now create the **Mount target**.

```bash
aws efs create-mount-target --file-system-id $EFS_ID \
    --subnet-id $SUBNET_ID --security-groups $NFS_SG_ID
```
This will output the following data, note down the `IpAddress`.

```json
{
    "OwnerId": "268674271179",
    "MountTargetId": "fsmt-0955bbcc67dccca55",
    "FileSystemId": "fs-051523e63a4561e3d",
    "SubnetId": "subnet-0b0a5941",
    "LifeCycleState": "creating",
    "IpAddress": "172.31.23.189",
    "NetworkInterfaceId": "eni-057847b7e1c9adf1f",
    "AvailabilityZoneId": "use1-az4",
    "AvailabilityZoneName": "us-east-1a",
    "VpcId": "vpc-7918c403"
}
```
#### Step 3: Create the EC2 Ubuntu instance

First open the AWS console, go to **EC2 -> Instances -> Launch instances**. In the **Application and OS Images** section, select `Ubuntu` and then from the dropdown select `Ubuntu 20.04 LTS (HVM)`. Note the AMI ID at the bottom. 

![25000000-ami-id](/images/25000000-ami-id.png)

A) Create a new EC2 instance named `myServer1` in `us-east-1a`. and store the IP address of the instance in a variable.

```bash
MYSERVER1_ID=$(aws ec2 run-instances \
    --image-id ami-08d4ac5b634553e16 \
    --count 1 \
    --instance-type t2.micro \
    --key-name myKeyPair \
    --security-group-ids $DMZ_SG_ID \
    --subnet-id $SUBNET_ID \
    --associate-public-ip-address \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=myServer1}]" \
    --query "Instances[0].InstanceId" --output text) && \
    echo "Created EC2 instance myServer1: $MYSERVER1_ID"
```
B) Get the IP address from instance metadata.

```bash
MYSERVER1_IP=$(aws ec2 describe-instances \
    --instance-ids $MYSERVER1_ID \
    --query "Reservations[0].Instances[0].PublicIpAddress" --output text) && \
    echo "EC2 instance myServer1 IP: $MYSERVER1_IP"
```

C) Check if the instance state is running and status checks are complete.

```bash
aws ec2 wait instance-status-ok --instance-ids $MYSERVER1_ID
```

#### Step 4: Login to the EC2 instance

Make sure you have a keypair downloaded and parth is specified correctly below

```bash
ssh -i ~/.ssh/myKeyPair.pem ubuntu@$MYSERVER1_IP
```
> In case you get `SSH Permission are too open` error, change the file's permission to `600` by running `chmod 600 ~/.ssh/myKeyPair.pem`

Install the NFS Utilities
  
```bash
sudo apt install nfs-common -y && \
    sudo systemctl status nfs-utils
```

#### Step 5: Mount the EFS File System

Next, mount the EFS drive to the EC2 instance. Replace <EFS IP> with the IP address of the Mount target from step 2C above

```bash
mkdir efs
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS IP>:/ efs
```

Run the following command in the EC2 terminal to see the EFS drive mounted.

```bash
lsblk
```

![25000000-check-efs-mount-lsblk](/images/25000000-check-efs-mount-lsblk.png)


#### Step 6: Test the mounted EFS Drive

A) Download a file and save them in EFS drive.

```bash
sudo curl -X GET https://cataas.com/cat -o efs/myFile.jpg
```

B) List the files

```
ls -al efs
```

![25000000-output-server1](/images/25000000-output-server1.png)

B) Logout from `myServer1`

```bash
exit
```

C) Now back in your AWS CLI, create a new instance named `myServer2` in `us-east-1a`.

```bash
MYSERVER2_ID=$(aws ec2 run-instances \
    --image-id ami-08d4ac5b634553e16 \
    --count 1 \
    --instance-type t2.micro \
    --key-name myKeyPair \
    --security-group-ids $DMZ_SG_ID \
    --subnet-id $SUBNET_ID \
    --associate-public-ip-address \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=myServer2}]" \
    --query "Instances[0].InstanceId" --output text) && \
    echo "Created EC2 instance myServer2: $MYSERVER2_ID"
```
Get the IP address from instance metadata.

```bash
MYSERVER2_IP=$(aws ec2 describe-instances \
    --instance-ids $MYSERVER2_ID \
    --query "Reservations[0].Instances[0].PublicIpAddress" --output text) && \
    echo "EC2 instance myServer2 IP: $MYSERVER2_IP"
```

D) Check if the instance state is running and status checks are complete.

```bash
aws ec2 wait instance-status-ok --instance-ids $MYSERVER2_ID
```

E) After the above command wait is complete, login to the EC2 instance `myServer2`

```bash
ssh -i ~/.ssh/myKeyPair.pem ubuntu@$MYSERVER2_IP
```

F) Follow the [mount instructions in Step 5](#step-5-mount-the-efs-file-system) above to mount the EFS drive.


Finally, run `ls -al efs` to see the files in the EFS drive that was downloaded from `myServer1`.

![25000000-efs-check-server2](/images/25000000-efs-check-server2.png)

Logout from EC2 instance `myServer2` before proceeding to next step by running `exit`

#### Step 7: Clean up

Clean up the EC2 instances and wait for them to terminate.

```bash
aws ec2 terminate-instances --instance-ids $MYSERVER1_ID $MYSERVER2_ID && \
  aws ec2 wait instance-terminated --instance-ids $MYSERVER1_ID $MYSERVER2_ID
```

Next, we need to delete the **Mount target**. First, fetch ID of the Mount target. You can also manually get this from the output in step 2C.

```bash
MOUNT_TARGET_ID=$(aws efs describe-mount-targets \
    --file-system-id $EFS_ID \
    --query "MountTargets[0].MountTargetId" --output text) && \
    echo "EFS Mount Target ID: $MOUNT_TARGET_ID"
```

Then, delete the Mount target.

```bash
aws efs delete-mount-target --mount-target-id $MOUNT_TARGET_ID
```

Finally, delete the EFS file system.

```bash
aws efs delete-file-system --file-system-id $EFS_ID
```

---

# Migrate a DB to RDS using AWS Database Migration Service (DMS)

URL: https://cloudbytes.dev/aws-academy/migrate-a-db-to-rds-using-aws-database-migration-service-dms
Category: AWS Academy
Published: 2024-03-03
Author: Rehan Haider
Tags: aws

> Use AWS Database Migration Service (DMS) to migrate a database to Amazon RDS by configuring a replication instance, source and target endpoints, and a migration task.

AWS Database Migration Service (DMS) is a managed service that enables you to migrate databases to AWS quickly and securely. You can use DMS to migrate your data to and from most widely used commercial and open-source databases such as Oracle, PostgreSQL, MySQL, MariaDB, and Amazon Aurora. DMS can also be used to migrate data between on-premises and cloud databases.


## Project Overview

In this project, we will use AWS Database Migration Service (DMS) to migrate a database to Amazon RDS. We will create a simple MySQL database on an EC2 instance and then migrate it to an Amazon RDS MySQL instance using DMS.


### Steps to migrate a database to Amazon RDS using DMS

There are six main steps to migrate a database to Amazon RDS using DMS:

1. [Setup the source database](#a-setup-the-source-database) - This is not required if your already have a database. We will create a simple MySQL DB on an EC2 instance for this tutorial.
2. [Setup the target database](#b-setup-the-target-database) - We will create an Amazon RDS MySQL instance.
3. [Create a replication instance](#c-create-a-replication-instance) - This is a managed service that you can use to migrate your data to and from most widely used commercial and open-source databases such as Oracle, PostgreSQL, MySQL, MariaDB, and Amazon Aurora.
4. [Create source and target endpoints](#d-create-source-and-target-endpoints) - These are the source and target databases that needs to be identified in DMS for migration.
5. [Create a migration task](#e-create-a-migration-task) - This is the final step where you create a migration task to migrate the data from the source to the target database.
6. [Execute the migration task](#e-create-a-migration-task) - This is the final step where you execute the migration task to migrate the data from the source to the target database.

Furthermore, we will:
7. [Test the migration](#g-testing-the-migration) - We will test the migration by connecting to the RDS instance using a MySQL client and check if the data is migrated.
8. [Configure Change Data Capture (CDC)](#h-configure-change-data-capture-cdc) - We will configure Change Data Capture (CDC) to capture changes in the source database and replicate them to the target database.
9. [Test the replication with CDC](#i-test-the-replication-with-cdc) - We will test the replication with CDC by making changes to the source database and checking if the changes are replicated to the target database.


## A: Setup the source database

We will create a simple MySQL database on an EC2 instance for this tutorial. You can skip this step if you already have a database.
The EC2 instance will be in us-east-2 (Ohio) region with Ubuntu 22.04 and the MySQL database will be installed on it.

### Launch an EC2 instance
To begin, go to the EC2 dashboard and click on "Launch Instance".

**Step 1**: Set the "Name and tags" as "MySQL-Source-DB".
![Set the "Name and tags" as "MySQL-Source-DB"](/images/47500000-01-create-ec2-name.png)

**Step 2**: Choose an Amazon Machine Image (AMI). We will choose Ubuntu Server 22.04 LTS (HVM), SSD Volume Type. Ensure that "Architecture" is "64-bit (x86)".
![Choose an Amazon Machine Image (AMI)](/images/47500000-02-create-ec2-os.png)

**Step 3**: Choose an Instance Type. We will choose "t2.micro".
![Choose an Instance Type](/images/47500000-03-create-ec2-instance-type.png)

**Step 4**: Create or reuse a Key Pair to login.
![Create or reuse a Key Pair to login](/images/47500000-04-create-ec2-key-pair.png)

**Step 5**: In the "Network settings", click on "Edit" button and select the default VPC. Leave the subnet as default and select "Auto-assign Public IP" as "Enable".
![In the "Network settings", click on "Edit" button and select the default VPC](/images/47500000-05-create-ec2-network-settings.png)

**Step 6**: Select "Create security group" and set the "Security group name" as "source-db-sg". Add two rules: one for SSH and one for MySQL/Aurora. You can leave the source as "Anywhere" for this tutorial.
![Select "Create security group" and set the "Security group name" as "source-db-sg"](/images/47500000-06-create-ec2-security-group.png)

**Step 7**: In the "Configure storage" section, leave the default settings and click "Next".
![In the "Configure storage" section, leave the default settings and click "Next"](/images/47500000-07-create-ec2-storage.png)

**Step 8**: Click on the "Launch instance" button to launch the EC2 instance.

### Setting up the MySQL database

After the instance is created, connect to the EC2 instance using either SSH or "EC2 Instance Connect" by clicking on the "Connect" button in EC2 dashboard.


**Step 1**: Login to the EC2 instance, to login using SSH, use the following command:
```bash
ssh -i "your-key.pem" ubuntu@instance-public-ip
```

**Step 2**: Update the package list and install MySQL server using the following commands:
```bash
sudo apt update && sudo apt upgrade -y
```

**Step 3**: Install MySQL server using the following command:
```bash
sudo apt install mysql-server -y
```

**Step 4**: Start the MySQL service and enable it to start on boot using the following commands:
```bash
sudo systemctl start mysql && sudo systemctl enable mysql
```

**Step 5**: Start and secure the MySQL installation using the following command:
```bash
sudo mysql_secure_installation
```

The following questions will be asked:

- **Would you like to setup VALIDATE PASSWORD component? (Press y|Y for Yes, any other key for No)** : Press "y" for "Yes".
- **There are three levels of password validation policy: LOW, MEDIUM, and STRONG. Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG** : Press "0" for "LOW".
- **Remove anonymous users? (Press y|Y for Yes, any other key for No) :** Press "y" for "Yes".
- **Disallow root login remotely? (Press y|Y for Yes, any other key for No) :** Press "y" for "Yes".
- **Remove test database and access to it? (Press y|Y for Yes, any other key for No) :** Press "y" for "Yes".
- **Reload privilege tables now? (Press y|Y for Yes, any other key for No) :** Press "y" for "Yes".

**Step 6**: Login to the MySQL server using the following command:
```bash
sudo mysql -u root -p
```
When asked for a password, press "Enter" as we have not set a password yet.

**Step 7**: Create a new database and a new user using the following commands:
```sql
CREATE DATABASE source_db;
CREATE USER 'source_user'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON source_db.* TO 'source_user'@'%';
FLUSH PRIVILEGES;
```
Replace `password` with a your password. We will use this user to connect to the source database from the DMS service. 

> Note: The "source_user" is created with the wildcard "%" to allow connections from any IP address. This is not recommended for production environments. In production, you should limit the connections to specific IP addresses.

!!! tip
    We disabled connecting with the root user remotely in Step 5 hence we have to create a new user to connect remotely the source database. This is a security best practice.

**Step 8**: Allow remote connection to the MySQL server by editing the MySQL configuration file using the following command:
```bash
sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf
```
Scroll down to the "bind-address" line, currently this is set to "127.0.0.1" which is the localhost and limits connections only from localhost. Change this to "0.0.0.0" to allow connections from any IP address. Save and close the file.

This change is required to allow the DMS service to connect to the MySQL server.

!!! warning
    It is not recommended to allow connections from any IP address in a production environment. Ideally, you should configure the "bind-address" to the IP address of the DMS replication instance.


**Step 9**: Restart the MySQL service using the following command:
```bash
sudo systemctl restart mysql
```

### Load sample data into the source database

We will load some sample data into the source database to migrate it to the target database.

**Step 1**: Login to the MySQL server using the following command:
```bash
sudo mysql -u source_user -p
```
When asked for a password, enter the password you set for the "source_user".

**Step 2**: Check if the "source_db" database exists using the following command:
```sql
SHOW DATABASES;
```
![Check if the "source_db" database exists](/images/47500000-08-create-ec2-mysql-show-databases.png)

**Step 3**: Use the "source_db" database using the following command:
```sql
USE source_db;
```

**Step 4**: Create a table a new table named "pets" using the following command:
```sql
CREATE TABLE pets (
    name VARCHAR(20),
    owner VARCHAR(20),
    breed VARCHAR(20),
    gender CHAR(1),
    birth DATE,
    death DATE
);
```

**Step 5**: Insert some sample data into the "pets" table using the following command:
```sql
INSERT INTO pets VALUES ('Puffball','Diane','hamster','f','2001-03-30', '2005-04-30');
INSERT INTO pets VALUES ('Fluffy','Harold','cat','f','2005-02-04', NULL);
INSERT INTO pets VALUES ('Claws','Gwen','dog','m','2015-03-17', '2021-08-24');
INSERT INTO pets VALUES ('Buffy','Harold','dog','f','1989-05-13', '1997-06-23');
INSERT INTO pets VALUES ('Fang','Benny','dog','m','1990-08-27', '1998-08-27');
INSERT INTO pets VALUES ('Bowser','Diane','dog','m','1979-08-31', '1995-07-29');
INSERT INTO pets VALUES ('Chirpy','Gwen','bird','f','1998-09-11', NULL);
INSERT INTO pets VALUES ('Whistler','Gwen','bird','m', '1997-12-09', NULL);
INSERT INTO pets VALUES ('Slim','Benny','snake','m','1996-04-29', '2000-06-29');
INSERT INTO pets VALUES ('Snowball','Diane','cat','f','1999-03-30', '2015-04-30');
```

**Step 6**: Check if the data is inserted into the "pets" table using the following command:
```sql
SELECT * FROM pets;
```

![Check if the data is inserted into the "pets" table](/images/47500000-09-create-ec2-mysql-show-pets.png)

The source database is now setup and ready for migration.

## B: Setup the target database

We will create an Amazon RDS MySQL instance as the target database. The RDS instance can be in a different region than the source database, for this tutorial, we will keep it in Ohio region.

### Create an Amazon RDS MySQL instance

To begin, go to the RDS dashboard and click on "Create database".

**Step 1**: Choose a database creation method. Select "Standard Create".

**Step 2**: Select the database engine. Select "MySQL".

**Step 3**: In the "Templates" section, select "Free tier".

**Step 4**: In the "Settings" section, set the "DB instance identifier" as "mysql-target-db". Leave the "Master username" as "admin" and set as Master password of your choice.

**Step 5**: In the "Instance configuration" section, select "Burstable classes" and "db.t3.micro".

**Step 6**: In the "Storage" section, leave the default settings and click "Next".

**Step 7**: Change the "Public access" as "Yes". This will allow the DMS service to connect to the RDS instance.

**Step 8**: Scroll down to the bottom and click "Create database".

## C: Create a replication instance

We will create a replication instance which is a managed service that you can use to migrate your data to and from most widely used commercial and open-source databases such as Oracle, PostgreSQL, MySQL, MariaDB, and Amazon Aurora.

To begin, navigate to DMS and on the left panel click on "Replication instances" under "Migrate data" section.

**Step 1**: To begin, navigate to DMS and on the left panel click on "Replication instances" under "Migrate data" section.

**Step 2**: Click on "Create replication instance".

**Step 3**: In the "Settings" section, set the "Name" to "mysql-replication-instance". Leave the others as blank.
![In the "Settings" section, set the "Name" to "mysql-replication-instance"](/images/47500000-10-create-replication-instance.png)

**Step 4**: In the "Instance configuration", leave "Instance class" and "Engine version" unchanged. And set the "High availability" as "Dev or test workload (Single-AZ)".
![In the "Instance configuration", leave "Instance class" and "Engine version" unchanged](/images/47500000-11-create-replication-instance-config.png)

**Step 5**: Leave "Storage" and "Connectivity" settings unchanged and click "Create replication instance".

## D: Create source and target endpoints

We will create source and target endpoints which are the source and target databases that needs to be identified in DMS for migration.

### Create a source endpoint
**Step 1**: To begin, navigate to DMS and on the left panel click on "Endpoints" under "Migrate data" section.

**Step 2**: Click on "Create endpoint".

**Step 3**: In "Endpoint type", select "Source".

**Step 4**: In Endpoint identifier, set the "Endpoint identifier" as "mysql-source-endpoint". Choose "Source engine" as "MySQL".

**Step 5**: Now you should see a "Access to endpoint" subsection. Select "Provide access information manually". Set the "Server name" as the public IP of the EC2 instance where the source database is running. Set the "Port" as "3306". Set the "Username" as "source_user" and "Password" as the password you set for the "source_user".
![Set the "Server name" as the public IP of the EC2 instance where the source database is running](/images/47500000-12-create-source-endpoint.png)


**Step 6**: Click on the "Endpoint settings" and check "Use endpoint connection attributes". In "Extra connection attributes" enter `initstms=SET FOREIGN_KEY_CHECKS=0;`, then scroll to bottom and click "Create endpoint".
![Click on the "Endpoint settings" and check "Use endpoint connection attributes"](/images/47500000-13-create-source-endpoint-config.png)

!!! note 
    The `initstms=SET FOREIGN_KEY_CHECKS=0;` is used to disable foreign key checks during the migration. This is required as the DMS service will migrate the data in a specific order and foreign key checks can cause issues during the migration.

**Step 7**: Once the source endpoint is created, you should see the status as "Active". Click on the "mysql-source-endpoint" to see the details. Move to the "Connections" tab and click on "Test connections". In the subsequent page, click on "Run test" to test if the Replication instance can access the source database.

![Click on the "mysql-source-endpoint" to see the details](/images/47500000-14-create-source-endpoint-test-connection.png)


### Create a target endpoint

**Step 1**: Click on "Create endpoint".

**Step 2**: In "Endpoint type", select "Target". Since we are migrating to an RDS instance, select "RDS" as the target engine. Then choose the RDS instance that we created earlier as the target endpoint.

![Choose the RDS instance that we created earlier as the target endpoint](/images/47500000-15-create-target-endpoint.png)

**Step 3**: In "Endpoint configuration", set the "Endpoint identifier" as "mysql-target-endpoint". Choose the Target engine as "MySQL".
![In "Endpoint configuration", set the "Endpoint identifier" as "mysql-target-endpoint"](/images/47500000-16-create-target-endpoint-config.png)

**Step 4**: Change the "Access to endpoint" settings as "Provide access information manually". The "Server name" and port should populate automatically. Set the "Username" as "admin" and "Password" as the password you set for the RDS instance.

**Step 5**: Leave the "SSL mode" as "None" and click "Create endpoint".
![Change the "Access to endpoint" settings as "Provide access information manually"](/images/47500000-17-create-target-endpoint-config.png)

!!! note
    If the full load migration fails then try setting the "Extra connection attributes" as `initstms=SET FOREIGN_KEY_CHECKS=0;` as we did for the source endpoint.


**Step 6**: Once the target endpoint is created, you should see the status as "Active". Click on the "mysql-target-endpoint" to see the details. Move to the "Connections" tab and click on "Test connections". In the subsequent page, click on "Run test" to test if the Replication instance can access the source database.

![Click on the "mysql-target-endpoint" to see the details](/images/47500000-18-create-target-endpoint-test-connection.png)


## E: Create a migration task

We will create a migration task to migrate the data from the source to the target database.

**Step 1**: To begin, navigate to DMS and on the left panel click on "Database migration tTasks" under "Migrate data" section.

**Step 2**: Click on "Create task".

**Step 3**: In the "Task configuration" section, 

- set the "Task identifier" as "mysql-migration-task". 
- Choose the "Replication instance" as "mysql-replication-instance".
- Choose the "Source endpoint" as "mysql-source-endpoint".
- Choose the "Target endpoint" as "mysql-target-endpoint".
- Choose the "Migration type" as "Migrate existing data".

![In the "Task configuration" section, set the "Task identifier" as "mysql-migration-task"](/images/47500000-19-create-migration-task.png)


**Step 4**: Leave the "Task settings" as default.

**Step 5**: Under "Table mappings", click on "Add new selection rule". Set the "Schema name" as "source_db" and "Table name" as "pets". Leave the "Action" as "Include".

**Step 6**: Under "Premigration assessment", uncheck "Turn on premigration assessment".

**Step 7**: Leave "Migration task startup configuration" as "Automatically on create" and then click "Create task".

## F: Execute the migration task

If you had selected "Migration task startup configuration" as "Automatically start on create", the task will execute automatically. Based on the amount of data, the migration task can take some time to complete. Once the task is started, you should see the status as "Load complete".

![Based on the amount of data, the migration task can take some time to start](/images/47500000-20-create-migration-task-status.png)

## G: Testing the migration

To test the migration, connect to the RDS instance using a MySQL client and check if the data is migrated.

**Step 1**: Connect to the RDS instance using a MySQL client. You can use the MySQL Workbench or any other MySQL client. 

You can use the following command to connect to the RDS instance using the MySQL client:
```bash
mysql -h <rds-endpoint> -u admin -p
```
Replace `<rds-endpoint>` with the endpoint of the RDS instance. Enter the password when prompted.

**Step 2**: Check if the "source_db" database exists using the following command:
```sql
SHOW DATABASES;
```
![Check if the "source_db" database exists](/images/47500000-21-create-rds-mysql-show-databases.png)

**Step 3**: Use the "source_db" database using the following command:
```sql
USE source_db;
```

**Step 4**: Check if the data is inserted into the "pets" table using the following command:
```sql
SELECT * FROM pets;
```
![Check if the data is inserted into the "pets" table](/images/47500000-22-create-rds-mysql-show-pets.png)

The data is successfully migrated from the source database to the target database.

## H: Configure Change Data Capture (CDC)

Change Data Capture (CDC) is a feature of DMS that captures changes in the source database and replicates them to the target database. CDC can be used to keep the target database in sync with the source database.

To configure CDC, you need to create a new migration task with CDC enabled. But first, let's login to the target RDS instance and delete the changes made by DMS.

### Reset the target RDS instance

**Step 1**: Connect to the RDS instance using a MySQL client.
You can use the following command to connect to the RDS instance using the MySQL client:
```bash
mysql -h <rds-endpoint> -u admin -p
```
Replace `<rds-endpoint>` with the endpoint of the RDS instance. Enter the password when prompted.

**Step 2**: See the databases that are present in the RDS instance using the following command:
```sql
SHOW DATABASES;
```
![See the databases that are present in the RDS instance](/images/47500000-23-create-rds-mysql-show-databases.png)

**Step 3**: Delete the `awsdms_control` and `source_db` databases using the following commands:
```sql
DROP DATABASE awsdms_control;
DROP DATABASE source_db;
```

### Create a migration task with CDC enabled

**Step 1**: To begin, navigate to DMS and on the left panel click on "Database migration Tasks" under "Migrate data" section.

**Step 2**: Click on "Create task".

**Step 3**: In the "Task configuration" section,
- set the "Task identifier" as "mysql-cdc-migration-task".
- Choose the "Replication instance" as "mysql-replication-instance".
- Choose the "Source endpoint" as "mysql-source-endpoint".
- Choose the "Target endpoint" as "mysql-target-endpoint".
- Choose the "Migration type" as "Migrate existing data and replicate ongoing changes".

**Step 4**: Leave the "Task settings" as default.

**Step 5**: Under "Table mappings", click on "Add new selection rule". Set the "Schema name" as "source_db" and "Table name" as "pets". Leave the "Action" as "Include".

**Step 6**: Under "Premigration assessment", uncheck "Turn on premigration assessment".

**Step 7**: Leave "Migration task startup configuration" as "Automatically on create" and then click "Create task".

At this point you the full load migration will start and once it is complete, the CDC will start capturing changes in the source database and replicating them to the target database.

You can check if the full load migration is complete by following the steps in the "[Testing the migration](#g-testing-the-migration)" section.

## I: Test the replication with CDC

We will test the replication with CDC by making changes to the source database and checking if the changes are replicated to the target database.

**Step 1**: Connect to the source database using a MySQL client. You can use the following command to connect to the source database using the MySQL client:
```bash
mysql -h <source-endpoint> -u source_user -p
```

**Step 2**: Insert a new record into the "pets" table using the following command:
```sql
USE source_db;
INSERT INTO pets VALUES ('Scooby','Shaggy','dog','m','2000-03-30', NULL);
```

**Step 3**: Go back to the RDS instance and check if the new record is replicated to the target database using the following command:
```sql
USE source_db;
SELECT * FROM pets;
```
![Check if the new record is replicated to the target database](/images/47500000-24-create-rds-mysql-show-pets-cdc.png)

The new record is successfully replicated from the source database to the target database using CDC.

---

# Run Selenium and Chrome on WSL2 using Python and Selenium webdriver

URL: https://cloudbytes.dev/snippets/run-selenium-and-chrome-on-wsl2
Category: Snippets
Published: 2023-12-05
Author: Rehan Haider
Tags: ubuntu, selenium, wsl2, python

> A guide to installating, configuring and running Selenium and Chrome for Testing on Windows Subsystem for Linux (WSL2) and run tests using Python and Selenium webdriver.

[TOC]


[Selenium](https://www.selenium.dev/) combined with Headless [Chrome](https://developers.google.com/web/updates/2017/04/headless-chrome) used to be great tool for creating automated UI tests for web applications. 

Google recently launched [Chrome for Testing](https://developer.chrome.com/blog/chrome-for-testing/), a version of Chrome specifically designed for automated testing. It is available for Windows, Linux and Mac.

This solves a major issue that developers had with finding a compatible versions of Chrome and Chromedriver.

With Selenium libraries, Python can be used to create and run automated browser-based tests & tasks. 
This guide will show you how to install, configure and run Selenium and Chrome on WSL2 using Python and Selenium webdriver.

## Step 1: Install WSL2

On Windows 10 version 2004 or higher (Build 19041 and above) or windows 11, run the below. 

```powershell
wsl --install
```
This will take care of all the steps required, i.e.

1. Enable Windows Virtualisation Layer and WSL2
2. Update the Linux kernel to the latest version
3. Install the default Linux distribution, i.e. latest Ubuntu (Currently Ubuntu 20.04)

![Install WSL](/images/99999966-install-wsl.gif)

Then type `wsl` in your terminal and press enter to login to WSL2. 

> !!! warning " NOTE: All codeblocks below are formatted as multi-line commands so the entire block needs to be copy pasted and not line by line."

Ensure you go to your home directory, update the repository and any packages

**a) Change the working directory to the user home directory.**
```bash
cd "$HOME"
```

**b) Update the repository and any packages**
```bash
sudo apt update && sudo apt upgrade -y
```

## Step 2: Install latest Chrome for Testing (for Linux)

This version of Chrome is not available in Ubuntu's official APT repository, so we will download the zipped files directly from Google and install it. 

**a) Download the latest Chrome file**
```bash
meta_data=$(curl 'https://googlechromelabs.github.io/chrome-for-testing/\
last-known-good-versions-with-downloads.json') /
wget $(echo "$meta_data" | jq -r '.channels.Stable.downloads.chrome[0].url')
```

**b) Install Chrome dependencies**
```bash
sudo apt install ca-certificates fonts-liberation \
    libappindicator3-1 libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 \
    libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 \
    libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 \
    libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
    libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 \
    libxrandr2 libxrender1 libxss1 libxtst6 lsb-release wget xdg-utils -y
```

**c) Install/Unzip Chrome**
The downloaded zip file contains unpackaged Chrome binary files. We just need to unpack them.

```bash
unzip chrome-linux64.zip
```

**c) Check if you Chrome is working**
```bash
./chrome-linux64/chrom --version
```
![chrome-version](/images/99999966-03-chrome-version.png)



## Step 3: Install compatible Chromedriver
Thankfully, the process of downloading and installing Chrome driver has become much simpler. We can use the same JSON API to get the compatible version of Chromedriver.

**b) Download the latest Chromedriver**
```bash
meta_data=$(curl 'https://googlechromelabs.github.io/chrome-for-testing/\
last-known-good-versions-with-downloads.json') /
wget $(echo "$meta_data" | jq -r '.channels.Stable.downloads.chromedriver[0].url')
```


**d) Unzip the binary file**
```bash
unzip chromedriver-linux64.zip
```

## Step 4: Configure Python and Install Selenium

Selenium webdriver is available as a Python package, but before installation we need to do some prep.

### Configure a Python virtual environment

Next, we need to install `venv`, choose the Python version based on what you have installed.
```
sudo apt install python3-venv -y
```

Then create a virtual environment
```bash
python3 -m venv .venv
```

Finally, activate the virtual environment by running 
```bash
source .venv/bin/activate
```

You should see your terminal change to the below with `(.venv)` in the prompt.

### Install Selenium

After activating the virtual environment, install Selenium using the pip command.

```bash
pip install selenium
```

>!!! tip "It is necessary to activate the Python virtual environment by running `source .venv/bin/activate` before executing the above command"


## Step 5: Run Selenium

Finally we're ready to start running our automated tests. We write a simple Python script to run Selenium and Chrome/Chromium.

Create a new folder, `selenium` and open VSCode by running the below
```bash
mkdir -p "selenium" && cd "selenium" && code .
```

**a) The Python program**

```python
"""
# Filename: run_selenium.py
"""

## Run selenium and chrome driver to scrape data from cloudbytes.dev
import time
import os.path
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

## Setup chrome options
chrome_options = Options()
chrome_options.add_argument("--headless") # Ensure GUI is off
chrome_options.add_argument("--no-sandbox")

# Set path to chrome/chromedriver as per your configuration
homedir = os.path.expanduser("~")
chrome_options.binary_location = f"{homedir}/chrome-linux64/chrome"
webdriver_service = Service(f"{homedir}/chromedriver/stable/chromedriver")

# Choose Chrome Browser
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)

# Get page
browser.get("https://cloudbytes.dev")

# Extract description from page and print
description = browser.find_element(By.NAME, "description").get_attribute("content")
print(f"{description}")

#Wait for 10 seconds
time.sleep(10)
browser.quit()
```

**b) Run the program**

Go back to your terminal and type `python3 selenium/run_selenium.py`

![run-selenium](/images/99999966-run-selenium.png)


## Creating a script to automate the process

We can merge the above steps to create a simple bash script to help you automate this entire process. Save this to a file named `install-selenium.sh` and then make it executable by running `chmod +x install-selenium`.


```bash
#!/usr/bin/bash

echo "Changing to home directory..."
pushd "$HOME"

echo "Update the repository and any packages..."
sudo apt update && sudo apt upgrade -y

echo "Install prerequisite system packages..."
sudo apt install wget curl unzip jq -y

# Set metadata for Google Chrome repository...
meta_data=$(curl 'https://googlechromelabs.github.io/chrome-for-testing/'\
'last-known-good-versions-with-downloads.json')


echo "Download the latest Chrome binary..."
wget $(echo "$meta_data" | jq -r '.channels.Stable.downloads.chrome[0].url')

echo "Install Chrome dependencies..."
sudo apt install ca-certificates fonts-liberation \
    libappindicator3-1 libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 \
    libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 \
    libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 \
    libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
    libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 \
    libxrandr2 libxrender1 libxss1 libxtst6 lsb-release wget xdg-utils -y


echo "Unzip the binary file..."
unzip chrome-linux64.zip


echo "Downloading latest Chromedriver..."
wget $(echo "$meta_data" | jq -r '.channels.Stable.downloads.chromedriver[0].url')

echo "Unzip the binary file and make it executable..."
unzip chromedriver-linux64.zip

echo "Install Selenium..."
python3 -m pip install selenium

echo "Removing archive files"
rm chrome-linux64.zip  chromedriver-linux64.zip

popd
```

Alternatively, this script is also available on [GitHub as a repository](https://github.com/rehanhaider/selenium-wsl2-ubuntu.git).

---

# Add CORS configuration to a S3 bucket using AWS CDK

URL: https://cloudbytes.dev/aws-academy/add-cors-configuration-to-a-s3-bucket-using-aws-cdk
Category: AWS Academy
Published: 2023-11-05
Author: Rehan Haider
Tags: aws, cdk, python

> Learn how to setup and configure CORS for S3 buckets using CDK

Cross-Origin Resource Sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served. 

Since all S3 buckets use `https://s3.amazonaws.com` as the domain, you need to configure CORS to allow access from other domains so be able to access the objects in the S3 bucket.

E.g. if a user tries to upload a file to an S3 bucket from a web page hosted on `https://example.com`, the browser will block the request unless CORS is configured to allow access from `https://example.com`.


## Setup CORS for S3 bucket

You can configure CORS on the S3 bucket by using the `cors` property of the `Bucket` construct.


```python
# filename: cdk_app/s3_stack.py
from aws_cdk import (
    Stack,
    aws_s3 as s3,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Sset the CORS configuration
            cors=[
                {
                    "allowedMethods": [
                        s3.HttpMethods.PUT,
                    ],
                    "allowedOrigins": ["https://www.example.com"],
                    "allowedHeaders": ["*"],
                }
            ],
        )
```

In the above code, we have set the CORS configuration to allow PUT requests from `https://www.example.com` with any headers. This will allow users to upload files to the S3 bucket from `https://www.example.com`.


## Configure multiple CORS rules

You can configure multiple CORS rules by adding multiple dictionaries to the `cors` property.

```python
# filename: cdk_app/s3_stack.py
from aws_cdk import (
    Stack,
    aws_s3 as s3,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Sset the CORS configuration
            cors=[
                {
                    "allowedMethods": [
                        s3.HttpMethods.PUT,
                    ],
                    "allowedOrigins": ["https://www.example.com"],
                    "allowedHeaders": ["*"],
                },
                {
                    "allowedMethods": [
                        s3.HttpMethods.GET,
                    ],
                    "allowedOrigins": ["https://www.example.com"],
                    "allowedHeaders": ["*"],
                },
            ],
        )

---

# How to create a lambda function using AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/how-to-create-a-lambda-function-using-aws-cdk-in-python
Category: AWS Academy
Published: 2023-11-05
Author: Rehan Haider
Tags: aws, cdk, python

> A complete guide to creating a lambda function using AWS CDK

The easiest way to create a lambda function using AWS CDK is to use the `Function` construct from the `aws_lambda` module. Let's see how to use it.

## Create a simple lambda function

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)

from constructs import Construct


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        inline_code = """
def handler(event, context):
    return {
        "statusCode": 200,
        "body": "Hello from Lambda!"
    }
    """
        # 👆🏽 formatting & indentation is not a mistake

        my_lambda = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="index.handler",
            code=_lambda.Code.from_inline(inline_code),
        )

```

You can create the app by modifying the `app.py` file as follows:

```python
# filename: app.py

import aws_cdk as cdk
from cdk_app.lambda_stack import LambdaStack

app = cdk.App()

lambda_stack = LambdaStack(app, "LambdaStack")

app.synth()
```

To deploy, run `cdk deploy`. 

Once deployed, you can go to the AWS Console and check the lambda function. You should see the following:

![Lambda inline function](/images/50002000-01-fn-inline-code.gif)

### What's happening here?

We used what is known as an **inline code** to create the lambda function. Effectively, the code we want to execute is passed as a string to the `code` property of the `Function` construct. 

The `Function` construct takes the following parameters:

- `scope`: The scope in which the construct is created. In our case, it is the `LambdaStack` class.
- `id`: The ID of the construct. This ID is used within CDK to uniquely identify the construct. It is also used as the logical ID in CloudFormation templates.
- `runtime`: The runtime environment for the lambda function. In our case, it is Python 3.10. Other versions of Python are also supported. You can also use other runtimes such as Node.js, Java, Go, etc. 
- `handler`: The name of the handler function. In our case, it is `index.handler`. This means that the `handler` function is defined in the `index.py` file.
- `code`: The code to execute. In our case, it is the `inline_code` variable.


But using inline code is not the best way to create a lambda function. It is fine for simple demonstrations, but for anything more complex, you should use a file or a Docker image. Let's see how to do that.


## Create a lambda function from a file

To create a lambda function from a file, you need to use the `Code.from_asset` method. This method takes the path to the file as an argument. 

Now one important point, the lambda code needs to be in its own directory. So, you need to create a directory and put the lambda code in that directory.

E.g. in our case, we will create a `cdk_app/lambda` directory and create a file named `index.py` in that directory. The contents of the file will be as follows:

```python
# filename: cdk_app/lambda/index.py

def handler(event, context):
    return {
        "statusCode": 200,
        "body": "Hello from Lambda!",
    }
```

Now, we can create the lambda function as follows:

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)

from constructs import Construct


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_lambda = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="index.handler",
            code=_lambda.Code.from_asset("cdk_app/lambda"),
        )
```

Notice that we have used the `Code.from_asset` method to create the lambda function. We have passed the path to the `lambda` directory as an argument to the method.

Now, when you run `cdk deploy`, you will see that the lambda function is created from the `index.py` file.

## Set environment variables

You can set environment variables for the lambda function by using the `environment` property of the `Function` construct.

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)

from constructs import Construct


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_lambda = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="index.handler",
            code=_lambda.Code.from_asset("cdk_app/lambda"),
            # 👇🏽 set environment variables
            environment={
                "ENVIRONMENT": "PROD",
            },
        )
```

## Set memory size and timeout

By default lambda provide 128 MB of memory and 3 seconds of timeout. That means if your lambda function takes more than 3 seconds to execute, it will timeout.

If your lambda function requires more memory or more time, you can set the `memory_size` and `timeout` properties of the `Function` construct.


```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
    Duration,
)

from constructs import Construct


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_lambda = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="index.handler",
            code=_lambda.Code.from_asset("cdk_app/lambda"),
            # 👇🏽 set memory size and timeout
            memory_size=256,
            timeout=Duration.seconds(10),
        )
```


## Additional resources for lambda functions

Or learn some advanced methods on how to create lambda functions:

1. [Import an existing lambda function]({filename}50002010-cdk-fn-import-lambda.md)
2. [Using lambda layers]({filename}50002020-cdk-fn-lambda_layers.md)
3. [Using Lambda PythonFunction to create lambda functions]({filename}50002030-cdk-fn-lambda-python-deps.md)
4. [Running Lambda using custom Docker container]({filename}50002040-cdk-fn-lambda-aws-docker.md)

---

# How to import an existing lambda function using AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/how-to-import-an-existing-lambda-function-using-aws-cdk-in-python
Category: AWS Academy
Published: 2023-11-05
Author: Rehan Haider
Tags: aws, cdk, python

> A guide to importing an existing lambda function using AWS CDK in Python and applying changes to it

We learnt [how to create a new lambda function using AWS CDK in Python]({filename}50002000-cdk-fn-create-lambda.md) in the previous post. 

But in a lot of cases, you might already have a lambda function that was created elsewhere and you want to import into your CDK app and apply changes to it. In this post, we will see how to do that.

There are three ways to import an existing lambda function, can use:

1. `from_function_arn`: This relies on the ARN of the lambda function.
2. `from_function_name`: You can also import the lambda function using its name.
3. `from_function_attributes`: This is typically used to import a lambda function that was created in a different AWS account or a different region.


## Import an existing lambda function using its ARN

To import an existing lambda function using its ARN, you can use the `from_function_arn` method of the `Function` construct. Let's see how to do that.

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)

from constructs import Construct


class LambdaStack(Stack):

    FN_ARN = "arn:aws:lambda:us-east-1:123456789012:function:my-lambda"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_lambda = _lambda.Function.from_function_arn(
            self,
            id="MyLambda",
            function_arn=self.FN_ARN,
        )
```

The `from_function_arn` method takes only three parameters:

1. `scope`: The scope of the construct. In this case, it is the `LambdaStack` class.
2. `id`: The ID that will be used within CDK/CloudFormation. In this case, we set it to `MyLambda` but it could be anything.
3. `function_arn`: The ARN of the lambda function that you want to import.

## Import an existing lambda function using its name


To import an existing lambda function using its name, you can use the `from_function_name` method of the `Function` construct. Let's see how to do that.

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)

from constructs import Construct


class LambdaStack(Stack):

    FN_NAME = "my-lambda"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_lambda = _lambda.Function.from_function_name(
            self,
            id="MyLambda",
            function_name=self.FN_NAME,
        )
```

The `from_function_name` method takes only three parameters:

1. `scope`: The scope of the construct. In this case, it is the `LambdaStack` class.
2. `id`: The ID that will be used within CDK/CloudFormation. In this case, we set it to `MyLambda` but it could be anything.
3. `function_name`: The name of the lambda function that you want to import.


## Import an existing lambda function using its attributes

This is typically used to import a lambda function when you have the Lambda function's ARN and you also need to specify or override other attributes. E.g. for instance, if you need to specify the execution role's ARN because you want to grant permissions or interact with the role in some way within your CDK app, this method would be more suitable.

To import an existing lambda function using its attributes, you can use the `from_function_attributes` method of the `Function` construct. Let's see how to do that.

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
    aws_iam as iam,
)

from constructs import Construct


class LambdaStack(Stack):
    FN_ARN = "arn:aws:lambda:us-east-1:123456789012:function:my-lambda"
    ROLE_ARN = "arn:aws:iam::123456789012:role/my-lambda-role"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_role = iam.Role.from_role_arn(
            self,
            id="MyRole",
            role_arn=self.ROLE_ARN,
        )

        my_lambda = _lambda.Function.from_function_attributes(
            self,
            id="MyLambda",
            function_arn=self.FN_ARN,
            role=my_role,
        )
```

In the above we provided the `role` parameter to the `from_function_attributes` method. This is because we wanted to override the execution role of the lambda function. To do that, we first imported the role using the `Role.from_role_arn` method.

---

# Import an existing S3 bucket in CDK

URL: https://cloudbytes.dev/aws-academy/import-an-existing-s3-bucket-in-cdk
Category: AWS Academy
Published: 2023-11-05
Author: Rehan Haider
Tags: aws, cdk, python

> How to import an existing S3 bucket using different methods in CDK using Python

We learnt how to [create an S3 bucket using CDK]({filename}50001000-cdk-s3-create-s3-bucket.md) in the previous guide. In this guide, we will learn how to import an existing S3 bucket in CDK using Python.

## Import an existing S3 bucket

There are three ways to import an existing S3 bucket in CDK:

1. `Bucket.from_bucket_name`: Import the bucket using bucket name
2. `Bucket.from_bucket_arn`: Import the bucket using bucket ARN
3. `Bucket.from_bucket_attributes`: Import the bucket using bucket attributes


### Import an existing S3 bucket using bucket name

You can import an existing S3 bucket using the `from_bucket_name` method of the `Bucket` construct. 

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"
    BUCKET_NAME = "my-existing-bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket.from_bucket_name(
            self,
            id=self.BUCKET_ID, # 👈🏽 Used to identify the bucket within CDK
            bucket_name=self.BUCKET_NAME, # 👈🏽 Name of the existing bucket
        )
```


### Import an existing S3 bucket using bucket ARN

If you have the ARN of the existing S3 bucket, e.g. from [another stack]({filename}50000050-cdk-multiple-stacks.md), you can import the bucket using the `from_bucket_arn` method of the `Bucket` construct. 

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"
    BUCKET_ARN = "arn:aws:s3:::my-existing-bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket.from_bucket_arn(
            self,
            id=self.BUCKET_ID, # 👈🏽 Used to identify the bucket within CDK
            bucket_arn=self.BUCKET_ARN, # 👈🏽 ARN of the existing bucket
        )
```

### Import an existing S3 bucket using bucket attributes

The above two methods are useful as long as the bucket is in the same region as the stack. If the bucket is in a different region, you can use the `from_bucket_attributes` method of the `Bucket` construct. 

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"
    BUCKET_NAME = "my-existing-bucket"
    BUCKET_REGION = "us-east-1"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket.from_bucket_attributes(
            self,
            id=self.BUCKET_ID, # 👈🏽 Used to identify the bucket within CDK
            bucket_name=self.BUCKET_NAME, # 👈🏽 Name of the existing bucket
            region=self.BUCKET_REGION, # 👈🏽 Region of the existing bucket
        )
```

---

# Manage Python dependencies in AWS Lambda using AWS CDK

URL: https://cloudbytes.dev/aws-academy/manage-python-dependencies-in-aws-lambda-using-aws-cdk
Category: AWS Academy
Published: 2023-11-05
Author: Rehan Haider
Tags: aws, cdk, python

> We look at how to package, install and manage Python dependencies in AWS Lambda using AWS CDK

We looked at how we can [install Python packages beyond the ones available by default in AWS Lambda]({filename}50002020-cdk-fn-lambda_layers.md#create-a-lambda-layer-in-aws-cdk-using-python-to-handle-dependencies). 

But I have always found it a bit cumbersome and unelegant to use Lambda layers to handle dependencies. Intead, I will show you an alternative way to handle Python dependencies in AWS Lambda using AWS CDK using an L2 construct called `PythonFunction`.

This is not availble in the `aws_cdk.aws_lambda` module, so we will have to install it using pip:

```bash
pip install aws-cdk.aws-lambda-python-alpha
```

## Create a lambda function using PythonFunction

```python
# filename: cdk_app/lambda_stack.py
from aws_cdk import (
    Stack,
)
# 👇🏽 import the python_alpha module
from aws_cdk import aws_lambda_python_alpha as python

from constructs import Construct


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 create a python lambda function
        my_lambda = _lambda.PythonFunction(
            self,
            id="MyLambda",
            entry="cdk_app/lambda", # 👈🏽 Required
            runtime=_lambda.Runtime.PYTHON_3_10, # 👈🏽 Required
            index="index.py", # 👈🏽 Optional, defaults to index.py
            handler="handler", # 👈🏽 Optional, defaults to handler
        )
```

In the above code we use `PythonFunction` to define a lambda function. It takes the following parameters:

1. `scope`: The scope of the construct. In our case, it is the `LambdaStack` class.
2. `id`: The id of the construct. In our case, it is `MyLambda`, this will be used to refer to the function in CloudFormation templates
3. `entry`: The path to the directory where the lambda function is located. In our case, it is `lambda`
4. `index`: The name of the file that contains the lambda function. In our case, it is `index.py`
5. `handler`: The name of the handler function. In our case, it is `handler`

By default, the Construct will look for a `requirements.txt` file within the `entry` directory and install the dependencies listed in it. 

Let's create the `cdk_app/lambda/index.py` file:

```python
# filename: cdk_app/lambda/index.py

import requests

def handler(event, context):
    response = requests.get("https://jsonplaceholder.typicode.com/todos/1")
    return {
        "statusCode": 200,
        "body": response.json()
    }
```

We put a `requirements.txt` file in the same directory as our `index.py` file with the following contents:

```text
requests
```

Now let's create the `app.py` file:

```python
# filename: app.py

import aws_cdk as cdk
from cdk_app.lambda_stack import LambdaStack

app = cdk.App()

lambda_stack = LambdaStack(app, "LambdaStack")

app.synth()
```

Run `cdk deploy` to deploy the stack. 

Using this method, you just need to capture your dependencies in a `requirements.txt` file and the construct will take care of the rest.

---

# Using Lambda Layers with AWS CDK in Python

URL: https://cloudbytes.dev/aws-academy/using-lambda-layers-with-aws-cdk-in-python
Category: AWS Academy
Published: 2023-11-05
Author: Rehan Haider
Tags: aws, cdk, python

> Using Lambda layers with AWS CDK in Python to handle dependencies and share code between lambda functions

While we created some simple lambda functions in a [previous post]({filename}50002000-cdk-fn-create-lambda.md), but in most cases, you will need to use some external libraries or dependencies in your lambda functions.

E.g., let's modify our `cdk_app/lambda/index.file` to use the `requests` library:

```python
# filename: cdk_app/lambda/index.py

import requests

def handler(event, context):
    response = requests.get("https://jsonplaceholder.typicode.com/todos/1")
    return {
        "statusCode": 200,
        "body": response.json()
    }
```
We put a `requirements.txt` file in the same directory as our `index.py` file with the following contents:

```text
# filename: cdk_app/lambda/requirements.txt
requests
```

Then we can create a lambda function using this code as follows:

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)

from constructs import Construct


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_lambda = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="index.handler",
            code=_lambda.Code.from_asset("cdk_app/lambda"),
        )
```

After deploying the stack if you try to invoke the lambda function, you will get an error:

![lambda requirement import error](/images/50002020-01-fn-lambda-import-error.png)


This is because by default, the lambda function does not have the `requests` library installed. To fix this, we need to create a lambda layer that contains the `requests` library and then add that layer to our lambda function.

## When to use lambda layers

Lambda layers are useful in the following scenarios:

1. [**Handle dependencies**](#create-a-lambda-layer-in-aws-cdk-using-python-to-handle-dependencies): If you need to use some external libraries or dependencies in your lambda function, you can put those dependencies in a lambda layer and then add that layer to your lambda function.
2. **Reuse code between lambda functions**: If you have some common code that you want to use in multiple lambda functions, you can put that code in a lambda layer and then add that layer to all the lambda functions that need to use that code.



## Create a lambda layer in AWS CDK using Python to handle dependencies

Essentially, handling dependencies means we have to download the dependencies and put them in a folder. Then we need to zip that folder and upload it to AWS.

We will create a function within our stack that will do this work for us, and then import that layer into our lambda function.

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)

from constructs import Construct

import os, subprocess  # 👈🏽 needed to download dependencies


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_lambda = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="index.handler",
            code=_lambda.Code.from_asset("cdk_app/lambda"),
            layers=[self.create_dependencies_layer(self.stack_name, "lambda/index")],
        )

    def create_dependencies_layer(self, project_name, function_name: str) -> _lambda.LayerVersion:
        requirements_file = "cdk_app/lambda/requirements.txt"  # 👈🏽 point to requirements.txt
        output_dir = f".build/app"  # 👈🏽 a temporary directory to store the dependencies

        if not os.environ.get("SKIP_PIP"):
            # 👇🏽 download the dependencies and store them in the output_dir
            subprocess.check_call(f"pip install -r {requirements_file} -t {output_dir}/python".split())

        layer_id = f"{project_name}-{function_name}-dependencies"  # 👈🏽 a unique id for the layer
        layer_code = _lambda.Code.from_asset(output_dir)  # 👈🏽 import the dependencies / code

        my_layer = _lambda.LayerVersion(
            self,
            layer_id,
            code=layer_code,
        )

        return my_layer
```

Let's go through the code above:

```python
import os, subprocess  # 👈🏽 needed to download dependencies
```

We imported the `os` and `subprocess` modules. We will use these modules to download the dependencies and store them in a temporary directory.

```python
        my_lambda = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="index.handler",
            code=_lambda.Code.from_asset("cdk_app/lambda"),
            layers=[self.create_dependencies_layer(self.stack_name, "lambda/index")],
        )
```

Here, we defined the `layers` which is to be passed as a list. We are calling the `create_dependencies_layer` function to create the layer.

```python
    def create_dependencies_layer(self, project_name, function_name: str) -> _lambda.LayerVersion:
        requirements_file = "cdk_app/lambda/requirements.txt"  # 👈🏽 point to requirements.txt
        output_dir = f".build/app"  # 👈🏽 a temporary directory to store the dependencies

        if not os.environ.get("SKIP_PIP"):
            # 👇🏽 download the dependencies and store them in the output_dir
            subprocess.check_call(f"pip install -r {requirements_file} -t {output_dir}/python".split())

        layer_id = f"{project_name}-{function_name}-dependencies"  # 👈🏽 a unique id for the layer
        layer_code = _lambda.Code.from_asset(output_dir)  # 👈🏽 import the dependencies / code

        my_layer = _lambda.LayerVersion(
            self,
            layer_id,
            code=layer_code,
        )

        return my_layer
```

Here, we are creating the `create_dependencies_layer` function. This functions runs the following steps:

1. It defines the `requirements_file` variable which points to the `requirements.txt` file.
2. It defines the `output_dir` variable which points to a temporary directory where we will store the dependencies.
3. It uses `subprocess` to run commands to downloads the dependencies and stores them in the `output_dir` directory.
4. It defines the `layer_id` variable which is a unique id for the layer.
5. It defines the `layer_code` variable which points to the `output_dir` directory.
6. It creates the `my_layer` variable which is an instance of the `LayerVersion` construct.
7. It returns the `my_layer` variable.

Now if you run `cdk deploy`, you will see that the lambda function is created successfully and you can invoke it successfully as well.

![Succes lambda layer](/images/50002020-02-fn-layer-success-dependency.png)


## Reuse code between lambda functions

Another major use case is when you write some code that you want to reuse between multiple lambda functions. E.g. some utility function or helper function that you want to use in multiple lambda functions.

Similar to the previous example, we can create a layer by importing the code and then add that layer to our lambda functions.

Let's create a `helpers.py` file in the `cdk_app/utils` directory with the following contents:

```python
# filename: cdk_app/utils/helpers.py

def hello_world():
    return "Hello World"
```

Now we can create a layer using this code as follows:

```python
# filename: cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
)

from constructs import Construct


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_layer = _lambda.LayerVersion(
            self,
            "MyLayer",
            code=_lambda.Code.from_asset("cdk_app/utils"),
        )


        my_lambda = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="index.handler",
            code=_lambda.Code.from_asset("cdk_app/lambda"),
            layers=[my_layer],
        )
```


Now we can use the `hello_world` function in our lambda function as follows:

```python
# filename: cdk_app/lambda/index.py

from utils.helpers import hello_world  # 👈🏽 import the helper function

def handler(event, context):

    return {
        "statusCode": 200,
        "body": hello_world()  # 👈🏽 use the helper function
    }
```

This is a very simple example, but you can imagine that you can put a lot of code in the `helpers.py` file and then use that code in multiple lambda functions.


## Conclusion
While layers are a very useful feature of lambda functions, they are not a silver bullet. You should use them wisely and only when needed. If you have a lot of code in your lambda function, you should consider using a different approach such as using a container image instead of a lambda function.

---

# Configure event notifications using EventBridge for S3 buckets using CDK

URL: https://cloudbytes.dev/aws-academy/configure-event-notifications-using-eventbridge-for-s3-buckets-using-cdk
Category: AWS Academy
Published: 2023-11-04
Author: Rehan Haider
Tags: aws, cdk, python

> Guide to configure event notifications using EventBridge for S3 buckets using CDK

There are two ways to configure event notifications on the S3 bucket:

1. Using EventBridge notifications. This is covered in this guide
2. Using the `add_event_notification` method of the `Bucket` construct. This is covered in the [next guide]({filename}50001060-cdk-s3-event-notifications.md)

![S3 event notifications](/images/50001050-02-event-notification-options.png)


## Configure EventBridge notifications

To configure EventBridge notifications, we will need to do the following:

1. **Turn on sending of S3 events to EventBridge**: This is a one time activity, which can be done either from console or through CDK.
2. **Create an EventBridge rule**: We define the rule using a pattern that specifies the event types to listen for and the targets to send the events to
3. **Create an EventBridge target**: We attach a target to the rule that specifies the target resource to send the events to. In this case, we will add a simple Lambda function as a target.


### Import the required modules
We need to import the `aws_lambda` and `aws_events` modules to create the Lambda function and the EventBridge rule respectively.

```python
# filename: cdk_app/s3_stack.py#part-1

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
    aws_lambda as _lambda,
    aws_events as events,  # 👈🏽 Import the events module
    aws_events_targets as targets,  # 👈🏽 Import the events_targets module
)

from constructs import Construct
```

### Initialise the stack and create the S3 bucket

We will initialise the stack and create the S3 bucket as we did in the previous examples. We will also turn on sending of S3 events to EventBridge.

```python
class S3Stack(Stack):

    BUCKET_ID = "MyS3Bucket"
    EVENT_RULE_ID = "MyS3BucketEventRule"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 Create the S3 bucket
        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            removal_policy=RemovalPolicy.DESTROY,
            event_bridge_enabled=True,  # 👈🏽 Enable EventBridge notifications
        )
```

### Create the Lambda function

We will create a simple Lambda function that prints "File uploaded" when invoked.

```python
        # 👇🏽 Create the Lambda function
        _lambda.Function(self, 
            id="MyFirstLambda",
            runtime=_lambda.Runtime.PYTHON_3_7,
            code=_lambda.Code.from_inline("def main(event, context):\n\tprint('File Uploaded')"),
            handler="index.main",
        )
```

### Create the EventBridge rule

We will create an EventBridge rule that listens for events from the S3 bucket and sends them to the Lambda function.

```python
        # 👇🏽 Create an EventBridge rule
        event_rule = events.Rule(
            self,
            id=self.EVENT_RULE_ID,
            event_pattern=events.EventPattern(
                source=["aws.s3"], # 👈🏽 Listen for events from S3
                detail_type=["Object Created"], # 👈🏽 List of event types to listen for
                detail={
                    "buckets": {
                        "name": [my_bucket.bucket_name], # 👈🏽 List of buckets to listen to
                    },
                },
            ),
        )

        # 👇🏽 Add lambda function as a target for the EventBridge rule
        event_rule.add_target(targets.LambdaFunction(my_lambda_fn))
```

### Deploy the app

After the above changes, the full code will look like this:

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
    aws_lambda as _lambda,
    aws_events as events,  # 👈🏽 Import the events module
    aws_events_targets as targets,  # 👈🏽 Import the events_targets module
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"
    EVENT_RULE_ID = "MyS3BucketEventRule"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            removal_policy=RemovalPolicy.DESTROY,
        )

        # 👇🏽 Create the Lambda function
        my_lambda_fn = _lambda.Function(
            self,
            id="MyLambdaFn",
            runtime=_lambda.Runtime.PYTHON_3_7,
            code=_lambda.Code.from_inline("def main(event, context):\n\tprint('File uploaded!')"),
            handler="index.main",
        )

        # 👇🏽 Create an EventBridge rule
        event_rule = events.Rule(
            self,
            id=self.EVENT_RULE_ID,
            event_pattern=events.EventPattern(
                source=["aws.s3"],
                detail_type=["Object Created"],
                detail={
                    "bucket": {
                        "name": [my_bucket.bucket_name],
                    },
                },
            ),
        )

        # 👇🏽 Add lambda function as a target for the EventBridge rule
        event_rule.add_target(targets.LambdaFunction(my_lambda_fn))
```


Now you can deploy the app by running `cdk deploy`. This will create the S3 bucket, the Lambda function and the EventBridge rule.

![S3 event bridge notification](/images/50001050-01-event-bridge-s3-notification.png)


### Test the app
Let's upload a file to the S3 bucket and see if the Lambda function is invoked.

![S3 upload file](/images/50001050-03-upload-to-s3.gif)

If you check your CloudWatch logs, you will see that the Lambda function was invoked and it printed "File uploaded!".

![S3 upload event bridge notification](/images/50001050-04-event-bridge-success.png)

---

# Configure event notifications using S3 buckets notifications

URL: https://cloudbytes.dev/aws-academy/configure-event-notifications-using-s3-buckets-notifications
Category: AWS Academy
Published: 2023-11-04
Author: Rehan Haider
Tags: aws, cdk, python

> Guide to configure bucket notifications for S3 buckets using CDK

Apart from [using EventBridge to gather events from S3 buckets and send them to a target]({filename}50001050-cdk-s3-eventbridge-notifications.md), you can also configure notifications on the S3 bucket itself. This is done using the `add_event_notification` method of the `Bucket` construct.


## Configure S3 bucket notifications

Bucket notifications can be configured using the `add_event_notification` method of the `Bucket` construct. This method takes two parameters:


### Import the required modules

We need to import the `s3_notifications` module to create the S3 bucket notification.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    aws_s3_notifications as s3n,  # 👈🏽 Import the s3 notifications module
    RemovalPolicy,
    aws_lambda as _lambda,
)

from constructs import Construct
```

### Initialise the stack and create the S3 bucket

We will initialise the stack and create the S3 bucket as we did in the previous examples. 

```python
class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Block all public access
            removal_policy=RemovalPolicy.DESTROY,
        )
```

### Create the Lambda function

We will create a simple Lambda function that will be triggered when an object is created in the S3 bucket. The Lambda function will simply print `File Uploaded` message to console.

```python
        # 👇🏽 Create a lambda function
        my_lambda_fn = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_7,
            code=_lambda.Code.from_inline("def main(event, context):\n\tprint('File Uploaded')"),
            handler="index.main",
        )
```

### Create the S3 bucket notification

We use the `add_event_notifications` method of the `Bucket` construct to create the notification. This method takes three parameters:

1. `event`: The type of event that will trigger the notification. We will use `s3.EventType.OBJECT_CREATED` to trigger the notification when an object is created in the bucket.
2. `destination`: The destination of the notification. We will use the Lambda function we created earlier as the destination.
3. `filters`: The filters to apply to the notification. We can use this to specify the `prefix` and `suffix` of the object key that will trigger the notification. E.g. if all objects are being stored in a folder named `images` we can use `prefix="images/"` to trigger the notification.

```python
        # 👇🏽 Create the S3 bucket notification
        my_bucket.add_event_notification(
            s3.EventType.OBJECT_CREATED, # 👈🏽 Trigger the notification when an object is created
            s3n.LambdaDestination(my_lambda_fn), # 👈🏽 Use the Lambda function as the destination
        )
```

### Deploy the stack

After the above changes, the full code will look like this:


```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    aws_s3_notifications as s3n,  # 👈🏽 Import the s3 notifications module
    RemovalPolicy,
    aws_lambda as _lambda,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Block all public access
            removal_policy=RemovalPolicy.DESTROY,
        )

        # 👇🏽 Create a lambda function
        my_lambda_fn = _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_7,
            code=_lambda.Code.from_inline("def main(event, context):\n\tprint('File Uploaded')"),
            handler="index.main",
        )

        # 👇🏽 Configure bucket notifications
        my_bucket.add_event_notification(
            s3.EventType.OBJECT_CREATED,
            s3n.LambdaDestination(my_lambda_fn),
        )
```

You can use `cdk deploy` to deploy the app.

The app will look like the below.

```python
# filename: app.py

import aws_cdk as cdk
from cdk_app.s3_stack import S3Stack

app = cdk.App()

s3_stack = S3Stack(app, "S3Stack")

app.synth()
```

## Configuring filters in event notifications

Let's say we wanted to configure a filter wheere if a new object is created in `uploads/` folder in S3 bucket and is a `.png` file, it triggers the notification. We can do this by adding a `prefix` and `suffix` to the notification.

```python
        # 👇🏽 Create the S3 bucket notification
        my_bucket.add_event_notification(
            s3.EventType.OBJECT_CREATED, # 👈🏽 Trigger the notification when an object is created
            s3n.LambdaDestination(my_lambda_fn), # 👈🏽 Use the Lambda function as the destination
            # 👇🏽 Add a prefix and suffix to the notification
            prefix="uploads/",
            suffix=".png",
        )
```

## EventBridge vs S3 bucket notifications

While both are acceptable ways to configure notifications for S3 buckets, I personally prefer using EventBridge as it provides more flexibility.

Also, Bucket Notification are configured as a workaround and you would notice in the AWS console that the notification is configured using a Lambda to configure the S3. This is because S3 bucket notifications are not natively supported by CDK. CDK uses a Lambda function to configure the S3 bucket notification.

---

# Configure lifecycle rules for S3 buckets using CDK

URL: https://cloudbytes.dev/aws-academy/configure-lifecycle-rules-for-s3-buckets-using-cdk
Category: AWS Academy
Published: 2023-11-04
Author: Rehan Haider
Tags: aws, cdk, python

> How to configure S3 lifecycle rules like expiration and transition

Lifecycle rules allow you to configure the lifecycle of objects in your S3 bucket. You can configure lifecycle rules to expire or transition objects to different storage classes.


## Configure Lifecycle Rules

You can configure lifecycle rules on the S3 bucket by using the `add_lifecycle_rule` method of the `Bucket` construct.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
    Duration,  # 👈🏽 Import the Duration class
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Set the lifecycle rules
            lifecycle_rules=[
                s3.LifecycleRule(
                    enabled=True,
                    expiration=Duration.days(14),
                )
            ],
            removal_policy=RemovalPolicy.DESTROY,
        )
```

In the above code, we have set the following lifecycle rules:

1. `enabled=True` - Enable the lifecycle rule
2. `expiration=Duration.days(14)` - Delete the objects after 14 days

You also configure transition rules by setting the `transitions` property of the `LifecycleRule` class.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
    Duration,  # 👈🏽 Import the Duration class
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            self.BUCKET_ID,
            # 👇🏽 Set the lifecycle rules
            lifecycle_rules=[
                s3.LifecycleRule(
                    enabled=True,
                    transitions=[
                        s3.Transition(
                            storage_class=s3.StorageClass.INFREQUENT_ACCESS,
                            transition_after=Duration.days(30),
                        ),
                        s3.Transition(
                            storage_class=s3.StorageClass.GLACIER,
                            transition_after=Duration.days(60),
                        ),
                    ],
                )
            ],
            removal_policy=RemovalPolicy.DESTROY,
        )
```

In the above example, we have set the following transition rules:

1. `storage_class=s3.StorageClass.INFREQUENT_ACCESS` - Move the objects to infrequent access storage class after 30 days
2. `storage_class=s3.StorageClass.GLACIER` - Move the objects to Glacier storage class after 60 days

---

# Configure public access control for S3 buckets using CDK

URL: https://cloudbytes.dev/aws-academy/configure-public-access-control-for-s3-buckets-using-cdk
Category: AWS Academy
Published: 2023-11-04
Author: Rehan Haider
Tags: aws, cdk, python

> How to configure S3 public access control

AWS provider the ability to [control public access to S3 buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html). 

There are four properties that can be set to control public access to S3 buckets:

1. `BlockPublicAcls`: Specifies if Amazon S3 should restrict public access control lists (ACLs) for this bucket and its objects
2. `BlockPublicPolicy`: Specifies if Amazon S3 should restrict public bucket policies for this bucket
3. `IgnorePublicAcls`: Specifies if Amazon S3 should ignore public ACLs for this bucket and its objects
4. `RestrictPublicBuckets`: Specifies whether Amazon S3 should restrict public bucket policies for this bucket


You can either configure them individually, or all together using the `BlockPublicAccess.BLOCK_ALL` configuration.


### Configure Access Control

You can configure access control on the S3 bucket by setting the `block_public_access` property of the `Bucket` construct to `BlockPublicAccess.BLOCK_ALL`.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Block all public access
            block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
            removal_policy=RemovalPolicy.DESTROY,
        )
```

This block all public access to the bucket and its objects.

---

# Configure S3 encryption using CDK

URL: https://cloudbytes.dev/aws-academy/configure-s3-encryption-using-cdk
Category: AWS Academy
Published: 2023-11-04
Author: Rehan Haider
Tags: aws, cdk, python

> How to configure S3 encryption for objects using CDK including SSE-S3, SSE-KMS, and SSE-C

AWS S3 encryption is the ability to [encrypt objects stored in S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html). 

AWS S3 supports the follow encryption states:

1. `UNENCRYPTED`: - This is deprecated and no longer use for any new buckets. 
2. `S3_MANAGED`: [(SSE-S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-s3-encryption.html) - S3 will use S3 managed keys for server side encryption. This is the default encryption state for all new buckets and will be used if no encryption state is specified.
3. `KMS_MANAGED`: [(SSE-KMS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-kms-encryption.html) - Server Side encryption with S3 will use KMS managed keys that you have created using AWS KMS service
4. `KMS`:[(SSE-C)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html) - S3 will use customer managed keys using an external KMS service
5. `DSSE_MANAGED`: [(DSSE-KMS)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-dsse-encryption.html) - S3 uses Dual Layer Server-Side Encryption (DSSE) with AWS KMS managed keys
5. `DSSE`: [(DSSE-C)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html) - S3 uses Double Server-Side Encryption (SSE) with customer provided keys from an external KMS service


## Configure S3 encryption using CDK

You can enable encryption on the S3 bucket by setting the `encryption` property of the `Bucket` construct to `BucketEncryption.S3_MANAGED` which is the default way of encrypting data in S3.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy, 
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Bucket encryption will use S3 managed keys
            encryption=s3.BucketEncryption.S3_MANAGED,
            removal_policy=RemovalPolicy.DESTROY,
        )
```

If you want to use KMS managed keys, you can use the `encryption_key` property of the `Bucket` construct to specify the KMS key to use for encryption. 

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy, 
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 Create a KMS key to use for encryption
        kms_key = kms.Key(self, "MyKmsKey")

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Bucket encryption will use KMS managed keys
            encryption=s3.BucketEncryption.KMS_MANAGED,
            # 👇🏽 Specify the KMS key to use for encryption
            encryption_key=kms_key,
            removal_policy=RemovalPolicy.DESTROY,
        )
```

---

# Configure S3 versioning using CDK

URL: https://cloudbytes.dev/aws-academy/configure-s3-versioning-using-cdk
Category: AWS Academy
Published: 2023-11-04
Author: Rehan Haider
Tags: aws, cdk, python

> How to configure S3 versioning using CDK

AWS S3 versioning is the ability to [keep multiple versions of an object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html) in one bucket. 

By default, versioning is disabled, however, if you enable it you cannot disable it. You can only suspend it.


## Configure S3 versioning using CDK

You can enable versioning on the S3 bucket by setting the `versioned` property of the `Bucket` construct to `True`.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy, 
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            versioned=True,  # 👈🏽 Enable versioning
            removal_policy=RemovalPolicy.DESTROY,
        )
```

Configure the CDK app to use the `S3Stack` stack.

```python
# filename: app.py

import aws_cdk as cdk
from cdk_app.s3_stack import S3Stack

app = cdk.App()

s3_stack = S3Stack(app, "S3Stack")

app.synth()
```

To deploy the stack run `cdk deploy`.

If you go to the AWS console and check the S3 bucket, you will see that versioning is enabled.

![S3 versioning enabled](/images/50001010-01-s3-versioning-enabled.png)

## Suspending versioning

You can suspend versioning by setting the `versioned` property of the `Bucket` construct to `False`.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            versioned=False,  # 👈🏽 Suspend versioning
            removal_policy=RemovalPolicy.DESTROY,
        )
```

![S3 versioning suspended](/images/50001010-02-s3-versioning-suspended.png)


## Conclusion

Currently CDK doesn't support configuring MFADelete for S3 buckets. There is a [feature request](https://github.com/aws/aws-cdk/issues/5247) for it, however, most methods are workarounds.

---

# Create S3 bucket using CDK

URL: https://cloudbytes.dev/aws-academy/create-s3-bucket-using-cdk
Category: AWS Academy
Published: 2023-11-04
Author: Rehan Haider
Tags: aws, cdk, python

> How to create an S3 bucket in CDK using Python

This guide will walk you through the process of creating and configuring an S3 bucket using the AWS CDK in Python. 

After creating the S3 bucket, we will also learn how to configure the following:

1. Retention Policies
2. Versioning
3. Encryption
4. Access Control
5. Lifecycle Rules
6. Event Notifications

## Create an S3 bucket

CDK has both [L1 and L2 constructs]({filename}50000040-cdk-constructs.md) for S3 buckets that can be used to create an S3 bucket.

> L1 constructs are low-level constructs that map directly to the underlying CloudFormation resources. L2 constructs are high-level constructs that provide a simpler API to work with.

### Create an S3 bucket using L1 construct (CfnBucket)

Let's start by creating an S3 bucket using the L1 construct. 

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.CfnBucket(
            self,
            id=self.BUCKET_ID,
        )
```

The above code will create an S3 bucket with an automatically generated unique name.

You can create the CDK app by modifying the `app.py` file as follows:

```python
# filename: app.py

import aws_cdk as cdk
from cdk_app.s3_stack import S3Stack

app = cdk.App()

s3_stack = S3Stack(app, "S3Stack")

app.synth()
```

Deploy by running `cdk deploy`.

!!! warning
    If you destroy the stack by running `cdk destroy`, the S3 bucket will not be deleted. This is because the default removal policy for S3 buckets is `Retain`. You will manually have to delete the S3 bucket from the AWS console.


### Create an S3 bucket using L2 construct (Bucket)

While using L1 `CfnBucket` construct is a valid way to create an S3 bucket, it is not the recommended way as `Bucket` provies a more abstraction.

Let's create an S3 bucket using the L2 `Bucket` construct.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # 👇🏽 Use Bucket instead of CfnBucket
        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
        )
```

The above code will create an S3 bucket with a unique name. You can deploy the app by running `cdk deploy`.

As in previous example, if you destroy the stack by running `cdk destroy`, the S3 bucket will not be deleted. This is because the default removal policy for S3 buckets is `Retain`. You will manually have to delete the S3 bucket from the AWS console.

### Configure Removal Policies

What if you want the bucket to be automatically deleted when you destroy the stack? You can do this by setting the `removal_policy` property of the `Bucket` construct to `DESTROY`.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,  # 👈🏽 Import the RemovalPolicy class
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            # 👇🏽 Set the removal policy to destroy
            removal_policy=RemovalPolicy.DESTROY,
        )
```

In the above code, we have set the removal policy to `DESTROY`. This means that when you destroy the stack, the S3 bucket will also be deleted.


### Configure Bucket Name

By default, the bucket name is automatically generated. You can specify a custom bucket name by setting the `bucket_name` property of the `Bucket` construct.

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_bucket = s3.Bucket(
            self,
            id=self.BUCKET_ID,
            removal_policy=RemovalPolicy.DESTROY,
            # 👇🏽 Set the bucket name
            bucket_name="my-custom-bucket-name",
        )
```

The `bucket_name` value must be globally unique. If you try to deploy the app with a bucket name that already exists, you will get an error.


## Additional Configuration for S3 buckets

You also also read how to configure the following for S3 buckets:

1. [Import an existing S3 bucket]({filename}50001005-cdk-s3-import-existing-bucket.md)
2. [Versioning]({filename}50001010-cdk-s3-configure-versioning.md)
3. [Encryption]({filename}50001020-cdk-s3-configure-encryption.md)
4. [Access Control]({filename}50001030-cdk-s3-access-control.md)
5. [Lifecycle Rules]({filename}50001040-cdk-s3-lifecycle-rules.md)
6. [Event Notifications using EventBridge]({filename}50001050-cdk-s3-eventbridge-notifications.md)
7. [Event Notificationsusing Bucket Notifications]({filename}50001060-cdk-s3-event-notifications.md)

---

# How to get the ARN of a resource using AWS CDK

URL: https://cloudbytes.dev/aws-academy/how-to-get-the-arn-of-a-resource-using-aws-cdk
Category: AWS Academy
Published: 2023-10-29
Author: Rehan Haider
Tags: aws, cdk, python

> A guide on how to get the ARN of a resource using AWS CDK

ARN (Amazon Resource Name) is a unique identifier automatically assigned to every AWS resource when it is created. The ARN is used to uniquely identify the resource across all of AWS including accounts, regions, and services.

# ARN Format

As explained in the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html), an ARN has the following format depending on the resource type:

```
arn:partition:service:region:account-id:resource-id
arn:partition:service:region:account-id:resource-type/resource-id
arn:partition:service:region:account-id:resource-type:resource-id
```

In the above format, the following placeholders are used:

1. `partition` - The partition that the resource is in. For standard AWS regions, the partition is `aws`. If you have resources in other partitions, the partition is `aws-cn` for China and `aws-us-gov` for AWS GovCloud (US).
2. `service` - The service namespace that identifies the AWS product (for example, `s3`, `iam`, `codecommit`, `ec2`, etc.).
3. `region` - The AWS Region that the resource resides in. For example, `us-east-1`.
4. `account-id`: The ID of the AWS account that owns the resource, without the hyphens. For example, `123456789012`.
5. `resource-type` - The resource type (for example, `instance`, `bucket`, `user`, etc.).
6. `resource-id`: The resource ID. This depends on the service namespace. For example, an Amazon S3 bucket is named using the path style `bucket_name`, and so is identified by `bucket_name`.


## How to get the ARN of a resource using AWS CDK

There are a few ways to get the ARN of a resource using AWS CDK. Some of them are:

1. [Using `<resource>_arn` property](#using-resource_arn-property)
2. [Using `attr_arn` method from CFN resource](#using-attr_arn-method-fromm-cfn-resource)
3. Using GetAtt method from Fn class

### Using `<resource>_arn` property

To get the ARN of a resource using AWS CDK, you can use the `<resource>_arn` property. For example, to get the ARN of an S3 bucket, you can use the `bucket_arn` property as shown below:

```python
# filename: cdk_app/s3_stack.py
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from aws_cdk import CfnOutput  # 👈🏽 Import the CfnOutput class
from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        myBucket = s3.Bucket(self, id=self.BUCKET_ID, removal_policy=RemovalPolicy.DESTROY)
        bucket_arn = myBucket.bucket_arn  # 👈🏽 Get the ARN of the bucket

        # 👇🏽 Print the bucket ARN to console
        print(f"Bucker ARN: {bucket_arn}")

        # 👇🏽 Output the bucket ARN to use in other stacks
        CfnOutput(self, "S3BucketARN", value=myBucket.bucket_arn, export_name="MyS3BucketARN")
```

Similarly, to get the ARN of a DynamoDB table, you can use the `table_arn` property as shown below:

```python
# filename: cdk_app/dynamodb_stack.py
from aws_cdk import (
    Stack,
    aws_dynamodb as ddb,
    RemovalPolicy,
)

from constructs import Construct


class DynamoDBStack(Stack):
    TABLE_ID = "MyDynamoDBTable"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        myTable = ddb.Table(
            self,
            id=self.TABLE_ID,
            partition_key={"name": "id", "type": ddb.AttributeType.STRING},
            removal_policy=RemovalPolicy.DESTROY,
        )

        # 👇🏽 Print the table ARN to console
        print(f"Table ARN: {myTable.table_arn}")
```

### Using `attr_arn` method fromm CFN resource

You can also use the `attr_arn` method from the [L1 CFN]({filename}50000040-cdk-constructs.md) resource to get the ARN of a resource. For example, let's modify our `s3_stack.py` to use the `attr_arn` method as shown below:


```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from aws_cdk import CfnOutput  # 👈🏽 Import the CfnOutput class
from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        myBucket = s3.Bucket(self, id=self.BUCKET_ID, removal_policy=RemovalPolicy.DESTROY)

        # 👇🏽 Get the CFN Bucket resource. 
        cfn_bucket: s3.CfnBucket = myBucket.node.default_child

        bucket_arn = cfn_bucket.attr_arn # 👈🏽 Get the ARN of the bucket
        
        # 👇🏽 Output the bucket ARN
        CfnOutput(self, id="S3BucketARN", value=bucket_arn, export_name="MyS3BucketARN")
```

In the above code, we are using the `node.default_child` property to get the CFN resource for the S3 bucket. Then we are using the `attr_arn` method to get the ARN of the bucket.

!!! note 
    Notice we hinted the type of the `cfn_bucket` variable as `s3.CfnBucket`. This is because the `node.default_child` property returns a generic `CfnResource` type. We need to hint the type to `s3.CfnBucket` to get access to the `attr_arn` method.

## Using GetAtt method from Fn class

You can also use the `Fn.get_att` method to get the ARN of a resource. For example, let's modify our `s3_stack.py` to use the `Fn.get_att` method as shown below:

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
    Fn,  # 👈🏽 Import the Fn class
)

from aws_cdk import CfnOutput  # 👈🏽 Import the CfnOutput class
from constructs import Construct


class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        myBucket = s3.Bucket(self, id=self.BUCKET_ID, removal_policy=RemovalPolicy.DESTROY)

        # 👇🏽 Get the CFN Bucket resource
        cfn_bucket: s3.CfnBucket = myBucket.node.default_child

        bucket_arn = Fn.get_att(cfn_bucket.logical_id, "Arn").to_string()

        # 👇🏽 Output the bucket ARN
        CfnOutput(self, id="S3BucketARN", value=bucket_arn, export_name="MyS3BucketARN")
```

To use the `Fn.get_att` method, you need to pass the logical ID of the resource and the attribute name as arguments. We also need to convert the output of the `Fn.get_att` method to a string using the `to_string` method.


## Conclusion

The above methods are common ways of getting the ARN of a resource using AWS CDK. Using the `<resource>_arn` property is the easiest way to get the ARN of a resource. However, if you need to get the ARN of a resource that doesn't have a L2 construct yet and is not supported by AWS CDK, you can use the `attr_arn` method from the CFN resource or the `Fn.get_att` method from the Fn class.

```python
        myBucket = s3.Bucket(self, id=self.BUCKET_ID, removal_policy=RemovalPolicy.DESTROY)
        bucket_arn = myBucket.bucket_arn  # 👈🏽 Get the ARN of the bucket
```

---

# Get your AWS Account ID using AWS CLI

URL: https://cloudbytes.dev/aws-academy/get-your-aws-account-id-using-aws-cli
Category: AWS Academy
Published: 2023-10-28
Author: Rehan Haider
Tags: aws, linux

> A guide to how to get your AWS Account ID using AWS CLI

If you have not already done so, [install and configure AWS CLI]({filename}12500000-aws-cli-intro.md). 

## Get your AWS Account ID

To get your AWS Account ID, run the following command:

```bash
aws sts get-caller-identity --query Account --output text
```

This command uses the Security Token Service (STS) get-caller-identity function, which returns details about the IAM user or role making the call. The --query Account fetches only the Account ID, and --output text ensures the result is displayed as plain text.

![STS caller identify](/images/12501000-01-cli-id-output-text.png)


### Understanding the output

The output of the `get-caller-identity` command provides three pieces of information:

* `UserId`: The unique identifier for the entity making the call. For an IAM user, this is the user's unique ID.
* `Account`: Your AWS Account ID.
* `Arn`: The Amazon Resource Name (ARN) of the IAM user or role making the call.

By using the `--query Account` parameter, we specifically extract the Account value.


### Saving the output to file

You can save the output of the `get-caller-identity` command to a file using the `>` operator as shown below:

```bash
aws sts get-caller-identity --query Account --output text > account-id.txt
```

This will save the output to a file called `account-id.txt` in the current directory.

![STS caller identify](/images/12501000-02-cli-id-to-file.png)

---

# How to install and configure AWS CLI on Linux with Autocompletion

URL: https://cloudbytes.dev/aws-academy/how-to-install-and-configure-aws-cli-on-linux-with-autocompletion
Category: AWS Academy
Published: 2023-10-28
Author: Rehan Haider
Tags: aws, linux

> A guide to how to install and configure AWS CLI on Ubuntu Linux with Autocompletion turned on

[TOC]

AWS provides several tools to help you manage and automate your AWS environment. Some of the key ones are

1. **AWS CLI** - The command line interface for AWS
2. **AWS SAM CLI** - The command line interface for AWS Serverless Application Model
3. **AWS CDK** - The AWS Cloud Development Kit
4. **AWS Chalice** - The AWS Serverless Development Framework

## What is AWS CLI?

[AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) (Command Line Interface) is an [open source tool hosted on GitHub](https://github.com/aws/aws-cli) that allows you to interact with AWS services from the command line shell on Linux, Mac OS, or Windows. 

You can use AWS CLI on either [bash](https://www.gnu.org/software/bash/), [zsh](http://www.zsh.org/), or [tcsh](https://www.tcsh.org/) shells on Linux/MacOS and PowerShell on Windows. Additionally, AWS CLI is installed by default on all AWS Linux EC2 instances.

AWS CLI can manage all IaaS (Infrastructure as a Service) services that are available in AWS Management Console.

## Installing AWS CLI on Ubuntu Linux

**Step 1)** First, update your Ubuntu Linux system and install `unzip` and `curl` packages.

```bash
sudo apt update && sudo apt upgrade -y && sudo apt install unzip curl -y
```

**Step 2)** Then download & unzip AWS CLI.

```bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
   && unzip awscliv2.zip
```

**Step 3)** Finally, install AWS CLI.

```bash
sudo ./aws/install
```

Verify that you can use the AWS CLI by running the following command:

```bash
aws --version
```

## Configuring AWS CLI for use

AWS CLI relies on **"Programmatic Access"** credentials to access AWS services.

Ideally, you should create an IAM User Account with only programmatic access to use with AWS CLI as shown below.

### Create a new IAM User Account with Admin Access

1. Login to [AWS Management Console](https://console.aws.amazon.com/) and navigate to [IAM](https://console.aws.amazon.com/iam/home)
2. On the left navigation pane, click on **Users**
3. CLick on **Add users**
4. Choose a username & select only **Programmatic access** under **Select AWS access type**, then click **Next: Permissions**

![Add AWS User](/images/12500000-aws-iam-type.png)

5. Then click on **Attach existing policies** and select **AdministratorAccess** from the list of available policies. Then click **Next: Tags**
6. You can leave the **Tags** empty and click **Next: Review**
7. Click **Create user** to create your new IAM User Account

Keep this window open for now, and notice the **Access key ID** & **Secret access key**. This will be needed in next step.

> !!! danger "WARNING: Never store this credentials anywhere or share them with anyone. An attacker can user your credentials to create AWS resources in your account. If you need to reconfigure, you can generate a new credentials from IAM screen."

![Add AWS User](/images/12500000-aws-new-iam-user.png)



### Configure AWS CLI to use the new IAM User Account
Open a terminal and run the following command:

```bash
aws configure
```

This will start an interactive session, copy paste your access keys and secret access keys that was generated in previous step when prompted

```bash
aws configure
AWS Access Key ID [None]: <your access key id>
AWS Secret Access Key [None]: <your secret access key>
Default region name [None]: us-east-1
Default output format [None]: json
```

## Configuring AWS CLI for Autocompletion

AWS CLI relies on a module named `aws_completer` for autocompletion. This module should be installed while installing AWS CLI however, for it to work correctly it requires

1. `aws_completer` to be on the `PATH`
2. Enable command completion in the shell

### Ensure `aws_completer` is added to the PATH

First, check if the `aws_completer` is already on path by running the following command:

```bash
which aws_completer
```

This should result in the following output:
![Which AWS Completer](/images/12500000-which-aws-completer.png)


If you get the above output, it means that the `aws_completer` is already on the `PATH`. So you can skip to [Enable command completion in the shell](#enable-command-completion-in-the-shell) section.

Otherwise if you don't see any output, it means that the `aws_completer` is not on the `PATH`, follow the steps below to add it to the `PATH`.

### Add `aws_completer` to the PATH

**Step 1**: Find `aws_completer` executable file by running the following command:

```bash
find / -name aws_completer
```

This will search for the `aws_completer` executable file in your filesystem. E.g. if you scroll through the results, you should see something similar to the following output:

![Find AWS Completer](/images/12500000-aws-completer-path.png)

**Step 2**: Identify your shell and add the `aws_completer` to the `PATH`. 

Run `echo $SHELL` to see what shell you are using. 

![Echo Shell](/images/12500000-shell-type.png)

If you are using some other shell, you will get a different output.

**Step 3**: Find the shell configuration file for your shell.

Depending on the shell you're using, your shell's profile file will be one of the following:

- **Bash**: `.bash_profile`, `.bash_login`, or `.profile`
- **Zsh**: `.zshrc`
- **Tcsh**: `.tcshrc`, `.cshrc`, or `.login`

Find your shell's profile file by running the following command and look for profile file as per above

```bash
ls -a ~/.bash_profile ~/.bash_login ~/.profile ~/.zshrc ~/.tcshrc ~/.cshrc ~/.login
```

You will get a bunch of "*No such file or directory*" errors except for the shell profile file.

E.g. in my case, I am using `bash` and my profile file is `.profile` thus running the above command will result in the following output:

![Find Shell Profile](/images/12500000-bash-profile.png)

**Step 4**: Add the `aws_completer` to the `PATH`

Now open the shell profile using any text editor, e.g. `vi` or `nano` and add the following line to the end of the file and replace `<path to aws_completer directory>` with the path to the `aws_completer` executable file discovered in step 2:


```text
export PATH=<path to aws_completer directory>:$PATH
```

**Step 5**: Restart your shell

Depending upon the shell, restart your shell by running the following command by replacing `<your profile file>` with the name of your shell profile file as per step 4:


```text
source ~/<your profile file>
```

E.g. in my case, this would be `source ~/.profile`

### Enable command completion in the shell

After you have added the `aws_completer` to the `PATH`, you need to perform a few steps to enable command completion depending on your shell.

- **bash**: Open the `.bashrc` file in your home directory and add the following line to the end of the file:

```text
complete -C '<path to aws_completer directory>/aws_completer' aws
```

E.g. in my case, the above would be 

```text
complete -C '/usr/local/bin/aws_completer' aws
```

- **zsh**: Open the `.zshrc` file in your home directory and add the following line to the end of the file:

```text
autoload bashcompinit && bashcompinit
autoload -Uz compinit && compinit
complete -C '<path to aws_completer directory>/aws_completer' aws
```

- **tcsh**: Open the `.tcshrc` file in your home directory and add the following line to the end of the file:

```text
complete aws 'p/*/`aws_completer`/'
```

### Verify that the command completion is working

Reload your shell configuration file, replace `<your profile file>` with the appropriate shell configure file

```text
source ~/<your shell config file>
```

Then type `aws s3` and press `TAB` to see the list of available commands.

You AWS CLI is configure and autocomplete is working.

---

# Managing dependency between stacks in CDK

URL: https://cloudbytes.dev/aws-academy/managing-dependency-between-stacks-in-cdk
Category: AWS Academy
Published: 2023-10-28
Author: Rehan Haider
Tags: aws, cdk, python

> How to specify dependency between stacks in CDK

Managing infrastructure means dealing with interconnected resources. In the AWS Cloud Development Kit (CDK), this often translates to specifying dependencies between different stacks. This post will guide you on how to define and manage these dependencies effectively within the CDK.


## Why Specify Dependencies?

The main use of specifying dependencies is to ensure that resources are deployed in the correct order. There are three main reasons why you would want to specify dependencies. For example, a Lambda function depends on an S3 bucket. The Lambda function needs to be deployed after the S3 bucket is created. This is because the Lambda function needs to access the S3 bucket. If the Lambda function is deployed before the S3 bucket, it will fail to access the S3 bucket.

Another reason is to ensure that resources are deleted in the correct order. For example, if you delete the S3 bucket before the Lambda function, the Lambda function will fail to access the S3 bucket. This is because the S3 bucket no longer exists.

## How to Specify Dependencies?

There are two ways to specify dependencies in CDK. The first way is to use the `add_dependency` method. The second way is to use the `depends_on` property.

### 1. Using the `add_dependency` Method

Taking the code example in previous post that talks about [how to import stack outputs]({filename}50000080-cdk-how-to-import-output.md), let's create a stack that creates an S3 bucket and exports its ARN as an Output.

You can use the `add_dependency` method to specify this dependency.

```python
import aws_cdk as cdk

from cdk_app.s3_stack import S3Stack
from cdk_app.lambda_stack import LambdaStack

app = cdk.App()

# LambdaStack depends on S3Stack

s3_stack = S3Stack(app, "S3Stack")
lambda_stack = LambdaStack(app, "LambdaStack")

# 👇🏽 Add the dependency to ensure S3Stack is deployed first
lambda_stack.add_dependency(s3_stack)

app.synth()
```

### 2. Using the `depends_on` Property

This is typically used if there a stack depends on more than one stack. For example, if you have a stack that contains a Lambda function that accesses an S3 bucket and a DynamoDB table, you can use the `depends_on` property to specify the dependencies.

You will need to specify the dependencies as a list of stacks.

```python
import aws_cdk as cdk

from cdk_app.s3_stack import S3Stack
from cdk_app.lambda_stack import LambdaStack
from cdk_app.dynamodb_stack import DynamoDBStack

app = cdk.App()

# LambdaStack depends on S3Stack

s3_stack = S3Stack(app, "S3Stack")
dynanodb_stack = DynamoDBStack(app, "DynamoDBStack")

# 👇🏽 Specify that the stack depends on both s3_stack and dynamodb_stack
lambda_stack = LambdaStack(app, "LambdaStack", depends_on=[s3_stack, dynamodb_stack])

app.synth()
```

You can use the below for the dynamodb_stack.py file:

```python
# filename: cdk_app/dynamodb_stack.py

from aws_cdk import (
    Stack,
    aws_dynamodb as ddb,
    RemovalPolicy,
)
from constructs import Construct

class DynamoDBStack(Stack):
    TABLE_ID = "MyDynamoDBTable"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        myTable = ddb.Table(
            self,
            id=self.TABLE_ID,
            partition_key={"name": "id", "type": ddb.AttributeType.STRING},
            removal_policy=RemovalPolicy.DESTROY,
        )
```

## Limitations of CDK Dependencies

There are a few limitations of CDK dependencies that you should be aware of.

1. **Circular Dependencies**: Be cautious to avoid circular dependencies where Stack A depends on Stack B and vice versa. This will result in an error.
2. **Cross-Region Dependencies**: CDK doesn't support specifying dependencies across stacks in different AWS regions. For such scenarios, consider other synchronization methods or manually coordinating deployments.

---

# CDK Output: How to Output data from a stack

URL: https://cloudbytes.dev/aws-academy/cdk-output-how-to-output-data-from-a-stack
Category: AWS Academy
Published: 2023-10-27
Author: Rehan Haider
Tags: aws, cdk, python

> Explanation of concept of Outputs, how to use them to share data to other stacks

[TOC]

Previously, we learnt how to [create multiple stacks]({filename}50000050-cdk-multiple-stacks.md). For most applications that you would build using CDK, you would need to share data between the stacks. For example, you might want to create an S3 bucket in one stack and then a Lambda function in another stack that uses that S3 bucket. 

In this case, you would need to share the name of the S3 bucket between the stacks. This is where Outputs come in.

## What are Outputs?

In AWS CloudFormation (which the CDK leverages under the hood), Outputs are a way to export specific values from a stack. These values can be anything: an S3 bucket name, a database connection string, or even a computed value.

Outputs are especially useful when:

1. Linking Multiple Stacks: They allow one stack to use a resource from another stack.
2. External Usage: When you want to use a specific value from your cloud infrastructure in an external system or application.

## How to create an Output in CDK?

To create an Output in CDK, you need to use the `CfnOutput` class. This class is available in the `aws_cdk` module.

Create a new file called `cdk_app/s3_stack.py` and add the following code to it:

```python
# cdk_app/s3_stack.py
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from aws_cdk import CfnOutput  # 👈🏽 Import the CfnOutput class

from constructs import Construct

class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        myBucket = s3.Bucket(self, id=self.BUCKET_ID, removal_policy=RemovalPolicy.DESTROY)

        # 👇🏽 Output the bucket ARN
        CfnOutput(self, id="S3BucketARN", value=myBucket.bucket_arn, export_name="MyS3BucketARN")
```

In the above example, we are creating an S3 bucket and then exporting its ARN as an Output. 

CfnOutput takes the following parameters:

1. `scope`: The scope of the Output. In this case, we are using the current stack as the scope.
2. `id`: The ID of the Output. This is used to uniquely identify the Output within the stack.
3. `value`: The value of the Output. This can be a string, a number, or even a complex object.
4. `export_name`: The name of the Output. This is used to uniquely identify the Output across stacks.

### Viewing Outputs

Let's modify our `app.py` file to view the outputs of our stack. 

```python
# app.py

import aws_cdk as cdk

from cdk_app.s3_stack import S3Stack

app = cdk.App()

S3Stack(app, "S3Stack")

app.synth()
```

Now, run `cdk deploy` to deploy the stack. Once the stack is deployed, you will see the following output:

![CDK deploy CfnOutput](/images/50000070-01-cdk-deploy-output.png)

You can also view the outputs of a stack using the AWS Console. Go to the CloudFormation service and select your stack. Then, click on the Outputs tab. You will see the following:

![CDK CloudFormation Outputs](/images/50000070-02-cdk-console-output.png)

### What happens during to Output during `cdk synth`?

Let's try printing the Output by modifying our `cdk_app/s3_stack.py` file:

```python
# cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from aws_cdk import CfnOutput  # 👈🏽 Import the CfnOutput class

from constructs import Construct

class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        myBucket = s3.Bucket(self, id=self.BUCKET_ID, removal_policy=RemovalPolicy.DESTROY)

        # 👇🏽 Print the bucket ARN
        print(myBucket.bucket_arn)

        # 👇🏽 Output the bucket ARN
        CfnOutput(self, id="S3BucketARN", value=myBucket.bucket_arn, export_name="MyS3BucketARN")
```

Now run `cdk synth`, we get the following:

![CDK synth CfnOutput](/images/50000070-03-cdk-synth-output.png)

So what is this `Token` that is being printed? Token is a placeholder value that is replaced with the actual value by CloudFormation during deployment.

So, that means that the value of the Output is not known before deployments and cannot be accessed in our code. We can use a reference but CDK will not be able to resolve it during synth hence if you put in conditional logic based on the value of the Output, it will not work.

### Print Output values to a file in CDK

Sometimes, you might want to print the values of the Outputs to a file. For example, you might want to print the values of the Outputs to a file and then use that file in your CI/CD pipeline. 

To do add modify your deployment command as shown below:

```bash
cdk deploy <stack-name> --outputs-file ./output.json
```

This will print the values of the Outputs to a file called `output.json` in the current directory.

![CDK Output to file](/images/50000070-04-cdk-output-file.png)

---

# How to Import Stack Output from another stack

URL: https://cloudbytes.dev/aws-academy/how-to-import-stack-output-from-another-stack
Category: AWS Academy
Published: 2023-10-27
Author: Rehan Haider
Tags: aws, cdk, python

> A guide to importing Stack Outputs and using them as Cross Stack references

In the previous article, I explained how to export data from a stack using Outputs. Outputs are a way to export specific values from a stack. These values can be anything: an S3 bucket name, a database connection string, or even a computed value. 

Output can be imported by another stack as a reference helping you access resources created in another stack.

## How to import an Output in CDK?

While printing the outputs is useful, the real power of Outputs is when you use them in other stacks. 

We need to use the `Fn.import_value` function to import the value of an Output. This function is available in the `aws_cdk` module.


### Export the Output
Let's first create the S3 bucket stack from the previous article. Create a new file called `cdk_app/s3_stack.py` and add the following code to it:

```python
# filename: cdk_app/s3_stack.py

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from aws_cdk import CfnOutput  # 👈🏽 Import the CfnOutput class
from constructs import Construct

class S3Stack(Stack):
    BUCKET_ID = "MyS3Bucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        myBucket = s3.Bucket(self, id=self.BUCKET_ID, removal_policy=RemovalPolicy.DESTROY)

        # 👇🏽 Output the bucket ARN
        CfnOutput(self, id="S3BucketARN", value=myBucket.bucket_arn, export_name="MyS3BucketARN")
```

In the above example, we are creating an S3 bucket and then exporting its ARN as an Output.


### Import the Output

Now, let's create a new stack that will import the S3 bucket ARN. Create a new file called `cdk_app/lambda_stack.py` and add the following code to it:

```python
# cdk_app/lambda_stack.py

from aws_cdk import (
    Stack,
    aws_lambda as _lambda,
    aws_s3 as s3,
)

from aws_cdk import Fn  # 👈🏽 Import the Fn class this contains the import_value method

from constructs import Construct


class LambdaStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        bucket_arn = Fn.import_value("MyS3BucketARN")

        myBucket = s3.Bucket.from_bucket_arn(self, id="MyImportedBucket", bucket_arn)

        _lambda.Function(
            self,
            id="MyLambdaFn",
            # 👇🏽 Pass the bucket name as an environment variable
            environment={"BUCKET_NAME": myBucket.bucket_name}, 
            runtime=_lambda.Runtime.PYTHON_3_10,
            code=_lambda.Code.from_asset("./"), # 👈🏽 Use the current directory as the source
            handler="index.main", # 👈🏽 Filename is index.py and the function is called main
        )
```

In the above example, we imported the S3 bucket ARN using the `Fn.import_value` method. We then used the `from_bucket_arn` method to create a reference to the S3 bucket.

We then created a Lambda function and passed the bucket name as an environment variable.

Now we create the Lambda function in the `cdk_app/lambda/index.py` file:

```python
# cdk_app/lambda/index.py

import os

bucket_name = os.environ["BUCKET_NAME"]

def main(event, context):
    print(f"Bucket Name: {bucket_name}")

    return {
        "statusCode": 200,
        "body": bucket_name,
    }
```


### Deploy the stacks

Now, let's modify our `app.py` file to deploy both stacks:

```python
# app.py

import aws_cdk as cdk

from cdk_app.s3_stack import S3Stack
from cdk_app.lambda_stack import LambdaStack

app = cdk.App()

# LambdaStack depends on S3Stack

s3_stack = S3Stack(app, "S3Stack")
lambda_stack = LambdaStack(app, "LambdaStack")

# 👇🏽 Add the dependency to ensure S3Stack is deployed first
lambda_stack.add_dependency(s3_stack)

app.synth()
```

Note that we have added a dependency between the two stacks. This is because the Lambda function depends on the S3 bucket so we need to ensure that the S3 bucket is deployed first.

Now, run `cdk deploy --all` to deploy the stacks.


### Testing the stacks
Go to the AWS Console and run the Lambda function. You will see the following output:

![Lambda function output](/images/50000080-01-lambda-read-output.png)

---

# Basic AWS CDK Commands - list, diff, deploy, destroy

URL: https://cloudbytes.dev/aws-academy/basic-aws-cdk-commands-list-diff-deploy-destroy
Category: AWS Academy
Published: 2023-10-25
Author: Rehan Haider
Tags: aws, cdk, python

> An introduction to some of the basic AWS CDK commands such as list, diff, deploy, and destroy

CDK has a set of basic commands that facilitate the management, deployment, and interactions of your cloud applications. This post will introduce you to some basic CDK commands that are essential for any developer starting their journey with the AWS CDK.

## Initialize a CDK project

To initialize a CDK project, run the following command:

```bash
cdk init app --language python
```

This needs to be run in an empty directory. This command will create a new directory called `cdk_app` with the following:

1. **CDK App**: Create a new CDK app in the current directory. It includes a `cdk.json` file that includes the configuration for the CDK app
2. **Stacks**: Create a folder called `cdk-app` in the current directory that includes the `stacks` that will be deployed
3. **Python Virtual Environment**: Create a Python virtual environment for the app in the .venv folder. It also creates a `requirements.txt` file that includes the Python dependencies for the app
4. **Git**: Create a git repository for the app including a `.gitignore` file

You can read more about this in the [Creating a new CDK app with Python]({filename}50000020-cdk-new-app.md) post.

## List the stacks

In most cases, you will have multiple stacks in your CDK app. To list all the stacks in your CDK app, run the following command:

```bash
cdk ls
```

You can also use `cdk list` instead of `cdk ls`.

## Synthesize the CloudFormation templates

Once you have created your CDK app, you can synthesize the CloudFormation templates for your stacks. To synthesize the CloudFormation templates, run the following command:

```bash
cdk synth
```

This will synthesize the CloudFormation templates for all the stacks in your CDK app. You can also synthesize the CloudFormation templates for a specific stack by running the following command:

```bash
cdk synth <stack_name>
```

Read more about working with multiple stacks in the [Working with multiple stacks]({filename}50000050-cdk-multiple-stacks.md) post.

## Compare between the current and the deployed stacks

After you have made changes to your CDK app, you can compare the changes between the current and the deployed stacks. To compare the changes, run the following command:

```bash
cdk diff
```

This will compare the changes between the current and the deployed stacks. You can also compare the changes for a specific stack by running the following command:

```bash
cdk diff <stack_name>
```

## Deploy the stacks

After you have made changes to your CDK app, you can deploy the stacks. To deploy the stacks, run the following command:

```bash
cdk deploy
```

This will deploy all the stacks in your CDK app. You can also deploy a specific stack by running the following command:

```bash
cdk deploy <stack_name>
```

## Destroy the stacks

If you want to delete all the resources created by your CDK app, you can destroy the stacks. To destroy the stacks, run the following command:

```bash
cdk destroy
```

This will destroy all the stacks in your CDK app. You can also destroy a specific stack by running the following command:

```bash
cdk destroy <stack_name>
```

---

# Creating multiple stacks in AWS CDK

URL: https://cloudbytes.dev/aws-academy/creating-multiple-stacks-in-aws-cdk
Category: AWS Academy
Published: 2023-10-24
Author: Rehan Haider
Tags: aws, cdk, python

> Guide to the process of creating and managing multiple stacks in a single AWS CDK application

In AWS CDK (Cloud Development Kit), a stack is a deployable unit that represents a collection of AWS resources. Sometimes, there's a need to manage multiple stacks either for logical separation, different environments, or to manage AWS resource limits. 

In this post, we will create multiple stacks in the app and deploy them.

## Why create multiple stacks?

1. **Logical Separation**: You might want to separate resources logically, like networking resources in one stack and database resources in another.
2. **Resource Limits**: AWS CloudFormation has a limit on the number of resources per stack; dividing resources among multiple stacks can be a solution.
3. **Environment Management**: Manage different environments like development, staging, and production with distinct stacks
4. **Maintainability**: It's easier to maintain multiple stacks than a single stack with a large number of resources.

## Creating multiple stacks

Start a new project by running the following command:

```bash
cdk init app --language python
```

### 1. Create the first stack

Rename the `cdk_app_stack.py` file to `first_stack.py` and modify the code as follows:

```python
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from constructs import Construct

class FirstStack(Stack):

    BUCKET_ID = "MyFirstBucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        s3.Bucket(self, id="MyFirstBucket", removal_policy=RemovalPolicy.DESTROY)
```

This is the same code that we used in the [previous post]({filename}50000030-cdk-update-app.md) to create a bucket with a destroy policy.


### 2. Create the second stack

Create a new file called `second_stack.py` under the `cdk_app` folder and modify the code as follows:

```python
from aws_cdk import (
    Stack,
    aws_lambda as _lambda
)

from constructs import Construct

class SecondStack(Stack):
    
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        _lambda.Function(self, 
            id="MyFirstLambda",
            runtime=_lambda.Runtime.PYTHON_3_7,
            code=_lambda.Code.from_inline("def main(event, context):\n\tprint('Hello World')"),
            handler="index.main",
        )
```

This simple stack creates a Lambda function that prints "Hello World" to the console when invoked.

### 3. Modify the app.py file

The original `app.py` only has the default single stack that was created when we initialized the project. This stack no longer exists, so we need to modify the `app.py` file to include the two new stacks.

We first import the two new stacks:

```python
from cdk_app.first_stack import FirstStack
from cdk_app.second_stack import SecondStack
```

Then we add the two stacks to the app:

```python
app = cdk.App()

FirstStack(app, "FirstStack")
SecondStack(app, "SecondStack")
```

The final `app.py` file looks like this:

```python
import os
import aws_cdk as cdk

from cdk_app.first_stack import FirstStack
from cdk_app.second_stack import SecondStack


app = cdk.App()

FirstStack(app, "FirstStack")
SecondStack(app, "SecondStack")

app.synth()
```

### 4. Synthesize multiple stacks

Synth of the stacks is done in the same way as before:

```bash
cdk synth
```

This will synthesise both stacks and create the CloudFormation templates for in the `cdk.out` folder.

### 5. Deploy multiple stacks

To get the list of stacks that are available in the app, run the following command:

```bash
cdk ls
```
![cdk ls output](/images/50000050-01-cdk-ls-output.png)


While deploying, we can either deploy only one stack, or all stacks together. To deploy a single stack, we have to specify the stack name:

```bash
cdk deploy FirstStack
```

And to deploy all stacks, we can simply run the following command:

```bash
cdk deploy --all
```

### 6. Destroy multiple stacks

To destroy a single stack, we have to specify the stack name:

```bash
cdk destroy FirstStack
```

And to destroy all stacks, we can simply run the following command:

```bash
cdk destroy --all
```

---

# Fix or configure Git authentication in WSL2

URL: https://cloudbytes.dev/snippets/fix-or-configure-git-authentication-in-wsl2
Category: Snippets
Published: 2023-10-24
Author: Rehan Haider
Tags: git, wsl2

> Configure Git authentication in WSL2 and avoid entering credentials every time

If you are using Git in WSL2, you might have noticed that you have to enter your username and password when working with private repositories or every time you push to a remote repository. This occurs because Git is not able to access the credentials stored in the Windows Credential Manager.

![Git auth error wsl2](/images/99999954-01-git-auth-error.png)



Let's see how to fix this issue.

## Pre-requisites

You need the following to complete this guide:

1. WSL2 installed and configured [(Guide to configure WSL2 on Windows 10 or Windows 11)]({filename}99999965-install-wsl2.md)
2. Git CLI is installed - download from [here](https://git-scm.com/downloads)


## Configure Git authentication in WSL2

Open the WSL2 terminal and follow the steps below

### 1. Git configuration / config file

First, set your name by running the following command:

```bash
git config --global user.name "Your Name"
```

Next, set your email address by running the following command:

```bash
git config --global user.email "youremail@domain.com"
```

### 2. Configure Git Credential Manager

This part is important, and it depends on the version of Git installed on the Windows part of your OS (Not WSL2).

#### 2.1. Check Git version

Open the Microsoft Terminal / Powershell / CMD and run the following command:

```bash
git --version
```

![Git version windows](/images/99999954-02-git-version-windows.png)


#### 2.2. Configure Git Credential Manager

Now go back to the **WSL2 terminal** and based on the Git version, run the appropriate command below:

* If the Git version is greater than `v2.39.0`, run the following command:

```bash
git config --global credential.helper "/mnt/c/Program\ Files/Git/mingw64/bin/git-credential-manager.exe"
```

* If the Git version is between `v2.36.1`, and `v2.39.0` run the following command:

```bash
git config --global credential.helper "/mnt/c/Program\ Files/Git/mingw64/libexec/git-core/git-credential-manager.exe"
```

* If, the Git version is less than `v2.36.1`, run the following command:

```bash
git config --global credential.helper "/mnt/c/Program\ Files/Git/mingw64/bin/git-credential-manager-core.exe"
```

### 3. Test Git authentication

Now, try to clone a private repository or push to a remote repository. You should not be prompted for credentials.

---

# CDK Bootstrap: Setting up your AWS account for CDK

URL: https://cloudbytes.dev/aws-academy/cdk-bootstrap-setting-up-your-aws-account-for-cdk
Category: AWS Academy
Published: 2023-10-22
Author: Rehan Haider
Tags: aws, cdk

> This article explains what is CDK Bootstrap and how to setup your AWS account for use with CDK

Before you can start using CDK, you need to configure your AWS account for use with CDK. This initial setup of AWS environment is done by `cdk bootstrap`` command. In this post, we'll delve into what CDK bootstrap does, why it's necessary, and how to use it effectively.

## What is CDK Bootstrap?

Bootstrap is a process that creates resources in your AWS account that are necessary for CDK to work. At its core, the CDK bootstrap is an initializer for your AWS environment, prepping it for subsequent CDK deployments.

### Why Do We Need to Bootstrap?

When you deploy a CDK stack, you're essentially asking the CDK to:

1. Synthesize an AWS CloudFormation template from your high-level code.
2. Store any necessary assets (Lambda code bundles, Docker images, etc.).
3. Use CloudFormation to deploy the defined resources based on the synthesized template and the stored assets.

To facilitate these steps, the CDK requires an environment where it can reliably store assets and manage the deployment. Bootstrapping creates that environment.

### Components of CDK Bootstrap

When you run the cdk bootstrap command, several resources are created:

1. **S3 Bucket**: This bucket is used to store assets for your CDK apps, such as Lambda deployment packages, Docker images, or CloudFormation templates. It's named cdktoolkit-stagingbucket-[unique ID].
2. **Ephemeral CloudFormation Stack**: Named CDKToolkit, this stack manages the resources required by the CDK, including the aforementioned S3 bucket and the IAM roles.
3. **IAM Roles**: The bootstrap process sets up roles that allow the CDK and CloudFormation to create and manage resources on your behalf.

![CDK Bootstrap cloudformation stack](/images/50000010-01-cdk-bootstrap-stack.gif)


## How to Bootstrap Your AWS Account

1. **Initialization**: If you have already configured AWS CDK, and you're deploying a CDK app for the first time in an AWS environment (or a specific AWS region/account combination), you'll need to run:

```bash
cdk bootstrap
```

---

# Creating a new CDK app with Python

URL: https://cloudbytes.dev/aws-academy/creating-a-new-cdk-app-with-python
Category: AWS Academy
Published: 2023-10-22
Author: Rehan Haider
Tags: aws, cdk, python

> How to create a new CDK app that uses Python as the programming language

In this post, we'll create a new CDK app that uses Python as the programming language. 

!!! note
    Ensure that [AWS CDK is installed & configured]({filename}00000100-cdk-installing-cdk-sam-cli.md) before proceeding.


## Creating a new CDK app

a) Create a new directory for the CDK app and navigate to it:

```bash
mkdir cdk-app && cd cdk-app
```


b) To create a new CDK app, run the following command:

```bash
cdk init app --language python
```

The above command will create the following:

1. **CDK App**: Create a new CDK app in the current directory. It includes a `cdk.json` file that includes the configuration for the CDK app
2. **Stacks**: Create a folder called `cdk-app` in the current directory that includes the `stacks` that will be deployed
3. **Python Virtual Environment**: Create a Python virtual environment for the app in the .venv folder. It also creates a `requirements.txt` file that includes the Python dependencies for the app
4. **Git**: Create a git repository for the app including a `.gitignore` file


c) Activate the Python virtual environment:

```bash
source .venv/bin/activate
```

If you are using Windows, run the following command instead:

```powershell
./source.bat
```

d) Install the Python dependencies:

```bash
pip install -r requirements.txt -r requirements-dev.txt
```

## Modifying the CDK app

a) Open the `app.py` file. This file includes the basic CDK app that includes a single stack called `CdkAppStack`. 

```python
import aws_cdk as cdk

from cdk_app.my_stack import MyStack

app = cdk.App()
my_stack = MyStack(
    app,
    "MyStack",
)

app.synth()
```

In the above code, we are importing the `MyStack` class from the `cdk_app/my_stack.py` file.

b) Open the `MyStack.py` file. This file includes the `MyStack` class that extends the `cdk.Stack` class. 

```python
from aws_cdk import (
    Stack,
)
from constructs import Construct

class MyStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # The code that defines your stack goes here

```

Currently, the `MyStack` class does not include any resources. Let's add a S3 bucket to the stack.

c) Modify the `my_stack.py` file as follows:

```python
from aws_cdk import (
    Stack,
    aws_s3 as s3,
)

from constructs import Construct

class MyStack(Stack):

    BUCKET_ID = "MyFirstBucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # The code that defines your stack goes here
        s3.Bucket(self, id=self.BUCKET_ID)

```
Change the `BUCKET_NAME` variable to a name of your choice. This is not the name of the S3 bucket but the ID by which the S3 bucket will be referenced in the CDK app.

## Deploying the CDK app

a) Run `cdk synth` to synthesize the CDK app. Synth will generate a CloudFormation template for the CDK app. You can see the generated template in the `cdk.out` folder.

b) Run `cdk deploy` to deploy the CDK app. CDK will deploy the S3 bucket to your AWS account. You can see the deployed S3 bucket in the AWS Console.

![Deployed S3 bucket](/images/50000020-01-cdk-deploy-cloudformation.png)

You can see the deployed S3 bucket in the AWS Console.

## Destroying the CDK app

To destroy the CDK app, run the following command:

```bash
cdk destroy
```

However, this will not delete the S3 bucket. You have to manually delete the S3 bucket from the AWS Console.
This is the default behaviour of CDK so you don't accidentally delete S3 buckets that contain important data.

---

# Setup your dev environment for use with AWS

URL: https://cloudbytes.dev/aws-academy/setup-your-dev-environment-for-use-with-aws
Category: AWS Academy
Published: 2023-10-22
Author: Rehan Haider
Tags: aws

> Instructions on how to setup the optimial development environment for use with AWS and this course

To complete this course, configure your system as per the instructions below per your OS. I personally use **Windows 10/11** with **WSL2** and **Ubuntu 22.04 LTS**. 

However, follow the instructions below as per your OS of choice:

1. **[WSL2 on Windows 10/11](#wsl2-on-windows-1011)**
2. **[Ubuntu Linux](#ubuntu-linux)**
3. **[Windows 10/11](#windows-1011)**
4. **[MacOS](#macos)**

We will install the following tools:

1. **VSCode** for editing code
2. **Docker Desktop** for running containers
3. **AWS CLI** for interacting with AWS


After that we will install **AWS CDK** for creating and managing AWS resources.

## WSL2 on Windows 10/11

a) Install the following prerequisites:

1. Install **Windows Terminal** from **[Microsoft Store](https://apps.microsoft.com/store/detail/windows-terminal/9N0DX20HK701?hl=en-us&gl=US)**.
2. Install VSCode from **[Microsoft VSCode Website](https://code.visualstudio.com/download)**
3. Install **[Docker for Desktop](https://www.docker.com/products/docker-desktop/)**

b) After that, follow the below instructions to install and configure WSL2 on Windows 10/11.

1. Install **WSL2 (Windows Subsystem for Linux 2)** on you PC by following **[these instructions]({filename}/articles/99999965-install-wsl2.md)**
2. Open ***Windows Terminal*** and run `wsl` to login to your **WSL2 environment**

c) Once you're logged into your WSL2 environment, follow the below instructions to install and configure AWS CLI.

1. Follow **[these instructions]({filename}/aws/12500000-aws-cli-intro.md)** to install and configure **AWS CLI**


Now you can proceed to [CDK installation instructions below](#install-aws-cdk).


## Ubuntu Linux

a) Install the following prerequisites:

1. Install VSCode from **[Microsoft VSCode Website](https://code.visualstudio.com/download)**
2. Install **[Docker for Desktop](https://www.docker.com/products/docker-desktop/)**

b) After that, follow the below instructions to install and configure AWS CLI.

1. Follow **[these instructions]({filename}/aws/12500000-aws-cli-intro.md)** to install and configure **AWS CLI**


Now you can proceed to [CDK installation instructions below](#install-aws-cdk).

## Windows 10/11

a) Install the following prerequisites:

1. Install **Windows Terminal** from **[Microsoft Store](https://apps.microsoft.com/store/detail/windows-terminal/9N0DX20HK701?hl=en-us&gl=US)**.
2. Install VSCode from **[Microsoft VSCode Website](https://code.visualstudio.com/download)**
3. Install **[Docker for Desktop](https://www.docker.com/products/docker-desktop/)**


b) After that, follow the below instructions to install and configure AWS CLI.

1. Download and install the **[AWS CLI MSI Installer](https://awscli.amazonaws.com/AWSCLIV2.msi)**
2. Use these **[instructions to configure AWS CLI for usage]({filename}/aws/12500000-aws-cli-intro.md#configuring-aws-cli-for-use)**

Now you can proceed to [CDK installation instructions below](#install-aws-cdk).



## MacOS

a) Install the following prerequisites:

1. Install VSCode from **[Microsoft VSCode Website](https://code.visualstudio.com/download)**
2. Install **[Docker for Desktop](https://www.docker.com/products/docker-desktop/)**

b) Proceed to **AWS CLI** installation instructions below. You need `sudo` access.

To install **AWS CLI** on **MacOS**, first download the latest AWS CLI package from AWS by running

```bash
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
```

Next, install the package by running

```bash
sudo installer -pkg AWSCLIV2.pkg -target /
```

Verify that AWS CLI has been installed & added to path by running

```bash
aws --version
```
c) You have to configure **[AWS CLI for usage by following these instructions]({filename}/aws/12500000-aws-cli-intro.md#configuring-aws-cli-for-use)**


Now you can proceed to [CDK installation instructions below](#install-aws-cdk).

## Install AWS CDK

Now that we have installed all the prerequisites, [we can install AWS CDK by following this guide]({filename}/aws/00000100-cdk-installing-cdk-sam-cli.md).

---

# Understanding Constructs in the AWS CDK

URL: https://cloudbytes.dev/aws-academy/understanding-constructs-in-the-aws-cdk
Category: AWS Academy
Published: 2023-10-22
Author: Rehan Haider
Tags: aws, cdk, python

> Dive deep into the fundamental building blocks of the AWS Cloud Development Kit: Constructs

Constructs are fundamental building blocks of the AWS Cloud Development Kit. They represent cloud components in the form of programmable classes. When AWS CDK apps are synthesized, these Constructs are translated into CloudFormation templates which AWS can then deploy.

## What are Constructs?

Constructs are essentially classes that can be reused to create cloud resources. The stack that you define in your CDK app is a collection of initialised Constructs.

In previous posts, [we created a new CDK App]({filename}50000020-cdk-new-app.md) which used the `Bucket` Construct to create an S3 bucket.

```python
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from constructs import Construct

class CdkAppStack(Stack):

    BUCKET_ID = "MyFirstBucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Create an S3 bucket
        s3.Bucket(self, id=self.BUCKET_ID) # This is a Construct
```

## What are the different types of Constructs?

![Types of constructs](/images/50000040-01-cdk-constructs-types.png)

As shown in the diagram above, there are three types of Constructs:

1. **L1 Constructs: CFN Resources**: These are low-level constructs that map directly to CloudFormation resources . Each L1 Construct represents one CloudFormation resource type, such as an S3 Bucket or an EC2 Instance. These L1 Constucts are the foundation of all other Constructs and are automatically generated from the [CloudFormation Resource Specification](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html).

Example:
```python
from aws_cdk.aws_s3 import CfnBucket
bucket = CfnBucket(self, "MyBucket", bucket_name="my-bucket-name")
```

2. **L2 Constructs:  Curated Constructs**: These are higher-level abstractions that provide sensible defaults and ease-of-use. They encapsulate multiple L1 or L2 Constructs, providing additional functionality and a more intuitive interface. The difference between L1 and L2 Constructs is that L2 Constructs are created by AWS CDK developers and usually have certain defaults set. For example, the `Bucket` Construct is an L2 Construct that encapsulates the `CfnBucket` L1 Construct. The `Bucket` Construct automatically computes the `bucket_name` based on app name and bucket ID.

Example:
```python
from aws_cdk.aws_s3 import Bucket
bucket = Bucket(self, "MyBucket")
```

3. **L3 Constructs: Patterns**: These are the highest-level abstractions that are not part of the core AWS CDK library. They are usually created by AWS CDK developers and are available as separate libraries. These Constructs are not part of the core AWS CDK library because they are not generic enough to be used by everyone. 

Typically, L3 constructs are not part of the standar AWS CDK library and needs to be installed. You can find a list of available L3 constructs on the [AWS Constructs Hub](https://constructs.dev).


## Creating your own Constructs

You can create your own Constructs by extending the `Construct` class. 

Let's create a custom Construct that creates an S3 bucket with a lifecycle policy that deletes objects after 30 days.

```python
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from constructs import Construct

class CustomBucket(Construct):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Create an S3 bucket
        bucket = s3.Bucket(self, id="MyBucket", removal_policy=RemovalPolicy.DESTROY)

        # Add lifecycle policy to delete objects after 30 days
        bucket.add_lifecycle_rule(expiration=Duration.days(30))
```

Now, we can use this custom Construct in our CDK app as follows:

```python
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy,
)

from constructs import Construct

class CdkAppStack(Stack):

    BUCKET_ID = "MyFirstBucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Create an S3 bucket
        CustomBucket(self, id=self.BUCKET_ID)
```

You can deploy this app by running `cdk deploy`. When you destroy the stack by running `cdk destroy`.

---

# Update an existing CDK app

URL: https://cloudbytes.dev/aws-academy/update-an-existing-cdk-app
Category: AWS Academy
Published: 2023-10-22
Author: Rehan Haider
Tags: aws, cdk, python

> How to update an existing CDK app

In this post, we'll update an existing CDK app that uses Python as the programming language.

If you haven't created a new CDK app yet, follow the steps in [Creating a new CDK app with Python]({filename}50000020-cdk-new-app.md).

!!! note
    Ensure that [AWS CDK is installed & configured]({filename}00000100-cdk-installing-cdk-sam-cli.md) before proceeding.

## Updating an existing CDK app

We can simply update the stack bby modifying the `cdk_app/cdk_app_stack.py` file.

For example, we can change the retention policy by modifying the `cdk_app/cdk_app_stack.py` file as follows:

```python

from aws_cdk import (
    Stack,
    aws_s3 as s3,
    RemovalPolicy, # New Import
)

from constructs import Construct

class CdkAppStack(Stack):

    BUCKET_ID = "MyFirstBucket"

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Updated code. This will delete the bucket when the stack is deleted
        s3.Bucket(self, id="MyFirstBucket", removal_policy=RemovalPolicy.DESTROY)
```

Check the changes by running `cdk diff`. As you can see below, there is a change in Bucket policy being implemented.

![CDK diff](/images/50000030-01-cdk-cdk-diff-changes.png)


Now run `cdk deploy` to deploy the changes. When you destroy the stack by running `cdk destroy`, you'll notice that the bucket is deleted as well.

---

# Install and configure AWS Cloud Development Kit (AWS CDK)

URL: https://cloudbytes.dev/aws-academy/install-and-configure-aws-cloud-development-kit-aws-cdk
Category: AWS Academy
Published: 2023-10-15
Author: Rehan Haider
Tags: aws, cdk

> A guide to how to install and configure AWS CDK on Windows 10/11, MacOS, and WSL2/Linux.

In this article, we will learn how to install and configure AWS CDK and SAM CLI. This is a prerequisite for the AWS CDK series. There are two steps to this process:
1. Install NVM and Node.js
2. Install AWS CDK CLI using NPM
3. Install AWS SAM CLI

Now you may think why do we need to install NVM and Node.js instead when we will use Python? 

Well, AWS CDK CLI is mainly available as a NPM package so installing it from NPM is the easiest way. This allows us to use the CDK CLI to create and manage CDK projects. 

The actual CDK project will be written in Python but we will use the CDK CLI to create and manage the project.

## Prerequisites

1. **AWS CLI**: Follow the guide [here]({filename}/aws/12500000-aws-cli-intro.md) to install and configure AWS CLI for your operating system.
2. **AWS SAM CLI**: Follow the guide [here](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) to install and configure AWS SAM CLI for your operating system.


## Install NPM & Node.js

1. [For WSL2 on Windows 10/11 or Linux](#for-wsl2-on-windows-1011)
2. [For Windows 10/11](#for-windows-1011)
3. [For MacOS](#for-macos) 


### For WSL2 on Windows 10/11 or Linux
We will be using Ubuntu 22.04 LTS on WSL2 on Windows 10/11 for this series. But this guide should work for any other version. We will install NVM, Node.js, AWS CDK, and AWS SAM CLI.

a) Let's first make sure curl is installed.
```bash
sudo apt-get install curl -y
```

b) Then, install NVM & NPM using the following command.
```bash
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash
```

c) Then, restart the shell then run the below command to verify that NVM is installed.
```bash
nvm --version
```

d) Now, install Node.js using the following command. I prefeer installing the LTS version to ensure a bug-free experience.
```bash
nvm install --lts
```

![Install Node.js using NVM](/images/00000000-01-wsl2-nvm-nodejs.png)

After this, jump to the [Install AWS CDK CLI](#install-aws-cdk-cli) section.

### For Windows 10/11

If WSL2 is not available, you can install Node.js on Windows 10/11 directly.

a) First, install `nvm-windows` by downloading the installer from [here](https://github.com/coreybutler/nvm-windows/releases/download/1.1.11/nvm-setup.exe).
![Install nvm-windows](/images/00000000-02-nvm-windows-installer.png)


b) Confirm the installation by running the following command in PowerShell.
```powershell
nvm --version
```

c) Finally, install Node.js using the following command.
```powershell
nvm install lts
```

![Install Node.js using nvm-windows](/images/00000000-03-windows-nvm-nodejs.png)

After this, jump to the [Install AWS CDK CLI](#install-aws-cdk-cli) section.

### For MacOS

a) For MacOS, you need `homebrew` installed. If you don't have it installed, you can install it using the following command.

```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

b) Add `homebrew` to your PATH by running the following command.
```bash
echo "# Homebrew\nexport PATH=/opt/homebrew/bin:\$PATH" >> .zshrc
```

c) Then, restart the shell.
```bash
source ~/.zshrc
```

d) Install nvm using the following command.
```bash
brew install nvm
```

e) We will need to create a directory for nvm to store its files. Run the following command to create the directory.
```bash
mkdir ~/.nvm
```

f) Then, nvm to to your `~/.zshrc` profile.
```bash
echo "export NVM_DIR=~/.nvm\nsource \$(brew --prefix nvm)/nvm.sh" >> .zshrc
```

g) Restart the shell again.
```bash
source ~/.zshrc
```

h) Finally, install Node.js using the following command.
```bash
nvm install --lts
```

## Install AWS CDK CLI

Now that we have installed NVM and Node.js, we can install AWS CDK CLI using NPM.

Run the following command to install AWS CDK CLI.
```bash
npm install -g aws-cdk
```

![Install AWS CDK CLI](/images/00000000-04-aws-cdk-install.png)

---

# CDK API Gateway with Custom Domain

URL: https://cloudbytes.dev/aws-academy/cdk-api-gateway-with-custom-domain
Category: AWS Academy
Published: 2022-10-24
Author: Rehan Haider
Tags: aws, cdk, python

> Create API Gateway and connect it with a custom domain / subdomain using Route53

Let me begin by stating it took almost 3 weeks to figure this out and yet the AWS CDK Documentation on API Gateway is so bad, I was able to get everything working except the `base_path`. I do appreciate any pointers if you may have them. 

If you know what API Gateway is, TL;DR jump to [creating the API Gateway with CDK](#create-a-new-api-gateway).

## What is API Gateway?

API Gateway is a serverless service from AWS that helps you create API Endpoints which can be connected with other AWS services such as Lambda, Step Functions, etc. 

API Gateway forms the foundation of serverless design that allows developers to create APIs that are infinitely scalable and easily connectable to other AWS services that are used to build serverless applications. 

![API Gateway Architecture](/images/87500000-01-api-gateway-architecture.png)

Users and consumers from around the globe can be given an app or a website that calls these APIs to authenticate users, and fetch data from the backend which can also be serverless giving massive cost savings and scalability.

API Gateway gives you the ability to create 3 types of APIs
1. REST APIs (part of API Gateway V1)
2. HTTP APIs (part of API Gateway V2)
3. Websockets APIs (part of API Gateway V2)

## What is Route53?

Route53 is a DNS service from AWS that allows you to create custom domains and subdomains for your applications. It also allows you to register domains and manage DNS records for your domains.

## Connect API Gateway to a custom domain

When you create an API Gateway, by default it provides you with a URL that looks like this

```http
https://<api-id>.execute-api.<region>.amazonaws.com/<stage>
```
The API ID is a unique identifier for you API Gateway and is a random string of characters that changes every time you deploy your API Gateway. This is not ideal if you want to give the endpoint to your users or customers.

In these cases you would want to create a custom domain that is easy to remember. 


To do so, you need to do the following

1. Create a certificate in Amazon Certificate Manager (ACM) that maps to the domain you want to use. This step requires you to also create records in Route53 to verify domain ownership
2. Create an API Gateway
3. Attach the domain and certificate to the API Gateway
4. Create an Route53 A record alias that maps to the domain 

For this example I am going to use the sample domain `example.com` and map the API gateway to a custom domain `api.example.com`.


### Pre-requisites

1. You need to own a domain name registered with AWS Route53. 
2. Have [AWS Cloud Development Kit (CDK)](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) Installed

### Create a new project

Open your terminal and create a new directory for your project

```
mkdir api_route53 && cd api_route53
```

Next create a new CDK Project by running the following command

```
cdk init app --language=python
```

Open the folder in VSCode 

```
code .
```

You should see the following project structure already created

```
.
├── README.md
├── api_route53
│   ├── __init__.py
│   └── api_route53_stack.py
├── app.py
├── cdk.json
├── requirements-dev.txt
├── requirements.txt
├── source.bat
└── tests
```

Finally, install the required dependencies

```
pip install -r requirements.txt
```

Also install the following library - more on this later

```
pip install aws-cdk.aws-lambda-python-alpha
```


### Initialise the Stack

Open the file `api_route53/api_route53_stack.py` and import the libaries we will need and initialise the stack

```python
# api_route53/api_route53_stack.py :: Step 1
from aws_cdk import (
    Stack,
    aws_certificatemanager as acm,
    aws_route53 as route53,
    aws_apigateway as apigateway,
    aws_lambda as _lambda,
    aws_lambda_python_alpha as lambda_python,
    aws_route53_targets as targets,
)

from constructs import Construct


class ApiRoute53Stack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # The code that defines your stack goes here
```

We have defined and initialised a stack named `ApiRoute53Stack` that will contain the definition of the environment we want to create. 

### Create the ACM Certificate

First we fetch the hosted zone for the domain we want to use. 

```python
# api_route53/api_route53_stack.py :: Step 2

        hosted_zone = route53.HostedZone.from_lookup(self, "HostedZone", domain_name="example.com")

```

Next, let's create the certificate and validate it using DNS validation method.

```python
# api_route53/api_route53_stack.py :: Step 3
    
        certificate = acm.DnsValidatedCertificate(
            self,
            "ApiCertificate",
            domain_name="api.example.com",
            hosted_zone=hosted_zone,
            region="us-east-1",
        )
```
We have used us-east-1 to create the certificate because we intend to create `edge-optimised` API Gateway for which the certificate must be created in us-east-1. If you choose to create a `regional` API Gateway the certificate must reside in the region where API Gateway is created.

### Create the lambla function

A) Create a new file `api_route53/lambda_function.py` and add the following code

```python
# api_route53/lambda_function.py
import json

def lambda_handler(event, context):
    print(event)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }
```

B) Now back in the file `api_route53/api_route53_stack.py` we can create a lambda function that will be used by the API Gateway

```python
# api_route53/api_route53_stack.py :: Step 4

        # Create a lambda function
        handler = _lambda.Function(
            self,
            "ApiHandler",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="lambda_function.lambda_handler",
            code=_lambda.Code.from_asset("api_route53"),
        )
```

Here we chose Python 3.10 as our runtime, and the handler is the function in the lambda file that will be called when the API Gateway is invoked. The code is loaded from the `api_route53` directory.

# Create the API Gateway

Now we create an api gateway and
1. Attach a lambda function to it
2. Add a `domain_name` to the API Gateway and map it to the certificate we created earlier

```python
# api_route53/api_route53_stack.py :: Step 5
        # Create an API Gateway
        api = apigateway.LambdaRestApi(
            self,
            "ApiGateway",
            handler=handler,
            domain_name=apigateway.DomainNameOptions(
                domain_name="api.example.com",
                certificate=certificate,
                security_policy=apigateway.SecurityPolicy.TLS_1_2,
                endpoint_type=apigateway.EndpointType.EDGE,
            )
        )
```

Now keep in mind that API Gateway can have multiple Endpoints and creating the above `domain_name` property does not remove the default Endpoint in form on `https://<api-id>.execute-api.<region>.amazonaws.com/<stage>`, instead you will have two endpoints.

### Create the Route53 A record

Finally, we create a Route53 A record that maps the domain to the API Gateway

```python
# api_route53/api_route53_stack.py :: Step 6

        # Create a Route53 record
        route53.ARecord(
            self,
            "ApiRecord",
            record_name="api",
            zone=hosted_zone,
            target=route53.RecordTarget.from_alias(targets.ApiGateway(api)),
        )
```

### Deploy the stack

First we need to create the app by editing the `./app.py` file

```python
# app.py

import aws_cdk as cdk

from api_route53.api_route53_stack import ApiRoute53Stack


app = cdk.App()
env = cdk.Environment(account="<YOUR_ACCOUNT_ID>", region="us-east-1")

ApiRoute53Stack(app, "ApiRoute53Stack", env=env)

app.synth()
```

Finally, deploy the stack by running

```
cdk deploy
```

### Test the API

You can test the API by running the following command

```
curl https://api.example.com/api
```

This should return the following response

```
Hello from Lambda!
```

## Conclusion

The overall code for the API Stack is below. You can also find the code sample in this [Github Repo](https://github.com/rehanhaider/aws-cdk-code-samples/tree/main/apigw_route53)

```python
# api_route53/api_route53_stack.py

from aws_cdk import (
    Stack,
    aws_certificatemanager as acm,
    aws_route53 as route53,
    aws_apigateway as apigateway,
    aws_lambda as _lambda,
    aws_route53_targets as targets,
)

from constructs import Construct


class ApiRoute53Stack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # The code that defines your stack goes here

        # Fetch the hosted zone
        hosted_zone = route53.HostedZone.from_lookup(self, "HostedZone", domain_name="taskman.click")

        # Create a certificate
        certificate = acm.DnsValidatedCertificate(
            self,
            "ApiCertificate",
            domain_name="apix.taskman.click",
            hosted_zone=hosted_zone,
            region="us-east-1",
        )

        # Create a lambda function
        handler = _lambda.Function(
            self,
            "ApiHandler",
            runtime=_lambda.Runtime.PYTHON_3_10,
            handler="lambda_function.lambda_handler",
            code=_lambda.Code.from_asset("api_route53"),
        )
            
        # Create an API Gateway
        api = apigateway.LambdaRestApi(
            self,
            "ApiGateway",
            handler=handler,
            domain_name=apigateway.DomainNameOptions(
                domain_name="apix.taskman.click",
                certificate=certificate,
                security_policy=apigateway.SecurityPolicy.TLS_1_2,
                endpoint_type=apigateway.EndpointType.EDGE,
            )
        )

        # Create a Route53 record
        route53.ARecord(
            self,
            "ApiRecord",
            record_name="apix",
            zone=hosted_zone,
            target=route53.RecordTarget.from_alias(targets.ApiGateway(api)),
        )
```

---

# Run Jupyter Notebooks with Python Virtual Environments

URL: https://cloudbytes.dev/snippets/run-jupyter-notebooks-with-python-virtual-environments
Category: Snippets
Published: 2022-10-10
Author: Rehan Haider
Tags: python, jupyter

> Create an isolated Python virtual environment and use it with Jupyter Notebooks

Jupyter Notebooks are a great explaratory tool for writing code and testing out ideas. They are specially useful for data science and machine learning projects. However, like most Python projects, Jupyter Notebooks and dependencies can also get messy and hard to manage. One way to keep things organized is to use Python virtual environments. 

In this guide, we will see how to create a virtual environment and use it with Jupyter Notebooks.

## Install Jupyter Notebook

In case you want to install Jupyter Notebook from scratch, follow the steps in this [guide to install & run Jupyter Notebook]({filename}99999958-run-jupyter-from-terminal.md).

## Create a virtual environment

We will use `venv` to create a virtual environment. You can also use `conda` if you prefer.

A) Ensure you have `venv` installed. If not, run the following command:

```bash
sudo apt install python3-venv
```

B) Create a new virtual environment. We will call it `.env`. 

```bash
python3 -m venv .env
```

C) Activate the virtual environment. 

```bash
source .env/bin/activate
```

## Add the virtual environment to Jupyter Notebooks

A) Install `ipykernel` in the virtual environment. 

```bash
pip install ipykernel
```

B) Add the virtual environment to kernel list

```bash
python3 -m ipykernel install --user --name=.env
```

C) Start the notebook and select the virtual environment from the kernel list.

```bash
jupyter notebook
```

You should see the `.env` kernel in the list when you create a new notebook.

![Jupyter Notebook with virtual environment](/images/99999955-01-list-of-kernel.png)

---

# Cross Account Access to AWS Services using IAM Roles

URL: https://cloudbytes.dev/aws-academy/cross-account-access-to-aws-services-using-iam-roles
Category: AWS Academy
Published: 2022-08-13
Author: Rehan Haider
Tags: aws

> Access another AWS account temporarily using an IAM role and STS

Think of a situation where you want to give someone access to your AWS account temporarily but you don't want to create an IAM user for them. 

Just like other AWS services, you can create an IAM role with the permissions and have someone assume the role from their own account. 

![Cross account access using role](/images/45000000-01-cross-account-access.png)

As described above, we will create an IAM role in the trusting account (the account that will give access) and then use the role to assume a role in the trusted account (the account that will be granted access).


## Create an IAM Role in the Trusting Account

a) Go to the **IAM console** then navigate to the **Roles** section. Click on the **Create role** button.

b) Select `AWS account` in **Trusted entity type**, select `Another AWS account` in **An AWS account** section and then enter the `Account ID` in **Account ID** section as shown below. 

![Select trusted entity aws iam](/images/45000000-02-select-trusted-entity.png)

Click on **Next** button at the bottom right corner.

c) In **Add permissions** section, you can select the permissions you want to grant to the role. E.g. in this case I will grant **AmazonS3ReadOnlyAccess** permission to the role that will give the user the ability to list and view the contents of any S3 bucket. Then click on **Next** button at the bottom right corner.

d) In **Role details** choose a **Role name**, e.g. `my-s3-x-account-role` and then click on **Create role** button at the bottom right corner.

## Create a Policy in the Trusted Account

a) Now we need to create a policy in the trusted account. Login with a user that has the ability to create IAM policies. 

b) Go to the **IAM console** then navigate to the **Policies** section. Click on the **Create policy** button.

c) Switch to the JSON editor and paste the following code into it.

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<trusting-account>:role/<role-name>"
        }
    ]
}
```

Replace `<trusting-account>` with the account ID of the trusting account and `<role-name>` with the name of the role that you created in the trusting account, e.g. `my-s3-x-account-role` above.

Then click on **Next: Tags**.

d) In **Tags** section, you can add some tags to the policy. Then click on **Next: Review** button at the bottom right corner.

e) Choose a name for the policy, e.g. `my-s3-x-account-policy` and then click on **Create policy** button at the bottom right corner. You should see the permissions we granted above. Click on **Create** once done.

![Trusted account policy](/images/45000000-03-trusted-account-policy.png)

## Attach the policy to a user

a) Now we need to attach the policy to a user (or group) in the trusting account. Navigate to **Users** section in the **IAM console**.

b) Select the user you want to attach the policy to and then click on **Edit** button.

c) Under **Permissions** tab, click on **Add permissions** button. 

d) In **Grant permissions**, click on **Attach existing policies directly** and then select the policy you created above.

![Attach policy to user](/images/45000000-04-attach-permission.png)

e) Click on **Next: Review**, then click on **Add permissions** button.

## Assume the role in the trusted account

a) Login to the IAM user account that needs to be granted access.

b) Click on the *Settings* menu on top right corner, then click on "**Switch role**" button. Then in the **Switch Role** page, click on `Switch Role` button.

c) In the ensuring dialogue, 

- choose the ID of `trusting-account` as **Account**, 
- type the name of the role you created in the trusting account as **Role**,
- then click on **Switch Role** button.

d) Now you should see the new role in the **Account** dropdown.

![Assume role in trusted account](/images/45000000-05-switch-role-success.png)

---

# How to Install and Run WordPress on an EC2 Instance

URL: https://cloudbytes.dev/aws-academy/how-to-install-and-run-wordpress-on-an-ec2-instance
Category: AWS Academy
Published: 2022-08-12
Author: Rehan Haider
Tags: aws, linux, wordpress

> A detailed guide on how to configure, install, and run WordPress on an EC2 instance. This guide will cover installation of NGINX, PHP, MySQL, and WordPress (LEMP stack) on Amazon Linux 2

There are few technologies that divides the Web Developers as sharply as [WordPress](https://wordpress.org/). Launched in 2003, WordPress is a content management system (CMS) for creating and managing web sites that is **estimated to power more than 42% of websites** in the world.

It is loved by users because of it's Free and Open Source (FOSS) nature but hated by developers due to it's clunky and bloated codebase under the hood. 

Nevertheless, WordPress is going nowhere hence in this guide we will cover how to install, configure, and run WordPress on an EC2 instance.

## How to install WordPress on an EC2 instance?

We will do the following steps to install WordPress on an EC2 instance:

- [How to install WordPress on an EC2 instance?](#how-to-install-wordpress-on-an-ec2-instance)
  - [Create an EC2 instance](#create-an-ec2-instance)
  - [Install NGINX](#install-nginx)
  - [Install PHP](#install-php)
  - [Configure NGINX to serve PHP](#configure-nginx-to-serve-php)
  - [Install MySQL/MariaDB](#install-mysqlmariadb)
  - [Configure \& Secure MySQL/MariaDB](#configure--secure-mysqlmariadb)
  - [Create the database](#create-the-database)
  - [Install WordPress](#install-wordpress)
  - [Configure WordPress](#configure-wordpress)
  - [Start WordPress](#start-wordpress)

### Create an EC2 instance

We will use the instructions in this [previous guide]({filename}/aws/18750200-create-ec2-instance-using-cli.md) to create an EC2 instance using AWS CLI. 

If you wish to use AWS Management Console instead, follow the steps in [this guide]({filename}/aws/18750100-create-ec2-instance-console.md).


We will use `Amazon Linux 2` AMI and `t2.micro` instance for this tutorial.


a) Run the below script to create an EC2 instance:

```bash
AMI_ID=ami-090fa75af13c156b4
echo "Using AMI_ID: $AMI_ID"


INSTANCE_TYPE=t2.micro
echo "Using INSTANCE_TYPE: $INSTANCE_TYPE"

aws ec2 create-key-pair \
    --key-name my-key-pair \
    --query 'KeyMaterial' \
    --output text > my-key-pair.pem

chmod 400 my-key-pair.pem
echo "Created my-key-pair.pem"

SECURITY_GROUP=$(aws ec2 create-security-group \
    --group-name "my-web-sg" \
    --description "Web security group" \
    --query 'GroupId' \
    --output text) && \
echo "Security group created with id $SECURITY_GROUP"

aws ec2 authorize-security-group-ingress \
    --group-id $SECURITY_GROUP \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
    --group-id $SECURITY_GROUP \
    --protocol tcp \
    --port 80 \
    --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
    --group-id $SECURITY_GROUP \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0


SUBNET_ID=$(aws ec2 describe-subnets \
    --filters "Name=availability-zone,Values=us-east-1a" \
    --query "Subnets[0].SubnetId" --output text) && \
    echo "Subnet ID for us-east-1a: $SUBNET_ID"


INSTANCE_ID=$(aws ec2 run-instances \
    --image-id $AMI_ID \
    --count 1 \
    --instance-type $INSTANCE_TYPE \
    --key-name my-key-pair \
    --security-group-ids $SECURITY_GROUP \
    --subnet-id $SUBNET_ID \
    --associate-public-ip-address \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=my-ec2-instance}]' \
    --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":20,"VolumeType":"gp2"}}]' \
    --query 'Instances[0].InstanceId' \
    --output text) && \
echo "Instance launched with id $INSTANCE_ID"
```

b) Next, let's assign a static Elastic IP address to the instance.

```bash
aws ec2 allocate-address \
    --domain vpc \
    --query 'AllocationId' \
    --output text > my-eip.txt

aws ec2 associate-address \
    --allocation-id $(cat my-eip.txt) \
    --instance-id $INSTANCE_ID
```

c) Get the IP address of the instance:

```bash
INSTANCE_IP=$(aws ec2 describe-instances \
    --instance-ids $INSTANCE_ID \
    --query 'Reservations[0].Instances[0].PublicIpAddress' \
    --output text)
echo "Instance IP: $INSTANCE_IP"
```

d) Login to the EC2 instance:

```bash
ssh -i my-key-pair.pem ec2-user@$INSTANCE_IP
```
Type `yes` to accept the RSA fingerprint and login to the server.

You should see the welcome message similar to below:

![Login to EC2 Instance](/images/22575000-01-login-to-ec2-instance.png)


e) Update the packages by running the following command:

```bash
sudo yum update -y
```


f) Use the which command to confirm `amazon-linux-extras` is installed:

```bash
which amazon-linux-extras
```

You should see the following output:

```bash
/usr/bin/amazon-linux-extras
```

If you get an error, install the `amazon-linux-extras` package:

```bash
sudo yum install -y amazon-linux-extras
```

### Install NGINX

a) Check if NGINX is available from `amazon-linux-extras` repository:

```bash
sudo amazon-linux-extras list | grep nginx
```
This will list the latest version of NGINX available in the repository and its installation name.

![Check nginx amazon linux extras](/images/22575000-02-check-nginx-amazon-extras.png)

b) Enable the NGINX package for installation:

```bash
sudo amazon-linux-extras enable nginx1
```

This will print out the modules that have been enabled.

c) Install NGINX:

```bash
sudo yum clean metadata && sudo yum install nginx -y
```

d) Confirm that NGINX is installed by checking the version:

```bash
nginx -v
```

e) Start NGINX:

```bash
sudo systemctl start nginx
```

Now if you open the IP address of the instance in your browser, you should see the following message:

![NGINX is running](/images/22575000-03-nginx-is-running.png)

f) Configure NGINX to start on boot:

```bash
sudo systemctl enable nginx
```

### Install PHP

a) Check the versions of PHP available in the repository:

```bash
sudo amazon-linux-extras list | grep php
```

![Check php versions](/images/22575000-04-php-versions.png)

b) Enable the PHP package for installation:

```bash
sudo amazon-linux-extras enable php8.0
```

c) Install PHP:

```bash
sudo yum clean metadata && sudo yum install yum install php-cli php-pdo php-fpm php-mysqlnd -y
```

d) Confirm that PHP is installed by checking the version:

```bash
php -v
```

e) Start PHP:

```bash
sudo systemctl start php-fpm
```

f) Configure PHP to start on boot:

```bash
sudo systemctl enable php-fpm
```

### Configure NGINX to serve PHP

By default NGINX is configured to use `/usr/share/nginx/html` as the web root with the directory owned by `root` user. 

So if you wanted to edit the `index.html` file, you would need to run the below command, you will get an `Permission denied` error message:

```bash
echo "Hello World" > /usr/share/nginx/html/index.html
```

To fix this we will need to change the ownership of the web root to `nginx` user and add `ec2-user` to the group.

a) First create a new group called `nginx`:

```bash
sudo groupadd www-data
```

b) Add `ec2-user` to the group:

```bash
sudo usermod -a -G nginx ec2-user
```

c) Change the ownership of the web root to `www-data` user:

```bash
sudo chown -R ec2-user:nginx /usr/share/nginx/html
```
d) Logout and login again to pick up the group and new permissions:

```bash
exit
```

e) Restart NGINX:

```bash
sudo systemctl restart nginx
```

f) Create a new file called `phpinfo.php` in the web root that calls `phpinfo()`:

```bash
echo "<?php phpinfo(); ?>" > /usr/share/nginx/html/phpinfo.php
```

g) Now open the `<ip-address>/phpinfo.php` in your browser and you should see the following message:

![PHP Info](/images/22575000-05-php-info.png)


h) Delete the `phpinfo.php` file. It contains sensitive information about your system that you should not share.

```bash
rm /usr/share/nginx/html/phpinfo.php
```

### Install MySQL/MariaDB

MySQL is not really an open source software now, instead we will use the open source fork `MariaDB`.

a) Check the versions of MariaDB available in the repository:

```bash
sudo amazon-linux-extras list | grep mariadb
```

b) Enable the MariaDB package for installation:

```bash
sudo amazon-linux-extras enable mariadb10.5
```

c) Install MariaDB:

```bash
sudo yum clean metadata && sudo yum install mariadb -y
```

d) Confirm that MariaDB is installed by checking the version:

```bash
mysql --version
```

e) Start MariaDB:

```bash
sudo systemctl start mariadb
```

f) Configure MariaDB to start on boot:

```bash
sudo systemctl enable mariadb
```

### Configure & Secure MySQL/MariaDB

a) Start the interactive MariaDB installation shell:

```bash
sudo mysql_secure_installation
```

b) When prompted for current the root password, press `Enter` to accept the default. By default, there is no password set.

c) Next, you will be prompted to `Switch to unix_socket authentication [Y/n]`, type `Y` to accept. This will allow you to connect to the database using your EC2 command line directly.

d) Then change the root password to something secure. Type `Y` then enter the new password and reconfirm.

e) Next, you will be prompted to `Remove anonymous users [Y/n]`, type `Y` to accept.

f) Next, you will be prompted to `Disallow root login remotely [Y/n]`, type `Y` to accept.

g) Next, you will be prompted to `Remove test database and access to it [Y/n]`, type `Y` to accept.

h) Next, you will be prompted to `Reload privilege tables now [Y/n]`, type `Y` to accept.

This will complete the installation.

### Create the database 

a) Login to the MariaDB shell:

```bash
mysql -u root -p
```

b) Create a new database called `dbase`:

```bash
CREATE DATABASE dbase;
```

Exit the MariaDB shell:

```bash
exit
```

### Install WordPress

a) Download the latest WordPress release from [wordpress.org](https://wordpress.org/latest.zip).

```bash
wget https://wordpress.org/latest.zip
```

b) Unzip the WordPress release:

```bash
unzip latest.zip
```

c) Move the WordPress files to the web root:

```bash
mv wordpress/* /usr/share/nginx/html
```

d) Change the ownership of the web root to `nginx` user:

```bash
sudo chown -R ec2-user:nginx /usr/share/nginx/html
```

d) Delete the WordPress files:

```bash
rm -rf wordpress
```

### Configure WordPress

a) Navigate to the web root

```bash
cd /usr/share/nginx/html
```

b) Create a new file called `wp-config.php` from `wp-config-sample.php`:

```bash
cp wp-config-sample.php wp-config.php
```

c) Edit the `wp-config.php` 

```bash
nano wp-config.php
```
Make sure you don't use `sudo` when editing the file.

d) Make the following edits to the `wp-config.php` file:

1. Replace `database_name_here` with `dbase`.
2. Replace `username_here` with `root`.
3. Replace `password_here` with the password you set earlier.

Press `Ctrl+X` and `Y` followed by `Enter` to save and exit.

### Start WordPress

To start WordPress you will need to open the `<ip-address>/wp-admin/install.php` in your browser. You should see the following:

![Start WordPress](/images/22575000-07-start-wordpress.png)

a) On `Information Needed` tab, provide the requested information and click `Install WordPress`. This should complete the installation process. Now you should be able to access the `<ip-address>/wp-admin/` in your browser and login using the username and password you set. 

b) Open the `<ip-address>` in your browser to see the newly created site.

![New Site](/images/22575000-09-new-site.png)

---

# How to create an AWS EC2 instance using AWS CLI

URL: https://cloudbytes.dev/aws-academy/how-to-create-an-aws-ec2-instance-using-aws-cli
Category: AWS Academy
Published: 2022-08-09
Author: Rehan Haider
Tags: aws

> A guide to create an EC2 instance using AWS CLI and login to the server

[TOC]

EC2 (Elastic Cloud Compute) is the original IaaS cloud service launched by Amazon in the mid-2000s which since then has launched the new era of cloud computing.

EC2 allows you to create an manage a virtual machine (VM) in cloud without having to worry about the hardware or virtualisation software.

## How to create an EC2 instance?

We will create an EC2 instance in this tutorial using the [AWS CLI](#create-an-ec2-instance-using-aws-cli).

If you're looking for a tutorial on how to create EC2 instance using the AWS Console, please see below

1. [Create an EC2 instance using the AWS CLI]({filename}/aws/18750100-create-ec2-instance-console.md)

### Create an EC2 instance using AWS CLI

The syntax for creating an EC2 instance using AWS CLI is as follows:

```
aws ec2 run-instances \
    --image-id <ami-id> \
    --count 1 \
    --instance-type <instance-type> \
    --key-name <keypair-name> \
    --security-groups <security-group-name> \
    --subnet-id <subnet-id> \
    --associate-public-ip-address \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=<name>}]' \
    --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":20,"VolumeType":"gp2"}}]'
```

Hence to create an EC2 instance, we need the following:

1. [Choose AMI ID](#1-choose-ami-id): AMI ID of the image we want to use. In this case, we will use `ami-090fa75af13c156b4`. You can find this in AWS console while launching an instance manually as shown [here](#5-choose-the-ami)
2. [Choose Instance Type](#2-choose-instance-type): The instance type that we want to use. In this case, we will use `t2.micro`
3. [Create Key-Pair](#3-create-key-pair): Create a key pair to use, we will name it `my-key-pair`
4. [Create Security Group](#4-create-security-group): Create a security group that allows inbound traffic on SSH, HTTP, and HTTPS ports. We will name it `my-web-sg`
5. [Choose Subnet](#5-choose-subnet): The subnet that we want to use considering the region and availability zone. In this case, we want to use subnet corresponding to `us-east-1a`
6. [Associate Public IP](#6-7-8-associate-public-ip-tag-the-instance-configure-storage): We will allow to associate a public IP address to the instance.
7. [Tag the instance](#6-7-8-associate-public-ip-tag-the-instance-configure-storage): We will tag the instance with the name `my-ec2-instance`
8. [Configure Storage](#6-7-8-associate-public-ip-tag-the-instance-configure-storage): We will configure the storage to have 20GB of storage and type `gp2`
9. [Launch the instance](#9-launch-the-instance): Launch the instance

#### 1. Choose AMI ID

We want to find the details of the AMI ID of **Amazon Linux 2 AMI** in `us-east-1` region. While you can try to look it up using the CLI and `describe-images` command, it's way simpler just looking it up in the AWS console. E.g. you can see the AMI ID at the bottom in the below image the AMI ID

![18750000-05-choose-ec2-ami](/images/18750200-05-choose-ec2-ami.png)


We will use this AMI ID to create an EC2 instance. Run the below to store it in a variable.

```bash
AMI_ID=ami-090fa75af13c156b4
```

#### 2. Choose Instance Type

We will simply use `t2.micro` as the instance type. Let's store this in a variable.

```bash
INSTANCE_TYPE=t2.micro
```

#### 3. Create Key-Pair

a) We will create a key pair to use. We will name it `my-key-pair`.

```bash
aws ec2 create-key-pair \
    --key-name my-key-pair \
    --query 'KeyMaterial' \
    --output text > my-key-pair.pem
```

This stores the key pair in a file named `my-key-pair.pem` in the current directory.

b) Next we need to correct the permissions of the file.

```bash
chmod 400 my-key-pair.pem
```

#### 4. Create Security Group

a) Let's first create a security group named `my-web-sg` with description `Web security group` and store the ID in a variable.

```bash
SECURITY_GROUP=$(aws ec2 create-security-group \
    --group-name "my-web-sg" \
    --description "Web security group" \
    --query 'GroupId' \
    --output text) && \
echo "Security group created with id $SECURITY_GROUP"
```

b) Now, let's add inbound rules to the security group. We will add rules for SSH, HTTP, and HTTPS. First, let's add SSH rule.

```bash
aws ec2 authorize-security-group-ingress \
    --group-id $SECURITY_GROUP \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0
```

c) Now add HTTP rule.

```bash
aws ec2 authorize-security-group-ingress \
    --group-id $SECURITY_GROUP \
    --protocol tcp \
    --port 80 \
    --cidr 0.0.0.0/0
```

d) Finally, add HTTPS rule.

```bash
aws ec2 authorize-security-group-ingress \
    --group-id $SECURITY_GROUP \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0
```

#### 5. Choose Subnet

Let's choose the subnet corresponding to `us-east-1a` availability zone and store it in a variable

```bash
SUBNET_ID=$(aws ec2 describe-subnets \
    --filters "Name=availability-zone,Values=us-east-1a" \
    --query "Subnets[0].SubnetId" --output text) && \
    echo "Subnet ID for us-east-1a: $SUBNET_ID"
```

#### 6, 7, 8. Associate Public IP, Tag the instance, Configure Storage

We will directly configure these in out final command


#### 9. Launch the instance

```bash
INSTANCE_ID=$(aws ec2 run-instances \
    --image-id $AMI_ID \
    --count 1 \
    --instance-type $INSTANCE_TYPE \
    --key-name my-key-pair \
    --security-group-ids $SECURITY_GROUP \
    --subnet-id $SUBNET_ID \
    --associate-public-ip-address \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=my-ec2-instance}]' \
    --block-device-mappings '[{"DeviceName":"/dev/xvda","Ebs":{"VolumeSize":20,"VolumeType":"gp2"}}]' \
    --query 'Instances[0].InstanceId' \
    --output text) && \
echo "Instance launched with id $INSTANCE_ID"
```

Get the instance IP address

```bash
INSTANCE_IP=$(aws ec2 describe-instances \
    --instance-ids $INSTANCE_ID \
    --query "Reservations[0].Instances[0].PublicIpAddress" --output text) && \
    echo "EC2 instance myServer1 IP: $INSTANCE_IP"
```



## Logging into the EC2 instance

a) With he IP address, we can log into the instance. Open a terminal and run the following command to log into the instance.

```bash
ssh -i my-key-pair.pem ec2-user@$INSTANCE_IP
```

You may be asked for confirmation similar to below, type Yes and press enter.

```text
The authenticity of host '54.173.196.189 (54.173.196.189)' can't be established.
ECDSA key fingerprint is SHA256:jm9pK6nGCsOVkQKfeQTG080hrb3G8Y2k1jeDwkNF4Og.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
```

This should log you into the instance.

---

# How to create an AWS EC2 instance using AWS Console

URL: https://cloudbytes.dev/aws-academy/how-to-create-an-aws-ec2-instance-using-aws-console
Category: AWS Academy
Published: 2022-08-09
Author: Rehan Haider
Tags: aws

> A guide to create an EC2 instance using AWS Management console and login to the server

[TOC]

EC2 (Elastic Cloud Compute) is the original IaaS cloud service launched by Amazon in the mid-2000s which since then has launched the new era of cloud computing.

EC2 allows you to create an manage a virtual machine (VM) in cloud without having to worry about the hardware or virtualisation software.

## How to create an EC2 instance?

We will create an EC2 instance in this tutorial using the [AWS Management Console](#create-an-ec2-instance-using-aws-management-console). 

If you're looking for a tutorial on how to create EC2 instance using the AWS Console, please see below

1. [Create an EC2 instance using the AWS CLI]({filename}/aws/18750200-create-ec2-instance-using-cli.md)

### Create an EC2 instance using AWS Management Console

To create an EC2 instance using AWS Management Console, we need to do the following:

1. [Go to the AWS Management Console](#1-go-to-the-aws-management-console)
2. [Choose a region](#2-choose-a-region): In this tutorial, we will use us-east-1.
3. [Launch the instance](#3-launch-the-instance): Click on the Launch Instance button.
4. [Choose Name](#4-choose-name): Give the instance a name.
5. [Choose the AMI](#5-choose-the-ami): In this tutorial, we will use the Amazon Linux AMI.
6. [Choose the instance type](#6-choose-the-instance-type): In this tutorial, we will use t2.micro.
7. [Choose the key pair](#7-choose-the-key-pair): The key pair that you want to use to log into the instance.
8. [Choose the Availability Zone / Subnet](#8-choose-the-availability-zone-subnet): In this tutorial, we will use us-east-1a.
9. [Choose the security group](#9-choose-the-security-group): In this tutorial, we will create a new security group that permits SSH access.
10. [Choose Storage](#10-choose-storage): In this tutorial, we will use an `gp2` volume of 20GB
11. [Launch the instance](#11-launch-the-instance): Click on the Launch Instance button.


#### 1. Go to the AWS Management Console

a) Login to the [AWS Management Console](https://console.aws.amazon.com) and search for EC2 and then go to the EC2 console.

![18750000-01-open-aws-console](/images/18750100-01-open-aws-console.gif)

#### 2. Choose a region

a) From the dropdown in top right corner, select the region you want to use.

![18750000-02-select-region](/images/18750100-02-select-region.gif)

#### 3. Launch the instance

a) Click on the **Instances** on the left navigation panel  
b) Click on the 🔽 icon to the left of **Launch instances** button to expand the Instances section  
c) Click on the **Launch Instances** button  

![18750000-03-launch-instances](/images/18750100-03-launch-instances.png)

This will launch the interactive launch wizard.

#### 4. Choose Name

a) Under "**Name and tags**" enter a name for the instance. E.g., I chose the name `my-ec2-instance`

![18750000-04-name-instance](/images/18750100-04-name-instance.png)


#### 5. Choose the AMI

a) Under "**Application and OS Images (Amazon Machine Image)**" choose "**Amazon Linux**" by click on it  
b) This will automatically select the latset version of the AMI & platform Architecture

![18750000-05-choose-ec2-ami](/images/18750100-05-choose-ec2-ami.png)

#### 6. Choose the instance type

a) Under "**Instance type**" choose "**t2.micro**"

![18750000-06-choose-instance-type](/images/18750100-06-choose-instance-type.png)

#### 7. Choose the key pair

a) Under "**Key pair (login)**" if you already have a key pair, select it.   
b) If you dont, click on **Create a new key pair**

- In the **Create key pair** dialog, choose a **Key pair name**, leave the other options as default. Then click on **Create key pair** button at the bottom
- Click on the `refresh` button on the right of the *Select* dropdown to refresh the list of keypairs
- Select the key pair you created from the dropdown

#### 8. Choose the Availability Zone / Subnet

a) In the **Network settings**, click on **Edit** button  
b) Under Subnet, select the subnet you want to use. E.g., I chose the one that mapped to `us-east-1a`  

![18750000-07-subnets](/images/18750100-07-subnets.png)

c) Under, **Firewall (security groups)**, click on **Create security Group** radio button  
d) Enter a name for the security group. E.g., I chose the name `my-web-sg` the change the description to `Web security group`  


#### 9. Choose the security group
e) Under **Inbound secruity groups rules**, click on **Add security group rule**

- Change the type to `ssh`
- Change the source to `Anywhere`

f) Click on **Add security group rule** again, 

- Change the type to `http`
- Change the source to `Anywhere`

g) f) Click on **Add security group rule** again, 

- Change the type to `https`
- Change the source to `Anywhere`

![18750000-08-sg-inbound](/images/18750100-08-sg-inbound.png)

#### 10. Choose Storage

a) Under Configure storage, change the size to `20GB` and the type to `gp2`

![18750000-09-storage](/images/18750100-09-storage.png)


#### 11. Launch the instance

a) Scroll down to the bottom of the page and click on **Launch instance** button under **Summary**  
b) Wait for the instance to be created, after success, scroll down to the bottom and click on **View all instances** button 
c) If you don't see anything, hit the refresh icon at the top  

![18750000-11-launch-instance](/images/18750100-11-launch-instance.png)

Now your EC2 instance is created. Select the instance, at the bottom you should see the public IPv4 address. You can use this address to access the instance.

![18750000-12-public-ip](/images/18750100-12-public-ip.png)

## Logging into the EC2 instance

a) With he IP address, we can log into the instance. Open a terminal and run the following command to log into the instance.

```bash
ssh -i my-key-pair.pem ec2-user@$INSTANCE_IP
```

You may be asked for confirmation similar to below, type Yes and press enter.

```text
The authenticity of host '54.173.196.189 (54.173.196.189)' can't be established.
ECDSA key fingerprint is SHA256:jm9pK6nGCsOVkQKfeQTG080hrb3G8Y2k1jeDwkNF4Og.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
```

This should log you into the instance.

---

# Run Flask Apps on Elastic Beanstalk

URL: https://cloudbytes.dev/aws-academy/run-flask-apps-on-elastic-beanstalk
Category: AWS Academy
Published: 2022-07-30
Author: Rehan Haider
Tags: aws, python

> Create a simple Flask app and run on AWS Elastic Beanstalk

[AWS Elastic Beanstalk](https://aws.amazon.com/elasticbeanstalk/) is a service from AWS that makes it easier for developers who want to focus on their code to deploy their applications to AWS. 

Deploying an app to EC2 is a lot of System Admin work. It involves 

- setting up a new instance, 
- configuring the instance, 
- configuring storage,
- installing the runtime, 
- installing dependencies, 
- configuring web servers and ports,
- configure scalability, and load balancing.

After you've done all that, you somehow need to transfer your files to the VM and then start the app and ensure the app starts on reboot. And even if you manage to do all that you need to figure out how to troubleshoot and debug your app in case of failures. 

AWS Elastic Beanstalk abstracts all of that and reduces developers task to only uploading the code and customisations. 

In this guide we will look at how to deploy a sample [Flask](https://github.com/pallets/flask) app to AWS Elastic Beanstalk.

## Deploying a Flask App to AWS Elastic Beanstalk

To deploy a Flask app to AWS Elastic Beanstalk, you need to do the following in order:

1. **[Create your Flask app & provide Python dependencies](#1-create-the-flask-app)**: This will be your code that will be deployed to AWS Elastic Beanstalk along with `requirements.txt`
2. **[Zip your app](#2-zip-the-files)**: Zip all of your source code and dependencies into a single file.
3. **[Upload the zip to S3](#3-upload-the-files-to-s3)**: The zip file needs to be staged to S3 before deployment
4. **[Create Elastic Beanstalk Application](#4-create-the-elastic-beanstalk-application)**: This will deploy your code and create an application, but you must create the environment for your application to work.
5. **[Create Elastic Beanstalk Environment](#5-create-the-elastic-beanstalk-environment)**: Create the environment & start the application.


### 1. Create the Flask App

a) We will use the simple flask app below. Create a new file named `application.py` and paste the following code into it.

> The name of the file containing your code must be `application.py` the `Flask` object too should be name `application`

```python
# filename: application.py
from flask import Flask
application = Flask(__name__) # This needs to be named `application`


@application.route("/")
def index():
    return "<h1>Hello World!</h1>"


@application.route("/another")
def another():
    return "<h1>Another Page!</h1>"


if __name__ == "__main__":
    application.run(debug=True)
```

b) You also need to capture the dependecies in a file named `requirements.txt` and paste the following code into it.

```text
Flask
```
### 2. Zip the files

a) Install zip utility.

```bash
sudo apt-get install zip
```

b) Then zip the files.

```bash
zip -r aws-flask-app.zip application.py requirements.txt
```

### 3. Upload the files to S3

a) First you need to create an S3 bucket where these artefacts will be staged before deployment to Elastic Beanstalk.

```bash
aws elasticbeanstalk create-storage-location
```

This will output something similar to the following:

```
{
  "S3Bucket": "elasticbeanstalk-us-east-1-123456789012"
}
```

b) Then upload the artefacts to the S3 bucket mentioned above by running:

```bash
aws s3 cp aws-flask-app.zip s3://<S3 Bucket>/aws-flask-app.zip
```

### 4. Create the Elastic Beanstalk Application

a) To create an application run the following command.

```bash
aws elasticbeanstalk create-application-version \
    --application-name flask-app \
    --version-label v1 \
    --source-bundle S3Bucket="<S3 Bucket>",S3Key="aws-flask-app.zip" \
    --auto-create-application
```

Here we create a new application called `flask-app` and deploy the version `v1` of the application, any subsequent revisions will have to have a different version label.

We also provided the S3 bucket and key where the zip file is stored. 

The `--auto-create-application` flag will create the application if it does not exist.

This command has now created an app, that you can see in AWS Management Console.

![Beanstalk app created](/images/40000000-01-beanstalk-app-created.png)

### 5. Create the Elastic Beanstalk Environment

a) First you need to create an options file. Create a new file named `options.txt` and paste the following code into it.

```text
[
    {
        "Namespace": "aws:autoscaling:launchconfiguration",
        "OptionName": "IamInstanceProfile",
        "Value": "aws-elasticbeanstalk-ec2-role"
    }
]
```

This defines the IAM role that will be used to launch the instance.

b) Then create the environment by running.

```bash
aws elasticbeanstalk create-environment \
    --application-name flask-app \
    --environment-name flask-app-env \
    --version-label v1 \
    --solution-stack-name "64bit Amazon Linux 2 v3.3.15 running Python 3.8" \
    --option-settings file://options.txt
```

The application name must match the application name we specified in previous step. 
And the version label should match the version of the app that you want to deploy.

``--solution-stack-name`` is the name of runtime you want to use. Apart from the one that we specified, you can find [other supported platforms here](https://docs.aws.amazon.com/elasticbeanstalk/latest/platforms/platforms-supported.html)

This command has now created an environment and begun to launch it.

![beanstalk env created](/images/40000000-02-beanstalk-env-created.png)

If you click on the environment in the AWS Management Console, you will see the following:

![beanstalk env starting](/images/40000000-03-beanstalk-env-starting.png)

After a while you will see the following that confirms your app has been deployed and launched successfully

![beanstalk success](/images/40000000-04-beanstalk-success.png)

## Testing the App

On the left hand panel, under `flask-app-env`, you will see an option `Go to environment`. Click on it. 
This will open a URL in your browser which displays the Flask app you created. 

### 1. Getting the Beanstalk Endpoint URL using CLI

a) To fetch the URL of the Beanstalk endpoint, you can use the following command.

```bash
aws elasticbeanstalk describe-environments \
    --environment-name flask-app-env \
    --version-label v1 \
    --query 'Environments[0].EndpointURL'
```

This will display the URL of the Beanstalk endpoint.


## Updating the app

To updated you need to follow the same steps as above, except you need to update the version label. Follow steps 1, 2, and 3 to upload your new code to S3. 

### 1. Update the application

a) We create `v2` version of the application by running.

```bash
aws elasticbeanstalk create-application-version \
    --application-name flask-app \
    --version-label v2 \
    --source-bundle S3Bucket="<S3 Bucket>",S3Key="aws-flask-app.zip" \
    --auto-create-application
```

Note the `--version-label` is `v2` now.

### 2. Update the environment

a) We update the environment by running.

```bash
aws elasticbeanstalk update-environment \
    --environment-name flask-app-env \
    --version-label v2 \
    --option-settings file://options.txt
```

Note the application name and version labels correspond to the application & version you want to deploy.


## Delete the app

You need to first delete the environment then the application

To delete the environment, run the following command.

```bash
aws elasticbeanstalk terminate-environment \
    --environment-name flask-app-env
```

To delete the application, run the following command.

```bash
aws elasticbeanstalk delete-application \
    --application-name flask-app
```

To check if there are any applications left, run the following command.

```bash
aws elasticbeanstalk describe-application-versions
```

---

# Mount AWS Credentials on VSCode Devcontainers

URL: https://cloudbytes.dev/snippets/mount-aws-credentials-on-vscode-devcontainers
Category: Snippets
Published: 2022-06-28
Author: Rehan Haider
Tags: python, aws, vscode

> A guide to install, configure and run selenium in Jupyter Notebook on WSL2 or Ubuntu

I've written in past about my [preference of using VSCode's devcontainers for developing Python]({filename}99999997-replace-python-venv-with-vscode-devcontainers.md) applications. 

While I work extensively with Python & AWS, one of the problems I've faced is that everytime I create a new devcontainer or rebuild one, I need to enter my AWS credentials again.

So I set about fixing this problem. I explored the idea of creating environmental variables and loading them directly into the container. I thought about copying the credentials file itself onto the container during build. But of of them weren't clean ideas and had their own problems. 

Turns out all it requires is a simple statement on `devcontainer.json` file. 

## How to mount AWS credentials on VSCode Devcontainers

Edit the `.devcontainer/devcontainer.json` file, add the following line after `build` instructions:

```json
	"mounts": [
		"source=${localEnv:HOME}/.aws,target=/home/vscode/.aws,type=bind,consistency=cached"
	],
```
> This will only work if you're using WSL2-backed devcontainers.

If you are using pure Windows based devcontainers, the above instructions may not work as I discovered after following [this guide](https://prabhatsharma.in/blog/vscode-dev-container-aws-credentials/).

---

# Run selenium in Jupyter Notebook on WSL2 or Ubuntu

URL: https://cloudbytes.dev/snippets/run-selenium-in-jupyter-notebook-on-wsl2-or-ubuntu
Category: Snippets
Published: 2022-06-26
Author: Rehan Haider
Tags: python, selenium, wsl2, linux

> A guide to install, configure and run selenium in Jupyter Notebook on WSL2 or Ubuntu

We've seen in past how to [install and run Selenium with Python]({filename}99999966-run-selenium-wsl2.md), in this guide we will try to do the same in Jupyter Notebook on WSL2.

The instructions should be same for both WSL2 and Ubuntu.

## Install WSL2

Follow the steps in this [guide to install WSL2]({filename}99999965-install-wsl2.md).

## Install Jupyter Notebook

Follow the steps in this [guide to install & run Jupyter Notebook]({filename}99999958-run-jupyter-from-terminal.md).

## Install & configure Selenium

Run the script in this guide to automatically [install and configure Selenium]({filename}99999966-run-selenium-wsl2.md#creating-a-script-to-automate-the-process). 

You can also download [this script from GitHub](https://github.com/rehanhaider/selenium-wsl2-ubuntu)

If for some reason the script is stuck, press `Ctrl+C` to stop the script and rerun it. Sometimes, chromedriver download crashes for no reason.

## Run Selenium

A) Start the notebook and import the dependencies.

```juypter
# In [1]
## Run selenium and chrome driver to scrape data from cloudbytes.dev
import time
import os.path
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
```

B) Next, set the chrome options.

```jupyter
# In [2]
## Setup chrome options
chrome_options = Options()
chrome_options.add_argument("--headless") # Ensure GUI is off
chrome_options.add_argument("--no-sandbox")
```

C) Now, set the chromedriver and Chrome browser path. Make sure you change `cloudbytes` below to your username

```jupyter
# In [3]

# Set path to chromedriver as per your configuration
homedir = os.path.expanduser("~")
webdriver_service = Service(f"{homedir}/chromedriver/stable/chromedriver")

# Choose Chrome Browser
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
```

D) Fetch the page.

```jupyter
# In [4]
browser.get("https://cloudbytes.dev")
```

E) Print the description

```jupyter
# In [5]

description = browser.find_element(By.NAME, "description").get_attribute("content")
print(f"{description}")
```

D) Exit the browser

```jupyter
# In [6]
browser.quit()
```

---

# Running Jupyter Notebook from Terminal

URL: https://cloudbytes.dev/snippets/running-jupyter-notebook-from-terminal
Category: Snippets
Published: 2022-06-26
Author: Rehan Haider
Tags: python, jupyter, wsl2, linux

> How to install, and run Jupyter Notebook from Terminal

In this specific scenario, we're going to run Jupyter Notebook from WSL2, however, the method can be used in any other Linux environment.


## Install Jupyter Notebook

To do so, we will need to

1. [Update the packages to latest](#update-system-packages)
2. [Ensure Python is installed](#check-if-python-is-installed)
3. [Ensure pip is installed](#check-if-pip-is-installed)
4. [Install Jupyter Notebook & dependencies](#install-jupyter-notebook-dependencies)
5. [Run Jupyter Notebook](#run-jupyter-notebook)


### Update system packages

Run the following command in the terminal to update the system packages:

```bash
sudo apt update && sudo apt upgrade -y
```

### Check if Python is installed

Run the below to check the Python version:

```bash
python3 --version
```

If you get an error, then you need to install Python by running the following command:

```bash
sudo apt install python3 -y
```

### Check if pip is installed

Run the below to check the pip version:

```bash
python3 -m pip --version
```
If you get an error, then you need to install pip by running the following command:

```bash
sudo apt install python3-pip -y
```

If you already have pip installed, upgrade it to the latest version by running the following command:

```bash
python3 -m pip install --upgrade pip
```


### Install Jupyter Notebook & dependencies
Run the following command to install Jupyter Notebook:

```bash
python3 -m pip install jupyter
```

## Run Jupyter Notebook

Logout & login again and open the terminal.Run the following command to start Jupyter Notebook:

```bash
jupyter notebook
```

You should see a message similar to below:

![99999958-run-jupyter-notebook](/images/99999958-run-jupyter-notebook.png)

Copy one of two the URLs highlighted as shown above and open it in a browser window. 
This should start the Jupyter Notebook.

---

# Make Ubuntu Fullscreen on Windows Hyper-V

URL: https://cloudbytes.dev/snippets/make-ubuntu-fullscreen-on-windows-hyper-v
Category: Snippets
Published: 2022-03-12
Author: Rehan Haider
Tags: linux, windows

> How to run Ubuntu with full resolution in fullscreen mode on Windows Hyper-V

I previously explained how to install Ubuntu 20.04 in a VM on Windows using Hyper-V. However, by default the VM display will not run in full resolution. 

![99999959-ubuntu-hyperv-low-res](/images/99999959-ubuntu-hyperv-low-res.png)


## How to run Ubuntu 20.04 in full resolution & fullscreen mode

First, open the terminal and run the following command to open grub settings using nano editor.

```bash
sudo nano /etc/default/grub
```

Then, change the `GRUB_CMDLINE_LINUX_DEFAULT` variable to the following:

```bash
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash video=hyperv_fb:1920x1080"
```

![99999959-ubuntu-grub-update](/images/99999959-ubuntu-grub-update.png)


Press `Ctrl+X` to exit nano editor and then `Y` followed by `Enter` to save the changes.

Next, run the following command to update grub settings.

```bash
sudo update-grub
```

Finally, restart Ubuntu by running the following command.
```bash
sudo reboot
```

Now, you have Ubuntu running in full screen mode with full resolution.

![99999959-ubuntu-hyperv-fullscreen](/images/99999959-ubuntu-hyperv-fullscreen.png)

---

# Install Ubuntu in a VM on Windows using Hyper-V

URL: https://cloudbytes.dev/snippets/install-ubuntu-in-a-vm-on-windows-using-hyper-v
Category: Snippets
Published: 2022-02-23
Author: Rehan Haider
Tags: linux, windows

> Step by step guide to install Ubuntu 20.04 in a VM on Windows using Hyper-V

I wrote earlier about how to install [Ubuntu 20.04 using WSL2 on Windows]({filename}99999965-install-wsl2.md), and also how to [configure Hyper-V on Windows 10/11 Home edition]({filename}99999961-enable-hyperv-windows10-home.md). IN this post I will show you how to install Ubuntu 20.04 in a VM on Windows using Hyper-V.

You need to have Hyper-V enabled on your Windows 10/11, you can follow [this guide]({filename}99999961-enable-hyperv-windows10-home.md) to enable it.

## Download the Ubuntu ISO

Download the **Ubuntu 20.04** ISO from [here](https://ubuntu.com/download/desktop).

## Create a New VM

### Step 1: Open the Hyper-V Manager

Open the start menu and search for "**Hyper-V Manager**". Click on "**Open**", to start the **Hyper-V Manager**.

![open-hyperv-manager](/images/99999960-open-hyperv-manager.png)


### Step 2: Create a new VM
Next, in the **Hyper-V Manager**, click on "**New**" in "**Actions**" panel on right, then select "**Virtual Machine**". This will start the "**New Virtual Machine Wizard**"

![hyperv-new](/images/99999960-hyperv-new.png)

### Step 3: Configure the VM
1. **Before You Begin** - This is an informational panel, click on **Next** to continue.
2. **Specify Name and Location** - Enter a name for the VM. You can leave the location to default. Press **Next**.
3. **Specify Generation** - Select the generation of the VM, choose "**Generation 2**" and press **Next**.
4. **Assign Memory** - Select the amount of memory to assign to the VM. You can leave the default, I chose 2048 MB. Press **Next**.
5. **Configure Networking** - Click on **Connection** and choose **Default Switch**. Press **Next**.
6. **Connect Virtual Hard Disk** - You can leave the defaults. Press **Next**.
7. **Installation Options** - Select the "**Install an operating system from bootable image file**, browse to select the Ubuntu ISO you downloaded earlier, and press **Next**.
8. **Summary** - Click on **Finish** to finish configuration and create a VM.


Now you should see a new VM in the "Virtual Machines" list.

![virtual-machines-list](/images/99999960-virtual-machines-list.png)

### Configure Boot Options

Before you can start you Ubuntu VM, you need to configure UEFI settings. Right click on the VM you created and select "**Settings**".

In the settings menu that opens, click on **Security**. Then under **Secure Boot**, check the **Enable Secure Boot** checkbox then in **Template** dropdown, choose **Microsoft UEFI Certificate Authority**.

Now click on **Apply** and then on **OK** to close the settings menu.

### Start the VM

Now, right click on the VM you want to start, then click on **Connect**. In the dialog that opens, click on **Start**.

![ubuntu-demo-conn](/images/99999960-ubuntu-demo-conn.png)

If you see a boot menu, don't do anything it will skip in a few seconds. 

On the first boot, the Installation Wizard will guide you. Follow the steps to complete the Ubuntu Installation. 

## Configure Ubuntu Installation

Follow the below steps, these are recommended by me personally, but you can choose as per your convinience.

1. **Welcome** - Select the Language (English), then click on **Install Ubuntu**.
2. **Keyboard layout** - Select your keyboard layout and language and press **Continue**. I chose defaults (English US as both language and keyboard layout).
3. **Updates and other software** - Leave the defaults, press **Continue**.
4. **Installation type** - Leave the defaults, press **Install Now**. In the confirmation dialogue for **Write the changes to disks**, click on **Continue**.
5. **Where are you** - You can choose your timezone, usually the default is correct. Press **Continue**.
6. **Who are you** - You can choose your name, username, password. The computer name is generated automatically, but you can override and choose as per your need. Press **Continue**. 

This will start the installation process, it will take a few minutes to complete. After the installation is complete, you will get a prompt to restart the VM. Click on **Restart**. 

If restart process is stuck, go back to the Hyper-V Manager and right click on VM and click on **Turn off**. Then right click again, click on "**Connect**" and then click on "**Start**".

### Configure your Ubuntu Profile

On the first login a configuration will guide you through profile settings. 

1. **Connect Your Online Accounts** - Choose one of the accounts in the option or skip by pressing "**Skip**" on top right.
2. **Livepatch** - You can setup up Livepatch or skip by pressing "**Next**" on top right.
3. **Help improve Ubuntu** - I chose "**No, don't send system info** to disable telemetry. Click on **Next**.
4. **Privacy** - I chose to turn off Location Services. Click on **Next**.
5. **You're ready to go** - Choose any additional software you want to install. Click on **Done**.

This should complete the setup. 

![ubuntu-dektop-small](/images/99999960-ubuntu-dektop-small.png)

### [Bonus] Configure your Ubuntu run in fullscreen mode with full resolution

If you notice above, the resolution and aspect of the VM is not the same as the monitor. You will also not be able to change this in the display settings. 

To run the VM in fullscreen mode with full resolution, follow the instructions in this guide: [Make Ubuntu Fullscreen on Windows Hyper-V]({filename}99999959-ubuntu-hyperv-fullscreen.md).

---

# Enable Hyper-V on Windows 10/11 Home

URL: https://cloudbytes.dev/snippets/enable-hyper-v-on-windows-10-11-home
Category: Snippets
Published: 2022-02-22
Author: Rehan Haider
Tags: linux, windows

> Step by step guide to enable Hyper-V on Windows 10 Home or Windows 11 Home

Windows 10/11 Home edition doesn't come with Hyper-V by default, if you try to enable it from "Turn on Windows features" option from control panel you would not find Hyper-V listed.

This is because Hyper-V is a Professional and Enterprise edition feature, but it is [possible to enable it from the command line](https://docs.microsoft.com/en-us/answers/questions/29175/installation-of-hyper-v-on-windows-10-home.html).

![windows-features](/images/99999961-windows-features.png)

## Enable Hyper-V from command line

We're going to use the [Windows Terminal](https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701) so make sure you install it from the [Windows Store](https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701).

### Step 1. Check Minimum System Requirements

Your PC should support Hardware virtualisation for Hyper-V to work. 

If you do not have hardware virtualisation, you will need to use an alternative such as [VirtualBox](https://www.virtualbox.org/) or [VMware Workstation Player](https://www.vmware.com/products/workstation-player.html).

!!! note "Note: Many Windows 10 PCs—and all PCs that come preinstalled with Windows 11—already have virtualization enabled, so you may not need to follow these steps."

For Windows 10, run the followind command in the terminal
```powershell
Get-ComputerInfo -property "HyperV*"
```
You should get the below output

![windows 10 virtualisation](/images/99999961-virtualisation.png)

This means that your PC supports hardware virtualisation.

### Step 2. Enable Hyper-V

Create a file on your PC called "enable-hyperv.bat" and paste the following code in it.

```powershell
pushd "%~dp0"
dir /b %SystemRoot%\servicing\Packages\*Hyper-V*.mum >hyper-v.txt
for /f %%i in ('findstr /i . hyper-v.txt 2^>nul') do dism /online /norestart /add-package:"%SystemRoot%\servicing\Packages\%%i"
del hyper-v.txt
Dism /online /enable-feature /featurename:Microsoft-Hyper-V -All /LimitAccess /ALL
pause
```
Now run the batch file as an administrator, as shown below.
![enable-hyperv](/images/99999961-enable-hyperv.png)

This will go through several steps and will take some time to complete. Though it might seem it's repeating the same steps, let it complete without interruption.

Once the process is complete, you should see the following message asking for confirmation to reboot your PC. Press Y to reboot. 

![enable-hyperv-complete](/images/99999961-enable-hyperv-complete.png)

### Step 3. Check if Hyper-V is enabled

Open your teminal and run `optionalfeatures` to see the status of Windows features. You should be able to see a Hyper-V feature listed now. 

![hyperv-enabled](/images/99999961-hyperv-enabled.png)


## Starting Hyper-V Manager

Go to the start menu and search for Hyper-V, open the Hyper-V Manager. 

![open-hyperv-manager](/images/99999961-open-hyperv-manager.png)

From here you can start creating VMs, creating a new VM, or even creating a new VM template.

For a quick start, follow this [guide to installing Ubuntu on Hyper-V]({filename}99999960-install-ubuntu-on-windows.md)

---

# Run Chrome extensions with Python Selenium on AWS Lambda

URL: https://cloudbytes.dev/snippets/run-chrome-extensions-with-python-selenium-on-aws-lambda
Category: Snippets
Published: 2021-12-25
Author: Rehan Haider
Tags: python, aws, selenium

> A detailed guide to use Selenium and Chrome with extensions on AWS Lambda

!!! warning "UPDATE 29 March 2022: **As of this update, the instructions in this article may not work on AWS Lambda. Please refer to [this discussion on GitHub](https://github.com/CloudBytesDotDev/CloudBytes.dev/discussions/103) for more details and alternative options.**"

I earlier wrote about how to [run Chrome AWS Lambda using Python and Selenium webdriver]({filename}99999982-run-selenium-in-aws-lambda.md), but running Chrome with extensions is a different ball game. So let's unpack the problem first, and then we'll get to the solution. 

Chrome, when started in headless mode will start without browser UI, it is just a webpage viewport sans anything else. 

![chrome-comparison](/images/99999962-chrome-comparison.png)

> You can take a screenshot by running the command `google-chrome --headless --disable-gpu --screenshot https://cloudbytes.dev`


## Why your Chrome extensions are not working using Selenium in headless mode?

As demonstrated above, because your Chrome is running in headless mode, it will not have any UI and thus no extensions are loaded. This is not a bug, it is a feature as explained here on [chromium website](https://bugs.chromium.org/p/chromium/issues/detail?id=706008).

So to be able to load extensions, you need to run Chrome in non-headless mode. Which is problematic considering AWS Lambda doesn't have a display so you cannot really run Chrome GUI. 

**Or can you?**\
Yes you can, of course you can, I'll show you how.


## How to run Chrome with extensions in AWS Lambda

For this example, a reader asked to try to run [GoFullPage](https://chrome.google.com/webstore/detail/gofullpage-full-page-scre/fdpohaocaechififmbbbbbknoalclacl?hl=en) extension in AWS Lambda. This extension relies on user-interaction thus presents a complex problem.

Let's try and break this problem down.

1. Extensions do not work in Chrome headless mode, thus you need to run Chrome in non-headless mode, i.e. with a display
2. AWS Lambda doesn't have a display, so you need a virtual display to run Chrome GUI. We will use `Xvfb` with `pyvirtualdisplay` wrapper to do this
3. The extension relies on user-interaction, but, Selenium cannot be used for these interactions since it restricts user interaction to DOM elements and doesn't allow sending hotkeys to browser. Thus we will need to create a virtual keyboard to send keys to browser. In this case I chose to use `pyautogui`
4. PyAutoGUI is a Python wrapper around the `Xlib` library and relies on several linux packages that are NOT AVAILABLE on AWS Lambda's default image that uses Amazon Linux 2 (derivative of CentOS) . So we need to use Debian based image on AWS Lambda to run this example. I chose to use Python Buster image.

Now with that out of the way, let's get started.

## Setting up the development environment

**Step 1**: You need [VSCode](https://code.visualstudio.com/download), [Docker Desktop](https://www.docker.com/products/docker-desktop), and WSL2 as the development environment. You can find instructions on how to setup WSL2 [here]({filename}99999965-install-wsl2.md)


**Step 2**: Start the VScode editor

1. Start the terminal and login to WSL2 by running `wsl`
2. Make a new directory `mkdir selenium-aws` and cd into it `cd selenium-aws`
3. Launch the VS Code editor by running `code .`

![start-vscode-wsl2](/images/99999962-start-vscode-wsl2.gif)

**Step 3**: Reopen the folder in a devcontainer

1. While in VScode, press `Ctrl + Shift + P` to open command palette
2. Choose `Reopen in Container` from the drop down menu
3. Then click on `Show All Definitions`
4. Choose `Docker in Docker` from the drop down menu (Do not select `Docker from Docker`)
5. Leave the default selections and choose OK in the next two dialogues

![create-devcontainer](/images/99999962-create-devcontainer.gif)


Next, install the following:

**Step 4**: [Install AWS CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md#install-aws-cli)

**Step 5**: [Install SAM CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md#install-aws-sam-cli)

And finally, configure AWS CLI as per below

**Step 6**: [Configure AWS & AWS CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md#configuring-aws-aws-cli)

## Running Chrome Extensions using Selenium in AWS Lambda
Unlike a [previous guide]({filename}99999983-run-lambda-on-container-sam.md#create-a-new-app) we'll use a manual SAM templates to create a new Lambda app. 

Your folder structure should look like below

```
.
├── __init__.py
├── events
│   └── event.json
├── src
│   ├── __init__.py
│   ├── app.py
│   ├── Dockerfile
│   ├── GoFullPage.crx
│   ├── install_chrome.sh
│   ├── install_driver.sh
│   └── requirements.txt
├── samconfig.toml
└── template.yaml
```
### a) \_\_init\_\_.py
Both the `__init__.py` files should be empty

### b) events/event.json
We will use a basic event structure that will trigger our lambda. The contents should be
```json
{
    "body": "{\"message\": \"hello world\"}",
    "resource": "/{proxy+}",
    "path": "/path/to/resource",
    "httpMethod": "POST",
    "isBase64Encoded": false,
    "queryStringParameters": {
        "foo": "bar"
    },
    "pathParameters": {
        "proxy": "/path/to/resource"
    },
    "stageVariables": {
        "baz": "qux"
    },
    "headers": {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, sdch",
        "Accept-Language": "en-US,en;q=0.8",
        "Cache-Control": "max-age=0",
        "CloudFront-Forwarded-Proto": "https",
        "CloudFront-Is-Desktop-Viewer": "true",
        "CloudFront-Is-Mobile-Viewer": "false",
        "CloudFront-Is-SmartTV-Viewer": "false",
        "CloudFront-Is-Tablet-Viewer": "false",
        "CloudFront-Viewer-Country": "US",
        "Host": "1234567890.execute-api.us-east-1.amazonaws.com",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Custom User Agent String",
        "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)",
        "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==",
        "X-Forwarded-For": "127.0.0.1, 127.0.0.2",
        "X-Forwarded-Port": "443",
        "X-Forwarded-Proto": "https"
    },
    "requestContext": {
        "accountId": "123456789012",
        "resourceId": "123456",
        "stage": "prod",
        "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
        "requestTime": "09/Apr/2015:12:34:56 +0000",
        "requestTimeEpoch": 1428582896000,
        "identity": {
            "cognitoIdentityPoolId": null,
            "accountId": null,
            "cognitoIdentityId": null,
            "caller": null,
            "accessKey": null,
            "sourceIp": "127.0.0.1",
            "cognitoAuthenticationType": null,
            "cognitoAuthenticationProvider": null,
            "userArn": null,
            "userAgent": "Custom User Agent String",
            "user": null
        },
        "path": "/prod/path/to/resource",
        "resourcePath": "/{proxy+}",
        "httpMethod": "POST",
        "apiId": "1234567890",
        "protocol": "HTTP/1.1"
    }
}
```

### c) template.yaml

```yaml
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  python3.8
  Selenium on Lambda
Globals:
  Function:
    Timeout: 120

Resources:
  SeleniumFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      Events:
        Selenium:
          Type: Api 
          Properties:
            Path: /twitter
            Method: get
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./src
      DockerTag: python3.9-Selenium

Outputs:
  SeleniumApi:
    Description: "API Gateway endpoint URL for Prod stage for Selenium function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/selenium/"
  SeleniumFunction:
    Description: "Selenium Lambda Function ARN"
    Value: !GetAtt Selenium.Arn
```

### d) src/Dockerfile
Our Dockerfile needs to do the following

1. Start from the python:buster image
2. Install AWS Lambda dependencies to run the Lambda function on custom image
3. Install Lambda Runtime Interface Client to implement Lambda Runtime API
4. Copy the extension, app.py and requirements.txt to the Docker image
5. Install the python dependencies
6. Install Chrome Browser to auto install Chromium dependencies
7. Install latest Chromium Browser
8. Install latest Chromedriver
9. Install Xvfb and dependencies
10. Configure Lambda Runtime API to execute the Lambda function


```Dockerfile
# Define function directory
ARG FUNCTION_DIR="/function"
ARG RUNTIME_VERSION="3.9"


FROM ubuntu:latest as base-image

RUN apt-get update && DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC
RUN apt-get install -y g++ make cmake unzip libcurl4-openssl-dev
RUN apt-get install -y python3 python3-pip 
RUN apt-get install xvfb python3-tk python3-dev -y
RUN apt-get install curl wget -y

ARG FUNCTION_DIR
# Create function directory
RUN mkdir -p ${FUNCTION_DIR}
# Copy function code


RUN pip install \
    --target ${FUNCTION_DIR} \
    awslambdaric

# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Set working directory to function root directory
WORKDIR ${FUNCTION_DIR}

# Copy setup & other temporary files
COPY requirements.txt /tmp/
#COPY GoFullPage.crx /opt/
RUN pip install --upgrade pip -q
RUN pip install -r /tmp/requirements.txt -q
COPY install_chrome.sh /tmp/
RUN /bin/bash /tmp/install_chrome.sh
COPY install_driver.sh /tmp/
RUN /bin/bash /tmp/install_driver.sh
COPY install_chromium.sh /tmp/
RUN /bin/bash /tmp/install_chromium.sh
COPY app.py ${FUNCTION_DIR}

COPY GoFullPage.crx /opt/
RUN ls -al /opt/chrome/stable/

ENTRYPOINT [ "python3", "-m", "awslambdaric" ]
CMD [ "app.handler" ]
```

### e) src/GoFullPage.crx

[Go Full Page](https://chrome.google.com/webstore/detail/gofullpage-full-page-scre/fdpohaocaechififmbbbbbknoalclacl?hl=en) is the chrome extension that we will use in this demo. 

There are many ways to download Chrome extensions, in this case I recommend running the below command

```bash
curl -L "https://clients2.google.com/service/update2/crx?response=redirect&\
os=win&arch=x64&os_arch=x86_64&nacl_arch=x86-64&prod=chromiumcrx&\
prodchannel=beta&prodversion=79.0.3945.53&lang=ru&acceptformat=crx3\
&x=id%3Dfdpohaocaechififmbbbbbknoalclacl%26installsource%3Dondemand%26uc" -o GoFullPage.crx
```

### f) src/install_chrome.sh

Next we install Chrome browser

```bash
#!/bin/bash

apt-get update && apt-get upgrade -y

echo "Download the latest Chrome .deb file..."
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb -q

echo "Install Google Chrome..."
dpkg -i google-chrome-stable_current_amd64.deb

echo "Fix dependencies..."
apt-get --fix-broken install -y

```

Make sure you make this file executable by running the following command

```bash
chmod +x src/install_chrome.sh
```

### g) src/install_chromium.sh

Now with dependencies installed we can install Chromium browser

```bash
#!/bin/bash

echo "Downloading Chromium"
mkdir -p "/opt/chrome/stable"
curl -Lo "/opt/chrome/stable/chrome-linux.zip" \
    "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/\
    o/Linux_x64%2F954502%2Fchrome-linux.zip?generation=1640815524872726&alt=media"
unzip -q "/opt/chrome/stable/chrome-linux.zip" -d "/opt/chrome/stable/"
ls -al /opt/chrome/stable/chrome-linux
mv /opt/chrome/stable/chrome-linux/* /opt/chrome/stable/
rm -rf /opt/chrome/stable/chrome-linux /opt/chrome/stable/chrome-linux.zip
```

Make this file executable by running 

```bash
chmod +x src/install_chromium.sh
```

### h) src/install_driver.sh

Now we install a compatible chrome driver. The below script 

1. Gets the version of Chrome installed, 
2. Then gets the latest version of the chromedriver available, 
3. Compares if the versions are the same
4. Downloads the latest chromedriver if the version match
5. If not, it will exit with an error

If you have used the `install_chrome.sh` script to install Chrome, the versions should match. 

```bash
echo "Getting Chrome version..."
chrome_version=($(google-chrome-stable --version))
version=${chrome_version[2]}
chrome_version=${version%.*}
echo "Chrome version: ${chrome_version}"

echo "Getting latest chromedriver version"
chromedriver_version_full=$(curl "https://chromedriver.storage.googleapis.com/LATEST_RELEASE")
version=${chromedriver_version_full}
chromedriver_version=${version%.*}
echo "Chromedriver version: ${chromedriver_version}"

if [ "${chrome_version}" == "$chromedriver_version" ]; then
    echo "Compatible Chromedriver is available..."
    echo "Proceeding with installation..."
else
    echo "Compabible Chromedriver not available...exiting"
    exit 1
fi

echo "Downloading latest Chromedriver..."
mkdir -p "/opt/chromedriver/stable/"

 curl "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/\
      o/Linux_x64%2F954502%2Fchromedriver_linux64.zip?generation=1640815530134396&alt=media" \
  -H 'authority: www.googleapis.com' \
  -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'dnt: 1' \
  -H 'upgrade-insecure-requests: 1' \
  -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)\
       Chrome/96.0.4664.110 Safari/537.36" \
  -H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/\
      apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" \
  -H "x-client-data: CLO1yQEIhrbJAQiktskBCMG2yQEIqZ3KAQjRoMoBCMCXywEI6vLLAQie+csBCNf8ywEI5oTMAQi1\
      hcwBCMuJzAEI0IvMAQisjswBCJqPzAEI0o/MAQjakMwBCMmSzAEIoZPMAQjHk8wBCIqUzAEY5KDLAQ==" \
  -H 'sec-fetch-site: none' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-user: ?1' \
  -H 'sec-fetch-dest: document' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8,ms;q=0.7' \
  --compressed > /opt/chromedriver/stable/chromedriver_linux64.zip



unzip -q "/opt/chromedriver/stable/chromedriver_linux64.zip" \
    -d "/opt/chromedriver/stable/"

mv /opt/chromedriver/stable/chromedriver_linux64/chromedriver /opt/chromedriver/stable/chromedriver

chmod +x "/opt/chromedriver/stable/chromedriver"
rm -rf "/opt/chromedriver/stable/chromedriver_linux64.zip"

echo "Chrome & Chromedriver installed"
```

Again, make sure you make this file executable by running the following command

```bash
chmod +x src/install_driver.sh
```

### i) src/app.py

The app.py file needs model the following user behavior 
1. Open the browser with the extension installed
2. Open `www.example.com`
3. Close extension welcome page
4. Start screenshot capture by pressing `Shift + Alt + P`
5. Navigate to the screenshot page
6. Download the screenshot to the default downloads directory by clicking on download button
7. Close the browser
8. Upload the screenshot(s) to S3

![user-behaviour](/images/99999962-user-behaviour.gif)

We achive this by the following code. 

```python
# src/app.py
import time
import glob
import os
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from pyvirtualdisplay import Display
#from pyvirtualdisplay.smartdisplay import SmartDisplay



def handler(event=None, context=None):
    display = Display(visible=False, extra_args=[':25'], size=(2560, 1440), backend="xvfb") 
    display.start()
    print('Started Display')
    #Pyautogui requires os.environ["Display"] variable to be set. 
    import pyautogui

    chrome_options = Options()
    # Headless environment starts without browser UI so no extensions
    #chrome_options.add_argument("--headless") 
    chrome_options.binary_location = "/opt/chrome/stable/chrome"
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--disable-dev-tools")
    #chrome_options.add_argument("--no-zygote") #This will not load the extension
    #chrome_options.add_argument("--single-process") #Single process will break the app
    chrome_options.add_argument("window-size=2560x1440")
    chrome_options.add_argument("--remote-debugging-port=9222")
    chrome_options.add_argument("--user-data-dir=/tmp/chrome-user-data")
    chrome_options.add_extension("/opt/GoFullPage.crx")
    download_directory = {"download.default_directory": "/tmp/"}
    chrome_options.add_experimental_option("prefs", download_directory)
    webdriver_service = Service("/opt/chromedriver/stable/chromedriver")
    browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)

    browser.get("https://example.com")
    # Open Extension options
    print("Open Extension options...")
    browser.switch_to.window(browser.window_handles[1])
    browser.get("chrome-extension://fdpohaocaechififmbbbbbknoalclacl/options.html")

    # Provide Download Permission
    print("Provide Download Permission...")
    browser.find_element(By.ID, "perm-toggle").click()
    browser.find_element(By.NAME, "downloads").click()
    browser.switch_to.active_element
    time.sleep(1)
    pyautogui.press("tab")
    time.sleep(1)
    pyautogui.press("enter")

    # Close options
    print("Close options...")
    print(len(browser.window_handles)) #Expected 2
    browser.close()
    print(len(browser.window_handles)) #Expected 1
    time.sleep(1)
    # Take screenshot
    print("Take screenshot...")
    browser.switch_to.window(browser.window_handles[0])
    pyautogui.hotkey("shift", "alt", "p")
    time.sleep(5)
    print(len(browser.window_handles)) #Expected 2
    browser.switch_to.window(browser.window_handles[1])
    time.sleep(1)
    browser.find_element(By.ID, "btn-download").click()


    time.sleep(5)
    browser.quit()

    # importing earlier conflicts with selenium actions
    import boto3
    s3 = boto3.client("s3")
    BUCKET_NAME = "cloudbytes.dev" # replace with your bucket name

    for image in glob.iglob("/tmp/*.png"): 
        s3.upload_file(image, BUCKET_NAME, os.path.basename(image))

    return {"status":"success"}

```

Make sure you replace the `BUCKET_NAME` in the code with your bucket name. 


### j) src/requirements.txt
This will contain the python dependencies required for the Lambda function

```text
selenium
pyvirtualdisplay
pillow
keyboard
pyautogui
python-xlib
boto3
```

## Build & test Lambda app to run Chrome with extension

To build, just run the following command

```bash
sam build
```

This will result in a message similar to this (the build process typically takes a few minutes given your internet speed)

![sam-build-output](/images/99999962-sam-build-output.png)

Finally, to test the Lambda function, run the following command

```text
sam local invoke
```

This will run the Lambda function locally and display the following output

![sam-invoke-output](/images/99999962-sam-invoke-output.png)


### Check the results
Go to your AWS console and navigate to the S3 bucket that you chose in the step (i) above.

You should see the screenshot(s) that you uploaded to S3 for each test execution. 

![aws-s3-screenshots](/images/99999962-aws-s3-screenshots.png)


## Deploying to AWS Lambda

Deploying to AWS Lambda is as simple as running the below

```bash
sam deploy --guided
```

This will launch a guided deployment process, you can use the following:

```text
Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Found
        Reading default arguments  :  Success

        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [selaws]: 
        AWS Region [us-east-1]: 
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [y/N]: 
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: 
        #Preserves the state of previously provisioned resources when an operation fails
        Disable rollback [y/N]: 
        SeleniumFunction may not have authorization defined, Is this okay? [y/N]: y
        Save arguments to configuration file [Y/n]: 
        SAM configuration file [samconfig.toml]: 
        SAM configuration environment [default]: 
```

Now you can run the Lambda app from the AWS console. Alternatively, you can also run the Lambda function by calling the API we created by using the following command

```bash
curl https://<api-id>.execute-api.us-east-1.amazonaws.com/Prod/selenium/
```

You can get the API ID from deployment output of `sam deploy` as shown below:

![lambda-api-url](/images/99999962-lambda-api-url.png)


## Final Code

The above code is available on [GitHub in this repository](https://github.com/rehanhaider/selenium-aws-chrome-extension).

---

# How to install Node.js and NPM on WSL2

URL: https://cloudbytes.dev/snippets/how-to-install-node-js-and-npm-on-wsl2
Category: Snippets
Published: 2021-12-12
Author: Rehan Haider
Tags: wsl2, node

> A short guide to installing Node.js and NPM on WSL2

If you are using WSL2 / Ubuntu, you can try to install Node.js by running "sudo apt install nodejs", but this will not install the latest version of Node.js. You can the check the version of Node.js available in Ubuntu's default repository by running the below command:

```bash
apt list | grep nodejs
```
![apt-list-nodejs](/images/99999964-apt-list-nodejs.png)

This guide will walk you through the steps to install the latest version of Node.js on WSL2.

## Install NVM (Node Version Manager)

Install the latest version of NVM by running the following command:
```bash
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
```

>!!! note "NOTE: You can get the latest versionof the above command by vising the [NVM GitHub Repository](https://github.com/nvm-sh/nvm#installing-and-updating)"

Confirm your installation and check version of NVM by running the following command:
```bash
nvm --version
```
## Populate the NVM list

Fetch all versions available in NVM by running the following command:
```bash
nvm ls-remote
```

## Install Node.js & NPM

Now you can install the latest version of Node.js by running:
```bash
nvm install node
```
This will install both the latest version of Node.js and NPM.

`node` above is an alias for the latest version of Node.js. To install a specific version of Node.js, replace node by the version and run the command, e.g. to install ` v16.13.1` run:

```bash
nvm install v16.13.1
```

Now confirm the version of Node.js installed by running the following command:
```bash
node --version
```
You can also check the version of NPM by running the following command:
```bash
npm --version
```

---

# Which Python Implementation you should use (Cpython, PyPy, etc.)?

URL: https://cloudbytes.dev/snippets/which-python-implementation-you-should-use-cpython-pypy-etc
Category: Snippets
Published: 2021-12-12
Author: Rehan Haider
Tags: python

> A short introduction to the different Python implementations.

Python as you know it is not just a programming language. The Python that you download from the official [Python.org](https://www.python.org/) website is a **reference implementation**. What that means is it implements the Python language specifications, as defined by the [Python Software Foundation](https://www.python.org/dev/).

But Python.org's reference implementation is not the only Python implementation available. There are many other some with very specialised use cases and for beginners sometimes this can get confusing. So let's take a look at some key Python implementations available and when


## CPython
![python-logo](/images/99999963-python-logo.png)

[CPython](https://www.python.org/downloads/)The officiis Python implementation that is used by the Python Software Foundation. Written in C and Python, it is the most popular Python implementation and is used by the vast majority of Python developers. CPython is considered the most mature and "production-quality" Python implementation.

If you're starting out with Python, you should definitely start with CPython as you're least likely to encounter any issues with it.

## PyPy
![pypy-logo](/images/99999963-pypy-logo.png)

[PyPy](https://www.pypy.org/download.html) is a Python implementation written in Python (specifically RPython) and is a replacement for CPython. PyPy's main utility is that it is **really fast**, in fact, it claims to be [almost 5x faster than CPython](https://speed.pypy.org/).

However, PyPy can run most Python code except for when the code depends on CPython extensions which results in either inability to run or significant loss of performance.

PyPy is intended for advanced users who want to optimise their code for performance.

## Jython
![jython-logo](/images/99999963-jython-logo.png)

[Jython](https://www.jython.org/download/) is a Python implementation written in Python and Java and is designed to run on Java platforms. The key use case for Jython is its ability to import Java classes and that Jython compiles the Python code into Java bytecode which can be run on Java Virtual Machines (JVM).

The typical use case of Jython is when Java classes are needed to be imported, e.g. you could build an Android app using a mix of Jython which can import Java Android packages and a toolkit like Kivy. 

## CircuitPython
![circuitpython-logo](/images/99999963-circuitpython-logo.png)

[CircuitPython](https://circuitpython.org/) is maintained by [Adafruit](https://www.adafruit.com/) and is designed to run on certain microcontroller hardware such as the [Adafruit Feather M0](https://www.adafruit.com/product/3317) and [Adafruit Feather M4](https://www.adafruit.com/product/3316). Is is written in C and is not exlusive to Adafruit Microcontrollers and can be used for other supported microcontroller hardware as well. 

## Other notable Python implementations
1. [Numba](http://numba.pydata.org/) is a NumPy aware JIT compiler that can compile a subset of Python code into machine code for faster execution
2. [Pyston](https://www.pyston.org/) is a relatively new alternate Python implementation designed to be a drop-in replacement for CPython and optimised for performance with claimed improvements of 30% in speed
3. [RPython](https://rpython.readthedocs.io/en/latest/) is a restricted version of Python is a subset of CPython and is designed to be a framework for creating dynamic languages

## Conclusion
For most purposes, **CPython** is the implementation you should be using, unless you have a specific reason to use another implementation as described above.

---

# How to install WSL2 on Windows 10/11

URL: https://cloudbytes.dev/snippets/how-to-install-wsl2-on-windows-10-11
Category: Snippets
Published: 2021-11-26
Author: Rehan Haider
Tags: wsl2, windows

> A guide to install WSL2 on Windows 10/11

[TOC]

The process of installing WSL2 can differ slightly depending upon the version of Windows you are using, so choose the [easy way](#installing-wsl2-on-windows-1011-the-easy-way) below if you're running the latest updates of Windows 10/11, otherwise, follow [these instructions](#installing-wsl2-on-windows-1011-the-hard-way) below to install WSL2 manually. 

To install WSL2, you must be running the following versions of Windows 10/11

* For x64 systems: Version 1903 or higher, with Build 18362 or higher
* For ARM64 systems: Version 2004 or higher, with Build 19041 or higher.
* Builds lower than 18362 do not support WSL 2. You will need to update your version of windows. 

## Installing WSL2 on Windows 10/11 (The easy way)
If you're running Windows 10 version 2004 or higher (Build 19041 and above) or windows 11, installation is as simple as running the below command.

```powershell
wsl --install
```
This will take care of all the steps required, i.e.

1. Enable Windows Virtualisation Layer and WSL2
2. Update the Linux kernel to the latest version
3. Install the default Linux distribution, i.e. latest Ubuntu (Currently Ubuntu 20.04)

![Install WSL](/images/99999965-install-wsl.gif)

Then type `wsl` in your terminal and press enter to login to WSL2. 


## Installing WSL2 on Windows 10/11 (The hard way)
If you're running Windows 10 version 1903 or lower (Build 18362 and below), you will need to install WSL2 manually.


### Step 1: Enable Windows Subsystem for Linux (WSL)

Open the Windows Terminal or Powershell, and type the following command to enable WSL:

```powershell
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
```

### Step 2: Enable Windows Virtual Machine Platform

In the Windows Terminal or Powershell, type the following command to enable Windows Virtual Machine Platform:
```powershell
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart
```

### Step 3: Update the Linux kernel to the latest version

This requires you to download the **WSL2 Linux kernel update MSI package**, choose the appropriate version from below, and install it.
1. [WSL2 Linux kernel update MSI package for x64 systems](https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi)
2. [WSL2 Linux kernel update MSI package for ARM64 systems](https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_arm64.msi)


### Step 4: Set WSL2 as the default version
Installing WSL2 in Step #3 doesn't change the default version of WSL from 1 to 2. To change the default version of WSL, you must run the following command in terminal or powershell:

```powershell
wsl --set-default-version 2
```

### Step 6: Install your preferred Linux distribution

You can choose from 10+ distributions available on Microsoft Store. To install, open Microsoft Store, and search for the Linux distribution you want to install then click on the **Get** button.

---

# WSL2: Find and Delete Zone.Identifier files

URL: https://cloudbytes.dev/snippets/wsl2-find-and-delete-zone-identifier-files
Category: Snippets
Published: 2021-11-10
Author: Rehan Haider
Tags: linux, wsl2, ubuntu, windows

> Code snippet to find and delete Zone.Identifier files that gets auto-generated while copying files to WSL2

**TL;DR:**: Run the below code snippet to find and delete Zone.Identifier files that gets auto-generated while copying files to WSL2

```bash
find . -name "*:Zone.Identifier" -type f -delete
```

If you ended up here chances are these pesky `*:Zone.Identifier` files have broken something in your workflow. Otherwise they are harmless files that are generated while downloading a file by browsers & Windows explorer to store metadata about the file being downloaded. 

The technical details are unnecessary for this post and most use cases, but suffice to say its is a NTFS feature and just identifies the course of the file by using one of the [preidentified Security Zones that are defined by Microsoft](https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/ms537183(v=vs.85)?redirectedfrom=MSDN). 

But because the file name contains a `:` colon, which actually is not a valid character in a file name, it can break certain automated workflows and scripts. For most cases, specially on WSL / Linux system you can just delete the file without any thought. 

To do so, just run the following command in the root of the directory where you want to search and delete these files, .e.g. 

```bash
cd "~/Downloads && find . -name "*:Zone.Identifier" -type f -delete"
```

## Microsoft says they have fixed it, but users think otherwise. 

Issues [#4609](https://github.com/microsoft/WSL/issues/4609) and [#7456](https://github.com/microsoft/WSL/issues/7456) on the official WSL repository provides more details, however, this issue is definitely not fixed in Windows 11.

---

# Configure logging in AWS Lambda to CloudWatch using Python

URL: https://cloudbytes.dev/snippets/configure-logging-in-aws-lambda-to-cloudwatch-using-python
Category: Snippets
Published: 2021-11-06
Author: Rehan Haider
Tags: aws, python

> How to log events to CloudWatch during AWS Lambda execution using Python

Logging events to CloudWatch log stream during execution is as simple as normal logging. You might be tempted to use `print`, and it may work, if it's not a good practice. 

Rather you should be using the `logging` module. 

Below in a rather simple example, I write a Lambda function that logs a "Hello, today is <today's date\>!" message to CloudWatch log stream. 

## Getting Started
You should follow the steps in this [SAM guide to setup a basic Lambda function]({filename}99999984-deploy-serverless-apps-with-aws-sam.md) to setup a Lambda function.


## Writing a Lambda function to log to CloudWatch
Edit the Lambda function under `hellow_world/app.py`, and change the code to the following

```python
import logging
from datetime import datetime

logging.basicConfig()
logger = logging.getLogger("HELLO")
logger.setLevel(logging.INFO)

date_today = datetime.today().strftime("%Y-%m-%d")

def lambda_handler(event, context):
    logger.info(f"Hello, today is {date_today}!")
```

Then execute this by following the instructions here [instructions here]({filename}99999984-deploy-serverless-apps-with-aws-sam.md##test-the-app).

Now you can go to the Lambda function's CloudWatch log stream and see the message being printed everytime you execute the funcion.

---

# Launching Pelican Algolia plugin for Pelican

URL: https://cloudbytes.dev/snippets/launching-pelican-algolia-plugin-for-pelican
Category: Snippets
Published: 2021-11-05
Author: Rehan Haider
Tags: python, pelican, algolia

> Launching Pelican Algolia plugin, an open source software published on PyPi, that can help integrate Algolia Search in your Pelican blog.

Pelican's preferred search tool, Tipue Search, is defunct. It is no longer maintained and the website has been shut down. 

So I decided to write a plugin for Pelican to integrate Algolia Search, and release it as a opensource software. 

The Github repository is available here: [pelican-algolia](https://github.com/rehanhaider/pelican-algolia)
The PyPi repository is available here: [pelican-algolia](https://pypi.org/project/pelican-algolia/)


## Installation of Pelican Algolia plugin
Installation is easy. Just install the plugin from PyPi.

```bash
pip install pelican-algolia
```

## Usage of Pelican Algolia plugin

Detailed instructions are available on the [GitHub repository](https://github.com/rehanhaider/pelican-algolia), however in summary you need to set the following in your pelican configuration file (`pelicanconf.py` or `publishconf.py` depending on your usage):

```python
# Algolia Publish Data

# Admin key is sensitive so fetching it from environment variable is recommended
import os
ALGOLIA_ADMIN_API_KEY = os.environ.get("ALGOLIA_ADMIN_API_KEY")


ALGOLIA_APP_ID = "<Your Algolia App ID>"
ALGOLIA_SEARCH_API_KEY = "<Your Search-only Api Key>"
ALGOLIA_INDEX_NAME = "<You Algolia App Index name>"
```

### What gets uploaded to Algolia

For every article that gets published, the following record gets uploaded to Algolia:

1. **Slug**: The slug of the article, e.g. the slug of this article is `launching-pelican-algolia-plugin-for-pelican` that can be seen in the URL of this article.
2. **Title**: Title of the article
3. **URL**: Full URL of the article, this is needed to ensure the results have correct links.
4. **tags**: Tags assigned to an article
5. **category**: Category assigned to an article
6. **content**: Full content of your article

With slug used as the primary key to avoid duplication.

## Creating a search box
To make a working search box, you need to do two things

1. Create a [HTML Search box](#1-html-search-box)
2. Include the [Javascript code](#2-javascript-code) in your HTML page


### 1. HTML Search box
Here's what I have on [CloudBytes/dev>](https://cloudbytes.dev)

```html
    <div class="algolia-hits">
        <div class="algolia-form-wrap">
            <input placeholder="Search for bytes ..." type="text" id="algolia-input" class="algolia-input"
                autocomplete="off" />
            <span id="search-close" class="algolia-close btn btn-red">X</span>
        </div>
    </div>
```
Note the `id=algolia-input` in the `<input>` tag above, this needs to be added to the JavaScript code below.

### 2. Javascript code
Place this right at the bottom of you HTML just before closing </html> tag.
Replace the `autocomplete('#algolia-input', ... ) with the ID of input field shown above.

The code below includes JINJA syntax so it needs to be added to your HTML templates to ensure they are correctly replaced during generation.

```javascript
<!-- Include AlgoliaSearch JS Client and autocomplete.js library -->
<script src="https://cdn.jsdelivr.net/algoliasearch/3/algoliasearch.min.js"></script>
<script src="https://cdn.jsdelivr.net/autocomplete.js/0/autocomplete.min.js"></script>
<!-- Initialize autocomplete menu -->
<script>
    var client = algoliasearch("{{ALGOLIA_APP_ID}}", "{{ALGOLIA_SEARCH_API_KEY}}");
    var index = client.initIndex('{{ALGOLIA_INDEX_NAME}}');
    //initialize autocomplete on search input (ID selector must match)
    autocomplete('#algolia-input',
        { hint: false }, {
        source: autocomplete.sources.hits(index, { hitsPerPage: 10 }),
        //value to be displayed in input control after user's suggestion selection
        displayKey: 'title',
        //hash of templates used when rendering dataset
        templates: {
            //'suggestion' templating function used to render a single suggestion
            suggestion: function (suggestion) {
                return '<a class="algolia-hit" href="/' + suggestion.url + '">' +
                    '<h4>' +
                    '<span class="hit-title">' + suggestion._highlightResult.title.value + '</span>' +
                    '</h4>' +
                    '</a>'
            }
        }
    });
</script>
```

## What next?

On the horizon is the ability to choose which records gets published. For now, the plugin will publish all records.

---

# Limit article tags to allowed list in Pelican SSG

URL: https://cloudbytes.dev/snippets/limit-article-tags-to-allowed-list-in-pelican-ssg
Category: Snippets
Published: 2021-11-04
Author: Rehan Haider
Tags: python, pelican

> How to limit article tags to only an allowed list in Pelican SSG and not end up with too many tags.

Pelican makes it easy to create tags for your articles, where you can simply define them as part of the article's metadata.

E.g., this posts has two tags: "python" and "pelican" which are defined in the posts's markdown metadata.

```markdown
Title: Limit tags and categories to allowed list in Pelican SSG
Date: 2021-11-04
Category: Snippets
Tags: python, pelican
Author: Rehan Haider
```

But this also means you can end up with too many tags they can get out of control. 

I try to provide a simple way to limit the tags to a list of allowed tags below. 

## How to limit tags in Pelican

First, create a variables in `pelicanconf.py` file (the Pelican configuration file). This will contain the list of allowed tags, e.g. below:

```python
# Add this to you pelicanconf.py file

ALLOWED_TAGS = ('python', 'pelican','firebase','aws')
```

Once these are defined, they can be called be used directly in your SSG theme templates for the Pelican website, e.g. in the `article.html` template to limit the categories to only the ones from the list above, we can create a small JINJA snippet to do this:

```html
<div class="article-tags">
    {% for tag in article.tags %}
    {% if tag in ALLOWED_TAGS %}
    <a href="/tags/{{tag}}"><{{tag}}</span></a>
    {% endif %}
    {% endfor %}
</div>
```
This will ensure that only the tags from the list defined in `pelicanconf.py` are displayed, and the rest are suppressed.

### Why not limit the tags at source?
Two reasons, the first is that Pelican's content is written in markdown, which is a plain text format so one cannot stop someone from adding a tag that wasn't approved in the article metadata. 

The second is, you can potentially stop Pelican from processing a file if it contains a tag that isn't approved listed in the `pelicanconf.py` file, but this just seems a bit messy specially if you have multple authors considering not all of them will be well versed with Pelican's internals and also the fact that this adds an additional layer of processing which can potentially slow down the generation of the website.

But the main reason I wouldn't limit it at generation is because pelican's internals are a bit fragile and I've managed to break it in a few ways, so I'm not sure if it's worth the risk.


If you have questions / suggestions feel free to create a new discussion on [CloudBytes/dev> Github discussions](https://github.com/CloudBytesDotDev/CloudBytes.dev/discussions)

---

# Find and test invalid links and 404 errors using Python & Pytest

URL: https://cloudbytes.dev/snippets/find-and-test-invalid-links-and-404-errors-using-python-pytest
Category: Snippets
Published: 2021-11-01
Author: Rehan Haider
Tags: python

> A short guide to scrape your own website, find invalid links, and highlight links that result in 404 errors

TL;DR: Go to the [solution](#the-solution-workflow-to-validate-the-links) directly.

So, I recently managed to break something on [CloudBytes/dev>](https://cloudbytes.dev). All the internal links on the site were broken, and I published the website. 

I only noticed the error when the number of 404 erros increased significantly in the analytics report. So I set about creating a Python script to find the broken links and highlight them during the [Continous Integration process I have setup]({filename}99999971-building-cicd-pipelines-with-github-actions.md). 

## The Solution Workflow to validate the links

I came up with the following set of steps to first populate the links and then validate if they exist. 

To begin with: 

1. Fetch the sitemap.xml file from the website & create a list of all links on the website
2. For each link in the list, check if it exists on the website and fetch the webpage
3. Find all the links in the webpage
4. Then request the webpage for each link and check if it exists
5. If the page exists, then add it to the list of valid links


## Pytest Program to Scrawl & Test Website

```python
import pytest
import requests
from bs4 import BeautifulSoup

BASE_URL = "http://localhost:8080"
SITE_URL = "https://cloudbytes.dev"

def get_sitemap_links():
    """
    This function gets all links from the sitemap
    """
    sitemap_url = BASE_URL + "/sitemap.xml"
    sitemap_response = requests.get(sitemap_url)
    sitemap_soup = BeautifulSoup(sitemap_response.text, "lxml")
    sitemap_links = sitemap_soup.find_all("loc")

    sitemap_urls = []
    for link in sitemap_links:
        url = link.text.replace(SITE_URL, BASE_URL)
        sitemap_urls.append(url)

    return sitemap_urls


def get_page_links(url):
    """
    This function gets all links from a page
    """
    page_response = requests.get(url)
    page_soup = BeautifulSoup(page_response.text, "html5lib")
    page_links = page_soup.find_all("a")

    page_urls = []
    for link in page_links:
        url = link.get("href")
        if url is not None:
            if url.startswith("/"):
                url = BASE_URL + url
            elif url.startswith(BASE_URL):
                page_urls.append(url)
            else:
                pass

    return page_urls


def test_internal_links():
    """
    This function tests all internal links in the URLs on the sitemap
    """
    sitemap_urls = get_sitemap_links()
    valid_urls = []
    for url in sitemap_urls:
        page_urls = get_page_links(url)
        for page_url in page_urls:
            if page_url not in valid_urls:
                response = requests.get(page_url)
                assert response.status_code == 200
                valid_urls.append(page_url)
```

### Explanation

**Step 0:** Set the Site URL and Base URL

I set BASE_URL to localhost and SITE_URL to cloudbytes.dev. The reason for doing both is that in the CI process I use the localhost server to run the tests, but you can use the same program above to test a live website with minor changes.

```python
BASE_URL = "http://localhost:8080"
SITE_URL = "https://cloudbytes.dev"
```

**Step 1:** Fetch the sitemap.xml file from the website & create a list of all links on the website

```python
def get_sitemap_links():
    """
    This function gets all links from the sitemap
    """
    sitemap_url = BASE_URL + "/sitemap.xml"
    sitemap_response = requests.get(sitemap_url)
    sitemap_soup = BeautifulSoup(sitemap_response.text, "lxml")
    sitemap_links = sitemap_soup.find_all("loc")

    sitemap_urls = []
    for link in sitemap_links:
        url = link.text.replace(SITE_URL, BASE_URL)
        sitemap_urls.append(url)

    return sitemap_urls
```

I also create a function to get all the links from a webpage passed as an argument
```python
def get_page_links(url):
    """
    This function gets all links from a page
    """

    page_response = requests.get(url)
    page_soup = BeautifulSoup(page_response.text, "html5lib")
    page_links = page_soup.find_all("a")

    page_urls = []
    for link in page_links:
        url = link.get("href")
        if url is not None:
            if url.startswith("/"):
                url = BASE_URL + url
            elif url.startswith(BASE_URL):
                page_urls.append(url)
            else:
                pass

    return page_urls
```

**Step 2** For each link in the list, check if it exists on the website and fetch the webpage

We do this in the `test_internal_links` function, where we get all the links in the sitemap

```python
    sitemap_urls = get_sitemap_links()
```

**Step 3** Find all the links in the webpage
```python
    for url in sitemap_urls:
        page_urls = get_page_links(url)
```

**Step 4** Then request the webpage for each link and check if it exists
```python
        for page_url in page_urls:
            if page_url not in valid_urls:
                response = requests.get(page_url)
```

**Step 5** If the page exists, then add it to the list of valid links
```python
                assert response.status_code == 200
                valid_urls.append(page_url)
```

Finally, run this script by running the following command (You need to have `pytest` installed):

```bash
pytest
```

And this will scrape through the entire website and check if all the internal links are valid.

---

# Build CI/CD pipelines using artifacts with GitHub Actions

URL: https://cloudbytes.dev/snippets/build-ci-cd-pipelines-using-artifacts-with-github-actions
Category: Snippets
Published: 2021-10-30
Author: Rehan Haider
Tags: github, pelican, python

> Create a CI/CD pipeline using GitHub Actions to build, test, and deploy you website

This guide will differ from other guides about GitHub Actions in that it will be focused on how to transfer the build artifacts & outputs across multiple steps. 

If you know what you're doing, you can skip the introduction and go straight to the [Building CI/CD pipelines with GitHub Actions]({filename}99999971-building-cicd-pipelines-with-github-actions.md#using-GitHub-artifacts) guide.


## What is a CI/CD pipeline?
CI/CD pipeline is a sequence of steps that are executed to build, test, and deploy your website. It stands for Continuous Integration/Continuous Deployment.

Typically CI/CD pipelines are built by DevOps professionals using variety of tools such as GitHub Action, AWS CodePipeline, Jenkins, and others.

![CI/CD pipeline](/images/99999971-ci-cd-workflow.png)

The above figure shows a typical CI/CD pipeline but depending on use case there could be fewer or more steps. 

CI/CD pipelines makes it easier for a developer to build, test, and deploy their website or app by automating the steps associated with the deployement, e.g., in the above workflow, as soon as the developer pushes their code to the repository, the pipeline starts executing the steps associated with the build, test, and deployement.


### What is a GitHub Action?
[GitHub Action](https://github.com/features/actions) is GitHub's native CI/CD workflow management tool. It uses YAML files to declaratively define the steps that needs to be executed. 

Like other tools, GitHub Actions provides an extensive set of tools and options to build CI/CD pipelines. 

## Deploying a Static Website to firebase
[CloudBytes/dev>](https://cloudbytes.dev) is built on [JAMStack]({filename}99999996-what-is-jamstack.md) architecture and uses GitHub Actions to deploy the static website generated by Pelican to Firebase Hosting. 

The simplest way to do that is by using the below GitHub Workflow. 

```yaml
# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches:
      - main

jobs:
  build_and_deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: rehanhaider/pelican-build-action@v0.1.10
        env:
          PELICAN_CONFIG_FILE: app/publishconf.py
          PELICAN_CONTENT_FOLDER: app/content
      - uses: FirebaseExtended/action-hosting-deploy@v0
        with:
          repoToken: "${{ secrets.GITHUB_TOKEN }}"
          firebaseServiceAccount: "${{ secrets.FIREBASE_SERVICE_ACCOUNT_CLOUDBYTES_PROD }}"
          channelId: live
          projectId: cloudbytes-prod
```

The above workflow is execetud `on` the `push` event of the `main` branch. It has only one job that does both build and deployment of the website. However, this will present a problem if we wanted to introduce additional jobs in the pipeline because each job runs on a separate container. So if we wanted to split build and deployment into two jobs, we would need to repeat the build process twice because the first job would  build the website but the second job will not have access to the output. 

So solve this, you need to use a GitHub Actions feature called [GitHub Actions Artifacts]({filename}99999971-building-cicd-pipelines-with-github-actions.md#using-GitHub-artifacts) to transfer the build artifacts from one job to another.

## Using GitHub Artifacts
GitHub Artifacts can be used to transfer the build outputs and artifacts between two jobs. To do that you need to 
1. Upload you build artifacts that includes the "output" and configuration files
2. Then add the build step as a dependency on the next job
3. And finally download the artifact

The above workflow can be modified to the following:
```yaml
name: Deployment

on:
  push:
    branches:
      - main

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Build Pelican Website
        uses: rehanhaider/pelican-build-action@v0.1.10
        env:
          PELICAN_CONFIG_FILE: app/publishconf.py
          PELICAN_CONTENT_FOLDER: app/content
      - name: Upload the build output
        uses: actions/upload-artifact@v2
        with:
          name: build-output
          path: |
            output/
            .firebaserc
            firebase.json
          retention-days: 1
  deploy:
    name: Deploy
    needs: [build]
    runs-on: ubuntu-latest
    steps:
      - name: Download the build output
        uses: actions/download-artifact@v2
        with:
          name: build-output
      - name: Deploy to Firebase
        uses: FirebaseExtended/action-hosting-deploy@v0
        with:
          repoToken: "${{ secrets.GITHUB_TOKEN }}"
          firebaseServiceAccount: "${{ secrets.FIREBASE_SERVICE_ACCOUNT_CLOUDBYTES_PROD }}"
          channelId: live
          projectId: cloudbytes-prod
```
In the above example, we used `upload-artifact@v2` action, named it `build-output` and added the files & directories under `path`. For pelican we needed the output folder, in this case `output`, and the two Firebase configuration files. 

Then during the deployment step, we used `download-artifact@v2` action to download the `build-output` and subsequently deployed the result to Firebase using `FirebaseExtended/action-hosting-deploy@v0` action.

Once the above workflow is defined, any code that is pushed to the `main` branch will trigger a GitHub Action and the workflow will look something like this:

![Github Action Output](/images/99999971-github-action-output.png)


### Benefits of using GitHub Action artifacts
The biggest benefit of using GitHub Actions artifacts is that it allows you to transfer the build artifacts from one job to another. This is useful when you want to split the piepline into several steps and add or remove steps in future. E.g. in the above workflow, we can add a new job called `test` that runs the tests without having to change the existing steps in the workflow.

---

# Create a Pelican plugin to minify HTML, CSS, and JS

URL: https://cloudbytes.dev/snippets/pelican-plugin-to-minify-html-css-and-js
Category: Snippets
Published: 2021-10-26
Author: Rehan Haider
Tags: pelican, python

> Write a Pelican plugin to minify HTML, CSS, and JS files without breaking your website

TL;DR - [Minify you Pelican Website](#)

[Minification](https://developers.google.com/speed/docs/insights/MinifyResources) of wesite resources is an essential step for good PageSpeed / Lighthouse scores, and as I've bragged in past, [CloudBytes/dev>](https://cloudbytes.dev) scores a perfect 100. 

![CloudBytes.dev PageSpeed Score](/images/99999972-PageSpeed-Score-(Small).png)

This is partly because Website generated by [SSGs]({filename}99999996-what-is-jamstack.md) are generally faster than the ones generated using CMS solutions like WordPress, etc. But the main reason is non-redundant JS & CSS code. 

To make websites faster still, you minify the contents of all the HTML, CSS & JS files which typically results in smaller file sizes that consequently means faster load times for users. 

![Minify Reduction](/images/99999972-minify_reduction.png)

## Minify your Pelican Website

There are a lot of options in Python to minify your webassets, but none of them work. Either they break your website (e.g. remove embedded JS code) or are outdated and not maintained. I personally tried the below in dev environment:

1. ❌ [Unofficial Pelican Plugin: css-html-js-minify](https://github.com/getpelican/pelican-plugins/tree/master/css-html-js-minify): Broke the website 💔, removed embedded JS, deleted some CSS variables & class definitions.
2. ❌[Official Pelican Plugin](https://github.com/pelican-plugins/webassets): Extremely promising, is useless, and doesn't work. Becase it doesn't minify HTML pages, and it relies on additional 3rd party modules for minification and those modules are either unmaintained ([cssmin](https://github.com/zacharyvoase/cssmin),css_yui, etc) or simply doesn't work and breaks the site ([cssutils](http://cthedot.de/cssutils/)), or requires NPM/NodeJS modules that needs to be installed manually ([cleancss](https://github.com/clean-css/clean-css))
3. ❌Non-Python packages such as [html-minifier](https://github.com/kangax/html-minifier), [minimize](https://github.com/Swaagie/minimize), etc. These could have been used but the hassle of integrating them into the CI/CD workflow was too much
4. ✅ [minify-html](https://pypi.org/project/minify-html/): This seemed to work without breaking the website. [Minify HTML](https://github.com/wilsonzlin/minify-html) is a  Rust app but with APIs available in several languages (Python, Ruby, NodeJS, Java, etc) which made it easy to write a simple plugin.


## Writing the Minification Plugin

First step is to install the minify-html package. Run the following command
```bash
pip install minify_html
```



Then create a plugin, using the below code
```python
import minify_html
import glob
import os
import sys
import logging

from pelican import signals

logger = logging.getLogger()


def main(pelican):
    for file in glob.iglob(pelican.output_path + "/**/*.html", recursive=True):
        print(f"Processing {file}")
        try:
            with open(file, "r", encoding="utf-8") as html:
                minified = minify_html.minify(html.read(), do_not_minify_doctype=True)
            with open(file, "w", encoding="utf-8") as html:
                html.write(minified)
        except Exception as error:
            logging.error(error)


def register():
    signals.finalized.connect(main)
```

Add it to the plugins list in `pelicanconf.py` and you're good to go with a website that is almost 20-30% faster

---

# Firebase Hosting Redirects using Wildcard

URL: https://cloudbytes.dev/snippets/firebase-hosting-redirects-using-wildcard
Category: Snippets
Published: 2021-10-24
Author: Rehan Haider
Tags: firebase, pelican, python

> Configure redirects on Firebase Hosting using wildcards for groups of URLs

[Firebase Hosting](https://firebase.google.com/docs/hosting) is a popular choice for hosting [Jamstack]({filename}99999996-what-is-jamstack.md) websites, mostly because it is free but also because it is [developer friendly]({filename}99999992-automate-pelican-firebase-hosting.md) and has almost all features that you might want.


One of the most vital features Firebase Hosting has is the ability to do **URL Redirects** including 301 redirects. 

If you have a firebase project, you can configure Firebase Hosting following [these steps]({filename}99999992-automate-pelican-firebase-hosting.md#2-create-configure-the-firebase-project).

Then, if you want to redirect a URL `www.example.com/articles/my-brilliant-article` to `www.example.com/post/my-brilliant-post`  you just need to add a directive in `firebase.json` file that was created as part of the above configuration

```json
{
  "hosting": {
    "public": "output",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ],
    "redirects": [
        {
            "source": "/articles/my-brilliant-article",
            "destination": "/posts/my-brilliant-post",
            "type": 301
        }
    ]
}
```

It's as simple as that. 

But what if you wanted to rename your article category to post, i.e. everything that was `www.example.com/article/some-url` now has to become `www.example.com/post/same-url`. 

You might be tempted to use regex or **wilcard* but it won't work because regex helps in selection of the URLs not naming what it will be in future, essentially what you want to do is capture the `some-url` in the above example and use it as `same-url` in the redirect. 

# Using Variables in Firebase Hosting Redirects
You can define a variable as part of the redirect url by appending it with colon (:), e.g. `:path`. 

So the above example would become,
```json
{
  "hosting": {
    "public": "output",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ],
    "redirects": [
        {
            "source": "/articles/:path",
            "destination": "/posts/:path",
            "type": 301
        }
    ]
}
```

This will redirect all URLs that match the source pattern starting with `/articles/` to `/posts` while keeping the path of the file same. 

If you were looking for some other solution, feel free to create a new discussion from the link below.

---

# Develop remotely on Raspberry Pi using VSCode Remote SSH

URL: https://cloudbytes.dev/snippets/develop-remotely-on-raspberry-pi-using-vscode-remote-ssh
Category: Snippets
Published: 2021-10-16
Author: Rehan Haider
Tags: python, raspberrypi

> How to connect and develop remotely on Raspberry PI by SSHing in using VSCode

So you got yourself a Raspberry Pi and want to develop something on it. 

Raspberry Pi is of course lightweight, and consumes very little power but that also makes it quite unsuitable for developing anything small apps and running lightweight code editors such as Geany, Thonny etc. 

![Raspberry Pi](/images/99999974-raspberry-pi.png)

You could use VIM, but then you need to login to your Raspberry Pi using either VNC Viewer, or SSH into Raspberry Pi using some form of terminal. But for a VSCode fan such as me, that's not an acceptable option specially because I don't want to reconfigure all of my preferences. 

So the compromise is, you can simply use VSCode on your PC and use VSCOde to SSH in to the Raspberry PI and develop remotely. 

## Add Raspberry Pi as recognised hosts

Fire up your VSCode and first search and install the Microsoft provided official [Remote - SSH extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh). 

Then open the command Palette by pressing `Ctrl + Shift + P`, search for `Remote-SSH: Connect to a Host` then click on *"+ Add New SSH Host..."*.

Then in the prompt type `ssh pi@raspberrypi`, assuming you username is `pi` and the hostname of your Raspberry Pi is `raspberrypi`. Then press Enter.

![SSH Raspberry Pi](/images/99999974-ssh-command.png)


When prompted to select SSH configuration file to update, choose the one under `C:\Users\YourName\.ssh\config`.

## Connect to Raspberry Pi

Then, open the Command Palette again by typing `Ctrl + Shift + P`, and search for `Remote-SSH: Connect to a Host` again. 
Choose `raspberrypi` from the dropdowns. 

![SSH Raspberry Pi](/images/99999974-ssh-connect.png)

You will be prompted for password for your user on Raspberry Pi, enter the password. This will setup the VSCode Remote Server on the Raspberry PI. 

Once you're connected you should see the `SSH: raspberrypi` on bottom left part of your VSCode. 

## Opening a particular folder
Open the explorer on VSCode and you should see the following message. Click on *Open Folder*
![SSH Raspberry Pi](/images/99999974-ssh-explorer.png)


You will get a dropdown, navigate to the folder you want to open and then press OK. Youl will be prompted for password again. And you should now be connected to Raspberry Pi and able to develop on it remotely. 
![SSH Raspberry Pi](/images/99999974-ssh-vscode.png)

---

# Received & return a file from in-memory buffer using FastAPI

URL: https://cloudbytes.dev/snippets/received-return-a-file-from-in-memory-buffer-using-fastapi
Category: Snippets
Published: 2021-10-15
Author: Rehan Haider
Tags: python, fastapi

> How to receive a file to the in-memory buffer and then return the file from buffer using FastAPI without saving it to disk.

FastAPI is fast becoming the go-to choice to write APIs using Python mostly due to its asynchronous nature. 

FastAPI by default will use `JSONResponse` method to return responses, however, it has the ability to return several custom responses including `HTMLResponse` and `FileResponse`. However, both of these messages returns files that are saved on the disk and requires a `PATH`. 

E.g. from [FastAPI Documentation](https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse), 
```python
from fastapi import FastAPI
from fastapi.responses import FileResponse

file_path = "sample-file.mp4"
app = FastAPI()


@app.get("/", response_class=FileResponse)
async def main():
    return file_path
```

So what if you wanted to send a file that is currently in the memory buffer, directly without the additional step of saving it on the disk? 

## Why simply using StreamingResponse is not enough? 
The right way of sending a file from memory is by using `StreamingResponse`, but `StreamingResponse` requires an iterator object, e.g.
```python
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

some_file_path = "large-video-file.mp4"
app = FastAPI()


@app.get("/")
def main():
    def iterfile():  
        with open(some_file_path, mode="rb") as file_like:  
            yield from file_like  

    return StreamingResponse(iterfile(), media_type="video/mp4")
```

But in reality the files e.g. images, etc. that you work with will rarely be an iterator object. Thus you are swapping one workaround with another workaround. 

## Using StreamingResponse correctly
Instead what we will do is, 
1. Receive the image directly in memory
2. Apply a `blur` PIL filter to the image method to the image
3. Return the image directly without saving


```python
from fastapi import FastAPI, File, UploadFile
from fastapi.responses import StreamingResponse
from io import BytesIO

app = FastAPI()

@app.post("/")
def image_filter(img: UploadFile = File(...)):
    original_image = Image.open(img.file)
    original_image = original_image.filter(ImageFilter.BLUR)

    filtered_image = BytesIO()
    original_image.save(filtered_image, "JPEG")
    filtered_image.seek(0)

    return StreamingResponse(filtered_image, media_type="image/jpeg")
```

### Testing the API
Save the above code in a file named app.py

You need to install the following libraries for this to work
```
pip install fastapi
pip install "uvicorn[standard]"
pip install Pillow
pip install python-multipart
```

Then fire up the FastAPI app by running
```bash
uvicorn app:app --reload
```

This should start the app on `127.0.0.1/8000` as shown below

![Uvicorn run](/images/99999975-uvicorn_run.png)

Open the swaggerUI using any browser by openign the link `127.0.0.1/8000/docs`, then click on try it out, then choose a image from and press Execute. 

![Swagger UI](/images/99999975-swagger_ui.png)
After that if you scroll below you should see a blurred image. 

An example of before and after is shown below, (but really this is just an excuse to show you a cat pic)
![Blurred Cat](/images/99999975-cat_pic.jpg)

---

# Set the default command in Python Typer CLI

URL: https://cloudbytes.dev/snippets/set-the-default-command-in-python-typer-cli
Category: Snippets
Published: 2021-10-01
Author: Rehan Haider
Tags: python, typer

> A short guide to setting a default command using callback when making multiple commands in Python Typer

TL;DR - jump to the [solution](#setting-a-default-command).

As I mentioned previously, [Typer is great]({filename}99999977-disable-python-typer-cli-autocompletion.md), but it's documentation isn't. 

While building a Typer app that has only one `@app.command()` decorated function that function is treated as the default command.

```python
import typer

app = typer.Typer(add_completion=False)

@app.command()
def foo(lat: float = None, long: float = None, method: str = None):
    typer.echo(f"{lat}, {long}, {method}")

if __name__ == "__main__":
    app()

```

E.g. if you run the above program 
```bash
/usr/bin/python3 main.py --lat 20.5 --long 88.3 --method cartesian
```

You will get an output like below
```text
20.5 88.3 cartesian
```

This is the expected behaviour because Typer has set foo() as the default action if the program is run. But if you create two commands like the example below

```python
import typer

app = typer.Typer(add_completion=False)


@app.command()
def foo(lat: float = None, long: float = None, method: str = None):
    typer.echo(f"{lat}, {long}, {method}")


@app.command()
def bar():
    typer.echo("I'm just here to mess things up...")


if __name__ == "__main__":
    app()
```

If you re-run this program

```bash
/usr/bin/python3 main.py --lat 20.5 --long 88.3 --method cartesian
```

You will rightly get an error as below stating there are no such options
```text
Usage: main.py [OPTIONS] COMMAND [ARGS]...
Try 'main.py --help' for help.

Error: No such option: --lat
```

The reason is, since there are two commands, typer is expecting to see at least one of them. 
```bash
/usr/bin/python3 main.py --help
```
Checking using help option

As you can see below, there are no default options and two commands, and Typer doesn't know which one to use as default. 
```text
Usage: main.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  bar
  foo
```

## Setting a default command
The answer to out problems? Callbacks!

Typer has a callback functionality allows the developer to create CLI parameters for the main CLI application itself.

So in our example, we will make two changes
1. change foo() decorator from `@app.command()` to `@app.callback()`
2. Add `invoke_without_command=True` argument to the above


```python
import typer

app = typer.Typer(add_completion=False)


@app.callback(invoke_without_command=True)
def foo(lat: float = None, long: float = None, method: str = None):
    typer.echo(f"{lat}, {long}, {method}")


@app.command()
def bar():
    typer.echo("I'm just here to mess things up...")


if __name__ == "__main__":
    app()
```

If you check the help again, 

```text
Usage: main.py [OPTIONS] COMMAND [ARGS]...

Options:
  --lat FLOAT
  --long FLOAT
  --method TEXT
  --help         Show this message and exit.

Commands:
  bar
```
You don't see `foo()` since that has become the default action and the `--lat`, `--long`, & `--method` are added as default options. 

If you run the program again
```bash
/usr/bin/python3 main.py --lat 20.5 --long 88.3 --method cartesian
```

You will get an output like below
```text
20.5 88.3 cartesian
```

---

# Disable the default completion options in Python Typer CLI

URL: https://cloudbytes.dev/snippets/disable-the-default-completion-options-in-python-typer-cli
Category: Snippets
Published: 2021-09-25
Author: Rehan Haider
Tags: python, typer

> A short guide to disable the default options such as install-completion and show-completion in Python Typer CLI, a popular CLI building tool

TL;DR - jump to the [solution](#disable-completion-option).

[Typer](https://typer.tiangolo.com/) is a great! But it's documentation isn't. 

So I ran into a challenge while building a simple app where I didn't intend to provide the users autocompletion options and was wondering how to disable the typical output that it prints out while invoking the program

```text
Usage: app.py [OPTIONS]

Options:
  --lat FLOAT
  --long FLOAT
  --method TEXT
  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.
  --help                          Show this message and exit.
```

As you can see, it is rather a simple app where only 3 inputs are provided and it just looks cluttered. So I wanted to disable it. 

Unfortunately, as I mentioned earlier, the documentation didn't talk about it. 

## Disable completion option

So eventually it took a bit of reading the source course to figure out how to disable it. The trick is to pass `add_completion=False` argument while initialising the `typer.Typer` app, as shown below

```python
import typer

app = typer.Typer(add_completion=False)

@app.command()
def foo(lat: float = None, long: float = None, method: str = None):
    typer.echo(f"{lat}, {long}, {method}")



if __name__ == "__main__":
    app()
```

And now the output unsurprisingly looks like

```text
Usage: app.py [OPTIONS]

Options:
  --lat FLOAT
  --long FLOAT
  --method TEXT
  --help         Show this message and exit.
```

---

# Get username, hostname and home directory using Python

URL: https://cloudbytes.dev/snippets/get-username-hostname-and-home-directory-using-python
Category: Snippets
Published: 2021-09-22
Author: Rehan Haider
Tags: python

> A quick guide to fetching the username, hostname and home directory using Python on both Windows & Linux

For a variety of reason your Python app might want to know the username of the logged in user along with a few other details such as path to their home directory and their systems hostname. 

In Python, you can use use [getpass](https://docs.python.org/3/library/getpass.html) library to fetch these.

## Get the username

Run the below to get the username

```python
import getpass
username = getpass.getuser()

print(f"Hello {username}")
```
### Output
On Linux you see
![python find username linux](/images/99999978-username-linux.png)

This will also work on Windows
![python find username windows](/images/99999978-username-windows.png)

## Get the path to home directory

```python
import os.path

homedir = os.path.expanduser("~")
print(homedir)
```

### Output
On Linux
![python find homedir linux](/images/99999978-homedir-linux.png)

And on Windows
![python find homedir windows](/images/99999978-homedir-windows.png)


## Get the hostname

```python
import socket
hostname = socket.gethostname()
print(hostname)
```
### Output
On Linux
![python find hostname linux](/images/99999978-hostname-linux.png)

And on Windows
![python find hostname windows](/images/99999978-hostname-windows.png)

---

# AWS CDK: Building a EventBridge scheduled Lambda that reads and writes to s3

URL: https://cloudbytes.dev/snippets/aws-cdk-building-a-eventbridge-scheduled-lambda-that-reads-and-writes-to-s3
Category: Snippets
Published: 2021-08-12
Author: Rehan Haider
Tags: aws, python

> A guide to building a serverless app using EventBridge, Lambda & s3 while introducing how to create, configure, test and deploy a CDK project using Python

[TOC]

AWS Cloud Development Kit, also known as [CDK](https://aws.amazon.com/cdk/), is an Open Source Software Development Framework that is maintained by AWS.

CDK is supposed to be the umbrella SDK from AWS, which can also easily integrate with [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html) and also [AWS Chalice](https://aws.github.io/chalice/), two other open source SDKs that AWS provides. 

Now, there are hundreds of simple guides that explains how to setup a simple Lambda app using CDK, but in this article we'll build a bit more complex example. 

## The CDK App that we will build
We'll build a Lambda app that runs at a specified time (using EventBridge). The Lambda app will read a file from S3 bucket that contains latitude and longiture stored in a CSV format. Then the Lambda app will fetch the sunrise & sunset times for those coordinates and save it to another file in S3. 

![CDK App architecure](/images/99999981-suntimes.png)

### Components
Before we get into the weeds, let's look at what are the components that we will need to configure

1. **Sun timings API**: We'll use a public API endpoint, `https://sunrise-sunset.org/api`
2. **EventBridge**: An event generated at a specified time using Cron. This event will be used to invoke Lambda. EventBridge needs to have permissing to send target the Lambda function for invocation
3. **S3 Object with Coordinates**: A file that contains a list of coordinates. 
4. **S3 Object with output**: This will be created by Lambda
5. **Lambda Function**: A Lambda function that reads the list of coordinates from S3, fetches the sunrise / sunset times for them, converts them to JSON and saves it in S3. Lambda will require read & write permission to S3. To read and write from S3 we will use AWS Boto Library

## Setting up the development environment
You need [Docker](https://docs.docker.com/get-docker/) & [VSCode](https://code.visualstudio.com/download) to be installed on your system for this guide. Download fromt he provided links and install. 

Then follow the following steps. 

**Step 1**: Install Python using [these instructions]({filename}99999987-how-to-check-python-version.md).

**Step 2**: [Install AWS CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md#install-aws-cli)

**Step 3**: [Configure AWS & AWS CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md#configuring-aws-aws-cli)

**Step 4**: [Install and configure AWS CDK](https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html#getting_started_install)

**Step 5**: Bootstrap the CDK, i.e. configure your AWS account to be able to use CDK. 
To do this first run the following AWS CLI command
```bash
aws sts get-caller-identity
```

This will produce an output similar to below that will provide the AWS account number, and user-id.
```json
{
    "UserId": "XXXXXXXXXXXXXX",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:user/cloudbytes"
}
```


Then run the below command replacing account & userID
```bash
cdk bootstrap aws://<Account>/<Region>
```
> You can get your default region by running `aws configure get region`

## Setting up the project
Create a new folder for the project, CDK requires an empty folder to initialise

```
mkdir cdk-tutorial
cd cdk-tutorial
```

After that, create a new CDK project by running
```bash
cdk init app --language python
```
> `app` in the above is a template, don't change it

This will produce a bunch of files in the below structure
```
├── README.md
├── app.py
├── cdk.json
├── cdk_tutorial
│   ├── __init__.py
│   └── cdk_tutorial_stack.py
├── requirements.txt
├── setup.py
└── source.bat
```

It will also create a virtual environment, to active run the below on Linux / MacOS
```bash
source .venv/bin/activate
```

Or on Windows, run
```powershell
.venv\Scripts\activate.bat
```

Then finally run the below to install the project dependencies,

```bash
python3 -m pip install -r requirements.txt
```
> These are the libraries used by your project, not the actual Lambda app. We will define that later


## Modifying the structure
Now the above initialisation has created 3 important files

1. `app.py`: This contains the initialisation of the app itself. This doesn't require any change unless you are changing the name of the app itself.
2. `cdk_tutorial/cdk_tutorial_stack.py`: This is where we will define our app construct, i.e. the services that we will use, the permissions that we need, etc. 
3. `setup.py`: This contains certain project information such as libraries etc. 
   
### Adding Lambda Handler
Create a folder named `lambda` and under the folder create two files
1. `__init__.py`: Should be an empty file
2. `index.py`: Leave the contents blank for now

## Defining the lambda function
Go to `cdk_tutorial/cdk_tutorial_stack.py`, and change the imports as per below

```python
from aws_cdk import core as cdk, aws_lambda, aws_events_targets as targets, aws_events as events, aws_s3 as s3
import subprocess
```

Then change the main program as per below

```python
# app.py
class CdkTutorialStack(cdk.Stack):

    def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Define a lambda function
        lambdaFn = aws_lambda.Function(
            self,
            "cdk-tutorial",
            code=aws_lambda.Code.from_asset("lambda"),
            handler="index.main",
            runtime=aws_lambda.Runtime.PYTHON_3_10,
            layers=[self.create_dependencies_layer(self.stack_name, "lambda/index")],
        )
```
Here we use `code` variable to import our code from `lambda` folder, then define Lambda handler to be `main()` method under `index.py` file by using `handler="index.main"`.

We also define the runtime to be Python 3.10 and a layer that is explained later. 

## Define EventBridge Schedules and Lambda access permission
Add the below code under where we defined lambda function

```python
        rule = events.Rule(
            self,
            "Run Daily at 21:30 hrs UTC",
            # UTC + 0 time. ~3 AM IST
            schedule=events.Schedule.cron(minute="30", hour="21", week_day="*", month="*", year="*"),
        )

        rule.add_target(targets.LambdaFunction(lambdaFn))
```

This simple definition has two parts
1. Where we define an Eventbridge rule that runs everyday as 9:30 PM UTC. The event schedule is always expressed in UTC timezone
2. We add a target to the lambda function that we created, `lambdaFn`

That's it, two statements to create and then give permission to EventBridge to invoke a Lambda function.

## Giving Lambda access to S3 bucket to read and write files

For this you need the name of the bucket that you want access to, in this example let's use `cloudbytes-dev`, you can replace it with bucket of your choice that you are using.

To provide access, we first define the bucket construct and then grant read & write permission as per below
```python
        my_bucket = s3.Bucket.from_bucket_name(self, "Bucket", "cloudbytes-dev")
        my_bucket.grant_read_write(lambdaFn)
```
## Create a dependencies layer in Lambda CDK
For our program we are going to use Python's `requests` library, however, this is not available by default on Lambda, so we need to upload this while creating our project. 

To do that, we need to do two things, first, add the below function in your `app.py`

```python
    def create_dependencies_layer(self, project_name, function_name: str) -> aws_lambda.LayerVersion:
        requirements_file = f"requirements.app.txt"
        output_dir = f".build/app"

        if not os.environ.get("SKIP_PIP"):
            subprocess.check_call(f"pip install -r {requirements_file} -t {output_dir}/python".split())

        layer_id = f"{project_name}-{function_name}-dependencies"
        layer_code = aws_lambda.Code.from_asset(output_dir)

        return aws_lambda.LayerVersion(self, layer_id, code=layer_code)
```

Then create a file named `requirements.app.txt` in your project root (where requirements.txt is) and add the following in that file
```
requests==2.26.0
boto3==1.18.24
```

(Optional) Install these in your virtual environment by running

```bash
python3 -m pip install -r requirements.app.txt
```
## Upload the coordinates file to S3
Copy the below in a file and name it `coordinates.csv`

```csv
latitude,longitude
-33.865143,151.209900
22.572645,88.363892
```
Upload this file to the S3 bucket that you are going to use

## Writing the lambda app
Go back to you `lambda` folder that we craete earlier and opent he `index.py` file

Add the following code to the file
```python
from botocore import endpoint
import requests
import os
import boto3
import csv
import json


def main(event, context):
    s3 = boto3.client("s3")
    bucket = "cloudbytes-dev"
    file_name = "coordinates.csv"
    # Download the coordinate files from S3
    s3.download_file(bucket, file_name, f"/tmp/{file_name}")

    coordinates = []

    with open(f"/tmp/{file_name}", "r") as file:
        rows = csv.DictReader(file)
        for row in rows:
            endpoint = f"https://api.sunrise-sunset.org/json?lat={row['latitude']}&lng={row['longitude']}"
            response = requests.get(endpoint)
            row["sunrise"] = response.json()["sunrise"]
            row["sunset"] = response.json()["sunset"]
            coordinates.append(row)

        output_file = "suntimes.json"
        with open(f"/tmp/{output_file}", "w") as file:
            file.write(json.dumps(str(coordinates)))

        # Upload the output file to S3
        s3.upload_file(f"/tmp/{output_file}", bucket, f"{output_file}")
```

## Deploy the CDK project
Before deploying we first need to synthesis (or generate the project CloudFormation template) by running
```bash
cdk synth
```

This will generate a lengthy CloudFormation output in the console.

Finally, we deploy the project by running
```bash
cdk deploy
```

If asked for a confirmation, press `Y` to deploy. 

Now you can go to your AWS console, go to lambda section and test the function. 

## Cleanup & Destroy the CDK project
We created several resources and policies as part of this tutorial, instead of deleting them one by one, just run 

```bash
cdk destroy
```

This will clean up the project competely. 

And this is how you build a complex Lambda application on AWS

---

# Run Selenium in AWS Lambda for UI testing

URL: https://cloudbytes.dev/snippets/run-selenium-in-aws-lambda-for-ui-testing
Category: Snippets
Published: 2021-08-12
Author: Rehan Haider
Tags: aws, selenium, python

> A guide about how to run selenium using headless chrome & chromium webdriver to automate UI testing

[TOC]

**[LAST UPDATED: 29-July-2022]**

Let me begin by expressing my frustration 😡😡😡 with the fact that AWS doesn't have a pre-configured selenium image for **Lambda** on their public ECR marketplace. [Selenium](https://selenium.dev) is the go-to tool for UI testing and for building many kinds of bots but running it on **Lambda** is complicated. 

The easiest method is to use [SAM CLI for **Docker for Lambda**]({filename}99999983-run-lambda-on-container-sam.md) to create an image with **Selenium**, **Chrome / Chromium headless** and **webdriver**, but given the way Lambda restricts the environment making it work on Selenium is quite difficult but not impossible. 

In this tutorial I will provide a guide on how to do exactly that. 

## Prerequisites

Follow [these instructions to setup your development environment]({filename}/aws/00000000-setting-up-dev-env.md).

It will guide you to install and configure AWS CLI & SAM CLI.

## Create the app

Follow the instructions in [this guide to create Lambda with Docker]({filename}99999983-run-lambda-on-container-sam.md#create-a-new-app).

Your folder structure should look like below. 
```
.
├── README.md
├── __init__.py
├── events
│   └── event.json
├── hello_world
│   ├── Dockerfile
│   ├── __init__.py
│   ├── app.py
│   └── requirements.txt
├── template.yaml
└── tests
    ├── __init__.py
    └── unit
        ├── __init__.py
        └── test_handler.py
```

## Customising the app

First, change the name of `hello-world` directory to `src`.

### __init__.py
Both the `__init__.py` files should be empty

### Events: events/event.json
Leave the contents of the `event.json` file unchanged.

### Application: src/app.py
We write a simple Python program that uses selenium webdriver to scape a website. 

Change the contents of the file to below. 

```python
## Run selenium and chrome driver to scrape data from cloudbytes.dev
import time
import json
import os.path
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options



def handler(event=None, context=None):
    chrome_options = webdriver.ChromeOptions()
    chrome_options.binary_location = "/opt/chrome/chrome"
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--disable-dev-tools")
    chrome_options.add_argument("--no-zygote")
    chrome_options.add_argument("--single-process")
    chrome_options.add_argument("window-size=2560x1440")
    chrome_options.add_argument("--user-data-dir=/tmp/chrome-user-data")
    chrome_options.add_argument("--remote-debugging-port=9222")
    #chrome_options.add_argument("--data-path=/tmp/chrome-user-data")
    #chrome_options.add_argument("--disk-cache-dir=/tmp/chrome-user-data")
    chrome = webdriver.Chrome("/opt/chromedriver", options=chrome_options)
    chrome.get("https://cloudbytes.dev/")
    description = chrome.find_element(By.NAME, "description").get_attribute("content")
    print(description)
    return {
        "statusCode": 200,
        "body": json.dumps(
            {
                "message": description,
            }
        ),
    }
```
### Python Dependencies: src/requirements.txt

Capture the app dependencies in `requirements.txt`
```
selenium
requests
pandas
```

### Chrome dependencies: src/chrome-deps.txt

Create a file named `src/chrome-deps.txt` with the following contents

```text
acl adwaita-cursor-theme adwaita-icon-theme alsa-lib at-spi2-atk at-spi2-core 
atk avahi-libs cairo cairo-gobject colord-libs cryptsetup-libs cups-libs dbus 
dbus-libs dconf desktop-file-utils device-mapper device-mapper-libs elfutils-default-yama-scope
elfutils-libs emacs-filesystem fribidi gdk-pixbuf2 glib-networking gnutls graphite2 
gsettings-desktop-schemas gtk-update-icon-cache gtk3 harfbuzz hicolor-icon-theme hwdata jasper-libs
jbigkit-libs json-glib kmod kmod-libs lcms2 libX11 libX11-common libXau libXcomposite libXcursor 
libXdamage libXext libXfixes libXft libXi libXinerama libXrandr libXrender libXtst libXxf86vm libdrm 
libepoxy liberation-fonts liberation-fonts-common liberation-mono-fonts liberation-narrow-fonts 
liberation-sans-fonts liberation-serif-fonts libfdisk libglvnd libglvnd-egl libglvnd-glx libgusb 
libidn libjpeg-turbo libmodman libpciaccess libproxy libsemanage libsmartcols libsoup libthai libtiff 
libusbx libutempter libwayland-client libwayland-cursor libwayland-egl libwayland-server libxcb 
libxkbcommon libxshmfence lz4 mesa-libEGL mesa-libGL mesa-libgbm mesa-libglapi nettle pango pixman 
qrencode-libs rest shadow-utils systemd systemd-libs trousers ustr util-linux vulkan 
vulkan-filesystem wget which xdg-utils xkeyboard-config
```

### Dockerfile: src/Dockerfile

Change the contents of the file to below.


```Dockerfile
FROM public.ecr.aws/lambda/python:3.9 as stage

# Hack to install chromium dependencies
RUN yum install -y -q sudo unzip

# Find the version of latest stable build of chromium from below
# https://omahaproxy.appspot.com/
# Then follow the instructions here in below URL 
# to download old builds of Chrome/Chromium that are stable
# Current stable version of Chromium
ENV CHROMIUM_VERSION=1002910 


# Install Chromium
COPY install-browser.sh /tmp/
RUN /usr/bin/bash /tmp/install-browser.sh

FROM public.ecr.aws/lambda/python:3.9 as base

COPY chrome-deps.txt /tmp/
RUN yum install -y $(cat /tmp/chrome-deps.txt)

# Install Python dependencies for function
COPY requirements.txt /tmp/
RUN python3 -m pip install --upgrade pip -q
RUN python3 -m pip install -r /tmp/requirements.txt -q 


COPY --from=stage /opt/chrome /opt/chrome
COPY --from=stage /opt/chromedriver /opt/chromedriver
COPY app.py ${LAMBDA_TASK_ROOT}

CMD [ "app.handler" ]

```

### Script to install browser: src/install-browser.sh
Create a file at `src/install-browser.sh`. We will use a simple shell script to install the latest Chrome and Chrome webdriver.

```bash
#!/bin/bash


echo "Downloading Chromium..."
curl "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/\
Linux_x64%2F$CHROMIUM_VERSION%2Fchrome-linux.zip?generation=1652397748160413&alt=media" > /tmp/chromium.zip

unzip /tmp/chromium.zip -d /tmp/
mv /tmp/chrome-linux/ /opt/chrome

curl "https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/\
Linux_x64%2F$CHROMIUM_VERSION%2Fchromedriver_linux64.zip?generation=1652397753719852&alt=media" > /tmp/chromedriver_linux64.zip

unzip /tmp/chromedriver_linux64.zip -d /tmp/
mv /tmp/chromedriver_linux64/chromedriver /opt/chromedriver
```

Then run the below command to make the script executable.

```bash
chmod +x src/install-browser.sh
```

### template.yaml
Change the contents to below. Based on the complexity of your app, you may need to increase the memory and timeout values under Globals:Function.


```yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  python3.9

  Sample SAM Template for selenium

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 120
    MemorySize: 2048

Resources:
  SeleniumFunction:
    Type: AWS::Serverless::Function 
    Properties:
      PackageType: Image
      Architectures:
        - x86_64
      Events:
        Selenium:
          Type: Api 
          Properties:
            Path: /selenium
            Method: get
    Metadata:
      Dockerfile: Dockerfile
      DockerContext: ./src
      DockerTag: python3.9-v1

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  SeleniumApi:
    Description: "API Gateway endpoint URL for Prod stage for Seleniumc function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/selenium/"
  SeleniumFunction:
    Description: "Selenium Lambda Function ARN"
    Value: !GetAtt SeleniumFunction.Arn
  SeleniumFunctionIamRole:
    Description: "Implicit IAM Role created for Selenium function"
    Value: !GetAtt SeleniumFunctionRole.Arn
```

## Build & test the app

To build the app run, 
```bash
sam build
```

To test run
```bash
sam local invoke
```
### Output
You should see something similar to below depending on the URL you scraped

![sam local invoke success](/images/99999982-sam_local_invoke_success.png)


## Deploy the app

To deploy the app for the first time run,

```bash
sam deploy --guided
```

This will start the interactive deployment to Lambda. You can use options as shown below.

![sam deploy guided](/images/99999982-sam-deploy-guided.png)

This will also create a `samconfig.toml` file that will contain these configurations.

Next time after you build the app, just run `sam deploy` to deploy the app.

After a successful deployment, you should see something similar to below. Note the API URL in the output at the bottom.

![99999982-sam-deploy-success](/images/99999982-sam-deploy-success.png)

## Test the app

Using the API URL from the output, you can test the app by running

```bash
curl -X GET <API URL>
```

## Cleanup
To delete the app, run `sam delete`.

## Using the GitHub repository directly

You need AWS SAM CLI installed and AWS credentials configured.

Open your terminal and run the following command to clone the [repository](https://github.com/rehanhaider/selenium-in-aws-lambda).

```git
git clone https://github.com/rehanhaider/selenium-in-aws-lambda.git
```

Navigate to the app directory. 

```bash
cd selenium-in-aws-lambda/selenium
```

Build the app.

```bash
sam build
```

Test the app locally.

```bash
sam local invoke
```

Deploy the app to AWS.

```bash
sam deploy --guided
```

---

# Run AWS Lambda using custom docker container

URL: https://cloudbytes.dev/snippets/run-aws-lambda-using-custom-docker-container
Category: Snippets
Published: 2021-08-08
Author: Rehan Haider
Tags: aws, python

> Learn how to use a custom docker container to run Lambda functions on AWS.

[TOC]

**[LAST UPDATED: 29-July-2022]**

I wrote about [building and deploying a AWS Lambda using SAM CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md) previously.

In this guide, we try to run a Lambda function inside a container. 

## Why run Lambda in Docker? 

For a very simple reason, Lambda runtimes are standardised environments where you can only use what they provide and they do not provide a lot. E.g. if your application required any binary to be installed you coudn't do that on Lambda. 

But in 2020 Re:invent, AWS launched [Container Image Support for Lambda](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/) for container images up to 10 GB in size. While for this may not be important for "one-off functions", but for many use cases such as machine learning models etc, the developmental workflow typically includes Dockers and that is where it gets tricky deploying them to AWS Lambda.

## Setting up the development environment. 

You need [Docker](https://docs.docker.com/get-docker/) & [VSCode](https://code.visualstudio.com/download) to be installed on your system for this guide. Download fromt he provided links and install. 

Then follow the following steps. 

**Step 1**: Install Python using [these instructions]({filename}99999987-how-to-check-python-version.md).

**Step 2**: [Install AWS CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md#install-aws-cli)

**Step 3**: [Install SAM CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md#install-aws-sam-cli)

**Step 4**: [Configure AWS & AWS CLI]({filename}99999984-deploy-serverless-apps-with-aws-sam.md#configuring-aws-aws-cli)


## Create a new app

Run the below in your terminal to create a new SAM application
```bash
sam init --package-type Image
```

This will start the interactive session to create your app. Choose Option as per below

```bash
Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1

Choose an AWS Quick Start application template
        1 - Hello World Example
        2 - Machine Learning
Template: 1

Which runtime would you like to use?
        1 - dotnet6
        2 - dotnet5.0
        3 - dotnetcore3.1
        4 - go1.x
        5 - java11
        6 - java8.al2
        7 - java8
        8 - nodejs16.x
        9 - nodejs14.x
        10 - nodejs12.x
        11 - python3.9
        12 - python3.8
        13 - python3.7
        14 - python3.6
        15 - ruby2.7
Runtime: 11

Based on your selections, the only dependency manager available is pip.
We will proceed copying the template using pip.

Would you like to enable X-Ray tracing on the function(s) in your application?  [y/N]:  

Project name [sam-app]: selenium

Cloning from https://github.com/aws/aws-sam-cli-app-templates (process may take a moment)

    -----------------------
    Generating application:
    -----------------------
    Name: selenium
    Base Image: amazon/python3.9-base
    Architectures: x86_64
    Dependency Manager: pip
    Output Directory: .

    Next steps can be found in the README file at ./selenium/README.md

```

### Understanding the SAM generated application template

First, go to the `selenium` directory.

```bash
cd selenium
```

You should see the following files

```
.
├── README.md
├── __init__.py
├── events
│   └── event.json
├── hello_world
│   ├── Dockerfile
│   ├── __init__.py
│   ├── app.py
│   └── requirements.txt
├── template.yaml
└── tests
    ├── __init__.py
    └── unit
        ├── __init__.py
        └── test_handler.py
```

[Compared to the standard Lambda example]({filename}99999984-deploy-serverless-apps-with-aws-sam.md), this has an additional file, the `Dockerfile` that contains the instructions to build the container where the lambda will be executed. 

```Dockerfile
FROM public.ecr.aws/lambda/python:3.9

COPY app.py requirements.txt ./

RUN python3.9 -m pip install -r requirements.txt -t .

# Command can be overwritten by providing a different command in the template directly.
CMD ["app.lambda_handler"]
```

The first thing you notice is, this image is building on top of an image from [AWS's pubic container registry](https://gallery.ecr.aws/lambda/).

> !!! tip "NOTE: You can also use non-AWS images such as those based on Alpine or Debian, however, the container image must include [Lambda Runtime API](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html). So if you use a non-AWS image, you will need to add them manually otherwise your app will not work."

And the final line is responsible for running the `lambda_handler()` function that in defined under `twitter/hello_world/app.py`. 

## Build the project

To build the app, run the following
```bash
sam build
```
> You need Docker & Python3.9 to be installed for this to work

![Sam build success](/images/99999983-sam-build-success.png)

### Test the build
To test if you application is working correctly, run
```text
sam local invoke
```
> Again, you need Docker & Python3.9 to be installed for this to work

You should see the following output:

![Sam local invoke success](/images/99999983-sam-local-invoke-success.png)

## Deploy the project

Now there are three more steps that needs to be performed, but in our case, SAM CLI will do them in one go. 

These steps are (again, we don't need to do them if using SAM CLI)
1. We need to rename the tag of our docker container to push it to the repository. 
2. Login from Docker CLI to ECR repository
3. Push the image to ECR repository

All we need to do now is
```bash
sam deploy --guided
```

This will start an interactive delployment session, choose options as below (blank means leave the defaults)

```
Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Not found

        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [sam-app]: hello-world
        AWS Region [us-east-1]: 
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [y/N]: 
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: y
        #Preserves the state of previously provisioned resources when an operation fails
        Disable rollback [y/N]: 
        HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
        Save arguments to configuration file [Y/n]: 
        SAM configuration file [samconfig.toml]: 
        SAM configuration environment [default]: 
```

After that, if will further ask for confirmation on creating ECR repository:

```
Create managed ECR repositories for all functions? [Y/n]: Y
```

This will deploy your app to AWS and you should see a final confirmation output similar to below. Note the URL of the API highlighted in the output.

![sam deploy success](/images/99999983-sam-deploy-success.png)

## Test the deployed app

Fetch the URL of the API from the output above and send a GET request to the API using the below command. 

```bash
curl -X GET https://lndcelxeyg.execute-api.us-east-1.amazonaws.com/Prod/hello
```

In response, you should see:

```json
{
  "message": "hello world!"
}
```

## Clean up and delete the app

To delete the app, run the following

```bash
sam delete
```
You will be asked for below confirmations:

```text
Are you sure you want to delete the stack hello-world in the region us-east-1 ? [y/N]: y
  Are you sure you want to delete the folder hello-world in S3 which contains the artifacts? [y/N]: y
  Found ECR Companion Stack hello-world-20953121-CompanionStack
  
  Do you you want to delete the ECR companion stack hello-world-20953121-CompanionStack 
  in the region us-east-1 ? [y/N]: y
  
  ECR repository helloworld20953121/helloworldfunction19d43fc4repo may not be empty. 
  Do you want to delete the repository and all the images in it ? [y/N]: y
```

You app is deleted.

---

# Build & Deploy serverless apps on AWS with SAM CLI

URL: https://cloudbytes.dev/snippets/build-deploy-serverless-apps-on-aws-with-sam-cli
Category: Snippets
Published: 2021-08-02
Author: Rehan Haider
Tags: aws, python

> Detailed step by step guide on how to use SAM CLI to build serverless apps.

[TOC]

AWS changed the game in 2015 when [they launched AWS Lambda](https://aws.amazon.com/blogs/compute/aws-lambda-is-generally-available/) which enabled developers build applications without needed a server. 

Since then, Lambda and Serverless in general has grown in leaps and bounds with launch of [API Gateway](https://aws.amazon.com/api-gateway/), among many other services that allows developers to build full-fledged serverless apps. 

> While CloudFormation (an IaaC tool) has existed for years, it is usually cumbersome to write CloudFormation templates with simplest ones going upto hundres of lines of codes. This had led to developers preferring  third-party tools such as Terraform or Pulumi, but with AWS launching their **SAM CLI** that is changing fast. 

Within AWS Ecosystem, there are a plethora of options available, the details of which we will cover in another lesson, but the two that we will use for this lesson are

1. [AWC CLI](https://aws.amazon.com/cli/): This allows a developer to manage all of their AWS services using CLI
2. [AWS SAM-CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-reference.html#serverless-sam-cli): This is a separate CLI tool that covers a subset of AWS CLI but is also a wrapper on CloudFormation and in fact generate a CloudFormation template as part of its build process.

In this tutorial, we will use AWS CLI and SAM-CLI to build a simple Lambda application using Python 3.8 and deploy it to AWS. 

The application of our choice is a Lambda function behind API Gateway that will return `Hello World` message when invoked. 

![AWS Lambda API Gateway Hello world](/images/99999984-sam-getting-started-hello-world.png)

## Setting up the development environment

Download and install Python 3.9 for your OS from [Python Website](https://www.python.org/ftp/python/3.9.6/). 

> !!! tip "IMPORTANT: The latest supported Python version on AWS Lambda is 3.8, thus you need 3.9 to be able to build your Lambda application"

You can refer to [this tutorial]({filename}99999987-how-to-check-python-version.md#2-how-to-install-python) if you need help in installing python correctly. 

Check your Python version to confirm
```bash
python --version
```

### Install AWS CLI

Installation method varies by OS. Steps below. 

**On Windows**
Download the [64-bit installer](https://awscli.amazonaws.com/AWSCLIV2.msi) and run to install. 

**On Linux**, run 
```bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
   && unzip awscliv2.zip
```

Then Install the AWS CLI by running the following command:
```bash
sudo ./aws/install
```

**On MacOS**
Download the [MacOS PKG](https://awscli.amazonaws.com/AWSCLIV2.pkg) and run to install. 

To check if the installation was successful, run a version check
```bash
aws --version
```

### Install AWS SAM CLI

Again, installation varies by Operating System, choose from below as appropriate. 

**On Windows**
Download and run the [64-bit installer](https://github.com/aws/aws-sam-cli/releases/latest/download/AWS_SAM_CLI_64_PY3.msi).

**On Linux**, run the below
```bash
wget "https://github.com/aws/aws-sam-cli/releases/latest/download/aws-sam-cli-linux-x86_64.zip" \
   -O "awssam.zip" \
   && unzip awssam.zip -d sam \
   && sudo ./sam/install
```

**On MacOS** run
```bash
brew tap aws/tap
brew install aws-sam-cli
```

Verify the installation by running a version check
```bash
sam --version
```

## Configuring AWS & AWS CLI

Both AWS CLI and SAM CLI relies on **"Programmatic Access"** credentials. So you will need to either create a new user or use your own Access Keys. 

Ideally, create a user with only programmatic access as shown below. 

### Create a new admin user with programmatic access

1. Login to your [AWS Console Home](https://console.aws.amazon.com/) and navigate to [IAM](https://console.aws.amazon.com/iam/home)
2. On the left pane, click on **Users**
3. Then Click on **Add users**
4. Choose a username & select only *Programmatic access* under **Select AWS access type**, then click **Next: Permissions**
   
![Create an AWS IAM user](/images/99999984-aws-iam-type.png)

Then click on **Attach existing policies directly** and choose **AdministratorAccess** then click on **Next: Tags**. 

Leave the tags blank for now, and click on **Next: Review** then click on **Create user**. This will create a new user.

Keep this window open for now, and notice the Access key ID & Secret access key. This will be needed in next step.

> !!! danger "WARNING: Never store this credentials anywhere or share them with anyone. An attacker can user your credentials to create AWS resources in your account. If you need to reconfigure, you can generate a new credentials from IAM screen. "

![AWS New IAM User](/images/99999984-aws-new-iam-user.png)

### Configure AWS CLI

Open a terminal window and run the below to start the configuration
```bash
aws configure
```

This will start an interactive session, copy paste your access keys and secret access keys when prompted
```bash
aws configure
AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXXXX
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXXXX
Default region name [None]: us-east-1
Default output format [None]: 
```

## Create a new app

Now run the below to create a new SAM App

```bash
sam init
```

This will prompt you to choose between Quick Start & Custom Template, choose 1 and press Enter followed by choose Zip (1). 
Then finally, choose python3.8 (option 2) when asked to choose the runtime as shown below. Leave the name as default (sam-app). 


```
Which template source would you like to use?
        1 - AWS Quick Start Templates
        2 - Custom Template Location
Choice: 1
What package type would you like to use?
        1 - Zip (artifact is a zip uploaded to S3)
        2 - Image (artifact is an image uploaded to an ECR image repository)
Package type: 1

Which runtime would you like to use?
        1 - nodejs14.x
        2 - python3.9
        3 - ruby2.7
        4 - go1.x
        5 - java11
        6 - dotnetcore3.1
        7 - nodejs12.x
        8 - nodejs10.x
        9 - python3.8
        10 - python3.7
        11 - python3.6
        12 - python2.7
        13 - ruby2.5
        14 - java8.al2
        15 - java8
        16 - dotnetcore2.1
Runtime: 2

Project name [sam-app]: 

Cloning from https://github.com/aws/aws-sam-cli-app-templates
``` 

Once the clone process is complete, you will be prompted to choose the application template, choose 1 - Hello World Example, as shown below

```bash
AWS quick start application templates:
        1 - Hello World Example
        2 - EventBridge Hello World
        3 - EventBridge App from scratch (100+ Event Schemas)
        4 - Step Functions Sample App (Stock Trader)
        5 - Elastic File System Sample App
Template selection: 1

    -----------------------
    Generating application:
    -----------------------
    Name: sam-app
    Runtime: python3.8
    Dependency Manager: pip
    Application Template: hello-world
    Output Directory: .
    
    Next steps can be found in the README file at ./sam-app/README.md
```

## Understandign the project structure

Go to the `sam-app` directory
```bash
cd sam-app
```
You should see the following files
```bash
.
├── README.md
├── __init__.py
├── events
│   └── event.json
├── hello_world
│   ├── __init__.py
│   ├── app.py
│   └── requirements.txt
├── template.yaml
└── tests
```

The key files to look at are
1. `event.json`: Invocation events that can be used to invoke the Lambda function
2. `hello_world/app.py`: Contains the code for the Lambda function
3. `template.yaml`: The SAM's template that declare the AWS resources that will be used for this app

## Building the app

With SAM CLI, building the app is easy, just run
```bash
sam build
```
> **NOTE**: If your Python versio is not 3.8, you will get an error on the build. Reinstall Python 3.8 version. 

You should get a `Build Succeeded` message along with a few other debug messages, if the build completed successfully. You can see the built lambda function in `.aws-sam` folder which will contain the dependencies and the app.py application. 

## Deploy the app
Now to deploy the app run
```bash
sam deploy --guided
```

This will first throw a warning
```text
Looking for config file [samconfig.toml] :  Not found
```

But then if will continue and ask you a few questions, choose as per below

```text
        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [sam-app]: 
        AWS Region [us-east-1]: 
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [y/N]: y
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: y
        #Preserves the state of previously provisioned resources when an operation fails
        Disable rollback [y/N]: 
        HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
        Save arguments to configuration file [Y/n]: 
        SAM configuration file [samconfig.toml]: 
        SAM configuration environment [default]: 
```

If there are any unreferenced ECR repositories, it will ask you to confirm the deletion.
```bash
         Delete the unreferenced repositories listed above when deploying? [y/N]: y
```


SAM CLI will print out a lot of debug information, but you should get a prompt asking for confirmation
```bash
Deploy this changeset? [y/N]: y
```

Finally you app is deployed. You should see a final output similar to the below
```text
CloudFormation outputs from deployed stack
---------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                   
---------------------------------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole                                                                             
Description         Implicit IAM Role created for Hello World function                                                    
Value               arn:aws:iam::268674271179:role/sam-app-HelloWorldFunctionRole-M6IZJ8JLWL61                            

Key                 HelloWorldApi                                                                                         
Description         API Gateway endpoint URL for Prod stage for Hello World function                                      
Value               https://jkasd6ja8.execute-api.us-east-1.amazonaws.com/Prod/hello/                                    

Key                 HelloWorldFunction                                                                                    
Description         Hello World Lambda Function ARN                                                                       
Value               arn:aws:lambda:us-east-1:1364247115578:function:sam-app-HelloWorldFunction-AxJdejTmhKx3                
---------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - sam-app in us-east-1
```

## Test the app
In the output above, note the URL in the format `https://<api-id>.execute-api.us-east-1.amazonaws.com/Prod/hello/ `, you should have got a similar URL, copy that and run the below 

```bash
curl https://<api-id>.execute-api.us-east-1.amazonaws.com/Prod/hello/
```

This should produce the output

```json
{"message": "hello world"}
```

Congratulations, you just deployed your Serverless app using SAM-CLI. Go to AWS Console and navigate to Lambda and see the app configuration. 

## Delete the App
Now you have created the app but SAM CLI doesn't provide you a method to delete it. For this we will use AWS CLI.

```bash
aws cloudformation list-stacks
```

Stack is basically a cloudformation term for a combination of resources that you've created together. This should list all of your stacks + 2 more (one for AWS CLI and another for SAM CLI). 

Notice the first one has `StackName` value as  `sam-app`, we will now delete this. 

> If you get stuck press 'q' to exit. 

Now runt he below to delete the app
```bash
aws cloudformation delete-stack --stack-name sam-app
```

## Confirm deletion

List all your stacks again

```bash
aws cloudformation list-stacks
```

You should still see the stack sam-app, however, if notice the `StackStatus` key, this should be `DELETE_COMPLETE`.

---

# What is Anaconda for Python?

URL: https://cloudbytes.dev/snippets/what-is-anaconda-for-python
Category: Snippets
Published: 2021-08-01
Author: Rehan Haider
Tags: python

> A short introduction to Anaconda for Python, how to configure and use Anaconda.

Anaconda is a Python distribution platform. What that means is Anaconda takes the core Python and packages it along with some utilities pre-configured targeted at data science / machine learning applications. 

This makes it easy for developers who are building applications that rely on many popular data science python packages such as Numpy, Pandas, etc. to start working as soon as Anaconda is installed. 

## Why is Anaconda needed?

If you happen to download and install Python from [python.org](https://python.org) website, by default it comes with only the standard python libraries. 

So developers need to download additional libraries using tools such as `pip` or similar package managers. And if you have too many of them, the packages many a times conflict with each other. Python tries to solve this by using [venv]({filename}99999999-create-a-python-virtual-environment.md) which is largely a command line utility. 

An alternative approach is using Anaconda which has a GUI and its `conda` package manager. But the key appeal of Anaconda is its dependency management whereby for any environment, it will ensure there are not package conflicts. 

Additionally, Anaconda comes with multiple additional SDKs bundled together to make development easier such as Jupyter, SciPy, etc. 

![Anaconda bundle](/images/99999985-anaconda.png)

## How to install Anaconda

To install Anaconda, just go to its [Individual Edition](https://www.anaconda.com/products/individual) page and download it for your OS then run the executable to install.

Anaconda Individual Edition is free of cost but is limited for use by individuals and not organisations. For organisations, there are several other editions which have their own license fee.

---

# Python For Loops, using range vs enumerate

URL: https://cloudbytes.dev/snippets/python-for-loops-using-range-vs-enumerate
Category: Snippets
Published: 2021-07-31
Author: Rehan Haider
Tags: python

> How to use python for loops, using range vs enumerate

Python for loops are different to the conventional programming languages in a key aspect, its syntax doesn't use an iterator. 

But that is a feature not a bug, the '*Pythonic*' way to write for loop is to run it over a range or a list by item and not by using an index to refer to the next element like C or Java. E.g.

```python
fruits = ['apple', 'orange', 'banana', 'tomato', 'cucumber']

for fruit in fruits:
    print(fruit)
```

Will print out
```bash
apple
orange
banana
tomato
cucumber
```

If we wanted to do the same in C, not withstanding the complexities, it would look like,
```c
for (int i = 0, i < 5, i++)
{
    printf("%s", fruits[i])
}
```

## Range and Enumerate functions

But there are many scenarios where you might need to iterate using index. For such cases, Python has two in-builts functions `range()` and `enumerate()` that provides this feature. 

### range()

Range is used to iterate over a sequence of numbers, e.g. to print 0 - 4, 

```python
for i in range(5):
    print(i)
```

We can use this to iterate over out fruits list with an iterator

```python
fruits = ['apple', 'orange', 'banana', 'tomato', 'cucumber']

for i in range(5):
    print(fruits[i])
```

But you now have a new problem, in the first example, we didn't have to worry about the length of the list, with range you do. You can still solve it by usng `len()` method

```python
fruits = ['apple', 'orange', 'banana', 'tomato', 'cucumber']
for i in range(len(fruits)):
    print(fruits[i])
```

### enumerate()

Instead of calculating the length and iterating over the list, we can also use enumerate() to get the same results.

```python
fruits = ['apple', 'orange', 'banana', 'tomato', 'cucumber']
for i, item in enumerate(fruits):
    print(f"Using iterator: {fruits[i]}")
    print(f"Using item: {item}")
```

## Range vs Enumerate: What should you use? 

As with most things, the answer is, it depends! And more often than not, it will end up being a personal choice. 

But from a performance perspective, we can test it. 

### Performance testing range() and enumerate()

Let's start by setting up a baseline. We will generate a list with 10000 integers, and then compute if each one of them are prime number of not and add them to an output list. 

#### Baseline: Simple iteration
```python
from time import time

# start the time counter
start = time()

def prime(num):
    '''Function to compute if a number is prime'''
    for i in range(2, int(num/2)+1):
        if num % i == 0:
            return False
    return True

# Generating a list of integers
n = 9999
inputs = [i for i in range(n)]
outputs = []

for number in inputs:
    outputs.append(f"{number} is prime? {prime(number)}")

end = time()

print(end - start)
```

For me it took ~44.6 secs

#### Using range()

```python
from time import time

# start the time counter
start = time()

def prime(num):
    '''Function to compute if a number is prime'''
    for i in range(2, int(num/2)+1):
        if num % i == 0:
            return False
    return True

# Generating a list of integers
n = 9999
inputs = [i for i in range(n)]
outputs = []

for i in range(len(inputs)):
    outputs.append(f"{inputs[i]} is prime? {prime(inputs[i])}")
    
end = time()

print(end - start)
```

Took roughly ~47.7 seconds

### Using enumerate()

```python
from time import time

# start the time counter
start = time()

def prime(num):
    '''Function to compute if a number is prime'''
    for i in range(2, int(num/2)+1):
        if num % i == 0:
            return False
    return True

# Generating a list of integers
n = 9999
inputs = [i for i in range(n)]
outputs = []

for i, num in enumerate(inputs):
    outputs.append(f"{i} is prime? {prime(num)}")
    
end = time()

print(end - start)
```
This took about ~45.76 seconds


## Conclusion

If you're iterating over a list, enumerate is probably the most optimal option.

---

# How to Install, Check Python Version and Update it

URL: https://cloudbytes.dev/snippets/how-to-install-check-python-version-and-update-it
Category: Snippets
Published: 2021-07-25
Author: Rehan Haider
Tags: python

> A simple guide on how to Install Python, how to check Python version installed and upgrade Python to the latest version

[TOC]

Python is one of the easiest languages to learn and has powerful extensibility which makes it the most popular programming loved by beginners, data-scientists, academics, and web-developers. 

![Python popular programming language](/images/99999987-top-languages.png)

Part of the reason on Python's extensibility is the opensource nature of the language where community developes features, libraries, and packages. 

But that comes at the cost of [several distributions](https://wiki.python.org/moin/PythonDistributions) apart from the Official one available from [Python.org](https://python.org), each one of them optimised for different purpose, e.g.

1. **The Official CPython distribution**: The most authentic version of Python implemented in C for best performance
2. **PyPy**: Python implemented using Python, best known for its JIT
3. **Anaconda**: Targeted towards data scientists focused on resolving package conflicts 
4. **IPython**: An interactive implementation of Python upon which Jupyter and other interpreters are based upon. 

## 1. Check the version of Python installed

On Windows/Linux/MacOS, run the below command to print the version of your Python installed on the system

```powershell
python --version
```

!!! note "On some Linux / MacOS distributions where both Python 2 & Python 3 are installed, you will need to identify which one you want to check individually by  ```python3 --version```"

If you do not get the version, it means Python is not installed. To install Python, follow the instructions below.


## 2. How to install Python? 

So when you ask the question, "How to install Python", you first need to ask the question "Which version of Python to install?".

The answer, in most cases is the [Official version of Python](https://www.python.org/downloads/). 

!!! warning "Do not install Python from Microsoft Store as adding it to PATH can be problematic due to the the folder structure followed by Microsoft."

### Install the latest Python On Windows

Just go to the [downloads section on Python.org](https://www.python.org/downloads/), click on `Download Python 3.x.x` to download the latest installer. 

After that run the installer, on the first screen make sure to select "Add Python 3.x to PATH". After that you can use default selections and follow the instructions to install. 

### Install the latest Python on Linux / MacOS

You don't need to. On most Linux & MacOS distributions, Python is installed by default. 

But if you want the latest versions, follow the below instructions.

On Mac, go to [downloads section on Python.org](https://www.python.org/downloads/mac-osx/) and download the installer for the latest stable release. Run the installer and follow the instructions to install.

On Linux, just run the below (with major version) to install the latest, e.g. to install Python 3.8 run
```bash
sudo apt-get install python3.8
```
> Python version on Linux repositories are usually behind one version 


## 3. Upgrade Python version

On Windows & Mac you need to download and install the latest version again from [Python.org](https://python.org/downloads) Website. 

On Linux run the below the get the latest supported version by the Operating System.

```bash
sudo apt-get update
sudo apt-get upgrade
```

---

# Convert a Pelican Website to PWA using Workbox

URL: https://cloudbytes.dev/snippets/convert-a-pelican-website-to-pwa-using-workbox
Category: Snippets
Published: 2021-07-24
Author: Rehan Haider
Tags: pelican, javascript, python, pwa

> A guide to converting a satic website generated by Pelican into a Progressive Web App using Google's Workbox library

[TOC]

There has always been 2 types of apps since internet was invented

1. Apps that ran on the desktop (Windows / Linux / Mac) that was developed using a variety of programming languages (C/C++, Java, etc.)
2. Apps that ran in browsers, typically built with HTML with 

With smartphones entering the fray, this quickly started to change with native Android and iOS apps being made using typically different technologies, e.g. Java/Kotlin for Android and Objective-C/Swift for iOS. 

While native smartphone apps were able to leverage the smartphone features such as camera, GPS, and other sensors. Web apps were limited in this regard. 

This also meant most app developers had 3 codebases, one for Android, one for iOS, and one for web. Several attempts has been made to simplify this and make a "cross platform development framework", such as [React Native](https://reactnative.dev/), [Ionic](https://ionicframework.com/), Xamarin, etc. 

Except the recently launched [Flutter](https://flutter.dev/), all of them faced the same problem, apps were trans-compiled into native app format resulting in performance loss due to an abstraction layer. 


## Introducing the PWA
The concept of PWA has been around since the release of iPhone, but the current incarnation was only built in around 2015 by engineers at Google working on Chrome. 

PWAs are called "progressive" because they can take advantage of new features supported by modern browsers and attempts to solve one of the biggest challenges in modern app development, cross-platform apps with native features and performance. 

![PWA Native Cross-platform](/images/99999988-pwa-native.png)

Static Websites generated using SSGs such as Pelican, Hugo, Gatsby, etc. are perfect candidate for conversion to PWA simply because they are easiest to convert. 

## Prerequisites

The only prerequisite is that a "Service Worker" definition that is a JavaScript file. But this file needs to be in the root of the website, i.e. `"/"` otherwise its scope will be limited. 

*Step 1*: Create a file named `SW.js` in your `content/extras` folder and add the below content in the file

```javascript
//extras/SW.js
console.log("I'm you service worker")
```

*Step 2*: To ensure this file is stored in the root folder of your website, edit the `pelicanconf.py` and add the following line 

```python
STATIC_PATHS = ["images", "extra/SW.js"]

EXTRA_PATH_METADATA = {"extra/SW.js": {"path": "SW.js"}}
```

This will first establish `SW.js` as a static file and then add instructions to copy it from `extras` to `root`. by using `EXTRA_PATH_METADATA`. 

## Create the PWA manifest & Add the icons
The manifest is a JSON file that outlines how your PWA will behave. Create a file named `site.webmanifest` and place it under your static / asset folder. 

The content of the manifest is quite self-explanatory and should contain
```json
{
    "name": "<your website name>",
    "short_name": "<your website short name>",
    "icons": [
        {
            "src": "img/android-chrome-192x192.png",
            "sizes": "192x192",
            "type": "image/png"
        },
        {
            "src": "img/maskable_icon.png",
            "sizes": "512x512",
            "type": "image/png",
            "purpose": "any maskable"
        }
    ],
    "start_url": "/",
    "display": "standalone",
    "orientation": "portrait",
    "background_color": "#ffffff",
    "theme_color": "#ffffff"
}
```
The above files also identifies the icons that should be used. Place the icons with the above names and sizes in your `image` folder. You can use [maskable.app](https://maskable.app/) to create one. 

### Add the manifest link to your tempalte
By adding the below line (make sure to change the path as appropriate)
```html
<link rel="manifest" href="/assets/site.webmanifest">
```

### Test your manifest
Open your website in either Chrome or Edge browser, press F12 or go to Developer Tools, and then click on Application Tab. If you've configured the manifest correctly, you shold see something similar to the below

![PWA Manifest](/images/99999988-manifest-dev-tools.png)


## Register the service worker
This has to be in your main theme JavaScript application file. E.g. in CloudBytes's case the `app.js` is stored under `//assets/js` folder. If you view source, you should see an HTML header that links this

```html
<script defer src="/assets/js/app.js"></script>
```

> If you don't already have it, create an `app.js` as per above and add the header in your Pelican layouts. 

In the `app.js` add the below snippet to register your service worker file

```javascript
// #app.js
// Check that service workers are supported
if ('serviceWorker' in navigator) {
    // Use the window load event to keep the page load performant
    window.addEventListener('load', () => {
        navigator.serviceWorker.register('/SW.js');
    });
};
```
### Test the Service Worker configuration
If you have configured it correctly, go to the `Service Worker` in section in Application tab of the developer tools, if there are no errors it means your service worker is configured correctly. 

Additionally, in the Dev Tools Console Tab, your should see the message
```
I'm you service worker
```
If you got this, your service worker is registered successfully, but it still needs configuration.

## Configure the Service Worker

Right now the service worker isn't doing anything except print to console, we need to configure the service worker to cache requests, and static files. 

To do that we will use [Workbox](https://developers.google.com/web/tools/workbox), a library developed by Google to make it easier to create PWAs. 

### Caching Pages as they are visited
Change the contents of `SW.js` to the below

```javascript
importScripts('https://storage.googleapis.com/workbox-cdn/releases/6.1.5/workbox-sw.js');

// Cache page navigations (html) with a Network First strategy
workbox.routing.registerRoute(
    // Check to see if the request is a navigation to a new page
    ({ request }) => request.mode === 'navigate',
    // Use a Network First caching strategy
    new workbox.strategies.NetworkFirst({
        // Put all cached files in a cache named 'pages'
        cacheName: 'pages',
        plugins: [
            // Ensure that only requests that result in a 200 status are cached
            new workbox.cacheableResponse.CacheableResponsePlugin({
                statuses: [200],
            }),
        ],
    }),
);
```
This snippet makes sure that whenever a page is requested, it first tries to download it from internet, if the service worker if not able to fetch a response, it will serve the page from the cache it has built. Additionally, it caches pages only if they are successfully fetched. 

### Cache static files

Add the below snippet to enable caching of stylesheets and scripts. 


```javascript
// Cache CSS, JS, and Web Worker requests with a Stale While Revalidate strategy
workbox.routing.registerRoute(
    // Check to see if the request's destination is style for stylesheets, script for JavaScript, or worker for web worker
    ({ request }) =>
        request.destination === 'style' ||
        request.destination === 'script' ||
        request.destination === 'worker',
    // Use a Stale While Revalidate caching strategy
    new workbox.strategies.StaleWhileRevalidate({
        // Put all cached files in a cache named 'assets'
        cacheName: 'assets',
        plugins: [
            // Ensure that only requests that result in a 200 status are cached
            new workbox.cacheableResponse.CacheableResponsePlugin({
                statuses: [200],
            }),
        ],
    }),
);
```

### Cache Images

Add the below snippet to cache images that are successfully fetched. 

```javascript
// Cache images with a Cache First strategy
workbox.routing.registerRoute(
    // Check to see if the request's destination is style for an image
    ({ request }) => request.destination === 'image',
    // Use a Cache First caching strategy
    new workbox.strategies.CacheFirst({
        // Put all cached files in a cache named 'images'
        cacheName: 'images',
        plugins: [
            // Ensure that only requests that result in a 200 status are cached
            new workbox.cacheableResponse.CacheableResponsePlugin({
                statuses: [200],
            }),
            // Don't cache more than 50 items, and expire them after 30 days
            new workbox.expiration.ExpirationPlugin({
                maxEntries: 50,
                maxAgeSeconds: 60 * 60 * 24 * 30, // 30 Days
            }),
        ],
    }),
);

```

## Test your PWA
After you have configured the above, open your website in using Chrome browser in your smartphone. You should get a small banner at the bottom asking if you want to add your website to the homescreen. Clicking on it will install your website like an app.

---

# How to use Git and GitHub to contributed to Open Source Projects

URL: https://cloudbytes.dev/snippets/how-to-use-git-and-github-to-contributed-to-open-source-projects
Category: Snippets
Published: 2021-07-24
Author: Rehan Haider
Tags: github

> A guide to Git and GitHub basics, and using them correctly to contribute to open source projects on GitHub

[TOC]

If you're a budding developer Git and GitHub is going to be your best friend and your worst enemy that you cannot get away from. So, it's better to learn what it does and how it does. 

Jokes aside, Git is a kind of source-code version control tool, i.e. if multiple people are working on the same application and developing different modules, Git / or any other version control system is used to ensure the parallel work does not conflict with each other. 

## Git Vs GitHub

Git specifically is an opensource tool developed by Linus Torvalds (the same guy to made Linux Kernel) around 2005, and is the de-facto version control tool that is used nowadays. 

GitHub on the other hand uses this open source tool to host code repositories that can be used by people like you and me. 

Git can also be connected with mutliple other respository hosting providers such as GitHub, GitLab, Bitbucket etc. 

## Install & Configure Git
Git can be downloaded and installed from the [Git-SCM website](https://git-scm.com/downloads).
Register on GitHub with your email and note your username, e.g. my username on GitHub is [`rehanhaider`](https://github.com/rehanhaider)

Then set your username in Git on your system by running
```bash
git config --global user.name "<your username>"
```

Next, set your commit email ID, this should match your GitHub ID. 
```bash
git config --global user.email "email@example.com"
```

Finally, follow [this guide on GitHub to cache your credentials](https://docs.github.com/en/get-started/getting-started-with-git/caching-your-github-credentials-in-git) so you don't have to enter your username / password everytime you use GitHub. 

### Git / GitHub basics
There is a lot of things you can do with Git, but we'll keep the focus on actions that you will encounter on a day to day basis. 

One key concept to keep in mind is that your system keeps a "*local*" copy of a repository that needs to be synced with the "*origin*" i.e. GitHub server. 

A typical Git workflow looks like below.

![Git / GitHub workflow and branches](/images/99999989-git-workflow-svg.svg)

Where the "*main*" branch is usually not changed directly and each node is a "*commit*". Developers create a branch, make their changes, and then create a "*pull request*" or *PR* to ask the owner of main branch to review their changes and merge the code. 

### Create a new repository
You either create a new repository on GitHub directly by clicking on the ➕ sign on top right, and choosing new repository. 
![Create a GitHub Repo](/images/99999989-create-a-repo.png).

### Fork a repository
Most of the times, you don't want to start from scratch and instead build on top of already existing and opensource tools / software. E.g. if you want to build your own copy of [CloudBytes](https://cloudbytes.dev), you can do so by "forking" the publicly available [CloudBytes source code on GitHub](https://github.com/CloudBytesDotDev/CloudBytes.dev).

To fork a repository, go to [CloudBytes source code on GitHub](https://github.com/CloudBytesDotDev/CloudBytes.dev) or any other repo that you want to fork and click on the fork button on top right. 

### Clone a repository
Now a copy or fork of the repository has been created on GitHub, but you need to create a local copy for you to be able to make changes. This is called cloning, e.g. to clone the forked copy of CloudBytes repo

```bash
git clone https://github.com/<your-username>/CloudBytes.dev.git
```

### Create a Branch
To create a new branch and checkout (switch over) to it, run the below


```bash
git checkout -b <branch-name>
```

### Stage the changes
Save your changes in the staging area by running the below command

```bash
git add . 
```
This will start tracking any new file that you may have created and save all the changes you have made. 

### Commit the changes
So far your changes are only saved in staging area, i.e. they haven't been added to your branch. To add your changes to the branch run

```bash
git commit -m "Describe your changes"
```

### Push the Changes
Now your changes are *committed* but they are only available on your local system and not on GitHub. This will require you to *push* your code to GitHub (or *origin*)

If you're pushing your changes in the current branch for the fist time, then run

```bash
git push --set-upstream origin <branch-name>
```

After the first time, you can just run `git push` to push your changes to GitHub.

## Pull Requests
Now your code is merged into your own branch, but not in the "upstream" branch that is the *main* branch. To do that you need to create a pull request with your changes. That can be done on GitHub by visiting your *forked repo*, clicking on the branches, find your branch in the list and click on New Pull request. 

![Create a pull request](/images/99999989-create-a-pull-request.png)

### Merging the changes
Now the owner of the repository will get a notification that there is an open *PR*. They can review the changes and decide to merge it into the *main* branch or send it back for further changes.

---

# Add a Table of Contents using Markdown in Pelican

URL: https://cloudbytes.dev/snippets/add-a-table-of-contents-using-markdown-in-pelican
Category: Snippets
Published: 2021-07-18
Author: Rehan Haider
Tags: python, pelican, markdown

> A guide to customising Markdown to add table of contents in Python Pelican

I've spoken about how [Pelican documentation is incoherent]({filename}99999998-fixing-pelican-sitemap-error-on-google-search-console.md). One of the key features that Pelican keeps hinting towards but never explains in detail is how to customise the Markdown output. 

Pelican uses [Pygments](https://pygments.org/) as its code syntax highlighter, something that they mention several times. And Pelican uses [Python-Markdown](https://python-markdown.github.io/) to convert the Markdown to HTML, again something that is not explicitly mentioned just hinted at. 

**So how do you customise Markdown output?**
By using the official [Table of Content](https://python-markdown.github.io/extensions/toc/) extension from Python-Markdown. A key feature, not even hinted at in the Pelican Documentation. 


## How to add a TOC to Pelican?

Pelican uses `MARKDOWN` dictionary to store the configuration you want to use for `Python-Markdown`. To add a TOC, append this snippet in your `pelicanconf.py` 

```python
MARKDOWN = {
    "extension_configs": {
        # Needed for code syntax highlighting
        "markdown.extensions.codehilite": {"css_class": "highlight"},
        "markdown.extensions.extra": {},
        "markdown.extensions.meta": {},
        # This is for enabling the TOC generation
        "markdown.extensions.toc": {"title": "Table of Contents"},
    },
    "output_format": "html5",
}
```

After that, you just need to add the shortcode `[TOC]` in  your markdown file where you want to inser the TOC. 

This will do two things

1. Add the table of contents to the output of your article
2. Add the class name `toc` to the table of contents that you can format using CSS

## How is the Table of Content built?

After adding `[TOC]` in your markdown document, it is replaced by the nested list of headers in you documents, e.g., 

```md
[TOC]

# 1 Main header
Content under main header
## 1.1 Secondary header
COntent under secondary header
```

will be replaced by 

```html
<div class="toc">
  <ul>
    <li><a href="#1-main-header">1 Main header</a></li>
      <ul>
        <li><a href="#11-secondary-header">1.1 Secondary header</a></li>
      </ul>
  </ul>
</div>
<h1 id="1-main-header">Header 1</h1>
<p>Content under main header</p>
<h2 id="11-secondary-header">Header 2</h2>
<p>Content under secondary header</p>
```

You can use a bit of CSS to format your TOC by adding the below to your CSS stylesheets
```css
.toc {
    border-radius: 0.5em;
    margin-bottom: 1em;
    background: #222831;
    padding: .5em;
    margin-top: 1em;
    top: 30px;
    box-shadow: rgba(0, 0, 0, 0.7) 0px 10px 20px 0px;
}

.toc ul {
    list-style: none;
    padding: 0.5rem 1rem;
    margin: 0;
}

.toc ul li {
    padding: .25em;
}

.toc ul li a {
    color: #498afb;
    font-weight: 500;
    transition: color .4s;
}

.toc ul li a:hover {
    color: #9166cc;
    transition: color .4s;
    border-bottom: 1px solid #9166cc;
}

.toc ul li ul {
    font-size: .75em;
    font-weight: 500;
    margin-left: 5px;;;;;
}

.toc ul li ul a {
    color: #b2becd;
}
```
This is how you add a table of contents to a markdown document in Pelican.

---

# Clean URLs in Pelican Sitemap using Python

URL: https://cloudbytes.dev/snippets/clean-urls-in-pelican-sitemap-using-python
Category: Snippets
Published: 2021-07-17
Author: Rehan Haider
Tags: python, pelican

> Learn to use Python to remove '.html' extensions and clean URLs in sitemaps generated by Pelican

A common problem that I faced using [Jamstack]({filename}99999996-what-is-jamstack.md) static site generators is having clean URLs. 

## What is Clean URL? 

A "Clean URL" is basically page address that doesn't have any extension such as `.html`, `.php` or trailing slashes `/` at the end of the URL. 

You run into challenges ranging from: 

1. Lack of full support for clean URLs in the Static Site Generator (SSG). E.g. typically Clean URLs are configured by using a property of webservers and browsers where by default any webserver will return `index.html` inside the folder if you access the folder directly. 
2. If you configure clean URLs by using the technique above, the URLs withh have trailing `/` slashes and "pages" will continue to have `.html`. 
3. If you use URL rewrite rules on the hosting webserver (e.g. Firebae provides a very easy to configure method to generate clean URLs), your SSG generated sitemap will be incorrect and contain `.html`

So, you basically end up between a rock and a hard place. 

### Why are Clean URLs important? 
There are two related aspects. 

1. **Clean URLs look better**. Take for example the URL of this page `https://uberpython.com/articles/clean-urls-in-sitemap-using-python`. You can understand from the URL that this is an article and the topic. With a `.html` at the end, nothing will change but it just looks ugly. 
2. **Clean URLs are better for SEO**: Widely known fact is because Clean URLs look better and increases the accessibility of a page, thus also increases the SEO score of a page

## How to create a Clean URLs? 

### Clean URLs in Firebase Hosting
The best method that I found is to use URL rewrite rules to generate Clean URLs, e.g. in if you're using Firebae Hositng, change the `firebase.json` to the below

```json
{
  "hosting": {
    "public": "output",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ],
    "cleanUrls": true,
    "trailingSlash": false
  }
}
```
That's all it takes, Firebase takes care of all the internal configuration in the webserver. 

## Solving the Sitemap '.html' Problem

After you have configured clean URLs in your webserver, if you don't correct your Sitemap, Google & other search engines will struggle to crawl and index your website correctly and that can have negative impact on your search engine rankings. 

To solve this, we will do the following

1. Let the SSG generate the sitemap that includes `.html` extension
2. Run a small Python snippet to replace that

Create a file named `fix_sitemap.py` witht he below contents and place it in your Pelican root directly (the folder where you have `publishconf.py`)

```python
# fix_sitemap.py
def fix_sitemap():
    try:
        with open("output/sitemap.xml", "r") as file:
            original = file.read()
            corrected = original.replace(".html", "")
    except Exception as e:
        print(f"Opening sitemap failed with error: {e}")

    try:
        with open("output/sitemap.xml", "w") as file:
            file.write(corrected)
    except Exception as e:
        print(f"Saving sitemap failed with error: {e}")
```
A rather simple solution where we simplly searched for all `.html` and removed it from the sitemap. No XML parsing, no tree navigation. 

Now go your Pelican's configuration in `publishconf.py` and add these lines right at the bottom
```python
from fix_sitemap import fix_sitemap
fix_sitemap()
```

This will ensure everytime you publish your website, the sitemap is updated correctly.

---

# Build WebAPIs with Python using Flask & FastAPI

URL: https://cloudbytes.dev/books/build-webapis-with-python-using-flask-fastapi
Category: Books
Published: 2021-07-15
Author: Rehan Haider
Tags: python, fastapi, flask, heroku, github, jupyter

> Beginner's guide to building and interacting with APIs using Python Flask and FastAPI

A hands-on project based guide to building APIs designed for beginners who have never built an API before or professionals who want a quick intro to FastAPI or Flask.

The book uses Python libraries such as Flask microframework that is used by the likes of Netflix, Airbnb, Uber, Instagram, etc. making its way up to modern framework like FastAPI, which is on par to any with NodeJS, and Go in terms of performance and quickly being adopted as the #1 API tools written in Python.

You will also learn efficient routing, type-hinting, data transfer, HTTP messages, form data handling, REST API design, and data validation techniques.

In this book you will learn

1. Fundamentals of APIs
2. Introduction to tools for API development
3. Flask backend development using REST architecture
4. Connect to Front-end designed using Jinja2 templates
5. FastAPI backed / API design
6. Asynchronous API development

Get it today at [Amazon](https://www.amazon.com/dp/B09BJLKM6F) or [Kindle](https://www.amazon.com/dp/B09BJLKM6F) !

---

# Auto Deploy Pelican Websites to Firebase Hosting

URL: https://cloudbytes.dev/snippets/auto-deploy-pelican-websites-to-firebase-hosting
Category: Snippets
Published: 2021-07-13
Author: Rehan Haider
Tags: firebase, python, pelican

> A step by step guide to installing and configuring Pelican and hosting the final blog on Firebase Hosting

[TOC]

In a previous article I wrote about [how you can host a simple Pelican static website on Github Pages]({filename}99999995-automate-pelican-github-pages.md) and automate the deployment process using Github Action. 

GitHub Pages is brilliant and extremely useful for a simple blog or small webites, but quickly becomes limited in features if you're trying to build anything serious. 

For starters, if you recall from the [Jamstack explanation]({filename}99999996-what-is-jamstack.md) and its principles, you rely on third party APIs. E.g. if you want to build a user management system into your website you will need to use a Auth API from a third party such as Okta, Azure, or AWS. 

This is where Firebase has a massive advantage by providing an integrated end to end development framework. Thus if you want to build more dynamic features into your Pelican / Jamstack website, you may want to use Firebase for your hosting purposes. 

In this guide, we will discuss how to setup Continous Deployments (CD) to Firebase Hosting so that your changes are deployed automatically on `git push`. 

## Getting started

After discovering VSCode devcontainers, I've just stopped using Python's virtual environment. So we will use devcontainers to clone the repository and prepare the development environment. 

So in this tutorial, we will

1. Set up the Pelican development environment insite a container and make a simple webpage using the default theme
2. Create a Firebase project that will be used to host the website
3. Create GitHub secrets that will be used to deploy the Pelican output to Firebase Hosting
4. Then create the action to deploy automatically to Firebase Hosting
5. Push the codebase back to GitHub repository & watch the fun unfold

## The workflow

We will use the following setup & automation to automate the deployment process.
![pelican ci cd](/images/99999992-pelican-ci-cd.png)

## 1. Setting up Pelican 
Use the instructions in [this guide on how to install Pelican in a VSCode devcontainer]({filename}99999993-install-pelican-in-devcontainer.md) and create a small blog. 

Then capture your dependencies by running

```bash
pip freeze > requirements.txt
```

## 2. Create & configure the Firebase Project
Visit the [Firebase Console Home](https://firebase.google.com/) page and register for an account, or sign-in if you already have an account. After that, click on "*Create a Project*". 

![Create a firebase project](/images/99999992-firebase-create-project.png)

Give your project a name and then follow the instructuions to complete the setup.

After that you need to create two filê in workspace

`.firebaserc`: Contains the project list and aliases. If you open it you would see something like

```json
{
  "projects": {
    "default": "<your project name>"
  }
}
```

Instead of "cloudbytes-prod" you should see the project you chose during the configuration.

`firebase.json`: Contains the configuration of your services, 

```json
{
  "hosting": {
    "public": "output",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ]
  }
}
```

## 3. Create GitHub Secrets

[GitHub Secrets](https://docs.github.com/en/actions/reference/encrypted-secrets) is GitHub's Key Management System (KMS) that excrypts and stores API keys so it can be used in your projects without being visible to anybody, even you. 

We need to store the credentials to Firebase Hosting Project as a secret in GitHub so it can be used to push your code directly to Firebase. 

To do that 
### 3.1 Get the Service Account JSON

Go to Firebase -> Select your Project -> Click on the ⚙ Settings icon on the left navigation page, then click on *Project Settings*. 

Then click on the *Service accounts* tab, this will create a Service Account for your project. Service Account credentials are used to interact with Firebase Services. 

Scroll to the bottom -> Click on *Generate new private key* then in the popup box, click again on *Generate key*. 

You will be prompted to save the file ending in `.json` extension. 


!!! danger "WARNING: Never share or upload this service account credentials including in GitHub repository"

The right way to handle such credentials is via encrypted secrets. 

### 3.2 Store the secret in GitHub Secrets

Open you GitHub repository and on the `Settings` tab, scroll down and click on `Secrets` in the navigation pange on left. 

Then click on `New repository secret` button on the top right. Give it any memorable name, e.g. `FIREBASE_SERVICE_ACCOUNT` and paste the contents of the service account file that you download in previos section then click `Add secret` to save. 

![Github Repository Secret](/images/99999992-github_repository_secret.png)

## 4. Create the action to deploy automatically to Firebase Hosting

Now we have all the building blocks ready, except the GitHub action definition. In VSCode, create a file at the path `.github/workflows/deploy-to-firebase.yml`.

Add the following content in the file
```yaml
name: Deploy

on:
  push:
    branches:
      - main

jobs:
  build_and_deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: rehanhaider/pelican-build-action@v0.1.11
      env:
        PELICAN_CONFIG_FILE: app/publishconf.py
        PELICAN_CONTENT_FOLDER: app/content
    - uses: FirebaseExtended/action-hosting-deploy@v0
      with:
        repoToken: '${{ secrets.GITHUB_TOKEN }}'
        firebaseServiceAccount: '${{ secrets.FIREBASE_SERVICE_ACCOUNT}}'
        channelId: live
        projectId: <enter your project id>
```

YAML is a declarative syntax where it is easy to understand what is going on. 
In the first part above, we define that the action will work `on` `push` into the `main` `branch` of your GitHub repository. 

Then we created a `job`, and named it `build_and_deploy` that will run on `ubuntu-latest` version of operating system. 

After that we defined the `steps` that need to be followed which are, 

1. Use `actions/checkout@v2` to access the branch of your repository which contains your code
2. Use another action that I built named `rehanhaider/pelican-build-action@v0.1.11` that will install all of your dependencies stored in `requirements.txt` and execute the `pelican content` command to generate the output in a folder named `output`
3. The above action is performed using the environmental variables under `env` that contains the path to the config file and the content folder where the markdown content is stored. 
4. Finally, we use official Firebase action to deploy the contents of `output` folder that the action will create into Firebase hosting 
5. The above action will be performed with a `repoToken` that is provided by GitHub automatically, the `FIREBASE_SERVICE_ACCOUNT` credentials that we stored in previous section, into the `live` channel and finally the `projectId` that you should change to the name of the project you had created on Firebase

## 5. Push the Code to GitHub and watch the fun unfold

Open the VSCode terminal and run the below command to add all your files to your GitHub repository tracking
```bash
git add .
```

Commit your changes to the repository by running
```bash
git commit -m "My cool comment"
```

Then push your code to the GitHub by running
```bash
git push
```

Now go to GitHub.com and browse to your repository page, click on `Actions` tab you will see the action being executed. Click on the action to see more details. 

![Github action success results](/images/99999992-github_action_results.png)

Now your Pelican Blog is setup for auto deployment to firebase, everytime you push your code.

---

# Beginner's guide to installing Pelican in a Container

URL: https://cloudbytes.dev/snippets/beginners-guide-to-installing-pelican-in-a-container
Category: Snippets
Published: 2021-07-13
Author: Rehan Haider
Tags: python, pelican, vscode

> Step by step guide on how to install and configure Pelican in a Docker Container for development for developement os static websites following Jamstack architecture

Pelican is a type of Static Site Generator (SSG) written in Python that follows the Jamstack architecture pattern. It is easily the most popular Python based SSG with Nikola and Lektor coming in a distant second and third. 

**Pelican's popularity is largely due to the fact that**<br>
a. It is written in Python<br>
b. It has a rich ecosystem of plugins and themes<br>
c. It uses Jinja2 as templating language, making it extremely easy to build your own theme<br>
d. It is extremely stable, having been around for almost a decade and well documented<br>
e. It is actively maintained by close to 400 contributors

![Pelican active maintained github](/images/99999993-pelican_github_activity.png)

## Getting Started

First create your GitHub repository that will host the codebase. Then

1. Fire up VSCode and press `Shift+Ctrl+P` to bring up the command palette. <br>
2. Search for *"Clone Repository in Container Volume"*, then follow the steps to select the repository you want to clone. <br>
3. When asked to choose a container configuration, select *"Show all definition"* and search for Python 3, <br>
4. then select Python 3.9 from the dropdown. <br>
5. `[OPTIONAL]` Also, choose to install NodeJS as well by selecting the checkbox when promtpted. 

> While we don't need NodeJS for Pelican, you will need NPM (which gets installed together as a bundle) in future to work directly with Firebase. However, this is an optional step and not required for this tutorial. 

If the container was created you would see a folder named `.devcontainer` created in VSCode explorer pane. This folder contains two files, a `Dockerfile` that contains the configuration of your docker and a `.devcontainer.json` that stores the preferences for the VSCode workspace you're in currently. 

You should now have a devcontainer configured. 

Open the VSCode terminal, you will notice you're inside a Linux machine. To verify, run
```bash
cat /etc/*-release
```

This will display something similar to the below
```bash
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```

## Installing Pelican in the devcontainer

In VSCode, bring up your terminal and then install Pelican by running

```bash
pip install "pelican[markdown]"
```

This will install pelican and all its dependencies including support for markdown

### Step 3: Setup Pelican

In the terminal type the below to trigger Pelican setup

```bash
pelican-quickstart
```

This starts the interactive setup where you are asked a series of question, choose as per below 👇🏽 (blank means leave the default by pressing Enter)

```none
> Where do you want to create your new web site? [.] 
> What will be the title of this web site? MyPelicanWebsite
> Who will be the author of this web site? Me
> What will be the default language of this web site? [en] 
> Do you want to specify a URL prefix? e.g., https://example.com   (Y/n) N
> Do you want to enable article pagination? (Y/n) Y
> How many articles per page do you want? [10] 
> What is your time zone? [Europe/Paris] 
> Do you want to generate a tasks.py/Makefile to automate generation and publishing? (Y/n) 
> Do you want to upload your website using FTP? (y/N) 
> Do you want to upload your website using SSH? (y/N) 
> Do you want to upload your website using Dropbox? (y/N) 
> Do you want to upload your website using S3? (y/N) 
> Do you want to upload your website using Rackspace Cloud Files? (y/N) 
> Do you want to upload your website using GitHub Pages? (y/N) 
Done. Your new project is available at /workspaces/myWorkingFolder
```

We haven't used the default GitHub Pages setting because it only publishes the output but doesn't automate the process of building.

You content is stored in the "content" folder, create a file named `my-first-post.md` in that folder with the below contents

```markdown
Title: My First automated blog post
Date: 2021-07-10
Category: Snippets
Tags: github, python, pelican
Author: Me
Summary: A guide to configuring automated Continuous Deployment (CD) of \
static site generated by Pelican SSG from GitHub repository to GitHub pages using GitHub Actions

## This is my first blog post
And it looks wonderful!
```

## Test your Pelican website

To generate and serve your Pelican Website, run the below command

```bashs
make devserver
```

This will build your blog in the `output` folder and start serving it on `localhost:8000`.

Start your browser and open the URL `localhost:8000`, and congrats 🎉🙌🏽🎇🎊🎈 you blog is up and running.

![Pelican blog default theme](/images/99999995-pelican-blog.png)

---

# Configure Firebase hosting emulator

URL: https://cloudbytes.dev/snippets/configure-firebase-hosting-emulator
Category: Snippets
Published: 2021-07-11
Author: Rehan Haider
Tags: firebase, github, python

> Use Firebase hosting emulator to locally test your static website generated by Hugo, Gatsby, Pelican and other SSGs

If you're hosting your static website or  [Jamstack]({filename}99999996-what-is-jamstack.md) webapp on firebase, you don't need to wait to deploy it to see how it will behave. You can do that while developing by installing [Firebase Emulators](https://firebase.google.com/docs/emulator-suite). 

Firebase offers local emulators for almost all of its services ranging from Auth, Hosting to even Realtime Database. You can use these emulators to develop features even before deploying them to Firebase. 

![Firebase emulator](/images/99999994-firebase_emulator.png)

Let's learn how to test your static website or a Jamstack website you have created using tools such as Pelican, Gatsby, etc. 

## Setting up the environment

Assuming you have your source code stored in a GitHub repository, fire up VSCode and press `Shift+Ctrl+P` to bring up the command palette. Then type/search for `Remote-Containers: Clone Repository in Container Volume` and press enter.  Then choose the repository you want to clone, followed by the branch you want to clone. 

If your devcontainer configuration is not already defined, it will as you to *Select a container configuration definition*, choose one based on the SSG you're using and its dependencies, e.g. in case of Pelican, choose Python 3, then choose the version as 3.9, and opt to install NodeJS as well since that is a requirement for Firebase CLI. 

If you're not using devcontainers for some absurd reason, install NPM from the below link. 

```http
https://nodejs.org/en/download/
```

## Setup Firebase CLI

To install the latest version of Firebase CLI, run the below in your VSCode Terminal

```bash
curl -sL firebase.tools | bash
```

This will automatically download and run a installation script that will detect your version of OS, and install the appropriate CLI version. 

### Login into Firebase CLI

To login, run the below command

```bash
firebase login
```

This will start the login process and open a link in your browser. After logging in using our firebase credentials you will be routed back and should see the below (redacted confidential data)

![Firebase login successful message](/images/99999994-firebase-cli-login.png)

### Firebase CLI Initial Setup

The firebase project is not yet setup. Let's first check the projects that you have already created by running

```bash
firebase projects:list
```

Assuming you already have some projects created you will see a list similar to below

![Firebase list all projects](/images/99999994-firebase_project_list.png)

**Step 1**: Then let's setup your projects locally, to begin that run

```bash
firebase init 
```

**Step 2**: This will bring up a list of options to initialise, use the arrow keys to navigate to Emulators and press Space to select it, then press enter. 

**Step 3**: Next it will ask for the project you want to use, choose the one you need. 

**Step 4**: Then it will ask to choose the Firebase emulators you want to setup, navigate to Hosting Emulator using arrow keys, press Space to select and then press Enter.

**Step 5**: Select the port you want the emulator to use, I chose 8080, since I already use the default port 5000 for some other apps. 

**Step 6**: It will next ask if you want to enable the Emulator UI, choose the default (Yes), then leave the next question about Emulator port to default and press enter

**Step 7**: Finally, when prompted if you want to download the emulators now, Type 'Y' and then press Enter.

Now your Firebase CLI is configured to use the Hosting Emulator. 

![Firebase emulator setup](/images/99999994-firebase_emulator_setup.png)

## Review the Firebase CLI configuration

The above configuration would have created two files

`.firebaserc`: Contains the project list and aliases. If you open it you would see something like

```json
{
  "projects": {
    "default": "cloudbytes-prod"
  }
}
```

Instead of "cloudbytes-prod" you should see the project you chose during the configuration.

`firebase.json`: Contains the configuration of your services, 

```json
{
  "hosting": {
    "public": "output",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ]
  },
  "emulators": {
    "hosting": {
      "port": 8080
    },
    "ui": {
      "enabled": true
    }
  }
}
```

The above file stores configuration for

1. **hosting**: with "public" key holding the name of the folder that contains all the files you want to host. Make sure this path points to the correct folder.
2. **emulators**: the configuration of our emulator that states, "Hosting Emulator" is enabled at port "8080" and UI is enabled

## Run Firebase Hosting Emulator

To start your hosting emulator run 

```bash
firebase emulators:start --only hosting
```

This will start your emulator using the "default" project select in `.firebaserc` and the "public" folder selected in `firebase.json` at port 8080. 

![Firebae hosting emulator start](/images/99999994-firebase_hosting_emulator_start.png)

To view the served website, open the URL `localhost:8080` in your browser.

---

# Automate deployment of Pelican Website to GitHub Pages

URL: https://cloudbytes.dev/snippets/automate-deployment-of-pelican-website-to-github-pages
Category: Snippets
Published: 2021-07-10
Author: Rehan Haider
Tags: github, python, pelican

> A guide to configuring automated Continuous Deployment (CD) of static site generated by Pelican SSG from GitHub repository to GitHub pages using GitHub Actions

Before moving forward, let's just accept that all of us use GitHub as our code repository. 
There are good reasons why? From free public and private repositories, to free GitHub Actions and free static hosting in form of [GitHub Pages](https://pages.github.com/).

## What is GitHub Pages

If you haven't heard of GitHub Pages, it is a service from GitHub that allows you to host a website or documentation directly from your repository. 
All you need to do is edit your changes, and push to the repository and your changes will be live in seconds.

This becomes specially relevant with the increased adoption of [Jamstack]({filename}99999996-what-is-jamstack.md) tools such as Pelican, Hugo, etc. 

But here's the problem. GitHub pages works only on three folders

1. **main** branch: The root of default main branch. Inconvenient since root folder will typically contain all of your code
2. **main/docs** directory: The doc folder in main branch. This could be possible but this would mean generating the content manually and then pushing it to repository
3. **gh-pages** branch: This could work but you need to copy only the output folder so that the full code is not copied

## How to automate deployment

Here's what we will do, we will setup a VSCode *devcontainer*, install Pelican, and then publish the website to GitHub pages. 

### Step 1: Prepare the development environment

To begin, create a new repository on GitHub. 

Then fire up VSCode, press `Shift + Ctrl + P` to bring up the command pallette, then type *"Clone repository in Container Volume"* to clone your repository in a container. 

This will ask if you want to clone a repository from GitHub, select the option, followed by the repository you have created. After that, choose the main as the branch you want to clone. 

After that you will be asked to choose a container configuration, select "Show all definitions", search for Python 3, and choose Python 3.9 from the dropdowns, when asked, choose to install NodeJS as well. 

We now have a [devcontainer]({filename}99999997-replace-python-venv-with-vscode-devcontainers.md).

### Step 2: Install Pelican and dependencies

In VSCode, bring up your terminal and then install Pelican by running

```bash
pip install "pelican[markdown]"
```

### Step 3: Setup Pelican

In the terminal type the below to trigger Pelican setup

```bash
pelican-quickstart
```

This starts the interactive setup where you are asked a series of question, choose as per below

```none
> Where do you want to create your new web site? [.] 
> What will be the title of this web site? MyPelicanWebsite
> Who will be the author of this web site? Me
> What will be the default language of this web site? [en] 
> Do you want to specify a URL prefix? e.g., https://example.com   (Y/n) N
> Do you want to enable article pagination? (Y/n) Y
> How many articles per page do you want? [10] 
> What is your time zone? [Europe/Paris] 
> Do you want to generate a tasks.py/Makefile to automate generation and publishing? (Y/n) 
> Do you want to upload your website using FTP? (y/N) 
> Do you want to upload your website using SSH? (y/N) 
> Do you want to upload your website using Dropbox? (y/N) 
> Do you want to upload your website using S3? (y/N) 
> Do you want to upload your website using Rackspace Cloud Files? (y/N) 
> Do you want to upload your website using GitHub Pages? (y/N) 
Done. Your new project is available at /workspaces/pelican-test
```

We haven't used the default GitHub Pages setting because it only publishes the output but doesn't automate the process of building.

You content is stored in the "content" folder, create a file named `my-first-post.md` in that folder with the below contents

```markdown
Title: My First automated blog post
Date: 2021-07-10
Category: Snippets
Tags: github, python, pelican
Author: Me
Summary: A guide to configuring automated Continuous Deployment (CD) of static site \
generated by Pelican SSG from GitHub repository to GitHub pages using GitHub Actions

## This is my first blog post
And it looks wonderful!
```

### Step 4: Test your Pelican website

To generate and serve your Pelican Website, run the below command

```bash
make devserver
```

This will build your blog in the `output` folder and start serving it on `localhost:8000`.

Start your browser and open the URL `localhost:8000`, and congrats 🎉🙌🏽🎇🎊🎈 you blog is up and running.

![Pelican blog default theme](/images/99999995-pelican-blog.png)

## Configuring GitHub action for automated deployment

Create a file at the path `.github/workflows/pelican.yml`, with the following contents

```yaml
name: Deploy

on:
  # Trigger the workflow on push on main branch,
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: justgoodin/pelican-to-github-pages@v1.0.2
      env:
        GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
        GH_PAGES_CNAME: ${{secrets.DOMAIN_CNAME}}
```

This action uses a GitHub Marketplace action published by me `justgoodin/pelican-to-github-pages@v1.0.1` to build your website and copy the output folder contents to `gh-pages` branch. 

### Configuring Secrets

GITHUB_TOKEN is a default action and you don't need to configure it, however, you will need to configure the DOMAIN_CNAME secret if you want to use a custom domain. In this example we will not configure since we will use a github.io subdomain. 

You can read more about the action at the [GitHub Marketplace](https://github.com/marketplace/actions/pelican-to-github-pages).

## Deploying the Pelican

Go back to your VSCode terminal and run the below commands to push your repository to GitHub. 

```bash
pip freeze > requirements.txt
git add .
git commit -m "My first Pelican blog"
git push
```

This will automatically create and run an action in GitHub > Actions tab of you repository

If everything went alright, you should see

![Github action success](/images/99999995-github-action-success.png)

## Publishing the website for the first and only time

Go to your repository on GitHub, on the repository navigation go to "Settings", then scroll down and click on Pages, under Source, choose `gh-pages` branch, and click on save. Shown below

![Github pages settings](/images/99999995-github-pages-settings.png)

If you go to the URL listed above source, your website is published. Hurray!! 👏🏽🥳

---

# What is Jamstack and why should you be using it?

URL: https://cloudbytes.dev/snippets/what-is-jamstack-and-why-should-you-be-using-it
Category: Snippets
Published: 2021-07-08
Author: Rehan Haider
Tags: python, javascript, pelican, netlify

> An introduction to Jamstack concepts and why should you be using it instead of wordpress.

Jamstack is a serverless web-app design concept which derives its JAM from JavaScript, API, and Markup. Even though it contains the work "-stack" it's not a framework instead it specifies a architecture pattern for designing websites that does not require a server at the backend resulting in massive performance improvements and lower cost. 

This architectural pattern can be implemented by a combination of technologies for each one of JavaScript, API, and Markup. 

<img src="/images/99999996-JAM.png" alt="jamstack architecture" width="75%" caption="Fig. 1: Jamstack frameworks / tools combination">

## So what makes a Jamstack?

This is achieved by 

1. **Markup / Frontend**: Uses a static website generator (SSG)  such as [Pelican](https://docs.getpelican.com/en/latest/), [Hugo](https://gohugo.io/), [Gatsby](https://www.gatsbyjs.com/), or [Next.js](https://nextjs.org/), etc. to convert a frontend template designed using [Angular](https://angular.io/), [Svelte](https://svelte.dev/), [Jinja2](https://jinja.palletsprojects.com/en/3.0.x/), or other templating languages into simple static HTML which can be served by using static website hosting such as [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html), [Netlify](https://www.netlify.com/), or [Firebase](https://firebase.google.com/), etc, eliminating the need for server based hosting which is considerably more expensive. 
2. **API / Backend **: Instead of using a server for processing business logic, e.g. authentication, Jamstack specifies using APIs such as that provided by Firebase, [Okta](https://www.okta.com/), etc. to handle user management. Similarly, more complex business logic that be coded in any language of one's choice using [AWS Lambda](https://aws.amazon.com/lambda/), [Google Cloud Functions](https://cloud.google.com/functions), etc. and used as as API by the frontend using, wait for it, 
3. **JavaScript / Event Handling**: JavaScript is used to handle user interaction and trigger events accordingly. Similarly, when an API response is received, JavaScript is used to render the output to the users and make the website dynamic. 

Alternatively, there are headless CMS such as Ghost, Strapi, etc. that can do a combination of above together. 

## What makes Jamstack different?

One can argue there is nothing new in "Jamstack", and older technology stacks used to function in the early days where every page was static and there was no server side rendering. 

As websites became more complex and technology matured we saw solutions like WordPress, Joomla, etc. coming into picture which would remove many repetitive tasks and function as a full Content Management System (CMS). 

But as these solutions become more powerful, it's compute requirements increased as well. A typical blog with reasonable audience running on WordPress would require 2 vCPU and 1 GB RAM minimum, which will cost between $5 - $10. Although not a big number, the correlation of cost to the scalability will largely be linear due to the server side rendering requirements. 

Compare that with a blog built using Jamstack architecture, the blog can run on the free GitHub pages upto more than 5,000 concurrent users before even considering more serious alternatives. 

Additionally, because of the lack of moving parts, testing becomes a lot easier and along with that building a CI/CD pipeline too. 

Finally, because you're using a bunch of static pages, you can use Content Delivery Network such as CloudFlare, AWS CloudFront, etc. to deliver your pages from close to where the users are resulting in sub-second page load times. 

<img src="/images/99999996-cdn.png" alt="jamstack content delivery network" width="75%" caption="Fig. 2: Jamstack frameworks content delivery network">

## When to use Jamstack? 

Theoretically, you can build almost anything using Jamstack architecture by utilising services such as [Firebase](https://firebase.google.com/), [AWS Amplify](https://aws.amazon.com/amplify/), [Supabase](https://supabase.io/), etc. 

E.g., you can replace a WordPress blog completely with Jamstack and still get 100 Rating on [Google PageSpeed Insights](https://developers.google.com/speed/pagespeed/insights/) without breaking a sweat, since, the webpages are generated ahead of time, there is 0 lag due to server side rendering (SSR). 

<img src="/images/99999996-pagespeed.png" alt="UberPython pagespeed results" caption="Fig. 3: UberPython pagespeed results">

Static Site Generators (SSGs) Such as Hugo, [Pelican](https://docs.getpelican.com/en/latest/), [Gatsby](https://www.gatsbyjs.com/), [11ty](https://www.11ty.dev/), etc. provide a rich plugin and theme ecosystem which can get your started with less technical knowledge than what is required for implementing WordPress. 

But that is not all, more complex use cases such as websites that offer courses, or other SaaS services can also be implemented easily. 

## Which combination to choose?

That depends on several factors and one usually ends up doing some trial and error before finalising on one. But you can follow some guidelines below that we used to arrive at the stack for [CloudBytes](https://cloudbytes.dev/).

* **Speed**: If your website has thousands of pages, use Hugo which is written in Go and designed to be really fast but has a steep learning curve. 
* **Familiar with React**: Use [Next.js](https://nextjs.org/) is a full React based framework that comes with kitchen sink. The alternative is [Gatsby](https://www.gatsbyjs.com/) which is equally popular and is a lot more customisable. 
* **Familiar with Ruby**: [Jekyll](https://jekyllrb.com/) is that you should be using. It is a fantastic SSG that is used by GitHub to serve static Github.io pages. 
* **Familiar with Python**: [Pelican](https://blog.getpelican.com/) and [Nikola](http://www.getnikola.com/) are the frontrunners that are based on Python
* **Familiar with JavaScript**: Most of the options above are based on JavaScript frameworks, but if you want a bit more freedom, [Eleventy](https://11ty.dev/) is your best choice with excellent templating support for almost any language (Nunjucks, Haml, Pug, Liquid, etc.) and plugin ecosystem.
* **Designed for Blog**: [Hexo](https://hexo.io/), [Jekyll](https://jekyllrb.com/), and [Pelican](https://blog.getpelican.com/) are best suited for use as blogging platform for it's support of Markdown and Liquid tags and flexibility. 
* **Documentation**: [MkDocs](http://www.mkdocs.org/) and [Docsify](https://docsify.js.org/) are best suited for documentation websites 

All of the options above are excellent in themselves with pros and cons in choosing one. 

This can be then combined with APIs for other functions such as authentication, email, messaging, queuing, payments, etc.

---

# Use VSCode Devcontainers instead of Python venv

URL: https://cloudbytes.dev/snippets/use-vscode-devcontainers-instead-of-python-venv
Category: Snippets
Published: 2021-07-07
Author: Rehan Haider
Tags: python, vscode, github

> VSCode Devcontainers are game changers that makes Python dependency management much easier.

I wrote about [why you need Python virtual environments]({filename}99999999-create-a-python-virtual-environment.md) and how to [create them]({filename}99999999-create-a-python-virtual-environment.md). 
All Python developers end up using some kind of environment manager like `venv` for any meaningful development effort. 

VSCode, a few years ago, released a concept called **devcontainers** that takes away the pain of managing many virtual environments for Python and other languages such as NodeJS, etc.

## VSCode's venv killer

Devcontainers or remote containers, work by opening you project folder inside a Docker Container giving you the flexibility of both Keeping the files on your system, and working in a secluded container with its own libraries and packages sanboxed from your other projects. 

This allows developers to use these Docker containers as a full-featured development environment with workspace files mounted from the local file system. 

![VSCode devcontainer architecture](/images/99999997-architecture-containers.png)

Each such devcontainer also acts like a workspace and can have its own set of extensions, and preferences configured.

## How to use devcontainers

First, the appropriate version of [Docker Desktop](https://www.docker.com/products/docker-desktop), If you're using a Windows system, you need to [install WSL2]({filename}99999965-install-wsl2.md) and enable [Docker WSL2 backend](https://aka.ms/vscode-remote/containers/docker-wsl2) is recommended. 

Once you have these setup, open VSCode and from Getting started page, click on "Open Folder" to open the folder where your project is stored.

After that either click on the "Open Remote Window" button on bottom left (two overlapping arroheads) or press `Ctrl + Shift + P` to open the command palette and choose "Reopen in Container" 

![VSCode remote container](/images/99999997-remote-container.png)

It will then ask you to choose one from ready-to-use configurations. Let's choose Python3 & PostgreSQL. This will trigger two actions

1. Create a folder named `.devcontainer` in your project root directory that with a `Dockerfile` and a `.devcontainer.json` that contains your user configuration
2. Start building the Docker container for you to use

Once the build process is complete, at the bottom left, you will see `Dev Container: Python3 & PostgreSQL` and you files will be still present.

## Preparing the environment

Now you have access to a functional actual Docker container running Debian Linux with Python and PostgreSQL already installed. 

If you are using `requirements.txt` to keep the list of dependencies, you will need to install your dependencies for the first time by opening the Terminal within VSCode and running

```bash
pip install requirements.txt
```

Or you can install them one by one and run the below to create a `requirements.txt`.

```bash
pip freeze > requirements.txt
```

## Enabling automatic installation

To enable VScode Remote Containers to install your dependencies, you should choose to rebuild the container, browse to the `.devcontainer` folder and open the `Dockerfile`

You need to uncomment the below block as shown below 👇🏽

```dockerfile
# [Optional] If your requirements rarely change, uncomment this section to add them to the image.
COPY requirements.txt /tmp/pip-tmp/
RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
   && rm -rf /tmp/pip-tmp
```

This will reinstall any dependencies that you may have noted under `requirements.txt`. 

> Make sure you have requirements.txt file present before doing the above, attempt to rebuild the container without the file present will result in failure

### Pro-tip
If you use your GitHub profile to login into VSCode, you can push your changes back to you Github account without entering your credentials or any additional configuration

---

# Configure & Fix Pelican Sitemap Error on Google Search Console

URL: https://cloudbytes.dev/snippets/configure-fix-pelican-sitemap-error-on-google-search-console
Category: Snippets
Published: 2021-07-04
Author: Rehan Haider
Tags: python, pelican

> A quick guide to correctly configurating sitemaps in Pelican & fixing URL errors thrown up by Google Search Console

**TL;DR** - Set the SITEURL variable in `publishconf.py` and use `publishconf.py` to generate your website instead of the default `pelicanconf.py`. This will fix your Google Search Console sitemap error. 

If you have have recently started using Pelican and have struggled with it comprehensive but incoherent documentation. One of the problems you might encounter is handing the SITEURL errors both while working on your local system and on the deployed webserver. 

## What is SITEURL?

Pelican's SITEURL refers to the URL of your website, e.g. example.com, or blog.example.com. In our website's case the SITEURL is "https://cloudbytes.dev". 

Pelican needs the SITEURL to define the `href` links in  your website correctly while publishing the website. 

## How to define SITEURL?

Make sure it is configured correctly in two places

1. `pelicanconf.py` : This file is typically used for local development. Running `pelican content` or `make html` or `invoke livereload` uses this file by **default**. 
2. `publishconf.py`: The setting in this file overrides the ones in `pelicanconf.py` and is meant for generating the static website for deployment. But Pelican doesn't use this by default. You need to run either `pelican content -s publishconf.py` or `make publish` to use this file as your settings. 

## Errors that you may encounter

If you don't configure SITEURL correctly the website will work on your system but not on the hosting provider you are using for your final website. 

Typically you may encounter the below error

![Pelican sitemap error on Google Search Console](/images/99999998-google-search-console-error.png)

This is due to malformed URLs, and the typical reason is incorrect SITEURL configuration.

---

# Create a Python virtual environment using venv

URL: https://cloudbytes.dev/snippets/create-a-python-virtual-environment-using-venv
Category: Snippets
Published: 2021-07-04
Author: Rehan Haider
Tags: python

> A quick guide to why you need a virtual environment, how to create, activate and manage

Python is the most popular programming language in the world. It Developers love it due to its versatility and the flexibility of using it in diverse ways. This is possible due to a plethora of Python packages published on [PyPi](https://pypi.org/) and many others. There are so many Python packages that you almost can find a package to do something, as illustrated by my favourite web-comic, [xkcd](https://xkcd.com/353/).

![XKCD - Python](https://imgs.xkcd.com/comics/python.png)

These packages or libraries can also depend on other packages, e.g. a very popular library used by data science professionals, [Pandas](https://pandas.pydata.org/), uses and builds on top of 3 other packages, Numpy, python-dateutil, and pytz. 

## The problem

But this diversity could also create problems for developers due to conflicts in the dependencies of multiple libraries, e.g. installing Pandas ends up installing 10+ other packages due. This could also cause conflicts between versions of dependencies. 

## Python Virtual Environment

This is where virtual environments can help. You can create different instances of Python specific for the application you're building without them conflicting with each other. 

![Python virtual environments](/images/99999999-python-virtual-environment.webp) 

### Create a virtual environment

Navigate to the folder that you want to place the virtual environment in and run `venv` module as shown below 👇🏽

```bash
python3 -m venv new-env
```

> `venv` is the recommended module for managing virtual environments now and `virtualenv` has been deprecated by Python 

This will create folder named `new-env` and place the virtual environment inside it including the Python interpreter, the standard library along with other supporting files. 

### Activate the virtual environment

After creating the virtual environment, you will need to activate it to be able to use it

On Windows, run:

```powershell
new-env\Scripts\activate
```

On Unix or MacOS, run:

```bash
source new-env/bin/activate
```

### Use the virtual environment

After creating the virtual environment, you will notice `(new-env)` in the terminal prompt you are using. 

You can install any package using

```bash
python3 -m pip install <package-name>
```

If you have added the Python directory to path, you also use the below 

```bash
pip install <package-name>
```

### Deactivate the virtual environment

To deactivate, simply run

```bash
deactivate
```