Felipe Trindade RSS Feed

Bridging the gap between IaC and GitOps with ArgoCD and Terraform

Wed, 22 Apr 2026 11:45:32 GMT

Hi there! Around 2 months ago, I published the following blog post: Bridging the gap between IaC and GitOps with FluxCD and Terraform. Now I'm continuing the "Bridge the Gap" saga but, this time using Terraform and ArgoCD instead.

As always, the entire code used in this post can be accessed in my GitHub Repository.

Bridge the Gap between IaC and ArgoCD

I’ll skip the explanation of what I mean by “bridge the gap between IaC and GitOps,” since this was fully covered in the Bridging the gap between IaC and GitOps with FluxCD and Terraform blog post. For this reason, I won’t repeat myself and will instead jump directly into the technical explanation!

Scenario

Our infrastructure goal is to have a new Kubernetes cluster deployed from scratch and bootstrap it with some good-to-have addons:

ALB Controller: To provision and manage Network Load Balance to expose Kubernetes apps.
External-DNS: To automatically create records in Route53 to expose our apps (domain level).
External-Secrets: To create Kubernetes secrets based on secrets in AWS Secrets Manager.
Istio: Implements the Gateway API controller.
Karpenter: Handles cluster node scalability.
Metrics-Server: Collects metrics of the pods and allows us to scale them based on these metrics (e.g., CPU and Memory).

Deploying the Infrastructure

The infrastructure needs to be deployed in a two-phase process when using Terraform. This is a limitation of the Kubernetes provider for Terraform, since it's required to have an EKS cluster provisioned before using the provider.

Based on that, first we need to deploy the core infrastructure (usually network related and EKS cluster) before moving to the Kubernetes cluster bootstrap.

So basically we should:

First deploy only core infrastructure (this can be easily achieved by using the --target flag in terraform apply CLI)
Then deploy the rest of the infrastructure.

It's important to mention that after the core infrastructure is deploy there is no need to deploy the infrastructure in a two-phase approach, since the EKS cluster is already in place, we are good to deploy the infrastructure entirely.

ArgoCD Bootstrap

ArgoCD will be installed by the cluster using the helm_release resource from the Helm provider for Terraform. After ArgoCD is installed into the cluster, Terraform will never manage ArgoCD version anymore, since we can create an ArgoCD Application that will manage ArgoCD installation. We will get there...

Ok, ArgoCD installed, now what? Well, we can deploy an ArgoCD Application that will deploy other Applications. That's the App of Apps pattern, used to bootstrap the cluster installation. In this case, this Application will deploy all the manifests inside 'k8s/bootstrap' folder. This folder contains basically three manifests:

App-of-Apps ArgoCD ApplicationSet to manage installation of addons (e.g., external-dns, karpenter...)
App-of-Apps ArgoCD ApplicationSet to manage installation of apps (e.g., your enterprise workloads...)
ArgoCD ApplicationSet to manage ArgoCD's own installation, so we completely remove Terraform responsability to manage ArgoCD installation! So, the helm_release will become a "ghost" resource, only used during the first execution of the second phase terraform apply.

Bridging the gap with ArgoCD ApplicationSet

So far, we covered the ArgoCD bootstrap part but, what about the part that really matters? How can we pass values that comes from infrastructure to the manifests.

ArgoCD offers a custom resource called ApplicationSet that can be used to deploy a custom resource Application (that is templated) in a specific cluster given inputs in the spec.generators field. The magical part is that, because the Application is templated, we can use values that come from the spec.generators part. So, if we can somehow control what goes into the generators we can dynamically inject values into the Application custom resource.

The ApplicationSet custom resource is usually used to, with a single manifest, deploy an Application into different Kubernetes clusters (remember that ArgoCD can manage multiple clusters!). We can manually register these clusters, but as DevOps, we avoid manual steps, so we can declaratively create a Kubernetes secret that contains information about this Kubernetes cluster (endpoint, name, TLS configuration). Within this secret, we can also add metadata (labels and annotations).

Given that, bridging the gap became simple, since we can create this Kubernetes secret using Terraform to reference the cluster and inject metadata (within the annotations, for example) with values about our infrastructure: Vpc ID, SQS ARN, ACM Certificate ARN...

Let's take two scenarios into consideration: deploy a Helm Chart that needs infrastructure information and deploy Kustomize manifests that needs infrastructure information.

Scenario 1: Helm Chart

Let's take, for example, the AWS ALB Controller that requires information about the infrastructure: cluster name, vpc id and AWS region.

We can create an ApplicationSet (remember that this ApplicationSet will be deployed automatically by the Addons App-of-Apps). This ApplicationSet will use information that comes from the metadata (annotations) of the Kubernetes Secret containing the cluster information. These metadata were added by the kubernetes_secret_v1 resource.

Scenario 2: Kustomize

Let's take as an example the EC2NodeClass resource of Karpenter. Suppose we already deployed Karpenter using Helm Chart, but now we would like to deploy the specific custom resources of Karpenter (such as EC2NodeClass and NodePool). We would like to pass to the EC2NodeClass the IAM Role that the EC2 created by Karpenter should use, so let's bridge the gap!

First, we create an ApplicationSet that will deploy the kustomize resources. Then, we use the Kustomize feature to patch the manifests with the values that comes from the metadata of the Kubernetes cluster Secret, which was created by Terraform.

Try it!

This is one of the things that explaining won't help much: you must try and see it for yourself. Demo it! The repository is ready to be used. Take your time to read the manifests and understand how things are connected!

Cya

Hope you enjoyed this blog post! See you in the next post! 👋

Self-Managed and AWS-Managed RAG System

Sat, 28 Feb 2026 10:40:32 GMT

Today we are going to do something different than what I usually post here. We are going to deploy a Self-Managed and AWS-managed RAG System to explore it more and understand more about RAG systems. If you are not familiar with it, don't worry, we will get there! It's essential to note that I'm not a Machine Learning Engineer and definitely not an expert in this matter, but I've participated in several projects involving the RAG system in AWS, and I believe I can explain it in a way that other DevOps professionals could benefit from.

Check out the code used on this GitHub Repository.

RAG explained

RAG (Retrieval Augmented Generation) is a pattern used with LLM to get answers given external knowledge/context to the LLM. Why is this important? Well, have you ever had an interaction with an LLM that couldn't provide a meaningful response? Maybe you asked about something that was specific to your company, and the LLM hallucinated or gave a wrong answer. This happened because the LLM didn't have the knowledge about what you were asking, maybe because the context you have is private and the LLM has no access (e.g., information that only exists in internal documents of your company), or maybe because the LLM was not trained with that piece of data (outdated data). The RAG system allows us to get better responses from the LLM given an external data collection. So, to summarize, RAG is a technique used to retrieve data from an external data collection where this data is augmented (injected into LLM prompt) at run-time and the LLM generates a response given that extra piece of context. This technique is very common to be implemented in AI customer chatbots (RAG is used to retrieve information about refunds, shipping, policies...) and AI assistants (use RAG to have more context about a PDF, file you uploaded).

A RAG system has two phases:

Indexing Phase: This part we get the data collection (PDFs, Web pages, internal documentation...) and chunk the text into smaller pieces, embed each chunk into a vector, and finally store in a vector database
Querying Phase: The application will use the vector database to retrieve the most similar chunks and inject this in the prompt in run-time.

Now, it's possible that you have multiple questions: What is this Vector Database? How was this database populated? What is this "similar search"? What is a vector? Don't worry, I plan to cover all these technicalities, so keep reading, and we will get there!

Visually, the querying phase with a RAG system goes like this:

First, the user asks something to the application, let's suppose an AI Chatbot. Let's say the user asked, "The shirt I bought arrived yesterday, but it's too big for me. Can I exchange it for a smaller size?"
The app will get the user's question, transform it into a vector using an embedding model, and send this vector to the vector database to perform a similarity search. In this case, it will understand it's about Returns, Refunds, Exchanges and retrieve information about the return policy of the company.
With the response from the vector database, the application can now inject this into a prompt (about the return policy) to an LLM to respond to the user's question.
The application will return the LLM response with a more concise and better context response.

The indexing phase with a RAG system goes like this:

First, we collect all the documents we would like to index (PDF, texts, URL...) and store them somewhere (e.g., object store service like S3)
Convert the data to text and perform cleanup on the data
Chunk the converted text data into multiple chunks using a chunking strategy and add metadata to it
Create the embeddings for each chunk
Store vector embeddings, original text, and metadata in an index in the vector database

Technicalities

Let's define some jargon that was used!

Vector: If you went to college in a STEM field (Science, Technology, Engineering, Math) you studied this for sure! A vector is simply a list of numbers living in a N-dimentional space. It's basically a generic math object that represents something (in physics, we use a lot to represent the direction of a force, for example).
Embedding: It is a numerical representation of data that captures its relevant qualities in a way that ML algorithms can process. In other words, think of it as the mathematical representation of something (image, text, audio...) that captures the relevant qualities, such as: meaning/topic when it's a text, shapes/textures in pictures... We need to convert the data to numbers; that's how we can perform mathematical logic in ML. The data is embedded in n-dimensional space.
Vector Embedding: Sometimes you will see people only saying "embeddings" instead of "vector embedding" because in the ML context, embeddings are almost always used in the vector form. The process of transforming the data to a vector embedding is performed by an embedding model, i.e., a machine learning model that is specialized in performing embedding.
Vector Similarity: Now that we have our data transformed into a vector embeddings, we can mathematically perform operations to know if one vector is similar to another vector. There are multiple methods to perform this, but the main ones are: dot product, cosine similarity, and Euclidean distance. The differences between them are outside the scope of this blog post. Why is this important? Simple! We can retrieve from a text information about refunds even if the user said I want my money back, because they are similar.
Chunking: Documents are too long to embed in a single vector, that's why chunking comes to rescue! Chunking means splitting the data into multiple smaller pieces that will later be embedded into vector embeddings. But there is a trade-off: smaller chunkings give a more precise retrieval but might miss context, while larger chunkings have more context but might be noisier. So the Machine Learning Engineer needs to use techniques to find the best chunking strategy.
Vector Database: Traditional databases are not optimized to deal with high-dimensional data and perform ANN (Approximate Nearest Neighbor) at scale, i.e. vector embedding requires optimized databases to work with. Top Vector Databases are: Qdrant, Pgvector (an extension of PostgreSQL), Milvus, ElasticSearch.
ANN: Approximate Nearest Neighbor is very popular algorithm that performs efficient search at scale. It's important not confuse it with the similarity metrics mentioned before (dot product, cosine similarity, and Euclidean distance). While the similarity metric answers "How similar are these two vectors?", the ANN answers "How do we find the top-K most similar vectors?". Imagine having to compare the embedded vector with millions of embedded vectors. This is an O(n) operation (using Big-O notation), and it's not the fast method... That's where ANN comes into play, by sacrificing a bit of accuracy (that's why it has Approximate in its name!) to gain speed.
Metadata: So far, everytime I talked about similarity and retrieval I only mentioned the data itself, but in reality it's very common in professional RAG systems to also use metadadata. During the indexing phase for each chunk, you usually store three things: the data (e.g., the text of the chunked context), the embedding vector (the numeric representation), and the metadata. The metadata holds attributes about the chunk that can be used to filter during the querying phase. Example of metadata fields are: language, region, source, title, section, page number, department, owner, tenant id...

I think this is the minimum you need to know in order to understand the basics of RAG. The Machine Learning Engineer needs to know way more than this and also apply the best strategy for chunking (size and overlap of each chunk), embedding (which embedding model to use), retrieval (how similar the words need to be and filters to use), and much more.

Hands On

Let's deploy the RAG architecture to serve a customer-facing chatbot responsible for answering eBay clients' questions. Ebay offers extensive documentation for helping sellers and buyers: more than 10 different pages talking about different topics. That's a great dataset for us! For our self-managed RAG system, I will open each page and download the page as PDF, for the AWS-managed solution, I will simply use the URL of the website. This way, I can present two different ways of dealing with input data for RAG.

Self-Managed RAG System

For the self-managed RAG system, I propose an event-based architecture that, every time a new PDF is added to the S3 Bucket, the entire process starts and does the parsing (PDF to text), chunking, and embedding process. Since the PDFs are small and the duration to run these processes is not long, it makes sense to trigger the orchestration (Step Function Workflow) of the process using the S3 upload event as input, run each process in Lambda (serverless, cheap, easy to maintain), and store it in the Vector Database. For the database, I will use Qdrant, since it's an open-source solution with great community support. For simplicity (this is now prod-ready design...) I will run the database in EC2 (data stored in EBS volume) managed by ECS. I void the usage of Fargate because I would need to use NFS for storing the data, and Qdrant docs explicitly say that Qdrant does not work with Network file systems. And of course, an API to receive the user question, perform similarity search in the vector database, and talk to a generative LLM to get a response (given a prompt and returned data from the vector database). For this, I will stick with the classic API Gateway + Lambda approach! Oh, and we can even leverage the new API Gateway streaming feature.

The orchestration is actually very simple: extract the PDF S3 Object key from the event and trigger the pdf-to-text Lambda, then the chunking, and finally the embedding. Clean, simple, and it works! Each PDF gets a doc_id and is stored in an organized way in the S3 bucket:

raw/      # Prefix used to PDF files uploaded by employees
clean/    # JSON containing the extracted text from PDF
chunks/   # JSONL (each line is a JSON) containing the chunk

AWS managed RAG System

Well, for the AWS-managed RAG system the things get... way simple! Bedrock Knowledge Bases is an AWS service that simplifies by A LOT the RAG system workflow. Indeed, you only need to define two things: the storage configuration (where the data will be stored - vector database) and the data source (where the data comes from). In our case we will use WEB Crawler data source (URLs) and the AWS Documentation tell us that the only supported vector database supported is OpenSearch Serverless, so that's it. In the date I'm writing this blog post (Feb 2026) the Web Crawler data source is in preview release and is subject to change, but let's use it anyway. Here we would still need to use an API to perform the RAG process but since Knowledge Bases offers an UI to test your knowledge base I will stick with it for this hands on experiment.

Proposed Architecture

Based on that, we have our proposed architecture for this project!

It's very important to say that this is not a silver bullet architecture that works great for all scenarios. This is simply a recommended architecture for this hands-on blog post!

Deployment

As always, the code for this hands-on experiment is in my GitHub Repository. As you might imagine, there are several parts in this project:

pdf-to-text Lambda code
chunking Lambda code
embedding Lambda code
API Lambda code
Terraform code
GitHub Actions CI/CD code

And pasting all this code here would be too much, so I will intentionally not paste any code here and encourage you to check the repository. Since it's all Terraform, feel free to deploy in your environment and test it!

Results

I've used a very simple prompt for the generative LLM that goes like this:

You are answering questions using only the provided eBay help-center context.
Treat the context as the only allowed source of truth.
If the answer is not clearly supported by the context, say:
'I do not have enough information in the provided knowledge base to answer that clearly.'
Do not guess.
Do not invent UI steps, actions, deadlines, or policies.
Do not combine buyer and seller flows unless the context explicitly says both apply.
If the user question is buyer-specific, prefer buyer instructions.
If the user question is seller-specific, prefer seller instructions.
If the role is unclear and the context differs by role, say so.
Ignore irrelevant context.
Return only plain English text, short and factual, in one paragraph.

Question: $query$

Context: $search_results$

Answer the question using only the context above.

where $query$ corresponds to the user input question and $search_results$ the results of the similarity search in the vector database.

Results - Self-Managed RAG

I've input several questions into the system and compiled all the answers here, some examples below:

Q: My package didn't arrive. What do I do?

A: If your item hasn't arrived, check your order confirmation email
for the expected delivery date and shipping address, and select
“View order details” to see the latest tracking updates. If the
estimated delivery date has passed, report that your item hasn't
arrived by contacting the seller through your order confirmation
email, selecting “I haven't received my item yet,” and choosing
“No, I want to contact the seller.” You have up to 30 calendar days
from the estimated delivery date to do so, and the seller should
respond within 3 business days with an update, replacement, or refund.

and

Q: Item arrived broken, do I open a return or something else?

A: If an item arrives broken, you should go to your Purchases
and select “See request details” to close your request. Select
a reason for closing the return and add details if you want to,
then confirm.

Sometimes the responses were on point, sometimes they looked weird. That's because we need to tune the system, perform better cleaning (PDF to text)... Not the perfect system, but the idea has been proven. And it works! In the logs of the API, you can even see the chunk returned by the similarity search operation.

Results - AWS Managed

I didn't compile all the responses that came from the AWS Knowledge Base, but I screenshot some responses.

For these questions, very similar results.

Cya!

Well, I hope you liked this blog post!

See you around! 👋

Bridging the gap between IaC and GitOps with FluxCD and Terraform

Mon, 05 Jan 2026 11:45:32 GMT

This blog post is a very special one for two main reasons:

It marks two full years of my blog posts! On average, I delivered one technical blog post per month (some months I posted two!).
This post tackles a real problem that I've faced for years, and never really liked how others addressed this issue… I'm really proud of the proposed solution that will be described here, and I'm pretty sure it has room for improvement, but it's a good start.

As always, the entire code used in this post can be accessed in my GitHub Repository

Bridge the Gap between IaC and FluxCD

Let's explain a bit about the problem. Cloud resources are created with the IaC tool, and they need to be referenced in Kubernetes (e.g., Bucket name, Certificate ARN, Subnet IDs, SQS Queue ARN…). But how can these values be passed to the Kubernetes resources (e.g., Deployments, Helm Chart Values…)?

The "classic" solution for this is to store these in AWS Secrets Manager or Parameter Store and use External Secrets to create Kubernetes Secrets that will be consumed by your application as environment variables. The reality is that this pattern does not work in all cases, since it's very common the need to pass the value directly to a Helm Chart custom values, to manifests that do not support reading the value from a secret (e.g. ALB Controller needs VPC ID that is passed as args to the controller pod) or even in annotations/labels of resources.

The solution proposed here allows us to perform substitution on placeholder values using our GitOps Tool (in this case, FluxCD), and it works by:

Creating a ConfigMap that will substitute Kustomize values using the postBuild feature of FluxCD.
Create a specific ConfigMap per service that will be applied as custom values for the Helm Chart.

This way, we are able to bridge the gap between IaC and GitOps by replacing the values in Kubernetes YAML manifests (created with Kustomize) and custom values for Helm Charts.

I bet you are still very confused with the summary of the explanation above, so let's deep dive with real examples and explain this step by step.

Scenario

Our infrastructure goal is to have a new Kubernetes cluster deployed from scratch and bootstrap it with some good-to-have addons:

ALB Controller: To provision and manage Network Load Balance to expose Kubernetes apps.
Authentik: A complete IdP platform to set up SSO. I've already talked about SSO and Authentik in the Hands-on SSO with Terraform and Authentik blog post!
External-DNS: To automatically create records in Route53 to expose our apps (domain level).
External-Secrets: To create Kubernetes secrets based on secrets in AWS Secrets Manager.
Headlamp: A dashboard for monitoring our cluster.
Istio: Implements the Gateway API controller.
Karpenter: Handles cluster node scalability.
Metrics-Server: Collects metrics of the pods and allows us to scale them based on these metrics (e.g. CPU and Memory).
Reloader: A controller to watch changes in ConfigMap and Secrets and do rolling updates on pods.

Deploying the Infrastructure

Based on that, first we need to deploy the core infrastructure (usually network related and EKS cluster) before moving to the Kubernetes cluster bootstrap.

You might be surprised that, based on the flow above, the creation of Kubernetes Namespaces is being managed by Terraform, but we will get there. For now, you just need to understand that the Terraform deployment should be performed in a two-phase approach:

Deploy only core infrastructure (this can be easily achieved by using the --target flag in terraform apply CLI)
Deploy the rest of the infrastructure

FluxCD Bootstrap

FluxCD Documentation suggests several ways for bootstrapping FluxCD. I personally like the Terraform approach, and I'll stick with it. The flux_bootstrap_git resource of the FluxCD provider is responsible for installing FluxCD in the Kubernetes cluster and push the manifests related to FluxCD installation to your git repository, which is aligned with a GitOps approach.

Our GitOps Controller (FluxCD) will be responsible for installing all the tools in the Kubernetes cluster using a GitOps approach. Based on our scenario, FluxCD should install the resources in the following order:

Some important comments:

I think it's a best practice to manage CRDs separately, since this avoids some racing conditions (e.g. application tries to use ExternalSecret CR but the CRDs were not installed yet).
Karpenter should be the first addon to be installed in a Kubernetes cluster, since it will guarantee cluster scalability. If you try to deploy Karpenter and several others addons at the same time, it is not guaranteed that the initial nodes of the cluster will have enough memory to install all addons and Karpenter, so it's better to be safe and make sure Karpenter is installed first and can guarantee the cluster scalability.
Based on my experience, if you try to deploy ALB Controller and several other services at the same time, the deployments might be stuck because of the following error:

Internal error occurred: failed calling webhook "mservice.elbv2.k8s.aws":
failed to call webhook:
Post "https://aws-load-balancer-webhook-service.kube-system.svc:443/mutate-v1-service?timeout=10s":
no endpoints available for service "aws-load-balancer-webhook-service"

When installing it first, I saw no errors.

I'm considering addons all services that are "internal" to Kubernetes, i.e. tools that handle monitoring, scalability, automation... and apps as real apps that will be used by your clients (e.g. APIs, Webservers...).

Addons/Apps installation

I have a preference for installing external tools (i.e. tools that I don't manage, such as open-source projects) using Helm Chart instead of managing each Kubernetes YAML Manifest individually (Deployment, PDB, Service, ConfigMap...). Because of that, I'll use the HelmRelease Flux's Custom Resource to install these resources. But, there is always a need to manage some pure YAML manifests (e.g. ExternalSecret, HttpRoute - some Helm Charts only support Ingress...), for these, I will use the Kustomization Flux's Custom Resource. Also, some of these addons need to install CRDs and the best practice is to install and manage these separately. The flow chart below explains the order of deployments I'd like FluxCD to run when creating a new cluster from scratch.

The following flow summarizes the main idea of how to Bridge the Gap between IaC and GitOps:

Notice that we can create the namespace with FluxCD, but there is a race condition problem. From the moment you FluxCD bootstrap fisishes, Terraform will try to create a ConfigMap in a namespace that does not exist yet (it takes time for FluxCD to deploy stuff). So if we move the namespace creation to Terraform we won't have this problem! Sure, we still add some hacky stuff (e.g. null_resource with "sleep") or retry the Terraform execution after FluxCD has correctly provisioned the namespace, but I personally don't like this idea much.

Let's take real examples to explain this.

If you REALLY want to understand how this was done, take your time to check these 6 examples:

Storage Class: Single manifest.
Reloader: Installation via HelmRelease without custom values.
Karpenter: Installation via HelmRelease. It needs non-sensitive custom values non-related to infra and related to infra (that are created by Terraform General ConfigMap and Terraform specific ConfigMap).
Metrics Server: Installation via HelmRelease. It needs non-sensitive custom values non-related to infra.
Authentik: Installation via HelmRelease. It needs non-sensitive custom values non-related to infra and related to infra. It also needs sensitive data.
Headlamp: Installation via HelmRelease. It needs non-sensitive custom values non-related to infra and sensitive data.

Cya

Hope you enjoyed this blog post! Happy New Year, and see you in the next post! 👋

Ephemeral Environment for serverless backend

Tue, 09 Dec 2025 17:20:32 GMT

Today, I will cover a strategy for ephemeral environments for backend applications hosted in AWS Lambda behind an API Gateway. So today's goal is to deploy a short-lived environment that exists during the pull request lifecycle, helping the developers and testers make sure the code change implements what is expected, without having to merge to the dev/qa environment to test the changes on a lower environment.

For this demo, I will use AWS CDK (with Typescript) and GitHub Actions.

All the code used for this demo is available in my GitHub Repository!

Architecture

The architecture is fairly simple: an API Gateway with a custom domain (e.g., api.mydomain.com) that points to a Lambda Function (using proxy integration) that serves as an API.

The proposed solution in this blog post has some considerations:

The Lambda uses AWS Lambda Web Adapter in order to run APIs on AWS Lambda (e.g., fastapi, flask, next, express...).
Although I'll be using FastAPI (Python), you can use any other API framework that is compatible with the Lambda Web Adapter mentioned.
The proposed solution is agnostic in terms of the IaC tool. I'll be using CDK, but the same result could be achieved with Terraform, Pulumi, or even AWS CLI. Here, what matters is the proposed idea.

So, based on the architecture, you should expect to have an endpoint deployed that your web page or other service would call (e.g., HTTP GET api.mydomain.com/users/123). The API Gateway has a root / resource and a /{proxy+} that will be used to proxy all routes to the Lambda, and the Lambda will handle the request. The API Gateway will target a Lambda Alias that points to the environment (i.e., dev).

Flow of the Ephemeral Environment

The ephemeral environment deployed will have the same lifespan as the pull request, so we are aiming for the following flow:

Developer opens a Pull Request with code changes
GitHub Actions runs a workflow that builds and deploys the image to the ECR repository. It creates a lambda alias (e.g., pr-4), updates the $LATEST version of the lambda to the image that was just pushed, adds the REMOVE_BASE_PATH (more on that on step 3) environment variable, and creates a new Lambda version pointing to the lambda alias created. Then, it creates a new API Gateway with Lambda alias proxy integration (e.g., pr-4).

The biggest reason to not use $LATEST version of the Lambda directly is that new versions are created from the $LATEST version, so it doesn't make sense to modify the $LATEST version (that would point to dev) when deploying to the ephemeral environment.

A comment is added to the pull request with the link to access the ephemeral environment.

It is important to mention that the ephemeral environment here is using a path-based model, instead of a subdomain-based model (e.g., pr-4.api.mydomain.com). This is because during Pull Request, we are creating a new API Gateway proxy resource with pr-4 as the name. So every request that has pr-4 in the path will be forwarded to the ephemeral environment lambda alias. This is the main reason for us to use the REMOVE_BASE_PATH: the API calls to the ephemeral environment will be routed by the newly created API Gateway resource (i.e., pr-4 in the example). So we need to remove this base path before sending the payload to the API code, otherwise, we would expect a not found response.

Developer/QA/Tester checks the ephemeral environment and verifies the changes
Pull Request gets approved and merged
GitHub Action runs a workflow to delete the ephemeral environment (i.e., deletes Lambda alias, version, and API Gateway resource) and the comment in the Pull Request with the URL.

This is a simple flow that is very powerful and can be extended to meet different needs.

Suppose the following Pull Request that creates a new endpoint /random that generates a pseudo-random number. During the Pull Request, the endpoint will be available in the ephemeral environment:

But if you try to access the dev environment, the endpoint will not be there:

⚠️ Since I was using fastapi I could have taken a screenshot of the /docs page (Swagger UI) to show that the new endpoint exists on the ephemeral environment, right? Well... the issue is that the Swagger page makes a request to its own API to fetch the openapi.json file, and this request does not have the path prefix (i.e. pr-4), which uses the dev Lambda alias. In other words, the ephemeral environment won’t work for this specific use case. Maybe there’s some custom logic you could implement to fix this, but I couldn’t find a straightforward workaround.

Code

I don't think it will be beneficial to copy and paste code from the demo GitHub Repository. So I recommend that you go over the code there and deploy it yourself!

Cya!

Hope you liked this blog post!

See you around! 👋

Using ALB and Cognito to add authentication to your apps

Wed, 26 Nov 2025 11:45:32 GMT

As a DevOps, the security should always be a concern when deploying applications and for that reason you should add a minimum layer of authentication in front of authless applications before deploying publicly (specially if they are internal tools). A good "hyped" example of application that does not implement authentation and forces you to manage it is MLflow Tracking Server that recommends you to add authentication using a reverse proxy (e.g. NGINX) via proxy authentication headers. In other words, instead of authentication being handled by the application, we should handle it on the infrastructure side.

The AWS-native way of implementing authentication on the infrastructure side is checking if the user is authenticaticate on the Load Balancer (ALB). A common strategy is to link a Cognito User Pool on the ALB actions that redirects unauthenticated users (i.e. cookie is not present) to the Cognito hosted UI to the user perform authentication. In this scenario, the Cognito act as the IdP (if these terms are scary to you I recommend reading the SSO blog post) and the ALB act as the OIDC client.

It's important to mention that since we are using Cognito as the IdP we can leverage Cognito capabilities and use the federated sign-in of Cognito to allow users to perform login using external providers, such as Google, Okta or even AWS IAM Identity Center. The later is specially interesting for us when the application that you would like to deploy should be internal, i.e. used only by the internal/development team. Since, AWS Identity Center is considered the best practice for managing user access in AWS environment and it's a free service, it is very handy to use it as an SSO panel for AWS and internal applications.

The main goal of this blog post is to set up a public application (I will use Nginx for simplicity, but it could be any other public application) using AWS Fargate (deployed in an ECS cluster) exposed by an ALB and add authentication via Cognito (either via Cognito User Pool or federated SAML with AWS IAM Identity).

All the code used is available in the GitHub repository.

CDK code

For simplicity a single Stack will be used to deploy the entire infrastructure: VPC, ALB, Cognito, Fargate, ECS Cluster... Here we go:

// main.ts
import * as cdk from "aws-cdk-lib";
import { WorkloadStack } from "./stack/workload";
import { devWorkloadConfig } from "./config";

const app = new cdk.App();

new WorkloadStack(app, "DevWorkloadStack", { env: devWorkloadConfig.env, config: devWorkloadConfig });

Our workload stack is defined as:

import * as cdk from "aws-cdk-lib";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import { Construct } from "constructs";
import { WorkloadConfig } from "../config";
import * as ecs from "aws-cdk-lib/aws-ecs";
import * as alb from "aws-cdk-lib/aws-elasticloadbalancingv2";
import * as acm from "aws-cdk-lib/aws-certificatemanager";
import * as route53 from "aws-cdk-lib/aws-route53";
import * as route53Targets from "aws-cdk-lib/aws-route53-targets";
import * as iam from "aws-cdk-lib/aws-iam";
import * as cognito from "aws-cdk-lib/aws-cognito";
import * as albActions from "aws-cdk-lib/aws-elasticloadbalancingv2-actions";

export interface NetworkStackProps extends cdk.StackProps {
  config: WorkloadConfig;
}

interface IUserPool {
  userPool: cognito.UserPool;
  userPoolClient: cognito.UserPoolClient;
  userPoolDomain: cognito.UserPoolDomain;
}

export class WorkloadStack extends cdk.Stack {
  config: WorkloadConfig;
  domain: route53.IPublicHostedZone;
  certificate: acm.Certificate;

  constructor(scope: Construct, id: string, props: NetworkStackProps) {
    super(scope, id, props);

    this.config = props.config;
    this.domain = route53.PublicHostedZone.fromLookup(this, "PublicHostedZone", {
      domainName: props.config.domainName,
    });
    this.certificate = new acm.Certificate(this, "WildcardCertificate", {
      domainName: this.config.domainName,
      subjectAlternativeNames: [`*.${this.config.domainName}`],
      validation: acm.CertificateValidation.fromDns(this.domain),
    });

    const vpc = this.createVpc();

    const cluster = this.createEcsCluster(vpc);
    const { publicAlb, httpsListener } = this.createPublicAlb(vpc);
    const userPool = this.createCognitoAuth();
    this.createAuthlessFargateApp(vpc, cluster, publicAlb, httpsListener, userPool);
  }

  createVpc(): ec2.Vpc {
    const vpc = new ec2.Vpc(this, "Vpc", {
      subnetConfiguration: [
        {
          cidrMask: 24,
          name: "public",
          subnetType: ec2.SubnetType.PUBLIC,
        },
        {
          cidrMask: 24,
          name: "private",
          subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
        },
        {
          cidrMask: 28,
          name: "isolated",
          subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
        },
      ],
      natGateways: 1, // It reduces costs but also reduces HA
      maxAzs: this.config.maxAzs,
    });
    return vpc;
  }

  createEcsCluster(vpc: ec2.Vpc): ecs.Cluster {
    return new ecs.Cluster(this, "EcsCluster", {
      vpc,
      enableFargateCapacityProviders: true,
    });
  }

  createPublicAlb(vpc: ec2.Vpc): {
    publicAlb: alb.ApplicationLoadBalancer;
    httpsListener: alb.ApplicationListener;
  } {
    const publicAlb = new alb.ApplicationLoadBalancer(this, "PublicAlb", {
      vpc,
      vpcSubnets: vpc.selectSubnets({
        subnetType: ec2.SubnetType.PUBLIC,
      }),
      internetFacing: true,
    });
    publicAlb.addRedirect();
    const httpsListener = publicAlb.addListener("HttpsListener", {
      protocol: alb.ApplicationProtocol.HTTPS,
      sslPolicy: alb.SslPolicy.RECOMMENDED_TLS,
      port: 443,
      defaultAction: alb.ListenerAction.fixedResponse(404, {
        messageBody: "No services configured",
      }),
      certificates: [this.certificate],
    });

    return {
      publicAlb,
      httpsListener,
    };
  }

  createCognitoAuth(): IUserPool {
    const COGNITO_DOMAIN = `auth.${this.config.domainName}`;

    // Reference: https://repost.aws/knowledge-center/cognito-custom-domain-errors
    const dummyARecordParentDomain = new route53.ARecord(this, "CognitoAuthDummyARecord", {
      zone: this.domain,
      recordName: "",
      target: route53.RecordTarget.fromIpAddresses("8.8.8.8"),
    });

    const userPool = new cognito.UserPool(this, "CognitoAuthUserPool", {
      selfSignUpEnabled: false,
      passwordPolicy: {
        minLength: 8,
        requireDigits: true,
        requireLowercase: true,
        requireUppercase: true,
        requireSymbols: false,
      },
      signInAliases: {
        email: true,
      },
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });
    userPool.node.addDependency(dummyARecordParentDomain);

    const userPoolDomain = new cognito.UserPoolDomain(this, "CognitoAuthUserPoolDomain", {
      userPool,
      customDomain: {
        certificate: this.certificate,
        domainName: COGNITO_DOMAIN,
      },
      managedLoginVersion: cognito.ManagedLoginVersion.CLASSIC_HOSTED_UI,
    });
    userPoolDomain.node.addDependency(dummyARecordParentDomain);

    const samlProviderName = "AuthlessFargateApp";
    let supportedIdentityProviders;
    if (this.config.samlMetadataUrl) {
      supportedIdentityProviders = [cognito.UserPoolClientIdentityProvider.custom(samlProviderName)];
    } else {
      supportedIdentityProviders = [cognito.UserPoolClientIdentityProvider.COGNITO];
    }

    const userPoolClient = new cognito.UserPoolClient(this, "CognitoAuthClient", {
      userPool,
      supportedIdentityProviders,
      oAuth: {
        flows: {
          authorizationCodeGrant: true,
          implicitCodeGrant: true,
        },
        scopes: [cognito.OAuthScope.OPENID],
        callbackUrls: [
          `https://${this.config.appSubdomain}.${this.config.domainName}/oauth2/idpresponse`,
          `https://${this.config.appSubdomain}.${this.config.domainName}`,
        ],
        logoutUrls: [`https://${this.config.appSubdomain}.${this.config.domainName}`],
      },
      generateSecret: true,
    });

    new route53.ARecord(this, "CognitoAuthAliasRecord", {
      zone: this.domain,
      recordName: COGNITO_DOMAIN,
      target: route53.RecordTarget.fromAlias(new route53Targets.UserPoolDomainTarget(userPoolDomain)),
    });

    if (this.config.samlMetadataUrl) {
      new cognito.UserPoolIdentityProviderSaml(this, "SamlProvider", {
        userPool,
        name: samlProviderName,
        metadata: cognito.UserPoolIdentityProviderSamlMetadata.url(this.config.samlMetadataUrl),
        attributeMapping: {
          email: cognito.ProviderAttribute.other("email"),
        },
      });
    }

    return {
      userPool,
      userPoolClient,
      userPoolDomain,
    };
  }

  createAuthlessFargateApp(
    vpc: ec2.Vpc,
    cluster: ecs.Cluster,
    publicAlb: alb.ApplicationLoadBalancer,
    httpsListener: alb.ApplicationListener,
    userPool: IUserPool,
  ) {
    const FULL_APP_HOST = `${this.config.appSubdomain}.${this.config.domainName}`;

    const taskDefinition = new ecs.TaskDefinition(this, "AuthlessAppTaskDefinition", {
      cpu: "512",
      memoryMiB: "1024",
      compatibility: ecs.Compatibility.FARGATE,
    });
    taskDefinition.addContainer("nginx", {
      portMappings: [{ containerPort: 80 }],
      image: ecs.ContainerImage.fromRegistry("nginx:latest"),
      logging: ecs.LogDriver.awsLogs({
        streamPrefix: "nginx",
      }),
    });

    const fargateService = new ecs.FargateService(this, "AuthlessFargateService", {
      cluster,
      taskDefinition: taskDefinition,
      desiredCount: 1,
      assignPublicIp: false,
      vpcSubnets: vpc.selectSubnets({
        subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
      }),
    });
    fargateService.taskDefinition.executionRole?.addManagedPolicy(
      iam.ManagedPolicy.fromAwsManagedPolicyName("service-role/AmazonECSTaskExecutionRolePolicy"),
    );

    const targetGroup = new alb.ApplicationTargetGroup(this, "AppTargetGroup", {
      vpc,
      targetType: alb.TargetType.IP,
      protocol: alb.ApplicationProtocol.HTTP,
      port: 80,
      targets: [fargateService],
      deregistrationDelay: cdk.Duration.seconds(10),
    });

    const action = new albActions.AuthenticateCognitoAction({
      userPool: userPool.userPool,
      userPoolClient: userPool.userPoolClient,
      userPoolDomain: userPool.userPoolDomain,
      next: alb.ListenerAction.forward([targetGroup]),
    });
    httpsListener.addAction("AppRule", {
      priority: 1,
      action,
      conditions: [alb.ListenerCondition.hostHeaders([FULL_APP_HOST])],
    });

    new route53.ARecord(this, "AuthlessAppRecord", {
      target: route53.RecordTarget.fromAlias(new route53Targets.LoadBalancerTarget(publicAlb)),
      zone: this.domain,
      recordName: FULL_APP_HOST,
    });
  }
}

And WorkloadConfig is defined as:

export type WorkloadConfig = {
  env: {
    account: string;
    region: string;
  };
  vpcCidrBlock: string;
  maxAzs: number;
  domainName: string;
  appSubdomain: string;
  samlMetadataUrl?: string;
};

And real values, should look like:

import { WorkloadConfig } from "./types";

export const config: WorkloadConfig = {
  env: {
    account: "937168356724",
    region: "us-east-1",
  },
  vpcCidrBlock: "10.5.0.0/16",
  maxAzs: 2,
  domainName: "demosfelipetrindade.lat",
  appSubdomain: "app",
  samlMetadataUrl:
    "https://portal.sso.us-east-1.amazonaws.com/saml/metadata/OTM3MTY4MzU2NzI0X2lucy03MjIzMGJjNWY5MDQzYzVj",
};

Hands-on

Ok! Now that you know how the code looks like, let's deploy this! I'd recommend you to clone my GitHub repository and follow along.

Install dependencies

You can use mise to install all developer dependencies.

mise install

Then, install NodeJS dependencies:

yarn

Export AWS credentials to connect to the Shared Assets account in our console
Change the config values to fit your use-case

Change the src/config/config.dev.ts to fit your use case. The parameter samlMetadataUrl is optional, and if not provided, the authentication will be via Cognito User Pool; otherwise, if the parameter is passed, the authentication will be via SAML 2.0 using IAM Identity Center as IdP.

⚠️ The AWS does not support creating Identity Center applications programatically, meaning that the application should be created manually. I suggest first deploying the application without the samlMetadataUrl parameter, then proceeding to add it. This will be explained in the following steps.

Go to your application endpoint and check if it's redirecting you to log in via Cognito User Pool.

You can create a user in the user pool and validate that this user can actually log in to the application.

Deploy the DevWorkloadStack

yarn cdk deploy DevWorkloadStack

After the deployment, go to the AWS Console > CloudFormation > DevWorkloadStack > Resources > Copy the Physical ID of the CognitoAuthUserPool resource. We are going to use it on step 8.

Go to the AWS Identity Center

Select Application of type SAML 2.0

Configure the SAML 2.0 Application

The Application Start URL should be set to: https://{appSubdomain}.{domainName}
The Application ACS URL should be set to: https://{appSubdomain}.{domainName}/saml2/idpresponse. The Cognito documentation specificies the /saml2/idpresponse path.
The Application SAML audience should be set to: urn:amazon:cognito:sp:{awsRegion}:{userPoolId} based on the Cognito documentation.

Make sure to substitute the variable in the brackets with the correct value:

{appSubdomain}: The value used in your src/config/config.dev.ts file
{domainName}: The value used in your src/config/config.dev.ts file
{awsRegion}: The value used in your src/config/config.dev.ts file (env.region).
{userPoolId}: The value you copied on step 4.

Copy the IAM Identity Center SAML metadata URL to use in the following step.

Edit attribute mapping

Map:

Subject to ${user:subject} with persistent format.
email to ${user:email} with basic format.

Deploy the stack with the samlMetadataUrl

Add the samlMetadataUrl parameter to the src/config/config.dev.ts and then deploy the stack again:

yarn cdk deploy DevWorkloadStack

Assign the Application to the identity center user

Go to your application endpoint and check if it's redirecting you to log in via Identity Center.

⚠️ Even after the deployment is successful, it might take a couple of minutes for Cognito to redirect to Identity Center.

Check if the application is accessible after the login

Cya!

I hope you liked this new blog post! ArgoCD is an amazing tool that allows us to achieve pure GitOps principles and it's extremely used in the DevOps world.

See you around! 👋

Using end-to-end IAM Auth to connect to RDS using RDS Proxy with AWS CDK

Thu, 09 Oct 2025 10:40:32 GMT

The goal of this blog post is to deploy a Lambda Function inside the VPC and make this Lambda Function connect to an RDS Cluster using Authentication. To increase the resiliency and reduce spikes of connections in the database, we are also going to add an RDS proxy.

All the code used in this blog post is available in my GitHub Repository.

IAM Authentication

In AWS, there are two ways of authenticating in RDS databases:

Using static credentials of users in the database (i.e., master password or user created within the database)
Using dynamic credentials, i.e., using IAM Authentication

There are several benefits of using dynamic credentials, such as: no need to handle password rotation and easy integration with AWS services (AWS Lambda, ECS Task, EC2...), since these services already assume an IAM Role that can be granted access to the database. Your security team will be very happy with you!

When using IAM Authentication, your application will issue a token (temporary password) by using the generate_db_auth_token action. This token will be used by your application as the password of the DB user that can use IAM Auth (more on that soon).

This token has a lifetime of 15 minutes and can only be used once, but once this token is used and you are connected to the database, there is no "lifetime" involved. You are in!

The steps needed to use IAM Authentication are:

Allow RDS Cluster/Instance to use IAM Authentication
Create a user in the DB that will be used during IAM Authentication. The way to create this user depends on the engine (PostgreSQL, MySQL, MariaDB) of the database.
Create an IAM Role that will be used by your application. The policy of this role should allow rds-db:connect action on the ARN of the DB user.

With that being done, your application can issue the token to get the password to connect to the database using the created DB user.

There are some limitations, or at least, points of attention in case you decide to use IAM Authentication. The most relevant, in my opinion, is the fact that IAM database authentication throttles connections at 200 connections per second, and the fact that CloudWatch and CloudTrail don't log IAM authentication.

Two hundred connections per second might look scary and a huge spike, but when we are talking about Lambda Functions (serverless) that create connections on every new lambda context creation, this can easily be a reality. To solve this problem, we can add a pool system that will manage the connections to the database to avoid this problem!

There are several open source database pool systems that you can use, e.g., pgBouncer (for PostgreSQL). In case you don't want to self-deploy and manage your DB pool system, you can use the AWS managed solution: RDS Proxy.

RDS Proxy

RDS Proxy is an AWS solution to manage a database pool of connections. It's a service that is easily integrated with RDS (unfortunately, it does not work with self-managed databases - e.g., deployed in EC2) and can help you with unpredictable surges in database traffic, as mentioned, this is very common in serverless scenarios.

RDS Proxy can connect to databases using static credentials to the database or with IAM Authentication. You know what this means? That we can have an end-to-end integration (App -> RDS Proxy -> Database) using IAM Authentication! And that's exactly what we are going to do in this blog post.

Architecture

We are going to deploy all this infrastructure using AWS CDK. Ready?

Talk is cheap, show me the code

Let's start with our application code. A simple Lambda Python code that connects to the database and runs a simple query.

import os

import boto3
import psycopg2
from dotenv import load_dotenv


def get_db_password():
    client = boto3.client("rds", region_name=os.getenv("AWS_REGION"))
    token = client.generate_db_auth_token(
        DBHostname=os.getenv("DB_HOST"),
        Port=os.getenv("DB_PORT"),
        DBUsername=os.getenv("DB_USER"),
        Region=os.getenv("AWS_REGION"),
    )
    print(f"{token = }")

    return token


def handler(event, context):
    print("------ LAMBDA STARTED ------")

    try:
        if os.getenv("USE_IAM_AUTH").lower() == "true":
            password = get_db_password()
        else:
            password = os.getenv("DB_PASSWORD")
        conn = psycopg2.connect(
            host=os.getenv("DB_HOST"),
            user=os.getenv("DB_USER"),
            password=password,
            port=os.getenv("DB_PORT"),
            dbname=os.getenv("DB_NAME"),
            sslmode="require",
        )

        cursor = conn.cursor()
        cursor.execute("SELECT current_database(), current_user;")
        record = cursor.fetchall()[0]
        db_name, db_user = record
        print(f"Connected to database '{db_name}' with user '{db_user}'...")
    except Exception as e:
        raise e


if __name__ == "__main__":
    if os.getenv("LOCAL_DEVELOPMENT"):
        load_dotenv()
        handler({}, {})

The code is simple. You can connect via static credentials or with IAM Auth (if the environment variable USE_IAM_AUTH is set to TRUE).

Now, let's take a look at the CDK Code.

import { ArnFormat, Duration, Fn, Stack, StackProps, Tags } from "aws-cdk-lib";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as rds from "aws-cdk-lib/aws-rds";
import { Construct } from "constructs";
import * as rdsSql from "cdk-rds-sql";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as iam from "aws-cdk-lib/aws-iam";
import * as ecrAssets from "aws-cdk-lib/aws-ecr-assets";
import * as path from "path";

export class IamAuthStack extends Stack {
  vpc: ec2.Vpc;

  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    Tags.of(this).add("Repo", "github.com/felipelaptrin/iam-authentication-rds");

    this.vpc = this.setVpc();
    const dbName = "app";
    const dbUser = "iamuserapp";

    const { cluster, clusterProxy, clusterProxySg, clusterProxyId } = this.createDatabase(dbName, dbUser);
    this.setRdsIamAuthentication(dbName, dbUser, cluster);
    this.setBackend(dbUser, cluster, clusterProxy, clusterProxySg, clusterProxyId);
  }

  setVpc(): ec2.Vpc {
    const vpc = new ec2.Vpc(this, "Vpc", {
      ipAddresses: ec2.IpAddresses.cidr("10.220.0.0/16"),
      maxAzs: 2,
      natGateways: 1,
      subnetConfiguration: [
        {
          cidrMask: 24,
          name: "public",
          subnetType: ec2.SubnetType.PUBLIC,
        },
        {
          cidrMask: 24,
          name: "private",
          subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
        },
        {
          cidrMask: 24,
          name: "databases",
          subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
        },
      ],
    });

    return vpc;
  }

  createDatabase(
    dbName: string,
    dbUser: string,
  ): {
    cluster: rds.DatabaseCluster;
    clusterProxy: rds.CfnDBProxy;
    clusterProxySg: ec2.SecurityGroup;
    clusterProxyId: string;
  } {
    const cluster = new rds.DatabaseCluster(this, "DatabaseCluster", {
      vpc: this.vpc,
      iamAuthentication: true,
      serverlessV2MaxCapacity: 2,
      serverlessV2MinCapacity: 0.5,
      autoMinorVersionUpgrade: true,
      engine: rds.DatabaseClusterEngine.auroraPostgres({
        version: rds.AuroraPostgresEngineVersion.VER_17_4,
      }),
      writer: rds.ClusterInstance.serverlessV2("writer"),
      vpcSubnets: this.vpc.selectSubnets({
        subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
      }),
      defaultDatabaseName: dbName,
      deletionProtection: false,
    });

    const proxyRole = new iam.Role(this, "PostgresClusterRdsProxyRole", {
      assumedBy: new iam.ServicePrincipal("rds.amazonaws.com"),
    });
    cluster.secret!.grantRead(proxyRole);
    cluster.grantConnect(proxyRole, dbUser);

    const clusterProxySg = new ec2.SecurityGroup(this, "PostgresClusterRdsProxySecurityGroup", {
      vpc: this.vpc,
    });
    cluster.connections.allowFrom(clusterProxySg, ec2.Port.POSTGRES);

    // L2 Construct rds.DatabaseProxy does not support modifying the DefaulAuthScheme to IAM. So we should use L1 construct instead
    const clusterProxy = new rds.CfnDBProxy(this, "PgProxyIamE2E", {
      dbProxyName: "PostgreSqlProxy",
      roleArn: proxyRole.roleArn,
      engineFamily: "POSTGRESQL",
      vpcSubnetIds: this.vpc.isolatedSubnets.map((subnet) => subnet.subnetId),
      vpcSecurityGroupIds: [clusterProxySg.securityGroupId],
      requireTls: true,
      defaultAuthScheme: "IAM_AUTH",
    });

    new rds.CfnDBProxyTargetGroup(this, "PgProxyTG", {
      dbProxyName: clusterProxy.dbProxyName!,
      targetGroupName: "default",
      connectionPoolConfigurationInfo: {},
      dbClusterIdentifiers: [cluster.clusterIdentifier],
    });

    const clusterProxyId = Fn.select(1, Fn.split("db-proxy:", clusterProxy.attrDbProxyArn));

    return {
      cluster,
      clusterProxy,
      clusterProxySg,
      clusterProxyId,
    };
  }

  setRdsIamAuthentication(dbName: string, dbUser: string, dbCluster: rds.DatabaseCluster): void {
    const provider = new rdsSql.Provider(this, "Provider", {
      vpc: this.vpc,
      cluster: dbCluster,
      secret: dbCluster.secret!,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
      },
    });

    new rdsSql.Sql(this, "CreateIamAuthUser", {
      provider: provider,
      statement: `
        CREATE ROLE ${dbUser} LOGIN;
        GRANT rds_iam TO ${dbUser};
        GRANT CONNECT ON DATABASE ${dbName} TO ${dbUser};
        GRANT USAGE ON SCHEMA public TO ${dbUser};
      `,
    });
  }

  setBackend(
    dbUser: string,
    dbCluster: rds.DatabaseCluster,
    dbProxy: rds.CfnDBProxy,
    dbProxySg: ec2.SecurityGroup,
    dbProxyId: string,
  ): void {
    const lambdaFunction = new lambda.DockerImageFunction(this, "BackendFunction", {
      vpc: this.vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
      },
      code: lambda.DockerImageCode.fromImageAsset(path.join(__dirname, "../backend"), {
        platform: ecrAssets.Platform.LINUX_AMD64,
      }),
      environment: {
        USE_IAM_AUTH: "TRUE",
        DB_HOST: dbProxy.attrEndpoint,
        DB_NAME: dbCluster.secret!.secretValueFromJson("dbname").unsafeUnwrap(),
        DB_PORT: dbCluster.secret!.secretValueFromJson("port").unsafeUnwrap(),
        DB_USER: dbUser,
      },
      timeout: Duration.seconds(10),
      memorySize: 1024,
      functionName: "TestIamAuth",
    });
    const rdsProxyDbUserArn = Stack.of(this).formatArn({
      service: "rds-db",
      resource: "dbuser",
      resourceName: `${dbProxyId}/iamuserapp`,
      arnFormat: ArnFormat.COLON_RESOURCE_NAME,
    });
    lambdaFunction.addToRolePolicy(
      new iam.PolicyStatement({
        actions: ["rds-db:connect"],
        resources: [rdsProxyDbUserArn],
      }),
    );
    lambdaFunction.connections.allowTo(dbProxySg, ec2.Port.POSTGRES);
  }
}

In the CDK code we are creating:

VPC with public (for the NAT Gateway), private (for CDK-RDS-SQL module), and isolated (for RDS Cluster and RDS Proxy)
Serverless RDS Cluster (but it didn't have to be serverless)
RDS Proxy (here I've used a L1 construct since the current L2 construct of RDS Proxy does not support IAM Authentication)
Docker Lambda Function

Pay attention to the following security group rules:

RDS Cluster should allow inbound traffic coming from the RDS Proxy
RDS Proxy should allow inbound traffic coming from the Lambda Function

This is because we are aiming for the following flow: (Lambda Function -> RDS Proxy -> RDS Cluster).

The creation of the DB user that will be used in the IAM Auth was created using cdk-rds-sql construct. This is needed because the RDS sits in an isolated subnet, meaning that when the code runs, it will run on our machine, and our machine can't connect to the database without a VPN. The idea behind CDK-rDS-SQL is to deploy a Lambda Function inside the VPC that can reach the database and execute queries.

Cya!

That's it! We've implemented a full end-to-end IAM Auth to connect to RDS using RDS Proxy! This is the basic idea, and you can adapt this to your needs (e.g., add more perms to the DB user to actually use the database). I hope you've liked it!

See you around! 👋

Ephemeral Environment for frontend

Thu, 04 Sep 2025 17:20:32 GMT

Today I will cover a strategy for ephemeral environments for frontend (static websites) applications, i.e., a short-lived frontend deployment will exist during the pull request lifecycle, and this will help the developers and testers make sure the code change implements what is expected, without having to merge to dev environment to test.

For this demo I will use Terraform, GitHub Actions and AWS (CloudFront + S3 for the frontend). In case you do not know how to deploy a static frontend application, I recommend you check two blog posts I've posted in the past: Deploying a static website on AWS using S3 and CloudFront and Deploying a serverless, cheap, and scalable infrastructure using Terraform in AWS .

All the code used for this demo is available in my GitHub Repository!

Architecture

The architecture of this demo is fairly simple but has some interesting points to be commented on.

This demo will not deploy a backend application, but usually the static website interacts with a backend application, so that's why I added it in the architecture diagram. The code for this demo involves the deployment of a dev environment frontend and an ephemeral environment frontend. This is not a must-have, but I wanted to isolate the dev environment from the ephemeral environment, especially because the ephemeral environment will require a Lambda@Edge function to rewrite the path (more on that later), and the dev deployment does not need it (at least in my code it doesn't). The idea is that the ephemeral frontend environment will use the dev backend.

Our goal is to have an ephemeral environment (frontend) for every Pull Request opened and access the environment using a dedicated (subdomain-based) URL. So, the idea is actually very simple:

Have a dedicated CloudFront + S3 infrastructure for the ephemeral environment. This is very cheap!
Set up an ephemeral.<DOMAIN> certificate for the CloudFront and create the domain records pointing to the CloudFront.
For every Pull Request, build the code and push it to the S3 bucket that serves as the origin of this distribution. Objects in S3 will be prefixed by the pull request number, e.g., pr-10.
A viewer-request Lambda@Edge function will be used to rewrite the URI to the appropriate folder, i.e., if the user accesses pr-15.ephemeral.<DOMAIN>, the Lambda will fetch /pr-15/index.html in the bucket (and the same logic applies to all the other static assets).

Flow

The ephemeral environment deployed will have the same lifespan as the pull request, so we are aiming for the following flow:

Developer opens a Pull Request with code changes
GitHub Actions runs a workflow that builds and deploys the code to the S3 bucket (prefixed by the pull request number). This workflow runs for every new commit in the pull request, i.e. the ephemeral environment will always be up-to-date with the pull request code.
A comment is added to the pull request with the link to access the ephemeral environment

Developer/QA/Tester checks the ephemeral environment and verifies the changes
Pull Request gets approved and merged
GitHub Action runs a workflow to delete the ephemeral environment (i.e., deletes files from S3) and the comment in the Pull Request with the URL.

This is a simple flow that is very powerful and can be extended to meet different needs. In this demo, I've used Vite template and deployed it to the dev environment.

When the Pull Request was opened to modify the message, the following ephemeral environment was deployed:

Check the URL of each screenshot! As you can see, I'm deploying an ephemeral environment for the frontend. You can even check my Pull Request here.

Code

I don't think it will be beneficial to copy and paste code from the demo GitHub Repository. So I recommend that you go over the code there and deploy it yourself!

Considerations

I think some DevOps Engineers might disagree that what we have here is an ephemeral environment, since there is already an environment in place (S3 + CloudFront + Lambda@Edge deployed beforehand). I don't agree with this point of view since, in the end, we have an isolated environment per pull request. Also, CloudFront is a slow service, in the sense that it takes a long time to be provisioned, so I don't think it's smart to create it on the fly. Also, by having dedicated ephemeral environment resources already deployed, we speed up the deployment and avoid some quota limitations. These services are billed based on usage, so if no pull request is made, then you basically won't get billed.

The frontend code used in this project was a Vite template. In a real-world application, things are way more complex, so you should expect changes in the CI/CD, such as adding environment variables before building the frontend, or end-to-end tests, integration with SSO...

There are several other ways of achieving the same thing I did here. This solution is not a silver bullet and has pros/cons over other solutions. Be critical and adapt to the situation.

Cya!

Hope you liked this blog post! Ephemeral environment is a really interesting topic and the strategy to deploy it varies a lot based on the compute/data stack used (e.g. Kubernetes, ECS, Lambda, EC2...). Here I presented an ephemeral environment strategy for S3 + CloudFront (frontend) deployments.

See you around! 👋

AWS CDK for Terraform users

Tue, 01 Jul 2025 10:40:32 GMT

This one is going to be a very short blog post since it's a high-level guide on how to understand CDK from a Terraform user perspective. This is the guide I would like to have read the first time I worked with CDK.

I think a good thing when learning a new tool is to bring concepts, ideas, and mindset of similar tools you've worked with before. This helps you understand the pros and cons of each tool and how they handle the same problem. So for this guide, I'm expecting you to have experience with Terraform. I'm assuming you have no experience with Pulumi, CDK for Terraform, or tools that use programming languages to create infrastructure.

AWS CDK

AWS CDK (AWS Cloud Developer Kit) is a framework that allows you to define code in your favorite programming language (e.g. Typescript, Python - check the full list of supported languages in the documentation). Behind the scenes, CDK uses an AWS service called CloudFormation, that allows you to provision infrastructure using templates written in JSON/YAML.

Because CDK uses programming languages instead of a DSL (domain-specific language) like Terraform, you have way more flexibility to define your infrastructure: create functions, classes, methods, custom logic, apply DRY concepts, code patterns... This is great, especially if the DevOps has a developer background.

It's not required to know CloudFormation in order to use CDK. If you know it it's amazing since the learning curve will be smoother, but I wrote this post supposing you have no idea what CloudFormation is. For now, what you need to know is that when using CDK you write code in your favorite (and supported) programming language and then this code is transformed into a YAML/JSON that will be used by CloudFormation to deploy your infrastructure.

Bootstrapping

We refer to "bootstrap" as the process of preparing the environment before actual work begins. The idea is that certain setup steps are needed in advance to ensure everything is ready for productive work.

You might be thinking that in Terraform there is no such thing as "bootstrapping" and you are right. Well, partially. If you want to have a professional Terraform code you need to:

Store state remotely (when using AWS we use S3 Bucket)
Support state locking (when using AWS we used to use DynamoDB Table but this is now deprecated and you should use S3 State Locking)

Terraform calls this backend, but it practice it's a bootstrap proccess: need to have something in place (S3 Bucket with State Locking) before actually starting your Terraform code. This is bootstrapping. I know that some of you might say "hey, but I can start to work locally and only then migrate my state to S3 bucket"... Yes, that's true but if you start your Terraform code from day one following best practices you can treat the S3 Bucket creation as a "bootstrap" proccess.

Differently from Terraform, bootstrapping in CDK is a pre requisite: you MUST bootstrap your AWS environment in order to use CDK. When I say environment I'm talking about an AWS Region in a specific AWS Account. So the bootstrapping process must be performed in EVERY AWS environment that you plan to use CDK. This process is usually a manual process (or in a pipeline) and it's easily done using aws cli.

The bootstrap process will create a CloudFormation stack (named CDKToolkit) to deploy resources that are needed by CDK: S3 bucket (to store files assets - such as inline lambda functions, website files, CloudFormation stacks), Parameter Store (tracks bootstrap environment metadata - used internally by CDK), ECR repository (to build and deploy images locally), IAM Role/policy (Role to perform deployments, upload files/assets to S3, push image to ECR)... In case you are confused, don't worry. You probably won't need to use these resources. Just think these are needed resources in order to have CDK working in your AWS environment. In reality, some of these resources might not even be used by CDK, it will depend on how your code will be written.

The process of bootstrapping, as mentioned, is as easy as running a single AWS CLI command:

cdk bootstrap aws://<AWS_ACCOUNT_NUMBER>/<AWS_REGION>

Well, this is actually the simplest way to bootstrap an AWS Environment but you can use several other flags.

State

When using Infrastructure as Code (IaC), our goal is idempotency, i.e. running the same code multiple times should consistently produce the same infrastructure. In other words, applying the same code should always result in the same outcome. Behind the scenes, the IaC tool should compare the desired state (defined in code) with the current state (what's deployed), detect any differences (drift), and make the necessary changes to align them.

In Terraform, this process is managed through state files stored in S3 Bucket. A state file is a JSON-formatted file that stores metadata about the resources Terraform has deployed, including a one-to-one mapping between each deployed resource and the corresponding resource instance defined in the code.

This file is crucial because it enables Terraform to track what it created, how it was configured, and how it relates to the desired configuration. In short, Terraform uses three things to deploy infrastructure correctly:

The state file (current known state)
The desired configuration (your code)
The actual deployed state (what's in your cloud provider).

By comparing these, Terraform can determine what needs to change and apply only the necessary updates.

As already mentioned, CDK uses CloudFormation, which is an AWS-managed service, meaning that we are not responsible for managing the state (e.g. storing in an S3 bucket) because this is completely managed by CloudFormation. All you need to worry about is your CDK code and CloudFormation will be responsible for managing the resources.

CDK Concepts

We are getting somewhere! Now, let's understand CDK core concepts and how it's related to the code. I will use Typescript for this demo.

The core idea of AWS CDK is to define an application (App) that contains one or more stacks, and each stack is composed of constructs (reusable building blocks that define AWS resources). Since CDK translates your code into a CloudFormation template, so many terms used in CDK, like stack, originate from CloudFormation itself.

App

The entry point of CDK is the App, which provides a context for all the resources that will be instantiated by the app.

const app = new cdk.App();

The app will be used to create a stack.

Constructs

A construct is the building block of CDK, i.e. what is really gonna be used to create resources inside stacks.

A construct in Typescript is simply a class that expects three main arguments:

scope: The construct's parent or owner, either a stack or another construct. It decides where this construct lives in the construct tree. For example, you can create a custom construct to set up a common API using API Gateway and Lambda, and inside it, you’d use the API GW and Lambda constructs. Most of the time, you’ll just pass this (or self in Python), which refers to the current construct.
id: A unique ID for the construct within its scope. This helps CDK and CloudFormation keep track of resources. Be careful with this. If you change the ID, CloudFormation will think it’s a new resource and replace the old one.
props: These are the settings or options you pass to configure the construct. For example, in an L2 (keep reading and I will explain this!) S3 construct, you could set a versioned flag to true if you want the bucket to keep versions of files.

A construct represents one - or more - AWS resources, depending on the level of abstraction of the construct. There are three levels of abstraction for the constructs:

L1 (Cfn): The lower level of abstraction is using the CloudFormation resource directly (e.g. S3 bucket). The L1 constructs are also known as Cfn and are usually avoided when a higher level construct is available because they are lower level components with no validation, default values (beyond what is default what CloudFront already has), no best practices, hard to maintain.

import * as cdk from 'aws-cdk-lib';

class HelloCdkStack extends Stack {
  constructor(scope: App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const bucket = new s3.CfnBucket(this, "MyBucket", {
        bucketName: "MyBucket"
    });
  }
}

L2: Most times you are going to use L2 constructs because they are L1 constructs with default values, best practices, and methods for easily using the resource. Let's take for example the S3 Bucket L2 construct. It applies best practice by default and allows you to use several methods for managing lifecycle, and access (read/write access to Roles)... You can think of them as Terraform community modules for simple resources (e.g. S3, KMS).

import * as cdk from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';

class HelloCdkStack extends Stack {
  constructor(scope: App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'MyBucket', {
        versioned: true
    });
  }
}

L3: The highest level of abstraction. L3 constructs implements common AWS patterns (API Gateway with Lambda, ECS with Fargate Service and Load Balancer, S3 with CloudFront...). Usually creates several resources and are also equivalent to Terraform complex modules (e.g. CloudFront S3 CDN). But nothing stops you from building your own construct (e.g. an S3 bucket with compliance that should be used as a "default" bucket in your CDK code). You can find several L3 constructs in the Construct Hub.

CDK Deployment

There are several methods to deploy your infrastructure, such as:

Manual Deployment: Easiest and simplest to start working with CDK. You simply run cdk deploy and that's it. Well, you shouldn't use this approach when working in a team. Ideally, you want to make sure everyone has visibility of the changes so usually a CI/CD system is the way to go.
CI/CD Deployment: You can use your favorite CI/CD tools (e.g. GitHub Actions, CircleCI, GitLab) to set up pipelines to deploy your CID code.
CDK Pipelines Deployment: I've made an entire blog post to talk only about this method. We can use CDK to define our CI/CD using AWS Codepipeline. This allows us to have CDK as the truly single source of truth in our infrastructure!

Cya

Hope you enjoyed this blog post! It was very short but this would definitely save me hours from reading the HUGE CDK documentation. From now on I expect you to have the essential CDK knowledge to get started with CDK. See you in the next post! 👋

Set up Headscale with own DERP server in AWS

Mon, 02 Jun 2025 10:40:32 GMT

Some blog posts ago I talked about bastion host and private resources. I always mentioned VPN but never wrote a blog post about how to deploy one. Today I'm finally doing that. The goal of this blog post is to deploy a VPN (Headscale) and access private resources deployed in AWS.

Hub-Spoke and Mesh VPN

A VPN (Virtual Private Network) server is a mechanism to allow secure connection to public or private resources (such as corporate network, internal systems) using a public network.

Traditional VPNs follow the "hub-and-spoke" model, i.e. all clients connect to a central gateway and this gateway forwards the traffic to the target destination. This model works well and it's still heavily used today but it has some downsides:

Multiple locations: Modern and big companies have way more than a single place/location of interest. They usually have multiple offices, cloud providers (AWS, Azure, GCP...), and deployment in several regions/AZ. This means that several tunnels will need to be created among these locations which can be a pain to maintain and configure.
Latency: If all the traffic needs to go through a central gateway server it will increase latency and affect the user experience when trying to connect to the resources. Maybe the user is physically located close to the resource it wants to access but the traffic will still need to go to the VPN gateway (that might be far from the client).
Single point of failure: If the VPN gateway goes down, the traffic also goes down and this means no connection to the resources.

What if we implement a peer-to-peer design? Where every node could directly reach each other? That's the Mesh VPN! This concept is not new but lately, it's becoming more popular. It was not a preferred approach of the IT team due to the difficulty to deploy/configure/maintain, resource-intensive (especially for devices with limited CPU/RAM) but lately Mesh VPNs (such as Tailscale, Nebula, Zerotier) started to grow and become popular. In a mesh network, every node would need to create several tunnels and know how to reach other nodes (i.e. their IP in a world where devices rarely have static IP addresses). The first problem can be solved by using a fast, modern, and lightweight VPN tunnel, such as WireGuard (the big brain behind Tailscale/Headscale). The second is solvable by adding a new component: control plane. The control plane (also known as the coordinator server) is the main contact point of every node in the mesh VPN. The control plane is responsible for gathering all public keys of every node in the mesh network and sharing them with all the other nodes. Please notice that the control plane does not forward traffic or act as a main gateway. The control plane only provides information/metadata about the network!

The client in the mesh VPN can connect with its peers by using several advanced network techniques such as NAT Traversal (basically the VPN client will perform a Hole Punch) and DERP servers (when NAT traversal is not possible to be done the traffic goes through a public server called DERP - the connection is slower than doing a NAT Traversal but it's better than not having the connection!).

Headscale

In this demo we are going to deploy the following infrastructure:

Notice the following components:

Headscale Control Plane: Tailscale control plane is not open-source and is a hosted service run by Tailscale Inc. The open-source version of it is called Headscale. We are going to deploy the Headscale in an EC2 server exposed by a Caddy web server that is going to automatically manage and issue a public certificate using Let's Encrypt. Because we don't want to keep connecting to this EC2 to have access to the terminal to run the Headscale CLI to add/remove clients to the VPN we are also going to install Headscale Admin a simple GUI to interact with Headscale via an API Key. Please notice that the Tailscale client is in fact open-source and used to connect to a Tailnet (Tailscale Network).
Subnet Router: The subnet router is a Tailscale client that will serve as a router to private networks when you can not install a Tailscale client in the private resources. A good example of that is an RDS instance. We do not have access to the operational system of the RDS, so we can not install Tailscale there. So the solution to this is to deploy a Tailscale client that has access to the RDS and route traffic to the RDS. The best practice is to deploy the subnet router in a public subnet with a public IP address to ensure direct connection and optimal performance. If you deploy the subnet router in a private subnet it will need to use the AWS NAT Gateway which is a Hard NAT that will prevent direct connections (forcing it to use the DERP server - which is slow).
DERP Server: Ideally you should deploy one DERP server per region to have multiple DERP servers available and minimize latency to the clients. In reality, the DERP server is not required in this setup but it's highly recommended to deploy your own DERP server. If you do not use your own DERP server, it will use a Tailscale DERP servers, which means the traffic will go through the Tailscale infrastructure. The traffic is encrypted, and Tailscale won't be able to read the packages but if you came to the point of deploying your own control plane, then you probably want to keep everything on your side and control all the traffic.
Private Hosted Zone: As we already discussed in previous blog posts, it's not recommended to expose dev tools (e.g. ArgoCD, Grafana...) and ideally your developers should access them via VPN. To be one step extra in the best practices, we can deploy our own internal hosted zone, i.e. a DNS that will only work for the hosts that are within the internal AWS VPC. In other words, a hostname like grafana.internal will only be resolved by those that are within the VPC. A common alternative scenario is to have a public hosted zone that points to a private IP, i.e. anyone will be able to resolve the domain (e.g. grafana.mydomain.tld but only does that are internal to the VPC will be able to have network access to it). Here we are doing an extra security step by deploying our own private hosted zone.
Private Service: To simulate a private service that should only be accessed by those devices inside the VPC, I will deploy an NGINX server in a private EC2 instance.
RDS Instance: A database will also be deployed in a private subnet.

Hands-On

There is one thing that you should have in place before the hands-on: make sure you have a hosted zone configured in Route 53.

Since the code for this demo is very big I won't be copying and pasting it here, please check it in my GitHub Repository.

Install dependencies of the repository

devbox shell

Initialize Terraform

cd infrastructure
terraform init

Create a file to define variables

You can check the README of the repo to see the required variables (or check variables.tf file directly).

Here is one example of a valid file (modify it for your use case)

# vars.tfvars
aws_region = "us-east-1"
domain = "mydomain.tld"
headscale_acme_email = "me@mydomain.tld"
headscale_hostname = "headscale.mydomain.tld"

Once again: make sure the domain is already deployed as a public hosted zone in Route 53.

Apply Terraform and create the infrastructure

Remember to have valid AWS credentials in your terminal!

terraform apply --var-file=vars.tfvars

Wait until Headscale gets deployed

The user_data script will run in all EC2. You should wait for the user_data of the headscale control plane gets ready. You can check if the admin (headscale community console) is available. It is available in the /admin route, i.e. if you defined headscale_hostname as headscale.mydomain.tld, then you will need to wait https://headscale.mydomain.tld/admin to be available.

Access Headscale Admin

Use the API Key generated by the Headscale server. You can get the API Key value in Parameter Store via AWS Console or via AWS CLI running the command aws ssm get-parameter --with-decryption --name "/headscale/headscale-admin/api-key" --query "Parameter.Value" --output text. Refresh the page when you save the API Key value.

You can also get the API Key from the AWS Console.

Authorize Subnet Router route

Go to the Routes session and authorize the route created by the Subnet Router.

Create a new user for yourself

Give it a name and issue a PreAuth Key, we will use it in the next step.

Connect to Tailnet (Tailscale network)

Now that you have the PreAuth Key for your user, let's connect using the CLI. Substitute the <PRE_AUTH_KEY> with the one you generated in the previous step and <HEADSCALE_HOSTNAME> with what you defined on the Terraform vars.

tailscale up --accept-dns --accept-routes \
 --login-server=https://<HEADSCALE_HOSTNAME> \
  --auth-key=<PRE_AUTH_KEY>

PS: You need to have Tailscale client installed!

Run tailscale status. You will see the subnet-router there!

If you check the Headscale Admin dashboard it should look like this:

Reach private server

We deployed an EC2 with an NGINX installed (pretend it's a real application). The internal Route53 Hosted zone has a DNS record called service.internal that points to this service. Open your browser and try to reach http://service.internal/.

The Subnet Router is being configured as the DNS server of the Tailnet in the Headscale configuration. Because of that, we are also installing dnsmasq to resolve all ".internal" domains, that's why you are able to resolve service.internal.

Access Database

I'm using DBeaver to connect to the MySQL database. Check AWS Secrets Manager for the secret that contains the root credential.

Disconnect from the Tailnet

tailscale down

Try to reach the private server or the database! You won't be able to reach, us as we expected!

Cya!

I hope you've like it and that this demo and explanation was helpful!

See you around! 👋

Hands-on SSO with Terraform and Authentik

Sat, 03 May 2025 10:40:32 GMT

It's been a while since I wanted to write this blog post! I do not intend to follow best practices to deploy Authentik or be extremely detailed about how SAML/OIDC works in depth but I will give you a good overview and flows to explain how it works. The idea of this post is to explain how SAML/OAuth/OIDC works, and how to configure a SAML and OIDC application using Authentik. I highly recommend you check the GitHub repository that contains the code used in the blog post!

At the end of this blog post you should be able to create the following infrastructure:

SSO and common scenario

As I have mentioned in previous blog posts, my preferred way to understand a topic is by answering generic questions about it. Once the basics are clear, it's easier to deep into the topic. This time, I will stick to this method, but before getting into the questions, I will propose a scenario. So, without further ado:

Imagine working for a company that uses several apps/services. It doesn't really matter what services you use, but if you are a more visual person, you can think of apps such as SalesForce, PowerBI, ArgoCD, Gmail, etc. The most dummy way to manage credentials to all these apps is to create an account (probably IT would do this for you) in every service and then you would have to set up a password for each application. This is tedious because of several reasons:

Now you have to memorize a bunch of credentials and manage the rotating (change of password) of each one when needed. Sure, you can use a vault to store your credentials, but still, very tedious.
You don't have an easy way to check which apps you have access to. You would need to save all the URLs in your browser bookmark.
For each new service that you want to have access to, you will have to create a new password, manage the rotation, and memorize/save the URL to access the application.

This is tedious, kinda-insecure (do all apps implement good password policies and 2FA?), and time-consuming. Ideally what you want is to have SSO (Single Sign-On), which is a single place that will manage your access to the apps. To allow SSO you will need a component (technically it's called an Identity Provider) responsible for:

Centralized user identities, i.e. the users of your organization will be registered in a central place (a "centralized database"). This way your users only need to worry about these single credentials.
Handling all the communication/security between itself and the users and itself and the applications. Here is exactly where SAML and OIDC come into play.

SAML

SAML (Security Assertion Markup Language) is a standard to exchange authentication and authorization between two parts using XML. In our previous scenario, SAML would be used by the Identity Provider and the applications.

To use SAML, both parts must "speak" SAML, i.e. they should implement SAML in their code. There are three SAML versions (1.0, 1.1, and 2.0), where SAML 2.0 is the latest and by far the most widely used version. When you hear someone saying "SAML" they are probably talking about the 2.0 version.

SAML Concepts

SAML uses several technical terms that can intimidate you when trying to learn about it. Let's define them using simple words, then I will show you the classic sequence diagram.

Identity Provider (IdP): It's the heart of the SSO! It's responsible for storing the identities of the users and contains the "logic" to speak with the applications and users.
Service Provider (SP): The SP is the application that users will access! As the name suggests, a service provider will provide a service for the user (e.g. SalesForce, Gmail, PowerBI...). This is exactly what the user wants to use. The user is not interested in the IdP part, he wants to access the applications that will be accessible by using the IdP (instead of a specific single user/password created for this service). The service provider must be able to communicate using SAML protocol with the IdP, i.e. you need to confirm in the documentation of the application if it supports SAML.
SAML Request: Corresponds to the XML message that the SP will send to the IdP. In plain English, this would be something such as "Hey IdP, there is a user that is trying to log in. Can you validate his credentials? I will wait for you to validate his credentials (authentication) and log him in if you say the credentials are valid, and I will only allow him to do things he is allowed to (authorization)". You might be wondering, "Why would the SP trust another entity to validate if the user can log in into it?". This is because the SP and the IdP were previously configured to allow this using SAML protocol (more on that later).
SAML Response: Corresponds to the XML message that the IdP will send to the SP. In plain English, this would be something such as "Hey SP, I validated that these user credentials are correct, but make them valid only for the next X minutes/hours. This user is actually John Doe, and should have the role 'admin'. You can validate that this message was issued by me and didn't suffer a man-in-the-middle attack by checking my digital signature".
Binding: The IdP and SP communicate via the browser exchanging HTTPS messages. We call "binding" the mechanism to transport these messages. The most common ways of exchanging these messages are via HTTP POST (SAML message is sent via HTML form with base64 encoding) or HTTP redirect (SAML message is sent via HTTP GET with URL query string). You might be thinking... which one should I choose? This should be mentioned in the SP documentation!
Assertion: It is the main part of the SAML response. The SAML response contains assertions (usually it only contains a single one) and more things (such as issuer, and status...), but the most important information is the assertion, which contains important pieces of information: subject (who the user is - an important field here is the NameID that identifies the user usually via email or id), conditions (time window of validity of the assertion), issuer (the IdP), attributes (email, roles, department)... The assertion is signed with IdP digital certificate keys, so this can be validated to make sure the message was not tampered.
Assertion Consumer Service (ACS): The ACS is simply the SP endpoint (e.g. https:/myservice.com/saml) that the IdP will send the XML data. Remember that SAML is a protocol that uses XML messages and these messages are transported using the HTTP protocol. You should check this endpoint in the documentation of your SP. In other words, the SP is the entire application, while the ACS is the endpoint of the SP.
SAML Flow: SAML allows two flows: SP-initiated and IdP-initiated. As the name suggests, in the SP-initiated flow the service provider (application) is the one that starts the communication with the Identity Provider (IdP). This flow happens if the user accesses the SP URL directly (and clicks to log in using the IdP). The IdP-initiated flow is the opposite, the IdP starts the SAML flow and basically redirects the user to the ACS. This flow usually happens when the user is already logged in and from the IdP Portal clicks to access an application. In practice, the only technical difference between these two flows is that the IdP-initiated flow won't have a Relay State (more on that below).
Relay State: Imagine the user bookmark a specific URL page of the SP (e.g. a specific Dashboard in PowerBI, a specific App in ArgoCD...). If he accesses this link directly the SP-initiated flow will be used. And if RelayState didn't exist, you would expect, at the end of the flow, the user to be redirected to the homepage of the service provided, not the specific URL that he wanted to access at first. The RelayState is basically the information of the page that the user wants to access when the SAML login is completed. This information can't be lost during the entire flow, so the RelayState information is passed from the start of the SAML request until the SP validates the SAML assertion and redirects the user to the page.
Metadata: When configuring SAML it's common to see the SP or IdP exposing its metadata, i.e. information about itself that should be known by the other part (URL, certificates, binding...), allowing a more automatic configuration. This usually can be exported as an XML file or exposed via HTTP endpoint.

Now that you know all these technical words let's jump to the diagrams!

Here is the SP-initiated flow.

Notice that the SP initiated the SAML flow, that's why the flow is called SP-initiated. It's important to understand that in both flows, the SP and IdP do not communicate directly, this is all done by the user browser (front channel), with redirects and HTTP requests to the endpoints (e.g. ACS). Now let's check the IdP-initiated SAML flow.

The IdP-initiated SAML flow starts with the IdP already generating the SAML response with assertion, so a "SAML request" does not exist in the IdP-initiated flow.

OIDC

Similarly to SAML, OIDC (OpenID Connect) is a standard for exchanging authentication and authorization using JSON. This standard was built on top of OAuth: a standard that only implemented authorization (that's why we don't use OAuth for SSO). OAuth's first version (1.0) was born in 2009 but nowadays it's rarely used, since OAuth 2.0 (created in 2013) is more flexible and simpler to implement. OIDC implements the authentication layer of OAuth 2.0 and it's a common standard used with SSO these days. The general rule is: that new apps use OIDC, while legacy/enterprise apps use SAML.

OAuth/OIDC Concepts

Let's check some concepts that are present in both OAuth and OIDC.

Basically, there are four actors involved:

Resource Owner: The user that owns data/resources and would like to grant access to another part. This is usually a human (me, you, etc) that would like to share information with a system. Have you ever seen the "Log in with Gmail" instead of creating from scratch your user in a platform? That's it, you are the owner of your information (name, email, phone number) and would like to share these. Resource Owner is also known as simply "User".
Resource Server: This corresponds to the server/service that contains the resources (e.g. Google Drive, Grafana...).
Client The client is the application that wants to access the resources on behalf of the resource owner (user). Here is a practical example: I usually use DrawIO to create the diagrams for my blog post. I can integrate the diagram created with Google Drive (so I don't have to store it locally). DrawIO is the client (because he wants to access my Google Drive on my behalf) and Google Drive is the Resource Server.
Authorization Server: It's responsible for issuing access tokens for the client to access the resources. In the DrawIO example I just mentioned, this would be the Google Accounts service. Sometimes the Authorization Server and Resource Server is the same thing (same system). Although this is not technically 100% accurate the Authorization Server is also known as Identity Provider (IdP).

Before the OAuth/OIDC flow starts the client and authorization server should be configured beforehand.

Client type: The client can be placed in one of the following categories when adding a new client to the authorization server: confidential client and public client (non-confidential). The concept of public here doesn't mean publicly accessible, it means an application that cannot safely store secrets. A SPA can not store secrets securely because the only way for the application to store this is in the source code, which is available if you inspect the website. Because of this, confidential clients (i.e. server-side hosted applications) are the only type of clients that store client secret. Public clients use PKCE (Proof Key for Code Exchange) a one-time code challenge.
Client ID and Client Secret: The authentication server needs to create in its configuration a new application/client. This gives us a client ID (nonsensitive) and a client secret (sensitive - if the client is confidential). These are used later to obtain the access token when using the authorization code flow (keep reading and you will understand this!).

Some important concepts in the flow:

Authorization Code: Corresponds to a short-lived (around 10 minutes), one-time-only use code that is used to prove to the client that the user logged in and gave permissions. The authorization code is meaningless to the resource server and it's obtained with browser communication (front channel), which is less secure (is it secure, don't get me wrong, but it's considered less secure than a server to server - i.e. back channel - communication), that's why the authorization code is not sufficient to access the protected resources.
Scopes: We don't want the client to access the resource server with permissions to execute whatever he wants, we would like to limit the access. Using the DrawIO example, we would like to grant DrawIO (the client) to see/create its own configuration data in the user Google Drive(https://www.googleapis.com/auth/drive.appdata scope) but not view the user photos, videos, and albums in Google Photos (https://www.googleapis.com/auth/drive.photos.readonly scope). The user needs to be aware of the actions that the client will do on the user's behalf, so that's why a consent screen is usually presented after the user logs in to the authorization server.
Redirect URI: The Redirect URI (also known as callback URL) is the URL that the user needs to reach back with the Authorization code that was given by the Authorization Server. The flow is stateless and the communication happens using the browser as an intermediate layer (except for the Access Token exchange), so the client needs to specify this URL when the OAuth/OIDC process starts.
Access Token: Differently from the Authorization Code, the Access Token contains the "real credential" that proves authorization to access the protected resources. It's often a signed JWT that can be validated by the resource server. This access token is limited to the scopes authorized (consented) by the user and it's obtained between the direct communication of the client and authorization server (back channel) when the client passes the client ID, client secret, and authorization code. Notice that this value can not fetched by the user, since this communication happens between the client and the authorization server.
Response Type: The response type specifies what flow the client expects the authorization server to use. OAuth/OIDC supports several flows but the most common is the Authorization Code Flow (the one I'm commenting on here!). You might have heard about Implicit flow but this is deprecated and not recommended anymore (it only uses the front channel to exchange tokens).

If you understand these concepts, you are ready to understand the OAuth 2.0 Authorization Code Flow below.

As mentioned, OIDC is built on top of Auth 2.0, so it introduces new concepts/jargon that do not exist in OAuth 2.0 that are helpful when it comes to authentication:

ID Token: A signed JWT that contains information about the user (name, email, unique ID...) that is issued by the Authorization Server. The key-value pair inside the JWT that represent the user is called claim.
/userinfo: The OIDC standard specifies an endpoint called /userinfo in the authorization server so that the client can get more information about the user if needed. Yes, the ID Token already has information about the user, but sometimes the client needs extra information that is not available in the claim. In these cases, the client will reach the /userinfo route (passing the Access Token - since it's a protected route) to get more information, if needed.
openid scope: To specify that the OIDC flow will be used, the client needs to specify the openid scope. Scope, as we just saw, is something present in OAuth 2.0, but the openid didn't exist.
/.well-known: The Authorization Server exposes all the URL capabilities, and metadata that the client needs to interact via OIDC in a route called /.well-known (instead of specifying the Auth URL, Token URL, and Userinfo Url).

You will notice that the OIDC flow below will be very similar to the OAuth 2.0.

Hands-On

The goal of this demo is to show how to configure:

Authentik as an Identity Provider
AWS as a Service Provider using SAML
Grafana as Client using OIDC
Configure Caddy as a web proxy

To achieve that I will use Terraform and deploy resources in AWS. For simplicity and to make this demo cheap I will deploy Grafana and Authentik in the same EC2. This is obviously a bad practice and you should not use this in production! Since the code is kinda big I will only paste the step-by-step guide to run the demo. Please check the repository for the code.

There are some things we need to know before hands-on starts:

Grafana only supports SAML in the Enterprise edition, so we are going to use OIDC, that is available to everyone.
AWS Identity Center can only be configured in the Management Account, so if you are planning to run this demo, make sure you are in the management account!
Make sure you have a hosted zone configured in Route 53

Install dependencies of the repository

devbox shell

Initialize Terraform

terraform init

Create a file to define variables

You can check the README below to see the variables that are required (or check variables.tf file directly).

Here is one example of a valid file (modify it for your use case)

# vars.tfvars
aws_region = "us-east-1"
domain = "mydomain.com"
use_staging_certificate = true
authentik_email = "me@mail.com"

Apply Terraform and create the infrastructure

Remember to have valid AWS credentials in your terminal!

terraform apply --var-file=vars.tfvars

This command will provision the VPC, and EC2 with elastic (static) IP and create the record in Route53 to allow Let's Encrypt to validate we own the domain (Caddy will manage and create this certificate for us). The EC2 will run the user_data script to configure Authentik and Grafana.

Wait for infrastructure to get ready

Open your browser and keep checking if the Authentik server is already ready (check the endpoint by running terraform output authentik_endpoint). It will take a couple of minutes (most likely less than 10) to be available because the user_data script needs to take a couple of minutes to install Authentik and expose using Caddy. Also, the DNS to reach the DNS needs to propagate to the network.

You can check how the progress is going by connecting via Session Manager (AWS Console) to the EC2 and running the following command:

sudo su
echo "Check Cloud Init Script"
cat /var/log/cloud-init-output.log
echo "Check Caddy Status"
systemctl status caddy

Go to AWS Console to get Identity Center URL

Now, open the AWS Console and navigate to the IAM Identity Center. Go to Settings and click Change Identity source

Select External Identity Source

We will be using Authentik as the IdP of AWS (our SP).

Copy the SP metadata provided

We are going to use these values in the Terraform Authentik configuration soon so do not close this window yet!

Get credentials to Authentik

Soon we are going to configure Authentik using Terraform. Authentik Terraform provider requires a Token to execute HTTP calls to execute actions and configure it. Also, we will need to configure Grafana's OAuth provider using the created client ID and client secret. We can get these values from the outputs of the main Terraform code we just executed.

Run the Terraform command to get these values:

terraform output authentik_endpoint
terraform output authentik_token
terraform output grafana_oauth_client_id
terraform output grafana_oauth_client_secret

If you would like to log in as admin you will use the email used to configure Authentik and the password available in the authentik_password output (terraform output authentik_password).

Initialize the Authentik Terraform module

cd authentik
terraform init

Create a file to define variables

The Authentik terraform module needs some variables to be defined. Here is an example:

# authentik/vars.tfvars
authentik_url = "https://sso.mydomain.com/" # Use the Terraform output
authentik_token = "123ABC" # Use the Terraform output
authentik_insecure = true # Set to true if you set use_staging_certificate to true

grafana_url = "https://monitoring.mydomain.top" # Use the Terraform output
grafana_client_id = "1a2b3c" # Use the Terraform output
grafana_client_secret = "123abc" # Use the Terraform output

aws_sign_in_url = "https://ABC.awsapps.com/start" # Use the AWS value
aws_acs_url ="https://us-east-1.signin.aws.amazon.com/platform/saml/acs/DEF" # Use the AWS value
aws_issuer_url = "https://us-east-1.signin.aws.amazon.com/platform/saml/GHI" # Use the AWS value

set_scim = false
aws_scim_url = ""
aws_scim_token = ""

We are not setting SCIM yet. We need to first set up the external Identity Provider for the Identity Center before configuring the SCIM.

Apply Authentik configuration

If Authentik is already available you can now configure it using Terraform

terraform apply --var-file=vars.tfvars

Here we are going to create:

SAML provider for AWS
OIDC provider for Grafana
AWS and Grafana application

Some important things to mention:

Grafana documentation specifies the redirect URI
AWS Identity Center does not support encrypted SAML assertion
AWS Identity Center supports post binding

Upload SAML metadata and IdP Cert to AWS External Identity Provider configuration

You will notice that when applying the module two files were generated: authentik/cert.pem and authentik/metadata.xml. These files can be obtained in the Authentik but for simplicity, we extracted them via Terraform. Upload these to the AWS Console:

authentik/metadata.xml should be uploaded in the "IdP SAML metadata" field
authentik/cert.pem should be uploaded in the "IdP certificate" field

PS: I noticed a bug that I couldn't fix that it might happend here. The first Terraform apply execution of the authentik module can generate and empty the metadata.xml file. If this happens simply rerun the terraform apply --var-file=vars.tfvars command and the file will be created correctly. If the file is empty AWS will complain (of course!).

Enable SCIM

Go to AWS Console on the Identity Center Settings page. You will see a message to enable SCIM. Click to enable. An endpoint and a token will be generated. Copy these values because we are going to need it the next step.

Deploy SCIM via Terraform

Modify the authentik/vars.tfvars file to enable SCIM. The SCIM variables should look like this:

set_scim = true # This should have "true" value now!
aws_scim_url = "https://scim.AWS_REGION.amazonaws.com/ABCDEF/scim/v2"
aws_scim_token = "GIANT_TOKEN"

Rerun the apply

terraform apply --var-file=vars.tfvars

This will create:

SCIM provider for AWS. AWS says that even when using an external identity provider you still need to provision all applicable users and groups into AWS Identity Center. This is very tedious, but here SCIM comes to the rescue! SCIM is a standard to simplify the exchange of user identity information between identity centers. Since AWS supports SCIM we can use and take advantage of it! In other words, every time a user (that is from the AWS group) is created, Authentik will do an API call to the AWS SCIM server to also create this user there. The AWS SCIM implementation requires some specification and has some constraints so that's why we carefully used a Python script to specify the fields we need to pass to the AWS SCIM provider. Also, another important thing to mention is that the AWS external Identity Center requires us to use email as the NameID to identify the user.
Two users: "dev" (that should only have ReadOnly permissions in the AWS account and Viewer permission in Grafana) and "admin" (should have AdministratorAccess in the AWS account and Admin permission in Grafana). We are creating passwords for these users using Terraform, this is not ideal in a real-world scenario: once again, this is a demo.
Groups: AWS groups (all users that need AWS access should be placed here since in a real-world scenario we don't want to create users that do not need AWS access to be created there), Admin AWS group, Dev AWS group, Grafana Admin group, Granafa Viewer group.

Manually sync the SCIM

In case SCIM was not synced (i.e. users were not created in AWS IAM Identity Center) we should manually sync before moving on. Log in as admin (the email you set up and password of authentik_password in the main terraform code) and go to Applications > Providers > aws-scim > hit the sync button. Now, you can see in the AWS IAM Identity Center console (Users and Groups section) the users and groups created by the Authentik Terraform module.

Initialize the AWS Organizations Terraform module

In the previous module, we created the users and groups. Now, from the AWS side, we need to configure this group to access the AWS account and configure their permissions. First, let's initialize the Terraform module.

cd ../organizations
terraform init

Create a file to define variables

# organizations/vars.tfvars
aws_region = "us-east-1"
instance_arn = "arn:aws:sso:::instance/ssoins-abc123"
identity_store_id = "GHI"

The instance_arn and identity_store_id variables can be obtained in the settings page of the Identity Center.

Deploy Organizations module

terraform apply --var-file=vars.tfvars

Here we create the permission sets (for administrators and developers) and assign these permission sets to the management account.

You should be able to see the following application on the Authentik home screen:

Now, let's log in the AWS using the Dev and Admin to see the differences. First, let's use the Dev user:

Now let's check with the Admin user:

We will see the same behavior when using accessing Grafana. First, let's use Dev user:

Then, admin user:

Load test your ECS deployment

Tue, 01 Apr 2025 12:05:32 GMT

In this blog post we are going to deploy an API using ECS and enable auto-scaling using AWS CDK. We will load test our application using Locust and validate in the AWS Console that our application is able to scale. In reality, a bunch of services and components will be created along the way!

As always, the code used here is available in my GitHub Repository.

Proposed Architecture

For this demo I will use a very similar architecture that I proposed on my Multi-account deployment using AWS CDK blog post. For simplicity I will skip the CodePipeline deployment and deploy everything manually, but you can easily modify the provided code here to deploy using CodePipeline if you followed the Multi-Account with CDK blog post.

Some important points to mention about the proposed architecture:

Expose application using ALB: The API will be deployed in a private subnet and the service will be exposed via Application Load Balancer (ALB). The Security Group of the API needs to allow inbound traffic from the ALB.
Cross Account Image Registry: The artifacts (docker images) will be deployed in a centralized account that can be used by other workload accounts (e.g. Dev, Staging, Prod...). This is important to assure artifact promotion and was already discussed in the Consistent deployments - a real-world case scenario blog post. Since the API is deployed in a private subnet it required to deploy a VPC Endpoint or NAT Gateway (I will use this!) to reach ECR service to pull the docker image.
HTTPs handled by ALB: The users of the API will communicate over HTTPs with the load balancer which is responsible for handling the HTTPs (using a certificate issued by AWS Certificate Manager) connection, while HTTP will be used in the communication between ALB and API (SSL termination at ALB).
Route53 should be created in advance: I will consider that a Route53 hosted zone is already in place and ready to use. If you do not have a domain I highly recommend you to buy a cheap one for testing and experimenting (you can easily find it for under 2 dollars per year!).

API

I will develop a super simple API that has two routes:

GET /api/health: A simple health check. It's a must-have route to allow the ALB to check if the task is healthy.
GET /api/count-prime/{limit}: Calculates the number of prime numbers lower than a given value. This route is enough to consume some CPU and it will work great during our load test.

Let's check our simple API code:

import time
import os
from typing import Optional

from fastapi import FastAPI, HTTPException
from fastapi.responses import JSONResponse

import random

app = FastAPI(
    title="Backend",
    description="An API with endpoints designed to simulate CPU and memory load.",
    version="1.0.0",
    root_path="/api",
)


@app.get("/health", summary="Healthcheck Endpoint")
async def healthcheck() -> dict:
    return {"status": "ok"}


def is_prime(n):
    """Check if a number is prime."""
    if n < 2:
        return False
    # Only test up to the square root of n for efficiency
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True


def count_primes(limit):
    """Count the number of prime numbers below the given limit."""
    count = 0
    for num in range(2, limit):
        if is_prime(num):
            count += 1
    return count


@app.get("/count-prime/{limit}", summary="Simulate CPU Load")
async def simulate_cpu(limit: int) -> dict:
    start_time = time.time()
    result = count_primes(limit)
    duration = time.time() - start_time

    return {"limit": limit, "result": result, "durationSeconds": duration}


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(
        "main:app", host="0.0.0.0", port=8000, workers=os.getenv("NUMBER_OF_WORKERS", 3)
    )

The github repository containing this code also contains some other pieces, such as requirements.txt (dependencies) and Dockerfile.

Load Test

Since you are already familiar with the API code, we are ready to understand the load testing. We are going to use one of the most widely used frameworks to load test: locust. The documentation is great and it's simple to use. If you have never used it, don't worry, you will see how easy it is now:

import json

import logging
from locust import HttpUser, task, between
import random

logger = logging.getLogger("logger")
logging.basicConfig(
    level=getattr(logging, "INFO", logging.INFO),
    format="%(asctime)s - %(levelname)s - %(message)s",
)


class Test(HttpUser):
    wait_time = between(1, 3)

    @task
    def prime(self):
        limit = random.randint(10_000, 20_000)
        logger.info(f"Calculating the number of prime number below {limit}...")
        response = self.client.get(f"/api/count-prime/{limit}", name="count-prime")
        response.raise_for_status()

When running locust we should specify the max number of users to simulate and the spawn rate (i.e. how fast new users will be created). The load test starts with 0 users and goes up to the maximum number of users defined in the test using the spawn rate defined (e.g. 1 new user per second).

Each user in our load test waits between 1 and 3 before sending an HTTP request to our API (and this will keep repeating until the end of the load test). The limit number to calculate the number of prime is a random number between 10,000 and 20,000.

IaaC with CDK

We are going to create two stacks in our CDK code:

EcrStack: Created ECR repository that will contain the API image. Should be deployed in the Shared Assets account.
EcsStack: Create the entire ECS infrastructure, VPC, Certificates, Route53 record... Should be deployed in the Development account.

// main.ts
#!/usr/bin/env node
import * as cdk from "aws-cdk-lib";
import { EcsStack } from "./stack/ecs";
import { EcrStack } from "./stack/ecr";
import { devConfig, sharedAssetsConfig } from "./config";

const app = new cdk.App();
new EcsStack(app, "EcsStack", devConfig);
new EcrStack(app, "EcrStack", sharedAssetsConfig);

The main code basically instantiate both stacks. Let's check in depth the EcrStack:

// stack/ecr.ts
import * as cdk from "aws-cdk-lib";
import {
  Repository,
  RepositoryEncryption,
  TagMutability,
  TagStatus,
} from "aws-cdk-lib/aws-ecr";
import { Construct } from "constructs";
import { AwsAccount } from "../config";
import { AccountPrincipal, CompositePrincipal } from "aws-cdk-lib/aws-iam";

export interface EcrProps extends cdk.StackProps {
  ecrRepository: string[];
  maxAgeOtherTagsInDays: number;
}

export class EcrStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: EcrProps) {
    super(scope, id, props);

    const awsAccount = [AwsAccount.Development];
    const allAwsAccounts = awsAccount.map(
      (accountId) => new AccountPrincipal(accountId),
    );
    const principals = new CompositePrincipal(...allAwsAccounts);

    for (const repositoryName of props.ecrRepository) {
      const repository = new Repository(this, repositoryName, {
        repositoryName: repositoryName,
        removalPolicy: cdk.RemovalPolicy.DESTROY,
        imageTagMutability: TagMutability.MUTABLE,
        encryption: RepositoryEncryption.KMS,
      });
      repository.grantPull(principals);
      repository.addLifecycleRule({
        tagPrefixList: ["dev"],
        maxImageCount: 1,
        rulePriority: 1,
      });
      repository.addLifecycleRule({
        description: `Delete images with other tags after ${props.maxAgeOtherTagsInDays} days`,
        tagStatus: TagStatus.TAGGED,
        maxImageAge: cdk.Duration.days(props.maxAgeOtherTagsInDays),
        tagPatternList: ["*"],
        rulePriority: 4,
      });
    }
    cdk.Tags.of(this).add("CreatedBy", "CDK");
  }
}

The stack is simple, it basically creates ECR repositories (in our case we will create only a single repository) and allows cross-account pull from the Development account. Some lifecycle rule policies were also created to reduce costs. The tagging strategy used here is tag based on the environment name, i.e. tag named dev will be used by the Development environment while a tag named prod should be used in the Production environment.

Now, let's check how the EcsStack looks in depth:

// stack/ecs.ts
import * as cdk from "aws-cdk-lib";
import {
  Certificate,
  CertificateValidation,
  KeyAlgorithm,
} from "aws-cdk-lib/aws-certificatemanager";
import { IpAddresses, Port, SubnetType, Vpc } from "aws-cdk-lib/aws-ec2";
import {
  Cluster,
  Compatibility,
  ContainerImage,
  ContainerInsights,
  EcrImage,
  FargateService,
  LogDriver,
  TaskDefinition,
} from "aws-cdk-lib/aws-ecs";
import {
  ApplicationListener,
  ApplicationLoadBalancer,
  ApplicationProtocol,
  ApplicationTargetGroup,
  HealthCheck,
  ListenerAction,
  ListenerCondition,
  SslPolicy,
} from "aws-cdk-lib/aws-elasticloadbalancingv2";
import {
  ARecord,
  HostedZone,
  IPublicHostedZone,
  RecordTarget,
} from "aws-cdk-lib/aws-route53";
import { Construct } from "constructs";
import { AwsAccount, Environment } from "../config";
import { Repository } from "aws-cdk-lib/aws-ecr";
import { TaskRole } from "aws-cdk-lib/aws-stepfunctions";
import { ManagedPolicy } from "aws-cdk-lib/aws-iam";
import { LoadBalancerTarget } from "aws-cdk-lib/aws-route53-targets";

export interface EcsProps extends cdk.StackProps {
  envName: Environment;
  vpcCidrBlock: string;
  maxAzs: number;
  domain: string;
  backendConfig: EcsFargateServiceDefinition;
}

export interface EcsFargateServiceDefinition {
  name: string;
  cpu: string;
  memoryMiB: string;
  scalability: {
    minCapacity: number;
    maxCapacity: number;
    cpuThresholdPercentage?: number;
  };
  port: number;
  healthCheck: HealthCheck;
  listenerRules: {
    priority: number;
    pathPattern: string;
  };
}

export class EcsStack extends cdk.Stack {
  props: EcsProps;
  hostedZone: IPublicHostedZone;
  vpc: Vpc;

  constructor(scope: Construct, id: string, props: EcsProps) {
    super(scope, id, props);
    this.props = props;

    this.vpc = this.setVpc();

    this.hostedZone = HostedZone.fromLookup(this, "Zone", {
      domainName: props.domain,
    });

    const cluster = new Cluster(this, "EcsCluster", {
      vpc: this.vpc,
      containerInsightsV2: ContainerInsights.ENABLED,
    });
    const { alb, httpsListener } = this.createAlb(this.vpc);

    const backendRepositoryArn = `arn:aws:ecr:${this.props.env?.region}:${AwsAccount.SharedAssets}:repository/${props.backendConfig.name}`;
    const backendImage = ContainerImage.fromEcrRepository(
      Repository.fromRepositoryArn(
        this,
        props.backendConfig.name,
        backendRepositoryArn,
      ),
      this.props.envName,
    );
    this.createFargateService(
      cluster,
      backendImage,
      alb,
      httpsListener,
      this.props.backendConfig,
    );
  }

  setVpc(): Vpc {
    const vpc = new Vpc(this, "Vpc", {
      ipAddresses: IpAddresses.cidr(this.props.vpcCidrBlock),
      maxAzs: this.props.maxAzs,
      subnetConfiguration: [
        {
          cidrMask: 22,
          name: "PrivateForServices",
          subnetType: SubnetType.PRIVATE_WITH_EGRESS,
        },
        {
          cidrMask: 22,
          name: "PublicForServices",
          subnetType: SubnetType.PUBLIC,
        },
      ],
      natGateways: 1,
    });

    return vpc;
  }

  createAlb(vpc: Vpc) {
    const certificate = new Certificate(this, "EcsCertificate", {
      domainName: this.props.domain,
      keyAlgorithm: KeyAlgorithm.RSA_2048,
      validation: CertificateValidation.fromDns(this.hostedZone),
    });

    const alb = new ApplicationLoadBalancer(
      this,
      "EcsApplicationLoadBalancer",
      {
        vpc,
        internetFacing: true,
      },
    );
    alb.connections.allowToAnyIpv4(Port.tcp(80));
    alb.connections.allowToAnyIpv4(Port.tcp(443));
    alb.addListener("Http", {
      protocol: ApplicationProtocol.HTTP,
      port: 80,
      open: true,
      defaultAction: ListenerAction.redirect({
        port: "443",
        protocol: "HTTPS",
        permanent: true,
      }),
    });
    const httpsListener = alb.addListener("Https", {
      protocol: ApplicationProtocol.HTTPS,
      port: 443,
      open: true,
      certificates: [certificate],
      defaultAction: ListenerAction.fixedResponse(404, {
        messageBody:
          "Please configure services to redicted to the ALB correctly",
      }),
      sslPolicy: SslPolicy.RECOMMENDED_TLS,
    });

    return { alb, httpsListener };
  }

  createFargateService(
    cluster: Cluster,
    image: EcrImage,
    loadBalancer: ApplicationLoadBalancer,
    listener: ApplicationListener,
    definition: EcsFargateServiceDefinition,
  ): FargateService {
    const taskDefinition = new TaskDefinition(
      this,
      `EcsServiceTaskDefinition${definition.name}`,
      {
        cpu: definition.cpu,
        compatibility: Compatibility.FARGATE,
        memoryMiB: definition.memoryMiB,
      },
    );
    taskDefinition.addContainer(definition.name, {
      image: image,
      portMappings: [{ containerPort: definition.port }],
      logging: LogDriver.awsLogs({
        streamPrefix: definition.name,
      }),
    });
    taskDefinition.executionRole?.addManagedPolicy(
      ManagedPolicy.fromAwsManagedPolicyName(
        "service-role/AmazonECSTaskExecutionRolePolicy",
      ),
    );

    const service = new FargateService(this, `EcsService${definition.name}`, {
      cluster,
      taskDefinition: taskDefinition,
    });
    loadBalancer.connections.allowTo(service, Port.tcp(definition.port));

    const scaling = service.autoScaleTaskCount({
      maxCapacity: definition.scalability.maxCapacity,
      minCapacity: definition.scalability.minCapacity,
    });
    const cpuMaxMetric = service.metricCpuUtilization({
      statistic: "Maximum",
      period: cdk.Duration.minutes(1),
    });
    const cpuAverageMetric = service.metricCpuUtilization({
      statistic: "Average",
      period: cdk.Duration.minutes(1),
    });

    scaling.scaleOnMetric(`${definition.name}CpuMaxScale`, {
      metric: cpuMaxMetric,
      scalingSteps: [
        {
          lower: 0,
          upper: definition.scalability.cpuThresholdPercentage,
          change: -1,
        },
        {
          lower: definition.scalability.cpuThresholdPercentage,
          upper: 70,
          change: +1,
        },
        { lower: 70, change: +2 },
      ],
      evaluationPeriods: 1,
      cooldown: cdk.Duration.seconds(60),
    });
    scaling.scaleOnMetric(`${definition.name}CpuAverageScale`, {
      metric: cpuAverageMetric,
      scalingSteps: [
        {
          lower: 0,
          upper: definition.scalability.cpuThresholdPercentage,
          change: -1,
        },
        {
          lower: definition.scalability.cpuThresholdPercentage,
          upper: 70,
          change: +1,
        },
        { lower: 70, change: +2 },
      ],
      evaluationPeriods: 1,
      cooldown: cdk.Duration.seconds(60),
    });

    const targetGroup = new ApplicationTargetGroup(
      this,
      `${definition.name}TargetGroup`,
      {
        vpc: this.vpc,
        protocol: ApplicationProtocol.HTTP,
        port: definition.port,
        healthCheck: definition.healthCheck,
        targets: [service],
      },
    );

    listener.addAction(`${definition.name}Action`, {
      conditions: [
        ListenerCondition.hostHeaders([this.props.domain]),
        ListenerCondition.pathPatterns([definition.listenerRules.pathPattern]),
      ],
      priority: definition.listenerRules.priority,
      action: ListenerAction.forward([targetGroup]),
    });

    new ARecord(this, `${this.props.domain}ARecord`, {
      target: RecordTarget.fromAlias(new LoadBalancerTarget(loadBalancer)),
      zone: this.hostedZone,
      recordName: this.props.domain,
    });

    return service;
  }
}

This stack is way bigger than the ECR stack so let's explain this step-by-step:

The Stack class contains three methods:
- setVpc: Created a VPC with NAT Gateway, Public Subnets, Private Subnets, and Route Tables.
- createAlb: Create an ALB with an HTTPS listener (with a Certificate attached to it) and an HTTP listener (that redirects to HTTPS). The ALB allows HTTP (port 80) and HTTPS (port 443) traffic from anywhere.
- createFargateService: Creates the taskDefinition, container definition, and auto scaling policies for the ECS service.
There are three ways to auto-scale your ECS cluster:
- Target Tracking: Simply select a metric and a target value to scale and ECS will manage everything - e.g. 70% average CPU - and the alarms will try to keep the metric as close to the specified value.
- Step scaling: You will manage the creation of CloudWatch Alarms, the rules to scale and steps of scale-in/scale-out, e.g. create an alarm that monitors average memory and reduce 1 task if below 30% / add 1 task if above 60% / add 2 tasks if above 75%.
- Predictive scaling: You specify a metric and AWS will scale based on historical data of the metric selected. Predictive scaling only scales up, meaning that you will need other scaling methods to scale down. Often used when there is cyclical traffic in our application.
I chose to not use the ApplicationLoadBalancedFargateService construct because it creates a single ALB per service created and my idea was to use a single ALB per ECS Cluster, so we could add other services (e.g. frontend) that would reuse this ALB. Also notice that I've used a path-based routing instead of a subdomain-based routing (i.e. /api forwards to the backend service, while the / could forward to the frontend service for example).

Now, let's just define the configuration for each stack.

// config/dev.ts
import { Protocol } from "aws-cdk-lib/aws-elasticloadbalancingv2";
import { AwsAccount } from ".";
import { EcsProps } from "../stack/ecs";

export const devConfig: EcsProps = {
  env: {
    account: AwsAccount.Development,
    region: "us-east-1",
  },
  domain: "demosfelipetrindade.top",
  envName: "dev",
  vpcCidrBlock: "10.200.0.0/16",
  maxAzs: 2,
  backendConfig: {
    name: "backend",
    cpu: "1024",
    memoryMiB: "2048",
    port: 8000,
    scalability: {
      minCapacity: 2,
      maxCapacity: 6,
      cpuThresholdPercentage: 40,
    },
    healthCheck: {
      path: "/api/health",
      protocol: Protocol.HTTP,
    },
    listenerRules: {
      priority: 100,
      pathPattern: "/api/*",
    },
  },
};

and

// config/sharedAssets,ts
import { AwsAccount } from ".";
import { EcrProps } from "../stack/ecr";

export const sharedAssetsConfig: EcrProps = {
  env: {
    account: AwsAccount.SharedAssets,
    region: "us-east-1",
  },
  ecrRepository: ["backend"],
  maxAgeOtherTagsInDays: 7,
};

Running our demo!

The commands to reproduce this demo are available in my GitHub Repository. So let's focus on the results of the demo and suppose that the ECR contains the API image and that the ECS service is already running as expected.

For this load test, I decided to use a spawn rate of 0.25 users per second and 80 max users. I let the test run for almost an hour. Here are the Locust results.

During our test, 116k requests were made to the API, and only 2 requests failed to return to the user and the 99%ile was 260ms. Let's now analyze the charts (RPS, Response Time, and Number of Users).

We can observe that there were some peaks in the 95th percentile but for the most time, the 50th percentile was pretty stable.

Now, let's analyze the infrastructure side: CPU and memory usage of the ECS Service during the load test.

We can see that the application was sensitive in terms of CPU but the memory didn't change much (didn't even reach 10% utilization), that's why we used scaling based on CPU for this application. The Max CPU Utilization varied and peaked several times around 40% (our target set threshold) but after a while, the value dropped significantly. Why? That's because more tasks were added to the service to make sure the CPU goes below 40%. This becomes way more visible in the chart of a number of tasks running.

It's important to comprehend the correlation between CPU increase and number of tasks running. When the CPU peaks, ECS scales up the service. When the CPU goes down, ECS scales down the cluster. The scaling (up and down) of ECS is managed by CloudWatch Alarms that monitor a metric (in our case CPU Average and CPU Maximum) and perform an action when the alarm is in alarm state. The ECS service sends metrics (CPU, Memory...) to CloudWatch in 1-minute intervals. We created four alarms:

Let's take a look at how the Average CPU metric (to scale up) behaves:

This alarm was in alarm state a single time when the average CPU peaked and went beyond the 40% threshold. The alarm to scale down based on the Average CPU will be the opposite of the alarm to scale up! Let's check:

Now let's check the Alarm related to the scaling up of the Maximum CPU metric:

Notice how this metric was triggered several times! As you expect the policy to scale down based on maximum CPU usage will be the opposite.

You might be wondering why the alarms were not triggered when the CPU reached 40% immediately. Well, the reason for that is that it takes time for ECS to send the metrics, CloudWatch to evaluate and EventBridge trigger the scale up.

Cya!

Well, I hope you liked this blog post. See you around! 👋

Terraform CI/CD with GitHub Actions and pre-commit

Wed, 05 Mar 2025 12:55:32 GMT

In today's blog post, I would like to propose a simple, yet powerful, CI/CD pipeline to be used in Terraform projects. From my experience, I've been in several projects with no automated deployment for terraform, lack of code standards, and use of bad practices. Setting up a good CI/CD for Terraform is not difficult but has some important parts that most don't see. Let's deep dive into a great CI/CD pipeline that goes beyond terraform plan and terraform apply actions.

What makes a good CI/CD?

As you probably know CI/CD stands for Continuous Improvement (allow developer to always improve/fix the source code) and Continuous Deployment (allow developers to deploy the changes with ease).

During our Terraform CI, you would like to:

Lint the code: We would like to run checks to make sure the code is valid and can be built/deployed with no issues, i.e. check syntax errors, and common bugs...
Run security/compliance checks: We should shift security left as much as we can to avoid adding security issues to our code.
Format the code: Developers have different preferences when it comes to formatting the code, which can make the code look inconsistent. Adding a code formatter standardizes the way the code looks and improves readability.
Run tests: Make sure that what is being added won't break the current code.
Plan changes: We would like to see what actions will be performed by the code, i.e. the terraform plan

During the Terraform CD other actions are performed:

Deploy the infrastructure: The source code gets deployed into the environments. It's a common practice to not deploy to all environments (Development, Staging, Production) all at once. Some companies use a manual approach (validate staging is working before manually approving production deployment) while others use an automatic approach based on metrics (latency, availability...).
Run Post-Deployment Tests: Load tests or E2E tests after the infrastructure is deployed guarantee that everything is working as expected.
Notify the team Send messages via Slack, MS Teams, and Email to notify important people about the deployment.

Do not treat these actions as a source of truth, since they vary depending on the company, but this serves as a good baseline.

Local development

When possible it's interesting to shift to the left the CI checks, i.e. run checks before pushing the code to the remote repository. If CI checks are simple and easy to run locally on the developing machine they should run on the developer machine too (not only during the CI pipeline) because:

It removes frustration from the developer that needs to push code and wait for CI to run and check if it passed
Speed up the development process
Reduce CI costs

Sometimes the checks take long and it's better to run checks in the CI, but if it's not CPU/Memory/Time consuming they should also run in the developer machine.

A great tool to automate the CI checks in the developer machine is Pre-Commit. We basically define a manifest with the hooks that are going to run before the git commit is executed and if it fails it will give the reasons and try to remediate, otherwise if it passes the commit is made, so you also end up with a better Git history too!

Another common problem when contributing to a project as a developer is to make sure our local environment contains all the needed tools/binaries to run the project. Actually, not only having the tools/binaries are needed, but it's very common to also enforce specific versions. If you follow my blog you know that I'm a bit fan of Devbox and mise-en-place. For this blog post, I will use mise-en-place. With these tools, you can define a manifest with needed tools and versions needed and install all these tools with a simple bash command.

Helper Tools

During CI and Local Development

Our development environment and CI will rely on running the pre-commit check in the repo, this guarantees that even if the developer bypasses the pre-commit check, the CI will guarantee that the code is compliant with the best practices created for the repo. Several tools and commands will be used during the CI:

terraform fmt: Make sure the code is formatted and easy to read.
terraform validate: Validate runs checks that verify whether a configuration is syntactically valid and internally consistent, regardless of any provided variables or existing state.
terraform-docs: Automatically updates the README markdown that contains the documentation of the terraform code auto-generated using terraform-docs
terraform providers lock: Consults registries to write provider dependency information into the dependency lock file (.terraform.lock.hcl).
tflint: tflint allows us to find possible errors and enforce best practices.
trivy: Find vulnerabilities, misconfigurations, and exposed secrets using trivy

For simplicity, I won't be running Tests in terraform but you can use Terratest.

During CD

Our CD will be simple and will simply run terraform apply. Since I won't deploy any real application I didn't load tests or send notifications but you can use some great actions, such as Slack action.

Hands-on!

Finally, it's time to check how to implement the proposed CI/CD! Once again, you can check the full code on my GitHub Repository.

├── config/     # Backend configuration per AWS environment
├── modules/    # Custom Terraform modules
├── src/        # Main Terraform code to deploy infrastructure
└── vars/       # Variables values specific for every environment

Then let's check the tools needed to run the project.

# .mise.toml
[tools]
terraform = "1.10.5"
pre-commit = "4.0.1"
terraform-docs = "0.19.0"
tflint = "0.55.1"
trivy = "0.59.1"
jq = "1.7.1"

Installing these tools is as easy as running mise install.

Now, let's check our pre-commit file.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.97.3
    hooks:
      - id: terraform_fmt
      - id: terraform_docs
        args:
          - --hook-config=--create-file-if-not-exist=true
          - "--args=--lockfile=false"
      - id: terraform_validate
        args:
          - --hook-config=--retry-once-with-cleanup=true
      - id: terraform_providers_lock
        args:
          - --args=-platform=linux_amd64
          - --args=-platform=linux_arm64
          - --args=-platform=darwin_arm64
          - --args=-platform=darwin_amd64
          - --hook-config=--mode=always-regenerate-lockfile
      - id: terraform_tflint
      - id: terraform_trivy
        args:
          - --hook-config=--parallelism-limit=1
          - --args=--severity HIGH,CRITICAL
          - --args=--ignorefile=__GIT_WORKING_DIR__/.trivyignore

The repository that contains the hooks is owned by Anton Babenko: a huge Terraform hero! Notice that our pre-commit uses all the tools mentioned previously.

It's important to mention that the developer needs to run pre-commit install in order to use pre-commit.

With everything set, let's try to create a simple SQS queue and commit in the Git repository!

Notice that it failed because:

Documentation was not updated, so it updated and is waiting for us to add to the commit
SQS should be deployed with SSE enabled to increase security, here is the fix (that needs to be manually performed):

// src/main.tf
resource "aws_sqs_queue" "terraform_queue" {
 name                    = "test"
 sqs_managed_sse_enabled = true
}

Great! Now we have a local environment that applies best practices! Let's move to GitHub Actions. Since we have multiple environments (in my demo I will use two environments: development and production) you can imagine that repetitive actions will be performed (e.g. terraform plan will be executed in every environment), because of that we can create a composite action to simplify our CI/CD code.

# .github/actions/terraform/action.yaml
name: Terraform Action
description: Run Terraform Plan or Apply

inputs:
  action:
    description: "Terraform action to perform. Allowed values: plan, apply"
    required: true
  environment:
    description: "Account name that is being deployed. Allowed values: DEV, PROD"
    required: true
  working_directory:
    description: Path to the directory containing the terraform files
    default: src
  aws_region:
    type: string
    description: AWS region to use
    required: true
  tfconfig_path:
    type: string
    description: Path to the HCL file that configured Terraform backend
    required: true
  tfvars_path:
    type: string
    description: Path to the TFVARS file that contains the values for variables
    required: true
  aws_role_arn:
    type: string
    description: AWS IAM Role that will be used to assume a role using OIDC
  github_token:
    type: string
    description: GitHub Token generated during GitHub Workflow runs used to comment the plan in the Pull Request. Only needed if 'action' is 'plan'
    required: false

runs:
  using: "composite"
  steps:
    - name: Install Mise
      uses: jdx/mise-action@v2

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: ${{ inputs.aws_role_arn }}
        role-session-name: Github-Actions
        aws-region: ${{ inputs.aws_region }}

    - name: Initialize terraform
      shell: bash
      working-directory: ${{ inputs.working_directory }}
      run: terraform init -backend-config=$GITHUB_WORKSPACE/${{ inputs.tfconfig_path }}

    - name: Terraform Plan
      if: ${{ inputs.action == 'plan' && github.ref != 'refs/heads/main' }}
      shell: bash
      working-directory: ${{ inputs.working_directory }}
      run: terraform plan -var-file=$GITHUB_WORKSPACE/${{ inputs.tfvars_path }} -out .planfile

    - name: Comment Terraform plan in PR
      if: ${{ inputs.action == 'plan' && github.ref != 'refs/heads/main' }}
      uses: borchero/terraform-plan-comment@v2
      with:
        working-directory: ${{ inputs.working_directory }}
        token: ${{ inputs.github_token }}
        planfile: .planfile
        header: "[ ${{ inputs.environment }} - ${{ inputs.aws_region }}] Terraform Plan"

    - name: Terraform Apply
      if: ${{ inputs.action == 'apply' }}
      shell: bash
      working-directory: ${{ inputs.working_directory }}
      run: terraform apply -auto-approve -var-file=$GITHUB_WORKSPACE/${{ inputs.tfvars_path }}

The terraform plan will be displayed as a comment on the Pull Request like that:

And here is our main workflow

# .github/workflows/main.yaml
name: "Terraform"

on:
  push:
    branches:
      - main
  pull_request:

permissions:
  contents: write
  pull-requests: write
  id-token: write

jobs:
  ci:
    if: github.event_name == 'pull_request' && github.ref != 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Install Mise
        uses: jdx/mise-action@v2

      - name: Run Pre-Commit
        run: pre-commit run --all-files

  plan-dev-us-east-1:
    name: "[DEV - us-east-1] Terraform Plan"
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Terraform Plan
        uses: ./.github/actions/terraform
        with:
          action: plan
          environment: DEV
          aws_region: us-east-1
          tfconfig_path: config/dev.hcl
          tfvars_path: vars/dev/us-east-1.tfvars
          aws_role_arn: ${{ vars.DEV_TERRAFORM_ROLE }}
          github_token: ${{ github.token }}

  apply-dev-us-east-1:
    needs:
      - plan-dev-us-east-1
    name: "[DEV - us-east-1] Terraform Apply"
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    environment:
      name: dev

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Terraform Apply
        uses: ./.github/actions/terraform
        with:
          action: apply
          environment: DEV
          aws_region: us-east-1
          tfconfig_path: config/dev.hcl
          tfvars_path: vars/dev/us-east-1.tfvars
          aws_role_arn: ${{ vars.DEV_TERRAFORM_ROLE }}
          github_token: ${{ github.token }}

  plan-prod-us-east-1:
    name: "[PROD - us-east-1] Terraform Plan"
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Terraform Plan
        uses: ./.github/actions/terraform
        with:
          action: plan
          environment: PROD
          aws_region: us-east-1
          tfconfig_path: config/prod.hcl
          tfvars_path: vars/prod/us-east-1.tfvars
          aws_role_arn: ${{ vars.PROD_TERRAFORM_ROLE }}
          github_token: ${{ github.token }}

  apply-prod-us-east-1:
    needs:
      - apply-dev-us-east-1
      - plan-prod-us-east-1
    name: "[PROD - us-east-1] Terraform Apply"
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    environment:
      name: prod

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Terraform Apply
        uses: ./.github/actions/terraform
        with:
          action: apply
          environment: PROD
          aws_region: us-east-1
          tfconfig_path: config/prod.hcl
          tfvars_path: vars/prod/us-east-1.tfvars
          aws_role_arn: ${{ vars.PROD_TERRAFORM_ROLE }}
          github_token: ${{ github.token }}

If you are a visual person, like me, here is the workflow that we just created in the GitHub console:

Please notice that I'm using GitHub Environment and the prod requires manual approval to deploy to production!

Cya!

I hope you've liked it and the explanation was helpful!

See you around! 👋

Deploy a bastion host with Terraform or AWS CDK

Sat, 01 Feb 2025 12:55:32 GMT

As you probably know the best practice is to deploy resources (such as databases and servers) within a private network and if they need to be exposed you should expose them using Load Balancer to handle the requests and traffic distribution.

Some resources, such as databases, you don't want ever to expose publicly. A common request in tech companies is to allow developers to connect to development databases deployed in the cloud provider. Some companies also create processes for allowing access in production in case of an urgent need!

The problem is that since databases are deployed within a private network, the developers won't be able to connect to the database. There are two common options here:

Use a VPN to connect the developers to the private network
Use a bastion host to connect to the private resources

This blog post aims at the latter and you will see how easy and simple it is to deploy a bastion host securely! All the source code used in this repository is available in my GitHub Repository.

Bastion Host and Systems Manager

A bastion Host (also known as jump box) is simply a server that has access to a private network but it's exposed publicly in a way that users (usually developers) can connect to it. The connectivity is usually done using SSH for Linux-based instances or RDP if it's a Windows instance. You would also need to configure the firewall of the instance correctly to allow inbound traffic from developers' IPs (and change this constantly, since internet providers usually offer dynamic IP) or from anywhere (too permissive - turning your bastion into a possible source of attacks). This is the standard way to deploy a bastion host in a VPS.

In AWS we can deploy bastion hosts using the approach mentioned above or in a different way, that relies on a service called AWS Systems Manager (as known as SSM). It is a big AWS service used to manage servers (e.g. EC2, edge devices, on-premise servers): execute scripts, perform updates, install tools... It is a great service to manage nodes at scale!

For this blog post, the only thing we need to know about SSM is:

Systems Manager Agent: A node can only be managed by SSM if it has an agent (a piece of software that runs as a daemon) installed and running in the machine. This agent is called Systems Manager Agent and its source code is open and available on GitHub. This agent can be installed in EC2, Virtual Machines, edge devices... When using EC2 machines you can use AMIs that already have this agent preinstalled or install it yourself (commonly done using the EC2 UserData script). Notice that the agent needs to access the SSM Endpoint (publicly or via VPC Endpoint) to be able to communicate with the service, so make sure the EC2 instance is in a public subnet with Internet Gateway configured properly or in a private subnet with access to NAT Gateway or VPC Endpoint configured.
Session Manager: As mentioned, SSM allows us to run several different actions in a machine that contains the SSM Agent installed. These actions are called Documents, and we are interested in the AWS-StartPortForwardingSessionToRemoteHost document that is part of the Session Manager tool. The Session Manager is a secure way to manage/access your servers via web-based sessions (via AWS Console) or AWS CLI (but you will need to install a plugin), in other words, it's a "replacement" for using pure SSH connection, which means that now you can rely on SSM to handle security and traffic (no need to manage firewall rules anymore). Notice that this is only possible because the SSM Agent is handling all of that, so it's a must-have. Instead of managing firewall and SSH keys, the way we manage/control who will have access to the instance is via IAM policies on two fronts: permission of the EC2/On-prem instance to allow usage of SSM service (on-premise servers use a hybrid activation method) and permission of the IAM User/Role that will access the instance.

Demo time!

Let's demo the Bastion Host deployment using Terraform and CDK! To test that the bastion host is working I will also deploy an RDS instance (I will use Postgres) in a private subnet. Our goal is to deploy a bastion instance in a private subnet, be able to connect to it using SSM Session Manager, and connect to the RDS using a GUI (such as DBeaver) and CLI (psql). The following resources will be created in Terraform and CDK:

VPC
EC2 to serve as Bastion
IAM Roles
Single instance RDS (Postgres)

Terraform

For simplicity, I will rely on Terraform community modules to deploy the needed resources. Let's check the variables of our code:

// variables.tf
variable "aws_region" {
  description = "Region that AWS resources will be deployed"
  type        = string
  default     = "us-east-1"
}

variable "vpc_cidr" {
  description = "VPC CIDR to use"
  type        = string
  default     = "10.123.0.0/16"
}

variable "azs" {
  description = "Number of AZs to use in the VPC"
  type        = number
  default     = 3
}

variable "instance_type" {
  description = "Defines the instance type of the EC2 bastion host"
  type        = string
  default     = "t3.nano"
}

And then the data sources:

// data.tf
data "aws_availability_zones" "available" {}

data "aws_ec2_instance_type" "this" {
  instance_type = var.instance_type
}

data "aws_ami" "ubuntu_latest" {
  most_recent = true
  owners      = ["099720109477"] # Canonical's AWS account ID (maintainer of Ubuntu)

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-*-*-${local.instance_architecture == "x86_64" ? "amd64" : "arm64"}-server-*"]
  }

  filter {
    name   = "architecture"
    values = ["${local.instance_architecture}"]
  }
}

And now the constants file:

// locals.tf
locals {
  azs                   = slice(data.aws_availability_zones.available.names, 0, var.azs)
  instance_architecture = contains(data.aws_ec2_instance_type.this.supported_architectures, "arm64") ? "arm64" : "x86_64"
}

Finally, here is the creation of the resources:

// main.tf
module "vpc" {
  source          = "terraform-aws-modules/vpc/aws"
  version         = "5.16.0"
  name            = "bastion-terraform"
  azs             = local.azs
  cidr            = var.vpc_cidr
  private_subnets = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k + 4)]

  database_subnets                   = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k + 8)]
  create_database_subnet_route_table = true
  create_database_nat_gateway_route  = true

  enable_nat_gateway = true
  single_nat_gateway = true
}

module "bastion_sg" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "5.3.0"

  name = "bastion-terraform-sg"

  vpc_id = module.vpc.vpc_id
  egress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 65535
      protocol    = "tcp"
      description = "Allow all outbound traffic"
      cidr_blocks = "0.0.0.0/0"
    },
  ]
}

module "bastion" {
  source  = "terraform-aws-modules/ec2-instance/aws"
  version = "5.7.1"

  name                   = "bastion-terraform"
  instance_type          = var.instance_type
  vpc_security_group_ids = [module.bastion_sg.security_group_id]
  subnet_id              = module.vpc.private_subnets[0]
  ignore_ami_changes     = false
  ami                    = data.aws_ami.ubuntu_latest.id

  create_iam_instance_profile = true
  iam_role_policies = {
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }

  user_data = <<EOF
#!/bin/bash

echo "Installing SSM Agent"
sudo snap install amazon-ssm-agent --classic
sudo snap list amazon-ssm-agent
sudo snap start amazon-ssm-agent
sudo snap services amazon-ssm-agent
EOF
}


module "database_sg" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "5.3.0"

  name = "database-terraform-sg"

  vpc_id = module.vpc.vpc_id
  ingress_with_source_security_group_id = [
    {
      from_port                = 5432
      to_port                  = 5432
      protocol                 = "tcp"
      description              = "Allow inbound traffic on default Postgres port"
      source_security_group_id = module.bastion_sg.security_group_id
    },
  ]
  egress_with_cidr_blocks = [
    {
      from_port   = 0
      to_port     = 65535
      protocol    = "tcp"
      description = "Allow all outbound traffic"
      cidr_blocks = "0.0.0.0/0"
    },
  ]
}

module "database" {
  source  = "terraform-aws-modules/rds/aws"
  version = "6.10.0"

  identifier = "database-terraform"
  db_name    = "main"

  engine               = "postgres"
  engine_version       = "14"
  family               = "postgres14"
  major_engine_version = "14"
  instance_class       = "db.t4g.large"
  allocated_storage    = 5

  username = "postgres"

  db_subnet_group_name   = module.vpc.database_subnet_group
  vpc_security_group_ids = [module.database_sg.security_group_id]

  skip_final_snapshot = true
  deletion_protection = false
}

Notice that we are getting the AMI image from data.tf and based on the var.instance_type we are getting the correct Ubuntu AMI for the bastion instance. Also, we are manually installing the SSM Agent via EC2 User Data script.

Deploying that is as easy as running terraform apply (or tofu apply).

Now, after the infrastructure is deployed we can create a simple script to automatically:

Get EC2 Instance ID based on the Bastion name
Get RDS Endpoint, master Username/Password (in Secrets Manager), Port
Use AWS-StartPortForwardingSessionToRemoteHost document to connect to RDS

export BASTION_NAME="bastion-terraform"
export DATABASE_NAME="database-terraform"
export AWS_REGION="us-east-1"

SECRET_ARN=$(aws rds describe-db-instances --db-instance-identifier \
  $DATABASE_NAME --query 'DBInstances[0].MasterUserSecret.SecretArn' --output text)
SECRET_VALUE_RAW=$(aws secretsmanager get-secret-value --secret-id $SECRET_ARN | jq '.SecretString')
DB_USERNAME=$(echo $SECRET_VALUE_RAW | jq -r | jq -r ".username")
DB_PASSWORD=$(echo $SECRET_VALUE_RAW | jq -r | jq -r ".password")
DB_PORT=$(aws rds describe-db-instances --db-instance-identifier \
  $DATABASE_NAME --query 'DBInstances[0].Endpoint.Port' --output text)
DB_DATABASE=$(aws rds describe-db-instances \
  --db-instance-identifier $DATABASE_NAME \
  --query 'DBInstances[0].DBName' --output text)
echo "Connect to the database using the following inputs:"
echo "\n\nHost => localhost\nDatabase => $DB_DATABASE\nUSERNAME => $DB_USERNAME\nPASSWORD => $DB_PASSWORD\nPORT => $DB_PORT"

CLUSTER_HOST=$(aws rds describe-db-instances \
  --db-instance-identifier $DATABASE_NAME \
  --query 'DBInstances[0].Endpoint.Address' --output text)
BASTION_INSTANCE_ID=$(aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=$BASTION_NAME" \
  --filters "Name=instance-state-name,Values=running" \
  --query "Reservations[*].Instances[*].InstanceId" --output text)
aws --region=us-east-1 ssm start-session \
  --target $BASTION_INSTANCE_ID \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters "{\"host\":[\"$CLUSTER_HOST\"],\"portNumber\":[\"5432\"], \"localPortNumber\":[\"$DB_PORT\"]}"

This script will output the following:

You can then use DBeaver or PSQL to connect to the database. Notice that your terminal will be "blocked" with a pending session with the bastion host. If you close or cancel the command the connection is lost and you won't be able to reach the database. This makes possible to access the database as it was running locally.

Connecting via GUI is as simple as inputing the information that came from the script output.

It's also simple to connect via PSQL.

We can also connect to the instance terminal using SSM Session Manager:

BASTION_INSTANCE_ID=$(aws ec2 describe-instances \
    --filters "Name=tag:Name,Values=$BASTION_NAME" \
    --filters "Name=instance-state-name,Values=running" \
    --query "Reservations[*].Instances[*].InstanceId" \
    --output text)
aws ssm start-session --target $BASTION_INSTANCE_ID --region us-east-1

Notice that we didn't specify a document to be used. This is because if no document is given it will launch the shell to manage the node by default.

CDK

For CDK a single Stack will be created using Typescript.

// mais.ts
import "source-map-support/register";
import * as cdk from "aws-cdk-lib";
import { Stack } from "./stack";
import { InstanceClass, InstanceSize, InstanceType } from "aws-cdk-lib/aws-ec2";

const app = new cdk.App();

new Stack(app, "BastionStack", {
  env: {
    region: "us-east-1",
  },
  instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.SMALL),
  vpcCidrBlock: "10.130.0.0/16",
});

The main code instantiate the Stack that was created in a separate file.

import * as cdk from "aws-cdk-lib";
import {
  BastionHostLinux,
  InstanceClass,
  InstanceSize,
  IpAddresses,
  Port,
  SubnetType,
  Vpc,
} from "aws-cdk-lib/aws-ec2";
import { Construct } from "constructs";
import { InstanceType } from "aws-cdk-lib/aws-ec2";
import {
  Credentials,
  DatabaseInstance,
  DatabaseInstanceEngine,
  PostgresEngineVersion,
} from "aws-cdk-lib/aws-rds";

export interface BastionProps extends cdk.StackProps {
  vpcCidrBlock: string;
  instanceType: InstanceType;
}

export class Stack extends cdk.Stack {
  public readonly vpc: Vpc;

  constructor(scope: Construct, id: string, props: BastionProps) {
    super(scope, id, props);

    this.vpc = new Vpc(this, "Vpc", {
      ipAddresses: IpAddresses.cidr(props.vpcCidrBlock),
      subnetConfiguration: [
        {
          cidrMask: 22,
          name: "Private",
          subnetType: SubnetType.PRIVATE_WITH_EGRESS,
        },
        {
          cidrMask: 22,
          name: "PrivateForDB",
          subnetType: SubnetType.PRIVATE_ISOLATED,
        },
        {
          cidrMask: 22,
          name: "Public",
          subnetType: SubnetType.PUBLIC,
        },
      ],
      natGateways: 1,
    });

    const database = new DatabaseInstance(this, "Database", {
      vpc: this.vpc,
      engine: DatabaseInstanceEngine.postgres({
        version: PostgresEngineVersion.VER_14,
      }),
      instanceType: InstanceType.of(InstanceClass.T4G, InstanceSize.LARGE),
      multiAz: false,
      allocatedStorage: 5,
      vpcSubnets: {
        subnetType: SubnetType.PRIVATE_ISOLATED,
      },
      credentials: Credentials.fromGeneratedSecret("postgres"),
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      databaseName: "main",
      instanceIdentifier: "database-cdk",
    });

    const bastion = new BastionHostLinux(this, "BastionEc2", {
      vpc: this.vpc,
      instanceName: "bastion-cdk",
      instanceType: props.instanceType,
      subnetSelection: {
        subnetType: SubnetType.PRIVATE_WITH_EGRESS,
      },
    });
    bastion.connections.allowTo(database, Port.tcp(5432));
  }
}

Cya!

Well, I hope you liked this blog post. See you around! 👋

Learn AWS Step Functions from practical examples with Terraform

Mon, 06 Jan 2025 12:55:32 GMT

AWS Step Functions is an amazing AWS service that allows you to create state machines (workflows) to orchestrate processes, microservices, jobs, etc. It's commonly used for orchestrating data pipelines and performing machine learning training/executions but it can be used whatever way fits your needs.

I would like to go over this service and explain to you how it works, what you can do with it, and how to use it with several practical examples that you can reproduce using Terraform.

State Machine

A state machine is comprised of several event-driven steps (known as state) that can modify/interact with the flow of the execution or execute a task (action) in AWS (basically an API call to an AWS service). Explaining in simple words, a state machine is a bunch of actions that are executed in a sequential order (unless Parallel state is used - more on that later!) and that you can decide the actions to execute based on logic/data. In AWS, a state machine is defined via a JSON that specifies the list of states, order, handling errors, and content... Usually, this JSON is not built manually and it's common to use the AWS Console to generate this JSON file that represents the state machine: an image worth more than a million sentences.

We can easily transform a cooking recipe into a state machine, let's take for example a cake recipe: we start by preheating the oven, then we get a bowl, then we add all the ingredients (eggs, milk, butter, baking powder, sugar...), mix the ingredients, pour the batter in a baking pan inside the oven, wait for 30 minutes and check if the cake is baked, if not, wait 5 more minutes and check again (keep checking until the cake is baked), once baked remove from the oven. Well, I'm not a cook so this recipe might not be 100% accurate but the importance here is to understand the flow of actions, which would visually be something like this:

Every state machine can be executed with an input event, that can be (and most of the time is) relevant to coordinate the flow or introduce data into the workflow. For example, in your cake state machine, we could have expected an input of the flavor of the cake. If it's a chocolate cake, we would add that ingredient to the batter.

When creating a state machine in AWS you must select what type the workflow will be: Standard or Express. They have different use cases and are completely different in a way that we can express their difference in a few points:

Execution: Standard workflows are exactly-once, while express workflows are at-least-once.
Execution rate: Standard workflows have a 2,000 executions per second rate (and 4000 per state transition, i.e. going from one state to another), while express workflows can run 100,000 executions per second (and unlimited transition rate).
Duration: Your workflow can run up to one year in the standard mode, while the express is up to five minutes.
Billing: Standard workflows are billed based on the number of transitions, while in express machines you pay based on the duration and memory of executions (behind the scenes Express workflows use Lambda - that's why the billing is like a Lambda!).
Debugging: Visual and history execution is available when using standard workflows, while express workflows use CloudWatch.

Express workflows are intended to be used for high-volume, short-duration, low-latency workflows where the at-least-once execution strategy is not a problem, such as real-time events and microservices coordination. In every other scenario, you will be using the standard workflow type.

States

As mentioned, there are two types of states:

Flow: There are only seven flow states (at the time that I'm writing this blog post) that are used to manipulate the flow of execution of the state machine. You can, for example, use the Choice block to add a conditional block (if-else logic), add a Parallel block to execute actions in parallel, or use the Pass block to perform data manipulation in the events that triggered the Step Function.

Actions: Hundreds of API calls to AWS services are available in the action states. Several services offer blocks to perform actions, such as invoking Lambda Functions, manipulating objects in S3, modifying data in the DynamoDB table, configuring Route53 records, using Bedrock models, executing Athena queries, running Glue jobs... The list of available actions is huge! It is important to mention that when you create a state machine you need to define an IAM Role that will be assumed by this state machine to run all the state actions and make sure the permissions are set correctly. Please do not confuse this role with the role used by the execution of some state actions, e.g. imagine you would like to trigger a Lambda using the state machine; for that, you will need to grant lambda:Invoke permission to the IAM Role associated with the state machine, but the Lambda will run using it's configured IAM Role!

Data Flow and JSON Query Languages

Data is passed from the previous state to the following state and can be modified, transformed, and filtered during runtime using JSON data languages: JSONata or JSONPath.

JSONPath

When using JSONPath there are 5 fields that you could pass to the state:

InputPath: Filters the JSON input from the previous task.
Parameters: Argument of the state itself. Notice that this is different than the InputPath. Think of Parameters as the input to the API call (e.g. payload of the S3 PutObject state), while the InputPath is all the JSON data available that can or can not be used to create the Parameters.
ResultSelector: Here you can select part of the result of the state that you would like to carry forward.
ResultPath: You can combine the initial state input and the ResultSelector to build a new JSON and pass forward.
OutputPath: Finally you will build the end JSON, filtering the ResultPath JSON data, that will be passed to the next state.

JSONPath allows the usage of some intrinsic functions to manipulate the JSON data. The functions are very limited but handful and can sometimes substitute the need to develop a Lambda Function only to manipulate the data, such as:

Array operations: Select array item, check array length...
String operations: Split by delimiter
Math operations: Generate random numbers, sum numbers...
Generic operations: String concat/template

and much more... Please check the official documentation to get to know the complete set of available functions. Due to the limitations of JSONPath and the fact that it's not the recommended query language anymore, I will prefer to use JSONata (although in the final exercise I will provide the code for JSONata and JSONPath).

When using JSONPath there are two syntaxes that you need to know:

For accessing the data using JSON path you will use the $. operator to access the data. For accessing the context object (information about the state machine, state, execution, and task) you will use the $$. operator.
For using the intrinsic functions you will use the States.<FunctionName>(arguments).

Let's use an example for easy understanding. Let's suppose the following input for your state:

{
  "name": "Felipe",
  "occupation": {
    "title": "DevOps Engineer"
  },
  "hobbies": ["cooking", "hiking"]
}

We can access the job title with $.occupation.title and get the first hobby (cooking) with State.ArrayGetItem($.hobbies, 0).

Every JSON key that will access a JSON data using JSONPath or that will use an intrinsic function should end with .$ so AWS can be aware that it needs to apply JSONPath logic there.

JSONPath also offers usage of Variables, i.e. values that you can store to use later in future states!

JSONata

The JSONata is the new supported way to query JSON data in Step Functions and it was introduced in the re:Invent 2024. Before that only JSONPath was available and used, that's why you may still see more content using JSONPath out there.

The JSONata format also simplifies the fields of the States. Instead of five fields to configure, we have only two:

Arguments: Represents the inputs parameter for task state, it's equivalent to the Parameters of JSONPath.
Outputs: Corresponds to the JSON data that will be passed to the following state, it's similar to the OutputPath.

Differently from the JSONPath, the JSONata offers more complex and rich query transformations that can be fully checked in their documentation.

There is a single syntax when using JSONata: {% <expression> %}. The expression can be any valid JSONata expression or a special and reserved keyword: states. With $states you can access four data:

input: Original input to the state
result: API result
errorOutput: Error Output in a Catch
context: Context object

Exercise 01: Should I Deploy? (JSONata)

Let's put into action all the theory that was just explained! This will be the first exercise of this blog post! All the Terraform code for deploying this is in the GitHub Repository.

If you are a DevOps Engineer or even a Software Engineer you probably hear about people saying that you should not deploy on Fridays, the end of the working day, before a holiday... For this exercise we will use Should I Deploy API to the deployment of the infrastructure via the AWS keys of an IAM User. The idea is to check the response of the API: if we get a red light we will attach AWS Deny All Policy to the user, otherwise, we will remove this permission. Imagine that the AWS Access Keys of this IAM User will be used in a CI/CD to deploy Terraform code.

The state machine has the following graph:

Notice that it performs exactly what we discussed above. It is also checking two important cases: if it's allowed to deploy and the deny policy is already not attached, and if it should not deploy and the policy is already attached. This is needed because the detach user policy action will raise an error if it tries to detach a policy that is not attached to the user!

For this case, the input of the Step Function is not relevant, it's just a cron-based trigger. The state machine starts with the HTTP Endpoint state with the following configuration:

The HTTP Endpoint state requires us to pass an EventBridge Connection that contains the credentials (we are using a public API that does not need credential but we still need to create this resource).

We are also specifying the API Endpoint (https://shouldideploy.today/api) and the HTTP method (GET). The only relevant information from the JSON response of the API is the shouldideploy key: a boolean that says if we should deploy or not. Notice how I'm passing this as the following JSON:

{
  "deploy": "{% $states.result.ResponseBody.shouldideploy %}"
}

I'm using JSONata to extrat the response of the API ($states.result) the desired key. This becomes clear when we check an execution of this State Machine. Let's check the entire output of this state.

I'm also exporting the deploy as a variable since I plan to use this in the future states (choice).

By default, the entire task output is passed to the following state but I wanted to pass only the ResponseBody.shouldideploy value.

Moving to the next state: ListAttachedUserPolicies. We will check all the IAM policies attached to this user.

The ListAttachedUserPolicies API call returns a JSON that the only relevant field is AttachedPolicies: an array of objects, where each object is composed of PolicyArn and PolicyName. JSONata allows us to extract values from a key of an array of objects and return this as a list of values. Then we can use the in comparison to check if a given value is in that list:

{
  "policyAttached": "{% 'arn:aws:iam::aws:policy/AWSDenyAll' in $states.result.AttachedPolicies.PolicyArn %}"
}

Notice how $states.result.AttachedPolicies.PolicyArn is powerful! This would be a pain to do using JSONPath, since the JSONPath in Step Functions do not allow some operators that could help us in this case. Check the state input/output during execution:

Now, with the two relevant pieces of information in our hands (deploy and policyAttached) we can define what action will be executed:

If we can not deploy and the policy is not attached -> We would like to attach the deny policy
If we can deploy and the policy is attached -> We would like to detach the deny policy
If we can deploy and policy is not attached OR if we can not deploy and policy is attached -> We are already done and nothing should be done

Notice how we are using the variable previously created ($deploy) and also using the ListAttachedUserPolicies output which is now the input of our choice state ($states.input.policyAttached). I also could have exported the policyAttached as a variable and used it in the choice state!

Both AttachUserPolicy and DetachUserPolicy accept the exact same input: the ARN of the policy and the IAM User name.

Here is a final workflow execution in a case where the IAM User was blocked from deploying but now it's allowed to deploy (i.e. the state machine should detach the deny policy):

Sync jobs, callback, and pooling pattern

By default, Step Functions execute states and move to the following state when it gets a response. Sometimes this behavior is not desired, such as:

You would like to wait for an execution of an ECS Task or Batch Job
You would like to treat some asynchronous jobs as synchronous
You would like to continue the execution only after a manual approval/action

Sync Jobs

Some job execution in AWS is asynchronous, i.e. the API to start the job (e.g. Glue StartJobRun, Batch SubmitJob, ECS RunTask) and return a successful response, which means that the action to initiate the job was successful, but does not monitor the job itself (i.e. if the job execution will fail or succeed).

When adding these types of states in the state machine, you can optionally mark these jobs as synchronous and AWS will internally use pooling (EventBridge events and polls APIs) to monitor the job status. This is as simple as marking a checkbox in the Step Functions Console or as simple as adding .sync at the end of the resource name (note that not all Jobs accept this - please check the documentation) in the JSON definition, e.g.:

"Glue StartJobRun": {
  "Type": "Task",
  "Resource": "arn:aws:states:::glue:startJobRun.sync",
  "Arguments": {
    "JobName": "myJobName"
  },
  "Next": "Batch SubmitJob"
}

You could consider the Lambda Invoke as a job, but in reality, AWS treats this synchronous by default, so if you run the aws lambda invoke command, the command will be pending until the Lambda finishes (success or error). So this is the reason why Lambdas are not considered as an async job.

This feature is only available for Standard Step Functions.

Task Token Callback

Some states allow you to pause the execution of the workflow and wait for a token to be returned (up to 1 year!). When this feature is enabled, the state can pass a token (available in the context JSON), e.g. if it's an SQS message it can pass this token in the message, if it's an SQS topic it can pass in the message of the topic, if it's a Lambda invoke state it can pass the Lambda in the Payload.

The step function execution will wait until it receives a SendTaskSuccess or SendTaskFailure API action (via AWS CLI or SDK) passing the token as input. This becomes very handy when you need manual approval or wait for external events!

This feature is only available for Standard Step Functions.

Pooling Pattern

Some services can not be turned into sync jobs (e.g. AWS Transcribe) and task token callback can be overkill or unnecessary for your use case, so a common pattern is to create a pooling system using wait state, choice state, and an AWS API state to check the status of a job. The pooling pattern uses Choice, a Wait, and an API Action to check the status of the job: Wait X seconds Check status using the API call If the status is ready, move to another state to deal with a success response. If it's not ready, go to Wait step again. If the status is a failure, move to another state to deal with the failure.

Exercise 02: Manual approval for creating a new IAM User

This is the second exercise of this blog post! All the Terraform code for deploying this is in the GitHub Repository. From now on, I won't comment on every detail about the workflow, I highly encourage you to deploy the code in your AWS account and check, test, and explore. If I explain in deep detail every exercise this post will be tedious and extremely long (more than it already is).

Imagine a company that is constantly growing and would like to create AWS users (IAM users) fast but only if the manager approves it. Also, the manager can select if that person will be assigned with admin permissions or developer permission. Here is the proposed Step Function diagram:

When the Step Function gets triggered with the username (string) to create an email is forwarded (via SNS topic) to the manager.

The manager will use AWS CLI to approve or deny the creation of the new user and decide if the user will be an admin or not.

# Deny
aws stepfunctions send-task-failure --task-token $TASK_TOKEN

# Approve as admin
aws stepfunctions send-task-success \
 --task-output '{"admin": true}' \
    --task-token $TASK_TOKEN

# Approve as developer
aws stepfunctions send-task-success \
 --task-output '{"admin": false}' \
    --task-token $TASK_TOKEN

Running the send-task-success action with admin set to true we will have the user created as expected!

In a real-world scenario, you would create something more sophisticated (e.g. send only the URL to approve/deny integrating with another system that will perform the Step Functions send-task-success or send-task-failure). For learning purposes the example above works great!

Exercise 03: Bucket replication with pooling system

This is the third exercise of this blog post! All the Terraform code for deploying this is in the GitHub Repository.

As commented earlier, some AWS actions can not be turned into synchronous jobs. In this case, the solution is to implement a pooling system which is a loop that after the run job action you will:

Get the status of the job
Check the status of the job
If the job is still running sleep for X seconds
If the job finished with success/error you will handle these cases

Imagine you would like to manually trigger a bucket replication using Data Sync. Since the DataSync StartTaskExecution action does not allow synchronous jobs we will implement the pool system to wait until it finishes with success/error. For simplicity, we will simply exit the step function execution with success or error but you could implement additional logic that makes sense in your use case.

Here is the Step Function graph:

Once the step function gets triggered the DataSync task will start and the workflow will check the task execution state every 10s. The easiest way to visualize that the pooling system is working is by checking the state view.

Notice the loop: Wait 10s -> DescribeTaskExecution -> Job Complete?.

Exercise 04: Profile picture resize

This is the second exercise of this blog post! All the Terraform code for deploying this is in the GitHub Repository. This will be the only exercise which I've created two definitions: one in JSONata and another in JSONPath, so you can deploy as you wish and check the differences. The goal of this exercise is to show you how to deploy with Terraform.

Imagine the following scenario: your platform accepts profile pictures of the users but you need to standardize the profile picture size to use in two cases: small pictures (128x128 px) and big pictures (512x512 px). You would like the user to upload it's profile picture (PNG file) and then resize the image of the profile picture automatically. We could automate that using the following infrastructure.

When the bucket receives new profile pictures it will send this information to the SQS Queue via S3 events notification. The Step Function will consume the queue using EventBridge Pipe (since there is no direct integration between SQS and Step Function directly).

The following Step Function will be used:

Notice that the Step Function will consume the SQS Queue in batches, i.e. the input of the Step Function workflow can be more than one event about the image uploaded, that's why we are using the Map flow state. For every profile picture uploaded in the input event of the workflow we would like to (in parallel) execute the lambda that will resize the profile picture to the desired size. We store in the DynamoDB the S3 key of the resized profile picture, this will be used in the backend application (out of scope of this project). Finally, when the images are resized we can delete the original profile picture, since it's not helpful anymore.

The main entry file of the Terraform code is available below:

// main.tf
locals {
  project = "resize-profile-picture"
}

##############################
##### S3 + SQS
##############################
resource "aws_s3_bucket" "this" {
  bucket_prefix = "profile-pictures"
}

resource "aws_sqs_queue" "this" {
  name = "profile-pictures"
}

resource "aws_sqs_queue_policy" "this" {
  queue_url = aws_sqs_queue.this.id
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Id" : "AllowS3Notification",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "Service" : "s3.amazonaws.com"
        },
        "Action" : [
          "SQS:SendMessage"
        ],
        "Resource" : aws_sqs_queue.this.arn,
        "Condition" : {
          "ArnLike" : {
            "aws:SourceArn" : aws_s3_bucket.this.arn
          }
        }
      }
    ]
  })
}

resource "aws_s3_bucket_notification" "this" {
  bucket = aws_s3_bucket.this.id

  queue {
    queue_arn     = aws_sqs_queue.this.arn
    events        = ["s3:ObjectCreated:*"]
    filter_prefix = "uploaded/"
  }
}

##############################
##### DYNAMODB TABLE
##############################
resource "aws_dynamodb_table" "this" {
  name         = "user"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "userId"

  attribute {
    name = "userId"
    type = "S"
  }
}

##############################
##### LAMBDA & ECR
##############################
resource "aws_ecr_repository" "this" {
  name                 = "${local.project}"
  image_tag_mutability = "MUTABLE"
}

resource "docker_image" "this" {
  name = "${aws_ecr_repository.this.repository_url}:latest"

  build {
    context  = "${path.cwd}/../lambda/resize-image"
    platform = var.lambda_architecture == "arm64" ? "linux/arm64" : "linux/amd64"
  }
  force_remove = true
}

resource "docker_registry_image" "this" {
  name          = docker_image.this.name
  keep_remotely = false
}

resource "aws_lambda_function" "this" {
  depends_on = [docker_registry_image.this]

  function_name = local.project
  role          = aws_iam_role.lambda.arn
  package_type  = "Image"
  image_uri     = "${aws_ecr_repository.this.repository_url}:latest"
  architectures = [var.lambda_architecture]
  timeout       = 10

  environment {
    variables = {
      BUCKET_NAME = aws_s3_bucket.this.id
    }
  }
}

resource "aws_iam_role" "lambda" {
  name = "lambda-${local.project}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_cloudwatch_log_group" "lambda" {
  name = "/aws/lambda/${local.project}"
}

resource "aws_iam_policy" "lambda" {
  name = "lambda-${local.project}"
  path = "/"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid = "S3Actions",
        Action = [
          "s3:List*",
          "s3:Get*",
          "s3:Put*",
        ]
        Effect = "Allow"
        Resource = [
          "${aws_s3_bucket.this.arn}",
          "${aws_s3_bucket.this.arn}/*"
        ]
      },
      {
        Sid = "CloudWatchActions"
        Effect = "Allow",
        Action = [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
        Resource = "${aws_cloudwatch_log_group.lambda.arn}:*"
    }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda" {
  role       = aws_iam_role.lambda.name
  policy_arn = aws_iam_policy.lambda.arn
}

##############################
##### STEP FUNCTIONS
##############################
resource "aws_iam_role" "sfn" {
  name = "sfn-${local.project}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "states.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_policy" "sfn" {
  name = "sfn-${local.project}"
  path = "/"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid = "S3Actions",
        Action = [
          "s3:List*",
          "s3:Get*",
          "s3:Put*",
          "s3:Delete*",
        ]
        Effect = "Allow"
        Resource = [
          "${aws_s3_bucket.this.arn}",
          "${aws_s3_bucket.this.arn}/*"
        ]
      },
      {
        Sid = "LambdaActions",
        Action = [
          "lambda:InvokeFunction",
        ]
        Effect   = "Allow"
        Resource = [
          "${aws_lambda_function.this.arn}",
          "${aws_lambda_function.this.arn}*"
        ]
      },
      {
        Sid = "DynamoDBActions",
        Action = [
          "dynamodb:UpdateItem",
        ]
        Effect   = "Allow"
        Resource = "${aws_dynamodb_table.this.arn}"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "sfn" {
  role       = aws_iam_role.sfn.name
  policy_arn = aws_iam_policy.sfn.arn
}

resource "aws_sfn_state_machine" "this" {
  name     = "${local.project}"
  role_arn = aws_iam_role.sfn.arn
  definition = templatefile(
    "machine-definition/resize-image/${var.query_language}.json",
    {
      LAMBDA_NAME         = aws_lambda_function.this.id
      DYNAMODB_TABLE_NAME = aws_dynamodb_table.this.id
    }
  )
}

##############################
##### EVENTBRIDGE PIPE
##############################
resource "aws_iam_role" "pipe" {
  name = "pipe-${local.project}"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "pipes.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_policy" "pipe" {
  name = "pipe-${local.project}"
  path = "/"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid = "SqsAccess",
        Action = [
          "sqs:ReceiveMessage",
          "sqs:DeleteMessage",
          "sqs:GetQueueAttributes"
        ]
        Effect   = "Allow"
        Resource = "${aws_sqs_queue.this.arn}"
      },
      {
        Sid = "StepFunctionsAccess",
        Action = [
          "states:StartExecution",
          "states:StartSyncExecution"
        ]
        Effect   = "Allow"
        Resource = "${aws_sfn_state_machine.this.arn}"
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "pipe" {
  role       = aws_iam_role.pipe.name
  policy_arn = aws_iam_policy.pipe.arn
}

resource "aws_pipes_pipe" "this" {
  name     = "trigger-${local.project}-sfn"
  role_arn = aws_iam_role.pipe.arn
  source   = aws_sqs_queue.this.arn
  target   = aws_sfn_state_machine.this.arn

  source_parameters {
    sqs_queue_parameters {
      batch_size = 5
      maximum_batching_window_in_seconds = 5
    }
  }

  target_parameters {
    step_function_state_machine_parameters {
      invocation_type = "FIRE_AND_FORGET"
    }
  }
}

Let's break down this code into pieces. We created the:

S3 bucket to upload the profile pictures and store the small and big sizes.
SQS Queue that will receive the S3 events when new profile pictures get uploaded. We also make sure that the Queue Policy allows the SQS:SendMessage action from the S3 bucket.
DynamoDB table that contains metadata about the resized profile picture
ECR repository and Lambda Function to run the resize profile picture code (more on that later). The kreuzwerker/docker provider was used to build and push the Lambda code from the IaC code. The Lambda Function was granted write permissions to the S3 bucket and CloudWatch (for logging).
Step Function was created and used the JSON definition as a template (i.e. the JSON code contains placeholders that are substituted during IaC deployment: LAMBDA_NAME and DYNAMODB_TABLE_NAME). The reason why this was done is that it allowed us to reference resources created by IaC with IaC code reference instead of hardcoding them (ARNs, resource names...) in the JSON definition, which allows us to use a single JSON definition for all environments, instead of creating a hardcoded JSON definition per environment (Dev/Staging/Production). The Step Function was granted the needed permissions to interact with the S3 bucket, Lambda Function, and DynamoDB Table.
EventBridge Pipe to trigger Step Functions from SQS Queue.

The code used some variables:

// variables.tf
variable "aws_region" {
  description = "AWS region to deploy the resources"
  type        = string
  default     = "us-east-1"
}

variable "lambda_architecture" {
  description = "Instruction set architecture for your Lambda function. Valid values are 'x86_64' 'arm64'. Since we are also deploying the image via local deployemnt this should correspond to your computer architecture"
  type        = string
  default     = "arm64"
}

variable "query_language" {
  description = "Query Language to use in the Step Functions"
  type        = string
  default     = "jsonpath"

  validation {
    condition     = contains(["jsonata", "jsonpath"], var.query_language)
    error_message = "Valid values for var: test_variable are: 'jsonata', 'jsonpath'"
  }
}

I also mentioned about the Lambda Code. Let's take a look.

import io
import os

import boto3
from PIL import Image

s3 = boto3.client('s3', region_name=os.getenv("AWS_REGION", "us-east-1"))

def resize_image_s3(
    bucket_name: str,
    input_key: str,
    output_key: str,
    size: int
) -> None:
    print("Downloading uploaded profile picture...")
    response = s3.get_object(Bucket=bucket_name, Key=input_key)
    image_data = response['Body'].read()

    print(f"Resizing image to {size}x{size}...")
    image = Image.open(io.BytesIO(image_data))
    image_resized = image.resize((size, size))
    buffer = io.BytesIO()
    image_resized.save(buffer, format=image.format)
    buffer.seek(0)

    print("Uploading resized profile picture to S3...")
    s3.put_object(
        Bucket=bucket_name,
        Key=output_key,
        Body=buffer,
        ContentType=response['ContentType']
    )

def lambda_handler(event, context):
    print(f"{event = }")
    print(f"{context = }")

    try:
        bucket_name = event["body"]["s3"]["bucket"]["name"]
        size = event["size"]
        uploaded_profile_picture_key = event["body"]["s3"]["object"]["key"]
        picture_name = uploaded_profile_picture_key.split("/")[-1]
        user_id = picture_name.split(".png")[0]
        resized_profile_picture_key = f"{user_id}/{size}.png"

        resize_image_s3(
            bucket_name,
            uploaded_profile_picture_key,
            resized_profile_picture_key,
            size
        )
        return {
            "statusCode": 200,
            "data": {
                "message": "Image resized!",
                "userId": user_id,
                "bucket": bucket_name,
                "uploadedKey" : uploaded_profile_picture_key,
                "resizedKey": resized_profile_picture_key
            }
        }
    except Exception as e:
        print(e)
        raise Exception(f"Something went wrong: {e}")

It is a simple code that resizes the image using PIL library.

Cya!

I hope you've liked it and that this demo and explanation was helpful! Remember that reading this blog post is not enough, I highly encourage you to deploy the examples mentioned here and also create your own workflow!

See you around! 👋

Deploying a serverless, cheap, and scalable infrastructure using Terraform in AWS

Wed, 04 Dec 2024 10:40:32 GMT

In today's blog post, I would like to comment on costs, scalability, and my thoughts about an initial architecture for a new company/product. All the code of this blog post will be available on GitHub.

I've worked with several different clients and it still shocks me how some care about technology more than the business. They focus so much on using new technologies, hyped tools, and writing their own solutions for problems that already exist that they simply ignore the economic and business parts of the organization.

During my time working in new startups, I notice that:

Time is key: The company is still trying to validate an idea or a product and needs to do that fast. You don't need to deliver a full/complete product, only the minimum viable product (MVP) to validate if what the company is trying to do is feasible.
Money is tight: Money is not infinite, the company needs to spend the money in the correct place. Use it wisely and remember that (developer) time is money.
You are not a MANG It's so common to see entrepreneurs thinking that their idea is so extraordinary that they will be the next MANG (Meta, Apple, Netflix, Google). Well, I expect them to grow, but let's keep our feet on the ground. It's amazing to read MANG's tech blogs and see the solution they are using for their problems (i.e. tools, strategies...) but remember you are NOT a FANG. Their blog post means: that if you were us, facing the same problems that we are, maybe, and just maybe, the proposed solution here will work for you. Start small, grow constantly, and modify your infrastructure and software as you grow. There is a well-known sentence in the software engineering community that says "Premature Optimization is the root of all evil".

It's shocking the amount of time that I saw over complex architectures in startups that don't even have clients: Kubernetes, multi-regional deployment, extra costs involved in compliance (we not needed)... This only increases the overall costs of the startup and reduces the time to market of the product.

Here, I will propose a simple, scalable and cheap infrastructure, ideal for most startups. This is 100% opiniated, and that's fine, it might not be viable in all scenarios but it's a good start for a good amount of startups. I have DevOps friends that do not like to use managed services at all, and hate serverless and Lambdas and that's completely ok, just make sure you are aware of the pros and cons of each approach.

Full Stack Architecture

A basic full-stack architecture is composed of at least three parts:

Backend: A piece of software that will be responsible for handling the bussiness logic
Database: The backend needs to store information in a database
Frontend: Website that will display information for the users. It's always communicating with the backend to get data.

Once again, this is the basic infrastructure. We are missing several other parts (e.g. caching, load balancing, authentication/authorization, observability...) that are still relevant but out of the scope of this blog post.

The classic (not talking about the old days) approach for deploying this infrastructure is:

Deploy the Database (SQL - such as PostgreSQL - or NoSQL - such as MongoDB)
Deploy the API and expose it by an API Gateway/Load Balancer (e.g. Kong, NGINX)
Deploy the frontend by exposing the assets via a web server (e.g. NGINX)

This is a perfect approach, nothing is wrong but notice that this involves having a server running 100% of the time to respond to incoming requests (shutting down all the servers to reduce costs means downtime for clients) and you also will need to manage scalability (if a sudden spike of traffic comes, you need to spin up more servers).

Once again, this might be a huge cost for a new startup that barely has clients. Here is a more cost-effective approach:

Use a serverless, pay for what you use, database: DynamoDB. Notice that this is a NoSQL database and might not fit the needs of your application. For new accounts, AWS offers a 12-month free tier for RDS small databases, this might be a good alternative.
Deploy the API using Lambda Function and expose it using API Gateway, you only pay for the traffic, storage of Lambda code (ECR or S3 zip object), and Lambda execution. For Lambda, AWS offers a monthly always-free tier of 1 million requests and up to 400,000 GB-seconds of execution (which means 111 hours of execution in a month when using a 1GB X86 architecture Lambda). Amazon API Gateway has a 12-month free tier of 1 million requests per month, but even without the free tier, the cost is $1.00 per million requests.
Deploy the frontend using Amazon S3 for storing the website assets and distributing it at edge locations using CloudFront. The always free tier of CloudFront is great, with 1TB of data transfer out and 10 million requests per month. S3 will be basically free since we expect to cache all the static content in CloudFront and the size of the website assets is small.

Serverless

Serverless, by definition, is the lack of servers, which is contradictory because you must have CPU, Memory, and Disk in order to run any software. So, what does it mean? Well, it basically means that your application will run in a way that you won't even see the servers running your application; you won't need to manage the virtual servers running or code; and you won't need to worry about managing failures, restarts and scalability.

Serverless was highly related to "pay as you go" and "pay by what you use". Meaning that, if you don't use it, you won't pay for it. This is partially true since we have serverless products in AWS that will charge you even if you are not using them (e.g. RDS Serverless, Fargate). So that's why I rather stick with the "you don't manage servers" instead of thinking it's free if you don't use it.

Cloud is to On-Premise as Virtual Servers is to Serverless. If you understand this sentence then you understand the serverless concept.

Frontend: S3 + CloudFront

I've already talked about S3 and CloudFront to deploy a website in a previous blog post called Deploying a static website on AWS using S3 and CloudFront but I will talk about it again. I highly recommend you read the CloudFront section and subsections. Be aware that this method of deploying a website only works if your Javascript code is CRS (Client Side Rendering), so you can't use frameworks that are SSR (e.g. NextJS), only CSR (e.g. React). Another super important thing to mention is that when using SSL/TLS certificates with CloudFront you must issue the certificate in the us-east-1 region.

There are two easy ways of deploying a website using S3:

I could make a comparison between these methods but I really don't see a reason why you would prefer only S3 with website hosting configuration enabled. Why? Well, check the disadvantages of using this method:

It does not support HTTPS. This is a big NO, but let's keep going...
Limited to a specific location, which may cause delays to users in different geo locations
Bad SEO
S3 bucket needs to be public

Because of that I really don't think it's worth it to even use the S3 website hosting option when you can use the CloudFront with S3 origin as an alternative. With the latter you have all the benefits that are cons in S3 website hosting plus:

Advanced caching behavior
Geographic restriction
Custom headers (good for SEO, and security)

So the idea here is to create a CloudFront distribution and select the S3 bucket as origin. This bucket will contain the build of our frontend website. When a distribution gets created you will get a URL (something like d111111abcdef8.cloudfront.net) that is your endpoint for accessing your website. To simplify the way of accesing your website, you can create an A record (with alias) pointing to this distribution (e.g. app.mydomain.com).

Let's take a step back and talk about how CloudFront accesses the origin (S3 bucket in our case). In a single CloudFront distribution, you can have several origins (e.g. ELB, API Gateway, S3 bucket) and the way you control which origin will be used is by controlling the behavior of the distribution and you can define access patterns based on the URL path to decide what origin will be used. You can also specify a default behavior, which, in our use case it's enough, since there is only a single origin in our distribution.

When controlling the behavior we can specify other things, such as:

Protocol policy: HTTP and HTTPS; redirect HTTP to HTTPS; only HTTPS.
Allowed HTTP methods: GET, HEAD; GET, HEAD, OPTIONS; GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE.
Caching Policy You can use a custom or a managed caching policy (that defines TTL, Headers, Cookies, and Queries string)
Function Association: CloudFront Functions or Lambda@Edge functions to run based on the response/request. It's useful if you would like to run operations such as URL redirects/rewrites, manipulate the header, access the HTTP body request... I've used Lambda@Edge in a previous blog post to perform URL rewrite!

Backend: API Gateway + Lambda

Lambda offers a way of running a code for a short (up to 15 minutes) duration and you only pay for the storage of the function (in S3 as a ZIP file or in ECR as a Docker Image) and the duration of the Lambda execution. This becomes very interesting when you are working with API, which usually takes a sub-second response time. By default, the default concurrency Lambda limit is 1000, i.e. you can have up to 1000 Lambdas running at the same time in your account, but you can increase this values since it's a soft-limit.

API Gateway offers a secure HTTP endpoint for invoking your Lambda function, while also managing high call volumes through traffic throttling and automating the validation and authorization of API requests. The idea is to proxy all requests to the Lambda. An API Gateway can be one of the following types: HTTP, REST, and WebSocket. The first two are suitable for your API use case and the REST API is basically an HTTP with batteries included (API keys, per-client throttling, AWS WAF integration, caching...). REST API Gateway can also be deployed privately and at the edge, while HTTP is only available publicly in a regional deployment. The REST API comes with more features but it comes with a price, at the first tier of requests (~300million per month - which is likely to be your use-case) it costs $3.5/million of requests, while the HTTP costs a fraction of it: $1/million of requests. To summarize, HTTP API Gateway is cheaper and has lower latency, while REST API Gateway offers advanced features but it's more expensive.

One of the great advantages of using Lambdas is the famous "cold start" that can indeed make your application slower. Cold start varies from under 100ms to over 1s but typically it's under 1s, it depends on several things, such as the number of dependencies, the amount of RAM allocated to the function, the programming language used...

You would need to adapt your code to be "Lambda-compliant" which is something that varies based on the programming language used, so the best thing to do is to directly check the documentation.

Remember that our API Gateway will receive requests from a browser, so we need to be aware of CORS, which means that your code needs to return some headers for the OPTIONS method (more on that in the code).

Database: DynamoDB

DynamoDB is a blazing-fast NoSQL key-value database proprietary service of AWS. It's completely serverless, you don't need to manage anything, you use the service as it is and only pay for storage and transactions (read and writes) in the table. When I say blazing fast I really mean it, it offers a single-digit millisecond latency but it's an expensive service, especially if you do not design your table correctly.

A table in DynamoDB is schemaless, but you need to define a primary key (hash key) that must be unique in the entire table. The primary key can be a single key (partition key) or a composite key (partition key and sort key). The design of a DynamoDB table is completely off this discussion but to make you curious, it's a common strategy to denormalize data, single there are no joins in DynamoDB, so it's a completely different way of design when compared to common SQL tables. Because you pay for reading/writing in the table, a well-designed is so important, since you definitely don't want to avoid scan operation.

Demo Time

Infrastructure as Code

Let's create the entire infrastructure of this code using Terraform! I decided to maximize the amount of community modules when creating the resources to make the code easier to read. If you rather check some "pure" Terraform code you can check this repository that has a similar infrastructure to the one we are creating right now.

Let's define the providers (AWS) that we will use in your terraform code. Notice that I've defined a specific AWS provider to the us-east-1 region (do you rember the TLS/SSL certificates when using CloudFront?).

# providers.tf
provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Project    = "Serverless Infra"
      Repository = "https://github.com/felipelaptrin/serverless-infra-blog"
    }
  }
}

provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"

  default_tags {
    tags = {
      Project    = "Serverless Infra"
      Repository = "https://github.com/felipelaptrin/serverless-infra-blog"
    }
  }
}

Below, I declared all the variables that will be used, notice that only two variables are needed: domain (your domain) and frontend_bucket_name (the bucket name that will contain the frontend assets).

# variables.tf
########################################
##### GENERAL
########################################
variable "aws_region" {
  type        = string
  description = "AWS region to deploy the infrastructure"
  default     = "us-east-1"
}

variable "domain" {
  type        = string
  description = "The domain of your project"
}

variable "logs_retention" {
  type        = number
  description = "CloudWatch Group log retention"
  default     = 90
}

########################################
##### NETWORKING
########################################
variable "vpc_deploy" {
  type        = bool
  description = "Controls the deployment of the VPC resources (VPC, Subnets, Internet Gateway, Route Table...). If you already have a VPC deployed, set this variable to false and set 'vpc_id' variable."
  default     = true
}

variable "vpc_id" {
  type        = string
  description = "VPC ID of the already deployed VPC in your account. To use this, set vpc_deploy to false."
  default     = ""
}

variable "vpc_name" {
  type        = string
  description = "Name of the VPC to deploy"
  default     = "PoC"
}

variable "vpc_cidr" {
  type        = string
  description = "CIDR of the VPC to create. Please use a /16 mask for high compatibility with this module."
  default     = "10.50.0.0/16"
}

########################################
##### BACKEND
########################################
variable "backend_subdomain" {
  type        = string
  description = "Subdomain where the API Gateway will be exposed, i.e. https://{backend_subdomain}/{domain}"
  default     = "api"
}

variable "lambda_name" {
  type        = string
  description = "Name of the Lambda Function"
  default     = "backend-api"
}

variable "lambda_memory" {
  type        = number
  description = "Amount of memory that should be used in the Lambda"
  default     = 256
}

variable "lambda_architecture" {
  type        = string
  description = "Architecture that the Lambda function will run"
  default     = "arm64"

  validation {
    condition     = contains(["x86_64", "arm64"], var.lambda_architecture)
    error_message = "Valid values for var: test_variable are: 'x86_64' and 'arm64'."
  }
}

variable "lambda_timeout" {
  type        = number
  description = "Timeout in seconds of the Lambda"
  default     = 5
}

########################################
##### DATABASE
########################################
variable "table_name" {
  type        = string
  description = "Name of the DynamoDB table"
  default     = "table"
}

variable "table_attributes" {
  type        = list(map(string))
  description = "Attributes of the DynamoDB table"
  default = [
    {
      name = "UserId",
      type = "S",
    },
  ]
}

variable "table_hash_key" {
  type        = string
  description = "Hash key of the DynamodDB table"
  default     = "UserId"
}

########################################
##### FRONTEND
########################################
variable "frontend_subdomain" {
  type        = string
  description = "Subdomain that the Website will be exposed, i.e. https://{frontend_subdomain}/{domain}"
  default     = "app"
}

variable "frontend_bucket_name" {
  type        = string
  description = "Name of the S3 bucket that will contains the frontend website"
}

Now, let's check some "helper" files (data sources and locals).

# locals.tf
locals {
  vpc_id          = var.vpc_deploy == true ? module.vpc[0].vpc_id : var.vpc_id
  vpc_ip          = cidrhost(var.vpc_cidr, 0)
  all_subnets     = [for i in range(4) : cidrsubnet(var.vpc_cidr, 8, i + 1)]
  public_subnets  = slice(local.all_subnets, 0, 2)
  private_subnets = slice(local.all_subnets, 2, 4)
  account_id      = data.aws_caller_identity.current.account_id
}

# data.tf
data "aws_caller_identity" "current" {}

data "aws_route53_zone" "this" {
  name = var.domain
}

And finally, let's check the main code.

# main.tf

########################################
##### NETWORKING
########################################
module "vpc" {
  count = var.vpc_deploy == true ? 1 : 0

  source  = "terraform-aws-modules/vpc/aws"
  version = "5.15.0"

  name = var.vpc_name
  cidr = var.vpc_cidr

  azs             = ["${var.aws_region}a", "${var.aws_region}b"]
  private_subnets = local.private_subnets
  public_subnets  = local.public_subnets
}

########################################
##### DATABASE
########################################
module "dynamodb-table" {
  source  = "terraform-aws-modules/dynamodb-table/aws"
  version = "4.2.0"

  name         = var.table_name
  attributes   = var.table_attributes
  hash_key     = var.table_hash_key
  billing_mode = "PAY_PER_REQUEST"
}

########################################
##### BACKEND
########################################
module "ecr" {
  source  = "terraform-aws-modules/ecr/aws"
  version = "2.3.0"

  repository_name                 = "backend-api"
  repository_image_tag_mutability = "MUTABLE"
  repository_image_scan_on_push   = false
  create_lifecycle_policy         = false
}


module "lambda_function" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "7.14.0"

  function_name                     = var.lambda_name
  description                       = "Lambda Function based API serving as backend for the app"
  create_package                    = false
  architectures                     = [var.lambda_architecture]
  image_uri                         = "${module.ecr.repository_url}:latest"
  package_type                      = "Image"
  cloudwatch_logs_retention_in_days = var.logs_retention

  attach_network_policy = true
  vpc_subnet_ids        = module.vpc[0].private_subnets

  memory_size = var.lambda_memory
  timeout     = var.lambda_timeout
  environment_variables = {
    TABLE_NAME        = var.table_name
    FRONTEND_ENDPOINT = "https://${var.frontend_subdomain}.${var.domain}"
  }

  create_current_version_allowed_triggers = false
  allowed_triggers = {
    ApiGateway = {
      service    = "apigateway",
      source_arn = "arn:aws:execute-api:${var.aws_region}:${local.account_id}:${module.api_gateway.api_id}/*/*"
    }
  }

  attach_policy_json = true
  policy_json = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "dynamodb:Batch*",
          "dynamodb:GetItem",
          "dynamodb:PutItem",
          "dynamodb:UpdateItem",
          "dynamodb:Query",
          "dynamodb:Scan",
        ]
        Effect   = "Allow"
        Resource = "${module.dynamodb-table.dynamodb_table_arn}"
      },
    ]
  })
}

module "api_gateway" {
  source  = "terraform-aws-modules/apigateway-v2/aws"
  version = "5.2.0"

  name             = "backend-gateway"
  description      = "HTTP API Gateway for the Lambda-based API"
  protocol_type    = "HTTP"
  domain_name      = "${var.backend_subdomain}.${var.domain}"
  hosted_zone_name = var.domain
  subdomains       = [var.backend_subdomain]

  cors_configuration = {
    allow_headers = ["*"]
    allow_methods = ["*"]
    allow_origins = ["https://${var.frontend_subdomain}.${var.domain}"]
  }

  routes = {
    "$default" = {
      integration = {
        uri                    = "arn:aws:lambda:${var.aws_region}:${local.account_id}:function:${var.lambda_name}" // module.lambda_function.lambda_function_arn
        payload_format_version = "2.0"
      }
    }
  }
}

########################################
##### FRONTEND
########################################
module "s3_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "4.2.2"

  bucket = var.frontend_bucket_name

  attach_policy = true
  policy = jsonencode({
    "Version" : "2008-10-17",
    "Id" : "PolicyForCloudFrontPrivateContent",
    "Statement" : [
      {
        "Sid" : "AllowCloudFrontServicePrincipal",
        "Effect" : "Allow",
        "Principal" : {
          "Service" : "cloudfront.amazonaws.com"
        },
        "Action" : "s3:GetObject",
        "Resource" : "arn:aws:s3:::${var.frontend_bucket_name}/*",
        "Condition" : {
          "StringEquals" : {
            "AWS:SourceArn" : "arn:aws:cloudfront::${local.account_id}:distribution/${module.cdn.cloudfront_distribution_id}"
          }
        }
      }
    ]
  })

  providers = {
    aws = aws.us_east_1
  }
}

module "acm" {
  source  = "terraform-aws-modules/acm/aws"
  version = "5.1.1"

  domain_name       = var.domain
  validation_method = "DNS"
  zone_id           = data.aws_route53_zone.this.id

  subject_alternative_names = [
    "${var.frontend_subdomain}.${var.domain}",
  ]
  wait_for_validation = true

  providers = {
    aws = aws.us_east_1
  }
}

module "cdn" {
  source  = "terraform-aws-modules/cloudfront/aws"
  version = "3.4.1"

  aliases             = ["${var.frontend_subdomain}.${var.domain}"]
  comment             = "CDN of Frontend"
  price_class         = "PriceClass_All"
  is_ipv6_enabled     = true
  default_root_object = "index.html"

  create_origin_access_control = true
  origin_access_control = {
    s3_bucket_frontend = {
      description      = "Frontend assets bucket"
      origin_type      = "s3"
      signing_behavior = "always",
      signing_protocol = "sigv4"
    }
  }
  origin = {
    s3 = {
      domain_name           = "${var.frontend_bucket_name}.s3.us-east-1.amazonaws.com"
      origin_access_control = "s3_bucket_frontend"
    }
  }

  default_cache_behavior = {
    target_origin_id       = "s3"
    viewer_protocol_policy = "redirect-to-https"
    allowed_methods        = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods         = ["GET", "HEAD"]
    cache_policy_id        = "658327ea-f89d-4fab-a63d-7e88639e58f6" # CachingOptimized - Recommended for S3
    use_forwarded_values   = false
    compress               = true
  }

  viewer_certificate = {
    acm_certificate_arn      = module.acm.acm_certificate_arn
    ssl_support_method       = "sni-only"
    minimum_protocol_version = "TLSv1.2_2021"
  }
}

module "records" {
  source  = "terraform-aws-modules/route53/aws//modules/records"
  version = "4.1.0"

  zone_name = data.aws_route53_zone.this.name

  records = [
    {
      name = "${var.frontend_subdomain}"
      type = "A"
      alias = {
        name    = module.cdn.cloudfront_distribution_domain_name
        zone_id = module.cdn.cloudfront_distribution_hosted_zone_id
      }
    }
  ]
}

To recap, the following was created:

VPC (for the entire project)
CloudFront distribution (S3 bucket as origin)
S3 bucket (frontend assets)
API Gateway (Lambda as Target)
ECR repository (store backend Docker image)
Lambda (serve as Backend)
ACM Certificates (for HTTPs in the API Gateway and CloudFront)
Route53 records (for frontend and backend endpoints)
DynamoDB table

If you try to run this for the first time it will fail. Why? Because the code will attempt to create an ECR repository and use the latest image in this repository to provision the Lambda. This is impossible to happen. What needs to be done is: create ECR repository -> push docker image to repository -> deploy Lambda function.

A simple option to make this deployement work is: apply terraform -> fail -> push image to ECR repo -> apply terraform again.

Backend Code Example (Golang)

I will show a quick Golang code that when receives a POST request adds records in the DynamoDB table and OPTIONS for the CORS.

package main

import (
  "context"
  "encoding/json"
  "fmt"
  "log"
  "os"
  "time"

  "github.com/aws/aws-lambda-go/events"
  "github.com/aws/aws-lambda-go/lambda"
  "github.com/aws/aws-sdk-go-v2/aws"
  "github.com/aws/aws-sdk-go-v2/config"
  "github.com/aws/aws-sdk-go-v2/service/dynamodb"
  "github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
  "github.com/aws/aws-sdk-go-v2/service/sts"
)

type User struct {
  UserId    string `json:"userId"`
  Name      string `json:"name"`
  Email     string `json:"email"`
  CreatedAt string `json:"createdAt"`
}

func getEnv(key, fallback string) string {
  value, exists := os.LookupEnv(key)
  if !exists {
    value = fallback
  }
  return value
}

var dynamoClient *dynamodb.Client
var tableName = getEnv("TABLE_NAME", "table")

func init() {
  cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion(getEnv("AWS_REGION", "us-east-1")))
  if err != nil {
    log.Fatalf("Failed to load AWS config: %v", err)
  }

  dynamoClient = dynamodb.NewFromConfig(cfg)

  stsClient := sts.NewFromConfig(cfg)
  identity, err := stsClient.GetCallerIdentity(context.TODO(), &sts.GetCallerIdentityInput{})
  if err != nil {
    log.Printf("Failed to get caller identity: %v", err)
  } else {
    log.Printf("Lambda is running as: %s", *identity.Arn)
  }
}

func handler(request events.APIGatewayV2HTTPRequest) (events.APIGatewayProxyResponse, error) {
  if request.RequestContext.HTTP.Method == "OPTIONS" {
    return events.APIGatewayProxyResponse{
      Headers: map[string]string{
        "Access-Control-Allow-Headers": "Content-Type",
        "Access-Control-Allow-Origin":  os.Getenv("FRONTEND_ENDPOINT"),
        "Access-Control-Allow-Methods": "OPTIONS,POST,GET",
      },
      StatusCode: 200,
    }, nil
  }
  if request.RequestContext.HTTP.Method == "POST" {
    return createUser(request)
  }

  return events.APIGatewayProxyResponse{
    Body:       "Method not supported",
    StatusCode: 405,
  }, nil
}

func createUser(request events.APIGatewayV2HTTPRequest) (events.APIGatewayProxyResponse, error) {
  var user User

  if err := json.Unmarshal([]byte(request.Body), &user); err != nil {
    return events.APIGatewayProxyResponse{
      Body:       fmt.Sprintf("Invalid request body: %v", err),
      StatusCode: 400,
    }, nil
  }

  if user.UserId == "" {
    return events.APIGatewayProxyResponse{
      Body:       "UserId must be provided",
      StatusCode: 400,
    }, nil
  }

  getItemOutput, err := dynamoClient.GetItem(context.TODO(), &dynamodb.GetItemInput{
    TableName: aws.String(tableName),
    Key: map[string]types.AttributeValue{
      "UserId": &types.AttributeValueMemberS{Value: user.UserId},
    },
  })
  if err != nil {
    return events.APIGatewayProxyResponse{
      Body:       fmt.Sprintf("Error checking for existing user: %v", err),
      StatusCode: 500,
    }, nil
  }

  if getItemOutput.Item != nil {
    return events.APIGatewayProxyResponse{
      Body:       "User already exists",
      StatusCode: 400,
    }, nil
  }

  user.CreatedAt = time.Now().UTC().Format(time.RFC3339)

  item := map[string]types.AttributeValue{
    "UserId":    &types.AttributeValueMemberS{Value: user.UserId},
    "Name":      &types.AttributeValueMemberS{Value: user.Name},
    "Email":     &types.AttributeValueMemberS{Value: user.Email},
    "CreatedAt": &types.AttributeValueMemberS{Value: user.CreatedAt},
  }

  _, err = dynamoClient.PutItem(context.TODO(), &dynamodb.PutItemInput{
    TableName: aws.String(tableName),
    Item:      item,
  })
  if err != nil {
    return events.APIGatewayProxyResponse{
      Body:       fmt.Sprintf("Failed to save user: %v", err),
      StatusCode: 500,
    }, nil
  }

  responseBody, _ := json.Marshal(map[string]string{
    "message": "User created successfully",
    "userId":  user.UserId,
  })

  return events.APIGatewayProxyResponse{
    Body:       string(responseBody),
    StatusCode: 201,
  }, nil
}

func main() {
  lambda.Start(handler)
}

Cya!

Well, I hope you liked this blog post. See you around! 👋

Email, protocols, and Amazon SES

Fri, 08 Nov 2024 12:55:32 GMT

The goal of today's post is to comment more about the Amazon SES service, email infrastructure, protocols, and domain. The entire source code of this demo is available in my GitHub repository.

How does email work?

Before we dive deep into Amazon SES, we need to take a step back and understand how email works, its concepts, and the protocols involved in the process.

Well, you know what an email is: an electronic message, a "digital letter." But how am I able to send a message from my personal email (e.g., myemail@something.com) to yours (e.g., youremail@otherthing.com)? Let's deep-dive.

When you create an email account in an email provider (e.g., Gmail, Outlook, Yahoo), you are now relying on the provider's infrastructure to handle email operations, such as receiving, sending, filtering/detecting spam, etc. Under the hood, they use several protocols to make sure everything works fine for you.

Protocol for sending emails

The standard protocol for sending mail is known as SMTP (Simple Mail Transfer Protocol). It is a network protocol that defines how servers should exchange information to send mail to another server. It creates a standard in the way the emails should navigate the internet from the sender to the recipient. Notice that SMTP is NOT a protocol that specifies how the mail should be received, this belongs to other protocols that will be covered in the next section.

The SMTP protocol was specified in several different RFC (publications about the base of the internet), such as RFC 772, RFC 780, RFC 788 and RFC 821. As time passed, bugs were found, errors were fixed more secure strategies were created and new RFCs were presented to enhance SMPT, such as RFC 5321. Could you imagine emails being sent without encryption (over TLS/SSL) these days? SMTP protocol was created more than 10 years before the invention of SSL, so that's why updates in these standards are so important.

Now that we have the protocol in place we need software that implements this protocol, i.e. we need an SMTP server. There are plenty of open-source SMTP servers out there, and the most open-source famous one is Postfix. As you probably know, ports are just numbers and you can basically run any service in any port (e.g. SSH by default works on port 22 but you can run in any other port) but there are standards that are always good to know and follow if possible. There are four SMTP ports that you should know about:

25: This is the default SMTP port and it's often blocked by your ISP to prevent spam.
587: This is the standard SMTP port when used with TLS encryption (SMTPS).
465: This was the original port intended to be used with SMTPS but it's considered deprecated, although some providers may still offer this option.
2525: In case the ports above are blocked (by ISP or firewall rules) it is commonly used this port.

When you open your Gmail email, for example, you are using the default Gmail's SMTP server (smpt.gmail.com).

Notice that for interacting with the SMTP server you need an SMTP client. When using Gmail, for example, everything is abstracted in the browser and you don't even realize these technicities. But you can also use SMTP programmatically, e.g. smtplib for Python, sendmail.

Protocol for receiving emails

There are two main protocols for receiving mails: POP3 (Post Office Protocol version 3) and IMAP (Internet Message Access Protocol). I have already used the IMAP with Python in my configuring an email forward to enhance your phone security in case of a robbery blog post.

Similarly to the SMTP, POP3 and IMAP were proposed in several different RFCs that I will not go into detail. Both protocols have the same goal: retrieve mail, but they do that using different ways and approaches. The main difference between these protocols is that IMAP reads the emails from the email server without deleting them, while POP3 downloads and deletes the emails from the server, because of that, POP3 downloads the emails from the server allowing the emails to be accessible offline, while in IMAP it doesn't.

With the protocol established we need, once again, software to implement it, following all the rules and conventions. By default, these servers use the following ports:

143: IMAP.
993: IMAP over SSL.
110: POP3.
995: POP3 over SSL.

There are several open-source software that implement these protocols, such as Wildduck and Dovecot.

Domain

When you send someone an email (e.g. youremail@otherthing.com) how does it know "how to get" to your inbox? Domain resolution. Computers do not understand names, only IPs, that's why DNS exists. otherthing.com is a domain and the nameservers will tell you the IP of the email servers. If you have a domain and would like to setup an email you would need to configure some records in your DNS. Some records are important to be set in your domain such as:

Email record: A MX record that will specify the mail server(s).
SPF: A TXT record used to specify what servers are authorized to send emails on behalf of your domain (the mail provider that will send the emails). It helps receiving servers verify that emails claiming to come from your domain are sent by legitimate sources.
DKIM: A TXT record that contains the public key. The emails from your domain will be signed by your private key and people that receive the email can verify if it was signed by you by checking it using the public key. This is usually generated by the email provider and you only need to add to your DNS.
DMARC: A TXT record that specifies what should email servers do (nothing/quarantine/discard) if they receive an email that failed SPF/DKIM checks.

Some of these records were set in my configuring a free email provider for your domain using AWS Route53 and Zoho blog post.

Missing pieces

The real email infrastructure with all components of a real provider (e.g. Gmail) is really big and complex but there are so many more parts that are not relevant for this post, such as:

Email Storage: Store all the mail, and attachments somewhere that should be accessible by the user.
Anti-Spam and Anti-Virus filters: Smart detection of phishing, false, and spam mail.
Web interface: Access your inbox via browser.

SES

Amazon SES (Simple Email Service) is a service responsible for sending emails using its interface/API or using SMTP. You may probably be thinking: "Ok... can't I simply set up my own SMTP server and send emails using it?". Indeed you can, but there are a few catches: you would need to make sure your server is scalable and secure; would need to manage retries and assure high deliverability; would need to establish IP addresses with a good reputation (otherwise your emails will always arrive in the receipt spam folder); would need to set up feedback mechanism (bounced and undelivered emails). That's why services like Amazon SES, SendGrid, Mailchimp, and Mailgun exist, for you to focus only on your application.

Be aware that SES by default puts your environment in a sandbox mode, where you can only send emails to emails verified by SES. This is ok for testing but for production, this makes no sense, so you should open a ticket to AWS informing that you would like to move out from the sandbox mode, specifying why and how you going to send email.

Identity

There are two ways of proving that you own the email (validate identity) you would like to send the emails from:

Domain validation: To prove that you own the domain you will need to set up DKIM records. You can either choose Easy DKIM (AWS will generate a public-private key for you to use and will ask you to add the public key in your DNS) or BYODKIM (you provide the private key instead of relying on AWS to create one for you). And later you can add SPF and DMARC records to reduce the chances of your email going to the spam folder. It's very common for companies to use an email provider (e.g. Outlook, Google Workspace) with the domain (e.g. example.com) and create a subdomain for sending an email (e.g. mail.example.com or marketing.example.com).
Email address validation: To prove that you own that email address, AWS will send a mail to the provided email and you should confirm it by clicking the confirmation link provided. By confirming that you own the email address, you are allowing Amazon SES to send emails using your email address in the "From" field.

There are a few differences in choosing one over another:

Set up complexity: Email address identity is way faster to set up when compared to domain identity.
Deliverability: Domain identity is always used to achieve optimal deliverability because the receipt can always validate if the email using DNS records. You can also send emails from any subdomain or email address of the domain identity, while when using email address identity you can only send from the specified email.

Sending emails using SES

When using Amazon SES there are two ways to send emails:

API: Amazon SES exposes a public API that you can use in three ways: making direct HTTPS requests, using AWS SDK or using AWS CLI. The first one (HTTPS request) is less commonly used because you will need to handle authentication, signing of requests and manually construct the request. The second one (AWS SDK) is very common, especially if your application is already running inside AWS (Lambda, ECS, EC2) and it's pretty easy to use with your favorite programming language (e.g. Python, Javascript). The later (AWS CLI) is commonly used in scripts and testing. In all these ways you need to make sure that the IAM User/IAM Role that is sending the requests has the correct IAM permission.
SMTP interface: The SMTP protocol can be used to send emails using SMTP protocol. You will need to specify the SMTP server endpoint, the SMTP port and SMTP user and password (create an IAM user that has permission to use Amazon SES and issue SMTP credentials).

Demo time!

For this demo, I will use AWS CDK to create the needed infrastructure. You can also run this demo with me, you just need to adapt the inputs (e.g. your domain name). The only thing that I expect from you is that your AWS region is CDK bootstrapped and the domain in Route53 is already configured (I will not configure this in this code because this domain is already being used in other personal demos). If you do not own a domain I highly recommend you buy a cheap one just for testing (you can get a cheap domain for $2/year or even less than that).

I will use Amazon SES in a region that it's still in Sandbox mode, i.e. it can only send emails to identities verified in Amazon SES. So because of that, I will create two identities, an email address identity (personal email) and a domain identity (using the domain in Route53).

CDK Code

This will be a simple code that will use a single stack to create two email identities, the first will be validated using the domain, while the other is a personal email that will be validated using email address validation. Let's check the main (entry point) of our CDK project.

// src/main.ts
#!/usr/bin/env ts-node
import "source-map-support/register";
import * as cdk from "aws-cdk-lib";
import { SesStack } from "./stack/ses";
import { devProps } from "./config";

const app = new cdk.App();

new SesStack(app, "SesStack", devProps);

Before checking the values of the stack let's first take a look at the stack definition.

// src/stack/ses.ts
import { Stack, StackProps, Tags } from "aws-cdk-lib";
import { Construct } from "constructs";
import {
  DkimIdentity,
  EasyDkimSigningKeyLength,
  EmailIdentity,
  Identity,
  MailFromBehaviorOnMxFailure,
} from "aws-cdk-lib/aws-ses";
import { HostedZone, TxtRecord } from "aws-cdk-lib/aws-route53";

export interface SesStackProps extends StackProps {
  domain: string;
  email: string;
}

export class SesStack extends Stack {
  constructor(scope: Construct, id: string, props: SesStackProps) {
    super(scope, id, props);

    this.setEmailIdentity(props.email);
    this.setDomainIdentity(props.domain);

    Tags.of(this).add("CreatedBy", "CDK");
  }

  private setEmailIdentity(email: string): void {
    new EmailIdentity(this, "EmailIdentity", {
 identity: Identity.email(email),
    });
  }

  private setDomainIdentity(domain: string): void {
    const hostedZone = HostedZone.fromLookup(this, "Zone", { domainName: domain });
    new EmailIdentity(this, "DomainIdentity", {
 identity: Identity.publicHostedZone(hostedZone),
 dkimIdentity: DkimIdentity.easyDkim(EasyDkimSigningKeyLength.RSA_2048_BIT),
 mailFromBehaviorOnMxFailure: MailFromBehaviorOnMxFailure.USE_DEFAULT_VALUE,
 dkimSigning: true,
    });
    new TxtRecord(this, "DmarkRecord", {
 recordName: "_dmarc",
 values: ["v=DMARC1;p=quarantine"],
 zone: hostedZone,
    });
    new TxtRecord(this, "SpfRecord", {
 recordName: "",
 values: ["v=spf1 include:amazonses.com -all"],
 zone: hostedZone,
    });
  }
}

And finally, let's check the values for this stack.

// src/config/dev.config.ts
import { SesStackProps } from "../stack/ses";

export const devProps: SesStackProps = {
 domain: "demosfelipetrindade.top",
 email: "yourpersonalemail@gmail.com",
 env: {
 account: "937168356724",
 region: "us-east-1",
  },
};

There you go! Deploying this stack is super easy and you only need to run cdk deploy with the appropriate IAM permissions to the desired account. Of course you will need to change these values to fit your use case.

A few important points to mention:

Notice that in the code we didn't have to configure the DKIM record. Why? Actually, because my domain is being managed by Route53 I've used the publicHostedZone method of CDK that already setup the DKIM in the hosted zone. If you used the domain method the Amazon SES would provide you with the DKIM records for you to add in your DNS. Indeed, after deployment you will see the DNS records there:

You will need to confirm the personal email address identity in your inbox. It will be similar to the following.

Let's test!

We can easily test by sending a test email using AWS CLI or SMTP. Why not check how it works using both approaches?

Sending emails using Amazon SES API

Let's use a simple bash script for sending these emails:

#!/bin/bash

# PARAMETERS
SENDER="noreply@demosfelipetrindade.top"
RECIPIENT="yourpersonalemail@gmail.com"
SUBJECT="Test Email from Amazon SES"
BODY_HTML="<html><body><h1>Test Email</h1><p>This is an <strong>Amazon SES</strong> test email!</p></body></html>"
AWS_REGION="us-east-1"

Amazon SES send-email \
 --region "$AWS_REGION" \
    --from "$SENDER" \
 --destination "ToAddresses=$RECIPIENT" \
    --message "Subject={Data=$SUBJECT,Charset=utf-8},Body={Html={Data=$BODY_HTML,Charset=utf-8}}"

if [ $? -eq 0 ]; then
    echo "Email sent successfully!"
else
    echo "Failed to send email."
fi

Notice that because I validated the entire domain I can send email using any email address within that domain, in this case, I'm using noreply. After executing this script the following email will show up in your email inbox.

We can check the details of this email to see extra information.

. Notice the mailed-by field! The value is amazonses.com. Is it possible to change this? Yes it is tottaly possible. This is called the EMAIL FROM field. By default if not specified AWS uses Amazon SES domain. This is out of the scope of this demo but it's good to know.

Sending emails using SMTP

Sending email using SMTP requires more work at first. I'll create using the AWS Console an IAM User with SMTP credentials. Ideally you should save this in Secrets Manager for your application to use (remember that at the moment it is not recommended to create SMTP credentials using ephemeral/temporary AWS credentials).

Now that we have the SMTP user and SMTP password let's use a simple Python code to send emails.

import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

# Parameters
smtp_server = "email-smtp.us-east-1.amazonaws.com"
smtp_port = 587
smtp_username = "YOUR_SMTP_USERNAME"
smtp_password = "YOUT_SMTP_PASSWORD"
sender = "noreply@demosfelipetrindade.top"
recipient = "yourpersonalemail@gmail.com"
subject = "Test Email from SMTP"
message = """
Test SMTP Email
This is an Amazon SES SMTP test email!
"""
body_html = f"<html><body><h1>Test SMTP Email</h1><p>{message}</p></body></html>"

msg = MIMEMultipart("alternative")
msg["Subject"] = subject
msg["From"] = sender
msg["To"] = recipient

part1 = MIMEText(message, "plain")
part2 = MIMEText(body_html, "html")

msg.attach(part1)
msg.attach(part2)

try:
    with smtplib.SMTP(smtp_server, smtp_port) as server:
 server.starttls()
 server.login(smtp_username, smtp_password)
 server.sendmail(sender, recipient, msg.as_string())
    print("Email sent successfully!")
except Exception as e:
    print(f"Failed to send email: {e}")

After code execution, you should see the following in your inbox:

Cya!

I hope you've like it and that this demo and explanation was helpful!

See you around! 👋

RSS as a tool to stay up-to-date with tools, news, and posts

Mon, 21 Oct 2024 12:05:32 GMT

We all know that the technology world evolves FAST. The amount of new tools created weekly, updates, bug fixes, new features, tech blog posts... It's impossible to manually keep track of all these things without an automatic way. That's where we are going to use RSS in our favor! Maybe the word "RSS" is not familiar to you but I bet you have already seen its icon.

RSS

RSS (it can mean several things, such as Really Simple Syndication or Rich Site Summary) is one of the elders of the internet, created in 1999. We are talking about a time when internet speed was around 127 kilobits per second. Downloading a 3MB audio took three minutes back then! You can imagine how slow it was to surf the web back then...

Today our internet speed is way faster (at least 1000x) so doing "ordinary" things, such as navigating the internet, uploading attachments in emails, watching videos, and other activities are way faster than it was more than two decades ago.

With that being said, can you imagine the pain of opening a website only to check if a new post/news was published in the feed? The idea is that the website could publish posts/news in a different type of feed called RSS. This special feed could be interpreted by a feed reader that would allow the user to know when something new was published and the data transfer would also be significantly smaller. Today RSS is more used to save time, and as you know time is valuable. Notice that RSS is not magical, it relies on the website to implement the RSS feed so it can be consumed by the feed reader (also known as feed aggregator). Today, some fancy feed readers can automatically scrape a website and turn it into an "RSS feed" (but it might be a payed feature in our feed reader).

Something very common in tech blogs is to, instead of implementing an RSS feed, create a newsletter, where you get the new blog posts via e-mail. Ideally, though, both should be presented!

The RSS feed is just an XML formatted plain text, as you can see in my website's RSS. As you can see, this is pretty difficult for us to read but very easy for a machine to understand. So every time this XML changes with a new section, the feed reader knows that the website has a new post.

Feed readers

There are several feed readers out there but in general, they can be divided into two categories: desktop and web apps. Desktop readers, such as RSS Guard or Fluent Reader are "off-line" readers, which means you install them on your computer and needs your computer to access the feed reader. A more handy solution (because you simply need to access it using any web browser) is web feed readers such as Feedly or Inoreader. Choose the option that fits your needs the most. One important thing to mention about web feed readers is that they are usually paid (but offer an always free version with limited features).

This is not an ad or recommendation: I use Feedly and for now, it works fine for my use case. It does not support some features in the free plan but I tolerate. But you should really use any tool that fits your needs.

RSS feeds suggestion

Currently, I divide my RSS feed into two categories:

Tech-blogs: Mostly tech companies blogs presenting solutions, problems, and tools. Some of the tech blogs I follow are AWS Architecture Blog, Engineering at Meta, LinkedIn Blog, Slack Engineering, Spotify Engineering. I also recommend personal tech blogs, such as SamWho and mine (of course!).
Tools releases: This helps me stay tuned about releases of the main tools I use. Some of the tools I follow are Opentofu, Terragrunt, CDK.

Some important things to mention:

Today is very common for tech people to publish things in blogs like Medium and Dev.To. I still think RSS is relevant and does not substitute them, use both in your favor!
The "tools release" DOES NOT replace Dependabot at all. You should still use it in your projects to update the versions of your dependencies.
It's common for tools to offer release notes (e.g. Fastapi) and upgrading notes (e.g. ArgoCD Upgrading guide). You should still read them, and treat the RSS as just a "preview".
Some GitHub repositories have INTENSE release cycles, with beta, dev, and test versions being deployed frequently, this will spam your RSS feed a bit.

Cya!

This was a not-so-common blog post from my side. Less technical but still important to comment on. This might be extremely helpful if you struggle to stay up-to-date with tools and tech articles.

See you around! 👋

Multi-account deployment using AWS CDK

Mon, 30 Sep 2024 16:40:30 GMT

AWS CDK is a great tool to deploy infrastructure in AWS using your favorite programming language but deploying to multiple accounts using AWS CodePipeline can be tricky. In this blog post, we will cover the idea behind it and we are going to implement this solution using AWS CDK and AWS CodePipeline. All the code used in this blog post is available in my GitHub repository.

Multi Account Architecture

The goal of this blog post is to use AWS best practices and separate AWS accounts based on their functionality and deploy infrastructure from a single AWS account to several AWS accounts. To summarize, the below architecture will be created in this blog post:

Let's comment about it:

GitHub Actions: GitHub Actions will be used only for CI purposes: check the linting and run tests of our AWS CDK code.
AWS Accounts: We will use two AWS accounts for workloads (development and production) and one account that will contain the CD pipeline to deploy to the workload accounts and resources shared by every account (shared assets). The workload accounts usually contain services like RDS, EC2, EKS, Lambda, S3, and so on, while the shared assets accounts will contain the Pipelines for deploying to the workload account and common resources to the workload accounts (e.g. images in ECR to allow artifact promotion, common scripts in S3 buckets...).
CodePipeline: It will deploy services in the desired AWS account based on the AWS CDK code created.
AWS Connector for GitHub: A CodeStar connection will be created between GitHub repository and AWS, this will install a GitHub App that will trigger a CodePipeline execution based on commits pushed to the GitHub repository.
CloudFormation: AWS CDK is an abstraction over CloudFormation, in the end, all the AWS CDK code will be transformed (synth) into CloudFormation stacks.

Notice that the ECR and S3 created are just for exemplifying in the code how to create resources in these accounts.

AWS CDK

This is not an AWS CDK tutorial but I need to comment some important things before diving into the code. CDK, as mentioned, relies on the CloudFormation service, so the more you know about CloudFormation the better will be to understand how it works, although it's not necessary.

For CDK to work you need some resources in your account (such as IAM Roles that CloudFormation will use to deploy/lookup resources and publish files). The step of creating these resources before start to work on the CDK code is called bootstrap. This is a manual step that will provision a CloudFormation Stack (called CDKToolkit) in the region you are bootstrapping.

You can use any programming language supported by AWS CDK my recommendation is Typescript, which is the default programming language used to write AWS CDK code, and this is the programming language that I will use in this demo.

In our code we will create 4 stacks:

Pipeline stack: This will be responsible for creating the CodePipeline resources to cross-account deploy to all environments (shared-assets and workload). This will be deployed in the shared-assets account.
Shared Assets stack: Resources deployed in the shared assets account.
Workload (Development) stack: Resources deployed in the development account.
Workload (Production) stack: Resources deployed in the production account.

Notice that this is just one possible way of deploying things, in reality, you could have divided this in several different ways, multiple stacks per account, and so on. The idea here is to present only the solution and the idea that makes this deployment possible.

Most of the time when creating CDK resources we will use L2 constructs that can be found in the API Reference documentation. What makes AWS CDK deployment easy is the existence of the pipelines module (do not confuse with the CodePipeline module) that is an abstraction over CI/CD AWS Services to create a deployment pipeline for your own AWS CDK (notice, this means that AWS CDK will be responsible for creating its own CD deployment pipeline!). We will see it in action.

Hands-on!

Let's discuss all tools used in this project (the latter will be CDK and the code of this demo).

Devbox

I always use Devbox in my projects. I HATE when I on-board a project that it's not clear the dependencies (and their version) that I need to have installed to contribute to the project and make it work locally. That's why I love Devbox: an abstraction of Nix Packages that installs locally (in a folder called .devbox) all the dependencies of the project in an explicit way. I've used the following devbox file (devbox.json):

{
  "$schema": "https://raw.githubusercontent.com/jetify-com/devbox/0.12.0/.schema/devbox.schema.json",
  "packages": [
    "nodejs@22.8.0",
    "awscli2@2.17.42",
    "yarn@1.22.22",
    "go-task@3.38.0"
  ],
  "shell": {
    "init_hook": [
      "echo 'Please use Taskfile scripts.'"
    ]
  }
}

As you can see, I'm using NodeJS, AWS CLI, Yarn, and Task. The first three dependencies are obvious why we are using, but you might wondering about Task...

Task

Task is a modern approach to Makefiles written in Go. It supports a declarative YAML manifest (called Taskfile) which you specify the scripts and commands that you would like to use. I'm using Task here to make it easier the CDK bootstrap process. Indeed, this will only be done once, but I highly recommend this to be documented or even scripted to make it repeatable in case it's needed in the future. What we want here is to bootstrap all the three AWS accounts (shared assets, development, and production) but there is one thing: we need to allow the shared assets account to assume a role with Administrator access to be assumed in the workload accounts. To simplify all of this, the following Taskfile was created:

version: '3'

anchors:
  check_aws_credentials: &check_aws_credentials
     sh: 'aws sts get-caller-identity'
     msg: "AWS credentials are not valid or are expired!"

env:
  AWS_REGION: us-east-2 # Modify this!
  AWS_SHARED_ACCOUNT: "730335516527" # Modify this!

tasks:
  init:
    desc: Install all code dependencies using YARN
    silent: true
    cmds:
      - yarn
  bootstrap-shared-assets:
    desc: Bootstrap the shared account (contains shared assets and CI/CD pipeline). Make sure AWS credentials are set.
    silent: true
    preconditions:
      - *check_aws_credentials
    cmds:
      - echo "Bootstrapping the {{.AWS_SHARED_ACCOUNT}} account in the {{.AWS_REGION}} region..."
      - yarn cdk bootstrap aws://{{.AWS_SHARED_ACCOUNT}}/{{.AWS_REGION}}
  bootstrap-workload:
    desc: Bootstrap an workload account to allow deployments from the shared/ci-cd AWS account.  Make sure AWS credentials are set.
    silent: true
    preconditions:
      - *check_aws_credentials
    vars:
      AWS_ACCOUNT:
        sh: aws sts get-caller-identity --query Account --output text
    cmds:
      - echo "Bootstrapping the {{.AWS_ACCOUNT}} account in the {{.AWS_REGION}} region..."
      - |
 yarn cdk bootstrap aws://{{.AWS_ACCOUNT}}/{{.AWS_REGION}} \
 --trust {{.AWS_SHARED_ACCOUNT}} --trust-for-lookup {{.AWS_SHARED_ACCOUNT}} \
 --cloudformation-execution-policies "arn:aws:iam::aws:policy/AdministratorAccess"

When running task --list we have the following output (that summarizes well the tasks we have defined in the Taskfile).

So, with the AWS Accounts in place, we need to perform:

Export the credentials of the Shared Assets account and run task bootstrap-shared-assets.
Export the credentials of the Development account and run task bootstrap-workload.
Export the credentials of the Production account and run task bootstrap-workload.

I encourage you to check the Taskfile code to notice that the cdk bootstrap command used in the workload account have extra parameters that allow the cross-account access!

AWS CodeStar connection

Follow the AWS Documentation. This must be performed in the CI-CD/Shared account. You must be the owner of the GitHub repository/organization.

This is a step that must be performed via the console, even the CLI option provided by the documentation mentions that you still need to go to the console to activate the connection, so I think it's easier to simply create everything manually.

CDK

I'm going to divide the source code into three parts:

Pipeline: Pipeline configuration and stack to create the CodePipeline that will deploy CDK code.
Stacks: Creates resources
Configuration: Values used to configure the stacks.

Let's check our code.

// main.ts
import "source-map-support/register";
import * as cdk from "aws-cdk-lib";
import { CodePipelineStack } from "./pipeline/codepipeline";
import { pipelineProps } from "./config/pipeline";

const app = new cdk.App();

new CodePipelineStack(app, "MultiAccountStack", pipelineProps);

The entry point of our project defines a stack called MultiAccountStack. This stack is the codepipeline stack that will deploy CDK and will also deploy the other stack in the corresponding accounts.

// pipeline/codepipeline.ts
import { Stack, StackProps, Stage, StageProps, Tags } from "aws-cdk-lib";
import { CodePipeline, CodePipelineSource, ManualApprovalStep, ShellStep } from "aws-cdk-lib/pipelines";
import { Construct } from "constructs";
import { SharedAssetsProps, SharedAssetsStack } from "../stack/shared-assets";
import { WorkflowStack, WorkloadProps } from "../stack/workload";
import { developmentProps } from "../config/development";
import { productionProps } from "../config/production";
import { sharedAssetsProps } from "../config/shared-assets";

export interface PipelineProps extends StackProps {
  gitBranch: string;
  githubOwner: string;
  githubRepository: string;
  codeStarConnectionArn: string;
}

export class SharedAssetsPipelineStage extends Stage {
  constructor(scope: Construct, id: string, sharedAssetsProps: SharedAssetsProps, props?: StageProps) {
    super(scope, id, props);

    new SharedAssetsStack(this, "StageStack", sharedAssetsProps);
  }
}

export class WorkloadPipelineStage extends Stage {
  constructor(scope: Construct, id: string, workloadProps: WorkloadProps, props?: StageProps) {
    super(scope, id, props);

    new WorkflowStack(this, "WorkflowStack", workloadProps);
  }
}

export class CodePipelineStack extends Stack {
  constructor(scope: Construct, id: string, props: PipelineProps) {
    super(scope, id, props);

    const branch = props.gitBranch;
    const githubRepository = `${props.githubOwner}/${props.githubRepository}`;
    const githubConnection = CodePipelineSource.connection(githubRepository, branch, {
 connectionArn: props.codeStarConnectionArn,
    });

    const pipeline = new CodePipeline(this, "PipelineStack", {
 pipelineName: "MultiAccountPipeline",
 synth: new ShellStep("Synth", {
 input: githubConnection,
 commands: ["yarn", "yarn run cdk synth"],
 primaryOutputDirectory: "cdk.out",
      }),
 selfMutation: true,
 crossAccountKeys: true,
    });

    pipeline.addStage(new SharedAssetsPipelineStage(this, "SharedAssetsStage", sharedAssetsProps));
    pipeline.addStage(new WorkloadPipelineStage(this, "DevelopmentStage", developmentProps));
    pipeline.addStage(new WorkloadPipelineStage(this, "ProductionStage", productionProps), {
 pre: [new ManualApprovalStep("Release to production")],
    });

    Tags.of(this).add("CreatedBy", "CDK");
  }
}

In the end, this code defines a class (CodePipelineStack) that extends a stack and defines a pipeline with stages to deploy to shared assets account (using shared-assets stack), development account, and production account (using workload stack).

// stack/shared-assets.ts
import { RemovalPolicy, Stack, StackProps, Tags } from "aws-cdk-lib";
import { Repository, TagMutability } from "aws-cdk-lib/aws-ecr";
import { Construct } from "constructs";
import { AccountPrincipal, CompositePrincipal } from "aws-cdk-lib/aws-iam";
import { AwsAccount } from "../config/types";

export interface SharedAssetsProps extends StackProps {
  ecrRepository: string[];
}

export class SharedAssetsStack extends Stack {
  constructor(scope: Construct, id: string, props: SharedAssetsProps) {
    super(scope, id, props);

    const allAwsAccounts = Object.values(AwsAccount).map((accountId) => new AccountPrincipal(accountId));

    for (const repositoryName of props.ecrRepository) {
      const repository = new Repository(this, repositoryName, {
 repositoryName: repositoryName,
 removalPolicy: RemovalPolicy.DESTROY,
 imageTagMutability: TagMutability.IMMUTABLE,
 imageScanOnPush: true,
      });
      const principals = new CompositePrincipal(...allAwsAccounts);
      repository.grantPullPush(principals);
    }

    Tags.of(this).add("CreatedBy", "CDK");
  }
}

This is a "dummy" stack that creates ECR repositories.

// stack/workload.ts
import { RemovalPolicy, Stack, StackProps, Tags } from "aws-cdk-lib";
import { Construct } from "constructs";
import { BlockPublicAccess, Bucket } from "aws-cdk-lib/aws-s3";
import { Environment } from "../config/types";

export interface WorkloadProps extends StackProps {
  environment: Environment;
}

export class WorkflowStack extends Stack {
  constructor(scope: Construct, id: string, props: WorkloadProps) {
    super(scope, id, props);

    new Bucket(this, `${props.environment}-bucket-ingestion`, {
 enforceSSL: true,
 blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
 removalPolicy: RemovalPolicy.DESTROY,
    });

    Tags.of(this).add("CreatedBy", "CDK");
  }
}

This is a "dummy" stack that creates an S3 bucket.

It's important to understand that in the pipeline/codepipeline.ts file, the configs are being passed to the stack, controlling where (environment) the stack will be deployed. Let's check, for example, the development config values.

// config/development.ts
import { WorkloadProps } from "../stack/workload";
import { AwsAccount } from "./types";

export const developmentProps: WorkloadProps = {
 environment: "development",
 env: {
 account: AwsAccount.Development,
 region: "us-east-2",
  },
};

Basically, it add values to the props of the stack. The same will happen to ALL the other environments: we are defining our infrastructure via code.

Commenting and pasting all the code here doesn't make sense, since at this point the idea of the code, how it works, and the CDK deployment in multi-account was already explained. I highly encourage you to check the source code.

The chicken and the egg problem

Notice one thing, we are defining the stack that creates the Codepipeline to create the code. But it won't deploy the CDK code if the stack does not exist. We are facing the chicken and egg problem. To solve this, we should deploy (only this time) manually the codepipeline stack.

After this is done, all the pushes of the code in the main branch will trigger Codepipeline execution and the CDK code will be deployed. We can easily do that by exporting the credentials to the shared-assets account and running yarn cdk deploy MultiAccountStack.

Cya

Hope you enjoyed this blog post! See you in the next post! 👋

Using IaC to configure private resources

Sun, 18 Aug 2024 13:44:32 GMT

The demo GitHub repository with all the code for this project can be accessed here.

The goal of this blog post is to present two ways of configuring private resources. First, let's define what I mean by private resources. Private resources are resources that are not accessible directly via the internet, i.e. are resources that are contained in a private network. When I say "resources" I'm talking about configurations for applications/services (that are typically a Terraform resource). A good example is a DB/User in a database (which is typically deployed in a private network and can't be accessed directly), but it could have been a configuration for Harbor, Keycloak or any other service that can be configured. For this demo we want to create a DB in a database service.

Notice that I'm not talking about resources that are deployed privately but can be configured because the API is public. For example, the creation of these services (e.g. Amazon RDS) is in general not a problem because they are being created in a cloud provider (e.g. AWS) and the API for creating these resources is public (but requires authentication). Configuring all "cloud-related" topics for these services is not a problem (e.g. replication, backup, instance size...) but the internal configuration of this service (e.g. database creation, db user creation) requires you to connect to these services and do the configuration yourself.

It is important to have these services in a private network because it's a security risk to expose them publicly so it's not worth the risk of having it exposed to easy access versus the risk that you will face in case of an attack.

In this blog post, I will cover two common approaches to this problem:

SSH Proxy: Using a bastion host as to SSH-proxy the connection to the private resource
Self Host: Instead of using the bastion as a proxy, the entire CI/CD to configure these resources can be deployed inside the private network

For this project, I'll be using Vultr Cloud and Opentofu (the open-source version of Terraform). I wouldn't skip this in case you are looking for an AWS approach, what matters here are the concepts, that you can easily replicate to AWS scenario.

SSH-proxy

This is a well-known solution when users need to connect to private resources. The idea is to deploy a virtual machine publicly so only the developers have access and connect to this instance using SSH. Once the connection is established, the developers can access all the resources deployed inside the internal network (of course, if the firewall allows it). Ideally, you should also restrict the IPs that can connect to the bastion, but one bad thing is that usually, internet service providers (ISP) give you a dynamic IP. You can have a plan that gives you a static IP but it's usually way more expensive.

This is usually a cheap approach because you typically deploy a virtual machine that is small since this machine won't run workloads of application and will serve only as a network proxy for your internal network.

Let's check how can we create a private network with a bastion host and database.

# variables.tf
variable "public_ssh_key" {
  description = "Public SSH key to attach to the authorized_keys of the bastion host"
  type        = string
}

A single variable will be used here: the public SSH key to attach to the authorized_keys file of the bastion host.

# main.tf
locals {
  region = "mia" # Miami
  label  = "ssh-proxy"
}

############################
###### Networking
############################
resource "vultr_vpc" "this" {
  region         = local.region
  v4_subnet      = "10.0.0.0"
  v4_subnet_mask = 24
}

resource "vultr_firewall_group" "bastion" {
  description = "Bastion Firewall"
}

resource "vultr_firewall_rule" "bastion" {
  firewall_group_id = vultr_firewall_group.bastion.id
  protocol          = "tcp"
  ip_type           = "v4"
  subnet            = "0.0.0.0"
  subnet_size       = 0
  port              = "22"
  notes             = "Allow all SSH from anywhere"
}
############################
###### Virual Machine
############################
resource "vultr_ssh_key" "bastion" {
  name    = "bastion"
  ssh_key = var.public_ssh_key
}

resource "vultr_instance" "bastion" {
  region              = local.region
  vpc_ids             = [vultr_vpc.this.id]
  plan                = "vc2-1c-1gb"
  os_id               = data.vultr_os.ubuntu.id
  backups             = "disabled"
  label               = local.label
  firewall_group_id   = vultr_firewall_group.bastion.id
  ssh_key_ids         = [vultr_ssh_key.bastion.id]
  enable_ipv6         = true
  disable_public_ipv4 = false
  ddos_protection     = false
  activation_email    = false
}

############################
###### Database
############################
resource "vultr_database" "this" {
  region                  = local.region
  vpc_id                  = vultr_vpc.this.id
  plan                    = "vultr-dbaas-hobbyist-cc-1-25-1"
  database_engine         = "pg"
  database_engine_version = "15"
  label                   = local.label
  trusted_ips = [
    "${vultr_instance.bastion.internal_ip}/32",
  ]
}

resource "postgresql_database" "this" {
  name = "demodb"
}

# versions.tf
terraform {
  required_providers {
    vultr = {
      source  = "vultr/vultr"
      version = "2.21.0"
    }
    postgresql = {
      source  = "cyrilgdn/postgresql"
      version = "1.22.0"
    }
  }
}

provider "vultr" {}

module "db_tunnel" {
  source  = "flaupretre/tunnel/ssh"
  version = "2.2.1"

  target_host = vultr_database.this.host
  target_port = vultr_database.this.port

  gateway_host = vultr_instance.bastion.main_ip
  gateway_user = "root"
}

provider "postgresql" {
  host     = module.db_tunnel.host
  port     = module.db_tunnel.port
  username = vultr_database.this.user
  password = vultr_database.this.password
  database = vultr_database.this.dbname
}

# data.tf
data "vultr_os" "ubuntu" {
  filter {
    name   = "name"
    values = ["Ubuntu 22.04 LTS x64"]
  }
}

Notice that for simplicity, I'm allowing the bastion instance firewall to receive traffic from anywhere but this is not a best practice in terms of security, as commented before.

Terraform does not have a built-in feature to SSH-proxy, that's why you need to implement it. There is a really good module called tunnel that I've used to create this SSH-proxy tunnel and reference the database host as the "localhost" of the runner.

Self-Hosted

The self-hosted approach is a very interesting approach for configuring private resources. You are fully responsible for the security, availability, operating system, and software tools presented in the runner. For the demo, I will use GitHub Action self-hosted runner.

There are two important considerations:

This might be an expensive solution for smaller teams in terms of maintenance and servers. Developers are expensive, maybe it is not suitable for your team to "spend" developer time to configure, maintain and deploy self-hosted runners. Also, it's unlikely that there will be jobs running late at night, so paying for an infrastructure (remember that a server to run a CI/CD will usually require more CPU/Memory to run tests, build, and stress-test) that is not in use can be expensive for a small company. Of course, you can create automation to turn on and off the servers but, again, more developer work.
Do not use self-hosted runners if your repository is public. This is because if the repository is public anyone can fork your repository and create a pull request that executes code in a workflow, which can be treated as a security risk.

The code for provisioning this infrastructure is really similar to the one we've used in the SSH-proxy.

# variables.tf
variable "repo_name" {
  description = "Github repository name including the owner (e.g. felipelaptrin/my-repo)"
  type        = string
}

variable "public_ssh_key" {
  description = "Public SSH key to attach to the authorized_keys of the self-hosted runner host"
  type        = string
}

variable "github_actions_version" {
  description = "Version that the self-hosted runner will use. You can check all version here: https://github.com/actions/runner/releases"
  type        = string
  default     = "v2.319.1"
}

Notice that in addition to the SSH key to connect to the self-hosted runner host (in this case only to debug if needed), there are two more variables: the GitHub runner version and the repository name that the self runner will be configured (needed to retrieve the runner token to connect to GitHub and register the runner).

# main.tf
locals {
  region                 = "mia" # Miami
  label                  = "self-hosted"
  github_actions_version = split("v", var.github_actions_version)[1]
  repo_name              = split("/", var.repo_name)[1]
}

############################
###### Networking
############################
resource "vultr_vpc" "this" {
  region         = local.region
  v4_subnet      = "10.0.0.0"
  v4_subnet_mask = 24
}

resource "vultr_firewall_group" "runner" {
  description = "Github self-runner Firewall"
}

resource "vultr_firewall_rule" "runner" {
  firewall_group_id = vultr_firewall_group.runner.id
  protocol          = "tcp"
  ip_type           = "v4"
  subnet            = "0.0.0.0"
  subnet_size       = 0
  port              = "22"
  notes             = "Allow all SSH from anywhere"
}

############################
###### Virual Machine
############################
resource "vultr_ssh_key" "runner" {
  name    = "runner"
  ssh_key = var.public_ssh_key
}

resource "vultr_instance" "runner" {
  region              = local.region
  vpc_ids             = [vultr_vpc.this.id]
  plan                = "vc2-1c-1gb"
  os_id               = data.vultr_os.ubuntu.id
  backups             = "disabled"
  label               = local.label
  firewall_group_id   = vultr_firewall_group.runner.id
  ssh_key_ids         = [vultr_ssh_key.runner.id]
  enable_ipv6         = true
  disable_public_ipv4 = false
  ddos_protection     = false
  activation_email    = false
  user_data           = <<EOF
#!/bin/bash
echo "ubuntu ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
cd /home/ubuntu && mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux.tar.gz -L https://github.com/actions/runner/releases/download/${var.github_actions_version}/actions-runner-linux-x64-${local.github_actions_version}.tar.gz
tar xzf ./actions-runner-linux.tar.gz
sudo chown -R ubuntu /home/ubuntu/actions-runner
sudo -u ubuntu ./config.sh --name vultr --replace --url https://github.com/${var.repo_name} --token ${sensitive(data.github_actions_registration_token.this.token)}
sudo -u ubuntu ./run.sh
  EOF

  lifecycle {
    ignore_changes = [
      user_data
     ]
  }
}

############################
###### Database
############################
resource "vultr_database" "this" {
  region                  = local.region
  vpc_id                  = vultr_vpc.this.id
  plan                    = "vultr-dbaas-hobbyist-cc-1-25-1"
  database_engine         = "pg"
  database_engine_version = "15"
  label                   = local.label
  trusted_ips = [
    "${vultr_instance.runner.internal_ip}/32",
  ]
}

resource "postgresql_database" "this" {
  name = "demodb"
}

# versions.tf
terraform {
  required_providers {
    vultr = {
      source  = "vultr/vultr"
      version = "2.21.0"
    }
    github = {
      source  = "integrations/github"
      version = "6.2.3"
    }
    postgresql = {
      source  = "cyrilgdn/postgresql"
      version = "1.22.0"
    }
  }
}

provider "vultr" {}

provider "github" {}

provider "postgresql" {
  host     = vultr_database.this.host
  port     = vultr_database.this.port
  username = vultr_database.this.user
  password = vultr_database.this.password
  database = vultr_database.this.dbname
}

# data.tf
data "vultr_os" "ubuntu" {
  filter {
    name   = "name"
    values = ["Ubuntu 22.04 LTS x64"]
  }
}

data "github_actions_registration_token" "this" {
  repository = local.repo_name
}

Notice that now we are directly connecting to the database: not proxy, just direct connection.

CI/CD using GitHub Actions

We don't want to apply things manually, let's use a pipeline workflow to deploy our infrastructure using GitHub Actions! The idea is simple:

During pull request: Plan the actions that will be performed and commit the dependency lock file.
When pull request is merged: Deploy the infrastructure.

Two jobs were created (one for ssh-proxy and the other one for self-hosted) and to not repeat code two composite actions (plan and deploy) were also created. Let's check the actions first

# .github/actions/plan/action.yaml
name: plan
description: Plan IaC

inputs:
  folder:
    description: Name of the folder that terraform will run
    required: true

runs:
  using: "composite"
  steps:
    - name: Install devbox
      uses: jetify-com/devbox-install-action@v0.11.0

    - name: Configure Git
      shell: bash
      run: |
        git config --global user.name "github-actions[bot]"
        git config --global user.email "github-actions[bot]@users.noreply.github.com"

    - uses: webfactory/ssh-agent@v0.9.0
      with:
        ssh-private-key: ${{ env.SSH_PRIVATE_KEY }}

    - name: Plan
      shell: bash
      env:
        FOLDER: ${{ inputs.folder }}
        VULTR_API_KEY: ${{ env.VULTR_API_KEY }}
        AWS_ACCESS_KEY: ${{ env.AWS_ACCESS_KEY }}
        AWS_SECRET_ACCESS_KEY: ${{ env.AWS_SECRET_ACCESS_KEY }}
        GITHUB_TOKEN: ${{ env.PAT_GITHUB }}
      run: |
        devbox run tofu -chdir=$FOLDER init
        devbox run tofu -chdir=$FOLDER plan -var-file=variables.tfvars

    - name: Install GH CLI
      uses: dev-hanz-ops/install-gh-cli-action@v0.1.0
      with:
        gh-cli-version: 2.54.0

    - name: Commit .terraform.lock.hcl files
      working-directory: ${{ inputs.folder }}
      if: github.ref != 'refs/heads/main'
      env:
        GITHUB_TOKEN: ${{ env.GITHUB_TOKEN }}
      shell: bash
      run: |
        if [ -n "$(git status --porcelain)" ]; then
          gh pr checkout ${{ github.event.pull_request.number }}
          git add .
          git commit -m "Add .terraform.lock.hcl files"
          git push
        else
          echo "Nothing to commit."
        fi

# .github/actions/apply/action.yaml
name: apply
description: Apply IaC

inputs:
  folder:
    description: Name of the folder that opentofu will run
    required: true

runs:
  using: "composite"
  steps:
    - name: Install devbox
      uses: jetify-com/devbox-install-action@v0.11.0

    - uses: webfactory/ssh-agent@v0.9.0
      with:
        ssh-private-key: ${{ env.SSH_PRIVATE_KEY }}

    - name: Apply Opentofu
      shell: bash
      env:
        FOLDER: ${{ inputs.folder }}
        VULTR_API_KEY: ${{ env.VULTR_API_KEY }}
        AWS_ACCESS_KEY: ${{ env.AWS_ACCESS_KEY }}
        AWS_SECRET_ACCESS_KEY: ${{ env.AWS_SECRET_ACCESS_KEY }}
        GITHUB_TOKEN: ${{ env.GITHUB_TOKEN }}
      run: |
        devbox run tofu -chdir=$FOLDER init
        devbox run tofu -chdir=$FOLDER apply -var-file=variables.tfvars -auto-approve

Notice that several env vars are required to fully plan/run Opentofu:

VULTR_API_KEY: Required to authenticate with Vultr Cloud. This API Key is safely stored in the GitHub Secrets of the repository.
AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY: I didn't paste the backend.tf code because it's not relevant to the post, but I've configured a Vultr Object Storage (similar to S3) to store the state of Opentofu and these environment variables are required. These keys are safely stored in the GitHub Secrets of the repository.
GITHUB_TOKEN: Notice that in the plan action, I'm using two different values for this environment value. The first one is automatically created by GitHub and it's being used to use GitHub Bot to commit the dependency lock files of Opentofu. The second value is my GitHub PAT token (Personal Access Token) that is being used to get the token to register the self-hosted runner (check the data.tf file of the self-hosted manifests). This PAT token is safely stored in the GitHub Secrets of the repository.
SSH_PRIVATE_KEY: This is the private SSH key that can be used to connect to the bastion host and self-hosted runer. The latter is necessary only in case of debugging since all the self-runner configuration is being done in the user_data script.

Now, let's check how the workflow looks like

# .github/workflows/main.yaml
name: "Opentofu Actions"
on:
  pull_request:
  push:
    branches:
      - main

env:
  AWS_ACCESS_KEY: "${{ secrets.AWS_ACCESS_KEY }}"
  AWS_SECRET_ACCESS_KEY: "${{ secrets.AWS_SECRET_ACCESS_KEY }}"
  VULTR_API_KEY: "${{ secrets.VULTR_API_KEY }}"
  SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}

permissions:
  contents: write

jobs:
  ssh-proxy:
    runs-on: ubuntu-latest
    steps:
      - name: "Checkout"
        uses: actions/checkout@v4

      - name: Plan ssh-proxy
        if: github.ref != 'refs/heads/main'
        uses: ./.github/actions/plan
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PAT_GITHUB: ${{ secrets.PAT_GITHUB }}
        with:
          folder: ssh-proxy

      - name: Apply ssh-proxy
        if: github.ref == 'refs/heads/main'
        uses: ./.github/actions/apply
        env:
          GITHUB_TOKEN: ${{ secrets.PAT_GITHUB }}
        with:
          folder: ssh-proxy

  self-hosted:
    runs-on: self-hosted
    steps:
      - name: "Checkout"
        uses: actions/checkout@v4

      - name: Plan self-hosted
        if: github.ref != 'refs/heads/main'
        uses: ./.github/actions/plan
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PAT_GITHUB: ${{ secrets.PAT_GITHUB }}
        with:
          folder: self-hosted

      - name: Apply self-hosted
        if: github.ref == 'refs/heads/main'
        uses: ./.github/actions/apply
        env:
          GITHUB_TOKEN: ${{ secrets.PAT_GITHUB }}
        with:
          folder: self-hosted

Great!

Wait, we are not over! Did you notice the chicken and the egg problem when using self-hosted? Well, we want to run a CI/CD pipeline to deploy our infrastructure. This infrastructure declares the runner that the CI/CD will run (self-hosted). When you commit this, GitHub Actions will try to run this on the self-hosted runners (that does not exist) and it's trying to create itself. Not good.

How can we solve that? This is what we call a bootstrap process. We can do this in a two-step process:

Comment the resource postgresql_database.this (data.tf file) and modify the runs-on input of the self-hosted job to use ubuntu-latest.
Apply the infrastructure (via GitHub Actions managed runners). After done, uncomment the resource postgresql_database.this and add back self-hosted input.

That's one way of doing this. You can also use scrips or have a completely separate workflow only for setting the self-hosted runners to solve this problem.

Cya

Hope you enjoyed this blog post! See you in the next post! 👋

Deploying public and private applications in Kubernetes

Mon, 12 Aug 2024 10:40:32 GMT

The goal of this blog post is to deploy a Kubernetes cluster with a public and a private application. This is a common requirement in several companies because usually their main service is exposed publicly but internal tools (e.g. Grafana, Airflow) shouldn't be exposed publicly, even if they require authentication, so are deployed internally/privately.

Several tools will be used during the process. Cert Manager, Kubernetes Replicator, External DNS and NGINX Ingress will help us achieve this goal. This will be explained in detail during the blog post, so don't freak out!

For this demo, I will use Vultr to provision the Kubernetes cluster. If you are using a different cloud provider (such as AWS) I would still read this post, since everything here is valid and I will comment exactly where things need to change if you are using a different cloud provider.

All the manifests used here are in the GitHub repository.

Ingress and NGINX Ingress Controller

Instead of exposing every Kubernetes application using a LoadBalancer service type, which would require the deployment of a single LoadBalancer per application in your cloud provider, a good practice is to reuse the LoadBalancer to expose multiple services and route these internally using an Ingress.

An Ingress only works with an Ingress Controller, which is not deployed in Kubernetes by default. There are several controllers available out there. One of the most popular options available is the NGINX Ingress Controller, which is cloud-agnostic and open-source.

The NGINX Ingress Controller will be responsible for requesting a Load Balancer (layer 4) to your cloud provider and will manage the routing to your services. So imagine the following scenario (for simplicity consider all apps are public):

Application A: Configured in the ingress manifest to be accessible in the app-a.mydomain.com.
Application B: Configured in the ingress manifest to be accessible in the app-b.mydomain.com.

If you resolve these names (e.g. using dig or nslookup) you will see that they resolve to the same IP, because it's indeed a single Load Balancer exposed and all the routing is happening inside Kubernetes by the Ingress controller (in this case NGINX Ingress). This will be demoed, don't worry!

It's also in the ingress manifest that we define if the application will be exposed securely (using a TLS certificate). If HTTPS is configured we need to specify the name of the Kubernetes secret that contains the certificates in the ingress manifest.

Let's Encrypt and Cert Manager

Now that we know that we can use NGINX Ingress Controller to expose our applications using a single Load Balancer and we can specify in the ingress manifest the certificates of the TLS the question now is how the TLS certificate gets generated.

You can self-sign your certificate but who is going to trust you? By default browsers and operational systems come with a list of CA (Certificate Authorities) that are trustable to issue certificates (e.g. Firefox included certificates). If you self-sign this certificate you will need to add yourself as a valid CA in the computers of the users (which is something that is not practical, but that can be done in an on-premise environment). To summarize, a self-signed certificate is not practical even for on-premise/private environments is something tedious.

Another option is to buy a certificate from one of these CA companies (e.g. Digicert or Segtigo - this is not an advertisement, I'm just giving examples and I honestly recommend certificates that can be issued for free by LetsEncrypt) but this is usually expensive.

A really popular and free option is to issue certificates from a nonprofit CA company called Let's Encrypt. According to W3Techs, around 53% of the websites use a certificate issued by Let's Encrypt!

For a certificate to be issued to your domain/website you will need to prove that you own the domain (this is called a challenge). Let's Encrypt uses ACME protocol to verify that you control the given domain. There are several ACME clients that you can use to issue the certificate, for Kubernetes the most popular tool is Cert Manager. Let's Encrypt accepts the following challenges from the ACME protocol:

HTTP-01: Let's Encrypt gives you a token and you must expose this token in a given route/path (/.well-known/acme-challenge/<TOKEN>) in your website. Let's Encrypt will try to retrieve (several times from different servers) this token and check if this is the same token he sent to your ACME client. If it's the same then a new certificate will be issued, otherwise it will fail, meaning that you probably don't own/manage the domain you are trying to issue a certificate. Notice that your application must be public so Let's Encrypt can reach your server.
DNS-01: Instead of managing a route/path to expose the token in your application, DNS-01 challenge will verify that you own the domain by checking TXT records in your domain. Let's Encrypt gives you a TXT value to add to the domain and will check if the value was updated as expected, if so, it means you own the domain and the certificate will be issued. DNS-01 allows issuing certificates with wildcards, while HTTP-01 doesn't.

Some things that are important to mention:

The tokens/TXT values are not sensitive. They are only used to prove that you own the domain. Users who resolve the TXT record (DNS-01) or access the path containing the token (HTTP-01) can't do anything with this information
The certificate received is public and the private key is created by the ACME client (in our case Cert-Manager) and will be stored in a Kubernetes secret (once the certificate is issued, the public certificate will also be added to the secret)
The encryption and decryption part is done in the NGINX Ingress Controller pod
The certificate is valid for 90 days and needs to be renewed before this time
You can issue 5 certificates per week, this is a hard rate limit and you can not change it, so if you are still testing your application you should use the staging environment of Let's Encrypt

Both solutions are possible to be used in production but DNS-01 is preferred over HTTP-01. First, because it works for public and private applications. Second, because you can issue the certificate before the application is deployed, so no period of missing the certificate. We are going to use DNS-01 challenge since we have private applications and because we are going to issue a wildcard certificate. We will need to provide credentials to Cert Manager so it can go to our DNS and add the TXT record.

Kubernetes Replicator

As commented, when you issue the certificate, Cert Manager will create a Kubernetes secret containing the private and public keys that will be used by the ingress. As we know, Kubernetes secrets are namespace-scoped so if you have multiple applications in different namespaces there are a few options when using DNS-01:

Issue one certificate per every application, i.e. one certificate for app-a.mydomain.com, another for app-b.mydomain.com and so on (this is exactly what you have to do if you use HTTP-01)
Issue one wildcard certificate per every namespace, i.e. a certificate for *.mydomain.com in the apps namespace, *.mydomain.com in the monitoring namespace and so on.
Issue a single wildcard certificate and replicate this secret in every namespace needed

I'll use the third approach, especially because this will reduce the number of certificates issued, which can be critical for deployments in a big Kubernetes cluster. Since there is no built-in solution for Kubernetes to replicate a Kubernetes secret we need a tool for that. I will use Kubernetes Reflector but other tools can be used, such as Kubernetes Replicator. The usage is as simple as installing the tool and adding annotations to the secret you want to replicate.

DNS records and External DNS

We commented that ingress allows us to expose our applications (internal-facing or public-facing) using a Load Balancer and routing using an ingress controller internal to the cluster. We need to add these names (e.g. app-a.mydomain.com) to our DNS so clients can resolve this name into an IP.

We definitely don't want to manage this by hand, since this is tedious and prone to error. We can automate this task by using a well-known Kubernetes tool called External DNS. External DNS will be responsible for adding the DNS records to the DNS automatically!

The configuration is simple and it connects to several DNS providers, such as AWS, GCP, Azure, Linode, Digital Ocean, Vultr... We simply need to add permissions to connect and execute changes in the DNS and we are good to go! External DNS automatically monitors created ingresses.

Demo

Let's demo with a step-by-step approach. I will use Helm Chart with Kustomize to deploy the applications, since using GitOps (e.g. using ArgoCD) is out of the scope of this demo. For the infrastructure, I will use Terraform.

Creating the Kubernetes cluster

I'm using Vultr to deploy my Kubernetes cluster, so let's use Vultr Terraform provider.

# versions.tf
terraform {
  required_providers {
    vultr = {
      source = "vultr/vultr"
      version = "2.21.0"
    }
  }
}

provider "vultr" {}

Creating a Kubernetes cluster is as easy as providing a single terraform resource

# main.tf
resource "vultr_kubernetes" "this" {
    region  = "mia" # Miami
    label   = "terraform"
    version = "v1.30.0+1"

    node_pools {
        node_quantity = 1
        plan          = "vc2-2c-4gb"
        label         = "vke-nodepool"
        auto_scaler   = false
    }
}

output "cluster_id" {
  description = "ID of Kubernetes cluster"
  value       = vultr_kubernetes.this.id
}

To interact with Vultr API you need to set your environment variable to use the API key of your account (e.g. export VULTR_API_KEY=BRYAAFNWHYEFY2ZHGTQSDZCHEA). Make sure you have Terraform or Opentofu (I will use OpenTofu - which is the open-source version of Terraform) and run:

tofu init
tofu apply

You can use Vultr-CLI to get the kubeconfig (it will).

vultr-cli kubernetes config $(tofu output -raw cluster_id)

Installing Kubernetes Reflector

Let's use the Helm Chart to install the Kubernetes Reflector

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: reflector
  labels:
    app.kubernetes.io/name: reflector
---
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
helmCharts:
  - name: reflector
    repo: https://emberstack.github.io/helm-charts
    releaseName: reflector
    namespace: reflector
    version: 7.1.288

Installing NGINX Ingress Controller

We will need two Load Balancers: one for the public apps and another one for the private apps, because of that we will deploy the NGINX Ingress Controller two times.

NGINX pod will request to your Cloud Provider the Load Balancers so you will need to provide credentials to the pod to interact with the cloud. For Vultr this is as simple as providing the API token. For AWS you could provide the AWS Access Keys (this is not a best practice), IRSA or Pod Identity.

Notice that these credentials will also be used in the cert-manager (to create TXT records) and external-dns (to create records for the apps) so we could create these two secrets manually but let's create only a single secret and replicate using Kubernetes Reflector.

kubectl create namespace cert-manager
kubectl create namespace ingress-nginx
kubectl create secret generic vultr-credentials -n ingress-nginx --from-literal=apiKey=$VULTR_API_KEY
kubectl label secret vultr-credentials -n ingress-nginx created-manually=true
kubectl annotate secret vultr-credentials -n ingress-nginx reflector.v1.k8s.emberstack.com/reflection-auto-namespaces=kube-system,cert-manager
kubectl annotate secret vultr-credentials -n ingress-nginx reflector.v1.k8s.emberstack.com/reflection-allowed="true"
kubectl annotate secret vultr-credentials -n ingress-nginx reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true"

The Load Balancer creation can be configured using annotations that are specific to the cloud provider (e.g. AWS and Vultr). Private Load Balancers require special annotation.

Let's check the manifests for deploying NGINX Ingress Controller:

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
---
# values-private.yaml
controller:
  electionID: ingress-controller-leader-private
  ingressClass: nginx-private
  ingressClassResource:
    name: nginx-private
    controllerValue: k8s.io/private-ingress-nginx
  service:
    external:
      enabled: false
    internal:
      enabled: true
      annotations:
        service.beta.kubernetes.io/vultr-loadbalancer-vpc: true
        service.beta.kubernetes.io/vultr-loadbalancer-backend-protocol: "http"
---
# values-public.yaml
controller:
  electionID: ingress-controller-leader-public
  ingressClass: nginx-public
  ingressClassResource:
    name: nginx-public
    controllerValue: k8s.io/public-ingress-nginx
---
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ingress-nginx
resources:
  - namespace.yaml
helmCharts:
  - name: ingress-nginx
    repo: https://kubernetes.github.io/ingress-nginx
    releaseName: ingress-nginx-public
    namespace: ingress-nginx
    version: 4.11.1
    valuesFile: values-public.yaml
    includeCRDs: true
  - name: ingress-nginx
    repo: https://kubernetes.github.io/ingress-nginx
    releaseName: ingress-nginx-private
    namespace: ingress-nginx
    version: 4.11.1
    valuesFile: values-private.yaml
    includeCRDs: true

It might take a while for the Load Balancer to get created. You can run kubectl get service -n ingress-nginx and check if the Load Balancers services already have an IP assigned.

Installing Cert-Manager

Once again, let's use Helm Chart to install Cert Manager. One important thing to mention, when installing Cert Manager several CRDs are installed, such as ClusterIssuer and Certificate. We need to create a cluster issuer for Let's Encrypt and a wildcard certificate to be used in all namespaces.

When creating the cluster issuer using DNS-01 we need to define the DNS provider that we are going to use. There are several options available and the configuration of each one is specific to the provider. For Vultr, we need to use the webhook method, so we are going to install the Vultr webhook for Cert Manager. This will already provide the cluster issuer for us, but in case you are curious to know how the cluster issuer manifest would look for AWS it is like the following:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
  namespace: cert-manager
spec:
  acme:
    email: youremail@domain.com
    server: https://acme-v02.api.letsencrypt.org/directory # you can use staging URL for testing
    privateKeySecretRef:
      name: issuer-key
    solvers:
    - dns01:
        route53:
          region: us-east-1
          hostedZoneID: Z101361A2CEA1ZXYAMN60

So now, let's install Cert Manager.

# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager
  labels:
    app.kubernetes.io/name: cert-manager
---
# certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-tls
  namespace: default
spec:
  issuerRef:
    name: vultr-letsencrypt-prod # created by vultr webhook
    kind: ClusterIssuer
  duration: 2160h # 90 days
  renewBefore: 240h # 10 days
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  dnsNames:
    - "*.demosfelipetrindade.top"
  secretName: ca-tsl
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
---
# values.yaml
installCRDs: true
---
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - certificate.yaml
  - vultr-webhook.yaml
helmCharts:
  - name: cert-manager
    repo: https://charts.jetstack.io
    releaseName: cert-manager
    namespace: cert-manager
    valuesFile: values.yaml
    version: 1.15.2

Notice that I didn't include here the values of the vultr-webhook.yaml. You can check in the GitHub repository but I rather not paste it here because it's really big (the deployed Helm Chart of Vultr Webhook is old so I've used helm template to get the manifests from the source code).

It will take a while for the certificate to become valid. You can check the flow of issuing a certificate in the documentation but you basically:

You create the Certificate (kubectl get certificate)
The Certificate creates a CertificateRequest (kubectl get CertificateRequest)
An Order gets created (kubectl get order)
A Challenge gets created (kubectl get challenge)

During the certificate is being issued you can run kubectl get and kubectl describe to get the state of the resources. If you open your DNS provider you will see that a TXT record will be created and after validation it will be removed.

Installing External-DNS

Differently than Cert-Manager, here we don't need to instantiate CRDs, but we need to set credentials to the cloud provider.

# values.yaml
provider:
  name: vultr
env:
  - name: VULTR_API_KEY
    valueFrom:
      secretKeyRef:
        key: apiKey
        name: vultr-credentials # secret replicated
---
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
  - name: external-dns
    repo: https://kubernetes-sigs.github.io/external-dns/
    releaseName: external-dns
    namespace: kube-system
    valuesFile: values.yaml
    version: 1.14.5

Deploying a public application

Let's deploy a really simple application using Nginx as the base image and show the text "Hello, from public-app". We are going to deploy a Service of kind ClusterIp, a Deployment and an Ingress.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: public-app
  name: public-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: public-app
  template:
    metadata:
      labels:
        app: public-app
    spec:
      containers:
      - image: nginx
        name: nginx
        command:
          - sh
          - "-c"
          - "echo 'Hello, from public-app' > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"
---
# service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: public-app
  name: public-app
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: public-app
  type: ClusterIP
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-app
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx-public
  tls:
    - secretName: ca-tsl
      hosts:
        - public-app.demosfelipetrindade.top
  rules:
  - host: public-app.demosfelipetrindade.top
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: public-app
            port:
              number: 80
---
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml

Notice that we are using the public ingress class created in the NGINX Ingress Controller installation (nginx-public).

Deploying a private application

The manifests for deploying the private application are almost identical to the the public one, the only change will be the name of the application being deployed, the message displayed and the ingress class name (nginx-private)

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: private-app
  name: private-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: private-app
  template:
    metadata:
      labels:
        app: private-app
    spec:
      containers:
      - image: nginx
        name: nginx
        command:
          - sh
          - "-c"
          - "echo 'Hello, from private-app' > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"
---
# service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: private-app
  name: private-app
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: private-app
  type: ClusterIP
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: private-app
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx-private
  tls:
    - secretName: ca-tsl
      hosts:
        - private-app.demosfelipetrindade.top
  rules:
  - host: private-app.demosfelipetrindade.top
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: private-app
            port:
              number: 80
---
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml

Checking the applications

You can inspect the logs (kubectl logs) of the external-dns pod in the kube-system namespace and check that it will detect the new ingress and will create records in the DNS provider. Notice that it might take a while for the DNS to propage the changes and the applications to be accessible.

Now access the public application URL (in my case https://public-app.demosfelipetrindade.top). Check the message displayed!

Now let's check if we are able to access the private application URL (in my case https://private-app.demosfelipetrindade.top).

We can't, as expected! So let's see if we can access the private application within the cluster. For that, let's run a pod that has curl installed and curl the private app URL.

kubectl run curl --image=nginx:alpine
kubectl exec -it curl -- curl https://private-app.demosfelipetrindade.top

Bingo! The private application is accessible within the cluster! A great improvement would be to deploy a VPN in this network and access the private applications using the VPN but this is out of the scope of this blog post.

Cya!

Well, I hope you liked this blog post. We covered several tools used in real Kubernetes environments and explained in detail why they are needed and how they all are used together!

See you around! 👋

Image proxy cache using Harbor

Wed, 07 Aug 2024 21:40:32 GMT

You can check the GitHub repository an access all the manifests used in this blog post.

A common problem when dealing with highly dynamic and/or big Kubernetes cluster is facing pull limits from Dockerhub. The pull rate limits are:

100 pulls per 6 hours per IP (for unauthenticated users)
200 pulls per 6 hours (for authenticated users)

This is usually enough for small clusters, but dynamic and big clusters face issues. A simple solution is implementing a proxy cache project with Harbor. Notice that there are several benefits, besides avoiding the low rate pull limit from Dockerhub, of implementing an image proxy cache, such as:

Reduce egress traffic: NAT is usually charged by the amount of traffic
Faster pod startup: The image is already cached inside the cluster
Improve reliability: If the image is cached in the cluster, you do not depend on the external registry

How does a proxy cache work?

Harbor allows proxy cache to Dockerhub and Harbor repositories. Since our use case is to proxy cache from DockerHub I will only mention DockerHub, but keep in mind that it also applies to Harbor registries (in case you want to have a private image registry).

A proxy cache basically does the following:

If the image is not cached in the registry (Harbor), it will proxy the pull image registry to Dockerhub and cache the image locally
If the image is already cached, Harbor doesn't need to send the request to Dockerhub and will serve the cached image

In reality, there are some extra steps that Harbor does before serving the cache image, such as:

Checking if the image still exists in Dockerhub. If it doesn't, the image is not served
Checking if the image was updated. If it wasn't it serves cache, if it was it will proxy the request to Dockerhub (notice that this commonly happens if you use latest tag)
If DockerHub is not reachable, it will serve the cache image

Local Kubernetes setup

I will demo this locally using Kind. I highly recommend you check my blog post on how to set up a local Kubernetes cluster with LoadBalancer and name resolution in MacOs and Linux. I will reuse all the components present in the blog post. Since I don't want to repeat myself and keep things dry, I highly recommend you read it (or skim the text).

Demo time

Basic local Kubernetes cluster setup

Once again, some of these steps were explained in detail in my previous blog post, I won't go deeper into the explanation here

Create a docker network

docker network create --subnet 172.100.0.0/16 custom-kind-network

Create a Kind cluster

Using the following Kind configuration

# kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.30.2
  - role: worker
    image: kindest/node:v1.30.2

run the command:

KIND_EXPERIMENTAL_DOCKER_NETWORK=custom-kind-network kind create cluster --config kind.yaml

Install MetalLB

Let's install MetalLB using Helm Chart

helm repo add metallb https://metallb.github.io/metallb
helm install metallb -n metallb --create-namespace metallb/metallb
sleep 5
kubectl wait -n metallb -l app.kubernetes.io/component=controller --for=condition=ready pod --timeout=120s

Now that MetalLB is installed let's create our LBs IPs

# ip-adress-pool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ip-pool
  namespace: metallb
spec:
  addresses:
    - 172.100.150.0-172.100.150.10

and announce using the L2 ARP protocol

# advertisement.yaml
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: advertisement-l2
  namespace: metallb
spec:
  ipAddressPools:
    - ip-pool

Apply these manifests:

kubectl apply -f ip-adress-pool.yaml
kubectl apply -f advertisement.yaml

Install NGINX Ingress

Once again, let's use Helm to install NGINX Ingress

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx -n ingress-nginx --create-namespace ingress-nginx/ingress-nginx

You should see that an external-ip is being used by the NGINX service (kubectl get service -n ingress-nginx).

Harbor installation

The only configuration that needs to be changed for this demo (at the end of this blog post I will comment on several improvements that must be done in a real production environment) is related to the ingress, but I will also modify the admin password to be admin by simplicity (the classic admin/admin).

# values.yaml
externalURL: https://harbor.ingress.local
harborAdminPassword: admin
expose:
  type: ingress
  ingress:
    controller: default
    className: nginx
    hosts:
      core: harbor.ingress.local
  tls:
    certSource: auto

Notice that you will need to add harbor.ingress.local to your /etc/hosts file.

INGRESS_LB_IP=$(kubectl get svc ingress-nginx-controller -n ingress-nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "$INGRESS_LB_IP harbor.ingress.local" | sudo tee -a /etc/hosts

And finally install Harbor

helm repo add harbor https://helm.goharbor.io
helm repo update
helm install harbor -n harbor --create-namespace --values values.yaml --version 1.15.0 harbor/harbor

Wait for it to get ready and access https://harbor.ingress.local UI interface using username admin and password admin.

After login, this will be the first page you will see

Creating a proxy cache project

I will show you how to create via UI the proxy cache project but I will also provide the Terraform code.

Let's go to the Registries tab.

Now, let's create the endpoint for Dockerhub

Now we can create a new proxy cache project using the registry we just created

Instead of manually managing your Harbor configuration, a better approach is to create all of this using infrastructure as a code. I highly recommend Terraform/OpenTofu in these cases. You can simply use the Harbor provider created by the community and have all configurations as a code. Let's create a simple Terraform module for that

# version.tf
terraform {
  required_providers {
    harbor = {
      source  = "goharbor/harbor"
      version = "3.10.14"
    }
  }
}

provider "harbor" {
  url      = var.url
  username = var.username
  password = var.password
}

# variables.tf
variable "url" {
  description = "The url of harbor"
  type        = string
}

variable "username" {
  description = "The username to be used to access harbor"
  type        = string
}

variable "password" {
  description = "The password to be used to access harbor"
  type        = string
}

variable "proxy_project_name" {
  description = "The name of the project that will be created in harbor."
  type        = string
  default     = "proxy_cache"
}

# main.tf
resource "harbor_registry" "docker" {
  name          = "docker-hub"
  provider_name = "docker-hub"
  endpoint_url  = "https://hub.docker.com"
}

resource "harbor_project" "proxy" {
  name        = var.proxy_project_name
  registry_id = harbor_registry.docker.registry_id
}

# vars.tfvars
url ="http://harbor.ingress.local"
username ="admin"
password ="admin"

We are simply going to run (using OpenTofu):

tofu init
tofu apply -var-file=vars.tfvars

Using the harbor proxy cache

Ok, so let's recap what we have so far. We have a local Kubernetes cluster created using Kind with Harbor installed and configured with a proxy cache repository that is proxying DockerHub. Harbor is exposed by our MetalLB Load balancer and routed by the NGINX Ingress.

Now we would like to run an application and see Harbor proxy working in action. Let's run the following demos:

Deploy an application that uses Kustomize
Deploy an application that uses Helm Chart
Deploy an application whose image is already presented in the node

When we are using a private image registry (remember that we didn't check the Public box while creating the proxy cache project) we need to tell Kubernetes "Hey, use these credentials to pull the image from the registry". We do that by:

Pointing the image to the desired registry and repository. This looks obvious, but what I want to say is: that we are not going to directly use DockerHub images, we are going to use Harbor as a registry. This means that we will need to add a prefix to DockerHub images to point to the Harbor project (<harbor_server_name>/<proxy_cache_name>), in our use case this will be equivalent to harbor.ingress.local/proxy_cache.
Define the Kubernetes secret name that contains the credentials to access the private registry in the imagePullSecrets property of container definition

For simplicity, I'll use the admin user credentials (that have permission to pull images from the proxy cache project). Notice that this secret needs to be created in every namespace that our apps will require the usage of Harbor registry, in our case we will deploy all apps in the default namespace. Let's create this secret by running

kubectl create secret -n default docker-registry regcred-harbor \
  --docker-server=harbor.ingress.local \
  --docker-username=admin \
  --docker-password=admin

Ok, now let's comment about a tricky step that we will need to do to make this work locally. In a real environment with real TLS certificates, you wouldn't have this problem, but since we are running this locally with self-signed TLS certificates (or even without) this problem will appear. Kubernetes, by default, won't pull from "insecure" registries and because of that, we will need to configure the Kubernetes nodes to trust our private Harbor registry.

This configuration is done in the container runtime that is being used by the Kubernetes cluster, in our case, Kind only supports containerd, and for containerd, this configuration is defined in the /etc/containerd/config.toml file.

Let's connect your our Kubernetes node. We should do this to every worker node of our cluster, but since we only have 1 worker node this procedure will only be done once.

docker exec -it kind-worker bash

Let's install some utilities for editing the /etc/containerd/config.toml file

apt-get update
apt install vim -y
vim /etc/containerd/config.toml

and add the following lines to the file

[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.ingress.local"]
    endpoint = ["https://harbor.ingress.local"]
  [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.ingress.local".tls]
    insecure_skip_verify = true
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.ingress.local".auth]
  username = "admin"
  password = "admin"

and finally restart containerd so these changes can be applied:

systemctl restart containerd

Deploying an application that uses Kustomize

Let's deploy an NGINX Kubernetes Deployment manifest using Kustomize. A common pattern when using Kustomize is to have a base folder that will be reused by the overlays folder (usually the overlays folders represent an environment, e.g. development). We would apply generators and patches in the overlays folder but always use the base resources as a reference.

Let's suppose the Kubernetes deployment manifest is in the base folder. One option is to manually modify the prefix of the image there to point to our Harbor registry. It's possible but unlikely that all environments (overlays) will use the same endpoint. So a better idea is to add this as a patch. Let's create a patch for adding the registry credential and the prefix to the image.

# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-with-kustomize
  name: app-with-kustomize
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-with-kustomize
  template:
    metadata:
      labels:
        app: app-with-kustomize
    spec:
      containers:
        - image: nginx:1.26-alpine
          name: nginx
---
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

Now let's check the overlays folder for the local environment:

# local/harbor-image-prefix.yaml
apiVersion: builtin
kind: PrefixTransformer
metadata:
  name: image-prefix
prefix: harbor.ingress.local/proxy_cache/
fieldSpecs:
  - path: spec/template/spec/containers/image
---
# local/harbor-regcred.yaml
- op: add
  path: /spec/template/spec/imagePullSecrets
  value:
    - name: regcred-harbor
---
# local/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
resources:
  - ../base
patches:
  - path: harbor-regcred.yaml
    target:
      group: apps
      version: v1
      kind: Deployment
      name: app-with-kustomize
transformers:
  - harbor-image-prefix.yaml

Let's apply this using kubectl apply -k local and describe the pod.

Let's check the proxy_cache project in the Harbor UI to see if a new registry was created there.

Perfect!

Deploying an application that uses Helm

When using Helm, there are two possibilities: you created the Helm chart, you are using someone else Helm chart. I'm not going to detail the first option, since you have total control over the templates used you can simply add the variable to add the imagePullSecrets and also a variable to add a prefix to the image. The second option might be scary: what if the chart owner didn't add this variable? Well, it's possible but unlikely since these are very common things in a well-built Helm chart. Let's use Grafana chart as an example.

As you can see in the default values section, there are two variables that we are going to use: global.imageRegistry, global.imagePullSecrets. So let's create our custom values for this chart:

# values.yaml
global:
  imageRegistry: harbor.ingress.local/proxy_cache
  imagePullSecrets:
    - regcred-harbor

And install it by running

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install grafana grafana/grafana -n default --version 8.4.1 --values values.yaml

Let's describe the pod and check how it went:

Finally, let's check if the registry was created in Harbor

Perfect!

Deploy an application whose image is already presented in the node

Now we are going to deploy an application using an image directly from Dockerhub and modifying it to use Harbor proxy cache and see how it's going to behave, it might surprise you.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: dummy
  name: dummy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dummy
  strategy: {}
  template:
    metadata:
      labels:
        app: dummy
    spec:
      containers:
      - image: httpd
        name: httpd

Let's apply that and see that indeed it worked as expected. The pod is running. Now, let's modify this deployment a bit to use the Harbor proxy cache and the registry private credential.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: dummy
  name: dummy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dummy
  strategy: {}
  template:
    metadata:
      labels:
        app: dummy
    spec:
      imagePullSecrets:
      - name: regcred-harbor
      containers:
      - image: harbor.ingress.local/proxy_cache/httpd
        name: httpd

Now, let's describe the pod and see if it pulled the image correctly:

Now if you check Harbor UI you will see that the httpd image is not there! Why? The image was already present in our node! So kubelet queried Harbor to resolve the name of the image (in our case httpd:latest) to an image digest. Since the image digest wanted was the same as the one cached locally (if I waited a time long enough to make the digest different - i.e. a new image was pushed to latest tag - this would be different!) then kubelet used the cached image (presented on the node).

Let's run an experiment to prove our point. Let's connect to the Kubernetes node and delete the image presented on the node.

docker exec -it kind-worker bash
crictl rmi harbor.ingress.local/proxy_cache/httpd

Now, let's roll out the deployment (or simply delete the pod, downtime is irrelevant here) and check if the image will now be present in Harbor.

Bingo!

Cya

Hope you enjoyed this blog post! Please, don't copy and paste this to directly deploy to the production environment. There are SEVERAL things that you would need to do to be in a production-ready state. Here are a few tips before deploying this in a real environment.

Use TLS and certificates: Enhance your security by using TLS/HTTPS and certificates. Adding insecure registries to Kubernetes nodes is not a good practice.
Configure external database for Harbor: The configurations for Harbor (such as SSO, and users) are all stored in the database. By default, the Helm chart deploys a Postgres database for you but it's more secure to use an external Postgres database (it can even be a database that you run inside Kubernetes with an Operator).
Don't Harbor credentials explicit in Terraform variables: A good option is to store this somewhere safe (AWS Secrets Manager, SOPS, Hashicorp Vault...) and pass only the name of the secret that contains the credentials. This way you can use the correct provider to pull this information.
Don't use admin user credentials as image pull secret: Create a user with limited permissions and use these credentials

See you in the next post! 👋

Local Kubernetes deployment with LoadBalancer and name resolution in MacOs and Linux

Sun, 28 Jul 2024 20:20:32 GMT

The goal of this post is to teach you (explaining the concepts, problems and doing a demo) how to create a local Kubernetes cluster, expose your application using an Load Balancer and manage the traffic using Ingress using Linux and MacOs.

I will slowly explain the logic behind the setup and the need for every tool used here. If you want to check the code directly you can take a look on the GitHub repository. Let's start!

Local Kubernetes cluster

There are several tools to run Kubernetes locally, such as Kind, k3s, k3d, Minikube. My personal choice is always Kind because it's lightweight (Kind stands for Kubernetes in Docker) because it runs in containers instead of virtual machines and is one of the few tools (if not the only) that allows you to run a multi node cluster (multi control plane and multi worker nodes), which is very handy to test affinity, tolerance, rolling restarts... So we are going to use Kind to provision a local Kubernetes cluster.

Since Kind uses containers we also need to choose a container runtime to run our containers (nodes of the Kubernetes cluster). I will stick to Docker Desktop for MacOs and Linux, which comes by default when you intall Docker.

For MacOs there are other options, such as Colima or Orbstack. Due to Docker Inc licence agreement change (Docker Desktop is no longer free for enterprise usage), some engineers decided to use Colima, since an open-source project, to setup the container environment. In my case Docker Desktop is fine, specially because for the Kubernetes set up I'll comment in this blog post I will need a tool that is only supported in Docker Desktop.

Docker container network

To run containers in MacOs you will need to run them inside a Linux virtual machine (VM), since Docker containers run o a Linux kernel. This means that Docker Desktop and Colima, for exameple, will create VMs to run your containers.

This reminds me of Matrix. In fact, when running a container in MacOs we are running the container inside a Virtual Machine. So in MacOs, when you run a container with port-forward (e.g. docker container run nginx -p 3000:80) the traffic gets routed to the VM and then the VM routes the request to the container inside it.

So, using MacOs, because the containers run inside a Virtual Machine we can't directly access their network interface, there is no traffic by default to the VM. We could configure the iptable and routing rules to send traffic from MacOs to the VM and then set forward traffic rules to the Kind network. This is tedious. I way better and automatic approach of managing that is to install docker-mac-net-connect. This tool allow us to directly connect to the ip of the container.

Let's give an example to make it easier to understand. Without the tool installed I will run an nginx container without port-forward (docker container run -d --name nginx nginx). Without port-forward you won't be able to access this container. Let's check the IP of the container (NGINX_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' nginx)) and try to run a curl (curl $NGINX_IP). You will get a timeout. You simply can't reach the container, there is no traffic. Now, install the docker-mac-net-connect` tool and you will see that you are able to get a response from the NGINX container! Don't forget to kill the container we created for testing (docker rm -f nginx).

If you are a Linux user, you can ignore all the previous explanation. In Linux, you have direct access to the docker network interface and can access the containers directly by IP!

MetalLB

This setup is completely local (similar to an on-premise environment), there is no cloud involved so if you want to expose your application using a LoadBalancer you will need to have a tool to create this LoadBalancer for you inside your cluster. The most popular tool is, by far, MetalLB. MetalLb provides a network load balancer implementation, i.e. allows you to create Kubernetes services of type LoadBalancer in clusters that don't run on a cloud provider.

To use MetalLb you will need to define two things (considering you have already installed it in your cluster):

The IPs that MetalLb will assign to the load balancer. Notice that for our use-case, this IP must be in the same network space (network CIDR) of our local Kubernetes cluster. This is done by instanciating a Kubernetes resource of kind IPAddressPool (a custom resource that comes with MetalLB installation).
Announce the IPs of the load balancer to make the network beyond our cluster aware of the IPs of the LoadBalancers. This can be achieve using two routing protocols: ARP/NDP (OSI Layer 2) or BGP. This is done by instanciating a Kubernetes resource of kind L2Advertisement for ARP. BGP set up is more specific and we won't use in this demo, so I won't comment to make this post less confusing.

To summarize, MetalLb allow us to expose your applications (demo comes soon!). So if we deploy an application (e.g. Kubernetes Deployment) and expose this application using Kubernetes service of type LoadBalancer, MetalLB will assign one IP for our service and user using this IP can access our applications.

Ingress Controller

Let's be honest, directly using IPs to access application is something tedious and no one does that. We access applications using names (e.g. myapp.mydomain.tld). Also, if we expose every application with a load balancer we might run out of IPs.

A common practice is to re-use the LoadBalancer and internally (inside Kubernets) forward the traffic to the requested service. There are several cloud-agnostic ingress controllers, such as NGINX Ingress and Traefik. Since I'm more familiar with NGINX Ingress will use it in this post.

Demo time

For this demo I will assume you have:

For a better experience, please check the content of GitHub repository containing all code needed to run this demo.

Creating the Kind Cluster

The MetalLB IPs need to be in the range of the IPs of the cluster. I good strategy is to before the Kubernetes cluster creation to create the range of IPs we want our Kubernetes cluster to be in. This way the Kubernetes manifest for the MetalLB ip pool (IPAddressPool) won't need to be modified. This docker network will be used by our Kind cluster on the following step.

docker network create --subnet 172.100.0.0/16 custom-kind-network

Now, let's create our local Kind cluster with two nodes: one control-plane and one worker.

cat <<EOF > /tmp/kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.30.2
- role: worker
  image: kindest/node:v1.30.2
EOF
KIND_EXPERIMENTAL_DOCKER_NETWORK=custom-kind-network kind create cluster --config /tmp/kind-config.yaml

You can check the nodes of the cluster by running docker ps.

Installing MetalLB

Install MetalLB using Helm.

helm repo add metallb https://metallb.github.io/metallb
helm install metallb -n metallb --create-namespace metallb/metallb
sleep 5
kubectl wait -n metallb -l app.kubernetes.io/component=controller --for=condition=ready pod --timeout=120s

Now, let's create a range of IPs (that must be inside the docker network we just created).

kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ip-pool
  namespace: metallb
spec:
  addresses:
  - 172.100.150.0-172.100.150.10
EOF

Notice that we are allocating 10 IPs for load balancer in our local cluster, which is more than enought for this demo.

Now, using ARP protocol, let's announce these IPs.

kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: advertisement-l2
  namespace: metallb
spec:
  ipAddressPools:
  - ip-pool
EOF

Install NGINX Ingress

Install NGINX Ingress using Helm.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx -n ingress-nginx --create-namespace ingress-nginx/ingress-nginx

Run kubectl get service -n ingress-nginx to notice that NGINX Ingress create a service of type Load Balancer (and MetalLB assigned an external ip to this Load Balancer in the range we specified before). This will be used later!

Deploy application with LoadBalancer service

Let's deploy an application that uses LoadBalancer service type.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-load-balancer
  name: app-load-balancer
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-load-balancer
  template:
    metadata:
      labels:
        app: app-load-balancer
    spec:
      containers:
      - image: nginx
        name: nginx
        command:
          - sh
          - -c
          - "echo 'Hello, from app-load-balancer' > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"
EOF

And for the service:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  labels:
    app: app-load-balancer
  name: app-load-balancer
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: app-load-balancer
  type: LoadBalancer
EOF

Now, let's check if our application was correctly provisioned with a Load Balancer service with an external IP (run kubectl get service -n default).

Now, let's check if we can access this service via Load Balancer external IP (172.100.150.1).

Deploy application with Ingress

Let's deploy an application that uses the NGINX Ingress. Starting with the deployment manifest:

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-ingress
  name: app-ingress
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-ingress
  template:
    metadata:
      labels:
        app: app-ingress
    spec:
      containers:
      - image: nginx
        name: nginx
        command:
          - sh
          - "-c"
          - "echo 'Hello, from app-ingress' > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"
EOF

The service

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  labels:
    app: app-ingress
  name: app-ingress
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: app-ingress
  type: ClusterIP
EOF

And finally the ingress

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: app-ingress.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-ingress
            port:
              number: 80
EOF

The app-ingress application is exposed by the NGINX Ingress Load Balancer and routed accordingly by the Ingress resource. In order to locally resolve "app-ingress.local" we will need to add this entry to our /etc/hosts file. If it was a real Kubernetes deployment we would add an A record to the DNS server. To be honest, you can setup a local DNS server (using dnsmasq for example) to specify that all .local names should be resolved to the IP of the NGINX Load Balancer. This is specially useful if you have several services exposed locally (/etc/hosts does not support wildcard, so we would need to add an entry for every service we expose via Ingress).

Get the NGINX Ingress IP

INGRESS_LB_IP=$(kubectl get svc ingress-nginx-controller -n ingress-nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "Add $INGRESS_LB_IP to /etc/hosts"

Now add this IP to the /etc/hosts file

echo "$INGRESS_LB_IP app-ingress.local" | sudo tee -a /etc/hosts

Now access http://app-ingress.local. Check if the application is accessible.

If you are done with the demo just run the command below to destroy the Kubernetes cluster.

kind delete cluster

Cya

Hope you enjoyed this blog post! There only a few resources online explaining how to do that using MacOs, because the VMs is tricky. This demo works for both Linux and MacOs and I hope you found it helpful!

See you in the next post! 👋

Consistent deployments - a real-world case scenario

Sun, 09 Jun 2024 11:20:32 GMT

Recently I was part of a project that didn't have consistent deployments (infra and application-related). I'll go over some problems faced by the client and the proposed solution for that.

If you would like to directly check the code used in this blog post, please check the following GitHub repositories:

Deployment state

The client had two repositories: one for infrastructure and another one for all the applications (Lambdas and Glue jobs). The CI/CD of the application repository was responsible for running tests, building the Docker image, pushing the image (latest tag) to the ECR of the environment, and updating Lambda code to use this new code version. The CI/CD of the infra repository was responsible for planning the changes and then deploying the changes to the infrastructure (Lambda, Glue jobs and other things).

Before we start to comment on it I would like to clarify some concepts (I will use these words a lot during this blog post):

Artifacts (assets): It represents the application. In our case Docker images (used in the Lambdas). It could be another thing, such as a JAR file, a zip file containing application code, or Python scripts for Glue jobs...
Manifest: Declarative configuration of the infrastructure. In our case, it will be Terraform files, but it could have been CDK code, CloudFormation template, Pulumi code...

At first glance, this looks perfectly fine. Indeed, I've seen that multiple times in multiple clients, but there are several problems here and I would highlight four:

Deployments are not atomic. The configuration/infra and artifact should be deployed simultaneously: either both are successfully deployed, or neither is.

Ensuring that both the application code (artifact) and its configuration are deployed together reduces the risk of incompatibilities because partial deployments can lead to failures and unexpected behaviors if the configuration does not match the deployed code. Also, managing dependencies and orchestrating deployments to ensure they happen simultaneously can be challenging.

Imagine the scenario where a Lambda needs to access a new cloud resource (e.g. S3, SQS...). If the application code is updated before the IAM Role has the needed permissions it can lead to failures. We want to deploy the new code with the needed permissions simultaneously. The same problem would happen if it was a new code that requires more memory/CPU or a new environment variable.

Different artifacts are being used in different environments. There is no artifact promotion involved.

For every environment, a new Docker image is being built and pushed to ECR. Deploying the same artifact that was tested ensures the environment closely mirrors the one in which the artifact was validated (i.e. production environment should use the same artifact that was tested in staging, that should use the same artifact that was tested in development). This reduces the risk of deploying untested or modified artifacts.

Even when using a Dockerfile it's possible to generate two different artifacts. There are potential causes for that, such as:

Environment differences: Development applications might have additional debugging tools or dependencies that are not present in production.
External resources: If the build process relies on external resources that can change over time this can lead to non-deterministic builds.
Dependency version: When not specifying versions when installing dependencies (e.g. apt install, npm install, pip install) the latest version will be used, which can be different from development and production depending on when the build ran.

Even if you are not satisfied with this answer, there is a cost impact. We are multiplying our costs based on the number of environments. We are storing the "same" build over different environments, instead of having a centralized place to store the artifacts.

There is not an easy way to revert configuration/infra and artifact.

It's a common requirement for deployment systems to be able to rollback in case of issues with the new release. The way to do it on the presented scenario is to revert the infra change (revert PR on the infrastructure repository) and revert the application change (revert PR on the application repository). We would still wait for the CI/CD to run to generate new assets. Oh, and we would have to wait for deployment in development, staging, and production (assuming only 3 environments).

Every time a new deployment is done, manifests get outdated (drift is shown)

Since the application repository has a CD that forces the update of a new Lambda, this will generate a drift in the infra repository, because the infra repository state applied in the past a different manifest (even if the Docker image is the same as the hash of the image is different - it's a different artifact!).

Looking for solutions

We can improve by a lot the deployment reliability/consistency, let's check some important points to consider.

Application versioning

If you are using the same Docker tag for your deployments you are not versioning your application (e.g. using dev or latest for the development environment). This is a big problem. If you are using the same image tag you don't know what version of your application is running and don't have an easy way to run a previous version.

There are two widely used options to version your artifacts: Semantic Versioning (SemVer) and Git Hash.

SemVer uses a part number system MAJOR.MINOR.PATCH (e.g. 1.5.4). You are probably very familiar with SemVer because it's widely used in the IT world. It's readable (it's just tree numbers!), easy to communicate ("We just deployed version X.Y.Z"), and predictable (if it's a major change you can expect breaking changes). On the other hand, the initial overhead to implement can be challenging and you don't know what a given version represents in code (unless you use Git tags).

Git hash is even more familiar to you because every developer knows (at least that's what I expect...) Git. A SHA-1 hash that is unique and identifies a commit in a Git repository. It's commonly used because it maps to a specific point in the repository's history (just do git checkout <GIT_HASH> and you can see it!), making it easier to understand exactly what code is being used, and it's easy to implement in CI/CD pipelines since you don't need to worry if the change is a breaking change, minor change or a patch/hotfix to name your version, it's simply the git hash. On the other hand, the readability is terrible ("Hey, I'm deploying version 7b2d773") and it's meaningless (it's a hash and that's it - you can't get good information from it).

I've worked with both approaches and my personal choice is SemVer, although the initial setup is harder, I think it pays off in the long run, but of course, this depends on several things, such as the deadline, personal preference of the team, maturity of the team (not everyone knows if they should increment a major, a minor or a patch).

IaaC as the source of truth

Drifts are happening because we are not treating the infrastructure repository as the source of truth. We shouldn't deploy a new application version from the application repository. From the artifacts point of view, the application repository should only build new artifacts.

The ultimate goal is to have the infrastructure repository in a 1:1 relationship with what we have deployed. In other words, the infrastructure repository should be the source of truth when it comes to what we have deployed in the environments. This is completely achievable but hard, especially if we are talking about ephemeral deployments, but this is a subject for another time.

The artifact version should be managed by the application repository! Well, it can be done manually, but we - as DevOps - hate manual things, and automation is the way to go. The CD of the application repository should open a Pull Request to modify the version of the application in the manifests. E.g. if you are using Terraform it can simply be a change on the variable in the var.tfvars file that corresponds to the version of the application Lambda. To be honest, the CD can directly push to the main/master branch of the infrastructure repository but this can be very risky and seen as not a good practice by some DevOps Engineers.

If the change does not require infrastructure change (IAM permission, new environment variable, changes on the memory/cpu), the developers can simply merge the PR. Otherwise, changes can be implemented on this pull request to deploy infra and code changes at the same time. Notice that team coordination is ALWAYS important. If a new version needs to be deployed and this new version requires infrastructure change this needs to be aware by all the team (dev + ops).

Monorepo x Multi-Repo

I mentioned that the client used a multi-repo approach: one for the infrastructure and another for the applications (a single repository per application would also have the same impact). A possible solution is to use a monorepo approach: infrastructure and application code on the same git repository.

A single pull request that changes infra and application code would do the trick. Atomic changes are a big pro of using a monorepo strategy.

To be honest, I'm not a big fan of having applications and infrastructure together on the same repository. I rather have a clear separation using different repositories. This way you can have an easier way to controll access to codebase, an independent development cycle, and higher performance (faster cloning, building, testing).

If you go for a monorepo approach you will need more collaboration from your team, i.e., if the application team wants to deploy a new code that uses a new environment variable, it needs to collaborate with the infrastructure team to add this environment variable simultaneously.

Centralized assets

As we commented, deploying the "same" assets in different accounts it's not a good practice. Ideally, you should have a centralized place to put your assets (scripts, images). Many companies use the production environment for that and I don't like that.

The best practice is to have a specific AWS account only for the assets. This account can also be used to host all CI/CD (if you decide to use CodePipeline, CodeBuild, or CodeDeploy).

All the other environments (e.g. dev, staging, prod) should be able (cross-account permissions) to pull assets from this account!

Hotfix workflow

Imagine the following scenario: the new application version was tested in dev, staging, and production. It's running fine for some time but suddenly you start to see several errors related to your application and you NEED to rollback the version in production.

A hotfix workflow is a workflow that runs directly in production to fix things as fast as possible. You don't want to run tests, build (remember that the old asset already exists!), and other things. You just want to fix the problem as fast as possible. You might see some DevOps that allows you to add new code (instead of rolling back the version) in the hotfix workflow, this is also possible, but I think rolling back is safer.

Hands-on

I'll propose the following solution:

Multi-repo approach
Application repository builds and push assets to assets account and creates a pull request (remember to enable that in the GitHub Actions settings of the repository) to the infrastructure repository to update image being used
Git hash to version images

I wouldn't write this and not demo what I'm saying! To demo this I chose the following stack:

AWS: For the Cloud provider.
ECR: For the container registry.
Python: For the Application (Lambda) code. It's a dummy application, don't worry!
Terraform: For the infrastructure deployment.
GitHub Actions: For the CI/CD.

Some things were created beforehand because they were used in another personal projects or because manual creation is the way to go:

IAM Identity Provider: GitHub
IAM Role: Role that will be assumed by GitHub CI/CD using OIDC
Github Actions Environments: 3 environments that require manual approval (assets, dev and prod)
GIthub Secrets: Added PAT_GITHUB_TOKEN to the application repository (needed to create pull requests on the infrastructure repository)
S3 Buckets: To store Terraform state

Application

Let's check the application code:

import os

import cowsay


def lambda_handler(event, context):
    print(f"event ==> ${event}")
    cowsay.cow(os.environ["TO_SAY"])

It's a simple application that uses cowsay! The code will run as a dockerized lambda that has the following Dockerfile:

FROM public.ecr.aws/lambda/python:3.11
COPY requirements.txt  .
RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
COPY .  .
CMD [ "main.lambda_handler" ]

The CI/CD contains two jobs (build-and-push and infra-pr):

name: Application CI/CD

on:
  push:
    branches:
      - main
  pull_request:

permissions:
  id-token: write
  contents: write
  pull-requests: write

env:
  AWS_ACCOUNT_ID: "730335516527"
  AWS_IAM_ROLE_OIDC_NAME: "GitHubActions"
  AWS_REGION: "us-east-1"
  ECR_REPO_NAME: "consistent-deployments"
  INFRA_REPOSITORY: felipelaptrin/consistent-deployments-infra-blog

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Build Docker image
        env:
          ECR_REPOSITORY: ${{ env.ECR_REPO_NAME }}
          IMAGE_TAG: ${{ github.sha }}
        run: docker build -t $ECR_REPOSITORY:$IMAGE_TAG .

      - name: Configure AWS Credentials
        if: github.ref == 'refs/heads/main'
        uses: aws-actions/configure-aws-credentials@v3
        with:
          role-to-assume: arn:aws:iam::${{ env.AWS_ACCOUNT_ID }}:role/${{ env.AWS_IAM_ROLE_OIDC_NAME }}
          aws-region: ${{ env.AWS_REGION }}
          role-session-name: GitHubActions

      - name: Push image to Amazon ECR
        if: github.ref == 'refs/heads/main'
        env:
          ECR_REGISTRY: ${{ env.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com
          ECR_REPOSITORY: ${{ env.ECR_REPO_NAME }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          aws ecr get-login-password --region ${{ env.AWS_REGION }} | docker login --username AWS --password-stdin ${{ env.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com
          docker tag $ECR_REPOSITORY:$IMAGE_TAG $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

  infra-pr:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    needs: [build-and-push]
    steps:
      - name: Checkout the infrastructure repository
        uses: actions/checkout@v4
        with:
          repository: ${{ env.INFRA_REPOSITORY }}
          token: ${{ secrets.PAT_GITHUB_TOKEN }}

      - name: Set up Git
        run: |
          git config --global user.name 'github-actions[bot]'
          git config --global user.email 'github-actions[bot]@users.noreply.github.com'

      - name: Update app_version (tfvars file)
        env:
          VERSION: ${{ github.sha }}
        run: |
          sed -i -E 's/(app_version\s*=\s*").*"/\1'"${VERSION}"'"/' dev/dev.tfvars
          sed -i -E 's/(app_version\s*=\s*").*"/\1'"${VERSION}"'"/' prod/prod.tfvars

      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v4
        with:
            token: ${{ secrets.PAT_GITHUB_TOKEN }}
            commit-message: Update application version to ${{ github.sha }}
            title: Update application version to ${{ github.sha }}
            body: This was created by Application CI/CD.
            base: main
            branch: feat/${{ github.sha }}

Infrastructure

The infrastructure contains two modules: ECR and Lambda. The ECR module contains the following resources:

# ECR
locals {
  account_ids = {
    dev    = "937168356724"
    prod   = "654654203090"
    assets = "730335516527"
  }
  base_name = "consistent-deployments"
}

resource "aws_ecr_repository" "this" {
  name                 = local.base_name
  image_tag_mutability = "IMMUTABLE"
}

resource "aws_ecr_repository_policy" "this" {
  repository = aws_ecr_repository.this.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          AWS = values(local.account_ids)
        }
        Action = [
          "ecr:BatchCheckLayerAvailability",
          "ecr:BatchGetImage",
          "ecr:GetDownloadUrlForLayer"
        ]
      },
      {
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
        Action = [
          "ecr:BatchCheckLayerAvailability",
          "ecr:BatchGetImage",
          "ecr:GetDownloadUrlForLayer"
        ],
        Condition = {
          StringLike = {
            "aws:sourceArn": [ for account in values(local.account_ids) : "arn:aws:lambda:${var.aws_region}:${account}:function:*" ]
          }
        }
      }
    ]
  })
}

The Lambda module contains the following resources:

# LAMBDA
locals {
  account_ids = {
    dev    = "937168356724"
    prod   = "654654203090"
    assets = "730335516527"
  }
  base_name = "consistent-deployments"
}

data "aws_iam_policy_document" "this" {
  count = var.deploy ? 1 : 0
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "this" {
  count = var.deploy ? 1 : 0

  name               = local.base_name
  assume_role_policy = data.aws_iam_policy_document.this[0].json
}

resource "aws_iam_role_policy" "this" {
  count = var.deploy ? 1 : 0

  name = "lambda-policy"
  role = aws_iam_role.this[0].id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Effect   = "Allow"
        Resource = "*"
      },
      {
        Action = [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ]
        Effect = "Allow"
        Resource = [
          "arn:aws:ecr:${var.aws_region}:${local.account_ids.assets}:repository/${local.base_name}"
        ]
      }
    ]
  })
}

resource "aws_lambda_function" "this" {
  count = var.deploy ? 1 : 0

  function_name = "${var.environment}-${local.base_name}"
  role          = aws_iam_role.this[0].arn
  image_uri     = "${var.ecr_repo_url}:${var.app_version}"
  package_type  = "Image"
  environment {
    variables = {
      TO_SAY = "Hello from Lambda!"
    }
  }
}

We have two GitHub Actions workflows. The first one is for the normal deployment of the infrastructure that contains two jobs per environment: one for generating the plan (tf plan) of the deployment and another one for deploying. The second is the hotfix workflow.

name: Infrastructure CI/CD

on:
  push:
    branches:
      - main
  pull_request:

permissions:
  id-token: write
  contents: read

jobs:
  plan-assets:
    name: Plan assets infra deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Plan assets deployment
        uses: ./.github/actions/plan-infra
        with:
          environment: assets
          working_directory: assets
          aws_account_id: 730335516527

  deploy-assets:
    if: github.ref == 'refs/heads/main'
    name: Deploy assets infra
    needs: [plan-assets]
    runs-on: ubuntu-latest
    environment:
      name: assets
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Deploy assets deployment
        uses: ./.github/actions/deploy-infra
        with:
          environment: assets
          working_directory: assets
          aws_account_id: 730335516527

  plan-dev:
    name: Plan dev infra deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Plan dev deployment
        uses: ./.github/actions/plan-infra
        with:
          environment: dev
          working_directory: dev
          aws_account_id: 937168356724

  deploy-dev:
    if: github.ref == 'refs/heads/main'
    name: Deploy dev infra
    needs: [plan-dev, deploy-assets]
    runs-on: ubuntu-latest
    environment:
      name: dev
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Plan dev deployment
        uses: ./.github/actions/deploy-infra
        with:
          environment: dev
          working_directory: dev
          aws_account_id: 937168356724

  plan-prod:
    name: Plan prod infra deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Plan prod deployment
        uses: ./.github/actions/plan-infra
        with:
          environment: prod
          working_directory: prod
          aws_account_id: 654654203090

  deploy-prod:
    if: github.ref == 'refs/heads/main'
    name: Deploy prod infra
    needs: [plan-prod, deploy-dev]
    runs-on: ubuntu-latest
    environment:
      name: prod
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Plan prod deployment
        uses: ./.github/actions/deploy-infra
        with:
          environment: prod
          working_directory: prod
          aws_account_id: 654654203090

These jobs are very repetitive, so to make it easier to reuse and avoid repetition the following actions were created:

name: plan-environment
description: Plan Opentofu
inputs:
  environment:
    description: Environment name - accepts `dev` and `prod`
    required: true
  working_directory:
    description: Working directory
    required: true
  aws_account_id:
    description: AWS Account ID to run
    required: true
  aws_iam_role_oidc_name:
    description: Name of the IAM Role that will be assumed via OIDC
    default: "GitHubActions"
  aws_region:
    description: AWS region
    default: "us-east-1"
  opentofu_version:
    description: OpenTofu version to use
    default: 1.7.2

runs:
  using: "composite"
  steps:
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v3
      with:
        role-to-assume: arn:aws:iam::${{ inputs.aws_account_id }}:role/${{ inputs.aws_iam_role_oidc_name }}
        aws-region: ${{ inputs.aws_region }}
        role-session-name: GitHubActions

    - uses: opentofu/setup-opentofu@v1
      with:
        tofu_wrapper: false
        tofu_version: ${{ inputs.opentofu_version }}

    - name: Opentofu init
      shell: bash
      working-directory: ${{ inputs.working_directory }}
      run: tofu init

    - name: Opentofu plan
      shell: bash
      working-directory: ${{ inputs.working_directory }}
      run: |
 if find . -maxdepth 1 -name "*.tfvars" | grep -q .; then
 tofu plan -out tf-${{ inputs.environment }}.plan -lock=false -var-file="${{ inputs.environment }}.tfvars"
 else
 tofu plan -out tf-${{ inputs.environment }}.plan -lock=false
 fi

    - name: Upload Opentofu plan file as artifact
      uses: actions/upload-artifact@v4
      with:
        name: tfplan-${{ inputs.environment }}
        path: ${{ inputs.working_directory }}/tf-${{ inputs.environment }}.plan

and

name: deploy-environment
description: Deploy Opentofu
inputs:
  environment:
    description: Environment name - accepts `dev` and `prod`
    required: true
  working_directory:
    description: Working directory
    required: true
  aws_account_id:
    description: AWS Account ID to run
    required: true
  aws_iam_role_oidc_name:
    description: Name of the IAM Role that will be assumed via OIDC
    default: "GitHubActions"
  aws_region:
    description: AWS region
    default: "us-east-1"
  opentofu_version:
    description: OpenTofu version to use
    default: 1.7.2

runs:
  using: "composite"
  steps:
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v3
      with:
        role-to-assume: arn:aws:iam::${{ inputs.aws_account_id }}:role/${{ inputs.aws_iam_role_oidc_name }}
        aws-region: ${{ inputs.aws_region }}
        role-session-name: GitHubActions

    - uses: opentofu/setup-opentofu@v1
      with:
        tofu_wrapper: false
        tofu_version: ${{ inputs.opentofu_version }}

    - name: Opentofu init
      shell: bash
      working-directory: ${{ inputs.working_directory }}
      run: tofu init

    - name: Download Opentofu plan from previous action
      uses: actions/download-artifact@v4
      with:
        name: tfplan-${{ inputs.environment }}
        path: ${{ inputs.working_directory }}

    - name: OpenTofu Apply
      shell: bash
      working-directory: ${{ inputs.working_directory }}
      run: tofu apply -input=false -lock-timeout=600s tf-${{ inputs.environment }}.plan

The hotfix workflow is the following:

name: Hotflow CI/CD

on:
  workflow_dispatch:
    inputs:
      rollback_version:
        description: Version that should be used to rollback the application
        required: true

permissions:
  id-token: write
  contents: write

jobs:
  plan-prod:
    name: Plan prod infra deployment
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Rollback app_version
        env:
          VERSION: ${{ github.event.inputs.rollback_version }}
        run: sed -i -E 's/(app_version\s*=\s*").*"/\1'"${VERSION}"'"/' prod/prod.tfvars

      - name: Plan prod deployment
        uses: ./.github/actions/plan-infra
        with:
          environment: prod
          working_directory: prod
          aws_account_id: 654654203090

  deploy-prod:
    if: github.ref == 'refs/heads/main'
    name: Deploy prod infra
    needs: [plan-prod]
    runs-on: ubuntu-latest
    environment:
      name: prod
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Deploy prod deployment
        uses: ./.github/actions/deploy-infra
        with:
          environment: prod
          working_directory: prod
          aws_account_id: 654654203090

      - name: Set up Git
        run: |
 git config --global user.name 'github-actions[bot]'
 git config --global user.email 'github-actions[bot]@users.noreply.github.com'

      - name: Rollback app_version
        env:
          VERSION: ${{ github.event.inputs.rollback_version }}
        run: sed -i -E 's/(app_version\s*=\s*").*"/\1'"${VERSION}"'"/' prod/prod.tfvars

      - name: Push changes
        uses: ad-m/github-push-action@v0.8.0
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          branch: main

Assets account

The assets account will only use the ECR module:

# main.tf

module "this" {
  source      = "../modules/ecr"
  aws_region  = "us-east-1"
  environment = "assets"
}

Dev account

The dev account will use only the Lambda module.

# main.tf

data "terraform_remote_state" "assets" {
  backend = "s3"
  config = {
    bucket = "terraform-states-730335516527"
    key    = "consistent-deployments.tfstate"
    region = "us-east-1"
  }
}

module "this" {
  source       = "../modules/lambda"
  aws_region   = "us-east-1"
  environment  = "dev"
  ecr_repo_url = data.terraform_remote_state.assets.outputs.ecr_repo_url
  deploy       = true
  app_version  = var.app_version
}

Notice that we are reading from the remote state in the assets environment of the ECR repository that was created.

# dev.tfvars

app_version="fea18f2595ef250eee020aee633666f349a13973"

Prod account

Prod account is exactly the same as the dev account.

# main.tf

data "terraform_remote_state" "assets" {
  backend = "s3"
  config = {
    bucket = "terraform-states-730335516527"
    key    = "consistent-deployments.tfstate"
    region = "us-east-1"
  }
}

module "this" {
  source       = "../modules/lambda"
  aws_region   = "us-east-1"
  environment  = "prod"
  ecr_repo_url = data.terraform_remote_state.assets.outputs.ecr_repo_url
  deploy       = true
  app_version  = var.app_version
}

And the prod.tfvars is shown below:

# dev.tfvars

app_version="fea18f2595ef250eee020aee633666f349a13973"

Notice that this value is managed by the PR created by the application CI/CD!

Final flow

So, the final flow, based on this new proposed solution is:

The application developer opens a PR on the application repository to add new features to the code
Application PR is approved and merged
The Docker image is created and pushed to ECR (tagged based on git commit hash) on the assets account
A pull request is created in the infrastructure repository to update app_version in the tfvars file to use the new image
If no infra changes are required, the application team can approve the update of the new Lambda version, otherwise, the infra team add changes to this PR
Infra PR is approved and merged
Deploy to assets account to create ECR repository (requires manual approval)
Deploy to the dev account to create a new Lambda version (requires manual approval)
Deploy to prod account to create new Lambda version (requires manual approval)

Cya

That's all! Hope this brought some insights for you about consistent deployments. Hope to see you in the next on the next post!

On-premise Kubernetes cluster management using Rancher and Terraform

Thu, 06 Jun 2024 01:40:32 GMT

If you want to go directly to the code/source code you can check my GitHub repository. Otherwise, let's start to comment about Rancher, Vagrant, OpenTofu/Terraform!

Rancher

I usually like to understand a tool by asking questions such as: what is the problem that the tool is trying to solve? How the tool can solve this problem? So let's go directly to these questions:

The problem: Managing Kubernetes at scale is complex and a real challenge. You need to manage things such as multi-cluster, standardization, security/access, and deploy applications consistently across clusters.
What is Rancher?: Rancher is an open-source platform to simplify the deployment, management and operation of Kubernetes clusters (cloud, on-prem or hybrid). It makes it easier to solve the mentioned problem because:

It acts as a centralized tool for managing all clusters
You can use GUI and CLI
Cluster provisioning is all done by Rancher, which is perfect for on-premise clusters
Provides authentication and RBAC for access/permission control
Provides built-in monitoring and logging compatible with Prometheus and Grafana

and many more!

Basic Diagram: I'm a very visual person, so even simple diagrams help me visualize things.

Deployment and Installation

Rancher can be deployed in several cloud providers (AWS, GCP, Azure, Linode, DigitalOcean...) or bare metal. You can use your preferred IaC tool (such as Terraform, Pulumi or even Ansible) to configure and deploy the environment that will have Rancher installed. In this blog post, I will use a virtual machine, provisioned via Vagrant, that will deploy Rancher using Terraform. It doesn't matter the deployment method you are going to use, you just need to make sure that Rancher can reach (network) the Kubernetes cluster that it's going to provision/manage.

Rancher can be installed in several different ways. There is no right or wrong, each one has pros and cons when compared to other methods. Let's check a few ways of doing that:

Multi-cluster: For the production environment you NEED high availability, so you should set up at least three (3) nodes when using RKE and two (2) nodes when using K3s (don't freak out about RKE or K3s, I will comment about them soon!). Using Helm as a Kubernetes Package Manager makes your life easier to install Rancher on multiple nodes on the management Kubernetes cluster that will have Rancher installed.

Single-node: As you can imagine, a single node running Rancher is good for testing, cost-saving and non-critical environments, not for production. For testing and demonstration purposes you can even install Rancher using Docker!

Ok.. Now that we know that we can deploy in a multi-cluster or single-node way (in this demo I will use a single-node approach), where we are installing Rancher? And how we are going to use Helm to install Rancher if we don't even have a running Kubernetes cluster at first? To answer this let's comment about bootstrapping.

Bootstrapping

Bootstrap is the initial process of setting up a system/environment with basic components, dependencies, and configuration. To install Rancher you will need a Kubernetes cluster. I know... this is confusing, you will need a Kubernetes cluster to install a tool that creates/manages Kubernetes clusters. And this is part of the bootstrap. You will need to create manually/script/IaC a Kubernetes cluster to run Rancher. You can use several Kubernetes engines to host Rancher: RKE (also known as RKE1), RKE2, K3s, EKS, AKS and GKE. The last three engines are cloud-specific, so if you are going for on-premise you should focus on RKE1, RKE2 or K3s.

One very important thing: It is NOT recommended to run workloads in this cluster. This cluster should be used entirely for Rancher and will serve as the control plane for managing the other clusters. This cluster is also known as the "Rancher Manager Cluster".

Vagrant

Since I don't have physical server (or a home lab) I will create virtual machines using a hypervisor. Instead of manually creating these machines using the hypervisor interface/GUI I will use Vagrant to automate this task. Once again let's target these questions:

The problem: Managing the creation, access and configuration of Virtual Machines is a tedious task that can be automated!
What is Vagrant?: Vagrant is a tool that allows us to declaratively define a manifest (called Vagrantfile) with the configuration of the VMs that we would like to create. Notice that Vagrant is not a hypervisor, it sits between the hypervisor and the user.

As commented, we still need a hypervisor. There are several hypervisors in the market. For this tutorial I decided to use VMware Fusion Pro. I could have used any other hypervisor (e.g. Virtualbox is probably the most famous one) to deploy the VMs but I think VMware Fusion is a great option because it's one of the few free (for personal use) hypervisors that you can use with M-series Macbooks and x86 architectures.

Installation

Set up Vagrant with VMware Fusion Pro integration is fairly simple but a bit tricky because it requires you to install several things. The step-by-step guide is the following:

Install Vagrant
Install VMWare Fusion Pro (it requires you to create a free account on Broadcom)
Install VMware Utility (this will provide VMware provider plugin - that we are installing next - various functionalities)
Install Vagrant VMware plugin

Configuration

To configure the VMs I will use a Vagrantfile that will consume inputs from a YAML file.

# Vagrantfile
require 'yaml'
settings = YAML.load_file(File.join(File.dirname(__FILE__), 'vagrant.yaml'))

Vagrant.configure("2") do |config|
  config.vm.box = settings['box_name']

  settings['vm'].each do |vm_config|
    config.vm.define vm_config['name'] do |vm|
      vm.vm.hostname = vm_config['name']
      vm.vm.network "private_network", ip: vm_config['ip']
      vm.vm.synced_folder ".", "/vagrant", disabled: false

      vm.vm.provider "vmware_fusion" do |vb|
        vb.memory = vm_config['memory']
        vb.cpus = vm_config['cpus']
      end
    end
  end
end

If you are not familiar with Vagrant please don't get scared. It's actually fairly easy to read this. For every input in the vagrant.yaml file we will create a VM in a private network with a given private IP and a given amount of CPU and memory. Easy, right? Now let's check the vagrant.yaml file:

box_name: "bento/ubuntu-22.04"
vm:
- name: "rancher-manager"
  ip: "192.168.110.10"
  memory: "4096"
  cpus: "1"
- name: "node01"
  ip: "192.168.110.11"
  memory: "4096"
  cpus: "1"
- name: "node02"
  ip: "192.168.110.12"
  memory: "2048"
  cpus: "1"
- name: "node03"
  ip: "192.168.110.13"
  memory: "1024"
  cpus: "1"

I'm defining the OS of the VMs (you can check all the available OS in the Vagrant Box website) and the configuration of each VM. In our case, four VMs: one for the Rancher manager cluster and others for the Kubernetes cluster that Rancher will create (one VM will assume the role of the etcd, another for the controlplane and the last one for the worker role).

Deployment

Vagrant deployment is as easy as running vagrant up. That's it. That's all.

OpenTofu (Terraform)

Once again, here we go:

The problem: I could write an entire blog post commenting about infrastructure as code (IaaC) and its benefits. To be direct to the point I will say that manually creating infrastructure is terrible: not reproducible, hard to review, prone to errors, there is no history of actions performed...
What OpenTofu is?: OpenTofu is the open-source version of Terraform (which is no longer open source since 2023). It allows us to declaratively define manifests (in JSON or HCL - a domain-specific language) it has the concepts of "blocks" to create resources.

Here we will use OpenTofu to install Rancher a configure Kubernetes nodes. Although Terraform was created to manage infrastructure it can be used to manage configuration in machines. Ansible is a more specific tool for this purpose but I think Terraform/OpenTofu can be perfectly used here.

Configuration

Let's start with the variables of the OpenTofu code:

// variables.tf
variable "k3s_version" {
  description = "K3s version that will be used in the management cluster (Rancher). Check all available versions on GitHub releases page: https://github.com/k3s-io/k3s/releases"
  type        = string
}

variable "cert_manager_version" {
  description = "Cert Manager version that will be deployed, as a Rancher dependency. Check all available versions on Github releases page: https://github.com/cert-manager/cert-manager/releases"
  type        = string
}

variable "rancher_version" {
  description = "Rancher version that will be deployed, as a Rancher dependency. Check all available versions on Github releases page: https://github.com/rancher/rancher/releases"
  type        = string
}

variable "rancher_endpoint" {
  description = "URL that points to the Rancher server"
  type        = string
}

variable "management_cluster" {
  description = "Defines the management cluster that will be created."
  type = object({
    host        = string
    user        = string
    password    = string
    port        = number
    private_key = string
  })
}

variable "managed_cluster" {
  description = "Defines all the clusters that will be created. Also defines all the nodes of the cluster."
  type = map(
    object({
      kubernetes_version = string
      nodes = list(object({
        host        = string
        user        = string
        password    = string
        port        = number
        private_key = string
        role        = list(string) # Accept values: etcd, controlplane, worker
  })) }))
}

Just a few variables: versions of K3s (I decided to use K3s instead of RKE), Cert Manager (needed to install Rancher) and Rancher. Also the definition of the Rancher endpoint and the configuration of the node that will contain the Rancher installation and the nodes that Rancher will create the managed Kubernetes cluster.

A completely valid configuration can be seen below:

// vars.tfvars
k3s_version          = "v1.28.10+k3s1"
cert_manager_version = "v1.14.5"
rancher_version      = "v2.8.4"
rancher_endpoint     = "192.168.110.10.sslip.io"
management_cluster = {
  host        = "127.0.0.1",
  user        = "vagrant"
  password    = "vagrant"
  port        = 2222
  private_key = "/Users/felipe/Desktop/folders/personal/rancher/.vagrant/machines/rancher-manager/vmware_fusion/private_key"
}
managed_cluster = {
  development = {
    kubernetes_version = "v1.28.10+rke2r1"
    nodes = [{
      host        = "127.0.0.1",
      user        = "vagrant"
      password    = "vagrant"
      port        = 2200
      private_key = "/Users/felipe/Desktop/folders/personal/rancher/.vagrant/machines/node01/vmware_fusion/private_key",
      role        = ["controlplane"]
      },{
      host        = "127.0.0.1",
      user        = "vagrant"
      password    = "vagrant"
      port        = 2201
      private_key = "/Users/felipe/Desktop/folders/personal/rancher/.vagrant/machines/node02/vmware_fusion/private_key"
      role        = ["etcd"]
      },
      {
      host        = "127.0.0.1",
      user        = "vagrant"
      password    = "vagrant"
      port        = 2202
      private_key = "/Users/felipe/Desktop/folders/personal/rancher/.vagrant/machines/node03/vmware_fusion/private_key"
      role        = ["worker"]
      }]
  }
}

PS: Make sure to edit managed_cluster.<cluster-name>.nodes[] accordingly. You can run vagrant ssh-config (after the VMs are created) to get the host, user, password, port and private_key.

The following providers will be used:

// versions.tf
terraform {
  required_version = "~> 1.7"
  required_providers {
    rancher2 = {
      source  = "rancher/rancher2"
      version = "4.1.0"
    }
  }
}

provider "local" {}

provider "rancher2" {
  alias = "bootstrap"

  api_url = "https://${var.rancher_endpoint}"
  insecure  = true
  bootstrap = true
}

provider "rancher2" {
  alias = "admin"

  api_url   = rancher2_bootstrap.this.url
  token_key = rancher2_bootstrap.this.token
  insecure  = true
}

Notice that we are using Rancher (we will use that to create a managed cluster). You might ask why two providers for Rancher. This is needed because one of the providers will deal with the bootstrapping of the Rancher (change the initial admin password, set Rancher endpoint, create and manage the rotation of the Rancher token) and the other we will use to manage Rancher itself (using the token created by the bootstrap provider).

Now the most important part: the resources created by OpenTofu.

// main.tf
locals {
  bootstrap_remote_folder_path = "/tmp/bootstrap"
  bootstrap_folder_path        = "${path.module}/bootstrap"
  rancher_install_command      = "/bin/bash ${local.bootstrap_remote_folder_path}/install-manager.sh ${var.k3s_version} ${var.cert_manager_version} ${var.rancher_version} ${local.bootstrap_remote_folder_path}"
  nodes = flatten([
    for cluster_name, cluster_value in var.managed_cluster : [
      for index, node_value in cluster_value.nodes :
      merge({ "cluster" : cluster_name }, node_value)
    ]
  ])
}

data "archive_file" "check_if_bootstrap_needs_to_run" {
  type        = "zip"
  source_dir  = local.bootstrap_folder_path
  output_path = "/tmp/k3s-bootstrap.zip"
}

resource "null_resource" "bootstrap_rancher" {
  connection {
    type        = "ssh"
    host        = var.management_cluster["host"]
    user        = var.management_cluster["user"]
    password    = var.management_cluster["password"]
    port        = var.management_cluster["port"]
    private_key = file(var.management_cluster["private_key"])
  }

  provisioner "file" {
    source      = local.bootstrap_folder_path
    destination = local.bootstrap_remote_folder_path
  }

  # We can´t have a single 'remote-exec' for running 2 commands because it will
  # consider that this resource was successful if the last inline command run fine
  provisioner "remote-exec" {
    inline = [
      local.rancher_install_command,
    ]
  }

  provisioner "remote-exec" {
    inline = [
      "rm -rf ${local.bootstrap_remote_folder_path}",
    ]
  }

  triggers = {
    script_hash     = data.archive_file.check_if_bootstrap_needs_to_run.output_sha
    install_command = md5(local.rancher_install_command)
  }
}

resource "rancher2_bootstrap" "this" {
  depends_on       = [null_resource.bootstrap_rancher]
  provider         = rancher2.bootstrap
  initial_password = "ChangedByTerraform"
  telemetry        = false
}

resource "rancher2_cluster_v2" "this" {
  for_each = var.managed_cluster

  provider           = rancher2.admin
  name               = each.key
  kubernetes_version = each.value.kubernetes_version
}

resource "null_resource" "bootstrap_node" {
  depends_on = [null_resource.bootstrap_rancher, rancher2_cluster_v2.this]
  for_each   = { for k, v in local.nodes : k => v }
  connection {
    type        = "ssh"
    host        = each.value["host"]
    user        = each.value.user
    password    = each.value.password
    port        = each.value.port
    private_key = file(each.value.private_key)
  }

  provisioner "file" {
    source      = local.bootstrap_folder_path
    destination = local.bootstrap_remote_folder_path
  }

  provisioner "remote-exec" {
    inline = [
      "/bin/bash ${local.bootstrap_remote_folder_path}/install-node.sh",
      "${rancher2_cluster_v2.this[each.value.cluster].cluster_registration_token[0].insecure_node_command} ${join(" ", [for x in each.value.role : format("--%s", x)])}",
    ]
  }

  provisioner "remote-exec" {
    inline = [
      "rm -rf ${local.bootstrap_remote_folder_path}",
    ]
  }

  triggers = {
    always = timestamp()
  }
}

Notice that the code does the following:

Copy the install-manager.sh (we will get there soon!) script to the node that will have Rancher installed and run it
Creates the managed cluster by Rancher
Copy the install-node.sh (once again, we will get there!) script to all the managed nodes and run the command to add these nodes to the newly created cluster by Rancher (once Rancher detects the nodes it will install Kubernetes in these nodes and manage everything for you!)

Now, let's comment about these two scripts:

# install-manager.sh
#!/bin/bash
set -euo pipefail

# Description: This bash script install K3s, Cert Manager and Rancher. It's supposed to
#              be executed in the node of the management cluster.

# Arguments: This bash script expects the follow arguments in order:
# 1) K3S_VERSION: Version of K3S
# 2) CERT_MANAGER_VERSION: Version of K3S
# 3) RANCHER_VERSION: Version of K3S
# 4) HELM_VALUES_RANCHER_FOLDER_PATH: Path of the folder that contains the custom "values.yaml" file for the Rancher Helm installation

if [ "$#" -ne 4 ]; then
  echo "Error: Exactly 4 arguments are required."
  echo "Usage: $0 <K3S_VERSION> <CERT_MANAGER_VERSION> <RANCHER_VERSION> <HELM_VALUES_RANCHER_FOLDER_PATH>"
  exit 1
fi

K3S_VERSION=$1
CERT_MANAGER_VERSION=$2
RANCHER_VERSION=$3
HELM_VALUES_RANCHER_FOLDER_PATH=$4

# Basic tools
sudo apt-get update
sudo apt install curl jq -y

# K3s Installation
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$K3S_VERSION sh -s - server --cluster-init
export KUBECONFIG="$HOME/.kube/config"
echo "export KUBECONFIG=$KUBECONFIG" >> $HOME/.bashrc
mkdir -p $HOME/.kube
sudo k3s kubectl config view --raw > $HOME/.kube/config
sudo chmod 600 $HOME/.kube/config

# Kubernetes
## Helm
curl -fsSL -o /tmp/get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod +x /tmp/get_helm.sh
/tmp/get_helm.sh

## Cert Manager
CERT_MANAGER_NAMESPACE=cert-manager
CERT_MANAGER_HELM_RELEASE=cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/$CERT_MANAGER_VERSION/cert-manager.crds.yaml
helm repo add jetstack https://charts.jetstack.io
helm repo update
if helm status -n $CERT_MANAGER_NAMESPACE $CERT_MANAGER_HELM_RELEASE > /dev/null 2>&1; then
  echo "Release '$CERT_MANAGER_HELM_RELEASE' exists."
  version=$(helm list -n $CERT_MANAGER_NAMESPACE -o json -a | jq -r .[].app_version)
  if [[ $version != $CERT_MANAGER_VERSION ]]; then
    echo "Upgrading release '$CERT_MANAGER_HELM_RELEASE' to version '$CERT_MANAGER_VERSION'..."
    helm upgrade $CERT_MANAGER_HELM_RELEASE jetstack/cert-manager \
      --namespace $CERT_MANAGER_NAMESPACE --version $CERT_MANAGER_VERSION
  fi
else
  echo "Release '$CERT_MANAGER_HELM_RELEASE' does not exist. Installing version '$CERT_MANAGER_VERSION'..."
  helm install $CERT_MANAGER_HELM_RELEASE jetstack/cert-manager \
    --namespace $CERT_MANAGER_NAMESPACE --version $CERT_MANAGER_VERSION --create-namespace
fi
kubectl -n $CERT_MANAGER_NAMESPACE rollout status deploy/cert-manager

## Rancher
RANCHER_NAMESPACE=cattle-system
RANCHER_HELM_RELEASE=rancher
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
if helm status -n $RANCHER_NAMESPACE $RANCHER_HELM_RELEASE > /dev/null 2>&1; then
  echo "Release '$RANCHER_HELM_RELEASE' exists."
  version=$(helm list -n $RANCHER_NAMESPACE -o json -a | jq -r .[0].app_version)
  if [[ $version != $RANCHER_VERSION ]]; then
    echo "Upgrading release '$RANCHER_HELM_RELEASE' to version '$RANCHER_VERSION'..."
    helm upgrade $RANCHER_HELM_RELEASE rancher-latest/rancher \
      --namespace $RANCHER_NAMESPACE --version $RANCHER_VERSION \
      -f $HELM_VALUES_RANCHER_FOLDER_PATH/values.yaml
  fi
else
  echo "Release '$RANCHER_HELM_RELEASE' does not exist. Installing version '$RANCHER_VERSION'..."
  helm install $RANCHER_HELM_RELEASE rancher-latest/rancher \
    --namespace $RANCHER_NAMESPACE --version $RANCHER_VERSION \
    -f $HELM_VALUES_RANCHER_FOLDER_PATH/values.yaml --create-namespace
  kubectl -n $RANCHER_NAMESPACE rollout status deploy/rancher
  sleep 30
fi
kubectl -n $RANCHER_NAMESPACE rollout status deploy/rancher

# Clean up
rm -rf HELM_VALUES_RANCHER_FOLDER_PATH

This script is tedious to read but it, as the comment in the script says, installs K3s, Cert Manager (via Helm Chart) and Rancher (via Helm Chart).

I'm using custom values for the Helm Chart.

# values.yaml
hostname: 192.168.110.10.sslip.io
replicas: 1
bootstrapPassword: ChangedByTerraform

Notice that the hostname is the static IP that we set in the vagrant.yaml file! Also, the bootstrapPassword will be modified by the bootstrap Rancher provider of OpenTofu.

Finally, the install-node.sh script!

# install-node.sh
#!/bin/bash
set -euo pipefail

# Description: This bash script applies best practices for Kubernetes nodes.

# Disable Swap (Kubernetes requirements)
sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab

# Install recommended packages
sudo apt-get update
sudo apt-get install -y curl tar

It's recommended to set the nodes with some Kubernetes configurations, such as disable swap.

Deployment

Deploying all of this using OpenTofu is as simple as running tofu init and tofu apply --var-file="vars.tfvars". After it's deployed you can simply access 192.168.110.10.sslip.io and verify the cluster deployment. The username to access the Rancher application is admin and the password can be found on the terraform.tfstate file (since the bootstrap provider modified the initial admin password).

Cya

Hope you enjoyed this blog post! This time the post was really hands-on and not so focused on the theorical part of it, but I hope it's a bit clear how you can deploy Kubernetes clusters using Rancher and OpenTofu!

See you in the next post! 👋

Configuring an email forward to enhance your phone security in case of a robbery

Sat, 04 May 2024 19:23:32 GMT

Today we use our phones for absolutely everything and to access everything, such as internet banking, social networks, email, phone calls... And this might be very risky.

In case you want to go directly to the Git repository, please check here.

The risk

I remember that I saw a piece of news about a man who had his phone robbed. The thing is that the phone was unblocked. The robber looked for bank apps installed on the phone and for every bank app the robber started the "forget password" procedure. The reset password email was sent to the email of the person who was robbed (the email account was logged into the phone) and the robber had access to his bank account, stealing hundreds of dollars.

The bank security system to reset the password must be more secure, I agree with you. But this doesn't change the fact that the robber with access to the phone and SIM card can reset several accounts of this person (bank apps, social networks, e-commerce account...).

Because of that, I created a new Gmail account that only receives the subject of the emails received by my main and secondary emails and removed the main and secondary email accounts from my phone. This way, even if I had the bad luck of getting robbed with my phone unlocked I know that at least the robber won't have access to my email, making it very hard for him to reset the password of my bank account. In fact, he will see in my phone that new emails are arriving but since it's only the title of the email (the content won't be forwarded) the robber won't receive the code or link to reset the password of my account.

Proposed Architecture

The gear icon in the diagram above is composed of several components that will be deployed in the cloud (I used AWS). I will comment on the cost of this project later (spoiler: basically free).

From time to time (cron-based) a code written in Python, using the IMAP protocol, will communicate with the main/secondary email, get the recent emails and forward only the title of these emails to the receiver email account using pub-sub pattern.

This entire infrastructure was created using AWS CDK.

CDK Code

All the infrastructure was created using a single stack:

// stack.ts
import { Duration, Stack, StackProps, Tags } from "aws-cdk-lib";
import { Construct } from "constructs";
import { DockerImageFunction, DockerImageCode } from "aws-cdk-lib/aws-lambda";
import path from "path";
import { StringParameter } from "aws-cdk-lib/aws-ssm";
import { InfraConfig } from "./config";
import { Rule, RuleTargetInput, Schedule } from "aws-cdk-lib/aws-events";
import { LambdaFunction } from "aws-cdk-lib/aws-events-targets";
import { Topic } from "aws-cdk-lib/aws-sns";
import { EmailSubscription } from "aws-cdk-lib/aws-sns-subscriptions";

export class EmailForwarderStack extends Stack {
  constructor(scope: Construct, id: string, config: InfraConfig, props?: StackProps) {
    super(scope, id, props);

    const topic = new Topic(this, "Topic", {
      topicName: "email-forwarder",
    });
    topic.addSubscription(new EmailSubscription(config.targetEmail));

    const lambda = new DockerImageFunction(this, "Lambda", {
      description:
        "Cron-based Lambda that checks new email of Gmail accounts and forward the title and sender of the email to another Gmail email",
      functionName: "email-forwarder",
      code: DockerImageCode.fromImageAsset(path.join(__dirname, "../../", "lambda")),
      environment: {
        SNS_TOPIC_ARN: topic.topicArn,
      },
      timeout: Duration.seconds(config.lambda.timeoutInSeconds),
      memorySize: config.lambda.memoryInMiB,
      deadLetterTopic: topic,
      architecture: config.lambda.architecture,
    });
    topic.grantPublish(lambda);

    for (const email of config.sourceEmail) {
      const [username, domain] = email.split("@");

      const parameterEmail = new StringParameter(this, `Email${username}`, {
        parameterName: `/email/${domain}/${username}/email`,
        stringValue: email,
      });
      parameterEmail.grantRead(lambda);
      const parameterPassword = new StringParameter(this, `Email${username}Password`, {
        parameterName: `/email/${domain}/${username}/password`,
        stringValue: "ChangeMe",
      });
      parameterPassword.grantRead(lambda);
      const parameterId = new StringParameter(this, `Email${username}LatestId`, {
        parameterName: `/email/${domain}/${username}/latest-id`,
        stringValue: "ThisWillBeManagedByTheLambda",
      });
      parameterId.grantRead(lambda);
      parameterId.grantWrite(lambda);

      const unit = config.lambda.triggerRateInMinutes == 1 ? "minute" : "minutes"
      const triggerRule = new Rule(this, `TriggerRule${username}`, {
        schedule: Schedule.expression(`rate(${config.lambda.triggerRateInMinutes} ${unit})`),
      });
      triggerRule.addTarget(
        new LambdaFunction(lambda, {
          event: RuleTargetInput.fromObject({
            parameterStore: {
              lastEmailIdRead: parameterId.parameterName,
              email: parameterEmail.parameterName,
              password: parameterPassword.parameterName,
            },
          }),
        }),
      );
    }

    Tags.of(this).add("Repository", "https://github.com/felipelaptrin/email-forwarder");
    Tags.of(this).add("ManagedBy", "CDK");
  }
}

As you can see we are deploy the following resources:

SNS Topic: The topic will be used to forward the subjects of the emails to the receiver Gmail account. So in this topic there will be a single subscriber (the receiver account).
Lambda: It's the "brain" of the project, will contain the code responsible for reading the emails from the main/secondary Gmail account and publish this message into the SNS topic. Notice that I'm using a monorepo and the application code of the lambda sits on the same repository and we are building the Docker image of the Lambda during CDK deployment.
Parameters: We are going to store the credentials to the main/secondary account and the last ID read by the Lambda (more on that later).
EventBridge Rule: This allows us to trigger the Lambda passing as parameter the name of the parameter store that has the credentials of the emails.

This code is completely generic (things are not hardcoded) and you can customize it by editing the config.ts file.

import { Architecture } from "aws-cdk-lib/aws-lambda";

export interface LambdaProps {
  timeoutInSeconds: number;
  memoryInMiB: number;
  triggerRateInMinutes: number;
  architecture: Architecture;
}

export interface InfraConfig {
  sourceEmail: string[];
  targetEmail: string;
  lambda: LambdaProps;
}

export const config: InfraConfig = {
  sourceEmail: ["felipelaptrin@gmail.com", "paotrindadepao@gmail.com"],
  targetEmail: "emailsfelipetrindade@gmail.com",
  lambda: {
    timeoutInSeconds: 60,
    memoryInMiB: 256,
    triggerRateInMinutes: 15,
    architecture: Architecture.ARM_64, // This must match your architecture (since we are building during deployment)
  },
};

Notice that you can completely modify your use case.

Lambda code

I decided to go for a more OOP-oriented code and created the following classes:

# ssm.py
import boto3
from envs import AWS_REGION

ssm = boto3.client('ssm', region_name=AWS_REGION)

class ParameterStore:

    def get_parameter(self, name: str) -> str:
        try:
            response = ssm.get_parameter(Name=name)
            value = response['Parameter']['Value']
            return value.strip()

        except Exception as e:
            print(f'Error fetching parameter from SSM Parameter Store: {str(e)}')

    def update_parameter(self, name: str, value: str) -> None:
        try:
            ssm.put_parameter(
                Name=name,
                Value=value,
                Overwrite=True
            )
        except Exception as e:
            print(f'Error trying to update parameter from SSM Parameter Store: {str(e)}')

This class is responsible for retrieving and recording data from/in the Parameter Store. I also created a class to interact with the SNS topic.

# sns.py
import boto3
from email_reader import Email
from envs import AWS_REGION, SNS_TOPIC_ARN

sns = boto3.client('sns', region_name=AWS_REGION)

class SNS:
    def send_message(email: Email, original_email: str) -> None:
        try:
            sns.publish(
                TargetArn=SNS_TOPIC_ARN,
                Subject=f"[{email.sender}] {email.title}",
                Message=f"Check email content in the {original_email} email."
            )

        except Exception as e:
            print(f'Error publish message to topic: {str(e)}')

The last class I created was responsible for managing the Email read.

# email_reader.py
import email
import imaplib
from dataclasses import dataclass
from typing import List

from ssm import ParameterStore

imap = imaplib.IMAP4_SSL("imap.gmail.com")

@dataclass
class Email:
    id: int
    title: str
    sender: str

@dataclass
class Parameters:
    lastEmailIdRead: str
    email: str
    password: str

class EmailReader:
    def __init__(self, parameters: Parameters):
        self.parameters = parameters
        self.parameter_store = ParameterStore()
        self.email = self.parameter_store.get_parameter(self.parameters.email)
        self.imap = self.login()
        self.latest_email_read = self.get_latest_email_read()


    def login(self) -> imaplib.IMAP4_SSL:
        imap = imaplib.IMAP4_SSL("imap.gmail.com")

        password = self.parameter_store.get_parameter(self.parameters.password)
        imap.login(self.email, password)

        return imap

    @staticmethod
    def __decode_str(string: str):
        """Decode email header string."""
        decoded_header = email.header.decode_header(string)
        result = []
        for decoded_part, charset in decoded_header:
            if isinstance(decoded_part, bytes):
                if charset:
                    decoded_part = decoded_part.decode(charset)
                else:
                    decoded_part = decoded_part.decode()
            result.append(decoded_part)
        return ''.join(result)

    def read_emails(self) -> List[Email]:
        imap = self.imap
        emails = []

        emails_to_read_ids = [
            str(id_as_int).encode('ascii')
            for id_as_int
            in range(self.latest_email_read+1, self.get_latest_email_receive()+1)
        ]
        print(f"There are {len(emails_to_read_ids)} to be read...")

        for email_id in emails_to_read_ids:
            _, msg_data = imap.fetch(email_id, '(RFC822)')
            imap.store(email_id, '-FLAGS', '\\Seen') # Mark as unread

            for response_part in msg_data:
                if isinstance(response_part, tuple):
                    msg = email.message_from_bytes(response_part[1])
                    sender = self.__decode_str(msg['from'].split(" <")[0])
                    read_email = Email(
                        id=email_id,
                        title= self.__decode_str(msg['subject']),
                        sender = sender,
                    )
                    print(f"Email ==> {read_email}")
                    emails.append(read_email)
        return emails

    def update_latest_email_read(self, most_recent_email_id: int):
        try:
            print("Updating parameter store ")
            self.parameter_store.update_parameter(
                self.parameters.lastEmailIdRead,
                str(most_recent_email_id)
            )
        except Exception as e:
            raise Exception(e)

    def get_latest_email_read(self) -> int:
        try:
            value = self.parameter_store.get_parameter(
                name=self.parameters.lastEmailIdRead
            )
            # The below if statement will only be executed once (first run after infra is created)
            if value == "ThisWillBeManagedByTheLambda":
                print("Bootstraping the Parameter store with ID value")
                self.update_latest_email_read(self.get_latest_email_receive())
                return self.get_latest_email_receive()
            return int(value)
        except Exception as e:
            print(e)

    def get_latest_email_receive(self) -> int:
        imap = self.imap
        imap.select('INBOX')
        _, email_ids = imap.search(None, 'ALL')
        email_ids = email_ids[0].split()

        latest_email_id = email_ids[-1]
        return int(latest_email_id.decode('ascii'))

Something that I think it's worth mentioning:

IMAP reads emails by the index of the email. So the first email your Gmail got has index 1, the second has index 2 and so on until the most recent email you have.
If we store the last read email ID (I used Parameter Store for that) we can check the most recent email ID and the last email ID read and forward only those that were not read.
I don't want you to deal with this ID thing manually, so the first time the code runs (after you deploy the infra), it will set the last email read as the last email you received (so the first time it runs it won't email you, it will only setup the Parameter Store).

And finally, the entrypoint of our lambda:

# main.py
import os

from email_reader import EmailReader, Parameters
from envs import (
    PARAMETER_STORE_EMAIL,
    PARAMETER_STORE_EMAIL_PASSWORD,
    PARAMETER_STORE_LAST_EMAIL_ID_READ,
)
from sns import SNS


def lambda_handler(event, context):
    print(f"Event ==> {event}")
    print(f"Context ==> {context}")

    reader = EmailReader(Parameters(**event["parameterStore"]))
    emails = reader.read_emails()
    if emails:
        for email in emails:
            print("Sending email...")
            SNS.send_message(email, reader.email)
        max_id = max(emails, key=lambda email: email.id).id
        reader.update_latest_email_read(int(max_id.decode('ascii')))

if(os.getenv('LOCAL_DEVELOPMENT') == 'TRUE'):
   print("---- Running the code locally ----")
   lambda_handler({
       "parameterStore": {
           "lastEmailIdRead": PARAMETER_STORE_LAST_EMAIL_ID_READ,
           "email": PARAMETER_STORE_EMAIL,
           "password": PARAMETER_STORE_EMAIL_PASSWORD,
          }
    }, None)

Now that you know how the infrastructure and code works let's check how you can implement that too.

Pricing

AWS offers a generous free tier for the services we use:

Parameter Store: Standard parameters are free. The interaction (retrieve/update value) with the parameter costs $0.05 per 10,000 interactions.
EventBridge Rule: Free for our use case.
SNS: 1,000 notifications are free, after that $2 per 100,000 notifications.
Lambda: This varies based on the memory and execution time of the Lambda. Using a Lambda of 512MB and 5s for each execution and running every 1 minute we will use around 110,000GB-s/month. The free tier is 1 million invocations per month and 400,000Gb-s of compute time. So for us, this will be free.

So this is very likely to be free for you!

Deployment

To make it more user-friendly I will do a step-by-step guide to help you setup all of this. Follow this in order!

Gmail settings

The first thing we need to do is to access your Gmail settings and make sure the IMAP is enabled (check the Forwarding and POP/IMAP tab).

After that let's create a new App Password for us to use with the IMAP.

Access Google Settings

Search for App Passwords

Create a new app password and copy the password value after creation

Make sure to do all these steps in the email accounts you want to secure, in my case I have two (so I did this for both accounts).

Infrastructure deployment

To make the development live easier I set up DevBox in the project. So you only need to have DevBox installed and run devbox shell to install all the dependencies of the project (NodeJS, Yarn...).

Install the dependencies

devbox shell

Move to the infrastructure folder

cd infrastructure

Install NodeJS dependencies

yarn

Modify the config.ts file accordingly
Deploy the infrastructure

yarn cdk deploy

Make you have AWS credentials setup in place and that your account-region is already CDK bootstraped.

Check the receiver email and accept the subscribe email from Amazon SNS

Go to the AWS console and update the parameter(s) that contain your password (e.g. /email/gmail.com/<YOUR_EMAIL>/password) to used the app password you copied.

Final result

Finally! Now we are receiving our forwarded emails as expected!

Cya!

I hope you liked this new blog post! See you around! 👋

ArgoCD beyond the basics

Fri, 26 Apr 2024 11:45:32 GMT

This blog post expects you to be familiar with the basic usage of ArgoCD. I will assume you are already familiar with the following topics:

You can demo almost this entire blog post! Go check out the ArgoCD Demo Repository.

GitOps

Since I will cover more advanced concepts of GitOps using ArgoCD I assume that you will already know what GitOps is. At the beginning of my DevOps career I thought that GitOps was just a buzzword to say: put everything under an SCM (most common is Git). Well, I was wrong that it is just a buzzword but I was partially right about the git part.

The truth is that GitOps follows four well-defined principles:

Declarative approach: You must not do things manually. All changes must be done declaratively via a manifest file (usually YAML).
All artifacts should be stored in a SCM system: Manifest must be versioned and stored in an SCM system (usually Git), which is considered the source of truth. A common practice is to have a repository only for storing these manifests (application code is placed in another repository).
The GitOps operator constantly pulls changes from the artifacts repository: Desired state and manifests are pulled from the source of the truth repository.
The GitOps operator runs a reconciliation loop: observe, diff, act: The goal is to always make sure that our system is in sync with the desired state stored under the SCM system.

This blog post will consider our GitOps infrastructure will be represented by: ArgoCD (our GitOps Operator) and the artifacts will be stored in GitHub.

App of Apps pattern

When going for the declarative setup of ArgoCD (which is something that I recommend you to do, instead of creating things manually - let's follow GitOps principles!) you can create a manifest for defining ArgoCD Applications. The ArgoCD Application will basically say "Hey ArgoCD, I have manifests for running a new app in a given format (e.g. Helm Chart, pure YAML files, Kustomize...) and they are stored under a given Git repository. Please install this application".

Some questions can show up:

Where this ArgoCD Application manifest will be stored?: Let's follow GitOps principles and store this under a Git repository.
How ArgoCD will notice that I added this ArgoCD Application manifest and it will create my application?: By default, it won't be aware. We have two common options: apply the ArgoCD Application manually or let ArgoCD automatically apply this application.
How can ArgoCD automatically apply the ArgoCD Application file?: Using the App of Apps pattern.

Imagine you want to create a new application and want it to be deployed in a Kubernetes cluster. Here is a summary of the steps that you probably will do:

Create a pull request to add the ArgoCD Application manifest to the GitHub repository
If approved by your team, merge the content to the master/main branch
Connect to the Kubernetes cluster and apply the ArgoCD Application manifest (e.g. kubectl apply -f my-application.yaml) and now ArgoCD is aware of the new application you want to install

After that, ArgoCD will fetch the manifests (Helm, Kustomize, pure YAML...) to create your application (because you pointed out where they were in the ArgoCD Application manifest) and it will apply them to be deployed in the cluster.

There is nothing wrong with the above flow but we can improve it. The fact that we need to connect to the Kubernetes cluster and manually apply commands is prone to error. We can improve that and do that automatically. One way of solving this is to add all the needed steps in the CD pipeline but I'm not a big fan of that because you are allowing an external agent (e.g. GitHub Actions runner) to connect directly to your cluster and apply manifest there, you already have a GitOps Agent (ArgoCD) so why not use it? We can use the App of Apps pattern to dynamically create new applications in the cluster. So the workflow will be something like this:

Create a pull request to add the ArgoCD Application manifest to the GitHub repository
If approved by your team, merge the content to master/main branch
App of Apps application will apply the new ArgoCD Application

Notice that the App of Apps pattern simply uses the ArgoCD Application to point to a place that contains other ArgoCD Applications. And you can use that to also organize your ArgoCD deployment in groups. You can create an App of Apps for all the applications deployed by the data team/machine learning team/devops teams...

We still have a problem here: the chicken and the egg problem! Imagine a newly created cluster. I added the App of Apps manifest to the Git repository. Nothing is going to happen because this manifest was not applied to the cluster! This means that we need to apply once (or you can add this in a bootstrap pipeline) the app of apps application manifest. So the flow would be:

Create the App of Apps ArgoCD Application
Apply the manifest in the cluster (manually/CD/bootstrap pipeline)
ArgoCD creates the ArgoCD Application (App of Apps application)
All the following ArgoCD Applications that the App of Apps application is aware (the source) will be automatically applied by it

In the end, the App of Apps pattern is just an ArgoCD Application that points to ArgoCD Applications and it looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dev-app-of-apps-applications
  namespace: argocd
spec:
  destination:
    name: in-cluster
    namespace: argocd
  source:
    path: argocd/applications/dev
    repoURL: https://github.com/felipelaptrin/argocd-blog.git
    targetRevision: main
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Some things to be aware of:

spec.source.repoURL: is the link to the Git repository (source of the truth).
spec.source.path: is the path of the Git repository that the App of Apps Application will be "aware".
spec.source.targetRevision: is the branch of the Git repository that the App of Apps Application should take into consideration.

And the ArgoCD Applications will look like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dev-dummy-app
  namespace: argocd
spec:
  destination:
    name: cluster-dev
    namespace: default
  source:
    path: apps/dummy
    repoURL: https://github.com/felipelaptrin/argocd-blog.git
    targetRevision: main
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Multi-Cluster Management

Before deep diving into multi-cluster management, it's important to understand an important concept in ArgoCD called AppProject. The AppProject specifies where (which cluster and namespace) ArgoCD Applications inside this AppProject can be deployed. ArgoCD default installation comes with an AppProject called default that allows all applications in this group to be installed in any namespaces.

With that being said, imagine that we have a multi-environment infrastructure (dev, staging and production). We want to have completely isolated environments so each environment has its own Kubernetes Cluster. How can we use ArgoCD to help us deploy these applications in each environment? There are two very common solutions for that (none of these is better than the other, they will have pros and cons - there is no silver bullet!):

Single ArgoCD per cluster: In this scenario every ArgoCD instance will deploy to its cluster, providing an extra layer of isolation but usually this will require more work time to manage.

Multi-Cluster single ArgoCD: Here we have a single ArgoCD instance deploying to multiple Kubernetes clusters, leveraging features like SSO integration and RBAC. Ideally, you would have a separate cluster only for this ArgoCD (a completely separate control plane for managing this ArgoCD). But this is expensive and people usually use the production cluster to deploy this ArgoCD instance.

These are only two common configurations that we see DevOps Engineers doing. Of course, there are several other possible configurations (e.g. multiple ArgoCD instances that control multiple deployment clusters).

Managing external clusters

If we want to go for the single ArgoCD per cluster configuration we need to create some resources on the cluster that has ArgoCD installed and the external cluster (cluster that does not have ArgoCD installed and will be managed by ArgoCD).

On the external cluster we are going to:

Create a ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: argocd-manager
  namespace: kube-system
secrets:
  - name: argocd-manager-token

Create a Secret of type kubernetes.io/service-account-token and link it with the ServiceAccount created

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  annotations:
    kubernetes.io/service-account.name: argocd-manager
  name: argocd-manager-token
  namespace: kube-system

Associate a ClusterRole with required ArgoCD permissions (you can use cluster-admin instead) using a ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd-manager-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd-manager-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argocd-manager-role
subjects:
- kind: ServiceAccount
  name: argocd-manager
  namespace: kube-system

When the secret gets created you can inspect its value (kubectl get secret argocd-manager-token -n kube-system -o yaml), because the cluster that contains ArgoCD installed will need to know the ca.crt and token value. Since these are secret values we will create a Secret and add the label argocd.argoproj.io/secret-type (with value cluster) on the management cluster, so ArgoCD will understand that this secret is related to an external cluster.

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  labels:
    argocd.argoproj.io/secret-type: cluster
  name: cluster-dev
  namespace: argocd
stringData:
  config: |
    {
      "bearerToken": "eyJhbGciOiJSUzI1NiIsImtpZCI6InNJZlBrMXkxTkVpM1EtZzBKYmdsd0JldjBQZ1JBWktxQ1BmNHRjTmstMTQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhcmdvY2QtbWFuYWdlci10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhcmdvY2QtbWFuYWdlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjI2ZTc4NDBhLTg2M2UtNDE4YS1iNzZhLTk2YjE3YzhiYmQwYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTphcmdvY2QtbWFuYWdlciJ9.UcSVUwaTMf91V7S8p-8V1DCnXcDXodvtyeXeLkWEkTYQjtDk52SNJE2xofBQhNUVKLoIivSzX5mBcFpgS2V-iLOrqv-4Y9A47OBwXx-fNWkCifquz0c7hnqilGOYW6fI9X5PrmTL4-f6VVddmPX0vYbC-o-rf-G5HXbXJUgggTTZ7BKSHApRFNov8mX6RYxjMyVnNbYGHSIDnA8TtPpiibt13onOAZULnv-jDj23PVqtkk-RvxeqvtVNkyxxF4jpQzHF9uVIvOB-GDGj7iuDXZG-_CdTTar02dA9hLTpX6cn1gDhsq6iNGEMrLJtjZUKJJedyQHytG5QDu5H-VioCg",
      "tlsClientConfig": {
        "insecure":false,
        "caData":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJYmYxMDRmNUZYLzB3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBME1qTXlNRFUxTURkYUZ3MHpOREEwTWpFeU1UQXdNRGRhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUR5RmxvZjBSN0lIWUI1cWRBVGFhNnc0eXN2MlNwekdEdk5oM3I2UnluL1FhZEtvYkUrblF3Q3BVV1gKTHJ6NVNJYlYvSWlWUGV0WEliVjBVTktEdXRxaW0zajRzaDd4cUd6MmlQeU5JU1p6N3RqUGg5YXVkcVdvbzlFcQpQcWR2eXpHd1ZncDVCYytKaW1kTGlGOVFTSXVOMnFMMTAyUjdjVHNBZ2Q3QjcwWjVuWkhhVHd1eWw4QmNHVGw4ClFla0lyZTBUYm9ndGJzaGdWQkZxejMxM0hEaDJadExSYkEzN3VHZjY0NGdVdkNnWUpQQjV1cEZrcFN5dGZodUgKUG9TSkFTYkNablNRR1IrV1d3YzJuVVZPZ0plU3BWZTZRMks0eUR5azVqTWdEc2wzVVRwL0JSTDh6aXhMTXVjeApXTkFtOEo1RWdTOVJGK2VrcFFISFdVa05SeHRQQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJTRWd3NnQ3akJ0VHpUWjhnU2xwTVpUaHJhc0xqQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQzlZcXUzRTdnOQpaZ2VkL3FTN052cmtGK05taUpKUXE3VStka3BuU0N1V3BiZVN5ZWZWaGZ6U1p6bE9sTFRsUFN0NFNnKzRqUnJRCjdrdDFsM0FHcEtJdHJob0sycUZWZktYZmMyc0RER2ovZEJTMFpUSGw5ZEJGVDNwZjMxSU9XQ1o4SFg3M080U0wKN0s1bXhIek1NYnRWbGZNZTlNckJUUVlMQzNFK0RoSy81RW80VkhCV1p2SnBDOTFlVWdSNTUyT1YwV2xtNmpTZQoyekRMc1hjVzhvQk43dytZNzYvNWUwVWJCSXpENUlEU3owZ1ZWa05od0trR3lXS3luWmljcG5kY2ltZE51SDhhCm9IQVptSEM3bytObk9weWV4dmJFcFEzRmZPa0wxODN3ckJORWo1T1dUOHFoakd2YmVSWW5RaER3MVQ2aHRFOGEKdWNKUG11UkFkK2VtCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
      }
    }
  name: cluster-dev
  server: https://cluster-dev-control-plane:6443

Some points that I would like to comment on:

bearerToken: The value that is being used is the base64 decoded value of the output of the kubectl get secret argocd-manager-token -n kube-system -o yaml command.
server: We are using two Kind clusters and since it works with docker under the hood I'm leveraging the fact that Kind creates a docker bridge network to run the clusters. That's why I can reference the dev cluster (the one that does not contain ArgoCD installed) by its name. In case you are still confused run docker inspect cluster-dev-control-plane and check the network section.
caData: That's the exact value of ca.crt field returned by the kubectl get secret argocd-manager-token -n kube-system -o yaml command.

ApplicationSet

When using an ArgoCD Application we create a single deployment in a given cluster. What would happen if we have multiple Kubernetes clusters or different namespaces? We would have to create a single ArgoCD Application for each deployment, which can be repetitive and prone to error. In these cases, we can use an ApplicationSet.

An ApplicationSet automates the creation of multiple ArgoCD Applications at once using a template. The easiest way to comprehend this is to simply see the manifest:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: common
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - env: dev
        cluster: cluster-dev
      - env: prod
        cluster: in-cluster
  template:
    metadata:
      name: '{{env}}-common-app'
      namespace: argocd
    spec:
      project: default
      destination:
        name: '{{cluster}}'
        namespace: default
      source:
        path: apps/common
        repoURL: https://github.com/felipelaptrin/argocd-blog.git
        targetRevision: feat/generator
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Notice that the spec.template section is a templated ArgoCD Application that replaces the variables with the corresponding variable in the list of generators (spec.generators). There are several generators that you can use, such as list, git, cluster, matrix, merge, pull request... I don't think it's worth it to deep dive into all of them in this blog post, in case you need more info you can directly check in the documentation.

Releases Promotion

A golden rule of DevOps is to never deploy directly to production before testing this in other environments. It's likely to have more than one environment, such as dev, staging, qa, production... Different companies will have different environments, some will have only dev and prod, others dev, staging and prod, and so on. But the important thing to understand is that first we deploy to one environment, check how it goes and only after we validate we can deploy to the following environment.

How can we organize our repository (source of truth) to be a well-organized multi-environment GitOps repository? There are three common approaches to this question:

Environment per folder: Each folder of the repository represents an environment, i.e. the dev folder contains all the manifests related to the development environment and so on. This is a really good and highly recommended approach.

Environment per repository: Each git repository represents a single environment. This is a more isolated scenario, especially when you only want a special group to be able to deploy in production. When using this approach, it's common to have only two repositories, one for non-production environments and another one for production environments (that only a special team can work with).

Environment per branch: It's very similar to Git flow. Each branch corresponds to a given environment, i.e. the develop branch deploys to the dev environment and so on. This is an approach that is NOT recommended. There are several problems using this approach such as: pull requests and merges between branches are painful and very problematic (git conflicts, unwanted changes, wrong order of changes...), having a large number of environments requires a lot of maintenance, it's easy to create environment-specific code, creating drifts between environments, this approach per branch goes against Kubernetes ecosystem (such as Helm/Kustomize).

Of course, some companies might use a combination of these approaches, but in general, these are the most commonly used ways to have a GitOps repository.

CI integration with GitOps

The CI (Continuous Integration) pipeline is usually responsible for linting, running tests, checking code coverage, building the docker image, push the docker image to a registry (such as DockerHub or EKS). How does this integrate with GitOps?

Remember one of the most important GitOps principles is that everything needs to be declaratively stored (commonly) in a Git repository. Our CI must then make modifications in our repository, i.e. committing changes. There are two common ways of achieving that: asynchronously (which is a pure GitOps way of doing it) and synchronously.

Async GitOps pipeline

Using an async GitOps pipeline means that ArgoCD Applications have auto-sync enabled, which means that if a change is detected in the manifests it will auto-apply this change (remember that the repository is considered the source of truth).

So here is the flow:

Tests run to ensure code quality
The docker image is created
The docker image is pushed to the registry (e.g. Dockerhub, ECR)
The new docker image tag is committed in the manifests repository
ArgoCD detects that there is a change in the repository and will apply the changes in the application (remember that auto-sync needs to be enabled). Notice that this is an async event.
Changes are applied in the Kubernetes cluster
Kubernetes cluster pulls new docker image from registry

Sync GitOps pipeline

Some companies rather have a more synchronous approach, where developers can see the end-to-end process of deployment (because in the previous scenario, you would have to open ArgoCD to check if a new deployment was finished). This pipeline also allows us to perform actions after the deployment is done such as running smoke tests, sending a Slack message, and executing a script (this can also be achieved using ArgoCD Posy-Sync hooks/sync waves or a pipeline tool such as Argo Workflows - more on that later).

The flow is similar to the previous one but now we monitor and run commands in ArgoCD:

Tests run to ensure code quality
The docker image is created
The docker image is pushed to the registry (e.g. Dockerhub, ECR)
The new docker image is committed in the manifests repository
Using ArgoCD CLI the CI forces the ArgoCD Application to sync (remember that auto-sync is disabled).
Changes are applied in the Kubernetes cluster
Kubernetes cluster pulls the new docker image from the registry
We monitor the status of the ArgoCD Application until it gets healthy. If it doesn't get healthy the CI pipeline fails and the DevOps team needs to troubleshoot the error.
Extra steps can be performed after a successful deployment (e.g. send a Slack message to the team to let them know that a new release was deployed).

ArgoCD Image Updater

The pipelines presented above are usually different from company to company depending on the needs of the company. Extra steps can exist and tools can change. A common example of that is to use special tools for updating the image tag of the applications such as ArgoCD Image Updater. Be aware that in the period that I'm writing this blog post (April 2024), this tool is still under active development, with possibly a lot of breaking changes from release to release and should be tested on non-critical environments.

ArgoCD Image Updater works by monitoring new image tags in the registry and committing the image tag change in the repository.

Some important things to know about this tool:

It only works integrated with ArgoCD
Your manifests need to be written using Kustomize or Helm Chart
By default, the image updater will not manage the image update unless you explicitly tell him to do (configuring labels)
You can specify image version constraints based on the application

Sync Operations

As we commented before, a common scenario is the need to run actions (e.g. database migration, sending email/message, running integration tests...) before and after the deployment. We can easily achieve this using Sync Phases and/or Sync Waves.

Hook Phases and Lifecycle

There are five (5) sync phases. They are executed in the following order:

PreSync: Executed before syncing the ArgoCD Application.
Sync: Executed at the same time as the ArgoCD Application sync.
Skip: Tells ArgoCD to not apply the manifest.
PostSync: Executed after the success (healthy state) of the sync operation.
SyncFail: Executed after the failed sync operation.

When configuring hooks we can define a deletion policy, i.e. when we want this hook to be deleted. The are three (3) deletion policies:

HookSucceeded: The hook is deleted after the hook succeeded.
HookFailed: The hook is deleted after the hook fails.
BeforeHookCreation: Any existing hook resource is deleted before the new one is created.

There is nothing "special" with the hook, it's just a Kubernetes resource that will run in a given period (sync phase) and the lifecycle is controlled by ArgoCD. We configure all of this in the annotations of the resource (metadata.annotations). Let's check one example of a hook that runs before the sync (PreSync) and should be deleted (to not run again in the future) if it ran successfully:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: database-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded

One good thing about using the argocd.argoproj.io/hook-delete-policy annotation instead of using the ttlSecondsAfterFinished (Kubernetes Job field) is that if the resource is destroyed ArgoCD won't "complain" showing a drift!

Sync Waves

Sync waves allow us to modify the order in which the resources/hooks are applied. A wave is simply a number that defines when the resource should be applied (defined by the argocd.argoproj.io/sync-wave annotation). The wave is applied from the lowest value (you can even use negative numbers) to the highest value. By default, resources and hooks have wave set to zero (0). So using sync waves we can modify the order the resources/hooks will be applied.

We can combine sync waves and hooks to achieve maximum flexibility. Imagine the following scenario: we want to run a database migration and only after the migration is done we want to send a message to Slack saying the the deployment will start. In this case, we can set a sync wave value of the database migration job lower than the slack message job (e.g. -1 and 0 or 1 and 2). You can use sync phase annotations in most Kubernetes resources, but usually, they are assigned to Kubernetes Job and/or Argo Workflows.

ArgoCD uses the following order of precedence when applying the resources:

Phase (i.e. PreSync phase happens before the Sync Phase)
The sync wave (i.e. from lowest to highest value)
By Kubernetes kind (e.g. namespace should be created first than other Kubernetes resources, followed by custom resources)
By name

After the order is established ArgoCD will apply wave by wave in order. There is a delay between the sync of waves (default is 2 seconds) that can be configured by the ARGOCD_SYNC_WAVE_DELAY environment variable.

Let's give some examples so you understand exactly how ArgoCD orders theses resources. Imagine we want to deploy an ArgoCD Application with the following resources:

Job (Slack message before deployment): Wave set to -5.
Job (database migration): Wave set to -10.
Namespace: Wave default (0).
Deployment (application): Wave default (0).
Job (Slack message after deployment): Wave set to 5.

ArgoCD would first create the namespace (ArgoCD is smart enough to understand that namespace is required to be applied first because the following jobs will be deployed on this namespace), then run the database migration job, then Slack message, then deployment and finally the slack message after deployment.

We can have more flexibility using sync waves combined with sync phases:

Job (database migration): Presync wave set to 1.
Job (Slack message before deployment): PreSync with wave set to 2.
Namespace: Wave default (0).
Deployment (application): Wave default (0).
Job (Slack message after deployment): Posy Sync with default wave (0) - because the is no order job to run in this phase.

Below you can see the above example in a YouTube video. Pay attention to the order of execution. Also, notice that resources with hooks have an anchor symbol.

Sync Windows

Sync windows define the period that you want to enable/disable sync on applications/cluster/namespace. A common example is to block deployments/sync during the weekend, allow only deployments/syncs during weekdays, or allow only deployment during late night (e.g. 4AM) to not impact users.

When defining a sync window you need to specify when the sync window starts (cron expression), how long should it last, if it should allow or deny the sync operations, the timezone, if manual syncs are still allowed... In case of having more than one sync window, the deny has always preference.

Sync windows are defined on the project level (AppProject object). Check the example below.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
spec:
  syncWindows:
  - kind: allow
    schedule: '10 1 * * *'
    duration: 1h
    applications:
    - '*-prod'
    manualSync: true
  - kind: deny
    schedule: '0 22 * * *'
    timeZone: "Europe/Amsterdam"
    duration: 1h
    namespaces:
    - default
  - kind: allow
    schedule: '0 23 * * *'
    duration: 1h
    clusters:
    - in-cluster
    - cluster1

The rules for sync windows are summarized below:

Cya!

I hope you liked this new blog post! ArgoCD is an amazing tool that allows us to achieve pure GitOps principles and it's extremely used in the DevOps world.

See you around! 👋

Hands-on OpenTelemetry with Python and NodeJS

Mon, 04 Mar 2024 16:10:32 GMT

The goal of this blog post is to demystify OpenTelemetry using applications written in Python and NodeJS (Typescript). We will check how to add telemetry (logs, metrics, traces) to our application and we will use Opentelemetry when possible.

I coded two simple APIs:

Date Service: Written in Python (using fastapi) that runs on port 3000. Expose the route /date that returns the current day, 0 for Monday, 1 for Tuesday, and it keeps going until 6 for Sunday.
Gym Service: Written in NodeJS and Typescript (using express) that runs on port 8000. Expose the route /gym that returns a JSON saying if you should or should not go to the gym based on the current day. It calls the Date Service to know what's the current day. In case you are curious the business rule applied here is simple: we should go to the gym when it's a weekday.

I will try to link everything I'm doing with the documentation so I can teach you where to look when you need to develop yourself.

Oh, and something very important, if you are not familiar with OpenTelemetry I highly recommend you to check my previous blog post that goes into detail about what OTel is, its goal, concepts and much more.

Applications

Python API: Date Service

import logging
from datetime import datetime

from fastapi import FastAPI, HTTPException

# Logger
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

# App
app = FastAPI()

# Routes
@app.get("/health")
def healthcheck():
    logger.info("Checking status of application...")
    return {
        "App Name": "Date Service",
        "status": "success",
        "message": "OK"
    }


@app.get("/date")
def get_date():
    try:
        day = datetime.today().weekday()
        logger.info(f"Successfully able to get current day ({day})")
        return {
            "status": "success",
            "date": day
        }
    except Exception as e:
        logger.error(f"Not able to get current day: {e}")
        return HTTPException(500, {
            "status": "failed",
            "message": "Something went wrong... Try again"
        })

As you can see a straightforward code. Notice that we are not logging in JSON format (this is important because in the NodeJS API, I will use JSON so we will see the differences in the following steps).

NodeJS API: Gym Service

import express from 'express'
import bodyParser from 'body-parser'
import cors from 'cors'
import { createLogger, format, transports } from 'winston'
import axios from 'axios'

// Logger
const { combine, timestamp, json } = format
const logger = createLogger({
  format: combine(
    timestamp(),
    json()
  ),
  transports: [
    new transports.Console()
  ]
})

// App
const app = express()

app.use(express.json())
app.use(cors())
app.use(bodyParser.json())
app.listen(8000)

// Routes
app.get('/health', (req, res) => {
  logger.info("Checking status of application...")
  res.send({
    "App Name": "Gym Service",
    "status": "success",
    "message": "OK"
  })
})

app.get('/gym', async (req, res) => {
  try {
    const dateResponse: any = await axios.get(
      `http://${process.env.DATE_SERVICE_HOSTNAME}:${process.env.DATE_SERVICE_PORT}/date`
    )
    logger.info("Able to get the date from date service")
    if (dateResponse.data.date === 5 || dateResponse.data.date === 6) {
      res.send({
        status: "success",
        message: "Day off! No need to go to the gym today..."
      })
      return
    } else {
      res.send({
        status: "success",
        message: "Let's work out! You need to go to the gym today..."
      })
      return
    }

  } catch (error) {
    logger.error(`Unable to get current date: ${String(error)}`)
    res.status(500).send({
      status: "failed",
      message: "Something went wrong... Try again later."
    })
    return
  }
})

A bit bigger code, when compared to the Date Service but still small and easy to understand. Notice that here we are logging in JSON format.

Deployment

We can easily deploy these applications using Docker and orchestrate this deployment locally using Docker Compose. A simple docker compose up --build will run all the things we need!

Dockerfile

For the Date Service:

FROM python:3.11.8-slim
WORKDIR /app
RUN pip install pipenv
COPY Pipfile .
COPY Pipfile.lock .
RUN pipenv install --deploy
COPY . .
CMD ["pipenv", "run", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "3000", "--no-access-log"]

For the Gym Service:

FROM node:20-alpine as build
WORKDIR /app
COPY package.json ./
RUN yarn
COPY . .
RUN yarn build


FROM node:20-alpine as runner
WORKDIR /app
COPY package.json ./
RUN yarn --prod
COPY --from=build /app/dist ./
EXPOSE 8000
CMD ["node", "index.js"]

Docker Compose

We can deploy these services in the same bridge network in a Docker Compose as follows:

version: "3"

services:
  date:
    container_name: date-service
    build:
      context: ./services/date
    networks:
      - observability
    ports:
      - 3000:3000

  gym:
    container_name: gym-service
    build:
      context: ./services/gym
    networks:
      - observability
    environment:
      DATE_SERVICE_HOSTNAME: date-service
      DATE_SERVICE_PORT: 3000
    ports:
      - 8000:8000

networks:
  observability:
    name: observability

Instrumentation

Ok, so far I only explained the applications and how they are being deployed locally. Now let's start the interesting part: instrumentation.

OpenTelemetry only exposes SDKs, APIs and specs to standardize observability. It is not part of OpenTelemetry to deal with storage and visualization (called observability backend, or simply backend). There are several tools in the market, I will point some that we can use:

There are two other categories of tools (that are not used for instrumentation) but are usually used in observability projects that I would like to mention:

Visualization: Help you visualize all the metrics, logs, and traces in a single place. It consumes (data connection) the other observability backends used. Popular open-source tools are Grafana and Kibana.
Alerts: Let developers know when issues are happening in the applications using alert tools such as AlertManager, Keep
Collectors (or Agents): These tools are used to collect the instrumentation data (logs, metrics, metrics). The idea is that your application shouldn't worry manage the retries, process (transformations), and export telemetry to the backend service, it should only expose and another tool should manage this: this is where collectors come in. Some open-source collectors are Fluentbit (for collecting logs), Logstash (for collecting logs), OpenTelemetry Collector (for collecting Logs, Metrics and Traces), Vector (Logs, metrics), and Grafana Agent.
APM Platforms: APM (Application Performance Monitoring) in simple words is just a fancy word to say "observability", commonly it's used to talk about tools/services that in a single place can help you understand how your application is performing (using traces, logs, metrics). The most famous APM platforms in the market are Datadog, New Relic, and Dynatrace. There is also an open-source APM that it's an alternative to all these paid services: SigNoz.

As you can see, there are infinite ways to set up an observability stack. The community even likes to create acronyms for popular stacks: such as ELK (Elasticsearch - Logstash - Kibana).

For this blog post, I chose to use the following backends:

Collector: OpenTelemetry Collector (as a collector)
Logs Collector: Fluentbit
Logs: Grafana Loki
Metrics: Prometheus
Traces: Grafana Tempo
Visualization: Grafana

So without further ado, let's start!

Notice that we are using Fluentbit to collect logs of the applications. Why? Because at the moment I'm writing this blog post the SDK status for Python is in an experimental state and the Javascript is under development. Although the SDK spec is stable, the open-source community is still implementing the spec in the programming languages. I could have used OpenTelemetry to export logs in Python but I wanted to apply the same collect strategy for the NodeJS application, which is not possible at the moment I'm writing this blog post.

Logging

Actually, we have already instrumented our code with logs, but now we need to set up the infrastructure to forward them to the backend. This will be done in three (3) steps: deploy the logs backend (Grafana Loki), deploy fluentbit for collecting the logs and configure Grafana to visualize these logs.

Grafana Loki

Loki can be deployed in several modes. For simplicity, we will deploy in the monolithic mode using a local configuration.

loki:
  image: grafana/loki:2.9.0
  container_name: loki
  networks:
    - observability
  ports:
    - 3100:3100
    - 7946:7946
    - 9095:9095
  command:
    - -config.file=/etc/loki/local-config.yaml
    - -print-config-stderr=true
  healthcheck:
    test: wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1
    interval: 15s
    timeout: 5s
    retries: 5

Fluentbit

Currently, Date service is logging to STDOUT in a non-serialized format (not JSON) and Gym service in JSON format to STDOUT.

To collect logs with Fluentbit we will need to:

Deploy Fluentbit

This can be easily achievable by adding a new service to our Docker Compose file.

fluent-bit:
  image: fluent/fluent-bit:2.2.0
  container_name: fluent-bit
  networks:
    - observability
  volumes:
    - ./config/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    - ./config/fluent-bit-parsers.conf:/fluent-bit/etc/fluent-bit-parsers.conf
  ports:
    - "24224:24224/tcp"
    - "24224:24224/udp"
  command: -c /fluent-bit/etc/fluent-bit.conf -log.level=debug

Notice that the fluent-bit.conf and fluent-bit-parsers.conf are two configuration files for the fluent-bit. I will comment about these files in the next section.

Probably the most common way to collect logs with Fluentbit is to read logs from a file. Even if your application is configured to send logs to STDOUT, depending on where your application is running, the tool that manages the deployment (e.g. Docker and Kubernetes) will get the STDOUT logs and dump them in a file, so you can read this file using fluentbit.

I'm using a Mac and the Docker logs folder of the containers is not available easily (in Linux you could easily do this by collecting the log files inside the /var/lib/docker/containers folder). So instead of reading a file, I will configure the logs of the Docker to use fluentbit as the logging driver. Even though fluentbit is not present in the list of logging drivers we can use the fluentd logging driver. This works because both projects are very similar, the main differences are the fact that fluentbit is written only in C (instead of C and Ruby for fluentd) and it uses way less memory. So we can configure fluentd driver with the following configuration (this configuration will be added to Date and Gym Service):

logging:
  driver: fluentd
  options: localhost:24224
  fluentd-async: true

Configure Fluentbit pipeline

Fluentbit configuration of pipelines can be done via the classic way (conf files) or the "modern" way (YAML files), and I went for the first approach.

Let's first create a dummy pipeline to inspect how the logs will be seen by fluentbit:

# fluent-bit.conf

# https://docs.fluentbit.io/manual/pipeline/inputs/forward
[INPUTS]
  Name            forward
  Listen          0.0.0.0
  Port            24224

# https://docs.fluentbit.io/manual/pipeline/filters/standard-output
[OUTPUT]
  name            stdout
  match           *

We got the following sample logs:

Date Service:

[[1709125948.000000000, {}], {"container_name"=>"/date-service", "source"=>"stderr", "log"=>"2024-02-28 13:12:28,235 - INFO - Successfully able to get current day (2)", "container_id"=>"a58d767d02d37feaf384733808e879a4980c7cce6ac8524addacb58ad4e09253"}]

Gym Service:

[[1709125948.000000000, {}], {"source"=>"stdout", "log"=>"{"level":"info","message":"Able to get the date from date service","timestamp":"2024-02-28T13:12:28.241Z"}", "container_id"=>"c402b0f01f439e6d76b94b71757894f974dc43ab3250907dbd930c90f26389a3", "container_name"=>"/gym-service"}]

We can clearly see the Fluentbit event format in these logs: [[TIMESTAMP, METADATA], MESSAGE]. Notice that the message contains some keys to identify where the logs came from, such as container_id and container_name. Also, notice that the log key is the real log message that we were expecting. We need to transform the message into more structured (correctly parse it) data to send to the backend and would also need to forward these logs to the Loki backend.

# fluent-bit.conf

# https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file#config_section
[SERVICE]
  Parsers_File   /fluent-bit/etc/fluent-bit-parsers.conf

# https://docs.fluentbit.io/manual/pipeline/inputs/forward
[INPUTS]
  Name           forward
  Tag            docker
  Listen         0.0.0.0
  Port           24224

# https://docs.fluentbit.io/manual/pipeline/filters/parser
[FILTER]
  Name           parser
  Match          docker
  Key_Name       log
  Parser         date-service
  Reserve_Data   True

[FILTER]
  Name           parser
  Match          docker
  Key_Name       log
  Parser         gym-service
  Reserve_Data   True

# https://docs.fluentbit.io/manual/pipeline/filters/standard-output
[OUTPUT]
  name           stdout
  match          docker

# https://docs.fluentbit.io/manual/pipeline/outputs/loki
[OUTPUT]
  name loki
  match *
  host loki
  port 3100
  labels collector=fluentbit, container_name=$container_name

# fluent-bit-parsers.conf

# https://docs.fluentbit.io/manual/v/1.3/parser/json
[PARSER]
  Name           gym-service
  Format         json
  Time_Key       timestamp
  Time_Format    %Y-%m-%dT%H:%M:%S %z

# https://docs.fluentbit.io/manual/pipeline/parsers/regular-expression
[PARSER]
  Name           date-service
  Format         regex
  Regex          ^(?<time>[^ ]*) - (?<level>[^ ]*) - (?<message>.*)$
  Time_Key       time
  Time_Format    %Y-%m-%d %H:%M:%S
  Time_Keep      On

Now we can see in fluentbit the following events:

Date Service:

[[1709130293.000000000, {}], {"container_name"=>"/date-service", "source"=>"stderr", "log"=>"2024-02-28 14:24:53,777 - INFO - Successfully able to get current day (2)", "container_id"=>"b2d627623047133424981c15f67381c66c57f661f6869f60b7b0241e71500e59"}]

Gym Service:

[[1709130293.000000000, {}], {"level"=>"info", "message"=>"Able to get the date from date service", "timestamp"=>"2024-02-28T14:24:53.782Z", "container_id"=>"c402b0f01f439e6d76b94b71757894f974dc43ab3250907dbd930c90f26389a3", "container_name"=>"/gym-service", "source"=>"stdout"}]

Good job! 😀

Grafana

As you probably expect... we will also deploy Grafana via Docker Compose. We are going to deploy Grafana and automatically set up a data source to point to Loki.

grafana:
  image: grafana/grafana:10.2.1
  container_name: grafana
  networks:
    - observability
  environment:
    - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
    - GF_AUTH_ANONYMOUS_ENABLED=true
    - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
  ports:
    - "3001:3000" # Using 3001 because 3000 is already being used by Date Service
  volumes:
    - ./config/grafana.yaml:/etc/grafana/provisioning/datasources/ds.yaml
  entrypoint:
    - sh
    - -euc
    - /run.sh

Where the config file is the data source file pointing to loki:

apiVersion: 1
datasources:
  # https://grafana.com/docs/grafana/latest/datasources/loki/#provision-the-loki-data-source
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    basicAuth: false

Now you should be seeing logs from Grafana in the explorer page.

Traces

Traces SDKs are stable for Python and Javascript so we will definitely use OpenTelemetry here! We can divide this into five (5) steps: deploy trace backend (Grafana Tempo), add Tempo data source to Grafana, deploy OpenTelemetry Collector, instrument Date service, and instrument Gym service.

Grafana Tempo

We are going to deploy Grafana Tempo in the monolithic mode and configure some things such as: allowing it to receive traces via OTLP (via HTTP) and configuring Grafana Tempo storage to be local (you don't want to do that in production, I'm just using a simple configuration since Tempo configuration is not the goal of this blog post).

tempo:
  image: grafana/tempo:2.3.1
  container_name: tempo
  networks:
    - observability
  ports:
    - 3200:3200
    - 4318:4318 # OTLP http receiver
  volumes:
    - ./config/tempo.yaml:/etc/tempo/config.yaml
  command: -config.file=/etc/tempo/config.yaml

And tempo.yaml is the configuration file for Grafana Tempo:

stream_over_http_enabled: true

# https://grafana.com/docs/tempo/latest/configuration/#server
server:
  http_listen_port: 3200
  log_level: info

# https://grafana.com/docs/tempo/latest/configuration/#query-frontend
query_frontend:
  search:
    duration_slo: 5s
  trace_by_id:
    duration_slo: 5s

# https://grafana.com/docs/tempo/latest/configuration/#distributor
distributor:
  receivers:
    otlp:
      protocols:
        http:

# https://grafana.com/docs/tempo/latest/configuration/#storage
storage:
  trace:
    backend: local
    wal:
      path: /tmp/tempo/wal             # where to store the the wal locally
    local:
      path: /tmp/tempo/blocks

Grafana Tempo data source

To add a new data source, as we saw earlier, we should add a new entry in the grafana.yaml file:

apiVersion: 1
datasources:
  # https://grafana.com/docs/grafana/latest/datasources/loki/#provision-the-loki-data-source
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    basicAuth: false
  # https://grafana.com/docs/grafana/latest/datasources/tempo/configure-tempo-data-source/#provision-the-data-source
  - name: Tempo
    type: tempo
    url: http://tempo:3200

OpenTelemetry Collector

We can deploy the OTel Collector in Docker Compose.

otel-collector:
    image: otel/opentelemetry-collector:0.95.0
    container_name: otel-collector
    command: [ "--config=/etc/otel-collector.yaml" ]
    networks:
      - observability
    volumes:
      - ./config/otel-collector.yaml:/etc/otel-collector.yaml

The otel-collector.yaml contains the OTel Collector configuration. The idea is that we want to receive data from the applications (Date and Gym service) via HTTP using OTLP protocol (there are other protocols, such as gRPC), and then we want to forward these traces to Grafana Tempo.

# https://opentelemetry.io/docs/collector/configuration/#receivers
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

# https://opentelemetry.io/docs/collector/configuration/#exporters
exporters:
  otlphttp:
    endpoint: http://tempo:4318

# https://opentelemetry.io/docs/collector/configuration/#pipelines
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]

Date Service (Python)

Finally! Now that all the infrastructure is in place we can start to work on the application. As I promised I will instrument the code automatically and manually. Let's start with the automatic instrumentation for Python.

We need to install the packages opentelemetry-distro (installs API, SDK and Bootstrap packages) and opentelemetry-exporter-otlp. We will also need to add other packages to allow auto instrumentation of the packages we are using in our project. We could check these packages in the registry and install them or simply run opentelemetry-bootstrap -a install`, which will automatically check our dependencies and install these packages.

We need to run this Opentelemetry auto instrumentation agent with our application so instead of running uvicorn main:app to run the application we will need to call the agent first opentelemetry-instrument uvicorn main:app. So our new Dockefile will be (just modified the CMD):

FROM python:3.11.8-slim
WORKDIR /app
RUN pip install pipenv
COPY Pipfile .
COPY Pipfile.lock .
RUN pipenv install --deploy
COPY . .
CMD ["pipenv", "run", "opentelemetry-instrument", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "3000", "--no-access-log"]

We should also configure the name of the service in the OpenTelemetry SDK by configuring the environment variables OTEL_SERVICE_NAME. We should also configure the application to send data to the collector by configuring the OTEL_TRACES_EXPORTER (by default it's already OTLP, which is the exporter that we are going to use to communicate with the OTel Collector), the OTEL_EXPORTER_OTLP_PROTOCOL (to specify that the protocol used is HTTP), and the OTEL_EXPORTER_OTLP_ENDPOINT (configuration of the OTLP exporter).

# docker-compose.yaml (inside date-service)
environment:
  OTEL_SERVICE_NAME: date-service
  OTEL_TRACES_EXPORTER: otlp
  OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318

Now you can see traces in the explorer session in Grafana. Now let's manually instrument the Python application.

import logging
from datetime import datetime

from fastapi import FastAPI, HTTPException
from opentelemetry import trace

# Logger
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

# Opentelemetry
tracer = trace.get_tracer("diceroller.tracer")

# App
app = FastAPI()

# Routes
@app.get("/health")
def healthcheck():
    logger.info("Checking status of application...")
    return {
        "App Name": "Date Service",
        "status": "success",
        "message": "OK"
    }


@app.get("/date")
def get_date():
    with tracer.start_as_current_span("date") as span:
        try:
            day = datetime.today().weekday()
            span.set_attribute("date.value", day)
            logger.info(f"Successfully able to get current day ({day})")
            return {
                "status": "success",
                "date": day
            }
        except Exception as e:
            logger.error(f"Not able to get current day: {e}")
            span.set_attribute("date.value", None)
            return HTTPException(500, {
                "status": "failed",
                "message": "Something went wrong... Try again"
            })

Now we can see this new span from Grafana:

Gym Service (NodeJS)

Here, we are going to do the same thing we did in the Python service above: add automatic instrumentation and add manual instrumentation.

For the automatic instrumentation, we need to install @opentelemetry/api and @opentelemetry/auto-instrumentations-node. To auto instrument express framework we will install @opentelemetry/instrumentation-http and @opentelemetry/instrumentation-express. To run the auto instrumentation agent we will also need to modify the command that starts the application to add the flag --require @opentelemetry/auto-instrumentations-node/register. This means that our new Dockerfile will be:

FROM node:20-alpine as build

WORKDIR /app
COPY package.json ./
RUN yarn
COPY . .
RUN yarn build


FROM node:20-alpine as runner

WORKDIR /app
COPY package.json ./
RUN yarn --prod
COPY --from=build /app/dist ./
EXPOSE 8000
CMD ["node", "--require", "@opentelemetry/auto-instrumentations-node/register", "index.js"]

The environment variables are also needed:

# docker-compose.yaml (inside gym-service)
environment:
  OTEL_SERVICE_NAME: gym-service
  OTEL_TRACES_EXPORTER: otlp
  OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318

Now let's manually instrument the application. The following packages need to be installed: @opentelemetry/sdk-node, @opentelemetry/api, @opentelemetry/sdk-trace-node. And since we want to export the traces via OTLP we will also need to install @opentelemetry/exporter-trace-otlp-proto to use the OTLP exporter to the OpenTelemetry Collector. We can add a new span to the trace manually:

import express from 'express'
import bodyParser from 'body-parser'
import cors from 'cors'
import { createLogger, format, transports } from 'winston'
import axios from 'axios'
import { trace } from '@opentelemetry/api'

// Logger
const { combine, timestamp, json } = format
const logger = createLogger({
  format: combine(
    timestamp(),
    json()
  ),
  transports: [
    new transports.Console()
  ]
})

// OpenTelemetry
const tracer = trace.getTracer('gym.tracer')

// App
const app = express()

app.use(express.json())
app.use(cors())
app.use(bodyParser.json())
app.listen(8000)

// Routes
app.get('/health', (req, res) => {
  logger.info("Checking status of application...")
  res.send({
    "App Name": "Gym Service",
    "status": "success",
    "message": "OK"
  })
})

app.get('/gym', async (req, res) => {
  const span = tracer.startSpan('gym')
  try {
    const dateResponse: any = await axios.get(
      `http://${process.env.DATE_SERVICE_HOSTNAME}:${process.env.DATE_SERVICE_PORT}/date`
    )
    logger.info("Able to get the date from date service")
    if (dateResponse.data.date === 5 || dateResponse.data.date === 6) {
      span.setAttribute("gym.shouldGo", false)
      res.send({
        status: "success",
        message: "Day off! No need to go to the gym today..."
      })
      return
    } else {
      span.setAttribute("gym.shouldGo", true)
      res.send({
        status: "success",
        message: "Let's work out! You need to go to the gym today..."
      })
      return
    }

  } catch (error) {
    logger.error(`Unable to get current date: ${String(error)}`)
    res.status(500).send({
      status: "failed",
      message: "Something went wrong... Try again later."
    })
    return
  } finally {
    span.end()
  }
})

Now, when checking Grafana we can see two cool things: a new span being added manually and the full trace showing spans of our two services!

Metrics

Metrics SDKs are also stable for Python and Javascript. Just like what we did in the traces section, we are going to perform five (5) steps: deploy metric backend (Prometheus), add Prometheus data source to Grafana, configure the OpenTelemetry Collector, instrument Date service, and instrument Gym service.

Prometheus

Prometheus deployment will be done via Docker Compose and it's very easy for basic configuration:

prometheus:
  image: prom/prometheus:v2.50.0
  container_name: prometheus
  networks:
    - observability
  ports:
    - 9090:9090
  volumes:
    - ./config/prometheus.yaml:/etc/prometheus/config.yaml
  command: --config.file=/etc/prometheus/config.yaml --enable-feature=otlp-write-receiver

Prometheus is best used as a pull-based system, which means, he is best used when it goes directly to the applications and collects the metrics instead of the system/collector sending the metrics to Prometheus. But, since we want to use the OTLP protocol we can configure the OTLP Receiver in Prometheus, allowing it to receive metrics via an HTTP request (this is considered an inefficient way of ingesting metrics).

The config file used was simple, just to allow Prometheus to auto-collect its logs.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
        - localhost:9090

OpenTelemetry Collector

The OpenTelemetry Collector needs to be configured to forward metrics to the OTLP endpoint of Prometheus. This is easily done by editing the Otel Collector configuration file:

# https://opentelemetry.io/docs/collector/configuration/#receivers
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

# https://opentelemetry.io/docs/collector/configuration/#exporters
exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318

  otlphttp/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp

# https://opentelemetry.io/docs/collector/configuration/#pipelines
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/tempo]
    metrics:
      receivers: [otlp]
      exporters: [otlphttp/prometheus]

Notice that we can use several exporters/receivers, if we want to use the same type of exporter/receiver we just need the /<NAME> suffix. The <NAME> can be anything you want: a number, or a name... but I used the name of the service to be more meaningful and easy to understand.

Date Service (Python)

Adding a metric is as easy as adding a trace, but before adding the metric we need to properly configure the application to export the metrics correctly via OTLP to the OpenTelemetry Collect with the following environment variables:

# docker-compose.yaml (inside date-service)
environment:
  OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: http/protobuf

Now, let's manually create the metrics in our application. It will be similar to what we did in the traces session: get a meter, create a metric instrument (I chose a counter to count the number of requests the /date endpoint received), modify the metric instrument (adding one to the metric when the /date endpoint is called).

import logging
from datetime import datetime

from fastapi import FastAPI, HTTPException
from opentelemetry import metrics, trace

# Logger
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

# Opentelemetry
tracer = trace.get_tracer("date.tracer")
meter = metrics.get_meter("date.meter")

date_counter = meter.create_counter(
    "requests_counter_get_date",
    description="Counts the number of requests that the /date endpoint received"
)

# App
app = FastAPI()

# Routes
@app.get("/health")
def healthcheck():
    logger.info("Checking status of application...")
    return {
        "App Name": "Date Service",
        "status": "success",
        "message": "OK"
    }


@app.get("/date")
def get_date():
    date_counter.add(1)
    with tracer.start_as_current_span("date") as span:
        try:
            day = datetime.today().weekday()
            span.set_attribute("date.value", day)
            logger.info(f"Successfully able to get current day ({day})")
            return {
                "status": "success",
                "date": day
            }
        except Exception as e:
            logger.error(f"Not able to get current day: {e}")
            span.set_attribute("date.value", None)
            return HTTPException(500, {
                "status": "failed",
                "message": "Something went wrong... Try again"
            })

If we deploy the stack using Docker Compose and make some requests to localhost:3000/date we will see the metric in Grafana, as you can see below:

Gym Service (NodeJS)

As you can already tell, we are going to perform the exact same things: add the environment variable, create a meter, create a metric instrument, and modify the metric instrument.

# docker-compose.yaml (inside gym-service)
environment:
  OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: http/protobuf

Before adding the code we will also need to add @opentelemetry/sdk-metrics package.

import express from 'express'
import bodyParser from 'body-parser'
import cors from 'cors'
import { createLogger, format, transports } from 'winston'
import axios from 'axios'
import { trace, metrics } from '@opentelemetry/api'
import {
  MeterProvider,
  PeriodicExportingMetricReader
} from '@opentelemetry/sdk-metrics'
import { Resource } from '@opentelemetry/resources'
import opentelemetry from '@opentelemetry/api'
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto'
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions'
// Logger
const { combine, timestamp, json } = format
const logger = createLogger({
  format: combine(
    timestamp(),
    json()
  ),
  transports: [
    new transports.Console()
  ]
})

// OpenTelemetry
const myServiceMeterProvider = new MeterProvider({
  resource: Resource.default().merge(
    new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: process.env.OTEL_SERVICE_NAME
    })
  ),
  readers: [new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
  })],
})

// Set this MeterProvider to be global to the app being instrumented.
opentelemetry.metrics.setGlobalMeterProvider(myServiceMeterProvider)

const tracer = trace.getTracer('gym.tracer')
const meter = metrics.getMeter('gym.meter')
const gymCounter = meter.createCounter("requests_counter_get_gym", {
  description: "Counts the number of requests that the /gym endpoint received"
})

// App
const app = express()

app.use(express.json())
app.use(cors())
app.use(bodyParser.json())
app.listen(8000)

// Routes
app.get('/health', (req, res) => {
  logger.info("Checking status of application...")
  res.send({
    "App Name": "Gym Service",
    "status": "success",
    "message": "OK"
  })
})

app.get('/gym', async (req, res) => {
  const span = tracer.startSpan('gym')
  gymCounter.add(1)
  try {
    const dateResponse: any = await axios.get(
      `http://${process.env.DATE_SERVICE_HOSTNAME}:${process.env.DATE_SERVICE_PORT}/date`
    )
    logger.info("Able to get the date from date service")
    if (dateResponse.data.date === 5 || dateResponse.data.date === 6) {
      span.setAttribute("gym.shouldGo", false)
      res.send({
        status: "success",
        message: "Day off! No need to go to the gym today..."
      })
      return
    } else {
      span.setAttribute("gym.shouldGo", true)
      res.send({
        status: "success",
        message: "Let's work out! You need to go to the gym today..."
      })
      return
    }

  } catch (error) {
    logger.error(`Unable to get current date: ${String(error)}`)
    res.status(500).send({
      status: "failed",
      message: "Something went wrong... Try again later."
    })
    return
  } finally {
    span.end()
  }
})

And we can visualize this metric in Grafana.

Cya!

I hope you liked this new blog post! OpenTelemetry is the hype topic of the moment about observability and the hype is well deserved!

You can check the entire code repository or check the individual PR below:

See you around! 👋

Get started with OpenTelemetry right now!

Fri, 23 Feb 2024 10:40:32 GMT

OpenTelemetry (as known as OTel) is an open-source project (maintained by CNCF, the same foundation that maintains Kubernetes) that aims to help developers instrument their code in a more standardized way. This project is a merge of two other projects: OpenTracing and OpenCensus. It provides several components, most notably:

APIs and SDK: per programming language for generating and emitting telemetry
Collector: a component that receives, processes and export telemetry to the observability backend
OTLP: a protocol for transmitting telemetry data

This documentation was divided into the following parts (in order):

Concepts: Some frequently used terms in observability and OTel
Signals: We will talk about the main components (traces, metrics, logs) used to make an application observable
Instrumentation: Here we will cover what needs to be done in your application to emit signals
OpenTelemetry: Architecture, reasons to exist, components...

Concepts

OTel requires you to know several important words used in observability, the most important ones for you to know are:

Observability: Observability lets us observe our system, making it easier to answer the question "Why is this happening?". It's very helpful to troubleshoot and handle problems. To have observability your code must be instrumented.
Observability Backend: OTel does not provide storage and visualization for collected telemetry data, that's where an observability backend is needed. Examples of backends: Jaeger, Prometheus, Zipkin, Loki.
Instrumentation: The application code emits signals such as logs, metrics and traces.
Signals: Categories of telemetry supported. At the moment it supports traces, metrics, logs and baggage.
Logs: Timestamped messages that help you understand at a code level what happened. Since they lack contextual information (such as where they were called from in a distributed system), they are not extremely useful for tracking code alone but are way more useful when included as part of a span.
Span: Represents a unit of work or operation such as a request made, or query made to the database. It contains name, log message and attributes.
Trace: It's the "path" or "route" made by requests that propagate among services. Traces are what improve visibility in distributed systems/microservice architectures. Traces are composed of one or more spans.
Metrics: Aggregations over time about your infrastructure/application. It can be related to the infrastructure itself, like CPU usage, memory usage, and requests per second or it can be related to the business, like sales per second.

Signals

Traces

Traces are super important to understand the full "path" that a request takes in your application, especially if you are using a microservice, mesh or multiple-service architecture. Technically, a trace is just a collection of JSONs (each one represents a span) where each JSON contains a name, log, attributes and timestamps. You can think of spans as structured logs on steroids (with context).

Example of a span JSON:

{
    "name": "Hello-Salutations",
    "context": {
        "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df2",
        "span_id": "0x93564f51e1abe1c2",
    },
    "parent_id": "0x051581bf3cb55c13",
    "start_time": "2022-04-29T18:52:58.114492Z",
    "end_time": "2022-04-29T18:52:58.114631Z",
    "attributes": {
        "http.route": "/cart",
        "http.server_name": "frontend",
        "http.scheme": "http",
        "http.status_code": "200",
        "net.peer.port": "10243"
    },
    "events": [
        {
            "name": "hey there!",
            "timestamp": "2022-04-29T18:52:58.114561Z",
            "attributes": {
                "event_attributes": 1
            }
        }
    ],
}

The "magic" that allows us to trace the requests is the context and parent_id field. With these fields, the backend tracing tool can organize the spans and create the full trace of the request. Some points about this:

If the parent_id is empty it means that it is the initial span (root)
If two spans have the same trace_id it means they belong to the same trace
One span is a child of another span if the parent_id of the first span is the same as the span_id of the latter.

Traces should be stored in a proper backend, popular open-source tools for that are Jaeger, Zipkin, Grafana Tempo and SigNoz. Some of these tools are just the backend for storing the traces and do not provide a graphical interface to see the traces. In these scenarios, you can always use a dashboard tool that can integrate with the backend to display the traces, a popular tool for that is Grafana.

A visual example of a trace can be seen below (please click on the picture to open in a separate tab and be able to visualize the image properly):

In the above example, you can see a trace of a POST HTTP request in the /poll endpoint. The request took 42.04ms and is composed of 130 spans (you would see the rest if you scroll down the UI of the trace). You have complete visibility of how long each process took and can tackle bottlenecks and understand the behavior of the request. One of the last traces there is a query to insert data to the database, let's check what this span looks like:

The span contains several useful pieces of information to help us debug and understand of what is going on: query made, the user used to connect to the database, how long the query took...

Span telemetry components

Four parts are necessary to instrument our code to use traces:

Tracer: Creates spans containing more information about what's happening for a given operation (e.g. query to the database). They are created by a tracer provider.
Tracer Provider: It's the factory that creates a Tracer. Usually, it's a class that needs to be instantiated once but in some language SDKs, a global tracer is already initialized automatically.
Trace Exporter: Send traces to a consumer such as STDOUT, OTel Collector or open source/vendor backend.
Trace Context: As we mentioned, this is what makes it possible to correlate spans no matter where they were generated, enabling distributed tracing. A context (the context object in the span example) contains IDs that are related to the service that called/calls the request. Each context is associated with a span!

Span Information

Span contains the following information:

Name
Parent ID (empty for root spans)
Start and End timestamps
Context
Attributes (key-value pairs containing metadata)
Events (logs about a meaningful event that happened during the span execution)
Links (way to associate spans with one another)
Status (set when there is a known error in the app code, accept values Unset, Ok and Error)
Kind (accept values Client, Server, Internal, Producer or Consumer)

Metrics

A metric is a measurement of a service in a given time that is extremely useful for monitoring the performance and availability of applications. OTel measurements are captured by metric instruments that are defined by:

Name
Kind: Here are the following types of metric instruments:
- Counter: A value that accumulates over time and only goes up. E.g. Count the number of HTTP requests with 5XX status code
- Asynchronous Counter: An asynchronous counter
- UpDownCounter: A value that accumulates over time but can also decrease. E.g. Number of items in a queue, Number of active requests
- Asynchronous UpDownCounter: An asynchronous updowncounter
- Gauge: Measures a current value at the time it is read. Notice that this is similar to UpDownCounter but the difference is that here you don't care about the rate of change. E.g. Amount of memory being used
- Histogram: Aggregation of values, useful when you are interested in statistic values. E.g. Request latency/duration
Unit (Optional)
Description (Optional)

Metric telemetry components

The instrument our code to support metrics the following components are required (you will note a clear similarity with the Span components):

Meter: Creates metrics instruments, capturing measurements about the service at runtime (e.g. CPU usage). It is created by a Meter Provider.
Meter Provider: Factory to create Meter. Usually, it's a class that needs to be instantiated once but in some language SDKs, a global meter provider is already initialized automatically.
Metric Exporter: Send metrics to a consumer such as STDOUT, OTel Collector or open source/vendor backend.

Logs

Logs are just timestamped text records with metadata. It's recommended to be structured (like a JSON) but it can be unstructured. They may also be attached to spans for extra visibility. The rule is: if it's not part of a trace or is not a metric then it is a log. A sample log (not in a JSON format):

I, [2021-02-23T13:26:23.505892 #22473]  INFO -- : [6459ffe1-ea53-4044-aaa3-bf902868f730] Started GET "/" for ::1 at 2021-02-23 13:26:23 -0800

Another example of a standardized log (JSON format) is:

{
  "timestamp": "2024-02-23 12:58:30.949",
  "level": "info",
  "message": "Creating a new poll",
  "container_name": "/poll",
  "source": "stdout",
  "container_id": "21aa4fa96624ee698e319e7e8b6571de58"
}

OTel approach to logs is a bit different than the approach to traces or metrics. Because logging solutions are widespread in all programming languages, OTel will be "hidden" behind these logging solutions. In other words, as a developer, you will choose your favorite logger library (it needs to be compatible with OpenTelemetry) and configure it to use a log appender/bridge to emit logs to OpenTelemetry LogRecordExporter. All the log components, such as Logger, Logger Provider and Log Record Exporter will be managed by the logger library that you use.

Baggage

Allow a way to propagate information across traces/signals. Since spans are immutable once created and can be exported before you need information on them you can use baggage to solve this problem. It should be used for storing non-sensitive data (it will be passed in HTTP headers). Common use cases are for account identification, user IDs, product IDs, and origin IPs. Baggage is not the same as span attributes!

Instrumentation

Once again: we need to instrument our code to emit signals and have observability in the system... but how can we do that? We can do that in two ways: automatic or manual. In the automatic way you don't even need to touch the code base of your application, while manually you will need to change it.

At first sight, the automatic way looks way better, this is not true, it's just different. Of course, there is a trade-off but in the manual configuration, you can have a way more granular approach, which can help with your observability. For most languages you can use both approaches at the same time, gaining direct insights from automatic and granular observability with the manual instrumentation! Let's take a closer look.

Automatic Instrumentation

The automatic configuration will require you to add dependencies/packages to your application (API, SDK and instrumentation tools), add new environment variables to the application (name of the application, an endpoint to export the signals...) and modify the way the application starts (e.g. in Python instead of python main.py it will be opentelemetry-instrument python main.py, in JavaScript instead of node index.js it will be node --require @opentelemetry/auto-instrumentations-node/register index.js). So it's a change you will make in the Dockerfile and dependencies of the application, you won't need to change a single line of code!

Manual Instrumentation

If you decide to use the manual configuration you will need to manually instrument the application to call OpenTelemetry APIs. It turns out that most of the applications don't need this because the code mostly relies on some dependencies (e.g. web frameworks such as Express, Flask, Spring Boot) so you won't even need to deal with OTel APIs. In these scenarios, there are two possibilities:

Natively Instrumented: the dependency you use can directly call OTel APIs (e.g. NextJS). There are only a few libraries that are natively instrumented, you will likely have to go for the next possibility.
Integration Libraries: The open-source community created a library that integrates your dependency and OTel API. In case you want to check these libraries your best friend is the OpenTelemetry Registry. Search for the name of the dependency you are using and check if the open-source community created the integration library!

OpenTelemetry

What is the point of Open Telemetry?

It's better to start by understanding how code was instrumented before OTel, so we can understand why OTel exists.

As commented before, you can only have observability if your code is instrumented. To do so your code must send data (logs, metrics, traces) to an observability backend. There are several observability backend tools (open sources like Jaeger and Zipkin or paid services like Datadog and New Relic) and for each tool, there will be specific libraries/agents for emitting the data. It means that if we instrumented our code using Jaeger but now we need to change to Zipkin we would have to re-instrument the code again using different libraries, different codes, different collectors, and different agent. That's where OpenTelemetry comes in.

OpenTelemetry goal is to provide standard SDKs, APIs and tools for sending data to an observability backend. Otel is not an observability backend service, it supports exporting data to the observability backend tools (open source or not). In other words, OTel wants to standardize the way we instrument the application code and standardize the data format of generated telemetry data. Once the telemetry data is in a consistent format we can send it to any observability backend (such as Jaeger, Prometheus, Signoz...).

So OTel does two important things: allows you to own the data you generate rather than be stuck with a proprietary data format or tool and allows you to learn a single set of APIs and conventions.

The OpenTelemetry Architecture

The diagram above uses generic elements to represent each category and this is the important thing to understand. You can easily modify the blocks to real tools, e.g.

OTel Components

The main components of OTel are:

Specification: Defines cross-language requirements for all implementations, defining API, SDK and OTLP (OpenTelemetry Protocol).
Collector: Proxy that receives, processes and exports telemetry data to a backend. It supports data in multiple formats (OTLP, Jaeger, Prometheus...).
Language SDKs: It lets you use OTel API to generate telemetry data for your favorite language and export to your favorite backend.
Instrumentation Libraries: Supports popular libraries and frameworks for generating telemetry.
Automatic Instrumentation: Provides a way to instrument the code without changing a single line of code. It will add a minimum telemetry for you.
K8S Operator: It's a Kubernetes Operator that manages the OTel Collector and auto-instrumentation.

Collector

The OpenTelemetry Collector is a vendor-agnostic proxy that can receive, process, and export telemetry data. It supports receiving telemetry data in multiple formats (for example, OTLP, Jaeger, Prometheus, as well as many commercial/proprietary tools) and sending data to one or more backends. It also supports processing and filtering telemetry data before it gets exported.

The OTel Collector is NOT necessary. You can configure your application to emit signals and export to the backend tools directly or you can configure it to send to the collector and then the collector forwards to the backend tools.

Well, if it's optional, why would you want to use the collector then? Using a collector is more scalable!

Configuration

The configuration of the collector is done through a YAML file. In the collector, we can configure a pipeline with the following elements:

Receivers: Define the sources of the telemetry
Processors: Applies transformations to the collected data. This includes actions such as filtering, renaming, dropping, recalculating...
Exporters: Define where the telemetry (after the processor is applied) will be sent
Connectors: Acts as an exporter and receiver because it will connect two pipelines (consuming data as an exporter and emitting data as a receiver)

All these components must be enabled in the service session of the YAML manifest. An example of a valid configuration is the following:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:

exporters:
  otlp:
    endpoint: otelcol:4317

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

OTLP

The OpenTelemetry Protocol (OTLP) defines how the telemetry data will be transported between the source (application) and destination (observability backend). It will define the encoded, protocol (it accepts HTTP and gRPC) of the payloads.

In simple terms, the idea is to define a specification (common language) that can be spoken by all parts: the application, the collector (that can be an OpenTelemetry Colletor or any other alternative, such as Grafana Agent) and the observability backend (Loki, Prometheus, Grafana Tempo...). If all parts follow the same specification (language) you can simply change the elements (e.g. switch from Jaeger to Tempo) without having to modify the application code (maybe you will need to modify environment variables but that's all).

If you want to check in depth I recommend you to check the official documentation.

Cya!

Well, we covered a lot of what OTel is, the main components, concepts, and architecture... I highly recommend you to check some very useful links above:

Play around, get familiar with the documentation, take notes and start to use OpenTelemetry in your applications!

I intend to make another blog post applying the concepts we covered here so stay tuned! Hope you like it and see you around! 👋

Configuring a free email provider for your domain using AWS Route53 and Zoho

Thu, 08 Feb 2024 10:40:32 GMT

If you own a domain (in my case felipetrindade.com) you can easily configure a free email provider and start using your custom email (e.g. contact@felipetrindade.com). This is so much more professional!

Stack

Infra as Code

As you probably know, DevOps Engineers hate to do manual things. That's why I'm going to avoid as maximum as I can manual steps and I will configure our custom email provider using Pulumi with Typescript as Infrastructure as Code tool.

DNS

I will consider that your domain is already being managed by AWS Route53. This is not necessary, you can manage your domain in the registrar (the website you bought your domain, e.g. GoDaddy, Namecheap...), but since the code I wrote using Pulumi is specific this is a pre-requirement for following this post.

Email provider

There are myriad options of email providers out there, such as Google Workspace, Zoho, ProtonMail, Outlook, AOL, Yahoo... What we want is a free email provider (or one that has a free plan that is enough for our use case).

I'm very used to Gmail, but I'm not willing to pay around $5 per month to use barely use Gmail (Google Workspace). Oh, just for reference I paid $15 (second year) for a full year for my domain on NameCheap. And I pay around $0.5 per month to maintain my static blog/website. So domain + website cost me around $21 per year... This is the same as 4 months of Google Workspace in the cheapest plan. So, I don't think it is worth it (for my use case).

Zoho provided a good free plan for my use case, but another good alternative is Proton.

Zoho configuration

Configuring Zoho is straightforward, you need to create the account, follow the steps and soon you will be presented with the following screen:

After configuring this DNS record (we will get there!) the following screen will be displayed:

Pulumi code

To configure Zoho to be the email provider for your domain you need to create TXT and MX records in your DNS (AWS Route53). I created a generic class for this:

// email.ts
import { route53 } from "@pulumi/aws"
import { ICommonProps } from "./commons"

export interface ICommonProps {
  domainName: string
  accountId: string
  zoneId: string
}

interface Record {
  type: "MX" | "TXT"
  subdomain?: string
  value: string[]
}

export interface IEmail {
  record: Record[]
  commonProps: ICommonProps
}

export class Email {
  id: string
  props: IEmail

  constructor(id: string, props: IEmail) {
    this.id = id
    this.props = props
    this.setRecords()
  }

  setRecords(): void {
    for(let i=0; i<this.props.record.length; i++){
      const subdomain = this.props.record[i].subdomain ?? '';
      new route53.Record(`${this.id}-${i}-record`, {
        zoneId: this.props.commonProps.zoneId,
        name: subdomain,
        type: this.props.record[i].type,
        records: this.props.record[i].value,
        ttl: 300
      })
    }
  }
}

We need to instantiate this class in two parts. The first TXT DNS record and then we apply the Pulumi configuration and after it's configured we apply the rest of the records.

// index.ts
import { Email } from "./email"

export const commonProps: ICommonProps = {
  domainName: "felipetrindade.com",
  accountId: "1111111111111",
  zoneId: "Z123456789ABCDEFGHI00",
}

export const emailConfig: IEmail = {
  record: [{
    type: "TXT",
    value: ["zoho-verification=zb57142340.zmverify.zoho.com"]
  }],
  commonProps: commonProps,
}

new Email("email", emailConfig)

After you apply this (pulumi up), the second DNS screen will be displayed and you can edit this file to be:

// index.ts
import { Email } from "./email"

export const commonProps: ICommonProps = {
  domainName: "felipetrindade.com",
  accountId: "1111111111111",
  zoneId: "Z123456789ABCDEFGHI00",
}

export const emailConfig: IEmail = {
  record: [{
    type: "TXT",
    value: [
      "zoho-verification=zb57142340.zmverify.zoho.com",
      "v=spf1 include:zoho.com ~all",
    ],
    },{
    type: "MX",
    value: [
      "10 mx.zoho.com",
      "20 mx2.zoho.com",
      "50 mx3.zoho.com",
    ],
    },{
    type: "TXT",
    subdomain: "zmail._domainkey",
    value: ["v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCnFUY1z6/mZNsJXDBic4c+lxD0HHAy/iKQybHKLcvxAGPKjoTGW/wtlH+ncsKtSV2QDACinwXAVqP0j8eJtOKnABCwklr2y3IoC/MzrCzMk9ef3ni1Vg8ktB6Ig5KKZGcmhn8GWFj+xXa/ezVc/7KSgABCogwt0DDYZCE+QvK4wIDAQAB"]
  }],
  commonProps: commonProps,
}

new Email("email", emailConfig)

Email Forwarder

In case you want to set up a Contact Me` page on your website you will need to configure an email forwarder to forward the message to your email provider (in our case, Zoho).

I did that on my website, as you can check here. I've used EmailJS which gives me 200 emails per month in the free plan. This is more than enough for my website!

PS: Recently Zoho announced that POP/IMAP will only be available on the paid plan ($1 month).

Cya!

Hope you liked this post! Stay tuned for the following posts!

Deploying a static website on AWS using S3 and CloudFront

Thu, 01 Feb 2024 15:40:32 GMT

In this post, I'll comment, discuss and deep-dive about how to deploy a static web application using S3 and CloudFront. Actually, this was the way I deployed this website/blog!

Considerations regarding the tools used

I've used some tools to help me deploy the static web application. To be honest, I think this is the less important part here, I could have used other tools to do the same job. The most important thing is to understand why the tool is being used, if you understand the reason you can replace it with another tool easily.

CI/CD: I've used GitHub Actions to perform the CI/CI workflow of the project for the infrastructure as code (IaC) and for the website (static web application). You could have used any continuous integration tool here, such as GitLab CI/CD, CircleCI, Jenkins... I decided to use GitHub Actions to manage the CI/CD because it's already integrated with GitHub, the free tier is enough for my use case and it is a managed service (and I was not in the mood to self-manage and deploy an open source CICD tool). The workflow was performing the following tasks:
- For the infrastructure: Deploying the cloud resources used in the project, such as S3 bucket, CloudFront distribution, and Route53 records. Don't worry! I will cover all these services, just keep reading!
- For the web app: Building the source code, generating the static files, pushing the static files to S3 and invalidating the CloudFront distribution.
Infrastructure as Code: All the cloud resources created could have been created manually in the console, but what is the point of it? This is not reproducible, difficult to document the changes, hard to keep track of the changes. I decided to use Pulumi to create the documentation. I chose Pulumi because I was curious to give it a try and be able to compare it with other IaC tools that I've used in other projects, such as Terraform/OpenTofu, Terragrunt, AWS CDK and SAM.
Web app: Here we have so many options to build a web application, so many different stacks/frameworks are possible to be used here, such as pure HTML/CSS project, React, NextJS... I decided to use Gatsby, which is a web framework that uses React behind the scenes.

Architecture

A visual approach to what we are going to deploy is available below.

The following AWS Services are being used:

S3: Service that stores objects, in our case, it will serve as the storage of our static web app.
ACM: It's a service that will provide the TLS certificate that will make our website secure (using HTTPS).
CloudFront: It is a CDN service, that will cache the static files of our website and deliver them to the clients (users).
Route53: To create records that allow the users to access the website using our domain.

S3

The Simple Store Service, widely known as S3, is one of the most famous/oldest AWS services. It is an object storage service that automatically manages scalability, data availability, high availability, data replication... A cool feature of S3 is that you can set an S3 bucket to host a static website, so you can simply build the website code and throw it in the bucket, set the bucket to be public, add a bucket policy that allows anyone to get objects from the bucket and you have a website up and "running"!

That's it. It's that straightforward. But there are some catches that it is important to mention and be aware of. S3 website endpoints do not support HTTPS, which is a big bad thing, even if it's just a static website, the users might feel that the website is not safe, secure, trustworthy and won't put much credibility into it. Also, your website will be now hosted in the AWS region where the S3 bucket was created. This means that if you, for example, created the bucket in us-east-1 (N. California) and a user from China wants to access the website, all the requests would have to go to the N. California servers, so the latency can be higher than expected. If it's a simple static website this might not even be noticed by the user, but if it contains videos, this can be very noticeable. And no, creating several buckets across different regions is not a good idea to solve the problem.

Bucket Policy

Since there is no such thing as an IAM Role for CloudFront, the way we allow CloudFront to get the assets from the S3 Bucket is by setting a bucket policy that allows the CloudFront service (we can add, and we will, a condition to only allow a specific CloudFront distribution, instead of the entire service) to pull objects from the bucket. By doing that we guarantee that users are not able to access the website through the S3 Website endpoint, only through the CloudFront distribution. A configuration regarding access should also be done from the CloudFormation side, we will cover that in a bit!

CloudFront

Wouldn't it be nice if we were able to cache our website very close to the users across the world and also allow HTTPS in our website? That's where CloudFront comes in!

CloudFront is the CDN (Content Delivery Network) of AWS. If you used AWS at least once you probably know that resources are deployed in regions (such as us-east-1, ap-south-1, eu-west-2...), and this means that the server that hosts the service is running in that region. So if you work for a company based in India and most of your clients are also from India, it probably makes sense to host this in India too (e.g. ap-south-1), because the data center that contains the server that is running your application will be closer to the client, resulting in lower latency and better user experience in general. Of course, there are some points to consider, such as the cost of the Indian region being more expensive than the US region, but that's the general idea in terms of performance.

But if you want to deploy static content (e.g. HTML, CSS, JS, Images, Videos...) you can deploy these assets in data centers that are closer to the clients, called edge locations - also called CloudFront POP (point of presence). These edge locations will cache your assets so the delivery to the client will be way faster! These edge locations exist all around the world so it means that even if you only host your application (e.g. the API that is used by your website) in one region, parts of the website (that are using the CloudFront to deliver static content) will be fast to the client.

Data center levels

There are three data center levels when we talk about CloudFront: edge locations (also known as POP), regional edge caches and the origin.

Let's take a look at each type and how they related:

Edge Location (POP): This is the closest server to the client and can serve the content directly to the users.
Regional Edge Caches: It has a larger cache when compared to the POP so the objects persist longer than the POP.
Origin: The "place" that contains the original content (in our case, the static files of the website). A common origin (and this is the origin that we will use) is to host the content in an S3 bucket, but you can also use Elastic Load Balancer, an HTTP server in an EC2 instance, and many other options as a CloudFront origin.

Distribution Price Class

We commented that the edge locations exist all around the world, and that's a fact, but you don't need to use all the edge locations. You can select the Price Class of your CloudFront distribution.

Edge locations are grouped in regions (North America, South America, Europe...). Price Class is a group of regions that will be used in your distribution. There are three Price Classes:

Price Class All: Uses all POP in all regions.
Price Class 200: Uses all POP in all regions, except South America, Australia and New Zealand regions.
Price Class 100: Uses POP only in North America, Europe and Israel.

Based on that, you should ideally use a Price Class that covers most (or all) of your clients, reducing the latency they will see and optimizing the user experience. It is tempting to select Price Class All, so you will cover the entire world! But be aware that one of the highest costs in CloudFront is the data transfer going out of CloudFront to the users and this depends on the region. The cheapest regions (North America and Europe) cost $0.085 per GB transferred and the most expensive region (Hong Kong, Indonesia...) costs $0.120 per GB transferred. As we gonna comment in the pricing section, if your website has little access you won't even be charged because the free tier is very generous, so it's ok to use Price Class All.

If you select Price Class 100, it means that users from Japan (for example), won't get the content from an edge location. To be honest, there is a chance that they can be served by an edge location (that's how AWS works) but you will pay the least expensive region in your price class.

Caching Flow

We can sum up the caching flow using the following flow chart below:

You might be wondering: what happens if I update the content? Does it mean that the clients will get the cached content until eventually the cache is not available and the content needs to be fetched from the origin? Yes, that's exactly what happens. If you want to update the content and force it to show to your users you can invalidate the cache! This is something that we will do in our CI/CD: build the website, push the assets to the S3 bucket and invalidate the cache. But don't worry, we will still get there.

HTTPS

To distribute the content using HTTPS it is necessary to have a certificate. You can use the AWS Certificate Manager (ACM) service to create a certificate and "attach" to the CloudFront distribution. This way, your website will support HTTPS using TLS protocol.

Alias

When a CloudFront distribution is created, it automatically provides a domain name for you to access the distribution, something like d139253ab29.cloudfront.net. This domain name is really bad in terms of usability and you definitely shouldn't use it directly, especially in production.

The idea is to create an A record in your Route53 that points to a more friendly name (in a domain that you own), such as felipetrindade.com, blog.felipetrindade.com or www.felipetrindade.com.

Access Control

To allow CloudFront to authenticate with S3 as origin two ways can be used: Origin Access Control (OAC) and Origin Access Identity (OAI). The latest is the legacy method, so we will give preference to the OAC method. The main difference between OAC and OAI is that OAC enhances security (using short-term credentials behind the scenes) and allows you to use KMS with CloudFront and S3 bucket. Using OAC, we will allow the CloudFront distribution to access the S3 using authenticated requests as we want.

Lambda@Edge

CloudFront allows us to specify the default root object of our website, usually the index.html, but this only works on the root of the website (e.g. https://www.felipetrindade.com -> https://www.felipetrindade.com/index.html). It does not work on any subdirectory level (e.g. https://www.felipetrindade.com/hello-world).

Let me be more specific here, if you access the root page of the website (e.g. https://www.felipetrindade.com) and click on any post it will work normally, let's say that you click on the Hello, World! post, you will see that the URL of the browser will be modified to https://felipetrindade.com/hello-world. But if you reload the page of the post or simply try to access it from a link (https://www.felipetrindade.com/hello-world) you will get a 403 Access Denied error. But if you add index.html at the end of the URL it is gonna work perfectly (https://felipetrindade.com/hello-world/index.html).

The explanation for this problem is fairly simple. S3 authenticates with CloudFront using the OAC, sending requests to the S3 Rest API. The API does not allow redirection to the default index page. So when the user goes directly to a subpage (e.g. https://felipetrindade.com/hello-world) it will request to S3 an object with hello-world as the object key and this key does not exist, but the key hello-world/index.html exists! So that's why it is going to work if we add the index.html at the end of the URL.

This behavior is such a TERRIBLE user experience and we don't want to deploy such a bad thing. Luckily CloudFront provides us something called Lambda@Edge. Basically, we are going to deploy Lambda Functions at edge locations and these Lambda functions can alter the response or request between the user and edge location or between edge location and request. So it can be used to do A/B testing, add headers, dynamically route based on headers/cookies/query strings and so on. You can set the Lambda to run in four types of events:

Viewer request: User request to edge location.
Viewer response: Edge location request to the user.
Origin request: Edge location request to origin.
Origin response: Origin request to edge location.

In our use case, we want to modify the Viewer Request, since we want to modify the URI so the Edge Location can effectively retrieve the sub-page! We are going to implement that using NodeJS and it is as simple as 10 lines of code: get the URL from the events objects (JSON) and return the URL with index.html appended.

Certificate Manager

AWS Certificate Manager is the service responsible for provisioning the TLS certificates. So you don't need to go through the burden of purchasing/issuing the certificate, renewing and uploading it to the servers. It is a really handy service and the best part is that public certificates are free, so you don't pay extra for it.

One tricky thing about the integration between ACM and CloudFront is that the certificate MUST be issued in the N. Virginia region (us-east-1).

Route53

This is the DNS service managed by AWS, where you can buy a domain and manage the records for the domain, allowing name resolution for hostnames you own. One thing that is important to mention is that you DO NOT have to buy your domain from AWS to use it. If your domain was purchased from websites like GoDaddy, Namecheap, HostGator or any other registrar you can delegate your domain to be managed by Route53. This tutorial is out of the scope of this blog post but it is a straightforward process, it is easy to find tutorials or videos that go through it. The idea is that you will create a public or private (in our example the website is public, so I created a public one) zone in Route53.

Pricing

Not sure if you noticed but the infrastructure we built is 100% serverless: we only pay for what we use and the amount of traffic we receive, let's take a deeper look at the costs:

Route53

We will pay $0.50 to manage a public hosted zone. This is one of the highest costs of the project. There are other costs involved, but they are so low (if your website has low traffic) that you can consider it to be zero, e.g. $0.40 per 1 million requests to resolve the host of your website (in my case felipetrindade.com).

We pay for the assets that we put in S3. The storage cost is barely zero, for reference, the total size of my website is under 10MB (you pay only $0.023 per GB). It's true that we also pay for API operations in S3 but it is only $0.004 per 10,000 GET (which is likely to be the most used operation in your website).

As we commented before, public certificates are free. 😄

CloudFront

The CloudFront always free tier is generous so if your website does not have much access you probably won't even pay a single penny on that. The always free tier offers you 1TB of data transfer out, 10 million HTTP/HTTPS requests each month and this is more than enough for a personal website with low access. But in case you are curious to know more about the CloudFront pricing, here we go:

HTTP requests from users: from $0.0075 to $0.016 per 10,000 HTTP requests
HTTPS requests from users: from$0.01 to $0.022 per 10,000 HTTPS requests
Data Transfer from CloudFront to users: from $0.085 to $0.12 per GB transferred

Lambda@Edge

We are charged $0.60 per 1 million requests and, since we are going to use a 128MB Lambda, $0.00000625125 for every second used. So considering that the Lambda will run for 100ms per execution (usually it's way less), this means that 1 million executions of the Lambda will cost us $1.23. So cheap!

Total Cost

In the end, for a small project, you probably going to pay only for the Route53 public hosted zone. So the monthly cost for your project will be only $0.50!

Infrastructure code

Well, my entire website code is publicly available on GitHub! You can take your time to check the code, what is being done and how things were created. You can also find valuable information in the README.md files of the frontend and infrastructure folders.

I rather not talk about the website code, although you can check what I'm doing by taking a look at the Github repository. I rather comment on the infrastructure code. It's going to be a code specific for Pulumi but it is easy to "migrate" the code from Pulumi to Terraform or any other IaC tool since the valid thing here is to understand the resources being used.

Pulumi

I divided the infrastructure into two parts:

Frontend: The architecture diagram you saw earlier, is composed of ACM Certificate, CloudFront, S3 bucket, and Route53 records.
GitHub Actions OIDC: In the CI/CD we need to perform some actions (e.g. push assets to the S3 bucket, invalidate CloudFront distribution) in the AWS account. There are two common ways to deal with this in terms of authentication:
- Create an IAM User for the GitHub Actions, issue an access key/secret, store it in the secrets of GitHub actions, and perform authentication using these key pairs. The major problem with this approach in my opinion is that you are using static credentials, you will need to care about rotating these, so that's why we are not going through this path.
- Create an Identity Provider in AWS that allows GitHub Actions to authenticate with AWS via OIDC and creates an IAM Role that GitHub Actions will be able to assume. The credentials are dynamic, so it's an improvement in terms of security! We are going to implement this!

Frontend

Pulumi creates resources when you instantiate the classes that represent the resource. My idea was to put all the frontend components into a single class called Frontend and instantiate in the index.ts file. So I ended up with the following file:

// frontend.ts
import * as archive from "@pulumi/archive";
import { s3, acm, cloudfront, route53, lambda, iam, cloudwatch, Provider } from '@pulumi/aws'
import { interpolate, asset } from '@pulumi/pulumi'

import { accountId } from "./commons"

export interface IFrontend {
  domainName: string
  lambdaAtEdgeLogGroupRetention: number
}


export class Frontend {
  id: string
  props: IFrontend
  certificate: acm.Certificate
  distribution: cloudfront.Distribution
  bucket: s3.Bucket
  zoneId: Promise<string>
  lambda: lambda.Function
  provider: Provider

  constructor(id: string, props: IFrontend) {
    this.id = id
    this.props = props
    this.zoneId = this.getZone()
    this.certificate = this.getCertificate()
    this.bucket = this.getS3Bucket()
    this.lambda = this.getLambdaAtEdge()
    this.distribution = this.getDistribution()
    this.provider = new Provider("us-east-1", { region: "us-east-1" })

    this.setBucketPolicy()
    this.setRoute53()
  }

  getCertificate(): acm.Certificate {
    const certificate = new acm.Certificate(`${this.id}-certificate`, {
      domainName: `${this.props.domainName}`,
      subjectAlternativeNames: [
        this.props.domainName,
        `*.${this.props.domainName}`,
      ],
      validationMethod: "DNS"
    }, { provider: this.provider })
    const certificateValidation = new route53.Record(`${this.id}-certificate`, {
      zoneId: this.zoneId,
      name: certificate.domainValidationOptions[0].resourceRecordName,
      records: [certificate.domainValidationOptions[0].resourceRecordValue],
      type: certificate.domainValidationOptions[0].resourceRecordType,
      ttl: 60
    })
    new acm.CertificateValidation(`${this.id}-certificate-validation`, {
      certificateArn: certificate.arn,
      validationRecordFqdns: [certificateValidation.fqdn]
    })

    return certificate
  }

  getS3Bucket(): s3.Bucket {
    const bucket = new s3.Bucket(`${this.id}-bucket`, {
      website: {
        indexDocument: "index.html",
        errorDocument: "index.html",
      },
    })

    return bucket
  }

  getDistribution(): cloudfront.Distribution {
    const originAccessControl = new cloudfront.OriginAccessControl(`${this.id}-origin-access-control`, {
      originAccessControlOriginType: "s3",
      signingBehavior: "always",
      signingProtocol: "sigv4"
    })
    const distribution = new cloudfront.Distribution(`${this.id}-cloudfront-distribution`, {
      enabled: true,
      comment: "Cloudfront distribution of my personal website and blog",
      origins: [{
        domainName: this.bucket.bucketRegionalDomainName,
        originAccessControlId: originAccessControl.id,
        originId: "s3OriginId",
      }],
      priceClass: "PriceClass_All",
      defaultCacheBehavior: {
        allowedMethods: [
          "DELETE",
          "GET",
          "HEAD",
          "OPTIONS",
          "PATCH",
          "POST",
          "PUT",
        ],
        cachedMethods: [
          "GET",
          "HEAD",
        ],
        targetOriginId: "s3OriginId",
        viewerProtocolPolicy: "redirect-to-https",
        cachePolicyId: "658327ea-f89d-4fab-a63d-7e88639e58f6", // Managed-CachingOptimized
        lambdaFunctionAssociations: [{
          eventType: "viewer-request",
          lambdaArn: this.lambda.qualifiedArn
        }]
      },
      restrictions: {
        geoRestriction: {
          restrictionType: "none",
          locations: [],
        }
      },
      viewerCertificate: {
        cloudfrontDefaultCertificate: false,
        acmCertificateArn: this.certificate.arn,
        sslSupportMethod: "sni-only",
        minimumProtocolVersion: "TLSv1.2_2021",
      },
      customErrorResponses: [{
        errorCode: 404,
        responsePagePath: "/404.html",
        responseCode: 404,
        errorCachingMinTtl: 3600,
      }, {
        errorCode: 403,
        responsePagePath: "/404.html",
        responseCode: 404,
        errorCachingMinTtl: 3600,
      }],
      defaultRootObject: "index.html",
      aliases: [
        this.props.domainName,
        `www.${this.props.domainName}`
      ],
    })

    return distribution
  }

  setBucketPolicy(): void {
    const bucketPolicyDocument = interpolate`{
      "Version": "2008-10-17",
      "Id": "PolicyForCloudFrontPrivateContent",
      "Statement": [
        {
          "Sid": "AllowCloudFrontServicePrincipal",
          "Effect": "Allow",
          "Principal": {
            "Service": "cloudfront.amazonaws.com"
          },
          "Action": "s3:GetObject",
          "Resource": "${this.bucket.arn}/*",
          "Condition": {
            "StringEquals": {
              "AWS:SourceArn": "arn:aws:cloudfront::${accountId}:distribution/${this.distribution.id}"
            }
          }
        }
      ]}`
    new s3.BucketPolicy(`${this.id}-bucket-policy`, {
      bucket: this.bucket.id,
      policy: bucketPolicyDocument,
    })
  }

  getZone(): Promise<string> {
    const zone = route53.getZone({
      name: this.props.domainName
    })
    const zoneId = zone.then(zone => zone.zoneId)

    return zoneId
  }

  setRoute53(): void {
    new route53.Record(`${this.id}-record`, {
      zoneId: this.zoneId,
      name: this.props.domainName,
      type: "A",
      aliases: [{
        name: this.distribution.domainName,
        zoneId: this.distribution.hostedZoneId,
        evaluateTargetHealth: false
      }]
    })

    new route53.Record(`${this.id}-record-www`, {
      zoneId: this.zoneId,
      name: `www.${this.props.domainName}`,
      type: "A",
      aliases: [{
        name: this.distribution.domainName,
        zoneId: this.distribution.hostedZoneId,
        evaluateTargetHealth: false
      }]
    })
  }

  archiveLambdaCode(): string {
    const archiveFile = "lambda/lambda_at_edge.zip"
    archive.getFile({
      type: "zip",
      sourceFile: "lambda/index.js",
      outputPath: archiveFile
    })

    return archiveFile
  }

  getLambdaAtEdge(): lambda.Function {
    const archiveFile = this.archiveLambdaCode()
    const trustRelationship = interpolate`{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Service": [
            "lambda.amazonaws.com",
            "edgelambda.amazonaws.com"
        ]},
        "Action": "sts:AssumeRole"
      }]
    }`

    const policy = new iam.Policy(`${this.id}-lambda-policy`, {
      policy: interpolate`{
        "Version": "2012-10-17",
        "Statement": [{
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": ["s3:List", "s3:Get*"],
          "Resource": [
            "${this.bucket.arn}",
            "${this.bucket.arn}/*"
          ]
        }]
      }`
    })

    const role = new iam.Role(`${this.id}-lambda-at-edge`, {
      assumeRolePolicy: trustRelationship,
      managedPolicyArns: ["arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole", policy.arn],
    })

    const logGroup = new cloudwatch.LogGroup(`${this.id}-lambda-at-edge-log-group`, {
      retentionInDays: this.props.lambdaAtEdgeLogGroupRetention,
    })

    const lambda_function = new lambda.Function(`${this.id}`, {
      description: "Runs at edge to add '/index.html' in every uri, allowing users to access paths",
      role: role.arn,
      code: new asset.FileArchive(archiveFile),
      runtime: "nodejs20.x",
      handler: "index.handler",
      architectures: ["x86_64"],
      publish: true,
      loggingConfig: {
        logGroup: logGroup.name,
        logFormat: "Text"
      }
    }, { provider: this.provider })

    new lambda.Permission(`${this.id}-lambda-permission`, {
      action: "lambda:InvokeFunction",
      statementId: "AllowExecutionFromCloudFront",
      function: lambda_function,
      principal: "edgelambda.amazonaws.com",
    })

    return lambda_function
  }
}

The Lambda@Edge code is the following:

// lambda/index.html
'use strict';
exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request;
  const uri = request.uri;

  if (uri.endsWith('/')) {
    request.uri += 'index.html';
  } else if (!uri.includes('.')) {
    request.uri += '/index.html';
  }

  callback(null, request);
};

I used a commons.ts file to export helpful functions and values:

// commons.ts
import { getCallerIdentity } from '@pulumi/aws'

const current = getCallerIdentity({});
export const accountId = current.then(current => current.accountId);

And instantiated my Frontend class in the index.ts:

// index.ts
import { Frontend } from "./frontend"
import { frontendConfig, githubOidcConfig } from "./config"
import { GitHubOidc } from "./oidc"

new GitHubOidc("oidc", githubOidcConfig)
const frontend = new Frontend("frontend", frontendConfig)

// Export - Slack Outputs - Used during CI
export const cloudfrontDistribution = frontend.distribution.id
export const frontendS3Bucket = frontend.bucket.bucket

And the configs (parameters of my stack), I created the config.ts file:

// config.ts
import { IFrontend } from './frontend'
import { IGitHubOidc } from './oidc'


export const frontendConfig: IFrontend = {
  domainName: "felipetrindade.com",
}

export const githubOidcConfig: IGitHubOidc = {
  roleName: "GithubActionsOidcWebsite",
  organization: "felipelaptrin",
  repository: "felipetrindade.com",
}

Github Actions OIDC

Now, let's take a look at the class that creates the GitHub Actions identity provider.

// oidc.ts
import { iam } from "@pulumi/aws"
import { accountId } from "./commons"
import { interpolate } from "@pulumi/pulumi"

export interface IGitHubOidc {
  roleName: string
  organization: string
  repository: string
}

export class GitHubOidc {
  id: string
  props: IGitHubOidc
  role: iam.Role

  constructor(id: string, props: IGitHubOidc) {
    this.id = id
    this.props = props
    this.setIdentityProvider()
    this.role = this.getOidcRole()
  }

  setIdentityProvider(): void {
    new iam.OpenIdConnectProvider(`${this.id}-identity-provider`, {
      url: "https://token.actions.githubusercontent.com",
      clientIdLists: ["sts.amazonaws.com"],
      thumbprintLists: [
        "1b511abead59c6ce207077c0bf0e0043b1382612"
      ]
    })
  }

  getOidcRole(): iam.Role {
    const trustRelationship = interpolate`{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::${accountId}:oidc-provider/token.actions.githubusercontent.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
          },
          "StringLike": {
            "token.actions.githubusercontent.com:sub": "repo:${this.props.organization}/${this.props.repository}:*"
          }
        }
      }]
    }`

    const role = new iam.Role(`${this.id}-role`, {
      name: this.props.roleName,
      description: "Role assumed by the GitHub Actions to deploy resources related to the website/blog",
      assumeRolePolicy: trustRelationship,
      managedPolicyArns: ["arn:aws:iam::aws:policy/AdministratorAccess"],
    })

    return role
  }
}

CI/CD

Finally! Let's check the CI/CD. We can divide the CI/CD pipeline into two jobs:

Infrastructure deployment: Here, we need to install Pulumi, check if there are drifts, show the actions that it will perform and finally deploy the infrastructure (the latest step only when merged in the main branch).
Frontend deployment: We need to build the assets, push them to the S3 bucket and invalidate the CloudFront cache.

There is a clear dependency in these jobs: the infrastructure needs to be deployed first, only after the infrastructure deployment we can effectively deploy the website assets to CloudFront.

GitHub Actions Workflow

The final CI/CD for this project is the following:

name: CI/CD

on:
  push:
    branches:
      - main
  pull_request:

permissions:
  id-token: write # This is required for requesting the JWT
  contents: read  # This is required for actions/checkout

jobs:
  infra:
    runs-on: ubuntu-latest
    outputs:
      frontend-bucket: ${{ steps.frontend-bucket.outputs.s3 }}
      cloudfront-distribution: ${{ steps.cloudfront-distribution.outputs.cloudfront }}
    env:
      AWS_REGION: ${{ vars.AWS_REGION }}
      PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Assume Role via OIDC
        uses: aws-actions/configure-aws-credentials@v3
        with:
          role-to-assume: ${{ vars.IAM_ROLE_ARN_OIDC_GITHUB }}
          role-session-name: Github-Actions
          aws-region: ${{ vars.AWS_REGION }}

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: 21.6.1

      - name: Install node dependencies
        working-directory: infrastructure
        run: npm install

      - name: Pulumi installation
        run: |
          curl -fsSL https://get.pulumi.com | sh
          export PATH=$PATH:/root/.pulumi/bin

      - name: Pulumi Login using S3 as backend
        run: pulumi login ${{ vars.PULUMI_BACKEND_S3_BUCKET }}

      - name: Pulumi Preview
        working-directory: infrastructure
        run: pulumi preview --diff --refresh -s stack

      - name: Get Frontend S3 Bucket
        id: frontend-bucket
        working-directory: infrastructure
        run: |
          OUTPUT=$(pulumi stack output frontendS3Bucket -s stack)
          echo "s3=$OUTPUT" >> "$GITHUB_OUTPUT"

      - name: Get CloudFront Distribution ID
        id: cloudfront-distribution
        working-directory: infrastructure
        run: |
          OUTPUT=$(pulumi stack output cloudfrontDistribution -s stack)
          echo "cloudfront=$OUTPUT" >> "$GITHUB_OUTPUT"

      - name: Pulumi Deploy
        if: github.ref == 'refs/heads/main'
        working-directory: infrastructure
        run: pulumi up --yes --refresh -s stack

  frontend:
    runs-on: ubuntu-latest
    needs: infra
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Assume Role via OIDC
        uses: aws-actions/configure-aws-credentials@v3
        with:
          role-to-assume: ${{ vars.IAM_ROLE_ARN_OIDC_GITHUB }}
          role-session-name: Github-Actions
          aws-region: ${{ vars.AWS_REGION }}

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: 21.6.1

      - name: Install node dependencies
        working-directory: frontend
        run: |
          npm install
          npm install -g gatsby-cli

      - name: Build website
        working-directory: frontend
        run: gatsby build

      - name: Push assets to S3
        if: github.ref == 'refs/heads/main'
        working-directory: frontend/public
        run: |
          S3=s3://${{ needs.infra.outputs.frontend-bucket }}
          aws s3 sync . $S3

      - name: Invalidate CloudFront Cache
        if: github.ref == 'refs/heads/main'
        run: |
          aws cloudfront create-invalidation \
            --distribution-id ${{ needs.infra.outputs.cloudfront-distribution }} \
            --paths "/*"

I don't have much to comment about the CI/CD since I've already made my considerations and explained the steps needed so it should be straightforward to understand what is being done. But I'd like to comment on the Pulumi outputs.

In the infra job, we are creating all the cloud resources needed (actually this is not correct... there are some manual steps needed that I will comment on in the next section). There are two outputs that we need to get from these resources created: the bucket name and the CloudFront distribution ID.

Instead of hardcoding these values or using any other manual approach to get these values directly from the Pulumi stack output IaC tools usually have an output section where you can get metadata about the resources created and if you are using a tool that does not support that you can (using IaC code) store the needed values in the AWS Parameter Store and read the value from the CI.

Secrets and Variables

Instead of hardcoding some important values in the workflow YAML directly I decided to store these values in the Secrets and Variables. This gives me more flexibility (e.g. if something changes I do not need to open a PR to change the value) and security.

The following variables were set in the repository:

AWS_REGION: The AWS region where all the resources will be deployed.
PULUMI_BACKEND_S3_BUCKET: This is the S3 bucket that will store the Pulumi stack information and metadata. I will comment on this in the following sections.
IAM_ROLE_ARN_OIDC_GITHUB: The IAM Role ARN of the role that Github will use to authenticate with AWS and perform the deployments in the account. Please notice that this can't be fetched from Pulumi outputs, since we need to have access to the Pulumi backend (in our case S3 bucket) to retrieve this information!

And the following secret was set:

PULUMI_CONFIG_PASSPHRASE: Pulumi uses this password to protect and lock your configuration values and secrets. This will be commented on soon.

The chicken and egg problem

I'm not sure if you noticed, but in this project, we faced the chicken and egg problem (actually two times!). If you didn't notice, let me explain these problems.

CI/CD Pipelines

We want to deploy our infrastructure using CI/CD pipelines but the CI/CD needs to assume a role but the role will be created by the CI/CD... Houston, we have a problem.

There are some solutions to tackle this problem, let me point two:

Create resources related to OIDC manually: After manually creating the resources that allow GitHub Actions to authenticate in AWS assuming the IAM Role you can import the manually created resources to your IaC code.
Deploy the stack manually a single time: This involves no manual steps to be done, which is nice but someone (usually a DevOps from the infrastructure team) needs to apply the Pulumi stack manually once. After that, there is no need to apply things manually since the CI/CD pipeline will manage it.

Both options have pros and cons, I chose the second option. So I ran locally:

pulumi up

IaC backend

Once again, we want to deploy our infrastructure using IaC but the IaC requires you to set up a backend to store the metadata about the stack you will deploy. You can't create the IaC backend using IaC because you need it in place before start to use it.

Something important to notice is that this is a common thing for basically all IaC tools out there, if you use Terraform/OpenTofu you will need to set the backend provider (in AWS you probably will use DynamoDB table to perform state lock and S3 bucket to store the remote state), if you use AWS CDK you will need to perform the CDK Bootstrap command...

In this project, using Pulumi, it was necessary to create the S3 bucket to store the Pulumi stack metadata. I decided to create this manually and after the creation, I added the S3 URI in the GitHub Actions Variables (PULUMI_BACKEND_S3_BUCKET).

Also, the initial setup of the stack is to define the name of the stack and the password for the configuration (used in GitHub Actions Secret PULUMI_CONFIG_PASSPHRASE). To do that the following Pulumi command was used:

pulumi stack init --secrets-provider=passphrase -s stack

Back in the old days

Cloud and managed services are great! It makes our lives easier and simple but at the same time, it creates a layer of abstraction that makes it difficult to understand exactly what is happening behind the scenes. So I'd like to comment on how people used to deploy static websites back then. I don't plan to deep dive here because this is not the goal of this blog post, but I think it's worth it to mention some steps needed:

Buy/rent a server/virtual machine
Configure the firewall of the machine to allow traffic on port 80/443
Setup a web server using tools such as NGINX, Apache, Tomcat
Configure the SSL/TLS certificate to allow a secure connection using HTTPS. You could buy the certificate from a Certificate Authority (CA) or use a nonprofit CA (e.g. Let's Encrypt) and install the certificate on your web server. Oh, and remember that SSL/TLS certificates expire so you would need to manage this manually or install a more automatic way to renew it (e.g. Certbot, used to auto-renew certificates issued by Let's Encrypt).
Setup a CI/CD pipeline probably using a bash script that connects to the server using SSH

Well, this is just one example of how this used to be done. Of course, other tools and other steps can be introduced here but as a general view, it's fine.

Cya!

Thanks for taking the time to read this blog post. Hope you liked it and feel free to message me if you want to chat about it. Hope to see you in the next blog post!

Hello, World!

Fri, 26 Jan 2024 10:40:32 GMT

It's been a while since I started to think about writing a blog. And well... the time has finally come!

Motivation

One thing that I love to do, and I know this is not a common thing that people in the software engineering field like to do, is document things. Why? Well, I hate (although it's quite common in our area...) to get frustrated trying to understand how things come together, how they relate, how things work behind the scenes and having a hard time trying to understand what is the real motivation behind things being used.

And after hours of googling, asking friends and watching videos you finally understood it all. And it was probably something that you said:

Oh, so it was just that? Why this was not said in the videos/documentation/explanations online? It was so simple but people was not explaining that!

The click in your mind happened that you finally said: I finally got it!. It's a mindblowing experience that I bet every software developer has been through, and it's simply magical.

Although this experience is incredible I still feel it's a burden to go through it. Wouldn't it be better if a good article existed and spared you all the pain of learning/understanding what you wanted?

This. This is my real motivation. To write easy-to-understand guides/articles, simplify hard concepts and teach people the way I think about several topics.

The internet era and the lack of good content

I love the internet so much, don't get me wrong, but I hate so much the amount of tech material/content that we can find online. I notice that there are two scenarios:

Too much content online, but 95% of it is just basic knowledge (or too repetitive) and because of the huge amount of content, it's hard to filter and find a good one that deep dives or simply explains it in a different way
Niched topic lacking content online

In either scenario, it will be difficult to find good content that deep dives into the topic you are looking for.

My goal is to try as max as I can to avoid too basic discussions or only talk about the obvious parts. Being obvious sometimes is necessary, but I will try my best to be obvious only when needed to guarantee that readers of the blog will be on the same page before jumping into a more complex/advanced topic/real explanation.

My way of doing things

I'm a very visual guy. I rather have one image/diagram rather than 2 pages of text. Also, I'm a very hands-on person, I like to learn by doing, to understand how things work in real life.

There is a famous saying that goes like this:

Give a man a fish, he eats for a day. Teach a man how to fish, he eats for a lifetime

And I'm 100% like that. If I need to code something, I won't simply copy and paste, I will invite you to the documentation, add references and explain why I used that function/class. This is powerful, I don't want to "show off", copy and paste what I did and say "It works man, trust me". I want you to understand what I did, how I did it, how you can improve it and how you can do it yourself, going directly to the source.

So, to sum up, you should expect:

Articles with diagrams, pictures and flowcharts (for all the visual guys, just like me)
Opinionated based articles
How-to guides
Connecting the dots between different topics
Experiments and POCs

Hope you enjoy it and see you around! 👋