↓ Skip to main content
Natan Godoy

Lab 01: Car API on Kubernetes (learning lab)

3 mins

Problem
#

This is a learning lab, not professional experience with Kubernetes. The API (car_api) and the DevOps part (Kubernetes, Helm and GitOps) come from the pycodebr training, with the DevOps part by Mateus Muller. The goal was to follow the training and reproduce the infrastructure and the pipeline, with the adjustments listed below.

View the repository on GitLab

Project architecture
#

Project architectureGitLab CIimages, chart and GitOpsECRimages and Helm chartGitOps repositoryFlux reconciles the HelmReleaseAWS · EKS (us-east-1)FluxALB Ingresscar-apiDeployment and ServiceExternal SecretsSSM Parameter StoreRDS PostgreSQLLocal environment · kindKong · Ingresscar-api · 3 réplicasPostgreSQL
Conceptual diagram based on the implementation documented in the GitLab repository.
  • Local (kind): a kind cluster with one control-plane and two workers. Kong receives requests through an Ingress and forwards them to the car-api Service. PostgreSQL runs in the cluster as a StatefulSet.
  • AWS (EKS): an EKS cluster on version 1.34, with a node group of t4g.small instances. The Ingress uses the ALB from the AWS Load Balancer Controller. The database is an RDS PostgreSQL 17.6 instance in private subnets.
  • Delivery: the pipeline builds the images and the Helm chart, pushes them to ECR, and updates the version in the GitOps repository. Flux reconciles the HelmRelease in the cluster.

Technical decisions and reasoning
#

  • Ingress on the active path, Gateway API outside it: each environment’s values enable Ingress: kong locally and alb on AWS. The chart’s HTTPRoute is disabled. A Gateway API manifest exists in the local directory, but it is not part of the path set by the values.
  • Secrets through External Secrets: on AWS, DATABASE_URL and JWT_SECRET_KEY come from SSM parameters, through a ClusterSecretStore. The chart creates the Secret from them. This keeps credentials out of the chart.
  • GitOps with Flux: the pipeline does not apply manifests to the cluster. It updates the HelmRelease version in the GitOps repository, and Flux reconciles it. The cluster state is versioned in a separate repository.
  • AWS Terraform: VPC with three AZs, public and private subnets, and one NAT gateway. EKS with CoreDNS, kube-proxy, VPC CNI and the pod identity agent. RDS in a private subnet, an OIDC role for the CI, and add-ons for the Load Balancer Controller, Cluster Autoscaler, metrics server and External Secrets.
  • Adjustments to the training: three car-api replicas locally (the reference uses one); external-dns disabled in Terraform; car search by color and filter by owner in the API code. Of the files present in both repositories, 40 are identical to the official one. The others differ in versions, names and configuration.

Real challenges
#

  • Race in the Load Balancer Controller webhook: Terraform disables enableServiceMutatorWebhook. The Service webhook is registered before the controller pods have endpoints, and that interfered with other Helm releases in the same apply. A comment in the code documents the reason.

What I would do differently
#

  • One repository for code and pipeline: the .gitlab-ci.yml runs in the car_api repository. The copy inside project-03 is identical, but in that folder it does not run, because the labs repository has no CI at its root.
  • Image version: the build job uses the fixed tag 1.0.0. The VERSION variable is computed but not used.
  • Test values in GitOps: the HelmRelease contains test credentials in plain text. I would move everything to SSM and remove these values from the repository.
  • Exposure: the ALB Ingress is enabled, but with an example host and no TLS in the HelmRelease. I would only expose the service with a domain and a certificate configured.

Technologies
#

FastAPI, SQLAlchemy, Alembic, PostgreSQL, Docker, Kubernetes (kind and EKS), Helm, Flux, External Secrets, Kong, AWS (EKS, RDS, ECR, SSM, ALB), Terraform, GitLab CI, MkDocs.