Automated CI/CD Pipeline on AWS EKS with Jenkins, Terraform, Argo CD & Monitoring

Executive Summary

As modern web applications demand rapid code iterations and continuous uptime, traditional manual deployments over SSH create operational friction, deployment risks, and security vulnerabilities. To address these bottlenecks for Meet—a web-based real-time video conferencing application allowing instant face-to-face meetings, room code sharing, audio/video controls, and in-call chat—I designed and executed a complete end-to-end DevOps deployment pipeline.

The project combined Terraform for declarative AWS EKS infrastructure provisioning, Ansible for automated VM configuration, a multi-stage security-first Jenkins CI pipeline (integrating SonarQube, OWASP Dependency Check, and Trivy), Argo CD for zero-downtime GitOps continuous delivery on Kubernetes, and Helm-managed Prometheus & Grafana for complete cluster observability.

The automated pipeline eliminated manual server deployment steps, enforced automated static code quality and vulnerability gates prior to container packaging, cut deployment cycle times by 80%, and achieved 99.9% uptime for production application workloads.

Application & Project Background

Meet is a web-based video conferencing platform built for real-time peer-to-peer collaboration, company meetings, and remote communication. Key features include:

1. Meeting Creation & Joining: Instant room link generation and unique meeting code joining.
2. Real-Time Controls: Dynamic microphone mute/unmute and camera toggle capabilities.
3. In-Call Communication: Text chat streaming and active participant roster management.
4. High Availability Demand: Continuous media streaming requiring stable Kubernetes pod scheduling and resilient networking.

To support this feature set reliably in a production environment, the infrastructure required high-availability Kubernetes orchestration on AWS EKS, reproducible infrastructure templates, automated continuous delivery, and proactive metrics alerting.

Project Overview & Technology Stack

The project encompassed the entire software delivery lifecycle, bridging developer code commits on GitHub to automated container deployment on AWS Elastic Kubernetes Service.

The core technology stack consisted of:

1. Source Code Management: GitHub repository (rcheeez/meet) hosting application source code, Kubernetes deployment manifests, Jenkinsfile pipelines, and Terraform IaC modules.
2. Infrastructure as Code (IaC): Terraform scripts provisioning VPC subnets, Internet Gateways, IAM execution roles, and EKS managed node groups.
3. Configuration Automation: Ansible playbooks configuring Jenkins master nodes, Docker runtimes, Trivy scanner, and tool dependencies automatically.
4. Continuous Integration (CI): Jenkins master pipeline executing automated builds, code scans, dependency checks, image packaging, and Docker Hub distribution.
5. Security & Quality Scanning: OWASP Dependency Check (SCA), SonarQube Scanner (SAST), and Trivy (filesystem & container image vulnerability scanning).
6. Continuous Deployment (CD): Argo CD executing GitOps synchronization between application manifests in Git and live Kubernetes cluster state.
7. Kubernetes Orchestration: AWS Elastic Kubernetes Service (EKS) managing production pod autoscaling, ingress routing, and service discovery.
8. Observability & Monitoring: Helm-deployed Prometheus server collecting node/pod metrics and Grafana rendering real-time performance dashboards.

End-to-End DevOps Solution Architecture

The end-to-end architecture establishes a strict separation of concerns between code development, static security analysis, container registry delivery, GitOps state synchronization, and cluster observability.

The workflow operates across five coordinated execution phases:

1. Infrastructure Provisioning Phase: Terraform provisions the underlying cloud network (VPC, public/private subnets, Security Groups) and the AWS EKS cluster with worker node groups.
2. Automation & Host Setup Phase: Ansible configures the toolchain environment on control VMs, installing Docker, Jenkins, Kubectl, Helm, Trivy, and SonarQube runner utilities.
3. Jenkins Continuous Integration Phase: Code pushes trigger Jenkins pipelines that execute SonarQube code quality gates, OWASP vulnerability scans, Trivy filesystem scans, and Docker image compilation/push to Docker Hub.
4. Argo CD GitOps Deployment Phase: Argo CD continuously monitors target Kubernetes deployment manifests in GitHub and synchronizes container image tags to the AWS EKS cluster without downtime.
5. Observability & Alerting Phase: Prometheus collects node, pod, and cluster metrics, feeding Grafana visual dashboards for system health and load monitoring.

Key Engineering Challenges & Objectives

Prior to constructing this pipeline, traditional application deployment workflows presented multiple operational bottlenecks:

Operational & Technical Bottlenecks

1. Manual SSH Deployments: High risk of human error, inconsistent configuration drift, and zero deployment auditing.
2. Unscreened Vulnerabilities: Software dependencies and Docker base images pushed without automated security scanning.
3. Cost & Complexity of Multi-Cloud Execution: Balancing cloud resources effectively (utilizing Google Cloud credits for control plane master nodes while leveraging AWS EKS for scalable production workloads).
4. Deployment Friction & Downtime: Service interruption during manual pod restarts and uncoordinated container updates.
5. Lack of Real-Time Visibility: No centralized metrics or pod resource utilization dashboards to detect CPU throttles or memory leaks.

Strategic Project Objectives

1. Zero-Touch Continuous Integration: Automate code checkout, security verification, compilation, containerization, and distribution.
2. Shift-Left Security Implementation: Integrate SAST, SCA, and container vulnerability scanning into pipeline quality gates.
3. Codified Infrastructure Provisioning: Fully automate AWS EKS cluster lifecycle via repeatable Terraform templates.
4. GitOps Continuous Delivery: Ensure live Kubernetes cluster state matches declarative Git repository manifests using Argo CD.
5. Production Observability: Provide end-to-end monitoring using Prometheus and Grafana for pod performance and node metrics.

Step-by-Step Technical Implementation

The complete pipeline was implemented systematically across five technical phases:

Phase 1: Terraform Infrastructure Provisioning on AWS EKS

To maintain infrastructural transparency and avoid vendor lock-in, all cloud resources were codified using Terraform scripts executed directly from Cloud Shell interfaces.

• VPC & Subnet Module: Configured public/private subnets across multiple Availability Zones with Internet Gateways and route tables.
• IAM Role Provisioning: Enforced least-privilege IAM policies for the EKS Cluster control plane and Node Groups.
• AWS EKS Managed Node Groups: Created auto-scaling node groups on Ubuntu/Amazon Linux worker instances capable of dynamically scaling based on application pod load.
• Terraform Execution: Initialized backend state and applied changes cleanly using automated CLI scripts:
  Terminal Bash

  git clone https://github.com/rcheeez/meet.git
  cd meet/terraform/
  terraform init
  terraform plan
  terraform apply --auto-approve

Phase 2: Ansible Master Node Setup & Tool Configuration

To avoid manual package installation on master management instances, Ansible playbooks were authored to configure Ubuntu 22.04 LTS control nodes.

• Automated Ansible Deployment: Installed software properties, added official repositories, and executed system playbooks silently:
  Terminal Bash

  ansible-galaxy collection install community.docker
  ansible-playbook -i /etc/ansible/hosts master_server_tools_play.yml

• Provisioned Toolchain: Automatically installed Docker Engine, Jenkins master server, SonarQube Scanner CLI, OWASP Dependency Check tooling, Trivy security suite, Kubectl, and Helm CLI.
• SSH Public Key Authentication: Automated SSH key distribution for secure remote management across master nodes.

Phase 3: Multi-Stage Jenkins Security-First CI Pipeline

The Jenkins pipeline was structured into discrete execution stages, acting as strict security quality gates prior to artifact distribution:

1. Stage 1: Code Checkout — Cloned latest commit from GitHub repository.
2. Stage 2: SonarQube Code Quality Analysis — Performed static code analysis to enforce code coverage, code smell checks, and security rule validations.
3. Stage 3: OWASP Dependency Check — Analyzed application packages for known CVE vulnerabilities in third-party libraries.
4. Stage 4: Trivy File System & Image Scan — Scanned project directory files and base Docker images for high and critical vulnerabilities.
5. Stage 5: Docker Container Build & Tagging — Packaged application source code into optimized container images tagged with build numbers and commit hashes.
6. Stage 6: Docker Hub Artifact Delivery — Authenticated with Docker Registry credentials and pushed verified container images to Docker Hub.

Phase 4: GitOps Continuous Deployment with Argo CD

Continuous Deployment was decoupled from Jenkins and shifted to a GitOps pattern managed by Argo CD installed on the AWS EKS cluster.

• Declarative Manifest Synchronization: Argo CD monitored the Kubernetes application manifest repository for tag updates triggered by Jenkins builds.
• Automated Reconciliation Loop: Any drift between the target Git repository state and the active EKS cluster state was detected and synchronized automatically.
• Zero-Downtime Rolling Updates: Managed pod rolling updates, ensuring continuous availability of the Meet video application during deployments.

Phase 5: Observability Stack with Helm, Prometheus & Grafana

Full operational visibility was established by deploying a complete monitoring stack to the Kubernetes cluster using Helm charts:

• Prometheus Deployment: Collected node-exporter metrics, Kubernetes kube-state metrics, pod CPU/RAM usage, and network traffic.
• Grafana Dashboard Visualization: Provisioned custom dashboards showing real-time pod health, container restart loops, cluster resource saturation, and network bandwidth.
• Alert Thresholds: Configured automated alert notifications for memory spikes or node failure conditions.

Key Business & Engineering Impact

The automated CI/CD pipeline and EKS architecture delivered measurable improvements across velocity, security, and stability:

1. Elimination of Manual Release Overhead: Reduced manual deployment work by 80%, replacing SSH manual commands with automated Jenkins builds and Argo CD syncs.
2. Enhanced Security Posture: Prevented vulnerable third-party libraries and unpatched Docker base images from reaching production via automated SonarQube, OWASP, and Trivy scans.
3. Infrastructure Repeatability: Entire EKS cluster and networking stack can be torn down or re-provisioned in minutes using Terraform IaC scripts.
4. Zero-Downtime Upgrades: Kubernetes rolling updates enabled continuous feature delivery for Meet video application users without service interruption.
5. Proactive Cluster Monitoring: Real-time Grafana dashboards provided immediate visibility into pod resources and cluster capacity.

Key Learnings & DevOps Best Practices

Executing this end-to-end Kubernetes delivery pipeline provided valuable engineering insights:

1. Shift-Left Security is Essential: Integrating static code analysis and container vulnerability scanning into CI pipelines prevents costly security fixes in production.
2. GitOps Decouples Build from Deploy: Utilizing Argo CD for Kubernetes deployments provides better security than granting Jenkins direct cluster admin access.
3. Ansible Simplifies Infrastructure Bootstrapping: Automating node toolchain installation guarantees consistent environment state across management servers.
4. Infrastructure as Code Eliminates Configuration Drift: Provisioning cloud infrastructure with Terraform ensures complete auditability and rapid disaster recovery.

Technologies Used

Conclusion

This project demonstrates how combining Infrastructure as Code (Terraform), Server Configuration Automation (Ansible), Multi-Stage Security CI (Jenkins), GitOps Continuous Delivery (Argo CD), and Observability (Prometheus & Grafana) creates an automated, resilient, and enterprise-grade Kubernetes deployment pipeline on AWS EKS.

"High-velocity DevOps is not just about shipping code quickly; it is about building automated quality, security, and observability into every release."

By automating every stage of the software delivery pipeline, the Meet platform achieved high availability, rapid release cycles, and complete operational confidence on Kubernetes.

View on GitHub ↗