Teaching Reference · Software Engineering

The Classic CI/CD Pipeline

A systematic walkthrough of the five-stage pipeline — from local development through GitHub, Docker, and cloud deployment — with practical implementation for GCP-native stacks.

Local → GitHub → CI → Docker → Cloud Run FastAPI · Next.js · GCP GitHub Actions Artifact Registry
Stage 1
Local Dev
Code + hooks
Stage 2
GitHub
PR + branch rules
Stage 3
CI Build
Test + lint
Stage 4
Registry
Versioned images
Stage 5
Cloud Deploy
Canary + promote
Before we begin · Foundation concepts

Background for students new to Git, GitHub, and Docker

The CI/CD pipeline sits on top of several tools that professional developers use every day. If these are new to you, start here. Understanding the purpose of each tool makes the pipeline feel logical rather than overwhelming.

Version control — Git

Git
A tool that tracks every change you make to your code, over time. It runs on your local computer. Every time you save a meaningful change, you create a "commit" — a snapshot of your code at that moment, with a message explaining what changed.
Git is like "Track Changes" in Microsoft Word, but for an entire project folder — and far more powerful. Every version is saved, and you can go back to any of them.
git commit
The act of saving a snapshot of your code. You write a short message describing what you did — for example, git commit -m "Add Stripe checkout endpoint". This snapshot is stored permanently in your project's history.
git push
Sending your local commits up to a shared server (GitHub) so others can see them, and so they are safely backed up. "Push" means you are pushing your work out from your machine to the internet.
Committing is like saving a Word document on your laptop. Pushing is like uploading it to Google Drive so your team can access it.
branch
A parallel copy of your code where you can work independently without affecting the main version. When your feature is ready, you merge it back. The main production-ready branch is typically called main. Developers work on short-lived branches like feature/stripe-checkout.
Think of a document you duplicate before making big edits. You work on the copy, and only merge it back into the original when it's ready.

GitHub — the shared home for code

GitHub
A website (github.com) that stores your Git project in the cloud and adds collaboration tools on top. It is not the same as Git — Git is the tool, GitHub is the platform that hosts your project and lets teams work together.
Git is the engine. GitHub is the garage where you park and share the car with your team.
repository (repo)
The project folder — all your code, history, and settings in one place. Your GLSChineseHub repository contains your FastAPI backend, Next.js frontend, and all the configuration files that make the CI/CD pipeline work.
Pull Request (PR)
A formal request to merge your branch into another branch (usually main). Before the merge happens, teammates can review the changes, leave comments, and approve or reject. The name "Pull Request" means you are asking the team to "pull" your changes in.
A PR is like submitting a draft document to an editor for review before it gets published. Nobody can publish until the editor approves.
branch protection
Rules on GitHub that prevent code from being merged unless certain conditions are met — such as requiring a teammate's approval, or requiring all automated tests to pass. This is the "policy enforcement" that makes GitHub more than just file storage.
GitHub Actions
An automation system built into GitHub. You write instructions (in .yml files) that tell GitHub to automatically run tasks whenever something happens — like "whenever someone pushes to the develop branch, run all the tests." This is the engine of the CI/CD pipeline.

Docker — packaging your application

Docker
A tool that packages your application and everything it needs to run — code, libraries, settings — into a single portable unit called a container. The container runs identically on your laptop, on the CI server, and in the cloud. "It works on my machine" stops being an excuse.
Docker is like a shipping container. The goods inside are your app. The container fits on any ship (any computer), and the contents arrive exactly as packed — every time.
Dockerfile
A text file with step-by-step instructions for building your container. It describes the starting point (e.g., "start with Python 3.11"), what to install, what files to copy in, and what command to run when the container starts. One Dockerfile = one recipe for one image.
image vs container
An image is the blueprint. A container is a running instance of that blueprint. You build the image once. You can run many containers from the same image.
An image is like a cake recipe. A container is a cake you baked from that recipe. You can bake as many cakes as you like from the same recipe, and they are all identical.
docker-compose
A tool for running multiple containers together on your local machine. Your project needs a FastAPI backend, a Next.js frontend, and a Postgres database — all three running at once. docker-compose.yml defines them and starts them all with one command: docker-compose up.
registry
A storage system for Docker images — similar to how GitHub stores code, a registry stores images. Google's version is called Artifact Registry. When CI builds your image, it pushes it to the registry. When Cloud Run deploys, it pulls from the registry.

Why do we use .yml sometimes and .md other times?

This is one of the most common points of confusion for new developers. Both are plain text files — but they serve completely different purposes and are read by completely different tools.

.yml / .yaml
YAML — "Yet Another Markup Language"
Purpose: instructions for machines. YAML is a structured data format that computers read to understand configuration and automation rules. It uses indentation (spaces) to show structure — a wrong space breaks the file.

In CI/CD, GitHub Actions reads your .yml files to know what steps to run, in what order, and under what conditions. The computer executes these instructions literally.
ci.yml deploy.yml docker-compose.yml .pre-commit-config.yaml
.md
Markdown — lightweight document format
Purpose: readable documents for humans. Markdown is a simple way to write formatted text — headings, bold, lists, links — using plain characters like # for headings and ** for bold. GitHub automatically renders .md files as formatted pages.

In a repo, .md files are documentation — they explain the project to humans, not to computers.
README.md CLAUDE.md pull_request_template.md CONTRIBUTING.md
Rule of thumb If a computer reads and executes it → .yml. If a human reads it to understand something → .md. A README.md tells a new developer what the project is. A ci.yml tells GitHub Actions what to do when code is pushed.

Other file types you will encounter in a CI/CD repo:

ExtensionNamePurposeRead by
.yml / .yamlYAMLAutomation instructions, configurationGitHub Actions, Docker Compose, pre-commit
.mdMarkdownHuman-readable documentationDevelopers, GitHub (renders as web page)
DockerfileDockerfileInstructions for building a Docker imageDocker engine
.envEnv fileSecret values for local developmentYour app at runtime — never committed to Git
.tomlTOMLPython tool configuration (ruff, black, mypy)Python tooling
.jsonJSONStructured data, Node.js confignpm, APIs, many tools

The three principles that hold everything together

01
Artifact immutability — Build once, deploy everywhere. The same Docker image moves through staging and production. Environment differences come from config and secrets injection, never from rebuilding.
02
Promotion gates — Nothing bypasses a stage. The pipeline enforces this by only deploying what passed the previous gate. Code never skips an environment.
03
Secrets segregation — Local uses .env, CI uses GitHub Secrets, Cloud uses Secret Manager or Vault. No secret ever travels as a value baked into an image layer.
Stage 1

Local Development

The developer's machine is where the feedback loop must be fastest. Everything that can break in CI should surface here first.

docker-compose

Run the same Docker image locally via docker-compose up. Local source code is mounted as a volume for hot reload. Env vars come from a .env file — gitignored, never committed.

Pre-commit hooks

Scripts that run automatically before git commit finalizes. If any hook fails, the commit is blocked — code never reaches GitHub.

Unit tests locally

Run pytest and npm test before pushing. The same tests that run in CI should run locally — no surprises.

Pre-commit hooks — the tool breakdown

ToolLanguageTypeWhat it does
blackPythonFormatterAuto-rewrites code to consistent style. Opinionated — no config needed.
ruffPythonLinterReplaces flake8 + isort. 100× faster (Rust). Flags unused imports, bad patterns.
mypyPythonType checkerCatches type errors that linters miss. Slower — often run in CI not commit.
eslintJS/TSLinterRules + auto-fix for JavaScript and TypeScript. Catches bugs and bad patterns.
prettierJS/TS/JSONFormatterOpinionated style rewriter for the JS ecosystem.
tsc --noEmitTypeScriptType checkerChecks type annotations without emitting compiled output.
detect-secretsAnySecurityScans every staged file for API keys, tokens, and private key patterns.
Key distinction: Formatters (Black, Prettier) rewrite code automatically — no human decision needed. Linters (Ruff, ESLint) flag problems and leave the fix to the developer. Both are complementary — most projects use both.

Pre-commit config (.pre-commit-config.yaml)

.pre-commit-config.yaml YAML
repos:
  - repo: https://github.com/psf/black
    rev: 24.3.0
    hooks:
      - id: black

  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.4.1
    hooks:
      - id: ruff
        args: [--fix]

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: detect-private-key
      - id: trailing-whitespace
      - id: end-of-file-fixer

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
Stage 2

GitHub — Policy Enforcement

GitHub is your policy enforcement layer, not just storage. Branch protection rules, PR gates, and Actions triggers form the quality gate between a developer's machine and the build system.

Branch structure

feature/*
short-lived
→
develop
integration
→
main
prod-ready
→
v1.4.2
tagged release

Branch protection rules

main (strictest)

Require PR before merge · 1+ approvals · status checks must pass · branches must be up to date · no bypass allowed.

develop

Status checks must pass (CI green) · allow direct push for solo developer · no unreviewed merges from external contributors.

GitHub Secrets

Settings → Secrets and Variables → Actions. Scope secrets to environments — create staging and production environments, restrict which branches deploy to each.

GitHub Actions trigger (.github/workflows/ci.yml)

.github/workflows/ci.yml — trigger block YAML
on:
  push:
    branches: [develop]       # every push to develop triggers CI
  pull_request:
    branches: [main, develop]  # every PR targeting either branch
Stage 3

CI Build — GitHub Actions

CI has one job: produce a verified, immutable artifact. Two parallel jobs run simultaneously — backend (Python/FastAPI) and frontend (Next.js). Total CI time equals whichever takes longer.

Key insight: The services: block spins up a real Postgres container alongside the test runner on the same Docker network. Your test code connects to localhost:5432 as if Postgres were local. Without a health check, tests may start before Postgres is ready and fail with a connection error rather than a real test failure.

Complete ci.yml

.github/workflows/ci.yml YAML
name: CI

on:
  push:
    branches: [develop]
  pull_request:
    branches: [main, develop]

jobs:
  backend:
    name: Python / FastAPI
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER: testuser
          POSTGRES_PASSWORD: testpass
          POSTGRES_DB: gls_test
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    env:
      DATABASE_URL: postgresql://testuser:testpass@localhost:5432/gls_test
      ENVIRONMENT: test

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: pip

      - name: Install dependencies
        run: pip install -r requirements.txt ruff black pytest pytest-asyncio httpx

      - name: Run migrations
        run: alembic upgrade head

      - name: Lint (ruff)
        run: ruff check .

      - name: Format check (black)
        run: black --check .   # --check fails if files need reformatting

      - name: Run tests
        run: pytest tests/ -v

  frontend:
    name: Next.js / TypeScript
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: npm

      - name: Install dependencies
        working-directory: ./frontend
        run: npm ci   # clean lockfile install — faster, deterministic

      - name: Lint (ESLint)
        working-directory: ./frontend
        run: npm run lint

      - name: Type check
        working-directory: ./frontend
        run: npx tsc --noEmit

      - name: Build check
        working-directory: ./frontend
        run: npm run build
        env:
          NEXT_PUBLIC_API_URL: http://localhost:8000

Step-by-step breakdown — backend

01
actions/checkout@v4 — clones the repo into the runner. Without this, the runner has no code.
02
setup-python with cache: pip — cache key is derived from requirements.txt hash. pip only re-downloads packages when that file changes. Saves 30–60 seconds per run.
03
alembic upgrade head — runs before tests. The test Postgres is empty on startup — migrations apply the schema so test queries work against real tables.
04
ruff check . — lints entire codebase. Any rule violation fails the job immediately.
05
black --check . — the --check flag is critical. Without it, black would reformat files in the runner and then pass — useless. With it, CI fails if any file would be reformatted, forcing the developer to run black locally first.
06
pytest tests/ -v — full test suite against live test Postgres. -v gives verbose output so failed test names are visible in the CI log.

Step-by-step breakdown — frontend

01
npm ci vs npm install — npm ci deletes node_modules and installs exactly what's in package-lock.json. No resolution, no version negotiation. Fails if lockfile is out of sync with package.json.
02
tsc --noEmit — type-checks the entire TypeScript codebase without emitting compiled output. Catches type errors that ESLint won't catch.
03
npm run build — full Next.js production build. Catches dynamic import failures, missing env vars, API route misconfigurations, and bundle size issues that neither type checking nor linting will find.
Stage 4

Docker Registry

The registry is your artifact store and the single source of truth for what gets deployed. Images are immutable once pushed — a SHA-tagged image is never rebuilt.

One Dockerfile, three contexts. The same Dockerfile produces the same image. What differs is how and where it runs: locally via docker-compose (with volume mounts and .env), in CI via docker build (with GitHub secrets), and in Cloud Run (with Secret Manager). The image itself is always identical.

Tagging strategy — SHA tag vs Semver tag

A Docker image tag is a label attached to a specific image, so you can refer to it by name later. The same image can carry multiple tags simultaneously — like a product that has both a serial number and a model name printed on the box. Understanding the difference between SHA and semver tags is essential for safe deployments and reliable rollback.

Primary tag — always applied
SHA tag
backend:abc12ef
What is a SHA? SHA stands for "Secure Hash Algorithm." Git automatically generates a unique SHA fingerprint for every single commit you make — based on the exact content of the code. It looks like a long string of letters and numbers: a3f92bc1d88e4f301c5b2e9a7d6c0e42f1b83d90. In practice we use just the first 7 characters: abc12ef.
Why is it useful as a tag? Because it is a permanent, unique fingerprint of a specific commit. No two commits ever share the same SHA. When your Docker image is tagged with a SHA, you can always trace it back to exactly which line of code was in that image.
Real scenario A bug is reported in production on a Tuesday. You check Cloud Run — it is running revision backend:abc12ef. You search your GitHub commits for abc12ef and find the exact commit, the exact developer, the exact line of code that caused it. No guessing.
Applied: On every single push to CI. Every image built in your pipeline carries a SHA tag. This is non-negotiable.
Secondary tag — on release only
Semver tag
backend:v1.4.2
What is Semver? Semver stands for "Semantic Versioning." It is a widely adopted convention for numbering software versions using three numbers: MAJOR.MINOR.PATCH. Each number has a specific meaning that communicates to humans what kind of change was made.
v1.4.2
MAJOR · 1
Breaking changes. Old clients may stop working. Increment when you change an API in an incompatible way.
MINOR · 4
New features, backwards compatible. Existing clients still work. Increment when you add something new.
PATCH · 2
Bug fixes only, backwards compatible. Nothing new, nothing broken. Increment when you fix something.
Applied: Only when a developer creates an official Git release tag (e.g. git tag v1.4.2). Not every commit — only intentional milestones. Helps stakeholders, clients, and the team track what version is running.
How SHA and semver tags work together on the same image

One image can carry multiple tags at the same time. When you cut a release, CI applies both — the SHA for machine traceability, and the semver for human readability. They both point to the exact same image bytes.

# The same image pushed with two tags:
docker build \
  -t .../backend:abc12ef \   # SHA tag — always
  -t .../backend:v1.4.2 \    # Semver tag — on release only
  ./backend

# Both tags point to identical image bytes in the registry:
.../backend:abc12ef  →  sha256:f3a9b2c1...
.../backend:v1.4.2   →  sha256:f3a9b2c1...  # same digest
Rule — never deploy "latest" to production

latest is a mutable tag — it silently moves to a new image every time you push. If your Cloud Run service is configured to use backend:latest, then a bad push at 2am automatically becomes production. Rollback becomes impossible because the tag no longer points to the old image.

✗ WRONG
gcloud run deploy backend \
  --image .../backend:latest
Mutable — could be any image
✓ CORRECT
gcloud run deploy backend \
  --image .../backend:abc12ef
Immutable — exact known commit

Quick reference — all three tag types

Tag typeExampleMutable?Applied whenUsed by
SHA tag backend:abc12ef No — immutable Every CI push Cloud Run deploy, rollback, debugging
Semver tag backend:v1.4.2 No — immutable On official git release tag only Stakeholders, changelogs, milestone tracking
latest tag backend:latest Yes — dangerous Every push (auto) Local dev only — never production

Build and push job (deploy.yml)

.github/workflows/deploy.yml — build-and-push job YAML
jobs:
  build-and-push:
    name: Build + push images
    runs-on: ubuntu-latest

    outputs:
      sha: ${{ steps.tag.outputs.SHA }}

    steps:
      - uses: actions/checkout@v4

      - id: auth
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}

      - name: Configure Docker for Artifact Registry
        run: gcloud auth configure-docker asia-east1-docker.pkg.dev --quiet

      - name: Set image tag
        id: tag
        run: echo "SHA=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

      - name: Build backend image
        run: |
          docker build \
            -t asia-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/images/backend:${{ steps.tag.outputs.SHA }} \
            ./backend

      - name: Push backend image
        run: |
          docker push \
            asia-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/images/backend:${{ steps.tag.outputs.SHA }}

      - name: Build frontend image
        run: |
          docker build \
            -t asia-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/images/frontend:${{ steps.tag.outputs.SHA }} \
            ./frontend \
            --build-arg NEXT_PUBLIC_API_URL=${{ secrets.STAGING_API_URL }}

      - name: Push frontend image
        run: |
          docker push \
            asia-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/images/frontend:${{ steps.tag.outputs.SHA }}

GCP Artifact Registry setup (one-time)

terminal BASH
# Create the repository
gcloud artifacts repositories create images \
  --repository-format=docker \
  --location=asia-east1 \
  --description="GLS_CH_Hub container images"

# Enable vulnerability scanning
gcloud services enable containerscanning.googleapis.com

# Set retention policy — keep last 10 images per service
gcloud artifacts repositories set-cleanup-policies images \
  --location=asia-east1 \
  --policy='[{"name":"keep-last-10","action":{"type":"Keep"},"condition":{"tagState":"tagged","mostRecentVersions":{"keepCount":10}}}]'
Stage 5

Cloud Deployment — Cloud Run

Deployment happens in two hops: staging first, then production. The same image that passed staging goes to production — not a different build. Traffic splitting enables safe canary deployments with instant rollback.

How Cloud Run revisions work

Every deploy = new revision

A revision is defined by image SHA + env vars + secret versions. Change any of the three and you get a new revision. Old revisions stay available at 0% traffic.

Traffic splitting

Route traffic percentages to specific revisions. Deploy new revision at 0%, canary to 10%, watch metrics, promote to 100%. Blast radius contained at each step.

Instant rollback

Old revision never deleted — just at 0% traffic. Rollback = one command pointing 100% traffic back to previous revision. No rebuild, no retest, under 30 seconds.

Staging deploy + smoke test

.github/workflows/deploy.yml — staging job YAML
  deploy-staging:
    name: Deploy to staging
    needs: build-and-push
    runs-on: ubuntu-latest

    steps:
      - id: auth
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}

      - name: Deploy backend to staging
        run: |
          gcloud run deploy gls-backend-staging \
            --image asia-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/images/backend:${{ needs.build-and-push.outputs.sha }} \
            --region asia-east1 \
            --platform managed \
            --set-secrets DATABASE_URL=staging-db-url:latest,STRIPE_SECRET_KEY=stripe-key-staging:latest \
            --set-env-vars ENVIRONMENT=staging \
            --allow-unauthenticated

      - name: Smoke test staging
        run: |
          sleep 10
          curl --fail https://gls-backend-staging-xxxx.run.app/health
          curl --fail https://gls-frontend-staging-xxxx.run.app/

Production canary deploy

.github/workflows/deploy.yml — production job YAML
  deploy-production:
    name: Deploy to production
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production   # requires manual approval in GitHub

    steps:
      - id: auth
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}

      # Deploy new revision with NO traffic yet
      - name: Deploy to prod (no traffic)
        run: |
          gcloud run deploy gls-backend-prod \
            --image asia-east1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/images/backend:${{ needs.build-and-push.outputs.sha }} \
            --region asia-east1 \
            --no-traffic \
            --tag canary \
            --set-secrets DATABASE_URL=prod-db-url:latest,STRIPE_SECRET_KEY=stripe-key-prod:latest,PINECONE_API_KEY=pinecone-key:latest \
            --set-env-vars ENVIRONMENT=production

      # Send 10% to new revision — canary phase
      - name: Canary — 10% traffic
        run: |
          gcloud run services update-traffic gls-backend-prod \
            --region asia-east1 \
            --to-tags canary=10

      - name: Monitor canary (60s)
        run: sleep 60

      # Promote fully
      - name: Promote to 100%
        run: |
          gcloud run services update-traffic gls-backend-prod \
            --region asia-east1 \
            --to-tags canary=100

Rollback command

terminal — instant rollback BASH
# Point 100% of traffic back to any previous revision immediately
gcloud run services update-traffic gls-backend-prod \
  --region asia-east1 \
  --to-revisions=PREVIOUS-REVISION-NAME=100

Required health endpoint (FastAPI)

backend/main.py Python
@app.get("/health")
def health():
    # Simple, fast, no auth, no DB call
    # curl --fail will exit non-zero if this returns anything other than 200
    return {"status": "ok", "env": settings.ENVIRONMENT}
Secret injection at runtime: The --set-secrets flag syntax stripe-key-prod:latest means Cloud Run pulls the latest version from Secret Manager at container start. Your app reads secrets with os.environ.get("STRIPE_SECRET_KEY") — no SDK needed, no secret baked into the image.

Secrets across all five stages

ContextSecrets sourceHow it works
Local dev.env filegitignored, on your machine only, loaded by docker-compose
GitHub Actions CIGitHub Secrets${{ secrets.NAME }} — scoped to repo or environment
Docker imageBuild args (NEXT_PUBLIC_* only)Baked at build time — only for Next.js public vars. Never for API keys.
Cloud Run stagingGCP Secret Manager--set-secrets KEY=secret-name:latest — injected at container start
Cloud Run productionGCP Secret ManagerSame pattern — separate secret names/values from staging

Recommended repo layout

GLSChineseHub/ TREE
GLSChineseHub/
├── .github/
│   ├── workflows/
│   │   ├── ci.yml          ← Stage 3: runs on push/PR
│   │   └── deploy.yml      ← Stages 4+5: runs on merge to main
│   └── pull_request_template.md
├── backend/             ← FastAPI app
│   ├── Dockerfile
│   ├── requirements.txt
│   └── main.py
├── frontend/            ← Next.js app
│   ├── Dockerfile
│   └── package.json
├── .pre-commit-config.yaml  ← Stage 1 hooks
├── docker-compose.yml       ← Stage 1 local dev
└── .env.example             ← template — never commit .env itself