Distributed Task Queue with Retry & Dead Letter Handling

A production-grade asynchronous job processing system designed for reliability, observability, and horizontal scalability.

Overview

The Problem

Every production backend eventually needs to handle slow, unreliable, or resource-intensive operations outside the request-response cycle:

Sending emails and webhooks to third-party services that may be down
Generating PDFs and reports that take 5-30 seconds
Processing and resizing images uploaded by users
Handling spikes in traffic without dropping requests

Doing this synchronously means slow APIs, lost work on crashes, no retry capability, and zero visibility into what's failing.

The Solution

This system decouples job submission from job execution using a message queue architecture:

User Request (50ms) → Queue → Workers Process Asynchronously → Result Stored

Jobs are submitted via a REST API and immediately queued. Background workers pull jobs at their own pace, with automatic retries, failure tracking, and dead letter handling for permanently failed jobs.

This is the same pattern used internally at Stripe (webhook delivery), Uber (trip processing), Amazon (order fulfillment), and Netflix (video encoding).

Architecture

                                    ┌──────────────────┐
                                    │   Prometheus      │
                                    │   :9090           │
                                    └────────┬─────────┘
                                             │ scrapes
                                    ┌────────▼─────────┐
                                    │    Grafana        │
                                    │    :3000          │
                                    └──────────────────┘

┌──────────┐     ┌──────────────┐     ┌──────────────────────┐     ┌───────────────────┐
│  Client   │────▶│   FastAPI    │────▶│      RabbitMQ         │────▶│  Celery Workers    │
│           │     │   :8000      │     │      :5672            │     │                   │
└──────────┘     └──────┬───────┘     │                      │     │  ┌─────────────┐  │
                        │              │  ┌────────────────┐  │     │  │  critical    │  │
                        │              │  │ critical queue  │  │     │  │  (4 workers) │  │
                        ▼              │  │ default  queue  │  │     │  ├─────────────┤  │
                 ┌──────────────┐     │  │ bulk     queue  │  │     │  │  default     │  │
                 │  PostgreSQL  │     │  │ dlq      queue  │  │     │  │  (8 workers) │  │
                 │  :5432       │     │  └────────────────┘  │     │  └─────────────┘  │
                 │              │     └──────────────────────┘     └─────────┬─────────┘
                 │  jobs        │                                            │
                 │  job_logs    │◀───────────────────────────────────────────┘
                 └──────────────┘             status updates
                        ▲
                        │
                 ┌──────────────┐
                 │    Redis     │
                 │    :6379     │
                 │              │
                 │ idempotency  │
                 │ rate_limits  │
                 │ circuit_brk  │
                 │ celery_results│
                 └──────────────┘

Request Lifecycle

1. Client POSTs to /api/v1/jobs
2. Middleware injects Correlation ID (propagated through entire lifecycle)
3. Rate Limiter: client_id under 100 req/min? → 429 if not
4. Circuit Breaker: task_type circuit open? → 503 if yes
5. Idempotency: seen this key before? → return existing job (200)
6. Create Job in PostgreSQL (status: pending)
7. Dispatch to RabbitMQ via Celery (status: queued)
8. Return 201 with job_id in ~50ms

--- Worker Side ---

9.  Worker picks message from priority queue
10. before_start(): idempotency check, set status → running
11. Execute task (image resize / PDF gen / webhook delivery)
12. Success → on_success(): status → completed, store result
    OR
    Transient failure → on_retry(): increment retry_count, exponential backoff
    OR
    Max retries exhausted → on_failure(): status → dead_lettered, publish to DLQ

Key Features

Job Processing

Feature	Description
3 Task Types	Image processing (Pillow), PDF generation (ReportLab), Webhook delivery (httpx + HMAC signing)
Async by Design	Submit via REST, get job ID instantly, poll for status
Priority Queues	Jobs routed to `critical` (1-3), `default` (4-7), or `bulk` (8-10) queues
Configurable per Job	Each job can set its own `max_retries`, `priority`, and `callback_url`

Reliability & Fault Tolerance

Feature	Description
Exponential Backoff	`min(600s, 60s * 2^retry) + jitter` — prevents thundering herd
Dead Letter Queue	Failed jobs (max retries exhausted) captured with full error context
Idempotency	Redis SET NX (fast path) + PostgreSQL unique constraint (safety net)
At-Least-Once Delivery	`acks_late=True` — message re-delivered if worker crashes mid-task
Error Classification	`TransientError` (retry) vs `PermanentError` (fail immediately, no retry)
Circuit Breaker	Per-task-type, 3 states: closed → open → half_open. Prevents cascading failures

Observability

Feature	Description
8 Prometheus Metrics	Submission rate, completion status, P50/P95/P99 latency, retry rate, DLQ depth, circuit breaker state
Grafana Dashboard	7 pre-built panels, auto-provisioned on startup
Structured Logging	JSON via structlog, every log line includes correlation_id, job_id, task_type
5 Alert Rules	DLQ depth, failure rate, latency, retry rate, rate limit rejections
Full Audit Trail	`job_logs` table records every state transition with timestamps

Infrastructure

Feature	Description
Docker Compose	8 services, one command to start everything
Terraform	AWS deployment: VPC, ECS Fargate, RDS, ElastiCache, CloudWatch, auto-scaling
Health Checks	`/health` (liveness) and `/ready` (checks DB, Redis, RabbitMQ connectivity)
Worker Isolation	Separate pools for critical vs default queues — no priority inversion

System Design Deep Dive

Retry Strategy

Attempt 1: Execute immediately
  ↓ failure (TransientError)
Attempt 2: Wait ~60s  (60 * 2^0 + jitter)
  ↓ failure
Attempt 3: Wait ~120s (60 * 2^1 + jitter)
  ↓ failure
Attempt 4: Wait ~240s (60 * 2^2 + jitter)
  ↓ failure — max retries (3) exhausted
  ↓
Dead Letter Queue → alert fired → human reviews

Why jitter? Without it, if 1000 webhook deliveries fail because the upstream is down, all 1000 will retry at exactly the same time — creating a thundering herd that makes things worse. Random jitter spreads retries over time.

Circuit Breaker Pattern

CLOSED (normal) ──── failure rate > 50% ────▶ OPEN (rejecting)
    ▲                 (min 10 calls)              │
    │                                             │ 30s cooldown
    │                                             ▼
    └──── probe succeeds ──── HALF_OPEN (one request allowed)
                                    │
                          probe fails ──▶ OPEN (rejecting)

Why? If a webhook endpoint is down, sending 1000 requests to it wastes resources and makes recovery slower. The circuit breaker detects the pattern and fails fast (503) until the upstream recovers.

Idempotency Implementation

POST /api/v1/jobs { idempotency_key: "order-123-email" }

Step 1: Redis SET NX "idempotency:order-123-email" → job_id (TTL: 24h)
  - If SET succeeds → key is new → create job
  - If SET fails → key exists → lookup existing job → return it (200, not 201)

Step 2: PostgreSQL UNIQUE constraint on idempotency_key
  - Safety net for race conditions if Redis SET NX succeeds on two nodes simultaneously

Priority Queue Design

Priority 1-3  →  "critical" queue  →  worker-critical (4 concurrency)
Priority 4-7  →  "default"  queue  →  worker-default  (8 concurrency)
Priority 8-10 →  "bulk"     queue  →  worker-default  (8 concurrency)

+ RabbitMQ x-max-priority: 10 for fine-grained ordering within each queue

Why two tiers? Queue-level isolation prevents bulk image processing jobs from starving urgent webhook deliveries. Within each queue, RabbitMQ's native priority handles ordering.

Project Structure

distributed-task-queue/
├── src/
│   ├── config.py                     # Pydantic Settings — all config from env vars
│   ├── logging_config.py             # structlog JSON + correlation ID injection
│   │
│   ├── api/                          # HTTP Layer
│   │   ├── main.py                   # FastAPI app factory + Prometheus instrumentation
│   │   ├── deps.py                   # Dependency injection (DB, Redis, services)
│   │   ├── middleware.py             # Correlation ID middleware + request logging
│   │   ├── schemas.py               # Pydantic request/response models
│   │   └── routers/
│   │       ├── jobs.py               # POST /api/v1/jobs, GET, retry endpoint
│   │       └── health.py             # /health (liveness), /ready (readiness)
│   │
│   ├── worker/                       # Background Processing Layer
│   │   ├── celery_app.py             # Celery config, queue definitions, task routing
│   │   ├── callbacks.py              # DLQ handler (logs + alerts)
│   │   └── tasks/
│   │       ├── base.py               # BaseTask — retry logic, idempotency, DB updates, metrics
│   │       ├── image_processing.py   # Pillow resize/compress/convert
│   │       ├── pdf_generation.py     # ReportLab document builder
│   │       └── webhook_delivery.py   # httpx + HMAC signing + timeout handling
│   │
│   ├── domain/                       # Core Domain
│   │   ├── models.py                 # SQLAlchemy ORM: Job, JobLog
│   │   ├── enums.py                  # JobStatus, JobType, Priority, queue mapping
│   │   └── repository.py            # Job CRUD + audit log creation
│   │
│   ├── services/                     # Business Logic
│   │   ├── job_service.py            # Orchestrates: rate limit → circuit breaker → idempotency → create → dispatch
│   │   ├── idempotency.py            # Redis SET NX with TTL
│   │   ├── rate_limiter.py           # Sliding window counter per client_id
│   │   └── circuit_breaker.py        # Per-task-type, Redis-backed, 3-state machine
│   │
│   └── infra/                        # Infrastructure Clients
│       ├── database.py               # Async SQLAlchemy engine + session factory
│       ├── redis_client.py           # Redis connection pool
│       └── prometheus.py             # Custom metric definitions (8 metrics)
│
├── tests/
│   ├── conftest.py                   # Fixtures: fake Redis, SQLite, mocked Celery, HTTP client
│   ├── unit/
│   │   ├── test_schemas.py           # Pydantic validation (valid/invalid inputs)
│   │   ├── test_idempotency.py       # SET NX behavior, TTL, key reuse
│   │   ├── test_rate_limiter.py      # Sliding window, quota exhaustion, client isolation
│   │   ├── test_circuit_breaker.py   # State transitions, threshold, cooldown, independence
│   │   └── test_job_service.py       # Create, idempotency hit, rate limit, list, get
│   └── integration/
│       ├── test_api_jobs.py          # Full HTTP round-trips, pagination, 429, 422
│       └── test_retry_dlq.py         # Status transitions, retry counting, DLQ flow, audit logs
│
├── monitoring/
│   ├── prometheus.yml                # Scrape config for API + RabbitMQ
│   ├── alerting_rules.yml            # 5 alert rules (DLQ, failure rate, latency, etc.)
│   └── grafana/
│       ├── dashboards/
│       │   └── task_queue.json       # 7-panel dashboard (auto-provisioned)
│       └── provisioning/
│           ├── datasources/datasource.yml
│           └── dashboards/dashboards.yml
│
├── terraform/                        # AWS Infrastructure as Code
│   ├── main.tf                       # Root module — wires VPC, RDS, ElastiCache, ECS, monitoring
│   ├── variables.tf                  # All configurable parameters
│   ├── outputs.tf                    # API endpoint, DB endpoint, Redis endpoint
│   ├── modules/
│   │   ├── vpc/main.tf               # VPC, subnets, NAT gateway, route tables
│   │   ├── rds/main.tf               # PostgreSQL RDS, security groups, auto-generated password
│   │   ├── elasticache/main.tf       # Redis cluster, subnet group
│   │   ├── ecs/main.tf               # Fargate cluster, ALB, task definitions, services, IAM
│   │   └── monitoring/main.tf        # CloudWatch alarms, auto-scaling policies
│   └── environments/
│       ├── dev.tfvars                # Minimal resources (t3.micro)
│       └── prod.tfvars               # Production sizing (t3.medium, 3 API + 4 workers)
│
├── alembic/                          # Database Migrations
│   ├── env.py                        # Reads DATABASE_URL from env, converts async→sync driver
│   └── versions/
│       └── 001_initial_schema.py     # Jobs + job_logs tables with indexes
│
├── docker-compose.yml                # 8 services: API, 2 workers, RabbitMQ, Postgres, Redis, Prometheus, Grafana
├── Dockerfile.api                    # API image (uvicorn, 4 workers)
├── Dockerfile.worker                 # Worker image (Celery + Pillow + ReportLab)
├── pyproject.toml                    # Project metadata, pytest config, ruff config
├── requirements/
│   ├── base.txt                      # Production dependencies
│   └── dev.txt                       # Test dependencies (pytest, fakeredis, httpx)
├── .env.example                      # All environment variables documented
└── .gitignore

Quick Start

Prerequisites

Docker Desktop installed and running

Start Everything

git clone https://github.com/malav-250/distributed-task-queue.git
cd distributed-task-queue

# Start all 8 services
docker compose up --build -d

# Run database migrations
docker compose exec api alembic upgrade head

# Verify
curl http://localhost:8000/health

Access Points

Service	URL	Credentials
API Docs (Swagger)	http://localhost:8000/docs	—
Grafana Dashboard	http://localhost:3000	admin / admin
RabbitMQ Management	http://localhost:15672	guest / guest
Prometheus	http://localhost:9090	—
API Metrics	http://localhost:8000/metrics	—

Stop Everything

docker compose down        # Stop containers (keep data)
docker compose down -v     # Stop and delete all data

API Reference

Submit a Job

POST /api/v1/jobs

Request Body:

{
  "job_type": "webhook_delivery",        // Required: image_processing | pdf_generation | webhook_delivery
  "payload": {                            // Required: task-specific data
    "url": "https://httpbin.org/post",
    "body": {"event": "order.completed", "order_id": "ORD-123"},
    "secret": "whsec_signing_key",
    "timeout": 15
  },
  "client_id": "my-service",             // Required: used for rate limiting
  "priority": 1,                          // Optional: 1 (critical) to 10 (low), default: 5
  "max_retries": 5,                       // Optional: 0-10, default: 3
  "idempotency_key": "order-123-notify",  // Optional: prevents duplicate processing
  "callback_url": "https://my.app/hook"   // Optional: notified on completion
}

Response (201 Created):

{
  "job_id": "4b1c5555-8737-4d9c-8cd7-1ca0d1876c4e",
  "status": "queued",
  "created_at": "2026-04-05T04:13:31.060974Z"
}

Error Responses:

Status	When
`200`	Idempotency key already exists — returns existing job
`422`	Invalid request body (bad job_type, priority out of range)
`429`	Rate limit exceeded — includes `Retry-After` header
`503`	Circuit breaker open for this task type

Task-Specific Payloads

Image Processing

{
  "job_type": "image_processing",
  "payload": {
    "image_url": "https://example.com/photo.jpg",
    "operations": ["resize", "compress"],
    "width": 640,
    "height": 480,
    "format": "JPEG",
    "quality": 85
  },
  "client_id": "my-service"
}

PDF Generation

{
  "job_type": "pdf_generation",
  "payload": {
    "title": "Monthly Invoice",
    "content": [
      {"type": "heading", "text": "Invoice #12345"},
      {"type": "paragraph", "text": "Amount due: $1,250.00"},
      {"type": "paragraph", "text": "Due date: April 30, 2026"}
    ],
    "author": "Billing System",
    "page_size": "A4"
  },
  "client_id": "billing-service"
}

Webhook Delivery

{
  "job_type": "webhook_delivery",
  "payload": {
    "url": "https://partner-api.example.com/webhooks",
    "method": "POST",
    "body": {"event": "payment.succeeded", "amount": 1250},
    "headers": {"X-Custom-Header": "value"},
    "secret": "whsec_abc123",
    "timeout": 30
  },
  "client_id": "payment-service",
  "priority": 1
}

When a secret is provided, the payload is signed with HMAC-SHA256 and the signature is sent as X-Webhook-Signature: sha256=<hex>.

Check Job Status

GET /api/v1/jobs/{job_id}

Response:

{
  "job_id": "4b1c5555-8737-4d9c-8cd7-1ca0d1876c4e",
  "job_type": "webhook_delivery",
  "status": "completed",
  "priority": 1,
  "retry_count": 0,
  "max_retries": 3,
  "payload": {"url": "...", "body": {"event": "order.completed"}},
  "result": {"status_code": 200, "delivery_time_ms": 245.3},
  "error_message": null,
  "created_at": "2026-04-05T04:13:31Z",
  "started_at": "2026-04-05T04:13:31Z",
  "completed_at": "2026-04-05T04:13:32Z"
}

List Jobs (Paginated)

GET /api/v1/jobs?client_id=my-service&status=failed&page=1&page_size=20

Retry Dead-Lettered Job

POST /api/v1/jobs/{job_id}/retry

Resets retry_count to 0, re-queues the job. Only works for jobs in dead_lettered status.

Testing Guide

Run the Test Suite

pip install -r requirements/dev.txt
pytest tests/ -v --cov=src --cov-report=term-missing

Manual Testing Scenarios

1. Submit All 3 Job Types:

# Webhook (critical priority)
curl -X POST http://localhost:8000/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_type":"webhook_delivery","payload":{"url":"https://httpbin.org/post","body":{"event":"test"}},"client_id":"demo","priority":1}'

# PDF (normal priority)
curl -X POST http://localhost:8000/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_type":"pdf_generation","payload":{"title":"Test","content":[{"type":"paragraph","text":"Hello World"}]},"client_id":"demo"}'

# Image (low priority)
curl -X POST http://localhost:8000/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_type":"image_processing","payload":{"image_url":"https://picsum.photos/800/600","operations":["resize"],"width":400,"height":300},"client_id":"demo","priority":8}'

2. Test Idempotency (Duplicate Prevention):

# First call → 201 Created
curl -w "\nStatus: %{http_code}\n" -X POST http://localhost:8000/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_type":"webhook_delivery","payload":{"url":"https://httpbin.org/post","body":{}},"client_id":"test","idempotency_key":"unique-123"}'

# Same key → 200 OK (returns same job_id, no duplicate)
curl -w "\nStatus: %{http_code}\n" -X POST http://localhost:8000/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_type":"webhook_delivery","payload":{"url":"https://httpbin.org/post","body":{}},"client_id":"test","idempotency_key":"unique-123"}'

3. Test Failure → Retry → Dead Letter:

# Submit to a URL that always returns 500 (max 2 retries)
curl -X POST http://localhost:8000/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"job_type":"webhook_delivery","payload":{"url":"https://httpbin.org/status/500","body":{"test":true}},"client_id":"fail-test","max_retries":2}'

# Check status after 5+ minutes — will show dead_lettered with error
curl http://localhost:8000/api/v1/jobs/{job_id}

4. Test Rate Limiting:

# Submit 101+ requests rapidly from same client_id
for i in $(seq 1 105); do
  code=$(curl -s -o /dev/null -w "%{http_code}" -X POST http://localhost:8000/api/v1/jobs \
    -H "Content-Type: application/json" \
    -d '{"job_type":"pdf_generation","payload":{"title":"t","content":[{"type":"paragraph","text":"x"}]},"client_id":"flood-test"}')
  echo "Request $i: $code"
done
# Last few should return 429

5. Watch Worker Logs:

docker compose logs worker-critical --tail 30 -f
docker compose logs worker-default --tail 30 -f

6. Check RabbitMQ Queues:

Open http://localhost:15672 (guest/guest) → Queues tab. You'll see critical, default, bulk, dlq with message counts.

7. Check Grafana Metrics:

Open http://localhost:3000 (admin/admin) → "Distributed Task Queue Dashboard". Shows submission rate, latency percentiles, failures, circuit breaker state.

Configuration

All configuration is via environment variables. See .env.example.

Variable	Default	Description
`DATABASE_URL`	`postgresql+asyncpg://taskqueue:taskqueue@localhost:5432/taskqueue`	PostgreSQL connection
`CELERY_BROKER_URL`	`amqp://guest:guest@localhost:5672//`	RabbitMQ broker
`REDIS_URL`	`redis://localhost:6379/1`	Redis for services
`CELERY_RESULT_BACKEND`	`redis://localhost:6379/0`	Celery result storage
`RATE_LIMIT_PER_MINUTE`	`100`	Max jobs per client/minute
`DEFAULT_MAX_RETRIES`	`3`	Default retry count
`RETRY_BASE_DELAY`	`60`	Base backoff delay (seconds)
`RETRY_BACKOFF_MAX`	`600`	Max backoff cap (seconds)
`CB_FAILURE_THRESHOLD`	`0.5`	Circuit breaker: failure rate to trip
`CB_WINDOW_SECONDS`	`60`	Circuit breaker: monitoring window
`CB_COOLDOWN_SECONDS`	`30`	Circuit breaker: cooldown before half-open
`CB_MIN_CALLS`	`10`	Circuit breaker: min calls before evaluating
`IDEMPOTENCY_TTL_SECONDS`	`86400`	Idempotency key TTL (24 hours)
`LOG_LEVEL`	`INFO`	Logging level
`LOG_FORMAT`	`json`	`json` (production) or `console` (development)

Monitoring & Observability

Prometheus Metrics

Metric	Type	Labels	Description
`jobs_submitted_total`	Counter	job_type, client_id, priority	Total jobs submitted
`jobs_completed_total`	Counter	job_type, status	Jobs completed (completed/failed)
`job_duration_seconds`	Histogram	job_type	Processing time (P50/P95/P99)
`job_retries_total`	Counter	job_type	Total retry attempts
`jobs_dead_lettered_total`	Counter	job_type	Jobs moved to DLQ
`circuit_breaker_state`	Gauge	task_type	0=closed, 1=open, 2=half_open
`rate_limit_rejected_total`	Counter	client_id	Rate-limited requests
`worker_heartbeat_timestamp`	Gauge	worker_id	Last worker heartbeat

Alert Rules

Alert	Condition	Severity
`HighDLQDepth`	DLQ > 10 jobs for 5 min	Critical
`HighJobFailureRate`	>10% failure rate for 5 min	Warning
`HighJobLatency`	P95 > 300s for 10 min	Warning
`HighRetryRate`	>1 retry/sec for 10 min	Warning
`RateLimitRejections`	>5 rejections/sec for 5 min	Info

Failure Scenarios & Recovery

Scenario	What Happens	Recovery
Worker crashes mid-task	`acks_late=True` → RabbitMQ re-delivers message to another worker	Automatic. Idempotency check in `before_start()` prevents duplicate if job already completed
Upstream 5xx (transient)	`TransientError` → Celery retries with exponential backoff	Automatic. Up to `max_retries` attempts
Upstream 4xx (permanent)	`PermanentError` → job goes directly to `failed` status, no retry	Manual review. Fix payload and retry via `POST /api/v1/jobs/{id}/retry`
Max retries exhausted	Job moves to `dead_lettered` status, published to DLQ	Alert fires. Admin reviews via API, fixes issue, calls retry endpoint
Database connection lost	SQLAlchemy pool handles reconnection. `/ready` reports degraded	Automatic recovery when DB comes back
Redis down	Idempotency falls back to PostgreSQL unique constraint. Rate limiter fails open	Degraded mode — still works, just without Redis-speed dedup
RabbitMQ down	New job submissions fail with 500. Existing queued messages persist on disk	Jobs resume processing when RabbitMQ recovers
User submits duplicate	Idempotency key detected → returns existing job (200), no new work created	By design — client receives the original job response

Design Trade-offs

Decision	Choice	Alternative	Why This Choice
Broker	RabbitMQ	AWS SQS, Redis, Kafka	Free for local dev, native priority queues, management UI. SQS for production (Terraform provided)
Result Backend	Redis	PostgreSQL	Redis is 10-100x faster for high-throughput result polling. Avoids adding read load to the primary DB
Worker DB Access	Sync SQLAlchemy	Async	Celery workers are synchronous (fork-based). Async would add complexity with no benefit
Idempotency	Redis + PG constraint	DB-only or Redis-only	Redis for speed (<1ms). PG unique index as safety net for race conditions across API replicas
Priority	Queue isolation + in-queue priority	Single priority queue	Two-tier prevents starvation. Bulk jobs physically cannot block critical webhooks
Retry	Celery native	Custom retry loop	Battle-tested, handles edge cases (worker death, broker reconnection). `acks_late` is the key
Circuit Breaker	Redis-backed	In-memory	Shared across all API instances. In-memory would be per-process, inconsistent
Logging	structlog (JSON)	stdlib logging	Machine-parseable, correlation ID propagation, works with ELK/CloudWatch/Datadog
Migrations	Alembic	Raw SQL	Standard for SQLAlchemy, supports auto-generation, rollbacks, and version tracking

Scaling Strategy

Horizontal Scaling

                    ┌─── API Instance 1 (4 uvicorn workers) ───┐
Load Balancer ──────┼─── API Instance 2 (4 uvicorn workers) ───┼──▶ RabbitMQ ──▶ Workers
                    └─── API Instance 3 (4 uvicorn workers) ───┘
                                                                      │
                                                               ┌──────┴──────┐
                                                               │ Worker Pool │
                                                               │ Scale by    │
                                                               │ queue depth │
                                                               └─────────────┘

API: Stateless — scale behind a load balancer. Each instance runs 4 uvicorn workers
Workers: Scale independently per queue. Add worker-critical replicas for webhook spikes
Auto-scaling trigger: Queue depth. When queue_depth > threshold for N minutes → scale out

Vertical Scaling

PostgreSQL: Increase db_pool_size and db_max_overflow
Workers: Increase -c (concurrency) for CPU-bound, or --pool=gevent for I/O-bound tasks

Database Scaling

Read replicas for GET /api/v1/jobs listing queries
Partition job_logs by month for large installations
Archive completed jobs older than 30 days to cold storage (S3)

AWS Deployment (Terraform)

cd terraform
terraform init
terraform plan -var-file=environments/dev.tfvars
terraform apply -var-file=environments/dev.tfvars

What Gets Provisioned

Resource	Service	Sizing (dev)	Sizing (prod)
VPC	Networking	2 AZs, public + private subnets	Same
RDS PostgreSQL	Job storage	db.t3.micro	db.t3.medium
ElastiCache Redis	Caching + rate limiting	cache.t3.micro	cache.t3.medium
ECS Fargate (API)	HTTP service	1 task, 256 CPU, 512 MB	3 tasks, 512 CPU, 1 GB
ECS Fargate (Worker)	Job processing	1 task, 256 CPU, 512 MB	4 tasks, 1024 CPU, 2 GB
ALB	Load balancer	Single	With health checks
CloudWatch	Monitoring	CPU alarms	CPU + auto-scaling

Estimated Monthly Cost (Dev)

RDS db.t3.micro: ~$15
ElastiCache cache.t3.micro: ~$12
ECS Fargate (2 tasks): ~$20
ALB: ~$16
NAT Gateway: ~$32
Total: ~$95/month

Database Schema

`jobs` Table

Column	Type	Description
`id`	UUID (PK)	Unique job identifier
`idempotency_key`	VARCHAR(255), UNIQUE	Client-provided dedup key
`client_id`	VARCHAR(128), INDEXED	For rate limiting
`job_type`	VARCHAR(50)	image_processing, pdf_generation, webhook_delivery
`priority`	SMALLINT (1-10)	1=critical, 10=low
`status`	VARCHAR(20)	pending → queued → running → completed/failed/dead_lettered
`payload`	JSONB	Task-specific input
`result`	JSONB	Task output on completion
`retry_count`	SMALLINT	Current attempt number
`max_retries`	SMALLINT	Max allowed retries
`error_message`	TEXT	Last error description
`celery_task_id`	VARCHAR(255)	Celery task correlation
`callback_url`	VARCHAR(2048)	Optional completion webhook
`created_at`	TIMESTAMPTZ	Job creation time
`updated_at`	TIMESTAMPTZ	Last modification
`started_at`	TIMESTAMPTZ	When worker picked it up
`completed_at`	TIMESTAMPTZ	Terminal state timestamp

`job_logs` Table (Audit Trail)

Column	Type	Description
`id`	BIGSERIAL (PK)	Auto-increment
`job_id`	UUID (FK → jobs)	Parent job
`event`	VARCHAR(50)	created, queued, started, retried, completed, failed, dead_lettered
`detail`	JSONB	Event-specific context
`created_at`	TIMESTAMPTZ	Event timestamp

Tech Stack

Layer	Technology	Why
API Framework	FastAPI	Async, auto-docs, Pydantic validation, dependency injection
Task Queue	Celery 5.4	Industry standard, battle-tested, rich ecosystem
Message Broker	RabbitMQ 3.13	Priority queues, management UI, persistence, free
Database	PostgreSQL 16	JSONB for flexible payloads, robust indexing, ACID
Cache / Services	Redis 7	Sub-millisecond ops for idempotency, rate limiting, results
Monitoring	Prometheus + Grafana	Industry standard, free, rich query language
Image Processing	Pillow 10.4	Standard Python imaging library
PDF Generation	ReportLab 4.2	Production PDF library, no browser dependency
HTTP Client	httpx	Async-capable, timeout handling, modern API
Logging	structlog	JSON output, context propagation, zero config
Migrations	Alembic	SQLAlchemy native, version tracking, auto-generation
IaC	Terraform	Multi-cloud, declarative, module system
Containers	Docker Compose	Single-command local development

License

MIT

Built with distributed systems principles for production reliability.

Latest commit

History

Folders and files

Repository files navigation