A comprehensive repository for learning Python DevOps practices with enterprise-grade CI/CD pipeline, security scanning, and AWS integration.

python-devops-course/
├── src/                           # Source code
│   ├── aws/                      # AWS management tools
│   │   └── aws_iam_manager.py    # Complete AWS IAM role & policy management
│   ├── cli/                      # Command line interfaces
│   │   ├── helloclick.py         # Tokenizer CLI tool
│   │   ├── hello-click2.py       # Enhanced CLI with name processing
│   │   ├── gcli.py               # File search utility
│   │   ├── hello.py              # AWS S3 buckets listing tool
│   │   └── lambda_function.py    # AWS Lambda function example
│   └── utils/                    # Utility functions
│       ├── magic_stuff.py        # Magic utility functions
│       └── marco.py              # Marco utility functions
├── tests/                        # Test suite
│   ├── test_helloclick.py       # Tokenizer tests
│   └── test_gcli.py              # File search tests
├── scripts/                      # Deployment and automation scripts
│   ├── deploy_lambda.py          # Full Lambda deployment with role creation
│   ├── deploy_lambda_simple.py  # Simplified Lambda deployment
│   └── deploy_with_role.py       # Lambda deployment with existing role
├── lambda_packages/              # Lambda deployment packages
│   └── python-devops-lambda.zip  # Current deployment package
├── docs/                         # Documentation
│   └── AWS_IAM_SETUP.md          # AWS setup guide
├── requirements.txt              # Dependencies (pylint, click, pytest, boto3, ipython, pandas)
├── Makefile                      # Build automation with 15+ commands
├── pytest.ini                   # Test configuration
├── .gitignore                    # Version control exclusions
├── README.md                     # Project overview
└── PROJECT_STRUCTURE.md          # Detailed structure guide

• Security Scanning: Bandit, Semgrep, Safety with SARIF uploads
• Dependency Review: Automated vulnerability scanning for PRs
• SARIF Integration: Security results surface in GitHub Security tab
• Conditional Failure: Soft failures on dev, hard failures on main

• Parallel Testing: pytest-xdist with conditional strategies
• Build Once, Deploy Many: Artifact reuse across environments
• Smart Caching: pip cache with dependency path monitoring
• Fail-Fast Control: Resilient to single environment failures

• GitHub Environments: Approval gates and environment-specific secrets
• Matrix Strategies: Parallel deployment to multiple environments
• Environment-Specific: Different configurations per environment
• URL Tracking: Deployment URL history and monitoring

• Job Summaries: Rich markdown summaries in GitHub UI
• Composite Actions: DRY principle for reusable workflows
• Ruff Integration: 10-100x faster linting than pylint
• Artifact Management: Conditional uploads with retention policies

# Install dependencies
make install

# Run tests
make test

# Setup AWS IAM
make aws-setup

# Get help
make help

A modern, responsive web interface for monitoring deployment status across all environments in real-time.

• Real-time Status Monitoring: Visual status indicators for all environments (ephemeral, dev, staging, prod)
• Color-coded Environment Cards: Green (success), Red (failure), Yellow (pending)
• Detailed Deployment Information: Region, last deployed time, duration, commit hash
• Summary Statistics: Overview of successful, failed, and pending deployments
• Responsive Design: Works on desktop and mobile devices
• RESTful API: Complete API for deployment status management

The dashboard shows real-time deployment status with color-coded environment cards, detailed deployment information, and summary statistics.

# Install dependencies
pip install flask

# Run the dashboard
python run_dashboard.py

# Access the dashboard
open http://localhost:5001

• GET / - Main dashboard interface
• GET /api/deployments - All deployment statuses
• GET /api/deployments/<env> - Specific environment status
• POST /api/deployments/<env>/status - Update environment status
• GET /api/health - Health check
• GET /api/summary - Deployment summary statistics

Each environment displays:

• Environment name and status badge
• AWS region and deployment details
• Last deployment timestamp
• Deployment duration and commit hash
• Direct link to environment URL

• Total deployments across all environments
• Successful deployment count
• Failed deployment count
• Pending deployment count

• Refresh button for real-time updates
• Last updated timestamp
• Responsive grid layout
• Smooth animations and transitions

The dashboard integrates seamlessly with the CI/CD pipeline:

1. Automatic Status Updates: Deployment jobs update status via API
2. Real-time Monitoring: View deployment progress across environments
3. Failure Detection: Immediate visual feedback on deployment failures
4. Environment Tracking: Monitor deployment history and performance

For detailed documentation, see DEPLOYMENT_DASHBOARD.md.

• AWS IAM Manager: Complete Cloud9 service role and policy management
• Features: Role creation, policy attachment, user management
• Documentation: See docs/AWS_IAM_SETUP.md

• Hello Click: Interactive CLI tools with Click framework
• Features: Tokenization, command line interfaces, AWS integration
• Examples:
  • helloclick.py - Text tokenization CLI tool
  • gcli.py - File search utility with glob patterns
  • hello.py - AWS S3 buckets listing tool
  • hello-click2.py - Enhanced CLI with name processing
  • lambda_function.py - AWS Lambda function example

• Flask Application: Modern web application with deployment dashboard
• Features: Real-time deployment monitoring, RESTful API, responsive UI
• Components:
  • application.py - Main Flask web application
  • deployment_api.py - Deployment dashboard API server
  • templates/ - HTML templates for web interface
  • templates/deployment-dashboard.html - Modern deployment dashboard UI

• Magic Functions: Utility functions and helpers
• Features: Common functionality, helper scripts
• Examples: magic_stuff.py, marco.py

• Python 3.8+
• AWS CLI configured (for AWS features)
• Virtual environment

# Clone repository
git clone <repository-url>
cd python-devops-course

# Install dependencies
make install

# Activate virtual environment
source venv/bin/activate

# Run all tests
make test

# Run specific test suites
make test-cli
make test-aws

# Run with coverage
make test

# Lint code
make lint

# Format code
make format

# Clean up
make clean

• Cloud9 Service Role: Automated role creation with proper policies
• User Management: IAM user creation and role attachment
• Security: AWS managed policies and best practices

# Configure AWS credentials
aws configure

# Setup AWS IAM
make aws-setup

# Test AWS functionality
make aws-test

• ✅ Cloud9 service role creation
• ✅ AWS managed policy attachment
• ✅ User management and role assumption
• ✅ Access key creation
• ✅ Comprehensive error handling

# Security scanning with SARIF uploads
- name: Bandit (SARIF)
  run: bandit -r src -f sarif -o bandit.sarif

- name: Semgrep (SARIF)  
  run: semgrep ci --config p/ci --sarif --output semgrep.sarif

- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v3

# Fast feedback on PRs, comprehensive on main
- name: Test with pytest
  run: |
    if [ "${{ github.ref }}" == "refs/heads/main" ]; then
      pytest -q --maxfail=0 -n auto --durations=15
    else
      pytest -q --maxfail=1 -n auto --durations=15
    fi

strategy:
  fail-fast: false
  matrix:
    include:
      - environment: ephemeral
        aws_region: us-east-1
        timeout: 300
      - environment: dev
        aws_region: us-east-1
        timeout: 600
      - environment: staging
        aws_region: us-east-1
        timeout: 900
      - environment: prod
        aws_region: us-west-2
        timeout: 1200

# DRY Python setup
- uses: ./.github/actions/python-setup

# Environment-specific post-deploy checks
- uses: ./.github/actions/post-deploy-checks
  with:
    environment: ${{ matrix.environment }}

• Project Structure: PROJECT_STRUCTURE.md
• AWS Setup: docs/AWS_IAM_SETUP.md
• Code Examples: See src/ directory

1. Fork the repository
2. Create a feature branch
3. Follow the project structure
4. Write tests for new functionality
5. Run linting and formatting
6. Submit a pull request

Create buildspec.yml in your project root:

version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.11
  pre_build:
    commands:
      - echo Installing dependencies...
      - pip install -r requirements.txt
  build:
    commands:
      - echo Running tests...
      - make test
      - echo Running linting...
      - make lint
      - echo Building web application...
      - cd src/web && python -c "import application; print('✅ Web app ready')"
  post_build:
    commands:
      - echo Build completed successfully
artifacts:
  files:
    - '**/*'
  base-directory: .

# Create CodeBuild project
aws codebuild create-project \
  --name python-devops-build \
  --source type=GITHUB,location=https://github.com/your-username/python-devops-course \
  --artifacts type=NO_ARTIFACTS \
  --environment type=LINUX_CONTAINER,image=aws/codebuild/python:3.11 \
  --service-role arn:aws:iam::YOUR_ACCOUNT:role/CodeBuildServiceRole

# Start build
aws codebuild start-build --project-name python-devops-build

# Create deployment package
mkdir -p eb-deploy
cp -r src/web/* eb-deploy/
cp requirements.txt eb-deploy/
cp Procfile eb-deploy/

# Create Procfile for Elastic Beanstalk
echo "web: gunicorn application:app --bind 0.0.0.0:8000" > eb-deploy/Procfile

# Create .ebextensions for configuration
mkdir -p eb-deploy/.ebextensions

Create eb-deploy/.ebextensions/01_packages.config:

packages:
  yum:
    git: []

Create eb-deploy/.ebextensions/02_python.config:

option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: application:app
  aws:elasticbeanstalk:application:environment:
    PYTHONPATH: "/var/app/current"

# Install EB CLI
pip install awsebcli

# Initialize Elastic Beanstalk
eb init python-devops-app

# Create environment
eb create python-devops-env

# Deploy application
eb deploy

# Open in browser
eb open

# Set environment variables
eb setenv AWS_DEFAULT_REGION=us-east-1

# Scale application
eb scale 2

# View logs
eb logs

# Terminate environment
eb terminate

# .github/workflows/ship.yml
name: Deploy to AWS
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1
    
    - name: Deploy to Elastic Beanstalk
      run: |
        pip install awsebcli
        eb deploy python-devops-env

# View application logs
aws logs describe-log-groups --log-group-name-prefix /aws/elasticbeanstalk

# Monitor metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/ElasticBeanstalk \
  --metric-name ApplicationRequestsTotal \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-02T00:00:00Z \
  --period 3600 \
  --statistics Sum

# Check application health
eb health

# View detailed health
eb health --refresh

# Monitor specific instances
eb status --verbose

# .ebextensions/03_loadbalancer.config
option_settings:
  aws:elbv2:loadbalancer:
    IdleTimeout: 60
  aws:autoscaling:launchconfiguration:
    InstanceType: t3.micro

# .ebextensions/04_database.config
option_settings:
  aws:rds:dbinstance:
    DBInstanceClass: db.t3.micro
    DBAllocatedStorage: 20

• CodeBuild: Build and test automation
• Elastic Beanstalk: Application deployment
• CloudWatch: Monitoring and logging
• Load Balancer: Traffic distribution
• Auto Scaling: Handle traffic spikes
• Health Checks: Application monitoring
• SSL/TLS: Secure connections
• Domain: Custom domain setup

This project is part of the Python DevOps Course.

• 🤖 Master GenAI Engineering - Build Production AI Systems
• 🦀 Learn Professional Rust - Industry-Grade Development
• 📊 AWS AI & Analytics - Scale Your ML in Cloud
• ⚡ Production GenAI on AWS - Deploy at Enterprise Scale
• 🛠️ Rust DevOps Mastery - Automate Everything

• 💼 Production ML Program - Complete MLOps & Cloud Mastery
• 🎯 Start Learning Now - Fast-Track Your ML Career
• 🏢 Trusted by Fortune 500 Teams

Learn end-to-end ML engineering from industry veterans at PAIML.COM