Skip to content

burkaslarry/python_mysql_http

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

20 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Python MySQL/PostgreSQL FastAPI Project

πŸš€ API Introduction

A production-ready REST API built with FastAPI that provides dynamic CRUD operations for any MySQL or PostgreSQL database table. The API features:

  • βœ… Multi-Database Support: MySQL (via aiomysql) or PostgreSQL (via asyncpg)
  • βœ… Dynamic CRUD Operations: CREATE, READ, UPDATE, DELETE on any table
  • βœ… Resiliency Engine: Automatic exponential backoff retry logic (200ms β†’ 1600ms)
  • βœ… CSV Operations: Batch import/export for data management
  • βœ… SQL Safety: Parameterized queries prevent SQL injection
  • βœ… Comprehensive Error Handling: Standardized JSON error responses
  • βœ… Async/Await: High-performance asynchronous operations
  • βœ… CORS Enabled: Cross-origin resource sharing support
  • βœ… Docker Ready: Containerized deployment support

πŸ“‹ API Endpoints

Method Endpoint Description
POST /api/{tableName} CREATE a new record
GET /api/{tableName}/query READ all records
POST /api/{tableName}/query READ filtered records
PUT /api/{tableName} UPDATE existing records
DELETE /api/{tableName} DELETE records
POST /api/batch/import CSV bulk import
GET /api/batch/{tableName}/export CSV export

πŸ“¦ Project Structure

/app/
β”œβ”€β”€ core/                    # Business logic layer
β”‚   β”œβ”€β”€ resiliency.py       # Retry engine with exponential backoff
β”‚   β”œβ”€β”€ database.py         # Query execution & parameterized statements
β”‚   └── config.py           # Database configuration (MySQL/PostgreSQL)
β”œβ”€β”€ routes/                  # HTTP endpoint handlers
β”‚   β”œβ”€β”€ crud.py             # CREATE, READ, UPDATE, DELETE endpoints
β”‚   └── csv_routes.py       # CSV import/export endpoints
β”œβ”€β”€ services/                # Business logic utilities
β”‚   └── csv_service.py      # CSV processing & validation
β”œβ”€β”€ schemas/                 # Pydantic data models
β”‚   └── requests.py         # Request body validation
└── utils/                   # Helper functions
    └── error_handler.py    # Standardized response formatting

main.py                      # FastAPI application entry point
requirements.txt             # Python dependencies
Dockerfile                   # Docker container configuration
deploy.sh                    # Deployment automation script
test_main.py                 # Unit tests
test_comprehensive.py        # Comprehensive test suite

βš™οΈ Configuration & Setup

Option 1: Local Development (Quick Start)

Step 1: Clone & Install Dependencies

# Clone the repository
git clone https://github.com/burkaslarry/python_mysql_http.git
cd python_mysql_http

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Step 2: Choose Your Database & Create .env File

For MySQL:

cat > .env << EOF
# Database Type: mysql or postgresql
DB_TYPE=mysql

# MySQL Configuration
DB_HOST=localhost
DB_PORT=3306
DB_USER=root
DB_PASSWORD=your_password
DB_NAME=test_db
EOF

For PostgreSQL:

cat > .env << EOF
# Database Type: mysql or postgresql
DB_TYPE=postgresql

# PostgreSQL Configuration
DB_POSTGRES_HOST=localhost
DB_POSTGRES_PORT=5432
DB_POSTGRES_USER=postgres
DB_POSTGRES_PASSWORD=your_password
DB_POSTGRES_NAME=test_db
EOF

Step 3: Create Database & Run API

# For MySQL: Create test database
mysql -u root -p -e "CREATE DATABASE test_db;"

# For PostgreSQL: Create test database
psql -U postgres -c "CREATE DATABASE test_db;"

# Start the API server
python main.py
# or with auto-reload:
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Step 4: Verify Installation

# Test the API
curl http://localhost:8000/
# Expected response: {"message": "Hello, World"}

🌐 Remote Database Setup

MySQL Remote Configuration

1. AWS RDS MySQL Setup

# .env file configuration
cat > .env << EOF
DB_TYPE=mysql
DB_HOST=mydb.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com
DB_PORT=3306
DB_USER=admin
DB_PASSWORD=YourSecurePassword123
DB_NAME=production_db
EOF

AWS RDS Setup Steps:

  1. Go to AWS RDS Console β†’ Create Database β†’ MySQL
  2. Choose Multi-AZ for high availability
  3. Set Master username: admin
  4. Set Master password: YourSecurePassword123
  5. Create security group allowing port 3306
  6. Copy endpoint to DB_HOST in .env

2. Other MySQL Hosting Providers

Provider Setup Link Cost
DigitalOcean https://www.digitalocean.com/products/managed-databases $15/month
Google Cloud SQL https://cloud.google.com/sql/docs/mysql $3.50/month
Azure Database https://azure.microsoft.com/en-us/services/mysql/ $20/month
Aiven https://aiven.io/mysql Free tier + $10/month

PostgreSQL Remote Configuration

1. AWS RDS PostgreSQL Setup

# .env file configuration
cat > .env << EOF
DB_TYPE=postgresql
DB_POSTGRES_HOST=mydb.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com
DB_POSTGRES_PORT=5432
DB_POSTGRES_USER=postgres
DB_POSTGRES_PASSWORD=YourSecurePassword123
DB_POSTGRES_NAME=production_db
EOF

AWS RDS Setup Steps:

  1. Go to AWS RDS Console β†’ Create Database β†’ PostgreSQL
  2. Choose Multi-AZ for high availability
  3. Set Master username: postgres
  4. Set Master password: YourSecurePassword123
  5. Create security group allowing port 5432
  6. Copy endpoint to DB_POSTGRES_HOST in .env

2. Other PostgreSQL Hosting Providers

Provider Setup Link Cost
DigitalOcean https://www.digitalocean.com/products/managed-databases $15/month
Heroku PostgreSQL https://www.heroku.com/postgres Free tier + $9/month
Google Cloud SQL https://cloud.google.com/sql/docs/postgres $3.50/month
Aiven https://aiven.io/postgresql Free tier + $10/month
Supabase https://supabase.com Free tier included

πŸ”„ Environment Variable Reference

Complete .env Example:

# Database Selection
DB_TYPE=mysql              # mysql or postgresql

# MySQL Settings (if DB_TYPE=mysql)
DB_HOST=localhost
DB_PORT=3306
DB_USER=root
DB_PASSWORD=password
DB_NAME=test_db

# PostgreSQL Settings (if DB_TYPE=postgresql)
DB_POSTGRES_HOST=localhost
DB_POSTGRES_PORT=5432
DB_POSTGRES_USER=postgres
DB_POSTGRES_PASSWORD=password
DB_POSTGRES_NAME=test_db

# API Server Settings
PORT=8000
ENVIRONMENT=development    # development or production
LOG_LEVEL=INFO

πŸš€ Deployment & Automation Scripts

Automated Deployment Script

Create deploy.sh in your project root:

#!/bin/bash

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

echo -e "${YELLOW}=== Python MySQL/PostgreSQL FastAPI Deployment ===${NC}\n"

show_menu() {
    echo "Choose an option:"
    echo "1. Run locally (development)"
    echo "2. Run tests"
    echo "3. Build Docker image"
    echo "4. Deploy to remote server"
    echo "5. Deploy to Heroku"
    echo "6. Exit"
    read -p "Enter choice [1-6]: " choice
}

run_local() {
    echo -e "${YELLOW}Starting API server locally...${NC}"
    python3 main.py
}

run_tests() {
    echo -e "${YELLOW}Running test suite...${NC}"
    python3 -m pytest test_main.py test_comprehensive.py -v --tb=short
    if [ $? -eq 0 ]; then
        echo -e "${GREEN}βœ“ All tests passed!${NC}\n"
    else
        echo -e "${RED}βœ— Tests failed!${NC}\n"
    fi
}

build_docker() {
    echo -e "${YELLOW}Building Docker image...${NC}"
    read -p "Enter Docker image name (default: my-api): " image_name
    image_name=${image_name:-my-api}
    
    docker build -t $image_name:latest .
    
    if [ $? -eq 0 ]; then
        echo -e "${GREEN}βœ“ Docker image built: $image_name:latest${NC}"
        echo "To run: docker run -p 8000:8000 -e DB_HOST=... $image_name:latest"
    else
        echo -e "${RED}βœ— Docker build failed!${NC}"
    fi
}

deploy_aws_ec2() {
    read -p "Enter EC2 instance IP/hostname: " ec2_host
    read -p "Enter SSH key path (default: ~/.ssh/id_rsa): " ssh_key
    ssh_key=${ssh_key:-~/.ssh/id_rsa}
    
    echo -e "${YELLOW}Deploying to AWS EC2: $ec2_host${NC}"
    
    ssh -i $ssh_key ec2-user@$ec2_host << 'DEPLOY'
        # Install Docker on Amazon Linux
        sudo yum update -y
        sudo amazon-linux-extras install docker -y
        sudo systemctl start docker
        sudo systemctl enable docker
        
        # Create app directory
        mkdir -p ~/app && cd ~/app
        
        # Create .env file (customize these values)
        cat > .env << 'ENV'
DB_TYPE=mysql
DB_HOST=your_remote_db_host
DB_PORT=3306
DB_USER=your_user
DB_PASSWORD=your_password
DB_NAME=your_database
PORT=8000
ENV
        
        # Pull and run Docker image
        docker pull your_docker_registry/my-api:latest
        docker run -d --restart always \
            --name my-api \
            -p 8000:8000 \
            --env-file .env \
            your_docker_registry/my-api:latest
DEPLOY
    
    echo -e "${GREEN}βœ“ Deployed to AWS EC2${NC}"
    echo "API available at: http://$ec2_host:8000"
}

deploy_digitalocean() {
    read -p "Enter Droplet IP address: " do_host
    read -p "Enter SSH key path (default: ~/.ssh/id_rsa): " ssh_key
    ssh_key=${ssh_key:-~/.ssh/id_rsa}
    
    echo -e "${YELLOW}Deploying to DigitalOcean: $do_host${NC}"
    
    ssh -i $ssh_key root@$do_host << 'DEPLOY'
        # Update system
        apt-get update && apt-get upgrade -y
        
        # Install Docker
        curl -fsSL https://get.docker.com -o get-docker.sh
        sh get-docker.sh
        systemctl start docker
        systemctl enable docker
        
        # Create app directory
        mkdir -p /opt/app && cd /opt/app
        
        # Create .env file
        cat > .env << 'ENV'
DB_TYPE=postgresql
DB_POSTGRES_HOST=your_remote_db_host
DB_POSTGRES_PORT=5432
DB_POSTGRES_USER=postgres
DB_POSTGRES_PASSWORD=your_password
DB_POSTGRES_NAME=your_database
PORT=8000
ENV
        
        # Pull and run Docker image
        docker pull your_docker_registry/my-api:latest
        docker run -d --restart always \
            --name my-api \
            -p 8000:8000 \
            --env-file .env \
            your_docker_registry/my-api:latest
DEPLOY
    
    echo -e "${GREEN}βœ“ Deployed to DigitalOcean${NC}"
    echo "API available at: http://$do_host:8000"
}

deploy_heroku() {
    echo -e "${YELLOW}=== Heroku Deployment ===${NC}\n"
    
    if ! command -v heroku &> /dev/null; then
        echo "Installing Heroku CLI..."
        curl https://cli-assets.heroku.com/install.sh | sh
    fi
    
    read -p "Enter Heroku app name: " heroku_app
    
    echo "Logging into Heroku..."
    heroku login
    
    echo "Creating Heroku app..."
    heroku create $heroku_app 2>/dev/null || true
    
    echo "Setting environment variables..."
    heroku config:set --app $heroku_app \
        DB_TYPE=postgresql \
        DB_POSTGRES_HOST=your_db_host \
        DB_POSTGRES_USER=your_user \
        DB_POSTGRES_PASSWORD=your_password \
        DB_POSTGRES_NAME=your_database
    
    echo "Deploying to Heroku..."
    git push heroku main
    
    echo -e "${GREEN}βœ“ Deployed to Heroku${NC}"
    echo "API available at: https://$heroku_app.herokuapp.com"
}

# Main loop
while true; do
    show_menu
    
    case $choice in
        1)
            run_local
            ;;
        2)
            run_tests
            ;;
        3)
            build_docker
            ;;
        4)
            echo ""
            echo "Choose deployment platform:"
            echo "1. AWS EC2"
            echo "2. DigitalOcean"
            echo "3. Linode (manual SSH)"
            echo "4. Google Cloud (manual SSH)"
            echo "5. Azure (manual SSH)"
            read -p "Enter choice [1-5]: " deploy_choice
            
            case $deploy_choice in
                1) deploy_aws_ec2 ;;
                2) deploy_digitalocean ;;
                *)
                    echo -e "${YELLOW}For other platforms, use manual SSH deployment${NC}"
                    echo "Commands: git clone, install docker, pull image, run container"
                    ;;
            esac
            ;;
        5)
            deploy_heroku
            ;;
        6)
            echo -e "${GREEN}Goodbye!${NC}"
            exit 0
            ;;
        *)
            echo -e "${RED}Invalid choice. Please try again.${NC}"
            ;;
    esac
done

Make executable and run:

chmod +x deploy.sh
./deploy.sh

πŸ§ͺ Testing

Quick API Test

# Start the server in background
python main.py &

# Test root endpoint
curl http://localhost:8000/

# Test with database
curl -X POST http://localhost:8000/api/users \
  -H "Content-Type: application/json" \
  -d '{"input": {"name": "John", "email": "john@example.com"}}'

Comprehensive Testing

# Run all tests
python -m pytest test_main.py test_comprehensive.py -v

# Run specific test file
python -m pytest test_comprehensive.py -v

# Run with coverage report
python -m pytest --cov=app test_main.py

Test Results: βœ… 25/25 tests passing


🐳 Docker Deployment

Build Docker Image

docker build -t my-api:latest .

Run Docker Container

docker run -d \
  -p 8000:8000 \
  -e DB_TYPE=mysql \
  -e DB_HOST=mysql-db \
  -e DB_PORT=3306 \
  -e DB_USER=root \
  -e DB_PASSWORD=password \
  -e DB_NAME=test_db \
  --name my-api \
  my-api:latest

# Check logs
docker logs my-api

# Stop container
docker stop my-api

Docker Compose (Local Development)

Create docker-compose.yml:

version: '3.8'

services:
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: test_db
    ports:
      - "3306:3306"
    volumes:
      - mysql_data:/var/lib/mysql

  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: test_db
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  api:
    build: .
    ports:
      - "8000:8000"
    environment:
      DB_TYPE: mysql
      DB_HOST: mysql
      DB_PORT: 3306
      DB_USER: root
      DB_PASSWORD: password
      DB_NAME: test_db
    depends_on:
      - mysql
      - postgres

volumes:
  mysql_data:
  postgres_data:

Run with Docker Compose:

docker-compose up -d
# API available at: http://localhost:8000
# MySQL: localhost:3306
# PostgreSQL: localhost:5432

🌍 Cloud Hosting Recommendations

Platform Type Cost Setup Time Best For
AWS EC2 + RDS VM + Managed DB $10-50/month 20 min Enterprise, auto-scaling
DigitalOcean Droplet + Managed DB $5-20/month 10 min Startups, simplicity
Heroku PaaS $7-50/month 5 min Quick deployment
Railway.app Modern PaaS Free-$20/month 3 min Fastest setup
Render Modern PaaS Free-$19/month 5 min Simple & fast
Google Cloud Run Serverless Pay-per-use 15 min Sporadic traffic
Azure App Service PaaS $10-50/month 20 min Microsoft ecosystem

Recommended for Quick Start: Railway.app or Render
Recommended for Production: AWS or DigitalOcean


πŸ“š API Examples

Create a Record

curl -X POST http://localhost:8000/api/users \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "name": "John Doe",
      "email": "john@example.com",
      "age": 30
    }
  }'

Read All Records

curl http://localhost:8000/api/users/query

Read Filtered Records

curl -X POST http://localhost:8000/api/users/query \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "age": 30
    }
  }'

Update Records

curl -X PUT http://localhost:8000/api/users \
  -H "Content-Type: application/json" \
  -d '{
    "input": {"age": 31},
    "where": {"id": 1}
  }'

Delete Records

curl -X DELETE http://localhost:8000/api/users \
  -H "Content-Type: application/json" \
  -d '{
    "input": {"id": 1}
  }'

CSV Bulk Import

curl -X POST http://localhost:8000/api/batch/import \
  -F "file=@users.csv"

CSV Export

curl http://localhost:8000/api/batch/users/export -o users.csv

πŸ“‹ Production Checklist

  • Database configured (MySQL or PostgreSQL)
  • All tests passing (python -m pytest test_main.py -v)
  • Environment variables set in .env (or secrets management)
  • Docker image built and tested
  • API endpoints tested with curl or Postman
  • Error handling verified
  • Database backups configured
  • Logging enabled and monitored
  • Security group/firewall rules configured
  • SSL/TLS certificate installed (if using domain)
  • Monitoring and alerts set up
  • Rate limiting configured (if needed)

πŸŽ“ Documentation References


πŸ“ž Support & Troubleshooting

Common Issues:

  1. "Database connection failed"

    • Verify database is running and accessible
    • Check .env file configuration matches database
    • Confirm database credentials are correct
    • For remote databases, check security group/firewall allows connection
  2. "Port 8000 already in use"

    • Change port: uvicorn main:app --port 8001
    • Or kill process: lsof -i :8000 | grep LISTEN | awk '{print $2}' | xargs kill -9
  3. "Module not found"

    • Activate virtual environment: source venv/bin/activate
    • Install dependencies: pip install -r requirements.txt
  4. "Cannot connect to remote database"

    • Check network connectivity: telnet db_host db_port
    • Verify security group allows your IP
    • Confirm .env has correct host/port

For more details, see API_DOCUMENTATION.md section.


πŸŽ‰ Quick Start

  1. Clone repository

    git clone https://github.com/burkaslarry/python_mysql_http.git
    cd python_mysql_http
  2. Install dependencies

    pip install -r requirements.txt
  3. Configure database

    cat > .env << EOF
    DB_TYPE=mysql
    DB_HOST=localhost
    DB_USER=root
    DB_PASSWORD=your_password
    DB_NAME=test_db
    EOF
  4. Create database

    mysql -u root -p -e "CREATE DATABASE test_db;"
  5. Run tests

    python -m pytest test_main.py -v
  6. Start API

    python main.py
  7. Deploy

    ./deploy.sh  # Interactive deployment menu

βœ… Status

  • Tests: βœ… 25/25 passing
  • Databases: βœ… MySQL & PostgreSQL
  • Documentation: βœ… Complete
  • Docker: βœ… Production ready
  • Deployment: βœ… Automated scripts

Version: 1.0.0
Last Updated: 2025-12-30
Status: Production Ready βœ…

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors