SageMaker training infrastructure for the image classification sample: data and model buckets, the SageMaker pipeline (validation, preprocessing, three trainers, evaluation, ensemble, fairness check, clinical quality gate, registration), the Model Registry, a Model Card and cost controls.

1. Creates a KMS key with rotation for the stack's buckets, logs and SageMaker jobs
2. Creates 6 S3 buckets (raw data, processed data, scripts, model artifacts, inference results, monitoring), plus an SBOM bucket when enable_sbom_bucket = true (default) and a CloudTrail bucket when enable_cloudtrail = true (default false). force_destroy is false on all of them
3. Creates the SageMaker execution role (carrying the workload permissions boundary when permissions_boundary_arn is set)
4. Creates the Model Package Group and, with enable_model_card = true (default), a Model Card
5. Creates the SageMaker pipeline from pipeline.json.tftpl: 9 top-level steps (ValidateDataset, PreprocessData, three training steps, EvaluateAllModels, CreateEnsembleModel, BiasCheck, and the Condition step), whose branches are RegisterEnsembleModel or Fail
6. Creates an EventBridge rule that starts the pipeline with RetrainingReason=data_upload when the .batch_complete marker is written to the raw data bucket
7. With enable_upload_quarantine = true (default), creates the upload quarantine Lambda (see Upload quarantine)
8. Creates CloudWatch log groups, metric filters and a training dashboard, and an AWS Budgets budget (monthly_budget_usd, default 200)
9. Optionally creates a managed MLflow tracking server (enable_mlflow, default false) and a multi-region CloudTrail trail (enable_cloudtrail, default false)

An EventBridge rule on the raw-data bucket's Object Created events under training_data_path invokes a Lambda once per uploaded object. EventBridge is the same event source the pipeline trigger uses: S3 allows one notification configuration per bucket, and the bucket's sends everything to EventBridge. The Lambda:

• keeps an object that has an allowed extension (.jpg, .jpeg, .png, the extensions validation and preprocessing read), opens with Pillow as the format that extension names (verified and fully decoded, so truncated files fail), and is at least quarantine_min_image_size_px (112, the validation step's hard minimum) on both sides;
• otherwise copies it to <quarantine_prefix><original key> (default quarantine/), writes <quarantine_prefix><original key>.reason.json with the reason, ETag, size and time, deletes the original, and adds 1 to the QuarantinedImages metric (namespace <project_name>/DataQuality, dimension Bucket);
• ignores the .batch_complete marker, keys outside training_data_path and anything already under quarantine_prefix.

quarantine_prefix must sit outside training_data_path (a variable validation checks it), so neither the pipeline trigger nor the validation and preprocessing inputs read quarantined files. The bucket is versioned, so the deleted original stays recoverable as a noncurrent version.

The function uses the terraform-aws-lambda module: KMS-encrypted environment and log group (log_retention_days), active X-Ray tracing, an SQS dead-letter queue encrypted with the project key plus a queue-depth alarm, reserved concurrency (upload_quarantine_reserved_concurrency, 10) and the workload permissions boundary. Its role can read and delete only under training_data_path, write only under quarantine_prefix, and publish metrics only to its namespace. Pillow comes from a layer built from the same zip and script as the inference Lambda's layer (ops-scripts/build_lambda_layer.sh, Pillow 12.3.0); make layer builds it, and the apply builds it when missing.

The checks run as files arrive, while the marker starts the pipeline as soon as the upload script writes it, so a large upload can start the pipeline before the Lambda has reached every file. The validation step checks every image again and fails the run on a bad one, so nothing unchecked reaches training. For a clean first run, wait for the Lambda to catch up (its CloudWatch Invocations metric flattens) before writing the marker, or rerun the pipeline.

• Patient-grouped split. Preprocessing resizes images to preprocessing_target_size (512) and splits train, validation and test by BreakHis patient id, so one patient's images never cross splits.
• Thresholds tuned on validation. Evaluation and the ensemble tune their decision thresholds on the validation split; the clinical gate is scored on the test split at that threshold.
• Fairness gate on the ensemble. BiasCheck runs Fairlearn over the ensemble's own test predictions at its tuned threshold, per fairness_gate.sensitive_feature (magnification). Fewer than two subgroups is "not evaluable" and fails the gate unless allow_not_evaluable = true (default false; meant for small demo datasets).
• Gate. The Condition step requires clinical_quality_gate (accuracy 0.85, recall 0.95, precision 0.80, AUC 0.90) and fairness_gate.max_disparity (0.10). A pass registers the ensemble as PendingManualApproval; a miss ends in the Fail step.
• Per-execution artifacts. Evaluation, ensemble and bias outputs go to <prefix>/<pipeline-execution-id>/ in the model artifacts bucket, and the registered package's ModelDataUrl points at that execution's model.tar.gz.
• Lineage. RetrainingReason, DatasetVersion and CodeCommitSha are pipeline parameters recorded on the model package. The Terraform variables only set their defaults; the upload rule passes data_upload and the drift rule in stack-inference passes drift_detected.
• Network isolation. enable_network_isolation = true (default) runs the training jobs without outbound network access; they read ImageNet weights from s3://<scripts-bucket>/pretrained-weights/. Processing jobs are not isolated because they install packages at start-up. vpc_config optionally places the jobs in your subnets.
• Instance types. The shipped terraform.tfvars trains on ml.c5.2xlarge (CPU) with 2 epochs, sized for a smoke test. Instance types starting ml.g or ml.p use the GPU training image; they need a SageMaker training quota.

1. stack-backend-setup applied and backend.hcl written (make backend-config)
2. Terraform ~> 1.15 and AWS CLI v2
3. After apply: the scripts uploaded (make upload-scripts), the ImageNet weights uploaded when network isolation is on, and training data uploaded with scripts/data_uploader.sh
4. training_data_path set to the prefix the uploader writes to (medical_image_data/, the shipped value), and auto_trigger_marker_key to the marker it writes (medical_image_data/.batch_complete)

From the repository root:

make deploy-training     # terraform init -backend-config=backend.hcl, plan -out, apply
make upload-scripts      # pipeline scripts to the scripts bucket

# Network isolation (default): upload ImageNet weights once (no-op when isolation is off)
make weights

# Upload a dataset (breast_benign/ and breast_malignant/ folders); the marker starts the pipeline
./scripts/data_uploader.sh data/breakhis

make deploy-training passes state_bucket_name, state_bucket_region and permissions_boundary_arn from the bootstrap outputs as TF_VAR_*.

map(object({
    script_name     = string
    tar_file        = string
    step_name       = string
    hyperparameters = map(string)
    volume_size     = number
    instance_count  = number
    max_runtime     = number
    instance_type   = string
    enable_cache    = bool
    cache_expiry    = string
  }))

map(object({
    instance_type  = string
    instance_count = number
    volume_size    = number
    script_name    = string
    step_name      = string
    step_type      = string
    enable_cache   = bool
    cache_expiry   = string
  }))

object({
    force_destroy       = bool
    enable_versioning   = bool
    block_public_access = bool
  })

{
  "block_public_access": true,
  "enable_versioning": true,
  "force_destroy": false
}

object({
    accuracy  = number
    recall    = number
    precision = number
    auc_roc   = number
  })

{
  "accuracy": 0.85,
  "auc_roc": 0.9,
  "precision": 0.8,
  "recall": 0.95
}

object({
    content_type     = string
    compression_type = string
    s3_data_type     = string
    s3_input_mode    = string
    s3_upload_mode   = string
  })

{
  "compression_type": "None",
  "content_type": "application/x-image",
  "s3_data_type": "S3Prefix",
  "s3_input_mode": "File",
  "s3_upload_mode": "EndOfJob"
}

object({
    max_disparity     = number
    sensitive_feature = string
  })

{
  "max_disparity": 0.1,
  "sensitive_feature": "magnification"
}

map(object({
    name = string
  }))

object({
    tensorflow_gpu_tag       = string
    tensorflow_cpu_tag       = string
    tensorflow_inference_tag = string
  })

{
  "tensorflow_cpu_tag": "2.19.0-cpu-py312-ubuntu22.04-sagemaker",
  "tensorflow_gpu_tag": "2.19.0-gpu-py312-cu125-ubuntu22.04-sagemaker",
  "tensorflow_inference_tag": "2.19.0-cpu-py312-ubuntu22.04-sagemaker"
}

object({
    subnet_ids         = list(string)
    security_group_ids = list(string)
  })

stack-training/
|-- auto_trigger.tf           # EventBridge rule, target and role that start the pipeline on upload
|-- backend.tf                # Partial S3 backend (values come from backend.hcl)
|-- backend.hcl.example       # Template for backend.hcl
|-- cloudtrail_budgets.tf     # Optional CloudTrail trail and bucket, AWS Budgets budget
|-- data.tf                   # Caller identity and SageMaker prebuilt image lookups
|-- lambda/upload_quarantine/ # Upload quarantine Lambda source
|-- main.tf                   # KMS, S3 buckets, execution role, registry, Model Card, MLflow, pipeline
|-- monitor.tf                # CloudWatch log groups, metric filters and training dashboard
|-- outputs.tf                # Buckets, pipeline, registry, IAM, KMS, MLflow, SBOM outputs
|-- pipeline.json.tftpl       # SageMaker pipeline definition (rendered by templatefile)
|-- providers.tf              # AWS provider with default_tags
|-- quarantine.tf             # Upload quarantine Lambda, rule, role, layer and DLQ
|-- sbom.tf                   # SBOM bucket for the patched inference image builds
|-- terraform.tfvars          # Values for this environment
|-- terraform.tfvars.example  # Annotated example values
|-- variables.tf              # Input variables
`-- versions.tf               # Terraform and provider version constraints