---
title: Maintainers playbook
description: Playbook for active maintainers in Powertools for AWS Lambda (Python)
---

<!--  markdownlint-disable MD043 -->

## Overview

!!! note "Please treat this content as a living document."

This is document explains who the maintainers are, their responsibilities, and how they should be doing it. If you're interested in contributing, see [CONTRIBUTING](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CONTRIBUTING.md){target="_blank"}.

## Current Maintainers

| Maintainer        | GitHub ID                                                                               | Affiliation |
| ----------------- | --------------------------------------------------------------------------------------- | ----------- |
| Leandro Damascena | [leandrodamascena](https://github.com/leandrodamascena){target="_blank" rel="nofollow"} | Amazon      |

## AWS Contributors

Active contributors from AWS who help maintain and improve this project.

| Contributor       | GitHub ID                                                                               |
| ----------------- | --------------------------------------------------------------------------------------- |
| Ana Falcão        | [anafalcao](https://github.com/anafalcao){target="_blank" rel="nofollow"}               |

## Emeritus

Previous active maintainers who contributed to this project.

| Maintainer        | GitHub ID                                                                       | Affiliation |
| ----------------- | ------------------------------------------------------------------------------- | ----------- |
| Alexander Schueren| [am29d](https://github.com/am29d){target="_blank" rel="nofollow"}               | Amazon      |
| Heitor Lessa      | [heitorlessa](https://github.com/heitorlessa){target="_blank" rel="nofollow"}   | Adyen       |
| Michal Ploski     | [mploski](https://github.com/mploski){target="_blank" rel="nofollow"}           | Splunk      |
| Nicolas Moutschen | [nmoutschen](https://github.com/nmoutschen){target="_blank" rel="nofollow"}     | Apollo      |
| Ruben Fonseca     | [rubenfonseca](https://github.com/rubenfonseca){target="_blank" rel="nofollow"} | N/A         |
| Simon Thulbourn   | [sthulb](https://github.com/sthulb){target="_blank" rel="nofollow"}             | N/A         |
| Tom McCarthy      | [cakepietoast](https://github.com/cakepietoast){target="_blank" rel="nofollow"} | MongoDB     |

## Labels

These are the most common labels used by maintainers to triage issues, pull requests (PR), and for project management:

| Label                  | Usage                                                                                    | Notes                                                           |
| ---------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| triage                 | New issues that require maintainers review                                               | Issue template                                                  |
| bug                    | Unexpected, reproducible and unintended software behavior                                | PR/Release automation; Doc snippets are excluded;               |
| not-a-bug              | New and existing bug reports incorrectly submitted as bug                                | Analytics                                                       |
| documentation          | Documentation improvements                                                               | PR/Release automation; Doc additions, fixes, etc.;              |
| feature-request        | New or enhancements to existing features                                                 | Issue template                                                  |
| typing                 | New or enhancements to static typing                                                     | Issue template                                                  |
| RFC                    | Technical design documents related to a feature request                                  | Issue template                                                  |
| bug-upstream           | Bug caused by upstream dependency                                                        |                                                                 |
| help wanted            | Tasks you want help from anyone to move forward                                          | Bandwidth, complex topics, etc.                                 |
| need-customer-feedback | Tasks that need more feedback before proceeding                                          | 80/20% rule, uncertain, etc.                                    |
| need-more-information  | Missing information before making any calls                                              |                                                                 |
| need-documentation     | PR is missing or has incomplete documentation                                            |                                                                 |
| need-issue             | PR is missing a related issue for tracking change                                        | PR automation                                                   |
| need-rfc               | Feature request requires a RFC to improve discussion                                     |                                                                 |
| pending-release        | Merged changes that will be available soon                                               | Release automation auto-closes/notifies it                      |
| revisit-in-3-months    | Blocked issues/PRs that need to be revisited                                             | Often related to `need-customer-feedback`, prioritization, etc. |
| breaking-change        | Changes that will cause customer impact and need careful triage                          |                                                                 |
| do-not-merge           | PRs that are blocked for varying reasons                                                 | Timeline is uncertain                                           |
| size/XS                | PRs between 0-9 LOC                                                                      | PR automation                                                   |
| size/S                 | PRs between 10-29 LOC                                                                    | PR automation                                                   |
| size/M                 | PRs between 30-99 LOC                                                                    | PR automation                                                   |
| size/L                 | PRs between 100-499 LOC                                                                  | PR automation                                                   |
| size/XL                | PRs between 500-999 LOC, often PRs that grown with feedback                              | PR automation                                                   |
| size/XXL               | PRs with 1K+ LOC, largely documentation related                                          | PR automation                                                   |
| tests                  | PRs that add or change tests                                                             | PR automation                                                   |
| `<utility>`            | PRs related to a Powertools for AWS Lambda (Python) utility, e.g. `parameters`, `tracer` | PR automation                                                   |
| feature                | New features or minor changes                                                            | PR/Release automation                                           |
| dependencies           | Changes that touch dependencies, e.g. Dependabot, etc.                                   | PR/ automation                                                  |
| github-actions         | Changes in GitHub workflows                                                              | PR automation                                                   |
| github-templates       | Changes in GitHub issue/PR templates                                                     | PR automation                                                   |
| internal               | Changes in governance and chores (linting setup, baseline, etc.)                         | PR automation                                                   |
| tech-debt              | Changes in tech debt                                                                     |                                                                 |
| customer-reference     | Authorization to use company name in our documentation                                   | Public Relations                                                |
| community-content      | Suggested content to feature in our documentation                                        | Public Relations                                                |

## Maintainer Responsibilities

Maintainers are active and visible members of the community, and have [maintain-level permissions on a repository](https://docs.github.com/en/organizations/managing-access-to-your-organizations-repositories/repository-permission-levels-for-an-organization){target="_blank" rel="nofollow"}. Use those privileges to serve the community and evolve code as follows.

Be aware of recurring ambiguous situations and [document them](#common-scenarios) to help your fellow maintainers.

### Uphold Code of Conduct

<!-- markdownlint-disable-next-line MD013 -->
Model the behavior set forward by the [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank"} and raise any violations to other maintainers and admins. There could be unusual circumstances where inappropriate behavior does not immediately fall within the [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank"}.

These might be nuanced and should be handled with extra care - when in doubt, do not engage and reach out to other maintainers and admins.

### Prioritize Security

Security is your number one priority. Maintainer's Github keys must be password protected securely and any reported security vulnerabilities are addressed before features or bugs.

Note that this repository is monitored and supported 24/7 by Amazon Security, see [Reporting a Vulnerability](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/SECURITY.md){target="_blank"} for details.

### Review Pull Requests

Review pull requests regularly, comment, suggest, reject, merge and close. Accept only high quality pull-requests. Provide code reviews and guidance on incoming pull requests.

PRs are [labeled](#labels) based on file changes and semantic title. Pay attention to whether labels reflect the current state of the PR and correct accordingly.

Use and enforce [semantic versioning](https://semver.org/) pull request titles, as these will be used for [CHANGELOG](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CHANGELOG.md){target="_blank"} and [Release notes](https://github.com/aws-powertools/powertools-lambda-python/releases) - make sure they communicate their intent at the human level.

> TODO: This is an area we want to automate using the new GitHub GraphQL API.

For issues linked to a PR, make sure `pending release` label is applied to them when merging. [Upon release](#releasing-a-new-version), these issues will be notified which release version contains their change.

See [Common scenarios](#common-scenarios) section for additional guidance.

### Triage New Issues

Manage [labels](#labels), review issues regularly, and create new labels as needed by the project. Remove `triage` label when you're able to confirm the validity of a request, a bug can be reproduced, etc. Give priority to the original author for implementation, unless it is a sensitive task that is best handled by maintainers.

> TODO: This is an area we want to automate using the new GitHub GraphQL API.

Make sure issues are assigned to our [board of activities](https://github.com/orgs/awslabs/projects/51/) and have the right [status](https://docs.powertools.aws.dev/lambda/python/latest/roadmap/#roadmap-status-definition).

Use our [labels](#labels) to signal good first issues to new community members, and to set expectation that this might need additional feedback from the author, other customers, experienced community members and/or maintainers.

Be aware of [casual contributors](https://opensource.com/article/17/10/managing-casual-contributors) and recurring contributors. Provide the experience and attention you wish you had if you were starting in open source.

See [Common scenarios](#common-scenarios) section for additional guidance.

### Triage Bug Reports

Be familiar with [our definition of bug](#is-that-a-bug). If it's not a bug, you can close it or adjust its title and labels - always communicate the reason accordingly.

For bugs caused by upstream dependencies, replace `bug` with `bug-upstream` label. Ask the author whether they'd like to raise the issue upstream or if they prefer us to do so.

Assess the impact and make the call on whether we need an emergency release. Contact other [maintainers](#current-maintainers) when in doubt.

See [Common scenarios](#common-scenarios) section for additional guidance.

### Triage RFCs

RFC is a collaborative process to help us get to the most optimal solution given the context. Their purpose is to ensure everyone understands what this context is, their trade-offs, and alternative solutions that were part of the research before implementation begins.

Make sure you ask these questions in mind when reviewing:

- Does it use our [RFC template](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=RFC%2Ctriage&template=rfc.yml&title=RFC%3A+TITLE)?
- Does the match our [Tenets](https://docs.powertools.aws.dev/lambda/python/latest/#tenets)?
- Does the proposal address the use case? If so, is the recommended usage explicit?
- Does it focus on the mechanics to solve the use case over fine-grained implementation details?
- Can anyone familiar with the code base implement it?
- If approved, are they interested in contributing? Do they need any guidance?
- Does this significantly increase the overall project maintenance? Do we have the skills to maintain it?
- If we can't take this use case, are there alternative projects we could recommend? Or does it call for a new project altogether?

When necessary, be upfront that the time to review, approve, and implement a RFC can vary - see [Contribution is stuck](#contribution-is-stuck). Some RFCs may be further updated after implementation, as certain areas become clearer.

Some examples using our initial and new RFC templates: #92, #94, #95, #991, #1226

### Releasing a new version

!!! note "Only maintainers with write access to this repository can release a new version"

Releasing a new version is a multi-step process that takes up to 2 hours to complete. Most steps are automated - you provide inputs to trigger the release and monitor progress.

**Prerequisites**: Ensure the commit history in the `develop` branch is up to date, commit messages are semantic, and include their respective area (e.g., `feat(logger): <change>`, `chore(ci): ...`).

<!-- markdownlint-disable MD013 -->
**Release Steps**:

1. **Run [end-to-end tests](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/run-e2e-tests.yml)** and ensure they pass
2. **Trigger Release v3 workflow** - Run the [`Release v3`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/release-v3.yml) workflow with two inputs: the new Powertools version (check [latest version](https://pypi.org/project/aws-lambda-powertools/)) and the new Lambda Layer version number. To find the new Lambda Layer version number, go to this [page](https://docs.powertools.aws.dev/lambda/python/latest/#python-313) and increase the version by one (N + 1).
3. **Monitor the release workflow** - it runs tests, publishes to PyPI, deploys Lambda layers to Beta and Prod environments across all commercial regions, runs canary tests, and updates documentation. If it fails, see the [Re-run partial failed Release workflow](#re-run-partial-failed-release-workflow) section
4. **Review and merge documentation/version PRs** - two PRs will be created to update documentation and bump version files. Review, approve and merge both (order doesn't matter)
5. **Deploy GovCloud layers** - Run the [`Layer Deployment (GovCloud)`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/layer_govcloud.yml) workflow with `develop` branch, `Prod` environment, and the Layer version number from step 2.
6. **Deploy China layers** - Run the [`Layer Deployment (Partitions)`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/layers_partitions.yml) workflow with `develop` branch, `Prod` environment, and the Layer version number from step 2.
7. **Update documentation** - Run the [`Rebuild latest docs`](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/rebuild_latest_docs.yml) workflow with `develop` branch and the PyPI package version

Once complete, you can start drafting the release notes to let customers know **what changed and what's in it for them (a.k.a why they should care)**. We have guidelines in the [release notes](#drafting-release-notes) section so you know what good looks like.

#### Re-run partial failed Release workflow

While workflows are designed to be stable, failures can occur during the release process. If the release workflow fails, you have two recovery options:

**Option 1: Re-run failed jobs** The Release v3 pipeline contains several steps, and one of them is `publish_layer`. This step is responsible for building and deploying public Lambda layers. If this step fails due to CloudFormation errors (we deploy approximately 600 layers per release and cannot control CloudFormation quotas), you can re-run only the failed jobs. This will retry the deployment and typically resolves quota-related issues.
<!-- markdownlint-enable MD013 -->

**Option 2: Re-trigger the entire workflow**
If the release fails due to workflow modifications or permission issues that prevent re-running failed jobs, trigger the Release v3 workflow again.

!!! important "Avoid PyPI errors"
    Make sure to select `Skip publishing to PyPI` as it can't publish more than once. This is useful for semi-failed releases when rerunning the entire workflow to avoid duplicate publishes

#### Release process visualized

Every release makes hundreds of checks, security scans, canaries and deployments - all of these are automated.

This is a close visual representation of the main steps (GitHub Actions UI should be the source of truth), along with the approximate time it takes for each key step to complete.

<!-- use 8s to graph steps faster than 8s like seal/checksum that actually takes 0s -->

```mermaid
gantt

title      Release process
dateFormat HH:mm
axisFormat %H:%M

Release commit   : milestone, m1, 10:00,2m

section Seal
    Bump release version        : active, 8s
    Prevent source tampering    : active, 43s
section QA
    Quality checks              : active, 2.2m
section Build
    Checksum                    : active, 8s
    Build release artifact      : active, 39s
    Seal                        : active, 8s
section Provenance
    Attest build                : active, 8s
    Sign attestation            : active, attestation, 10:06, 8s

section Release
    Checksum                    : active, 8s
    PyPi temp credentials       : active, 8s
    Publish PyPi                : active, pypi, 10:07, 29s

PyPi release : milestone, m2, 10:07,1s

section Git release
    Checksum                    : active, after pypi, 8s
    Git Tag                     : active, 8s
    Bump package version        : active, 8s
    Create PR                   : active, 8s
    Upload attestation          : active, 8s

section Layer release
    Build (x86_64+ARM)             : active, layer_build, 10:08, 6m
    Deploy Beta                 : active, layer_beta, after layer_build, 6.3m
    Deploy Prod                 : active, layer_prod, after layer_beta, 6.3m

Layer release : milestone, m3, 10:26,1s

section SAR release
    Deploy Beta                 : active, sar_beta, after layer_beta, 2.2m
    Deploy Prod                 : active, sar_prod, after sar_beta, 2.2m

SAR release : milestone, m4, 10:25,1s

section Docs
    Create PR (Layer ARN)       : active, after layer_prod, 8s
    Release versioned docs      : active, 2.2m

Documentation release : milestone, m5, 10:28,1m

section SSM Parameters
    Update SSM Parameters       : active, 8s

SSM Parameters : milestone, m6, 10:28,1m

section Post-release
    Close pending issues        : active, 8s

section GovCloud
    Publish GovCloud layers (Gamma)    : active, 8s
    Publish GovCloud layers (Prod)    : active, 8s
    GovCloud layers published : milestone, m7


section China
    Publish China layers (Gamma)    : active, 8s
    Publish China layers (Prod)    : active, 8s
    China layers published : milestone, m8

Release complete : milestone, m9, 10:31,2m
```

#### Drafting release notes

!!! info "Make sure the release workflow completed before you edit release notes."

Visit the [Releases page](https://github.com/aws-powertools/powertools-lambda-python/releases) and choose the edit pencil button.

Make sure the `tag` field reflects the new version you're releasing, the target branch field is set to `develop`, and `release title` matches your tag e.g., `v1.26.0`.

You'll notice we group all changes based on their [labels](#labels) like `feature`, `bug`, `documentation`, etc.
!!! question inline end "Spotted a typo?"

    Edit the respective PR title/labels and run the [Release Drafter workflow](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/release-drafter.yml).

!!! question "All good, what's next?"

The best part comes now!

Replace the placeholder `[Human readable summary of changes]` with what you'd like to communicate to customers what this release is all about.

!!! tip inline end "Always put yourself in the customers shoes. Most read the first sentence only to know whether this is for them."

These are some questions to keep in mind when drafting your first or future release notes:

- **Can customers briefly understand the main changes in less than 30s?**
    - _tip: first paragraph is punchy and optimizes for dependabot-like notifications._
- **Are we calling out key contributor(s) to this release?**
- **Is it clear what each change enables/unlocks and before?**
    - _tip: use present and active voice; lead with the answer._
- **Does it include a link to the documentation for each main change?**
    - _tip: release explains what a change unblocks/enables (before/after), docs go in details_
- **Is code snippet better in text or [graphic](https://carbon.now.sh)?**
- **Does code snippet focus on the change only?**
    - _tip: release snippets highlight functionality, no need to be functional (that's docs)_

Once you're happy, hit `Publish release` 🎉🎉🎉.

### Releasing an alpha release

You can manually trigger the `pre-release` workflow to publish alpha releases _(`prerelease`)_ to PyPi when needed.

### Run end to end tests

E2E tests are run on every push to `develop` or manually via [run-e2e-tests workflow](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/run-e2e-tests.yml).

To run locally, you need [AWS CDK CLI](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites) and an [account bootstrapped](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html) (`cdk bootstrap`). With a default AWS CLI profile configured, or `AWS_PROFILE` environment variable set, run `make e2e tests`.

### Releasing a documentation hotfix

You can rebuild the latest documentation without a full release via this [GitHub Actions Workflow](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/rebuild_latest_docs.yml). Choose `Run workflow`, keep `develop` as the branch, and input the latest Powertools for AWS Lambda (Python) version available.

This workflow will update both user guide and API documentation.

### Maintain Overall Health of the Repo

> TODO: Coordinate renaming `develop` to `main`

Keep the `develop` branch at production quality at all times. Backport features as needed. Cut release branches and tags to enable future patches.

### Manage Roadmap

See [Roadmap section](https://docs.powertools.aws.dev/lambda/python/latest/roadmap/)

Ensure the repo highlights features that should be elevated to the project roadmap. Be clear about the feature’s status, priority, target version, and whether or not it should be elevated to the roadmap.

### Add Continuous Integration Checks

Add integration checks that validate pull requests and pushes to ease the burden on Pull Request reviewers. Continuously revisit areas of improvement to reduce operational burden in all parties involved.

### Negative Impact on the Project

<!-- markdownlint-disable MD013 -->
Actions that negatively impact the project will be handled by the admins, in coordination with other maintainers, in balance with the urgency of the issue. Examples would be [Code of Conduct](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/CODE_OF_CONDUCT.md){target="_blank"} violations, deliberate harmful or malicious actions, spam, monopolization, and security risks.
<!-- markdownlint-enable MD013 -->

### Becoming a maintainer

In 2023, we will revisit this. We need to improve our understanding of how other projects are doing, their mechanisms to promote key contributors, and how they interact daily.

We suspect this process might look similar to the [OpenSearch project](https://github.com/opensearch-project/.github/blob/main/MAINTAINERS.md#becoming-a-maintainer){target="_blank" rel="nofollow"}.

## Common scenarios

These are recurring ambiguous situations that new and existing maintainers may encounter. They serve as guidance. It is up to each maintainer to follow, adjust, or handle in a different manner as long as [our conduct is consistent](#uphold-code-of-conduct)

### Contribution is stuck

A contribution can get stuck often due to lack of bandwidth and language barrier. For bandwidth issues, check whether the author needs help. Make sure you get their permission before pushing code into their existing PR - do not create a new PR unless strictly necessary.

For language barrier and others, offer a 1:1 chat to get them unblocked. Often times, English might not be their primary language, and writing in public might put them off, or come across not the way they intended to be.

In other cases, you may have constrained capacity. Use `help wanted` label when you want to signal other maintainers and external contributors that you could use a hand to move it forward.

### Insufficient feedback or information

When in doubt, use `need-more-information` or `need-customer-feedback` labels to signal more context and feedback are necessary before proceeding. You can also use `revisit-in-3-months` label when you expect it might take a while to gather enough information before you can decide.

### Crediting contributions

We credit all contributions as part of each [release note](https://github.com/aws-powertools/powertools-lambda-python/releases){target="_blank"} as an automated process. If you find  contributors are missing from the release note you're producing, please add them manually.

### Is that a bug?

A bug produces incorrect or unexpected results at runtime that differ from its intended behavior. Bugs must be reproducible. They directly affect customers experience at runtime despite following its recommended usage.

Documentation snippets, use of internal components, or unadvertised functionalities are not considered bugs.

### Mentoring contributions

Always favor mentoring issue authors to contribute, unless they're not interested or the implementation is sensitive (_e.g., complexity, time to release, etc._).

Make use of `help wanted` and `good first issue` to signal additional contributions the community can help.

### Long running issues or PRs

Try offering a 1:1 call in the attempt to get to a mutual understanding and clarify areas that maintainers could help.

In the rare cases where both parties don't have the bandwidth or expertise to continue, it's best to use the `revisit-in-3-months` label. By then, see if it's possible to break the PR or issue in smaller chunks, and eventually close if there is no progress.

## E2E framework

### Structure

Our E2E framework relies on [Pytest fixtures](https://docs.pytest.org/en/6.2.x/fixture.html){target="_blank" rel="nofollow"} to coordinate infrastructure and test parallelization - see [Test Parallelization](#test-runner-parallelization) and [CDK CLI Parallelization](#cdk-cli-parallelization).

**tests/e2e structure**

```shell
.
├── __init__.py
├── conftest.py # builds Lambda Layer once
├── logger
│   ├── __init__.py
│   ├── conftest.py  # deploys LoggerStack
│   ├── handlers
│   │   └── basic_handler.py
│   ├── infrastructure.py # LoggerStack definition
│   └── test_logger.py
├── metrics
│   ├── __init__.py
│   ├── conftest.py  # deploys MetricsStack
│   ├── handlers
│   │   ├── basic_handler.py
│   │   └── cold_start.py
│   ├── infrastructure.py # MetricsStack definition
│   └── test_metrics.py
├── tracer
│   ├── __init__.py
│   ├── conftest.py  # deploys TracerStack
│   ├── handlers
│   │   ├── async_capture.py
│   │   └── basic_handler.py
│   ├── infrastructure.py  # TracerStack definition
│   └── test_tracer.py
└── utils
    ├── __init__.py
    ├── data_builder  # build_service_name(), build_add_dimensions_input, etc.
    ├── data_fetcher  # get_traces(), get_logs(), get_lambda_response(), etc.
    ├── infrastructure.py # base infrastructure like deploy logic, etc.
```

Where:

- **`<feature>/infrastructure.py`**. Uses CDK to define the infrastructure a given feature needs.
- **`<feature>/handlers/`**. Lambda function handlers to build, deploy, and exposed as stack output in PascalCase (e.g., `BasicHandler`).
- **`utils/`**. Test utilities to build data and fetch AWS data to ease assertion
- **`conftest.py`**. Deploys and deletes a given feature infrastructure. Hierarchy matters:
    - **Top-level (`e2e/conftest`)**. Builds Lambda Layer only once and blocks I/O across all CPU workers.
    - **Feature-level (`e2e/<feature>/conftest`)**. Deploys stacks in parallel and make them independent of each other.

### Mechanics

Under [`BaseInfrastructure`](https://github.com/aws-powertools/powertools-lambda-python/blob/develop/tests/e2e/utils/infrastructure.py){target="_blank"}, we hide the complexity of deployment and delete coordination under `deploy`, `delete`, and `create_lambda_functions` methods.

This allows us to benefit from test and deployment parallelization, use IDE step-through debugging for a single test, run one, subset, or all tests and only deploy their related infrastructure, without any custom configuration.

> Class diagram to understand abstraction built when defining a new stack (`LoggerStack`)

```mermaid
classDiagram
    class InfrastructureProvider {
        <<interface>>
        +deploy() Dict
        +delete()
        +create_resources()
        +create_lambda_functions() Dict~Functions~
    }

    class BaseInfrastructure {
        +deploy() Dict
        +delete()
        +create_lambda_functions() Dict~Functions~
        +add_cfn_output()
    }

    class TracerStack {
        +create_resources()
    }

    class LoggerStack {
        +create_resources()
    }

    class MetricsStack {
        +create_resources()
    }

    class EventHandlerStack {
        +create_resources()
    }

    InfrastructureProvider <|-- BaseInfrastructure : implement
    BaseInfrastructure <|-- TracerStack : inherit
    BaseInfrastructure <|-- LoggerStack : inherit
    BaseInfrastructure <|-- MetricsStack : inherit
    BaseInfrastructure <|-- EventHandlerStack : inherit
```

### Authoring a new feature E2E test

Imagine you're going to create E2E for Event Handler feature for the first time. Keep the following mental model when reading:

```mermaid
graph LR
    A["1. Define infrastructure"]-->B["2. Deploy/Delete infrastructure"]-->C["3.Access Stack outputs" ]
```

#### 1. Define infrastructure

We use CDK as our Infrastructure as Code tool of choice. Before you start using CDK, you'd take the following steps:

1. Create `tests/e2e/event_handler/infrastructure.py` file
2. Create a new class `EventHandlerStack` and inherit from `BaseInfrastructure`
3. Override `create_resources` method and define your infrastructure using CDK
4. (Optional) Create a Lambda function under `handlers/alb_handler.py`

> Excerpt `tests/e2e/event_handler/infrastructure.py`

```python
class EventHandlerStack(BaseInfrastructure):
    def create_resources(self):
        functions = self.create_lambda_functions()

        self._create_alb(function=functions["AlbHandler"])
        ...

    def _create_alb(self, function: Function):
        vpc = ec2.Vpc.from_lookup(
            self.stack,
            "VPC",
            is_default=True,
            region=self.region,
        )

        alb = elbv2.ApplicationLoadBalancer(self.stack, "ALB", vpc=vpc, internet_facing=True)
        CfnOutput(self.stack, "ALBDnsName", value=alb.load_balancer_dns_name)
        ...
```

> Excerpt `tests/e2e/event_handler/handlers/alb_handler.py`

```python
from aws_lambda_powertools.event_handler import ALBResolver, Response, content_types

app = ALBResolver()


@app.get("/todos")
def hello():
    return Response(
        status_code=200,
        content_type=content_types.TEXT_PLAIN,
        body="Hello world",
        cookies=["CookieMonster", "MonsterCookie"],
        headers={"Foo": ["bar", "zbr"]},
    )


def lambda_handler(event, context):
    return app.resolve(event, context)
```

#### 2. Deploy/Delete infrastructure when tests run

We need to create a Pytest fixture for our new feature under `tests/e2e/event_handler/conftest.py`.

This will instruct Pytest to deploy our infrastructure when our tests start, and delete it when they complete whether tests are successful or not. Note that this file will not need any modification in the future.

> Excerpt `conftest.py` for Event Handler

```python
import pytest

from tests.e2e.event_handler.infrastructure import EventHandlerStack


@pytest.fixture(autouse=True, scope="module")
def infrastructure():
    """Setup and teardown logic for E2E test infrastructure

    Yields
    ------
    Dict[str, str]
        CloudFormation Outputs from deployed infrastructure
    """
    stack = EventHandlerStack()
    try:
        yield stack.deploy()
    finally:
        stack.delete()

```

#### 3. Access stack outputs for E2E tests

Within our tests, we should now have access to the `infrastructure` fixture we defined earlier in `tests/e2e/event_handler/conftest.py`.

We can access any Stack Output using pytest dependency injection.

> Excerpt `tests/e2e/event_handler/test_header_serializer.py`

```python
@pytest.fixture
def alb_basic_listener_endpoint(infrastructure: dict) -> str:
    dns_name = infrastructure.get("ALBDnsName")
    port = infrastructure.get("ALBBasicListenerPort", "")
    return f"http://{dns_name}:{port}"


def test_alb_headers_serializer(alb_basic_listener_endpoint):
    # GIVEN
    url = f"{alb_basic_listener_endpoint}/todos"
    ...
```

### Internals

#### Test runner parallelization

Besides speed, we parallelize our end-to-end tests to ease asserting async side-effects may take a while per test too, _e.g., traces to become available_.

The following diagram demonstrates the process we take every time you use `make e2e` locally or at CI:

```mermaid
graph TD
    A[make e2e test] -->Spawn{"Split and group tests <br>by feature and CPU"}

    Spawn -->|Worker0| Worker0_Start["Load tests"]
    Spawn -->|Worker1| Worker1_Start["Load tests"]
    Spawn -->|WorkerN| WorkerN_Start["Load tests"]

    Worker0_Start -->|Wait| LambdaLayer["Lambda Layer build"]
    Worker1_Start -->|Wait| LambdaLayer["Lambda Layer build"]
    WorkerN_Start -->|Wait| LambdaLayer["Lambda Layer build"]

    LambdaLayer -->|Worker0| Worker0_Deploy["Launch feature stack"]
    LambdaLayer -->|Worker1| Worker1_Deploy["Launch feature stack"]
    LambdaLayer -->|WorkerN| WorkerN_Deploy["Launch feature stack"]

    Worker0_Deploy -->|Worker0| Worker0_Tests["Run tests"]
    Worker1_Deploy -->|Worker1| Worker1_Tests["Run tests"]
    WorkerN_Deploy -->|WorkerN| WorkerN_Tests["Run tests"]

    Worker0_Tests --> ResultCollection
    Worker1_Tests --> ResultCollection
    WorkerN_Tests --> ResultCollection

    ResultCollection{"Wait for workers<br/>Collect test results"}
    ResultCollection --> TestEnd["Report results"]
    ResultCollection --> DeployEnd["Delete Stacks"]
```

#### CDK CLI parallelization

For CDK CLI to work with [independent CDK Apps](https://docs.aws.amazon.com/cdk/v2/guide/apps.html){target="_blank"}, we specify an output directory when synthesizing our stack and deploy from said output directory.

```mermaid
flowchart TD
    subgraph "Deploying distinct CDK Apps"
        EventHandlerInfra["Event Handler CDK App"] --> EventHandlerSynth
        TracerInfra["Tracer CDK App"] --> TracerSynth
       EventHandlerSynth["cdk synth --out cdk.out/event_handler"] --> EventHandlerDeploy["cdk deploy --app cdk.out/event_handler"]

       TracerSynth["cdk synth --out cdk.out/tracer"] --> TracerDeploy["cdk deploy --app cdk.out/tracer"]
    end
```

We create the typical CDK `app.py` at runtime when tests run, since we know which feature and Python version we're dealing with (locally or at CI).

> Excerpt `cdk_app_V39.py` for Event Handler created at deploy phase

```python
from tests.e2e.event_handler.infrastructure import EventHandlerStack
stack = EventHandlerStack()
stack.create_resources()
stack.app.synth()
```

When we run E2E tests for a single feature or all of them, our `cdk.out` looks like this:

```shell
total 8
drwxr-xr-x  18 lessa  staff   576B Sep  6 15:38 event-handler
drwxr-xr-x   3 lessa  staff    96B Sep  6 15:08 layer_build
-rw-r--r--   1 lessa  staff    32B Sep  6 15:08 layer_build.diff
drwxr-xr-x  18 lessa  staff   576B Sep  6 15:38 logger
drwxr-xr-x  18 lessa  staff   576B Sep  6 15:38 metrics
drwxr-xr-x  22 lessa  staff   704B Sep  9 10:52 tracer
```

```mermaid
classDiagram
    class CdkOutDirectory {
        feature_name/
        layer_build/
        layer_build.diff
    }

    class EventHandler {
        manifest.json
        stack_outputs.json
        cdk_app_V39.py
        asset.uuid/
        ...
    }

    class StackOutputsJson {
        BasicHandlerArn: str
        ALBDnsName: str
        ...
    }

    CdkOutDirectory <|-- EventHandler : feature_name/
    StackOutputsJson <|-- EventHandler
```

Where:

- **`<feature>`**. Contains CDK Assets, CDK `manifest.json`, our `cdk_app_<PyVersion>.py` and `stack_outputs.json`
- **`layer_build`**. Contains our Lambda Layer source code built once, used by all stacks independently
- **`layer_build.diff`**. Contains a hash on whether our source code has changed to speed up further deployments and E2E tests

Together, all of this allows us to use Pytest like we would for any project, use CDK CLI and its [context methods](https://docs.aws.amazon.com/cdk/v2/guide/context.html#context_methods){target="_blank"} (`from_lookup`), and use step-through debugging for a single E2E test without any extra configuration.

> NOTE: VSCode doesn't support debugging processes spawning sub-processes (like CDK CLI does w/ shell and CDK App). Maybe [this works](https://stackoverflow.com/a/65339352){target="_blank" rel="nofollow"}. PyCharm works just fine.
