Skip to content
This repository was archived by the owner on Mar 23, 2026. It is now read-only.

fix lambda state reset method to shutdown esm workers correctly#13671

Merged
thrau merged 1 commit intomainfrom
fix-state-reset-on-lambda
Feb 2, 2026
Merged

fix lambda state reset method to shutdown esm workers correctly#13671
thrau merged 1 commit intomainfrom
fix-state-reset-on-lambda

Conversation

@thrau
Copy link
Copy Markdown
Member

@thrau thrau commented Jan 31, 2026

Motivation

While working on a project, I noticed that after calling localstack state reset, the logs indicated that ESM workers were not shut down correctly. I have a simple SQS <- ESM -> Lambda setup, and after resetting the state, I continued to get error messages that look like they're coming from the worker:

2026-01-31T14:34:46.821  INFO --- [et.reactor-1] localstack.request.http    : POST /_localstack/state/reset => 200
2026-01-31T14:34:47.230 ERROR --- [functhread12] l.s.l.e.esm_worker         : Error while polling messages for event source arn:aws:sqs:us-east-1:000000000000:messages-queue: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the ReceiveMessage operation: The specified queue does not exist.
2026-01-31T14:34:48.377 ERROR --- [functhread12] l.s.l.e.esm_worker         : Error while polling messages for event source arn:aws:sqs:us-east-1:000000000000:messages-queue: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the ReceiveMessage operation: The specified queue does not exist.

This PR simply replicates what on_before_stop does, which is iterating over the ESM workers and stopping them. Additionally, we're now also clearing the state that holds references to the workers.

Changes

  • ESM workers are now shut down and removed correctly when resetting lambda state

Tests

Tested manually with a simple project:

  • Create a simple SQS/Lambda ESM setup, observe the esm worker logs that they are polling from the queue
  • call localstack state reset, and observe that the logs are no longer producing the error output
  • re-deploy the same resources again and make sure it still works (ensuring no breaking state side-effects)

Related

@thrau thrau added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes labels Jan 31, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 31, 2026

Test Results - Preflight, Unit

23 088 tests  ±0   21 229 ✅ ±0   6m 7s ⏱️ -1s
     1 suites ±0    1 859 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit 1eef829. ± Comparison against base commit 59e81ce.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 31, 2026

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 2s ⏱️ +2s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit 1eef829. ± Comparison against base commit 59e81ce.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 31, 2026

Test Results (amd64) - Integration, Bootstrap

    5 files      5 suites   2h 0m 52s ⏱️
3 767 tests 3 492 ✅ 275 💤 0 ❌
3 773 runs  3 492 ✅ 281 💤 0 ❌

Results for commit 1eef829.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 31, 2026

LocalStack Community integration with Pro

    2 files  ±    0      2 suites  ±0   1h 30m 33s ⏱️ - 28m 8s
3 743 tests  - 1 435  3 464 ✅  - 1 326  279 💤  - 109  0 ❌ ±0 
3 745 runs   - 1 435  3 464 ✅  - 1 326  281 💤  - 109  0 ❌ ±0 

Results for commit 1eef829. ± Comparison against base commit 59e81ce.

This pull request removes 1435 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

joe4dev
joe4dev approved these changes Feb 2, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@joe4dev joe4dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this state reset fix 🩹

sidenote: async pollers for event invokes and provisioned concurrency might need similar considerations

localstack-core/localstack/services/lambda_/provider.py
def on_before_state_reset(self):
for esm_worker in self.esm_workers.values():
esm_worker.stop_for_shutdown()
self.esm_workers = {}
Copy link
Copy Markdown
Member

@joe4dev joe4dev Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: Thank you for explicitly resetting that state (which is done implicitly upon stopping relying on its ephemeral nature) ✨

@thrau thrau merged commit a8c43f2 into main Feb 2, 2026
54 of 59 checks passed
@thrau thrau deleted the fix-state-reset-on-lambda branch February 2, 2026 12:26
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@dominikschubert dominikschubert Awaiting requested review from dominikschubert

@dfangl dfangl Awaiting requested review from dfangl

@gregfurman gregfurman Awaiting requested review from gregfurman

1 more reviewer

@joe4dev joe4dev joe4dev approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes semver: patch Non-breaking changes which can be included in patch releases

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@thrau @joe4dev