Skip to content

Added Kafka worker-side consumer logic - #342

Merged
tylerharter merged 8 commits into
open-lambda:mainfrom
Yashwanth-Ranjan-Singaravel:feature/kafka-trigger-poc
Oct 20, 2025
Merged

tylerharter merged 8 commits into
open-lambda:mainfrom
Yashwanth-Ranjan-Singaravel:feature/kafka-trigger-poc

Conversation

@Yashwanth-Ranjan-Singaravel

Copy link
Copy Markdown
Contributor

Added logic to run Kafka consumers during startup of workers that can act as triggers to invoke lambda functions on new messages from configured brokers.

@Yashwanth-Ranjan-Singaravel

Copy link
Copy Markdown
Contributor Author

I'll definitely clean up the code by removing redundant comments and logging statements before I mark this as ready!

@tylerharter tylerharter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

Comment thread go/boss/lambdastore/store.go Outdated
Comment thread go/common/lambdaConfig.go Outdated
Comment thread go/common/lambdaConfig.go Outdated
Comment thread go/worker/lambda/lambdaFunction.go Outdated
Comment thread go/worker/event/server.go Outdated
Comment thread go/worker/event/server.go Outdated
Comment thread go/worker/event/kafkaServer.go Outdated
Comment thread go/worker/event/kafkaServer.go Outdated
Comment thread go/worker/event/kafkaServer.go Outdated
Comment thread go/worker/event/kafkaServer.go Outdated
Comment thread go/common/lambdaConfig.go Outdated
}

if header.Name == LambdaConfigFilename {
// Check if this is the config file handle exact matches (ol.yaml)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

separate PR

@tylerharter tylerharter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

Comment thread go/worker/event/kafkaServer.go Outdated
Comment thread go/worker/event/kafkaServer.go
Comment thread go/worker/event/kafkaServer.go Outdated
Comment thread go/worker/event/kafkaServer.go Outdated
Comment thread go/worker/event/kafkaServer.go Outdated
Comment thread go/worker/event/kafkaServer.go
Comment thread go/worker/event/server.go Outdated
Comment thread go/worker/event/server.go Outdated
Comment thread go/worker/event/server.go Outdated
Comment thread go/worker/event/server.go Outdated
@Yashwanth-Ranjan-Singaravel
Yashwanth-Ranjan-Singaravel marked this pull request as ready for review October 9, 2025 10:13
Comment thread go/worker/event/server.go Outdated
slog.Info("Received kill signal, cleaning up.")
shutdown(pidPath, s)

// Clean up all resources in order

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant with other case? Let's have one way we cleanup (probably the other code). For that to run, though, Main needs to return. Which means ListenAndServe needs to return. Which means we need to close the mux. But perhaps that is tricky if we're using the default? Bridget already has feedback about using non-defaults for both the servers: https://github.com/open-lambda/open-lambda/pull/343/files. Let's coordinate.

@tylerharter
tylerharter merged commit 90f6cbd into open-lambda:main Oct 20, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants