v2: MarQS powered job executions#1149
Merged
Merged
Conversation
|
6cb5e64 to
861807d
Compare
matt-aitken
reviewed
Jun 6, 2024
matt-aitken
reviewed
Jun 6, 2024
matt-aitken
approved these changes
Jun 6, 2024
f1aff16 to
373206d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is the start of the conversion of v2 run executions being powered by MarQS instead of graphile worker. This is currently shipped behind two flags, both of which need to be turned on to use MarQS instead of graphile:
V2_MARQS_ENABLED=1env varv2MarqsEnabled = truefor an organizationWhen
V2_MARQS_ENABLED=1is set, concurrency settings will be double-written to both MarQS and the legacy graphile system. Enabling MarQS v2 first before enabling orgs allows for the gradual migration to MarQS instead of having to do a "stop the world" migration.This PR also contains a much improved MarQS "queue choice" algorithm, that will be used for v3 and v2. Other relevant environment variables for operation v2 MarQS:
DEFAULT_ORG_EXECUTION_CONCURRENCY_LIMITenv var is used by MarQS v2V2_MARQS_DEFAULT_ENV_CONCURRENCYcontrols the default concurrency per RuntimeEnvironment. This should be set very high as the current graphile system doesn't have the concept of environment-level concurrency, only org.V2_MARQS_CONSUMER_POOL_ENABLEDcontrols whether to start a consumer pool.V2_MARQS_CONSUMER_POOL_SIZEcontrols the number of consumers the pool should manageV2_MARQS_CONSUMER_POLL_INTERVAL_MScontrols how often each consumer pollsV2_MARQS_QUEUE_SELECTION_COUNTcontrols how many queues MarQS will consider on each dequeueing attemptV2_MARQS_VERBOSEsetting this to"1"will log out a bunch of MarQS stuff