Skip to content

v2: MarQS powered job executions#1149

Merged
ericallam merged 8 commits into
mainfrom
v2/graphile-to-marqs
Jun 6, 2024
Merged

v2: MarQS powered job executions#1149
ericallam merged 8 commits into
mainfrom
v2/graphile-to-marqs

Conversation

@ericallam

Copy link
Copy Markdown
Member

This PR is the start of the conversion of v2 run executions being powered by MarQS instead of graphile worker. This is currently shipped behind two flags, both of which need to be turned on to use MarQS instead of graphile:

  • Set V2_MARQS_ENABLED=1 env var
  • Set the v2MarqsEnabled = true for an organization

When V2_MARQS_ENABLED=1 is set, concurrency settings will be double-written to both MarQS and the legacy graphile system. Enabling MarQS v2 first before enabling orgs allows for the gradual migration to MarQS instead of having to do a "stop the world" migration.

This PR also contains a much improved MarQS "queue choice" algorithm, that will be used for v3 and v2. Other relevant environment variables for operation v2 MarQS:

  • The existing DEFAULT_ORG_EXECUTION_CONCURRENCY_LIMIT env var is used by MarQS v2
  • V2_MARQS_DEFAULT_ENV_CONCURRENCY controls the default concurrency per RuntimeEnvironment. This should be set very high as the current graphile system doesn't have the concept of environment-level concurrency, only org.
  • V2_MARQS_CONSUMER_POOL_ENABLED controls whether to start a consumer pool.
  • V2_MARQS_CONSUMER_POOL_SIZE controls the number of consumers the pool should manage
  • V2_MARQS_CONSUMER_POLL_INTERVAL_MS controls how often each consumer polls
  • V2_MARQS_QUEUE_SELECTION_COUNT controls how many queues MarQS will consider on each dequeueing attempt
  • V2_MARQS_VERBOSE setting this to "1" will log out a bunch of MarQS stuff

@changeset-bot

changeset-bot Bot commented Jun 6, 2024

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 373206d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ericallam ericallam force-pushed the v2/graphile-to-marqs branch from 6cb5e64 to 861807d Compare June 6, 2024 12:22
Comment thread apps/webapp/app/v3/marqs/index.server.ts Outdated
Comment thread apps/webapp/app/v3/marqs/index.server.ts Outdated
@ericallam ericallam force-pushed the v2/graphile-to-marqs branch from f1aff16 to 373206d Compare June 6, 2024 12:56
@ericallam ericallam merged commit 8fc8f57 into main Jun 6, 2024
@ericallam ericallam deleted the v2/graphile-to-marqs branch June 6, 2024 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants