Pin apm-sdks-benchmarks and add tooling to automatically update pin - #12153
Open
sarahchen6 wants to merge 3 commits into
Open
Pin apm-sdks-benchmarks and add tooling to automatically update pin#12153sarahchen6 wants to merge 3 commits into
sarahchen6 wants to merge 3 commits into
Conversation
…utomatically update pin
Contributor
🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
This comment has been minimized.
This comment has been minimized.
sarahchen6
marked this pull request as ready for review
August 6, 2026 20:44
sarahchen6
requested review from
AlexeyKuznetsov-DD,
PerfectSlayer and
erikayasuda
and removed request for
a team
August 6, 2026 20:44
Contributor
There was a problem hiding this comment.
More details
The new updater correctly replaces all five pinned GitLab includes and the runtime clone SHA, rejects malformed commit inputs, and is idempotent when rerun. Workflow syntax parsed successfully and its token, checkout, commit-headless, and PR steps match existing repository automation; no concrete behavioral regression was identified.
📊 Validated against 7 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit 761a752 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 761a752366
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
AlexeyKuznetsov-DD
approved these changes
Aug 6, 2026
AlexeyKuznetsov-DD
left a comment
Contributor
There was a problem hiding this comment.
LGTM, but I proposed optional improvement to use references like *apm_sdks_benchmarks_sha that will reduce future PRs to one line.
Comment on lines
+6
to
22
| # Refs pinned by the "Update apm-sdks-benchmarks reference" workflow | ||
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | ||
| file: '.gitlab/ci-java-spring-petclinic-parallel.yml' | ||
| ref: 'main' | ||
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | ||
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | ||
| file: '.gitlab/ci-java-load-parallel.yml' | ||
| ref: 'main' | ||
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | ||
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | ||
| file: '.gitlab/ci-java-startup-parallel.yml' | ||
| ref: 'main' | ||
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | ||
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | ||
| file: '.gitlab/ci-java-dacapo-parallel.yml' | ||
| ref: 'main' | ||
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | ||
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | ||
| file: '.gitlab/ci-java-post-pr-comment.yml' | ||
| ref: 'main' | ||
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | ||
| - local: ".gitlab/java-benchmark-configs.yml" |
Contributor
There was a problem hiding this comment.
To reduce future PRs, this can be used:
Suggested change
| # Refs pinned by the "Update apm-sdks-benchmarks reference" workflow | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-spring-petclinic-parallel.yml' | |
| ref: 'main' | |
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-load-parallel.yml' | |
| ref: 'main' | |
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-startup-parallel.yml' | |
| ref: 'main' | |
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-dacapo-parallel.yml' | |
| ref: 'main' | |
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-post-pr-comment.yml' | |
| ref: 'main' | |
| ref: 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | |
| - local: ".gitlab/java-benchmark-configs.yml" | |
| # Refs pinned by the "Update apm-sdks-benchmarks reference" workflow | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-spring-petclinic-parallel.yml' | |
| ref: &apm_sdks_benchmarks_sha 'f59ec6a23d07cb63650a2c8c694a45e479ab0dc2' | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-load-parallel.yml' | |
| ref: *apm_sdks_benchmarks_sha | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-startup-parallel.yml' | |
| ref: *apm_sdks_benchmarks_sha | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-dacapo-parallel.yml' | |
| ref: *apm_sdks_benchmarks_sha | |
| - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' | |
| file: '.gitlab/ci-java-post-pr-comment.yml' | |
| ref: *apm_sdks_benchmarks_sha | |
| - local: ".gitlab/java-benchmark-configs.yml" |
| - generate-signing-key | ||
|
|
||
| variables: | ||
| APM_SDKS_BENCHMARKS_SHA: "f59ec6a23d07cb63650a2c8c694a45e479ab0dc2" # pinned by "Update apm-sdks-benchmarks reference" workflow |
Contributor
There was a problem hiding this comment.
Suggested change
| APM_SDKS_BENCHMARKS_SHA: "f59ec6a23d07cb63650a2c8c694a45e479ab0dc2" # pinned by "Update apm-sdks-benchmarks reference" workflow | |
| APM_SDKS_BENCHMARKS_SHA: *apm_sdks_benchmarks_sha |
Comment on lines
+1
to
+50
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| # This script updates the apm-sdks-benchmarks include references and runtime | ||
| # clone SHA used by GitLab CI. | ||
| # Usage: tooling/update_apm_sdks_benchmarks_reference.sh <commit-sha> | ||
|
|
||
| if [[ $# -ne 1 || ! "$1" =~ ^[0-9a-f]{40}$ ]]; then | ||
| echo "Usage: $0 <40-character commit SHA>" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| REF="$1" | ||
| GITLAB_TARGET=".gitlab-ci.yml" | ||
| PROJECT="DataDog/apm-reliability/apm-sdks-benchmarks" | ||
| REFERENCE_PATTERN="( - project: '$PROJECT'\n file: '[^']+'\n ref: ')(?:main|[0-9a-f]{40})(')" | ||
| SHA_PATTERN='( APM_SDKS_BENCHMARKS_SHA: ")[0-9a-f]{40}(")' | ||
|
|
||
| if [[ ! -f "$GITLAB_TARGET" ]]; then | ||
| echo "Error: Target file $GITLAB_TARGET does not exist" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| REFERENCE_COUNT=$(REFERENCE_PATTERN="$REFERENCE_PATTERN" perl -0ne ' | ||
| $count++ while /$ENV{REFERENCE_PATTERN}/g; | ||
| END { print $count // 0 } | ||
| ' "$GITLAB_TARGET") | ||
| SHA_COUNT=$(SHA_PATTERN="$SHA_PATTERN" perl -0ne ' | ||
| $count++ while /$ENV{SHA_PATTERN}/g; | ||
| END { print $count // 0 } | ||
| ' "$GITLAB_TARGET") | ||
|
|
||
| if [[ "$REFERENCE_COUNT" -eq 0 ]]; then | ||
| echo "Error: No $PROJECT references found" >&2 | ||
| exit 1 | ||
| fi | ||
| if [[ "$SHA_COUNT" -eq 0 ]]; then | ||
| echo "Error: No APM_SDKS_BENCHMARKS_SHA variables found" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| REFERENCE_PATTERN="$REFERENCE_PATTERN" REF="$REF" perl -0pi -e ' | ||
| s/$ENV{REFERENCE_PATTERN}/${1}$ENV{REF}${2}/g | ||
| ' "$GITLAB_TARGET" | ||
| SHA_PATTERN="$SHA_PATTERN" REF="$REF" perl -0pi -e ' | ||
| s/$ENV{SHA_PATTERN}/${1}$ENV{REF}${2}/g | ||
| ' "$GITLAB_TARGET" | ||
|
|
||
| echo "Updated $REFERENCE_COUNT apm-sdks-benchmarks references and" \ | ||
| "$SHA_COUNT runtime clone SHA variables to $REF" |
Contributor
There was a problem hiding this comment.
Suggested change
| #!/usr/bin/env bash | |
| set -euo pipefail | |
| # This script updates the apm-sdks-benchmarks include references and runtime | |
| # clone SHA used by GitLab CI. | |
| # Usage: tooling/update_apm_sdks_benchmarks_reference.sh <commit-sha> | |
| if [[ $# -ne 1 || ! "$1" =~ ^[0-9a-f]{40}$ ]]; then | |
| echo "Usage: $0 <40-character commit SHA>" >&2 | |
| exit 1 | |
| fi | |
| REF="$1" | |
| GITLAB_TARGET=".gitlab-ci.yml" | |
| PROJECT="DataDog/apm-reliability/apm-sdks-benchmarks" | |
| REFERENCE_PATTERN="( - project: '$PROJECT'\n file: '[^']+'\n ref: ')(?:main|[0-9a-f]{40})(')" | |
| SHA_PATTERN='( APM_SDKS_BENCHMARKS_SHA: ")[0-9a-f]{40}(")' | |
| if [[ ! -f "$GITLAB_TARGET" ]]; then | |
| echo "Error: Target file $GITLAB_TARGET does not exist" >&2 | |
| exit 1 | |
| fi | |
| REFERENCE_COUNT=$(REFERENCE_PATTERN="$REFERENCE_PATTERN" perl -0ne ' | |
| $count++ while /$ENV{REFERENCE_PATTERN}/g; | |
| END { print $count // 0 } | |
| ' "$GITLAB_TARGET") | |
| SHA_COUNT=$(SHA_PATTERN="$SHA_PATTERN" perl -0ne ' | |
| $count++ while /$ENV{SHA_PATTERN}/g; | |
| END { print $count // 0 } | |
| ' "$GITLAB_TARGET") | |
| if [[ "$REFERENCE_COUNT" -eq 0 ]]; then | |
| echo "Error: No $PROJECT references found" >&2 | |
| exit 1 | |
| fi | |
| if [[ "$SHA_COUNT" -eq 0 ]]; then | |
| echo "Error: No APM_SDKS_BENCHMARKS_SHA variables found" >&2 | |
| exit 1 | |
| fi | |
| REFERENCE_PATTERN="$REFERENCE_PATTERN" REF="$REF" perl -0pi -e ' | |
| s/$ENV{REFERENCE_PATTERN}/${1}$ENV{REF}${2}/g | |
| ' "$GITLAB_TARGET" | |
| SHA_PATTERN="$SHA_PATTERN" REF="$REF" perl -0pi -e ' | |
| s/$ENV{SHA_PATTERN}/${1}$ENV{REF}${2}/g | |
| ' "$GITLAB_TARGET" | |
| echo "Updated $REFERENCE_COUNT apm-sdks-benchmarks references and" \ | |
| "$SHA_COUNT runtime clone SHA variables to $REF" | |
| #!/usr/bin/env bash | |
| set -euo pipefail | |
| # This script updates the shared SHA used by the apm-sdks-benchmarks includes | |
| # and runtime clone in GitLab CI. | |
| # Usage: tooling/update_apm_sdks_benchmarks_reference.sh <commit-sha> | |
| if [[ $# -ne 1 || ! "$1" =~ ^[0-9a-f]{40}$ ]]; then | |
| echo "Usage: $0 <40-character commit SHA>" >&2 | |
| exit 1 | |
| fi | |
| REF="$1" | |
| GITLAB_TARGET=".gitlab-ci.yml" | |
| PROJECT="DataDog/apm-reliability/apm-sdks-benchmarks" | |
| REFERENCE_PATTERN="( - project: '$PROJECT'\n file: '[^']+'\n ref: &apm_sdks_benchmarks_sha ')(?:main|[0-9a-f]{40})(')" | |
| SHA_PATTERN=' APM_SDKS_BENCHMARKS_SHA: \*apm_sdks_benchmarks_sha' | |
| if [[ ! -f "$GITLAB_TARGET" ]]; then | |
| echo "Error: Target file $GITLAB_TARGET does not exist" >&2 | |
| exit 1 | |
| fi | |
| REFERENCE_COUNT=$(REFERENCE_PATTERN="$REFERENCE_PATTERN" perl -0ne ' | |
| $count++ while /$ENV{REFERENCE_PATTERN}/g; | |
| END { print $count // 0 } | |
| ' "$GITLAB_TARGET") | |
| SHA_COUNT=$(SHA_PATTERN="$SHA_PATTERN" perl -0ne ' | |
| $count++ while /$ENV{SHA_PATTERN}/g; | |
| END { print $count // 0 } | |
| ' "$GITLAB_TARGET") | |
| if [[ "$REFERENCE_COUNT" -ne 1 ]]; then | |
| echo "Error: Expected one anchored $PROJECT reference, found $REFERENCE_COUNT" >&2 | |
| exit 1 | |
| fi | |
| if [[ "$SHA_COUNT" -ne 1 ]]; then | |
| echo "Error: Expected APM_SDKS_BENCHMARKS_SHA to use the shared anchor" >&2 | |
| exit 1 | |
| fi | |
| REFERENCE_PATTERN="$REFERENCE_PATTERN" REF="$REF" perl -0pi -e ' | |
| s/$ENV{REFERENCE_PATTERN}/${1}$ENV{REF}${2}/g | |
| ' "$GITLAB_TARGET" | |
| echo "Updated the shared apm-sdks-benchmarks SHA to $REF" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Pin our
apm-sdks-benchmarksreference to the current latest commit SHA onmain. Add tooling to update this pinned SHA.Motivation
Currently, any change made to
apm-sdks-benchmarkswill take immediate effect in this repo. We should pin this reference to protect CI and releases.Additional Notes
The tooling automation will run once a week at the same frequency in which we update our gradle dependencies. It can also be manually triggered. The PR also goes in tandem with https://github.com/DataDog/apm-sdks-benchmarks/pull/228 (merged) which allows reading
apm-sdks-benchmarksrepo and https://github.com/DataDog/apm-sdks-benchmarks/pull/230 (waiting on review) which applies the logic to clone the repo at the specified SHA to thepost-pr-commentjob.Testing
The
tooling/update_apm_sdks_benchmarks_reference.shscript works locally, and the.github/workflows/update-apm-sdks-benchmarks-reference.yamlis based on existing working GitHub workflows. However, the workflow cannot be tested end-to-end until it lands onmasterbecause thedd-octo-stspermissions are scoped to and passed frommaster.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]