Add a skill for flake analysis#187530
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new skill markdown file (SKILL.md) for analyzing GitHub flake issues in the flutter/flutter repository, outlining instructions, rules, and steps to collect failing builds and fetch logs. Feedback on the changes includes correcting minor grammatical and typographical errors, as well as parameterizing hardcoded values in the curl payload to allow dynamic replacement of the bucket and builder names.
| -H 'sec-fetch-site: cross-site' \ | ||
| -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36' \ | ||
| -H 'x-return-encrypted-headers: all' \ | ||
| --data-raw '{"builder":{"project":"flutter","bucket":"try","builder":"Linux repo_checks"},"buildNumber":<buildNumber>,"mask":{"fields":"id,builder,builderInfo,number,canceledBy,createdBy,createTime,startTime,endTime,cancelTime,status,statusDetails,summaryMarkdown,output,steps,tags,schedulingTimeout,executionTimeout,gracePeriod,ancestorIds,retriable"}}' |
Contributor
There was a problem hiding this comment.
The curl command payload has hardcoded values for "bucket" ("try") and "builder" ("Linux repo_checks"). Since the flaking builds can come from different builders (e.g., Windows plugin_test in prod), these should be parameterized as <bucket> and <check_name> so the agent knows to replace them dynamically based on the URL being analyzed.
Suggested change
| --data-raw '{"builder":{"project":"flutter","bucket":"try","builder":"Linux repo_checks"},"buildNumber":<buildNumber>,"mask":{"fields":"id,builder,builderInfo,number,canceledBy,createdBy,createTime,startTime,endTime,cancelTime,status,statusDetails,summaryMarkdown,output,steps,tags,schedulingTimeout,executionTimeout,gracePeriod,ancestorIds,retriable"}}' | |
| --data-raw '{"builder":{"project":"flutter","bucket":"<bucket>","builder":"<check_name>"},"buildNumber":<buildNumber>,"mask":{"fields":"id,builder,builderInfo,number,canceledBy,createdBy,createTime,startTime,endTime,cancelTime,status,statusDetails,summaryMarkdown,output,steps,tags,schedulingTimeout,executionTimeout,gracePeriod,ancestorIds,retriable"}}' |
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
reidbaker
requested changes
Jun 3, 2026
Contributor
reidbaker
left a comment
There was a problem hiding this comment.
You also need to add yourself to CODEOWNERs for this skill.
| # Flake analysis instructions | ||
|
|
||
| *Context* | ||
| flutter/flutter repo has the following folder structure. |
Contributor
There was a problem hiding this comment.
I would eventually want this out of the skill but for a first pass I think it can stay in.
Member
Author
There was a problem hiding this comment.
I'm fine removing now if you're not opposed. I think it made sense in your packages skill but makes less sense for a skill living inside flutter proper
|
|
||
| For example to find information about the issue https://github.com/flutter/flutter/issues/174116 you can call `gh issue view 174116 --repo flutter/flutter --json=author,body,id,labels,number,state,title,url` | ||
|
|
||
| The title will contain the name of the flaking check, and the body will contain urls which link to instances where the check failed or flaked, each on a new line, directly after "Flaky builds:". You should also inspect the comments and find each comment made by "login": "fluttergithubbot", and extract the urls from there as well (the format will be the same, with an example flake at the top, and the full list immediately after the "Flaky builds:" header). |
Contributor
There was a problem hiding this comment.
Not all github issues are flake issues. If this skill is called on an issue that is not a flake it should short circuit out. Can you add a section for how to determine if an issue is a flake issue and if the issue does not meet the criteria then stop working and tell the user.
Member
Author
There was a problem hiding this comment.
Added short circuit case
| <!-- TODO include a .gitignored directory where agent can dump local files in flutter/packages --> | ||
|
|
||
| ``` | ||
| curl 'https://cr-buildbucket.appspot.com/prpc/buildbucket.v2.Builds/GetBuild' \ |
Contributor
There was a problem hiding this comment.
I think this came from a skill I posted. Eventually we will want to make this have less specific information.
Member
Author
There was a problem hiding this comment.
Changed to be slightly more general (for bucket (prod/try) and check name)
reidbaker
approved these changes
Jun 3, 2026
This was referenced Jun 4, 2026
This was referenced Jun 5, 2026
auto-submit Bot
pushed a commit
to flutter/packages
that referenced
this pull request
Jun 5, 2026
Roll Flutter from 2ba5420a7049 to 1bdf4af29076 (43 revisions) flutter/flutter@2ba5420...1bdf4af 2026-06-05 engine-flutter-autoroll@skia.org Roll Packages from 03352b5 to 61bdbb4 (5 revisions) (flutter/flutter#187612) 2026-06-05 engine-flutter-autoroll@skia.org Roll Skia from 6e003d7f69c8 to a47a9a2c8ae5 (1 revision) (flutter/flutter#187610) 2026-06-05 engine-flutter-autoroll@skia.org Roll Dart SDK from aad8be4ce307 to 6a9a0efe66eb (10 revisions) (flutter/flutter#187609) 2026-06-05 engine-flutter-autoroll@skia.org Roll Skia from 494f1bf55f51 to 6e003d7f69c8 (2 revisions) (flutter/flutter#187607) 2026-06-05 engine-flutter-autoroll@skia.org Roll Fuchsia Linux SDK from ZE1Jy9CtVVi-tjBAE... to N_LiSaBSUsE2LDZgG... (flutter/flutter#187597) 2026-06-05 engine-flutter-autoroll@skia.org Roll Skia from 59556fdb8c33 to 494f1bf55f51 (2 revisions) (flutter/flutter#187596) 2026-06-04 engine-flutter-autoroll@skia.org Roll Skia from 8eb107046fd5 to 59556fdb8c33 (1 revision) (flutter/flutter#187590) 2026-06-04 34871572+gmackall@users.noreply.github.com Remove `embedded_android_views_integration_test.dart` (flutter/flutter#187465) 2026-06-04 burak.karahan@mail.ru Remove Material imports from rendering editable tests (flutter/flutter#186951) 2026-06-04 jason-simmons@users.noreply.github.com [Impeller] Wait for the Vulkan device to become idle before destroying Vulkan objects in the AHBSwapchainImplVK destructor (flutter/flutter#187477) 2026-06-04 chris@bracken.jp [iOS] Eliminate unnecessary redeclaration of FlutterDisplayLink (flutter/flutter#187557) 2026-06-04 engine-flutter-autoroll@skia.org Roll Skia from 928ded2a31af to 8eb107046fd5 (1 revision) (flutter/flutter#187583) 2026-06-04 34871572+gmackall@users.noreply.github.com Log stdout in adb.dart (flutter/flutter#187531) 2026-06-04 mvincentong@gmail.com Clarify RouterDelegate popRoute bubbling (flutter/flutter#186875) 2026-06-04 engine-flutter-autoroll@skia.org Roll Skia from 928ded2a31af to 8eb107046fd5 (1 revision) (flutter/flutter#187584) 2026-06-04 1063596+reidbaker@users.noreply.github.com Add updating-android-sdk agent skill for rolling Android SDK in CIPD (flutter/flutter#187576) 2026-06-04 Rusino@users.noreply.github.com Fixing alignment issue (flutter/flutter#187518) 2026-06-04 brackenavaron@gmail.com [Material Cross Imports] Clean up Material Divider usages (flutter/flutter#187300) 2026-06-04 engine-flutter-autoroll@skia.org Roll Skia from cecc0e0da9ae to 928ded2a31af (6 revisions) (flutter/flutter#187574) 2026-06-04 31859944+LongCatIsLooong@users.noreply.github.com Use swift demangle to verify internal Swift symbols (flutter/flutter#186835) 2026-06-04 1063596+reidbaker@users.noreply.github.com Add android 37 platform and build tools to script for android cipd bundle creation (flutter/flutter#187571) 2026-06-04 jason-simmons@users.noreply.github.com [Impeller] Increase the precision of the IPSampleWithTileModeOES coords parameter to match the input coordinates in the tiled_texture_fill_external shader (flutter/flutter#187545) 2026-06-04 engine-flutter-autoroll@skia.org Roll Packages from b11504f to 03352b5 (4 revisions) (flutter/flutter#187569) 2026-06-04 iinozemtsev@google.com Roll Dart SDK to Dart 3.13 beta2 (flutter/flutter#187555) 2026-06-04 engine-flutter-autoroll@skia.org Roll Skia from 611e3f8ceb93 to cecc0e0da9ae (1 revision) (flutter/flutter#187562) 2026-06-04 6655696+guidezpl@users.noreply.github.com Add step to bootstrap Flutter tool in coverage workflow (flutter/flutter#187199) 2026-06-04 engine-flutter-autoroll@skia.org Roll Skia from 4fdb859c8da7 to 611e3f8ceb93 (4 revisions) (flutter/flutter#187554) 2026-06-04 engine-flutter-autoroll@skia.org Roll Skia from 0020aae33f63 to 4fdb859c8da7 (2 revisions) (flutter/flutter#187552) 2026-06-04 engine-flutter-autoroll@skia.org Roll Fuchsia Linux SDK from ap7MhLX4TdpWRrLS_... to ZE1Jy9CtVVi-tjBAE... (flutter/flutter#187550) 2026-06-04 stuartmorgan@google.com Add vector_math to package issue template (flutter/flutter#187536) 2026-06-04 jason-simmons@users.noreply.github.com Manual roll Dart SDK from d39850bf4a01 to 3b70b98fa7c0 (flutter/flutter#187519) 2026-06-04 engine-flutter-autoroll@skia.org Roll Skia from d625048c853a to 0020aae33f63 (20 revisions) (flutter/flutter#187539) 2026-06-04 smille2003@yandex.ru [Impeller][Windows] fix black screen on OpenGL fallback (flutter/flutter#187288) 2026-06-04 97480502+b-luk@users.noreply.github.com Fix unintentionally joined path contours (flutter/flutter#187522) 2026-06-03 matt.boetger@gmail.com fix: resolve issue #177379 by using lazy buildDirectory.dir() API in build.gradle template (flutter/flutter#187127) 2026-06-03 34871572+gmackall@users.noreply.github.com Add a skill for flake analysis (flutter/flutter#187530) 2026-06-03 30870216+gaaclarke@users.noreply.github.com adds linux impeller project flag (flutter/flutter#186982) 2026-06-03 codedoctor@linwood.dev Add support for stylus buttons (flutter/flutter#183369) 2026-06-03 46920873+gabrimatic@users.noreply.github.com Prevent Cubic transform from looping on out-of-range input (flutter/flutter#185875) 2026-06-03 bdero@google.com [Impeller] Reland: Allow attaching specific texture mip levels and slices (flutter/flutter#187470) 2026-06-03 kjlubick@users.noreply.github.com [skia] Update image deserial proc (flutter/flutter#185041) 2026-06-03 112751483+shivanshu877@users.noreply.github.com docs: update Impeller advanced blend docs for framebuffer fetch (flutter/flutter#185457) 2026-06-03 ahmedsameha1@gmail.com Handle#6537 fifth grouped tests (flutter/flutter#183720) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Example output for issue #185070
Flake Analysis Report:
android_engine_vulkan_testsThis report analyzes the failures of the
android_engine_vulkan_testsshard in theflutter/flutterrepository across 15 recent failed builds between Build 1658 and Build 2495.Executive Summary
The failures in the
android_engine_vulkan_testsbuilder fall into three distinct categories:5ba6df7acef516f7dab78587cf30ce20). This strongly points to a frame buffer rendering or screenshot capture failure (e.g., capturing a completely blank, black, or transparent frame) on the Vulkan emulator.Build Failure Summary Table
5ba6df7acef516f7dab78587cf30ce20Failed to uninstall app(Process exited abnormally)Failed to uninstall app(Process exited abnormally)5ba6df7acef516f7dab78587cf30ce20Failed to uninstall app(Process exited abnormally)Failed to uninstall app(Process exited abnormally)Failed to uninstall app,nativeDriver has not been initializedFailed to uninstall app,Failed to fulfill WaitFor due to remote error5ba6df7acef516f7dab78587cf30ce205ba6df7acef516f7dab78587cf30ce205ba6df7acef516f7dab78587cf30ce205ba6df7acef516f7dab78587cf30ce20Detailed Findings
1. Skia Gold Image Mismatch (Primary Blocker)
In builds 2495, 2313, 1965, 1813, 1691, and 1658, the following 5 tests failed because the screenshot captured during the test did not match any approved golden image:
android_engine_test.vulkan.external_texture_surface_texture_smiley_faceandroid_engine_test.vulkan.external_texture_surface_producer_smiley_faceandroid_engine_test.vulkan.platform_view_tap_color_change_initialandroid_engine_test.vulkan.fluttered_rendered_blue_rectangleandroid_engine_test.vulkan.texture_layer_hybrid_composition_platform_view.blue_orange_gradient_portraitImportant
Identical Hash Anomaly:
For all 5 tests, the screenshot captured had the exact same hash:
5ba6df7acef516f7dab78587cf30ce20.Since these tests render completely different UI components (e.g. a blue rectangle vs. a blue/orange gradient vs. a smiley face texture), it is mathematically impossible for them to produce the same valid image digest. This indicates that the screenshot capture mechanism (or the emulator's Vulkan rendering context) returned a blank, black, or default-cleared buffer.
2. Infrastructure / Emulator Issues (Transient Flakes)
In builds 2337, 2317, 2283, 2280, 2268, and 2198, the tests failed due to VM/emulator communication errors:
Failed to uninstall appfollowed byProcess exited abnormally with exit code 1.Local 'nativeDriver' has not been initializedandFailed to fulfill NativeCommand due to remote error.Failed to fulfill WaitFor due to remote error.These are standard emulator health and connectivity flakes, where the test runner loses connection to the emulator device during apk installation/uninstallation or command dispatch.
Recommendations & Next Steps
Investigate if the screenshotting code in the Flutter Driver / engine Vulkan integration tests is correctly capturing the Vulkan swapchain. The identical hash suggests a blank/empty frame buffer is being captured.
Check the Flutter Gold console for digest
5ba6df7acef516f7dab78587cf30ce20to visually confirm if the images are blank/black.To address the secondary infrastructure failures, consider adding retry logic for emulator ADB connection, or updating the Android emulator image version used on the LUCI bots.