From 03fe7e72b68ee5970b9e09d8d83ad35797d6a37b Mon Sep 17 00:00:00 2001
From: runningcode There are two authentication mechanisms for using Fladle. It is recommended to use user authentication on local development machines to avoid sharing credentials and a service account on CI. Credentials are stored in Create a service account. Service accounts aren't subject to spam checks or captcha prompts, which could
+ otherwise block your CI builds. Create a service account with an Editor role in the
+ Google Cloud Platform console. Enable required APIs. After logging in using the service account: In the Google Developers Console API Library
+ page, enable the Google Cloud Testing API and Cloud Tool Results API. To enable these APIs, type these API names into
+ the search box at the top of the console, and then click Enable API on the overview page for that API. After creating the account go to Configure the Fladle extension serviceAccountCredentials to point to the credentials. Above instructions are based on Google instruction for authenticating with CI. See also Flank's instructions for authenticating with a service account. Breaking API Change Previously, only app modules could be picked up as root-level modules in Fulladle. This has changed, now either app or library modules can be picked up (though we try app modules first). If a library module is picked up as a the root-level module, it must specifiy a Breaking API Change Use lazy properties Fixes #92 PR Thanks pawelpasterz Breaking API change Add time units for timeout. Breaking API Change additionalTestApks now uses ListProperty instead of the previous Map. This is to allow for lazy configuration of the provided files. Warning Minimum supported Gradle version is now 5.4. Breaking API change debugApk and instrumentationApk now use Lazy Property API to avoid resolving at configuration time. Breaking API Change serviceAccountCredentials now uses Lazy Property API. See Configuration for details on how to set it. PR Warning Minimum required Gradle Version is now 5.1. Warning Dropped support for Flank 7.X and lower. Previous users of The following configuration options must be set inside the fladle block. See the sample configuration below. There is also a groovy sample and a kotlin sample. User Authentication is also possible. See Authentication. Optionally, the serviceAccountCredentials can be set with environment variables but then the projectId parameter must be set. See Authentication for more info. Note Set the variant to automatically configure for testing. A build variant is a combination of buildFlavor and buildType.
+This must also be set when testing against a non-default variant.
+For example: 'debug' or 'freeDebug'.
+Put this inside your Fladle block. Whether or not we should use the android test orchestrator to run this tests.
+Set this to true when the build.gradle file includes Set multiple testTargets to be run by Flank. These are used to whitelist or blacklist test classes, test cases and test annotations.
+See Google Cloud Firebase docs for more information. A list of devices to run the tests against. When list is empty, a default device will be used. Each device entry is a map. The valid keys in the map are The projectId is a unique identifier which can be found in the project's URL: Need a different Flank version? Specify it with To use a snapshot: Need more than 50 shards? Use Flank To use a different version: Specify custom flank maven coordinates. This is a string representing the path to the app's debug apk.
+Supports wildcard characters.
+Optional, prefer to set variant. This is a string representing the path to the app's instrumentaiton apk.
+Supports wildcard characters.
+Optional, prefer to set variant.
+InstrumenationApk should not be set when using roboScript. Paths to additional test configurations.
+Order matters. A test apk is run with the nearest previous listed app apk.
+For library modules, add them to the list with a Whether to automatically log in using a preconfigured google account. More Info Environment variables are mirrored as extra options to the Deprecated Use The maximum number of shards. Fladle will throw an error when used together with maxTestShards or numUniformShards. The maximum number of shards. Fladle will throw an error when used together with testShards or numUniformShards. The amount of time tests within a shard should take.
+When set to > 0, the shard count is dynamically set based on time up to the maximum limit defined by maxTestShards
+2 minutes (120) is recommended.
+default: -1 (unlimited) The number of times to repeat each test. Uses Flank's default value when not specified. Give a name to a custom flank task and configure its options. The name is appended to the end of the flank task. For example In the above example, the configuration is inherited from the outer fladle config but with the repeatTests property set to 100. Running The cloud storage location for historical test runs. This must be set in order to use smart sharding. The amount of shards used is set by The history name for your test results (an arbitrary string label; default: the application's label from the APK manifest). All tests which use the same history name will have their results grouped together in the Firebase console in a time-ordered test history list. The number of times to retry failed tests. Default is 0. Max is 10.
+Setting the value to 1 will mean that test are retried once. If the test fails then succeeds after the retry the run
+will be marked as "successful". The matrix with a flaky test will be marked as flaky. The ABI split of the application that should be tested (e.g. "x86"). Only required if the application under test uses ABI splits and the debug APK is selected automatically (via variant) instead of manually (via debugApk). If the application uses ABI splits, and this property isn't specified, an arbitrary ABI split will be selected. A list of paths that will be copied from the device's storage to the designated results bucket after the test is complete. These must be absolute paths under List of regex that is matched against bucket paths (for example: The max time test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 45m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed.
+Examples:
+* 1h -> 1 hour
+* 5m -> 5 minutes
+* 200s -> 200 seconds
+* 100 -> 100 seconds Enable video recording during the test. Enabled by default. Monitor and record performance metrics: CPU, memory, network usage, and FPS (game-loop only). Enabled by default. The name of a Google Cloud Storage bucket where raw test results will be stored. Keeps the full path of downloaded files from a Google Cloud Storage bucket. Required when file names are not unique. Disabled by default. The name of a unique Google Cloud Storage object within the results bucket where raw test results will be stored. The default is a timestamp with a random suffix. Disables sharding. All tests will run on the same device. Useful for parameterized tests which do not support sharding. (default: false) Disables smart flank JUnit XML uploading. Useful for preventing timing data from being updated. (default: false)
+What is Smart Flank? The fully-qualified Java class name of the instrumentation test runner (default: the test manifest is parsed to determine the class name). The local directory to store the test results. Folder is DELETED before each run to ensure only artifacts from the new run are saved. This directory is relative to the working directory of Flank which is by default Always run - these tests are inserted at the beginning of every shard. Useful if you need to grant permissions or login before other tests run The flags below are only available with Flank 20.05.0 or higher. The max time this test run can execute before it is cancelled. s (seconds), m (minutes), h (hours) suffixes are acceptable, mixes like 1h45m are currently not supported (default: unlimited).
+Examples: Always return successful task completion even when there are failed tests. Useful when parsing JUnit XML to determine failure. (default: false) Specifies the number of shards into which you want to evenly distribute test cases. The shards are run in parallel on separate devices. For example, if your test execution contains 20 test cases and you specify four shards, each shard executes five test cases. The number of shards should be less than the total number of test cases. The number of shards specified must be >= 1 and <= 50. This option cannot be used along maxTestShards and is not compatible with smart sharding (Smart Flank). If you want to take benefits of smart sharding use maxTestShards instead. (default: null) A key-value map of additional details to attach to the test matrix.(clientDetails in Google Cloud Docs) Arbitrary key-value pairs may be attached to a test matrix to provide additional context about the tests being run. When consuming the test results, such as in Cloud Functions or a CI system, these details can add additional context such as a link to the corresponding pull request. (Access Client Details).
+These can be used to provide additional context about the environment where the tests are being run. A list of device-path: file-path pairs that indicate the device paths to push files to the device before starting tests, and the paths of files to push. Device paths must be under absolute, whitelisted paths (${EXTERNAL_STORAGE}, or ${ANDROID_DATA}/local/tmp). Source file paths may be in the local filesystem or in Google Cloud Storage (gs://…). The name of the network traffic profile, for example LTE, HSPA, etc, which consists of a set of parameters to emulate network conditions when running the test (default: no network shaping; see available profiles listed by the The path to a Robo Script JSON file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. You can guide the Robo test to perform specific actions by recording a Robo Script in Android Studio and then specifying this argument. Learn more at DOCS. List of robo_directives that you can use to customize the behavior of Robo test. The type specifies the action type of the directive, which may take on values click, text or ignore. Each directive is list of String = [type, key, value]. Each key should be the Android resource name of a target UI element and each value should be the text input for that element. Values are only permitted for text type elements, so no value should be specified for click and ignore type elements. Output style of execution status. May be one of [ Default value is single. Flank provides two ways for parsing junit xml results.
+New way uses google api instead of merging xml files, but can generate slightly different output format.
+This flag allows fallback for legacy xml junit results parsing Enables creating an additional local junit result on local storage with failure nodes on passed flaky tests. Runs a sanityRobo test.
+ Set default test time expressed in seconds, used for calculating shards. (default: 120.0s) Set default parameterized class test time expressed in seconds, used for calculating shards. (default: 2x defaultTestTime => 240s) A list of up to 100 additional APKs to install, in addition to those being directly tested. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. Enable using average time from previous tests duration when using SmartShard and tests did not run before. (default: false) Disable flank results upload on gcloud storage. (default: false) Specifies a group of packages, classes, and/or test cases to run in each shard (a group of test cases).
+The shards are run in parallel on separate devices. You can use this option up to 50 times to specify multiple shards when one or more physical devices are selected,
+or up to 500 times when no physical devices are selected.
+Note: If you include the flags Whether to grant runtime permissions on the device before the test begins. By default, all permissions are granted. PERMISSIONS must be one of: all, none The type of test to run. TYPE must be one of: instrumentation, robo, game-loop. Use if you want to be sure there is only one type of tests being run (flank enables to run mixed types of test in one run). A list of game-loop scenario labels (default: None). Each game-loop scenario may be labeled in the APK manifest file with one or more arbitrary strings, creating logical groupings (e.g. GPU_COMPATIBILITY_TESTS).
+If A list of game-loop scenario numbers which will be run as part of the test (default: all scenarios).
+A maximum of 1024 scenarios may be specified in one test matrix, but the maximum number may also be limited by the overall test A list of one or two Android OBB file names which will be copied to each test device before the tests will run (default: None).
+Each OBB file name must conform to the format as specified by Android (e.g. A list of OBB required filenames. OBB file name must conform to the format as specified by Android e.g.
+ If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected.
+Normally, 2 or more attempts are made if a potential infrastructure issue is detected.
+This feature is for latency sensitive workloads. The incidence of execution failures may be significantly greater for
+fail-fast matrices and support is more limited because of that expectation. Appending additional option to flank root yaml. This option is useful when you would like to test options before official fladle support is added.
+Multiple options are supported. Single option Multiple options Allow appending additional config to gcloud root yaml. This option is useful when you would like to test option
+before it is available on Fladle. Supports both single and multiple properties. Single option Multiple options Enable to automatically build the app and test APKs before Enable to return immediately after invoking tests, without waiting for results. (default: The app APK and the instrumentation APK are expected to have already been generated before calling runFlank. To generate APKs, run You can also have Fladle build them for you by using the If you receive an error like this, it is likely caused by invalid fladle extension confiuration.
+The syntax was changed in the If you receive a similar error, please check configuration for a sample configuration. Still having trouble? Check the #flank channel in the Firebase Community Slack The Gradle Plugin for Firebase Test Lab and Flank. Fladle is the easiest way to scale your instrumentation tests using Firebase Test Lab. Apply the gradle plugin and runFlank! You can easily run many hours of instrumentation tests using hundreds of devices in Test Lab and get the results in just few minutes. Fladle simplifies the configuration necessary to scale your tests using Firebase Test Lab and Flank. Fladle is mentioned in Fragmented Podcast #163. Skip to ~29 minutes. Flank is a parallel test runner for Firebase Test Lab. Also read this medium post. Thank you Doist for your sponsorship! The following companies test their Android app using Fladle. Want to appear here? File an issue or open a PR Multi module testing can be done by manually specifying additionalTestApks or applying the Fulladle plugin to automacally gather all the additional test apks. See also this article for a full setup and instructions including integration with CI. Apply the Fulladle plugin at the root of the project. Configure the Fladle extension. Warning If using buildFlavors or testing against a non default variant, you will need to specify the variant you want to test in the fulladleModuleConfig block. Run the tests.
+ First assemble all your debug apks and test apks.
+ Note When using flavors, make sure to assemble your buildVariants as well. Run Flank!
+ Fulladle will pick Flank configurations from the All of the above configurations are optional, Flank will default to the top-level configurations if you don't override anything here. For details about these configurations, refer to configuration docs. You may want to exclude a library module from testing when using Fulladle. You can do so by setting the Fulladle does not provide the ability to control which module ends up as the root-level module or as an additional module. Either one of app modules or library modules can become a root-level module. If a library module ends up as a root-level module, it needs to specify a The root-level configuration (e.g. Fulladle might still have some rough edges, but we'd love feedback. Please join us in the Firebase Community Slack with any feedback you may have.
+You can also file Fladle Github issues. When filing a bug report, please include the Flank version number, the Fladle version number and the output of the following: Using Fladle takes 3 steps: Apply the Fladle plugin. Follow instructions here Root Application module Configure Authentication using these steps. Warning If using buildFlavors or testing against a non default variant, variant must also configured Run your tests! First assemble your debug apk and test apk.
+ Note When using flavors, make sure to assemble your buildVariants. Run Flank!
+ Here are some recipes to use to achieve various goals in flank. For additional recipes or suggestions, please file an
+issue on Github. This recipe will keep track of test durations automatically on firebase test lab and try to split up test runs in to 120 second shards up to maximum of 50 shards. Use Gradle's dynamic version syntax to declare a dynamic version. Warning Dynamic versions lead to non-reproducible builds since Gradle will check for new versions periodically based on how long versions are cached. Create a local release branch from Update Update the current version and next version in Take one last look
+ Commit all local changes
+ Create a tag and push it
+ Pushing the tag automatically triggers the release workflow which: Creates a GitHub Release with auto-generated notes Release to Maven Central Merge the release branch to master
+ Update Commit your changes
+ Push your changes
+ By default, results are placed in the A merged junit report is available in On failures only, an html report with links to failed shards is also generated in the results directory with the name A cost report is also available The Gradle Plugin for Firebase Test Lab and Flank. Fladle is the easiest way to scale your instrumentation tests using Firebase Test Lab. Apply the gradle plugin and runFlank! You can easily run many hours of instrumentation tests using hundreds of devices in Test Lab and get the results in just few minutes. Fladle simplifies the configuration necessary to scale your tests using Firebase Test Lab and Flank. Fladle is mentioned in Fragmented Podcast #163. Skip to ~29 minutes. Flank is a parallel test runner for Firebase Test Lab. Also read this medium post. Thank you Doist for your sponsorship! Sponsor Fladle! The following companies test their Android app using Fladle. Want to appear here? File an issue or open a PR There are two authentication mechanisms for using Fladle. It is recommended to use user authentication on local development machines to avoid sharing credentials and a service account on CI. Credentials are stored in Create a service account. Service accounts aren't subject to spam checks or captcha prompts, which could otherwise block your CI builds. Create a service account with an Editor role in the Google Cloud Platform console. Enable required APIs. After logging in using the service account: In the Google Developers Console API Library page, enable the Google Cloud Testing API and Cloud Tool Results API. To enable these APIs, type these API names into the search box at the top of the console, and then click Enable API on the overview page for that API. After creating the account go to Configure the Fladle extension serviceAccountCredentials to point to the credentials. Above instructions are based on Google instruction for authenticating with CI. See also Flank's instructions for authenticating with a service account. Breaking API Change Previously, only app modules could be picked up as root-level modules in Fulladle. This has changed, now either app or library modules can be picked up (though we try app modules first). If a library module is picked up as a the root-level module, it must specifiy a Breaking API Change Use lazy properties Fixes #92 PR Thanks pawelpasterz Breaking API change Add time units for timeout. Breaking API Change additionalTestApks now uses ListProperty instead of the previous Map. This is to allow for lazy configuration of the provided files. Warning Minimum supported Gradle version is now 5.4. Breaking API change debugApk and instrumentationApk now use Lazy Property API to avoid resolving at configuration time. Breaking API Change serviceAccountCredentials now uses Lazy Property API. See Configuration for details on how to set it. PR Warning Minimum required Gradle Version is now 5.1. Warning Dropped support for Flank 7.X and lower. Previous users of 404 - Not found
+
+ Authentication
+User authentication
+~/.flank.
+
+./gradlew flankAuth./gradlew runFlankService account credentials
+
+
+Keys, click Add Key -> Create a new Key. Select JSON for the key type. This will download the json credentials.
+
Changelog
+Unreleased
+0.21.0
+
+
+0.20.0
+
+
+0.19.0
+
+
+0.18.0
+
+0.17.5
+
+
+0.17.4
+
+
+abi option to support selecting a particular debug APK in projects that use ABI splits to produce multiple APKs. PR0.17.2
+
+
+0.17.0
+
+
+0.16.3
+
+
+main in JavaExec task.0.16.2
+
+
+0.16.1
+
+
+debugApk through either the root fladle block or the module's own fulladleModuleConfig block.0.16.0
+
+
+0.15.1
+
+
+smartFlankGcsPath to samples Fixes #2360.15.0
+
+
+async flag. PR.dependOnAssemble PR Thanks asadsalman0.14.1
+
+
+0.14.0
+
+
+testShards Fixes #204 PR#212 Thanks pawelpasterz0.13.1
+
+
+
+
+default-test-timedefault-class-test-timeadditional-apksuse-average-test-time-for-new-testsdisable-results-upload0.13.0
+
+
+0.12.1
+
+
+0.12.0
+
+
+full-junit-result and legacy-junit-result. #170 Fixes #157. Thanks MatthewTPage0.11.0
+
+
+additional-test-apks for modules that have tests. PR0.10.2
+
+
+timeoutMin has been renamed to testTimeout. PR Thanks pawelpasterz0.10.1
+
+
+0.10.0
+
+
+0.9.4
+
+
+0.9.2
+0.9.1
+
+
+0.9.0
+
+
+0.8.1
+
+0.8.0
+
+
+List<Map<String, String>> instead of a List<Device>. See the [#README.md] for an example. PR Thanks zlippard.keep-file-path. PR Thanks tahirhajizada.0.7.0
+
+
+0.6.7
+
+
+0.6.6
+
+
+0.6.5
+
+
+0.6.4
+
+
+0.6.3
+
+
+0.6.2
+
+0.6.1
+
+0.6.0
+
+
+0.5.2
+
+
+0.5.1
+
+
+0.5.0
+
+
+0.4.1
+
+
+0.4.0
+
+
+Breaking Changes:
+clearPackageData = true will now need to use:
+environmentVariables = [
+ "clearPackageData": "true"
+]
+0.3.8
+
+
+0.3.7
+
+
+0.3.6
+
+
+0.3.5
+
+
+0.3.4
+
+
+0.3.3
+
+
+0.3.2
+
+
+0.3.1
+
+
+0.3.0
+
+
+0.2.12
+
+
+0.2.11
+
+
+0.2.10
+
+
+0.2.9
+
+
+
+
+
+
+
+
+
+
Configuration
+serviceAccountCredentials
+serviceAccountCredentials = project.layout.projectDirectory.file("flank-gradle-5cf02dc90531.json")
+serviceAccountCredentials.set("project.layout.projectDirectory.file("flank-gradle-5cf02dc90531.json")
+variant
+variant must be set if using buildFlavors in order to automatically configure the debugApk and testApk.variant = "freeDebug"
+variant.set("freeDebug")
+Sample Configuration
+fladle {
+ // Required parameters
+ serviceAccountCredentials = project.layout.projectDirectory.file("flank-gradle-5cf02dc90531.json")
+ variant = "freeDebug"
+
+ // Optional parameters
+ useOrchestrator = false
+ environmentVariables = [
+ "clearPackageData": "true"
+ ]
+ directoriesToPull = [
+ "/sdcard/screenshots"
+ ]
+ filesToDownload = [
+ ".*/screenshots/.*"
+ ]
+ testTargets = [
+ "class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView"
+ ]
+ testTimeout = "15m"
+ recordVideo = false
+ performanceMetrics = false
+ devices = [
+ [ "model": "SmallPhone.arm", "version": "28" ],
+ [ "model": "MediumPhone.arm", "version": "33" ]
+ ]
+ projectId("flank-gradle")
+ flankVersion = "23.10.1"
+ debugApk = "$buildDir/outputs/apk/debug/sample-debug.apk"
+ instrumentationApk = "$buildDir/outputs/apk/androidTest/debug/sample-debug-androidTest.apk"
+ additionalTestApks = [
+ "- app: $buildDir/outputs/apk/debug/sample-debug.apk",
+ " test: $buildDir/outputs/apk/androidTest/debug/sample2-debug-androidTest.apk",
+ "- test: ${rootProject.buildDir}/database/module/outputs/apk/database-module-androidTest.apk"
+ ]
+ autoGoogleLogin = true
+ maxTestShards = 8 //or numUniformShards=5 cannot use both.
+ shardTime = 120
+ smartFlankGcsPath = "gs://tmp_flank/flank/test_app_android.xml"
+ configs {
+ oranges {
+ useOrchestrator.set(false)
+ testTargets.set(project.provider { [
+ "class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#runAndFail"
+ ] })
+ }
+ }
+ resultsBucket("my-results-bucket-name")
+ keepFilePath = true
+ runTimout = "45m"
+ ignoreFailedTests = false
+ disableSharding = false
+ smartFlankDisableUpload = false
+ testRunnerClass = "com.foo.TestRunner"
+ localResultsDir = "flank"
+ clientDetails = [
+ "key1": "value1",
+ "key2": "value2"
+ ]
+ testTargetsAlwaysRun = [
+ "com.example.TestSuite#test1",
+ "com.example.TestSuite#test2"
+ ]
+ otherFiles = [
+ "/sdcard/dir1/file1.txt": "/my/example/path/file1.txt",
+ "/sdcard/dir2/file2.txt": "/my/example/path/file2.txt"
+ ]
+ networkProfile = "LTE"
+ roboDirectives = [
+ ["click", "button1", ""],
+ ["ignore", "button2"],
+ ["text", "field1", "my text"],
+ ]
+ outputStyle = 'multi'
+ legacyJunitResult = false
+ fullJunitResult = false
+ additionalApks = [
+ "gs://path/to/app1.apk",
+ "localPath/to/app2.apk"
+ ]
+ defaultTestTime = 5.3
+ defaultClassTestTime = 180.5
+ useAverageTestTimeForNewTests = true
+ disableResultsUpload = true
+ grantPermissions = "none"
+ type = "game-loop"
+ scenarioLabels = [
+ "label1",
+ "label2"
+ ]
+ scenarioNumbers = [ 1, 23, 52 ]
+ obbFiles = [
+ "local/file/path/test1.obb",
+ "local/file/path/test2.obb"
+ ]
+ obbNames = [
+ "patch.0300110.com.example.android.obb",
+ "patch.0300111.com.example.android.obb"
+ ]
+ testTargetsForShard = [
+ "package com.package1.for.shard1, com.package2.for.shard1",
+ "class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2",
+ "class com.foo.ClassForShard3; package com.package.for.shard3"
+ ]
+ failFast = true
+ dependOnAssemble = true
+}
+useOrchestrator
+testOptions.execution 'ANDROID_TEST_ORCHESTRATOR'useOrchestrator = true
+useOrchestrator.set(true)
+testTargets
+testTargets = [
+ "class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView"
+]
+testTargets.set(listOf(
+ "class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView"
+))
+devices
+model, version, orientation, and locale. When a key is not set or is null, a default value will be used.devices = [
+ [ "model": "MediumPhone.arm", "version": "26" ],
+ [ "model": "MediumPhone.arm", "version": "33" ]
+]
+devices.set(listOf(
+ mapOf("model" to "MediumPhone.arm", "version" to "26" ),
+ mapOf("model" to "MediumPhone.arm", "version" to "33" )
+))
+projectId
+https://console.firebase.google.com/project/<projectId>
+This is automatically discovered based on the service credential by default.projectId = "flank-gradle"
+projectId.set("flank-gradle")
+flankVersion
+flankVersion.flankVersion = "flank_snapshot"`
+flankVersion.set("flank_snapshot")
+8.1.0.flankVersion = "23.10.1"
+flankVersion.set("23.10.1")
+flankCoordinates
+flankCoordinates = "com.github.flank:flank"
+flankCoordinates.set("com.github.flank:flank")
+debugApk
+debugApk = project.provider { "${buildDir.toString()}/outputs/apk/debug/*.apk" }
+debugApk.set(project.provider { "${buildDir.toString()}/outputs/apk/debug/*.apk" })
+instrumentationApk
+instrumentationApk = project.provider { "${buildDir.toString()}/outputs/apk/androidTest/debug/*.apk" }
+instrumentationApk.set(project.provider { "${buildDir.toString()}/outputs/apk/androidTest/debug/*.apk" })
+additionalTestApks
+- test: in front. For test apks which belong to an application module, add them with test:.
+It is not required to list an app apk here. If there is no app apk listed in additionalTestApks, the test apks are run against the debugApk.additionalTestApks.value(project.provider { [
+"- app: ../main/app/build/output/apk/debug/app.apk",
+" test: ../main/app/build/output/apk/androidTest/debug/app-test.apk",
+"- app: ../sample/app/build/output/apk/debug/sample-app.apk",
+" test: ../sample/app/build/output/apk/androidTest/debug/sample-app-test.apk",
+"- test: ../feature/room/build/output/apk/androidTest/debug/feature-room-test.apk",
+"- test: ../library/databases/build/output/apk/androidTest/debug/sample-databases-test.apk"
+]})
+additionalTestApks.value(project.provider { listOf(
+"- app: ../main/app/build/output/apk/debug/app.apk",
+" test: ../main/app/build/output/apk/androidTest/debug/app-test.apk",
+"- app: ../sample/app/build/output/apk/debug/sample-app.apk",
+" test: ../sample/app/build/output/apk/androidTest/debug/sample-app-test.apk",
+"- test: ../feature/room/build/output/apk/androidTest/debug/feature-room-test.apk",
+"- test: ../library/databases/build/output/apk/androidTest/debug/sample-databases-test.apk"
+)})
+autoGoogleLogin
+autoGoogleLogin = false
+autoGoogleLogin.set(false)
+environmentVariables
+am instrument -e KEY1 VALUE1 … command and passed to your test runner (typically AndroidJUnitRunner). ExamplesenvironmentVariables = [
+ // Whether or not to remove all shared state from your device's CPU and memory after each test. [More info](https://developer.android.com/training/testing/junit-runner)
+ "clearPackageData": "true"
+]
+environmentVariables = mapOf(
+ // Whether or not to remove all shared state from your device's CPU and memory after each test. [More info](https://developer.android.com/training/testing/junit-runner)
+ "clearPackageData" to "true"
+)
+testShards
+maxTestShards instead.testShards = 5
+testShards.set(5)
+maxTestShards
+maxTestShards = 8
+maxTestShards.set(8)
+shardTime
+shardTime = 120
+shardTime.set(120)
+repeatTests
+repeatTests = 1
+repeatTests.set(1)
+configs
+runFlank becomes runFlank<name>.configs {
+ repeatOneHundred {
+ // DSL sugar for container elements is missing (= syntax): https://github.com/gradle/gradle/issues/9987
+ repeatTests.set(100)
+ }
+}
+configs {
+ create("repeatOneHundred") {
+ repeatTests.set(100)
+ }
+}
+runFlankRepeateOneHundred will execute this custom configuration.smartFlankGcsPath
+maxTestShards. The bucket (such as flank_data in the example) must already exist in order for it to be used.smartFlankGcsPath = 'gs://flank_data/results/JUnitReport.xml'
+smartFlankGcsPath.set("gs://flank_data/results/JUnitReport.xml")
+resultsHistoryName
+resultsHistoryName = 'android-history'
+resultsHistoryName.set("android-history")
+flakyTestAttempts
+flakyTestAttempts = 0
+flakyTestAttempts.set(0)
+abi
+abi = "arm64-v8a"
+abi.set("arm64-v8a")
+directoriesToPull
+/sdcard or /data/local/tmp. Path names are restricted to the characters a-zA-Z0-9_-./+. The paths /sdcard and /data will be made available and treated as implicit path substitutions. E.g. if /sdcard on a particular device does not map to external storage, the system will replace it with the external storage path prefix for that device.directoriesToPull = [
+ '/sdcard/tempDir1', '/data/local/tmp/tempDir2'
+]
+directoriesToPull.set(listOf(
+ "/sdcard/tempDir1", "/data/local/tmp/tempDir2"
+))
+filesToDownload
+2019-01-09_00:13:06.106000_YCKl/shard_0/SmallPhone.arm-28-en-portrait/bugreport.txt) for files to be downloaded after a flank run. The results are downloaded to the APP_MODULE/build/fladle/RESULTS directory where RESULTS can be set by localResultsDir var otherwise defaulting to results/.filesToDownload = [
+ '.*/sdcard/tempDir1/.*', '.*/data/local/tmp/tempDir2/.*'
+]
+filesToDownload.set(listOf(
+ ".*/sdcard/tempDir1/.*", ".*/data/local/tmp/tempDir2/.*"
+))
+testTimeout
+testTimeout = "1h"
+testTimeout.set("1h")
+recordVideo
+recordVideo = true
+recordVideo.set(true)
+performanceMetrics
+performanceMetrics = true
+performanceMetrics.set(true)
+resultsBucket
+resultsBucket = "my-gcs-bucket-name"
+resultsBucket.set("my-gcs-bucket-name")
+keepFilePath
+keepFilePath = false
+keepFilePath.set(false)
+resultsDir
+resultsDir = "result-dir-${getTimeStamp()}"
+resultsDir.set("result-dir-${getTimeStamp()}")
+disableSharding
+disableSharding = false
+disableSharding.set(false)
+smartFlankDisableUpload
+smartFlankDisableUpload = false
+smartFlankDisableUpload.set(false)
+testRunnerClass
+testRunnerClass = "com.example.MyCustomTestRunner"
+testRunnerClass.set("com.example.MyCustomTestRunner")
+localResultsDir
+build/fladle or build/fladle/<flavorname>.localResultsDir = "my-results-dir"
+localResultsDir.set("my-results-dir")
+testTargetsAlwaysRun
+testTargetsAlwaysRun = [
+ 'class com.example.MyTestClass'
+]
+testTargetsAlwaysRun.set(listOf(
+ "class com.example.MyTestClass"
+))
+runTimeout
+
+
+runTimeout = "15m"
+runTimeout.set("15m")
+ignoreFailedTests
+ignoreFailedTests = false
+ignoreFailedTests.set(false)
+numUniformShards
+numUniformShards = 50
+numUniformShards.set(50)
+clientDetails
+clientDetails = [
+ "test-type": "PR",
+ "build-number": "132"
+]
+clientDetails.set(mapOf(
+ "test-type" to "PR",
+ "build-number" to "132"
+))
+otherFiles
+otherFiles = [
+ "/sdcard/dir1/file1.txt": "local/file.txt",
+ "/sdcard/dir2/file2.jpg": "gs://bucket/file.jpg",
+]
+otherFiles.set(mapOf(
+ "/sdcard/dir1/file1.txt" to "local/file.txt",
+ "/sdcard/dir2/file2.jpg" to "gs://bucket/file.jpg",
+))
+networkProfile
+flank test network-profiles list command). This feature only works on physical devices.networkProfile = "LTE"
+networkProfile.set("LTE")
+roboScript
+roboScript = "my-robo-script.json"
+roboScript.set("my-robo-script.json")
+roboDirectives
+roboDirectives = [
+ ["test, "input_resource_name", "message"],
+ ["click, "button_resource_name", ""],
+]
+roboDirectives.set(listOf(
+ listOf("test", "input_resource_name", "message"),
+ listOf("click", "button_resource_name", ""),
+))
+outputStyle
+verbose, multi, single].
+For runs with only one test execution the default value is 'verbose', in other cases 'multi' is used as the default. The output style 'multi' is not displayed correctly on consoles which don't support ANSI codes, to avoid corrupted output use single or verbose.multi displays separated status for each shard execution in separated line, lines are updated over time. Similar to gradle output when running multiple tasks in parallel. Requires ANSI codes support.single displays shortened status of all executions in single line. Similar to gcloud output when running with sharding. Should work on any console.outputSyle = "single"
+outputStyle.set("single")
+legacyJunitResult
+legacyJunitResult = false
+legacyJunitResult.set(false)
+fullJunitResult
+fullJunitResult = false
+fullJunitResult.set(false)
+sanityRobo
+instrumentationApk, roboDirectives, roboScript and additionalTestApks must be blank or empty.sanityRobo = true
+sanityRobo.set(true)
+defaultTestTime
+defaultTestTime = 1.2
+defaultTestTime.set(1.2)
+defaultClassTestTime
+defaultClassTestTime = 245.5
+defaultClassTestTime.set(245,5)
+additionalApks
+additionalApks = [
+ "gs://path/to/app1.apk",
+ "localPath/to/app2.apk"
+]
+additionalApks.set(
+ project.provider {
+ listOf("gs://path/to/app1.apk", "localPath/to/app2.apk")
+ }
+)
+useAverageTestTimeForNewTests
+useAverageTestTimeForNewTests = true
+useAverageTestTimeForNewTests.set(true)
+disableResultsUpload
+disableResultsUpload = true
+disableResultsUpload.set(true)
+testTargetsForShard
+environmentVariables or testTargets when running testTargetsForShard, the flags are applied to all the shards you create.
+You can also specify multiple packages, classes, or test cases in the same shard by separating each item with a comma.
+To specify both package and class in the same shard, separate package and class with semi-colons.testTargetsForShard = [
+ "package com.package1.for.shard1, com.package2.for.shard1",
+ "class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2",
+ "class com.foo.ClassForShard3; package com.package.for.shard3"
+ ]
+testTargetsForShard.set(
+ project.provider {
+ listOf(
+ "package com.package1.for.shard1, com.package2.for.shard1",
+ "class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2",
+ "class com.foo.ClassForShard3; package com.package.for.shard3"
+ )
+ }
+)
+grantPermissions
+grantPermissions = "none"
+grantPermissions.set("none")
+type
+type = "game-loop"
+type.set("game-loop")
+scenarioLabels
+--scenario-numbers and --scenario-labels are specified together, Firebase Test Lab will first execute each scenario from --scenario-numbers.
+It will then expand each given scenario label into a list of scenario numbers marked with that label, and execute those scenarios.scenarioLabels = [
+ "label1",
+ "label2"
+]
+scenarioLabels.set(
+ project.provider {
+ listOf("label1", "label2")
+ }
+)
+scenarioNumbers
+--timeout setting.scenarioNumbers = [ 1, 23, 52 ]
+scenarioNumbers.set(
+ project.provider {
+ listOf(1, 23, 52)
+ }
+)
+obbFiles
+[main|patch].0300110.com.example.android.obb) and will be installed into <shared-storage>/Android/obb/<package-name>/ on the test device.obbFiles = [
+ "local/file/path/test1.obb",
+ "local/file/path/test2.obb"
+]
+obbFiles.set(
+ project.provider {
+ listOf(
+ "local/file/path/test1.obb",
+ "local/file/path/test2.obb"
+ )
+ }
+)
+obbNames
+[main|patch].0300110.com.example.android.obb which will be installed into <shared-storage>/Android/obb/<package-name>/ on the device.obbNames = [
+ "patch.0300110.com.example.android.obb",
+ "patch.0300111.com.example.android.obb"
+]
+obbNames.set(
+ project.provider {
+ listOf(
+ "patch.0300110.com.example.android.obb",
+ "patch.0300111.com.example.android.obb"
+ )
+ }
+)
+failFast
+failFast = true
+failFast.set(true)
+additionalFlankOptions
+additionalFlankOptions = "new-property: true"
+additionalFlankOptions.set("new-property: true")
+additionalFlankOptions = """
+ new-property: true
+ other-new-property: force
+""".stripIndent()
+additionalFlankOptions.set("""
+ new-property: true
+ other-new-property: force
+""".trimIndent())
+additionalGcloudOptions
+additionalGcloudOptions = "new-property: true"
+additionalGcloudOptions.set("new-property: true")
+additionalGcloudOptions = """
+ new-property: true
+ other-new-property: force
+ """.stripIndent()
+additionalGcloudOptions.set("""
+ new-property: true
+ other-new-property: force
+""".trimIndent())
+dependOnAssemble
+runFlank executes. (default: false)dependOnAssemble = true
+dependOnAssemble.set(true)
+async
+false)async = true
+async.set(true)
+
+
FAQ
+Error APK file not found
+assembleDebug and assembleDebugAndroidTest before running runFlank. dependOnAssemble property.No signature of method
+0.9.X releases in order to avoid touching files during the configuration phase.
+No signature of method: flank_4vvjv7w3oopge32w1tl9cs6e4.fladle() is applicable for argument types: (flank_4vvjv7w3oopge32w1tl9cs6e4$_run_closure1) values: [flank_4vvjv7w3oopge32w1tl9cs6e4$_run_closure1@649a2315]
+ Possible solutions: file(java.lang.Object), find(), findAll(), file(java.lang.Object, org.gradle.api.PathValidation), files([Ljava.lang.Object;), findAll(groovy.lang.Closure)
+Debugging
+./gradlew runFlank -PdumpShards Will dump shards and exit the process without running the tests../gradlew printYml Will print out the current yaml configuration to be passed to Flank.More help?
+
+
Welcome to Fladle
+New? Get Started Here
+Flank
+Corporate Sponsors
+
Trusted by
+
+
+

+
Multi-module testing
+Fulladle Plugin
+
+
+plugins {
+ id 'com.osacky.fulladle' version '0.20.0'
+}
+plugins {
+ id("com.osacky.fulladle") version "0.20.0"
+}
+fladle {
+ serviceAccountCredentials = project.layout.projectDirectory.file("flank-gradle-service-account.json")
+}
+fladle {
+ serviceAccountCredentials.set(project.layout.projectDirectory.file("flank-gradle-service-account.json"))
+}
+./gradlew assembleDebug assembleDebugAndroidTest
+./gradlew :app:assembleFreeDebug :app:assembleFreeDebugAndroidTest./gradlew runFlank
+Overriding configurations in modules
+fladle block in the root build.gradle file. You may want to override some of these configurations for certain modules, you can add the following block to any Android library module to override its configurations:fulladleModuleConfig {
+ clientDetails = [
+ "test-type": "PR",
+ "build-number": "132"
+ ]
+ maxTestShards = 3
+ environmentVariables = [
+ "clearPackageData": "true"
+ ]
+ debugApk = "app.apk"
+ variant = "vanillaDebug"
+}
+fulladleModuleConfig {
+ clientDetails.set(mapOf(
+ "test-type" to "PR",
+ "build-number" to "132",
+ ))
+ maxTestShards.set(3)
+ environmentVariables.set(mapOf(
+ "clearPackageData" to "true"
+ ))
+ debugApk.set("app.apk")
+ variant.set("vanillaDebug")
+}
+Disabling a module
+enabled configuration in the module's fulladleModuleConfig block like so:fulladleModuleConfig {
+ enabled = false
+}
+fulladleModuleConfig {
+ enabled.set(false)
+}
+Overriding root-level config
+debugApk in its fladle or fulladleModuleConfig block.maxTestShards) can also be overridden in the fulladleModuleConfig block of whatever module gets picked as the root module.Troubleshooting
+./gradlew printYml./gradlew runFlank -PdumpShards
+
Quick Start
+
+
+
+
+
+
+
+
+ build.gradlebuildscript {
+ dependencies {
+ classpath "com.osacky.flank.gradle:fladle:0.20.0"
+ }
+}
+buildscript {
+ dependencies {
+ classpath("com.osacky.flank.gradle:fladle:0.20.0")
+ }
+}
+build.gradleapply plugin: "com.android.application"
+apply plugin: "com.osacky.fladle"
+plugins {
+ id ("com.android.application")
+ id ("com.osacky.fladle")
+}
+./gradlew :app:assembleDebug :app:assembleDebugAndroidTest
+./gradlew :app:assembleFreeDebug :app:assembleFreeDebugAndroidTest./gradlew runFlank
+
+
Recipes
+Smartly shard tests in 120 second shards across a maximum of 50 shards.
+fladle {
+ maxTestShards = 50
+ shardTime = 120
+ smartFlankGcsPath = "gs://fladle-results/smart-flank/JUnitReport.xml"
+}
+fladle {
+ maxTestShards.set(50)
+ shardTime.set(120)
+ smartFlankGcsPath.set("gs://fladle-results/smart-flank/JUnitReport.xml")
+}
+Run different tests on different devices with different Gradle tasks.
+./gradlew runFlankPerfTests will execute the performance tests against a MediumPhone.arm
+./gradlew runFlankRegresssionTests will execute the regressions tests against a SmallPhone.armfladle {
+ configs {
+ perfTests {
+ devices.set([
+ ["model" : "MediumPhone.arm", "version" : "28"],
+ ["model" : "MediumPhone.arm", "version" : "28"]
+ ])
+ testTargets.set([
+ "class com.sample.MyPerformanceTest"
+ ])
+ }
+ regressionTests {
+ devices.set([
+ [ "model" : "SmallPhone.arm", "version" : "28"]
+ ])
+ testTargets.set([
+ "class com.sample.MyRegressionTest"
+ ])
+ }
+ }
+}
+fladle {
+ configs {
+ create("perfTests") {
+ devices.set(listOf(
+ mapOf("model" to "MediumPhone.arm", "version" to "28" ),
+ mapOf("model" to "MediumPhone.arm", "version" to "28")
+ ))
+ testTargets.set(listOf(
+ "class com.sample.MyPerformanceTest"
+ ))
+ }
+ create("regressionTests") {
+ devices.set(listOf(
+ mapOf("model" to "SmallPhone.arm", "version" to "28" )
+ ))
+ testTargets.set(listOf(
+ "class com.sample.MyRegressionTest"
+ ))
+ }
+ }
+}
+Always use the latest version of Flank
+flankVersion = "23.+"
+flankVersion.set("23.+")
+
+
Releasing
+
+
+master
+git checkout master
+git pull
+git checkout -b release_0.20.1
+version in fladle-plugin/build.gradle.kts (remove -SNAPSHOT)
+version = "0.20.1"
+mkdocs.yml:
+extra:
+ fladle:
+ release: '0.20.1'
+ next_release: 'REPLACE_WITH_NEXT_VERSION_NUMBER'
+git diff
+git commit -am "Prepare 0.20.1 release"
+git tag v0.20.1
+git push origin v0.20.1
+
+
+
+
+
+
+
+
+
+
+git checkout master
+git pull
+git merge --no-ff release_0.20.1
+version in fladle-plugin/build.gradle.kts (increase version and add -SNAPSHOT)
+version = "REPLACE_WITH_NEXT_VERSION_NUMBER-SNAPSHOT"
+git commit -am "Prepare for next development iteration"
+git push
+
+
Results
+build/fladle/results/<matrix name> directly.JUnitReport.xmlHtmlErrorReport.html.CostReport.txt.~/.flank.
"},{"location":"authentication/#service-account-credentials","title":"Service account credentials","text":"./gradlew flankAuth./gradlew runFlank
Keys, click Add Key -> Create a new Key. Select JSON for the key type. This will download the json credentials.
"},{"location":"changelog/#0200","title":"0.20.0","text":"
"},{"location":"changelog/#0190","title":"0.19.0","text":"
"},{"location":"changelog/#0180","title":"0.18.0","text":"
"},{"location":"changelog/#0175","title":"0.17.5","text":"
"},{"location":"changelog/#0174","title":"0.17.4","text":"
"},{"location":"changelog/#0172","title":"0.17.2","text":"abi option to support selecting a particular debug APK in projects that use ABI splits to produce multiple APKs. PR
"},{"location":"changelog/#0170","title":"0.17.0","text":"
"},{"location":"changelog/#0163","title":"0.16.3","text":"
"},{"location":"changelog/#0162","title":"0.16.2","text":"main in JavaExec task.
"},{"location":"changelog/#0161","title":"0.16.1","text":"
debugApk through either the root fladle block or the module's own fulladleModuleConfig block.
"},{"location":"changelog/#0151","title":"0.15.1","text":"
"},{"location":"changelog/#0150","title":"0.15.0","text":"smartFlankGcsPath to samples Fixes #236
"},{"location":"changelog/#0141","title":"0.14.1","text":"async flag. PR.dependOnAssemble PR Thanks asadsalman
"},{"location":"changelog/#0140","title":"0.14.0","text":"
"},{"location":"changelog/#0131","title":"0.13.1","text":"testShards Fixes #204 PR#212 Thanks pawelpasterz
"},{"location":"changelog/#0130","title":"0.13.0","text":"
default-test-timedefault-class-test-timeadditional-apksuse-average-test-time-for-new-testsdisable-results-upload
"},{"location":"changelog/#0120","title":"0.12.0","text":"
"},{"location":"changelog/#0110","title":"0.11.0","text":"full-junit-result and legacy-junit-result. #170 Fixes #157. Thanks MatthewTPage
"},{"location":"changelog/#0102","title":"0.10.2","text":"additional-test-apks for modules that have tests. PR
timeoutMin has been renamed to testTimeout. PR Thanks pawelpasterz
"},{"location":"changelog/#0100","title":"0.10.0","text":"
"},{"location":"changelog/#092","title":"0.9.2","text":"
"},{"location":"changelog/#081","title":"0.8.1","text":"
"},{"location":"changelog/#080","title":"0.8.0","text":"additionalTestApks. PR Thanks japplin.resultsDir. PR
"},{"location":"changelog/#070","title":"0.7.0","text":"List<Map<String, String>> instead of a List<Device>. See the [#README.md] for an example. PR Thanks zlippard.keep-file-path. PR Thanks tahirhajizada.
"},{"location":"changelog/#067","title":"0.6.7","text":"
"},{"location":"changelog/#066","title":"0.6.6","text":"
"},{"location":"changelog/#065","title":"0.6.5","text":"
"},{"location":"changelog/#064","title":"0.6.4","text":"
"},{"location":"changelog/#063","title":"0.6.3","text":"
"},{"location":"changelog/#062","title":"0.6.2","text":"
"},{"location":"changelog/#061","title":"0.6.1","text":"
"},{"location":"changelog/#060","title":"0.6.0","text":"
"},{"location":"changelog/#052","title":"0.5.2","text":"
"},{"location":"changelog/#051","title":"0.5.1","text":"
"},{"location":"changelog/#050","title":"0.5.0","text":"
"},{"location":"changelog/#041","title":"0.4.1","text":"
"},{"location":"changelog/#040","title":"0.4.0","text":"
"},{"location":"changelog/#breaking-changes","title":"Breaking Changes:","text":"clearPackageData = true will now need to use: environmentVariables = [\n \"clearPackageData\": \"true\"\n]\n
The following configuration options must be set inside the fladle block. See the sample configuration below. There is also a groovy sample and a kotlin sample.
"},{"location":"configuration/#serviceaccountcredentials","title":"serviceAccountCredentials","text":"User Authentication is also possible. See Authentication.
GroovyserviceAccountCredentials = project.layout.projectDirectory.file(\"flank-gradle-5cf02dc90531.json\")\n Kotlin serviceAccountCredentials.set(\"project.layout.projectDirectory.file(\"flank-gradle-5cf02dc90531.json\")\n Optionally, the serviceAccountCredentials can be set with environment variables but then the projectId parameter must be set.
See Authentication for more info.
"},{"location":"configuration/#variant","title":"variant","text":"Note
variant must be set if using buildFlavors in order to automatically configure the debugApk and testApk.
Set the variant to automatically configure for testing. A build variant is a combination of buildFlavor and buildType. This must also be set when testing against a non-default variant. For example: 'debug' or 'freeDebug'. Put this inside your Fladle block.
Groovyvariant = \"freeDebug\"\n Kotlin variant.set(\"freeDebug\")\n"},{"location":"configuration/#sample-configuration","title":"Sample Configuration","text":"fladle {\n// Required parameters\nserviceAccountCredentials = project.layout.projectDirectory.file(\"flank-gradle-5cf02dc90531.json\")\nvariant = \"freeDebug\"\n\n// Optional parameters\nuseOrchestrator = false\nenvironmentVariables = [\n\"clearPackageData\": \"true\"\n]\ndirectoriesToPull = [\n\"/sdcard/screenshots\"\n]\nfilesToDownload = [\n\".*/screenshots/.*\"\n]\ntestTargets = [\n\"class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView\"\n]\ntestTimeout = \"15m\"\nrecordVideo = false\nperformanceMetrics = false\ndevices = [\n[ \"model\": \"SmallPhone.arm\", \"version\": \"28\" ],\n[ \"model\": \"MediumPhone.arm\", \"version\": \"33\" ]\n]\nprojectId(\"flank-gradle\")\nflankVersion = \"23.10.1\"\ndebugApk = \"$buildDir/outputs/apk/debug/sample-debug.apk\"\ninstrumentationApk = \"$buildDir/outputs/apk/androidTest/debug/sample-debug-androidTest.apk\"\nadditionalTestApks = [\n\"- app: $buildDir/outputs/apk/debug/sample-debug.apk\",\n\" test: $buildDir/outputs/apk/androidTest/debug/sample2-debug-androidTest.apk\",\n\"- test: ${rootProject.buildDir}/database/module/outputs/apk/database-module-androidTest.apk\"\n]\nautoGoogleLogin = true\nmaxTestShards = 8 //or numUniformShards=5 cannot use both.\nshardTime = 120\nsmartFlankGcsPath = \"gs://tmp_flank/flank/test_app_android.xml\"\nconfigs {\noranges {\nuseOrchestrator.set(false)\ntestTargets.set(project.provider { [\n\"class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#runAndFail\"\n] })\n}\n}\nresultsBucket(\"my-results-bucket-name\")\nkeepFilePath = true\nrunTimout = \"45m\"\nignoreFailedTests = false\ndisableSharding = false\nsmartFlankDisableUpload = false\ntestRunnerClass = \"com.foo.TestRunner\"\nlocalResultsDir = \"flank\"\nclientDetails = [\n\"key1\": \"value1\",\n\"key2\": \"value2\"\n]\ntestTargetsAlwaysRun = [\n\"com.example.TestSuite#test1\",\n\"com.example.TestSuite#test2\"\n]\notherFiles = [\n\"/sdcard/dir1/file1.txt\": \"/my/example/path/file1.txt\",\n\"/sdcard/dir2/file2.txt\": \"/my/example/path/file2.txt\"\n]\nnetworkProfile = \"LTE\"\nroboDirectives = [\n[\"click\", \"button1\", \"\"],\n[\"ignore\", \"button2\"],\n[\"text\", \"field1\", \"my text\"],\n]\noutputStyle = 'multi'\nlegacyJunitResult = false\nfullJunitResult = false\nadditionalApks = [\n\"gs://path/to/app1.apk\",\n\"localPath/to/app2.apk\"\n]\ndefaultTestTime = 5.3\ndefaultClassTestTime = 180.5\nuseAverageTestTimeForNewTests = true\ndisableResultsUpload = true\ngrantPermissions = \"none\"\ntype = \"game-loop\"\nscenarioLabels = [\n\"label1\",\n\"label2\" ]\nscenarioNumbers = [ 1, 23, 52 ]\nobbFiles = [\n\"local/file/path/test1.obb\",\n\"local/file/path/test2.obb\"\n]\nobbNames = [\n\"patch.0300110.com.example.android.obb\",\n\"patch.0300111.com.example.android.obb\"\n]\ntestTargetsForShard = [\n\"package com.package1.for.shard1, com.package2.for.shard1\",\n\"class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2\",\n\"class com.foo.ClassForShard3; package com.package.for.shard3\"\n]\nfailFast = true\ndependOnAssemble = true\n}\n"},{"location":"configuration/#useorchestrator","title":"useOrchestrator","text":"Whether or not we should use the android test orchestrator to run this tests. Set this to true when the build.gradle file includes testOptions.execution 'ANDROID_TEST_ORCHESTRATOR'
useOrchestrator = true\n Kotlin useOrchestrator.set(true)\n"},{"location":"configuration/#testtargets","title":"testTargets","text":"Set multiple testTargets to be run by Flank. These are used to whitelist or blacklist test classes, test cases and test annotations. See Google Cloud Firebase docs for more information.
GroovytestTargets = [\n\"class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView\"\n]\n Kotlin testTargets.set(listOf(\n\"class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView\"\n))\n"},{"location":"configuration/#devices","title":"devices","text":"A list of devices to run the tests against. When list is empty, a default device will be used. Each device entry is a map. The valid keys in the map are model, version, orientation, and locale. When a key is not set or is null, a default value will be used.
devices = [\n[ \"model\": \"MediumPhone.arm\", \"version\": \"26\" ],\n[ \"model\": \"MediumPhone.arm\", \"version\": \"33\" ]\n]\n Kotlin devices.set(listOf(\nmapOf(\"model\" to \"MediumPhone.arm\", \"version\" to \"26\" ),\nmapOf(\"model\" to \"MediumPhone.arm\", \"version\" to \"33\" )\n))\n"},{"location":"configuration/#projectid","title":"projectId","text":"The projectId is a unique identifier which can be found in the project's URL: https://console.firebase.google.com/project/<projectId> This is automatically discovered based on the service credential by default.
projectId = \"flank-gradle\"\n Kotlin projectId.set(\"flank-gradle\")\n"},{"location":"configuration/#flankversion","title":"flankVersion","text":"Need a different Flank version? Specify it with flankVersion.
To use a snapshot:
GroovyflankVersion = \"flank_snapshot\"`\n Kotlin flankVersion.set(\"flank_snapshot\")\n Need more than 50 shards? Use Flank 8.1.0.
To use a different version:
GroovyflankVersion = \"23.10.1\"\n Kotlin flankVersion.set(\"23.10.1\")\n"},{"location":"configuration/#flankcoordinates","title":"flankCoordinates","text":"Specify custom flank maven coordinates.
GroovyflankCoordinates = \"com.github.flank:flank\"\n Kotlin flankCoordinates.set(\"com.github.flank:flank\")\n"},{"location":"configuration/#debugapk","title":"debugApk","text":"This is a string representing the path to the app's debug apk. Supports wildcard characters. Optional, prefer to set variant.
GroovydebugApk = project.provider { \"${buildDir.toString()}/outputs/apk/debug/*.apk\" }\n Kotlin debugApk.set(project.provider { \"${buildDir.toString()}/outputs/apk/debug/*.apk\" })\n"},{"location":"configuration/#instrumentationapk","title":"instrumentationApk","text":"This is a string representing the path to the app's instrumentaiton apk. Supports wildcard characters. Optional, prefer to set variant. InstrumenationApk should not be set when using roboScript.
GroovyinstrumentationApk = project.provider { \"${buildDir.toString()}/outputs/apk/androidTest/debug/*.apk\" }\n Kotlin instrumentationApk.set(project.provider { \"${buildDir.toString()}/outputs/apk/androidTest/debug/*.apk\" })\n"},{"location":"configuration/#additionaltestapks","title":"additionalTestApks","text":"Paths to additional test configurations. Order matters. A test apk is run with the nearest previous listed app apk. For library modules, add them to the list with a - test: in front. For test apks which belong to an application module, add them with test:. It is not required to list an app apk here. If there is no app apk listed in additionalTestApks, the test apks are run against the debugApk.
additionalTestApks.value(project.provider { [\n\"- app: ../main/app/build/output/apk/debug/app.apk\",\n\" test: ../main/app/build/output/apk/androidTest/debug/app-test.apk\",\n\"- app: ../sample/app/build/output/apk/debug/sample-app.apk\",\n\" test: ../sample/app/build/output/apk/androidTest/debug/sample-app-test.apk\",\n\"- test: ../feature/room/build/output/apk/androidTest/debug/feature-room-test.apk\",\n\"- test: ../library/databases/build/output/apk/androidTest/debug/sample-databases-test.apk\"\n]})\n Kotlin additionalTestApks.value(project.provider { listOf(\n\"- app: ../main/app/build/output/apk/debug/app.apk\",\n\" test: ../main/app/build/output/apk/androidTest/debug/app-test.apk\",\n\"- app: ../sample/app/build/output/apk/debug/sample-app.apk\",\n\" test: ../sample/app/build/output/apk/androidTest/debug/sample-app-test.apk\",\n\"- test: ../feature/room/build/output/apk/androidTest/debug/feature-room-test.apk\",\n\"- test: ../library/databases/build/output/apk/androidTest/debug/sample-databases-test.apk\"\n)})\n"},{"location":"configuration/#autogooglelogin","title":"autoGoogleLogin","text":"Whether to automatically log in using a preconfigured google account. More Info
GroovyautoGoogleLogin = false\n Kotlin autoGoogleLogin.set(false)\n"},{"location":"configuration/#environmentvariables","title":"environmentVariables","text":"Environment variables are mirrored as extra options to the am instrument -e KEY1 VALUE1 \u2026 command and passed to your test runner (typically AndroidJUnitRunner). Examples
environmentVariables = [\n// Whether or not to remove all shared state from your device's CPU and memory after each test. [More info](https://developer.android.com/training/testing/junit-runner)\n\"clearPackageData\": \"true\"\n]\n Kotlin environmentVariables = mapOf(\n// Whether or not to remove all shared state from your device's CPU and memory after each test. [More info](https://developer.android.com/training/testing/junit-runner)\n\"clearPackageData\" to \"true\"\n)\n"},{"location":"configuration/#testshards","title":"testShards","text":"Deprecated
Use maxTestShards instead.
The maximum number of shards. Fladle will throw an error when used together with maxTestShards or numUniformShards.
GroovytestShards = 5\n Kotlin testShards.set(5)\n"},{"location":"configuration/#maxtestshards","title":"maxTestShards","text":"The maximum number of shards. Fladle will throw an error when used together with testShards or numUniformShards.
GroovymaxTestShards = 8\n Kotlin maxTestShards.set(8)\n"},{"location":"configuration/#shardtime","title":"shardTime","text":"The amount of time tests within a shard should take. When set to > 0, the shard count is dynamically set based on time up to the maximum limit defined by maxTestShards 2 minutes (120) is recommended. default: -1 (unlimited)
GroovyshardTime = 120\n Kotlin shardTime.set(120)\n"},{"location":"configuration/#repeattests","title":"repeatTests","text":"The number of times to repeat each test. Uses Flank's default value when not specified.
GroovyrepeatTests = 1\n Kotlin repeatTests.set(1)\n"},{"location":"configuration/#configs","title":"configs","text":"Give a name to a custom flank task and configure its options. The name is appended to the end of the flank task. For example runFlank becomes runFlank<name>.
configs {\nrepeatOneHundred {\n// DSL sugar for container elements is missing (= syntax): https://github.com/gradle/gradle/issues/9987\nrepeatTests.set(100)\n}\n}\n Kotlin configs {\ncreate(\"repeatOneHundred\") {\nrepeatTests.set(100)\n}\n}\n In the above example, the configuration is inherited from the outer fladle config but with the repeatTests property set to 100. Running runFlankRepeateOneHundred will execute this custom configuration.
The cloud storage location for historical test runs. This must be set in order to use smart sharding. The amount of shards used is set by maxTestShards. The bucket (such as flank_data in the example) must already exist in order for it to be used.
smartFlankGcsPath = 'gs://flank_data/results/JUnitReport.xml'\n Kotlin smartFlankGcsPath.set(\"gs://flank_data/results/JUnitReport.xml\")\n"},{"location":"configuration/#resultshistoryname","title":"resultsHistoryName","text":"The history name for your test results (an arbitrary string label; default: the application's label from the APK manifest). All tests which use the same history name will have their results grouped together in the Firebase console in a time-ordered test history list.
GroovyresultsHistoryName = 'android-history'\n Kotlin resultsHistoryName.set(\"android-history\")\n"},{"location":"configuration/#flakytestattempts","title":"flakyTestAttempts","text":"The number of times to retry failed tests. Default is 0. Max is 10. Setting the value to 1 will mean that test are retried once. If the test fails then succeeds after the retry the run will be marked as \"successful\". The matrix with a flaky test will be marked as flaky.
GroovyflakyTestAttempts = 0\n Kotlin flakyTestAttempts.set(0)\n"},{"location":"configuration/#abi","title":"abi","text":"The ABI split of the application that should be tested (e.g. \"x86\"). Only required if the application under test uses ABI splits and the debug APK is selected automatically (via variant) instead of manually (via debugApk).
If the application uses ABI splits, and this property isn't specified, an arbitrary ABI split will be selected.
Groovyabi = \"arm64-v8a\"\n Kotlin abi.set(\"arm64-v8a\")\n"},{"location":"configuration/#directoriestopull","title":"directoriesToPull","text":"A list of paths that will be copied from the device's storage to the designated results bucket after the test is complete. These must be absolute paths under /sdcard or /data/local/tmp. Path names are restricted to the characters a-zA-Z0-9_-./+. The paths /sdcard and /data will be made available and treated as implicit path substitutions. E.g. if /sdcard on a particular device does not map to external storage, the system will replace it with the external storage path prefix for that device.
directoriesToPull = [\n'/sdcard/tempDir1', '/data/local/tmp/tempDir2'\n]\n Kotlin directoriesToPull.set(listOf(\n\"/sdcard/tempDir1\", \"/data/local/tmp/tempDir2\"\n))\n"},{"location":"configuration/#filestodownload","title":"filesToDownload","text":"List of regex that is matched against bucket paths (for example: 2019-01-09_00:13:06.106000_YCKl/shard_0/SmallPhone.arm-28-en-portrait/bugreport.txt) for files to be downloaded after a flank run. The results are downloaded to the APP_MODULE/build/fladle/RESULTS directory where RESULTS can be set by localResultsDir var otherwise defaulting to results/.
filesToDownload = [\n'.*/sdcard/tempDir1/.*', '.*/data/local/tmp/tempDir2/.*'\n]\n Kotlin filesToDownload.set(listOf(\n\".*/sdcard/tempDir1/.*\", \".*/data/local/tmp/tempDir2/.*\"\n))\n"},{"location":"configuration/#testtimeout","title":"testTimeout","text":"The max time test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 45m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. Examples: * 1h -> 1 hour * 5m -> 5 minutes * 200s -> 200 seconds * 100 -> 100 seconds
GroovytestTimeout = \"1h\"\n Kotlin testTimeout.set(\"1h\")\n"},{"location":"configuration/#recordvideo","title":"recordVideo","text":"Enable video recording during the test. Enabled by default.
GroovyrecordVideo = true\n Kotlin recordVideo.set(true)\n"},{"location":"configuration/#performancemetrics","title":"performanceMetrics","text":"Monitor and record performance metrics: CPU, memory, network usage, and FPS (game-loop only). Enabled by default.
GroovyperformanceMetrics = true\n Kotlin performanceMetrics.set(true)\n"},{"location":"configuration/#resultsbucket","title":"resultsBucket","text":"The name of a Google Cloud Storage bucket where raw test results will be stored.
GroovyresultsBucket = \"my-gcs-bucket-name\"\n Kotlin resultsBucket.set(\"my-gcs-bucket-name\")\n"},{"location":"configuration/#keepfilepath","title":"keepFilePath","text":"Keeps the full path of downloaded files from a Google Cloud Storage bucket. Required when file names are not unique. Disabled by default.
GroovykeepFilePath = false\n Kotlin keepFilePath.set(false)\n"},{"location":"configuration/#resultsdir","title":"resultsDir","text":"The name of a unique Google Cloud Storage object within the results bucket where raw test results will be stored. The default is a timestamp with a random suffix.
GroovyresultsDir = \"result-dir-${getTimeStamp()}\"\n Kotlin resultsDir.set(\"result-dir-${getTimeStamp()}\")\n"},{"location":"configuration/#disablesharding","title":"disableSharding","text":"Disables sharding. All tests will run on the same device. Useful for parameterized tests which do not support sharding. (default: false)
GroovydisableSharding = false\n Kotlin disableSharding.set(false)\n"},{"location":"configuration/#smartflankdisableupload","title":"smartFlankDisableUpload","text":"Disables smart flank JUnit XML uploading. Useful for preventing timing data from being updated. (default: false) What is Smart Flank?
GroovysmartFlankDisableUpload = false\n Kotlin smartFlankDisableUpload.set(false)\n"},{"location":"configuration/#testrunnerclass","title":"testRunnerClass","text":"The fully-qualified Java class name of the instrumentation test runner (default: the test manifest is parsed to determine the class name).
GroovytestRunnerClass = \"com.example.MyCustomTestRunner\"\n Kotlin testRunnerClass.set(\"com.example.MyCustomTestRunner\")\n"},{"location":"configuration/#localresultsdir","title":"localResultsDir","text":"The local directory to store the test results. Folder is DELETED before each run to ensure only artifacts from the new run are saved. This directory is relative to the working directory of Flank which is by default build/fladle or build/fladle/<flavorname>.
localResultsDir = \"my-results-dir\"\n Kotlin localResultsDir.set(\"my-results-dir\")\n"},{"location":"configuration/#testtargetsalwaysrun","title":"testTargetsAlwaysRun","text":"Always run - these tests are inserted at the beginning of every shard. Useful if you need to grant permissions or login before other tests run
GroovytestTargetsAlwaysRun = [\n'class com.example.MyTestClass'\n]\n Kotlin testTargetsAlwaysRun.set(listOf(\n\"class com.example.MyTestClass\"\n))\n The flags below are only available with Flank 20.05.0 or higher.
"},{"location":"configuration/#runtimeout","title":"runTimeout","text":"The max time this test run can execute before it is cancelled. s (seconds), m (minutes), h (hours) suffixes are acceptable, mixes like 1h45m are currently not supported (default: unlimited). Examples:
runTimeout = \"15m\"\n Kotlin runTimeout.set(\"15m\")\n"},{"location":"configuration/#ignorefailedtests","title":"ignoreFailedTests","text":"Always return successful task completion even when there are failed tests. Useful when parsing JUnit XML to determine failure. (default: false)
GroovyignoreFailedTests = false\n Kotlin ignoreFailedTests.set(false)\n"},{"location":"configuration/#numuniformshards","title":"numUniformShards","text":"Specifies the number of shards into which you want to evenly distribute test cases. The shards are run in parallel on separate devices. For example, if your test execution contains 20 test cases and you specify four shards, each shard executes five test cases. The number of shards should be less than the total number of test cases. The number of shards specified must be >= 1 and <= 50. This option cannot be used along maxTestShards and is not compatible with smart sharding (Smart Flank). If you want to take benefits of smart sharding use maxTestShards instead. (default: null)
GroovynumUniformShards = 50\n Kotlin numUniformShards.set(50)\n"},{"location":"configuration/#clientdetails","title":"clientDetails","text":"A key-value map of additional details to attach to the test matrix.(clientDetails in Google Cloud Docs) Arbitrary key-value pairs may be attached to a test matrix to provide additional context about the tests being run. When consuming the test results, such as in Cloud Functions or a CI system, these details can add additional context such as a link to the corresponding pull request. (Access Client Details). These can be used to provide additional context about the environment where the tests are being run.
GroovyclientDetails = [\n\"test-type\": \"PR\",\n\"build-number\": \"132\"\n]\n Kotlin clientDetails.set(mapOf(\n\"test-type\" to \"PR\",\n\"build-number\" to \"132\"\n))\n"},{"location":"configuration/#otherfiles","title":"otherFiles","text":"A list of device-path: file-path pairs that indicate the device paths to push files to the device before starting tests, and the paths of files to push. Device paths must be under absolute, whitelisted paths (${EXTERNAL_STORAGE}, or ${ANDROID_DATA}/local/tmp). Source file paths may be in the local filesystem or in Google Cloud Storage (gs://\u2026).
GroovyotherFiles = [\n\"/sdcard/dir1/file1.txt\": \"local/file.txt\",\n\"/sdcard/dir2/file2.jpg\": \"gs://bucket/file.jpg\",\n]\n Kotlin otherFiles.set(mapOf(\n\"/sdcard/dir1/file1.txt\" to \"local/file.txt\",\n\"/sdcard/dir2/file2.jpg\" to \"gs://bucket/file.jpg\",\n))\n"},{"location":"configuration/#networkprofile","title":"networkProfile","text":"The name of the network traffic profile, for example LTE, HSPA, etc, which consists of a set of parameters to emulate network conditions when running the test (default: no network shaping; see available profiles listed by the flank test network-profiles list command). This feature only works on physical devices.
networkProfile = \"LTE\"\n Kotlin networkProfile.set(\"LTE\")\n"},{"location":"configuration/#roboscript","title":"roboScript","text":"The path to a Robo Script JSON file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. You can guide the Robo test to perform specific actions by recording a Robo Script in Android Studio and then specifying this argument. Learn more at DOCS.
GroovyroboScript = \"my-robo-script.json\"\n Kotlin roboScript.set(\"my-robo-script.json\")\n"},{"location":"configuration/#robodirectives","title":"roboDirectives","text":"List of robo_directives that you can use to customize the behavior of Robo test. The type specifies the action type of the directive, which may take on values click, text or ignore. Each directive is list of String = [type, key, value]. Each key should be the Android resource name of a target UI element and each value should be the text input for that element. Values are only permitted for text type elements, so no value should be specified for click and ignore type elements.
GroovyroboDirectives = [\n[\"test, \"input_resource_name\", \"message\"],\n [\"click, \"button_resource_name\", \"\"],\n]\n Kotlin roboDirectives.set(listOf(\nlistOf(\"test\", \"input_resource_name\", \"message\"),\nlistOf(\"click\", \"button_resource_name\", \"\"),\n))\n"},{"location":"configuration/#outputstyle","title":"outputStyle","text":"Output style of execution status. May be one of [verbose, multi, single]. For runs with only one test execution the default value is 'verbose', in other cases 'multi' is used as the default. The output style 'multi' is not displayed correctly on consoles which don't support ANSI codes, to avoid corrupted output use single or verbose.
multi displays separated status for each shard execution in separated line, lines are updated over time. Similar to gradle output when running multiple tasks in parallel. Requires ANSI codes support.
single displays shortened status of all executions in single line. Similar to gcloud output when running with sharding. Should work on any console.
Default value is single.
GroovyoutputSyle = \"single\"\n Kotlin outputStyle.set(\"single\")\n"},{"location":"configuration/#legacyjunitresult","title":"legacyJunitResult","text":"Flank provides two ways for parsing junit xml results. New way uses google api instead of merging xml files, but can generate slightly different output format. This flag allows fallback for legacy xml junit results parsing
GroovylegacyJunitResult = false\n Kotlin legacyJunitResult.set(false)\n"},{"location":"configuration/#fulljunitresult","title":"fullJunitResult","text":"Enables creating an additional local junit result on local storage with failure nodes on passed flaky tests.
GroovyfullJunitResult = false\n Kotlin fullJunitResult.set(false)\n"},{"location":"configuration/#sanityrobo","title":"sanityRobo","text":"Runs a sanityRobo test. instrumentationApk, roboDirectives, roboScript and additionalTestApks must be blank or empty.
sanityRobo = true\n Kotlin sanityRobo.set(true)\n"},{"location":"configuration/#defaulttesttime","title":"defaultTestTime","text":"Set default test time expressed in seconds, used for calculating shards. (default: 120.0s)
GroovydefaultTestTime = 1.2\n Kotlin defaultTestTime.set(1.2)\n"},{"location":"configuration/#defaultclasstesttime","title":"defaultClassTestTime","text":"Set default parameterized class test time expressed in seconds, used for calculating shards. (default: 2x defaultTestTime => 240s)
GroovydefaultClassTestTime = 245.5\n Kotlin defaultClassTestTime.set(245,5)\n"},{"location":"configuration/#additionalapks","title":"additionalApks","text":"A list of up to 100 additional APKs to install, in addition to those being directly tested. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
GroovyadditionalApks = [\n\"gs://path/to/app1.apk\",\n\"localPath/to/app2.apk\"\n]\n Kotlin additionalApks.set(\nproject.provider {\nlistOf(\"gs://path/to/app1.apk\", \"localPath/to/app2.apk\")\n}\n)\n"},{"location":"configuration/#useaveragetesttimefornewtests","title":"useAverageTestTimeForNewTests","text":"Enable using average time from previous tests duration when using SmartShard and tests did not run before. (default: false)
GroovyuseAverageTestTimeForNewTests = true\n Kotlin useAverageTestTimeForNewTests.set(true)\n"},{"location":"configuration/#disableresultsupload","title":"disableResultsUpload","text":"Disable flank results upload on gcloud storage. (default: false)
GroovydisableResultsUpload = true\n Kotlin disableResultsUpload.set(true)\n"},{"location":"configuration/#testtargetsforshard","title":"testTargetsForShard","text":"Specifies a group of packages, classes, and/or test cases to run in each shard (a group of test cases). The shards are run in parallel on separate devices. You can use this option up to 50 times to specify multiple shards when one or more physical devices are selected, or up to 500 times when no physical devices are selected. Note: If you include the flags environmentVariables or testTargets when running testTargetsForShard, the flags are applied to all the shards you create. You can also specify multiple packages, classes, or test cases in the same shard by separating each item with a comma. To specify both package and class in the same shard, separate package and class with semi-colons.
testTargetsForShard = [\n \"package com.package1.for.shard1, com.package2.for.shard1\",\n \"class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2\",\n \"class com.foo.ClassForShard3; package com.package.for.shard3\"\n ]\n Kotlin testTargetsForShard.set(\n project.provider {\n listOf(\n \"package com.package1.for.shard1, com.package2.for.shard1\",\n \"class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2\",\n \"class com.foo.ClassForShard3; package com.package.for.shard3\"\n )\n }\n)\n"},{"location":"configuration/#grantpermissions","title":"grantPermissions","text":"Whether to grant runtime permissions on the device before the test begins. By default, all permissions are granted. PERMISSIONS must be one of: all, none
GroovygrantPermissions = \"none\"\n Kotlin grantPermissions.set(\"none\")\n"},{"location":"configuration/#type","title":"type","text":"The type of test to run. TYPE must be one of: instrumentation, robo, game-loop. Use if you want to be sure there is only one type of tests being run (flank enables to run mixed types of test in one run).
Groovytype = \"game-loop\"\n Kotlin type.set(\"game-loop\")\n"},{"location":"configuration/#scenariolabels","title":"scenarioLabels","text":"A list of game-loop scenario labels (default: None). Each game-loop scenario may be labeled in the APK manifest file with one or more arbitrary strings, creating logical groupings (e.g. GPU_COMPATIBILITY_TESTS). If --scenario-numbers and --scenario-labels are specified together, Firebase Test Lab will first execute each scenario from --scenario-numbers. It will then expand each given scenario label into a list of scenario numbers marked with that label, and execute those scenarios.
scenarioLabels = [\n\"label1\",\n\"label2\" ]\n Kotlin scenarioLabels.set(\nproject.provider {\nlistOf(\"label1\", \"label2\")\n}\n)\n"},{"location":"configuration/#scenarionumbers","title":"scenarioNumbers","text":"A list of game-loop scenario numbers which will be run as part of the test (default: all scenarios). A maximum of 1024 scenarios may be specified in one test matrix, but the maximum number may also be limited by the overall test --timeout setting.
scenarioNumbers = [ 1, 23, 52 ]\n Kotlin scenarioNumbers.set(\nproject.provider {\nlistOf(1, 23, 52)\n}\n)\n"},{"location":"configuration/#obbfiles","title":"obbFiles","text":"A list of one or two Android OBB file names which will be copied to each test device before the tests will run (default: None). Each OBB file name must conform to the format as specified by Android (e.g. [main|patch].0300110.com.example.android.obb) and will be installed into <shared-storage>/Android/obb/<package-name>/ on the test device.
obbFiles = [\n\"local/file/path/test1.obb\",\n\"local/file/path/test2.obb\"\n]\n Kotlin obbFiles.set(\nproject.provider {\nlistOf(\n\"local/file/path/test1.obb\",\n\"local/file/path/test2.obb\"\n)\n}\n)\n"},{"location":"configuration/#obbnames","title":"obbNames","text":"A list of OBB required filenames. OBB file name must conform to the format as specified by Android e.g. [main|patch].0300110.com.example.android.obb which will be installed into <shared-storage>/Android/obb/<package-name>/ on the device.
obbNames = [\n\"patch.0300110.com.example.android.obb\",\n\"patch.0300111.com.example.android.obb\"\n]\n Kotlin obbNames.set(\nproject.provider {\nlistOf(\n\"patch.0300110.com.example.android.obb\",\n\"patch.0300111.com.example.android.obb\"\n)\n}\n)\n"},{"location":"configuration/#failfast","title":"failFast","text":"If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected. Normally, 2 or more attempts are made if a potential infrastructure issue is detected. This feature is for latency sensitive workloads. The incidence of execution failures may be significantly greater for fail-fast matrices and support is more limited because of that expectation.
GroovyfailFast = true\n Kotlin failFast.set(true)\n"},{"location":"configuration/#additionalflankoptions","title":"additionalFlankOptions","text":"Appending additional option to flank root yaml. This option is useful when you would like to test options before official fladle support is added. Multiple options are supported.
Single option
GroovyadditionalFlankOptions = \"new-property: true\"\n Kotlin additionalFlankOptions.set(\"new-property: true\")\n Multiple options
GroovyadditionalFlankOptions = \"\"\"\n new-property: true\n other-new-property: force\n\"\"\".stripIndent()\n Kotlin additionalFlankOptions.set(\"\"\"\n new-property: true\n other-new-property: force\n\"\"\".trimIndent())\n"},{"location":"configuration/#additionalgcloudoptions","title":"additionalGcloudOptions","text":"Allow appending additional config to gcloud root yaml. This option is useful when you would like to test option before it is available on Fladle. Supports both single and multiple properties.
Single option
GroovyadditionalGcloudOptions = \"new-property: true\"\n Kotlin additionalGcloudOptions.set(\"new-property: true\")\n Multiple options
GroovyadditionalGcloudOptions = \"\"\"\n new-property: true\n other-new-property: force\n \"\"\".stripIndent()\n Kotlin additionalGcloudOptions.set(\"\"\"\n new-property: true\n other-new-property: force\n\"\"\".trimIndent())\n"},{"location":"configuration/#dependonassemble","title":"dependOnAssemble","text":"Enable to automatically build the app and test APKs before runFlank executes. (default: false)
dependOnAssemble = true\n Kotlin dependOnAssemble.set(true)\n"},{"location":"configuration/#async","title":"async","text":"Enable to return immediately after invoking tests, without waiting for results. (default: false)
async = true\n Kotlin async.set(true)\n"},{"location":"faq/","title":"FAQ","text":""},{"location":"faq/#error-apk-file-not-found","title":"Error APK file not found","text":"The app APK and the instrumentation APK are expected to have already been generated before calling runFlank. To generate APKs, run assembleDebug and assembleDebugAndroidTest before running runFlank.
You can also have Fladle build them for you by using the dependOnAssemble property.
If you receive an error like this, it is likely caused by invalid fladle extension confiuration. The syntax was changed in the 0.9.X releases in order to avoid touching files during the configuration phase.
No signature of method: flank_4vvjv7w3oopge32w1tl9cs6e4.fladle() is applicable for argument types: (flank_4vvjv7w3oopge32w1tl9cs6e4$_run_closure1) values: [flank_4vvjv7w3oopge32w1tl9cs6e4$_run_closure1@649a2315]\nPossible solutions: file(java.lang.Object), find(), findAll(), file(java.lang.Object, org.gradle.api.PathValidation), files([Ljava.lang.Object;), findAll(groovy.lang.Closure)\n If you receive a similar error, please check configuration for a sample configuration.
"},{"location":"faq/#debugging","title":"Debugging","text":"./gradlew runFlank -PdumpShards Will dump shards and exit the process without running the tests.
./gradlew printYml Will print out the current yaml configuration to be passed to Flank.
Still having trouble? Check the #flank channel in the Firebase Community Slack
"},{"location":"multi-module-testing/","title":"Multi-module testing","text":"Multi module testing can be done by manually specifying additionalTestApks or applying the Fulladle plugin to automacally gather all the additional test apks. See also this article for a full setup and instructions including integration with CI.
"},{"location":"multi-module-testing/#fulladle-plugin","title":"Fulladle Plugin","text":"Apply the Fulladle plugin at the root of the project.
Groovyplugins {\nid 'com.osacky.fulladle' version '0.20.0'\n}\n Kotlin plugins {\nid(\"com.osacky.fulladle\") version \"0.20.0\"\n}\n Configure the Fladle extension.
Groovyfladle {\nserviceAccountCredentials = project.layout.projectDirectory.file(\"flank-gradle-service-account.json\")\n}\n Kotlin fladle {\nserviceAccountCredentials.set(project.layout.projectDirectory.file(\"flank-gradle-service-account.json\"))\n}\n Warning
If using buildFlavors or testing against a non default variant, you will need to specify the variant you want to test in the fulladleModuleConfig block.
Run the tests. First assemble all your debug apks and test apks.
./gradlew assembleDebug assembleDebugAndroidTest\n Note
When using flavors, make sure to assemble your buildVariants as well.
./gradlew :app:assembleFreeDebug :app:assembleFreeDebugAndroidTest
Run Flank!
./gradlew runFlank\n Fulladle will pick Flank configurations from the fladle block in the root build.gradle file. You may want to override some of these configurations for certain modules, you can add the following block to any Android library module to override its configurations:
fulladleModuleConfig {\nclientDetails = [\n\"test-type\": \"PR\",\n\"build-number\": \"132\"\n]\nmaxTestShards = 3\nenvironmentVariables = [\n\"clearPackageData\": \"true\"\n]\ndebugApk = \"app.apk\"\nvariant = \"vanillaDebug\"\n}\n Kotlin fulladleModuleConfig {\nclientDetails.set(mapOf(\n\"test-type\" to \"PR\",\n\"build-number\" to \"132\",\n))\nmaxTestShards.set(3)\nenvironmentVariables.set(mapOf(\n\"clearPackageData\" to \"true\"\n))\ndebugApk.set(\"app.apk\")\nvariant.set(\"vanillaDebug\")\n}\n All of the above configurations are optional, Flank will default to the top-level configurations if you don't override anything here. For details about these configurations, refer to configuration docs.
"},{"location":"multi-module-testing/#disabling-a-module","title":"Disabling a module","text":"You may want to exclude a library module from testing when using Fulladle. You can do so by setting the enabled configuration in the module's fulladleModuleConfig block like so:
fulladleModuleConfig {\nenabled = false\n}\n Kotlin fulladleModuleConfig {\nenabled.set(false)\n}\n"},{"location":"multi-module-testing/#overriding-root-level-config","title":"Overriding root-level config","text":"Fulladle does not provide the ability to control which module ends up as the root-level module or as an additional module. Either one of app modules or library modules can become a root-level module. If a library module ends up as a root-level module, it needs to specify a debugApk in its fladle or fulladleModuleConfig block.
The root-level configuration (e.g. maxTestShards) can also be overridden in the fulladleModuleConfig block of whatever module gets picked as the root module.
Fulladle might still have some rough edges, but we'd love feedback. Please join us in the Firebase Community Slack with any feedback you may have. You can also file Fladle Github issues.
When filing a bug report, please include the Flank version number, the Fladle version number and the output of the following:
./gradlew printYml
./gradlew runFlank -PdumpShards
Using Fladle takes 3 steps:
Apply the Fladle plugin. Follow instructions here
Root build.gradle
buildscript {\ndependencies {\nclasspath \"com.osacky.flank.gradle:fladle:0.20.0\"\n}\n}\n Kotlin buildscript {\ndependencies {\nclasspath(\"com.osacky.flank.gradle:fladle:0.20.0\")\n}\n}\n Application module build.gradle
apply plugin: \"com.android.application\"\napply plugin: \"com.osacky.fladle\"\n Kotlin plugins {\nid (\"com.android.application\")\nid (\"com.osacky.fladle\")\n}\n Configure Authentication using these steps.
Warning
If using buildFlavors or testing against a non default variant, variant must also configured
Run your tests!
First assemble your debug apk and test apk.
./gradlew :app:assembleDebug :app:assembleDebugAndroidTest\n Note
When using flavors, make sure to assemble your buildVariants.
./gradlew :app:assembleFreeDebug :app:assembleFreeDebugAndroidTest
Run Flank!
./gradlew runFlank\n Here are some recipes to use to achieve various goals in flank. For additional recipes or suggestions, please file an issue on Github.
"},{"location":"recipes/#smartly-shard-tests-in-120-second-shards-across-a-maximum-of-50-shards","title":"Smartly shard tests in 120 second shards across a maximum of 50 shards.","text":"This recipe will keep track of test durations automatically on firebase test lab and try to split up test runs in to 120 second shards up to maximum of 50 shards.
Groovyfladle {\nmaxTestShards = 50\nshardTime = 120\nsmartFlankGcsPath = \"gs://fladle-results/smart-flank/JUnitReport.xml\"\n}\n Kotlin fladle {\nmaxTestShards.set(50)\nshardTime.set(120)\nsmartFlankGcsPath.set(\"gs://fladle-results/smart-flank/JUnitReport.xml\")\n}\n"},{"location":"recipes/#run-different-tests-on-different-devices-with-different-gradle-tasks","title":"Run different tests on different devices with different Gradle tasks.","text":"./gradlew runFlankPerfTests will execute the performance tests against a MediumPhone.arm ./gradlew runFlankRegresssionTests will execute the regressions tests against a SmallPhone.arm
fladle {\nconfigs {\nperfTests {\ndevices.set([\n[\"model\" : \"MediumPhone.arm\", \"version\" : \"28\"], [\"model\" : \"MediumPhone.arm\", \"version\" : \"28\"]\n])\ntestTargets.set([\n\"class com.sample.MyPerformanceTest\"\n])\n}\nregressionTests {\ndevices.set([\n[ \"model\" : \"SmallPhone.arm\", \"version\" : \"28\"]\n])\ntestTargets.set([\n\"class com.sample.MyRegressionTest\"\n])\n}\n}\n}\n Kotlin fladle {\nconfigs {\ncreate(\"perfTests\") {\ndevices.set(listOf(\nmapOf(\"model\" to \"MediumPhone.arm\", \"version\" to \"28\" ), mapOf(\"model\" to \"MediumPhone.arm\", \"version\" to \"28\")\n))\ntestTargets.set(listOf(\n\"class com.sample.MyPerformanceTest\"\n))\n}\ncreate(\"regressionTests\") {\ndevices.set(listOf(\nmapOf(\"model\" to \"SmallPhone.arm\", \"version\" to \"28\" )\n))\ntestTargets.set(listOf(\n\"class com.sample.MyRegressionTest\"\n))\n}\n}\n}\n"},{"location":"recipes/#always-use-the-latest-version-of-flank","title":"Always use the latest version of Flank","text":"Use Gradle's dynamic version syntax to declare a dynamic version.
Warning
Dynamic versions lead to non-reproducible builds since Gradle will check for new versions periodically based on how long versions are cached.
GroovyflankVersion = \"23.+\"\n Kotlin flankVersion.set(\"23.+\")\n"},{"location":"releasing/","title":"Releasing","text":"Create a local release branch from master
git checkout master\ngit pull\ngit checkout -b release_0.20.1\n Update version in fladle-plugin/build.gradle.kts (remove -SNAPSHOT)
version = \"0.20.1\"\n Update the current version and next version in mkdocs.yml:
extra:\n fladle:\n release: '0.20.1'\n next_release: 'REPLACE_WITH_NEXT_VERSION_NUMBER'\n Take one last look
git diff\n Commit all local changes
git commit -am \"Prepare 0.20.1 release\"\n Create a tag and push it
git tag v0.20.1\ngit push origin v0.20.1\n Pushing the tag automatically triggers the release workflow which:
Creates a GitHub Release with auto-generated notes
Release to Maven Central
Merge the release branch to master
git checkout master\ngit pull\ngit merge --no-ff release_0.20.1\n Update version in fladle-plugin/build.gradle.kts (increase version and add -SNAPSHOT)
version = \"REPLACE_WITH_NEXT_VERSION_NUMBER-SNAPSHOT\"\n Commit your changes
git commit -am \"Prepare for next development iteration\"\n Push your changes
git push\n By default, results are placed in the build/fladle/results/<matrix name> directly.
A merged junit report is available in JUnitReport.xml
On failures only, an html report with links to failed shards is also generated in the results directory with the name HtmlErrorReport.html.
A cost report is also available CostReport.txt.
To test the Fladle snapshot release you have two options:
"},{"location":"snapshots/#traditional","title":"Traditional","text":"Root build.gradle
buildscript {\nrepositories {\nmaven {\nurl \"https://oss.sonatype.org/content/repositories/snapshots/\"\n}\n}\ndependencies {\nclasspath \"com.osacky.flank.gradle:fladle:0.20.1-SNAPSHOT\"\n}\n}\n Project build.gradle
apply plugin: \"com.osacky.fladle\"\n"},{"location":"snapshots/#plugin-management","title":"Plugin Management","text":"settings.gradle
pluginManagement {\nrepositories {\nmaven {\nurl \"https://oss.sonatype.org/content/repositories/snapshots/\"\n}\ngradlePluginPortal()\n}\n}\n Android application build.gradle
plugins {\nid \"com.osacky.fladle\" version \"0.20.1-SNAPSHOT\"\n}\n"}]}
\ No newline at end of file
diff --git a/sitemap.xml b/sitemap.xml
new file mode 100644
index 00000000..0b3e1895
--- /dev/null
+++ b/sitemap.xml
@@ -0,0 +1,58 @@
+
+To test the Fladle snapshot release you have two options:
+Root build.gradle
+
buildscript {
+ repositories {
+ maven {
+ url "https://oss.sonatype.org/content/repositories/snapshots/"
+ }
+ }
+ dependencies {
+ classpath "com.osacky.flank.gradle:fladle:0.20.1-SNAPSHOT"
+ }
+}
+Project build.gradle
+
apply plugin: "com.osacky.fladle"
+settings.gradle
+
pluginManagement {
+ repositories {
+ maven {
+ url "https://oss.sonatype.org/content/repositories/snapshots/"
+ }
+ gradlePluginPortal()
+ }
+}
+Android application build.gradle
+
plugins {
+ id "com.osacky.fladle" version "0.20.1-SNAPSHOT"
+}
+