There are two authentication mechanisms for using Fladle.
+It is recommended to use user authentication on local development machines to avoid sharing credentials and a service account on CI.
+Credentials are stored in ~/.flank.
./gradlew flankAuth./gradlew runFlankCreate a service account. Service accounts aren't subject to spam checks or captcha prompts, which could + otherwise block your CI builds. Create a service account with an Editor role in the + Google Cloud Platform console.
+Enable required APIs. After logging in using the service account: In the Google Developers Console API Library + page, enable the Google Cloud Testing API and Cloud Tool Results API. To enable these APIs, type these API names into + the search box at the top of the console, and then click Enable API on the overview page for that API.
+After creating the account go to Keys, click Add Key -> Create a new Key. Select JSON for the key type. This will download the json credentials.
Configure the Fladle extension serviceAccountCredentials to point to the credentials.
+Above instructions are based on Google instruction for authenticating with CI.
+See also Flank's instructions for authenticating with a service account.
+ + + + + + +abi option to support selecting a particular debug APK in projects that use ABI splits to produce multiple APKs. PRmain in JavaExec task.Breaking API Change
+Previously, only app modules could be picked up as root-level modules in Fulladle. This has changed, now either app or library modules can be picked up (though we try app modules first). If a library module is picked up as a the root-level module, it must specifiy a debugApk through either the root fladle block or the module's own fulladleModuleConfig block.
smartFlankGcsPath to samples Fixes #236async flag. PR.dependOnAssemble PR Thanks asadsalmantestShards Fixes #204 PR#212 Thanks pawelpasterzdefault-test-timedefault-class-test-timeadditional-apksuse-average-test-time-for-new-testsdisable-results-uploadBreaking API Change
+Use lazy properties Fixes #92 PR Thanks pawelpasterz
+full-junit-result and legacy-junit-result. #170 Fixes #157. Thanks MatthewTPageadditional-test-apks for modules that have tests. PRBreaking API change
+Add time units for timeout. timeoutMin has been renamed to testTimeout. PR Thanks pawelpasterz
Breaking API Change
+additionalTestApks now uses ListProperty instead of the previous Map. This is to allow for lazy configuration of the provided files.
+Warning
+Minimum supported Gradle version is now 5.4.
+Breaking API change
+debugApk and instrumentationApk now use Lazy Property API to avoid resolving at configuration time.
+Breaking API Change
+serviceAccountCredentials now uses Lazy Property API. See Configuration for details on how to set it. PR
+Warning
+Minimum required Gradle Version is now 5.1.
+Warning
+Dropped support for Flank 7.X and lower.
+List<Map<String, String>> instead of a List<Device>. See the [#README.md] for an example. PR Thanks zlippard.keep-file-path. PR Thanks tahirhajizada.Previous users of clearPackageData = true will now need to use:
+
environmentVariables = [
+ "clearPackageData": "true"
+]
+The following configuration options must be set inside the fladle block. See the sample configuration below. There is also a groovy sample and a kotlin sample.
+User Authentication is also possible. See Authentication.
+serviceAccountCredentials = project.layout.projectDirectory.file("flank-gradle-5cf02dc90531.json")
+serviceAccountCredentials.set("project.layout.projectDirectory.file("flank-gradle-5cf02dc90531.json")
+Optionally, the serviceAccountCredentials can be set with environment variables but then the projectId parameter must be set.
+See Authentication for more info.
+Note
+variant must be set if using buildFlavors in order to automatically configure the debugApk and testApk.
Set the variant to automatically configure for testing. A build variant is a combination of buildFlavor and buildType. +This must also be set when testing against a non-default variant. +For example: 'debug' or 'freeDebug'. +Put this inside your Fladle block.
+variant = "freeDebug"
+variant.set("freeDebug")
+fladle {
+ // Required parameters
+ serviceAccountCredentials = project.layout.projectDirectory.file("flank-gradle-5cf02dc90531.json")
+ variant = "freeDebug"
+
+ // Optional parameters
+ useOrchestrator = false
+ environmentVariables = [
+ "clearPackageData": "true"
+ ]
+ directoriesToPull = [
+ "/sdcard/screenshots"
+ ]
+ filesToDownload = [
+ ".*/screenshots/.*"
+ ]
+ testTargets = [
+ "class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView"
+ ]
+ testTimeout = "15m"
+ recordVideo = false
+ performanceMetrics = false
+ devices = [
+ [ "model": "SmallPhone.arm", "version": "28" ],
+ [ "model": "MediumPhone.arm", "version": "33" ]
+ ]
+ projectId("flank-gradle")
+ flankVersion = "23.10.1"
+ debugApk = "$buildDir/outputs/apk/debug/sample-debug.apk"
+ instrumentationApk = "$buildDir/outputs/apk/androidTest/debug/sample-debug-androidTest.apk"
+ additionalTestApks = [
+ "- app: $buildDir/outputs/apk/debug/sample-debug.apk",
+ " test: $buildDir/outputs/apk/androidTest/debug/sample2-debug-androidTest.apk",
+ "- test: ${rootProject.buildDir}/database/module/outputs/apk/database-module-androidTest.apk"
+ ]
+ autoGoogleLogin = true
+ maxTestShards = 8 //or numUniformShards=5 cannot use both.
+ shardTime = 120
+ smartFlankGcsPath = "gs://tmp_flank/flank/test_app_android.xml"
+ configs {
+ oranges {
+ useOrchestrator.set(false)
+ testTargets.set(project.provider { [
+ "class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#runAndFail"
+ ] })
+ }
+ }
+ resultsBucket("my-results-bucket-name")
+ keepFilePath = true
+ runTimout = "45m"
+ ignoreFailedTests = false
+ disableSharding = false
+ smartFlankDisableUpload = false
+ testRunnerClass = "com.foo.TestRunner"
+ localResultsDir = "flank"
+ clientDetails = [
+ "key1": "value1",
+ "key2": "value2"
+ ]
+ testTargetsAlwaysRun = [
+ "com.example.TestSuite#test1",
+ "com.example.TestSuite#test2"
+ ]
+ otherFiles = [
+ "/sdcard/dir1/file1.txt": "/my/example/path/file1.txt",
+ "/sdcard/dir2/file2.txt": "/my/example/path/file2.txt"
+ ]
+ networkProfile = "LTE"
+ roboDirectives = [
+ ["click", "button1", ""],
+ ["ignore", "button2"],
+ ["text", "field1", "my text"],
+ ]
+ outputStyle = 'multi'
+ legacyJunitResult = false
+ fullJunitResult = false
+ additionalApks = [
+ "gs://path/to/app1.apk",
+ "localPath/to/app2.apk"
+ ]
+ defaultTestTime = 5.3
+ defaultClassTestTime = 180.5
+ useAverageTestTimeForNewTests = true
+ disableResultsUpload = true
+ grantPermissions = "none"
+ type = "game-loop"
+ scenarioLabels = [
+ "label1",
+ "label2"
+ ]
+ scenarioNumbers = [ 1, 23, 52 ]
+ obbFiles = [
+ "local/file/path/test1.obb",
+ "local/file/path/test2.obb"
+ ]
+ obbNames = [
+ "patch.0300110.com.example.android.obb",
+ "patch.0300111.com.example.android.obb"
+ ]
+ testTargetsForShard = [
+ "package com.package1.for.shard1, com.package2.for.shard1",
+ "class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2",
+ "class com.foo.ClassForShard3; package com.package.for.shard3"
+ ]
+ failFast = true
+ dependOnAssemble = true
+}
+Whether or not we should use the android test orchestrator to run this tests.
+Set this to true when the build.gradle file includes testOptions.execution 'ANDROID_TEST_ORCHESTRATOR'
useOrchestrator = true
+useOrchestrator.set(true)
+Set multiple testTargets to be run by Flank. These are used to whitelist or blacklist test classes, test cases and test annotations. +See Google Cloud Firebase docs for more information.
+testTargets = [
+ "class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView"
+]
+testTargets.set(listOf(
+ "class com.osacky.flank.gradle.sample.ExampleInstrumentedTest#seeView"
+))
+A list of devices to run the tests against. When list is empty, a default device will be used. Each device entry is a map. The valid keys in the map are model, version, orientation, and locale. When a key is not set or is null, a default value will be used.
devices = [
+ [ "model": "MediumPhone.arm", "version": "26" ],
+ [ "model": "MediumPhone.arm", "version": "33" ]
+]
+devices.set(listOf(
+ mapOf("model" to "MediumPhone.arm", "version" to "26" ),
+ mapOf("model" to "MediumPhone.arm", "version" to "33" )
+))
+The projectId is a unique identifier which can be found in the project's URL: https://console.firebase.google.com/project/<projectId>
+This is automatically discovered based on the service credential by default.
projectId = "flank-gradle"
+projectId.set("flank-gradle")
+Need a different Flank version? Specify it with flankVersion.
To use a snapshot:
+flankVersion = "flank_snapshot"`
+flankVersion.set("flank_snapshot")
+Need more than 50 shards? Use Flank 8.1.0.
To use a different version:
+flankVersion = "23.10.1"
+flankVersion.set("23.10.1")
+Specify custom flank maven coordinates.
+flankCoordinates = "com.github.flank:flank"
+flankCoordinates.set("com.github.flank:flank")
+This is a string representing the path to the app's debug apk. +Supports wildcard characters. +Optional, prefer to set variant.
+debugApk = project.provider { "${buildDir.toString()}/outputs/apk/debug/*.apk" }
+debugApk.set(project.provider { "${buildDir.toString()}/outputs/apk/debug/*.apk" })
+This is a string representing the path to the app's instrumentaiton apk. +Supports wildcard characters. +Optional, prefer to set variant. +InstrumenationApk should not be set when using roboScript.
+instrumentationApk = project.provider { "${buildDir.toString()}/outputs/apk/androidTest/debug/*.apk" }
+instrumentationApk.set(project.provider { "${buildDir.toString()}/outputs/apk/androidTest/debug/*.apk" })
+Paths to additional test configurations.
+Order matters. A test apk is run with the nearest previous listed app apk.
+For library modules, add them to the list with a - test: in front. For test apks which belong to an application module, add them with test:.
+It is not required to list an app apk here. If there is no app apk listed in additionalTestApks, the test apks are run against the debugApk.
additionalTestApks.value(project.provider { [
+"- app: ../main/app/build/output/apk/debug/app.apk",
+" test: ../main/app/build/output/apk/androidTest/debug/app-test.apk",
+"- app: ../sample/app/build/output/apk/debug/sample-app.apk",
+" test: ../sample/app/build/output/apk/androidTest/debug/sample-app-test.apk",
+"- test: ../feature/room/build/output/apk/androidTest/debug/feature-room-test.apk",
+"- test: ../library/databases/build/output/apk/androidTest/debug/sample-databases-test.apk"
+]})
+additionalTestApks.value(project.provider { listOf(
+"- app: ../main/app/build/output/apk/debug/app.apk",
+" test: ../main/app/build/output/apk/androidTest/debug/app-test.apk",
+"- app: ../sample/app/build/output/apk/debug/sample-app.apk",
+" test: ../sample/app/build/output/apk/androidTest/debug/sample-app-test.apk",
+"- test: ../feature/room/build/output/apk/androidTest/debug/feature-room-test.apk",
+"- test: ../library/databases/build/output/apk/androidTest/debug/sample-databases-test.apk"
+)})
+Whether to automatically log in using a preconfigured google account. More Info
+autoGoogleLogin = false
+autoGoogleLogin.set(false)
+Environment variables are mirrored as extra options to the am instrument -e KEY1 VALUE1 … command and passed to your test runner (typically AndroidJUnitRunner). Examples
environmentVariables = [
+ // Whether or not to remove all shared state from your device's CPU and memory after each test. [More info](https://developer.android.com/training/testing/junit-runner)
+ "clearPackageData": "true"
+]
+environmentVariables = mapOf(
+ // Whether or not to remove all shared state from your device's CPU and memory after each test. [More info](https://developer.android.com/training/testing/junit-runner)
+ "clearPackageData" to "true"
+)
+Deprecated
+Use maxTestShards instead.
The maximum number of shards. Fladle will throw an error when used together with maxTestShards or numUniformShards.
+testShards = 5
+testShards.set(5)
+The maximum number of shards. Fladle will throw an error when used together with testShards or numUniformShards.
+maxTestShards = 8
+maxTestShards.set(8)
+The amount of time tests within a shard should take. +When set to > 0, the shard count is dynamically set based on time up to the maximum limit defined by maxTestShards +2 minutes (120) is recommended. +default: -1 (unlimited)
+shardTime = 120
+shardTime.set(120)
+The number of times to repeat each test. Uses Flank's default value when not specified.
+repeatTests = 1
+repeatTests.set(1)
+Give a name to a custom flank task and configure its options. The name is appended to the end of the flank task. For example runFlank becomes runFlank<name>.
configs {
+ repeatOneHundred {
+ // DSL sugar for container elements is missing (= syntax): https://github.com/gradle/gradle/issues/9987
+ repeatTests.set(100)
+ }
+}
+configs {
+ create("repeatOneHundred") {
+ repeatTests.set(100)
+ }
+}
+In the above example, the configuration is inherited from the outer fladle config but with the repeatTests property set to 100. Running runFlankRepeateOneHundred will execute this custom configuration.
The cloud storage location for historical test runs. This must be set in order to use smart sharding. The amount of shards used is set by maxTestShards. The bucket (such as flank_data in the example) must already exist in order for it to be used.
smartFlankGcsPath = 'gs://flank_data/results/JUnitReport.xml'
+smartFlankGcsPath.set("gs://flank_data/results/JUnitReport.xml")
+The history name for your test results (an arbitrary string label; default: the application's label from the APK manifest). All tests which use the same history name will have their results grouped together in the Firebase console in a time-ordered test history list.
+resultsHistoryName = 'android-history'
+resultsHistoryName.set("android-history")
+The number of times to retry failed tests. Default is 0. Max is 10. +Setting the value to 1 will mean that test are retried once. If the test fails then succeeds after the retry the run +will be marked as "successful". The matrix with a flaky test will be marked as flaky.
+flakyTestAttempts = 0
+flakyTestAttempts.set(0)
+The ABI split of the application that should be tested (e.g. "x86"). Only required if the application under test uses ABI splits and the debug APK is selected automatically (via variant) instead of manually (via debugApk).
+If the application uses ABI splits, and this property isn't specified, an arbitrary ABI split will be selected.
+abi = "arm64-v8a"
+abi.set("arm64-v8a")
+A list of paths that will be copied from the device's storage to the designated results bucket after the test is complete. These must be absolute paths under /sdcard or /data/local/tmp. Path names are restricted to the characters a-zA-Z0-9_-./+. The paths /sdcard and /data will be made available and treated as implicit path substitutions. E.g. if /sdcard on a particular device does not map to external storage, the system will replace it with the external storage path prefix for that device.
directoriesToPull = [
+ '/sdcard/tempDir1', '/data/local/tmp/tempDir2'
+]
+directoriesToPull.set(listOf(
+ "/sdcard/tempDir1", "/data/local/tmp/tempDir2"
+))
+List of regex that is matched against bucket paths (for example: 2019-01-09_00:13:06.106000_YCKl/shard_0/SmallPhone.arm-28-en-portrait/bugreport.txt) for files to be downloaded after a flank run. The results are downloaded to the APP_MODULE/build/fladle/RESULTS directory where RESULTS can be set by localResultsDir var otherwise defaulting to results/.
filesToDownload = [
+ '.*/sdcard/tempDir1/.*', '.*/data/local/tmp/tempDir2/.*'
+]
+filesToDownload.set(listOf(
+ ".*/sdcard/tempDir1/.*", ".*/data/local/tmp/tempDir2/.*"
+))
+The max time test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 45m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. +Examples: +* 1h -> 1 hour +* 5m -> 5 minutes +* 200s -> 200 seconds +* 100 -> 100 seconds
+testTimeout = "1h"
+testTimeout.set("1h")
+Enable video recording during the test. Enabled by default.
+recordVideo = true
+recordVideo.set(true)
+Monitor and record performance metrics: CPU, memory, network usage, and FPS (game-loop only). Enabled by default.
+performanceMetrics = true
+performanceMetrics.set(true)
+The name of a Google Cloud Storage bucket where raw test results will be stored.
+resultsBucket = "my-gcs-bucket-name"
+resultsBucket.set("my-gcs-bucket-name")
+Keeps the full path of downloaded files from a Google Cloud Storage bucket. Required when file names are not unique. Disabled by default.
+keepFilePath = false
+keepFilePath.set(false)
+The name of a unique Google Cloud Storage object within the results bucket where raw test results will be stored. The default is a timestamp with a random suffix.
+resultsDir = "result-dir-${getTimeStamp()}"
+resultsDir.set("result-dir-${getTimeStamp()}")
+Disables sharding. All tests will run on the same device. Useful for parameterized tests which do not support sharding. (default: false)
+disableSharding = false
+disableSharding.set(false)
+Disables smart flank JUnit XML uploading. Useful for preventing timing data from being updated. (default: false) +What is Smart Flank?
+smartFlankDisableUpload = false
+smartFlankDisableUpload.set(false)
+The fully-qualified Java class name of the instrumentation test runner (default: the test manifest is parsed to determine the class name).
+testRunnerClass = "com.example.MyCustomTestRunner"
+testRunnerClass.set("com.example.MyCustomTestRunner")
+The local directory to store the test results. Folder is DELETED before each run to ensure only artifacts from the new run are saved. This directory is relative to the working directory of Flank which is by default build/fladle or build/fladle/<flavorname>.
localResultsDir = "my-results-dir"
+localResultsDir.set("my-results-dir")
+Always run - these tests are inserted at the beginning of every shard. Useful if you need to grant permissions or login before other tests run
+testTargetsAlwaysRun = [
+ 'class com.example.MyTestClass'
+]
+testTargetsAlwaysRun.set(listOf(
+ "class com.example.MyTestClass"
+))
+The flags below are only available with Flank 20.05.0 or higher.
+The max time this test run can execute before it is cancelled. s (seconds), m (minutes), h (hours) suffixes are acceptable, mixes like 1h45m are currently not supported (default: unlimited). +Examples:
+runTimeout = "15m"
+runTimeout.set("15m")
+Always return successful task completion even when there are failed tests. Useful when parsing JUnit XML to determine failure. (default: false)
+ignoreFailedTests = false
+ignoreFailedTests.set(false)
+Specifies the number of shards into which you want to evenly distribute test cases. The shards are run in parallel on separate devices. For example, if your test execution contains 20 test cases and you specify four shards, each shard executes five test cases. The number of shards should be less than the total number of test cases. The number of shards specified must be >= 1 and <= 50. This option cannot be used along maxTestShards and is not compatible with smart sharding (Smart Flank). If you want to take benefits of smart sharding use maxTestShards instead. (default: null)
+numUniformShards = 50
+numUniformShards.set(50)
+A key-value map of additional details to attach to the test matrix.(clientDetails in Google Cloud Docs) Arbitrary key-value pairs may be attached to a test matrix to provide additional context about the tests being run. When consuming the test results, such as in Cloud Functions or a CI system, these details can add additional context such as a link to the corresponding pull request. (Access Client Details). +These can be used to provide additional context about the environment where the tests are being run.
+clientDetails = [
+ "test-type": "PR",
+ "build-number": "132"
+]
+clientDetails.set(mapOf(
+ "test-type" to "PR",
+ "build-number" to "132"
+))
+A list of device-path: file-path pairs that indicate the device paths to push files to the device before starting tests, and the paths of files to push. Device paths must be under absolute, whitelisted paths (${EXTERNAL_STORAGE}, or ${ANDROID_DATA}/local/tmp). Source file paths may be in the local filesystem or in Google Cloud Storage (gs://…).
+otherFiles = [
+ "/sdcard/dir1/file1.txt": "local/file.txt",
+ "/sdcard/dir2/file2.jpg": "gs://bucket/file.jpg",
+]
+otherFiles.set(mapOf(
+ "/sdcard/dir1/file1.txt" to "local/file.txt",
+ "/sdcard/dir2/file2.jpg" to "gs://bucket/file.jpg",
+))
+The name of the network traffic profile, for example LTE, HSPA, etc, which consists of a set of parameters to emulate network conditions when running the test (default: no network shaping; see available profiles listed by the flank test network-profiles list command). This feature only works on physical devices.
networkProfile = "LTE"
+networkProfile.set("LTE")
+The path to a Robo Script JSON file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. You can guide the Robo test to perform specific actions by recording a Robo Script in Android Studio and then specifying this argument. Learn more at DOCS.
+roboScript = "my-robo-script.json"
+roboScript.set("my-robo-script.json")
+List of robo_directives that you can use to customize the behavior of Robo test. The type specifies the action type of the directive, which may take on values click, text or ignore. Each directive is list of String = [type, key, value]. Each key should be the Android resource name of a target UI element and each value should be the text input for that element. Values are only permitted for text type elements, so no value should be specified for click and ignore type elements.
+roboDirectives = [
+ ["test, "input_resource_name", "message"],
+ ["click, "button_resource_name", ""],
+]
+roboDirectives.set(listOf(
+ listOf("test", "input_resource_name", "message"),
+ listOf("click", "button_resource_name", ""),
+))
+Output style of execution status. May be one of [verbose, multi, single].
+For runs with only one test execution the default value is 'verbose', in other cases 'multi' is used as the default. The output style 'multi' is not displayed correctly on consoles which don't support ANSI codes, to avoid corrupted output use single or verbose.
multi displays separated status for each shard execution in separated line, lines are updated over time. Similar to gradle output when running multiple tasks in parallel. Requires ANSI codes support.
single displays shortened status of all executions in single line. Similar to gcloud output when running with sharding. Should work on any console.
Default value is single.
+outputSyle = "single"
+outputStyle.set("single")
+Flank provides two ways for parsing junit xml results. +New way uses google api instead of merging xml files, but can generate slightly different output format. +This flag allows fallback for legacy xml junit results parsing
+legacyJunitResult = false
+legacyJunitResult.set(false)
+Enables creating an additional local junit result on local storage with failure nodes on passed flaky tests.
+fullJunitResult = false
+fullJunitResult.set(false)
+Runs a sanityRobo test.
+instrumentationApk, roboDirectives, roboScript and additionalTestApks must be blank or empty.
sanityRobo = true
+sanityRobo.set(true)
+Set default test time expressed in seconds, used for calculating shards. (default: 120.0s)
+defaultTestTime = 1.2
+defaultTestTime.set(1.2)
+Set default parameterized class test time expressed in seconds, used for calculating shards. (default: 2x defaultTestTime => 240s)
+defaultClassTestTime = 245.5
+defaultClassTestTime.set(245,5)
+A list of up to 100 additional APKs to install, in addition to those being directly tested. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
+additionalApks = [
+ "gs://path/to/app1.apk",
+ "localPath/to/app2.apk"
+]
+additionalApks.set(
+ project.provider {
+ listOf("gs://path/to/app1.apk", "localPath/to/app2.apk")
+ }
+)
+Enable using average time from previous tests duration when using SmartShard and tests did not run before. (default: false)
+useAverageTestTimeForNewTests = true
+useAverageTestTimeForNewTests.set(true)
+Disable flank results upload on gcloud storage. (default: false)
+disableResultsUpload = true
+disableResultsUpload.set(true)
+Specifies a group of packages, classes, and/or test cases to run in each shard (a group of test cases).
+The shards are run in parallel on separate devices. You can use this option up to 50 times to specify multiple shards when one or more physical devices are selected,
+or up to 500 times when no physical devices are selected.
+Note: If you include the flags environmentVariables or testTargets when running testTargetsForShard, the flags are applied to all the shards you create.
+You can also specify multiple packages, classes, or test cases in the same shard by separating each item with a comma.
+To specify both package and class in the same shard, separate package and class with semi-colons.
testTargetsForShard = [
+ "package com.package1.for.shard1, com.package2.for.shard1",
+ "class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2",
+ "class com.foo.ClassForShard3; package com.package.for.shard3"
+ ]
+testTargetsForShard.set(
+ project.provider {
+ listOf(
+ "package com.package1.for.shard1, com.package2.for.shard1",
+ "class com.foo.ClassForShard2#testMethod1, com.foo.ClassForShard2#testMethod2",
+ "class com.foo.ClassForShard3; package com.package.for.shard3"
+ )
+ }
+)
+Whether to grant runtime permissions on the device before the test begins. By default, all permissions are granted. PERMISSIONS must be one of: all, none
+grantPermissions = "none"
+grantPermissions.set("none")
+The type of test to run. TYPE must be one of: instrumentation, robo, game-loop. Use if you want to be sure there is only one type of tests being run (flank enables to run mixed types of test in one run).
+type = "game-loop"
+type.set("game-loop")
+A list of game-loop scenario labels (default: None). Each game-loop scenario may be labeled in the APK manifest file with one or more arbitrary strings, creating logical groupings (e.g. GPU_COMPATIBILITY_TESTS).
+If --scenario-numbers and --scenario-labels are specified together, Firebase Test Lab will first execute each scenario from --scenario-numbers.
+It will then expand each given scenario label into a list of scenario numbers marked with that label, and execute those scenarios.
scenarioLabels = [
+ "label1",
+ "label2"
+]
+scenarioLabels.set(
+ project.provider {
+ listOf("label1", "label2")
+ }
+)
+A list of game-loop scenario numbers which will be run as part of the test (default: all scenarios).
+A maximum of 1024 scenarios may be specified in one test matrix, but the maximum number may also be limited by the overall test --timeout setting.
scenarioNumbers = [ 1, 23, 52 ]
+scenarioNumbers.set(
+ project.provider {
+ listOf(1, 23, 52)
+ }
+)
+A list of one or two Android OBB file names which will be copied to each test device before the tests will run (default: None).
+Each OBB file name must conform to the format as specified by Android (e.g. [main|patch].0300110.com.example.android.obb) and will be installed into <shared-storage>/Android/obb/<package-name>/ on the test device.
obbFiles = [
+ "local/file/path/test1.obb",
+ "local/file/path/test2.obb"
+]
+obbFiles.set(
+ project.provider {
+ listOf(
+ "local/file/path/test1.obb",
+ "local/file/path/test2.obb"
+ )
+ }
+)
+A list of OBB required filenames. OBB file name must conform to the format as specified by Android e.g.
+[main|patch].0300110.com.example.android.obb which will be installed into <shared-storage>/Android/obb/<package-name>/ on the device.
obbNames = [
+ "patch.0300110.com.example.android.obb",
+ "patch.0300111.com.example.android.obb"
+]
+obbNames.set(
+ project.provider {
+ listOf(
+ "patch.0300110.com.example.android.obb",
+ "patch.0300111.com.example.android.obb"
+ )
+ }
+)
+If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected. +Normally, 2 or more attempts are made if a potential infrastructure issue is detected. +This feature is for latency sensitive workloads. The incidence of execution failures may be significantly greater for +fail-fast matrices and support is more limited because of that expectation.
+failFast = true
+failFast.set(true)
+Appending additional option to flank root yaml. This option is useful when you would like to test options before official fladle support is added. +Multiple options are supported.
+Single option
+additionalFlankOptions = "new-property: true"
+additionalFlankOptions.set("new-property: true")
+Multiple options
+additionalFlankOptions = """
+ new-property: true
+ other-new-property: force
+""".stripIndent()
+additionalFlankOptions.set("""
+ new-property: true
+ other-new-property: force
+""".trimIndent())
+Allow appending additional config to gcloud root yaml. This option is useful when you would like to test option +before it is available on Fladle. Supports both single and multiple properties.
+Single option
+additionalGcloudOptions = "new-property: true"
+additionalGcloudOptions.set("new-property: true")
+Multiple options
+additionalGcloudOptions = """
+ new-property: true
+ other-new-property: force
+ """.stripIndent()
+additionalGcloudOptions.set("""
+ new-property: true
+ other-new-property: force
+""".trimIndent())
+Enable to automatically build the app and test APKs before runFlank executes. (default: false)
dependOnAssemble = true
+dependOnAssemble.set(true)
+Enable to return immediately after invoking tests, without waiting for results. (default: false)
async = true
+async.set(true)
+
-
-[Sponsor Fladle!](https://github.com/sponsors/runningcode)
-
-## Trusted by
-
-The following companies test their Android app using Fladle.
-
-
-
-
-
-
-Want to appear here? [File an issue](https://github.com/runningcode/fladle/issues/new) or [open a PR](https://github.com/runningcode/fladle/edit/master/docs/index.md)
diff --git a/docs/multi-module-testing.md b/docs/multi-module-testing.md
deleted file mode 100644
index 5489dd64..00000000
--- a/docs/multi-module-testing.md
+++ /dev/null
@@ -1,125 +0,0 @@
-# Multi-module testing
-
-Multi module testing can be done by manually specifying [additionalTestApks](/fladle/configuration/#additionaltestapks) or applying the Fulladle plugin to automacally gather all the additional test apks. See also [this article](https://proandroiddev.com/android-code-coverage-on-firebase-test-lab-part-3-ci-cd-integration-10f729996c29) for a full setup and instructions including integration with CI.
-
-## Fulladle Plugin
-
-1. Apply the Fulladle plugin at the root of the project.
-
- === "Groovy"
- ``` groovy
- plugins {
- id 'com.osacky.fulladle' version '{{ fladle.current_release }}'
- }
- ```
- === "Kotlin"
- ``` kotlin
- plugins {
- id("com.osacky.fulladle") version "{{ fladle.current_release }}"
- }
- ```
-
-2. Configure the Fladle extension.
-
- ===! "Groovy"
- ``` groovy
- fladle {
- serviceAccountCredentials = project.layout.projectDirectory.file("flank-gradle-service-account.json")
- }
- ```
- === "Kotlin"
- ``` kotlin
- fladle {
- serviceAccountCredentials.set(project.layout.projectDirectory.file("flank-gradle-service-account.json"))
- }
- ```
-
- !!! Warning
- If using buildFlavors or testing against a non default variant, you will need to specify the variant you want to test in the fulladleModuleConfig block.
-
-3. Run the tests.
- First assemble all your debug apks and test apks.
- ``` bash
- ./gradlew assembleDebug assembleDebugAndroidTest
- ```
-
- !!! note
- When using flavors, make sure to assemble your buildVariants as well.
-
- `./gradlew :app:assembleFreeDebug :app:assembleFreeDebugAndroidTest`
-
- Run Flank!
- ``` bash
- ./gradlew runFlank
- ```
-
-### Overriding configurations in modules
-Fulladle will pick Flank configurations from the `fladle` block in the root `build.gradle` file. You may want to override some of these configurations for certain modules, you can add the following block to any Android library module to override its configurations:
-
-
-===! "Groovy"
- ``` groovy
- fulladleModuleConfig {
- clientDetails = [
- "test-type": "PR",
- "build-number": "132"
- ]
- maxTestShards = 3
- environmentVariables = [
- "clearPackageData": "true"
- ]
- debugApk = "app.apk"
- variant = "vanillaDebug"
- }
- ```
-=== "Kotlin"
- ``` kotlin
- fulladleModuleConfig {
- clientDetails.set(mapOf(
- "test-type" to "PR",
- "build-number" to "132",
- ))
- maxTestShards.set(3)
- environmentVariables.set(mapOf(
- "clearPackageData" to "true"
- ))
- debugApk.set("app.apk")
- variant.set("vanillaDebug")
- }
- ```
-All of the above configurations are optional, Flank will default to the top-level configurations if you don't override anything here. For details about these configurations, refer to [configuration docs](./configuration.md).
-
-#### Disabling a module
-You may want to exclude a library module from testing when using Fulladle. You can do so by setting the `enabled` configuration in the module's `fulladleModuleConfig` block like so:
-
-
-=== "Groovy"
- ``` groovy
- fulladleModuleConfig {
- enabled = false
- }
- ```
-=== "Kotlin"
- ``` kotlin
- fulladleModuleConfig {
- enabled.set(false)
- }
- ```
-
-
-### Overriding root-level config
-Fulladle does not provide the ability to control which module ends up as the root-level module or as an additional module. Either one of app modules or library modules can become a root-level module. If a library module ends up as a root-level module, it needs to specify a `debugApk` in its `fladle` or `fulladleModuleConfig` block.
-
-The root-level configuration (e.g. `maxTestShards`) can also be overridden in the `fulladleModuleConfig` block of whatever module gets picked as the root module.
-
-## Troubleshooting
-Fulladle might still have some rough edges, but we'd love feedback. Please join us in the [Firebase Community Slack](https://firebase.community/) with any feedback you may have.
-You can also file [Fladle Github issues](https://github.com/runningcode/fladle/issues).
-
-When filing a bug report, please include the Flank version number, the Fladle version number and the output of the following:
-
-`./gradlew printYml`
-
-`./gradlew runFlank -PdumpShards`
-
-
diff --git a/docs/quick-start.md b/docs/quick-start.md
deleted file mode 100644
index 0d6d5fa9..00000000
--- a/docs/quick-start.md
+++ /dev/null
@@ -1,63 +0,0 @@
-# Quick Start
-
-Using Fladle takes 3 steps:
-
-1. Apply the Fladle plugin. Follow instructions [here](https://plugins.gradle.org/plugin/com.osacky.fladle)
-
- Root `build.gradle`
-
- === "Groovy"
- ``` groovy
- buildscript {
- dependencies {
- classpath "com.osacky.flank.gradle:fladle:{{ fladle.current_release }}"
- }
- }
- ```
- === "Kotlin"
- ``` kotlin
- buildscript {
- dependencies {
- classpath("com.osacky.flank.gradle:fladle:{{ fladle.current_release }}")
- }
- }
- ```
-
- Application module `build.gradle`
-
- ===! "Groovy"
- ``` groovy
- apply plugin: "com.android.application"
- apply plugin: "com.osacky.fladle"
- ```
-
- === "Kotlin"
- ``` kotlin
- plugins {
- id ("com.android.application")
- id ("com.osacky.fladle")
- }
- ```
-
-2. Configure [Authentication using these steps.](../authentication)
-
- !!! Warning
- If using buildFlavors or testing against a non default variant, [variant must also configured](/fladle/configuration#variant)
-
-3. Run your tests!
-
- First assemble your debug apk and test apk.
- ``` bash
- ./gradlew :app:assembleDebug :app:assembleDebugAndroidTest
- ```
-
- !!! note
- When using flavors, make sure to assemble your buildVariants.
-
- `./gradlew :app:assembleFreeDebug :app:assembleFreeDebugAndroidTest`
-
- Run Flank!
- ``` bash
- ./gradlew runFlank
- ```
-
diff --git a/docs/recipes.md b/docs/recipes.md
deleted file mode 100644
index 83a7b6a8..00000000
--- a/docs/recipes.md
+++ /dev/null
@@ -1,103 +0,0 @@
-# Recipes
-
-Here are some recipes to use to achieve various goals in flank. For additional recipes or suggestions, please file an
-issue on Github.
-
-## Smartly shard tests in 120 second shards across a maximum of 50 shards.
-
-This recipe will keep track of test durations automatically on firebase test lab and try to split up test runs in to 120 second shards up to maximum of 50 shards.
-
-=== "Groovy"
- ``` groovy
- fladle {
- maxTestShards = 50
- shardTime = 120
- smartFlankGcsPath = "gs://fladle-results/smart-flank/JUnitReport.xml"
- }
- ```
-=== "Kotlin"
- ``` kotlin
- fladle {
- maxTestShards.set(50)
- shardTime.set(120)
- smartFlankGcsPath.set("gs://fladle-results/smart-flank/JUnitReport.xml")
- }
- ```
-
-
-## Run different tests on different devices with different Gradle tasks.
-
-`./gradlew runFlankPerfTests` will execute the performance tests against a MediumPhone.arm
-`./gradlew runFlankRegresssionTests` will execute the regressions tests against a SmallPhone.arm
-
-=== "Groovy"
- ``` groovy
- fladle {
- configs {
- perfTests {
- devices.set([
- ["model" : "MediumPhone.arm", "version" : "28"],
- ["model" : "MediumPhone.arm", "version" : "28"]
- ])
- testTargets.set([
- "class com.sample.MyPerformanceTest"
- ])
- }
- regressionTests {
- devices.set([
- [ "model" : "SmallPhone.arm", "version" : "28"]
- ])
- testTargets.set([
- "class com.sample.MyRegressionTest"
- ])
- }
- }
- }
- ```
-=== "Kotlin"
- ``` kotlin
- fladle {
- configs {
- create("perfTests") {
- devices.set(listOf(
- mapOf("model" to "MediumPhone.arm", "version" to "28" ),
- mapOf("model" to "MediumPhone.arm", "version" to "28")
- ))
- testTargets.set(listOf(
- "class com.sample.MyPerformanceTest"
- ))
- }
- create("regressionTests") {
- devices.set(listOf(
- mapOf("model" to "SmallPhone.arm", "version" to "28" )
- ))
- testTargets.set(listOf(
- "class com.sample.MyRegressionTest"
- ))
- }
- }
- }
- ```
-
-
-## Always use the latest version of Flank
-
-
-Use Gradle's [dynamic version syntax] to declare a dynamic version.
-
-
-!!! warning
- Dynamic versions lead to non-reproducible builds since Gradle will check for new versions periodically based on [how long versions are cached].
-
-=== "Groovy"
- ``` groovy
- flankVersion = "23.+"
- ```
-=== "Kotlin"
- ``` kotlin
- flankVersion.set("23.+")
- ```
-
-
-[dynamic version syntax]: https://docs.gradle.org/current/userguide/dynamic_versions.html#sub:declaring_dependency_with_dynamic_version
-[how long versions are cached]: https://docs.gradle.org/current/userguide/dynamic_versions.html#sec:controlling_dependency_caching_programmatically
\ No newline at end of file
diff --git a/docs/releasing.md b/docs/releasing.md
deleted file mode 100644
index 10ec8351..00000000
--- a/docs/releasing.md
+++ /dev/null
@@ -1,68 +0,0 @@
-# Releasing
-
-* Create a local release branch from `master`
-```bash
-git checkout master
-git pull
-git checkout -b release_{{ fladle.next_release }}
-```
-
-* Update `version` in `fladle-plugin/build.gradle.kts` (remove `-SNAPSHOT`)
-```kotlin
-version = "{{ fladle.next_release }}"
-```
-
-* Update the current version and next version in `mkdocs.yml`:
-```
-extra:
- fladle:
- release: '{{ fladle.next_release }}'
- next_release: 'REPLACE_WITH_NEXT_VERSION_NUMBER'
-```
-
-* Take one last look
-```
-git diff
-```
-
-* Commit all local changes
-```
-git commit -am "Prepare {{ fladle.next_release }} release"
-```
-
-* Create a tag and push it
-```bash
-git tag v{{ fladle.next_release }}
-git push origin v{{ fladle.next_release }}
-```
-
-Pushing the tag automatically triggers the [release workflow](https://github.com/runningcode/fladle/actions/workflows/gradle-release.yml) which:
-
-1. Publishes to Maven Central
-2. Publishes to Gradle Plugin Portal
-3. Creates a GitHub Release with auto-generated notes
-
-* Release to Maven Central
- * Login to Maven Central Repository: [https://central.sonatype.com/](https://central.sonatype.com/)
- * Click on **Publish**
-
-* Merge the release branch to master
-```
-git checkout master
-git pull
-git merge --no-ff release_{{ fladle.next_release }}
-```
-* Update `version` in `fladle-plugin/build.gradle.kts` (increase version and add `-SNAPSHOT`)
-```kotlin
-version = "REPLACE_WITH_NEXT_VERSION_NUMBER-SNAPSHOT"
-```
-
-* Commit your changes
-```
-git commit -am "Prepare for next development iteration"
-```
-
-* Push your changes
-```
-git push
-```
diff --git a/docs/results.md b/docs/results.md
deleted file mode 100644
index b36f6a10..00000000
--- a/docs/results.md
+++ /dev/null
@@ -1,11 +0,0 @@
-# Results
-
-By default, results are placed in the `build/fladle/results/The app APK and the instrumentation APK are expected to have already been generated before calling runFlank. To generate APKs, run assembleDebug and assembleDebugAndroidTest before running runFlank.
You can also have Fladle build them for you by using the dependOnAssemble property.
If you receive an error like this, it is likely caused by invalid fladle extension confiuration.
+The syntax was changed in the 0.9.X releases in order to avoid touching files during the configuration phase.
+
No signature of method: flank_4vvjv7w3oopge32w1tl9cs6e4.fladle() is applicable for argument types: (flank_4vvjv7w3oopge32w1tl9cs6e4$_run_closure1) values: [flank_4vvjv7w3oopge32w1tl9cs6e4$_run_closure1@649a2315]
+ Possible solutions: file(java.lang.Object), find(), findAll(), file(java.lang.Object, org.gradle.api.PathValidation), files([Ljava.lang.Object;), findAll(groovy.lang.Closure)
+If you receive a similar error, please check configuration for a sample configuration.
+./gradlew runFlank -PdumpShards Will dump shards and exit the process without running the tests.
./gradlew printYml Will print out the current yaml configuration to be passed to Flank.
Still having trouble? Check the #flank channel in the Firebase Community Slack
+ + + + + + +