add section on concurrency limits to dashboard/test/ui/README.md (#73154)

davidsbailey · web-flow · commit da2dabe7b2a6 · 2026-06-11T14:04:44.000-07:00
diff --git a/dashboard/test/ui/README.md b/dashboard/test/ui/README.md
@@ -28,6 +28,33 @@ The UI tests run as part of our deployment during the Deploy To Test (DTT) via `
 
 Each suite uploads its own status page (`test_status_{Safari_iPad_iPhone_UI,Chrome_Firefox_UI,Eyes}.html`) to the test machine and to S3.
 
+## Concurrency limits
+
+SauceLabs and Device Farm each have their own cost structure and concurrency
+limits. SauceLabs charges by maximum concurrency. Device Farm charges per test
+run, but also has separate concurrency limits for desktop and mobile within each
+AWS account. Device Farm limits can be increased at no additional cost by
+negotiating with AWS support based on current need an availability.
+
+Today, the relevant concurrency limits are:
+
+| Provider            | AWS Account | Concurrency | Usage                   |
+|---------------------|-------------|-------------|-------------------------|
+| SauceLabs           | —           | 65          | DTT + local development |
+| Device Farm Desktop | codeorg     | 150         | DTT + local development |
+| Device Farm Desktop | codeorg-dev | 150         | Drone (CI)              |
+
+Staying under concurrency limits in DTT is straightforward, because only one DTT
+runs at a time. Staying under limits in Drone is less predictable, because there
+is no limit to the number of concurrent Drone runs, and each concurrent drone
+run might not be running the maximum number of UI tests at a given time.
+Therefore, the simplest plan is to request additional increases to Device Farm
+Desktop (codeorg-dev) concurrency if/when we start seeing UI tests fail in Drone
+with concurrency errors like this one:
+```
+Cannot exceed 150 concurrent sessions. Currently active=130, pending=20, failed=0 (Selenium::WebDriver::Error::SessionNotCreatedError)
+```
+
 ## Local Setup
 
 ### On your machine: Chrome webdriver
diff --git a/lib/rake/ci.rake b/lib/rake/ci.rake
@@ -171,14 +171,6 @@ namespace :ci do
       container_features = `find ./features -name '*.feature' | sort`.split("\n").map {|f| f[2..]}
       eyes_features = `grep -lr '@eyes' features`.split("\n")
       container_eyes_features = container_features & eyes_features
-      # The concurrency limit for Device Farm desktop sessions in the
-      # codeorg-dev AWS account is 50. This concurrency limit is shared across
-      # all CI jobs. This limit is not shared by sessions running in local
-      # development or the chef-managed test environment, which run in the prod
-      # AWS account. This limit must be taken into consideration when choosing
-      # parallelism as well as when choosing to run on Device Farm vs SauceLabs
-      # or local chromedriver.
-      #
       # Use --local to configure the UI tests to run against localhost, and
       # --config to override the local webdriver with the remote provider
       # (Device Farm Chrome by default, SauceLabs under [use saucelabs] or any
diff --git a/lib/rake/test.rake b/lib/rake/test.rake
@@ -43,7 +43,7 @@ namespace :test do
       '-d', CDO.site_host('studio.code.org'),
       '-p', CDO.site_host('code.org'),
       '--db', # Ensure features that require database access are run even if the server name isn't "test"
-      '--parallel', '65', # The total saucelabs concurrency budget is 65
+      '--parallel', '65',
       '--magic_retry',
       '--with-status-page',
       '--fail_fast',
@@ -62,8 +62,6 @@ namespace :test do
   end
 
   timed_task_with_logging :devicefarm_desktop_ui do
-    # As of June 2026, our concurrency limit for desktop browser sessions in
-    # Device Farm within our prod AWS account is 150.
     ChatClient.log 'Running <b>dashboard</b> Chrome + Firefox UI tests...'
     failed_browser_count = RakeUtils.system_with_chat_logging(
       "cd #{dashboard_dir('test/ui')} &&",
