Repository navigation
Conversation
Lets an MCP client drive a real Android/iOS device (App Automate, Appium) or a desktop/mobile browser (Automate, WebDriver) step by step: start a session, read the screen, tap, type, swipe, press keys, manage the app, change device state, and end it with a passed/failed status. - Device catalog: listMobileDevices (App Automate list, exact names), listDedicatedDevices, getTeamDefaultDevices, uploadMobileApp, listUploadedApps, uploadMediaFile, getAppAutomatePlan. - Device sessions: 19 tools, including biometric, camera image injection, GPS, network profiles, file push/pull and read-only adb. - Browser sessions: 12 tools (navigate, click, type/select, scroll, tabs and frames, screenshot, page reading); BrowserStack Local via local: true. - src/http-server.ts: Streamable HTTP entry point with per-user Basic login verified against BrowserStack (fails closed); one server instance per session; sessions owned by a hash of username + key. - Guard rails: exact device/OS/browser names only, per-user session-start limit, open-session cap, waits around actions, credential scrubbing on every result and error, typed values never echoed, password fields masked. - Existing tools are unchanged; the new groups register beside them and can be switched off with BROWSERSTACK_DEVICE_SESSION_TOOLS=false. - docs/AGENT-DEVICE-AND-BROWSER-SESSIONS.md explains the feature for product and engineering readers; README and TOOLS.md link to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
- iOS back: the edge-pan gesture now starts exactly at x = 0 and moves in steps. A start at x = 2 or one long move is ignored by iOS; verified on iPhone 15 with iOS 17 and iOS 27. - Remove readDeviceScreen and readBrowserPage (including the page-HTML mode). Agents see the screen through screenshots; the server no longer hands out element locators or page HTML. - runAdbShellCommand accepts getprop only. pm list / dumpsys could show apps other customers left on shared public phones. - On-screen keyboard: tapOnDevice and typeOnDevice close the keyboard first when it covers the target element (iOS: by the keyboard's bounds; Android: before tapping a non-text element), then look the element up again. typeOnDevice gains hideKeyboard and says when the keyboard is left open. When an app offers no way to close the keyboard, the tools say so instead of reporting a tap that landed on the keyboard; a coordinate tap on the keyboard is noted in the result. setDeviceState hideKeyboard tries Done, a swipe down, then Return, and fails clearly if all three are ignored. - Live view: starting a device or browser session opens its live view in the local browser (https browserstack.com links only, no shell; local server only; BROWSERSTACK_OPEN_LIVE_VIEW=false turns it off). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found while re-running the failing cases end to end with an agent: - pressDeviceKey back on iOS compares the screen before and after the edge swipe. Screens an app draws with its own header instead of an iOS navigation bar ignore the system swipe; the tool now says so and points the agent at the app's own back control instead of reporting success. The swipe itself is unchanged (verified again in Settings). - manageDeviceApp launch/restart checks which app is in front afterwards. activateApp can return while the device stays on the home screen (an app restricted on a shared device); that is now an error, not "Launched". - The keyboard-blocked message suggests a route that needs no typing, or leaving the screen with the app's own back control, before reporting an app issue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… close via typeOnDevice
From a second round of internal re-testing:
- runAdbShellCommand needs a property ("getprop <property>"). A bare
getprop is refused by BrowserStack, which also accepts only some
properties on shared phones (verified live: ro.build.version.release and
ro.product.manufacturer pass; the model and SDK level are refused). Its
refusal now comes back as a clear message pointing at getDeviceSession
for the device name and OS version.
- swipeOnDevice on iOS moves the finger in five short steps. iOS ignores an
edge swipe made of one long move, so a manual swipe from x = 0 did not go
back; it now does (verified on iPhone 15, iOS 17).
- typeOnDevice hideKeyboard tries Return last, as setDeviceState does, and
says how the keyboard was closed. Without it, apps whose keyboard closes
only on Return were wrongly reported as having no way to close it. A tap
blocked by the keyboard now says Return was not pressed and points at
setDeviceState hideKeyboard instead of calling it an app issue outright.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closing the keyboard tried a downward swipe that started on the keys. iOS
reads a drag that starts on the keyboard as swipe-typing, so a word was
typed into the focused field ("TV" in internal testing; "to" in a live
repro on iPhone 15, iOS 27). That step now drags down above the keyboard
and stops short of it, so no key is ever touched. A test checks that no
pointer move lands on the keyboard while closing it.
Verified on iPhone 15, iOS 17 and 27: typed text stays exactly as typed
after setDeviceState hideKeyboard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An MCP client can now drive a real Android or iOS device or a real desktop or mobile browser on BrowserStack step by step. It starts a session, reads the screen, taps, types, swipes and checks the result, then ends the session with a passed/failed status. The server only exposes capabilities; deciding what to do stays with the agent.
📄 Start here: docs/AGENT-DEVICE-AND-BROWSER-SESSIONS.md covers what it does, how it works, setup, prompts to try, settings, safety and limits. It's written for product and engineering readers.
listMobileDevices,listDedicatedDevices,getTeamDefaultDevices,uploadMobileApp,listUploadedApps,uploadMediaFile,getAppAutomatePlanstartDeviceSession,getDeviceSession,endDeviceSession,takeDeviceScreenshot,tapOnDevice,typeOnDevice,swipeOnDevice,pressDeviceKey,manageDeviceApp,uninstallDeviceApp,setDeviceState,simulateBiometric,injectCameraImage,setDeviceLocation,setNetworkConditions,pushFileToDevice,pullFileFromDevice,runAdbShellCommandlistBrowsers,getAutomatePlan,startBrowserSession,getBrowserSession,endBrowserSession,navigateBrowser,clickInBrowser,typeInBrowser,scrollBrowser,switchBrowserContext,takeBrowserScreenshotnpm run start:http(Streamable HTTP, per-user login)src/http-server.tsDesign notes
startDeviceSessionaccepts.runAppLiveSessionis unchanged.BROWSERSTACK_SESSION_STARTS_PER_HOUR)BROWSERSTACK_DEVICE_SESSION_MAX)getprop, so nothing can list apps other customers left on shared phonesuninstallDeviceAppmarked destructiveBROWSERSTACK_DEVICE_SESSION_TOOLS=false.Pre-release feedback addressed (commits 2e9f00e, 151ca21, a431dd1 and the keyboard fix after them)
From internal testing on real devices. Re-checked with BrowserStack's own sample apps (BStackSampleApp, WikipediaSample) and bstackdemo.com:
pressDeviceKey backstarts the edge swipe exactly at x = 0 and moves in steps (a start at x = 2 or one long move is ignored by iOS).swipeOnDeviceon iOS also moves in steps, so a manual edge swipe goes back too. When a screen ignores the swipe (apps that draw their own header instead of an iOS navigation bar), the tool says the screen did not change and points to the app's own back controlpressDeviceKeyandswipeOnDevicefrom x = 0readDeviceScreenandreadBrowserPageremoved; agents use screenshotsreadBrowserPagerunAdbShellCommandacceptsgetprop <property>only. BrowserStack itself accepts only some properties; its refusal now comes back as a clear messagepm list packages,dumpsysand a baregetproprefused;getprop ro.build.version.releaseandro.product.manufacturerworktypeOnDevicegainshideKeyboard(Done, a drag above the keyboard, then Return last, assetDeviceState hideKeyboarddoes). When nothing closes it, the tool says so instead of reporting a tap that hit the keyboardBROWSERSTACK_OPEN_LIVE_VIEW=falseto turn off)setDeviceState hideKeyboard(live repro of the old step typed "to")manageDeviceApplaunch/restart fails if the app never reached the front instead of claiming "Launched"End-to-end agent runs (headless Claude Code, only this server loaded): BStackSampleApp on iPhone 15 (type, close keyboard, back, alert) passed; bstackdemo.com in Safari on iPhone 15 (sign in, two items, shipping form, stop before Submit) passed; WikipediaSample on Pixel 8 searched fine (its old build can no longer load articles, an app limitation).
Testing
tscandeslintare clean. The new unit tests cover keyboard handling, opening the live view, the iOS back swipe, credential scrubbing, strict resolution, session ownership, the rate limit, HTTP login (including fail-closed), the upload guards, every action against a fake driver, and registration.buildFailureThemes/tfaRcaCollaborationthat already fail onmain, so CI will show them.getprop, file push/pull, GPS.<select>, tabs, iframes, scrolling.localhostpage reached through BrowserStack Local.Before merging
🤖 Generated with Claude Code