Skip to content

Commit b2cb16d

Browse files
DjDeveloperrDjDeveloperr
committed
Add service client functions for simulator interactions and tap target resolution
- Implemented various service functions for HTTP requests to interact with the simulator, including JSON and byte handling. - Added performance monitoring capabilities with `service_performance_json` and `run_stats_watch`. - Introduced tap target resolution logic in `tap_target.rs` to handle touch interactions based on accessibility snapshots. - Enhanced `NativeBridge` to support interactive accessibility snapshots. - Updated FFI bindings to accommodate new parameters for accessibility snapshot functions. - Improved documentation in `SKILL.md` to reflect new command usage and features for agent interactions.
1 parent 90617b9 commit b2cb16d

39 files changed

Lines changed: 7632 additions & 5414 deletions

‎README.md‎

Lines changed: 9 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -98,45 +98,11 @@ simdeck use <udid>
9898
simdeck tap 0.5 0.5 --normalized
9999
simdeck tap "Continue"
100100
simdeck describe --format agent --max-depth 2 --interactive
101+
simdeck press @e3
102+
simdeck snapshot --format agent --max-depth 2 -i
101103
simdeck --device <other-udid> describe --format agent --max-depth 2
102104
```
103105

104-
## Daemon
105-
106-
Manage the project daemon explicitly when needed:
107-
108-
```sh
109-
simdeck daemon start
110-
simdeck daemon restart
111-
simdeck daemon status
112-
simdeck daemon stop
113-
simdeck daemon killall
114-
```
115-
116-
`simdeck daemon` manages the normal per-project warm process. `daemon killall`
117-
stops SimDeck daemons across all workspaces.
118-
119-
Use software H.264 when the hardware encoder is unavailable, busy, or starved
120-
by screen recording:
121-
122-
```sh
123-
simdeck daemon start --video-codec software
124-
```
125-
126-
Restart the CoreSimulator service layer when `simctl` reports a stale service
127-
version or the live display gets stuck before the first frame:
128-
129-
```sh
130-
simdeck core-simulator restart
131-
```
132-
133-
You can also start or stop the CoreSimulator service layer explicitly:
134-
135-
```sh
136-
simdeck core-simulator start
137-
simdeck core-simulator shutdown
138-
```
139-
140106
## CLI
141107

142108
```sh
@@ -161,12 +127,17 @@ simdeck stream --frames 120 > stream.h264
161127
simdeck describe
162128
simdeck describe --format agent --max-depth 4
163129
simdeck describe --format agent --max-depth 4 --interactive
130+
simdeck snapshot --format agent --max-depth 4 -i
164131
simdeck describe --point 120,240
165132
simdeck wait-for --label "Welcome" --timeout-ms 5000
133+
simdeck wait --label "Welcome" --timeout-ms 5000
166134
simdeck assert --id login.button --source auto --max-depth 8
167135
simdeck tap 120 240
168136
simdeck tap --label "Continue" --wait-timeout-ms 5000
137+
simdeck tap --id com.apple.settings.screenTime --expect-id BackButton
169138
simdeck tap "Continue"
139+
simdeck press @e3
140+
simdeck back
170141
simdeck swipe 200 700 200 200
171142
simdeck gesture scroll-down
172143
simdeck pinch --start-distance 160 --end-distance 80
@@ -184,7 +155,7 @@ simdeck button action --duration-ms 1000
184155
simdeck button digital-crown
185156
simdeck crown --delta 50
186157
simdeck button left-side-button
187-
simdeck batch --step "tap --label Continue" --step "type 'hello'" --step "wait-for --label hello"
158+
simdeck batch --step "tap --label Continue --expect-label Done" --step "type 'hello'" --step "back"
188159
simdeck dismiss-keyboard
189160
simdeck button software-keyboard
190161
simdeck home
@@ -205,35 +176,7 @@ metadata.
205176
`simdeck use <udid>` stores a default simulator for the current project
206177
directory. Most device commands accept `[<udid>]`; when it is omitted, SimDeck
207178
uses `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, the saved project default,
208-
or the only booted simulator, in that order. The old explicit-UDID form still
209-
works for every command.
210-
211-
`boot` uses SimDeck's private CoreSimulator boot path so it can start devices
212-
without launching Simulator.app. If that private path is unavailable, the
213-
command returns the CoreSimulator error instead of falling back to
214-
`xcrun simctl boot`.
215-
216-
Android emulators appear in `simdeck list` with IDs like
217-
`android:SimDeck_Pixel_8_API_36`. For Android IDs, lifecycle, install, launch,
218-
URL, screenshot, logs, UIAutomator `describe`, tap, swipe, text, key, home, app
219-
switcher, rotation, pasteboard, and browser live view route through the Android
220-
SDK tools (`emulator` and `adb`) plus the emulator gRPC screenshot stream for
221-
live video. `simdeck stream` remains iOS-only because it writes the iOS H.264
222-
transport stream.
223-
224-
`stream` writes an Annex B H.264 elementary stream to stdout for diagnostics or
225-
external tools such as `ffplay`.
226-
227-
`describe` uses the project daemon to prefer React Native, NativeScript,
228-
Flutter, or UIKit in-app inspectors, then falls back to the built-in private
229-
CoreSimulator accessibility bridge. Use `--format agent` or
230-
`--format compact-json` for
231-
lower-token hierarchy dumps, and add `--interactive`/`-i` when an agent only
232-
needs actionable elements plus their ancestors. Set a project default with
233-
`simdeck use <udid>` so agent commands can use short forms like
234-
`simdeck tap "Continue"` and `simdeck describe --format agent --max-depth 2`.
235-
Coordinate commands accept screen coordinates from the accessibility tree by
236-
default; pass `--normalized` to send `0.0..1.0` coordinates directly.
179+
or the only booted simulator, in that order.
237180

238181
## JS/TS Tests
239182

@@ -257,12 +200,6 @@ healthy, and only stops daemons it started itself. Pass `udid` to `connect()`
257200
to make it the default for session methods; each method still accepts an
258201
explicit UDID as the first argument when needed.
259202

260-
Run common Maestro YAML flows against the same daemon-backed simulator API:
261-
262-
```sh
263-
simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro
264-
```
265-
266203
## NativeScript Inspector
267204

268205
NativeScript apps can connect directly to the running server from JS and expose
@@ -318,13 +255,6 @@ void main() {
318255
}
319256
```
320257

321-
## VS Code
322-
323-
Install the `nativescript.simdeck-vscode` extension from the VS Code Marketplace, then
324-
run `SimDeck: Open Simulator View` from the Command Palette. The extension
325-
opens the simulator inside a VS Code panel and auto-starts the local daemon
326-
when it is not already reachable.
327-
328258
## Contributing
329259

330260
Contributors should read [CONTRIBUTING.md](CONTRIBUTING.md) for local build

‎cli/XCWAccessibilityBridge.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ NS_ASSUME_NONNULL_BEGIN
77
+ (nullable NSDictionary *)accessibilitySnapshotForSimulatorUDID:(NSString *)udid
88
atPoint:(nullable NSValue *)pointValue
99
maxDepth:(NSUInteger)maxDepth
10+
interactiveOnly:(BOOL)interactiveOnly
1011
error:(NSError * _Nullable * _Nullable)error;
1112

1213
@end

‎cli/XCWAccessibilityBridge.m‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1353,6 +1353,7 @@ @implementation XCWAccessibilityBridge
13531353
+ (nullable NSDictionary *)accessibilitySnapshotForSimulatorUDID:(NSString *)udid
13541354
atPoint:(nullable NSValue *)pointValue
13551355
maxDepth:(NSUInteger)maxDepth
1356+
interactiveOnly:(BOOL)interactiveOnly
13561357
error:(NSError * _Nullable __autoreleasing *)error {
13571358
if (![self.class loadAndValidate:error]) {
13581359
return nil;
@@ -1421,7 +1422,7 @@ + (nullable NSDictionary *)accessibilitySnapshotForSimulatorUDID:(NSString *)udi
14211422
}
14221423
// Shallow snapshots power fast agent describe loops. Keep the expensive
14231424
// multi-root recovery pass for full trees, or when frontmost lookup fails.
1424-
BOOL shouldRecoverRoots = pointValue == nil && (translation == nil || maxDepth > 2);
1425+
BOOL shouldRecoverRoots = pointValue == nil && (translation == nil || (!interactiveOnly && maxDepth > 2));
14251426
if (shouldRecoverRoots) {
14261427
NSMutableDictionary<NSNumber *, NSMutableDictionary *> *candidatesByKey = [NSMutableDictionary dictionary];
14271428
if (translation != nil) {

‎cli/native/XCWNativeBridge.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ xcw_native_owned_bytes xcw_native_screen_recording_mp4(const char * _Nonnull udi
5656
char * _Nullable xcw_native_start_screen_recording(const char * _Nonnull udid, char * _Nullable * _Nullable error_message);
5757
xcw_native_owned_bytes xcw_native_stop_screen_recording(const char * _Nonnull recording_id, char * _Nullable * _Nullable error_message);
5858
char * _Nullable xcw_native_recent_logs(const char * _Nonnull udid, double seconds, size_t limit, char * _Nullable * _Nullable error_message);
59-
char * _Nullable xcw_native_accessibility_snapshot(const char * _Nonnull udid, bool has_point, double x, double y, size_t max_depth, char * _Nullable * _Nullable error_message);
59+
char * _Nullable xcw_native_accessibility_snapshot(const char * _Nonnull udid, bool has_point, double x, double y, size_t max_depth, bool interactive_only, char * _Nullable * _Nullable error_message);
6060
bool xcw_native_send_touch(const char * _Nonnull udid, double x, double y, const char * _Nonnull phase, char * _Nullable * _Nullable error_message);
6161
bool xcw_native_send_key(const char * _Nonnull udid, uint16_t key_code, uint32_t modifiers, char * _Nullable * _Nullable error_message);
6262
bool xcw_native_send_key_event(const char * _Nonnull udid, uint16_t key_code, bool down, char * _Nullable * _Nullable error_message);

‎cli/native/XCWNativeBridge.m‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ @interface XCWNativeAccessibilitySnapshotRequest : NSObject
7777
@property (nonatomic, assign) double x;
7878
@property (nonatomic, assign) double y;
7979
@property (nonatomic, assign) NSUInteger maxDepth;
80+
@property (nonatomic, assign) BOOL interactiveOnly;
8081
@property (nonatomic, assign) char *result;
8182
@property (nonatomic, assign) char *serializationError;
8283
@property (nonatomic, strong) NSError *snapshotError;
@@ -94,6 +95,7 @@ - (void)performSnapshot {
9495
NSDictionary *snapshot = [XCWAccessibilityBridge accessibilitySnapshotForSimulatorUDID:self.udid
9596
atPoint:pointValue
9697
maxDepth:self.maxDepth
98+
interactiveOnly:self.interactiveOnly
9799
error:&error];
98100
if (snapshot == nil) {
99101
self.snapshotError = error;
@@ -607,14 +609,15 @@ xcw_native_owned_bytes xcw_native_stop_screen_recording(const char *recording_id
607609
}
608610
}
609611

610-
char *xcw_native_accessibility_snapshot(const char *udid, bool has_point, double x, double y, size_t max_depth, char **error_message) {
612+
char *xcw_native_accessibility_snapshot(const char *udid, bool has_point, double x, double y, size_t max_depth, bool interactive_only, char **error_message) {
611613
@autoreleasepool {
612614
XCWNativeAccessibilitySnapshotRequest *request = [XCWNativeAccessibilitySnapshotRequest new];
613615
request.udid = XCWStringFromCString(udid);
614616
request.hasPoint = has_point;
615617
request.x = x;
616618
request.y = y;
617619
request.maxDepth = max_depth;
620+
request.interactiveOnly = interactive_only;
618621

619622
NSThread *accessibilityThread = XCWNativeAccessibilityThread();
620623
if (NSThread.currentThread == accessibilityThread) {

‎client/src/api/controls.ts‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,19 @@ async function postSimulatorAction(
3939
action: string,
4040
payload?: LaunchPayload | OpenUrlPayload,
4141
): Promise<SimulatorMetadata | null> {
42+
if (action === "launch" || action === "open-url") {
43+
const response = await apiRequest<{
44+
ok: boolean;
45+
simulator?: SimulatorMetadata | null;
46+
}>(`/api/simulators/${encodeURIComponent(udid)}/action`, {
47+
method: "POST",
48+
body: JSON.stringify({
49+
action: action === "open-url" ? "openUrl" : "launch",
50+
...payload,
51+
}),
52+
});
53+
return response.simulator ?? null;
54+
}
4255
const response = await apiRequest<SimulatorResponse | { ok: boolean }>(
4356
`/api/simulators/${udid}/${action}`,
4457
{
@@ -66,6 +79,16 @@ export function launchSimulatorBundle(udid: string, payload: LaunchPayload) {
6679
return postSimulatorAction(udid, "launch", payload);
6780
}
6881

82+
export function toggleSimulatorAppearance(udid: string) {
83+
return apiRequest<{ ok: boolean }>(
84+
`/api/simulators/${encodeURIComponent(udid)}/action`,
85+
{
86+
method: "POST",
87+
body: JSON.stringify({ action: "toggleAppearance" }),
88+
},
89+
);
90+
}
91+
6992
export function uploadSimulatorApp(
7093
udid: string,
7194
file: File,

0 commit comments

Comments
 (0)