feat(copilot): server-side mothership tool/vfs/file metrics by Sg312 · Pull Request #5071 · simstudioai/sim

Sg312 · 2026-06-15T22:12:31Z

Summary

Mship observability updates
Companion: https://github.com/simstudioai/copilot/pull/306

Type of Change

New feature

Testing

Manual

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…gging - copilot.tool.duration/calls, vfs.materialize, file.read metrics on Sim's MeterProvider (shared buckets, bounded labels, name-capped) at the tool/VFS/file-read boundaries - metrics-v1 TS mirror + sync-metrics-contract script + GENERATORS entry; regen trace-attributes/events mirrors (adds gen_ai.system) - deployment.environment from APPCONFIG_ENVIRONMENT (production -> prod) so a single Grafana $env spans Sim + Go without a new infra env var

vercel · 2026-06-15T22:12:37Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jun 15, 2026 11:18pm

greptile-apps · 2026-06-15T22:20:50Z

Greptile Summary

This PR adds server-side OTel metrics to Sim's copilot layer (tool execution, VFS materialization, and file reads) so Go and Sim emit identical metric names, label keys, and histogram bucket boundaries — enabling a unified Go∪Sim query set in Grafana. It also fixes the deployment environment tag by inserting APPCONFIG_ENVIRONMENT into the fallback chain so deployed tiers no longer incorrectly tag everything as "production" via the NODE_ENV fallback.

Introduces apps/sim/lib/copilot/request/metrics.ts as the first OTel metrics consumer on the Sim side, with a lazy-init instruments() that mirrors Go's histogram bucket boundaries; a new scripts/sync-metrics-contract.ts code-gen script keeps metric names in sync with the Go JSON Schema contract.
Generated contract files (metrics-v1.ts, trace-attributes-v1.ts, trace-events-v1.ts) are updated to reflect upstream Go-side additions (context.transform, router config, messaging attributes) and removals (bedrock-specific attributes).

Confidence Score: 4/5

The change is additive telemetry only — no data paths, auth flows, or user-facing behavior is modified.

Two instrumentation call sites (workspace-vfs.ts and executor.ts) silently drop metrics when the inner operation throws, meaning error durations never appear in the histogram and p50/p99 will be skewed toward happy-path timings. Additionally, file-reader.ts records raw record.size (stored input bytes) rather than actual output bytes, which diverges for resized images and text-extracted documents. Neither issue affects runtime correctness or user-facing behavior, but the dashboards built on these metrics will have structural blind spots from day one.

apps/sim/lib/copilot/vfs/workspace-vfs.ts and apps/sim/lib/copilot/request/tools/executor.ts — both emit metrics only after a successful await, so thrown errors produce no signal. apps/sim/lib/copilot/vfs/file-reader.ts — the size argument uses source file size rather than output bytes.

Important Files Changed

Filename	Overview
apps/sim/instrumentation-node.ts	Adds deploymentEnvFromAppConfig to map APPCONFIG_ENVIRONMENT=production → prod so Sim's OTel resource tag matches Go's value space; inserted correctly in the fallback chain between DEPLOYMENT_ENVIRONMENT and NODE_ENV.
apps/sim/lib/copilot/generated/metrics-v1.ts	Auto-generated metric name enum from the Go-side JSON Schema contract; Metric const + MetricValues sorted array follow the same pattern as the existing trace-attribute generated files.
apps/sim/lib/copilot/request/metrics.ts	New module wiring OTel histograms/counters for tool, VFS, and file-read metrics; lazy instrument initialization and cardinality caps are well-designed, but metrics on error/throw paths in callers are silently dropped.
apps/sim/lib/copilot/request/tools/executor.ts	Adds startedAt/durationMs and calls recordSimToolMetric after executeToolAndReportInner; metric is skipped if the inner call throws (error outcomes produce no signal).
apps/sim/lib/copilot/vfs/file-reader.ts	Wraps startActiveSpan return in a variable, then calls recordFileRead; uses record.size ?? 0 (input size) rather than output bytes, which may overstate bytes for resized images and parseable documents.
apps/sim/lib/copilot/vfs/workspace-vfs.ts	Records per-phase and total VFS materialization durations after the span resolves; correctly reuses the computed totalMs in the existing logger call, but error paths that throw will not emit metrics.
scripts/sync-metrics-contract.ts	New code-gen script that reads the Go-side metrics-v1.schema.json and emits metrics-v1.ts; reuses the same PascalCase identifier + collision-detection pattern as the trace-attribute sync scripts.
apps/sim/lib/copilot/generated/trace-attributes-v1.ts	Auto-generated; adds new context.transform., copilot.vfs.phase, messaging, router, and server.address attributes while removing bedrock and router.bedrock_ entries in sync with the Go schema.
apps/sim/lib/copilot/generated/trace-events-v1.ts	Auto-generated; replaces BedrockInvokeRetryWithoutImages with ContextTransform in both the const object and the sorted values array.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Tool Call - executor.ts] -->|startedAt = Date.now| B[executeToolAndReportInner]
    B -->|success| C[durationMs = Date.now - startedAt]
    B -->|throws| X1[metric SKIPPED]
    C --> D[recordSimToolMetric name, outcome, durationMs]
    D --> E[metrics.ts instruments]
    E --> F[copilot.tool.calls counter]
    E --> G[copilot.tool.duration histogram]
    H[readFileRecord - file-reader.ts] -->|startedAt = Date.now| I[startActiveSpan callback]
    I -->|null on internal error| J[recordFileRead read_failed record.size]
    I -->|non-null result| K[recordFileRead success record.size]
    K -->|input size not output| E
    L[materializeVFS - workspace-vfs.ts] -->|start = Date.now| M[startActiveSpan callback]
    M -->|success| N[totalMs = Date.now - start]
    M -->|throws| X2[metrics SKIPPED]
    N --> O[recordVfsMaterialize per-phase + total]
    O --> E
    E --> P[OTLP to Mimir]
    P --> Q[Grafana Go+Sim union queries]

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Tool Call - executor.ts] -->|startedAt = Date.now| B[executeToolAndReportInner]
    B -->|success| C[durationMs = Date.now - startedAt]
    B -->|throws| X1[metric SKIPPED]
    C --> D[recordSimToolMetric name, outcome, durationMs]
    D --> E[metrics.ts instruments]
    E --> F[copilot.tool.calls counter]
    E --> G[copilot.tool.duration histogram]
    H[readFileRecord - file-reader.ts] -->|startedAt = Date.now| I[startActiveSpan callback]
    I -->|null on internal error| J[recordFileRead read_failed record.size]
    I -->|non-null result| K[recordFileRead success record.size]
    K -->|input size not output| E
    L[materializeVFS - workspace-vfs.ts] -->|start = Date.now| M[startActiveSpan callback]
    M -->|success| N[totalMs = Date.now - start]
    M -->|throws| X2[metrics SKIPPED]
    N --> O[recordVfsMaterialize per-phase + total]
    O --> E
    E --> P[OTLP to Mimir]
    P --> Q[Grafana Go+Sim union queries]

_{Reviews (1): Last reviewed commit: "fix(lint): fix lint" | Re-trigger Greptile}

fix(lint): fix lint

6f2c355

vercel Bot deployed to Preview June 15, 2026 22:12 View deployment

vercel Bot temporarily deployed to Preview June 15, 2026 22:13 Inactive

greptile-apps Bot reviewed Jun 15, 2026

View reviewed changes

Comment thread apps/sim/lib/copilot/vfs/workspace-vfs.ts Outdated

Comment thread apps/sim/lib/copilot/vfs/file-reader.ts

Sg312 changed the title ~~feat(copilot): server-side mothership tool/vfs/file metrics + $env ta…~~ feat(copilot): server-side mothership tool/vfs/file metrics Jun 15, 2026

fix(vfs): add error on vfs materialize fail

b27e4c2

vercel Bot temporarily deployed to Preview June 15, 2026 22:40 Inactive

improvement(mship): add user metadata to mship request

7803f47

vercel Bot temporarily deployed to Preview June 15, 2026 23:18 Inactive

Sg312 merged commit d538b76 into staging Jun 15, 2026
27 checks passed

waleedlatif1 mentioned this pull request Jun 16, 2026

v0.7.8: security hardening, code hygiene, MSFT oauth provider, new atlassian assets and google calendar tools #5088

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(copilot): server-side mothership tool/vfs/file metrics#5071

feat(copilot): server-side mothership tool/vfs/file metrics#5071
Sg312 merged 4 commits into
stagingfrom
dev

Sg312 commented Jun 15, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Sg312 commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 15, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Sg312 commented Jun 15, 2026 •

edited

Loading

vercel Bot commented Jun 15, 2026 •

edited

Loading