Skip to content

feat(webapp): split Models into Your models and Model library tabs#3958

Open
ericallam wants to merge 5 commits into
mainfrom
feature/tri-10025-on-the-models-page-create-a-new-table-for-your-models
Open

feat(webapp): split Models into Your models and Model library tabs#3958
ericallam wants to merge 5 commits into
mainfrom
feature/tri-10025-on-the-models-page-create-a-new-table-for-your-models

Conversation

@ericallam

@ericallam ericallam commented Jun 15, 2026

Copy link
Copy Markdown
Member

Summary

The Models page is now split into two tabs. Your models shows the models your project has actually used in the selected time range, with usage charts (cost over time, tokens over time, calls by model), a per-model table of calls / cost / avg TTFC / avg tokens-per-sec, and calls/tokens trend sparklines. Model library is the full catalog, reordered from alphabetical to a relevance-based provider order (Anthropic, OpenAI, Google, then the rest), newest models first within each provider, with a "New" badge on models released in the last 7 days.

One time-range selector drives the whole Your models tab, so the charts, the table, and the sparklines all share the same window. Opening a model shows its own metrics with an independent range picker and a "View in AI metrics" link that opens the AI metrics dashboard filtered to that model. The active tab is kept in the URL so it survives a refresh and is shareable.

Notes

Also fixes React "invalid DOM property" console warnings from the provider icons (the Llama and DeepSeek SVGs used raw fill-rule / clip-rule / clip-path attributes), which this page surfaces by rendering more provider icons.

Screenshots

Your models tab: usage charts and a per-model table with calls/tokens trend sparklines.

1-your-models-tab

Model library: provider-relevance ordering with a "New" badge on models released in the last 7 days.

2-model-library-tab

Model detail, Metrics tab: per-model range picker and a "View in AI metrics" link.

3-model-detail-metrics

View in AI metrics: the dashboard deep-linked and filtered to the selected model.

4-ai-metrics-filtered

@changeset-bot

changeset-bot Bot commented Jun 15, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e372c8b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The PR adds a "Your models" tab to the Models page with project-scoped usage metrics and prompt-cache insights. SVG provider icons are updated to use React camelCase attributes. A new UsageSparkline component renders recharts bar charts with portal tooltips. ModelRegistryPresenter adds provider-ranked catalog ordering, two ClickHouse queries (getProjectModelUsage, getModelUsageSparklines) with adaptive interval bucketing, and supporting Zod schemas. The route loader now parses time-window URL params, concurrently fetches project usage and sparkline data, and exports shouldRevalidate to skip refetches on tab-only navigation. The AI metrics dashboard gains a caching section with multiple widgets for cache hit rate, cached tokens, and estimated savings. The UI adds a tabbed layout with YourModelsTab containing usage charts and a sparkline table, updates ModelDetailPanel with deep-link support via aiMetricsBasePath and new caching metric widgets, adds a "New" badge for recently released models, and includes a modelLookup memo for variant-to-catalog mapping.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 46.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive The description comprehensively covers objectives, changes, features, and includes screenshots. However, it lacks the required checklist and testing/changelog sections from the template. Complete the PR description template by adding the checklist items, testing steps, and changelog sections as specified in the repository template.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: splitting the Models page into two tabs (Your models and Model library).
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/tri-10025-on-the-models-page-create-a-new-table-for-your-models

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint install timed out. The project may have too many dependencies for the sandbox.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot]

This comment was marked as resolved.

@ericallam ericallam marked this pull request as ready for review June 15, 2026 17:02
devin-ai-integration[bot]

This comment was marked as resolved.

The Your models sparklines use dynamic bucket sizes (6h at 7d, etc.), but the
tooltip assumed hourly buckets and showed wrong dates. Thread the bucket
interval and start through so each bar is labelled correctly.

Also pin the library tab cross-tenant p50 TTFC column to a fixed 7-day window
so it no longer follows the Your models time selector.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

Comment on lines +102 to +111
function sparklineBucketKeys(from: Date, to: Date, intervalSeconds: number): string[] {
const intervalMs = intervalSeconds * 1000;
const start = Math.floor(from.getTime() / intervalMs) * intervalMs;
const end = Math.floor(to.getTime() / intervalMs) * intervalMs;
const keys: string[] = [];
for (let t = start; t <= end; t += intervalMs) {
keys.push(new Date(t).toISOString().slice(0, 19).replace("T", " "));
}
return keys;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Sparkline bucket key alignment depends on ClickHouse UTC session timezone

The sparklineBucketKeys function at apps/webapp/app/presenters/v3/ModelRegistryPresenter.server.ts:102-111 generates UTC-formatted bucket strings via toISOString().slice(0, 19).replace('T', ' '). These must exactly match the string representation of ClickHouse's toStartOfInterval(start_time, INTERVAL N SECOND) at line 742. Since toStartOfInterval with SECOND intervals is epoch-aligned, the numeric computation matches. However, the string serialization of the ClickHouse DateTime result depends on the session timezone. If a self-hosted ClickHouse instance uses a non-UTC timezone, the bucket strings won't match the JS-generated keys, and all sparklines will silently show zeros (every lookup falls through to ?? 0). The hosted Trigger.dev service almost certainly uses UTC, but this could bite self-hosters.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

query: `
SELECT
response_model,
toStartOfInterval(start_time, INTERVAL ${intervalSeconds} SECOND) AS bucket,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Sparkline bucket keys use timezone-dependent DateTime strings instead of epoch timestamps

The getModelUsageSparklines ClickHouse query at apps/webapp/app/presenters/v3/ModelRegistryPresenter.server.ts:742 returns toStartOfInterval(start_time, INTERVAL n SECOND) AS bucket as a raw DateTime string. The #buildSparklineMap method then matches these strings against JS-generated UTC keys from sparklineBucketKeys (apps/webapp/app/presenters/v3/ModelRegistryPresenter.server.ts:108). If the ClickHouse server timezone is not UTC, the DateTime strings from ClickHouse (in server-local time) will not match the UTC strings from JS, causing every rowMap.get() lookup to miss and all sparklines to silently display zeros.

Every other presenter in the codebase avoids this by wrapping in toUnixTimestamp() for timezone-independent numeric buckets:

  • AgentDetailPresenter.server.ts:152: toUnixTimestamp(toStartOfInterval(...))
  • TaskDetailPresenter.server.ts:202: toUnixTimestamp(toStartOfInterval(...))
  • internal-packages/clickhouse/src/errors.ts:308: toUnixTimestamp(toStartOfInterval(...))
Prompt for agents
The sparkline query in getModelUsageSparklines (ModelRegistryPresenter.server.ts line 742) uses raw toStartOfInterval output as a DateTime string for bucket keys. This is timezone-dependent and breaks if ClickHouse server timezone is not UTC. The fix should follow the established pattern used by AgentDetailPresenter, TaskDetailPresenter, and errors.ts: wrap the bucket expression in toUnixTimestamp() to produce timezone-independent numeric epoch seconds. Then update sparklineBucketKeys() to produce numeric keys (epoch seconds) instead of formatted UTC strings, and update #buildSparklineMap to match on numeric keys. Alternatively, the simpler fix is to pass an explicit UTC timezone to toStartOfInterval: toStartOfInterval(start_time, INTERVAL n SECOND, 'UTC').
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Your models gets a cache-savings column and per-model cached-tokens and
cache-hit-rate views; the AI metrics dashboard gets a caching section
(hit rate, cached tokens, estimated savings, hit rate by model). Also makes
the Your models charts all time-series for consistency.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 1642b874-6fd9-41c1-b4f5-d43cd4dd5592

📥 Commits

Reviewing files that changed from the base of the PR and between b5a7a56 and e372c8b.

📒 Files selected for processing (4)
  • .server-changes/models-page-usage-tabs.md
  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
  • apps/webapp/app/presenters/v3/ModelRegistryPresenter.server.ts
  • apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.models._index/route.tsx
✅ Files skipped from review due to trivial changes (1)
  • .server-changes/models-page-usage-tabs.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • apps/webapp/app/presenters/v3/ModelRegistryPresenter.server.ts
  • apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.models._index/route.tsx
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (12)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 10)
  • GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 10)
  • GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp
  • GitHub Check: typecheck / typecheck
🧰 Additional context used
📓 Path-based instructions (7)
**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead

Import from @trigger.dev/sdk when writing Trigger.dev tasks. Never use @trigger.dev/sdk/v3 or deprecated client.defineJob

Files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
{packages/core,apps/webapp}/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use zod for validation in packages/core and apps/webapp

Files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use function declarations instead of default exports

**/*.{ts,tsx,js,jsx}: Prefer static imports over dynamic imports. Only use dynamic import() when circular dependencies cannot be resolved, code splitting is needed for performance, or the module must be loaded conditionally at runtime
Import subpaths only from packages/core (@trigger.dev/core), never import from the root

Files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
**/*.ts

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

**/*.ts: When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs
Do not use high-cardinality attributes in OTEL metrics such as UUIDs/IDs (envId, userId, runId, projectId, organizationId), unbounded integers (itemCount, batchSize, retryCount), timestamps (createdAt, startTime), or free-form strings (errorMessage, taskName, queueName)
When exporting OTEL metrics via OTLP to Prometheus, be aware that the exporter automatically adds unit suffixes to metric names (e.g., 'my_duration_ms' becomes 'my_duration_ms_milliseconds', 'my_counter' becomes 'my_counter_total'). Account for these transformations when writing Grafana dashboards or Prometheus queries

Files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
apps/webapp/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)

apps/webapp/**/*.{ts,tsx}: Access environment variables through the env export of env.server.ts instead of directly accessing process.env
Use subpath exports from @trigger.dev/core package instead of importing from the root @trigger.dev/core path

Use named constants for sentinel/placeholder values (e.g. const UNSET_VALUE = '__unset__') instead of raw string literals scattered across comparisons

Files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
apps/webapp/**/*.server.ts

📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md)

apps/webapp/**/*.server.ts: Never use request.signal for detecting client disconnects. Use getRequestAbortSignal() from app/services/httpAsyncStorage.server.ts instead, which is wired directly to Express res.on('close') and fires reliably
Access environment variables via env export from app/env.server.ts. Never use process.env directly
Always use findFirst instead of findUnique in Prisma queries. findUnique has an implicit DataLoader that batches concurrent calls and has active bugs even in Prisma 6.x (uppercase UUIDs returning null, composite key SQL correctness issues, 5-10x worse performance). findFirst is never batched and avoids this entire class of issues

Files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
**/*.{js,ts,tsx,jsx,css,json,md}

📄 CodeRabbit inference engine (AGENTS.md)

Use Prettier for code formatting and run pnpm run format before committing

Files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
🧠 Learnings (10)
📚 Learning: 2026-02-06T19:53:38.843Z
Learnt from: 0ski
Repo: triggerdotdev/trigger.dev PR: 2994
File: apps/webapp/app/presenters/v3/DeploymentListPresenter.server.ts:233-237
Timestamp: 2026-02-06T19:53:38.843Z
Learning: When constructing Vercel dashboard URLs from deployment IDs, always strip the dpl_ prefix from the ID. Implement this by transforming the ID with .replace(/^dpl_/, "") before concatenating into the URL: https://vercel.com/${teamSlug}/${projectName}/${cleanedDeploymentId}. Consider centralizing this logic in a small helper (e.g., getVercelDeploymentId(id) or a URL builder) and add tests to verify both prefixed and non-prefixed inputs.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-03-22T13:26:12.060Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3244
File: apps/webapp/app/components/code/TextEditor.tsx:81-86
Timestamp: 2026-03-22T13:26:12.060Z
Learning: In the triggerdotdev/trigger.dev codebase, do not flag `navigator.clipboard.writeText(...)` calls for `missing-await`/`unhandled-promise` issues. These clipboard writes are intentionally invoked without `await` and without `catch` handlers across the project; keep that behavior consistent when reviewing TypeScript/TSX files (e.g., usages like in `apps/webapp/app/components/code/TextEditor.tsx`).

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-03-22T19:24:14.403Z
Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 3187
File: apps/webapp/app/v3/services/alerts/deliverErrorGroupAlert.server.ts:200-204
Timestamp: 2026-03-22T19:24:14.403Z
Learning: In the triggerdotdev/trigger.dev codebase, webhook URLs are not expected to contain embedded credentials/secrets (e.g., fields like `ProjectAlertWebhookProperties` should only hold credential-free webhook endpoints). During code review, if you see logging or inclusion of raw webhook URLs in error messages, do not automatically treat it as a credential-leak/secrets-in-logs issue by default—first verify the URL does not contain embedded credentials (for example, no username/password in the URL, no obvious secret/token query params or fragments). If the URL is credential-free per this project’s conventions, allow the logging.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-05-18T08:21:27.694Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma error P1001 ("Can't reach database server") in TypeScript, don’t assume a single error shape. Prisma can surface P1001 via two different error classes/fields: `PrismaClientKnownRequestError` exposes it as `err.code === "P1001"` (common during mid-query connection drops), while `PrismaClientInitializationError` exposes it as `err.errorCode === "P1001"` (common on client startup failure). Therefore, predicates should use `err.code === "P1001" || err.errorCode === "P1001"`. Do not flag `err.code === "P1001"` as “unreachable/never matches,” as it is expected in production.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-05-18T08:21:27.694Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma errors for P1001 ("Can't reach database server"), do not assume it only appears under a single property name. Prisma may surface P1001 via either `PrismaClientKnownRequestError` (`err.code === "P1001"`, e.g., mid-query connection drops) or `PrismaClientInitializationError` (`err.errorCode === "P1001"`, e.g., client startup connection failure). To reliably detect the condition, check `err.code === "P1001" || err.errorCode === "P1001"`, and avoid review rules that would incorrectly flag `err.code === "P1001"` as unreachable/never-matching.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-06-13T19:53:13.759Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3937
File: packages/trigger-sdk/skills/realtime-and-frontend/SKILL.md:258-260
Timestamp: 2026-06-13T19:53:13.759Z
Learning: When reviewing code that uses `trigger.dev/react-hooks`’s `useRealtimeRun`, preserve the call signature where the first argument is the full realtime handle object (not `handle.id`). This is intentional to maintain type-safety and is consistent with the official docs; do not suggest changing the first argument from the handle object to `handle.id`.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-05-05T09:38:02.512Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3523
File: apps/webapp/app/routes/api.v3.batches.ts:178-181
Timestamp: 2026-05-05T09:38:02.512Z
Learning: When reviewing code that catches `ServiceValidationError` in `*.server.ts` files, do not blindly forward `error.status` to HTTP responses, because SVEs may be thrown with non-default statuses (e.g., 400/500) and forwarding them can cause client-visible behavioral regressions (e.g., surfacing 500s to clients). Prefer a safe default response status of `error.status ?? 422`, but only after confirming via the reachable call graph that the caught `ServiceValidationError` instances are expected to carry those non-default statuses; otherwise, normalize to `422` to avoid unexpected client-visible 5xx behavior.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-05-12T21:04:05.815Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3542
File: apps/webapp/app/components/sessions/v1/SessionStatus.tsx:1-3
Timestamp: 2026-05-12T21:04:05.815Z
Learning: In this Remix + TypeScript codebase, do not flag a server/client boundary violation when a file imports only types from a module matching `*.server`.

Specifically, it’s safe to import types using `import type { Foo } from "*.server"` or `import { type Foo } from "*.server"` because TypeScript erases type-only imports at compile time and they emit no JavaScript, so they won’t cross the Remix server/client bundle boundary.

Only raise the boundary concern for value imports (e.g., `import { Foo }` without `type`, or `import Foo`), since those produce JavaScript output.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-06-04T18:16:35.386Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3836
File: apps/supervisor/src/backpressure/backpressureMonitor.ts:3-5
Timestamp: 2026-06-04T18:16:35.386Z
Learning: When reviewing TypeScript in this repo, apply the rule “prefer type aliases over interfaces” only to data/object shapes and union/intersection type modeling. If an interface is being used as a behavioral contract for collaborators to implement (e.g., method-shape interfaces that define required behavior, such as `BackpressureLogger` / `BackpressureSignalSource` in `apps/supervisor/src/backpressure/backpressureMonitor.ts`), keep it as an `interface` and do not flag it as a type-alias-vs-interface violation.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts
📚 Learning: 2026-06-09T17:58:04.699Z
Learnt from: 0ski
Repo: triggerdotdev/trigger.dev PR: 3879
File: apps/webapp/app/models/vercelIntegration.server.ts:619-630
Timestamp: 2026-06-09T17:58:04.699Z
Learning: In this codebase, outbound raw `fetch` calls should typically rely on Node/undici’s default request timeout (about ~300s) rather than adding a per-call `AbortController` + `setTimeout` wrapper inside individual functions (e.g. in files like `apps/webapp/app/models/vercelIntegration.server.ts`). During code review, do not flag the absence of a per-call timeout on a single `fetch` as an issue; if per-call timeouts are needed, they should be implemented via a codebase-wide convention (e.g., a shared fetch wrapper or documented pattern) rather than ad-hoc per-function changes.

Applied to files:

  • apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts

title: "Cache hit rate over time",
query:
"SELECT\r\n timeBucket(),\r\n round(countIf(cached_read_tokens > 0) * 100.0 / count(), 1) AS cache_hit_pct,\r\n round(avg(cached_read_tokens), 0) AS avg_cached_tokens\r\nFROM\r\n llm_metrics\r\nGROUP BY\r\n timeBucket\r\nORDER BY\r\n timeBucket",
"SELECT timeBucket(), round(sum(cached_read_tokens) * 100.0 / (sum(input_tokens) + sum(cached_read_tokens)), 1) AS cache_hit_pct FROM llm_metrics GROUP BY timeBucket ORDER BY timeBucket",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

fd -t f "BuiltInDashboards.server.ts" apps/webapp

Repository: triggerdotdev/trigger.dev

Length of output: 127


🏁 Script executed:

cat -n apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts | sed -n '495,550p'

Repository: triggerdotdev/trigger.dev

Length of output: 2832


🏁 Script executed:

grep -r "ClickHouse\|llm_metrics\|timeBucket" apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts | head -20

Repository: triggerdotdev/trigger.dev

Length of output: 4239


🏁 Script executed:

cat -n apps/webapp/app/presenters/v3/BuiltInDashboards.server.ts | head -50

Repository: triggerdotdev/trigger.dev

Length of output: 2359


🏁 Script executed:

rg "nullIf|ifNull" apps/webapp --type ts -A 2 -B 2 | head -30

Repository: triggerdotdev/trigger.dev

Length of output: 1836


Normalize denominator handling in cache metric queries to prevent division-by-zero errors.

Lines 499 and 547 divide by sum(input_tokens) + sum(cached_read_tokens) without guarding against zero, while line 531 uses the + 1 sentinel workaround, which introduces bias when the denominator would be zero. Use consistent nullIf() / ifNull() handling across all three cache metrics to ensure they remain mathematically stable.

Suggested query adjustments
- "SELECT timeBucket(), round(sum(cached_read_tokens) * 100.0 / (sum(input_tokens) + sum(cached_read_tokens)), 1) AS cache_hit_pct FROM llm_metrics GROUP BY timeBucket ORDER BY timeBucket",
+ "SELECT timeBucket(), round(ifNull(sum(cached_read_tokens) * 100.0 / nullIf(sum(input_tokens) + sum(cached_read_tokens), 0), 0), 1) AS cache_hit_pct FROM llm_metrics GROUP BY timeBucket ORDER BY timeBucket",

- "SELECT timeBucket(), round(sum(cached_read_tokens) * (sum(input_cost) / (sum(input_tokens) + 1)) - sum(cached_read_cost), 4) AS cache_savings FROM llm_metrics WHERE cached_read_tokens > 0 GROUP BY timeBucket ORDER BY timeBucket",
+ "SELECT timeBucket(), round(ifNull(sum(cached_read_tokens) * (sum(input_cost) / nullIf(sum(input_tokens), 0)) - sum(cached_read_cost), 0), 4) AS cache_savings FROM llm_metrics WHERE cached_read_tokens > 0 GROUP BY timeBucket ORDER BY timeBucket",

- "SELECT response_model, round(sum(cached_read_tokens) * 100.0 / (sum(input_tokens) + sum(cached_read_tokens)), 1) AS cache_hit_pct, sum(cached_read_tokens) AS cached_tokens FROM llm_metrics GROUP BY response_model ORDER BY cached_tokens DESC LIMIT 20",
+ "SELECT response_model, round(ifNull(sum(cached_read_tokens) * 100.0 / nullIf(sum(input_tokens) + sum(cached_read_tokens), 0), 0), 1) AS cache_hit_pct, sum(cached_read_tokens) AS cached_tokens FROM llm_metrics GROUP BY response_model ORDER BY cached_tokens DESC LIMIT 20",

Also applies to: 531-531, 547-547

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

title: "Cache hit rate over time",
query:
"SELECT\r\n timeBucket(),\r\n round(countIf(cached_read_tokens > 0) * 100.0 / count(), 1) AS cache_hit_pct,\r\n round(avg(cached_read_tokens), 0) AS avg_cached_tokens\r\nFROM\r\n llm_metrics\r\nGROUP BY\r\n timeBucket\r\nORDER BY\r\n timeBucket",
"SELECT timeBucket(), round(sum(cached_read_tokens) * 100.0 / (sum(input_tokens) + sum(cached_read_tokens)), 1) AS cache_hit_pct FROM llm_metrics GROUP BY timeBucket ORDER BY timeBucket",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Division by zero in cache hit rate SQL queries produces NaN, replacing a previously safe formula

The new llm-cache-hit and llm-cache-by-model dashboard widgets, as well as the model detail panel's cache hit rate query (route.tsx:1174), all use sum(cached_read_tokens) * 100.0 / (sum(input_tokens) + sum(cached_read_tokens)). When a time bucket or model group has only rows where both input_tokens = 0 and cached_read_tokens = 0 (e.g. errored calls with no token counts), the denominator is 0, producing NaN in ClickHouse which can break chart rendering.

The old llm-cache-util widget at BuiltInDashboards.server.ts:493 used countIf(cached_read_tokens > 0) * 100.0 / count() which was safe since count() is always > 0. The llm-cache-savings widget correctly uses sum(input_tokens) + 1 as a denominator guard, but the cache-hit-rate queries lack any such protection.

Suggested change
"SELECT timeBucket(), round(sum(cached_read_tokens) * 100.0 / (sum(input_tokens) + sum(cached_read_tokens)), 1) AS cache_hit_pct FROM llm_metrics GROUP BY timeBucket ORDER BY timeBucket",
"SELECT timeBucket(), round(sum(cached_read_tokens) * 100.0 / greatest(sum(input_tokens) + sum(cached_read_tokens), 1), 1) AS cache_hit_pct FROM llm_metrics GROUP BY timeBucket ORDER BY timeBucket",
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant