improvement(logs): cleanup code#999
Merged
Merged
Conversation
v0.3.20: KB Tag fixes
…ypes (#919) * feat(execution-filesystem): system to pass files between blocks (#866) * feat(files): pass files between blocks * presigned URL for downloads * Remove latest migration before merge * starter block file upload wasn't getting logged * checkpoint in human readable form * checkpoint files / file type outputs * file downloads working for block outputs * checkpoint file download * fix type issues * remove filereference interface with simpler user file interface * show files in the tag dropdown for start block * more migration to simple url object, reduce presigned time to 5 min * Remove migration 0065_parallel_nightmare and related files - Deleted apps/sim/db/migrations/0065_parallel_nightmare.sql - Deleted apps/sim/db/migrations/meta/0065_snapshot.json - Removed 0065 entry from apps/sim/db/migrations/meta/_journal.json Preparing for merge with origin/staging and migration regeneration * add migration files * fix tests * Update apps/sim/lib/uploads/setup.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update apps/sim/lib/workflows/execution-file-storage.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update apps/sim/lib/workflows/execution-file-storage.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * cleanup types * fix lint * fix logs typing for file refs * open download in new tab * fixed * Update apps/sim/tools/index.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix file block * cleanup unused code * fix bugs * remove hacky file id logic * fix drag and drop * fix tests --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(trigger-mode): added trigger-mode to workflow_blocks table (#902) * fix(schedules-perms): use regular perm system to view/edit schedule info (#901) * fix(schedules-perms): use regular perm system to view schedule info * fix perms * improve logging * feat(webhooks): deprecate singular webhook block + add trigger mode to blocks (#903) * feat(triggers): added new trigger mode for blocks, added socket event, ran migrations * Rename old trigger/ directory to background/ * cleaned up, ensured that we display active webhook at the block-level * fix submenu in tag dropdown * keyboard nav on tag dropdown submenu * feat(triggers): add outlook to new triggers system * cleanup * add types to tag dropdown, type all outputs for tools and use that over block outputs * update doc generator to truly reflect outputs * fix docs * add trigger handler * fix active webhook tag * tag dropdown fix for triggers * remove trigger mode schema change * feat(execution-filesystem): system to pass files between blocks (#866) * feat(files): pass files between blocks * presigned URL for downloads * Remove latest migration before merge * starter block file upload wasn't getting logged * checkpoint in human readable form * checkpoint files / file type outputs * file downloads working for block outputs * checkpoint file download * fix type issues * remove filereference interface with simpler user file interface * show files in the tag dropdown for start block * more migration to simple url object, reduce presigned time to 5 min * Remove migration 0065_parallel_nightmare and related files - Deleted apps/sim/db/migrations/0065_parallel_nightmare.sql - Deleted apps/sim/db/migrations/meta/0065_snapshot.json - Removed 0065 entry from apps/sim/db/migrations/meta/_journal.json Preparing for merge with origin/staging and migration regeneration * add migration files * fix tests * Update apps/sim/lib/uploads/setup.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update apps/sim/lib/workflows/execution-file-storage.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update apps/sim/lib/workflows/execution-file-storage.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * cleanup types * fix lint * fix logs typing for file refs * open download in new tab * fixed * Update apps/sim/tools/index.ts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix file block * cleanup unused code * fix bugs * remove hacky file id logic * fix drag and drop * fix tests --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(trigger-mode): added trigger-mode to workflow_blocks table (#902) * fix(schedules-perms): use regular perm system to view/edit schedule info (#901) * fix(schedules-perms): use regular perm system to view schedule info * fix perms * improve logging * cleanup * prevent tooltip showing up on modal open * updated trigger config * fix type issues --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> * fix(helm): fix helm charts migrations using wrong image (#907) * fix(helm): fix helm charts migrations using wrong image * fixed migrations * feat(whitelist): add email & domain-based whitelisting for signups (#908) * improvement(helm): fix duplicate SOCKET_SERVER_URL and add additional envvars to template (#909) * improvement(helm): fix duplicate SOCKET_SERVER_URL and add additional envvars to template * rm serper & freestyle * improvement(tag-dropdown): typed tag dropdown values (#910) * fix(min-chunk): remove minsize for chunk (#911) * fix(min-chunk): remove minsize for chunk * fix tests * improvement(chunk-config): migrate unused default for consistency (#913) * fix(mailer): update mailer to use the EMAIL_DOMAIN (#914) * fix(mailer): update mailer to use the EMAIL_DOMAIn * add more * Improvement(cc): added cc to gmail and outlook (#900) * changed just gmail * bun run lint * fixed bcc * updated docs --------- Co-authored-by: Adam Gough <adamgough@Mac.attlocal.net> Co-authored-by: waleedlatif1 <walif6@gmail.com> * fix(email-validation): add email validation to prevent bouncing, fixed OTP validation (#916) * feat(email-validation): add email validation to prevent bouncing * removed suspicious patterns * fix(verification): fixed OTP verification * fix failing tests, cleanup * fix(otp): fix email not sending (#917) * fix(email): manual OTP instead of better-auth (#921) * fix(email): manual OTP instead of better-auth * lint --------- Co-authored-by: Vikhyath Mondreti <vikhyathvikku@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai> Co-authored-by: Adam Gough <77861281+aadamgough@users.noreply.github.com> Co-authored-by: Adam Gough <adamgough@Mac.attlocal.net>
…ers fixes v0.3.23: multiplayer variables, api key fixes, kb improvements, triggers fixes
v0.3.24: revert redis session management change
v0.3.25: oauth credentials sharing mechanism, workflow block error handling changes
…curity improvements v0.3.26: fix billing, bubble up workflow block errors, credentials security improvements
v0.3.27: oauth/webhook fixes, whitelabel fixes, code cleanups
attempt to fix build issues (#985)
Contributor
There was a problem hiding this comment.
Greptile Summary
This PR refactors the workflow execution logging system by consolidating and simplifying the database schema for better performance and maintainability. The changes rename the generic metadata column to executionData for better semantic clarity about its content (traceSpans and execution context). Multiple fragmented cost-related columns (total_cost, total_input_cost, total_output_cost, total_tokens) are consolidated into a single cost JSONB column that provides structured storage for complex cost breakdowns by model and token type.
The refactoring removes unused columns including block statistics (block_count, success_count, error_count, skipped_count) and the message column, reducing database bloat and schema complexity. This supports the implementation of lazy loading architecture where the logs list page fetches minimal column data for fast rendering, while detailed cost and execution information is only loaded when users open the sidebar for specific log entries.
The migration (0076_damp_vector) includes sophisticated data transformation logic using recursive CTEs to aggregate cost information from multiple sources (legacy columns, traceSpans, tokenBreakdown) into the new unified structure. The migration is designed to be safe for reruns with conditional column operations and comprehensive data preservation during the transition.
Confidence score: 3/5
- This PR involves complex database schema changes and data migration that could potentially cause issues if the migration logic has edge cases or performance problems
- Score reflects the sophisticated but risky nature of the recursive CTE migration logic and the potential for data loss or corruption during complex transformations
- Pay close attention to
apps/sim/db/migrations/0076_damp_vector.sql- the migration contains complex recursive queries and bulk operations that need thorough testing in production-like environments
4 files reviewed, 1 comment
5 tasks
arenadeveloper02
pushed a commit
to arenadeveloper02/p2-sim
that referenced
this pull request
Sep 19, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactor logs code:
Type of Change
Testing
Tested creating new logs. Tested custom migration with dry run script to migrate old logs.
Checklist