Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
6386e6b
updates
lakeesiv Jan 13, 2026
5dddb03
required
lakeesiv Jan 13, 2026
e80660f
trashy table viewer
lakeesiv Jan 13, 2026
4316f45
updates
lakeesiv Jan 13, 2026
7e4fc32
updates
lakeesiv Jan 13, 2026
0872314
filtering ui
lakeesiv Jan 14, 2026
9a3d563
updates
lakeesiv Jan 14, 2026
48ecb19
updates
lakeesiv Jan 14, 2026
ed807be
updates
lakeesiv Jan 14, 2026
8a8589e
one input mode
lakeesiv Jan 14, 2026
a919816
format
lakeesiv Jan 14, 2026
6605c88
fix lints
lakeesiv Jan 14, 2026
c1eef30
improved errors
lakeesiv Jan 14, 2026
a537ca7
updates
lakeesiv Jan 14, 2026
fc6dbcf
updates
lakeesiv Jan 14, 2026
48250f5
chages
lakeesiv Jan 14, 2026
c155d8a
doc strings
lakeesiv Jan 14, 2026
4d176c0
breaking down file
lakeesiv Jan 14, 2026
e287388
update comments with ai
lakeesiv Jan 14, 2026
dfa018f
updates
lakeesiv Jan 14, 2026
22f89cf
comments
lakeesiv Jan 14, 2026
8f9cf93
changes
lakeesiv Jan 14, 2026
4422a69
revert
lakeesiv Jan 15, 2026
15bef48
updates
lakeesiv Jan 15, 2026
cfbc8d7
dedupe
lakeesiv Jan 15, 2026
b3ca0c9
updates
lakeesiv Jan 15, 2026
df3e869
updates
lakeesiv Jan 15, 2026
96a3fe5
updates
lakeesiv Jan 15, 2026
cbb93c6
refactoring
lakeesiv Jan 15, 2026
c9373c7
renames & refactors
lakeesiv Jan 15, 2026
b08ce03
refactoring
lakeesiv Jan 15, 2026
ffad20e
updates
lakeesiv Jan 15, 2026
793c888
undo
lakeesiv Jan 15, 2026
c3afbae
update db
lakeesiv Jan 15, 2026
5a69d16
wand
lakeesiv Jan 15, 2026
fdc3af9
updates
lakeesiv Jan 15, 2026
80270ce
fix comments
lakeesiv Jan 15, 2026
57fbd2a
fixes
lakeesiv Jan 15, 2026
7f894ec
simplify comments
lakeesiv Jan 15, 2026
ed543a7
u[dates
lakeesiv Jan 15, 2026
e503408
renames
lakeesiv Jan 15, 2026
0a6312d
better comments
lakeesiv Jan 16, 2026
4665595
validation
lakeesiv Jan 16, 2026
d00997c
updates
lakeesiv Jan 16, 2026
cfffd05
updates
lakeesiv Jan 16, 2026
1a13762
updates
lakeesiv Jan 16, 2026
4490996
fix sorting
lakeesiv Jan 16, 2026
fef2d2c
fix appearnce
lakeesiv Jan 16, 2026
c94bb5a
updating prompt to make it user sort
lakeesiv Jan 16, 2026
e695007
rm
lakeesiv Jan 16, 2026
a940dd6
updates
lakeesiv Jan 16, 2026
271375d
rename
lakeesiv Jan 16, 2026
26d9662
comments
lakeesiv Jan 16, 2026
5173320
clean comments
lakeesiv Jan 16, 2026
ea72ab5
simplicifcaiton
lakeesiv Jan 16, 2026
42aa794
updates
lakeesiv Jan 16, 2026
897891e
updates
lakeesiv Jan 16, 2026
7093209
refactor
lakeesiv Jan 16, 2026
2e624c2
reduced type confusion
lakeesiv Jan 16, 2026
f90c9c7
undo
lakeesiv Jan 16, 2026
abb671e
rename
lakeesiv Jan 16, 2026
448b8f0
undo changes
lakeesiv Jan 16, 2026
e4dd14d
undo
lakeesiv Jan 16, 2026
cca1772
simplify
lakeesiv Jan 16, 2026
86c5e1b
updates
lakeesiv Jan 16, 2026
94c6795
updates
lakeesiv Jan 16, 2026
3d81c1c
revert
lakeesiv Jan 16, 2026
51d1b95
updates
lakeesiv Jan 16, 2026
c662a31
db updates
lakeesiv Jan 16, 2026
e7f4516
type fix
lakeesiv Jan 16, 2026
4357230
fix
lakeesiv Jan 17, 2026
0e1133f
fix error handling
lakeesiv Jan 17, 2026
8955915
updates
lakeesiv Jan 17, 2026
ea77790
docs
lakeesiv Jan 17, 2026
292cd39
docs
lakeesiv Jan 17, 2026
118e4f6
updates
lakeesiv Jan 17, 2026
7ca628d
rename
lakeesiv Jan 17, 2026
37b50cb
dedupe
lakeesiv Jan 17, 2026
4b6de03
revert
lakeesiv Jan 17, 2026
e79e9e7
Merge origin/main into lakees/db
lakeesiv Jan 17, 2026
8574d66
uncook
lakeesiv Jan 17, 2026
d0c3c6a
updates
lakeesiv Jan 17, 2026
6e8dc77
fix
lakeesiv Jan 17, 2026
87f8fcd
fix
lakeesiv Jan 17, 2026
f05f5bb
fix
lakeesiv Jan 17, 2026
a8e413a
fix
lakeesiv Jan 17, 2026
107679b
prepare merge
icecrasher321 Jan 22, 2026
8d43947
Merge staging into lakees/db
icecrasher321 Jan 22, 2026
1f682eb
readd migrations
icecrasher321 Jan 22, 2026
2d49de7
add back missed code
icecrasher321 Jan 22, 2026
2818b74
migrate enrichment logic to general abstraction
icecrasher321 Jan 22, 2026
1938818
address bugbot concerns
icecrasher321 Jan 22, 2026
be757a4
adhere to size limits for tables
icecrasher321 Jan 22, 2026
d9dbe93
remove conflicting migration
icecrasher321 Feb 11, 2026
f0a8d82
Merge remote-tracking branch 'origin/staging' into lakees/db
icecrasher321 Feb 11, 2026
0ffcce8
add back migrations
icecrasher321 Feb 11, 2026
ca70280
fix tables auth
icecrasher321 Feb 11, 2026
8bf3370
fix permissive auth
icecrasher321 Feb 11, 2026
c4986a9
fix lint
icecrasher321 Feb 11, 2026
c18fbcf
Merge branch 'staging' into lakees/db
waleedlatif1 Feb 19, 2026
b476b8d
reran migrations
waleedlatif1 Feb 19, 2026
592dd46
migrate to use tanstack query for all server state
waleedlatif1 Feb 19, 2026
f1938f0
update table-selector
waleedlatif1 Feb 19, 2026
7b0e030
update names
waleedlatif1 Feb 19, 2026
407f9c7
added tables to permission groups, updated subblock types
waleedlatif1 Feb 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
updates
  • Loading branch information
lakeesiv committed Jan 17, 2026
commit 6386e6b43786ad8aab93c7a98513aa8963d29195
250 changes: 250 additions & 0 deletions PLAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
# Table Block Implementation Plan

> Create a new "table" block type that enables users to define schemas and perform CRUD operations on lightweight, workspace/workflow-scoped tables stored in the existing PostgreSQL database using JSONB with application-level schema enforcement.

## Table of Contents

- [Architecture Overview](#architecture-overview)
- [Data Model](#data-model)
- [Implementation Files](#implementation-files)
- [Key Design Decisions](#key-design-decisions)
- [Limits and Limitations](#limits-and-limitations)
- [Implementation Checklist](#implementation-checklist)

## Architecture Overview

```mermaid
flowchart TB
subgraph UI [Block UI Layer]
TableBlock[Table Block]
SchemaEditor[Schema Editor SubBlock]
end

subgraph Tools [Tool Layer]
CreateTable[table_create]
Insert[table_insert]
Select[table_select]
Update[table_update]
Delete[table_delete]
DropTable[table_drop]
end

subgraph API [API Routes]
TableAPI["/api/tables"]
RowAPI["/api/tables/tableId/rows"]
end

subgraph DB [PostgreSQL]
SimTable[sim_table]
SimTableRow[sim_table_row]
end

TableBlock --> Tools
Tools --> API
API --> DB
```

## Data Model

Two new tables in the existing PostgreSQL database:

### `sim_table` - Table Definitions

| Column | Type | Description |
|--------|------|-------------|
| id | text | Primary key |
| workspace_id | text | FK to workspace |
| workflow_id | text | FK to workflow (nullable for workspace-scope) |
| name | text | Table name (unique per scope) |
| schema | jsonb | Column definitions with types/constraints |
| created_by | text | FK to user |
| created_at | timestamp | Creation time |
| updated_at | timestamp | Last update time |

### `sim_table_row` - Row Data

| Column | Type | Description |
|--------|------|-------------|
| id | text | Primary key |
| table_id | text | FK to sim_table |
| data | jsonb | Row data (validated against schema) |
| created_at | timestamp | Creation time |
| updated_at | timestamp | Last update time |

### Schema Format

**Example schema definition:**

```json
{
"columns": [
{ "name": "id", "type": "string", "primaryKey": true },
{ "name": "email", "type": "string", "required": true, "unique": true },
{ "name": "age", "type": "number" },
{ "name": "active", "type": "boolean", "default": true }
]
}
```

**Supported Types:** `string`, `number`, `boolean`, `date`, `json`

## Implementation Files

### 1. Database Schema

- `packages/db/schema.ts` - Add `simTable` and `simTableRow` table definitions
- Generate migration for the new tables

### 2. Tools (`apps/sim/tools/table/`)

| File | Purpose |
|------|---------|
| `types.ts` | Type definitions for params/responses |
| `create.ts` | Create table with schema |
| `insert.ts` | Insert row(s) with schema validation |
| `select.ts` | Query rows with filtering |
| `update.ts` | Update rows with schema validation |
| `delete.ts` | Delete rows |
| `drop.ts` | Drop table |
| `index.ts` | Barrel export |

### 3. Block Definition

- `apps/sim/blocks/blocks/table.ts` - Block config with:
- Operation dropdown (create, insert, select, update, delete, drop)
- Scope selector (workspace/workflow)
- Table selector (for existing tables)
- Schema editor (for create operation)
- Data/query inputs (operation-specific)

### 4. API Routes

- `apps/sim/app/api/tables/route.ts` - Create table, list tables
- `apps/sim/app/api/tables/[tableId]/route.ts` - Get/drop table
- `apps/sim/app/api/tables/[tableId]/rows/route.ts` - CRUD on rows

### 5. Schema Validation Library

- `apps/sim/lib/tables/schema.ts` - Schema validation utilities
- `apps/sim/lib/tables/types.ts` - Shared types

## Key Design Decisions

1. **Schema Enforcement**: Application-layer validation before database writes. JSONB stores data, but every insert/update validates against the table's schema.

2. **Concurrency**: PostgreSQL handles concurrent reads/writes natively. Row-level locking for updates.

3. **Indexing**: GIN index on `data` column for efficient JSONB queries. Additional indexes on `table_id` for fast row lookups.

4. **Scope Resolution**: Tables with `workflow_id = NULL` are workspace-scoped. Tables with `workflow_id` set are workflow-scoped.

5. **Table Selector**: New SubBlock type `table-selector` that fetches available tables based on current workspace/workflow context.

## Limits and Limitations

### Table Limits

| Limit | Free Plan | Pro Plan | Enterprise |
|-------|-----------|----------|------------|
| Tables per workspace | 10 | 50 | Unlimited |
| Tables per workflow | 5 | 20 | Unlimited |
| Columns per table | 50 | 100 | 200 |

### Row Limits

| Limit | Free Plan | Pro Plan | Enterprise |
|-------|-----------|----------|------------|
| Rows per table | 10,000 | 100,000 | 1,000,000 |
| Batch insert size | 100 | 500 | 1,000 |
| Batch update/delete size | 100 | 500 | 1,000 |

### Size Limits

| Limit | Value | Rationale |
|-------|-------|-----------|
| Column name length | 64 chars | PostgreSQL identifier limit |
| Table name length | 64 chars | PostgreSQL identifier limit |
| String field max length | 65,535 chars | ~64KB per text field |
| JSON field max size | 1 MB | PostgreSQL JSONB practical limit |
| Single row max size | 2 MB | Reasonable row size limit |
| Total table data size | Based on plan | Tied to workspace storage quota |

### Query Limits

| Limit | Value | Notes |
|-------|-------|-------|
| Default page size | 100 rows | Can be overridden up to max |
| Max page size | 1,000 rows | Prevents memory issues |
| Max filter conditions | 20 | AND/OR conditions combined |
| Query timeout | 30 seconds | Prevents long-running queries |
| Max concurrent queries per table | 50 | Rate limiting per table |

### Schema Constraints

| Constraint | Limit |
|------------|-------|
| Primary key columns | 1 (single column only) |
| Unique constraints | 5 per table |
| Required (NOT NULL) columns | Unlimited |
| Default values | Supported for all types |
| Foreign keys | Not supported (v1) |
| Computed columns | Not supported (v1) |
| Indexes | Auto-created for primary key and unique columns |

### Data Type Specifications

| Type | Storage | Min | Max | Notes |
|------|---------|-----|-----|-------|
| `string` | text | 0 chars | 65,535 chars | UTF-8 encoded |
| `number` | double precision | -1.7e308 | 1.7e308 | IEEE 754 double |
| `boolean` | boolean | - | - | true/false |
| `date` | timestamp | 4713 BC | 294276 AD | ISO 8601 format |
| `json` | jsonb | - | 1 MB | Nested objects/arrays |

### Operational Limitations

1. **No Transactions Across Tables**: Each operation is atomic to a single table. Cross-table transactions are not supported.

2. **No JOINs**: Cannot join data between tables. Use workflow logic to combine data from multiple tables.

3. **No Triggers/Hooks**: No automatic actions on insert/update/delete. Use workflow blocks for reactive logic.

4. **No Full-Text Search**: Basic filtering only. For full-text search, use the Knowledge Base feature.

5. **No Schema Migrations**: Schema changes require dropping and recreating the table (with data loss). Future versions may support additive migrations.

6. **Query Complexity**: Only basic operators supported:
- Comparison: `=`, `!=`, `>`, `<`, `>=`, `<=`
- String: `LIKE`, `ILIKE`, `STARTS_WITH`, `ENDS_WITH`, `CONTAINS`
- Logical: `AND`, `OR`, `NOT`
- Null checks: `IS NULL`, `IS NOT NULL`
- Array: `IN`, `NOT IN`

### Performance Characteristics

| Operation | Expected Latency | Notes |
|-----------|------------------|-------|
| Insert (single row) | < 50ms | With schema validation |
| Insert (batch 100) | < 200ms | Parallel validation |
| Select (indexed) | < 20ms | Primary key or unique column |
| Select (filtered, 1K rows) | < 100ms | With GIN index |
| Update (single row) | < 50ms | By primary key |
| Delete (single row) | < 30ms | By primary key |

### Storage Accounting

- Table storage counts toward workspace storage quota
- Calculated as: `sum(row_data_size) + schema_overhead`
- Schema overhead: ~1KB per table
- Row overhead: ~100 bytes per row (metadata, timestamps)

## Implementation Checklist

- [ ] Add `simTable` and `simTableRow` to `packages/db/schema.ts` and generate migration
- [ ] Create `apps/sim/lib/tables/` with schema validation and types
- [ ] Create `apps/sim/tools/table/` with all 6 tool implementations
- [ ] Register tools in `apps/sim/tools/registry.ts`
- [ ] Create API routes for tables and rows CRUD operations
- [ ] Create `apps/sim/blocks/blocks/table.ts` block definition
- [ ] Register block in `apps/sim/blocks/registry.ts`
- [ ] Add `TableIcon` to `apps/sim/components/icons.tsx`
Loading