Greptile Overview

Summary

Replaces token estimation heuristics with tiktoken for accurate token counting in embeddings and chunking. This ensures compliance with OpenAI's 8,191 token limit per embedding request.

Key changes:

• Integrated tiktoken library for precise token counting matching OpenAI's behavior
• Replaced fixed batch sizes (50 items) with token-aware batching (8,000 tokens/batch)
• Reduced JSON chunk sizes from 2000→1000 tokens (target) and 3000→1500 tokens (max) for safer margins
• Added support for JSON/YAML file uploads
• Added fallback to estimation when tiktoken fails

Issues found:

• Memory leak: tiktoken encodings are cached but never freed - clearEncodingCache() function exists but is never called
• Type safety: as any assertion bypasses TypeScript safety

Confidence Score: 3/5

• Safe to merge with minor memory leak that should be addressed post-merge
• Core logic is sound and improves accuracy significantly. However, tiktoken encodings are never freed causing a memory leak in long-running processes. The cache is small (typically 1-3 models) so impact is limited, but should be fixed. Type assertion issue is minor.
• apps/sim/lib/tokenization/estimators.ts - needs cleanup mechanism for encoding cache

Important Files Changed

File Analysis

Sequence Diagram

sequenceDiagram
    participant Client
    participant API
    participant EmbeddingUtils
    participant Tokenization
    participant Tiktoken
    participant OpenAI

    Client->>API: Upload JSON or YAML file
    API->>API: Validate file extension
    Note over API: json, yaml, yml now allowed
    
    API->>JsonYamlChunker: chunk content
    JsonYamlChunker->>Tokenization: getAccurateTokenCount
    Tokenization->>Tiktoken: encode text
    Tiktoken-->>Tokenization: token count
    Note over JsonYamlChunker: Reduced chunk sizes<br/>1000 target 1500 max
    JsonYamlChunker-->>API: chunks array
    
    API->>EmbeddingUtils: generateEmbeddings
    EmbeddingUtils->>Tokenization: batchByTokenLimit with 8000 max
    Tokenization->>Tiktoken: count tokens for each text
    Tiktoken-->>Tokenization: token counts
    Tokenization-->>EmbeddingUtils: batches array
    Note over Tokenization: Token-aware batching<br/>replaces fixed batches
    
    loop For each batch
        EmbeddingUtils->>OpenAI: Request embeddings
        Note over EmbeddingUtils,OpenAI: Max 8000 tokens per batch
        OpenAI-->>EmbeddingUtils: embeddings array
    end
    
    EmbeddingUtils-->>API: all embeddings
    API-->>Client: Success response

_{7 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}