Summary

This issue proposes a comprehensive ETag strategy for JavaScriptSolidServer aligned with HTTP caching best practices. ETags are critical for efficient caching, bandwidth reduction, and preventing mid-air collisions during concurrent edits.

Difficulty: 35/100
Estimated Effort: 2-3 days
Dependencies: None

Current State Analysis

ETag Generation

File: src/storage/filesystem.js line 32

etag: `"${crypto.createHash('md5').update(stats.mtime.toISOString() + stats.size).digest('hex')}"`

Current approach: MD5 hash of mtime + size (metadata-based)

Conditional Request Handling

File: src/utils/conditional.js (154 lines)

✅ Well implemented:

• If-Match header for safe updates (412 on mismatch)
• If-None-Match for GET/HEAD (304 Not Modified)
• If-None-Match for PUT/POST (create-only with *)
• Wildcard (*) support
• Proper normalization (strips W/ prefix, quotes)

Cache-Control Headers

Current usage:

Missing: No Cache-Control on regular resource responses.

Last-Modified Header

Status: ❌ Not implemented

mtime is available from stat() but not exposed as Last-Modified header.

Issues Identified

1. Strong ETag Mismatch

Severity: MEDIUM

Current ETags are formatted as strong ("abc123") but are generated from metadata, not content. Per RFC 7232:

A strong validator is representation metadata that changes value whenever a change occurs to the representation data that would be observable in the payload body of a 200 (OK) response to GET.

Problem: If a file is modified but mtime is preserved (e.g., touch -m), the ETag won't change even though content changed.

2. Missing Content-Based ETags for Dynamic Content

Severity: MEDIUM

Content negotiation transforms stored content:

• Turtle → JSON-LD conversion
• JSON-LD → Turtle conversion
• HTML data island extraction

These transformations produce different byte streams, but may use the same source file ETag.

3. No Last-Modified Header

Severity: LOW

Some clients prefer Last-Modified over ETags. Both should be provided per best practices.

4. Container Listing ETags

Severity: MEDIUM

Container listings are dynamically generated. Current implementation may use directory mtime, but this doesn't reflect:

• File additions/deletions
• Nested container changes
• ACL changes affecting visibility

5. Synchronous Hash Calculation

Severity: LOW (Performance)

Runs synchronously on every stat() call. For high-traffic servers, this could become a bottleneck.

6. Cache-Control Strategy Missing

Severity: MEDIUM

No systematic Cache-Control headers on resource responses. This means:

• Browsers may cache indefinitely (heuristic caching)
• Or revalidate on every request (no caching benefit)
• CDNs can't optimize caching

Web Best Practices

RFC 7232 - Conditional Requests

• Strong ETags: Byte-for-byte identical representations
• Weak ETags: Semantically equivalent (use W/ prefix)
• If-Match: For safe mutations (optimistic concurrency)
• If-None-Match: For caching (GET) or create-only (PUT)

RFC 7234 - HTTP Caching

• Cache-Control: Primary caching directive
• ETag + Cache-Control: Work together for efficient revalidation
• Last-Modified: Fallback for clients not supporting ETags

Industry Recommendations

Proposed Strategy

1. ETag Generation Tiers

Tier 1: Strong ETag (content-based)

• Use for: Static files where content hash is feasible
• Algorithm: SHA-256, base64url encoded, 27 chars

Tier 2: Weak ETag (metadata-based)

• Use for: Large files, dynamic content, containers
• Format: W/"hash" with mtime + size + extras

Tier 3: Version ETag (for transformed content)

• Use for: Content negotiation results
• Format: W/"hash" derived from source ETag + transformation type

2. ETag Strategy by Resource Type

3. Cache-Control Strategy

4. Last-Modified Header

Add Last-Modified to all resource responses using stats.mtime.toUTCString().

5. Vary Header for Content Negotiation

When content negotiation is enabled, add:

Vary: Accept, Accept-Language

This tells caches that different Accept headers produce different responses.

Implementation Plan

Phase 1: Foundation

• Create src/utils/etag.js with tiered generation functions
• Create src/utils/caching.js with cache profiles
• Replace MD5 with SHA-256 in etag generation
• Switch large files to weak ETags

Phase 2: Headers

• Add Last-Modified header to all resource responses
• Implement Cache-Control profiles by resource type
• Add Vary header for conneg responses

Phase 3: Content-Based ETags

• Implement content hashing for small files (<1MB threshold)
• Cache computed ETags to avoid repeated hashing
• Add async ETag computation option

Phase 4: Container ETags

• Improve container ETag calculation (include child count, newest mtime)
• Consider membership hash for accurate container ETags

Phase 5: Conneg ETags

• Generate distinct ETags for transformed content
• Include transformation type in ETag calculation

Configuration Options

{
  "etag": {
    "algorithm": "sha256",
    "strongThreshold": 1048576,
    "cacheEtags": true,
    "cacheMaxSize": 10000
  },
  "caching": {
    "defaultProfile": "resource",
    "staticMaxAge": 3600,
    "immutableAssets": false
  }
}

Comparison Matrix

Current vs Proposed

Solid Ecosystem Comparison

Testing Plan

Unit Tests

• Strong ETag format validation
• Weak ETag format validation
• Different ETags for conneg transforms
• 304 responses for matching ETags
• 412 responses for If-Match mismatch

Integration Tests

• CDN compatibility (Cloudflare, Fastly)
• Browser caching behavior
• Concurrent edit scenarios (If-Match)
• Solid app compatibility (SolidOS, Penny, etc.)

Security Considerations

1. ETag as fingerprint: ETags can be used to track users across requests. Mitigated by using private in Cache-Control.

2. Timing attacks: Content-based ETags reveal if content changed. This is inherent to caching and generally acceptable.

3. ETag collision: SHA-256 with 162+ bits is collision-resistant. MD5 collisions are feasible but unlikely to be exploited via ETags.

References

• RFC 7232 - Conditional Requests
• RFC 7234 - HTTP Caching
• MDN - ETag
• MDN - HTTP Conditional Requests
• Cloudflare - ETag Headers
• Fastly - ETags: What they are
• Google Web Fundamentals - HTTP Caching

Related Issues

• Roadmap: Production Readiness for v0.1 Release - Code Quality & Cleanup #100 - Production Readiness (ETag caching mentioned in performance section)
• Feature: Docker Support - Dockerfile, Compose, and Container Registry #99 - Docker (caching affects container behavior)