Firehose phase 2: follows hose (kind 3) + relay-URL harvesting#10
Merged
Conversation
- src/hoses/event.js: extract the shared structural + schnorr-signature
check (eventId, verifySignature); profiles hose refactored to use it.
- src/hoses/follows.js (kind 3): verify signature, derive the social-graph
shape { pubkey, follows:[hex…], created_at, count } from p tags (64-hex,
lowercased, deduped), latest-wins upsert (the legacy followshose blindly
replaced — an out-of-order older list could clobber a newer one). Owns
its indexes (pubkey, follows, created_at). Harvests relay URLs from the
legacy relay map in kind-3 content into the relays directory via
$setOnInsert (discovery only — never overwrites health metrics;
best-effort, never fails ingest).
- src/indexer.js: register the follows hose; LEGACY_KINDS drops to [10002].
The switch removes kind 3 from the legacy set automatically.
- tests: new follows-hose tests (verify/parse/harvest/canonicalize);
plan tests updated for the now-hose-owned kind 3.
Fixes in-degree for kind-3 events the legacy raw-upsert path wrote without
a top-level follows array. npm test 34/34.
Closes #9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9. Phase 2 of the firehose rework (after #7) — adds the follows hose and retires the legacy raw-upsert path for kind 3.
What
src/hoses/event.js— extract the shared structural + schnorr-signature check (eventId,verifySignature) every hose uses; the profiles hose now delegates to it (DRY, −18 lines there).src/hoses/follows.js(kind 3):{ pubkey, follows:[hex…], created_at, count }fromptags (64-hex, lowercased, deduped). This is the shapefollowerCount(in-degree) andenrichCountsneed.replaceOne'd, so an out-of-order older contact list could clobber a newer one.pubkey,follows(reverse lookup),created_at.content, canonicalizes (trailing slash on origins; ws/wss only), and$setOnInserts into therelaysdirectory — discovery only, never overwrites health metrics, best-effort (never fails ingest).src/indexer.js: register the follows hose;LEGACY_KINDSdrops to[10002]. TheHOSESswitch removes kind 3 from the legacy set automatically, soHOSES=profiles,followsis now meaningful.Why it's also a fix
The phase-1 legacy fallback wrote raw kind-3 events for kind 3, which have no top-level
followsarray — so follower counts didn't accrue for them. Routing kind 3 through the derived-shape hose fixes in-degree.Verification
npm test— 34/34 (new follows-hose tests: verify, wrong kind, tampered, malformed;parseFollowsdedupe/hex;canonicalizeRelayUrl;relayUrlsFrom; plan tests updated for hose-owned kind 3).{relay}only, and a pre-existing relay'sonline/uptimeleft untouched by the harvest.Out of scope
Relay-lists hose (kind 10002) — phase 3 · relay-health prober — phase 4 · deploy/retire legacy server hoses — phase 5.