Skip to content

fix(web): block disallowed Meta crawler before SSR - #1592

Open
brendan-kellam wants to merge 2 commits into
mainfrom
brendan/block-meta-externalagent
Open

fix(web): block disallowed Meta crawler before SSR#1592
brendan-kellam wants to merge 2 commits into
mainfrom
brendan/block-meta-externalagent

Conversation

@brendan-kellam

@brendan-kellam brendan-kellam commented Aug 14, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Enforce the existing robots.txt policy for meta-externalagent in the Next.js proxy.
  • Return 403 before the request reaches authentication, database work, or React server rendering.
  • Preserve normal browser traffic and the existing legacy URL redirect behavior.

Finding

Production ALB logs for the 2026-08-14 traffic surge contain 700,083 requests from 16:00-21:00 UTC. Of those, 563,237 were successful /browse requests spanning 198,239 distinct pathnames. User-agent attribution shows meta-externalagent generated 562,846 of them: 99.93% of browse traffic and about 80% of all web traffic in the window.

The application robots.txt already explicitly disallows meta-externalagent because Sourcebot exposes an effectively unbounded file/revision/commit URL space, but robots directives are advisory and the crawler is ignoring them. During the same window, post-major-GC old-space continued to rise with cumulative requests. Blocking this crawler does not replace the underlying retention fixes, but it removes the verified production load amplifier that is driving the current heap growth rate and liveness pressure.

Remediation

The proxy now recognizes meta-externalagent case-insensitively and returns 403 before SSR. This makes the documented crawler policy enforceable even when the crawler does not honor robots.txt, while leaving link-preview fetchers and ordinary browsers unchanged.

Test plan

  • yarn workspace @sourcebot/web test --run src/proxy.test.ts (4 tests passed)
  • yarn workspace @sourcebot/web exec eslint src/proxy.ts src/proxy.test.ts
  • Regression cases cover the observed Meta user-agent forms, case-insensitive matching, ordinary browser passthrough, and the existing legacy organization-prefix redirect.

Note

Cursor Bugbot is generating a summary for commit 54012b1. Configure here.

Summary by CodeRabbit

  • New Features
    • Added protection against requests from crawlers that do not follow the site’s crawler policy.
    • Blocked requests receive a clear HTTP 403 response before page rendering.
  • Bug Fixes
    • Preserved existing redirects for legacy organization-prefixed URLs.
  • Documentation
    • Added an unreleased changelog entry describing the crawler filtering update.

@coderabbitai

coderabbitai Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d5cc9b7e-87fc-44ae-8396-6d16b3d69b8f

📥 Commits

Reviewing files that changed from the base of the PR and between f3b61aa and add2053.

📒 Files selected for processing (3)
  • CHANGELOG.md
  • packages/web/src/proxy.test.ts
  • packages/web/src/proxy.ts

Walkthrough

The proxy now blocks requests from meta-externalagent before server rendering or redirects. Tests cover blocked crawler requests, ordinary browser requests, and legacy organization-prefix redirects. The changelog records the fix.

Changes

Proxy crawler filtering

Layer / File(s) Summary
Crawler filter and proxy behavior
packages/web/src/proxy.ts, packages/web/src/proxy.test.ts, CHANGELOG.md		 The proxy lowercases the user-agent, returns HTTP 403 for meta-externalagent, preserves existing redirects for other requests, and adds coverage and changelog documentation.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to add20

The change blocks the specified crawler before server rendering while preserving ordinary browser traffic and the legacy redirect; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes blocking the disallowed Meta crawler before server-side rendering.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch brendan/block-meta-externalagent

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@brendan-kellam