Skip to content

Leaderboard migration#1907

Open
SamXop123 wants to merge 4 commits into
recodehive:mainfrom
SamXop123:leaderboard-migration
Open

Leaderboard migration#1907
SamXop123 wants to merge 4 commits into
recodehive:mainfrom
SamXop123:leaderboard-migration

Conversation

@SamXop123

Copy link
Copy Markdown
Contributor

Whats implemented:

  1. Backend Cron Scraper: The Express backend runs a background scheduled cron job (every 2 hours) that scrapes public, unarchived organization repositories, filters for merged PRs matching the "recode" label, aggregates repository statistics, and counts discussions.
  2. Point & Streak Aggregation: Contributors are scored according to Recode rules (Level 1 = 10, Level 2 = 30, Level 3 = 50 points). Streaks are calculated based on active daily consecutive PR merge dates.
  3. Atomic Cache Storage: Scraper data is written atomically to leaderboard.json and stats.json inside the cache directory (compatible with persistent volume mounts).
  4. Backend API: Serves the cached datasets instantly via /api/leaderboard and /api/stats.
  5. Frontend Context Hook: The frontend React context (CommunityStatsProvider) fetches from these two backend API routes in parallel on page load. Client-side sorting and date filtering (week/month/year/all-time) are preserved in memory for instant, responsive UI transitions.
image

Files Changed:

Backend (leaderboard repository)

  • server.js: Setup API routes, CORS configs, and initialized startup checks.
  • utils/cacheHelper.js [NEW]: Implements atomic file writes (writeJsonAtomic) to prevent corruption, file exists checks, and fallback JSON models.
  • functions/generateLeaderboard.js: Replaced legacy scraper. Implements GraphQL repository traversal, label scoring weight filters, active streak checks, and discussions count queries.
  • functions/generateCALeaderboard.js: Integrates atomic writes and generation locks.
  • jobs/updateOSLeaderboard.js: Schedules OS leaderboard cron jobs.
  • jobs/updateCALeaderboard.js: Standardizes cron patterns.

Frontend (this repository)

  • docusaurus.config.ts: Declares backendApiUrl customField and removes deprecated gitToken client exposing.
  • src/lib/statsProvider.tsx: Removes client-side scraping pipelines. Fetches and parses backend API endpoints and binds to context states.
  • src/services/githubService.ts: Removes dead code path helper methods.

API Contract

A. GET /api/leaderboard

Returns the contributors array and legacy compatibility aliases.

{
  "success": true,
  "updatedAt": 1781548394726,
  "lastUpdated": 1781548394726,
  "generated": true,
  "contributors": [
    {
      "username": "contributor1",
      "avatar": "https://github.com/contributor1.png",
      "profile": "https://github.com/contributor1",
      "points": 60,
      "prs": 2,
      "prDetails": [
        {
          "title": "PR 2 - Docs",
          "url": "https://github.com/recodehive/repo1/pull/2",
          "mergedAt": "2026-06-15T12:00:00Z",
          "repoName": "repo1",
          "number": 2,
          "points": 50
        }
      ],
      "streak": 2
    }
  ]
}

B. GET /api/stats

{
  "success": true,
  "updatedAt": 1781548394731,
  "lastUpdated": 1781548394731,
  "totalStars": 984,
  "totalForks": 1107,
  "totalContributors": 467,
  "totalRepositories": 10,
  "publicRepositories": 9,
  "discussionsCount": 42
}

Migration Strategy

  1. Backend Deploy (Railway): Deploy backend and map CACHE_DIR to a persistent volume (e.g. /data) to preserve cached listings across restarts. Set PORT (default 5000) and the private GIT_TOKEN.
  2. Frontend Config: Expose BACKEND_API_URL pointing to the deployed backend server domain.
  3. Authentication: Clerk login gate remains unchanged. Discussions panel continues fetching via restricted token until a discussions backend proxy endpoint is introduced in a future PR.

Testing Performed

  • Backend Unit Tests: Verified repository discovery, unarchived filters, point weights, and active streak resets using mock datasets.
  • E2E Headless Browser Testing: Used Puppeteer to verify:
    • Podium: Rendered 1st, 2nd, and 3rd rank performer cards correctly.
    • Table: Populated ranking rows, usernames, avatars, and badges.
    • Stats Cards: Displayed correct community values.
    • PR Modal: Clicking a row count badge successfully launched the PR List Modal with correct titles, URLs, and points.

Fixes #1362

Type of Change

  • New feature (e.g., new page, component, or functionality)
  • Bug fix (non-breaking change that fixes an issue)
  • UI/UX improvement (design, layout, or styling updates)
  • Performance optimization (e.g., code splitting, caching)
  • Documentation update (README, contribution guidelines, etc.)
  • Other (please specify): backend migration

Checklist

  • My code follows the style guidelines of this project.
  • I have tested my changes across major browsers and devices
  • My changes do not generate new console warnings or errors .
  • I ran npm run build and attached screenshot(s) in this PR.
  • This is already assigned Issue to me, not an unassigned issue.

@vercel

vercel Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

@SamXop123 is attempting to deploy a commit to the recode Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

Copy link
Copy Markdown

Thank you for submitting your pull request! 🙌 We'll review it as soon as possible. The estimated time for response is 5–8 hrs.

In the meantime, please provide all necessary screenshots and make sure you run - npm build run , command and provide a screenshot, a video recording, or an image of the update you made below, which helps speed up the review and assignment. If you have questions, reach out to LinkedIn. Your contributions are highly appreciated!😊

Note: I maintain the repo issue every day twice at 8:00 AM IST and 9:00 PM IST. If your PR goes stale for more than one day, you can tag and comment on this same issue by tagging @sanjay-kv.

We are here to help you on this journey of open source. Consistent 20 contributions are eligible for sponsorship 💰

🎁 check our list of amazing people we sponsored so far: GitHub Sponsorship. ✨

📚Your perks for contribution to this community 👇🏻

  1. Get free Consultation use code recode50 to get free: Mentorship for free.

  2. Get the Ebook for free use code recode at checkout: Data Science cheatsheet for Beginners.

  3. Check out this weekly Newsletter: Sanjay's Newsletter.

If there are any specific instructions or feedback regarding your PR, we'll provide them here. Thanks again for your contribution! 😊

@github-actions github-actions Bot added in-review The current changes are in review and would need approval and testing before merging level 1 10 points recode this is label for leaderboard labels Jun 17, 2026
@github-actions github-actions Bot added this to the recode:launch 3.0 milestone Jun 17, 2026
@github-actions github-actions Bot added area:backend-sql for issues related to the experimintal support of MsSql as backend enhancement New feature or request level 2 30 points labels Jun 17, 2026
@github-actions

Copy link
Copy Markdown

✅ Synchronized metadata from Issue #1362:

  • Labels: enhancement, area:backend-sql, level 2, recode
  • Milestone: recode:launch 3.0

@SamXop123

Copy link
Copy Markdown
Contributor Author

The leaderboard repo pr: http://github.com/recodehive/leaderboard/pull/1

@SamXop123

Copy link
Copy Markdown
Contributor Author

@iitzIrFan this was a multiple repo issue as suggested by @Adez017 so i had to create two pr for backend migration

@iitzIrFan

Copy link
Copy Markdown
Member

Okay makes scenes now, @Adez017 will review soon!

@Adez017

Adez017 commented Jun 17, 2026

Copy link
Copy Markdown
Member

Okay makes scenes now, @Adez017 will review soon!

will review it in the morning @iitzIrFan , need to take a look closer onwards

@SamXop123

Copy link
Copy Markdown
Contributor Author

it works perfectly fine on local testing. just needs a few production tests

a little help for reviewers for production tsting:

  • deploy the backend branch to railway.
  • add git token and cache directory (CACHE_DIR=/data)
  • mount a persistent volm at /data
  • and start the service
  • then we can verify the /api/leaderboard & /api/stats endpoints
  • on frontend we set BACKEND_API_URL=<railway-backend-url>
  • deploy that branch as preview
  • then we can visit the leaderboard page to actually verify.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates the website’s leaderboard/community stats data flow away from client-side GitHub scraping toward a dedicated backend API, aligning with the goal of reducing GitHub rate-limit pressure and improving page-load responsiveness.

Changes:

  • Added a backendApiUrl Docusaurus customFields value for configuring the backend base URL.
  • Updated CommunityStatsProvider to fetch /api/leaderboard and /api/stats from the backend in parallel instead of scraping GitHub on the client.
  • Removed now-obsolete organization stats scraping methods from GitHubService.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
docusaurus.config.ts Adds backendApiUrl to browser-exposed config; (still) exposes gitToken via customFields.
src/lib/statsProvider.tsx Replaces client-side scraping pipeline with backend fetch + mapping into existing context state.
src/services/githubService.ts Removes org stats scraping/aggregation methods that are no longer used by the frontend.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/lib/statsProvider.tsx
Comment on lines 166 to +167
const token = customFields?.gitToken || "";
const backendApiUrl = (customFields?.backendApiUrl as string) || "http://localhost:5000";
Comment thread src/lib/statsProvider.tsx
Comment on lines 314 to 317
data: {
contributors: contributorsArray,
rawStats: { totalPRs: totalMergedPRs },
rawStats: { totalPRs: statsData.totalContributors },
},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Adez017 I need your review on merged PR
https://github.com/recodehive/leaderboard/pull/1/changes

This is the backend code.

Comment thread docusaurus.config.ts Outdated
Comment thread docusaurus.config.ts
Comment on lines +286 to 288
backendApiUrl: process.env.BACKEND_API_URL || "http://localhost:5000",
gitToken: process.env.DOCUSAURUS_GIT_TOKEN,
clerkPublishableKey: process.env.VITE_CLERK_PUBLISHABLE_KEY || "",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think now we don't need to mount the git token , let me know if it is required or we can move forward and remove this one @SamXop123

@vercel

vercel Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
recode-website Ready Ready Preview, Comment Jun 18, 2026 2:16am

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

✅ Synchronized metadata from Issue #1362:

  • Labels: enhancement, area:backend-sql, level 2, recode
  • Milestone: recode:launch 3.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend-sql for issues related to the experimintal support of MsSql as backend enhancement New feature or request in-review The current changes are in review and would need approval and testing before merging level 1 10 points level 2 30 points recode this is label for leaderboard

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

💡[Feature]: Leaderboard Update

5 participants