XML sitemap parser designed to extract and process millions of URLs while bypassing most modern anti-bot protections. Supports plain and compressed XML, unlimited nested sitemaps, multi-threading, multiple inputs, CloudScraper integration, fingerprint randomization, proxy/user agent rotation, auto stealth mode, and detailed monitoring.

A high effective golang library for parsing big-sized sitemaps and avoiding high memory usage. The sitemap parser was written on golang without external dependencies.

A Sitemap Generator written in PHP

A Multisite Robots.txt Manager - Quickly and easily manage all robots.txt files on a WordPress Multisite Website Network.

sitemap-generator-github Author: Bocaletto Luca A small toolkit for regenerating sitemap.xml for your GitHub Pages portfolio. This repo provides three interchangeable methods—pick the one that fits your workflow best: Local Bash script (gen-sitemap.sh) Local Node.js script (generate-sitemap.js) GitHub Actions workflow...

Making static html pages of your SPA for search bots with storing results in S3 (or on local filesystem).

Parse robots.txt and traverse sitemaps.

Domain Mapping

A CLI utility for generating sitemaps given a particular URL

Are you tired of manually checking your website's sitemaps for updates? Do you want to be notified as soon as something is out of date? Look no further! Our powerful and efficient sitemap monitoring script is here to save the day.

site map generator for websites that serve static content or auto-generated indexes from Nginx

[READ-ONLY] Redirect incoming requests to a random URL from a sitemap.

Sitemap CLI for fast and automated sitemap.xml generation

Dynamic sitemap generator for Go

PHP sitemap generator

Just a scanner that checks if pages in your sitemap are available.