A robust and professional web scraping tool built with Python to extract product data from paginated websites. It uses requests and BeautifulSoup with logging, retry logic, and CSV export capabilities.

• 🌐 Fetch HTML pages with custom browser headers.
• 🛒 Extract product details:
  • Product name
  • Product price
  • Product link
• 🔄 *Retry logic for failed requests.
• 📄 Scrape multiple pages automatically.
• 💾 Save results to CSV file.
• 📊 Logging for progress and error tracking.
• 🛠 Easily customizable CSS selectors for any website structure.

• Python 3.x
• Python libraries:
  • requests
  • beautifulsoup4
  • pandas (optional for CSV formatting)

Install dependencies:

pip install requests beautifulsoup4 pandas

1. Open Web Scraper Code.py.

2. Modify the BASE_URL to target the website you want to scrape.

3. Adjust pagination in scrape_all_pages(start, end).

4. Run the script:

python Web Scraper Code.py The script will log progress and save all scraped products to:

products.csv

✅ Always check the website's robots.txt before scraping.

✅ Use time.sleep() between requests to avoid overwhelming servers.

✅ Use headers to mimic a real browser.

⚡ For dynamic content (JS-loaded pages), consider Selenium.

🔧 Customize CSS selectors in parse_products() for each website.

🗂 For large datasets, you can save output to JSON or a database.

• The script logs:

• URL fetch attempts

• Status codes and errors

• Number of products found per page

• CSV save confirmation

• You can extend this project to:

• Scrape multiple websites simultaneously

• Schedule scraping tasks with cron or task scheduler

• Visualize product trends with Matplotlib or Seaborn

• Integrate with APIs or dashboards for real-time updates