bpo-35922: Fix RobotFileParser when robots.txt is invalid#11791

Merged

taleinat merged 16 commits into

python:masterfrom

remilapeyre:fix-robotfileparser-for-invalid-files

Jun 16, 2019

Commits on Feb 8, 2019

Fix RobotFileParser when robot.txt is invalid
Rémi Lapeyre
committed

Commits on Jun 9, 2019

Rename test case

Rémi Lapeyre
and
taleinat
authored
Return None explicitly
Rémi Lapeyre
committed
Add tests for bad user-agent and missing attributes
Rémi Lapeyre
committed

Commits on Jun 11, 2019

Always test craw_delay and request_rate
Rémi Lapeyre
committed
Remove NoRequestRateAndCrawlDelayTest
Rémi Lapeyre
committed
Remove useless attributes in DifferentAgentTest
Rémi Lapeyre
committed
Remove Misc/python-embed.pc
Rémi Lapeyre
committed
Fix EmptyFileTest
Rémi Lapeyre
committed
add a NEWS entry
taleinat
committed