Skip to content

bpo-35922: Fix RobotFileParser when robots.txt is invalid #11791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
taleinat merged 16 commits into from
Jun 16, 2019
Merged

bpo-35922: Fix RobotFileParser when robots.txt is invalid #11791

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
e7b777f
Fix RobotFileParser when robot.txt is invalid
Feb 8, 2019
decb931
Rename test case
Jun 9, 2019
b219f9d
Return None explicitly
Jun 9, 2019
f417727
Add tests for bad user-agent and missing attributes
Jun 9, 2019
00b918a
Always test craw_delay and request_rate
Jun 11, 2019
8680e41
Remove NoRequestRateAndCrawlDelayTest
Jun 11, 2019
1e2a03e
Remove useless attributes in DifferentAgentTest
Jun 11, 2019
4693573
Remove Misc/python-embed.pc
Jun 11, 2019
3125ee9
Fix EmptyFileTest
Jun 11, 2019
0988fb2
add a NEWS entry
taleinat Jun 11, 2019
20dce06
tidy up test_request_rate code
taleinat Jun 12, 2019
5d64344
fix BaseRequestRateTest and make EmptyFileTest inherit it
taleinat Jun 12, 2019
aaade01
improve NEWS entry formatting
taleinat Jun 12, 2019
4e6a9bd
Update 2019-06-11-19-34-29.bpo-35922.rxpzWr.rst
taleinat Jun 12, 2019
d2e9b69
fix NEWS entry formatting
taleinat Jun 12, 2019
67a6ee2
address Berker Peksag's review comments
taleinat Jun 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix EmptyFileTest
  • Loading branch information
Rémi Lapeyre
Rémi Lapeyre committed Jun 11, 2019
commit 3125ee9ba2c4ae422c0cb5688bdaaffb56869319
1 change: 1 addition & 0 deletions Lib/test/test_robotparser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ class RejectAllRobotsTest(BaseRobotTest, unittest.TestCase):

class EmptyFileTest(BaseRobotTest, unittest.TestCase):
robots_txt = ""
good = ['/foo']


class BaseRequestRateTest(BaseRobotTest):
Expand Down