You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I found some website like "sony.co.jp" requests User-Agent field in headers when I try to fetch their robots.txt. Unless they returns 403 Forbidden status code and RobotTextParser fails to read robots.txt. Currently there is no way to specify headers so I propose add headers arguments in constructor.
Activity
eamanu commentedon Sep 9, 2019
Hello I open an issue from a PR that I think need discussion:
I write here the @fuwaraido's PR description:
orsenthil commentedon Jul 5, 2026
Addressed here #103753