robotparser do not allow some url which are allowed

Bug report

Bug description:

I was using robotparser to check some websites and I realized that the there was a strange behavior. I gave it some urls that I knew could be crawled and it return me False. After some debugging I realize that the url was ok but I think there is a problem in the function that is reading the url.

Line 62 of https://github.com/python/cpython/blob/main/Lib/urllib/robotparser.py

f = urllib.request.urlopen(self.url)

returned my an error code 404, but actually the url/robots.txt file was present and I could see it online.

What I did was to add these two lines of code, the error disappear and the function told me the url could be crawled, as I expected. :

header= {'User-Agent': '*'}  
req = urllib.request.Request(url=self.url, headers=header) 
f = urllib.request.urlopen(req)

Could you give me a feedback on that? Do you think it is correct and could be fixed in the main version?

CPython versions tested on:

3.8, 3.9

Operating systems tested on:

Windows

Can you please provide a URL to reproduce this locally?

Yes, sure. here my code.

If I try it using the original function it returns False. If I add the header (see above) it return True

  url='https://www.researchgate.net/profile/Federica-Gervasoni'
  
  parse_url=urlparse(url)
  base_url=parse_url.scheme + '://' + parse_url.netloc

  #Check robots
  rp = urllib.robotparser.RobotFileParser()
  rp.set_url(base_url + '/robots.txt')
  rp.read()
  print(rp.modified())

  if rp.can_fetch("*", url):
    print(f"{url}")
  else:
    print(f"Crowling cannot be performed {url}")

The robotparser module is old, it is based on the pre-standard document http://www.robotstxt.org/orig.html. It does not support all features of RFC 9309, which was only published in 2022. In particularly, it does not support the $ special character.

So, this is a request for a new feature. Unfortunately, the full support of RFC 9309 requires changing the internal structure of the module.

In RFC 9309:

2.3.1.3. "Unavailable" Status

"Unavailable" means the crawler tries to fetch the robots.txt file and the server responds with status codes indicating that the resource in question is unavailable. For example, in the context of HTTP, such status codes are in the 400-499 range.

If a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server.

The robotparser implementation treats status codes 401 and 403 as "all disallowe" (it was so from beginning). It MAY treat them as "all allowed", but the current behavior does not violate RFC 9309. We can add an option to allow full access in such cases if there are good reasons to keep the current behavior.

Addressed here #103753

Navigation Menu

Repository navigation

robotparser do not allow some url which are allowed #110333

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

robotparser do not allow some url which are allowed #110333

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

sobolevn commented on Oct 4, 2023

FedeGerva commented on Oct 4, 2023

serhiy-storchaka commented on Sep 4, 2025

serhiy-storchaka commented on May 4, 2026

orsenthil commented on Jul 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions