robotparser should support specifying SSL context

@berkerpeksag

BPO	43597
Nosy	@berkerpeksag, @tchinmai7
PRs	bpo-43597: add ability to specify sslcontext in RobotFileParser.read() #24984 bpo-43597: add ability to specify sslcontext in RobotFileParser.read() #24986

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-22.23:08:07.176>
labels = ['type-feature', 'library', '3.10']
title = 'robotparser should support specifying SSL context'
updated_at = <Date 2021-04-07.01:56:13.794>
user = 'https://github.com/Tchinmai7'

bugs.python.org fields:

activity = <Date 2021-04-07.01:56:13.794>
actor = 'Tchinmai7'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-03-22.23:08:07.176>
creator = 'Tchinmai7'
dependencies = []
files = []
hgrepos = []
issue_num = 43597
keywords = ['patch']
message_count = 3.0
messages = ['389352', '390395', '390396']
nosy_count = 2.0
nosy_names = ['berker.peksag', 'Tchinmai7']
pr_nums = ['24984', '24986']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue43597'
versions = ['Python 3.10']

IMO this could be enhanced by adding a sslcontext parameter to read method

a sample change would it could look like

def read(self, sslcontext=None):
    """Reads the robots.txt URL and feeds it to the parser."""
    try:
        if sslcontext:
           f = urllib.request.urlopen(self.url, context=sslcontext)
        else:
           f = urllib.request.urlopen(self.url)
    except urllib.error.HTTPError as err:
        if err.code in (401, 403):
            self.disallow_all = True
        elif err.code >= 400 and err.code < 500:
            self.allow_all = True
    else:
        raw = f.read()
        self.parse(raw.decode("utf-8").splitlines())

Happy to send a PR if this proposal makes sense.

I'm not opposing to the idea, but what's the practical use case here? I haven't seen a case where you needed to pass a custom SSLContext in order to fetch the robots.txt file.

I am writing a web scraper, that runs in a container that has CA-Certificates stored in a non-standard location. The Ca-Certificates are managed by Certifi. By allowing to override the sslcontext, it is possible for the user to construct a sslcontext and pass it in.

Navigation Menu

Repository navigation

robotparser should support specifying SSL context #87763

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

robotparser should support specifying SSL context #87763

Description

Activity

Tchinmai7 commented on Mar 22, 2021

berkerpeksag commented on Apr 7, 2021

Tchinmai7 commented on Apr 7, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions