Skip to content

Add headers parameter on RobotFileParser #82248

Closed

Description

@eamanu
mannequin
BPO 38067
Nosy @eamanu
PRs
  • some website requests User-Agent field in headers #15726
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2019-09-09.12:28:17.485>
    labels = ['type-feature', 'library', '3.9']
    title = 'Add headers parameter on RobotFileParser'
    updated_at = <Date 2019-09-09.12:28:17.485>
    user = 'https://github.com/eamanu'

    bugs.python.org fields:

    activity = <Date 2019-09-09.12:28:17.485>
    actor = 'eamanu'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2019-09-09.12:28:17.485>
    creator = 'eamanu'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 38067
    keywords = []
    message_count = 1.0
    messages = ['351440']
    nosy_count = 1.0
    nosy_names = ['eamanu']
    pr_nums = ['15726']
    priority = 'normal'
    resolution = None
    stage = None
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue38067'
    versions = ['Python 3.9']

    Activity

    1. eamanu commented on Sep 9, 2019

      eamanumannequin
      MannequinAuthor

      Hello I open an issue from a PR that I think need discussion:

      I write here the @fuwaraido's PR description:

      I found some website like "sony.co.jp" requests User-Agent field in headers when I try to fetch their robots.txt. Unless they returns 403 Forbidden status code and RobotTextParser fails to read robots.txt. Currently there is no way to specify headers so I propose add headers arguments in constructor.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      type-featureA feature request or enhancement
      on Sep 9, 2019
    3. transferred this issue fromon Apr 10, 2022
    4. orsenthil commented on Jul 5, 2026

      @orsenthil
      Member

      Addressed here #103753

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions