Skip to content

Is it possible to change the proxyUrl with the beforeRequest hook? #77

Open

Description

@szmarczak

Hey!

First off all thanks for this amazing package!

As the title suggests, is there any way to set the proxyUrl through the beforeRequest hook?
Ideally I'd like to do this on the beforeRetry hook but we cannot access the options object from there.

It should sort of act like a proxy rotation system - if the request fails, call a function which returns a new proxy and set it as the new proxyUrl.

Thanks!

Originally posted by @PhilBookst in #76

Activity

  1. self-assigned this
    on Sep 11, 2022
  2. szmarczak commented on Sep 11, 2022

    @szmarczak
    ContributorAuthor

    It should be possible to do that, I'll add a test for this. You should be able to override error.options.context.proxyUrl.

  3. VoidMonk commented on Jul 7, 2023

    @VoidMonk

    Any way to remove the set proxy before a retry?

    Setting error.options.context.proxyUrl to undefined, null or an empty string in the beforeRetry hook still makes a retry request with the originally set proxyUrl.

    @szmarczak We'll appreciate any suggestions or unit test results from you. Thanks.

  4. yovanoc commented on Sep 23, 2023

    @yovanoc

    Would like to know this too, thanks

  5. yovanoc commented on Jan 18, 2024

    @yovanoc

    as I understand, if we extends the got-scraping client our hooks will be called after got-scraping ones, so the proxy hook is called before we can change the proxyUrl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions