Repository navigation
Add tunnel CONNECT response headers to httplib / http.client #69152
Description
| BPO | 24964 |
|---|---|
| Nosy | @terryjreedy, @vadmium, @OneMoreZanuda |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2015-08-30.17:41:50.140>
labels = ['type-feature', 'library', '3.11']
title = 'Add tunnel CONNECT response headers to httplib / http.client'
updated_at = <Date 2021-05-15.21:19:25.060>
user = 'https://bugs.python.org/thomasbelhalfaoui'bugs.python.org fields:
activity = <Date 2021-05-15.21:19:25.060>
actor = 'alexey.namyotkin'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-08-30.17:41:50.140>
creator = 'thomas.belhalfaoui'
dependencies = []
files = ['40301', '40371']
hgrepos = []
issue_num = 24964
keywords = ['patch']
message_count = 10.0
messages = ['249361', '249373', '249741', '249803', '249897', '249906', '249996', '250085', '393729', '393730']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'martin.panter', 'thomas.belhalfaoui', 'alexey.namyotkin']
pr_nums = ['26152']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24964'
versions = ['Python 3.11']
Activity
thomasbelhalfaoui commentedon Aug 30, 2015
When using httplib / http.client to connect to an HTTPS website through a proxy (by making a tunnel with a CONNECT request), there is no way to retrieve the HTTP headers which the proxy sends back in response to that CONNECT request.
This becomes a problem when using rotating proxy providers like ProxyMesh, who send useful information in those headers (for instance, "X-ProxyMesh-IP" contains the IP address of the proxy, which is necessary to keep the same address throughout the session).
It would be nice to save those headers in a property of the HTTPConnection class (e.g. self._tunnel_response_headers), which would be set up inside the _tunnel method (as proposed in the attached patch, lines 748 and 827-831). This would allow to get the headers back and/or pass them to a higher-level library (such as requests).
vadmium commentedon Aug 30, 2015
Such a change would involve adding a new API, so should go into a new version of Python.
Thomas: a diff rather than a full copy of the changed file would be more convenient. Also, if this gets accepted, test cases and documentation would be needed.
It is also useful to get the header of an unsuccessful CONNECT response. For example, see bpo-7291, where the Proxy-Authenticate header of the proxy’s 407 response needs to be accessible. In that issue, I started working on a patch tht may also be useful here. From memory, usage would be a bit like this:
Thomas, let me know if this would be useful for you, and I can try and dig up my patch.
thomasbelhalfaoui commentedon Sep 4, 2015
Martin: Thanks for your quick answer (and sorry for sending the whole file) !
I think it is indeed a good idea to detach the proxy connection and treat it as any other connection, as you did in your patch. It would be great if you would be able to dig it up !
terryjreedy commentedon Sep 4, 2015
Thomas, please sign a contributor agreement for your patches to be considered.
https://www.python.org/psf/contrib/
https://www.python.org/psf/contrib/contrib-form/
vadmium commentedon Sep 5, 2015
This is the patch I had in mind. It looks like it only implements the detach() method, so we would still need to add support for passing in the tunnel details to the HTTPSConnection constructor.
This patch would allow doing stuff at a lower level than the existing tunnel functionality. The patch includes a test case for getting the proxy’s response header fields, and another test case illustrating how a plain text HTTP 2 upgrade could work.
thomasbelhalfaoui commentedon Sep 5, 2015
Terry: Thanks for the form, I just filled it.
Martin: Thanks for sending your patch. I will dive into it, and try to figure out how to add support for passing in the tunnel details to the HTTPSConnection constructor.
thomasbelhalfaoui commentedon Sep 6, 2015
Martin, I went through your patch and made some simple tests, and I have a couple of questions.
So I tweaked the "getresponse" function so that it does not call "self.close()" (i.e. the connection stays open after the CONNECT request) in that case, and it seems to works fine.
Do you think it makes sense or am I missing something ?
vadmium commentedon Sep 7, 2015
The real problem is when _tunnel() internally calls getresponse(), it notices the connection cannot be reused for another request, and closes the socket object. Perhaps I should rethink my logic; maybe move sock and detach() to HTTPResponse.
With some rough experimentation, passing tunnel through the HTTPConnection (plain text HTTP) constructor seems to work for me. However if you meant HTTPSConnection (over TLS) instead, you will probably need to manually do the wrap_socket() step. Maybe that’s why your connection is being dropped.
terryjreedy commentedon May 15, 2021
Alexey, to repeat what I said to Thomas above: please sign a contributor agreement for your patches to be considered.
https://www.python.org/psf/contrib/
https://www.python.org/psf/contrib/contrib-form/
OneMoreZanuda commentedon May 15, 2021
Thanks, Terry. I signed it.
gh-69152: Add _proxy_response_headers attribute to HTTPConnection (#2…
Merge branch 'main' into inlinecomp2
pythongh-69152: Add _proxy_response_headers attribute to HTTPConnecti…
gh-69152: add method get_proxy_response_headers to HTTPConnection cla…
gpshead commentedon May 16, 2023
thanks for the contribution!
Merge branch 'main' into supermon