Example Domain

-HTTPX -

HTTPX 1.0 — Prelease.

+A complete HTTP toolkit for Python. Supporting both client & server, and available in either sync or async flavors. -A next-generation HTTP client for Python. -

@@ -46,105 +40,73 @@ Now, let's get started: '\n\n\nExample Domain...' ``` -Or, using the command-line client. +```{ .python .ahttpx .hidden } +>>> import ahttpx -```shell -# The command line client is an optional dependency. -$ pip install 'httpx[cli]' +>>> r = await ahttpx.get('https://www.example.org/') +>>> r + +>>> r.status_code +200 +>>> r.headers['content-type'] +'text/html; charset=UTF-8' +>>> r.text +'\n\n\nExample Domain...' ``` -Which now allows us to use HTTPX directly from the command-line... - -![httpx --help](img/httpx-help.png) - -Sending a request... - -![httpx http://httpbin.org/json](img/httpx-request.png) - -## Features +*Serving responses as the server...* -HTTPX builds on the well-established usability of `requests`, and gives you: +

httpx

-* A broadly [requests-compatible API](compatibility.md). -* Standard synchronous interface, but with [async support if you need it](async.md). -* HTTP/1.1 [and HTTP/2 support](http2.md). -* Ability to make requests directly to [WSGI applications](advanced/transports.md#wsgi-transport) or [ASGI applications](advanced/transports.md#asgi-transport). -* Strict timeouts everywhere. -* Fully type annotated. -* 100% test coverage. - -Plus all the standard features of `requests`... - -* International Domains and URLs -* Keep-Alive & Connection Pooling -* Sessions with Cookie Persistence -* Browser-style SSL Verification -* Basic/Digest Authentication -* Elegant Key/Value Cookies -* Automatic Decompression -* Automatic Content Decoding -* Unicode Response Bodies -* Multipart File Uploads -* HTTP(S) Proxy Support -* Connection Timeouts -* Streaming Downloads -* .netrc Support -* Chunked Requests - -## Documentation - -For a run-through of all the basics, head over to the [QuickStart](quickstart.md). - -For more advanced topics, see the **Advanced** section, -the [async support](async.md) section, or the [HTTP/2](http2.md) section. - -The [Developer Interface](api.md) provides a comprehensive API reference. - -To find out about tools that integrate with HTTPX, see [Third Party Packages](third_party_packages.md). - -## Dependencies +```{ .python .httpx } +>>> import httpx -The HTTPX project relies on these excellent libraries: +>>> def app(request): +... content = httpx.HTML('hello, world.') +... return httpx.Response(200, content=content) -* `httpcore` - The underlying transport implementation for `httpx`. - * `h11` - HTTP/1.1 support. -* `certifi` - SSL certificates. -* `idna` - Internationalized domain name support. -* `sniffio` - Async library autodetection. +>>> httpx.run(app) +Serving on http://127.0.0.1:8080/ (Press CTRL+C to quit) +``` -As well as these optional installs: +```{ .python .ahttpx .hidden } +>>> import ahttpx -* `h2` - HTTP/2 support. *(Optional, with `httpx[http2]`)* -* `socksio` - SOCKS proxy support. *(Optional, with `httpx[socks]`)* -* `rich` - Rich terminal support. *(Optional, with `httpx[cli]`)* -* `click` - Command line client support. *(Optional, with `httpx[cli]`)* -* `brotli` or `brotlicffi` - Decoding for "brotli" compressed responses. *(Optional, with `httpx[brotli]`)* -* `zstandard` - Decoding for "zstd" compressed responses. *(Optional, with `httpx[zstd]`)* +>>> async def app(request): +... content = httpx.HTML('hello, world.') +... return httpx.Response(200, content=content) -A huge amount of credit is due to `requests` for the API layout that -much of this work follows, as well as to `urllib3` for plenty of design -inspiration around the lower-level networking details. +>>> await httpx.run(app) +Serving on http://127.0.0.1:8080/ (Press CTRL+C to quit) +``` -## Installation +--- -Install with pip: +# Documentation + +* [Quickstart](quickstart.md) +* [Clients](clients.md) +* [Servers](servers.md) +* [Requests](requests.md) +* [Responses](responses.md) +* [URLs](urls.md) +* [Headers](headers.md) +* [Content Types](content-types.md) +* [Streams](streams.md) +* [Connections](connections.md) +* [Parsers](parsers.md) +* [Network Backends](networking.md) -```shell -$ pip install httpx -``` +--- -Or, to include the optional HTTP/2 support, use: +# Collaboration -```shell -$ pip install httpx[http2] -``` +The repository for this project is currently private. -To include the optional brotli and zstandard decoders support, use: +We’re looking at creating paid opportunities for working on open source software *which are properly compensated, flexible & well balanced.* -```shell -$ pip install httpx[brotli,zstd] -``` +If you're interested in a position working on this project, please send an intro: *kim@encode.io* -HTTPX requires Python 3.9+ +--- -[sync-support]: https://github.com/encode/httpx/issues/572 +

This design work is not yet licensed for reuse.
— 🦋 —

diff --git a/docs/logging.md b/docs/logging.md deleted file mode 100644 index b3c5781727..0000000000 --- a/docs/logging.md +++ /dev/null @@ -1,81 +0,0 @@ -# Logging - -If you need to inspect the internal behaviour of `httpx`, you can use Python's standard logging to output information about the underlying network behaviour. - -For example, the following configuration... - -```python -import logging -import httpx - -logging.basicConfig( - format="%(levelname)s [%(asctime)s] %(name)s - %(message)s", - datefmt="%Y-%m-%d %H:%M:%S", - level=logging.DEBUG -) - -httpx.get("https://www.example.com") -``` - -Will send debug level output to the console, or wherever `stdout` is directed too... - -``` -DEBUG [2024-09-28 17:27:40] httpcore.connection - connect_tcp.started host='www.example.com' port=443 local_address=None timeout=5.0 socket_options=None -DEBUG [2024-09-28 17:27:41] httpcore.connection - connect_tcp.complete return_value= -DEBUG [2024-09-28 17:27:41] httpcore.connection - start_tls.started ssl_context=SSLContext(verify=True) server_hostname='www.example.com' timeout=5.0 -DEBUG [2024-09-28 17:27:41] httpcore.connection - start_tls.complete return_value= -DEBUG [2024-09-28 17:27:41] httpcore.http11 - send_request_headers.started request= -DEBUG [2024-09-28 17:27:41] httpcore.http11 - send_request_headers.complete -DEBUG [2024-09-28 17:27:41] httpcore.http11 - send_request_body.started request= -DEBUG [2024-09-28 17:27:41] httpcore.http11 - send_request_body.complete -DEBUG [2024-09-28 17:27:41] httpcore.http11 - receive_response_headers.started request= -DEBUG [2024-09-28 17:27:41] httpcore.http11 - receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Content-Encoding', b'gzip'), (b'Accept-Ranges', b'bytes'), (b'Age', b'407727'), (b'Cache-Control', b'max-age=604800'), (b'Content-Type', b'text/html; charset=UTF-8'), (b'Date', b'Sat, 28 Sep 2024 13:27:42 GMT'), (b'Etag', b'"3147526947+gzip"'), (b'Expires', b'Sat, 05 Oct 2024 13:27:42 GMT'), (b'Last-Modified', b'Thu, 17 Oct 2019 07:18:26 GMT'), (b'Server', b'ECAcc (dcd/7D43)'), (b'Vary', b'Accept-Encoding'), (b'X-Cache', b'HIT'), (b'Content-Length', b'648')]) -INFO [2024-09-28 17:27:41] httpx - HTTP Request: GET https://www.example.com "HTTP/1.1 200 OK" -DEBUG [2024-09-28 17:27:41] httpcore.http11 - receive_response_body.started request= -DEBUG [2024-09-28 17:27:41] httpcore.http11 - receive_response_body.complete -DEBUG [2024-09-28 17:27:41] httpcore.http11 - response_closed.started -DEBUG [2024-09-28 17:27:41] httpcore.http11 - response_closed.complete -DEBUG [2024-09-28 17:27:41] httpcore.connection - close.started -DEBUG [2024-09-28 17:27:41] httpcore.connection - close.complete -``` - -Logging output includes information from both the high-level `httpx` logger, and the network-level `httpcore` logger, which can be configured separately. - -For handling more complex logging configurations you might want to use the dictionary configuration style... - -```python -import logging.config -import httpx - -LOGGING_CONFIG = { - "version": 1, - "handlers": { - "default": { - "class": "logging.StreamHandler", - "formatter": "http", - "stream": "ext://sys.stderr" - } - }, - "formatters": { - "http": { - "format": "%(levelname)s [%(asctime)s] %(name)s - %(message)s", - "datefmt": "%Y-%m-%d %H:%M:%S", - } - }, - 'loggers': { - 'httpx': { - 'handlers': ['default'], - 'level': 'DEBUG', - }, - 'httpcore': { - 'handlers': ['default'], - 'level': 'DEBUG', - }, - } -} - -logging.config.dictConfig(LOGGING_CONFIG) -httpx.get('https://www.example.com') -``` - -The exact formatting of the debug logging may be subject to change across different versions of `httpx` and `httpcore`. If you need to rely on a particular format it is recommended that you pin installation of these packages to fixed versions. diff --git a/docs/networking.md b/docs/networking.md new file mode 100644 index 0000000000..6375fdf291 --- /dev/null +++ b/docs/networking.md @@ -0,0 +1,381 @@ +# Network Backends + +The lowest level network abstractions in `httpx` are the `NetworkBackend` and `NetworkStream` classes. These provide a consistent interface onto the operations for working with a network stream, typically over a TCP connection. Different runtimes (threaded, trio & asyncio) are supported via alternative implementations of the core interface. + +--- + +## `NetworkBackend()` + +The default backend is instantiated via the `NetworkBackend` class... + +

httpx

+ +```{ .python .httpx } +>>> net = httpx.NetworkBackend() +>>> net + +``` + +```{ .python .ahttpx .hidden } +>>> net = ahttpx.NetworkBackend() +>>> net + +``` + +### `.connect(host, port)` + +A TCP stream is created using the `connect` method... + +

httpx

+ +```{ .python .httpx } +>>> net = httpx.NetworkBackend() +>>> stream = net.connect("www.encode.io", 80) +>>> stream + +``` + +```{ .python .ahttpx .hidden } +>>> net = ahttpx.NetworkBackend() +>>> stream = await net.connect("www.encode.io", 80) +>>> stream + +``` + +Streams support being used in a context managed style. The cleanest approach to resource management is to use `.connect(...)` in the context of a `with` block. + +

httpx

+ +```{ .python .httpx } +>>> net = httpx.NetworkBackend() +>>> with net.connect("dev.encode.io", 80) as stream: +>>> ... +>>> stream + +``` + +```{ .python .ahttpx .hidden } +>>> net = ahttpx.NetworkBackend() +>>> async with await net.connect("dev.encode.io", 80) as stream: +>>> ... +>>> stream + +``` + +## `NetworkStream(sock)` + +The `NetworkStream` class provides TCP stream abstraction, by providing a thin wrapper around a socket instance. + +Network streams do not provide any built-in thread or task locking. +Within `httpx` thread and task saftey is handled at the `Connection` layer. + +### `.read(max_bytes=None)` + +Read up to `max_bytes` bytes of data from the network stream. +If no limit is provided a default value of 64KB will be used. + +### `.write(data)` + +Write the given bytes of `data` to the network stream. + +### `.start_tls(ctx, hostname)` + +Upgrade a stream to TLS (SSL) connection for sending secure `https://` requests. + +`` + +### `.get_extra_info(key)` + +Return information about the underlying resource. May include... + +* `"client_addr"` - Return the client IP and port. +* `"server_addr"` - Return the server IP and port. +* `"ssl_object"` - Return an `ssl.SSLObject` instance. +* `"socket"` - Access the raw socket instance. + +### `.close()` + +Close the network stream. For TLS streams this will attempt to send a closing handshake before terminating the conmection. + +

httpx

+ +```{ .python .httpx } +>>> net = httpx.NetworkBackend() +>>> stream = net.connect("dev.encode.io", 80) +>>> try: +>>> ... +>>> finally: +>>> stream.close() +>>> stream + +``` + +```{ .python .ahttpx .hidden } +>>> net = ahttpx.NetworkBackend() +>>> stream = await net.connect("dev.encode.io", 80) +>>> try: +>>> ... +>>> finally: +>>> await stream.close() +>>> stream + +``` + +--- + +## Timeouts + +Network timeouts are handled using a context block API. + +This [design approach](https://vorpus.org/blog/timeouts-and-cancellation-for-humans) avoids timeouts needing to passed around throughout the stack, and provides an obvious and natural API to dealing with timeout contexts. + +### timeout(duration) + +The timeout context manager can be used to wrap socket operations anywhere in the stack. + +Here's an example of enforcing an overall 3 second timeout on a request. + +

httpx

+ +```{ .python .httpx } +>>> with httpx.Client() as cli: +>>> with httpx.timeout(3.0): +>>> res = cli.get('https://www.example.com') +>>> print(res) +``` + +```{ .python .ahttpx .hidden } +>>> async with ahttpx.Client() as cli: +>>> async with ahttpx.timeout(3.0): +>>> res = await cli.get('https://www.example.com') +>>> print(res) +``` + +Timeout contexts provide an API allowing for deadlines to be cancelled. + +### .cancel() + +In this example we enforce a 3 second timeout on *receiving the start of* a streaming HTTP response... + +

httpx

+ +```{ .python .httpx } +>>> with httpx.Client() as cli: +>>> with httpx.timeout(3.0) as t: +>>> with cli.stream('https://www.example.com') as r: +>>> t.cancel() +>>> print(">>>", res) +>>> for chunk in r.stream: +>>> print("...", chunk) +``` + +```{ .python .ahttpx .hidden } +>>> async with ahttpx.Client() as cli: +>>> async with ahttpx.timeout(3.0) as t: +>>> async with await cli.stream('https://www.example.com') as r: +>>> t.cancel() +>>> print(">>>", res) +>>> async for chunk in r.stream: +>>> print("...", chunk) +``` + +--- + +## Sending HTTP requests + +Let's take a look at how we can work directly with a network backend to send an HTTP request, and recieve an HTTP response. + +

httpx

+ +```{ .python .httpx } +import httpx +import ssl +import truststore + +net = httpx.NetworkBackend() +ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT) +req = b'\r\n'.join([ + b'GET / HTTP/1.1', + b'Host: www.example.com', + b'User-Agent: python/dev', + b'Connection: close', + b'', + b'', +]) + +# Use a 10 second overall timeout for the entire request/response. +with httpx.timeout(10.0): + # Use a 3 second timeout for the initial connection. + with httpx.timeout(3.0) as t: + # Open the connection & establish SSL. + with net.connect("www.example.com", 443) as stream: + stream.start_tls(ctx, hostname="www.example.com") + t.cancel() + # Send the request & read the response. + stream.write(req) + buffer = [] + while part := stream.read(): + buffer.append(part) + resp = b''.join(buffer) +``` + +```{ .python .ahttpx .hidden } +import ahttpx +import ssl +import truststore + +net = ahttpx.NetworkBackend() +ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT) +req = b'\r\n'.join([ + b'GET / HTTP/1.1', + b'Host: www.example.com', + b'User-Agent: python/dev', + b'Connection: close', + b'', + b'', +]) + +# Use a 10 second overall timeout for the entire request/response. +async with ahttpx.timeout(10.0): + # Use a 3 second timeout for the initial connection. + async with ahttpx.timeout(3.0) as t: + # Open the connection & establish SSL. + async with await net.connect("www.example.com", 443) as stream: + await stream.start_tls(ctx, hostname="www.example.com") + t.cancel() + # Send the request & read the response. + await stream.write(req) + buffer = [] + while part := await stream.read(): + buffer.append(part) + resp = b''.join(buffer) +``` + +The example above is somewhat contrived, there's no HTTP parsing implemented so we can't actually determine when the response is complete. We're using a `Connection: close` header to request that the server close the connection once the response is complete. + +A more complete example would require proper HTTP parsing. The `Connection` class implements an HTTP request/response interface, layered over a `NetworkStream`. + +--- + +## Custom network backends + +The interface for implementing custom network backends is provided by two classes... + +### `NetworkBackendInterface` + +The abstract interface implemented by `NetworkBackend`. See above for details. + +### `NetworkStreamInterface` + +The abstract interface implemented by `NetworkStream`. See above for details. + +### An example backend + +We can use these interfaces to implement custom functionality. For example, here we're providing a network backend that logs all the ingoing and outgoing bytes. + +

httpx

+ +```{ .python .httpx } +class RecordingBackend(httpx.NetworkBackendInterface): + def __init__(self): + self._backend = NetworkBackend() + + def connect(self, host, port): + # Delegate creating connections to the default + # network backend, and return a wrapped stream. + stream = self._backend.connect(host, port) + return RecordingStream(stream) + + +class RecordingStream(httpx.NetworkStreamInterface): + def __init__(self, stream): + self._stream = stream + + def read(self, max_bytes: int = None): + # Print all incoming data to the terminal. + data = self._stream.read(max_bytes) + lines = data.decode('ascii', errors='replace').splitlines() + for line in lines: + print("<<< ", line) + return data + + def write(self, data): + # Print all outgoing data to the terminal. + lines = data.decode('ascii', errors='replace').splitlines() + for line in lines: + print(">>> ", line) + self._stream.write(data) + + def start_tls(ctx, hostname): + self._stream.start_tls(ctx, hostname) + + def get_extra_info(key): + return self._stream.get_extra_info(key) + + def close(): + self._stream.close() +``` + +```{ .python .ahttpx .hidden } +class RecordingBackend(ahhtpx.NetworkBackendInterface): + def __init__(self): + self._backend = NetworkBackend() + + async def connect(self, host, port): + # Delegate creating connections to the default + # network backend, and return a wrapped stream. + stream = await self._backend.connect(host, port) + return RecordingStream(stream) + + +class RecordingStream(ahttpx.NetworkStreamInterface): + def __init__(self, stream): + self._stream = stream + + async def read(self, max_bytes: int = None): + # Print all incoming data to the terminal. + data = await self._stream.read(max_bytes) + lines = data.decode('ascii', errors='replace').splitlines() + for line in lines: + print("<<< ", line) + return data + + async def write(self, data): + # Print all outgoing data to the terminal. + lines = data.decode('ascii', errors='replace').splitlines() + for line in lines: + print(">>> ", line) + await self._stream.write(data) + + async def start_tls(ctx, hostname): + await self._stream.start_tls(ctx, hostname) + + def get_extra_info(key): + return self._stream.get_extra_info(key) + + async def close(): + await self._stream.close() +``` + +We can now instantiate a client using this network backend. + +

httpx

+ +```{ .python .httpx } +>>> transport = httpx.ConnectionPool(backend=RecordingBackend()) +>>> cli = httpx.Client(transport=transport) +>>> cli.get('https://www.example.com') +``` + +```{ .python .ahttpx .hidden } +>>> transport = ahttpx.ConnectionPool(backend=RecordingBackend()) +>>> cli = ahttpx.Client(transport=transport) +>>> await cli.get('https://www.example.com') +``` + +Custom network backends can also be used to provide functionality such as handling DNS caching for name lookups, or connecting via a UNIX domain socket instead of a TCP connection. + +--- + +← [Parsers](parsers.md) + diff --git a/docs/overrides/partials/nav.html b/docs/overrides/partials/nav.html deleted file mode 100644 index d5a413f013..0000000000 --- a/docs/overrides/partials/nav.html +++ /dev/null @@ -1,54 +0,0 @@ -{% import "partials/nav-item.html" as item with context %} - - - {% set class = "md-nav md-nav--primary" %} - {% if "navigation.tabs" in features %} - {% set class = class ~ " md-nav--lifted" %} - {% endif %} - {% if "toc.integrate" in features %} - {% set class = class ~ " md-nav--integrated" %} - {% endif %} - - - - \ No newline at end of file diff --git a/docs/parsers.md b/docs/parsers.md new file mode 100644 index 0000000000..3416c923f4 --- /dev/null +++ b/docs/parsers.md @@ -0,0 +1,110 @@ +# Parsers + +### Client + +

httpx

+ +```{ .python .httpx } +stream = httpx.DuplexStream( + b'HTTP/1.1 200 OK\r\n' + b'Content-Length: 23\r\n' + b'Content-Type: application/json\r\n' + b'\r\n' + b'{"msg": "hello, world"}' +) +p = ahttpx.HTTPParser(stream, mode='CLIENT') + +# Send the request... +p.send_method_line(b'GET', b'/', b'HTTP/1.1') +p.send_headers([(b'Host', b'www.example.com')]) +p.send_body(b'') + +# Receive the response... +protocol, code, reason_phase = p.recv_status_line() +headers = p.recv_headers() +body = b'' +while buffer := p.recv_body(): + body += buffer +``` + +```{ .python .ahttpx .hidden } +stream = ahttpx.DuplexStream( + b'HTTP/1.1 200 OK\r\n' + b'Content-Length: 23\r\n' + b'Content-Type: application/json\r\n' + b'\r\n' + b'{"msg": "hello, world"}' +) +p = ahttpx.HTTPParser(stream, mode='CLIENT') + +# Send the request... +await p.send_method_line(b'GET', b'/', b'HTTP/1.1') +await p.send_headers([(b'Host', b'www.example.com')]) +await p.send_body(b'') + +# Receive the response... +protocol, code, reason_phase = await p.recv_status_line() +headers = await p.recv_headers() +body = b'' +while buffer := await p.recv_body(): + body += buffer +``` + +### Server + +

httpx

+ +```{ .python .httpx } +stream = httpx.DuplexStream( + b'GET / HTTP/1.1\r\n' + b'Host: www.example.com\r\n' + b'\r\n' +) +p = httpx.HTTPParser(stream, mode='SERVER') + +# Receive the request... +method, target, protocol = p.recv_method_line() +headers = p.recv_headers() +body = b'' +while buffer := p.recv_body(): + body += buffer + +# Send the response... +p.send_status_line(b'HTTP/1.1', 200, b'OK') +p.send_headers([ + (b'Content-Length', b'23'), + (b'Content-Type', b'application/json') +]) +p.send_body(b'{"msg": "hello, world"}') +p.send_body(b'') +``` + +```{ .python .ahttpx .hidden } +stream = ahttpx.DuplexStream( + b'GET / HTTP/1.1\r\n' + b'Host: www.example.com\r\n' + b'\r\n' +) +p = ahttpx.HTTPParser(stream, mode='SERVER') + +# Receive the request... +method, target, protocol = await p.recv_method_line() +headers = await p.recv_headers() +body = b'' +while buffer := await p.recv_body(): + body += buffer + +# Send the response... +await p.send_status_line(b'HTTP/1.1', 200, b'OK') +await p.send_headers([ + (b'Content-Length', b'23'), + (b'Content-Type', b'application/json') +]) +await p.send_body(b'{"msg": "hello, world"}') +await p.send_body(b'') +``` + +--- + +← [Connections](connections.md) +[Low Level Networking](networking.md) → diff --git a/docs/quickstart.md b/docs/quickstart.md index e140b53cd7..c3a6068253 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -1,146 +1,207 @@ # QuickStart -First, start by importing HTTPX: +Install using ... -```pycon +

httpx

+ +```{ .shell .httpx } +$ pip install --pre httpx +``` + +```{ .shell .ahttpx .hidden } +$ pip install --pre ahttpx +``` + +First, start by importing `httpx`... + +

httpx

+ +```{ .python .httpx } >>> import httpx ``` +```{ .python .ahttpx .hidden } +>>> import ahttpx +``` + Now, let’s try to get a webpage. -```pycon +

httpx

+ +```{ .python .httpx } >>> r = httpx.get('https://httpbin.org/get') >>> r ``` -Similarly, to make an HTTP POST request: +```{ .python .ahttpx .hidden } +>>> r = await ahttpx.get('https://httpbin.org/get') +>>> r + +``` + +To make an HTTP `POST` request, including some content... -```pycon ->>> r = httpx.post('https://httpbin.org/post', data={'key': 'value'}) +

httpx

+ +```{ .python .httpx } +>>> form = httpx.Form({'key': 'value'}) +>>> r = httpx.post('https://httpbin.org/post', content=form) +``` + +```{ .python .ahttpx .hidden } +>>> form = httpx.Form({'key': 'value'}) +>>> r = await ahttpx.post('https://httpbin.org/post', content=form) ``` -The PUT, DELETE, HEAD, and OPTIONS requests all follow the same style: +Shortcut methods for `PUT`, `PATCH`, and `DELETE` requests follow the same style... + +

httpx

-```pycon ->>> r = httpx.put('https://httpbin.org/put', data={'key': 'value'}) +```{ .python .httpx } +>>> r = httpx.put('https://httpbin.org/put', content=form) +>>> r = httpx.patch('https://httpbin.org/patch', content=form) >>> r = httpx.delete('https://httpbin.org/delete') ->>> r = httpx.head('https://httpbin.org/get') ->>> r = httpx.options('https://httpbin.org/get') +``` + +```{ .python .ahttpx .hidden } +>>> r = await ahttpx.put('https://httpbin.org/put', content=form) +>>> r = await ahttpx.patch('https://httpbin.org/patch', content=form) +>>> r = await ahttpx.delete('https://httpbin.org/delete') ``` ## Passing Parameters in URLs -To include URL query parameters in the request, use the `params` keyword: +To include URL query parameters in the request, construct a URL using the `params` keyword... + +

httpx

-```pycon +```{ .python .httpx } >>> params = {'key1': 'value1', 'key2': 'value2'} ->>> r = httpx.get('https://httpbin.org/get', params=params) +>>> url = httpx.URL('https://httpbin.org/get', params=params) +>>> r = httpx.get(url) ``` -To see how the values get encoding into the URL string, we can inspect the -resulting URL that was used to make the request: - -```pycon ->>> r.url -URL('https://httpbin.org/get?key2=value2&key1=value1') +```{ .python .ahttpx .hidden } +>>> params = {'key1': 'value1', 'key2': 'value2'} +>>> url = ahttpx.URL('https://httpbin.org/get', params=params) +>>> r = await ahttpx.get(url) ``` -You can also pass a list of items as a value: +You can also pass a list of items as a value... -```pycon +

httpx

+ +```{ .python .httpx } >>> params = {'key1': 'value1', 'key2': ['value2', 'value3']} ->>> r = httpx.get('https://httpbin.org/get', params=params) ->>> r.url -URL('https://httpbin.org/get?key1=value1&key2=value2&key2=value3') +>>> url = httpx.URL('https://httpbin.org/get', params=params) +>>> r = httpx.get(url) ``` -## Response Content +```{ .python .ahttpx .hidden } +>>> params = {'key1': 'value1', 'key2': ['value2', 'value3']} +>>> url = ahttpx.URL('https://httpbin.org/get', params=params) +>>> r = await ahttpx.get(url) +``` -HTTPX will automatically handle decoding the response content into Unicode text. +## Custom Headers -```pycon ->>> r = httpx.get('https://www.example.org/') ->>> r.text -'\n\n\nExample Domain...' -``` +To include additional headers in the outgoing request, use the `headers` keyword argument... -You can inspect what encoding will be used to decode the response. +

httpx

+ +```{ .python .httpx } +>>> url = 'https://httpbin.org/headers' +>>> headers = {'User-Agent': 'my-app/0.0.1'} +>>> r = httpx.get(url, headers=headers) +``` -```pycon ->>> r.encoding -'UTF-8' +```{ .python .ahttpx .hidden } +>>> url = 'https://httpbin.org/headers' +>>> headers = {'User-Agent': 'my-app/0.0.1'} +>>> r = await ahttpx.get(url, headers=headers) ``` -In some cases the response may not contain an explicit encoding, in which case HTTPX -will attempt to automatically determine an encoding to use. +--- + +## Response Content -```pycon ->>> r.encoding -None +HTTPX will automatically handle decoding the response content into unicode text. + +

httpx

+ +```{ .python .httpx } +>>> r = httpx.get('https://www.example.org/') >>> r.text '\n\n\nExample Domain...' ``` -If you need to override the standard behaviour and explicitly set the encoding to -use, then you can do that too. - -```pycon ->>> r.encoding = 'ISO-8859-1' +```{ .python .ahttpx .hidden } +>>> r = await ahttpx.get('https://www.example.org/') +>>> r.text +'\n\n\nExample Domain...' ``` ## Binary Response Content -The response content can also be accessed as bytes, for non-text responses: +The response content can also be accessed as bytes, for non-text responses. + +

httpx

-```pycon ->>> r.content +```{ .python .httpx } +>>> r.body b'\n\n\nExample Domain...' ``` -Any `gzip` and `deflate` HTTP response encodings will automatically -be decoded for you. If `brotlipy` is installed, then the `brotli` response -encoding will be supported. If `zstandard` is installed, then `zstd` -response encodings will also be supported. - -For example, to create an image from binary data returned by a request, you can use the following code: - -```pycon ->>> from PIL import Image ->>> from io import BytesIO ->>> i = Image.open(BytesIO(r.content)) +```{ .python .ahttpx .hidden } +>>> r.body +b'\n\n\nExample Domain...' ``` ## JSON Response Content Often Web API responses will be encoded as JSON. -```pycon ->>> r = httpx.get('https://api.github.com/events') +

httpx

+ +```{ .python .httpx } +>>> r = httpx.get('https://httpbin.org/get') >>> r.json() -[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...' ... }}] +{'args': {}, 'headers': {'Host': 'httpbin.org', 'User-Agent': 'dev', 'X-Amzn-Trace-Id': 'Root=1-679814d5-0f3d46b26686f5013e117085'}, 'origin': '21.35.60.128', 'url': 'https://httpbin.org/get'} ``` -## Custom Headers - -To include additional headers in the outgoing request, use the `headers` keyword argument: - -```pycon ->>> url = 'https://httpbin.org/headers' ->>> headers = {'user-agent': 'my-app/0.0.1'} ->>> r = httpx.get(url, headers=headers) +```{ .python .ahttpx .hidden } +>>> r = await ahttpx.get('https://httpbin.org/get') +>>> await r.json() +{'args': {}, 'headers': {'Host': 'httpbin.org', 'User-Agent': 'dev', 'X-Amzn-Trace-Id': 'Root=1-679814d5-0f3d46b26686f5013e117085'}, 'origin': '21.35.60.128', 'url': 'https://httpbin.org/get'} ``` +--- + ## Sending Form Encoded Data -Some types of HTTP requests, such as `POST` and `PUT` requests, can include data -in the request body. One common way of including that is as form-encoded data, -which is used for HTML forms. +Some types of HTTP requests, such as `POST` and `PUT` requests, can include data in the request body. One common way of including that is as form-encoded data, which is used for HTML forms. + +

httpx

+ +```{ .python .httpx } +>>> form = httpx.Form({'key1': 'value1', 'key2': 'value2'}) +>>> r = httpx.post("https://httpbin.org/post", content=form) +>>> r.json() +{ + ... + "form": { + "key2": "value2", + "key1": "value1" + }, + ... +} +``` -```pycon ->>> data = {'key1': 'value1', 'key2': 'value2'} ->>> r = httpx.post("https://httpbin.org/post", data=data) ->>> print(r.text) +```{ .python .ahttpx .hidden } +>>> form = ahttpx.Form({'key1': 'value1', 'key2': 'value2'}) +>>> r = await ahttpx.post("https://httpbin.org/post", content=form) +>>> await r.json() { ... "form": { @@ -153,10 +214,28 @@ which is used for HTML forms. Form encoded data can also include multiple values from a given key. -```pycon ->>> data = {'key1': ['value1', 'value2']} ->>> r = httpx.post("https://httpbin.org/post", data=data) ->>> print(r.text) +

httpx

+ +```{ .python .httpx } +>>> form = httpx.Form({'key1': ['value1', 'value2']}) +>>> r = httpx.post("https://httpbin.org/post", content=form) +>>> r.json() +{ + ... + "form": { + "key1": [ + "value1", + "value2" + ] + }, + ... +} +``` + +```{ .python .ahttpx .hidden } +>>> form = ahttpx.Form({'key1': ['value1', 'value2']}) +>>> r = await ahttpx.post("https://httpbin.org/post", content=form) +>>> await r.json() { ... "form": { @@ -171,34 +250,31 @@ Form encoded data can also include multiple values from a given key. ## Sending Multipart File Uploads -You can also upload files, using HTTP multipart encoding: +You can also upload files, using HTTP multipart encoding. + +

httpx

-```pycon ->>> with open('report.xls', 'rb') as report_file: -... files = {'upload-file': report_file} -... r = httpx.post("https://httpbin.org/post", files=files) ->>> print(r.text) +```{ .python .httpx } +>>> files = httpx.Files({'upload': httpx.File('uploads/report.xls')}) +>>> r = httpx.post("https://httpbin.org/post", content=files) +>>> r.json() { ... "files": { - "upload-file": "<... binary content ...>" + "upload": "<... binary content ...>" }, ... } ``` -You can also explicitly set the filename and content type, by using a tuple -of items for the file value: - -```pycon ->>> with open('report.xls', 'rb') as report_file: -... files = {'upload-file': ('report.xls', report_file, 'application/vnd.ms-excel')} -... r = httpx.post("https://httpbin.org/post", files=files) ->>> print(r.text) +```{ .python .ahttpx .hidden } +>>> files = ahttpx.Files({'upload': httpx.File('uploads/report.xls')}) +>>> r = await ahttpx.post("https://httpbin.org/post", content=files) +>>> await r.json() { ... "files": { - "upload-file": "<... binary content ...>" + "upload": "<... binary content ...>" }, ... } @@ -206,16 +282,36 @@ of items for the file value: If you need to include non-file data fields in the multipart form, use the `data=...` parameter: -```pycon ->>> data = {'message': 'Hello, world!'} ->>> with open('report.xls', 'rb') as report_file: -... files = {'file': report_file} -... r = httpx.post("https://httpbin.org/post", data=data, files=files) ->>> print(r.text) +

httpx

+ +```{ .python .httpx } +>>> form = {'message': 'Hello, world!'} +>>> files = {'upload': httpx.File('uploads/report.xls')} +>>> data = httpx.MultiPart(form=form, files=files) +>>> r = httpx.post("https://httpbin.org/post", content=data) +>>> r.json() { ... "files": { - "file": "<... binary content ...>" + "upload": "<... binary content ...>" + }, + "form": { + "message": "Hello, world!", + }, + ... +} +``` + +```{ .python .ahttpx .hidden } +>>> form = {'message': 'Hello, world!'} +>>> files = {'upload': httpx.File('uploads/report.xls')} +>>> data = ahttpx.MultiPart(form=form, files=files) +>>> r = await ahttpx.post("https://httpbin.org/post", content=data) +>>> await r.json() +{ + ... + "files": { + "upload": "<... binary content ...>" }, "form": { "message": "Hello, world!", @@ -229,10 +325,31 @@ If you need to include non-file data fields in the multipart form, use the `data Form encoded data is okay if all you need is a simple key-value data structure. For more complicated data structures you'll often want to use JSON encoding instead. -```pycon +

httpx

+ +```{ .python .httpx } >>> data = {'integer': 123, 'boolean': True, 'list': ['a', 'b', 'c']} ->>> r = httpx.post("https://httpbin.org/post", json=data) ->>> print(r.text) +>>> r = httpx.post("https://httpbin.org/post", content=httpx.JSON(data)) +>>> r.json() +{ + ... + "json": { + "boolean": true, + "integer": 123, + "list": [ + "a", + "b", + "c" + ] + }, + ... +} +``` + +```{ .python .ahttpx .hidden } +>>> data = {'integer': 123, 'boolean': True, 'list': ['a', 'b', 'c']} +>>> r = await ahttpx.post("https://httpbin.org/post", content=httpx.JSON(data)) +>>> await r.json() { ... "json": { @@ -253,88 +370,92 @@ For more complicated data structures you'll often want to use JSON encoding inst For other encodings, you should use the `content=...` parameter, passing either a `bytes` type or a generator that yields `bytes`. -```pycon +

httpx

+ +```{ .python .httpx } >>> content = b'Hello, world' >>> r = httpx.post("https://httpbin.org/post", content=content) ``` +```{ .python .ahttpx .hidden } +>>> content = b'Hello, world' +>>> r = await ahttpx.post("https://httpbin.org/post", content=content) +``` + You may also want to set a custom `Content-Type` header when uploading binary data. +--- + ## Response Status Codes We can inspect the HTTP status code of the response: -```pycon +

httpx

+ +```{ .python .httpx } >>> r = httpx.get('https://httpbin.org/get') >>> r.status_code 200 ``` -HTTPX also includes an easy shortcut for accessing status codes by their text phrase. - -```pycon ->>> r.status_code == httpx.codes.OK -True -``` - -We can raise an exception for any responses which are not a 2xx success code: - -```pycon ->>> not_found = httpx.get('https://httpbin.org/status/404') ->>> not_found.status_code -404 ->>> not_found.raise_for_status() -Traceback (most recent call last): - File "/Users/tomchristie/GitHub/encode/httpcore/httpx/models.py", line 837, in raise_for_status - raise HTTPStatusError(message, response=self) -httpx._exceptions.HTTPStatusError: 404 Client Error: Not Found for url: https://httpbin.org/status/404 -For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404 +```{ .python .ahttpx .hidden } +>>> r = await ahttpx.get('https://httpbin.org/get') +>>> r.status_code +200 ``` -Any successful response codes will return the `Response` instance rather than raising an exception. +## Response Headers -```pycon ->>> r.raise_for_status() -``` +The response headers are available as a dictionary-like interface. -The method returns the response instance, allowing you to use it inline. For example: +

httpx

-```pycon ->>> r = httpx.get('...').raise_for_status() ->>> data = httpx.get('...').raise_for_status().json() +```{ .python .httpx } +>>> r.headers + ``` -## Response Headers - -The response headers are available as a dictionary-like interface. - -```pycon +```{ .python .ahttpx .hidden } >>> r.headers -Headers({ - 'content-encoding': 'gzip', - 'transfer-encoding': 'chunked', - 'connection': 'close', - 'server': 'nginx/1.0.4', - 'x-runtime': '148ms', - 'etag': '"e1ca502697e5c9317743dc078f67693f"', - 'content-type': 'application/json' -}) + ``` The `Headers` data type is case-insensitive, so you can use any capitalization. -```pycon ->>> r.headers['Content-Type'] +

httpx

+ +```{ .python .httpx } +>>> r.headers.get('Content-Type') 'application/json' >>> r.headers.get('content-type') 'application/json' ``` -Multiple values for a single response header are represented as a single comma-separated value, as per [RFC 7230](https://tools.ietf.org/html/rfc7230#section-3.2): +```{ .python .ahttpx .hidden } +>>> r.headers.get('Content-Type') +'application/json' + +>>> r.headers.get('content-type') +'application/json' +``` -> A recipient MAY combine multiple header fields with the same field name into one “field-name: field-value” pair, without changing the semantics of the message, by appending each subsequent field-value to the combined field value in order, separated by a comma. +--- ## Streaming Responses @@ -342,206 +463,22 @@ For large downloads you may want to use streaming responses that do not load the You can stream the binary content of the response... -```pycon ->>> with httpx.stream("GET", "https://www.example.com") as r: -... for data in r.iter_bytes(): -... print(data) -``` - -Or the text of the response... +

httpx

-```pycon +```{ .python .httpx } >>> with httpx.stream("GET", "https://www.example.com") as r: -... for text in r.iter_text(): -... print(text) -``` - -Or stream the text, on a line-by-line basis... - -```pycon ->>> with httpx.stream("GET", "https://www.example.com") as r: -... for line in r.iter_lines(): -... print(line) -``` - -HTTPX will use universal line endings, normalising all cases to `\n`. - -In some cases you might want to access the raw bytes on the response without applying any HTTP content decoding. In this case any content encoding that the web server has applied such as `gzip`, `deflate`, `brotli`, or `zstd` will -not be automatically decoded. - -```pycon ->>> with httpx.stream("GET", "https://www.example.com") as r: -... for chunk in r.iter_raw(): -... print(chunk) -``` - -If you're using streaming responses in any of these ways then the `response.content` and `response.text` attributes will not be available, and will raise errors if accessed. However you can also use the response streaming functionality to conditionally load the response body: - -```pycon ->>> with httpx.stream("GET", "https://www.example.com") as r: -... if int(r.headers['Content-Length']) < TOO_LONG: -... r.read() -... print(r.text) -``` - -## Cookies - -Any cookies that are set on the response can be easily accessed: - -```pycon ->>> r = httpx.get('https://httpbin.org/cookies/set?chocolate=chip') ->>> r.cookies['chocolate'] -'chip' -``` - -To include cookies in an outgoing request, use the `cookies` parameter: - -```pycon ->>> cookies = {"peanut": "butter"} ->>> r = httpx.get('https://httpbin.org/cookies', cookies=cookies) ->>> r.json() -{'cookies': {'peanut': 'butter'}} -``` - -Cookies are returned in a `Cookies` instance, which is a dict-like data structure -with additional API for accessing cookies by their domain or path. - -```pycon ->>> cookies = httpx.Cookies() ->>> cookies.set('cookie_on_domain', 'hello, there!', domain='httpbin.org') ->>> cookies.set('cookie_off_domain', 'nope.', domain='example.org') ->>> r = httpx.get('http://httpbin.org/cookies', cookies=cookies) ->>> r.json() -{'cookies': {'cookie_on_domain': 'hello, there!'}} -``` - -## Redirection and History - -By default, HTTPX will **not** follow redirects for all HTTP methods, although -this can be explicitly enabled. - -For example, GitHub redirects all HTTP requests to HTTPS. - -```pycon ->>> r = httpx.get('http://github.com/') ->>> r.status_code -301 ->>> r.history -[] ->>> r.next_request - -``` - -You can modify the default redirection handling with the `follow_redirects` parameter: - -```pycon ->>> r = httpx.get('http://github.com/', follow_redirects=True) ->>> r.url -URL('https://github.com/') ->>> r.status_code -200 ->>> r.history -[] -``` - -The `history` property of the response can be used to inspect any followed redirects. -It contains a list of any redirect responses that were followed, in the order -in which they were made. - -## Timeouts - -HTTPX defaults to including reasonable timeouts for all network operations, -meaning that if a connection is not properly established then it should always -raise an error rather than hanging indefinitely. - -The default timeout for network inactivity is five seconds. You can modify the -value to be more or less strict: - -```pycon ->>> httpx.get('https://github.com/', timeout=0.001) -``` - -You can also disable the timeout behavior completely... - -```pycon ->>> httpx.get('https://github.com/', timeout=None) -``` - -For advanced timeout management, see [Timeout fine-tuning](advanced/timeouts.md#fine-tuning-the-configuration). - -## Authentication - -HTTPX supports Basic and Digest HTTP authentication. - -To provide Basic authentication credentials, pass a 2-tuple of -plaintext `str` or `bytes` objects as the `auth` argument to the request -functions: - -```pycon ->>> httpx.get("https://example.com", auth=("my_user", "password123")) -``` - -To provide credentials for Digest authentication you'll need to instantiate -a `DigestAuth` object with the plaintext username and password as arguments. -This object can be then passed as the `auth` argument to the request methods -as above: - -```pycon ->>> auth = httpx.DigestAuth("my_user", "password123") ->>> httpx.get("https://example.com", auth=auth) - -``` - -## Exceptions - -HTTPX will raise exceptions if an error occurs. - -The most important exception classes in HTTPX are `RequestError` and `HTTPStatusError`. - -The `RequestError` class is a superclass that encompasses any exception that occurs -while issuing an HTTP request. These exceptions include a `.request` attribute. - -```python -try: - response = httpx.get("https://www.example.com/") -except httpx.RequestError as exc: - print(f"An error occurred while requesting {exc.request.url!r}.") -``` - -The `HTTPStatusError` class is raised by `response.raise_for_status()` on responses which are not a 2xx success code. -These exceptions include both a `.request` and a `.response` attribute. - -```python -response = httpx.get("https://www.example.com/") -try: - response.raise_for_status() -except httpx.HTTPStatusError as exc: - print(f"Error response {exc.response.status_code} while requesting {exc.request.url!r}.") +... for data in r.stream: +... print(data) ``` -There is also a base class `HTTPError` that includes both of these categories, and can be used -to catch either failed requests, or 4xx and 5xx responses. - -You can either use this base class to catch both categories... - -```python -try: - response = httpx.get("https://www.example.com/") - response.raise_for_status() -except httpx.HTTPError as exc: - print(f"Error while requesting {exc.request.url!r}.") +```{ .python .ahttpx .hidden } +>>> async with ahttpx.stream("GET", "https://www.example.com") as r: +... async for data in r.stream: +... print(data) ``` -Or handle each case explicitly... - -```python -try: - response = httpx.get("https://www.example.com/") - response.raise_for_status() -except httpx.RequestError as exc: - print(f"An error occurred while requesting {exc.request.url!r}.") -except httpx.HTTPStatusError as exc: - print(f"Error response {exc.response.status_code} while requesting {exc.request.url!r}.") -``` +--- -For a full list of available exceptions, see [Exceptions (API Reference)](exceptions.md). +← [Home](index.md) +[Clients](clients.md) → + \ No newline at end of file diff --git a/docs/requests.md b/docs/requests.md new file mode 100644 index 0000000000..7f27125189 --- /dev/null +++ b/docs/requests.md @@ -0,0 +1,178 @@ +# Requests + +The core elements of an HTTP request are the `method`, `url`, `headers` and `body`. + +

httpx

+ +```{ .python .httpx } +>>> req = httpx.Request('GET', 'https://www.example.com/') +>>> req + +>>> req.method +'GET' +>>> req.url + +>>> req.headers + +>>> req.body +b'' +``` + +```{ .python .ahttpx .hidden } +>>> req = ahttpx.Request('GET', 'https://www.example.com/') +>>> req + +>>> req.method +'GET' +>>> req.url + +>>> req.headers + +>>> req.body +b'' +``` + +## Working with the request headers + +The following headers have automatic behavior with `Requests` instances... + +* `Host` - A `Host` header must always be included on a request. This header is automatically populated from the `url`, using the `url.netloc` property. +* `Content-Length` - Requests including a request body must always include either a `Content-Length` header or a `Transfer-Encoding: chunked` header. This header is automatically populated if `content` is not `None` and the content is a known size. +* `Transfer-Encoding` - Requests automatically include a `Transfer-Encoding: chunked` header if `content` is not `None` and the content is an unkwown size. +* `Content-Type` - Requests automatically include a `Content-Type` header if `content` is set using the [Content Type] API. + +## Working with the request body + +Including binary data directly... + +

httpx

+ +```{ .python .httpx } +>>> headers = {'Content-Type': 'application/json'} +>>> content = json.dumps(...) +>>> httpx.Request('POST', 'https://echo.encode.io/', content=content) +``` + +```{ .python .ahttpx .hidden } +>>> headers = {'Content-Type': 'application/json'} +>>> content = json.dumps(...) +>>> ahttpx.Request('POST', 'https://echo.encode.io/', content=content) +``` + +## Working with content types + +Including JSON request content... + +

httpx

+ +```{ .python .httpx } +>>> data = httpx.JSON(...) +>>> httpx.Request('POST', 'https://echo.encode.io/', content=data) +``` + +```{ .python .ahttpx .hidden } +>>> data = ahttpx.JSON(...) +>>> ahttpx.Request('POST', 'https://echo.encode.io/', content=data) +``` + +Including form encoded request content... + +

httpx

+ +```{ .python .httpx } +>>> data = httpx.Form(...) +>>> httpx.Request('PUT', 'https://echo.encode.io/', content=data) +``` + +```{ .python .ahttpx .hidden } +>>> data = ahttpx.Form(...) +>>> ahttpx.Request('PUT', 'https://echo.encode.io/', content=data) +``` + +Including multipart file uploads... + +

httpx

+ +```{ .python .httpx } +>>> form = httpx.MultiPart(form={...}, files={...}) +>>> with httpx.Request('POST', 'https://echo.encode.io/', content=form) as req: +>>> req.headers +{...} +>>> req.stream + +``` + +```{ .python .ahttpx .hidden } +>>> form = ahttpx.MultiPart(form={...}, files={...}) +>>> async with ahttpx.Request('POST', 'https://echo.encode.io/', content=form) as req: +>>> req.headers +{...} +>>> req.stream + +``` + +Including direct file uploads... + +

httpx

+ +```{ .python .httpx } +>>> file = httpx.File('upload.json') +>>> with httpx.Request('POST', 'https://echo.encode.io/', content=file) as req: +>>> req.headers +{...} +>>> req.stream + +``` + +```{ .python .ahttpx .hidden } +>>> file = ahttpx.File('upload.json') +>>> async with ahttpx.Request('POST', 'https://echo.encode.io/', content=file) as req: +>>> req.headers +{...} +>>> req.stream + +``` + +## Accessing request content + +*In progress...* + +

httpx

+ +```{ .python .httpx } +>>> data = request.json() +``` + +```{ .python .ahttpx .hidden } +>>> data = await request.json() +``` + +... + +

httpx

+ +```{ .python .httpx } +>>> form = request.form() +``` + +```{ .python .ahttpx .hidden } +>>> form = await request.form() +``` + +... + +

httpx

+ +```{ .python .httpx } +>>> files = request.files() +``` + +```{ .python .ahttpx .hidden } +>>> files = await request.files() +``` + +--- + +← [Servers](servers.md) +[Responses](responses.md) → + diff --git a/docs/responses.md b/docs/responses.md new file mode 100644 index 0000000000..58ef2e494d --- /dev/null +++ b/docs/responses.md @@ -0,0 +1,131 @@ +# Responses + +The core elements of an HTTP response are the `status_code`, `headers` and `body`. + +

httpx

+ +```{ .python .httpx } +>>> resp = httpx.Response(200, headers={'Content-Type': 'text/plain'}, content=b'hello, world') +>>> resp + +>>> resp.status_code +200 +>>> resp.headers + +>>> resp.body +b'hello, world' +``` + +```{ .python .ahttpx .hidden } +>>> resp = ahttpx.Response(200, headers={'Content-Type': 'text/plain'}, content=b'hello, world') +>>> resp + +>>> resp.status_code +200 +>>> resp.headers + +>>> resp.body +b'hello, world' +``` + +## Working with the response headers + +The following headers have automatic behavior with `Response` instances... + +* `Content-Length` - Responses including a response body must always include either a `Content-Length` header or a `Transfer-Encoding: chunked` header. This header is automatically populated if `content` is not `None` and the content is a known size. +* `Transfer-Encoding` - Responses automatically include a `Transfer-Encoding: chunked` header if `content` is not `None` and the content is an unkwown size. +* `Content-Type` - Responses automatically include a `Content-Type` header if `content` is set using the [Content Type] API. + +## Working with content types + +Including HTML content... + +

httpx

+ +```{ .python .httpx } +>>> content = httpx.HTML('......') +>>> response = httpx.Response(200, content=content) +``` + +```{ .python .ahttpx .hidden } +>>> content = ahttpx.HTML('......') +>>> response = ahttpx.Response(200, content=content) +``` + +Including plain text content... + +

httpx

+ +```{ .python .httpx } +>>> content = httpx.Text('hello, world') +>>> response = httpx.Response(200, content=content) +``` + +```{ .python .ahttpx .hidden } +>>> content = ahttpx.Text('hello, world') +>>> response = ahttpx.Response(200, content=content) +``` + +Including JSON data... + +

httpx

+ +```{ .python .httpx } +>>> content = httpx.JSON({'message': 'hello, world'}) +>>> response = httpx.Response(200, content=content) +``` + +```{ .python .ahttpx .hidden } +>>> content = ahttpx.JSON({'message': 'hello, world'}) +>>> response = ahttpx.Response(200, content=content) +``` + +Including content from a file... + +

httpx

+ +```{ .python .httpx } +>>> content = httpx.File('index.html') +>>> with httpx.Response(200, content=content) as response: +... pass +``` + +```{ .python .ahttpx .hidden } +>>> content = ahttpx.File('index.html') +>>> async with ahttpx.Response(200, content=content) as response: +... pass +``` + +## Accessing response content + +... + +

httpx

+ +```{ .python .httpx } +>>> response.body +``` + +```{ .python .ahttpx .hidden } +>>> response.body +``` + +... + +

httpx

+ +```{ .python .httpx } +>>> response.text +... +``` + +```{ .python .ahttpx .hidden } +>>> response.text +... +``` + +--- + +← [Requests](requests.md) +[URLs](urls.md) → + diff --git a/docs/servers.md b/docs/servers.md new file mode 100644 index 0000000000..57e79c33b8 --- /dev/null +++ b/docs/servers.md @@ -0,0 +1,85 @@ +# Servers + +The HTTP server provides a simple request/response API. +This gives you a lightweight way to build web applications or APIs. + +### `serve_http(endpoint)` + +

httpx

+ +```{ .python .httpx } +>>> website = """ +... +... +... +... +... +...

hello, world

+... +... +... """ + +>>> def hello_world(request): +... content = httpx.HTML(website) +... return httpx.Response(200, content=content) + +>>> with httpx.serve_http(hello_world) as server: +... print(f"Serving on {server.url} (Press CTRL+C to quit)") +... server.wait() +Serving on http://127.0.0.1:8080/ (Press CTRL+C to quit) +``` + +```{ .python .ahttpx .hidden } +>>> import httpx + +>>> website = """ +... +... +... +... +... +...

hello, world

+... +... +... """ + +>>> async def hello_world(request): +... if request.path != '/': +... content = httpx.Text("Not found") +... return httpx.Response(404, content=content) +... content = httpx.HTML(website) +... return httpx.Response(200, content=content) + +>>> async with httpx.serve_http(hello_world) as server: +... print(f"Serving on {server.url} (Press CTRL+C to quit)") +... await server.wait() +Serving on http://127.0.0.1:8080/ (Press CTRL+C to quit) +``` + +--- + +*Docs in progress...* + +--- + +← [Clients](clients.md) +[Requests](requests.md) → + diff --git a/docs/streams.md b/docs/streams.md new file mode 100644 index 0000000000..53c32d6821 --- /dev/null +++ b/docs/streams.md @@ -0,0 +1,88 @@ +# Streams + +Streams provide a minimal file-like interface for reading bytes from a data source. They are used as the abstraction for reading the body of a request or response. + +The interfaces here are simplified versions of Python's standard I/O operations. + +## Stream + +The base `Stream` class. The core of the interface is a subset of Python's `io.IOBase`... + +* `.read(size=-1)` - *(bytes)* Return the bytes from the data stream. If the `size` argument is omitted or negative then the entire stream will be read. If `size` is an positive integer then the call returns at most `size` bytes. A return value of `b''` indicates the end of the stream has been reached. +* `.write(self, data: bytes)` - *None* Write the given bytes to the data stream. May raise `NotImplmentedError` if this is not a writeable stream. +* `.close()` - Close the stream. Any further operations will raise a `ValueError`. + +Additionally, the following property is also defined... + +* `.size` - *(int or None)* Return an integer indicating the size of the stream, or `None` if the size is unknown. When working with HTTP this is used to either set a `Content-Length: ` header, or a `Content-Encoding: chunked` header. + +The `Stream` interface and `ContentType` interface are related, with streams being used as the abstraction for the bytewise representation, and content types being used to encapsulate the parsed data structure. + +For example, encoding some `JSON` data... + +```python +>>> data = httpx.JSON({'name': 'zelda', 'score': '478'}) +>>> stream = data.encode() +>>> stream.read() +b'{"name":"zelda","score":"478"}' +>>> stream.content_type +'application/json' +``` + +--- + +## ByteStream + +A byte stream returning fixed byte content. Similar to Python's `io.BytesIO` class. + +```python +>>> s = httpx.ByteStream(b'{"msg": "Hello, world!"}') +>>> s.read() +b'{"msg": "Hello, world!"}' +``` + +## FileStream + +A byte stream returning content from a file. + +The standard pattern for instantiating a `FileStream` is to use `File` as a context manager: + +```python +>>> with httpx.File('upload.json') as s: +... s.read() +b'{"msg": "Hello, world!"}' +``` + +## MultiPartStream + +A byte stream returning multipart upload data. + +The standard pattern for instantiating a `MultiPartStream` is to use `MultiPart` as a context manager: + +```python +>>> files = {'avatar-upload': 'image.png'} +>>> with httpx.MultiPart(files=files) as s: +... s.read() +# ... +``` + +## HTTPStream + +A byte stream returning unparsed content from an HTTP request or response. + +```python +>>> with httpx.Client() as cli: +... r = cli.get('https://www.example.com/') +... r.stream.read() +# ... +``` + +## GZipStream + +... + +--- + +← [Content Types](content-types.md) +[Connections](connections.md) → + diff --git a/docs/templates/base.html b/docs/templates/base.html new file mode 100644 index 0000000000..22fe4d3796 --- /dev/null +++ b/docs/templates/base.html @@ -0,0 +1,186 @@ + + + + + + httpx + + + + + + + + + + + + + +

+ {{ content }} +

+ + \ No newline at end of file diff --git a/docs/third_party_packages.md b/docs/third_party_packages.md deleted file mode 100644 index 253c312f58..0000000000 --- a/docs/third_party_packages.md +++ /dev/null @@ -1,107 +0,0 @@ -# Third Party Packages - -As HTTPX usage grows, there is an expanding community of developers building tools and libraries that integrate with HTTPX, or depend on HTTPX. Here are some of them. - - - -## Plugins - -### Hishel - -[GitHub](https://github.com/karpetrosyan/hishel) - [Documentation](https://hishel.com/) - -An elegant HTTP Cache implementation for HTTPX and HTTP Core. - -### HTTPX-Auth - -[GitHub](https://github.com/Colin-b/httpx_auth) - [Documentation](https://colin-b.github.io/httpx_auth/) - -Provides authentication classes to be used with HTTPX's [authentication parameter](advanced/authentication.md#customizing-authentication). - -### httpx-caching - -[Github](https://github.com/johtso/httpx-caching) - -This package adds caching functionality to HTTPX - -### httpx-secure - -[GitHub](https://github.com/Zaczero/httpx-secure) - -Drop-in SSRF protection for httpx with DNS caching and custom validation support. - -### httpx-socks - -[GitHub](https://github.com/romis2012/httpx-socks) - -Proxy (HTTP, SOCKS) transports for httpx. - -### httpx-sse - -[GitHub](https://github.com/florimondmanca/httpx-sse) - -Allows consuming Server-Sent Events (SSE) with HTTPX. - -### httpx-retries - -[GitHub](https://github.com/will-ockmore/httpx-retries) - [Documentation](https://will-ockmore.github.io/httpx-retries/) - -A retry layer for HTTPX. - -### httpx-ws - -[GitHub](https://github.com/frankie567/httpx-ws) - [Documentation](https://frankie567.github.io/httpx-ws/) - -WebSocket support for HTTPX. - -### pytest-HTTPX - -[GitHub](https://github.com/Colin-b/pytest_httpx) - [Documentation](https://colin-b.github.io/pytest_httpx/) - -Provides a [pytest](https://docs.pytest.org/en/latest/) fixture to mock HTTPX within test cases. - -### RESPX - -[GitHub](https://github.com/lundberg/respx) - [Documentation](https://lundberg.github.io/respx/) - -A utility for mocking out HTTPX. - -### rpc.py - -[Github](https://github.com/abersheeran/rpc.py) - [Documentation](https://github.com/abersheeran/rpc.py#rpcpy) - -A fast and powerful RPC framework based on ASGI/WSGI. Use HTTPX as the client of the RPC service. - -## Libraries with HTTPX support - -### Authlib - -[GitHub](https://github.com/lepture/authlib) - [Documentation](https://docs.authlib.org/en/latest/) - -A python library for building OAuth and OpenID Connect clients and servers. Includes an [OAuth HTTPX client](https://docs.authlib.org/en/latest/client/httpx.html). - -### Gidgethub - -[GitHub](https://github.com/brettcannon/gidgethub) - [Documentation](https://gidgethub.readthedocs.io/en/latest/index.html) - -An asynchronous GitHub API library. Includes [HTTPX support](https://gidgethub.readthedocs.io/en/latest/httpx.html). - -### httpdbg - -[GitHub](https://github.com/cle-b/httpdbg) - [Documentation](https://httpdbg.readthedocs.io/) - -A tool for python developers to easily debug the HTTP(S) client requests in a python program. - -### VCR.py - -[GitHub](https://github.com/kevin1024/vcrpy) - [Documentation](https://vcrpy.readthedocs.io/) - -Record and repeat requests. - -## Gists - -### urllib3-transport - -[GitHub](https://gist.github.com/florimondmanca/d56764d78d748eb9f73165da388e546e) - -This public gist provides an example implementation for a [custom transport](advanced/transports.md#custom-transports) implementation on top of the battle-tested [`urllib3`](https://urllib3.readthedocs.io) library. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md deleted file mode 100644 index a2ca15f564..0000000000 --- a/docs/troubleshooting.md +++ /dev/null @@ -1,63 +0,0 @@ -# Troubleshooting - -This page lists some common problems or issues you could encounter while developing with HTTPX, as well as possible solutions. - -## Proxies - ---- - -### "`The handshake operation timed out`" on HTTPS requests when using a proxy - -**Description**: When using a proxy and making an HTTPS request, you see an exception looking like this: - -```console -httpx.ProxyError: _ssl.c:1091: The handshake operation timed out -``` - -**Similar issues**: [encode/httpx#1412](https://github.com/encode/httpx/issues/1412), [encode/httpx#1433](https://github.com/encode/httpx/issues/1433) - -**Resolution**: it is likely that you've set up your proxies like this... - -```python -mounts = { - "http://": httpx.HTTPTransport(proxy="http://myproxy.org"), - "https://": httpx.HTTPTransport(proxy="https://myproxy.org"), -} -``` - -Using this setup, you're telling HTTPX to connect to the proxy using HTTP for HTTP requests, and using HTTPS for HTTPS requests. - -But if you get the error above, it is likely that your proxy doesn't support connecting via HTTPS. Don't worry: that's a [common gotcha](advanced/proxies.md#http-proxies). - -Change the scheme of your HTTPS proxy to `http://...` instead of `https://...`: - -```python -mounts = { - "http://": httpx.HTTPTransport(proxy="http://myproxy.org"), - "https://": httpx.HTTPTransport(proxy="http://myproxy.org"), -} -``` - -This can be simplified to: - -```python -proxy = "http://myproxy.org" -with httpx.Client(proxy=proxy) as client: - ... -``` - -For more information, see [Proxies: FORWARD vs TUNNEL](advanced/proxies.md#forward-vs-tunnel). - ---- - -### Error when making requests to an HTTPS proxy - -**Description**: your proxy _does_ support connecting via HTTPS, but you are seeing errors along the lines of... - -```console -httpx.ProxyError: [SSL: PRE_MAC_LENGTH_TOO_LONG] invalid alert (_ssl.c:1091) -``` - -**Similar issues**: [encode/httpx#1424](https://github.com/encode/httpx/issues/1424). - -**Resolution**: HTTPX does not properly support HTTPS proxies at this time. If that's something you're interested in having, please see [encode/httpx#1434](https://github.com/encode/httpx/issues/1434) and consider lending a hand there. diff --git a/docs/urls.md b/docs/urls.md new file mode 100644 index 0000000000..ef56b18495 --- /dev/null +++ b/docs/urls.md @@ -0,0 +1,240 @@ +# URLs + +The `URL` class handles URL validation and parsing. + +

httpx

+ +```{ .python .httpx } +>>> url = httpx.URL('https://www.example.com/') +>>> url + +``` + +```{ .python .ahttpx .hidden } +>>> url = ahttpx.URL('https://www.example.com/') +>>> url + +``` + +URL components are normalised, following the same rules as internet browsers. + +

httpx

+ +```{ .python .httpx } +>>> url = httpx.URL('https://www.EXAMPLE.com:443/path/../main') +>>> url + +``` + +```{ .python .ahttpx .hidden } +>>> url = ahttpx.URL('https://www.EXAMPLE.com:443/path/../main') +>>> url + +``` + +Both absolute and relative URLs are valid. + +

httpx

+ +```{ .python .httpx } +>>> url = httpx.URL('/README.md') +>>> url + +``` + +```{ .python .ahttpx .hidden } +>>> url = ahttpx.URL('/README.md') +>>> url + +``` + +Coercing a URL to a `str` will always result in a printable ASCII string. + +

httpx

+ +```{ .python .httpx } +>>> url = httpx.URL('https://example.com/path to here?search=🦋') +>>> str(url) +'https://example.com/path%20to%20here?search=%F0%9F%A6%8B' +``` + +```{ .python .ahttpx .hidden } +>>> url = ahttpx.URL('https://example.com/path to here?search=🦋') +>>> str(url) +'https://example.com/path%20to%20here?search=%F0%9F%A6%8B' +``` + +### URL components + +The following properties are available for accessing the component parts of a URL. + +* `.scheme` - *str. ASCII. Normalised to lowercase.* +* `.userinfo` - *str. ASCII. URL encoded.* +* `.username` - *str. Unicode.* +* `.password` - *str. Unicode.* +* `.host` - *str. ASCII. IDNA encoded.* +* `.port` - *int or None. Scheme default ports are normalised to None.* +* `.authority` - *str. ASCII. IDNA encoded. Eg. "example.com", "example.com:1337", "xn--p1ai".* +* `.path` - *str. Unicode.* +* `.query` - *str. ASCII. URL encoded.* +* `.target` - *str. ASCII. URL encoded.* +* `.fragment` - *str. ASCII. URL encoded.* + +A parsed representation of the query parameters is accessible with the `.params` property. + +* `.params` - [`QueryParams`](#query-parameters) + +URLs can be instantiated from their components... + +

httpx

+ +```{ .python .httpx } +>>> httpx.URL(scheme="https", host="example.com", path="/") + +``` + +```{ .python .ahttpx .hidden } +>>> ahttpx.URL(scheme="https", host="example.com", path="/") + +``` + +Or using both the string form and query parameters... + +

httpx

+ +```{ .python .httpx } +>>> httpx.URL("https://example.com/", params={"search": "some text"}) + +``` + +```{ .python .ahttpx .hidden } +>>> ahttpx.URL("https://example.com/", params={"search": "some text"}) + +``` + +### Modifying URLs + +Instances of `URL` are immutable, meaning their value cannot be changed. Instead new modified instances may be created. + +* `.copy_with(**components)` - *Return a new URL, updating one or more components. Eg. `url = url.copy_with(scheme="https")`*. +* `.copy_set_param(key, value)` - *Return a new URL, setting a query parameter. Eg. `url = url.copy_set_param("sort_by", "price")`*. +* `.copy_append_param(key, value)` - *Return a new URL, setting or appending a query parameter. Eg. `url = url.copy_append_param("tag", "sale")`*. +* `.copy_remove_param(key)` - *Return a new URL, removing a query parameter. Eg. `url = url.copy_remove_param("max_price")`*. +* `.copy_update_params(params)` - *Return a new URL, updating the query parameters. Eg. `url = url.copy_update_params({"color_scheme": "dark"})`*. +* `.join(url)` - *Return a new URL, given this URL as the base and another URL as the target. Eg. `url = url.join("../navigation")`*. + +--- + +## Query Parameters + +The `QueryParams` class provides an immutable multi-dict for accessing URL query parameters. + +They can be instantiated from a dictionary. + +

httpx

+ +```{ .python .httpx } +>>> params = httpx.QueryParams({"color": "black", "size": "medium"}) +>>> params + +``` + +```{ .python .ahttpx .hidden } +>>> params = ahttpx.QueryParams({"color": "black", "size": "medium"}) +>>> params + +``` + +Multiple values for a single key are valid. + +

httpx

+ +```{ .python .httpx } +>>> params = httpx.QueryParams({"filter": ["60GHz", "75GHz", "100GHz"]}) +>>> params + +``` + +```{ .python .ahttpx .hidden } +>>> params = ahttpx.QueryParams({"filter": ["60GHz", "75GHz", "100GHz"]}) +>>> params + +``` + +They can also be instantiated directly from a query string. + +

httpx

+ +```{ .python .httpx } +>>> params = httpx.QueryParams("color=black&size=medium") +>>> params + +``` + +```{ .python .ahttpx .hidden } +>>> params = ahttpx.QueryParams("color=black&size=medium") +>>> params + +``` + +Keys and values are always represented as strings. + +

httpx

+ +```{ .python .httpx } +>>> params = httpx.QueryParams("sort_by=published&author=natalie") +>>> params["sort_by"] +'published' +``` + +```{ .python .ahttpx .hidden } +>>> params = ahttpx.QueryParams("sort_by=published&author=natalie") +>>> params["sort_by"] +'published' +``` + +When coercing query parameters to strings you'll see the same escaping behavior as HTML form submissions. The result will always be a printable ASCII string. + +

httpx

+ +```{ .python .httpx } +>>> params = httpx.QueryParams({"email": "user@example.com", "search": "How HTTP works!"}) +>>> str(params) +'email=user%40example.com&search=How+HTTP+works%21' +``` + +```{ .python .ahttpx .hidden } +>>> params = ahttpx.QueryParams({"email": "user@example.com", "search": "How HTTP works!"}) +>>> str(params) +'email=user%40example.com&search=How+HTTP+works%21' +``` + +### Accessing query parameters + +Query parameters are accessed using a standard dictionary style interface... + +* `.get(key, default=None)` - *Return the value for a given key, or a default value. If multiple values for the key are present, only the first will be returned.* +* `.keys()` - *Return the unique keys of the query parameters. Each key will be a `str` instance.* +* `.values()` - *Return the values of the query parameters. Each value will be a list of one or more `str` instances.* +* `.items()` - *Return the key value pairs of the query parameters. Each item will be a two-tuple including a `str` instance as the key, and a list of one or more `str` instances as the value.* + +The following methods are also available for accessing query parameters as a multidict... + +* `.get_all(key)` - *Return all the values for a given key. Returned as a list of zero or more `str` instances.* +* `.multi_items()` - *Return the key value pairs of the query parameters. Each item will be a two-tuple `(str, str)`. Repeated keys may occur.* +* `.multi_dict()` - *Return the query parameters as a dictionary, with each value being a list of one or more `str` instances.* + +### Modifying query parameters + +The following methods can be used to create modified query parameter instances... + +* `.copy_set(key, value)` +* `.copy_append(key, value)` +* `.copy_remove(key)` +* `.copy_update(params)` + +--- + +← [Responses](responses.md) +[Headers](headers.md) → + \ No newline at end of file diff --git a/httpx/__init__.py b/httpx/__init__.py deleted file mode 100644 index 63225040ba..0000000000 --- a/httpx/__init__.py +++ /dev/null @@ -1,106 +0,0 @@ -from .__version__ import __description__, __title__, __version__ -from ._api import * -from ._auth import * -from ._client import * -from ._config import * -from ._content import * -from ._exceptions import * -from ._models import * -from ._status_codes import * -from ._transports import * -from ._types import * -from ._urls import * - -try: - from ._main import main -except ImportError: # pragma: no cover - - def main() -> None: # type: ignore - import sys - - print( - "The httpx command line client could not run because the required " - "dependencies were not installed.\nMake sure you've installed " - "everything with: pip install 'httpx[cli]'" - ) - sys.exit(1) - - -__all__ = [ - "__description__", - "__title__", - "__version__", - "ASGITransport", - "AsyncBaseTransport", - "AsyncByteStream", - "AsyncClient", - "AsyncHTTPTransport", - "Auth", - "BaseTransport", - "BasicAuth", - "ByteStream", - "Client", - "CloseError", - "codes", - "ConnectError", - "ConnectTimeout", - "CookieConflict", - "Cookies", - "create_ssl_context", - "DecodingError", - "delete", - "DigestAuth", - "FunctionAuth", - "get", - "head", - "Headers", - "HTTPError", - "HTTPStatusError", - "HTTPTransport", - "InvalidURL", - "Limits", - "LocalProtocolError", - "main", - "MockTransport", - "NetRCAuth", - "NetworkError", - "options", - "patch", - "PoolTimeout", - "post", - "ProtocolError", - "Proxy", - "ProxyError", - "put", - "QueryParams", - "ReadError", - "ReadTimeout", - "RemoteProtocolError", - "request", - "Request", - "RequestError", - "RequestNotRead", - "Response", - "ResponseNotRead", - "stream", - "StreamClosed", - "StreamConsumed", - "StreamError", - "SyncByteStream", - "Timeout", - "TimeoutException", - "TooManyRedirects", - "TransportError", - "UnsupportedProtocol", - "URL", - "USE_CLIENT_DEFAULT", - "WriteError", - "WriteTimeout", - "WSGITransport", -] - - -__locals = locals() -for __name in __all__: - if not __name.startswith("__"): - setattr(__locals[__name], "__module__", "httpx") # noqa diff --git a/httpx/__version__.py b/httpx/__version__.py deleted file mode 100644 index 801bfacf67..0000000000 --- a/httpx/__version__.py +++ /dev/null @@ -1,3 +0,0 @@ -__title__ = "httpx" -__description__ = "A next generation HTTP client, for Python 3." -__version__ = "0.28.1" diff --git a/httpx/_api.py b/httpx/_api.py deleted file mode 100644 index c3cda1ecda..0000000000 --- a/httpx/_api.py +++ /dev/null @@ -1,438 +0,0 @@ -from __future__ import annotations - -import typing -from contextlib import contextmanager - -from ._client import Client -from ._config import DEFAULT_TIMEOUT_CONFIG -from ._models import Response -from ._types import ( - AuthTypes, - CookieTypes, - HeaderTypes, - ProxyTypes, - QueryParamTypes, - RequestContent, - RequestData, - RequestFiles, - TimeoutTypes, -) -from ._urls import URL - -if typing.TYPE_CHECKING: - import ssl # pragma: no cover - - -__all__ = [ - "delete", - "get", - "head", - "options", - "patch", - "post", - "put", - "request", - "stream", -] - - -def request( - method: str, - url: URL | str, - *, - params: QueryParamTypes | None = None, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - follow_redirects: bool = False, - verify: ssl.SSLContext | str | bool = True, - trust_env: bool = True, -) -> Response: - """ - Sends an HTTP request. - - **Parameters:** - - * **method** - HTTP method for the new `Request` object: `GET`, `OPTIONS`, - `HEAD`, `POST`, `PUT`, `PATCH`, or `DELETE`. - * **url** - URL for the new `Request` object. - * **params** - *(optional)* Query parameters to include in the URL, as a - string, dictionary, or sequence of two-tuples. - * **content** - *(optional)* Binary content to include in the body of the - request, as bytes or a byte iterator. - * **data** - *(optional)* Form data to include in the body of the request, - as a dictionary. - * **files** - *(optional)* A dictionary of upload files to include in the - body of the request. - * **json** - *(optional)* A JSON serializable object to include in the body - of the request. - * **headers** - *(optional)* Dictionary of HTTP headers to include in the - request. - * **cookies** - *(optional)* Dictionary of Cookie items to include in the - request. - * **auth** - *(optional)* An authentication class to use when sending the - request. - * **proxy** - *(optional)* A proxy URL where all the traffic should be routed. - * **timeout** - *(optional)* The timeout configuration to use when sending - the request. - * **follow_redirects** - *(optional)* Enables or disables HTTP redirects. - * **verify** - *(optional)* Either `True` to use an SSL context with the - default CA bundle, `False` to disable verification, or an instance of - `ssl.SSLContext` to use a custom context. - * **trust_env** - *(optional)* Enables or disables usage of environment - variables for configuration. - - **Returns:** `Response` - - Usage: - - ``` - >>> import httpx - >>> response = httpx.request('GET', 'https://httpbin.org/get') - >>> response - - ``` - """ - with Client( - cookies=cookies, - proxy=proxy, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) as client: - return client.request( - method=method, - url=url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - auth=auth, - follow_redirects=follow_redirects, - ) - - -@contextmanager -def stream( - method: str, - url: URL | str, - *, - params: QueryParamTypes | None = None, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - follow_redirects: bool = False, - verify: ssl.SSLContext | str | bool = True, - trust_env: bool = True, -) -> typing.Iterator[Response]: - """ - Alternative to `httpx.request()` that streams the response body - instead of loading it into memory at once. - - **Parameters**: See `httpx.request`. - - See also: [Streaming Responses][0] - - [0]: /quickstart#streaming-responses - """ - with Client( - cookies=cookies, - proxy=proxy, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) as client: - with client.stream( - method=method, - url=url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - auth=auth, - follow_redirects=follow_redirects, - ) as response: - yield response - - -def get( - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - follow_redirects: bool = False, - verify: ssl.SSLContext | str | bool = True, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - trust_env: bool = True, -) -> Response: - """ - Sends a `GET` request. - - **Parameters**: See `httpx.request`. - - Note that the `data`, `files`, `json` and `content` parameters are not available - on this function, as `GET` requests should not include a request body. - """ - return request( - "GET", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - proxy=proxy, - follow_redirects=follow_redirects, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) - - -def options( - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - follow_redirects: bool = False, - verify: ssl.SSLContext | str | bool = True, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - trust_env: bool = True, -) -> Response: - """ - Sends an `OPTIONS` request. - - **Parameters**: See `httpx.request`. - - Note that the `data`, `files`, `json` and `content` parameters are not available - on this function, as `OPTIONS` requests should not include a request body. - """ - return request( - "OPTIONS", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - proxy=proxy, - follow_redirects=follow_redirects, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) - - -def head( - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - follow_redirects: bool = False, - verify: ssl.SSLContext | str | bool = True, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - trust_env: bool = True, -) -> Response: - """ - Sends a `HEAD` request. - - **Parameters**: See `httpx.request`. - - Note that the `data`, `files`, `json` and `content` parameters are not available - on this function, as `HEAD` requests should not include a request body. - """ - return request( - "HEAD", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - proxy=proxy, - follow_redirects=follow_redirects, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) - - -def post( - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - follow_redirects: bool = False, - verify: ssl.SSLContext | str | bool = True, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - trust_env: bool = True, -) -> Response: - """ - Sends a `POST` request. - - **Parameters**: See `httpx.request`. - """ - return request( - "POST", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - proxy=proxy, - follow_redirects=follow_redirects, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) - - -def put( - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - follow_redirects: bool = False, - verify: ssl.SSLContext | str | bool = True, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - trust_env: bool = True, -) -> Response: - """ - Sends a `PUT` request. - - **Parameters**: See `httpx.request`. - """ - return request( - "PUT", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - proxy=proxy, - follow_redirects=follow_redirects, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) - - -def patch( - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - follow_redirects: bool = False, - verify: ssl.SSLContext | str | bool = True, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - trust_env: bool = True, -) -> Response: - """ - Sends a `PATCH` request. - - **Parameters**: See `httpx.request`. - """ - return request( - "PATCH", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - proxy=proxy, - follow_redirects=follow_redirects, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) - - -def delete( - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | None = None, - proxy: ProxyTypes | None = None, - follow_redirects: bool = False, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - verify: ssl.SSLContext | str | bool = True, - trust_env: bool = True, -) -> Response: - """ - Sends a `DELETE` request. - - **Parameters**: See `httpx.request`. - - Note that the `data`, `files`, `json` and `content` parameters are not available - on this function, as `DELETE` requests should not include a request body. - """ - return request( - "DELETE", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - proxy=proxy, - follow_redirects=follow_redirects, - verify=verify, - timeout=timeout, - trust_env=trust_env, - ) diff --git a/httpx/_auth.py b/httpx/_auth.py deleted file mode 100644 index 9d24faed99..0000000000 --- a/httpx/_auth.py +++ /dev/null @@ -1,348 +0,0 @@ -from __future__ import annotations - -import hashlib -import os -import re -import time -import typing -from base64 import b64encode -from urllib.request import parse_http_list - -from ._exceptions import ProtocolError -from ._models import Cookies, Request, Response -from ._utils import to_bytes, to_str, unquote - -if typing.TYPE_CHECKING: # pragma: no cover - from hashlib import _Hash - - -__all__ = ["Auth", "BasicAuth", "DigestAuth", "FunctionAuth", "NetRCAuth"] - - -class Auth: - """ - Base class for all authentication schemes. - - To implement a custom authentication scheme, subclass `Auth` and override - the `.auth_flow()` method. - - If the authentication scheme does I/O such as disk access or network calls, or uses - synchronization primitives such as locks, you should override `.sync_auth_flow()` - and/or `.async_auth_flow()` instead of `.auth_flow()` to provide specialized - implementations that will be used by `Client` and `AsyncClient` respectively. - """ - - requires_request_body = False - requires_response_body = False - - def auth_flow(self, request: Request) -> typing.Generator[Request, Response, None]: - """ - Execute the authentication flow. - - To dispatch a request, `yield` it: - - ``` - yield request - ``` - - The client will `.send()` the response back into the flow generator. You can - access it like so: - - ``` - response = yield request - ``` - - A `return` (or reaching the end of the generator) will result in the - client returning the last response obtained from the server. - - You can dispatch as many requests as is necessary. - """ - yield request - - def sync_auth_flow( - self, request: Request - ) -> typing.Generator[Request, Response, None]: - """ - Execute the authentication flow synchronously. - - By default, this defers to `.auth_flow()`. You should override this method - when the authentication scheme does I/O and/or uses concurrency primitives. - """ - if self.requires_request_body: - request.read() - - flow = self.auth_flow(request) - request = next(flow) - - while True: - response = yield request - if self.requires_response_body: - response.read() - - try: - request = flow.send(response) - except StopIteration: - break - - async def async_auth_flow( - self, request: Request - ) -> typing.AsyncGenerator[Request, Response]: - """ - Execute the authentication flow asynchronously. - - By default, this defers to `.auth_flow()`. You should override this method - when the authentication scheme does I/O and/or uses concurrency primitives. - """ - if self.requires_request_body: - await request.aread() - - flow = self.auth_flow(request) - request = next(flow) - - while True: - response = yield request - if self.requires_response_body: - await response.aread() - - try: - request = flow.send(response) - except StopIteration: - break - - -class FunctionAuth(Auth): - """ - Allows the 'auth' argument to be passed as a simple callable function, - that takes the request, and returns a new, modified request. - """ - - def __init__(self, func: typing.Callable[[Request], Request]) -> None: - self._func = func - - def auth_flow(self, request: Request) -> typing.Generator[Request, Response, None]: - yield self._func(request) - - -class BasicAuth(Auth): - """ - Allows the 'auth' argument to be passed as a (username, password) pair, - and uses HTTP Basic authentication. - """ - - def __init__(self, username: str | bytes, password: str | bytes) -> None: - self._auth_header = self._build_auth_header(username, password) - - def auth_flow(self, request: Request) -> typing.Generator[Request, Response, None]: - request.headers["Authorization"] = self._auth_header - yield request - - def _build_auth_header(self, username: str | bytes, password: str | bytes) -> str: - userpass = b":".join((to_bytes(username), to_bytes(password))) - token = b64encode(userpass).decode() - return f"Basic {token}" - - -class NetRCAuth(Auth): - """ - Use a 'netrc' file to lookup basic auth credentials based on the url host. - """ - - def __init__(self, file: str | None = None) -> None: - # Lazily import 'netrc'. - # There's no need for us to load this module unless 'NetRCAuth' is being used. - import netrc - - self._netrc_info = netrc.netrc(file) - - def auth_flow(self, request: Request) -> typing.Generator[Request, Response, None]: - auth_info = self._netrc_info.authenticators(request.url.host) - if auth_info is None or not auth_info[2]: - # The netrc file did not have authentication credentials for this host. - yield request - else: - # Build a basic auth header with credentials from the netrc file. - request.headers["Authorization"] = self._build_auth_header( - username=auth_info[0], password=auth_info[2] - ) - yield request - - def _build_auth_header(self, username: str | bytes, password: str | bytes) -> str: - userpass = b":".join((to_bytes(username), to_bytes(password))) - token = b64encode(userpass).decode() - return f"Basic {token}" - - -class DigestAuth(Auth): - _ALGORITHM_TO_HASH_FUNCTION: dict[str, typing.Callable[[bytes], _Hash]] = { - "MD5": hashlib.md5, - "MD5-SESS": hashlib.md5, - "SHA": hashlib.sha1, - "SHA-SESS": hashlib.sha1, - "SHA-256": hashlib.sha256, - "SHA-256-SESS": hashlib.sha256, - "SHA-512": hashlib.sha512, - "SHA-512-SESS": hashlib.sha512, - } - - def __init__(self, username: str | bytes, password: str | bytes) -> None: - self._username = to_bytes(username) - self._password = to_bytes(password) - self._last_challenge: _DigestAuthChallenge | None = None - self._nonce_count = 1 - - def auth_flow(self, request: Request) -> typing.Generator[Request, Response, None]: - if self._last_challenge: - request.headers["Authorization"] = self._build_auth_header( - request, self._last_challenge - ) - - response = yield request - - if response.status_code != 401 or "www-authenticate" not in response.headers: - # If the response is not a 401 then we don't - # need to build an authenticated request. - return - - for auth_header in response.headers.get_list("www-authenticate"): - if auth_header.lower().startswith("digest "): - break - else: - # If the response does not include a 'WWW-Authenticate: Digest ...' - # header, then we don't need to build an authenticated request. - return - - self._last_challenge = self._parse_challenge(request, response, auth_header) - self._nonce_count = 1 - - request.headers["Authorization"] = self._build_auth_header( - request, self._last_challenge - ) - if response.cookies: - Cookies(response.cookies).set_cookie_header(request=request) - yield request - - def _parse_challenge( - self, request: Request, response: Response, auth_header: str - ) -> _DigestAuthChallenge: - """ - Returns a challenge from a Digest WWW-Authenticate header. - These take the form of: - `Digest realm="realm@host.com",qop="auth,auth-int",nonce="abc",opaque="xyz"` - """ - scheme, _, fields = auth_header.partition(" ") - - # This method should only ever have been called with a Digest auth header. - assert scheme.lower() == "digest" - - header_dict: dict[str, str] = {} - for field in parse_http_list(fields): - key, value = field.strip().split("=", 1) - header_dict[key] = unquote(value) - - try: - realm = header_dict["realm"].encode() - nonce = header_dict["nonce"].encode() - algorithm = header_dict.get("algorithm", "MD5") - opaque = header_dict["opaque"].encode() if "opaque" in header_dict else None - qop = header_dict["qop"].encode() if "qop" in header_dict else None - return _DigestAuthChallenge( - realm=realm, nonce=nonce, algorithm=algorithm, opaque=opaque, qop=qop - ) - except KeyError as exc: - message = "Malformed Digest WWW-Authenticate header" - raise ProtocolError(message, request=request) from exc - - def _build_auth_header( - self, request: Request, challenge: _DigestAuthChallenge - ) -> str: - hash_func = self._ALGORITHM_TO_HASH_FUNCTION[challenge.algorithm.upper()] - - def digest(data: bytes) -> bytes: - return hash_func(data).hexdigest().encode() - - A1 = b":".join((self._username, challenge.realm, self._password)) - - path = request.url.raw_path - A2 = b":".join((request.method.encode(), path)) - # TODO: implement auth-int - HA2 = digest(A2) - - nc_value = b"%08x" % self._nonce_count - cnonce = self._get_client_nonce(self._nonce_count, challenge.nonce) - self._nonce_count += 1 - - HA1 = digest(A1) - if challenge.algorithm.lower().endswith("-sess"): - HA1 = digest(b":".join((HA1, challenge.nonce, cnonce))) - - qop = self._resolve_qop(challenge.qop, request=request) - if qop is None: - # Following RFC 2069 - digest_data = [HA1, challenge.nonce, HA2] - else: - # Following RFC 2617/7616 - digest_data = [HA1, challenge.nonce, nc_value, cnonce, qop, HA2] - - format_args = { - "username": self._username, - "realm": challenge.realm, - "nonce": challenge.nonce, - "uri": path, - "response": digest(b":".join(digest_data)), - "algorithm": challenge.algorithm.encode(), - } - if challenge.opaque: - format_args["opaque"] = challenge.opaque - if qop: - format_args["qop"] = b"auth" - format_args["nc"] = nc_value - format_args["cnonce"] = cnonce - - return "Digest " + self._get_header_value(format_args) - - def _get_client_nonce(self, nonce_count: int, nonce: bytes) -> bytes: - s = str(nonce_count).encode() - s += nonce - s += time.ctime().encode() - s += os.urandom(8) - - return hashlib.sha1(s).hexdigest()[:16].encode() - - def _get_header_value(self, header_fields: dict[str, bytes]) -> str: - NON_QUOTED_FIELDS = ("algorithm", "qop", "nc") - QUOTED_TEMPLATE = '{}="{}"' - NON_QUOTED_TEMPLATE = "{}={}" - - header_value = "" - for i, (field, value) in enumerate(header_fields.items()): - if i > 0: - header_value += ", " - template = ( - QUOTED_TEMPLATE - if field not in NON_QUOTED_FIELDS - else NON_QUOTED_TEMPLATE - ) - header_value += template.format(field, to_str(value)) - - return header_value - - def _resolve_qop(self, qop: bytes | None, request: Request) -> bytes | None: - if qop is None: - return None - qops = re.split(b", ?", qop) - if b"auth" in qops: - return b"auth" - - if qops == [b"auth-int"]: - raise NotImplementedError("Digest auth-int support is not yet implemented") - - message = f'Unexpected qop value "{qop!r}" in digest auth' - raise ProtocolError(message, request=request) - - -class _DigestAuthChallenge(typing.NamedTuple): - realm: bytes - nonce: bytes - algorithm: str - opaque: bytes | None - qop: bytes | None diff --git a/httpx/_client.py b/httpx/_client.py deleted file mode 100644 index 13cd933673..0000000000 --- a/httpx/_client.py +++ /dev/null @@ -1,2019 +0,0 @@ -from __future__ import annotations - -import datetime -import enum -import logging -import time -import typing -import warnings -from contextlib import asynccontextmanager, contextmanager -from types import TracebackType - -from .__version__ import __version__ -from ._auth import Auth, BasicAuth, FunctionAuth -from ._config import ( - DEFAULT_LIMITS, - DEFAULT_MAX_REDIRECTS, - DEFAULT_TIMEOUT_CONFIG, - Limits, - Proxy, - Timeout, -) -from ._decoders import SUPPORTED_DECODERS -from ._exceptions import ( - InvalidURL, - RemoteProtocolError, - TooManyRedirects, - request_context, -) -from ._models import Cookies, Headers, Request, Response -from ._status_codes import codes -from ._transports.base import AsyncBaseTransport, BaseTransport -from ._transports.default import AsyncHTTPTransport, HTTPTransport -from ._types import ( - AsyncByteStream, - AuthTypes, - CertTypes, - CookieTypes, - HeaderTypes, - ProxyTypes, - QueryParamTypes, - RequestContent, - RequestData, - RequestExtensions, - RequestFiles, - SyncByteStream, - TimeoutTypes, -) -from ._urls import URL, QueryParams -from ._utils import URLPattern, get_environment_proxies - -if typing.TYPE_CHECKING: - import ssl # pragma: no cover - -__all__ = ["USE_CLIENT_DEFAULT", "AsyncClient", "Client"] - -# The type annotation for @classmethod and context managers here follows PEP 484 -# https://www.python.org/dev/peps/pep-0484/#annotating-instance-and-class-methods -T = typing.TypeVar("T", bound="Client") -U = typing.TypeVar("U", bound="AsyncClient") - - -def _is_https_redirect(url: URL, location: URL) -> bool: - """ - Return 'True' if 'location' is a HTTPS upgrade of 'url' - """ - if url.host != location.host: - return False - - return ( - url.scheme == "http" - and _port_or_default(url) == 80 - and location.scheme == "https" - and _port_or_default(location) == 443 - ) - - -def _port_or_default(url: URL) -> int | None: - if url.port is not None: - return url.port - return {"http": 80, "https": 443}.get(url.scheme) - - -def _same_origin(url: URL, other: URL) -> bool: - """ - Return 'True' if the given URLs share the same origin. - """ - return ( - url.scheme == other.scheme - and url.host == other.host - and _port_or_default(url) == _port_or_default(other) - ) - - -class UseClientDefault: - """ - For some parameters such as `auth=...` and `timeout=...` we need to be able - to indicate the default "unset" state, in a way that is distinctly different - to using `None`. - - The default "unset" state indicates that whatever default is set on the - client should be used. This is different to setting `None`, which - explicitly disables the parameter, possibly overriding a client default. - - For example we use `timeout=USE_CLIENT_DEFAULT` in the `request()` signature. - Omitting the `timeout` parameter will send a request using whatever default - timeout has been configured on the client. Including `timeout=None` will - ensure no timeout is used. - - Note that user code shouldn't need to use the `USE_CLIENT_DEFAULT` constant, - but it is used internally when a parameter is not included. - """ - - -USE_CLIENT_DEFAULT = UseClientDefault() - - -logger = logging.getLogger("httpx") - -USER_AGENT = f"python-httpx/{__version__}" -ACCEPT_ENCODING = ", ".join( - [key for key in SUPPORTED_DECODERS.keys() if key != "identity"] -) - - -class ClientState(enum.Enum): - # UNOPENED: - # The client has been instantiated, but has not been used to send a request, - # or been opened by entering the context of a `with` block. - UNOPENED = 1 - # OPENED: - # The client has either sent a request, or is within a `with` block. - OPENED = 2 - # CLOSED: - # The client has either exited the `with` block, or `close()` has - # been called explicitly. - CLOSED = 3 - - -class BoundSyncStream(SyncByteStream): - """ - A byte stream that is bound to a given response instance, and that - ensures the `response.elapsed` is set once the response is closed. - """ - - def __init__( - self, stream: SyncByteStream, response: Response, start: float - ) -> None: - self._stream = stream - self._response = response - self._start = start - - def __iter__(self) -> typing.Iterator[bytes]: - for chunk in self._stream: - yield chunk - - def close(self) -> None: - elapsed = time.perf_counter() - self._start - self._response.elapsed = datetime.timedelta(seconds=elapsed) - self._stream.close() - - -class BoundAsyncStream(AsyncByteStream): - """ - An async byte stream that is bound to a given response instance, and that - ensures the `response.elapsed` is set once the response is closed. - """ - - def __init__( - self, stream: AsyncByteStream, response: Response, start: float - ) -> None: - self._stream = stream - self._response = response - self._start = start - - async def __aiter__(self) -> typing.AsyncIterator[bytes]: - async for chunk in self._stream: - yield chunk - - async def aclose(self) -> None: - elapsed = time.perf_counter() - self._start - self._response.elapsed = datetime.timedelta(seconds=elapsed) - await self._stream.aclose() - - -EventHook = typing.Callable[..., typing.Any] - - -class BaseClient: - def __init__( - self, - *, - auth: AuthTypes | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - follow_redirects: bool = False, - max_redirects: int = DEFAULT_MAX_REDIRECTS, - event_hooks: None | (typing.Mapping[str, list[EventHook]]) = None, - base_url: URL | str = "", - trust_env: bool = True, - default_encoding: str | typing.Callable[[bytes], str] = "utf-8", - ) -> None: - event_hooks = {} if event_hooks is None else event_hooks - - self._base_url = self._enforce_trailing_slash(URL(base_url)) - - self._auth = self._build_auth(auth) - self._params = QueryParams(params) - self.headers = Headers(headers) - self._cookies = Cookies(cookies) - self._timeout = Timeout(timeout) - self.follow_redirects = follow_redirects - self.max_redirects = max_redirects - self._event_hooks = { - "request": list(event_hooks.get("request", [])), - "response": list(event_hooks.get("response", [])), - } - self._trust_env = trust_env - self._default_encoding = default_encoding - self._state = ClientState.UNOPENED - - @property - def is_closed(self) -> bool: - """ - Check if the client being closed - """ - return self._state == ClientState.CLOSED - - @property - def trust_env(self) -> bool: - return self._trust_env - - def _enforce_trailing_slash(self, url: URL) -> URL: - if url.raw_path.endswith(b"/"): - return url - return url.copy_with(raw_path=url.raw_path + b"/") - - def _get_proxy_map( - self, proxy: ProxyTypes | None, allow_env_proxies: bool - ) -> dict[str, Proxy | None]: - if proxy is None: - if allow_env_proxies: - return { - key: None if url is None else Proxy(url=url) - for key, url in get_environment_proxies().items() - } - return {} - else: - proxy = Proxy(url=proxy) if isinstance(proxy, (str, URL)) else proxy - return {"all://": proxy} - - @property - def timeout(self) -> Timeout: - return self._timeout - - @timeout.setter - def timeout(self, timeout: TimeoutTypes) -> None: - self._timeout = Timeout(timeout) - - @property - def event_hooks(self) -> dict[str, list[EventHook]]: - return self._event_hooks - - @event_hooks.setter - def event_hooks(self, event_hooks: dict[str, list[EventHook]]) -> None: - self._event_hooks = { - "request": list(event_hooks.get("request", [])), - "response": list(event_hooks.get("response", [])), - } - - @property - def auth(self) -> Auth | None: - """ - Authentication class used when none is passed at the request-level. - - See also [Authentication][0]. - - [0]: /quickstart/#authentication - """ - return self._auth - - @auth.setter - def auth(self, auth: AuthTypes) -> None: - self._auth = self._build_auth(auth) - - @property - def base_url(self) -> URL: - """ - Base URL to use when sending requests with relative URLs. - """ - return self._base_url - - @base_url.setter - def base_url(self, url: URL | str) -> None: - self._base_url = self._enforce_trailing_slash(URL(url)) - - @property - def headers(self) -> Headers: - """ - HTTP headers to include when sending requests. - """ - return self._headers - - @headers.setter - def headers(self, headers: HeaderTypes) -> None: - client_headers = Headers( - { - b"Accept": b"*/*", - b"Accept-Encoding": ACCEPT_ENCODING.encode("ascii"), - b"Connection": b"keep-alive", - b"User-Agent": USER_AGENT.encode("ascii"), - } - ) - client_headers.update(headers) - self._headers = client_headers - - @property - def cookies(self) -> Cookies: - """ - Cookie values to include when sending requests. - """ - return self._cookies - - @cookies.setter - def cookies(self, cookies: CookieTypes) -> None: - self._cookies = Cookies(cookies) - - @property - def params(self) -> QueryParams: - """ - Query parameters to include in the URL when sending requests. - """ - return self._params - - @params.setter - def params(self, params: QueryParamTypes) -> None: - self._params = QueryParams(params) - - def build_request( - self, - method: str, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Request: - """ - Build and return a request instance. - - * The `params`, `headers` and `cookies` arguments - are merged with any values set on the client. - * The `url` argument is merged with any `base_url` set on the client. - - See also: [Request instances][0] - - [0]: /advanced/clients/#request-instances - """ - url = self._merge_url(url) - headers = self._merge_headers(headers) - cookies = self._merge_cookies(cookies) - params = self._merge_queryparams(params) - extensions = {} if extensions is None else extensions - if "timeout" not in extensions: - timeout = ( - self.timeout - if isinstance(timeout, UseClientDefault) - else Timeout(timeout) - ) - extensions = dict(**extensions, timeout=timeout.as_dict()) - return Request( - method, - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - extensions=extensions, - ) - - def _merge_url(self, url: URL | str) -> URL: - """ - Merge a URL argument together with any 'base_url' on the client, - to create the URL used for the outgoing request. - """ - merge_url = URL(url) - if merge_url.is_relative_url: - # To merge URLs we always append to the base URL. To get this - # behaviour correct we always ensure the base URL ends in a '/' - # separator, and strip any leading '/' from the merge URL. - # - # So, eg... - # - # >>> client = Client(base_url="https://www.example.com/subpath") - # >>> client.base_url - # URL('https://www.example.com/subpath/') - # >>> client.build_request("GET", "/path").url - # URL('https://www.example.com/subpath/path') - merge_raw_path = self.base_url.raw_path + merge_url.raw_path.lstrip(b"/") - return self.base_url.copy_with(raw_path=merge_raw_path) - return merge_url - - def _merge_cookies(self, cookies: CookieTypes | None = None) -> CookieTypes | None: - """ - Merge a cookies argument together with any cookies on the client, - to create the cookies used for the outgoing request. - """ - if cookies or self.cookies: - merged_cookies = Cookies(self.cookies) - merged_cookies.update(cookies) - return merged_cookies - return cookies - - def _merge_headers(self, headers: HeaderTypes | None = None) -> HeaderTypes | None: - """ - Merge a headers argument together with any headers on the client, - to create the headers used for the outgoing request. - """ - merged_headers = Headers(self.headers) - merged_headers.update(headers) - return merged_headers - - def _merge_queryparams( - self, params: QueryParamTypes | None = None - ) -> QueryParamTypes | None: - """ - Merge a queryparams argument together with any queryparams on the client, - to create the queryparams used for the outgoing request. - """ - if params or self.params: - merged_queryparams = QueryParams(self.params) - return merged_queryparams.merge(params) - return params - - def _build_auth(self, auth: AuthTypes | None) -> Auth | None: - if auth is None: - return None - elif isinstance(auth, tuple): - return BasicAuth(username=auth[0], password=auth[1]) - elif isinstance(auth, Auth): - return auth - elif callable(auth): - return FunctionAuth(func=auth) - else: - raise TypeError(f'Invalid "auth" argument: {auth!r}') - - def _build_request_auth( - self, - request: Request, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - ) -> Auth: - auth = ( - self._auth if isinstance(auth, UseClientDefault) else self._build_auth(auth) - ) - - if auth is not None: - return auth - - username, password = request.url.username, request.url.password - if username or password: - return BasicAuth(username=username, password=password) - - return Auth() - - def _build_redirect_request(self, request: Request, response: Response) -> Request: - """ - Given a request and a redirect response, return a new request that - should be used to effect the redirect. - """ - method = self._redirect_method(request, response) - url = self._redirect_url(request, response) - headers = self._redirect_headers(request, url, method) - stream = self._redirect_stream(request, method) - cookies = Cookies(self.cookies) - return Request( - method=method, - url=url, - headers=headers, - cookies=cookies, - stream=stream, - extensions=request.extensions, - ) - - def _redirect_method(self, request: Request, response: Response) -> str: - """ - When being redirected we may want to change the method of the request - based on certain specs or browser behavior. - """ - method = request.method - - # https://tools.ietf.org/html/rfc7231#section-6.4.4 - if response.status_code == codes.SEE_OTHER and method != "HEAD": - method = "GET" - - # Do what the browsers do, despite standards... - # Turn 302s into GETs. - if response.status_code == codes.FOUND and method != "HEAD": - method = "GET" - - # If a POST is responded to with a 301, turn it into a GET. - # This bizarre behaviour is explained in 'requests' issue 1704. - if response.status_code == codes.MOVED_PERMANENTLY and method == "POST": - method = "GET" - - return method - - def _redirect_url(self, request: Request, response: Response) -> URL: - """ - Return the URL for the redirect to follow. - """ - location = response.headers["Location"] - - try: - url = URL(location) - except InvalidURL as exc: - raise RemoteProtocolError( - f"Invalid URL in location header: {exc}.", request=request - ) from None - - # Handle malformed 'Location' headers that are "absolute" form, have no host. - # See: https://github.com/encode/httpx/issues/771 - if url.scheme and not url.host: - url = url.copy_with(host=request.url.host) - - # Facilitate relative 'Location' headers, as allowed by RFC 7231. - # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource') - if url.is_relative_url: - url = request.url.join(url) - - # Attach previous fragment if needed (RFC 7231 7.1.2) - if request.url.fragment and not url.fragment: - url = url.copy_with(fragment=request.url.fragment) - - return url - - def _redirect_headers(self, request: Request, url: URL, method: str) -> Headers: - """ - Return the headers that should be used for the redirect request. - """ - headers = Headers(request.headers) - - if not _same_origin(url, request.url): - if not _is_https_redirect(request.url, url): - # Strip Authorization headers when responses are redirected - # away from the origin. (Except for direct HTTP to HTTPS redirects.) - headers.pop("Authorization", None) - - # Update the Host header. - headers["Host"] = url.netloc.decode("ascii") - - if method != request.method and method == "GET": - # If we've switch to a 'GET' request, then strip any headers which - # are only relevant to the request body. - headers.pop("Content-Length", None) - headers.pop("Transfer-Encoding", None) - - # We should use the client cookie store to determine any cookie header, - # rather than whatever was on the original outgoing request. - headers.pop("Cookie", None) - - return headers - - def _redirect_stream( - self, request: Request, method: str - ) -> SyncByteStream | AsyncByteStream | None: - """ - Return the body that should be used for the redirect request. - """ - if method != request.method and method == "GET": - return None - - return request.stream - - def _set_timeout(self, request: Request) -> None: - if "timeout" not in request.extensions: - timeout = ( - self.timeout - if isinstance(self.timeout, UseClientDefault) - else Timeout(self.timeout) - ) - request.extensions = dict(**request.extensions, timeout=timeout.as_dict()) - - -class Client(BaseClient): - """ - An HTTP client, with connection pooling, HTTP/2, redirects, cookie persistence, etc. - - It can be shared between threads. - - Usage: - - ```python - >>> client = httpx.Client() - >>> response = client.get('https://example.org') - ``` - - **Parameters:** - - * **auth** - *(optional)* An authentication class to use when sending - requests. - * **params** - *(optional)* Query parameters to include in request URLs, as - a string, dictionary, or sequence of two-tuples. - * **headers** - *(optional)* Dictionary of HTTP headers to include when - sending requests. - * **cookies** - *(optional)* Dictionary of Cookie items to include when - sending requests. - * **verify** - *(optional)* Either `True` to use an SSL context with the - default CA bundle, `False` to disable verification, or an instance of - `ssl.SSLContext` to use a custom context. - * **http2** - *(optional)* A boolean indicating if HTTP/2 support should be - enabled. Defaults to `False`. - * **proxy** - *(optional)* A proxy URL where all the traffic should be routed. - * **timeout** - *(optional)* The timeout configuration to use when sending - requests. - * **limits** - *(optional)* The limits configuration to use. - * **max_redirects** - *(optional)* The maximum number of redirect responses - that should be followed. - * **base_url** - *(optional)* A URL to use as the base when building - request URLs. - * **transport** - *(optional)* A transport class to use for sending requests - over the network. - * **trust_env** - *(optional)* Enables or disables usage of environment - variables for configuration. - * **default_encoding** - *(optional)* The default encoding to use for decoding - response text, if no charset information is included in a response Content-Type - header. Set to a callable for automatic character set detection. Default: "utf-8". - """ - - def __init__( - self, - *, - auth: AuthTypes | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - trust_env: bool = True, - http1: bool = True, - http2: bool = False, - proxy: ProxyTypes | None = None, - mounts: None | (typing.Mapping[str, BaseTransport | None]) = None, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - follow_redirects: bool = False, - limits: Limits = DEFAULT_LIMITS, - max_redirects: int = DEFAULT_MAX_REDIRECTS, - event_hooks: None | (typing.Mapping[str, list[EventHook]]) = None, - base_url: URL | str = "", - transport: BaseTransport | None = None, - default_encoding: str | typing.Callable[[bytes], str] = "utf-8", - ) -> None: - super().__init__( - auth=auth, - params=params, - headers=headers, - cookies=cookies, - timeout=timeout, - follow_redirects=follow_redirects, - max_redirects=max_redirects, - event_hooks=event_hooks, - base_url=base_url, - trust_env=trust_env, - default_encoding=default_encoding, - ) - - if http2: - try: - import h2 # noqa - except ImportError: # pragma: no cover - raise ImportError( - "Using http2=True, but the 'h2' package is not installed. " - "Make sure to install httpx using `pip install httpx[http2]`." - ) from None - - allow_env_proxies = trust_env and transport is None - proxy_map = self._get_proxy_map(proxy, allow_env_proxies) - - self._transport = self._init_transport( - verify=verify, - cert=cert, - trust_env=trust_env, - http1=http1, - http2=http2, - limits=limits, - transport=transport, - ) - self._mounts: dict[URLPattern, BaseTransport | None] = { - URLPattern(key): None - if proxy is None - else self._init_proxy_transport( - proxy, - verify=verify, - cert=cert, - trust_env=trust_env, - http1=http1, - http2=http2, - limits=limits, - ) - for key, proxy in proxy_map.items() - } - if mounts is not None: - self._mounts.update( - {URLPattern(key): transport for key, transport in mounts.items()} - ) - - self._mounts = dict(sorted(self._mounts.items())) - - def _init_transport( - self, - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - trust_env: bool = True, - http1: bool = True, - http2: bool = False, - limits: Limits = DEFAULT_LIMITS, - transport: BaseTransport | None = None, - ) -> BaseTransport: - if transport is not None: - return transport - - return HTTPTransport( - verify=verify, - cert=cert, - trust_env=trust_env, - http1=http1, - http2=http2, - limits=limits, - ) - - def _init_proxy_transport( - self, - proxy: Proxy, - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - trust_env: bool = True, - http1: bool = True, - http2: bool = False, - limits: Limits = DEFAULT_LIMITS, - ) -> BaseTransport: - return HTTPTransport( - verify=verify, - cert=cert, - trust_env=trust_env, - http1=http1, - http2=http2, - limits=limits, - proxy=proxy, - ) - - def _transport_for_url(self, url: URL) -> BaseTransport: - """ - Returns the transport instance that should be used for a given URL. - This will either be the standard connection pool, or a proxy. - """ - for pattern, transport in self._mounts.items(): - if pattern.matches(url): - return self._transport if transport is None else transport - - return self._transport - - def request( - self, - method: str, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Build and send a request. - - Equivalent to: - - ```python - request = client.build_request(...) - response = client.send(request, ...) - ``` - - See `Client.build_request()`, `Client.send()` and - [Merging of configuration][0] for how the various parameters - are merged with client-level configuration. - - [0]: /advanced/clients/#merging-of-configuration - """ - if cookies is not None: - message = ( - "Setting per-request cookies=<...> is being deprecated, because " - "the expected behaviour on cookie persistence is ambiguous. Set " - "cookies directly on the client instance instead." - ) - warnings.warn(message, DeprecationWarning, stacklevel=2) - - request = self.build_request( - method=method, - url=url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - timeout=timeout, - extensions=extensions, - ) - return self.send(request, auth=auth, follow_redirects=follow_redirects) - - @contextmanager - def stream( - self, - method: str, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> typing.Iterator[Response]: - """ - Alternative to `httpx.request()` that streams the response body - instead of loading it into memory at once. - - **Parameters**: See `httpx.request`. - - See also: [Streaming Responses][0] - - [0]: /quickstart#streaming-responses - """ - request = self.build_request( - method=method, - url=url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - timeout=timeout, - extensions=extensions, - ) - response = self.send( - request=request, - auth=auth, - follow_redirects=follow_redirects, - stream=True, - ) - try: - yield response - finally: - response.close() - - def send( - self, - request: Request, - *, - stream: bool = False, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - ) -> Response: - """ - Send a request. - - The request is sent as-is, unmodified. - - Typically you'll want to build one with `Client.build_request()` - so that any client-level configuration is merged into the request, - but passing an explicit `httpx.Request()` is supported as well. - - See also: [Request instances][0] - - [0]: /advanced/clients/#request-instances - """ - if self._state == ClientState.CLOSED: - raise RuntimeError("Cannot send a request, as the client has been closed.") - - self._state = ClientState.OPENED - follow_redirects = ( - self.follow_redirects - if isinstance(follow_redirects, UseClientDefault) - else follow_redirects - ) - - self._set_timeout(request) - - auth = self._build_request_auth(request, auth) - - response = self._send_handling_auth( - request, - auth=auth, - follow_redirects=follow_redirects, - history=[], - ) - try: - if not stream: - response.read() - - return response - - except BaseException as exc: - response.close() - raise exc - - def _send_handling_auth( - self, - request: Request, - auth: Auth, - follow_redirects: bool, - history: list[Response], - ) -> Response: - auth_flow = auth.sync_auth_flow(request) - try: - request = next(auth_flow) - - while True: - response = self._send_handling_redirects( - request, - follow_redirects=follow_redirects, - history=history, - ) - try: - try: - next_request = auth_flow.send(response) - except StopIteration: - return response - - response.history = list(history) - response.read() - request = next_request - history.append(response) - - except BaseException as exc: - response.close() - raise exc - finally: - auth_flow.close() - - def _send_handling_redirects( - self, - request: Request, - follow_redirects: bool, - history: list[Response], - ) -> Response: - while True: - if len(history) > self.max_redirects: - raise TooManyRedirects( - "Exceeded maximum allowed redirects.", request=request - ) - - for hook in self._event_hooks["request"]: - hook(request) - - response = self._send_single_request(request) - try: - for hook in self._event_hooks["response"]: - hook(response) - response.history = list(history) - - if not response.has_redirect_location: - return response - - request = self._build_redirect_request(request, response) - history = history + [response] - - if follow_redirects: - response.read() - else: - response.next_request = request - return response - - except BaseException as exc: - response.close() - raise exc - - def _send_single_request(self, request: Request) -> Response: - """ - Sends a single request, without handling any redirections. - """ - transport = self._transport_for_url(request.url) - start = time.perf_counter() - - if not isinstance(request.stream, SyncByteStream): - raise RuntimeError( - "Attempted to send an async request with a sync Client instance." - ) - - with request_context(request=request): - response = transport.handle_request(request) - - assert isinstance(response.stream, SyncByteStream) - - response.request = request - response.stream = BoundSyncStream( - response.stream, response=response, start=start - ) - self.cookies.extract_cookies(response) - response.default_encoding = self._default_encoding - - logger.info( - 'HTTP Request: %s %s "%s %d %s"', - request.method, - request.url, - response.http_version, - response.status_code, - response.reason_phrase, - ) - - return response - - def get( - self, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `GET` request. - - **Parameters**: See `httpx.request`. - """ - return self.request( - "GET", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - def options( - self, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send an `OPTIONS` request. - - **Parameters**: See `httpx.request`. - """ - return self.request( - "OPTIONS", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - def head( - self, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `HEAD` request. - - **Parameters**: See `httpx.request`. - """ - return self.request( - "HEAD", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - def post( - self, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `POST` request. - - **Parameters**: See `httpx.request`. - """ - return self.request( - "POST", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - def put( - self, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `PUT` request. - - **Parameters**: See `httpx.request`. - """ - return self.request( - "PUT", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - def patch( - self, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `PATCH` request. - - **Parameters**: See `httpx.request`. - """ - return self.request( - "PATCH", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - def delete( - self, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `DELETE` request. - - **Parameters**: See `httpx.request`. - """ - return self.request( - "DELETE", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - def close(self) -> None: - """ - Close transport and proxies. - """ - if self._state != ClientState.CLOSED: - self._state = ClientState.CLOSED - - self._transport.close() - for transport in self._mounts.values(): - if transport is not None: - transport.close() - - def __enter__(self: T) -> T: - if self._state != ClientState.UNOPENED: - msg = { - ClientState.OPENED: "Cannot open a client instance more than once.", - ClientState.CLOSED: ( - "Cannot reopen a client instance, once it has been closed." - ), - }[self._state] - raise RuntimeError(msg) - - self._state = ClientState.OPENED - - self._transport.__enter__() - for transport in self._mounts.values(): - if transport is not None: - transport.__enter__() - return self - - def __exit__( - self, - exc_type: type[BaseException] | None = None, - exc_value: BaseException | None = None, - traceback: TracebackType | None = None, - ) -> None: - self._state = ClientState.CLOSED - - self._transport.__exit__(exc_type, exc_value, traceback) - for transport in self._mounts.values(): - if transport is not None: - transport.__exit__(exc_type, exc_value, traceback) - - -class AsyncClient(BaseClient): - """ - An asynchronous HTTP client, with connection pooling, HTTP/2, redirects, - cookie persistence, etc. - - It can be shared between tasks. - - Usage: - - ```python - >>> async with httpx.AsyncClient() as client: - >>> response = await client.get('https://example.org') - ``` - - **Parameters:** - - * **auth** - *(optional)* An authentication class to use when sending - requests. - * **params** - *(optional)* Query parameters to include in request URLs, as - a string, dictionary, or sequence of two-tuples. - * **headers** - *(optional)* Dictionary of HTTP headers to include when - sending requests. - * **cookies** - *(optional)* Dictionary of Cookie items to include when - sending requests. - * **verify** - *(optional)* Either `True` to use an SSL context with the - default CA bundle, `False` to disable verification, or an instance of - `ssl.SSLContext` to use a custom context. - * **http2** - *(optional)* A boolean indicating if HTTP/2 support should be - enabled. Defaults to `False`. - * **proxy** - *(optional)* A proxy URL where all the traffic should be routed. - * **timeout** - *(optional)* The timeout configuration to use when sending - requests. - * **limits** - *(optional)* The limits configuration to use. - * **max_redirects** - *(optional)* The maximum number of redirect responses - that should be followed. - * **base_url** - *(optional)* A URL to use as the base when building - request URLs. - * **transport** - *(optional)* A transport class to use for sending requests - over the network. - * **trust_env** - *(optional)* Enables or disables usage of environment - variables for configuration. - * **default_encoding** - *(optional)* The default encoding to use for decoding - response text, if no charset information is included in a response Content-Type - header. Set to a callable for automatic character set detection. Default: "utf-8". - """ - - def __init__( - self, - *, - auth: AuthTypes | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - http1: bool = True, - http2: bool = False, - proxy: ProxyTypes | None = None, - mounts: None | (typing.Mapping[str, AsyncBaseTransport | None]) = None, - timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, - follow_redirects: bool = False, - limits: Limits = DEFAULT_LIMITS, - max_redirects: int = DEFAULT_MAX_REDIRECTS, - event_hooks: None | (typing.Mapping[str, list[EventHook]]) = None, - base_url: URL | str = "", - transport: AsyncBaseTransport | None = None, - trust_env: bool = True, - default_encoding: str | typing.Callable[[bytes], str] = "utf-8", - ) -> None: - super().__init__( - auth=auth, - params=params, - headers=headers, - cookies=cookies, - timeout=timeout, - follow_redirects=follow_redirects, - max_redirects=max_redirects, - event_hooks=event_hooks, - base_url=base_url, - trust_env=trust_env, - default_encoding=default_encoding, - ) - - if http2: - try: - import h2 # noqa - except ImportError: # pragma: no cover - raise ImportError( - "Using http2=True, but the 'h2' package is not installed. " - "Make sure to install httpx using `pip install httpx[http2]`." - ) from None - - allow_env_proxies = trust_env and transport is None - proxy_map = self._get_proxy_map(proxy, allow_env_proxies) - - self._transport = self._init_transport( - verify=verify, - cert=cert, - trust_env=trust_env, - http1=http1, - http2=http2, - limits=limits, - transport=transport, - ) - - self._mounts: dict[URLPattern, AsyncBaseTransport | None] = { - URLPattern(key): None - if proxy is None - else self._init_proxy_transport( - proxy, - verify=verify, - cert=cert, - trust_env=trust_env, - http1=http1, - http2=http2, - limits=limits, - ) - for key, proxy in proxy_map.items() - } - if mounts is not None: - self._mounts.update( - {URLPattern(key): transport for key, transport in mounts.items()} - ) - self._mounts = dict(sorted(self._mounts.items())) - - def _init_transport( - self, - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - trust_env: bool = True, - http1: bool = True, - http2: bool = False, - limits: Limits = DEFAULT_LIMITS, - transport: AsyncBaseTransport | None = None, - ) -> AsyncBaseTransport: - if transport is not None: - return transport - - return AsyncHTTPTransport( - verify=verify, - cert=cert, - trust_env=trust_env, - http1=http1, - http2=http2, - limits=limits, - ) - - def _init_proxy_transport( - self, - proxy: Proxy, - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - trust_env: bool = True, - http1: bool = True, - http2: bool = False, - limits: Limits = DEFAULT_LIMITS, - ) -> AsyncBaseTransport: - return AsyncHTTPTransport( - verify=verify, - cert=cert, - trust_env=trust_env, - http1=http1, - http2=http2, - limits=limits, - proxy=proxy, - ) - - def _transport_for_url(self, url: URL) -> AsyncBaseTransport: - """ - Returns the transport instance that should be used for a given URL. - This will either be the standard connection pool, or a proxy. - """ - for pattern, transport in self._mounts.items(): - if pattern.matches(url): - return self._transport if transport is None else transport - - return self._transport - - async def request( - self, - method: str, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Build and send a request. - - Equivalent to: - - ```python - request = client.build_request(...) - response = await client.send(request, ...) - ``` - - See `AsyncClient.build_request()`, `AsyncClient.send()` - and [Merging of configuration][0] for how the various parameters - are merged with client-level configuration. - - [0]: /advanced/clients/#merging-of-configuration - """ - - if cookies is not None: # pragma: no cover - message = ( - "Setting per-request cookies=<...> is being deprecated, because " - "the expected behaviour on cookie persistence is ambiguous. Set " - "cookies directly on the client instance instead." - ) - warnings.warn(message, DeprecationWarning, stacklevel=2) - - request = self.build_request( - method=method, - url=url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - timeout=timeout, - extensions=extensions, - ) - return await self.send(request, auth=auth, follow_redirects=follow_redirects) - - @asynccontextmanager - async def stream( - self, - method: str, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> typing.AsyncIterator[Response]: - """ - Alternative to `httpx.request()` that streams the response body - instead of loading it into memory at once. - - **Parameters**: See `httpx.request`. - - See also: [Streaming Responses][0] - - [0]: /quickstart#streaming-responses - """ - request = self.build_request( - method=method, - url=url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - timeout=timeout, - extensions=extensions, - ) - response = await self.send( - request=request, - auth=auth, - follow_redirects=follow_redirects, - stream=True, - ) - try: - yield response - finally: - await response.aclose() - - async def send( - self, - request: Request, - *, - stream: bool = False, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - ) -> Response: - """ - Send a request. - - The request is sent as-is, unmodified. - - Typically you'll want to build one with `AsyncClient.build_request()` - so that any client-level configuration is merged into the request, - but passing an explicit `httpx.Request()` is supported as well. - - See also: [Request instances][0] - - [0]: /advanced/clients/#request-instances - """ - if self._state == ClientState.CLOSED: - raise RuntimeError("Cannot send a request, as the client has been closed.") - - self._state = ClientState.OPENED - follow_redirects = ( - self.follow_redirects - if isinstance(follow_redirects, UseClientDefault) - else follow_redirects - ) - - self._set_timeout(request) - - auth = self._build_request_auth(request, auth) - - response = await self._send_handling_auth( - request, - auth=auth, - follow_redirects=follow_redirects, - history=[], - ) - try: - if not stream: - await response.aread() - - return response - - except BaseException as exc: - await response.aclose() - raise exc - - async def _send_handling_auth( - self, - request: Request, - auth: Auth, - follow_redirects: bool, - history: list[Response], - ) -> Response: - auth_flow = auth.async_auth_flow(request) - try: - request = await auth_flow.__anext__() - - while True: - response = await self._send_handling_redirects( - request, - follow_redirects=follow_redirects, - history=history, - ) - try: - try: - next_request = await auth_flow.asend(response) - except StopAsyncIteration: - return response - - response.history = list(history) - await response.aread() - request = next_request - history.append(response) - - except BaseException as exc: - await response.aclose() - raise exc - finally: - await auth_flow.aclose() - - async def _send_handling_redirects( - self, - request: Request, - follow_redirects: bool, - history: list[Response], - ) -> Response: - while True: - if len(history) > self.max_redirects: - raise TooManyRedirects( - "Exceeded maximum allowed redirects.", request=request - ) - - for hook in self._event_hooks["request"]: - await hook(request) - - response = await self._send_single_request(request) - try: - for hook in self._event_hooks["response"]: - await hook(response) - - response.history = list(history) - - if not response.has_redirect_location: - return response - - request = self._build_redirect_request(request, response) - history = history + [response] - - if follow_redirects: - await response.aread() - else: - response.next_request = request - return response - - except BaseException as exc: - await response.aclose() - raise exc - - async def _send_single_request(self, request: Request) -> Response: - """ - Sends a single request, without handling any redirections. - """ - transport = self._transport_for_url(request.url) - start = time.perf_counter() - - if not isinstance(request.stream, AsyncByteStream): - raise RuntimeError( - "Attempted to send a sync request with an AsyncClient instance." - ) - - with request_context(request=request): - response = await transport.handle_async_request(request) - - assert isinstance(response.stream, AsyncByteStream) - response.request = request - response.stream = BoundAsyncStream( - response.stream, response=response, start=start - ) - self.cookies.extract_cookies(response) - response.default_encoding = self._default_encoding - - logger.info( - 'HTTP Request: %s %s "%s %d %s"', - request.method, - request.url, - response.http_version, - response.status_code, - response.reason_phrase, - ) - - return response - - async def get( - self, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `GET` request. - - **Parameters**: See `httpx.request`. - """ - return await self.request( - "GET", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - async def options( - self, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send an `OPTIONS` request. - - **Parameters**: See `httpx.request`. - """ - return await self.request( - "OPTIONS", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - async def head( - self, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `HEAD` request. - - **Parameters**: See `httpx.request`. - """ - return await self.request( - "HEAD", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - async def post( - self, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `POST` request. - - **Parameters**: See `httpx.request`. - """ - return await self.request( - "POST", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - async def put( - self, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `PUT` request. - - **Parameters**: See `httpx.request`. - """ - return await self.request( - "PUT", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - async def patch( - self, - url: URL | str, - *, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `PATCH` request. - - **Parameters**: See `httpx.request`. - """ - return await self.request( - "PATCH", - url, - content=content, - data=data, - files=files, - json=json, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - async def delete( - self, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, - follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, - timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, - extensions: RequestExtensions | None = None, - ) -> Response: - """ - Send a `DELETE` request. - - **Parameters**: See `httpx.request`. - """ - return await self.request( - "DELETE", - url, - params=params, - headers=headers, - cookies=cookies, - auth=auth, - follow_redirects=follow_redirects, - timeout=timeout, - extensions=extensions, - ) - - async def aclose(self) -> None: - """ - Close transport and proxies. - """ - if self._state != ClientState.CLOSED: - self._state = ClientState.CLOSED - - await self._transport.aclose() - for proxy in self._mounts.values(): - if proxy is not None: - await proxy.aclose() - - async def __aenter__(self: U) -> U: - if self._state != ClientState.UNOPENED: - msg = { - ClientState.OPENED: "Cannot open a client instance more than once.", - ClientState.CLOSED: ( - "Cannot reopen a client instance, once it has been closed." - ), - }[self._state] - raise RuntimeError(msg) - - self._state = ClientState.OPENED - - await self._transport.__aenter__() - for proxy in self._mounts.values(): - if proxy is not None: - await proxy.__aenter__() - return self - - async def __aexit__( - self, - exc_type: type[BaseException] | None = None, - exc_value: BaseException | None = None, - traceback: TracebackType | None = None, - ) -> None: - self._state = ClientState.CLOSED - - await self._transport.__aexit__(exc_type, exc_value, traceback) - for proxy in self._mounts.values(): - if proxy is not None: - await proxy.__aexit__(exc_type, exc_value, traceback) diff --git a/httpx/_config.py b/httpx/_config.py deleted file mode 100644 index 467a6c90ae..0000000000 --- a/httpx/_config.py +++ /dev/null @@ -1,248 +0,0 @@ -from __future__ import annotations - -import os -import typing - -from ._models import Headers -from ._types import CertTypes, HeaderTypes, TimeoutTypes -from ._urls import URL - -if typing.TYPE_CHECKING: - import ssl # pragma: no cover - -__all__ = ["Limits", "Proxy", "Timeout", "create_ssl_context"] - - -class UnsetType: - pass # pragma: no cover - - -UNSET = UnsetType() - - -def create_ssl_context( - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - trust_env: bool = True, -) -> ssl.SSLContext: - import ssl - import warnings - - import certifi - - if verify is True: - if trust_env and os.environ.get("SSL_CERT_FILE"): # pragma: nocover - ctx = ssl.create_default_context(cafile=os.environ["SSL_CERT_FILE"]) - elif trust_env and os.environ.get("SSL_CERT_DIR"): # pragma: nocover - ctx = ssl.create_default_context(capath=os.environ["SSL_CERT_DIR"]) - else: - # Default case... - ctx = ssl.create_default_context(cafile=certifi.where()) - elif verify is False: - ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - elif isinstance(verify, str): # pragma: nocover - message = ( - "`verify=` is deprecated. " - "Use `verify=ssl.create_default_context(cafile=...)` " - "or `verify=ssl.create_default_context(capath=...)` instead." - ) - warnings.warn(message, DeprecationWarning) - if os.path.isdir(verify): - return ssl.create_default_context(capath=verify) - return ssl.create_default_context(cafile=verify) - else: - ctx = verify - - if cert: # pragma: nocover - message = ( - "`cert=...` is deprecated. Use `verify=` instead," - "with `.load_cert_chain()` to configure the certificate chain." - ) - warnings.warn(message, DeprecationWarning) - if isinstance(cert, str): - ctx.load_cert_chain(cert) - else: - ctx.load_cert_chain(*cert) - - return ctx - - -class Timeout: - """ - Timeout configuration. - - **Usage**: - - Timeout(None) # No timeouts. - Timeout(5.0) # 5s timeout on all operations. - Timeout(None, connect=5.0) # 5s timeout on connect, no other timeouts. - Timeout(5.0, connect=10.0) # 10s timeout on connect. 5s timeout elsewhere. - Timeout(5.0, pool=None) # No timeout on acquiring connection from pool. - # 5s timeout elsewhere. - """ - - def __init__( - self, - timeout: TimeoutTypes | UnsetType = UNSET, - *, - connect: None | float | UnsetType = UNSET, - read: None | float | UnsetType = UNSET, - write: None | float | UnsetType = UNSET, - pool: None | float | UnsetType = UNSET, - ) -> None: - if isinstance(timeout, Timeout): - # Passed as a single explicit Timeout. - assert connect is UNSET - assert read is UNSET - assert write is UNSET - assert pool is UNSET - self.connect = timeout.connect # type: typing.Optional[float] - self.read = timeout.read # type: typing.Optional[float] - self.write = timeout.write # type: typing.Optional[float] - self.pool = timeout.pool # type: typing.Optional[float] - elif isinstance(timeout, tuple): - # Passed as a tuple. - self.connect = timeout[0] - self.read = timeout[1] - self.write = None if len(timeout) < 3 else timeout[2] - self.pool = None if len(timeout) < 4 else timeout[3] - elif not ( - isinstance(connect, UnsetType) - or isinstance(read, UnsetType) - or isinstance(write, UnsetType) - or isinstance(pool, UnsetType) - ): - self.connect = connect - self.read = read - self.write = write - self.pool = pool - else: - if isinstance(timeout, UnsetType): - raise ValueError( - "httpx.Timeout must either include a default, or set all " - "four parameters explicitly." - ) - self.connect = timeout if isinstance(connect, UnsetType) else connect - self.read = timeout if isinstance(read, UnsetType) else read - self.write = timeout if isinstance(write, UnsetType) else write - self.pool = timeout if isinstance(pool, UnsetType) else pool - - def as_dict(self) -> dict[str, float | None]: - return { - "connect": self.connect, - "read": self.read, - "write": self.write, - "pool": self.pool, - } - - def __eq__(self, other: typing.Any) -> bool: - return ( - isinstance(other, self.__class__) - and self.connect == other.connect - and self.read == other.read - and self.write == other.write - and self.pool == other.pool - ) - - def __repr__(self) -> str: - class_name = self.__class__.__name__ - if len({self.connect, self.read, self.write, self.pool}) == 1: - return f"{class_name}(timeout={self.connect})" - return ( - f"{class_name}(connect={self.connect}, " - f"read={self.read}, write={self.write}, pool={self.pool})" - ) - - -class Limits: - """ - Configuration for limits to various client behaviors. - - **Parameters:** - - * **max_connections** - The maximum number of concurrent connections that may be - established. - * **max_keepalive_connections** - Allow the connection pool to maintain - keep-alive connections below this point. Should be less than or equal - to `max_connections`. - * **keepalive_expiry** - Time limit on idle keep-alive connections in seconds. - """ - - def __init__( - self, - *, - max_connections: int | None = None, - max_keepalive_connections: int | None = None, - keepalive_expiry: float | None = 5.0, - ) -> None: - self.max_connections = max_connections - self.max_keepalive_connections = max_keepalive_connections - self.keepalive_expiry = keepalive_expiry - - def __eq__(self, other: typing.Any) -> bool: - return ( - isinstance(other, self.__class__) - and self.max_connections == other.max_connections - and self.max_keepalive_connections == other.max_keepalive_connections - and self.keepalive_expiry == other.keepalive_expiry - ) - - def __repr__(self) -> str: - class_name = self.__class__.__name__ - return ( - f"{class_name}(max_connections={self.max_connections}, " - f"max_keepalive_connections={self.max_keepalive_connections}, " - f"keepalive_expiry={self.keepalive_expiry})" - ) - - -class Proxy: - def __init__( - self, - url: URL | str, - *, - ssl_context: ssl.SSLContext | None = None, - auth: tuple[str, str] | None = None, - headers: HeaderTypes | None = None, - ) -> None: - url = URL(url) - headers = Headers(headers) - - if url.scheme not in ("http", "https", "socks5", "socks5h"): - raise ValueError(f"Unknown scheme for proxy URL {url!r}") - - if url.username or url.password: - # Remove any auth credentials from the URL. - auth = (url.username, url.password) - url = url.copy_with(username=None, password=None) - - self.url = url - self.auth = auth - self.headers = headers - self.ssl_context = ssl_context - - @property - def raw_auth(self) -> tuple[bytes, bytes] | None: - # The proxy authentication as raw bytes. - return ( - None - if self.auth is None - else (self.auth[0].encode("utf-8"), self.auth[1].encode("utf-8")) - ) - - def __repr__(self) -> str: - # The authentication is represented with the password component masked. - auth = (self.auth[0], "********") if self.auth else None - - # Build a nice concise representation. - url_str = f"{str(self.url)!r}" - auth_str = f", auth={auth!r}" if auth else "" - headers_str = f", headers={dict(self.headers)!r}" if self.headers else "" - return f"Proxy({url_str}{auth_str}{headers_str})" - - -DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=5.0) -DEFAULT_LIMITS = Limits(max_connections=100, max_keepalive_connections=20) -DEFAULT_MAX_REDIRECTS = 20 diff --git a/httpx/_content.py b/httpx/_content.py deleted file mode 100644 index 6f479a0885..0000000000 --- a/httpx/_content.py +++ /dev/null @@ -1,240 +0,0 @@ -from __future__ import annotations - -import inspect -import warnings -from json import dumps as json_dumps -from typing import ( - Any, - AsyncIterable, - AsyncIterator, - Iterable, - Iterator, - Mapping, -) -from urllib.parse import urlencode - -from ._exceptions import StreamClosed, StreamConsumed -from ._multipart import MultipartStream -from ._types import ( - AsyncByteStream, - RequestContent, - RequestData, - RequestFiles, - ResponseContent, - SyncByteStream, -) -from ._utils import peek_filelike_length, primitive_value_to_str - -__all__ = ["ByteStream"] - - -class ByteStream(AsyncByteStream, SyncByteStream): - def __init__(self, stream: bytes) -> None: - self._stream = stream - - def __iter__(self) -> Iterator[bytes]: - yield self._stream - - async def __aiter__(self) -> AsyncIterator[bytes]: - yield self._stream - - -class IteratorByteStream(SyncByteStream): - CHUNK_SIZE = 65_536 - - def __init__(self, stream: Iterable[bytes]) -> None: - self._stream = stream - self._is_stream_consumed = False - self._is_generator = inspect.isgenerator(stream) - - def __iter__(self) -> Iterator[bytes]: - if self._is_stream_consumed and self._is_generator: - raise StreamConsumed() - - self._is_stream_consumed = True - if hasattr(self._stream, "read"): - # File-like interfaces should use 'read' directly. - chunk = self._stream.read(self.CHUNK_SIZE) - while chunk: - yield chunk - chunk = self._stream.read(self.CHUNK_SIZE) - else: - # Otherwise iterate. - for part in self._stream: - yield part - - -class AsyncIteratorByteStream(AsyncByteStream): - CHUNK_SIZE = 65_536 - - def __init__(self, stream: AsyncIterable[bytes]) -> None: - self._stream = stream - self._is_stream_consumed = False - self._is_generator = inspect.isasyncgen(stream) - - async def __aiter__(self) -> AsyncIterator[bytes]: - if self._is_stream_consumed and self._is_generator: - raise StreamConsumed() - - self._is_stream_consumed = True - if hasattr(self._stream, "aread"): - # File-like interfaces should use 'aread' directly. - chunk = await self._stream.aread(self.CHUNK_SIZE) - while chunk: - yield chunk - chunk = await self._stream.aread(self.CHUNK_SIZE) - else: - # Otherwise iterate. - async for part in self._stream: - yield part - - -class UnattachedStream(AsyncByteStream, SyncByteStream): - """ - If a request or response is serialized using pickle, then it is no longer - attached to a stream for I/O purposes. Any stream operations should result - in `httpx.StreamClosed`. - """ - - def __iter__(self) -> Iterator[bytes]: - raise StreamClosed() - - async def __aiter__(self) -> AsyncIterator[bytes]: - raise StreamClosed() - yield b"" # pragma: no cover - - -def encode_content( - content: str | bytes | Iterable[bytes] | AsyncIterable[bytes], -) -> tuple[dict[str, str], SyncByteStream | AsyncByteStream]: - if isinstance(content, (bytes, str)): - body = content.encode("utf-8") if isinstance(content, str) else content - content_length = len(body) - headers = {"Content-Length": str(content_length)} if body else {} - return headers, ByteStream(body) - - elif isinstance(content, Iterable) and not isinstance(content, dict): - # `not isinstance(content, dict)` is a bit oddly specific, but it - # catches a case that's easy for users to make in error, and would - # otherwise pass through here, like any other bytes-iterable, - # because `dict` happens to be iterable. See issue #2491. - content_length_or_none = peek_filelike_length(content) - - if content_length_or_none is None: - headers = {"Transfer-Encoding": "chunked"} - else: - headers = {"Content-Length": str(content_length_or_none)} - return headers, IteratorByteStream(content) # type: ignore - - elif isinstance(content, AsyncIterable): - headers = {"Transfer-Encoding": "chunked"} - return headers, AsyncIteratorByteStream(content) - - raise TypeError(f"Unexpected type for 'content', {type(content)!r}") - - -def encode_urlencoded_data( - data: RequestData, -) -> tuple[dict[str, str], ByteStream]: - plain_data = [] - for key, value in data.items(): - if isinstance(value, (list, tuple)): - plain_data.extend([(key, primitive_value_to_str(item)) for item in value]) - else: - plain_data.append((key, primitive_value_to_str(value))) - body = urlencode(plain_data, doseq=True).encode("utf-8") - content_length = str(len(body)) - content_type = "application/x-www-form-urlencoded" - headers = {"Content-Length": content_length, "Content-Type": content_type} - return headers, ByteStream(body) - - -def encode_multipart_data( - data: RequestData, files: RequestFiles, boundary: bytes | None -) -> tuple[dict[str, str], MultipartStream]: - multipart = MultipartStream(data=data, files=files, boundary=boundary) - headers = multipart.get_headers() - return headers, multipart - - -def encode_text(text: str) -> tuple[dict[str, str], ByteStream]: - body = text.encode("utf-8") - content_length = str(len(body)) - content_type = "text/plain; charset=utf-8" - headers = {"Content-Length": content_length, "Content-Type": content_type} - return headers, ByteStream(body) - - -def encode_html(html: str) -> tuple[dict[str, str], ByteStream]: - body = html.encode("utf-8") - content_length = str(len(body)) - content_type = "text/html; charset=utf-8" - headers = {"Content-Length": content_length, "Content-Type": content_type} - return headers, ByteStream(body) - - -def encode_json(json: Any) -> tuple[dict[str, str], ByteStream]: - body = json_dumps( - json, ensure_ascii=False, separators=(",", ":"), allow_nan=False - ).encode("utf-8") - content_length = str(len(body)) - content_type = "application/json" - headers = {"Content-Length": content_length, "Content-Type": content_type} - return headers, ByteStream(body) - - -def encode_request( - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: Any | None = None, - boundary: bytes | None = None, -) -> tuple[dict[str, str], SyncByteStream | AsyncByteStream]: - """ - Handles encoding the given `content`, `data`, `files`, and `json`, - returning a two-tuple of (, ). - """ - if data is not None and not isinstance(data, Mapping): - # We prefer to separate `content=` - # for raw request content, and `data=` for url encoded or - # multipart form content. - # - # However for compat with requests, we *do* still support - # `data=` usages. We deal with that case here, treating it - # as if `content=<...>` had been supplied instead. - message = "Use 'content=<...>' to upload raw bytes/text content." - warnings.warn(message, DeprecationWarning, stacklevel=2) - return encode_content(data) - - if content is not None: - return encode_content(content) - elif files: - return encode_multipart_data(data or {}, files, boundary) - elif data: - return encode_urlencoded_data(data) - elif json is not None: - return encode_json(json) - - return {}, ByteStream(b"") - - -def encode_response( - content: ResponseContent | None = None, - text: str | None = None, - html: str | None = None, - json: Any | None = None, -) -> tuple[dict[str, str], SyncByteStream | AsyncByteStream]: - """ - Handles encoding the given `content`, returning a two-tuple of - (, ). - """ - if content is not None: - return encode_content(content) - elif text is not None: - return encode_text(text) - elif html is not None: - return encode_html(html) - elif json is not None: - return encode_json(json) - - return {}, ByteStream(b"") diff --git a/httpx/_decoders.py b/httpx/_decoders.py deleted file mode 100644 index 899dfada87..0000000000 --- a/httpx/_decoders.py +++ /dev/null @@ -1,393 +0,0 @@ -""" -Handlers for Content-Encoding. - -See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding -""" - -from __future__ import annotations - -import codecs -import io -import typing -import zlib - -from ._exceptions import DecodingError - -# Brotli support is optional -try: - # The C bindings in `brotli` are recommended for CPython. - import brotli -except ImportError: # pragma: no cover - try: - # The CFFI bindings in `brotlicffi` are recommended for PyPy - # and other environments. - import brotlicffi as brotli - except ImportError: - brotli = None - - -# Zstandard support is optional -try: - import zstandard -except ImportError: # pragma: no cover - zstandard = None # type: ignore - - -class ContentDecoder: - def decode(self, data: bytes) -> bytes: - raise NotImplementedError() # pragma: no cover - - def flush(self) -> bytes: - raise NotImplementedError() # pragma: no cover - - -class IdentityDecoder(ContentDecoder): - """ - Handle unencoded data. - """ - - def decode(self, data: bytes) -> bytes: - return data - - def flush(self) -> bytes: - return b"" - - -class DeflateDecoder(ContentDecoder): - """ - Handle 'deflate' decoding. - - See: https://stackoverflow.com/questions/1838699 - """ - - def __init__(self) -> None: - self.first_attempt = True - self.decompressor = zlib.decompressobj() - - def decode(self, data: bytes) -> bytes: - was_first_attempt = self.first_attempt - self.first_attempt = False - try: - return self.decompressor.decompress(data) - except zlib.error as exc: - if was_first_attempt: - self.decompressor = zlib.decompressobj(-zlib.MAX_WBITS) - return self.decode(data) - raise DecodingError(str(exc)) from exc - - def flush(self) -> bytes: - try: - return self.decompressor.flush() - except zlib.error as exc: # pragma: no cover - raise DecodingError(str(exc)) from exc - - -class GZipDecoder(ContentDecoder): - """ - Handle 'gzip' decoding. - - See: https://stackoverflow.com/questions/1838699 - """ - - def __init__(self) -> None: - self.decompressor = zlib.decompressobj(zlib.MAX_WBITS | 16) - - def decode(self, data: bytes) -> bytes: - try: - return self.decompressor.decompress(data) - except zlib.error as exc: - raise DecodingError(str(exc)) from exc - - def flush(self) -> bytes: - try: - return self.decompressor.flush() - except zlib.error as exc: # pragma: no cover - raise DecodingError(str(exc)) from exc - - -class BrotliDecoder(ContentDecoder): - """ - Handle 'brotli' decoding. - - Requires `pip install brotlipy`. See: https://brotlipy.readthedocs.io/ - or `pip install brotli`. See https://github.com/google/brotli - Supports both 'brotlipy' and 'Brotli' packages since they share an import - name. The top branches are for 'brotlipy' and bottom branches for 'Brotli' - """ - - def __init__(self) -> None: - if brotli is None: # pragma: no cover - raise ImportError( - "Using 'BrotliDecoder', but neither of the 'brotlicffi' or 'brotli' " - "packages have been installed. " - "Make sure to install httpx using `pip install httpx[brotli]`." - ) from None - - self.decompressor = brotli.Decompressor() - self.seen_data = False - self._decompress: typing.Callable[[bytes], bytes] - if hasattr(self.decompressor, "decompress"): - # The 'brotlicffi' package. - self._decompress = self.decompressor.decompress # pragma: no cover - else: - # The 'brotli' package. - self._decompress = self.decompressor.process # pragma: no cover - - def decode(self, data: bytes) -> bytes: - if not data: - return b"" - self.seen_data = True - try: - return self._decompress(data) - except brotli.error as exc: - raise DecodingError(str(exc)) from exc - - def flush(self) -> bytes: - if not self.seen_data: - return b"" - try: - if hasattr(self.decompressor, "finish"): - # Only available in the 'brotlicffi' package. - - # As the decompressor decompresses eagerly, this - # will never actually emit any data. However, it will potentially throw - # errors if a truncated or damaged data stream has been used. - self.decompressor.finish() # pragma: no cover - return b"" - except brotli.error as exc: # pragma: no cover - raise DecodingError(str(exc)) from exc - - -class ZStandardDecoder(ContentDecoder): - """ - Handle 'zstd' RFC 8878 decoding. - - Requires `pip install zstandard`. - Can be installed as a dependency of httpx using `pip install httpx[zstd]`. - """ - - # inspired by the ZstdDecoder implementation in urllib3 - def __init__(self) -> None: - if zstandard is None: # pragma: no cover - raise ImportError( - "Using 'ZStandardDecoder', ..." - "Make sure to install httpx using `pip install httpx[zstd]`." - ) from None - - self.decompressor = zstandard.ZstdDecompressor().decompressobj() - self.seen_data = False - - def decode(self, data: bytes) -> bytes: - assert zstandard is not None - self.seen_data = True - output = io.BytesIO() - try: - output.write(self.decompressor.decompress(data)) - while self.decompressor.eof and self.decompressor.unused_data: - unused_data = self.decompressor.unused_data - self.decompressor = zstandard.ZstdDecompressor().decompressobj() - output.write(self.decompressor.decompress(unused_data)) - except zstandard.ZstdError as exc: - raise DecodingError(str(exc)) from exc - return output.getvalue() - - def flush(self) -> bytes: - if not self.seen_data: - return b"" - ret = self.decompressor.flush() # note: this is a no-op - if not self.decompressor.eof: - raise DecodingError("Zstandard data is incomplete") # pragma: no cover - return bytes(ret) - - -class MultiDecoder(ContentDecoder): - """ - Handle the case where multiple encodings have been applied. - """ - - def __init__(self, children: typing.Sequence[ContentDecoder]) -> None: - """ - 'children' should be a sequence of decoders in the order in which - each was applied. - """ - # Note that we reverse the order for decoding. - self.children = list(reversed(children)) - - def decode(self, data: bytes) -> bytes: - for child in self.children: - data = child.decode(data) - return data - - def flush(self) -> bytes: - data = b"" - for child in self.children: - data = child.decode(data) + child.flush() - return data - - -class ByteChunker: - """ - Handles returning byte content in fixed-size chunks. - """ - - def __init__(self, chunk_size: int | None = None) -> None: - self._buffer = io.BytesIO() - self._chunk_size = chunk_size - - def decode(self, content: bytes) -> list[bytes]: - if self._chunk_size is None: - return [content] if content else [] - - self._buffer.write(content) - if self._buffer.tell() >= self._chunk_size: - value = self._buffer.getvalue() - chunks = [ - value[i : i + self._chunk_size] - for i in range(0, len(value), self._chunk_size) - ] - if len(chunks[-1]) == self._chunk_size: - self._buffer.seek(0) - self._buffer.truncate() - return chunks - else: - self._buffer.seek(0) - self._buffer.write(chunks[-1]) - self._buffer.truncate() - return chunks[:-1] - else: - return [] - - def flush(self) -> list[bytes]: - value = self._buffer.getvalue() - self._buffer.seek(0) - self._buffer.truncate() - return [value] if value else [] - - -class TextChunker: - """ - Handles returning text content in fixed-size chunks. - """ - - def __init__(self, chunk_size: int | None = None) -> None: - self._buffer = io.StringIO() - self._chunk_size = chunk_size - - def decode(self, content: str) -> list[str]: - if self._chunk_size is None: - return [content] if content else [] - - self._buffer.write(content) - if self._buffer.tell() >= self._chunk_size: - value = self._buffer.getvalue() - chunks = [ - value[i : i + self._chunk_size] - for i in range(0, len(value), self._chunk_size) - ] - if len(chunks[-1]) == self._chunk_size: - self._buffer.seek(0) - self._buffer.truncate() - return chunks - else: - self._buffer.seek(0) - self._buffer.write(chunks[-1]) - self._buffer.truncate() - return chunks[:-1] - else: - return [] - - def flush(self) -> list[str]: - value = self._buffer.getvalue() - self._buffer.seek(0) - self._buffer.truncate() - return [value] if value else [] - - -class TextDecoder: - """ - Handles incrementally decoding bytes into text - """ - - def __init__(self, encoding: str = "utf-8") -> None: - self.decoder = codecs.getincrementaldecoder(encoding)(errors="replace") - - def decode(self, data: bytes) -> str: - return self.decoder.decode(data) - - def flush(self) -> str: - return self.decoder.decode(b"", True) - - -class LineDecoder: - """ - Handles incrementally reading lines from text. - - Has the same behaviour as the stdllib splitlines, - but handling the input iteratively. - """ - - def __init__(self) -> None: - self.buffer: list[str] = [] - self.trailing_cr: bool = False - - def decode(self, text: str) -> list[str]: - # See https://docs.python.org/3/library/stdtypes.html#str.splitlines - NEWLINE_CHARS = "\n\r\x0b\x0c\x1c\x1d\x1e\x85\u2028\u2029" - - # We always push a trailing `\r` into the next decode iteration. - if self.trailing_cr: - text = "\r" + text - self.trailing_cr = False - if text.endswith("\r"): - self.trailing_cr = True - text = text[:-1] - - if not text: - # NOTE: the edge case input of empty text doesn't occur in practice, - # because other httpx internals filter out this value - return [] # pragma: no cover - - trailing_newline = text[-1] in NEWLINE_CHARS - lines = text.splitlines() - - if len(lines) == 1 and not trailing_newline: - # No new lines, buffer the input and continue. - self.buffer.append(lines[0]) - return [] - - if self.buffer: - # Include any existing buffer in the first portion of the - # splitlines result. - lines = ["".join(self.buffer) + lines[0]] + lines[1:] - self.buffer = [] - - if not trailing_newline: - # If the last segment of splitlines is not newline terminated, - # then drop it from our output and start a new buffer. - self.buffer = [lines.pop()] - - return lines - - def flush(self) -> list[str]: - if not self.buffer and not self.trailing_cr: - return [] - - lines = ["".join(self.buffer)] - self.buffer = [] - self.trailing_cr = False - return lines - - -SUPPORTED_DECODERS = { - "identity": IdentityDecoder, - "gzip": GZipDecoder, - "deflate": DeflateDecoder, - "br": BrotliDecoder, - "zstd": ZStandardDecoder, -} - - -if brotli is None: - SUPPORTED_DECODERS.pop("br") # pragma: no cover -if zstandard is None: - SUPPORTED_DECODERS.pop("zstd") # pragma: no cover diff --git a/httpx/_exceptions.py b/httpx/_exceptions.py deleted file mode 100644 index dd7fb6cdb3..0000000000 --- a/httpx/_exceptions.py +++ /dev/null @@ -1,377 +0,0 @@ -""" -Our exception hierarchy: - -* HTTPError - x RequestError - + TransportError - - TimeoutException - · ConnectTimeout - · ReadTimeout - · WriteTimeout - · PoolTimeout - - NetworkError - · ConnectError - · ReadError - · WriteError - · CloseError - - ProtocolError - · LocalProtocolError - · RemoteProtocolError - - ProxyError - - UnsupportedProtocol - + DecodingError - + TooManyRedirects - x HTTPStatusError -* InvalidURL -* CookieConflict -* StreamError - x StreamConsumed - x StreamClosed - x ResponseNotRead - x RequestNotRead -""" - -from __future__ import annotations - -import contextlib -import typing - -if typing.TYPE_CHECKING: - from ._models import Request, Response # pragma: no cover - -__all__ = [ - "CloseError", - "ConnectError", - "ConnectTimeout", - "CookieConflict", - "DecodingError", - "HTTPError", - "HTTPStatusError", - "InvalidURL", - "LocalProtocolError", - "NetworkError", - "PoolTimeout", - "ProtocolError", - "ProxyError", - "ReadError", - "ReadTimeout", - "RemoteProtocolError", - "RequestError", - "RequestNotRead", - "ResponseNotRead", - "StreamClosed", - "StreamConsumed", - "StreamError", - "TimeoutException", - "TooManyRedirects", - "TransportError", - "UnsupportedProtocol", - "WriteError", - "WriteTimeout", -] - - -class HTTPError(Exception): - """ - Base class for `RequestError` and `HTTPStatusError`. - - Useful for `try...except` blocks when issuing a request, - and then calling `.raise_for_status()`. - - For example: - - ``` - try: - response = httpx.get("https://www.example.com") - response.raise_for_status() - except httpx.HTTPError as exc: - print(f"HTTP Exception for {exc.request.url} - {exc}") - ``` - """ - - def __init__(self, message: str) -> None: - super().__init__(message) - self._request: Request | None = None - - @property - def request(self) -> Request: - if self._request is None: - raise RuntimeError("The .request property has not been set.") - return self._request - - @request.setter - def request(self, request: Request) -> None: - self._request = request - - -class RequestError(HTTPError): - """ - Base class for all exceptions that may occur when issuing a `.request()`. - """ - - def __init__(self, message: str, *, request: Request | None = None) -> None: - super().__init__(message) - # At the point an exception is raised we won't typically have a request - # instance to associate it with. - # - # The 'request_context' context manager is used within the Client and - # Response methods in order to ensure that any raised exceptions - # have a `.request` property set on them. - self._request = request - - -class TransportError(RequestError): - """ - Base class for all exceptions that occur at the level of the Transport API. - """ - - -# Timeout exceptions... - - -class TimeoutException(TransportError): - """ - The base class for timeout errors. - - An operation has timed out. - """ - - -class ConnectTimeout(TimeoutException): - """ - Timed out while connecting to the host. - """ - - -class ReadTimeout(TimeoutException): - """ - Timed out while receiving data from the host. - """ - - -class WriteTimeout(TimeoutException): - """ - Timed out while sending data to the host. - """ - - -class PoolTimeout(TimeoutException): - """ - Timed out waiting to acquire a connection from the pool. - """ - - -# Core networking exceptions... - - -class NetworkError(TransportError): - """ - The base class for network-related errors. - - An error occurred while interacting with the network. - """ - - -class ReadError(NetworkError): - """ - Failed to receive data from the network. - """ - - -class WriteError(NetworkError): - """ - Failed to send data through the network. - """ - - -class ConnectError(NetworkError): - """ - Failed to establish a connection. - """ - - -class CloseError(NetworkError): - """ - Failed to close a connection. - """ - - -# Other transport exceptions... - - -class ProxyError(TransportError): - """ - An error occurred while establishing a proxy connection. - """ - - -class UnsupportedProtocol(TransportError): - """ - Attempted to make a request to an unsupported protocol. - - For example issuing a request to `ftp://www.example.com`. - """ - - -class ProtocolError(TransportError): - """ - The protocol was violated. - """ - - -class LocalProtocolError(ProtocolError): - """ - A protocol was violated by the client. - - For example if the user instantiated a `Request` instance explicitly, - failed to include the mandatory `Host:` header, and then issued it directly - using `client.send()`. - """ - - -class RemoteProtocolError(ProtocolError): - """ - The protocol was violated by the server. - - For example, returning malformed HTTP. - """ - - -# Other request exceptions... - - -class DecodingError(RequestError): - """ - Decoding of the response failed, due to a malformed encoding. - """ - - -class TooManyRedirects(RequestError): - """ - Too many redirects. - """ - - -# Client errors - - -class HTTPStatusError(HTTPError): - """ - The response had an error HTTP status of 4xx or 5xx. - - May be raised when calling `response.raise_for_status()` - """ - - def __init__(self, message: str, *, request: Request, response: Response) -> None: - super().__init__(message) - self.request = request - self.response = response - - -class InvalidURL(Exception): - """ - URL is improperly formed or cannot be parsed. - """ - - def __init__(self, message: str) -> None: - super().__init__(message) - - -class CookieConflict(Exception): - """ - Attempted to lookup a cookie by name, but multiple cookies existed. - - Can occur when calling `response.cookies.get(...)`. - """ - - def __init__(self, message: str) -> None: - super().__init__(message) - - -# Stream exceptions... - -# These may occur as the result of a programming error, by accessing -# the request/response stream in an invalid manner. - - -class StreamError(RuntimeError): - """ - The base class for stream exceptions. - - The developer made an error in accessing the request stream in - an invalid way. - """ - - def __init__(self, message: str) -> None: - super().__init__(message) - - -class StreamConsumed(StreamError): - """ - Attempted to read or stream content, but the content has already - been streamed. - """ - - def __init__(self) -> None: - message = ( - "Attempted to read or stream some content, but the content has " - "already been streamed. For requests, this could be due to passing " - "a generator as request content, and then receiving a redirect " - "response or a secondary request as part of an authentication flow." - "For responses, this could be due to attempting to stream the response " - "content more than once." - ) - super().__init__(message) - - -class StreamClosed(StreamError): - """ - Attempted to read or stream response content, but the request has been - closed. - """ - - def __init__(self) -> None: - message = "Attempted to read or stream content, but the stream has been closed." - super().__init__(message) - - -class ResponseNotRead(StreamError): - """ - Attempted to access streaming response content, without having called `read()`. - """ - - def __init__(self) -> None: - message = ( - "Attempted to access streaming response content," - " without having called `read()`." - ) - super().__init__(message) - - -class RequestNotRead(StreamError): - """ - Attempted to access streaming request content, without having called `read()`. - """ - - def __init__(self) -> None: - message = ( - "Attempted to access streaming request content," - " without having called `read()`." - ) - super().__init__(message) - - -@contextlib.contextmanager -def request_context( - request: Request | None = None, -) -> typing.Iterator[None]: - """ - A context manager that can be used to attach the given request context - to any `RequestError` exceptions that are raised within the block. - """ - try: - yield - except RequestError as exc: - if request is not None: - exc.request = request - raise exc diff --git a/httpx/_main.py b/httpx/_main.py deleted file mode 100644 index cffa4bb7db..0000000000 --- a/httpx/_main.py +++ /dev/null @@ -1,506 +0,0 @@ -from __future__ import annotations - -import functools -import json -import sys -import typing - -import click -import pygments.lexers -import pygments.util -import rich.console -import rich.markup -import rich.progress -import rich.syntax -import rich.table - -from ._client import Client -from ._exceptions import RequestError -from ._models import Response -from ._status_codes import codes - -if typing.TYPE_CHECKING: - import httpcore # pragma: no cover - - -def print_help() -> None: - console = rich.console.Console() - - console.print("[bold]HTTPX :butterfly:", justify="center") - console.print() - console.print("A next generation HTTP client.", justify="center") - console.print() - console.print( - "Usage: [bold]httpx[/bold] [cyan] [OPTIONS][/cyan] ", justify="left" - ) - console.print() - - table = rich.table.Table.grid(padding=1, pad_edge=True) - table.add_column("Parameter", no_wrap=True, justify="left", style="bold") - table.add_column("Description") - table.add_row( - "-m, --method [cyan]METHOD", - "Request method, such as GET, POST, PUT, PATCH, DELETE, OPTIONS, HEAD.\n" - "[Default: GET, or POST if a request body is included]", - ) - table.add_row( - "-p, --params [cyan] ...", - "Query parameters to include in the request URL.", - ) - table.add_row( - "-c, --content [cyan]TEXT", "Byte content to include in the request body." - ) - table.add_row( - "-d, --data [cyan] ...", "Form data to include in the request body." - ) - table.add_row( - "-f, --files [cyan] ...", - "Form files to include in the request body.", - ) - table.add_row("-j, --json [cyan]TEXT", "JSON data to include in the request body.") - table.add_row( - "-h, --headers [cyan] ...", - "Include additional HTTP headers in the request.", - ) - table.add_row( - "--cookies [cyan] ...", "Cookies to include in the request." - ) - table.add_row( - "--auth [cyan]", - "Username and password to include in the request. Specify '-' for the password" - " to use a password prompt. Note that using --verbose/-v will expose" - " the Authorization header, including the password encoding" - " in a trivially reversible format.", - ) - - table.add_row( - "--proxy [cyan]URL", - "Send the request via a proxy. Should be the URL giving the proxy address.", - ) - - table.add_row( - "--timeout [cyan]FLOAT", - "Timeout value to use for network operations, such as establishing the" - " connection, reading some data, etc... [Default: 5.0]", - ) - - table.add_row("--follow-redirects", "Automatically follow redirects.") - table.add_row("--no-verify", "Disable SSL verification.") - table.add_row( - "--http2", "Send the request using HTTP/2, if the remote server supports it." - ) - - table.add_row( - "--download [cyan]FILE", - "Save the response content as a file, rather than displaying it.", - ) - - table.add_row("-v, --verbose", "Verbose output. Show request as well as response.") - table.add_row("--help", "Show this message and exit.") - console.print(table) - - -def get_lexer_for_response(response: Response) -> str: - content_type = response.headers.get("Content-Type") - if content_type is not None: - mime_type, _, _ = content_type.partition(";") - try: - return typing.cast( - str, pygments.lexers.get_lexer_for_mimetype(mime_type.strip()).name - ) - except pygments.util.ClassNotFound: # pragma: no cover - pass - return "" # pragma: no cover - - -def format_request_headers(request: httpcore.Request, http2: bool = False) -> str: - version = "HTTP/2" if http2 else "HTTP/1.1" - headers = [ - (name.lower() if http2 else name, value) for name, value in request.headers - ] - method = request.method.decode("ascii") - target = request.url.target.decode("ascii") - lines = [f"{method} {target} {version}"] + [ - f"{name.decode('ascii')}: {value.decode('ascii')}" for name, value in headers - ] - return "\n".join(lines) - - -def format_response_headers( - http_version: bytes, - status: int, - reason_phrase: bytes | None, - headers: list[tuple[bytes, bytes]], -) -> str: - version = http_version.decode("ascii") - reason = ( - codes.get_reason_phrase(status) - if reason_phrase is None - else reason_phrase.decode("ascii") - ) - lines = [f"{version} {status} {reason}"] + [ - f"{name.decode('ascii')}: {value.decode('ascii')}" for name, value in headers - ] - return "\n".join(lines) - - -def print_request_headers(request: httpcore.Request, http2: bool = False) -> None: - console = rich.console.Console() - http_text = format_request_headers(request, http2=http2) - syntax = rich.syntax.Syntax(http_text, "http", theme="ansi_dark", word_wrap=True) - console.print(syntax) - syntax = rich.syntax.Syntax("", "http", theme="ansi_dark", word_wrap=True) - console.print(syntax) - - -def print_response_headers( - http_version: bytes, - status: int, - reason_phrase: bytes | None, - headers: list[tuple[bytes, bytes]], -) -> None: - console = rich.console.Console() - http_text = format_response_headers(http_version, status, reason_phrase, headers) - syntax = rich.syntax.Syntax(http_text, "http", theme="ansi_dark", word_wrap=True) - console.print(syntax) - syntax = rich.syntax.Syntax("", "http", theme="ansi_dark", word_wrap=True) - console.print(syntax) - - -def print_response(response: Response) -> None: - console = rich.console.Console() - lexer_name = get_lexer_for_response(response) - if lexer_name: - if lexer_name.lower() == "json": - try: - data = response.json() - text = json.dumps(data, indent=4) - except ValueError: # pragma: no cover - text = response.text - else: - text = response.text - - syntax = rich.syntax.Syntax(text, lexer_name, theme="ansi_dark", word_wrap=True) - console.print(syntax) - else: - console.print(f"<{len(response.content)} bytes of binary data>") - - -_PCTRTT = typing.Tuple[typing.Tuple[str, str], ...] -_PCTRTTT = typing.Tuple[_PCTRTT, ...] -_PeerCertRetDictType = typing.Dict[str, typing.Union[str, _PCTRTTT, _PCTRTT]] - - -def format_certificate(cert: _PeerCertRetDictType) -> str: # pragma: no cover - lines = [] - for key, value in cert.items(): - if isinstance(value, (list, tuple)): - lines.append(f"* {key}:") - for item in value: - if key in ("subject", "issuer"): - for sub_item in item: - lines.append(f"* {sub_item[0]}: {sub_item[1]!r}") - elif isinstance(item, tuple) and len(item) == 2: - lines.append(f"* {item[0]}: {item[1]!r}") - else: - lines.append(f"* {item!r}") - else: - lines.append(f"* {key}: {value!r}") - return "\n".join(lines) - - -def trace( - name: str, info: typing.Mapping[str, typing.Any], verbose: bool = False -) -> None: - console = rich.console.Console() - if name == "connection.connect_tcp.started" and verbose: - host = info["host"] - console.print(f"* Connecting to {host!r}") - elif name == "connection.connect_tcp.complete" and verbose: - stream = info["return_value"] - server_addr = stream.get_extra_info("server_addr") - console.print(f"* Connected to {server_addr[0]!r} on port {server_addr[1]}") - elif name == "connection.start_tls.complete" and verbose: # pragma: no cover - stream = info["return_value"] - ssl_object = stream.get_extra_info("ssl_object") - version = ssl_object.version() - cipher = ssl_object.cipher() - server_cert = ssl_object.getpeercert() - alpn = ssl_object.selected_alpn_protocol() - console.print(f"* SSL established using {version!r} / {cipher[0]!r}") - console.print(f"* Selected ALPN protocol: {alpn!r}") - if server_cert: - console.print("* Server certificate:") - console.print(format_certificate(server_cert)) - elif name == "http11.send_request_headers.started" and verbose: - request = info["request"] - print_request_headers(request, http2=False) - elif name == "http2.send_request_headers.started" and verbose: # pragma: no cover - request = info["request"] - print_request_headers(request, http2=True) - elif name == "http11.receive_response_headers.complete": - http_version, status, reason_phrase, headers = info["return_value"] - print_response_headers(http_version, status, reason_phrase, headers) - elif name == "http2.receive_response_headers.complete": # pragma: no cover - status, headers = info["return_value"] - http_version = b"HTTP/2" - reason_phrase = None - print_response_headers(http_version, status, reason_phrase, headers) - - -def download_response(response: Response, download: typing.BinaryIO) -> None: - console = rich.console.Console() - console.print() - content_length = response.headers.get("Content-Length") - with rich.progress.Progress( - "[progress.description]{task.description}", - "[progress.percentage]{task.percentage:>3.0f}%", - rich.progress.BarColumn(bar_width=None), - rich.progress.DownloadColumn(), - rich.progress.TransferSpeedColumn(), - ) as progress: - description = f"Downloading [bold]{rich.markup.escape(download.name)}" - download_task = progress.add_task( - description, - total=int(content_length or 0), - start=content_length is not None, - ) - for chunk in response.iter_bytes(): - download.write(chunk) - progress.update(download_task, completed=response.num_bytes_downloaded) - - -def validate_json( - ctx: click.Context, - param: click.Option | click.Parameter, - value: typing.Any, -) -> typing.Any: - if value is None: - return None - - try: - return json.loads(value) - except json.JSONDecodeError: # pragma: no cover - raise click.BadParameter("Not valid JSON") - - -def validate_auth( - ctx: click.Context, - param: click.Option | click.Parameter, - value: typing.Any, -) -> typing.Any: - if value == (None, None): - return None - - username, password = value - if password == "-": # pragma: no cover - password = click.prompt("Password", hide_input=True) - return (username, password) - - -def handle_help( - ctx: click.Context, - param: click.Option | click.Parameter, - value: typing.Any, -) -> None: - if not value or ctx.resilient_parsing: - return - - print_help() - ctx.exit() - - -@click.command(add_help_option=False) -@click.argument("url", type=str) -@click.option( - "--method", - "-m", - "method", - type=str, - help=( - "Request method, such as GET, POST, PUT, PATCH, DELETE, OPTIONS, HEAD. " - "[Default: GET, or POST if a request body is included]" - ), -) -@click.option( - "--params", - "-p", - "params", - type=(str, str), - multiple=True, - help="Query parameters to include in the request URL.", -) -@click.option( - "--content", - "-c", - "content", - type=str, - help="Byte content to include in the request body.", -) -@click.option( - "--data", - "-d", - "data", - type=(str, str), - multiple=True, - help="Form data to include in the request body.", -) -@click.option( - "--files", - "-f", - "files", - type=(str, click.File(mode="rb")), - multiple=True, - help="Form files to include in the request body.", -) -@click.option( - "--json", - "-j", - "json", - type=str, - callback=validate_json, - help="JSON data to include in the request body.", -) -@click.option( - "--headers", - "-h", - "headers", - type=(str, str), - multiple=True, - help="Include additional HTTP headers in the request.", -) -@click.option( - "--cookies", - "cookies", - type=(str, str), - multiple=True, - help="Cookies to include in the request.", -) -@click.option( - "--auth", - "auth", - type=(str, str), - default=(None, None), - callback=validate_auth, - help=( - "Username and password to include in the request. " - "Specify '-' for the password to use a password prompt. " - "Note that using --verbose/-v will expose the Authorization header, " - "including the password encoding in a trivially reversible format." - ), -) -@click.option( - "--proxy", - "proxy", - type=str, - default=None, - help="Send the request via a proxy. Should be the URL giving the proxy address.", -) -@click.option( - "--timeout", - "timeout", - type=float, - default=5.0, - help=( - "Timeout value to use for network operations, such as establishing the " - "connection, reading some data, etc... [Default: 5.0]" - ), -) -@click.option( - "--follow-redirects", - "follow_redirects", - is_flag=True, - default=False, - help="Automatically follow redirects.", -) -@click.option( - "--no-verify", - "verify", - is_flag=True, - default=True, - help="Disable SSL verification.", -) -@click.option( - "--http2", - "http2", - type=bool, - is_flag=True, - default=False, - help="Send the request using HTTP/2, if the remote server supports it.", -) -@click.option( - "--download", - type=click.File("wb"), - help="Save the response content as a file, rather than displaying it.", -) -@click.option( - "--verbose", - "-v", - type=bool, - is_flag=True, - default=False, - help="Verbose. Show request as well as response.", -) -@click.option( - "--help", - is_flag=True, - is_eager=True, - expose_value=False, - callback=handle_help, - help="Show this message and exit.", -) -def main( - url: str, - method: str, - params: list[tuple[str, str]], - content: str, - data: list[tuple[str, str]], - files: list[tuple[str, click.File]], - json: str, - headers: list[tuple[str, str]], - cookies: list[tuple[str, str]], - auth: tuple[str, str] | None, - proxy: str, - timeout: float, - follow_redirects: bool, - verify: bool, - http2: bool, - download: typing.BinaryIO | None, - verbose: bool, -) -> None: - """ - An HTTP command line client. - Sends a request and displays the response. - """ - if not method: - method = "POST" if content or data or files or json else "GET" - - try: - with Client(proxy=proxy, timeout=timeout, http2=http2, verify=verify) as client: - with client.stream( - method, - url, - params=list(params), - content=content, - data=dict(data), - files=files, # type: ignore - json=json, - headers=headers, - cookies=dict(cookies), - auth=auth, - follow_redirects=follow_redirects, - extensions={"trace": functools.partial(trace, verbose=verbose)}, - ) as response: - if download is not None: - download_response(response, download) - else: - response.read() - if response.content: - print_response(response) - - except RequestError as exc: - console = rich.console.Console() - console.print(f"[red]{type(exc).__name__}[/red]: {exc}") - sys.exit(1) - - sys.exit(0 if response.is_success else 1) diff --git a/httpx/_models.py b/httpx/_models.py deleted file mode 100644 index 2cc86321a4..0000000000 --- a/httpx/_models.py +++ /dev/null @@ -1,1277 +0,0 @@ -from __future__ import annotations - -import codecs -import datetime -import email.message -import json as jsonlib -import re -import typing -import urllib.request -from collections.abc import Mapping -from http.cookiejar import Cookie, CookieJar - -from ._content import ByteStream, UnattachedStream, encode_request, encode_response -from ._decoders import ( - SUPPORTED_DECODERS, - ByteChunker, - ContentDecoder, - IdentityDecoder, - LineDecoder, - MultiDecoder, - TextChunker, - TextDecoder, -) -from ._exceptions import ( - CookieConflict, - HTTPStatusError, - RequestNotRead, - ResponseNotRead, - StreamClosed, - StreamConsumed, - request_context, -) -from ._multipart import get_multipart_boundary_from_content_type -from ._status_codes import codes -from ._types import ( - AsyncByteStream, - CookieTypes, - HeaderTypes, - QueryParamTypes, - RequestContent, - RequestData, - RequestExtensions, - RequestFiles, - ResponseContent, - ResponseExtensions, - SyncByteStream, -) -from ._urls import URL -from ._utils import to_bytes_or_str, to_str - -__all__ = ["Cookies", "Headers", "Request", "Response"] - -SENSITIVE_HEADERS = {"authorization", "proxy-authorization"} - - -def _is_known_encoding(encoding: str) -> bool: - """ - Return `True` if `encoding` is a known codec. - """ - try: - codecs.lookup(encoding) - except LookupError: - return False - return True - - -def _normalize_header_key(key: str | bytes, encoding: str | None = None) -> bytes: - """ - Coerce str/bytes into a strictly byte-wise HTTP header key. - """ - return key if isinstance(key, bytes) else key.encode(encoding or "ascii") - - -def _normalize_header_value(value: str | bytes, encoding: str | None = None) -> bytes: - """ - Coerce str/bytes into a strictly byte-wise HTTP header value. - """ - if isinstance(value, bytes): - return value - if not isinstance(value, str): - raise TypeError(f"Header value must be str or bytes, not {type(value)}") - return value.encode(encoding or "ascii") - - -def _parse_content_type_charset(content_type: str) -> str | None: - # We used to use `cgi.parse_header()` here, but `cgi` became a dead battery. - # See: https://peps.python.org/pep-0594/#cgi - msg = email.message.Message() - msg["content-type"] = content_type - return msg.get_content_charset(failobj=None) - - -def _parse_header_links(value: str) -> list[dict[str, str]]: - """ - Returns a list of parsed link headers, for more info see: - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Link - The generic syntax of those is: - Link: < uri-reference >; param1=value1; param2="value2" - So for instance: - Link; '; type="image/jpeg",;' - would return - [ - {"url": "http:/.../front.jpeg", "type": "image/jpeg"}, - {"url": "http://.../back.jpeg"}, - ] - :param value: HTTP Link entity-header field - :return: list of parsed link headers - """ - links: list[dict[str, str]] = [] - replace_chars = " '\"" - value = value.strip(replace_chars) - if not value: - return links - for val in re.split(", *<", value): - try: - url, params = val.split(";", 1) - except ValueError: - url, params = val, "" - link = {"url": url.strip("<> '\"")} - for param in params.split(";"): - try: - key, value = param.split("=") - except ValueError: - break - link[key.strip(replace_chars)] = value.strip(replace_chars) - links.append(link) - return links - - -def _obfuscate_sensitive_headers( - items: typing.Iterable[tuple[typing.AnyStr, typing.AnyStr]], -) -> typing.Iterator[tuple[typing.AnyStr, typing.AnyStr]]: - for k, v in items: - if to_str(k.lower()) in SENSITIVE_HEADERS: - v = to_bytes_or_str("[secure]", match_type_of=v) - yield k, v - - -class Headers(typing.MutableMapping[str, str]): - """ - HTTP headers, as a case-insensitive multi-dict. - """ - - def __init__( - self, - headers: HeaderTypes | None = None, - encoding: str | None = None, - ) -> None: - self._list = [] # type: typing.List[typing.Tuple[bytes, bytes, bytes]] - - if isinstance(headers, Headers): - self._list = list(headers._list) - elif isinstance(headers, Mapping): - for k, v in headers.items(): - bytes_key = _normalize_header_key(k, encoding) - bytes_value = _normalize_header_value(v, encoding) - self._list.append((bytes_key, bytes_key.lower(), bytes_value)) - elif headers is not None: - for k, v in headers: - bytes_key = _normalize_header_key(k, encoding) - bytes_value = _normalize_header_value(v, encoding) - self._list.append((bytes_key, bytes_key.lower(), bytes_value)) - - self._encoding = encoding - - @property - def encoding(self) -> str: - """ - Header encoding is mandated as ascii, but we allow fallbacks to utf-8 - or iso-8859-1. - """ - if self._encoding is None: - for encoding in ["ascii", "utf-8"]: - for key, value in self.raw: - try: - key.decode(encoding) - value.decode(encoding) - except UnicodeDecodeError: - break - else: - # The else block runs if 'break' did not occur, meaning - # all values fitted the encoding. - self._encoding = encoding - break - else: - # The ISO-8859-1 encoding covers all 256 code points in a byte, - # so will never raise decode errors. - self._encoding = "iso-8859-1" - return self._encoding - - @encoding.setter - def encoding(self, value: str) -> None: - self._encoding = value - - @property - def raw(self) -> list[tuple[bytes, bytes]]: - """ - Returns a list of the raw header items, as byte pairs. - """ - return [(raw_key, value) for raw_key, _, value in self._list] - - def keys(self) -> typing.KeysView[str]: - return {key.decode(self.encoding): None for _, key, value in self._list}.keys() - - def values(self) -> typing.ValuesView[str]: - values_dict: dict[str, str] = {} - for _, key, value in self._list: - str_key = key.decode(self.encoding) - str_value = value.decode(self.encoding) - if str_key in values_dict: - values_dict[str_key] += f", {str_value}" - else: - values_dict[str_key] = str_value - return values_dict.values() - - def items(self) -> typing.ItemsView[str, str]: - """ - Return `(key, value)` items of headers. Concatenate headers - into a single comma separated value when a key occurs multiple times. - """ - values_dict: dict[str, str] = {} - for _, key, value in self._list: - str_key = key.decode(self.encoding) - str_value = value.decode(self.encoding) - if str_key in values_dict: - values_dict[str_key] += f", {str_value}" - else: - values_dict[str_key] = str_value - return values_dict.items() - - def multi_items(self) -> list[tuple[str, str]]: - """ - Return a list of `(key, value)` pairs of headers. Allow multiple - occurrences of the same key without concatenating into a single - comma separated value. - """ - return [ - (key.decode(self.encoding), value.decode(self.encoding)) - for _, key, value in self._list - ] - - def get(self, key: str, default: typing.Any = None) -> typing.Any: - """ - Return a header value. If multiple occurrences of the header occur - then concatenate them together with commas. - """ - try: - return self[key] - except KeyError: - return default - - def get_list(self, key: str, split_commas: bool = False) -> list[str]: - """ - Return a list of all header values for a given key. - If `split_commas=True` is passed, then any comma separated header - values are split into multiple return strings. - """ - get_header_key = key.lower().encode(self.encoding) - - values = [ - item_value.decode(self.encoding) - for _, item_key, item_value in self._list - if item_key.lower() == get_header_key - ] - - if not split_commas: - return values - - split_values = [] - for value in values: - split_values.extend([item.strip() for item in value.split(",")]) - return split_values - - def update(self, headers: HeaderTypes | None = None) -> None: # type: ignore - headers = Headers(headers) - for key in headers.keys(): - if key in self: - self.pop(key) - self._list.extend(headers._list) - - def copy(self) -> Headers: - return Headers(self, encoding=self.encoding) - - def __getitem__(self, key: str) -> str: - """ - Return a single header value. - - If there are multiple headers with the same key, then we concatenate - them with commas. See: https://tools.ietf.org/html/rfc7230#section-3.2.2 - """ - normalized_key = key.lower().encode(self.encoding) - - items = [ - header_value.decode(self.encoding) - for _, header_key, header_value in self._list - if header_key == normalized_key - ] - - if items: - return ", ".join(items) - - raise KeyError(key) - - def __setitem__(self, key: str, value: str) -> None: - """ - Set the header `key` to `value`, removing any duplicate entries. - Retains insertion order. - """ - set_key = key.encode(self._encoding or "utf-8") - set_value = value.encode(self._encoding or "utf-8") - lookup_key = set_key.lower() - - found_indexes = [ - idx - for idx, (_, item_key, _) in enumerate(self._list) - if item_key == lookup_key - ] - - for idx in reversed(found_indexes[1:]): - del self._list[idx] - - if found_indexes: - idx = found_indexes[0] - self._list[idx] = (set_key, lookup_key, set_value) - else: - self._list.append((set_key, lookup_key, set_value)) - - def __delitem__(self, key: str) -> None: - """ - Remove the header `key`. - """ - del_key = key.lower().encode(self.encoding) - - pop_indexes = [ - idx - for idx, (_, item_key, _) in enumerate(self._list) - if item_key.lower() == del_key - ] - - if not pop_indexes: - raise KeyError(key) - - for idx in reversed(pop_indexes): - del self._list[idx] - - def __contains__(self, key: typing.Any) -> bool: - header_key = key.lower().encode(self.encoding) - return header_key in [key for _, key, _ in self._list] - - def __iter__(self) -> typing.Iterator[typing.Any]: - return iter(self.keys()) - - def __len__(self) -> int: - return len(self._list) - - def __eq__(self, other: typing.Any) -> bool: - try: - other_headers = Headers(other) - except ValueError: - return False - - self_list = [(key, value) for _, key, value in self._list] - other_list = [(key, value) for _, key, value in other_headers._list] - return sorted(self_list) == sorted(other_list) - - def __repr__(self) -> str: - class_name = self.__class__.__name__ - - encoding_str = "" - if self.encoding != "ascii": - encoding_str = f", encoding={self.encoding!r}" - - as_list = list(_obfuscate_sensitive_headers(self.multi_items())) - as_dict = dict(as_list) - - no_duplicate_keys = len(as_dict) == len(as_list) - if no_duplicate_keys: - return f"{class_name}({as_dict!r}{encoding_str})" - return f"{class_name}({as_list!r}{encoding_str})" - - -class Request: - def __init__( - self, - method: str, - url: URL | str, - *, - params: QueryParamTypes | None = None, - headers: HeaderTypes | None = None, - cookies: CookieTypes | None = None, - content: RequestContent | None = None, - data: RequestData | None = None, - files: RequestFiles | None = None, - json: typing.Any | None = None, - stream: SyncByteStream | AsyncByteStream | None = None, - extensions: RequestExtensions | None = None, - ) -> None: - self.method = method.upper() - self.url = URL(url) if params is None else URL(url, params=params) - self.headers = Headers(headers) - self.extensions = {} if extensions is None else dict(extensions) - - if cookies: - Cookies(cookies).set_cookie_header(self) - - if stream is None: - content_type: str | None = self.headers.get("content-type") - headers, stream = encode_request( - content=content, - data=data, - files=files, - json=json, - boundary=get_multipart_boundary_from_content_type( - content_type=content_type.encode(self.headers.encoding) - if content_type - else None - ), - ) - self._prepare(headers) - self.stream = stream - # Load the request body, except for streaming content. - if isinstance(stream, ByteStream): - self.read() - else: - # There's an important distinction between `Request(content=...)`, - # and `Request(stream=...)`. - # - # Using `content=...` implies automatically populated `Host` and content - # headers, of either `Content-Length: ...` or `Transfer-Encoding: chunked`. - # - # Using `stream=...` will not automatically include *any* - # auto-populated headers. - # - # As an end-user you don't really need `stream=...`. It's only - # useful when: - # - # * Preserving the request stream when copying requests, eg for redirects. - # * Creating request instances on the *server-side* of the transport API. - self.stream = stream - - def _prepare(self, default_headers: dict[str, str]) -> None: - for key, value in default_headers.items(): - # Ignore Transfer-Encoding if the Content-Length has been set explicitly. - if key.lower() == "transfer-encoding" and "Content-Length" in self.headers: - continue - self.headers.setdefault(key, value) - - auto_headers: list[tuple[bytes, bytes]] = [] - - has_host = "Host" in self.headers - has_content_length = ( - "Content-Length" in self.headers or "Transfer-Encoding" in self.headers - ) - - if not has_host and self.url.host: - auto_headers.append((b"Host", self.url.netloc)) - if not has_content_length and self.method in ("POST", "PUT", "PATCH"): - auto_headers.append((b"Content-Length", b"0")) - - self.headers = Headers(auto_headers + self.headers.raw) - - @property - def content(self) -> bytes: - if not hasattr(self, "_content"): - raise RequestNotRead() - return self._content - - def read(self) -> bytes: - """ - Read and return the request content. - """ - if not hasattr(self, "_content"): - assert isinstance(self.stream, typing.Iterable) - self._content = b"".join(self.stream) - if not isinstance(self.stream, ByteStream): - # If a streaming request has been read entirely into memory, then - # we can replace the stream with a raw bytes implementation, - # to ensure that any non-replayable streams can still be used. - self.stream = ByteStream(self._content) - return self._content - - async def aread(self) -> bytes: - """ - Read and return the request content. - """ - if not hasattr(self, "_content"): - assert isinstance(self.stream, typing.AsyncIterable) - self._content = b"".join([part async for part in self.stream]) - if not isinstance(self.stream, ByteStream): - # If a streaming request has been read entirely into memory, then - # we can replace the stream with a raw bytes implementation, - # to ensure that any non-replayable streams can still be used. - self.stream = ByteStream(self._content) - return self._content - - def __repr__(self) -> str: - class_name = self.__class__.__name__ - url = str(self.url) - return f"<{class_name}({self.method!r}, {url!r})>" - - def __getstate__(self) -> dict[str, typing.Any]: - return { - name: value - for name, value in self.__dict__.items() - if name not in ["extensions", "stream"] - } - - def __setstate__(self, state: dict[str, typing.Any]) -> None: - for name, value in state.items(): - setattr(self, name, value) - self.extensions = {} - self.stream = UnattachedStream() - - -class Response: - def __init__( - self, - status_code: int, - *, - headers: HeaderTypes | None = None, - content: ResponseContent | None = None, - text: str | None = None, - html: str | None = None, - json: typing.Any = None, - stream: SyncByteStream | AsyncByteStream | None = None, - request: Request | None = None, - extensions: ResponseExtensions | None = None, - history: list[Response] | None = None, - default_encoding: str | typing.Callable[[bytes], str] = "utf-8", - ) -> None: - self.status_code = status_code - self.headers = Headers(headers) - - self._request: Request | None = request - - # When follow_redirects=False and a redirect is received, - # the client will set `response.next_request`. - self.next_request: Request | None = None - - self.extensions = {} if extensions is None else dict(extensions) - self.history = [] if history is None else list(history) - - self.is_closed = False - self.is_stream_consumed = False - - self.default_encoding = default_encoding - - if stream is None: - headers, stream = encode_response(content, text, html, json) - self._prepare(headers) - self.stream = stream - if isinstance(stream, ByteStream): - # Load the response body, except for streaming content. - self.read() - else: - # There's an important distinction between `Response(content=...)`, - # and `Response(stream=...)`. - # - # Using `content=...` implies automatically populated content headers, - # of either `Content-Length: ...` or `Transfer-Encoding: chunked`. - # - # Using `stream=...` will not automatically include any content headers. - # - # As an end-user you don't really need `stream=...`. It's only - # useful when creating response instances having received a stream - # from the transport API. - self.stream = stream - - self._num_bytes_downloaded = 0 - - def _prepare(self, default_headers: dict[str, str]) -> None: - for key, value in default_headers.items(): - # Ignore Transfer-Encoding if the Content-Length has been set explicitly. - if key.lower() == "transfer-encoding" and "content-length" in self.headers: - continue - self.headers.setdefault(key, value) - - @property - def elapsed(self) -> datetime.timedelta: - """ - Returns the time taken for the complete request/response - cycle to complete. - """ - if not hasattr(self, "_elapsed"): - raise RuntimeError( - "'.elapsed' may only be accessed after the response " - "has been read or closed." - ) - return self._elapsed - - @elapsed.setter - def elapsed(self, elapsed: datetime.timedelta) -> None: - self._elapsed = elapsed - - @property - def request(self) -> Request: - """ - Returns the request instance associated to the current response. - """ - if self._request is None: - raise RuntimeError( - "The request instance has not been set on this response." - ) - return self._request - - @request.setter - def request(self, value: Request) -> None: - self._request = value - - @property - def http_version(self) -> str: - try: - http_version: bytes = self.extensions["http_version"] - except KeyError: - return "HTTP/1.1" - else: - return http_version.decode("ascii", errors="ignore") - - @property - def reason_phrase(self) -> str: - try: - reason_phrase: bytes = self.extensions["reason_phrase"] - except KeyError: - return codes.get_reason_phrase(self.status_code) - else: - return reason_phrase.decode("ascii", errors="ignore") - - @property - def url(self) -> URL: - """ - Returns the URL for which the request was made. - """ - return self.request.url - - @property - def content(self) -> bytes: - if not hasattr(self, "_content"): - raise ResponseNotRead() - return self._content - - @property - def text(self) -> str: - if not hasattr(self, "_text"): - content = self.content - if not content: - self._text = "" - else: - decoder = TextDecoder(encoding=self.encoding or "utf-8") - self._text = "".join([decoder.decode(self.content), decoder.flush()]) - return self._text - - @property - def encoding(self) -> str | None: - """ - Return an encoding to use for decoding the byte content into text. - The priority for determining this is given by... - - * `.encoding = <>` has been set explicitly. - * The encoding as specified by the charset parameter in the Content-Type header. - * The encoding as determined by `default_encoding`, which may either be - a string like "utf-8" indicating the encoding to use, or may be a callable - which enables charset autodetection. - """ - if not hasattr(self, "_encoding"): - encoding = self.charset_encoding - if encoding is None or not _is_known_encoding(encoding): - if isinstance(self.default_encoding, str): - encoding = self.default_encoding - elif hasattr(self, "_content"): - encoding = self.default_encoding(self._content) - self._encoding = encoding or "utf-8" - return self._encoding - - @encoding.setter - def encoding(self, value: str) -> None: - """ - Set the encoding to use for decoding the byte content into text. - - If the `text` attribute has been accessed, attempting to set the - encoding will throw a ValueError. - """ - if hasattr(self, "_text"): - raise ValueError( - "Setting encoding after `text` has been accessed is not allowed." - ) - self._encoding = value - - @property - def charset_encoding(self) -> str | None: - """ - Return the encoding, as specified by the Content-Type header. - """ - content_type = self.headers.get("Content-Type") - if content_type is None: - return None - - return _parse_content_type_charset(content_type) - - def _get_content_decoder(self) -> ContentDecoder: - """ - Returns a decoder instance which can be used to decode the raw byte - content, depending on the Content-Encoding used in the response. - """ - if not hasattr(self, "_decoder"): - decoders: list[ContentDecoder] = [] - values = self.headers.get_list("content-encoding", split_commas=True) - for value in values: - value = value.strip().lower() - try: - decoder_cls = SUPPORTED_DECODERS[value] - decoders.append(decoder_cls()) - except KeyError: - continue - - if len(decoders) == 1: - self._decoder = decoders[0] - elif len(decoders) > 1: - self._decoder = MultiDecoder(children=decoders) - else: - self._decoder = IdentityDecoder() - - return self._decoder - - @property - def is_informational(self) -> bool: - """ - A property which is `True` for 1xx status codes, `False` otherwise. - """ - return codes.is_informational(self.status_code) - - @property - def is_success(self) -> bool: - """ - A property which is `True` for 2xx status codes, `False` otherwise. - """ - return codes.is_success(self.status_code) - - @property - def is_redirect(self) -> bool: - """ - A property which is `True` for 3xx status codes, `False` otherwise. - - Note that not all responses with a 3xx status code indicate a URL redirect. - - Use `response.has_redirect_location` to determine responses with a properly - formed URL redirection. - """ - return codes.is_redirect(self.status_code) - - @property - def is_client_error(self) -> bool: - """ - A property which is `True` for 4xx status codes, `False` otherwise. - """ - return codes.is_client_error(self.status_code) - - @property - def is_server_error(self) -> bool: - """ - A property which is `True` for 5xx status codes, `False` otherwise. - """ - return codes.is_server_error(self.status_code) - - @property - def is_error(self) -> bool: - """ - A property which is `True` for 4xx and 5xx status codes, `False` otherwise. - """ - return codes.is_error(self.status_code) - - @property - def has_redirect_location(self) -> bool: - """ - Returns True for 3xx responses with a properly formed URL redirection, - `False` otherwise. - """ - return ( - self.status_code - in ( - # 301 (Cacheable redirect. Method may change to GET.) - codes.MOVED_PERMANENTLY, - # 302 (Uncacheable redirect. Method may change to GET.) - codes.FOUND, - # 303 (Client should make a GET or HEAD request.) - codes.SEE_OTHER, - # 307 (Equiv. 302, but retain method) - codes.TEMPORARY_REDIRECT, - # 308 (Equiv. 301, but retain method) - codes.PERMANENT_REDIRECT, - ) - and "Location" in self.headers - ) - - def raise_for_status(self) -> Response: - """ - Raise the `HTTPStatusError` if one occurred. - """ - request = self._request - if request is None: - raise RuntimeError( - "Cannot call `raise_for_status` as the request " - "instance has not been set on this response." - ) - - if self.is_success: - return self - - if self.has_redirect_location: - message = ( - "{error_type} '{0.status_code} {0.reason_phrase}' for url '{0.url}'\n" - "Redirect location: '{0.headers[location]}'\n" - "For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/{0.status_code}" - ) - else: - message = ( - "{error_type} '{0.status_code} {0.reason_phrase}' for url '{0.url}'\n" - "For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/{0.status_code}" - ) - - status_class = self.status_code // 100 - error_types = { - 1: "Informational response", - 3: "Redirect response", - 4: "Client error", - 5: "Server error", - } - error_type = error_types.get(status_class, "Invalid status code") - message = message.format(self, error_type=error_type) - raise HTTPStatusError(message, request=request, response=self) - - def json(self, **kwargs: typing.Any) -> typing.Any: - return jsonlib.loads(self.content, **kwargs) - - @property - def cookies(self) -> Cookies: - if not hasattr(self, "_cookies"): - self._cookies = Cookies() - self._cookies.extract_cookies(self) - return self._cookies - - @property - def links(self) -> dict[str | None, dict[str, str]]: - """ - Returns the parsed header links of the response, if any - """ - header = self.headers.get("link") - if header is None: - return {} - - return { - (link.get("rel") or link.get("url")): link - for link in _parse_header_links(header) - } - - @property - def num_bytes_downloaded(self) -> int: - return self._num_bytes_downloaded - - def __repr__(self) -> str: - return f"" - - def __getstate__(self) -> dict[str, typing.Any]: - return { - name: value - for name, value in self.__dict__.items() - if name not in ["extensions", "stream", "is_closed", "_decoder"] - } - - def __setstate__(self, state: dict[str, typing.Any]) -> None: - for name, value in state.items(): - setattr(self, name, value) - self.is_closed = True - self.extensions = {} - self.stream = UnattachedStream() - - def read(self) -> bytes: - """ - Read and return the response content. - """ - if not hasattr(self, "_content"): - self._content = b"".join(self.iter_bytes()) - return self._content - - def iter_bytes(self, chunk_size: int | None = None) -> typing.Iterator[bytes]: - """ - A byte-iterator over the decoded response content. - This allows us to handle gzip, deflate, brotli, and zstd encoded responses. - """ - if hasattr(self, "_content"): - chunk_size = len(self._content) if chunk_size is None else chunk_size - for i in range(0, len(self._content), max(chunk_size, 1)): - yield self._content[i : i + chunk_size] - else: - decoder = self._get_content_decoder() - chunker = ByteChunker(chunk_size=chunk_size) - with request_context(request=self._request): - for raw_bytes in self.iter_raw(): - decoded = decoder.decode(raw_bytes) - for chunk in chunker.decode(decoded): - yield chunk - decoded = decoder.flush() - for chunk in chunker.decode(decoded): - yield chunk # pragma: no cover - for chunk in chunker.flush(): - yield chunk - - def iter_text(self, chunk_size: int | None = None) -> typing.Iterator[str]: - """ - A str-iterator over the decoded response content - that handles both gzip, deflate, etc but also detects the content's - string encoding. - """ - decoder = TextDecoder(encoding=self.encoding or "utf-8") - chunker = TextChunker(chunk_size=chunk_size) - with request_context(request=self._request): - for byte_content in self.iter_bytes(): - text_content = decoder.decode(byte_content) - for chunk in chunker.decode(text_content): - yield chunk - text_content = decoder.flush() - for chunk in chunker.decode(text_content): - yield chunk # pragma: no cover - for chunk in chunker.flush(): - yield chunk - - def iter_lines(self) -> typing.Iterator[str]: - decoder = LineDecoder() - with request_context(request=self._request): - for text in self.iter_text(): - for line in decoder.decode(text): - yield line - for line in decoder.flush(): - yield line - - def iter_raw(self, chunk_size: int | None = None) -> typing.Iterator[bytes]: - """ - A byte-iterator over the raw response content. - """ - if self.is_stream_consumed: - raise StreamConsumed() - if self.is_closed: - raise StreamClosed() - if not isinstance(self.stream, SyncByteStream): - raise RuntimeError("Attempted to call a sync iterator on an async stream.") - - self.is_stream_consumed = True - self._num_bytes_downloaded = 0 - chunker = ByteChunker(chunk_size=chunk_size) - - with request_context(request=self._request): - for raw_stream_bytes in self.stream: - self._num_bytes_downloaded += len(raw_stream_bytes) - for chunk in chunker.decode(raw_stream_bytes): - yield chunk - - for chunk in chunker.flush(): - yield chunk - - self.close() - - def close(self) -> None: - """ - Close the response and release the connection. - Automatically called if the response body is read to completion. - """ - if not isinstance(self.stream, SyncByteStream): - raise RuntimeError("Attempted to call a sync close on an async stream.") - - if not self.is_closed: - self.is_closed = True - with request_context(request=self._request): - self.stream.close() - - async def aread(self) -> bytes: - """ - Read and return the response content. - """ - if not hasattr(self, "_content"): - self._content = b"".join([part async for part in self.aiter_bytes()]) - return self._content - - async def aiter_bytes( - self, chunk_size: int | None = None - ) -> typing.AsyncIterator[bytes]: - """ - A byte-iterator over the decoded response content. - This allows us to handle gzip, deflate, brotli, and zstd encoded responses. - """ - if hasattr(self, "_content"): - chunk_size = len(self._content) if chunk_size is None else chunk_size - for i in range(0, len(self._content), max(chunk_size, 1)): - yield self._content[i : i + chunk_size] - else: - decoder = self._get_content_decoder() - chunker = ByteChunker(chunk_size=chunk_size) - with request_context(request=self._request): - async for raw_bytes in self.aiter_raw(): - decoded = decoder.decode(raw_bytes) - for chunk in chunker.decode(decoded): - yield chunk - decoded = decoder.flush() - for chunk in chunker.decode(decoded): - yield chunk # pragma: no cover - for chunk in chunker.flush(): - yield chunk - - async def aiter_text( - self, chunk_size: int | None = None - ) -> typing.AsyncIterator[str]: - """ - A str-iterator over the decoded response content - that handles both gzip, deflate, etc but also detects the content's - string encoding. - """ - decoder = TextDecoder(encoding=self.encoding or "utf-8") - chunker = TextChunker(chunk_size=chunk_size) - with request_context(request=self._request): - async for byte_content in self.aiter_bytes(): - text_content = decoder.decode(byte_content) - for chunk in chunker.decode(text_content): - yield chunk - text_content = decoder.flush() - for chunk in chunker.decode(text_content): - yield chunk # pragma: no cover - for chunk in chunker.flush(): - yield chunk - - async def aiter_lines(self) -> typing.AsyncIterator[str]: - decoder = LineDecoder() - with request_context(request=self._request): - async for text in self.aiter_text(): - for line in decoder.decode(text): - yield line - for line in decoder.flush(): - yield line - - async def aiter_raw( - self, chunk_size: int | None = None - ) -> typing.AsyncIterator[bytes]: - """ - A byte-iterator over the raw response content. - """ - if self.is_stream_consumed: - raise StreamConsumed() - if self.is_closed: - raise StreamClosed() - if not isinstance(self.stream, AsyncByteStream): - raise RuntimeError("Attempted to call an async iterator on a sync stream.") - - self.is_stream_consumed = True - self._num_bytes_downloaded = 0 - chunker = ByteChunker(chunk_size=chunk_size) - - with request_context(request=self._request): - async for raw_stream_bytes in self.stream: - self._num_bytes_downloaded += len(raw_stream_bytes) - for chunk in chunker.decode(raw_stream_bytes): - yield chunk - - for chunk in chunker.flush(): - yield chunk - - await self.aclose() - - async def aclose(self) -> None: - """ - Close the response and release the connection. - Automatically called if the response body is read to completion. - """ - if not isinstance(self.stream, AsyncByteStream): - raise RuntimeError("Attempted to call an async close on a sync stream.") - - if not self.is_closed: - self.is_closed = True - with request_context(request=self._request): - await self.stream.aclose() - - -class Cookies(typing.MutableMapping[str, str]): - """ - HTTP Cookies, as a mutable mapping. - """ - - def __init__(self, cookies: CookieTypes | None = None) -> None: - if cookies is None or isinstance(cookies, dict): - self.jar = CookieJar() - if isinstance(cookies, dict): - for key, value in cookies.items(): - self.set(key, value) - elif isinstance(cookies, list): - self.jar = CookieJar() - for key, value in cookies: - self.set(key, value) - elif isinstance(cookies, Cookies): - self.jar = CookieJar() - for cookie in cookies.jar: - self.jar.set_cookie(cookie) - else: - self.jar = cookies - - def extract_cookies(self, response: Response) -> None: - """ - Loads any cookies based on the response `Set-Cookie` headers. - """ - urllib_response = self._CookieCompatResponse(response) - urllib_request = self._CookieCompatRequest(response.request) - - self.jar.extract_cookies(urllib_response, urllib_request) # type: ignore - - def set_cookie_header(self, request: Request) -> None: - """ - Sets an appropriate 'Cookie:' HTTP header on the `Request`. - """ - urllib_request = self._CookieCompatRequest(request) - self.jar.add_cookie_header(urllib_request) - - def set(self, name: str, value: str, domain: str = "", path: str = "/") -> None: - """ - Set a cookie value by name. May optionally include domain and path. - """ - kwargs = { - "version": 0, - "name": name, - "value": value, - "port": None, - "port_specified": False, - "domain": domain, - "domain_specified": bool(domain), - "domain_initial_dot": domain.startswith("."), - "path": path, - "path_specified": bool(path), - "secure": False, - "expires": None, - "discard": True, - "comment": None, - "comment_url": None, - "rest": {"HttpOnly": None}, - "rfc2109": False, - } - cookie = Cookie(**kwargs) # type: ignore - self.jar.set_cookie(cookie) - - def get( # type: ignore - self, - name: str, - default: str | None = None, - domain: str | None = None, - path: str | None = None, - ) -> str | None: - """ - Get a cookie by name. May optionally include domain and path - in order to specify exactly which cookie to retrieve. - """ - value = None - for cookie in self.jar: - if cookie.name == name: - if domain is None or cookie.domain == domain: - if path is None or cookie.path == path: - if value is not None: - message = f"Multiple cookies exist with name={name}" - raise CookieConflict(message) - value = cookie.value - - if value is None: - return default - return value - - def delete( - self, - name: str, - domain: str | None = None, - path: str | None = None, - ) -> None: - """ - Delete a cookie by name. May optionally include domain and path - in order to specify exactly which cookie to delete. - """ - if domain is not None and path is not None: - return self.jar.clear(domain, path, name) - - remove = [ - cookie - for cookie in self.jar - if cookie.name == name - and (domain is None or cookie.domain == domain) - and (path is None or cookie.path == path) - ] - - for cookie in remove: - self.jar.clear(cookie.domain, cookie.path, cookie.name) - - def clear(self, domain: str | None = None, path: str | None = None) -> None: - """ - Delete all cookies. Optionally include a domain and path in - order to only delete a subset of all the cookies. - """ - args = [] - if domain is not None: - args.append(domain) - if path is not None: - assert domain is not None - args.append(path) - self.jar.clear(*args) - - def update(self, cookies: CookieTypes | None = None) -> None: # type: ignore - cookies = Cookies(cookies) - for cookie in cookies.jar: - self.jar.set_cookie(cookie) - - def __setitem__(self, name: str, value: str) -> None: - return self.set(name, value) - - def __getitem__(self, name: str) -> str: - value = self.get(name) - if value is None: - raise KeyError(name) - return value - - def __delitem__(self, name: str) -> None: - return self.delete(name) - - def __len__(self) -> int: - return len(self.jar) - - def __iter__(self) -> typing.Iterator[str]: - return (cookie.name for cookie in self.jar) - - def __bool__(self) -> bool: - for _ in self.jar: - return True - return False - - def __repr__(self) -> str: - cookies_repr = ", ".join( - [ - f"" - for cookie in self.jar - ] - ) - - return f"" - - class _CookieCompatRequest(urllib.request.Request): - """ - Wraps a `Request` instance up in a compatibility interface suitable - for use with `CookieJar` operations. - """ - - def __init__(self, request: Request) -> None: - super().__init__( - url=str(request.url), - headers=dict(request.headers), - method=request.method, - ) - self.request = request - - def add_unredirected_header(self, key: str, value: str) -> None: - super().add_unredirected_header(key, value) - self.request.headers[key] = value - - class _CookieCompatResponse: - """ - Wraps a `Request` instance up in a compatibility interface suitable - for use with `CookieJar` operations. - """ - - def __init__(self, response: Response) -> None: - self.response = response - - def info(self) -> email.message.Message: - info = email.message.Message() - for key, value in self.response.headers.multi_items(): - # Note that setting `info[key]` here is an "append" operation, - # not a "replace" operation. - # https://docs.python.org/3/library/email.compat32-message.html#email.message.Message.__setitem__ - info[key] = value - return info diff --git a/httpx/_multipart.py b/httpx/_multipart.py deleted file mode 100644 index b4761af9b2..0000000000 --- a/httpx/_multipart.py +++ /dev/null @@ -1,300 +0,0 @@ -from __future__ import annotations - -import io -import mimetypes -import os -import re -import typing -from pathlib import Path - -from ._types import ( - AsyncByteStream, - FileContent, - FileTypes, - RequestData, - RequestFiles, - SyncByteStream, -) -from ._utils import ( - peek_filelike_length, - primitive_value_to_str, - to_bytes, -) - -_HTML5_FORM_ENCODING_REPLACEMENTS = {'"': "%22", "\\": "\\\\"} -_HTML5_FORM_ENCODING_REPLACEMENTS.update( - {chr(c): "%{:02X}".format(c) for c in range(0x1F + 1) if c != 0x1B} -) -_HTML5_FORM_ENCODING_RE = re.compile( - r"|".join([re.escape(c) for c in _HTML5_FORM_ENCODING_REPLACEMENTS.keys()]) -) - - -def _format_form_param(name: str, value: str) -> bytes: - """ - Encode a name/value pair within a multipart form. - """ - - def replacer(match: typing.Match[str]) -> str: - return _HTML5_FORM_ENCODING_REPLACEMENTS[match.group(0)] - - value = _HTML5_FORM_ENCODING_RE.sub(replacer, value) - return f'{name}="{value}"'.encode() - - -def _guess_content_type(filename: str | None) -> str | None: - """ - Guesses the mimetype based on a filename. Defaults to `application/octet-stream`. - - Returns `None` if `filename` is `None` or empty. - """ - if filename: - return mimetypes.guess_type(filename)[0] or "application/octet-stream" - return None - - -def get_multipart_boundary_from_content_type( - content_type: bytes | None, -) -> bytes | None: - if not content_type or not content_type.startswith(b"multipart/form-data"): - return None - # parse boundary according to - # https://www.rfc-editor.org/rfc/rfc2046#section-5.1.1 - if b";" in content_type: - for section in content_type.split(b";"): - if section.strip().lower().startswith(b"boundary="): - return section.strip()[len(b"boundary=") :].strip(b'"') - return None - - -class DataField: - """ - A single form field item, within a multipart form field. - """ - - def __init__(self, name: str, value: str | bytes | int | float | None) -> None: - if not isinstance(name, str): - raise TypeError( - f"Invalid type for name. Expected str, got {type(name)}: {name!r}" - ) - if value is not None and not isinstance(value, (str, bytes, int, float)): - raise TypeError( - "Invalid type for value. Expected primitive type," - f" got {type(value)}: {value!r}" - ) - self.name = name - self.value: str | bytes = ( - value if isinstance(value, bytes) else primitive_value_to_str(value) - ) - - def render_headers(self) -> bytes: - if not hasattr(self, "_headers"): - name = _format_form_param("name", self.name) - self._headers = b"".join( - [b"Content-Disposition: form-data; ", name, b"\r\n\r\n"] - ) - - return self._headers - - def render_data(self) -> bytes: - if not hasattr(self, "_data"): - self._data = to_bytes(self.value) - - return self._data - - def get_length(self) -> int: - headers = self.render_headers() - data = self.render_data() - return len(headers) + len(data) - - def render(self) -> typing.Iterator[bytes]: - yield self.render_headers() - yield self.render_data() - - -class FileField: - """ - A single file field item, within a multipart form field. - """ - - CHUNK_SIZE = 64 * 1024 - - def __init__(self, name: str, value: FileTypes) -> None: - self.name = name - - fileobj: FileContent - - headers: dict[str, str] = {} - content_type: str | None = None - - # This large tuple based API largely mirror's requests' API - # It would be good to think of better APIs for this that we could - # include in httpx 2.0 since variable length tuples(especially of 4 elements) - # are quite unwieldly - if isinstance(value, tuple): - if len(value) == 2: - # neither the 3rd parameter (content_type) nor the 4th (headers) - # was included - filename, fileobj = value - elif len(value) == 3: - filename, fileobj, content_type = value - else: - # all 4 parameters included - filename, fileobj, content_type, headers = value # type: ignore - else: - filename = Path(str(getattr(value, "name", "upload"))).name - fileobj = value - - if content_type is None: - content_type = _guess_content_type(filename) - - has_content_type_header = any("content-type" in key.lower() for key in headers) - if content_type is not None and not has_content_type_header: - # note that unlike requests, we ignore the content_type provided in the 3rd - # tuple element if it is also included in the headers requests does - # the opposite (it overwrites the headerwith the 3rd tuple element) - headers["Content-Type"] = content_type - - if isinstance(fileobj, io.StringIO): - raise TypeError( - "Multipart file uploads require 'io.BytesIO', not 'io.StringIO'." - ) - if isinstance(fileobj, io.TextIOBase): - raise TypeError( - "Multipart file uploads must be opened in binary mode, not text mode." - ) - - self.filename = filename - self.file = fileobj - self.headers = headers - - def get_length(self) -> int | None: - headers = self.render_headers() - - if isinstance(self.file, (str, bytes)): - return len(headers) + len(to_bytes(self.file)) - - file_length = peek_filelike_length(self.file) - - # If we can't determine the filesize without reading it into memory, - # then return `None` here, to indicate an unknown file length. - if file_length is None: - return None - - return len(headers) + file_length - - def render_headers(self) -> bytes: - if not hasattr(self, "_headers"): - parts = [ - b"Content-Disposition: form-data; ", - _format_form_param("name", self.name), - ] - if self.filename: - filename = _format_form_param("filename", self.filename) - parts.extend([b"; ", filename]) - for header_name, header_value in self.headers.items(): - key, val = f"\r\n{header_name}: ".encode(), header_value.encode() - parts.extend([key, val]) - parts.append(b"\r\n\r\n") - self._headers = b"".join(parts) - - return self._headers - - def render_data(self) -> typing.Iterator[bytes]: - if isinstance(self.file, (str, bytes)): - yield to_bytes(self.file) - return - - if hasattr(self.file, "seek"): - try: - self.file.seek(0) - except io.UnsupportedOperation: - pass - - chunk = self.file.read(self.CHUNK_SIZE) - while chunk: - yield to_bytes(chunk) - chunk = self.file.read(self.CHUNK_SIZE) - - def render(self) -> typing.Iterator[bytes]: - yield self.render_headers() - yield from self.render_data() - - -class MultipartStream(SyncByteStream, AsyncByteStream): - """ - Request content as streaming multipart encoded form data. - """ - - def __init__( - self, - data: RequestData, - files: RequestFiles, - boundary: bytes | None = None, - ) -> None: - if boundary is None: - boundary = os.urandom(16).hex().encode("ascii") - - self.boundary = boundary - self.content_type = "multipart/form-data; boundary=%s" % boundary.decode( - "ascii" - ) - self.fields = list(self._iter_fields(data, files)) - - def _iter_fields( - self, data: RequestData, files: RequestFiles - ) -> typing.Iterator[FileField | DataField]: - for name, value in data.items(): - if isinstance(value, (tuple, list)): - for item in value: - yield DataField(name=name, value=item) - else: - yield DataField(name=name, value=value) - - file_items = files.items() if isinstance(files, typing.Mapping) else files - for name, value in file_items: - yield FileField(name=name, value=value) - - def iter_chunks(self) -> typing.Iterator[bytes]: - for field in self.fields: - yield b"--%s\r\n" % self.boundary - yield from field.render() - yield b"\r\n" - yield b"--%s--\r\n" % self.boundary - - def get_content_length(self) -> int | None: - """ - Return the length of the multipart encoded content, or `None` if - any of the files have a length that cannot be determined upfront. - """ - boundary_length = len(self.boundary) - length = 0 - - for field in self.fields: - field_length = field.get_length() - if field_length is None: - return None - - length += 2 + boundary_length + 2 # b"--{boundary}\r\n" - length += field_length - length += 2 # b"\r\n" - - length += 2 + boundary_length + 4 # b"--{boundary}--\r\n" - return length - - # Content stream interface. - - def get_headers(self) -> dict[str, str]: - content_length = self.get_content_length() - content_type = self.content_type - if content_length is None: - return {"Transfer-Encoding": "chunked", "Content-Type": content_type} - return {"Content-Length": str(content_length), "Content-Type": content_type} - - def __iter__(self) -> typing.Iterator[bytes]: - for chunk in self.iter_chunks(): - yield chunk - - async def __aiter__(self) -> typing.AsyncIterator[bytes]: - for chunk in self.iter_chunks(): - yield chunk diff --git a/httpx/_status_codes.py b/httpx/_status_codes.py deleted file mode 100644 index 133a6231a5..0000000000 --- a/httpx/_status_codes.py +++ /dev/null @@ -1,162 +0,0 @@ -from __future__ import annotations - -from enum import IntEnum - -__all__ = ["codes"] - - -class codes(IntEnum): - """HTTP status codes and reason phrases - - Status codes from the following RFCs are all observed: - - * RFC 7231: Hypertext Transfer Protocol (HTTP/1.1), obsoletes 2616 - * RFC 6585: Additional HTTP Status Codes - * RFC 3229: Delta encoding in HTTP - * RFC 4918: HTTP Extensions for WebDAV, obsoletes 2518 - * RFC 5842: Binding Extensions to WebDAV - * RFC 7238: Permanent Redirect - * RFC 2295: Transparent Content Negotiation in HTTP - * RFC 2774: An HTTP Extension Framework - * RFC 7540: Hypertext Transfer Protocol Version 2 (HTTP/2) - * RFC 2324: Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0) - * RFC 7725: An HTTP Status Code to Report Legal Obstacles - * RFC 8297: An HTTP Status Code for Indicating Hints - * RFC 8470: Using Early Data in HTTP - """ - - def __new__(cls, value: int, phrase: str = "") -> codes: - obj = int.__new__(cls, value) - obj._value_ = value - - obj.phrase = phrase # type: ignore[attr-defined] - return obj - - def __str__(self) -> str: - return str(self.value) - - @classmethod - def get_reason_phrase(cls, value: int) -> str: - try: - return codes(value).phrase # type: ignore - except ValueError: - return "" - - @classmethod - def is_informational(cls, value: int) -> bool: - """ - Returns `True` for 1xx status codes, `False` otherwise. - """ - return 100 <= value <= 199 - - @classmethod - def is_success(cls, value: int) -> bool: - """ - Returns `True` for 2xx status codes, `False` otherwise. - """ - return 200 <= value <= 299 - - @classmethod - def is_redirect(cls, value: int) -> bool: - """ - Returns `True` for 3xx status codes, `False` otherwise. - """ - return 300 <= value <= 399 - - @classmethod - def is_client_error(cls, value: int) -> bool: - """ - Returns `True` for 4xx status codes, `False` otherwise. - """ - return 400 <= value <= 499 - - @classmethod - def is_server_error(cls, value: int) -> bool: - """ - Returns `True` for 5xx status codes, `False` otherwise. - """ - return 500 <= value <= 599 - - @classmethod - def is_error(cls, value: int) -> bool: - """ - Returns `True` for 4xx or 5xx status codes, `False` otherwise. - """ - return 400 <= value <= 599 - - # informational - CONTINUE = 100, "Continue" - SWITCHING_PROTOCOLS = 101, "Switching Protocols" - PROCESSING = 102, "Processing" - EARLY_HINTS = 103, "Early Hints" - - # success - OK = 200, "OK" - CREATED = 201, "Created" - ACCEPTED = 202, "Accepted" - NON_AUTHORITATIVE_INFORMATION = 203, "Non-Authoritative Information" - NO_CONTENT = 204, "No Content" - RESET_CONTENT = 205, "Reset Content" - PARTIAL_CONTENT = 206, "Partial Content" - MULTI_STATUS = 207, "Multi-Status" - ALREADY_REPORTED = 208, "Already Reported" - IM_USED = 226, "IM Used" - - # redirection - MULTIPLE_CHOICES = 300, "Multiple Choices" - MOVED_PERMANENTLY = 301, "Moved Permanently" - FOUND = 302, "Found" - SEE_OTHER = 303, "See Other" - NOT_MODIFIED = 304, "Not Modified" - USE_PROXY = 305, "Use Proxy" - TEMPORARY_REDIRECT = 307, "Temporary Redirect" - PERMANENT_REDIRECT = 308, "Permanent Redirect" - - # client error - BAD_REQUEST = 400, "Bad Request" - UNAUTHORIZED = 401, "Unauthorized" - PAYMENT_REQUIRED = 402, "Payment Required" - FORBIDDEN = 403, "Forbidden" - NOT_FOUND = 404, "Not Found" - METHOD_NOT_ALLOWED = 405, "Method Not Allowed" - NOT_ACCEPTABLE = 406, "Not Acceptable" - PROXY_AUTHENTICATION_REQUIRED = 407, "Proxy Authentication Required" - REQUEST_TIMEOUT = 408, "Request Timeout" - CONFLICT = 409, "Conflict" - GONE = 410, "Gone" - LENGTH_REQUIRED = 411, "Length Required" - PRECONDITION_FAILED = 412, "Precondition Failed" - REQUEST_ENTITY_TOO_LARGE = 413, "Request Entity Too Large" - REQUEST_URI_TOO_LONG = 414, "Request-URI Too Long" - UNSUPPORTED_MEDIA_TYPE = 415, "Unsupported Media Type" - REQUESTED_RANGE_NOT_SATISFIABLE = 416, "Requested Range Not Satisfiable" - EXPECTATION_FAILED = 417, "Expectation Failed" - IM_A_TEAPOT = 418, "I'm a teapot" - MISDIRECTED_REQUEST = 421, "Misdirected Request" - UNPROCESSABLE_ENTITY = 422, "Unprocessable Entity" - LOCKED = 423, "Locked" - FAILED_DEPENDENCY = 424, "Failed Dependency" - TOO_EARLY = 425, "Too Early" - UPGRADE_REQUIRED = 426, "Upgrade Required" - PRECONDITION_REQUIRED = 428, "Precondition Required" - TOO_MANY_REQUESTS = 429, "Too Many Requests" - REQUEST_HEADER_FIELDS_TOO_LARGE = 431, "Request Header Fields Too Large" - UNAVAILABLE_FOR_LEGAL_REASONS = 451, "Unavailable For Legal Reasons" - - # server errors - INTERNAL_SERVER_ERROR = 500, "Internal Server Error" - NOT_IMPLEMENTED = 501, "Not Implemented" - BAD_GATEWAY = 502, "Bad Gateway" - SERVICE_UNAVAILABLE = 503, "Service Unavailable" - GATEWAY_TIMEOUT = 504, "Gateway Timeout" - HTTP_VERSION_NOT_SUPPORTED = 505, "HTTP Version Not Supported" - VARIANT_ALSO_NEGOTIATES = 506, "Variant Also Negotiates" - INSUFFICIENT_STORAGE = 507, "Insufficient Storage" - LOOP_DETECTED = 508, "Loop Detected" - NOT_EXTENDED = 510, "Not Extended" - NETWORK_AUTHENTICATION_REQUIRED = 511, "Network Authentication Required" - - -# Include lower-case styles for `requests` compatibility. -for code in codes: - setattr(codes, code._name_.lower(), int(code)) diff --git a/httpx/_transports/__init__.py b/httpx/_transports/__init__.py deleted file mode 100644 index 7a321053b2..0000000000 --- a/httpx/_transports/__init__.py +++ /dev/null @@ -1,15 +0,0 @@ -from .asgi import * -from .base import * -from .default import * -from .mock import * -from .wsgi import * - -__all__ = [ - "ASGITransport", - "AsyncBaseTransport", - "BaseTransport", - "AsyncHTTPTransport", - "HTTPTransport", - "MockTransport", - "WSGITransport", -] diff --git a/httpx/_transports/asgi.py b/httpx/_transports/asgi.py deleted file mode 100644 index 2bc4efae0e..0000000000 --- a/httpx/_transports/asgi.py +++ /dev/null @@ -1,187 +0,0 @@ -from __future__ import annotations - -import typing - -from .._models import Request, Response -from .._types import AsyncByteStream -from .base import AsyncBaseTransport - -if typing.TYPE_CHECKING: # pragma: no cover - import asyncio - - import trio - - Event = typing.Union[asyncio.Event, trio.Event] - - -_Message = typing.MutableMapping[str, typing.Any] -_Receive = typing.Callable[[], typing.Awaitable[_Message]] -_Send = typing.Callable[ - [typing.MutableMapping[str, typing.Any]], typing.Awaitable[None] -] -_ASGIApp = typing.Callable[ - [typing.MutableMapping[str, typing.Any], _Receive, _Send], typing.Awaitable[None] -] - -__all__ = ["ASGITransport"] - - -def is_running_trio() -> bool: - try: - # sniffio is a dependency of trio. - - # See https://github.com/python-trio/trio/issues/2802 - import sniffio - - if sniffio.current_async_library() == "trio": - return True - except ImportError: # pragma: nocover - pass - - return False - - -def create_event() -> Event: - if is_running_trio(): - import trio - - return trio.Event() - - import asyncio - - return asyncio.Event() - - -class ASGIResponseStream(AsyncByteStream): - def __init__(self, body: list[bytes]) -> None: - self._body = body - - async def __aiter__(self) -> typing.AsyncIterator[bytes]: - yield b"".join(self._body) - - -class ASGITransport(AsyncBaseTransport): - """ - A custom AsyncTransport that handles sending requests directly to an ASGI app. - - ```python - transport = httpx.ASGITransport( - app=app, - root_path="/submount", - client=("1.2.3.4", 123) - ) - client = httpx.AsyncClient(transport=transport) - ``` - - Arguments: - - * `app` - The ASGI application. - * `raise_app_exceptions` - Boolean indicating if exceptions in the application - should be raised. Default to `True`. Can be set to `False` for use cases - such as testing the content of a client 500 response. - * `root_path` - The root path on which the ASGI application should be mounted. - * `client` - A two-tuple indicating the client IP and port of incoming requests. - ``` - """ - - def __init__( - self, - app: _ASGIApp, - raise_app_exceptions: bool = True, - root_path: str = "", - client: tuple[str, int] = ("127.0.0.1", 123), - ) -> None: - self.app = app - self.raise_app_exceptions = raise_app_exceptions - self.root_path = root_path - self.client = client - - async def handle_async_request( - self, - request: Request, - ) -> Response: - assert isinstance(request.stream, AsyncByteStream) - - # ASGI scope. - scope = { - "type": "http", - "asgi": {"version": "3.0"}, - "http_version": "1.1", - "method": request.method, - "headers": [(k.lower(), v) for (k, v) in request.headers.raw], - "scheme": request.url.scheme, - "path": request.url.path, - "raw_path": request.url.raw_path.split(b"?")[0], - "query_string": request.url.query, - "server": (request.url.host, request.url.port), - "client": self.client, - "root_path": self.root_path, - } - - # Request. - request_body_chunks = request.stream.__aiter__() - request_complete = False - - # Response. - status_code = None - response_headers = None - body_parts = [] - response_started = False - response_complete = create_event() - - # ASGI callables. - - async def receive() -> dict[str, typing.Any]: - nonlocal request_complete - - if request_complete: - await response_complete.wait() - return {"type": "http.disconnect"} - - try: - body = await request_body_chunks.__anext__() - except StopAsyncIteration: - request_complete = True - return {"type": "http.request", "body": b"", "more_body": False} - return {"type": "http.request", "body": body, "more_body": True} - - async def send(message: typing.MutableMapping[str, typing.Any]) -> None: - nonlocal status_code, response_headers, response_started - - if message["type"] == "http.response.start": - assert not response_started - - status_code = message["status"] - response_headers = message.get("headers", []) - response_started = True - - elif message["type"] == "http.response.body": - assert not response_complete.is_set() - body = message.get("body", b"") - more_body = message.get("more_body", False) - - if body and request.method != "HEAD": - body_parts.append(body) - - if not more_body: - response_complete.set() - - try: - await self.app(scope, receive, send) - except Exception: # noqa: PIE-786 - if self.raise_app_exceptions: - raise - - response_complete.set() - if status_code is None: - status_code = 500 - if response_headers is None: - response_headers = {} - - assert response_complete.is_set() - assert status_code is not None - assert response_headers is not None - - stream = ASGIResponseStream(body_parts) - - return Response(status_code, headers=response_headers, stream=stream) diff --git a/httpx/_transports/base.py b/httpx/_transports/base.py deleted file mode 100644 index 66fd99d702..0000000000 --- a/httpx/_transports/base.py +++ /dev/null @@ -1,86 +0,0 @@ -from __future__ import annotations - -import typing -from types import TracebackType - -from .._models import Request, Response - -T = typing.TypeVar("T", bound="BaseTransport") -A = typing.TypeVar("A", bound="AsyncBaseTransport") - -__all__ = ["AsyncBaseTransport", "BaseTransport"] - - -class BaseTransport: - def __enter__(self: T) -> T: - return self - - def __exit__( - self, - exc_type: type[BaseException] | None = None, - exc_value: BaseException | None = None, - traceback: TracebackType | None = None, - ) -> None: - self.close() - - def handle_request(self, request: Request) -> Response: - """ - Send a single HTTP request and return a response. - - Developers shouldn't typically ever need to call into this API directly, - since the Client class provides all the higher level user-facing API - niceties. - - In order to properly release any network resources, the response - stream should *either* be consumed immediately, with a call to - `response.stream.read()`, or else the `handle_request` call should - be followed with a try/finally block to ensuring the stream is - always closed. - - Example usage: - - with httpx.HTTPTransport() as transport: - req = httpx.Request( - method=b"GET", - url=(b"https", b"www.example.com", 443, b"/"), - headers=[(b"Host", b"www.example.com")], - ) - resp = transport.handle_request(req) - body = resp.stream.read() - print(resp.status_code, resp.headers, body) - - - Takes a `Request` instance as the only argument. - - Returns a `Response` instance. - """ - raise NotImplementedError( - "The 'handle_request' method must be implemented." - ) # pragma: no cover - - def close(self) -> None: - pass - - -class AsyncBaseTransport: - async def __aenter__(self: A) -> A: - return self - - async def __aexit__( - self, - exc_type: type[BaseException] | None = None, - exc_value: BaseException | None = None, - traceback: TracebackType | None = None, - ) -> None: - await self.aclose() - - async def handle_async_request( - self, - request: Request, - ) -> Response: - raise NotImplementedError( - "The 'handle_async_request' method must be implemented." - ) # pragma: no cover - - async def aclose(self) -> None: - pass diff --git a/httpx/_transports/default.py b/httpx/_transports/default.py deleted file mode 100644 index fc8c70970a..0000000000 --- a/httpx/_transports/default.py +++ /dev/null @@ -1,406 +0,0 @@ -""" -Custom transports, with nicely configured defaults. - -The following additional keyword arguments are currently supported by httpcore... - -* uds: str -* local_address: str -* retries: int - -Example usages... - -# Disable HTTP/2 on a single specific domain. -mounts = { - "all://": httpx.HTTPTransport(http2=True), - "all://*example.org": httpx.HTTPTransport() -} - -# Using advanced httpcore configuration, with connection retries. -transport = httpx.HTTPTransport(retries=1) -client = httpx.Client(transport=transport) - -# Using advanced httpcore configuration, with unix domain sockets. -transport = httpx.HTTPTransport(uds="socket.uds") -client = httpx.Client(transport=transport) -""" - -from __future__ import annotations - -import contextlib -import typing -from types import TracebackType - -if typing.TYPE_CHECKING: - import ssl # pragma: no cover - - import httpx # pragma: no cover - -from .._config import DEFAULT_LIMITS, Limits, Proxy, create_ssl_context -from .._exceptions import ( - ConnectError, - ConnectTimeout, - LocalProtocolError, - NetworkError, - PoolTimeout, - ProtocolError, - ProxyError, - ReadError, - ReadTimeout, - RemoteProtocolError, - TimeoutException, - UnsupportedProtocol, - WriteError, - WriteTimeout, -) -from .._models import Request, Response -from .._types import AsyncByteStream, CertTypes, ProxyTypes, SyncByteStream -from .._urls import URL -from .base import AsyncBaseTransport, BaseTransport - -T = typing.TypeVar("T", bound="HTTPTransport") -A = typing.TypeVar("A", bound="AsyncHTTPTransport") - -SOCKET_OPTION = typing.Union[ - typing.Tuple[int, int, int], - typing.Tuple[int, int, typing.Union[bytes, bytearray]], - typing.Tuple[int, int, None, int], -] - -__all__ = ["AsyncHTTPTransport", "HTTPTransport"] - -HTTPCORE_EXC_MAP: dict[type[Exception], type[httpx.HTTPError]] = {} - - -def _load_httpcore_exceptions() -> dict[type[Exception], type[httpx.HTTPError]]: - import httpcore - - return { - httpcore.TimeoutException: TimeoutException, - httpcore.ConnectTimeout: ConnectTimeout, - httpcore.ReadTimeout: ReadTimeout, - httpcore.WriteTimeout: WriteTimeout, - httpcore.PoolTimeout: PoolTimeout, - httpcore.NetworkError: NetworkError, - httpcore.ConnectError: ConnectError, - httpcore.ReadError: ReadError, - httpcore.WriteError: WriteError, - httpcore.ProxyError: ProxyError, - httpcore.UnsupportedProtocol: UnsupportedProtocol, - httpcore.ProtocolError: ProtocolError, - httpcore.LocalProtocolError: LocalProtocolError, - httpcore.RemoteProtocolError: RemoteProtocolError, - } - - -@contextlib.contextmanager -def map_httpcore_exceptions() -> typing.Iterator[None]: - global HTTPCORE_EXC_MAP - if len(HTTPCORE_EXC_MAP) == 0: - HTTPCORE_EXC_MAP = _load_httpcore_exceptions() - try: - yield - except Exception as exc: - mapped_exc = None - - for from_exc, to_exc in HTTPCORE_EXC_MAP.items(): - if not isinstance(exc, from_exc): - continue - # We want to map to the most specific exception we can find. - # Eg if `exc` is an `httpcore.ReadTimeout`, we want to map to - # `httpx.ReadTimeout`, not just `httpx.TimeoutException`. - if mapped_exc is None or issubclass(to_exc, mapped_exc): - mapped_exc = to_exc - - if mapped_exc is None: # pragma: no cover - raise - - message = str(exc) - raise mapped_exc(message) from exc - - -class ResponseStream(SyncByteStream): - def __init__(self, httpcore_stream: typing.Iterable[bytes]) -> None: - self._httpcore_stream = httpcore_stream - - def __iter__(self) -> typing.Iterator[bytes]: - with map_httpcore_exceptions(): - for part in self._httpcore_stream: - yield part - - def close(self) -> None: - if hasattr(self._httpcore_stream, "close"): - self._httpcore_stream.close() - - -class HTTPTransport(BaseTransport): - def __init__( - self, - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - trust_env: bool = True, - http1: bool = True, - http2: bool = False, - limits: Limits = DEFAULT_LIMITS, - proxy: ProxyTypes | None = None, - uds: str | None = None, - local_address: str | None = None, - retries: int = 0, - socket_options: typing.Iterable[SOCKET_OPTION] | None = None, - ) -> None: - import httpcore - - proxy = Proxy(url=proxy) if isinstance(proxy, (str, URL)) else proxy - ssl_context = create_ssl_context(verify=verify, cert=cert, trust_env=trust_env) - - if proxy is None: - self._pool = httpcore.ConnectionPool( - ssl_context=ssl_context, - max_connections=limits.max_connections, - max_keepalive_connections=limits.max_keepalive_connections, - keepalive_expiry=limits.keepalive_expiry, - http1=http1, - http2=http2, - uds=uds, - local_address=local_address, - retries=retries, - socket_options=socket_options, - ) - elif proxy.url.scheme in ("http", "https"): - self._pool = httpcore.HTTPProxy( - proxy_url=httpcore.URL( - scheme=proxy.url.raw_scheme, - host=proxy.url.raw_host, - port=proxy.url.port, - target=proxy.url.raw_path, - ), - proxy_auth=proxy.raw_auth, - proxy_headers=proxy.headers.raw, - ssl_context=ssl_context, - proxy_ssl_context=proxy.ssl_context, - max_connections=limits.max_connections, - max_keepalive_connections=limits.max_keepalive_connections, - keepalive_expiry=limits.keepalive_expiry, - http1=http1, - http2=http2, - socket_options=socket_options, - ) - elif proxy.url.scheme in ("socks5", "socks5h"): - try: - import socksio # noqa - except ImportError: # pragma: no cover - raise ImportError( - "Using SOCKS proxy, but the 'socksio' package is not installed. " - "Make sure to install httpx using `pip install httpx[socks]`." - ) from None - - self._pool = httpcore.SOCKSProxy( - proxy_url=httpcore.URL( - scheme=proxy.url.raw_scheme, - host=proxy.url.raw_host, - port=proxy.url.port, - target=proxy.url.raw_path, - ), - proxy_auth=proxy.raw_auth, - ssl_context=ssl_context, - max_connections=limits.max_connections, - max_keepalive_connections=limits.max_keepalive_connections, - keepalive_expiry=limits.keepalive_expiry, - http1=http1, - http2=http2, - ) - else: # pragma: no cover - raise ValueError( - "Proxy protocol must be either 'http', 'https', 'socks5', or 'socks5h'," - f" but got {proxy.url.scheme!r}." - ) - - def __enter__(self: T) -> T: # Use generics for subclass support. - self._pool.__enter__() - return self - - def __exit__( - self, - exc_type: type[BaseException] | None = None, - exc_value: BaseException | None = None, - traceback: TracebackType | None = None, - ) -> None: - with map_httpcore_exceptions(): - self._pool.__exit__(exc_type, exc_value, traceback) - - def handle_request( - self, - request: Request, - ) -> Response: - assert isinstance(request.stream, SyncByteStream) - import httpcore - - req = httpcore.Request( - method=request.method, - url=httpcore.URL( - scheme=request.url.raw_scheme, - host=request.url.raw_host, - port=request.url.port, - target=request.url.raw_path, - ), - headers=request.headers.raw, - content=request.stream, - extensions=request.extensions, - ) - with map_httpcore_exceptions(): - resp = self._pool.handle_request(req) - - assert isinstance(resp.stream, typing.Iterable) - - return Response( - status_code=resp.status, - headers=resp.headers, - stream=ResponseStream(resp.stream), - extensions=resp.extensions, - ) - - def close(self) -> None: - self._pool.close() - - -class AsyncResponseStream(AsyncByteStream): - def __init__(self, httpcore_stream: typing.AsyncIterable[bytes]) -> None: - self._httpcore_stream = httpcore_stream - - async def __aiter__(self) -> typing.AsyncIterator[bytes]: - with map_httpcore_exceptions(): - async for part in self._httpcore_stream: - yield part - - async def aclose(self) -> None: - if hasattr(self._httpcore_stream, "aclose"): - await self._httpcore_stream.aclose() - - -class AsyncHTTPTransport(AsyncBaseTransport): - def __init__( - self, - verify: ssl.SSLContext | str | bool = True, - cert: CertTypes | None = None, - trust_env: bool = True, - http1: bool = True, - http2: bool = False, - limits: Limits = DEFAULT_LIMITS, - proxy: ProxyTypes | None = None, - uds: str | None = None, - local_address: str | None = None, - retries: int = 0, - socket_options: typing.Iterable[SOCKET_OPTION] | None = None, - ) -> None: - import httpcore - - proxy = Proxy(url=proxy) if isinstance(proxy, (str, URL)) else proxy - ssl_context = create_ssl_context(verify=verify, cert=cert, trust_env=trust_env) - - if proxy is None: - self._pool = httpcore.AsyncConnectionPool( - ssl_context=ssl_context, - max_connections=limits.max_connections, - max_keepalive_connections=limits.max_keepalive_connections, - keepalive_expiry=limits.keepalive_expiry, - http1=http1, - http2=http2, - uds=uds, - local_address=local_address, - retries=retries, - socket_options=socket_options, - ) - elif proxy.url.scheme in ("http", "https"): - self._pool = httpcore.AsyncHTTPProxy( - proxy_url=httpcore.URL( - scheme=proxy.url.raw_scheme, - host=proxy.url.raw_host, - port=proxy.url.port, - target=proxy.url.raw_path, - ), - proxy_auth=proxy.raw_auth, - proxy_headers=proxy.headers.raw, - proxy_ssl_context=proxy.ssl_context, - ssl_context=ssl_context, - max_connections=limits.max_connections, - max_keepalive_connections=limits.max_keepalive_connections, - keepalive_expiry=limits.keepalive_expiry, - http1=http1, - http2=http2, - socket_options=socket_options, - ) - elif proxy.url.scheme in ("socks5", "socks5h"): - try: - import socksio # noqa - except ImportError: # pragma: no cover - raise ImportError( - "Using SOCKS proxy, but the 'socksio' package is not installed. " - "Make sure to install httpx using `pip install httpx[socks]`." - ) from None - - self._pool = httpcore.AsyncSOCKSProxy( - proxy_url=httpcore.URL( - scheme=proxy.url.raw_scheme, - host=proxy.url.raw_host, - port=proxy.url.port, - target=proxy.url.raw_path, - ), - proxy_auth=proxy.raw_auth, - ssl_context=ssl_context, - max_connections=limits.max_connections, - max_keepalive_connections=limits.max_keepalive_connections, - keepalive_expiry=limits.keepalive_expiry, - http1=http1, - http2=http2, - ) - else: # pragma: no cover - raise ValueError( - "Proxy protocol must be either 'http', 'https', 'socks5', or 'socks5h'," - f" but got {proxy.url.scheme!r}." - ) - - async def __aenter__(self: A) -> A: # Use generics for subclass support. - await self._pool.__aenter__() - return self - - async def __aexit__( - self, - exc_type: type[BaseException] | None = None, - exc_value: BaseException | None = None, - traceback: TracebackType | None = None, - ) -> None: - with map_httpcore_exceptions(): - await self._pool.__aexit__(exc_type, exc_value, traceback) - - async def handle_async_request( - self, - request: Request, - ) -> Response: - assert isinstance(request.stream, AsyncByteStream) - import httpcore - - req = httpcore.Request( - method=request.method, - url=httpcore.URL( - scheme=request.url.raw_scheme, - host=request.url.raw_host, - port=request.url.port, - target=request.url.raw_path, - ), - headers=request.headers.raw, - content=request.stream, - extensions=request.extensions, - ) - with map_httpcore_exceptions(): - resp = await self._pool.handle_async_request(req) - - assert isinstance(resp.stream, typing.AsyncIterable) - - return Response( - status_code=resp.status, - headers=resp.headers, - stream=AsyncResponseStream(resp.stream), - extensions=resp.extensions, - ) - - async def aclose(self) -> None: - await self._pool.aclose() diff --git a/httpx/_transports/mock.py b/httpx/_transports/mock.py deleted file mode 100644 index 8c418f59e0..0000000000 --- a/httpx/_transports/mock.py +++ /dev/null @@ -1,43 +0,0 @@ -from __future__ import annotations - -import typing - -from .._models import Request, Response -from .base import AsyncBaseTransport, BaseTransport - -SyncHandler = typing.Callable[[Request], Response] -AsyncHandler = typing.Callable[[Request], typing.Coroutine[None, None, Response]] - - -__all__ = ["MockTransport"] - - -class MockTransport(AsyncBaseTransport, BaseTransport): - def __init__(self, handler: SyncHandler | AsyncHandler) -> None: - self.handler = handler - - def handle_request( - self, - request: Request, - ) -> Response: - request.read() - response = self.handler(request) - if not isinstance(response, Response): # pragma: no cover - raise TypeError("Cannot use an async handler in a sync Client") - return response - - async def handle_async_request( - self, - request: Request, - ) -> Response: - await request.aread() - response = self.handler(request) - - # Allow handler to *optionally* be an `async` function. - # If it is, then the `response` variable need to be awaited to actually - # return the result. - - if not isinstance(response, Response): - response = await response - - return response diff --git a/httpx/_transports/wsgi.py b/httpx/_transports/wsgi.py deleted file mode 100644 index 8592ffe017..0000000000 --- a/httpx/_transports/wsgi.py +++ /dev/null @@ -1,149 +0,0 @@ -from __future__ import annotations - -import io -import itertools -import sys -import typing - -from .._models import Request, Response -from .._types import SyncByteStream -from .base import BaseTransport - -if typing.TYPE_CHECKING: - from _typeshed import OptExcInfo # pragma: no cover - from _typeshed.wsgi import WSGIApplication # pragma: no cover - -_T = typing.TypeVar("_T") - - -__all__ = ["WSGITransport"] - - -def _skip_leading_empty_chunks(body: typing.Iterable[_T]) -> typing.Iterable[_T]: - body = iter(body) - for chunk in body: - if chunk: - return itertools.chain([chunk], body) - return [] - - -class WSGIByteStream(SyncByteStream): - def __init__(self, result: typing.Iterable[bytes]) -> None: - self._close = getattr(result, "close", None) - self._result = _skip_leading_empty_chunks(result) - - def __iter__(self) -> typing.Iterator[bytes]: - for part in self._result: - yield part - - def close(self) -> None: - if self._close is not None: - self._close() - - -class WSGITransport(BaseTransport): - """ - A custom transport that handles sending requests directly to an WSGI app. - The simplest way to use this functionality is to use the `app` argument. - - ``` - client = httpx.Client(app=app) - ``` - - Alternatively, you can setup the transport instance explicitly. - This allows you to include any additional configuration arguments specific - to the WSGITransport class: - - ``` - transport = httpx.WSGITransport( - app=app, - script_name="/submount", - remote_addr="1.2.3.4" - ) - client = httpx.Client(transport=transport) - ``` - - Arguments: - - * `app` - The WSGI application. - * `raise_app_exceptions` - Boolean indicating if exceptions in the application - should be raised. Default to `True`. Can be set to `False` for use cases - such as testing the content of a client 500 response. - * `script_name` - The root path on which the WSGI application should be mounted. - * `remote_addr` - A string indicating the client IP of incoming requests. - ``` - """ - - def __init__( - self, - app: WSGIApplication, - raise_app_exceptions: bool = True, - script_name: str = "", - remote_addr: str = "127.0.0.1", - wsgi_errors: typing.TextIO | None = None, - ) -> None: - self.app = app - self.raise_app_exceptions = raise_app_exceptions - self.script_name = script_name - self.remote_addr = remote_addr - self.wsgi_errors = wsgi_errors - - def handle_request(self, request: Request) -> Response: - request.read() - wsgi_input = io.BytesIO(request.content) - - port = request.url.port or {"http": 80, "https": 443}[request.url.scheme] - environ = { - "wsgi.version": (1, 0), - "wsgi.url_scheme": request.url.scheme, - "wsgi.input": wsgi_input, - "wsgi.errors": self.wsgi_errors or sys.stderr, - "wsgi.multithread": True, - "wsgi.multiprocess": False, - "wsgi.run_once": False, - "REQUEST_METHOD": request.method, - "SCRIPT_NAME": self.script_name, - "PATH_INFO": request.url.path, - "QUERY_STRING": request.url.query.decode("ascii"), - "SERVER_NAME": request.url.host, - "SERVER_PORT": str(port), - "SERVER_PROTOCOL": "HTTP/1.1", - "REMOTE_ADDR": self.remote_addr, - } - for header_key, header_value in request.headers.raw: - key = header_key.decode("ascii").upper().replace("-", "_") - if key not in ("CONTENT_TYPE", "CONTENT_LENGTH"): - key = "HTTP_" + key - environ[key] = header_value.decode("ascii") - - seen_status = None - seen_response_headers = None - seen_exc_info = None - - def start_response( - status: str, - response_headers: list[tuple[str, str]], - exc_info: OptExcInfo | None = None, - ) -> typing.Callable[[bytes], typing.Any]: - nonlocal seen_status, seen_response_headers, seen_exc_info - seen_status = status - seen_response_headers = response_headers - seen_exc_info = exc_info - return lambda _: None - - result = self.app(environ, start_response) - - stream = WSGIByteStream(result) - - assert seen_status is not None - assert seen_response_headers is not None - if seen_exc_info and seen_exc_info[0] and self.raise_app_exceptions: - raise seen_exc_info[1] - - status_code = int(seen_status.split()[0]) - headers = [ - (key.encode("ascii"), value.encode("ascii")) - for key, value in seen_response_headers - ] - - return Response(status_code, headers=headers, stream=stream) diff --git a/httpx/_types.py b/httpx/_types.py deleted file mode 100644 index 704dfdffc8..0000000000 --- a/httpx/_types.py +++ /dev/null @@ -1,114 +0,0 @@ -""" -Type definitions for type checking purposes. -""" - -from http.cookiejar import CookieJar -from typing import ( - IO, - TYPE_CHECKING, - Any, - AsyncIterable, - AsyncIterator, - Callable, - Dict, - Iterable, - Iterator, - List, - Mapping, - Optional, - Sequence, - Tuple, - Union, -) - -if TYPE_CHECKING: # pragma: no cover - from ._auth import Auth # noqa: F401 - from ._config import Proxy, Timeout # noqa: F401 - from ._models import Cookies, Headers, Request # noqa: F401 - from ._urls import URL, QueryParams # noqa: F401 - - -PrimitiveData = Optional[Union[str, int, float, bool]] - -URLTypes = Union["URL", str] - -QueryParamTypes = Union[ - "QueryParams", - Mapping[str, Union[PrimitiveData, Sequence[PrimitiveData]]], - List[Tuple[str, PrimitiveData]], - Tuple[Tuple[str, PrimitiveData], ...], - str, - bytes, -] - -HeaderTypes = Union[ - "Headers", - Mapping[str, str], - Mapping[bytes, bytes], - Sequence[Tuple[str, str]], - Sequence[Tuple[bytes, bytes]], -] - -CookieTypes = Union["Cookies", CookieJar, Dict[str, str], List[Tuple[str, str]]] - -TimeoutTypes = Union[ - Optional[float], - Tuple[Optional[float], Optional[float], Optional[float], Optional[float]], - "Timeout", -] -ProxyTypes = Union["URL", str, "Proxy"] -CertTypes = Union[str, Tuple[str, str], Tuple[str, str, str]] - -AuthTypes = Union[ - Tuple[Union[str, bytes], Union[str, bytes]], - Callable[["Request"], "Request"], - "Auth", -] - -RequestContent = Union[str, bytes, Iterable[bytes], AsyncIterable[bytes]] -ResponseContent = Union[str, bytes, Iterable[bytes], AsyncIterable[bytes]] -ResponseExtensions = Mapping[str, Any] - -RequestData = Mapping[str, Any] - -FileContent = Union[IO[bytes], bytes, str] -FileTypes = Union[ - # file (or bytes) - FileContent, - # (filename, file (or bytes)) - Tuple[Optional[str], FileContent], - # (filename, file (or bytes), content_type) - Tuple[Optional[str], FileContent, Optional[str]], - # (filename, file (or bytes), content_type, headers) - Tuple[Optional[str], FileContent, Optional[str], Mapping[str, str]], -] -RequestFiles = Union[Mapping[str, FileTypes], Sequence[Tuple[str, FileTypes]]] - -RequestExtensions = Mapping[str, Any] - -__all__ = ["AsyncByteStream", "SyncByteStream"] - - -class SyncByteStream: - def __iter__(self) -> Iterator[bytes]: - raise NotImplementedError( - "The '__iter__' method must be implemented." - ) # pragma: no cover - yield b"" # pragma: no cover - - def close(self) -> None: - """ - Subclasses can override this method to release any network resources - after a request/response cycle is complete. - """ - - -class AsyncByteStream: - async def __aiter__(self) -> AsyncIterator[bytes]: - raise NotImplementedError( - "The '__aiter__' method must be implemented." - ) # pragma: no cover - yield b"" # pragma: no cover - - async def aclose(self) -> None: - pass diff --git a/httpx/_utils.py b/httpx/_utils.py deleted file mode 100644 index 7fe827da4d..0000000000 --- a/httpx/_utils.py +++ /dev/null @@ -1,242 +0,0 @@ -from __future__ import annotations - -import ipaddress -import os -import re -import typing -from urllib.request import getproxies - -from ._types import PrimitiveData - -if typing.TYPE_CHECKING: # pragma: no cover - from ._urls import URL - - -def primitive_value_to_str(value: PrimitiveData) -> str: - """ - Coerce a primitive data type into a string value. - - Note that we prefer JSON-style 'true'/'false' for boolean values here. - """ - if value is True: - return "true" - elif value is False: - return "false" - elif value is None: - return "" - return str(value) - - -def get_environment_proxies() -> dict[str, str | None]: - """Gets proxy information from the environment""" - - # urllib.request.getproxies() falls back on System - # Registry and Config for proxies on Windows and macOS. - # We don't want to propagate non-HTTP proxies into - # our configuration such as 'TRAVIS_APT_PROXY'. - proxy_info = getproxies() - mounts: dict[str, str | None] = {} - - for scheme in ("http", "https", "all"): - if proxy_info.get(scheme): - hostname = proxy_info[scheme] - mounts[f"{scheme}://"] = ( - hostname if "://" in hostname else f"http://{hostname}" - ) - - no_proxy_hosts = [host.strip() for host in proxy_info.get("no", "").split(",")] - for hostname in no_proxy_hosts: - # See https://curl.haxx.se/libcurl/c/CURLOPT_NOPROXY.html for details - # on how names in `NO_PROXY` are handled. - if hostname == "*": - # If NO_PROXY=* is used or if "*" occurs as any one of the comma - # separated hostnames, then we should just bypass any information - # from HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and always ignore - # proxies. - return {} - elif hostname: - # NO_PROXY=.google.com is marked as "all://*.google.com, - # which disables "www.google.com" but not "google.com" - # NO_PROXY=google.com is marked as "all://*google.com, - # which disables "www.google.com" and "google.com". - # (But not "wwwgoogle.com") - # NO_PROXY can include domains, IPv6, IPv4 addresses and "localhost" - # NO_PROXY=example.com,::1,localhost,192.168.0.0/16 - if "://" in hostname: - mounts[hostname] = None - elif is_ipv4_hostname(hostname): - mounts[f"all://{hostname}"] = None - elif is_ipv6_hostname(hostname): - mounts[f"all://[{hostname}]"] = None - elif hostname.lower() == "localhost": - mounts[f"all://{hostname}"] = None - else: - mounts[f"all://*{hostname}"] = None - - return mounts - - -def to_bytes(value: str | bytes, encoding: str = "utf-8") -> bytes: - return value.encode(encoding) if isinstance(value, str) else value - - -def to_str(value: str | bytes, encoding: str = "utf-8") -> str: - return value if isinstance(value, str) else value.decode(encoding) - - -def to_bytes_or_str(value: str, match_type_of: typing.AnyStr) -> typing.AnyStr: - return value if isinstance(match_type_of, str) else value.encode() - - -def unquote(value: str) -> str: - return value[1:-1] if value[0] == value[-1] == '"' else value - - -def peek_filelike_length(stream: typing.Any) -> int | None: - """ - Given a file-like stream object, return its length in number of bytes - without reading it into memory. - """ - try: - # Is it an actual file? - fd = stream.fileno() - # Yup, seems to be an actual file. - length = os.fstat(fd).st_size - except (AttributeError, OSError): - # No... Maybe it's something that supports random access, like `io.BytesIO`? - try: - # Assuming so, go to end of stream to figure out its length, - # then put it back in place. - offset = stream.tell() - length = stream.seek(0, os.SEEK_END) - stream.seek(offset) - except (AttributeError, OSError): - # Not even that? Sorry, we're doomed... - return None - - return length - - -class URLPattern: - """ - A utility class currently used for making lookups against proxy keys... - - # Wildcard matching... - >>> pattern = URLPattern("all://") - >>> pattern.matches(httpx.URL("http://example.com")) - True - - # Witch scheme matching... - >>> pattern = URLPattern("https://") - >>> pattern.matches(httpx.URL("https://example.com")) - True - >>> pattern.matches(httpx.URL("http://example.com")) - False - - # With domain matching... - >>> pattern = URLPattern("https://example.com") - >>> pattern.matches(httpx.URL("https://example.com")) - True - >>> pattern.matches(httpx.URL("http://example.com")) - False - >>> pattern.matches(httpx.URL("https://other.com")) - False - - # Wildcard scheme, with domain matching... - >>> pattern = URLPattern("all://example.com") - >>> pattern.matches(httpx.URL("https://example.com")) - True - >>> pattern.matches(httpx.URL("http://example.com")) - True - >>> pattern.matches(httpx.URL("https://other.com")) - False - - # With port matching... - >>> pattern = URLPattern("https://example.com:1234") - >>> pattern.matches(httpx.URL("https://example.com:1234")) - True - >>> pattern.matches(httpx.URL("https://example.com")) - False - """ - - def __init__(self, pattern: str) -> None: - from ._urls import URL - - if pattern and ":" not in pattern: - raise ValueError( - f"Proxy keys should use proper URL forms rather " - f"than plain scheme strings. " - f'Instead of "{pattern}", use "{pattern}://"' - ) - - url = URL(pattern) - self.pattern = pattern - self.scheme = "" if url.scheme == "all" else url.scheme - self.host = "" if url.host == "*" else url.host - self.port = url.port - if not url.host or url.host == "*": - self.host_regex: typing.Pattern[str] | None = None - elif url.host.startswith("*."): - # *.example.com should match "www.example.com", but not "example.com" - domain = re.escape(url.host[2:]) - self.host_regex = re.compile(f"^.+\\.{domain}$") - elif url.host.startswith("*"): - # *example.com should match "www.example.com" and "example.com" - domain = re.escape(url.host[1:]) - self.host_regex = re.compile(f"^(.+\\.)?{domain}$") - else: - # example.com should match "example.com" but not "www.example.com" - domain = re.escape(url.host) - self.host_regex = re.compile(f"^{domain}$") - - def matches(self, other: URL) -> bool: - if self.scheme and self.scheme != other.scheme: - return False - if ( - self.host - and self.host_regex is not None - and not self.host_regex.match(other.host) - ): - return False - if self.port is not None and self.port != other.port: - return False - return True - - @property - def priority(self) -> tuple[int, int, int]: - """ - The priority allows URLPattern instances to be sortable, so that - we can match from most specific to least specific. - """ - # URLs with a port should take priority over URLs without a port. - port_priority = 0 if self.port is not None else 1 - # Longer hostnames should match first. - host_priority = -len(self.host) - # Longer schemes should match first. - scheme_priority = -len(self.scheme) - return (port_priority, host_priority, scheme_priority) - - def __hash__(self) -> int: - return hash(self.pattern) - - def __lt__(self, other: URLPattern) -> bool: - return self.priority < other.priority - - def __eq__(self, other: typing.Any) -> bool: - return isinstance(other, URLPattern) and self.pattern == other.pattern - - -def is_ipv4_hostname(hostname: str) -> bool: - try: - ipaddress.IPv4Address(hostname.split("/")[0]) - except Exception: - return False - return True - - -def is_ipv6_hostname(hostname: str) -> bool: - try: - ipaddress.IPv6Address(hostname.split("/")[0]) - except Exception: - return False - return True diff --git a/httpx/py.typed b/httpx/py.typed deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/mkdocs.yml b/mkdocs.yml deleted file mode 100644 index 86ca1e53b7..0000000000 --- a/mkdocs.yml +++ /dev/null @@ -1,61 +0,0 @@ -site_name: HTTPX -site_description: A next-generation HTTP client for Python. -site_url: https://www.python-httpx.org/ - -theme: - name: 'material' - custom_dir: 'docs/overrides' - palette: - - scheme: 'default' - media: '(prefers-color-scheme: light)' - toggle: - icon: 'material/lightbulb' - name: "Switch to dark mode" - - scheme: 'slate' - media: '(prefers-color-scheme: dark)' - primary: 'blue' - toggle: - icon: 'material/lightbulb-outline' - name: 'Switch to light mode' - -repo_name: encode/httpx -repo_url: https://github.com/encode/httpx/ -edit_uri: "" - -nav: - - Introduction: 'index.md' - - QuickStart: 'quickstart.md' - - Advanced: - - Clients: 'advanced/clients.md' - - Authentication: 'advanced/authentication.md' - - SSL: 'advanced/ssl.md' - - Proxies: 'advanced/proxies.md' - - Timeouts: 'advanced/timeouts.md' - - Resource Limits: 'advanced/resource-limits.md' - - Event Hooks: 'advanced/event-hooks.md' - - Transports: 'advanced/transports.md' - - Text Encodings: 'advanced/text-encodings.md' - - Extensions: 'advanced/extensions.md' - - Guides: - - Async Support: 'async.md' - - HTTP/2 Support: 'http2.md' - - Logging: 'logging.md' - - Requests Compatibility: 'compatibility.md' - - Troubleshooting: 'troubleshooting.md' - - API Reference: - - Developer Interface: 'api.md' - - Exceptions: 'exceptions.md' - - Environment Variables: 'environment_variables.md' - - Community: - - Third Party Packages: 'third_party_packages.md' - - Contributing: 'contributing.md' - - Code of Conduct: 'code_of_conduct.md' - -markdown_extensions: - - admonition - - codehilite: - css_class: highlight - - mkautodoc - -extra_css: - - css/custom.css diff --git a/pyproject.toml b/pyproject.toml index fc3e95ea74..dec723e8bd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,26 +1,21 @@ [build-system] -requires = ["hatchling", "hatch-fancy-pypi-readme"] +requires = ["hatchling"] build-backend = "hatchling.build" [project] name = "httpx" -description = "The next generation HTTP client." -license = "BSD-3-Clause" -requires-python = ">=3.9" +description = "HTTP, for Python." +version = "1.0.dev3" +requires-python = ">=3.11" authors = [ { name = "Tom Christie", email = "tom@tomchristie.com" }, ] classifiers = [ "Development Status :: 4 - Beta", "Environment :: Web Environment", - "Framework :: AsyncIO", - "Framework :: Trio", "Intended Audience :: Developers", - "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3 :: Only", - "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", @@ -29,104 +24,4 @@ classifiers = [ ] dependencies = [ "certifi", - "httpcore==1.*", - "anyio", - "idna", ] -dynamic = ["readme", "version"] - -[project.optional-dependencies] -brotli = [ - "brotli; platform_python_implementation == 'CPython'", - "brotlicffi; platform_python_implementation != 'CPython'", -] -cli = [ - "click==8.*", - "pygments==2.*", - "rich>=10,<15", -] -http2 = [ - "h2>=3,<5", -] -socks = [ - "socksio==1.*", -] -zstd = [ - "zstandard>=0.18.0", -] - -[project.scripts] -httpx = "httpx:main" - -[project.urls] -Changelog = "https://github.com/encode/httpx/blob/master/CHANGELOG.md" -Documentation = "https://www.python-httpx.org" -Homepage = "https://github.com/encode/httpx" -Source = "https://github.com/encode/httpx" - -[tool.hatch.version] -path = "httpx/__version__.py" - -[tool.hatch.build.targets.sdist] -include = [ - "/httpx", - "/CHANGELOG.md", - "/README.md", - "/tests", -] - -[tool.hatch.metadata.hooks.fancy-pypi-readme] -content-type = "text/markdown" - -[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]] -path = "README.md" - -[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]] -text = "\n## Release Information\n\n" - -[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]] -path = "CHANGELOG.md" -pattern = "\n(###.+?\n)## " - -[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]] -text = "\n---\n\n[Full changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md)\n" - -[[tool.hatch.metadata.hooks.fancy-pypi-readme.substitutions]] -pattern = 'src="(docs/img/.*?)"' -replacement = 'src="https://raw.githubusercontent.com/encode/httpx/master/\1"' - -[tool.ruff.lint] -select = ["E", "F", "I", "B", "PIE"] -ignore = ["B904", "B028"] - -[tool.ruff.lint.isort] -combine-as-imports = true - -[tool.ruff.lint.per-file-ignores] -"__init__.py" = ["F403", "F405"] - -[tool.mypy] -ignore_missing_imports = true -strict = true - -[[tool.mypy.overrides]] -module = "tests.*" -disallow_untyped_defs = false -check_untyped_defs = true - -[tool.pytest.ini_options] -addopts = "-rxXs" -filterwarnings = [ - "error", - "ignore: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.:RuntimeWarning", - # See: https://github.com/agronholm/anyio/issues/508 - "ignore: trio.MultiError is deprecated since Trio 0.22.0:trio.TrioDeprecationWarning" -] -markers = [ - "copied_from(source, changes=None): mark test as copied from somewhere else, along with a description of changes made to accodomate e.g. our test setup", - "network: marks tests which require network connection. Used in 3rd-party build environments that have network disabled." -] - -[tool.coverage.run] -omit = ["venv/*"] -include = ["httpx/*", "tests/*"] diff --git a/requirements.txt b/requirements.txt index ebc6ea7fc5..f4d4bb3809 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,29 +1,17 @@ -# We're pinning our tooling, because it's an environment we can strictly control. -# On the other hand, we're not pinning package dependencies, because our tests -# needs to pass with the latest version of the packages. -# Reference: https://github.com/encode/httpx/pull/1721#discussion_r661241588 --e .[brotli,cli,http2,socks,zstd] +-e . -# Optional charset auto-detection -# Used in our test cases -chardet==5.2.0 +# Build... +build==1.2.2 -# Documentation -mkdocs==1.6.1 -mkautodoc==0.2.0 -mkdocs-material==9.6.18 +# Test... +mypy==1.15.0 +pytest==8.3.5 +pytest-cov==6.1.1 -# Packaging -build==1.3.0 -twine==6.1.0 +# Sync & Async mirroring... +unasync==0.6.0 -# Tests & Linting -coverage[toml]==7.10.6 -cryptography==45.0.7 -mypy==1.17.1 -pytest==8.4.1 -ruff==0.12.11 -trio==0.31.0 -trio-typing==0.10.0 -trustme==1.2.1 -uvicorn==0.35.0 +# Documentation... +click==8.2.1 +jinja2==3.1.6 +markdown==3.8 diff --git a/scripts/build b/scripts/build index 92378cb942..c7e14690a0 100755 --- a/scripts/build +++ b/scripts/build @@ -1,13 +1,32 @@ -#!/bin/sh -e +#!/bin/sh +PKG=$1 + +if [ "$PKG" != "httpx" ] && [ "$PKG" != "ahttpx" ] ; then + echo "build [httpx|ahttpx]" + exit 1 +fi + +export PREFIX="" if [ -d 'venv' ] ; then - PREFIX="venv/bin/" -else - PREFIX="" + export PREFIX="venv/bin/" +fi + +# Create pyproject-httpx.toml and pyproject-ahttpx.toml +cp pyproject.toml pyproject-httpx.toml +cat pyproject-httpx.toml | sed 's/name = "httpx"/name = "ahttpx"/' > pyproject-ahttpx.toml + +# Build the releases +if [ "$PKG" == "httpx" ]; then + ${PREFIX}python -m build +fi +if [ "$PKG" == "ahttpx" ]; then + cp pyproject-ahttpx.toml pyproject.toml + ${PREFIX}python -m build + cp pyproject-httpx.toml pyproject.toml fi -set -x +# Clean up +rm pyproject-httpx.toml pyproject-ahttpx.toml -${PREFIX}python -m build -${PREFIX}twine check dist/* -${PREFIX}mkdocs build +echo $PKG \ No newline at end of file diff --git a/scripts/check b/scripts/check deleted file mode 100755 index a4bce0948e..0000000000 --- a/scripts/check +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/sh -e - -export PREFIX="" -if [ -d 'venv' ] ; then - export PREFIX="venv/bin/" -fi -export SOURCE_FILES="httpx tests" - -set -x - -./scripts/sync-version -${PREFIX}ruff format $SOURCE_FILES --diff -${PREFIX}mypy $SOURCE_FILES -${PREFIX}ruff check $SOURCE_FILES diff --git a/scripts/clean b/scripts/clean deleted file mode 100755 index f08d02c41c..0000000000 --- a/scripts/clean +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/sh -e - -if [ -d 'dist' ] ; then - rm -r dist -fi -if [ -d 'site' ] ; then - rm -r site -fi -if [ -d 'htmlcov' ] ; then - rm -r htmlcov -fi -if [ -d 'httpx.egg-info' ] ; then - rm -r httpx.egg-info -fi diff --git a/scripts/coverage b/scripts/coverage deleted file mode 100755 index 25a2691074..0000000000 --- a/scripts/coverage +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/sh -e - -export PREFIX="" -if [ -d 'venv' ] ; then - export PREFIX="venv/bin/" -fi -export SOURCE_FILES="httpx tests" - -set -x - -${PREFIX}coverage report --show-missing --skip-covered --fail-under=100 diff --git a/scripts/docs b/scripts/docs index 4ac3beb7a6..8c53da4786 100755 --- a/scripts/docs +++ b/scripts/docs @@ -1,10 +1,153 @@ -#!/bin/sh -e +#!venv/bin/python +import pathlib +import posixpath -export PREFIX="" -if [ -d 'venv' ] ; then - export PREFIX="venv/bin/" -fi +import click +import ghp_import +import logging +import httpx +import jinja2 +import markdown -set -x +import xml.etree.ElementTree as etree -${PREFIX}mkdocs serve + +pages = { + '/': 'docs/index.md', + '/quickstart': 'docs/quickstart.md', + '/clients': 'docs/clients.md', + '/servers': 'docs/servers.md', + '/requests': 'docs/requests.md', + '/responses': 'docs/responses.md', + '/urls': 'docs/urls.md', + '/headers': 'docs/headers.md', + '/content-types': 'docs/content-types.md', + '/streams': 'docs/streams.md', + '/connections': 'docs/connections.md', + '/parsers': 'docs/parsers.md', + '/networking': 'docs/networking.md', + '/about': 'docs/about.md', +} + +def path_to_url(path): + if path == "index.md": + return "/" + return f"/{path[:-3]}" + + +class URLsProcessor(markdown.treeprocessors.Treeprocessor): + def __init__(self, state): + self.state = state + + def run(self, root: etree.Element) -> etree.Element: + for element in root.iter(): + if element.tag == 'a': + key = 'href' + elif element.tag == 'img': + key = 'src' + else: + continue + + url_or_path = element.get(key) + if url_or_path is not None: + output_url = self.rewrite_url(url_or_path) + element.set(key, output_url) + + return root + + def rewrite_url(self, href: str) -> str: + if not href.endswith('.md'): + return href + + current_url = path_to_url(self.state.file) + linked_url = path_to_url(href) + return posixpath.relpath(linked_url, start=current_url) + + +class BuildState: + def __init__(self): + self.file = '' + + +state = BuildState() +env = jinja2.Environment( + loader=jinja2.FileSystemLoader('docs/templates'), + autoescape=False +) +template = env.get_template('base.html') +md = markdown.Markdown(extensions=['fenced_code']) +md.treeprocessors.register( + item=URLsProcessor(state), + name='urls', + priority=10, +) + + +def not_found(): + text = httpx.Text('Not Found') + return httpx.Response(404, content=text) + + +def web_server(request): + if request.url.path not in pages: + return not_found() + + file = pages[request.url.path] + text = pathlib.Path(file).read_text() + + state.file = file + content = md.convert(text) + html = template.render(content=content).encode('utf-8') + content = httpx.HTML(html) + return httpx.Response(200, content=html) + + +@click.group() +def main(): + pass + + +@main.command() +def build(): + pathlib.Path("build").mkdir(exist_ok=True) + + for url, path in pages.items(): + basename = url.lstrip("/") + output = f"build/{basename}.html" if basename else "build/index.html" + text = pathlib.Path(path).read_text() + content = md.convert(text) + html = template.render(content=content) + pathlib.Path(output).write_text(html) + print(f"Built {output}") + + +@main.command() +def serve(): + logging.basicConfig( + format="%(levelname)s [%(asctime)s] %(name)s - %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=logging.INFO + ) + + with httpx.serve_http(web_server) as server: + server.wait() + + +@main.command() +def deploy(): + ghp_import.ghp_import( + "build", + mesg="Documentation deploy", + remote="origin", + branch="gh-pages", + push=True, + force=False, + use_shell=False, + no_history=False, + nojekyll=True, + ) + print(f"Deployed to GitHub") + + +if __name__ == "__main__": + main() diff --git a/scripts/install b/scripts/install index 58b632403b..1b531e57cd 100755 --- a/scripts/install +++ b/scripts/install @@ -1,19 +1,13 @@ -#!/bin/sh -e - -# Use the Python executable provided from the `-p` option, or a default. -[ "$1" = "-p" ] && PYTHON=$2 || PYTHON="python3" - -REQUIREMENTS="requirements.txt" -VENV="venv" +#!/bin/sh set -x if [ -z "$GITHUB_ACTIONS" ]; then - "$PYTHON" -m venv "$VENV" - PIP="$VENV/bin/pip" + python3 -m venv venv + PIP="venv/bin/pip" else PIP="pip" fi "$PIP" install -U pip -"$PIP" install -r "$REQUIREMENTS" +"$PIP" install -r requirements.txt diff --git a/scripts/lint b/scripts/lint deleted file mode 100755 index 6d096d760b..0000000000 --- a/scripts/lint +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/sh -e - -export PREFIX="" -if [ -d 'venv' ]; then - export PREFIX="venv/bin/" -fi -export SOURCE_FILES="httpx tests" - -set -x - -${PREFIX}ruff check --fix $SOURCE_FILES -${PREFIX}ruff format $SOURCE_FILES diff --git a/scripts/publish b/scripts/publish index c0184471a3..6e6955f5e3 100755 --- a/scripts/publish +++ b/scripts/publish @@ -1,26 +1,15 @@ -#!/bin/sh -e +#!/bin/sh -VERSION_FILE="httpx/__version__.py" +PKG=$1 -if [ -d 'venv' ] ; then - PREFIX="venv/bin/" -else - PREFIX="" -fi - -if [ ! -z "$GITHUB_ACTIONS" ]; then - git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com" - git config --local user.name "GitHub Action" - - VERSION=`grep __version__ ${VERSION_FILE} | grep -o '[0-9][^"]*'` - - if [ "refs/tags/${VERSION}" != "${GITHUB_REF}" ] ; then - echo "GitHub Ref '${GITHUB_REF}' did not match package version '${VERSION}'" +if [ "$PKG" != "httpx" ] && [ "$PKG" != "ahttpx" ] ; then + echo "publish [httpx|ahttpx]" exit 1 - fi fi -set -x - -${PREFIX}twine upload dist/* -${PREFIX}mkdocs gh-deploy --force +export PREFIX="" +if [ -d 'venv' ] ; then + export PREFIX="venv/bin/" +fi +${PREFIX}pip install -q twine +${PREFIX}twine upload dist/$PKG-* diff --git a/scripts/sync-version b/scripts/sync-version deleted file mode 100755 index c8eefe1deb..0000000000 --- a/scripts/sync-version +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/sh -e - -SEMVER_REGEX="([0-9]+)\.([0-9]+)\.([0-9]+)(?:-([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?(?:\+[0-9A-Za-z-]+)?" -CHANGELOG_VERSION=$(grep -o -E $SEMVER_REGEX CHANGELOG.md | sed -n 2p) -VERSION=$(grep -o -E $SEMVER_REGEX httpx/__version__.py | head -1) -echo "CHANGELOG_VERSION: $CHANGELOG_VERSION" -echo "VERSION: $VERSION" -if [ "$CHANGELOG_VERSION" != "$VERSION" ]; then - echo "Version in changelog does not match version in httpx/__version__.py!" - exit 1 -fi diff --git a/scripts/test b/scripts/test index d778a544d2..1e0812cd07 100755 --- a/scripts/test +++ b/scripts/test @@ -5,14 +5,6 @@ if [ -d 'venv' ] ; then export PREFIX="venv/bin/" fi -set -ex - -if [ -z $GITHUB_ACTIONS ]; then - scripts/check -fi - -${PREFIX}coverage run -m pytest "$@" - -if [ -z $GITHUB_ACTIONS ]; then - scripts/coverage -fi +${PREFIX}mypy src/httpx +${PREFIX}mypy src/ahttpx +${PREFIX}pytest --cov src/httpx tests diff --git a/scripts/unasync b/scripts/unasync new file mode 100755 index 0000000000..67d66b5c88 --- /dev/null +++ b/scripts/unasync @@ -0,0 +1,29 @@ +#!venv/bin/python +import unasync + +unasync.unasync_files( + fpath_list = [ + "src/ahttpx/__init__.py", + "src/ahttpx/__version__.py", + "src/ahttpx/_client.py", + "src/ahttpx/_content.py", + "src/ahttpx/_headers.py", + "src/ahttpx/_parsers.py", + "src/ahttpx/_pool.py", + "src/ahttpx/_quickstart.py", + "src/ahttpx/_response.py", + "src/ahttpx/_request.py", + "src/ahttpx/_server.py", + "src/ahttpx/_streams.py", + "src/ahttpx/_urlencode.py", + "src/ahttpx/_urlparse.py", + "src/ahttpx/_urls.py", + ], + rules = [ + unasync.Rule( + "src/ahttpx/", + "src/httpx/", + additional_replacements={"ahttpx": "httpx"} + ), + ] +) diff --git a/src/ahttpx/__init__.py b/src/ahttpx/__init__.py new file mode 100644 index 0000000000..2a2e58acd8 --- /dev/null +++ b/src/ahttpx/__init__.py @@ -0,0 +1,62 @@ +from ._client import * # Client +from ._content import * # Content, File, Files, Form, HTML, JSON, MultiPart, Text +from ._headers import * # Headers +from ._network import * # NetworkBackend, NetworkStream, timeout +from ._parsers import * # HTTPParser, ProtocolError +from ._pool import * # Connection, ConnectionPool, Transport +from ._quickstart import * # get, post, put, patch, delete +from ._response import * # Response +from ._request import * # Request +from ._streams import * # ByteStream, DuplexStream, FileStream, HTTPStream, Stream +from ._server import * # serve_http, run +from ._urlencode import * # quote, unquote, urldecode, urlencode +from ._urls import * # QueryParams, URL + + +__all__ = [ + "ByteStream", + "Client", + "Connection", + "ConnectionPool", + "Content", + "delete", + "DuplexStream", + "File", + "FileStream", + "Files", + "Form", + "get", + "Headers", + "HTML", + "HTTPParser", + "HTTPStream", + "JSON", + "MultiPart", + "NetworkBackend", + "NetworkStream", + "open_connection", + "post", + "ProtocolError", + "put", + "patch", + "Response", + "Request", + "run", + "serve_http", + "Stream", + "Text", + "timeout", + "Transport", + "QueryParams", + "quote", + "unquote", + "URL", + "urldecode", + "urlencode", +] + + +__locals = locals() +for __name in __all__: + if not __name.startswith('__'): + setattr(__locals[__name], "__module__", "httpx") diff --git a/src/ahttpx/_client.py b/src/ahttpx/_client.py new file mode 100644 index 0000000000..6326ac5de9 --- /dev/null +++ b/src/ahttpx/_client.py @@ -0,0 +1,156 @@ +import types +import typing + +from ._content import Content +from ._headers import Headers +from ._pool import ConnectionPool, Transport +from ._request import Request +from ._response import Response +from ._streams import Stream +from ._urls import URL + +__all__ = ["Client"] + + +class Client: + def __init__( + self, + url: URL | str | None = None, + headers: Headers | typing.Mapping[str, str] | None = None, + transport: Transport | None = None, + ): + if url is None: + url = "" + if headers is None: + headers = {"User-Agent": "dev"} + if transport is None: + transport = ConnectionPool() + + self.url = URL(url) + self.headers = Headers(headers) + self.transport = transport + self.via = RedirectMiddleware(self.transport) + + def build_request( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Request: + return Request( + method=method, + url=self.url.join(url), + headers=self.headers.copy_update(headers), + content=content, + ) + + async def request( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + request = self.build_request(method, url, headers=headers, content=content) + async with await self.via.send(request) as response: + await response.read() + return response + + async def stream( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + request = self.build_request(method, url, headers=headers, content=content) + return await self.via.send(request) + + async def get( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + ): + return await self.request("GET", url, headers=headers) + + async def post( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + return await self.request("POST", url, headers=headers, content=content) + + async def put( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + return await self.request("PUT", url, headers=headers, content=content) + + async def patch( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + return await self.request("PATCH", url, headers=headers, content=content) + + async def delete( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + ): + return await self.request("DELETE", url, headers=headers) + + async def close(self): + await self.transport.close() + + async def __aenter__(self): + return self + + async def __aexit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None + ): + await self.close() + + def __repr__(self): + return f"" + + +class RedirectMiddleware(Transport): + def __init__(self, transport: Transport) -> None: + self._transport = transport + + def is_redirect(self, response: Response) -> bool: + return ( + response.status_code in (301, 302, 303, 307, 308) + and "Location" in response.headers + ) + + def build_redirect_request(self, request: Request, response: Response) -> Request: + raise NotImplementedError() + + async def send(self, request: Request) -> Response: + while True: + response = await self._transport.send(request) + + if not self.is_redirect(response): + return response + + # If we have a redirect, then we read the body of the response. + # Ensures that the HTTP connection is available for a new + # request/response cycle. + await response.read() + await response.close() + + # We've made a request-response and now need to issue a redirect request. + request = self.build_redirect_request(request, response) + + async def close(self): + pass diff --git a/src/ahttpx/_content.py b/src/ahttpx/_content.py new file mode 100644 index 0000000000..45774fbfb8 --- /dev/null +++ b/src/ahttpx/_content.py @@ -0,0 +1,379 @@ +import json +import os +import typing + +from ._streams import Stream, ByteStream, FileStream, MultiPartStream +from ._urlencode import urldecode, urlencode + +__all__ = [ + "Content", + "Form", + "File", + "Files", + "JSON", + "MultiPart", + "Text", + "HTML", +] + +# https://github.com/nginx/nginx/blob/master/conf/mime.types +_content_types = { + ".json": "application/json", + ".js": "application/javascript", + ".html": "text/html", + ".css": "text/css", + ".png": "image/png", + ".jpeg": "image/jpeg", + ".jpg": "image/jpeg", + ".gif": "image/gif", +} + + +class Content: + def encode(self) -> Stream: + raise NotImplementedError() + + def content_type(self) -> str: + raise NotImplementedError() + + +class Form(typing.Mapping[str, str], Content): + """ + HTML form data, as an immutable multi-dict. + Form parameters, as a multi-dict. + """ + + def __init__( + self, + form: ( + typing.Mapping[str, str | typing.Sequence[str]] + | typing.Sequence[tuple[str, str]] + | str + | None + ) = None, + ) -> None: + d: dict[str, list[str]] = {} + + if form is None: + d = {} + elif isinstance(form, str): + d = urldecode(form) + elif isinstance(form, typing.Mapping): + # Convert dict inputs like: + # {"a": "123", "b": ["456", "789"]} + # To dict inputs where values are always lists, like: + # {"a": ["123"], "b": ["456", "789"]} + d = {k: [v] if isinstance(v, str) else list(v) for k, v in form.items()} + else: + # Convert list inputs like: + # [("a", "123"), ("a", "456"), ("b", "789")] + # To a dict representation, like: + # {"a": ["123", "456"], "b": ["789"]} + for k, v in form: + d.setdefault(k, []).append(v) + + self._dict = d + + # Content API + + def encode(self) -> Stream: + content = str(self).encode("ascii") + return ByteStream(content) + + def content_type(self) -> str: + return "application/x-www-form-urlencoded" + + # Dict operations + + def keys(self) -> typing.KeysView[str]: + return self._dict.keys() + + def values(self) -> typing.ValuesView[str]: + return {k: v[0] for k, v in self._dict.items()}.values() + + def items(self) -> typing.ItemsView[str, str]: + return {k: v[0] for k, v in self._dict.items()}.items() + + def get(self, key: str, default: typing.Any = None) -> typing.Any: + if key in self._dict: + return self._dict[key][0] + return default + + # Multi-dict operations + + def multi_items(self) -> list[tuple[str, str]]: + multi_items: list[tuple[str, str]] = [] + for k, v in self._dict.items(): + multi_items.extend([(k, i) for i in v]) + return multi_items + + def multi_dict(self) -> dict[str, list[str]]: + return {k: list(v) for k, v in self._dict.items()} + + def get_list(self, key: str) -> list[str]: + return list(self._dict.get(key, [])) + + # Update operations + + def copy_set(self, key: str, value: str) -> "Form": + d = self.multi_dict() + d[key] = [value] + return Form(d) + + def copy_append(self, key: str, value: str) -> "Form": + d = self.multi_dict() + d[key] = d.get(key, []) + [value] + return Form(d) + + def copy_remove(self, key: str) -> "Form": + d = self.multi_dict() + d.pop(key, None) + return Form(d) + + # Accessors & built-ins + + def __getitem__(self, key: str) -> str: + return self._dict[key][0] + + def __contains__(self, key: typing.Any) -> bool: + return key in self._dict + + def __iter__(self) -> typing.Iterator[str]: + return iter(self.keys()) + + def __len__(self) -> int: + return len(self._dict) + + def __bool__(self) -> bool: + return bool(self._dict) + + def __hash__(self) -> int: + return hash(str(self)) + + def __eq__(self, other: typing.Any) -> bool: + return ( + isinstance(other, Form) and + sorted(self.multi_items()) == sorted(other.multi_items()) + ) + + def __str__(self) -> str: + return urlencode(self.multi_dict()) + + def __repr__(self) -> str: + return f"" + + +class File(Content): + """ + Wrapper class used for files in uploads and multipart requests. + """ + + def __init__(self, path: str): + self._path = path + + def name(self) -> str: + return os.path.basename(self._path) + + def size(self) -> int: + return os.path.getsize(self._path) + + def encode(self) -> Stream: + fin = open(self._path, 'rb') + return FileStream(self._path, fin) + + def content_type(self) -> str: + _, ext = os.path.splitext(self._path) + ct = _content_types.get(ext, "application/octet-stream") + if ct.startswith('text/'): + ct += "; charset='utf-8'" + return ct + + def __lt__(self, other: typing.Any) -> bool: + return isinstance(other, File) and other._path < self._path + + def __eq__(self, other: typing.Any) -> bool: + return isinstance(other, File) and other._path == self._path + + def __repr__(self) -> str: + return f"" + + +class Files(typing.Mapping[str, File], Content): + """ + File parameters, as a multi-dict. + """ + + def __init__( + self, + files: ( + typing.Mapping[str, File | typing.Sequence[File]] + | typing.Sequence[tuple[str, File]] + | None + ) = None, + boundary: str = '' + ) -> None: + d: dict[str, list[File]] = {} + + if files is None: + d = {} + elif isinstance(files, typing.Mapping): + d = {k: [v] if isinstance(v, File) else list(v) for k, v in files.items()} + else: + d = {} + for k, v in files: + d.setdefault(k, []).append(v) + + self._dict = d + self._boundary = boundary or os.urandom(16).hex() + + # Standard dict interface + def keys(self) -> typing.KeysView[str]: + return self._dict.keys() + + def values(self) -> typing.ValuesView[File]: + return {k: v[0] for k, v in self._dict.items()}.values() + + def items(self) -> typing.ItemsView[str, File]: + return {k: v[0] for k, v in self._dict.items()}.items() + + def get(self, key: str, default: typing.Any = None) -> typing.Any: + if key in self._dict: + return self._dict[key][0] + return None + + # Multi dict interface + def multi_items(self) -> list[tuple[str, File]]: + multi_items: list[tuple[str, File]] = [] + for k, v in self._dict.items(): + multi_items.extend([(k, i) for i in v]) + return multi_items + + def multi_dict(self) -> dict[str, list[File]]: + return {k: list(v) for k, v in self._dict.items()} + + def get_list(self, key: str) -> list[File]: + return list(self._dict.get(key, [])) + + # Content interface + def encode(self) -> Stream: + return MultiPart(files=self).encode() + + def content_type(self) -> str: + return f"multipart/form-data; boundary={self._boundary}" + + # Builtins + def __getitem__(self, key: str) -> File: + return self._dict[key][0] + + def __contains__(self, key: typing.Any) -> bool: + return key in self._dict + + def __iter__(self) -> typing.Iterator[str]: + return iter(self.keys()) + + def __len__(self) -> int: + return len(self._dict) + + def __bool__(self) -> bool: + return bool(self._dict) + + def __eq__(self, other: typing.Any) -> bool: + return ( + isinstance(other, Files) and + sorted(self.multi_items()) == sorted(other.multi_items()) + ) + + def __repr__(self) -> str: + return f"" + + +class JSON(Content): + def __init__(self, data: typing.Any) -> None: + self._data = data + + def encode(self) -> Stream: + content = json.dumps( + self._data, + ensure_ascii=False, + separators=(",", ":"), + allow_nan=False + ).encode("utf-8") + return ByteStream(content) + + def content_type(self) -> str: + return "application/json" + + def __repr__(self) -> str: + return f"" + + +class Text(Content): + def __init__(self, text: str) -> None: + self._text = text + + def encode(self) -> Stream: + content = self._text.encode("utf-8") + return ByteStream(content) + + def content_type(self) -> str: + return "text/plain; charset='utf-8'" + + def __repr__(self) -> str: + return f"" + + +class HTML(Content): + def __init__(self, text: str) -> None: + self._text = text + + def encode(self) -> Stream: + content = self._text.encode("utf-8") + return ByteStream(content) + + def content_type(self) -> str: + return "text/html; charset='utf-8'" + + def __repr__(self) -> str: + return f"" + + +class MultiPart(Content): + def __init__( + self, + form: ( + Form + | typing.Mapping[str, str | typing.Sequence[str]] + | typing.Sequence[tuple[str, str]] + | str + | None + ) = None, + files: ( + Files + | typing.Mapping[str, File | typing.Sequence[File]] + | typing.Sequence[tuple[str, File]] + | None + ) = None, + boundary: str | None = None + ): + self._form = form if isinstance(form , Form) else Form(form) + self._files = files if isinstance(files, Files) else Files(files) + self._boundary = os.urandom(16).hex() if boundary is None else boundary + + @property + def form(self) -> Form: + return self._form + + @property + def files(self) -> Files: + return self._files + + def encode(self) -> Stream: + form = [(key, value) for key, value in self._form.items()] + files = [(key, file._path) for key, file in self._files.items()] + return MultiPartStream(form, files, boundary=self._boundary) + + def content_type(self) -> str: + return f"multipart/form-data; boundary={self._boundary}" + + def __repr__(self) -> str: + return f"" diff --git a/src/ahttpx/_headers.py b/src/ahttpx/_headers.py new file mode 100644 index 0000000000..dade8058d0 --- /dev/null +++ b/src/ahttpx/_headers.py @@ -0,0 +1,243 @@ +import re +import typing + + +__all__ = ["Headers"] + + +VALID_HEADER_CHARS = ( + "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "abcdefghijklmnopqrstuvwxyz" + "0123456789" + "!#$%&'*+-.^_`|~" +) + + +# TODO... +# +# * Comma folded values, eg. `Vary: ...` +# * Multiple Set-Cookie headers. +# * Non-ascii support. +# * Ordering, including `Host` header exception. + + +def headername(name: str) -> str: + if name.strip(VALID_HEADER_CHARS) or not name: + raise ValueError(f"Invalid HTTP header name {name!r}.") + return name + + +def headervalue(value: str) -> str: + value = value.strip(" ") + if not value or not value.isascii() or not value.isprintable(): + raise ValueError(f"Invalid HTTP header value {value!r}.") + return value + + +class Headers(typing.Mapping[str, str]): + def __init__( + self, + headers: typing.Mapping[str, str] | typing.Sequence[tuple[str, str]] | None = None, + ) -> None: + # {'accept': ('Accept', '*/*')} + d: dict[str, str] = {} + + if isinstance(headers, typing.Mapping): + # Headers({ + # 'Content-Length': '1024', + # 'Content-Type': 'text/plain; charset=utf-8', + # ) + d = {headername(k): headervalue(v) for k, v in headers.items()} + elif headers is not None: + # Headers([ + # ('Location', 'https://www.example.com'), + # ('Set-Cookie', 'session_id=3498jj489jhb98jn'), + # ]) + d = {headername(k): headervalue(v) for k, v in headers} + + self._dict = d + + def keys(self) -> typing.KeysView[str]: + """ + Return all the header keys. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.keys()) == ["Accept", "User-Agent"] + """ + return self._dict.keys() + + def values(self) -> typing.ValuesView[str]: + """ + Return all the header values. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.values()) == ["*/*", "python/httpx"] + """ + return self._dict.values() + + def items(self) -> typing.ItemsView[str, str]: + """ + Return all headers as (key, value) tuples. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.items()) == [("Accept", "*/*"), ("User-Agent", "python/httpx")] + """ + return self._dict.items() + + def get(self, key: str, default: typing.Any = None) -> typing.Any: + """ + Get a value from the query param for a given key. If the key occurs + more than once, then only the first value is returned. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert h.get("User-Agent") == "python/httpx" + """ + for k, v in self._dict.items(): + if k.lower() == key.lower(): + return v + return default + + def copy_set(self, key: str, value: str) -> "Headers": + """ + Return a new Headers instance, setting the value of a key. + + Usage: + + h = httpx.Headers({"Expires": "0"}) + h = h.copy_set("Expires", "Wed, 21 Oct 2015 07:28:00 GMT") + assert h == httpx.Headers({"Expires": "Wed, 21 Oct 2015 07:28:00 GMT"}) + """ + l = [] + seen = False + + # Either insert... + for k, v in self._dict.items(): + if k.lower() == key.lower(): + l.append((key, value)) + seen = True + else: + l.append((k, v)) + + # Or append... + if not seen: + l.append((key, value)) + + return Headers(l) + + def copy_remove(self, key: str) -> "Headers": + """ + Return a new Headers instance, removing the value of a key. + + Usage: + + h = httpx.Headers({"Accept": "*/*"}) + h = h.copy_remove("Accept") + assert h == httpx.Headers({}) + """ + h = {k: v for k, v in self._dict.items() if k.lower() != key.lower()} + return Headers(h) + + def copy_update(self, update: "Headers" | typing.Mapping[str, str] | None) -> "Headers": + """ + Return a new Headers instance, removing the value of a key. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + h = h.copy_update({"Accept-Encoding": "gzip"}) + assert h == httpx.Headers({"Accept": "*/*", "Accept-Encoding": "gzip", "User-Agent": "python/httpx"}) + """ + if update is None: + return self + + new = update if isinstance(update, Headers) else Headers(update) + + # Remove updated items using a case-insensitive approach... + keys = set([key.lower() for key in new.keys()]) + h = {k: v for k, v in self._dict.items() if k.lower() not in keys} + + # Perform the actual update... + h.update(dict(new)) + + return Headers(h) + + def __getitem__(self, key: str) -> str: + match = key.lower() + for k, v in self._dict.items(): + if k.lower() == match: + return v + raise KeyError(key) + + def __contains__(self, key: typing.Any) -> bool: + match = key.lower() + return any(k.lower() == match for k in self._dict.keys()) + + def __iter__(self) -> typing.Iterator[str]: + return iter(self.keys()) + + def __len__(self) -> int: + return len(self._dict) + + def __bool__(self) -> bool: + return bool(self._dict) + + def __eq__(self, other: typing.Any) -> bool: + self_lower = {k.lower(): v for k, v in self.items()} + other_lower = {k.lower(): v for k, v in Headers(other).items()} + return self_lower == other_lower + + def __repr__(self) -> str: + return f"" + + +def parse_opts_header(header: str) -> tuple[str, dict[str, str]]: + # The Content-Type header is described in RFC 2616 'Content-Type' + # https://datatracker.ietf.org/doc/html/rfc2616#section-14.17 + + # The 'type/subtype; parameter' format is described in RFC 2616 'Media Types' + # https://datatracker.ietf.org/doc/html/rfc2616#section-3.7 + + # Parameter quoting is described in RFC 2616 'Transfer Codings' + # https://datatracker.ietf.org/doc/html/rfc2616#section-3.6 + + header = header.strip() + content_type = '' + params = {} + + # Match the content type (up to the first semicolon or end) + match = re.match(r'^([^;]+)', header) + if match: + content_type = match.group(1).strip().lower() + rest = header[match.end():] + else: + return '', {} + + # Parse parameters, accounting for quoted strings + param_pattern = re.compile(r''' + ;\s* # Semicolon + optional whitespace + (?P[^=;\s]+) # Parameter key + = # Equal sign + (?P # Parameter value: + "(?:[^"\\]|\\.)*" # Quoted string with escapes + | # OR + [^;]* # Unquoted string (until semicolon) + ) + ''', re.VERBOSE) + + for match in param_pattern.finditer(rest): + key = match.group('key').lower() + value = match.group('value').strip() + if value.startswith('"') and value.endswith('"'): + # Remove surrounding quotes and unescape + value = re.sub(r'\\(.)', r'\1', value[1:-1]) + params[key] = value + + return content_type, params diff --git a/src/ahttpx/_network.py b/src/ahttpx/_network.py new file mode 100644 index 0000000000..957e036167 --- /dev/null +++ b/src/ahttpx/_network.py @@ -0,0 +1,120 @@ +import asyncio +import ssl +import types +import typing + +import certifi + +from ._streams import Stream + + +__all__ = ["NetworkBackend", "NetworkStream", "timeout"] + + +class NetworkStream(Stream): + def __init__( + self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter, address: str = '' + ) -> None: + self._reader = reader + self._writer = writer + self._address = address + self._tls = False + self._closed = False + + async def read(self, size: int = -1) -> bytes: + if size < 0: + size = 64 * 1024 + return await self._reader.read(size) + + async def write(self, buffer: bytes) -> None: + self._writer.write(buffer) + await self._writer.drain() + + async def close(self) -> None: + if not self._closed: + self._writer.close() + await self._writer.wait_closed() + self._closed = True + + def __repr__(self): + description = "" + description += " TLS" if self._tls else "" + description += " CLOSED" if self._closed else "" + return f"" + + def __del__(self): + if not self._closed: + import warnings + warnings.warn("NetworkStream was garbage collected without being closed.") + + # Context managed usage... + async def __aenter__(self) -> "NetworkStream": + return self + + async def __aexit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None, + ): + await self.close() + + +class NetworkServer: + def __init__(self, host: str, port: int, server: asyncio.Server): + self.host = host + self.port = port + self._server = server + + # Context managed usage... + async def __aenter__(self) -> "NetworkServer": + return self + + async def __aexit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None, + ): + self._server.close() + await self._server.wait_closed() + + +class NetworkBackend: + def __init__(self, ssl_ctx: ssl.SSLContext | None = None): + self._ssl_ctx = self.create_default_context() if ssl_ctx is None else ssl_ctx + + def create_default_context(self) -> ssl.SSLContext: + import certifi + return ssl.create_default_context(cafile=certifi.where()) + + async def connect(self, host: str, port: int) -> NetworkStream: + """ + Connect to the given address, returning a Stream instance. + """ + address = f"{host}:{port}" + reader, writer = await asyncio.open_connection(host, port) + return NetworkStream(reader, writer, address=address) + + async def connect_tls(self, host: str, port: int, hostname: str = '') -> NetworkStream: + """ + Connect to the given address, returning a Stream instance. + """ + address = f"{host}:{port}" + reader, writer = await asyncio.open_connection(host, port) + await writer.start_tls(self._ssl_ctx, server_hostname=hostname) + return NetworkStream(reader, writer, address=address) + + async def serve(self, host: str, port: int, handler: typing.Callable[[NetworkStream], None]) -> NetworkServer: + async def callback(reader, writer): + stream = NetworkStream(reader, writer) + await handler(stream) + + server = await asyncio.start_server(callback, host, port) + return NetworkServer(host, port, server) + + +Semaphore = asyncio.Semaphore +Lock = asyncio.Lock +timeout = asyncio.timeout +sleep = asyncio.sleep diff --git a/src/ahttpx/_parsers.py b/src/ahttpx/_parsers.py new file mode 100644 index 0000000000..8a52a56fdf --- /dev/null +++ b/src/ahttpx/_parsers.py @@ -0,0 +1,515 @@ +import enum + +from ._streams import Stream + +__all__ = ['HTTPParser', 'Mode', 'ProtocolError'] + + +# TODO... + +# * Upgrade +# * CONNECT + +# * Support 'Expect: 100 Continue' +# * Add 'Error' state transitions +# * Add tests to trickle data +# * Add type annotations + +# * Optional... HTTP/1.0 support +# * Read trailing headers on Transfer-Encoding: chunked. Not just '\r\n'. +# * When writing Transfer-Encoding: chunked, split large writes into buffer size. +# * When reading Transfer-Encoding: chunked, handle incomplete reads from large chunk sizes. +# * .read() doesn't document if will always return maximum available. + +# * validate method, target, protocol in request line +# * validate protocol, status_code, reason_phrase in response line +# * validate name, value on headers + + +class State(enum.Enum): + WAIT = 0 + SEND_METHOD_LINE = 1 + SEND_STATUS_LINE = 2 + SEND_HEADERS = 3 + SEND_BODY = 4 + RECV_METHOD_LINE = 5 + RECV_STATUS_LINE = 6 + RECV_HEADERS = 7 + RECV_BODY = 8 + DONE = 9 + CLOSED = 10 + + +class Mode(enum.Enum): + CLIENT = 0 + SERVER = 1 + + +# The usual transitions will be... + +# IDLE, IDLE +# SEND_HEADERS, IDLE +# SEND_BODY, IDLE +# DONE, IDLE +# DONE, SEND_HEADERS +# DONE, SEND_BODY +# DONE, DONE + +# Then either back to IDLE, IDLE +# or move to CLOSED, CLOSED + +# 1. It is also valid for the server to start +# sending the response without waiting for the +# complete request. +# 2. 1xx status codes are interim states, and +# transition from SEND_HEADERS back to IDLE +# 3. ... + +class ProtocolError(Exception): + pass + + +class HTTPParser: + """ + Usage... + + client = HTTPParser(writer, reader) + client.send_method_line() + client.send_headers() + client.send_body() + client.recv_status_line() + client.recv_headers() + client.recv_body() + client.complete() + client.close() + """ + def __init__(self, stream: Stream, mode: str) -> None: + self.stream = stream + self.parser = ReadAheadParser(stream) + self.mode = {'CLIENT': Mode.CLIENT, 'SERVER': Mode.SERVER}[mode] + + # Track state... + if self.mode == Mode.CLIENT: + self.send_state: State = State.SEND_METHOD_LINE + self.recv_state: State = State.WAIT + else: + self.recv_state = State.RECV_METHOD_LINE + self.send_state = State.WAIT + + # Track message framing... + self.send_content_length: int | None = 0 + self.recv_content_length: int | None = 0 + self.send_seen_length = 0 + self.recv_seen_length = 0 + + # Track connection keep alive... + self.send_keep_alive = True + self.recv_keep_alive = True + + # Special states... + self.processing_1xx = False + + async def send_method_line(self, method: bytes, target: bytes, protocol: bytes) -> None: + """ + Send the initial request line: + + >>> p.send_method_line(b'GET', b'/', b'HTTP/1.1') + + Sending state will switch to SEND_HEADERS state. + """ + if self.send_state != State.SEND_METHOD_LINE: + msg = f"Called 'send_method_line' in invalid state {self.send_state}" + raise ProtocolError(msg) + + # Send initial request line, eg. "GET / HTTP/1.1" + if protocol != b'HTTP/1.1': + raise ProtocolError("Sent unsupported protocol version") + data = b" ".join([method, target, protocol]) + b"\r\n" + await self.stream.write(data) + + self.send_state = State.SEND_HEADERS + self.recv_state = State.RECV_STATUS_LINE + + async def send_status_line(self, protocol: bytes, status_code: int, reason: bytes) -> None: + """ + Send the initial response line: + + >>> p.send_method_line(b'HTTP/1.1', 200, b'OK') + + Sending state will switch to SEND_HEADERS state. + """ + if self.send_state != State.SEND_STATUS_LINE: + msg = f"Called 'send_status_line' in invalid state {self.send_state}" + raise ProtocolError(msg) + + # Send initial request line, eg. "GET / HTTP/1.1" + if protocol != b'HTTP/1.1': + raise ProtocolError("Sent unsupported protocol version") + status_code_bytes = str(status_code).encode('ascii') + data = b" ".join([protocol, status_code_bytes, reason]) + b"\r\n" + await self.stream.write(data) + + self.send_state = State.SEND_HEADERS + + async def send_headers(self, headers: list[tuple[bytes, bytes]]) -> None: + """ + Send the request headers: + + >>> p.send_headers([(b'Host', b'www.example.com')]) + + Sending state will switch to SEND_BODY state. + """ + if self.send_state != State.SEND_HEADERS: + msg = f"Called 'send_headers' in invalid state {self.send_state}" + raise ProtocolError(msg) + + # Update header state + seen_host = False + for name, value in headers: + lname = name.lower() + if lname == b'host': + seen_host = True + elif lname == b'content-length': + self.send_content_length = bounded_int( + value, + max_digits=20, + exc_text="Sent invalid Content-Length" + ) + elif lname == b'connection' and value == b'close': + self.send_keep_alive = False + elif lname == b'transfer-encoding' and value == b'chunked': + self.send_content_length = None + + if self.mode == Mode.CLIENT and not seen_host: + raise ProtocolError("Request missing 'Host' header") + + # Send request headers + lines = [name + b": " + value + b"\r\n" for name, value in headers] + data = b"".join(lines) + b"\r\n" + await self.stream.write(data) + + self.send_state = State.SEND_BODY + + async def send_body(self, body: bytes) -> None: + """ + Send the request body. An empty bytes argument indicates the end of the stream: + + >>> p.send_body(b'') + + Sending state will switch to DONE. + """ + if self.send_state != State.SEND_BODY: + msg = f"Called 'send_body' in invalid state {self.send_state}" + raise ProtocolError(msg) + + if self.send_content_length is None: + # Transfer-Encoding: chunked + self.send_seen_length += len(body) + marker = f'{len(body):x}\r\n'.encode('ascii') + await self.stream.write(marker + body + b'\r\n') + + else: + # Content-Length: xxx + self.send_seen_length += len(body) + if self.send_seen_length > self.send_content_length: + msg = 'Too much data sent for declared Content-Length' + raise ProtocolError(msg) + if self.send_seen_length < self.send_content_length and body == b'': + msg = 'Not enough data sent for declared Content-Length' + raise ProtocolError(msg) + if body: + await self.stream.write(body) + + if body == b'': + # Handle body close + self.send_state = State.DONE + + async def recv_method_line(self) -> tuple[bytes, bytes, bytes]: + """ + Receive the initial request method line: + + >>> method, target, protocol = p.recv_status_line() + + Receive state will switch to RECV_HEADERS. + """ + if self.recv_state != State.RECV_METHOD_LINE: + msg = f"Called 'recv_method_line' in invalid state {self.recv_state}" + raise ProtocolError(msg) + + # Read initial response line, eg. "GET / HTTP/1.1" + exc_text = "reading request method line" + line = await self.parser.read_until(b"\r\n", max_size=4096, exc_text=exc_text) + method, target, protocol = line.split(b" ", 2) + if protocol != b'HTTP/1.1': + raise ProtocolError("Received unsupported protocol version") + + self.recv_state = State.RECV_HEADERS + self.send_state = State.SEND_STATUS_LINE + return method, target, protocol + + async def recv_status_line(self) -> tuple[bytes, int, bytes]: + """ + Receive the initial response status line: + + >>> protocol, status_code, reason_phrase = p.recv_status_line() + + Receive state will switch to RECV_HEADERS. + """ + if self.recv_state != State.RECV_STATUS_LINE: + msg = f"Called 'recv_status_line' in invalid state {self.recv_state}" + raise ProtocolError(msg) + + # Read initial response line, eg. "HTTP/1.1 200 OK" + exc_text = "reading response status line" + line = await self.parser.read_until(b"\r\n", max_size=4096, exc_text=exc_text) + protocol, status_code_str, reason_phrase = line.split(b" ", 2) + if protocol != b'HTTP/1.1': + raise ProtocolError("Received unsupported protocol version") + + status_code = bounded_int( + status_code_str, + max_digits=3, + exc_text="Received invalid status code" + ) + if status_code < 100: + raise ProtocolError("Received invalid status code") + # 1xx status codes preceed the final response status code + self.processing_1xx = status_code < 200 + + self.recv_state = State.RECV_HEADERS + return protocol, status_code, reason_phrase + + async def recv_headers(self) -> list[tuple[bytes, bytes]]: + """ + Receive the response headers: + + >>> headers = p.recv_status_line() + + Receive state will switch to RECV_BODY by default. + Receive state will revert to RECV_STATUS_CODE for interim 1xx responses. + """ + if self.recv_state != State.RECV_HEADERS: + msg = f"Called 'recv_headers' in invalid state {self.recv_state}" + raise ProtocolError(msg) + + # Read response headers + headers = [] + exc_text = "reading response headers" + while line := await self.parser.read_until(b"\r\n", max_size=4096, exc_text=exc_text): + name, value = line.split(b":", 1) + value = value.strip(b" ") + headers.append((name, value)) + + # Update header state + seen_host = False + for name, value in headers: + lname = name.lower() + if lname == b'host': + seen_host = True + elif lname == b'content-length': + self.recv_content_length = bounded_int( + value, + max_digits=20, + exc_text="Received invalid Content-Length" + ) + elif lname == b'connection' and value == b'close': + self.recv_keep_alive = False + elif lname == b'transfer-encoding' and value == b'chunked': + self.recv_content_length = None + + if self.mode == Mode.SERVER and not seen_host: + raise ProtocolError("Request missing 'Host' header") + + if self.processing_1xx: + # 1xx status codes preceed the final response status code + self.processing_1xx = False + self.recv_state = State.RECV_STATUS_LINE + else: + self.recv_state = State.RECV_BODY + return headers + + async def recv_body(self) -> bytes: + """ + Receive the response body. An empty byte string indicates the end of the stream: + + >>> buffer = bytearray() + >>> while body := p.recv_body() + >>> buffer.extend(body) + + The server will switch to DONE. + """ + if self.recv_state != State.RECV_BODY: + msg = f"Called 'recv_body' in invalid state {self.recv_state}" + raise ProtocolError(msg) + + if self.recv_content_length is None: + # Transfer-Encoding: chunked + exc_text = 'reading chunk size' + line = await self.parser.read_until(b"\r\n", max_size=4096, exc_text=exc_text) + sizestr, _, _ = line.partition(b";") + + exc_text = "Received invalid chunk size" + size = bounded_hex(sizestr, max_digits=8, exc_text=exc_text) + if size > 0: + body = await self.parser.read(size=size) + exc_text = 'reading chunk data' + await self.parser.read_until(b"\r\n", max_size=2, exc_text=exc_text) + self.recv_seen_length += len(body) + else: + body = b'' + exc_text = 'reading chunk termination' + await self.parser.read_until(b"\r\n", max_size=2, exc_text=exc_text) + + else: + # Content-Length: xxx + remaining = self.recv_content_length - self.recv_seen_length + size = min(remaining, 4096) + body = await self.parser.read(size=size) + self.recv_seen_length += len(body) + if self.recv_seen_length < self.recv_content_length and body == b'': + msg = 'Not enough data received for declared Content-Length' + raise ProtocolError(msg) + + if body == b'': + # Handle body close + self.recv_state = State.DONE + return body + + async def complete(self): + is_fully_complete = self.send_state == State.DONE and self.recv_state == State.DONE + is_keepalive = self.send_keep_alive and self.recv_keep_alive + + if not (is_fully_complete and is_keepalive): + await self.close() + return + + if self.mode == Mode.CLIENT: + self.send_state = State.SEND_METHOD_LINE + self.recv_state = State.WAIT + else: + self.recv_state = State.RECV_METHOD_LINE + self.send_state = State.WAIT + + self.send_content_length = 0 + self.recv_content_length = 0 + self.send_seen_length = 0 + self.recv_seen_length = 0 + self.send_keep_alive = True + self.recv_keep_alive = True + self.processing_1xx = False + + async def close(self): + if self.send_state != State.CLOSED: + self.send_state = State.CLOSED + self.recv_state = State.CLOSED + await self.stream.close() + + def is_idle(self) -> bool: + return ( + self.send_state == State.SEND_METHOD_LINE or + self.recv_state == State.RECV_METHOD_LINE + ) + + def is_closed(self) -> bool: + return self.send_state == State.CLOSED + + def description(self) -> str: + return { + State.SEND_METHOD_LINE: "idle", + State.CLOSED: "closed", + }.get(self.send_state, "active") + + def __repr__(self) -> str: + cl_state = self.send_state.name + sr_state = self.recv_state.name + detail = f"client {cl_state}, server {sr_state}" + return f'' + + +class ReadAheadParser: + """ + A buffered I/O stream, with methods for read-ahead parsing. + """ + def __init__(self, stream: Stream) -> None: + self._buffer = b'' + self._stream = stream + self._chunk_size = 4096 + + async def _read_some(self) -> bytes: + if self._buffer: + ret, self._buffer = self._buffer, b'' + return ret + return await self._stream.read(self._chunk_size) + + def _push_back(self, buffer): + assert self._buffer == b'' + self._buffer = buffer + + async def read(self, size: int) -> bytes: + """ + Read and return up to 'size' bytes from the stream, with I/O buffering provided. + + * Returns b'' to indicate connection close. + """ + buffer = bytearray() + while len(buffer) < size: + chunk = await self._read_some() + if not chunk: + break + buffer.extend(chunk) + + if len(buffer) > size: + buffer, push_back = buffer[:size], buffer[size:] + self._push_back(bytes(push_back)) + return bytes(buffer) + + async def read_until(self, marker: bytes, max_size: int, exc_text: str) -> bytes: + """ + Read and return bytes from the stream, delimited by marker. + + * The marker is not included in the return bytes. + * The marker is consumed from the I/O stream. + * Raises `ProtocolError` if the stream closes before a marker occurance. + * Raises `ProtocolError` if marker did not occur within 'max_size + len(marker)' bytes. + """ + buffer = bytearray() + while len(buffer) <= max_size: + chunk = await self._read_some() + if not chunk: + # stream closed before marker found. + raise ProtocolError(f"Stream closed early {exc_text}") + start_search = max(len(buffer) - len(marker), 0) + buffer.extend(chunk) + index = buffer.find(marker, start_search) + + if index > max_size: + # marker was found, though 'max_size' exceeded. + raise ProtocolError(f"Exceeded maximum size {exc_text}") + elif index >= 0: + endindex = index + len(marker) + self._push_back(bytes(buffer[endindex:])) + return bytes(buffer[:index]) + + raise ProtocolError(f"Exceeded maximum size {exc_text}") + + +def bounded_int(intstr: bytes, max_digits: int, exc_text: str): + if len(intstr) > max_digits: + # Length of bytestring exceeds maximum. + raise ProtocolError(exc_text) + if len(intstr.strip(b'0123456789')) != 0: + # Contains invalid characters. + raise ProtocolError(exc_text) + + return int(intstr) + + +def bounded_hex(hexstr: bytes, max_digits: int, exc_text: str): + if len(hexstr) > max_digits: + # Length of bytestring exceeds maximum. + raise ProtocolError(exc_text) + if len(hexstr.strip(b'0123456789abcdefABCDEF')) != 0: + # Contains invalid characters. + raise ProtocolError(exc_text) + + return int(hexstr, base=16) diff --git a/src/ahttpx/_pool.py b/src/ahttpx/_pool.py new file mode 100644 index 0000000000..f712cfac27 --- /dev/null +++ b/src/ahttpx/_pool.py @@ -0,0 +1,284 @@ +import time +import typing +import types + +from ._content import Content +from ._headers import Headers +from ._network import Lock, NetworkBackend, Semaphore +from ._parsers import HTTPParser +from ._response import Response +from ._request import Request +from ._streams import HTTPStream, Stream +from ._urls import URL + + +__all__ = [ + "Transport", + "ConnectionPool", + "Connection", + "open_connection", +] + + +class Transport: + async def send(self, request: Request) -> Response: + raise NotImplementedError() + + async def close(self): + pass + + async def request( + self, + method: str, + url: URL | str, + headers: Headers | dict[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + request = Request(method, url, headers=headers, content=content) + async with await self.send(request) as response: + await response.read() + return response + + async def stream( + self, + method: str, + url: URL | str, + headers: Headers | dict[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + request = Request(method, url, headers=headers, content=content) + response = await self.send(request) + return response + + +class ConnectionPool(Transport): + def __init__(self, backend: NetworkBackend | None = None): + if backend is None: + backend = NetworkBackend() + + self._connections: list[Connection] = [] + self._network_backend = backend + self._limit_concurrency = Semaphore(100) + self._closed = False + + # Public API... + async def send(self, request: Request) -> Response: + if self._closed: + raise RuntimeError("ConnectionPool is closed.") + + # TODO: concurrency limiting + await self._cleanup() + connection = await self._get_connection(request) + response = await connection.send(request) + return response + + async def close(self): + self._closed = True + closing = list(self._connections) + self._connections = [] + for conn in closing: + await conn.close() + + # Create or reuse connections as required... + async def _get_connection(self, request: Request) -> "Connection": + # Attempt to reuse an existing connection. + url = request.url + origin = URL(scheme=url.scheme, host=url.host, port=url.port) + now = time.monotonic() + for conn in self._connections: + if conn.origin() == origin and conn.is_idle() and not conn.is_expired(now): + return conn + + # Or else create a new connection. + conn = await open_connection( + origin, + hostname=request.headers["Host"], + backend=self._network_backend + ) + self._connections.append(conn) + return conn + + # Connection pool management... + async def _cleanup(self) -> None: + now = time.monotonic() + for conn in list(self._connections): + if conn.is_expired(now): + await conn.close() + if conn.is_closed(): + self._connections.remove(conn) + + @property + def connections(self) -> typing.List['Connection']: + return [c for c in self._connections] + + def description(self) -> str: + counts = {"active": 0} + for status in [c.description() for c in self._connections]: + counts[status] = counts.get(status, 0) + 1 + return ", ".join(f"{count} {status}" for status, count in counts.items()) + + # Builtins... + def __repr__(self) -> str: + return f"" + + def __del__(self): + if not self._closed: + import warnings + warnings.warn("ConnectionPool was garbage collected without being closed.") + + async def __aenter__(self) -> "ConnectionPool": + return self + + async def __aexit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None, + ) -> None: + await self.close() + + +class Connection(Transport): + def __init__(self, stream: Stream, origin: URL | str): + self._stream = stream + self._origin = URL(origin) + self._keepalive_duration = 5.0 + self._idle_expiry = time.monotonic() + self._keepalive_duration + self._request_lock = Lock() + self._parser = HTTPParser(stream, mode='CLIENT') + + # API for connection pool management... + def origin(self) -> URL: + return self._origin + + def is_idle(self) -> bool: + return self._parser.is_idle() + + def is_expired(self, when: float) -> bool: + return self._parser.is_idle() and when > self._idle_expiry + + def is_closed(self) -> bool: + return self._parser.is_closed() + + def description(self) -> str: + return self._parser.description() + + # API entry points... + async def send(self, request: Request) -> Response: + #async with self._request_lock: + # try: + await self._send_head(request) + await self._send_body(request) + code, headers = await self._recv_head() + stream = HTTPStream(self._recv_body, self._complete) + # TODO... + return Response(code, headers=headers, content=stream) + # finally: + # await self._cycle_complete() + + async def close(self) -> None: + async with self._request_lock: + await self._close() + + # Top-level API for working directly with a connection. + async def request( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + url = self._origin.join(url) + request = Request(method, url, headers=headers, content=content) + async with await self.send(request) as response: + await response.read() + return response + + async def stream( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + url = self._origin.join(url) + request = Request(method, url, headers=headers, content=content) + return await self.send(request) + + # Send the request... + async def _send_head(self, request: Request) -> None: + method = request.method.encode('ascii') + target = request.url.target.encode('ascii') + protocol = b'HTTP/1.1' + await self._parser.send_method_line(method, target, protocol) + headers = [ + (k.encode('ascii'), v.encode('ascii')) + for k, v in request.headers.items() + ] + await self._parser.send_headers(headers) + + async def _send_body(self, request: Request) -> None: + while data := await request.stream.read(64 * 1024): + await self._parser.send_body(data) + await self._parser.send_body(b'') + + # Receive the response... + async def _recv_head(self) -> tuple[int, Headers]: + _, code, _ = await self._parser.recv_status_line() + h = await self._parser.recv_headers() + headers = Headers([ + (k.decode('ascii'), v.decode('ascii')) + for k, v in h + ]) + return code, headers + + async def _recv_body(self) -> bytes: + return await self._parser.recv_body() + + # Request/response cycle complete... + async def _complete(self) -> None: + await self._parser.complete() + self._idle_expiry = time.monotonic() + self._keepalive_duration + + async def _close(self) -> None: + await self._parser.close() + + # Builtins... + def __repr__(self) -> str: + return f"" + + async def __aenter__(self) -> "Connection": + return self + + async def __aexit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None, + ): + await self.close() + + +async def open_connection( + url: URL | str, + hostname: str = '', + backend: NetworkBackend | None = None, + ) -> Connection: + + if isinstance(url, str): + url = URL(url) + + if url.scheme not in ("http", "https"): + raise ValueError("URL scheme must be 'http://' or 'https://'.") + if backend is None: + backend = NetworkBackend() + + host = url.host + port = url.port or {"http": 80, "https": 443}[url.scheme] + + if url.scheme == "https": + stream = await backend.connect_tls(host, port, hostname) + else: + stream = await backend.connect(host, port) + + return Connection(stream, url) diff --git a/src/ahttpx/_quickstart.py b/src/ahttpx/_quickstart.py new file mode 100644 index 0000000000..8b6e12ff4c --- /dev/null +++ b/src/ahttpx/_quickstart.py @@ -0,0 +1,49 @@ +import typing + +from ._client import Client +from ._content import Content +from ._headers import Headers +from ._streams import Stream +from ._urls import URL + + +__all__ = ['get', 'post', 'put', 'patch', 'delete'] + + +async def get( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, +): + async with Client() as client: + return await client.request("GET", url=url, headers=headers) + +async def post( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, +): + async with Client() as client: + return await client.request("POST", url, headers=headers, content=content) + +async def put( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, +): + async with Client() as client: + return await client.request("PUT", url, headers=headers, content=content) + +async def patch( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, +): + async with Client() as client: + return await client.request("PATCH", url, headers=headers, content=content) + +async def delete( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, +): + async with Client() as client: + return await client.request("DELETE", url=url, headers=headers) diff --git a/src/ahttpx/_request.py b/src/ahttpx/_request.py new file mode 100644 index 0000000000..78b82282d0 --- /dev/null +++ b/src/ahttpx/_request.py @@ -0,0 +1,93 @@ +import types +import typing + +from ._content import Content +from ._streams import ByteStream, Stream +from ._headers import Headers +from ._urls import URL + +__all__ = ["Request"] + + +class Request: + def __init__( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + self.method = method + self.url = URL(url) + self.headers = Headers(headers) + self.stream: Stream = ByteStream(b"") + + # https://datatracker.ietf.org/doc/html/rfc2616#section-14.23 + # RFC 2616, Section 14.23, Host. + # + # A client MUST include a Host header field in all HTTP/1.1 request messages. + if "Host" not in self.headers: + self.headers = self.headers.copy_set("Host", self.url.netloc) + + if content is not None: + if isinstance(content, bytes): + self.stream = ByteStream(content) + elif isinstance(content, Stream): + self.stream = content + elif isinstance(content, Content): + ct = content.content_type() + self.stream = content.encode() + self.headers = self.headers.copy_set("Content-Type", ct) + else: + raise TypeError(f'Expected `Content | Stream | bytes | None` got {type(content)}') + + # https://datatracker.ietf.org/doc/html/rfc2616#section-4.3 + # RFC 2616, Section 4.3, Message Body. + # + # The presence of a message-body in a request is signaled by the + # inclusion of a Content-Length or Transfer-Encoding header field in + # the request's message-headers. + content_length: int | None = self.stream.size + if content_length is None: + self.headers = self.headers.copy_set("Transfer-Encoding", "chunked") + elif content_length > 0: + self.headers = self.headers.copy_set("Content-Length", str(content_length)) + + elif method in ("POST", "PUT", "PATCH"): + # https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.2 + # RFC 7230, Section 3.3.2, Content Length. + # + # A user agent SHOULD send a Content-Length in a request message when no + # Transfer-Encoding is sent and the request method defines a meaning for + # an enclosed payload body. For example, a Content-Length header field is + # normally sent in a POST request even when the value is 0. + # (indicating an empty payload body). + self.headers = self.headers.copy_set("Content-Length", "0") + + @property + def body(self) -> bytes: + if not hasattr(self, '_body'): + raise RuntimeError("'.body' cannot be accessed without calling '.read()'") + return self._body + + async def read(self) -> bytes: + if not hasattr(self, '_body'): + self._body = await self.stream.read() + self.stream = ByteStream(self._body) + return self._body + + async def close(self) -> None: + await self.stream.close() + + async def __aenter__(self): + return self + + async def __aexit__(self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None + ): + await self.close() + + def __repr__(self): + return f"" diff --git a/src/ahttpx/_response.py b/src/ahttpx/_response.py new file mode 100644 index 0000000000..db1de832e4 --- /dev/null +++ b/src/ahttpx/_response.py @@ -0,0 +1,158 @@ +import types +import typing + +from ._content import Content +from ._streams import ByteStream, Stream +from ._headers import Headers, parse_opts_header + +__all__ = ["Response"] + +# We're using the same set as stdlib `http.HTTPStatus` here... +# +# https://github.com/python/cpython/blob/main/Lib/http/__init__.py +_codes = { + 100: "Continue", + 101: "Switching Protocols", + 102: "Processing", + 103: "Early Hints", + 200: "OK", + 201: "Created", + 202: "Accepted", + 203: "Non-Authoritative Information", + 204: "No Content", + 205: "Reset Content", + 206: "Partial Content", + 207: "Multi-Status", + 208: "Already Reported", + 226: "IM Used", + 300: "Multiple Choices", + 301: "Moved Permanently", + 302: "Found", + 303: "See Other", + 304: "Not Modified", + 305: "Use Proxy", + 307: "Temporary Redirect", + 308: "Permanent Redirect", + 400: "Bad Request", + 401: "Unauthorized", + 402: "Payment Required", + 403: "Forbidden", + 404: "Not Found", + 405: "Method Not Allowed", + 406: "Not Acceptable", + 407: "Proxy Authentication Required", + 408: "Request Timeout", + 409: "Conflict", + 410: "Gone", + 411: "Length Required", + 412: "Precondition Failed", + 413: "Content Too Large", + 414: "URI Too Long", + 415: "Unsupported Media Type", + 416: "Range Not Satisfiable", + 417: "Expectation Failed", + 418: "I'm a Teapot", + 421: "Misdirected Request", + 422: "Unprocessable Content", + 423: "Locked", + 424: "Failed Dependency", + 425: "Too Early", + 426: "Upgrade Required", + 428: "Precondition Required", + 429: "Too Many Requests", + 431: "Request Header Fields Too Large", + 451: "Unavailable For Legal Reasons", + 500: "Internal Server Error", + 501: "Not Implemented", + 502: "Bad Gateway", + 503: "Service Unavailable", + 504: "Gateway Timeout", + 505: "HTTP Version Not Supported", + 506: "Variant Also Negotiates", + 507: "Insufficient Storage", + 508: "Loop Detected", + 510: "Not Extended", + 511: "Network Authentication Required", +} + + +class Response: + def __init__( + self, + status_code: int, + *, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + self.status_code = status_code + self.headers = Headers(headers) + self.stream: Stream = ByteStream(b"") + + if content is not None: + if isinstance(content, bytes): + self.stream = ByteStream(content) + elif isinstance(content, Stream): + self.stream = content + elif isinstance(content, Content): + ct = content.content_type() + self.stream = content.encode() + self.headers = self.headers.copy_set("Content-Type", ct) + else: + raise TypeError(f'Expected `Content | Stream | bytes | None` got {type(content)}') + + # https://datatracker.ietf.org/doc/html/rfc2616#section-4.3 + # RFC 2616, Section 4.3, Message Body. + # + # All 1xx (informational), 204 (no content), and 304 (not modified) responses + # MUST NOT include a message-body. All other responses do include a + # message-body, although it MAY be of zero length. + if status_code >= 200 and status_code != 204 and status_code != 304: + content_length: int | None = self.stream.size + if content_length is None: + self.headers = self.headers.copy_set("Transfer-Encoding", "chunked") + else: + self.headers = self.headers.copy_set("Content-Length", str(content_length)) + + @property + def reason_phrase(self): + return _codes.get(self.status_code, "Unknown Status Code") + + @property + def body(self) -> bytes: + if not hasattr(self, '_body'): + raise RuntimeError("'.body' cannot be accessed without calling '.read()'") + return self._body + + @property + def text(self) -> str: + if not hasattr(self, '_body'): + raise RuntimeError("'.text' cannot be accessed without calling '.read()'") + if not hasattr(self, '_text'): + ct = self.headers.get('Content-Type', '') + media, opts = parse_opts_header(ct) + charset = 'utf-8' + if media.startswith('text/'): + charset = opts.get('charset', 'utf-8') + self._text = self._body.decode(charset) + return self._text + + async def read(self) -> bytes: + if not hasattr(self, '_body'): + self._body = await self.stream.read() + return self._body + + async def close(self) -> None: + await self.stream.close() + + async def __aenter__(self): + return self + + async def __aexit__(self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None + ): + await self.close() + + def __repr__(self): + return f"" diff --git a/src/ahttpx/_server.py b/src/ahttpx/_server.py new file mode 100644 index 0000000000..a9103cc97f --- /dev/null +++ b/src/ahttpx/_server.py @@ -0,0 +1,126 @@ +import contextlib +import logging +import time + +from ._content import Text +from ._parsers import HTTPParser +from ._request import Request +from ._response import Response +from ._network import NetworkBackend, sleep +from ._streams import HTTPStream + +__all__ = [ + "serve_http", "run" +] + +logger = logging.getLogger("httpx.server") + + +class ConnectionClosed(Exception): + pass + + +class HTTPConnection: + def __init__(self, stream, endpoint): + self._stream = stream + self._endpoint = endpoint + self._parser = HTTPParser(stream, mode='SERVER') + self._keepalive_duration = 5.0 + self._idle_expiry = time.monotonic() + self._keepalive_duration + + # API entry points... + async def handle_requests(self): + try: + while not self._parser.is_closed(): + method, url, headers = await self._recv_head() + stream = HTTPStream(self._recv_body, self._complete) + # TODO: Handle endpoint exceptions + async with Request(method, url, headers=headers, content=stream) as request: + try: + response = await self._endpoint(request) + status_line = f"{request.method} {request.url.target} [{response.status_code} {response.reason_phrase}]" + logger.info(status_line) + except Exception: + logger.error("Internal Server Error", exc_info=True) + content = Text("Internal Server Error") + err = Response(code=500, content=content) + await self._send_head(err) + await self._send_body(err) + else: + await self._send_head(response) + await self._send_body(response) + except Exception: + logger.error("Internal Server Error", exc_info=True) + + async def close(self): + self._parser.close() + + # Receive the request... + async def _recv_head(self) -> tuple[str, str, list[tuple[str, str]]]: + method, target, _ = await self._parser.recv_method_line() + m = method.decode('ascii') + t = target.decode('ascii') + headers = await self._parser.recv_headers() + h = [ + (k.decode('latin-1'), v.decode('latin-1')) + for k, v in headers + ] + return m, t, h + + async def _recv_body(self): + return await self._parser.recv_body() + + # Return the response... + async def _send_head(self, response: Response): + protocol = b"HTTP/1.1" + status = response.status_code + reason = response.reason_phrase.encode('ascii') + await self._parser.send_status_line(protocol, status, reason) + headers = [ + (k.encode('ascii'), v.encode('ascii')) + for k, v in response.headers.items() + ] + await self._parser.send_headers(headers) + + async def _send_body(self, response: Response): + while data := await response.stream.read(64 * 1024): + await self._parser.send_body(data) + await self._parser.send_body(b'') + + # Start it all over again... + async def _complete(self): + await self._parser.complete + self._idle_expiry = time.monotonic() + self._keepalive_duration + + +class HTTPServer: + def __init__(self, host, port): + self.url = f"http://{host}:{port}/" + + async def wait(self): + while(True): + await sleep(1) + + +@contextlib.asynccontextmanager +async def serve_http(endpoint): + async def handler(stream): + connection = HTTPConnection(stream, endpoint) + await connection.handle_requests() + + logging.basicConfig( + format="%(levelname)s [%(asctime)s] %(name)s - %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=logging.DEBUG + ) + + backend = NetworkBackend() + async with await backend.serve("127.0.0.1", 8080, handler) as server: + server = HTTPServer(server.host, server.port) + logger.info(f"Serving on {server.url} (Press CTRL+C to quit)") + yield server + + +async def run(app): + async with await serve_http(app) as server: + server.wait() diff --git a/src/ahttpx/_streams.py b/src/ahttpx/_streams.py new file mode 100644 index 0000000000..3cf779898c --- /dev/null +++ b/src/ahttpx/_streams.py @@ -0,0 +1,223 @@ +import io +import typing +import types +import os + + +class Stream: + async def read(self, size: int=-1) -> bytes: + raise NotImplementedError() + + async def write(self, data: bytes) -> None: + raise NotImplementedError() + + async def close(self) -> None: + raise NotImplementedError() + + @property + def size(self) -> int | None: + return None + + async def __aenter__(self): + return self + + async def __aexit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None + ): + await self.close() + + +class ByteStream(Stream): + def __init__(self, data: bytes = b''): + self._buffer = io.BytesIO(data) + self._size = len(data) + + async def read(self, size: int=-1) -> bytes: + return self._buffer.read(size) + + async def close(self) -> None: + self._buffer.close() + + @property + def size(self) -> int | None: + return self._size + + +class DuplexStream(Stream): + """ + DuplexStream supports both `read` and `write` operations, + which are applied to seperate buffers. + + This stream can be used for testing network parsers. + """ + + def __init__(self, data: bytes = b''): + self._read_buffer = io.BytesIO(data) + self._write_buffer = io.BytesIO() + + async def read(self, size: int=-1) -> bytes: + return self._read_buffer.read(size) + + async def write(self, buffer: bytes): + return self._write_buffer.write(buffer) + + async def close(self) -> None: + self._read_buffer.close() + self._write_buffer.close() + + def input_bytes(self) -> bytes: + return self._read_buffer.getvalue() + + def output_bytes(self) -> bytes: + return self._write_buffer.getvalue() + + +class FileStream(Stream): + def __init__(self, path: str, fin: typing.Any) -> None: + self._path = path + self._fin = fin + + async def read(self, size: int=-1) -> bytes: + return self._fin.read(size) + + async def close(self) -> None: + self._fin.close() + + @property + def size(self) -> int | None: + return os.path.getsize(self._path) + + +class HTTPStream(Stream): + def __init__(self, next_chunk, complete): + self._next_chunk = next_chunk + self._complete = complete + self._buffer = io.BytesIO() + + async def read(self, size=-1) -> bytes: + sections = [] + length = 0 + + # If we have any data in the buffer read that and clear the buffer. + buffered = self._buffer.read() + if buffered: + sections.append(buffered) + length += len(buffered) + self._buffer.seek(0) + self._buffer.truncate(0) + + # Read each chunk in turn. + while (size < 0) or (length < size): + section = await self._next_chunk() + sections.append(section) + length += len(section) + if section == b'': + break + + # If we've more data than requested, then push some back into the buffer. + output = b''.join(sections) + if size > -1 and len(output) > size: + output, remainder = output[:size], output[size:] + self._buffer.write(remainder) + self._buffer.seek(0) + + return output + + async def close(self) -> None: + self._buffer.close() + if self._complete is not None: + await self._complete() + + +class MultiPartStream(Stream): + def __init__(self, form: list[tuple[str, str]], files: list[tuple[str, str]], boundary=''): + self._form = list(form) + self._files = list(files) + self._boundary = boundary or os.urandom(16).hex() + # Mutable state... + self._form_progress = list(self._form) + self._files_progress = list(self._files) + self._fin: typing.Any = None + self._complete = False + self._buffer = io.BytesIO() + + async def read(self, size=-1) -> bytes: + sections = [] + length = 0 + + # If we have any data in the buffer read that and clear the buffer. + buffered = self._buffer.read() + if buffered: + sections.append(buffered) + length += len(buffered) + self._buffer.seek(0) + self._buffer.truncate(0) + + # Read each multipart section in turn. + while (size < 0) or (length < size): + section = await self._read_next_section() + sections.append(section) + length += len(section) + if section == b'': + break + + # If we've more data than requested, then push some back into the buffer. + output = b''.join(sections) + if size > -1 and len(output) > size: + output, remainder = output[:size], output[size:] + self._buffer.write(remainder) + self._buffer.seek(0) + + return output + + async def _read_next_section(self) -> bytes: + if self._form_progress: + # return a form item + key, value = self._form_progress.pop(0) + name = key.translate({10: "%0A", 13: "%0D", 34: "%22"}) + return ( + f"--{self._boundary}\r\n" + f'Content-Disposition: form-data; name="{name}"\r\n' + f"\r\n" + f"{value}\r\n" + ).encode("utf-8") + elif self._files_progress and self._fin is None: + # return start of a file item + key, value = self._files_progress.pop(0) + self._fin = open(value, 'rb') + name = key.translate({10: "%0A", 13: "%0D", 34: "%22"}) + filename = os.path.basename(value) + return ( + f"--{self._boundary}\r\n" + f'Content-Disposition: form-data; name="{name}"; filename="{filename}"\r\n' + f"\r\n" + ).encode("utf-8") + elif self._fin is not None: + chunk = await self._fin.read(64*1024) + if chunk != b'': + # return some bytes from file + return chunk + else: + # return end of file item + await self._fin.close() + self._fin = None + return b"\r\n" + elif not self._complete: + # return final section of multipart + self._complete = True + return f"--{self._boundary}--\r\n".encode("utf-8") + # return EOF marker + return b"" + + async def close(self) -> None: + if self._fin is not None: + await self._fin.close() + self._fin = None + self._buffer.close() + + @property + def size(self) -> int | None: + return None diff --git a/src/ahttpx/_urlencode.py b/src/ahttpx/_urlencode.py new file mode 100644 index 0000000000..1a83b620a6 --- /dev/null +++ b/src/ahttpx/_urlencode.py @@ -0,0 +1,85 @@ +import re + +__all__ = ["quote", "unquote", "urldecode", "urlencode"] + + +# Matchs a sequence of one or more '%xx' escapes. +PERCENT_ENCODED_REGEX = re.compile("(%[A-Fa-f0-9][A-Fa-f0-9])+") + +# https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 +SAFE = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~" + + +def urlencode(multidict, safe=SAFE): + pairs = [] + for key, values in multidict.items(): + pairs.extend([(key, value) for value in values]) + + safe += "+" + pairs = [(k.replace(" ", "+"), v.replace(" ", "+")) for k, v in pairs] + + return "&".join( + f"{quote(key, safe)}={quote(val, safe)}" + for key, val in pairs + ) + + +def urldecode(string): + parts = [part.partition("=") for part in string.split("&") if part] + pairs = [ + (unquote(key), unquote(val)) + for key, _, val in parts + ] + + pairs = [(k.replace("+", " "), v.replace("+", " ")) for k, v in pairs] + + ret = {} + for k, v in pairs: + ret.setdefault(k, []).append(v) + return ret + + +def quote(string, safe=SAFE): + # Fast path if the string is already safe. + if not string.strip(safe): + return string + + # Replace any characters not in the safe set with '%xx' escape sequences. + return "".join([ + char if char in safe else percent(char) + for char in string + ]) + + +def unquote(string): + # Fast path if the string is not quoted. + if '%' not in string: + return string + + # Unquote. + parts = [] + current_position = 0 + for match in re.finditer(PERCENT_ENCODED_REGEX, string): + start_position, end_position = match.start(), match.end() + matched_text = match.group(0) + # Include any text up to the '%xx' escape sequence. + if start_position != current_position: + leading_text = string[current_position:start_position] + parts.append(leading_text) + + # Decode the '%xx' escape sequence. + hex = matched_text.replace('%', '') + decoded = bytes.fromhex(hex).decode('utf-8') + parts.append(decoded) + current_position = end_position + + # Include any text after the final '%xx' escape sequence. + if current_position != len(string): + trailing_text = string[current_position:] + parts.append(trailing_text) + + return "".join(parts) + + +def percent(c): + return ''.join(f"%{b:02X}" for b in c.encode("utf-8")) diff --git a/httpx/_urlparse.py b/src/ahttpx/_urlparse.py similarity index 95% rename from httpx/_urlparse.py rename to src/ahttpx/_urlparse.py index bf190fd560..612892fa8b 100644 --- a/httpx/_urlparse.py +++ b/src/ahttpx/_urlparse.py @@ -16,15 +16,14 @@ required. """ -from __future__ import annotations - import ipaddress import re import typing -import idna -from ._exceptions import InvalidURL +class InvalidURL(ValueError): + pass + MAX_URL_LENGTH = 65536 @@ -183,7 +182,7 @@ def netloc(self) -> str: ] ) - def copy_with(self, **kwargs: str | None) -> ParseResult: + def copy_with(self, **kwargs: str | None) -> "ParseResult": if not kwargs: return self @@ -376,20 +375,28 @@ def encode_host(host: str) -> str: raise InvalidURL(f"Invalid IPv6 address: {host!r}") return host[1:-1] - elif host.isascii(): - # Regular ASCII hostnames - # - # From https://datatracker.ietf.org/doc/html/rfc3986/#section-3.2.2 - # - # reg-name = *( unreserved / pct-encoded / sub-delims ) - WHATWG_SAFE = '"`{}%|\\' - return quote(host.lower(), safe=SUB_DELIMS + WHATWG_SAFE) + elif not host.isascii(): + try: + import idna # type: ignore + except ImportError: + raise InvalidURL( + f"Cannot handle URL with IDNA hostname: {host!r}. " + f"Package 'idna' is not installed." + ) + + # IDNA hostnames + try: + return idna.encode(host.lower()).decode("ascii") + except idna.IDNAError: + raise InvalidURL(f"Invalid IDNA hostname: {host!r}") - # IDNA hostnames - try: - return idna.encode(host.lower()).decode("ascii") - except idna.IDNAError: - raise InvalidURL(f"Invalid IDNA hostname: {host!r}") + # Regular ASCII hostnames + # + # From https://datatracker.ietf.org/doc/html/rfc3986/#section-3.2.2 + # + # reg-name = *( unreserved / pct-encoded / sub-delims ) + WHATWG_SAFE = '"`{}%|\\' + return quote(host.lower(), safe=SUB_DELIMS + WHATWG_SAFE) def normalize_port(port: str | int | None, scheme: str) -> int | None: diff --git a/httpx/_urls.py b/src/ahttpx/_urls.py similarity index 67% rename from httpx/_urls.py rename to src/ahttpx/_urls.py index 301d0874d5..4ae4464e25 100644 --- a/httpx/_urls.py +++ b/src/ahttpx/_urls.py @@ -1,15 +1,11 @@ from __future__ import annotations import typing -from urllib.parse import parse_qs, unquote, urlencode -import idna - -from ._types import QueryParamTypes from ._urlparse import urlparse -from ._utils import primitive_value_to_str +from ._urlencode import unquote, urldecode, urlencode -__all__ = ["URL", "QueryParams"] +__all__ = ["QueryParams", "URL"] class URL: @@ -74,7 +70,7 @@ class URL: themselves. """ - def __init__(self, url: URL | str = "", **kwargs: typing.Any) -> None: + def __init__(self, url: "URL" | str = "", **kwargs: typing.Any) -> None: if kwargs: allowed = { "scheme": str, @@ -83,7 +79,7 @@ def __init__(self, url: URL | str = "", **kwargs: typing.Any) -> None: "userinfo": bytes, "host": str, "port": int, - "netloc": bytes, + "netloc": str, "path": str, "query": bytes, "raw_path": bytes, @@ -131,14 +127,6 @@ def scheme(self) -> str: """ return self._uri_reference.scheme - @property - def raw_scheme(self) -> bytes: - """ - The raw bytes representation of the URL scheme, such as b"http", b"https". - Always normalised to lowercase. - """ - return self._uri_reference.scheme.encode("ascii") - @property def userinfo(self) -> bytes: """ @@ -169,7 +157,7 @@ def password(self) -> str: def host(self) -> str: """ The URL host as a string. - Always normalized to lowercase, with IDNA hosts decoded into unicode. + Always normalized to lowercase. Possibly IDNA encoded. Examples: @@ -177,42 +165,15 @@ def host(self) -> str: assert url.host == "www.example.org" url = httpx.URL("http://中国.icom.museum") - assert url.host == "中国.icom.museum" + assert url.host == "xn--fiqs8s" url = httpx.URL("http://xn--fiqs8s.icom.museum") - assert url.host == "中国.icom.museum" + assert url.host == "xn--fiqs8s" url = httpx.URL("https://[::ffff:192.168.0.1]") assert url.host == "::ffff:192.168.0.1" """ - host: str = self._uri_reference.host - - if host.startswith("xn--"): - host = idna.decode(host) - - return host - - @property - def raw_host(self) -> bytes: - """ - The raw bytes representation of the URL host. - Always normalized to lowercase, and IDNA encoded. - - Examples: - - url = httpx.URL("http://www.EXAMPLE.org") - assert url.raw_host == b"www.example.org" - - url = httpx.URL("http://中国.icom.museum") - assert url.raw_host == b"xn--fiqs8s.icom.museum" - - url = httpx.URL("http://xn--fiqs8s.icom.museum") - assert url.raw_host == b"xn--fiqs8s.icom.museum" - - url = httpx.URL("https://[::ffff:192.168.0.1]") - assert url.raw_host == b"::ffff:192.168.0.1" - """ - return self._uri_reference.host.encode("ascii") + return self._uri_reference.host @property def port(self) -> int | None: @@ -231,7 +192,7 @@ def port(self) -> int | None: return self._uri_reference.port @property - def netloc(self) -> bytes: + def netloc(self) -> str: """ Either `` or `:` as bytes. Always normalized to lowercase, and IDNA encoded. @@ -239,7 +200,7 @@ def netloc(self) -> bytes: This property may be used for generating the value of a request "Host" header. """ - return self._uri_reference.netloc.encode("ascii") + return self._uri_reference.netloc @property def path(self) -> str: @@ -272,7 +233,7 @@ def query(self) -> bytes: return query.encode("ascii") @property - def params(self) -> QueryParams: + def params(self) -> "QueryParams": """ The URL query parameters, neatly parsed and packaged into an immutable multidict representation. @@ -280,7 +241,7 @@ def params(self) -> QueryParams: return QueryParams(self._uri_reference.query) @property - def raw_path(self) -> bytes: + def target(self) -> str: """ The complete URL path and query string as raw bytes. Used as the target when constructing HTTP requests. @@ -291,10 +252,10 @@ def raw_path(self) -> bytes: Host: www.example.org Connection: close """ - path = self._uri_reference.path or "/" + target = self._uri_reference.path or "/" if self._uri_reference.query is not None: - path += "?" + self._uri_reference.query - return path.encode("ascii") + target += "?" + self._uri_reference.query + return target @property def fragment(self) -> str: @@ -324,7 +285,7 @@ def is_relative_url(self) -> bool: """ return not self.is_absolute_url - def copy_with(self, **kwargs: typing.Any) -> URL: + def copy_with(self, **kwargs: typing.Any) -> "URL": """ Copy this URL, returning a new URL with some components altered. Accepts the same set of parameters as the components that are made @@ -339,19 +300,22 @@ def copy_with(self, **kwargs: typing.Any) -> URL: """ return URL(self, **kwargs) - def copy_set_param(self, key: str, value: typing.Any = None) -> URL: - return self.copy_with(params=self.params.set(key, value)) + def copy_set_param(self, key: str, value: typing.Any = None) -> "URL": + return self.copy_with(params=self.params.copy_set(key, value)) - def copy_add_param(self, key: str, value: typing.Any = None) -> URL: - return self.copy_with(params=self.params.add(key, value)) + def copy_append_param(self, key: str, value: typing.Any = None) -> "URL": + return self.copy_with(params=self.params.copy_append(key, value)) - def copy_remove_param(self, key: str) -> URL: - return self.copy_with(params=self.params.remove(key)) + def copy_remove_param(self, key: str) -> "URL": + return self.copy_with(params=self.params.copy_remove(key)) - def copy_merge_params(self, params: QueryParamTypes) -> URL: - return self.copy_with(params=self.params.merge(params)) + def copy_merge_params( + self, + params: "QueryParams" | dict[str, str | list[str]] | list[tuple[str, str]] | None, + ) -> "URL": + return self.copy_with(params=self.params.copy_update(params)) - def join(self, url: URL | str) -> URL: + def join(self, url: "URL" | str) -> "URL": """ Return an absolute URL, using this URL as the base. @@ -375,46 +339,7 @@ def __str__(self) -> str: return str(self._uri_reference) def __repr__(self) -> str: - scheme, userinfo, host, port, path, query, fragment = self._uri_reference - - if ":" in userinfo: - # Mask any password component. - userinfo = f"{userinfo.split(':')[0]}:[secure]" - - authority = "".join( - [ - f"{userinfo}@" if userinfo else "", - f"[{host}]" if ":" in host else host, - f":{port}" if port is not None else "", - ] - ) - url = "".join( - [ - f"{self.scheme}:" if scheme else "", - f"//{authority}" if authority else "", - path, - f"?{query}" if query is not None else "", - f"#{fragment}" if fragment is not None else "", - ] - ) - - return f"{self.__class__.__name__}({url!r})" - - @property - def raw(self) -> tuple[bytes, bytes, int, bytes]: # pragma: nocover - import collections - import warnings - - warnings.warn("URL.raw is deprecated.") - RawURL = collections.namedtuple( - "RawURL", ["raw_scheme", "raw_host", "port", "raw_path"] - ) - return RawURL( - raw_scheme=self.raw_scheme, - raw_host=self.raw_host, - port=self.port, - raw_path=self.raw_path, - ) + return f"" class QueryParams(typing.Mapping[str, str]): @@ -422,43 +347,35 @@ class QueryParams(typing.Mapping[str, str]): URL query parameters, as a multi-dict. """ - def __init__(self, *args: QueryParamTypes | None, **kwargs: typing.Any) -> None: - assert len(args) < 2, "Too many arguments." - assert not (args and kwargs), "Cannot mix named and unnamed arguments." - - value = args[0] if args else kwargs - - if value is None or isinstance(value, (str, bytes)): - value = value.decode("ascii") if isinstance(value, bytes) else value - self._dict = parse_qs(value, keep_blank_values=True) - elif isinstance(value, QueryParams): - self._dict = {k: list(v) for k, v in value._dict.items()} + def __init__( + self, + params: ( + "QueryParams" | dict[str, str | list[str]] | list[tuple[str, str]] | str | None + ) = None, + ) -> None: + d: dict[str, list[str]] = {} + + if params is None: + d = {} + elif isinstance(params, str): + d = urldecode(params) + elif isinstance(params, QueryParams): + d = params.multi_dict() + elif isinstance(params, dict): + # Convert dict inputs like: + # {"a": "123", "b": ["456", "789"]} + # To dict inputs where values are always lists, like: + # {"a": ["123"], "b": ["456", "789"]} + d = {k: [v] if isinstance(v, str) else list(v) for k, v in params.items()} else: - dict_value: dict[typing.Any, list[typing.Any]] = {} - if isinstance(value, (list, tuple)): - # Convert list inputs like: - # [("a", "123"), ("a", "456"), ("b", "789")] - # To a dict representation, like: - # {"a": ["123", "456"], "b": ["789"]} - for item in value: - dict_value.setdefault(item[0], []).append(item[1]) - else: - # Convert dict inputs like: - # {"a": "123", "b": ["456", "789"]} - # To dict inputs where values are always lists, like: - # {"a": ["123"], "b": ["456", "789"]} - dict_value = { - k: list(v) if isinstance(v, (list, tuple)) else [v] - for k, v in value.items() - } - - # Ensure that keys and values are neatly coerced to strings. - # We coerce values `True` and `False` to JSON-like "true" and "false" - # representations, and coerce `None` values to the empty string. - self._dict = { - str(k): [primitive_value_to_str(item) for item in v] - for k, v in dict_value.items() - } + # Convert list inputs like: + # [("a", "123"), ("a", "456"), ("b", "789")] + # To a dict representation, like: + # {"a": ["123", "456"], "b": ["789"]} + for k, v in params: + d.setdefault(k, []).append(v) + + self._dict = d def keys(self) -> typing.KeysView[str]: """ @@ -509,7 +426,10 @@ def multi_items(self) -> list[tuple[str, str]]: multi_items.extend([(k, i) for i in v]) return multi_items - def get(self, key: typing.Any, default: typing.Any = None) -> typing.Any: + def multi_dict(self) -> dict[str, list[str]]: + return {k: list(v) for k, v in self._dict.items()} + + def get(self, key: str, default: typing.Any = None) -> typing.Any: """ Get a value from the query param for a given key. If the key occurs more than once, then only the first value is returned. @@ -520,7 +440,7 @@ def get(self, key: typing.Any, default: typing.Any = None) -> typing.Any: assert q.get("a") == "123" """ if key in self._dict: - return self._dict[str(key)][0] + return self._dict[key][0] return default def get_list(self, key: str) -> list[str]: @@ -532,9 +452,9 @@ def get_list(self, key: str) -> list[str]: q = httpx.QueryParams("a=123&a=456&b=789") assert q.get_list("a") == ["123", "456"] """ - return list(self._dict.get(str(key), [])) + return list(self._dict.get(key, [])) - def set(self, key: str, value: typing.Any = None) -> QueryParams: + def copy_set(self, key: str, value: str) -> "QueryParams": """ Return a new QueryParams instance, setting the value of a key. @@ -546,25 +466,25 @@ def set(self, key: str, value: typing.Any = None) -> QueryParams: """ q = QueryParams() q._dict = dict(self._dict) - q._dict[str(key)] = [primitive_value_to_str(value)] + q._dict[key] = [value] return q - def add(self, key: str, value: typing.Any = None) -> QueryParams: + def copy_append(self, key: str, value: str) -> "QueryParams": """ Return a new QueryParams instance, setting or appending the value of a key. Usage: q = httpx.QueryParams("a=123") - q = q.add("a", "456") + q = q.append("a", "456") assert q == httpx.QueryParams("a=123&a=456") """ q = QueryParams() q._dict = dict(self._dict) - q._dict[str(key)] = q.get_list(key) + [primitive_value_to_str(value)] + q._dict[key] = q.get_list(key) + [value] return q - def remove(self, key: str) -> QueryParams: + def copy_remove(self, key: str) -> QueryParams: """ Return a new QueryParams instance, removing the value of a key. @@ -579,31 +499,36 @@ def remove(self, key: str) -> QueryParams: q._dict.pop(str(key), None) return q - def merge(self, params: QueryParamTypes | None = None) -> QueryParams: + def copy_update( + self, + params: ( + "QueryParams" | dict[str, str | list[str]] | list[tuple[str, str]] | None + ) = None, + ) -> "QueryParams": """ Return a new QueryParams instance, updated with. Usage: q = httpx.QueryParams("a=123") - q = q.merge({"b": "456"}) + q = q.copy_update({"b": "456"}) assert q == httpx.QueryParams("a=123&b=456") q = httpx.QueryParams("a=123") - q = q.merge({"a": "456", "b": "789"}) + q = q.copy_update({"a": "456", "b": "789"}) assert q == httpx.QueryParams("a=456&b=789") """ q = QueryParams(params) q._dict = {**self._dict, **q._dict} return q - def __getitem__(self, key: typing.Any) -> str: + def __getitem__(self, key: str) -> str: return self._dict[key][0] def __contains__(self, key: typing.Any) -> bool: return key in self._dict - def __iter__(self) -> typing.Iterator[typing.Any]: + def __iter__(self) -> typing.Iterator[str]: return iter(self.keys()) def __len__(self) -> int: @@ -621,21 +546,7 @@ def __eq__(self, other: typing.Any) -> bool: return sorted(self.multi_items()) == sorted(other.multi_items()) def __str__(self) -> str: - return urlencode(self.multi_items()) + return urlencode(self.multi_dict()) def __repr__(self) -> str: - class_name = self.__class__.__name__ - query_string = str(self) - return f"{class_name}({query_string!r})" - - def update(self, params: QueryParamTypes | None = None) -> None: - raise RuntimeError( - "QueryParams are immutable since 0.18.0. " - "Use `q = q.merge(...)` to create an updated copy." - ) - - def __setitem__(self, key: str, value: str) -> None: - raise RuntimeError( - "QueryParams are immutable since 0.18.0. " - "Use `q = q.set(key, value)` to create an updated copy." - ) + return f"" diff --git a/src/httpx/__init__.py b/src/httpx/__init__.py new file mode 100644 index 0000000000..2a2e58acd8 --- /dev/null +++ b/src/httpx/__init__.py @@ -0,0 +1,62 @@ +from ._client import * # Client +from ._content import * # Content, File, Files, Form, HTML, JSON, MultiPart, Text +from ._headers import * # Headers +from ._network import * # NetworkBackend, NetworkStream, timeout +from ._parsers import * # HTTPParser, ProtocolError +from ._pool import * # Connection, ConnectionPool, Transport +from ._quickstart import * # get, post, put, patch, delete +from ._response import * # Response +from ._request import * # Request +from ._streams import * # ByteStream, DuplexStream, FileStream, HTTPStream, Stream +from ._server import * # serve_http, run +from ._urlencode import * # quote, unquote, urldecode, urlencode +from ._urls import * # QueryParams, URL + + +__all__ = [ + "ByteStream", + "Client", + "Connection", + "ConnectionPool", + "Content", + "delete", + "DuplexStream", + "File", + "FileStream", + "Files", + "Form", + "get", + "Headers", + "HTML", + "HTTPParser", + "HTTPStream", + "JSON", + "MultiPart", + "NetworkBackend", + "NetworkStream", + "open_connection", + "post", + "ProtocolError", + "put", + "patch", + "Response", + "Request", + "run", + "serve_http", + "Stream", + "Text", + "timeout", + "Transport", + "QueryParams", + "quote", + "unquote", + "URL", + "urldecode", + "urlencode", +] + + +__locals = locals() +for __name in __all__: + if not __name.startswith('__'): + setattr(__locals[__name], "__module__", "httpx") diff --git a/src/httpx/_client.py b/src/httpx/_client.py new file mode 100644 index 0000000000..2dd54fd34d --- /dev/null +++ b/src/httpx/_client.py @@ -0,0 +1,156 @@ +import types +import typing + +from ._content import Content +from ._headers import Headers +from ._pool import ConnectionPool, Transport +from ._request import Request +from ._response import Response +from ._streams import Stream +from ._urls import URL + +__all__ = ["Client"] + + +class Client: + def __init__( + self, + url: URL | str | None = None, + headers: Headers | typing.Mapping[str, str] | None = None, + transport: Transport | None = None, + ): + if url is None: + url = "" + if headers is None: + headers = {"User-Agent": "dev"} + if transport is None: + transport = ConnectionPool() + + self.url = URL(url) + self.headers = Headers(headers) + self.transport = transport + self.via = RedirectMiddleware(self.transport) + + def build_request( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Request: + return Request( + method=method, + url=self.url.join(url), + headers=self.headers.copy_update(headers), + content=content, + ) + + def request( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + request = self.build_request(method, url, headers=headers, content=content) + with self.via.send(request) as response: + response.read() + return response + + def stream( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + request = self.build_request(method, url, headers=headers, content=content) + return self.via.send(request) + + def get( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + ): + return self.request("GET", url, headers=headers) + + def post( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + return self.request("POST", url, headers=headers, content=content) + + def put( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + return self.request("PUT", url, headers=headers, content=content) + + def patch( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + return self.request("PATCH", url, headers=headers, content=content) + + def delete( + self, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + ): + return self.request("DELETE", url, headers=headers) + + def close(self): + self.transport.close() + + def __enter__(self): + return self + + def __exit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None + ): + self.close() + + def __repr__(self): + return f"" + + +class RedirectMiddleware(Transport): + def __init__(self, transport: Transport) -> None: + self._transport = transport + + def is_redirect(self, response: Response) -> bool: + return ( + response.status_code in (301, 302, 303, 307, 308) + and "Location" in response.headers + ) + + def build_redirect_request(self, request: Request, response: Response) -> Request: + raise NotImplementedError() + + def send(self, request: Request) -> Response: + while True: + response = self._transport.send(request) + + if not self.is_redirect(response): + return response + + # If we have a redirect, then we read the body of the response. + # Ensures that the HTTP connection is available for a new + # request/response cycle. + response.read() + response.close() + + # We've made a request-response and now need to issue a redirect request. + request = self.build_redirect_request(request, response) + + def close(self): + pass diff --git a/src/httpx/_content.py b/src/httpx/_content.py new file mode 100644 index 0000000000..45774fbfb8 --- /dev/null +++ b/src/httpx/_content.py @@ -0,0 +1,379 @@ +import json +import os +import typing + +from ._streams import Stream, ByteStream, FileStream, MultiPartStream +from ._urlencode import urldecode, urlencode + +__all__ = [ + "Content", + "Form", + "File", + "Files", + "JSON", + "MultiPart", + "Text", + "HTML", +] + +# https://github.com/nginx/nginx/blob/master/conf/mime.types +_content_types = { + ".json": "application/json", + ".js": "application/javascript", + ".html": "text/html", + ".css": "text/css", + ".png": "image/png", + ".jpeg": "image/jpeg", + ".jpg": "image/jpeg", + ".gif": "image/gif", +} + + +class Content: + def encode(self) -> Stream: + raise NotImplementedError() + + def content_type(self) -> str: + raise NotImplementedError() + + +class Form(typing.Mapping[str, str], Content): + """ + HTML form data, as an immutable multi-dict. + Form parameters, as a multi-dict. + """ + + def __init__( + self, + form: ( + typing.Mapping[str, str | typing.Sequence[str]] + | typing.Sequence[tuple[str, str]] + | str + | None + ) = None, + ) -> None: + d: dict[str, list[str]] = {} + + if form is None: + d = {} + elif isinstance(form, str): + d = urldecode(form) + elif isinstance(form, typing.Mapping): + # Convert dict inputs like: + # {"a": "123", "b": ["456", "789"]} + # To dict inputs where values are always lists, like: + # {"a": ["123"], "b": ["456", "789"]} + d = {k: [v] if isinstance(v, str) else list(v) for k, v in form.items()} + else: + # Convert list inputs like: + # [("a", "123"), ("a", "456"), ("b", "789")] + # To a dict representation, like: + # {"a": ["123", "456"], "b": ["789"]} + for k, v in form: + d.setdefault(k, []).append(v) + + self._dict = d + + # Content API + + def encode(self) -> Stream: + content = str(self).encode("ascii") + return ByteStream(content) + + def content_type(self) -> str: + return "application/x-www-form-urlencoded" + + # Dict operations + + def keys(self) -> typing.KeysView[str]: + return self._dict.keys() + + def values(self) -> typing.ValuesView[str]: + return {k: v[0] for k, v in self._dict.items()}.values() + + def items(self) -> typing.ItemsView[str, str]: + return {k: v[0] for k, v in self._dict.items()}.items() + + def get(self, key: str, default: typing.Any = None) -> typing.Any: + if key in self._dict: + return self._dict[key][0] + return default + + # Multi-dict operations + + def multi_items(self) -> list[tuple[str, str]]: + multi_items: list[tuple[str, str]] = [] + for k, v in self._dict.items(): + multi_items.extend([(k, i) for i in v]) + return multi_items + + def multi_dict(self) -> dict[str, list[str]]: + return {k: list(v) for k, v in self._dict.items()} + + def get_list(self, key: str) -> list[str]: + return list(self._dict.get(key, [])) + + # Update operations + + def copy_set(self, key: str, value: str) -> "Form": + d = self.multi_dict() + d[key] = [value] + return Form(d) + + def copy_append(self, key: str, value: str) -> "Form": + d = self.multi_dict() + d[key] = d.get(key, []) + [value] + return Form(d) + + def copy_remove(self, key: str) -> "Form": + d = self.multi_dict() + d.pop(key, None) + return Form(d) + + # Accessors & built-ins + + def __getitem__(self, key: str) -> str: + return self._dict[key][0] + + def __contains__(self, key: typing.Any) -> bool: + return key in self._dict + + def __iter__(self) -> typing.Iterator[str]: + return iter(self.keys()) + + def __len__(self) -> int: + return len(self._dict) + + def __bool__(self) -> bool: + return bool(self._dict) + + def __hash__(self) -> int: + return hash(str(self)) + + def __eq__(self, other: typing.Any) -> bool: + return ( + isinstance(other, Form) and + sorted(self.multi_items()) == sorted(other.multi_items()) + ) + + def __str__(self) -> str: + return urlencode(self.multi_dict()) + + def __repr__(self) -> str: + return f"" + + +class File(Content): + """ + Wrapper class used for files in uploads and multipart requests. + """ + + def __init__(self, path: str): + self._path = path + + def name(self) -> str: + return os.path.basename(self._path) + + def size(self) -> int: + return os.path.getsize(self._path) + + def encode(self) -> Stream: + fin = open(self._path, 'rb') + return FileStream(self._path, fin) + + def content_type(self) -> str: + _, ext = os.path.splitext(self._path) + ct = _content_types.get(ext, "application/octet-stream") + if ct.startswith('text/'): + ct += "; charset='utf-8'" + return ct + + def __lt__(self, other: typing.Any) -> bool: + return isinstance(other, File) and other._path < self._path + + def __eq__(self, other: typing.Any) -> bool: + return isinstance(other, File) and other._path == self._path + + def __repr__(self) -> str: + return f"" + + +class Files(typing.Mapping[str, File], Content): + """ + File parameters, as a multi-dict. + """ + + def __init__( + self, + files: ( + typing.Mapping[str, File | typing.Sequence[File]] + | typing.Sequence[tuple[str, File]] + | None + ) = None, + boundary: str = '' + ) -> None: + d: dict[str, list[File]] = {} + + if files is None: + d = {} + elif isinstance(files, typing.Mapping): + d = {k: [v] if isinstance(v, File) else list(v) for k, v in files.items()} + else: + d = {} + for k, v in files: + d.setdefault(k, []).append(v) + + self._dict = d + self._boundary = boundary or os.urandom(16).hex() + + # Standard dict interface + def keys(self) -> typing.KeysView[str]: + return self._dict.keys() + + def values(self) -> typing.ValuesView[File]: + return {k: v[0] for k, v in self._dict.items()}.values() + + def items(self) -> typing.ItemsView[str, File]: + return {k: v[0] for k, v in self._dict.items()}.items() + + def get(self, key: str, default: typing.Any = None) -> typing.Any: + if key in self._dict: + return self._dict[key][0] + return None + + # Multi dict interface + def multi_items(self) -> list[tuple[str, File]]: + multi_items: list[tuple[str, File]] = [] + for k, v in self._dict.items(): + multi_items.extend([(k, i) for i in v]) + return multi_items + + def multi_dict(self) -> dict[str, list[File]]: + return {k: list(v) for k, v in self._dict.items()} + + def get_list(self, key: str) -> list[File]: + return list(self._dict.get(key, [])) + + # Content interface + def encode(self) -> Stream: + return MultiPart(files=self).encode() + + def content_type(self) -> str: + return f"multipart/form-data; boundary={self._boundary}" + + # Builtins + def __getitem__(self, key: str) -> File: + return self._dict[key][0] + + def __contains__(self, key: typing.Any) -> bool: + return key in self._dict + + def __iter__(self) -> typing.Iterator[str]: + return iter(self.keys()) + + def __len__(self) -> int: + return len(self._dict) + + def __bool__(self) -> bool: + return bool(self._dict) + + def __eq__(self, other: typing.Any) -> bool: + return ( + isinstance(other, Files) and + sorted(self.multi_items()) == sorted(other.multi_items()) + ) + + def __repr__(self) -> str: + return f"" + + +class JSON(Content): + def __init__(self, data: typing.Any) -> None: + self._data = data + + def encode(self) -> Stream: + content = json.dumps( + self._data, + ensure_ascii=False, + separators=(",", ":"), + allow_nan=False + ).encode("utf-8") + return ByteStream(content) + + def content_type(self) -> str: + return "application/json" + + def __repr__(self) -> str: + return f"" + + +class Text(Content): + def __init__(self, text: str) -> None: + self._text = text + + def encode(self) -> Stream: + content = self._text.encode("utf-8") + return ByteStream(content) + + def content_type(self) -> str: + return "text/plain; charset='utf-8'" + + def __repr__(self) -> str: + return f"" + + +class HTML(Content): + def __init__(self, text: str) -> None: + self._text = text + + def encode(self) -> Stream: + content = self._text.encode("utf-8") + return ByteStream(content) + + def content_type(self) -> str: + return "text/html; charset='utf-8'" + + def __repr__(self) -> str: + return f"" + + +class MultiPart(Content): + def __init__( + self, + form: ( + Form + | typing.Mapping[str, str | typing.Sequence[str]] + | typing.Sequence[tuple[str, str]] + | str + | None + ) = None, + files: ( + Files + | typing.Mapping[str, File | typing.Sequence[File]] + | typing.Sequence[tuple[str, File]] + | None + ) = None, + boundary: str | None = None + ): + self._form = form if isinstance(form , Form) else Form(form) + self._files = files if isinstance(files, Files) else Files(files) + self._boundary = os.urandom(16).hex() if boundary is None else boundary + + @property + def form(self) -> Form: + return self._form + + @property + def files(self) -> Files: + return self._files + + def encode(self) -> Stream: + form = [(key, value) for key, value in self._form.items()] + files = [(key, file._path) for key, file in self._files.items()] + return MultiPartStream(form, files, boundary=self._boundary) + + def content_type(self) -> str: + return f"multipart/form-data; boundary={self._boundary}" + + def __repr__(self) -> str: + return f"" diff --git a/src/httpx/_headers.py b/src/httpx/_headers.py new file mode 100644 index 0000000000..dade8058d0 --- /dev/null +++ b/src/httpx/_headers.py @@ -0,0 +1,243 @@ +import re +import typing + + +__all__ = ["Headers"] + + +VALID_HEADER_CHARS = ( + "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "abcdefghijklmnopqrstuvwxyz" + "0123456789" + "!#$%&'*+-.^_`|~" +) + + +# TODO... +# +# * Comma folded values, eg. `Vary: ...` +# * Multiple Set-Cookie headers. +# * Non-ascii support. +# * Ordering, including `Host` header exception. + + +def headername(name: str) -> str: + if name.strip(VALID_HEADER_CHARS) or not name: + raise ValueError(f"Invalid HTTP header name {name!r}.") + return name + + +def headervalue(value: str) -> str: + value = value.strip(" ") + if not value or not value.isascii() or not value.isprintable(): + raise ValueError(f"Invalid HTTP header value {value!r}.") + return value + + +class Headers(typing.Mapping[str, str]): + def __init__( + self, + headers: typing.Mapping[str, str] | typing.Sequence[tuple[str, str]] | None = None, + ) -> None: + # {'accept': ('Accept', '*/*')} + d: dict[str, str] = {} + + if isinstance(headers, typing.Mapping): + # Headers({ + # 'Content-Length': '1024', + # 'Content-Type': 'text/plain; charset=utf-8', + # ) + d = {headername(k): headervalue(v) for k, v in headers.items()} + elif headers is not None: + # Headers([ + # ('Location', 'https://www.example.com'), + # ('Set-Cookie', 'session_id=3498jj489jhb98jn'), + # ]) + d = {headername(k): headervalue(v) for k, v in headers} + + self._dict = d + + def keys(self) -> typing.KeysView[str]: + """ + Return all the header keys. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.keys()) == ["Accept", "User-Agent"] + """ + return self._dict.keys() + + def values(self) -> typing.ValuesView[str]: + """ + Return all the header values. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.values()) == ["*/*", "python/httpx"] + """ + return self._dict.values() + + def items(self) -> typing.ItemsView[str, str]: + """ + Return all headers as (key, value) tuples. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.items()) == [("Accept", "*/*"), ("User-Agent", "python/httpx")] + """ + return self._dict.items() + + def get(self, key: str, default: typing.Any = None) -> typing.Any: + """ + Get a value from the query param for a given key. If the key occurs + more than once, then only the first value is returned. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert h.get("User-Agent") == "python/httpx" + """ + for k, v in self._dict.items(): + if k.lower() == key.lower(): + return v + return default + + def copy_set(self, key: str, value: str) -> "Headers": + """ + Return a new Headers instance, setting the value of a key. + + Usage: + + h = httpx.Headers({"Expires": "0"}) + h = h.copy_set("Expires", "Wed, 21 Oct 2015 07:28:00 GMT") + assert h == httpx.Headers({"Expires": "Wed, 21 Oct 2015 07:28:00 GMT"}) + """ + l = [] + seen = False + + # Either insert... + for k, v in self._dict.items(): + if k.lower() == key.lower(): + l.append((key, value)) + seen = True + else: + l.append((k, v)) + + # Or append... + if not seen: + l.append((key, value)) + + return Headers(l) + + def copy_remove(self, key: str) -> "Headers": + """ + Return a new Headers instance, removing the value of a key. + + Usage: + + h = httpx.Headers({"Accept": "*/*"}) + h = h.copy_remove("Accept") + assert h == httpx.Headers({}) + """ + h = {k: v for k, v in self._dict.items() if k.lower() != key.lower()} + return Headers(h) + + def copy_update(self, update: "Headers" | typing.Mapping[str, str] | None) -> "Headers": + """ + Return a new Headers instance, removing the value of a key. + + Usage: + + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + h = h.copy_update({"Accept-Encoding": "gzip"}) + assert h == httpx.Headers({"Accept": "*/*", "Accept-Encoding": "gzip", "User-Agent": "python/httpx"}) + """ + if update is None: + return self + + new = update if isinstance(update, Headers) else Headers(update) + + # Remove updated items using a case-insensitive approach... + keys = set([key.lower() for key in new.keys()]) + h = {k: v for k, v in self._dict.items() if k.lower() not in keys} + + # Perform the actual update... + h.update(dict(new)) + + return Headers(h) + + def __getitem__(self, key: str) -> str: + match = key.lower() + for k, v in self._dict.items(): + if k.lower() == match: + return v + raise KeyError(key) + + def __contains__(self, key: typing.Any) -> bool: + match = key.lower() + return any(k.lower() == match for k in self._dict.keys()) + + def __iter__(self) -> typing.Iterator[str]: + return iter(self.keys()) + + def __len__(self) -> int: + return len(self._dict) + + def __bool__(self) -> bool: + return bool(self._dict) + + def __eq__(self, other: typing.Any) -> bool: + self_lower = {k.lower(): v for k, v in self.items()} + other_lower = {k.lower(): v for k, v in Headers(other).items()} + return self_lower == other_lower + + def __repr__(self) -> str: + return f"" + + +def parse_opts_header(header: str) -> tuple[str, dict[str, str]]: + # The Content-Type header is described in RFC 2616 'Content-Type' + # https://datatracker.ietf.org/doc/html/rfc2616#section-14.17 + + # The 'type/subtype; parameter' format is described in RFC 2616 'Media Types' + # https://datatracker.ietf.org/doc/html/rfc2616#section-3.7 + + # Parameter quoting is described in RFC 2616 'Transfer Codings' + # https://datatracker.ietf.org/doc/html/rfc2616#section-3.6 + + header = header.strip() + content_type = '' + params = {} + + # Match the content type (up to the first semicolon or end) + match = re.match(r'^([^;]+)', header) + if match: + content_type = match.group(1).strip().lower() + rest = header[match.end():] + else: + return '', {} + + # Parse parameters, accounting for quoted strings + param_pattern = re.compile(r''' + ;\s* # Semicolon + optional whitespace + (?P[^=;\s]+) # Parameter key + = # Equal sign + (?P # Parameter value: + "(?:[^"\\]|\\.)*" # Quoted string with escapes + | # OR + [^;]* # Unquoted string (until semicolon) + ) + ''', re.VERBOSE) + + for match in param_pattern.finditer(rest): + key = match.group('key').lower() + value = match.group('value').strip() + if value.startswith('"') and value.endswith('"'): + # Remove surrounding quotes and unescape + value = re.sub(r'\\(.)', r'\1', value[1:-1]) + params[key] = value + + return content_type, params diff --git a/src/httpx/_network.py b/src/httpx/_network.py new file mode 100644 index 0000000000..5ea9bb5472 --- /dev/null +++ b/src/httpx/_network.py @@ -0,0 +1,243 @@ +import concurrent.futures +import contextlib +import contextvars +import select +import socket +import ssl +import threading +import time +import types +import typing + +from ._streams import Stream + + +__all__ = ["NetworkBackend", "NetworkStream", "timeout"] + +_timeout_stack: contextvars.ContextVar[list[float]] = contextvars.ContextVar("timeout_context", default=[]) + + +@contextlib.contextmanager +def timeout(duration: float) -> typing.Iterator[None]: + """ + A context managed timeout API. + + with timeout(1.0): + ... + """ + now = time.monotonic() + until = now + duration + stack = typing.cast(list[float], _timeout_stack.get()) + stack = [until] + stack + token = _timeout_stack.set(stack) + try: + yield + finally: + _timeout_stack.reset(token) + + +def get_current_timeout() -> float | None: + stack = _timeout_stack.get() + if not stack: + return None + soonest = min(stack) + now = time.monotonic() + remaining = soonest - now + if remaining <= 0.0: + raise TimeoutError() + return remaining + + +class NetworkStream(Stream): + def __init__(self, sock: socket.socket, address: tuple[str, int]) -> None: + self._socket = sock + self._address = address + self._is_tls = False + self._is_closed = False + + @property + def host(self) -> str: + return self._address[0] + + @property + def port(self) -> int: + return self._address[1] + + def read(self, size: int = -1) -> bytes: + if size < 0: + size = 64 * 1024 + timeout = get_current_timeout() + self._socket.settimeout(timeout) + content = self._socket.recv(size) + return content + + def write(self, buffer: bytes) -> None: + while buffer: + timeout = get_current_timeout() + self._socket.settimeout(timeout) + n = self._socket.send(buffer) + buffer = buffer[n:] + + def close(self) -> None: + if not self._is_closed: + self._is_closed = True + self._socket.close() + + def __repr__(self): + description = "" + description += " TLS" if self._is_tls else "" + description += " CLOSED" if self._is_closed else "" + return f"" + + def __del__(self): + if not self._is_closed: + import warnings + warnings.warn(f"NetworkStream was garbage collected without being closed.") + + def __enter__(self) -> "NetworkStream": + return self + + def __exit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None, + ): + self.close() + + +class NetworkListener: + def __init__(self, sock: socket.socket, address: tuple[str, int]) -> None: + self._server_socket = sock + self._address = address + self._is_closed = False + + @property + def host(self): + return self._address[0] + + @property + def port(self): + return self._address[1] + + def accept(self) -> NetworkStream | None: + """ + Blocks until an incoming connection is accepted, and returns the NetworkStream. + Stops blocking and returns `None` once the listener is closed. + """ + while not self._is_closed: + r, _, _ = select.select([self._server_socket], [], [], 3) + if r: + sock, address = self._server_socket.accept() + return NetworkStream(sock, address) + return None + + def close(self): + self._is_closed = True + self._server_socket.close() + + def __del__(self): + if not self._is_closed: + import warnings + warnings.warn("NetworkListener was garbage collected without being closed.") + + def __enter__(self) -> "NetworkListener": + return self + + def __exit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None, + ): + self.close() + + +class NetworkServer: + def __init__(self, listener: NetworkListener, handler: typing.Callable[[NetworkStream], None]) -> None: + self.listener = listener + self.handler = handler + self._max_workers = 5 + self._executor = None + self._thread = None + self._streams = list[NetworkStream] + + @property + def host(self): + return self.listener.host + + @property + def port(self): + return self.listener.port + + def __enter__(self): + self._executor = concurrent.futures.ThreadPoolExecutor(max_workers=self._max_workers) + self._executor.submit(self._serve) + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + self.listener.close() + self._executor.shutdown(wait=True) + + def _serve(self): + while stream := self.listener.accept(): + self._executor.submit(self._handler, stream) + + def _handler(self, stream): + try: + self.handler(stream) + finally: + stream.close() + + +class NetworkBackend: + def __init__(self, ssl_ctx: ssl.SSLContext | None = None): + self._ssl_ctx = self.create_default_context() if ssl_ctx is None else ssl_ctx + + def create_default_context(self) -> ssl.SSLContext: + import certifi + return ssl.create_default_context(cafile=certifi.where()) + + def connect(self, host: str, port: int) -> NetworkStream: + """ + Connect to the given address, returning a NetworkStream instance. + """ + address = (host, port) + timeout = get_current_timeout() + sock = socket.create_connection(address, timeout=timeout) + return NetworkStream(sock, address) + + def connect_tls(self, host: str, port: int, hostname: str = '') -> NetworkStream: + """ + Connect to the given address, returning a NetworkStream instance. + """ + address = (host, port) + hostname = hostname or host + timeout = get_current_timeout() + sock = socket.create_connection(address, timeout=timeout) + sock = self._ssl_ctx.wrap_socket(sock, server_hostname=hostname) + return NetworkStream(sock, address) + + def listen(self, host: str, port: int) -> NetworkListener: + """ + List on the given address, returning a NetworkListener instance. + """ + address = (host, port) + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + sock.bind(address) + sock.listen(5) + sock.setblocking(False) + return NetworkListener(sock, address) + + def serve(self, host: str, port: int, handler: typing.Callable[[NetworkStream], None]) -> NetworkServer: + listener = self.listen(host, port) + return NetworkServer(listener, handler) + + def __repr__(self): + return "" + + +Semaphore = threading.Semaphore +Lock = threading.Lock +sleep = time.sleep diff --git a/src/httpx/_parsers.py b/src/httpx/_parsers.py new file mode 100644 index 0000000000..830fccd901 --- /dev/null +++ b/src/httpx/_parsers.py @@ -0,0 +1,515 @@ +import enum + +from ._streams import Stream + +__all__ = ['HTTPParser', 'Mode', 'ProtocolError'] + + +# TODO... + +# * Upgrade +# * CONNECT + +# * Support 'Expect: 100 Continue' +# * Add 'Error' state transitions +# * Add tests to trickle data +# * Add type annotations + +# * Optional... HTTP/1.0 support +# * Read trailing headers on Transfer-Encoding: chunked. Not just '\r\n'. +# * When writing Transfer-Encoding: chunked, split large writes into buffer size. +# * When reading Transfer-Encoding: chunked, handle incomplete reads from large chunk sizes. +# * .read() doesn't document if will always return maximum available. + +# * validate method, target, protocol in request line +# * validate protocol, status_code, reason_phrase in response line +# * validate name, value on headers + + +class State(enum.Enum): + WAIT = 0 + SEND_METHOD_LINE = 1 + SEND_STATUS_LINE = 2 + SEND_HEADERS = 3 + SEND_BODY = 4 + RECV_METHOD_LINE = 5 + RECV_STATUS_LINE = 6 + RECV_HEADERS = 7 + RECV_BODY = 8 + DONE = 9 + CLOSED = 10 + + +class Mode(enum.Enum): + CLIENT = 0 + SERVER = 1 + + +# The usual transitions will be... + +# IDLE, IDLE +# SEND_HEADERS, IDLE +# SEND_BODY, IDLE +# DONE, IDLE +# DONE, SEND_HEADERS +# DONE, SEND_BODY +# DONE, DONE + +# Then either back to IDLE, IDLE +# or move to CLOSED, CLOSED + +# 1. It is also valid for the server to start +# sending the response without waiting for the +# complete request. +# 2. 1xx status codes are interim states, and +# transition from SEND_HEADERS back to IDLE +# 3. ... + +class ProtocolError(Exception): + pass + + +class HTTPParser: + """ + Usage... + + client = HTTPParser(writer, reader) + client.send_method_line() + client.send_headers() + client.send_body() + client.recv_status_line() + client.recv_headers() + client.recv_body() + client.complete() + client.close() + """ + def __init__(self, stream: Stream, mode: str) -> None: + self.stream = stream + self.parser = ReadAheadParser(stream) + self.mode = {'CLIENT': Mode.CLIENT, 'SERVER': Mode.SERVER}[mode] + + # Track state... + if self.mode == Mode.CLIENT: + self.send_state: State = State.SEND_METHOD_LINE + self.recv_state: State = State.WAIT + else: + self.recv_state = State.RECV_METHOD_LINE + self.send_state = State.WAIT + + # Track message framing... + self.send_content_length: int | None = 0 + self.recv_content_length: int | None = 0 + self.send_seen_length = 0 + self.recv_seen_length = 0 + + # Track connection keep alive... + self.send_keep_alive = True + self.recv_keep_alive = True + + # Special states... + self.processing_1xx = False + + def send_method_line(self, method: bytes, target: bytes, protocol: bytes) -> None: + """ + Send the initial request line: + + >>> p.send_method_line(b'GET', b'/', b'HTTP/1.1') + + Sending state will switch to SEND_HEADERS state. + """ + if self.send_state != State.SEND_METHOD_LINE: + msg = f"Called 'send_method_line' in invalid state {self.send_state}" + raise ProtocolError(msg) + + # Send initial request line, eg. "GET / HTTP/1.1" + if protocol != b'HTTP/1.1': + raise ProtocolError("Sent unsupported protocol version") + data = b" ".join([method, target, protocol]) + b"\r\n" + self.stream.write(data) + + self.send_state = State.SEND_HEADERS + self.recv_state = State.RECV_STATUS_LINE + + def send_status_line(self, protocol: bytes, status_code: int, reason: bytes) -> None: + """ + Send the initial response line: + + >>> p.send_method_line(b'HTTP/1.1', 200, b'OK') + + Sending state will switch to SEND_HEADERS state. + """ + if self.send_state != State.SEND_STATUS_LINE: + msg = f"Called 'send_status_line' in invalid state {self.send_state}" + raise ProtocolError(msg) + + # Send initial request line, eg. "GET / HTTP/1.1" + if protocol != b'HTTP/1.1': + raise ProtocolError("Sent unsupported protocol version") + status_code_bytes = str(status_code).encode('ascii') + data = b" ".join([protocol, status_code_bytes, reason]) + b"\r\n" + self.stream.write(data) + + self.send_state = State.SEND_HEADERS + + def send_headers(self, headers: list[tuple[bytes, bytes]]) -> None: + """ + Send the request headers: + + >>> p.send_headers([(b'Host', b'www.example.com')]) + + Sending state will switch to SEND_BODY state. + """ + if self.send_state != State.SEND_HEADERS: + msg = f"Called 'send_headers' in invalid state {self.send_state}" + raise ProtocolError(msg) + + # Update header state + seen_host = False + for name, value in headers: + lname = name.lower() + if lname == b'host': + seen_host = True + elif lname == b'content-length': + self.send_content_length = bounded_int( + value, + max_digits=20, + exc_text="Sent invalid Content-Length" + ) + elif lname == b'connection' and value == b'close': + self.send_keep_alive = False + elif lname == b'transfer-encoding' and value == b'chunked': + self.send_content_length = None + + if self.mode == Mode.CLIENT and not seen_host: + raise ProtocolError("Request missing 'Host' header") + + # Send request headers + lines = [name + b": " + value + b"\r\n" for name, value in headers] + data = b"".join(lines) + b"\r\n" + self.stream.write(data) + + self.send_state = State.SEND_BODY + + def send_body(self, body: bytes) -> None: + """ + Send the request body. An empty bytes argument indicates the end of the stream: + + >>> p.send_body(b'') + + Sending state will switch to DONE. + """ + if self.send_state != State.SEND_BODY: + msg = f"Called 'send_body' in invalid state {self.send_state}" + raise ProtocolError(msg) + + if self.send_content_length is None: + # Transfer-Encoding: chunked + self.send_seen_length += len(body) + marker = f'{len(body):x}\r\n'.encode('ascii') + self.stream.write(marker + body + b'\r\n') + + else: + # Content-Length: xxx + self.send_seen_length += len(body) + if self.send_seen_length > self.send_content_length: + msg = 'Too much data sent for declared Content-Length' + raise ProtocolError(msg) + if self.send_seen_length < self.send_content_length and body == b'': + msg = 'Not enough data sent for declared Content-Length' + raise ProtocolError(msg) + if body: + self.stream.write(body) + + if body == b'': + # Handle body close + self.send_state = State.DONE + + def recv_method_line(self) -> tuple[bytes, bytes, bytes]: + """ + Receive the initial request method line: + + >>> method, target, protocol = p.recv_status_line() + + Receive state will switch to RECV_HEADERS. + """ + if self.recv_state != State.RECV_METHOD_LINE: + msg = f"Called 'recv_method_line' in invalid state {self.recv_state}" + raise ProtocolError(msg) + + # Read initial response line, eg. "GET / HTTP/1.1" + exc_text = "reading request method line" + line = self.parser.read_until(b"\r\n", max_size=4096, exc_text=exc_text) + method, target, protocol = line.split(b" ", 2) + if protocol != b'HTTP/1.1': + raise ProtocolError("Received unsupported protocol version") + + self.recv_state = State.RECV_HEADERS + self.send_state = State.SEND_STATUS_LINE + return method, target, protocol + + def recv_status_line(self) -> tuple[bytes, int, bytes]: + """ + Receive the initial response status line: + + >>> protocol, status_code, reason_phrase = p.recv_status_line() + + Receive state will switch to RECV_HEADERS. + """ + if self.recv_state != State.RECV_STATUS_LINE: + msg = f"Called 'recv_status_line' in invalid state {self.recv_state}" + raise ProtocolError(msg) + + # Read initial response line, eg. "HTTP/1.1 200 OK" + exc_text = "reading response status line" + line = self.parser.read_until(b"\r\n", max_size=4096, exc_text=exc_text) + protocol, status_code_str, reason_phrase = line.split(b" ", 2) + if protocol != b'HTTP/1.1': + raise ProtocolError("Received unsupported protocol version") + + status_code = bounded_int( + status_code_str, + max_digits=3, + exc_text="Received invalid status code" + ) + if status_code < 100: + raise ProtocolError("Received invalid status code") + # 1xx status codes preceed the final response status code + self.processing_1xx = status_code < 200 + + self.recv_state = State.RECV_HEADERS + return protocol, status_code, reason_phrase + + def recv_headers(self) -> list[tuple[bytes, bytes]]: + """ + Receive the response headers: + + >>> headers = p.recv_status_line() + + Receive state will switch to RECV_BODY by default. + Receive state will revert to RECV_STATUS_CODE for interim 1xx responses. + """ + if self.recv_state != State.RECV_HEADERS: + msg = f"Called 'recv_headers' in invalid state {self.recv_state}" + raise ProtocolError(msg) + + # Read response headers + headers = [] + exc_text = "reading response headers" + while line := self.parser.read_until(b"\r\n", max_size=4096, exc_text=exc_text): + name, value = line.split(b":", 1) + value = value.strip(b" ") + headers.append((name, value)) + + # Update header state + seen_host = False + for name, value in headers: + lname = name.lower() + if lname == b'host': + seen_host = True + elif lname == b'content-length': + self.recv_content_length = bounded_int( + value, + max_digits=20, + exc_text="Received invalid Content-Length" + ) + elif lname == b'connection' and value == b'close': + self.recv_keep_alive = False + elif lname == b'transfer-encoding' and value == b'chunked': + self.recv_content_length = None + + if self.mode == Mode.SERVER and not seen_host: + raise ProtocolError("Request missing 'Host' header") + + if self.processing_1xx: + # 1xx status codes preceed the final response status code + self.processing_1xx = False + self.recv_state = State.RECV_STATUS_LINE + else: + self.recv_state = State.RECV_BODY + return headers + + def recv_body(self) -> bytes: + """ + Receive the response body. An empty byte string indicates the end of the stream: + + >>> buffer = bytearray() + >>> while body := p.recv_body() + >>> buffer.extend(body) + + The server will switch to DONE. + """ + if self.recv_state != State.RECV_BODY: + msg = f"Called 'recv_body' in invalid state {self.recv_state}" + raise ProtocolError(msg) + + if self.recv_content_length is None: + # Transfer-Encoding: chunked + exc_text = 'reading chunk size' + line = self.parser.read_until(b"\r\n", max_size=4096, exc_text=exc_text) + sizestr, _, _ = line.partition(b";") + + exc_text = "Received invalid chunk size" + size = bounded_hex(sizestr, max_digits=8, exc_text=exc_text) + if size > 0: + body = self.parser.read(size=size) + exc_text = 'reading chunk data' + self.parser.read_until(b"\r\n", max_size=2, exc_text=exc_text) + self.recv_seen_length += len(body) + else: + body = b'' + exc_text = 'reading chunk termination' + self.parser.read_until(b"\r\n", max_size=2, exc_text=exc_text) + + else: + # Content-Length: xxx + remaining = self.recv_content_length - self.recv_seen_length + size = min(remaining, 4096) + body = self.parser.read(size=size) + self.recv_seen_length += len(body) + if self.recv_seen_length < self.recv_content_length and body == b'': + msg = 'Not enough data received for declared Content-Length' + raise ProtocolError(msg) + + if body == b'': + # Handle body close + self.recv_state = State.DONE + return body + + def complete(self): + is_fully_complete = self.send_state == State.DONE and self.recv_state == State.DONE + is_keepalive = self.send_keep_alive and self.recv_keep_alive + + if not (is_fully_complete and is_keepalive): + self.close() + return + + if self.mode == Mode.CLIENT: + self.send_state = State.SEND_METHOD_LINE + self.recv_state = State.WAIT + else: + self.recv_state = State.RECV_METHOD_LINE + self.send_state = State.WAIT + + self.send_content_length = 0 + self.recv_content_length = 0 + self.send_seen_length = 0 + self.recv_seen_length = 0 + self.send_keep_alive = True + self.recv_keep_alive = True + self.processing_1xx = False + + def close(self): + if self.send_state != State.CLOSED: + self.send_state = State.CLOSED + self.recv_state = State.CLOSED + self.stream.close() + + def is_idle(self) -> bool: + return ( + self.send_state == State.SEND_METHOD_LINE or + self.recv_state == State.RECV_METHOD_LINE + ) + + def is_closed(self) -> bool: + return self.send_state == State.CLOSED + + def description(self) -> str: + return { + State.SEND_METHOD_LINE: "idle", + State.CLOSED: "closed", + }.get(self.send_state, "active") + + def __repr__(self) -> str: + cl_state = self.send_state.name + sr_state = self.recv_state.name + detail = f"client {cl_state}, server {sr_state}" + return f'' + + +class ReadAheadParser: + """ + A buffered I/O stream, with methods for read-ahead parsing. + """ + def __init__(self, stream: Stream) -> None: + self._buffer = b'' + self._stream = stream + self._chunk_size = 4096 + + def _read_some(self) -> bytes: + if self._buffer: + ret, self._buffer = self._buffer, b'' + return ret + return self._stream.read(self._chunk_size) + + def _push_back(self, buffer): + assert self._buffer == b'' + self._buffer = buffer + + def read(self, size: int) -> bytes: + """ + Read and return up to 'size' bytes from the stream, with I/O buffering provided. + + * Returns b'' to indicate connection close. + """ + buffer = bytearray() + while len(buffer) < size: + chunk = self._read_some() + if not chunk: + break + buffer.extend(chunk) + + if len(buffer) > size: + buffer, push_back = buffer[:size], buffer[size:] + self._push_back(bytes(push_back)) + return bytes(buffer) + + def read_until(self, marker: bytes, max_size: int, exc_text: str) -> bytes: + """ + Read and return bytes from the stream, delimited by marker. + + * The marker is not included in the return bytes. + * The marker is consumed from the I/O stream. + * Raises `ProtocolError` if the stream closes before a marker occurance. + * Raises `ProtocolError` if marker did not occur within 'max_size + len(marker)' bytes. + """ + buffer = bytearray() + while len(buffer) <= max_size: + chunk = self._read_some() + if not chunk: + # stream closed before marker found. + raise ProtocolError(f"Stream closed early {exc_text}") + start_search = max(len(buffer) - len(marker), 0) + buffer.extend(chunk) + index = buffer.find(marker, start_search) + + if index > max_size: + # marker was found, though 'max_size' exceeded. + raise ProtocolError(f"Exceeded maximum size {exc_text}") + elif index >= 0: + endindex = index + len(marker) + self._push_back(bytes(buffer[endindex:])) + return bytes(buffer[:index]) + + raise ProtocolError(f"Exceeded maximum size {exc_text}") + + +def bounded_int(intstr: bytes, max_digits: int, exc_text: str): + if len(intstr) > max_digits: + # Length of bytestring exceeds maximum. + raise ProtocolError(exc_text) + if len(intstr.strip(b'0123456789')) != 0: + # Contains invalid characters. + raise ProtocolError(exc_text) + + return int(intstr) + + +def bounded_hex(hexstr: bytes, max_digits: int, exc_text: str): + if len(hexstr) > max_digits: + # Length of bytestring exceeds maximum. + raise ProtocolError(exc_text) + if len(hexstr.strip(b'0123456789abcdefABCDEF')) != 0: + # Contains invalid characters. + raise ProtocolError(exc_text) + + return int(hexstr, base=16) diff --git a/src/httpx/_pool.py b/src/httpx/_pool.py new file mode 100644 index 0000000000..7193f8d81c --- /dev/null +++ b/src/httpx/_pool.py @@ -0,0 +1,284 @@ +import time +import typing +import types + +from ._content import Content +from ._headers import Headers +from ._network import Lock, NetworkBackend, Semaphore +from ._parsers import HTTPParser +from ._response import Response +from ._request import Request +from ._streams import HTTPStream, Stream +from ._urls import URL + + +__all__ = [ + "Transport", + "ConnectionPool", + "Connection", + "open_connection", +] + + +class Transport: + def send(self, request: Request) -> Response: + raise NotImplementedError() + + def close(self): + pass + + def request( + self, + method: str, + url: URL | str, + headers: Headers | dict[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + request = Request(method, url, headers=headers, content=content) + with self.send(request) as response: + response.read() + return response + + def stream( + self, + method: str, + url: URL | str, + headers: Headers | dict[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + request = Request(method, url, headers=headers, content=content) + response = self.send(request) + return response + + +class ConnectionPool(Transport): + def __init__(self, backend: NetworkBackend | None = None): + if backend is None: + backend = NetworkBackend() + + self._connections: list[Connection] = [] + self._network_backend = backend + self._limit_concurrency = Semaphore(100) + self._closed = False + + # Public API... + def send(self, request: Request) -> Response: + if self._closed: + raise RuntimeError("ConnectionPool is closed.") + + # TODO: concurrency limiting + self._cleanup() + connection = self._get_connection(request) + response = connection.send(request) + return response + + def close(self): + self._closed = True + closing = list(self._connections) + self._connections = [] + for conn in closing: + conn.close() + + # Create or reuse connections as required... + def _get_connection(self, request: Request) -> "Connection": + # Attempt to reuse an existing connection. + url = request.url + origin = URL(scheme=url.scheme, host=url.host, port=url.port) + now = time.monotonic() + for conn in self._connections: + if conn.origin() == origin and conn.is_idle() and not conn.is_expired(now): + return conn + + # Or else create a new connection. + conn = open_connection( + origin, + hostname=request.headers["Host"], + backend=self._network_backend + ) + self._connections.append(conn) + return conn + + # Connection pool management... + def _cleanup(self) -> None: + now = time.monotonic() + for conn in list(self._connections): + if conn.is_expired(now): + conn.close() + if conn.is_closed(): + self._connections.remove(conn) + + @property + def connections(self) -> typing.List['Connection']: + return [c for c in self._connections] + + def description(self) -> str: + counts = {"active": 0} + for status in [c.description() for c in self._connections]: + counts[status] = counts.get(status, 0) + 1 + return ", ".join(f"{count} {status}" for status, count in counts.items()) + + # Builtins... + def __repr__(self) -> str: + return f"" + + def __del__(self): + if not self._closed: + import warnings + warnings.warn("ConnectionPool was garbage collected without being closed.") + + def __enter__(self) -> "ConnectionPool": + return self + + def __exit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None, + ) -> None: + self.close() + + +class Connection(Transport): + def __init__(self, stream: Stream, origin: URL | str): + self._stream = stream + self._origin = URL(origin) + self._keepalive_duration = 5.0 + self._idle_expiry = time.monotonic() + self._keepalive_duration + self._request_lock = Lock() + self._parser = HTTPParser(stream, mode='CLIENT') + + # API for connection pool management... + def origin(self) -> URL: + return self._origin + + def is_idle(self) -> bool: + return self._parser.is_idle() + + def is_expired(self, when: float) -> bool: + return self._parser.is_idle() and when > self._idle_expiry + + def is_closed(self) -> bool: + return self._parser.is_closed() + + def description(self) -> str: + return self._parser.description() + + # API entry points... + def send(self, request: Request) -> Response: + #async with self._request_lock: + # try: + self._send_head(request) + self._send_body(request) + code, headers = self._recv_head() + stream = HTTPStream(self._recv_body, self._complete) + # TODO... + return Response(code, headers=headers, content=stream) + # finally: + # await self._cycle_complete() + + def close(self) -> None: + with self._request_lock: + self._close() + + # Top-level API for working directly with a connection. + def request( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + url = self._origin.join(url) + request = Request(method, url, headers=headers, content=content) + with self.send(request) as response: + response.read() + return response + + def stream( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ) -> Response: + url = self._origin.join(url) + request = Request(method, url, headers=headers, content=content) + return self.send(request) + + # Send the request... + def _send_head(self, request: Request) -> None: + method = request.method.encode('ascii') + target = request.url.target.encode('ascii') + protocol = b'HTTP/1.1' + self._parser.send_method_line(method, target, protocol) + headers = [ + (k.encode('ascii'), v.encode('ascii')) + for k, v in request.headers.items() + ] + self._parser.send_headers(headers) + + def _send_body(self, request: Request) -> None: + while data := request.stream.read(64 * 1024): + self._parser.send_body(data) + self._parser.send_body(b'') + + # Receive the response... + def _recv_head(self) -> tuple[int, Headers]: + _, code, _ = self._parser.recv_status_line() + h = self._parser.recv_headers() + headers = Headers([ + (k.decode('ascii'), v.decode('ascii')) + for k, v in h + ]) + return code, headers + + def _recv_body(self) -> bytes: + return self._parser.recv_body() + + # Request/response cycle complete... + def _complete(self) -> None: + self._parser.complete() + self._idle_expiry = time.monotonic() + self._keepalive_duration + + def _close(self) -> None: + self._parser.close() + + # Builtins... + def __repr__(self) -> str: + return f"" + + def __enter__(self) -> "Connection": + return self + + def __exit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None, + ): + self.close() + + +def open_connection( + url: URL | str, + hostname: str = '', + backend: NetworkBackend | None = None, + ) -> Connection: + + if isinstance(url, str): + url = URL(url) + + if url.scheme not in ("http", "https"): + raise ValueError("URL scheme must be 'http://' or 'https://'.") + if backend is None: + backend = NetworkBackend() + + host = url.host + port = url.port or {"http": 80, "https": 443}[url.scheme] + + if url.scheme == "https": + stream = backend.connect_tls(host, port, hostname) + else: + stream = backend.connect(host, port) + + return Connection(stream, url) diff --git a/src/httpx/_quickstart.py b/src/httpx/_quickstart.py new file mode 100644 index 0000000000..1a975301a3 --- /dev/null +++ b/src/httpx/_quickstart.py @@ -0,0 +1,49 @@ +import typing + +from ._client import Client +from ._content import Content +from ._headers import Headers +from ._streams import Stream +from ._urls import URL + + +__all__ = ['get', 'post', 'put', 'patch', 'delete'] + + +def get( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, +): + with Client() as client: + return client.request("GET", url=url, headers=headers) + +def post( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, +): + with Client() as client: + return client.request("POST", url, headers=headers, content=content) + +def put( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, +): + with Client() as client: + return client.request("PUT", url, headers=headers, content=content) + +def patch( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, +): + with Client() as client: + return client.request("PATCH", url, headers=headers, content=content) + +def delete( + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, +): + with Client() as client: + return client.request("DELETE", url=url, headers=headers) diff --git a/src/httpx/_request.py b/src/httpx/_request.py new file mode 100644 index 0000000000..1b739b1872 --- /dev/null +++ b/src/httpx/_request.py @@ -0,0 +1,93 @@ +import types +import typing + +from ._content import Content +from ._streams import ByteStream, Stream +from ._headers import Headers +from ._urls import URL + +__all__ = ["Request"] + + +class Request: + def __init__( + self, + method: str, + url: URL | str, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + self.method = method + self.url = URL(url) + self.headers = Headers(headers) + self.stream: Stream = ByteStream(b"") + + # https://datatracker.ietf.org/doc/html/rfc2616#section-14.23 + # RFC 2616, Section 14.23, Host. + # + # A client MUST include a Host header field in all HTTP/1.1 request messages. + if "Host" not in self.headers: + self.headers = self.headers.copy_set("Host", self.url.netloc) + + if content is not None: + if isinstance(content, bytes): + self.stream = ByteStream(content) + elif isinstance(content, Stream): + self.stream = content + elif isinstance(content, Content): + ct = content.content_type() + self.stream = content.encode() + self.headers = self.headers.copy_set("Content-Type", ct) + else: + raise TypeError(f'Expected `Content | Stream | bytes | None` got {type(content)}') + + # https://datatracker.ietf.org/doc/html/rfc2616#section-4.3 + # RFC 2616, Section 4.3, Message Body. + # + # The presence of a message-body in a request is signaled by the + # inclusion of a Content-Length or Transfer-Encoding header field in + # the request's message-headers. + content_length: int | None = self.stream.size + if content_length is None: + self.headers = self.headers.copy_set("Transfer-Encoding", "chunked") + elif content_length > 0: + self.headers = self.headers.copy_set("Content-Length", str(content_length)) + + elif method in ("POST", "PUT", "PATCH"): + # https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.2 + # RFC 7230, Section 3.3.2, Content Length. + # + # A user agent SHOULD send a Content-Length in a request message when no + # Transfer-Encoding is sent and the request method defines a meaning for + # an enclosed payload body. For example, a Content-Length header field is + # normally sent in a POST request even when the value is 0. + # (indicating an empty payload body). + self.headers = self.headers.copy_set("Content-Length", "0") + + @property + def body(self) -> bytes: + if not hasattr(self, '_body'): + raise RuntimeError("'.body' cannot be accessed without calling '.read()'") + return self._body + + def read(self) -> bytes: + if not hasattr(self, '_body'): + self._body = self.stream.read() + self.stream = ByteStream(self._body) + return self._body + + def close(self) -> None: + self.stream.close() + + def __enter__(self): + return self + + def __exit__(self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None + ): + self.close() + + def __repr__(self): + return f"" diff --git a/src/httpx/_response.py b/src/httpx/_response.py new file mode 100644 index 0000000000..abfec81029 --- /dev/null +++ b/src/httpx/_response.py @@ -0,0 +1,158 @@ +import types +import typing + +from ._content import Content +from ._streams import ByteStream, Stream +from ._headers import Headers, parse_opts_header + +__all__ = ["Response"] + +# We're using the same set as stdlib `http.HTTPStatus` here... +# +# https://github.com/python/cpython/blob/main/Lib/http/__init__.py +_codes = { + 100: "Continue", + 101: "Switching Protocols", + 102: "Processing", + 103: "Early Hints", + 200: "OK", + 201: "Created", + 202: "Accepted", + 203: "Non-Authoritative Information", + 204: "No Content", + 205: "Reset Content", + 206: "Partial Content", + 207: "Multi-Status", + 208: "Already Reported", + 226: "IM Used", + 300: "Multiple Choices", + 301: "Moved Permanently", + 302: "Found", + 303: "See Other", + 304: "Not Modified", + 305: "Use Proxy", + 307: "Temporary Redirect", + 308: "Permanent Redirect", + 400: "Bad Request", + 401: "Unauthorized", + 402: "Payment Required", + 403: "Forbidden", + 404: "Not Found", + 405: "Method Not Allowed", + 406: "Not Acceptable", + 407: "Proxy Authentication Required", + 408: "Request Timeout", + 409: "Conflict", + 410: "Gone", + 411: "Length Required", + 412: "Precondition Failed", + 413: "Content Too Large", + 414: "URI Too Long", + 415: "Unsupported Media Type", + 416: "Range Not Satisfiable", + 417: "Expectation Failed", + 418: "I'm a Teapot", + 421: "Misdirected Request", + 422: "Unprocessable Content", + 423: "Locked", + 424: "Failed Dependency", + 425: "Too Early", + 426: "Upgrade Required", + 428: "Precondition Required", + 429: "Too Many Requests", + 431: "Request Header Fields Too Large", + 451: "Unavailable For Legal Reasons", + 500: "Internal Server Error", + 501: "Not Implemented", + 502: "Bad Gateway", + 503: "Service Unavailable", + 504: "Gateway Timeout", + 505: "HTTP Version Not Supported", + 506: "Variant Also Negotiates", + 507: "Insufficient Storage", + 508: "Loop Detected", + 510: "Not Extended", + 511: "Network Authentication Required", +} + + +class Response: + def __init__( + self, + status_code: int, + *, + headers: Headers | typing.Mapping[str, str] | None = None, + content: Content | Stream | bytes | None = None, + ): + self.status_code = status_code + self.headers = Headers(headers) + self.stream: Stream = ByteStream(b"") + + if content is not None: + if isinstance(content, bytes): + self.stream = ByteStream(content) + elif isinstance(content, Stream): + self.stream = content + elif isinstance(content, Content): + ct = content.content_type() + self.stream = content.encode() + self.headers = self.headers.copy_set("Content-Type", ct) + else: + raise TypeError(f'Expected `Content | Stream | bytes | None` got {type(content)}') + + # https://datatracker.ietf.org/doc/html/rfc2616#section-4.3 + # RFC 2616, Section 4.3, Message Body. + # + # All 1xx (informational), 204 (no content), and 304 (not modified) responses + # MUST NOT include a message-body. All other responses do include a + # message-body, although it MAY be of zero length. + if status_code >= 200 and status_code != 204 and status_code != 304: + content_length: int | None = self.stream.size + if content_length is None: + self.headers = self.headers.copy_set("Transfer-Encoding", "chunked") + else: + self.headers = self.headers.copy_set("Content-Length", str(content_length)) + + @property + def reason_phrase(self): + return _codes.get(self.status_code, "Unknown Status Code") + + @property + def body(self) -> bytes: + if not hasattr(self, '_body'): + raise RuntimeError("'.body' cannot be accessed without calling '.read()'") + return self._body + + @property + def text(self) -> str: + if not hasattr(self, '_body'): + raise RuntimeError("'.text' cannot be accessed without calling '.read()'") + if not hasattr(self, '_text'): + ct = self.headers.get('Content-Type', '') + media, opts = parse_opts_header(ct) + charset = 'utf-8' + if media.startswith('text/'): + charset = opts.get('charset', 'utf-8') + self._text = self._body.decode(charset) + return self._text + + def read(self) -> bytes: + if not hasattr(self, '_body'): + self._body = self.stream.read() + return self._body + + def close(self) -> None: + self.stream.close() + + def __enter__(self): + return self + + def __exit__(self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None + ): + self.close() + + def __repr__(self): + return f"" diff --git a/src/httpx/_server.py b/src/httpx/_server.py new file mode 100644 index 0000000000..95226d9914 --- /dev/null +++ b/src/httpx/_server.py @@ -0,0 +1,126 @@ +import contextlib +import logging +import time + +from ._content import Text +from ._parsers import HTTPParser +from ._request import Request +from ._response import Response +from ._network import NetworkBackend, sleep +from ._streams import HTTPStream + +__all__ = [ + "serve_http", "run" +] + +logger = logging.getLogger("httpx.server") + + +class ConnectionClosed(Exception): + pass + + +class HTTPConnection: + def __init__(self, stream, endpoint): + self._stream = stream + self._endpoint = endpoint + self._parser = HTTPParser(stream, mode='SERVER') + self._keepalive_duration = 5.0 + self._idle_expiry = time.monotonic() + self._keepalive_duration + + # API entry points... + def handle_requests(self): + try: + while not self._parser.is_closed(): + method, url, headers = self._recv_head() + stream = HTTPStream(self._recv_body, self._complete) + # TODO: Handle endpoint exceptions + with Request(method, url, headers=headers, content=stream) as request: + try: + response = self._endpoint(request) + status_line = f"{request.method} {request.url.target} [{response.status_code} {response.reason_phrase}]" + logger.info(status_line) + except Exception: + logger.error("Internal Server Error", exc_info=True) + content = Text("Internal Server Error") + err = Response(code=500, content=content) + self._send_head(err) + self._send_body(err) + else: + self._send_head(response) + self._send_body(response) + except Exception: + logger.error("Internal Server Error", exc_info=True) + + def close(self): + self._parser.close() + + # Receive the request... + def _recv_head(self) -> tuple[str, str, list[tuple[str, str]]]: + method, target, _ = self._parser.recv_method_line() + m = method.decode('ascii') + t = target.decode('ascii') + headers = self._parser.recv_headers() + h = [ + (k.decode('latin-1'), v.decode('latin-1')) + for k, v in headers + ] + return m, t, h + + def _recv_body(self): + return self._parser.recv_body() + + # Return the response... + def _send_head(self, response: Response): + protocol = b"HTTP/1.1" + status = response.status_code + reason = response.reason_phrase.encode('ascii') + self._parser.send_status_line(protocol, status, reason) + headers = [ + (k.encode('ascii'), v.encode('ascii')) + for k, v in response.headers.items() + ] + self._parser.send_headers(headers) + + def _send_body(self, response: Response): + while data := response.stream.read(64 * 1024): + self._parser.send_body(data) + self._parser.send_body(b'') + + # Start it all over again... + def _complete(self): + self._parser.complete + self._idle_expiry = time.monotonic() + self._keepalive_duration + + +class HTTPServer: + def __init__(self, host, port): + self.url = f"http://{host}:{port}/" + + def wait(self): + while(True): + sleep(1) + + +@contextlib.contextmanager +def serve_http(endpoint): + def handler(stream): + connection = HTTPConnection(stream, endpoint) + connection.handle_requests() + + logging.basicConfig( + format="%(levelname)s [%(asctime)s] %(name)s - %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + level=logging.DEBUG + ) + + backend = NetworkBackend() + with backend.serve("127.0.0.1", 8080, handler) as server: + server = HTTPServer(server.host, server.port) + logger.info(f"Serving on {server.url} (Press CTRL+C to quit)") + yield server + + +def run(app): + with serve_http(app) as server: + server.wait() diff --git a/src/httpx/_streams.py b/src/httpx/_streams.py new file mode 100644 index 0000000000..c88a63f6f8 --- /dev/null +++ b/src/httpx/_streams.py @@ -0,0 +1,223 @@ +import io +import typing +import types +import os + + +class Stream: + def read(self, size: int=-1) -> bytes: + raise NotImplementedError() + + def write(self, data: bytes) -> None: + raise NotImplementedError() + + def close(self) -> None: + raise NotImplementedError() + + @property + def size(self) -> int | None: + return None + + def __enter__(self): + return self + + def __exit__( + self, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: types.TracebackType | None = None + ): + self.close() + + +class ByteStream(Stream): + def __init__(self, data: bytes = b''): + self._buffer = io.BytesIO(data) + self._size = len(data) + + def read(self, size: int=-1) -> bytes: + return self._buffer.read(size) + + def close(self) -> None: + self._buffer.close() + + @property + def size(self) -> int | None: + return self._size + + +class DuplexStream(Stream): + """ + DuplexStream supports both `read` and `write` operations, + which are applied to seperate buffers. + + This stream can be used for testing network parsers. + """ + + def __init__(self, data: bytes = b''): + self._read_buffer = io.BytesIO(data) + self._write_buffer = io.BytesIO() + + def read(self, size: int=-1) -> bytes: + return self._read_buffer.read(size) + + def write(self, buffer: bytes): + return self._write_buffer.write(buffer) + + def close(self) -> None: + self._read_buffer.close() + self._write_buffer.close() + + def input_bytes(self) -> bytes: + return self._read_buffer.getvalue() + + def output_bytes(self) -> bytes: + return self._write_buffer.getvalue() + + +class FileStream(Stream): + def __init__(self, path: str, fin: typing.Any) -> None: + self._path = path + self._fin = fin + + def read(self, size: int=-1) -> bytes: + return self._fin.read(size) + + def close(self) -> None: + self._fin.close() + + @property + def size(self) -> int | None: + return os.path.getsize(self._path) + + +class HTTPStream(Stream): + def __init__(self, next_chunk, complete): + self._next_chunk = next_chunk + self._complete = complete + self._buffer = io.BytesIO() + + def read(self, size=-1) -> bytes: + sections = [] + length = 0 + + # If we have any data in the buffer read that and clear the buffer. + buffered = self._buffer.read() + if buffered: + sections.append(buffered) + length += len(buffered) + self._buffer.seek(0) + self._buffer.truncate(0) + + # Read each chunk in turn. + while (size < 0) or (length < size): + section = self._next_chunk() + sections.append(section) + length += len(section) + if section == b'': + break + + # If we've more data than requested, then push some back into the buffer. + output = b''.join(sections) + if size > -1 and len(output) > size: + output, remainder = output[:size], output[size:] + self._buffer.write(remainder) + self._buffer.seek(0) + + return output + + def close(self) -> None: + self._buffer.close() + if self._complete is not None: + self._complete() + + +class MultiPartStream(Stream): + def __init__(self, form: list[tuple[str, str]], files: list[tuple[str, str]], boundary=''): + self._form = list(form) + self._files = list(files) + self._boundary = boundary or os.urandom(16).hex() + # Mutable state... + self._form_progress = list(self._form) + self._files_progress = list(self._files) + self._fin: typing.Any = None + self._complete = False + self._buffer = io.BytesIO() + + def read(self, size=-1) -> bytes: + sections = [] + length = 0 + + # If we have any data in the buffer read that and clear the buffer. + buffered = self._buffer.read() + if buffered: + sections.append(buffered) + length += len(buffered) + self._buffer.seek(0) + self._buffer.truncate(0) + + # Read each multipart section in turn. + while (size < 0) or (length < size): + section = self._read_next_section() + sections.append(section) + length += len(section) + if section == b'': + break + + # If we've more data than requested, then push some back into the buffer. + output = b''.join(sections) + if size > -1 and len(output) > size: + output, remainder = output[:size], output[size:] + self._buffer.write(remainder) + self._buffer.seek(0) + + return output + + def _read_next_section(self) -> bytes: + if self._form_progress: + # return a form item + key, value = self._form_progress.pop(0) + name = key.translate({10: "%0A", 13: "%0D", 34: "%22"}) + return ( + f"--{self._boundary}\r\n" + f'Content-Disposition: form-data; name="{name}"\r\n' + f"\r\n" + f"{value}\r\n" + ).encode("utf-8") + elif self._files_progress and self._fin is None: + # return start of a file item + key, value = self._files_progress.pop(0) + self._fin = open(value, 'rb') + name = key.translate({10: "%0A", 13: "%0D", 34: "%22"}) + filename = os.path.basename(value) + return ( + f"--{self._boundary}\r\n" + f'Content-Disposition: form-data; name="{name}"; filename="{filename}"\r\n' + f"\r\n" + ).encode("utf-8") + elif self._fin is not None: + chunk = self._fin.read(64*1024) + if chunk != b'': + # return some bytes from file + return chunk + else: + # return end of file item + self._fin.close() + self._fin = None + return b"\r\n" + elif not self._complete: + # return final section of multipart + self._complete = True + return f"--{self._boundary}--\r\n".encode("utf-8") + # return EOF marker + return b"" + + def close(self) -> None: + if self._fin is not None: + self._fin.close() + self._fin = None + self._buffer.close() + + @property + def size(self) -> int | None: + return None diff --git a/src/httpx/_urlencode.py b/src/httpx/_urlencode.py new file mode 100644 index 0000000000..1a83b620a6 --- /dev/null +++ b/src/httpx/_urlencode.py @@ -0,0 +1,85 @@ +import re + +__all__ = ["quote", "unquote", "urldecode", "urlencode"] + + +# Matchs a sequence of one or more '%xx' escapes. +PERCENT_ENCODED_REGEX = re.compile("(%[A-Fa-f0-9][A-Fa-f0-9])+") + +# https://datatracker.ietf.org/doc/html/rfc3986#section-2.3 +SAFE = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~" + + +def urlencode(multidict, safe=SAFE): + pairs = [] + for key, values in multidict.items(): + pairs.extend([(key, value) for value in values]) + + safe += "+" + pairs = [(k.replace(" ", "+"), v.replace(" ", "+")) for k, v in pairs] + + return "&".join( + f"{quote(key, safe)}={quote(val, safe)}" + for key, val in pairs + ) + + +def urldecode(string): + parts = [part.partition("=") for part in string.split("&") if part] + pairs = [ + (unquote(key), unquote(val)) + for key, _, val in parts + ] + + pairs = [(k.replace("+", " "), v.replace("+", " ")) for k, v in pairs] + + ret = {} + for k, v in pairs: + ret.setdefault(k, []).append(v) + return ret + + +def quote(string, safe=SAFE): + # Fast path if the string is already safe. + if not string.strip(safe): + return string + + # Replace any characters not in the safe set with '%xx' escape sequences. + return "".join([ + char if char in safe else percent(char) + for char in string + ]) + + +def unquote(string): + # Fast path if the string is not quoted. + if '%' not in string: + return string + + # Unquote. + parts = [] + current_position = 0 + for match in re.finditer(PERCENT_ENCODED_REGEX, string): + start_position, end_position = match.start(), match.end() + matched_text = match.group(0) + # Include any text up to the '%xx' escape sequence. + if start_position != current_position: + leading_text = string[current_position:start_position] + parts.append(leading_text) + + # Decode the '%xx' escape sequence. + hex = matched_text.replace('%', '') + decoded = bytes.fromhex(hex).decode('utf-8') + parts.append(decoded) + current_position = end_position + + # Include any text after the final '%xx' escape sequence. + if current_position != len(string): + trailing_text = string[current_position:] + parts.append(trailing_text) + + return "".join(parts) + + +def percent(c): + return ''.join(f"%{b:02X}" for b in c.encode("utf-8")) diff --git a/src/httpx/_urlparse.py b/src/httpx/_urlparse.py new file mode 100644 index 0000000000..612892fa8b --- /dev/null +++ b/src/httpx/_urlparse.py @@ -0,0 +1,534 @@ +""" +An implementation of `urlparse` that provides URL validation and normalization +as described by RFC3986. + +We rely on this implementation rather than the one in Python's stdlib, because: + +* It provides more complete URL validation. +* It properly differentiates between an empty querystring and an absent querystring, + to distinguish URLs with a trailing '?'. +* It handles scheme, hostname, port, and path normalization. +* It supports IDNA hostnames, normalizing them to their encoded form. +* The API supports passing individual components, as well as the complete URL string. + +Previously we relied on the excellent `rfc3986` package to handle URL parsing and +validation, but this module provides a simpler alternative, with less indirection +required. +""" + +import ipaddress +import re +import typing + + +class InvalidURL(ValueError): + pass + + +MAX_URL_LENGTH = 65536 + +# https://datatracker.ietf.org/doc/html/rfc3986.html#section-2.3 +UNRESERVED_CHARACTERS = ( + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~" +) +SUB_DELIMS = "!$&'()*+,;=" + +PERCENT_ENCODED_REGEX = re.compile("%[A-Fa-f0-9]{2}") + +# https://url.spec.whatwg.org/#percent-encoded-bytes + +# The fragment percent-encode set is the C0 control percent-encode set +# and U+0020 SPACE, U+0022 ("), U+003C (<), U+003E (>), and U+0060 (`). +FRAG_SAFE = "".join( + [chr(i) for i in range(0x20, 0x7F) if i not in (0x20, 0x22, 0x3C, 0x3E, 0x60)] +) + +# The query percent-encode set is the C0 control percent-encode set +# and U+0020 SPACE, U+0022 ("), U+0023 (#), U+003C (<), and U+003E (>). +QUERY_SAFE = "".join( + [chr(i) for i in range(0x20, 0x7F) if i not in (0x20, 0x22, 0x23, 0x3C, 0x3E)] +) + +# The path percent-encode set is the query percent-encode set +# and U+003F (?), U+0060 (`), U+007B ({), and U+007D (}). +PATH_SAFE = "".join( + [ + chr(i) + for i in range(0x20, 0x7F) + if i not in (0x20, 0x22, 0x23, 0x3C, 0x3E) + (0x3F, 0x60, 0x7B, 0x7D) + ] +) + +# The userinfo percent-encode set is the path percent-encode set +# and U+002F (/), U+003A (:), U+003B (;), U+003D (=), U+0040 (@), +# U+005B ([) to U+005E (^), inclusive, and U+007C (|). +USERNAME_SAFE = "".join( + [ + chr(i) + for i in range(0x20, 0x7F) + if i + not in (0x20, 0x22, 0x23, 0x3C, 0x3E) + + (0x3F, 0x60, 0x7B, 0x7D) + + (0x2F, 0x3A, 0x3B, 0x3D, 0x40, 0x5B, 0x5C, 0x5D, 0x5E, 0x7C) + ] +) +PASSWORD_SAFE = "".join( + [ + chr(i) + for i in range(0x20, 0x7F) + if i + not in (0x20, 0x22, 0x23, 0x3C, 0x3E) + + (0x3F, 0x60, 0x7B, 0x7D) + + (0x2F, 0x3A, 0x3B, 0x3D, 0x40, 0x5B, 0x5C, 0x5D, 0x5E, 0x7C) + ] +) +# Note... The terminology 'userinfo' percent-encode set in the WHATWG document +# is used for the username and password quoting. For the joint userinfo component +# we remove U+003A (:) from the safe set. +USERINFO_SAFE = "".join( + [ + chr(i) + for i in range(0x20, 0x7F) + if i + not in (0x20, 0x22, 0x23, 0x3C, 0x3E) + + (0x3F, 0x60, 0x7B, 0x7D) + + (0x2F, 0x3B, 0x3D, 0x40, 0x5B, 0x5C, 0x5D, 0x5E, 0x7C) + ] +) + + +# {scheme}: (optional) +# //{authority} (optional) +# {path} +# ?{query} (optional) +# #{fragment} (optional) +URL_REGEX = re.compile( + ( + r"(?:(?P{scheme}):)?" + r"(?://(?P{authority}))?" + r"(?P{path})" + r"(?:\?(?P{query}))?" + r"(?:#(?P{fragment}))?" + ).format( + scheme="([a-zA-Z][a-zA-Z0-9+.-]*)?", + authority="[^/?#]*", + path="[^?#]*", + query="[^#]*", + fragment=".*", + ) +) + +# {userinfo}@ (optional) +# {host} +# :{port} (optional) +AUTHORITY_REGEX = re.compile( + ( + r"(?:(?P{userinfo})@)?" r"(?P{host})" r":?(?P{port})?" + ).format( + userinfo=".*", # Any character sequence. + host="(\\[.*\\]|[^:@]*)", # Either any character sequence excluding ':' or '@', + # or an IPv6 address enclosed within square brackets. + port=".*", # Any character sequence. + ) +) + + +# If we call urlparse with an individual component, then we need to regex +# validate that component individually. +# Note that we're duplicating the same strings as above. Shock! Horror!! +COMPONENT_REGEX = { + "scheme": re.compile("([a-zA-Z][a-zA-Z0-9+.-]*)?"), + "authority": re.compile("[^/?#]*"), + "path": re.compile("[^?#]*"), + "query": re.compile("[^#]*"), + "fragment": re.compile(".*"), + "userinfo": re.compile("[^@]*"), + "host": re.compile("(\\[.*\\]|[^:]*)"), + "port": re.compile(".*"), +} + + +# We use these simple regexs as a first pass before handing off to +# the stdlib 'ipaddress' module for IP address validation. +IPv4_STYLE_HOSTNAME = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$") +IPv6_STYLE_HOSTNAME = re.compile(r"^\[.*\]$") + + +class ParseResult(typing.NamedTuple): + scheme: str + userinfo: str + host: str + port: int | None + path: str + query: str | None + fragment: str | None + + @property + def authority(self) -> str: + return "".join( + [ + f"{self.userinfo}@" if self.userinfo else "", + f"[{self.host}]" if ":" in self.host else self.host, + f":{self.port}" if self.port is not None else "", + ] + ) + + @property + def netloc(self) -> str: + return "".join( + [ + f"[{self.host}]" if ":" in self.host else self.host, + f":{self.port}" if self.port is not None else "", + ] + ) + + def copy_with(self, **kwargs: str | None) -> "ParseResult": + if not kwargs: + return self + + defaults = { + "scheme": self.scheme, + "authority": self.authority, + "path": self.path, + "query": self.query, + "fragment": self.fragment, + } + defaults.update(kwargs) + return urlparse("", **defaults) + + def __str__(self) -> str: + authority = self.authority + return "".join( + [ + f"{self.scheme}:" if self.scheme else "", + f"//{authority}" if authority else "", + self.path, + f"?{self.query}" if self.query is not None else "", + f"#{self.fragment}" if self.fragment is not None else "", + ] + ) + + +def urlparse(url: str = "", **kwargs: str | None) -> ParseResult: + # Initial basic checks on allowable URLs. + # --------------------------------------- + + # Hard limit the maximum allowable URL length. + if len(url) > MAX_URL_LENGTH: + raise InvalidURL("URL too long") + + # If a URL includes any ASCII control characters including \t, \r, \n, + # then treat it as invalid. + if any(char.isascii() and not char.isprintable() for char in url): + char = next(char for char in url if char.isascii() and not char.isprintable()) + idx = url.find(char) + error = ( + f"Invalid non-printable ASCII character in URL, {char!r} at position {idx}." + ) + raise InvalidURL(error) + + # Some keyword arguments require special handling. + # ------------------------------------------------ + + # Coerce "port" to a string, if it is provided as an integer. + if "port" in kwargs: + port = kwargs["port"] + kwargs["port"] = str(port) if isinstance(port, int) else port + + # Replace "netloc" with "host and "port". + if "netloc" in kwargs: + netloc = kwargs.pop("netloc") or "" + kwargs["host"], _, kwargs["port"] = netloc.partition(":") + + # Replace "username" and/or "password" with "userinfo". + if "username" in kwargs or "password" in kwargs: + username = quote(kwargs.pop("username", "") or "", safe=USERNAME_SAFE) + password = quote(kwargs.pop("password", "") or "", safe=PASSWORD_SAFE) + kwargs["userinfo"] = f"{username}:{password}" if password else username + + # Replace "raw_path" with "path" and "query". + if "raw_path" in kwargs: + raw_path = kwargs.pop("raw_path") or "" + kwargs["path"], seperator, kwargs["query"] = raw_path.partition("?") + if not seperator: + kwargs["query"] = None + + # Ensure that IPv6 "host" addresses are always escaped with "[...]". + if "host" in kwargs: + host = kwargs.get("host") or "" + if ":" in host and not (host.startswith("[") and host.endswith("]")): + kwargs["host"] = f"[{host}]" + + # If any keyword arguments are provided, ensure they are valid. + # ------------------------------------------------------------- + + for key, value in kwargs.items(): + if value is not None: + if len(value) > MAX_URL_LENGTH: + raise InvalidURL(f"URL component '{key}' too long") + + # If a component includes any ASCII control characters including \t, \r, \n, + # then treat it as invalid. + if any(char.isascii() and not char.isprintable() for char in value): + char = next( + char for char in value if char.isascii() and not char.isprintable() + ) + idx = value.find(char) + error = ( + f"Invalid non-printable ASCII character in URL {key} component, " + f"{char!r} at position {idx}." + ) + raise InvalidURL(error) + + # Ensure that keyword arguments match as a valid regex. + if not COMPONENT_REGEX[key].fullmatch(value): + raise InvalidURL(f"Invalid URL component '{key}'") + + # The URL_REGEX will always match, but may have empty components. + url_match = URL_REGEX.match(url) + assert url_match is not None + url_dict = url_match.groupdict() + + # * 'scheme', 'authority', and 'path' may be empty strings. + # * 'query' may be 'None', indicating no trailing "?" portion. + # Any string including the empty string, indicates a trailing "?". + # * 'fragment' may be 'None', indicating no trailing "#" portion. + # Any string including the empty string, indicates a trailing "#". + scheme = kwargs.get("scheme", url_dict["scheme"]) or "" + authority = kwargs.get("authority", url_dict["authority"]) or "" + path = kwargs.get("path", url_dict["path"]) or "" + query = kwargs.get("query", url_dict["query"]) + frag = kwargs.get("fragment", url_dict["fragment"]) + + # The AUTHORITY_REGEX will always match, but may have empty components. + authority_match = AUTHORITY_REGEX.match(authority) + assert authority_match is not None + authority_dict = authority_match.groupdict() + + # * 'userinfo' and 'host' may be empty strings. + # * 'port' may be 'None'. + userinfo = kwargs.get("userinfo", authority_dict["userinfo"]) or "" + host = kwargs.get("host", authority_dict["host"]) or "" + port = kwargs.get("port", authority_dict["port"]) + + # Normalize and validate each component. + # We end up with a parsed representation of the URL, + # with components that are plain ASCII bytestrings. + parsed_scheme: str = scheme.lower() + parsed_userinfo: str = quote(userinfo, safe=USERINFO_SAFE) + parsed_host: str = encode_host(host) + parsed_port: int | None = normalize_port(port, scheme) + + has_scheme = parsed_scheme != "" + has_authority = ( + parsed_userinfo != "" or parsed_host != "" or parsed_port is not None + ) + validate_path(path, has_scheme=has_scheme, has_authority=has_authority) + if has_scheme or has_authority: + path = normalize_path(path) + + parsed_path: str = quote(path, safe=PATH_SAFE) + parsed_query: str | None = None if query is None else quote(query, safe=QUERY_SAFE) + parsed_frag: str | None = None if frag is None else quote(frag, safe=FRAG_SAFE) + + # The parsed ASCII bytestrings are our canonical form. + # All properties of the URL are derived from these. + return ParseResult( + parsed_scheme, + parsed_userinfo, + parsed_host, + parsed_port, + parsed_path, + parsed_query, + parsed_frag, + ) + + +def encode_host(host: str) -> str: + if not host: + return "" + + elif IPv4_STYLE_HOSTNAME.match(host): + # Validate IPv4 hostnames like #.#.#.# + # + # From https://datatracker.ietf.org/doc/html/rfc3986/#section-3.2.2 + # + # IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet + try: + ipaddress.IPv4Address(host) + except ipaddress.AddressValueError: + raise InvalidURL(f"Invalid IPv4 address: {host!r}") + return host + + elif IPv6_STYLE_HOSTNAME.match(host): + # Validate IPv6 hostnames like [...] + # + # From https://datatracker.ietf.org/doc/html/rfc3986/#section-3.2.2 + # + # "A host identified by an Internet Protocol literal address, version 6 + # [RFC3513] or later, is distinguished by enclosing the IP literal + # within square brackets ("[" and "]"). This is the only place where + # square bracket characters are allowed in the URI syntax." + try: + ipaddress.IPv6Address(host[1:-1]) + except ipaddress.AddressValueError: + raise InvalidURL(f"Invalid IPv6 address: {host!r}") + return host[1:-1] + + elif not host.isascii(): + try: + import idna # type: ignore + except ImportError: + raise InvalidURL( + f"Cannot handle URL with IDNA hostname: {host!r}. " + f"Package 'idna' is not installed." + ) + + # IDNA hostnames + try: + return idna.encode(host.lower()).decode("ascii") + except idna.IDNAError: + raise InvalidURL(f"Invalid IDNA hostname: {host!r}") + + # Regular ASCII hostnames + # + # From https://datatracker.ietf.org/doc/html/rfc3986/#section-3.2.2 + # + # reg-name = *( unreserved / pct-encoded / sub-delims ) + WHATWG_SAFE = '"`{}%|\\' + return quote(host.lower(), safe=SUB_DELIMS + WHATWG_SAFE) + + +def normalize_port(port: str | int | None, scheme: str) -> int | None: + # From https://tools.ietf.org/html/rfc3986#section-3.2.3 + # + # "A scheme may define a default port. For example, the "http" scheme + # defines a default port of "80", corresponding to its reserved TCP + # port number. The type of port designated by the port number (e.g., + # TCP, UDP, SCTP) is defined by the URI scheme. URI producers and + # normalizers should omit the port component and its ":" delimiter if + # port is empty or if its value would be the same as that of the + # scheme's default." + if port is None or port == "": + return None + + try: + port_as_int = int(port) + except ValueError: + raise InvalidURL(f"Invalid port: {port!r}") + + # See https://url.spec.whatwg.org/#url-miscellaneous + default_port = {"ftp": 21, "http": 80, "https": 443, "ws": 80, "wss": 443}.get( + scheme + ) + if port_as_int == default_port: + return None + return port_as_int + + +def validate_path(path: str, has_scheme: bool, has_authority: bool) -> None: + """ + Path validation rules that depend on if the URL contains + a scheme or authority component. + + See https://datatracker.ietf.org/doc/html/rfc3986.html#section-3.3 + """ + if has_authority: + # If a URI contains an authority component, then the path component + # must either be empty or begin with a slash ("/") character." + if path and not path.startswith("/"): + raise InvalidURL("For absolute URLs, path must be empty or begin with '/'") + + if not has_scheme and not has_authority: + # If a URI does not contain an authority component, then the path cannot begin + # with two slash characters ("//"). + if path.startswith("//"): + raise InvalidURL("Relative URLs cannot have a path starting with '//'") + + # In addition, a URI reference (Section 4.1) may be a relative-path reference, + # in which case the first path segment cannot contain a colon (":") character. + if path.startswith(":"): + raise InvalidURL("Relative URLs cannot have a path starting with ':'") + + +def normalize_path(path: str) -> str: + """ + Drop "." and ".." segments from a URL path. + + For example: + + normalize_path("/path/./to/somewhere/..") == "/path/to" + """ + # Fast return when no '.' characters in the path. + if "." not in path: + return path + + components = path.split("/") + + # Fast return when no '.' or '..' components in the path. + if "." not in components and ".." not in components: + return path + + # https://datatracker.ietf.org/doc/html/rfc3986#section-5.2.4 + output: list[str] = [] + for component in components: + if component == ".": + pass + elif component == "..": + if output and output != [""]: + output.pop() + else: + output.append(component) + return "/".join(output) + + +def PERCENT(string: str) -> str: + return "".join([f"%{byte:02X}" for byte in string.encode("utf-8")]) + + +def percent_encoded(string: str, safe: str) -> str: + """ + Use percent-encoding to quote a string. + """ + NON_ESCAPED_CHARS = UNRESERVED_CHARACTERS + safe + + # Fast path for strings that don't need escaping. + if not string.rstrip(NON_ESCAPED_CHARS): + return string + + return "".join( + [char if char in NON_ESCAPED_CHARS else PERCENT(char) for char in string] + ) + + +def quote(string: str, safe: str) -> str: + """ + Use percent-encoding to quote a string, omitting existing '%xx' escape sequences. + + See: https://www.rfc-editor.org/rfc/rfc3986#section-2.1 + + * `string`: The string to be percent-escaped. + * `safe`: A string containing characters that may be treated as safe, and do not + need to be escaped. Unreserved characters are always treated as safe. + See: https://www.rfc-editor.org/rfc/rfc3986#section-2.3 + """ + parts = [] + current_position = 0 + for match in re.finditer(PERCENT_ENCODED_REGEX, string): + start_position, end_position = match.start(), match.end() + matched_text = match.group(0) + # Add any text up to the '%xx' escape sequence. + if start_position != current_position: + leading_text = string[current_position:start_position] + parts.append(percent_encoded(leading_text, safe=safe)) + + # Add the '%xx' escape sequence. + parts.append(matched_text) + current_position = end_position + + # Add any text after the final '%xx' escape sequence. + if current_position != len(string): + trailing_text = string[current_position:] + parts.append(percent_encoded(trailing_text, safe=safe)) + + return "".join(parts) diff --git a/src/httpx/_urls.py b/src/httpx/_urls.py new file mode 100644 index 0000000000..4ae4464e25 --- /dev/null +++ b/src/httpx/_urls.py @@ -0,0 +1,552 @@ +from __future__ import annotations + +import typing + +from ._urlparse import urlparse +from ._urlencode import unquote, urldecode, urlencode + +__all__ = ["QueryParams", "URL"] + + +class URL: + """ + url = httpx.URL("HTTPS://jo%40email.com:a%20secret@müller.de:1234/pa%20th?search=ab#anchorlink") + + assert url.scheme == "https" + assert url.username == "jo@email.com" + assert url.password == "a secret" + assert url.userinfo == b"jo%40email.com:a%20secret" + assert url.host == "müller.de" + assert url.raw_host == b"xn--mller-kva.de" + assert url.port == 1234 + assert url.netloc == b"xn--mller-kva.de:1234" + assert url.path == "/pa th" + assert url.query == b"?search=ab" + assert url.raw_path == b"/pa%20th?search=ab" + assert url.fragment == "anchorlink" + + The components of a URL are broken down like this: + + https://jo%40email.com:a%20secret@müller.de:1234/pa%20th?search=ab#anchorlink + [scheme] [ username ] [password] [ host ][port][ path ] [ query ] [fragment] + [ userinfo ] [ netloc ][ raw_path ] + + Note that: + + * `url.scheme` is normalized to always be lowercased. + + * `url.host` is normalized to always be lowercased. Internationalized domain + names are represented in unicode, without IDNA encoding applied. For instance: + + url = httpx.URL("http://中国.icom.museum") + assert url.host == "中国.icom.museum" + url = httpx.URL("http://xn--fiqs8s.icom.museum") + assert url.host == "中国.icom.museum" + + * `url.raw_host` is normalized to always be lowercased, and is IDNA encoded. + + url = httpx.URL("http://中国.icom.museum") + assert url.raw_host == b"xn--fiqs8s.icom.museum" + url = httpx.URL("http://xn--fiqs8s.icom.museum") + assert url.raw_host == b"xn--fiqs8s.icom.museum" + + * `url.port` is either None or an integer. URLs that include the default port for + "http", "https", "ws", "wss", and "ftp" schemes have their port + normalized to `None`. + + assert httpx.URL("http://example.com") == httpx.URL("http://example.com:80") + assert httpx.URL("http://example.com").port is None + assert httpx.URL("http://example.com:80").port is None + + * `url.userinfo` is raw bytes, without URL escaping. Usually you'll want to work + with `url.username` and `url.password` instead, which handle the URL escaping. + + * `url.raw_path` is raw bytes of both the path and query, without URL escaping. + This portion is used as the target when constructing HTTP requests. Usually you'll + want to work with `url.path` instead. + + * `url.query` is raw bytes, without URL escaping. A URL query string portion can + only be properly URL escaped when decoding the parameter names and values + themselves. + """ + + def __init__(self, url: "URL" | str = "", **kwargs: typing.Any) -> None: + if kwargs: + allowed = { + "scheme": str, + "username": str, + "password": str, + "userinfo": bytes, + "host": str, + "port": int, + "netloc": str, + "path": str, + "query": bytes, + "raw_path": bytes, + "fragment": str, + "params": object, + } + + # Perform type checking for all supported keyword arguments. + for key, value in kwargs.items(): + if key not in allowed: + message = f"{key!r} is an invalid keyword argument for URL()" + raise TypeError(message) + if value is not None and not isinstance(value, allowed[key]): + expected = allowed[key].__name__ + seen = type(value).__name__ + message = f"Argument {key!r} must be {expected} but got {seen}" + raise TypeError(message) + if isinstance(value, bytes): + kwargs[key] = value.decode("ascii") + + if "params" in kwargs: + # Replace any "params" keyword with the raw "query" instead. + # + # Ensure that empty params use `kwargs["query"] = None` rather + # than `kwargs["query"] = ""`, so that generated URLs do not + # include an empty trailing "?". + params = kwargs.pop("params") + kwargs["query"] = None if not params else str(QueryParams(params)) + + if isinstance(url, str): + self._uri_reference = urlparse(url, **kwargs) + elif isinstance(url, URL): + self._uri_reference = url._uri_reference.copy_with(**kwargs) + else: + raise TypeError( + "Invalid type for url. Expected str or httpx.URL," + f" got {type(url)}: {url!r}" + ) + + @property + def scheme(self) -> str: + """ + The URL scheme, such as "http", "https". + Always normalised to lowercase. + """ + return self._uri_reference.scheme + + @property + def userinfo(self) -> bytes: + """ + The URL userinfo as a raw bytestring. + For example: b"jo%40email.com:a%20secret". + """ + return self._uri_reference.userinfo.encode("ascii") + + @property + def username(self) -> str: + """ + The URL username as a string, with URL decoding applied. + For example: "jo@email.com" + """ + userinfo = self._uri_reference.userinfo + return unquote(userinfo.partition(":")[0]) + + @property + def password(self) -> str: + """ + The URL password as a string, with URL decoding applied. + For example: "a secret" + """ + userinfo = self._uri_reference.userinfo + return unquote(userinfo.partition(":")[2]) + + @property + def host(self) -> str: + """ + The URL host as a string. + Always normalized to lowercase. Possibly IDNA encoded. + + Examples: + + url = httpx.URL("http://www.EXAMPLE.org") + assert url.host == "www.example.org" + + url = httpx.URL("http://中国.icom.museum") + assert url.host == "xn--fiqs8s" + + url = httpx.URL("http://xn--fiqs8s.icom.museum") + assert url.host == "xn--fiqs8s" + + url = httpx.URL("https://[::ffff:192.168.0.1]") + assert url.host == "::ffff:192.168.0.1" + """ + return self._uri_reference.host + + @property + def port(self) -> int | None: + """ + The URL port as an integer. + + Note that the URL class performs port normalization as per the WHATWG spec. + Default ports for "http", "https", "ws", "wss", and "ftp" schemes are always + treated as `None`. + + For example: + + assert httpx.URL("http://www.example.com") == httpx.URL("http://www.example.com:80") + assert httpx.URL("http://www.example.com:80").port is None + """ + return self._uri_reference.port + + @property + def netloc(self) -> str: + """ + Either `` or `:` as bytes. + Always normalized to lowercase, and IDNA encoded. + + This property may be used for generating the value of a request + "Host" header. + """ + return self._uri_reference.netloc + + @property + def path(self) -> str: + """ + The URL path as a string. Excluding the query string, and URL decoded. + + For example: + + url = httpx.URL("https://example.com/pa%20th") + assert url.path == "/pa th" + """ + path = self._uri_reference.path or "/" + return unquote(path) + + @property + def query(self) -> bytes: + """ + The URL query string, as raw bytes, excluding the leading b"?". + + This is necessarily a bytewise interface, because we cannot + perform URL decoding of this representation until we've parsed + the keys and values into a QueryParams instance. + + For example: + + url = httpx.URL("https://example.com/?filter=some%20search%20terms") + assert url.query == b"filter=some%20search%20terms" + """ + query = self._uri_reference.query or "" + return query.encode("ascii") + + @property + def params(self) -> "QueryParams": + """ + The URL query parameters, neatly parsed and packaged into an immutable + multidict representation. + """ + return QueryParams(self._uri_reference.query) + + @property + def target(self) -> str: + """ + The complete URL path and query string as raw bytes. + Used as the target when constructing HTTP requests. + + For example: + + GET /users?search=some%20text HTTP/1.1 + Host: www.example.org + Connection: close + """ + target = self._uri_reference.path or "/" + if self._uri_reference.query is not None: + target += "?" + self._uri_reference.query + return target + + @property + def fragment(self) -> str: + """ + The URL fragments, as used in HTML anchors. + As a string, without the leading '#'. + """ + return unquote(self._uri_reference.fragment or "") + + @property + def is_absolute_url(self) -> bool: + """ + Return `True` for absolute URLs such as 'http://example.com/path', + and `False` for relative URLs such as '/path'. + """ + # We don't use `.is_absolute` from `rfc3986` because it treats + # URLs with a fragment portion as not absolute. + # What we actually care about is if the URL provides + # a scheme and hostname to which connections should be made. + return bool(self._uri_reference.scheme and self._uri_reference.host) + + @property + def is_relative_url(self) -> bool: + """ + Return `False` for absolute URLs such as 'http://example.com/path', + and `True` for relative URLs such as '/path'. + """ + return not self.is_absolute_url + + def copy_with(self, **kwargs: typing.Any) -> "URL": + """ + Copy this URL, returning a new URL with some components altered. + Accepts the same set of parameters as the components that are made + available via properties on the `URL` class. + + For example: + + url = httpx.URL("https://www.example.com").copy_with( + username="jo@gmail.com", password="a secret" + ) + assert url == "https://jo%40email.com:a%20secret@www.example.com" + """ + return URL(self, **kwargs) + + def copy_set_param(self, key: str, value: typing.Any = None) -> "URL": + return self.copy_with(params=self.params.copy_set(key, value)) + + def copy_append_param(self, key: str, value: typing.Any = None) -> "URL": + return self.copy_with(params=self.params.copy_append(key, value)) + + def copy_remove_param(self, key: str) -> "URL": + return self.copy_with(params=self.params.copy_remove(key)) + + def copy_merge_params( + self, + params: "QueryParams" | dict[str, str | list[str]] | list[tuple[str, str]] | None, + ) -> "URL": + return self.copy_with(params=self.params.copy_update(params)) + + def join(self, url: "URL" | str) -> "URL": + """ + Return an absolute URL, using this URL as the base. + + Eg. + + url = httpx.URL("https://www.example.com/test") + url = url.join("/new/path") + assert url == "https://www.example.com/new/path" + """ + from urllib.parse import urljoin + + return URL(urljoin(str(self), str(URL(url)))) + + def __hash__(self) -> int: + return hash(str(self)) + + def __eq__(self, other: typing.Any) -> bool: + return isinstance(other, (URL, str)) and str(self) == str(URL(other)) + + def __str__(self) -> str: + return str(self._uri_reference) + + def __repr__(self) -> str: + return f"" + + +class QueryParams(typing.Mapping[str, str]): + """ + URL query parameters, as a multi-dict. + """ + + def __init__( + self, + params: ( + "QueryParams" | dict[str, str | list[str]] | list[tuple[str, str]] | str | None + ) = None, + ) -> None: + d: dict[str, list[str]] = {} + + if params is None: + d = {} + elif isinstance(params, str): + d = urldecode(params) + elif isinstance(params, QueryParams): + d = params.multi_dict() + elif isinstance(params, dict): + # Convert dict inputs like: + # {"a": "123", "b": ["456", "789"]} + # To dict inputs where values are always lists, like: + # {"a": ["123"], "b": ["456", "789"]} + d = {k: [v] if isinstance(v, str) else list(v) for k, v in params.items()} + else: + # Convert list inputs like: + # [("a", "123"), ("a", "456"), ("b", "789")] + # To a dict representation, like: + # {"a": ["123", "456"], "b": ["789"]} + for k, v in params: + d.setdefault(k, []).append(v) + + self._dict = d + + def keys(self) -> typing.KeysView[str]: + """ + Return all the keys in the query params. + + Usage: + + q = httpx.QueryParams("a=123&a=456&b=789") + assert list(q.keys()) == ["a", "b"] + """ + return self._dict.keys() + + def values(self) -> typing.ValuesView[str]: + """ + Return all the values in the query params. If a key occurs more than once + only the first item for that key is returned. + + Usage: + + q = httpx.QueryParams("a=123&a=456&b=789") + assert list(q.values()) == ["123", "789"] + """ + return {k: v[0] for k, v in self._dict.items()}.values() + + def items(self) -> typing.ItemsView[str, str]: + """ + Return all items in the query params. If a key occurs more than once + only the first item for that key is returned. + + Usage: + + q = httpx.QueryParams("a=123&a=456&b=789") + assert list(q.items()) == [("a", "123"), ("b", "789")] + """ + return {k: v[0] for k, v in self._dict.items()}.items() + + def multi_items(self) -> list[tuple[str, str]]: + """ + Return all items in the query params. Allow duplicate keys to occur. + + Usage: + + q = httpx.QueryParams("a=123&a=456&b=789") + assert list(q.multi_items()) == [("a", "123"), ("a", "456"), ("b", "789")] + """ + multi_items: list[tuple[str, str]] = [] + for k, v in self._dict.items(): + multi_items.extend([(k, i) for i in v]) + return multi_items + + def multi_dict(self) -> dict[str, list[str]]: + return {k: list(v) for k, v in self._dict.items()} + + def get(self, key: str, default: typing.Any = None) -> typing.Any: + """ + Get a value from the query param for a given key. If the key occurs + more than once, then only the first value is returned. + + Usage: + + q = httpx.QueryParams("a=123&a=456&b=789") + assert q.get("a") == "123" + """ + if key in self._dict: + return self._dict[key][0] + return default + + def get_list(self, key: str) -> list[str]: + """ + Get all values from the query param for a given key. + + Usage: + + q = httpx.QueryParams("a=123&a=456&b=789") + assert q.get_list("a") == ["123", "456"] + """ + return list(self._dict.get(key, [])) + + def copy_set(self, key: str, value: str) -> "QueryParams": + """ + Return a new QueryParams instance, setting the value of a key. + + Usage: + + q = httpx.QueryParams("a=123") + q = q.set("a", "456") + assert q == httpx.QueryParams("a=456") + """ + q = QueryParams() + q._dict = dict(self._dict) + q._dict[key] = [value] + return q + + def copy_append(self, key: str, value: str) -> "QueryParams": + """ + Return a new QueryParams instance, setting or appending the value of a key. + + Usage: + + q = httpx.QueryParams("a=123") + q = q.append("a", "456") + assert q == httpx.QueryParams("a=123&a=456") + """ + q = QueryParams() + q._dict = dict(self._dict) + q._dict[key] = q.get_list(key) + [value] + return q + + def copy_remove(self, key: str) -> QueryParams: + """ + Return a new QueryParams instance, removing the value of a key. + + Usage: + + q = httpx.QueryParams("a=123") + q = q.remove("a") + assert q == httpx.QueryParams("") + """ + q = QueryParams() + q._dict = dict(self._dict) + q._dict.pop(str(key), None) + return q + + def copy_update( + self, + params: ( + "QueryParams" | dict[str, str | list[str]] | list[tuple[str, str]] | None + ) = None, + ) -> "QueryParams": + """ + Return a new QueryParams instance, updated with. + + Usage: + + q = httpx.QueryParams("a=123") + q = q.copy_update({"b": "456"}) + assert q == httpx.QueryParams("a=123&b=456") + + q = httpx.QueryParams("a=123") + q = q.copy_update({"a": "456", "b": "789"}) + assert q == httpx.QueryParams("a=456&b=789") + """ + q = QueryParams(params) + q._dict = {**self._dict, **q._dict} + return q + + def __getitem__(self, key: str) -> str: + return self._dict[key][0] + + def __contains__(self, key: typing.Any) -> bool: + return key in self._dict + + def __iter__(self) -> typing.Iterator[str]: + return iter(self.keys()) + + def __len__(self) -> int: + return len(self._dict) + + def __bool__(self) -> bool: + return bool(self._dict) + + def __hash__(self) -> int: + return hash(str(self)) + + def __eq__(self, other: typing.Any) -> bool: + if not isinstance(other, self.__class__): + return False + return sorted(self.multi_items()) == sorted(other.multi_items()) + + def __str__(self) -> str: + return urlencode(self.multi_dict()) + + def __repr__(self) -> str: + return f"" diff --git a/tests/client/__init__.py b/tests/client/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/tests/client/test_async_client.py b/tests/client/test_async_client.py deleted file mode 100644 index 8d7eaa3c58..0000000000 --- a/tests/client/test_async_client.py +++ /dev/null @@ -1,375 +0,0 @@ -from __future__ import annotations - -import typing -from datetime import timedelta - -import pytest - -import httpx - - -@pytest.mark.anyio -async def test_get(server): - url = server.url - async with httpx.AsyncClient(http2=True) as client: - response = await client.get(url) - assert response.status_code == 200 - assert response.text == "Hello, world!" - assert response.http_version == "HTTP/1.1" - assert response.headers - assert repr(response) == "" - assert response.elapsed > timedelta(seconds=0) - - -@pytest.mark.parametrize( - "url", - [ - pytest.param("invalid://example.org", id="scheme-not-http(s)"), - pytest.param("://example.org", id="no-scheme"), - pytest.param("http://", id="no-host"), - ], -) -@pytest.mark.anyio -async def test_get_invalid_url(server, url): - async with httpx.AsyncClient() as client: - with pytest.raises((httpx.UnsupportedProtocol, httpx.LocalProtocolError)): - await client.get(url) - - -@pytest.mark.anyio -async def test_build_request(server): - url = server.url.copy_with(path="/echo_headers") - headers = {"Custom-header": "value"} - async with httpx.AsyncClient() as client: - request = client.build_request("GET", url) - request.headers.update(headers) - response = await client.send(request) - - assert response.status_code == 200 - assert response.url == url - - assert response.json()["Custom-header"] == "value" - - -@pytest.mark.anyio -async def test_post(server): - url = server.url - async with httpx.AsyncClient() as client: - response = await client.post(url, content=b"Hello, world!") - assert response.status_code == 200 - - -@pytest.mark.anyio -async def test_post_json(server): - url = server.url - async with httpx.AsyncClient() as client: - response = await client.post(url, json={"text": "Hello, world!"}) - assert response.status_code == 200 - - -@pytest.mark.anyio -async def test_stream_response(server): - async with httpx.AsyncClient() as client: - async with client.stream("GET", server.url) as response: - body = await response.aread() - - assert response.status_code == 200 - assert body == b"Hello, world!" - assert response.content == b"Hello, world!" - - -@pytest.mark.anyio -async def test_access_content_stream_response(server): - async with httpx.AsyncClient() as client: - async with client.stream("GET", server.url) as response: - pass - - assert response.status_code == 200 - with pytest.raises(httpx.ResponseNotRead): - response.content # noqa: B018 - - -@pytest.mark.anyio -async def test_stream_request(server): - async def hello_world() -> typing.AsyncIterator[bytes]: - yield b"Hello, " - yield b"world!" - - async with httpx.AsyncClient() as client: - response = await client.post(server.url, content=hello_world()) - assert response.status_code == 200 - - -@pytest.mark.anyio -async def test_cannot_stream_sync_request(server): - def hello_world() -> typing.Iterator[bytes]: # pragma: no cover - yield b"Hello, " - yield b"world!" - - async with httpx.AsyncClient() as client: - with pytest.raises(RuntimeError): - await client.post(server.url, content=hello_world()) - - -@pytest.mark.anyio -async def test_raise_for_status(server): - async with httpx.AsyncClient() as client: - for status_code in (200, 400, 404, 500, 505): - response = await client.request( - "GET", server.url.copy_with(path=f"/status/{status_code}") - ) - - if 400 <= status_code < 600: - with pytest.raises(httpx.HTTPStatusError) as exc_info: - response.raise_for_status() - assert exc_info.value.response == response - else: - assert response.raise_for_status() is response - - -@pytest.mark.anyio -async def test_options(server): - async with httpx.AsyncClient() as client: - response = await client.options(server.url) - assert response.status_code == 200 - assert response.text == "Hello, world!" - - -@pytest.mark.anyio -async def test_head(server): - async with httpx.AsyncClient() as client: - response = await client.head(server.url) - assert response.status_code == 200 - assert response.text == "" - - -@pytest.mark.anyio -async def test_put(server): - async with httpx.AsyncClient() as client: - response = await client.put(server.url, content=b"Hello, world!") - assert response.status_code == 200 - - -@pytest.mark.anyio -async def test_patch(server): - async with httpx.AsyncClient() as client: - response = await client.patch(server.url, content=b"Hello, world!") - assert response.status_code == 200 - - -@pytest.mark.anyio -async def test_delete(server): - async with httpx.AsyncClient() as client: - response = await client.delete(server.url) - assert response.status_code == 200 - assert response.text == "Hello, world!" - - -@pytest.mark.anyio -async def test_100_continue(server): - headers = {"Expect": "100-continue"} - content = b"Echo request body" - - async with httpx.AsyncClient() as client: - response = await client.post( - server.url.copy_with(path="/echo_body"), headers=headers, content=content - ) - - assert response.status_code == 200 - assert response.content == content - - -@pytest.mark.anyio -async def test_context_managed_transport(): - class Transport(httpx.AsyncBaseTransport): - def __init__(self) -> None: - self.events: list[str] = [] - - async def aclose(self): - # The base implementation of httpx.AsyncBaseTransport just - # calls into `.aclose`, so simple transport cases can just override - # this method for any cleanup, where more complex cases - # might want to additionally override `__aenter__`/`__aexit__`. - self.events.append("transport.aclose") - - async def __aenter__(self): - await super().__aenter__() - self.events.append("transport.__aenter__") - - async def __aexit__(self, *args): - await super().__aexit__(*args) - self.events.append("transport.__aexit__") - - transport = Transport() - async with httpx.AsyncClient(transport=transport): - pass - - assert transport.events == [ - "transport.__aenter__", - "transport.aclose", - "transport.__aexit__", - ] - - -@pytest.mark.anyio -async def test_context_managed_transport_and_mount(): - class Transport(httpx.AsyncBaseTransport): - def __init__(self, name: str) -> None: - self.name: str = name - self.events: list[str] = [] - - async def aclose(self): - # The base implementation of httpx.AsyncBaseTransport just - # calls into `.aclose`, so simple transport cases can just override - # this method for any cleanup, where more complex cases - # might want to additionally override `__aenter__`/`__aexit__`. - self.events.append(f"{self.name}.aclose") - - async def __aenter__(self): - await super().__aenter__() - self.events.append(f"{self.name}.__aenter__") - - async def __aexit__(self, *args): - await super().__aexit__(*args) - self.events.append(f"{self.name}.__aexit__") - - transport = Transport(name="transport") - mounted = Transport(name="mounted") - async with httpx.AsyncClient( - transport=transport, mounts={"http://www.example.org": mounted} - ): - pass - - assert transport.events == [ - "transport.__aenter__", - "transport.aclose", - "transport.__aexit__", - ] - assert mounted.events == [ - "mounted.__aenter__", - "mounted.aclose", - "mounted.__aexit__", - ] - - -def hello_world(request): - return httpx.Response(200, text="Hello, world!") - - -@pytest.mark.anyio -async def test_client_closed_state_using_implicit_open(): - client = httpx.AsyncClient(transport=httpx.MockTransport(hello_world)) - - assert not client.is_closed - await client.get("http://example.com") - - assert not client.is_closed - await client.aclose() - - assert client.is_closed - # Once we're close we cannot make any more requests. - with pytest.raises(RuntimeError): - await client.get("http://example.com") - - # Once we're closed we cannot reopen the client. - with pytest.raises(RuntimeError): - async with client: - pass # pragma: no cover - - -@pytest.mark.anyio -async def test_client_closed_state_using_with_block(): - async with httpx.AsyncClient(transport=httpx.MockTransport(hello_world)) as client: - assert not client.is_closed - await client.get("http://example.com") - - assert client.is_closed - with pytest.raises(RuntimeError): - await client.get("http://example.com") - - -def unmounted(request: httpx.Request) -> httpx.Response: - data = {"app": "unmounted"} - return httpx.Response(200, json=data) - - -def mounted(request: httpx.Request) -> httpx.Response: - data = {"app": "mounted"} - return httpx.Response(200, json=data) - - -@pytest.mark.anyio -async def test_mounted_transport(): - transport = httpx.MockTransport(unmounted) - mounts = {"custom://": httpx.MockTransport(mounted)} - - async with httpx.AsyncClient(transport=transport, mounts=mounts) as client: - response = await client.get("https://www.example.com") - assert response.status_code == 200 - assert response.json() == {"app": "unmounted"} - - response = await client.get("custom://www.example.com") - assert response.status_code == 200 - assert response.json() == {"app": "mounted"} - - -@pytest.mark.anyio -async def test_async_mock_transport(): - async def hello_world(request: httpx.Request) -> httpx.Response: - return httpx.Response(200, text="Hello, world!") - - transport = httpx.MockTransport(hello_world) - - async with httpx.AsyncClient(transport=transport) as client: - response = await client.get("https://www.example.com") - assert response.status_code == 200 - assert response.text == "Hello, world!" - - -@pytest.mark.anyio -async def test_cancellation_during_stream(): - """ - If any BaseException is raised during streaming the response, then the - stream should be closed. - - This includes: - - * `asyncio.CancelledError` (A subclass of BaseException from Python 3.8 onwards.) - * `trio.Cancelled` - * `KeyboardInterrupt` - * `SystemExit` - - See https://github.com/encode/httpx/issues/2139 - """ - stream_was_closed = False - - def response_with_cancel_during_stream(request): - class CancelledStream(httpx.AsyncByteStream): - async def __aiter__(self) -> typing.AsyncIterator[bytes]: - yield b"Hello" - raise KeyboardInterrupt() - yield b", world" # pragma: no cover - - async def aclose(self) -> None: - nonlocal stream_was_closed - stream_was_closed = True - - return httpx.Response( - 200, headers={"Content-Length": "12"}, stream=CancelledStream() - ) - - transport = httpx.MockTransport(response_with_cancel_during_stream) - - async with httpx.AsyncClient(transport=transport) as client: - with pytest.raises(KeyboardInterrupt): - await client.get("https://www.example.com") - assert stream_was_closed - - -@pytest.mark.anyio -async def test_server_extensions(server): - url = server.url - async with httpx.AsyncClient(http2=True) as client: - response = await client.get(url) - assert response.status_code == 200 - assert response.extensions["http_version"] == b"HTTP/1.1" diff --git a/tests/client/test_auth.py b/tests/client/test_auth.py deleted file mode 100644 index 72674e6f4b..0000000000 --- a/tests/client/test_auth.py +++ /dev/null @@ -1,772 +0,0 @@ -""" -Integration tests for authentication. - -Unit tests for auth classes also exist in tests/test_auth.py -""" - -import hashlib -import netrc -import os -import sys -import threading -import typing -from urllib.request import parse_keqv_list - -import anyio -import pytest - -import httpx - -from ..common import FIXTURES_DIR - - -class App: - """ - A mock app to test auth credentials. - """ - - def __init__(self, auth_header: str = "", status_code: int = 200) -> None: - self.auth_header = auth_header - self.status_code = status_code - - def __call__(self, request: httpx.Request) -> httpx.Response: - headers = {"www-authenticate": self.auth_header} if self.auth_header else {} - data = {"auth": request.headers.get("Authorization")} - return httpx.Response(self.status_code, headers=headers, json=data) - - -class DigestApp: - def __init__( - self, - algorithm: str = "SHA-256", - send_response_after_attempt: int = 1, - qop: str = "auth", - regenerate_nonce: bool = True, - ) -> None: - self.algorithm = algorithm - self.send_response_after_attempt = send_response_after_attempt - self.qop = qop - self._regenerate_nonce = regenerate_nonce - self._response_count = 0 - - def __call__(self, request: httpx.Request) -> httpx.Response: - if self._response_count < self.send_response_after_attempt: - return self.challenge_send(request) - - data = {"auth": request.headers.get("Authorization")} - return httpx.Response(200, json=data) - - def challenge_send(self, request: httpx.Request) -> httpx.Response: - self._response_count += 1 - nonce = ( - hashlib.sha256(os.urandom(8)).hexdigest() - if self._regenerate_nonce - else "ee96edced2a0b43e4869e96ebe27563f369c1205a049d06419bb51d8aeddf3d3" - ) - challenge_data = { - "nonce": nonce, - "qop": self.qop, - "opaque": ( - "ee6378f3ee14ebfd2fff54b70a91a7c9390518047f242ab2271380db0e14bda1" - ), - "algorithm": self.algorithm, - "stale": "FALSE", - } - challenge_str = ", ".join( - '{}="{}"'.format(key, value) - for key, value in challenge_data.items() - if value - ) - - headers = { - "www-authenticate": f'Digest realm="httpx@example.org", {challenge_str}', - } - return httpx.Response(401, headers=headers) - - -class RepeatAuth(httpx.Auth): - """ - A mock authentication scheme that requires clients to send - the request a fixed number of times, and then send a last request containing - an aggregation of nonces that the server sent in 'WWW-Authenticate' headers - of intermediate responses. - """ - - requires_request_body = True - - def __init__(self, repeat: int) -> None: - self.repeat = repeat - - def auth_flow( - self, request: httpx.Request - ) -> typing.Generator[httpx.Request, httpx.Response, None]: - nonces = [] - - for index in range(self.repeat): - request.headers["Authorization"] = f"Repeat {index}" - response = yield request - nonces.append(response.headers["www-authenticate"]) - - key = ".".join(nonces) - request.headers["Authorization"] = f"Repeat {key}" - yield request - - -class ResponseBodyAuth(httpx.Auth): - """ - A mock authentication scheme that requires clients to send an 'Authorization' - header, then send back the contents of the response in the 'Authorization' - header. - """ - - requires_response_body = True - - def __init__(self, token: str) -> None: - self.token = token - - def auth_flow( - self, request: httpx.Request - ) -> typing.Generator[httpx.Request, httpx.Response, None]: - request.headers["Authorization"] = self.token - response = yield request - data = response.text - request.headers["Authorization"] = data - yield request - - -class SyncOrAsyncAuth(httpx.Auth): - """ - A mock authentication scheme that uses a different implementation for the - sync and async cases. - """ - - def __init__(self) -> None: - self._lock = threading.Lock() - self._async_lock = anyio.Lock() - - def sync_auth_flow( - self, request: httpx.Request - ) -> typing.Generator[httpx.Request, httpx.Response, None]: - with self._lock: - request.headers["Authorization"] = "sync-auth" - yield request - - async def async_auth_flow( - self, request: httpx.Request - ) -> typing.AsyncGenerator[httpx.Request, httpx.Response]: - async with self._async_lock: - request.headers["Authorization"] = "async-auth" - yield request - - -@pytest.mark.anyio -async def test_basic_auth() -> None: - url = "https://example.org/" - auth = ("user", "password123") - app = App() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": "Basic dXNlcjpwYXNzd29yZDEyMw=="} - - -@pytest.mark.anyio -async def test_basic_auth_with_stream() -> None: - """ - See: https://github.com/encode/httpx/pull/1312 - """ - url = "https://example.org/" - auth = ("user", "password123") - app = App() - - async with httpx.AsyncClient( - transport=httpx.MockTransport(app), auth=auth - ) as client: - async with client.stream("GET", url) as response: - await response.aread() - - assert response.status_code == 200 - assert response.json() == {"auth": "Basic dXNlcjpwYXNzd29yZDEyMw=="} - - -@pytest.mark.anyio -async def test_basic_auth_in_url() -> None: - url = "https://user:password123@example.org/" - app = App() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url) - - assert response.status_code == 200 - assert response.json() == {"auth": "Basic dXNlcjpwYXNzd29yZDEyMw=="} - - -@pytest.mark.anyio -async def test_basic_auth_on_session() -> None: - url = "https://example.org/" - auth = ("user", "password123") - app = App() - - async with httpx.AsyncClient( - transport=httpx.MockTransport(app), auth=auth - ) as client: - response = await client.get(url) - - assert response.status_code == 200 - assert response.json() == {"auth": "Basic dXNlcjpwYXNzd29yZDEyMw=="} - - -@pytest.mark.anyio -async def test_custom_auth() -> None: - url = "https://example.org/" - app = App() - - def auth(request: httpx.Request) -> httpx.Request: - request.headers["Authorization"] = "Token 123" - return request - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": "Token 123"} - - -def test_netrc_auth_credentials_exist() -> None: - """ - When netrc auth is being used and a request is made to a host that is - in the netrc file, then the relevant credentials should be applied. - """ - netrc_file = str(FIXTURES_DIR / ".netrc") - url = "http://netrcexample.org" - app = App() - auth = httpx.NetRCAuth(netrc_file) - - with httpx.Client(transport=httpx.MockTransport(app), auth=auth) as client: - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == { - "auth": "Basic ZXhhbXBsZS11c2VybmFtZTpleGFtcGxlLXBhc3N3b3Jk" - } - - -def test_netrc_auth_credentials_do_not_exist() -> None: - """ - When netrc auth is being used and a request is made to a host that is - not in the netrc file, then no credentials should be applied. - """ - netrc_file = str(FIXTURES_DIR / ".netrc") - url = "http://example.org" - app = App() - auth = httpx.NetRCAuth(netrc_file) - - with httpx.Client(transport=httpx.MockTransport(app), auth=auth) as client: - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == {"auth": None} - - -@pytest.mark.skipif( - sys.version_info >= (3, 11), - reason="netrc files without a password are valid from Python >= 3.11", -) -def test_netrc_auth_nopassword_parse_error() -> None: # pragma: no cover - """ - Python has different netrc parsing behaviours with different versions. - For Python < 3.11 a netrc file with no password is invalid. In this case - we want to allow the parse error to be raised. - """ - netrc_file = str(FIXTURES_DIR / ".netrc-nopassword") - with pytest.raises(netrc.NetrcParseError): - httpx.NetRCAuth(netrc_file) - - -@pytest.mark.anyio -async def test_auth_disable_per_request() -> None: - url = "https://example.org/" - auth = ("user", "password123") - app = App() - - async with httpx.AsyncClient( - transport=httpx.MockTransport(app), auth=auth - ) as client: - response = await client.get(url, auth=None) - - assert response.status_code == 200 - assert response.json() == {"auth": None} - - -def test_auth_hidden_url() -> None: - url = "http://example-username:example-password@example.org/" - expected = "URL('http://example-username:[secure]@example.org/')" - assert url == httpx.URL(url) - assert expected == repr(httpx.URL(url)) - - -@pytest.mark.anyio -async def test_auth_hidden_header() -> None: - url = "https://example.org/" - auth = ("example-username", "example-password") - app = App() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert "'authorization': '[secure]'" in str(response.request.headers) - - -@pytest.mark.anyio -async def test_auth_property() -> None: - app = App() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - assert client.auth is None - - client.auth = ("user", "password123") - assert isinstance(client.auth, httpx.BasicAuth) - - url = "https://example.org/" - response = await client.get(url) - assert response.status_code == 200 - assert response.json() == {"auth": "Basic dXNlcjpwYXNzd29yZDEyMw=="} - - -@pytest.mark.anyio -async def test_auth_invalid_type() -> None: - app = App() - - with pytest.raises(TypeError): - client = httpx.AsyncClient( - transport=httpx.MockTransport(app), - auth="not a tuple, not a callable", # type: ignore - ) - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - with pytest.raises(TypeError): - await client.get(auth="not a tuple, not a callable") # type: ignore - - with pytest.raises(TypeError): - client.auth = "not a tuple, not a callable" # type: ignore - - -@pytest.mark.anyio -async def test_digest_auth_returns_no_auth_if_no_digest_header_in_response() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = App() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": None} - assert len(response.history) == 0 - - -def test_digest_auth_returns_no_auth_if_alternate_auth_scheme() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - auth_header = "Token ..." - app = App(auth_header=auth_header, status_code=401) - - client = httpx.Client(transport=httpx.MockTransport(app)) - response = client.get(url, auth=auth) - - assert response.status_code == 401 - assert response.json() == {"auth": None} - assert len(response.history) == 0 - - -@pytest.mark.anyio -async def test_digest_auth_200_response_including_digest_auth_header() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - auth_header = 'Digest realm="realm@host.com",qop="auth",nonce="abc",opaque="xyz"' - app = App(auth_header=auth_header, status_code=200) - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": None} - assert len(response.history) == 0 - - -@pytest.mark.anyio -async def test_digest_auth_401_response_without_digest_auth_header() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = App(auth_header="", status_code=401) - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 401 - assert response.json() == {"auth": None} - assert len(response.history) == 0 - - -@pytest.mark.parametrize( - "algorithm,expected_hash_length,expected_response_length", - [ - ("MD5", 64, 32), - ("MD5-SESS", 64, 32), - ("SHA", 64, 40), - ("SHA-SESS", 64, 40), - ("SHA-256", 64, 64), - ("SHA-256-SESS", 64, 64), - ("SHA-512", 64, 128), - ("SHA-512-SESS", 64, 128), - ], -) -@pytest.mark.anyio -async def test_digest_auth( - algorithm: str, expected_hash_length: int, expected_response_length: int -) -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp(algorithm=algorithm) - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert len(response.history) == 1 - - authorization = typing.cast(typing.Dict[str, typing.Any], response.json())["auth"] - scheme, _, fields = authorization.partition(" ") - assert scheme == "Digest" - - response_fields = [field.strip() for field in fields.split(",")] - digest_data = dict(field.split("=") for field in response_fields) - - assert digest_data["username"] == '"user"' - assert digest_data["realm"] == '"httpx@example.org"' - assert "nonce" in digest_data - assert digest_data["uri"] == '"/"' - assert len(digest_data["response"]) == expected_response_length + 2 # extra quotes - assert len(digest_data["opaque"]) == expected_hash_length + 2 - assert digest_data["algorithm"] == algorithm - assert digest_data["qop"] == "auth" - assert digest_data["nc"] == "00000001" - assert len(digest_data["cnonce"]) == 16 + 2 - - -@pytest.mark.anyio -async def test_digest_auth_no_specified_qop() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp(qop="") - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert len(response.history) == 1 - - authorization = typing.cast(typing.Dict[str, typing.Any], response.json())["auth"] - scheme, _, fields = authorization.partition(" ") - assert scheme == "Digest" - - response_fields = [field.strip() for field in fields.split(",")] - digest_data = dict(field.split("=") for field in response_fields) - - assert "qop" not in digest_data - assert "nc" not in digest_data - assert "cnonce" not in digest_data - assert digest_data["username"] == '"user"' - assert digest_data["realm"] == '"httpx@example.org"' - assert len(digest_data["nonce"]) == 64 + 2 # extra quotes - assert digest_data["uri"] == '"/"' - assert len(digest_data["response"]) == 64 + 2 - assert len(digest_data["opaque"]) == 64 + 2 - assert digest_data["algorithm"] == "SHA-256" - - -@pytest.mark.parametrize("qop", ("auth, auth-int", "auth,auth-int", "unknown,auth")) -@pytest.mark.anyio -async def test_digest_auth_qop_including_spaces_and_auth_returns_auth(qop: str) -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp(qop=qop) - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert len(response.history) == 1 - - -@pytest.mark.anyio -async def test_digest_auth_qop_auth_int_not_implemented() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp(qop="auth-int") - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - with pytest.raises(NotImplementedError): - await client.get(url, auth=auth) - - -@pytest.mark.anyio -async def test_digest_auth_qop_must_be_auth_or_auth_int() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp(qop="not-auth") - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - with pytest.raises(httpx.ProtocolError): - await client.get(url, auth=auth) - - -@pytest.mark.anyio -async def test_digest_auth_incorrect_credentials() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp(send_response_after_attempt=2) - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 401 - assert len(response.history) == 1 - - -@pytest.mark.anyio -async def test_digest_auth_reuses_challenge() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response_1 = await client.get(url, auth=auth) - response_2 = await client.get(url, auth=auth) - - assert response_1.status_code == 200 - assert response_2.status_code == 200 - - assert len(response_1.history) == 1 - assert len(response_2.history) == 0 - - -@pytest.mark.anyio -async def test_digest_auth_resets_nonce_count_after_401() -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response_1 = await client.get(url, auth=auth) - assert response_1.status_code == 200 - assert len(response_1.history) == 1 - - first_nonce = parse_keqv_list( - response_1.request.headers["Authorization"].split(", ") - )["nonce"] - first_nc = parse_keqv_list( - response_1.request.headers["Authorization"].split(", ") - )["nc"] - - # with this we now force a 401 on a subsequent (but initial) request - app.send_response_after_attempt = 2 - - # we expect the client again to try to authenticate, - # i.e. the history length must be 1 - response_2 = await client.get(url, auth=auth) - assert response_2.status_code == 200 - assert len(response_2.history) == 1 - - second_nonce = parse_keqv_list( - response_2.request.headers["Authorization"].split(", ") - )["nonce"] - second_nc = parse_keqv_list( - response_2.request.headers["Authorization"].split(", ") - )["nc"] - - assert first_nonce != second_nonce # ensures that the auth challenge was reset - assert ( - first_nc == second_nc - ) # ensures the nonce count is reset when the authentication failed - - -@pytest.mark.parametrize( - "auth_header", - [ - 'Digest realm="httpx@example.org", qop="auth"', # missing fields - 'Digest realm="httpx@example.org", qop="auth,au', # malformed fields list - ], -) -@pytest.mark.anyio -async def test_async_digest_auth_raises_protocol_error_on_malformed_header( - auth_header: str, -) -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = App(auth_header=auth_header, status_code=401) - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - with pytest.raises(httpx.ProtocolError): - await client.get(url, auth=auth) - - -@pytest.mark.parametrize( - "auth_header", - [ - 'Digest realm="httpx@example.org", qop="auth"', # missing fields - 'Digest realm="httpx@example.org", qop="auth,au', # malformed fields list - ], -) -def test_sync_digest_auth_raises_protocol_error_on_malformed_header( - auth_header: str, -) -> None: - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = App(auth_header=auth_header, status_code=401) - - with httpx.Client(transport=httpx.MockTransport(app)) as client: - with pytest.raises(httpx.ProtocolError): - client.get(url, auth=auth) - - -@pytest.mark.anyio -async def test_async_auth_history() -> None: - """ - Test that intermediate requests sent as part of an authentication flow - are recorded in the response history. - """ - url = "https://example.org/" - auth = RepeatAuth(repeat=2) - app = App(auth_header="abc") - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": "Repeat abc.abc"} - - assert len(response.history) == 2 - resp1, resp2 = response.history - assert resp1.json() == {"auth": "Repeat 0"} - assert resp2.json() == {"auth": "Repeat 1"} - - assert len(resp2.history) == 1 - assert resp2.history == [resp1] - - assert len(resp1.history) == 0 - - -def test_sync_auth_history() -> None: - """ - Test that intermediate requests sent as part of an authentication flow - are recorded in the response history. - """ - url = "https://example.org/" - auth = RepeatAuth(repeat=2) - app = App(auth_header="abc") - - with httpx.Client(transport=httpx.MockTransport(app)) as client: - response = client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": "Repeat abc.abc"} - - assert len(response.history) == 2 - resp1, resp2 = response.history - assert resp1.json() == {"auth": "Repeat 0"} - assert resp2.json() == {"auth": "Repeat 1"} - - assert len(resp2.history) == 1 - assert resp2.history == [resp1] - - assert len(resp1.history) == 0 - - -class ConsumeBodyTransport(httpx.MockTransport): - async def handle_async_request(self, request: httpx.Request) -> httpx.Response: - assert isinstance(request.stream, httpx.AsyncByteStream) - [_ async for _ in request.stream] - return self.handler(request) # type: ignore[return-value] - - -@pytest.mark.anyio -async def test_digest_auth_unavailable_streaming_body(): - url = "https://example.org/" - auth = httpx.DigestAuth(username="user", password="password123") - app = DigestApp() - - async def streaming_body() -> typing.AsyncIterator[bytes]: - yield b"Example request body" # pragma: no cover - - async with httpx.AsyncClient(transport=ConsumeBodyTransport(app)) as client: - with pytest.raises(httpx.StreamConsumed): - await client.post(url, content=streaming_body(), auth=auth) - - -@pytest.mark.anyio -async def test_async_auth_reads_response_body() -> None: - """ - Test that we can read the response body in an auth flow if `requires_response_body` - is set. - """ - url = "https://example.org/" - auth = ResponseBodyAuth("xyz") - app = App() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": '{"auth":"xyz"}'} - - -def test_sync_auth_reads_response_body() -> None: - """ - Test that we can read the response body in an auth flow if `requires_response_body` - is set. - """ - url = "https://example.org/" - auth = ResponseBodyAuth("xyz") - app = App() - - with httpx.Client(transport=httpx.MockTransport(app)) as client: - response = client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": '{"auth":"xyz"}'} - - -@pytest.mark.anyio -async def test_async_auth() -> None: - """ - Test that we can use an auth implementation specific to the async case, to - support cases that require performing I/O or using concurrency primitives (such - as checking a disk-based cache or fetching a token from a remote auth server). - """ - url = "https://example.org/" - auth = SyncOrAsyncAuth() - app = App() - - async with httpx.AsyncClient(transport=httpx.MockTransport(app)) as client: - response = await client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": "async-auth"} - - -def test_sync_auth() -> None: - """ - Test that we can use an auth implementation specific to the sync case. - """ - url = "https://example.org/" - auth = SyncOrAsyncAuth() - app = App() - - with httpx.Client(transport=httpx.MockTransport(app)) as client: - response = client.get(url, auth=auth) - - assert response.status_code == 200 - assert response.json() == {"auth": "sync-auth"} diff --git a/tests/client/test_client.py b/tests/client/test_client.py deleted file mode 100644 index 657839018a..0000000000 --- a/tests/client/test_client.py +++ /dev/null @@ -1,462 +0,0 @@ -from __future__ import annotations - -import typing -from datetime import timedelta - -import chardet -import pytest - -import httpx - - -def autodetect(content): - return chardet.detect(content).get("encoding") - - -def test_get(server): - url = server.url - with httpx.Client(http2=True) as http: - response = http.get(url) - assert response.status_code == 200 - assert response.url == url - assert response.content == b"Hello, world!" - assert response.text == "Hello, world!" - assert response.http_version == "HTTP/1.1" - assert response.encoding == "utf-8" - assert response.request.url == url - assert response.headers - assert response.is_redirect is False - assert repr(response) == "" - assert response.elapsed > timedelta(0) - - -@pytest.mark.parametrize( - "url", - [ - pytest.param("invalid://example.org", id="scheme-not-http(s)"), - pytest.param("://example.org", id="no-scheme"), - pytest.param("http://", id="no-host"), - ], -) -def test_get_invalid_url(server, url): - with httpx.Client() as client: - with pytest.raises((httpx.UnsupportedProtocol, httpx.LocalProtocolError)): - client.get(url) - - -def test_build_request(server): - url = server.url.copy_with(path="/echo_headers") - headers = {"Custom-header": "value"} - - with httpx.Client() as client: - request = client.build_request("GET", url) - request.headers.update(headers) - response = client.send(request) - - assert response.status_code == 200 - assert response.url == url - - assert response.json()["Custom-header"] == "value" - - -def test_build_post_request(server): - url = server.url.copy_with(path="/echo_headers") - headers = {"Custom-header": "value"} - - with httpx.Client() as client: - request = client.build_request("POST", url) - request.headers.update(headers) - response = client.send(request) - - assert response.status_code == 200 - assert response.url == url - - assert response.json()["Content-length"] == "0" - assert response.json()["Custom-header"] == "value" - - -def test_post(server): - with httpx.Client() as client: - response = client.post(server.url, content=b"Hello, world!") - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_post_json(server): - with httpx.Client() as client: - response = client.post(server.url, json={"text": "Hello, world!"}) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_stream_response(server): - with httpx.Client() as client: - with client.stream("GET", server.url) as response: - content = response.read() - assert response.status_code == 200 - assert content == b"Hello, world!" - - -def test_stream_iterator(server): - body = b"" - - with httpx.Client() as client: - with client.stream("GET", server.url) as response: - for chunk in response.iter_bytes(): - body += chunk - - assert response.status_code == 200 - assert body == b"Hello, world!" - - -def test_raw_iterator(server): - body = b"" - - with httpx.Client() as client: - with client.stream("GET", server.url) as response: - for chunk in response.iter_raw(): - body += chunk - - assert response.status_code == 200 - assert body == b"Hello, world!" - - -def test_cannot_stream_async_request(server): - async def hello_world() -> typing.AsyncIterator[bytes]: # pragma: no cover - yield b"Hello, " - yield b"world!" - - with httpx.Client() as client: - with pytest.raises(RuntimeError): - client.post(server.url, content=hello_world()) - - -def test_raise_for_status(server): - with httpx.Client() as client: - for status_code in (200, 400, 404, 500, 505): - response = client.request( - "GET", server.url.copy_with(path=f"/status/{status_code}") - ) - if 400 <= status_code < 600: - with pytest.raises(httpx.HTTPStatusError) as exc_info: - response.raise_for_status() - assert exc_info.value.response == response - assert exc_info.value.request.url.path == f"/status/{status_code}" - else: - assert response.raise_for_status() is response - - -def test_options(server): - with httpx.Client() as client: - response = client.options(server.url) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_head(server): - with httpx.Client() as client: - response = client.head(server.url) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_put(server): - with httpx.Client() as client: - response = client.put(server.url, content=b"Hello, world!") - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_patch(server): - with httpx.Client() as client: - response = client.patch(server.url, content=b"Hello, world!") - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_delete(server): - with httpx.Client() as client: - response = client.delete(server.url) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_base_url(server): - base_url = server.url - with httpx.Client(base_url=base_url) as client: - response = client.get("/") - assert response.status_code == 200 - assert response.url == base_url - - -def test_merge_absolute_url(): - client = httpx.Client(base_url="https://www.example.com/") - request = client.build_request("GET", "http://www.example.com/") - assert request.url == "http://www.example.com/" - - -def test_merge_relative_url(): - client = httpx.Client(base_url="https://www.example.com/") - request = client.build_request("GET", "/testing/123") - assert request.url == "https://www.example.com/testing/123" - - -def test_merge_relative_url_with_path(): - client = httpx.Client(base_url="https://www.example.com/some/path") - request = client.build_request("GET", "/testing/123") - assert request.url == "https://www.example.com/some/path/testing/123" - - -def test_merge_relative_url_with_dotted_path(): - client = httpx.Client(base_url="https://www.example.com/some/path") - request = client.build_request("GET", "../testing/123") - assert request.url == "https://www.example.com/some/testing/123" - - -def test_merge_relative_url_with_path_including_colon(): - client = httpx.Client(base_url="https://www.example.com/some/path") - request = client.build_request("GET", "/testing:123") - assert request.url == "https://www.example.com/some/path/testing:123" - - -def test_merge_relative_url_with_encoded_slashes(): - client = httpx.Client(base_url="https://www.example.com/") - request = client.build_request("GET", "/testing%2F123") - assert request.url == "https://www.example.com/testing%2F123" - - client = httpx.Client(base_url="https://www.example.com/base%2Fpath") - request = client.build_request("GET", "/testing") - assert request.url == "https://www.example.com/base%2Fpath/testing" - - -def test_context_managed_transport(): - class Transport(httpx.BaseTransport): - def __init__(self) -> None: - self.events: list[str] = [] - - def close(self): - # The base implementation of httpx.BaseTransport just - # calls into `.close`, so simple transport cases can just override - # this method for any cleanup, where more complex cases - # might want to additionally override `__enter__`/`__exit__`. - self.events.append("transport.close") - - def __enter__(self): - super().__enter__() - self.events.append("transport.__enter__") - - def __exit__(self, *args): - super().__exit__(*args) - self.events.append("transport.__exit__") - - transport = Transport() - with httpx.Client(transport=transport): - pass - - assert transport.events == [ - "transport.__enter__", - "transport.close", - "transport.__exit__", - ] - - -def test_context_managed_transport_and_mount(): - class Transport(httpx.BaseTransport): - def __init__(self, name: str) -> None: - self.name: str = name - self.events: list[str] = [] - - def close(self): - # The base implementation of httpx.BaseTransport just - # calls into `.close`, so simple transport cases can just override - # this method for any cleanup, where more complex cases - # might want to additionally override `__enter__`/`__exit__`. - self.events.append(f"{self.name}.close") - - def __enter__(self): - super().__enter__() - self.events.append(f"{self.name}.__enter__") - - def __exit__(self, *args): - super().__exit__(*args) - self.events.append(f"{self.name}.__exit__") - - transport = Transport(name="transport") - mounted = Transport(name="mounted") - with httpx.Client(transport=transport, mounts={"http://www.example.org": mounted}): - pass - - assert transport.events == [ - "transport.__enter__", - "transport.close", - "transport.__exit__", - ] - assert mounted.events == [ - "mounted.__enter__", - "mounted.close", - "mounted.__exit__", - ] - - -def hello_world(request): - return httpx.Response(200, text="Hello, world!") - - -def test_client_closed_state_using_implicit_open(): - client = httpx.Client(transport=httpx.MockTransport(hello_world)) - - assert not client.is_closed - client.get("http://example.com") - - assert not client.is_closed - client.close() - - assert client.is_closed - - # Once we're close we cannot make any more requests. - with pytest.raises(RuntimeError): - client.get("http://example.com") - - # Once we're closed we cannot reopen the client. - with pytest.raises(RuntimeError): - with client: - pass # pragma: no cover - - -def test_client_closed_state_using_with_block(): - with httpx.Client(transport=httpx.MockTransport(hello_world)) as client: - assert not client.is_closed - client.get("http://example.com") - - assert client.is_closed - with pytest.raises(RuntimeError): - client.get("http://example.com") - - -def echo_raw_headers(request: httpx.Request) -> httpx.Response: - data = [ - (name.decode("ascii"), value.decode("ascii")) - for name, value in request.headers.raw - ] - return httpx.Response(200, json=data) - - -def test_raw_client_header(): - """ - Set a header in the Client. - """ - url = "http://example.org/echo_headers" - headers = {"Example-Header": "example-value"} - - client = httpx.Client( - transport=httpx.MockTransport(echo_raw_headers), headers=headers - ) - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == [ - ["Host", "example.org"], - ["Accept", "*/*"], - ["Accept-Encoding", "gzip, deflate, br, zstd"], - ["Connection", "keep-alive"], - ["User-Agent", f"python-httpx/{httpx.__version__}"], - ["Example-Header", "example-value"], - ] - - -def unmounted(request: httpx.Request) -> httpx.Response: - data = {"app": "unmounted"} - return httpx.Response(200, json=data) - - -def mounted(request: httpx.Request) -> httpx.Response: - data = {"app": "mounted"} - return httpx.Response(200, json=data) - - -def test_mounted_transport(): - transport = httpx.MockTransport(unmounted) - mounts = {"custom://": httpx.MockTransport(mounted)} - - client = httpx.Client(transport=transport, mounts=mounts) - - response = client.get("https://www.example.com") - assert response.status_code == 200 - assert response.json() == {"app": "unmounted"} - - response = client.get("custom://www.example.com") - assert response.status_code == 200 - assert response.json() == {"app": "mounted"} - - -def test_all_mounted_transport(): - mounts = {"all://": httpx.MockTransport(mounted)} - - client = httpx.Client(mounts=mounts) - - response = client.get("https://www.example.com") - assert response.status_code == 200 - assert response.json() == {"app": "mounted"} - - -def test_server_extensions(server): - url = server.url.copy_with(path="/http_version_2") - with httpx.Client(http2=True) as client: - response = client.get(url) - assert response.status_code == 200 - assert response.extensions["http_version"] == b"HTTP/1.1" - - -def test_client_decode_text_using_autodetect(): - # Ensure that a 'default_encoding=autodetect' on the response allows for - # encoding autodetection to be used when no "Content-Type: text/plain; charset=..." - # info is present. - # - # Here we have some french text encoded with ISO-8859-1, rather than UTF-8. - text = ( - "Non-seulement Despréaux ne se trompait pas, mais de tous les écrivains " - "que la France a produits, sans excepter Voltaire lui-même, imprégné de " - "l'esprit anglais par son séjour à Londres, c'est incontestablement " - "Molière ou Poquelin qui reproduit avec l'exactitude la plus vive et la " - "plus complète le fond du génie français." - ) - - def cp1252_but_no_content_type(request): - content = text.encode("ISO-8859-1") - return httpx.Response(200, content=content) - - transport = httpx.MockTransport(cp1252_but_no_content_type) - with httpx.Client(transport=transport, default_encoding=autodetect) as client: - response = client.get("http://www.example.com") - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.encoding == "ISO-8859-1" - assert response.text == text - - -def test_client_decode_text_using_explicit_encoding(): - # Ensure that a 'default_encoding="..."' on the response is used for text decoding - # when no "Content-Type: text/plain; charset=..."" info is present. - # - # Here we have some french text encoded with ISO-8859-1, rather than UTF-8. - text = ( - "Non-seulement Despréaux ne se trompait pas, mais de tous les écrivains " - "que la France a produits, sans excepter Voltaire lui-même, imprégné de " - "l'esprit anglais par son séjour à Londres, c'est incontestablement " - "Molière ou Poquelin qui reproduit avec l'exactitude la plus vive et la " - "plus complète le fond du génie français." - ) - - def cp1252_but_no_content_type(request): - content = text.encode("ISO-8859-1") - return httpx.Response(200, content=content) - - transport = httpx.MockTransport(cp1252_but_no_content_type) - with httpx.Client(transport=transport, default_encoding=autodetect) as client: - response = client.get("http://www.example.com") - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.encoding == "ISO-8859-1" - assert response.text == text diff --git a/tests/client/test_cookies.py b/tests/client/test_cookies.py deleted file mode 100644 index f0c8352593..0000000000 --- a/tests/client/test_cookies.py +++ /dev/null @@ -1,168 +0,0 @@ -from http.cookiejar import Cookie, CookieJar - -import pytest - -import httpx - - -def get_and_set_cookies(request: httpx.Request) -> httpx.Response: - if request.url.path == "/echo_cookies": - data = {"cookies": request.headers.get("cookie")} - return httpx.Response(200, json=data) - elif request.url.path == "/set_cookie": - return httpx.Response(200, headers={"set-cookie": "example-name=example-value"}) - else: - raise NotImplementedError() # pragma: no cover - - -def test_set_cookie() -> None: - """ - Send a request including a cookie. - """ - url = "http://example.org/echo_cookies" - cookies = {"example-name": "example-value"} - - client = httpx.Client( - cookies=cookies, transport=httpx.MockTransport(get_and_set_cookies) - ) - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == {"cookies": "example-name=example-value"} - - -def test_set_per_request_cookie_is_deprecated() -> None: - """ - Sending a request including a per-request cookie is deprecated. - """ - url = "http://example.org/echo_cookies" - cookies = {"example-name": "example-value"} - - client = httpx.Client(transport=httpx.MockTransport(get_and_set_cookies)) - with pytest.warns(DeprecationWarning): - response = client.get(url, cookies=cookies) - - assert response.status_code == 200 - assert response.json() == {"cookies": "example-name=example-value"} - - -def test_set_cookie_with_cookiejar() -> None: - """ - Send a request including a cookie, using a `CookieJar` instance. - """ - - url = "http://example.org/echo_cookies" - cookies = CookieJar() - cookie = Cookie( - version=0, - name="example-name", - value="example-value", - port=None, - port_specified=False, - domain="", - domain_specified=False, - domain_initial_dot=False, - path="/", - path_specified=True, - secure=False, - expires=None, - discard=True, - comment=None, - comment_url=None, - rest={"HttpOnly": ""}, - rfc2109=False, - ) - cookies.set_cookie(cookie) - - client = httpx.Client( - cookies=cookies, transport=httpx.MockTransport(get_and_set_cookies) - ) - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == {"cookies": "example-name=example-value"} - - -def test_setting_client_cookies_to_cookiejar() -> None: - """ - Send a request including a cookie, using a `CookieJar` instance. - """ - - url = "http://example.org/echo_cookies" - cookies = CookieJar() - cookie = Cookie( - version=0, - name="example-name", - value="example-value", - port=None, - port_specified=False, - domain="", - domain_specified=False, - domain_initial_dot=False, - path="/", - path_specified=True, - secure=False, - expires=None, - discard=True, - comment=None, - comment_url=None, - rest={"HttpOnly": ""}, - rfc2109=False, - ) - cookies.set_cookie(cookie) - - client = httpx.Client( - cookies=cookies, transport=httpx.MockTransport(get_and_set_cookies) - ) - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == {"cookies": "example-name=example-value"} - - -def test_set_cookie_with_cookies_model() -> None: - """ - Send a request including a cookie, using a `Cookies` instance. - """ - - url = "http://example.org/echo_cookies" - cookies = httpx.Cookies() - cookies["example-name"] = "example-value" - - client = httpx.Client(transport=httpx.MockTransport(get_and_set_cookies)) - client.cookies = cookies - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == {"cookies": "example-name=example-value"} - - -def test_get_cookie() -> None: - url = "http://example.org/set_cookie" - - client = httpx.Client(transport=httpx.MockTransport(get_and_set_cookies)) - response = client.get(url) - - assert response.status_code == 200 - assert response.cookies["example-name"] == "example-value" - assert client.cookies["example-name"] == "example-value" - - -def test_cookie_persistence() -> None: - """ - Ensure that Client instances persist cookies between requests. - """ - client = httpx.Client(transport=httpx.MockTransport(get_and_set_cookies)) - - response = client.get("http://example.org/echo_cookies") - assert response.status_code == 200 - assert response.json() == {"cookies": None} - - response = client.get("http://example.org/set_cookie") - assert response.status_code == 200 - assert response.cookies["example-name"] == "example-value" - assert client.cookies["example-name"] == "example-value" - - response = client.get("http://example.org/echo_cookies") - assert response.status_code == 200 - assert response.json() == {"cookies": "example-name=example-value"} diff --git a/tests/client/test_event_hooks.py b/tests/client/test_event_hooks.py deleted file mode 100644 index 78fb0484e6..0000000000 --- a/tests/client/test_event_hooks.py +++ /dev/null @@ -1,228 +0,0 @@ -import pytest - -import httpx - - -def app(request: httpx.Request) -> httpx.Response: - if request.url.path == "/redirect": - return httpx.Response(303, headers={"server": "testserver", "location": "/"}) - elif request.url.path.startswith("/status/"): - status_code = int(request.url.path[-3:]) - return httpx.Response(status_code, headers={"server": "testserver"}) - - return httpx.Response(200, headers={"server": "testserver"}) - - -def test_event_hooks(): - events = [] - - def on_request(request): - events.append({"event": "request", "headers": dict(request.headers)}) - - def on_response(response): - events.append({"event": "response", "headers": dict(response.headers)}) - - event_hooks = {"request": [on_request], "response": [on_response]} - - with httpx.Client( - event_hooks=event_hooks, transport=httpx.MockTransport(app) - ) as http: - http.get("http://127.0.0.1:8000/", auth=("username", "password")) - - assert events == [ - { - "event": "request", - "headers": { - "host": "127.0.0.1:8000", - "user-agent": f"python-httpx/{httpx.__version__}", - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - }, - }, - { - "event": "response", - "headers": {"server": "testserver"}, - }, - ] - - -def test_event_hooks_raising_exception(server): - def raise_on_4xx_5xx(response): - response.raise_for_status() - - event_hooks = {"response": [raise_on_4xx_5xx]} - - with httpx.Client( - event_hooks=event_hooks, transport=httpx.MockTransport(app) - ) as http: - try: - http.get("http://127.0.0.1:8000/status/400") - except httpx.HTTPStatusError as exc: - assert exc.response.is_closed - - -@pytest.mark.anyio -async def test_async_event_hooks(): - events = [] - - async def on_request(request): - events.append({"event": "request", "headers": dict(request.headers)}) - - async def on_response(response): - events.append({"event": "response", "headers": dict(response.headers)}) - - event_hooks = {"request": [on_request], "response": [on_response]} - - async with httpx.AsyncClient( - event_hooks=event_hooks, transport=httpx.MockTransport(app) - ) as http: - await http.get("http://127.0.0.1:8000/", auth=("username", "password")) - - assert events == [ - { - "event": "request", - "headers": { - "host": "127.0.0.1:8000", - "user-agent": f"python-httpx/{httpx.__version__}", - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - }, - }, - { - "event": "response", - "headers": {"server": "testserver"}, - }, - ] - - -@pytest.mark.anyio -async def test_async_event_hooks_raising_exception(): - async def raise_on_4xx_5xx(response): - response.raise_for_status() - - event_hooks = {"response": [raise_on_4xx_5xx]} - - async with httpx.AsyncClient( - event_hooks=event_hooks, transport=httpx.MockTransport(app) - ) as http: - try: - await http.get("http://127.0.0.1:8000/status/400") - except httpx.HTTPStatusError as exc: - assert exc.response.is_closed - - -def test_event_hooks_with_redirect(): - """ - A redirect request should trigger additional 'request' and 'response' event hooks. - """ - - events = [] - - def on_request(request): - events.append({"event": "request", "headers": dict(request.headers)}) - - def on_response(response): - events.append({"event": "response", "headers": dict(response.headers)}) - - event_hooks = {"request": [on_request], "response": [on_response]} - - with httpx.Client( - event_hooks=event_hooks, - transport=httpx.MockTransport(app), - follow_redirects=True, - ) as http: - http.get("http://127.0.0.1:8000/redirect", auth=("username", "password")) - - assert events == [ - { - "event": "request", - "headers": { - "host": "127.0.0.1:8000", - "user-agent": f"python-httpx/{httpx.__version__}", - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - }, - }, - { - "event": "response", - "headers": {"location": "/", "server": "testserver"}, - }, - { - "event": "request", - "headers": { - "host": "127.0.0.1:8000", - "user-agent": f"python-httpx/{httpx.__version__}", - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - }, - }, - { - "event": "response", - "headers": {"server": "testserver"}, - }, - ] - - -@pytest.mark.anyio -async def test_async_event_hooks_with_redirect(): - """ - A redirect request should trigger additional 'request' and 'response' event hooks. - """ - - events = [] - - async def on_request(request): - events.append({"event": "request", "headers": dict(request.headers)}) - - async def on_response(response): - events.append({"event": "response", "headers": dict(response.headers)}) - - event_hooks = {"request": [on_request], "response": [on_response]} - - async with httpx.AsyncClient( - event_hooks=event_hooks, - transport=httpx.MockTransport(app), - follow_redirects=True, - ) as http: - await http.get("http://127.0.0.1:8000/redirect", auth=("username", "password")) - - assert events == [ - { - "event": "request", - "headers": { - "host": "127.0.0.1:8000", - "user-agent": f"python-httpx/{httpx.__version__}", - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - }, - }, - { - "event": "response", - "headers": {"location": "/", "server": "testserver"}, - }, - { - "event": "request", - "headers": { - "host": "127.0.0.1:8000", - "user-agent": f"python-httpx/{httpx.__version__}", - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - }, - }, - { - "event": "response", - "headers": {"server": "testserver"}, - }, - ] diff --git a/tests/client/test_headers.py b/tests/client/test_headers.py deleted file mode 100755 index 47f5a4d731..0000000000 --- a/tests/client/test_headers.py +++ /dev/null @@ -1,293 +0,0 @@ -#!/usr/bin/env python3 - -import pytest - -import httpx - - -def echo_headers(request: httpx.Request) -> httpx.Response: - data = {"headers": dict(request.headers)} - return httpx.Response(200, json=data) - - -def echo_repeated_headers_multi_items(request: httpx.Request) -> httpx.Response: - data = {"headers": list(request.headers.multi_items())} - return httpx.Response(200, json=data) - - -def echo_repeated_headers_items(request: httpx.Request) -> httpx.Response: - data = {"headers": list(request.headers.items())} - return httpx.Response(200, json=data) - - -def test_client_header(): - """ - Set a header in the Client. - """ - url = "http://example.org/echo_headers" - headers = {"Example-Header": "example-value"} - - client = httpx.Client(transport=httpx.MockTransport(echo_headers), headers=headers) - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == { - "headers": { - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "example-header": "example-value", - "host": "example.org", - "user-agent": f"python-httpx/{httpx.__version__}", - } - } - - -def test_header_merge(): - url = "http://example.org/echo_headers" - client_headers = {"User-Agent": "python-myclient/0.2.1"} - request_headers = {"X-Auth-Token": "FooBarBazToken"} - client = httpx.Client( - transport=httpx.MockTransport(echo_headers), headers=client_headers - ) - response = client.get(url, headers=request_headers) - - assert response.status_code == 200 - assert response.json() == { - "headers": { - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "host": "example.org", - "user-agent": "python-myclient/0.2.1", - "x-auth-token": "FooBarBazToken", - } - } - - -def test_header_merge_conflicting_headers(): - url = "http://example.org/echo_headers" - client_headers = {"X-Auth-Token": "FooBar"} - request_headers = {"X-Auth-Token": "BazToken"} - client = httpx.Client( - transport=httpx.MockTransport(echo_headers), headers=client_headers - ) - response = client.get(url, headers=request_headers) - - assert response.status_code == 200 - assert response.json() == { - "headers": { - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "host": "example.org", - "user-agent": f"python-httpx/{httpx.__version__}", - "x-auth-token": "BazToken", - } - } - - -def test_header_update(): - url = "http://example.org/echo_headers" - client = httpx.Client(transport=httpx.MockTransport(echo_headers)) - first_response = client.get(url) - client.headers.update( - {"User-Agent": "python-myclient/0.2.1", "Another-Header": "AThing"} - ) - second_response = client.get(url) - - assert first_response.status_code == 200 - assert first_response.json() == { - "headers": { - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "host": "example.org", - "user-agent": f"python-httpx/{httpx.__version__}", - } - } - - assert second_response.status_code == 200 - assert second_response.json() == { - "headers": { - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "another-header": "AThing", - "connection": "keep-alive", - "host": "example.org", - "user-agent": "python-myclient/0.2.1", - } - } - - -def test_header_repeated_items(): - url = "http://example.org/echo_headers" - client = httpx.Client(transport=httpx.MockTransport(echo_repeated_headers_items)) - response = client.get(url, headers=[("x-header", "1"), ("x-header", "2,3")]) - - assert response.status_code == 200 - - echoed_headers = response.json()["headers"] - # as per RFC 7230, the whitespace after a comma is insignificant - # so we split and strip here so that we can do a safe comparison - assert ["x-header", ["1", "2", "3"]] in [ - [k, [subv.lstrip() for subv in v.split(",")]] for k, v in echoed_headers - ] - - -def test_header_repeated_multi_items(): - url = "http://example.org/echo_headers" - client = httpx.Client( - transport=httpx.MockTransport(echo_repeated_headers_multi_items) - ) - response = client.get(url, headers=[("x-header", "1"), ("x-header", "2,3")]) - - assert response.status_code == 200 - - echoed_headers = response.json()["headers"] - assert ["x-header", "1"] in echoed_headers - assert ["x-header", "2,3"] in echoed_headers - - -def test_remove_default_header(): - """ - Remove a default header from the Client. - """ - url = "http://example.org/echo_headers" - - client = httpx.Client(transport=httpx.MockTransport(echo_headers)) - del client.headers["User-Agent"] - - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == { - "headers": { - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "host": "example.org", - } - } - - -def test_header_does_not_exist(): - headers = httpx.Headers({"foo": "bar"}) - with pytest.raises(KeyError): - del headers["baz"] - - -def test_header_with_incorrect_value(): - with pytest.raises( - TypeError, - match=f"Header value must be str or bytes, not {type(None)}", - ): - httpx.Headers({"foo": None}) # type: ignore - - -def test_host_with_auth_and_port_in_url(): - """ - The Host header should only include the hostname, or hostname:port - (for non-default ports only). Any userinfo or default port should not - be present. - """ - url = "http://username:password@example.org:80/echo_headers" - - client = httpx.Client(transport=httpx.MockTransport(echo_headers)) - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == { - "headers": { - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "host": "example.org", - "user-agent": f"python-httpx/{httpx.__version__}", - "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - } - } - - -def test_host_with_non_default_port_in_url(): - """ - If the URL includes a non-default port, then it should be included in - the Host header. - """ - url = "http://username:password@example.org:123/echo_headers" - - client = httpx.Client(transport=httpx.MockTransport(echo_headers)) - response = client.get(url) - - assert response.status_code == 200 - assert response.json() == { - "headers": { - "accept": "*/*", - "accept-encoding": "gzip, deflate, br, zstd", - "connection": "keep-alive", - "host": "example.org:123", - "user-agent": f"python-httpx/{httpx.__version__}", - "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - } - } - - -def test_request_auto_headers(): - request = httpx.Request("GET", "https://www.example.org/") - assert "host" in request.headers - - -def test_same_origin(): - origin = httpx.URL("https://example.com") - request = httpx.Request("GET", "HTTPS://EXAMPLE.COM:443") - - client = httpx.Client() - headers = client._redirect_headers(request, origin, "GET") - - assert headers["Host"] == request.url.netloc.decode("ascii") - - -def test_not_same_origin(): - origin = httpx.URL("https://example.com") - request = httpx.Request("GET", "HTTP://EXAMPLE.COM:80") - - client = httpx.Client() - headers = client._redirect_headers(request, origin, "GET") - - assert headers["Host"] == origin.netloc.decode("ascii") - - -def test_is_https_redirect(): - url = httpx.URL("https://example.com") - request = httpx.Request( - "GET", "http://example.com", headers={"Authorization": "empty"} - ) - - client = httpx.Client() - headers = client._redirect_headers(request, url, "GET") - - assert "Authorization" in headers - - -def test_is_not_https_redirect(): - url = httpx.URL("https://www.example.com") - request = httpx.Request( - "GET", "http://example.com", headers={"Authorization": "empty"} - ) - - client = httpx.Client() - headers = client._redirect_headers(request, url, "GET") - - assert "Authorization" not in headers - - -def test_is_not_https_redirect_if_not_default_ports(): - url = httpx.URL("https://example.com:1337") - request = httpx.Request( - "GET", "http://example.com:9999", headers={"Authorization": "empty"} - ) - - client = httpx.Client() - headers = client._redirect_headers(request, url, "GET") - - assert "Authorization" not in headers diff --git a/tests/client/test_properties.py b/tests/client/test_properties.py deleted file mode 100644 index f9ca9f247f..0000000000 --- a/tests/client/test_properties.py +++ /dev/null @@ -1,68 +0,0 @@ -import httpx - - -def test_client_base_url(): - client = httpx.Client() - client.base_url = "https://www.example.org/" - assert isinstance(client.base_url, httpx.URL) - assert client.base_url == "https://www.example.org/" - - -def test_client_base_url_without_trailing_slash(): - client = httpx.Client() - client.base_url = "https://www.example.org/path" - assert isinstance(client.base_url, httpx.URL) - assert client.base_url == "https://www.example.org/path/" - - -def test_client_base_url_with_trailing_slash(): - client = httpx.Client() - client.base_url = "https://www.example.org/path/" - assert isinstance(client.base_url, httpx.URL) - assert client.base_url == "https://www.example.org/path/" - - -def test_client_headers(): - client = httpx.Client() - client.headers = {"a": "b"} - assert isinstance(client.headers, httpx.Headers) - assert client.headers["A"] == "b" - - -def test_client_cookies(): - client = httpx.Client() - client.cookies = {"a": "b"} - assert isinstance(client.cookies, httpx.Cookies) - mycookies = list(client.cookies.jar) - assert len(mycookies) == 1 - assert mycookies[0].name == "a" and mycookies[0].value == "b" - - -def test_client_timeout(): - expected_timeout = 12.0 - client = httpx.Client() - - client.timeout = expected_timeout - - assert isinstance(client.timeout, httpx.Timeout) - assert client.timeout.connect == expected_timeout - assert client.timeout.read == expected_timeout - assert client.timeout.write == expected_timeout - assert client.timeout.pool == expected_timeout - - -def test_client_event_hooks(): - def on_request(request): - pass # pragma: no cover - - client = httpx.Client() - client.event_hooks = {"request": [on_request]} - assert client.event_hooks == {"request": [on_request], "response": []} - - -def test_client_trust_env(): - client = httpx.Client() - assert client.trust_env - - client = httpx.Client(trust_env=False) - assert not client.trust_env diff --git a/tests/client/test_proxies.py b/tests/client/test_proxies.py deleted file mode 100644 index 3e4090dcec..0000000000 --- a/tests/client/test_proxies.py +++ /dev/null @@ -1,265 +0,0 @@ -import httpcore -import pytest - -import httpx - - -def url_to_origin(url: str) -> httpcore.URL: - """ - Given a URL string, return the origin in the raw tuple format that - `httpcore` uses for it's representation. - """ - u = httpx.URL(url) - return httpcore.URL(scheme=u.raw_scheme, host=u.raw_host, port=u.port, target="/") - - -def test_socks_proxy(): - url = httpx.URL("http://www.example.com") - - for proxy in ("socks5://localhost/", "socks5h://localhost/"): - client = httpx.Client(proxy=proxy) - transport = client._transport_for_url(url) - assert isinstance(transport, httpx.HTTPTransport) - assert isinstance(transport._pool, httpcore.SOCKSProxy) - - async_client = httpx.AsyncClient(proxy=proxy) - async_transport = async_client._transport_for_url(url) - assert isinstance(async_transport, httpx.AsyncHTTPTransport) - assert isinstance(async_transport._pool, httpcore.AsyncSOCKSProxy) - - -PROXY_URL = "http://[::1]" - - -@pytest.mark.parametrize( - ["url", "proxies", "expected"], - [ - ("http://example.com", {}, None), - ("http://example.com", {"https://": PROXY_URL}, None), - ("http://example.com", {"http://example.net": PROXY_URL}, None), - # Using "*" should match any domain name. - ("http://example.com", {"http://*": PROXY_URL}, PROXY_URL), - ("https://example.com", {"http://*": PROXY_URL}, None), - # Using "example.com" should match example.com, but not www.example.com - ("http://example.com", {"http://example.com": PROXY_URL}, PROXY_URL), - ("http://www.example.com", {"http://example.com": PROXY_URL}, None), - # Using "*.example.com" should match www.example.com, but not example.com - ("http://example.com", {"http://*.example.com": PROXY_URL}, None), - ("http://www.example.com", {"http://*.example.com": PROXY_URL}, PROXY_URL), - # Using "*example.com" should match example.com and www.example.com - ("http://example.com", {"http://*example.com": PROXY_URL}, PROXY_URL), - ("http://www.example.com", {"http://*example.com": PROXY_URL}, PROXY_URL), - ("http://wwwexample.com", {"http://*example.com": PROXY_URL}, None), - # ... - ("http://example.com:443", {"http://example.com": PROXY_URL}, PROXY_URL), - ("http://example.com", {"all://": PROXY_URL}, PROXY_URL), - ("http://example.com", {"http://": PROXY_URL}, PROXY_URL), - ("http://example.com", {"all://example.com": PROXY_URL}, PROXY_URL), - ("http://example.com", {"http://example.com": PROXY_URL}, PROXY_URL), - ("http://example.com", {"http://example.com:80": PROXY_URL}, PROXY_URL), - ("http://example.com:8080", {"http://example.com:8080": PROXY_URL}, PROXY_URL), - ("http://example.com:8080", {"http://example.com": PROXY_URL}, PROXY_URL), - ( - "http://example.com", - { - "all://": PROXY_URL + ":1", - "http://": PROXY_URL + ":2", - "all://example.com": PROXY_URL + ":3", - "http://example.com": PROXY_URL + ":4", - }, - PROXY_URL + ":4", - ), - ( - "http://example.com", - { - "all://": PROXY_URL + ":1", - "http://": PROXY_URL + ":2", - "all://example.com": PROXY_URL + ":3", - }, - PROXY_URL + ":3", - ), - ( - "http://example.com", - {"all://": PROXY_URL + ":1", "http://": PROXY_URL + ":2"}, - PROXY_URL + ":2", - ), - ], -) -def test_transport_for_request(url, proxies, expected): - mounts = {key: httpx.HTTPTransport(proxy=value) for key, value in proxies.items()} - client = httpx.Client(mounts=mounts) - - transport = client._transport_for_url(httpx.URL(url)) - - if expected is None: - assert transport is client._transport - else: - assert isinstance(transport, httpx.HTTPTransport) - assert isinstance(transport._pool, httpcore.HTTPProxy) - assert transport._pool._proxy_url == url_to_origin(expected) - - -@pytest.mark.anyio -@pytest.mark.network -async def test_async_proxy_close(): - try: - transport = httpx.AsyncHTTPTransport(proxy=PROXY_URL) - client = httpx.AsyncClient(mounts={"https://": transport}) - await client.get("http://example.com") - finally: - await client.aclose() - - -@pytest.mark.network -def test_sync_proxy_close(): - try: - transport = httpx.HTTPTransport(proxy=PROXY_URL) - client = httpx.Client(mounts={"https://": transport}) - client.get("http://example.com") - finally: - client.close() - - -def test_unsupported_proxy_scheme(): - with pytest.raises(ValueError): - httpx.Client(proxy="ftp://127.0.0.1") - - -@pytest.mark.parametrize( - ["url", "env", "expected"], - [ - ("http://google.com", {}, None), - ( - "http://google.com", - {"HTTP_PROXY": "http://example.com"}, - "http://example.com", - ), - # Auto prepend http scheme - ("http://google.com", {"HTTP_PROXY": "example.com"}, "http://example.com"), - ( - "http://google.com", - {"HTTP_PROXY": "http://example.com", "NO_PROXY": "google.com"}, - None, - ), - # Everything proxied when NO_PROXY is empty/unset - ( - "http://127.0.0.1", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": ""}, - "http://localhost:123", - ), - # Not proxied if NO_PROXY matches URL. - ( - "http://127.0.0.1", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "127.0.0.1"}, - None, - ), - # Proxied if NO_PROXY scheme does not match URL. - ( - "http://127.0.0.1", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "https://127.0.0.1"}, - "http://localhost:123", - ), - # Proxied if NO_PROXY scheme does not match host. - ( - "http://127.0.0.1", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "1.1.1.1"}, - "http://localhost:123", - ), - # Not proxied if NO_PROXY matches host domain suffix. - ( - "http://courses.mit.edu", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "mit.edu"}, - None, - ), - # Proxied even though NO_PROXY matches host domain *prefix*. - ( - "https://mit.edu.info", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "mit.edu"}, - "http://localhost:123", - ), - # Not proxied if one item in NO_PROXY case matches host domain suffix. - ( - "https://mit.edu.info", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "mit.edu,edu.info"}, - None, - ), - # Not proxied if one item in NO_PROXY case matches host domain suffix. - # May include whitespace. - ( - "https://mit.edu.info", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "mit.edu, edu.info"}, - None, - ), - # Proxied if no items in NO_PROXY match. - ( - "https://mit.edu.info", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "mit.edu,mit.info"}, - "http://localhost:123", - ), - # Proxied if NO_PROXY domain doesn't match. - ( - "https://foo.example.com", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "www.example.com"}, - "http://localhost:123", - ), - # Not proxied for subdomains matching NO_PROXY, with a leading ".". - ( - "https://www.example1.com", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": ".example1.com"}, - None, - ), - # Proxied, because NO_PROXY subdomains only match if "." separated. - ( - "https://www.example2.com", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "ample2.com"}, - "http://localhost:123", - ), - # No requests are proxied if NO_PROXY="*" is set. - ( - "https://www.example3.com", - {"ALL_PROXY": "http://localhost:123", "NO_PROXY": "*"}, - None, - ), - ], -) -@pytest.mark.parametrize("client_class", [httpx.Client, httpx.AsyncClient]) -def test_proxies_environ(monkeypatch, client_class, url, env, expected): - for name, value in env.items(): - monkeypatch.setenv(name, value) - - client = client_class() - transport = client._transport_for_url(httpx.URL(url)) - - if expected is None: - assert transport == client._transport - else: - assert transport._pool._proxy_url == url_to_origin(expected) - - -@pytest.mark.parametrize( - ["proxies", "is_valid"], - [ - ({"http": "http://127.0.0.1"}, False), - ({"https": "http://127.0.0.1"}, False), - ({"all": "http://127.0.0.1"}, False), - ({"http://": "http://127.0.0.1"}, True), - ({"https://": "http://127.0.0.1"}, True), - ({"all://": "http://127.0.0.1"}, True), - ], -) -def test_for_deprecated_proxy_params(proxies, is_valid): - mounts = {key: httpx.HTTPTransport(proxy=value) for key, value in proxies.items()} - - if not is_valid: - with pytest.raises(ValueError): - httpx.Client(mounts=mounts) - else: - httpx.Client(mounts=mounts) - - -def test_proxy_with_mounts(): - proxy_transport = httpx.HTTPTransport(proxy="http://127.0.0.1") - client = httpx.Client(mounts={"http://": proxy_transport}) - - transport = client._transport_for_url(httpx.URL("http://example.com")) - assert transport == proxy_transport diff --git a/tests/client/test_queryparams.py b/tests/client/test_queryparams.py deleted file mode 100644 index 1c6d587309..0000000000 --- a/tests/client/test_queryparams.py +++ /dev/null @@ -1,35 +0,0 @@ -import httpx - - -def hello_world(request: httpx.Request) -> httpx.Response: - return httpx.Response(200, text="Hello, world") - - -def test_client_queryparams(): - client = httpx.Client(params={"a": "b"}) - assert isinstance(client.params, httpx.QueryParams) - assert client.params["a"] == "b" - - -def test_client_queryparams_string(): - client = httpx.Client(params="a=b") - assert isinstance(client.params, httpx.QueryParams) - assert client.params["a"] == "b" - - client = httpx.Client() - client.params = "a=b" - assert isinstance(client.params, httpx.QueryParams) - assert client.params["a"] == "b" - - -def test_client_queryparams_echo(): - url = "http://example.org/echo_queryparams" - client_queryparams = "first=str" - request_queryparams = {"second": "dict"} - client = httpx.Client( - transport=httpx.MockTransport(hello_world), params=client_queryparams - ) - response = client.get(url, params=request_queryparams) - - assert response.status_code == 200 - assert response.url == "http://example.org/echo_queryparams?first=str&second=dict" diff --git a/tests/client/test_redirects.py b/tests/client/test_redirects.py deleted file mode 100644 index f65827134c..0000000000 --- a/tests/client/test_redirects.py +++ /dev/null @@ -1,447 +0,0 @@ -import typing - -import pytest - -import httpx - - -def redirects(request: httpx.Request) -> httpx.Response: - if request.url.scheme not in ("http", "https"): - raise httpx.UnsupportedProtocol(f"Scheme {request.url.scheme!r} not supported.") - - if request.url.path == "/redirect_301": - status_code = httpx.codes.MOVED_PERMANENTLY - content = b"here" - headers = {"location": "https://example.org/"} - return httpx.Response(status_code, headers=headers, content=content) - - elif request.url.path == "/redirect_302": - status_code = httpx.codes.FOUND - headers = {"location": "https://example.org/"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/redirect_303": - status_code = httpx.codes.SEE_OTHER - headers = {"location": "https://example.org/"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/relative_redirect": - status_code = httpx.codes.SEE_OTHER - headers = {"location": "/"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/malformed_redirect": - status_code = httpx.codes.SEE_OTHER - headers = {"location": "https://:443/"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/invalid_redirect": - status_code = httpx.codes.SEE_OTHER - raw_headers = [(b"location", "https://😇/".encode("utf-8"))] - return httpx.Response(status_code, headers=raw_headers) - - elif request.url.path == "/no_scheme_redirect": - status_code = httpx.codes.SEE_OTHER - headers = {"location": "//example.org/"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/multiple_redirects": - params = httpx.QueryParams(request.url.query) - count = int(params.get("count", "0")) - redirect_count = count - 1 - status_code = httpx.codes.SEE_OTHER if count else httpx.codes.OK - if count: - location = "/multiple_redirects" - if redirect_count: - location += f"?count={redirect_count}" - headers = {"location": location} - else: - headers = {} - return httpx.Response(status_code, headers=headers) - - if request.url.path == "/redirect_loop": - status_code = httpx.codes.SEE_OTHER - headers = {"location": "/redirect_loop"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/cross_domain": - status_code = httpx.codes.SEE_OTHER - headers = {"location": "https://example.org/cross_domain_target"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/cross_domain_target": - status_code = httpx.codes.OK - data = { - "body": request.content.decode("ascii"), - "headers": dict(request.headers), - } - return httpx.Response(status_code, json=data) - - elif request.url.path == "/redirect_body": - status_code = httpx.codes.PERMANENT_REDIRECT - headers = {"location": "/redirect_body_target"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/redirect_no_body": - status_code = httpx.codes.SEE_OTHER - headers = {"location": "/redirect_body_target"} - return httpx.Response(status_code, headers=headers) - - elif request.url.path == "/redirect_body_target": - data = { - "body": request.content.decode("ascii"), - "headers": dict(request.headers), - } - return httpx.Response(200, json=data) - - elif request.url.path == "/cross_subdomain": - if request.headers["Host"] != "www.example.org": - status_code = httpx.codes.PERMANENT_REDIRECT - headers = {"location": "https://www.example.org/cross_subdomain"} - return httpx.Response(status_code, headers=headers) - else: - return httpx.Response(200, text="Hello, world!") - - elif request.url.path == "/redirect_custom_scheme": - status_code = httpx.codes.MOVED_PERMANENTLY - headers = {"location": "market://details?id=42"} - return httpx.Response(status_code, headers=headers) - - if request.method == "HEAD": - return httpx.Response(200) - - return httpx.Response(200, html="Hello, world!") - - -def test_redirect_301(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.post("https://example.org/redirect_301", follow_redirects=True) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/" - assert len(response.history) == 1 - - -def test_redirect_302(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.post("https://example.org/redirect_302", follow_redirects=True) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/" - assert len(response.history) == 1 - - -def test_redirect_303(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.get("https://example.org/redirect_303", follow_redirects=True) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/" - assert len(response.history) == 1 - - -def test_next_request(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - request = client.build_request("POST", "https://example.org/redirect_303") - response = client.send(request, follow_redirects=False) - assert response.status_code == httpx.codes.SEE_OTHER - assert response.url == "https://example.org/redirect_303" - assert response.next_request is not None - - response = client.send(response.next_request, follow_redirects=False) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/" - assert response.next_request is None - - -@pytest.mark.anyio -async def test_async_next_request(): - async with httpx.AsyncClient(transport=httpx.MockTransport(redirects)) as client: - request = client.build_request("POST", "https://example.org/redirect_303") - response = await client.send(request, follow_redirects=False) - assert response.status_code == httpx.codes.SEE_OTHER - assert response.url == "https://example.org/redirect_303" - assert response.next_request is not None - - response = await client.send(response.next_request, follow_redirects=False) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/" - assert response.next_request is None - - -def test_head_redirect(): - """ - Contrary to Requests, redirects remain enabled by default for HEAD requests. - """ - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.head("https://example.org/redirect_302", follow_redirects=True) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/" - assert response.request.method == "HEAD" - assert len(response.history) == 1 - assert response.text == "" - - -def test_relative_redirect(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.get( - "https://example.org/relative_redirect", follow_redirects=True - ) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/" - assert len(response.history) == 1 - - -def test_malformed_redirect(): - # https://github.com/encode/httpx/issues/771 - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.get( - "http://example.org/malformed_redirect", follow_redirects=True - ) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org:443/" - assert len(response.history) == 1 - - -def test_invalid_redirect(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - with pytest.raises(httpx.RemoteProtocolError): - client.get("http://example.org/invalid_redirect", follow_redirects=True) - - -def test_no_scheme_redirect(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.get( - "https://example.org/no_scheme_redirect", follow_redirects=True - ) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/" - assert len(response.history) == 1 - - -def test_fragment_redirect(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.get( - "https://example.org/relative_redirect#fragment", follow_redirects=True - ) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/#fragment" - assert len(response.history) == 1 - - -def test_multiple_redirects(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - response = client.get( - "https://example.org/multiple_redirects?count=20", follow_redirects=True - ) - assert response.status_code == httpx.codes.OK - assert response.url == "https://example.org/multiple_redirects" - assert len(response.history) == 20 - assert response.history[0].url == "https://example.org/multiple_redirects?count=20" - assert response.history[1].url == "https://example.org/multiple_redirects?count=19" - assert len(response.history[0].history) == 0 - assert len(response.history[1].history) == 1 - - -@pytest.mark.anyio -async def test_async_too_many_redirects(): - async with httpx.AsyncClient(transport=httpx.MockTransport(redirects)) as client: - with pytest.raises(httpx.TooManyRedirects): - await client.get( - "https://example.org/multiple_redirects?count=21", follow_redirects=True - ) - - -def test_sync_too_many_redirects(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - with pytest.raises(httpx.TooManyRedirects): - client.get( - "https://example.org/multiple_redirects?count=21", follow_redirects=True - ) - - -def test_redirect_loop(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - with pytest.raises(httpx.TooManyRedirects): - client.get("https://example.org/redirect_loop", follow_redirects=True) - - -def test_cross_domain_redirect_with_auth_header(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "https://example.com/cross_domain" - headers = {"Authorization": "abc"} - response = client.get(url, headers=headers, follow_redirects=True) - assert response.url == "https://example.org/cross_domain_target" - assert "authorization" not in response.json()["headers"] - - -def test_cross_domain_https_redirect_with_auth_header(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "http://example.com/cross_domain" - headers = {"Authorization": "abc"} - response = client.get(url, headers=headers, follow_redirects=True) - assert response.url == "https://example.org/cross_domain_target" - assert "authorization" not in response.json()["headers"] - - -def test_cross_domain_redirect_with_auth(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "https://example.com/cross_domain" - response = client.get(url, auth=("user", "pass"), follow_redirects=True) - assert response.url == "https://example.org/cross_domain_target" - assert "authorization" not in response.json()["headers"] - - -def test_same_domain_redirect(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "https://example.org/cross_domain" - headers = {"Authorization": "abc"} - response = client.get(url, headers=headers, follow_redirects=True) - assert response.url == "https://example.org/cross_domain_target" - assert response.json()["headers"]["authorization"] == "abc" - - -def test_same_domain_https_redirect_with_auth_header(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "http://example.org/cross_domain" - headers = {"Authorization": "abc"} - response = client.get(url, headers=headers, follow_redirects=True) - assert response.url == "https://example.org/cross_domain_target" - assert response.json()["headers"]["authorization"] == "abc" - - -def test_body_redirect(): - """ - A 308 redirect should preserve the request body. - """ - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "https://example.org/redirect_body" - content = b"Example request body" - response = client.post(url, content=content, follow_redirects=True) - assert response.url == "https://example.org/redirect_body_target" - assert response.json()["body"] == "Example request body" - assert "content-length" in response.json()["headers"] - - -def test_no_body_redirect(): - """ - A 303 redirect should remove the request body. - """ - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "https://example.org/redirect_no_body" - content = b"Example request body" - response = client.post(url, content=content, follow_redirects=True) - assert response.url == "https://example.org/redirect_body_target" - assert response.json()["body"] == "" - assert "content-length" not in response.json()["headers"] - - -def test_can_stream_if_no_redirect(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "https://example.org/redirect_301" - with client.stream("GET", url, follow_redirects=False) as response: - pass - assert response.status_code == httpx.codes.MOVED_PERMANENTLY - assert response.headers["location"] == "https://example.org/" - - -class ConsumeBodyTransport(httpx.MockTransport): - def handle_request(self, request: httpx.Request) -> httpx.Response: - assert isinstance(request.stream, httpx.SyncByteStream) - list(request.stream) - return self.handler(request) # type: ignore[return-value] - - -def test_cannot_redirect_streaming_body(): - client = httpx.Client(transport=ConsumeBodyTransport(redirects)) - url = "https://example.org/redirect_body" - - def streaming_body() -> typing.Iterator[bytes]: - yield b"Example request body" # pragma: no cover - - with pytest.raises(httpx.StreamConsumed): - client.post(url, content=streaming_body(), follow_redirects=True) - - -def test_cross_subdomain_redirect(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - url = "https://example.com/cross_subdomain" - response = client.get(url, follow_redirects=True) - assert response.url == "https://www.example.org/cross_subdomain" - - -def cookie_sessions(request: httpx.Request) -> httpx.Response: - if request.url.path == "/": - cookie = request.headers.get("Cookie") - if cookie is not None: - content = b"Logged in" - else: - content = b"Not logged in" - return httpx.Response(200, content=content) - - elif request.url.path == "/login": - status_code = httpx.codes.SEE_OTHER - headers = { - "location": "/", - "set-cookie": ( - "session=eyJ1c2VybmFtZSI6ICJ0b21; path=/; Max-Age=1209600; " - "httponly; samesite=lax" - ), - } - return httpx.Response(status_code, headers=headers) - - else: - assert request.url.path == "/logout" - status_code = httpx.codes.SEE_OTHER - headers = { - "location": "/", - "set-cookie": ( - "session=null; path=/; expires=Thu, 01 Jan 1970 00:00:00 GMT; " - "httponly; samesite=lax" - ), - } - return httpx.Response(status_code, headers=headers) - - -def test_redirect_cookie_behavior(): - client = httpx.Client( - transport=httpx.MockTransport(cookie_sessions), follow_redirects=True - ) - - # The client is not logged in. - response = client.get("https://example.com/") - assert response.url == "https://example.com/" - assert response.text == "Not logged in" - - # Login redirects to the homepage, setting a session cookie. - response = client.post("https://example.com/login") - assert response.url == "https://example.com/" - assert response.text == "Logged in" - - # The client is logged in. - response = client.get("https://example.com/") - assert response.url == "https://example.com/" - assert response.text == "Logged in" - - # Logout redirects to the homepage, expiring the session cookie. - response = client.post("https://example.com/logout") - assert response.url == "https://example.com/" - assert response.text == "Not logged in" - - # The client is not logged in. - response = client.get("https://example.com/") - assert response.url == "https://example.com/" - assert response.text == "Not logged in" - - -def test_redirect_custom_scheme(): - client = httpx.Client(transport=httpx.MockTransport(redirects)) - with pytest.raises(httpx.UnsupportedProtocol) as e: - client.post("https://example.org/redirect_custom_scheme", follow_redirects=True) - assert str(e.value) == "Scheme 'market' not supported." - - -@pytest.mark.anyio -async def test_async_invalid_redirect(): - async with httpx.AsyncClient(transport=httpx.MockTransport(redirects)) as client: - with pytest.raises(httpx.RemoteProtocolError): - await client.get( - "http://example.org/invalid_redirect", follow_redirects=True - ) diff --git a/tests/common.py b/tests/common.py deleted file mode 100644 index 064c25a645..0000000000 --- a/tests/common.py +++ /dev/null @@ -1,4 +0,0 @@ -import pathlib - -TESTS_DIR = pathlib.Path(__file__).parent -FIXTURES_DIR = TESTS_DIR / "fixtures" diff --git a/tests/concurrency.py b/tests/concurrency.py deleted file mode 100644 index a8ed55829a..0000000000 --- a/tests/concurrency.py +++ /dev/null @@ -1,15 +0,0 @@ -""" -Async environment-agnostic concurrency utilities that are only used in tests. -""" - -import asyncio - -import sniffio -import trio - - -async def sleep(seconds: float) -> None: - if sniffio.current_async_library() == "trio": - await trio.sleep(seconds) # pragma: no cover - else: - await asyncio.sleep(seconds) diff --git a/tests/conftest.py b/tests/conftest.py deleted file mode 100644 index 858bca1397..0000000000 --- a/tests/conftest.py +++ /dev/null @@ -1,287 +0,0 @@ -import asyncio -import json -import os -import threading -import time -import typing - -import pytest -import trustme -from cryptography.hazmat.backends import default_backend -from cryptography.hazmat.primitives.serialization import ( - BestAvailableEncryption, - Encoding, - PrivateFormat, - load_pem_private_key, -) -from uvicorn.config import Config -from uvicorn.server import Server - -import httpx -from tests.concurrency import sleep - -ENVIRONMENT_VARIABLES = { - "SSL_CERT_FILE", - "SSL_CERT_DIR", - "HTTP_PROXY", - "HTTPS_PROXY", - "ALL_PROXY", - "NO_PROXY", - "SSLKEYLOGFILE", -} - - -@pytest.fixture(scope="function", autouse=True) -def clean_environ(): - """Keeps os.environ clean for every test without having to mock os.environ""" - original_environ = os.environ.copy() - os.environ.clear() - os.environ.update( - { - k: v - for k, v in original_environ.items() - if k not in ENVIRONMENT_VARIABLES and k.lower() not in ENVIRONMENT_VARIABLES - } - ) - yield - os.environ.clear() - os.environ.update(original_environ) - - -Message = typing.Dict[str, typing.Any] -Receive = typing.Callable[[], typing.Awaitable[Message]] -Send = typing.Callable[ - [typing.Dict[str, typing.Any]], typing.Coroutine[None, None, None] -] -Scope = typing.Dict[str, typing.Any] - - -async def app(scope: Scope, receive: Receive, send: Send) -> None: - assert scope["type"] == "http" - if scope["path"].startswith("/slow_response"): - await slow_response(scope, receive, send) - elif scope["path"].startswith("/status"): - await status_code(scope, receive, send) - elif scope["path"].startswith("/echo_body"): - await echo_body(scope, receive, send) - elif scope["path"].startswith("/echo_binary"): - await echo_binary(scope, receive, send) - elif scope["path"].startswith("/echo_headers"): - await echo_headers(scope, receive, send) - elif scope["path"].startswith("/redirect_301"): - await redirect_301(scope, receive, send) - elif scope["path"].startswith("/json"): - await hello_world_json(scope, receive, send) - else: - await hello_world(scope, receive, send) - - -async def hello_world(scope: Scope, receive: Receive, send: Send) -> None: - await send( - { - "type": "http.response.start", - "status": 200, - "headers": [[b"content-type", b"text/plain"]], - } - ) - await send({"type": "http.response.body", "body": b"Hello, world!"}) - - -async def hello_world_json(scope: Scope, receive: Receive, send: Send) -> None: - await send( - { - "type": "http.response.start", - "status": 200, - "headers": [[b"content-type", b"application/json"]], - } - ) - await send({"type": "http.response.body", "body": b'{"Hello": "world!"}'}) - - -async def slow_response(scope: Scope, receive: Receive, send: Send) -> None: - await send( - { - "type": "http.response.start", - "status": 200, - "headers": [[b"content-type", b"text/plain"]], - } - ) - await sleep(1.0) # Allow triggering a read timeout. - await send({"type": "http.response.body", "body": b"Hello, world!"}) - - -async def status_code(scope: Scope, receive: Receive, send: Send) -> None: - status_code = int(scope["path"].replace("/status/", "")) - await send( - { - "type": "http.response.start", - "status": status_code, - "headers": [[b"content-type", b"text/plain"]], - } - ) - await send({"type": "http.response.body", "body": b"Hello, world!"}) - - -async def echo_body(scope: Scope, receive: Receive, send: Send) -> None: - body = b"" - more_body = True - - while more_body: - message = await receive() - body += message.get("body", b"") - more_body = message.get("more_body", False) - - await send( - { - "type": "http.response.start", - "status": 200, - "headers": [[b"content-type", b"text/plain"]], - } - ) - await send({"type": "http.response.body", "body": body}) - - -async def echo_binary(scope: Scope, receive: Receive, send: Send) -> None: - body = b"" - more_body = True - - while more_body: - message = await receive() - body += message.get("body", b"") - more_body = message.get("more_body", False) - - await send( - { - "type": "http.response.start", - "status": 200, - "headers": [[b"content-type", b"application/octet-stream"]], - } - ) - await send({"type": "http.response.body", "body": body}) - - -async def echo_headers(scope: Scope, receive: Receive, send: Send) -> None: - body = { - name.capitalize().decode(): value.decode() - for name, value in scope.get("headers", []) - } - await send( - { - "type": "http.response.start", - "status": 200, - "headers": [[b"content-type", b"application/json"]], - } - ) - await send({"type": "http.response.body", "body": json.dumps(body).encode()}) - - -async def redirect_301(scope: Scope, receive: Receive, send: Send) -> None: - await send( - {"type": "http.response.start", "status": 301, "headers": [[b"location", b"/"]]} - ) - await send({"type": "http.response.body"}) - - -@pytest.fixture(scope="session") -def cert_authority(): - return trustme.CA() - - -@pytest.fixture(scope="session") -def localhost_cert(cert_authority): - return cert_authority.issue_cert("localhost") - - -@pytest.fixture(scope="session") -def cert_pem_file(localhost_cert): - with localhost_cert.cert_chain_pems[0].tempfile() as tmp: - yield tmp - - -@pytest.fixture(scope="session") -def cert_private_key_file(localhost_cert): - with localhost_cert.private_key_pem.tempfile() as tmp: - yield tmp - - -@pytest.fixture(scope="session") -def cert_encrypted_private_key_file(localhost_cert): - # Deserialize the private key and then reserialize with a password - private_key = load_pem_private_key( - localhost_cert.private_key_pem.bytes(), password=None, backend=default_backend() - ) - encrypted_private_key_pem = trustme.Blob( - private_key.private_bytes( - Encoding.PEM, - PrivateFormat.TraditionalOpenSSL, - BestAvailableEncryption(password=b"password"), - ) - ) - with encrypted_private_key_pem.tempfile() as tmp: - yield tmp - - -class TestServer(Server): - @property - def url(self) -> httpx.URL: - protocol = "https" if self.config.is_ssl else "http" - return httpx.URL(f"{protocol}://{self.config.host}:{self.config.port}/") - - def install_signal_handlers(self) -> None: - # Disable the default installation of handlers for signals such as SIGTERM, - # because it can only be done in the main thread. - pass # pragma: nocover - - async def serve(self, sockets=None): - self.restart_requested = asyncio.Event() - - loop = asyncio.get_event_loop() - tasks = { - loop.create_task(super().serve(sockets=sockets)), - loop.create_task(self.watch_restarts()), - } - await asyncio.wait(tasks) - - async def restart(self) -> None: # pragma: no cover - # This coroutine may be called from a different thread than the one the - # server is running on, and from an async environment that's not asyncio. - # For this reason, we use an event to coordinate with the server - # instead of calling shutdown()/startup() directly, and should not make - # any asyncio-specific operations. - self.started = False - self.restart_requested.set() - while not self.started: - await sleep(0.2) - - async def watch_restarts(self) -> None: # pragma: no cover - while True: - if self.should_exit: - return - - try: - await asyncio.wait_for(self.restart_requested.wait(), timeout=0.1) - except asyncio.TimeoutError: - continue - - self.restart_requested.clear() - await self.shutdown() - await self.startup() - - -def serve_in_thread(server: TestServer) -> typing.Iterator[TestServer]: - thread = threading.Thread(target=server.run) - thread.start() - try: - while not server.started: - time.sleep(1e-3) - yield server - finally: - server.should_exit = True - thread.join() - - -@pytest.fixture(scope="session") -def server() -> typing.Iterator[TestServer]: - config = Config(app=app, lifespan="off", loop="asyncio") - server = TestServer(config=config) - yield from serve_in_thread(server) diff --git a/tests/fixtures/.netrc b/tests/fixtures/.netrc deleted file mode 100644 index ed65ee7d3d..0000000000 --- a/tests/fixtures/.netrc +++ /dev/null @@ -1,3 +0,0 @@ -machine netrcexample.org -login example-username -password example-password \ No newline at end of file diff --git a/tests/fixtures/.netrc-nopassword b/tests/fixtures/.netrc-nopassword deleted file mode 100644 index 5575bee704..0000000000 --- a/tests/fixtures/.netrc-nopassword +++ /dev/null @@ -1,2 +0,0 @@ -machine netrcexample.org -login example-username diff --git a/tests/models/__init__.py b/tests/models/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/tests/models/test_cookies.py b/tests/models/test_cookies.py deleted file mode 100644 index f7abe11ad4..0000000000 --- a/tests/models/test_cookies.py +++ /dev/null @@ -1,98 +0,0 @@ -import http - -import pytest - -import httpx - - -def test_cookies(): - cookies = httpx.Cookies({"name": "value"}) - assert cookies["name"] == "value" - assert "name" in cookies - assert len(cookies) == 1 - assert dict(cookies) == {"name": "value"} - assert bool(cookies) is True - - del cookies["name"] - assert "name" not in cookies - assert len(cookies) == 0 - assert dict(cookies) == {} - assert bool(cookies) is False - - -def test_cookies_update(): - cookies = httpx.Cookies() - more_cookies = httpx.Cookies() - more_cookies.set("name", "value", domain="example.com") - - cookies.update(more_cookies) - assert dict(cookies) == {"name": "value"} - assert cookies.get("name", domain="example.com") == "value" - - -def test_cookies_with_domain(): - cookies = httpx.Cookies() - cookies.set("name", "value", domain="example.com") - cookies.set("name", "value", domain="example.org") - - with pytest.raises(httpx.CookieConflict): - cookies["name"] - - cookies.clear(domain="example.com") - assert len(cookies) == 1 - - -def test_cookies_with_domain_and_path(): - cookies = httpx.Cookies() - cookies.set("name", "value", domain="example.com", path="/subpath/1") - cookies.set("name", "value", domain="example.com", path="/subpath/2") - cookies.clear(domain="example.com", path="/subpath/1") - assert len(cookies) == 1 - cookies.delete("name", domain="example.com", path="/subpath/2") - assert len(cookies) == 0 - - -def test_multiple_set_cookie(): - jar = http.cookiejar.CookieJar() - headers = [ - ( - b"Set-Cookie", - b"1P_JAR=2020-08-09-18; expires=Tue, 08-Sep-2099 18:33:35 GMT; " - b"path=/; domain=.example.org; Secure", - ), - ( - b"Set-Cookie", - b"NID=204=KWdXOuypc86YvRfBSiWoW1dEXfSl_5qI7sxZY4umlk4J35yNTeNEkw15" - b"MRaujK6uYCwkrtjihTTXZPp285z_xDOUzrdHt4dj0Z5C0VOpbvdLwRdHatHAzQs7" - b"7TsaiWY78a3qU9r7KP_RbSLvLl2hlhnWFR2Hp5nWKPsAcOhQgSg; expires=Mon, " - b"08-Feb-2099 18:33:35 GMT; path=/; domain=.example.org; HttpOnly", - ), - ] - request = httpx.Request("GET", "https://www.example.org") - response = httpx.Response(200, request=request, headers=headers) - - cookies = httpx.Cookies(jar) - cookies.extract_cookies(response) - - assert len(cookies) == 2 - - -def test_cookies_can_be_a_list_of_tuples(): - cookies_val = [("name1", "val1"), ("name2", "val2")] - - cookies = httpx.Cookies(cookies_val) - - assert len(cookies.items()) == 2 - for k, v in cookies_val: - assert cookies[k] == v - - -def test_cookies_repr(): - cookies = httpx.Cookies() - cookies.set(name="foo", value="bar", domain="http://blah.com") - cookies.set(name="fizz", value="buzz", domain="http://hello.com") - - assert repr(cookies) == ( - "," - " ]>" - ) diff --git a/tests/models/test_headers.py b/tests/models/test_headers.py deleted file mode 100644 index a87a446784..0000000000 --- a/tests/models/test_headers.py +++ /dev/null @@ -1,219 +0,0 @@ -import pytest - -import httpx - - -def test_headers(): - h = httpx.Headers([("a", "123"), ("a", "456"), ("b", "789")]) - assert "a" in h - assert "A" in h - assert "b" in h - assert "B" in h - assert "c" not in h - assert h["a"] == "123, 456" - assert h.get("a") == "123, 456" - assert h.get("nope", default=None) is None - assert h.get_list("a") == ["123", "456"] - - assert list(h.keys()) == ["a", "b"] - assert list(h.values()) == ["123, 456", "789"] - assert list(h.items()) == [("a", "123, 456"), ("b", "789")] - assert h.multi_items() == [("a", "123"), ("a", "456"), ("b", "789")] - assert list(h) == ["a", "b"] - assert dict(h) == {"a": "123, 456", "b": "789"} - assert repr(h) == "Headers([('a', '123'), ('a', '456'), ('b', '789')])" - assert h == [("a", "123"), ("b", "789"), ("a", "456")] - assert h == [("a", "123"), ("A", "456"), ("b", "789")] - assert h == {"a": "123", "A": "456", "b": "789"} - assert h != "a: 123\nA: 456\nb: 789" - - h = httpx.Headers({"a": "123", "b": "789"}) - assert h["A"] == "123" - assert h["B"] == "789" - assert h.raw == [(b"a", b"123"), (b"b", b"789")] - assert repr(h) == "Headers({'a': '123', 'b': '789'})" - - -def test_header_mutations(): - h = httpx.Headers() - assert dict(h) == {} - h["a"] = "1" - assert dict(h) == {"a": "1"} - h["a"] = "2" - assert dict(h) == {"a": "2"} - h.setdefault("a", "3") - assert dict(h) == {"a": "2"} - h.setdefault("b", "4") - assert dict(h) == {"a": "2", "b": "4"} - del h["a"] - assert dict(h) == {"b": "4"} - assert h.raw == [(b"b", b"4")] - - -def test_copy_headers_method(): - headers = httpx.Headers({"custom": "example"}) - headers_copy = headers.copy() - assert headers == headers_copy - assert headers is not headers_copy - - -def test_copy_headers_init(): - headers = httpx.Headers({"custom": "example"}) - headers_copy = httpx.Headers(headers) - assert headers == headers_copy - - -def test_headers_insert_retains_ordering(): - headers = httpx.Headers({"a": "a", "b": "b", "c": "c"}) - headers["b"] = "123" - assert list(headers.values()) == ["a", "123", "c"] - - -def test_headers_insert_appends_if_new(): - headers = httpx.Headers({"a": "a", "b": "b", "c": "c"}) - headers["d"] = "123" - assert list(headers.values()) == ["a", "b", "c", "123"] - - -def test_headers_insert_removes_all_existing(): - headers = httpx.Headers([("a", "123"), ("a", "456")]) - headers["a"] = "789" - assert dict(headers) == {"a": "789"} - - -def test_headers_delete_removes_all_existing(): - headers = httpx.Headers([("a", "123"), ("a", "456")]) - del headers["a"] - assert dict(headers) == {} - - -def test_headers_dict_repr(): - """ - Headers should display with a dict repr by default. - """ - headers = httpx.Headers({"custom": "example"}) - assert repr(headers) == "Headers({'custom': 'example'})" - - -def test_headers_encoding_in_repr(): - """ - Headers should display an encoding in the repr if required. - """ - headers = httpx.Headers({b"custom": "example ☃".encode("utf-8")}) - assert repr(headers) == "Headers({'custom': 'example ☃'}, encoding='utf-8')" - - -def test_headers_list_repr(): - """ - Headers should display with a list repr if they include multiple identical keys. - """ - headers = httpx.Headers([("custom", "example 1"), ("custom", "example 2")]) - assert ( - repr(headers) == "Headers([('custom', 'example 1'), ('custom', 'example 2')])" - ) - - -def test_headers_decode_ascii(): - """ - Headers should decode as ascii by default. - """ - raw_headers = [(b"Custom", b"Example")] - headers = httpx.Headers(raw_headers) - assert dict(headers) == {"custom": "Example"} - assert headers.encoding == "ascii" - - -def test_headers_decode_utf_8(): - """ - Headers containing non-ascii codepoints should default to decoding as utf-8. - """ - raw_headers = [(b"Custom", "Code point: ☃".encode("utf-8"))] - headers = httpx.Headers(raw_headers) - assert dict(headers) == {"custom": "Code point: ☃"} - assert headers.encoding == "utf-8" - - -def test_headers_decode_iso_8859_1(): - """ - Headers containing non-UTF-8 codepoints should default to decoding as iso-8859-1. - """ - raw_headers = [(b"Custom", "Code point: ÿ".encode("iso-8859-1"))] - headers = httpx.Headers(raw_headers) - assert dict(headers) == {"custom": "Code point: ÿ"} - assert headers.encoding == "iso-8859-1" - - -def test_headers_decode_explicit_encoding(): - """ - An explicit encoding may be set on headers in order to force a - particular decoding. - """ - raw_headers = [(b"Custom", "Code point: ☃".encode("utf-8"))] - headers = httpx.Headers(raw_headers) - headers.encoding = "iso-8859-1" - assert dict(headers) == {"custom": "Code point: â\x98\x83"} - assert headers.encoding == "iso-8859-1" - - -def test_multiple_headers(): - """ - `Headers.get_list` should support both split_commas=False and split_commas=True. - """ - h = httpx.Headers([("set-cookie", "a, b"), ("set-cookie", "c")]) - assert h.get_list("Set-Cookie") == ["a, b", "c"] - - h = httpx.Headers([("vary", "a, b"), ("vary", "c")]) - assert h.get_list("Vary", split_commas=True) == ["a", "b", "c"] - - -@pytest.mark.parametrize("header", ["authorization", "proxy-authorization"]) -def test_sensitive_headers(header): - """ - Some headers should be obfuscated because they contain sensitive data. - """ - value = "s3kr3t" - h = httpx.Headers({header: value}) - assert repr(h) == "Headers({'%s': '[secure]'})" % header - - -@pytest.mark.parametrize( - "headers, output", - [ - ([("content-type", "text/html")], [("content-type", "text/html")]), - ([("authorization", "s3kr3t")], [("authorization", "[secure]")]), - ([("proxy-authorization", "s3kr3t")], [("proxy-authorization", "[secure]")]), - ], -) -def test_obfuscate_sensitive_headers(headers, output): - as_dict = {k: v for k, v in output} - headers_class = httpx.Headers({k: v for k, v in headers}) - assert repr(headers_class) == f"Headers({as_dict!r})" - - -@pytest.mark.parametrize( - "value, expected", - ( - ( - '; rel=front; type="image/jpeg"', - [{"url": "http:/.../front.jpeg", "rel": "front", "type": "image/jpeg"}], - ), - ("", [{"url": "http:/.../front.jpeg"}]), - (";", [{"url": "http:/.../front.jpeg"}]), - ( - '; type="image/jpeg",;', - [ - {"url": "http:/.../front.jpeg", "type": "image/jpeg"}, - {"url": "http://.../back.jpeg"}, - ], - ), - ("", []), - ), -) -def test_parse_header_links(value, expected): - all_links = httpx.Response(200, headers={"link": value}).links.values() - assert all(link in all_links for link in expected) - - -def test_parse_header_links_no_link(): - all_links = httpx.Response(200).links - assert all_links == {} diff --git a/tests/models/test_queryparams.py b/tests/models/test_queryparams.py deleted file mode 100644 index 29b2ca634d..0000000000 --- a/tests/models/test_queryparams.py +++ /dev/null @@ -1,136 +0,0 @@ -import pytest - -import httpx - - -@pytest.mark.parametrize( - "source", - [ - "a=123&a=456&b=789", - {"a": ["123", "456"], "b": 789}, - {"a": ("123", "456"), "b": 789}, - [("a", "123"), ("a", "456"), ("b", "789")], - (("a", "123"), ("a", "456"), ("b", "789")), - ], -) -def test_queryparams(source): - q = httpx.QueryParams(source) - assert "a" in q - assert "A" not in q - assert "c" not in q - assert q["a"] == "123" - assert q.get("a") == "123" - assert q.get("nope", default=None) is None - assert q.get_list("a") == ["123", "456"] - - assert list(q.keys()) == ["a", "b"] - assert list(q.values()) == ["123", "789"] - assert list(q.items()) == [("a", "123"), ("b", "789")] - assert len(q) == 2 - assert list(q) == ["a", "b"] - assert dict(q) == {"a": "123", "b": "789"} - assert str(q) == "a=123&a=456&b=789" - assert repr(q) == "QueryParams('a=123&a=456&b=789')" - assert httpx.QueryParams({"a": "123", "b": "456"}) == httpx.QueryParams( - [("a", "123"), ("b", "456")] - ) - assert httpx.QueryParams({"a": "123", "b": "456"}) == httpx.QueryParams( - "a=123&b=456" - ) - assert httpx.QueryParams({"a": "123", "b": "456"}) == httpx.QueryParams( - {"b": "456", "a": "123"} - ) - assert httpx.QueryParams() == httpx.QueryParams({}) - assert httpx.QueryParams([("a", "123"), ("a", "456")]) == httpx.QueryParams( - "a=123&a=456" - ) - assert httpx.QueryParams({"a": "123", "b": "456"}) != "invalid" - - q = httpx.QueryParams([("a", "123"), ("a", "456")]) - assert httpx.QueryParams(q) == q - - -def test_queryparam_types(): - q = httpx.QueryParams(None) - assert str(q) == "" - - q = httpx.QueryParams({"a": True}) - assert str(q) == "a=true" - - q = httpx.QueryParams({"a": False}) - assert str(q) == "a=false" - - q = httpx.QueryParams({"a": ""}) - assert str(q) == "a=" - - q = httpx.QueryParams({"a": None}) - assert str(q) == "a=" - - q = httpx.QueryParams({"a": 1.23}) - assert str(q) == "a=1.23" - - q = httpx.QueryParams({"a": 123}) - assert str(q) == "a=123" - - q = httpx.QueryParams({"a": [1, 2]}) - assert str(q) == "a=1&a=2" - - -def test_empty_query_params(): - q = httpx.QueryParams({"a": ""}) - assert str(q) == "a=" - - q = httpx.QueryParams("a=") - assert str(q) == "a=" - - q = httpx.QueryParams("a") - assert str(q) == "a=" - - -def test_queryparam_update_is_hard_deprecated(): - q = httpx.QueryParams("a=123") - with pytest.raises(RuntimeError): - q.update({"a": "456"}) - - -def test_queryparam_setter_is_hard_deprecated(): - q = httpx.QueryParams("a=123") - with pytest.raises(RuntimeError): - q["a"] = "456" - - -def test_queryparam_set(): - q = httpx.QueryParams("a=123") - q = q.set("a", "456") - assert q == httpx.QueryParams("a=456") - - -def test_queryparam_add(): - q = httpx.QueryParams("a=123") - q = q.add("a", "456") - assert q == httpx.QueryParams("a=123&a=456") - - -def test_queryparam_remove(): - q = httpx.QueryParams("a=123") - q = q.remove("a") - assert q == httpx.QueryParams("") - - -def test_queryparam_merge(): - q = httpx.QueryParams("a=123") - q = q.merge({"b": "456"}) - assert q == httpx.QueryParams("a=123&b=456") - q = q.merge({"a": "000", "c": "789"}) - assert q == httpx.QueryParams("a=000&b=456&c=789") - - -def test_queryparams_are_hashable(): - params = ( - httpx.QueryParams("a=123"), - httpx.QueryParams({"a": 123}), - httpx.QueryParams("b=456"), - httpx.QueryParams({"b": 456}), - ) - - assert len(set(params)) == 2 diff --git a/tests/models/test_requests.py b/tests/models/test_requests.py deleted file mode 100644 index b31fe007be..0000000000 --- a/tests/models/test_requests.py +++ /dev/null @@ -1,241 +0,0 @@ -import pickle -import typing - -import pytest - -import httpx - - -def test_request_repr(): - request = httpx.Request("GET", "http://example.org") - assert repr(request) == "" - - -def test_no_content(): - request = httpx.Request("GET", "http://example.org") - assert "Content-Length" not in request.headers - - -def test_content_length_header(): - request = httpx.Request("POST", "http://example.org", content=b"test 123") - assert request.headers["Content-Length"] == "8" - - -def test_iterable_content(): - class Content: - def __iter__(self): - yield b"test 123" # pragma: no cover - - request = httpx.Request("POST", "http://example.org", content=Content()) - assert request.headers == {"Host": "example.org", "Transfer-Encoding": "chunked"} - - -def test_generator_with_transfer_encoding_header(): - def content() -> typing.Iterator[bytes]: - yield b"test 123" # pragma: no cover - - request = httpx.Request("POST", "http://example.org", content=content()) - assert request.headers == {"Host": "example.org", "Transfer-Encoding": "chunked"} - - -def test_generator_with_content_length_header(): - def content() -> typing.Iterator[bytes]: - yield b"test 123" # pragma: no cover - - headers = {"Content-Length": "8"} - request = httpx.Request( - "POST", "http://example.org", content=content(), headers=headers - ) - assert request.headers == {"Host": "example.org", "Content-Length": "8"} - - -def test_url_encoded_data(): - request = httpx.Request("POST", "http://example.org", data={"test": "123"}) - request.read() - - assert request.headers["Content-Type"] == "application/x-www-form-urlencoded" - assert request.content == b"test=123" - - -def test_json_encoded_data(): - request = httpx.Request("POST", "http://example.org", json={"test": 123}) - request.read() - - assert request.headers["Content-Type"] == "application/json" - assert request.content == b'{"test":123}' - - -def test_headers(): - request = httpx.Request("POST", "http://example.org", json={"test": 123}) - - assert request.headers == { - "Host": "example.org", - "Content-Type": "application/json", - "Content-Length": "12", - } - - -def test_read_and_stream_data(): - # Ensure a request may still be streamed if it has been read. - # Needed for cases such as authentication classes that read the request body. - request = httpx.Request("POST", "http://example.org", json={"test": 123}) - request.read() - assert request.stream is not None - assert isinstance(request.stream, typing.Iterable) - content = b"".join(list(request.stream)) - assert content == request.content - - -@pytest.mark.anyio -async def test_aread_and_stream_data(): - # Ensure a request may still be streamed if it has been read. - # Needed for cases such as authentication classes that read the request body. - request = httpx.Request("POST", "http://example.org", json={"test": 123}) - await request.aread() - assert request.stream is not None - assert isinstance(request.stream, typing.AsyncIterable) - content = b"".join([part async for part in request.stream]) - assert content == request.content - - -def test_cannot_access_streaming_content_without_read(): - # Ensure that streaming requests - def streaming_body() -> typing.Iterator[bytes]: # pragma: no cover - yield b"" - - request = httpx.Request("POST", "http://example.org", content=streaming_body()) - with pytest.raises(httpx.RequestNotRead): - request.content # noqa: B018 - - -def test_transfer_encoding_header(): - async def streaming_body(data: bytes) -> typing.AsyncIterator[bytes]: - yield data # pragma: no cover - - data = streaming_body(b"test 123") - - request = httpx.Request("POST", "http://example.org", content=data) - assert "Content-Length" not in request.headers - assert request.headers["Transfer-Encoding"] == "chunked" - - -def test_ignore_transfer_encoding_header_if_content_length_exists(): - """ - `Transfer-Encoding` should be ignored if `Content-Length` has been set explicitly. - See https://github.com/encode/httpx/issues/1168 - """ - - def streaming_body(data: bytes) -> typing.Iterator[bytes]: - yield data # pragma: no cover - - data = streaming_body(b"abcd") - - headers = {"Content-Length": "4"} - request = httpx.Request("POST", "http://example.org", content=data, headers=headers) - assert "Transfer-Encoding" not in request.headers - assert request.headers["Content-Length"] == "4" - - -def test_override_host_header(): - headers = {"host": "1.2.3.4:80"} - - request = httpx.Request("GET", "http://example.org", headers=headers) - assert request.headers["Host"] == "1.2.3.4:80" - - -def test_override_accept_encoding_header(): - headers = {"Accept-Encoding": "identity"} - - request = httpx.Request("GET", "http://example.org", headers=headers) - assert request.headers["Accept-Encoding"] == "identity" - - -def test_override_content_length_header(): - async def streaming_body(data: bytes) -> typing.AsyncIterator[bytes]: - yield data # pragma: no cover - - data = streaming_body(b"test 123") - headers = {"Content-Length": "8"} - - request = httpx.Request("POST", "http://example.org", content=data, headers=headers) - assert request.headers["Content-Length"] == "8" - - -def test_url(): - url = "http://example.org" - request = httpx.Request("GET", url) - assert request.url.scheme == "http" - assert request.url.port is None - assert request.url.path == "/" - assert request.url.raw_path == b"/" - - url = "https://example.org/abc?foo=bar" - request = httpx.Request("GET", url) - assert request.url.scheme == "https" - assert request.url.port is None - assert request.url.path == "/abc" - assert request.url.raw_path == b"/abc?foo=bar" - - -def test_request_picklable(): - request = httpx.Request("POST", "http://example.org", json={"test": 123}) - pickle_request = pickle.loads(pickle.dumps(request)) - assert pickle_request.method == "POST" - assert pickle_request.url.path == "/" - assert pickle_request.headers["Content-Type"] == "application/json" - assert pickle_request.content == b'{"test":123}' - assert pickle_request.stream is not None - assert request.headers == { - "Host": "example.org", - "Content-Type": "application/json", - "content-length": "12", - } - - -@pytest.mark.anyio -async def test_request_async_streaming_content_picklable(): - async def streaming_body(data: bytes) -> typing.AsyncIterator[bytes]: - yield data - - data = streaming_body(b"test 123") - request = httpx.Request("POST", "http://example.org", content=data) - pickle_request = pickle.loads(pickle.dumps(request)) - with pytest.raises(httpx.RequestNotRead): - pickle_request.content # noqa: B018 - with pytest.raises(httpx.StreamClosed): - await pickle_request.aread() - - request = httpx.Request("POST", "http://example.org", content=data) - await request.aread() - pickle_request = pickle.loads(pickle.dumps(request)) - assert pickle_request.content == b"test 123" - - -def test_request_generator_content_picklable(): - def content() -> typing.Iterator[bytes]: - yield b"test 123" # pragma: no cover - - request = httpx.Request("POST", "http://example.org", content=content()) - pickle_request = pickle.loads(pickle.dumps(request)) - with pytest.raises(httpx.RequestNotRead): - pickle_request.content # noqa: B018 - with pytest.raises(httpx.StreamClosed): - pickle_request.read() - - request = httpx.Request("POST", "http://example.org", content=content()) - request.read() - pickle_request = pickle.loads(pickle.dumps(request)) - assert pickle_request.content == b"test 123" - - -def test_request_params(): - request = httpx.Request("GET", "http://example.com", params={}) - assert str(request.url) == "http://example.com" - - request = httpx.Request( - "GET", "http://example.com?c=3", params={"a": "1", "b": "2"} - ) - assert str(request.url) == "http://example.com?a=1&b=2" - - request = httpx.Request("GET", "http://example.com?a=1", params={}) - assert str(request.url) == "http://example.com" diff --git a/tests/models/test_responses.py b/tests/models/test_responses.py deleted file mode 100644 index d2972da5bd..0000000000 --- a/tests/models/test_responses.py +++ /dev/null @@ -1,1040 +0,0 @@ -import json -import pickle -import typing - -import chardet -import pytest - -import httpx - - -class StreamingBody: - def __iter__(self): - yield b"Hello, " - yield b"world!" - - -def streaming_body() -> typing.Iterator[bytes]: - yield b"Hello, " - yield b"world!" - - -async def async_streaming_body() -> typing.AsyncIterator[bytes]: - yield b"Hello, " - yield b"world!" - - -def autodetect(content): - return chardet.detect(content).get("encoding") - - -def test_response(): - response = httpx.Response( - 200, - content=b"Hello, world!", - request=httpx.Request("GET", "https://example.org"), - ) - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Hello, world!" - assert response.request.method == "GET" - assert response.request.url == "https://example.org" - assert not response.is_error - - -def test_response_content(): - response = httpx.Response(200, content="Hello, world!") - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Hello, world!" - assert response.headers == {"Content-Length": "13"} - - -def test_response_text(): - response = httpx.Response(200, text="Hello, world!") - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Hello, world!" - assert response.headers == { - "Content-Length": "13", - "Content-Type": "text/plain; charset=utf-8", - } - - -def test_response_html(): - response = httpx.Response(200, html="Hello, world!") - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Hello, world!" - assert response.headers == { - "Content-Length": "39", - "Content-Type": "text/html; charset=utf-8", - } - - -def test_response_json(): - response = httpx.Response(200, json={"hello": "world"}) - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert str(response.json()) == "{'hello': 'world'}" - assert response.headers == { - "Content-Length": "17", - "Content-Type": "application/json", - } - - -def test_raise_for_status(): - request = httpx.Request("GET", "https://example.org") - - # 2xx status codes are not an error. - response = httpx.Response(200, request=request) - response.raise_for_status() - - # 1xx status codes are informational responses. - response = httpx.Response(101, request=request) - assert response.is_informational - with pytest.raises(httpx.HTTPStatusError) as exc_info: - response.raise_for_status() - assert str(exc_info.value) == ( - "Informational response '101 Switching Protocols' for url 'https://example.org'\n" - "For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101" - ) - - # 3xx status codes are redirections. - headers = {"location": "https://other.org"} - response = httpx.Response(303, headers=headers, request=request) - assert response.is_redirect - with pytest.raises(httpx.HTTPStatusError) as exc_info: - response.raise_for_status() - assert str(exc_info.value) == ( - "Redirect response '303 See Other' for url 'https://example.org'\n" - "Redirect location: 'https://other.org'\n" - "For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/303" - ) - - # 4xx status codes are a client error. - response = httpx.Response(403, request=request) - assert response.is_client_error - assert response.is_error - with pytest.raises(httpx.HTTPStatusError) as exc_info: - response.raise_for_status() - assert str(exc_info.value) == ( - "Client error '403 Forbidden' for url 'https://example.org'\n" - "For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403" - ) - - # 5xx status codes are a server error. - response = httpx.Response(500, request=request) - assert response.is_server_error - assert response.is_error - with pytest.raises(httpx.HTTPStatusError) as exc_info: - response.raise_for_status() - assert str(exc_info.value) == ( - "Server error '500 Internal Server Error' for url 'https://example.org'\n" - "For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500" - ) - - # Calling .raise_for_status without setting a request instance is - # not valid. Should raise a runtime error. - response = httpx.Response(200) - with pytest.raises(RuntimeError): - response.raise_for_status() - - -def test_response_repr(): - response = httpx.Response( - 200, - content=b"Hello, world!", - ) - assert repr(response) == "" - - -def test_response_content_type_encoding(): - """ - Use the charset encoding in the Content-Type header if possible. - """ - headers = {"Content-Type": "text-plain; charset=latin-1"} - content = "Latin 1: ÿ".encode("latin-1") - response = httpx.Response( - 200, - content=content, - headers=headers, - ) - assert response.text == "Latin 1: ÿ" - assert response.encoding == "latin-1" - - -def test_response_default_to_utf8_encoding(): - """ - Default to utf-8 encoding if there is no Content-Type header. - """ - content = "おはようございます。".encode("utf-8") - response = httpx.Response( - 200, - content=content, - ) - assert response.text == "おはようございます。" - assert response.encoding == "utf-8" - - -def test_response_fallback_to_utf8_encoding(): - """ - Fallback to utf-8 if we get an invalid charset in the Content-Type header. - """ - headers = {"Content-Type": "text-plain; charset=invalid-codec-name"} - content = "おはようございます。".encode("utf-8") - response = httpx.Response( - 200, - content=content, - headers=headers, - ) - assert response.text == "おはようございます。" - assert response.encoding == "utf-8" - - -def test_response_no_charset_with_ascii_content(): - """ - A response with ascii encoded content should decode correctly, - even with no charset specified. - """ - content = b"Hello, world!" - headers = {"Content-Type": "text/plain"} - response = httpx.Response( - 200, - content=content, - headers=headers, - ) - assert response.status_code == 200 - assert response.encoding == "utf-8" - assert response.text == "Hello, world!" - - -def test_response_no_charset_with_utf8_content(): - """ - A response with UTF-8 encoded content should decode correctly, - even with no charset specified. - """ - content = "Unicode Snowman: ☃".encode("utf-8") - headers = {"Content-Type": "text/plain"} - response = httpx.Response( - 200, - content=content, - headers=headers, - ) - assert response.text == "Unicode Snowman: ☃" - assert response.encoding == "utf-8" - - -def test_response_no_charset_with_iso_8859_1_content(): - """ - A response with ISO 8859-1 encoded content should decode correctly, - even with no charset specified, if autodetect is enabled. - """ - content = "Accented: Österreich abcdefghijklmnopqrstuzwxyz".encode("iso-8859-1") - headers = {"Content-Type": "text/plain"} - response = httpx.Response( - 200, content=content, headers=headers, default_encoding=autodetect - ) - assert response.text == "Accented: Österreich abcdefghijklmnopqrstuzwxyz" - assert response.charset_encoding is None - - -def test_response_no_charset_with_cp_1252_content(): - """ - A response with Windows 1252 encoded content should decode correctly, - even with no charset specified, if autodetect is enabled. - """ - content = "Euro Currency: € abcdefghijklmnopqrstuzwxyz".encode("cp1252") - headers = {"Content-Type": "text/plain"} - response = httpx.Response( - 200, content=content, headers=headers, default_encoding=autodetect - ) - assert response.text == "Euro Currency: € abcdefghijklmnopqrstuzwxyz" - assert response.charset_encoding is None - - -def test_response_non_text_encoding(): - """ - Default to attempting utf-8 encoding for non-text content-type headers. - """ - headers = {"Content-Type": "image/png"} - response = httpx.Response( - 200, - content=b"xyz", - headers=headers, - ) - assert response.text == "xyz" - assert response.encoding == "utf-8" - - -def test_response_set_explicit_encoding(): - headers = { - "Content-Type": "text-plain; charset=utf-8" - } # Deliberately incorrect charset - response = httpx.Response( - 200, - content="Latin 1: ÿ".encode("latin-1"), - headers=headers, - ) - response.encoding = "latin-1" - assert response.text == "Latin 1: ÿ" - assert response.encoding == "latin-1" - - -def test_response_force_encoding(): - response = httpx.Response( - 200, - content="Snowman: ☃".encode("utf-8"), - ) - response.encoding = "iso-8859-1" - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Snowman: â\x98\x83" - assert response.encoding == "iso-8859-1" - - -def test_response_force_encoding_after_text_accessed(): - response = httpx.Response( - 200, - content=b"Hello, world!", - ) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Hello, world!" - assert response.encoding == "utf-8" - - with pytest.raises(ValueError): - response.encoding = "UTF8" - - with pytest.raises(ValueError): - response.encoding = "iso-8859-1" - - -def test_read(): - response = httpx.Response( - 200, - content=b"Hello, world!", - ) - - assert response.status_code == 200 - assert response.text == "Hello, world!" - assert response.encoding == "utf-8" - assert response.is_closed - - content = response.read() - - assert content == b"Hello, world!" - assert response.content == b"Hello, world!" - assert response.is_closed - - -def test_empty_read(): - response = httpx.Response(200) - - assert response.status_code == 200 - assert response.text == "" - assert response.encoding == "utf-8" - assert response.is_closed - - content = response.read() - - assert content == b"" - assert response.content == b"" - assert response.is_closed - - -@pytest.mark.anyio -async def test_aread(): - response = httpx.Response( - 200, - content=b"Hello, world!", - ) - - assert response.status_code == 200 - assert response.text == "Hello, world!" - assert response.encoding == "utf-8" - assert response.is_closed - - content = await response.aread() - - assert content == b"Hello, world!" - assert response.content == b"Hello, world!" - assert response.is_closed - - -@pytest.mark.anyio -async def test_empty_aread(): - response = httpx.Response(200) - - assert response.status_code == 200 - assert response.text == "" - assert response.encoding == "utf-8" - assert response.is_closed - - content = await response.aread() - - assert content == b"" - assert response.content == b"" - assert response.is_closed - - -def test_iter_raw(): - response = httpx.Response( - 200, - content=streaming_body(), - ) - - raw = b"" - for part in response.iter_raw(): - raw += part - assert raw == b"Hello, world!" - - -def test_iter_raw_with_chunksize(): - response = httpx.Response(200, content=streaming_body()) - parts = list(response.iter_raw(chunk_size=5)) - assert parts == [b"Hello", b", wor", b"ld!"] - - response = httpx.Response(200, content=streaming_body()) - parts = list(response.iter_raw(chunk_size=7)) - assert parts == [b"Hello, ", b"world!"] - - response = httpx.Response(200, content=streaming_body()) - parts = list(response.iter_raw(chunk_size=13)) - assert parts == [b"Hello, world!"] - - response = httpx.Response(200, content=streaming_body()) - parts = list(response.iter_raw(chunk_size=20)) - assert parts == [b"Hello, world!"] - - -def test_iter_raw_doesnt_return_empty_chunks(): - def streaming_body_with_empty_chunks() -> typing.Iterator[bytes]: - yield b"Hello, " - yield b"" - yield b"world!" - yield b"" - - response = httpx.Response(200, content=streaming_body_with_empty_chunks()) - - parts = list(response.iter_raw()) - assert parts == [b"Hello, ", b"world!"] - - -def test_iter_raw_on_iterable(): - response = httpx.Response( - 200, - content=StreamingBody(), - ) - - raw = b"" - for part in response.iter_raw(): - raw += part - assert raw == b"Hello, world!" - - -def test_iter_raw_on_async(): - response = httpx.Response( - 200, - content=async_streaming_body(), - ) - - with pytest.raises(RuntimeError): - list(response.iter_raw()) - - -def test_close_on_async(): - response = httpx.Response( - 200, - content=async_streaming_body(), - ) - - with pytest.raises(RuntimeError): - response.close() - - -def test_iter_raw_increments_updates_counter(): - response = httpx.Response(200, content=streaming_body()) - - num_downloaded = response.num_bytes_downloaded - for part in response.iter_raw(): - assert len(part) == (response.num_bytes_downloaded - num_downloaded) - num_downloaded = response.num_bytes_downloaded - - -@pytest.mark.anyio -async def test_aiter_raw(): - response = httpx.Response(200, content=async_streaming_body()) - - raw = b"" - async for part in response.aiter_raw(): - raw += part - assert raw == b"Hello, world!" - - -@pytest.mark.anyio -async def test_aiter_raw_with_chunksize(): - response = httpx.Response(200, content=async_streaming_body()) - - parts = [part async for part in response.aiter_raw(chunk_size=5)] - assert parts == [b"Hello", b", wor", b"ld!"] - - response = httpx.Response(200, content=async_streaming_body()) - - parts = [part async for part in response.aiter_raw(chunk_size=13)] - assert parts == [b"Hello, world!"] - - response = httpx.Response(200, content=async_streaming_body()) - - parts = [part async for part in response.aiter_raw(chunk_size=20)] - assert parts == [b"Hello, world!"] - - -@pytest.mark.anyio -async def test_aiter_raw_on_sync(): - response = httpx.Response( - 200, - content=streaming_body(), - ) - - with pytest.raises(RuntimeError): - [part async for part in response.aiter_raw()] - - -@pytest.mark.anyio -async def test_aclose_on_sync(): - response = httpx.Response( - 200, - content=streaming_body(), - ) - - with pytest.raises(RuntimeError): - await response.aclose() - - -@pytest.mark.anyio -async def test_aiter_raw_increments_updates_counter(): - response = httpx.Response(200, content=async_streaming_body()) - - num_downloaded = response.num_bytes_downloaded - async for part in response.aiter_raw(): - assert len(part) == (response.num_bytes_downloaded - num_downloaded) - num_downloaded = response.num_bytes_downloaded - - -def test_iter_bytes(): - response = httpx.Response(200, content=b"Hello, world!") - - content = b"" - for part in response.iter_bytes(): - content += part - assert content == b"Hello, world!" - - -def test_iter_bytes_with_chunk_size(): - response = httpx.Response(200, content=streaming_body()) - parts = list(response.iter_bytes(chunk_size=5)) - assert parts == [b"Hello", b", wor", b"ld!"] - - response = httpx.Response(200, content=streaming_body()) - parts = list(response.iter_bytes(chunk_size=13)) - assert parts == [b"Hello, world!"] - - response = httpx.Response(200, content=streaming_body()) - parts = list(response.iter_bytes(chunk_size=20)) - assert parts == [b"Hello, world!"] - - -def test_iter_bytes_with_empty_response(): - response = httpx.Response(200, content=b"") - parts = list(response.iter_bytes()) - assert parts == [] - - -def test_iter_bytes_doesnt_return_empty_chunks(): - def streaming_body_with_empty_chunks() -> typing.Iterator[bytes]: - yield b"Hello, " - yield b"" - yield b"world!" - yield b"" - - response = httpx.Response(200, content=streaming_body_with_empty_chunks()) - - parts = list(response.iter_bytes()) - assert parts == [b"Hello, ", b"world!"] - - -@pytest.mark.anyio -async def test_aiter_bytes(): - response = httpx.Response( - 200, - content=b"Hello, world!", - ) - - content = b"" - async for part in response.aiter_bytes(): - content += part - assert content == b"Hello, world!" - - -@pytest.mark.anyio -async def test_aiter_bytes_with_chunk_size(): - response = httpx.Response(200, content=async_streaming_body()) - parts = [part async for part in response.aiter_bytes(chunk_size=5)] - assert parts == [b"Hello", b", wor", b"ld!"] - - response = httpx.Response(200, content=async_streaming_body()) - parts = [part async for part in response.aiter_bytes(chunk_size=13)] - assert parts == [b"Hello, world!"] - - response = httpx.Response(200, content=async_streaming_body()) - parts = [part async for part in response.aiter_bytes(chunk_size=20)] - assert parts == [b"Hello, world!"] - - -def test_iter_text(): - response = httpx.Response( - 200, - content=b"Hello, world!", - ) - - content = "" - for part in response.iter_text(): - content += part - assert content == "Hello, world!" - - -def test_iter_text_with_chunk_size(): - response = httpx.Response(200, content=b"Hello, world!") - parts = list(response.iter_text(chunk_size=5)) - assert parts == ["Hello", ", wor", "ld!"] - - response = httpx.Response(200, content=b"Hello, world!!") - parts = list(response.iter_text(chunk_size=7)) - assert parts == ["Hello, ", "world!!"] - - response = httpx.Response(200, content=b"Hello, world!") - parts = list(response.iter_text(chunk_size=7)) - assert parts == ["Hello, ", "world!"] - - response = httpx.Response(200, content=b"Hello, world!") - parts = list(response.iter_text(chunk_size=13)) - assert parts == ["Hello, world!"] - - response = httpx.Response(200, content=b"Hello, world!") - parts = list(response.iter_text(chunk_size=20)) - assert parts == ["Hello, world!"] - - -@pytest.mark.anyio -async def test_aiter_text(): - response = httpx.Response( - 200, - content=b"Hello, world!", - ) - - content = "" - async for part in response.aiter_text(): - content += part - assert content == "Hello, world!" - - -@pytest.mark.anyio -async def test_aiter_text_with_chunk_size(): - response = httpx.Response(200, content=b"Hello, world!") - parts = [part async for part in response.aiter_text(chunk_size=5)] - assert parts == ["Hello", ", wor", "ld!"] - - response = httpx.Response(200, content=b"Hello, world!") - parts = [part async for part in response.aiter_text(chunk_size=13)] - assert parts == ["Hello, world!"] - - response = httpx.Response(200, content=b"Hello, world!") - parts = [part async for part in response.aiter_text(chunk_size=20)] - assert parts == ["Hello, world!"] - - -def test_iter_lines(): - response = httpx.Response( - 200, - content=b"Hello,\nworld!", - ) - content = list(response.iter_lines()) - assert content == ["Hello,", "world!"] - - -@pytest.mark.anyio -async def test_aiter_lines(): - response = httpx.Response( - 200, - content=b"Hello,\nworld!", - ) - - content = [] - async for line in response.aiter_lines(): - content.append(line) - assert content == ["Hello,", "world!"] - - -def test_sync_streaming_response(): - response = httpx.Response( - 200, - content=streaming_body(), - ) - - assert response.status_code == 200 - assert not response.is_closed - - content = response.read() - - assert content == b"Hello, world!" - assert response.content == b"Hello, world!" - assert response.is_closed - - -@pytest.mark.anyio -async def test_async_streaming_response(): - response = httpx.Response( - 200, - content=async_streaming_body(), - ) - - assert response.status_code == 200 - assert not response.is_closed - - content = await response.aread() - - assert content == b"Hello, world!" - assert response.content == b"Hello, world!" - assert response.is_closed - - -def test_cannot_read_after_stream_consumed(): - response = httpx.Response( - 200, - content=streaming_body(), - ) - - content = b"" - for part in response.iter_bytes(): - content += part - - with pytest.raises(httpx.StreamConsumed): - response.read() - - -@pytest.mark.anyio -async def test_cannot_aread_after_stream_consumed(): - response = httpx.Response( - 200, - content=async_streaming_body(), - ) - - content = b"" - async for part in response.aiter_bytes(): - content += part - - with pytest.raises(httpx.StreamConsumed): - await response.aread() - - -def test_cannot_read_after_response_closed(): - response = httpx.Response( - 200, - content=streaming_body(), - ) - - response.close() - with pytest.raises(httpx.StreamClosed): - response.read() - - -@pytest.mark.anyio -async def test_cannot_aread_after_response_closed(): - response = httpx.Response( - 200, - content=async_streaming_body(), - ) - - await response.aclose() - with pytest.raises(httpx.StreamClosed): - await response.aread() - - -@pytest.mark.anyio -async def test_elapsed_not_available_until_closed(): - response = httpx.Response( - 200, - content=async_streaming_body(), - ) - - with pytest.raises(RuntimeError): - response.elapsed # noqa: B018 - - -def test_unknown_status_code(): - response = httpx.Response( - 600, - ) - assert response.status_code == 600 - assert response.reason_phrase == "" - assert response.text == "" - - -def test_json_with_specified_encoding(): - data = {"greeting": "hello", "recipient": "world"} - content = json.dumps(data).encode("utf-16") - headers = {"Content-Type": "application/json, charset=utf-16"} - response = httpx.Response( - 200, - content=content, - headers=headers, - ) - assert response.json() == data - - -def test_json_with_options(): - data = {"greeting": "hello", "recipient": "world", "amount": 1} - content = json.dumps(data).encode("utf-16") - headers = {"Content-Type": "application/json, charset=utf-16"} - response = httpx.Response( - 200, - content=content, - headers=headers, - ) - assert response.json(parse_int=str)["amount"] == "1" - - -@pytest.mark.parametrize( - "encoding", - [ - "utf-8", - "utf-8-sig", - "utf-16", - "utf-16-be", - "utf-16-le", - "utf-32", - "utf-32-be", - "utf-32-le", - ], -) -def test_json_without_specified_charset(encoding): - data = {"greeting": "hello", "recipient": "world"} - content = json.dumps(data).encode(encoding) - headers = {"Content-Type": "application/json"} - response = httpx.Response( - 200, - content=content, - headers=headers, - ) - assert response.json() == data - - -@pytest.mark.parametrize( - "encoding", - [ - "utf-8", - "utf-8-sig", - "utf-16", - "utf-16-be", - "utf-16-le", - "utf-32", - "utf-32-be", - "utf-32-le", - ], -) -def test_json_with_specified_charset(encoding): - data = {"greeting": "hello", "recipient": "world"} - content = json.dumps(data).encode(encoding) - headers = {"Content-Type": f"application/json; charset={encoding}"} - response = httpx.Response( - 200, - content=content, - headers=headers, - ) - assert response.json() == data - - -@pytest.mark.parametrize( - "headers, expected", - [ - ( - {"Link": "; rel='preload'"}, - {"preload": {"rel": "preload", "url": "https://example.com"}}, - ), - ( - {"Link": '; rel="hub", ; rel="self"'}, - { - "hub": {"url": "/hub", "rel": "hub"}, - "self": {"url": "/resource", "rel": "self"}, - }, - ), - ], -) -def test_link_headers(headers, expected): - response = httpx.Response( - 200, - content=None, - headers=headers, - ) - assert response.links == expected - - -@pytest.mark.parametrize("header_value", (b"deflate", b"gzip", b"br")) -def test_decode_error_with_request(header_value): - headers = [(b"Content-Encoding", header_value)] - broken_compressed_body = b"xxxxxxxxxxxxxx" - with pytest.raises(httpx.DecodingError): - httpx.Response( - 200, - headers=headers, - content=broken_compressed_body, - ) - - with pytest.raises(httpx.DecodingError): - httpx.Response( - 200, - headers=headers, - content=broken_compressed_body, - request=httpx.Request("GET", "https://www.example.org/"), - ) - - -@pytest.mark.parametrize("header_value", (b"deflate", b"gzip", b"br")) -def test_value_error_without_request(header_value): - headers = [(b"Content-Encoding", header_value)] - broken_compressed_body = b"xxxxxxxxxxxxxx" - with pytest.raises(httpx.DecodingError): - httpx.Response(200, headers=headers, content=broken_compressed_body) - - -def test_response_with_unset_request(): - response = httpx.Response(200, content=b"Hello, world!") - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Hello, world!" - assert not response.is_error - - -def test_set_request_after_init(): - response = httpx.Response(200, content=b"Hello, world!") - - response.request = httpx.Request("GET", "https://www.example.org") - - assert response.request.method == "GET" - assert response.request.url == "https://www.example.org" - - -def test_cannot_access_unset_request(): - response = httpx.Response(200, content=b"Hello, world!") - - with pytest.raises(RuntimeError): - response.request # noqa: B018 - - -def test_generator_with_transfer_encoding_header(): - def content() -> typing.Iterator[bytes]: - yield b"test 123" # pragma: no cover - - response = httpx.Response(200, content=content()) - assert response.headers == {"Transfer-Encoding": "chunked"} - - -def test_generator_with_content_length_header(): - def content() -> typing.Iterator[bytes]: - yield b"test 123" # pragma: no cover - - headers = {"Content-Length": "8"} - response = httpx.Response(200, content=content(), headers=headers) - assert response.headers == {"Content-Length": "8"} - - -def test_response_picklable(): - response = httpx.Response( - 200, - content=b"Hello, world!", - request=httpx.Request("GET", "https://example.org"), - ) - pickle_response = pickle.loads(pickle.dumps(response)) - assert pickle_response.is_closed is True - assert pickle_response.is_stream_consumed is True - assert pickle_response.next_request is None - assert pickle_response.stream is not None - assert pickle_response.content == b"Hello, world!" - assert pickle_response.status_code == 200 - assert pickle_response.request.url == response.request.url - assert pickle_response.extensions == {} - assert pickle_response.history == [] - - -@pytest.mark.anyio -async def test_response_async_streaming_picklable(): - response = httpx.Response(200, content=async_streaming_body()) - pickle_response = pickle.loads(pickle.dumps(response)) - with pytest.raises(httpx.ResponseNotRead): - pickle_response.content # noqa: B018 - with pytest.raises(httpx.StreamClosed): - await pickle_response.aread() - assert pickle_response.is_stream_consumed is False - assert pickle_response.num_bytes_downloaded == 0 - assert pickle_response.headers == {"Transfer-Encoding": "chunked"} - - response = httpx.Response(200, content=async_streaming_body()) - await response.aread() - pickle_response = pickle.loads(pickle.dumps(response)) - assert pickle_response.is_stream_consumed is True - assert pickle_response.content == b"Hello, world!" - assert pickle_response.num_bytes_downloaded == 13 - - -def test_response_decode_text_using_autodetect(): - # Ensure that a 'default_encoding="autodetect"' on the response allows for - # encoding autodetection to be used when no "Content-Type: text/plain; charset=..." - # info is present. - # - # Here we have some french text encoded with ISO-8859-1, rather than UTF-8. - text = ( - "Non-seulement Despréaux ne se trompait pas, mais de tous les écrivains " - "que la France a produits, sans excepter Voltaire lui-même, imprégné de " - "l'esprit anglais par son séjour à Londres, c'est incontestablement " - "Molière ou Poquelin qui reproduit avec l'exactitude la plus vive et la " - "plus complète le fond du génie français." - ) - content = text.encode("ISO-8859-1") - response = httpx.Response(200, content=content, default_encoding=autodetect) - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - # The encoded byte string is consistent with either ISO-8859-1 or - # WINDOWS-1252. Versions <6.0 of chardet claim the former, while chardet - # 6.0 detects the latter. - assert response.encoding in ("ISO-8859-1", "WINDOWS-1252") - assert response.text == text - - -def test_response_decode_text_using_explicit_encoding(): - # Ensure that a 'default_encoding="..."' on the response is used for text decoding - # when no "Content-Type: text/plain; charset=..."" info is present. - # - # Here we have some french text encoded with Windows-1252, rather than UTF-8. - # https://en.wikipedia.org/wiki/Windows-1252 - text = ( - "Non-seulement Despréaux ne se trompait pas, mais de tous les écrivains " - "que la France a produits, sans excepter Voltaire lui-même, imprégné de " - "l'esprit anglais par son séjour à Londres, c'est incontestablement " - "Molière ou Poquelin qui reproduit avec l'exactitude la plus vive et la " - "plus complète le fond du génie français." - ) - content = text.encode("cp1252") - response = httpx.Response(200, content=content, default_encoding="cp1252") - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.encoding == "cp1252" - assert response.text == text diff --git a/tests/models/test_url.py b/tests/models/test_url.py deleted file mode 100644 index 03072e8f5c..0000000000 --- a/tests/models/test_url.py +++ /dev/null @@ -1,863 +0,0 @@ -import pytest - -import httpx - -# Tests for `httpx.URL` instantiation and property accessors. - - -def test_basic_url(): - url = httpx.URL("https://www.example.com/") - - assert url.scheme == "https" - assert url.userinfo == b"" - assert url.netloc == b"www.example.com" - assert url.host == "www.example.com" - assert url.port is None - assert url.path == "/" - assert url.query == b"" - assert url.fragment == "" - - assert str(url) == "https://www.example.com/" - assert repr(url) == "URL('https://www.example.com/')" - - -def test_complete_url(): - url = httpx.URL("https://example.org:123/path/to/somewhere?abc=123#anchor") - assert url.scheme == "https" - assert url.host == "example.org" - assert url.port == 123 - assert url.path == "/path/to/somewhere" - assert url.query == b"abc=123" - assert url.raw_path == b"/path/to/somewhere?abc=123" - assert url.fragment == "anchor" - - assert str(url) == "https://example.org:123/path/to/somewhere?abc=123#anchor" - assert ( - repr(url) == "URL('https://example.org:123/path/to/somewhere?abc=123#anchor')" - ) - - -def test_url_with_empty_query(): - """ - URLs with and without a trailing `?` but an empty query component - should preserve the information on the raw path. - """ - url = httpx.URL("https://www.example.com/path") - assert url.path == "/path" - assert url.query == b"" - assert url.raw_path == b"/path" - - url = httpx.URL("https://www.example.com/path?") - assert url.path == "/path" - assert url.query == b"" - assert url.raw_path == b"/path?" - - -def test_url_no_scheme(): - url = httpx.URL("://example.com") - assert url.scheme == "" - assert url.host == "example.com" - assert url.path == "/" - - -def test_url_no_authority(): - url = httpx.URL("http://") - assert url.scheme == "http" - assert url.host == "" - assert url.path == "/" - - -# Tests for percent encoding across path, query, and fragment... - - -@pytest.mark.parametrize( - "url,raw_path,path,query,fragment", - [ - # URL with unescaped chars in path. - ( - "https://example.com/!$&'()*+,;= abc ABC 123 :/[]@", - b"/!$&'()*+,;=%20abc%20ABC%20123%20:/[]@", - "/!$&'()*+,;= abc ABC 123 :/[]@", - b"", - "", - ), - # URL with escaped chars in path. - ( - "https://example.com/!$&'()*+,;=%20abc%20ABC%20123%20:/[]@", - b"/!$&'()*+,;=%20abc%20ABC%20123%20:/[]@", - "/!$&'()*+,;= abc ABC 123 :/[]@", - b"", - "", - ), - # URL with mix of unescaped and escaped chars in path. - # WARNING: This has the incorrect behaviour, adding the test as an interim step. - ( - "https://example.com/ %61%62%63", - b"/%20%61%62%63", - "/ abc", - b"", - "", - ), - # URL with unescaped chars in query. - ( - "https://example.com/?!$&'()*+,;= abc ABC 123 :/[]@?", - b"/?!$&'()*+,;=%20abc%20ABC%20123%20:/[]@?", - "/", - b"!$&'()*+,;=%20abc%20ABC%20123%20:/[]@?", - "", - ), - # URL with escaped chars in query. - ( - "https://example.com/?!$&%27()*+,;=%20abc%20ABC%20123%20:%2F[]@?", - b"/?!$&%27()*+,;=%20abc%20ABC%20123%20:%2F[]@?", - "/", - b"!$&%27()*+,;=%20abc%20ABC%20123%20:%2F[]@?", - "", - ), - # URL with mix of unescaped and escaped chars in query. - ( - "https://example.com/?%20%97%98%99", - b"/?%20%97%98%99", - "/", - b"%20%97%98%99", - "", - ), - # URL encoding characters in fragment. - ( - "https://example.com/#!$&'()*+,;= abc ABC 123 :/[]@?#", - b"/", - "/", - b"", - "!$&'()*+,;= abc ABC 123 :/[]@?#", - ), - ], -) -def test_path_query_fragment(url, raw_path, path, query, fragment): - url = httpx.URL(url) - assert url.raw_path == raw_path - assert url.path == path - assert url.query == query - assert url.fragment == fragment - - -def test_url_query_encoding(): - url = httpx.URL("https://www.example.com/?a=b c&d=e/f") - assert url.raw_path == b"/?a=b%20c&d=e/f" - - url = httpx.URL("https://www.example.com/?a=b+c&d=e/f") - assert url.raw_path == b"/?a=b+c&d=e/f" - - url = httpx.URL("https://www.example.com/", params={"a": "b c", "d": "e/f"}) - assert url.raw_path == b"/?a=b+c&d=e%2Ff" - - -def test_url_params(): - url = httpx.URL("https://example.org:123/path/to/somewhere", params={"a": "123"}) - assert str(url) == "https://example.org:123/path/to/somewhere?a=123" - assert url.params == httpx.QueryParams({"a": "123"}) - - url = httpx.URL( - "https://example.org:123/path/to/somewhere?b=456", params={"a": "123"} - ) - assert str(url) == "https://example.org:123/path/to/somewhere?a=123" - assert url.params == httpx.QueryParams({"a": "123"}) - - -# Tests for username and password - - -@pytest.mark.parametrize( - "url,userinfo,username,password", - [ - # username and password in URL. - ( - "https://username:password@example.com", - b"username:password", - "username", - "password", - ), - # username and password in URL with percent escape sequences. - ( - "https://username%40gmail.com:pa%20ssword@example.com", - b"username%40gmail.com:pa%20ssword", - "username@gmail.com", - "pa ssword", - ), - ( - "https://user%20name:p%40ssword@example.com", - b"user%20name:p%40ssword", - "user name", - "p@ssword", - ), - # username and password in URL without percent escape sequences. - ( - "https://username@gmail.com:pa ssword@example.com", - b"username%40gmail.com:pa%20ssword", - "username@gmail.com", - "pa ssword", - ), - ( - "https://user name:p@ssword@example.com", - b"user%20name:p%40ssword", - "user name", - "p@ssword", - ), - ], -) -def test_url_username_and_password(url, userinfo, username, password): - url = httpx.URL(url) - assert url.userinfo == userinfo - assert url.username == username - assert url.password == password - - -# Tests for different host types - - -def test_url_valid_host(): - url = httpx.URL("https://example.com/") - assert url.host == "example.com" - - -def test_url_normalized_host(): - url = httpx.URL("https://EXAMPLE.com/") - assert url.host == "example.com" - - -def test_url_percent_escape_host(): - url = httpx.URL("https://exam le.com/") - assert url.host == "exam%20le.com" - - -def test_url_ipv4_like_host(): - """rare host names used to quality as IPv4""" - url = httpx.URL("https://023b76x43144/") - assert url.host == "023b76x43144" - - -# Tests for different port types - - -def test_url_valid_port(): - url = httpx.URL("https://example.com:123/") - assert url.port == 123 - - -def test_url_normalized_port(): - # If the port matches the scheme default it is normalized to None. - url = httpx.URL("https://example.com:443/") - assert url.port is None - - -def test_url_invalid_port(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL("https://example.com:abc/") - assert str(exc.value) == "Invalid port: 'abc'" - - -# Tests for path handling - - -def test_url_normalized_path(): - url = httpx.URL("https://example.com/abc/def/../ghi/./jkl") - assert url.path == "/abc/ghi/jkl" - - -def test_url_escaped_path(): - url = httpx.URL("https://example.com/ /🌟/") - assert url.raw_path == b"/%20/%F0%9F%8C%9F/" - - -def test_url_leading_dot_prefix_on_absolute_url(): - url = httpx.URL("https://example.com/../abc") - assert url.path == "/abc" - - -def test_url_leading_dot_prefix_on_relative_url(): - url = httpx.URL("../abc") - assert url.path == "../abc" - - -# Tests for query parameter percent encoding. -# -# Percent-encoding in `params={}` should match browser form behavior. - - -def test_param_with_space(): - # Params passed as form key-value pairs should be form escaped, - # Including the special case of "+" for space seperators. - url = httpx.URL("http://webservice", params={"u": "with spaces"}) - assert str(url) == "http://webservice?u=with+spaces" - - -def test_param_requires_encoding(): - # Params passed as form key-value pairs should be escaped. - url = httpx.URL("http://webservice", params={"u": "%"}) - assert str(url) == "http://webservice?u=%25" - - -def test_param_with_percent_encoded(): - # Params passed as form key-value pairs should always be escaped, - # even if they include a valid escape sequence. - # We want to match browser form behaviour here. - url = httpx.URL("http://webservice", params={"u": "with%20spaces"}) - assert str(url) == "http://webservice?u=with%2520spaces" - - -def test_param_with_existing_escape_requires_encoding(): - # Params passed as form key-value pairs should always be escaped, - # even if they include a valid escape sequence. - # We want to match browser form behaviour here. - url = httpx.URL("http://webservice", params={"u": "http://example.com?q=foo%2Fa"}) - assert str(url) == "http://webservice?u=http%3A%2F%2Fexample.com%3Fq%3Dfoo%252Fa" - - -# Tests for query parameter percent encoding. -# -# Percent-encoding in `url={}` should match browser URL bar behavior. - - -def test_query_with_existing_percent_encoding(): - # Valid percent encoded sequences should not be double encoded. - url = httpx.URL("http://webservice?u=phrase%20with%20spaces") - assert str(url) == "http://webservice?u=phrase%20with%20spaces" - - -def test_query_requiring_percent_encoding(): - # Characters that require percent encoding should be encoded. - url = httpx.URL("http://webservice?u=phrase with spaces") - assert str(url) == "http://webservice?u=phrase%20with%20spaces" - - -def test_query_with_mixed_percent_encoding(): - # When a mix of encoded and unencoded characters are present, - # characters that require percent encoding should be encoded, - # while existing sequences should not be double encoded. - url = httpx.URL("http://webservice?u=phrase%20with spaces") - assert str(url) == "http://webservice?u=phrase%20with%20spaces" - - -# Tests for invalid URLs - - -def test_url_invalid_hostname(): - """ - Ensure that invalid URLs raise an `httpx.InvalidURL` exception. - """ - with pytest.raises(httpx.InvalidURL): - httpx.URL("https://😇/") - - -def test_url_excessively_long_url(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL("https://www.example.com/" + "x" * 100_000) - assert str(exc.value) == "URL too long" - - -def test_url_excessively_long_component(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL("https://www.example.com", path="/" + "x" * 100_000) - assert str(exc.value) == "URL component 'path' too long" - - -def test_url_non_printing_character_in_url(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL("https://www.example.com/\n") - assert str(exc.value) == ( - "Invalid non-printable ASCII character in URL, '\\n' at position 24." - ) - - -def test_url_non_printing_character_in_component(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL("https://www.example.com", path="/\n") - assert str(exc.value) == ( - "Invalid non-printable ASCII character in URL path component, " - "'\\n' at position 1." - ) - - -# Test for url components - - -def test_url_with_components(): - url = httpx.URL(scheme="https", host="www.example.com", path="/") - - assert url.scheme == "https" - assert url.userinfo == b"" - assert url.host == "www.example.com" - assert url.port is None - assert url.path == "/" - assert url.query == b"" - assert url.fragment == "" - - assert str(url) == "https://www.example.com/" - - -def test_urlparse_with_invalid_component(): - with pytest.raises(TypeError) as exc: - httpx.URL(scheme="https", host="www.example.com", incorrect="/") - assert str(exc.value) == "'incorrect' is an invalid keyword argument for URL()" - - -def test_urlparse_with_invalid_scheme(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL(scheme="~", host="www.example.com", path="/") - assert str(exc.value) == "Invalid URL component 'scheme'" - - -def test_urlparse_with_invalid_path(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL(scheme="https", host="www.example.com", path="abc") - assert str(exc.value) == "For absolute URLs, path must be empty or begin with '/'" - - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL(path="//abc") - assert str(exc.value) == "Relative URLs cannot have a path starting with '//'" - - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL(path=":abc") - assert str(exc.value) == "Relative URLs cannot have a path starting with ':'" - - -def test_url_with_relative_path(): - # This path would be invalid for an absolute URL, but is valid as a relative URL. - url = httpx.URL(path="abc") - assert url.path == "abc" - - -# Tests for `httpx.URL` python built-in operators. - - -def test_url_eq_str(): - """ - Ensure that `httpx.URL` supports the equality operator. - """ - url = httpx.URL("https://example.org:123/path/to/somewhere?abc=123#anchor") - assert url == "https://example.org:123/path/to/somewhere?abc=123#anchor" - assert str(url) == url - - -def test_url_set(): - """ - Ensure that `httpx.URL` instances can be used in sets. - """ - urls = ( - httpx.URL("http://example.org:123/path/to/somewhere"), - httpx.URL("http://example.org:123/path/to/somewhere/else"), - ) - - url_set = set(urls) - - assert all(url in urls for url in url_set) - - -# Tests for TypeErrors when instantiating `httpx.URL`. - - -def test_url_invalid_type(): - """ - Ensure that invalid types on `httpx.URL()` raise a `TypeError`. - """ - - class ExternalURLClass: # representing external URL class - pass - - with pytest.raises(TypeError): - httpx.URL(ExternalURLClass()) # type: ignore - - -def test_url_with_invalid_component(): - with pytest.raises(TypeError) as exc: - httpx.URL(scheme="https", host="www.example.com", incorrect="/") - assert str(exc.value) == "'incorrect' is an invalid keyword argument for URL()" - - -# Tests for `URL.join()`. - - -def test_url_join(): - """ - Some basic URL joining tests. - """ - url = httpx.URL("https://example.org:123/path/to/somewhere") - assert url.join("/somewhere-else") == "https://example.org:123/somewhere-else" - assert ( - url.join("somewhere-else") == "https://example.org:123/path/to/somewhere-else" - ) - assert ( - url.join("../somewhere-else") == "https://example.org:123/path/somewhere-else" - ) - assert url.join("../../somewhere-else") == "https://example.org:123/somewhere-else" - - -def test_relative_url_join(): - url = httpx.URL("/path/to/somewhere") - assert url.join("/somewhere-else") == "/somewhere-else" - assert url.join("somewhere-else") == "/path/to/somewhere-else" - assert url.join("../somewhere-else") == "/path/somewhere-else" - assert url.join("../../somewhere-else") == "/somewhere-else" - - -def test_url_join_rfc3986(): - """ - URL joining tests, as-per reference examples in RFC 3986. - - https://tools.ietf.org/html/rfc3986#section-5.4 - """ - - url = httpx.URL("http://example.com/b/c/d;p?q") - - assert url.join("g") == "http://example.com/b/c/g" - assert url.join("./g") == "http://example.com/b/c/g" - assert url.join("g/") == "http://example.com/b/c/g/" - assert url.join("/g") == "http://example.com/g" - assert url.join("//g") == "http://g" - assert url.join("?y") == "http://example.com/b/c/d;p?y" - assert url.join("g?y") == "http://example.com/b/c/g?y" - assert url.join("#s") == "http://example.com/b/c/d;p?q#s" - assert url.join("g#s") == "http://example.com/b/c/g#s" - assert url.join("g?y#s") == "http://example.com/b/c/g?y#s" - assert url.join(";x") == "http://example.com/b/c/;x" - assert url.join("g;x") == "http://example.com/b/c/g;x" - assert url.join("g;x?y#s") == "http://example.com/b/c/g;x?y#s" - assert url.join("") == "http://example.com/b/c/d;p?q" - assert url.join(".") == "http://example.com/b/c/" - assert url.join("./") == "http://example.com/b/c/" - assert url.join("..") == "http://example.com/b/" - assert url.join("../") == "http://example.com/b/" - assert url.join("../g") == "http://example.com/b/g" - assert url.join("../..") == "http://example.com/" - assert url.join("../../") == "http://example.com/" - assert url.join("../../g") == "http://example.com/g" - - assert url.join("../../../g") == "http://example.com/g" - assert url.join("../../../../g") == "http://example.com/g" - - assert url.join("/./g") == "http://example.com/g" - assert url.join("/../g") == "http://example.com/g" - assert url.join("g.") == "http://example.com/b/c/g." - assert url.join(".g") == "http://example.com/b/c/.g" - assert url.join("g..") == "http://example.com/b/c/g.." - assert url.join("..g") == "http://example.com/b/c/..g" - - assert url.join("./../g") == "http://example.com/b/g" - assert url.join("./g/.") == "http://example.com/b/c/g/" - assert url.join("g/./h") == "http://example.com/b/c/g/h" - assert url.join("g/../h") == "http://example.com/b/c/h" - assert url.join("g;x=1/./y") == "http://example.com/b/c/g;x=1/y" - assert url.join("g;x=1/../y") == "http://example.com/b/c/y" - - assert url.join("g?y/./x") == "http://example.com/b/c/g?y/./x" - assert url.join("g?y/../x") == "http://example.com/b/c/g?y/../x" - assert url.join("g#s/./x") == "http://example.com/b/c/g#s/./x" - assert url.join("g#s/../x") == "http://example.com/b/c/g#s/../x" - - -def test_resolution_error_1833(): - """ - See https://github.com/encode/httpx/issues/1833 - """ - url = httpx.URL("https://example.com/?[]") - assert url.join("/") == "https://example.com/" - - -# Tests for `URL.copy_with()`. - - -def test_copy_with(): - url = httpx.URL("https://www.example.com/") - assert str(url) == "https://www.example.com/" - - url = url.copy_with() - assert str(url) == "https://www.example.com/" - - url = url.copy_with(scheme="http") - assert str(url) == "http://www.example.com/" - - url = url.copy_with(netloc=b"example.com") - assert str(url) == "http://example.com/" - - url = url.copy_with(path="/abc") - assert str(url) == "http://example.com/abc" - - -def test_url_copywith_authority_subcomponents(): - copy_with_kwargs = { - "username": "username", - "password": "password", - "port": 444, - "host": "example.net", - } - url = httpx.URL("https://example.org") - new = url.copy_with(**copy_with_kwargs) - assert str(new) == "https://username:password@example.net:444" - - -def test_url_copywith_netloc(): - copy_with_kwargs = { - "netloc": b"example.net:444", - } - url = httpx.URL("https://example.org") - new = url.copy_with(**copy_with_kwargs) - assert str(new) == "https://example.net:444" - - -def test_url_copywith_userinfo_subcomponents(): - copy_with_kwargs = { - "username": "tom@example.org", - "password": "abc123@ %", - } - url = httpx.URL("https://example.org") - new = url.copy_with(**copy_with_kwargs) - assert str(new) == "https://tom%40example.org:abc123%40%20%@example.org" - assert new.username == "tom@example.org" - assert new.password == "abc123@ %" - assert new.userinfo == b"tom%40example.org:abc123%40%20%" - - -def test_url_copywith_invalid_component(): - url = httpx.URL("https://example.org") - with pytest.raises(TypeError): - url.copy_with(pathh="/incorrect-spelling") - with pytest.raises(TypeError): - url.copy_with(userinfo="should be bytes") - - -def test_url_copywith_urlencoded_path(): - url = httpx.URL("https://example.org") - url = url.copy_with(path="/path to somewhere") - assert url.path == "/path to somewhere" - assert url.query == b"" - assert url.raw_path == b"/path%20to%20somewhere" - - -def test_url_copywith_query(): - url = httpx.URL("https://example.org") - url = url.copy_with(query=b"a=123") - assert url.path == "/" - assert url.query == b"a=123" - assert url.raw_path == b"/?a=123" - - -def test_url_copywith_raw_path(): - url = httpx.URL("https://example.org") - url = url.copy_with(raw_path=b"/some/path") - assert url.path == "/some/path" - assert url.query == b"" - assert url.raw_path == b"/some/path" - - url = httpx.URL("https://example.org") - url = url.copy_with(raw_path=b"/some/path?") - assert url.path == "/some/path" - assert url.query == b"" - assert url.raw_path == b"/some/path?" - - url = httpx.URL("https://example.org") - url = url.copy_with(raw_path=b"/some/path?a=123") - assert url.path == "/some/path" - assert url.query == b"a=123" - assert url.raw_path == b"/some/path?a=123" - - -def test_url_copywith_security(): - """ - Prevent unexpected changes on URL after calling copy_with (CVE-2021-41945) - """ - with pytest.raises(httpx.InvalidURL): - httpx.URL("https://u:p@[invalid!]//evilHost/path?t=w#tw") - - url = httpx.URL("https://example.com/path?t=w#tw") - bad = "https://xxxx:xxxx@xxxxxxx/xxxxx/xxx?x=x#xxxxx" - with pytest.raises(httpx.InvalidURL): - url.copy_with(scheme=bad) - - -# Tests for copy-modifying-parameters methods. -# -# `URL.copy_set_param()` -# `URL.copy_add_param()` -# `URL.copy_remove_param()` -# `URL.copy_merge_params()` - - -def test_url_set_param_manipulation(): - """ - Some basic URL query parameter manipulation. - """ - url = httpx.URL("https://example.org:123/?a=123") - assert url.copy_set_param("a", "456") == "https://example.org:123/?a=456" - - -def test_url_add_param_manipulation(): - """ - Some basic URL query parameter manipulation. - """ - url = httpx.URL("https://example.org:123/?a=123") - assert url.copy_add_param("a", "456") == "https://example.org:123/?a=123&a=456" - - -def test_url_remove_param_manipulation(): - """ - Some basic URL query parameter manipulation. - """ - url = httpx.URL("https://example.org:123/?a=123") - assert url.copy_remove_param("a") == "https://example.org:123/" - - -def test_url_merge_params_manipulation(): - """ - Some basic URL query parameter manipulation. - """ - url = httpx.URL("https://example.org:123/?a=123") - assert url.copy_merge_params({"b": "456"}) == "https://example.org:123/?a=123&b=456" - - -# Tests for IDNA hostname support. - - -@pytest.mark.parametrize( - "given,idna,host,raw_host,scheme,port", - [ - ( - "http://中国.icom.museum:80/", - "http://xn--fiqs8s.icom.museum:80/", - "中国.icom.museum", - b"xn--fiqs8s.icom.museum", - "http", - None, - ), - ( - "http://Königsgäßchen.de", - "http://xn--knigsgchen-b4a3dun.de", - "königsgäßchen.de", - b"xn--knigsgchen-b4a3dun.de", - "http", - None, - ), - ( - "https://faß.de", - "https://xn--fa-hia.de", - "faß.de", - b"xn--fa-hia.de", - "https", - None, - ), - ( - "https://βόλος.com:443", - "https://xn--nxasmm1c.com:443", - "βόλος.com", - b"xn--nxasmm1c.com", - "https", - None, - ), - ( - "http://ශ්‍රී.com:444", - "http://xn--10cl1a0b660p.com:444", - "ශ්‍රී.com", - b"xn--10cl1a0b660p.com", - "http", - 444, - ), - ( - "https://نامه‌ای.com:4433", - "https://xn--mgba3gch31f060k.com:4433", - "نامه‌ای.com", - b"xn--mgba3gch31f060k.com", - "https", - 4433, - ), - ], - ids=[ - "http_with_port", - "unicode_tr46_compat", - "https_without_port", - "https_with_port", - "http_with_custom_port", - "https_with_custom_port", - ], -) -def test_idna_url(given, idna, host, raw_host, scheme, port): - url = httpx.URL(given) - assert url == httpx.URL(idna) - assert url.host == host - assert url.raw_host == raw_host - assert url.scheme == scheme - assert url.port == port - - -def test_url_unescaped_idna_host(): - url = httpx.URL("https://中国.icom.museum/") - assert url.raw_host == b"xn--fiqs8s.icom.museum" - - -def test_url_escaped_idna_host(): - url = httpx.URL("https://xn--fiqs8s.icom.museum/") - assert url.raw_host == b"xn--fiqs8s.icom.museum" - - -def test_url_invalid_idna_host(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL("https://☃.com/") - assert str(exc.value) == "Invalid IDNA hostname: '☃.com'" - - -# Tests for IPv4 hostname support. - - -def test_url_valid_ipv4(): - url = httpx.URL("https://1.2.3.4/") - assert url.host == "1.2.3.4" - - -def test_url_invalid_ipv4(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL("https://999.999.999.999/") - assert str(exc.value) == "Invalid IPv4 address: '999.999.999.999'" - - -# Tests for IPv6 hostname support. - - -def test_ipv6_url(): - url = httpx.URL("http://[::ffff:192.168.0.1]:5678/") - - assert url.host == "::ffff:192.168.0.1" - assert url.netloc == b"[::ffff:192.168.0.1]:5678" - - -def test_url_valid_ipv6(): - url = httpx.URL("https://[2001:db8::ff00:42:8329]/") - assert url.host == "2001:db8::ff00:42:8329" - - -def test_url_invalid_ipv6(): - with pytest.raises(httpx.InvalidURL) as exc: - httpx.URL("https://[2001]/") - assert str(exc.value) == "Invalid IPv6 address: '[2001]'" - - -@pytest.mark.parametrize("host", ["[::ffff:192.168.0.1]", "::ffff:192.168.0.1"]) -def test_ipv6_url_from_raw_url(host): - url = httpx.URL(scheme="https", host=host, port=443, path="/") - - assert url.host == "::ffff:192.168.0.1" - assert url.netloc == b"[::ffff:192.168.0.1]" - assert str(url) == "https://[::ffff:192.168.0.1]/" - - -@pytest.mark.parametrize( - "url_str", - [ - "http://127.0.0.1:1234", - "http://example.com:1234", - "http://[::ffff:127.0.0.1]:1234", - ], -) -@pytest.mark.parametrize("new_host", ["[::ffff:192.168.0.1]", "::ffff:192.168.0.1"]) -def test_ipv6_url_copy_with_host(url_str, new_host): - url = httpx.URL(url_str).copy_with(host=new_host) - - assert url.host == "::ffff:192.168.0.1" - assert url.netloc == b"[::ffff:192.168.0.1]:1234" - assert str(url) == "http://[::ffff:192.168.0.1]:1234" diff --git a/tests/models/test_whatwg.py b/tests/models/test_whatwg.py deleted file mode 100644 index 14af682586..0000000000 --- a/tests/models/test_whatwg.py +++ /dev/null @@ -1,52 +0,0 @@ -# The WHATWG have various tests that can be used to validate the URL parsing. -# -# https://url.spec.whatwg.org/ - -import json - -import pytest - -from httpx._urlparse import urlparse - -# URL test cases from... -# https://github.com/web-platform-tests/wpt/blob/master/url/resources/urltestdata.json -with open("tests/models/whatwg.json", "r", encoding="utf-8") as input: - test_cases = json.load(input) - test_cases = [ - item - for item in test_cases - if not isinstance(item, str) and not item.get("failure") - ] - - -@pytest.mark.parametrize("test_case", test_cases) -def test_urlparse(test_case): - if test_case["href"] in ("a: foo.com", "lolscheme:x x#x%20x"): - # Skip these two test cases. - # WHATWG cases where are not using percent-encoding for the space character. - # Anyone know what's going on here? - return - - p = urlparse(test_case["href"]) - - # Test cases include the protocol with the trailing ":" - protocol = p.scheme + ":" - # Include the square brackets for IPv6 addresses. - hostname = f"[{p.host}]" if ":" in p.host else p.host - # The test cases use a string representation of the port. - port = "" if p.port is None else str(p.port) - # I have nothing to say about this one. - path = p.path - # The 'search' and 'hash' components in the whatwg tests are semantic, not literal. - # Our parsing differentiates between no query/hash and empty-string query/hash. - search = "" if p.query in (None, "") else "?" + str(p.query) - hash = "" if p.fragment in (None, "") else "#" + str(p.fragment) - - # URL hostnames are case-insensitive. - # We normalize these, unlike the WHATWG test cases. - assert protocol == test_case["protocol"] - assert hostname.lower() == test_case["hostname"].lower() - assert port == test_case["port"] - assert path == test_case["pathname"] - assert search == test_case["search"] - assert hash == test_case["hash"] diff --git a/tests/models/whatwg.json b/tests/models/whatwg.json deleted file mode 100644 index 85a5140f8a..0000000000 --- a/tests/models/whatwg.json +++ /dev/null @@ -1,9746 +0,0 @@ -[ - "See ../README.md for a description of the format.", - { - "input": "http://example\t.\norg", - "base": "http://example.org/foo/bar", - "href": "http://example.org/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://user:pass@foo:21/bar;par?b#c", - "base": "http://example.org/foo/bar", - "href": "http://user:pass@foo:21/bar;par?b#c", - "origin": "http://foo:21", - "protocol": "http:", - "username": "user", - "password": "pass", - "host": "foo:21", - "hostname": "foo", - "port": "21", - "pathname": "/bar;par", - "search": "?b", - "hash": "#c" - }, - { - "input": "https://test:@test", - "base": null, - "href": "https://test@test/", - "origin": "https://test", - "protocol": "https:", - "username": "test", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https://:@test", - "base": null, - "href": "https://test/", - "origin": "https://test", - "protocol": "https:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "non-special://test:@test/x", - "base": null, - "href": "non-special://test@test/x", - "origin": "null", - "protocol": "non-special:", - "username": "test", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/x", - "search": "", - "hash": "" - }, - { - "input": "non-special://:@test/x", - "base": null, - "href": "non-special://test/x", - "origin": "null", - "protocol": "non-special:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/x", - "search": "", - "hash": "" - }, - { - "input": "http:foo.com", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/foo.com", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/foo.com", - "search": "", - "hash": "" - }, - { - "input": "\t :foo.com \n", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:foo.com", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:foo.com", - "search": "", - "hash": "" - }, - { - "input": " foo.com ", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/foo.com", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/foo.com", - "search": "", - "hash": "" - }, - { - "input": "a:\t foo.com", - "base": "http://example.org/foo/bar", - "href": "a: foo.com", - "origin": "null", - "protocol": "a:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": " foo.com", - "search": "", - "hash": "" - }, - { - "input": "http://f:21/ b ? d # e ", - "base": "http://example.org/foo/bar", - "href": "http://f:21/%20b%20?%20d%20#%20e", - "origin": "http://f:21", - "protocol": "http:", - "username": "", - "password": "", - "host": "f:21", - "hostname": "f", - "port": "21", - "pathname": "/%20b%20", - "search": "?%20d%20", - "hash": "#%20e" - }, - { - "input": "lolscheme:x x#x x", - "base": null, - "href": "lolscheme:x x#x%20x", - "protocol": "lolscheme:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "x x", - "search": "", - "hash": "#x%20x" - }, - { - "input": "http://f:/c", - "base": "http://example.org/foo/bar", - "href": "http://f/c", - "origin": "http://f", - "protocol": "http:", - "username": "", - "password": "", - "host": "f", - "hostname": "f", - "port": "", - "pathname": "/c", - "search": "", - "hash": "" - }, - { - "input": "http://f:0/c", - "base": "http://example.org/foo/bar", - "href": "http://f:0/c", - "origin": "http://f:0", - "protocol": "http:", - "username": "", - "password": "", - "host": "f:0", - "hostname": "f", - "port": "0", - "pathname": "/c", - "search": "", - "hash": "" - }, - { - "input": "http://f:00000000000000/c", - "base": "http://example.org/foo/bar", - "href": "http://f:0/c", - "origin": "http://f:0", - "protocol": "http:", - "username": "", - "password": "", - "host": "f:0", - "hostname": "f", - "port": "0", - "pathname": "/c", - "search": "", - "hash": "" - }, - { - "input": "http://f:00000000000000000000080/c", - "base": "http://example.org/foo/bar", - "href": "http://f/c", - "origin": "http://f", - "protocol": "http:", - "username": "", - "password": "", - "host": "f", - "hostname": "f", - "port": "", - "pathname": "/c", - "search": "", - "hash": "" - }, - { - "input": "http://f:b/c", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://f: /c", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://f:\n/c", - "base": "http://example.org/foo/bar", - "href": "http://f/c", - "origin": "http://f", - "protocol": "http:", - "username": "", - "password": "", - "host": "f", - "hostname": "f", - "port": "", - "pathname": "/c", - "search": "", - "hash": "" - }, - { - "input": "http://f:fifty-two/c", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://f:999999/c", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "non-special://f:999999/c", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://f: 21 / b ? d # e ", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "" - }, - { - "input": " \t", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "" - }, - { - "input": ":foo.com/", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:foo.com/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:foo.com/", - "search": "", - "hash": "" - }, - { - "input": ":foo.com\\", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:foo.com/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:foo.com/", - "search": "", - "hash": "" - }, - { - "input": ":", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:", - "search": "", - "hash": "" - }, - { - "input": ":a", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:a", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:a", - "search": "", - "hash": "" - }, - { - "input": ":/", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:/", - "search": "", - "hash": "" - }, - { - "input": ":\\", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:/", - "search": "", - "hash": "" - }, - { - "input": ":#", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:#", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:", - "search": "", - "hash": "" - }, - { - "input": "#", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar#", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "#/", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar#/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "#/" - }, - { - "input": "#\\", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar#\\", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "#\\" - }, - { - "input": "#;?", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar#;?", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "#;?" - }, - { - "input": "?", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar?", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "/", - "base": "http://example.org/foo/bar", - "href": "http://example.org/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": ":23", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:23", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:23", - "search": "", - "hash": "" - }, - { - "input": "/:23", - "base": "http://example.org/foo/bar", - "href": "http://example.org/:23", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/:23", - "search": "", - "hash": "" - }, - { - "input": "\\x", - "base": "http://example.org/foo/bar", - "href": "http://example.org/x", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/x", - "search": "", - "hash": "" - }, - { - "input": "\\\\x\\hello", - "base": "http://example.org/foo/bar", - "href": "http://x/hello", - "origin": "http://x", - "protocol": "http:", - "username": "", - "password": "", - "host": "x", - "hostname": "x", - "port": "", - "pathname": "/hello", - "search": "", - "hash": "" - }, - { - "input": "::", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/::", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/::", - "search": "", - "hash": "" - }, - { - "input": "::23", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/::23", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/::23", - "search": "", - "hash": "" - }, - { - "input": "foo://", - "base": "http://example.org/foo/bar", - "href": "foo://", - "origin": "null", - "protocol": "foo:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "", - "search": "", - "hash": "" - }, - { - "input": "http://a:b@c:29/d", - "base": "http://example.org/foo/bar", - "href": "http://a:b@c:29/d", - "origin": "http://c:29", - "protocol": "http:", - "username": "a", - "password": "b", - "host": "c:29", - "hostname": "c", - "port": "29", - "pathname": "/d", - "search": "", - "hash": "" - }, - { - "input": "http::@c:29", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/:@c:29", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/:@c:29", - "search": "", - "hash": "" - }, - { - "input": "http://&a:foo(b]c@d:2/", - "base": "http://example.org/foo/bar", - "href": "http://&a:foo(b%5Dc@d:2/", - "origin": "http://d:2", - "protocol": "http:", - "username": "&a", - "password": "foo(b%5Dc", - "host": "d:2", - "hostname": "d", - "port": "2", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://::@c@d:2", - "base": "http://example.org/foo/bar", - "href": "http://:%3A%40c@d:2/", - "origin": "http://d:2", - "protocol": "http:", - "username": "", - "password": "%3A%40c", - "host": "d:2", - "hostname": "d", - "port": "2", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://foo.com:b@d/", - "base": "http://example.org/foo/bar", - "href": "http://foo.com:b@d/", - "origin": "http://d", - "protocol": "http:", - "username": "foo.com", - "password": "b", - "host": "d", - "hostname": "d", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://foo.com/\\@", - "base": "http://example.org/foo/bar", - "href": "http://foo.com//@", - "origin": "http://foo.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo.com", - "hostname": "foo.com", - "port": "", - "pathname": "//@", - "search": "", - "hash": "" - }, - { - "input": "http:\\\\foo.com\\", - "base": "http://example.org/foo/bar", - "href": "http://foo.com/", - "origin": "http://foo.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo.com", - "hostname": "foo.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:\\\\a\\b:c\\d@foo.com\\", - "base": "http://example.org/foo/bar", - "href": "http://a/b:c/d@foo.com/", - "origin": "http://a", - "protocol": "http:", - "username": "", - "password": "", - "host": "a", - "hostname": "a", - "port": "", - "pathname": "/b:c/d@foo.com/", - "search": "", - "hash": "" - }, - { - "input": "http://a:b@c\\", - "base": null, - "href": "http://a:b@c/", - "origin": "http://c", - "protocol": "http:", - "username": "a", - "password": "b", - "host": "c", - "hostname": "c", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ws://a@b\\c", - "base": null, - "href": "ws://a@b/c", - "origin": "ws://b", - "protocol": "ws:", - "username": "a", - "password": "", - "host": "b", - "hostname": "b", - "port": "", - "pathname": "/c", - "search": "", - "hash": "" - }, - { - "input": "foo:/", - "base": "http://example.org/foo/bar", - "href": "foo:/", - "origin": "null", - "protocol": "foo:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "foo:/bar.com/", - "base": "http://example.org/foo/bar", - "href": "foo:/bar.com/", - "origin": "null", - "protocol": "foo:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/bar.com/", - "search": "", - "hash": "" - }, - { - "input": "foo://///////", - "base": "http://example.org/foo/bar", - "href": "foo://///////", - "origin": "null", - "protocol": "foo:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "///////", - "search": "", - "hash": "" - }, - { - "input": "foo://///////bar.com/", - "base": "http://example.org/foo/bar", - "href": "foo://///////bar.com/", - "origin": "null", - "protocol": "foo:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "///////bar.com/", - "search": "", - "hash": "" - }, - { - "input": "foo:////://///", - "base": "http://example.org/foo/bar", - "href": "foo:////://///", - "origin": "null", - "protocol": "foo:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//://///", - "search": "", - "hash": "" - }, - { - "input": "c:/foo", - "base": "http://example.org/foo/bar", - "href": "c:/foo", - "origin": "null", - "protocol": "c:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/foo", - "search": "", - "hash": "" - }, - { - "input": "//foo/bar", - "base": "http://example.org/foo/bar", - "href": "http://foo/bar", - "origin": "http://foo", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/bar", - "search": "", - "hash": "" - }, - { - "input": "http://foo/path;a??e#f#g", - "base": "http://example.org/foo/bar", - "href": "http://foo/path;a??e#f#g", - "origin": "http://foo", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/path;a", - "search": "??e", - "hash": "#f#g" - }, - { - "input": "http://foo/abcd?efgh?ijkl", - "base": "http://example.org/foo/bar", - "href": "http://foo/abcd?efgh?ijkl", - "origin": "http://foo", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/abcd", - "search": "?efgh?ijkl", - "hash": "" - }, - { - "input": "http://foo/abcd#foo?bar", - "base": "http://example.org/foo/bar", - "href": "http://foo/abcd#foo?bar", - "origin": "http://foo", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/abcd", - "search": "", - "hash": "#foo?bar" - }, - { - "input": "[61:24:74]:98", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/[61:24:74]:98", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/[61:24:74]:98", - "search": "", - "hash": "" - }, - { - "input": "http:[61:27]/:foo", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/[61:27]/:foo", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/[61:27]/:foo", - "search": "", - "hash": "" - }, - { - "input": "http://[1::2]:3:4", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://2001::1", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://2001::1]", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://2001::1]:80", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://[2001::1]", - "base": "http://example.org/foo/bar", - "href": "http://[2001::1]/", - "origin": "http://[2001::1]", - "protocol": "http:", - "username": "", - "password": "", - "host": "[2001::1]", - "hostname": "[2001::1]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://[::127.0.0.1]", - "base": "http://example.org/foo/bar", - "href": "http://[::7f00:1]/", - "origin": "http://[::7f00:1]", - "protocol": "http:", - "username": "", - "password": "", - "host": "[::7f00:1]", - "hostname": "[::7f00:1]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://[::127.0.0.1.]", - "base": "http://example.org/foo/bar", - "failure": true - }, - { - "input": "http://[0:0:0:0:0:0:13.1.68.3]", - "base": "http://example.org/foo/bar", - "href": "http://[::d01:4403]/", - "origin": "http://[::d01:4403]", - "protocol": "http:", - "username": "", - "password": "", - "host": "[::d01:4403]", - "hostname": "[::d01:4403]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://[2001::1]:80", - "base": "http://example.org/foo/bar", - "href": "http://[2001::1]/", - "origin": "http://[2001::1]", - "protocol": "http:", - "username": "", - "password": "", - "host": "[2001::1]", - "hostname": "[2001::1]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:/example.com/", - "base": "http://example.org/foo/bar", - "href": "http://example.org/example.com/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "ftp:/example.com/", - "base": "http://example.org/foo/bar", - "href": "ftp://example.com/", - "origin": "ftp://example.com", - "protocol": "ftp:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https:/example.com/", - "base": "http://example.org/foo/bar", - "href": "https://example.com/", - "origin": "https://example.com", - "protocol": "https:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "madeupscheme:/example.com/", - "base": "http://example.org/foo/bar", - "href": "madeupscheme:/example.com/", - "origin": "null", - "protocol": "madeupscheme:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "file:/example.com/", - "base": "http://example.org/foo/bar", - "href": "file:///example.com/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "file://example:1/", - "base": null, - "failure": true - }, - { - "input": "file://example:test/", - "base": null, - "failure": true - }, - { - "input": "file://example%/", - "base": null, - "failure": true - }, - { - "input": "file://[example]/", - "base": null, - "failure": true - }, - { - "input": "ftps:/example.com/", - "base": "http://example.org/foo/bar", - "href": "ftps:/example.com/", - "origin": "null", - "protocol": "ftps:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "gopher:/example.com/", - "base": "http://example.org/foo/bar", - "href": "gopher:/example.com/", - "origin": "null", - "protocol": "gopher:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "ws:/example.com/", - "base": "http://example.org/foo/bar", - "href": "ws://example.com/", - "origin": "ws://example.com", - "protocol": "ws:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "wss:/example.com/", - "base": "http://example.org/foo/bar", - "href": "wss://example.com/", - "origin": "wss://example.com", - "protocol": "wss:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "data:/example.com/", - "base": "http://example.org/foo/bar", - "href": "data:/example.com/", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "javascript:/example.com/", - "base": "http://example.org/foo/bar", - "href": "javascript:/example.com/", - "origin": "null", - "protocol": "javascript:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "mailto:/example.com/", - "base": "http://example.org/foo/bar", - "href": "mailto:/example.com/", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "http:example.com/", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/example.com/", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/example.com/", - "search": "", - "hash": "" - }, - { - "input": "ftp:example.com/", - "base": "http://example.org/foo/bar", - "href": "ftp://example.com/", - "origin": "ftp://example.com", - "protocol": "ftp:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https:example.com/", - "base": "http://example.org/foo/bar", - "href": "https://example.com/", - "origin": "https://example.com", - "protocol": "https:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "madeupscheme:example.com/", - "base": "http://example.org/foo/bar", - "href": "madeupscheme:example.com/", - "origin": "null", - "protocol": "madeupscheme:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "ftps:example.com/", - "base": "http://example.org/foo/bar", - "href": "ftps:example.com/", - "origin": "null", - "protocol": "ftps:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "gopher:example.com/", - "base": "http://example.org/foo/bar", - "href": "gopher:example.com/", - "origin": "null", - "protocol": "gopher:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "ws:example.com/", - "base": "http://example.org/foo/bar", - "href": "ws://example.com/", - "origin": "ws://example.com", - "protocol": "ws:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "wss:example.com/", - "base": "http://example.org/foo/bar", - "href": "wss://example.com/", - "origin": "wss://example.com", - "protocol": "wss:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "data:example.com/", - "base": "http://example.org/foo/bar", - "href": "data:example.com/", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "javascript:example.com/", - "base": "http://example.org/foo/bar", - "href": "javascript:example.com/", - "origin": "null", - "protocol": "javascript:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "mailto:example.com/", - "base": "http://example.org/foo/bar", - "href": "mailto:example.com/", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "/a/b/c", - "base": "http://example.org/foo/bar", - "href": "http://example.org/a/b/c", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/a/b/c", - "search": "", - "hash": "" - }, - { - "input": "/a/ /c", - "base": "http://example.org/foo/bar", - "href": "http://example.org/a/%20/c", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/a/%20/c", - "search": "", - "hash": "" - }, - { - "input": "/a%2fc", - "base": "http://example.org/foo/bar", - "href": "http://example.org/a%2fc", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/a%2fc", - "search": "", - "hash": "" - }, - { - "input": "/a/%2f/c", - "base": "http://example.org/foo/bar", - "href": "http://example.org/a/%2f/c", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/a/%2f/c", - "search": "", - "hash": "" - }, - { - "input": "#β", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar#%CE%B2", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "#%CE%B2" - }, - { - "input": "data:text/html,test#test", - "base": "http://example.org/foo/bar", - "href": "data:text/html,test#test", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "text/html,test", - "search": "", - "hash": "#test" - }, - { - "input": "tel:1234567890", - "base": "http://example.org/foo/bar", - "href": "tel:1234567890", - "origin": "null", - "protocol": "tel:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "1234567890", - "search": "", - "hash": "" - }, - "# Based on https://felixfbecker.github.io/whatwg-url-custom-host-repro/", - { - "input": "ssh://example.com/foo/bar.git", - "base": "http://example.org/", - "href": "ssh://example.com/foo/bar.git", - "origin": "null", - "protocol": "ssh:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/bar.git", - "search": "", - "hash": "" - }, - "# Based on http://trac.webkit.org/browser/trunk/LayoutTests/fast/url/file.html", - { - "input": "file:c:\\foo\\bar.html", - "base": "file:///tmp/mock/path", - "href": "file:///c:/foo/bar.html", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/c:/foo/bar.html", - "search": "", - "hash": "" - }, - { - "input": " File:c|////foo\\bar.html", - "base": "file:///tmp/mock/path", - "href": "file:///c:////foo/bar.html", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/c:////foo/bar.html", - "search": "", - "hash": "" - }, - { - "input": "C|/foo/bar", - "base": "file:///tmp/mock/path", - "href": "file:///C:/foo/bar", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "/C|\\foo\\bar", - "base": "file:///tmp/mock/path", - "href": "file:///C:/foo/bar", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "//C|/foo/bar", - "base": "file:///tmp/mock/path", - "href": "file:///C:/foo/bar", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "//server/file", - "base": "file:///tmp/mock/path", - "href": "file://server/file", - "protocol": "file:", - "username": "", - "password": "", - "host": "server", - "hostname": "server", - "port": "", - "pathname": "/file", - "search": "", - "hash": "" - }, - { - "input": "\\\\server\\file", - "base": "file:///tmp/mock/path", - "href": "file://server/file", - "protocol": "file:", - "username": "", - "password": "", - "host": "server", - "hostname": "server", - "port": "", - "pathname": "/file", - "search": "", - "hash": "" - }, - { - "input": "/\\server/file", - "base": "file:///tmp/mock/path", - "href": "file://server/file", - "protocol": "file:", - "username": "", - "password": "", - "host": "server", - "hostname": "server", - "port": "", - "pathname": "/file", - "search": "", - "hash": "" - }, - { - "input": "file:///foo/bar.txt", - "base": "file:///tmp/mock/path", - "href": "file:///foo/bar.txt", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/foo/bar.txt", - "search": "", - "hash": "" - }, - { - "input": "file:///home/me", - "base": "file:///tmp/mock/path", - "href": "file:///home/me", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/home/me", - "search": "", - "hash": "" - }, - { - "input": "//", - "base": "file:///tmp/mock/path", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "///", - "base": "file:///tmp/mock/path", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "///test", - "base": "file:///tmp/mock/path", - "href": "file:///test", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "file://test", - "base": "file:///tmp/mock/path", - "href": "file://test/", - "protocol": "file:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "file://localhost", - "base": "file:///tmp/mock/path", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "file://localhost/", - "base": "file:///tmp/mock/path", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "file://localhost/test", - "base": "file:///tmp/mock/path", - "href": "file:///test", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "test", - "base": "file:///tmp/mock/path", - "href": "file:///tmp/mock/test", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/tmp/mock/test", - "search": "", - "hash": "" - }, - { - "input": "file:test", - "base": "file:///tmp/mock/path", - "href": "file:///tmp/mock/test", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/tmp/mock/test", - "search": "", - "hash": "" - }, - "# Based on http://trac.webkit.org/browser/trunk/LayoutTests/fast/url/script-tests/path.js", - { - "input": "http://example.com/././foo", - "base": null, - "href": "http://example.com/foo", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/./.foo", - "base": null, - "href": "http://example.com/.foo", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/.foo", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/.", - "base": null, - "href": "http://example.com/foo/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/./", - "base": null, - "href": "http://example.com/foo/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/bar/..", - "base": null, - "href": "http://example.com/foo/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/bar/../", - "base": null, - "href": "http://example.com/foo/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/..bar", - "base": null, - "href": "http://example.com/foo/..bar", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/..bar", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/bar/../ton", - "base": null, - "href": "http://example.com/foo/ton", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/ton", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/bar/../ton/../../a", - "base": null, - "href": "http://example.com/a", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/a", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/../../..", - "base": null, - "href": "http://example.com/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/../../../ton", - "base": null, - "href": "http://example.com/ton", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/ton", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/%2e", - "base": null, - "href": "http://example.com/foo/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/%2e%2", - "base": null, - "href": "http://example.com/foo/%2e%2", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/%2e%2", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/%2e./%2e%2e/.%2e/%2e.bar", - "base": null, - "href": "http://example.com/%2e.bar", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%2e.bar", - "search": "", - "hash": "" - }, - { - "input": "http://example.com////../..", - "base": null, - "href": "http://example.com//", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/bar//../..", - "base": null, - "href": "http://example.com/foo/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo/bar//..", - "base": null, - "href": "http://example.com/foo/bar/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo/bar/", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo", - "base": null, - "href": "http://example.com/foo", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/%20foo", - "base": null, - "href": "http://example.com/%20foo", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%20foo", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo%", - "base": null, - "href": "http://example.com/foo%", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo%", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo%2", - "base": null, - "href": "http://example.com/foo%2", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo%2", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo%2zbar", - "base": null, - "href": "http://example.com/foo%2zbar", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo%2zbar", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo%2Â©zbar", - "base": null, - "href": "http://example.com/foo%2%C3%82%C2%A9zbar", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo%2%C3%82%C2%A9zbar", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo%41%7a", - "base": null, - "href": "http://example.com/foo%41%7a", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo%41%7a", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo\t\u0091%91", - "base": null, - "href": "http://example.com/foo%C2%91%91", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo%C2%91%91", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo%00%51", - "base": null, - "href": "http://example.com/foo%00%51", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foo%00%51", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/(%28:%3A%29)", - "base": null, - "href": "http://example.com/(%28:%3A%29)", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/(%28:%3A%29)", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/%3A%3a%3C%3c", - "base": null, - "href": "http://example.com/%3A%3a%3C%3c", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%3A%3a%3C%3c", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/foo\tbar", - "base": null, - "href": "http://example.com/foobar", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/foobar", - "search": "", - "hash": "" - }, - { - "input": "http://example.com\\\\foo\\\\bar", - "base": null, - "href": "http://example.com//foo//bar", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "//foo//bar", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/%7Ffp3%3Eju%3Dduvgw%3Dd", - "base": null, - "href": "http://example.com/%7Ffp3%3Eju%3Dduvgw%3Dd", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%7Ffp3%3Eju%3Dduvgw%3Dd", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/@asdf%40", - "base": null, - "href": "http://example.com/@asdf%40", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/@asdf%40", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/你好你好", - "base": null, - "href": "http://example.com/%E4%BD%A0%E5%A5%BD%E4%BD%A0%E5%A5%BD", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%E4%BD%A0%E5%A5%BD%E4%BD%A0%E5%A5%BD", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/‥/foo", - "base": null, - "href": "http://example.com/%E2%80%A5/foo", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%E2%80%A5/foo", - "search": "", - "hash": "" - }, - { - "input": "http://example.com//foo", - "base": null, - "href": "http://example.com/%EF%BB%BF/foo", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%EF%BB%BF/foo", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/‮/foo/‭/bar", - "base": null, - "href": "http://example.com/%E2%80%AE/foo/%E2%80%AD/bar", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%E2%80%AE/foo/%E2%80%AD/bar", - "search": "", - "hash": "" - }, - "# Based on http://trac.webkit.org/browser/trunk/LayoutTests/fast/url/script-tests/relative.js", - { - "input": "http://www.google.com/foo?bar=baz#", - "base": null, - "href": "http://www.google.com/foo?bar=baz#", - "origin": "http://www.google.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.google.com", - "hostname": "www.google.com", - "port": "", - "pathname": "/foo", - "search": "?bar=baz", - "hash": "" - }, - { - "input": "http://www.google.com/foo?bar=baz# »", - "base": null, - "href": "http://www.google.com/foo?bar=baz#%20%C2%BB", - "origin": "http://www.google.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.google.com", - "hostname": "www.google.com", - "port": "", - "pathname": "/foo", - "search": "?bar=baz", - "hash": "#%20%C2%BB" - }, - { - "input": "data:test# »", - "base": null, - "href": "data:test#%20%C2%BB", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "test", - "search": "", - "hash": "#%20%C2%BB" - }, - { - "input": "http://www.google.com", - "base": null, - "href": "http://www.google.com/", - "origin": "http://www.google.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.google.com", - "hostname": "www.google.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://192.0x00A80001", - "base": null, - "href": "http://192.168.0.1/", - "origin": "http://192.168.0.1", - "protocol": "http:", - "username": "", - "password": "", - "host": "192.168.0.1", - "hostname": "192.168.0.1", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://www/foo%2Ehtml", - "base": null, - "href": "http://www/foo%2Ehtml", - "origin": "http://www", - "protocol": "http:", - "username": "", - "password": "", - "host": "www", - "hostname": "www", - "port": "", - "pathname": "/foo%2Ehtml", - "search": "", - "hash": "" - }, - { - "input": "http://www/foo/%2E/html", - "base": null, - "href": "http://www/foo/html", - "origin": "http://www", - "protocol": "http:", - "username": "", - "password": "", - "host": "www", - "hostname": "www", - "port": "", - "pathname": "/foo/html", - "search": "", - "hash": "" - }, - { - "input": "http://user:pass@/", - "base": null, - "failure": true - }, - { - "input": "http://%25DOMAIN:foobar@foodomain.com/", - "base": null, - "href": "http://%25DOMAIN:foobar@foodomain.com/", - "origin": "http://foodomain.com", - "protocol": "http:", - "username": "%25DOMAIN", - "password": "foobar", - "host": "foodomain.com", - "hostname": "foodomain.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:\\\\www.google.com\\foo", - "base": null, - "href": "http://www.google.com/foo", - "origin": "http://www.google.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.google.com", - "hostname": "www.google.com", - "port": "", - "pathname": "/foo", - "search": "", - "hash": "" - }, - { - "input": "http://foo:80/", - "base": null, - "href": "http://foo/", - "origin": "http://foo", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://foo:81/", - "base": null, - "href": "http://foo:81/", - "origin": "http://foo:81", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo:81", - "hostname": "foo", - "port": "81", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "httpa://foo:80/", - "base": null, - "href": "httpa://foo:80/", - "origin": "null", - "protocol": "httpa:", - "username": "", - "password": "", - "host": "foo:80", - "hostname": "foo", - "port": "80", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://foo:-80/", - "base": null, - "failure": true - }, - { - "input": "https://foo:443/", - "base": null, - "href": "https://foo/", - "origin": "https://foo", - "protocol": "https:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https://foo:80/", - "base": null, - "href": "https://foo:80/", - "origin": "https://foo:80", - "protocol": "https:", - "username": "", - "password": "", - "host": "foo:80", - "hostname": "foo", - "port": "80", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ftp://foo:21/", - "base": null, - "href": "ftp://foo/", - "origin": "ftp://foo", - "protocol": "ftp:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ftp://foo:80/", - "base": null, - "href": "ftp://foo:80/", - "origin": "ftp://foo:80", - "protocol": "ftp:", - "username": "", - "password": "", - "host": "foo:80", - "hostname": "foo", - "port": "80", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "gopher://foo:70/", - "base": null, - "href": "gopher://foo:70/", - "origin": "null", - "protocol": "gopher:", - "username": "", - "password": "", - "host": "foo:70", - "hostname": "foo", - "port": "70", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "gopher://foo:443/", - "base": null, - "href": "gopher://foo:443/", - "origin": "null", - "protocol": "gopher:", - "username": "", - "password": "", - "host": "foo:443", - "hostname": "foo", - "port": "443", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ws://foo:80/", - "base": null, - "href": "ws://foo/", - "origin": "ws://foo", - "protocol": "ws:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ws://foo:81/", - "base": null, - "href": "ws://foo:81/", - "origin": "ws://foo:81", - "protocol": "ws:", - "username": "", - "password": "", - "host": "foo:81", - "hostname": "foo", - "port": "81", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ws://foo:443/", - "base": null, - "href": "ws://foo:443/", - "origin": "ws://foo:443", - "protocol": "ws:", - "username": "", - "password": "", - "host": "foo:443", - "hostname": "foo", - "port": "443", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ws://foo:815/", - "base": null, - "href": "ws://foo:815/", - "origin": "ws://foo:815", - "protocol": "ws:", - "username": "", - "password": "", - "host": "foo:815", - "hostname": "foo", - "port": "815", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "wss://foo:80/", - "base": null, - "href": "wss://foo:80/", - "origin": "wss://foo:80", - "protocol": "wss:", - "username": "", - "password": "", - "host": "foo:80", - "hostname": "foo", - "port": "80", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "wss://foo:81/", - "base": null, - "href": "wss://foo:81/", - "origin": "wss://foo:81", - "protocol": "wss:", - "username": "", - "password": "", - "host": "foo:81", - "hostname": "foo", - "port": "81", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "wss://foo:443/", - "base": null, - "href": "wss://foo/", - "origin": "wss://foo", - "protocol": "wss:", - "username": "", - "password": "", - "host": "foo", - "hostname": "foo", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "wss://foo:815/", - "base": null, - "href": "wss://foo:815/", - "origin": "wss://foo:815", - "protocol": "wss:", - "username": "", - "password": "", - "host": "foo:815", - "hostname": "foo", - "port": "815", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:/example.com/", - "base": null, - "href": "http://example.com/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ftp:/example.com/", - "base": null, - "href": "ftp://example.com/", - "origin": "ftp://example.com", - "protocol": "ftp:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https:/example.com/", - "base": null, - "href": "https://example.com/", - "origin": "https://example.com", - "protocol": "https:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "madeupscheme:/example.com/", - "base": null, - "href": "madeupscheme:/example.com/", - "origin": "null", - "protocol": "madeupscheme:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "file:/example.com/", - "base": null, - "href": "file:///example.com/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "ftps:/example.com/", - "base": null, - "href": "ftps:/example.com/", - "origin": "null", - "protocol": "ftps:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "gopher:/example.com/", - "base": null, - "href": "gopher:/example.com/", - "origin": "null", - "protocol": "gopher:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "ws:/example.com/", - "base": null, - "href": "ws://example.com/", - "origin": "ws://example.com", - "protocol": "ws:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "wss:/example.com/", - "base": null, - "href": "wss://example.com/", - "origin": "wss://example.com", - "protocol": "wss:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "data:/example.com/", - "base": null, - "href": "data:/example.com/", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "javascript:/example.com/", - "base": null, - "href": "javascript:/example.com/", - "origin": "null", - "protocol": "javascript:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "mailto:/example.com/", - "base": null, - "href": "mailto:/example.com/", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/example.com/", - "search": "", - "hash": "" - }, - { - "input": "http:example.com/", - "base": null, - "href": "http://example.com/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ftp:example.com/", - "base": null, - "href": "ftp://example.com/", - "origin": "ftp://example.com", - "protocol": "ftp:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https:example.com/", - "base": null, - "href": "https://example.com/", - "origin": "https://example.com", - "protocol": "https:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "madeupscheme:example.com/", - "base": null, - "href": "madeupscheme:example.com/", - "origin": "null", - "protocol": "madeupscheme:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "ftps:example.com/", - "base": null, - "href": "ftps:example.com/", - "origin": "null", - "protocol": "ftps:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "gopher:example.com/", - "base": null, - "href": "gopher:example.com/", - "origin": "null", - "protocol": "gopher:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "ws:example.com/", - "base": null, - "href": "ws://example.com/", - "origin": "ws://example.com", - "protocol": "ws:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "wss:example.com/", - "base": null, - "href": "wss://example.com/", - "origin": "wss://example.com", - "protocol": "wss:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "data:example.com/", - "base": null, - "href": "data:example.com/", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "javascript:example.com/", - "base": null, - "href": "javascript:example.com/", - "origin": "null", - "protocol": "javascript:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - { - "input": "mailto:example.com/", - "base": null, - "href": "mailto:example.com/", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "example.com/", - "search": "", - "hash": "" - }, - "# Based on http://trac.webkit.org/browser/trunk/LayoutTests/fast/url/segments-userinfo-vs-host.html", - { - "input": "http:@www.example.com", - "base": null, - "href": "http://www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:/@www.example.com", - "base": null, - "href": "http://www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://@www.example.com", - "base": null, - "href": "http://www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:a:b@www.example.com", - "base": null, - "href": "http://a:b@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "a", - "password": "b", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:/a:b@www.example.com", - "base": null, - "href": "http://a:b@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "a", - "password": "b", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://a:b@www.example.com", - "base": null, - "href": "http://a:b@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "a", - "password": "b", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://@pple.com", - "base": null, - "href": "http://pple.com/", - "origin": "http://pple.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "pple.com", - "hostname": "pple.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http::b@www.example.com", - "base": null, - "href": "http://:b@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "b", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:/:b@www.example.com", - "base": null, - "href": "http://:b@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "b", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://:b@www.example.com", - "base": null, - "href": "http://:b@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "b", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:/:@/www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http://user@/www.example.com", - "base": null, - "failure": true - }, - { - "input": "http:@/www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http:/@/www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http://@/www.example.com", - "base": null, - "failure": true - }, - { - "input": "https:@/www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http:a:b@/www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http:/a:b@/www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http://a:b@/www.example.com", - "base": null, - "failure": true - }, - { - "input": "http::@/www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http:a:@www.example.com", - "base": null, - "href": "http://a@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "a", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:/a:@www.example.com", - "base": null, - "href": "http://a@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "a", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://a:@www.example.com", - "base": null, - "href": "http://a@www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "a", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://www.@pple.com", - "base": null, - "href": "http://www.@pple.com/", - "origin": "http://pple.com", - "protocol": "http:", - "username": "www.", - "password": "", - "host": "pple.com", - "hostname": "pple.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http:@:www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http:/@:www.example.com", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "http://@:www.example.com", - "base": null, - "failure": true - }, - { - "input": "http://:@www.example.com", - "base": null, - "href": "http://www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "# Others", - { - "input": "/", - "base": "http://www.example.com/test", - "href": "http://www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "/test.txt", - "base": "http://www.example.com/test", - "href": "http://www.example.com/test.txt", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/test.txt", - "search": "", - "hash": "" - }, - { - "input": ".", - "base": "http://www.example.com/test", - "href": "http://www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "..", - "base": "http://www.example.com/test", - "href": "http://www.example.com/", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "test.txt", - "base": "http://www.example.com/test", - "href": "http://www.example.com/test.txt", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/test.txt", - "search": "", - "hash": "" - }, - { - "input": "./test.txt", - "base": "http://www.example.com/test", - "href": "http://www.example.com/test.txt", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/test.txt", - "search": "", - "hash": "" - }, - { - "input": "../test.txt", - "base": "http://www.example.com/test", - "href": "http://www.example.com/test.txt", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/test.txt", - "search": "", - "hash": "" - }, - { - "input": "../aaa/test.txt", - "base": "http://www.example.com/test", - "href": "http://www.example.com/aaa/test.txt", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/aaa/test.txt", - "search": "", - "hash": "" - }, - { - "input": "../../test.txt", - "base": "http://www.example.com/test", - "href": "http://www.example.com/test.txt", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/test.txt", - "search": "", - "hash": "" - }, - { - "input": "中/test.txt", - "base": "http://www.example.com/test", - "href": "http://www.example.com/%E4%B8%AD/test.txt", - "origin": "http://www.example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example.com", - "hostname": "www.example.com", - "port": "", - "pathname": "/%E4%B8%AD/test.txt", - "search": "", - "hash": "" - }, - { - "input": "http://www.example2.com", - "base": "http://www.example.com/test", - "href": "http://www.example2.com/", - "origin": "http://www.example2.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example2.com", - "hostname": "www.example2.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "//www.example2.com", - "base": "http://www.example.com/test", - "href": "http://www.example2.com/", - "origin": "http://www.example2.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.example2.com", - "hostname": "www.example2.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "file:...", - "base": "http://www.example.com/test", - "href": "file:///...", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/...", - "search": "", - "hash": "" - }, - { - "input": "file:..", - "base": "http://www.example.com/test", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "file:a", - "base": "http://www.example.com/test", - "href": "file:///a", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/a", - "search": "", - "hash": "" - }, - { - "input": "file:.", - "base": null, - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "file:.", - "base": "http://www.example.com/test", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "# Based on http://trac.webkit.org/browser/trunk/LayoutTests/fast/url/host.html", - "Basic canonicalization, uppercase should be converted to lowercase", - { - "input": "http://ExAmPlE.CoM", - "base": "http://other.com/", - "href": "http://example.com/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://example example.com", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://Goo%20 goo%7C|.com", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[:]", - "base": "http://other.com/", - "failure": true - }, - "U+3000 is mapped to U+0020 (space) which is disallowed", - { - "input": "http://GOO\u00a0\u3000goo.com", - "base": "http://other.com/", - "failure": true - }, - "Other types of space (no-break, zero-width, zero-width-no-break) are name-prepped away to nothing. U+200B, U+2060, and U+FEFF, are ignored", - { - "input": "http://GOO\u200b\u2060\ufeffgoo.com", - "base": "http://other.com/", - "href": "http://googoo.com/", - "origin": "http://googoo.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "googoo.com", - "hostname": "googoo.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "Leading and trailing C0 control or space", - { - "input": "\u0000\u001b\u0004\u0012 http://example.com/\u001f \u000d ", - "base": null, - "href": "http://example.com/", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "Ideographic full stop (full-width period for Chinese, etc.) should be treated as a dot. U+3002 is mapped to U+002E (dot)", - { - "input": "http://www.foo。bar.com", - "base": "http://other.com/", - "href": "http://www.foo.bar.com/", - "origin": "http://www.foo.bar.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "www.foo.bar.com", - "hostname": "www.foo.bar.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "Invalid unicode characters should fail... U+FDD0 is disallowed; %ef%b7%90 is U+FDD0", - { - "input": "http://\ufdd0zyx.com", - "base": "http://other.com/", - "failure": true - }, - "This is the same as previous but escaped", - { - "input": "http://%ef%b7%90zyx.com", - "base": "http://other.com/", - "failure": true - }, - "U+FFFD", - { - "input": "https://\ufffd", - "base": null, - "failure": true - }, - { - "input": "https://%EF%BF%BD", - "base": null, - "failure": true - }, - { - "input": "https://x/\ufffd?\ufffd#\ufffd", - "base": null, - "href": "https://x/%EF%BF%BD?%EF%BF%BD#%EF%BF%BD", - "origin": "https://x", - "protocol": "https:", - "username": "", - "password": "", - "host": "x", - "hostname": "x", - "port": "", - "pathname": "/%EF%BF%BD", - "search": "?%EF%BF%BD", - "hash": "#%EF%BF%BD" - }, - "Domain is ASCII, but a label is invalid IDNA", - { - "input": "http://a.b.c.xn--pokxncvks", - "base": null, - "failure": true - }, - { - "input": "http://10.0.0.xn--pokxncvks", - "base": null, - "failure": true - }, - "IDNA labels should be matched case-insensitively", - { - "input": "http://a.b.c.XN--pokxncvks", - "base": null, - "failure": true - }, - { - "input": "http://a.b.c.Xn--pokxncvks", - "base": null, - "failure": true - }, - { - "input": "http://10.0.0.XN--pokxncvks", - "base": null, - "failure": true - }, - { - "input": "http://10.0.0.xN--pokxncvks", - "base": null, - "failure": true - }, - "Test name prepping, fullwidth input should be converted to ASCII and NOT IDN-ized. This is 'Go' in fullwidth UTF-8/UTF-16.", - { - "input": "http://Ｇｏ.com", - "base": "http://other.com/", - "href": "http://go.com/", - "origin": "http://go.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "go.com", - "hostname": "go.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "URL spec forbids the following. https://www.w3.org/Bugs/Public/show_bug.cgi?id=24257", - { - "input": "http://％４１.com", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://%ef%bc%85%ef%bc%94%ef%bc%91.com", - "base": "http://other.com/", - "failure": true - }, - "...%00 in fullwidth should fail (also as escaped UTF-8 input)", - { - "input": "http://％００.com", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://%ef%bc%85%ef%bc%90%ef%bc%90.com", - "base": "http://other.com/", - "failure": true - }, - "Basic IDN support, UTF-8 and UTF-16 input should be converted to IDN", - { - "input": "http://你好你好", - "base": "http://other.com/", - "href": "http://xn--6qqa088eba/", - "origin": "http://xn--6qqa088eba", - "protocol": "http:", - "username": "", - "password": "", - "host": "xn--6qqa088eba", - "hostname": "xn--6qqa088eba", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https://faß.ExAmPlE/", - "base": null, - "href": "https://xn--fa-hia.example/", - "origin": "https://xn--fa-hia.example", - "protocol": "https:", - "username": "", - "password": "", - "host": "xn--fa-hia.example", - "hostname": "xn--fa-hia.example", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "sc://faß.ExAmPlE/", - "base": null, - "href": "sc://fa%C3%9F.ExAmPlE/", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "fa%C3%9F.ExAmPlE", - "hostname": "fa%C3%9F.ExAmPlE", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "Invalid escaped characters should fail and the percents should be escaped. https://www.w3.org/Bugs/Public/show_bug.cgi?id=24191", - { - "input": "http://%zz%66%a.com", - "base": "http://other.com/", - "failure": true - }, - "If we get an invalid character that has been escaped.", - { - "input": "http://%25", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://hello%00", - "base": "http://other.com/", - "failure": true - }, - "Escaped numbers should be treated like IP addresses if they are.", - { - "input": "http://%30%78%63%30%2e%30%32%35%30.01", - "base": "http://other.com/", - "href": "http://192.168.0.1/", - "origin": "http://192.168.0.1", - "protocol": "http:", - "username": "", - "password": "", - "host": "192.168.0.1", - "hostname": "192.168.0.1", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://%30%78%63%30%2e%30%32%35%30.01%2e", - "base": "http://other.com/", - "href": "http://192.168.0.1/", - "origin": "http://192.168.0.1", - "protocol": "http:", - "username": "", - "password": "", - "host": "192.168.0.1", - "hostname": "192.168.0.1", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://192.168.0.257", - "base": "http://other.com/", - "failure": true - }, - "Invalid escaping in hosts causes failure", - { - "input": "http://%3g%78%63%30%2e%30%32%35%30%2E.01", - "base": "http://other.com/", - "failure": true - }, - "A space in a host causes failure", - { - "input": "http://192.168.0.1 hello", - "base": "http://other.com/", - "failure": true - }, - { - "input": "https://x x:12", - "base": null, - "failure": true - }, - "Fullwidth and escaped UTF-8 fullwidth should still be treated as IP", - { - "input": "http://０Ｘｃ０．０２５０．０１", - "base": "http://other.com/", - "href": "http://192.168.0.1/", - "origin": "http://192.168.0.1", - "protocol": "http:", - "username": "", - "password": "", - "host": "192.168.0.1", - "hostname": "192.168.0.1", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "Domains with empty labels", - { - "input": "http://./", - "base": null, - "href": "http://./", - "origin": "http://.", - "protocol": "http:", - "username": "", - "password": "", - "host": ".", - "hostname": ".", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://../", - "base": null, - "href": "http://../", - "origin": "http://..", - "protocol": "http:", - "username": "", - "password": "", - "host": "..", - "hostname": "..", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "Non-special domains with empty labels", - { - "input": "h://.", - "base": null, - "href": "h://.", - "origin": "null", - "protocol": "h:", - "username": "", - "password": "", - "host": ".", - "hostname": ".", - "port": "", - "pathname": "", - "search": "", - "hash": "" - }, - "Broken IPv6", - { - "input": "http://[www.google.com]/", - "base": null, - "failure": true - }, - { - "input": "http://[google.com]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[::1.2.3.4x]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[::1.2.3.]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[::1.2.]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[::.1.2]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[::1.]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[::.1]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://[::%31]", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://%5B::1]", - "base": "http://other.com/", - "failure": true - }, - "Misc Unicode", - { - "input": "http://foo:💩@example.com/bar", - "base": "http://other.com/", - "href": "http://foo:%F0%9F%92%A9@example.com/bar", - "origin": "http://example.com", - "protocol": "http:", - "username": "foo", - "password": "%F0%9F%92%A9", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/bar", - "search": "", - "hash": "" - }, - "# resolving a fragment against any scheme succeeds", - { - "input": "#", - "base": "test:test", - "href": "test:test#", - "origin": "null", - "protocol": "test:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "test", - "search": "", - "hash": "" - }, - { - "input": "#x", - "base": "mailto:x@x.com", - "href": "mailto:x@x.com#x", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "x@x.com", - "search": "", - "hash": "#x" - }, - { - "input": "#x", - "base": "data:,", - "href": "data:,#x", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": ",", - "search": "", - "hash": "#x" - }, - { - "input": "#x", - "base": "about:blank", - "href": "about:blank#x", - "origin": "null", - "protocol": "about:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "blank", - "search": "", - "hash": "#x" - }, - { - "input": "#x:y", - "base": "about:blank", - "href": "about:blank#x:y", - "origin": "null", - "protocol": "about:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "blank", - "search": "", - "hash": "#x:y" - }, - { - "input": "#", - "base": "test:test?test", - "href": "test:test?test#", - "origin": "null", - "protocol": "test:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "test", - "search": "?test", - "hash": "" - }, - "# multiple @ in authority state", - { - "input": "https://@test@test@example:800/", - "base": "http://doesnotmatter/", - "href": "https://%40test%40test@example:800/", - "origin": "https://example:800", - "protocol": "https:", - "username": "%40test%40test", - "password": "", - "host": "example:800", - "hostname": "example", - "port": "800", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https://@@@example", - "base": "http://doesnotmatter/", - "href": "https://%40%40@example/", - "origin": "https://example", - "protocol": "https:", - "username": "%40%40", - "password": "", - "host": "example", - "hostname": "example", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "non-az-09 characters", - { - "input": "http://`{}:`{}@h/`{}?`{}", - "base": "http://doesnotmatter/", - "href": "http://%60%7B%7D:%60%7B%7D@h/%60%7B%7D?`{}", - "origin": "http://h", - "protocol": "http:", - "username": "%60%7B%7D", - "password": "%60%7B%7D", - "host": "h", - "hostname": "h", - "port": "", - "pathname": "/%60%7B%7D", - "search": "?`{}", - "hash": "" - }, - "byte is ' and url is special", - { - "input": "http://host/?'", - "base": null, - "href": "http://host/?%27", - "origin": "http://host", - "protocol": "http:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/", - "search": "?%27", - "hash": "" - }, - { - "input": "notspecial://host/?'", - "base": null, - "href": "notspecial://host/?'", - "origin": "null", - "protocol": "notspecial:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/", - "search": "?'", - "hash": "" - }, - "# Credentials in base", - { - "input": "/some/path", - "base": "http://user@example.org/smth", - "href": "http://user@example.org/some/path", - "origin": "http://example.org", - "protocol": "http:", - "username": "user", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/some/path", - "search": "", - "hash": "" - }, - { - "input": "", - "base": "http://user:pass@example.org:21/smth", - "href": "http://user:pass@example.org:21/smth", - "origin": "http://example.org:21", - "protocol": "http:", - "username": "user", - "password": "pass", - "host": "example.org:21", - "hostname": "example.org", - "port": "21", - "pathname": "/smth", - "search": "", - "hash": "" - }, - { - "input": "/some/path", - "base": "http://user:pass@example.org:21/smth", - "href": "http://user:pass@example.org:21/some/path", - "origin": "http://example.org:21", - "protocol": "http:", - "username": "user", - "password": "pass", - "host": "example.org:21", - "hostname": "example.org", - "port": "21", - "pathname": "/some/path", - "search": "", - "hash": "" - }, - "# a set of tests designed by zcorpan for relative URLs with unknown schemes", - { - "input": "i", - "base": "sc:sd", - "failure": true - }, - { - "input": "i", - "base": "sc:sd/sd", - "failure": true - }, - { - "input": "i", - "base": "sc:/pa/pa", - "href": "sc:/pa/i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/pa/i", - "search": "", - "hash": "" - }, - { - "input": "i", - "base": "sc://ho/pa", - "href": "sc://ho/i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "ho", - "hostname": "ho", - "port": "", - "pathname": "/i", - "search": "", - "hash": "" - }, - { - "input": "i", - "base": "sc:///pa/pa", - "href": "sc:///pa/i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/pa/i", - "search": "", - "hash": "" - }, - { - "input": "../i", - "base": "sc:sd", - "failure": true - }, - { - "input": "../i", - "base": "sc:sd/sd", - "failure": true - }, - { - "input": "../i", - "base": "sc:/pa/pa", - "href": "sc:/i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/i", - "search": "", - "hash": "" - }, - { - "input": "../i", - "base": "sc://ho/pa", - "href": "sc://ho/i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "ho", - "hostname": "ho", - "port": "", - "pathname": "/i", - "search": "", - "hash": "" - }, - { - "input": "../i", - "base": "sc:///pa/pa", - "href": "sc:///i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/i", - "search": "", - "hash": "" - }, - { - "input": "/i", - "base": "sc:sd", - "failure": true - }, - { - "input": "/i", - "base": "sc:sd/sd", - "failure": true - }, - { - "input": "/i", - "base": "sc:/pa/pa", - "href": "sc:/i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/i", - "search": "", - "hash": "" - }, - { - "input": "/i", - "base": "sc://ho/pa", - "href": "sc://ho/i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "ho", - "hostname": "ho", - "port": "", - "pathname": "/i", - "search": "", - "hash": "" - }, - { - "input": "/i", - "base": "sc:///pa/pa", - "href": "sc:///i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/i", - "search": "", - "hash": "" - }, - { - "input": "?i", - "base": "sc:sd", - "failure": true - }, - { - "input": "?i", - "base": "sc:sd/sd", - "failure": true - }, - { - "input": "?i", - "base": "sc:/pa/pa", - "href": "sc:/pa/pa?i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/pa/pa", - "search": "?i", - "hash": "" - }, - { - "input": "?i", - "base": "sc://ho/pa", - "href": "sc://ho/pa?i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "ho", - "hostname": "ho", - "port": "", - "pathname": "/pa", - "search": "?i", - "hash": "" - }, - { - "input": "?i", - "base": "sc:///pa/pa", - "href": "sc:///pa/pa?i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/pa/pa", - "search": "?i", - "hash": "" - }, - { - "input": "#i", - "base": "sc:sd", - "href": "sc:sd#i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "sd", - "search": "", - "hash": "#i" - }, - { - "input": "#i", - "base": "sc:sd/sd", - "href": "sc:sd/sd#i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "sd/sd", - "search": "", - "hash": "#i" - }, - { - "input": "#i", - "base": "sc:/pa/pa", - "href": "sc:/pa/pa#i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/pa/pa", - "search": "", - "hash": "#i" - }, - { - "input": "#i", - "base": "sc://ho/pa", - "href": "sc://ho/pa#i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "ho", - "hostname": "ho", - "port": "", - "pathname": "/pa", - "search": "", - "hash": "#i" - }, - { - "input": "#i", - "base": "sc:///pa/pa", - "href": "sc:///pa/pa#i", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/pa/pa", - "search": "", - "hash": "#i" - }, - "# make sure that relative URL logic works on known typically non-relative schemes too", - { - "input": "about:/../", - "base": null, - "href": "about:/", - "origin": "null", - "protocol": "about:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "data:/../", - "base": null, - "href": "data:/", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "javascript:/../", - "base": null, - "href": "javascript:/", - "origin": "null", - "protocol": "javascript:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "mailto:/../", - "base": null, - "href": "mailto:/", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "# unknown schemes and their hosts", - { - "input": "sc://ñ.test/", - "base": null, - "href": "sc://%C3%B1.test/", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%C3%B1.test", - "hostname": "%C3%B1.test", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "sc://%/", - "base": null, - "href": "sc://%/", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%", - "hostname": "%", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "sc://@/", - "base": null, - "failure": true - }, - { - "input": "sc://te@s:t@/", - "base": null, - "failure": true - }, - { - "input": "sc://:/", - "base": null, - "failure": true - }, - { - "input": "sc://:12/", - "base": null, - "failure": true - }, - { - "input": "x", - "base": "sc://ñ", - "href": "sc://%C3%B1/x", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%C3%B1", - "hostname": "%C3%B1", - "port": "", - "pathname": "/x", - "search": "", - "hash": "" - }, - "# unknown schemes and backslashes", - { - "input": "sc:\\../", - "base": null, - "href": "sc:\\../", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "\\../", - "search": "", - "hash": "" - }, - "# unknown scheme with path looking like a password", - { - "input": "sc::a@example.net", - "base": null, - "href": "sc::a@example.net", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": ":a@example.net", - "search": "", - "hash": "" - }, - "# unknown scheme with bogus percent-encoding", - { - "input": "wow:%NBD", - "base": null, - "href": "wow:%NBD", - "origin": "null", - "protocol": "wow:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "%NBD", - "search": "", - "hash": "" - }, - { - "input": "wow:%1G", - "base": null, - "href": "wow:%1G", - "origin": "null", - "protocol": "wow:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "%1G", - "search": "", - "hash": "" - }, - "# unknown scheme with non-URL characters", - { - "input": "wow:\uFFFF", - "base": null, - "href": "wow:%EF%BF%BF", - "origin": "null", - "protocol": "wow:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "%EF%BF%BF", - "search": "", - "hash": "" - }, - { - "input": "http://example.com/\uD800\uD801\uDFFE\uDFFF\uFDD0\uFDCF\uFDEF\uFDF0\uFFFE\uFFFF?\uD800\uD801\uDFFE\uDFFF\uFDD0\uFDCF\uFDEF\uFDF0\uFFFE\uFFFF", - "base": null, - "href": "http://example.com/%EF%BF%BD%F0%90%9F%BE%EF%BF%BD%EF%B7%90%EF%B7%8F%EF%B7%AF%EF%B7%B0%EF%BF%BE%EF%BF%BF?%EF%BF%BD%F0%90%9F%BE%EF%BF%BD%EF%B7%90%EF%B7%8F%EF%B7%AF%EF%B7%B0%EF%BF%BE%EF%BF%BF", - "origin": "http://example.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.com", - "hostname": "example.com", - "port": "", - "pathname": "/%EF%BF%BD%F0%90%9F%BE%EF%BF%BD%EF%B7%90%EF%B7%8F%EF%B7%AF%EF%B7%B0%EF%BF%BE%EF%BF%BF", - "search": "?%EF%BF%BD%F0%90%9F%BE%EF%BF%BD%EF%B7%90%EF%B7%8F%EF%B7%AF%EF%B7%B0%EF%BF%BE%EF%BF%BF", - "hash": "" - }, - "Forbidden host code points", - { - "input": "sc://a\u0000b/", - "base": null, - "failure": true - }, - { - "input": "sc://a b/", - "base": null, - "failure": true - }, - { - "input": "sc://ab", - "base": null, - "failure": true - }, - { - "input": "sc://a[b/", - "base": null, - "failure": true - }, - { - "input": "sc://a\\b/", - "base": null, - "failure": true - }, - { - "input": "sc://a]b/", - "base": null, - "failure": true - }, - { - "input": "sc://a^b", - "base": null, - "failure": true - }, - { - "input": "sc://a|b/", - "base": null, - "failure": true - }, - "Forbidden host codepoints: tabs and newlines are removed during preprocessing", - { - "input": "foo://ho\u0009st/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href":"foo://host/", - "password": "", - "pathname": "/", - "port":"", - "protocol": "foo:", - "search": "", - "username": "" - }, - { - "input": "foo://ho\u000Ast/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href":"foo://host/", - "password": "", - "pathname": "/", - "port":"", - "protocol": "foo:", - "search": "", - "username": "" - }, - { - "input": "foo://ho\u000Dst/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href":"foo://host/", - "password": "", - "pathname": "/", - "port":"", - "protocol": "foo:", - "search": "", - "username": "" - }, - "Forbidden domain code-points", - { - "input": "http://a\u0000b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0001b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0002b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0003b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0004b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0005b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0006b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0007b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0008b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u000Bb/", - "base": null, - "failure": true - }, - { - "input": "http://a\u000Cb/", - "base": null, - "failure": true - }, - { - "input": "http://a\u000Eb/", - "base": null, - "failure": true - }, - { - "input": "http://a\u000Fb/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0010b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0011b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0012b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0013b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0014b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0015b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0016b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0017b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0018b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u0019b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u001Ab/", - "base": null, - "failure": true - }, - { - "input": "http://a\u001Bb/", - "base": null, - "failure": true - }, - { - "input": "http://a\u001Cb/", - "base": null, - "failure": true - }, - { - "input": "http://a\u001Db/", - "base": null, - "failure": true - }, - { - "input": "http://a\u001Eb/", - "base": null, - "failure": true - }, - { - "input": "http://a\u001Fb/", - "base": null, - "failure": true - }, - { - "input": "http://a b/", - "base": null, - "failure": true - }, - { - "input": "http://a%b/", - "base": null, - "failure": true - }, - { - "input": "http://ab", - "base": null, - "failure": true - }, - { - "input": "http://a[b/", - "base": null, - "failure": true - }, - { - "input": "http://a]b/", - "base": null, - "failure": true - }, - { - "input": "http://a^b", - "base": null, - "failure": true - }, - { - "input": "http://a|b/", - "base": null, - "failure": true - }, - { - "input": "http://a\u007Fb/", - "base": null, - "failure": true - }, - "Forbidden domain codepoints: tabs and newlines are removed during preprocessing", - { - "input": "http://ho\u0009st/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href":"http://host/", - "password": "", - "pathname": "/", - "port":"", - "protocol": "http:", - "search": "", - "username": "" - }, - { - "input": "http://ho\u000Ast/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href":"http://host/", - "password": "", - "pathname": "/", - "port":"", - "protocol": "http:", - "search": "", - "username": "" - }, - { - "input": "http://ho\u000Dst/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href":"http://host/", - "password": "", - "pathname": "/", - "port":"", - "protocol": "http:", - "search": "", - "username": "" - }, - "Encoded forbidden domain codepoints in special URLs", - { - "input": "http://ho%00st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%01st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%02st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%03st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%04st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%05st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%06st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%07st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%08st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%09st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%0Ast/", - "base": null, - "failure": true - }, - { - "input": "http://ho%0Bst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%0Cst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%0Dst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%0Est/", - "base": null, - "failure": true - }, - { - "input": "http://ho%0Fst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%10st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%11st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%12st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%13st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%14st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%15st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%16st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%17st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%18st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%19st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%1Ast/", - "base": null, - "failure": true - }, - { - "input": "http://ho%1Bst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%1Cst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%1Dst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%1Est/", - "base": null, - "failure": true - }, - { - "input": "http://ho%1Fst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%20st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%23st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%25st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%2Fst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%3Ast/", - "base": null, - "failure": true - }, - { - "input": "http://ho%3Cst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%3Est/", - "base": null, - "failure": true - }, - { - "input": "http://ho%3Fst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%40st/", - "base": null, - "failure": true - }, - { - "input": "http://ho%5Bst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%5Cst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%5Dst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%7Cst/", - "base": null, - "failure": true - }, - { - "input": "http://ho%7Fst/", - "base": null, - "failure": true - }, - "Allowed host/domain code points", - { - "input": "http://!\"$&'()*+,-.;=_`{}~/", - "base": null, - "href": "http://!\"$&'()*+,-.;=_`{}~/", - "origin": "http://!\"$&'()*+,-.;=_`{}~", - "protocol": "http:", - "username": "", - "password": "", - "host": "!\"$&'()*+,-.;=_`{}~", - "hostname": "!\"$&'()*+,-.;=_`{}~", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "sc://\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008\u000B\u000C\u000E\u000F\u0010\u0011\u0012\u0013\u0014\u0015\u0016\u0017\u0018\u0019\u001A\u001B\u001C\u001D\u001E\u001F\u007F!\"$%&'()*+,-.;=_`{}~/", - "base": null, - "href": "sc://%01%02%03%04%05%06%07%08%0B%0C%0E%0F%10%11%12%13%14%15%16%17%18%19%1A%1B%1C%1D%1E%1F%7F!\"$%&'()*+,-.;=_`{}~/", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%01%02%03%04%05%06%07%08%0B%0C%0E%0F%10%11%12%13%14%15%16%17%18%19%1A%1B%1C%1D%1E%1F%7F!\"$%&'()*+,-.;=_`{}~", - "hostname": "%01%02%03%04%05%06%07%08%0B%0C%0E%0F%10%11%12%13%14%15%16%17%18%19%1A%1B%1C%1D%1E%1F%7F!\"$%&'()*+,-.;=_`{}~", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "# Hosts and percent-encoding", - { - "input": "ftp://example.com%80/", - "base": null, - "failure": true - }, - { - "input": "ftp://example.com%A0/", - "base": null, - "failure": true - }, - { - "input": "https://example.com%80/", - "base": null, - "failure": true - }, - { - "input": "https://example.com%A0/", - "base": null, - "failure": true - }, - { - "input": "ftp://%e2%98%83", - "base": null, - "href": "ftp://xn--n3h/", - "origin": "ftp://xn--n3h", - "protocol": "ftp:", - "username": "", - "password": "", - "host": "xn--n3h", - "hostname": "xn--n3h", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "https://%e2%98%83", - "base": null, - "href": "https://xn--n3h/", - "origin": "https://xn--n3h", - "protocol": "https:", - "username": "", - "password": "", - "host": "xn--n3h", - "hostname": "xn--n3h", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "# tests from jsdom/whatwg-url designed for code coverage", - { - "input": "http://127.0.0.1:10100/relative_import.html", - "base": null, - "href": "http://127.0.0.1:10100/relative_import.html", - "origin": "http://127.0.0.1:10100", - "protocol": "http:", - "username": "", - "password": "", - "host": "127.0.0.1:10100", - "hostname": "127.0.0.1", - "port": "10100", - "pathname": "/relative_import.html", - "search": "", - "hash": "" - }, - { - "input": "http://facebook.com/?foo=%7B%22abc%22", - "base": null, - "href": "http://facebook.com/?foo=%7B%22abc%22", - "origin": "http://facebook.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "facebook.com", - "hostname": "facebook.com", - "port": "", - "pathname": "/", - "search": "?foo=%7B%22abc%22", - "hash": "" - }, - { - "input": "https://localhost:3000/jqueryui@1.2.3", - "base": null, - "href": "https://localhost:3000/jqueryui@1.2.3", - "origin": "https://localhost:3000", - "protocol": "https:", - "username": "", - "password": "", - "host": "localhost:3000", - "hostname": "localhost", - "port": "3000", - "pathname": "/jqueryui@1.2.3", - "search": "", - "hash": "" - }, - "# tab/LF/CR", - { - "input": "h\tt\nt\rp://h\to\ns\rt:9\t0\n0\r0/p\ta\nt\rh?q\tu\ne\rry#f\tr\na\rg", - "base": null, - "href": "http://host:9000/path?query#frag", - "origin": "http://host:9000", - "protocol": "http:", - "username": "", - "password": "", - "host": "host:9000", - "hostname": "host", - "port": "9000", - "pathname": "/path", - "search": "?query", - "hash": "#frag" - }, - "# Stringification of URL.searchParams", - { - "input": "?a=b&c=d", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar?a=b&c=d", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "?a=b&c=d", - "searchParams": "a=b&c=d", - "hash": "" - }, - { - "input": "??a=b&c=d", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar??a=b&c=d", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "??a=b&c=d", - "searchParams": "%3Fa=b&c=d", - "hash": "" - }, - "# Scheme only", - { - "input": "http:", - "base": "http://example.org/foo/bar", - "href": "http://example.org/foo/bar", - "origin": "http://example.org", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/foo/bar", - "search": "", - "searchParams": "", - "hash": "" - }, - { - "input": "http:", - "base": "https://example.org/foo/bar", - "failure": true - }, - { - "input": "sc:", - "base": "https://example.org/foo/bar", - "href": "sc:", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "", - "search": "", - "searchParams": "", - "hash": "" - }, - "# Percent encoding of fragments", - { - "input": "http://foo.bar/baz?qux#foo\bbar", - "base": null, - "href": "http://foo.bar/baz?qux#foo%08bar", - "origin": "http://foo.bar", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo.bar", - "hostname": "foo.bar", - "port": "", - "pathname": "/baz", - "search": "?qux", - "searchParams": "qux=", - "hash": "#foo%08bar" - }, - { - "input": "http://foo.bar/baz?qux#foo\"bar", - "base": null, - "href": "http://foo.bar/baz?qux#foo%22bar", - "origin": "http://foo.bar", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo.bar", - "hostname": "foo.bar", - "port": "", - "pathname": "/baz", - "search": "?qux", - "searchParams": "qux=", - "hash": "#foo%22bar" - }, - { - "input": "http://foo.bar/baz?qux#foobar", - "base": null, - "href": "http://foo.bar/baz?qux#foo%3Ebar", - "origin": "http://foo.bar", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo.bar", - "hostname": "foo.bar", - "port": "", - "pathname": "/baz", - "search": "?qux", - "searchParams": "qux=", - "hash": "#foo%3Ebar" - }, - { - "input": "http://foo.bar/baz?qux#foo`bar", - "base": null, - "href": "http://foo.bar/baz?qux#foo%60bar", - "origin": "http://foo.bar", - "protocol": "http:", - "username": "", - "password": "", - "host": "foo.bar", - "hostname": "foo.bar", - "port": "", - "pathname": "/baz", - "search": "?qux", - "searchParams": "qux=", - "hash": "#foo%60bar" - }, - "# IPv4 parsing (via https://github.com/nodejs/node/pull/10317)", - { - "input": "http://1.2.3.4/", - "base": "http://other.com/", - "href": "http://1.2.3.4/", - "origin": "http://1.2.3.4", - "protocol": "http:", - "username": "", - "password": "", - "host": "1.2.3.4", - "hostname": "1.2.3.4", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://1.2.3.4./", - "base": "http://other.com/", - "href": "http://1.2.3.4/", - "origin": "http://1.2.3.4", - "protocol": "http:", - "username": "", - "password": "", - "host": "1.2.3.4", - "hostname": "1.2.3.4", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://192.168.257", - "base": "http://other.com/", - "href": "http://192.168.1.1/", - "origin": "http://192.168.1.1", - "protocol": "http:", - "username": "", - "password": "", - "host": "192.168.1.1", - "hostname": "192.168.1.1", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://192.168.257.", - "base": "http://other.com/", - "href": "http://192.168.1.1/", - "origin": "http://192.168.1.1", - "protocol": "http:", - "username": "", - "password": "", - "host": "192.168.1.1", - "hostname": "192.168.1.1", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://192.168.257.com", - "base": "http://other.com/", - "href": "http://192.168.257.com/", - "origin": "http://192.168.257.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "192.168.257.com", - "hostname": "192.168.257.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://256", - "base": "http://other.com/", - "href": "http://0.0.1.0/", - "origin": "http://0.0.1.0", - "protocol": "http:", - "username": "", - "password": "", - "host": "0.0.1.0", - "hostname": "0.0.1.0", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://256.com", - "base": "http://other.com/", - "href": "http://256.com/", - "origin": "http://256.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "256.com", - "hostname": "256.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://999999999", - "base": "http://other.com/", - "href": "http://59.154.201.255/", - "origin": "http://59.154.201.255", - "protocol": "http:", - "username": "", - "password": "", - "host": "59.154.201.255", - "hostname": "59.154.201.255", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://999999999.", - "base": "http://other.com/", - "href": "http://59.154.201.255/", - "origin": "http://59.154.201.255", - "protocol": "http:", - "username": "", - "password": "", - "host": "59.154.201.255", - "hostname": "59.154.201.255", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://999999999.com", - "base": "http://other.com/", - "href": "http://999999999.com/", - "origin": "http://999999999.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "999999999.com", - "hostname": "999999999.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://10000000000", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://10000000000.com", - "base": "http://other.com/", - "href": "http://10000000000.com/", - "origin": "http://10000000000.com", - "protocol": "http:", - "username": "", - "password": "", - "host": "10000000000.com", - "hostname": "10000000000.com", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://4294967295", - "base": "http://other.com/", - "href": "http://255.255.255.255/", - "origin": "http://255.255.255.255", - "protocol": "http:", - "username": "", - "password": "", - "host": "255.255.255.255", - "hostname": "255.255.255.255", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://4294967296", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://0xffffffff", - "base": "http://other.com/", - "href": "http://255.255.255.255/", - "origin": "http://255.255.255.255", - "protocol": "http:", - "username": "", - "password": "", - "host": "255.255.255.255", - "hostname": "255.255.255.255", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://0xffffffff1", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://256.256.256.256", - "base": "http://other.com/", - "failure": true - }, - { - "input": "https://0x.0x.0", - "base": null, - "href": "https://0.0.0.0/", - "origin": "https://0.0.0.0", - "protocol": "https:", - "username": "", - "password": "", - "host": "0.0.0.0", - "hostname": "0.0.0.0", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "More IPv4 parsing (via https://github.com/jsdom/whatwg-url/issues/92)", - { - "input": "https://0x100000000/test", - "base": null, - "failure": true - }, - { - "input": "https://256.0.0.1/test", - "base": null, - "failure": true - }, - "# file URLs containing percent-encoded Windows drive letters (shouldn't work)", - { - "input": "file:///C%3A/", - "base": null, - "href": "file:///C%3A/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C%3A/", - "search": "", - "hash": "" - }, - { - "input": "file:///C%7C/", - "base": null, - "href": "file:///C%7C/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C%7C/", - "search": "", - "hash": "" - }, - { - "input": "file://%43%3A", - "base": null, - "failure": true - }, - { - "input": "file://%43%7C", - "base": null, - "failure": true - }, - { - "input": "file://%43|", - "base": null, - "failure": true - }, - { - "input": "file://C%7C", - "base": null, - "failure": true - }, - { - "input": "file://%43%7C/", - "base": null, - "failure": true - }, - { - "input": "https://%43%7C/", - "base": null, - "failure": true - }, - { - "input": "asdf://%43|/", - "base": null, - "failure": true - }, - { - "input": "asdf://%43%7C/", - "base": null, - "href": "asdf://%43%7C/", - "origin": "null", - "protocol": "asdf:", - "username": "", - "password": "", - "host": "%43%7C", - "hostname": "%43%7C", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "# file URLs relative to other file URLs (via https://github.com/jsdom/whatwg-url/pull/60)", - { - "input": "pix/submit.gif", - "base": "file:///C:/Users/Domenic/Dropbox/GitHub/tmpvar/jsdom/test/level2/html/files/anchor.html", - "href": "file:///C:/Users/Domenic/Dropbox/GitHub/tmpvar/jsdom/test/level2/html/files/pix/submit.gif", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/Users/Domenic/Dropbox/GitHub/tmpvar/jsdom/test/level2/html/files/pix/submit.gif", - "search": "", - "hash": "" - }, - { - "input": "..", - "base": "file:///C:/", - "href": "file:///C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "..", - "base": "file:///", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "# More file URL tests by zcorpan and annevk", - { - "input": "/", - "base": "file:///C:/a/b", - "href": "file:///C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "/", - "base": "file://h/C:/a/b", - "href": "file://h/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "h", - "hostname": "h", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "/", - "base": "file://h/a/b", - "href": "file://h/", - "protocol": "file:", - "username": "", - "password": "", - "host": "h", - "hostname": "h", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "//d:", - "base": "file:///C:/a/b", - "href": "file:///d:", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/d:", - "search": "", - "hash": "" - }, - { - "input": "//d:/..", - "base": "file:///C:/a/b", - "href": "file:///d:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/d:/", - "search": "", - "hash": "" - }, - { - "input": "..", - "base": "file:///ab:/", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "..", - "base": "file:///1:/", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "", - "base": "file:///test?test#test", - "href": "file:///test?test", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "?test", - "hash": "" - }, - { - "input": "file:", - "base": "file:///test?test#test", - "href": "file:///test?test", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "?test", - "hash": "" - }, - { - "input": "?x", - "base": "file:///test?test#test", - "href": "file:///test?x", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "?x", - "hash": "" - }, - { - "input": "file:?x", - "base": "file:///test?test#test", - "href": "file:///test?x", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "?x", - "hash": "" - }, - { - "input": "#x", - "base": "file:///test?test#test", - "href": "file:///test?test#x", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "?test", - "hash": "#x" - }, - { - "input": "file:#x", - "base": "file:///test?test#test", - "href": "file:///test?test#x", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "?test", - "hash": "#x" - }, - "# File URLs and many (back)slashes", - { - "input": "file:\\\\//", - "base": null, - "href": "file:////", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - { - "input": "file:\\\\\\\\", - "base": null, - "href": "file:////", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - { - "input": "file:\\\\\\\\?fox", - "base": null, - "href": "file:////?fox", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "?fox", - "hash": "" - }, - { - "input": "file:\\\\\\\\#guppy", - "base": null, - "href": "file:////#guppy", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "#guppy" - }, - { - "input": "file://spider///", - "base": null, - "href": "file://spider///", - "protocol": "file:", - "username": "", - "password": "", - "host": "spider", - "hostname": "spider", - "port": "", - "pathname": "///", - "search": "", - "hash": "" - }, - { - "input": "file:\\\\localhost//", - "base": null, - "href": "file:////", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - { - "input": "file:///localhost//cat", - "base": null, - "href": "file:///localhost//cat", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/localhost//cat", - "search": "", - "hash": "" - }, - { - "input": "file://\\/localhost//cat", - "base": null, - "href": "file:////localhost//cat", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//localhost//cat", - "search": "", - "hash": "" - }, - { - "input": "file://localhost//a//../..//", - "base": null, - "href": "file://///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "///", - "search": "", - "hash": "" - }, - { - "input": "/////mouse", - "base": "file:///elephant", - "href": "file://///mouse", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "///mouse", - "search": "", - "hash": "" - }, - { - "input": "\\//pig", - "base": "file://lion/", - "href": "file:///pig", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/pig", - "search": "", - "hash": "" - }, - { - "input": "\\/localhost//pig", - "base": "file://lion/", - "href": "file:////pig", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//pig", - "search": "", - "hash": "" - }, - { - "input": "//localhost//pig", - "base": "file://lion/", - "href": "file:////pig", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//pig", - "search": "", - "hash": "" - }, - { - "input": "/..//localhost//pig", - "base": "file://lion/", - "href": "file://lion//localhost//pig", - "protocol": "file:", - "username": "", - "password": "", - "host": "lion", - "hostname": "lion", - "port": "", - "pathname": "//localhost//pig", - "search": "", - "hash": "" - }, - { - "input": "file://", - "base": "file://ape/", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "# File URLs with non-empty hosts", - { - "input": "/rooibos", - "base": "file://tea/", - "href": "file://tea/rooibos", - "protocol": "file:", - "username": "", - "password": "", - "host": "tea", - "hostname": "tea", - "port": "", - "pathname": "/rooibos", - "search": "", - "hash": "" - }, - { - "input": "/?chai", - "base": "file://tea/", - "href": "file://tea/?chai", - "protocol": "file:", - "username": "", - "password": "", - "host": "tea", - "hostname": "tea", - "port": "", - "pathname": "/", - "search": "?chai", - "hash": "" - }, - "# Windows drive letter handling with the 'file:' base URL", - { - "input": "C|", - "base": "file://host/dir/file", - "href": "file://host/C:", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:", - "search": "", - "hash": "" - }, - { - "input": "C|", - "base": "file://host/D:/dir1/dir2/file", - "href": "file://host/C:", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:", - "search": "", - "hash": "" - }, - { - "input": "C|#", - "base": "file://host/dir/file", - "href": "file://host/C:#", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:", - "search": "", - "hash": "" - }, - { - "input": "C|?", - "base": "file://host/dir/file", - "href": "file://host/C:?", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:", - "search": "", - "hash": "" - }, - { - "input": "C|/", - "base": "file://host/dir/file", - "href": "file://host/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "C|\n/", - "base": "file://host/dir/file", - "href": "file://host/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "C|\\", - "base": "file://host/dir/file", - "href": "file://host/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "C", - "base": "file://host/dir/file", - "href": "file://host/dir/C", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/dir/C", - "search": "", - "hash": "" - }, - { - "input": "C|a", - "base": "file://host/dir/file", - "href": "file://host/dir/C|a", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/dir/C|a", - "search": "", - "hash": "" - }, - "# Windows drive letter quirk in the file slash state", - { - "input": "/c:/foo/bar", - "base": "file:///c:/baz/qux", - "href": "file:///c:/foo/bar", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/c:/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "/c|/foo/bar", - "base": "file:///c:/baz/qux", - "href": "file:///c:/foo/bar", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/c:/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "file:\\c:\\foo\\bar", - "base": "file:///c:/baz/qux", - "href": "file:///c:/foo/bar", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/c:/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "/c:/foo/bar", - "base": "file://host/path", - "href": "file://host/c:/foo/bar", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/c:/foo/bar", - "search": "", - "hash": "" - }, - "# Do not drop the host in the presence of a drive letter", - { - "input": "file://example.net/C:/", - "base": null, - "href": "file://example.net/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "example.net", - "hostname": "example.net", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "file://1.2.3.4/C:/", - "base": null, - "href": "file://1.2.3.4/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "1.2.3.4", - "hostname": "1.2.3.4", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "file://[1::8]/C:/", - "base": null, - "href": "file://[1::8]/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "[1::8]", - "hostname": "[1::8]", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - "# Copy the host from the base URL in the following cases", - { - "input": "C|/", - "base": "file://host/", - "href": "file://host/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "/C:/", - "base": "file://host/", - "href": "file://host/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "file:C:/", - "base": "file://host/", - "href": "file://host/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "file:/C:/", - "base": "file://host/", - "href": "file://host/C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "host", - "hostname": "host", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - "# Copy the empty host from the input in the following cases", - { - "input": "//C:/", - "base": "file://host/", - "href": "file:///C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "file://C:/", - "base": "file://host/", - "href": "file:///C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "///C:/", - "base": "file://host/", - "href": "file:///C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "file:///C:/", - "base": "file://host/", - "href": "file:///C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - "# Windows drive letter quirk (no host)", - { - "input": "file:/C|/", - "base": null, - "href": "file:///C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - { - "input": "file://C|/", - "base": null, - "href": "file:///C:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/C:/", - "search": "", - "hash": "" - }, - "# file URLs without base URL by Rimas Misevičius", - { - "input": "file:", - "base": null, - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "file:?q=v", - "base": null, - "href": "file:///?q=v", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "?q=v", - "hash": "" - }, - { - "input": "file:#frag", - "base": null, - "href": "file:///#frag", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "#frag" - }, - "# file: drive letter cases from https://crbug.com/1078698", - { - "input": "file:///Y:", - "base": null, - "href": "file:///Y:", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/Y:", - "search": "", - "hash": "" - }, - { - "input": "file:///Y:/", - "base": null, - "href": "file:///Y:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/Y:/", - "search": "", - "hash": "" - }, - { - "input": "file:///./Y", - "base": null, - "href": "file:///Y", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/Y", - "search": "", - "hash": "" - }, - { - "input": "file:///./Y:", - "base": null, - "href": "file:///Y:", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/Y:", - "search": "", - "hash": "" - }, - { - "input": "\\\\\\.\\Y:", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - "# file: drive letter cases from https://crbug.com/1078698 but lowercased", - { - "input": "file:///y:", - "base": null, - "href": "file:///y:", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/y:", - "search": "", - "hash": "" - }, - { - "input": "file:///y:/", - "base": null, - "href": "file:///y:/", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/y:/", - "search": "", - "hash": "" - }, - { - "input": "file:///./y", - "base": null, - "href": "file:///y", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/y", - "search": "", - "hash": "" - }, - { - "input": "file:///./y:", - "base": null, - "href": "file:///y:", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/y:", - "search": "", - "hash": "" - }, - { - "input": "\\\\\\.\\y:", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - "# Additional file URL tests for (https://github.com/whatwg/url/issues/405)", - { - "input": "file://localhost//a//../..//foo", - "base": null, - "href": "file://///foo", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "///foo", - "search": "", - "hash": "" - }, - { - "input": "file://localhost////foo", - "base": null, - "href": "file://////foo", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "////foo", - "search": "", - "hash": "" - }, - { - "input": "file:////foo", - "base": null, - "href": "file:////foo", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//foo", - "search": "", - "hash": "" - }, - { - "input": "file:///one/two", - "base": "file:///", - "href": "file:///one/two", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/one/two", - "search": "", - "hash": "" - }, - { - "input": "file:////one/two", - "base": "file:///", - "href": "file:////one/two", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//one/two", - "search": "", - "hash": "" - }, - { - "input": "//one/two", - "base": "file:///", - "href": "file://one/two", - "protocol": "file:", - "username": "", - "password": "", - "host": "one", - "hostname": "one", - "port": "", - "pathname": "/two", - "search": "", - "hash": "" - }, - { - "input": "///one/two", - "base": "file:///", - "href": "file:///one/two", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/one/two", - "search": "", - "hash": "" - }, - { - "input": "////one/two", - "base": "file:///", - "href": "file:////one/two", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//one/two", - "search": "", - "hash": "" - }, - { - "input": "file:///.//", - "base": "file:////", - "href": "file:////", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - "File URL tests for https://github.com/whatwg/url/issues/549", - { - "input": "file:.//p", - "base": null, - "href": "file:////p", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//p", - "search": "", - "hash": "" - }, - { - "input": "file:/.//p", - "base": null, - "href": "file:////p", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//p", - "search": "", - "hash": "" - }, - "# IPv6 tests", - { - "input": "http://[1:0::]", - "base": "http://example.net/", - "href": "http://[1::]/", - "origin": "http://[1::]", - "protocol": "http:", - "username": "", - "password": "", - "host": "[1::]", - "hostname": "[1::]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://[0:1:2:3:4:5:6:7:8]", - "base": "http://example.net/", - "failure": true - }, - { - "input": "https://[0::0::0]", - "base": null, - "failure": true - }, - { - "input": "https://[0:.0]", - "base": null, - "failure": true - }, - { - "input": "https://[0:0:]", - "base": null, - "failure": true - }, - { - "input": "https://[0:1:2:3:4:5:6:7.0.0.0.1]", - "base": null, - "failure": true - }, - { - "input": "https://[0:1.00.0.0.0]", - "base": null, - "failure": true - }, - { - "input": "https://[0:1.290.0.0.0]", - "base": null, - "failure": true - }, - { - "input": "https://[0:1.23.23]", - "base": null, - "failure": true - }, - "# Empty host", - { - "input": "http://?", - "base": null, - "failure": true - }, - { - "input": "http://#", - "base": null, - "failure": true - }, - "Port overflow (2^32 + 81)", - { - "input": "http://f:4294967377/c", - "base": "http://example.org/", - "failure": true - }, - "Port overflow (2^64 + 81)", - { - "input": "http://f:18446744073709551697/c", - "base": "http://example.org/", - "failure": true - }, - "Port overflow (2^128 + 81)", - { - "input": "http://f:340282366920938463463374607431768211537/c", - "base": "http://example.org/", - "failure": true - }, - "# Non-special-URL path tests", - { - "input": "sc://ñ", - "base": null, - "href": "sc://%C3%B1", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%C3%B1", - "hostname": "%C3%B1", - "port": "", - "pathname": "", - "search": "", - "hash": "" - }, - { - "input": "sc://ñ?x", - "base": null, - "href": "sc://%C3%B1?x", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%C3%B1", - "hostname": "%C3%B1", - "port": "", - "pathname": "", - "search": "?x", - "hash": "" - }, - { - "input": "sc://ñ#x", - "base": null, - "href": "sc://%C3%B1#x", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%C3%B1", - "hostname": "%C3%B1", - "port": "", - "pathname": "", - "search": "", - "hash": "#x" - }, - { - "input": "#x", - "base": "sc://ñ", - "href": "sc://%C3%B1#x", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%C3%B1", - "hostname": "%C3%B1", - "port": "", - "pathname": "", - "search": "", - "hash": "#x" - }, - { - "input": "?x", - "base": "sc://ñ", - "href": "sc://%C3%B1?x", - "origin": "null", - "protocol": "sc:", - "username": "", - "password": "", - "host": "%C3%B1", - "hostname": "%C3%B1", - "port": "", - "pathname": "", - "search": "?x", - "hash": "" - }, - { - "input": "sc://?", - "base": null, - "href": "sc://?", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "", - "search": "", - "hash": "" - }, - { - "input": "sc://#", - "base": null, - "href": "sc://#", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "", - "search": "", - "hash": "" - }, - { - "input": "///", - "base": "sc://x/", - "href": "sc:///", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "////", - "base": "sc://x/", - "href": "sc:////", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - { - "input": "////x/", - "base": "sc://x/", - "href": "sc:////x/", - "protocol": "sc:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//x/", - "search": "", - "hash": "" - }, - { - "input": "tftp://foobar.com/someconfig;mode=netascii", - "base": null, - "href": "tftp://foobar.com/someconfig;mode=netascii", - "origin": "null", - "protocol": "tftp:", - "username": "", - "password": "", - "host": "foobar.com", - "hostname": "foobar.com", - "port": "", - "pathname": "/someconfig;mode=netascii", - "search": "", - "hash": "" - }, - { - "input": "telnet://user:pass@foobar.com:23/", - "base": null, - "href": "telnet://user:pass@foobar.com:23/", - "origin": "null", - "protocol": "telnet:", - "username": "user", - "password": "pass", - "host": "foobar.com:23", - "hostname": "foobar.com", - "port": "23", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "ut2004://10.10.10.10:7777/Index.ut2", - "base": null, - "href": "ut2004://10.10.10.10:7777/Index.ut2", - "origin": "null", - "protocol": "ut2004:", - "username": "", - "password": "", - "host": "10.10.10.10:7777", - "hostname": "10.10.10.10", - "port": "7777", - "pathname": "/Index.ut2", - "search": "", - "hash": "" - }, - { - "input": "redis://foo:bar@somehost:6379/0?baz=bam&qux=baz", - "base": null, - "href": "redis://foo:bar@somehost:6379/0?baz=bam&qux=baz", - "origin": "null", - "protocol": "redis:", - "username": "foo", - "password": "bar", - "host": "somehost:6379", - "hostname": "somehost", - "port": "6379", - "pathname": "/0", - "search": "?baz=bam&qux=baz", - "hash": "" - }, - { - "input": "rsync://foo@host:911/sup", - "base": null, - "href": "rsync://foo@host:911/sup", - "origin": "null", - "protocol": "rsync:", - "username": "foo", - "password": "", - "host": "host:911", - "hostname": "host", - "port": "911", - "pathname": "/sup", - "search": "", - "hash": "" - }, - { - "input": "git://github.com/foo/bar.git", - "base": null, - "href": "git://github.com/foo/bar.git", - "origin": "null", - "protocol": "git:", - "username": "", - "password": "", - "host": "github.com", - "hostname": "github.com", - "port": "", - "pathname": "/foo/bar.git", - "search": "", - "hash": "" - }, - { - "input": "irc://myserver.com:6999/channel?passwd", - "base": null, - "href": "irc://myserver.com:6999/channel?passwd", - "origin": "null", - "protocol": "irc:", - "username": "", - "password": "", - "host": "myserver.com:6999", - "hostname": "myserver.com", - "port": "6999", - "pathname": "/channel", - "search": "?passwd", - "hash": "" - }, - { - "input": "dns://fw.example.org:9999/foo.bar.org?type=TXT", - "base": null, - "href": "dns://fw.example.org:9999/foo.bar.org?type=TXT", - "origin": "null", - "protocol": "dns:", - "username": "", - "password": "", - "host": "fw.example.org:9999", - "hostname": "fw.example.org", - "port": "9999", - "pathname": "/foo.bar.org", - "search": "?type=TXT", - "hash": "" - }, - { - "input": "ldap://localhost:389/ou=People,o=JNDITutorial", - "base": null, - "href": "ldap://localhost:389/ou=People,o=JNDITutorial", - "origin": "null", - "protocol": "ldap:", - "username": "", - "password": "", - "host": "localhost:389", - "hostname": "localhost", - "port": "389", - "pathname": "/ou=People,o=JNDITutorial", - "search": "", - "hash": "" - }, - { - "input": "git+https://github.com/foo/bar", - "base": null, - "href": "git+https://github.com/foo/bar", - "origin": "null", - "protocol": "git+https:", - "username": "", - "password": "", - "host": "github.com", - "hostname": "github.com", - "port": "", - "pathname": "/foo/bar", - "search": "", - "hash": "" - }, - { - "input": "urn:ietf:rfc:2648", - "base": null, - "href": "urn:ietf:rfc:2648", - "origin": "null", - "protocol": "urn:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "ietf:rfc:2648", - "search": "", - "hash": "" - }, - { - "input": "tag:joe@example.org,2001:foo/bar", - "base": null, - "href": "tag:joe@example.org,2001:foo/bar", - "origin": "null", - "protocol": "tag:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "joe@example.org,2001:foo/bar", - "search": "", - "hash": "" - }, - "Serialize /. in path", - { - "input": "non-spec:/.//", - "base": null, - "href": "non-spec:/.//", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - { - "input": "non-spec:/..//", - "base": null, - "href": "non-spec:/.//", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - { - "input": "non-spec:/a/..//", - "base": null, - "href": "non-spec:/.//", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//", - "search": "", - "hash": "" - }, - { - "input": "non-spec:/.//path", - "base": null, - "href": "non-spec:/.//path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//path", - "search": "", - "hash": "" - }, - { - "input": "non-spec:/..//path", - "base": null, - "href": "non-spec:/.//path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//path", - "search": "", - "hash": "" - }, - { - "input": "non-spec:/a/..//path", - "base": null, - "href": "non-spec:/.//path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//path", - "search": "", - "hash": "" - }, - { - "input": "/.//path", - "base": "non-spec:/p", - "href": "non-spec:/.//path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//path", - "search": "", - "hash": "" - }, - { - "input": "/..//path", - "base": "non-spec:/p", - "href": "non-spec:/.//path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//path", - "search": "", - "hash": "" - }, - { - "input": "..//path", - "base": "non-spec:/p", - "href": "non-spec:/.//path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//path", - "search": "", - "hash": "" - }, - { - "input": "a/..//path", - "base": "non-spec:/p", - "href": "non-spec:/.//path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//path", - "search": "", - "hash": "" - }, - { - "input": "", - "base": "non-spec:/..//p", - "href": "non-spec:/.//p", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//p", - "search": "", - "hash": "" - }, - { - "input": "path", - "base": "non-spec:/..//p", - "href": "non-spec:/.//path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "//path", - "search": "", - "hash": "" - }, - "Do not serialize /. in path", - { - "input": "../path", - "base": "non-spec:/.//p", - "href": "non-spec:/path", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/path", - "search": "", - "hash": "" - }, - "# percent encoded hosts in non-special-URLs", - { - "input": "non-special://%E2%80%A0/", - "base": null, - "href": "non-special://%E2%80%A0/", - "protocol": "non-special:", - "username": "", - "password": "", - "host": "%E2%80%A0", - "hostname": "%E2%80%A0", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "non-special://H%4fSt/path", - "base": null, - "href": "non-special://H%4fSt/path", - "protocol": "non-special:", - "username": "", - "password": "", - "host": "H%4fSt", - "hostname": "H%4fSt", - "port": "", - "pathname": "/path", - "search": "", - "hash": "" - }, - "# IPv6 in non-special-URLs", - { - "input": "non-special://[1:2:0:0:5:0:0:0]/", - "base": null, - "href": "non-special://[1:2:0:0:5::]/", - "protocol": "non-special:", - "username": "", - "password": "", - "host": "[1:2:0:0:5::]", - "hostname": "[1:2:0:0:5::]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "non-special://[1:2:0:0:0:0:0:3]/", - "base": null, - "href": "non-special://[1:2::3]/", - "protocol": "non-special:", - "username": "", - "password": "", - "host": "[1:2::3]", - "hostname": "[1:2::3]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "non-special://[1:2::3]:80/", - "base": null, - "href": "non-special://[1:2::3]:80/", - "protocol": "non-special:", - "username": "", - "password": "", - "host": "[1:2::3]:80", - "hostname": "[1:2::3]", - "port": "80", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "non-special://[:80/", - "base": null, - "failure": true - }, - { - "input": "blob:https://example.com:443/", - "base": null, - "href": "blob:https://example.com:443/", - "origin": "https://example.com", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "https://example.com:443/", - "search": "", - "hash": "" - }, - { - "input": "blob:http://example.org:88/", - "base": null, - "href": "blob:http://example.org:88/", - "origin": "http://example.org:88", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "http://example.org:88/", - "search": "", - "hash": "" - }, - { - "input": "blob:d3958f5c-0777-0845-9dcf-2cb28783acaf", - "base": null, - "href": "blob:d3958f5c-0777-0845-9dcf-2cb28783acaf", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "d3958f5c-0777-0845-9dcf-2cb28783acaf", - "search": "", - "hash": "" - }, - { - "input": "blob:", - "base": null, - "href": "blob:", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "", - "search": "", - "hash": "" - }, - "blob: in blob:", - { - "input": "blob:blob:", - "base": null, - "href": "blob:blob:", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "blob:", - "search": "", - "hash": "" - }, - { - "input": "blob:blob:https://example.org/", - "base": null, - "href": "blob:blob:https://example.org/", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "blob:https://example.org/", - "search": "", - "hash": "" - }, - "Non-http(s): in blob:", - { - "input": "blob:about:blank", - "base": null, - "href": "blob:about:blank", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "about:blank", - "search": "", - "hash": "" - }, - { - "input": "blob:file://host/path", - "base": null, - "href": "blob:file://host/path", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "file://host/path", - "search": "", - "hash": "" - }, - { - "input": "blob:ftp://host/path", - "base": null, - "href": "blob:ftp://host/path", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "ftp://host/path", - "search": "", - "hash": "" - }, - { - "input": "blob:ws://example.org/", - "base": null, - "href": "blob:ws://example.org/", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "ws://example.org/", - "search": "", - "hash": "" - }, - { - "input": "blob:wss://example.org/", - "base": null, - "href": "blob:wss://example.org/", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "wss://example.org/", - "search": "", - "hash": "" - }, - "Percent-encoded http: in blob:", - { - "input": "blob:http%3a//example.org/", - "base": null, - "href": "blob:http%3a//example.org/", - "origin": "null", - "protocol": "blob:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "http%3a//example.org/", - "search": "", - "hash": "" - }, - "Invalid IPv4 radix digits", - { - "input": "http://0x7f.0.0.0x7g", - "base": null, - "href": "http://0x7f.0.0.0x7g/", - "protocol": "http:", - "username": "", - "password": "", - "host": "0x7f.0.0.0x7g", - "hostname": "0x7f.0.0.0x7g", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://0X7F.0.0.0X7G", - "base": null, - "href": "http://0x7f.0.0.0x7g/", - "protocol": "http:", - "username": "", - "password": "", - "host": "0x7f.0.0.0x7g", - "hostname": "0x7f.0.0.0x7g", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "Invalid IPv4 portion of IPv6 address", - { - "input": "http://[::127.0.0.0.1]", - "base": null, - "failure": true - }, - "Uncompressed IPv6 addresses with 0", - { - "input": "http://[0:1:0:1:0:1:0:1]", - "base": null, - "href": "http://[0:1:0:1:0:1:0:1]/", - "protocol": "http:", - "username": "", - "password": "", - "host": "[0:1:0:1:0:1:0:1]", - "hostname": "[0:1:0:1:0:1:0:1]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "http://[1:0:1:0:1:0:1:0]", - "base": null, - "href": "http://[1:0:1:0:1:0:1:0]/", - "protocol": "http:", - "username": "", - "password": "", - "host": "[1:0:1:0:1:0:1:0]", - "hostname": "[1:0:1:0:1:0:1:0]", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - "Percent-encoded query and fragment", - { - "input": "http://example.org/test?\u0022", - "base": null, - "href": "http://example.org/test?%22", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?%22", - "hash": "" - }, - { - "input": "http://example.org/test?\u0023", - "base": null, - "href": "http://example.org/test?#", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "http://example.org/test?\u003C", - "base": null, - "href": "http://example.org/test?%3C", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?%3C", - "hash": "" - }, - { - "input": "http://example.org/test?\u003E", - "base": null, - "href": "http://example.org/test?%3E", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?%3E", - "hash": "" - }, - { - "input": "http://example.org/test?\u2323", - "base": null, - "href": "http://example.org/test?%E2%8C%A3", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?%E2%8C%A3", - "hash": "" - }, - { - "input": "http://example.org/test?%23%23", - "base": null, - "href": "http://example.org/test?%23%23", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?%23%23", - "hash": "" - }, - { - "input": "http://example.org/test?%GH", - "base": null, - "href": "http://example.org/test?%GH", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?%GH", - "hash": "" - }, - { - "input": "http://example.org/test?a#%EF", - "base": null, - "href": "http://example.org/test?a#%EF", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?a", - "hash": "#%EF" - }, - { - "input": "http://example.org/test?a#%GH", - "base": null, - "href": "http://example.org/test?a#%GH", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?a", - "hash": "#%GH" - }, - "URLs that require a non-about:blank base. (Also serve as invalid base tests.)", - { - "input": "a", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "a/", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "a//", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - "Bases that don't fail to parse but fail to be bases", - { - "input": "test-a-colon.html", - "base": "a:", - "failure": true - }, - { - "input": "test-a-colon-b.html", - "base": "a:b", - "failure": true - }, - "Other base URL tests, that must succeed", - { - "input": "test-a-colon-slash.html", - "base": "a:/", - "href": "a:/test-a-colon-slash.html", - "protocol": "a:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test-a-colon-slash.html", - "search": "", - "hash": "" - }, - { - "input": "test-a-colon-slash-slash.html", - "base": "a://", - "href": "a:///test-a-colon-slash-slash.html", - "protocol": "a:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test-a-colon-slash-slash.html", - "search": "", - "hash": "" - }, - { - "input": "test-a-colon-slash-b.html", - "base": "a:/b", - "href": "a:/test-a-colon-slash-b.html", - "protocol": "a:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test-a-colon-slash-b.html", - "search": "", - "hash": "" - }, - { - "input": "test-a-colon-slash-slash-b.html", - "base": "a://b", - "href": "a://b/test-a-colon-slash-slash-b.html", - "protocol": "a:", - "username": "", - "password": "", - "host": "b", - "hostname": "b", - "port": "", - "pathname": "/test-a-colon-slash-slash-b.html", - "search": "", - "hash": "" - }, - "Null code point in fragment", - { - "input": "http://example.org/test?a#b\u0000c", - "base": null, - "href": "http://example.org/test?a#b%00c", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?a", - "hash": "#b%00c" - }, - { - "input": "non-spec://example.org/test?a#b\u0000c", - "base": null, - "href": "non-spec://example.org/test?a#b%00c", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/test", - "search": "?a", - "hash": "#b%00c" - }, - { - "input": "non-spec:/test?a#b\u0000c", - "base": null, - "href": "non-spec:/test?a#b%00c", - "protocol": "non-spec:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "?a", - "hash": "#b%00c" - }, - "First scheme char - not allowed: https://github.com/whatwg/url/issues/464", - { - "input": "10.0.0.7:8080/foo.html", - "base": "file:///some/dir/bar.html", - "href": "file:///some/dir/10.0.0.7:8080/foo.html", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/some/dir/10.0.0.7:8080/foo.html", - "search": "", - "hash": "" - }, - "Subsequent scheme chars - not allowed", - { - "input": "a!@$*=/foo.html", - "base": "file:///some/dir/bar.html", - "href": "file:///some/dir/a!@$*=/foo.html", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/some/dir/a!@$*=/foo.html", - "search": "", - "hash": "" - }, - "First and subsequent scheme chars - allowed", - { - "input": "a1234567890-+.:foo/bar", - "base": "http://example.com/dir/file", - "href": "a1234567890-+.:foo/bar", - "protocol": "a1234567890-+.:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "foo/bar", - "search": "", - "hash": "" - }, - "IDNA ignored code points in file URLs hosts", - { - "input": "file://a\u00ADb/p", - "base": null, - "href": "file://ab/p", - "protocol": "file:", - "username": "", - "password": "", - "host": "ab", - "hostname": "ab", - "port": "", - "pathname": "/p", - "search": "", - "hash": "" - }, - { - "input": "file://a%C2%ADb/p", - "base": null, - "href": "file://ab/p", - "protocol": "file:", - "username": "", - "password": "", - "host": "ab", - "hostname": "ab", - "port": "", - "pathname": "/p", - "search": "", - "hash": "" - }, - "IDNA hostnames which get mapped to 'localhost'", - { - "input": "file://loC𝐀𝐋𝐇𝐨𝐬𝐭/usr/bin", - "base": null, - "href": "file:///usr/bin", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/usr/bin", - "search": "", - "hash": "" - }, - "Empty host after the domain to ASCII", - { - "input": "file://\u00ad/p", - "base": null, - "failure": true - }, - { - "input": "file://%C2%AD/p", - "base": null, - "failure": true - }, - { - "input": "file://xn--/p", - "base": null, - "failure": true - }, - "https://bugzilla.mozilla.org/show_bug.cgi?id=1647058", - { - "input": "#link", - "base": "https://example.org/##link", - "href": "https://example.org/#link", - "protocol": "https:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/", - "search": "", - "hash": "#link" - }, - "UTF-8 percent-encode of C0 control percent-encode set and supersets", - { - "input": "non-special:cannot-be-a-base-url-\u0000\u0001\u001F\u001E\u007E\u007F\u0080", - "base": null, - "hash": "", - "host": "", - "hostname": "", - "href": "non-special:cannot-be-a-base-url-%00%01%1F%1E~%7F%C2%80", - "origin": "null", - "password": "", - "pathname": "cannot-be-a-base-url-%00%01%1F%1E~%7F%C2%80", - "port": "", - "protocol": "non-special:", - "search": "", - "username": "" - }, - { - "input": "https://www.example.com/path{\u007Fpath.html?query'\u007F=query#fragment<\u007Ffragment", - "base": null, - "hash": "#fragment%3C%7Ffragment", - "host": "www.example.com", - "hostname": "www.example.com", - "href": "https://www.example.com/path%7B%7Fpath.html?query%27%7F=query#fragment%3C%7Ffragment", - "origin": "https://www.example.com", - "password": "", - "pathname": "/path%7B%7Fpath.html", - "port": "", - "protocol": "https:", - "search": "?query%27%7F=query", - "username": "" - }, - { - "input": "https://user:pass[\u007F@foo/bar", - "base": "http://example.org", - "hash": "", - "host": "foo", - "hostname": "foo", - "href": "https://user:pass%5B%7F@foo/bar", - "origin": "https://foo", - "password": "pass%5B%7F", - "pathname": "/bar", - "port": "", - "protocol": "https:", - "search": "", - "username": "user" - }, - "Tests for the distinct percent-encode sets", - { - "input": "foo:// !\"$%&'()*+,-.;<=>@[\\]^_`{|}~@host/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href": "foo://%20!%22$%&'()*+,-.%3B%3C%3D%3E%40%5B%5C%5D%5E_%60%7B%7C%7D~@host/", - "origin": "null", - "password": "", - "pathname": "/", - "port":"", - "protocol": "foo:", - "search": "", - "username": "%20!%22$%&'()*+,-.%3B%3C%3D%3E%40%5B%5C%5D%5E_%60%7B%7C%7D~" - }, - { - "input": "wss:// !\"$%&'()*+,-.;<=>@[]^_`{|}~@host/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href": "wss://%20!%22$%&'()*+,-.%3B%3C%3D%3E%40%5B%5D%5E_%60%7B%7C%7D~@host/", - "origin": "wss://host", - "password": "", - "pathname": "/", - "port":"", - "protocol": "wss:", - "search": "", - "username": "%20!%22$%&'()*+,-.%3B%3C%3D%3E%40%5B%5D%5E_%60%7B%7C%7D~" - }, - { - "input": "foo://joe: !\"$%&'()*+,-.:;<=>@[\\]^_`{|}~@host/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href": "foo://joe:%20!%22$%&'()*+,-.%3A%3B%3C%3D%3E%40%5B%5C%5D%5E_%60%7B%7C%7D~@host/", - "origin": "null", - "password": "%20!%22$%&'()*+,-.%3A%3B%3C%3D%3E%40%5B%5C%5D%5E_%60%7B%7C%7D~", - "pathname": "/", - "port":"", - "protocol": "foo:", - "search": "", - "username": "joe" - }, - { - "input": "wss://joe: !\"$%&'()*+,-.:;<=>@[]^_`{|}~@host/", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href": "wss://joe:%20!%22$%&'()*+,-.%3A%3B%3C%3D%3E%40%5B%5D%5E_%60%7B%7C%7D~@host/", - "origin": "wss://host", - "password": "%20!%22$%&'()*+,-.%3A%3B%3C%3D%3E%40%5B%5D%5E_%60%7B%7C%7D~", - "pathname": "/", - "port":"", - "protocol": "wss:", - "search": "", - "username": "joe" - }, - { - "input": "foo://!\"$%&'()*+,-.;=_`{}~/", - "base": null, - "hash": "", - "host": "!\"$%&'()*+,-.;=_`{}~", - "hostname": "!\"$%&'()*+,-.;=_`{}~", - "href":"foo://!\"$%&'()*+,-.;=_`{}~/", - "origin": "null", - "password": "", - "pathname": "/", - "port":"", - "protocol": "foo:", - "search": "", - "username": "" - }, - { - "input": "wss://!\"$&'()*+,-.;=_`{}~/", - "base": null, - "hash": "", - "host": "!\"$&'()*+,-.;=_`{}~", - "hostname": "!\"$&'()*+,-.;=_`{}~", - "href":"wss://!\"$&'()*+,-.;=_`{}~/", - "origin": "wss://!\"$&'()*+,-.;=_`{}~", - "password": "", - "pathname": "/", - "port":"", - "protocol": "wss:", - "search": "", - "username": "" - }, - { - "input": "foo://host/ !\"$%&'()*+,-./:;<=>@[\\]^_`{|}~", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href": "foo://host/%20!%22$%&'()*+,-./:;%3C=%3E@[\\]^_%60%7B|%7D~", - "origin": "null", - "password": "", - "pathname": "/%20!%22$%&'()*+,-./:;%3C=%3E@[\\]^_%60%7B|%7D~", - "port":"", - "protocol": "foo:", - "search": "", - "username": "" - }, - { - "input": "wss://host/ !\"$%&'()*+,-./:;<=>@[\\]^_`{|}~", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href": "wss://host/%20!%22$%&'()*+,-./:;%3C=%3E@[/]^_%60%7B|%7D~", - "origin": "wss://host", - "password": "", - "pathname": "/%20!%22$%&'()*+,-./:;%3C=%3E@[/]^_%60%7B|%7D~", - "port":"", - "protocol": "wss:", - "search": "", - "username": "" - }, - { - "input": "foo://host/dir/? !\"$%&'()*+,-./:;<=>?@[\\]^_`{|}~", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href": "foo://host/dir/?%20!%22$%&'()*+,-./:;%3C=%3E?@[\\]^_`{|}~", - "origin": "null", - "password": "", - "pathname": "/dir/", - "port":"", - "protocol": "foo:", - "search": "?%20!%22$%&'()*+,-./:;%3C=%3E?@[\\]^_`{|}~", - "username": "" - }, - { - "input": "wss://host/dir/? !\"$%&'()*+,-./:;<=>?@[\\]^_`{|}~", - "base": null, - "hash": "", - "host": "host", - "hostname": "host", - "href": "wss://host/dir/?%20!%22$%&%27()*+,-./:;%3C=%3E?@[\\]^_`{|}~", - "origin": "wss://host", - "password": "", - "pathname": "/dir/", - "port":"", - "protocol": "wss:", - "search": "?%20!%22$%&%27()*+,-./:;%3C=%3E?@[\\]^_`{|}~", - "username": "" - }, - { - "input": "foo://host/dir/# !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~", - "base": null, - "hash": "#%20!%22#$%&'()*+,-./:;%3C=%3E?@[\\]^_%60{|}~", - "host": "host", - "hostname": "host", - "href": "foo://host/dir/#%20!%22#$%&'()*+,-./:;%3C=%3E?@[\\]^_%60{|}~", - "origin": "null", - "password": "", - "pathname": "/dir/", - "port":"", - "protocol": "foo:", - "search": "", - "username": "" - }, - { - "input": "wss://host/dir/# !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~", - "base": null, - "hash": "#%20!%22#$%&'()*+,-./:;%3C=%3E?@[\\]^_%60{|}~", - "host": "host", - "hostname": "host", - "href": "wss://host/dir/#%20!%22#$%&'()*+,-./:;%3C=%3E?@[\\]^_%60{|}~", - "origin": "wss://host", - "password": "", - "pathname": "/dir/", - "port":"", - "protocol": "wss:", - "search": "", - "username": "" - }, - "Ensure that input schemes are not ignored when resolving non-special URLs", - { - "input": "abc:rootless", - "base": "abc://host/path", - "hash": "", - "host": "", - "hostname": "", - "href":"abc:rootless", - "password": "", - "pathname": "rootless", - "port":"", - "protocol": "abc:", - "search": "", - "username": "" - }, - { - "input": "abc:rootless", - "base": "abc:/path", - "hash": "", - "host": "", - "hostname": "", - "href":"abc:rootless", - "password": "", - "pathname": "rootless", - "port":"", - "protocol": "abc:", - "search": "", - "username": "" - }, - { - "input": "abc:rootless", - "base": "abc:path", - "hash": "", - "host": "", - "hostname": "", - "href":"abc:rootless", - "password": "", - "pathname": "rootless", - "port":"", - "protocol": "abc:", - "search": "", - "username": "" - }, - { - "input": "abc:/rooted", - "base": "abc://host/path", - "hash": "", - "host": "", - "hostname": "", - "href":"abc:/rooted", - "password": "", - "pathname": "/rooted", - "port":"", - "protocol": "abc:", - "search": "", - "username": "" - }, - "Empty query and fragment with blank should throw an error", - { - "input": "#", - "base": null, - "failure": true, - "relativeTo": "any-base" - }, - { - "input": "?", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - "Last component looks like a number, but not valid IPv4", - { - "input": "http://1.2.3.4.5", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://1.2.3.4.5.", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://0..0x300/", - "base": null, - "failure": true - }, - { - "input": "http://0..0x300./", - "base": null, - "failure": true - }, - { - "input": "http://256.256.256.256.256", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://256.256.256.256.256.", - "base": "http://other.com/", - "failure": true - }, - { - "input": "http://1.2.3.08", - "base": null, - "failure": true - }, - { - "input": "http://1.2.3.08.", - "base": null, - "failure": true - }, - { - "input": "http://1.2.3.09", - "base": null, - "failure": true - }, - { - "input": "http://09.2.3.4", - "base": null, - "failure": true - }, - { - "input": "http://09.2.3.4.", - "base": null, - "failure": true - }, - { - "input": "http://01.2.3.4.5", - "base": null, - "failure": true - }, - { - "input": "http://01.2.3.4.5.", - "base": null, - "failure": true - }, - { - "input": "http://0x100.2.3.4", - "base": null, - "failure": true - }, - { - "input": "http://0x100.2.3.4.", - "base": null, - "failure": true - }, - { - "input": "http://0x1.2.3.4.5", - "base": null, - "failure": true - }, - { - "input": "http://0x1.2.3.4.5.", - "base": null, - "failure": true - }, - { - "input": "http://foo.1.2.3.4", - "base": null, - "failure": true - }, - { - "input": "http://foo.1.2.3.4.", - "base": null, - "failure": true - }, - { - "input": "http://foo.2.3.4", - "base": null, - "failure": true - }, - { - "input": "http://foo.2.3.4.", - "base": null, - "failure": true - }, - { - "input": "http://foo.09", - "base": null, - "failure": true - }, - { - "input": "http://foo.09.", - "base": null, - "failure": true - }, - { - "input": "http://foo.0x4", - "base": null, - "failure": true - }, - { - "input": "http://foo.0x4.", - "base": null, - "failure": true - }, - { - "input": "http://foo.09..", - "base": null, - "hash": "", - "host": "foo.09..", - "hostname": "foo.09..", - "href":"http://foo.09../", - "password": "", - "pathname": "/", - "port":"", - "protocol": "http:", - "search": "", - "username": "" - }, - { - "input": "http://0999999999999999999/", - "base": null, - "failure": true - }, - { - "input": "http://foo.0x", - "base": null, - "failure": true - }, - { - "input": "http://foo.0XFfFfFfFfFfFfFfFfFfAcE123", - "base": null, - "failure": true - }, - { - "input": "http://💩.123/", - "base": null, - "failure": true - }, - "U+0000 and U+FFFF in various places", - { - "input": "https://\u0000y", - "base": null, - "failure": true - }, - { - "input": "https://x/\u0000y", - "base": null, - "hash": "", - "host": "x", - "hostname": "x", - "href": "https://x/%00y", - "password": "", - "pathname": "/%00y", - "port": "", - "protocol": "https:", - "search": "", - "username": "" - }, - { - "input": "https://x/?\u0000y", - "base": null, - "hash": "", - "host": "x", - "hostname": "x", - "href": "https://x/?%00y", - "password": "", - "pathname": "/", - "port": "", - "protocol": "https:", - "search": "?%00y", - "username": "" - }, - { - "input": "https://x/?#\u0000y", - "base": null, - "hash": "#%00y", - "host": "x", - "hostname": "x", - "href": "https://x/?#%00y", - "password": "", - "pathname": "/", - "port": "", - "protocol": "https:", - "search": "", - "username": "" - }, - { - "input": "https://\uFFFFy", - "base": null, - "failure": true - }, - { - "input": "https://x/\uFFFFy", - "base": null, - "hash": "", - "host": "x", - "hostname": "x", - "href": "https://x/%EF%BF%BFy", - "password": "", - "pathname": "/%EF%BF%BFy", - "port": "", - "protocol": "https:", - "search": "", - "username": "" - }, - { - "input": "https://x/?\uFFFFy", - "base": null, - "hash": "", - "host": "x", - "hostname": "x", - "href": "https://x/?%EF%BF%BFy", - "password": "", - "pathname": "/", - "port": "", - "protocol": "https:", - "search": "?%EF%BF%BFy", - "username": "" - }, - { - "input": "https://x/?#\uFFFFy", - "base": null, - "hash": "#%EF%BF%BFy", - "host": "x", - "hostname": "x", - "href": "https://x/?#%EF%BF%BFy", - "password": "", - "pathname": "/", - "port": "", - "protocol": "https:", - "search": "", - "username": "" - }, - { - "input": "non-special:\u0000y", - "base": null, - "hash": "", - "host": "", - "hostname": "", - "href": "non-special:%00y", - "password": "", - "pathname": "%00y", - "port": "", - "protocol": "non-special:", - "search": "", - "username": "" - }, - { - "input": "non-special:x/\u0000y", - "base": null, - "hash": "", - "host": "", - "hostname": "", - "href": "non-special:x/%00y", - "password": "", - "pathname": "x/%00y", - "port": "", - "protocol": "non-special:", - "search": "", - "username": "" - }, - { - "input": "non-special:x/?\u0000y", - "base": null, - "hash": "", - "host": "", - "hostname": "", - "href": "non-special:x/?%00y", - "password": "", - "pathname": "x/", - "port": "", - "protocol": "non-special:", - "search": "?%00y", - "username": "" - }, - { - "input": "non-special:x/?#\u0000y", - "base": null, - "hash": "#%00y", - "host": "", - "hostname": "", - "href": "non-special:x/?#%00y", - "password": "", - "pathname": "x/", - "port": "", - "protocol": "non-special:", - "search": "", - "username": "" - }, - { - "input": "non-special:\uFFFFy", - "base": null, - "hash": "", - "host": "", - "hostname": "", - "href": "non-special:%EF%BF%BFy", - "password": "", - "pathname": "%EF%BF%BFy", - "port": "", - "protocol": "non-special:", - "search": "", - "username": "" - }, - { - "input": "non-special:x/\uFFFFy", - "base": null, - "hash": "", - "host": "", - "hostname": "", - "href": "non-special:x/%EF%BF%BFy", - "password": "", - "pathname": "x/%EF%BF%BFy", - "port": "", - "protocol": "non-special:", - "search": "", - "username": "" - }, - { - "input": "non-special:x/?\uFFFFy", - "base": null, - "hash": "", - "host": "", - "hostname": "", - "href": "non-special:x/?%EF%BF%BFy", - "password": "", - "pathname": "x/", - "port": "", - "protocol": "non-special:", - "search": "?%EF%BF%BFy", - "username": "" - }, - { - "input": "non-special:x/?#\uFFFFy", - "base": null, - "hash": "#%EF%BF%BFy", - "host": "", - "hostname": "", - "href": "non-special:x/?#%EF%BF%BFy", - "password": "", - "pathname": "x/", - "port": "", - "protocol": "non-special:", - "search": "", - "username": "" - }, - { - "input": "", - "base": null, - "failure": true, - "relativeTo": "non-opaque-path-base" - }, - { - "input": "https://example.com/\"quoted\"", - "base": null, - "hash": "", - "host": "example.com", - "hostname": "example.com", - "href": "https://example.com/%22quoted%22", - "origin": "https://example.com", - "password": "", - "pathname": "/%22quoted%22", - "port": "", - "protocol": "https:", - "search": "", - "username": "" - }, - { - "input": "https://a%C2%ADb/", - "base": null, - "hash": "", - "host": "ab", - "hostname": "ab", - "href": "https://ab/", - "origin": "https://ab", - "password": "", - "pathname": "/", - "port": "", - "protocol": "https:", - "search": "", - "username": "" - }, - { - "comment": "Empty host after domain to ASCII", - "input": "https://\u00AD/", - "base": null, - "failure": true - }, - { - "input": "https://%C2%AD/", - "base": null, - "failure": true - }, - { - "input": "https://xn--/", - "base": null, - "failure": true - }, - "Non-special schemes that some implementations might incorrectly treat as special", - { - "input": "data://example.com:8080/pathname?search#hash", - "base": null, - "href": "data://example.com:8080/pathname?search#hash", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "example.com:8080", - "hostname": "example.com", - "port": "8080", - "pathname": "/pathname", - "search": "?search", - "hash": "#hash" - }, - { - "input": "data:///test", - "base": null, - "href": "data:///test", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "data://test/a/../b", - "base": null, - "href": "data://test/b", - "origin": "null", - "protocol": "data:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/b", - "search": "", - "hash": "" - }, - { - "input": "data://:443", - "base": null, - "failure": true - }, - { - "input": "data://test:test", - "base": null, - "failure": true - }, - { - "input": "data://[:1]", - "base": null, - "failure": true - }, - { - "input": "javascript://example.com:8080/pathname?search#hash", - "base": null, - "href": "javascript://example.com:8080/pathname?search#hash", - "origin": "null", - "protocol": "javascript:", - "username": "", - "password": "", - "host": "example.com:8080", - "hostname": "example.com", - "port": "8080", - "pathname": "/pathname", - "search": "?search", - "hash": "#hash" - }, - { - "input": "javascript:///test", - "base": null, - "href": "javascript:///test", - "origin": "null", - "protocol": "javascript:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "javascript://test/a/../b", - "base": null, - "href": "javascript://test/b", - "origin": "null", - "protocol": "javascript:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/b", - "search": "", - "hash": "" - }, - { - "input": "javascript://:443", - "base": null, - "failure": true - }, - { - "input": "javascript://test:test", - "base": null, - "failure": true - }, - { - "input": "javascript://[:1]", - "base": null, - "failure": true - }, - { - "input": "mailto://example.com:8080/pathname?search#hash", - "base": null, - "href": "mailto://example.com:8080/pathname?search#hash", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "example.com:8080", - "hostname": "example.com", - "port": "8080", - "pathname": "/pathname", - "search": "?search", - "hash": "#hash" - }, - { - "input": "mailto:///test", - "base": null, - "href": "mailto:///test", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "mailto://test/a/../b", - "base": null, - "href": "mailto://test/b", - "origin": "null", - "protocol": "mailto:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/b", - "search": "", - "hash": "" - }, - { - "input": "mailto://:443", - "base": null, - "failure": true - }, - { - "input": "mailto://test:test", - "base": null, - "failure": true - }, - { - "input": "mailto://[:1]", - "base": null, - "failure": true - }, - { - "input": "intent://example.com:8080/pathname?search#hash", - "base": null, - "href": "intent://example.com:8080/pathname?search#hash", - "origin": "null", - "protocol": "intent:", - "username": "", - "password": "", - "host": "example.com:8080", - "hostname": "example.com", - "port": "8080", - "pathname": "/pathname", - "search": "?search", - "hash": "#hash" - }, - { - "input": "intent:///test", - "base": null, - "href": "intent:///test", - "origin": "null", - "protocol": "intent:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "intent://test/a/../b", - "base": null, - "href": "intent://test/b", - "origin": "null", - "protocol": "intent:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/b", - "search": "", - "hash": "" - }, - { - "input": "intent://:443", - "base": null, - "failure": true - }, - { - "input": "intent://test:test", - "base": null, - "failure": true - }, - { - "input": "intent://[:1]", - "base": null, - "failure": true - }, - { - "input": "urn://example.com:8080/pathname?search#hash", - "base": null, - "href": "urn://example.com:8080/pathname?search#hash", - "origin": "null", - "protocol": "urn:", - "username": "", - "password": "", - "host": "example.com:8080", - "hostname": "example.com", - "port": "8080", - "pathname": "/pathname", - "search": "?search", - "hash": "#hash" - }, - { - "input": "urn:///test", - "base": null, - "href": "urn:///test", - "origin": "null", - "protocol": "urn:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "urn://test/a/../b", - "base": null, - "href": "urn://test/b", - "origin": "null", - "protocol": "urn:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/b", - "search": "", - "hash": "" - }, - { - "input": "urn://:443", - "base": null, - "failure": true - }, - { - "input": "urn://test:test", - "base": null, - "failure": true - }, - { - "input": "urn://[:1]", - "base": null, - "failure": true - }, - { - "input": "turn://example.com:8080/pathname?search#hash", - "base": null, - "href": "turn://example.com:8080/pathname?search#hash", - "origin": "null", - "protocol": "turn:", - "username": "", - "password": "", - "host": "example.com:8080", - "hostname": "example.com", - "port": "8080", - "pathname": "/pathname", - "search": "?search", - "hash": "#hash" - }, - { - "input": "turn:///test", - "base": null, - "href": "turn:///test", - "origin": "null", - "protocol": "turn:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "turn://test/a/../b", - "base": null, - "href": "turn://test/b", - "origin": "null", - "protocol": "turn:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/b", - "search": "", - "hash": "" - }, - { - "input": "turn://:443", - "base": null, - "failure": true - }, - { - "input": "turn://test:test", - "base": null, - "failure": true - }, - { - "input": "turn://[:1]", - "base": null, - "failure": true - }, - { - "input": "stun://example.com:8080/pathname?search#hash", - "base": null, - "href": "stun://example.com:8080/pathname?search#hash", - "origin": "null", - "protocol": "stun:", - "username": "", - "password": "", - "host": "example.com:8080", - "hostname": "example.com", - "port": "8080", - "pathname": "/pathname", - "search": "?search", - "hash": "#hash" - }, - { - "input": "stun:///test", - "base": null, - "href": "stun:///test", - "origin": "null", - "protocol": "stun:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/test", - "search": "", - "hash": "" - }, - { - "input": "stun://test/a/../b", - "base": null, - "href": "stun://test/b", - "origin": "null", - "protocol": "stun:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/b", - "search": "", - "hash": "" - }, - { - "input": "stun://:443", - "base": null, - "failure": true - }, - { - "input": "stun://test:test", - "base": null, - "failure": true - }, - { - "input": "stun://[:1]", - "base": null, - "failure": true - }, - { - "input": "w://x:0", - "base": null, - "href": "w://x:0", - "origin": "null", - "protocol": "w:", - "username": "", - "password": "", - "host": "x:0", - "hostname": "x", - "port": "0", - "pathname": "", - "search": "", - "hash": "" - }, - { - "input": "west://x:0", - "base": null, - "href": "west://x:0", - "origin": "null", - "protocol": "west:", - "username": "", - "password": "", - "host": "x:0", - "hostname": "x", - "port": "0", - "pathname": "", - "search": "", - "hash": "" - }, - "Scheme relative path starting with multiple slashes", - { - "input": "///test", - "base": "http://example.org/", - "href": "http://test/", - "protocol": "http:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "///\\//\\//test", - "base": "http://example.org/", - "href": "http://test/", - "protocol": "http:", - "username": "", - "password": "", - "host": "test", - "hostname": "test", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "///example.org/path", - "base": "http://example.org/", - "href": "http://example.org/path", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/path", - "search": "", - "hash": "" - }, - { - "input": "///example.org/../path", - "base": "http://example.org/", - "href": "http://example.org/path", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/path", - "search": "", - "hash": "" - }, - { - "input": "///example.org/../../", - "base": "http://example.org/", - "href": "http://example.org/", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "///example.org/../path/../../", - "base": "http://example.org/", - "href": "http://example.org/", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "///example.org/../path/../../path", - "base": "http://example.org/", - "href": "http://example.org/path", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/path", - "search": "", - "hash": "" - }, - { - "input": "/\\/\\//example.org/../path", - "base": "http://example.org/", - "href": "http://example.org/path", - "protocol": "http:", - "username": "", - "password": "", - "host": "example.org", - "hostname": "example.org", - "port": "", - "pathname": "/path", - "search": "", - "hash": "" - }, - { - "input": "///abcdef/../", - "base": "file:///", - "href": "file:///", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - }, - { - "input": "/\\//\\/a/../", - "base": "file:///", - "href": "file://////", - "protocol": "file:", - "username": "", - "password": "", - "host": "", - "hostname": "", - "port": "", - "pathname": "////", - "search": "", - "hash": "" - }, - { - "input": "//a/../", - "base": "file:///", - "href": "file://a/", - "protocol": "file:", - "username": "", - "password": "", - "host": "a", - "hostname": "a", - "port": "", - "pathname": "/", - "search": "", - "hash": "" - } -] diff --git a/tests/test_api.py b/tests/test_api.py deleted file mode 100644 index 225f384ede..0000000000 --- a/tests/test_api.py +++ /dev/null @@ -1,102 +0,0 @@ -import typing - -import pytest - -import httpx - - -def test_get(server): - response = httpx.get(server.url) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Hello, world!" - assert response.http_version == "HTTP/1.1" - - -def test_post(server): - response = httpx.post(server.url, content=b"Hello, world!") - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_post_byte_iterator(server): - def data() -> typing.Iterator[bytes]: - yield b"Hello" - yield b", " - yield b"world!" - - response = httpx.post(server.url, content=data()) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_post_byte_stream(server): - class Data(httpx.SyncByteStream): - def __iter__(self): - yield b"Hello" - yield b", " - yield b"world!" - - response = httpx.post(server.url, content=Data()) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_options(server): - response = httpx.options(server.url) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_head(server): - response = httpx.head(server.url) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_put(server): - response = httpx.put(server.url, content=b"Hello, world!") - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_patch(server): - response = httpx.patch(server.url, content=b"Hello, world!") - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_delete(server): - response = httpx.delete(server.url) - assert response.status_code == 200 - assert response.reason_phrase == "OK" - - -def test_stream(server): - with httpx.stream("GET", server.url) as response: - response.read() - - assert response.status_code == 200 - assert response.reason_phrase == "OK" - assert response.text == "Hello, world!" - assert response.http_version == "HTTP/1.1" - - -def test_get_invalid_url(): - with pytest.raises(httpx.UnsupportedProtocol): - httpx.get("invalid://example.org") - - -# check that httpcore isn't imported until we do a request -def test_httpcore_lazy_loading(server): - import sys - - # unload our module if it is already loaded - if "httpx" in sys.modules: - del sys.modules["httpx"] - del sys.modules["httpcore"] - import httpx - - assert "httpcore" not in sys.modules - _response = httpx.get(server.url) - assert "httpcore" in sys.modules diff --git a/tests/test_asgi.py b/tests/test_asgi.py deleted file mode 100644 index ffbc91bc00..0000000000 --- a/tests/test_asgi.py +++ /dev/null @@ -1,224 +0,0 @@ -import json - -import pytest - -import httpx - - -async def hello_world(scope, receive, send): - status = 200 - output = b"Hello, World!" - headers = [(b"content-type", "text/plain"), (b"content-length", str(len(output)))] - - await send({"type": "http.response.start", "status": status, "headers": headers}) - await send({"type": "http.response.body", "body": output}) - - -async def echo_path(scope, receive, send): - status = 200 - output = json.dumps({"path": scope["path"]}).encode("utf-8") - headers = [(b"content-type", "text/plain"), (b"content-length", str(len(output)))] - - await send({"type": "http.response.start", "status": status, "headers": headers}) - await send({"type": "http.response.body", "body": output}) - - -async def echo_raw_path(scope, receive, send): - status = 200 - output = json.dumps({"raw_path": scope["raw_path"].decode("ascii")}).encode("utf-8") - headers = [(b"content-type", "text/plain"), (b"content-length", str(len(output)))] - - await send({"type": "http.response.start", "status": status, "headers": headers}) - await send({"type": "http.response.body", "body": output}) - - -async def echo_body(scope, receive, send): - status = 200 - headers = [(b"content-type", "text/plain")] - - await send({"type": "http.response.start", "status": status, "headers": headers}) - more_body = True - while more_body: - message = await receive() - body = message.get("body", b"") - more_body = message.get("more_body", False) - await send({"type": "http.response.body", "body": body, "more_body": more_body}) - - -async def echo_headers(scope, receive, send): - status = 200 - output = json.dumps( - {"headers": [[k.decode(), v.decode()] for k, v in scope["headers"]]} - ).encode("utf-8") - headers = [(b"content-type", "text/plain"), (b"content-length", str(len(output)))] - - await send({"type": "http.response.start", "status": status, "headers": headers}) - await send({"type": "http.response.body", "body": output}) - - -async def raise_exc(scope, receive, send): - raise RuntimeError() - - -async def raise_exc_after_response(scope, receive, send): - status = 200 - output = b"Hello, World!" - headers = [(b"content-type", "text/plain"), (b"content-length", str(len(output)))] - - await send({"type": "http.response.start", "status": status, "headers": headers}) - await send({"type": "http.response.body", "body": output}) - raise RuntimeError() - - -@pytest.mark.anyio -async def test_asgi_transport(): - async with httpx.ASGITransport(app=hello_world) as transport: - request = httpx.Request("GET", "http://www.example.com/") - response = await transport.handle_async_request(request) - await response.aread() - assert response.status_code == 200 - assert response.content == b"Hello, World!" - - -@pytest.mark.anyio -async def test_asgi_transport_no_body(): - async with httpx.ASGITransport(app=echo_body) as transport: - request = httpx.Request("GET", "http://www.example.com/") - response = await transport.handle_async_request(request) - await response.aread() - assert response.status_code == 200 - assert response.content == b"" - - -@pytest.mark.anyio -async def test_asgi(): - transport = httpx.ASGITransport(app=hello_world) - async with httpx.AsyncClient(transport=transport) as client: - response = await client.get("http://www.example.org/") - - assert response.status_code == 200 - assert response.text == "Hello, World!" - - -@pytest.mark.anyio -async def test_asgi_urlencoded_path(): - transport = httpx.ASGITransport(app=echo_path) - async with httpx.AsyncClient(transport=transport) as client: - url = httpx.URL("http://www.example.org/").copy_with(path="/user@example.org") - response = await client.get(url) - - assert response.status_code == 200 - assert response.json() == {"path": "/user@example.org"} - - -@pytest.mark.anyio -async def test_asgi_raw_path(): - transport = httpx.ASGITransport(app=echo_raw_path) - async with httpx.AsyncClient(transport=transport) as client: - url = httpx.URL("http://www.example.org/").copy_with(path="/user@example.org") - response = await client.get(url) - - assert response.status_code == 200 - assert response.json() == {"raw_path": "/user@example.org"} - - -@pytest.mark.anyio -async def test_asgi_raw_path_should_not_include_querystring_portion(): - """ - See https://github.com/encode/httpx/issues/2810 - """ - transport = httpx.ASGITransport(app=echo_raw_path) - async with httpx.AsyncClient(transport=transport) as client: - url = httpx.URL("http://www.example.org/path?query") - response = await client.get(url) - - assert response.status_code == 200 - assert response.json() == {"raw_path": "/path"} - - -@pytest.mark.anyio -async def test_asgi_upload(): - transport = httpx.ASGITransport(app=echo_body) - async with httpx.AsyncClient(transport=transport) as client: - response = await client.post("http://www.example.org/", content=b"example") - - assert response.status_code == 200 - assert response.text == "example" - - -@pytest.mark.anyio -async def test_asgi_headers(): - transport = httpx.ASGITransport(app=echo_headers) - async with httpx.AsyncClient(transport=transport) as client: - response = await client.get("http://www.example.org/") - - assert response.status_code == 200 - assert response.json() == { - "headers": [ - ["host", "www.example.org"], - ["accept", "*/*"], - ["accept-encoding", "gzip, deflate, br, zstd"], - ["connection", "keep-alive"], - ["user-agent", f"python-httpx/{httpx.__version__}"], - ] - } - - -@pytest.mark.anyio -async def test_asgi_exc(): - transport = httpx.ASGITransport(app=raise_exc) - async with httpx.AsyncClient(transport=transport) as client: - with pytest.raises(RuntimeError): - await client.get("http://www.example.org/") - - -@pytest.mark.anyio -async def test_asgi_exc_after_response(): - transport = httpx.ASGITransport(app=raise_exc_after_response) - async with httpx.AsyncClient(transport=transport) as client: - with pytest.raises(RuntimeError): - await client.get("http://www.example.org/") - - -@pytest.mark.anyio -async def test_asgi_disconnect_after_response_complete(): - disconnect = False - - async def read_body(scope, receive, send): - nonlocal disconnect - - status = 200 - headers = [(b"content-type", "text/plain")] - - await send( - {"type": "http.response.start", "status": status, "headers": headers} - ) - more_body = True - while more_body: - message = await receive() - more_body = message.get("more_body", False) - - await send({"type": "http.response.body", "body": b"", "more_body": False}) - - # The ASGI spec says of the Disconnect message: - # "Sent to the application when a HTTP connection is closed or if receive is - # called after a response has been sent." - # So if receive() is called again, the disconnect message should be received - message = await receive() - disconnect = message.get("type") == "http.disconnect" - - transport = httpx.ASGITransport(app=read_body) - async with httpx.AsyncClient(transport=transport) as client: - response = await client.post("http://www.example.org/", content=b"example") - - assert response.status_code == 200 - assert disconnect - - -@pytest.mark.anyio -async def test_asgi_exc_no_raise(): - transport = httpx.ASGITransport(app=raise_exc, raise_app_exceptions=False) - async with httpx.AsyncClient(transport=transport) as client: - response = await client.get("http://www.example.org/") - - assert response.status_code == 500 diff --git a/tests/test_auth.py b/tests/test_auth.py deleted file mode 100644 index 6b6df922ea..0000000000 --- a/tests/test_auth.py +++ /dev/null @@ -1,308 +0,0 @@ -""" -Unit tests for auth classes. - -Integration tests also exist in tests/client/test_auth.py -""" - -from urllib.request import parse_keqv_list - -import pytest - -import httpx - - -def test_basic_auth(): - auth = httpx.BasicAuth(username="user", password="pass") - request = httpx.Request("GET", "https://www.example.com") - - # The initial request should include a basic auth header. - flow = auth.sync_auth_flow(request) - request = next(flow) - assert request.headers["Authorization"].startswith("Basic") - - # No other requests are made. - response = httpx.Response(content=b"Hello, world!", status_code=200) - with pytest.raises(StopIteration): - flow.send(response) - - -def test_digest_auth_with_200(): - auth = httpx.DigestAuth(username="user", password="pass") - request = httpx.Request("GET", "https://www.example.com") - - # The initial request should not include an auth header. - flow = auth.sync_auth_flow(request) - request = next(flow) - assert "Authorization" not in request.headers - - # If a 200 response is returned, then no other requests are made. - response = httpx.Response(content=b"Hello, world!", status_code=200) - with pytest.raises(StopIteration): - flow.send(response) - - -def test_digest_auth_with_401(): - auth = httpx.DigestAuth(username="user", password="pass") - request = httpx.Request("GET", "https://www.example.com") - - # The initial request should not include an auth header. - flow = auth.sync_auth_flow(request) - request = next(flow) - assert "Authorization" not in request.headers - - # If a 401 response is returned, then a digest auth request is made. - headers = { - "WWW-Authenticate": 'Digest realm="...", qop="auth", nonce="...", opaque="..."' - } - response = httpx.Response( - content=b"Auth required", status_code=401, headers=headers, request=request - ) - request = flow.send(response) - assert request.headers["Authorization"].startswith("Digest") - - # No other requests are made. - response = httpx.Response(content=b"Hello, world!", status_code=200) - with pytest.raises(StopIteration): - flow.send(response) - - -def test_digest_auth_with_401_nonce_counting(): - auth = httpx.DigestAuth(username="user", password="pass") - request = httpx.Request("GET", "https://www.example.com") - - # The initial request should not include an auth header. - flow = auth.sync_auth_flow(request) - request = next(flow) - assert "Authorization" not in request.headers - - # If a 401 response is returned, then a digest auth request is made. - headers = { - "WWW-Authenticate": 'Digest realm="...", qop="auth", nonce="...", opaque="..."' - } - response = httpx.Response( - content=b"Auth required", status_code=401, headers=headers, request=request - ) - first_request = flow.send(response) - assert first_request.headers["Authorization"].startswith("Digest") - - # Each subsequent request contains the digest header by default... - request = httpx.Request("GET", "https://www.example.com") - flow = auth.sync_auth_flow(request) - second_request = next(flow) - assert second_request.headers["Authorization"].startswith("Digest") - - # ... and the client nonce count (nc) is increased - first_nc = parse_keqv_list(first_request.headers["Authorization"].split(", "))["nc"] - second_nc = parse_keqv_list(second_request.headers["Authorization"].split(", "))[ - "nc" - ] - assert int(first_nc, 16) + 1 == int(second_nc, 16) - - # No other requests are made. - response = httpx.Response(content=b"Hello, world!", status_code=200) - with pytest.raises(StopIteration): - flow.send(response) - - -def set_cookies(request: httpx.Request) -> httpx.Response: - headers = { - "Set-Cookie": "session=.session_value...", - "WWW-Authenticate": 'Digest realm="...", qop="auth", nonce="...", opaque="..."', - } - if request.url.path == "/auth": - return httpx.Response( - content=b"Auth required", status_code=401, headers=headers - ) - else: - raise NotImplementedError() # pragma: no cover - - -def test_digest_auth_setting_cookie_in_request(): - url = "https://www.example.com/auth" - client = httpx.Client(transport=httpx.MockTransport(set_cookies)) - request = client.build_request("GET", url) - - auth = httpx.DigestAuth(username="user", password="pass") - flow = auth.sync_auth_flow(request) - request = next(flow) - assert "Authorization" not in request.headers - - response = client.get(url) - assert len(response.cookies) > 0 - assert response.cookies["session"] == ".session_value..." - - request = flow.send(response) - assert request.headers["Authorization"].startswith("Digest") - assert request.headers["Cookie"] == "session=.session_value..." - - # No other requests are made. - response = httpx.Response( - content=b"Hello, world!", status_code=200, request=request - ) - with pytest.raises(StopIteration): - flow.send(response) - - -def test_digest_auth_rfc_2069(): - # Example from https://datatracker.ietf.org/doc/html/rfc2069#section-2.4 - # with corrected response from https://www.rfc-editor.org/errata/eid749 - - auth = httpx.DigestAuth(username="Mufasa", password="CircleOfLife") - request = httpx.Request("GET", "https://www.example.com/dir/index.html") - - # The initial request should not include an auth header. - flow = auth.sync_auth_flow(request) - request = next(flow) - assert "Authorization" not in request.headers - - # If a 401 response is returned, then a digest auth request is made. - headers = { - "WWW-Authenticate": ( - 'Digest realm="testrealm@host.com", ' - 'nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093", ' - 'opaque="5ccc069c403ebaf9f0171e9517f40e41"' - ) - } - response = httpx.Response( - content=b"Auth required", status_code=401, headers=headers, request=request - ) - request = flow.send(response) - assert request.headers["Authorization"].startswith("Digest") - assert 'username="Mufasa"' in request.headers["Authorization"] - assert 'realm="testrealm@host.com"' in request.headers["Authorization"] - assert ( - 'nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093"' in request.headers["Authorization"] - ) - assert 'uri="/dir/index.html"' in request.headers["Authorization"] - assert ( - 'opaque="5ccc069c403ebaf9f0171e9517f40e41"' in request.headers["Authorization"] - ) - assert ( - 'response="1949323746fe6a43ef61f9606e7febea"' - in request.headers["Authorization"] - ) - - # No other requests are made. - response = httpx.Response(content=b"Hello, world!", status_code=200) - with pytest.raises(StopIteration): - flow.send(response) - - -def test_digest_auth_rfc_7616_md5(monkeypatch): - # Example from https://datatracker.ietf.org/doc/html/rfc7616#section-3.9.1 - - def mock_get_client_nonce(nonce_count: int, nonce: bytes) -> bytes: - return "f2/wE4q74E6zIJEtWaHKaf5wv/H5QzzpXusqGemxURZJ".encode() - - auth = httpx.DigestAuth(username="Mufasa", password="Circle of Life") - monkeypatch.setattr(auth, "_get_client_nonce", mock_get_client_nonce) - - request = httpx.Request("GET", "https://www.example.com/dir/index.html") - - # The initial request should not include an auth header. - flow = auth.sync_auth_flow(request) - request = next(flow) - assert "Authorization" not in request.headers - - # If a 401 response is returned, then a digest auth request is made. - headers = { - "WWW-Authenticate": ( - 'Digest realm="http-auth@example.org", ' - 'qop="auth, auth-int", ' - "algorithm=MD5, " - 'nonce="7ypf/xlj9XXwfDPEoM4URrv/xwf94BcCAzFZH4GiTo0v", ' - 'opaque="FQhe/qaU925kfnzjCev0ciny7QMkPqMAFRtzCUYo5tdS"' - ) - } - response = httpx.Response( - content=b"Auth required", status_code=401, headers=headers, request=request - ) - request = flow.send(response) - assert request.headers["Authorization"].startswith("Digest") - assert 'username="Mufasa"' in request.headers["Authorization"] - assert 'realm="http-auth@example.org"' in request.headers["Authorization"] - assert 'uri="/dir/index.html"' in request.headers["Authorization"] - assert "algorithm=MD5" in request.headers["Authorization"] - assert ( - 'nonce="7ypf/xlj9XXwfDPEoM4URrv/xwf94BcCAzFZH4GiTo0v"' - in request.headers["Authorization"] - ) - assert "nc=00000001" in request.headers["Authorization"] - assert ( - 'cnonce="f2/wE4q74E6zIJEtWaHKaf5wv/H5QzzpXusqGemxURZJ"' - in request.headers["Authorization"] - ) - assert "qop=auth" in request.headers["Authorization"] - assert ( - 'opaque="FQhe/qaU925kfnzjCev0ciny7QMkPqMAFRtzCUYo5tdS"' - in request.headers["Authorization"] - ) - assert ( - 'response="8ca523f5e9506fed4657c9700eebdbec"' - in request.headers["Authorization"] - ) - - # No other requests are made. - response = httpx.Response(content=b"Hello, world!", status_code=200) - with pytest.raises(StopIteration): - flow.send(response) - - -def test_digest_auth_rfc_7616_sha_256(monkeypatch): - # Example from https://datatracker.ietf.org/doc/html/rfc7616#section-3.9.1 - - def mock_get_client_nonce(nonce_count: int, nonce: bytes) -> bytes: - return "f2/wE4q74E6zIJEtWaHKaf5wv/H5QzzpXusqGemxURZJ".encode() - - auth = httpx.DigestAuth(username="Mufasa", password="Circle of Life") - monkeypatch.setattr(auth, "_get_client_nonce", mock_get_client_nonce) - - request = httpx.Request("GET", "https://www.example.com/dir/index.html") - - # The initial request should not include an auth header. - flow = auth.sync_auth_flow(request) - request = next(flow) - assert "Authorization" not in request.headers - - # If a 401 response is returned, then a digest auth request is made. - headers = { - "WWW-Authenticate": ( - 'Digest realm="http-auth@example.org", ' - 'qop="auth, auth-int", ' - "algorithm=SHA-256, " - 'nonce="7ypf/xlj9XXwfDPEoM4URrv/xwf94BcCAzFZH4GiTo0v", ' - 'opaque="FQhe/qaU925kfnzjCev0ciny7QMkPqMAFRtzCUYo5tdS"' - ) - } - response = httpx.Response( - content=b"Auth required", status_code=401, headers=headers, request=request - ) - request = flow.send(response) - assert request.headers["Authorization"].startswith("Digest") - assert 'username="Mufasa"' in request.headers["Authorization"] - assert 'realm="http-auth@example.org"' in request.headers["Authorization"] - assert 'uri="/dir/index.html"' in request.headers["Authorization"] - assert "algorithm=SHA-256" in request.headers["Authorization"] - assert ( - 'nonce="7ypf/xlj9XXwfDPEoM4URrv/xwf94BcCAzFZH4GiTo0v"' - in request.headers["Authorization"] - ) - assert "nc=00000001" in request.headers["Authorization"] - assert ( - 'cnonce="f2/wE4q74E6zIJEtWaHKaf5wv/H5QzzpXusqGemxURZJ"' - in request.headers["Authorization"] - ) - assert "qop=auth" in request.headers["Authorization"] - assert ( - 'opaque="FQhe/qaU925kfnzjCev0ciny7QMkPqMAFRtzCUYo5tdS"' - in request.headers["Authorization"] - ) - assert ( - 'response="753927fa0e85d155564e2e272a28d1802ca10daf4496794697cf8db5856cb6c1"' - in request.headers["Authorization"] - ) - - # No other requests are made. - response = httpx.Response(content=b"Hello, world!", status_code=200) - with pytest.raises(StopIteration): - flow.send(response) diff --git a/tests/test_client.py b/tests/test_client.py new file mode 100644 index 0000000000..c26f6ba8db --- /dev/null +++ b/tests/test_client.py @@ -0,0 +1,112 @@ +import json +import httpx +import pytest + + +def echo(request): + request.read() + response = httpx.Response(200, content=httpx.JSON({ + 'method': request.method, + 'query-params': dict(request.url.params.items()), + 'content-type': request.headers.get('Content-Type'), + 'json': json.loads(request.body) if request.body else None, + })) + return response + + +@pytest.fixture +def client(): + with httpx.Client() as client: + yield client + + +@pytest.fixture +def server(): + with httpx.serve_http(echo) as server: + yield server + + +def test_client(client): + assert repr(client) == "" + + +def test_get(client, server): + r = client.get(server.url) + assert r.status_code == 200 + assert r.body == b'{"method":"GET","query-params":{},"content-type":null,"json":null}' + assert r.text == '{"method":"GET","query-params":{},"content-type":null,"json":null}' + + +def test_post(client, server): + data = httpx.JSON({"data": 123}) + r = client.post(server.url, content=data) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'POST', + 'query-params': {}, + 'content-type': 'application/json', + 'json': {"data": 123}, + } + + +def test_put(client, server): + data = httpx.JSON({"data": 123}) + r = client.put(server.url, content=data) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'PUT', + 'query-params': {}, + 'content-type': 'application/json', + 'json': {"data": 123}, + } + + +def test_patch(client, server): + data = httpx.JSON({"data": 123}) + r = client.patch(server.url, content=data) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'PATCH', + 'query-params': {}, + 'content-type': 'application/json', + 'json': {"data": 123}, + } + + +def test_delete(client, server): + r = client.delete(server.url) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'DELETE', + 'query-params': {}, + 'content-type': None, + 'json': None, + } + + +def test_request(client, server): + r = client.request("GET", server.url) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'GET', + 'query-params': {}, + 'content-type': None, + 'json': None, + } + + +def test_stream(client, server): + with client.stream("GET", server.url) as r: + assert r.status_code == 200 + r.read() + assert json.loads(r.body) == { + 'method': 'GET', + 'query-params': {}, + 'content-type': None, + 'json': None, + } + + +def test_get_with_invalid_scheme(client): + with pytest.raises(ValueError): + client.get("nope://www.example.com") diff --git a/tests/test_config.py b/tests/test_config.py deleted file mode 100644 index 22abd4c22c..0000000000 --- a/tests/test_config.py +++ /dev/null @@ -1,184 +0,0 @@ -import ssl -import typing -from pathlib import Path - -import certifi -import pytest - -import httpx - - -def test_load_ssl_config(): - context = httpx.create_ssl_context() - assert context.verify_mode == ssl.VerifyMode.CERT_REQUIRED - assert context.check_hostname is True - - -def test_load_ssl_config_verify_non_existing_file(): - with pytest.raises(IOError): - context = httpx.create_ssl_context() - context.load_verify_locations(cafile="/path/to/nowhere") - - -def test_load_ssl_with_keylog(monkeypatch: typing.Any) -> None: - monkeypatch.setenv("SSLKEYLOGFILE", "test") - context = httpx.create_ssl_context() - assert context.keylog_filename == "test" - - -def test_load_ssl_config_verify_existing_file(): - context = httpx.create_ssl_context() - context.load_verify_locations(capath=certifi.where()) - assert context.verify_mode == ssl.VerifyMode.CERT_REQUIRED - assert context.check_hostname is True - - -def test_load_ssl_config_verify_directory(): - context = httpx.create_ssl_context() - context.load_verify_locations(capath=Path(certifi.where()).parent) - assert context.verify_mode == ssl.VerifyMode.CERT_REQUIRED - assert context.check_hostname is True - - -def test_load_ssl_config_cert_and_key(cert_pem_file, cert_private_key_file): - context = httpx.create_ssl_context() - context.load_cert_chain(cert_pem_file, cert_private_key_file) - assert context.verify_mode == ssl.VerifyMode.CERT_REQUIRED - assert context.check_hostname is True - - -@pytest.mark.parametrize("password", [b"password", "password"]) -def test_load_ssl_config_cert_and_encrypted_key( - cert_pem_file, cert_encrypted_private_key_file, password -): - context = httpx.create_ssl_context() - context.load_cert_chain(cert_pem_file, cert_encrypted_private_key_file, password) - assert context.verify_mode == ssl.VerifyMode.CERT_REQUIRED - assert context.check_hostname is True - - -def test_load_ssl_config_cert_and_key_invalid_password( - cert_pem_file, cert_encrypted_private_key_file -): - with pytest.raises(ssl.SSLError): - context = httpx.create_ssl_context() - context.load_cert_chain( - cert_pem_file, cert_encrypted_private_key_file, "password1" - ) - - -def test_load_ssl_config_cert_without_key_raises(cert_pem_file): - with pytest.raises(ssl.SSLError): - context = httpx.create_ssl_context() - context.load_cert_chain(cert_pem_file) - - -def test_load_ssl_config_no_verify(): - context = httpx.create_ssl_context(verify=False) - assert context.verify_mode == ssl.VerifyMode.CERT_NONE - assert context.check_hostname is False - - -def test_SSLContext_with_get_request(server, cert_pem_file): - context = httpx.create_ssl_context() - context.load_verify_locations(cert_pem_file) - response = httpx.get(server.url, verify=context) - assert response.status_code == 200 - - -def test_limits_repr(): - limits = httpx.Limits(max_connections=100) - expected = ( - "Limits(max_connections=100, max_keepalive_connections=None," - " keepalive_expiry=5.0)" - ) - assert repr(limits) == expected - - -def test_limits_eq(): - limits = httpx.Limits(max_connections=100) - assert limits == httpx.Limits(max_connections=100) - - -def test_timeout_eq(): - timeout = httpx.Timeout(timeout=5.0) - assert timeout == httpx.Timeout(timeout=5.0) - - -def test_timeout_all_parameters_set(): - timeout = httpx.Timeout(connect=5.0, read=5.0, write=5.0, pool=5.0) - assert timeout == httpx.Timeout(timeout=5.0) - - -def test_timeout_from_nothing(): - timeout = httpx.Timeout(None) - assert timeout.connect is None - assert timeout.read is None - assert timeout.write is None - assert timeout.pool is None - - -def test_timeout_from_none(): - timeout = httpx.Timeout(timeout=None) - assert timeout == httpx.Timeout(None) - - -def test_timeout_from_one_none_value(): - timeout = httpx.Timeout(None, read=None) - assert timeout == httpx.Timeout(None) - - -def test_timeout_from_one_value(): - timeout = httpx.Timeout(None, read=5.0) - assert timeout == httpx.Timeout(timeout=(None, 5.0, None, None)) - - -def test_timeout_from_one_value_and_default(): - timeout = httpx.Timeout(5.0, pool=60.0) - assert timeout == httpx.Timeout(timeout=(5.0, 5.0, 5.0, 60.0)) - - -def test_timeout_missing_default(): - with pytest.raises(ValueError): - httpx.Timeout(pool=60.0) - - -def test_timeout_from_tuple(): - timeout = httpx.Timeout(timeout=(5.0, 5.0, 5.0, 5.0)) - assert timeout == httpx.Timeout(timeout=5.0) - - -def test_timeout_from_config_instance(): - timeout = httpx.Timeout(timeout=5.0) - assert httpx.Timeout(timeout) == httpx.Timeout(timeout=5.0) - - -def test_timeout_repr(): - timeout = httpx.Timeout(timeout=5.0) - assert repr(timeout) == "Timeout(timeout=5.0)" - - timeout = httpx.Timeout(None, read=5.0) - assert repr(timeout) == "Timeout(connect=None, read=5.0, write=None, pool=None)" - - -def test_proxy_from_url(): - proxy = httpx.Proxy("https://example.com") - - assert str(proxy.url) == "https://example.com" - assert proxy.auth is None - assert proxy.headers == {} - assert repr(proxy) == "Proxy('https://example.com')" - - -def test_proxy_with_auth_from_url(): - proxy = httpx.Proxy("https://username:password@example.com") - - assert str(proxy.url) == "https://example.com" - assert proxy.auth == ("username", "password") - assert proxy.headers == {} - assert repr(proxy) == "Proxy('https://example.com', auth=('username', '********'))" - - -def test_invalid_proxy_scheme(): - with pytest.raises(ValueError): - httpx.Proxy("invalid://example.com") diff --git a/tests/test_content.py b/tests/test_content.py index 9bfe983722..ae3158e916 100644 --- a/tests/test_content.py +++ b/tests/test_content.py @@ -1,518 +1,285 @@ -import io -import typing - -import pytest - import httpx +import os +import tempfile -method = "POST" -url = "https://www.example.com" - - -@pytest.mark.anyio -async def test_empty_content(): - request = httpx.Request(method, url) - assert isinstance(request.stream, httpx.SyncByteStream) - assert isinstance(request.stream, httpx.AsyncByteStream) - - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) - - assert request.headers == {"Host": "www.example.com", "Content-Length": "0"} - assert sync_content == b"" - assert async_content == b"" - - -@pytest.mark.anyio -async def test_bytes_content(): - request = httpx.Request(method, url, content=b"Hello, world!") - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) - - assert request.headers == {"Host": "www.example.com", "Content-Length": "13"} - assert sync_content == b"Hello, world!" - assert async_content == b"Hello, world!" - # Support 'data' for compat with requests. - with pytest.warns(DeprecationWarning): - request = httpx.Request(method, url, data=b"Hello, world!") # type: ignore - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) +# HTML - assert request.headers == {"Host": "www.example.com", "Content-Length": "13"} - assert sync_content == b"Hello, world!" - assert async_content == b"Hello, world!" +def test_html(): + html = httpx.HTML("Hello, world") + stream = html.encode() + content_type = html.content_type() -@pytest.mark.anyio -async def test_bytesio_content(): - request = httpx.Request(method, url, content=io.BytesIO(b"Hello, world!")) - assert isinstance(request.stream, typing.Iterable) - assert not isinstance(request.stream, typing.AsyncIterable) + assert stream.read() == b'Hello, world' + assert content_type == "text/html; charset='utf-8'" - content = b"".join(list(request.stream)) - assert request.headers == {"Host": "www.example.com", "Content-Length": "13"} - assert content == b"Hello, world!" +# Text +def test_text(): + text = httpx.Text("Hello, world") -@pytest.mark.anyio -async def test_async_bytesio_content(): - class AsyncBytesIO: - def __init__(self, content: bytes) -> None: - self._idx = 0 - self._content = content + stream = text.encode() + content_type = text.content_type() - async def aread(self, chunk_size: int) -> bytes: - chunk = self._content[self._idx : self._idx + chunk_size] - self._idx = self._idx + chunk_size - return chunk + assert stream.read() == b'Hello, world' + assert content_type == "text/plain; charset='utf-8'" - async def __aiter__(self): - yield self._content # pragma: no cover - request = httpx.Request(method, url, content=AsyncBytesIO(b"Hello, world!")) - assert not isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) +# JSON - content = b"".join([part async for part in request.stream]) +def test_json(): + data = httpx.JSON({'data': 123}) - assert request.headers == { - "Host": "www.example.com", - "Transfer-Encoding": "chunked", - } - assert content == b"Hello, world!" - - -@pytest.mark.anyio -async def test_iterator_content(): - def hello_world() -> typing.Iterator[bytes]: - yield b"Hello, " - yield b"world!" - - request = httpx.Request(method, url, content=hello_world()) - assert isinstance(request.stream, typing.Iterable) - assert not isinstance(request.stream, typing.AsyncIterable) - - content = b"".join(list(request.stream)) + stream = data.encode() + content_type = data.content_type() - assert request.headers == { - "Host": "www.example.com", - "Transfer-Encoding": "chunked", - } - assert content == b"Hello, world!" - - with pytest.raises(httpx.StreamConsumed): - list(request.stream) + assert stream.read() == b'{"data":123}' + assert content_type == "application/json" - # Support 'data' for compat with requests. - with pytest.warns(DeprecationWarning): - request = httpx.Request(method, url, data=hello_world()) # type: ignore - assert isinstance(request.stream, typing.Iterable) - assert not isinstance(request.stream, typing.AsyncIterable) - content = b"".join(list(request.stream)) +# Form - assert request.headers == { - "Host": "www.example.com", - "Transfer-Encoding": "chunked", +def test_form(): + f = httpx.Form("a=123&a=456&b=789") + assert str(f) == "a=123&a=456&b=789" + assert repr(f) == "" + assert f.multi_dict() == { + "a": ["123", "456"], + "b": ["789"] } - assert content == b"Hello, world!" - - -@pytest.mark.anyio -async def test_aiterator_content(): - async def hello_world() -> typing.AsyncIterator[bytes]: - yield b"Hello, " - yield b"world!" - - request = httpx.Request(method, url, content=hello_world()) - assert not isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - content = b"".join([part async for part in request.stream]) - assert request.headers == { - "Host": "www.example.com", - "Transfer-Encoding": "chunked", +def test_form_from_dict(): + f = httpx.Form({ + "a": ["123", "456"], + "b": "789" + }) + assert str(f) == "a=123&a=456&b=789" + assert repr(f) == "" + assert f.multi_dict() == { + "a": ["123", "456"], + "b": ["789"] } - assert content == b"Hello, world!" - with pytest.raises(httpx.StreamConsumed): - [part async for part in request.stream] - # Support 'data' for compat with requests. - with pytest.warns(DeprecationWarning): - request = httpx.Request(method, url, data=hello_world()) # type: ignore - assert not isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - - content = b"".join([part async for part in request.stream]) - - assert request.headers == { - "Host": "www.example.com", - "Transfer-Encoding": "chunked", +def test_form_from_list(): + f = httpx.Form([("a", "123"), ("a", "456"), ("b", "789")]) + assert str(f) == "a=123&a=456&b=789" + assert repr(f) == "" + assert f.multi_dict() == { + "a": ["123", "456"], + "b": ["789"] } - assert content == b"Hello, world!" -@pytest.mark.anyio -async def test_json_content(): - request = httpx.Request(method, url, json={"Hello": "world!"}) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) +def test_empty_form(): + f = httpx.Form() + assert str(f) == '' + assert repr(f) == "" + assert f.multi_dict() == {} - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) - assert request.headers == { - "Host": "www.example.com", - "Content-Length": "18", - "Content-Type": "application/json", - } - assert sync_content == b'{"Hello":"world!"}' - assert async_content == b'{"Hello":"world!"}' +def test_form_accessors(): + f = httpx.Form([("a", "123"), ("a", "456"), ("b", "789")]) + assert "a" in f + assert "A" not in f + assert "c" not in f + assert f["a"] == "123" + assert f.get("a") == "123" + assert f.get("nope", default=None) is None -@pytest.mark.anyio -async def test_urlencoded_content(): - request = httpx.Request(method, url, data={"Hello": "world!"}) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) +def test_form_dict(): + f = httpx.Form([("a", "123"), ("a", "456"), ("b", "789")]) + assert list(f.keys()) == ["a", "b"] + assert list(f.values()) == ["123", "789"] + assert list(f.items()) == [("a", "123"), ("b", "789")] + assert list(f) == ["a", "b"] + assert dict(f) == {"a": "123", "b": "789"} - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) - - assert request.headers == { - "Host": "www.example.com", - "Content-Length": "14", - "Content-Type": "application/x-www-form-urlencoded", - } - assert sync_content == b"Hello=world%21" - assert async_content == b"Hello=world%21" +def test_form_multidict(): + f = httpx.Form([("a", "123"), ("a", "456"), ("b", "789")]) + assert f.get_list("a") == ["123", "456"] + assert f.multi_items() == [("a", "123"), ("a", "456"), ("b", "789")] + assert f.multi_dict() == {"a": ["123", "456"], "b": ["789"]} -@pytest.mark.anyio -async def test_urlencoded_boolean(): - request = httpx.Request(method, url, data={"example": True}) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) +def test_form_builtins(): + f = httpx.Form([("a", "123"), ("a", "456"), ("b", "789")]) + assert len(f) == 2 + assert bool(f) + assert hash(f) + assert f == httpx.Form([("a", "123"), ("a", "456"), ("b", "789")]) - assert request.headers == { - "Host": "www.example.com", - "Content-Length": "12", - "Content-Type": "application/x-www-form-urlencoded", - } - assert sync_content == b"example=true" - assert async_content == b"example=true" +def test_form_copy_operations(): + f = httpx.Form([("a", "123"), ("a", "456"), ("b", "789")]) + assert f.copy_set("a", "abc") == httpx.Form([("a", "abc"), ("b", "789")]) + assert f.copy_append("a", "abc") == httpx.Form([("a", "123"), ("a", "456"), ("a", "abc"), ("b", "789")]) + assert f.copy_remove("a") == httpx.Form([("b", "789")]) -@pytest.mark.anyio -async def test_urlencoded_none(): - request = httpx.Request(method, url, data={"example": None}) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) +def test_form_encode(): + form = httpx.Form({'email': 'address@example.com'}) + assert form['email'] == "address@example.com" - assert request.headers == { - "Host": "www.example.com", - "Content-Length": "8", - "Content-Type": "application/x-www-form-urlencoded", - } - assert sync_content == b"example=" - assert async_content == b"example=" + stream = form.encode() + content_type = form.content_type() + assert stream.read() == b"email=address%40example.com" + assert content_type == "application/x-www-form-urlencoded" -@pytest.mark.anyio -async def test_urlencoded_list(): - request = httpx.Request(method, url, data={"example": ["a", 1, True]}) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) +# Files - assert request.headers == { - "Host": "www.example.com", - "Content-Length": "32", - "Content-Type": "application/x-www-form-urlencoded", - } - assert sync_content == b"example=a&example=1&example=true" - assert async_content == b"example=a&example=1&example=true" - - -@pytest.mark.anyio -async def test_multipart_files_content(): - files = {"file": io.BytesIO(b"")} - headers = {"Content-Type": "multipart/form-data; boundary=+++"} - request = httpx.Request( - method, - url, - files=files, - headers=headers, - ) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) +def test_files(): + f = httpx.Files() + assert f.multi_dict() == {} + assert repr(f) == "" - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) - assert request.headers == { - "Host": "www.example.com", - "Content-Length": "138", - "Content-Type": "multipart/form-data; boundary=+++", - } - assert sync_content == b"".join( - [ - b"--+++\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--+++--\r\n", - ] - ) - assert async_content == b"".join( - [ - b"--+++\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--+++--\r\n", +def test_files_from_dict(): + f = httpx.Files({ + "a": [ + httpx.File("123.json"), + httpx.File("456.json"), + ], + "b": httpx.File("789.json") + }) + assert f.multi_dict() == { + "a": [ + httpx.File("123.json"), + httpx.File("456.json"), + ], + "b": [ + httpx.File("789.json"), ] + } + assert repr(f) == ( + "), ('a', ), ('b', )]>" ) -@pytest.mark.anyio -async def test_multipart_data_and_files_content(): - data = {"message": "Hello, world!"} - files = {"file": io.BytesIO(b"")} - headers = {"Content-Type": "multipart/form-data; boundary=+++"} - request = httpx.Request(method, url, data=data, files=files, headers=headers) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) - - assert request.headers == { - "Host": "www.example.com", - "Content-Length": "210", - "Content-Type": "multipart/form-data; boundary=+++", - } - assert sync_content == b"".join( - [ - b"--+++\r\n", - b'Content-Disposition: form-data; name="message"\r\n', - b"\r\n", - b"Hello, world!\r\n", - b"--+++\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--+++--\r\n", - ] - ) - assert async_content == b"".join( - [ - b"--+++\r\n", - b'Content-Disposition: form-data; name="message"\r\n', - b"\r\n", - b"Hello, world!\r\n", - b"--+++\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--+++--\r\n", +def test_files_from_list(): + f = httpx.Files([ + ("a", httpx.File("123.json")), + ("a", httpx.File("456.json")), + ("b", httpx.File("789.json")) + ]) + assert f.multi_dict() == { + "a": [ + httpx.File("123.json"), + httpx.File("456.json"), + ], + "b": [ + httpx.File("789.json"), ] + } + assert repr(f) == ( + "), ('a', ), ('b', )]>" ) -@pytest.mark.anyio -async def test_empty_request(): - request = httpx.Request(method, url, data={}, files={}) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) - - assert request.headers == {"Host": "www.example.com", "Content-Length": "0"} - assert sync_content == b"" - assert async_content == b"" - - -def test_invalid_argument(): - with pytest.raises(TypeError): - httpx.Request(method, url, content=123) # type: ignore - - with pytest.raises(TypeError): - httpx.Request(method, url, content={"a": "b"}) # type: ignore - - -@pytest.mark.anyio -async def test_multipart_multiple_files_single_input_content(): - files = [ - ("file", io.BytesIO(b"")), - ("file", io.BytesIO(b"")), +def test_files_accessors(): + f = httpx.Files([ + ("a", httpx.File("123.json")), + ("a", httpx.File("456.json")), + ("b", httpx.File("789.json")) + ]) + assert "a" in f + assert "A" not in f + assert "c" not in f + assert f["a"] == httpx.File("123.json") + assert f.get("a") == httpx.File("123.json") + assert f.get("nope", default=None) is None + + +def test_files_dict(): + f = httpx.Files([ + ("a", httpx.File("123.json")), + ("a", httpx.File("456.json")), + ("b", httpx.File("789.json")) + ]) + assert list(f.keys()) == ["a", "b"] + assert list(f.values()) == [httpx.File("123.json"), httpx.File("789.json")] + assert list(f.items()) == [("a", httpx.File("123.json")), ("b", httpx.File("789.json"))] + assert list(f) == ["a", "b"] + assert dict(f) == {"a": httpx.File("123.json"), "b": httpx.File("789.json")} + + +def test_files_multidict(): + f = httpx.Files([ + ("a", httpx.File("123.json")), + ("a", httpx.File("456.json")), + ("b", httpx.File("789.json")) + ]) + assert f.get_list("a") == [ + httpx.File("123.json"), + httpx.File("456.json"), ] - headers = {"Content-Type": "multipart/form-data; boundary=+++"} - request = httpx.Request(method, url, files=files, headers=headers) - assert isinstance(request.stream, typing.Iterable) - assert isinstance(request.stream, typing.AsyncIterable) - - sync_content = b"".join(list(request.stream)) - async_content = b"".join([part async for part in request.stream]) - - assert request.headers == { - "Host": "www.example.com", - "Content-Length": "271", - "Content-Type": "multipart/form-data; boundary=+++", - } - assert sync_content == b"".join( - [ - b"--+++\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--+++\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--+++--\r\n", - ] - ) - assert async_content == b"".join( - [ - b"--+++\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--+++\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--+++--\r\n", + assert f.multi_items() == [ + ("a", httpx.File("123.json")), + ("a", httpx.File("456.json")), + ("b", httpx.File("789.json")), + ] + assert f.multi_dict() == { + "a": [ + httpx.File("123.json"), + httpx.File("456.json"), + ], + "b": [ + httpx.File("789.json"), ] - ) - - -@pytest.mark.anyio -async def test_response_empty_content(): - response = httpx.Response(200) - assert isinstance(response.stream, typing.Iterable) - assert isinstance(response.stream, typing.AsyncIterable) - - sync_content = b"".join(list(response.stream)) - async_content = b"".join([part async for part in response.stream]) - - assert response.headers == {} - assert sync_content == b"" - assert async_content == b"" - - -@pytest.mark.anyio -async def test_response_bytes_content(): - response = httpx.Response(200, content=b"Hello, world!") - assert isinstance(response.stream, typing.Iterable) - assert isinstance(response.stream, typing.AsyncIterable) - - sync_content = b"".join(list(response.stream)) - async_content = b"".join([part async for part in response.stream]) - - assert response.headers == {"Content-Length": "13"} - assert sync_content == b"Hello, world!" - assert async_content == b"Hello, world!" - - -@pytest.mark.anyio -async def test_response_iterator_content(): - def hello_world() -> typing.Iterator[bytes]: - yield b"Hello, " - yield b"world!" - - response = httpx.Response(200, content=hello_world()) - assert isinstance(response.stream, typing.Iterable) - assert not isinstance(response.stream, typing.AsyncIterable) - - content = b"".join(list(response.stream)) - - assert response.headers == {"Transfer-Encoding": "chunked"} - assert content == b"Hello, world!" - - with pytest.raises(httpx.StreamConsumed): - list(response.stream) - - -@pytest.mark.anyio -async def test_response_aiterator_content(): - async def hello_world() -> typing.AsyncIterator[bytes]: - yield b"Hello, " - yield b"world!" - - response = httpx.Response(200, content=hello_world()) - assert not isinstance(response.stream, typing.Iterable) - assert isinstance(response.stream, typing.AsyncIterable) - - content = b"".join([part async for part in response.stream]) - - assert response.headers == {"Transfer-Encoding": "chunked"} - assert content == b"Hello, world!" - - with pytest.raises(httpx.StreamConsumed): - [part async for part in response.stream] - - -def test_response_invalid_argument(): - with pytest.raises(TypeError): - httpx.Response(200, content=123) # type: ignore - - -def test_ensure_ascii_false_with_french_characters(): - data = {"greeting": "Bonjour, ça va ?"} - response = httpx.Response(200, json=data) - assert "ça va" in response.text, ( - "ensure_ascii=False should preserve French accented characters" - ) - assert response.headers["Content-Type"] == "application/json" - - -def test_separators_for_compact_json(): - data = {"clé": "valeur", "liste": [1, 2, 3]} - response = httpx.Response(200, json=data) - assert response.text == '{"clé":"valeur","liste":[1,2,3]}', ( - "separators=(',', ':') should produce a compact representation" - ) - assert response.headers["Content-Type"] == "application/json" - + } -def test_allow_nan_false(): - data_with_nan = {"nombre": float("nan")} - data_with_inf = {"nombre": float("inf")} - with pytest.raises( - ValueError, match="Out of range float values are not JSON compliant" - ): - httpx.Response(200, json=data_with_nan) - with pytest.raises( - ValueError, match="Out of range float values are not JSON compliant" - ): - httpx.Response(200, json=data_with_inf) +def test_files_builtins(): + f = httpx.Files([ + ("a", httpx.File("123.json")), + ("a", httpx.File("456.json")), + ("b", httpx.File("789.json")) + ]) + assert len(f) == 2 + assert bool(f) + assert f == httpx.Files([ + ("a", httpx.File("123.json")), + ("a", httpx.File("456.json")), + ("b", httpx.File("789.json")), + ]) + + +def test_multipart(): + with tempfile.NamedTemporaryFile() as f: + f.write(b"Hello, world") + f.seek(0) + + multipart = httpx.MultiPart( + form={'email': 'me@example.com'}, + files={'upload': httpx.File(f.name)}, + boundary='BOUNDARY', + ) + assert multipart.form['email'] == "me@example.com" + assert multipart.files['upload'] == httpx.File(f.name) + + fname = os.path.basename(f.name).encode('utf-8') + stream = multipart.encode() + content_type = multipart.content_type() + + content_type == "multipart/form-data; boundary=BOUNDARY" + content = stream.read() + assert content == ( + b'--BOUNDARY\r\n' + b'Content-Disposition: form-data; name="email"\r\n' + b'\r\n' + b'me@example.com\r\n' + b'--BOUNDARY\r\n' + b'Content-Disposition: form-data; name="upload"; filename="' + fname + b'"\r\n' + b'\r\n' + b'Hello, world\r\n' + b'--BOUNDARY--\r\n' + ) diff --git a/tests/test_decoders.py b/tests/test_decoders.py deleted file mode 100644 index 9ffaba189d..0000000000 --- a/tests/test_decoders.py +++ /dev/null @@ -1,355 +0,0 @@ -from __future__ import annotations - -import io -import typing -import zlib - -import chardet -import pytest -import zstandard as zstd - -import httpx - - -def test_deflate(): - """ - Deflate encoding may use either 'zlib' or 'deflate' in the wild. - - https://stackoverflow.com/questions/1838699/how-can-i-decompress-a-gzip-stream-with-zlib#answer-22311297 - """ - body = b"test 123" - compressor = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS) - compressed_body = compressor.compress(body) + compressor.flush() - - headers = [(b"Content-Encoding", b"deflate")] - response = httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - assert response.content == body - - -def test_zlib(): - """ - Deflate encoding may use either 'zlib' or 'deflate' in the wild. - - https://stackoverflow.com/questions/1838699/how-can-i-decompress-a-gzip-stream-with-zlib#answer-22311297 - """ - body = b"test 123" - compressed_body = zlib.compress(body) - - headers = [(b"Content-Encoding", b"deflate")] - response = httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - assert response.content == body - - -def test_gzip(): - body = b"test 123" - compressor = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16) - compressed_body = compressor.compress(body) + compressor.flush() - - headers = [(b"Content-Encoding", b"gzip")] - response = httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - assert response.content == body - - -def test_brotli(): - body = b"test 123" - compressed_body = b"\x8b\x03\x80test 123\x03" - - headers = [(b"Content-Encoding", b"br")] - response = httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - assert response.content == body - - -def test_zstd(): - body = b"test 123" - compressed_body = zstd.compress(body) - - headers = [(b"Content-Encoding", b"zstd")] - response = httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - assert response.content == body - - -def test_zstd_decoding_error(): - compressed_body = "this_is_not_zstd_compressed_data" - - headers = [(b"Content-Encoding", b"zstd")] - with pytest.raises(httpx.DecodingError): - httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - - -def test_zstd_empty(): - headers = [(b"Content-Encoding", b"zstd")] - response = httpx.Response(200, headers=headers, content=b"") - assert response.content == b"" - - -def test_zstd_truncated(): - body = b"test 123" - compressed_body = zstd.compress(body) - - headers = [(b"Content-Encoding", b"zstd")] - with pytest.raises(httpx.DecodingError): - httpx.Response( - 200, - headers=headers, - content=compressed_body[1:3], - ) - - -def test_zstd_multiframe(): - # test inspired by urllib3 test suite - data = ( - # Zstandard frame - zstd.compress(b"foo") - # skippable frame (must be ignored) - + bytes.fromhex( - "50 2A 4D 18" # Magic_Number (little-endian) - "07 00 00 00" # Frame_Size (little-endian) - "00 00 00 00 00 00 00" # User_Data - ) - # Zstandard frame - + zstd.compress(b"bar") - ) - compressed_body = io.BytesIO(data) - - headers = [(b"Content-Encoding", b"zstd")] - response = httpx.Response(200, headers=headers, content=compressed_body) - response.read() - assert response.content == b"foobar" - - -def test_multi(): - body = b"test 123" - - deflate_compressor = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS) - compressed_body = deflate_compressor.compress(body) + deflate_compressor.flush() - - gzip_compressor = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16) - compressed_body = ( - gzip_compressor.compress(compressed_body) + gzip_compressor.flush() - ) - - headers = [(b"Content-Encoding", b"deflate, gzip")] - response = httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - assert response.content == body - - -def test_multi_with_identity(): - body = b"test 123" - compressed_body = b"\x8b\x03\x80test 123\x03" - - headers = [(b"Content-Encoding", b"br, identity")] - response = httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - assert response.content == body - - headers = [(b"Content-Encoding", b"identity, br")] - response = httpx.Response( - 200, - headers=headers, - content=compressed_body, - ) - assert response.content == body - - -@pytest.mark.anyio -async def test_streaming(): - body = b"test 123" - compressor = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16) - - async def compress(body: bytes) -> typing.AsyncIterator[bytes]: - yield compressor.compress(body) - yield compressor.flush() - - headers = [(b"Content-Encoding", b"gzip")] - response = httpx.Response( - 200, - headers=headers, - content=compress(body), - ) - assert not hasattr(response, "body") - assert await response.aread() == body - - -@pytest.mark.parametrize("header_value", (b"deflate", b"gzip", b"br", b"identity")) -def test_empty_content(header_value): - headers = [(b"Content-Encoding", header_value)] - response = httpx.Response( - 200, - headers=headers, - content=b"", - ) - assert response.content == b"" - - -@pytest.mark.parametrize("header_value", (b"deflate", b"gzip", b"br", b"identity")) -def test_decoders_empty_cases(header_value): - headers = [(b"Content-Encoding", header_value)] - response = httpx.Response(content=b"", status_code=200, headers=headers) - assert response.read() == b"" - - -@pytest.mark.parametrize("header_value", (b"deflate", b"gzip", b"br")) -def test_decoding_errors(header_value): - headers = [(b"Content-Encoding", header_value)] - compressed_body = b"invalid" - with pytest.raises(httpx.DecodingError): - request = httpx.Request("GET", "https://example.org") - httpx.Response(200, headers=headers, content=compressed_body, request=request) - - with pytest.raises(httpx.DecodingError): - httpx.Response(200, headers=headers, content=compressed_body) - - -@pytest.mark.parametrize( - ["data", "encoding"], - [ - ((b"Hello,", b" world!"), "ascii"), - ((b"\xe3\x83", b"\x88\xe3\x83\xa9", b"\xe3", b"\x83\x99\xe3\x83\xab"), "utf-8"), - ((b"Euro character: \x88! abcdefghijklmnopqrstuvwxyz", b""), "cp1252"), - ((b"Accented: \xd6sterreich abcdefghijklmnopqrstuvwxyz", b""), "iso-8859-1"), - ], -) -@pytest.mark.anyio -async def test_text_decoder_with_autodetect(data, encoding): - async def iterator() -> typing.AsyncIterator[bytes]: - nonlocal data - for chunk in data: - yield chunk - - def autodetect(content): - return chardet.detect(content).get("encoding") - - # Accessing `.text` on a read response. - response = httpx.Response(200, content=iterator(), default_encoding=autodetect) - await response.aread() - assert response.text == (b"".join(data)).decode(encoding) - - # Streaming `.aiter_text` iteratively. - # Note that if we streamed the text *without* having read it first, then - # we won't get a `charset_normalizer` guess, and will instead always rely - # on utf-8 if no charset is specified. - text = "".join([part async for part in response.aiter_text()]) - assert text == (b"".join(data)).decode(encoding) - - -@pytest.mark.anyio -async def test_text_decoder_known_encoding(): - async def iterator() -> typing.AsyncIterator[bytes]: - yield b"\x83g" - yield b"\x83" - yield b"\x89\x83x\x83\x8b" - - response = httpx.Response( - 200, - headers=[(b"Content-Type", b"text/html; charset=shift-jis")], - content=iterator(), - ) - - await response.aread() - assert "".join(response.text) == "トラベル" - - -def test_text_decoder_empty_cases(): - response = httpx.Response(200, content=b"") - assert response.text == "" - - response = httpx.Response(200, content=[b""]) - response.read() - assert response.text == "" - - -@pytest.mark.parametrize( - ["data", "expected"], - [((b"Hello,", b" world!"), ["Hello,", " world!"])], -) -def test_streaming_text_decoder( - data: typing.Iterable[bytes], expected: list[str] -) -> None: - response = httpx.Response(200, content=iter(data)) - assert list(response.iter_text()) == expected - - -def test_line_decoder_nl(): - response = httpx.Response(200, content=[b""]) - assert list(response.iter_lines()) == [] - - response = httpx.Response(200, content=[b"", b"a\n\nb\nc"]) - assert list(response.iter_lines()) == ["a", "", "b", "c"] - - # Issue #1033 - response = httpx.Response( - 200, content=[b"", b"12345\n", b"foo ", b"bar ", b"baz\n"] - ) - assert list(response.iter_lines()) == ["12345", "foo bar baz"] - - -def test_line_decoder_cr(): - response = httpx.Response(200, content=[b"", b"a\r\rb\rc"]) - assert list(response.iter_lines()) == ["a", "", "b", "c"] - - response = httpx.Response(200, content=[b"", b"a\r\rb\rc\r"]) - assert list(response.iter_lines()) == ["a", "", "b", "c"] - - # Issue #1033 - response = httpx.Response( - 200, content=[b"", b"12345\r", b"foo ", b"bar ", b"baz\r"] - ) - assert list(response.iter_lines()) == ["12345", "foo bar baz"] - - -def test_line_decoder_crnl(): - response = httpx.Response(200, content=[b"", b"a\r\n\r\nb\r\nc"]) - assert list(response.iter_lines()) == ["a", "", "b", "c"] - - response = httpx.Response(200, content=[b"", b"a\r\n\r\nb\r\nc\r\n"]) - assert list(response.iter_lines()) == ["a", "", "b", "c"] - - response = httpx.Response(200, content=[b"", b"a\r", b"\n\r\nb\r\nc"]) - assert list(response.iter_lines()) == ["a", "", "b", "c"] - - # Issue #1033 - response = httpx.Response(200, content=[b"", b"12345\r\n", b"foo bar baz\r\n"]) - assert list(response.iter_lines()) == ["12345", "foo bar baz"] - - -def test_invalid_content_encoding_header(): - headers = [(b"Content-Encoding", b"invalid-header")] - body = b"test 123" - - response = httpx.Response( - 200, - headers=headers, - content=body, - ) - assert response.content == body diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py deleted file mode 100644 index 60c8721c02..0000000000 --- a/tests/test_exceptions.py +++ /dev/null @@ -1,63 +0,0 @@ -from __future__ import annotations - -import typing - -import httpcore -import pytest - -import httpx - -if typing.TYPE_CHECKING: # pragma: no cover - from conftest import TestServer - - -def test_httpcore_all_exceptions_mapped() -> None: - """ - All exception classes exposed by HTTPCore are properly mapped to an HTTPX-specific - exception class. - """ - expected_mapped_httpcore_exceptions = { - value.__name__ - for _, value in vars(httpcore).items() - if isinstance(value, type) - and issubclass(value, Exception) - and value is not httpcore.ConnectionNotAvailable - } - - httpx_exceptions = { - value.__name__ - for _, value in vars(httpx).items() - if isinstance(value, type) and issubclass(value, Exception) - } - - unmapped_exceptions = expected_mapped_httpcore_exceptions - httpx_exceptions - - if unmapped_exceptions: # pragma: no cover - pytest.fail(f"Unmapped httpcore exceptions: {unmapped_exceptions}") - - -def test_httpcore_exception_mapping(server: TestServer) -> None: - """ - HTTPCore exception mapping works as expected. - """ - impossible_port = 123456 - with pytest.raises(httpx.ConnectError): - httpx.get(server.url.copy_with(port=impossible_port)) - - with pytest.raises(httpx.ReadTimeout): - httpx.get( - server.url.copy_with(path="/slow_response"), - timeout=httpx.Timeout(5, read=0.01), - ) - - -def test_request_attribute() -> None: - # Exception without request attribute - exc = httpx.ReadTimeout("Read operation timed out") - with pytest.raises(RuntimeError): - exc.request # noqa: B018 - - # Exception with request attribute - request = httpx.Request("GET", "https://www.example.com") - exc = httpx.ReadTimeout("Read operation timed out", request=request) - assert exc.request == request diff --git a/tests/test_exported_members.py b/tests/test_exported_members.py deleted file mode 100644 index 8d9c8a74ca..0000000000 --- a/tests/test_exported_members.py +++ /dev/null @@ -1,13 +0,0 @@ -import httpx - - -def test_all_imports_are_exported() -> None: - included_private_members = ["__description__", "__title__", "__version__"] - assert httpx.__all__ == sorted( - ( - member - for member in vars(httpx).keys() - if not member.startswith("_") or member in included_private_members - ), - key=str.casefold, - ) diff --git a/tests/test_headers.py b/tests/test_headers.py new file mode 100644 index 0000000000..6ebb99dca2 --- /dev/null +++ b/tests/test_headers.py @@ -0,0 +1,109 @@ +import httpx +import pytest + + +def test_headers_from_dict(): + headers = httpx.Headers({ + 'Content-Length': '1024', + 'Content-Type': 'text/plain; charset=utf-8', + }) + assert headers['Content-Length'] == '1024' + assert headers['Content-Type'] == 'text/plain; charset=utf-8' + + +def test_headers_from_list(): + headers = httpx.Headers([ + ('Location', 'https://www.example.com'), + ('Set-Cookie', 'session_id=3498jj489jhb98jn'), + ]) + assert headers['Location'] == 'https://www.example.com' + assert headers['Set-Cookie'] == 'session_id=3498jj489jhb98jn' + + +def test_header_keys(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.keys()) == ["Accept", "User-Agent"] + + +def test_header_values(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.values()) == ["*/*", "python/httpx"] + + +def test_header_items(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert list(h.items()) == [("Accept", "*/*"), ("User-Agent", "python/httpx")] + + +def test_header_get(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert h.get("User-Agent") == "python/httpx" + assert h.get("user-agent") == "python/httpx" + assert h.get("missing") is None + + +def test_header_copy_set(): + h = httpx.Headers({"Expires": "0"}) + h = h.copy_set("Expires", "Wed, 21 Oct 2015 07:28:00 GMT") + assert h == httpx.Headers({"Expires": "Wed, 21 Oct 2015 07:28:00 GMT"}) + + h = httpx.Headers({"Expires": "0"}) + h = h.copy_set("expires", "Wed, 21 Oct 2015 07:28:00 GMT") + assert h == httpx.Headers({"Expires": "Wed, 21 Oct 2015 07:28:00 GMT"}) + + +def test_header_copy_remove(): + h = httpx.Headers({"Accept": "*/*"}) + h = h.copy_remove("Accept") + assert h == httpx.Headers({}) + + h = httpx.Headers({"Accept": "*/*"}) + h = h.copy_remove("accept") + assert h == httpx.Headers({}) + + +def test_header_getitem(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert h["User-Agent"] == "python/httpx" + assert h["user-agent"] == "python/httpx" + with pytest.raises(KeyError): + h["missing"] + + +def test_header_contains(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert "User-Agent" in h + assert "user-agent" in h + assert "missing" not in h + + +def test_header_bool(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert bool(h) + h = httpx.Headers() + assert not bool(h) + + +def test_header_iter(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert [k for k in h] == ["Accept", "User-Agent"] + + +def test_header_len(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert len(h) == 2 + + +def test_header_repr(): + h = httpx.Headers({"Accept": "*/*", "User-Agent": "python/httpx"}) + assert repr(h) == "" + + +def test_header_invalid_name(): + with pytest.raises(ValueError): + httpx.Headers({"Accept\n": "*/*"}) + + +def test_header_invalid_value(): + with pytest.raises(ValueError): + httpx.Headers({"Accept": "*/*\n"}) diff --git a/tests/test_main.py b/tests/test_main.py deleted file mode 100644 index b1a77d485b..0000000000 --- a/tests/test_main.py +++ /dev/null @@ -1,187 +0,0 @@ -import os -import typing - -from click.testing import CliRunner - -import httpx - - -def splitlines(output: str) -> typing.Iterable[str]: - return [line.strip() for line in output.splitlines()] - - -def remove_date_header(lines: typing.Iterable[str]) -> typing.Iterable[str]: - return [line for line in lines if not line.startswith("date:")] - - -def test_help(): - runner = CliRunner() - result = runner.invoke(httpx.main, ["--help"]) - assert result.exit_code == 0 - assert "A next generation HTTP client." in result.output - - -def test_get(server): - url = str(server.url) - runner = CliRunner() - result = runner.invoke(httpx.main, [url]) - assert result.exit_code == 0 - assert remove_date_header(splitlines(result.output)) == [ - "HTTP/1.1 200 OK", - "server: uvicorn", - "content-type: text/plain", - "Transfer-Encoding: chunked", - "", - "Hello, world!", - ] - - -def test_json(server): - url = str(server.url.copy_with(path="/json")) - runner = CliRunner() - result = runner.invoke(httpx.main, [url]) - assert result.exit_code == 0 - assert remove_date_header(splitlines(result.output)) == [ - "HTTP/1.1 200 OK", - "server: uvicorn", - "content-type: application/json", - "Transfer-Encoding: chunked", - "", - "{", - '"Hello": "world!"', - "}", - ] - - -def test_binary(server): - url = str(server.url.copy_with(path="/echo_binary")) - runner = CliRunner() - content = "Hello, world!" - result = runner.invoke(httpx.main, [url, "-c", content]) - assert result.exit_code == 0 - assert remove_date_header(splitlines(result.output)) == [ - "HTTP/1.1 200 OK", - "server: uvicorn", - "content-type: application/octet-stream", - "Transfer-Encoding: chunked", - "", - f"<{len(content)} bytes of binary data>", - ] - - -def test_redirects(server): - url = str(server.url.copy_with(path="/redirect_301")) - runner = CliRunner() - result = runner.invoke(httpx.main, [url]) - assert result.exit_code == 1 - assert remove_date_header(splitlines(result.output)) == [ - "HTTP/1.1 301 Moved Permanently", - "server: uvicorn", - "location: /", - "Transfer-Encoding: chunked", - "", - ] - - -def test_follow_redirects(server): - url = str(server.url.copy_with(path="/redirect_301")) - runner = CliRunner() - result = runner.invoke(httpx.main, [url, "--follow-redirects"]) - assert result.exit_code == 0 - assert remove_date_header(splitlines(result.output)) == [ - "HTTP/1.1 301 Moved Permanently", - "server: uvicorn", - "location: /", - "Transfer-Encoding: chunked", - "", - "HTTP/1.1 200 OK", - "server: uvicorn", - "content-type: text/plain", - "Transfer-Encoding: chunked", - "", - "Hello, world!", - ] - - -def test_post(server): - url = str(server.url.copy_with(path="/echo_body")) - runner = CliRunner() - result = runner.invoke(httpx.main, [url, "-m", "POST", "-j", '{"hello": "world"}']) - assert result.exit_code == 0 - assert remove_date_header(splitlines(result.output)) == [ - "HTTP/1.1 200 OK", - "server: uvicorn", - "content-type: text/plain", - "Transfer-Encoding: chunked", - "", - '{"hello":"world"}', - ] - - -def test_verbose(server): - url = str(server.url) - runner = CliRunner() - result = runner.invoke(httpx.main, [url, "-v"]) - assert result.exit_code == 0 - assert remove_date_header(splitlines(result.output)) == [ - "* Connecting to '127.0.0.1'", - "* Connected to '127.0.0.1' on port 8000", - "GET / HTTP/1.1", - f"Host: {server.url.netloc.decode('ascii')}", - "Accept: */*", - "Accept-Encoding: gzip, deflate, br, zstd", - "Connection: keep-alive", - f"User-Agent: python-httpx/{httpx.__version__}", - "", - "HTTP/1.1 200 OK", - "server: uvicorn", - "content-type: text/plain", - "Transfer-Encoding: chunked", - "", - "Hello, world!", - ] - - -def test_auth(server): - url = str(server.url) - runner = CliRunner() - result = runner.invoke(httpx.main, [url, "-v", "--auth", "username", "password"]) - print(result.output) - assert result.exit_code == 0 - assert remove_date_header(splitlines(result.output)) == [ - "* Connecting to '127.0.0.1'", - "* Connected to '127.0.0.1' on port 8000", - "GET / HTTP/1.1", - f"Host: {server.url.netloc.decode('ascii')}", - "Accept: */*", - "Accept-Encoding: gzip, deflate, br, zstd", - "Connection: keep-alive", - f"User-Agent: python-httpx/{httpx.__version__}", - "Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=", - "", - "HTTP/1.1 200 OK", - "server: uvicorn", - "content-type: text/plain", - "Transfer-Encoding: chunked", - "", - "Hello, world!", - ] - - -def test_download(server): - url = str(server.url) - runner = CliRunner() - with runner.isolated_filesystem(): - runner.invoke(httpx.main, [url, "--download", "index.txt"]) - assert os.path.exists("index.txt") - with open("index.txt", "r") as input_file: - assert input_file.read() == "Hello, world!" - - -def test_errors(): - runner = CliRunner() - result = runner.invoke(httpx.main, ["invalid://example.org"]) - assert result.exit_code == 1 - assert splitlines(result.output) == [ - "UnsupportedProtocol: Request URL has an unsupported protocol 'invalid://'.", - ] diff --git a/tests/test_multipart.py b/tests/test_multipart.py deleted file mode 100644 index 764f85a253..0000000000 --- a/tests/test_multipart.py +++ /dev/null @@ -1,469 +0,0 @@ -from __future__ import annotations - -import io -import tempfile -import typing - -import pytest - -import httpx - - -def echo_request_content(request: httpx.Request) -> httpx.Response: - return httpx.Response(200, content=request.content) - - -@pytest.mark.parametrize(("value,output"), (("abc", b"abc"), (b"abc", b"abc"))) -def test_multipart(value, output): - client = httpx.Client(transport=httpx.MockTransport(echo_request_content)) - - # Test with a single-value 'data' argument, and a plain file 'files' argument. - data = {"text": value} - files = {"file": io.BytesIO(b"")} - response = client.post("http://127.0.0.1:8000/", data=data, files=files) - boundary = response.request.headers["Content-Type"].split("boundary=")[-1] - boundary_bytes = boundary.encode("ascii") - - assert response.status_code == 200 - assert response.content == b"".join( - [ - b"--" + boundary_bytes + b"\r\n", - b'Content-Disposition: form-data; name="text"\r\n', - b"\r\n", - b"abc\r\n", - b"--" + boundary_bytes + b"\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--" + boundary_bytes + b"--\r\n", - ] - ) - - -@pytest.mark.parametrize( - "header", - [ - "multipart/form-data; boundary=+++; charset=utf-8", - "multipart/form-data; charset=utf-8; boundary=+++", - "multipart/form-data; boundary=+++", - "multipart/form-data; boundary=+++ ;", - 'multipart/form-data; boundary="+++"; charset=utf-8', - 'multipart/form-data; charset=utf-8; boundary="+++"', - 'multipart/form-data; boundary="+++"', - 'multipart/form-data; boundary="+++" ;', - ], -) -def test_multipart_explicit_boundary(header: str) -> None: - client = httpx.Client(transport=httpx.MockTransport(echo_request_content)) - - files = {"file": io.BytesIO(b"")} - headers = {"content-type": header} - response = client.post("http://127.0.0.1:8000/", files=files, headers=headers) - boundary_bytes = b"+++" - - assert response.status_code == 200 - assert response.request.headers["Content-Type"] == header - assert response.content == b"".join( - [ - b"--" + boundary_bytes + b"\r\n", - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n', - b"Content-Type: application/octet-stream\r\n", - b"\r\n", - b"\r\n", - b"--" + boundary_bytes + b"--\r\n", - ] - ) - - -@pytest.mark.parametrize( - "header", - [ - "multipart/form-data; charset=utf-8", - "multipart/form-data; charset=utf-8; ", - ], -) -def test_multipart_header_without_boundary(header: str) -> None: - client = httpx.Client(transport=httpx.MockTransport(echo_request_content)) - - files = {"file": io.BytesIO(b"")} - headers = {"content-type": header} - response = client.post("http://127.0.0.1:8000/", files=files, headers=headers) - - assert response.status_code == 200 - assert response.request.headers["Content-Type"] == header - - -@pytest.mark.parametrize(("key"), (b"abc", 1, 2.3, None)) -def test_multipart_invalid_key(key): - client = httpx.Client(transport=httpx.MockTransport(echo_request_content)) - - data = {key: "abc"} - files = {"file": io.BytesIO(b"")} - with pytest.raises(TypeError) as e: - client.post( - "http://127.0.0.1:8000/", - data=data, - files=files, - ) - assert "Invalid type for name" in str(e.value) - assert repr(key) in str(e.value) - - -@pytest.mark.parametrize(("value"), (object(), {"key": "value"})) -def test_multipart_invalid_value(value): - client = httpx.Client(transport=httpx.MockTransport(echo_request_content)) - - data = {"text": value} - files = {"file": io.BytesIO(b"")} - with pytest.raises(TypeError) as e: - client.post("http://127.0.0.1:8000/", data=data, files=files) - assert "Invalid type for value" in str(e.value) - - -def test_multipart_file_tuple(): - client = httpx.Client(transport=httpx.MockTransport(echo_request_content)) - - # Test with a list of values 'data' argument, - # and a tuple style 'files' argument. - data = {"text": ["abc"]} - files = {"file": ("name.txt", io.BytesIO(b""))} - response = client.post("http://127.0.0.1:8000/", data=data, files=files) - boundary = response.request.headers["Content-Type"].split("boundary=")[-1] - boundary_bytes = boundary.encode("ascii") - - assert response.status_code == 200 - assert response.content == b"".join( - [ - b"--" + boundary_bytes + b"\r\n", - b'Content-Disposition: form-data; name="text"\r\n', - b"\r\n", - b"abc\r\n", - b"--" + boundary_bytes + b"\r\n", - b'Content-Disposition: form-data; name="file"; filename="name.txt"\r\n', - b"Content-Type: text/plain\r\n", - b"\r\n", - b"\r\n", - b"--" + boundary_bytes + b"--\r\n", - ] - ) - - -@pytest.mark.parametrize("file_content_type", [None, "text/plain"]) -def test_multipart_file_tuple_headers(file_content_type: str | None) -> None: - file_name = "test.txt" - file_content = io.BytesIO(b"") - file_headers = {"Expires": "0"} - - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - files = {"file": (file_name, file_content, file_content_type, file_headers)} - - request = httpx.Request("POST", url, headers=headers, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Content-Length": str(len(request.content)), - } - assert request.content == ( - f'--BOUNDARY\r\nContent-Disposition: form-data; name="file"; ' - f'filename="{file_name}"\r\nExpires: 0\r\nContent-Type: ' - f"text/plain\r\n\r\n\r\n--BOUNDARY--\r\n" - "".encode("ascii") - ) - - -def test_multipart_headers_include_content_type() -> None: - """ - Content-Type from 4th tuple parameter (headers) should - override the 3rd parameter (content_type) - """ - file_name = "test.txt" - file_content = io.BytesIO(b"") - file_content_type = "text/plain" - file_headers = {"Content-Type": "image/png"} - - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - files = {"file": (file_name, file_content, file_content_type, file_headers)} - - request = httpx.Request("POST", url, headers=headers, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Content-Length": str(len(request.content)), - } - assert request.content == ( - f'--BOUNDARY\r\nContent-Disposition: form-data; name="file"; ' - f'filename="{file_name}"\r\nContent-Type: ' - f"image/png\r\n\r\n\r\n--BOUNDARY--\r\n" - "".encode("ascii") - ) - - -def test_multipart_encode(tmp_path: typing.Any) -> None: - path = str(tmp_path / "name.txt") - with open(path, "wb") as f: - f.write(b"") - - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - data = { - "a": "1", - "b": b"C", - "c": ["11", "22", "33"], - "d": "", - "e": True, - "f": "", - } - with open(path, "rb") as input_file: - files = {"file": ("name.txt", input_file)} - - request = httpx.Request("POST", url, headers=headers, data=data, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Content-Length": str(len(request.content)), - } - assert request.content == ( - '--BOUNDARY\r\nContent-Disposition: form-data; name="a"\r\n\r\n1\r\n' - '--BOUNDARY\r\nContent-Disposition: form-data; name="b"\r\n\r\nC\r\n' - '--BOUNDARY\r\nContent-Disposition: form-data; name="c"\r\n\r\n11\r\n' - '--BOUNDARY\r\nContent-Disposition: form-data; name="c"\r\n\r\n22\r\n' - '--BOUNDARY\r\nContent-Disposition: form-data; name="c"\r\n\r\n33\r\n' - '--BOUNDARY\r\nContent-Disposition: form-data; name="d"\r\n\r\n\r\n' - '--BOUNDARY\r\nContent-Disposition: form-data; name="e"\r\n\r\ntrue\r\n' - '--BOUNDARY\r\nContent-Disposition: form-data; name="f"\r\n\r\n\r\n' - '--BOUNDARY\r\nContent-Disposition: form-data; name="file";' - ' filename="name.txt"\r\n' - "Content-Type: text/plain\r\n\r\n\r\n" - "--BOUNDARY--\r\n" - "".encode("ascii") - ) - - -def test_multipart_encode_unicode_file_contents() -> None: - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - files = {"file": ("name.txt", b"")} - - request = httpx.Request("POST", url, headers=headers, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Content-Length": str(len(request.content)), - } - assert request.content == ( - b'--BOUNDARY\r\nContent-Disposition: form-data; name="file";' - b' filename="name.txt"\r\n' - b"Content-Type: text/plain\r\n\r\n\r\n" - b"--BOUNDARY--\r\n" - ) - - -def test_multipart_encode_files_allows_filenames_as_none() -> None: - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - files = {"file": (None, io.BytesIO(b""))} - - request = httpx.Request("POST", url, headers=headers, data={}, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Content-Length": str(len(request.content)), - } - assert request.content == ( - '--BOUNDARY\r\nContent-Disposition: form-data; name="file"\r\n\r\n' - "\r\n--BOUNDARY--\r\n" - "".encode("ascii") - ) - - -@pytest.mark.parametrize( - "file_name,expected_content_type", - [ - ("example.json", "application/json"), - ("example.txt", "text/plain"), - ("no-extension", "application/octet-stream"), - ], -) -def test_multipart_encode_files_guesses_correct_content_type( - file_name: str, expected_content_type: str -) -> None: - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - files = {"file": (file_name, io.BytesIO(b""))} - - request = httpx.Request("POST", url, headers=headers, data={}, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Content-Length": str(len(request.content)), - } - assert request.content == ( - f'--BOUNDARY\r\nContent-Disposition: form-data; name="file"; ' - f'filename="{file_name}"\r\nContent-Type: ' - f"{expected_content_type}\r\n\r\n\r\n--BOUNDARY--\r\n" - "".encode("ascii") - ) - - -def test_multipart_encode_files_allows_bytes_content() -> None: - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - files = {"file": ("test.txt", b"", "text/plain")} - - request = httpx.Request("POST", url, headers=headers, data={}, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Content-Length": str(len(request.content)), - } - assert request.content == ( - '--BOUNDARY\r\nContent-Disposition: form-data; name="file"; ' - 'filename="test.txt"\r\n' - "Content-Type: text/plain\r\n\r\n\r\n" - "--BOUNDARY--\r\n" - "".encode("ascii") - ) - - -def test_multipart_encode_files_allows_str_content() -> None: - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - files = {"file": ("test.txt", "", "text/plain")} - - request = httpx.Request("POST", url, headers=headers, data={}, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Content-Length": str(len(request.content)), - } - assert request.content == ( - '--BOUNDARY\r\nContent-Disposition: form-data; name="file"; ' - 'filename="test.txt"\r\n' - "Content-Type: text/plain\r\n\r\n\r\n" - "--BOUNDARY--\r\n" - "".encode("ascii") - ) - - -def test_multipart_encode_files_raises_exception_with_StringIO_content() -> None: - url = "https://www.example.com" - files = {"file": ("test.txt", io.StringIO("content"), "text/plain")} - with pytest.raises(TypeError): - httpx.Request("POST", url, data={}, files=files) # type: ignore - - -def test_multipart_encode_files_raises_exception_with_text_mode_file() -> None: - url = "https://www.example.com" - with tempfile.TemporaryFile(mode="w") as upload: - files = {"file": ("test.txt", upload, "text/plain")} - with pytest.raises(TypeError): - httpx.Request("POST", url, data={}, files=files) # type: ignore - - -def test_multipart_encode_non_seekable_filelike() -> None: - """ - Test that special readable but non-seekable filelike objects are supported. - In this case uploads with use 'Transfer-Encoding: chunked', instead of - a 'Content-Length' header. - """ - - class IteratorIO(io.IOBase): - def __init__(self, iterator: typing.Iterator[bytes]) -> None: - self._iterator = iterator - - def read(self, *args: typing.Any) -> bytes: - return b"".join(self._iterator) - - def data() -> typing.Iterator[bytes]: - yield b"Hello" - yield b"World" - - url = "https://www.example.com/" - headers = {"Content-Type": "multipart/form-data; boundary=BOUNDARY"} - fileobj: typing.Any = IteratorIO(data()) - files = {"file": fileobj} - - request = httpx.Request("POST", url, headers=headers, files=files) - request.read() - - assert request.headers == { - "Host": "www.example.com", - "Content-Type": "multipart/form-data; boundary=BOUNDARY", - "Transfer-Encoding": "chunked", - } - assert request.content == ( - b"--BOUNDARY\r\n" - b'Content-Disposition: form-data; name="file"; filename="upload"\r\n' - b"Content-Type: application/octet-stream\r\n" - b"\r\n" - b"HelloWorld\r\n" - b"--BOUNDARY--\r\n" - ) - - -def test_multipart_rewinds_files(): - with tempfile.TemporaryFile() as upload: - upload.write(b"Hello, world!") - - transport = httpx.MockTransport(echo_request_content) - client = httpx.Client(transport=transport) - - files = {"file": upload} - response = client.post("http://127.0.0.1:8000/", files=files) - assert response.status_code == 200 - assert b"\r\nHello, world!\r\n" in response.content - - # POSTing the same file instance a second time should have the same content. - files = {"file": upload} - response = client.post("http://127.0.0.1:8000/", files=files) - assert response.status_code == 200 - assert b"\r\nHello, world!\r\n" in response.content - - -class TestHeaderParamHTML5Formatting: - def test_unicode(self): - filename = "n\u00e4me" - expected = b'filename="n\xc3\xa4me"' - files = {"upload": (filename, b"")} - request = httpx.Request("GET", "https://www.example.com", files=files) - assert expected in request.read() - - def test_ascii(self): - filename = "name" - expected = b'filename="name"' - files = {"upload": (filename, b"")} - request = httpx.Request("GET", "https://www.example.com", files=files) - assert expected in request.read() - - def test_unicode_escape(self): - filename = "hello\\world\u0022" - expected = b'filename="hello\\\\world%22"' - files = {"upload": (filename, b"")} - request = httpx.Request("GET", "https://www.example.com", files=files) - assert expected in request.read() - - def test_unicode_with_control_character(self): - filename = "hello\x1a\x1b\x1c" - expected = b'filename="hello%1A\x1b%1C"' - files = {"upload": (filename, b"")} - request = httpx.Request("GET", "https://www.example.com", files=files) - assert expected in request.read() diff --git a/tests/test_network.py b/tests/test_network.py new file mode 100644 index 0000000000..e6ce92563b --- /dev/null +++ b/tests/test_network.py @@ -0,0 +1,101 @@ +import httpx +import pytest + + +def echo(stream): + while buffer := stream.read(): + stream.write(buffer) + + +@pytest.fixture +def server(): + net = httpx.NetworkBackend() + with net.serve("127.0.0.1", 8080, echo) as server: + yield server + + +def test_network_backend(): + net = httpx.NetworkBackend() + assert repr(net) == "" + + +def test_network_backend_connect(server): + net = httpx.NetworkBackend() + stream = net.connect(server.host, server.port) + try: + assert repr(stream) == f"" + stream.write(b"Hello, world.") + content = stream.read() + assert content == b"Hello, world." + finally: + stream.close() + + +def test_network_backend_context_managed(server): + net = httpx.NetworkBackend() + with net.connect(server.host, server.port) as stream: + stream.write(b"Hello, world.") + content = stream.read() + assert content == b"Hello, world." + assert repr(stream) == f"" + + +def test_network_backend_timeout(server): + net = httpx.NetworkBackend() + with httpx.timeout(0.0): + with pytest.raises(TimeoutError): + with net.connect(server.host, server.port) as stream: + pass + + with httpx.timeout(10.0): + with net.connect(server.host, server.port) as stream: + pass + + +# >>> net = httpx.NetworkBackend() +# >>> stream = net.connect("dev.encode.io", 80) +# >>> try: +# >>> ... +# >>> finally: +# >>> stream.close() +# >>> stream +# + +# import httpx +# import ssl +# import truststore + +# net = httpx.NetworkBackend() +# ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT) +# req = b'\r\n'.join([ +# b'GET / HTTP/1.1', +# b'Host: www.example.com', +# b'User-Agent: python/dev', +# b'Connection: close', +# b'', +# ]) + +# # Use a 10 second overall timeout for the entire request/response. +# with timeout(10.0): +# # Use a 3 second timeout for the initial connection. +# with timeout(3.0) as t: +# # Open the connection & establish SSL. +# with net.open_stream("www.example.com", 443) as stream: +# stream.start_tls(ctx, hostname="www.example.com") +# t.cancel() +# # Send the request & read the response. +# stream.write(req) +# buffer = [] +# while part := stream.read(): +# buffer.append(part) +# resp = b''.join(buffer) + + +# def test_fixture(tcp_echo_server): +# host, port = (tcp_echo_server.host, tcp_echo_server.port) + +# net = httpx.NetworkBackend() +# with net.connect(host, port) as stream: +# stream.write(b"123") +# buffer = stream.read() +# assert buffer == b"123" diff --git a/tests/test_parsers.py b/tests/test_parsers.py new file mode 100644 index 0000000000..e2a321e2a2 --- /dev/null +++ b/tests/test_parsers.py @@ -0,0 +1,748 @@ +import httpx +import pytest + + +class TrickleIO(httpx.Stream): + def __init__(self, stream: httpx.Stream): + self._stream = stream + + def read(self, size) -> bytes: + return self._stream.read(1) + + def write(self, data: bytes) -> None: + self._stream.write(data) + + def close(self) -> None: + self._stream.close() + + +def test_parser(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 12\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"hello, world" + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"POST", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Content-Type", b"application/json"), + (b"Content-Length", b"23"), + ]) + p.send_body(b'{"msg": "hello, world"}') + p.send_body(b'') + + assert stream.input_bytes() == ( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 12\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"hello, world" + ) + assert stream.output_bytes() == ( + b"POST / HTTP/1.1\r\n" + b"Host: example.com\r\n" + b"Content-Type: application/json\r\n" + b"Content-Length: 23\r\n" + b"\r\n" + b'{"msg": "hello, world"}' + ) + + protocol, code, reason_phase = p.recv_status_line() + headers = p.recv_headers() + body = p.recv_body() + terminator = p.recv_body() + + assert protocol == b'HTTP/1.1' + assert code == 200 + assert reason_phase == b'OK' + assert headers == [ + (b'Content-Length', b'12'), + (b'Content-Type', b'text/plain'), + ] + assert body == b'hello, world' + assert terminator == b'' + + assert not p.is_idle() + p.complete() + assert p.is_idle() + + +def test_parser_server(): + stream = httpx.DuplexStream( + b"GET / HTTP/1.1\r\n" + b"Host: www.example.com\r\n" + b"\r\n" + ) + + p = httpx.HTTPParser(stream, mode='SERVER') + method, target, protocol = p.recv_method_line() + headers = p.recv_headers() + body = p.recv_body() + + assert method == b'GET' + assert target == b'/' + assert protocol == b'HTTP/1.1' + assert headers == [ + (b'Host', b'www.example.com'), + ] + assert body == b'' + + p.send_status_line(b"HTTP/1.1", 200, b"OK") + p.send_headers([ + (b"Content-Type", b"application/json"), + (b"Content-Length", b"23"), + ]) + p.send_body(b'{"msg": "hello, world"}') + p.send_body(b'') + + assert stream.input_bytes() == ( + b"GET / HTTP/1.1\r\n" + b"Host: www.example.com\r\n" + b"\r\n" + ) + assert stream.output_bytes() == ( + b"HTTP/1.1 200 OK\r\n" + b"Content-Type: application/json\r\n" + b"Content-Length: 23\r\n" + b"\r\n" + b'{"msg": "hello, world"}' + ) + + assert not p.is_idle() + p.complete() + assert p.is_idle() + + +def test_parser_trickle(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 12\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"hello, world" + ) + + p = httpx.HTTPParser(TrickleIO(stream), mode='CLIENT') + p.send_method_line(b"POST", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Content-Type", b"application/json"), + (b"Content-Length", b"23"), + ]) + p.send_body(b'{"msg": "hello, world"}') + p.send_body(b'') + + assert stream.input_bytes() == ( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 12\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"hello, world" + ) + assert stream.output_bytes() == ( + b"POST / HTTP/1.1\r\n" + b"Host: example.com\r\n" + b"Content-Type: application/json\r\n" + b"Content-Length: 23\r\n" + b"\r\n" + b'{"msg": "hello, world"}' + ) + + protocol, code, reason_phase = p.recv_status_line() + headers = p.recv_headers() + body = p.recv_body() + terminator = p.recv_body() + + assert protocol == b'HTTP/1.1' + assert code == 200 + assert reason_phase == b'OK' + assert headers == [ + (b'Content-Length', b'12'), + (b'Content-Type', b'text/plain'), + ] + assert body == b'hello, world' + assert terminator == b'' + + +def test_parser_transfer_encoding_chunked(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Type: text/plain\r\n" + b"Transfer-Encoding: chunked\r\n" + b"\r\n" + b"c\r\n" + b"hello, world\r\n" + b"0\r\n\r\n" + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"POST", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Content-Type", b"application/json"), + (b"Transfer-Encoding", b"chunked"), + ]) + p.send_body(b'{"msg": "hello, world"}') + p.send_body(b'') + + assert stream.input_bytes() == ( + b"HTTP/1.1 200 OK\r\n" + b"Content-Type: text/plain\r\n" + b"Transfer-Encoding: chunked\r\n" + b"\r\n" + b"c\r\n" + b"hello, world\r\n" + b"0\r\n\r\n" + ) + assert stream.output_bytes() == ( + b"POST / HTTP/1.1\r\n" + b"Host: example.com\r\n" + b"Content-Type: application/json\r\n" + b"Transfer-Encoding: chunked\r\n" + b"\r\n" + b'17\r\n' + b'{"msg": "hello, world"}\r\n' + b'0\r\n\r\n' + ) + + protocol, code, reason_phase = p.recv_status_line() + headers = p.recv_headers() + body = p.recv_body() + terminator = p.recv_body() + + assert protocol == b'HTTP/1.1' + assert code == 200 + assert reason_phase == b'OK' + assert headers == [ + (b'Content-Type', b'text/plain'), + (b'Transfer-Encoding', b'chunked'), + ] + assert body == b'hello, world' + assert terminator == b'' + + +def test_parser_transfer_encoding_chunked_trickle(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Type: text/plain\r\n" + b"Transfer-Encoding: chunked\r\n" + b"\r\n" + b"c\r\n" + b"hello, world\r\n" + b"0\r\n\r\n" + ) + + p = httpx.HTTPParser(TrickleIO(stream), mode='CLIENT') + p.send_method_line(b"POST", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Content-Type", b"application/json"), + (b"Transfer-Encoding", b"chunked"), + ]) + p.send_body(b'{"msg": "hello, world"}') + p.send_body(b'') + + assert stream.input_bytes() == ( + b"HTTP/1.1 200 OK\r\n" + b"Content-Type: text/plain\r\n" + b"Transfer-Encoding: chunked\r\n" + b"\r\n" + b"c\r\n" + b"hello, world\r\n" + b"0\r\n\r\n" + ) + assert stream.output_bytes() == ( + b"POST / HTTP/1.1\r\n" + b"Host: example.com\r\n" + b"Content-Type: application/json\r\n" + b"Transfer-Encoding: chunked\r\n" + b"\r\n" + b'17\r\n' + b'{"msg": "hello, world"}\r\n' + b'0\r\n\r\n' + ) + + protocol, code, reason_phase = p.recv_status_line() + headers = p.recv_headers() + body = p.recv_body() + terminator = p.recv_body() + + assert protocol == b'HTTP/1.1' + assert code == 200 + assert reason_phase == b'OK' + assert headers == [ + (b'Content-Type', b'text/plain'), + (b'Transfer-Encoding', b'chunked'), + ] + assert body == b'hello, world' + assert terminator == b'' + + +def test_parser_repr(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Type: application/json\r\n" + b"Content-Length: 23\r\n" + b"\r\n" + b'{"msg": "hello, world"}' + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + assert repr(p) == "" + + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + assert repr(p) == "" + + p.send_headers([(b"Host", b"example.com")]) + assert repr(p) == "" + + p.send_body(b'') + assert repr(p) == "" + + p.recv_status_line() + assert repr(p) == "" + + p.recv_headers() + assert repr(p) == "" + + p.recv_body() + assert repr(p) == "" + + p.recv_body() + assert repr(p) == "" + + p.complete() + assert repr(p) == "" + + +def test_parser_invalid_transitions(): + stream = httpx.DuplexStream() + + with pytest.raises(httpx.ProtocolError): + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b'GET', b'/', b'HTTP/1.1') + p.send_method_line(b'GET', b'/', b'HTTP/1.1') + + with pytest.raises(httpx.ProtocolError): + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_headers([]) + + with pytest.raises(httpx.ProtocolError): + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_body(b'') + + with pytest.raises(httpx.ProtocolError): + reader = httpx.ByteStream(b'HTTP/1.1 200 OK\r\n') + p = httpx.HTTPParser(stream, mode='CLIENT') + p.recv_status_line() + + with pytest.raises(httpx.ProtocolError): + p = httpx.HTTPParser(stream, mode='CLIENT') + p.recv_headers() + + with pytest.raises(httpx.ProtocolError): + p = httpx.HTTPParser(stream, mode='CLIENT') + p.recv_body() + + +def test_parser_invalid_status_line(): + # ... + stream = httpx.DuplexStream(b'...') + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([(b"Host", b"example.com")]) + p.send_body(b'') + + msg = 'Stream closed early reading response status line' + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_status_line() + + # ... + stream = httpx.DuplexStream(b'HTTP/1.1' + b'x' * 5000) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([(b"Host", b"example.com")]) + p.send_body(b'') + + msg = 'Exceeded maximum size reading response status line' + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_status_line() + + # ... + stream = httpx.DuplexStream(b'HTTP/1.1' + b'x' * 5000 + b'\r\n') + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([(b"Host", b"example.com")]) + p.send_body(b'') + + msg = 'Exceeded maximum size reading response status line' + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_status_line() + + +def test_parser_sent_unsupported_protocol(): + # Currently only HTTP/1.1 is supported. + stream = httpx.DuplexStream() + + p = httpx.HTTPParser(stream, mode='CLIENT') + msg = 'Sent unsupported protocol version' + with pytest.raises(httpx.ProtocolError, match=msg): + p.send_method_line(b"GET", b"/", b"HTTP/1.0") + + +def test_parser_recv_unsupported_protocol(): + # Currently only HTTP/1.1 is supported. + stream = httpx.DuplexStream(b"HTTP/1.0 200 OK\r\n") + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + msg = 'Received unsupported protocol version' + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_status_line() + + +def test_parser_large_body(): + body = b"x" * 6988 + + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 6988\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + body + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([(b"Host", b"example.com")]) + p.send_body(b'') + + # Checkout our buffer sizes. + p.recv_status_line() + p.recv_headers() + assert len(p.recv_body()) == 4096 + assert len(p.recv_body()) == 2892 + assert len(p.recv_body()) == 0 + + +def test_parser_stream_large_body(): + body = b"x" * 6956 + + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Transfer-Encoding: chunked\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"1b2c\r\n" + body + b'\r\n0\r\n\r\n' + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([(b"Host", b"example.com")]) + p.send_body(b'') + + # Checkout our buffer sizes. + p.recv_status_line() + p.recv_headers() + # assert len(p.recv_body()) == 4096 + # assert len(p.recv_body()) == 2860 + assert len(p.recv_body()) == 6956 + assert len(p.recv_body()) == 0 + + +def test_parser_not_enough_data_received(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 188\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"truncated" + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([(b"Host", b"example.com")]) + p.send_body(b'') + + # Checkout our buffer sizes. + p.recv_status_line() + p.recv_headers() + p.recv_body() + msg = 'Not enough data received for declared Content-Length' + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_body() + + +def test_parser_not_enough_data_sent(): + stream = httpx.DuplexStream() + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"POST", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Content-Type", b"application/json"), + (b"Content-Length", b"23"), + ]) + p.send_body(b'{"msg": "too smol"}') + msg = 'Not enough data sent for declared Content-Length' + with pytest.raises(httpx.ProtocolError, match=msg): + p.send_body(b'') + + +def test_parser_too_much_data_sent(): + stream = httpx.DuplexStream() + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"POST", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Content-Type", b"application/json"), + (b"Content-Length", b"19"), + ]) + msg = 'Too much data sent for declared Content-Length' + with pytest.raises(httpx.ProtocolError, match=msg): + p.send_body(b'{"msg": "too chonky"}') + + +def test_parser_missing_host_header(): + stream = httpx.DuplexStream() + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + msg = "Request missing 'Host' header" + with pytest.raises(httpx.ProtocolError, match=msg): + p.send_headers([]) + + +def test_client_connection_close(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 12\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"hello, world" + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Connection", b"close"), + ]) + p.send_body(b'') + + protocol, code, reason_phase = p.recv_status_line() + headers = p.recv_headers() + body = p.recv_body() + terminator = p.recv_body() + + assert protocol == b'HTTP/1.1' + assert code == 200 + assert reason_phase == b"OK" + assert headers == [ + (b'Content-Length', b'12'), + (b'Content-Type', b'text/plain'), + ] + assert body == b"hello, world" + assert terminator == b"" + + assert repr(p) == "" + + p.complete() + assert repr(p) == "" + assert p.is_closed() + + +def test_server_connection_close(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 12\r\n" + b"Content-Type: text/plain\r\n" + b"Connection: close\r\n" + b"\r\n" + b"hello, world" + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([(b"Host", b"example.com")]) + p.send_body(b'') + + protocol, code, reason_phase = p.recv_status_line() + headers = p.recv_headers() + body = p.recv_body() + terminator = p.recv_body() + + assert protocol == b'HTTP/1.1' + assert code == 200 + assert reason_phase == b"OK" + assert headers == [ + (b'Content-Length', b'12'), + (b'Content-Type', b'text/plain'), + (b'Connection', b'close'), + ] + assert body == b"hello, world" + assert terminator == b"" + + assert repr(p) == "" + p.complete() + assert repr(p) == "" + + +def test_invalid_status_code(): + stream = httpx.DuplexStream( + b"HTTP/1.1 99 OK\r\n" + b"Content-Length: 12\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"hello, world" + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Connection", b"close"), + ]) + p.send_body(b'') + + msg = "Received invalid status code" + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_status_line() + + +def test_1xx_status_code(): + stream = httpx.DuplexStream( + b"HTTP/1.1 103 Early Hints\r\n" + b"Link: ; rel=preload; as=style\r\n" + b"Link: ; rel=preload; as=script\r\n" + b"\r\n" + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: 12\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"hello, world" + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([(b"Host", b"example.com")]) + p.send_body(b'') + + protocol, code, reason_phase = p.recv_status_line() + headers = p.recv_headers() + + assert protocol == b'HTTP/1.1' + assert code == 103 + assert reason_phase == b'Early Hints' + assert headers == [ + (b'Link', b'; rel=preload; as=style'), + (b'Link', b'; rel=preload; as=script'), + ] + + protocol, code, reason_phase = p.recv_status_line() + headers = p.recv_headers() + body = p.recv_body() + terminator = p.recv_body() + + assert protocol == b'HTTP/1.1' + assert code == 200 + assert reason_phase == b"OK" + assert headers == [ + (b'Content-Length', b'12'), + (b'Content-Type', b'text/plain'), + ] + assert body == b"hello, world" + assert terminator == b"" + + +def test_received_invalid_content_length(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Content-Length: -999\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"hello, world" + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Connection", b"close"), + ]) + p.send_body(b'') + + p.recv_status_line() + msg = "Received invalid Content-Length" + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_headers() + + +def test_sent_invalid_content_length(): + stream = httpx.DuplexStream() + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + msg = "Sent invalid Content-Length" + with pytest.raises(httpx.ProtocolError, match=msg): + # Limited to 20 digits. + # 100 million terabytes should be enough for anyone. + p.send_headers([ + (b"Host", b"example.com"), + (b"Content-Length", b"100000000000000000000"), + ]) + + +def test_received_invalid_characters_in_chunk_size(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Transfer-Encoding: chunked\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"0xFF\r\n..." + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Connection", b"close"), + ]) + p.send_body(b'') + + p.recv_status_line() + p.recv_headers() + msg = "Received invalid chunk size" + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_body() + + +def test_received_oversized_chunk(): + stream = httpx.DuplexStream( + b"HTTP/1.1 200 OK\r\n" + b"Transfer-Encoding: chunked\r\n" + b"Content-Type: text/plain\r\n" + b"\r\n" + b"FFFFFFFFFF\r\n..." + ) + + p = httpx.HTTPParser(stream, mode='CLIENT') + p.send_method_line(b"GET", b"/", b"HTTP/1.1") + p.send_headers([ + (b"Host", b"example.com"), + (b"Connection", b"close"), + ]) + p.send_body(b'') + + p.recv_status_line() + p.recv_headers() + msg = "Received invalid chunk size" + with pytest.raises(httpx.ProtocolError, match=msg): + p.recv_body() diff --git a/tests/test_pool.py b/tests/test_pool.py new file mode 100644 index 0000000000..04cd0246e5 --- /dev/null +++ b/tests/test_pool.py @@ -0,0 +1,126 @@ +import httpx +import pytest + + +def hello_world(request): + content = httpx.Text('Hello, world.') + return httpx.Response(200, content=content) + + +@pytest.fixture +def server(): + with httpx.serve_http(hello_world) as server: + yield server + + +def test_connection_pool_request(server): + with httpx.ConnectionPool() as pool: + assert repr(pool) == "" + assert len(pool.connections) == 0 + + r = pool.request("GET", server.url) + + assert r.status_code == 200 + assert repr(pool) == "" + assert len(pool.connections) == 1 + + +def test_connection_pool_connection_close(server): + with httpx.ConnectionPool() as pool: + assert repr(pool) == "" + assert len(pool.connections) == 0 + + r = pool.request("GET", server.url, headers={"Connection": "close"}) + + # TODO: Really we want closed connections proactively removed from the pool, + assert r.status_code == 200 + assert repr(pool) == "" + assert len(pool.connections) == 1 + + +def test_connection_pool_stream(server): + with httpx.ConnectionPool() as pool: + assert repr(pool) == "" + assert len(pool.connections) == 0 + + with pool.stream("GET", server.url) as r: + assert r.status_code == 200 + assert repr(pool) == "" + assert len(pool.connections) == 1 + r.read() + + assert repr(pool) == "" + assert len(pool.connections) == 1 + + +def test_connection_pool_cannot_request_after_closed(server): + with httpx.ConnectionPool() as pool: + pool + + with pytest.raises(RuntimeError): + pool.request("GET", server.url) + + +def test_connection_pool_should_have_managed_lifespan(server): + pool = httpx.ConnectionPool() + with pytest.warns(UserWarning): + del pool + + +def test_connection_request(server): + with httpx.open_connection(server.url) as conn: + assert repr(conn) == f"" + + r = conn.request("GET", "/") + + assert r.status_code == 200 + assert repr(conn) == f"" + + +def test_connection_stream(server): + with httpx.open_connection(server.url) as conn: + assert repr(conn) == f"" + with conn.stream("GET", "/") as r: + assert r.status_code == 200 + assert repr(conn) == f"" + r.read() + assert repr(conn) == f"" + + +# # with httpx.open_connection("https://www.example.com/") as conn: +# # r = conn.request("GET", "/") + +# # >>> pool = httpx.ConnectionPool() +# # >>> pool +# # + +# # >>> with httpx.open_connection_pool() as pool: +# # >>> res = pool.request("GET", "https://www.example.com") +# # >>> res, pool +# # , + +# # >>> with httpx.open_connection_pool() as pool: +# # >>> with pool.stream("GET", "https://www.example.com") as res: +# # >>> res, pool +# # , + +# # >>> with httpx.open_connection_pool() as pool: +# # >>> req = httpx.Request("GET", "https://www.example.com") +# # >>> with pool.send(req) as res: +# # >>> res.body() +# # >>> res, pool +# # , + +# # >>> with httpx.open_connection_pool() as pool: +# # >>> pool.close() +# # + +# # with httpx.open_connection("https://www.example.com/") as conn: +# # with conn.upgrade("GET", "/feed", {"Upgrade": "WebSocket") as stream: +# # ... + +# # with httpx.open_connection("http://127.0.0.1:8080") as conn: +# # with conn.upgrade("CONNECT", "www.encode.io:443") as stream: +# # stream.start_tls(ctx, hostname="www.encode.io") +# # ... + diff --git a/tests/test_quickstart.py b/tests/test_quickstart.py new file mode 100644 index 0000000000..55c34b1b5a --- /dev/null +++ b/tests/test_quickstart.py @@ -0,0 +1,78 @@ +import json +import httpx +import pytest + + +def echo(request): + request.read() + response = httpx.Response(200, content=httpx.JSON({ + 'method': request.method, + 'query-params': dict(request.url.params.items()), + 'content-type': request.headers.get('Content-Type'), + 'json': json.loads(request.body) if request.body else None, + })) + return response + + +@pytest.fixture +def server(): + with httpx.serve_http(echo) as server: + yield server + + +def test_get(server): + r = httpx.get(server.url) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'GET', + 'query-params': {}, + 'content-type': None, + 'json': None, + } + + +def test_post(server): + data = httpx.JSON({"data": 123}) + r = httpx.post(server.url, content=data) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'POST', + 'query-params': {}, + 'content-type': 'application/json', + 'json': {"data": 123}, + } + + +def test_put(server): + data = httpx.JSON({"data": 123}) + r = httpx.put(server.url, content=data) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'PUT', + 'query-params': {}, + 'content-type': 'application/json', + 'json': {"data": 123}, + } + + +def test_patch(server): + data = httpx.JSON({"data": 123}) + r = httpx.patch(server.url, content=data) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'PATCH', + 'query-params': {}, + 'content-type': 'application/json', + 'json': {"data": 123}, + } + + +def test_delete(server): + r = httpx.delete(server.url) + assert r.status_code == 200 + assert json.loads(r.body) == { + 'method': 'DELETE', + 'query-params': {}, + 'content-type': None, + 'json': None, + } diff --git a/tests/test_request.py b/tests/test_request.py new file mode 100644 index 0000000000..a69e1d1358 --- /dev/null +++ b/tests/test_request.py @@ -0,0 +1,79 @@ +import httpx + + +class ByteIterator: + def __init__(self, buffer=b""): + self._buffer = buffer + + def next(self) -> bytes: + buffer = self._buffer + self._buffer = b'' + return buffer + + +def test_request(): + r = httpx.Request("GET", "https://example.com") + + assert repr(r) == "" + assert r.method == "GET" + assert r.url == "https://example.com" + assert r.headers == { + "Host": "example.com" + } + assert r.read() == b"" + +def test_request_bytes(): + content = b"Hello, world" + r = httpx.Request("POST", "https://example.com", content=content) + + assert repr(r) == "" + assert r.method == "POST" + assert r.url == "https://example.com" + assert r.headers == { + "Host": "example.com", + "Content-Length": "12", + } + assert r.read() == b"Hello, world" + + +def test_request_stream(): + i = ByteIterator(b"Hello, world") + stream = httpx.HTTPStream(i.next, None) + r = httpx.Request("POST", "https://example.com", content=stream) + + assert repr(r) == "" + assert r.method == "POST" + assert r.url == "https://example.com" + assert r.headers == { + "Host": "example.com", + "Transfer-Encoding": "chunked", + } + assert r.read() == b"Hello, world" + + +def test_request_json(): + data = httpx.JSON({"msg": "Hello, world"}) + r = httpx.Request("POST", "https://example.com", content=data) + + assert repr(r) == "" + assert r.method == "POST" + assert r.url == "https://example.com" + assert r.headers == { + "Host": "example.com", + "Content-Length": "22", + "Content-Type": "application/json", + } + assert r.read() == b'{"msg":"Hello, world"}' + + +def test_request_empty_post(): + r = httpx.Request("POST", "https://example.com") + + assert repr(r) == "" + assert r.method == "POST" + assert r.url == "https://example.com" + assert r.headers == { + "Host": "example.com", + "Content-Length": "0", + } + assert r.read() == b'' diff --git a/tests/test_response.py b/tests/test_response.py new file mode 100644 index 0000000000..d25ebeb211 --- /dev/null +++ b/tests/test_response.py @@ -0,0 +1,64 @@ +import httpx + + +class ByteIterator: + def __init__(self, buffer=b""): + self._buffer = buffer + + def next(self) -> bytes: + buffer = self._buffer + self._buffer = b'' + return buffer + + +def test_response(): + r = httpx.Response(200) + + assert repr(r) == "" + assert r.status_code == 200 + assert r.headers == {'Content-Length': '0'} + assert r.read() == b"" + + +def test_response_204(): + r = httpx.Response(204) + + assert repr(r) == "" + assert r.status_code == 204 + assert r.headers == {} + assert r.read() == b"" + + +def test_response_bytes(): + content = b"Hello, world" + r = httpx.Response(200, content=content) + + assert repr(r) == "" + assert r.headers == { + "Content-Length": "12", + } + assert r.read() == b"Hello, world" + + +def test_response_stream(): + i = ByteIterator(b"Hello, world") + stream = httpx.HTTPStream(i.next, None) + r = httpx.Response(200, content=stream) + + assert repr(r) == "" + assert r.headers == { + "Transfer-Encoding": "chunked", + } + assert r.read() == b"Hello, world" + + +def test_response_json(): + data = httpx.JSON({"msg": "Hello, world"}) + r = httpx.Response(200, content=data) + + assert repr(r) == "" + assert r.headers == { + "Content-Length": "22", + "Content-Type": "application/json", + } + assert r.read() == b'{"msg":"Hello, world"}' diff --git a/tests/test_status_codes.py b/tests/test_status_codes.py deleted file mode 100644 index 13314db788..0000000000 --- a/tests/test_status_codes.py +++ /dev/null @@ -1,27 +0,0 @@ -import httpx - - -def test_status_code_as_int(): - # mypy doesn't (yet) recognize that IntEnum members are ints, so ignore it here - assert httpx.codes.NOT_FOUND == 404 # type: ignore[comparison-overlap] - assert str(httpx.codes.NOT_FOUND) == "404" - - -def test_status_code_value_lookup(): - assert httpx.codes(404) == 404 - - -def test_status_code_phrase_lookup(): - assert httpx.codes["NOT_FOUND"] == 404 - - -def test_lowercase_status_code(): - assert httpx.codes.not_found == 404 # type: ignore - - -def test_reason_phrase_for_status_code(): - assert httpx.codes.get_reason_phrase(404) == "Not Found" - - -def test_reason_phrase_for_unknown_status_code(): - assert httpx.codes.get_reason_phrase(499) == "" diff --git a/tests/test_streams.py b/tests/test_streams.py new file mode 100644 index 0000000000..70c7244099 --- /dev/null +++ b/tests/test_streams.py @@ -0,0 +1,82 @@ +import pytest +import httpx + + +def test_stream(): + i = httpx.Stream() + with pytest.raises(NotImplementedError): + i.read() + + with pytest.raises(NotImplementedError): + i.close() + + i.size == None + + +def test_bytestream(): + data = b'abc' + s = httpx.ByteStream(data) + assert s.size == 3 + assert s.read() == b'abc' + + s = httpx.ByteStream(data) + assert s.read(1) == b'a' + assert s.read(1) == b'b' + assert s.read(1) == b'c' + assert s.read(1) == b'' + + +def test_filestream(tmp_path): + path = tmp_path / "example.txt" + path.write_bytes(b"hello world") + + with httpx.File(path).encode() as s: + assert s.size == 11 + assert s.read() == b'hello world' + + with httpx.File(path).encode() as s: + assert s.read(5) == b'hello' + assert s.read(5) == b' worl' + assert s.read(5) == b'd' + assert s.read(5) == b'' + + with httpx.File(path).encode() as s: + assert s.read(5) == b'hello' + + + +def test_multipartstream(tmp_path): + path = tmp_path / 'example.txt' + path.write_bytes(b'hello world' + b'x' * 50) + + expected = b''.join([ + b'--boundary\r\n', + b'Content-Disposition: form-data; name="email"\r\n', + b'\r\n', + b'heya@example.com\r\n', + b'--boundary\r\n', + b'Content-Disposition: form-data; name="upload"; filename="example.txt"\r\n', + b'\r\n', + b'hello world' + ( b'x' * 50) + b'\r\n', + b'--boundary--\r\n', + ]) + + form = [('email', 'heya@example.com')] + files = [('upload', str(path))] + with httpx.MultiPartStream(form, files, boundary='boundary') as s: + assert s.size is None + assert s.read() == expected + + with httpx.MultiPartStream(form, files, boundary='boundary') as s: + assert s.read(50) == expected[:50] + assert s.read(50) == expected[50:100] + assert s.read(50) == expected[100:150] + assert s.read(50) == expected[150:200] + assert s.read(50) == expected[200:250] + + with httpx.MultiPartStream(form, files, boundary='boundary') as s: + assert s.read(50) == expected[:50] + assert s.read(50) == expected[50:100] + assert s.read(50) == expected[100:150] + assert s.read(50) == expected[150:200] + s.close() # test close during open file diff --git a/tests/test_timeouts.py b/tests/test_timeouts.py deleted file mode 100644 index 666cc8e376..0000000000 --- a/tests/test_timeouts.py +++ /dev/null @@ -1,55 +0,0 @@ -import pytest - -import httpx - - -@pytest.mark.anyio -async def test_read_timeout(server): - timeout = httpx.Timeout(None, read=1e-6) - - async with httpx.AsyncClient(timeout=timeout) as client: - with pytest.raises(httpx.ReadTimeout): - await client.get(server.url.copy_with(path="/slow_response")) - - -@pytest.mark.anyio -async def test_write_timeout(server): - timeout = httpx.Timeout(None, write=1e-6) - - async with httpx.AsyncClient(timeout=timeout) as client: - with pytest.raises(httpx.WriteTimeout): - data = b"*" * 1024 * 1024 * 100 - await client.put(server.url.copy_with(path="/slow_response"), content=data) - - -@pytest.mark.anyio -@pytest.mark.network -async def test_connect_timeout(server): - timeout = httpx.Timeout(None, connect=1e-6) - - async with httpx.AsyncClient(timeout=timeout) as client: - with pytest.raises(httpx.ConnectTimeout): - # See https://stackoverflow.com/questions/100841/ - await client.get("http://10.255.255.1/") - - -@pytest.mark.anyio -async def test_pool_timeout(server): - limits = httpx.Limits(max_connections=1) - timeout = httpx.Timeout(None, pool=1e-4) - - async with httpx.AsyncClient(limits=limits, timeout=timeout) as client: - with pytest.raises(httpx.PoolTimeout): - async with client.stream("GET", server.url): - await client.get(server.url) - - -@pytest.mark.anyio -async def test_async_client_new_request_send_timeout(server): - timeout = httpx.Timeout(1e-6) - - async with httpx.AsyncClient(timeout=timeout) as client: - with pytest.raises(httpx.TimeoutException): - await client.send( - httpx.Request("GET", server.url.copy_with(path="/slow_response")) - ) diff --git a/tests/test_urlencode.py b/tests/test_urlencode.py new file mode 100644 index 0000000000..42ba45ac37 --- /dev/null +++ b/tests/test_urlencode.py @@ -0,0 +1,33 @@ +import httpx + + +def test_urlencode(): + qs = "a=name%40example.com&a=456&b=7+8+9&c" + d = httpx.urldecode(qs) + assert d == { + "a": ["name@example.com", "456"], + "b": ["7 8 9"], + "c": [""] + } + + +def test_urldecode(): + d = { + "a": ["name@example.com", "456"], + "b": ["7 8 9"], + "c": [""] + } + qs = httpx.urlencode(d) + assert qs == "a=name%40example.com&a=456&b=7+8+9&c=" + + +def test_urlencode_empty(): + qs = "" + d = httpx.urldecode(qs) + assert d == {} + + +def test_urldecode_empty(): + d = {} + qs = httpx.urlencode(d) + assert qs == "" diff --git a/tests/test_urls.py b/tests/test_urls.py new file mode 100644 index 0000000000..ad72935273 --- /dev/null +++ b/tests/test_urls.py @@ -0,0 +1,164 @@ +import httpx +import pytest + + +def test_url(): + url = httpx.URL('https://www.example.com/') + assert str(url) == "https://www.example.com/" + + +def test_url_repr(): + url = httpx.URL('https://www.example.com/') + assert repr(url) == "" + + +def test_url_params(): + url = httpx.URL('https://www.example.com/', params={"a": "b", "c": "d"}) + assert str(url) == "https://www.example.com/?a=b&c=d" + + +def test_url_normalisation(): + url = httpx.URL('https://www.EXAMPLE.com:443/path/../main') + assert str(url) == 'https://www.example.com/main' + + +def test_url_relative(): + url = httpx.URL('/README.md') + assert str(url) == '/README.md' + + +def test_url_escaping(): + url = httpx.URL('https://example.com/path to here?search=🦋') + assert str(url) == 'https://example.com/path%20to%20here?search=%F0%9F%A6%8B' + + +def test_url_components(): + url = httpx.URL(scheme="https", host="example.com", path="/") + assert str(url) == 'https://example.com/' + + +# QueryParams + +def test_queryparams(): + params = httpx.QueryParams({"color": "black", "size": "medium"}) + assert str(params) == 'color=black&size=medium' + + +def test_queryparams_repr(): + params = httpx.QueryParams({"color": "black", "size": "medium"}) + assert repr(params) == "" + + +def test_queryparams_list_of_values(): + params = httpx.QueryParams({"filter": ["60GHz", "75GHz", "100GHz"]}) + assert str(params) == 'filter=60GHz&filter=75GHz&filter=100GHz' + + +def test_queryparams_from_str(): + params = httpx.QueryParams("color=black&size=medium") + assert str(params) == 'color=black&size=medium' + + +def test_queryparams_access(): + params = httpx.QueryParams("sort_by=published&author=natalie") + assert params["sort_by"] == 'published' + + +def test_queryparams_escaping(): + params = httpx.QueryParams({"email": "user@example.com", "search": "How HTTP works!"}) + assert str(params) == 'email=user%40example.com&search=How+HTTP+works%21' + + +def test_queryparams_empty(): + q = httpx.QueryParams({"a": ""}) + assert str(q) == "a=" + + q = httpx.QueryParams("a=") + assert str(q) == "a=" + + q = httpx.QueryParams("a") + assert str(q) == "a=" + + +def test_queryparams_set(): + q = httpx.QueryParams("a=123") + q = q.copy_set("a", "456") + assert q == httpx.QueryParams("a=456") + + +def test_queryparams_append(): + q = httpx.QueryParams("a=123") + q = q.copy_append("a", "456") + assert q == httpx.QueryParams("a=123&a=456") + + +def test_queryparams_remove(): + q = httpx.QueryParams("a=123") + q = q.copy_remove("a") + assert q == httpx.QueryParams("") + + +def test_queryparams_merge(): + q = httpx.QueryParams("a=123") + q = q.copy_update({"b": "456"}) + assert q == httpx.QueryParams("a=123&b=456") + q = q.copy_update({"a": "000", "c": "789"}) + assert q == httpx.QueryParams("a=000&b=456&c=789") + + +def test_queryparams_are_hashable(): + params = ( + httpx.QueryParams("a=123"), + httpx.QueryParams({"a": "123"}), + httpx.QueryParams("b=456"), + httpx.QueryParams({"b": "456"}), + ) + + assert len(set(params)) == 2 + + +@pytest.mark.parametrize( + "source", + [ + "a=123&a=456&b=789", + {"a": ["123", "456"], "b": "789"}, + {"a": ("123", "456"), "b": "789"}, + [("a", "123"), ("a", "456"), ("b", "789")], + (("a", "123"), ("a", "456"), ("b", "789")), + ], +) +def test_queryparams_misc(source): + q = httpx.QueryParams(source) + assert "a" in q + assert "A" not in q + assert "c" not in q + assert q["a"] == "123" + assert q.get("a") == "123" + assert q.get("nope", default=None) is None + assert q.get_list("a") == ["123", "456"] + assert bool(q) + + assert list(q.keys()) == ["a", "b"] + assert list(q.values()) == ["123", "789"] + assert list(q.items()) == [("a", "123"), ("b", "789")] + assert len(q) == 2 + assert list(q) == ["a", "b"] + assert dict(q) == {"a": "123", "b": "789"} + assert str(q) == "a=123&a=456&b=789" + assert httpx.QueryParams({"a": "123", "b": "456"}) == httpx.QueryParams( + [("a", "123"), ("b", "456")] + ) + assert httpx.QueryParams({"a": "123", "b": "456"}) == httpx.QueryParams( + "a=123&b=456" + ) + assert httpx.QueryParams({"a": "123", "b": "456"}) == httpx.QueryParams( + {"b": "456", "a": "123"} + ) + assert httpx.QueryParams() == httpx.QueryParams({}) + assert httpx.QueryParams([("a", "123"), ("a", "456")]) == httpx.QueryParams( + "a=123&a=456" + ) + assert httpx.QueryParams({"a": "123", "b": "456"}) != "invalid" + + q = httpx.QueryParams([("a", "123"), ("a", "456")]) + assert httpx.QueryParams(q) == q diff --git a/tests/test_utils.py b/tests/test_utils.py deleted file mode 100644 index f9c215f65a..0000000000 --- a/tests/test_utils.py +++ /dev/null @@ -1,150 +0,0 @@ -import json -import logging -import os -import random - -import pytest - -import httpx -from httpx._utils import URLPattern, get_environment_proxies - - -@pytest.mark.parametrize( - "encoding", - ( - "utf-32", - "utf-8-sig", - "utf-16", - "utf-8", - "utf-16-be", - "utf-16-le", - "utf-32-be", - "utf-32-le", - ), -) -def test_encoded(encoding): - content = '{"abc": 123}'.encode(encoding) - response = httpx.Response(200, content=content) - assert response.json() == {"abc": 123} - - -def test_bad_utf_like_encoding(): - content = b"\x00\x00\x00\x00" - response = httpx.Response(200, content=content) - with pytest.raises(json.decoder.JSONDecodeError): - response.json() - - -@pytest.mark.parametrize( - ("encoding", "expected"), - ( - ("utf-16-be", "utf-16"), - ("utf-16-le", "utf-16"), - ("utf-32-be", "utf-32"), - ("utf-32-le", "utf-32"), - ), -) -def test_guess_by_bom(encoding, expected): - content = '\ufeff{"abc": 123}'.encode(encoding) - response = httpx.Response(200, content=content) - assert response.json() == {"abc": 123} - - -def test_logging_request(server, caplog): - caplog.set_level(logging.INFO) - with httpx.Client() as client: - response = client.get(server.url) - assert response.status_code == 200 - - assert caplog.record_tuples == [ - ( - "httpx", - logging.INFO, - 'HTTP Request: GET http://127.0.0.1:8000/ "HTTP/1.1 200 OK"', - ) - ] - - -def test_logging_redirect_chain(server, caplog): - caplog.set_level(logging.INFO) - with httpx.Client(follow_redirects=True) as client: - response = client.get(server.url.copy_with(path="/redirect_301")) - assert response.status_code == 200 - - assert caplog.record_tuples == [ - ( - "httpx", - logging.INFO, - "HTTP Request: GET http://127.0.0.1:8000/redirect_301" - ' "HTTP/1.1 301 Moved Permanently"', - ), - ( - "httpx", - logging.INFO, - 'HTTP Request: GET http://127.0.0.1:8000/ "HTTP/1.1 200 OK"', - ), - ] - - -@pytest.mark.parametrize( - ["environment", "proxies"], - [ - ({}, {}), - ({"HTTP_PROXY": "http://127.0.0.1"}, {"http://": "http://127.0.0.1"}), - ( - {"https_proxy": "http://127.0.0.1", "HTTP_PROXY": "https://127.0.0.1"}, - {"https://": "http://127.0.0.1", "http://": "https://127.0.0.1"}, - ), - ({"all_proxy": "http://127.0.0.1"}, {"all://": "http://127.0.0.1"}), - ({"TRAVIS_APT_PROXY": "http://127.0.0.1"}, {}), - ({"no_proxy": "127.0.0.1"}, {"all://127.0.0.1": None}), - ({"no_proxy": "192.168.0.0/16"}, {"all://192.168.0.0/16": None}), - ({"no_proxy": "::1"}, {"all://[::1]": None}), - ({"no_proxy": "localhost"}, {"all://localhost": None}), - ({"no_proxy": "github.com"}, {"all://*github.com": None}), - ({"no_proxy": ".github.com"}, {"all://*.github.com": None}), - ({"no_proxy": "http://github.com"}, {"http://github.com": None}), - ], -) -def test_get_environment_proxies(environment, proxies): - os.environ.update(environment) - - assert get_environment_proxies() == proxies - - -@pytest.mark.parametrize( - ["pattern", "url", "expected"], - [ - ("http://example.com", "http://example.com", True), - ("http://example.com", "https://example.com", False), - ("http://example.com", "http://other.com", False), - ("http://example.com:123", "http://example.com:123", True), - ("http://example.com:123", "http://example.com:456", False), - ("http://example.com:123", "http://example.com", False), - ("all://example.com", "http://example.com", True), - ("all://example.com", "https://example.com", True), - ("http://", "http://example.com", True), - ("http://", "https://example.com", False), - ("all://", "https://example.com:123", True), - ("", "https://example.com:123", True), - ], -) -def test_url_matches(pattern, url, expected): - pattern = URLPattern(pattern) - assert pattern.matches(httpx.URL(url)) == expected - - -def test_pattern_priority(): - matchers = [ - URLPattern("all://"), - URLPattern("http://"), - URLPattern("http://example.com"), - URLPattern("http://example.com:123"), - ] - random.shuffle(matchers) - assert sorted(matchers) == [ - URLPattern("http://example.com:123"), - URLPattern("http://example.com"), - URLPattern("http://"), - URLPattern("all://"), - ] diff --git a/tests/test_wsgi.py b/tests/test_wsgi.py deleted file mode 100644 index dc2b52885a..0000000000 --- a/tests/test_wsgi.py +++ /dev/null @@ -1,203 +0,0 @@ -from __future__ import annotations - -import sys -import typing -import wsgiref.validate -from functools import partial -from io import StringIO - -import pytest - -import httpx - -if typing.TYPE_CHECKING: # pragma: no cover - from _typeshed.wsgi import StartResponse, WSGIApplication, WSGIEnvironment - - -def application_factory(output: typing.Iterable[bytes]) -> WSGIApplication: - def application(environ, start_response): - status = "200 OK" - - response_headers = [ - ("Content-type", "text/plain"), - ] - - start_response(status, response_headers) - - for item in output: - yield item - - return wsgiref.validate.validator(application) - - -def echo_body( - environ: WSGIEnvironment, start_response: StartResponse -) -> typing.Iterable[bytes]: - status = "200 OK" - output = environ["wsgi.input"].read() - - response_headers = [ - ("Content-type", "text/plain"), - ] - - start_response(status, response_headers) - - return [output] - - -def echo_body_with_response_stream( - environ: WSGIEnvironment, start_response: StartResponse -) -> typing.Iterable[bytes]: - status = "200 OK" - - response_headers = [("Content-Type", "text/plain")] - - start_response(status, response_headers) - - def output_generator(f: typing.IO[bytes]) -> typing.Iterator[bytes]: - while True: - output = f.read(2) - if not output: - break - yield output - - return output_generator(f=environ["wsgi.input"]) - - -def raise_exc( - environ: WSGIEnvironment, - start_response: StartResponse, - exc: type[Exception] = ValueError, -) -> typing.Iterable[bytes]: - status = "500 Server Error" - output = b"Nope!" - - response_headers = [ - ("Content-type", "text/plain"), - ] - - try: - raise exc() - except exc: - exc_info = sys.exc_info() - start_response(status, response_headers, exc_info) - - return [output] - - -def log_to_wsgi_log_buffer(environ, start_response): - print("test1", file=environ["wsgi.errors"]) - environ["wsgi.errors"].write("test2") - return echo_body(environ, start_response) - - -def test_wsgi(): - transport = httpx.WSGITransport(app=application_factory([b"Hello, World!"])) - client = httpx.Client(transport=transport) - response = client.get("http://www.example.org/") - assert response.status_code == 200 - assert response.text == "Hello, World!" - - -def test_wsgi_upload(): - transport = httpx.WSGITransport(app=echo_body) - client = httpx.Client(transport=transport) - response = client.post("http://www.example.org/", content=b"example") - assert response.status_code == 200 - assert response.text == "example" - - -def test_wsgi_upload_with_response_stream(): - transport = httpx.WSGITransport(app=echo_body_with_response_stream) - client = httpx.Client(transport=transport) - response = client.post("http://www.example.org/", content=b"example") - assert response.status_code == 200 - assert response.text == "example" - - -def test_wsgi_exc(): - transport = httpx.WSGITransport(app=raise_exc) - client = httpx.Client(transport=transport) - with pytest.raises(ValueError): - client.get("http://www.example.org/") - - -def test_wsgi_http_error(): - transport = httpx.WSGITransport(app=partial(raise_exc, exc=RuntimeError)) - client = httpx.Client(transport=transport) - with pytest.raises(RuntimeError): - client.get("http://www.example.org/") - - -def test_wsgi_generator(): - output = [b"", b"", b"Some content", b" and more content"] - transport = httpx.WSGITransport(app=application_factory(output)) - client = httpx.Client(transport=transport) - response = client.get("http://www.example.org/") - assert response.status_code == 200 - assert response.text == "Some content and more content" - - -def test_wsgi_generator_empty(): - output = [b"", b"", b"", b""] - transport = httpx.WSGITransport(app=application_factory(output)) - client = httpx.Client(transport=transport) - response = client.get("http://www.example.org/") - assert response.status_code == 200 - assert response.text == "" - - -def test_logging(): - buffer = StringIO() - transport = httpx.WSGITransport(app=log_to_wsgi_log_buffer, wsgi_errors=buffer) - client = httpx.Client(transport=transport) - response = client.post("http://www.example.org/", content=b"example") - assert response.status_code == 200 # no errors - buffer.seek(0) - assert buffer.read() == "test1\ntest2" - - -@pytest.mark.parametrize( - "url, expected_server_port", - [ - pytest.param("http://www.example.org", "80", id="auto-http"), - pytest.param("https://www.example.org", "443", id="auto-https"), - pytest.param("http://www.example.org:8000", "8000", id="explicit-port"), - ], -) -def test_wsgi_server_port(url: str, expected_server_port: str) -> None: - """ - SERVER_PORT is populated correctly from the requested URL. - """ - hello_world_app = application_factory([b"Hello, World!"]) - server_port: str | None = None - - def app(environ, start_response): - nonlocal server_port - server_port = environ["SERVER_PORT"] - return hello_world_app(environ, start_response) - - transport = httpx.WSGITransport(app=app) - client = httpx.Client(transport=transport) - response = client.get(url) - assert response.status_code == 200 - assert response.text == "Hello, World!" - assert server_port == expected_server_port - - -def test_wsgi_server_protocol(): - server_protocol = None - - def app(environ, start_response): - nonlocal server_protocol - server_protocol = environ["SERVER_PROTOCOL"] - start_response("200 OK", [("Content-Type", "text/plain")]) - return [b"success"] - - transport = httpx.WSGITransport(app=app) - with httpx.Client(transport=transport, base_url="http://testserver") as client: - response = client.get("/") - - assert response.status_code == 200 - assert response.text == "success" - assert server_protocol == "HTTP/1.1"