diff --git a/.agents/docs/design-cancel-stop-token.md b/.agents/docs/design-cancel-stop-token.md new file mode 100644 index 0000000..671a1fa --- /dev/null +++ b/.agents/docs/design-cancel-stop-token.md @@ -0,0 +1,216 @@ +# 设计方案:请求取消(`std::stop_token`),基于 PR #23 + +| | | +|---|---| +| 基线 | `origin/master @ f461940`(0.3.3) | +| 输入 | [PR #23](https://github.com/mcpplibs/tinyhttps/pull/23) `feat/cancel @ 8996984`(作者 yspbwx2010,`maintainerCanModify: true`) | +| 落地方式 | 在 PR #23 上追加 commit,不另开 PR | +| 文档日期 | 2026-10-02 | + +**标注约定**:✅ 已核验源码 · 🔁 已在本地复现或跑通 · ❓ 推断,未实测 + +--- + +## 0. TL;DR + +- **应该进库。** 库外做不到:socket 是 `HttpClient` 的私有状态,`HttpClient` 本身又不是线程安全的。库里也已经有半个先例:`download_to_file` 的 `isCancelled` 只在 body 的块与块之间检查。 +- **PR #23 的机制是对的,保留:** 等待只发生在 `Socket::wait_fd`,有 token 时按 50 ms 分片 `poll`;`bio_recv` 读之前先等。**它缺的是 token 的生命周期规则**,H1 就是这个缺口造成的 bug:经 https 代理复用连接时,一个 POST 会被执行两次。 +- **架构收敛成三条不变式:** token 只放在传输链最底层的 `Socket` 上;token 只在调用期间挂着,连接回池时摘掉;不传 token 时行为和 0.3.3 完全一样。 +- **改动量:** 在 PR 基础上,`src/` 约 +15 行(必需部分 +7 行),另加测试和 CI。必需部分已经做成原型跑通(§7)。 +- **CI 目前对这个功能只有一个配置有效:** Linux + gcc 16。Windows job 不编译 `test_cancel`;Linux libc++(llvm 20/22)连链接都过不了;没有 macOS job。 + +### 决策(2026-10-02 均按建议采纳,执行见 [exec-plan-0.3.4-cancel.md](exec-plan-0.3.4-cancel.md)) + +| # | 决策 | 建议 | +|---|---|---| +| D1 | `download_to_file` 是否同时加 `std::stop_token` 和 `DownloadToFileResult::cancelled` | **加**。理由见 §3.3 | +| D2 | 删掉 PR 给 `proxy_connect` 加的 `stop` 参数 | **删**。它是遗留的包装函数,只有 `proxy_tunnel` 内部需要 token | +| D3 | CI 新增 Linux llvm@22 job、Windows 跑 hermetic 测试 | **加**。macOS 视 mcpp 支持情况定(❓) | +| D4 | 写操作阻塞(对端不读)仍不可取消 | **本 PR 不做**,列为已知限制。后续单独做非阻塞发送 | + +--- + +## 1. 该不该进库 + +**结论:进。** + +| 论点 | 说明 | +|---|---| +| 库外无解 | 调用方拿不到 fd。唯一的绕法是把 `send` 丢到 detached 线程里等 `readTimeoutMs`。这期间 `HttpClient` 不能再用(不是线程安全的,`http.cppm` 里 `HttpClient` 的注释写明了),连接也一直占着,服务器还在继续生成响应 | +| 已有先例 | `download_to_file(..., isCancelled)` ✅ 说明库已经承认取消是它的职责。但 `isCancelled` 只在 body 块之间检查,等响应头、TLS 握手这些阶段照样卡死,正是 PR 要解决的问题 | +| 标准词汇 | `std::stop_token` 是 C++20 标准类型,和 `std::jthread` 天然配合,今后也能对接 `std::execution` 的 stop token。不需要自造回调或传 `atomic*` | +| 成本可控 | 不传 token 时零开销(I3)。传了 token 时,每个进行中的调用每秒多 20 次 `poll` 唤醒 | + +**反方意见及回应:**“tiny 库不该长功能”。这里不是加功能,而是补全已有的取消语义(`isCancelled` 覆盖不全),并让三个入口一致。这和 #15 合并三个入口、只保留一份读取逻辑是同一个思路。 + +--- + +## 2. 架构:三条不变式 + +``` +HttpClient::send / send_stream / download_to_file ← 唯一接收 token 的地方 + │ perform_exchange: sock->set_stop(stop) ← 挂上(每次调用、每次重试) + │ PooledConnection::keep(): set_stop({}) ← 回池时摘下;drop() 直接销毁 + ▼ +TlsSocket::set_stop ──转发──► lower_(https 代理隧道)──转发──► Socket + │ 自己不存 token + ▼ +Socket::stop_ ── wait_fd():分片 poll,片与片之间检查 stop_requested() + ── bio_recv:有 token 时先 wait_readable(-1) 再 recv + ── connect:每个地址、每次 DNS 回退前检查 +``` + +| 不变式 | 内容 | 违反时的后果 | +|---|---|---| +| **I1 只存在底层** | token 只存在于传输链最底层的 `Socket`。`TlsSocket` 不存 token,只往下转发(不管下面是 `socket_` 还是 `lower_`) | **H1** 🔁:https 代理下 `set_stop` 只设了用不到的 `socket_`,复用的隧道一直带着第一个请求的 token | +| **I2 只在调用期间存在** | 进入 exchange 时挂上,`keep()` 把连接还回池时摘掉。**连接池里的连接永远不带 token** | **M1**:直连情况下现在不会出错(下次请求会覆盖),但只要以后加一个在挂 token 之前等待池中连接的路径(比如空闲探活),就会踩到和 H1 一样的坑 | +| **I3 没有 token 时行为不变** | `stop_possible()==false` 时 `wait_fd` 就是原来的单次 `poll_fd`,`bio_recv` 不预先等待 | 已满足 ✅ | + +**为什么 token 必须作为状态存在 `Socket` 上,而不是作为参数一路传下去:** mbedtls 的 BIO 回调只有一个 `void* ctx`,在这里就是 `Socket*` ✅。握手和 TLS record 的后半段都是从 `bio_recv` 里阻塞的,没有别的通道能把 token 送进去。既然状态没法避免,就用 I1 和 I2 限定它在哪里、活多久。 + +### 考虑过、没有采用的方案 + +| 方案 | 不采用的原因 | +|---|---| +| `HttpClient::cancel()` 成员函数 | 需要从别的线程访问连接池,而 `HttpClient` 不是线程安全的。token 是不加锁就能跨线程传递的唯一渠道 | +| `std::stop_callback` + `shutdown(fd)` | 优点:没有轮询,还能覆盖阻塞的写(D4)。缺点:回调在发起 stop 的线程上执行,会和调用线程里 `drop()` 等所有关闭 fd 的路径竞争,必须加锁;另外 connect 进行中被 `shutdown` 时,macOS、Windows、openkal 上的语义都没有验证 ❓。**留作解决 D4 时的候选方案** | +| 第二个 fd(wake-up pipe)加 `poll` | Windows 没有 pipe;PR 作者说 openkal 上两个 fd 的 `poll` 会忙等(❓,未独立核实) | + +--- + +## 3. 在 PR #23 基础上:保留、修改、删除 + +### 3.1 保留(✅ 已逐项核验) + +- `Socket::wait_fd` 的分片 `poll`(50 ms,固定值,不做成配置项);`bio_recv` 的预等待;connect 前、每个地址、DNS 回退前的 stop 检查。 +- `perform_exchange` 的 `fail()`:stop 优先于真实错误,并在这里 drop 连接;重试循环开头检查 stop;取消后不跟重定向。 +- `HttpResponse::cancelled`,以及它在响应头到达前后的两种形态(见 §4)。 +- `proxy_tunnel` 尾部的默认参数 `stop`(内部需要)。 +- `Socket` 的 move 会带走 `stop_`;`close()` 不清 `stop_`,所以 `set_stop` 必须在 `connect` 之前调用,现在的顺序是对的 ✅。 + +### 3.2 修改 + +| 项 | 改动 | 行数 | 状态 | +|---|---|---|---| +| **H1** | `TlsSocket::set_stop` 先转发给 `lower_`,再设 `socket_`(I1) | +3 | 🔁 原型通过 | +| **M1** | `PooledConnection::keep()` 把池中连接的 token 设为 `{}`(I2)。`drop()` 直接销毁连接,不用处理 | +5 | 🔁 原型通过 | +| **M2** | `test_cancel.cpp` 里 `import std` 的位置按标准库区分(§6.2) | +6 | 🔁 llvm 22 和 gcc 16 都通过 | +| 文档 | CHANGELOG 的 “Existing callers are unaffected” 改为:普通调用不受影响,但取 `&HttpClient::send` 成员函数指针的代码需要调整 | — | — | + +### 3.3 D1:`download_to_file` 加 token(建议) + +三个入口共用 `perform_exchange` 和 `read_body` ✅。PR 只改了其中两个入口,`download_to_file` 仍然只有 `isCancelled`,又回到了 #15 那张“修复只落到部分入口”的矩阵。具体改法: + +- 签名加尾部默认参数:`download_to_file(url, dest, onProgress = nullptr, isCancelled = nullptr, std::stop_token stop = {})`。 +- `DownloadToFileResult::cancelled`:`isCancelled` 或 token 任一触发都置为 true。`error` 文本保持现有的 `"cancelled"` 不变,原来匹配文本的代码不受影响。 +- 实现:`perform_exchange(..., stop)`;body 的 sink 里检查 stop;重定向递归时把 stop 传下去。预计约 +15 行。 +- `isCancelled` 保留,不标记废弃。它是轮询式谓词,和 token 不冲突。 + +### 3.4 D2:删除(建议) + +- `proxy_connect(..., std::stop_token)`:这是只返回 `Socket` 的遗留包装函数,库内部不用它。加参数只会扩大导出面(YAGNI)。 + +--- + +## 4. 语义契约(README 和 CHANGELOG 要写清楚) + +| 情形 | `statusCode` / `statusText` | `bodyComplete` / `bodyError` | `cancelled` | 连接 | +|---|---|---|---|---| +| 调用开始前 token 已经 stop | `0` / `"Cancelled"` | false / `"Cancelled"` | true | 不发起任何连接 | +| 响应头到达前取消(connect、代理、握手、等响应头) | `0` / `"Cancelled"` | false / `"Cancelled"` | true | drop | +| 响应头到达后取消(读 body、流式回调间隙) | 服务器给的值 | false / `"cancelled"` | true | drop | +| token 从未 stop | 和 0.3.3 一样 | 和 0.3.3 一样 | false | 和 0.3.3 一样 | + +其他规则: + +- `ok()` 不看 `cancelled`,就像它也不看 `bodyComplete`。取消的契约是那个 bool,不是文本。 +- 取消后不做 stale-connection 重试,也不跟重定向。 +- 真实失败和 stop 同时发生时,报告为取消。 +- 响应时间约 50 ms(一个分片)。 +- `close_notify` 只在握手完成之后的取消中发送。 + +**已知限制:**`getaddrinfo` 和 `socks5://` 对目标地址的解析不可中断;对端不读导致的阻塞写不可中断(socket 在 connect 之后被设回阻塞模式,`socket.cppm:257` ✅,见 D4);openkal 上 connect 是同步完成的。 + +--- + +## 5. 测试 + +PR 已有 13 个用例,覆盖各阶段的首次建连 ✅。**缺口在“复用连接”这一维度**,H1 就漏在这里。补成下面这个矩阵: + +| 新增用例 | 覆盖 | 状态 | +|---|---|---| +| 经 https 代理复用:第一个请求结束后它的 token 被 stop,第二个 POST 在目标服务器上**只执行一次**,代理只被连一次 | H1 / I1 | 🔁 修复前 3 次请求、2 次代理连接;修复后通过 | +| 经 https 代理复用:第二个请求自己的 token 在 500 ms 内生效 | H1 / I1 | 🔁 修复前要等满 3000 ms 读超时 | +| 同样的场景走 http:// 代理和直连(对照组) | I1 | 🔁 通过 | +| README 的 jthread 写法:请求完成后 jthread 析构(会 `request_stop`),下一个请求复用连接并成功 | I2 | 🔁(https 代理那一组就是这个写法) | +| `download_to_file`:等响应头时取消、读 body 时取消;`cancelled==true`;目标文件的处理和现有 `isCancelled` 一致 | D1 | 待写 | +| SOCKS5 代理在等握手应答时取消 | 补齐代理种类 | 待写(PR 只覆盖了 http 和 https 代理) | + +时间断言使用 `kPromptMs = 500`。Windows 和 macOS 的 runner 比较慢,如果出现偶发失败,再按平台放宽,不要预先放宽。 + +--- + +## 6. CI 覆盖 + +### 6.1 现状 ✅ + +| job | 工具链 | 跑什么 | 对本功能的覆盖 | +|---|---|---|---| +| linux mcpp | gcc 16.1.0(mcpp 首次运行时引导安装的默认工具链) | 全部 `mcpp test` | ✅ 唯一有效的覆盖 | +| openkal | llvm + openkal libc++ | `examples/openkal` 的 `mcpp run` | 只编译库,不跑 `test_cancel` | +| windows msvc / llvm@20.1.7 | MSVC STL | **只跑 `mcpp test test_ca_store`** | ❌ 根本不编译 `test_cancel` | +| linux libc++ | — | — | ❌ 而且 `test_cancel` 在 llvm 20/22 上链接失败(§6.2) | +| macOS | — | — | ❌ 但 README 声称支持 macOS | + +### 6.2 libc++ 链接问题(M2)🔁 + +- 在 llvm 20.1.7 和 22.1.8(libc++)上,先 `#include ` 再 `import std;`,然后调用 `stop_source::request_stop()`,链接时报 `undefined hidden symbol __atomic_unique_lock<...>::__set_locked_bit`。llvm 23.1.0 和 gcc 16 正常。这是工具链 bug,不是本库的问题。 +- 仓库测试文件的惯例是先 include 后 import,因为 gcc 必须这样:反过来在 gcc 16 上会报 `redefinition of std::byte` 等错误 🔁。 +- 解决办法:在 `test_cancel.cpp` 里按标准库区分顺序(已验证 llvm 22 和 gcc 16 都能 16/16 通过): + +```cpp +// libc++ (≤22) fails to link stop_source::request_stop() when +// was included textually before `import std`; libstdc++ needs the opposite. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif +``` + +- llvm 20 在 Linux 上两种顺序都编不过(import 在前时报 `requires clause differs`)🔁,显式实例化 `__atomic_unique_lock` 也不行 🔁,因此 **libc++ 只把 22 纳入 CI 矩阵**(Linux 和 macOS 都用 llvm@22.1.8;Windows 上的 llvm@20.1.7 用的是 MSVC STL,不受影响)。 +- README 的 jthread 示例对“include 和 import std 混用、用 libc++ ≤22”的使用者同样会链接失败,需要在 README 里加一行说明。 + +### 6.3 目标 + +| 改动 | 内容 | 状态 | +|---|---|---| +| Linux job 改成矩阵 `[gcc@16.1.0, llvm@22.1.8]`,都跑全部 `mcpp test` | 覆盖 libc++。llvm 22 也是 mcpp 本地常见的默认工具链 | 🔁 本地 llvm 22 + M2 修复通过 | +| Windows 两个 job 追加 `mcpp test test_cancel`,最好再加 `test_framing`、`test_pool`、`test_proxy` | 覆盖 `WSAPoll` 分片和 Windows 上 connect 的等待 | ❓ 测试服务器有 `_WIN32` 分支(`proxy_test_server.hpp:28`),但从没在 Windows 上编译过,先试,编不过再移植 | +| 新增 macOS job(macos-14),跑 hermetic 测试 | PR 作者明确说没在 macOS 上跑过;macOS 上 `poll` 对进行中的 connect 的语义需要实测 | ❓ 要先确认 mcpp 的 macOS 工具链可用。如果不可用,就作为 follow-up 单独开 issue,不阻塞本 PR | +| openkal | 不变。库被编进 example,签名的编译已经覆盖到 | — | + +--- + +## 7. 落地:在 PR #23 上追加的 commit + +原型补丁(H1 + M1 + M2 + 3 个复现测试)已经在 PR 分支上跑通:`test_cancel` 16/16、`test_pool`、`test_proxy`、`test_framing` 通过(gcc 16),`test_cancel` 在 llvm 22 上也通过。之前那一轮(H1 + 复现测试)的全量 `mcpp test` 是 8/8 🔁。 + +| # | commit | 依赖 | +|---|---|---| +| 1 | `fix(tls): a stop token reaches the session a tunnelled one runs over`:H1,以及 https/http/直连三组复用测试 | — | +| 2 | `fix(http): a pooled connection carries no stop token`:M1,以及 jthread 析构场景的测试 | 1 | +| 3 | `test(cancel): import std before the includes under libc++`:M2 | — | +| 4 | `feat(download): download_to_file takes a stop token`:D1,以及 `DownloadToFileResult::cancelled` 和测试 | 2,D1 批准后 | +| 5 | `refactor(proxy): proxy_connect does not take a stop token`:D2 | D2 批准后 | +| 6 | `ci: linux on llvm 22 as well; windows runs the hermetic tests`(再加 macOS job,如果可行):D3 | 3 | +| 7 | `docs: cancellation contract, download_to_file, libc++ note` | 4、5 | + +顺序说明:1 和 2 是正确性修复,最先落地;3 必须在 6 之前,否则新加的 llvm job 第一次跑就会失败;6 失败时(比如 Windows 测试服务器不可移植),可以把 Windows 部分拆成 follow-up,不阻塞合并。 diff --git a/.agents/docs/exec-plan-0.3.0.md b/.agents/docs/exec-plan-0.3.0.md new file mode 100644 index 0000000..79250fb --- /dev/null +++ b/.agents/docs/exec-plan-0.3.0.md @@ -0,0 +1,183 @@ +# 执行计划 —— 0.3.0:把连接池不变量写进类型里 + +| | | +|---|---| +| 输入 | [`fix-plan-issue-15-16.md`](fix-plan-issue-15-16.md)(19 项)+ openkal 体系适配 | +| 基线 | `origin/master @ 2cec1c1`(0.2.10) | +| 目标 | **0.3.0** —— 单 PR 全量实现 P0 + P1 + P2 + P3-A/B/D | +| 分支 | `fix/pool-invariants-and-sigpipe` | + +--- + +## 0. 为什么是一个 PR、为什么是 0.3.0 + +原方案建议拆 4 个 PR(0.2.11 → 0.2.12 → 0.2.13 → 0.3.0)。这里合并成一个,理由是**依赖方向**而不是省事: + +- P1 的十处补丁,和 P3-A 的 RAII guard,**是同一件事的两种写法**。先合 P1 再合 P3-A,等于把十个 `connectionClose = true` 写进去、下一个 PR 再全部删掉。 +- P2-C1/C2/C4(状态行、header 循环、Content-Length)在三个入口各有一份拷贝。P3-B 把三份合成一份之后,这三项**只需要改一处**;不合并就要改九处再删六处。 +- §1.1 的矩阵证明:**分批修 = 每批只落到一两份拷贝上**。`#14` 就是这样引入 A1/A2 回归的。一个 PR 是让这张矩阵不再存在的唯一方式。 + +版本取 **0.3.0** 而不是 0.2.11:`HttpResponse` / `HttpClientConfig` 加了字段(非破坏),严格解析会把过去静默接受的畸形响应变成显式错误(**行为变更**)。语义化版本下,行为变更进 minor。 + +**无感升级的边界**(release note 必须点名): +- 现有代码**不需要改一行**就能编译:只加字段,不改签名,`ok()` 语义不变。 +- 过去被静默接受的畸形响应现在会报错 —— 这不是新 bug,是过去在静默损坏数据。 + +--- + +## 1. 八个视角,各自要什么 + +| 视角 | 要求 | 落在哪 | +|---|---|---| +| **架构** | 池不变量不能靠十个早退路径各自记得维护 | P3-A `PooledConnection`(默认丢弃)+ P3-B 三份 reader 合一 | +| **稳定性** | 库不得杀死宿主进程;不得因一个 header 分配 2 GiB | P0(`MSG_NOSIGNAL`/`SO_NOSIGPIPE`)、P2-E2(分配上限) | +| **优雅简洁** | 一个概念一处实现 | 删 `parse_hex`;`read_response_head` / `read_body` 各一份;`parse_status_line` 一份 | +| **用户体验** | 调用方要能知道 body 被截断了;空闲连接被服务端关掉不该变成 `No response` | P2-F1 `bodyComplete`+`bodyError`、P3-D 陈旧连接重试 | +| **兼容性** | 只加字段,不改签名,`ok()` 不变 | `HttpResponse` / `HttpClientConfig` 追加字段;`TlsSocket::read` 保留 | +| **跨平台** | Linux / macOS / Windows / Termux / **openkal** 全绿 | `#ifdef` 而非 `if constexpr`(5e7d66f 的教训);平台事实集中在 `platform.cppm` | +| **一致性** | `send` / `send_stream` / `download_to_file` 三个入口行为逐条相同 | P3-B 之后三者共用同一个 head/body reader | +| **无感升级** | 升级不需要改代码 | 见 §0 | + +--- + +## 2. 任务依赖图 + +``` +T0 分支 + 计划 ✔ + │ + ├─ T1 P0 SIGPIPE(socket.cppm) 必须最先 + │ │ P0-1 send() + MSG_NOSIGNAL + │ │ P0-2 socket() 后 setsockopt(SO_NOSIGPIPE) + │ │ 理由:T5 让 pool_.erase() 在更多路径上执行, + │ │ erase → ~TlsSocket → close_notify → send()。 + │ │ T1 没做之前,T5 等于扩大 SIGPIPE 触发面。 + │ │ + │ ├─ T2 P2-D1/D2(tls.cppm / http.cppm 的 write_all) + │ │ read_some() 三态:Data / WouldBlock / Eof / Error + │ │ write_all 遇 WANT_WRITE 改为 wait_writable + 总时限 + │ │ (T3 的 reader 建立在三态之上) + │ │ + │ └─ T3 P3-B 统一 reader(http.cppm) ← 吸收 P2-C1/C2/C4 + P1-D1 + │ parse_status_line(导出,严格) + │ read_response_head → expected + │ read_body → BodyOutcome{Complete|Truncated|ClosedByPeer|Stopped} + │ ├ 吸收 P1-B2(parse_chunk_size_line,删 parse_hex) + │ ├ 吸收 P2-C3(块后 CRLF 校验) + │ └ 吸收 P2-E2(分配上限,chunk 分片读) + │ │ + │ ├─ T4 P3-A PooledConnection(默认丢弃)← 吸收 P1-A/B/C 全部十项 + │ │ │ 递归重定向:递归前必须 keep() 或 drop(),不能让析构跨过递归 + │ │ │ + │ │ ├─ T5 P2-E1 bounded drain(重定向/非 2xx/文件打不开时省一次握手) + │ │ ├─ T6 P2-F1/F2 bodyComplete + bodyError(statusText 不再被覆盖) + │ │ └─ T7 P3-D 陈旧连接重试(一次,仅在零响应字节时) + │ │ + │ └─ T8 纯单测:parse_status_line / parse_chunk_size_line / parse_content_length + │ + ├─ T9 集成测试:进程内 mbedtls TLS listener(T4 后) + │ tinyhttps 只支持 https,本地明文 listener 用不了 → 必须自带 TLS 服务端 + │ T1..T11 用例见 §4 + │ + ├─ T10 openkal 体系(与 T1..T9 并行调研,结论落地在 T1/platform.cppm) + │ + └─ T11 文档 / release note / README ← 依赖全部 + └─ T12 CI 全绿 → 自审 → PR → release → gtc 补 gitcode → mcpp-index → 生态验证 +``` + +**关键路径**:T1 → T3 → T4 → T9 → T12。 +**可并行**:T8、T10 与 T2..T7 无共享状态。 + +--- + +## 3. 原方案 19 项 → 本计划落点 + +| 原 ID | 落在 | 备注 | +|---|---|---| +| P0-1 / P0-2 | T1 | 逐字采纳 | +| P1-A1/A2/A3、B1、C1/C2/C3/C4/C5 | **T4** | 十项全部由 `PooledConnection` 默认丢弃消除,不逐条打补丁 | +| P1-B2 | T3 | 并入统一 chunk reader;`parse_hex` 删除 | +| P1-D1 | T3 | `ResponseHead::contentLength` 只有 `int64_t` 一种类型 | +| P2-C1 | T3 | 新增导出 `parse_status_line` | +| P2-C2 | T3 | `read_response_head` 全程用 `read_complete_line` | +| P2-C3 | T3 | 块后 CRLF 必须是空行 | +| P2-C4 | T3 | 统一 `parse_content_length` | +| P2-E1 | T5 | `drain_body`,上限 64 KiB | +| P2-E2 | T3 | `maxResponseBodyBytes`(默认 64 MiB);chunk 按固定缓冲分片读,不按块大小分配 | +| P2-D1 | T2 | `TlsSocket::read_some` 三态;`read()` 保留 | +| P2-D2 | T2 | `write_all` 用 `wait_writable` | +| P2-F1 | T6 | `bodyComplete`,`ok()` 不变(方案 a,非破坏) | +| P2-F2 | T6 | `statusText` 不再被错误信息覆盖;原因进 `bodyError` | +| P3-A | T4 | 采纳 | +| P3-B | T3 | 采纳(head + body 都合) | +| P3-C 带缓冲的读 | **不做** | 见 §5 | +| P3-D | T7 | 采纳 curl 策略 + `retryOnStaleConnection` 开关(默认 true) | + +### 两个 review 决策点的裁定 + +- **P2-F1 `ok()` 是否要求 `bodyComplete`** → **不要求**(方案 a)。目标写明「无感升级」;让 `ok()` 变严会让现有代码在**升级后**行为改变,与之直接冲突。截断可见性由新字段提供。 +- **P3-D 是否限制到幂等方法** → **不限制**(curl 策略),加 `retryOnStaleConnection` 开关。重试窗口是「一个响应字节都没收到」,此时服务端已发 FIN、应用层没看到这个请求;限制到幂等方法等于对 LLM/API 客户端(POST 为主)这个修复基本无效。 + +--- + +## 4. 测试矩阵 + +### 4.1 纯单测(无需服务端)—— T8 + +`parse_status_line` / `parse_chunk_size_line` / `parse_content_length` / `append_within_limit` 表驱动。 +其中 `parse_status_line` 必须覆盖 issue #15 的原始输入 `"BBBB 999 XHTTP/1.1 200 OK"` → 拒绝。 + +### 4.2 集成测试 —— T9 + +**为什么必须自带 TLS 服务端**:`send/send_stream/download_to_file` 三个入口开头都有 +`if (parsed.scheme != "https") return "Only HTTPS is supported"`,本地明文 listener 根本进不去。 +所以测试自带一个进程内 mbedtls TLS listener(自签证书,`verifySsl = false`), +`MBEDTLS_SSL_SRV_C` / `MBEDTLS_PEM_PARSE_C` 在依赖的 mbedtls 3.6.1 配置里已开启(已核验)。 + +| # | 用例 | 断言 | 覆盖 | +|---|---|---|---| +| T1 | 服务端发一半 CL body 后挂住 | 请求 2 `statusCode == 200` | P1-A1 | +| T2 | 同上,**并断言服务端 accept 次数 == 2** | 唯一能抓住「看起来对了」的断言 | P1-A1 | +| T3 | chunked 版本同上 | 同 T1+T2 | P1-A3 | +| T4 | 服务端发一半后关连接 | 不崩;进程未被信号杀死 | #16 | +| T5 | `SIGPIPE` 默认处置下跑 T4(子进程) | `WIFSIGNALED == false`,退出码 ≠ 141 | P0 | +| T6 | `Content-Length: 4294967296` | 不得当成 0;连接不得回池 | P1-D1 | +| T7 | 302 + CL body + keep-alive → 同 host | 内容正确 + accept 次数正确 | P1-C3 | +| T8 | 404 + error body + keep-alive,同 client 再发一个 | 第二个请求正确 | P1-C4 | +| T9 | chunk 头是 `zz` / 空行 | 报错,不得当成终止块 | P1-B2 | +| T10 | 状态行 `BBBB 999 XHTTP/1.1 200 OK` | `Invalid status line`,不是 999 | P2-C1 | +| T11 | 截断的 body | `bodyComplete == false` | P2-F1 | +| T12 | 正常 keep-alive 两请求 | accept 次数 == **1**(不能因为修得太狠而丢掉复用) | 回归护栏 | +| T13 | 服务端在两请求之间关闭空闲连接 | 请求 2 成功,accept 次数 == 2 | P3-D | + +**T12 与 T2 是一对**:T2 保证脏连接被丢弃,T12 保证干净连接被保留。只有 T2 会让「关掉连接池」这种过度修复也通过。 + +--- + +## 5. 明确不做的,和为什么 + +| 项 | 不做的理由 | +|---|---| +| **P3-C 带缓冲的读** | 原方案自陈「不要当成性能修复来卖」,其主要理由是「让『这条连接干净吗』可回答」——而 T4 的 guard + T3 的严格 framing 已经把这个问题回答了:干净 = `read_body` 返回 `Complete`。缓冲区是第二种表达,不是必需的一种。单独立项。 | +| **openkal 原生 socket 后端** | 见 §6。 | + +--- + +## 6. openkal 体系 + +调研 T10 输出决定落点;先记录已核验的事实: + +- `openkal-musl/port/src/okm_net.c:546-550`:`MSG_NOSIGNAL` 被**接受并忽略**(注释原文:"asks that a signal not be raised. There are no signals here"),`rest = flags & ~(MSG_NOSIGNAL | MSG_DONTWAIT)`,其余 flag 返回 `-ENOSYS`。 + ⇒ **P0-1 在 openkal 体系上是正确且无副作用的**,而且必须写成 `#ifdef MSG_NOSIGNAL`(否则 macOS 分支编不过,见 `5e7d66f`)。 +- `SO_NOSIGPIPE` 是 BSD/macOS 的拼法,musl 头文件不定义 ⇒ P0-2 在 openkal-musl 上自动不编入。 + +--- + +## 7. 交付顺序 + +1. T1 → T2 → T3 → T4 → T5/T6/T7(实现) +2. T8 / T9(测试) +3. T10 结论落地 +4. T11 文档 + release note +5. 本地 `mcpp build && mcpp test` 全绿 → 推 PR → CI 全绿 → 自审 → 合并 +6. tag + GitHub release → `gtc` 本地补 gitcode 资源(不等 release CI) +7. mcpp-index 增加 0.3.0 条目 → 生态真实验证 diff --git a/.agents/docs/exec-plan-0.3.4-cancel.md b/.agents/docs/exec-plan-0.3.4-cancel.md new file mode 100644 index 0000000..b065c49 --- /dev/null +++ b/.agents/docs/exec-plan-0.3.4-cancel.md @@ -0,0 +1,101 @@ +# 执行计划 —— 0.3.4:在途请求可取消,三个入口一致,CI 覆盖库支持的全部平台 + +| | | +|---|---| +| 输入 | [`design-cancel-stop-token.md`](design-cancel-stop-token.md):D1–D4 均按建议采纳 | +| 基线 | `origin/master @ f461940`(0.3.3) | +| 落地 | 全部做在 PR #23 上(`yspbwx2010:feat/cancel`,`maintainerCanModify: true`),squash 合入 | +| 目标版本 | **0.3.4** | +| 跨仓库 | `mcpplibs/tinyhttps` → gitcode `mcpp-res/tinyhttps`(CN 镜像)→ `mcpplibs/mcpp-index` | + +--- + +## 0. 为什么是 0.3.4 + +- 只新增,不改已有行为:`send`、`send_stream`、`download_to_file` 都只加了尾部默认参数;`HttpResponse` 和 `DownloadToFileResult` 各在末尾追加一个字段;不传 token 时逐字节不变(不变式 I3)。 +- 依赖写成 caret 范围 `tinyhttps = "^0.3.0"` 的项目会自动解析到它。**更正(发布后实测):** mcpp 里裸写的 `"0.3.1"` 是精确版本,不会解析到 0.3.4;`"0.3.x"` 不是 mcpp 的合法写法。0.3.3 的发布说明里也沿用了这个说法。已用已发布的 index 实测:`"^0.3.1"` 在 GLOBAL 和 CN 两个源下都解析到 0.3.4。 +- 唯一的源码不兼容点是对 `&HttpClient::send` 取成员函数指针(参数个数变了)。在 CHANGELOG 里点名说明。 + +## 1. 八个视角 + +| 视角 | 要求 | 落在哪 | +|---|---|---| +| **架构** | token 存在哪里、存活多久要有规则,不能靠每个调用点自己记得 | I1:只存在传输链最底层的 `Socket`,`TlsSocket` 只往下转发。I2:只在一次调用期间挂着,`PooledConnection::keep()` 把连接还回池时摘下 | +| **稳定性** | 取消不能引发重复执行、fd 泄漏或卡死 | 取消后不重试、不跟重定向;连接 drop;fd 计数测试;复用连接测试覆盖直连、http 代理、https 代理三种 | +| **优雅简洁** | 只有一套机制:在 `wait_fd` 里分片 `poll` | 不引入 pipe、shutdown 或回调;不增加配置项;`proxy_connect` 不加参数 | +| **用户体验** | 用 `std::jthread` 就能直接取消;结果用 bool 判断,不用匹配文本 | README 示例;两个结果结构都有 `cancelled`;README 说明 libc++ ≤22 的工具链问题 | +| **兼容性** | 升级不需要改代码 | §0;`HttpResponse{204, "No Content", {}, {}}` 这种聚合初始化仍然能用(mcpp-index 的测试断言了这一点) | +| **跨平台** | Linux(gcc、libc++)、macOS、Windows(msvc、clang)、openkal(linux-gnu、windows-musl)都要实测 | §4 CI 矩阵;openkal 示例增加一段不联网的取消检查 | +| **一致性** | 三个入口取消语义相同 | `download_to_file` 加 token(D1);`isCancelled` 保留 | +| **无感升级** | 不传 token 时行为和 0.3.3 一样;不需要改依赖版本号 | I3;0.3.4 是补丁版本 | + +## 2. 任务依赖图 + +``` +T0 计划(本文)+ 设计文档纳入 PR + │ + ├─ T1 src:I1(tls.cppm set_stop 转发给 lower_) + │ └─ T2 src:I2(http.cppm PooledConnection::keep 时摘下 token) + │ └─ T3 src:D1 download_to_file(…, stop) + DownloadToFileResult::cancelled + │ └─ T4 src:D2 proxy_connect 去掉 stop 参数;源码注释 + │ + ├─ T5 测试(可与 T1–T4 并行编写,依赖它们才能通过) + │ a. 复用连接矩阵:直连 / http 代理 / https 代理 × {旧 token 已 stop, 新 token 生效} + │ b. SOCKS5 等待握手应答时取消 + │ c. download_to_file:等响应头时取消、读 body 时取消、已 stop 的 token + │ d. libc++ 下 import std 的位置(M2) + │ + ├─ T6 openkal:examples/openkal 增加不联网的取消检查(本地监听 + 握手中取消) + │ 运行时版本跟 mcpp-index 的 pins.toml 对齐;增加 windows-musl(Linux 上构建,windows-2022 上原生运行;本地用 wine) + │ + ├─ T7 CI(依赖 T5d、T6) + │ linux {gcc@16.1.0, llvm@22.1.8} 跑全部测试 · macos-15 llvm@22.1.8 跑全部测试 + │ windows {msvc, llvm@20.1.7} 跑不联网的测试 · openkal {x86_64-linux-gnu, x86_64-windows-musl} + │ + ├─ T8 文档:README、CHANGELOG 0.3.4、mcpp.toml version、README 安装行 + │ + └─ T9 合入与发版(依赖全部) + CI 全绿 → 自审 → squash 合入 → tag 0.3.4 + GitHub release + → 本地 gtc 上传 gitcode mcpp-res/tinyhttps 的 0.3.4 tarball(核对 sha256) + → mcpp-index PR:pkgs/t/tinyhttps.lua 加 0.3.4(三个平台), + tests/examples/tinyhttps 升到 0.3.4 并加取消的 API 检查 + → 用已发布的 index 实测:新建项目解析到 0.3.4,GLOBAL 和 CN 两个源都验证 +``` + +**关键路径**:T1 → T2 → T5 → T7 → T9。T3/T4、T6、T8 可以和关键路径并行。 + +## 3. 测试清单(test_cancel) + +| 用例 | 覆盖 | +|---|---| +| PR 原有 13 个 | 首次建连各阶段、不重试、重定向、fd 泄漏、connect、http/https 代理 | +| `APooledTunnelCarriesNoTokenFromTheLastCall`,参数为直连 / http / https | I1 + I2,H1 的回归测试(一个 POST 只能执行一次) | +| `TheSecondCallsTokenReachesAReusedTunnel`,参数为直连 / http / https | I1 | +| `ASocks5ProxyThatNeverAnswersIsAbandoned` | 代理种类补齐 | +| `ADownloadWaitingForTheHeadersIsAbandoned` / `…ForTheBody…` / `…AlreadyStopped…` | D1 | + +## 4. CI 矩阵(目标) + +| job | 工具链 | 跑什么 | +|---|---|---| +| linux | gcc@16.1.0 / llvm@22.1.8 | 全部 `mcpp test`(`test_download`、`test_resolver` 需联网)+ 模板冒烟 | +| macos-15 | llvm@22.1.8 | 全部 `mcpp test` | +| windows-2022 | msvc / llvm@20.1.7 | `test_ca_store`,以及不联网的 `test_framing`、`test_pool`、`test_proxy`、`test_cancel`、`test_tls_verify` | +| openkal | llvm@22.1.8 + openkal-llvm-runtime | `examples/openkal` 在 x86_64-linux-gnu 上原生跑、x86_64-windows-musl 在 Linux 上构建、在 windows-2022 上原生跑(CI 装 wine 要 4–30 多分钟,且不稳定)| + +某个平台上测试编不过时,**修测试,不删平台**。确实做不到的,单独开 issue 记下原因,并在 job 里写清楚。 + +## 5. 过程中发现并修复 + +- **x86_64-windows-musl 上 TLS 从来没通过。** mbedTLS 只有识别到 glibc 才用 `getrandom`,否则读 `/dev/urandom`,而 openkal-windows 没有这个文件。在 musl 平台上改用 musl 的 `getentropy`(`cfg(c-abi = "musl")` → `TINYHTTPS_GETENTROPY`)🔁。 +- **Windows 目标没有名字解析**(openkal 没有解析器接口),已登记 [openkal-musl#46](https://github.com/mcpplibs/openkal-musl/issues/46),README 已注明。 +- **`test_pool` 在 Windows 上编不过**(`` 的 `max` 宏),因为以前从没在 Windows 上编译过;已修复。 + +## 6. 验收 + +- [x] PR #23 的 CI 全绿(10 个 job,`10e28b0`) +- [x] 自审:H1、M1、M2、D1、D2 全部落地;PR 描述更新 +- [x] squash 合入(`156edeb`);tag `0.3.4`;GitHub release +- [x] gitcode 上的 `tinyhttps-0.3.4.tar.gz` 与 GitHub archive 的 sha256 一致(`26e0e496…41e9`) +- [x] mcpp-index 合入(mcpplibs/mcpp-index#504);`validate` 和 `openkal-compat`(tinyhttps 这一项,四个 target 均为 runs)通过 +- [x] 新建项目用已发布的 index 解析到 0.3.4(`"^0.3.1"`),GLOBAL 和 CN 两个源都能构建并运行,并对 httpbin 做了真实的进行中取消(301 ms) diff --git a/.agents/docs/exec-plan-0.3.5.md b/.agents/docs/exec-plan-0.3.5.md new file mode 100644 index 0000000..e903164 --- /dev/null +++ b/.agents/docs/exec-plan-0.3.5.md @@ -0,0 +1,171 @@ +# 执行计划 —— 0.3.5:额外信任的 CA,TLS 握手有时限,Windows CI 恢复 + +| | | +|---|---| +| 输入 | [PR #24](https://github.com/mcpplibs/tinyhttps/pull/24) `feat/extra-ca-certs @ bd76c9e` · [PR #25](https://github.com/mcpplibs/tinyhttps/pull/25) `fix/tls-handshake-timeout @ cfdab98`(作者都是 yspbwx2010,`maintainerCanModify: true`) | +| 基线 | `origin/master @ 2cc7310`(0.3.4) | +| 落地 | 两个 PR 各自 squash 合入;对 #25 的修正直接追加在 #25 上(已推送 `97a0657`) | +| 目标版本 | **0.3.5**(D1–D4 按建议采纳) | +| 跨仓库 | `mcpplibs/tinyhttps` → gitcode `mcpp-res/tinyhttps`(CN 镜像)→ `mcpplibs/mcpp-index` | +| 文档日期 | 2026-10-10 | + +**标注约定**:✅ 已核验源码 · 🔁 已在本地复现或跑通 · ❓ 推断,未实测 + +--- + +## 0. TL;DR + +- **两个 PR 的方向都对,设计也符合本项目的约定**:只新增、尾部默认参数、新字段放最后、错误写进 `statusText`、状态只放在传输链最底层的 `Socket`(沿用 0.3.4 的 I1)、测试不联网(hermetic)。和业界通行做法也一致(§2)。 +- **#24 可以直接合**,不需要追加 commit。剩下几处小问题不影响合入(§3.3)。 +- **#25 有一个回归,已修复并推到 PR 上** 🔁:handshake 的等待改成了 `poll`,信号会让它以 EINTR 返回,即使 handler 装的是 SA_RESTART 也一样。结果是装了信号 handler 的进程(profiler、定时器)握手会被误报成 “TLS handshake timed out”。修复前新加的用例在 100 ms 就失败了(时限是 1000 ms)。 +- **Windows CI 红是 runner 镜像的变化,和两个 PR 都无关** ✅:win22/20261004.326 带的是 Git for Windows 2.56,原来硬编码的 `mingw64/etc/ssl/certs/ca-bundle.crt` 不存在了。修复已随 #25 一起推送:向 git 查询 bundle 的路径,查不到再在安装目录里搜索。 +- **合入顺序是 #25 → #24**:#25 带着 CI 修复,先合入后 master 才能变绿;#24 再同步 master,同时解决 CHANGELOG 的冲突。 + +### 待你决定 + +| # | 决策 | 建议 | +|---|---|---| +| D1 | 版本号用 0.3.5 还是 0.4.0 | **0.3.5**。#25 严格说改变了行为(握手超过 `connectTimeoutMs` 会失败),不符合 0.3.4 “补丁版本只新增”的约定。但它修的是一个永久卡死的问题;而且 0.4.0 不会被 `^0.3.x` 解析到,最需要这个修复的正是已经在用 `^0.3.x` 的项目。在 CHANGELOG 里用 “Behaviour change” 单列说明 | +| D2 | `connectTimeoutMs` 继续按阶段各自计时,还是改成整个建连过程共用一个 deadline | **0.3.5 保持按阶段计时**(和现有的代理应答等待一致,PR 已在文档里写明)。共用 deadline 和 curl 的语义一样,更符合直觉,留到 0.4 统一做(§4) | +| D3 | #24 是否同时提供内存中的 PEM(如 `extraCaPem`) | **这次不做**,作为后续单独的一项。路径方式已经覆盖代理、企业 CA 这些主要场景 | +| D4 | `Socket::set_deadline / deadline_hit / wait_before_recv` 都是 public(因为整个类是 export 的) | **接受**。`set_stop / stop_possible` 已经是这样,跨 partition 用 friend 在各家编译器上的风险更大;注释已写明它们是给 TLS 层用的 | + +--- + +## 1. CI 失败分析(run 37911364113 / job 113757995241) + +| 项 | 内容 | +|---|---| +| 现象 | `run (x86_64-windows-musl, above openkal, on Windows)` 运行 4 秒后失败:`cp: cannot stat '/c/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt'` ✅ | +| 根因 | runner 镜像从 win22/20260927.320 换成了 win22/20261004.326,Git for Windows 也从 2.55 升到 2.56 ✅。2.56 把环境从 MINGW64 换成了 UCRT64,bundle 现在在 `/c/Program Files/Git/ucrt64/etc/ssl/certs/ca-bundle.crt`(新 CI 日志里打印出来的路径)✅ | +| 影响 | 所有分支的这个 job 都会失败,在 smoke 程序运行之前就挂了。和 #24、#25 的代码都无关 | +| 修复 | 已推送到 #25 的 `bf99dbd`:先用 `git config --system http.sslcainfo` 查路径(经 `cygpath` 转换),查不到就在 `/c/Program Files/Git` 下搜索 `*/ssl/certs/ca-bundle.crt`,还找不到就报 `::error::` 并给出明确原因。同时打印 `git --version` 和最终用到的路径,方便下次镜像再变时定位 | +| 验证 | #25 的 CI run 37968516327:10 个 job 全绿,这个 job 用 `git config` 查到了 ucrt64 下的路径;smoke 输出 `cancellation: ok (stopped at 200 ms, returned after 261 ms)` ✅ | +| 顺带发现 | 这个 job 里 `framing parsers: ok` 和 `cancellation: ok` 之间有约 39 s 的空档,10-05 master 的运行里也有,所以不是这两个 PR 引入的。空档出现在计时开始之前,原因没有查 ❓,列入 §4 | + +--- + +## 2. 设计评审 + +### 2.1 是否符合本项目的约定 + +| 约定(来源) | #24 | #25 | +|---|---|---| +| 只新增:尾部默认参数、新字段放最后(0.3.4 §0) | ✅ `extraCaFile` 放在最后;`proxy_tunnel` 加了尾部默认参数 | ✅ `connect_over` 加了尾部默认参数;CHANGELOG 点名说明了成员函数指针的不兼容 | +| 状态只放最底层 `Socket`,`TlsSocket` 只往下转发(0.3.4 I1) | 不涉及 | ✅ `set_deadline` / `deadline_hit` 都经 `lower_` 往下转发,https 代理隧道的情况有测试覆盖 | +| 状态只在调用期间存在(0.3.4 I2) | 不涉及 | ✅ 握手结束就清掉 deadline,有测试覆盖 | +| 不配置时行为不变(0.3.4 I3) | ✅ 字段为空时一行代码都不走 | ⚠️ 默认的 10 s 会生效,这是有意的行为变化(见 D1) | +| 失败原因写进 `statusText`,能直接看懂 | ✅ 错误信息里带文件路径 | ✅ 显示 `TLS handshake timed out`;代理那一跳还会带上代理地址 | +| 测试不联网,能在 Windows 上跑 | ✅ 已加进 Windows 的 hermetic 列表 | ⚠️ **原 PR 没有加**,现已补上(`bf99dbd`) | +| 不引入新的环境变量或全局状态 | ✅ 只增加一个字段 | ✅ 不新增配置项 | + +### 2.2 语义和业界做法对照 + +**#24 `extraCaFile`:在默认 store 之外再追加信任的 CA** + +| 实现 | 语义 | +|---|---| +| Node `NODE_EXTRA_CA_CERTS` | **追加**,和 #24 完全一样,名字也对应 | +| reqwest `add_root_certificate` | **追加**到内置根证书 | +| curl `--cacert` / `CURLOPT_CAINFO`、Python requests `verify=path`、Go `tls.Config.RootCAs` | **替换**(本库对应的是 `SSL_CERT_FILE`) | + +结论:库里原本只有“替换”的方式(`SSL_CERT_FILE`),#24 补上了“追加”,这是业界常见的另一半。文件不可用时直接失败、不跳过,这个判断对:跳过的话,错误最后会变成一个不提文件的证书校验失败。 + +**#25 握手时限并入 `connectTimeoutMs`** + +| 实现 | 握手的时限 | +|---|---| +| curl `CURLOPT_CONNECTTIMEOUT` | 包含 TLS 握手,并且整个建连阶段共用一个时限 | +| Go `http.Transport.TLSHandshakeTimeout` | 单独一项,`DefaultTransport` 默认 10 s | +| #25 | 包含握手,但**每个阶段各自计时** | + +结论:并入 `connectTimeoutMs`(不新增配置项)和 curl 一致,默认 10 s 也和 Go 的默认值相同。差别在于各阶段各自计时,最坏情况下总时间是 N × timeout。PR 和 README 都已写明(见 D2)。 + +### 2.3 API 好不好用 + +| 点 | 评价 | +|---|---| +| `cfg.extraCaFile = "/etc/corp/root-ca.pem";` 一行就能用 | 好。代理那一跳自动生效,不用再配置一次 | +| 只接受文件路径 | 证书打包进程序、或者从配置中心拿到的场景要先落盘再用(D3) | +| 只能指定一个文件 | 和 Node 一样,可以把多个证书拼成一个文件,够用 | +| `verifySsl=false` 时文件读不到也会失败 | 有意设计(所有机器上报同样的错),README 里再补一句说明会更好 | +| 握手超时不需要任何改动就生效 | 好,解决的是默认配置下的卡死问题 | +| `connect_over(..., handshakeTimeoutMs = -1)` 和 `connect(..., timeoutMs, ...)` 的默认值不对称 | 为了兼容只能这样:`connect_over` 原本就不带超时 | + +--- + +## 3. 问题清单与处理 + +### 3.1 #25(已推送:`cfdab98..97a0657`) + +| # | 级别 | 问题 | 处理 | +|---|---|---|---| +| H1 | 高 | EINTR 被当成 deadline 到期。`wait_before_recv` 只要等待失败就设置 `deadline_hit_`,不看时钟。没有 token 时只调用一次 `poll`,信号一来就返回。master 上走的是阻塞的 `recv`,在 SA_RESTART 下会自动重启,所以这是回归 🔁 | `3013b98`:提前返回就按剩余时间继续等,只有时钟确实到了才算 deadline 到期。新用例 `ASignalDuringTheHandshakeIsNotATimeout`:时限 1000 ms,每 100 ms 用 `pthread_kill` 给握手线程发一次 SIGUSR1。修复前 100 ms 就失败,修复后通过 🔁 | +| M1 | 中 | `test_handshake_timeout` 没进 Windows 的 hermetic 列表,`WSAPoll` 路径没人测 | `bf99dbd` | +| M2 | 中 | Windows openkal job 的 bundle 路径(§1) | `bf99dbd` | +| L1 | 低 | CHANGELOG 没提导出的 `TlsSocket::connect` 现在也会限制握手时间 | `97a0657` | +| L2 | 低 | 移动构造和移动赋值复制了 `deadline_`,但没复制 `deadline_hit_` | `3013b98` | +| — | 已知限制 | 握手期间的 `send` 是阻塞的;`readTimeoutMs` 管不到阻塞的写(README 里的说法和实际不符) | 不在本版本处理,单列一项(§4) | + +本地结果(Linux, LLVM 22)🔁:`test_handshake_timeout` 9/9、`test_cancel` 24、`test_proxy` 43、`test_pool` 27、`test_tls_verify` 6、`test_framing` 17,全部通过。 + +### 3.2 #24:可以直接合入,不追加 commit + +本地结果 🔁:`test_extra_ca` 11/11,其余测试不受影响。#24 和 #25 合在一起后,全部测试也都通过(`test_extra_ca` + `test_handshake_timeout` 一起跑过)。 + +### 3.3 #24 的小问题(合入后视情况再处理,不阻塞本版本) + +- 在 Linux 上,路径是目录时 `ifstream` 也能打开,错误信息变成 “cannot parse”,不是 “cannot read”。加一个 `is_regular_file` 判断就能修。 +- 部分证书解析失败(`ret > 0`)不会报错,和默认 bundle 的处理一致,README 可以补一句。 +- 每建一个新连接都要重新读、重新解析一次文件。默认 bundle 也是这样,暂时不缓存。 + +--- + +## 4. 0.3.5 之后(不在本版本范围内) + +| 项 | 说明 | +|---|---| +| 写操作不阻塞 | `bio_send` / `write_all` 在阻塞 socket 上永远拿不到 WANT_WRITE,所以 `readTimeoutMs` 管不到阻塞的写,握手期间的写也不受 deadline 约束。这是 0.3.4 D4 留下的问题 | +| 整个建连共用一个 deadline(D2) | DNS、TCP(每个地址)、代理应答、两次握手共用一个 `connectTimeoutMs`。行为有变化,适合 0.4 | +| `extraCaPem`(D3) | 直接从内存加载 PEM | +| openkal Windows smoke 的 39 s 空档 | 见 §1,master 上已有,原因待查 | +| `poll_fd` 统一处理 EINTR | TCP connect 和代理应答的等待也有同样的问题,H1 只修了握手。可以在 `poll_fd` 一处统一修 | + +--- + +## 5. 任务依赖图 + +``` +T0 计划(本文)← 你在这里审 + │ + ├─ T1 #25 追加 commit(已推送 97a0657)→ CI 全绿 ✅(run 37968516327) + │ + ├─ T2 更新 #25 的 PR 描述:Not covered 里关于 EINTR 的说法改成“已修复”;补充 CI 的变化 + │ + ├─ T3 squash 合入 #25(依赖 T1、T2) + │ + ├─ T4 #24 同步 master:合并两边 CHANGELOG 的 “Unreleased” 段 → CI 全绿 → squash 合入 + │ + ├─ T5 发版 commit:CHANGELOG 的 Unreleased → 0.3.5;mcpp.toml version;README 安装行(mcpp add / 依赖示例) + │ + └─ T6 发布(依赖全部) + tag 0.3.5 + GitHub release + → gitcode mcpp-res/tinyhttps 上传 0.3.5 tarball(核对和 GitHub archive 的 sha256 一致) + → mcpp-index PR:pkgs/t/tinyhttps.lua 加 0.3.5(GLOBAL / CN,三个平台), + tests/examples/tinyhttps 升到 0.3.5,加一条 `extraCaFile` 的 API 检查 + → 用已发布的 index 实测:新建项目写 "^0.3.1" 能解析到 0.3.5,GLOBAL 和 CN 两个源都验证 +``` + +**关键路径**:T1(CI)→ T3 → T4 → T5 → T6。 + +## 6. 验收 + +- [x] #24、#25 以及两者合并后,hermetic 测试在本地全部通过 🔁 +- [x] H1 能复现,修复后用例通过 🔁;已推送到 #25 +- [x] #25 CI 全绿(run 37968516327,10/10),包括 Windows openkal 和 Windows hermetic(`test_handshake_timeout`) +- [x] #25 squash 合入(`9d4315e`);#24 同步 master(合并时 CHANGELOG 和 Windows 测试列表都有冲突)后 CI 10/10 全绿,squash 合入(`a0f7749`) +- [x] 版本号 0.3.5(D1);发版 commit `6153cdd` 的 master CI 10/10 全绿;tag `0.3.5`,GitHub release +- [x] gitcode `mcpp-res/tinyhttps` 0.3.5 的 tarball 和 GitHub archive 的 sha256 一致(`6f75f9e6…58a3`) +- [x] mcpp-index 合入(mcpplibs/mcpp-index#517,`bf7e87c`):PR 的全部检查通过,包括 openkal measure 和 linux/macos/windows workspace;合入后 Publish Index Artifact、validate、deploy-site 也都通过 +- [x] 真实验证:用已发布的 index 新建项目,依赖写 `"^0.3.1"`,在 GLOBAL 和 CN 两个源下(都清了缓存)都解析到 0.3.5。真实 HTTPS 请求返回 200;握手不回应的本地对端在 `connectTimeoutMs=500` 下约 520 ms 报 `TLS handshake timed out`;`extraCaFile` 文件不存在时,报错里带路径;指向系统 bundle 时请求正常返回 200 +- 注:mcpp 在一个源失败时会自动换另一个源,所以“切到某个源跑通”证明不了用的是哪个源。两个 URL 内容一致,是直接下载后比对 sha256 得出的 diff --git a/.agents/docs/fix-plan-issue-15-16.md b/.agents/docs/fix-plan-issue-15-16.md new file mode 100644 index 0000000..7f5a45b --- /dev/null +++ b/.agents/docs/fix-plan-issue-15-16.md @@ -0,0 +1,771 @@ +# tinyhttps 修复 / 优化方案 —— issue #15 · #16 及连带发现 + +| | | +|---|---| +| 基准版本 | `origin/master @ 2cec1c1`,`mcpp.toml` version = **0.2.10** | +| 目标版本 | **0.2.11**(P0+P1)/ 0.3.0(P3 结构性) | +| 关联 issue | [#15](https://github.com/mcpplibs/tinyhttps/issues/15) 连接池污染 · [#16](https://github.com/mcpplibs/tinyhttps/issues/16) SIGPIPE | +| 文档日期 | 2026-09-06 | + +> **动手前先 `git pull`。** 当前工作区在 `626c9d0`(0.2.8),落后 origin/master 两个提交(`965d805` #9、`2cec1c1` #14)。本文所有行号对应 **0.2.10**;本地 `src/http.cppm` 只有 1032 行,对不上。 + +**标注约定**:= 已逐行核验源码 · = 已独立复现 · = 推断,未实测 + +--- + +## 0. TL;DR + +两个 issue **都是库的真 bug**,不是使用方法错误。核验过程中另外发现 **7 处同类问题**,其中 3 处在 `download_to_file_impl` 的**正常路径**上(重定向、非 2xx、文件打开失败),1 处是服务端可单凭一个 header 触发的整数截断。 + +合计 **19 个修复项**,分 4 档: + +| 档 | 内容 | 项数 | 改动量 | 风险 | +|---|---|---|---|---| +| **P0** | SIGPIPE(#16)—— 库会杀死宿主进程 | 2 | ~10 行 | 零(成功路径不变) | +| **P1** | 连接池不变量(#15 + 4 处新发现) | 10 | ~30 行 | 低(只改早退路径) | +| **P2** | 纵深防御 + 解析加固 + API 缺口 | 9 | ~150 行 | 中(有 1 项 API 变更) | +| **P3** | 结构性重构(让这类 bug 编译期就写不出来) | 4 | 大 | 需单独 PR | + +**建议提交顺序**:P0 单独一个 PR 先合 → P1 一个 PR → 发 0.2.11 → P2 分批 → P3 单独立项。 + +**P0 必须排在 P1 之前**:P1 的修复会让 `pool_.erase()` 在更多路径上执行,而 `erase` 会析构 `TlsSocket` → `close_notify` → `send()`。P0 没修之前,P1 等于**扩大** SIGPIPE 的触发面。 + +--- + +## 1. 问题全景 + +### 1.1 三个入口的加固覆盖矩阵 + +库里有三份几乎平行的响应读取实现:`send_impl`、`send_stream`、`download_to_file_impl`。历次修复各自只落到其中一两份: + +| 加固措施 | 引入自 | `send_impl` | `send_stream` | `download_to_file_impl` | +|---|---|---|---|---| +| `parse_content_length`(严格) | #14 | `:516` | `:783` | `:1042-1048` 手写摘数字 | +| `contentLength` 用 `int64_t` | — | `:488` **`int`** | `:756` | `:1024` | +| `parse_chunk_size_line`(严格) | #9 | `:541` 仍是 `parse_hex` | `:831` | `:1134` | +| `read_complete_line`(区分超时/EOF) | #9 | no | | `:1118/1145/1177` | +| 块后 CRLF 校验 | #9 | `:556` 丢弃返回值 | `:850` 丢弃返回值 | `:1177-1185` | +| 状态行 `HTTP/` 前缀校验 | — | no | | no | +| 早退时清理连接池 | — | 2 处 | 3 处 | **5 处** | +| 失败时能告诉调用方 | — | 无字段 | 无字段 | `result.error` | + +**这张表本身就是最强的论据**:每一次修复都是在一份拷贝上打补丁,另外两份继续带病。`#14` 引入 P1-A1/A2 两处新回归,正是这个模式的最新一次复发。**不做 P3,下一次还会有。** + +### 1.2 四个根因 + +| 根因 | 说明 | 派生问题 | +|---|---|---| +| **R1 池不变量没有被代码表达** | 「可复用」被编码成 `is_valid()`(fd 开着),真正的条件是「流干净 + 对端还在」。这个条件靠 10 个早退路径各自记得设 `connectionClose` 来维护 | P1 全部 | +| **R2 `read_line` 空返回三义** | `read_line`(`:169-191`)在**超时 / EOF / 真的读到空行**三种情况下都返回 `""`,调用方无从区分 | P1-B2、P2-C1 | +| **R3 `TlsSocket::read` 返回 0 二义** | `tls.cppm:124-130`:**「对端关闭」和「WANT_READ/WANT_WRITE 暂时无数据」都返回 0**。`read() <= 0` 因此分不清 EOF 与阻塞 | P2-D1 | +| **R4 库在自己的 fd 上用裸 `send`** | 缺 `MSG_NOSIGNAL`,把信号处置的责任推给了调用方 | P0 | + +R2/R3 是同一个毛病的两个实例:**用一个哨兵值(空串 / 0)表示多种语义**。`#9` 已经引入了 `expected` / `optional` 版本的正确做法,只是没铺开。 + +--- + +## 2. P0 —— SIGPIPE(issue #16) + +### 2.1 判定 + +**真 bug,最高优先级。** 一个库在自己管理的 fd 上调 `send()`,却让宿主进程被 `SIGPIPE` 杀死(exit 141),是库的责任。 + +**关键补充证据(issue 里没有)**:mbedtls 3.6.1 自己在 `library/net_sockets.c:114` 的 `net_prepare()` 里有 `signal(SIGPIPE, SIG_IGN)`,但 tinyhttps 在 `tls.cppm:82/94/240` 用 `mbedtls_ssl_set_bio()` 挂了自定义 BIO,**全仓库不存在任何 `mbedtls_net_init` / `mbedtls_net_context` 调用**(已 grep 确认)。 + +> 也就是说:换掉 mbedtls 网络层时,把它自带的那层防护一并丢掉了,却没补等价物。**这是引入型缺陷,不是「从来没考虑过」。** +> +> 反过来看这也是个改进机会:mbedtls 的做法(进程级 `SIG_IGN`)其实很粗暴,库不该改宿主的全局信号处置。**用 `MSG_NOSIGNAL` 比 mbedtls 原来的做法更干净。** + +独立复现(`sigpipe_probe.c`,不依赖构建整个库): + +``` +[A] flags = 0 → KILLED by signal 13 (Broken pipe) → exit status 141 ← 与报告者实测一致 +[B] flags = MSG_NOSIGNAL → send 返回 -1/EPIPE,进程正常退出 +``` + +### 2.2 修复 + +全库 fd 只有**一个创建点**(`socket.cppm:125`),`proxy.cppm` 也复用 `Socket` 类,所以只需两处改动。 + +#### P0-1 `src/socket.cppm:176-179` —— 发送时抑制信号(Linux / Android / Termux) + +```diff + int write(const char* buf, int len) { + if (!is_valid()) return -1; +- return static_cast(::send(fd_, buf, len, 0)); ++ // A write to a socket whose peer has gone away raises SIGPIPE, and a ++ // program that has not disarmed it — the default — is killed rather ++ // than told. MSG_NOSIGNAL turns that into EPIPE, which this codebase ++ // already knows how to carry: bio_send (tls.cppm:49-51) maps a ++ // non-positive return to MBEDTLS_ERR_NET_SEND_FAILED, and ++ // TlsSocket::close (tls.cppm:151) already ignores what close_notify ++ // returns. Nothing downstream needs to change. ++ // ++ // #ifdef, not `if constexpr`: the macro does not exist on Windows, so ++ // both arms of an `if constexpr` would have to compile and one cannot. ++ // This is the trap 5e7d66f fixed in the resolver stubs. ++#ifdef MSG_NOSIGNAL ++ return static_cast(::send(fd_, buf, len, MSG_NOSIGNAL)); ++#else ++ return static_cast(::send(fd_, buf, len, 0)); ++#endif + } +``` + +#### P0-2 `src/socket.cppm:125` 之后 —— macOS / *BSD 没有 `MSG_NOSIGNAL` + +```diff + bool connect_addrinfo(struct addrinfo* result, int timeoutMs) { + for (auto* rp = result; rp != nullptr; rp = rp->ai_next) { + SocketHandle fd = ::socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); + if (fd == INVALID_SOCKET_FD) { + continue; + } + ++ // macOS and the BSDs spell it as a socket option instead of a send ++ // flag. Best-effort: a failure here only means the socket keeps the ++ // default disposition, which is what it had before this line. ++#ifdef SO_NOSIGPIPE ++ int nosigpipe = 1; ++ ::setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, ++ reinterpret_cast(&nosigpipe), sizeof(nosigpipe)); ++#endif ++ + // Set non-blocking + if (!set_non_blocking(fd, true)) { +``` + +**平台矩阵**:Linux/Android 走 `MSG_NOSIGNAL`;macOS/BSD 走 `SO_NOSIGPIPE`;Windows 两个宏都不存在 → 自动走 `#else`,且本来就没有 `SIGPIPE`,无需额外分支。`socket.cppm:8-9` 已经在非 Windows 下 include 了 ``,两个宏都在里面,**不需要新增 include**。 + +### 2.3 为什么这个修复是自包含的 + +失败会沿着**已经存在的**路径传播,零新增分支: + +``` +Socket::write → -1 (EPIPE) + → bio_send (tls.cppm:49-51) 已有 if (ret <= 0) return MBEDTLS_ERR_NET_SEND_FAILED; + → mbedtls_ssl_close_notify 返回错误 + → TlsSocket::close (tls.cppm:151) 本来就忽略返回值 +``` + +**成功路径逐字节不变。** + +--- + +## 3. P1 —— 连接池不变量(issue #15 + 4 处新发现) + +### 3.1 判定 + +**真 bug,不是使用方法错误。** + +| 报告者做的 | 是否合法 | +|---|---| +| 单个 `HttpClient` 跑多个请求 | 这正是连接池的用途 | +| `keepAlive` 用默认值(true) | 库的默认值 | +| `readTimeoutMs = 1000` | 激进但合法;**超时是正常事件,不是 API 误用** | +| 触发条件:服务端发一半 / 空闲后关连接 | **服务端的常规行为** | + +规避手段只有「关掉 keep-alive」或「每次换 client」—— 都是放弃这个特性本身。**用户无法在正确使用 API 的前提下绕开 → 定义上就是库的 bug。** + +### 3.2 完整泄漏路径清单(10 条) + +`connectionClose` 列 = 走到清理点时该标志的值。 + +#### A. `send_stream` —— **0.2.10 新引入的回归** + +| ID | 位置 | 触发 | 现状 | 修复 | +|---|---|---|---|---| +| **A1** | `:878-880` | `wait_readable` 超时 → `break` | `false` → `:909` 保留 socket | 设 `connectionClose = true` | +| **A2** | `:884-885` | `read() <= 0` → `break` | `false` → 同上 | 设 `connectionClose = true` | +| **A3** | `:846-848` | `read_exact` 失败 → `break` | `false` → 同上 | 设 `connectionClose = true` | + +**回归定位** :`git log -S "A DECLARED LENGTH IS READ AND THE READER THEN STOPS"` → `2cec1c1` (#14)。0.2.3 的对应分支是: + +```cpp +} else { + // Not chunked — read until connection closes + connectionClose = true; // ← 无条件设置,池永远安全 +``` + +`#14` 为修「keepAlive 下 error body 晚到一个 readTimeout」新增了 Content-Length 分支,**顺手把这句无条件赋值拿掉了**。A1/A2 是 0.2.10 才有的新回归;A3 是旧疾(0.2.3 同样存在)。 + +#### B. `send_impl` + +| ID | 位置 | 触发 | 现状 | 修复 | +|---|---|---|---|---| +| **B1** | `:550-551` | `read_exact` 失败 → `break` | `false` → `:582` 不清理 | 设 `connectionClose = true` | +| **B2** | `:541-545` | `parse_hex("") == 0` → 当成终止块 | **把超时当成 body 正常结束** | 换 `parse_chunk_size_line` | + +B2 细节:`parse_hex`(`:233-243`)遇到不认识的字符是 `break` 返回**已累积的值**,空串返回 0。所以 + +- 超时的空行 → `0` → `:542` 认为是终止块 → 池被污染 **且静默报告成功** +- 残留的 `"BBBB"` → `0xBBBB = 48059` → `read_exact` 去等 48059 个永不到来的字节 + +`parse_chunk_size_line`(`:277-288`)用 `from_chars` 全串校验,两种情况都返回 `nullopt`。**`send_impl:541` 是全库最后一个 `parse_hex` 调用点**,改完可以直接删掉 `parse_hex`。 + +#### C. `download_to_file_impl` + +| ID | 位置 | 触发 | 现状 | 来源 | +|---|---|---|---|---| +| **C1** | `:1166-1171` | chunk 数据 `read_exact` 失败 → **直接 return** | 跳过 `:1223` 尾声 | issue #15 | +| **C2** | `:1195-1200` | CL body `read_exact` 失败 → **直接 return** | 同上 | issue #15 | +| **C3** | `:1060-1075` | **重定向** | **正常路径** | 本次发现 | +| **C4** | `:1077-1081` | **非 2xx 返回** | **正常路径** | 本次发现 | +| **C5** | `:1092-1096` | 输出文件打不开 | | 本次发现 | + +C1/C2 的对照:同一个函数里相邻的 `:1150-1153` 和 `:1180-1184` **写对了**(`sock->close(); pool_.erase(poolKey);`),这本身就证明是漏写而非取舍。 + +##### C3 是本次分析最值得注意的一条 + +```cpp +1062: // Drain any body to keep connection clean +1063: if (connectionClose) { +1064: sock->close(); +1065: pool_.erase(poolKey); +1066: } +``` + +**注释说要 drain,但代码里没有任何 drain。** 只有在服务端主动送 `Connection: close` 时才清理。于是: + +服务端回 `302` + `Content-Length: 123` + 一段 HTML + `keep-alive`(**这是绝大多数服务端的重定向响应**) +→ 123 字节 body 一个都没读 +→ socket 带着这 123 字节回池 +→ `:1073` 递归调用 `download_to_file_impl` +→ 若重定向到同一 host,`:954` 拿到这个被污染的 socket +→ `:1002` `read_line` 把重定向 body 的头几十个字节当成状态行 + +**这就是 #15 的失效模式,发生在完全正常的重定向路径上,不需要任何超时或截断。** `send_impl` 没有这个问题,因为它是先读完 body(`:524-579`)再处理重定向(`:588`);`download_to_file_impl` 的顺序反过来。 + +C4 同理:非 2xx 时 error body 未读就返回,socket 回池,同一 client 的下个请求中招。 +C5 同理:文件打不开时 body 未读。 + +#### D. 整数截断 + +| ID | 位置 | 问题 | +|---|---|---| +| **D1** | `:488` | `int contentLength = -1;`,而 `send_stream:756` / `download:1024` 都是 `std::int64_t` | + +`:516` `contentLength = parse_content_length(valStr).value_or(-1);` —— 右边是 `int64_t`,**赋给 `int` 静默截断**。 + +服务端发 `Content-Length: 4294967296`(2³²)→ 截断成 `int` **0** → `:560` `if (contentLength > 0)` 不成立 → **一个字节都不读** → `:582` `connectionClose` 为 false → **socket 带着 4 GiB 未读 body 回池**。 + +**这是唯一一条服务端单凭一个 header 就能触发的池污染,不需要超时、不需要截断、不需要任何时序配合。** + +顺带:`:561` `response.body.resize(contentLength)` 在 `contentLength` 接近 `INT_MAX` 时会先分配 2 GiB(见 P2-E2)。 + +### 3.3 修复 diff + +#### P1-A `src/http.cppm` —— `send_stream` 三处 + +```diff +@@ -844,7 +844,10 @@ (chunked 分支) + std::string chunkData(chunkSize, '\0'); + if (!read_exact(*sock, chunkData.data(), chunkSize, config_.readTimeoutMs)) { ++ // The body stopped mid-chunk: bytes are still owed on this ++ // socket, so it must not go back into the pool. ++ connectionClose = true; + break; + } +@@ -877,10 +880,12 @@ (Content-Length 分支) + while (!stopped && remaining > 0) { + if (!sock->wait_readable(config_.readTimeoutMs)) { ++ connectionClose = true; // remaining > 0: bytes still owed + break; + } + ... + int ret = sock->read(buf, want); +- if (ret <= 0) break; ++ // End of stream before the declared length — the framing is ++ // broken either way, so the connection cannot be reused. ++ if (ret <= 0) { connectionClose = true; break; } +``` + +> 已核验:`:909-912` 的清理本来就 key 在 `connectionClose || stopped` 上;正常终止路径(`remaining > 0` 循环条件自然结束、`:841` 终止块 break)**不经过**这三个出口。**成功路径零变化。** + +#### P1-B `src/http.cppm` —— `send_impl` 两处 + +```diff +@@ -538,15 +538,21 @@ + while (!sizeLine.empty() && (sizeLine.back() == ' ' || sizeLine.back() == '\t')) { + sizeLine.pop_back(); + } + +- int chunkSize = parse_hex(sizeLine); +- if (chunkSize == 0) { ++ // A CHUNK HEADER THAT DOES NOT PARSE IS NOT A TERMINAL CHUNK. ++ // parse_hex returned 0 for an empty line, and read_line returns ++ // an empty line on a timeout — so a stream that was cut short ++ // read as one that ended cleanly, and the socket went back into ++ // the pool with the rest of the body still owed on it. ++ // #9 established parse_chunk_size_line for exactly this; it ++ // reached send_stream and download_to_file but not this reader. ++ auto parsedChunkSize = parse_chunk_size_line(sizeLine); ++ if (!parsedChunkSize) { ++ connectionClose = true; ++ break; ++ } ++ int chunkSize = static_cast(*parsedChunkSize); ++ if (chunkSize == 0) { + read_line(*sock, config_.readTimeoutMs); + break; + } + + std::string chunkData(chunkSize, '\0'); + if (!read_exact(*sock, chunkData.data(), chunkSize, config_.readTimeoutMs)) { ++ connectionClose = true; + break; + } +``` + +改完后 `parse_hex`(`:233-243`)已无调用点 → **一并删除**。 + +#### P1-C `src/http.cppm` —— `download_to_file_impl` 五处 + +C1 / C2(照搬相邻 `:1151-1152` 的写法): + +```diff +@@ -1166,6 +1166,8 @@ + if (!read_exact(*sock, buf, toRead, config_.readTimeoutMs)) { + result.error = "Read error during chunked transfer"; + ofs.close(); + result.bytesWritten = downloaded; ++ sock->close(); ++ pool_.erase(poolKey); + return result; + } +@@ -1195,6 +1197,8 @@ + if (!read_exact(*sock, buf, toRead, config_.readTimeoutMs)) { + result.error = "Read error"; + ofs.close(); + result.bytesWritten = downloaded; ++ sock->close(); ++ pool_.erase(poolKey); + return result; + } +``` + +C3 / C4 / C5 —— 最小改法是**无条件丢弃连接**(见 P2-E1 有更省的做法): + +```diff +@@ -1059,10 +1063,12 @@ (重定向) + if (result.statusCode >= 300 && result.statusCode < 400 && + !location.empty() && redirectCount < config_.maxRedirects) { +- // Drain any body to keep connection clean +- if (connectionClose) { +- sock->close(); +- pool_.erase(poolKey); +- } ++ // The comment here used to promise a drain that was never written: ++ // a 3xx normally carries a short body, and leaving it unread put a ++ // socket with owed bytes back into the pool — which the recursive ++ // call below then picks up for the very next request. Until there ++ // is a bounded drain (see P2-E1), the connection is not reusable. ++ sock->close(); ++ pool_.erase(poolKey); + +@@ -1077,7 +1083,8 @@ (非 2xx) + if (result.statusCode < 200 || result.statusCode >= 300) { + result.error = "HTTP " + std::to_string(result.statusCode); +- if (connectionClose) { sock->close(); pool_.erase(poolKey); } ++ sock->close(); pool_.erase(poolKey); // error body was never read + return result; + } + +@@ -1092,7 +1099,8 @@ (文件打不开) + if (!ofs) { + result.error = "Cannot open file: " + destFile.string(); +- if (connectionClose) { sock->close(); pool_.erase(poolKey); } ++ sock->close(); pool_.erase(poolKey); // body was never read + return result; + } +``` + +> **review 决策点**:C3 用「无条件丢连接」换正确性,代价是每次重定向多一次 TLS 握手。若在意,P2-E1 提供一个有上限的 drain 助手,可把这三处改回「drain 成功就留、失败就丢」。**建议先合无条件版本(正确性优先),drain 作为后续优化。** + +#### P1-D `src/http.cppm:488` —— 类型对齐 + +```diff + bool chunked = false; +- int contentLength = -1; ++ std::int64_t contentLength = -1; // matches send_stream:756 / download:1024 + bool connectionClose = false; +``` + +同时 `:560-565` 需要跟着调整(`resize` 的参数类型 + 上限,见 P2-E2)。 + +--- + +## 4. P2 —— 纵深防御 · 解析加固 · API 缺口 + +> P2 的价值:**即使 P1 将来又出漏洞,残留字节也不会被伪装成一个合法响应。** #15 里最危险的两档(静默数据损坏)都会退化成可见错误。 + +### P2-C1 状态行必须以 `HTTP/` 开头 + +三个入口(`:456-484`、`:725`、`:1010-1020`)**都不校验前缀**,状态码提取还是「挑数字、静默跳过非数字」: + +``` +"BBB…B 999 XHTTP/1.1 200 OK" + ↑ 第一个空格 → rest = "999 XHTTP/1.1 200 OK" + ↑ 第二个空格 → code = "999", text = "XHTTP/1.1 200 OK" +``` + +这就是 issue #15 里 `statusCode=999` 的完整来源。而且 `X9Y9Z9` 也会被解析成 `999`。 + +建议抽一个共用的严格解析器: + +```cpp +struct StatusLine { int code; std::string text; }; + +// A status line is HTTP-version SP status-code [SP reason] (RFC 9112 §4). +// Rejecting anything else is what stops another response's leftover bytes +// from being read as a status line — see issue #15. +static std::optional parse_status_line(std::string_view line) { + if (!line.starts_with("HTTP/")) return std::nullopt; + auto sp = line.find(' '); + if (sp == std::string_view::npos) return std::nullopt; + auto rest = line.substr(sp + 1); + auto sp2 = rest.find(' '); + auto codeStr = (sp2 == std::string_view::npos) ? rest : rest.substr(0, sp2); + if (codeStr.size() != 3) return std::nullopt; // exactly three digits + int code {}; + auto [end, ec] = std::from_chars(codeStr.data(), codeStr.data() + codeStr.size(), code); + if (ec != std::errc{} || end != codeStr.data() + codeStr.size()) return std::nullopt; + return StatusLine{ code, sp2 == std::string_view::npos + ? std::string{} : std::string(rest.substr(sp2 + 1)) }; +} +``` + +三个入口统一改用它,失败一律 `pool_.erase` + `Invalid status line`。**导出它,理由和 `parse_chunk_size_line` 一样:这是 framing 里可以脱离服务端单测的那一半。** + +### P2-C2 header 循环区分「超时」与「真空行」 + +`:491-495` / `:758-762` / `:1030-1032` 三处都是: + +```cpp +std::string headerLine = read_line(*sock, config_.readTimeoutMs); +if (headerLine.empty()) break; // "End of headers" +``` + +读 header 期间超时 → `read_line` 返回空 → 当成 header 读完 → 拿一份**残缺的 header 集合**去读 body(`contentLength` 还是 -1、`chunked` 还是 false)→ 走「读到连接关闭为止」分支。 + +那条分支会设 `connectionClose = true`,所以**池是安全的**,但**响应是静默错误的**:header 少一半,body 里混进剩下的 header 文本。 + +修复:三处都换成 `read_complete_line`(`:193-213`,返回 `expected`),`unexpected` → 明确报错并丢连接。**根因 R2 的正解,`#9` 已经写好了工具,只是没铺开。** + +### P2-C3 块后 CRLF 校验 + +`send_impl:556` 和 `send_stream:850` 都是 `read_line(...)` 丢弃返回值;`download_to_file_impl:1177-1185` 是校验的(`"Missing CRLF after chunk data"`)。块长度算错时前两者会直接错位到下一个块头而不报错。照 download 的写法补齐。 + +### P2-C4 `download_to_file_impl` 的 Content-Length 解析 + +`:1042-1048` 仍是手写摘数字循环: + +```cpp +if (iequals(key, "Content-Length")) { + contentLength = 0; + for (char c : valStr) { if (c >= '0' && c <= '9') contentLength = contentLength * 10 + (c - '0'); } +} +``` + +`:245-252` 的注释亲口描述的正是这个缺陷(「`12abc` 是 12,`abc` 是 0,溢出静默回绕」),`#14` 修了 `send_impl`/`send_stream`,**唯独漏了这里**。换成 `parse_content_length(valStr).value_or(-1)`。 + +### P2-E1 有上限的 body drain(可选,给 C3/C4 省一次握手) + +```cpp +// Consume a body we are not going to hand to the caller, so the connection +// stays reusable. Bounded: past the cap it is cheaper to reconnect than to +// keep reading, and an endless body must not be able to stall us. +// Returns false if the body could not be fully consumed — drop the connection. +static bool drain_body(TlsSocket& sock, bool chunked, std::int64_t contentLength, + int timeoutMs, std::int64_t cap = 64 * 1024); +``` + +有了它,C3/C4 可以写成 `if (!drain_body(...)) { sock->close(); pool_.erase(poolKey); }`。 + +### P2-E2 分配上限(拒绝服务端诱导的巨额分配) + +- `send_impl:561` `response.body.resize(contentLength)` —— `contentLength` 最大到 `INT64_MAX`,`resize` 会抛 `std::length_error` / `std::bad_alloc`。**库没有文档说 `send()` 会抛异常**,调用方大概率没接。 +- `send_impl:549` / `send_stream:845` `std::string chunkData(chunkSize, '\0')` —— 服务端一个 `7fffffff` 块头就能让客户端分配 2 GiB。 + +建议:加一个 `HttpClientConfig::maxResponseBodyBytes`(默认比如 64 MiB),超过就当作 framing 错误拒绝;chunk 数据改成按固定缓冲区分片读,不按块大小一次性分配。 + +### P2-D1 `TlsSocket::read` 区分 EOF 与 would-block(根因 R3) + +`tls.cppm:120-134`:`PEER_CLOSE_NOTIFY`、`ret == 0`、`WANT_READ`、`WANT_WRITE` **全部返回 0**。调用方(`read_exact:157`、`read_line:178`、`send_stream:885`)因此分不清「对端关了」和「暂时没数据」,只好用「等一下再试一次」的土办法遮盖。 + +建议引入三态: + +```cpp +enum class ReadStatus { Data, WouldBlock, Eof, Error }; +``` + +或保留 `int` 但用 `-2` 表示 would-block。 + +> **与 P1 的关系**:P1-A2 把 `read() <= 0` 当成「流结束」丢连接。若那个 0 其实是瞬时 would-block,我们只是白丢一条连接 —— **正确性不受影响,只损失一点复用率**。所以 P1 不依赖 P2-D1,但修了 D1 之后 P1-A2 才精确。实践中 socket 在握手后被设回阻塞模式(`socket.cppm`),`WANT_*` 极少出现,影响应该很小,**未实测**。 + +### P2-D2 `write_all` 的 WANT_WRITE 处理 + +`http.cppm:216-230`: + +```cpp +int ret = sock.write(...); +if (ret < 0) return false; +if (ret == 0) { + ret = sock.write(...); // 立刻重试一次,不等待 + if (ret <= 0) return false; // 再失败就放弃 +} +``` + +`TlsSocket::write`(`tls.cppm:140-142`)在 `WANT_READ/WANT_WRITE` 时返回 0。所以 TLS 背压下会「忙重试一次然后放弃」→ 用户看到莫名其妙的 `"Write failed"`。应该改成 `wait_writable(timeoutMs)` 后重试,并给总时长设上限。**未实测,按代码推断。** + +### P2-F1 调用方无法得知 body 被截断了(API 缺口) + +**这是 P1 修完之后剩下的最大问题。** + +P1 只保证了「坏连接不会毒害下一个请求」,但当前请求返回给调用方的是: + +- `send_impl` CL 短读(`:562-565`)→ 返回真实的 `statusCode=200` + 半截 body,**没有任何截断标记** +- `send_impl` / `send_stream` chunked 中断 → 同上 +- `DownloadToFileResult` **有** `error` 字段 ,`HttpResponse` **没有** + +也就是说:**一个被截断的 200 响应,和一个完整的 200 响应,调用方分不出来。** + +建议(**加字段,不改现有语义,非破坏性**): + +```cpp +export struct HttpResponse { + int statusCode { 0 }; + std::string statusText; + std::map headers; + std::string body; + + // False when the body ended before its framing said it would — a read + // timeout, an EOF mid-body, a chunk header that did not parse. The status + // line and headers are still what the server sent; `body` is a prefix. + // DownloadToFileResult has always carried this as `error`; send() and + // send_stream() had no way to say it. + bool bodyComplete { true }; + + bool ok() const { return statusCode >= 200 && statusCode < 300; } + // 若担心旧代码只看 ok(),可考虑让 ok() 也要求 bodyComplete —— 但这是行为变更, + // 需要单独讨论。默认保持 ok() 语义不变。 +}; +``` + +在 P1 设 `connectionClose = true` 的每一处,同时设 `response.bodyComplete = false`。 + +> **review 决策点**:(a) 只加 `bodyComplete` 字段、`ok()` 不变(建议,非破坏性);(b) 让 `ok()` 也要求 `bodyComplete`(更安全,但会让现有代码行为变化)。 + +### P2-F2 `send_stream` 出错时会覆盖真实的 statusText + +`:833` `response.statusText = "Invalid chunk size: " + sizeLine;` —— 把服务端真实的 `"OK"` 覆盖掉了。有了 P2-F1 的 `bodyComplete` 之后,应该改成保留 `statusText`,把原因放进新字段(或一个 `bodyError` 字符串)。 + +--- + +## 5. P3 —— 结构性重构(建议单独立项) + +> P1 是逐条堵漏。**但 §1.1 的矩阵说明:这个 bug 类会随着每次新增代码复发。** `#14` 就是最新一次。P3 的目标是让它写不出来。 + +### P3-A `PooledConn` —— 把默认行为从「留」翻成「丢」 最高性价比 + +当前所有 10 个 bug 的形状**完全一样**:一条早退路径**什么都没做**,而「什么都没做」的默认结果是**把脏连接留在池里**。 + +```cpp +// The pool's invariant — a pooled connection owes no bytes and its peer is +// still there — is not something is_valid() can check. So make dropping the +// default: any path that leaves without an explicit keep() drops the socket. +// Every defect in issue #15 was a path that did nothing. +class PooledConn { +public: + PooledConn(std::map& pool, std::string key) + : pool_(pool), key_(std::move(key)) {} + ~PooledConn() { + if (!keep_) { + if (auto it = pool_.find(key_); it != pool_.end()) { + it->second.close(); + pool_.erase(it); + } + } + } + // Call only where the body was consumed to its declared end and the + // response did not say Connection: close. + void keep() { keep_ = true; } + PooledConn(const PooledConn&) = delete; + PooledConn& operator=(const PooledConn&) = delete; +private: + std::map& pool_; + std::string key_; + bool keep_ { false }; +}; +``` + +用法:每个入口开头建一个,只在 body **正常读完**的那唯一一处调 `keep()`。10 个 bug 一次全消,而且以后新增任何早退路径**天然安全**。 + +> **两个必须注意的陷阱(review 重点)**: +> +> 1. **递归重定向**。`send_impl:612` `return send_impl(redirectReq, redirectCount + 1);` 发生在 guard 析构**之前**。若重定向到同一 host,内层调用会往池里放一条新连接,外层 guard 析构时会把它**误删**。 +> → 递归前必须先 `guard.keep()` 或 `guard.release()`。`download_to_file_impl:1073` 同理。 +> **现状没有这个 bug**,因为现在的清理(`:582-585`)在重定向块之前执行 —— 改成 RAII 时必须专门处理。 +> 2. **必须在 P0 之后做**。guard 析构 → `erase` → `~TlsSocket` → `close_notify` → `send()`。P0 没修就是在扩大 SIGPIPE 面。 + +### P3-B 三份 reader 合一 + +§1.1 的矩阵就是三份平行实现的代价。建议抽出: + +```cpp +struct ResponseHead { int statusCode; std::string statusText; + std::map headers; + bool chunked; std::int64_t contentLength; bool connectionClose; }; + +// 统一的 head 读取:状态行 + header,严格解析 +static std::expected read_response_head(TlsSocket&, int timeoutMs); + +// 统一的 body 读取,把 body 的去向抽象成 sink(string / SSE 回调 / ofstream) +enum class BodyEnd { Complete, Truncated, ClosedByPeer }; +static BodyEnd read_body(TlsSocket&, const ResponseHead&, int timeoutMs, + std::function sink); +``` + +`[[nodiscard]] BodyEnd` 让「读完之后要不要留连接」变成一个**必须处理的返回值**,而不是一个可以忘记设的 bool。**增量做法**:先抽 `read_response_head`(三处几乎逐字重复,最安全),body 之后再说。 + +### P3-C 带缓冲的读 + +`read_line` 现在是逐字节 `sock.read(&c, 1)`。除了慢(影响有限:`TlsSocket::wait_readable` 会先查 `mbedtls_ssl_get_bytes_avail`,所以并不是每字节一次 `poll()`,主要成本是每字节一次函数调用 —— **未做基准测试,不要当成性能修复来卖**),更重要的是: + +**一个显式的读缓冲,才能让「这条连接干净吗」这个问题真正可回答**(= 缓冲区空 + body 已按 framing 读完)。现在这个问题只能靠人肉追踪 10 条路径。这是 P3-C 的主要理由,性能是副产品。 + +### P3-D 池连接失效时自动重连重试(issue #15 里被当引子带过的那条) + +**即使 P0+P1+P2 全修完,这个场景依然会失败:** + +服务端在两次请求之间关闭空闲 keep-alive 连接(**常规行为**) +→ `:357` `is_valid()` 为 true(fd 还开着)→ 复用 +→ `:439` `write_all` 成功(第一个 TCP 写进缓冲区不报错) +→ `:447` `read_line` 返回空 +→ `:449` → `statusCode=0, statusText="No response"` + +报告者观察到「服务端日志里没有第二条连接、没有第二个请求」—— **客户端根本没重连**。 + +P0+P1 只是把「进程被杀」降级成「一个莫名其妙的 `No response`」。curl(`Curl_retry_request`)、Go `net/http`(`shouldRetryRequest`)、Python requests 都会在这种情况下静默重连重试一次,因为这个失败与请求内容无关,纯粹是池的簿记问题。 + +建议: + +``` +若 (连接取自池中的复用连接) 且 (在收到任何响应字节之前失败): + 丢弃连接 → 新建连接 → 重发一次(仅一次) +``` + +> **review 决策点 —— 要不要限制到幂等方法?** +> +> - Go 的策略保守:只重试幂等方法(GET/HEAD/OPTIONS/TRACE)或带 `Idempotency-Key` 的请求。 +> - curl 的策略宽松:只要连接是复用的就重试,不看方法。 +> +> **我倾向 curl 的做法**,理由:(1) tinyhttps 的主力场景是 LLM / API 客户端,POST 是绝对多数,限制到幂等方法等于这个修复基本没用;(2) 重试窗口极窄 —— 只在**一个响应字节都没收到**时才重试,此时服务端已经发过 FIN,应用层根本没看到这个请求;(3) 加一个 `HttpClientConfig::retryOnStaleConnection`(默认 true)让保守用户能关掉。 +> +> 残留风险:服务端「收到并处理了请求 → 还没回响应就崩了/关了」这种情况下会重复执行。窄,但非零。**这一条请你拍板。** + +--- + +## 6. 测试方案 + +### 6.1 现状 + +`tests/` 下只有 `test_download.cpp` 和 `test_resolver.cpp`,**零 keep-alive / 连接复用覆盖** —— 这就是 `#14` 的回归能全绿合进来的直接原因。 + +报告者主动提出「可以把 Content-Length 和 chunked 两个 case 写成对着本地 listener 的 gtest,风格照 `test_download.cpp`」,而且他手上已经有能跑的复现脚本(一个 Python server + 一个 `main.cpp` + `run.sh`)。**建议直接接受这个 offer**,这是最省事也最可信的路径。 + +### 6.2 必须覆盖的用例 + +| # | 用例 | 断言 | 覆盖 | +|---|---|---|---| +| T1 | 服务端发一半 CL body 后挂住 | 请求 2 `statusCode == 200`(不是 999/0/Invalid) | A1 | +| T2 | **同上,并断言服务端 TCP 连接数 == 2** | **唯一能抓住「50 个 B」那种「看起来对了」的 case 的断言** | A1 | +| T3 | chunked 版本同上 | 同 T1 + T2 | A3 | +| T4 | 服务端发一半后关连接 | 请求 2 不崩 + 断言**进程未被信号杀死** | #16 | +| T5 | **`SIGPIPE` 保持默认处置**跑 T4(子进程) | `WIFSIGNALED == false`,退出码 != 141 | P0 | +| T6 | `Content-Length: 4294967296` | 不得静默当成 0;连接不得回池 | D1 | +| T7 | 302 + `Content-Length` body + keep-alive → 同 host | 重定向后的下载内容正确 + 连接数正确 | **C3** | +| T8 | 404 + error body + keep-alive,同 client 再发一个请求 | 第二个请求正确 | **C4** | +| T9 | chunk 头是 `zz` / 空行 | 报错,不得当成终止块 | B2 | +| T10 | 状态行是 `BBBB 999 XHTTP/1.1 200 OK` | `Invalid status line`,不是 `999` | P2-C1 | +| T11 | 截断的 body | `bodyComplete == false` | P2-F1 | + +**T2 是整套测试里最关键的一条。** 没有连接数断言,issue #15 表格第三行(50 个 `B`,输出与对照组逐字节相同)**会静默通过**。 + +### 6.3 纯单元测试(不需要服务端) + +`parse_status_line`(P2-C1 新增)、`parse_chunk_size_line`、`parse_content_length` 都是导出的纯函数,可以直接表驱动单测。`#9` 导出它们就是为了这个 —— 沿用这个模式。 + +--- + +## 7. 发布计划与回归风险 + +| 版本 | 内容 | 风险 | +|---|---|---| +| **0.2.11** | P0 + P1(全部 10 项) | **低**。全部只改早退路径,成功路径逐字节不变 | +| 0.2.12 | P2-C1~C4(解析加固) | 中。严格解析会把过去「蒙混过关」的畸形响应变成显式错误 —— **这是本意,但要写进 release note** | +| 0.2.13 | P2-E/D/F(分配上限、would-block、`bodyComplete`) | 中。`bodyComplete` 是加字段,非破坏性 | +| 0.3.0 | P3(结构性 + 重试) | 高,单独立项 | + +**Release note 必须点名的两件事**: + +1. **0.2.10 的 `send_stream` 回归**(A1/A2 由 `#14` 引入)。用 0.2.10 且用了 SSE 流式接口的用户应尽快升级。 +2. **0.2.12 起,过去被静默接受的畸形响应会开始报错。** 这不是新 bug,是过去在静默损坏数据。 + +**P0 的兼容性**:需要在 Linux / macOS / Windows / Termux 四个平台各过一遍 CI。`MSG_NOSIGNAL` 和 `SO_NOSIGPIPE` 都在已 include 的 `` 里,不需要新增 include。注意 `5e7d66f` 的教训 —— **必须 `#ifdef`,不能 `if constexpr`**。 + +--- + +## 附录 A:核验方法 + +```bash +# 本地 checkout 是 0.2.8,issue 行号对应 0.2.10,必须取 origin/master +git show origin/master:src/http.cppm > v0210/http.cppm # 1235 行 +git show origin/master:src/socket.cppm > v0210/socket.cppm +git show origin/master:src/tls.cppm > v0210/tls.cppm + +# 全仓库确认无任何 SIGPIPE 处理 +git grep -n "MSG_NOSIGNAL\|SIGPIPE\|NOSIGPIPE" origin/master # 无命中 + +# A1/A2 回归定位 +git log --oneline -S "A DECLARED LENGTH IS READ AND THE READER THEN STOPS" -- src/http.cppm +# → 2cec1c1 (#14) +git show 0.2.3:src/http.cppm | sed -n '738,741p' # connectionClose = true 无条件 + +# mbedtls 自带的防护(被自定义 BIO 绕过) +grep -n "SIGPIPE" .../mbedtls-3.6.1/library/net_sockets.c # :114 signal(SIGPIPE, SIG_IGN) +git grep -n "mbedtls_net_init" origin/master -- src/ # 无命中 + +# 加固覆盖矩阵 +grep -n "parse_content_length\|parse_hex\|parse_chunk_size_line\|read_complete_line" v0210/http.cppm + +# SIGPIPE 机制独立复现 +gcc -o sigpipe_probe sigpipe_probe.c && ./sigpipe_probe +# flags=0 → signal 13 → exit 141;MSG_NOSIGNAL → 正常返回 -1/EPIPE +``` + +引用到的行号全部在 `origin/master @ 2cec1c1` (v0.2.10) 上逐条打开确认过。 + +## 附录 B:19 个修复项速查 + +| ID | 档 | 文件:行 | 一句话 | +|---|---|---|---| +| P0-1 | P0 | `socket.cppm:176-179` | `send` 加 `MSG_NOSIGNAL` | +| P0-2 | P0 | `socket.cppm:125` | macOS 加 `SO_NOSIGPIPE` | +| P1-A1 | P1 | `http.cppm:879` | `send_stream` CL 超时 → `connectionClose = true` | +| P1-A2 | P1 | `http.cppm:885` | `send_stream` CL EOF → `connectionClose = true` | +| P1-A3 | P1 | `http.cppm:847` | `send_stream` chunk 短读 → `connectionClose = true` | +| P1-B1 | P1 | `http.cppm:551` | `send_impl` chunk 短读 → `connectionClose = true` | +| P1-B2 | P1 | `http.cppm:541` | `parse_hex` → `parse_chunk_size_line`(并删除 `parse_hex`) | +| P1-C1 | P1 | `http.cppm:1166-1171` | download chunk 短读 → `close + erase` | +| P1-C2 | P1 | `http.cppm:1195-1200` | download CL 短读 → `close + erase` | +| P1-C3 | P1 | `http.cppm:1060-1075` | **重定向未 drain body** | +| P1-C4 | P1 | `http.cppm:1077-1081` | 非 2xx 未 drain body | +| P1-C5 | P1 | `http.cppm:1092-1096` | 文件打开失败未 drain body | +| P1-D1 | P1 | `http.cppm:488` | `int contentLength` → `int64_t`(整数截断) | +| P2-C1 | P2 | 三处状态行 | 校验 `HTTP/` 前缀 + 严格状态码 | +| P2-C2 | P2 | 三处 header 循环 | 用 `read_complete_line` 区分超时/空行 | +| P2-C3 | P2 | `:556` `:850` | 校验块后 CRLF | +| P2-C4 | P2 | `:1042-1048` | download 换 `parse_content_length` | +| P2-E2 | P2 | `:561` `:549` `:845` | 分配上限,防 2 GiB 分配 / 未文档化的异常 | +| P2-F1 | P2 | `HttpResponse` | 加 `bodyComplete` —— 调用方现在无法得知 body 被截断 | diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5e43dd0..1da4c4c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,42 +1,374 @@ name: CI +# THE BUILD TOOL IS NAMED HERE AND NOWHERE ELSE. +# +# This workflow used to install it through `.xlings.json', a workspace pin that +# `actions/checkout' places in the working directory before anything else runs. +# Two consequences followed and neither was visible in a passing log: +# +# * every `mcpp' invocation in this repository resolved to the pinned version +# rather than to the one the job had installed, because that pin is read +# from the current directory; +# * the pin named 0.0.13, from 2026-05. The index no longer serves the +# toolchain packages a build tool of that age asks for, so `mcpp build' +# ended at `package 'xim:glibc@>=2.39' not found' before compiling a line. +# +# The pin file is removed. The version is an environment variable, so the two +# places that need it, the install and the cache, cannot drift apart again. + on: push: branches: [master] pull_request: + # A CANARY, BECAUSE THIS ROTTED WHERE NOTHING WOULD LOOK. + # + # The breakage above occurred between 2026-07-11 and 2026-08-29 and nothing + # reported it: master received no pushes in that window, so the first run to + # meet it was a contributor's pull request, whose author had no way to + # distinguish a broken environment from a broken change. + schedule: + - cron: '0 6 * * 1' + workflow_dispatch: + +env: + # 2026.9.30.2 and not earlier: 2026.9.28.3 refused x86_64-windows-musl + # ("cannot be built on this host") before the dependency graph that supplies + # its system side, openkal-llvm-runtime, had been resolved. + MCPP_VERSION: 2026.9.30.2 + XLINGS_VERSION: v2026.8.17.2 + XLINGS_NON_INTERACTIVE: '1' jobs: + # ── Linux, under both standard libraries ───────────────────────────────────── + # + # WHY TWO TOOLCHAINS. The job used to run whatever mcpp installed by default on + # a fresh runner, which is gcc and libstdc++, and nothing ran the tests under + # libc++. libc++ is the standard library of every other job below — macOS and + # openkal — and of mcpp's own default where a developer has chosen llvm; the + # cancellation tests met a libc++ defect that libstdc++ does not have + # (tests/test_cancel.cpp, at the top), and no job here could have seen it. build: - name: build + test (linux x86_64, mcpp) + name: build + test (linux x86_64, ${{ matrix.toolchain }}) runs-on: ubuntu-latest + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + toolchain: ['gcc@16.1.0', 'llvm@22.1.8'] steps: - uses: actions/checkout@v4 - name: Install xlings - env: - XLINGS_VERSION: 0.4.31 run: | - tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" - curl -fsSL -o "/tmp/${tarball}" \ - "https://github.com/d2learn/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" - tar -xzf "/tmp/${tarball}" -C /tmp - "/tmp/xlings-${XLINGS_VERSION}-linux-x86_64/subos/default/bin/xlings" self install + # --retry-all-errors and not --retry alone: the first covers a + # transient HTTP status and a timeout, and what this step actually + # meets is a failure of the transport. Observed in this ecosystem as + # `curl: (35) Recv failure: Connection reset by peer', thirteen + # seconds into a job, before anything was built. + curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 \ + https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh \ + | bash -s "$XLINGS_VERSION" echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - - name: Refresh package index - run: xlings update + - name: Install mcpp + run: | + xlings update + xlings install "mcpp@$MCPP_VERSION" -y -g + # The version this job runs is the version it prints. A pin that is + # not read is the failure this file exists to stop repeating. + mcpp --version + # A runner outside China reaches the mirrors this names. The CN set is + # for a developer's machine and is slower or unreachable from here. + mcpp self config --mirror GLOBAL + + # The toolchain mcpp bootstraps is decided by the version pinned above, so + # that version and the toolchain are the whole of the key. + - name: Cache mcpp sandbox + uses: actions/cache@v4 + with: + path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry + key: mcpp-sandbox-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-${{ matrix.toolchain }} + + - name: Select the toolchain + run: | + spec='${{ matrix.toolchain }}' + mcpp toolchain install "${spec%@*}" "${spec#*@}" + mcpp toolchain default "$spec" + mcpp toolchain list - - name: Install workspace tools (.xlings.json → mcpp 0.0.13) - run: xlings install -y + - name: Build with mcpp + run: mcpp build + + # Most of these are hermetic: `test_framing` needs nothing, and `test_pool`, + # `test_proxy` and `test_cancel` script their own servers on 127.0.0.1. The + # eight in `test_download` and `test_resolver` reach httpbin.org and + # one.one.one.one; a network failure among them is distinguishable in the + # log by the endpoint it names. + - name: Run tests + run: mcpp test + + # A TEMPLATE IS PART OF THE RELEASE THAT SHIPS IT, AND `mcpp new` CAN ONLY + # REACH ONE THAT IS ALREADY PUBLISHED. Checking them after the release + # makes the first person to run `mcpp new` the one who finds out they do + # not compile, so this renders them the way the scaffolder does and builds + # them against this commit. It has already caught one: `import std` + # carries no `stdout` macro, so a progress bar flushed through it did not + # compile in a generated project while compiling fine in this repository. + - name: Smoke-test the project templates + run: bash tools/template_smoke.sh + + # ── macOS ──────────────────────────────────────────────────────────────────── + # + # THE README NAMES macOS, AND UNTIL THIS JOB NOTHING RAN THERE. The + # differences are real and in this library's own code: SO_NOSIGPIPE rather + # than MSG_NOSIGNAL (src/platform.cppm), /dev/urandom as mbedTLS's entropy, + # the system CA bundle's location, and poll(2) on a socket whose connect is in + # progress, which the cancellation slices wait on. + macos: + name: build + test (macos arm64, llvm@22.1.8) + runs-on: macos-15 + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + + - name: Install xlings + run: | + curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 \ + https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh \ + | bash -s "$XLINGS_VERSION" + echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" + + - name: Install mcpp + run: | + xlings update + xlings install "mcpp@$MCPP_VERSION" -y -g + mcpp --version + mcpp self config --mirror GLOBAL - name: Cache mcpp sandbox uses: actions/cache@v4 with: - path: ~/.xlings/data/xpkgs/xim-x-mcpp/0.0.13/registry - key: mcpp-sandbox-${{ runner.os }}-mcpp0.0.13 + path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry + key: mcpp-sandbox-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-llvm@22.1.8 + + - name: Select the toolchain + run: | + mcpp toolchain install llvm 22.1.8 + mcpp toolchain default llvm@22.1.8 + mcpp toolchain list - name: Build with mcpp run: mcpp build - name: Run tests run: mcpp test + + # ── The same sources, on a different kernel ABI ────────────────────────────── + # + # WHY THIS IS A SEPARATE JOB AND NOT A STEP. It resolves a different toolchain + # (llvm rather than gcc) and a different everything below the program — musl + # ported onto openkal, libc++ configured for that musl, and openkal-linux + # beneath both. Sharing a job would mean sharing a cache key between two + # stacks that have nothing in common but the source tree. + # + # WHAT IT CATCHES THAT THE JOB ABOVE CANNOT. `Socket::write` sends with + # MSG_NOSIGNAL and `connect_addrinfo` sets SO_NOSIGPIPE, and which of those + # exists is decided by the C library rather than by the operating system — + # openkal-musl defines the first and not the second. A `#ifdef` that is wrong + # about that compiles cleanly here and fails there, which is exactly how + # 5e7d66f reached master. + # + # FOUR TARGETS, THE ONES mcpp-index MEASURES tinyhttps ON (its + # tests/openkal/pins.toml). aarch64-linux-musl runs under qemu, through the + # runner examples/openkal names. x86_64-windows-musl is built here and run on + # Windows itself by the job after this one: wine took anywhere from four to + # more than thirty minutes to install on these runners, and Windows is the + # system the program is for. The Windows target is where every handshake used + # to fail for want of entropy (src/tls.cppm); it has no name resolution, so its + # network step reports "no network" and the cancellation check, which needs + # none, is what runs. + openkal: + name: build + run (${{ matrix.target }}, above openkal) + runs-on: ubuntu-24.04 + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + target: ['x86_64-linux-gnu', 'x86_64-linux-musl', 'x86_64-windows-musl', 'aarch64-linux-musl'] + steps: + - uses: actions/checkout@v4 + + - name: Install xlings + run: | + curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 \ + https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh \ + | bash -s "$XLINGS_VERSION" + echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" + + - name: Install mcpp + run: | + xlings update + xlings install "mcpp@$MCPP_VERSION" -y -g + mcpp --version + mcpp self config --mirror GLOBAL + + - name: Install the runner + if: matrix.target == 'aarch64-linux-musl' + timeout-minutes: 15 + env: + DEBIAN_FRONTEND: noninteractive + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq --no-install-recommends qemu-user > /dev/null + qemu-aarch64 --version | head -1 + + - name: Cache mcpp sandbox + uses: actions/cache@v4 + with: + path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry + key: mcpp-openkal-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-${{ matrix.target }} + + # The example builds the library from this checkout (`path = "../.."`) and + # takes the stack beneath it from the index, so what is tested is this + # commit against the published openkal packages. + # + # The toolchain is named, as mcpp-index's measurement names it, and + # installed beforehand: on a fresh runner x86_64-windows-musl was refused + # as "cannot be built on this host" while llvm was not yet installed, though + # the Linux targets, which install it on demand, built. + # + # It cancels a handshake against a local listener, which needs no network, + # and then makes one HTTPS request; it reports "no network" rather than + # failing when there is none, so a runner without egress reports "not run" + # instead of "broken". A certificate the client refused is a failure, not + # an absence of network. The build, the link, the framing parsers and the + # cancellation are checked either way. + - name: Install llvm + run: mcpp toolchain install llvm 22.1.8 + + - name: Build and run above openkal + if: matrix.target != 'x86_64-windows-musl' + working-directory: examples/openkal + run: | + set -o pipefail + mcpp run --toolchain llvm@22.1.8 --target '${{ matrix.target }}' 2>&1 | tee run.log + grep -q '^cancellation: ok' run.log + + # A static PE executable that imports only system DLLs, so it runs on any + # Windows as it is. + - name: Build for Windows above openkal + if: matrix.target == 'x86_64-windows-musl' + working-directory: examples/openkal + run: | + mcpp build --toolchain llvm@22.1.8 --target x86_64-windows-musl + mkdir -p "$RUNNER_TEMP/smoke" + cp target/x86_64-windows-musl/*/bin/smoke.exe "$RUNNER_TEMP/smoke/" + + - uses: actions/upload-artifact@v4 + if: matrix.target == 'x86_64-windows-musl' + with: + name: smoke-x86_64-windows-musl + path: ${{ runner.temp }}/smoke/smoke.exe + if-no-files-found: error + + openkal-windows: + name: run (x86_64-windows-musl, above openkal, on Windows) + needs: openkal + runs-on: windows-2022 + timeout-minutes: 15 + defaults: + run: + shell: bash + steps: + - uses: actions/download-artifact@v4 + with: + name: smoke-x86_64-windows-musl + + # Above openkal the Windows certificate store is not read (README, + # "Platforms"), so the program is given a bundle: Git for Windows's, copied + # beside it, because openkal resolves a relative name against the working + # directory and names no absolute path. + # + # Git is asked where its bundle is, and the install searched after that, + # rather than the directory named: the runner image that brought Git for + # Windows 2.56 has no `mingw64/etc/ssl/certs/ca-bundle.crt`, and the job + # failed there before the program ran. + - name: Run above openkal + run: | + set -o pipefail + git --version + ca=$(git config --system --get http.sslcainfo || true) + if [ -n "$ca" ]; then ca=$(cygpath -u "$ca"); fi + if [ ! -f "$ca" ]; then + ca=$(find "/c/Program Files/Git" -path '*/ssl/certs/ca-bundle.crt' -print -quit) + fi + if [ ! -f "$ca" ]; then + echo "::error::no CA bundle in the Git for Windows install" + exit 1 + fi + echo "CA bundle: $ca" + cp "$ca" ca-bundle.crt + SSL_CERT_FILE=ca-bundle.crt ./smoke.exe 2>&1 | tee run.log + grep -q '^cancellation: ok' run.log + + # ── Windows ────────────────────────────────────────────────────────────────── + # + # WHY THIS JOB EXISTS. Windows keeps no bundle file, so on a Windows Sockets + # build `load_ca_certs` reads the system's ROOT store through the Win32 API + # (src/ca_bundle.cppm). That code is compiled on no other platform, and a job + # elsewhere can show neither that it builds nor that the roots it returns + # verify a real certificate chain. `test_ca_store` asserts both. + # + # AND THE HERMETIC TESTS RUN HERE TOO. They script their own servers on + # 127.0.0.1, and Windows Sockets is where `poll` is `WSAPoll`, a connect in + # progress is reported differently, and the cancellation slices were never + # run before this job ran them. + windows: + name: build + test (windows x86_64, ${{ matrix.toolchain }}) + runs-on: windows-2022 + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + toolchain: ['msvc@system', 'llvm@20.1.7'] + defaults: + run: + shell: bash + steps: + - uses: actions/checkout@v4 + + - name: Install xlings + shell: pwsh + run: | + irm https://d2learn.org/xlings-install.ps1.txt | iex + "$env:USERPROFILE\.xlings\subos\current\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append + + - name: Install mcpp + run: | + xlings update + xlings install "mcpp@$MCPP_VERSION" -y -g + mcpp --version + mcpp self config --mirror GLOBAL + + - name: Select the toolchain + run: | + spec='${{ matrix.toolchain }}' + case "$spec" in + msvc*) mcpp toolchain default msvc ;; + *) mcpp toolchain install "${spec%@*}" "${spec#*@}" + mcpp toolchain default "$spec" ;; + esac + mcpp toolchain list + + - name: The system store verifies a public chain + run: | + mcpp test test_ca_store 2>&1 | tee tests.log + grep -q '^test_ca_store \.\.\. ok' tests.log + + - name: The hermetic tests + run: | + set -o pipefail + for t in test_framing test_tls_verify test_pool test_proxy test_cancel test_handshake_timeout test_extra_ca; do + mcpp test "$t" 2>&1 | tee "$t.log" + grep -q "^$t \.\.\. ok" "$t.log" + done diff --git a/.gitignore b/.gitignore index 1a3902c..4314c55 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,12 @@ build/ .xmake/ # mcpp build artefacts target/ +gcm.cache/ +# Resolution record, not a pin — the file says so itself: "It does not yet pin +# future builds: index dependencies are re-resolved from their constraints each +# time." A library's consumers re-resolve from its constraints, so committing it +# records one machine's resolution and changes nothing about anyone else's. +# Every other mcpplibs package ignores it, including the scaffold template. +mcpp.lock +# mcpp writes this beside the manifest on every build +compile_commands.json diff --git a/.xlings.json b/.xlings.json deleted file mode 100644 index 81146db..0000000 --- a/.xlings.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "workspace": { - "mcpp": "0.0.87" - } -} diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..fcd7dcf --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,373 @@ +# Changelog + +## 0.3.5 + +A handshake that never completes no longer holds a request for good, and a +private CA can be trusted alongside the default store. It is a patch release, so +a dependency written as a caret range, `tinyhttps = "^0.3.1"`, resolves to it; +the one change in behaviour, a handshake slower than `connectTimeoutMs` now +failing, is the end of a hang rather than a new failure mode, and is described +below. + +### TLS handshake time limit + +[#25](https://github.com/mcpplibs/tinyhttps/pull/25). The TLS handshake is +bounded by `connectTimeoutMs`. It had no limit: after the +TCP connection was up, a server (or an https:// proxy) that accepted it and never +sent its ServerHello held `send`, `send_stream` and `download_to_file` for good +unless the caller passed a stop token. + +* The handshake gives up once `connectTimeoutMs` has passed since it began, and + `statusText` is `TLS handshake timed out`. This covers the handshake with the + target, with an https:// proxy, and with the target inside an https:// proxy's + tunnel. The limit is on the handshake as a whole, so a peer that sends a byte + at a time does not hold it open. +* Behaviour change: a handshake slower than `connectTimeoutMs` (10 s by default) + now fails where it used to complete. Each step of setting up a connection (the + TCP connect, a proxy's reply, a handshake) has `connectTimeoutMs` to itself, so + the whole can take several times that. +* `TlsSocket::connect` bounds the handshake with the `timeoutMs` it already + took for the TCP connect, so a caller of it directly sees the change too. +* `TlsSocket::connect_over` takes a trailing `handshakeTimeoutMs`, `-1` (the + default) for no limit; a call by name compiles unchanged, a pointer to it needs + the new parameter in its type. `Socket` gains `set_deadline`, `deadline_hit` + and `wait_before_recv`. + +### Extra CA certificates + +[#24](https://github.com/mcpplibs/tinyhttps/pull/24). `HttpClientConfig::extraCaFile` is a path to a PEM file whose certificates are +trusted in addition to the default store, which `SSL_CERT_FILE` can only replace. +It is for a private CA or a proxy that re-signs TLS. + +* It applies wherever a trust store is built: the target, and the connection to + an `https://` proxy. +* A file that cannot be read or holds no certificate fails the connection, and + `statusText` names the file. +* `proxy_tunnel` takes the path as a trailing default argument, and `TlsSocket` + gains `set_extra_ca_file`. Nothing changes while the field is empty. + +### CI + +* The openkal job on Windows asks Git for Windows where its CA bundle is; 2.56 + moved it from `mingw64/` to `ucrt64/`, and the job failed before it ran. +* The Windows job runs `test_handshake_timeout` and `test_extra_ca` with the + other hermetic tests. + +## 0.3.4 + +A request in flight can be abandoned from another thread. Everything is added +and nothing changes without a token, so it is a patch release: a dependency +written as a caret range, `tinyhttps = "^0.3.0"`, resolves to it. A bare version +such as `"0.3.3"` is exact in mcpp and stays where it is. + +### Cancellation + +`send`, `send_stream` and `download_to_file` take an optional `std::stop_token` +as their last argument ([#23](https://github.com/mcpplibs/tinyhttps/pull/23)). + +* When the token is stopped the call returns within about 50 ms, and the + connection is closed instead of being returned to the pool. It covers the + connect, the exchange with an HTTP, https or SOCKS5 proxy, the TLS handshake, + waiting for the response head, reading the body and the gaps between + streaming callbacks. +* `HttpResponse::cancelled` says so. Before the status line arrives the response + is `statusCode` 0 with `statusText` and `bodyError` `Cancelled`; after it, + `statusCode` is the server's and `bodyError` is `cancelled`. `ok()` does not + look at `cancelled`, as it does not look at `bodyComplete`. +* `DownloadToFileResult::cancelled` is the same for a download, set by the token + or by `isCancelled`, with `error` `Cancelled` or `cancelled` as above. A + destination that failed first is reported as `writeFailed` instead. +* A cancelled request is not retried on a new connection, and its redirect is + not followed. +* A token belongs to its call. It is set on the socket at the bottom of the + connection (through an https:// proxy, the one beneath the tunnel) and taken + off when the connection goes back to the pool, so a token stopped after its + call returned — `std::jthread` stops its own in its destructor — does not + reach the next call on that connection. +* Not covered: name resolution (`getaddrinfo`, and the lookup `socks5://` does + for the target), and a write that is blocked because the server is not + reading. +* `proxy_tunnel` takes the token as a trailing default argument. `Socket` and + `TlsSocket` gain `set_stop`, and `Socket` gains `stop_possible`. +* A call through a pointer to `HttpClient::send`, `send_stream` or + `download_to_file` needs the new parameter in the pointer's type; a call by + name compiles unchanged. +* libc++ 20 and 22 fail to link `std::stop_source::request_stop()` in a file + that includes a standard header such as `` before `import std;`. + Import first in such a file; libc++ 23 and libstdc++ are not affected. + +### openkal + +* Where the C library is musl, the TLS random generator is seeded from musl's + `getentropy` rather than from mbedTLS's reading of `/dev/urandom`. Above + openkal on x86_64-windows-musl that file does not exist, and every handshake + failed with "CTR_DRBG - The entropy source failed". +* `examples/openkal` follows mcpp-index's openkal measurement: runtime 0.15.2, + and a cancellation check against a local listener that needs no network. CI + runs it on x86_64-linux-gnu, x86_64-linux-musl, aarch64-linux-musl (qemu) and + x86_64-windows-musl, which is built on Linux and run on Windows itself. + +### CI + +* Linux runs every test under gcc 16 and under llvm 22 with libc++. +* macOS (arm64, llvm 22) runs every test. Nothing ran on macOS before. +* Windows (msvc and llvm 20) runs the hermetic tests as well as `test_ca_store`. + +## 0.3.3 + +**Behaviour change, and the reason for this release:** `verifySsl = true` (the +default) now verifies the server certificate, and with no CA bundle to verify +against a connection fails instead of going ahead unverified +([#20](https://github.com/mcpplibs/tinyhttps/issues/20)). It is a patch release +so that every dependency written `tinyhttps = "0.3.x"` receives it. + +### Certificate verification + +`verifySsl = true` now verifies the server certificate. The handshake used to +complete with any certificate (a self-signed one, one for another host, an +expired one) because `MBEDTLS_SSL_VERIFY_OPTIONAL` was set and nothing read the +result afterwards. + +* The handshake fails when the chain is not trusted, the certificate has expired + or its name is not the host being connected to. The reason is in `statusText`, + for example `certificate verification failed: The certificate validity has + expired`; other handshake failures carry the mbedTLS error text. A refused TCP + connection still reads `Connection failed`. +* Behaviour change: with `verifySsl = true` and no CA bundle found + (`SSL_CERT_FILE` unset or unreadable, none of the system locations present), + the connection fails with a message saying so, where it used to go ahead + unverified. Point `SSL_CERT_FILE` at a PEM bundle, or set `verifySsl = false`. +* On Windows, when `SSL_CERT_FILE` is not set, the roots in the system `ROOT` + certificate store that are usable for TLS servers are read. That code is built + only for Windows Sockets targets, not for the musl one. MinGW builds need + `-lcrypt32`, as they already need `-lws2_32`. +* `examples/openkal` exits 1 when the certificate is refused; it used to read + every failure as an absence of network. CI builds and runs the Windows store + reader on windows-2022 under msvc and llvm (`tests/test_ca_store.cpp`). +* MSVC builds link `bcrypt`, from which mbedTLS draws entropy; its package asks + for it as `-lbcrypt`, which link.exe does not read. + +### Proxies + +Proxy URLs carry credentials, and a proxy that refuses the tunnel says why. + +`parse_proxy_url` split a URL at the first colon, so `user:pass@host:3128` came +out as host `user` and a port made of the digits of `pass@host:3128`. It also +dropped the scheme, which meant `https://proxy:8443` was spoken to as plain +HTTP. And every refusal, whatever the proxy said, reached the caller as +`Connection failed`. + +* `user:password@` in the proxy URL is percent-decoded and sent as a Basic + `Proxy-Authorization` header on the `CONNECT`. Brackets around an IPv6 host + are understood, and a port that is not 1..65535 is 0 rather than a truncation. +* `ProxyConfig` gains `scheme`, `hasCredentials`, `user` and `password`. + `parse_proxy_url` still returns the same type, and `host` and `port` keep + their meaning. +* A refused tunnel is reported in `statusText`: `proxy rejected the credentials: + 407 Proxy Authentication Required` when credentials were sent, + `proxy requires credentials: ...` when none were, `proxy refused CONNECT: ...` + for any other status. Any 2xx answer opens the tunnel; before, only 200 did. +* A scheme this library does not speak is an error (`proxy: unsupported scheme + 'socks4'`) where it used to be read as HTTP. +* New `proxy_tunnel` returns the tunnel together with the reason it could not + be made. `proxy_connect` keeps its signature and calls it, so it also accepts + any 2xx answer now. +* An `https://` proxy URL is spoken to over TLS: the session to the proxy + carries the `CONNECT`, and the session to the target runs inside the tunnel. + `TlsSocket` can now run over another `TlsSocket` (`connect_over` takes the + lower session), and its `wait_readable` asks that session rather than the + descriptor, because the bytes may already be decrypted there. The proxy + connection is set up like the one to the target, `verifySsl` included. The + default port for `https` is 443. +* SOCKS5 proxies: `socks5://` resolves the target's name here and sends the + proxy an address, `socks5h://` sends the name. An address literal is sent as + an address in both. The client offers no authentication, plus the + username/password method when the URL has credentials. Failures are reported + in `statusText` the same way as for an HTTP proxy. The default port is 1080. + SOCKS4 is not supported and, like any unknown scheme, is an error. +* `Socket::resolve_address` returns the address a name resolves to as raw bytes. +* The connection to an `https://` proxy is verified like the connection to the + target, and a refusal names both the hop and the reason, for example + `proxy: could not open a TLS connection to proxy:443: certificate + verification failed: ...`. The CONNECT, and any credentials, are sent only + after the proxy's certificate has verified. +* A failed handshake inside a tunnel closes the tunnel's session as well. + +## 0.3.2 + +`download_to_file` no longer reports success for a file the disk did not keep. + +It called `ofs.write` for each body chunk without looking at the stream, added +each chunk's size to `bytesWritten` from what the network delivered, and set +`bytesWritten` after `ofs.close()` without looking at that either. On a full disk +(`ENOSPC`) the transfer therefore succeeded: a 420,831,054 byte download left +220,979,200 bytes on disk while `bytesWritten` said 420,831,054 and `ok()` was +true, and the caller blamed the source (a checksum mismatch) for what was the +local disk. + +* Each chunk is flushed and the stream checked. On the first failure the + transfer stops reading, `error` becomes `write : ` (the reason + is `errno` as `std::generic_category` words it, for example `No space left on + device`), and the connection is dropped rather than returned to the pool with + the rest of the body still on it. Closing the file is checked the same way. +* `DownloadToFileResult::writeFailed` is new and is true when the fault is the + destination rather than the source: the write failed, the close failed, or the + file could not be opened (`error` is still `Cannot open file: ` for the + last). +* `DownloadToFileResult::bytesReceived` is new: the bytes of body the + connection delivered. +* `DownloadToFileResult::bytesWritten` changes meaning, and this is the one + behaviour change: it now counts bytes the file accepted, where it used to + count bytes the network delivered. The two are equal for every transfer that + did not fail to write. When a write fails, the chunk that failed is not + counted, though the file may hold part of it, so the value is a floor on the + file's size. + +## 0.3.1 + +The socket interface is selected by the C library rather than by the operating +system ([mcpp-index#435](https://github.com/mcpplibs/mcpp-index/issues/435)). +Above openkal on Windows the C library is musl and `winsock2.h` is not in the +dependency graph; the POSIX interface is used there, selected in `mcpp.toml` by +`cfg(c-abi = "musl")`. A Windows build with the platform's C runtime is +unchanged. `platform::is_windows` is renamed `platform::uses_winsock`. + +## 0.3.0 + +Closes [#15](https://github.com/mcpplibs/tinyhttps/issues/15) (a socket returned +to the pool with bytes still owed on it) and +[#16](https://github.com/mcpplibs/tinyhttps/issues/16) (a write to a departed +peer killing the host process). + +**#15 named two of the eleven paths that could leak a connection.** All eleven +are fixed here. Four of the other nine turned up while verifying the report — a +redirect, a non-2xx and a failed file open, none of which involve a timeout or a +truncation at all, plus a `Content-Length` past 32 bits — and two more (the +streaming reader's `Content-Length` exits) were regressions 0.2.10 had +introduced. + +### Read this first + +**If you are on 0.2.10 and use `send_stream`, upgrade.** 0.2.10 added a +`Content-Length` branch to the streaming reader whose two exits — a read timeout +and an end of stream — left the connection marked reusable where the branch they +replaced never did. A stream cut short therefore poisoned the next request on +that client. + +**Malformed responses that used to be accepted in silence now report an error.** +A status line that is not one, a chunk header that does not parse, a header +block cut short by a timeout: each of these used to produce a plausible-looking +result. They are not new failures — they are failures that were previously +invisible. `HttpResponse::bodyError` says which. + +**Nothing in the API was removed or changed shape.** Existing code compiles +unchanged, and `ok()` means exactly what it did. + +### Fixed + +- **A write to a socket whose peer has gone away no longer raises `SIGPIPE`** + (#16). `Socket::write` sends with `MSG_NOSIGNAL` where the platform has it and + sets `SO_NOSIGPIPE` on the descriptor where it does not. Previously a program + that had not disarmed the signal itself — the default — was killed with exit + status 141, most often from inside the pool's own clean-up. The library no + longer changes its host's signal disposition either, which is what mbedtls's + own network layer does and what replacing that layer had silently dropped. +- **A connection is returned to the pool only when the body was read to the end + its framing declared** (#15). Ten early-return paths each had to remember to + mark the connection unusable and several did not; the invariant is now carried + by a guard whose default is to drop, so a path that does nothing is safe. + The paths that were leaking: a stream that timed out or ended mid-body, a + chunk that was cut short, a chunked download that failed, a download whose + declared body was cut short, **a redirect whose body was never read**, **a + non-2xx whose error body was never read**, and **a download whose output file + could not be opened**. The last three are ordinary paths with no timeout and + no truncation involved. +- **A status line that is not a status line is rejected** rather than mined for + digits. `BBBB 999 XHTTP/1.1 200 OK` — leftover body bytes read as a status + line — used to be reported as an ordinary `999`. +- **A `Content-Length` past 32 bits is no longer truncated.** `send()` held it in + an `int`, so `4294967296` became `0`, the body was taken for empty, and the + socket went back into the pool with four gigabytes owed on it. This was the + only pool defect a server could trigger with a single header. +- **A chunk header that does not parse is not a terminal chunk.** `send()` still + used the salvaging hex parser, which returned `0` — the value that means "body + ends here" — for an empty line, and an empty line is what a timed-out read + produced. +- **A header block cut short by a timeout is an error**, not the end of the + headers. The reader went on to the body with half the headers and no framing. +- **The CRLF after chunk data is verified** in `send()` and `send_stream`, as it + already was in `download_to_file`. +- **`download_to_file` parses `Content-Length` strictly**, as the other two + readers already did. +- **A response with no body is not read for one.** 204 and 304 carry a + `Content-Length` describing what a `GET` would have returned; reading that many + bytes consumed the next response instead. Only `HEAD` was handled. +- **An interim 1xx response is read past rather than returned as the answer** + (RFC 9112 §2.1). A `103 Early Hints` — sent proactively by several CDNs — or a + `100 Continue` — sent for any request carrying `Expect: 100-continue` — was + reported to the caller as the result, and the real response was then read as + its body. Found by review, along with the fact that skipping the body of a 1xx + without also reading past it would have marked the connection clean with the + real response still on it: issue #15 arriving by another door. +- **An end of stream is no longer reported as a transport error.** The BIO under + mbedtls answered a peer's FIN with `MBEDTLS_ERR_NET_CONN_RESET`; mbedtls's own + BIO passes the zero through, and `ssl_fetch_input` tests for exactly that zero + to produce `SSL_CONN_EOF` (`ssl_msg.c:2251`). Since most servers close without + a TLS `close_notify`, a body whose framing *is* the close — legal per RFC 9112 + §6.3 — read as truncated: `download_to_file` set `error` and `ok()` returned + false for a file that had arrived complete and correct. +- **TLS back-pressure is waited on rather than retried once and abandoned.** A + slow peer used to reach the caller as an unexplained `"Write failed"`. + +### Added + +- `HttpResponse::bodyComplete` and `HttpResponse::bodyError` — whether the body + arrived in full, and why not. A truncated 200 and a complete 200 were + previously indistinguishable. `ok()` does not consult them, so no existing + behaviour changes. +- `HttpClientConfig::maxResponseBodyBytes` (default 64 MiB) — bounds what + `send()` will hold in memory. A server could previously induce an allocation of + any size by writing a header, and reach an undocumented `std::length_error`. + Does not bound `download_to_file` or `send_stream`, which stream. +- `HttpClientConfig::retryOnStaleConnection` (default true) — resend once when a + connection taken from the pool turns out to have been closed while idle, which + is routine server behaviour the client cannot see until it writes. Without it + such a request returned `statusCode = 0, "No response"` for a request the + server never saw. The window is one attempt, on a pooled connection, before a + single response byte has arrived. +- `parse_status_line`, exported and unit-testable without a server, alongside + `parse_content_length` and `parse_chunk_size_line`. +- `TlsSocket::read_some`, returning `Data`/`WouldBlock`/`Eof`/`Error`. `read()` + collapsed an end of stream and a transport that was not ready into the same + `0`, which is why callers could not tell a finished body from a stalled one. + `read()` is unchanged and still exported. +- `examples/openkal` — the library built above + [openkal](https://github.com/mcpplibs/openkal), making a real HTTPS request + through `kal_net_connect`. Run in CI. +- **Project templates**, scaffolded with `mcpp new --template tinyhttps[:name]`: + `fetch` (default), `download` and `stream`, one per entry point. + `tools/template_smoke.sh` renders and builds each of them against the working + tree — a template can only be checked before the release that ships it, not + after — and CI runs it. + +### Changed + +- `send`, `send_stream` and `download_to_file` share one status-line parser, one + header reader and one body reader. They previously carried three near-copies, + and every past hardening had landed on one or two of them — which is the whole + of why #15 had eight instances rather than one. +- A short 3xx or error body is drained (up to 64 KiB) so the connection survives + a redirect, instead of being dropped with it. +- `send_stream` no longer overwrites the server's reason phrase with a transfer + error; the reason goes to `bodyError`. + +### Tests + +`tests/` had no keep-alive coverage at all, which is why the 0.2.10 regression +was merged green. It now has a scripted in-process TLS server, and every pool +test asserts **how many TCP connections the server saw** as well as what came +back — the report for #15 contains a case whose output is byte-for-byte correct +and whose only symptom is the connection count. + +Each fix was checked by mutation: reverting it in the source makes specific tests +fail, and restoring it makes them pass. diff --git a/README.md b/README.md index a4d0481..6ee907d 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,8 @@ Minimal C++23 HTTP/HTTPS client library with SSE (Server-Sent Events) streaming - HTTP/HTTPS client with connection pooling (keep-alive) - SSE (Server-Sent Events) streaming -- Proxy support (HTTP CONNECT) +- Streaming downloads to a file, with progress and cancellation +- Proxy support (HTTP CONNECT over plain HTTP or TLS, with Basic authentication; SOCKS5) - C++23 modules ## Usage @@ -28,32 +29,221 @@ auto resp = client.send(mcpplibs::tinyhttps::HttpRequest::post( )); ``` +### Knowing whether you received all of it + +`ok()` reports what the server said. `bodyComplete` reports whether the body +arrived in full — a read that timed out, a connection that ended mid-body or a +chunk header that did not parse all leave a *prefix* of the body behind a +perfectly ordinary status code. + +```cpp +auto resp = client.send(request); +if (!resp.ok()) { /* the server refused: resp.statusCode */ } +else if (!resp.bodyComplete) { /* the transfer broke: resp.bodyError */ } +else { /* resp.body is all of it */ } +``` + +`ok()` deliberately does not consult `bodyComplete`, so existing `if (res.ok())` +means exactly what it did before. + +### Cancelling a request + +Pass a `std::stop_token` to `send`, `send_stream` or `download_to_file` and stop +its source from another thread. The call returns within about 50 ms with +`cancelled` set, and the connection is closed rather than pooled. + +```cpp +HttpResponse response; +std::jthread worker([&](std::stop_token stop) { response = client.send(request, stop); }); +// elsewhere: +worker.request_stop(); +``` + +If the status line had not arrived, `statusCode` is 0 and `statusText` is +`Cancelled`; otherwise they are the server's and `bodyError` is `cancelled`. +As with `bodyComplete`, `ok()` does not look at `cancelled`. A cancelled request +is not retried and its redirect is not followed. `download_to_file` reports the +same in `DownloadToFileResult::cancelled` and `error`, whether the token or its +`isCancelled` callback asked. + +It covers connecting, a proxy's handshake, the TLS handshake, waiting for the +response and reading it. Name resolution and a write blocked on a server that is +not reading cannot be interrupted. Without a token nothing changes. + +With libc++ 20 or 22, a file that includes a standard header such as `` +and then `import std;` fails to link `std::stop_source::request_stop()` (an +inline helper of libc++'s is never emitted). Put `import std;` before the +includes in that file, or use libc++ 23 or libstdc++. + +### Configuration + +| field | default | what it decides | +| --- | --- | --- | +| `connectTimeoutMs` | 10000 | setting up the connection: the TCP connect, each wait for a proxy's reply, and the TLS handshake, with the proxy and with the target (the handshake as a whole, not each read in it) | +| `readTimeoutMs` | 60000 | any single read, and the total wait on a blocked write | +| `verifySsl` | true | verify the server certificate against the CA bundle (`SSL_CERT_FILE`, else the Windows `ROOT` certificate store in Windows Sockets builds, else a system location); the connection fails if the certificate is not trusted, has expired or is for another host, or if no bundle is found | +| `keepAlive` | true | reuse connections between requests | +| `maxRedirects` | 10 | 0 disables redirect following | +| `maxResponseBodyBytes` | 64 MiB | the most `send()` will hold in memory; does not bound `download_to_file` or `send_stream` | +| `retryOnStaleConnection` | true | resend once when a pooled connection turns out to have been closed while idle | +| `extraCaFile` | empty | path of a PEM file of CA certificates to trust in addition to the default store, see below | +| `proxy` | none | proxy URL, see below | + +### Extra CA certificates + +A private CA, or a proxy that re-signs TLS, is not in the default store, and +`SSL_CERT_FILE` replaces that store instead of adding to it. Set `extraCaFile` +to a PEM file and its certificates are trusted as well as the default ones: + +```cpp +cfg.extraCaFile = "/etc/corp/root-ca.pem"; +``` + +It applies to the connection to an `https://` proxy as well as to the target. If +the file cannot be read or holds no certificate, the connection fails and +`statusText` names the file. The file adds to a default store and does not stand +in for one: with none found, the connection fails as described under `verifySsl`. + +### Proxies + +Set `proxy` to a URL and every request is tunnelled through it with `CONNECT`: + +```cpp +cfg.proxy = "http://user:pass@proxy.example:3128"; +``` + +The scheme is `http` (a bare `host:port` means the same), `https`, `socks5` or +`socks5h`. + +An `https://` proxy is reached over TLS first, the `CONNECT` is sent inside that +session, and the connection to the target is a second TLS session inside the +tunnel; the two connections are set up alike, `verifySsl` included. + +With `socks5` the target's name is resolved here and the proxy is given an +address; with `socks5h` the proxy is given the name and resolves it itself, +which is what to use when the proxy is the only thing that can see the target's +DNS. An address literal is sent as an address either way. The client offers the +SOCKS5 proxy no authentication, plus the username/password method when the URL +has credentials. SOCKS4 is not supported. + +Credentials are read from the URL and sent as Basic `Proxy-Authorization` (or as +the SOCKS5 username and password); percent-escape any character that is special +in a URL (`p%40ss` for `p@ss`). The port defaults to 8080, 443 for `https`, and +1080 for the SOCKS schemes. + +When the proxy refuses the tunnel, `statusCode` is 0 and `statusText` carries +its answer: `proxy rejected the credentials: 407 Proxy Authentication Required`, +`proxy refused CONNECT: 403 Forbidden`, `proxy refused CONNECT: connection +refused (SOCKS5 reply 5)`. Any other scheme (`socks4://`, say) fails the same +way instead of being read as HTTP. + +## Project templates + +The package ships starting points in `templates/`. Scaffold one with `mcpp new` +— the template comes from the library, so it is always the version you asked +for: + +```bash +mcpp new --list-templates tinyhttps # what this library provides +mcpp new myapp --template tinyhttps # the default (fetch) +mcpp new grab --template tinyhttps:download +mcpp new chat --template tinyhttps@0.3.0:stream +``` + +| Template | What it starts you with | +| --- | --- | +| `fetch` (default) | One request, and how to tell a complete answer from a truncated one | +| `download` | A file streamed to disk, with a progress bar, cancellation and `ETag`/`Last-Modified` | +| `stream` | Server-Sent Events — a streaming LLM chat completion, token by token | + +The selector is `[namespace.]name[@version][:template]`; omit `:template` for +the default. Templates are pure data — rendered and copied, with no hooks and no +script execution — and the placeholder vocabulary is mcpp's: +`{{project.name}}`, `{{template.package.name}}`, `{{template.package.version}}` +and a few more. + +A template is part of the release that ships it, so it has to be checked before +that release exists in the index: + +```bash +bash tools/template_smoke.sh # renders, builds and runs every template +``` + +It repoints each generated project at this checkout, so it verifies the +templates against the working tree rather than against whatever is published. +CI runs it. + +## Platforms + +Linux, macOS, Windows, Android/Termux — and above +[openkal](https://github.com/mcpplibs/openkal), the portable kernel ABI, where +the C library is musl ported onto openkal rather than the host's. + +`examples/openkal` builds this library from the checkout against the published +openkal stack and makes a real HTTPS request through it; CI runs it on every +push. See the platform table at the top of `src/platform.cppm` for what actually +differs between them. + +```bash +cd examples/openkal && mcpp run +``` + +Above openkal two things differ from a host build. Both follow from the layer +beneath and not from this library: + +- On Windows the socket interface is the POSIX one, because the C library is + musl, so the Windows `ROOT` certificate store is not read. Set `SSL_CERT_FILE` + to a PEM bundle of trusted roots; without one every HTTPS connection is + refused, and `statusText` says so. +- `connectTimeoutMs` does not bound the TCP connect. `kal_net_connect` has no + form that begins a connection and reports its outcome later, so the connect + completes or fails before it returns. A stop token is checked before the + connect, not during it. +- On Windows there is no name resolution: openkal has no resolver interface, + and musl's reads `/etc/resolv.conf`, which Windows does not have + ([openkal-musl#46](https://github.com/mcpplibs/openkal-musl/issues/46)). A URL + with an address in it connects. + +CI runs `examples/openkal` for x86_64-linux-gnu, x86_64-linux-musl, +aarch64-linux-musl (under qemu) and x86_64-windows-musl (built on Linux, run on +Windows). Locally, +`mcpp run --toolchain llvm@22.1.8 --target x86_64-windows-musl` runs it under wine. + ## 使用 mcpp 构建 ### 添加依赖 ```bash -mcpp add tinyhttps@0.2.2 +mcpp add tinyhttps@0.3.5 ``` 或在 `mcpp.toml` 中手动添加: ```toml [dependencies] -tinyhttps = "0.2.2" +tinyhttps = "0.3.5" ``` ### 构建 ```bash mcpp build +mcpp test ``` ### 代码示例 ```cpp import mcpplibs.tinyhttps; -// ... usage example + +mcpplibs::tinyhttps::HttpClient client; +auto result = client.download_to_file( + "https://example.com/big.tar.gz", "out/big.tar.gz", + [](std::int64_t total, std::int64_t done) { /* progress */ }); +if (!result.ok()) { + // result.error says why; result.writeFailed says the local file is at + // fault (a full disk, a refused write) and not the server. +} ``` ## License diff --git a/examples/openkal/mcpp.toml b/examples/openkal/mcpp.toml new file mode 100644 index 0000000..66dc732 --- /dev/null +++ b/examples/openkal/mcpp.toml @@ -0,0 +1,32 @@ +[package] +name = "tinyhttps-openkal" +version = "0.1.0" + +# ONE LINE IS WHAT MAKES THIS AN openkal PROGRAM, AND IT IS NOT ABOUT tinyhttps. +# +# `openkal-llvm-runtime` carries libc++, libc++abi, libunwind and compiler-rt +# configured for `openkal-musl`, which is musl ported onto the openkal kernel +# ABI. Naming it replaces the whole layer beneath this program — C library, +# C++ runtime and platform interface — and tinyhttps is compiled from the same +# sources either way. That is the claim this directory exists to check. +[dependencies] +tinyhttps = { path = "../.." } +openkal-llvm-runtime = "0.15.2" + +[targets.smoke] +kind = "bin" +main = "src/main.cpp" + +# An ordinary compiler. Where the headers, C library, C++ runtime and platform +# implementation come from is not stated here at all — it follows from the +# dependency above. +[toolchain] +default = "llvm@22.1.8" + +# The programs of the other targets this example is run for, as mcpp-index's +# openkal measurement runs them (tests/openkal/pins.toml there). +[target.x86_64-windows-musl] +runner = ["wine"] + +[target.aarch64-linux-musl] +runner = ["qemu-aarch64"] diff --git a/examples/openkal/src/main.cpp b/examples/openkal/src/main.cpp new file mode 100644 index 0000000..543439d --- /dev/null +++ b/examples/openkal/src/main.cpp @@ -0,0 +1,146 @@ +// tinyhttps above openkal — a smoke test that runs the whole stack. +// +// WHAT THIS PROVES THAT COMPILING DOES NOT. A build only shows that the headers +// were found. This binary is statically linked, contains openkal's own +// `kal_net_connect` and `kal_stream_read`, and makes a real HTTPS request +// through them: name resolution, TCP, a TLS 1.2 handshake against a public +// certificate chain, and the HTTP framing this library was rewritten around. +// +// Run it with `mcpp run` from this directory. It needs the network; when there +// is none it says so and exits 0, because a CI job without egress should report +// "not run" rather than "broken". A certificate the client refused is not an +// absence of network, and exits 1. +// The listener below is made with the C library's sockets, which openkal-musl +// provides on every target this example runs on. C headers only: they declare +// nothing `import std` also declares. +#include +#include +#include +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +// CANCELLATION ABOVE openkal, WITH NO NETWORK. +// +// A listener that never accepts: the connection completes in the backlog, the +// client sends its ClientHello and waits for an answer that is not coming. That +// wait is a `poll` in slices of 50 ms, and above openkal each slice is a bounded +// read (`kal_timeout_read`) rather than a readiness query, which is the part of +// the stack this checks. The read timeout is 30 s, so returning promptly can +// only be the token's doing. +bool cancellation_works() { + const int listener = ::socket(AF_INET, SOCK_STREAM, 0); + if (listener < 0) { std::println("cancellation: no socket"); return false; } + sockaddr_in addr {}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = 0; + socklen_t len = sizeof addr; + if (::bind(listener, reinterpret_cast(&addr), sizeof addr) != 0 + || ::listen(listener, 4) != 0 + || ::getsockname(listener, reinterpret_cast(&addr), &len) != 0) { + std::println("cancellation: no listener"); + ::close(listener); + return false; + } + const std::string url = "https://127.0.0.1:" + std::to_string(ntohs(addr.sin_port)) + "/"; + + https::HttpClientConfig config; + config.verifySsl = false; + config.connectTimeoutMs = 5000; + config.readTimeoutMs = 30000; + https::HttpClient client(config); + https::HttpRequest request; + request.method = https::Method::GET; + request.url = url; + + // A token stopped before the call: nothing is sent. + std::stop_source early; + early.request_stop(); + auto before = client.send(request, early.get_token()); + + // A token stopped while the handshake waits. + std::stop_source source; + std::jthread stopper([&] { + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + source.request_stop(); + }); + const auto start = std::chrono::steady_clock::now(); + auto during = client.send(request, source.get_token()); + const auto ms = std::chrono::duration_cast( + std::chrono::steady_clock::now() - start).count(); + ::close(listener); + + const bool ok = before.cancelled && before.statusCode == 0 + && during.cancelled && during.statusCode == 0 + && ms >= 150 && ms < 2000; + std::println("cancellation: {} (stopped at 200 ms, returned after {} ms)", + ok ? "ok" : "WRONG", ms); + return ok; +} + +} // namespace + +int main() { + https::Socket::platform_init(); + + // The parsers first: they need no network, and reaching them at all means + // every module of the library compiled and linked on this stack. + auto status = https::parse_status_line("HTTP/1.1 200 OK"); + auto length = https::parse_content_length("4294967296"); // past 32 bits + auto chunk = https::parse_chunk_size_line("1a"); + if (!status || status->code != 200 || !length || *length != 4294967296LL + || !chunk || *chunk != 26) { + std::println("framing parsers gave the wrong answers"); + return 1; + } + std::println("framing parsers: ok"); + + if (!cancellation_works()) return 1; + + https::HttpClientConfig config; + config.connectTimeoutMs = 15000; + config.readTimeoutMs = 20000; + https::HttpClient client(config); + + https::HttpRequest request; + request.method = https::Method::GET; + request.url = "https://httpbin.org/bytes/64"; + + auto response = client.send(request); + if (response.statusCode == 0) { + // A certificate the client refused is a verdict about this stack, not + // an absence of network, and is reported as one. Read as "no network", + // a CA bundle this stack cannot find would pass here unnoticed. + constexpr std::string_view refusals[] = { + "certificate verification failed", + "no CA certificate bundle", + "cannot parse the CA certificate bundle", + "TLS handshake failed", + }; + for (auto refusal : refusals) { + if (response.statusText.starts_with(refusal)) { + std::println("TLS refused: {}", response.statusText); + return 1; + } + } + std::println("no network ({}) — the build and the parsers are still verified", + response.statusText); + return 0; + } + + std::println("GET {} -> {} {} ({} bytes, complete={})", + request.url, response.statusCode, response.statusText, + response.body.size(), response.bodyComplete); + + const bool ok = response.statusCode == 200 + && response.bodyComplete + && response.body.size() == 64; + std::println("{}", ok ? "tinyhttps above openkal: ok" : "tinyhttps above openkal: WRONG"); + return ok ? 0 : 1; +} diff --git a/mcpp.lock b/mcpp.lock deleted file mode 100644 index 07de937..0000000 --- a/mcpp.lock +++ /dev/null @@ -1,9 +0,0 @@ -# Auto-generated by mcpp. Do not edit by hand. -version = 2 - -[package."compat.mbedtls"] -namespace = "compat" -version = "3.6.1" -source = "index+compat@3.6.1" -hash = "fnv1a:83dea67ac1379be3" - diff --git a/mcpp.toml b/mcpp.toml index ec01e93..911b9b3 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "tinyhttps" -version = "0.2.9" +version = "0.3.5" description = "Minimal C++23 HTTP/HTTPS client with SSE streaming support" license = "Apache-2.0" repo = "https://github.com/mcpplibs/tinyhttps" @@ -14,3 +14,11 @@ mbedtls = "3.6.1" [dev-dependencies.compat] gtest = "1.15.2" + +# The socket interface follows the C library rather than the operating system +# (src/platform.cppm). Where the C library is musl, including musl above openkal +# on Windows, it is the POSIX interface; the platform's own headers are not part +# of that graph. The entropy source follows it too: musl's `getentropy`, rather +# than mbedTLS's reading of /dev/urandom (src/tls.cppm). +[target.'cfg(c-abi = "musl")'.build] +cxxflags = ["-DTINYHTTPS_POSIX_SOCKETS", "-DTINYHTTPS_GETENTROPY"] diff --git a/src/ca_bundle.cppm b/src/ca_bundle.cppm index 577179f..f481d77 100644 --- a/src/ca_bundle.cppm +++ b/src/ca_bundle.cppm @@ -2,9 +2,28 @@ module; #include +// The same switch as socket.cppm. The store is read through the Win32 API, whose +// headers a build on the POSIX socket interface does not have. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#include +#pragma comment(lib, "crypt32.lib") +#endif + export module mcpplibs.tinyhttps:ca_bundle; import std; +import :platform; namespace mcpplibs::tinyhttps { @@ -24,6 +43,59 @@ auto read_file(const char* path) -> std::string { return result; } +#ifdef TINYHTTPS_WINSOCK +// Whether the root may be used to authenticate a server: no usage restriction at +// all, or serverAuth listed. Windows marks roots it no longer trusts for TLS +// servers through this restriction, and the store still holds them. +auto trusted_for_servers(const CERT_CONTEXT* c) -> bool { + DWORD n = 0; + if (!::CertGetEnhancedKeyUsage(c, 0, nullptr, &n)) { + return false; + } + std::vector buf(n); + auto* usage = reinterpret_cast(buf.data()); + if (!::CertGetEnhancedKeyUsage(c, 0, usage, &n)) { + return false; + } + if (usage->cUsageIdentifier == 0) { + // CRYPT_E_NOT_FOUND means no restriction; otherwise the root allows no use. + return ::GetLastError() == CRYPT_E_NOT_FOUND; + } + for (DWORD i = 0; i < usage->cUsageIdentifier; ++i) { + if (std::strcmp(usage->rgpszUsageIdentifier[i], szOID_PKIX_KP_SERVER_AUTH) == 0) { + return true; + } + } + return false; +} +#endif + +// The Windows ROOT store as one PEM string; empty elsewhere or on failure. +auto system_store_pem() -> std::string { + std::string pem; +#ifdef TINYHTTPS_WINSOCK + if (HCERTSTORE store = ::CertOpenSystemStoreW(0, L"ROOT")) { + for (const CERT_CONTEXT* c = nullptr; (c = ::CertEnumCertificatesInStore(store, c));) { + if (!trusted_for_servers(c)) { + continue; + } + DWORD n = 0; + if (!::CryptBinaryToStringA(c->pbCertEncoded, c->cbCertEncoded, + CRYPT_STRING_BASE64HEADER, nullptr, &n)) { + continue; + } + std::string one(n, '\0'); + if (::CryptBinaryToStringA(c->pbCertEncoded, c->cbCertEncoded, + CRYPT_STRING_BASE64HEADER, one.data(), &n)) { + pem.append(one, 0, n); + } + } + ::CertCloseStore(store, 0); + } +#endif + return pem; +} + } // anonymous namespace export auto load_ca_certs() -> std::string { @@ -37,6 +109,13 @@ export auto load_ca_certs() -> std::string { } } + // Windows has no bundle file; read the system's trusted roots instead. + if constexpr (platform::uses_winsock) { + if (auto pem = system_store_pem(); !pem.empty()) { + return pem; + } + } + std::vector ca_paths; // 2. Non-FHS prefixes (Termux et al. ship the bundle under $PREFIX). Probed diff --git a/src/http.cppm b/src/http.cppm index 67d700e..5cbdc73 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -1,3 +1,9 @@ +module; + +// `import std` carries no `errno`: it is a macro, and a module does not export +// macros. The download path reads it to say why the local file refused a write. +#include + export module mcpplibs.tinyhttps:http; import :tls; @@ -29,6 +35,34 @@ export struct HttpResponse { std::map headers; std::string body; + // FALSE WHEN THE BODY ENDED BEFORE ITS FRAMING SAID IT WOULD — a read that + // timed out, an end of stream mid-body, a chunk header that did not parse, + // or a streaming callback that asked to stop. + // + // The status line and the headers are still what the server sent; `body` is + // a prefix of what it promised. Before this field a truncated 200 and a + // complete 200 were indistinguishable to the caller: `DownloadToFileResult` + // has carried `error` throughout, and `send`/`send_stream` had no way to say + // it at all. + // + // `ok()` deliberately does NOT consult this. `ok()` answers "what did the + // server say", which is a question about the status code, and making it also + // answer "did I receive all of it" would change what existing code means + // without changing what it says. + bool bodyComplete { true }; + + // Why the body is incomplete, when it is. Empty whenever `bodyComplete`. + // + // Separate from `statusText`, which used to be overwritten with the reason — + // so a caller logging `statusText` on a 200 that broke mid-body saw + // "Invalid chunk size: zz" where the server had said "OK". + std::string bodyError; + + // True when the caller's stop token ended the request. Then `bodyComplete` + // is false, and the status line is the server's if it had arrived and + // 0 / "Cancelled" if not. + bool cancelled { false }; + bool ok() const { return statusCode >= 200 && statusCode < 300; } }; @@ -39,6 +73,47 @@ export struct HttpClientConfig { bool verifySsl { true }; bool keepAlive { true }; int maxRedirects { 10 }; // 0 = don't follow redirects + + // The most `send()` will accumulate in `HttpResponse::body` before treating + // the response as unreadable. + // + // It bounds one specific hazard: `send()` is the entry point that holds the + // whole body in memory, and the size it was about to allocate came from a + // header. A server answering `Content-Length: 9223372036854775807` used to + // reach `std::string::resize` with it — and this library documents no + // exception, so a caller had no reason to have caught the `length_error` + // that followed. + // + // It does NOT bound `download_to_file` (which streams to disk) or + // `send_stream` (which streams to a callback); a download larger than this + // is an ordinary thing to want. + std::int64_t maxResponseBodyBytes { 64ll * 1024 * 1024 }; + + // Whether to reconnect and send once more when a connection taken from the + // pool turns out to have been closed by the server. + // + // A server closing an idle keep-alive connection is routine, and the client + // cannot see it until it writes: the write lands in a local buffer and + // succeeds, and the read that follows returns nothing. Without a retry that + // becomes `statusCode = 0, statusText = "No response"` — for a request the + // server never saw. curl (`Curl_retry_request`), Go's `net/http` + // (`shouldRetryRequest`) and requests all send again in this window. + // + // The window is narrow by construction: only a connection that came from the + // pool, only before a single response byte has arrived, and only once. In + // that state the server has already sent its FIN, so nothing above its + // transport layer saw the request. Unlike Go, this is not restricted to + // idempotent methods — this library's traffic is predominantly POST to API + // endpoints, and restricting it there would make the retry inapplicable to + // nearly every request it exists to rescue. Set false to opt out. + bool retryOnStaleConnection { true }; + + // A PEM file of CA certificates to trust in addition to the default store + // (`SSL_CERT_FILE`, or else the system roots), for a private CA or a proxy that + // re-signs TLS. It applies to the connection to an https:// proxy as well. + // A file that cannot be read or holds no certificate fails the connection, + // with the path in `statusText`. Empty means none. + std::string extraCaFile; }; // Progress callback for streaming downloads: (totalBytes, downloadedBytes) @@ -48,11 +123,45 @@ export using DownloadProgressFn = std::function expectedBytes; std::string finalUrl; std::string etag; std::string lastModified; + + // Bytes of body the connection delivered, whether or not the file took them. + // Equal to `bytesWritten` unless `writeFailed`. + std::int64_t bytesReceived { 0 }; + + // TRUE WHEN THE FAULT IS THE DESTINATION AND NOT THE SOURCE — the file could + // not be opened, a write to it failed (a full disk, a quota, a device that + // refuses), or closing it reported that buffered data did not reach it. + // + // `error` then names the path and the reason, and the transfer stops at the + // first failure instead of reading on and discarding what it reads. It is + // what lets a caller tell "the server sent something wrong" from "this + // machine could not keep it" without matching text in `error`; a checksum + // failure downstream of a truncated file otherwise blames the wrong party. + bool writeFailed { false }; + + // True when the caller abandoned the transfer: `isCancelled` returned true + // or the stop token was stopped. `error` is then "Cancelled" if no status + // line had arrived and "cancelled" if one had, as for `HttpResponse`. A + // destination that failed first is reported as that instead. + bool cancelled { false }; + bool ok() const { return statusCode >= 200 && statusCode < 300 && error.empty(); } }; @@ -115,24 +224,37 @@ static ParsedUrl parse_url(std::string_view url) { return result; } +// Case-insensitive string comparison +static bool iequals(std::string_view a, std::string_view b) { + if (a.size() != b.size()) return false; + for (std::size_t i = 0; i < a.size(); ++i) { + char ca = a[i]; + char cb = b[i]; + if (ca >= 'A' && ca <= 'Z') ca += 32; + if (cb >= 'A' && cb <= 'Z') cb += 32; + if (ca != cb) return false; + } + return true; +} + // Check if user headers contain a key (case-insensitive) static bool has_header(const std::map& headers, std::string_view key) { for (const auto& [k, v] : headers) { - if (k.size() == key.size()) { - bool match = true; - for (std::size_t i = 0; i < k.size(); ++i) { - if (std::tolower(static_cast(k[i])) != - std::tolower(static_cast(key[i]))) { - match = false; - break; - } - } - if (match) return true; - } + if (iequals(k, key)) return true; } return false; } +// The value of a header, whatever case the server spelled it in. Empty when +// absent — which the three callers here can all treat as "not set". +static std::string find_header(const std::map& headers, + std::string_view key) { + for (const auto& [k, v] : headers) { + if (iequals(k, key)) return v; + } + return {}; +} + static std::string_view method_to_string(Method m) { switch (m) { case Method::GET: return "GET"; @@ -145,63 +267,172 @@ static std::string_view method_to_string(Method m) { return "GET"; } -// Read exactly n bytes from socket, using wait_readable for timeout -static bool read_exact(TlsSocket& sock, char* buf, int n, int timeoutMs) { - int total = 0; - while (total < n) { - if (!sock.wait_readable(timeoutMs)) { - return false; - } - int ret = sock.read(buf + total, n - total); - if (ret < 0) return false; - if (ret == 0) { - // Try again after wait - if (!sock.wait_readable(timeoutMs)) return false; - ret = sock.read(buf + total, n - total); - if (ret <= 0) return false; - } - total += ret; +// ───────────────────────────────────────────────────────────────────────────── +// Framing primitives — the half of HTTP that can be examined without a server. +// ───────────────────────────────────────────────────────────────────────────── + +// A status line is `HTTP-version SP status-code [SP reason]` (RFC 9112 §4.1), +// and anything else is not one. +// +// ALL THREE READERS USED TO EXTRACT THE CODE BY KEEPING THE DIGITS AND SKIPPING +// EVERYTHING ELSE, with no check that the line was a status line at all. That is +// where issue #15's `statusCode = 999` came from: a socket returned to the pool +// with a previous response's bytes still on it, and the next request read +// +// BBBB…B 999 XHTTP/1.1 200 OK +// +// as a perfectly ordinary 999. `X9Y9Z9` would have been 999 too. +// +// Exported for the same reason as `parse_chunk_size_line` and +// `parse_content_length`: it is testable without a network. +export struct StatusLine { + int code { 0 }; + std::string text; +}; + +export std::optional parse_status_line(std::string_view line) { + if (!line.starts_with("HTTP/")) return std::nullopt; + + auto sp = line.find(' '); + if (sp == std::string_view::npos) return std::nullopt; + auto rest = line.substr(sp + 1); + + auto sp2 = rest.find(' '); + auto codeStr = (sp2 == std::string_view::npos) ? rest : rest.substr(0, sp2); + + // Exactly three digits — the grammar says three, and accepting two or four + // is how a fragment of a body starts looking like a status code. + if (codeStr.size() != 3) return std::nullopt; + int code {}; + auto [end, error] = std::from_chars( + codeStr.data(), codeStr.data() + codeStr.size(), code); + if (error != std::errc{} || end != codeStr.data() + codeStr.size()) { + return std::nullopt; } - return true; + if (code < 100 || code > 599) return std::nullopt; + + return StatusLine{ code, + sp2 == std::string_view::npos + ? std::string{} + : std::string(rest.substr(sp2 + 1)) }; } -// Read a line (ending with \r\n) from socket -static std::string read_line(TlsSocket& sock, int timeoutMs) { - std::string line; - char c; - while (true) { - if (!sock.wait_readable(timeoutMs)) { - break; - } - int ret = sock.read(&c, 1); - if (ret < 0) break; - if (ret == 0) { - // Try once more - if (!sock.wait_readable(timeoutMs)) break; - ret = sock.read(&c, 1); - if (ret <= 0) break; - } - line += c; - if (line.size() >= 2 && line[line.size() - 2] == '\r' && line[line.size() - 1] == '\n') { - line.resize(line.size() - 2); - break; - } +// `Content-Length`, rejected rather than salvaged. +// +// Both readers parsed this by walking the characters and keeping the digits, so +// `12abc` was 12, `abc` was 0 — indistinguishable from a genuine `0` — and a +// value past the width of the accumulator wrapped in silence. A length is the +// number of bytes the reader will then trust, so a wrong one is not a cosmetic +// error: too small leaves the next response's bytes in the stream, and too +// large waits for bytes that are not coming. +export std::optional +parse_content_length(std::string_view value) { + // A field value may carry optional whitespace on either side (RFC 9110). + while (!value.empty() && (value.front() == ' ' || value.front() == '\t')) + value.remove_prefix(1); + while (!value.empty() && (value.back() == ' ' || value.back() == '\t')) + value.remove_suffix(1); + if (value.empty()) return std::nullopt; + + std::uint64_t parsed {}; + auto [end, error] = std::from_chars( + value.data(), value.data() + value.size(), parsed, 10); + if (error != std::errc{} || end != value.data() + value.size() + || parsed > static_cast( + std::numeric_limits::max())) { + return std::nullopt; } - return line; + return static_cast(parsed); } +export std::optional +parse_chunk_size_line(std::string_view line) { + if (line.empty()) return std::nullopt; + std::uint64_t value {}; + auto [end, error] = std::from_chars( + line.data(), line.data() + line.size(), value, 16); + if (error != std::errc{} || end != line.data() + line.size() + || value > static_cast( + std::numeric_limits::max())) { + return std::nullopt; + } + return static_cast(value); +} + +// A streaming request that fails is answered with an error document, not an +// event stream, so SseParser finds no event boundary in it and yields nothing. +// The bytes are still worth keeping: without them a caller can report the +// status line but never the reason. Bounded, so a server answering 5xx with an +// endless body cannot grow the buffer without limit. +export inline constexpr std::size_t stream_error_body_limit = 1024 * 1024; + +// Appends as much of `data` as `limit` still allows. Returns false once the +// buffer is full, so a caller can stop copying without tracking sizes itself. +export bool append_within_limit(std::string& buffer, std::string_view data, + std::size_t limit) { + if (buffer.size() >= limit) return false; + const std::size_t room = limit - buffer.size(); + // Not std::min: defines a `min` macro on Windows. + const std::size_t take = data.size() < room ? data.size() : room; + buffer.append(data.substr(0, take)); + return take == data.size(); +} + +// ───────────────────────────────────────────────────────────────────────────── +// Transport +// ───────────────────────────────────────────────────────────────────────────── + +// A header line or a chunk header past this length is a server that is not +// speaking HTTP. Without a bound, one such line grows a std::string until the +// process runs out of memory. +inline constexpr std::size_t max_line_length = 8192; + +// And a bound on how MANY such lines, which is a separate hazard from how long +// each one is. A header block is accumulated into a map, so a server sending an +// endless supply of distinct header lines — each one short and well formed — +// grows the client's memory without limit. A trailer section is discarded +// rather than kept, but an endless supply of trailers holds the caller's thread +// just as long, and every line resets the read timeout so nothing else stops +// it. Well past what any real response carries: a large one has forty. +inline constexpr int max_head_lines = 200; + +// Reads one CRLF-terminated line, distinguishing the three things an empty +// return used to mean. +// +// The older `read_line` returned `""` on a timeout, on an end of stream, AND on +// a genuine empty line — the blank line that ends a header block. So a header +// block that was cut short by a timeout read as a header block that had ended, +// and the reader went on to the body with half the headers and no idea it had +// done so. That is root cause R2 behind issue #15. +// +// `sawBytes`, when given, is set as soon as any byte of this line has arrived. +// A connection the server closed while it was idle produces none, which is the +// one condition under which resending the request is safe. static std::expected -read_complete_line(TlsSocket& sock, int timeoutMs) { +read_complete_line(TlsSocket& sock, int timeoutMs, bool* sawBytes = nullptr) { std::string line; char c {}; while (true) { if (!sock.wait_readable(timeoutMs)) { - return std::unexpected("timeout or EOF before CRLF"); + return std::unexpected("timed out before CRLF"); + } + auto r = sock.read_some(&c, 1); + if (r.status == ReadStatus::WouldBlock) { + // The transport had nothing decrypted yet. Wait once more; the + // socket is blocking after the handshake, so this is rare. + if (!sock.wait_readable(timeoutMs)) { + return std::unexpected("timed out before CRLF"); + } + r = sock.read_some(&c, 1); + } + if (r.status == ReadStatus::Eof) { + return std::unexpected("connection closed before CRLF"); } - int ret = sock.read(&c, 1); - if (ret <= 0) { - return std::unexpected("EOF before CRLF"); + if (r.status != ReadStatus::Data) { + return std::unexpected("read error before CRLF"); } + if (sawBytes) *sawBytes = true; + line += c; if (line.size() >= 2 && line[line.size() - 2] == '\r' @@ -209,66 +440,458 @@ read_complete_line(TlsSocket& sock, int timeoutMs) { line.resize(line.size() - 2); return line; } + if (line.size() > max_line_length) { + return std::unexpected("line exceeds " + std::to_string(max_line_length) + + " bytes without a CRLF"); + } } } -// Write all data to socket -static bool write_all(TlsSocket& sock, const std::string& data) { - int total = 0; - int len = static_cast(data.size()); - while (total < len) { - int ret = sock.write(data.c_str() + total, len - total); +// Writes the whole buffer, waiting when the TLS record layer cannot flush. +// +// `TlsSocket::write` reports back-pressure as 0. This used to answer that by +// retrying immediately once and giving up if the second attempt also returned 0 +// — so a slow peer reached the caller as an unexplained "Write failed". Waiting +// on the socket is what the 0 was asking for; the total wait is bounded by +// `timeoutMs` so a peer that never drains cannot stall the call. +static bool write_all(TlsSocket& sock, const std::string& data, int timeoutMs) { + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(timeoutMs); + std::size_t total = 0; + while (total < data.size()) { + int ret = sock.write(data.data() + total, + static_cast(data.size() - total)); if (ret < 0) return false; if (ret == 0) { - // Try again - ret = sock.write(data.c_str() + total, len - total); - if (ret <= 0) return false; + const auto left = std::chrono::duration_cast( + deadline - std::chrono::steady_clock::now()).count(); + if (left <= 0) return false; + if (!sock.wait_writable(static_cast(left))) return false; + continue; } - total += ret; + total += static_cast(ret); } return true; } -// Parse hex string to int -static int parse_hex(std::string_view s) { - int result = 0; - for (char c : s) { - result <<= 4; - if (c >= '0' && c <= '9') result |= (c - '0'); - else if (c >= 'a' && c <= 'f') result |= (c - 'a' + 10); - else if (c >= 'A' && c <= 'F') result |= (c - 'A' + 10); - else break; +// ───────────────────────────────────────────────────────────────────────────── +// One reader, used by all three entry points. +// +// There used to be three: `send`, `send_stream` and `download_to_file` each +// carried a near-copy of the status-line parse, the header loop and the body +// loop. Every hardening landed on one or two of them — strict `Content-Length` +// on two, strict chunk sizes on two, CRLF-after-chunk validation on one, pool +// clean-up on none — and the matrix of which fix reached which copy is the +// whole of issue #15's cause. `#14`, the most recent change to touch them, added +// a branch to one copy and introduced two regressions doing it. +// +// Below there is one status-line parse, one header loop and one body loop. +// ───────────────────────────────────────────────────────────────────────────── + +struct ResponseHead { + int statusCode { 0 }; + std::string statusText; + std::map headers; + bool chunked { false }; + std::int64_t contentLength { -1 }; + bool connectionClose { false }; +}; + +struct HeadError { + std::string message; + // Whether any byte of the response had arrived when this failed. False means + // the server never answered, which is what a connection closed while idle + // looks like from here. + bool sawBytes { false }; +}; + +static std::expected +read_response_head(TlsSocket& sock, int timeoutMs) { + ResponseHead head; + bool sawBytes = false; + + // AN INTERIM RESPONSE IS NOT THE ANSWER, AND STOPPING AT ONE POISONS THE + // POOL. RFC 9112 §2.1 requires a client to be able to read one or more 1xx + // responses before the final one, and each is a complete status line and + // header block of its own. + // + // Returning the 1xx as if it were the answer would report `103 Early Hints` + // (or `100 Continue`, which any server sends for a request carrying + // `Expect: 100-continue`) to the caller, and — because a 1xx carries no + // body — mark the connection clean while the REAL response sat unread on + // it. The next request through the same client would then read that + // response as its own, which is issue #15 arriving by another door. + // + // The `max_head_lines` cap below applies per interim response; the loop + // itself is bounded by the same count, so a server sending nothing but 1xx + // cannot hold the call open forever either. + for (int interim = 0; ; ++interim) { + if (interim >= max_head_lines) { + return std::unexpected(HeadError{ + "More than " + std::to_string(max_head_lines) + + " interim responses", true }); + } + auto statusLine = read_complete_line(sock, timeoutMs, &sawBytes); + if (!statusLine) { + return std::unexpected(HeadError{ "No response", sawBytes }); + } + auto status = parse_status_line(*statusLine); + if (!status) { + return std::unexpected(HeadError{ "Invalid status line", true }); + } + head.statusCode = status->code; + head.statusText = std::move(status->text); + + if (head.statusCode >= 200) break; + + // Discard the interim response's own header block and read the next + // status line. Its fields describe the interim response, not the + // answer, so keeping them would misreport the final one's headers. + for (int lines = 0; ; ++lines) { + if (lines >= max_head_lines) { + return std::unexpected(HeadError{ + "Interim header block exceeds " + + std::to_string(max_head_lines) + " lines", true }); + } + auto line = read_complete_line(sock, timeoutMs); + if (!line) { + return std::unexpected(HeadError{ + "Incomplete interim headers: " + line.error(), true }); + } + if (line->empty()) break; + } } - return result; -} -export std::optional -parse_chunk_size_line(std::string_view line) { - if (line.empty()) return std::nullopt; - std::uint64_t value {}; - auto [end, error] = std::from_chars( - line.data(), line.data() + line.size(), value, 16); - if (error != std::errc{} || end != line.data() + line.size() - || value > static_cast( - std::numeric_limits::max())) { - return std::nullopt; + for (int lines = 0; ; ++lines) { + if (lines >= max_head_lines) { + return std::unexpected(HeadError{ + "Header block exceeds " + std::to_string(max_head_lines) + + " lines", true }); + } + auto line = read_complete_line(sock, timeoutMs); + if (!line) { + return std::unexpected(HeadError{ + "Incomplete headers: " + line.error(), true }); + } + if (line->empty()) break; // the blank line that ends the block + + auto colon = line->find(':'); + if (colon == std::string::npos) continue; + std::string key = line->substr(0, colon); + std::string_view value = std::string_view(*line).substr(colon + 1); + while (!value.empty() && (value.front() == ' ' || value.front() == '\t')) { + value.remove_prefix(1); + } + std::string valStr(value); + + if (iequals(key, "Transfer-Encoding") && iequals(valStr, "chunked")) { + head.chunked = true; + } + if (iequals(key, "Content-Length")) { + // Rejected rather than salvaged. A malformed value leaves this at + // -1, which is the same state as an absent header and is a framing + // this reader already handles. + head.contentLength = parse_content_length(valStr).value_or(-1); + } + if (iequals(key, "Connection") && iequals(valStr, "close")) { + head.connectionClose = true; + } + head.headers[std::move(key)] = std::move(valStr); } - return static_cast(value); + + return head; } -// Case-insensitive string comparison -static bool iequals(std::string_view a, std::string_view b) { - if (a.size() != b.size()) return false; - for (std::size_t i = 0; i < a.size(); ++i) { - char ca = a[i]; - char cb = b[i]; - if (ca >= 'A' && ca <= 'Z') ca += 32; - if (cb >= 'A' && cb <= 'Z') cb += 32; - if (ca != cb) return false; +// Whether a response of this shape carries a body at all (RFC 9112 §6.3). +// +// A HEAD response, a 1xx, a 204 and a 304 carry none even when they declare a +// Content-Length — the field describes what a GET would have returned. Reading +// that many bytes would consume the next response instead; only HEAD was +// handled before. +static bool response_has_body(Method method, int statusCode) { + if (method == Method::HEAD) return false; + if (statusCode >= 100 && statusCode < 200) return false; + return statusCode != 204 && statusCode != 304; +} + +// WHERE A BODY ENDED, WHICH IS THE SAME QUESTION AS "IS THIS CONNECTION STILL +// USABLE". +// +// `Complete` is the only answer that leaves bytes owed to nobody. Every other +// answer means the stream is at an unknown offset, so the connection cannot be +// handed to the next request whatever its headers said. +enum class BodyEnd { + Complete, // read to the end its framing declared + Truncated, // the framing was broken or the body stopped short + ClosedByPeer, // no framing at all; the body ended when the connection did + Stopped, // the sink asked to stop, so bytes are still owed +}; + +struct BodyOutcome { + BodyEnd end { BodyEnd::Truncated }; + std::string error; + std::int64_t bytes { 0 }; +}; + +using BodySink = std::function; + +// Reads a body according to the framing its head declared, handing each slice to +// `sink`. Returning false from `sink` stops the read (`BodyEnd::Stopped`). +// +// `maxBytes` bounds what will be read. It exists because the size this function +// was about to trust came from the server: a `Content-Length` near INT64_MAX, or +// a single `7fffffff` chunk header, used to reach an allocation of that size. +// Nothing is allocated per chunk here — the body is read in fixed slices — so +// the bound is on the total handed to the sink rather than on any one buffer. +static BodyOutcome read_body(TlsSocket& sock, const ResponseHead& head, + int timeoutMs, std::int64_t maxBytes, + const BodySink& sink) { + BodyOutcome out; + char buf[8192]; + + // Reads at least one byte. A timeout on the wait, and a transport that stays + // unready across two waits, are both reported as `WouldBlock` — at that + // point bytes are still owed and the framing is broken either way. + auto pull = [&](std::size_t want) -> ReadResult { + for (int attempt = 0; attempt < 2; ++attempt) { + if (!sock.wait_readable(timeoutMs)) return { ReadStatus::WouldBlock, 0 }; + auto r = sock.read_some(buf, static_cast(want)); + if (r.status != ReadStatus::WouldBlock) return r; + } + return { ReadStatus::WouldBlock, 0 }; + }; + + auto emit = [&](int n) -> bool { + out.bytes += n; + return sink(std::string_view(buf, static_cast(n))); + }; + + auto too_large = [&] { + out.end = BodyEnd::Truncated; + out.error = "Body exceeds the configured limit of " + + std::to_string(maxBytes) + " bytes"; + }; + + if (head.chunked) { + while (true) { + auto sizeResult = read_complete_line(sock, timeoutMs); + if (!sizeResult) { + out.end = BodyEnd::Truncated; + out.error = "Invalid chunk size line: " + sizeResult.error(); + return out; + } + std::string sizeLine = std::move(*sizeResult); + if (auto semi = sizeLine.find(';'); semi != std::string::npos) { + sizeLine.resize(semi); + } + while (!sizeLine.empty() + && (sizeLine.back() == ' ' || sizeLine.back() == '\t')) { + sizeLine.pop_back(); + } + + // A CHUNK HEADER THAT DOES NOT PARSE IS NOT A TERMINAL CHUNK. The + // older `parse_hex` returned what it had accumulated when it met a + // character it did not recognise, and 0 for an empty line — and the + // older `read_line` returned an empty line on a timeout. So a stream + // that was cut short read as one that had ended cleanly, and the + // socket went back into the pool with the rest of the body owed. + auto chunkSize = parse_chunk_size_line(sizeLine); + if (!chunkSize) { + out.end = BodyEnd::Truncated; + out.error = "Invalid chunk size: " + sizeLine; + return out; + } + + if (*chunkSize == 0) { + // The trailer section, then the blank line that ends it. + // Bounded for the reason `max_head_lines` gives: every line + // resets the read timeout, so an endless supply of them holds + // this call open with nothing else to stop it. + for (int lines = 0; ; ++lines) { + if (lines >= max_head_lines) { + out.end = BodyEnd::Truncated; + out.error = "Trailer section exceeds " + + std::to_string(max_head_lines) + " lines"; + return out; + } + auto trailer = read_complete_line(sock, timeoutMs); + if (!trailer) { + out.end = BodyEnd::Truncated; + out.error = "Invalid chunk trailer: " + trailer.error(); + return out; + } + if (trailer->empty()) break; + } + out.end = BodyEnd::Complete; + return out; + } + + if (out.bytes + *chunkSize > maxBytes) { + too_large(); + return out; + } + + std::int64_t remaining = *chunkSize; + while (remaining > 0) { + const auto want = static_cast( + remaining < static_cast(sizeof buf) + ? remaining : static_cast(sizeof buf)); + auto r = pull(want); + if (r.status != ReadStatus::Data) { + out.end = BodyEnd::Truncated; + out.error = r.status == ReadStatus::Eof + ? "Connection closed in the middle of a chunk" + : "Read error in the middle of a chunk"; + return out; + } + remaining -= r.bytes; + if (!emit(r.bytes)) { out.end = BodyEnd::Stopped; return out; } + } + + // The CRLF that follows the data. Discarding it lets a wrong chunk + // length slide into the next chunk header without a word. + auto delimiter = read_complete_line(sock, timeoutMs); + if (!delimiter || !delimiter->empty()) { + out.end = BodyEnd::Truncated; + out.error = "Missing CRLF after chunk data"; + return out; + } + } + } + + if (head.contentLength >= 0) { + if (head.contentLength > maxBytes) { + too_large(); + return out; + } + std::int64_t remaining = head.contentLength; + while (remaining > 0) { + const auto want = static_cast( + remaining < static_cast(sizeof buf) + ? remaining : static_cast(sizeof buf)); + auto r = pull(want); + if (r.status != ReadStatus::Data) { + out.end = BodyEnd::Truncated; + out.error = r.status == ReadStatus::Eof + ? "Connection closed before the declared length" + : "Read timed out before the declared length"; + return out; + } + remaining -= r.bytes; + if (!emit(r.bytes)) { out.end = BodyEnd::Stopped; return out; } + } + out.end = BodyEnd::Complete; + return out; + } + + // Neither chunked nor a declared length: the body ends when the connection + // does, so there is nothing left to reuse afterwards. + while (true) { + auto r = pull(sizeof buf); + if (r.status == ReadStatus::Eof) { + out.end = BodyEnd::ClosedByPeer; + return out; + } + if (r.status != ReadStatus::Data) { + out.end = BodyEnd::Truncated; + out.error = "Read timed out before the connection closed"; + return out; + } + if (out.bytes + r.bytes > maxBytes) { + too_large(); + return out; + } + if (!emit(r.bytes)) { out.end = BodyEnd::Stopped; return out; } } - return true; } +// Consumes a body the caller is not going to be given, so that the connection +// stays usable. Returns false when it could not be consumed in full — then the +// connection must be dropped. +// +// Bounded: past the cap, reconnecting is cheaper than reading, and a body with +// no declared end must not be able to stall the call. A response with no framing +// of its own ends at the close, so there is nothing to salvage and this refuses +// without reading. +static bool drain_body(TlsSocket& sock, const ResponseHead& head, + int timeoutMs, std::int64_t cap = 64 * 1024) { + if (head.connectionClose) return false; + if (!head.chunked && head.contentLength < 0) return false; + if (!head.chunked && head.contentLength == 0) return true; + if (head.contentLength > cap) return false; + + auto outcome = read_body(sock, head, timeoutMs, cap, + [](std::string_view) { return true; }); + return outcome.end == BodyEnd::Complete; +} + +// THE POOL'S INVARIANT, MADE THE DEFAULT. +// +// A connection may be reused when it owes no bytes and its peer is still there. +// `TlsSocket::is_valid()` cannot check that — it answers "is the descriptor +// open" — so the condition was maintained by ten early-return paths each +// remembering to set a flag, and issue #15 is the list of the ones that did not. +// Every defect in it has the same shape: a path that did nothing, where doing +// nothing left a dirty socket in the pool for the next request to pick up. +// +// So dropping is what doing nothing means here. A path that leaves without +// calling `keep()` drops the connection, and a path added tomorrow is safe +// before anyone reviews it. +// +// Ordering: `drop()` destroys a `TlsSocket`, which sends `close_notify`, +// which writes to a socket whose peer may be gone. That write is why P0 — +// `MSG_NOSIGNAL` in `Socket::write` — had to land before this class existed; +// without it, making the drop path more common would have made issue #16 more +// common too. +class PooledConnection { +public: + PooledConnection(std::map& pool, std::string key) + : pool_(pool), key_(std::move(key)) {} + + ~PooledConnection() { drop(); } + + PooledConnection(const PooledConnection&) = delete; + PooledConnection& operator=(const PooledConnection&) = delete; + + // Call only where the body was read to the end its framing declared and the + // response did not ask for the connection to close. + // + // A connection in the pool carries no stop token. The call that used it + // set its own (`perform_exchange`), and the caller may stop that token + // after this call has returned — a `std::jthread` does so in its + // destructor — which must not reach whatever takes the connection next. + void keep() noexcept { + if (!armed_) return; + armed_ = false; + if (auto it = pool_.find(key_); it != pool_.end()) it->second.set_stop({}); + } + + // Drop now rather than at the end of the scope. Idempotent, and a no-op + // after `keep()`. + // + // Callers that recurse — the redirect paths — MUST settle the guard + // before recursing. The inner call puts its own connection into the pool + // under the same key when the redirect is to the same host, and a guard + // still armed when the outer scope ends would then delete it. + void drop() { + if (!armed_) return; + armed_ = false; + if (auto it = pool_.find(key_); it != pool_.end()) { + it->second.close(); + pool_.erase(it); + } + } + + // Drop what is there and arm again for what replaces it. The stale-connection + // retry is the one place where a guard outlives the socket it was made for. + void reset() { drop(); armed_ = true; } + +private: + std::map& pool_; + std::string key_; + bool armed_ { true }; +}; + export class HttpClient { public: // Thread-safety: HttpClient owns a mutable connection pool and is not synchronized. @@ -284,12 +907,21 @@ public: HttpClient(HttpClient&&) = default; HttpClient& operator=(HttpClient&&) = default; - HttpResponse send(const HttpRequest& request) { - return send_impl(request, 0); + // `stop` can be stopped from another thread to abandon the request: the + // call returns within about 50 ms with `cancelled` set, and the connection + // is closed rather than pooled. It cannot interrupt name resolution or a + // write blocked because the server is not reading. + HttpResponse send(const HttpRequest& request, std::stop_token stop = {}) { + return send_impl(request, 0, stop); } -private: - HttpResponse send_impl(const HttpRequest& request, int redirectCount) { + // Streaming SSE request — reads the response body incrementally, feeding + // chunks through SseParser to the caller's callback. The callback receives + // each SseEvent and returns true to continue or false to stop. + // `stop` is as for send(). + HttpResponse send_stream(const HttpRequest& request, SseCallbackFn callback, + std::stop_token stop = {}) { + if (stop.stop_requested()) return cancelled_response(); HttpResponse response; auto parsed = parse_url(request.url); @@ -299,51 +931,91 @@ private: return response; } - std::string poolKey = parsed.host + ":" + std::to_string(parsed.port); + const std::string poolKey = parsed.host + ":" + std::to_string(parsed.port); + PooledConnection guard(pool_, poolKey); - // Get or create connection - TlsSocket* sock = nullptr; - auto it = pool_.find(poolKey); - if (it != pool_.end() && it->second.is_valid()) { - sock = &it->second; - } else { - // Remove stale entry if exists - if (it != pool_.end()) { - pool_.erase(it); - } - // Create new connection - auto [insertIt, ok] = pool_.emplace(poolKey, TlsSocket{}); - sock = &insertIt->second; - bool connected = false; - if (config_.proxy.has_value()) { - auto proxyConf = parse_proxy_url(config_.proxy.value()); - auto tunnel = proxy_connect(proxyConf.host, proxyConf.port, - parsed.host, parsed.port, - config_.connectTimeoutMs); - if (tunnel.is_valid()) { - connected = sock->connect_over(std::move(tunnel), - parsed.host.c_str(), - config_.verifySsl); - } - } else { - connected = sock->connect(parsed.host.c_str(), parsed.port, - config_.connectTimeoutMs, config_.verifySsl); + auto exchange = perform_exchange(parsed, poolKey, + build_request(request, parsed), guard, stop); + if (!exchange.error.empty()) { + response.statusCode = 0; + response.statusText = exchange.error; + response.bodyComplete = false; + response.bodyError = exchange.error; + response.cancelled = exchange.cancelled; + return response; + } + + response.statusCode = exchange.head.statusCode; + response.statusText = exchange.head.statusText; + response.headers = exchange.head.headers; + + // A failed streaming request is answered with an error document rather + // than an event stream, so the parser finds nothing in it. Keeping the + // bytes is the only way a caller learns the reason. + const bool captureBody = !response.ok(); + SseParser parser; + + auto sink = [&](std::string_view data) -> bool { + if (stop.stop_requested()) return false; + if (captureBody) { + append_within_limit(response.body, data, stream_error_body_limit); } - if (!connected) { - pool_.erase(poolKey); - response.statusCode = 0; - response.statusText = "Connection failed"; - return response; + for (const auto& event : parser.feed(data)) { + if (!callback(event)) return false; } - } + return true; + }; + + finish_body(response, exchange, request.method, + /*maxBytes=*/std::numeric_limits::max(), + sink, guard, stop); + return response; + } - // Build request + // Download URL to file with streaming progress. + // Follows redirects. Calls onProgress periodically during body read. + // isCancelled is checked after each block — return true to abort. + // `stop` is as for send(), and also ends the wait for the response head. + DownloadToFileResult download_to_file( + const std::string& url, + const std::filesystem::path& destFile, + DownloadProgressFn onProgress = nullptr, + std::function isCancelled = nullptr, + std::stop_token stop = {}) + { + return download_to_file_impl(url, destFile, std::move(onProgress), + std::move(isCancelled), 0, stop); + } + + HttpClientConfig& config() { return config_; } + const HttpClientConfig& config() const { return config_; } + +private: + // What one request/response head exchange produced. `error` empty means + // `sock` and `head` are usable. + struct Exchange { + TlsSocket* sock { nullptr }; + ResponseHead head; + std::string error; + bool cancelled { false }; + }; + + static HttpResponse cancelled_response() { + HttpResponse response; + response.statusText = "Cancelled"; + response.bodyComplete = false; + response.bodyError = "Cancelled"; + response.cancelled = true; + return response; + } + + std::string build_request(const HttpRequest& request, const ParsedUrl& parsed) const { std::string reqStr; reqStr += method_to_string(request.method); reqStr += " "; reqStr += parsed.path; reqStr += " HTTP/1.1\r\n"; - // Add Host header (skip if user provided) + if (!has_header(request.headers, "Host")) { reqStr += "Host: "; reqStr += parsed.host; @@ -353,225 +1025,165 @@ private: } reqStr += "\r\n"; } - - // Add Content-Length if body present (skip if user provided) if (!request.body.empty() && !has_header(request.headers, "Content-Length")) { reqStr += "Content-Length: "; reqStr += std::to_string(request.body.size()); reqStr += "\r\n"; } - - // Add user headers for (const auto& [key, value] : request.headers) { reqStr += key; reqStr += ": "; reqStr += value; reqStr += "\r\n"; } - - // Add connection header (skip if user provided) if (!has_header(request.headers, "Connection")) { - if (config_.keepAlive) { - reqStr += "Connection: keep-alive\r\n"; - } else { - reqStr += "Connection: close\r\n"; - } + reqStr += config_.keepAlive ? "Connection: keep-alive\r\n" + : "Connection: close\r\n"; } - reqStr += "\r\n"; + reqStr += request.body; + return reqStr; + } - // Append body - if (!request.body.empty()) { - reqStr += request.body; - } - - // Send request - if (!write_all(*sock, reqStr)) { - pool_.erase(poolKey); - response.statusCode = 0; - response.statusText = "Write failed"; - return response; - } - - // Read status line - std::string statusLine = read_line(*sock, config_.readTimeoutMs); - if (statusLine.empty()) { - pool_.erase(poolKey); - response.statusCode = 0; - response.statusText = "No response"; - return response; - } - - // Parse status line: HTTP/1.1 200 OK - { - auto spacePos = statusLine.find(' '); - if (spacePos == std::string::npos) { - pool_.erase(poolKey); - response.statusCode = 0; - response.statusText = "Invalid status line"; - return response; + // False on failure, with `error` set when there is more to say than that the + // connection failed: a proxy that refused the tunnel says so in its own words. + bool open_connection(TlsSocket& sock, const ParsedUrl& parsed, std::string& error, + std::stop_token stop) { + sock.set_extra_ca_file(config_.extraCaFile); + if (config_.proxy.has_value()) { + auto tunnel = proxy_tunnel(parse_proxy_url(config_.proxy.value()), + parsed.host, parsed.port, + config_.connectTimeoutMs, config_.verifySsl, stop, + config_.extraCaFile); + if (!tunnel.ok()) { + error = std::move(tunnel.error); + return false; } - auto rest = std::string_view(statusLine).substr(spacePos + 1); - auto spacePos2 = rest.find(' '); - if (spacePos2 != std::string_view::npos) { - auto codeStr = rest.substr(0, spacePos2); - response.statusCode = 0; - for (char c : codeStr) { - if (c >= '0' && c <= '9') { - response.statusCode = response.statusCode * 10 + (c - '0'); - } - } - response.statusText = std::string(rest.substr(spacePos2 + 1)); - } else { - // No status text, just code - response.statusCode = 0; - for (char c : rest) { - if (c >= '0' && c <= '9') { - response.statusCode = response.statusCode * 10 + (c - '0'); - } - } + if (tunnel.proxyTls) { + return sock.connect_over(std::move(tunnel.proxyTls), parsed.host.c_str(), + config_.verifySsl, config_.connectTimeoutMs); } + return sock.connect_over(std::move(tunnel.socket), parsed.host.c_str(), + config_.verifySsl, config_.connectTimeoutMs); } + return sock.connect(parsed.host.c_str(), parsed.port, + config_.connectTimeoutMs, config_.verifySsl); + } - // Read headers - bool chunked = false; - int contentLength = -1; - bool connectionClose = false; - - while (true) { - std::string headerLine = read_line(*sock, config_.readTimeoutMs); - if (headerLine.empty()) { - break; // End of headers (empty line after stripping \r\n) - } + // Acquires a connection for `poolKey`, sends `reqStr` and reads the response + // head, leaving `guard` armed for whatever is in the pool. + // + // A connection the server closed while it was idle fails here and nowhere + // else: the write lands in a local buffer and succeeds, and the read that + // follows sees the FIN. Nothing above the server's transport layer saw the + // request, so sending it again on a fresh connection is safe — once, and + // only when no response byte has arrived. See `retryOnStaleConnection`. + Exchange perform_exchange(const ParsedUrl& parsed, const std::string& poolKey, + const std::string& reqStr, PooledConnection& guard, + std::stop_token stop = {}) { + // Drops the connection; a stop request outranks whatever went wrong. + auto fail = [&](std::string message) -> Exchange { + guard.drop(); + if (stop.stop_requested()) return { nullptr, {}, "Cancelled", true }; + return { nullptr, {}, std::move(message) }; + }; - auto colonPos = headerLine.find(':'); - if (colonPos != std::string::npos) { - std::string key = headerLine.substr(0, colonPos); - std::string_view value = std::string_view(headerLine).substr(colonPos + 1); - // Trim leading whitespace from value - while (!value.empty() && value[0] == ' ') { - value = value.substr(1); - } - std::string valStr(value); - response.headers[key] = valStr; + for (int attempt = 0; attempt < 2; ++attempt) { + if (stop.stop_requested()) return fail(""); + bool reused = false; + TlsSocket* sock = nullptr; - if (iequals(key, "Transfer-Encoding") && iequals(valStr, "chunked")) { - chunked = true; - } - if (iequals(key, "Content-Length")) { - contentLength = 0; - for (char c : valStr) { - if (c >= '0' && c <= '9') { - contentLength = contentLength * 10 + (c - '0'); - } - } - } - if (iequals(key, "Connection") && iequals(valStr, "close")) { - connectionClose = true; - } + auto it = pool_.find(poolKey); + if (it != pool_.end() && it->second.is_valid()) { + sock = &it->second; + reused = true; + } else { + if (it != pool_.end()) pool_.erase(it); + auto [inserted, ok] = pool_.emplace(poolKey, TlsSocket{}); + sock = &inserted->second; } - } - - // Read body - if (request.method == Method::HEAD) { - // HEAD responses have no body - } else if (chunked) { - // Chunked transfer encoding - while (true) { - std::string sizeLine = read_line(*sock, config_.readTimeoutMs); - // Strip any chunk extensions (after semicolon) - auto semiPos = sizeLine.find(';'); - if (semiPos != std::string::npos) { - sizeLine = sizeLine.substr(0, semiPos); - } - // Trim whitespace - while (!sizeLine.empty() && (sizeLine.back() == ' ' || sizeLine.back() == '\t')) { - sizeLine.pop_back(); + sock->set_stop(stop); + if (!reused) { + std::string openError; + if (!open_connection(*sock, parsed, openError, stop)) { + // The proxy's refusal, else the TLS session's reason, + // else the TCP connection failed and there is no more to say. + std::string why = std::move(openError); + if (why.empty()) why = sock->error(); + if (why.empty()) why = "Connection failed"; + return fail(std::move(why)); } + } - int chunkSize = parse_hex(sizeLine); - if (chunkSize == 0) { - // Read trailing \r\n after last chunk - read_line(*sock, config_.readTimeoutMs); - break; - } + const bool mayRetry = reused && attempt == 0 + && config_.retryOnStaleConnection; + + // A failed write needs no `sawBytes` test, and the asymmetry with + // the read below is not an oversight. `write_all` returns true only + // when every byte went out, so a false here means an INCOMPLETE + // request reached the server — and a conforming server cannot act on + // one: it is still waiting for the blank line, or for the + // Content-Length bytes it was promised. Resending on a fresh + // connection therefore yields exactly one execution. + if (!write_all(*sock, reqStr, config_.readTimeoutMs)) { + if (mayRetry) { guard.reset(); continue; } + return fail("Write failed"); + } - // Read chunk data - std::string chunkData(chunkSize, '\0'); - if (!read_exact(*sock, chunkData.data(), chunkSize, config_.readTimeoutMs)) { - break; - } - response.body += chunkData; + auto head = read_response_head(*sock, config_.readTimeoutMs); + if (head) return { sock, std::move(*head), {} }; - // Read trailing \r\n after chunk - read_line(*sock, config_.readTimeoutMs); - } - } else if (contentLength >= 0) { - // Read exactly contentLength bytes - if (contentLength > 0) { - response.body.resize(contentLength); - if (!read_exact(*sock, response.body.data(), contentLength, config_.readTimeoutMs)) { - pool_.erase(poolKey); - return response; - } - } - } else { - // Read until connection closed - connectionClose = true; - char buf[4096]; - while (true) { - if (!sock->wait_readable(config_.readTimeoutMs)) { - break; - } - int ret = sock->read(buf, sizeof(buf)); - if (ret <= 0) break; - response.body.append(buf, ret); - } + // Only a silent connection is safe to resend on. Once a byte has + // arrived the server has seen the request, and repeating it could + // repeat its effect. + if (mayRetry && !head.error().sawBytes) { guard.reset(); continue; } + return fail(head.error().message); } + return fail("No response"); + } - // Handle connection pooling - if (connectionClose) { - sock->close(); - pool_.erase(poolKey); + // Reads the body into `sink` and settles the pool guard by what the read + // found. The single place that decides whether a connection is reusable. + void finish_body(HttpResponse& response, Exchange& exchange, Method method, + std::int64_t maxBytes, const BodySink& sink, + PooledConnection& guard, std::stop_token stop = {}) { + if (!response_has_body(method, exchange.head.statusCode)) { + if (config_.keepAlive && !exchange.head.connectionClose) guard.keep(); + return; } - // Follow 3xx redirects if configured - if (config_.maxRedirects > 0 && - response.statusCode >= 300 && response.statusCode < 400 && - redirectCount < config_.maxRedirects) { - std::string location; - for (const auto& [k, v] : response.headers) { - if (iequals(k, "location")) { - location = v; - break; - } - } - if (!location.empty()) { - // Resolve relative URLs - if (location.starts_with("/")) { - location = parsed.scheme + "://" + parsed.host + - (parsed.port != 443 ? ":" + std::to_string(parsed.port) : "") + - location; + auto outcome = read_body(*exchange.sock, exchange.head, + config_.readTimeoutMs, maxBytes, sink); + switch (outcome.end) { + case BodyEnd::Complete: + if (config_.keepAlive && !exchange.head.connectionClose) guard.keep(); + break; + case BodyEnd::Stopped: + // Not an error, but bytes are still owed on the socket. + response.bodyComplete = false; + response.bodyError = "stopped by callback"; + if (stop.stop_requested()) { + response.bodyError = "cancelled"; + response.cancelled = true; } - HttpRequest redirectReq = request; - redirectReq.url = location; - // Change POST to GET on 301/302/303 (standard behavior) - if (response.statusCode != 307 && response.statusCode != 308) { - redirectReq.method = Method::GET; - redirectReq.body.clear(); + break; + case BodyEnd::ClosedByPeer: + // The body ended where it said it would; the connection did too. + break; + case BodyEnd::Truncated: + response.bodyComplete = false; + response.bodyError = outcome.error; + if (stop.stop_requested()) { + response.bodyError = "cancelled"; + response.cancelled = true; } - return send_impl(redirectReq, redirectCount + 1); - } + break; } - - return response; } -public: - // Streaming SSE request — reads response body incrementally, feeding - // chunks through SseParser to the caller's callback. The callback - // receives each SseEvent and returns true to continue or false to stop. - HttpResponse send_stream(const HttpRequest& request, SseCallbackFn callback) { + HttpResponse send_impl(const HttpRequest& request, int redirectCount, + std::stop_token stop) { + if (stop.stop_requested()) return cancelled_response(); HttpResponse response; auto parsed = parse_url(request.url); @@ -581,250 +1193,85 @@ public: return response; } - std::string poolKey = parsed.host + ":" + std::to_string(parsed.port); + const std::string poolKey = parsed.host + ":" + std::to_string(parsed.port); + PooledConnection guard(pool_, poolKey); - // Get or create connection - TlsSocket* sock = nullptr; - auto it = pool_.find(poolKey); - if (it != pool_.end() && it->second.is_valid()) { - sock = &it->second; - } else { - if (it != pool_.end()) { - pool_.erase(it); - } - auto [insertIt, ok] = pool_.emplace(poolKey, TlsSocket{}); - sock = &insertIt->second; - bool connected = false; - if (config_.proxy.has_value()) { - auto proxyConf = parse_proxy_url(config_.proxy.value()); - auto tunnel = proxy_connect(proxyConf.host, proxyConf.port, - parsed.host, parsed.port, - config_.connectTimeoutMs); - if (tunnel.is_valid()) { - connected = sock->connect_over(std::move(tunnel), - parsed.host.c_str(), - config_.verifySsl); - } - } else { - connected = sock->connect(parsed.host.c_str(), parsed.port, - config_.connectTimeoutMs, config_.verifySsl); - } - if (!connected) { - pool_.erase(poolKey); - response.statusCode = 0; - response.statusText = "Connection failed"; - return response; - } - } - - // Build request — same as send() - std::string reqStr; - reqStr += method_to_string(request.method); - reqStr += " "; - reqStr += parsed.path; - reqStr += " HTTP/1.1\r\n"; - if (!has_header(request.headers, "Host")) { - reqStr += "Host: "; - reqStr += parsed.host; - if (parsed.port != 443) { - reqStr += ":"; - reqStr += std::to_string(parsed.port); - } - reqStr += "\r\n"; - } - if (!request.body.empty() && !has_header(request.headers, "Content-Length")) { - reqStr += "Content-Length: "; - reqStr += std::to_string(request.body.size()); - reqStr += "\r\n"; - } - for (const auto& [key, value] : request.headers) { - reqStr += key; - reqStr += ": "; - reqStr += value; - reqStr += "\r\n"; - } - if (!has_header(request.headers, "Connection")) { - if (config_.keepAlive) { - reqStr += "Connection: keep-alive\r\n"; - } else { - reqStr += "Connection: close\r\n"; - } - } - reqStr += "\r\n"; - if (!request.body.empty()) { - reqStr += request.body; - } - - if (!write_all(*sock, reqStr)) { - pool_.erase(poolKey); - response.statusCode = 0; - response.statusText = "Write failed"; - return response; - } - - // Read status line - std::string statusLine = read_line(*sock, config_.readTimeoutMs); - if (statusLine.empty()) { - pool_.erase(poolKey); + auto exchange = perform_exchange(parsed, poolKey, + build_request(request, parsed), guard, stop); + if (!exchange.error.empty()) { response.statusCode = 0; - response.statusText = "No response"; + response.statusText = exchange.error; + response.bodyComplete = false; + response.bodyError = exchange.error; + response.cancelled = exchange.cancelled; return response; } - // Parse status line - { - auto spacePos = statusLine.find(' '); - if (spacePos == std::string::npos) { - pool_.erase(poolKey); - response.statusCode = 0; - response.statusText = "Invalid status line"; - return response; - } - auto rest = std::string_view(statusLine).substr(spacePos + 1); - auto spacePos2 = rest.find(' '); - if (spacePos2 != std::string_view::npos) { - auto codeStr = rest.substr(0, spacePos2); - response.statusCode = 0; - for (char c : codeStr) { - if (c >= '0' && c <= '9') { - response.statusCode = response.statusCode * 10 + (c - '0'); - } - } - response.statusText = std::string(rest.substr(spacePos2 + 1)); - } else { - response.statusCode = 0; - for (char c : rest) { - if (c >= '0' && c <= '9') { - response.statusCode = response.statusCode * 10 + (c - '0'); - } - } - } - } - - // Read headers - bool chunked = false; - bool connectionClose = false; - - while (true) { - std::string headerLine = read_line(*sock, config_.readTimeoutMs); - if (headerLine.empty()) { - break; - } - auto colonPos = headerLine.find(':'); - if (colonPos != std::string::npos) { - std::string key = headerLine.substr(0, colonPos); - std::string_view value = std::string_view(headerLine).substr(colonPos + 1); - while (!value.empty() && value[0] == ' ') { - value = value.substr(1); - } - std::string valStr(value); - response.headers[key] = valStr; - - if (iequals(key, "Transfer-Encoding") && iequals(valStr, "chunked")) { - chunked = true; - } - if (iequals(key, "Connection") && iequals(valStr, "close")) { - connectionClose = true; - } - } + response.statusCode = exchange.head.statusCode; + response.statusText = exchange.head.statusText; + response.headers = exchange.head.headers; + + // Reserving the declared length would let a server ask for an allocation + // by writing a header. Reserve a modest amount and let the string grow; + // `maxResponseBodyBytes` is what actually bounds the total. + if (exchange.head.contentLength > 0) { + constexpr std::int64_t reserveCap = 1024 * 1024; + response.body.reserve(static_cast( + exchange.head.contentLength < reserveCap + ? exchange.head.contentLength : reserveCap)); } - // Stream body incrementally, feeding chunks to SseParser - SseParser parser; - bool stopped = false; + finish_body(response, exchange, request.method, + config_.maxResponseBodyBytes, + [&](std::string_view data) { + response.body.append(data); + return true; + }, + guard, stop); - auto dispatch = [&](std::string_view data) -> bool { - auto events = parser.feed(data); - for (const auto& ev : events) { - if (!callback(ev)) { - stopped = true; - return false; - } - } - return true; - }; - - if (chunked) { - // Incrementally decode chunked transfer-encoding - while (!stopped) { - std::string sizeLine = read_line(*sock, config_.readTimeoutMs); - auto semiPos = sizeLine.find(';'); - if (semiPos != std::string::npos) { - sizeLine = sizeLine.substr(0, semiPos); - } - while (!sizeLine.empty() && (sizeLine.back() == ' ' || sizeLine.back() == '\t')) { - sizeLine.pop_back(); - } - - int chunkSize = parse_hex(sizeLine); - if (chunkSize == 0) { - // Terminal chunk — read trailing \r\n - read_line(*sock, config_.readTimeoutMs); - break; - } - - // Read chunk data - std::string chunkData(chunkSize, '\0'); - if (!read_exact(*sock, chunkData.data(), chunkSize, config_.readTimeoutMs)) { - break; - } - // Read trailing \r\n after chunk - read_line(*sock, config_.readTimeoutMs); - - if (!dispatch(chunkData)) { - break; - } - } - } else { - // Not chunked — read until connection closes - connectionClose = true; - char buf[4096]; - while (!stopped) { - if (!sock->wait_readable(config_.readTimeoutMs)) { - break; + // Follow 3xx redirects if configured. + if (config_.maxRedirects > 0 && !response.cancelled && + response.statusCode >= 300 && response.statusCode < 400 && + redirectCount < config_.maxRedirects) { + std::string location = find_header(response.headers, "Location"); + if (!location.empty()) { + if (location.starts_with("/")) { + location = parsed.scheme + "://" + parsed.host + + (parsed.port != 443 ? ":" + std::to_string(parsed.port) : "") + + location; } - int ret = sock->read(buf, sizeof(buf)); - if (ret <= 0) break; - if (!dispatch(std::string_view(buf, static_cast(ret)))) { - break; + HttpRequest redirectReq = request; + redirectReq.url = location; + // Change POST to GET on 301/302/303 (standard behavior) + if (response.statusCode != 307 && response.statusCode != 308) { + redirectReq.method = Method::GET; + redirectReq.body.clear(); } + // Settle before recursing: the inner call may put its own + // connection into the pool under this same key, and a guard + // still armed would delete it when this scope ends. + guard.drop(); + return send_impl(redirectReq, redirectCount + 1, stop); } } - // Clean up connection - if (connectionClose || stopped) { - sock->close(); - pool_.erase(poolKey); - } - return response; } - // Download URL to file with streaming progress. - // Follows redirects. Calls onProgress periodically during body read. - // isCancelled is checked after each block — return true to abort. - DownloadToFileResult download_to_file( - const std::string& url, - const std::filesystem::path& destFile, - DownloadProgressFn onProgress = nullptr, - std::function isCancelled = nullptr) - { - return download_to_file_impl(url, destFile, std::move(onProgress), - std::move(isCancelled), 0); - } - - HttpClientConfig& config() { return config_; } - const HttpClientConfig& config() const { return config_; } - -private: DownloadToFileResult download_to_file_impl( const std::string& url, const std::filesystem::path& destFile, DownloadProgressFn onProgress, std::function isCancelled, - int redirectCount) + int redirectCount, + std::stop_token stop) { DownloadToFileResult result; + if (stop.stop_requested()) { + result.error = "Cancelled"; + result.cancelled = true; + return result; + } auto parsed = parse_url(url); if (parsed.scheme != "https") { @@ -832,288 +1279,200 @@ private: return result; } - std::string poolKey = parsed.host + ":" + std::to_string(parsed.port); - - // Get or create connection - TlsSocket* sock = nullptr; - auto it = pool_.find(poolKey); - if (it != pool_.end() && it->second.is_valid()) { - sock = &it->second; - } else { - if (it != pool_.end()) pool_.erase(it); - auto [insertIt, ok] = pool_.emplace(poolKey, TlsSocket{}); - sock = &insertIt->second; - bool connected = false; - if (config_.proxy.has_value()) { - auto proxyConf = parse_proxy_url(config_.proxy.value()); - auto tunnel = proxy_connect(proxyConf.host, proxyConf.port, - parsed.host, parsed.port, - config_.connectTimeoutMs); - if (tunnel.is_valid()) { - connected = sock->connect_over(std::move(tunnel), - parsed.host.c_str(), - config_.verifySsl); - } - } else { - connected = sock->connect(parsed.host.c_str(), parsed.port, - config_.connectTimeoutMs, config_.verifySsl); - } - if (!connected) { - pool_.erase(poolKey); - result.error = "Connection failed"; - return result; - } - } + const std::string poolKey = parsed.host + ":" + std::to_string(parsed.port); + PooledConnection guard(pool_, poolKey); - // Build GET request - std::string reqStr = "GET "; - reqStr += parsed.path; - reqStr += " HTTP/1.1\r\nHost: "; - reqStr += parsed.host; - if (parsed.port != 443) { - reqStr += ":"; - reqStr += std::to_string(parsed.port); - } - reqStr += "\r\nUser-Agent: tinyhttps/1.0\r\nAccept: */*\r\n"; - reqStr += config_.keepAlive ? "Connection: keep-alive\r\n" : "Connection: close\r\n"; - reqStr += "\r\n"; + HttpRequest request; + request.method = Method::GET; + request.url = url; + request.headers = { {"User-Agent", "tinyhttps/1.0"}, {"Accept", "*/*"} }; - if (!write_all(*sock, reqStr)) { - pool_.erase(poolKey); - result.error = "Write failed"; + auto exchange = perform_exchange(parsed, poolKey, + build_request(request, parsed), guard, stop); + if (!exchange.error.empty()) { + result.error = exchange.error; + result.cancelled = exchange.cancelled; return result; } - // Read status line - std::string statusLine = read_line(*sock, config_.readTimeoutMs); - if (statusLine.empty()) { - pool_.erase(poolKey); - result.error = "No response"; - return result; - } + result.statusCode = exchange.head.statusCode; + const bool hasBody = response_has_body(Method::GET, exchange.head.statusCode); - // Parse status code - { - auto sp = statusLine.find(' '); - if (sp != std::string::npos) { - auto rest = std::string_view(statusLine).substr(sp + 1); - for (char c : rest) { - if (c >= '0' && c <= '9') - result.statusCode = result.statusCode * 10 + (c - '0'); - else break; + // Follow redirects. + if (result.statusCode >= 300 && result.statusCode < 400 && + redirectCount < config_.maxRedirects) { + std::string location = find_header(exchange.head.headers, "Location"); + if (!location.empty()) { + // A 3xx normally carries a short explanatory body, and the code + // here promised in a comment to "drain any body to keep the + // connection clean" without ever draining one. The socket went + // back into the pool with those bytes owed, and the recursive + // call below picked it up for the very next request — issue #15's + // failure mode, on a path with no timeout and no truncation. + const bool clean = !hasBody + || drain_body(*exchange.sock, exchange.head, config_.readTimeoutMs); + if (clean && config_.keepAlive && !exchange.head.connectionClose) { + guard.keep(); + } else { + guard.drop(); } - } - } - - // Read headers - bool chunked = false; - std::int64_t contentLength = -1; - bool connectionClose = false; - std::string location; - std::string etag; - std::string lastModified; - - while (true) { - std::string line = read_line(*sock, config_.readTimeoutMs); - if (line.empty()) break; - auto colon = line.find(':'); - if (colon == std::string::npos) continue; - std::string key = line.substr(0, colon); - std::string_view val = std::string_view(line).substr(colon + 1); - while (!val.empty() && val[0] == ' ') val = val.substr(1); - std::string valStr(val); - - if (iequals(key, "Transfer-Encoding") && iequals(valStr, "chunked")) - chunked = true; - if (iequals(key, "Content-Length")) { - contentLength = 0; - for (char c : valStr) { - if (c >= '0' && c <= '9') - contentLength = contentLength * 10 + (c - '0'); + if (stop.stop_requested()) { + result.error = "cancelled"; + result.cancelled = true; + return result; } - } - if (iequals(key, "Connection") && iequals(valStr, "close")) - connectionClose = true; - if (iequals(key, "Location")) - location = valStr; - if (iequals(key, "ETag")) - etag = valStr; - if (iequals(key, "Last-Modified")) - lastModified = valStr; - } - // Follow redirects - if (result.statusCode >= 300 && result.statusCode < 400 && - !location.empty() && redirectCount < config_.maxRedirects) { - // Drain any body to keep connection clean - if (connectionClose) { - sock->close(); - pool_.erase(poolKey); - } - // Resolve relative URL - if (location.starts_with("/")) { - location = parsed.scheme + "://" + parsed.host + - (parsed.port != 443 ? ":" + std::to_string(parsed.port) : "") + - location; + if (location.starts_with("/")) { + location = parsed.scheme + "://" + parsed.host + + (parsed.port != 443 ? ":" + std::to_string(parsed.port) : "") + + location; + } + return download_to_file_impl(location, destFile, std::move(onProgress), + std::move(isCancelled), redirectCount + 1, + stop); } - return download_to_file_impl(location, destFile, std::move(onProgress), - std::move(isCancelled), redirectCount + 1); } + // A non-2xx carries an error document that nobody is going to read. + // Draining it is what keeps the connection usable for the next request. if (result.statusCode < 200 || result.statusCode >= 300) { result.error = "HTTP " + std::to_string(result.statusCode); - if (connectionClose) { sock->close(); pool_.erase(poolKey); } + settle_without_body(exchange, hasBody, guard); return result; } result.finalUrl = url; - result.etag = std::move(etag); - result.lastModified = std::move(lastModified); - if (contentLength >= 0) result.expectedBytes = contentLength; + result.etag = find_header(exchange.head.headers, "ETag"); + result.lastModified = find_header(exchange.head.headers, "Last-Modified"); + if (exchange.head.contentLength >= 0) { + result.expectedBytes = exchange.head.contentLength; + } - // Open output file std::error_code ec; std::filesystem::create_directories(destFile.parent_path(), ec); std::ofstream ofs(destFile, std::ios::binary); if (!ofs) { result.error = "Cannot open file: " + destFile.string(); - if (connectionClose) { sock->close(); pool_.erase(poolKey); } + result.writeFailed = true; + settle_without_body(exchange, hasBody, guard); return result; } - std::int64_t totalBytes = contentLength > 0 ? contentLength : 0; - std::int64_t downloaded = 0; - - // Check cancellation between blocks - auto cancelled = [&]() -> bool { - if (isCancelled && isCancelled()) { - result.error = "cancelled"; - ofs.close(); - result.bytesWritten = downloaded; - sock->close(); - pool_.erase(poolKey); - return true; + // What the file refused, in the words of the system that refused it. + // `errno` is cleared before each operation that can set it, because the + // socket reads between writes leave their own values behind and a stale + // EAGAIN reported as the reason a disk filled is worse than no reason. + int writeErrno = 0; + bool writeFailed = false; + auto close_file = [&] { + errno = 0; + ofs.close(); + if (!writeFailed && ofs.fail()) { + writeFailed = true; + writeErrno = errno; } - return false; + }; + auto write_failure = [&] { + return "write " + destFile.string() + ": " + + (writeErrno != 0 + ? std::generic_category().message(writeErrno) + : std::string("stream error")); }; - // Read body and stream to file - if (chunked) { - while (true) { - if (cancelled()) return result; - auto sizeResult = read_complete_line( - *sock, config_.readTimeoutMs); - if (!sizeResult) { - result.error = "Invalid chunk size line: " - + std::move(sizeResult).error(); - result.bytesWritten = downloaded; - sock->close(); - pool_.erase(poolKey); - return result; - } - std::string sizeLine = std::move(*sizeResult); - auto semi = sizeLine.find(';'); - if (semi != std::string::npos) sizeLine = sizeLine.substr(0, semi); - while (!sizeLine.empty() && (sizeLine.back() == ' ' || sizeLine.back() == '\t')) - sizeLine.pop_back(); - - auto parsedChunkSize = parse_chunk_size_line(sizeLine); - if (!parsedChunkSize) { - result.error = "Invalid chunk size: " + sizeLine; - result.bytesWritten = downloaded; - sock->close(); - pool_.erase(poolKey); - return result; - } - int chunkSize = static_cast(*parsedChunkSize); - if (chunkSize == 0) { - for (;;) { - auto trailer = read_complete_line( - *sock, config_.readTimeoutMs); - if (!trailer) { - result.error = "Invalid chunk trailer: " - + std::move(trailer).error(); - result.bytesWritten = downloaded; - sock->close(); - pool_.erase(poolKey); - return result; - } - if (trailer->empty()) break; - } - break; - } + if (!hasBody) { + close_file(); + if (writeFailed) { + result.error = write_failure(); + result.writeFailed = true; + return result; + } + if (config_.keepAlive && !exchange.head.connectionClose) guard.keep(); + return result; + } - int remaining = chunkSize; - char buf[8192]; - while (remaining > 0) { - if (cancelled()) return result; - int toRead = remaining > static_cast(sizeof(buf)) - ? static_cast(sizeof(buf)) : remaining; - if (!read_exact(*sock, buf, toRead, config_.readTimeoutMs)) { - result.error = "Read error during chunked transfer"; - ofs.close(); - result.bytesWritten = downloaded; - return result; - } - ofs.write(buf, toRead); - downloaded += toRead; - remaining -= toRead; - if (onProgress) onProgress(totalBytes, downloaded); + const std::int64_t totalBytes = + exchange.head.contentLength > 0 ? exchange.head.contentLength : 0; + std::int64_t received = 0; + std::int64_t written = 0; + bool cancelled = false; + + auto outcome = read_body( + *exchange.sock, exchange.head, config_.readTimeoutMs, + std::numeric_limits::max(), + [&](std::string_view data) -> bool { + const auto size = static_cast(data.size()); + received += size; + + // Flushed per chunk, and the stream checked after it. A write + // into the stream's buffer succeeds whatever the disk has left; + // the failure surfaces at the flush that hands the buffer to the + // operating system, and without one here it surfaced at + // `close()`, after every chunk had been counted as written. + // `read_body` hands over slices of at most 8 KiB, about the size + // of the stream's own buffer, so this costs no more system calls + // than the buffering it replaces. + errno = 0; + ofs.write(data.data(), static_cast(data.size())); + ofs.flush(); + if (!ofs) { + writeFailed = true; + writeErrno = errno; + return false; // reading on would only discard what it reads } - auto delimiter = read_complete_line( - *sock, config_.readTimeoutMs); - if (!delimiter || !delimiter->empty()) { - result.error = "Missing CRLF after chunk data"; - result.bytesWritten = downloaded; - sock->close(); - pool_.erase(poolKey); - return result; - } - } - } else if (contentLength > 0) { - char buf[8192]; - std::int64_t remaining = contentLength; - while (remaining > 0) { - if (cancelled()) return result; - int toRead = remaining > static_cast(sizeof(buf)) - ? static_cast(sizeof(buf)) - : static_cast(remaining); - if (!read_exact(*sock, buf, toRead, config_.readTimeoutMs)) { - result.error = "Read error"; - ofs.close(); - result.bytesWritten = downloaded; - return result; + + written += size; + if (onProgress) onProgress(totalBytes, written); + if ((isCancelled && isCancelled()) || stop.stop_requested()) { + cancelled = true; + return false; } - ofs.write(buf, toRead); - downloaded += toRead; - remaining -= toRead; - if (onProgress) onProgress(totalBytes, downloaded); - } - } else { - connectionClose = true; - char buf[8192]; - while (true) { - if (cancelled()) return result; - if (!sock->wait_readable(config_.readTimeoutMs)) break; - int ret = sock->read(buf, sizeof(buf)); - if (ret <= 0) break; - ofs.write(buf, ret); - downloaded += ret; - if (onProgress) onProgress(totalBytes, downloaded); - } - } + return true; + }); - ofs.close(); - result.bytesWritten = downloaded; + close_file(); + result.bytesWritten = written; + result.bytesReceived = received; - if (connectionClose) { - sock->close(); - pool_.erase(poolKey); + switch (outcome.end) { + case BodyEnd::Complete: + if (config_.keepAlive && !exchange.head.connectionClose) guard.keep(); + break; + case BodyEnd::ClosedByPeer: + break; // the body ended where it said it would + case BodyEnd::Stopped: + result.error = cancelled ? "cancelled" : "stopped"; + break; + case BodyEnd::Truncated: + // A wait that the stop token ended reads as a truncated body. + cancelled = stop.stop_requested(); + result.error = cancelled ? "cancelled" : outcome.error; + break; + } + result.cancelled = cancelled; + + // The destination's failure outranks whatever the switch recorded: a + // stop that the file asked for is not a cancellation, and a body that + // arrived whole into a file that lost it has not succeeded. + if (writeFailed) { + result.error = write_failure(); + result.writeFailed = true; + result.cancelled = false; } - return result; } + // Settles the guard for a response whose body the caller will not be given: + // drain it if it is short enough, and drop the connection otherwise. + void settle_without_body(Exchange& exchange, bool hasBody, PooledConnection& guard) { + const bool clean = !hasBody + || drain_body(*exchange.sock, exchange.head, config_.readTimeoutMs); + if (clean && config_.keepAlive && !exchange.head.connectionClose) { + guard.keep(); + } else { + guard.drop(); + } + } + HttpClientConfig config_; std::map pool_; }; diff --git a/src/platform.cppm b/src/platform.cppm index 251bf52..dcf3ade 100644 --- a/src/platform.cppm +++ b/src/platform.cppm @@ -1,6 +1,15 @@ module; -#ifndef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifndef TINYHTTPS_WINSOCK #include #include #include @@ -14,9 +23,39 @@ import std; // mcpplibs.tinyhttps:platform — the one place OS-specific networking quirks // live. Everything else (socket.cppm, http.cppm…) stays portable and branches -// on `platform::is_windows` with `if constexpr`, never raw #ifdef. +// on `platform::uses_winsock` with `if constexpr`, never raw #ifdef. +// +// ── The platforms this library is known to run on, and what differs ────────── +// +// | platform | not raising SIGPIPE | notes | +// |-------------------|--------------------------|----------------------------| +// | Linux, Android | MSG_NOSIGNAL on send() | | +// | Termux | MSG_NOSIGNAL on send() | manual DNS, below | +// | macOS, the BSDs | SO_NOSIGPIPE on socket() | no MSG_NOSIGNAL | +// | Windows | nothing to do | no SIGPIPE, neither macro | +// | above openkal | MSG_NOSIGNAL, and moot | see below | +// +// ABOVE openkal there are no signals at all, so the hazard issue #16 +// describes cannot arise — and the same code is nevertheless correct there +// without a branch. openkal-musl defines MSG_NOSIGNAL +// (`musl/include/sys/socket.h:344`) and accepts it as a no-op, in its own words: +// "MSG_NOSIGNAL asks that a signal not be raised. There are no signals here, so +// the request is satisfied by there being nothing to raise" +// (`port/src/okm_net.c:546-550`); a flag it does not know it refuses with +// -ENOSYS (`:550`, `:592-593`). SO_NOSIGPIPE is a BSD spelling that musl does +// not define, so `socket.cppm`'s `#ifdef` for it simply does not compile in. +// +// One difference that IS live above openkal: connect() completes before it +// returns even on a non-blocking descriptor, because `kal_net_connect` has no +// form that begins a connection and reports its outcome later +// (`port/src/okm_net.c:454-467`). `Socket::connect_addrinfo` therefore never +// sees EINPROGRESS, never reaches its poll, and `connectTimeoutMs` is not +// enforced for the TCP connect on that stack. Nothing here can fix that — the +// bound would have to come from openkal — and nothing here needs to know it at +// compile time, which is why there is no `is_openkal`. // -// Sole concern today: hostname resolution on platforms where libc can't do it. +// Sole runtime concern today: hostname resolution on platforms where libc can't +// do it. // A musl-static binary on Android/Termux can't resolve through getaddrinfo() — // libc reads /etc/resolv.conf, but Android's /etc is read-only with no // resolv.conf; Termux keeps its nameservers in $PREFIX/etc/resolv.conf, which @@ -28,14 +67,14 @@ import std; namespace mcpplibs::tinyhttps::platform { // Compile-time platform flag for `if constexpr` at call sites. -export inline constexpr bool is_windows = -#ifdef _WIN32 +export inline constexpr bool uses_winsock = +#ifdef TINYHTTPS_WINSOCK true; #else false; #endif -#ifndef _WIN32 +#ifndef TINYHTTPS_WINSOCK namespace { @@ -155,7 +194,7 @@ inline std::vector dns_query_a(const std::string& server, const cha } // anonymous namespace -#endif // !_WIN32 +#endif // !TINYHTTPS_WINSOCK // True when libc's own resolver has a usable config (/etc/resolv.conf). When // false, callers should prefer resolve_fallback() to avoid a multi-second stall @@ -165,9 +204,9 @@ inline std::vector dns_query_a(const std::string& server, const cha // `if constexpr` still compiles its discarded branch, which would reference the // POSIX-only helpers above that don't exist on Windows. Concentrating that // preprocessor divergence here is exactly why this platform module exists; call -// sites elsewhere branch on `is_windows` with `if constexpr`. +// sites elsewhere branch on `uses_winsock` with `if constexpr`. export bool system_resolver_configured() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK return true; #else std::error_code ec; @@ -181,7 +220,7 @@ export bool system_resolver_configured() { // on query failure. A numeric host is returned unchanged. export std::vector resolve_fallback([[maybe_unused]] const char* host, [[maybe_unused]] int timeoutMs) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK return {}; #else if (is_numeric_host(host)) return { std::string(host) }; diff --git a/src/proxy.cppm b/src/proxy.cppm index e102de6..bd51d92 100644 --- a/src/proxy.cppm +++ b/src/proxy.cppm @@ -1,155 +1,401 @@ export module mcpplibs.tinyhttps:proxy; import :socket; +import :tls; import std; namespace mcpplibs::tinyhttps { +// A parsed proxy URL: where the proxy is, how to talk to it, and the credentials +// to send. `parse_proxy_url` fills it in; nothing validates it until +// `proxy_tunnel` tries to use it. export struct ProxyConfig { std::string host; int port { 8080 }; + + // Lower case: "http", "https" for a proxy that is itself reached over TLS, + // "socks5" or "socks5h". "http" when the URL has no scheme, which is how a + // bare `host:port` has always been read. With "socks5" this library + // resolves the target's name and gives the proxy an address; with "socks5h" + // it gives the proxy the name. + std::string scheme { "http" }; + + // From `user:password@` in the URL, percent-decoded. A URL with `@` and no + // password still has credentials; `hasCredentials` is what says so, because + // an empty user or an empty password is a legitimate value. + bool hasCredentials { false }; + std::string user; + std::string password; }; -// Parse "http://host:port" proxy URL +static int hex_digit(char c) { + if (c >= '0' && c <= '9') return c - '0'; + if (c >= 'a' && c <= 'f') return c - 'a' + 10; + if (c >= 'A' && c <= 'F') return c - 'A' + 10; + return -1; +} + +// "%40" is "@". A '%' that is not followed by two hex digits stays as written. +static std::string percent_decode(std::string_view in) { + std::string out; + out.reserve(in.size()); + for (std::size_t i = 0; i < in.size(); ++i) { + if (in[i] == '%' && i + 2 < in.size() && hex_digit(in[i + 1]) >= 0 + && hex_digit(in[i + 2]) >= 0) { + out += static_cast(hex_digit(in[i + 1]) * 16 + hex_digit(in[i + 2])); + i += 2; + } else { + out += in[i]; + } + } + return out; +} + +// Parse "[scheme://][user[:password]@]host[:port]". A path, query or fragment +// after the authority is ignored. IPv6 literals are written in brackets. The +// port defaults to 443 for https, 1080 for socks5 and socks5h, and 8080 +// otherwise. export ProxyConfig parse_proxy_url(std::string_view url) { ProxyConfig config; - // Strip scheme if present auto schemeEnd = url.find("://"); if (schemeEnd != std::string_view::npos) { + config.scheme.clear(); + for (char c : url.substr(0, schemeEnd)) { + config.scheme += (c >= 'A' && c <= 'Z') ? static_cast(c + 32) : c; + } url = url.substr(schemeEnd + 3); } - // Strip trailing path if present - auto pathStart = url.find('/'); - if (pathStart != std::string_view::npos) { - url = url.substr(0, pathStart); + url = url.substr(0, url.find_first_of("/?#")); + + // The last '@' separates the userinfo, so an unencoded '@' in a password + // still parses the way curl reads it. + if (auto at = url.rfind('@'); at != std::string_view::npos) { + auto userinfo = url.substr(0, at); + url = url.substr(at + 1); + auto colon = userinfo.find(':'); + config.hasCredentials = true; + config.user = percent_decode(userinfo.substr(0, colon)); + if (colon != std::string_view::npos) { + config.password = percent_decode(userinfo.substr(colon + 1)); + } } - // Check for port - auto colonPos = url.find(':'); - if (colonPos != std::string_view::npos) { - config.host = std::string(url.substr(0, colonPos)); - auto portStr = url.substr(colonPos + 1); - config.port = 0; - for (char c : portStr) { - if (c >= '0' && c <= '9') { - config.port = config.port * 10 + (c - '0'); - } + std::string_view portStr; + bool hasPort = false; + if (!url.empty() && url.front() == '[') { + auto close = url.find(']'); + config.host = std::string(url.substr(1, close == std::string_view::npos + ? std::string_view::npos : close - 1)); + if (close != std::string_view::npos && close + 1 < url.size() && url[close + 1] == ':') { + portStr = url.substr(close + 2); + hasPort = true; } + } else if (auto colon = url.find(':'); colon != std::string_view::npos) { + config.host = std::string(url.substr(0, colon)); + portStr = url.substr(colon + 1); + hasPort = true; } else { config.host = std::string(url); - config.port = 8080; + } + + if (config.scheme == "https") config.port = 443; + if (config.scheme.starts_with("socks")) config.port = 1080; + + if (hasPort && !portStr.empty()) { + // Anything but 1..65535 becomes 0, which no connect will accept. + long port = 0; + for (char c : portStr) { + if (c < '0' || c > '9' || (port = port * 10 + (c - '0')) > 65535) { port = 0; break; } + } + config.port = static_cast(port); } return config; } -// Read a line (ending with \r\n) from a plain Socket -static std::string read_line_plain(Socket& sock, int timeoutMs) { +static std::string base64_encode(std::string_view in) { + constexpr std::string_view table = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + std::string out; + for (std::size_t i = 0; i < in.size(); i += 3) { + unsigned v = static_cast(in[i]) << 16; + if (i + 1 < in.size()) v |= static_cast(in[i + 1]) << 8; + if (i + 2 < in.size()) v |= static_cast(in[i + 2]); + out += table[(v >> 18) & 63]; + out += table[(v >> 12) & 63]; + out += i + 1 < in.size() ? table[(v >> 6) & 63] : '='; + out += i + 2 < in.size() ? table[v & 63] : '='; + } + return out; +} + +// One line without its CRLF. nullopt on a timeout, an end of stream, or a line +// longer than any proxy sends. Reads a byte at a time on purpose: whatever +// follows the blank line belongs to the tunnel and must not be consumed. +template +static std::optional read_line(Stream& sock, int timeoutMs) { std::string line; char c; - while (true) { - if (!sock.wait_readable(timeoutMs)) { - break; - } + while (line.size() < 8192) { + if (!sock.wait_readable(timeoutMs)) return std::nullopt; int ret = sock.read(&c, 1); - if (ret < 0) break; if (ret == 0) { - if (!sock.wait_readable(timeoutMs)) break; + if (!sock.wait_readable(timeoutMs)) return std::nullopt; ret = sock.read(&c, 1); - if (ret <= 0) break; } + if (ret <= 0) return std::nullopt; line += c; - if (line.size() >= 2 && line[line.size() - 2] == '\r' && line[line.size() - 1] == '\n') { + if (line.size() >= 2 && line[line.size() - 2] == '\r' && line.back() == '\n') { line.resize(line.size() - 2); - break; + return line; } } - return line; + return std::nullopt; } -// Write all data to a plain Socket -static bool write_all_plain(Socket& sock, const std::string& data) { - int total = 0; - int len = static_cast(data.size()); - while (total < len) { - int ret = sock.write(data.c_str() + total, len - total); +template +static bool write_all(Stream& sock, std::string_view data, int timeoutMs) { + std::size_t total = 0; + while (total < data.size()) { + int ret = sock.write(data.data() + total, static_cast(data.size() - total)); if (ret < 0) return false; - if (ret == 0) { - ret = sock.write(data.c_str() + total, len - total); - if (ret <= 0) return false; - } - total += ret; + if (ret == 0 && !sock.wait_writable(timeoutMs)) return false; + total += static_cast(ret); } return true; } -// Connect through HTTP CONNECT proxy, returning a Socket connected to target through tunnel. -// On failure, returns an invalid (closed) Socket. -export Socket proxy_connect(std::string_view proxyHost, int proxyPort, - std::string_view targetHost, int targetPort, - int timeoutMs) { - Socket sock; - - // Step 1: Connect to proxy - std::string proxyHostStr(proxyHost); - if (!sock.connect(proxyHostStr.c_str(), proxyPort, timeoutMs)) { - return sock; - } - - // Step 2: Send CONNECT request - std::string request = "CONNECT "; - request += targetHost; - request += ":"; - request += std::to_string(targetPort); - request += " HTTP/1.1\r\nHost: "; - request += targetHost; - request += ":"; - request += std::to_string(targetPort); - request += "\r\n\r\n"; - - if (!write_all_plain(sock, request)) { - sock.close(); - return sock; - } - - // Step 3: Read response status line - std::string statusLine = read_line_plain(sock, timeoutMs); - if (statusLine.empty()) { - sock.close(); - return sock; - } - - // Parse status code from "HTTP/1.x 200 ..." - int statusCode = 0; - auto spacePos = statusLine.find(' '); - if (spacePos != std::string::npos) { - auto rest = std::string_view(statusLine).substr(spacePos + 1); - for (char c : rest) { - if (c >= '0' && c <= '9') { - statusCode = statusCode * 10 + (c - '0'); - } else { - break; - } +// "HTTP/1.1 407 Proxy Authentication Required" -> 407, or 0 when the line is +// not a status line. +static int status_code_of(std::string_view line) { + if (!line.starts_with("HTTP/1.") || line.size() < 12 || line[8] != ' ') return 0; + int code = 0; + for (char c : line.substr(9, 3)) { + if (c < '0' || c > '9') return 0; + code = code * 10 + (c - '0'); + } + return code; +} + +// Ask an HTTP proxy to open a tunnel to host:port. Returns an error message, or +// an empty string once the proxy has answered 2xx and the stream is the tunnel. +template +static std::string http_connect(Stream& sock, const ProxyConfig& proxy, + std::string_view host, int port, int timeoutMs) { + // An IPv6 literal needs its brackets back in the request target. + std::string authority = host.contains(':') ? "[" + std::string(host) + "]" : std::string(host); + authority += ":" + std::to_string(port); + + std::string request = "CONNECT " + authority + " HTTP/1.1\r\nHost: " + authority + "\r\n"; + if (proxy.hasCredentials) { + request += "Proxy-Authorization: Basic " + + base64_encode(proxy.user + ":" + proxy.password) + "\r\n"; + } + request += "\r\n"; + + if (!write_all(sock, request, timeoutMs)) return "proxy: could not send CONNECT"; + + auto statusLine = read_line(sock, timeoutMs); + int code = statusLine ? status_code_of(*statusLine) : 0; + if (code == 0) return "proxy: no HTTP response to CONNECT"; + if (code == 407) { + return std::string(proxy.hasCredentials ? "proxy rejected the credentials: " + : "proxy requires credentials: ") + + statusLine->substr(9); + } + if (code < 200 || code > 299) return "proxy refused CONNECT: " + statusLine->substr(9); + + // The rest of the head. A 2xx answer to CONNECT carries no body. + for (int i = 0; i < 100; ++i) { + auto header = read_line(sock, timeoutMs); + if (!header) return "proxy: response to CONNECT was cut short"; + if (header->empty()) return {}; + } + return "proxy: response to CONNECT has too many headers"; +} + +// SOCKS5 (RFC 1928), with the username/password method of RFC 1929. + +static bool read_exact(Socket& sock, unsigned char* buf, std::size_t n, int timeoutMs) { + std::size_t got = 0; + while (got < n) { + if (!sock.wait_readable(timeoutMs)) return false; + int ret = sock.read(reinterpret_cast(buf) + got, static_cast(n - got)); + if (ret <= 0) return false; // 0 is the proxy closing the connection + got += static_cast(ret); + } + return true; +} + +static std::string_view socks5_reply_text(unsigned char code) { + switch (code) { + case 1: return "general failure"; + case 2: return "connection not allowed by ruleset"; + case 3: return "network unreachable"; + case 4: return "host unreachable"; + case 5: return "connection refused"; + case 6: return "TTL expired"; + case 7: return "command not supported"; + case 8: return "address type not supported"; + } + return "unknown error"; +} + +// Returns an error message, or an empty string once the proxy has connected to +// the target and the stream is the tunnel. +static std::string socks5_connect(Socket& sock, const ProxyConfig& proxy, + std::string_view host, int port, int timeoutMs) { + const bool remoteDns = proxy.scheme == "socks5h"; + const std::string hostStr(host); + + // Everything that can be refused before a byte is sent is refused first. + if (proxy.hasCredentials && (proxy.user.size() > 255 || proxy.password.size() > 255)) { + return "proxy: SOCKS5 credentials are limited to 255 bytes each"; + } + // An address literal is always sent as an address; a name is sent as a name + // for socks5h and resolved here for socks5. + auto address = Socket::resolve_address(hostStr.c_str(), /*numericOnly=*/true); + if (address.empty() && remoteDns) { + if (host.empty() || host.size() > 255) return "proxy: host name does not fit a SOCKS5 request"; + } else if (address.empty()) { + address = Socket::resolve_address(hostStr.c_str()); + if (address.empty()) return "proxy: could not resolve " + hostStr; + } + + std::string greeting = proxy.hasCredentials ? std::string("\x05\x02\x00\x02", 4) + : std::string("\x05\x01\x00", 3); + if (!write_all(sock, greeting, timeoutMs)) return "proxy: could not send the SOCKS5 greeting"; + + unsigned char choice[2]; + if (!read_exact(sock, choice, 2, timeoutMs)) return "proxy: no SOCKS5 response"; + if (choice[0] != 5) return "proxy: not a SOCKS5 proxy"; + if (choice[1] == 0xFF) { + return proxy.hasCredentials + ? "proxy: SOCKS5 proxy accepts neither no authentication nor a username and password" + : "proxy requires credentials: no acceptable SOCKS5 authentication method"; + } + if (choice[1] == 2 && proxy.hasCredentials) { + std::string auth = "\x01"; + auth += static_cast(proxy.user.size()); + auth += proxy.user; + auth += static_cast(proxy.password.size()); + auth += proxy.password; + unsigned char status[2]; + if (!write_all(sock, auth, timeoutMs) || !read_exact(sock, status, 2, timeoutMs)) { + return "proxy: no response to the SOCKS5 username and password"; } + if (status[1] != 0) return "proxy rejected the credentials (SOCKS5 status " + std::to_string(status[1]) + ")"; + } else if (choice[1] != 0) { + return "proxy: SOCKS5 proxy chose an authentication method that was not offered"; } - if (statusCode != 200) { - sock.close(); - return sock; + std::string request("\x05\x01\x00", 3); + if (address.empty()) { + request += '\x03'; + request += static_cast(host.size()); + request += host; + } else { + request += address.size() == 4 ? '\x01' : '\x04'; + request.append(reinterpret_cast(address.data()), address.size()); } + request += static_cast(port >> 8); + request += static_cast(port & 0xFF); + if (!write_all(sock, request, timeoutMs)) return "proxy: could not send the SOCKS5 request"; - // Step 4: Read remaining response headers until empty line - while (true) { - std::string headerLine = read_line_plain(sock, timeoutMs); - if (headerLine.empty()) { - break; + unsigned char reply[4]; + if (!read_exact(sock, reply, 4, timeoutMs)) return "proxy: no SOCKS5 reply to the request"; + if (reply[0] != 5) return "proxy: not a SOCKS5 proxy"; + if (reply[1] != 0) { + return "proxy refused CONNECT: " + std::string(socks5_reply_text(reply[1])) + + " (SOCKS5 reply " + std::to_string(reply[1]) + ")"; + } + + // The bound address and port: nothing to use, but they are on the stream + // ahead of the tunnel and have to be read off it. + std::size_t bound = 0; + if (reply[3] == 1) { + bound = 4; + } else if (reply[3] == 4) { + bound = 16; + } else if (reply[3] == 3) { + unsigned char len; + if (!read_exact(sock, &len, 1, timeoutMs)) return "proxy: SOCKS5 reply was cut short"; + bound = len; + } else { + return "proxy: SOCKS5 reply has an unknown address type"; + } + unsigned char rest[258]; + if (!read_exact(sock, rest, bound + 2, timeoutMs)) return "proxy: SOCKS5 reply was cut short"; + return {}; +} + +// A connection to the target through a proxy. `error` is empty exactly when the +// tunnel is open; otherwise it says what the proxy did, in words fit for a +// caller to log. +// +// The tunnel is `socket`, except for an https:// proxy, where it is the TLS +// session to the proxy, `proxyTls`, and `socket` is unused. +export struct ProxyTunnel { + Socket socket; + std::unique_ptr proxyTls; + std::string error; + + [[nodiscard]] bool ok() const { + return error.empty() && (proxyTls ? proxyTls->is_valid() : socket.is_valid()); + } +}; + +// `verifySsl` and `extraCaFile` are for the connection to an https:// proxy, and +// are the same settings that govern the connection to the target. +export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, + std::string_view targetHost, int targetPort, + int timeoutMs, bool verifySsl = true, + std::stop_token stop = {}, + std::string_view extraCaFile = {}) { + ProxyTunnel tunnel; + tunnel.socket.set_stop(stop); + const std::string where = proxy.host + ":" + std::to_string(proxy.port); + + if (proxy.scheme == "http" || proxy.scheme == "socks5" || proxy.scheme == "socks5h") { + if (!tunnel.socket.connect(proxy.host.c_str(), proxy.port, timeoutMs)) { + tunnel.error = "proxy: could not connect to " + where; + return tunnel; } + tunnel.error = proxy.scheme == "http" + ? http_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs) + : socks5_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs); + if (!tunnel.error.empty()) tunnel.socket.close(); + } else if (proxy.scheme == "https") { + auto tls = std::make_unique(); + tls->set_stop(stop); + tls->set_extra_ca_file(std::string(extraCaFile)); + if (!tls->connect(proxy.host.c_str(), proxy.port, timeoutMs, verifySsl)) { + // The session says why when the TCP connection was up, which is + // where a proxy certificate that does not verify is refused. + tunnel.error = "proxy: could not open a TLS connection to " + where; + if (!tls->error().empty()) tunnel.error += ": " + tls->error(); + return tunnel; + } + tunnel.error = http_connect(*tls, proxy, targetHost, targetPort, timeoutMs); + if (tunnel.error.empty()) tunnel.proxyTls = std::move(tls); + } else { + tunnel.error = "proxy: unsupported scheme '" + proxy.scheme + "'"; } + return tunnel; +} - // Socket is now tunneled to the target - return sock; +// The original entry point, kept for callers outside this repository. It cannot +// say why it failed; `proxy_tunnel` can. +export Socket proxy_connect(std::string_view proxyHost, int proxyPort, + std::string_view targetHost, int targetPort, + int timeoutMs) { + ProxyConfig proxy; + proxy.host = std::string(proxyHost); + proxy.port = proxyPort; + auto tunnel = proxy_tunnel(proxy, targetHost, targetPort, timeoutMs); + return std::move(tunnel.socket); } } // namespace mcpplibs::tinyhttps diff --git a/src/socket.cppm b/src/socket.cppm index b480090..de56075 100644 --- a/src/socket.cppm +++ b/src/socket.cppm @@ -1,6 +1,15 @@ module; -#ifdef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK #include #include #pragma comment(lib, "ws2_32.lib") @@ -21,7 +30,7 @@ import :platform; namespace mcpplibs::tinyhttps { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK using SocketHandle = SOCKET; constexpr SocketHandle INVALID_SOCKET_FD = INVALID_SOCKET; #else @@ -43,7 +52,8 @@ public: // Move constructor Socket(Socket&& other) noexcept - : fd_(other.fd_) { + : fd_(other.fd_), stop_(std::move(other.stop_)), deadline_(other.deadline_) + , deadline_hit_(other.deadline_hit_) { other.fd_ = INVALID_SOCKET_FD; } @@ -52,6 +62,9 @@ public: if (this != &other) { close(); fd_ = other.fd_; + stop_ = std::move(other.stop_); + deadline_ = other.deadline_; + deadline_hit_ = other.deadline_hit_; other.fd_ = INVALID_SOCKET_FD; } return *this; @@ -61,6 +74,48 @@ public: return fd_ != INVALID_SOCKET_FD; } + // Once the token is stopped, every wait on this socket (connect included) + // ends within 50 ms and reports "not ready". + void set_stop(std::stop_token stop) { stop_ = std::move(stop); } + + [[nodiscard]] bool stop_possible() const { return stop_.stop_possible(); } + + // Bounds the TLS handshake: until it is cleared, `wait_before_recv` gives up + // once this time has passed. Cleared once the handshake is over. + void set_deadline(std::optional deadline) { + deadline_ = deadline; + deadline_hit_ = false; + } + + // True once `wait_before_recv` gave up because of the deadline. + [[nodiscard]] bool deadline_hit() const { return deadline_hit_; } + + // What `bio_recv` calls ahead of a `recv` that would otherwise block. True + // when a `recv` now will not block. With neither a token nor a deadline it + // answers true without waiting, so the plain `recv` is as it was. + // + // A wait that ends before the deadline is waited again for what is left: a + // signal ends `poll` with EINTR even under SA_RESTART, which the blocking + // `recv` this wait stands in front of would have restarted. The deadline is + // hit only when the clock says so. + bool wait_before_recv() { + if (!stop_.stop_possible() && !deadline_) return true; + for (;;) { + int wait = -1; + if (deadline_) { + const auto left = std::chrono::ceil( + *deadline_ - std::chrono::steady_clock::now()).count(); + wait = left > 0 ? static_cast(left) : 0; + } + if (wait_readable(wait)) return true; + if (stop_.stop_requested() || !deadline_) return false; + if (std::chrono::steady_clock::now() >= *deadline_) { + deadline_hit_ = true; + return false; + } + } + } + bool connect(const char* host, int port, int timeoutMs) { // Close existing connection if any if (is_valid()) { @@ -73,6 +128,7 @@ public: // musl-static build can't — its nameservers live in $PREFIX/etc/resolv.conf // which libc never reads — so fall back to a manual DNS query there. auto try_resolved = [&](const char* node, bool numeric) -> bool { + if (stop_.stop_requested()) return false; struct addrinfo hints{}; hints.ai_family = AF_UNSPEC; hints.ai_socktype = SOCK_STREAM; @@ -88,12 +144,13 @@ public: return ok; }; - if constexpr (platform::is_windows) { + if constexpr (platform::uses_winsock) { return try_resolved(host, /*numeric=*/false); } else { // Fall back to a manual DNS query when libc can't resolve (Termux: // nameservers live in $PREFIX/etc/resolv.conf, which libc ignores). auto try_manual = [&]() -> bool { + if (stop_.stop_requested()) return false; // DNS must be snappy: a UDP query to a working resolver answers // in well under a second. Cap it hard (independent of the much // larger connect timeout) so an intermittently-dropped packet to @@ -119,14 +176,93 @@ public: } } + // The first address `host` resolves to, as network-order bytes: four for + // IPv4, sixteen for IPv6, none when it does not resolve. IPv4 wins when both + // exist, which is what a SOCKS5 proxy of any age can be asked for. + // `numericOnly` accepts an address literal and nothing else, with no lookup. + static std::vector resolve_address(const char* host, bool numericOnly = false) { + struct addrinfo hints{}; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + if (numericOnly) hints.ai_flags = AI_NUMERICHOST; + + std::vector bytes; + struct addrinfo* result = nullptr; + if (::getaddrinfo(host, nullptr, &hints, &result) == 0 && result != nullptr) { + for (auto* rp = result; rp != nullptr; rp = rp->ai_next) { + if (rp->ai_family == AF_INET) { + auto* in = reinterpret_cast( + &reinterpret_cast(rp->ai_addr)->sin_addr); + bytes.assign(in, in + 4); + break; + } + if (rp->ai_family == AF_INET6 && bytes.empty()) { + auto* in = reinterpret_cast( + &reinterpret_cast(rp->ai_addr)->sin6_addr); + bytes.assign(in, in + 16); + } + } + ::freeaddrinfo(result); + } + + // As in `connect`: where libc has no resolver configuration (Termux), + // fall back to a manual query. + if constexpr (!platform::uses_winsock) { + if (bytes.empty() && !numericOnly) { + for (const auto& ip : platform::resolve_fallback(host, 2500)) { + bytes = resolve_address(ip.c_str(), true); + if (!bytes.empty()) break; + } + } + } + return bytes; + } + // Connect to the first reachable address in a resolved list. bool connect_addrinfo(struct addrinfo* result, int timeoutMs) { for (auto* rp = result; rp != nullptr; rp = rp->ai_next) { + if (stop_.stop_requested()) return false; SocketHandle fd = ::socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); if (fd == INVALID_SOCKET_FD) { continue; } + // macOS and the BSDs spell "do not raise a signal" as a socket + // option rather than as a send flag; `write` below carries the flag + // for the platforms that have one. + // + // NOTHING SELECTS THIS AND NOTHING MAY. It is not a feature, not + // a config field and not a runtime probe: the preprocessor reads the + // target's own and the answer is already complete. + // Measured on this machine — glibc: SO_NOSIGPIPE absent, + // MSG_NOSIGNAL 0x4000; musl (Termux, Alpine, openkal-musl): + // SO_NOSIGPIPE absent, MSG_NOSIGNAL 0x4000; Darwin/BSD: the reverse; + // Windows: neither, and no SIGPIPE to raise. + // + // Making it selectable would be actively wrong. A consumer who left + // it off on macOS would get exactly issue #16 — the process killed + // by a signal it never armed — and would get it silently, on a + // platform they may not build for themselves. A property that only + // prevents harm and costs one setsockopt is not a choice worth + // offering; there is no target where the name exists and setting it + // is undesirable. + // + // This does NOT touch the process's signal disposition, so a program + // that wants SIGPIPE on its own stdout still gets it. That is the + // whole reason to prefer this over mbedtls's `signal(SIGPIPE, + // SIG_IGN)`, which changes it for everything the host does. + // + // Best-effort: a failure leaves the socket with the disposition it + // had before this line. + // + // #ifdef, not `if constexpr`: the name does not exist on Linux or + // Windows, and both arms of an `if constexpr` must compile. +#ifdef SO_NOSIGPIPE + int nosigpipe = 1; + ::setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, + reinterpret_cast(&nosigpipe), sizeof(nosigpipe)); +#endif + // Set non-blocking if (!set_non_blocking(fd, true)) { close_handle(fd); @@ -139,13 +275,13 @@ public: if (rc == 0) { connected = true; } else { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK if (WSAGetLastError() == WSAEWOULDBLOCK) { #else if (errno == EINPROGRESS) { #endif // Wait for connection with timeout - if (poll_fd(fd, timeoutMs, false)) { + if (wait_fd(fd, timeoutMs, false)) { int err = 0; socklen_t len = sizeof(err); if (::getsockopt(fd, SOL_SOCKET, SO_ERROR, reinterpret_cast(&err), &len) == 0 && err == 0) { @@ -173,19 +309,49 @@ public: return static_cast(::recv(fd_, buf, len, 0)); } + // A WRITE TO A SOCKET WHOSE PEER HAS GONE AWAY RAISES SIGPIPE, AND A + // PROGRAM THAT HAS NOT DISARMED IT — THE DEFAULT — IS KILLED RATHER THAN + // TOLD. That is issue #16, and it is this library's defect rather than its + // caller's: the fd is one this class created, and the write that meets a + // dead peer is most often the `close_notify` the pool's own clean-up sends. + // + // mbedtls guards against this in `net_prepare` (net_sockets.c:114) with a + // process-wide `signal(SIGPIPE, SIG_IGN)`. This library replaces mbedtls's + // network layer with its own BIO and never calls `mbedtls_net_init`, so it + // dropped that guard without putting anything in its place. MSG_NOSIGNAL is + // the better replacement in any case: a library has no business changing + // its host's signal disposition. + // + // The failure travels along paths that already exist — `bio_send` maps a + // non-positive return to MBEDTLS_ERR_NET_SEND_FAILED, and `TlsSocket::close` + // already ignores what close_notify returns — so the successful path is + // unchanged byte for byte. + // + // #ifdef, not `if constexpr`: the macro does not exist on Windows (which has + // no SIGPIPE either) or on the BSDs (which use SO_NOSIGPIPE, set in + // `connect_addrinfo`), and both arms of an `if constexpr` must compile. That + // is the trap 5e7d66f fixed in the resolver stubs. + // + // Above openkal the flag is accepted and ignored — openkal has no signals at + // all (openkal-musl `port/src/okm_net.c:546-550`) — so this compiles and is + // correct there without a branch of its own. int write(const char* buf, int len) { if (!is_valid()) return -1; +#ifdef MSG_NOSIGNAL + return static_cast(::send(fd_, buf, len, MSG_NOSIGNAL)); +#else return static_cast(::send(fd_, buf, len, 0)); +#endif } bool wait_readable(int timeoutMs) { if (!is_valid()) return false; - return poll_fd(fd_, timeoutMs, true); + return wait_fd(fd_, timeoutMs, true); } bool wait_writable(int timeoutMs) { if (!is_valid()) return false; - return poll_fd(fd_, timeoutMs, false); + return wait_fd(fd_, timeoutMs, false); } [[nodiscard]] SocketHandle native_handle() const { @@ -200,23 +366,46 @@ public: } static void platform_init() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSADATA wsaData; WSAStartup(MAKEWORD(2, 2), &wsaData); #endif } static void platform_cleanup() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSACleanup(); #endif } private: SocketHandle fd_ = INVALID_SOCKET_FD; + std::stop_token stop_; + std::optional deadline_; + bool deadline_hit_ = false; + + // poll_fd in slices while a token is attached. A negative timeout waits + // without limit. Not std::min: defines a `min` macro. + bool wait_fd(SocketHandle fd, int timeoutMs, bool forRead) const { + if (!stop_.stop_possible()) return poll_fd(fd, timeoutMs, forRead); + constexpr int slice = 50; + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(timeoutMs); + while (!stop_.stop_requested()) { + int wait = slice; + if (timeoutMs >= 0) { + const auto left = std::chrono::duration_cast( + deadline - std::chrono::steady_clock::now()).count(); + if (left < slice) wait = left > 0 ? static_cast(left) : 0; + } + if (poll_fd(fd, wait, forRead)) return true; + if (timeoutMs >= 0 && std::chrono::steady_clock::now() >= deadline) return false; + } + return false; + } static bool set_non_blocking(SocketHandle fd, bool nonBlocking) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK u_long mode = nonBlocking ? 1 : 0; return ioctlsocket(fd, FIONBIO, &mode) == 0; #else @@ -232,7 +421,7 @@ private: } static bool poll_fd(SocketHandle fd, int timeoutMs, bool forRead) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSAPOLLFD pfd{}; pfd.fd = fd; pfd.events = forRead ? POLLIN : POLLOUT; @@ -248,7 +437,7 @@ private: } static void close_handle(SocketHandle fd) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK ::closesocket(fd); #else ::close(fd); diff --git a/src/tls.cppm b/src/tls.cppm index ac96e5a..112e633 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -7,6 +7,29 @@ module; #include #include +// mbedTLS draws its entropy from BCryptGenRandom on Windows. Its package asks +// for the library as `-lbcrypt`, which link.exe does not read, so the request is +// made here in the form every linker for this target reads, as socket.cppm does +// for ws2_32. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#pragma comment(lib, "bcrypt.lib") +#endif + +// WHERE THE C LIBRARY IS musl, THE ENTROPY IS THE C LIBRARY'S. +// +// mbedTLS's own platform source asks the kernel through `getrandom` only where +// it recognises glibc (`entropy_poll.c`: `__linux__ && __GLIBC__`); with musl it +// opens `/dev/urandom`. Above openkal that file is not a thing the C library can +// promise — openkal reaches entropy through its own `openkal.random` interface, +// and openkal-musl answers `getrandom` with it — so on x86_64-windows-musl every +// handshake failed before it began, with "CTR_DRBG - The entropy source failed". +// musl's `getentropy` is `getrandom` underneath on every system it runs on, +// which is the source mbedTLS would have chosen had it known. The macro comes +// from mcpp.toml's `cfg(c-abi = "musl")`, because musl defines none of its own. +#ifdef TINYHTTPS_GETENTROPY +#include +#endif + export module mcpplibs.tinyhttps:tls; import :socket; @@ -52,18 +75,83 @@ static int bio_send(void* ctx, const unsigned char* buf, size_t len) { return ret; } +// A ZERO IS PASSED THROUGH, AND THAT IS THE WHOLE CONTRACT. +// +// This used to answer a `recv` of 0 — a peer that sent FIN, which is how nearly +// every server ends a connection-close-delimited response — with +// MBEDTLS_ERR_NET_CONN_RESET. mbedtls's own BIO does not: `mbedtls_net_recv` +// returns `read()`'s result unchanged and reserves CONN_RESET for an actual +// ECONNRESET/EPIPE (net_sockets.c). The distinction is not cosmetic, because +// `mbedtls_ssl_fetch_input` tests for exactly this zero — +// +// if (ret == 0) { return MBEDTLS_ERR_SSL_CONN_EOF; } ssl_msg.c:2251, :2320 +// +// — and passes any negative value straight through. So the old mapping made an +// orderly end of stream indistinguishable from a transport error, one layer +// below `TlsSocket::read_some`, which then had to call it `Error`. A body whose +// framing IS the close then read as truncated: `download_to_file` set +// `result.error` and `ok()` returned false for a file that had arrived +// complete and correct. static int bio_recv(void* ctx, unsigned char* buf, size_t len) { auto* sock = static_cast(ctx); + // With a stop token or a handshake deadline, wait here so the recv below + // cannot block past a stop or the deadline. + if (!sock->wait_before_recv()) { + return MBEDTLS_ERR_NET_RECV_FAILED; + } int ret = sock->read(reinterpret_cast(buf), static_cast(len)); if (ret < 0) { return MBEDTLS_ERR_NET_RECV_FAILED; } - if (ret == 0) { - return MBEDTLS_ERR_NET_CONN_RESET; + return ret; // 0 means end of stream; mbedtls turns it into SSL_CONN_EOF +} + +#ifdef TINYHTTPS_GETENTROPY +// `getentropy` gives at most 256 bytes a call. +static int c_library_entropy(void*, unsigned char* out, size_t len) { + while (len > 0) { + const size_t n = len < 256 ? len : 256; + if (::getentropy(out, n) != 0) return MBEDTLS_ERR_ENTROPY_SOURCE_FAILED; + out += n; + len -= n; } - return ret; + return 0; +} +#endif + +static std::string mbedtls_message(int ret) { + char buf[200]; + mbedtls_strerror(ret, buf, sizeof buf); + return buf; +} + +// verify_info ends every reason with a newline; join them with "; ". +static std::string one_line(std::string text) { + while (!text.empty() && text.back() == '\n') text.pop_back(); + for (std::size_t pos = 0; (pos = text.find('\n', pos)) != std::string::npos;) { + text.replace(pos, 1, "; "); + } + return text; } +// WHAT A READ ENDED IN, WHICH `int` COULD NOT SAY. +// +// `TlsSocket::read` returned 0 for a peer that had closed AND for a transport +// that had no bytes ready yet, so every caller had to guess. They all guessed +// the same way — wait once more and try again — and a reader that cannot tell +// "the body ended here" from "nothing yet" cannot decide whether the connection +// is still reusable. That is root cause R3 behind issue #15. +// +// `Eof` is a fact about the stream and `WouldBlock` is a fact about this +// instant; naming them apart is what lets `read_body` below return `Complete` +// rather than a guess. +export enum class ReadStatus { Data, WouldBlock, Eof, Error }; + +export struct ReadResult { + ReadStatus status { ReadStatus::Error }; + int bytes { 0 }; // meaningful only when status == Data +}; + export class TlsSocket { public: TlsSocket() = default; @@ -76,11 +164,12 @@ public: // Move constructor TlsSocket(TlsSocket&& other) noexcept : socket_(std::move(other.socket_)) - , state_(std::move(other.state_)) { + , lower_(std::move(other.lower_)) + , state_(std::move(other.state_)) + , error_(std::move(other.error_)) + , extraCaFile_(std::move(other.extraCaFile_)) { // Re-bind BIO to point to our socket_ (not the moved-from one) - if (state_) { - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); - } + bind_bio(); } // Move assignment @@ -88,51 +177,114 @@ public: if (this != &other) { close(); socket_ = std::move(other.socket_); + lower_ = std::move(other.lower_); state_ = std::move(other.state_); + error_ = std::move(other.error_); + extraCaFile_ = std::move(other.extraCaFile_); // Re-bind BIO to point to our socket_ - if (state_) { - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); - } + bind_bio(); } return *this; } [[nodiscard]] bool is_valid() const { - return state_ != nullptr && socket_.is_valid(); + return state_ != nullptr && (lower_ ? lower_->is_valid() : socket_.is_valid()); + } + + // Why the last connect_over/connect failed once the TCP connection was up; + // empty if it failed earlier or has not failed. + [[nodiscard]] const std::string& error() const { return error_; } + + // Call before connect(); it covers the handshake and every later wait. + // + // THE TOKEN BELONGS TO THE SOCKET AT THE BOTTOM, AND ONLY THERE. Every wait + // ends in a `Socket` — this session's own, or, inside an https:// proxy's + // tunnel, the one beneath `lower_` — so that is where it is set. Setting + // only `socket_` left a reused tunnel holding the token of the call that + // opened it: a later call could not be cancelled, and once that first + // token was stopped every wait failed at once and the stale-connection + // retry sent a POST that had already arrived a second time. + void set_stop(std::stop_token stop) { + if (lower_) lower_->set_stop(stop); + socket_.set_stop(std::move(stop)); } + // A PEM file of roots to trust in addition to the default store. Call before + // connect(); setup fails if the file cannot be read or holds no certificate. + void set_extra_ca_file(std::string path) { extraCaFile_ = std::move(path); } + // Connect over an already-established Socket (e.g. a proxy tunnel). // Takes ownership of the socket and performs TLS handshake on top of it. - bool connect_over(Socket&& socket, const char* host, bool verifySsl) { + // The handshake gives up after `handshakeTimeoutMs`; a negative value waits + // without limit. + bool connect_over(Socket&& socket, const char* host, bool verifySsl, + int handshakeTimeoutMs = -1) { + error_.clear(); socket_ = std::move(socket); - return setup_tls(host, verifySsl); + return setup_tls(host, verifySsl, handshakeTimeoutMs); + } + + // Run the handshake inside another TLS session, which is how a client + // reaches a target through an https:// proxy: TLS to the proxy, CONNECT + // inside it, then this session to the target inside the tunnel. Takes + // ownership of `lower`, which must already be past its CONNECT. + bool connect_over(std::unique_ptr lower, const char* host, bool verifySsl, + int handshakeTimeoutMs = -1) { + error_.clear(); + lower_ = std::move(lower); + return setup_tls(host, verifySsl, handshakeTimeoutMs); } bool connect(const char* host, int port, int timeoutMs, bool verifySsl) { + error_.clear(); // Step 1: TCP connect via Socket if (!socket_.connect(host, port, timeoutMs)) { return false; } - return setup_tls(host, verifySsl); + return setup_tls(host, verifySsl, timeoutMs); } - int read(char* buf, int len) { - if (!is_valid()) return -1; + // The read that says which of the four things happened. Prefer it over + // `read` below wherever the answer changes what the caller does. + ReadResult read_some(char* buf, int len) { + if (!is_valid()) return { ReadStatus::Error, 0 }; int ret = mbedtls_ssl_read(&state_->ssl, reinterpret_cast(buf), static_cast(len)); - if (ret == MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY || ret == 0) { - return 0; // Connection closed + // Three spellings of "there is nothing more coming", and all three are + // an end of stream rather than a failure: the peer shut the session + // down politely, the transport reached its end (what `bio_recv`'s zero + // becomes), or mbedtls had nothing left to hand back. Most servers do + // NOT send close_notify before closing, so the middle one is the common + // case rather than the exotic one. + if (ret == MBEDTLS_ERR_SSL_PEER_CLOSE_NOTIFY + || ret == MBEDTLS_ERR_SSL_CONN_EOF + || ret == 0) { + return { ReadStatus::Eof, 0 }; } - if (ret < 0) { - if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) { - return 0; // Would block, treat as no data yet - } - return -1; + if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) { + return { ReadStatus::WouldBlock, 0 }; } - return ret; + if (ret < 0) return { ReadStatus::Error, 0 }; + return { ReadStatus::Data, ret }; + } + + // The older shape, kept because it is exported and callers outside this + // repository use it. It collapses Eof and WouldBlock back into 0, which is + // the ambiguity `read_some` exists to remove. + int read(char* buf, int len) { + auto r = read_some(buf, len); + switch (r.status) { + case ReadStatus::Data: return r.bytes; + case ReadStatus::Eof: + case ReadStatus::WouldBlock: return 0; + case ReadStatus::Error: return -1; + } + return -1; } + // Returns 0 for "the transport is not ready", which `write_all` answers by + // waiting on the socket rather than by retrying immediately. int write(const char* buf, int len) { if (!is_valid()) return -1; int ret = mbedtls_ssl_write(&state_->ssl, @@ -151,6 +303,8 @@ public: mbedtls_ssl_close_notify(&state_->ssl); state_.reset(); } + // Only after the close_notify above, which travels through it. + lower_.reset(); socket_.close(); } @@ -159,35 +313,103 @@ public: if (state_ && mbedtls_ssl_get_bytes_avail(&state_->ssl) > 0) { return true; } - return socket_.wait_readable(timeoutMs); + // Under another session the bytes may be decrypted and waiting there + // while the descriptor has nothing, so ask that session, not the fd. + return lower_ ? lower_->wait_readable(timeoutMs) : socket_.wait_readable(timeoutMs); + } + + // TLS back-pressure is a wait, not a failure. `mbedtls_ssl_write` reports + // WANT_WRITE when the record layer cannot flush, and `write` above turns + // that into 0; without something to wait on, `write_all` could only retry + // at once and then give up, which reached the caller as "Write failed". + bool wait_writable(int timeoutMs) { + return lower_ ? lower_->wait_writable(timeoutMs) : socket_.wait_writable(timeoutMs); } private: Socket socket_; + // The session this one runs inside, when there is one; `socket_` is unused + // then. Declared before `state_` so it is destroyed after it: the BIO of + // `state_` points at it. + std::unique_ptr lower_; std::unique_ptr state_; + std::string error_; + std::string extraCaFile_; + + // The session beneath this one, when there is one, is closed as well: a + // failed handshake to the target leaves nothing of the tunnel open. + bool fail(std::string message) { + error_ = std::move(message); + state_.reset(); + drop_transport(); + return false; + } + + // The BIO of a session that runs inside another is that session, which stays + // put when this object moves; the BIO of one on a socket is `socket_`, which + // moves with it. + void bind_bio() { + if (!state_) return; + if (lower_) { + mbedtls_ssl_set_bio(&state_->ssl, lower_.get(), send_over_tls, recv_over_tls, nullptr); + } else { + mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); + } + } - bool setup_tls(const char* host, bool verifySsl) { + static int send_over_tls(void* ctx, const unsigned char* buf, size_t len) { + int ret = static_cast(ctx)->write(reinterpret_cast(buf), + static_cast(len)); + if (ret < 0) return MBEDTLS_ERR_NET_SEND_FAILED; + if (ret == 0) return MBEDTLS_ERR_SSL_WANT_WRITE; + return ret; + } + + // As `bio_recv`: an end of stream is passed on as a zero. + static int recv_over_tls(void* ctx, unsigned char* buf, size_t len) { + auto r = static_cast(ctx)->read_some(reinterpret_cast(buf), + static_cast(len)); + switch (r.status) { + case ReadStatus::Data: return r.bytes; + case ReadStatus::Eof: return 0; + case ReadStatus::WouldBlock: return MBEDTLS_ERR_SSL_WANT_READ; + case ReadStatus::Error: break; + } + return MBEDTLS_ERR_NET_RECV_FAILED; + } + + void drop_transport() { + lower_.reset(); + socket_.close(); + } + + // As `set_stop`, the deadline belongs to the socket at the bottom, which + // inside a tunnel is the one beneath `lower_`. + void set_deadline(std::optional deadline) { + if (lower_) lower_->set_deadline(deadline); + else socket_.set_deadline(deadline); + } + + bool deadline_hit() const { return lower_ ? lower_->deadline_hit() : socket_.deadline_hit(); } + + bool setup_tls(const char* host, bool verifySsl, int handshakeTimeoutMs) { state_ = std::make_unique(); int ret = mbedtls_ctr_drbg_seed( +#ifdef TINYHTTPS_GETENTROPY + &state_->ctr_drbg, c_library_entropy, nullptr, +#else &state_->ctr_drbg, mbedtls_entropy_func, &state_->entropy, +#endif nullptr, 0); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); ret = mbedtls_ssl_config_defaults( &state_->conf, MBEDTLS_SSL_IS_CLIENT, MBEDTLS_SSL_TRANSPORT_STREAM, MBEDTLS_SSL_PRESET_DEFAULT); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); mbedtls_ssl_conf_rng(&state_->conf, mbedtls_ctr_drbg_random, &state_->ctr_drbg); @@ -197,6 +419,10 @@ private: // Load CA certs auto ca_pem = load_ca_certs(); + if (ca_pem.empty() && verifySsl) { + return fail("no CA certificate bundle found; set SSL_CERT_FILE to a PEM " + "file of trusted roots, or set verifySsl to false"); + } if (!ca_pem.empty()) { ret = mbedtls_x509_crt_parse( &state_->ca_cert, @@ -204,48 +430,68 @@ private: ca_pem.size() + 1); // +1 for null terminator required by mbedtls // ret > 0 means some certs failed to parse but others succeeded — acceptable if (ret < 0) { - state_.reset(); - socket_.close(); - return false; + return fail("cannot parse the CA certificate bundle: " + mbedtls_message(ret)); + } + } + // Roots the caller adds to the above. A file that cannot be used is an + // error, not something to skip: the caller named it because a server + // needs it, and skipping would surface later as a verification failure + // that does not mention the file. + if (!extraCaFile_.empty()) { + std::ifstream in(extraCaFile_, std::ios::binary); + if (!in) return fail("cannot read extra CA file '" + extraCaFile_ + "'"); + std::string extra((std::istreambuf_iterator(in)), {}); + ret = mbedtls_x509_crt_parse( + &state_->ca_cert, + reinterpret_cast(extra.c_str()), extra.size() + 1); + if (ret < 0) { + return fail("cannot parse extra CA file '" + extraCaFile_ + "': " + + mbedtls_message(ret)); } + } + if (!ca_pem.empty() || !extraCaFile_.empty()) { mbedtls_ssl_conf_ca_chain(&state_->conf, &state_->ca_cert, nullptr); } - // Certificate verification - // Use OPTIONAL (not REQUIRED) so handshake succeeds even if the CA - // bundle is incomplete; callers that need strict verification can - // inspect the verification result after handshake. - if (verifySsl) { - mbedtls_ssl_conf_authmode(&state_->conf, MBEDTLS_SSL_VERIFY_OPTIONAL); - } else { - mbedtls_ssl_conf_authmode(&state_->conf, MBEDTLS_SSL_VERIFY_NONE); - } + // Certificate verification. REQUIRED makes the handshake fail on a bad + // chain, an expired certificate or a name that is not `host`. + mbedtls_ssl_conf_authmode(&state_->conf, + verifySsl ? MBEDTLS_SSL_VERIFY_REQUIRED : MBEDTLS_SSL_VERIFY_NONE); ret = mbedtls_ssl_setup(&state_->ssl, &state_->conf); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); // Set hostname for SNI ret = mbedtls_ssl_set_hostname(&state_->ssl, host); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; + if (ret != 0) return fail(mbedtls_message(ret)); + + // Set BIO callbacks using our Socket, or the session beneath this one + bind_bio(); + + // Perform TLS handshake. The socket is blocking, so `bio_recv` waits + // against the deadline before it reads, and the deadline is taken off + // again before the connection is used. + std::optional deadline; + if (handshakeTimeoutMs >= 0) { + deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(handshakeTimeoutMs); } - - // Set BIO callbacks using our Socket - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); - - // Perform TLS handshake - while ((ret = mbedtls_ssl_handshake(&state_->ssl)) != 0) { - if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE) { - state_.reset(); - socket_.close(); - return false; + set_deadline(deadline); + while ((ret = mbedtls_ssl_handshake(&state_->ssl)) == MBEDTLS_ERR_SSL_WANT_READ + || ret == MBEDTLS_ERR_SSL_WANT_WRITE) {} + const bool timedOut = deadline_hit(); + set_deadline({}); + if (ret != 0) { + if (ret == MBEDTLS_ERR_X509_CERT_VERIFY_FAILED) { + char info[512] = {}; + mbedtls_x509_crt_verify_info(info, sizeof info, "", + mbedtls_ssl_get_verify_result(&state_->ssl)); + return fail("certificate verification failed: " + one_line(info)); + } + if (ret == MBEDTLS_ERR_NET_RECV_FAILED && timedOut) { + return fail("TLS handshake timed out"); } + return fail("TLS handshake failed: " + mbedtls_message(ret)); } return true; diff --git a/templates/download/mcpp.toml.in b/templates/download/mcpp.toml.in new file mode 100644 index 0000000..8226ca7 --- /dev/null +++ b/templates/download/mcpp.toml.in @@ -0,0 +1,9 @@ +[package] +name = "{{project.name}}" +version = "0.1.0" + +# The library this project was scaffolded from, pinned to the version that +# shipped the template. mcpp injects this line when a template omits it; it is +# written out here so the generated manifest says where the code came from. +[dependencies.{{template.package.namespace}}] +{{template.package.name}} = "{{template.package.version}}" diff --git a/templates/download/src/main.cpp.in b/templates/download/src/main.cpp.in new file mode 100644 index 0000000..3fc25ed --- /dev/null +++ b/templates/download/src/main.cpp.in @@ -0,0 +1,92 @@ +// {{project.name}} — scaffolded from {{template.package.selector}}@{{template.package.version}}:{{template.name}} +// +// A file downloaded to disk rather than into memory, with progress and a way to +// stop. `download_to_file` streams: the body never exists as one buffer, so the +// size of the file is not the size of the process. + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +std::string human(std::int64_t bytes) { + constexpr const char* units[] = { "B", "KiB", "MiB", "GiB", "TiB" }; + double value = static_cast(bytes); + std::size_t unit = 0; + while (value >= 1024.0 && unit + 1 < std::size(units)) { + value /= 1024.0; + ++unit; + } + return std::format("{:.1f} {}", value, units[unit]); +} + +void draw_progress(std::int64_t total, std::int64_t done) { + // A server need not declare a length: chunked and connection-close framings + // both report total == 0, and a bar with no end is a lie rather than a bar. + if (total <= 0) { + std::print(std::cout, "\r {} downloaded", human(done)); + std::cout << std::flush; + return; + } + constexpr int width = 32; + const int filled = static_cast(width * done / total); + // `std::print(std::cout, ...)` rather than `std::print(...)`: the second + // writes to the C stream `stdout`, and flushing that needs the `stdout` + // macro from , which `import std` does not carry. A progress bar + // that is not flushed does not appear until the download finishes. + std::print(std::cout, "\r [{}{}] {:>3}% {} / {}", + std::string(filled, '#'), std::string(width - filled, '.'), + 100 * done / total, human(done), human(total)); + std::cout << std::flush; +} + +} // namespace + +int main(int argc, char** argv) { + https::Socket::platform_init(); + + const std::string url = argc > 1 ? argv[1] : "https://httpbin.org/bytes/65536"; + const std::filesystem::path destination = + argc > 2 ? std::filesystem::path(argv[2]) : std::filesystem::path("out/download.bin"); + + https::HttpClientConfig config; + config.connectTimeoutMs = 15000; + config.readTimeoutMs = 30000; + https::HttpClient client(config); + + std::println("GET {}\n -> {}", url, destination.string()); + + // Set this from a signal handler, another thread, or a UI button. It is + // checked between blocks, so a cancelled download stops promptly and the + // connection is dropped rather than reused half-read. + std::atomic cancelled { false }; + + const auto started = std::chrono::steady_clock::now(); + auto result = client.download_to_file( + url, destination, + [](std::int64_t total, std::int64_t done) { draw_progress(total, done); }, + [&] { return cancelled.load(); }); + const auto elapsed = std::chrono::duration_cast( + std::chrono::steady_clock::now() - started); + std::println(""); + + if (!result.ok()) { + // `error` distinguishes a refusal ("HTTP 404") from a transfer that + // broke, and `bytesWritten` says how far it got either way. + std::println(std::cerr, "failed after {}: {}", + human(result.bytesWritten), result.error); + return 1; + } + + std::println("{} in {} ms", human(result.bytesWritten), elapsed.count()); + + // Kept so a later run can ask the server whether the file changed, with + // `If-None-Match` / `If-Modified-Since`. + if (!result.etag.empty()) std::println(" ETag: {}", result.etag); + if (!result.lastModified.empty()) std::println(" Last-Modified: {}", result.lastModified); + if (result.expectedBytes) std::println(" declared length: {}", *result.expectedBytes); + + return 0; +} diff --git a/templates/download/template.toml b/templates/download/template.toml new file mode 100644 index 0000000..2043362 --- /dev/null +++ b/templates/download/template.toml @@ -0,0 +1,3 @@ +[template] +description = "Download a file to disk with a progress bar, resumable-friendly metadata and cancellation" +post_message = "cd into the project, then `mcpp run -- [destination]`." diff --git a/templates/fetch/mcpp.toml.in b/templates/fetch/mcpp.toml.in new file mode 100644 index 0000000..8226ca7 --- /dev/null +++ b/templates/fetch/mcpp.toml.in @@ -0,0 +1,9 @@ +[package] +name = "{{project.name}}" +version = "0.1.0" + +# The library this project was scaffolded from, pinned to the version that +# shipped the template. mcpp injects this line when a template omits it; it is +# written out here so the generated manifest says where the code came from. +[dependencies.{{template.package.namespace}}] +{{template.package.name}} = "{{template.package.version}}" diff --git a/templates/fetch/src/main.cpp.in b/templates/fetch/src/main.cpp.in new file mode 100644 index 0000000..c2cd2ae --- /dev/null +++ b/templates/fetch/src/main.cpp.in @@ -0,0 +1,66 @@ +// {{project.name}} — scaffolded from {{template.package.selector}}@{{template.package.version}}:{{template.name}} +// +// One request, and the three separate questions worth asking about the answer. + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +int main(int argc, char** argv) { + // Required once per process. A no-op everywhere but Windows, where it is + // WSAStartup. + https::Socket::platform_init(); + + const std::string url = argc > 1 ? argv[1] : "https://httpbin.org/get"; + + // Defaults: 10s connect, 60s read, certificates verified, keep-alive on, + // up to 10 redirects. See HttpClientConfig for the rest. + https::HttpClient client; + + https::HttpRequest request; + request.method = https::Method::GET; + request.url = url; + request.headers.emplace("Accept", "application/json"); + + auto response = client.send(request); + + // ── 1. Did the request reach a server at all? ─────────────────────────── + // statusCode 0 means nothing was received: DNS, TCP, TLS or a write that + // failed. statusText says which. + if (response.statusCode == 0) { + std::println(std::cerr, "{}: {}", url, response.statusText); + return 1; + } + + // ── 2. What did the server say? ───────────────────────────────────────── + std::println("{} {}", response.statusCode, response.statusText); + for (const auto& [name, value] : response.headers) { + std::println(" {}: {}", name, value); + } + + // ── 3. Did all of the body arrive? ────────────────────────────────────── + // A read that timed out, a connection that ended mid-body or a chunk header + // that did not parse all leave a PREFIX of the body behind an ordinary + // status code. ok() answers question 2 and deliberately not this one. + if (!response.bodyComplete) { + std::println(std::cerr, "warning: body is incomplete — {}", response.bodyError); + } + + std::println("\n{}", response.body); + return response.ok() && response.bodyComplete ? 0 : 1; +} + +// A POST with a JSON body is one line: +// +// auto response = client.send(https::HttpRequest::post( +// "https://api.example.com/v1/things", R"({"name": "thing"})")); +// +// and an API key goes in the headers: +// +// https::HttpRequest request; +// request.method = https::Method::POST; +// request.url = "https://api.example.com/v1/things"; +// request.body = R"({"name": "thing"})"; +// request.headers.emplace("Content-Type", "application/json"); +// request.headers.emplace("Authorization", "Bearer " + key); diff --git a/templates/fetch/template.toml b/templates/fetch/template.toml new file mode 100644 index 0000000..852da9f --- /dev/null +++ b/templates/fetch/template.toml @@ -0,0 +1,4 @@ +[template] +description = "One HTTPS request, and how to tell a complete answer from a truncated one" +default = true +post_message = "cd into the project, then `mcpp run` — or `mcpp run -- https://your.api/endpoint`." diff --git a/templates/stream/mcpp.toml.in b/templates/stream/mcpp.toml.in new file mode 100644 index 0000000..8226ca7 --- /dev/null +++ b/templates/stream/mcpp.toml.in @@ -0,0 +1,9 @@ +[package] +name = "{{project.name}}" +version = "0.1.0" + +# The library this project was scaffolded from, pinned to the version that +# shipped the template. mcpp injects this line when a template omits it; it is +# written out here so the generated manifest says where the code came from. +[dependencies.{{template.package.namespace}}] +{{template.package.name}} = "{{template.package.version}}" diff --git a/templates/stream/src/main.cpp.in b/templates/stream/src/main.cpp.in new file mode 100644 index 0000000..b91874b --- /dev/null +++ b/templates/stream/src/main.cpp.in @@ -0,0 +1,128 @@ +// {{project.name}} — scaffolded from {{template.package.selector}}@{{template.package.version}}:{{template.name}} +// +// Server-Sent Events. `send_stream` hands each event to a callback as it +// arrives rather than returning a body at the end, which is what makes a +// token-by-token completion possible. +// +// The endpoint here is the OpenAI chat-completions shape, which is what almost +// every SSE-speaking API in this space implements; point OPENAI_BASE_URL at any +// compatible service. The mechanism is not specific to it — any `text/event-stream` +// endpoint works the same way. + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +// The one field this example needs out of each event's JSON payload. A real +// program uses a JSON library; extracting one string does not justify one. +std::optional content_delta(std::string_view json) { + constexpr std::string_view key = "\"content\":"; + auto at = json.find(key); + if (at == std::string_view::npos) return std::nullopt; + at = json.find('"', at + key.size()); + if (at == std::string_view::npos) return std::nullopt; + + std::string out; + for (std::size_t i = at + 1; i < json.size(); ++i) { + if (json[i] == '\\' && i + 1 < json.size()) { + switch (json[++i]) { + case 'n': out.push_back('\n'); break; + case 't': out.push_back('\t'); break; + case 'r': break; + case 'u': i += 4; break; // left as-is by this sketch + default: out.push_back(json[i]); + } + continue; + } + if (json[i] == '"') return out; + out.push_back(json[i]); + } + return std::nullopt; +} + +std::string env_or(const char* name, std::string fallback) { + const char* value = std::getenv(name); + return (value != nullptr && *value != '\0') ? std::string(value) : std::move(fallback); +} + +} // namespace + +int main(int argc, char** argv) { + https::Socket::platform_init(); + + const char* apiKey = std::getenv("OPENAI_API_KEY"); + if (apiKey == nullptr || *apiKey == '\0') { + std::println(std::cerr, + "OPENAI_API_KEY is not set.\n" + " export OPENAI_API_KEY=sk-...\n" + " export OPENAI_BASE_URL=https://api.openai.com # or any compatible service\n" + " mcpp run -- \"your question\""); + return 0; // not a failure: nothing was configured yet + } + + const std::string baseUrl = env_or("OPENAI_BASE_URL", "https://api.openai.com"); + const std::string model = env_or("OPENAI_MODEL", "gpt-4o-mini"); + const std::string prompt = argc > 1 ? argv[1] : "Say hello in one sentence."; + + // A streaming request is a long-lived one: the read timeout bounds the gap + // BETWEEN events, not the whole exchange, so it can stay modest. + https::HttpClientConfig config; + config.connectTimeoutMs = 15000; + config.readTimeoutMs = 60000; + https::HttpClient client(config); + + https::HttpRequest request; + request.method = https::Method::POST; + request.url = baseUrl + "/v1/chat/completions"; + request.headers.emplace("Content-Type", "application/json"); + request.headers.emplace("Authorization", std::string("Bearer ") + apiKey); + request.headers.emplace("Accept", "text/event-stream"); + // Concatenated rather than std::format'd. A format string emitting JSON has + // to double its braces, and a doubled opening brace is exactly how mcpp's + // template renderer opens a placeholder — so the format-string version of + // this line cannot survive being scaffolded. Concatenation has no such + // character; std::format is fine everywhere in the generated project. + request.body = "{\"model\":\"" + model + "\",\"stream\":true," + "\"messages\":[{\"role\":\"user\",\"content\":\"" + prompt + "\"}]}"; + + std::size_t events = 0; + auto response = client.send_stream(request, [&](const https::SseEvent& event) { + ++events; + // The sentinel that ends an OpenAI-shaped stream. Returning false stops + // the read; the connection is then dropped rather than reused, because + // bytes are still owed on it. + if (event.data == "[DONE]") return false; + if (auto delta = content_delta(event.data)) { + // Flushed through the C++ stream: `import std` carries no `stdout` + // macro, so a token printed without this would not appear until the + // stream ended — which defeats the point of streaming. + std::print(std::cout, "{}", *delta); + std::cout << std::flush; + } + return true; + }); + std::println(""); + + if (response.statusCode == 0) { + std::println(std::cerr, "request failed: {}", response.statusText); + return 1; + } + if (!response.ok()) { + // A failed streaming request is answered with an error document, not an + // event stream — so the parser yields nothing and the bytes land in + // `body` instead. Without them there is a status code and no reason. + std::println(std::cerr, "{} {}\n{}", + response.statusCode, response.statusText, response.body); + return 1; + } + + // `bodyComplete` is false when the stream ended before its framing said it + // would — and also when the callback above stopped it, which is the ordinary + // way an SSE exchange ends. `bodyError` distinguishes the two. + std::println(std::cerr, "\n[{} events, {}]", events, + response.bodyComplete ? "stream ended cleanly" : response.bodyError); + return 0; +} diff --git a/templates/stream/template.toml b/templates/stream/template.toml new file mode 100644 index 0000000..e7e306c --- /dev/null +++ b/templates/stream/template.toml @@ -0,0 +1,3 @@ +[template] +description = "Server-Sent Events: a streaming LLM chat completion, token by token" +post_message = "cd into the project, export OPENAI_API_KEY (and OPENAI_BASE_URL for an OpenAI-compatible service), then `mcpp run`." diff --git a/tests/extra_ca_certs.hpp b/tests/extra_ca_certs.hpp new file mode 100644 index 0000000..bfd8229 --- /dev/null +++ b/tests/extra_ca_certs.hpp @@ -0,0 +1,161 @@ +// Two private CAs, each with one server certificate (CN=localhost, SAN localhost +// and 127.0.0.1), for test_extra_ca.cpp. Valid for a century. The keys are here +// on purpose: they authenticate nothing and must never be used for anything. +// +// Made with openssl: a self-signed CA with basicConstraints CA:TRUE and +// keyUsage keyCertSign, and a leaf signed by it whose extensions are +// basicConstraints CA:FALSE, extendedKeyUsage serverAuth and +// subjectAltName DNS:localhost,IP:127.0.0.1. +#pragma once + +namespace extra_ca_test { + +inline constexpr const char* kCa1Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDLzCCAhegAwIBAgIUG2Zap6LiJwyHlkOt1Ko7G28l+DcwDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMTAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3Qg +Q0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALCFaWIMBSe8OolK +5ExxlX3LbNgg3XlOLxa604zHI0nLWFhKJalkticwwMaX4gcJQdETCXADq+V8qj48 +014c3uUC7xlb+INULhe2ocThipOSwhh4E1Vyv8ywNhgVEw8r+Lgl2ExaSIGzcx4O +2tWuOvzQHhe6JG8G5YRWVk5ydimhFGY6VdGzUPfHAID+3JEqr6RUeCct4Ehx2zbg +mkn7ZBu0PHtP2fjrc5KnVwXWDH91MbSfJ1WQvPq5kPbSpeeLoWvDznzpUjg+9aTB +Dbyxu+/5YVFy5jWWEpgOKakyKnfsQ6XTqzz6dvogAjjh3UWTHe01osluXqY8tjls +gapw+N8CAwEAAaNjMGEwHQYDVR0OBBYEFP3PSVqEY/jqpWJmAzftRY3zFZPtMB8G +A1UdIwQYMBaAFP3PSVqEY/jqpWJmAzftRY3zFZPtMA8GA1UdEwEB/wQFMAMBAf8w +DgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQAhMIwH6LSKiywbxaFK +RlUIqgThCIP9uUOZ0m/6jjgMwSJhU0IZ4COCewFVPEGepeqhNjDXHD9Y0EHbxDi5 +cCsk2fSz9xfsjBniI+xETlxGLoCrNDAmNEfPv8mT9UnNKG7KqLghQiHpMbx47895 +wTbr1OldFAlU8usR9pOigRc6j5cmf9b1Tu/piUTFsy5xoANPt/HRg3o/w0mfR8c5 +bHQ3LaqcQO+uBpKAy9guHpszKosOmbTX6AB5JBSyo70u3jx0PHgQuOJTEeKP1PBJ +51EyNShztp7bn+Rz0OLsMk2h6AMns7HB2m5gEIq/ELlaWDYruqiYFmGpqcm7deVd +v6DH +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf1Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDTzCCAjegAwIBAgIUO+WtedkpuiIwKSiJ20nYcEG4pNQwDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMTAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAj3gXKIHuAZ2h0DeMqxZX2ImgZJan +arOOw8vKg8yqyfgAeEtenmH42Hv4DAPhHVycq9DyXutiXVxQ+v1y7ABfG9RG2Z0u +Ueen2TqNDz5F6CrlUeG/Fgk3OVOWVeglcJpd8vqiVzrgRRtc4+uREGlu2YPusgSX +rbVL+6VARdEfypluaStOt1kJkQlLUxa5A7OpsXKIDLRnOKSHCyx1CCJNy3jwPmqz +vng6M/FYns/73TxqSZ8Wi7CRdCpNWjlwfJ0zunjeYdrGUchgVpbyWedexZzKwLMO +NFnjY20qPmN0ZIcXSg+IfZJcUKSD6t+kWZAOtcFwrO6xi7SWFQwNyskW3wIDAQAB +o4GMMIGJMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF +BwMBMBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATAdBgNVHQ4EFgQUNtDg1vH4 +jNeYzFRvdHWYhhdO6WEwHwYDVR0jBBgwFoAU/c9JWoRj+OqlYmYDN+1FjfMVk+0w +DQYJKoZIhvcNAQELBQADggEBAEUwKddb7KxhgZ5a41u78wRANsfkXHJEXX6l2IWL +cOAE9PU2LFj4muABzp7g6WA5rOF5Dt34E3PSB8EQIMTY6SBA9V6Y80/8uBlsF+Er +Labg6ABrvswpGPn7sc65m8uDUYdG5jm0P2lure9mV8kdiYURk0/eOIpdCp8CcuFx +iG1LsAMz9h0WAgbkfIzigDGx9TkvqTZNSfzvvObgGmeRfo++pLMEjifug9NBRHuN +dMiQ4NBvRv+HvJHGSAD1fpm8QvEiMk7spf+PyZbJZP34YZlbJfZJXvBxnTnJY6rD +amz7EBggI4kRiDGG5vxgyH2bYe9uzw01jb02QJyUvJqvOv4= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf1KeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCPeBcoge4BnaHQ +N4yrFlfYiaBklqdqs47Dy8qDzKrJ+AB4S16eYfjYe/gMA+EdXJyr0PJe62JdXFD6 +/XLsAF8b1EbZnS5R56fZOo0PPkXoKuVR4b8WCTc5U5ZV6CVwml3y+qJXOuBFG1zj +65EQaW7Zg+6yBJettUv7pUBF0R/KmW5pK063WQmRCUtTFrkDs6mxcogMtGc4pIcL +LHUIIk3LePA+arO+eDoz8Viez/vdPGpJnxaLsJF0Kk1aOXB8nTO6eN5h2sZRyGBW +lvJZ517FnMrAsw40WeNjbSo+Y3RkhxdKD4h9klxQpIPq36RZkA61wXCs7rGLtJYV +DA3KyRbfAgMBAAECggEAI/sGM86CFj2mO0XzLWUpGtFJPHDcA9KMV2Zgn7VQj952 +n7SnQiln80MD7l96sZZsZFq+rjdyL9/bH1c43x8oux39ts3ItBnY6o3Ymp4Pgi5f +3K008x3Uj7f62Aikexz7uthIpn/JhwtZvH9znh4TAtInptWoBJSXPcEv4VKVGoYI +HglTL1/TvIkyo2hwdLBqOuUF5PioHktG8MZH2ILqFJTfB5bWmOPXDMAEuZD9pkQK +LfyD9AntTRKVxRO/sXjHy27IXSKbDoZq4kqqLkxk0BGJpf4BnxxjZUWoqbQ7MZHs +MS26/fcpMWSJygYT8vMkAQYrGjrEkKQSvABFTmkegQKBgQDASz/4vpa+qPf9soX8 +n9+XXSPBeyNeNKpl2p0b+7h8skRIdi/gmttSiC9g/L2fuI2rpoikii1qxQ1XN+wA +LOcOrL5p/RJdGKAyu8n8cRSGFkhkOO5s2zLsSdXUHesVqWc8keI37DoJkySYEsiC +WI0MZNNK01cUFttD+nEqxgCicQKBgQC+/+3u+aWF1rumheoUV9pRk65F0jJRnTAQ +bfiRE7jWhs/LSYpJ1EOgc0OYaRnJPwKYuck4/Z7K3beNKPH3jPJ+89UvfRjHKE/3 +o89yACWZATh2XMjvZUUUmvn+Om+yMBxWhy1vyTJU+XhFVSIROofqg5xH3F3hMC6P +WIyJctVWTwKBgDUdE+BrFEw5c7Y5d1Td++5dJaEtmAlPVrmndrnh/4VB4CZ7rqHF +ZEsZnqyVYvMZENiWuStplz3ki9jJc4Bg5rlg8X9pDYd1Y9pfkF0QBvE6emhkaUbc +DObDRjK+yM5E6mUKjoeP0kAOyo7OsV9/ggYoW0xRr722yn2uUU/VPRuhAoGBALu/ +TX5cTdWa98g/2JH4rbUEqwkdDrNlBQsRXoYVUdsml3mnp1TdX0CILQjTPzKwSngg +H+lyfLXDbhtHBVqmZPRf8M4GdOXH/ZtanZ7dABg/t+W5XRbdgCM6F8VMYeFRI3n2 +1LYOmvgmZZAZmCxBUbs3dda5ilrCBvKHQ1YU2BpHAoGAGZVbkJ0lnCoXf3tyMBCZ +MliDj7Q6fSGd22ZAoYy/XGLmZUMi/ZK+PkdzVwT6CDwJY07vBDwUIj5qBHlmaHGS +qC7o3bmhdtC4R+eN4Gb6mdKB3wCdrRhx2HTBGDf4aoPwGtUw+kUdlve1L+Rwsdfc +zCIifQbNtFiGohWrskdamio= +-----END PRIVATE KEY----- +)PEM"; + +inline constexpr const char* kCa2Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDLzCCAhegAwIBAgIUe6yBH1jY6Ogs0HBiaUEsnbZlB54wDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMjAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3Qg +Q0EgMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJXfKHI6jLsG/yLn +UqcDb+PBuw4+C6eaYUwgJNJ5VNubXZpjg1JJxqBqOYFKJ69J5OULqBD2A3DWCAKh +vh6FOHeSdXlvwnZQN0fLwtnYcmL90XJaDrUwaFn/NrceqsgBo+pNXHcVb6OYwxQr +Np9/3kta6vbWp9cC8D6vCkmp2ehrz2UfBo4m6oh1Re2ATEBn0H2OD41LKyQslpF4 +CFkm2rIdkTsqlGtdoBkE+JyBX+eoZAlnRBRC5NxJDLEtDlxoPlbVRP/YgPeazv1g +tMgUvBhmC5AROU6oEpiXiGxas/3m3av/21shsq0iz+b+1QOF5gt9MdOEm2mxrxfF +hkf0pH0CAwEAAaNjMGEwHQYDVR0OBBYEFMV3Z2Qg+uzCT8vmKJV84SBhwzQQMB8G +A1UdIwQYMBaAFMV3Z2Qg+uzCT8vmKJV84SBhwzQQMA8GA1UdEwEB/wQFMAMBAf8w +DgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQBmve5C71LtGh24Sq3Q +kNsEROTQYwjbI1W1tEC5C/bqeUFLl4YxyzIHyGFiNSZ7aaaOSRrMEY2aCG2f7am0 +Rckygp6fFaTTAfWnsgH7fVxvBEwywNlLDaAU/co20cnFa1MfUO1MfstF56lqllgX +NHE/PAOCV/ZcO3DAmJ1lWbsJcXQKplbNlZIeEBwqJs2i3xpiJHxhckg95wyq2vnX +qtCBiGlK7LBawlHLAbYEXf4LTDAJvb5OFfb8vIJN5pn7PKvMhhMtp4i0r0nDgLkR +nP25ABxH8uw8KV7rCzp6i0LDSMRAU6pbs/C8kvMSv8BmF2FFK4OX4+29aIPH/utP +x/4p +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf2Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDTzCCAjegAwIBAgIUBQEev9nx9U9d1N63ZONvDn/2u9owDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMjAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4piCGItiClSpVhm3Rw2h/OTfgyE4 +vrEXqkEcHB7Oq4z23KeluS9r6eM/6i0ezRUTCI71Q6UBQ/n7mQUPb5ePHZygVqL+ +QEYGiO6xH2q3kB3oDzNixaPYUCg0u7WLCbv4E6E8ngqcpMSBTxTH/Be3AB1j0I9a +AH3xmVbY1E2V79uYhRVc7cQ0mNfXkj1K4G0AaPKuiwwVKnE7/z/BwA4vqWjH9jsV +s1JFhjPpydBY73VWEVE0Ou1ntBNoKGilsVJBs4lf4P8pu9hBS8H0PCAk784kVY8n +0Y1G2xU3fSash6HjljjKtWg+mudbu2edVG8pkv13JYxJNNIUCXLvRH6m4QIDAQAB +o4GMMIGJMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF +BwMBMBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATAdBgNVHQ4EFgQUSfHOwhMo +5rrgK8aDsQ4K7lvGJ5IwHwYDVR0jBBgwFoAUxXdnZCD67MJPy+YolXzhIGHDNBAw +DQYJKoZIhvcNAQELBQADggEBAIQ2kQBTG7wSFAa/fGup6cph+pV2ZVikoQUN0C9U +29uJaGALXNhx5RUDduPZP0i1Q9chYh55HcCT/CtbVlkO3X3BwLmgvqJel0O0nDnR +bIA90Ccs0wOYBY1bkWqcjqALJlddjeeerYdV3PulAp5uklf5ZWxgxl6XSucRW1fH +wvtBbQ7TirbYos0nZqAF4pxpMq4KLfWa7eAVMThL6ImiQyOYZsblMxZqAOPh8RIU +lmZG8/79aHwWyQdCC7p91dD5bCtOwrn1VZvTgmrzp8+TQnzqGuKAlUZEJXp/OF+a +qana9iMkkKQyE7y+DGZz09xKrvlh8l7kxJMES6ynN/XjITg= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf2KeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDimIIYi2IKVKlW +GbdHDaH85N+DITi+sReqQRwcHs6rjPbcp6W5L2vp4z/qLR7NFRMIjvVDpQFD+fuZ +BQ9vl48dnKBWov5ARgaI7rEfareQHegPM2LFo9hQKDS7tYsJu/gToTyeCpykxIFP +FMf8F7cAHWPQj1oAffGZVtjUTZXv25iFFVztxDSY19eSPUrgbQBo8q6LDBUqcTv/ +P8HADi+paMf2OxWzUkWGM+nJ0FjvdVYRUTQ67We0E2goaKWxUkGziV/g/ym72EFL +wfQ8ICTvziRVjyfRjUbbFTd9JqyHoeOWOMq1aD6a51u7Z51UbymS/XcljEk00hQJ +cu9EfqbhAgMBAAECggEAFmUu1HiwKae47cjQX64mAK3pNldZ+aTcmyfgvJnB4Nwn +iOLr4uvgLUdXyZAIj4nh7fJdJMCh6aNDHdpNboDK9QkGItHssZMKMc3vhRShzFVc +WmZAgJqjNjbnEVdNxNpGWEBTEqanWTSPuC5sXorEfDcrZpFIeqx37KkT+JWreD9r +DEsHVL51wGwD7KC06qv/IUbQXrfUtwJouof0X3iF/1bSmBTi3gVvyRpBaECRAZab +bOktlfKP/9FquA7KegY0HaaByXC073lj7pl3H9AmnYihXfTHSnxRMLq31IhpWzIN +Bbx/+vaVWgt/lWdedAVg8wVqlen4PSTstKYpq9WrHQKBgQD0SeekTgwjHfixaGm8 +7BoYHrszO32ua7BPAi8zXjbJi2SwFqyLlsw52FaYMMFxWRCarhtOKzUR0AnfB+Tp +HUsNlg5IOzeBXz26Care6Ymmx0jc1ZTrNMNoSTRMhZ4Cq16pD/mXSFNRQaBxP1Uy +ZTKzbz/UMZ6dbw+wY0lRp5gABQKBgQDtdXZHyPWns5OY34IYkO/cX3FY7K6BqqQz +8ciULN1v8bK32J8NXuuHRpbl+gJQ9ZwSzTyUBEW2EJArSmiVQBWLfITOv+Ubb34a +7RmiMO7ectqDsydpXyNr5GVvmJSN7ua/Bgse71ZEER+5T+Wk1IFrmc+dgWD7p1wX +DPoRGVruLQKBgGxA++extmsxcu2bKZBQyFd79tUDPfS3QBAjFaubyoeRv/LQeXH6 +R1Iy7lzO7ko+f78r/Gmtd/0GxfkR1H0BQCsrHfUgWL556C6y7geOUjOYCE/kTfPV +E+r41bY7WAQy0OX2cFJ7+H//pwvIVwbYFZmyaQStYG6/sqIlvW3z6MXJAoGAFNv1 +qPapRbMLqykvJ3NL7sLflR0tcyyHQhziLsoGJbhIp8e6qRUGgZqc1NTycSQyMqCe +yq3ZZdXspDDkIQBBJECq9fJ7cL6JiFn9n3bG9PmH4nvRYXs/BFJK6tOvcQxkbrFc +Nrp1jIMok0HGfYVBXS9kwoRD/bdE6q10Jh5hE30CgYEA1cUVunt7NYrY2NV+jF+1 +kp9Imom/F0ao4wowMmrcJTm1zgw4BehYaRoE7BVOxoq4Rc7SBgSlvb8aoy7EXuu7 +eieh8GE4WeiFGcDtCdpS82s/3IOeHwHo9I7wFa16r1jy/ooygVPM3RB8q7KOmfP+ +Hmwo4XQ7djlELruiEfglIt4= +-----END PRIVATE KEY----- +)PEM"; + +} // namespace extra_ca_test diff --git a/tests/proxy_test_server.hpp b/tests/proxy_test_server.hpp new file mode 100644 index 0000000..0d3d32d --- /dev/null +++ b/tests/proxy_test_server.hpp @@ -0,0 +1,242 @@ +// A proxy, in this process, that does exactly what a test tells it to. +// +// The proxy tests need the three things a real proxy does between accepting a +// connection and relaying bytes: read what the client asked for, answer it, and +// then splice the client to the target. A handler scripts the first two and +// `Peer::relay_to` does the third, so one test can make the proxy refuse, stall +// or hang up, and another can let a request through to the TLS server from +// tls_test_server.hpp and check what arrived. +// +// Plain TCP on 127.0.0.1, port chosen by the OS. It uses mbedtls's net layer +// for the same reason tls_test_server.hpp does: it is already a dependency and +// it is the same on every platform the library builds for. +#pragma once + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// Which socket interface this is built against; see socket.cppm. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK +#include +#include +#else +#include +#include +#endif + +namespace proxy_test { + +class Server; + +// The accepted connection, as the handler sees it. +class Peer { +public: + Peer(Server& server, mbedtls_net_context net) : server_(server), net_(net) {} + ~Peer() { mbedtls_net_free(&net_); } + Peer(const Peer&) = delete; + Peer& operator=(const Peer&) = delete; + + // Everything up to and including the blank line that ends an HTTP head, + // read a byte at a time so nothing past it is taken. Empty on a hang-up. + std::string read_head(); + + // Exactly `n` bytes, or empty when the client hangs up first. + std::string read_exact(std::size_t n); + + // Whatever arrives first, up to 4 KiB. Empty on a hang-up. + std::string read_some(); + + bool write(std::string_view data) { + std::size_t sent = 0; + while (sent < data.size()) { + int ret = mbedtls_net_send(&net_, + reinterpret_cast(data.data() + sent), + data.size() - sent); + if (ret <= 0) return false; + sent += static_cast(ret); + } + return true; + } + + void note(std::string text); + + // Splices this connection to 127.0.0.1:`port` until either side ends. + void relay_to(int port); + +private: + bool wait_readable(int ms) { return mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, ms) > 0; } + + Server& server_; + mbedtls_net_context net_; +}; + +class Server { +public: + using Handler = std::function; + + explicit Server(Handler handler) : handler_(std::move(handler)) { + mbedtls_net_init(&listener_); + if (mbedtls_net_bind(&listener_, "127.0.0.1", "0", MBEDTLS_NET_PROTO_TCP) != 0) { + failed_ = true; + return; + } + sockaddr_in addr {}; + socklen_t len = sizeof addr; + if (::getsockname(listener_.fd, reinterpret_cast(&addr), &len) != 0) { + failed_ = true; + return; + } + port_ = ntohs(addr.sin_port); + acceptor_ = std::thread([this] { accept_loop(); }); + } + + ~Server() { stop(); } + Server(const Server&) = delete; + Server& operator=(const Server&) = delete; + + void stop() { + if (stopping_.exchange(true)) return; + if (acceptor_.joinable()) acceptor_.join(); + for (auto& t : connections_) { + if (t.joinable()) t.join(); + } + connections_.clear(); + mbedtls_net_free(&listener_); + } + + [[nodiscard]] bool failed() const { return failed_; } + [[nodiscard]] int port() const { return port_; } + [[nodiscard]] bool stopping() const { return stopping_.load(); } + [[nodiscard]] int accepts() const { return accepts_.load(); } + + // The text a handler stored with `note`, for the test to assert on once the + // request has completed. + void note(std::string text) { + std::lock_guard lock(mutex_); + notes_.push_back(std::move(text)); + } + [[nodiscard]] std::vector notes() { + std::lock_guard lock(mutex_); + return notes_; + } + +private: + void accept_loop() { + while (!stopping_.load()) { + int ready = mbedtls_net_poll(&listener_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) break; + if (ready == 0) continue; + + mbedtls_net_context client {}; + mbedtls_net_init(&client); + if (mbedtls_net_accept(&listener_, &client, nullptr, 0, nullptr) != 0) { + mbedtls_net_free(&client); + continue; + } + accepts_.fetch_add(1); + connections_.emplace_back([this, client] { + Peer peer(*this, client); + handler_(peer); + }); + } + } + + Handler handler_; + mbedtls_net_context listener_ {}; + int port_ { 0 }; + bool failed_ { false }; + std::atomic stopping_ { false }; + std::atomic accepts_ { 0 }; + std::thread acceptor_; + std::vector connections_; + std::mutex mutex_; + std::vector notes_; +}; + +inline void Peer::note(std::string text) { server_.note(std::move(text)); } + +inline std::string Peer::read_head() { + std::string head; + unsigned char c {}; + while (head.find("\r\n\r\n") == std::string::npos) { + if (head.size() > 65536) return {}; + // 20 s is far beyond any test; the loop exists so `stop` is noticed. + bool ready = false; + for (int waited = 0; waited < 20000 && !ready; waited += 50) { + if (server_.stopping()) return {}; + ready = wait_readable(50); + } + if (!ready) return {}; + if (mbedtls_net_recv(&net_, &c, 1) <= 0) return {}; + head.push_back(static_cast(c)); + } + return head; +} + +inline std::string Peer::read_exact(std::size_t n) { + std::string out; + unsigned char c {}; + while (out.size() < n) { + bool ready = false; + for (int waited = 0; waited < 20000 && !ready; waited += 50) { + if (server_.stopping()) return {}; + ready = wait_readable(50); + } + if (!ready || mbedtls_net_recv(&net_, &c, 1) <= 0) return {}; + out.push_back(static_cast(c)); + } + return out; +} + +inline std::string Peer::read_some() { + unsigned char buf[4096]; + for (int waited = 0; waited < 20000; waited += 50) { + if (server_.stopping()) return {}; + if (!wait_readable(50)) continue; + int n = mbedtls_net_recv(&net_, buf, sizeof buf); + return n > 0 ? std::string(reinterpret_cast(buf), static_cast(n)) + : std::string(); + } + return {}; +} + +inline void Peer::relay_to(int port) { + mbedtls_net_context target {}; + mbedtls_net_init(&target); + if (mbedtls_net_connect(&target, "127.0.0.1", std::to_string(port).c_str(), + MBEDTLS_NET_PROTO_TCP) != 0) { + return; + } + // Round-robin over two sockets with a short poll each: slower than a + // select over both, and portable without a second code path. + unsigned char buf[4096]; + bool open = true; + while (open && !server_.stopping()) { + for (auto [from, to] : {std::pair{&net_, &target}, std::pair{&target, &net_}}) { + if (mbedtls_net_poll(from, MBEDTLS_NET_POLL_READ, 10) <= 0) continue; + int n = mbedtls_net_recv(from, buf, sizeof buf); + if (n <= 0) { open = false; break; } + for (int sent = 0; sent < n;) { + int w = mbedtls_net_send(to, buf + sent, static_cast(n - sent)); + if (w <= 0) { open = false; break; } + sent += w; + } + } + } + mbedtls_net_free(&target); +} + +} // namespace proxy_test diff --git a/tests/test_ca_store.cpp b/tests/test_ca_store.cpp new file mode 100644 index 0000000..38e2dd7 --- /dev/null +++ b/tests/test_ca_store.cpp @@ -0,0 +1,42 @@ +// The trust store the platform supplies is found with nothing configured, and a +// public certificate chain verifies against it. +// +// On a Windows Sockets build the store is the system's ROOT store, read through +// the Win32 API; elsewhere it is a bundle file at one of the known locations. +// Either way a default-constructed client verifies a public server. A host that +// cannot be reached is a skip, so a runner without egress reports "not run"; a +// certificate the client refused is a failure. +#include + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +TEST(SystemStore, IsFoundWithNothingConfigured) { + if (const char* file = std::getenv("SSL_CERT_FILE"); file && *file) { + GTEST_SKIP() << "SSL_CERT_FILE is set, so the system store is not consulted"; + } + const auto pem = https::load_ca_certs(); + EXPECT_FALSE(pem.empty()); + EXPECT_NE(pem.find("-----BEGIN CERTIFICATE-----"), std::string::npos); +} + +TEST(SystemStore, APublicChainVerifiesAgainstIt) { + https::Socket::platform_init(); + https::HttpClientConfig config; + config.connectTimeoutMs = 15000; + config.readTimeoutMs = 20000; + https::HttpClient client{config}; + + https::HttpRequest request; + request.url = "https://one.one.one.one/"; + const auto response = client.send(request); + + if (response.statusCode == 0 && response.statusText == "Connection failed") { + GTEST_SKIP() << "one.one.one.one is not reachable from this runner"; + } + EXPECT_NE(response.statusCode, 0) << response.statusText; +} diff --git a/tests/test_cancel.cpp b/tests/test_cancel.cpp new file mode 100644 index 0000000..9d677bc --- /dev/null +++ b/tests/test_cancel.cpp @@ -0,0 +1,812 @@ +// Abandoning a request from another thread through a std::stop_token. +// +// Every wait here is far longer than the test should take (readTimeoutMs is +// 30 s), so a request that returns promptly with `cancelled` set can only have +// been ended by the token, never by a timeout. + +// WHERE `import std` GOES DEPENDS ON THE STANDARD LIBRARY, AND ONLY HERE. +// +// Every other test file includes its headers and then imports std, which is the +// order libstdc++ needs: the other way round, gcc 16 rejects the headers as +// redefinitions of what the module already declared. This file is the one that +// calls `std::stop_source::request_stop()`, and libc++ 20 and 22 compile that +// call, after a textual or (gtest and the test +// servers include both), into a reference to an inline helper that nothing then +// emits: the link fails on `__atomic_unique_lock<...>::__set_locked_bit`. With +// the import first it links, and libc++ 23 links either way. Measured with the +// toolchains mcpp installs; see .agents/docs/design-cancel-stop-token.md §6.2. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif + +namespace https = mcpplibs::tinyhttps; + +namespace { + +using Clock = std::chrono::steady_clock; + +// Well above the 50 ms slice and still far below any timeout in play. +constexpr long long kPromptMs = 500; + +https::HttpClientConfig test_config(int readTimeoutMs = 30000) { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = readTimeoutMs; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::GET; + req.url = url; + return req; +} + +long long ms_between(Clock::time_point from, Clock::time_point to) { + return std::chrono::duration_cast(to - from).count(); +} + +// Requests a stop `delay` after `ready` first holds (at once when it is not +// given) and remembers when it did, so a test can measure from the stop to the +// return of the call. Waiting for the server to have seen the request keeps a +// slow machine from stopping a request that was never sent. +class Stopper { +public: + explicit Stopper(std::chrono::milliseconds delay, + std::function ready = {}) + : thread_([this, delay, ready = std::move(ready)] { + for (int i = 0; ready && !ready() && i < 1000 && !quit_.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + for (auto end = Clock::now() + delay; + Clock::now() < end && !quit_.load();) { + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + if (quit_.load()) return; + stoppedAt_.store(Clock::now().time_since_epoch().count()); + source.request_stop(); + }) {} + // A stop that has not come yet is abandoned, so a long delay costs nothing + // when the call it was meant to end has already returned. + ~Stopper() { + quit_ = true; + if (thread_.joinable()) thread_.join(); + } + + long long since_stop_ms(Clock::time_point returned) { + thread_.join(); + return ms_between(Clock::time_point(Clock::duration(stoppedAt_.load())), returned); + } + + std::stop_source source; + +private: + std::atomic stoppedAt_ { 0 }; + std::atomic quit_ { false }; + std::thread thread_; +}; + +// A TCP listener that never speaks TLS: it accepts, reads and discards, so a +// client's handshake waits for a ServerHello that is not coming. +class SilentListener { +public: + SilentListener() { + mbedtls_net_init(&listener_); + if (mbedtls_net_bind(&listener_, "127.0.0.1", "0", MBEDTLS_NET_PROTO_TCP) != 0) { + failed_ = true; + return; + } + sockaddr_in addr {}; + socklen_t len = sizeof addr; + if (::getsockname(listener_.fd, reinterpret_cast(&addr), &len) != 0) { + failed_ = true; + return; + } + port_ = ntohs(addr.sin_port); + thread_ = std::thread([this] { run(); }); + } + + ~SilentListener() { + release(); + if (thread_.joinable()) thread_.join(); + mbedtls_net_free(&listener_); + } + + [[nodiscard]] bool failed() const { return failed_; } + [[nodiscard]] std::string url() const { + return "https://127.0.0.1:" + std::to_string(port_) + "/"; + } + [[nodiscard]] bool peer_closed() const { return peerClosed_.load(); } + [[nodiscard]] bool received() const { return received_.load(); } + + // Lets go of the accepted connection and stops listening. + void release() { released_.store(true); } + +private: + void run() { + while (!released_.load()) { + const int ready = mbedtls_net_poll(&listener_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) return; + if (ready == 0) continue; + mbedtls_net_context client {}; + mbedtls_net_init(&client); + if (mbedtls_net_accept(&listener_, &client, nullptr, 0, nullptr) != 0) { + mbedtls_net_free(&client); + continue; + } + unsigned char buf[512]; + while (!released_.load()) { + const int r = mbedtls_net_poll(&client, MBEDTLS_NET_POLL_READ, 50); + if (r < 0) break; + if (r == 0) continue; + if (mbedtls_net_recv(&client, buf, sizeof buf) <= 0) { + peerClosed_.store(true); + break; + } + received_.store(true); + } + mbedtls_net_free(&client); + return; + } + } + + mbedtls_net_context listener_ {}; + int port_ { 0 }; + bool failed_ { false }; + std::atomic released_ { false }; + std::atomic peerClosed_ { false }; + std::atomic received_ { false }; + std::thread thread_; +}; + +// Waits up to `limit` for `predicate`, which is how a test learns that the server +// saw the connection end. +bool becomes_true(const std::function& predicate, + std::chrono::milliseconds limit = std::chrono::seconds(2)) { + const auto deadline = Clock::now() + limit; + while (!predicate() && Clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + return predicate(); +} + +class CancelTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +} // namespace + +// The case that motivates all of this: the request is written and the server +// has not answered. The server also records that the connection ended, which is +// what makes it stop working on a request nobody is waiting for. +TEST_F(CancelTest, ARequestWaitingForTheHeadersIsAbandoned) { + std::atomic peerClosed { false }; + tls_test::Server server([&](tls_test::Conn& conn, int) { + peerClosed = conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(200), [&] { return server.requests() >= 1; }); + + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + const auto returned = Clock::now(); + const auto latency = stopper.since_stop_ms(returned); + std::cout << "[ latency ] headers: " << latency << " ms after the stop" << std::endl; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_FALSE(res.bodyComplete); + EXPECT_LT(latency, kPromptMs); + EXPECT_TRUE(becomes_true([&] { return peerClosed.load(); })) + << "the connection was not closed"; +} + +TEST_F(CancelTest, AStreamWaitingForTheNextChunkIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n"); + conn.write("b\r\ndata: one\n\n\r\n"); + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + std::atomic events { 0 }; + Stopper stopper(std::chrono::milliseconds(100), [&] { return events.load() > 0; }); + + auto res = client.send_stream(get(server.url("/")), + [&](const https::SseEvent&) { ++events; return true; }, + stopper.source.get_token()); + const auto latency = stopper.since_stop_ms(Clock::now()); + std::cout << "[ latency ] stream: " << latency << " ms after the stop" << std::endl; + + EXPECT_EQ(res.statusCode, 200); + EXPECT_TRUE(res.cancelled); + EXPECT_FALSE(res.bodyComplete); + EXPECT_EQ(res.bodyError, "cancelled"); + EXPECT_EQ(events.load(), 1); + EXPECT_LT(latency, kPromptMs); +} + +// The handshake has no timeout of its own, so a server that accepts and says +// nothing holds the call for good unless the token ends it. The watchdog keeps +// a regression from hanging the suite: it lets go of the connection after 5 s. +TEST_F(CancelTest, AHandshakeThatNeverAnswersIsAbandoned) { + SilentListener listener; + ASSERT_FALSE(listener.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(200), [&] { return listener.received(); }); + + std::atomic done { false }; + std::atomic watchdogFired { false }; + std::thread watchdog([&] { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + if (!done.load()) { watchdogFired = true; listener.release(); } + }); + + auto res = client.send(get(listener.url()), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + watchdog.join(); + const auto latency = stopper.since_stop_ms(returned); + std::cout << "[ latency ] handshake: " << latency << " ms after the stop" << std::endl; + + EXPECT_FALSE(watchdogFired.load()) << "the handshake was not interrupted"; + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(latency, kPromptMs); + EXPECT_TRUE(becomes_true([&] { return listener.peer_closed(); })) + << "the connection was not closed"; +} + +TEST_F(CancelTest, ATokenAlreadyStoppedSendsNothing) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("hello")); + return true; + }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::HttpClient client(test_config()); + + auto res = client.send(get(server.url("/")), source.get_token()); + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + + auto streamed = client.send_stream(get(server.url("/")), + [](const https::SseEvent&) { return true; }, + source.get_token()); + EXPECT_TRUE(streamed.cancelled); + EXPECT_EQ(server.accepts(), 0); +} + +// The slices must not shorten or lengthen the timeout a token-carrying call +// already had. The token is stopped after 5 s, so a timeout that never comes +// fails the test rather than hanging the suite. +TEST_F(CancelTest, ATokenThatIsNeverStoppedLeavesTheTimeoutAlone) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config(/*readTimeoutMs=*/300)); + Stopper backstop(std::chrono::seconds(5)); + + const auto started = Clock::now(); + auto res = client.send(get(server.url("/")), backstop.source.get_token()); + const auto elapsed = ms_between(started, Clock::now()); + + EXPECT_FALSE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "No response"); + EXPECT_GT(elapsed, 250); + EXPECT_LT(elapsed, 1500); +} + +// A cancelled connection has an unknown amount of the response still on it, so +// it must not be handed to the next request. +TEST_F(CancelTest, ACancelledConnectionIsNotReused) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.wait_peer_close(); + return false; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto first = client.send(get(server.url("/")), stopper.source.get_token()); + EXPECT_TRUE(first.cancelled); + + auto second = client.send(get(server.url("/"))); + EXPECT_EQ(second.statusCode, 200) << "status text was: " << second.statusText; + EXPECT_EQ(second.body, "second"); + EXPECT_FALSE(second.cancelled); + EXPECT_EQ(server.accepts(), 2); +} + +// A pooled connection that fails before any response byte is normally retried +// on a new one. A request the caller has abandoned must not be sent again. +TEST_F(CancelTest, ACancelledRequestIsNotRetriedOnAFreshConnection) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write(tls_test::ok_response("first")); + return true; + } + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + EXPECT_EQ(client.send(get(server.url("/"))).statusCode, 200); + + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 2; }); + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + EXPECT_TRUE(res.cancelled); + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(server.requests(), 2); + EXPECT_EQ(server.accepts(), 1); +} + +namespace { + +int open_fds() { + std::error_code ec; + for (const char* dir : { "/proc/self/fd", "/dev/fd" }) { + if (!std::filesystem::exists(dir, ec)) continue; + int n = 0; + for (auto it = std::filesystem::directory_iterator(dir, ec); + !ec && it != std::filesystem::directory_iterator(); it.increment(ec)) { + ++n; + } + return n; + } + return -1; +} + +void cancel_stuck_requests(int times) { + std::atomic closed { 0 }; + tls_test::Server server([&](tls_test::Conn& conn, int) { + if (conn.wait_peer_close()) ++closed; + return false; + }); + https::HttpClient client(test_config()); + for (int i = 0; i < times; ++i) { + const int seen = server.requests(); + Stopper stopper(std::chrono::milliseconds(50), [&] { return server.requests() > seen; }); + client.send(get(server.url("/")), stopper.source.get_token()); + } + // The server's side of each connection is done before its descriptors are + // counted as gone. + for (int i = 0; i < 200 && closed.load() < times; ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + server.stop(); +} + +} // namespace + +TEST_F(CancelTest, CancelledRequestsDoNotLeakDescriptors) { + if (open_fds() < 0) GTEST_SKIP() << "no /proc/self/fd or /dev/fd"; + + cancel_stuck_requests(1); // anything the first request opens and keeps + const int before = open_fds(); + cancel_stuck_requests(20); + EXPECT_EQ(open_fds(), before); +} + +// A redirect whose body is still arriving when the stop comes has to be reported +// as it is, not followed: the next request would be one the caller abandoned. +TEST_F(CancelTest, ARedirectIsNotFollowedOnceTheRequestIsCancelled) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 302 Found\r\nLocation: /next\r\nContent-Length: 100\r\n\r\npartial"); + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 302); + EXPECT_EQ(res.bodyError, "cancelled"); + EXPECT_EQ(server.requests(), 1); +} + +// Connecting is a wait too, and nothing may be sent to an address once the stop +// has been asked for. The listener only has to exist: the kernel completes the +// connection for it, so an attempt shows as an accept. +TEST_F(CancelTest, AConnectAfterTheStopSendsNoSyn) { + tls_test::Server server([](tls_test::Conn&, int) { return false; }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::Socket sock; + sock.set_stop(source.get_token()); + + EXPECT_FALSE(sock.connect("127.0.0.1", server.port(), 4000)); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + EXPECT_EQ(server.accepts(), 0); +} + +// ── through a proxy ────────────────────────────────────────────────────────── + +namespace { + +https::HttpClientConfig proxied_config(const std::string& proxyUrl) { + auto cfg = test_config(); + cfg.proxy = proxyUrl; + return cfg; +} + +// Holds a connection for up to 5 s or until `done`, so a test whose cancellation +// fails sees the connection dropped (and an assertion fail) rather than a hang. +void hold(const std::atomic& done) { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +} + +} // namespace + +TEST_F(CancelTest, AProxyThatNeverAnswersConnectIsAbandoned) { + std::atomic sawConnect { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + sawConnect = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("http://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawConnect.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "Cancelled"); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +TEST_F(CancelTest, AnHttpsProxyThatNeverCompletesItsHandshakeIsAbandoned) { + std::atomic sawHello { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_some(); + sawHello = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("https://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawHello.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// The session to the target runs inside the one to the proxy, and the stop has +// to reach the proxy's session for the inner handshake to be interruptible. +TEST_F(CancelTest, AHandshakeInsideAnHttpsProxyTunnelIsAbandoned) { + std::atomic tunnelOpen { false }; + tls_test::Server proxy([&](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + tunnelOpen = true; + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("https://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(200), [&] { return tunnelOpen.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// ── a reused connection carries the current call's token ───────────────────── +// +// The token is set on a connection for one call and taken off when it goes back +// to the pool. Through an https:// proxy the waits happen in the session beneath +// the one the pool holds, and it was that session's token, set by the call that +// opened the tunnel, that every later call waited on. + +namespace { + +enum class Route { Direct, HttpProxy, HttpsProxy }; + +std::string route_name(const ::testing::TestParamInfo& info) { + switch (info.param) { + case Route::Direct: return "Direct"; + case Route::HttpProxy: return "HttpProxy"; + case Route::HttpsProxy: return "HttpsProxy"; + } + return "Unknown"; +} + +// The proxy in front of a target, when the route has one, and the client +// configuration that reaches the target through it. +class Path { +public: + Path(Route route, int targetPort) { + if (route == Route::HttpProxy) { + http_ = std::make_unique([targetPort](proxy_test::Peer& peer) { + if (peer.read_head().empty()) return; + peer.write("HTTP/1.1 200 Connection established\r\n\r\n"); + peer.relay_to(targetPort); + }); + } else if (route == Route::HttpsProxy) { + https_ = std::make_unique([targetPort](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.relay_to(targetPort); + return false; + }); + } + } + + [[nodiscard]] bool failed() const { + return (http_ && http_->failed()) || (https_ && https_->failed()); + } + + [[nodiscard]] https::HttpClientConfig config() const { + auto cfg = test_config(); + if (http_) cfg.proxy = "http://127.0.0.1:" + std::to_string(http_->port()); + if (https_) cfg.proxy = "https://127.0.0.1:" + std::to_string(https_->port()); + return cfg; + } + + // Connections the proxy accepted; the target's count when there is none. + [[nodiscard]] int proxy_accepts(const tls_test::Server& target) const { + if (http_) return http_->accepts(); + if (https_) return https_->accepts(); + return target.accepts(); + } + +private: + std::unique_ptr http_; + std::unique_ptr https_; +}; + +https::HttpRequest post(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::POST; + req.url = url; + req.body = "{}"; + return req; +} + +class ReuseTest : public CancelTest, public ::testing::WithParamInterface {}; + +} // namespace + +// THE README'S OWN PATTERN. A jthread requests a stop in its destructor, after +// the call it ran has returned and its connection is back in the pool. That stop +// must not reach the next call: through an https:// proxy it made every wait on +// the tunnel fail at once, and the stale-connection retry then sent a POST the +// target had already received a second time. +TEST_P(ReuseTest, AStopAfterTheCallReturnedDoesNotReachTheNextCall) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("done")); + return true; + }); + ASSERT_FALSE(target.failed()); + Path path(GetParam(), target.port()); + ASSERT_FALSE(path.failed()); + + https::HttpClient client(path.config()); + { + std::atomic returned { false }; + std::jthread worker([&](std::stop_token stop) { + auto first = client.send(post(target.url("/first")), stop); + EXPECT_EQ(first.statusCode, 200) << first.statusText; + EXPECT_FALSE(first.cancelled); + returned = true; + }); + while (!returned.load()) std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } // ~jthread: request_stop(), then join + + auto second = client.send(post(target.url("/second"))); + EXPECT_EQ(second.statusCode, 200) << second.statusText; + EXPECT_FALSE(second.cancelled); + + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(target.requests(), 2) << "the second POST reached the target more than once"; + EXPECT_EQ(target.accepts(), 1) << "the pooled connection was not reused"; + EXPECT_EQ(path.proxy_accepts(target), 1); +} + +// The other half: the second call's own token has to reach the connection it +// reuses, whatever the first call's token is doing. +TEST_P(ReuseTest, TheSecondCallsTokenReachesTheReusedConnection) { + tls_test::Server target([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write(tls_test::ok_response("first")); + return true; + } + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(target.failed()); + Path path(GetParam(), target.port()); + ASSERT_FALSE(path.failed()); + + https::HttpClient client(path.config()); + std::stop_source first; // never stopped, and alive throughout + EXPECT_EQ(client.send(get(target.url("/a")), first.get_token()).statusCode, 200); + + Stopper stopper(std::chrono::milliseconds(100), [&] { return target.requests() >= 2; }); + auto res = client.send(get(target.url("/b")), stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled) << res.statusText; + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); + EXPECT_EQ(target.accepts(), 1) << "the second call did not reuse the connection"; +} + +INSTANTIATE_TEST_SUITE_P(Routes, ReuseTest, + ::testing::Values(Route::Direct, Route::HttpProxy, Route::HttpsProxy), + route_name); + +TEST_F(CancelTest, ASocks5ProxyThatNeverAnswersTheGreetingIsAbandoned) { + std::atomic sawGreeting { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_some(); + sawGreeting = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("socks5://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawGreeting.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// ── download_to_file ───────────────────────────────────────────────────────── + +namespace { + +std::filesystem::path scratch_file(std::string_view name) { + auto path = std::filesystem::temp_directory_path() + / ("tinyhttps-cancel-" + std::string(name)); + std::error_code ec; + std::filesystem::remove(path, ec); + return path; +} + +} // namespace + +TEST_F(CancelTest, ADownloadWaitingForTheHeadersIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("head"); + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto res = client.download_to_file(server.url("/file"), file, nullptr, nullptr, + stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.error, "Cancelled"); + EXPECT_FALSE(res.ok()); + EXPECT_FALSE(std::filesystem::exists(file)) << "no file before a status line"; + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +TEST_F(CancelTest, ADownloadWaitingForTheRestOfTheBodyIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 100000\r\n\r\n"); + conn.write(std::string(1000, 'x')); + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("body"); + std::atomic written { 0 }; + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return written.load() >= 1000; }); + auto res = client.download_to_file(server.url("/file"), file, + [&](std::int64_t, std::int64_t now) { written = now; }, + nullptr, stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 200); + EXPECT_EQ(res.error, "cancelled"); + EXPECT_FALSE(res.writeFailed); + EXPECT_EQ(res.bytesWritten, 1000); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); + + std::error_code ec; + std::filesystem::remove(file, ec); +} + +TEST_F(CancelTest, ADownloadWithATokenAlreadyStoppedSendsNothing) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("never")); + return true; + }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::HttpClient client(test_config()); + auto res = client.download_to_file(server.url("/file"), scratch_file("none"), + nullptr, nullptr, source.get_token()); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.error, "Cancelled"); + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(server.accepts(), 0); +} + +// `isCancelled` and the token are two ways to say the same thing, and the result +// says it the same way for both. +TEST_F(CancelTest, IsCancelledSetsCancelledAsTheTokenDoes) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 100000\r\n\r\n"); + conn.write(std::string(1000, 'x')); + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("predicate"); + https::HttpClient client(test_config()); + auto res = client.download_to_file(server.url("/file"), file, nullptr, + [] { return true; }); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.error, "cancelled"); + EXPECT_EQ(res.statusCode, 200); + + std::error_code ec; + std::filesystem::remove(file, ec); +} diff --git a/tests/test_download.cpp b/tests/test_download.cpp index c862fbe..e7288cd 100644 --- a/tests/test_download.cpp +++ b/tests/test_download.cpp @@ -1,4 +1,10 @@ -// Test must include gtest before import std to avoid GCC module redefinition errors +// Live tests against a real HTTPS endpoint (httpbin.org). +// +// The framing parsers and the response contracts these used to also cover moved +// to `test_framing.cpp`, and the connection-pool behaviour to `test_pool.cpp`, +// which scripts its own server. What is left here is the part that genuinely +// needs the internet: a real certificate chain, a real chunked stream, a real +// redirect. #include import mcpplibs.tinyhttps; @@ -6,34 +12,84 @@ import std; namespace https = mcpplibs::tinyhttps; -TEST(ChunkedProtocol, RejectsEmptyInvalidAndOverflowSizeLines) { - EXPECT_FALSE(https::parse_chunk_size_line("").has_value()); - EXPECT_FALSE(https::parse_chunk_size_line("xyz").has_value()); - EXPECT_FALSE(https::parse_chunk_size_line("1g").has_value()); - EXPECT_FALSE(https::parse_chunk_size_line("FFFFFFFFFFFFFFFF").has_value()); -} +// Test that a failed streaming request carries its error body, against a real +// HTTPS endpoint. httpbin's /status/418 answers non-2xx with a body, which is +// exactly the shape SseParser cannot turn into events. + +class StreamErrorBodyLiveTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +// ON THE LIBRARY'S OWN DEFAULTS, AND THE READ TIMEOUT IS PART OF THE +// OBSERVATION RATHER THAN A SAFETY NET. +// +// The first form of this test set `keepAlive = false`, "so the server closes +// and the read loop ends". That comment was the defect: `send_stream` had no +// branch for a declared `Content-Length`, so a response that was not chunked +// was read until the connection closed — and on the defaults the server does +// not close. The body arrived, one full `readTimeoutMs` late. +// +// keepAlive = false status=418 body=135 elapsed 1370 ms +// keepAlive = true status=418 body=135 elapsed 9379 ms (timeout 8000) +// +// So the configuration under test is the default one, and the elapsed time is +// asserted. A test that closed the connection to make the loop end would be +// examining the one arrangement in which the defect does not appear. +TEST_F(StreamErrorBodyLiveTest, FailedStreamKeepsTheErrorBody) { + https::HttpClientConfig cfg; + cfg.connectTimeoutMs = 15000; + cfg.readTimeoutMs = 30000; + // keepAlive is left at its default, which is true. + https::HttpClient client(cfg); -TEST(ChunkedProtocol, AcceptsValidSizeAndTerminalChunk) { - ASSERT_TRUE(https::parse_chunk_size_line("1a").has_value()); - EXPECT_EQ(*https::parse_chunk_size_line("1a"), 26); - ASSERT_TRUE(https::parse_chunk_size_line("0").has_value()); - EXPECT_EQ(*https::parse_chunk_size_line("0"), 0); + https::HttpRequest req; + req.method = https::Method::GET; + req.url = "https://httpbin.org/status/418"; + + int events = 0; + const auto started = std::chrono::steady_clock::now(); + auto res = client.send_stream(req, [&](const https::SseEvent&) { + ++events; + return true; + }); + const auto elapsed = std::chrono::duration_cast( + std::chrono::steady_clock::now() - started).count(); + + EXPECT_EQ(res.statusCode, 418); + EXPECT_EQ(events, 0) << "an error document is not an event stream"; + EXPECT_FALSE(res.body.empty()) << "error body was dropped"; + EXPECT_NE(res.body.find("teapot"), std::string::npos); + // Generous against a slow runner and still an order of magnitude below the + // read timeout, which is what the defect consumed. + EXPECT_LT(elapsed, 15000) + << "the body arrived after " << elapsed + << " ms; a declared Content-Length was not honoured and the reader " + "waited for a close that keep-alive was never going to bring"; } -TEST(DownloadResultContract, CarriesTransferAndResponseMetadata) { - https::DownloadToFileResult result; - result.bytesWritten = 42; - result.expectedBytes = 42; - result.finalUrl = "https://example.test/final"; - result.etag = "\"abc\""; - result.lastModified = "Wed, 21 Oct 2015 07:28:00 GMT"; - - EXPECT_EQ(result.bytesWritten, 42); - ASSERT_TRUE(result.expectedBytes.has_value()); - EXPECT_EQ(*result.expectedBytes, 42); - EXPECT_EQ(result.finalUrl, "https://example.test/final"); - EXPECT_EQ(result.etag, "\"abc\""); - EXPECT_FALSE(result.lastModified.empty()); +// The success path, on the framing an event stream actually uses. A 2xx is not +// captured, so `body` stays empty and the bytes reach the parser — and it must +// still return promptly, since the chunked branch is the one this change did +// not restructure. +TEST_F(StreamErrorBodyLiveTest, AChunkedSuccessIsUnchangedAndPrompt) { + https::HttpClientConfig cfg; + cfg.connectTimeoutMs = 15000; + cfg.readTimeoutMs = 30000; + https::HttpClient client(cfg); + + https::HttpRequest req; + req.method = https::Method::GET; + req.url = "https://httpbin.org/stream/3"; + + const auto started = std::chrono::steady_clock::now(); + auto res = client.send_stream(req, [](const https::SseEvent&) { return true; }); + const auto elapsed = std::chrono::duration_cast( + std::chrono::steady_clock::now() - started).count(); + + EXPECT_EQ(res.statusCode, 200); + EXPECT_TRUE(res.body.empty()) << "a 2xx body is not captured; it is the caller's stream"; + EXPECT_LT(elapsed, 15000) << "the chunked reader did not terminate promptly"; } // Test download_to_file against a real HTTPS endpoint. diff --git a/tests/test_extra_ca.cpp b/tests/test_extra_ca.cpp new file mode 100644 index 0000000..f86875c --- /dev/null +++ b/tests/test_extra_ca.cpp @@ -0,0 +1,243 @@ +// `HttpClientConfig::extraCaFile` adds roots to the default trust store. +// +// The default store is whatever SSL_CERT_FILE names, so each test writes the +// bundles it needs to temp files and the machine's own store never enters into +// it. The servers present leaves signed by two private CAs (extra_ca_certs.hpp); +// nothing leaves 127.0.0.1. +#include +#include "extra_ca_certs.hpp" +#include "tls_test_server.hpp" + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; +namespace pki = extra_ca_test; + +namespace { + +void set_env(const char* name, const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s(name, value); +#else + ::setenv(name, value, 1); +#endif +} + +class ExtraCaTest : public ::testing::Test { +protected: + void SetUp() override { + https::Socket::platform_init(); + const char* old = std::getenv("SSL_CERT_FILE"); + hadOld_ = old != nullptr; + if (hadOld_) old_ = old; + } + + void TearDown() override { + set_env("SSL_CERT_FILE", hadOld_ ? old_.c_str() : ""); + std::error_code ec; + for (auto& path : files_) std::filesystem::remove(path, ec); + } + + // Writes `content` to a new temp file and returns its path. + std::string write_file(std::string_view content) { + auto path = std::filesystem::temp_directory_path() + / ("tinyhttps-extra-ca-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + "-" + std::to_string(files_.size()) + ".pem"); + std::ofstream(path, std::ios::binary) << content; + files_.push_back(path); + return path.string(); + } + + // Makes `content` the whole default store. + void default_store(std::string_view content) { + set_env("SSL_CERT_FILE", write_file(content).c_str()); + } + + static https::HttpClientConfig config(std::string extraCaFile = {}) { + https::HttpClientConfig cfg; // verifySsl is true by default + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + cfg.extraCaFile = std::move(extraCaFile); + return cfg; + } + + static https::HttpResponse get(const https::HttpClientConfig& cfg, const std::string& url) { + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = url; + return client.send(req); + } + +private: + std::vector files_; + std::string old_; + bool hadOld_ { false }; +}; + +bool answer_ok(tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("ok")); + return true; +} + +tls_test::Server server_of_ca1() { + return tls_test::Server(answer_ok, pki::kLeaf1Pem, pki::kLeaf1KeyPem); +} + +tls_test::Server server_of_ca2() { + return tls_test::Server(answer_ok, pki::kLeaf2Pem, pki::kLeaf2KeyPem); +} + +} // namespace + +TEST_F(ExtraCaTest, APrivateCaIsNotTrustedWithoutTheFile) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(tls_test::kCertPem); // a store that does not contain CA 1 + + auto r = get(config(), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("not correctly signed by the trusted CA"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, APrivateCaIsTrustedWithTheFile) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(tls_test::kCertPem); + + auto r = get(config(write_file(pki::kCa1Pem)), server.url("/")); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + EXPECT_EQ(r.body, "ok"); +} + +// The file adds to the default store; it does not replace it. +TEST_F(ExtraCaTest, TheDefaultStoreStaysInEffect) { + auto fromDefault = server_of_ca1(); + auto fromExtra = server_of_ca2(); + auto neither = tls_test::Server(answer_ok); // self-signed, in no store + ASSERT_FALSE(fromDefault.failed()); + ASSERT_FALSE(fromExtra.failed()); + ASSERT_FALSE(neither.failed()); + default_store(pki::kCa1Pem); + auto cfg = config(write_file(pki::kCa2Pem)); + + EXPECT_EQ(get(cfg, fromDefault.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, fromExtra.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, neither.url("/")).statusCode, 0); +} + +TEST_F(ExtraCaTest, AFileHoldingSeveralCertificatesTrustsAllOfThem) { + auto a = server_of_ca1(); + auto b = server_of_ca2(); + ASSERT_FALSE(a.failed()); + ASSERT_FALSE(b.failed()); + default_store(tls_test::kCertPem); + auto cfg = config(write_file(std::string(pki::kCa1Pem) + pki::kCa2Pem)); + + EXPECT_EQ(get(cfg, a.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, b.url("/")).statusCode, 200); +} + +// Both hops of a request through an https:// proxy use the file: the proxy's +// certificate comes from CA 2, which only the file names, and the target's from +// CA 1, which only the default store names. +TEST_F(ExtraCaTest, TheHopToAnHttpsProxyUsesTheFileToo) { + auto target = server_of_ca1(); + ASSERT_FALSE(target.failed()); + const int targetPort = target.port(); + auto proxy = tls_test::Server([targetPort](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.relay_to(targetPort); + return false; + }, pki::kLeaf2Pem, pki::kLeaf2KeyPem); + ASSERT_FALSE(proxy.failed()); + default_store(pki::kCa1Pem); + + auto cfg = config(write_file(pki::kCa2Pem)); + cfg.proxy = "https://127.0.0.1:" + std::to_string(proxy.port()); + auto r = get(cfg, target.url("/")); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + + cfg.extraCaFile.clear(); + r = get(cfg, target.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_TRUE(r.statusText.starts_with("proxy: could not open a TLS connection to 127.0.0.1:")) + << r.statusText; + EXPECT_NE(r.statusText.find("certificate verification failed"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileThatDoesNotExistIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); // the server would verify without the file + const auto missing = (std::filesystem::temp_directory_path() / "tinyhttps-no-such-ca.pem").string(); + + auto r = get(config(missing), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("cannot read extra CA file '" + missing + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileThatIsNotPemIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("this is not a certificate\n"); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileWhoseCertificateCannotBeParsedIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("-----BEGIN CERTIFICATE-----\nAAAA\n-----END CERTIFICATE-----\n"); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AnEmptyFileIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file(""); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +// The file is checked whether or not the server would be verified. +TEST_F(ExtraCaTest, AFileThatCannotBeUsedFailsEvenWithoutVerification) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("not a certificate\n"); + + auto cfg = config(path); + cfg.verifySsl = false; + auto r = get(cfg, server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +// SSL_CERT_FILE keeps its meaning when the field is unset. +TEST_F(ExtraCaTest, WithoutTheFieldNothingChanges) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + EXPECT_EQ(get(config(), server.url("/")).statusCode, 200); +} diff --git a/tests/test_framing.cpp b/tests/test_framing.cpp new file mode 100644 index 0000000..deb5d9f --- /dev/null +++ b/tests/test_framing.cpp @@ -0,0 +1,211 @@ +// The half of HTTP framing that can be examined without a server. +// +// `parse_chunk_size_line` and `parse_content_length` are exported so they can be +// tested here rather than through a network; `parse_status_line` joins them for +// the same reason. Every case below is a string a server can send. +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +// ── the status line ────────────────────────────────────────────────────────── + +TEST(StatusLine, AcceptsTheOrdinaryShape) { + auto parsed = https::parse_status_line("HTTP/1.1 200 OK"); + ASSERT_TRUE(parsed.has_value()); + EXPECT_EQ(parsed->code, 200); + EXPECT_EQ(parsed->text, "OK"); +} + +TEST(StatusLine, AReasonPhraseIsOptionalAndMayContainSpaces) { + auto none = https::parse_status_line("HTTP/1.1 204"); + ASSERT_TRUE(none.has_value()); + EXPECT_EQ(none->code, 204); + EXPECT_TRUE(none->text.empty()); + + auto phrase = https::parse_status_line("HTTP/1.0 404 Not Found Here"); + ASSERT_TRUE(phrase.has_value()); + EXPECT_EQ(phrase->code, 404); + EXPECT_EQ(phrase->text, "Not Found Here"); +} + +// THE ONE FROM ISSUE #15. +// +// A socket returned to the pool with the previous response's body still on it +// gave the next request this to read. The older parser looked for the first +// space, then the second, then kept whatever digits it found between them: +// +// "BBBB 999 XHTTP/1.1 200 OK" +// ^ first space -> rest = "999 XHTTP/1.1 200 OK" +// ^ second space -> code = "999" +// +// and reported a perfectly ordinary-looking 999 to the caller. +TEST(StatusLine, RejectsALineThatDoesNotBeginWithTheVersion) { + EXPECT_FALSE(https::parse_status_line("BBBB 999 XHTTP/1.1 200 OK").has_value()); + EXPECT_FALSE(https::parse_status_line("").has_value()); + EXPECT_FALSE(https::parse_status_line("200 OK").has_value()); + EXPECT_FALSE(https::parse_status_line("").has_value()); + // A run of digits with letters around it used to become 999 as well. + EXPECT_FALSE(https::parse_status_line("X9Y9Z9 999 OK").has_value()); +} + +TEST(StatusLine, RejectsAStatusCodeThatIsNotThreeDigits) { + EXPECT_FALSE(https::parse_status_line("HTTP/1.1 20 OK").has_value()); + EXPECT_FALSE(https::parse_status_line("HTTP/1.1 2000 OK").has_value()); + EXPECT_FALSE(https::parse_status_line("HTTP/1.1 20a OK").has_value()); + EXPECT_FALSE(https::parse_status_line("HTTP/1.1 +20 OK").has_value()); + EXPECT_FALSE(https::parse_status_line("HTTP/1.1").has_value()); + // Outside the range the grammar defines. 999 is what issue #15 reported, + // and it is not a status code either. + EXPECT_FALSE(https::parse_status_line("HTTP/1.1 099 OK").has_value()); + EXPECT_FALSE(https::parse_status_line("HTTP/1.1 999 OK").has_value()); +} + +TEST(StatusLine, AcceptsEveryClassTheGrammarDefines) { + for (int code : {100, 200, 204, 301, 302, 404, 418, 500, 599}) { + auto line = "HTTP/1.1 " + std::to_string(code) + " X"; + auto parsed = https::parse_status_line(line); + ASSERT_TRUE(parsed.has_value()) << line; + EXPECT_EQ(parsed->code, code); + } +} + +// ── chunk sizes ────────────────────────────────────────────────────────────── + +TEST(ChunkedProtocol, RejectsEmptyInvalidAndOverflowSizeLines) { + EXPECT_FALSE(https::parse_chunk_size_line("").has_value()); + EXPECT_FALSE(https::parse_chunk_size_line("xyz").has_value()); + EXPECT_FALSE(https::parse_chunk_size_line("1g").has_value()); + EXPECT_FALSE(https::parse_chunk_size_line("FFFFFFFFFFFFFFFF").has_value()); + // The two the older `parse_hex` turned into a terminal chunk: an empty line + // (what a timed-out read produced) and a line of letters. + EXPECT_FALSE(https::parse_chunk_size_line("zz").has_value()); +} + +TEST(ChunkedProtocol, AcceptsValidSizeAndTerminalChunk) { + ASSERT_TRUE(https::parse_chunk_size_line("1a").has_value()); + EXPECT_EQ(*https::parse_chunk_size_line("1a"), 26); + ASSERT_TRUE(https::parse_chunk_size_line("0").has_value()); + EXPECT_EQ(*https::parse_chunk_size_line("0"), 0); +} + +// ── content length ─────────────────────────────────────────────────────────── + +// `Content-Length` decides how many bytes a reader will trust, so a wrong +// answer is not cosmetic: too small leaves the next response's bytes in the +// stream and too large waits for bytes that are not coming. +// +// Both readers used to keep the digits and discard everything else. Measured, +// by compiling that parser on its own: +// +// "135" -> 135 +// "0" -> 0 +// "abc" -> 0 <- a refusal read as a real zero +// "12abc" -> 12 <- stops twelve bytes in +// "" -> 0 +// "-1" -> 1 <- the sign is discarded +// "99999999999999999999" -> 7766279631452241919 <- wraps, in silence +// +// The last two are the ones no amount of care at the call site could recover +// from, because the value it receives is a plausible number. +TEST(ContentLength, AcceptsAWellFormedValue) { + ASSERT_TRUE(https::parse_content_length("135").has_value()); + EXPECT_EQ(*https::parse_content_length("135"), 135); + // A field value may carry optional whitespace on either side. + ASSERT_TRUE(https::parse_content_length(" 135\t").has_value()); + EXPECT_EQ(*https::parse_content_length(" 135\t"), 135); +} + +// The one a salvaging parser cannot express. `abc` used to yield 0, which is +// indistinguishable from a server that genuinely declared an empty body — and +// the two call for opposite behaviour. +TEST(ContentLength, AZeroIsDistinguishableFromARefusal) { + ASSERT_TRUE(https::parse_content_length("0").has_value()); + EXPECT_EQ(*https::parse_content_length("0"), 0); + EXPECT_FALSE(https::parse_content_length("abc").has_value()); +} + +TEST(ContentLength, RejectsEmptyTrailingGarbageAndOverflow) { + EXPECT_FALSE(https::parse_content_length("").has_value()); + EXPECT_FALSE(https::parse_content_length(" ").has_value()); + // `12abc` used to be 12: the reader would then stop twelve bytes in and + // leave the rest of the body to be read as the next response. + EXPECT_FALSE(https::parse_content_length("12abc").has_value()); + EXPECT_FALSE(https::parse_content_length("-1").has_value()); + EXPECT_FALSE(https::parse_content_length("+1").has_value()); + // Past the width of the accumulator, which used to wrap in silence. + EXPECT_FALSE(https::parse_content_length("99999999999999999999").has_value()); +} + +// A value that fits in 64 bits but not in 32. `send_impl` held this in an `int` +// and truncated 2^32 to 0 in silence, which read as "the body is empty". +TEST(ContentLength, KeepsAValuePastThirtyTwoBits) { + auto parsed = https::parse_content_length("4294967296"); + ASSERT_TRUE(parsed.has_value()); + EXPECT_EQ(*parsed, 4294967296LL); +} + +// ── the bounded error-body buffer ──────────────────────────────────────────── + +TEST(StreamErrorBody, KeepsEverythingWhileUnderLimit) { + std::string buffer; + EXPECT_TRUE(https::append_within_limit(buffer, "abc", 8)); + EXPECT_TRUE(https::append_within_limit(buffer, "de", 8)); + EXPECT_EQ(buffer, "abcde"); +} + +TEST(StreamErrorBody, TruncatesTheChunkThatCrossesTheLimit) { + std::string buffer = "abc"; + EXPECT_FALSE(https::append_within_limit(buffer, "defgh", 5)); + EXPECT_EQ(buffer, "abcde"); +} + +TEST(StreamErrorBody, RefusesFurtherDataOnceFull) { + std::string buffer = "abcde"; + EXPECT_FALSE(https::append_within_limit(buffer, "f", 5)); + EXPECT_EQ(buffer, "abcde"); + // A zero limit must not append anything, not even an empty append. + std::string empty; + EXPECT_FALSE(https::append_within_limit(empty, "a", 0)); + EXPECT_TRUE(empty.empty()); +} + +// ── the response contract ──────────────────────────────────────────────────── + +// `bodyComplete` defaults to true, so code that never looks at it behaves as it +// did. That default is the whole of the compatibility claim for this field. +TEST(ResponseContract, ABodyIsCompleteUntilSomethingSaysOtherwise) { + https::HttpResponse response; + EXPECT_TRUE(response.bodyComplete); + EXPECT_TRUE(response.bodyError.empty()); +} + +// `ok()` answers what the server said, and deliberately does not consult +// `bodyComplete`. Making it stricter would change what existing `if (res.ok())` +// means without changing what it says. +TEST(ResponseContract, OkReportsTheStatusCodeAndNotTheTransfer) { + https::HttpResponse response; + response.statusCode = 200; + response.bodyComplete = false; + response.bodyError = "Connection closed before the declared length"; + EXPECT_TRUE(response.ok()); + EXPECT_FALSE(response.bodyComplete); +} + +TEST(DownloadResultContract, CarriesTransferAndResponseMetadata) { + https::DownloadToFileResult result; + result.bytesWritten = 42; + result.expectedBytes = 42; + result.finalUrl = "https://example.test/final"; + result.etag = "\"abc\""; + result.lastModified = "Wed, 21 Oct 2015 07:28:00 GMT"; + + EXPECT_EQ(result.bytesWritten, 42); + ASSERT_TRUE(result.expectedBytes.has_value()); + EXPECT_EQ(*result.expectedBytes, 42); + EXPECT_EQ(result.finalUrl, "https://example.test/final"); + EXPECT_EQ(result.etag, "\"abc\""); + EXPECT_FALSE(result.lastModified.empty()); +} diff --git a/tests/test_handshake_timeout.cpp b/tests/test_handshake_timeout.cpp new file mode 100644 index 0000000..a825612 --- /dev/null +++ b/tests/test_handshake_timeout.cpp @@ -0,0 +1,289 @@ +// The TLS handshake is bounded by connectTimeoutMs. +// +// Each case gives the client a peer that completes the TCP connection and then +// never answers the ClientHello, with no stop token to end the wait. The peer +// holds the connection for at most 5 s and then drops it, so a handshake that +// has no deadline fails the case (by the message and by the time it took) +// instead of hanging the suite. + +// As in test_cancel.cpp, `import std` goes first with libc++ because of the +// stop token in one of the cases, and after the includes otherwise. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" +#ifndef _WIN32 +#include +#include +#endif + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif + +namespace https = mcpplibs::tinyhttps; + +namespace { + +using Clock = std::chrono::steady_clock; + +// Far below the 5 s the peers hold on for, far above the 300 ms timeout. +constexpr long long kPromptMs = 3000; + +https::HttpClientConfig test_config() { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 300; + cfg.readTimeoutMs = 30000; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::GET; + req.url = url; + return req; +} + +long long since_ms(Clock::time_point start) { + return std::chrono::duration_cast(Clock::now() - start).count(); +} + +// Holds a connection for up to 5 s or until `done`. +void hold(const std::atomic& done) { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +} + +// A TCP listener that never speaks TLS: it reads the ClientHello and says nothing. +class Silent { +public: + Silent() : server_([this](proxy_test::Peer& peer) { + peer.read_some(); + hold(done_); + }) {} + ~Silent() { done_ = true; } + + [[nodiscard]] bool failed() const { return server_.failed(); } + [[nodiscard]] std::string url() const { + return "https://127.0.0.1:" + std::to_string(server_.port()) + "/"; + } + [[nodiscard]] std::string proxy_url(const char* scheme) const { + return std::string(scheme) + "://127.0.0.1:" + std::to_string(server_.port()); + } + +private: + std::atomic done_ { false }; + proxy_test::Server server_; +}; + +class HandshakeTimeoutTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +} // namespace + +TEST_F(HandshakeTimeoutTest, SendGivesUpOnAHandshakeThatNeverAnswers) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send(get(peer.url())); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, SendStreamGivesUpOnAHandshakeThatNeverAnswers) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send_stream(get(peer.url()), [](const https::SseEvent&) { return true; }); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A stop token that is never stopped takes the waits in slices; the deadline +// has to hold there as well. +TEST_F(HandshakeTimeoutTest, TheDeadlineHoldsWithAStopTokenAttached) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + std::stop_source never; + const auto start = Clock::now(); + auto res = client.send(get(peer.url()), never.get_token()); + + EXPECT_FALSE(res.cancelled); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, TheHandshakeWithAnHttpsProxyIsBounded) { + Silent proxy; + ASSERT_FALSE(proxy.failed()); + + auto cfg = test_config(); + cfg.proxy = proxy.proxy_url("https"); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_NE(res.statusText.find("proxy: could not open a TLS connection"), std::string::npos) + << res.statusText; + EXPECT_NE(res.statusText.find("TLS handshake timed out"), std::string::npos) + << res.statusText; + EXPECT_LT(since_ms(start), kPromptMs); +} + +// The session to the target runs inside the one to the proxy, which answered +// the CONNECT and is now silent. +TEST_F(HandshakeTimeoutTest, TheHandshakeInsideAnHttpsProxyTunnelIsBounded) { + tls_test::Server proxy([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(proxy.failed()); + + // The proxy's own handshake is real and has to fit in the timeout too. + auto cfg = test_config(); + cfg.connectTimeoutMs = 1000; + cfg.proxy = "https://127.0.0.1:" + std::to_string(proxy.port()); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, TheHandshakeInsideAnHttpProxyTunnelIsBounded) { + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.1 200 Connection established\r\n\r\n"); + peer.read_some(); + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + auto cfg = test_config(); + cfg.proxy = "http://127.0.0.1:" + std::to_string(proxy.port()); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + done = true; + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A peer that sends a byte at a time keeps every read short, so only a limit on +// the handshake as a whole ends it: the record header promises 64 bytes and the +// peer sends fewer than that, one per 100 ms. +TEST_F(HandshakeTimeoutTest, ABytePerReadDoesNotHoldTheHandshakeOpen) { + std::atomic done { false }; + proxy_test::Server peerServer([&](proxy_test::Peer& peer) { + peer.read_some(); + if (!peer.write(std::string("\x16\x03\x03\x00\x40", 5))) return; + for (int i = 0; i < 50 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + if (!peer.write("\x01")) return; + } + }); + ASSERT_FALSE(peerServer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:" + std::to_string(peerServer.port()) + "/")); + done = true; + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A signal whose handler was installed with SA_RESTART restarts a blocking +// `recv`, so it never ended a handshake before the deadline existed. The wait +// that the deadline puts ahead of each `recv` is a `poll`, which a signal ends +// with EINTR whatever the flag; that has to be waited out, not taken for the +// deadline. The signals go to this thread, the one in the handshake. +#ifndef _WIN32 +namespace { +void ignore_signal(int) {} +} + +TEST_F(HandshakeTimeoutTest, ASignalDuringTheHandshakeIsNotATimeout) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + struct sigaction action {}, previous {}; + action.sa_handler = ignore_signal; + action.sa_flags = SA_RESTART; + sigemptyset(&action.sa_mask); + ASSERT_EQ(sigaction(SIGUSR1, &action, &previous), 0); + + auto cfg = test_config(); + cfg.connectTimeoutMs = 1000; + https::HttpClient client(cfg); + const pthread_t self = pthread_self(); + std::atomic done { false }; + std::thread signaller([&] { + for (int i = 0; i < 5 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + if (!done.load()) pthread_kill(self, SIGUSR1); + } + }); + const auto start = Clock::now(); + auto res = client.send(get(peer.url())); + const auto took = since_ms(start); + done = true; + signaller.join(); + sigaction(SIGUSR1, &previous, nullptr); + + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_GE(took, 900) << "the handshake ended at a signal, not at the deadline"; + EXPECT_LT(took, kPromptMs); +} +#endif + +// connectTimeoutMs is for establishing the connection. Once the handshake is +// done a read waits as long as it always did: here 1600 ms of silence after a +// 1000 ms timeout, and then the bytes arrive. The timeout leaves room for a +// real handshake on a slow machine. +TEST_F(HandshakeTimeoutTest, TheDeadlineIsLiftedOnceTheHandshakeIsDone) { + tls_test::Server server([](tls_test::Conn& conn, int) { + std::this_thread::sleep_for(std::chrono::milliseconds(1600)); + conn.write("x"); + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::TlsSocket tls; + ASSERT_TRUE(tls.connect("127.0.0.1", server.port(), 1000, /*verifySsl=*/false)) << tls.error(); + const std::string request = "GET / HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"; + ASSERT_EQ(tls.write(request.data(), static_cast(request.size())), + static_cast(request.size())); + char byte = 0; + auto r = tls.read_some(&byte, 1); + + EXPECT_EQ(r.status, https::ReadStatus::Data); + EXPECT_EQ(byte, 'x'); +} diff --git a/tests/test_pool.cpp b/tests/test_pool.cpp new file mode 100644 index 0000000..995b795 --- /dev/null +++ b/tests/test_pool.cpp @@ -0,0 +1,903 @@ +// What the connection pool does with a response that went wrong. +// +// `tests/` had no keep-alive coverage at all, which is the direct reason the +// regression `#14` introduced (a `Content-Length` branch in `send_stream` that +// stopped marking the connection unusable) went in green. Every test here scripts +// a server, makes two requests through one `HttpClient`, and asserts BOTH what +// the second request received AND **how many TCP connections the server saw**. +// +// The connection count is the assertion that matters. Issue #15's own report +// contains a case whose output is byte-for-byte identical to a correct client's; +// the only observable difference was that the server logged one connection where +// it should have logged two. +#include +#include "tls_test_server.hpp" + +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifndef TINYHTTPS_WINSOCK +#include +#include +#include +#endif + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +https::HttpClientConfig test_config(int readTimeoutMs = 700) { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = readTimeoutMs; + cfg.keepAlive = true; // the configuration under test + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::GET; + req.url = url; + return req; +} + +class PoolTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +} // namespace + +// ── issue #15 · a declared length that stops halfway ───────────────────────── + +// The first response promises 100 bytes, delivers 50, and sends the rest AFTER +// the client has given up on it — which is what a server that is merely slow +// does. The read times out with 50 bytes still owed, and those 50 bytes then +// arrive on a socket nobody is reading. +// +// THE LATE SECOND HALF IS WHAT MAKES THIS A TEST. An earlier form of this +// server stalled and never sent the rest; the connection was then silent rather +// than dirty, and the stale-connection retry rescued the second request whether +// or not the pool guard worked. Verified by mutation: with the guard's +// destructor emptied, that form still passed and this one fails. +TEST_F(PoolTest, AShortContentLengthBodyDoesNotPoisonTheNextRequest) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 100\r\n\r\n"); + conn.write(std::string(50, 'B')); + std::this_thread::sleep_for(std::chrono::milliseconds(1300)); + conn.write(std::string(50, 'B')); // arrives after the read timeout + return true; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + auto first = client.send(get(server.url("/one"))); + EXPECT_EQ(first.statusCode, 200); + EXPECT_EQ(first.body, std::string(50, 'B')); + EXPECT_FALSE(first.bodyComplete) + << "50 of 100 bytes arrived; the caller has no other way to learn that"; + EXPECT_FALSE(first.bodyError.empty()); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 200) << "status text was: " << second.statusText; + EXPECT_EQ(second.body, "second"); + EXPECT_TRUE(second.bodyComplete); + + // THE ASSERTION THE BYTES CANNOT MAKE. + EXPECT_EQ(server.accepts(), 2) + << "the truncated connection was reused instead of being dropped"; +} + +TEST_F(PoolTest, AChunkedBodyThatStopsMidChunkDoesNotPoisonTheNextRequest) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n"); + conn.write("10\r\n"); // 16 bytes announced + conn.write("0123456"); // 7 delivered now + std::this_thread::sleep_for(std::chrono::milliseconds(1300)); + conn.write("789abcdef\r\n0\r\n\r\n"); // the rest, too late + return true; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + auto first = client.send(get(server.url("/one"))); + EXPECT_EQ(first.statusCode, 200); + EXPECT_FALSE(first.bodyComplete); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 200) << "status text was: " << second.statusText; + EXPECT_EQ(second.body, "second"); + EXPECT_EQ(server.accepts(), 2); +} + +// A chunk header that does not parse is not a terminal chunk. `parse_hex` +// returned 0 for `zz` — the same value a real `0\r\n` gives — so the reader +// declared the body finished and returned the socket with the rest still on it. +TEST_F(PoolTest, AnUnparsableChunkHeaderIsAnErrorAndNotAnEndOfBody) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n"); + conn.write("4\r\nabcd\r\n"); + conn.write("zz\r\n"); + // Whatever follows a header the client could not parse is still on + // the wire, and a reused connection would read it as a status line. + conn.write("4\r\nefgh\r\n0\r\n\r\n"); + return true; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + auto first = client.send(get(server.url("/one"))); + EXPECT_EQ(first.statusCode, 200); + EXPECT_EQ(first.body, "abcd"); + EXPECT_FALSE(first.bodyComplete) << "`zz` was accepted as a terminal chunk"; + EXPECT_NE(first.bodyError.find("zz"), std::string::npos); + // The server's real reason phrase survives; it used to be overwritten with + // the parse error, so a caller logging statusText on a 200 saw the error. + EXPECT_EQ(first.statusText, "OK"); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 200); + EXPECT_EQ(server.accepts(), 2); +} + +// A `Content-Length` past the width of an `int`. This was the one pool defect a +// server could trigger with a single header and no timing at all: the value was +// assigned to an `int`, 2^32 truncated to 0, the reader took that for an empty +// body and returned a socket with four gigabytes owed on it. +TEST_F(PoolTest, AContentLengthPastThirtyTwoBitsIsNotSilentlyZero) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 4294967296\r\n\r\n"); + conn.write("0123456789"); + std::this_thread::sleep_for(std::chrono::milliseconds(1300)); + conn.write("late-and-unread"); // still owed, and still arriving + return true; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + // The size cap is lifted here on purpose. With the default 64 MiB it would + // refuse this response before reading a byte — correct, and a different + // mechanism from the one under test. Lifting it puts the reader back on the + // path where the truncation used to happen, so what this test observes is + // the width of the field and nothing else. + // (`ABodyLargerThanTheConfiguredLimitIsRefused` covers the cap.) + auto cfg = test_config(); + // Parenthesised: , which the test servers include on Windows, + // defines a `max` macro. + cfg.maxResponseBodyBytes = (std::numeric_limits::max)(); + https::HttpClient client(cfg); + + auto first = client.send(get(server.url("/one"))); + EXPECT_EQ(first.statusCode, 200); + EXPECT_EQ(first.body, "0123456789") + << "the length truncated to zero, so the reader took the body for empty"; + EXPECT_FALSE(first.bodyComplete); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 200) << "status text was: " << second.statusText; + EXPECT_EQ(second.body, "second"); + EXPECT_EQ(server.accepts(), 2); +} + +// Leftover bytes must not be readable as a status line. This is the exact input +// from issue #15, where `BBBB…` produced `statusCode = 999`: the parser took the +// first run of digits it found and never checked that the line began with +// `HTTP/`. +TEST_F(PoolTest, AStatusLineThatIsNotOneIsRejectedRatherThanMined) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("BBBB 999 XHTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n"); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto response = client.send(get(server.url("/"))); + EXPECT_EQ(response.statusCode, 0) << "a body fragment was read as a status code"; + EXPECT_EQ(response.statusText, "Invalid status line"); +} + +// ── the other half: a clean connection must still be reused ────────────────── +// +// Without this, "drop everything, always" would pass every test above. The two +// assertions are a pair. +TEST_F(PoolTest, TwoCleanRequestsShareOneConnection) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + conn.write(tls_test::ok_response(index == 0 ? "first" : "second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + auto first = client.send(get(server.url("/one"))); + EXPECT_EQ(first.body, "first"); + EXPECT_TRUE(first.bodyComplete); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.body, "second"); + EXPECT_TRUE(second.bodyComplete); + + EXPECT_EQ(server.accepts(), 1) << "keep-alive stopped reusing connections"; +} + +TEST_F(PoolTest, KeepAliveOffDoesNotLeaveAConnectionInThePool) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + conn.write(tls_test::ok_response(index == 0 ? "first" : "second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + auto cfg = test_config(); + cfg.keepAlive = false; + https::HttpClient client(cfg); + + EXPECT_EQ(client.send(get(server.url("/one"))).body, "first"); + EXPECT_EQ(client.send(get(server.url("/two"))).body, "second"); + // The request said `Connection: close`; a server that answers without + // echoing it used to leave the socket pooled anyway. + EXPECT_EQ(server.accepts(), 2); +} + +// ── redirects and error bodies: the paths with no timeout and no truncation ── + +// A 302 that carries a body — which is what nearly every server sends — and +// keep-alive. The code here promised in a comment to "drain any body to keep +// connection clean" and drained nothing, so the recursive call below picked up a +// socket with the redirect's own HTML still on it. +TEST_F(PoolTest, ARedirectBodyIsConsumedBeforeTheConnectionIsReused) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 302 Found\r\nLocation: /final\r\n" + "Content-Length: 27\r\n\r\n" + "moved, see you"); // 27 bytes exactly + return true; + } + conn.write(tls_test::ok_response("REDIRECTED-CONTENT")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto response = client.send(get(server.url("/start"))); + + EXPECT_EQ(response.statusCode, 200) << "status text was: " << response.statusText; + EXPECT_EQ(response.body, "REDIRECTED-CONTENT") + << "the redirect's own body was read as the final response"; + EXPECT_TRUE(response.bodyComplete); +} + +TEST_F(PoolTest, ARedirectedDownloadReusesTheConnectionAfterDrainingTheBody) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 302 Found\r\nLocation: /final\r\n" + "Content-Length: 12\r\n\r\n" + "moved-along\n"); + return true; + } + conn.write(tls_test::ok_response("REDIRECTED-FILE")); + return true; + }); + ASSERT_FALSE(server.failed()); + + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_test"; + std::filesystem::create_directories(dir); + auto dest = dir / "redirected.bin"; + + https::HttpClient client(test_config()); + auto result = client.download_to_file(server.url("/start"), dest); + + ASSERT_TRUE(result.ok()) << "error: " << result.error; + std::ifstream in(dest, std::ios::binary); + std::string content((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); + EXPECT_EQ(content, "REDIRECTED-FILE"); + + // A short 3xx body is drained rather than thrown away with the connection, + // so the redirect costs no second handshake. Correctness above does not + // depend on this number; reuse does. + EXPECT_EQ(server.accepts(), 1); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} + +// A 404's body is never handed to the caller of download_to_file, and used to be +// left on the socket for whatever asked next. +TEST_F(PoolTest, AnErrorBodyIsConsumedBeforeTheConnectionIsReused) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 404 Not Found\r\nContent-Length: 9\r\n\r\n" + "not found"); + return true; + } + conn.write(tls_test::ok_response("after-404")); + return true; + }); + ASSERT_FALSE(server.failed()); + + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_404"; + std::filesystem::create_directories(dir); + + https::HttpClient client(test_config()); + auto failed = client.download_to_file(server.url("/missing"), dir / "x.bin"); + EXPECT_FALSE(failed.ok()); + EXPECT_EQ(failed.statusCode, 404); + EXPECT_EQ(failed.error, "HTTP 404"); + + auto next = client.send(get(server.url("/present"))); + EXPECT_EQ(next.statusCode, 200) << "status text was: " << next.statusText; + EXPECT_EQ(next.body, "after-404") + << "the 404's body was read as the next response"; + EXPECT_EQ(server.accepts(), 1); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} + +// ── a pooled connection the server has since closed ────────────────────────── + +// The scenario issue #15 mentions in passing and that none of the fixes above +// address: the server closes an idle keep-alive connection, which is routine. +// The client cannot see that until it writes, and the write succeeds into a +// local buffer. Without a retry the caller gets `statusCode = 0` / +// "No response" for a request the server never saw. +TEST_F(PoolTest, AConnectionClosedWhileIdleIsRetriedRatherThanReported) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + conn.write(tls_test::ok_response(index == 0 ? "first" : "second")); + return false; // answer, then close — as an idle-timeout would + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + auto first = client.send(get(server.url("/one"))); + EXPECT_EQ(first.statusCode, 200); + EXPECT_EQ(first.body, "first"); + + // Let the close reach us, so the pooled socket is genuinely dead. + std::this_thread::sleep_for(std::chrono::milliseconds(250)); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 200) + << "status text was: " << second.statusText + << " — a dead pooled connection was reported instead of replaced"; + EXPECT_EQ(second.body, "second"); + EXPECT_EQ(server.accepts(), 2); +} + +// THE OTHER HALF OF THE RETRY, AND THE ONE THAT KEEPS IT SAFE. +// +// The window closes the moment a byte arrives: past that the server has seen the +// request, and sending it again could repeat whatever it did. Here the server +// answers with a status line it never finishes and then closes — a failure, but +// one that must NOT be retried. +TEST_F(PoolTest, ARequestTheServerHasBegunAnsweringIsNotResent) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200"); // a head that never completes + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto response = client.send(get(server.url("/once"))); + + EXPECT_EQ(response.statusCode, 0); + EXPECT_EQ(server.requests(), 1) + << "a request the server had begun answering was sent a second time"; + EXPECT_EQ(server.accepts(), 1); +} + +TEST_F(PoolTest, TheRetryCanBeTurnedOff) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + conn.write(tls_test::ok_response(index == 0 ? "first" : "second")); + return false; + }); + ASSERT_FALSE(server.failed()); + + auto cfg = test_config(); + cfg.retryOnStaleConnection = false; + https::HttpClient client(cfg); + + EXPECT_EQ(client.send(get(server.url("/one"))).statusCode, 200); + std::this_thread::sleep_for(std::chrono::milliseconds(250)); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 0) + << "opting out of the retry must actually opt out"; +} + +// ── streaming ──────────────────────────────────────────────────────────────── + +// The regression `#14` introduced. `send_stream` gained a `Content-Length` +// branch whose two exits — a read timeout and an end of stream — left the +// connection marked reusable, which the branch they replaced never did. +TEST_F(PoolTest, AStreamCutShortDoesNotPoisonTheNextRequest) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n" + "Content-Length: 200\r\n\r\n"); + conn.write("data: one\n\n"); + conn.park(); + return false; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + int events = 0; + auto first = client.send_stream(get(server.url("/stream")), + [&](const https::SseEvent&) { ++events; return true; }); + EXPECT_EQ(first.statusCode, 200); + EXPECT_EQ(events, 1); + EXPECT_FALSE(first.bodyComplete); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 200) << "status text was: " << second.statusText; + EXPECT_EQ(second.body, "second"); + EXPECT_EQ(server.accepts(), 2); +} + +// A callback that stops early leaves bytes owed, so the connection cannot be +// reused either — but it is not an error, and the status line is untouched. +TEST_F(PoolTest, AStreamStoppedByItsCallbackReleasesTheConnection) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n"); + for (int i = 0; i < 20; ++i) conn.write("9\r\ndata: x\n\n\r\n"); + conn.park(); + return false; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + int events = 0; + auto first = client.send_stream(get(server.url("/stream")), + [&](const https::SseEvent&) { ++events; return false; }); + EXPECT_EQ(first.statusCode, 200); + EXPECT_EQ(first.statusText, "OK"); + EXPECT_EQ(events, 1); + EXPECT_FALSE(first.bodyComplete); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 200) << "status text was: " << second.statusText; + EXPECT_EQ(server.accepts(), 2); +} + +// ── allocation bounded by configuration rather than by a header ────────────── + +TEST_F(PoolTest, ABodyLargerThanTheConfiguredLimitIsRefused) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 100000\r\n\r\n"); + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + auto cfg = test_config(); + cfg.maxResponseBodyBytes = 1024; + https::HttpClient client(cfg); + + auto response = client.send(get(server.url("/big"))); + EXPECT_EQ(response.statusCode, 200); + EXPECT_TRUE(response.body.empty()) << "refused before reading, not after"; + EXPECT_FALSE(response.bodyComplete); + EXPECT_NE(response.bodyError.find("limit"), std::string::npos); +} + +// ── a body a HEAD response does not have ───────────────────────────────────── + +TEST_F(PoolTest, AHeadResponseDoesNotConsumeTheDeclaredLength) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + // A HEAD answer carries the length a GET would have returned, and + // no body. Reading it would consume the next response instead. + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 500\r\n\r\n"); + return true; + } + conn.write(tls_test::ok_response("after-head")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + https::HttpRequest head; + head.method = https::Method::HEAD; + head.url = server.url("/thing"); + auto headResponse = client.send(head); + EXPECT_EQ(headResponse.statusCode, 200); + EXPECT_TRUE(headResponse.body.empty()); + EXPECT_TRUE(headResponse.bodyComplete); + + auto next = client.send(get(server.url("/thing"))); + EXPECT_EQ(next.body, "after-head"); + EXPECT_EQ(server.accepts(), 1); +} + +TEST_F(PoolTest, A204CarriesNoBodyWhateverItsHeadersSay) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 204 No Content\r\nContent-Length: 42\r\n\r\n"); + return true; + } + conn.write(tls_test::ok_response("after-204")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto empty = client.send(get(server.url("/nothing"))); + EXPECT_EQ(empty.statusCode, 204); + EXPECT_TRUE(empty.body.empty()); + EXPECT_TRUE(empty.bodyComplete); + + EXPECT_EQ(client.send(get(server.url("/next"))).body, "after-204"); + EXPECT_EQ(server.accepts(), 1); +} + +// ── a body whose framing is the close, ended the way servers actually end it ─ + +// No Content-Length and no chunked encoding: the body ends when the connection +// does (RFC 9112 §6.3). The server then closes WITHOUT a TLS `close_notify`, +// which is what most servers do. +// +// The BIO under this client used to answer that FIN with +// MBEDTLS_ERR_NET_CONN_RESET — mbedtls's own BIO passes the zero through, and +// `ssl_fetch_input` tests for exactly that zero to produce SSL_CONN_EOF — so an +// orderly end of stream reached the reader as an error and a complete body read +// as truncated. +TEST_F(PoolTest, ABodyDelimitedByTheCloseIsCompleteWithoutACloseNotify) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\n"); + conn.write("connection-delimited-body"); + conn.close_hard(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto response = client.send(get(server.url("/"))); + + EXPECT_EQ(response.statusCode, 200); + EXPECT_EQ(response.body, "connection-delimited-body"); + EXPECT_TRUE(response.bodyComplete) << "bodyError was: " << response.bodyError; + EXPECT_TRUE(response.bodyError.empty()); +} + +// The same framing through `download_to_file`, where mistaking the close for an +// error is worse than cosmetic: it sets `result.error`, and `ok()` consults it. +TEST_F(PoolTest, ADownloadDelimitedByTheCloseReportsSuccess) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Type: application/octet-stream\r\n\r\n"); + conn.write("0123456789abcdef"); + conn.close_hard(); + return false; + }); + ASSERT_FALSE(server.failed()); + + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_close"; + std::filesystem::create_directories(dir); + auto dest = dir / "closed.bin"; + + https::HttpClient client(test_config()); + auto result = client.download_to_file(server.url("/blob"), dest); + + EXPECT_TRUE(result.ok()) + << "a file that arrived complete was reported as failed: " << result.error; + EXPECT_EQ(result.bytesWritten, 16); + EXPECT_EQ(std::filesystem::file_size(dest), 16u); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} + +// ── a destination that refuses the bytes ───────────────────────────────────── + +// `/dev/full` accepts an open and answers every write with ENOSPC, which is what +// a full disk does at the moment it fills — without needing one. +// +// Before 0.3.2 the transfer "succeeded": `ofs.write` was never checked, the +// count came from the network, and `bytesWritten` said the whole body when the +// file held none of it. The caller then blamed the source (a checksum mismatch) +// for what was the local disk. Verified by mutation: with the stream check +// removed, `ok()` is true and `bytesWritten` is the body's length. +#ifdef __linux__ +TEST_F(PoolTest, ADownloadIntoAFullDeviceFailsAsALocalWriteAndCountsNothing) { + if (!std::filesystem::exists("/dev/full")) GTEST_SKIP() << "no /dev/full"; + + const std::string payload(64 * 1024, 'F'); + tls_test::Server server([payload](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response(payload)); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto result = client.download_to_file(server.url("/big"), "/dev/full"); + + EXPECT_FALSE(result.ok()) << "a file that took nothing was reported as downloaded"; + EXPECT_TRUE(result.writeFailed) + << "the caller must be able to tell the disk from the source; error: " + << result.error; + EXPECT_EQ(result.statusCode, 200); + EXPECT_EQ(result.bytesWritten, 0); + EXPECT_GT(result.bytesReceived, 0); + EXPECT_LT(result.bytesReceived, static_cast(payload.size())) + << "the transfer read on after the first refused write"; + EXPECT_EQ(result.error.rfind("write /dev/full: ", 0), 0u) << result.error; + EXPECT_NE(result.error.find( + std::make_error_code(std::errc::no_space_on_device).message()), + std::string::npos) + << result.error; + + // The rest of the body was left on the socket, so the connection is not + // handed to the next request. The next request also shows the client is + // not left in a state where a good destination fails too. + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_full"; + std::filesystem::create_directories(dir); + auto dest = dir / "ok.bin"; + auto good = client.download_to_file(server.url("/big"), dest); + EXPECT_TRUE(good.ok()) << "error: " << good.error; + EXPECT_FALSE(good.writeFailed); + EXPECT_EQ(good.bytesWritten, static_cast(payload.size())); + EXPECT_EQ(good.bytesReceived, good.bytesWritten); + EXPECT_EQ(server.accepts(), 2) + << "the connection with an unread body was reused"; + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} +#endif + +// A destination that cannot be opened is the same kind of failure: nothing was +// wrong with what the server sent. +TEST_F(PoolTest, ADestinationThatCannotBeOpenedIsALocalFailure) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("payload")); + return true; + }); + ASSERT_FALSE(server.failed()); + + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_noopen"; + std::filesystem::create_directories(dir); + { + // A regular file where a directory is needed: no parent can be made. + std::ofstream blocker(dir / "blocker", std::ios::binary); + blocker << "not a directory"; + } + + https::HttpClient client(test_config()); + auto result = client.download_to_file(server.url("/x"), + dir / "blocker" / "out.bin"); + + EXPECT_FALSE(result.ok()); + EXPECT_TRUE(result.writeFailed) << "error: " << result.error; + EXPECT_EQ(result.error.rfind("Cannot open file: ", 0), 0u) << result.error; + EXPECT_EQ(result.bytesWritten, 0); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} + +// ── interim responses ──────────────────────────────────────────────────────── + +// A 1xx is not the answer. RFC 9112 §2.1 requires a client to read past one or +// more of them; stopping at one reports `103 Early Hints` as the result AND — +// because a 1xx carries no body — marks the connection clean while the real +// response is still sitting on it, for the next request to read as its own. +TEST_F(PoolTest, AnInterimResponseIsNotMistakenForTheAnswer) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write("HTTP/1.1 103 Early Hints\r\n" + "Link: ; rel=preload\r\n\r\n"); + conn.write(tls_test::ok_response("the-real-answer")); + return true; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + + auto first = client.send(get(server.url("/one"))); + EXPECT_EQ(first.statusCode, 200) << "statusText was: " << first.statusText; + EXPECT_EQ(first.body, "the-real-answer"); + EXPECT_TRUE(first.bodyComplete); + // The interim response's fields describe the interim response, not this one. + EXPECT_EQ(first.headers.count("Link"), 0u); + + auto second = client.send(get(server.url("/two"))); + EXPECT_EQ(second.statusCode, 200) << "statusText was: " << second.statusText; + EXPECT_EQ(second.body, "second") + << "the real response was left on the socket and read by the next request"; + EXPECT_EQ(server.accepts(), 1); +} + +// More than one, which the grammar allows and the loop has to survive. A +// `100 Continue` is what any server sends for a request carrying +// `Expect: 100-continue`, so a caller can provoke this with one header. +TEST_F(PoolTest, SeveralInterimResponsesInARowAreAllSkipped) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 100 Continue\r\n\r\n"); + conn.write("HTTP/1.1 103 Early Hints\r\nLink: ; rel=preload\r\n\r\n"); + conn.write("HTTP/1.1 103 Early Hints\r\nLink: ; rel=preload\r\n\r\n"); + conn.write(tls_test::ok_response("finally")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto response = client.send(get(server.url("/"))); + EXPECT_EQ(response.statusCode, 200) << "statusText was: " << response.statusText; + EXPECT_EQ(response.body, "finally"); + EXPECT_TRUE(response.bodyComplete); +} + +// ── how many lines, which is a different bound from how long each one is ───── + +// A header block is accumulated into a map, so an endless supply of short, +// well-formed header lines grows the client's memory without limit — and every +// line resets the read timeout, so nothing else stops it. +TEST_F(PoolTest, AnEndlessHeaderBlockIsRefusedRatherThanAccumulated) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\n"); + for (int i = 0; i < 400; ++i) { + conn.write("X-Filler-" + std::to_string(i) + ": aaaaaaaaaaaaaaaa\r\n"); + } + conn.write("\r\n"); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto response = client.send(get(server.url("/"))); + EXPECT_EQ(response.statusCode, 0); + EXPECT_NE(response.statusText.find("Header block"), std::string::npos) + << "statusText was: " << response.statusText; +} + +// The trailer section is discarded rather than kept, so it costs no memory — +// but an endless supply of trailers holds the call open just as long. +TEST_F(PoolTest, AnEndlessTrailerSectionIsRefused) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n"); + conn.write("4\r\nabcd\r\n"); + conn.write("0\r\n"); + for (int i = 0; i < 400; ++i) { + conn.write("X-Trailer-" + std::to_string(i) + ": v\r\n"); + } + conn.write("\r\n"); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto response = client.send(get(server.url("/"))); + EXPECT_EQ(response.statusCode, 200); + EXPECT_EQ(response.body, "abcd") << "the body itself was well formed"; + EXPECT_FALSE(response.bodyComplete); + EXPECT_NE(response.bodyError.find("Trailer"), std::string::npos) + << "bodyError was: " << response.bodyError; +} + +// ── issue #16 · SIGPIPE ────────────────────────────────────────────────────── + +#ifndef TINYHTTPS_WINSOCK +// THE TEST HAS TO FORK, AND THE REASON IS ITSELF EVIDENCE. +// +// `mbedtls_net_bind` calls `net_prepare`, which does `signal(SIGPIPE, SIG_IGN)` +// (net_sockets.c:114) — so merely having a test server in this binary disarms +// the signal for the whole process, and an in-process check would pass whether +// or not the library was fixed. That process-wide disarming is also precisely +// what tinyhttps loses by replacing mbedtls's network layer with its own BIO, +// which is why issue #16 exists at all. +// +// The child restores the default disposition, connects, and writes to a peer +// that has gone away. Unfixed, it dies with signal 13 and exit status 141 — +// which is what the reporter saw. +TEST(SigPipe, WritingToADepartedPeerReturnsInsteadOfKillingTheProcess) { + int listener = ::socket(AF_INET, SOCK_STREAM, 0); + ASSERT_GE(listener, 0); + int reuse = 1; + ::setsockopt(listener, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof reuse); + + sockaddr_in addr {}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = 0; + ASSERT_EQ(::bind(listener, reinterpret_cast(&addr), sizeof addr), 0); + ASSERT_EQ(::listen(listener, 4), 0); + + socklen_t len = sizeof addr; + ASSERT_EQ(::getsockname(listener, reinterpret_cast(&addr), &len), 0); + const int port = ntohs(addr.sin_port); + + const pid_t child = ::fork(); + ASSERT_GE(child, 0); + + if (child == 0) { + ::close(listener); + // Undo whatever else in this binary disarmed it. This is the disposition + // a program that has never thought about SIGPIPE has. + ::signal(SIGPIPE, SIG_DFL); + + mcpplibs::tinyhttps::Socket sock; + if (!sock.connect("127.0.0.1", port, 4000)) _exit(3); + + // Give the parent time to accept and slam the connection shut. + ::usleep(300 * 1000); + + // THE LOOP DOES NOT STOP AT THE FIRST FAILURE, AND THAT IS THE WHOLE + // POINT. A write to a socket that has just received an RST returns + // ECONNRESET and raises nothing; it is the write AFTER that one which + // gets EPIPE, and EPIPE is what SIGPIPE accompanies. A loop that broke + // on the first non-positive return would exit cleanly whether or not + // MSG_NOSIGNAL was there — measured: it did. + const std::string payload(4096, 'x'); + int last = 1; + for (int i = 0; i < 8; ++i) { + last = sock.write(payload.data(), static_cast(payload.size())); + ::usleep(20 * 1000); + } + _exit(last <= 0 ? 0 : 4); // 0 = the write reported failure, as it must + } + + sockaddr_in peer {}; + socklen_t peerLen = sizeof peer; + const int accepted = ::accept(listener, reinterpret_cast(&peer), &peerLen); + if (accepted >= 0) { + // Zero linger turns the close into an RST, so the child's next write + // meets a connection that is not merely half-closed but gone. + struct linger lg { 1, 0 }; + ::setsockopt(accepted, SOL_SOCKET, SO_LINGER, &lg, sizeof lg); + ::close(accepted); + } + ::close(listener); + + int status = 0; + ASSERT_EQ(::waitpid(child, &status, 0), child); + + ASSERT_FALSE(WIFSIGNALED(status)) + << "the child was killed by signal " << WTERMSIG(status) + << " (13 is SIGPIPE, which reaches a shell as exit status 141) — " + "Socket::write is sending without MSG_NOSIGNAL"; + EXPECT_EQ(WEXITSTATUS(status), 0) + << "child exit code 3 = could not connect, 4 = the write never failed"; +} +#endif // !TINYHTTPS_WINSOCK diff --git a/tests/test_proxy.cpp b/tests/test_proxy.cpp new file mode 100644 index 0000000..ca1d92f --- /dev/null +++ b/tests/test_proxy.cpp @@ -0,0 +1,758 @@ +// What the client does when it is told to go through a proxy. +// +// Each test stands up a proxy that behaves as the test scripts it and the TLS +// server from tls_test_server.hpp as the target, and asserts both what the +// caller got and what the proxy was sent. The second is the half a mock-free +// test cannot see: whether the CONNECT line named the right target, and what +// the Proxy-Authorization header said. +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +class ProxyTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +https::HttpClientConfig proxied(const std::string& proxyUrl) { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + cfg.proxy = proxyUrl; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.url = url; + return req; +} + +tls_test::Server hello_server() { + return tls_test::Server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("hello")); + return true; + }); +} + +// The value of a header in a request head, or "" when absent. +std::string header_of(const std::string& head, std::string_view name) { + std::string lower, key = std::string(name) + ":"; + for (char c : head) lower += static_cast(std::tolower(static_cast(c))); + for (char& c : key) c = static_cast(std::tolower(static_cast(c))); + auto pos = lower.find(key); + if (pos == std::string::npos) return {}; + pos += key.size(); + while (pos < head.size() && head[pos] == ' ') ++pos; + return head.substr(pos, head.find("\r\n", pos) - pos); +} + +constexpr std::string_view kEstablished = "HTTP/1.1 200 Connection established\r\n\r\n"; + +// A CONNECT proxy. With `expectAuth` set it answers 407 unless the request +// carries exactly that Proxy-Authorization value. Either way it records the +// request head, so the test can read back what the client sent. +proxy_test::Server::Handler connect_handler(int targetPort, std::string expectAuth = {}) { + return [=](proxy_test::Peer& peer) { + auto head = peer.read_head(); + peer.note(head); + if (head.empty()) return; + if (!expectAuth.empty() && header_of(head, "Proxy-Authorization") != expectAuth) { + peer.write("HTTP/1.1 407 Proxy Authentication Required\r\n" + "Proxy-Authenticate: Basic realm=\"test\"\r\n" + "Content-Length: 0\r\n\r\n"); + return; + } + peer.write(kEstablished); + peer.relay_to(targetPort); + }; +} + +std::string proxy_url(const proxy_test::Server& proxy, std::string_view userinfo = {}) { + std::string url = "http://"; + if (!userinfo.empty()) { url += userinfo; url += "@"; } + return url + "127.0.0.1:" + std::to_string(proxy.port()); +} + +} // namespace + +// ── parse_proxy_url ────────────────────────────────────────────────────────── + +TEST(ProxyUrl, BareHostAndPortIsAnHttpProxy) { + auto p = https::parse_proxy_url("proxy.example:3128"); + EXPECT_EQ(p.scheme, "http"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); + EXPECT_FALSE(p.hasCredentials); +} + +TEST(ProxyUrl, ThePortDefaultsAsItAlwaysHas) { + EXPECT_EQ(https::parse_proxy_url("http://proxy.example").port, 8080); +} + +TEST(ProxyUrl, APathQueryOrFragmentAfterTheAuthorityIsIgnored) { + auto p = https::parse_proxy_url("http://proxy.example:3128/some/path?x=1#f"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, UserAndPasswordAreSplitAndPercentDecoded) { + auto p = https::parse_proxy_url("http://al%40ice:p%3Ass%20w@proxy.example:3128"); + EXPECT_TRUE(p.hasCredentials); + EXPECT_EQ(p.user, "al@ice"); + EXPECT_EQ(p.password, "p:ss w"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, OnlyTheFirstColonSplitsUserFromPassword) { + auto p = https::parse_proxy_url("http://u:a:b@proxy.example:1"); + EXPECT_EQ(p.user, "u"); + EXPECT_EQ(p.password, "a:b"); +} + +TEST(ProxyUrl, AnUnencodedAtSignInThePasswordStaysInThePassword) { + auto p = https::parse_proxy_url("http://u:p@ss@proxy.example:1"); + EXPECT_EQ(p.password, "p@ss"); + EXPECT_EQ(p.host, "proxy.example"); +} + +TEST(ProxyUrl, AUserWithNoPasswordStillHasCredentials) { + auto p = https::parse_proxy_url("http://token@proxy.example:1"); + EXPECT_TRUE(p.hasCredentials); + EXPECT_EQ(p.user, "token"); + EXPECT_EQ(p.password, ""); +} + +TEST(ProxyUrl, ABadPercentEscapeIsLeftAsWritten) { + auto p = https::parse_proxy_url("http://u:50%@proxy.example:1"); + EXPECT_EQ(p.password, "50%"); +} + +TEST(ProxyUrl, AnIpv6LiteralKeepsItsColons) { + auto p = https::parse_proxy_url("http://[::1]:3128"); + EXPECT_EQ(p.host, "::1"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, TheSchemeIsLowerCased) { + EXPECT_EQ(https::parse_proxy_url("HTTP://proxy.example:1").scheme, "http"); +} + +TEST(ProxyUrl, APortOutOfRangeIsZeroRatherThanTruncated) { + EXPECT_EQ(https::parse_proxy_url("http://proxy.example:99999").port, 0); + EXPECT_EQ(https::parse_proxy_url("http://proxy.example:80x").port, 0); +} + +// ── CONNECT ────────────────────────────────────────────────────────────────── + +TEST_F(ProxyTest, ARequestGoesThroughAProxyThatAsksForNothing) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto notes = proxy.notes(); + ASSERT_EQ(notes.size(), 1u); + EXPECT_TRUE(notes[0].starts_with("CONNECT 127.0.0.1:" + std::to_string(target.port()) + " HTTP/1.1\r\n")); + EXPECT_EQ(header_of(notes[0], "Proxy-Authorization"), ""); +} + +TEST_F(ProxyTest, CredentialsInTheUrlBecomeABasicProxyAuthorization) { + auto target = hello_server(); + // base64("alice:s3cret") + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy, "alice:s3cret"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); +} + +TEST_F(ProxyTest, CredentialsArePercentDecodedBeforeTheyAreEncoded) { + auto target = hello_server(); + // base64("al@ice:p:ss w") + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxAaWNlOnA6c3Mgdw==")); + + https::HttpClient client(proxied(proxy_url(proxy, "al%40ice:p%3Ass%20w"))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); +} + +TEST_F(ProxyTest, WrongCredentialsAreReportedAsThatAndNotAsAFailedConnection) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy, "alice:wrong"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); + EXPECT_NE(resp.statusText.find("407"), std::string::npos); + EXPECT_EQ(target.accepts(), 0); // the tunnel was never opened +} + +TEST_F(ProxyTest, AProxyThatWantsCredentialsWeDidNotGiveSaysSo) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("requires credentials"), std::string::npos); +} + +TEST_F(ProxyTest, AnyOtherRefusalCarriesTheProxysStatus) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\n\r\n"); + }); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("refused CONNECT: 403 Forbidden"), std::string::npos); +} + +TEST_F(ProxyTest, AProxyThatHangsUpWithoutAnsweringIsAnError) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { peer.read_head(); }); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("no HTTP response"), std::string::npos); +} + +TEST_F(ProxyTest, AnyTwoHundredSeriesAnswerOpensTheTunnel) { + auto target = hello_server(); + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.0 204 No Content\r\nX-Note: a header or two\r\n\r\n"); + peer.relay_to(target.port()); + }); + + https::HttpClient client(proxied(proxy_url(proxy))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); +} + +TEST_F(ProxyTest, TwoRequestsShareOneTunnel) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + + https::HttpClient client(proxied(proxy_url(proxy))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} + +TEST_F(ProxyTest, ASchemeThisLibraryDoesNotSpeakIsNotQuietlyTreatedAsHttp) { + https::HttpClient client(proxied("socks4://127.0.0.1:9")); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("unsupported scheme 'socks4'"), std::string::npos); +} + +TEST_F(ProxyTest, TheOriginalProxyConnectStillReturnsATunnel) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + + auto tunnel = https::proxy_connect("127.0.0.1", proxy.port(), "127.0.0.1", + target.port(), 4000); + EXPECT_TRUE(tunnel.is_valid()); +} + +// ── an https:// proxy ──────────────────────────────────────────────────────── +// +// The proxy here is the TLS test server: the client opens TLS to it, sends +// CONNECT inside that, and then opens a second TLS session to the target inside +// the tunnel. Two sessions on one descriptor is what these tests exist to +// exercise, and the cases that matter are the ones where the two disagree about +// what is readable: a body larger than a record, and a second request that has +// to find the tunnel already open. + +namespace { + +// A recorder for what the TLS proxy saw. The handler runs on the server's +// threads; the test reads it after the response is in. +struct Seen { + std::mutex mutex; + std::vector heads; + void add(std::string head) { + std::lock_guard lock(mutex); + heads.push_back(std::move(head)); + } + std::vector get() { + std::lock_guard lock(mutex); + return heads; + } +}; + +tls_test::Server tls_connect_proxy(Seen& seen, int targetPort, std::string expectAuth = {}, + std::chrono::milliseconds gather = std::chrono::milliseconds::zero()) { + return tls_test::Server([&seen, targetPort, expectAuth, gather](tls_test::Conn& conn, int) { + seen.add(conn.request()); + if (!expectAuth.empty() + && header_of(conn.request(), "Proxy-Authorization") != expectAuth) { + conn.write("HTTP/1.1 407 Proxy Authentication Required\r\n" + "Proxy-Authenticate: Basic realm=\"test\"\r\n" + "Content-Length: 0\r\n\r\n"); + return false; + } + conn.write(kEstablished); + conn.relay_to(targetPort, gather); + return false; + }); +} + +} // namespace + +namespace { + +// Makes one certificate the only CA bundle the client reads, for the lifetime of +// the object, and restores what was there before. +class TrustOnly { +public: + explicit TrustOnly(std::string_view pem) + : path_(std::filesystem::temp_directory_path() + / ("tinyhttps-proxy-trust-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + ".pem")) { + if (const char* old = std::getenv("SSL_CERT_FILE")) { hadOld_ = true; old_ = old; } + std::ofstream(path_, std::ios::binary) << pem; + set(path_.string().c_str()); + } + ~TrustOnly() { + set(hadOld_ ? old_.c_str() : ""); + std::error_code ec; + std::filesystem::remove(path_, ec); + } + TrustOnly(const TrustOnly&) = delete; + TrustOnly& operator=(const TrustOnly&) = delete; + +private: + static void set(const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s("SSL_CERT_FILE", value); +#else + ::setenv("SSL_CERT_FILE", value, 1); +#endif + } + std::filesystem::path path_; + std::string old_; + bool hadOld_ { false }; +}; + +} // namespace + +TEST(ProxyUrl, AnHttpsProxyDefaultsToPort443) { + auto p = https::parse_proxy_url("https://proxy.example"); + EXPECT_EQ(p.scheme, "https"); + EXPECT_EQ(p.port, 443); + EXPECT_EQ(https::parse_proxy_url("https://proxy.example:8443").port, 8443); +} + +TEST_F(ProxyTest, ARequestGoesThroughAnHttpsProxy) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto heads = seen.get(); + ASSERT_EQ(heads.size(), 1u); + EXPECT_TRUE(heads[0].starts_with("CONNECT 127.0.0.1:" + std::to_string(target.port()) + " HTTP/1.1\r\n")); +} + +// The proxy hop is verified like the target hop, and its refusal names the hop +// and the reason. Only the wrong-name certificate is trusted, so the proxy's own +// certificate has no issuer the client knows; nothing, the CONNECT included, +// reaches a peer that did not verify. +TEST_F(ProxyTest, AnHttpsProxyWhoseCertificateDoesNotVerifyIsRefusedWithTheReason) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + ASSERT_FALSE(proxy.failed()); + TrustOnly bundle(tls_test::kWrongNameCertPem); + + auto cfg = proxied("https://127.0.0.1:" + std::to_string(proxy.port())); + cfg.verifySsl = true; + https::HttpClient client(cfg); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_TRUE(resp.statusText.starts_with("proxy: could not open a TLS connection to 127.0.0.1:")) + << resp.statusText; + EXPECT_NE(resp.statusText.find("certificate verification failed"), std::string::npos) + << resp.statusText; + EXPECT_TRUE(seen.get().empty()); +} + +TEST_F(ProxyTest, CredentialsGoToAnHttpsProxyInsideItsTlsSession) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port(), "Basic YWxpY2U6czNjcmV0"); + + https::HttpClient ok(proxied("https://alice:s3cret@127.0.0.1:" + std::to_string(proxy.port()))); + EXPECT_EQ(ok.send(get(target.url("/"))).statusCode, 200); + + https::HttpClient bad(proxied("https://alice:nope@127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = bad.send(get(target.url("/"))); + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); +} + +TEST_F(ProxyTest, ABodyLargerThanARecordCrossesBothSessions) { + std::string big(300 * 1024, 'x'); + for (std::size_t i = 0; i < big.size(); i += 97) big[i] = static_cast('a' + i % 26); + tls_test::Server target([&](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response(big)); + return true; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_TRUE(resp.bodyComplete); + EXPECT_TRUE(resp.body == big); +} + +TEST_F(ProxyTest, TwoRequestsShareOneTunnelThroughAnHttpsProxy) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} + +// The target writes the head and the body as two records and the proxy hands +// them on as one, so both arrive in a single record of the proxy session. The +// client reads the first through the proxy session and the descriptor is then +// empty while the second sits decrypted in that session: waiting on the +// descriptor, rather than on the session, would time out on a response that has +// already arrived. +TEST_F(ProxyTest, ABodyAlreadyDecryptedInTheProxySessionIsNotWaitedFor) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n"); + conn.write("hello"); + return true; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port(), {}, std::chrono::milliseconds(150)); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); +} + +TEST_F(ProxyTest, AStreamingRequestWorksThroughAnHttpsProxy) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n" + "Connection: close\r\n\r\ndata: one\n\ndata: two\n\n"); + return false; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + std::vector events; + auto resp = client.send_stream(get(target.url("/")), [&](const https::SseEvent& ev) { + events.push_back(ev.data); + return true; + }); + + EXPECT_EQ(resp.statusCode, 200); + ASSERT_EQ(events.size(), 2u); + EXPECT_EQ(events[0], "one"); + EXPECT_EQ(events[1], "two"); +} + +// The handshake has no timeout of its own, so the proxy ends it the way a real +// HTTP proxy does when it is sent something that is not HTTP: it reads the first +// bytes and hangs up. +TEST_F(ProxyTest, AnHttpsUrlToAPlainProxyFailsAtTheHandshake) { + auto target = hello_server(); + proxy_test::Server proxy([](proxy_test::Peer& peer) { peer.read_some(); }); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("TLS connection"), std::string::npos); +} + +TEST_F(ProxyTest, AnHttpUrlToATlsProxyIsAnErrorAndNotAHang) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("http://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_FALSE(resp.statusText.empty()); +} + +// ── SOCKS5 ─────────────────────────────────────────────────────────────────── + +namespace { + +// What a SOCKS5 proxy needs to know about how to behave. +struct Socks5Script { + int targetPort { 0 }; + // Empty: accept the no-authentication method. Otherwise "user:password", + // and the proxy insists on the username/password method. + std::string credentials; + // The reply code to the CONNECT request; 0 is success. + unsigned char replyCode { 0 }; + // The address type of the bound address in a success reply: 1, 3 or 4. + unsigned char boundType { 1 }; +}; + +std::string hex_of(std::string_view bytes) { + std::string out; + for (unsigned char c : bytes) out += std::format("{:02x}", c); + return out; +} + +// Notes, in order: "methods:", "auth::" when it was used, +// and "request::
:". +proxy_test::Server::Handler socks5_handler(Socks5Script script) { + return [script](proxy_test::Peer& peer) { + auto greet = peer.read_exact(2); + if (greet.size() != 2 || greet[0] != 5) return; + auto methods = peer.read_exact(static_cast(greet[1])); + peer.note("methods:" + hex_of(methods)); + + if (script.credentials.empty()) { + peer.write(std::string_view("\x05\x00", 2)); + } else { + if (methods.find('\x02') == std::string::npos) { + peer.write(std::string_view("\x05\xff", 2)); + return; + } + peer.write(std::string_view("\x05\x02", 2)); + auto ver = peer.read_exact(2); // 0x01, ULEN + if (ver.size() != 2) return; + auto user = peer.read_exact(static_cast(ver[1])); + auto plen = peer.read_exact(1); + if (plen.size() != 1) return; + auto pass = peer.read_exact(static_cast(plen[0])); + peer.note("auth:" + user + ":" + pass); + if (user + ":" + pass != script.credentials) { + peer.write(std::string_view("\x01\x01", 2)); + return; + } + peer.write(std::string_view("\x01\x00", 2)); + } + + auto head = peer.read_exact(4); // VER CMD RSV ATYP + if (head.size() != 4) return; + std::string addr; + std::size_t len = head[3] == 1 ? 4 : head[3] == 4 ? 16 : 0; + if (head[3] == 3) { + auto n = peer.read_exact(1); + if (n.size() != 1) return; + addr = peer.read_exact(static_cast(n[0])); + } else { + addr = peer.read_exact(len); + } + auto port = peer.read_exact(2); + if (port.size() != 2) return; + peer.note(std::format("request:{}:{}:{}", static_cast(head[3]), + head[3] == 3 ? addr : hex_of(addr), + (static_cast(port[0]) << 8) + | static_cast(port[1]))); + + std::string reply("\x05\x00\x00", 3); + reply[1] = static_cast(script.replyCode); + reply += static_cast(script.boundType); + if (script.boundType == 1) reply.append(4, '\x7f'); + else if (script.boundType == 4) reply.append(16, '\x01'); + else reply += std::string("\x0c") + "bind.example"; + reply.append("\x04\xd2", 2); + peer.write(reply); + if (script.replyCode == 0) peer.relay_to(script.targetPort); + }; +} + +std::string socks_url(std::string_view scheme, const proxy_test::Server& proxy, + std::string_view userinfo = {}) { + std::string url(scheme); + url += "://"; + if (!userinfo.empty()) { url += userinfo; url += "@"; } + return url + "127.0.0.1:" + std::to_string(proxy.port()); +} + +} // namespace + +TEST(ProxyUrl, SocksSchemesAreKeptAndDefaultToPort1080) { + auto a = https::parse_proxy_url("socks5://proxy.example"); + EXPECT_EQ(a.scheme, "socks5"); + EXPECT_EQ(a.port, 1080); + auto b = https::parse_proxy_url("SOCKS5H://u:p@proxy.example:9050"); + EXPECT_EQ(b.scheme, "socks5h"); + EXPECT_EQ(b.port, 9050); + EXPECT_EQ(b.user, "u"); +} + +TEST_F(ProxyTest, Socks5ConnectsToAnAddressLiteral) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port() })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto notes = proxy.notes(); + ASSERT_EQ(notes.size(), 2u); + EXPECT_EQ(notes[0], "methods:00"); // no credentials, so only "none" is offered + EXPECT_EQ(notes[1], "request:1:7f000001:" + std::to_string(target.port())); +} + +TEST_F(ProxyTest, Socks5hSendsAHostNameAndSocks5SendsAnAddress) { + auto target = hello_server(); + const std::string url = "https://localhost:" + std::to_string(target.port()) + "/"; + + proxy_test::Server remote(socks5_handler({ .targetPort = target.port() })); + https::HttpClient viaH(proxied(socks_url("socks5h", remote))); + EXPECT_EQ(viaH.send(get(url)).statusCode, 200); + ASSERT_EQ(remote.notes().size(), 2u); + EXPECT_EQ(remote.notes()[1], "request:3:localhost:" + std::to_string(target.port())); + + proxy_test::Server local(socks5_handler({ .targetPort = target.port() })); + https::HttpClient viaLocal(proxied(socks_url("socks5", local))); + EXPECT_EQ(viaLocal.send(get(url)).statusCode, 200); + ASSERT_EQ(local.notes().size(), 2u); + EXPECT_NE(local.notes()[1].substr(0, 10), "request:3:"); // an address, not the name +} + +TEST_F(ProxyTest, Socks5UsernameAndPasswordAreOfferedAndSent) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "al@ice:p:ss w" })); + + https::HttpClient client(proxied(socks_url("socks5h", proxy, "al%40ice:p%3Ass%20w"))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); + + auto notes = proxy.notes(); + ASSERT_GE(notes.size(), 2u); + EXPECT_EQ(notes[0], "methods:0002"); + EXPECT_EQ(notes[1], "auth:al@ice:p:ss w"); +} + +TEST_F(ProxyTest, Socks5WrongCredentialsAreReportedAsThat) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "alice:s3cret" })); + + https::HttpClient client(proxied(socks_url("socks5", proxy, "alice:wrong"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); + EXPECT_EQ(target.accepts(), 0); +} + +TEST_F(ProxyTest, Socks5ProxyThatWantsCredentialsWeDidNotGiveSaysSo) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "alice:s3cret" })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("requires credentials"), std::string::npos); +} + +TEST_F(ProxyTest, Socks5RefusalCarriesTheReplyCode) { + proxy_test::Server proxy(socks5_handler({ .replyCode = 5 })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("refused CONNECT: connection refused (SOCKS5 reply 5)"), + std::string::npos); +} + +TEST_F(ProxyTest, Socks5AnyBoundAddressTypeIsReadOffTheStream) { + // The reply to a successful CONNECT carries an address of one of three + // shapes. Misreading its length leaves bytes ahead of the TLS handshake + // and the request fails; reading it right is what a 200 here shows. + for (unsigned char type : {1, 3, 4}) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), .boundType = type })); + https::HttpClient client(proxied(socks_url("socks5", proxy))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); + } +} + +TEST_F(ProxyTest, Socks5ASpeakerOfAnotherProtocolIsNotMistakenForOne) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { + peer.read_exact(3); + peer.write("HTTP/1.1 400 Bad Request\r\n\r\n"); + }); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("not a SOCKS5 proxy"), std::string::npos); +} + +TEST_F(ProxyTest, Socks5hRefusesAHostNameThatDoesNotFitTheRequest) { + proxy_test::Server proxy(socks5_handler({})); + https::HttpClient client(proxied(socks_url("socks5h", proxy))); + auto resp = client.send(get("https://" + std::string(300, 'a') + ".example/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("does not fit a SOCKS5 request"), std::string::npos); +} + +TEST_F(ProxyTest, TwoRequestsShareOneSocks5Tunnel) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port() })); + + https::HttpClient client(proxied(socks_url("socks5h", proxy))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} diff --git a/tests/test_tls_verify.cpp b/tests/test_tls_verify.cpp new file mode 100644 index 0000000..ccaae34 --- /dev/null +++ b/tests/test_tls_verify.cpp @@ -0,0 +1,143 @@ +// A client with `verifySsl = true` refuses a server certificate it cannot trust. +// +// Each test serves one certificate from the in-process TLS server and trusts it +// (or not) by pointing SSL_CERT_FILE at a bundle written for the test, so the +// machine's own CA store never enters into it and nothing leaves 127.0.0.1. +// The certificates in tls_test_server.hpp are self-signed, which lets a trusted +// one be its own bundle and lets the expired and wrong-name ones differ from it +// in that one respect only. +#include +#include "tls_test_server.hpp" + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +void set_env(const char* name, const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s(name, value); +#else + ::setenv(name, value, 1); +#endif +} + +class TlsVerifyTest : public ::testing::Test { +protected: + void SetUp() override { + https::Socket::platform_init(); + const char* old = std::getenv("SSL_CERT_FILE"); + hadOld_ = old != nullptr; + if (hadOld_) old_ = old; + bundle_ = std::filesystem::temp_directory_path() + / ("tinyhttps-verify-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + ".pem"); + } + + void TearDown() override { + set_env("SSL_CERT_FILE", hadOld_ ? old_.c_str() : ""); + std::error_code ec; + std::filesystem::remove(bundle_, ec); + } + + // Makes `content` the only CA bundle the client will read. + void trust(std::string_view content) { + std::ofstream(bundle_, std::ios::binary) << content; + set_env("SSL_CERT_FILE", bundle_.string().c_str()); + } + + https::HttpResponse get(const tls_test::Server& server) { + https::HttpClientConfig cfg; // verifySsl is true by default + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = server.url("/"); + return client.send(req); + } + +private: + std::filesystem::path bundle_; + std::string old_; + bool hadOld_ { false }; +}; + +bool answer_ok(tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("ok")); + return true; +} + +} // namespace + +TEST_F(TlsVerifyTest, ACertificateIssuedByATrustedCaIsAccepted) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust(tls_test::kCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + EXPECT_EQ(r.body, "ok"); +} + +TEST_F(TlsVerifyTest, ACertificateNoTrustedCaIssuedIsRefused) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); // a bundle that does not contain the server's + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("certificate verification failed"), std::string::npos) + << r.statusText; + EXPECT_NE(r.statusText.find("not correctly signed by the trusted CA"), std::string::npos) + << r.statusText; +} + +TEST_F(TlsVerifyTest, AnExpiredCertificateIsRefused) { + tls_test::Server server(answer_ok, tls_test::kExpiredCertPem, tls_test::kExpiredKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kExpiredCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("expired"), std::string::npos) << r.statusText; +} + +TEST_F(TlsVerifyTest, ACertificateForAnotherHostnameIsRefused) { + tls_test::Server server(answer_ok, tls_test::kWrongNameCertPem, tls_test::kWrongNameKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("does not match"), std::string::npos) << r.statusText; +} + +TEST_F(TlsVerifyTest, ABundleThatCannotBeParsedIsReportedAsSuch) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust("this is not a certificate\n"); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("cannot parse the CA certificate bundle"), std::string::npos) + << r.statusText; +} + +TEST_F(TlsVerifyTest, VerificationCanStillBeTurnedOff) { + tls_test::Server server(answer_ok, tls_test::kExpiredCertPem, tls_test::kExpiredKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); + + https::HttpClientConfig cfg; + cfg.verifySsl = false; + cfg.connectTimeoutMs = 4000; + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = server.url("/"); + EXPECT_EQ(client.send(req).statusCode, 200); +} diff --git a/tests/tls_test_server.hpp b/tests/tls_test_server.hpp new file mode 100644 index 0000000..8017262 --- /dev/null +++ b/tests/tls_test_server.hpp @@ -0,0 +1,543 @@ +// A TLS server, in this process, that answers exactly what a test tells it to. +// +// WHY A TEST SERVER AND NOT AN ENDPOINT ON THE INTERNET. The defects in issue +// #15 are all about what the library does with a connection AFTER a response +// went wrong — a body that stops halfway, a chunk header that does not parse, a +// 302 whose body is never read. No public endpoint produces those on demand, +// and the assertion that matters most is not about a response at all: it is +// **how many TCP connections the server saw**. A socket returned to the pool +// dirty is invisible in the bytes a caller receives (issue #15's own report has +// a case whose output is byte-for-byte correct) and unmissable in the +// connection count. +// +// WHY TLS AND NOT PLAIN HTTP. All three entry points begin with +// `if (parsed.scheme != "https") return "Only HTTPS is supported"`, so a plain +// listener cannot be reached by the code under test at all. mbedtls is already a +// dependency and its server side is compiled in (MBEDTLS_SSL_SRV_C), so the +// server costs a certificate and about two hundred lines. +// +// The certificate below is self-signed and its private key is in this file on +// purpose: it authenticates nothing, exists only for 127.0.0.1, and the tests +// connect with `verifySsl = false`. It is not a secret and must never be used +// for anything. +#pragma once + +#include +#include +#include +#include +#include +#include + +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK +#include +#include +#else +#include +#include +#endif + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace tls_test { + +// Self-signed, CN=localhost, valid for a century. See the file comment. +inline constexpr const char* kCertPem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDJzCCAg+gAwIBAgIUexXC6epWFGkwN5tQmE6z/veNad0wDQYJKoZIhvcNAQEL +BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MCAXDTI2MDkwNTE4NDUxN1oYDzIxMjYw +ODEyMTg0NTE3WjAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEB +AQUAA4IBDwAwggEKAoIBAQDU5o+tqAgr2GUZc607VbubYhGNStjM2BIcADu8MZeZ +/pyclKaVlCQ5aCF0MyQBZWPBxOPCYblDHjZG0qZ5Hp4cw115yH5di2aa7U9BVQ9C +gPiH1xdX7fNECu8TqTGL7PaKgF2MguWSB05MQy1MrSKBwGf4r03tJkdxCR2wzEUN +Jswo6DrS2NzD61Xzs5djF+pOjNpnc/SpvTZ+axptbenm6UDziCnvCQXBFTsUvJVK +Cdb9ZPNdkoijtEsv625HXeNDvjwW02J5I9JxooSeCkBDMpr406toqMPq+C2sqTMO +kyNUD9LnwFLDkhy5+JcH7MlCGMZChpAUCU+k1hEdgAxfAgMBAAGjbzBtMB0GA1Ud +DgQWBBRqeelsU+P58i5W+sj/fEfOqBJ7hjAfBgNVHSMEGDAWgBRqeelsU+P58i5W ++sj/fEfOqBJ7hjAPBgNVHRMBAf8EBTADAQH/MBoGA1UdEQQTMBGCCWxvY2FsaG9z +dIcEfwAAATANBgkqhkiG9w0BAQsFAAOCAQEAFwgEVRw4Rqic7iXDSIF6SCofHTLL +7WlUEEgc6yOd6A3Z0KKSgnh1ty/4UwAWxWpcHNCpSI+A6YJaPwM9OJ0z3BOjhRHQ +p/vnAABJhoNBPUESUQyLRRkWMnhrAXaKiGughFkc1+oVqZwEvislcqxswBNxeCiM +kvQ7w1spMHf5bKLxg4hrjK+yj+M9YF8eZEtIQ9uZwV/AgQldt7K/6RMqb1LMcU3d +g47eY4vrNfyNPED6RSR6UwbatLzlotDSHmyQA0fme015Lxv0s511ZUSTvWbEm8rH +8/M9s4fhALVinK2IMktVEgFruo651BCUBNL17ki60uB29BGNhNu5zvprag== +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kKeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDU5o+tqAgr2GUZ +c607VbubYhGNStjM2BIcADu8MZeZ/pyclKaVlCQ5aCF0MyQBZWPBxOPCYblDHjZG +0qZ5Hp4cw115yH5di2aa7U9BVQ9CgPiH1xdX7fNECu8TqTGL7PaKgF2MguWSB05M +Qy1MrSKBwGf4r03tJkdxCR2wzEUNJswo6DrS2NzD61Xzs5djF+pOjNpnc/SpvTZ+ +axptbenm6UDziCnvCQXBFTsUvJVKCdb9ZPNdkoijtEsv625HXeNDvjwW02J5I9Jx +ooSeCkBDMpr406toqMPq+C2sqTMOkyNUD9LnwFLDkhy5+JcH7MlCGMZChpAUCU+k +1hEdgAxfAgMBAAECggEAPnGZdp+wNdv0WzC4gIy2x+5No5luWTaOqTPQUXRiOMKb +ALoA2iJnNYc7OK+/QcGRLsYm3152Th9QYBlsxl2almkew5dwqNM4NvyfoFPoc+MM +AhuuNxYNoclrMeMQKzBHZ3wa9Bl4aApIhsm1QaYOVuwuEpyoSIRPs/GuiaqAHkjX +3i+GK+kQHZc4p9pWaxRSXjxveLJ43SKpGbr0Wd3RrvcOiVJJxIHylH7CVw49SW8q +Pqp4fTHyIpAi+TaUcwcHPN+KdPSQCeii4qGHdGbAOoO4CBEr6R8mj8Yd0UbhMB9t +EuQkJyF+CUz/kL1wl1SObkxFOtrMQmAk2pnoqcHxcQKBgQDvkUsGSJyI4yNA64D0 +FzxI2OAVokFqrgA/OW4fAvGdSvaZcrLO682UCMMYGYwPypOmCoqXTCUa+Z6eXL6P +Wn+KdAGpHigMmHVUxRKoHqqKAfPY2wlwfRPa52GbDomPbB5IXofvrq1Wzez6KLzb +jGnXvWuhDAGW6YVkTSRN2Yj01QKBgQDjgQISUe/hriDiP80EcjRBqy0bAMiBn/TR +mcwGFgv0OFGealCc9AAH9/ahX0fMAhROwbn0Z5bvuM+TLtYRbsDbb7rFksyQauoJ +7FLIkxvIB+zD0u0mg1ceWpl0cm5XwvHDTU6vlcyfY2zr1WHAREeMSh7qJUkUle76 +8QB7fErmYwKBgGqj/YaVigCxQz8h1ixRr4cp604WBRKs0/VQ5kEtuUnwVadm1Euh +0chEwjuXG67n2SO/a4P/5ECGv2H1HOqJOV7zVs3mW6OHtir+8tgdloKKbfapQiFt +vAnkl6FDLl7GnRBP7Cj4U7bhQcz1l2QPtAnSCvgMdSStXLCSWkfBPuLlAoGAYPyn +iG3b6QcVx75RRZr2QiSadw+PawtpEE4Tl2igsf2sde927GJs/Pit8L+w2Pzt8WFP +SC09QHc2LtXhts7Tcvkf04iwosShf5d69z0Xs+AFBLqQFUIayrru/qCl/84AyTTU +3a/r6us1DRkLsi6pndofccxxalA88Ef736juOWcCgYEAsD22JvgmQJwNB0VOtPdh +5TEXjnyBsw4DCIkk13ADPm9TqrxBKE4zbF2E7tMrg8McKk09rPGgCrAkZXkAG3Kt +kXIBxmXf9BwXX0U27Owr6xpm75nNan4LZ9CxzXtK9jkLq9peAcphWYu7TRJK8G02 +VQKaSq3Gef7/KLR3YARmm5A= +-----END PRIVATE KEY----- +)PEM"; + +// Two more self-signed certificates, each wrong in one way, for the tests that +// check the client refuses them (test_tls_verify.cpp). Same terms as above: +// public keys, no secrets. +// +// kExpiredCertPem is CN/SAN localhost and 127.0.0.1, valid 2020-01-01 to 2020-01-02. +inline constexpr const char* kExpiredCertPem = R"PEM(-----BEGIN CERTIFICATE----- +MIIBeDCCAR6gAwIBAgIUQR4qNY0KJQYcYEO7585+Fy5Q/U8wCgYIKoZIzj0EAwIw +FDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIwMDEwMTAwMDAwMFoXDTIwMDEwMjAw +MDAwMFowFDESMBAGA1UEAwwJbG9jYWxob3N0MFkwEwYHKoZIzj0CAQYIKoZIzj0D +AQcDQgAEEb2pNZ4XE7aYNPdWF5+/h+6Ug+KoqKMnRQDgr4OfRvZPIH9NROFnEXnK +stqlISOxthgAmIkR1W3pLSj2BooVT6NOMEwwGgYDVR0RBBMwEYIJbG9jYWxob3N0 +hwR/AAABMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNSI5hk66wZ6dXb/N5+v +6H3CXr5+MAoGCCqGSM49BAMCA0gAMEUCIQD48yY+f7UE0URCJghr3Q+58od8BkxG +zvn+vOvdRjKNZwIgdfrfRWxHrQS/mfyZyzQwTWHHa/TXLD5Z+vlFNjSuAIk= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kExpiredKeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgP8T1y7ZRNYINaXf1 ++ciH6e0/avbCUNliO/brgmagw/OhRANCAAQRvak1nhcTtpg091YXn7+H7pSD4qio +oydFAOCvg59G9k8gf01E4WcRecqy2qUhI7G2GACYiRHVbektKPYGihVP +-----END PRIVATE KEY----- +)PEM"; + +// kWrongNameCertPem is valid for a century but names only wrong.example. +inline constexpr const char* kWrongNameCertPem = R"PEM(-----BEGIN CERTIFICATE----- +MIIBgTCCASagAwIBAgIUUaQJz12pEvdBKiHf8zQ3agib668wCgYIKoZIzj0EAwIw +GDEWMBQGA1UEAwwNd3JvbmcuZXhhbXBsZTAgFw0yNjEwMDEwMzU5NTZaGA8yMTI2 +MDkwNzAzNTk1NlowGDEWMBQGA1UEAwwNd3JvbmcuZXhhbXBsZTBZMBMGByqGSM49 +AgEGCCqGSM49AwEHA0IABAIy/jbI/tnfJmp4lDT9WtuvNNu3CapWaAAsXrXPBYVg +R5MyHwQhLGq9Uy+ZC9dpJI6R1+9b5NUCh5RfaOE4zYOjTDBKMBgGA1UdEQQRMA+C +DXdyb25nLmV4YW1wbGUwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUFN3zbGMq +Zu37qSve4CdZ6qXXSsAwCgYIKoZIzj0EAwIDSQAwRgIhAO/UR5PxHU2spE47HbQf +4qPyonqIlmGB1kp129etm0dYAiEAsjTvx61zqcTrES7R+zQOYykJ67PVTqqGkxIP +uv23Y9g= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kWrongNameKeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgdB0xMbmMhgDdvjON +ESmHq2CHTvE0SbWMP8P4Kt/GWpqhRANCAAQCMv42yP7Z3yZqeJQ0/VrbrzTbtwmq +VmgALF61zwWFYEeTMh8EISxqvVMvmQvXaSSOkdfvW+TVAoeUX2jhOM2D +-----END PRIVATE KEY----- +)PEM"; + +class Server; + +// One accepted connection, after its handshake. Handed to the test's handler. +class Conn { +public: + explicit Conn(Server& server) : server_(server) {} + + // Writes bytes verbatim. A test scripts a response by calling this, so a + // malformed status line or a body shorter than its Content-Length is as + // easy to produce as a correct one — which is the point. + bool write(std::string_view data) { + std::size_t sent = 0; + while (sent < data.size()) { + int ret = mbedtls_ssl_write( + &ssl_, reinterpret_cast(data.data() + sent), + data.size() - sent); + if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (ret <= 0) return false; + sent += static_cast(ret); + } + return true; + } + + // Stops answering while leaving the connection open, which is what a server + // that stalls mid-body looks like from the client. Returns when the server + // is stopped or after `limit`. + void park(std::chrono::milliseconds limit = std::chrono::seconds(20)); + + // Waits for the client to end the connection, discarding anything it sends. + // False when `limit` passes or the server is stopped first. + bool wait_peer_close(std::chrono::milliseconds limit = std::chrono::seconds(20)); + + // Ends the connection at the TLS layer, then at the transport. + void shutdown() { + if (closed_) return; + closed_ = true; + mbedtls_ssl_close_notify(&ssl_); + mbedtls_net_free(&net_); + } + + // Ends it at the transport only — a FIN with no `close_notify` first. + // + // THIS IS THE COMMON CASE, NOT THE EXOTIC ONE. The two-way TLS shutdown + // handshake is what the specification asks for and what few servers + // bother with; most simply close the socket. A client must read that as an + // end of stream, because for a body whose framing IS the close it is the + // only thing marking the end. + void close_hard() { + if (closed_) return; + closed_ = true; + mbedtls_net_free(&net_); + } + + // The request head this connection last read, for a handler that wants to + // answer differently per path. + const std::string& request() const { return request_; } + + // Splices what this connection decrypts to 127.0.0.1:`port`, and what comes + // back from there into it, until either side ends. This is what makes the + // server a proxy for the proxy tests: after it answers a CONNECT, the + // client's next bytes are the start of a second TLS session, and this + // carries them to the real target untouched. + // + // With `gather` set, what comes back from the target is held until the + // target has been quiet for that long and then sent as one write, so + // several records the target wrote in a row reach the client inside a single + // record of this session. + void relay_to(int port, std::chrono::milliseconds gather = std::chrono::milliseconds::zero()); + +private: + friend class Server; + + // Waits for a readable byte while staying responsive to `Server::stop`. + // + // A plain blocking read would be simpler and would hang the test binary: + // several tests leave a connection open with no further request on it, and + // the connection thread has to notice the server shutting down. The + // buffered-bytes check first is the same one `TlsSocket::wait_readable` + // makes — decrypted data can be waiting where the socket has nothing. + bool wait_readable(); + + // Reads one request: the head through its blank line, then as many body + // bytes as its Content-Length declares, so the stream stays aligned for the + // next request on a keep-alive connection. + bool read_request() { + request_.clear(); + unsigned char c {}; + while (request_.find("\r\n\r\n") == std::string::npos) { + if (!wait_readable()) return false; + int ret = mbedtls_ssl_read(&ssl_, &c, 1); + if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (ret <= 0) return false; + request_.push_back(static_cast(c)); + if (request_.size() > 65536) return false; + } + std::size_t bodyLen = declared_body_length(request_); + std::string sink; + while (sink.size() < bodyLen) { + if (!wait_readable()) return false; + unsigned char buf[4096]; + std::size_t want = bodyLen - sink.size(); + if (want > sizeof buf) want = sizeof buf; + int ret = mbedtls_ssl_read(&ssl_, buf, want); + if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (ret <= 0) return false; + sink.append(reinterpret_cast(buf), static_cast(ret)); + } + return true; + } + + static std::size_t declared_body_length(const std::string& head) { + // Case-insensitive search for the one header a test request may carry. + std::string lower; + lower.reserve(head.size()); + for (char c : head) lower.push_back(static_cast(std::tolower(static_cast(c)))); + auto pos = lower.find("content-length:"); + if (pos == std::string::npos) return 0; + pos += std::strlen("content-length:"); + std::size_t value = 0; + while (pos < head.size() && (head[pos] == ' ' || head[pos] == '\t')) ++pos; + while (pos < head.size() && head[pos] >= '0' && head[pos] <= '9') { + value = value * 10 + static_cast(head[pos] - '0'); + ++pos; + } + return value; + } + + Server& server_; + mbedtls_net_context net_ {}; + mbedtls_ssl_context ssl_ {}; + std::string request_; + bool closed_ { false }; +}; + +class Server { +public: + // Answers one request. `index` counts requests across all connections, so a + // handler can script "the first request stalls, the second succeeds". + // Return true to serve another request on the same connection. + using Handler = std::function; + + // Serves `certPem`/`keyPem`, the default pair unless a test needs another. + explicit Server(Handler handler, const char* certPem = kCertPem, + const char* keyPem = kKeyPem) + : handler_(std::move(handler)), certPem_(certPem), keyPem_(keyPem) { + mbedtls_net_init(&listener_); + // Port 0: the OS picks a free one, which is what lets tests run + // concurrently and on a machine where anything might already be bound. + if (mbedtls_net_bind(&listener_, "127.0.0.1", "0", MBEDTLS_NET_PROTO_TCP) != 0) { + failed_ = true; + return; + } + sockaddr_in addr {}; + socklen_t len = sizeof addr; + if (::getsockname(listener_.fd, reinterpret_cast(&addr), &len) != 0) { + failed_ = true; + return; + } + port_ = ntohs(addr.sin_port); + acceptor_ = std::thread([this] { accept_loop(); }); + } + + ~Server() { stop(); } + + Server(const Server&) = delete; + Server& operator=(const Server&) = delete; + + // The acceptor is joined before the listener is closed, and the connection + // threads before anything they touch is freed. Closing a descriptor another + // thread is blocked on is a race that mostly works, which is the worst kind + // to put in a test suite — so the accept loop polls instead. + void stop() { + if (stopping_.exchange(true)) return; + if (acceptor_.joinable()) acceptor_.join(); + for (auto& t : connections_) { + if (t.joinable()) t.join(); + } + connections_.clear(); + mbedtls_net_free(&listener_); + } + + [[nodiscard]] bool failed() const { return failed_; } + [[nodiscard]] int port() const { return port_; } + [[nodiscard]] std::string url(std::string_view path) const { + return "https://127.0.0.1:" + std::to_string(port_) + std::string(path); + } + + // THE ASSERTION THAT CATCHES WHAT THE BYTES DO NOT. A connection returned to + // the pool with a previous response still on it produces a correct-looking + // result and one fewer accept than a correct client would make. + [[nodiscard]] int accepts() const { return accepts_.load(); } + [[nodiscard]] int requests() const { return requests_.load(); } + [[nodiscard]] bool stopping() const { return stopping_.load(); } + +private: + void accept_loop() { + while (!stopping_.load()) { + int ready = mbedtls_net_poll(&listener_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) break; + if (ready == 0) continue; + + mbedtls_net_context client {}; + mbedtls_net_init(&client); + if (mbedtls_net_accept(&listener_, &client, nullptr, 0, nullptr) != 0) { + mbedtls_net_free(&client); + continue; + } + accepts_.fetch_add(1); + connections_.emplace_back([this, client]() mutable { serve(client); }); + } + } + + // Each connection carries its own TLS state. Sharing one `mbedtls_ssl_config` + // would mean sharing one CTR_DRBG across threads, and this build of mbedtls + // has no threading layer compiled in — so the isolation is a correctness + // requirement rather than tidiness. + void serve(mbedtls_net_context client) { + Conn conn(*this); + conn.net_ = client; + + mbedtls_entropy_context entropy; + mbedtls_ctr_drbg_context drbg; + mbedtls_ssl_config conf; + mbedtls_x509_crt cert; + mbedtls_pk_context key; + + mbedtls_entropy_init(&entropy); + mbedtls_ctr_drbg_init(&drbg); + mbedtls_ssl_config_init(&conf); + mbedtls_x509_crt_init(&cert); + mbedtls_pk_init(&key); + mbedtls_ssl_init(&conn.ssl_); + + auto cleanup = [&] { + mbedtls_ssl_free(&conn.ssl_); + mbedtls_pk_free(&key); + mbedtls_x509_crt_free(&cert); + mbedtls_ssl_config_free(&conf); + mbedtls_ctr_drbg_free(&drbg); + mbedtls_entropy_free(&entropy); + mbedtls_net_free(&conn.net_); + }; + + if (mbedtls_ctr_drbg_seed(&drbg, mbedtls_entropy_func, &entropy, nullptr, 0) != 0 + || mbedtls_x509_crt_parse(&cert, + reinterpret_cast(certPem_), + std::strlen(certPem_) + 1) != 0 + || mbedtls_pk_parse_key(&key, + reinterpret_cast(keyPem_), + std::strlen(keyPem_) + 1, nullptr, 0, + mbedtls_ctr_drbg_random, &drbg) != 0 + || mbedtls_ssl_config_defaults(&conf, MBEDTLS_SSL_IS_SERVER, + MBEDTLS_SSL_TRANSPORT_STREAM, MBEDTLS_SSL_PRESET_DEFAULT) != 0) { + cleanup(); + return; + } + + mbedtls_ssl_conf_rng(&conf, mbedtls_ctr_drbg_random, &drbg); + // The client caps itself at TLS 1.2 (tls.cppm), so this is the version + // that will be negotiated; saying so keeps the handshake off the 1.3 + // path entirely. + mbedtls_ssl_conf_max_tls_version(&conf, MBEDTLS_SSL_VERSION_TLS1_2); + mbedtls_ssl_conf_min_tls_version(&conf, MBEDTLS_SSL_VERSION_TLS1_2); + if (mbedtls_ssl_conf_own_cert(&conf, &cert, &key) != 0 + || mbedtls_ssl_setup(&conn.ssl_, &conf) != 0) { + cleanup(); + return; + } + mbedtls_ssl_set_bio(&conn.ssl_, &conn.net_, mbedtls_net_send, + mbedtls_net_recv, nullptr); + + int ret = 0; + while ((ret = mbedtls_ssl_handshake(&conn.ssl_)) != 0) { + if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE) { + cleanup(); + return; + } + } + + while (!stopping_.load()) { + if (!conn.read_request()) break; + const int index = requests_.fetch_add(1); + if (!handler_(conn, index)) break; + } + conn.shutdown(); + cleanup(); + } + + Handler handler_; + const char* certPem_; + const char* keyPem_; + mbedtls_net_context listener_ {}; + int port_ { 0 }; + bool failed_ { false }; + std::atomic stopping_ { false }; + std::atomic accepts_ { 0 }; + std::atomic requests_ { 0 }; + std::thread acceptor_; + std::vector connections_; +}; + +inline bool Conn::wait_readable() { + if (mbedtls_ssl_get_bytes_avail(&ssl_) > 0) return true; + for (int waited = 0; waited < 20000; waited += 50) { + if (server_.stopping()) return false; + int ready = mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, 50); + if (ready > 0) return true; + if (ready < 0) return false; + } + return false; +} + +inline void Conn::relay_to(int port, std::chrono::milliseconds gather) { + mbedtls_net_context target {}; + mbedtls_net_init(&target); + if (mbedtls_net_connect(&target, "127.0.0.1", std::to_string(port).c_str(), + MBEDTLS_NET_PROTO_TCP) != 0) { + return; + } + unsigned char buf[4096]; + std::string back; + while (!server_.stopping()) { + if (mbedtls_ssl_get_bytes_avail(&ssl_) > 0 + || mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, 10) > 0) { + int n = mbedtls_ssl_read(&ssl_, buf, sizeof buf); + if (n == MBEDTLS_ERR_SSL_WANT_READ || n == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (n <= 0) break; + int sent = 0; + while (sent < n) { + int w = mbedtls_net_send(&target, buf + sent, static_cast(n - sent)); + if (w <= 0) { n = 0; break; } + sent += w; + } + if (n == 0) break; + } + if (mbedtls_net_poll(&target, MBEDTLS_NET_POLL_READ, 10) > 0) { + int n = mbedtls_net_recv(&target, buf, sizeof buf); + if (n <= 0) break; + back.assign(reinterpret_cast(buf), static_cast(n)); + while (gather.count() > 0 && back.size() < 8192 + && mbedtls_net_poll(&target, MBEDTLS_NET_POLL_READ, + static_cast(gather.count())) > 0) { + n = mbedtls_net_recv(&target, buf, sizeof buf); + if (n <= 0) break; + back.append(reinterpret_cast(buf), static_cast(n)); + } + if (!write(back)) break; + } + } + mbedtls_net_free(&target); +} + +inline bool Conn::wait_peer_close(std::chrono::milliseconds limit) { + const auto deadline = std::chrono::steady_clock::now() + limit; + unsigned char buf[256]; + while (!server_.stopping() && std::chrono::steady_clock::now() < deadline) { + const int ready = mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) return true; + if (ready == 0) continue; + int ret = mbedtls_ssl_read(&ssl_, buf, sizeof buf); + if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (ret <= 0) return true; + } + return false; +} + +inline void Conn::park(std::chrono::milliseconds limit) { + const auto deadline = std::chrono::steady_clock::now() + limit; + while (!server_.stopping() && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +} + +// A response with a declared length and a body of exactly that length. +inline std::string ok_response(std::string_view body, + std::string_view extraHeaders = {}) { + std::string out = "HTTP/1.1 200 OK\r\nContent-Length: "; + out += std::to_string(body.size()); + out += "\r\n"; + out += extraHeaders; + out += "\r\n"; + out += body; + return out; +} + +} // namespace tls_test diff --git a/tools/template_smoke.sh b/tools/template_smoke.sh new file mode 100755 index 0000000..6eeeb3d --- /dev/null +++ b/tools/template_smoke.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# Render and build every template in templates/ against THIS checkout. +# +# WHY THIS EXISTS AND WHY IT IS NOT `mcpp new`. `mcpp new --template tinyhttps` +# resolves the package from the index, so it can only ever exercise a version +# that has already been released. A template is part of the release it ships in; +# checking it after publishing means the first person to run `mcpp new` is the +# one who finds out it does not compile. +# +# So this renders each template the way mcpp's scaffolder does — the same +# placeholder vocabulary, `.in` suffix stripped, everything else copied — and +# then repoints the generated dependency at this working tree. +# +# The vocabulary below is mcpp's, from `src/scaffold/template.cppm`: +# {{project.name}} {{project.namespace}} {{project.qualifiedName}} +# {{template.package.namespace}} {{template.package.name}} +# {{template.package.selector}} {{template.package.version}} {{template.name}} +# (`{{self.name}}` and `{{self.version}}` are a compatibility spelling of the +# two selector/version tokens and are not used here.) +# +# Usage: bash tools/template_smoke.sh (mcpp on PATH, or $MCPP) +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +MCPP_BIN="${MCPP:-$(command -v mcpp || true)}" +if [[ -z "$MCPP_BIN" || ! -x "$MCPP_BIN" ]]; then + echo "FATAL: put mcpp on PATH or set MCPP=/path/to/mcpp" >&2 + exit 1 +fi + +SELF_NS="$(sed -n 's/^namespace *= *"\([^"]*\)".*/\1/p' "$ROOT/mcpp.toml" | head -1)" +SELF_NAME="$(sed -n 's/^name *= *"\([^"]*\)".*/\1/p' "$ROOT/mcpp.toml" | head -1)" +SELF_VERSION="$(sed -n 's/^version *= *"\([^"]*\)".*/\1/p' "$ROOT/mcpp.toml" | head -1)" +echo "library: $SELF_NS.$SELF_NAME@$SELF_VERSION" + +# Generated projects live inside the repo so target/ keeps them out of git. +# The depth is fixed, so each one reaches the library root at ../../.. +TMP="$ROOT/target/template-smoke" +REL_ROOT="../../.." +rm -rf "$TMP" +mkdir -p "$TMP" + +fail=0 +for tdir in "$ROOT"/templates/*/; do + tname="$(basename "$tdir")" + proj="$TMP/smoke_$tname" + mkdir -p "$proj" + + if [[ ! -f "$tdir/template.toml" ]]; then + echo "FAIL: template '$tname' has no template.toml" >&2 + fail=1 + continue + fi + + while IFS= read -r -d '' f; do + rel="${f#"$tdir"}" + [[ "$rel" == "template.toml" ]] && continue # metadata, not content + dest="$proj/${rel%.in}" + mkdir -p "$(dirname "$dest")" + if [[ "$f" == *.in ]]; then + sed -e "s/{{project\.name}}/smoke_$tname/g" \ + -e "s/{{project\.namespace}}/$SELF_NS/g" \ + -e "s/{{project\.qualifiedName}}/$SELF_NS.smoke_$tname/g" \ + -e "s/{{template\.package\.namespace}}/$SELF_NS/g" \ + -e "s/{{template\.package\.name}}/$SELF_NAME/g" \ + -e "s/{{template\.package\.selector}}/$SELF_NAME/g" \ + -e "s/{{template\.package\.version}}/$SELF_VERSION/g" \ + -e "s/{{template\.name}}/$tname/g" "$f" > "$dest" + else + cp "$f" "$dest" + fi + done < <(find "$tdir" -type f -print0) + + # An unrendered placeholder is a token this script does not know and mcpp + # might not either. Catch it here rather than in the compiler's output. + if grep -rn "{{" "$proj" >/dev/null 2>&1; then + echo "FAIL: template '$tname' left an unrendered placeholder:" >&2 + grep -rn "{{" "$proj" >&2 + fail=1 + continue + fi + + # Build against this checkout rather than the index, which may not have this + # version yet. Relative on purpose: an absolute path needs TOML escaping on + # Windows. + sed -i.bak "s|^\( *\)$SELF_NAME *= *\"[^\"]*\"|\1$SELF_NAME = { path = \"$REL_ROOT\" }|" \ + "$proj/mcpp.toml" + rm -f "$proj/mcpp.toml.bak" + grep -q "path = \"$REL_ROOT\"" "$proj/mcpp.toml" || { + echo "FAIL: template '$tname' declares no '$SELF_NAME = \"...\"' dependency" >&2 + fail=1 + continue + } + + echo "=== template: $tname ===" + if [[ -f "$proj/src/main.cpp" ]]; then + # A bin template: build AND run it. Every template here exits 0 without + # network or credentials, so this stays honest in an offline CI job. + (cd "$proj" && "$MCPP_BIN" run) || { echo "FAIL: template '$tname'" >&2; fail=1; } + else + (cd "$proj" && "$MCPP_BIN" build) || { echo "FAIL: template '$tname'" >&2; fail=1; } + fi +done + +# Exactly one template may be the default; more than one is an error mcpp +# reports only when someone runs `mcpp new`. +defaults="$(grep -l "^default *= *true" "$ROOT"/templates/*/template.toml 2>/dev/null | wc -l)" +if [[ "$defaults" -gt 1 ]]; then + echo "FAIL: $defaults templates declare default = true; at most one may" >&2 + fail=1 +fi + +if [[ $fail -eq 0 ]]; then + echo "All templates render, build and run." +fi +exit $fail diff --git a/xmake.lua b/xmake.lua index 3bfce20..fd21acc 100644 --- a/xmake.lua +++ b/xmake.lua @@ -7,7 +7,7 @@ add_requires("gtest 1.15.2", { optional = true }) target("tinyhttps") set_kind("static") - set_version("0.1.0") + set_version("0.3.3") add_files("src/*.cppm", { public = true, install = true }) add_packages("mbedtls", { public = true }) set_policy("build.c++.modules", true)