diff --git a/.agents/docs/design-cancel-stop-token.md b/.agents/docs/design-cancel-stop-token.md new file mode 100644 index 0000000..671a1fa --- /dev/null +++ b/.agents/docs/design-cancel-stop-token.md @@ -0,0 +1,216 @@ +# 设计方案：请求取消（`std::stop_token`），基于 PR #23 + +| | | +|---|---| +| 基线 | `origin/master @ f461940`（0.3.3） | +| 输入 | [PR #23](https://github.com/mcpplibs/tinyhttps/pull/23) `feat/cancel @ 8996984`（作者 yspbwx2010，`maintainerCanModify: true`） | +| 落地方式 | 在 PR #23 上追加 commit，不另开 PR | +| 文档日期 | 2026-10-02 | + +**标注约定**：✅ 已核验源码 · 🔁 已在本地复现或跑通 · ❓ 推断，未实测 + +--- + +## 0. TL;DR + +- **应该进库。** 库外做不到：socket 是 `HttpClient` 的私有状态，`HttpClient` 本身又不是线程安全的。库里也已经有半个先例：`download_to_file` 的 `isCancelled` 只在 body 的块与块之间检查。 +- **PR #23 的机制是对的，保留：** 等待只发生在 `Socket::wait_fd`，有 token 时按 50 ms 分片 `poll`；`bio_recv` 读之前先等。**它缺的是 token 的生命周期规则**，H1 就是这个缺口造成的 bug：经 https 代理复用连接时，一个 POST 会被执行两次。 +- **架构收敛成三条不变式：** token 只放在传输链最底层的 `Socket` 上；token 只在调用期间挂着，连接回池时摘掉；不传 token 时行为和 0.3.3 完全一样。 +- **改动量：** 在 PR 基础上，`src/` 约 +15 行（必需部分 +7 行），另加测试和 CI。必需部分已经做成原型跑通（§7）。 +- **CI 目前对这个功能只有一个配置有效：** Linux + gcc 16。Windows job 不编译 `test_cancel`；Linux libc++（llvm 20/22）连链接都过不了；没有 macOS job。 + +### 决策（2026-10-02 均按建议采纳，执行见 [exec-plan-0.3.4-cancel.md](exec-plan-0.3.4-cancel.md)） + +| # | 决策 | 建议 | +|---|---|---| +| D1 | `download_to_file` 是否同时加 `std::stop_token` 和 `DownloadToFileResult::cancelled` | **加**。理由见 §3.3 | +| D2 | 删掉 PR 给 `proxy_connect` 加的 `stop` 参数 | **删**。它是遗留的包装函数，只有 `proxy_tunnel` 内部需要 token | +| D3 | CI 新增 Linux llvm@22 job、Windows 跑 hermetic 测试 | **加**。macOS 视 mcpp 支持情况定（❓） | +| D4 | 写操作阻塞（对端不读）仍不可取消 | **本 PR 不做**，列为已知限制。后续单独做非阻塞发送 | + +--- + +## 1. 该不该进库 + +**结论：进。** + +| 论点 | 说明 | +|---|---| +| 库外无解 | 调用方拿不到 fd。唯一的绕法是把 `send` 丢到 detached 线程里等 `readTimeoutMs`。这期间 `HttpClient` 不能再用（不是线程安全的，`http.cppm` 里 `HttpClient` 的注释写明了），连接也一直占着，服务器还在继续生成响应 | +| 已有先例 | `download_to_file(..., isCancelled)` ✅ 说明库已经承认取消是它的职责。但 `isCancelled` 只在 body 块之间检查，等响应头、TLS 握手这些阶段照样卡死，正是 PR 要解决的问题 | +| 标准词汇 | `std::stop_token` 是 C++20 标准类型，和 `std::jthread` 天然配合，今后也能对接 `std::execution` 的 stop token。不需要自造回调或传 `atomic*` | +| 成本可控 | 不传 token 时零开销（I3）。传了 token 时，每个进行中的调用每秒多 20 次 `poll` 唤醒 | + +**反方意见及回应：**“tiny 库不该长功能”。这里不是加功能，而是补全已有的取消语义（`isCancelled` 覆盖不全），并让三个入口一致。这和 #15 合并三个入口、只保留一份读取逻辑是同一个思路。 + +--- + +## 2. 架构：三条不变式 + +``` +HttpClient::send / send_stream / download_to_file ← 唯一接收 token 的地方 + │ perform_exchange: sock->set_stop(stop) ← 挂上（每次调用、每次重试） + │ PooledConnection::keep(): set_stop({}) ← 回池时摘下；drop() 直接销毁 + ▼ +TlsSocket::set_stop ──转发──► lower_（https 代理隧道）──转发──► Socket + │ 自己不存 token + ▼ +Socket::stop_ ── wait_fd()：分片 poll，片与片之间检查 stop_requested() + ── bio_recv：有 token 时先 wait_readable(-1) 再 recv + ── connect：每个地址、每次 DNS 回退前检查 +``` + +| 不变式 | 内容 | 违反时的后果 | +|---|---|---| +| **I1 只存在底层** | token 只存在于传输链最底层的 `Socket`。`TlsSocket` 不存 token，只往下转发（不管下面是 `socket_` 还是 `lower_`） | **H1** 🔁：https 代理下 `set_stop` 只设了用不到的 `socket_`，复用的隧道一直带着第一个请求的 token | +| **I2 只在调用期间存在** | 进入 exchange 时挂上，`keep()` 把连接还回池时摘掉。**连接池里的连接永远不带 token** | **M1**：直连情况下现在不会出错（下次请求会覆盖），但只要以后加一个在挂 token 之前等待池中连接的路径（比如空闲探活），就会踩到和 H1 一样的坑 | +| **I3 没有 token 时行为不变** | `stop_possible()==false` 时 `wait_fd` 就是原来的单次 `poll_fd`，`bio_recv` 不预先等待 | 已满足 ✅ | + +**为什么 token 必须作为状态存在 `Socket` 上，而不是作为参数一路传下去：** mbedtls 的 BIO 回调只有一个 `void* ctx`，在这里就是 `Socket*` ✅。握手和 TLS record 的后半段都是从 `bio_recv` 里阻塞的，没有别的通道能把 token 送进去。既然状态没法避免，就用 I1 和 I2 限定它在哪里、活多久。 + +### 考虑过、没有采用的方案 + +| 方案 | 不采用的原因 | +|---|---| +| `HttpClient::cancel()` 成员函数 | 需要从别的线程访问连接池，而 `HttpClient` 不是线程安全的。token 是不加锁就能跨线程传递的唯一渠道 | +| `std::stop_callback` + `shutdown(fd)` | 优点：没有轮询，还能覆盖阻塞的写（D4）。缺点：回调在发起 stop 的线程上执行，会和调用线程里 `drop()` 等所有关闭 fd 的路径竞争，必须加锁；另外 connect 进行中被 `shutdown` 时，macOS、Windows、openkal 上的语义都没有验证 ❓。**留作解决 D4 时的候选方案** | +| 第二个 fd（wake-up pipe）加 `poll` | Windows 没有 pipe；PR 作者说 openkal 上两个 fd 的 `poll` 会忙等（❓，未独立核实） | + +--- + +## 3. 在 PR #23 基础上：保留、修改、删除 + +### 3.1 保留（✅ 已逐项核验） + +- `Socket::wait_fd` 的分片 `poll`（50 ms，固定值，不做成配置项）；`bio_recv` 的预等待；connect 前、每个地址、DNS 回退前的 stop 检查。 +- `perform_exchange` 的 `fail()`：stop 优先于真实错误，并在这里 drop 连接；重试循环开头检查 stop；取消后不跟重定向。 +- `HttpResponse::cancelled`，以及它在响应头到达前后的两种形态（见 §4）。 +- `proxy_tunnel` 尾部的默认参数 `stop`（内部需要）。 +- `Socket` 的 move 会带走 `stop_`；`close()` 不清 `stop_`，所以 `set_stop` 必须在 `connect` 之前调用，现在的顺序是对的 ✅。 + +### 3.2 修改 + +| 项 | 改动 | 行数 | 状态 | +|---|---|---|---| +| **H1** | `TlsSocket::set_stop` 先转发给 `lower_`，再设 `socket_`（I1） | +3 | 🔁 原型通过 | +| **M1** | `PooledConnection::keep()` 把池中连接的 token 设为 `{}`（I2）。`drop()` 直接销毁连接，不用处理 | +5 | 🔁 原型通过 | +| **M2** | `test_cancel.cpp` 里 `import std` 的位置按标准库区分（§6.2） | +6 | 🔁 llvm 22 和 gcc 16 都通过 | +| 文档 | CHANGELOG 的 “Existing callers are unaffected” 改为：普通调用不受影响，但取 `&HttpClient::send` 成员函数指针的代码需要调整 | — | — | + +### 3.3 D1：`download_to_file` 加 token（建议） + +三个入口共用 `perform_exchange` 和 `read_body` ✅。PR 只改了其中两个入口，`download_to_file` 仍然只有 `isCancelled`，又回到了 #15 那张“修复只落到部分入口”的矩阵。具体改法： + +- 签名加尾部默认参数：`download_to_file(url, dest, onProgress = nullptr, isCancelled = nullptr, std::stop_token stop = {})`。 +- `DownloadToFileResult::cancelled`：`isCancelled` 或 token 任一触发都置为 true。`error` 文本保持现有的 `"cancelled"` 不变，原来匹配文本的代码不受影响。 +- 实现：`perform_exchange(..., stop)`；body 的 sink 里检查 stop；重定向递归时把 stop 传下去。预计约 +15 行。 +- `isCancelled` 保留，不标记废弃。它是轮询式谓词，和 token 不冲突。 + +### 3.4 D2：删除（建议） + +- `proxy_connect(..., std::stop_token)`：这是只返回 `Socket` 的遗留包装函数，库内部不用它。加参数只会扩大导出面（YAGNI）。 + +--- + +## 4. 语义契约（README 和 CHANGELOG 要写清楚） + +| 情形 | `statusCode` / `statusText` | `bodyComplete` / `bodyError` | `cancelled` | 连接 | +|---|---|---|---|---| +| 调用开始前 token 已经 stop | `0` / `"Cancelled"` | false / `"Cancelled"` | true | 不发起任何连接 | +| 响应头到达前取消（connect、代理、握手、等响应头） | `0` / `"Cancelled"` | false / `"Cancelled"` | true | drop | +| 响应头到达后取消（读 body、流式回调间隙） | 服务器给的值 | false / `"cancelled"` | true | drop | +| token 从未 stop | 和 0.3.3 一样 | 和 0.3.3 一样 | false | 和 0.3.3 一样 | + +其他规则： + +- `ok()` 不看 `cancelled`，就像它也不看 `bodyComplete`。取消的契约是那个 bool，不是文本。 +- 取消后不做 stale-connection 重试，也不跟重定向。 +- 真实失败和 stop 同时发生时，报告为取消。 +- 响应时间约 50 ms（一个分片）。 +- `close_notify` 只在握手完成之后的取消中发送。 + +**已知限制：**`getaddrinfo` 和 `socks5://` 对目标地址的解析不可中断；对端不读导致的阻塞写不可中断（socket 在 connect 之后被设回阻塞模式，`socket.cppm:257` ✅，见 D4）；openkal 上 connect 是同步完成的。 + +--- + +## 5. 测试 + +PR 已有 13 个用例，覆盖各阶段的首次建连 ✅。**缺口在“复用连接”这一维度**，H1 就漏在这里。补成下面这个矩阵： + +| 新增用例 | 覆盖 | 状态 | +|---|---|---| +| 经 https 代理复用：第一个请求结束后它的 token 被 stop，第二个 POST 在目标服务器上**只执行一次**，代理只被连一次 | H1 / I1 | 🔁 修复前 3 次请求、2 次代理连接；修复后通过 | +| 经 https 代理复用：第二个请求自己的 token 在 500 ms 内生效 | H1 / I1 | 🔁 修复前要等满 3000 ms 读超时 | +| 同样的场景走 http:// 代理和直连（对照组） | I1 | 🔁 通过 | +| README 的 jthread 写法：请求完成后 jthread 析构（会 `request_stop`），下一个请求复用连接并成功 | I2 | 🔁（https 代理那一组就是这个写法） | +| `download_to_file`：等响应头时取消、读 body 时取消；`cancelled==true`；目标文件的处理和现有 `isCancelled` 一致 | D1 | 待写 | +| SOCKS5 代理在等握手应答时取消 | 补齐代理种类 | 待写（PR 只覆盖了 http 和 https 代理） | + +时间断言使用 `kPromptMs = 500`。Windows 和 macOS 的 runner 比较慢，如果出现偶发失败，再按平台放宽，不要预先放宽。 + +--- + +## 6. CI 覆盖 + +### 6.1 现状 ✅ + +| job | 工具链 | 跑什么 | 对本功能的覆盖 | +|---|---|---|---| +| linux mcpp | gcc 16.1.0（mcpp 首次运行时引导安装的默认工具链） | 全部 `mcpp test` | ✅ 唯一有效的覆盖 | +| openkal | llvm + openkal libc++ | `examples/openkal` 的 `mcpp run` | 只编译库，不跑 `test_cancel` | +| windows msvc / llvm@20.1.7 | MSVC STL | **只跑 `mcpp test test_ca_store`** | ❌ 根本不编译 `test_cancel` | +| linux libc++ | — | — | ❌ 而且 `test_cancel` 在 llvm 20/22 上链接失败（§6.2） | +| macOS | — | — | ❌ 但 README 声称支持 macOS | + +### 6.2 libc++ 链接问题（M2）🔁 + +- 在 llvm 20.1.7 和 22.1.8（libc++）上，先 `#include ` 再 `import std;`，然后调用 `stop_source::request_stop()`，链接时报 `undefined hidden symbol __atomic_unique_lock<...>::__set_locked_bit`。llvm 23.1.0 和 gcc 16 正常。这是工具链 bug，不是本库的问题。 +- 仓库测试文件的惯例是先 include 后 import，因为 gcc 必须这样：反过来在 gcc 16 上会报 `redefinition of std::byte` 等错误 🔁。 +- 解决办法：在 `test_cancel.cpp` 里按标准库区分顺序（已验证 llvm 22 和 gcc 16 都能 16/16 通过）： + +```cpp +// libc++ (≤22) fails to link stop_source::request_stop() when +// was included textually before `import std`; libstdc++ needs the opposite. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif +``` + +- llvm 20 在 Linux 上两种顺序都编不过（import 在前时报 `requires clause differs`）🔁，显式实例化 `__atomic_unique_lock` 也不行 🔁，因此 **libc++ 只把 22 纳入 CI 矩阵**（Linux 和 macOS 都用 llvm@22.1.8；Windows 上的 llvm@20.1.7 用的是 MSVC STL，不受影响）。 +- README 的 jthread 示例对“include 和 import std 混用、用 libc++ ≤22”的使用者同样会链接失败，需要在 README 里加一行说明。 + +### 6.3 目标 + +| 改动 | 内容 | 状态 | +|---|---|---| +| Linux job 改成矩阵 `[gcc@16.1.0, llvm@22.1.8]`，都跑全部 `mcpp test` | 覆盖 libc++。llvm 22 也是 mcpp 本地常见的默认工具链 | 🔁 本地 llvm 22 + M2 修复通过 | +| Windows 两个 job 追加 `mcpp test test_cancel`，最好再加 `test_framing`、`test_pool`、`test_proxy` | 覆盖 `WSAPoll` 分片和 Windows 上 connect 的等待 | ❓ 测试服务器有 `_WIN32` 分支（`proxy_test_server.hpp:28`），但从没在 Windows 上编译过，先试，编不过再移植 | +| 新增 macOS job（macos-14），跑 hermetic 测试 | PR 作者明确说没在 macOS 上跑过；macOS 上 `poll` 对进行中的 connect 的语义需要实测 | ❓ 要先确认 mcpp 的 macOS 工具链可用。如果不可用，就作为 follow-up 单独开 issue，不阻塞本 PR | +| openkal | 不变。库被编进 example，签名的编译已经覆盖到 | — | + +--- + +## 7. 落地：在 PR #23 上追加的 commit + +原型补丁（H1 + M1 + M2 + 3 个复现测试）已经在 PR 分支上跑通：`test_cancel` 16/16、`test_pool`、`test_proxy`、`test_framing` 通过（gcc 16），`test_cancel` 在 llvm 22 上也通过。之前那一轮（H1 + 复现测试）的全量 `mcpp test` 是 8/8 🔁。 + +| # | commit | 依赖 | +|---|---|---| +| 1 | `fix(tls): a stop token reaches the session a tunnelled one runs over`：H1，以及 https/http/直连三组复用测试 | — | +| 2 | `fix(http): a pooled connection carries no stop token`：M1，以及 jthread 析构场景的测试 | 1 | +| 3 | `test(cancel): import std before the includes under libc++`：M2 | — | +| 4 | `feat(download): download_to_file takes a stop token`：D1，以及 `DownloadToFileResult::cancelled` 和测试 | 2，D1 批准后 | +| 5 | `refactor(proxy): proxy_connect does not take a stop token`：D2 | D2 批准后 | +| 6 | `ci: linux on llvm 22 as well; windows runs the hermetic tests`（再加 macOS job，如果可行）：D3 | 3 | +| 7 | `docs: cancellation contract, download_to_file, libc++ note` | 4、5 | + +顺序说明：1 和 2 是正确性修复，最先落地；3 必须在 6 之前，否则新加的 llvm job 第一次跑就会失败；6 失败时（比如 Windows 测试服务器不可移植），可以把 Windows 部分拆成 follow-up，不阻塞合并。 diff --git a/.agents/docs/exec-plan-0.3.0.md b/.agents/docs/exec-plan-0.3.0.md index ba9ad4b..79250fb 100644 --- a/.agents/docs/exec-plan-0.3.0.md +++ b/.agents/docs/exec-plan-0.3.0.md @@ -20,8 +20,8 @@ 版本取 **0.3.0** 而不是 0.2.11:`HttpResponse` / `HttpClientConfig` 加了字段(非破坏),严格解析会把过去静默接受的畸形响应变成显式错误(**行为变更**)。语义化版本下,行为变更进 minor。 **无感升级的边界**(release note 必须点名): -- ✅ 现有代码**不需要改一行**就能编译:只加字段,不改签名,`ok()` 语义不变。 -- ⚠️ 过去被静默接受的畸形响应现在会报错 —— 这不是新 bug,是过去在静默损坏数据。 +- 现有代码**不需要改一行**就能编译:只加字段,不改签名,`ok()` 语义不变。 +- 过去被静默接受的畸形响应现在会报错 —— 这不是新 bug,是过去在静默损坏数据。 --- @@ -45,7 +45,7 @@ ``` T0 分支 + 计划 ✔ │ - ├─ T1 P0 SIGPIPE（socket.cppm） ★ 必须最先 + ├─ T1 P0 SIGPIPE（socket.cppm） 必须最先 │ │ P0-1 send() + MSG_NOSIGNAL │ │ P0-2 socket() 后 setsockopt(SO_NOSIGPIPE) │ │ 理由：T5 让 pool_.erase() 在更多路径上执行， @@ -66,7 +66,7 @@ T0 分支 + 计划 ✔ │ └ 吸收 P2-E2（分配上限，chunk 分片读） │ │ │ ├─ T4 P3-A PooledConnection（默认丢弃）← 吸收 P1-A/B/C 全部十项 - │ │ │ ⚠ 递归重定向：递归前必须 keep() 或 drop()，不能让析构跨过递归 + │ │ │ 递归重定向：递归前必须 keep() 或 drop()，不能让析构跨过递归 │ │ │ │ │ ├─ T5 P2-E1 bounded drain（重定向/非 2xx/文件打不开时省一次握手） │ │ ├─ T6 P2-F1/F2 bodyComplete + bodyError（statusText 不再被覆盖） @@ -75,7 +75,7 @@ T0 分支 + 计划 ✔ │ └─ T8 纯单测：parse_status_line / parse_chunk_size_line / parse_content_length │ ├─ T9 集成测试:进程内 mbedtls TLS listener（T4 后） - │ ⚠ tinyhttps 只支持 https，本地明文 listener 用不了 → 必须自带 TLS 服务端 + │ tinyhttps 只支持 https，本地明文 listener 用不了 → 必须自带 TLS 服务端 │ T1..T11 用例见 §4 │ ├─ T10 openkal 体系(与 T1..T9 并行调研,结论落地在 T1/platform.cppm) @@ -136,7 +136,7 @@ T0 分支 + 计划 ✔ | # | 用例 | 断言 | 覆盖 | |---|---|---|---| | T1 | 服务端发一半 CL body 后挂住 | 请求 2 `statusCode == 200` | P1-A1 | -| T2 | 同上,**并断言服务端 accept 次数 == 2** | 🔴 唯一能抓住「看起来对了」的断言 | P1-A1 | +| T2 | 同上,**并断言服务端 accept 次数 == 2** | 唯一能抓住「看起来对了」的断言 | P1-A1 | | T3 | chunked 版本同上 | 同 T1+T2 | P1-A3 | | T4 | 服务端发一半后关连接 | 不崩;进程未被信号杀死 | #16 | | T5 | `SIGPIPE` 默认处置下跑 T4(子进程) | `WIFSIGNALED == false`,退出码 ≠ 141 | P0 | diff --git a/.agents/docs/exec-plan-0.3.4-cancel.md b/.agents/docs/exec-plan-0.3.4-cancel.md new file mode 100644 index 0000000..b065c49 --- /dev/null +++ b/.agents/docs/exec-plan-0.3.4-cancel.md @@ -0,0 +1,101 @@ +# 执行计划 —— 0.3.4：在途请求可取消，三个入口一致，CI 覆盖库支持的全部平台 + +| | | +|---|---| +| 输入 | [`design-cancel-stop-token.md`](design-cancel-stop-token.md)：D1–D4 均按建议采纳 | +| 基线 | `origin/master @ f461940`（0.3.3） | +| 落地 | 全部做在 PR #23 上（`yspbwx2010:feat/cancel`，`maintainerCanModify: true`），squash 合入 | +| 目标版本 | **0.3.4** | +| 跨仓库 | `mcpplibs/tinyhttps` → gitcode `mcpp-res/tinyhttps`（CN 镜像）→ `mcpplibs/mcpp-index` | + +--- + +## 0. 为什么是 0.3.4 + +- 只新增，不改已有行为：`send`、`send_stream`、`download_to_file` 都只加了尾部默认参数；`HttpResponse` 和 `DownloadToFileResult` 各在末尾追加一个字段；不传 token 时逐字节不变（不变式 I3）。 +- 依赖写成 caret 范围 `tinyhttps = "^0.3.0"` 的项目会自动解析到它。**更正（发布后实测）：** mcpp 里裸写的 `"0.3.1"` 是精确版本，不会解析到 0.3.4；`"0.3.x"` 不是 mcpp 的合法写法。0.3.3 的发布说明里也沿用了这个说法。已用已发布的 index 实测：`"^0.3.1"` 在 GLOBAL 和 CN 两个源下都解析到 0.3.4。 +- 唯一的源码不兼容点是对 `&HttpClient::send` 取成员函数指针（参数个数变了）。在 CHANGELOG 里点名说明。 + +## 1. 八个视角 + +| 视角 | 要求 | 落在哪 | +|---|---|---| +| **架构** | token 存在哪里、存活多久要有规则，不能靠每个调用点自己记得 | I1：只存在传输链最底层的 `Socket`，`TlsSocket` 只往下转发。I2：只在一次调用期间挂着，`PooledConnection::keep()` 把连接还回池时摘下 | +| **稳定性** | 取消不能引发重复执行、fd 泄漏或卡死 | 取消后不重试、不跟重定向；连接 drop；fd 计数测试；复用连接测试覆盖直连、http 代理、https 代理三种 | +| **优雅简洁** | 只有一套机制：在 `wait_fd` 里分片 `poll` | 不引入 pipe、shutdown 或回调；不增加配置项；`proxy_connect` 不加参数 | +| **用户体验** | 用 `std::jthread` 就能直接取消；结果用 bool 判断，不用匹配文本 | README 示例；两个结果结构都有 `cancelled`；README 说明 libc++ ≤22 的工具链问题 | +| **兼容性** | 升级不需要改代码 | §0；`HttpResponse{204, "No Content", {}, {}}` 这种聚合初始化仍然能用（mcpp-index 的测试断言了这一点） | +| **跨平台** | Linux（gcc、libc++）、macOS、Windows（msvc、clang）、openkal（linux-gnu、windows-musl）都要实测 | §4 CI 矩阵；openkal 示例增加一段不联网的取消检查 | +| **一致性** | 三个入口取消语义相同 | `download_to_file` 加 token（D1）；`isCancelled` 保留 | +| **无感升级** | 不传 token 时行为和 0.3.3 一样；不需要改依赖版本号 | I3；0.3.4 是补丁版本 | + +## 2. 任务依赖图 + +``` +T0 计划（本文）+ 设计文档纳入 PR + │ + ├─ T1 src：I1（tls.cppm set_stop 转发给 lower_） + │ └─ T2 src：I2（http.cppm PooledConnection::keep 时摘下 token） + │ └─ T3 src：D1 download_to_file(…, stop) + DownloadToFileResult::cancelled + │ └─ T4 src：D2 proxy_connect 去掉 stop 参数；源码注释 + │ + ├─ T5 测试（可与 T1–T4 并行编写，依赖它们才能通过） + │ a. 复用连接矩阵：直连 / http 代理 / https 代理 × {旧 token 已 stop, 新 token 生效} + │ b. SOCKS5 等待握手应答时取消 + │ c. download_to_file：等响应头时取消、读 body 时取消、已 stop 的 token + │ d. libc++ 下 import std 的位置（M2） + │ + ├─ T6 openkal：examples/openkal 增加不联网的取消检查（本地监听 + 握手中取消） + │ 运行时版本跟 mcpp-index 的 pins.toml 对齐；增加 windows-musl（Linux 上构建，windows-2022 上原生运行；本地用 wine） + │ + ├─ T7 CI（依赖 T5d、T6） + │ linux {gcc@16.1.0, llvm@22.1.8} 跑全部测试 · macos-15 llvm@22.1.8 跑全部测试 + │ windows {msvc, llvm@20.1.7} 跑不联网的测试 · openkal {x86_64-linux-gnu, x86_64-windows-musl} + │ + ├─ T8 文档：README、CHANGELOG 0.3.4、mcpp.toml version、README 安装行 + │ + └─ T9 合入与发版（依赖全部） + CI 全绿 → 自审 → squash 合入 → tag 0.3.4 + GitHub release + → 本地 gtc 上传 gitcode mcpp-res/tinyhttps 的 0.3.4 tarball（核对 sha256） + → mcpp-index PR：pkgs/t/tinyhttps.lua 加 0.3.4（三个平台）， + tests/examples/tinyhttps 升到 0.3.4 并加取消的 API 检查 + → 用已发布的 index 实测：新建项目解析到 0.3.4，GLOBAL 和 CN 两个源都验证 +``` + +**关键路径**：T1 → T2 → T5 → T7 → T9。T3/T4、T6、T8 可以和关键路径并行。 + +## 3. 测试清单（test_cancel） + +| 用例 | 覆盖 | +|---|---| +| PR 原有 13 个 | 首次建连各阶段、不重试、重定向、fd 泄漏、connect、http/https 代理 | +| `APooledTunnelCarriesNoTokenFromTheLastCall`，参数为直连 / http / https | I1 + I2，H1 的回归测试（一个 POST 只能执行一次） | +| `TheSecondCallsTokenReachesAReusedTunnel`，参数为直连 / http / https | I1 | +| `ASocks5ProxyThatNeverAnswersIsAbandoned` | 代理种类补齐 | +| `ADownloadWaitingForTheHeadersIsAbandoned` / `…ForTheBody…` / `…AlreadyStopped…` | D1 | + +## 4. CI 矩阵（目标） + +| job | 工具链 | 跑什么 | +|---|---|---| +| linux | gcc@16.1.0 / llvm@22.1.8 | 全部 `mcpp test`（`test_download`、`test_resolver` 需联网）+ 模板冒烟 | +| macos-15 | llvm@22.1.8 | 全部 `mcpp test` | +| windows-2022 | msvc / llvm@20.1.7 | `test_ca_store`，以及不联网的 `test_framing`、`test_pool`、`test_proxy`、`test_cancel`、`test_tls_verify` | +| openkal | llvm@22.1.8 + openkal-llvm-runtime | `examples/openkal` 在 x86_64-linux-gnu 上原生跑、x86_64-windows-musl 在 Linux 上构建、在 windows-2022 上原生跑（CI 装 wine 要 4–30 多分钟，且不稳定）| + +某个平台上测试编不过时，**修测试，不删平台**。确实做不到的，单独开 issue 记下原因，并在 job 里写清楚。 + +## 5. 过程中发现并修复 + +- **x86_64-windows-musl 上 TLS 从来没通过。** mbedTLS 只有识别到 glibc 才用 `getrandom`，否则读 `/dev/urandom`，而 openkal-windows 没有这个文件。在 musl 平台上改用 musl 的 `getentropy`（`cfg(c-abi = "musl")` → `TINYHTTPS_GETENTROPY`）🔁。 +- **Windows 目标没有名字解析**（openkal 没有解析器接口），已登记 [openkal-musl#46](https://github.com/mcpplibs/openkal-musl/issues/46)，README 已注明。 +- **`test_pool` 在 Windows 上编不过**（`` 的 `max` 宏），因为以前从没在 Windows 上编译过；已修复。 + +## 6. 验收 + +- [x] PR #23 的 CI 全绿（10 个 job，`10e28b0`） +- [x] 自审：H1、M1、M2、D1、D2 全部落地；PR 描述更新 +- [x] squash 合入（`156edeb`）；tag `0.3.4`；GitHub release +- [x] gitcode 上的 `tinyhttps-0.3.4.tar.gz` 与 GitHub archive 的 sha256 一致（`26e0e496…41e9`） +- [x] mcpp-index 合入（mcpplibs/mcpp-index#504）；`validate` 和 `openkal-compat`（tinyhttps 这一项，四个 target 均为 runs）通过 +- [x] 新建项目用已发布的 index 解析到 0.3.4（`"^0.3.1"`），GLOBAL 和 CN 两个源都能构建并运行，并对 httpbin 做了真实的进行中取消（301 ms） diff --git a/.agents/docs/exec-plan-0.3.5.md b/.agents/docs/exec-plan-0.3.5.md new file mode 100644 index 0000000..e903164 --- /dev/null +++ b/.agents/docs/exec-plan-0.3.5.md @@ -0,0 +1,171 @@ +# 执行计划 —— 0.3.5：额外信任的 CA，TLS 握手有时限，Windows CI 恢复 + +| | | +|---|---| +| 输入 | [PR #24](https://github.com/mcpplibs/tinyhttps/pull/24) `feat/extra-ca-certs @ bd76c9e` · [PR #25](https://github.com/mcpplibs/tinyhttps/pull/25) `fix/tls-handshake-timeout @ cfdab98`（作者都是 yspbwx2010，`maintainerCanModify: true`） | +| 基线 | `origin/master @ 2cc7310`（0.3.4） | +| 落地 | 两个 PR 各自 squash 合入；对 #25 的修正直接追加在 #25 上（已推送 `97a0657`） | +| 目标版本 | **0.3.5**（D1–D4 按建议采纳） | +| 跨仓库 | `mcpplibs/tinyhttps` → gitcode `mcpp-res/tinyhttps`（CN 镜像）→ `mcpplibs/mcpp-index` | +| 文档日期 | 2026-10-10 | + +**标注约定**：✅ 已核验源码 · 🔁 已在本地复现或跑通 · ❓ 推断，未实测 + +--- + +## 0. TL;DR + +- **两个 PR 的方向都对，设计也符合本项目的约定**：只新增、尾部默认参数、新字段放最后、错误写进 `statusText`、状态只放在传输链最底层的 `Socket`（沿用 0.3.4 的 I1）、测试不联网（hermetic）。和业界通行做法也一致（§2）。 +- **#24 可以直接合**，不需要追加 commit。剩下几处小问题不影响合入（§3.3）。 +- **#25 有一个回归，已修复并推到 PR 上** 🔁：handshake 的等待改成了 `poll`，信号会让它以 EINTR 返回，即使 handler 装的是 SA_RESTART 也一样。结果是装了信号 handler 的进程（profiler、定时器）握手会被误报成 “TLS handshake timed out”。修复前新加的用例在 100 ms 就失败了（时限是 1000 ms）。 +- **Windows CI 红是 runner 镜像的变化，和两个 PR 都无关** ✅：win22/20261004.326 带的是 Git for Windows 2.56，原来硬编码的 `mingw64/etc/ssl/certs/ca-bundle.crt` 不存在了。修复已随 #25 一起推送：向 git 查询 bundle 的路径，查不到再在安装目录里搜索。 +- **合入顺序是 #25 → #24**：#25 带着 CI 修复，先合入后 master 才能变绿；#24 再同步 master，同时解决 CHANGELOG 的冲突。 + +### 待你决定 + +| # | 决策 | 建议 | +|---|---|---| +| D1 | 版本号用 0.3.5 还是 0.4.0 | **0.3.5**。#25 严格说改变了行为（握手超过 `connectTimeoutMs` 会失败），不符合 0.3.4 “补丁版本只新增”的约定。但它修的是一个永久卡死的问题；而且 0.4.0 不会被 `^0.3.x` 解析到，最需要这个修复的正是已经在用 `^0.3.x` 的项目。在 CHANGELOG 里用 “Behaviour change” 单列说明 | +| D2 | `connectTimeoutMs` 继续按阶段各自计时，还是改成整个建连过程共用一个 deadline | **0.3.5 保持按阶段计时**（和现有的代理应答等待一致，PR 已在文档里写明）。共用 deadline 和 curl 的语义一样，更符合直觉，留到 0.4 统一做（§4） | +| D3 | #24 是否同时提供内存中的 PEM（如 `extraCaPem`） | **这次不做**，作为后续单独的一项。路径方式已经覆盖代理、企业 CA 这些主要场景 | +| D4 | `Socket::set_deadline / deadline_hit / wait_before_recv` 都是 public（因为整个类是 export 的） | **接受**。`set_stop / stop_possible` 已经是这样，跨 partition 用 friend 在各家编译器上的风险更大；注释已写明它们是给 TLS 层用的 | + +--- + +## 1. CI 失败分析（run 37911364113 / job 113757995241） + +| 项 | 内容 | +|---|---| +| 现象 | `run (x86_64-windows-musl, above openkal, on Windows)` 运行 4 秒后失败：`cp: cannot stat '/c/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt'` ✅ | +| 根因 | runner 镜像从 win22/20260927.320 换成了 win22/20261004.326，Git for Windows 也从 2.55 升到 2.56 ✅。2.56 把环境从 MINGW64 换成了 UCRT64，bundle 现在在 `/c/Program Files/Git/ucrt64/etc/ssl/certs/ca-bundle.crt`（新 CI 日志里打印出来的路径）✅ | +| 影响 | 所有分支的这个 job 都会失败，在 smoke 程序运行之前就挂了。和 #24、#25 的代码都无关 | +| 修复 | 已推送到 #25 的 `bf99dbd`：先用 `git config --system http.sslcainfo` 查路径（经 `cygpath` 转换），查不到就在 `/c/Program Files/Git` 下搜索 `*/ssl/certs/ca-bundle.crt`，还找不到就报 `::error::` 并给出明确原因。同时打印 `git --version` 和最终用到的路径，方便下次镜像再变时定位 | +| 验证 | #25 的 CI run 37968516327：10 个 job 全绿，这个 job 用 `git config` 查到了 ucrt64 下的路径；smoke 输出 `cancellation: ok (stopped at 200 ms, returned after 261 ms)` ✅ | +| 顺带发现 | 这个 job 里 `framing parsers: ok` 和 `cancellation: ok` 之间有约 39 s 的空档，10-05 master 的运行里也有，所以不是这两个 PR 引入的。空档出现在计时开始之前，原因没有查 ❓，列入 §4 | + +--- + +## 2. 设计评审 + +### 2.1 是否符合本项目的约定 + +| 约定（来源） | #24 | #25 | +|---|---|---| +| 只新增：尾部默认参数、新字段放最后（0.3.4 §0） | ✅ `extraCaFile` 放在最后；`proxy_tunnel` 加了尾部默认参数 | ✅ `connect_over` 加了尾部默认参数；CHANGELOG 点名说明了成员函数指针的不兼容 | +| 状态只放最底层 `Socket`，`TlsSocket` 只往下转发（0.3.4 I1） | 不涉及 | ✅ `set_deadline` / `deadline_hit` 都经 `lower_` 往下转发，https 代理隧道的情况有测试覆盖 | +| 状态只在调用期间存在（0.3.4 I2） | 不涉及 | ✅ 握手结束就清掉 deadline，有测试覆盖 | +| 不配置时行为不变（0.3.4 I3） | ✅ 字段为空时一行代码都不走 | ⚠️ 默认的 10 s 会生效，这是有意的行为变化（见 D1） | +| 失败原因写进 `statusText`，能直接看懂 | ✅ 错误信息里带文件路径 | ✅ 显示 `TLS handshake timed out`；代理那一跳还会带上代理地址 | +| 测试不联网，能在 Windows 上跑 | ✅ 已加进 Windows 的 hermetic 列表 | ⚠️ **原 PR 没有加**，现已补上（`bf99dbd`） | +| 不引入新的环境变量或全局状态 | ✅ 只增加一个字段 | ✅ 不新增配置项 | + +### 2.2 语义和业界做法对照 + +**#24 `extraCaFile`：在默认 store 之外再追加信任的 CA** + +| 实现 | 语义 | +|---|---| +| Node `NODE_EXTRA_CA_CERTS` | **追加**，和 #24 完全一样，名字也对应 | +| reqwest `add_root_certificate` | **追加**到内置根证书 | +| curl `--cacert` / `CURLOPT_CAINFO`、Python requests `verify=path`、Go `tls.Config.RootCAs` | **替换**（本库对应的是 `SSL_CERT_FILE`） | + +结论：库里原本只有“替换”的方式（`SSL_CERT_FILE`），#24 补上了“追加”，这是业界常见的另一半。文件不可用时直接失败、不跳过，这个判断对：跳过的话，错误最后会变成一个不提文件的证书校验失败。 + +**#25 握手时限并入 `connectTimeoutMs`** + +| 实现 | 握手的时限 | +|---|---| +| curl `CURLOPT_CONNECTTIMEOUT` | 包含 TLS 握手，并且整个建连阶段共用一个时限 | +| Go `http.Transport.TLSHandshakeTimeout` | 单独一项，`DefaultTransport` 默认 10 s | +| #25 | 包含握手，但**每个阶段各自计时** | + +结论：并入 `connectTimeoutMs`（不新增配置项）和 curl 一致，默认 10 s 也和 Go 的默认值相同。差别在于各阶段各自计时，最坏情况下总时间是 N × timeout。PR 和 README 都已写明（见 D2）。 + +### 2.3 API 好不好用 + +| 点 | 评价 | +|---|---| +| `cfg.extraCaFile = "/etc/corp/root-ca.pem";` 一行就能用 | 好。代理那一跳自动生效，不用再配置一次 | +| 只接受文件路径 | 证书打包进程序、或者从配置中心拿到的场景要先落盘再用（D3） | +| 只能指定一个文件 | 和 Node 一样，可以把多个证书拼成一个文件，够用 | +| `verifySsl=false` 时文件读不到也会失败 | 有意设计（所有机器上报同样的错），README 里再补一句说明会更好 | +| 握手超时不需要任何改动就生效 | 好，解决的是默认配置下的卡死问题 | +| `connect_over(..., handshakeTimeoutMs = -1)` 和 `connect(..., timeoutMs, ...)` 的默认值不对称 | 为了兼容只能这样：`connect_over` 原本就不带超时 | + +--- + +## 3. 问题清单与处理 + +### 3.1 #25（已推送：`cfdab98..97a0657`） + +| # | 级别 | 问题 | 处理 | +|---|---|---|---| +| H1 | 高 | EINTR 被当成 deadline 到期。`wait_before_recv` 只要等待失败就设置 `deadline_hit_`，不看时钟。没有 token 时只调用一次 `poll`，信号一来就返回。master 上走的是阻塞的 `recv`，在 SA_RESTART 下会自动重启，所以这是回归 🔁 | `3013b98`：提前返回就按剩余时间继续等，只有时钟确实到了才算 deadline 到期。新用例 `ASignalDuringTheHandshakeIsNotATimeout`：时限 1000 ms，每 100 ms 用 `pthread_kill` 给握手线程发一次 SIGUSR1。修复前 100 ms 就失败，修复后通过 🔁 | +| M1 | 中 | `test_handshake_timeout` 没进 Windows 的 hermetic 列表，`WSAPoll` 路径没人测 | `bf99dbd` | +| M2 | 中 | Windows openkal job 的 bundle 路径（§1） | `bf99dbd` | +| L1 | 低 | CHANGELOG 没提导出的 `TlsSocket::connect` 现在也会限制握手时间 | `97a0657` | +| L2 | 低 | 移动构造和移动赋值复制了 `deadline_`，但没复制 `deadline_hit_` | `3013b98` | +| — | 已知限制 | 握手期间的 `send` 是阻塞的；`readTimeoutMs` 管不到阻塞的写（README 里的说法和实际不符） | 不在本版本处理，单列一项（§4） | + +本地结果（Linux, LLVM 22）🔁：`test_handshake_timeout` 9/9、`test_cancel` 24、`test_proxy` 43、`test_pool` 27、`test_tls_verify` 6、`test_framing` 17，全部通过。 + +### 3.2 #24：可以直接合入，不追加 commit + +本地结果 🔁：`test_extra_ca` 11/11，其余测试不受影响。#24 和 #25 合在一起后，全部测试也都通过（`test_extra_ca` + `test_handshake_timeout` 一起跑过）。 + +### 3.3 #24 的小问题（合入后视情况再处理，不阻塞本版本） + +- 在 Linux 上，路径是目录时 `ifstream` 也能打开，错误信息变成 “cannot parse”，不是 “cannot read”。加一个 `is_regular_file` 判断就能修。 +- 部分证书解析失败（`ret > 0`）不会报错，和默认 bundle 的处理一致，README 可以补一句。 +- 每建一个新连接都要重新读、重新解析一次文件。默认 bundle 也是这样，暂时不缓存。 + +--- + +## 4. 0.3.5 之后（不在本版本范围内） + +| 项 | 说明 | +|---|---| +| 写操作不阻塞 | `bio_send` / `write_all` 在阻塞 socket 上永远拿不到 WANT_WRITE，所以 `readTimeoutMs` 管不到阻塞的写，握手期间的写也不受 deadline 约束。这是 0.3.4 D4 留下的问题 | +| 整个建连共用一个 deadline（D2） | DNS、TCP（每个地址）、代理应答、两次握手共用一个 `connectTimeoutMs`。行为有变化，适合 0.4 | +| `extraCaPem`（D3） | 直接从内存加载 PEM | +| openkal Windows smoke 的 39 s 空档 | 见 §1，master 上已有，原因待查 | +| `poll_fd` 统一处理 EINTR | TCP connect 和代理应答的等待也有同样的问题，H1 只修了握手。可以在 `poll_fd` 一处统一修 | + +--- + +## 5. 任务依赖图 + +``` +T0 计划（本文）← 你在这里审 + │ + ├─ T1 #25 追加 commit（已推送 97a0657）→ CI 全绿 ✅（run 37968516327） + │ + ├─ T2 更新 #25 的 PR 描述：Not covered 里关于 EINTR 的说法改成“已修复”；补充 CI 的变化 + │ + ├─ T3 squash 合入 #25（依赖 T1、T2） + │ + ├─ T4 #24 同步 master：合并两边 CHANGELOG 的 “Unreleased” 段 → CI 全绿 → squash 合入 + │ + ├─ T5 发版 commit：CHANGELOG 的 Unreleased → 0.3.5；mcpp.toml version；README 安装行（mcpp add / 依赖示例） + │ + └─ T6 发布（依赖全部） + tag 0.3.5 + GitHub release + → gitcode mcpp-res/tinyhttps 上传 0.3.5 tarball（核对和 GitHub archive 的 sha256 一致） + → mcpp-index PR：pkgs/t/tinyhttps.lua 加 0.3.5（GLOBAL / CN，三个平台）， + tests/examples/tinyhttps 升到 0.3.5，加一条 `extraCaFile` 的 API 检查 + → 用已发布的 index 实测：新建项目写 "^0.3.1" 能解析到 0.3.5，GLOBAL 和 CN 两个源都验证 +``` + +**关键路径**：T1（CI）→ T3 → T4 → T5 → T6。 + +## 6. 验收 + +- [x] #24、#25 以及两者合并后，hermetic 测试在本地全部通过 🔁 +- [x] H1 能复现，修复后用例通过 🔁；已推送到 #25 +- [x] #25 CI 全绿（run 37968516327，10/10），包括 Windows openkal 和 Windows hermetic（`test_handshake_timeout`） +- [x] #25 squash 合入（`9d4315e`）；#24 同步 master（合并时 CHANGELOG 和 Windows 测试列表都有冲突）后 CI 10/10 全绿，squash 合入（`a0f7749`） +- [x] 版本号 0.3.5（D1）；发版 commit `6153cdd` 的 master CI 10/10 全绿；tag `0.3.5`，GitHub release +- [x] gitcode `mcpp-res/tinyhttps` 0.3.5 的 tarball 和 GitHub archive 的 sha256 一致（`6f75f9e6…58a3`） +- [x] mcpp-index 合入（mcpplibs/mcpp-index#517，`bf7e87c`）：PR 的全部检查通过，包括 openkal measure 和 linux/macos/windows workspace；合入后 Publish Index Artifact、validate、deploy-site 也都通过 +- [x] 真实验证：用已发布的 index 新建项目，依赖写 `"^0.3.1"`，在 GLOBAL 和 CN 两个源下（都清了缓存）都解析到 0.3.5。真实 HTTPS 请求返回 200；握手不回应的本地对端在 `connectTimeoutMs=500` 下约 520 ms 报 `TLS handshake timed out`；`extraCaFile` 文件不存在时，报错里带路径；指向系统 bundle 时请求正常返回 200 +- 注：mcpp 在一个源失败时会自动换另一个源，所以“切到某个源跑通”证明不了用的是哪个源。两个 URL 内容一致，是直接下载后比对 sha256 得出的 diff --git a/.agents/docs/fix-plan-issue-15-16.md b/.agents/docs/fix-plan-issue-15-16.md index 0343ca6..7f5a45b 100644 --- a/.agents/docs/fix-plan-issue-15-16.md +++ b/.agents/docs/fix-plan-issue-15-16.md @@ -7,9 +7,9 @@ | 关联 issue | [#15](https://github.com/mcpplibs/tinyhttps/issues/15) 连接池污染 · [#16](https://github.com/mcpplibs/tinyhttps/issues/16) SIGPIPE | | 文档日期 | 2026-09-06 | -> ⚠️ **动手前先 `git pull`。** 当前工作区在 `626c9d0`(0.2.8),落后 origin/master 两个提交(`965d805` #9、`2cec1c1` #14)。本文所有行号对应 **0.2.10**;本地 `src/http.cppm` 只有 1032 行,对不上。 +> **动手前先 `git pull`。** 当前工作区在 `626c9d0`(0.2.8),落后 origin/master 两个提交(`965d805` #9、`2cec1c1` #14)。本文所有行号对应 **0.2.10**;本地 `src/http.cppm` 只有 1032 行,对不上。 -**标注约定**:✅ = 已逐行核验源码 · 🔬 = 已独立复现 · ⚠️ = 推断,未实测 +**标注约定**:= 已逐行核验源码 · = 已独立复现 · = 推断,未实测 --- @@ -34,20 +34,20 @@ ## 1. 问题全景 -### 1.1 三个入口的加固覆盖矩阵 ✅ +### 1.1 三个入口的加固覆盖矩阵 库里有三份几乎平行的响应读取实现:`send_impl`、`send_stream`、`download_to_file_impl`。历次修复各自只落到其中一两份: | 加固措施 | 引入自 | `send_impl` | `send_stream` | `download_to_file_impl` | |---|---|---|---|---| -| `parse_content_length`(严格) | #14 | ✅ `:516` | ✅ `:783` | ❌ `:1042-1048` 手写摘数字 | -| `contentLength` 用 `int64_t` | — | ❌ `:488` **`int`** | ✅ `:756` | ✅ `:1024` | -| `parse_chunk_size_line`(严格) | #9 | ❌ `:541` 仍是 `parse_hex` | ✅ `:831` | ✅ `:1134` | -| `read_complete_line`(区分超时/EOF) | #9 | ❌ | ❌ | ✅ `:1118/1145/1177` | -| 块后 CRLF 校验 | #9 | ❌ `:556` 丢弃返回值 | ❌ `:850` 丢弃返回值 | ✅ `:1177-1185` | -| 状态行 `HTTP/` 前缀校验 | — | ❌ | ❌ | ❌ | -| 早退时清理连接池 | — | ❌ 2 处 | ❌ 3 处 | ❌ **5 处** | -| 失败时能告诉调用方 | — | ❌ 无字段 | ❌ 无字段 | ✅ `result.error` | +| `parse_content_length`(严格) | #14 | `:516` | `:783` | `:1042-1048` 手写摘数字 | +| `contentLength` 用 `int64_t` | — | `:488` **`int`** | `:756` | `:1024` | +| `parse_chunk_size_line`(严格) | #9 | `:541` 仍是 `parse_hex` | `:831` | `:1134` | +| `read_complete_line`(区分超时/EOF) | #9 | no | | `:1118/1145/1177` | +| 块后 CRLF 校验 | #9 | `:556` 丢弃返回值 | `:850` 丢弃返回值 | `:1177-1185` | +| 状态行 `HTTP/` 前缀校验 | — | no | | no | +| 早退时清理连接池 | — | 2 处 | 3 处 | **5 处** | +| 失败时能告诉调用方 | — | 无字段 | 无字段 | `result.error` | **这张表本身就是最强的论据**:每一次修复都是在一份拷贝上打补丁,另外两份继续带病。`#14` 引入 P1-A1/A2 两处新回归,正是这个模式的最新一次复发。**不做 P3,下一次还会有。** @@ -66,7 +66,7 @@ R2/R3 是同一个毛病的两个实例:**用一个哨兵值(空串 / 0)表示 ## 2. P0 —— SIGPIPE(issue #16) -### 2.1 判定 🔬 +### 2.1 判定 **真 bug,最高优先级。** 一个库在自己管理的 fd 上调 `send()`,却让宿主进程被 `SIGPIPE` 杀死(exit 141),是库的责任。 @@ -137,7 +137,7 @@ R2/R3 是同一个毛病的两个实例:**用一个哨兵值(空串 / 0)表示 **平台矩阵**:Linux/Android 走 `MSG_NOSIGNAL`;macOS/BSD 走 `SO_NOSIGPIPE`;Windows 两个宏都不存在 → 自动走 `#else`,且本来就没有 `SIGPIPE`,无需额外分支。`socket.cppm:8-9` 已经在非 Windows 下 include 了 ``,两个宏都在里面,**不需要新增 include**。 -### 2.3 为什么这个修复是自包含的 ✅ +### 2.3 为什么这个修复是自包含的 失败会沿着**已经存在的**路径传播,零新增分支: @@ -154,16 +154,16 @@ Socket::write → -1 (EPIPE) ## 3. P1 —— 连接池不变量(issue #15 + 4 处新发现) -### 3.1 判定 ✅ +### 3.1 判定 **真 bug,不是使用方法错误。** | 报告者做的 | 是否合法 | |---|---| -| 单个 `HttpClient` 跑多个请求 | ✅ 这正是连接池的用途 | -| `keepAlive` 用默认值(true) | ✅ 库的默认值 | -| `readTimeoutMs = 1000` | ✅ 激进但合法;**超时是正常事件,不是 API 误用** | -| 触发条件:服务端发一半 / 空闲后关连接 | ✅ **服务端的常规行为** | +| 单个 `HttpClient` 跑多个请求 | 这正是连接池的用途 | +| `keepAlive` 用默认值(true) | 库的默认值 | +| `readTimeoutMs = 1000` | 激进但合法;**超时是正常事件,不是 API 误用** | +| 触发条件:服务端发一半 / 空闲后关连接 | **服务端的常规行为** | 规避手段只有「关掉 keep-alive」或「每次换 client」—— 都是放弃这个特性本身。**用户无法在正确使用 API 的前提下绕开 → 定义上就是库的 bug。** @@ -171,7 +171,7 @@ Socket::write → -1 (EPIPE) `connectionClose` 列 = 走到清理点时该标志的值。 -#### A. `send_stream` —— 🔴 **0.2.10 新引入的回归** +#### A. `send_stream` —— **0.2.10 新引入的回归** | ID | 位置 | 触发 | 现状 | 修复 | |---|---|---|---|---| @@ -179,7 +179,7 @@ Socket::write → -1 (EPIPE) | **A2** | `:884-885` | `read() <= 0` → `break` | `false` → 同上 | 设 `connectionClose = true` | | **A3** | `:846-848` | `read_exact` 失败 → `break` | `false` → 同上 | 设 `connectionClose = true` | -**回归定位** ✅:`git log -S "A DECLARED LENGTH IS READ AND THE READER THEN STOPS"` → `2cec1c1` (#14)。0.2.3 的对应分支是: +**回归定位** :`git log -S "A DECLARED LENGTH IS READ AND THE READER THEN STOPS"` → `2cec1c1` (#14)。0.2.3 的对应分支是: ```cpp } else { @@ -209,13 +209,13 @@ B2 细节:`parse_hex`(`:233-243`)遇到不认识的字符是 `break` 返回**已 |---|---|---|---|---| | **C1** | `:1166-1171` | chunk 数据 `read_exact` 失败 → **直接 return** | 跳过 `:1223` 尾声 | issue #15 | | **C2** | `:1195-1200` | CL body `read_exact` 失败 → **直接 return** | 同上 | issue #15 | -| **C3** | `:1060-1075` | **重定向** | 🆕 **正常路径** | 本次发现 | -| **C4** | `:1077-1081` | **非 2xx 返回** | 🆕 **正常路径** | 本次发现 | -| **C5** | `:1092-1096` | 输出文件打不开 | 🆕 | 本次发现 | +| **C3** | `:1060-1075` | **重定向** | **正常路径** | 本次发现 | +| **C4** | `:1077-1081` | **非 2xx 返回** | **正常路径** | 本次发现 | +| **C5** | `:1092-1096` | 输出文件打不开 | | 本次发现 | C1/C2 的对照:同一个函数里相邻的 `:1150-1153` 和 `:1180-1184` **写对了**(`sock->close(); pool_.erase(poolKey);`),这本身就证明是漏写而非取舍。 -##### 🆕 C3 是本次分析最值得注意的一条 +##### C3 是本次分析最值得注意的一条 ```cpp 1062: // Drain any body to keep connection clean @@ -239,7 +239,7 @@ C1/C2 的对照:同一个函数里相邻的 `:1150-1153` 和 `:1180-1184` **写 C4 同理:非 2xx 时 error body 未读就返回,socket 回池,同一 client 的下个请求中招。 C5 同理:文件打不开时 body 未读。 -#### D. 🆕 整数截断 +#### D. 整数截断 | ID | 位置 | 问题 | |---|---|---| @@ -280,7 +280,7 @@ C5 同理:文件打不开时 body 未读。 + if (ret <= 0) { connectionClose = true; break; } ``` -> ✅ 已核验:`:909-912` 的清理本来就 key 在 `connectionClose || stopped` 上;正常终止路径(`remaining > 0` 循环条件自然结束、`:841` 终止块 break)**不经过**这三个出口。**成功路径零变化。** +> 已核验:`:909-912` 的清理本来就 key 在 `connectionClose || stopped` 上;正常终止路径(`remaining > 0` 循环条件自然结束、`:841` 终止块 break)**不经过**这三个出口。**成功路径零变化。** #### P1-B `src/http.cppm` —— `send_impl` 两处 @@ -380,7 +380,7 @@ C3 / C4 / C5 —— 最小改法是**无条件丢弃连接**(见 P2-E1 有更省 } ``` -> 📌 **review 决策点**:C3 用「无条件丢连接」换正确性,代价是每次重定向多一次 TLS 握手。若在意,P2-E1 提供一个有上限的 drain 助手,可把这三处改回「drain 成功就留、失败就丢」。**建议先合无条件版本(正确性优先),drain 作为后续优化。** +> **review 决策点**:C3 用「无条件丢连接」换正确性,代价是每次重定向多一次 TLS 握手。若在意,P2-E1 提供一个有上限的 drain 助手,可把这三处改回「drain 成功就留、失败就丢」。**建议先合无条件版本(正确性优先),drain 作为后续优化。** #### P1-D `src/http.cppm:488` —— 类型对齐 @@ -399,7 +399,7 @@ C3 / C4 / C5 —— 最小改法是**无条件丢弃连接**(见 P2-E1 有更省 > P2 的价值:**即使 P1 将来又出漏洞,残留字节也不会被伪装成一个合法响应。** #15 里最危险的两档(静默数据损坏)都会退化成可见错误。 -### P2-C1 状态行必须以 `HTTP/` 开头 ✅ 🆕 +### P2-C1 状态行必须以 `HTTP/` 开头 三个入口(`:456-484`、`:725`、`:1010-1020`)**都不校验前缀**,状态码提取还是「挑数字、静默跳过非数字」: @@ -437,7 +437,7 @@ static std::optional parse_status_line(std::string_view line) { 三个入口统一改用它,失败一律 `pool_.erase` + `Invalid status line`。**导出它,理由和 `parse_chunk_size_line` 一样:这是 framing 里可以脱离服务端单测的那一半。** -### P2-C2 header 循环区分「超时」与「真空行」✅ +### P2-C2 header 循环区分「超时」与「真空行」 `:491-495` / `:758-762` / `:1030-1032` 三处都是: @@ -452,11 +452,11 @@ if (headerLine.empty()) break; // "End of headers" 修复:三处都换成 `read_complete_line`(`:193-213`,返回 `expected`),`unexpected` → 明确报错并丢连接。**根因 R2 的正解,`#9` 已经写好了工具,只是没铺开。** -### P2-C3 块后 CRLF 校验 ✅ +### P2-C3 块后 CRLF 校验 `send_impl:556` 和 `send_stream:850` 都是 `read_line(...)` 丢弃返回值;`download_to_file_impl:1177-1185` 是校验的(`"Missing CRLF after chunk data"`)。块长度算错时前两者会直接错位到下一个块头而不报错。照 download 的写法补齐。 -### P2-C4 `download_to_file_impl` 的 Content-Length 解析 ✅ +### P2-C4 `download_to_file_impl` 的 Content-Length 解析 `:1042-1048` 仍是手写摘数字循环: @@ -482,14 +482,14 @@ static bool drain_body(TlsSocket& sock, bool chunked, std::int64_t contentLength 有了它,C3/C4 可以写成 `if (!drain_body(...)) { sock->close(); pool_.erase(poolKey); }`。 -### P2-E2 分配上限(拒绝服务端诱导的巨额分配)🆕 ⚠️ +### P2-E2 分配上限(拒绝服务端诱导的巨额分配) - `send_impl:561` `response.body.resize(contentLength)` —— `contentLength` 最大到 `INT64_MAX`,`resize` 会抛 `std::length_error` / `std::bad_alloc`。**库没有文档说 `send()` 会抛异常**,调用方大概率没接。 - `send_impl:549` / `send_stream:845` `std::string chunkData(chunkSize, '\0')` —— 服务端一个 `7fffffff` 块头就能让客户端分配 2 GiB。 建议:加一个 `HttpClientConfig::maxResponseBodyBytes`(默认比如 64 MiB),超过就当作 framing 错误拒绝;chunk 数据改成按固定缓冲区分片读,不按块大小一次性分配。 -### P2-D1 `TlsSocket::read` 区分 EOF 与 would-block(根因 R3)✅ +### P2-D1 `TlsSocket::read` 区分 EOF 与 would-block(根因 R3) `tls.cppm:120-134`:`PEER_CLOSE_NOTIFY`、`ret == 0`、`WANT_READ`、`WANT_WRITE` **全部返回 0**。调用方(`read_exact:157`、`read_line:178`、`send_stream:885`)因此分不清「对端关了」和「暂时没数据」,只好用「等一下再试一次」的土办法遮盖。 @@ -501,9 +501,9 @@ enum class ReadStatus { Data, WouldBlock, Eof, Error }; 或保留 `int` 但用 `-2` 表示 would-block。 -> **与 P1 的关系**:P1-A2 把 `read() <= 0` 当成「流结束」丢连接。若那个 0 其实是瞬时 would-block,我们只是白丢一条连接 —— **正确性不受影响,只损失一点复用率**。所以 P1 不依赖 P2-D1,但修了 D1 之后 P1-A2 才精确。⚠️ 实践中 socket 在握手后被设回阻塞模式(`socket.cppm`),`WANT_*` 极少出现,影响应该很小,**未实测**。 +> **与 P1 的关系**:P1-A2 把 `read() <= 0` 当成「流结束」丢连接。若那个 0 其实是瞬时 would-block,我们只是白丢一条连接 —— **正确性不受影响,只损失一点复用率**。所以 P1 不依赖 P2-D1,但修了 D1 之后 P1-A2 才精确。实践中 socket 在握手后被设回阻塞模式(`socket.cppm`),`WANT_*` 极少出现,影响应该很小,**未实测**。 -### P2-D2 `write_all` 的 WANT_WRITE 处理 ✅ ⚠️ +### P2-D2 `write_all` 的 WANT_WRITE 处理 `http.cppm:216-230`: @@ -516,9 +516,9 @@ if (ret == 0) { } ``` -`TlsSocket::write`(`tls.cppm:140-142`)在 `WANT_READ/WANT_WRITE` 时返回 0。所以 TLS 背压下会「忙重试一次然后放弃」→ 用户看到莫名其妙的 `"Write failed"`。应该改成 `wait_writable(timeoutMs)` 后重试,并给总时长设上限。⚠️ **未实测,按代码推断。** +`TlsSocket::write`(`tls.cppm:140-142`)在 `WANT_READ/WANT_WRITE` 时返回 0。所以 TLS 背压下会「忙重试一次然后放弃」→ 用户看到莫名其妙的 `"Write failed"`。应该改成 `wait_writable(timeoutMs)` 后重试,并给总时长设上限。**未实测,按代码推断。** -### P2-F1 🆕 调用方无法得知 body 被截断了(API 缺口)✅ +### P2-F1 调用方无法得知 body 被截断了(API 缺口) **这是 P1 修完之后剩下的最大问题。** @@ -526,7 +526,7 @@ P1 只保证了「坏连接不会毒害下一个请求」,但当前请求返回 - `send_impl` CL 短读(`:562-565`)→ 返回真实的 `statusCode=200` + 半截 body,**没有任何截断标记** - `send_impl` / `send_stream` chunked 中断 → 同上 -- `DownloadToFileResult` **有** `error` 字段 ✅,`HttpResponse` **没有** ❌ +- `DownloadToFileResult` **有** `error` 字段 ,`HttpResponse` **没有** 也就是说:**一个被截断的 200 响应,和一个完整的 200 响应,调用方分不出来。** @@ -554,9 +554,9 @@ export struct HttpResponse { 在 P1 设 `connectionClose = true` 的每一处,同时设 `response.bodyComplete = false`。 -> 📌 **review 决策点**:(a) 只加 `bodyComplete` 字段、`ok()` 不变(建议,非破坏性);(b) 让 `ok()` 也要求 `bodyComplete`(更安全,但会让现有代码行为变化)。 +> **review 决策点**:(a) 只加 `bodyComplete` 字段、`ok()` 不变(建议,非破坏性);(b) 让 `ok()` 也要求 `bodyComplete`(更安全,但会让现有代码行为变化)。 -### P2-F2 🆕 `send_stream` 出错时会覆盖真实的 statusText ✅ +### P2-F2 `send_stream` 出错时会覆盖真实的 statusText `:833` `response.statusText = "Invalid chunk size: " + sizeLine;` —— 把服务端真实的 `"OK"` 覆盖掉了。有了 P2-F1 的 `bodyComplete` 之后,应该改成保留 `statusText`,把原因放进新字段(或一个 `bodyError` 字符串)。 @@ -566,7 +566,7 @@ export struct HttpResponse { > P1 是逐条堵漏。**但 §1.1 的矩阵说明:这个 bug 类会随着每次新增代码复发。** `#14` 就是最新一次。P3 的目标是让它写不出来。 -### P3-A `PooledConn` —— 把默认行为从「留」翻成「丢」 ★ 最高性价比 +### P3-A `PooledConn` —— 把默认行为从「留」翻成「丢」 最高性价比 当前所有 10 个 bug 的形状**完全一样**:一条早退路径**什么都没做**,而「什么都没做」的默认结果是**把脏连接留在池里**。 @@ -601,7 +601,7 @@ private: 用法:每个入口开头建一个,只在 body **正常读完**的那唯一一处调 `keep()`。10 个 bug 一次全消,而且以后新增任何早退路径**天然安全**。 -> ⚠️ **两个必须注意的陷阱(review 重点)**: +> **两个必须注意的陷阱(review 重点)**: > > 1. **递归重定向**。`send_impl:612` `return send_impl(redirectReq, redirectCount + 1);` 发生在 guard 析构**之前**。若重定向到同一 host,内层调用会往池里放一条新连接,外层 guard 析构时会把它**误删**。 > → 递归前必须先 `guard.keep()` 或 `guard.release()`。`download_to_file_impl:1073` 同理。 @@ -630,7 +630,7 @@ static BodyEnd read_body(TlsSocket&, const ResponseHead&, int timeoutMs, ### P3-C 带缓冲的读 -`read_line` 现在是逐字节 `sock.read(&c, 1)`。除了慢(⚠️ 影响有限:`TlsSocket::wait_readable` 会先查 `mbedtls_ssl_get_bytes_avail`,所以并不是每字节一次 `poll()`,主要成本是每字节一次函数调用 —— **未做基准测试,不要当成性能修复来卖**),更重要的是: +`read_line` 现在是逐字节 `sock.read(&c, 1)`。除了慢(影响有限:`TlsSocket::wait_readable` 会先查 `mbedtls_ssl_get_bytes_avail`,所以并不是每字节一次 `poll()`,主要成本是每字节一次函数调用 —— **未做基准测试,不要当成性能修复来卖**),更重要的是: **一个显式的读缓冲,才能让「这条连接干净吗」这个问题真正可回答**(= 缓冲区空 + body 已按 framing 读完)。现在这个问题只能靠人肉追踪 10 条路径。这是 P3-C 的主要理由,性能是副产品。 @@ -655,20 +655,20 @@ P0+P1 只是把「进程被杀」降级成「一个莫名其妙的 `No response` 丢弃连接 → 新建连接 → 重发一次(仅一次) ``` -> 📌 **review 决策点 —— 要不要限制到幂等方法?** +> **review 决策点 —— 要不要限制到幂等方法?** > > - Go 的策略保守:只重试幂等方法(GET/HEAD/OPTIONS/TRACE)或带 `Idempotency-Key` 的请求。 > - curl 的策略宽松:只要连接是复用的就重试,不看方法。 > > **我倾向 curl 的做法**,理由:(1) tinyhttps 的主力场景是 LLM / API 客户端,POST 是绝对多数,限制到幂等方法等于这个修复基本没用;(2) 重试窗口极窄 —— 只在**一个响应字节都没收到**时才重试,此时服务端已经发过 FIN,应用层根本没看到这个请求;(3) 加一个 `HttpClientConfig::retryOnStaleConnection`(默认 true)让保守用户能关掉。 > -> ⚠️ 残留风险:服务端「收到并处理了请求 → 还没回响应就崩了/关了」这种情况下会重复执行。窄,但非零。**这一条请你拍板。** +> 残留风险:服务端「收到并处理了请求 → 还没回响应就崩了/关了」这种情况下会重复执行。窄,但非零。**这一条请你拍板。** --- ## 6. 测试方案 -### 6.1 现状 ✅ +### 6.1 现状 `tests/` 下只有 `test_download.cpp` 和 `test_resolver.cpp`,**零 keep-alive / 连接复用覆盖** —— 这就是 `#14` 的回归能全绿合进来的直接原因。 @@ -679,7 +679,7 @@ P0+P1 只是把「进程被杀」降级成「一个莫名其妙的 `No response` | # | 用例 | 断言 | 覆盖 | |---|---|---|---| | T1 | 服务端发一半 CL body 后挂住 | 请求 2 `statusCode == 200`(不是 999/0/Invalid) | A1 | -| T2 | **同上,并断言服务端 TCP 连接数 == 2** | 🔴 **唯一能抓住「50 个 B」那种「看起来对了」的 case 的断言** | A1 | +| T2 | **同上,并断言服务端 TCP 连接数 == 2** | **唯一能抓住「50 个 B」那种「看起来对了」的 case 的断言** | A1 | | T3 | chunked 版本同上 | 同 T1 + T2 | A3 | | T4 | 服务端发一半后关连接 | 请求 2 不崩 + 断言**进程未被信号杀死** | #16 | | T5 | **`SIGPIPE` 保持默认处置**跑 T4(子进程) | `WIFSIGNALED == false`,退出码 != 141 | P0 | @@ -759,13 +759,13 @@ gcc -o sigpipe_probe sigpipe_probe.c && ./sigpipe_probe | P1-B2 | P1 | `http.cppm:541` | `parse_hex` → `parse_chunk_size_line`(并删除 `parse_hex`) | | P1-C1 | P1 | `http.cppm:1166-1171` | download chunk 短读 → `close + erase` | | P1-C2 | P1 | `http.cppm:1195-1200` | download CL 短读 → `close + erase` | -| P1-C3 | P1 | `http.cppm:1060-1075` | 🆕 **重定向未 drain body** | -| P1-C4 | P1 | `http.cppm:1077-1081` | 🆕 非 2xx 未 drain body | -| P1-C5 | P1 | `http.cppm:1092-1096` | 🆕 文件打开失败未 drain body | -| P1-D1 | P1 | `http.cppm:488` | 🆕 `int contentLength` → `int64_t`(整数截断) | -| P2-C1 | P2 | 三处状态行 | 🆕 校验 `HTTP/` 前缀 + 严格状态码 | +| P1-C3 | P1 | `http.cppm:1060-1075` | **重定向未 drain body** | +| P1-C4 | P1 | `http.cppm:1077-1081` | 非 2xx 未 drain body | +| P1-C5 | P1 | `http.cppm:1092-1096` | 文件打开失败未 drain body | +| P1-D1 | P1 | `http.cppm:488` | `int contentLength` → `int64_t`(整数截断) | +| P2-C1 | P2 | 三处状态行 | 校验 `HTTP/` 前缀 + 严格状态码 | | P2-C2 | P2 | 三处 header 循环 | 用 `read_complete_line` 区分超时/空行 | | P2-C3 | P2 | `:556` `:850` | 校验块后 CRLF | | P2-C4 | P2 | `:1042-1048` | download 换 `parse_content_length` | -| P2-E2 | P2 | `:561` `:549` `:845` | 🆕 分配上限,防 2 GiB 分配 / 未文档化的异常 | -| P2-F1 | P2 | `HttpResponse` | 🆕 加 `bodyComplete` —— 调用方现在无法得知 body 被截断 | +| P2-E2 | P2 | `:561` `:549` `:845` | 分配上限,防 2 GiB 分配 / 未文档化的异常 | +| P2-F1 | P2 | `HttpResponse` | 加 `bodyComplete` —— 调用方现在无法得知 body 被截断 | diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8104eea..1da4c4c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -31,15 +31,30 @@ on: workflow_dispatch: env: - MCPP_VERSION: 2026.8.29.1 + # 2026.9.30.2 and not earlier: 2026.9.28.3 refused x86_64-windows-musl + # ("cannot be built on this host") before the dependency graph that supplies + # its system side, openkal-llvm-runtime, had been resolved. + MCPP_VERSION: 2026.9.30.2 XLINGS_VERSION: v2026.8.17.2 XLINGS_NON_INTERACTIVE: '1' jobs: + # ── Linux, under both standard libraries ───────────────────────────────────── + # + # WHY TWO TOOLCHAINS. The job used to run whatever mcpp installed by default on + # a fresh runner, which is gcc and libstdc++, and nothing ran the tests under + # libc++. libc++ is the standard library of every other job below — macOS and + # openkal — and of mcpp's own default where a developer has chosen llvm; the + # cancellation tests met a libc++ defect that libstdc++ does not have + # (tests/test_cancel.cpp, at the top), and no job here could have seen it. build: - name: build + test (linux x86_64, mcpp) + name: build + test (linux x86_64, ${{ matrix.toolchain }}) runs-on: ubuntu-latest timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + toolchain: ['gcc@16.1.0', 'llvm@22.1.8'] steps: - uses: actions/checkout@v4 @@ -67,21 +82,28 @@ jobs: mcpp self config --mirror GLOBAL # The toolchain mcpp bootstraps is decided by the version pinned above, so - # that version is the whole of the key. + # that version and the toolchain are the whole of the key. - name: Cache mcpp sandbox uses: actions/cache@v4 with: path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry - key: mcpp-sandbox-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }} + key: mcpp-sandbox-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-${{ matrix.toolchain }} + + - name: Select the toolchain + run: | + spec='${{ matrix.toolchain }}' + mcpp toolchain install "${spec%@*}" "${spec#*@}" + mcpp toolchain default "$spec" + mcpp toolchain list - name: Build with mcpp run: mcpp build - # Most of these are hermetic: `test_framing` needs nothing, and `test_pool` - # scripts its own TLS server on 127.0.0.1. The eight in `test_download` - # and `test_resolver` reach httpbin.org and one.one.one.one; a network - # failure among them is distinguishable in the log by the endpoint it - # names. + # Most of these are hermetic: `test_framing` needs nothing, and `test_pool`, + # `test_proxy` and `test_cancel` script their own servers on 127.0.0.1. The + # eight in `test_download` and `test_resolver` reach httpbin.org and + # one.one.one.one; a network failure among them is distinguishable in the + # log by the endpoint it names. - name: Run tests run: mcpp test @@ -95,6 +117,52 @@ jobs: - name: Smoke-test the project templates run: bash tools/template_smoke.sh + # ── macOS ──────────────────────────────────────────────────────────────────── + # + # THE README NAMES macOS, AND UNTIL THIS JOB NOTHING RAN THERE. The + # differences are real and in this library's own code: SO_NOSIGPIPE rather + # than MSG_NOSIGNAL (src/platform.cppm), /dev/urandom as mbedTLS's entropy, + # the system CA bundle's location, and poll(2) on a socket whose connect is in + # progress, which the cancellation slices wait on. + macos: + name: build + test (macos arm64, llvm@22.1.8) + runs-on: macos-15 + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + + - name: Install xlings + run: | + curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 \ + https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh \ + | bash -s "$XLINGS_VERSION" + echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" + + - name: Install mcpp + run: | + xlings update + xlings install "mcpp@$MCPP_VERSION" -y -g + mcpp --version + mcpp self config --mirror GLOBAL + + - name: Cache mcpp sandbox + uses: actions/cache@v4 + with: + path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry + key: mcpp-sandbox-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-llvm@22.1.8 + + - name: Select the toolchain + run: | + mcpp toolchain install llvm 22.1.8 + mcpp toolchain default llvm@22.1.8 + mcpp toolchain list + + - name: Build with mcpp + run: mcpp build + + - name: Run tests + run: mcpp test + # ── The same sources, on a different kernel ABI ────────────────────────────── # # WHY THIS IS A SEPARATE JOB AND NOT A STEP. It resolves a different toolchain @@ -109,10 +177,24 @@ jobs: # openkal-musl defines the first and not the second. A `#ifdef` that is wrong # about that compiles cleanly here and fails there, which is exactly how # 5e7d66f reached master. + # + # FOUR TARGETS, THE ONES mcpp-index MEASURES tinyhttps ON (its + # tests/openkal/pins.toml). aarch64-linux-musl runs under qemu, through the + # runner examples/openkal names. x86_64-windows-musl is built here and run on + # Windows itself by the job after this one: wine took anywhere from four to + # more than thirty minutes to install on these runners, and Windows is the + # system the program is for. The Windows target is where every handshake used + # to fail for want of entropy (src/tls.cppm); it has no name resolution, so its + # network step reports "no network" and the cancellation check, which needs + # none, is what runs. openkal: - name: build + run (linux x86_64, above openkal) - runs-on: ubuntu-latest + name: build + run (${{ matrix.target }}, above openkal) + runs-on: ubuntu-24.04 timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + target: ['x86_64-linux-gnu', 'x86_64-linux-musl', 'x86_64-windows-musl', 'aarch64-linux-musl'] steps: - uses: actions/checkout@v4 @@ -130,20 +212,163 @@ jobs: mcpp --version mcpp self config --mirror GLOBAL + - name: Install the runner + if: matrix.target == 'aarch64-linux-musl' + timeout-minutes: 15 + env: + DEBIAN_FRONTEND: noninteractive + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq --no-install-recommends qemu-user > /dev/null + qemu-aarch64 --version | head -1 + - name: Cache mcpp sandbox uses: actions/cache@v4 with: path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry - key: mcpp-openkal-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }} + key: mcpp-openkal-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-${{ matrix.target }} # The example builds the library from this checkout (`path = "../.."`) and # takes the stack beneath it from the index, so what is tested is this # commit against the published openkal packages. # - # It makes one HTTPS request and reports "no network" rather than failing - # when there is none, so a runner without egress reports "not run" instead - # of "broken". The build, the link and the framing parsers are checked - # either way. + # The toolchain is named, as mcpp-index's measurement names it, and + # installed beforehand: on a fresh runner x86_64-windows-musl was refused + # as "cannot be built on this host" while llvm was not yet installed, though + # the Linux targets, which install it on demand, built. + # + # It cancels a handshake against a local listener, which needs no network, + # and then makes one HTTPS request; it reports "no network" rather than + # failing when there is none, so a runner without egress reports "not run" + # instead of "broken". A certificate the client refused is a failure, not + # an absence of network. The build, the link, the framing parsers and the + # cancellation are checked either way. + - name: Install llvm + run: mcpp toolchain install llvm 22.1.8 + - name: Build and run above openkal + if: matrix.target != 'x86_64-windows-musl' + working-directory: examples/openkal + run: | + set -o pipefail + mcpp run --toolchain llvm@22.1.8 --target '${{ matrix.target }}' 2>&1 | tee run.log + grep -q '^cancellation: ok' run.log + + # A static PE executable that imports only system DLLs, so it runs on any + # Windows as it is. + - name: Build for Windows above openkal + if: matrix.target == 'x86_64-windows-musl' working-directory: examples/openkal - run: mcpp run + run: | + mcpp build --toolchain llvm@22.1.8 --target x86_64-windows-musl + mkdir -p "$RUNNER_TEMP/smoke" + cp target/x86_64-windows-musl/*/bin/smoke.exe "$RUNNER_TEMP/smoke/" + + - uses: actions/upload-artifact@v4 + if: matrix.target == 'x86_64-windows-musl' + with: + name: smoke-x86_64-windows-musl + path: ${{ runner.temp }}/smoke/smoke.exe + if-no-files-found: error + + openkal-windows: + name: run (x86_64-windows-musl, above openkal, on Windows) + needs: openkal + runs-on: windows-2022 + timeout-minutes: 15 + defaults: + run: + shell: bash + steps: + - uses: actions/download-artifact@v4 + with: + name: smoke-x86_64-windows-musl + + # Above openkal the Windows certificate store is not read (README, + # "Platforms"), so the program is given a bundle: Git for Windows's, copied + # beside it, because openkal resolves a relative name against the working + # directory and names no absolute path. + # + # Git is asked where its bundle is, and the install searched after that, + # rather than the directory named: the runner image that brought Git for + # Windows 2.56 has no `mingw64/etc/ssl/certs/ca-bundle.crt`, and the job + # failed there before the program ran. + - name: Run above openkal + run: | + set -o pipefail + git --version + ca=$(git config --system --get http.sslcainfo || true) + if [ -n "$ca" ]; then ca=$(cygpath -u "$ca"); fi + if [ ! -f "$ca" ]; then + ca=$(find "/c/Program Files/Git" -path '*/ssl/certs/ca-bundle.crt' -print -quit) + fi + if [ ! -f "$ca" ]; then + echo "::error::no CA bundle in the Git for Windows install" + exit 1 + fi + echo "CA bundle: $ca" + cp "$ca" ca-bundle.crt + SSL_CERT_FILE=ca-bundle.crt ./smoke.exe 2>&1 | tee run.log + grep -q '^cancellation: ok' run.log + + # ── Windows ────────────────────────────────────────────────────────────────── + # + # WHY THIS JOB EXISTS. Windows keeps no bundle file, so on a Windows Sockets + # build `load_ca_certs` reads the system's ROOT store through the Win32 API + # (src/ca_bundle.cppm). That code is compiled on no other platform, and a job + # elsewhere can show neither that it builds nor that the roots it returns + # verify a real certificate chain. `test_ca_store` asserts both. + # + # AND THE HERMETIC TESTS RUN HERE TOO. They script their own servers on + # 127.0.0.1, and Windows Sockets is where `poll` is `WSAPoll`, a connect in + # progress is reported differently, and the cancellation slices were never + # run before this job ran them. + windows: + name: build + test (windows x86_64, ${{ matrix.toolchain }}) + runs-on: windows-2022 + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + toolchain: ['msvc@system', 'llvm@20.1.7'] + defaults: + run: + shell: bash + steps: + - uses: actions/checkout@v4 + + - name: Install xlings + shell: pwsh + run: | + irm https://d2learn.org/xlings-install.ps1.txt | iex + "$env:USERPROFILE\.xlings\subos\current\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append + + - name: Install mcpp + run: | + xlings update + xlings install "mcpp@$MCPP_VERSION" -y -g + mcpp --version + mcpp self config --mirror GLOBAL + + - name: Select the toolchain + run: | + spec='${{ matrix.toolchain }}' + case "$spec" in + msvc*) mcpp toolchain default msvc ;; + *) mcpp toolchain install "${spec%@*}" "${spec#*@}" + mcpp toolchain default "$spec" ;; + esac + mcpp toolchain list + + - name: The system store verifies a public chain + run: | + mcpp test test_ca_store 2>&1 | tee tests.log + grep -q '^test_ca_store \.\.\. ok' tests.log + + - name: The hermetic tests + run: | + set -o pipefail + for t in test_framing test_tls_verify test_pool test_proxy test_cancel test_handshake_timeout test_extra_ca; do + mcpp test "$t" 2>&1 | tee "$t.log" + grep -q "^$t \.\.\. ok" "$t.log" + done diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ff76b8..fcd7dcf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,237 @@ # Changelog +## 0.3.5 + +A handshake that never completes no longer holds a request for good, and a +private CA can be trusted alongside the default store. It is a patch release, so +a dependency written as a caret range, `tinyhttps = "^0.3.1"`, resolves to it; +the one change in behaviour, a handshake slower than `connectTimeoutMs` now +failing, is the end of a hang rather than a new failure mode, and is described +below. + +### TLS handshake time limit + +[#25](https://github.com/mcpplibs/tinyhttps/pull/25). The TLS handshake is +bounded by `connectTimeoutMs`. It had no limit: after the +TCP connection was up, a server (or an https:// proxy) that accepted it and never +sent its ServerHello held `send`, `send_stream` and `download_to_file` for good +unless the caller passed a stop token. + +* The handshake gives up once `connectTimeoutMs` has passed since it began, and + `statusText` is `TLS handshake timed out`. This covers the handshake with the + target, with an https:// proxy, and with the target inside an https:// proxy's + tunnel. The limit is on the handshake as a whole, so a peer that sends a byte + at a time does not hold it open. +* Behaviour change: a handshake slower than `connectTimeoutMs` (10 s by default) + now fails where it used to complete. Each step of setting up a connection (the + TCP connect, a proxy's reply, a handshake) has `connectTimeoutMs` to itself, so + the whole can take several times that. +* `TlsSocket::connect` bounds the handshake with the `timeoutMs` it already + took for the TCP connect, so a caller of it directly sees the change too. +* `TlsSocket::connect_over` takes a trailing `handshakeTimeoutMs`, `-1` (the + default) for no limit; a call by name compiles unchanged, a pointer to it needs + the new parameter in its type. `Socket` gains `set_deadline`, `deadline_hit` + and `wait_before_recv`. + +### Extra CA certificates + +[#24](https://github.com/mcpplibs/tinyhttps/pull/24). `HttpClientConfig::extraCaFile` is a path to a PEM file whose certificates are +trusted in addition to the default store, which `SSL_CERT_FILE` can only replace. +It is for a private CA or a proxy that re-signs TLS. + +* It applies wherever a trust store is built: the target, and the connection to + an `https://` proxy. +* A file that cannot be read or holds no certificate fails the connection, and + `statusText` names the file. +* `proxy_tunnel` takes the path as a trailing default argument, and `TlsSocket` + gains `set_extra_ca_file`. Nothing changes while the field is empty. + +### CI + +* The openkal job on Windows asks Git for Windows where its CA bundle is; 2.56 + moved it from `mingw64/` to `ucrt64/`, and the job failed before it ran. +* The Windows job runs `test_handshake_timeout` and `test_extra_ca` with the + other hermetic tests. + +## 0.3.4 + +A request in flight can be abandoned from another thread. Everything is added +and nothing changes without a token, so it is a patch release: a dependency +written as a caret range, `tinyhttps = "^0.3.0"`, resolves to it. A bare version +such as `"0.3.3"` is exact in mcpp and stays where it is. + +### Cancellation + +`send`, `send_stream` and `download_to_file` take an optional `std::stop_token` +as their last argument ([#23](https://github.com/mcpplibs/tinyhttps/pull/23)). + +* When the token is stopped the call returns within about 50 ms, and the + connection is closed instead of being returned to the pool. It covers the + connect, the exchange with an HTTP, https or SOCKS5 proxy, the TLS handshake, + waiting for the response head, reading the body and the gaps between + streaming callbacks. +* `HttpResponse::cancelled` says so. Before the status line arrives the response + is `statusCode` 0 with `statusText` and `bodyError` `Cancelled`; after it, + `statusCode` is the server's and `bodyError` is `cancelled`. `ok()` does not + look at `cancelled`, as it does not look at `bodyComplete`. +* `DownloadToFileResult::cancelled` is the same for a download, set by the token + or by `isCancelled`, with `error` `Cancelled` or `cancelled` as above. A + destination that failed first is reported as `writeFailed` instead. +* A cancelled request is not retried on a new connection, and its redirect is + not followed. +* A token belongs to its call. It is set on the socket at the bottom of the + connection (through an https:// proxy, the one beneath the tunnel) and taken + off when the connection goes back to the pool, so a token stopped after its + call returned — `std::jthread` stops its own in its destructor — does not + reach the next call on that connection. +* Not covered: name resolution (`getaddrinfo`, and the lookup `socks5://` does + for the target), and a write that is blocked because the server is not + reading. +* `proxy_tunnel` takes the token as a trailing default argument. `Socket` and + `TlsSocket` gain `set_stop`, and `Socket` gains `stop_possible`. +* A call through a pointer to `HttpClient::send`, `send_stream` or + `download_to_file` needs the new parameter in the pointer's type; a call by + name compiles unchanged. +* libc++ 20 and 22 fail to link `std::stop_source::request_stop()` in a file + that includes a standard header such as `` before `import std;`. + Import first in such a file; libc++ 23 and libstdc++ are not affected. + +### openkal + +* Where the C library is musl, the TLS random generator is seeded from musl's + `getentropy` rather than from mbedTLS's reading of `/dev/urandom`. Above + openkal on x86_64-windows-musl that file does not exist, and every handshake + failed with "CTR_DRBG - The entropy source failed". +* `examples/openkal` follows mcpp-index's openkal measurement: runtime 0.15.2, + and a cancellation check against a local listener that needs no network. CI + runs it on x86_64-linux-gnu, x86_64-linux-musl, aarch64-linux-musl (qemu) and + x86_64-windows-musl, which is built on Linux and run on Windows itself. + +### CI + +* Linux runs every test under gcc 16 and under llvm 22 with libc++. +* macOS (arm64, llvm 22) runs every test. Nothing ran on macOS before. +* Windows (msvc and llvm 20) runs the hermetic tests as well as `test_ca_store`. + +## 0.3.3 + +**Behaviour change, and the reason for this release:** `verifySsl = true` (the +default) now verifies the server certificate, and with no CA bundle to verify +against a connection fails instead of going ahead unverified +([#20](https://github.com/mcpplibs/tinyhttps/issues/20)). It is a patch release +so that every dependency written `tinyhttps = "0.3.x"` receives it. + +### Certificate verification + +`verifySsl = true` now verifies the server certificate. The handshake used to +complete with any certificate (a self-signed one, one for another host, an +expired one) because `MBEDTLS_SSL_VERIFY_OPTIONAL` was set and nothing read the +result afterwards. + +* The handshake fails when the chain is not trusted, the certificate has expired + or its name is not the host being connected to. The reason is in `statusText`, + for example `certificate verification failed: The certificate validity has + expired`; other handshake failures carry the mbedTLS error text. A refused TCP + connection still reads `Connection failed`. +* Behaviour change: with `verifySsl = true` and no CA bundle found + (`SSL_CERT_FILE` unset or unreadable, none of the system locations present), + the connection fails with a message saying so, where it used to go ahead + unverified. Point `SSL_CERT_FILE` at a PEM bundle, or set `verifySsl = false`. +* On Windows, when `SSL_CERT_FILE` is not set, the roots in the system `ROOT` + certificate store that are usable for TLS servers are read. That code is built + only for Windows Sockets targets, not for the musl one. MinGW builds need + `-lcrypt32`, as they already need `-lws2_32`. +* `examples/openkal` exits 1 when the certificate is refused; it used to read + every failure as an absence of network. CI builds and runs the Windows store + reader on windows-2022 under msvc and llvm (`tests/test_ca_store.cpp`). +* MSVC builds link `bcrypt`, from which mbedTLS draws entropy; its package asks + for it as `-lbcrypt`, which link.exe does not read. + +### Proxies + +Proxy URLs carry credentials, and a proxy that refuses the tunnel says why. + +`parse_proxy_url` split a URL at the first colon, so `user:pass@host:3128` came +out as host `user` and a port made of the digits of `pass@host:3128`. It also +dropped the scheme, which meant `https://proxy:8443` was spoken to as plain +HTTP. And every refusal, whatever the proxy said, reached the caller as +`Connection failed`. + +* `user:password@` in the proxy URL is percent-decoded and sent as a Basic + `Proxy-Authorization` header on the `CONNECT`. Brackets around an IPv6 host + are understood, and a port that is not 1..65535 is 0 rather than a truncation. +* `ProxyConfig` gains `scheme`, `hasCredentials`, `user` and `password`. + `parse_proxy_url` still returns the same type, and `host` and `port` keep + their meaning. +* A refused tunnel is reported in `statusText`: `proxy rejected the credentials: + 407 Proxy Authentication Required` when credentials were sent, + `proxy requires credentials: ...` when none were, `proxy refused CONNECT: ...` + for any other status. Any 2xx answer opens the tunnel; before, only 200 did. +* A scheme this library does not speak is an error (`proxy: unsupported scheme + 'socks4'`) where it used to be read as HTTP. +* New `proxy_tunnel` returns the tunnel together with the reason it could not + be made. `proxy_connect` keeps its signature and calls it, so it also accepts + any 2xx answer now. +* An `https://` proxy URL is spoken to over TLS: the session to the proxy + carries the `CONNECT`, and the session to the target runs inside the tunnel. + `TlsSocket` can now run over another `TlsSocket` (`connect_over` takes the + lower session), and its `wait_readable` asks that session rather than the + descriptor, because the bytes may already be decrypted there. The proxy + connection is set up like the one to the target, `verifySsl` included. The + default port for `https` is 443. +* SOCKS5 proxies: `socks5://` resolves the target's name here and sends the + proxy an address, `socks5h://` sends the name. An address literal is sent as + an address in both. The client offers no authentication, plus the + username/password method when the URL has credentials. Failures are reported + in `statusText` the same way as for an HTTP proxy. The default port is 1080. + SOCKS4 is not supported and, like any unknown scheme, is an error. +* `Socket::resolve_address` returns the address a name resolves to as raw bytes. +* The connection to an `https://` proxy is verified like the connection to the + target, and a refusal names both the hop and the reason, for example + `proxy: could not open a TLS connection to proxy:443: certificate + verification failed: ...`. The CONNECT, and any credentials, are sent only + after the proxy's certificate has verified. +* A failed handshake inside a tunnel closes the tunnel's session as well. + +## 0.3.2 + +`download_to_file` no longer reports success for a file the disk did not keep. + +It called `ofs.write` for each body chunk without looking at the stream, added +each chunk's size to `bytesWritten` from what the network delivered, and set +`bytesWritten` after `ofs.close()` without looking at that either. On a full disk +(`ENOSPC`) the transfer therefore succeeded: a 420,831,054 byte download left +220,979,200 bytes on disk while `bytesWritten` said 420,831,054 and `ok()` was +true, and the caller blamed the source (a checksum mismatch) for what was the +local disk. + +* Each chunk is flushed and the stream checked. On the first failure the + transfer stops reading, `error` becomes `write : ` (the reason + is `errno` as `std::generic_category` words it, for example `No space left on + device`), and the connection is dropped rather than returned to the pool with + the rest of the body still on it. Closing the file is checked the same way. +* `DownloadToFileResult::writeFailed` is new and is true when the fault is the + destination rather than the source: the write failed, the close failed, or the + file could not be opened (`error` is still `Cannot open file: ` for the + last). +* `DownloadToFileResult::bytesReceived` is new: the bytes of body the + connection delivered. +* `DownloadToFileResult::bytesWritten` changes meaning, and this is the one + behaviour change: it now counts bytes the file accepted, where it used to + count bytes the network delivered. The two are equal for every transfer that + did not fail to write. When a write fails, the chunk that failed is not + counted, though the file may hold part of it, so the value is a floor on the + file's size. + +## 0.3.1 + +The socket interface is selected by the C library rather than by the operating +system ([mcpp-index#435](https://github.com/mcpplibs/mcpp-index/issues/435)). +Above openkal on Windows the C library is musl and `winsock2.h` is not in the +dependency graph; the POSIX interface is used there, selected in `mcpp.toml` by +`cfg(c-abi = "musl")`. A Windows build with the platform's C runtime is +unchanged. `platform::is_windows` is renamed `platform::uses_winsock`. + ## 0.3.0 Closes [#15](https://github.com/mcpplibs/tinyhttps/issues/15) (a socket returned @@ -14,7 +246,7 @@ truncation at all, plus a `Content-Length` past 32 bits — and two more (the streaming reader's `Content-Length` exits) were regressions 0.2.10 had introduced. -### ⚠️ Read this first +### Read this first **If you are on 0.2.10 and use `send_stream`, upgrade.** 0.2.10 added a `Content-Length` branch to the streaming reader whose two exits — a read timeout diff --git a/README.md b/README.md index 693e738..6ee907d 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Minimal C++23 HTTP/HTTPS client library with SSE (Server-Sent Events) streaming - HTTP/HTTPS client with connection pooling (keep-alive) - SSE (Server-Sent Events) streaming - Streaming downloads to a file, with progress and cancellation -- Proxy support (HTTP CONNECT) +- Proxy support (HTTP CONNECT over plain HTTP or TLS, with Basic authentication; SOCKS5) - C++23 modules ## Usage @@ -46,17 +46,96 @@ else { /* resp.body is all of it */ } `ok()` deliberately does not consult `bodyComplete`, so existing `if (res.ok())` means exactly what it did before. +### Cancelling a request + +Pass a `std::stop_token` to `send`, `send_stream` or `download_to_file` and stop +its source from another thread. The call returns within about 50 ms with +`cancelled` set, and the connection is closed rather than pooled. + +```cpp +HttpResponse response; +std::jthread worker([&](std::stop_token stop) { response = client.send(request, stop); }); +// elsewhere: +worker.request_stop(); +``` + +If the status line had not arrived, `statusCode` is 0 and `statusText` is +`Cancelled`; otherwise they are the server's and `bodyError` is `cancelled`. +As with `bodyComplete`, `ok()` does not look at `cancelled`. A cancelled request +is not retried and its redirect is not followed. `download_to_file` reports the +same in `DownloadToFileResult::cancelled` and `error`, whether the token or its +`isCancelled` callback asked. + +It covers connecting, a proxy's handshake, the TLS handshake, waiting for the +response and reading it. Name resolution and a write blocked on a server that is +not reading cannot be interrupted. Without a token nothing changes. + +With libc++ 20 or 22, a file that includes a standard header such as `` +and then `import std;` fails to link `std::stop_source::request_stop()` (an +inline helper of libc++'s is never emitted). Put `import std;` before the +includes in that file, or use libc++ 23 or libstdc++. + ### Configuration | field | default | what it decides | | --- | --- | --- | -| `connectTimeoutMs` | 10000 | TCP connect | +| `connectTimeoutMs` | 10000 | setting up the connection: the TCP connect, each wait for a proxy's reply, and the TLS handshake, with the proxy and with the target (the handshake as a whole, not each read in it) | | `readTimeoutMs` | 60000 | any single read, and the total wait on a blocked write | -| `verifySsl` | true | verify the server certificate | +| `verifySsl` | true | verify the server certificate against the CA bundle (`SSL_CERT_FILE`, else the Windows `ROOT` certificate store in Windows Sockets builds, else a system location); the connection fails if the certificate is not trusted, has expired or is for another host, or if no bundle is found | | `keepAlive` | true | reuse connections between requests | | `maxRedirects` | 10 | 0 disables redirect following | | `maxResponseBodyBytes` | 64 MiB | the most `send()` will hold in memory; does not bound `download_to_file` or `send_stream` | | `retryOnStaleConnection` | true | resend once when a pooled connection turns out to have been closed while idle | +| `extraCaFile` | empty | path of a PEM file of CA certificates to trust in addition to the default store, see below | +| `proxy` | none | proxy URL, see below | + +### Extra CA certificates + +A private CA, or a proxy that re-signs TLS, is not in the default store, and +`SSL_CERT_FILE` replaces that store instead of adding to it. Set `extraCaFile` +to a PEM file and its certificates are trusted as well as the default ones: + +```cpp +cfg.extraCaFile = "/etc/corp/root-ca.pem"; +``` + +It applies to the connection to an `https://` proxy as well as to the target. If +the file cannot be read or holds no certificate, the connection fails and +`statusText` names the file. The file adds to a default store and does not stand +in for one: with none found, the connection fails as described under `verifySsl`. + +### Proxies + +Set `proxy` to a URL and every request is tunnelled through it with `CONNECT`: + +```cpp +cfg.proxy = "http://user:pass@proxy.example:3128"; +``` + +The scheme is `http` (a bare `host:port` means the same), `https`, `socks5` or +`socks5h`. + +An `https://` proxy is reached over TLS first, the `CONNECT` is sent inside that +session, and the connection to the target is a second TLS session inside the +tunnel; the two connections are set up alike, `verifySsl` included. + +With `socks5` the target's name is resolved here and the proxy is given an +address; with `socks5h` the proxy is given the name and resolves it itself, +which is what to use when the proxy is the only thing that can see the target's +DNS. An address literal is sent as an address either way. The client offers the +SOCKS5 proxy no authentication, plus the username/password method when the URL +has credentials. SOCKS4 is not supported. + +Credentials are read from the URL and sent as Basic `Proxy-Authorization` (or as +the SOCKS5 username and password); percent-escape any character that is special +in a URL (`p%40ss` for `p@ss`). The port defaults to 8080, 443 for `https`, and +1080 for the SOCKS schemes. + +When the proxy refuses the tunnel, `statusCode` is 0 and `statusText` carries +its answer: `proxy rejected the credentials: 407 Proxy Authentication Required`, +`proxy refused CONNECT: 403 Forbidden`, `proxy refused CONNECT: connection +refused (SOCKS5 reply 5)`. Any other scheme (`socks4://`, say) fails the same +way instead of being read as HTTP. ## Project templates @@ -109,19 +188,40 @@ differs between them. cd examples/openkal && mcpp run ``` +Above openkal two things differ from a host build. Both follow from the layer +beneath and not from this library: + +- On Windows the socket interface is the POSIX one, because the C library is + musl, so the Windows `ROOT` certificate store is not read. Set `SSL_CERT_FILE` + to a PEM bundle of trusted roots; without one every HTTPS connection is + refused, and `statusText` says so. +- `connectTimeoutMs` does not bound the TCP connect. `kal_net_connect` has no + form that begins a connection and reports its outcome later, so the connect + completes or fails before it returns. A stop token is checked before the + connect, not during it. +- On Windows there is no name resolution: openkal has no resolver interface, + and musl's reads `/etc/resolv.conf`, which Windows does not have + ([openkal-musl#46](https://github.com/mcpplibs/openkal-musl/issues/46)). A URL + with an address in it connects. + +CI runs `examples/openkal` for x86_64-linux-gnu, x86_64-linux-musl, +aarch64-linux-musl (under qemu) and x86_64-windows-musl (built on Linux, run on +Windows). Locally, +`mcpp run --toolchain llvm@22.1.8 --target x86_64-windows-musl` runs it under wine. + ## 使用 mcpp 构建 ### 添加依赖 ```bash -mcpp add tinyhttps@0.3.0 +mcpp add tinyhttps@0.3.5 ``` 或在 `mcpp.toml` 中手动添加： ```toml [dependencies] -tinyhttps = "0.3.0" +tinyhttps = "0.3.5" ``` ### 构建 @@ -140,7 +240,10 @@ mcpplibs::tinyhttps::HttpClient client; auto result = client.download_to_file( "https://example.com/big.tar.gz", "out/big.tar.gz", [](std::int64_t total, std::int64_t done) { /* progress */ }); -if (!result.ok()) { /* result.error */ } +if (!result.ok()) { + // result.error says why; result.writeFailed says the local file is at + // fault (a full disk, a refused write) and not the server. +} ``` ## License diff --git a/examples/openkal/mcpp.toml b/examples/openkal/mcpp.toml index f8d89c8..66dc732 100644 --- a/examples/openkal/mcpp.toml +++ b/examples/openkal/mcpp.toml @@ -11,7 +11,7 @@ version = "0.1.0" # sources either way. That is the claim this directory exists to check. [dependencies] tinyhttps = { path = "../.." } -openkal-llvm-runtime = "0.9.1" +openkal-llvm-runtime = "0.15.2" [targets.smoke] kind = "bin" @@ -22,3 +22,11 @@ main = "src/main.cpp" # dependency above. [toolchain] default = "llvm@22.1.8" + +# The programs of the other targets this example is run for, as mcpp-index's +# openkal measurement runs them (tests/openkal/pins.toml there). +[target.x86_64-windows-musl] +runner = ["wine"] + +[target.aarch64-linux-musl] +runner = ["qemu-aarch64"] diff --git a/examples/openkal/src/main.cpp b/examples/openkal/src/main.cpp index 6a63d8b..543439d 100644 --- a/examples/openkal/src/main.cpp +++ b/examples/openkal/src/main.cpp @@ -8,12 +8,84 @@ // // Run it with `mcpp run` from this directory. It needs the network; when there // is none it says so and exits 0, because a CI job without egress should report -// "not run" rather than "broken". +// "not run" rather than "broken". A certificate the client refused is not an +// absence of network, and exits 1. +// The listener below is made with the C library's sockets, which openkal-musl +// provides on every target this example runs on. C headers only: they declare +// nothing `import std` also declares. +#include +#include +#include +#include + import mcpplibs.tinyhttps; import std; namespace https = mcpplibs::tinyhttps; +namespace { + +// CANCELLATION ABOVE openkal, WITH NO NETWORK. +// +// A listener that never accepts: the connection completes in the backlog, the +// client sends its ClientHello and waits for an answer that is not coming. That +// wait is a `poll` in slices of 50 ms, and above openkal each slice is a bounded +// read (`kal_timeout_read`) rather than a readiness query, which is the part of +// the stack this checks. The read timeout is 30 s, so returning promptly can +// only be the token's doing. +bool cancellation_works() { + const int listener = ::socket(AF_INET, SOCK_STREAM, 0); + if (listener < 0) { std::println("cancellation: no socket"); return false; } + sockaddr_in addr {}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = 0; + socklen_t len = sizeof addr; + if (::bind(listener, reinterpret_cast(&addr), sizeof addr) != 0 + || ::listen(listener, 4) != 0 + || ::getsockname(listener, reinterpret_cast(&addr), &len) != 0) { + std::println("cancellation: no listener"); + ::close(listener); + return false; + } + const std::string url = "https://127.0.0.1:" + std::to_string(ntohs(addr.sin_port)) + "/"; + + https::HttpClientConfig config; + config.verifySsl = false; + config.connectTimeoutMs = 5000; + config.readTimeoutMs = 30000; + https::HttpClient client(config); + https::HttpRequest request; + request.method = https::Method::GET; + request.url = url; + + // A token stopped before the call: nothing is sent. + std::stop_source early; + early.request_stop(); + auto before = client.send(request, early.get_token()); + + // A token stopped while the handshake waits. + std::stop_source source; + std::jthread stopper([&] { + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + source.request_stop(); + }); + const auto start = std::chrono::steady_clock::now(); + auto during = client.send(request, source.get_token()); + const auto ms = std::chrono::duration_cast( + std::chrono::steady_clock::now() - start).count(); + ::close(listener); + + const bool ok = before.cancelled && before.statusCode == 0 + && during.cancelled && during.statusCode == 0 + && ms >= 150 && ms < 2000; + std::println("cancellation: {} (stopped at 200 ms, returned after {} ms)", + ok ? "ok" : "WRONG", ms); + return ok; +} + +} // namespace + int main() { https::Socket::platform_init(); @@ -29,6 +101,8 @@ int main() { } std::println("framing parsers: ok"); + if (!cancellation_works()) return 1; + https::HttpClientConfig config; config.connectTimeoutMs = 15000; config.readTimeoutMs = 20000; @@ -40,6 +114,21 @@ int main() { auto response = client.send(request); if (response.statusCode == 0) { + // A certificate the client refused is a verdict about this stack, not + // an absence of network, and is reported as one. Read as "no network", + // a CA bundle this stack cannot find would pass here unnoticed. + constexpr std::string_view refusals[] = { + "certificate verification failed", + "no CA certificate bundle", + "cannot parse the CA certificate bundle", + "TLS handshake failed", + }; + for (auto refusal : refusals) { + if (response.statusText.starts_with(refusal)) { + std::println("TLS refused: {}", response.statusText); + return 1; + } + } std::println("no network ({}) — the build and the parsers are still verified", response.statusText); return 0; diff --git a/mcpp.toml b/mcpp.toml index def4bb6..911b9b3 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "tinyhttps" -version = "0.3.0" +version = "0.3.5" description = "Minimal C++23 HTTP/HTTPS client with SSE streaming support" license = "Apache-2.0" repo = "https://github.com/mcpplibs/tinyhttps" @@ -14,3 +14,11 @@ mbedtls = "3.6.1" [dev-dependencies.compat] gtest = "1.15.2" + +# The socket interface follows the C library rather than the operating system +# (src/platform.cppm). Where the C library is musl, including musl above openkal +# on Windows, it is the POSIX interface; the platform's own headers are not part +# of that graph. The entropy source follows it too: musl's `getentropy`, rather +# than mbedTLS's reading of /dev/urandom (src/tls.cppm). +[target.'cfg(c-abi = "musl")'.build] +cxxflags = ["-DTINYHTTPS_POSIX_SOCKETS", "-DTINYHTTPS_GETENTROPY"] diff --git a/src/ca_bundle.cppm b/src/ca_bundle.cppm index 577179f..f481d77 100644 --- a/src/ca_bundle.cppm +++ b/src/ca_bundle.cppm @@ -2,9 +2,28 @@ module; #include +// The same switch as socket.cppm. The store is read through the Win32 API, whose +// headers a build on the POSIX socket interface does not have. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#include +#pragma comment(lib, "crypt32.lib") +#endif + export module mcpplibs.tinyhttps:ca_bundle; import std; +import :platform; namespace mcpplibs::tinyhttps { @@ -24,6 +43,59 @@ auto read_file(const char* path) -> std::string { return result; } +#ifdef TINYHTTPS_WINSOCK +// Whether the root may be used to authenticate a server: no usage restriction at +// all, or serverAuth listed. Windows marks roots it no longer trusts for TLS +// servers through this restriction, and the store still holds them. +auto trusted_for_servers(const CERT_CONTEXT* c) -> bool { + DWORD n = 0; + if (!::CertGetEnhancedKeyUsage(c, 0, nullptr, &n)) { + return false; + } + std::vector buf(n); + auto* usage = reinterpret_cast(buf.data()); + if (!::CertGetEnhancedKeyUsage(c, 0, usage, &n)) { + return false; + } + if (usage->cUsageIdentifier == 0) { + // CRYPT_E_NOT_FOUND means no restriction; otherwise the root allows no use. + return ::GetLastError() == CRYPT_E_NOT_FOUND; + } + for (DWORD i = 0; i < usage->cUsageIdentifier; ++i) { + if (std::strcmp(usage->rgpszUsageIdentifier[i], szOID_PKIX_KP_SERVER_AUTH) == 0) { + return true; + } + } + return false; +} +#endif + +// The Windows ROOT store as one PEM string; empty elsewhere or on failure. +auto system_store_pem() -> std::string { + std::string pem; +#ifdef TINYHTTPS_WINSOCK + if (HCERTSTORE store = ::CertOpenSystemStoreW(0, L"ROOT")) { + for (const CERT_CONTEXT* c = nullptr; (c = ::CertEnumCertificatesInStore(store, c));) { + if (!trusted_for_servers(c)) { + continue; + } + DWORD n = 0; + if (!::CryptBinaryToStringA(c->pbCertEncoded, c->cbCertEncoded, + CRYPT_STRING_BASE64HEADER, nullptr, &n)) { + continue; + } + std::string one(n, '\0'); + if (::CryptBinaryToStringA(c->pbCertEncoded, c->cbCertEncoded, + CRYPT_STRING_BASE64HEADER, one.data(), &n)) { + pem.append(one, 0, n); + } + } + ::CertCloseStore(store, 0); + } +#endif + return pem; +} + } // anonymous namespace export auto load_ca_certs() -> std::string { @@ -37,6 +109,13 @@ export auto load_ca_certs() -> std::string { } } + // Windows has no bundle file; read the system's trusted roots instead. + if constexpr (platform::uses_winsock) { + if (auto pem = system_store_pem(); !pem.empty()) { + return pem; + } + } + std::vector ca_paths; // 2. Non-FHS prefixes (Termux et al. ship the bundle under $PREFIX). Probed diff --git a/src/http.cppm b/src/http.cppm index a0832f6..5cbdc73 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -1,3 +1,9 @@ +module; + +// `import std` carries no `errno`: it is a macro, and a module does not export +// macros. The download path reads it to say why the local file refused a write. +#include + export module mcpplibs.tinyhttps:http; import :tls; @@ -52,6 +58,11 @@ export struct HttpResponse { // "Invalid chunk size: zz" where the server had said "OK". std::string bodyError; + // True when the caller's stop token ended the request. Then `bodyComplete` + // is false, and the status line is the server's if it had arrived and + // 0 / "Cancelled" if not. + bool cancelled { false }; + bool ok() const { return statusCode >= 200 && statusCode < 300; } }; @@ -96,6 +107,13 @@ export struct HttpClientConfig { // endpoints, and restricting it there would make the retry inapplicable to // nearly every request it exists to rescue. Set false to opt out. bool retryOnStaleConnection { true }; + + // A PEM file of CA certificates to trust in addition to the default store + // (`SSL_CERT_FILE`, or else the system roots), for a private CA or a proxy that + // re-signs TLS. It applies to the connection to an https:// proxy as well. + // A file that cannot be read or holds no certificate fails the connection, + // with the path in `statusText`. Empty means none. + std::string extraCaFile; }; // Progress callback for streaming downloads: (totalBytes, downloadedBytes) @@ -105,11 +123,45 @@ export using DownloadProgressFn = std::function expectedBytes; std::string finalUrl; std::string etag; std::string lastModified; + + // Bytes of body the connection delivered, whether or not the file took them. + // Equal to `bytesWritten` unless `writeFailed`. + std::int64_t bytesReceived { 0 }; + + // TRUE WHEN THE FAULT IS THE DESTINATION AND NOT THE SOURCE — the file could + // not be opened, a write to it failed (a full disk, a quota, a device that + // refuses), or closing it reported that buffered data did not reach it. + // + // `error` then names the path and the reason, and the transfer stops at the + // first failure instead of reading on and discarding what it reads. It is + // what lets a caller tell "the server sent something wrong" from "this + // machine could not keep it" without matching text in `error`; a checksum + // failure downstream of a truncated file otherwise blames the wrong party. + bool writeFailed { false }; + + // True when the caller abandoned the transfer: `isCancelled` returned true + // or the stop token was stopped. `error` is then "Cancelled" if no status + // line had arrived and "cancelled" if one had, as for `HttpResponse`. A + // destination that failed first is reported as that instead. + bool cancelled { false }; + bool ok() const { return statusCode >= 200 && statusCode < 300 && error.empty(); } }; @@ -786,7 +838,7 @@ static bool drain_body(TlsSocket& sock, const ResponseHead& head, // calling `keep()` drops the connection, and a path added tomorrow is safe // before anyone reviews it. // -// ⚠️ Ordering: `drop()` destroys a `TlsSocket`, which sends `close_notify`, +// Ordering: `drop()` destroys a `TlsSocket`, which sends `close_notify`, // which writes to a socket whose peer may be gone. That write is why P0 — // `MSG_NOSIGNAL` in `Socket::write` — had to land before this class existed; // without it, making the drop path more common would have made issue #16 more @@ -803,12 +855,21 @@ public: // Call only where the body was read to the end its framing declared and the // response did not ask for the connection to close. - void keep() noexcept { armed_ = false; } + // + // A connection in the pool carries no stop token. The call that used it + // set its own (`perform_exchange`), and the caller may stop that token + // after this call has returned — a `std::jthread` does so in its + // destructor — which must not reach whatever takes the connection next. + void keep() noexcept { + if (!armed_) return; + armed_ = false; + if (auto it = pool_.find(key_); it != pool_.end()) it->second.set_stop({}); + } // Drop now rather than at the end of the scope. Idempotent, and a no-op // after `keep()`. // - // ⚠️ Callers that recurse — the redirect paths — MUST settle the guard + // Callers that recurse — the redirect paths — MUST settle the guard // before recursing. The inner call puts its own connection into the pool // under the same key when the redirect is to the same host, and a guard // still armed when the outer scope ends would then delete it. @@ -846,14 +907,21 @@ public: HttpClient(HttpClient&&) = default; HttpClient& operator=(HttpClient&&) = default; - HttpResponse send(const HttpRequest& request) { - return send_impl(request, 0); + // `stop` can be stopped from another thread to abandon the request: the + // call returns within about 50 ms with `cancelled` set, and the connection + // is closed rather than pooled. It cannot interrupt name resolution or a + // write blocked because the server is not reading. + HttpResponse send(const HttpRequest& request, std::stop_token stop = {}) { + return send_impl(request, 0, stop); } // Streaming SSE request — reads the response body incrementally, feeding // chunks through SseParser to the caller's callback. The callback receives // each SseEvent and returns true to continue or false to stop. - HttpResponse send_stream(const HttpRequest& request, SseCallbackFn callback) { + // `stop` is as for send(). + HttpResponse send_stream(const HttpRequest& request, SseCallbackFn callback, + std::stop_token stop = {}) { + if (stop.stop_requested()) return cancelled_response(); HttpResponse response; auto parsed = parse_url(request.url); @@ -867,12 +935,13 @@ public: PooledConnection guard(pool_, poolKey); auto exchange = perform_exchange(parsed, poolKey, - build_request(request, parsed), guard); + build_request(request, parsed), guard, stop); if (!exchange.error.empty()) { response.statusCode = 0; response.statusText = exchange.error; response.bodyComplete = false; response.bodyError = exchange.error; + response.cancelled = exchange.cancelled; return response; } @@ -887,6 +956,7 @@ public: SseParser parser; auto sink = [&](std::string_view data) -> bool { + if (stop.stop_requested()) return false; if (captureBody) { append_within_limit(response.body, data, stream_error_body_limit); } @@ -898,21 +968,23 @@ public: finish_body(response, exchange, request.method, /*maxBytes=*/std::numeric_limits::max(), - sink, guard); + sink, guard, stop); return response; } // Download URL to file with streaming progress. // Follows redirects. Calls onProgress periodically during body read. // isCancelled is checked after each block — return true to abort. + // `stop` is as for send(), and also ends the wait for the response head. DownloadToFileResult download_to_file( const std::string& url, const std::filesystem::path& destFile, DownloadProgressFn onProgress = nullptr, - std::function isCancelled = nullptr) + std::function isCancelled = nullptr, + std::stop_token stop = {}) { return download_to_file_impl(url, destFile, std::move(onProgress), - std::move(isCancelled), 0); + std::move(isCancelled), 0, stop); } HttpClientConfig& config() { return config_; } @@ -925,8 +997,18 @@ private: TlsSocket* sock { nullptr }; ResponseHead head; std::string error; + bool cancelled { false }; }; + static HttpResponse cancelled_response() { + HttpResponse response; + response.statusText = "Cancelled"; + response.bodyComplete = false; + response.bodyError = "Cancelled"; + response.cancelled = true; + return response; + } + std::string build_request(const HttpRequest& request, const ParsedUrl& parsed) const { std::string reqStr; reqStr += method_to_string(request.method); @@ -963,15 +1045,26 @@ private: return reqStr; } - bool open_connection(TlsSocket& sock, const ParsedUrl& parsed) { + // False on failure, with `error` set when there is more to say than that the + // connection failed: a proxy that refused the tunnel says so in its own words. + bool open_connection(TlsSocket& sock, const ParsedUrl& parsed, std::string& error, + std::stop_token stop) { + sock.set_extra_ca_file(config_.extraCaFile); if (config_.proxy.has_value()) { - auto proxyConf = parse_proxy_url(config_.proxy.value()); - auto tunnel = proxy_connect(proxyConf.host, proxyConf.port, - parsed.host, parsed.port, - config_.connectTimeoutMs); - if (!tunnel.is_valid()) return false; - return sock.connect_over(std::move(tunnel), parsed.host.c_str(), - config_.verifySsl); + auto tunnel = proxy_tunnel(parse_proxy_url(config_.proxy.value()), + parsed.host, parsed.port, + config_.connectTimeoutMs, config_.verifySsl, stop, + config_.extraCaFile); + if (!tunnel.ok()) { + error = std::move(tunnel.error); + return false; + } + if (tunnel.proxyTls) { + return sock.connect_over(std::move(tunnel.proxyTls), parsed.host.c_str(), + config_.verifySsl, config_.connectTimeoutMs); + } + return sock.connect_over(std::move(tunnel.socket), parsed.host.c_str(), + config_.verifySsl, config_.connectTimeoutMs); } return sock.connect(parsed.host.c_str(), parsed.port, config_.connectTimeoutMs, config_.verifySsl); @@ -986,8 +1079,17 @@ private: // request, so sending it again on a fresh connection is safe — once, and // only when no response byte has arrived. See `retryOnStaleConnection`. Exchange perform_exchange(const ParsedUrl& parsed, const std::string& poolKey, - const std::string& reqStr, PooledConnection& guard) { + const std::string& reqStr, PooledConnection& guard, + std::stop_token stop = {}) { + // Drops the connection; a stop request outranks whatever went wrong. + auto fail = [&](std::string message) -> Exchange { + guard.drop(); + if (stop.stop_requested()) return { nullptr, {}, "Cancelled", true }; + return { nullptr, {}, std::move(message) }; + }; + for (int attempt = 0; attempt < 2; ++attempt) { + if (stop.stop_requested()) return fail(""); bool reused = false; TlsSocket* sock = nullptr; @@ -999,9 +1101,17 @@ private: if (it != pool_.end()) pool_.erase(it); auto [inserted, ok] = pool_.emplace(poolKey, TlsSocket{}); sock = &inserted->second; - if (!open_connection(*sock, parsed)) { - guard.drop(); - return { nullptr, {}, "Connection failed" }; + } + sock->set_stop(stop); + if (!reused) { + std::string openError; + if (!open_connection(*sock, parsed, openError, stop)) { + // The proxy's refusal, else the TLS session's reason, + // else the TCP connection failed and there is no more to say. + std::string why = std::move(openError); + if (why.empty()) why = sock->error(); + if (why.empty()) why = "Connection failed"; + return fail(std::move(why)); } } @@ -1017,8 +1127,7 @@ private: // connection therefore yields exactly one execution. if (!write_all(*sock, reqStr, config_.readTimeoutMs)) { if (mayRetry) { guard.reset(); continue; } - guard.drop(); - return { nullptr, {}, "Write failed" }; + return fail("Write failed"); } auto head = read_response_head(*sock, config_.readTimeoutMs); @@ -1028,18 +1137,16 @@ private: // arrived the server has seen the request, and repeating it could // repeat its effect. if (mayRetry && !head.error().sawBytes) { guard.reset(); continue; } - guard.drop(); - return { nullptr, {}, head.error().message }; + return fail(head.error().message); } - guard.drop(); - return { nullptr, {}, "No response" }; + return fail("No response"); } // Reads the body into `sink` and settles the pool guard by what the read // found. The single place that decides whether a connection is reusable. void finish_body(HttpResponse& response, Exchange& exchange, Method method, std::int64_t maxBytes, const BodySink& sink, - PooledConnection& guard) { + PooledConnection& guard, std::stop_token stop = {}) { if (!response_has_body(method, exchange.head.statusCode)) { if (config_.keepAlive && !exchange.head.connectionClose) guard.keep(); return; @@ -1055,6 +1162,10 @@ private: // Not an error, but bytes are still owed on the socket. response.bodyComplete = false; response.bodyError = "stopped by callback"; + if (stop.stop_requested()) { + response.bodyError = "cancelled"; + response.cancelled = true; + } break; case BodyEnd::ClosedByPeer: // The body ended where it said it would; the connection did too. @@ -1062,11 +1173,17 @@ private: case BodyEnd::Truncated: response.bodyComplete = false; response.bodyError = outcome.error; + if (stop.stop_requested()) { + response.bodyError = "cancelled"; + response.cancelled = true; + } break; } } - HttpResponse send_impl(const HttpRequest& request, int redirectCount) { + HttpResponse send_impl(const HttpRequest& request, int redirectCount, + std::stop_token stop) { + if (stop.stop_requested()) return cancelled_response(); HttpResponse response; auto parsed = parse_url(request.url); @@ -1080,12 +1197,13 @@ private: PooledConnection guard(pool_, poolKey); auto exchange = perform_exchange(parsed, poolKey, - build_request(request, parsed), guard); + build_request(request, parsed), guard, stop); if (!exchange.error.empty()) { response.statusCode = 0; response.statusText = exchange.error; response.bodyComplete = false; response.bodyError = exchange.error; + response.cancelled = exchange.cancelled; return response; } @@ -1109,10 +1227,10 @@ private: response.body.append(data); return true; }, - guard); + guard, stop); // Follow 3xx redirects if configured. - if (config_.maxRedirects > 0 && + if (config_.maxRedirects > 0 && !response.cancelled && response.statusCode >= 300 && response.statusCode < 400 && redirectCount < config_.maxRedirects) { std::string location = find_header(response.headers, "Location"); @@ -1133,7 +1251,7 @@ private: // connection into the pool under this same key, and a guard // still armed would delete it when this scope ends. guard.drop(); - return send_impl(redirectReq, redirectCount + 1); + return send_impl(redirectReq, redirectCount + 1, stop); } } @@ -1145,9 +1263,15 @@ private: const std::filesystem::path& destFile, DownloadProgressFn onProgress, std::function isCancelled, - int redirectCount) + int redirectCount, + std::stop_token stop) { DownloadToFileResult result; + if (stop.stop_requested()) { + result.error = "Cancelled"; + result.cancelled = true; + return result; + } auto parsed = parse_url(url); if (parsed.scheme != "https") { @@ -1164,9 +1288,10 @@ private: request.headers = { {"User-Agent", "tinyhttps/1.0"}, {"Accept", "*/*"} }; auto exchange = perform_exchange(parsed, poolKey, - build_request(request, parsed), guard); + build_request(request, parsed), guard, stop); if (!exchange.error.empty()) { result.error = exchange.error; + result.cancelled = exchange.cancelled; return result; } @@ -1191,6 +1316,11 @@ private: } else { guard.drop(); } + if (stop.stop_requested()) { + result.error = "cancelled"; + result.cancelled = true; + return result; + } if (location.starts_with("/")) { location = parsed.scheme + "://" + parsed.host + @@ -1198,7 +1328,8 @@ private: location; } return download_to_file_impl(location, destFile, std::move(onProgress), - std::move(isCancelled), redirectCount + 1); + std::move(isCancelled), redirectCount + 1, + stop); } } @@ -1222,34 +1353,85 @@ private: std::ofstream ofs(destFile, std::ios::binary); if (!ofs) { result.error = "Cannot open file: " + destFile.string(); + result.writeFailed = true; settle_without_body(exchange, hasBody, guard); return result; } - if (!hasBody) { + // What the file refused, in the words of the system that refused it. + // `errno` is cleared before each operation that can set it, because the + // socket reads between writes leave their own values behind and a stale + // EAGAIN reported as the reason a disk filled is worse than no reason. + int writeErrno = 0; + bool writeFailed = false; + auto close_file = [&] { + errno = 0; ofs.close(); + if (!writeFailed && ofs.fail()) { + writeFailed = true; + writeErrno = errno; + } + }; + auto write_failure = [&] { + return "write " + destFile.string() + ": " + + (writeErrno != 0 + ? std::generic_category().message(writeErrno) + : std::string("stream error")); + }; + + if (!hasBody) { + close_file(); + if (writeFailed) { + result.error = write_failure(); + result.writeFailed = true; + return result; + } if (config_.keepAlive && !exchange.head.connectionClose) guard.keep(); return result; } const std::int64_t totalBytes = exchange.head.contentLength > 0 ? exchange.head.contentLength : 0; - std::int64_t downloaded = 0; + std::int64_t received = 0; + std::int64_t written = 0; bool cancelled = false; auto outcome = read_body( *exchange.sock, exchange.head, config_.readTimeoutMs, std::numeric_limits::max(), [&](std::string_view data) -> bool { + const auto size = static_cast(data.size()); + received += size; + + // Flushed per chunk, and the stream checked after it. A write + // into the stream's buffer succeeds whatever the disk has left; + // the failure surfaces at the flush that hands the buffer to the + // operating system, and without one here it surfaced at + // `close()`, after every chunk had been counted as written. + // `read_body` hands over slices of at most 8 KiB, about the size + // of the stream's own buffer, so this costs no more system calls + // than the buffering it replaces. + errno = 0; ofs.write(data.data(), static_cast(data.size())); - downloaded += static_cast(data.size()); - if (onProgress) onProgress(totalBytes, downloaded); - if (isCancelled && isCancelled()) { cancelled = true; return false; } + ofs.flush(); + if (!ofs) { + writeFailed = true; + writeErrno = errno; + return false; // reading on would only discard what it reads + } + + written += size; + if (onProgress) onProgress(totalBytes, written); + if ((isCancelled && isCancelled()) || stop.stop_requested()) { + cancelled = true; + return false; + } return true; }); - ofs.close(); - result.bytesWritten = downloaded; + close_file(); + result.bytesWritten = written; + result.bytesReceived = received; switch (outcome.end) { case BodyEnd::Complete: @@ -1261,9 +1443,21 @@ private: result.error = cancelled ? "cancelled" : "stopped"; break; case BodyEnd::Truncated: - result.error = outcome.error; + // A wait that the stop token ended reads as a truncated body. + cancelled = stop.stop_requested(); + result.error = cancelled ? "cancelled" : outcome.error; break; } + result.cancelled = cancelled; + + // The destination's failure outranks whatever the switch recorded: a + // stop that the file asked for is not a cancellation, and a body that + // arrived whole into a file that lost it has not succeeded. + if (writeFailed) { + result.error = write_failure(); + result.writeFailed = true; + result.cancelled = false; + } return result; } diff --git a/src/platform.cppm b/src/platform.cppm index 685ffb0..dcf3ade 100644 --- a/src/platform.cppm +++ b/src/platform.cppm @@ -1,6 +1,15 @@ module; -#ifndef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifndef TINYHTTPS_WINSOCK #include #include #include @@ -14,7 +23,7 @@ import std; // mcpplibs.tinyhttps:platform — the one place OS-specific networking quirks // live. Everything else (socket.cppm, http.cppm…) stays portable and branches -// on `platform::is_windows` with `if constexpr`, never raw #ifdef. +// on `platform::uses_winsock` with `if constexpr`, never raw #ifdef. // // ── The platforms this library is known to run on, and what differs ────────── // @@ -26,7 +35,7 @@ import std; // | Windows | nothing to do | no SIGPIPE, neither macro | // | above openkal | MSG_NOSIGNAL, and moot | see below | // -// ⭐ ABOVE openkal there are no signals at all, so the hazard issue #16 +// ABOVE openkal there are no signals at all, so the hazard issue #16 // describes cannot arise — and the same code is nevertheless correct there // without a branch. openkal-musl defines MSG_NOSIGNAL // (`musl/include/sys/socket.h:344`) and accepts it as a no-op, in its own words: @@ -36,7 +45,7 @@ import std; // -ENOSYS (`:550`, `:592-593`). SO_NOSIGPIPE is a BSD spelling that musl does // not define, so `socket.cppm`'s `#ifdef` for it simply does not compile in. // -// ⚠️ One difference that IS live above openkal: connect() completes before it +// One difference that IS live above openkal: connect() completes before it // returns even on a non-blocking descriptor, because `kal_net_connect` has no // form that begins a connection and reports its outcome later // (`port/src/okm_net.c:454-467`). `Socket::connect_addrinfo` therefore never @@ -58,14 +67,14 @@ import std; namespace mcpplibs::tinyhttps::platform { // Compile-time platform flag for `if constexpr` at call sites. -export inline constexpr bool is_windows = -#ifdef _WIN32 +export inline constexpr bool uses_winsock = +#ifdef TINYHTTPS_WINSOCK true; #else false; #endif -#ifndef _WIN32 +#ifndef TINYHTTPS_WINSOCK namespace { @@ -185,7 +194,7 @@ inline std::vector dns_query_a(const std::string& server, const cha } // anonymous namespace -#endif // !_WIN32 +#endif // !TINYHTTPS_WINSOCK // True when libc's own resolver has a usable config (/etc/resolv.conf). When // false, callers should prefer resolve_fallback() to avoid a multi-second stall @@ -195,9 +204,9 @@ inline std::vector dns_query_a(const std::string& server, const cha // `if constexpr` still compiles its discarded branch, which would reference the // POSIX-only helpers above that don't exist on Windows. Concentrating that // preprocessor divergence here is exactly why this platform module exists; call -// sites elsewhere branch on `is_windows` with `if constexpr`. +// sites elsewhere branch on `uses_winsock` with `if constexpr`. export bool system_resolver_configured() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK return true; #else std::error_code ec; @@ -211,7 +220,7 @@ export bool system_resolver_configured() { // on query failure. A numeric host is returned unchanged. export std::vector resolve_fallback([[maybe_unused]] const char* host, [[maybe_unused]] int timeoutMs) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK return {}; #else if (is_numeric_host(host)) return { std::string(host) }; diff --git a/src/proxy.cppm b/src/proxy.cppm index e102de6..bd51d92 100644 --- a/src/proxy.cppm +++ b/src/proxy.cppm @@ -1,155 +1,401 @@ export module mcpplibs.tinyhttps:proxy; import :socket; +import :tls; import std; namespace mcpplibs::tinyhttps { +// A parsed proxy URL: where the proxy is, how to talk to it, and the credentials +// to send. `parse_proxy_url` fills it in; nothing validates it until +// `proxy_tunnel` tries to use it. export struct ProxyConfig { std::string host; int port { 8080 }; + + // Lower case: "http", "https" for a proxy that is itself reached over TLS, + // "socks5" or "socks5h". "http" when the URL has no scheme, which is how a + // bare `host:port` has always been read. With "socks5" this library + // resolves the target's name and gives the proxy an address; with "socks5h" + // it gives the proxy the name. + std::string scheme { "http" }; + + // From `user:password@` in the URL, percent-decoded. A URL with `@` and no + // password still has credentials; `hasCredentials` is what says so, because + // an empty user or an empty password is a legitimate value. + bool hasCredentials { false }; + std::string user; + std::string password; }; -// Parse "http://host:port" proxy URL +static int hex_digit(char c) { + if (c >= '0' && c <= '9') return c - '0'; + if (c >= 'a' && c <= 'f') return c - 'a' + 10; + if (c >= 'A' && c <= 'F') return c - 'A' + 10; + return -1; +} + +// "%40" is "@". A '%' that is not followed by two hex digits stays as written. +static std::string percent_decode(std::string_view in) { + std::string out; + out.reserve(in.size()); + for (std::size_t i = 0; i < in.size(); ++i) { + if (in[i] == '%' && i + 2 < in.size() && hex_digit(in[i + 1]) >= 0 + && hex_digit(in[i + 2]) >= 0) { + out += static_cast(hex_digit(in[i + 1]) * 16 + hex_digit(in[i + 2])); + i += 2; + } else { + out += in[i]; + } + } + return out; +} + +// Parse "[scheme://][user[:password]@]host[:port]". A path, query or fragment +// after the authority is ignored. IPv6 literals are written in brackets. The +// port defaults to 443 for https, 1080 for socks5 and socks5h, and 8080 +// otherwise. export ProxyConfig parse_proxy_url(std::string_view url) { ProxyConfig config; - // Strip scheme if present auto schemeEnd = url.find("://"); if (schemeEnd != std::string_view::npos) { + config.scheme.clear(); + for (char c : url.substr(0, schemeEnd)) { + config.scheme += (c >= 'A' && c <= 'Z') ? static_cast(c + 32) : c; + } url = url.substr(schemeEnd + 3); } - // Strip trailing path if present - auto pathStart = url.find('/'); - if (pathStart != std::string_view::npos) { - url = url.substr(0, pathStart); + url = url.substr(0, url.find_first_of("/?#")); + + // The last '@' separates the userinfo, so an unencoded '@' in a password + // still parses the way curl reads it. + if (auto at = url.rfind('@'); at != std::string_view::npos) { + auto userinfo = url.substr(0, at); + url = url.substr(at + 1); + auto colon = userinfo.find(':'); + config.hasCredentials = true; + config.user = percent_decode(userinfo.substr(0, colon)); + if (colon != std::string_view::npos) { + config.password = percent_decode(userinfo.substr(colon + 1)); + } } - // Check for port - auto colonPos = url.find(':'); - if (colonPos != std::string_view::npos) { - config.host = std::string(url.substr(0, colonPos)); - auto portStr = url.substr(colonPos + 1); - config.port = 0; - for (char c : portStr) { - if (c >= '0' && c <= '9') { - config.port = config.port * 10 + (c - '0'); - } + std::string_view portStr; + bool hasPort = false; + if (!url.empty() && url.front() == '[') { + auto close = url.find(']'); + config.host = std::string(url.substr(1, close == std::string_view::npos + ? std::string_view::npos : close - 1)); + if (close != std::string_view::npos && close + 1 < url.size() && url[close + 1] == ':') { + portStr = url.substr(close + 2); + hasPort = true; } + } else if (auto colon = url.find(':'); colon != std::string_view::npos) { + config.host = std::string(url.substr(0, colon)); + portStr = url.substr(colon + 1); + hasPort = true; } else { config.host = std::string(url); - config.port = 8080; + } + + if (config.scheme == "https") config.port = 443; + if (config.scheme.starts_with("socks")) config.port = 1080; + + if (hasPort && !portStr.empty()) { + // Anything but 1..65535 becomes 0, which no connect will accept. + long port = 0; + for (char c : portStr) { + if (c < '0' || c > '9' || (port = port * 10 + (c - '0')) > 65535) { port = 0; break; } + } + config.port = static_cast(port); } return config; } -// Read a line (ending with \r\n) from a plain Socket -static std::string read_line_plain(Socket& sock, int timeoutMs) { +static std::string base64_encode(std::string_view in) { + constexpr std::string_view table = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + std::string out; + for (std::size_t i = 0; i < in.size(); i += 3) { + unsigned v = static_cast(in[i]) << 16; + if (i + 1 < in.size()) v |= static_cast(in[i + 1]) << 8; + if (i + 2 < in.size()) v |= static_cast(in[i + 2]); + out += table[(v >> 18) & 63]; + out += table[(v >> 12) & 63]; + out += i + 1 < in.size() ? table[(v >> 6) & 63] : '='; + out += i + 2 < in.size() ? table[v & 63] : '='; + } + return out; +} + +// One line without its CRLF. nullopt on a timeout, an end of stream, or a line +// longer than any proxy sends. Reads a byte at a time on purpose: whatever +// follows the blank line belongs to the tunnel and must not be consumed. +template +static std::optional read_line(Stream& sock, int timeoutMs) { std::string line; char c; - while (true) { - if (!sock.wait_readable(timeoutMs)) { - break; - } + while (line.size() < 8192) { + if (!sock.wait_readable(timeoutMs)) return std::nullopt; int ret = sock.read(&c, 1); - if (ret < 0) break; if (ret == 0) { - if (!sock.wait_readable(timeoutMs)) break; + if (!sock.wait_readable(timeoutMs)) return std::nullopt; ret = sock.read(&c, 1); - if (ret <= 0) break; } + if (ret <= 0) return std::nullopt; line += c; - if (line.size() >= 2 && line[line.size() - 2] == '\r' && line[line.size() - 1] == '\n') { + if (line.size() >= 2 && line[line.size() - 2] == '\r' && line.back() == '\n') { line.resize(line.size() - 2); - break; + return line; } } - return line; + return std::nullopt; } -// Write all data to a plain Socket -static bool write_all_plain(Socket& sock, const std::string& data) { - int total = 0; - int len = static_cast(data.size()); - while (total < len) { - int ret = sock.write(data.c_str() + total, len - total); +template +static bool write_all(Stream& sock, std::string_view data, int timeoutMs) { + std::size_t total = 0; + while (total < data.size()) { + int ret = sock.write(data.data() + total, static_cast(data.size() - total)); if (ret < 0) return false; - if (ret == 0) { - ret = sock.write(data.c_str() + total, len - total); - if (ret <= 0) return false; - } - total += ret; + if (ret == 0 && !sock.wait_writable(timeoutMs)) return false; + total += static_cast(ret); } return true; } -// Connect through HTTP CONNECT proxy, returning a Socket connected to target through tunnel. -// On failure, returns an invalid (closed) Socket. -export Socket proxy_connect(std::string_view proxyHost, int proxyPort, - std::string_view targetHost, int targetPort, - int timeoutMs) { - Socket sock; - - // Step 1: Connect to proxy - std::string proxyHostStr(proxyHost); - if (!sock.connect(proxyHostStr.c_str(), proxyPort, timeoutMs)) { - return sock; - } - - // Step 2: Send CONNECT request - std::string request = "CONNECT "; - request += targetHost; - request += ":"; - request += std::to_string(targetPort); - request += " HTTP/1.1\r\nHost: "; - request += targetHost; - request += ":"; - request += std::to_string(targetPort); - request += "\r\n\r\n"; - - if (!write_all_plain(sock, request)) { - sock.close(); - return sock; - } - - // Step 3: Read response status line - std::string statusLine = read_line_plain(sock, timeoutMs); - if (statusLine.empty()) { - sock.close(); - return sock; - } - - // Parse status code from "HTTP/1.x 200 ..." - int statusCode = 0; - auto spacePos = statusLine.find(' '); - if (spacePos != std::string::npos) { - auto rest = std::string_view(statusLine).substr(spacePos + 1); - for (char c : rest) { - if (c >= '0' && c <= '9') { - statusCode = statusCode * 10 + (c - '0'); - } else { - break; - } +// "HTTP/1.1 407 Proxy Authentication Required" -> 407, or 0 when the line is +// not a status line. +static int status_code_of(std::string_view line) { + if (!line.starts_with("HTTP/1.") || line.size() < 12 || line[8] != ' ') return 0; + int code = 0; + for (char c : line.substr(9, 3)) { + if (c < '0' || c > '9') return 0; + code = code * 10 + (c - '0'); + } + return code; +} + +// Ask an HTTP proxy to open a tunnel to host:port. Returns an error message, or +// an empty string once the proxy has answered 2xx and the stream is the tunnel. +template +static std::string http_connect(Stream& sock, const ProxyConfig& proxy, + std::string_view host, int port, int timeoutMs) { + // An IPv6 literal needs its brackets back in the request target. + std::string authority = host.contains(':') ? "[" + std::string(host) + "]" : std::string(host); + authority += ":" + std::to_string(port); + + std::string request = "CONNECT " + authority + " HTTP/1.1\r\nHost: " + authority + "\r\n"; + if (proxy.hasCredentials) { + request += "Proxy-Authorization: Basic " + + base64_encode(proxy.user + ":" + proxy.password) + "\r\n"; + } + request += "\r\n"; + + if (!write_all(sock, request, timeoutMs)) return "proxy: could not send CONNECT"; + + auto statusLine = read_line(sock, timeoutMs); + int code = statusLine ? status_code_of(*statusLine) : 0; + if (code == 0) return "proxy: no HTTP response to CONNECT"; + if (code == 407) { + return std::string(proxy.hasCredentials ? "proxy rejected the credentials: " + : "proxy requires credentials: ") + + statusLine->substr(9); + } + if (code < 200 || code > 299) return "proxy refused CONNECT: " + statusLine->substr(9); + + // The rest of the head. A 2xx answer to CONNECT carries no body. + for (int i = 0; i < 100; ++i) { + auto header = read_line(sock, timeoutMs); + if (!header) return "proxy: response to CONNECT was cut short"; + if (header->empty()) return {}; + } + return "proxy: response to CONNECT has too many headers"; +} + +// SOCKS5 (RFC 1928), with the username/password method of RFC 1929. + +static bool read_exact(Socket& sock, unsigned char* buf, std::size_t n, int timeoutMs) { + std::size_t got = 0; + while (got < n) { + if (!sock.wait_readable(timeoutMs)) return false; + int ret = sock.read(reinterpret_cast(buf) + got, static_cast(n - got)); + if (ret <= 0) return false; // 0 is the proxy closing the connection + got += static_cast(ret); + } + return true; +} + +static std::string_view socks5_reply_text(unsigned char code) { + switch (code) { + case 1: return "general failure"; + case 2: return "connection not allowed by ruleset"; + case 3: return "network unreachable"; + case 4: return "host unreachable"; + case 5: return "connection refused"; + case 6: return "TTL expired"; + case 7: return "command not supported"; + case 8: return "address type not supported"; + } + return "unknown error"; +} + +// Returns an error message, or an empty string once the proxy has connected to +// the target and the stream is the tunnel. +static std::string socks5_connect(Socket& sock, const ProxyConfig& proxy, + std::string_view host, int port, int timeoutMs) { + const bool remoteDns = proxy.scheme == "socks5h"; + const std::string hostStr(host); + + // Everything that can be refused before a byte is sent is refused first. + if (proxy.hasCredentials && (proxy.user.size() > 255 || proxy.password.size() > 255)) { + return "proxy: SOCKS5 credentials are limited to 255 bytes each"; + } + // An address literal is always sent as an address; a name is sent as a name + // for socks5h and resolved here for socks5. + auto address = Socket::resolve_address(hostStr.c_str(), /*numericOnly=*/true); + if (address.empty() && remoteDns) { + if (host.empty() || host.size() > 255) return "proxy: host name does not fit a SOCKS5 request"; + } else if (address.empty()) { + address = Socket::resolve_address(hostStr.c_str()); + if (address.empty()) return "proxy: could not resolve " + hostStr; + } + + std::string greeting = proxy.hasCredentials ? std::string("\x05\x02\x00\x02", 4) + : std::string("\x05\x01\x00", 3); + if (!write_all(sock, greeting, timeoutMs)) return "proxy: could not send the SOCKS5 greeting"; + + unsigned char choice[2]; + if (!read_exact(sock, choice, 2, timeoutMs)) return "proxy: no SOCKS5 response"; + if (choice[0] != 5) return "proxy: not a SOCKS5 proxy"; + if (choice[1] == 0xFF) { + return proxy.hasCredentials + ? "proxy: SOCKS5 proxy accepts neither no authentication nor a username and password" + : "proxy requires credentials: no acceptable SOCKS5 authentication method"; + } + if (choice[1] == 2 && proxy.hasCredentials) { + std::string auth = "\x01"; + auth += static_cast(proxy.user.size()); + auth += proxy.user; + auth += static_cast(proxy.password.size()); + auth += proxy.password; + unsigned char status[2]; + if (!write_all(sock, auth, timeoutMs) || !read_exact(sock, status, 2, timeoutMs)) { + return "proxy: no response to the SOCKS5 username and password"; } + if (status[1] != 0) return "proxy rejected the credentials (SOCKS5 status " + std::to_string(status[1]) + ")"; + } else if (choice[1] != 0) { + return "proxy: SOCKS5 proxy chose an authentication method that was not offered"; } - if (statusCode != 200) { - sock.close(); - return sock; + std::string request("\x05\x01\x00", 3); + if (address.empty()) { + request += '\x03'; + request += static_cast(host.size()); + request += host; + } else { + request += address.size() == 4 ? '\x01' : '\x04'; + request.append(reinterpret_cast(address.data()), address.size()); } + request += static_cast(port >> 8); + request += static_cast(port & 0xFF); + if (!write_all(sock, request, timeoutMs)) return "proxy: could not send the SOCKS5 request"; - // Step 4: Read remaining response headers until empty line - while (true) { - std::string headerLine = read_line_plain(sock, timeoutMs); - if (headerLine.empty()) { - break; + unsigned char reply[4]; + if (!read_exact(sock, reply, 4, timeoutMs)) return "proxy: no SOCKS5 reply to the request"; + if (reply[0] != 5) return "proxy: not a SOCKS5 proxy"; + if (reply[1] != 0) { + return "proxy refused CONNECT: " + std::string(socks5_reply_text(reply[1])) + + " (SOCKS5 reply " + std::to_string(reply[1]) + ")"; + } + + // The bound address and port: nothing to use, but they are on the stream + // ahead of the tunnel and have to be read off it. + std::size_t bound = 0; + if (reply[3] == 1) { + bound = 4; + } else if (reply[3] == 4) { + bound = 16; + } else if (reply[3] == 3) { + unsigned char len; + if (!read_exact(sock, &len, 1, timeoutMs)) return "proxy: SOCKS5 reply was cut short"; + bound = len; + } else { + return "proxy: SOCKS5 reply has an unknown address type"; + } + unsigned char rest[258]; + if (!read_exact(sock, rest, bound + 2, timeoutMs)) return "proxy: SOCKS5 reply was cut short"; + return {}; +} + +// A connection to the target through a proxy. `error` is empty exactly when the +// tunnel is open; otherwise it says what the proxy did, in words fit for a +// caller to log. +// +// The tunnel is `socket`, except for an https:// proxy, where it is the TLS +// session to the proxy, `proxyTls`, and `socket` is unused. +export struct ProxyTunnel { + Socket socket; + std::unique_ptr proxyTls; + std::string error; + + [[nodiscard]] bool ok() const { + return error.empty() && (proxyTls ? proxyTls->is_valid() : socket.is_valid()); + } +}; + +// `verifySsl` and `extraCaFile` are for the connection to an https:// proxy, and +// are the same settings that govern the connection to the target. +export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, + std::string_view targetHost, int targetPort, + int timeoutMs, bool verifySsl = true, + std::stop_token stop = {}, + std::string_view extraCaFile = {}) { + ProxyTunnel tunnel; + tunnel.socket.set_stop(stop); + const std::string where = proxy.host + ":" + std::to_string(proxy.port); + + if (proxy.scheme == "http" || proxy.scheme == "socks5" || proxy.scheme == "socks5h") { + if (!tunnel.socket.connect(proxy.host.c_str(), proxy.port, timeoutMs)) { + tunnel.error = "proxy: could not connect to " + where; + return tunnel; } + tunnel.error = proxy.scheme == "http" + ? http_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs) + : socks5_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs); + if (!tunnel.error.empty()) tunnel.socket.close(); + } else if (proxy.scheme == "https") { + auto tls = std::make_unique(); + tls->set_stop(stop); + tls->set_extra_ca_file(std::string(extraCaFile)); + if (!tls->connect(proxy.host.c_str(), proxy.port, timeoutMs, verifySsl)) { + // The session says why when the TCP connection was up, which is + // where a proxy certificate that does not verify is refused. + tunnel.error = "proxy: could not open a TLS connection to " + where; + if (!tls->error().empty()) tunnel.error += ": " + tls->error(); + return tunnel; + } + tunnel.error = http_connect(*tls, proxy, targetHost, targetPort, timeoutMs); + if (tunnel.error.empty()) tunnel.proxyTls = std::move(tls); + } else { + tunnel.error = "proxy: unsupported scheme '" + proxy.scheme + "'"; } + return tunnel; +} - // Socket is now tunneled to the target - return sock; +// The original entry point, kept for callers outside this repository. It cannot +// say why it failed; `proxy_tunnel` can. +export Socket proxy_connect(std::string_view proxyHost, int proxyPort, + std::string_view targetHost, int targetPort, + int timeoutMs) { + ProxyConfig proxy; + proxy.host = std::string(proxyHost); + proxy.port = proxyPort; + auto tunnel = proxy_tunnel(proxy, targetHost, targetPort, timeoutMs); + return std::move(tunnel.socket); } } // namespace mcpplibs::tinyhttps diff --git a/src/socket.cppm b/src/socket.cppm index 12b1e7a..de56075 100644 --- a/src/socket.cppm +++ b/src/socket.cppm @@ -1,6 +1,15 @@ module; -#ifdef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK #include #include #pragma comment(lib, "ws2_32.lib") @@ -21,7 +30,7 @@ import :platform; namespace mcpplibs::tinyhttps { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK using SocketHandle = SOCKET; constexpr SocketHandle INVALID_SOCKET_FD = INVALID_SOCKET; #else @@ -43,7 +52,8 @@ public: // Move constructor Socket(Socket&& other) noexcept - : fd_(other.fd_) { + : fd_(other.fd_), stop_(std::move(other.stop_)), deadline_(other.deadline_) + , deadline_hit_(other.deadline_hit_) { other.fd_ = INVALID_SOCKET_FD; } @@ -52,6 +62,9 @@ public: if (this != &other) { close(); fd_ = other.fd_; + stop_ = std::move(other.stop_); + deadline_ = other.deadline_; + deadline_hit_ = other.deadline_hit_; other.fd_ = INVALID_SOCKET_FD; } return *this; @@ -61,6 +74,48 @@ public: return fd_ != INVALID_SOCKET_FD; } + // Once the token is stopped, every wait on this socket (connect included) + // ends within 50 ms and reports "not ready". + void set_stop(std::stop_token stop) { stop_ = std::move(stop); } + + [[nodiscard]] bool stop_possible() const { return stop_.stop_possible(); } + + // Bounds the TLS handshake: until it is cleared, `wait_before_recv` gives up + // once this time has passed. Cleared once the handshake is over. + void set_deadline(std::optional deadline) { + deadline_ = deadline; + deadline_hit_ = false; + } + + // True once `wait_before_recv` gave up because of the deadline. + [[nodiscard]] bool deadline_hit() const { return deadline_hit_; } + + // What `bio_recv` calls ahead of a `recv` that would otherwise block. True + // when a `recv` now will not block. With neither a token nor a deadline it + // answers true without waiting, so the plain `recv` is as it was. + // + // A wait that ends before the deadline is waited again for what is left: a + // signal ends `poll` with EINTR even under SA_RESTART, which the blocking + // `recv` this wait stands in front of would have restarted. The deadline is + // hit only when the clock says so. + bool wait_before_recv() { + if (!stop_.stop_possible() && !deadline_) return true; + for (;;) { + int wait = -1; + if (deadline_) { + const auto left = std::chrono::ceil( + *deadline_ - std::chrono::steady_clock::now()).count(); + wait = left > 0 ? static_cast(left) : 0; + } + if (wait_readable(wait)) return true; + if (stop_.stop_requested() || !deadline_) return false; + if (std::chrono::steady_clock::now() >= *deadline_) { + deadline_hit_ = true; + return false; + } + } + } + bool connect(const char* host, int port, int timeoutMs) { // Close existing connection if any if (is_valid()) { @@ -73,6 +128,7 @@ public: // musl-static build can't — its nameservers live in $PREFIX/etc/resolv.conf // which libc never reads — so fall back to a manual DNS query there. auto try_resolved = [&](const char* node, bool numeric) -> bool { + if (stop_.stop_requested()) return false; struct addrinfo hints{}; hints.ai_family = AF_UNSPEC; hints.ai_socktype = SOCK_STREAM; @@ -88,12 +144,13 @@ public: return ok; }; - if constexpr (platform::is_windows) { + if constexpr (platform::uses_winsock) { return try_resolved(host, /*numeric=*/false); } else { // Fall back to a manual DNS query when libc can't resolve (Termux: // nameservers live in $PREFIX/etc/resolv.conf, which libc ignores). auto try_manual = [&]() -> bool { + if (stop_.stop_requested()) return false; // DNS must be snappy: a UDP query to a working resolver answers // in well under a second. Cap it hard (independent of the much // larger connect timeout) so an intermittently-dropped packet to @@ -119,9 +176,52 @@ public: } } + // The first address `host` resolves to, as network-order bytes: four for + // IPv4, sixteen for IPv6, none when it does not resolve. IPv4 wins when both + // exist, which is what a SOCKS5 proxy of any age can be asked for. + // `numericOnly` accepts an address literal and nothing else, with no lookup. + static std::vector resolve_address(const char* host, bool numericOnly = false) { + struct addrinfo hints{}; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + if (numericOnly) hints.ai_flags = AI_NUMERICHOST; + + std::vector bytes; + struct addrinfo* result = nullptr; + if (::getaddrinfo(host, nullptr, &hints, &result) == 0 && result != nullptr) { + for (auto* rp = result; rp != nullptr; rp = rp->ai_next) { + if (rp->ai_family == AF_INET) { + auto* in = reinterpret_cast( + &reinterpret_cast(rp->ai_addr)->sin_addr); + bytes.assign(in, in + 4); + break; + } + if (rp->ai_family == AF_INET6 && bytes.empty()) { + auto* in = reinterpret_cast( + &reinterpret_cast(rp->ai_addr)->sin6_addr); + bytes.assign(in, in + 16); + } + } + ::freeaddrinfo(result); + } + + // As in `connect`: where libc has no resolver configuration (Termux), + // fall back to a manual query. + if constexpr (!platform::uses_winsock) { + if (bytes.empty() && !numericOnly) { + for (const auto& ip : platform::resolve_fallback(host, 2500)) { + bytes = resolve_address(ip.c_str(), true); + if (!bytes.empty()) break; + } + } + } + return bytes; + } + // Connect to the first reachable address in a resolved list. bool connect_addrinfo(struct addrinfo* result, int timeoutMs) { for (auto* rp = result; rp != nullptr; rp = rp->ai_next) { + if (stop_.stop_requested()) return false; SocketHandle fd = ::socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); if (fd == INVALID_SOCKET_FD) { continue; @@ -131,7 +231,7 @@ public: // option rather than as a send flag; `write` below carries the flag // for the platforms that have one. // - // ⭐ NOTHING SELECTS THIS AND NOTHING MAY. It is not a feature, not + // NOTHING SELECTS THIS AND NOTHING MAY. It is not a feature, not // a config field and not a runtime probe: the preprocessor reads the // target's own and the answer is already complete. // Measured on this machine — glibc: SO_NOSIGPIPE absent, @@ -175,13 +275,13 @@ public: if (rc == 0) { connected = true; } else { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK if (WSAGetLastError() == WSAEWOULDBLOCK) { #else if (errno == EINPROGRESS) { #endif // Wait for connection with timeout - if (poll_fd(fd, timeoutMs, false)) { + if (wait_fd(fd, timeoutMs, false)) { int err = 0; socklen_t len = sizeof(err); if (::getsockopt(fd, SOL_SOCKET, SO_ERROR, reinterpret_cast(&err), &len) == 0 && err == 0) { @@ -246,12 +346,12 @@ public: bool wait_readable(int timeoutMs) { if (!is_valid()) return false; - return poll_fd(fd_, timeoutMs, true); + return wait_fd(fd_, timeoutMs, true); } bool wait_writable(int timeoutMs) { if (!is_valid()) return false; - return poll_fd(fd_, timeoutMs, false); + return wait_fd(fd_, timeoutMs, false); } [[nodiscard]] SocketHandle native_handle() const { @@ -266,23 +366,46 @@ public: } static void platform_init() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSADATA wsaData; WSAStartup(MAKEWORD(2, 2), &wsaData); #endif } static void platform_cleanup() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSACleanup(); #endif } private: SocketHandle fd_ = INVALID_SOCKET_FD; + std::stop_token stop_; + std::optional deadline_; + bool deadline_hit_ = false; + + // poll_fd in slices while a token is attached. A negative timeout waits + // without limit. Not std::min: defines a `min` macro. + bool wait_fd(SocketHandle fd, int timeoutMs, bool forRead) const { + if (!stop_.stop_possible()) return poll_fd(fd, timeoutMs, forRead); + constexpr int slice = 50; + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(timeoutMs); + while (!stop_.stop_requested()) { + int wait = slice; + if (timeoutMs >= 0) { + const auto left = std::chrono::duration_cast( + deadline - std::chrono::steady_clock::now()).count(); + if (left < slice) wait = left > 0 ? static_cast(left) : 0; + } + if (poll_fd(fd, wait, forRead)) return true; + if (timeoutMs >= 0 && std::chrono::steady_clock::now() >= deadline) return false; + } + return false; + } static bool set_non_blocking(SocketHandle fd, bool nonBlocking) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK u_long mode = nonBlocking ? 1 : 0; return ioctlsocket(fd, FIONBIO, &mode) == 0; #else @@ -298,7 +421,7 @@ private: } static bool poll_fd(SocketHandle fd, int timeoutMs, bool forRead) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSAPOLLFD pfd{}; pfd.fd = fd; pfd.events = forRead ? POLLIN : POLLOUT; @@ -314,7 +437,7 @@ private: } static void close_handle(SocketHandle fd) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK ::closesocket(fd); #else ::close(fd); diff --git a/src/tls.cppm b/src/tls.cppm index 39494ca..112e633 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -7,6 +7,29 @@ module; #include #include +// mbedTLS draws its entropy from BCryptGenRandom on Windows. Its package asks +// for the library as `-lbcrypt`, which link.exe does not read, so the request is +// made here in the form every linker for this target reads, as socket.cppm does +// for ws2_32. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#pragma comment(lib, "bcrypt.lib") +#endif + +// WHERE THE C LIBRARY IS musl, THE ENTROPY IS THE C LIBRARY'S. +// +// mbedTLS's own platform source asks the kernel through `getrandom` only where +// it recognises glibc (`entropy_poll.c`: `__linux__ && __GLIBC__`); with musl it +// opens `/dev/urandom`. Above openkal that file is not a thing the C library can +// promise — openkal reaches entropy through its own `openkal.random` interface, +// and openkal-musl answers `getrandom` with it — so on x86_64-windows-musl every +// handshake failed before it began, with "CTR_DRBG - The entropy source failed". +// musl's `getentropy` is `getrandom` underneath on every system it runs on, +// which is the source mbedTLS would have chosen had it known. The macro comes +// from mcpp.toml's `cfg(c-abi = "musl")`, because musl defines none of its own. +#ifdef TINYHTTPS_GETENTROPY +#include +#endif + export module mcpplibs.tinyhttps:tls; import :socket; @@ -71,6 +94,11 @@ static int bio_send(void* ctx, const unsigned char* buf, size_t len) { // complete and correct. static int bio_recv(void* ctx, unsigned char* buf, size_t len) { auto* sock = static_cast(ctx); + // With a stop token or a handshake deadline, wait here so the recv below + // cannot block past a stop or the deadline. + if (!sock->wait_before_recv()) { + return MBEDTLS_ERR_NET_RECV_FAILED; + } int ret = sock->read(reinterpret_cast(buf), static_cast(len)); if (ret < 0) { return MBEDTLS_ERR_NET_RECV_FAILED; @@ -78,6 +106,34 @@ static int bio_recv(void* ctx, unsigned char* buf, size_t len) { return ret; // 0 means end of stream; mbedtls turns it into SSL_CONN_EOF } +#ifdef TINYHTTPS_GETENTROPY +// `getentropy` gives at most 256 bytes a call. +static int c_library_entropy(void*, unsigned char* out, size_t len) { + while (len > 0) { + const size_t n = len < 256 ? len : 256; + if (::getentropy(out, n) != 0) return MBEDTLS_ERR_ENTROPY_SOURCE_FAILED; + out += n; + len -= n; + } + return 0; +} +#endif + +static std::string mbedtls_message(int ret) { + char buf[200]; + mbedtls_strerror(ret, buf, sizeof buf); + return buf; +} + +// verify_info ends every reason with a newline; join them with "; ". +static std::string one_line(std::string text) { + while (!text.empty() && text.back() == '\n') text.pop_back(); + for (std::size_t pos = 0; (pos = text.find('\n', pos)) != std::string::npos;) { + text.replace(pos, 1, "; "); + } + return text; +} + // WHAT A READ ENDED IN, WHICH `int` COULD NOT SAY. // // `TlsSocket::read` returned 0 for a peer that had closed AND for a transport @@ -108,11 +164,12 @@ public: // Move constructor TlsSocket(TlsSocket&& other) noexcept : socket_(std::move(other.socket_)) - , state_(std::move(other.state_)) { + , lower_(std::move(other.lower_)) + , state_(std::move(other.state_)) + , error_(std::move(other.error_)) + , extraCaFile_(std::move(other.extraCaFile_)) { // Re-bind BIO to point to our socket_ (not the moved-from one) - if (state_) { - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); - } + bind_bio(); } // Move assignment @@ -120,33 +177,72 @@ public: if (this != &other) { close(); socket_ = std::move(other.socket_); + lower_ = std::move(other.lower_); state_ = std::move(other.state_); + error_ = std::move(other.error_); + extraCaFile_ = std::move(other.extraCaFile_); // Re-bind BIO to point to our socket_ - if (state_) { - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); - } + bind_bio(); } return *this; } [[nodiscard]] bool is_valid() const { - return state_ != nullptr && socket_.is_valid(); + return state_ != nullptr && (lower_ ? lower_->is_valid() : socket_.is_valid()); + } + + // Why the last connect_over/connect failed once the TCP connection was up; + // empty if it failed earlier or has not failed. + [[nodiscard]] const std::string& error() const { return error_; } + + // Call before connect(); it covers the handshake and every later wait. + // + // THE TOKEN BELONGS TO THE SOCKET AT THE BOTTOM, AND ONLY THERE. Every wait + // ends in a `Socket` — this session's own, or, inside an https:// proxy's + // tunnel, the one beneath `lower_` — so that is where it is set. Setting + // only `socket_` left a reused tunnel holding the token of the call that + // opened it: a later call could not be cancelled, and once that first + // token was stopped every wait failed at once and the stale-connection + // retry sent a POST that had already arrived a second time. + void set_stop(std::stop_token stop) { + if (lower_) lower_->set_stop(stop); + socket_.set_stop(std::move(stop)); } + // A PEM file of roots to trust in addition to the default store. Call before + // connect(); setup fails if the file cannot be read or holds no certificate. + void set_extra_ca_file(std::string path) { extraCaFile_ = std::move(path); } + // Connect over an already-established Socket (e.g. a proxy tunnel). // Takes ownership of the socket and performs TLS handshake on top of it. - bool connect_over(Socket&& socket, const char* host, bool verifySsl) { + // The handshake gives up after `handshakeTimeoutMs`; a negative value waits + // without limit. + bool connect_over(Socket&& socket, const char* host, bool verifySsl, + int handshakeTimeoutMs = -1) { + error_.clear(); socket_ = std::move(socket); - return setup_tls(host, verifySsl); + return setup_tls(host, verifySsl, handshakeTimeoutMs); + } + + // Run the handshake inside another TLS session, which is how a client + // reaches a target through an https:// proxy: TLS to the proxy, CONNECT + // inside it, then this session to the target inside the tunnel. Takes + // ownership of `lower`, which must already be past its CONNECT. + bool connect_over(std::unique_ptr lower, const char* host, bool verifySsl, + int handshakeTimeoutMs = -1) { + error_.clear(); + lower_ = std::move(lower); + return setup_tls(host, verifySsl, handshakeTimeoutMs); } bool connect(const char* host, int port, int timeoutMs, bool verifySsl) { + error_.clear(); // Step 1: TCP connect via Socket if (!socket_.connect(host, port, timeoutMs)) { return false; } - return setup_tls(host, verifySsl); + return setup_tls(host, verifySsl, timeoutMs); } // The read that says which of the four things happened. Prefer it over @@ -207,6 +303,8 @@ public: mbedtls_ssl_close_notify(&state_->ssl); state_.reset(); } + // Only after the close_notify above, which travels through it. + lower_.reset(); socket_.close(); } @@ -215,7 +313,9 @@ public: if (state_ && mbedtls_ssl_get_bytes_avail(&state_->ssl) > 0) { return true; } - return socket_.wait_readable(timeoutMs); + // Under another session the bytes may be decrypted and waiting there + // while the descriptor has nothing, so ask that session, not the fd. + return lower_ ? lower_->wait_readable(timeoutMs) : socket_.wait_readable(timeoutMs); } // TLS back-pressure is a wait, not a failure. `mbedtls_ssl_write` reports @@ -223,35 +323,93 @@ public: // that into 0; without something to wait on, `write_all` could only retry // at once and then give up, which reached the caller as "Write failed". bool wait_writable(int timeoutMs) { - return socket_.wait_writable(timeoutMs); + return lower_ ? lower_->wait_writable(timeoutMs) : socket_.wait_writable(timeoutMs); } private: Socket socket_; + // The session this one runs inside, when there is one; `socket_` is unused + // then. Declared before `state_` so it is destroyed after it: the BIO of + // `state_` points at it. + std::unique_ptr lower_; std::unique_ptr state_; + std::string error_; + std::string extraCaFile_; + + // The session beneath this one, when there is one, is closed as well: a + // failed handshake to the target leaves nothing of the tunnel open. + bool fail(std::string message) { + error_ = std::move(message); + state_.reset(); + drop_transport(); + return false; + } + + // The BIO of a session that runs inside another is that session, which stays + // put when this object moves; the BIO of one on a socket is `socket_`, which + // moves with it. + void bind_bio() { + if (!state_) return; + if (lower_) { + mbedtls_ssl_set_bio(&state_->ssl, lower_.get(), send_over_tls, recv_over_tls, nullptr); + } else { + mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); + } + } + + static int send_over_tls(void* ctx, const unsigned char* buf, size_t len) { + int ret = static_cast(ctx)->write(reinterpret_cast(buf), + static_cast(len)); + if (ret < 0) return MBEDTLS_ERR_NET_SEND_FAILED; + if (ret == 0) return MBEDTLS_ERR_SSL_WANT_WRITE; + return ret; + } + + // As `bio_recv`: an end of stream is passed on as a zero. + static int recv_over_tls(void* ctx, unsigned char* buf, size_t len) { + auto r = static_cast(ctx)->read_some(reinterpret_cast(buf), + static_cast(len)); + switch (r.status) { + case ReadStatus::Data: return r.bytes; + case ReadStatus::Eof: return 0; + case ReadStatus::WouldBlock: return MBEDTLS_ERR_SSL_WANT_READ; + case ReadStatus::Error: break; + } + return MBEDTLS_ERR_NET_RECV_FAILED; + } - bool setup_tls(const char* host, bool verifySsl) { + void drop_transport() { + lower_.reset(); + socket_.close(); + } + + // As `set_stop`, the deadline belongs to the socket at the bottom, which + // inside a tunnel is the one beneath `lower_`. + void set_deadline(std::optional deadline) { + if (lower_) lower_->set_deadline(deadline); + else socket_.set_deadline(deadline); + } + + bool deadline_hit() const { return lower_ ? lower_->deadline_hit() : socket_.deadline_hit(); } + + bool setup_tls(const char* host, bool verifySsl, int handshakeTimeoutMs) { state_ = std::make_unique(); int ret = mbedtls_ctr_drbg_seed( +#ifdef TINYHTTPS_GETENTROPY + &state_->ctr_drbg, c_library_entropy, nullptr, +#else &state_->ctr_drbg, mbedtls_entropy_func, &state_->entropy, +#endif nullptr, 0); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); ret = mbedtls_ssl_config_defaults( &state_->conf, MBEDTLS_SSL_IS_CLIENT, MBEDTLS_SSL_TRANSPORT_STREAM, MBEDTLS_SSL_PRESET_DEFAULT); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); mbedtls_ssl_conf_rng(&state_->conf, mbedtls_ctr_drbg_random, &state_->ctr_drbg); @@ -261,6 +419,10 @@ private: // Load CA certs auto ca_pem = load_ca_certs(); + if (ca_pem.empty() && verifySsl) { + return fail("no CA certificate bundle found; set SSL_CERT_FILE to a PEM " + "file of trusted roots, or set verifySsl to false"); + } if (!ca_pem.empty()) { ret = mbedtls_x509_crt_parse( &state_->ca_cert, @@ -268,48 +430,68 @@ private: ca_pem.size() + 1); // +1 for null terminator required by mbedtls // ret > 0 means some certs failed to parse but others succeeded — acceptable if (ret < 0) { - state_.reset(); - socket_.close(); - return false; + return fail("cannot parse the CA certificate bundle: " + mbedtls_message(ret)); } + } + // Roots the caller adds to the above. A file that cannot be used is an + // error, not something to skip: the caller named it because a server + // needs it, and skipping would surface later as a verification failure + // that does not mention the file. + if (!extraCaFile_.empty()) { + std::ifstream in(extraCaFile_, std::ios::binary); + if (!in) return fail("cannot read extra CA file '" + extraCaFile_ + "'"); + std::string extra((std::istreambuf_iterator(in)), {}); + ret = mbedtls_x509_crt_parse( + &state_->ca_cert, + reinterpret_cast(extra.c_str()), extra.size() + 1); + if (ret < 0) { + return fail("cannot parse extra CA file '" + extraCaFile_ + "': " + + mbedtls_message(ret)); + } + } + if (!ca_pem.empty() || !extraCaFile_.empty()) { mbedtls_ssl_conf_ca_chain(&state_->conf, &state_->ca_cert, nullptr); } - // Certificate verification - // Use OPTIONAL (not REQUIRED) so handshake succeeds even if the CA - // bundle is incomplete; callers that need strict verification can - // inspect the verification result after handshake. - if (verifySsl) { - mbedtls_ssl_conf_authmode(&state_->conf, MBEDTLS_SSL_VERIFY_OPTIONAL); - } else { - mbedtls_ssl_conf_authmode(&state_->conf, MBEDTLS_SSL_VERIFY_NONE); - } + // Certificate verification. REQUIRED makes the handshake fail on a bad + // chain, an expired certificate or a name that is not `host`. + mbedtls_ssl_conf_authmode(&state_->conf, + verifySsl ? MBEDTLS_SSL_VERIFY_REQUIRED : MBEDTLS_SSL_VERIFY_NONE); ret = mbedtls_ssl_setup(&state_->ssl, &state_->conf); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); // Set hostname for SNI ret = mbedtls_ssl_set_hostname(&state_->ssl, host); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; + if (ret != 0) return fail(mbedtls_message(ret)); + + // Set BIO callbacks using our Socket, or the session beneath this one + bind_bio(); + + // Perform TLS handshake. The socket is blocking, so `bio_recv` waits + // against the deadline before it reads, and the deadline is taken off + // again before the connection is used. + std::optional deadline; + if (handshakeTimeoutMs >= 0) { + deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(handshakeTimeoutMs); } - - // Set BIO callbacks using our Socket - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); - - // Perform TLS handshake - while ((ret = mbedtls_ssl_handshake(&state_->ssl)) != 0) { - if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE) { - state_.reset(); - socket_.close(); - return false; + set_deadline(deadline); + while ((ret = mbedtls_ssl_handshake(&state_->ssl)) == MBEDTLS_ERR_SSL_WANT_READ + || ret == MBEDTLS_ERR_SSL_WANT_WRITE) {} + const bool timedOut = deadline_hit(); + set_deadline({}); + if (ret != 0) { + if (ret == MBEDTLS_ERR_X509_CERT_VERIFY_FAILED) { + char info[512] = {}; + mbedtls_x509_crt_verify_info(info, sizeof info, "", + mbedtls_ssl_get_verify_result(&state_->ssl)); + return fail("certificate verification failed: " + one_line(info)); + } + if (ret == MBEDTLS_ERR_NET_RECV_FAILED && timedOut) { + return fail("TLS handshake timed out"); } + return fail("TLS handshake failed: " + mbedtls_message(ret)); } return true; diff --git a/tests/extra_ca_certs.hpp b/tests/extra_ca_certs.hpp new file mode 100644 index 0000000..bfd8229 --- /dev/null +++ b/tests/extra_ca_certs.hpp @@ -0,0 +1,161 @@ +// Two private CAs, each with one server certificate (CN=localhost, SAN localhost +// and 127.0.0.1), for test_extra_ca.cpp. Valid for a century. The keys are here +// on purpose: they authenticate nothing and must never be used for anything. +// +// Made with openssl: a self-signed CA with basicConstraints CA:TRUE and +// keyUsage keyCertSign, and a leaf signed by it whose extensions are +// basicConstraints CA:FALSE, extendedKeyUsage serverAuth and +// subjectAltName DNS:localhost,IP:127.0.0.1. +#pragma once + +namespace extra_ca_test { + +inline constexpr const char* kCa1Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDLzCCAhegAwIBAgIUG2Zap6LiJwyHlkOt1Ko7G28l+DcwDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMTAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3Qg +Q0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALCFaWIMBSe8OolK +5ExxlX3LbNgg3XlOLxa604zHI0nLWFhKJalkticwwMaX4gcJQdETCXADq+V8qj48 +014c3uUC7xlb+INULhe2ocThipOSwhh4E1Vyv8ywNhgVEw8r+Lgl2ExaSIGzcx4O +2tWuOvzQHhe6JG8G5YRWVk5ydimhFGY6VdGzUPfHAID+3JEqr6RUeCct4Ehx2zbg +mkn7ZBu0PHtP2fjrc5KnVwXWDH91MbSfJ1WQvPq5kPbSpeeLoWvDznzpUjg+9aTB +Dbyxu+/5YVFy5jWWEpgOKakyKnfsQ6XTqzz6dvogAjjh3UWTHe01osluXqY8tjls +gapw+N8CAwEAAaNjMGEwHQYDVR0OBBYEFP3PSVqEY/jqpWJmAzftRY3zFZPtMB8G +A1UdIwQYMBaAFP3PSVqEY/jqpWJmAzftRY3zFZPtMA8GA1UdEwEB/wQFMAMBAf8w +DgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQAhMIwH6LSKiywbxaFK +RlUIqgThCIP9uUOZ0m/6jjgMwSJhU0IZ4COCewFVPEGepeqhNjDXHD9Y0EHbxDi5 +cCsk2fSz9xfsjBniI+xETlxGLoCrNDAmNEfPv8mT9UnNKG7KqLghQiHpMbx47895 +wTbr1OldFAlU8usR9pOigRc6j5cmf9b1Tu/piUTFsy5xoANPt/HRg3o/w0mfR8c5 +bHQ3LaqcQO+uBpKAy9guHpszKosOmbTX6AB5JBSyo70u3jx0PHgQuOJTEeKP1PBJ +51EyNShztp7bn+Rz0OLsMk2h6AMns7HB2m5gEIq/ELlaWDYruqiYFmGpqcm7deVd +v6DH +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf1Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDTzCCAjegAwIBAgIUO+WtedkpuiIwKSiJ20nYcEG4pNQwDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMTAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAj3gXKIHuAZ2h0DeMqxZX2ImgZJan +arOOw8vKg8yqyfgAeEtenmH42Hv4DAPhHVycq9DyXutiXVxQ+v1y7ABfG9RG2Z0u +Ueen2TqNDz5F6CrlUeG/Fgk3OVOWVeglcJpd8vqiVzrgRRtc4+uREGlu2YPusgSX +rbVL+6VARdEfypluaStOt1kJkQlLUxa5A7OpsXKIDLRnOKSHCyx1CCJNy3jwPmqz +vng6M/FYns/73TxqSZ8Wi7CRdCpNWjlwfJ0zunjeYdrGUchgVpbyWedexZzKwLMO +NFnjY20qPmN0ZIcXSg+IfZJcUKSD6t+kWZAOtcFwrO6xi7SWFQwNyskW3wIDAQAB +o4GMMIGJMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF +BwMBMBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATAdBgNVHQ4EFgQUNtDg1vH4 +jNeYzFRvdHWYhhdO6WEwHwYDVR0jBBgwFoAU/c9JWoRj+OqlYmYDN+1FjfMVk+0w +DQYJKoZIhvcNAQELBQADggEBAEUwKddb7KxhgZ5a41u78wRANsfkXHJEXX6l2IWL +cOAE9PU2LFj4muABzp7g6WA5rOF5Dt34E3PSB8EQIMTY6SBA9V6Y80/8uBlsF+Er +Labg6ABrvswpGPn7sc65m8uDUYdG5jm0P2lure9mV8kdiYURk0/eOIpdCp8CcuFx +iG1LsAMz9h0WAgbkfIzigDGx9TkvqTZNSfzvvObgGmeRfo++pLMEjifug9NBRHuN +dMiQ4NBvRv+HvJHGSAD1fpm8QvEiMk7spf+PyZbJZP34YZlbJfZJXvBxnTnJY6rD +amz7EBggI4kRiDGG5vxgyH2bYe9uzw01jb02QJyUvJqvOv4= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf1KeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCPeBcoge4BnaHQ +N4yrFlfYiaBklqdqs47Dy8qDzKrJ+AB4S16eYfjYe/gMA+EdXJyr0PJe62JdXFD6 +/XLsAF8b1EbZnS5R56fZOo0PPkXoKuVR4b8WCTc5U5ZV6CVwml3y+qJXOuBFG1zj +65EQaW7Zg+6yBJettUv7pUBF0R/KmW5pK063WQmRCUtTFrkDs6mxcogMtGc4pIcL +LHUIIk3LePA+arO+eDoz8Viez/vdPGpJnxaLsJF0Kk1aOXB8nTO6eN5h2sZRyGBW +lvJZ517FnMrAsw40WeNjbSo+Y3RkhxdKD4h9klxQpIPq36RZkA61wXCs7rGLtJYV +DA3KyRbfAgMBAAECggEAI/sGM86CFj2mO0XzLWUpGtFJPHDcA9KMV2Zgn7VQj952 +n7SnQiln80MD7l96sZZsZFq+rjdyL9/bH1c43x8oux39ts3ItBnY6o3Ymp4Pgi5f +3K008x3Uj7f62Aikexz7uthIpn/JhwtZvH9znh4TAtInptWoBJSXPcEv4VKVGoYI +HglTL1/TvIkyo2hwdLBqOuUF5PioHktG8MZH2ILqFJTfB5bWmOPXDMAEuZD9pkQK +LfyD9AntTRKVxRO/sXjHy27IXSKbDoZq4kqqLkxk0BGJpf4BnxxjZUWoqbQ7MZHs +MS26/fcpMWSJygYT8vMkAQYrGjrEkKQSvABFTmkegQKBgQDASz/4vpa+qPf9soX8 +n9+XXSPBeyNeNKpl2p0b+7h8skRIdi/gmttSiC9g/L2fuI2rpoikii1qxQ1XN+wA +LOcOrL5p/RJdGKAyu8n8cRSGFkhkOO5s2zLsSdXUHesVqWc8keI37DoJkySYEsiC +WI0MZNNK01cUFttD+nEqxgCicQKBgQC+/+3u+aWF1rumheoUV9pRk65F0jJRnTAQ +bfiRE7jWhs/LSYpJ1EOgc0OYaRnJPwKYuck4/Z7K3beNKPH3jPJ+89UvfRjHKE/3 +o89yACWZATh2XMjvZUUUmvn+Om+yMBxWhy1vyTJU+XhFVSIROofqg5xH3F3hMC6P +WIyJctVWTwKBgDUdE+BrFEw5c7Y5d1Td++5dJaEtmAlPVrmndrnh/4VB4CZ7rqHF +ZEsZnqyVYvMZENiWuStplz3ki9jJc4Bg5rlg8X9pDYd1Y9pfkF0QBvE6emhkaUbc +DObDRjK+yM5E6mUKjoeP0kAOyo7OsV9/ggYoW0xRr722yn2uUU/VPRuhAoGBALu/ +TX5cTdWa98g/2JH4rbUEqwkdDrNlBQsRXoYVUdsml3mnp1TdX0CILQjTPzKwSngg +H+lyfLXDbhtHBVqmZPRf8M4GdOXH/ZtanZ7dABg/t+W5XRbdgCM6F8VMYeFRI3n2 +1LYOmvgmZZAZmCxBUbs3dda5ilrCBvKHQ1YU2BpHAoGAGZVbkJ0lnCoXf3tyMBCZ +MliDj7Q6fSGd22ZAoYy/XGLmZUMi/ZK+PkdzVwT6CDwJY07vBDwUIj5qBHlmaHGS +qC7o3bmhdtC4R+eN4Gb6mdKB3wCdrRhx2HTBGDf4aoPwGtUw+kUdlve1L+Rwsdfc +zCIifQbNtFiGohWrskdamio= +-----END PRIVATE KEY----- +)PEM"; + +inline constexpr const char* kCa2Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDLzCCAhegAwIBAgIUe6yBH1jY6Ogs0HBiaUEsnbZlB54wDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMjAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3Qg +Q0EgMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJXfKHI6jLsG/yLn +UqcDb+PBuw4+C6eaYUwgJNJ5VNubXZpjg1JJxqBqOYFKJ69J5OULqBD2A3DWCAKh +vh6FOHeSdXlvwnZQN0fLwtnYcmL90XJaDrUwaFn/NrceqsgBo+pNXHcVb6OYwxQr +Np9/3kta6vbWp9cC8D6vCkmp2ehrz2UfBo4m6oh1Re2ATEBn0H2OD41LKyQslpF4 +CFkm2rIdkTsqlGtdoBkE+JyBX+eoZAlnRBRC5NxJDLEtDlxoPlbVRP/YgPeazv1g +tMgUvBhmC5AROU6oEpiXiGxas/3m3av/21shsq0iz+b+1QOF5gt9MdOEm2mxrxfF +hkf0pH0CAwEAAaNjMGEwHQYDVR0OBBYEFMV3Z2Qg+uzCT8vmKJV84SBhwzQQMB8G +A1UdIwQYMBaAFMV3Z2Qg+uzCT8vmKJV84SBhwzQQMA8GA1UdEwEB/wQFMAMBAf8w +DgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQBmve5C71LtGh24Sq3Q +kNsEROTQYwjbI1W1tEC5C/bqeUFLl4YxyzIHyGFiNSZ7aaaOSRrMEY2aCG2f7am0 +Rckygp6fFaTTAfWnsgH7fVxvBEwywNlLDaAU/co20cnFa1MfUO1MfstF56lqllgX +NHE/PAOCV/ZcO3DAmJ1lWbsJcXQKplbNlZIeEBwqJs2i3xpiJHxhckg95wyq2vnX +qtCBiGlK7LBawlHLAbYEXf4LTDAJvb5OFfb8vIJN5pn7PKvMhhMtp4i0r0nDgLkR +nP25ABxH8uw8KV7rCzp6i0LDSMRAU6pbs/C8kvMSv8BmF2FFK4OX4+29aIPH/utP +x/4p +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf2Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDTzCCAjegAwIBAgIUBQEev9nx9U9d1N63ZONvDn/2u9owDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMjAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4piCGItiClSpVhm3Rw2h/OTfgyE4 +vrEXqkEcHB7Oq4z23KeluS9r6eM/6i0ezRUTCI71Q6UBQ/n7mQUPb5ePHZygVqL+ +QEYGiO6xH2q3kB3oDzNixaPYUCg0u7WLCbv4E6E8ngqcpMSBTxTH/Be3AB1j0I9a +AH3xmVbY1E2V79uYhRVc7cQ0mNfXkj1K4G0AaPKuiwwVKnE7/z/BwA4vqWjH9jsV +s1JFhjPpydBY73VWEVE0Ou1ntBNoKGilsVJBs4lf4P8pu9hBS8H0PCAk784kVY8n +0Y1G2xU3fSash6HjljjKtWg+mudbu2edVG8pkv13JYxJNNIUCXLvRH6m4QIDAQAB +o4GMMIGJMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF +BwMBMBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATAdBgNVHQ4EFgQUSfHOwhMo +5rrgK8aDsQ4K7lvGJ5IwHwYDVR0jBBgwFoAUxXdnZCD67MJPy+YolXzhIGHDNBAw +DQYJKoZIhvcNAQELBQADggEBAIQ2kQBTG7wSFAa/fGup6cph+pV2ZVikoQUN0C9U +29uJaGALXNhx5RUDduPZP0i1Q9chYh55HcCT/CtbVlkO3X3BwLmgvqJel0O0nDnR +bIA90Ccs0wOYBY1bkWqcjqALJlddjeeerYdV3PulAp5uklf5ZWxgxl6XSucRW1fH +wvtBbQ7TirbYos0nZqAF4pxpMq4KLfWa7eAVMThL6ImiQyOYZsblMxZqAOPh8RIU +lmZG8/79aHwWyQdCC7p91dD5bCtOwrn1VZvTgmrzp8+TQnzqGuKAlUZEJXp/OF+a +qana9iMkkKQyE7y+DGZz09xKrvlh8l7kxJMES6ynN/XjITg= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf2KeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDimIIYi2IKVKlW +GbdHDaH85N+DITi+sReqQRwcHs6rjPbcp6W5L2vp4z/qLR7NFRMIjvVDpQFD+fuZ +BQ9vl48dnKBWov5ARgaI7rEfareQHegPM2LFo9hQKDS7tYsJu/gToTyeCpykxIFP +FMf8F7cAHWPQj1oAffGZVtjUTZXv25iFFVztxDSY19eSPUrgbQBo8q6LDBUqcTv/ +P8HADi+paMf2OxWzUkWGM+nJ0FjvdVYRUTQ67We0E2goaKWxUkGziV/g/ym72EFL +wfQ8ICTvziRVjyfRjUbbFTd9JqyHoeOWOMq1aD6a51u7Z51UbymS/XcljEk00hQJ +cu9EfqbhAgMBAAECggEAFmUu1HiwKae47cjQX64mAK3pNldZ+aTcmyfgvJnB4Nwn +iOLr4uvgLUdXyZAIj4nh7fJdJMCh6aNDHdpNboDK9QkGItHssZMKMc3vhRShzFVc +WmZAgJqjNjbnEVdNxNpGWEBTEqanWTSPuC5sXorEfDcrZpFIeqx37KkT+JWreD9r +DEsHVL51wGwD7KC06qv/IUbQXrfUtwJouof0X3iF/1bSmBTi3gVvyRpBaECRAZab +bOktlfKP/9FquA7KegY0HaaByXC073lj7pl3H9AmnYihXfTHSnxRMLq31IhpWzIN +Bbx/+vaVWgt/lWdedAVg8wVqlen4PSTstKYpq9WrHQKBgQD0SeekTgwjHfixaGm8 +7BoYHrszO32ua7BPAi8zXjbJi2SwFqyLlsw52FaYMMFxWRCarhtOKzUR0AnfB+Tp +HUsNlg5IOzeBXz26Care6Ymmx0jc1ZTrNMNoSTRMhZ4Cq16pD/mXSFNRQaBxP1Uy +ZTKzbz/UMZ6dbw+wY0lRp5gABQKBgQDtdXZHyPWns5OY34IYkO/cX3FY7K6BqqQz +8ciULN1v8bK32J8NXuuHRpbl+gJQ9ZwSzTyUBEW2EJArSmiVQBWLfITOv+Ubb34a +7RmiMO7ectqDsydpXyNr5GVvmJSN7ua/Bgse71ZEER+5T+Wk1IFrmc+dgWD7p1wX +DPoRGVruLQKBgGxA++extmsxcu2bKZBQyFd79tUDPfS3QBAjFaubyoeRv/LQeXH6 +R1Iy7lzO7ko+f78r/Gmtd/0GxfkR1H0BQCsrHfUgWL556C6y7geOUjOYCE/kTfPV +E+r41bY7WAQy0OX2cFJ7+H//pwvIVwbYFZmyaQStYG6/sqIlvW3z6MXJAoGAFNv1 +qPapRbMLqykvJ3NL7sLflR0tcyyHQhziLsoGJbhIp8e6qRUGgZqc1NTycSQyMqCe +yq3ZZdXspDDkIQBBJECq9fJ7cL6JiFn9n3bG9PmH4nvRYXs/BFJK6tOvcQxkbrFc +Nrp1jIMok0HGfYVBXS9kwoRD/bdE6q10Jh5hE30CgYEA1cUVunt7NYrY2NV+jF+1 +kp9Imom/F0ao4wowMmrcJTm1zgw4BehYaRoE7BVOxoq4Rc7SBgSlvb8aoy7EXuu7 +eieh8GE4WeiFGcDtCdpS82s/3IOeHwHo9I7wFa16r1jy/ooygVPM3RB8q7KOmfP+ +Hmwo4XQ7djlELruiEfglIt4= +-----END PRIVATE KEY----- +)PEM"; + +} // namespace extra_ca_test diff --git a/tests/proxy_test_server.hpp b/tests/proxy_test_server.hpp new file mode 100644 index 0000000..0d3d32d --- /dev/null +++ b/tests/proxy_test_server.hpp @@ -0,0 +1,242 @@ +// A proxy, in this process, that does exactly what a test tells it to. +// +// The proxy tests need the three things a real proxy does between accepting a +// connection and relaying bytes: read what the client asked for, answer it, and +// then splice the client to the target. A handler scripts the first two and +// `Peer::relay_to` does the third, so one test can make the proxy refuse, stall +// or hang up, and another can let a request through to the TLS server from +// tls_test_server.hpp and check what arrived. +// +// Plain TCP on 127.0.0.1, port chosen by the OS. It uses mbedtls's net layer +// for the same reason tls_test_server.hpp does: it is already a dependency and +// it is the same on every platform the library builds for. +#pragma once + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// Which socket interface this is built against; see socket.cppm. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK +#include +#include +#else +#include +#include +#endif + +namespace proxy_test { + +class Server; + +// The accepted connection, as the handler sees it. +class Peer { +public: + Peer(Server& server, mbedtls_net_context net) : server_(server), net_(net) {} + ~Peer() { mbedtls_net_free(&net_); } + Peer(const Peer&) = delete; + Peer& operator=(const Peer&) = delete; + + // Everything up to and including the blank line that ends an HTTP head, + // read a byte at a time so nothing past it is taken. Empty on a hang-up. + std::string read_head(); + + // Exactly `n` bytes, or empty when the client hangs up first. + std::string read_exact(std::size_t n); + + // Whatever arrives first, up to 4 KiB. Empty on a hang-up. + std::string read_some(); + + bool write(std::string_view data) { + std::size_t sent = 0; + while (sent < data.size()) { + int ret = mbedtls_net_send(&net_, + reinterpret_cast(data.data() + sent), + data.size() - sent); + if (ret <= 0) return false; + sent += static_cast(ret); + } + return true; + } + + void note(std::string text); + + // Splices this connection to 127.0.0.1:`port` until either side ends. + void relay_to(int port); + +private: + bool wait_readable(int ms) { return mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, ms) > 0; } + + Server& server_; + mbedtls_net_context net_; +}; + +class Server { +public: + using Handler = std::function; + + explicit Server(Handler handler) : handler_(std::move(handler)) { + mbedtls_net_init(&listener_); + if (mbedtls_net_bind(&listener_, "127.0.0.1", "0", MBEDTLS_NET_PROTO_TCP) != 0) { + failed_ = true; + return; + } + sockaddr_in addr {}; + socklen_t len = sizeof addr; + if (::getsockname(listener_.fd, reinterpret_cast(&addr), &len) != 0) { + failed_ = true; + return; + } + port_ = ntohs(addr.sin_port); + acceptor_ = std::thread([this] { accept_loop(); }); + } + + ~Server() { stop(); } + Server(const Server&) = delete; + Server& operator=(const Server&) = delete; + + void stop() { + if (stopping_.exchange(true)) return; + if (acceptor_.joinable()) acceptor_.join(); + for (auto& t : connections_) { + if (t.joinable()) t.join(); + } + connections_.clear(); + mbedtls_net_free(&listener_); + } + + [[nodiscard]] bool failed() const { return failed_; } + [[nodiscard]] int port() const { return port_; } + [[nodiscard]] bool stopping() const { return stopping_.load(); } + [[nodiscard]] int accepts() const { return accepts_.load(); } + + // The text a handler stored with `note`, for the test to assert on once the + // request has completed. + void note(std::string text) { + std::lock_guard lock(mutex_); + notes_.push_back(std::move(text)); + } + [[nodiscard]] std::vector notes() { + std::lock_guard lock(mutex_); + return notes_; + } + +private: + void accept_loop() { + while (!stopping_.load()) { + int ready = mbedtls_net_poll(&listener_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) break; + if (ready == 0) continue; + + mbedtls_net_context client {}; + mbedtls_net_init(&client); + if (mbedtls_net_accept(&listener_, &client, nullptr, 0, nullptr) != 0) { + mbedtls_net_free(&client); + continue; + } + accepts_.fetch_add(1); + connections_.emplace_back([this, client] { + Peer peer(*this, client); + handler_(peer); + }); + } + } + + Handler handler_; + mbedtls_net_context listener_ {}; + int port_ { 0 }; + bool failed_ { false }; + std::atomic stopping_ { false }; + std::atomic accepts_ { 0 }; + std::thread acceptor_; + std::vector connections_; + std::mutex mutex_; + std::vector notes_; +}; + +inline void Peer::note(std::string text) { server_.note(std::move(text)); } + +inline std::string Peer::read_head() { + std::string head; + unsigned char c {}; + while (head.find("\r\n\r\n") == std::string::npos) { + if (head.size() > 65536) return {}; + // 20 s is far beyond any test; the loop exists so `stop` is noticed. + bool ready = false; + for (int waited = 0; waited < 20000 && !ready; waited += 50) { + if (server_.stopping()) return {}; + ready = wait_readable(50); + } + if (!ready) return {}; + if (mbedtls_net_recv(&net_, &c, 1) <= 0) return {}; + head.push_back(static_cast(c)); + } + return head; +} + +inline std::string Peer::read_exact(std::size_t n) { + std::string out; + unsigned char c {}; + while (out.size() < n) { + bool ready = false; + for (int waited = 0; waited < 20000 && !ready; waited += 50) { + if (server_.stopping()) return {}; + ready = wait_readable(50); + } + if (!ready || mbedtls_net_recv(&net_, &c, 1) <= 0) return {}; + out.push_back(static_cast(c)); + } + return out; +} + +inline std::string Peer::read_some() { + unsigned char buf[4096]; + for (int waited = 0; waited < 20000; waited += 50) { + if (server_.stopping()) return {}; + if (!wait_readable(50)) continue; + int n = mbedtls_net_recv(&net_, buf, sizeof buf); + return n > 0 ? std::string(reinterpret_cast(buf), static_cast(n)) + : std::string(); + } + return {}; +} + +inline void Peer::relay_to(int port) { + mbedtls_net_context target {}; + mbedtls_net_init(&target); + if (mbedtls_net_connect(&target, "127.0.0.1", std::to_string(port).c_str(), + MBEDTLS_NET_PROTO_TCP) != 0) { + return; + } + // Round-robin over two sockets with a short poll each: slower than a + // select over both, and portable without a second code path. + unsigned char buf[4096]; + bool open = true; + while (open && !server_.stopping()) { + for (auto [from, to] : {std::pair{&net_, &target}, std::pair{&target, &net_}}) { + if (mbedtls_net_poll(from, MBEDTLS_NET_POLL_READ, 10) <= 0) continue; + int n = mbedtls_net_recv(from, buf, sizeof buf); + if (n <= 0) { open = false; break; } + for (int sent = 0; sent < n;) { + int w = mbedtls_net_send(to, buf + sent, static_cast(n - sent)); + if (w <= 0) { open = false; break; } + sent += w; + } + } + } + mbedtls_net_free(&target); +} + +} // namespace proxy_test diff --git a/tests/test_ca_store.cpp b/tests/test_ca_store.cpp new file mode 100644 index 0000000..38e2dd7 --- /dev/null +++ b/tests/test_ca_store.cpp @@ -0,0 +1,42 @@ +// The trust store the platform supplies is found with nothing configured, and a +// public certificate chain verifies against it. +// +// On a Windows Sockets build the store is the system's ROOT store, read through +// the Win32 API; elsewhere it is a bundle file at one of the known locations. +// Either way a default-constructed client verifies a public server. A host that +// cannot be reached is a skip, so a runner without egress reports "not run"; a +// certificate the client refused is a failure. +#include + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +TEST(SystemStore, IsFoundWithNothingConfigured) { + if (const char* file = std::getenv("SSL_CERT_FILE"); file && *file) { + GTEST_SKIP() << "SSL_CERT_FILE is set, so the system store is not consulted"; + } + const auto pem = https::load_ca_certs(); + EXPECT_FALSE(pem.empty()); + EXPECT_NE(pem.find("-----BEGIN CERTIFICATE-----"), std::string::npos); +} + +TEST(SystemStore, APublicChainVerifiesAgainstIt) { + https::Socket::platform_init(); + https::HttpClientConfig config; + config.connectTimeoutMs = 15000; + config.readTimeoutMs = 20000; + https::HttpClient client{config}; + + https::HttpRequest request; + request.url = "https://one.one.one.one/"; + const auto response = client.send(request); + + if (response.statusCode == 0 && response.statusText == "Connection failed") { + GTEST_SKIP() << "one.one.one.one is not reachable from this runner"; + } + EXPECT_NE(response.statusCode, 0) << response.statusText; +} diff --git a/tests/test_cancel.cpp b/tests/test_cancel.cpp new file mode 100644 index 0000000..9d677bc --- /dev/null +++ b/tests/test_cancel.cpp @@ -0,0 +1,812 @@ +// Abandoning a request from another thread through a std::stop_token. +// +// Every wait here is far longer than the test should take (readTimeoutMs is +// 30 s), so a request that returns promptly with `cancelled` set can only have +// been ended by the token, never by a timeout. + +// WHERE `import std` GOES DEPENDS ON THE STANDARD LIBRARY, AND ONLY HERE. +// +// Every other test file includes its headers and then imports std, which is the +// order libstdc++ needs: the other way round, gcc 16 rejects the headers as +// redefinitions of what the module already declared. This file is the one that +// calls `std::stop_source::request_stop()`, and libc++ 20 and 22 compile that +// call, after a textual or (gtest and the test +// servers include both), into a reference to an inline helper that nothing then +// emits: the link fails on `__atomic_unique_lock<...>::__set_locked_bit`. With +// the import first it links, and libc++ 23 links either way. Measured with the +// toolchains mcpp installs; see .agents/docs/design-cancel-stop-token.md §6.2. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif + +namespace https = mcpplibs::tinyhttps; + +namespace { + +using Clock = std::chrono::steady_clock; + +// Well above the 50 ms slice and still far below any timeout in play. +constexpr long long kPromptMs = 500; + +https::HttpClientConfig test_config(int readTimeoutMs = 30000) { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = readTimeoutMs; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::GET; + req.url = url; + return req; +} + +long long ms_between(Clock::time_point from, Clock::time_point to) { + return std::chrono::duration_cast(to - from).count(); +} + +// Requests a stop `delay` after `ready` first holds (at once when it is not +// given) and remembers when it did, so a test can measure from the stop to the +// return of the call. Waiting for the server to have seen the request keeps a +// slow machine from stopping a request that was never sent. +class Stopper { +public: + explicit Stopper(std::chrono::milliseconds delay, + std::function ready = {}) + : thread_([this, delay, ready = std::move(ready)] { + for (int i = 0; ready && !ready() && i < 1000 && !quit_.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + for (auto end = Clock::now() + delay; + Clock::now() < end && !quit_.load();) { + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + if (quit_.load()) return; + stoppedAt_.store(Clock::now().time_since_epoch().count()); + source.request_stop(); + }) {} + // A stop that has not come yet is abandoned, so a long delay costs nothing + // when the call it was meant to end has already returned. + ~Stopper() { + quit_ = true; + if (thread_.joinable()) thread_.join(); + } + + long long since_stop_ms(Clock::time_point returned) { + thread_.join(); + return ms_between(Clock::time_point(Clock::duration(stoppedAt_.load())), returned); + } + + std::stop_source source; + +private: + std::atomic stoppedAt_ { 0 }; + std::atomic quit_ { false }; + std::thread thread_; +}; + +// A TCP listener that never speaks TLS: it accepts, reads and discards, so a +// client's handshake waits for a ServerHello that is not coming. +class SilentListener { +public: + SilentListener() { + mbedtls_net_init(&listener_); + if (mbedtls_net_bind(&listener_, "127.0.0.1", "0", MBEDTLS_NET_PROTO_TCP) != 0) { + failed_ = true; + return; + } + sockaddr_in addr {}; + socklen_t len = sizeof addr; + if (::getsockname(listener_.fd, reinterpret_cast(&addr), &len) != 0) { + failed_ = true; + return; + } + port_ = ntohs(addr.sin_port); + thread_ = std::thread([this] { run(); }); + } + + ~SilentListener() { + release(); + if (thread_.joinable()) thread_.join(); + mbedtls_net_free(&listener_); + } + + [[nodiscard]] bool failed() const { return failed_; } + [[nodiscard]] std::string url() const { + return "https://127.0.0.1:" + std::to_string(port_) + "/"; + } + [[nodiscard]] bool peer_closed() const { return peerClosed_.load(); } + [[nodiscard]] bool received() const { return received_.load(); } + + // Lets go of the accepted connection and stops listening. + void release() { released_.store(true); } + +private: + void run() { + while (!released_.load()) { + const int ready = mbedtls_net_poll(&listener_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) return; + if (ready == 0) continue; + mbedtls_net_context client {}; + mbedtls_net_init(&client); + if (mbedtls_net_accept(&listener_, &client, nullptr, 0, nullptr) != 0) { + mbedtls_net_free(&client); + continue; + } + unsigned char buf[512]; + while (!released_.load()) { + const int r = mbedtls_net_poll(&client, MBEDTLS_NET_POLL_READ, 50); + if (r < 0) break; + if (r == 0) continue; + if (mbedtls_net_recv(&client, buf, sizeof buf) <= 0) { + peerClosed_.store(true); + break; + } + received_.store(true); + } + mbedtls_net_free(&client); + return; + } + } + + mbedtls_net_context listener_ {}; + int port_ { 0 }; + bool failed_ { false }; + std::atomic released_ { false }; + std::atomic peerClosed_ { false }; + std::atomic received_ { false }; + std::thread thread_; +}; + +// Waits up to `limit` for `predicate`, which is how a test learns that the server +// saw the connection end. +bool becomes_true(const std::function& predicate, + std::chrono::milliseconds limit = std::chrono::seconds(2)) { + const auto deadline = Clock::now() + limit; + while (!predicate() && Clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + return predicate(); +} + +class CancelTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +} // namespace + +// The case that motivates all of this: the request is written and the server +// has not answered. The server also records that the connection ended, which is +// what makes it stop working on a request nobody is waiting for. +TEST_F(CancelTest, ARequestWaitingForTheHeadersIsAbandoned) { + std::atomic peerClosed { false }; + tls_test::Server server([&](tls_test::Conn& conn, int) { + peerClosed = conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(200), [&] { return server.requests() >= 1; }); + + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + const auto returned = Clock::now(); + const auto latency = stopper.since_stop_ms(returned); + std::cout << "[ latency ] headers: " << latency << " ms after the stop" << std::endl; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_FALSE(res.bodyComplete); + EXPECT_LT(latency, kPromptMs); + EXPECT_TRUE(becomes_true([&] { return peerClosed.load(); })) + << "the connection was not closed"; +} + +TEST_F(CancelTest, AStreamWaitingForTheNextChunkIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n"); + conn.write("b\r\ndata: one\n\n\r\n"); + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + std::atomic events { 0 }; + Stopper stopper(std::chrono::milliseconds(100), [&] { return events.load() > 0; }); + + auto res = client.send_stream(get(server.url("/")), + [&](const https::SseEvent&) { ++events; return true; }, + stopper.source.get_token()); + const auto latency = stopper.since_stop_ms(Clock::now()); + std::cout << "[ latency ] stream: " << latency << " ms after the stop" << std::endl; + + EXPECT_EQ(res.statusCode, 200); + EXPECT_TRUE(res.cancelled); + EXPECT_FALSE(res.bodyComplete); + EXPECT_EQ(res.bodyError, "cancelled"); + EXPECT_EQ(events.load(), 1); + EXPECT_LT(latency, kPromptMs); +} + +// The handshake has no timeout of its own, so a server that accepts and says +// nothing holds the call for good unless the token ends it. The watchdog keeps +// a regression from hanging the suite: it lets go of the connection after 5 s. +TEST_F(CancelTest, AHandshakeThatNeverAnswersIsAbandoned) { + SilentListener listener; + ASSERT_FALSE(listener.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(200), [&] { return listener.received(); }); + + std::atomic done { false }; + std::atomic watchdogFired { false }; + std::thread watchdog([&] { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + if (!done.load()) { watchdogFired = true; listener.release(); } + }); + + auto res = client.send(get(listener.url()), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + watchdog.join(); + const auto latency = stopper.since_stop_ms(returned); + std::cout << "[ latency ] handshake: " << latency << " ms after the stop" << std::endl; + + EXPECT_FALSE(watchdogFired.load()) << "the handshake was not interrupted"; + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(latency, kPromptMs); + EXPECT_TRUE(becomes_true([&] { return listener.peer_closed(); })) + << "the connection was not closed"; +} + +TEST_F(CancelTest, ATokenAlreadyStoppedSendsNothing) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("hello")); + return true; + }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::HttpClient client(test_config()); + + auto res = client.send(get(server.url("/")), source.get_token()); + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + + auto streamed = client.send_stream(get(server.url("/")), + [](const https::SseEvent&) { return true; }, + source.get_token()); + EXPECT_TRUE(streamed.cancelled); + EXPECT_EQ(server.accepts(), 0); +} + +// The slices must not shorten or lengthen the timeout a token-carrying call +// already had. The token is stopped after 5 s, so a timeout that never comes +// fails the test rather than hanging the suite. +TEST_F(CancelTest, ATokenThatIsNeverStoppedLeavesTheTimeoutAlone) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config(/*readTimeoutMs=*/300)); + Stopper backstop(std::chrono::seconds(5)); + + const auto started = Clock::now(); + auto res = client.send(get(server.url("/")), backstop.source.get_token()); + const auto elapsed = ms_between(started, Clock::now()); + + EXPECT_FALSE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "No response"); + EXPECT_GT(elapsed, 250); + EXPECT_LT(elapsed, 1500); +} + +// A cancelled connection has an unknown amount of the response still on it, so +// it must not be handed to the next request. +TEST_F(CancelTest, ACancelledConnectionIsNotReused) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.wait_peer_close(); + return false; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto first = client.send(get(server.url("/")), stopper.source.get_token()); + EXPECT_TRUE(first.cancelled); + + auto second = client.send(get(server.url("/"))); + EXPECT_EQ(second.statusCode, 200) << "status text was: " << second.statusText; + EXPECT_EQ(second.body, "second"); + EXPECT_FALSE(second.cancelled); + EXPECT_EQ(server.accepts(), 2); +} + +// A pooled connection that fails before any response byte is normally retried +// on a new one. A request the caller has abandoned must not be sent again. +TEST_F(CancelTest, ACancelledRequestIsNotRetriedOnAFreshConnection) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write(tls_test::ok_response("first")); + return true; + } + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + EXPECT_EQ(client.send(get(server.url("/"))).statusCode, 200); + + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 2; }); + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + EXPECT_TRUE(res.cancelled); + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(server.requests(), 2); + EXPECT_EQ(server.accepts(), 1); +} + +namespace { + +int open_fds() { + std::error_code ec; + for (const char* dir : { "/proc/self/fd", "/dev/fd" }) { + if (!std::filesystem::exists(dir, ec)) continue; + int n = 0; + for (auto it = std::filesystem::directory_iterator(dir, ec); + !ec && it != std::filesystem::directory_iterator(); it.increment(ec)) { + ++n; + } + return n; + } + return -1; +} + +void cancel_stuck_requests(int times) { + std::atomic closed { 0 }; + tls_test::Server server([&](tls_test::Conn& conn, int) { + if (conn.wait_peer_close()) ++closed; + return false; + }); + https::HttpClient client(test_config()); + for (int i = 0; i < times; ++i) { + const int seen = server.requests(); + Stopper stopper(std::chrono::milliseconds(50), [&] { return server.requests() > seen; }); + client.send(get(server.url("/")), stopper.source.get_token()); + } + // The server's side of each connection is done before its descriptors are + // counted as gone. + for (int i = 0; i < 200 && closed.load() < times; ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + server.stop(); +} + +} // namespace + +TEST_F(CancelTest, CancelledRequestsDoNotLeakDescriptors) { + if (open_fds() < 0) GTEST_SKIP() << "no /proc/self/fd or /dev/fd"; + + cancel_stuck_requests(1); // anything the first request opens and keeps + const int before = open_fds(); + cancel_stuck_requests(20); + EXPECT_EQ(open_fds(), before); +} + +// A redirect whose body is still arriving when the stop comes has to be reported +// as it is, not followed: the next request would be one the caller abandoned. +TEST_F(CancelTest, ARedirectIsNotFollowedOnceTheRequestIsCancelled) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 302 Found\r\nLocation: /next\r\nContent-Length: 100\r\n\r\npartial"); + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 302); + EXPECT_EQ(res.bodyError, "cancelled"); + EXPECT_EQ(server.requests(), 1); +} + +// Connecting is a wait too, and nothing may be sent to an address once the stop +// has been asked for. The listener only has to exist: the kernel completes the +// connection for it, so an attempt shows as an accept. +TEST_F(CancelTest, AConnectAfterTheStopSendsNoSyn) { + tls_test::Server server([](tls_test::Conn&, int) { return false; }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::Socket sock; + sock.set_stop(source.get_token()); + + EXPECT_FALSE(sock.connect("127.0.0.1", server.port(), 4000)); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + EXPECT_EQ(server.accepts(), 0); +} + +// ── through a proxy ────────────────────────────────────────────────────────── + +namespace { + +https::HttpClientConfig proxied_config(const std::string& proxyUrl) { + auto cfg = test_config(); + cfg.proxy = proxyUrl; + return cfg; +} + +// Holds a connection for up to 5 s or until `done`, so a test whose cancellation +// fails sees the connection dropped (and an assertion fail) rather than a hang. +void hold(const std::atomic& done) { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +} + +} // namespace + +TEST_F(CancelTest, AProxyThatNeverAnswersConnectIsAbandoned) { + std::atomic sawConnect { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + sawConnect = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("http://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawConnect.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "Cancelled"); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +TEST_F(CancelTest, AnHttpsProxyThatNeverCompletesItsHandshakeIsAbandoned) { + std::atomic sawHello { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_some(); + sawHello = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("https://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawHello.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// The session to the target runs inside the one to the proxy, and the stop has +// to reach the proxy's session for the inner handshake to be interruptible. +TEST_F(CancelTest, AHandshakeInsideAnHttpsProxyTunnelIsAbandoned) { + std::atomic tunnelOpen { false }; + tls_test::Server proxy([&](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + tunnelOpen = true; + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("https://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(200), [&] { return tunnelOpen.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// ── a reused connection carries the current call's token ───────────────────── +// +// The token is set on a connection for one call and taken off when it goes back +// to the pool. Through an https:// proxy the waits happen in the session beneath +// the one the pool holds, and it was that session's token, set by the call that +// opened the tunnel, that every later call waited on. + +namespace { + +enum class Route { Direct, HttpProxy, HttpsProxy }; + +std::string route_name(const ::testing::TestParamInfo& info) { + switch (info.param) { + case Route::Direct: return "Direct"; + case Route::HttpProxy: return "HttpProxy"; + case Route::HttpsProxy: return "HttpsProxy"; + } + return "Unknown"; +} + +// The proxy in front of a target, when the route has one, and the client +// configuration that reaches the target through it. +class Path { +public: + Path(Route route, int targetPort) { + if (route == Route::HttpProxy) { + http_ = std::make_unique([targetPort](proxy_test::Peer& peer) { + if (peer.read_head().empty()) return; + peer.write("HTTP/1.1 200 Connection established\r\n\r\n"); + peer.relay_to(targetPort); + }); + } else if (route == Route::HttpsProxy) { + https_ = std::make_unique([targetPort](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.relay_to(targetPort); + return false; + }); + } + } + + [[nodiscard]] bool failed() const { + return (http_ && http_->failed()) || (https_ && https_->failed()); + } + + [[nodiscard]] https::HttpClientConfig config() const { + auto cfg = test_config(); + if (http_) cfg.proxy = "http://127.0.0.1:" + std::to_string(http_->port()); + if (https_) cfg.proxy = "https://127.0.0.1:" + std::to_string(https_->port()); + return cfg; + } + + // Connections the proxy accepted; the target's count when there is none. + [[nodiscard]] int proxy_accepts(const tls_test::Server& target) const { + if (http_) return http_->accepts(); + if (https_) return https_->accepts(); + return target.accepts(); + } + +private: + std::unique_ptr http_; + std::unique_ptr https_; +}; + +https::HttpRequest post(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::POST; + req.url = url; + req.body = "{}"; + return req; +} + +class ReuseTest : public CancelTest, public ::testing::WithParamInterface {}; + +} // namespace + +// THE README'S OWN PATTERN. A jthread requests a stop in its destructor, after +// the call it ran has returned and its connection is back in the pool. That stop +// must not reach the next call: through an https:// proxy it made every wait on +// the tunnel fail at once, and the stale-connection retry then sent a POST the +// target had already received a second time. +TEST_P(ReuseTest, AStopAfterTheCallReturnedDoesNotReachTheNextCall) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("done")); + return true; + }); + ASSERT_FALSE(target.failed()); + Path path(GetParam(), target.port()); + ASSERT_FALSE(path.failed()); + + https::HttpClient client(path.config()); + { + std::atomic returned { false }; + std::jthread worker([&](std::stop_token stop) { + auto first = client.send(post(target.url("/first")), stop); + EXPECT_EQ(first.statusCode, 200) << first.statusText; + EXPECT_FALSE(first.cancelled); + returned = true; + }); + while (!returned.load()) std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } // ~jthread: request_stop(), then join + + auto second = client.send(post(target.url("/second"))); + EXPECT_EQ(second.statusCode, 200) << second.statusText; + EXPECT_FALSE(second.cancelled); + + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(target.requests(), 2) << "the second POST reached the target more than once"; + EXPECT_EQ(target.accepts(), 1) << "the pooled connection was not reused"; + EXPECT_EQ(path.proxy_accepts(target), 1); +} + +// The other half: the second call's own token has to reach the connection it +// reuses, whatever the first call's token is doing. +TEST_P(ReuseTest, TheSecondCallsTokenReachesTheReusedConnection) { + tls_test::Server target([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write(tls_test::ok_response("first")); + return true; + } + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(target.failed()); + Path path(GetParam(), target.port()); + ASSERT_FALSE(path.failed()); + + https::HttpClient client(path.config()); + std::stop_source first; // never stopped, and alive throughout + EXPECT_EQ(client.send(get(target.url("/a")), first.get_token()).statusCode, 200); + + Stopper stopper(std::chrono::milliseconds(100), [&] { return target.requests() >= 2; }); + auto res = client.send(get(target.url("/b")), stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled) << res.statusText; + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); + EXPECT_EQ(target.accepts(), 1) << "the second call did not reuse the connection"; +} + +INSTANTIATE_TEST_SUITE_P(Routes, ReuseTest, + ::testing::Values(Route::Direct, Route::HttpProxy, Route::HttpsProxy), + route_name); + +TEST_F(CancelTest, ASocks5ProxyThatNeverAnswersTheGreetingIsAbandoned) { + std::atomic sawGreeting { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_some(); + sawGreeting = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("socks5://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawGreeting.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// ── download_to_file ───────────────────────────────────────────────────────── + +namespace { + +std::filesystem::path scratch_file(std::string_view name) { + auto path = std::filesystem::temp_directory_path() + / ("tinyhttps-cancel-" + std::string(name)); + std::error_code ec; + std::filesystem::remove(path, ec); + return path; +} + +} // namespace + +TEST_F(CancelTest, ADownloadWaitingForTheHeadersIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("head"); + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto res = client.download_to_file(server.url("/file"), file, nullptr, nullptr, + stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.error, "Cancelled"); + EXPECT_FALSE(res.ok()); + EXPECT_FALSE(std::filesystem::exists(file)) << "no file before a status line"; + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +TEST_F(CancelTest, ADownloadWaitingForTheRestOfTheBodyIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 100000\r\n\r\n"); + conn.write(std::string(1000, 'x')); + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("body"); + std::atomic written { 0 }; + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return written.load() >= 1000; }); + auto res = client.download_to_file(server.url("/file"), file, + [&](std::int64_t, std::int64_t now) { written = now; }, + nullptr, stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 200); + EXPECT_EQ(res.error, "cancelled"); + EXPECT_FALSE(res.writeFailed); + EXPECT_EQ(res.bytesWritten, 1000); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); + + std::error_code ec; + std::filesystem::remove(file, ec); +} + +TEST_F(CancelTest, ADownloadWithATokenAlreadyStoppedSendsNothing) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("never")); + return true; + }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::HttpClient client(test_config()); + auto res = client.download_to_file(server.url("/file"), scratch_file("none"), + nullptr, nullptr, source.get_token()); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.error, "Cancelled"); + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(server.accepts(), 0); +} + +// `isCancelled` and the token are two ways to say the same thing, and the result +// says it the same way for both. +TEST_F(CancelTest, IsCancelledSetsCancelledAsTheTokenDoes) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 100000\r\n\r\n"); + conn.write(std::string(1000, 'x')); + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("predicate"); + https::HttpClient client(test_config()); + auto res = client.download_to_file(server.url("/file"), file, nullptr, + [] { return true; }); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.error, "cancelled"); + EXPECT_EQ(res.statusCode, 200); + + std::error_code ec; + std::filesystem::remove(file, ec); +} diff --git a/tests/test_extra_ca.cpp b/tests/test_extra_ca.cpp new file mode 100644 index 0000000..f86875c --- /dev/null +++ b/tests/test_extra_ca.cpp @@ -0,0 +1,243 @@ +// `HttpClientConfig::extraCaFile` adds roots to the default trust store. +// +// The default store is whatever SSL_CERT_FILE names, so each test writes the +// bundles it needs to temp files and the machine's own store never enters into +// it. The servers present leaves signed by two private CAs (extra_ca_certs.hpp); +// nothing leaves 127.0.0.1. +#include +#include "extra_ca_certs.hpp" +#include "tls_test_server.hpp" + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; +namespace pki = extra_ca_test; + +namespace { + +void set_env(const char* name, const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s(name, value); +#else + ::setenv(name, value, 1); +#endif +} + +class ExtraCaTest : public ::testing::Test { +protected: + void SetUp() override { + https::Socket::platform_init(); + const char* old = std::getenv("SSL_CERT_FILE"); + hadOld_ = old != nullptr; + if (hadOld_) old_ = old; + } + + void TearDown() override { + set_env("SSL_CERT_FILE", hadOld_ ? old_.c_str() : ""); + std::error_code ec; + for (auto& path : files_) std::filesystem::remove(path, ec); + } + + // Writes `content` to a new temp file and returns its path. + std::string write_file(std::string_view content) { + auto path = std::filesystem::temp_directory_path() + / ("tinyhttps-extra-ca-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + "-" + std::to_string(files_.size()) + ".pem"); + std::ofstream(path, std::ios::binary) << content; + files_.push_back(path); + return path.string(); + } + + // Makes `content` the whole default store. + void default_store(std::string_view content) { + set_env("SSL_CERT_FILE", write_file(content).c_str()); + } + + static https::HttpClientConfig config(std::string extraCaFile = {}) { + https::HttpClientConfig cfg; // verifySsl is true by default + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + cfg.extraCaFile = std::move(extraCaFile); + return cfg; + } + + static https::HttpResponse get(const https::HttpClientConfig& cfg, const std::string& url) { + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = url; + return client.send(req); + } + +private: + std::vector files_; + std::string old_; + bool hadOld_ { false }; +}; + +bool answer_ok(tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("ok")); + return true; +} + +tls_test::Server server_of_ca1() { + return tls_test::Server(answer_ok, pki::kLeaf1Pem, pki::kLeaf1KeyPem); +} + +tls_test::Server server_of_ca2() { + return tls_test::Server(answer_ok, pki::kLeaf2Pem, pki::kLeaf2KeyPem); +} + +} // namespace + +TEST_F(ExtraCaTest, APrivateCaIsNotTrustedWithoutTheFile) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(tls_test::kCertPem); // a store that does not contain CA 1 + + auto r = get(config(), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("not correctly signed by the trusted CA"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, APrivateCaIsTrustedWithTheFile) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(tls_test::kCertPem); + + auto r = get(config(write_file(pki::kCa1Pem)), server.url("/")); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + EXPECT_EQ(r.body, "ok"); +} + +// The file adds to the default store; it does not replace it. +TEST_F(ExtraCaTest, TheDefaultStoreStaysInEffect) { + auto fromDefault = server_of_ca1(); + auto fromExtra = server_of_ca2(); + auto neither = tls_test::Server(answer_ok); // self-signed, in no store + ASSERT_FALSE(fromDefault.failed()); + ASSERT_FALSE(fromExtra.failed()); + ASSERT_FALSE(neither.failed()); + default_store(pki::kCa1Pem); + auto cfg = config(write_file(pki::kCa2Pem)); + + EXPECT_EQ(get(cfg, fromDefault.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, fromExtra.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, neither.url("/")).statusCode, 0); +} + +TEST_F(ExtraCaTest, AFileHoldingSeveralCertificatesTrustsAllOfThem) { + auto a = server_of_ca1(); + auto b = server_of_ca2(); + ASSERT_FALSE(a.failed()); + ASSERT_FALSE(b.failed()); + default_store(tls_test::kCertPem); + auto cfg = config(write_file(std::string(pki::kCa1Pem) + pki::kCa2Pem)); + + EXPECT_EQ(get(cfg, a.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, b.url("/")).statusCode, 200); +} + +// Both hops of a request through an https:// proxy use the file: the proxy's +// certificate comes from CA 2, which only the file names, and the target's from +// CA 1, which only the default store names. +TEST_F(ExtraCaTest, TheHopToAnHttpsProxyUsesTheFileToo) { + auto target = server_of_ca1(); + ASSERT_FALSE(target.failed()); + const int targetPort = target.port(); + auto proxy = tls_test::Server([targetPort](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.relay_to(targetPort); + return false; + }, pki::kLeaf2Pem, pki::kLeaf2KeyPem); + ASSERT_FALSE(proxy.failed()); + default_store(pki::kCa1Pem); + + auto cfg = config(write_file(pki::kCa2Pem)); + cfg.proxy = "https://127.0.0.1:" + std::to_string(proxy.port()); + auto r = get(cfg, target.url("/")); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + + cfg.extraCaFile.clear(); + r = get(cfg, target.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_TRUE(r.statusText.starts_with("proxy: could not open a TLS connection to 127.0.0.1:")) + << r.statusText; + EXPECT_NE(r.statusText.find("certificate verification failed"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileThatDoesNotExistIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); // the server would verify without the file + const auto missing = (std::filesystem::temp_directory_path() / "tinyhttps-no-such-ca.pem").string(); + + auto r = get(config(missing), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("cannot read extra CA file '" + missing + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileThatIsNotPemIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("this is not a certificate\n"); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileWhoseCertificateCannotBeParsedIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("-----BEGIN CERTIFICATE-----\nAAAA\n-----END CERTIFICATE-----\n"); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AnEmptyFileIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file(""); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +// The file is checked whether or not the server would be verified. +TEST_F(ExtraCaTest, AFileThatCannotBeUsedFailsEvenWithoutVerification) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("not a certificate\n"); + + auto cfg = config(path); + cfg.verifySsl = false; + auto r = get(cfg, server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +// SSL_CERT_FILE keeps its meaning when the field is unset. +TEST_F(ExtraCaTest, WithoutTheFieldNothingChanges) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + EXPECT_EQ(get(config(), server.url("/")).statusCode, 200); +} diff --git a/tests/test_handshake_timeout.cpp b/tests/test_handshake_timeout.cpp new file mode 100644 index 0000000..a825612 --- /dev/null +++ b/tests/test_handshake_timeout.cpp @@ -0,0 +1,289 @@ +// The TLS handshake is bounded by connectTimeoutMs. +// +// Each case gives the client a peer that completes the TCP connection and then +// never answers the ClientHello, with no stop token to end the wait. The peer +// holds the connection for at most 5 s and then drops it, so a handshake that +// has no deadline fails the case (by the message and by the time it took) +// instead of hanging the suite. + +// As in test_cancel.cpp, `import std` goes first with libc++ because of the +// stop token in one of the cases, and after the includes otherwise. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" +#ifndef _WIN32 +#include +#include +#endif + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif + +namespace https = mcpplibs::tinyhttps; + +namespace { + +using Clock = std::chrono::steady_clock; + +// Far below the 5 s the peers hold on for, far above the 300 ms timeout. +constexpr long long kPromptMs = 3000; + +https::HttpClientConfig test_config() { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 300; + cfg.readTimeoutMs = 30000; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::GET; + req.url = url; + return req; +} + +long long since_ms(Clock::time_point start) { + return std::chrono::duration_cast(Clock::now() - start).count(); +} + +// Holds a connection for up to 5 s or until `done`. +void hold(const std::atomic& done) { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +} + +// A TCP listener that never speaks TLS: it reads the ClientHello and says nothing. +class Silent { +public: + Silent() : server_([this](proxy_test::Peer& peer) { + peer.read_some(); + hold(done_); + }) {} + ~Silent() { done_ = true; } + + [[nodiscard]] bool failed() const { return server_.failed(); } + [[nodiscard]] std::string url() const { + return "https://127.0.0.1:" + std::to_string(server_.port()) + "/"; + } + [[nodiscard]] std::string proxy_url(const char* scheme) const { + return std::string(scheme) + "://127.0.0.1:" + std::to_string(server_.port()); + } + +private: + std::atomic done_ { false }; + proxy_test::Server server_; +}; + +class HandshakeTimeoutTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +} // namespace + +TEST_F(HandshakeTimeoutTest, SendGivesUpOnAHandshakeThatNeverAnswers) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send(get(peer.url())); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, SendStreamGivesUpOnAHandshakeThatNeverAnswers) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send_stream(get(peer.url()), [](const https::SseEvent&) { return true; }); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A stop token that is never stopped takes the waits in slices; the deadline +// has to hold there as well. +TEST_F(HandshakeTimeoutTest, TheDeadlineHoldsWithAStopTokenAttached) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + std::stop_source never; + const auto start = Clock::now(); + auto res = client.send(get(peer.url()), never.get_token()); + + EXPECT_FALSE(res.cancelled); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, TheHandshakeWithAnHttpsProxyIsBounded) { + Silent proxy; + ASSERT_FALSE(proxy.failed()); + + auto cfg = test_config(); + cfg.proxy = proxy.proxy_url("https"); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_NE(res.statusText.find("proxy: could not open a TLS connection"), std::string::npos) + << res.statusText; + EXPECT_NE(res.statusText.find("TLS handshake timed out"), std::string::npos) + << res.statusText; + EXPECT_LT(since_ms(start), kPromptMs); +} + +// The session to the target runs inside the one to the proxy, which answered +// the CONNECT and is now silent. +TEST_F(HandshakeTimeoutTest, TheHandshakeInsideAnHttpsProxyTunnelIsBounded) { + tls_test::Server proxy([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(proxy.failed()); + + // The proxy's own handshake is real and has to fit in the timeout too. + auto cfg = test_config(); + cfg.connectTimeoutMs = 1000; + cfg.proxy = "https://127.0.0.1:" + std::to_string(proxy.port()); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, TheHandshakeInsideAnHttpProxyTunnelIsBounded) { + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.1 200 Connection established\r\n\r\n"); + peer.read_some(); + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + auto cfg = test_config(); + cfg.proxy = "http://127.0.0.1:" + std::to_string(proxy.port()); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + done = true; + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A peer that sends a byte at a time keeps every read short, so only a limit on +// the handshake as a whole ends it: the record header promises 64 bytes and the +// peer sends fewer than that, one per 100 ms. +TEST_F(HandshakeTimeoutTest, ABytePerReadDoesNotHoldTheHandshakeOpen) { + std::atomic done { false }; + proxy_test::Server peerServer([&](proxy_test::Peer& peer) { + peer.read_some(); + if (!peer.write(std::string("\x16\x03\x03\x00\x40", 5))) return; + for (int i = 0; i < 50 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + if (!peer.write("\x01")) return; + } + }); + ASSERT_FALSE(peerServer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:" + std::to_string(peerServer.port()) + "/")); + done = true; + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A signal whose handler was installed with SA_RESTART restarts a blocking +// `recv`, so it never ended a handshake before the deadline existed. The wait +// that the deadline puts ahead of each `recv` is a `poll`, which a signal ends +// with EINTR whatever the flag; that has to be waited out, not taken for the +// deadline. The signals go to this thread, the one in the handshake. +#ifndef _WIN32 +namespace { +void ignore_signal(int) {} +} + +TEST_F(HandshakeTimeoutTest, ASignalDuringTheHandshakeIsNotATimeout) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + struct sigaction action {}, previous {}; + action.sa_handler = ignore_signal; + action.sa_flags = SA_RESTART; + sigemptyset(&action.sa_mask); + ASSERT_EQ(sigaction(SIGUSR1, &action, &previous), 0); + + auto cfg = test_config(); + cfg.connectTimeoutMs = 1000; + https::HttpClient client(cfg); + const pthread_t self = pthread_self(); + std::atomic done { false }; + std::thread signaller([&] { + for (int i = 0; i < 5 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + if (!done.load()) pthread_kill(self, SIGUSR1); + } + }); + const auto start = Clock::now(); + auto res = client.send(get(peer.url())); + const auto took = since_ms(start); + done = true; + signaller.join(); + sigaction(SIGUSR1, &previous, nullptr); + + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_GE(took, 900) << "the handshake ended at a signal, not at the deadline"; + EXPECT_LT(took, kPromptMs); +} +#endif + +// connectTimeoutMs is for establishing the connection. Once the handshake is +// done a read waits as long as it always did: here 1600 ms of silence after a +// 1000 ms timeout, and then the bytes arrive. The timeout leaves room for a +// real handshake on a slow machine. +TEST_F(HandshakeTimeoutTest, TheDeadlineIsLiftedOnceTheHandshakeIsDone) { + tls_test::Server server([](tls_test::Conn& conn, int) { + std::this_thread::sleep_for(std::chrono::milliseconds(1600)); + conn.write("x"); + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::TlsSocket tls; + ASSERT_TRUE(tls.connect("127.0.0.1", server.port(), 1000, /*verifySsl=*/false)) << tls.error(); + const std::string request = "GET / HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"; + ASSERT_EQ(tls.write(request.data(), static_cast(request.size())), + static_cast(request.size())); + char byte = 0; + auto r = tls.read_some(&byte, 1); + + EXPECT_EQ(r.status, https::ReadStatus::Data); + EXPECT_EQ(byte, 'x'); +} diff --git a/tests/test_pool.cpp b/tests/test_pool.cpp index 164f1ee..995b795 100644 --- a/tests/test_pool.cpp +++ b/tests/test_pool.cpp @@ -13,7 +13,16 @@ #include #include "tls_test_server.hpp" -#ifndef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifndef TINYHTTPS_WINSOCK #include #include #include @@ -56,7 +65,7 @@ class PoolTest : public ::testing::Test { // does. The read times out with 50 bytes still owed, and those 50 bytes then // arrive on a socket nobody is reading. // -// ⚠️ THE LATE SECOND HALF IS WHAT MAKES THIS A TEST. An earlier form of this +// THE LATE SECOND HALF IS WHAT MAKES THIS A TEST. An earlier form of this // server stalled and never sent the rest; the connection was then silent rather // than dirty, and the stale-connection retry rescued the second request whether // or not the pool guard worked. Verified by mutation: with the guard's @@ -181,7 +190,9 @@ TEST_F(PoolTest, AContentLengthPastThirtyTwoBitsIsNotSilentlyZero) { // the width of the field and nothing else. // (`ABodyLargerThanTheConfiguredLimitIsRefused` covers the cap.) auto cfg = test_config(); - cfg.maxResponseBodyBytes = std::numeric_limits::max(); + // Parenthesised: , which the test servers include on Windows, + // defines a `max` macro. + cfg.maxResponseBodyBytes = (std::numeric_limits::max)(); https::HttpClient client(cfg); auto first = client.send(get(server.url("/one"))); @@ -615,6 +626,94 @@ TEST_F(PoolTest, ADownloadDelimitedByTheCloseReportsSuccess) { std::filesystem::remove_all(dir, ec); } +// ── a destination that refuses the bytes ───────────────────────────────────── + +// `/dev/full` accepts an open and answers every write with ENOSPC, which is what +// a full disk does at the moment it fills — without needing one. +// +// Before 0.3.2 the transfer "succeeded": `ofs.write` was never checked, the +// count came from the network, and `bytesWritten` said the whole body when the +// file held none of it. The caller then blamed the source (a checksum mismatch) +// for what was the local disk. Verified by mutation: with the stream check +// removed, `ok()` is true and `bytesWritten` is the body's length. +#ifdef __linux__ +TEST_F(PoolTest, ADownloadIntoAFullDeviceFailsAsALocalWriteAndCountsNothing) { + if (!std::filesystem::exists("/dev/full")) GTEST_SKIP() << "no /dev/full"; + + const std::string payload(64 * 1024, 'F'); + tls_test::Server server([payload](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response(payload)); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto result = client.download_to_file(server.url("/big"), "/dev/full"); + + EXPECT_FALSE(result.ok()) << "a file that took nothing was reported as downloaded"; + EXPECT_TRUE(result.writeFailed) + << "the caller must be able to tell the disk from the source; error: " + << result.error; + EXPECT_EQ(result.statusCode, 200); + EXPECT_EQ(result.bytesWritten, 0); + EXPECT_GT(result.bytesReceived, 0); + EXPECT_LT(result.bytesReceived, static_cast(payload.size())) + << "the transfer read on after the first refused write"; + EXPECT_EQ(result.error.rfind("write /dev/full: ", 0), 0u) << result.error; + EXPECT_NE(result.error.find( + std::make_error_code(std::errc::no_space_on_device).message()), + std::string::npos) + << result.error; + + // The rest of the body was left on the socket, so the connection is not + // handed to the next request. The next request also shows the client is + // not left in a state where a good destination fails too. + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_full"; + std::filesystem::create_directories(dir); + auto dest = dir / "ok.bin"; + auto good = client.download_to_file(server.url("/big"), dest); + EXPECT_TRUE(good.ok()) << "error: " << good.error; + EXPECT_FALSE(good.writeFailed); + EXPECT_EQ(good.bytesWritten, static_cast(payload.size())); + EXPECT_EQ(good.bytesReceived, good.bytesWritten); + EXPECT_EQ(server.accepts(), 2) + << "the connection with an unread body was reused"; + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} +#endif + +// A destination that cannot be opened is the same kind of failure: nothing was +// wrong with what the server sent. +TEST_F(PoolTest, ADestinationThatCannotBeOpenedIsALocalFailure) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("payload")); + return true; + }); + ASSERT_FALSE(server.failed()); + + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_noopen"; + std::filesystem::create_directories(dir); + { + // A regular file where a directory is needed: no parent can be made. + std::ofstream blocker(dir / "blocker", std::ios::binary); + blocker << "not a directory"; + } + + https::HttpClient client(test_config()); + auto result = client.download_to_file(server.url("/x"), + dir / "blocker" / "out.bin"); + + EXPECT_FALSE(result.ok()); + EXPECT_TRUE(result.writeFailed) << "error: " << result.error; + EXPECT_EQ(result.error.rfind("Cannot open file: ", 0), 0u) << result.error; + EXPECT_EQ(result.bytesWritten, 0); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} + // ── interim responses ──────────────────────────────────────────────────────── // A 1xx is not the answer. RFC 9112 §2.1 requires a client to read past one or @@ -719,7 +818,7 @@ TEST_F(PoolTest, AnEndlessTrailerSectionIsRefused) { // ── issue #16 · SIGPIPE ────────────────────────────────────────────────────── -#ifndef _WIN32 +#ifndef TINYHTTPS_WINSOCK // THE TEST HAS TO FORK, AND THE REASON IS ITSELF EVIDENCE. // // `mbedtls_net_bind` calls `net_prepare`, which does `signal(SIGPIPE, SIG_IGN)` @@ -801,4 +900,4 @@ TEST(SigPipe, WritingToADepartedPeerReturnsInsteadOfKillingTheProcess) { EXPECT_EQ(WEXITSTATUS(status), 0) << "child exit code 3 = could not connect, 4 = the write never failed"; } -#endif // !_WIN32 +#endif // !TINYHTTPS_WINSOCK diff --git a/tests/test_proxy.cpp b/tests/test_proxy.cpp new file mode 100644 index 0000000..ca1d92f --- /dev/null +++ b/tests/test_proxy.cpp @@ -0,0 +1,758 @@ +// What the client does when it is told to go through a proxy. +// +// Each test stands up a proxy that behaves as the test scripts it and the TLS +// server from tls_test_server.hpp as the target, and asserts both what the +// caller got and what the proxy was sent. The second is the half a mock-free +// test cannot see: whether the CONNECT line named the right target, and what +// the Proxy-Authorization header said. +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +class ProxyTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +https::HttpClientConfig proxied(const std::string& proxyUrl) { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + cfg.proxy = proxyUrl; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.url = url; + return req; +} + +tls_test::Server hello_server() { + return tls_test::Server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("hello")); + return true; + }); +} + +// The value of a header in a request head, or "" when absent. +std::string header_of(const std::string& head, std::string_view name) { + std::string lower, key = std::string(name) + ":"; + for (char c : head) lower += static_cast(std::tolower(static_cast(c))); + for (char& c : key) c = static_cast(std::tolower(static_cast(c))); + auto pos = lower.find(key); + if (pos == std::string::npos) return {}; + pos += key.size(); + while (pos < head.size() && head[pos] == ' ') ++pos; + return head.substr(pos, head.find("\r\n", pos) - pos); +} + +constexpr std::string_view kEstablished = "HTTP/1.1 200 Connection established\r\n\r\n"; + +// A CONNECT proxy. With `expectAuth` set it answers 407 unless the request +// carries exactly that Proxy-Authorization value. Either way it records the +// request head, so the test can read back what the client sent. +proxy_test::Server::Handler connect_handler(int targetPort, std::string expectAuth = {}) { + return [=](proxy_test::Peer& peer) { + auto head = peer.read_head(); + peer.note(head); + if (head.empty()) return; + if (!expectAuth.empty() && header_of(head, "Proxy-Authorization") != expectAuth) { + peer.write("HTTP/1.1 407 Proxy Authentication Required\r\n" + "Proxy-Authenticate: Basic realm=\"test\"\r\n" + "Content-Length: 0\r\n\r\n"); + return; + } + peer.write(kEstablished); + peer.relay_to(targetPort); + }; +} + +std::string proxy_url(const proxy_test::Server& proxy, std::string_view userinfo = {}) { + std::string url = "http://"; + if (!userinfo.empty()) { url += userinfo; url += "@"; } + return url + "127.0.0.1:" + std::to_string(proxy.port()); +} + +} // namespace + +// ── parse_proxy_url ────────────────────────────────────────────────────────── + +TEST(ProxyUrl, BareHostAndPortIsAnHttpProxy) { + auto p = https::parse_proxy_url("proxy.example:3128"); + EXPECT_EQ(p.scheme, "http"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); + EXPECT_FALSE(p.hasCredentials); +} + +TEST(ProxyUrl, ThePortDefaultsAsItAlwaysHas) { + EXPECT_EQ(https::parse_proxy_url("http://proxy.example").port, 8080); +} + +TEST(ProxyUrl, APathQueryOrFragmentAfterTheAuthorityIsIgnored) { + auto p = https::parse_proxy_url("http://proxy.example:3128/some/path?x=1#f"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, UserAndPasswordAreSplitAndPercentDecoded) { + auto p = https::parse_proxy_url("http://al%40ice:p%3Ass%20w@proxy.example:3128"); + EXPECT_TRUE(p.hasCredentials); + EXPECT_EQ(p.user, "al@ice"); + EXPECT_EQ(p.password, "p:ss w"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, OnlyTheFirstColonSplitsUserFromPassword) { + auto p = https::parse_proxy_url("http://u:a:b@proxy.example:1"); + EXPECT_EQ(p.user, "u"); + EXPECT_EQ(p.password, "a:b"); +} + +TEST(ProxyUrl, AnUnencodedAtSignInThePasswordStaysInThePassword) { + auto p = https::parse_proxy_url("http://u:p@ss@proxy.example:1"); + EXPECT_EQ(p.password, "p@ss"); + EXPECT_EQ(p.host, "proxy.example"); +} + +TEST(ProxyUrl, AUserWithNoPasswordStillHasCredentials) { + auto p = https::parse_proxy_url("http://token@proxy.example:1"); + EXPECT_TRUE(p.hasCredentials); + EXPECT_EQ(p.user, "token"); + EXPECT_EQ(p.password, ""); +} + +TEST(ProxyUrl, ABadPercentEscapeIsLeftAsWritten) { + auto p = https::parse_proxy_url("http://u:50%@proxy.example:1"); + EXPECT_EQ(p.password, "50%"); +} + +TEST(ProxyUrl, AnIpv6LiteralKeepsItsColons) { + auto p = https::parse_proxy_url("http://[::1]:3128"); + EXPECT_EQ(p.host, "::1"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, TheSchemeIsLowerCased) { + EXPECT_EQ(https::parse_proxy_url("HTTP://proxy.example:1").scheme, "http"); +} + +TEST(ProxyUrl, APortOutOfRangeIsZeroRatherThanTruncated) { + EXPECT_EQ(https::parse_proxy_url("http://proxy.example:99999").port, 0); + EXPECT_EQ(https::parse_proxy_url("http://proxy.example:80x").port, 0); +} + +// ── CONNECT ────────────────────────────────────────────────────────────────── + +TEST_F(ProxyTest, ARequestGoesThroughAProxyThatAsksForNothing) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto notes = proxy.notes(); + ASSERT_EQ(notes.size(), 1u); + EXPECT_TRUE(notes[0].starts_with("CONNECT 127.0.0.1:" + std::to_string(target.port()) + " HTTP/1.1\r\n")); + EXPECT_EQ(header_of(notes[0], "Proxy-Authorization"), ""); +} + +TEST_F(ProxyTest, CredentialsInTheUrlBecomeABasicProxyAuthorization) { + auto target = hello_server(); + // base64("alice:s3cret") + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy, "alice:s3cret"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); +} + +TEST_F(ProxyTest, CredentialsArePercentDecodedBeforeTheyAreEncoded) { + auto target = hello_server(); + // base64("al@ice:p:ss w") + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxAaWNlOnA6c3Mgdw==")); + + https::HttpClient client(proxied(proxy_url(proxy, "al%40ice:p%3Ass%20w"))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); +} + +TEST_F(ProxyTest, WrongCredentialsAreReportedAsThatAndNotAsAFailedConnection) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy, "alice:wrong"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); + EXPECT_NE(resp.statusText.find("407"), std::string::npos); + EXPECT_EQ(target.accepts(), 0); // the tunnel was never opened +} + +TEST_F(ProxyTest, AProxyThatWantsCredentialsWeDidNotGiveSaysSo) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("requires credentials"), std::string::npos); +} + +TEST_F(ProxyTest, AnyOtherRefusalCarriesTheProxysStatus) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\n\r\n"); + }); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("refused CONNECT: 403 Forbidden"), std::string::npos); +} + +TEST_F(ProxyTest, AProxyThatHangsUpWithoutAnsweringIsAnError) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { peer.read_head(); }); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("no HTTP response"), std::string::npos); +} + +TEST_F(ProxyTest, AnyTwoHundredSeriesAnswerOpensTheTunnel) { + auto target = hello_server(); + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.0 204 No Content\r\nX-Note: a header or two\r\n\r\n"); + peer.relay_to(target.port()); + }); + + https::HttpClient client(proxied(proxy_url(proxy))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); +} + +TEST_F(ProxyTest, TwoRequestsShareOneTunnel) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + + https::HttpClient client(proxied(proxy_url(proxy))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} + +TEST_F(ProxyTest, ASchemeThisLibraryDoesNotSpeakIsNotQuietlyTreatedAsHttp) { + https::HttpClient client(proxied("socks4://127.0.0.1:9")); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("unsupported scheme 'socks4'"), std::string::npos); +} + +TEST_F(ProxyTest, TheOriginalProxyConnectStillReturnsATunnel) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + + auto tunnel = https::proxy_connect("127.0.0.1", proxy.port(), "127.0.0.1", + target.port(), 4000); + EXPECT_TRUE(tunnel.is_valid()); +} + +// ── an https:// proxy ──────────────────────────────────────────────────────── +// +// The proxy here is the TLS test server: the client opens TLS to it, sends +// CONNECT inside that, and then opens a second TLS session to the target inside +// the tunnel. Two sessions on one descriptor is what these tests exist to +// exercise, and the cases that matter are the ones where the two disagree about +// what is readable: a body larger than a record, and a second request that has +// to find the tunnel already open. + +namespace { + +// A recorder for what the TLS proxy saw. The handler runs on the server's +// threads; the test reads it after the response is in. +struct Seen { + std::mutex mutex; + std::vector heads; + void add(std::string head) { + std::lock_guard lock(mutex); + heads.push_back(std::move(head)); + } + std::vector get() { + std::lock_guard lock(mutex); + return heads; + } +}; + +tls_test::Server tls_connect_proxy(Seen& seen, int targetPort, std::string expectAuth = {}, + std::chrono::milliseconds gather = std::chrono::milliseconds::zero()) { + return tls_test::Server([&seen, targetPort, expectAuth, gather](tls_test::Conn& conn, int) { + seen.add(conn.request()); + if (!expectAuth.empty() + && header_of(conn.request(), "Proxy-Authorization") != expectAuth) { + conn.write("HTTP/1.1 407 Proxy Authentication Required\r\n" + "Proxy-Authenticate: Basic realm=\"test\"\r\n" + "Content-Length: 0\r\n\r\n"); + return false; + } + conn.write(kEstablished); + conn.relay_to(targetPort, gather); + return false; + }); +} + +} // namespace + +namespace { + +// Makes one certificate the only CA bundle the client reads, for the lifetime of +// the object, and restores what was there before. +class TrustOnly { +public: + explicit TrustOnly(std::string_view pem) + : path_(std::filesystem::temp_directory_path() + / ("tinyhttps-proxy-trust-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + ".pem")) { + if (const char* old = std::getenv("SSL_CERT_FILE")) { hadOld_ = true; old_ = old; } + std::ofstream(path_, std::ios::binary) << pem; + set(path_.string().c_str()); + } + ~TrustOnly() { + set(hadOld_ ? old_.c_str() : ""); + std::error_code ec; + std::filesystem::remove(path_, ec); + } + TrustOnly(const TrustOnly&) = delete; + TrustOnly& operator=(const TrustOnly&) = delete; + +private: + static void set(const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s("SSL_CERT_FILE", value); +#else + ::setenv("SSL_CERT_FILE", value, 1); +#endif + } + std::filesystem::path path_; + std::string old_; + bool hadOld_ { false }; +}; + +} // namespace + +TEST(ProxyUrl, AnHttpsProxyDefaultsToPort443) { + auto p = https::parse_proxy_url("https://proxy.example"); + EXPECT_EQ(p.scheme, "https"); + EXPECT_EQ(p.port, 443); + EXPECT_EQ(https::parse_proxy_url("https://proxy.example:8443").port, 8443); +} + +TEST_F(ProxyTest, ARequestGoesThroughAnHttpsProxy) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto heads = seen.get(); + ASSERT_EQ(heads.size(), 1u); + EXPECT_TRUE(heads[0].starts_with("CONNECT 127.0.0.1:" + std::to_string(target.port()) + " HTTP/1.1\r\n")); +} + +// The proxy hop is verified like the target hop, and its refusal names the hop +// and the reason. Only the wrong-name certificate is trusted, so the proxy's own +// certificate has no issuer the client knows; nothing, the CONNECT included, +// reaches a peer that did not verify. +TEST_F(ProxyTest, AnHttpsProxyWhoseCertificateDoesNotVerifyIsRefusedWithTheReason) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + ASSERT_FALSE(proxy.failed()); + TrustOnly bundle(tls_test::kWrongNameCertPem); + + auto cfg = proxied("https://127.0.0.1:" + std::to_string(proxy.port())); + cfg.verifySsl = true; + https::HttpClient client(cfg); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_TRUE(resp.statusText.starts_with("proxy: could not open a TLS connection to 127.0.0.1:")) + << resp.statusText; + EXPECT_NE(resp.statusText.find("certificate verification failed"), std::string::npos) + << resp.statusText; + EXPECT_TRUE(seen.get().empty()); +} + +TEST_F(ProxyTest, CredentialsGoToAnHttpsProxyInsideItsTlsSession) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port(), "Basic YWxpY2U6czNjcmV0"); + + https::HttpClient ok(proxied("https://alice:s3cret@127.0.0.1:" + std::to_string(proxy.port()))); + EXPECT_EQ(ok.send(get(target.url("/"))).statusCode, 200); + + https::HttpClient bad(proxied("https://alice:nope@127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = bad.send(get(target.url("/"))); + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); +} + +TEST_F(ProxyTest, ABodyLargerThanARecordCrossesBothSessions) { + std::string big(300 * 1024, 'x'); + for (std::size_t i = 0; i < big.size(); i += 97) big[i] = static_cast('a' + i % 26); + tls_test::Server target([&](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response(big)); + return true; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_TRUE(resp.bodyComplete); + EXPECT_TRUE(resp.body == big); +} + +TEST_F(ProxyTest, TwoRequestsShareOneTunnelThroughAnHttpsProxy) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} + +// The target writes the head and the body as two records and the proxy hands +// them on as one, so both arrive in a single record of the proxy session. The +// client reads the first through the proxy session and the descriptor is then +// empty while the second sits decrypted in that session: waiting on the +// descriptor, rather than on the session, would time out on a response that has +// already arrived. +TEST_F(ProxyTest, ABodyAlreadyDecryptedInTheProxySessionIsNotWaitedFor) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n"); + conn.write("hello"); + return true; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port(), {}, std::chrono::milliseconds(150)); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); +} + +TEST_F(ProxyTest, AStreamingRequestWorksThroughAnHttpsProxy) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n" + "Connection: close\r\n\r\ndata: one\n\ndata: two\n\n"); + return false; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + std::vector events; + auto resp = client.send_stream(get(target.url("/")), [&](const https::SseEvent& ev) { + events.push_back(ev.data); + return true; + }); + + EXPECT_EQ(resp.statusCode, 200); + ASSERT_EQ(events.size(), 2u); + EXPECT_EQ(events[0], "one"); + EXPECT_EQ(events[1], "two"); +} + +// The handshake has no timeout of its own, so the proxy ends it the way a real +// HTTP proxy does when it is sent something that is not HTTP: it reads the first +// bytes and hangs up. +TEST_F(ProxyTest, AnHttpsUrlToAPlainProxyFailsAtTheHandshake) { + auto target = hello_server(); + proxy_test::Server proxy([](proxy_test::Peer& peer) { peer.read_some(); }); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("TLS connection"), std::string::npos); +} + +TEST_F(ProxyTest, AnHttpUrlToATlsProxyIsAnErrorAndNotAHang) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("http://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_FALSE(resp.statusText.empty()); +} + +// ── SOCKS5 ─────────────────────────────────────────────────────────────────── + +namespace { + +// What a SOCKS5 proxy needs to know about how to behave. +struct Socks5Script { + int targetPort { 0 }; + // Empty: accept the no-authentication method. Otherwise "user:password", + // and the proxy insists on the username/password method. + std::string credentials; + // The reply code to the CONNECT request; 0 is success. + unsigned char replyCode { 0 }; + // The address type of the bound address in a success reply: 1, 3 or 4. + unsigned char boundType { 1 }; +}; + +std::string hex_of(std::string_view bytes) { + std::string out; + for (unsigned char c : bytes) out += std::format("{:02x}", c); + return out; +} + +// Notes, in order: "methods:", "auth::" when it was used, +// and "request::
:". +proxy_test::Server::Handler socks5_handler(Socks5Script script) { + return [script](proxy_test::Peer& peer) { + auto greet = peer.read_exact(2); + if (greet.size() != 2 || greet[0] != 5) return; + auto methods = peer.read_exact(static_cast(greet[1])); + peer.note("methods:" + hex_of(methods)); + + if (script.credentials.empty()) { + peer.write(std::string_view("\x05\x00", 2)); + } else { + if (methods.find('\x02') == std::string::npos) { + peer.write(std::string_view("\x05\xff", 2)); + return; + } + peer.write(std::string_view("\x05\x02", 2)); + auto ver = peer.read_exact(2); // 0x01, ULEN + if (ver.size() != 2) return; + auto user = peer.read_exact(static_cast(ver[1])); + auto plen = peer.read_exact(1); + if (plen.size() != 1) return; + auto pass = peer.read_exact(static_cast(plen[0])); + peer.note("auth:" + user + ":" + pass); + if (user + ":" + pass != script.credentials) { + peer.write(std::string_view("\x01\x01", 2)); + return; + } + peer.write(std::string_view("\x01\x00", 2)); + } + + auto head = peer.read_exact(4); // VER CMD RSV ATYP + if (head.size() != 4) return; + std::string addr; + std::size_t len = head[3] == 1 ? 4 : head[3] == 4 ? 16 : 0; + if (head[3] == 3) { + auto n = peer.read_exact(1); + if (n.size() != 1) return; + addr = peer.read_exact(static_cast(n[0])); + } else { + addr = peer.read_exact(len); + } + auto port = peer.read_exact(2); + if (port.size() != 2) return; + peer.note(std::format("request:{}:{}:{}", static_cast(head[3]), + head[3] == 3 ? addr : hex_of(addr), + (static_cast(port[0]) << 8) + | static_cast(port[1]))); + + std::string reply("\x05\x00\x00", 3); + reply[1] = static_cast(script.replyCode); + reply += static_cast(script.boundType); + if (script.boundType == 1) reply.append(4, '\x7f'); + else if (script.boundType == 4) reply.append(16, '\x01'); + else reply += std::string("\x0c") + "bind.example"; + reply.append("\x04\xd2", 2); + peer.write(reply); + if (script.replyCode == 0) peer.relay_to(script.targetPort); + }; +} + +std::string socks_url(std::string_view scheme, const proxy_test::Server& proxy, + std::string_view userinfo = {}) { + std::string url(scheme); + url += "://"; + if (!userinfo.empty()) { url += userinfo; url += "@"; } + return url + "127.0.0.1:" + std::to_string(proxy.port()); +} + +} // namespace + +TEST(ProxyUrl, SocksSchemesAreKeptAndDefaultToPort1080) { + auto a = https::parse_proxy_url("socks5://proxy.example"); + EXPECT_EQ(a.scheme, "socks5"); + EXPECT_EQ(a.port, 1080); + auto b = https::parse_proxy_url("SOCKS5H://u:p@proxy.example:9050"); + EXPECT_EQ(b.scheme, "socks5h"); + EXPECT_EQ(b.port, 9050); + EXPECT_EQ(b.user, "u"); +} + +TEST_F(ProxyTest, Socks5ConnectsToAnAddressLiteral) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port() })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto notes = proxy.notes(); + ASSERT_EQ(notes.size(), 2u); + EXPECT_EQ(notes[0], "methods:00"); // no credentials, so only "none" is offered + EXPECT_EQ(notes[1], "request:1:7f000001:" + std::to_string(target.port())); +} + +TEST_F(ProxyTest, Socks5hSendsAHostNameAndSocks5SendsAnAddress) { + auto target = hello_server(); + const std::string url = "https://localhost:" + std::to_string(target.port()) + "/"; + + proxy_test::Server remote(socks5_handler({ .targetPort = target.port() })); + https::HttpClient viaH(proxied(socks_url("socks5h", remote))); + EXPECT_EQ(viaH.send(get(url)).statusCode, 200); + ASSERT_EQ(remote.notes().size(), 2u); + EXPECT_EQ(remote.notes()[1], "request:3:localhost:" + std::to_string(target.port())); + + proxy_test::Server local(socks5_handler({ .targetPort = target.port() })); + https::HttpClient viaLocal(proxied(socks_url("socks5", local))); + EXPECT_EQ(viaLocal.send(get(url)).statusCode, 200); + ASSERT_EQ(local.notes().size(), 2u); + EXPECT_NE(local.notes()[1].substr(0, 10), "request:3:"); // an address, not the name +} + +TEST_F(ProxyTest, Socks5UsernameAndPasswordAreOfferedAndSent) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "al@ice:p:ss w" })); + + https::HttpClient client(proxied(socks_url("socks5h", proxy, "al%40ice:p%3Ass%20w"))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); + + auto notes = proxy.notes(); + ASSERT_GE(notes.size(), 2u); + EXPECT_EQ(notes[0], "methods:0002"); + EXPECT_EQ(notes[1], "auth:al@ice:p:ss w"); +} + +TEST_F(ProxyTest, Socks5WrongCredentialsAreReportedAsThat) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "alice:s3cret" })); + + https::HttpClient client(proxied(socks_url("socks5", proxy, "alice:wrong"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); + EXPECT_EQ(target.accepts(), 0); +} + +TEST_F(ProxyTest, Socks5ProxyThatWantsCredentialsWeDidNotGiveSaysSo) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "alice:s3cret" })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("requires credentials"), std::string::npos); +} + +TEST_F(ProxyTest, Socks5RefusalCarriesTheReplyCode) { + proxy_test::Server proxy(socks5_handler({ .replyCode = 5 })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("refused CONNECT: connection refused (SOCKS5 reply 5)"), + std::string::npos); +} + +TEST_F(ProxyTest, Socks5AnyBoundAddressTypeIsReadOffTheStream) { + // The reply to a successful CONNECT carries an address of one of three + // shapes. Misreading its length leaves bytes ahead of the TLS handshake + // and the request fails; reading it right is what a 200 here shows. + for (unsigned char type : {1, 3, 4}) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), .boundType = type })); + https::HttpClient client(proxied(socks_url("socks5", proxy))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); + } +} + +TEST_F(ProxyTest, Socks5ASpeakerOfAnotherProtocolIsNotMistakenForOne) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { + peer.read_exact(3); + peer.write("HTTP/1.1 400 Bad Request\r\n\r\n"); + }); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("not a SOCKS5 proxy"), std::string::npos); +} + +TEST_F(ProxyTest, Socks5hRefusesAHostNameThatDoesNotFitTheRequest) { + proxy_test::Server proxy(socks5_handler({})); + https::HttpClient client(proxied(socks_url("socks5h", proxy))); + auto resp = client.send(get("https://" + std::string(300, 'a') + ".example/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("does not fit a SOCKS5 request"), std::string::npos); +} + +TEST_F(ProxyTest, TwoRequestsShareOneSocks5Tunnel) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port() })); + + https::HttpClient client(proxied(socks_url("socks5h", proxy))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} diff --git a/tests/test_tls_verify.cpp b/tests/test_tls_verify.cpp new file mode 100644 index 0000000..ccaae34 --- /dev/null +++ b/tests/test_tls_verify.cpp @@ -0,0 +1,143 @@ +// A client with `verifySsl = true` refuses a server certificate it cannot trust. +// +// Each test serves one certificate from the in-process TLS server and trusts it +// (or not) by pointing SSL_CERT_FILE at a bundle written for the test, so the +// machine's own CA store never enters into it and nothing leaves 127.0.0.1. +// The certificates in tls_test_server.hpp are self-signed, which lets a trusted +// one be its own bundle and lets the expired and wrong-name ones differ from it +// in that one respect only. +#include +#include "tls_test_server.hpp" + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +void set_env(const char* name, const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s(name, value); +#else + ::setenv(name, value, 1); +#endif +} + +class TlsVerifyTest : public ::testing::Test { +protected: + void SetUp() override { + https::Socket::platform_init(); + const char* old = std::getenv("SSL_CERT_FILE"); + hadOld_ = old != nullptr; + if (hadOld_) old_ = old; + bundle_ = std::filesystem::temp_directory_path() + / ("tinyhttps-verify-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + ".pem"); + } + + void TearDown() override { + set_env("SSL_CERT_FILE", hadOld_ ? old_.c_str() : ""); + std::error_code ec; + std::filesystem::remove(bundle_, ec); + } + + // Makes `content` the only CA bundle the client will read. + void trust(std::string_view content) { + std::ofstream(bundle_, std::ios::binary) << content; + set_env("SSL_CERT_FILE", bundle_.string().c_str()); + } + + https::HttpResponse get(const tls_test::Server& server) { + https::HttpClientConfig cfg; // verifySsl is true by default + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = server.url("/"); + return client.send(req); + } + +private: + std::filesystem::path bundle_; + std::string old_; + bool hadOld_ { false }; +}; + +bool answer_ok(tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("ok")); + return true; +} + +} // namespace + +TEST_F(TlsVerifyTest, ACertificateIssuedByATrustedCaIsAccepted) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust(tls_test::kCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + EXPECT_EQ(r.body, "ok"); +} + +TEST_F(TlsVerifyTest, ACertificateNoTrustedCaIssuedIsRefused) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); // a bundle that does not contain the server's + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("certificate verification failed"), std::string::npos) + << r.statusText; + EXPECT_NE(r.statusText.find("not correctly signed by the trusted CA"), std::string::npos) + << r.statusText; +} + +TEST_F(TlsVerifyTest, AnExpiredCertificateIsRefused) { + tls_test::Server server(answer_ok, tls_test::kExpiredCertPem, tls_test::kExpiredKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kExpiredCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("expired"), std::string::npos) << r.statusText; +} + +TEST_F(TlsVerifyTest, ACertificateForAnotherHostnameIsRefused) { + tls_test::Server server(answer_ok, tls_test::kWrongNameCertPem, tls_test::kWrongNameKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("does not match"), std::string::npos) << r.statusText; +} + +TEST_F(TlsVerifyTest, ABundleThatCannotBeParsedIsReportedAsSuch) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust("this is not a certificate\n"); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("cannot parse the CA certificate bundle"), std::string::npos) + << r.statusText; +} + +TEST_F(TlsVerifyTest, VerificationCanStillBeTurnedOff) { + tls_test::Server server(answer_ok, tls_test::kExpiredCertPem, tls_test::kExpiredKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); + + https::HttpClientConfig cfg; + cfg.verifySsl = false; + cfg.connectTimeoutMs = 4000; + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = server.url("/"); + EXPECT_EQ(client.send(req).statusCode, 200); +} diff --git a/tests/tls_test_server.hpp b/tests/tls_test_server.hpp index 3bd8e81..8017262 100644 --- a/tests/tls_test_server.hpp +++ b/tests/tls_test_server.hpp @@ -29,7 +29,16 @@ #include #include -#ifdef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK #include #include #else @@ -39,6 +48,7 @@ #include #include +#include #include #include #include @@ -100,6 +110,51 @@ VQKaSq3Gef7/KLR3YARmm5A= -----END PRIVATE KEY----- )PEM"; +// Two more self-signed certificates, each wrong in one way, for the tests that +// check the client refuses them (test_tls_verify.cpp). Same terms as above: +// public keys, no secrets. +// +// kExpiredCertPem is CN/SAN localhost and 127.0.0.1, valid 2020-01-01 to 2020-01-02. +inline constexpr const char* kExpiredCertPem = R"PEM(-----BEGIN CERTIFICATE----- +MIIBeDCCAR6gAwIBAgIUQR4qNY0KJQYcYEO7585+Fy5Q/U8wCgYIKoZIzj0EAwIw +FDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIwMDEwMTAwMDAwMFoXDTIwMDEwMjAw +MDAwMFowFDESMBAGA1UEAwwJbG9jYWxob3N0MFkwEwYHKoZIzj0CAQYIKoZIzj0D +AQcDQgAEEb2pNZ4XE7aYNPdWF5+/h+6Ug+KoqKMnRQDgr4OfRvZPIH9NROFnEXnK +stqlISOxthgAmIkR1W3pLSj2BooVT6NOMEwwGgYDVR0RBBMwEYIJbG9jYWxob3N0 +hwR/AAABMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNSI5hk66wZ6dXb/N5+v +6H3CXr5+MAoGCCqGSM49BAMCA0gAMEUCIQD48yY+f7UE0URCJghr3Q+58od8BkxG +zvn+vOvdRjKNZwIgdfrfRWxHrQS/mfyZyzQwTWHHa/TXLD5Z+vlFNjSuAIk= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kExpiredKeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgP8T1y7ZRNYINaXf1 ++ciH6e0/avbCUNliO/brgmagw/OhRANCAAQRvak1nhcTtpg091YXn7+H7pSD4qio +oydFAOCvg59G9k8gf01E4WcRecqy2qUhI7G2GACYiRHVbektKPYGihVP +-----END PRIVATE KEY----- +)PEM"; + +// kWrongNameCertPem is valid for a century but names only wrong.example. +inline constexpr const char* kWrongNameCertPem = R"PEM(-----BEGIN CERTIFICATE----- +MIIBgTCCASagAwIBAgIUUaQJz12pEvdBKiHf8zQ3agib668wCgYIKoZIzj0EAwIw +GDEWMBQGA1UEAwwNd3JvbmcuZXhhbXBsZTAgFw0yNjEwMDEwMzU5NTZaGA8yMTI2 +MDkwNzAzNTk1NlowGDEWMBQGA1UEAwwNd3JvbmcuZXhhbXBsZTBZMBMGByqGSM49 +AgEGCCqGSM49AwEHA0IABAIy/jbI/tnfJmp4lDT9WtuvNNu3CapWaAAsXrXPBYVg +R5MyHwQhLGq9Uy+ZC9dpJI6R1+9b5NUCh5RfaOE4zYOjTDBKMBgGA1UdEQQRMA+C +DXdyb25nLmV4YW1wbGUwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUFN3zbGMq +Zu37qSve4CdZ6qXXSsAwCgYIKoZIzj0EAwIDSQAwRgIhAO/UR5PxHU2spE47HbQf +4qPyonqIlmGB1kp129etm0dYAiEAsjTvx61zqcTrES7R+zQOYykJ67PVTqqGkxIP +uv23Y9g= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kWrongNameKeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgdB0xMbmMhgDdvjON +ESmHq2CHTvE0SbWMP8P4Kt/GWpqhRANCAAQCMv42yP7Z3yZqeJQ0/VrbrzTbtwmq +VmgALF61zwWFYEeTMh8EISxqvVMvmQvXaSSOkdfvW+TVAoeUX2jhOM2D +-----END PRIVATE KEY----- +)PEM"; + class Server; // One accepted connection, after its handshake. Handed to the test's handler. @@ -128,6 +183,10 @@ class Conn { // is stopped or after `limit`. void park(std::chrono::milliseconds limit = std::chrono::seconds(20)); + // Waits for the client to end the connection, discarding anything it sends. + // False when `limit` passes or the server is stopped first. + bool wait_peer_close(std::chrono::milliseconds limit = std::chrono::seconds(20)); + // Ends the connection at the TLS layer, then at the transport. void shutdown() { if (closed_) return; @@ -153,6 +212,18 @@ class Conn { // answer differently per path. const std::string& request() const { return request_; } + // Splices what this connection decrypts to 127.0.0.1:`port`, and what comes + // back from there into it, until either side ends. This is what makes the + // server a proxy for the proxy tests: after it answers a CONNECT, the + // client's next bytes are the start of a second TLS session, and this + // carries them to the real target untouched. + // + // With `gather` set, what comes back from the target is held until the + // target has been quiet for that long and then sent as one write, so + // several records the target wrote in a row reach the client inside a single + // record of this session. + void relay_to(int port, std::chrono::milliseconds gather = std::chrono::milliseconds::zero()); + private: friend class Server; @@ -225,7 +296,10 @@ class Server { // Return true to serve another request on the same connection. using Handler = std::function; - explicit Server(Handler handler) : handler_(std::move(handler)) { + // Serves `certPem`/`keyPem`, the default pair unless a test needs another. + explicit Server(Handler handler, const char* certPem = kCertPem, + const char* keyPem = kKeyPem) + : handler_(std::move(handler)), certPem_(certPem), keyPem_(keyPem) { mbedtls_net_init(&listener_); // Port 0: the OS picks a free one, which is what lets tests run // concurrently and on a machine where anything might already be bound. @@ -326,11 +400,11 @@ class Server { if (mbedtls_ctr_drbg_seed(&drbg, mbedtls_entropy_func, &entropy, nullptr, 0) != 0 || mbedtls_x509_crt_parse(&cert, - reinterpret_cast(kCertPem), - std::strlen(kCertPem) + 1) != 0 + reinterpret_cast(certPem_), + std::strlen(certPem_) + 1) != 0 || mbedtls_pk_parse_key(&key, - reinterpret_cast(kKeyPem), - std::strlen(kKeyPem) + 1, nullptr, 0, + reinterpret_cast(keyPem_), + std::strlen(keyPem_) + 1, nullptr, 0, mbedtls_ctr_drbg_random, &drbg) != 0 || mbedtls_ssl_config_defaults(&conf, MBEDTLS_SSL_IS_SERVER, MBEDTLS_SSL_TRANSPORT_STREAM, MBEDTLS_SSL_PRESET_DEFAULT) != 0) { @@ -370,6 +444,8 @@ class Server { } Handler handler_; + const char* certPem_; + const char* keyPem_; mbedtls_net_context listener_ {}; int port_ { 0 }; bool failed_ { false }; @@ -391,6 +467,60 @@ inline bool Conn::wait_readable() { return false; } +inline void Conn::relay_to(int port, std::chrono::milliseconds gather) { + mbedtls_net_context target {}; + mbedtls_net_init(&target); + if (mbedtls_net_connect(&target, "127.0.0.1", std::to_string(port).c_str(), + MBEDTLS_NET_PROTO_TCP) != 0) { + return; + } + unsigned char buf[4096]; + std::string back; + while (!server_.stopping()) { + if (mbedtls_ssl_get_bytes_avail(&ssl_) > 0 + || mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, 10) > 0) { + int n = mbedtls_ssl_read(&ssl_, buf, sizeof buf); + if (n == MBEDTLS_ERR_SSL_WANT_READ || n == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (n <= 0) break; + int sent = 0; + while (sent < n) { + int w = mbedtls_net_send(&target, buf + sent, static_cast(n - sent)); + if (w <= 0) { n = 0; break; } + sent += w; + } + if (n == 0) break; + } + if (mbedtls_net_poll(&target, MBEDTLS_NET_POLL_READ, 10) > 0) { + int n = mbedtls_net_recv(&target, buf, sizeof buf); + if (n <= 0) break; + back.assign(reinterpret_cast(buf), static_cast(n)); + while (gather.count() > 0 && back.size() < 8192 + && mbedtls_net_poll(&target, MBEDTLS_NET_POLL_READ, + static_cast(gather.count())) > 0) { + n = mbedtls_net_recv(&target, buf, sizeof buf); + if (n <= 0) break; + back.append(reinterpret_cast(buf), static_cast(n)); + } + if (!write(back)) break; + } + } + mbedtls_net_free(&target); +} + +inline bool Conn::wait_peer_close(std::chrono::milliseconds limit) { + const auto deadline = std::chrono::steady_clock::now() + limit; + unsigned char buf[256]; + while (!server_.stopping() && std::chrono::steady_clock::now() < deadline) { + const int ready = mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) return true; + if (ready == 0) continue; + int ret = mbedtls_ssl_read(&ssl_, buf, sizeof buf); + if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (ret <= 0) return true; + } + return false; +} + inline void Conn::park(std::chrono::milliseconds limit) { const auto deadline = std::chrono::steady_clock::now() + limit; while (!server_.stopping() && std::chrono::steady_clock::now() < deadline) { diff --git a/xmake.lua b/xmake.lua index 36a9d5a..fd21acc 100644 --- a/xmake.lua +++ b/xmake.lua @@ -7,7 +7,7 @@ add_requires("gtest 1.15.2", { optional = true }) target("tinyhttps") set_kind("static") - set_version("0.3.0") + set_version("0.3.3") add_files("src/*.cppm", { public = true, install = true }) add_packages("mbedtls", { public = true }) set_policy("build.c++.modules", true)