From c34f1ecbbc512226691ff866bab443a27b62f32e Mon Sep 17 00:00:00 2001 From: SPeak Date: Thu, 17 Sep 2026 17:51:50 +0800 Subject: [PATCH 01/15] 0.3.1 --- the socket interface follows the C library (#18) * 0.3.1 --- the socket interface follows the C library src/socket.cppm and src/platform.cppm selected Windows Sockets with _WIN32, which is a statement about the operating system. Which socket interface exists is decided by the C library: above openkal on Windows the C library is musl, winsock2.h is not in the graph, and the build failed (mcpp-index#435). TINYHTTPS_WINSOCK is now derived once, from _WIN32 and TINYHTTPS_POSIX_SOCKETS, and mcpp.toml defines the latter under cfg(c-abi = "musl"). A Windows build with the platform's C runtime is unchanged. platform::is_windows is renamed uses_winsock, which is what its one caller asks. Comments carry no emoji. * CHANGELOG: 0.3.1 * README: the install line names this release --- .agents/docs/exec-plan-0.3.0.md | 12 +-- .agents/docs/fix-plan-issue-15-16.md | 108 +++++++++++++-------------- CHANGELOG.md | 11 ++- README.md | 2 +- mcpp.toml | 9 ++- src/http.cppm | 4 +- src/platform.cppm | 31 +++++--- src/socket.cppm | 29 ++++--- tests/test_pool.cpp | 17 ++++- tests/tls_test_server.hpp | 11 ++- 10 files changed, 143 insertions(+), 91 deletions(-) diff --git a/.agents/docs/exec-plan-0.3.0.md b/.agents/docs/exec-plan-0.3.0.md index ba9ad4b..79250fb 100644 --- a/.agents/docs/exec-plan-0.3.0.md +++ b/.agents/docs/exec-plan-0.3.0.md @@ -20,8 +20,8 @@ 版本取 **0.3.0** 而不是 0.2.11:`HttpResponse` / `HttpClientConfig` 加了字段(非破坏),严格解析会把过去静默接受的畸形响应变成显式错误(**行为变更**)。语义化版本下,行为变更进 minor。 **无感升级的边界**(release note 必须点名): -- ✅ 现有代码**不需要改一行**就能编译:只加字段,不改签名,`ok()` 语义不变。 -- ⚠️ 过去被静默接受的畸形响应现在会报错 —— 这不是新 bug,是过去在静默损坏数据。 +- 现有代码**不需要改一行**就能编译:只加字段,不改签名,`ok()` 语义不变。 +- 过去被静默接受的畸形响应现在会报错 —— 这不是新 bug,是过去在静默损坏数据。 --- @@ -45,7 +45,7 @@ ``` T0 分支 + 计划 ✔ │ - ├─ T1 P0 SIGPIPE（socket.cppm） ★ 必须最先 + ├─ T1 P0 SIGPIPE（socket.cppm） 必须最先 │ │ P0-1 send() + MSG_NOSIGNAL │ │ P0-2 socket() 后 setsockopt(SO_NOSIGPIPE) │ │ 理由：T5 让 pool_.erase() 在更多路径上执行， @@ -66,7 +66,7 @@ T0 分支 + 计划 ✔ │ └ 吸收 P2-E2（分配上限，chunk 分片读） │ │ │ ├─ T4 P3-A PooledConnection（默认丢弃）← 吸收 P1-A/B/C 全部十项 - │ │ │ ⚠ 递归重定向：递归前必须 keep() 或 drop()，不能让析构跨过递归 + │ │ │ 递归重定向：递归前必须 keep() 或 drop()，不能让析构跨过递归 │ │ │ │ │ ├─ T5 P2-E1 bounded drain（重定向/非 2xx/文件打不开时省一次握手） │ │ ├─ T6 P2-F1/F2 bodyComplete + bodyError（statusText 不再被覆盖） @@ -75,7 +75,7 @@ T0 分支 + 计划 ✔ │ └─ T8 纯单测：parse_status_line / parse_chunk_size_line / parse_content_length │ ├─ T9 集成测试:进程内 mbedtls TLS listener（T4 后） - │ ⚠ tinyhttps 只支持 https，本地明文 listener 用不了 → 必须自带 TLS 服务端 + │ tinyhttps 只支持 https，本地明文 listener 用不了 → 必须自带 TLS 服务端 │ T1..T11 用例见 §4 │ ├─ T10 openkal 体系(与 T1..T9 并行调研,结论落地在 T1/platform.cppm) @@ -136,7 +136,7 @@ T0 分支 + 计划 ✔ | # | 用例 | 断言 | 覆盖 | |---|---|---|---| | T1 | 服务端发一半 CL body 后挂住 | 请求 2 `statusCode == 200` | P1-A1 | -| T2 | 同上,**并断言服务端 accept 次数 == 2** | 🔴 唯一能抓住「看起来对了」的断言 | P1-A1 | +| T2 | 同上,**并断言服务端 accept 次数 == 2** | 唯一能抓住「看起来对了」的断言 | P1-A1 | | T3 | chunked 版本同上 | 同 T1+T2 | P1-A3 | | T4 | 服务端发一半后关连接 | 不崩;进程未被信号杀死 | #16 | | T5 | `SIGPIPE` 默认处置下跑 T4(子进程) | `WIFSIGNALED == false`,退出码 ≠ 141 | P0 | diff --git a/.agents/docs/fix-plan-issue-15-16.md b/.agents/docs/fix-plan-issue-15-16.md index 0343ca6..7f5a45b 100644 --- a/.agents/docs/fix-plan-issue-15-16.md +++ b/.agents/docs/fix-plan-issue-15-16.md @@ -7,9 +7,9 @@ | 关联 issue | [#15](https://github.com/mcpplibs/tinyhttps/issues/15) 连接池污染 · [#16](https://github.com/mcpplibs/tinyhttps/issues/16) SIGPIPE | | 文档日期 | 2026-09-06 | -> ⚠️ **动手前先 `git pull`。** 当前工作区在 `626c9d0`(0.2.8),落后 origin/master 两个提交(`965d805` #9、`2cec1c1` #14)。本文所有行号对应 **0.2.10**;本地 `src/http.cppm` 只有 1032 行,对不上。 +> **动手前先 `git pull`。** 当前工作区在 `626c9d0`(0.2.8),落后 origin/master 两个提交(`965d805` #9、`2cec1c1` #14)。本文所有行号对应 **0.2.10**;本地 `src/http.cppm` 只有 1032 行,对不上。 -**标注约定**:✅ = 已逐行核验源码 · 🔬 = 已独立复现 · ⚠️ = 推断,未实测 +**标注约定**:= 已逐行核验源码 · = 已独立复现 · = 推断,未实测 --- @@ -34,20 +34,20 @@ ## 1. 问题全景 -### 1.1 三个入口的加固覆盖矩阵 ✅ +### 1.1 三个入口的加固覆盖矩阵 库里有三份几乎平行的响应读取实现:`send_impl`、`send_stream`、`download_to_file_impl`。历次修复各自只落到其中一两份: | 加固措施 | 引入自 | `send_impl` | `send_stream` | `download_to_file_impl` | |---|---|---|---|---| -| `parse_content_length`(严格) | #14 | ✅ `:516` | ✅ `:783` | ❌ `:1042-1048` 手写摘数字 | -| `contentLength` 用 `int64_t` | — | ❌ `:488` **`int`** | ✅ `:756` | ✅ `:1024` | -| `parse_chunk_size_line`(严格) | #9 | ❌ `:541` 仍是 `parse_hex` | ✅ `:831` | ✅ `:1134` | -| `read_complete_line`(区分超时/EOF) | #9 | ❌ | ❌ | ✅ `:1118/1145/1177` | -| 块后 CRLF 校验 | #9 | ❌ `:556` 丢弃返回值 | ❌ `:850` 丢弃返回值 | ✅ `:1177-1185` | -| 状态行 `HTTP/` 前缀校验 | — | ❌ | ❌ | ❌ | -| 早退时清理连接池 | — | ❌ 2 处 | ❌ 3 处 | ❌ **5 处** | -| 失败时能告诉调用方 | — | ❌ 无字段 | ❌ 无字段 | ✅ `result.error` | +| `parse_content_length`(严格) | #14 | `:516` | `:783` | `:1042-1048` 手写摘数字 | +| `contentLength` 用 `int64_t` | — | `:488` **`int`** | `:756` | `:1024` | +| `parse_chunk_size_line`(严格) | #9 | `:541` 仍是 `parse_hex` | `:831` | `:1134` | +| `read_complete_line`(区分超时/EOF) | #9 | no | | `:1118/1145/1177` | +| 块后 CRLF 校验 | #9 | `:556` 丢弃返回值 | `:850` 丢弃返回值 | `:1177-1185` | +| 状态行 `HTTP/` 前缀校验 | — | no | | no | +| 早退时清理连接池 | — | 2 处 | 3 处 | **5 处** | +| 失败时能告诉调用方 | — | 无字段 | 无字段 | `result.error` | **这张表本身就是最强的论据**:每一次修复都是在一份拷贝上打补丁,另外两份继续带病。`#14` 引入 P1-A1/A2 两处新回归,正是这个模式的最新一次复发。**不做 P3,下一次还会有。** @@ -66,7 +66,7 @@ R2/R3 是同一个毛病的两个实例:**用一个哨兵值(空串 / 0)表示 ## 2. P0 —— SIGPIPE(issue #16) -### 2.1 判定 🔬 +### 2.1 判定 **真 bug,最高优先级。** 一个库在自己管理的 fd 上调 `send()`,却让宿主进程被 `SIGPIPE` 杀死(exit 141),是库的责任。 @@ -137,7 +137,7 @@ R2/R3 是同一个毛病的两个实例:**用一个哨兵值(空串 / 0)表示 **平台矩阵**:Linux/Android 走 `MSG_NOSIGNAL`;macOS/BSD 走 `SO_NOSIGPIPE`;Windows 两个宏都不存在 → 自动走 `#else`,且本来就没有 `SIGPIPE`,无需额外分支。`socket.cppm:8-9` 已经在非 Windows 下 include 了 ``,两个宏都在里面,**不需要新增 include**。 -### 2.3 为什么这个修复是自包含的 ✅ +### 2.3 为什么这个修复是自包含的 失败会沿着**已经存在的**路径传播,零新增分支: @@ -154,16 +154,16 @@ Socket::write → -1 (EPIPE) ## 3. P1 —— 连接池不变量(issue #15 + 4 处新发现) -### 3.1 判定 ✅ +### 3.1 判定 **真 bug,不是使用方法错误。** | 报告者做的 | 是否合法 | |---|---| -| 单个 `HttpClient` 跑多个请求 | ✅ 这正是连接池的用途 | -| `keepAlive` 用默认值(true) | ✅ 库的默认值 | -| `readTimeoutMs = 1000` | ✅ 激进但合法;**超时是正常事件,不是 API 误用** | -| 触发条件:服务端发一半 / 空闲后关连接 | ✅ **服务端的常规行为** | +| 单个 `HttpClient` 跑多个请求 | 这正是连接池的用途 | +| `keepAlive` 用默认值(true) | 库的默认值 | +| `readTimeoutMs = 1000` | 激进但合法;**超时是正常事件,不是 API 误用** | +| 触发条件:服务端发一半 / 空闲后关连接 | **服务端的常规行为** | 规避手段只有「关掉 keep-alive」或「每次换 client」—— 都是放弃这个特性本身。**用户无法在正确使用 API 的前提下绕开 → 定义上就是库的 bug。** @@ -171,7 +171,7 @@ Socket::write → -1 (EPIPE) `connectionClose` 列 = 走到清理点时该标志的值。 -#### A. `send_stream` —— 🔴 **0.2.10 新引入的回归** +#### A. `send_stream` —— **0.2.10 新引入的回归** | ID | 位置 | 触发 | 现状 | 修复 | |---|---|---|---|---| @@ -179,7 +179,7 @@ Socket::write → -1 (EPIPE) | **A2** | `:884-885` | `read() <= 0` → `break` | `false` → 同上 | 设 `connectionClose = true` | | **A3** | `:846-848` | `read_exact` 失败 → `break` | `false` → 同上 | 设 `connectionClose = true` | -**回归定位** ✅:`git log -S "A DECLARED LENGTH IS READ AND THE READER THEN STOPS"` → `2cec1c1` (#14)。0.2.3 的对应分支是: +**回归定位** :`git log -S "A DECLARED LENGTH IS READ AND THE READER THEN STOPS"` → `2cec1c1` (#14)。0.2.3 的对应分支是: ```cpp } else { @@ -209,13 +209,13 @@ B2 细节:`parse_hex`(`:233-243`)遇到不认识的字符是 `break` 返回**已 |---|---|---|---|---| | **C1** | `:1166-1171` | chunk 数据 `read_exact` 失败 → **直接 return** | 跳过 `:1223` 尾声 | issue #15 | | **C2** | `:1195-1200` | CL body `read_exact` 失败 → **直接 return** | 同上 | issue #15 | -| **C3** | `:1060-1075` | **重定向** | 🆕 **正常路径** | 本次发现 | -| **C4** | `:1077-1081` | **非 2xx 返回** | 🆕 **正常路径** | 本次发现 | -| **C5** | `:1092-1096` | 输出文件打不开 | 🆕 | 本次发现 | +| **C3** | `:1060-1075` | **重定向** | **正常路径** | 本次发现 | +| **C4** | `:1077-1081` | **非 2xx 返回** | **正常路径** | 本次发现 | +| **C5** | `:1092-1096` | 输出文件打不开 | | 本次发现 | C1/C2 的对照:同一个函数里相邻的 `:1150-1153` 和 `:1180-1184` **写对了**(`sock->close(); pool_.erase(poolKey);`),这本身就证明是漏写而非取舍。 -##### 🆕 C3 是本次分析最值得注意的一条 +##### C3 是本次分析最值得注意的一条 ```cpp 1062: // Drain any body to keep connection clean @@ -239,7 +239,7 @@ C1/C2 的对照:同一个函数里相邻的 `:1150-1153` 和 `:1180-1184` **写 C4 同理:非 2xx 时 error body 未读就返回,socket 回池,同一 client 的下个请求中招。 C5 同理:文件打不开时 body 未读。 -#### D. 🆕 整数截断 +#### D. 整数截断 | ID | 位置 | 问题 | |---|---|---| @@ -280,7 +280,7 @@ C5 同理:文件打不开时 body 未读。 + if (ret <= 0) { connectionClose = true; break; } ``` -> ✅ 已核验:`:909-912` 的清理本来就 key 在 `connectionClose || stopped` 上;正常终止路径(`remaining > 0` 循环条件自然结束、`:841` 终止块 break)**不经过**这三个出口。**成功路径零变化。** +> 已核验:`:909-912` 的清理本来就 key 在 `connectionClose || stopped` 上;正常终止路径(`remaining > 0` 循环条件自然结束、`:841` 终止块 break)**不经过**这三个出口。**成功路径零变化。** #### P1-B `src/http.cppm` —— `send_impl` 两处 @@ -380,7 +380,7 @@ C3 / C4 / C5 —— 最小改法是**无条件丢弃连接**(见 P2-E1 有更省 } ``` -> 📌 **review 决策点**:C3 用「无条件丢连接」换正确性,代价是每次重定向多一次 TLS 握手。若在意,P2-E1 提供一个有上限的 drain 助手,可把这三处改回「drain 成功就留、失败就丢」。**建议先合无条件版本(正确性优先),drain 作为后续优化。** +> **review 决策点**:C3 用「无条件丢连接」换正确性,代价是每次重定向多一次 TLS 握手。若在意,P2-E1 提供一个有上限的 drain 助手,可把这三处改回「drain 成功就留、失败就丢」。**建议先合无条件版本(正确性优先),drain 作为后续优化。** #### P1-D `src/http.cppm:488` —— 类型对齐 @@ -399,7 +399,7 @@ C3 / C4 / C5 —— 最小改法是**无条件丢弃连接**(见 P2-E1 有更省 > P2 的价值:**即使 P1 将来又出漏洞,残留字节也不会被伪装成一个合法响应。** #15 里最危险的两档(静默数据损坏)都会退化成可见错误。 -### P2-C1 状态行必须以 `HTTP/` 开头 ✅ 🆕 +### P2-C1 状态行必须以 `HTTP/` 开头 三个入口(`:456-484`、`:725`、`:1010-1020`)**都不校验前缀**,状态码提取还是「挑数字、静默跳过非数字」: @@ -437,7 +437,7 @@ static std::optional parse_status_line(std::string_view line) { 三个入口统一改用它,失败一律 `pool_.erase` + `Invalid status line`。**导出它,理由和 `parse_chunk_size_line` 一样:这是 framing 里可以脱离服务端单测的那一半。** -### P2-C2 header 循环区分「超时」与「真空行」✅ +### P2-C2 header 循环区分「超时」与「真空行」 `:491-495` / `:758-762` / `:1030-1032` 三处都是: @@ -452,11 +452,11 @@ if (headerLine.empty()) break; // "End of headers" 修复:三处都换成 `read_complete_line`(`:193-213`,返回 `expected`),`unexpected` → 明确报错并丢连接。**根因 R2 的正解,`#9` 已经写好了工具,只是没铺开。** -### P2-C3 块后 CRLF 校验 ✅ +### P2-C3 块后 CRLF 校验 `send_impl:556` 和 `send_stream:850` 都是 `read_line(...)` 丢弃返回值;`download_to_file_impl:1177-1185` 是校验的(`"Missing CRLF after chunk data"`)。块长度算错时前两者会直接错位到下一个块头而不报错。照 download 的写法补齐。 -### P2-C4 `download_to_file_impl` 的 Content-Length 解析 ✅ +### P2-C4 `download_to_file_impl` 的 Content-Length 解析 `:1042-1048` 仍是手写摘数字循环: @@ -482,14 +482,14 @@ static bool drain_body(TlsSocket& sock, bool chunked, std::int64_t contentLength 有了它,C3/C4 可以写成 `if (!drain_body(...)) { sock->close(); pool_.erase(poolKey); }`。 -### P2-E2 分配上限(拒绝服务端诱导的巨额分配)🆕 ⚠️ +### P2-E2 分配上限(拒绝服务端诱导的巨额分配) - `send_impl:561` `response.body.resize(contentLength)` —— `contentLength` 最大到 `INT64_MAX`,`resize` 会抛 `std::length_error` / `std::bad_alloc`。**库没有文档说 `send()` 会抛异常**,调用方大概率没接。 - `send_impl:549` / `send_stream:845` `std::string chunkData(chunkSize, '\0')` —— 服务端一个 `7fffffff` 块头就能让客户端分配 2 GiB。 建议:加一个 `HttpClientConfig::maxResponseBodyBytes`(默认比如 64 MiB),超过就当作 framing 错误拒绝;chunk 数据改成按固定缓冲区分片读,不按块大小一次性分配。 -### P2-D1 `TlsSocket::read` 区分 EOF 与 would-block(根因 R3)✅ +### P2-D1 `TlsSocket::read` 区分 EOF 与 would-block(根因 R3) `tls.cppm:120-134`:`PEER_CLOSE_NOTIFY`、`ret == 0`、`WANT_READ`、`WANT_WRITE` **全部返回 0**。调用方(`read_exact:157`、`read_line:178`、`send_stream:885`)因此分不清「对端关了」和「暂时没数据」,只好用「等一下再试一次」的土办法遮盖。 @@ -501,9 +501,9 @@ enum class ReadStatus { Data, WouldBlock, Eof, Error }; 或保留 `int` 但用 `-2` 表示 would-block。 -> **与 P1 的关系**:P1-A2 把 `read() <= 0` 当成「流结束」丢连接。若那个 0 其实是瞬时 would-block,我们只是白丢一条连接 —— **正确性不受影响,只损失一点复用率**。所以 P1 不依赖 P2-D1,但修了 D1 之后 P1-A2 才精确。⚠️ 实践中 socket 在握手后被设回阻塞模式(`socket.cppm`),`WANT_*` 极少出现,影响应该很小,**未实测**。 +> **与 P1 的关系**:P1-A2 把 `read() <= 0` 当成「流结束」丢连接。若那个 0 其实是瞬时 would-block,我们只是白丢一条连接 —— **正确性不受影响,只损失一点复用率**。所以 P1 不依赖 P2-D1,但修了 D1 之后 P1-A2 才精确。实践中 socket 在握手后被设回阻塞模式(`socket.cppm`),`WANT_*` 极少出现,影响应该很小,**未实测**。 -### P2-D2 `write_all` 的 WANT_WRITE 处理 ✅ ⚠️ +### P2-D2 `write_all` 的 WANT_WRITE 处理 `http.cppm:216-230`: @@ -516,9 +516,9 @@ if (ret == 0) { } ``` -`TlsSocket::write`(`tls.cppm:140-142`)在 `WANT_READ/WANT_WRITE` 时返回 0。所以 TLS 背压下会「忙重试一次然后放弃」→ 用户看到莫名其妙的 `"Write failed"`。应该改成 `wait_writable(timeoutMs)` 后重试,并给总时长设上限。⚠️ **未实测,按代码推断。** +`TlsSocket::write`(`tls.cppm:140-142`)在 `WANT_READ/WANT_WRITE` 时返回 0。所以 TLS 背压下会「忙重试一次然后放弃」→ 用户看到莫名其妙的 `"Write failed"`。应该改成 `wait_writable(timeoutMs)` 后重试,并给总时长设上限。**未实测,按代码推断。** -### P2-F1 🆕 调用方无法得知 body 被截断了(API 缺口)✅ +### P2-F1 调用方无法得知 body 被截断了(API 缺口) **这是 P1 修完之后剩下的最大问题。** @@ -526,7 +526,7 @@ P1 只保证了「坏连接不会毒害下一个请求」,但当前请求返回 - `send_impl` CL 短读(`:562-565`)→ 返回真实的 `statusCode=200` + 半截 body,**没有任何截断标记** - `send_impl` / `send_stream` chunked 中断 → 同上 -- `DownloadToFileResult` **有** `error` 字段 ✅,`HttpResponse` **没有** ❌ +- `DownloadToFileResult` **有** `error` 字段 ,`HttpResponse` **没有** 也就是说:**一个被截断的 200 响应,和一个完整的 200 响应,调用方分不出来。** @@ -554,9 +554,9 @@ export struct HttpResponse { 在 P1 设 `connectionClose = true` 的每一处,同时设 `response.bodyComplete = false`。 -> 📌 **review 决策点**:(a) 只加 `bodyComplete` 字段、`ok()` 不变(建议,非破坏性);(b) 让 `ok()` 也要求 `bodyComplete`(更安全,但会让现有代码行为变化)。 +> **review 决策点**:(a) 只加 `bodyComplete` 字段、`ok()` 不变(建议,非破坏性);(b) 让 `ok()` 也要求 `bodyComplete`(更安全,但会让现有代码行为变化)。 -### P2-F2 🆕 `send_stream` 出错时会覆盖真实的 statusText ✅ +### P2-F2 `send_stream` 出错时会覆盖真实的 statusText `:833` `response.statusText = "Invalid chunk size: " + sizeLine;` —— 把服务端真实的 `"OK"` 覆盖掉了。有了 P2-F1 的 `bodyComplete` 之后,应该改成保留 `statusText`,把原因放进新字段(或一个 `bodyError` 字符串)。 @@ -566,7 +566,7 @@ export struct HttpResponse { > P1 是逐条堵漏。**但 §1.1 的矩阵说明:这个 bug 类会随着每次新增代码复发。** `#14` 就是最新一次。P3 的目标是让它写不出来。 -### P3-A `PooledConn` —— 把默认行为从「留」翻成「丢」 ★ 最高性价比 +### P3-A `PooledConn` —— 把默认行为从「留」翻成「丢」 最高性价比 当前所有 10 个 bug 的形状**完全一样**:一条早退路径**什么都没做**,而「什么都没做」的默认结果是**把脏连接留在池里**。 @@ -601,7 +601,7 @@ private: 用法:每个入口开头建一个,只在 body **正常读完**的那唯一一处调 `keep()`。10 个 bug 一次全消,而且以后新增任何早退路径**天然安全**。 -> ⚠️ **两个必须注意的陷阱(review 重点)**: +> **两个必须注意的陷阱(review 重点)**: > > 1. **递归重定向**。`send_impl:612` `return send_impl(redirectReq, redirectCount + 1);` 发生在 guard 析构**之前**。若重定向到同一 host,内层调用会往池里放一条新连接,外层 guard 析构时会把它**误删**。 > → 递归前必须先 `guard.keep()` 或 `guard.release()`。`download_to_file_impl:1073` 同理。 @@ -630,7 +630,7 @@ static BodyEnd read_body(TlsSocket&, const ResponseHead&, int timeoutMs, ### P3-C 带缓冲的读 -`read_line` 现在是逐字节 `sock.read(&c, 1)`。除了慢(⚠️ 影响有限:`TlsSocket::wait_readable` 会先查 `mbedtls_ssl_get_bytes_avail`,所以并不是每字节一次 `poll()`,主要成本是每字节一次函数调用 —— **未做基准测试,不要当成性能修复来卖**),更重要的是: +`read_line` 现在是逐字节 `sock.read(&c, 1)`。除了慢(影响有限:`TlsSocket::wait_readable` 会先查 `mbedtls_ssl_get_bytes_avail`,所以并不是每字节一次 `poll()`,主要成本是每字节一次函数调用 —— **未做基准测试,不要当成性能修复来卖**),更重要的是: **一个显式的读缓冲,才能让「这条连接干净吗」这个问题真正可回答**(= 缓冲区空 + body 已按 framing 读完)。现在这个问题只能靠人肉追踪 10 条路径。这是 P3-C 的主要理由,性能是副产品。 @@ -655,20 +655,20 @@ P0+P1 只是把「进程被杀」降级成「一个莫名其妙的 `No response` 丢弃连接 → 新建连接 → 重发一次(仅一次) ``` -> 📌 **review 决策点 —— 要不要限制到幂等方法?** +> **review 决策点 —— 要不要限制到幂等方法?** > > - Go 的策略保守:只重试幂等方法(GET/HEAD/OPTIONS/TRACE)或带 `Idempotency-Key` 的请求。 > - curl 的策略宽松:只要连接是复用的就重试,不看方法。 > > **我倾向 curl 的做法**,理由:(1) tinyhttps 的主力场景是 LLM / API 客户端,POST 是绝对多数,限制到幂等方法等于这个修复基本没用;(2) 重试窗口极窄 —— 只在**一个响应字节都没收到**时才重试,此时服务端已经发过 FIN,应用层根本没看到这个请求;(3) 加一个 `HttpClientConfig::retryOnStaleConnection`(默认 true)让保守用户能关掉。 > -> ⚠️ 残留风险:服务端「收到并处理了请求 → 还没回响应就崩了/关了」这种情况下会重复执行。窄,但非零。**这一条请你拍板。** +> 残留风险:服务端「收到并处理了请求 → 还没回响应就崩了/关了」这种情况下会重复执行。窄,但非零。**这一条请你拍板。** --- ## 6. 测试方案 -### 6.1 现状 ✅ +### 6.1 现状 `tests/` 下只有 `test_download.cpp` 和 `test_resolver.cpp`,**零 keep-alive / 连接复用覆盖** —— 这就是 `#14` 的回归能全绿合进来的直接原因。 @@ -679,7 +679,7 @@ P0+P1 只是把「进程被杀」降级成「一个莫名其妙的 `No response` | # | 用例 | 断言 | 覆盖 | |---|---|---|---| | T1 | 服务端发一半 CL body 后挂住 | 请求 2 `statusCode == 200`(不是 999/0/Invalid) | A1 | -| T2 | **同上,并断言服务端 TCP 连接数 == 2** | 🔴 **唯一能抓住「50 个 B」那种「看起来对了」的 case 的断言** | A1 | +| T2 | **同上,并断言服务端 TCP 连接数 == 2** | **唯一能抓住「50 个 B」那种「看起来对了」的 case 的断言** | A1 | | T3 | chunked 版本同上 | 同 T1 + T2 | A3 | | T4 | 服务端发一半后关连接 | 请求 2 不崩 + 断言**进程未被信号杀死** | #16 | | T5 | **`SIGPIPE` 保持默认处置**跑 T4(子进程) | `WIFSIGNALED == false`,退出码 != 141 | P0 | @@ -759,13 +759,13 @@ gcc -o sigpipe_probe sigpipe_probe.c && ./sigpipe_probe | P1-B2 | P1 | `http.cppm:541` | `parse_hex` → `parse_chunk_size_line`(并删除 `parse_hex`) | | P1-C1 | P1 | `http.cppm:1166-1171` | download chunk 短读 → `close + erase` | | P1-C2 | P1 | `http.cppm:1195-1200` | download CL 短读 → `close + erase` | -| P1-C3 | P1 | `http.cppm:1060-1075` | 🆕 **重定向未 drain body** | -| P1-C4 | P1 | `http.cppm:1077-1081` | 🆕 非 2xx 未 drain body | -| P1-C5 | P1 | `http.cppm:1092-1096` | 🆕 文件打开失败未 drain body | -| P1-D1 | P1 | `http.cppm:488` | 🆕 `int contentLength` → `int64_t`(整数截断) | -| P2-C1 | P2 | 三处状态行 | 🆕 校验 `HTTP/` 前缀 + 严格状态码 | +| P1-C3 | P1 | `http.cppm:1060-1075` | **重定向未 drain body** | +| P1-C4 | P1 | `http.cppm:1077-1081` | 非 2xx 未 drain body | +| P1-C5 | P1 | `http.cppm:1092-1096` | 文件打开失败未 drain body | +| P1-D1 | P1 | `http.cppm:488` | `int contentLength` → `int64_t`(整数截断) | +| P2-C1 | P2 | 三处状态行 | 校验 `HTTP/` 前缀 + 严格状态码 | | P2-C2 | P2 | 三处 header 循环 | 用 `read_complete_line` 区分超时/空行 | | P2-C3 | P2 | `:556` `:850` | 校验块后 CRLF | | P2-C4 | P2 | `:1042-1048` | download 换 `parse_content_length` | -| P2-E2 | P2 | `:561` `:549` `:845` | 🆕 分配上限,防 2 GiB 分配 / 未文档化的异常 | -| P2-F1 | P2 | `HttpResponse` | 🆕 加 `bodyComplete` —— 调用方现在无法得知 body 被截断 | +| P2-E2 | P2 | `:561` `:549` `:845` | 分配上限,防 2 GiB 分配 / 未文档化的异常 | +| P2-F1 | P2 | `HttpResponse` | 加 `bodyComplete` —— 调用方现在无法得知 body 被截断 | diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ff76b8..897790b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## 0.3.1 + +The socket interface is selected by the C library rather than by the operating +system ([mcpp-index#435](https://github.com/mcpplibs/mcpp-index/issues/435)). +Above openkal on Windows the C library is musl and `winsock2.h` is not in the +dependency graph; the POSIX interface is used there, selected in `mcpp.toml` by +`cfg(c-abi = "musl")`. A Windows build with the platform's C runtime is +unchanged. `platform::is_windows` is renamed `platform::uses_winsock`. + ## 0.3.0 Closes [#15](https://github.com/mcpplibs/tinyhttps/issues/15) (a socket returned @@ -14,7 +23,7 @@ truncation at all, plus a `Content-Length` past 32 bits — and two more (the streaming reader's `Content-Length` exits) were regressions 0.2.10 had introduced. -### ⚠️ Read this first +### Read this first **If you are on 0.2.10 and use `send_stream`, upgrade.** 0.2.10 added a `Content-Length` branch to the streaming reader whose two exits — a read timeout diff --git a/README.md b/README.md index 693e738..9a1ef7b 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,7 @@ mcpp add tinyhttps@0.3.0 ```toml [dependencies] -tinyhttps = "0.3.0" +tinyhttps = "0.3.1" ``` ### 构建 diff --git a/mcpp.toml b/mcpp.toml index def4bb6..d2837cd 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "tinyhttps" -version = "0.3.0" +version = "0.3.1" description = "Minimal C++23 HTTP/HTTPS client with SSE streaming support" license = "Apache-2.0" repo = "https://github.com/mcpplibs/tinyhttps" @@ -14,3 +14,10 @@ mbedtls = "3.6.1" [dev-dependencies.compat] gtest = "1.15.2" + +# The socket interface follows the C library rather than the operating system +# (src/platform.cppm). Where the C library is musl, including musl above openkal +# on Windows, it is the POSIX interface; the platform's own headers are not part +# of that graph. +[target.'cfg(c-abi = "musl")'.build] +cxxflags = ["-DTINYHTTPS_POSIX_SOCKETS"] diff --git a/src/http.cppm b/src/http.cppm index a0832f6..761155c 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -786,7 +786,7 @@ static bool drain_body(TlsSocket& sock, const ResponseHead& head, // calling `keep()` drops the connection, and a path added tomorrow is safe // before anyone reviews it. // -// ⚠️ Ordering: `drop()` destroys a `TlsSocket`, which sends `close_notify`, +// Ordering: `drop()` destroys a `TlsSocket`, which sends `close_notify`, // which writes to a socket whose peer may be gone. That write is why P0 — // `MSG_NOSIGNAL` in `Socket::write` — had to land before this class existed; // without it, making the drop path more common would have made issue #16 more @@ -808,7 +808,7 @@ public: // Drop now rather than at the end of the scope. Idempotent, and a no-op // after `keep()`. // - // ⚠️ Callers that recurse — the redirect paths — MUST settle the guard + // Callers that recurse — the redirect paths — MUST settle the guard // before recursing. The inner call puts its own connection into the pool // under the same key when the redirect is to the same host, and a guard // still armed when the outer scope ends would then delete it. diff --git a/src/platform.cppm b/src/platform.cppm index 685ffb0..dcf3ade 100644 --- a/src/platform.cppm +++ b/src/platform.cppm @@ -1,6 +1,15 @@ module; -#ifndef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifndef TINYHTTPS_WINSOCK #include #include #include @@ -14,7 +23,7 @@ import std; // mcpplibs.tinyhttps:platform — the one place OS-specific networking quirks // live. Everything else (socket.cppm, http.cppm…) stays portable and branches -// on `platform::is_windows` with `if constexpr`, never raw #ifdef. +// on `platform::uses_winsock` with `if constexpr`, never raw #ifdef. // // ── The platforms this library is known to run on, and what differs ────────── // @@ -26,7 +35,7 @@ import std; // | Windows | nothing to do | no SIGPIPE, neither macro | // | above openkal | MSG_NOSIGNAL, and moot | see below | // -// ⭐ ABOVE openkal there are no signals at all, so the hazard issue #16 +// ABOVE openkal there are no signals at all, so the hazard issue #16 // describes cannot arise — and the same code is nevertheless correct there // without a branch. openkal-musl defines MSG_NOSIGNAL // (`musl/include/sys/socket.h:344`) and accepts it as a no-op, in its own words: @@ -36,7 +45,7 @@ import std; // -ENOSYS (`:550`, `:592-593`). SO_NOSIGPIPE is a BSD spelling that musl does // not define, so `socket.cppm`'s `#ifdef` for it simply does not compile in. // -// ⚠️ One difference that IS live above openkal: connect() completes before it +// One difference that IS live above openkal: connect() completes before it // returns even on a non-blocking descriptor, because `kal_net_connect` has no // form that begins a connection and reports its outcome later // (`port/src/okm_net.c:454-467`). `Socket::connect_addrinfo` therefore never @@ -58,14 +67,14 @@ import std; namespace mcpplibs::tinyhttps::platform { // Compile-time platform flag for `if constexpr` at call sites. -export inline constexpr bool is_windows = -#ifdef _WIN32 +export inline constexpr bool uses_winsock = +#ifdef TINYHTTPS_WINSOCK true; #else false; #endif -#ifndef _WIN32 +#ifndef TINYHTTPS_WINSOCK namespace { @@ -185,7 +194,7 @@ inline std::vector dns_query_a(const std::string& server, const cha } // anonymous namespace -#endif // !_WIN32 +#endif // !TINYHTTPS_WINSOCK // True when libc's own resolver has a usable config (/etc/resolv.conf). When // false, callers should prefer resolve_fallback() to avoid a multi-second stall @@ -195,9 +204,9 @@ inline std::vector dns_query_a(const std::string& server, const cha // `if constexpr` still compiles its discarded branch, which would reference the // POSIX-only helpers above that don't exist on Windows. Concentrating that // preprocessor divergence here is exactly why this platform module exists; call -// sites elsewhere branch on `is_windows` with `if constexpr`. +// sites elsewhere branch on `uses_winsock` with `if constexpr`. export bool system_resolver_configured() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK return true; #else std::error_code ec; @@ -211,7 +220,7 @@ export bool system_resolver_configured() { // on query failure. A numeric host is returned unchanged. export std::vector resolve_fallback([[maybe_unused]] const char* host, [[maybe_unused]] int timeoutMs) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK return {}; #else if (is_numeric_host(host)) return { std::string(host) }; diff --git a/src/socket.cppm b/src/socket.cppm index 12b1e7a..5800588 100644 --- a/src/socket.cppm +++ b/src/socket.cppm @@ -1,6 +1,15 @@ module; -#ifdef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK #include #include #pragma comment(lib, "ws2_32.lib") @@ -21,7 +30,7 @@ import :platform; namespace mcpplibs::tinyhttps { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK using SocketHandle = SOCKET; constexpr SocketHandle INVALID_SOCKET_FD = INVALID_SOCKET; #else @@ -88,7 +97,7 @@ public: return ok; }; - if constexpr (platform::is_windows) { + if constexpr (platform::uses_winsock) { return try_resolved(host, /*numeric=*/false); } else { // Fall back to a manual DNS query when libc can't resolve (Termux: @@ -131,7 +140,7 @@ public: // option rather than as a send flag; `write` below carries the flag // for the platforms that have one. // - // ⭐ NOTHING SELECTS THIS AND NOTHING MAY. It is not a feature, not + // NOTHING SELECTS THIS AND NOTHING MAY. It is not a feature, not // a config field and not a runtime probe: the preprocessor reads the // target's own and the answer is already complete. // Measured on this machine — glibc: SO_NOSIGPIPE absent, @@ -175,7 +184,7 @@ public: if (rc == 0) { connected = true; } else { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK if (WSAGetLastError() == WSAEWOULDBLOCK) { #else if (errno == EINPROGRESS) { @@ -266,14 +275,14 @@ public: } static void platform_init() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSADATA wsaData; WSAStartup(MAKEWORD(2, 2), &wsaData); #endif } static void platform_cleanup() { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSACleanup(); #endif } @@ -282,7 +291,7 @@ private: SocketHandle fd_ = INVALID_SOCKET_FD; static bool set_non_blocking(SocketHandle fd, bool nonBlocking) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK u_long mode = nonBlocking ? 1 : 0; return ioctlsocket(fd, FIONBIO, &mode) == 0; #else @@ -298,7 +307,7 @@ private: } static bool poll_fd(SocketHandle fd, int timeoutMs, bool forRead) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK WSAPOLLFD pfd{}; pfd.fd = fd; pfd.events = forRead ? POLLIN : POLLOUT; @@ -314,7 +323,7 @@ private: } static void close_handle(SocketHandle fd) { -#ifdef _WIN32 +#ifdef TINYHTTPS_WINSOCK ::closesocket(fd); #else ::close(fd); diff --git a/tests/test_pool.cpp b/tests/test_pool.cpp index 164f1ee..db4970a 100644 --- a/tests/test_pool.cpp +++ b/tests/test_pool.cpp @@ -13,7 +13,16 @@ #include #include "tls_test_server.hpp" -#ifndef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifndef TINYHTTPS_WINSOCK #include #include #include @@ -56,7 +65,7 @@ class PoolTest : public ::testing::Test { // does. The read times out with 50 bytes still owed, and those 50 bytes then // arrive on a socket nobody is reading. // -// ⚠️ THE LATE SECOND HALF IS WHAT MAKES THIS A TEST. An earlier form of this +// THE LATE SECOND HALF IS WHAT MAKES THIS A TEST. An earlier form of this // server stalled and never sent the rest; the connection was then silent rather // than dirty, and the stale-connection retry rescued the second request whether // or not the pool guard worked. Verified by mutation: with the guard's @@ -719,7 +728,7 @@ TEST_F(PoolTest, AnEndlessTrailerSectionIsRefused) { // ── issue #16 · SIGPIPE ────────────────────────────────────────────────────── -#ifndef _WIN32 +#ifndef TINYHTTPS_WINSOCK // THE TEST HAS TO FORK, AND THE REASON IS ITSELF EVIDENCE. // // `mbedtls_net_bind` calls `net_prepare`, which does `signal(SIGPIPE, SIG_IGN)` @@ -801,4 +810,4 @@ TEST(SigPipe, WritingToADepartedPeerReturnsInsteadOfKillingTheProcess) { EXPECT_EQ(WEXITSTATUS(status), 0) << "child exit code 3 = could not connect, 4 = the write never failed"; } -#endif // !_WIN32 +#endif // !TINYHTTPS_WINSOCK diff --git a/tests/tls_test_server.hpp b/tests/tls_test_server.hpp index 3bd8e81..4170680 100644 --- a/tests/tls_test_server.hpp +++ b/tests/tls_test_server.hpp @@ -29,7 +29,16 @@ #include #include -#ifdef _WIN32 +// The socket interface is chosen by the C library, not by the operating +// system. On Windows with the platform's own C runtime it is Windows Sockets; +// where the C library is POSIX-shaped (openkal-musl, selected in mcpp.toml by +// `cfg(c-abi = "musl")`) it is the POSIX interface on every system, Windows +// included. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK #include #include #else From f26a77feed4f111e4959f33603ee8d78c4926200 Mon Sep 17 00:00:00 2001 From: SPeak Agent Date: Tue, 29 Sep 2026 23:19:00 +0800 Subject: [PATCH 02/15] 0.3.2 --- download_to_file reports the writes the disk refused (#19) * fix(http): download_to_file reports the writes the disk refused download_to_file called ofs.write for each body chunk without looking at the stream, added each chunk's size to bytesWritten from what the network delivered, and set bytesWritten after ofs.close() without looking at that either. On a full disk (ENOSPC) the transfer therefore succeeded: a 420,831,054 byte download left 220,979,200 bytes on disk while bytesWritten said 420,831,054 and ok() was true, and the caller blamed the source (a checksum mismatch) for what was the local disk. Each chunk is now flushed and the stream checked. On the first failure the transfer stops reading, error becomes "write : " with the reason taken from errno, and the connection is dropped instead of returned to the pool with the rest of the body still on it. Closing the file is checked the same way. Additive: DownloadToFileResult::writeFailed says the fault is the destination and not the source (a failed write, a failed close, or a file that could not be opened), and bytesReceived carries the network count. bytesWritten now counts bytes the file accepted; the two are equal for every transfer that did not fail to write. Tests: a download into /dev/full (Linux) against the in-process TLS server asserts not ok, writeFailed, bytesWritten == 0, the ENOSPC text, that reading stopped after the first refused chunk, and that the next request opens a new connection. Verified by mutation: with the stream checks removed it reports ok() and bytesWritten == 65536. * 0.3.2 --- download_to_file reports the writes the disk refused Version, CHANGELOG entry, and the README install lines name this release. * ci: bump mcpp to 2026.9.28.3 to satisfy the index floor The index now requires mcpp >= 2026.9.18.3, and MCPP_VERSION pinned 2026.8.29.1, so every `mcpp` invocation that needed the index ended at error: index requires mcpp >= 2026.9.18.3 but this is mcpp 2026.8.29.1 [E0006] That is the failure the "Smoke-test the project templates" step has reported on master since the scheduled run of 2026-09-21. MCPP_VERSION is the only place the version is named; both jobs' install steps and both cache keys read it. --------- Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/workflows/ci.yml | 2 +- CHANGELOG.md | 30 ++++++++++++ README.md | 9 ++-- mcpp.toml | 2 +- src/http.cppm | 102 ++++++++++++++++++++++++++++++++++++--- tests/test_pool.cpp | 88 +++++++++++++++++++++++++++++++++ 6 files changed, 222 insertions(+), 11 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8104eea..fba59e5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -31,7 +31,7 @@ on: workflow_dispatch: env: - MCPP_VERSION: 2026.8.29.1 + MCPP_VERSION: 2026.9.28.3 XLINGS_VERSION: v2026.8.17.2 XLINGS_NON_INTERACTIVE: '1' diff --git a/CHANGELOG.md b/CHANGELOG.md index 897790b..e3d5792 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,35 @@ # Changelog +## 0.3.2 + +`download_to_file` no longer reports success for a file the disk did not keep. + +It called `ofs.write` for each body chunk without looking at the stream, added +each chunk's size to `bytesWritten` from what the network delivered, and set +`bytesWritten` after `ofs.close()` without looking at that either. On a full disk +(`ENOSPC`) the transfer therefore succeeded: a 420,831,054 byte download left +220,979,200 bytes on disk while `bytesWritten` said 420,831,054 and `ok()` was +true, and the caller blamed the source (a checksum mismatch) for what was the +local disk. + +* Each chunk is flushed and the stream checked. On the first failure the + transfer stops reading, `error` becomes `write : ` (the reason + is `errno` as `std::generic_category` words it, for example `No space left on + device`), and the connection is dropped rather than returned to the pool with + the rest of the body still on it. Closing the file is checked the same way. +* `DownloadToFileResult::writeFailed` is new and is true when the fault is the + destination rather than the source: the write failed, the close failed, or the + file could not be opened (`error` is still `Cannot open file: ` for the + last). +* `DownloadToFileResult::bytesReceived` is new: the bytes of body the + connection delivered. +* `DownloadToFileResult::bytesWritten` changes meaning, and this is the one + behaviour change: it now counts bytes the file accepted, where it used to + count bytes the network delivered. The two are equal for every transfer that + did not fail to write. When a write fails, the chunk that failed is not + counted, though the file may hold part of it, so the value is a floor on the + file's size. + ## 0.3.1 The socket interface is selected by the C library rather than by the operating diff --git a/README.md b/README.md index 9a1ef7b..0ebe1bb 100644 --- a/README.md +++ b/README.md @@ -114,14 +114,14 @@ cd examples/openkal && mcpp run ### 添加依赖 ```bash -mcpp add tinyhttps@0.3.0 +mcpp add tinyhttps@0.3.2 ``` 或在 `mcpp.toml` 中手动添加： ```toml [dependencies] -tinyhttps = "0.3.1" +tinyhttps = "0.3.2" ``` ### 构建 @@ -140,7 +140,10 @@ mcpplibs::tinyhttps::HttpClient client; auto result = client.download_to_file( "https://example.com/big.tar.gz", "out/big.tar.gz", [](std::int64_t total, std::int64_t done) { /* progress */ }); -if (!result.ok()) { /* result.error */ } +if (!result.ok()) { + // result.error says why; result.writeFailed says the local file is at + // fault (a full disk, a refused write) and not the server. +} ``` ## License diff --git a/mcpp.toml b/mcpp.toml index d2837cd..5ee04a8 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "tinyhttps" -version = "0.3.1" +version = "0.3.2" description = "Minimal C++23 HTTP/HTTPS client with SSE streaming support" license = "Apache-2.0" repo = "https://github.com/mcpplibs/tinyhttps" diff --git a/src/http.cppm b/src/http.cppm index 761155c..5ef3e74 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -1,3 +1,9 @@ +module; + +// `import std` carries no `errno`: it is a macro, and a module does not export +// macros. The download path reads it to say why the local file refused a write. +#include + export module mcpplibs.tinyhttps:http; import :tls; @@ -105,11 +111,39 @@ export using DownloadProgressFn = std::function expectedBytes; std::string finalUrl; std::string etag; std::string lastModified; + + // Bytes of body the connection delivered, whether or not the file took them. + // Equal to `bytesWritten` unless `writeFailed`. + std::int64_t bytesReceived { 0 }; + + // TRUE WHEN THE FAULT IS THE DESTINATION AND NOT THE SOURCE — the file could + // not be opened, a write to it failed (a full disk, a quota, a device that + // refuses), or closing it reported that buffered data did not reach it. + // + // `error` then names the path and the reason, and the transfer stops at the + // first failure instead of reading on and discarding what it reads. It is + // what lets a caller tell "the server sent something wrong" from "this + // machine could not keep it" without matching text in `error`; a checksum + // failure downstream of a truncated file otherwise blames the wrong party. + bool writeFailed { false }; + bool ok() const { return statusCode >= 200 && statusCode < 300 && error.empty(); } }; @@ -1222,34 +1256,82 @@ private: std::ofstream ofs(destFile, std::ios::binary); if (!ofs) { result.error = "Cannot open file: " + destFile.string(); + result.writeFailed = true; settle_without_body(exchange, hasBody, guard); return result; } - if (!hasBody) { + // What the file refused, in the words of the system that refused it. + // `errno` is cleared before each operation that can set it, because the + // socket reads between writes leave their own values behind and a stale + // EAGAIN reported as the reason a disk filled is worse than no reason. + int writeErrno = 0; + bool writeFailed = false; + auto close_file = [&] { + errno = 0; ofs.close(); + if (!writeFailed && ofs.fail()) { + writeFailed = true; + writeErrno = errno; + } + }; + auto write_failure = [&] { + return "write " + destFile.string() + ": " + + (writeErrno != 0 + ? std::generic_category().message(writeErrno) + : std::string("stream error")); + }; + + if (!hasBody) { + close_file(); + if (writeFailed) { + result.error = write_failure(); + result.writeFailed = true; + return result; + } if (config_.keepAlive && !exchange.head.connectionClose) guard.keep(); return result; } const std::int64_t totalBytes = exchange.head.contentLength > 0 ? exchange.head.contentLength : 0; - std::int64_t downloaded = 0; + std::int64_t received = 0; + std::int64_t written = 0; bool cancelled = false; auto outcome = read_body( *exchange.sock, exchange.head, config_.readTimeoutMs, std::numeric_limits::max(), [&](std::string_view data) -> bool { + const auto size = static_cast(data.size()); + received += size; + + // Flushed per chunk, and the stream checked after it. A write + // into the stream's buffer succeeds whatever the disk has left; + // the failure surfaces at the flush that hands the buffer to the + // operating system, and without one here it surfaced at + // `close()`, after every chunk had been counted as written. + // `read_body` hands over slices of at most 8 KiB, about the size + // of the stream's own buffer, so this costs no more system calls + // than the buffering it replaces. + errno = 0; ofs.write(data.data(), static_cast(data.size())); - downloaded += static_cast(data.size()); - if (onProgress) onProgress(totalBytes, downloaded); + ofs.flush(); + if (!ofs) { + writeFailed = true; + writeErrno = errno; + return false; // reading on would only discard what it reads + } + + written += size; + if (onProgress) onProgress(totalBytes, written); if (isCancelled && isCancelled()) { cancelled = true; return false; } return true; }); - ofs.close(); - result.bytesWritten = downloaded; + close_file(); + result.bytesWritten = written; + result.bytesReceived = received; switch (outcome.end) { case BodyEnd::Complete: @@ -1264,6 +1346,14 @@ private: result.error = outcome.error; break; } + + // The destination's failure outranks whatever the switch recorded: a + // stop that the file asked for is not a cancellation, and a body that + // arrived whole into a file that lost it has not succeeded. + if (writeFailed) { + result.error = write_failure(); + result.writeFailed = true; + } return result; } diff --git a/tests/test_pool.cpp b/tests/test_pool.cpp index db4970a..eb91eaa 100644 --- a/tests/test_pool.cpp +++ b/tests/test_pool.cpp @@ -624,6 +624,94 @@ TEST_F(PoolTest, ADownloadDelimitedByTheCloseReportsSuccess) { std::filesystem::remove_all(dir, ec); } +// ── a destination that refuses the bytes ───────────────────────────────────── + +// `/dev/full` accepts an open and answers every write with ENOSPC, which is what +// a full disk does at the moment it fills — without needing one. +// +// Before 0.3.2 the transfer "succeeded": `ofs.write` was never checked, the +// count came from the network, and `bytesWritten` said the whole body when the +// file held none of it. The caller then blamed the source (a checksum mismatch) +// for what was the local disk. Verified by mutation: with the stream check +// removed, `ok()` is true and `bytesWritten` is the body's length. +#ifdef __linux__ +TEST_F(PoolTest, ADownloadIntoAFullDeviceFailsAsALocalWriteAndCountsNothing) { + if (!std::filesystem::exists("/dev/full")) GTEST_SKIP() << "no /dev/full"; + + const std::string payload(64 * 1024, 'F'); + tls_test::Server server([payload](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response(payload)); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + auto result = client.download_to_file(server.url("/big"), "/dev/full"); + + EXPECT_FALSE(result.ok()) << "a file that took nothing was reported as downloaded"; + EXPECT_TRUE(result.writeFailed) + << "the caller must be able to tell the disk from the source; error: " + << result.error; + EXPECT_EQ(result.statusCode, 200); + EXPECT_EQ(result.bytesWritten, 0); + EXPECT_GT(result.bytesReceived, 0); + EXPECT_LT(result.bytesReceived, static_cast(payload.size())) + << "the transfer read on after the first refused write"; + EXPECT_EQ(result.error.rfind("write /dev/full: ", 0), 0u) << result.error; + EXPECT_NE(result.error.find( + std::make_error_code(std::errc::no_space_on_device).message()), + std::string::npos) + << result.error; + + // The rest of the body was left on the socket, so the connection is not + // handed to the next request. The next request also shows the client is + // not left in a state where a good destination fails too. + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_full"; + std::filesystem::create_directories(dir); + auto dest = dir / "ok.bin"; + auto good = client.download_to_file(server.url("/big"), dest); + EXPECT_TRUE(good.ok()) << "error: " << good.error; + EXPECT_FALSE(good.writeFailed); + EXPECT_EQ(good.bytesWritten, static_cast(payload.size())); + EXPECT_EQ(good.bytesReceived, good.bytesWritten); + EXPECT_EQ(server.accepts(), 2) + << "the connection with an unread body was reused"; + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} +#endif + +// A destination that cannot be opened is the same kind of failure: nothing was +// wrong with what the server sent. +TEST_F(PoolTest, ADestinationThatCannotBeOpenedIsALocalFailure) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("payload")); + return true; + }); + ASSERT_FALSE(server.failed()); + + auto dir = std::filesystem::temp_directory_path() / "tinyhttps_pool_noopen"; + std::filesystem::create_directories(dir); + { + // A regular file where a directory is needed: no parent can be made. + std::ofstream blocker(dir / "blocker", std::ios::binary); + blocker << "not a directory"; + } + + https::HttpClient client(test_config()); + auto result = client.download_to_file(server.url("/x"), + dir / "blocker" / "out.bin"); + + EXPECT_FALSE(result.ok()); + EXPECT_TRUE(result.writeFailed) << "error: " << result.error; + EXPECT_EQ(result.error.rfind("Cannot open file: ", 0), 0u) << result.error; + EXPECT_EQ(result.bytesWritten, 0); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} + // ── interim responses ──────────────────────────────────────────────────────── // A 1xx is not the answer. RFC 9112 §2.1 requires a client to read past one or From 1d14c2377e6ce02338d16e3ce763979bf5ec9894 Mon Sep 17 00:00:00 2001 From: Cloud_Yun Date: Thu, 1 Oct 2026 13:02:48 +0900 Subject: [PATCH 03/15] fix(tls): refuse a server certificate that does not verify With verifySsl = true the TLS config used MBEDTLS_SSL_VERIFY_OPTIONAL, which lets the handshake finish whatever the certificate looks like, and nothing read mbedtls_ssl_get_verify_result afterwards. A self-signed certificate, one for another host and an expired one were all accepted. Use MBEDTLS_SSL_VERIFY_REQUIRED. A failed handshake now says why in the response's statusText: the text from mbedtls_x509_crt_verify_info for a verification failure, the mbedTLS error text for any other handshake error. A refused TCP connection still reads "Connection failed". When no CA bundle can be found and verifySsl is true, the connection now fails with a message that points at SSL_CERT_FILE, instead of going ahead without verification. Tests serve self-signed, expired and wrong-name certificates from the existing in-process server and trust them through SSL_CERT_FILE, so they need no network. On Windows, where there is no bundle file, load_ca_certs reads the roots in the system ROOT certificate store that are usable for TLS servers (serverAuth) when SSL_CERT_FILE is not set. That part is built only where platform::uses_winsock is true, and has not been run on Windows. --- CHANGELOG.md | 21 ++++++ README.md | 2 +- src/ca_bundle.cppm | 79 +++++++++++++++++++++ src/http.cppm | 4 +- src/tls.cppm | 88 +++++++++++++---------- tests/test_tls_verify.cpp | 143 ++++++++++++++++++++++++++++++++++++++ tests/tls_test_server.hpp | 60 ++++++++++++++-- 7 files changed, 354 insertions(+), 43 deletions(-) create mode 100644 tests/test_tls_verify.cpp diff --git a/CHANGELOG.md b/CHANGELOG.md index e3d5792..7b8b829 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,26 @@ # Changelog +## Unreleased + +`verifySsl = true` now verifies the server certificate. The handshake used to +complete with any certificate (a self-signed one, one for another host, an +expired one) because `MBEDTLS_SSL_VERIFY_OPTIONAL` was set and nothing read the +result afterwards. + +* The handshake fails when the chain is not trusted, the certificate has expired + or its name is not the host being connected to. The reason is in `statusText`, + for example `certificate verification failed: The certificate validity has + expired`; other handshake failures carry the mbedTLS error text. A refused TCP + connection still reads `Connection failed`. +* Behaviour change: with `verifySsl = true` and no CA bundle found + (`SSL_CERT_FILE` unset or unreadable, none of the system locations present), + the connection fails with a message saying so, where it used to go ahead + unverified. Point `SSL_CERT_FILE` at a PEM bundle, or set `verifySsl = false`. +* On Windows, when `SSL_CERT_FILE` is not set, the roots in the system `ROOT` + certificate store that are usable for TLS servers are read. That code is built + only for Windows Sockets targets, not for the musl one. MinGW builds need + `-lcrypt32`, as they already need `-lws2_32`. + ## 0.3.2 `download_to_file` no longer reports success for a file the disk did not keep. diff --git a/README.md b/README.md index 0ebe1bb..def0b1f 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ means exactly what it did before. | --- | --- | --- | | `connectTimeoutMs` | 10000 | TCP connect | | `readTimeoutMs` | 60000 | any single read, and the total wait on a blocked write | -| `verifySsl` | true | verify the server certificate | +| `verifySsl` | true | verify the server certificate against the CA bundle (`SSL_CERT_FILE`, else the Windows `ROOT` certificate store in Windows Sockets builds, else a system location); the connection fails if the certificate is not trusted, has expired or is for another host, or if no bundle is found | | `keepAlive` | true | reuse connections between requests | | `maxRedirects` | 10 | 0 disables redirect following | | `maxResponseBodyBytes` | 64 MiB | the most `send()` will hold in memory; does not bound `download_to_file` or `send_stream` | diff --git a/src/ca_bundle.cppm b/src/ca_bundle.cppm index 577179f..f481d77 100644 --- a/src/ca_bundle.cppm +++ b/src/ca_bundle.cppm @@ -2,9 +2,28 @@ module; #include +// The same switch as socket.cppm. The store is read through the Win32 API, whose +// headers a build on the POSIX socket interface does not have. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#include +#pragma comment(lib, "crypt32.lib") +#endif + export module mcpplibs.tinyhttps:ca_bundle; import std; +import :platform; namespace mcpplibs::tinyhttps { @@ -24,6 +43,59 @@ auto read_file(const char* path) -> std::string { return result; } +#ifdef TINYHTTPS_WINSOCK +// Whether the root may be used to authenticate a server: no usage restriction at +// all, or serverAuth listed. Windows marks roots it no longer trusts for TLS +// servers through this restriction, and the store still holds them. +auto trusted_for_servers(const CERT_CONTEXT* c) -> bool { + DWORD n = 0; + if (!::CertGetEnhancedKeyUsage(c, 0, nullptr, &n)) { + return false; + } + std::vector buf(n); + auto* usage = reinterpret_cast(buf.data()); + if (!::CertGetEnhancedKeyUsage(c, 0, usage, &n)) { + return false; + } + if (usage->cUsageIdentifier == 0) { + // CRYPT_E_NOT_FOUND means no restriction; otherwise the root allows no use. + return ::GetLastError() == CRYPT_E_NOT_FOUND; + } + for (DWORD i = 0; i < usage->cUsageIdentifier; ++i) { + if (std::strcmp(usage->rgpszUsageIdentifier[i], szOID_PKIX_KP_SERVER_AUTH) == 0) { + return true; + } + } + return false; +} +#endif + +// The Windows ROOT store as one PEM string; empty elsewhere or on failure. +auto system_store_pem() -> std::string { + std::string pem; +#ifdef TINYHTTPS_WINSOCK + if (HCERTSTORE store = ::CertOpenSystemStoreW(0, L"ROOT")) { + for (const CERT_CONTEXT* c = nullptr; (c = ::CertEnumCertificatesInStore(store, c));) { + if (!trusted_for_servers(c)) { + continue; + } + DWORD n = 0; + if (!::CryptBinaryToStringA(c->pbCertEncoded, c->cbCertEncoded, + CRYPT_STRING_BASE64HEADER, nullptr, &n)) { + continue; + } + std::string one(n, '\0'); + if (::CryptBinaryToStringA(c->pbCertEncoded, c->cbCertEncoded, + CRYPT_STRING_BASE64HEADER, one.data(), &n)) { + pem.append(one, 0, n); + } + } + ::CertCloseStore(store, 0); + } +#endif + return pem; +} + } // anonymous namespace export auto load_ca_certs() -> std::string { @@ -37,6 +109,13 @@ export auto load_ca_certs() -> std::string { } } + // Windows has no bundle file; read the system's trusted roots instead. + if constexpr (platform::uses_winsock) { + if (auto pem = system_store_pem(); !pem.empty()) { + return pem; + } + } + std::vector ca_paths; // 2. Non-FHS prefixes (Termux et al. ship the bundle under $PREFIX). Probed diff --git a/src/http.cppm b/src/http.cppm index 5ef3e74..440c0f6 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -1034,8 +1034,10 @@ private: auto [inserted, ok] = pool_.emplace(poolKey, TlsSocket{}); sock = &inserted->second; if (!open_connection(*sock, parsed)) { + std::string why = sock->error().empty() ? "Connection failed" + : sock->error(); guard.drop(); - return { nullptr, {}, "Connection failed" }; + return { nullptr, {}, std::move(why) }; } } diff --git a/src/tls.cppm b/src/tls.cppm index 39494ca..5937cae 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -78,6 +78,21 @@ static int bio_recv(void* ctx, unsigned char* buf, size_t len) { return ret; // 0 means end of stream; mbedtls turns it into SSL_CONN_EOF } +static std::string mbedtls_message(int ret) { + char buf[200]; + mbedtls_strerror(ret, buf, sizeof buf); + return buf; +} + +// verify_info ends every reason with a newline; join them with "; ". +static std::string one_line(std::string text) { + while (!text.empty() && text.back() == '\n') text.pop_back(); + for (std::size_t pos = 0; (pos = text.find('\n', pos)) != std::string::npos;) { + text.replace(pos, 1, "; "); + } + return text; +} + // WHAT A READ ENDED IN, WHICH `int` COULD NOT SAY. // // `TlsSocket::read` returned 0 for a peer that had closed AND for a transport @@ -108,7 +123,8 @@ public: // Move constructor TlsSocket(TlsSocket&& other) noexcept : socket_(std::move(other.socket_)) - , state_(std::move(other.state_)) { + , state_(std::move(other.state_)) + , error_(std::move(other.error_)) { // Re-bind BIO to point to our socket_ (not the moved-from one) if (state_) { mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); @@ -121,6 +137,7 @@ public: close(); socket_ = std::move(other.socket_); state_ = std::move(other.state_); + error_ = std::move(other.error_); // Re-bind BIO to point to our socket_ if (state_) { mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); @@ -133,14 +150,20 @@ public: return state_ != nullptr && socket_.is_valid(); } + // Why the last connect_over/connect failed once the TCP connection was up; + // empty if it failed earlier or has not failed. + [[nodiscard]] const std::string& error() const { return error_; } + // Connect over an already-established Socket (e.g. a proxy tunnel). // Takes ownership of the socket and performs TLS handshake on top of it. bool connect_over(Socket&& socket, const char* host, bool verifySsl) { + error_.clear(); socket_ = std::move(socket); return setup_tls(host, verifySsl); } bool connect(const char* host, int port, int timeoutMs, bool verifySsl) { + error_.clear(); // Step 1: TCP connect via Socket if (!socket_.connect(host, port, timeoutMs)) { return false; @@ -229,6 +252,14 @@ public: private: Socket socket_; std::unique_ptr state_; + std::string error_; + + bool fail(std::string message) { + error_ = std::move(message); + state_.reset(); + socket_.close(); + return false; + } bool setup_tls(const char* host, bool verifySsl) { state_ = std::make_unique(); @@ -236,22 +267,14 @@ private: int ret = mbedtls_ctr_drbg_seed( &state_->ctr_drbg, mbedtls_entropy_func, &state_->entropy, nullptr, 0); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); ret = mbedtls_ssl_config_defaults( &state_->conf, MBEDTLS_SSL_IS_CLIENT, MBEDTLS_SSL_TRANSPORT_STREAM, MBEDTLS_SSL_PRESET_DEFAULT); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); mbedtls_ssl_conf_rng(&state_->conf, mbedtls_ctr_drbg_random, &state_->ctr_drbg); @@ -261,6 +284,10 @@ private: // Load CA certs auto ca_pem = load_ca_certs(); + if (ca_pem.empty() && verifySsl) { + return fail("no CA certificate bundle found; set SSL_CERT_FILE to a PEM " + "file of trusted roots, or set verifySsl to false"); + } if (!ca_pem.empty()) { ret = mbedtls_x509_crt_parse( &state_->ca_cert, @@ -268,37 +295,22 @@ private: ca_pem.size() + 1); // +1 for null terminator required by mbedtls // ret > 0 means some certs failed to parse but others succeeded — acceptable if (ret < 0) { - state_.reset(); - socket_.close(); - return false; + return fail("cannot parse the CA certificate bundle: " + mbedtls_message(ret)); } mbedtls_ssl_conf_ca_chain(&state_->conf, &state_->ca_cert, nullptr); } - // Certificate verification - // Use OPTIONAL (not REQUIRED) so handshake succeeds even if the CA - // bundle is incomplete; callers that need strict verification can - // inspect the verification result after handshake. - if (verifySsl) { - mbedtls_ssl_conf_authmode(&state_->conf, MBEDTLS_SSL_VERIFY_OPTIONAL); - } else { - mbedtls_ssl_conf_authmode(&state_->conf, MBEDTLS_SSL_VERIFY_NONE); - } + // Certificate verification. REQUIRED makes the handshake fail on a bad + // chain, an expired certificate or a name that is not `host`. + mbedtls_ssl_conf_authmode(&state_->conf, + verifySsl ? MBEDTLS_SSL_VERIFY_REQUIRED : MBEDTLS_SSL_VERIFY_NONE); ret = mbedtls_ssl_setup(&state_->ssl, &state_->conf); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); // Set hostname for SNI ret = mbedtls_ssl_set_hostname(&state_->ssl, host); - if (ret != 0) { - state_.reset(); - socket_.close(); - return false; - } + if (ret != 0) return fail(mbedtls_message(ret)); // Set BIO callbacks using our Socket mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); @@ -306,9 +318,13 @@ private: // Perform TLS handshake while ((ret = mbedtls_ssl_handshake(&state_->ssl)) != 0) { if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE) { - state_.reset(); - socket_.close(); - return false; + if (ret == MBEDTLS_ERR_X509_CERT_VERIFY_FAILED) { + char info[512] = {}; + mbedtls_x509_crt_verify_info(info, sizeof info, "", + mbedtls_ssl_get_verify_result(&state_->ssl)); + return fail("certificate verification failed: " + one_line(info)); + } + return fail("TLS handshake failed: " + mbedtls_message(ret)); } } diff --git a/tests/test_tls_verify.cpp b/tests/test_tls_verify.cpp new file mode 100644 index 0000000..ccaae34 --- /dev/null +++ b/tests/test_tls_verify.cpp @@ -0,0 +1,143 @@ +// A client with `verifySsl = true` refuses a server certificate it cannot trust. +// +// Each test serves one certificate from the in-process TLS server and trusts it +// (or not) by pointing SSL_CERT_FILE at a bundle written for the test, so the +// machine's own CA store never enters into it and nothing leaves 127.0.0.1. +// The certificates in tls_test_server.hpp are self-signed, which lets a trusted +// one be its own bundle and lets the expired and wrong-name ones differ from it +// in that one respect only. +#include +#include "tls_test_server.hpp" + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +void set_env(const char* name, const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s(name, value); +#else + ::setenv(name, value, 1); +#endif +} + +class TlsVerifyTest : public ::testing::Test { +protected: + void SetUp() override { + https::Socket::platform_init(); + const char* old = std::getenv("SSL_CERT_FILE"); + hadOld_ = old != nullptr; + if (hadOld_) old_ = old; + bundle_ = std::filesystem::temp_directory_path() + / ("tinyhttps-verify-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + ".pem"); + } + + void TearDown() override { + set_env("SSL_CERT_FILE", hadOld_ ? old_.c_str() : ""); + std::error_code ec; + std::filesystem::remove(bundle_, ec); + } + + // Makes `content` the only CA bundle the client will read. + void trust(std::string_view content) { + std::ofstream(bundle_, std::ios::binary) << content; + set_env("SSL_CERT_FILE", bundle_.string().c_str()); + } + + https::HttpResponse get(const tls_test::Server& server) { + https::HttpClientConfig cfg; // verifySsl is true by default + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = server.url("/"); + return client.send(req); + } + +private: + std::filesystem::path bundle_; + std::string old_; + bool hadOld_ { false }; +}; + +bool answer_ok(tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("ok")); + return true; +} + +} // namespace + +TEST_F(TlsVerifyTest, ACertificateIssuedByATrustedCaIsAccepted) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust(tls_test::kCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + EXPECT_EQ(r.body, "ok"); +} + +TEST_F(TlsVerifyTest, ACertificateNoTrustedCaIssuedIsRefused) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); // a bundle that does not contain the server's + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("certificate verification failed"), std::string::npos) + << r.statusText; + EXPECT_NE(r.statusText.find("not correctly signed by the trusted CA"), std::string::npos) + << r.statusText; +} + +TEST_F(TlsVerifyTest, AnExpiredCertificateIsRefused) { + tls_test::Server server(answer_ok, tls_test::kExpiredCertPem, tls_test::kExpiredKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kExpiredCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("expired"), std::string::npos) << r.statusText; +} + +TEST_F(TlsVerifyTest, ACertificateForAnotherHostnameIsRefused) { + tls_test::Server server(answer_ok, tls_test::kWrongNameCertPem, tls_test::kWrongNameKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("does not match"), std::string::npos) << r.statusText; +} + +TEST_F(TlsVerifyTest, ABundleThatCannotBeParsedIsReportedAsSuch) { + tls_test::Server server(answer_ok); + ASSERT_FALSE(server.failed()); + trust("this is not a certificate\n"); + + auto r = get(server); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("cannot parse the CA certificate bundle"), std::string::npos) + << r.statusText; +} + +TEST_F(TlsVerifyTest, VerificationCanStillBeTurnedOff) { + tls_test::Server server(answer_ok, tls_test::kExpiredCertPem, tls_test::kExpiredKeyPem); + ASSERT_FALSE(server.failed()); + trust(tls_test::kWrongNameCertPem); + + https::HttpClientConfig cfg; + cfg.verifySsl = false; + cfg.connectTimeoutMs = 4000; + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = server.url("/"); + EXPECT_EQ(client.send(req).statusCode, 200); +} diff --git a/tests/tls_test_server.hpp b/tests/tls_test_server.hpp index 4170680..981822e 100644 --- a/tests/tls_test_server.hpp +++ b/tests/tls_test_server.hpp @@ -109,6 +109,51 @@ VQKaSq3Gef7/KLR3YARmm5A= -----END PRIVATE KEY----- )PEM"; +// Two more self-signed certificates, each wrong in one way, for the tests that +// check the client refuses them (test_tls_verify.cpp). Same terms as above: +// public keys, no secrets. +// +// kExpiredCertPem is CN/SAN localhost and 127.0.0.1, valid 2020-01-01 to 2020-01-02. +inline constexpr const char* kExpiredCertPem = R"PEM(-----BEGIN CERTIFICATE----- +MIIBeDCCAR6gAwIBAgIUQR4qNY0KJQYcYEO7585+Fy5Q/U8wCgYIKoZIzj0EAwIw +FDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTIwMDEwMTAwMDAwMFoXDTIwMDEwMjAw +MDAwMFowFDESMBAGA1UEAwwJbG9jYWxob3N0MFkwEwYHKoZIzj0CAQYIKoZIzj0D +AQcDQgAEEb2pNZ4XE7aYNPdWF5+/h+6Ug+KoqKMnRQDgr4OfRvZPIH9NROFnEXnK +stqlISOxthgAmIkR1W3pLSj2BooVT6NOMEwwGgYDVR0RBBMwEYIJbG9jYWxob3N0 +hwR/AAABMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNSI5hk66wZ6dXb/N5+v +6H3CXr5+MAoGCCqGSM49BAMCA0gAMEUCIQD48yY+f7UE0URCJghr3Q+58od8BkxG +zvn+vOvdRjKNZwIgdfrfRWxHrQS/mfyZyzQwTWHHa/TXLD5Z+vlFNjSuAIk= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kExpiredKeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgP8T1y7ZRNYINaXf1 ++ciH6e0/avbCUNliO/brgmagw/OhRANCAAQRvak1nhcTtpg091YXn7+H7pSD4qio +oydFAOCvg59G9k8gf01E4WcRecqy2qUhI7G2GACYiRHVbektKPYGihVP +-----END PRIVATE KEY----- +)PEM"; + +// kWrongNameCertPem is valid for a century but names only wrong.example. +inline constexpr const char* kWrongNameCertPem = R"PEM(-----BEGIN CERTIFICATE----- +MIIBgTCCASagAwIBAgIUUaQJz12pEvdBKiHf8zQ3agib668wCgYIKoZIzj0EAwIw +GDEWMBQGA1UEAwwNd3JvbmcuZXhhbXBsZTAgFw0yNjEwMDEwMzU5NTZaGA8yMTI2 +MDkwNzAzNTk1NlowGDEWMBQGA1UEAwwNd3JvbmcuZXhhbXBsZTBZMBMGByqGSM49 +AgEGCCqGSM49AwEHA0IABAIy/jbI/tnfJmp4lDT9WtuvNNu3CapWaAAsXrXPBYVg +R5MyHwQhLGq9Uy+ZC9dpJI6R1+9b5NUCh5RfaOE4zYOjTDBKMBgGA1UdEQQRMA+C +DXdyb25nLmV4YW1wbGUwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUFN3zbGMq +Zu37qSve4CdZ6qXXSsAwCgYIKoZIzj0EAwIDSQAwRgIhAO/UR5PxHU2spE47HbQf +4qPyonqIlmGB1kp129etm0dYAiEAsjTvx61zqcTrES7R+zQOYykJ67PVTqqGkxIP +uv23Y9g= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kWrongNameKeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgdB0xMbmMhgDdvjON +ESmHq2CHTvE0SbWMP8P4Kt/GWpqhRANCAAQCMv42yP7Z3yZqeJQ0/VrbrzTbtwmq +VmgALF61zwWFYEeTMh8EISxqvVMvmQvXaSSOkdfvW+TVAoeUX2jhOM2D +-----END PRIVATE KEY----- +)PEM"; + class Server; // One accepted connection, after its handshake. Handed to the test's handler. @@ -234,7 +279,10 @@ class Server { // Return true to serve another request on the same connection. using Handler = std::function; - explicit Server(Handler handler) : handler_(std::move(handler)) { + // Serves `certPem`/`keyPem`, the default pair unless a test needs another. + explicit Server(Handler handler, const char* certPem = kCertPem, + const char* keyPem = kKeyPem) + : handler_(std::move(handler)), certPem_(certPem), keyPem_(keyPem) { mbedtls_net_init(&listener_); // Port 0: the OS picks a free one, which is what lets tests run // concurrently and on a machine where anything might already be bound. @@ -335,11 +383,11 @@ class Server { if (mbedtls_ctr_drbg_seed(&drbg, mbedtls_entropy_func, &entropy, nullptr, 0) != 0 || mbedtls_x509_crt_parse(&cert, - reinterpret_cast(kCertPem), - std::strlen(kCertPem) + 1) != 0 + reinterpret_cast(certPem_), + std::strlen(certPem_) + 1) != 0 || mbedtls_pk_parse_key(&key, - reinterpret_cast(kKeyPem), - std::strlen(kKeyPem) + 1, nullptr, 0, + reinterpret_cast(keyPem_), + std::strlen(keyPem_) + 1, nullptr, 0, mbedtls_ctr_drbg_random, &drbg) != 0 || mbedtls_ssl_config_defaults(&conf, MBEDTLS_SSL_IS_SERVER, MBEDTLS_SSL_TRANSPORT_STREAM, MBEDTLS_SSL_PRESET_DEFAULT) != 0) { @@ -379,6 +427,8 @@ class Server { } Handler handler_; + const char* certPem_; + const char* keyPem_; mbedtls_net_context listener_ {}; int port_ { 0 }; bool failed_ { false }; From 08d85404d7e83958653765970e0f9fa9b46fc31b Mon Sep 17 00:00:00 2001 From: Cloud_Yun Date: Thu, 1 Oct 2026 13:06:21 +0900 Subject: [PATCH 04/15] feat(proxy): authenticate to HTTP proxies and report why a tunnel was refused parse_proxy_url split the URL at the first colon, so http://user:pass@proxy:3128 came out as host "user" and a port made of the digits that followed. The scheme was dropped as well, which meant an https:// proxy URL was spoken to as plain HTTP. And whatever the proxy answered to CONNECT, the caller saw "Connection failed". The URL is now parsed as scheme, userinfo, host and port. Credentials are percent-decoded and sent as a Basic Proxy-Authorization header. IPv6 hosts in brackets work, and a port outside 1..65535 is 0 instead of a truncation. ProxyConfig gains scheme, hasCredentials, user and password; host and port keep their meaning. A refused tunnel is reported in statusText: a 407 says whether credentials were sent and rejected or were needed and missing, and any other non-2xx carries the proxy's status line. Any 2xx opens the tunnel, where only 200 did. A scheme other than http is an error and is no longer read as HTTP. proxy_tunnel returns the tunnel together with the reason it could not be made. proxy_connect keeps its signature and calls it. Tests run the client against an in-process proxy and the existing TLS test server, and check both the result and what the proxy was sent. --- CHANGELOG.md | 29 ++++ README.md | 21 ++- src/http.cppm | 23 +-- src/proxy.cppm | 283 ++++++++++++++++++++++++------------ tests/proxy_test_server.hpp | 209 ++++++++++++++++++++++++++ tests/test_proxy.cpp | 280 +++++++++++++++++++++++++++++++++++ 6 files changed, 742 insertions(+), 103 deletions(-) create mode 100644 tests/proxy_test_server.hpp create mode 100644 tests/test_proxy.cpp diff --git a/CHANGELOG.md b/CHANGELOG.md index e3d5792..d32eac8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,34 @@ # Changelog +## Unreleased + +Proxy URLs carry credentials, and a proxy that refuses the tunnel says why. + +`parse_proxy_url` split a URL at the first colon, so `user:pass@host:3128` came +out as host `user` and a port made of the digits of `pass@host:3128`. It also +dropped the scheme, which meant `https://proxy:8443` was spoken to as plain +HTTP. And every refusal, whatever the proxy said, reached the caller as +`Connection failed`. + +* `user:password@` in the proxy URL is percent-decoded and sent as a Basic + `Proxy-Authorization` header on the `CONNECT`. Brackets around an IPv6 host + are understood, and a port that is not 1..65535 is 0 rather than a truncation. +* `ProxyConfig` gains `scheme`, `hasCredentials`, `user` and `password`. + `parse_proxy_url` still returns the same type, and `host` and `port` keep + their meaning. +* A refused tunnel is reported in `statusText`: `proxy rejected the credentials: + 407 Proxy Authentication Required` when credentials were sent, + `proxy requires credentials: ...` when none were, `proxy refused CONNECT: ...` + for any other status. Any 2xx answer opens the tunnel; before, only 200 did. +* A scheme other than `http` is an error (`proxy: unsupported scheme + 'socks4'`) where it used to be read as HTTP. That also ends a habit that + happened to work: a proxy that answers HTTP and SOCKS on one port could be + used by writing `socks5://` or `https://` in front of it, because the scheme + was ignored. +* New `proxy_tunnel` returns the tunnel together with the reason it could not + be made. `proxy_connect` keeps its signature and calls it, so it also accepts + any 2xx answer now. + ## 0.3.2 `download_to_file` no longer reports success for a file the disk did not keep. diff --git a/README.md b/README.md index 0ebe1bb..a9c4209 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Minimal C++23 HTTP/HTTPS client library with SSE (Server-Sent Events) streaming - HTTP/HTTPS client with connection pooling (keep-alive) - SSE (Server-Sent Events) streaming - Streaming downloads to a file, with progress and cancellation -- Proxy support (HTTP CONNECT) +- Proxy support (HTTP CONNECT, with Basic authentication) - C++23 modules ## Usage @@ -57,6 +57,25 @@ means exactly what it did before. | `maxRedirects` | 10 | 0 disables redirect following | | `maxResponseBodyBytes` | 64 MiB | the most `send()` will hold in memory; does not bound `download_to_file` or `send_stream` | | `retryOnStaleConnection` | true | resend once when a pooled connection turns out to have been closed while idle | +| `proxy` | none | proxy URL, see below | + +### Proxies + +Set `proxy` to a URL and every request is tunnelled through it with `CONNECT`: + +```cpp +cfg.proxy = "http://user:pass@proxy.example:3128"; +``` + +The scheme is `http`, and a bare `host:port` means the same. Credentials are +read from the URL and sent as Basic `Proxy-Authorization`; percent-escape any +character that is special in a URL (`p%40ss` for `p@ss`). The port defaults to +8080. + +When the proxy refuses the tunnel, `statusCode` is 0 and `statusText` carries +its answer: `proxy rejected the credentials: 407 Proxy Authentication Required`, +`proxy refused CONNECT: 403 Forbidden`. Any other scheme (`socks4://`, say) +fails the same way instead of being read as HTTP. ## Project templates diff --git a/src/http.cppm b/src/http.cppm index 5ef3e74..73d6b29 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -997,14 +997,18 @@ private: return reqStr; } - bool open_connection(TlsSocket& sock, const ParsedUrl& parsed) { + // False on failure, with `error` set when there is more to say than that the + // connection failed: a proxy that refused the tunnel says so in its own words. + bool open_connection(TlsSocket& sock, const ParsedUrl& parsed, std::string& error) { if (config_.proxy.has_value()) { - auto proxyConf = parse_proxy_url(config_.proxy.value()); - auto tunnel = proxy_connect(proxyConf.host, proxyConf.port, - parsed.host, parsed.port, - config_.connectTimeoutMs); - if (!tunnel.is_valid()) return false; - return sock.connect_over(std::move(tunnel), parsed.host.c_str(), + auto tunnel = proxy_tunnel(parse_proxy_url(config_.proxy.value()), + parsed.host, parsed.port, + config_.connectTimeoutMs); + if (!tunnel.ok()) { + error = std::move(tunnel.error); + return false; + } + return sock.connect_over(std::move(tunnel.socket), parsed.host.c_str(), config_.verifySsl); } return sock.connect(parsed.host.c_str(), parsed.port, @@ -1033,9 +1037,10 @@ private: if (it != pool_.end()) pool_.erase(it); auto [inserted, ok] = pool_.emplace(poolKey, TlsSocket{}); sock = &inserted->second; - if (!open_connection(*sock, parsed)) { + std::string openError; + if (!open_connection(*sock, parsed, openError)) { guard.drop(); - return { nullptr, {}, "Connection failed" }; + return { nullptr, {}, openError.empty() ? "Connection failed" : openError }; } } diff --git a/src/proxy.cppm b/src/proxy.cppm index e102de6..f5b2bc2 100644 --- a/src/proxy.cppm +++ b/src/proxy.cppm @@ -5,151 +5,248 @@ import std; namespace mcpplibs::tinyhttps { +// A parsed proxy URL: where the proxy is, how to talk to it, and the credentials +// to send. `parse_proxy_url` fills it in; nothing validates it until +// `proxy_tunnel` tries to use it. export struct ProxyConfig { std::string host; int port { 8080 }; + + // Lower case. "http" when the URL has no scheme, which is how a bare + // `host:port` has always been read. + std::string scheme { "http" }; + + // From `user:password@` in the URL, percent-decoded. A URL with `@` and no + // password still has credentials; `hasCredentials` is what says so, because + // an empty user or an empty password is a legitimate value. + bool hasCredentials { false }; + std::string user; + std::string password; }; -// Parse "http://host:port" proxy URL +static int hex_digit(char c) { + if (c >= '0' && c <= '9') return c - '0'; + if (c >= 'a' && c <= 'f') return c - 'a' + 10; + if (c >= 'A' && c <= 'F') return c - 'A' + 10; + return -1; +} + +// "%40" is "@". A '%' that is not followed by two hex digits stays as written. +static std::string percent_decode(std::string_view in) { + std::string out; + out.reserve(in.size()); + for (std::size_t i = 0; i < in.size(); ++i) { + if (in[i] == '%' && i + 2 < in.size() && hex_digit(in[i + 1]) >= 0 + && hex_digit(in[i + 2]) >= 0) { + out += static_cast(hex_digit(in[i + 1]) * 16 + hex_digit(in[i + 2])); + i += 2; + } else { + out += in[i]; + } + } + return out; +} + +// Parse "[scheme://][user[:password]@]host[:port]". A path, query or fragment +// after the authority is ignored. IPv6 literals are written in brackets. export ProxyConfig parse_proxy_url(std::string_view url) { ProxyConfig config; - // Strip scheme if present auto schemeEnd = url.find("://"); if (schemeEnd != std::string_view::npos) { + config.scheme.clear(); + for (char c : url.substr(0, schemeEnd)) { + config.scheme += (c >= 'A' && c <= 'Z') ? static_cast(c + 32) : c; + } url = url.substr(schemeEnd + 3); } - // Strip trailing path if present - auto pathStart = url.find('/'); - if (pathStart != std::string_view::npos) { - url = url.substr(0, pathStart); + url = url.substr(0, url.find_first_of("/?#")); + + // The last '@' separates the userinfo, so an unencoded '@' in a password + // still parses the way curl reads it. + if (auto at = url.rfind('@'); at != std::string_view::npos) { + auto userinfo = url.substr(0, at); + url = url.substr(at + 1); + auto colon = userinfo.find(':'); + config.hasCredentials = true; + config.user = percent_decode(userinfo.substr(0, colon)); + if (colon != std::string_view::npos) { + config.password = percent_decode(userinfo.substr(colon + 1)); + } } - // Check for port - auto colonPos = url.find(':'); - if (colonPos != std::string_view::npos) { - config.host = std::string(url.substr(0, colonPos)); - auto portStr = url.substr(colonPos + 1); - config.port = 0; - for (char c : portStr) { - if (c >= '0' && c <= '9') { - config.port = config.port * 10 + (c - '0'); - } + std::string_view portStr; + bool hasPort = false; + if (!url.empty() && url.front() == '[') { + auto close = url.find(']'); + config.host = std::string(url.substr(1, close == std::string_view::npos + ? std::string_view::npos : close - 1)); + if (close != std::string_view::npos && close + 1 < url.size() && url[close + 1] == ':') { + portStr = url.substr(close + 2); + hasPort = true; } + } else if (auto colon = url.find(':'); colon != std::string_view::npos) { + config.host = std::string(url.substr(0, colon)); + portStr = url.substr(colon + 1); + hasPort = true; } else { config.host = std::string(url); - config.port = 8080; + } + + if (hasPort && !portStr.empty()) { + // Anything but 1..65535 becomes 0, which no connect will accept. + long port = 0; + for (char c : portStr) { + if (c < '0' || c > '9' || (port = port * 10 + (c - '0')) > 65535) { port = 0; break; } + } + config.port = static_cast(port); } return config; } -// Read a line (ending with \r\n) from a plain Socket -static std::string read_line_plain(Socket& sock, int timeoutMs) { +static std::string base64_encode(std::string_view in) { + constexpr std::string_view table = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + std::string out; + for (std::size_t i = 0; i < in.size(); i += 3) { + unsigned v = static_cast(in[i]) << 16; + if (i + 1 < in.size()) v |= static_cast(in[i + 1]) << 8; + if (i + 2 < in.size()) v |= static_cast(in[i + 2]); + out += table[(v >> 18) & 63]; + out += table[(v >> 12) & 63]; + out += i + 1 < in.size() ? table[(v >> 6) & 63] : '='; + out += i + 2 < in.size() ? table[v & 63] : '='; + } + return out; +} + +// One line without its CRLF. nullopt on a timeout, an end of stream, or a line +// longer than any proxy sends. Reads a byte at a time on purpose: whatever +// follows the blank line belongs to the tunnel and must not be consumed. +template +static std::optional read_line(Stream& sock, int timeoutMs) { std::string line; char c; - while (true) { - if (!sock.wait_readable(timeoutMs)) { - break; - } + while (line.size() < 8192) { + if (!sock.wait_readable(timeoutMs)) return std::nullopt; int ret = sock.read(&c, 1); - if (ret < 0) break; if (ret == 0) { - if (!sock.wait_readable(timeoutMs)) break; + if (!sock.wait_readable(timeoutMs)) return std::nullopt; ret = sock.read(&c, 1); - if (ret <= 0) break; } + if (ret <= 0) return std::nullopt; line += c; - if (line.size() >= 2 && line[line.size() - 2] == '\r' && line[line.size() - 1] == '\n') { + if (line.size() >= 2 && line[line.size() - 2] == '\r' && line.back() == '\n') { line.resize(line.size() - 2); - break; + return line; } } - return line; + return std::nullopt; } -// Write all data to a plain Socket -static bool write_all_plain(Socket& sock, const std::string& data) { - int total = 0; - int len = static_cast(data.size()); - while (total < len) { - int ret = sock.write(data.c_str() + total, len - total); +template +static bool write_all(Stream& sock, std::string_view data, int timeoutMs) { + std::size_t total = 0; + while (total < data.size()) { + int ret = sock.write(data.data() + total, static_cast(data.size() - total)); if (ret < 0) return false; - if (ret == 0) { - ret = sock.write(data.c_str() + total, len - total); - if (ret <= 0) return false; - } - total += ret; + if (ret == 0 && !sock.wait_writable(timeoutMs)) return false; + total += static_cast(ret); } return true; } -// Connect through HTTP CONNECT proxy, returning a Socket connected to target through tunnel. -// On failure, returns an invalid (closed) Socket. -export Socket proxy_connect(std::string_view proxyHost, int proxyPort, - std::string_view targetHost, int targetPort, - int timeoutMs) { - Socket sock; - - // Step 1: Connect to proxy - std::string proxyHostStr(proxyHost); - if (!sock.connect(proxyHostStr.c_str(), proxyPort, timeoutMs)) { - return sock; +// "HTTP/1.1 407 Proxy Authentication Required" -> 407, or 0 when the line is +// not a status line. +static int status_code_of(std::string_view line) { + if (!line.starts_with("HTTP/1.") || line.size() < 12 || line[8] != ' ') return 0; + int code = 0; + for (char c : line.substr(9, 3)) { + if (c < '0' || c > '9') return 0; + code = code * 10 + (c - '0'); } + return code; +} - // Step 2: Send CONNECT request - std::string request = "CONNECT "; - request += targetHost; - request += ":"; - request += std::to_string(targetPort); - request += " HTTP/1.1\r\nHost: "; - request += targetHost; - request += ":"; - request += std::to_string(targetPort); - request += "\r\n\r\n"; - - if (!write_all_plain(sock, request)) { - sock.close(); - return sock; +// Ask an HTTP proxy to open a tunnel to host:port. Returns an error message, or +// an empty string once the proxy has answered 2xx and the stream is the tunnel. +template +static std::string http_connect(Stream& sock, const ProxyConfig& proxy, + std::string_view host, int port, int timeoutMs) { + // An IPv6 literal needs its brackets back in the request target. + std::string authority = host.contains(':') ? "[" + std::string(host) + "]" : std::string(host); + authority += ":" + std::to_string(port); + + std::string request = "CONNECT " + authority + " HTTP/1.1\r\nHost: " + authority + "\r\n"; + if (proxy.hasCredentials) { + request += "Proxy-Authorization: Basic " + + base64_encode(proxy.user + ":" + proxy.password) + "\r\n"; } + request += "\r\n"; + + if (!write_all(sock, request, timeoutMs)) return "proxy: could not send CONNECT"; - // Step 3: Read response status line - std::string statusLine = read_line_plain(sock, timeoutMs); - if (statusLine.empty()) { - sock.close(); - return sock; + auto statusLine = read_line(sock, timeoutMs); + int code = statusLine ? status_code_of(*statusLine) : 0; + if (code == 0) return "proxy: no HTTP response to CONNECT"; + if (code == 407) { + return std::string(proxy.hasCredentials ? "proxy rejected the credentials: " + : "proxy requires credentials: ") + + statusLine->substr(9); } + if (code < 200 || code > 299) return "proxy refused CONNECT: " + statusLine->substr(9); - // Parse status code from "HTTP/1.x 200 ..." - int statusCode = 0; - auto spacePos = statusLine.find(' '); - if (spacePos != std::string::npos) { - auto rest = std::string_view(statusLine).substr(spacePos + 1); - for (char c : rest) { - if (c >= '0' && c <= '9') { - statusCode = statusCode * 10 + (c - '0'); - } else { - break; - } - } + // The rest of the head. A 2xx answer to CONNECT carries no body. + for (int i = 0; i < 100; ++i) { + auto header = read_line(sock, timeoutMs); + if (!header) return "proxy: response to CONNECT was cut short"; + if (header->empty()) return {}; } + return "proxy: response to CONNECT has too many headers"; +} + +// A connection to the target through a proxy. `error` is empty exactly when +// `socket` is the tunnel; otherwise it says what the proxy did, in words fit +// for a caller to log. +export struct ProxyTunnel { + Socket socket; + std::string error; - if (statusCode != 200) { - sock.close(); - return sock; + [[nodiscard]] bool ok() const { return error.empty() && socket.is_valid(); } +}; + +export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, + std::string_view targetHost, int targetPort, + int timeoutMs) { + ProxyTunnel tunnel; + + if (proxy.scheme != "http") { + tunnel.error = "proxy: unsupported scheme '" + proxy.scheme + "'"; + return tunnel; } - // Step 4: Read remaining response headers until empty line - while (true) { - std::string headerLine = read_line_plain(sock, timeoutMs); - if (headerLine.empty()) { - break; - } + if (!tunnel.socket.connect(proxy.host.c_str(), proxy.port, timeoutMs)) { + tunnel.error = "proxy: could not connect to " + proxy.host + ":" + std::to_string(proxy.port); + return tunnel; } - // Socket is now tunneled to the target - return sock; + tunnel.error = http_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs); + if (!tunnel.error.empty()) tunnel.socket.close(); + return tunnel; +} + +// The original entry point, kept for callers outside this repository. It cannot +// say why it failed; `proxy_tunnel` can. +export Socket proxy_connect(std::string_view proxyHost, int proxyPort, + std::string_view targetHost, int targetPort, + int timeoutMs) { + ProxyConfig proxy; + proxy.host = std::string(proxyHost); + proxy.port = proxyPort; + auto tunnel = proxy_tunnel(proxy, targetHost, targetPort, timeoutMs); + return std::move(tunnel.socket); } } // namespace mcpplibs::tinyhttps diff --git a/tests/proxy_test_server.hpp b/tests/proxy_test_server.hpp new file mode 100644 index 0000000..3adaf6e --- /dev/null +++ b/tests/proxy_test_server.hpp @@ -0,0 +1,209 @@ +// A proxy, in this process, that does exactly what a test tells it to. +// +// The proxy tests need the three things a real proxy does between accepting a +// connection and relaying bytes: read what the client asked for, answer it, and +// then splice the client to the target. A handler scripts the first two and +// `Peer::relay_to` does the third, so one test can make the proxy refuse, stall +// or hang up, and another can let a request through to the TLS server from +// tls_test_server.hpp and check what arrived. +// +// Plain TCP on 127.0.0.1, port chosen by the OS. It uses mbedtls's net layer +// for the same reason tls_test_server.hpp does: it is already a dependency and +// it is the same on every platform the library builds for. +#pragma once + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// Which socket interface this is built against; see socket.cppm. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#define TINYHTTPS_WINSOCK 1 +#endif + +#ifdef TINYHTTPS_WINSOCK +#include +#include +#else +#include +#include +#endif + +namespace proxy_test { + +class Server; + +// The accepted connection, as the handler sees it. +class Peer { +public: + Peer(Server& server, mbedtls_net_context net) : server_(server), net_(net) {} + ~Peer() { mbedtls_net_free(&net_); } + Peer(const Peer&) = delete; + Peer& operator=(const Peer&) = delete; + + // Everything up to and including the blank line that ends an HTTP head, + // read a byte at a time so nothing past it is taken. Empty on a hang-up. + std::string read_head(); + + bool write(std::string_view data) { + std::size_t sent = 0; + while (sent < data.size()) { + int ret = mbedtls_net_send(&net_, + reinterpret_cast(data.data() + sent), + data.size() - sent); + if (ret <= 0) return false; + sent += static_cast(ret); + } + return true; + } + + void note(std::string text); + + // Splices this connection to 127.0.0.1:`port` until either side ends. + void relay_to(int port); + +private: + bool wait_readable(int ms) { return mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, ms) > 0; } + + Server& server_; + mbedtls_net_context net_; +}; + +class Server { +public: + using Handler = std::function; + + explicit Server(Handler handler) : handler_(std::move(handler)) { + mbedtls_net_init(&listener_); + if (mbedtls_net_bind(&listener_, "127.0.0.1", "0", MBEDTLS_NET_PROTO_TCP) != 0) { + failed_ = true; + return; + } + sockaddr_in addr {}; + socklen_t len = sizeof addr; + if (::getsockname(listener_.fd, reinterpret_cast(&addr), &len) != 0) { + failed_ = true; + return; + } + port_ = ntohs(addr.sin_port); + acceptor_ = std::thread([this] { accept_loop(); }); + } + + ~Server() { stop(); } + Server(const Server&) = delete; + Server& operator=(const Server&) = delete; + + void stop() { + if (stopping_.exchange(true)) return; + if (acceptor_.joinable()) acceptor_.join(); + for (auto& t : connections_) { + if (t.joinable()) t.join(); + } + connections_.clear(); + mbedtls_net_free(&listener_); + } + + [[nodiscard]] bool failed() const { return failed_; } + [[nodiscard]] int port() const { return port_; } + [[nodiscard]] bool stopping() const { return stopping_.load(); } + [[nodiscard]] int accepts() const { return accepts_.load(); } + + // The text a handler stored with `note`, for the test to assert on once the + // request has completed. + void note(std::string text) { + std::lock_guard lock(mutex_); + notes_.push_back(std::move(text)); + } + [[nodiscard]] std::vector notes() { + std::lock_guard lock(mutex_); + return notes_; + } + +private: + void accept_loop() { + while (!stopping_.load()) { + int ready = mbedtls_net_poll(&listener_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) break; + if (ready == 0) continue; + + mbedtls_net_context client {}; + mbedtls_net_init(&client); + if (mbedtls_net_accept(&listener_, &client, nullptr, 0, nullptr) != 0) { + mbedtls_net_free(&client); + continue; + } + accepts_.fetch_add(1); + connections_.emplace_back([this, client] { + Peer peer(*this, client); + handler_(peer); + }); + } + } + + Handler handler_; + mbedtls_net_context listener_ {}; + int port_ { 0 }; + bool failed_ { false }; + std::atomic stopping_ { false }; + std::atomic accepts_ { 0 }; + std::thread acceptor_; + std::vector connections_; + std::mutex mutex_; + std::vector notes_; +}; + +inline void Peer::note(std::string text) { server_.note(std::move(text)); } + +inline std::string Peer::read_head() { + std::string head; + unsigned char c {}; + while (head.find("\r\n\r\n") == std::string::npos) { + if (head.size() > 65536) return {}; + // 20 s is far beyond any test; the loop exists so `stop` is noticed. + bool ready = false; + for (int waited = 0; waited < 20000 && !ready; waited += 50) { + if (server_.stopping()) return {}; + ready = wait_readable(50); + } + if (!ready) return {}; + if (mbedtls_net_recv(&net_, &c, 1) <= 0) return {}; + head.push_back(static_cast(c)); + } + return head; +} + +inline void Peer::relay_to(int port) { + mbedtls_net_context target {}; + mbedtls_net_init(&target); + if (mbedtls_net_connect(&target, "127.0.0.1", std::to_string(port).c_str(), + MBEDTLS_NET_PROTO_TCP) != 0) { + return; + } + // Round-robin over two sockets with a short poll each: slower than a + // select over both, and portable without a second code path. + unsigned char buf[4096]; + bool open = true; + while (open && !server_.stopping()) { + for (auto [from, to] : {std::pair{&net_, &target}, std::pair{&target, &net_}}) { + if (mbedtls_net_poll(from, MBEDTLS_NET_POLL_READ, 10) <= 0) continue; + int n = mbedtls_net_recv(from, buf, sizeof buf); + if (n <= 0) { open = false; break; } + for (int sent = 0; sent < n;) { + int w = mbedtls_net_send(to, buf + sent, static_cast(n - sent)); + if (w <= 0) { open = false; break; } + sent += w; + } + } + } + mbedtls_net_free(&target); +} + +} // namespace proxy_test diff --git a/tests/test_proxy.cpp b/tests/test_proxy.cpp new file mode 100644 index 0000000..8c38e48 --- /dev/null +++ b/tests/test_proxy.cpp @@ -0,0 +1,280 @@ +// What the client does when it is told to go through a proxy. +// +// Each test stands up a proxy that behaves as the test scripts it and the TLS +// server from tls_test_server.hpp as the target, and asserts both what the +// caller got and what the proxy was sent. The second is the half a mock-free +// test cannot see: whether the CONNECT line named the right target, and what +// the Proxy-Authorization header said. +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +namespace { + +class ProxyTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +https::HttpClientConfig proxied(const std::string& proxyUrl) { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + cfg.proxy = proxyUrl; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.url = url; + return req; +} + +tls_test::Server hello_server() { + return tls_test::Server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("hello")); + return true; + }); +} + +// The value of a header in a request head, or "" when absent. +std::string header_of(const std::string& head, std::string_view name) { + std::string lower, key = std::string(name) + ":"; + for (char c : head) lower += static_cast(std::tolower(static_cast(c))); + for (char& c : key) c = static_cast(std::tolower(static_cast(c))); + auto pos = lower.find(key); + if (pos == std::string::npos) return {}; + pos += key.size(); + while (pos < head.size() && head[pos] == ' ') ++pos; + return head.substr(pos, head.find("\r\n", pos) - pos); +} + +constexpr std::string_view kEstablished = "HTTP/1.1 200 Connection established\r\n\r\n"; + +// A CONNECT proxy. With `expectAuth` set it answers 407 unless the request +// carries exactly that Proxy-Authorization value. Either way it records the +// request head, so the test can read back what the client sent. +proxy_test::Server::Handler connect_handler(int targetPort, std::string expectAuth = {}) { + return [=](proxy_test::Peer& peer) { + auto head = peer.read_head(); + peer.note(head); + if (head.empty()) return; + if (!expectAuth.empty() && header_of(head, "Proxy-Authorization") != expectAuth) { + peer.write("HTTP/1.1 407 Proxy Authentication Required\r\n" + "Proxy-Authenticate: Basic realm=\"test\"\r\n" + "Content-Length: 0\r\n\r\n"); + return; + } + peer.write(kEstablished); + peer.relay_to(targetPort); + }; +} + +std::string proxy_url(const proxy_test::Server& proxy, std::string_view userinfo = {}) { + std::string url = "http://"; + if (!userinfo.empty()) { url += userinfo; url += "@"; } + return url + "127.0.0.1:" + std::to_string(proxy.port()); +} + +} // namespace + +// ── parse_proxy_url ────────────────────────────────────────────────────────── + +TEST(ProxyUrl, BareHostAndPortIsAnHttpProxy) { + auto p = https::parse_proxy_url("proxy.example:3128"); + EXPECT_EQ(p.scheme, "http"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); + EXPECT_FALSE(p.hasCredentials); +} + +TEST(ProxyUrl, ThePortDefaultsAsItAlwaysHas) { + EXPECT_EQ(https::parse_proxy_url("http://proxy.example").port, 8080); +} + +TEST(ProxyUrl, APathQueryOrFragmentAfterTheAuthorityIsIgnored) { + auto p = https::parse_proxy_url("http://proxy.example:3128/some/path?x=1#f"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, UserAndPasswordAreSplitAndPercentDecoded) { + auto p = https::parse_proxy_url("http://al%40ice:p%3Ass%20w@proxy.example:3128"); + EXPECT_TRUE(p.hasCredentials); + EXPECT_EQ(p.user, "al@ice"); + EXPECT_EQ(p.password, "p:ss w"); + EXPECT_EQ(p.host, "proxy.example"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, OnlyTheFirstColonSplitsUserFromPassword) { + auto p = https::parse_proxy_url("http://u:a:b@proxy.example:1"); + EXPECT_EQ(p.user, "u"); + EXPECT_EQ(p.password, "a:b"); +} + +TEST(ProxyUrl, AnUnencodedAtSignInThePasswordStaysInThePassword) { + auto p = https::parse_proxy_url("http://u:p@ss@proxy.example:1"); + EXPECT_EQ(p.password, "p@ss"); + EXPECT_EQ(p.host, "proxy.example"); +} + +TEST(ProxyUrl, AUserWithNoPasswordStillHasCredentials) { + auto p = https::parse_proxy_url("http://token@proxy.example:1"); + EXPECT_TRUE(p.hasCredentials); + EXPECT_EQ(p.user, "token"); + EXPECT_EQ(p.password, ""); +} + +TEST(ProxyUrl, ABadPercentEscapeIsLeftAsWritten) { + auto p = https::parse_proxy_url("http://u:50%@proxy.example:1"); + EXPECT_EQ(p.password, "50%"); +} + +TEST(ProxyUrl, AnIpv6LiteralKeepsItsColons) { + auto p = https::parse_proxy_url("http://[::1]:3128"); + EXPECT_EQ(p.host, "::1"); + EXPECT_EQ(p.port, 3128); +} + +TEST(ProxyUrl, TheSchemeIsLowerCased) { + EXPECT_EQ(https::parse_proxy_url("HTTP://proxy.example:1").scheme, "http"); +} + +TEST(ProxyUrl, APortOutOfRangeIsZeroRatherThanTruncated) { + EXPECT_EQ(https::parse_proxy_url("http://proxy.example:99999").port, 0); + EXPECT_EQ(https::parse_proxy_url("http://proxy.example:80x").port, 0); +} + +// ── CONNECT ────────────────────────────────────────────────────────────────── + +TEST_F(ProxyTest, ARequestGoesThroughAProxyThatAsksForNothing) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto notes = proxy.notes(); + ASSERT_EQ(notes.size(), 1u); + EXPECT_TRUE(notes[0].starts_with("CONNECT 127.0.0.1:" + std::to_string(target.port()) + " HTTP/1.1\r\n")); + EXPECT_EQ(header_of(notes[0], "Proxy-Authorization"), ""); +} + +TEST_F(ProxyTest, CredentialsInTheUrlBecomeABasicProxyAuthorization) { + auto target = hello_server(); + // base64("alice:s3cret") + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy, "alice:s3cret"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); +} + +TEST_F(ProxyTest, CredentialsArePercentDecodedBeforeTheyAreEncoded) { + auto target = hello_server(); + // base64("al@ice:p:ss w") + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxAaWNlOnA6c3Mgdw==")); + + https::HttpClient client(proxied(proxy_url(proxy, "al%40ice:p%3Ass%20w"))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); +} + +TEST_F(ProxyTest, WrongCredentialsAreReportedAsThatAndNotAsAFailedConnection) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy, "alice:wrong"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); + EXPECT_NE(resp.statusText.find("407"), std::string::npos); + EXPECT_EQ(target.accepts(), 0); // the tunnel was never opened +} + +TEST_F(ProxyTest, AProxyThatWantsCredentialsWeDidNotGiveSaysSo) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port(), "Basic YWxpY2U6czNjcmV0")); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("requires credentials"), std::string::npos); +} + +TEST_F(ProxyTest, AnyOtherRefusalCarriesTheProxysStatus) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\n\r\n"); + }); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("refused CONNECT: 403 Forbidden"), std::string::npos); +} + +TEST_F(ProxyTest, AProxyThatHangsUpWithoutAnsweringIsAnError) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { peer.read_head(); }); + + https::HttpClient client(proxied(proxy_url(proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("no HTTP response"), std::string::npos); +} + +TEST_F(ProxyTest, AnyTwoHundredSeriesAnswerOpensTheTunnel) { + auto target = hello_server(); + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.0 204 No Content\r\nX-Note: a header or two\r\n\r\n"); + peer.relay_to(target.port()); + }); + + https::HttpClient client(proxied(proxy_url(proxy))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); +} + +TEST_F(ProxyTest, TwoRequestsShareOneTunnel) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + + https::HttpClient client(proxied(proxy_url(proxy))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} + +TEST_F(ProxyTest, ASchemeThisLibraryDoesNotSpeakIsNotQuietlyTreatedAsHttp) { + https::HttpClient client(proxied("socks4://127.0.0.1:9")); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("unsupported scheme 'socks4'"), std::string::npos); +} + +TEST_F(ProxyTest, TheOriginalProxyConnectStillReturnsATunnel) { + auto target = hello_server(); + proxy_test::Server proxy(connect_handler(target.port())); + + auto tunnel = https::proxy_connect("127.0.0.1", proxy.port(), "127.0.0.1", + target.port(), 4000); + EXPECT_TRUE(tunnel.is_valid()); +} From edff1ed82b055760c15a8069e782257609e9c957 Mon Sep 17 00:00:00 2001 From: Cloud_Yun Date: Thu, 1 Oct 2026 13:49:32 +0900 Subject: [PATCH 05/15] feat(proxy): reach an https:// proxy over TLS A proxy URL with the https scheme was spoken to as plain HTTP, which cannot work: such a proxy expects a TLS handshake before the CONNECT. TlsSocket can now run over another TlsSocket. For an https:// proxy the client opens TLS to the proxy, sends the CONNECT inside that session, and then runs the handshake with the target inside the tunnel. The BIO of the upper session is the lower one, and wait_readable asks it rather than the descriptor, since the bytes may be decrypted there while the socket has nothing. The lower session is closed after the upper one, so its close_notify goes out before the descriptor is closed. The proxy connection is set up like the one to the target, verifySsl included. The default port for an https proxy is 443. Credentials in the URL travel inside the proxy session. Tests put the existing TLS test server in front of the target as the proxy, which needs a relay in it, and cover a body larger than a record, keep-alive through the tunnel, SSE, a response that is already decrypted in the proxy session when the client goes to wait for it, and a scheme that does not match the proxy. --- CHANGELOG.md | 7 ++ README.md | 12 ++- src/http.cppm | 6 +- src/proxy.cppm | 56 +++++++---- src/tls.cppm | 87 +++++++++++++---- tests/proxy_test_server.hpp | 15 +++ tests/test_proxy.cpp | 183 ++++++++++++++++++++++++++++++++++++ tests/tls_test_server.hpp | 53 +++++++++++ 8 files changed, 378 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d32eac8..9517061 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,13 @@ HTTP. And every refusal, whatever the proxy said, reached the caller as * New `proxy_tunnel` returns the tunnel together with the reason it could not be made. `proxy_connect` keeps its signature and calls it, so it also accepts any 2xx answer now. +* An `https://` proxy URL is spoken to over TLS: the session to the proxy + carries the `CONNECT`, and the session to the target runs inside the tunnel. + `TlsSocket` can now run over another `TlsSocket` (`connect_over` takes the + lower session), and its `wait_readable` asks that session rather than the + descriptor, because the bytes may already be decrypted there. The proxy + connection is set up like the one to the target, `verifySsl` included. The + default port for `https` is 443. ## 0.3.2 diff --git a/README.md b/README.md index a9c4209..200629d 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Minimal C++23 HTTP/HTTPS client library with SSE (Server-Sent Events) streaming - HTTP/HTTPS client with connection pooling (keep-alive) - SSE (Server-Sent Events) streaming - Streaming downloads to a file, with progress and cancellation -- Proxy support (HTTP CONNECT, with Basic authentication) +- Proxy support (HTTP CONNECT, over plain HTTP or TLS, with Basic authentication) - C++23 modules ## Usage @@ -67,10 +67,12 @@ Set `proxy` to a URL and every request is tunnelled through it with `CONNECT`: cfg.proxy = "http://user:pass@proxy.example:3128"; ``` -The scheme is `http`, and a bare `host:port` means the same. Credentials are -read from the URL and sent as Basic `Proxy-Authorization`; percent-escape any -character that is special in a URL (`p%40ss` for `p@ss`). The port defaults to -8080. +The scheme is `http` (a bare `host:port` means the same) or `https`. An +`https://` proxy is reached over TLS first, the `CONNECT` is sent inside that +session, and the connection to the target is a second TLS session inside the +tunnel; the two connections are set up alike, `verifySsl` included. Credentials +are read from the URL and sent as Basic `Proxy-Authorization`; percent-escape any character that is special in +a URL (`p%40ss` for `p@ss`). The port defaults to 8080, or 443 for `https`. When the proxy refuses the tunnel, `statusCode` is 0 and `statusText` carries its answer: `proxy rejected the credentials: 407 Proxy Authentication Required`, diff --git a/src/http.cppm b/src/http.cppm index 73d6b29..7a46919 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -1003,11 +1003,15 @@ private: if (config_.proxy.has_value()) { auto tunnel = proxy_tunnel(parse_proxy_url(config_.proxy.value()), parsed.host, parsed.port, - config_.connectTimeoutMs); + config_.connectTimeoutMs, config_.verifySsl); if (!tunnel.ok()) { error = std::move(tunnel.error); return false; } + if (tunnel.proxyTls) { + return sock.connect_over(std::move(tunnel.proxyTls), parsed.host.c_str(), + config_.verifySsl); + } return sock.connect_over(std::move(tunnel.socket), parsed.host.c_str(), config_.verifySsl); } diff --git a/src/proxy.cppm b/src/proxy.cppm index f5b2bc2..db7e6c8 100644 --- a/src/proxy.cppm +++ b/src/proxy.cppm @@ -1,6 +1,7 @@ export module mcpplibs.tinyhttps:proxy; import :socket; +import :tls; import std; namespace mcpplibs::tinyhttps { @@ -12,8 +13,9 @@ export struct ProxyConfig { std::string host; int port { 8080 }; - // Lower case. "http" when the URL has no scheme, which is how a bare - // `host:port` has always been read. + // Lower case: "http", or "https" for a proxy that is itself reached over + // TLS. "http" when the URL has no scheme, which is how a bare `host:port` + // has always been read. std::string scheme { "http" }; // From `user:password@` in the URL, percent-decoded. A URL with `@` and no @@ -48,7 +50,8 @@ static std::string percent_decode(std::string_view in) { } // Parse "[scheme://][user[:password]@]host[:port]". A path, query or fragment -// after the authority is ignored. IPv6 literals are written in brackets. +// after the authority is ignored. IPv6 literals are written in brackets. The +// port defaults to 443 for https and to 8080 otherwise. export ProxyConfig parse_proxy_url(std::string_view url) { ProxyConfig config; @@ -94,6 +97,8 @@ export ProxyConfig parse_proxy_url(std::string_view url) { config.host = std::string(url); } + if (config.scheme == "https") config.port = 443; + if (hasPort && !portStr.empty()) { // Anything but 1..65535 becomes 0, which no connect will accept. long port = 0; @@ -207,33 +212,48 @@ static std::string http_connect(Stream& sock, const ProxyConfig& proxy, return "proxy: response to CONNECT has too many headers"; } -// A connection to the target through a proxy. `error` is empty exactly when -// `socket` is the tunnel; otherwise it says what the proxy did, in words fit -// for a caller to log. +// A connection to the target through a proxy. `error` is empty exactly when the +// tunnel is open; otherwise it says what the proxy did, in words fit for a +// caller to log. +// +// The tunnel is `socket`, except for an https:// proxy, where it is the TLS +// session to the proxy, `proxyTls`, and `socket` is unused. export struct ProxyTunnel { Socket socket; + std::unique_ptr proxyTls; std::string error; - [[nodiscard]] bool ok() const { return error.empty() && socket.is_valid(); } + [[nodiscard]] bool ok() const { + return error.empty() && (proxyTls ? proxyTls->is_valid() : socket.is_valid()); + } }; +// `verifySsl` is for the connection to an https:// proxy, and is the same +// setting that governs the connection to the target. export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, std::string_view targetHost, int targetPort, - int timeoutMs) { + int timeoutMs, bool verifySsl = true) { ProxyTunnel tunnel; + const std::string where = proxy.host + ":" + std::to_string(proxy.port); - if (proxy.scheme != "http") { + if (proxy.scheme == "http") { + if (!tunnel.socket.connect(proxy.host.c_str(), proxy.port, timeoutMs)) { + tunnel.error = "proxy: could not connect to " + where; + return tunnel; + } + tunnel.error = http_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs); + if (!tunnel.error.empty()) tunnel.socket.close(); + } else if (proxy.scheme == "https") { + auto tls = std::make_unique(); + if (!tls->connect(proxy.host.c_str(), proxy.port, timeoutMs, verifySsl)) { + tunnel.error = "proxy: could not open a TLS connection to " + where; + return tunnel; + } + tunnel.error = http_connect(*tls, proxy, targetHost, targetPort, timeoutMs); + if (tunnel.error.empty()) tunnel.proxyTls = std::move(tls); + } else { tunnel.error = "proxy: unsupported scheme '" + proxy.scheme + "'"; - return tunnel; - } - - if (!tunnel.socket.connect(proxy.host.c_str(), proxy.port, timeoutMs)) { - tunnel.error = "proxy: could not connect to " + proxy.host + ":" + std::to_string(proxy.port); - return tunnel; } - - tunnel.error = http_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs); - if (!tunnel.error.empty()) tunnel.socket.close(); return tunnel; } diff --git a/src/tls.cppm b/src/tls.cppm index 39494ca..21d5cdb 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -108,11 +108,10 @@ public: // Move constructor TlsSocket(TlsSocket&& other) noexcept : socket_(std::move(other.socket_)) + , lower_(std::move(other.lower_)) , state_(std::move(other.state_)) { // Re-bind BIO to point to our socket_ (not the moved-from one) - if (state_) { - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); - } + bind_bio(); } // Move assignment @@ -120,17 +119,16 @@ public: if (this != &other) { close(); socket_ = std::move(other.socket_); + lower_ = std::move(other.lower_); state_ = std::move(other.state_); // Re-bind BIO to point to our socket_ - if (state_) { - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); - } + bind_bio(); } return *this; } [[nodiscard]] bool is_valid() const { - return state_ != nullptr && socket_.is_valid(); + return state_ != nullptr && (lower_ ? lower_->is_valid() : socket_.is_valid()); } // Connect over an already-established Socket (e.g. a proxy tunnel). @@ -140,6 +138,15 @@ public: return setup_tls(host, verifySsl); } + // Run the handshake inside another TLS session, which is how a client + // reaches a target through an https:// proxy: TLS to the proxy, CONNECT + // inside it, then this session to the target inside the tunnel. Takes + // ownership of `lower`, which must already be past its CONNECT. + bool connect_over(std::unique_ptr lower, const char* host, bool verifySsl) { + lower_ = std::move(lower); + return setup_tls(host, verifySsl); + } + bool connect(const char* host, int port, int timeoutMs, bool verifySsl) { // Step 1: TCP connect via Socket if (!socket_.connect(host, port, timeoutMs)) { @@ -207,6 +214,8 @@ public: mbedtls_ssl_close_notify(&state_->ssl); state_.reset(); } + // Only after the close_notify above, which travels through it. + lower_.reset(); socket_.close(); } @@ -215,7 +224,9 @@ public: if (state_ && mbedtls_ssl_get_bytes_avail(&state_->ssl) > 0) { return true; } - return socket_.wait_readable(timeoutMs); + // Under another session the bytes may be decrypted and waiting there + // while the descriptor has nothing, so ask that session, not the fd. + return lower_ ? lower_->wait_readable(timeoutMs) : socket_.wait_readable(timeoutMs); } // TLS back-pressure is a wait, not a failure. `mbedtls_ssl_write` reports @@ -223,13 +234,55 @@ public: // that into 0; without something to wait on, `write_all` could only retry // at once and then give up, which reached the caller as "Write failed". bool wait_writable(int timeoutMs) { - return socket_.wait_writable(timeoutMs); + return lower_ ? lower_->wait_writable(timeoutMs) : socket_.wait_writable(timeoutMs); } private: Socket socket_; + // The session this one runs inside, when there is one; `socket_` is unused + // then. Declared before `state_` so it is destroyed after it: the BIO of + // `state_` points at it. + std::unique_ptr lower_; std::unique_ptr state_; + // The BIO of a session that runs inside another is that session, which stays + // put when this object moves; the BIO of one on a socket is `socket_`, which + // moves with it. + void bind_bio() { + if (!state_) return; + if (lower_) { + mbedtls_ssl_set_bio(&state_->ssl, lower_.get(), send_over_tls, recv_over_tls, nullptr); + } else { + mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); + } + } + + static int send_over_tls(void* ctx, const unsigned char* buf, size_t len) { + int ret = static_cast(ctx)->write(reinterpret_cast(buf), + static_cast(len)); + if (ret < 0) return MBEDTLS_ERR_NET_SEND_FAILED; + if (ret == 0) return MBEDTLS_ERR_SSL_WANT_WRITE; + return ret; + } + + // As `bio_recv`: an end of stream is passed on as a zero. + static int recv_over_tls(void* ctx, unsigned char* buf, size_t len) { + auto r = static_cast(ctx)->read_some(reinterpret_cast(buf), + static_cast(len)); + switch (r.status) { + case ReadStatus::Data: return r.bytes; + case ReadStatus::Eof: return 0; + case ReadStatus::WouldBlock: return MBEDTLS_ERR_SSL_WANT_READ; + case ReadStatus::Error: break; + } + return MBEDTLS_ERR_NET_RECV_FAILED; + } + + void drop_transport() { + lower_.reset(); + socket_.close(); + } + bool setup_tls(const char* host, bool verifySsl) { state_ = std::make_unique(); @@ -238,7 +291,7 @@ private: nullptr, 0); if (ret != 0) { state_.reset(); - socket_.close(); + drop_transport(); return false; } @@ -249,7 +302,7 @@ private: MBEDTLS_SSL_PRESET_DEFAULT); if (ret != 0) { state_.reset(); - socket_.close(); + drop_transport(); return false; } @@ -269,7 +322,7 @@ private: // ret > 0 means some certs failed to parse but others succeeded — acceptable if (ret < 0) { state_.reset(); - socket_.close(); + drop_transport(); return false; } mbedtls_ssl_conf_ca_chain(&state_->conf, &state_->ca_cert, nullptr); @@ -288,7 +341,7 @@ private: ret = mbedtls_ssl_setup(&state_->ssl, &state_->conf); if (ret != 0) { state_.reset(); - socket_.close(); + drop_transport(); return false; } @@ -296,18 +349,18 @@ private: ret = mbedtls_ssl_set_hostname(&state_->ssl, host); if (ret != 0) { state_.reset(); - socket_.close(); + drop_transport(); return false; } - // Set BIO callbacks using our Socket - mbedtls_ssl_set_bio(&state_->ssl, &socket_, bio_send, bio_recv, nullptr); + // Set BIO callbacks using our Socket, or the session beneath this one + bind_bio(); // Perform TLS handshake while ((ret = mbedtls_ssl_handshake(&state_->ssl)) != 0) { if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE) { state_.reset(); - socket_.close(); + drop_transport(); return false; } } diff --git a/tests/proxy_test_server.hpp b/tests/proxy_test_server.hpp index 3adaf6e..52ea42d 100644 --- a/tests/proxy_test_server.hpp +++ b/tests/proxy_test_server.hpp @@ -53,6 +53,9 @@ class Peer { // read a byte at a time so nothing past it is taken. Empty on a hang-up. std::string read_head(); + // Whatever arrives first, up to 4 KiB. Empty on a hang-up. + std::string read_some(); + bool write(std::string_view data) { std::size_t sent = 0; while (sent < data.size()) { @@ -180,6 +183,18 @@ inline std::string Peer::read_head() { return head; } +inline std::string Peer::read_some() { + unsigned char buf[4096]; + for (int waited = 0; waited < 20000; waited += 50) { + if (server_.stopping()) return {}; + if (!wait_readable(50)) continue; + int n = mbedtls_net_recv(&net_, buf, sizeof buf); + return n > 0 ? std::string(reinterpret_cast(buf), static_cast(n)) + : std::string(); + } + return {}; +} + inline void Peer::relay_to(int port) { mbedtls_net_context target {}; mbedtls_net_init(&target); diff --git a/tests/test_proxy.cpp b/tests/test_proxy.cpp index 8c38e48..22eced8 100644 --- a/tests/test_proxy.cpp +++ b/tests/test_proxy.cpp @@ -278,3 +278,186 @@ TEST_F(ProxyTest, TheOriginalProxyConnectStillReturnsATunnel) { target.port(), 4000); EXPECT_TRUE(tunnel.is_valid()); } + +// ── an https:// proxy ──────────────────────────────────────────────────────── +// +// The proxy here is the TLS test server: the client opens TLS to it, sends +// CONNECT inside that, and then opens a second TLS session to the target inside +// the tunnel. Two sessions on one descriptor is what these tests exist to +// exercise, and the cases that matter are the ones where the two disagree about +// what is readable: a body larger than a record, and a second request that has +// to find the tunnel already open. + +namespace { + +// A recorder for what the TLS proxy saw. The handler runs on the server's +// threads; the test reads it after the response is in. +struct Seen { + std::mutex mutex; + std::vector heads; + void add(std::string head) { + std::lock_guard lock(mutex); + heads.push_back(std::move(head)); + } + std::vector get() { + std::lock_guard lock(mutex); + return heads; + } +}; + +tls_test::Server tls_connect_proxy(Seen& seen, int targetPort, std::string expectAuth = {}, + std::chrono::milliseconds gather = std::chrono::milliseconds::zero()) { + return tls_test::Server([&seen, targetPort, expectAuth, gather](tls_test::Conn& conn, int) { + seen.add(conn.request()); + if (!expectAuth.empty() + && header_of(conn.request(), "Proxy-Authorization") != expectAuth) { + conn.write("HTTP/1.1 407 Proxy Authentication Required\r\n" + "Proxy-Authenticate: Basic realm=\"test\"\r\n" + "Content-Length: 0\r\n\r\n"); + return false; + } + conn.write(kEstablished); + conn.relay_to(targetPort, gather); + return false; + }); +} + +} // namespace + +TEST(ProxyUrl, AnHttpsProxyDefaultsToPort443) { + auto p = https::parse_proxy_url("https://proxy.example"); + EXPECT_EQ(p.scheme, "https"); + EXPECT_EQ(p.port, 443); + EXPECT_EQ(https::parse_proxy_url("https://proxy.example:8443").port, 8443); +} + +TEST_F(ProxyTest, ARequestGoesThroughAnHttpsProxy) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto heads = seen.get(); + ASSERT_EQ(heads.size(), 1u); + EXPECT_TRUE(heads[0].starts_with("CONNECT 127.0.0.1:" + std::to_string(target.port()) + " HTTP/1.1\r\n")); +} + +TEST_F(ProxyTest, CredentialsGoToAnHttpsProxyInsideItsTlsSession) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port(), "Basic YWxpY2U6czNjcmV0"); + + https::HttpClient ok(proxied("https://alice:s3cret@127.0.0.1:" + std::to_string(proxy.port()))); + EXPECT_EQ(ok.send(get(target.url("/"))).statusCode, 200); + + https::HttpClient bad(proxied("https://alice:nope@127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = bad.send(get(target.url("/"))); + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); +} + +TEST_F(ProxyTest, ABodyLargerThanARecordCrossesBothSessions) { + std::string big(300 * 1024, 'x'); + for (std::size_t i = 0; i < big.size(); i += 97) big[i] = static_cast('a' + i % 26); + tls_test::Server target([&](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response(big)); + return true; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_TRUE(resp.bodyComplete); + EXPECT_TRUE(resp.body == big); +} + +TEST_F(ProxyTest, TwoRequestsShareOneTunnelThroughAnHttpsProxy) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} + +// The target writes the head and the body as two records and the proxy hands +// them on as one, so both arrive in a single record of the proxy session. The +// client reads the first through the proxy session and the descriptor is then +// empty while the second sits decrypted in that session: waiting on the +// descriptor, rather than on the session, would time out on a response that has +// already arrived. +TEST_F(ProxyTest, ABodyAlreadyDecryptedInTheProxySessionIsNotWaitedFor) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n"); + conn.write("hello"); + return true; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port(), {}, std::chrono::milliseconds(150)); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); +} + +TEST_F(ProxyTest, AStreamingRequestWorksThroughAnHttpsProxy) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n" + "Connection: close\r\n\r\ndata: one\n\ndata: two\n\n"); + return false; + }); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + std::vector events; + auto resp = client.send_stream(get(target.url("/")), [&](const https::SseEvent& ev) { + events.push_back(ev.data); + return true; + }); + + EXPECT_EQ(resp.statusCode, 200); + ASSERT_EQ(events.size(), 2u); + EXPECT_EQ(events[0], "one"); + EXPECT_EQ(events[1], "two"); +} + +// The handshake has no timeout of its own, so the proxy ends it the way a real +// HTTP proxy does when it is sent something that is not HTTP: it reads the first +// bytes and hangs up. +TEST_F(ProxyTest, AnHttpsUrlToAPlainProxyFailsAtTheHandshake) { + auto target = hello_server(); + proxy_test::Server proxy([](proxy_test::Peer& peer) { peer.read_some(); }); + + https::HttpClient client(proxied("https://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("TLS connection"), std::string::npos); +} + +TEST_F(ProxyTest, AnHttpUrlToATlsProxyIsAnErrorAndNotAHang) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + + https::HttpClient client(proxied("http://127.0.0.1:" + std::to_string(proxy.port()))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_FALSE(resp.statusText.empty()); +} diff --git a/tests/tls_test_server.hpp b/tests/tls_test_server.hpp index 4170680..16ee5dc 100644 --- a/tests/tls_test_server.hpp +++ b/tests/tls_test_server.hpp @@ -48,6 +48,7 @@ #include #include +#include #include #include #include @@ -162,6 +163,18 @@ class Conn { // answer differently per path. const std::string& request() const { return request_; } + // Splices what this connection decrypts to 127.0.0.1:`port`, and what comes + // back from there into it, until either side ends. This is what makes the + // server a proxy for the proxy tests: after it answers a CONNECT, the + // client's next bytes are the start of a second TLS session, and this + // carries them to the real target untouched. + // + // With `gather` set, what comes back from the target is held until the + // target has been quiet for that long and then sent as one write, so + // several records the target wrote in a row reach the client inside a single + // record of this session. + void relay_to(int port, std::chrono::milliseconds gather = std::chrono::milliseconds::zero()); + private: friend class Server; @@ -400,6 +413,46 @@ inline bool Conn::wait_readable() { return false; } +inline void Conn::relay_to(int port, std::chrono::milliseconds gather) { + mbedtls_net_context target {}; + mbedtls_net_init(&target); + if (mbedtls_net_connect(&target, "127.0.0.1", std::to_string(port).c_str(), + MBEDTLS_NET_PROTO_TCP) != 0) { + return; + } + unsigned char buf[4096]; + std::string back; + while (!server_.stopping()) { + if (mbedtls_ssl_get_bytes_avail(&ssl_) > 0 + || mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, 10) > 0) { + int n = mbedtls_ssl_read(&ssl_, buf, sizeof buf); + if (n == MBEDTLS_ERR_SSL_WANT_READ || n == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (n <= 0) break; + int sent = 0; + while (sent < n) { + int w = mbedtls_net_send(&target, buf + sent, static_cast(n - sent)); + if (w <= 0) { n = 0; break; } + sent += w; + } + if (n == 0) break; + } + if (mbedtls_net_poll(&target, MBEDTLS_NET_POLL_READ, 10) > 0) { + int n = mbedtls_net_recv(&target, buf, sizeof buf); + if (n <= 0) break; + back.assign(reinterpret_cast(buf), static_cast(n)); + while (gather.count() > 0 && back.size() < 8192 + && mbedtls_net_poll(&target, MBEDTLS_NET_POLL_READ, + static_cast(gather.count())) > 0) { + n = mbedtls_net_recv(&target, buf, sizeof buf); + if (n <= 0) break; + back.append(reinterpret_cast(buf), static_cast(n)); + } + if (!write(back)) break; + } + } + mbedtls_net_free(&target); +} + inline void Conn::park(std::chrono::milliseconds limit) { const auto deadline = std::chrono::steady_clock::now() + limit; while (!server_.stopping() && std::chrono::steady_clock::now() < deadline) { From 0733e5d19da91da10a76878de3406984d8993d41 Mon Sep 17 00:00:00 2001 From: Cloud_Yun Date: Thu, 1 Oct 2026 13:14:04 +0900 Subject: [PATCH 06/15] feat(proxy): connect through SOCKS5 proxies socks5:// and socks5h:// proxy URLs are now understood. Both speak RFC 1928: the client offers "no authentication", and the username/password method of RFC 1929 as well when the URL carries credentials, then sends a CONNECT request and reads the reply, including the bound address that precedes the tunnel, whatever its type. The difference between the two schemes is who resolves the target's name. With socks5 the name is resolved here and the proxy is given an address; with socks5h the proxy is given the name. An address literal is sent as an address in both, and a name too long for the request is refused before anything is sent. Socket gains resolve_address for the first case, with the same manual-DNS fallback connect uses. A failure is reported in statusText as it is for an HTTP proxy: the credentials rejected, credentials needed and missing, or the proxy's reply code in words. SOCKS4 is not implemented and is an error, like any other unknown scheme. Tests script a SOCKS5 proxy byte for byte and check what it was sent for each scheme and for each authentication outcome. --- CHANGELOG.md | 14 ++- README.md | 29 +++-- src/proxy.cppm | 133 ++++++++++++++++++++- src/socket.cppm | 42 +++++++ tests/proxy_test_server.hpp | 18 +++ tests/test_proxy.cpp | 231 ++++++++++++++++++++++++++++++++++++ 6 files changed, 448 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9517061..a1d819b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,11 +20,8 @@ HTTP. And every refusal, whatever the proxy said, reached the caller as 407 Proxy Authentication Required` when credentials were sent, `proxy requires credentials: ...` when none were, `proxy refused CONNECT: ...` for any other status. Any 2xx answer opens the tunnel; before, only 200 did. -* A scheme other than `http` is an error (`proxy: unsupported scheme - 'socks4'`) where it used to be read as HTTP. That also ends a habit that - happened to work: a proxy that answers HTTP and SOCKS on one port could be - used by writing `socks5://` or `https://` in front of it, because the scheme - was ignored. +* A scheme this library does not speak is an error (`proxy: unsupported scheme + 'socks4'`) where it used to be read as HTTP. * New `proxy_tunnel` returns the tunnel together with the reason it could not be made. `proxy_connect` keeps its signature and calls it, so it also accepts any 2xx answer now. @@ -35,6 +32,13 @@ HTTP. And every refusal, whatever the proxy said, reached the caller as descriptor, because the bytes may already be decrypted there. The proxy connection is set up like the one to the target, `verifySsl` included. The default port for `https` is 443. +* SOCKS5 proxies: `socks5://` resolves the target's name here and sends the + proxy an address, `socks5h://` sends the name. An address literal is sent as + an address in both. The client offers no authentication, plus the + username/password method when the URL has credentials. Failures are reported + in `statusText` the same way as for an HTTP proxy. The default port is 1080. + SOCKS4 is not supported and, like any unknown scheme, is an error. +* `Socket::resolve_address` returns the address a name resolves to as raw bytes. ## 0.3.2 diff --git a/README.md b/README.md index 200629d..57a1634 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Minimal C++23 HTTP/HTTPS client library with SSE (Server-Sent Events) streaming - HTTP/HTTPS client with connection pooling (keep-alive) - SSE (Server-Sent Events) streaming - Streaming downloads to a file, with progress and cancellation -- Proxy support (HTTP CONNECT, over plain HTTP or TLS, with Basic authentication) +- Proxy support (HTTP CONNECT over plain HTTP or TLS, with Basic authentication; SOCKS5) - C++23 modules ## Usage @@ -67,17 +67,30 @@ Set `proxy` to a URL and every request is tunnelled through it with `CONNECT`: cfg.proxy = "http://user:pass@proxy.example:3128"; ``` -The scheme is `http` (a bare `host:port` means the same) or `https`. An -`https://` proxy is reached over TLS first, the `CONNECT` is sent inside that +The scheme is `http` (a bare `host:port` means the same), `https`, `socks5` or +`socks5h`. + +An `https://` proxy is reached over TLS first, the `CONNECT` is sent inside that session, and the connection to the target is a second TLS session inside the -tunnel; the two connections are set up alike, `verifySsl` included. Credentials -are read from the URL and sent as Basic `Proxy-Authorization`; percent-escape any character that is special in -a URL (`p%40ss` for `p@ss`). The port defaults to 8080, or 443 for `https`. +tunnel; the two connections are set up alike, `verifySsl` included. + +With `socks5` the target's name is resolved here and the proxy is given an +address; with `socks5h` the proxy is given the name and resolves it itself, +which is what to use when the proxy is the only thing that can see the target's +DNS. An address literal is sent as an address either way. The client offers the +SOCKS5 proxy no authentication, plus the username/password method when the URL +has credentials. SOCKS4 is not supported. + +Credentials are read from the URL and sent as Basic `Proxy-Authorization` (or as +the SOCKS5 username and password); percent-escape any character that is special +in a URL (`p%40ss` for `p@ss`). The port defaults to 8080, 443 for `https`, and +1080 for the SOCKS schemes. When the proxy refuses the tunnel, `statusCode` is 0 and `statusText` carries its answer: `proxy rejected the credentials: 407 Proxy Authentication Required`, -`proxy refused CONNECT: 403 Forbidden`. Any other scheme (`socks4://`, say) -fails the same way instead of being read as HTTP. +`proxy refused CONNECT: 403 Forbidden`, `proxy refused CONNECT: connection +refused (SOCKS5 reply 5)`. Any other scheme (`socks4://`, say) fails the same +way instead of being read as HTTP. ## Project templates diff --git a/src/proxy.cppm b/src/proxy.cppm index db7e6c8..c2061c9 100644 --- a/src/proxy.cppm +++ b/src/proxy.cppm @@ -13,9 +13,11 @@ export struct ProxyConfig { std::string host; int port { 8080 }; - // Lower case: "http", or "https" for a proxy that is itself reached over - // TLS. "http" when the URL has no scheme, which is how a bare `host:port` - // has always been read. + // Lower case: "http", "https" for a proxy that is itself reached over TLS, + // "socks5" or "socks5h". "http" when the URL has no scheme, which is how a + // bare `host:port` has always been read. With "socks5" this library + // resolves the target's name and gives the proxy an address; with "socks5h" + // it gives the proxy the name. std::string scheme { "http" }; // From `user:password@` in the URL, percent-decoded. A URL with `@` and no @@ -51,7 +53,8 @@ static std::string percent_decode(std::string_view in) { // Parse "[scheme://][user[:password]@]host[:port]". A path, query or fragment // after the authority is ignored. IPv6 literals are written in brackets. The -// port defaults to 443 for https and to 8080 otherwise. +// port defaults to 443 for https, 1080 for socks5 and socks5h, and 8080 +// otherwise. export ProxyConfig parse_proxy_url(std::string_view url) { ProxyConfig config; @@ -98,6 +101,7 @@ export ProxyConfig parse_proxy_url(std::string_view url) { } if (config.scheme == "https") config.port = 443; + if (config.scheme.starts_with("socks")) config.port = 1080; if (hasPort && !portStr.empty()) { // Anything but 1..65535 becomes 0, which no connect will accept. @@ -212,6 +216,121 @@ static std::string http_connect(Stream& sock, const ProxyConfig& proxy, return "proxy: response to CONNECT has too many headers"; } +// SOCKS5 (RFC 1928), with the username/password method of RFC 1929. + +static bool read_exact(Socket& sock, unsigned char* buf, std::size_t n, int timeoutMs) { + std::size_t got = 0; + while (got < n) { + if (!sock.wait_readable(timeoutMs)) return false; + int ret = sock.read(reinterpret_cast(buf) + got, static_cast(n - got)); + if (ret <= 0) return false; // 0 is the proxy closing the connection + got += static_cast(ret); + } + return true; +} + +static std::string_view socks5_reply_text(unsigned char code) { + switch (code) { + case 1: return "general failure"; + case 2: return "connection not allowed by ruleset"; + case 3: return "network unreachable"; + case 4: return "host unreachable"; + case 5: return "connection refused"; + case 6: return "TTL expired"; + case 7: return "command not supported"; + case 8: return "address type not supported"; + } + return "unknown error"; +} + +// Returns an error message, or an empty string once the proxy has connected to +// the target and the stream is the tunnel. +static std::string socks5_connect(Socket& sock, const ProxyConfig& proxy, + std::string_view host, int port, int timeoutMs) { + const bool remoteDns = proxy.scheme == "socks5h"; + const std::string hostStr(host); + + // Everything that can be refused before a byte is sent is refused first. + if (proxy.hasCredentials && (proxy.user.size() > 255 || proxy.password.size() > 255)) { + return "proxy: SOCKS5 credentials are limited to 255 bytes each"; + } + // An address literal is always sent as an address; a name is sent as a name + // for socks5h and resolved here for socks5. + auto address = Socket::resolve_address(hostStr.c_str(), /*numericOnly=*/true); + if (address.empty() && remoteDns) { + if (host.empty() || host.size() > 255) return "proxy: host name does not fit a SOCKS5 request"; + } else if (address.empty()) { + address = Socket::resolve_address(hostStr.c_str()); + if (address.empty()) return "proxy: could not resolve " + hostStr; + } + + std::string greeting = proxy.hasCredentials ? std::string("\x05\x02\x00\x02", 4) + : std::string("\x05\x01\x00", 3); + if (!write_all(sock, greeting, timeoutMs)) return "proxy: could not send the SOCKS5 greeting"; + + unsigned char choice[2]; + if (!read_exact(sock, choice, 2, timeoutMs)) return "proxy: no SOCKS5 response"; + if (choice[0] != 5) return "proxy: not a SOCKS5 proxy"; + if (choice[1] == 0xFF) { + return proxy.hasCredentials + ? "proxy: SOCKS5 proxy accepts neither no authentication nor a username and password" + : "proxy requires credentials: no acceptable SOCKS5 authentication method"; + } + if (choice[1] == 2 && proxy.hasCredentials) { + std::string auth = "\x01"; + auth += static_cast(proxy.user.size()); + auth += proxy.user; + auth += static_cast(proxy.password.size()); + auth += proxy.password; + unsigned char status[2]; + if (!write_all(sock, auth, timeoutMs) || !read_exact(sock, status, 2, timeoutMs)) { + return "proxy: no response to the SOCKS5 username and password"; + } + if (status[1] != 0) return "proxy rejected the credentials (SOCKS5 status " + std::to_string(status[1]) + ")"; + } else if (choice[1] != 0) { + return "proxy: SOCKS5 proxy chose an authentication method that was not offered"; + } + + std::string request("\x05\x01\x00", 3); + if (address.empty()) { + request += '\x03'; + request += static_cast(host.size()); + request += host; + } else { + request += address.size() == 4 ? '\x01' : '\x04'; + request.append(reinterpret_cast(address.data()), address.size()); + } + request += static_cast(port >> 8); + request += static_cast(port & 0xFF); + if (!write_all(sock, request, timeoutMs)) return "proxy: could not send the SOCKS5 request"; + + unsigned char reply[4]; + if (!read_exact(sock, reply, 4, timeoutMs)) return "proxy: no SOCKS5 reply to the request"; + if (reply[0] != 5) return "proxy: not a SOCKS5 proxy"; + if (reply[1] != 0) { + return "proxy refused CONNECT: " + std::string(socks5_reply_text(reply[1])) + + " (SOCKS5 reply " + std::to_string(reply[1]) + ")"; + } + + // The bound address and port: nothing to use, but they are on the stream + // ahead of the tunnel and have to be read off it. + std::size_t bound = 0; + if (reply[3] == 1) { + bound = 4; + } else if (reply[3] == 4) { + bound = 16; + } else if (reply[3] == 3) { + unsigned char len; + if (!read_exact(sock, &len, 1, timeoutMs)) return "proxy: SOCKS5 reply was cut short"; + bound = len; + } else { + return "proxy: SOCKS5 reply has an unknown address type"; + } + unsigned char rest[258]; + if (!read_exact(sock, rest, bound + 2, timeoutMs)) return "proxy: SOCKS5 reply was cut short"; + return {}; +} + // A connection to the target through a proxy. `error` is empty exactly when the // tunnel is open; otherwise it says what the proxy did, in words fit for a // caller to log. @@ -236,12 +355,14 @@ export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, ProxyTunnel tunnel; const std::string where = proxy.host + ":" + std::to_string(proxy.port); - if (proxy.scheme == "http") { + if (proxy.scheme == "http" || proxy.scheme == "socks5" || proxy.scheme == "socks5h") { if (!tunnel.socket.connect(proxy.host.c_str(), proxy.port, timeoutMs)) { tunnel.error = "proxy: could not connect to " + where; return tunnel; } - tunnel.error = http_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs); + tunnel.error = proxy.scheme == "http" + ? http_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs) + : socks5_connect(tunnel.socket, proxy, targetHost, targetPort, timeoutMs); if (!tunnel.error.empty()) tunnel.socket.close(); } else if (proxy.scheme == "https") { auto tls = std::make_unique(); diff --git a/src/socket.cppm b/src/socket.cppm index 5800588..c794f91 100644 --- a/src/socket.cppm +++ b/src/socket.cppm @@ -128,6 +128,48 @@ public: } } + // The first address `host` resolves to, as network-order bytes: four for + // IPv4, sixteen for IPv6, none when it does not resolve. IPv4 wins when both + // exist, which is what a SOCKS5 proxy of any age can be asked for. + // `numericOnly` accepts an address literal and nothing else, with no lookup. + static std::vector resolve_address(const char* host, bool numericOnly = false) { + struct addrinfo hints{}; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + if (numericOnly) hints.ai_flags = AI_NUMERICHOST; + + std::vector bytes; + struct addrinfo* result = nullptr; + if (::getaddrinfo(host, nullptr, &hints, &result) == 0 && result != nullptr) { + for (auto* rp = result; rp != nullptr; rp = rp->ai_next) { + if (rp->ai_family == AF_INET) { + auto* in = reinterpret_cast( + &reinterpret_cast(rp->ai_addr)->sin_addr); + bytes.assign(in, in + 4); + break; + } + if (rp->ai_family == AF_INET6 && bytes.empty()) { + auto* in = reinterpret_cast( + &reinterpret_cast(rp->ai_addr)->sin6_addr); + bytes.assign(in, in + 16); + } + } + ::freeaddrinfo(result); + } + + // As in `connect`: where libc has no resolver configuration (Termux), + // fall back to a manual query. + if constexpr (!platform::uses_winsock) { + if (bytes.empty() && !numericOnly) { + for (const auto& ip : platform::resolve_fallback(host, 2500)) { + bytes = resolve_address(ip.c_str(), true); + if (!bytes.empty()) break; + } + } + } + return bytes; + } + // Connect to the first reachable address in a resolved list. bool connect_addrinfo(struct addrinfo* result, int timeoutMs) { for (auto* rp = result; rp != nullptr; rp = rp->ai_next) { diff --git a/tests/proxy_test_server.hpp b/tests/proxy_test_server.hpp index 52ea42d..0d3d32d 100644 --- a/tests/proxy_test_server.hpp +++ b/tests/proxy_test_server.hpp @@ -53,6 +53,9 @@ class Peer { // read a byte at a time so nothing past it is taken. Empty on a hang-up. std::string read_head(); + // Exactly `n` bytes, or empty when the client hangs up first. + std::string read_exact(std::size_t n); + // Whatever arrives first, up to 4 KiB. Empty on a hang-up. std::string read_some(); @@ -183,6 +186,21 @@ inline std::string Peer::read_head() { return head; } +inline std::string Peer::read_exact(std::size_t n) { + std::string out; + unsigned char c {}; + while (out.size() < n) { + bool ready = false; + for (int waited = 0; waited < 20000 && !ready; waited += 50) { + if (server_.stopping()) return {}; + ready = wait_readable(50); + } + if (!ready || mbedtls_net_recv(&net_, &c, 1) <= 0) return {}; + out.push_back(static_cast(c)); + } + return out; +} + inline std::string Peer::read_some() { unsigned char buf[4096]; for (int waited = 0; waited < 20000; waited += 50) { diff --git a/tests/test_proxy.cpp b/tests/test_proxy.cpp index 22eced8..332f02f 100644 --- a/tests/test_proxy.cpp +++ b/tests/test_proxy.cpp @@ -461,3 +461,234 @@ TEST_F(ProxyTest, AnHttpUrlToATlsProxyIsAnErrorAndNotAHang) { EXPECT_EQ(resp.statusCode, 0); EXPECT_FALSE(resp.statusText.empty()); } + +// ── SOCKS5 ─────────────────────────────────────────────────────────────────── + +namespace { + +// What a SOCKS5 proxy needs to know about how to behave. +struct Socks5Script { + int targetPort { 0 }; + // Empty: accept the no-authentication method. Otherwise "user:password", + // and the proxy insists on the username/password method. + std::string credentials; + // The reply code to the CONNECT request; 0 is success. + unsigned char replyCode { 0 }; + // The address type of the bound address in a success reply: 1, 3 or 4. + unsigned char boundType { 1 }; +}; + +std::string hex_of(std::string_view bytes) { + std::string out; + for (unsigned char c : bytes) out += std::format("{:02x}", c); + return out; +} + +// Notes, in order: "methods:", "auth::" when it was used, +// and "request::
:". +proxy_test::Server::Handler socks5_handler(Socks5Script script) { + return [script](proxy_test::Peer& peer) { + auto greet = peer.read_exact(2); + if (greet.size() != 2 || greet[0] != 5) return; + auto methods = peer.read_exact(static_cast(greet[1])); + peer.note("methods:" + hex_of(methods)); + + if (script.credentials.empty()) { + peer.write(std::string_view("\x05\x00", 2)); + } else { + if (methods.find('\x02') == std::string::npos) { + peer.write(std::string_view("\x05\xff", 2)); + return; + } + peer.write(std::string_view("\x05\x02", 2)); + auto ver = peer.read_exact(2); // 0x01, ULEN + if (ver.size() != 2) return; + auto user = peer.read_exact(static_cast(ver[1])); + auto plen = peer.read_exact(1); + if (plen.size() != 1) return; + auto pass = peer.read_exact(static_cast(plen[0])); + peer.note("auth:" + user + ":" + pass); + if (user + ":" + pass != script.credentials) { + peer.write(std::string_view("\x01\x01", 2)); + return; + } + peer.write(std::string_view("\x01\x00", 2)); + } + + auto head = peer.read_exact(4); // VER CMD RSV ATYP + if (head.size() != 4) return; + std::string addr; + std::size_t len = head[3] == 1 ? 4 : head[3] == 4 ? 16 : 0; + if (head[3] == 3) { + auto n = peer.read_exact(1); + if (n.size() != 1) return; + addr = peer.read_exact(static_cast(n[0])); + } else { + addr = peer.read_exact(len); + } + auto port = peer.read_exact(2); + if (port.size() != 2) return; + peer.note(std::format("request:{}:{}:{}", static_cast(head[3]), + head[3] == 3 ? addr : hex_of(addr), + (static_cast(port[0]) << 8) + | static_cast(port[1]))); + + std::string reply("\x05\x00\x00", 3); + reply[1] = static_cast(script.replyCode); + reply += static_cast(script.boundType); + if (script.boundType == 1) reply.append(4, '\x7f'); + else if (script.boundType == 4) reply.append(16, '\x01'); + else reply += std::string("\x0c") + "bind.example"; + reply.append("\x04\xd2", 2); + peer.write(reply); + if (script.replyCode == 0) peer.relay_to(script.targetPort); + }; +} + +std::string socks_url(std::string_view scheme, const proxy_test::Server& proxy, + std::string_view userinfo = {}) { + std::string url(scheme); + url += "://"; + if (!userinfo.empty()) { url += userinfo; url += "@"; } + return url + "127.0.0.1:" + std::to_string(proxy.port()); +} + +} // namespace + +TEST(ProxyUrl, SocksSchemesAreKeptAndDefaultToPort1080) { + auto a = https::parse_proxy_url("socks5://proxy.example"); + EXPECT_EQ(a.scheme, "socks5"); + EXPECT_EQ(a.port, 1080); + auto b = https::parse_proxy_url("SOCKS5H://u:p@proxy.example:9050"); + EXPECT_EQ(b.scheme, "socks5h"); + EXPECT_EQ(b.port, 9050); + EXPECT_EQ(b.user, "u"); +} + +TEST_F(ProxyTest, Socks5ConnectsToAnAddressLiteral) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port() })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 200); + EXPECT_EQ(resp.body, "hello"); + auto notes = proxy.notes(); + ASSERT_EQ(notes.size(), 2u); + EXPECT_EQ(notes[0], "methods:00"); // no credentials, so only "none" is offered + EXPECT_EQ(notes[1], "request:1:7f000001:" + std::to_string(target.port())); +} + +TEST_F(ProxyTest, Socks5hSendsAHostNameAndSocks5SendsAnAddress) { + auto target = hello_server(); + const std::string url = "https://localhost:" + std::to_string(target.port()) + "/"; + + proxy_test::Server remote(socks5_handler({ .targetPort = target.port() })); + https::HttpClient viaH(proxied(socks_url("socks5h", remote))); + EXPECT_EQ(viaH.send(get(url)).statusCode, 200); + ASSERT_EQ(remote.notes().size(), 2u); + EXPECT_EQ(remote.notes()[1], "request:3:localhost:" + std::to_string(target.port())); + + proxy_test::Server local(socks5_handler({ .targetPort = target.port() })); + https::HttpClient viaLocal(proxied(socks_url("socks5", local))); + EXPECT_EQ(viaLocal.send(get(url)).statusCode, 200); + ASSERT_EQ(local.notes().size(), 2u); + EXPECT_NE(local.notes()[1].substr(0, 10), "request:3:"); // an address, not the name +} + +TEST_F(ProxyTest, Socks5UsernameAndPasswordAreOfferedAndSent) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "al@ice:p:ss w" })); + + https::HttpClient client(proxied(socks_url("socks5h", proxy, "al%40ice:p%3Ass%20w"))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); + + auto notes = proxy.notes(); + ASSERT_GE(notes.size(), 2u); + EXPECT_EQ(notes[0], "methods:0002"); + EXPECT_EQ(notes[1], "auth:al@ice:p:ss w"); +} + +TEST_F(ProxyTest, Socks5WrongCredentialsAreReportedAsThat) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "alice:s3cret" })); + + https::HttpClient client(proxied(socks_url("socks5", proxy, "alice:wrong"))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("rejected the credentials"), std::string::npos); + EXPECT_EQ(target.accepts(), 0); +} + +TEST_F(ProxyTest, Socks5ProxyThatWantsCredentialsWeDidNotGiveSaysSo) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), + .credentials = "alice:s3cret" })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("requires credentials"), std::string::npos); +} + +TEST_F(ProxyTest, Socks5RefusalCarriesTheReplyCode) { + proxy_test::Server proxy(socks5_handler({ .replyCode = 5 })); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("refused CONNECT: connection refused (SOCKS5 reply 5)"), + std::string::npos); +} + +TEST_F(ProxyTest, Socks5AnyBoundAddressTypeIsReadOffTheStream) { + // The reply to a successful CONNECT carries an address of one of three + // shapes. Misreading its length leaves bytes ahead of the TLS handshake + // and the request fails; reading it right is what a 200 here shows. + for (unsigned char type : {1, 3, 4}) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port(), .boundType = type })); + https::HttpClient client(proxied(socks_url("socks5", proxy))); + EXPECT_EQ(client.send(get(target.url("/"))).statusCode, 200); + } +} + +TEST_F(ProxyTest, Socks5ASpeakerOfAnotherProtocolIsNotMistakenForOne) { + proxy_test::Server proxy([](proxy_test::Peer& peer) { + peer.read_exact(3); + peer.write("HTTP/1.1 400 Bad Request\r\n\r\n"); + }); + + https::HttpClient client(proxied(socks_url("socks5", proxy))); + auto resp = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("not a SOCKS5 proxy"), std::string::npos); +} + +TEST_F(ProxyTest, Socks5hRefusesAHostNameThatDoesNotFitTheRequest) { + proxy_test::Server proxy(socks5_handler({})); + https::HttpClient client(proxied(socks_url("socks5h", proxy))); + auto resp = client.send(get("https://" + std::string(300, 'a') + ".example/")); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_NE(resp.statusText.find("does not fit a SOCKS5 request"), std::string::npos); +} + +TEST_F(ProxyTest, TwoRequestsShareOneSocks5Tunnel) { + auto target = hello_server(); + proxy_test::Server proxy(socks5_handler({ .targetPort = target.port() })); + + https::HttpClient client(proxied(socks_url("socks5h", proxy))); + EXPECT_EQ(client.send(get(target.url("/a"))).statusCode, 200); + EXPECT_EQ(client.send(get(target.url("/b"))).statusCode, 200); + + EXPECT_EQ(proxy.accepts(), 1); + EXPECT_EQ(target.accepts(), 1); +} From e52d78d72433f58248566706b0a3c88bd1574f4e Mon Sep 17 00:00:00 2001 From: SPeak Date: Thu, 1 Oct 2026 19:58:47 +0800 Subject: [PATCH 07/15] ci, examples: a refused certificate is a failure, and the Windows store is built and run Three additions on top of the contributor's fix, each a consequence of it. `examples/openkal` read every status 0 as "no network" and exited 0. With verification now enforced, a CA bundle the openkal stack cannot find would have passed there as a skipped run. A statusText that names a refused certificate, a missing or unparsable bundle, or a failed handshake now exits 1. The README states the two differences a build above openkal has: on Windows the socket interface is POSIX, so the ROOT store is not read and SSL_CERT_FILE is required; and connectTimeoutMs does not bound the TCP connect, because kal_net_connect has no deferred form. The Windows ROOT store reader is compiled on no platform CI covered. A windows-2022 job (msvc and llvm) builds the library and runs test_ca_store, which asserts that the store is found with nothing configured and that a public chain verifies against it. An unreachable host is a skip; a refusal is a failure. Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/workflows/ci.yml | 54 +++++++++++++++++++++++++++++++++-- README.md | 11 +++++++ examples/openkal/src/main.cpp | 18 +++++++++++- tests/test_ca_store.cpp | 42 +++++++++++++++++++++++++++ 4 files changed, 122 insertions(+), 3 deletions(-) create mode 100644 tests/test_ca_store.cpp diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fba59e5..724b69e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -142,8 +142,58 @@ jobs: # # It makes one HTTPS request and reports "no network" rather than failing # when there is none, so a runner without egress reports "not run" instead - # of "broken". The build, the link and the framing parsers are checked - # either way. + # of "broken". A certificate the client refused is a failure, not an + # absence of network. The build, the link and the framing parsers are + # checked either way. - name: Build and run above openkal working-directory: examples/openkal run: mcpp run + + # ── The Windows ROOT certificate store ─────────────────────────────────────── + # + # WHY THIS JOB EXISTS. Windows keeps no bundle file, so on a Windows Sockets + # build `load_ca_certs` reads the system's ROOT store through the Win32 API + # (src/ca_bundle.cppm). That code is compiled on no other platform, and a job + # elsewhere can show neither that it builds nor that the roots it returns + # verify a real certificate chain. `test_ca_store` asserts both. + windows: + name: build + test (windows x86_64, ${{ matrix.toolchain }}) + runs-on: windows-2022 + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + toolchain: ['msvc@system', 'llvm@20.1.7'] + defaults: + run: + shell: bash + steps: + - uses: actions/checkout@v4 + + - name: Install xlings + shell: pwsh + run: | + irm https://d2learn.org/xlings-install.ps1.txt | iex + "$env:USERPROFILE\.xlings\subos\current\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append + + - name: Install mcpp + run: | + xlings update + xlings install "mcpp@$MCPP_VERSION" -y -g + mcpp --version + mcpp self config --mirror GLOBAL + + - name: Select the toolchain + run: | + spec='${{ matrix.toolchain }}' + case "$spec" in + msvc*) mcpp toolchain default msvc ;; + *) mcpp toolchain install "${spec%@*}" "${spec#*@}" + mcpp toolchain default "$spec" ;; + esac + mcpp toolchain list + + - name: The system store verifies a public chain + run: | + mcpp test test_ca_store 2>&1 | tee tests.log + grep -q '^test_ca_store \.\.\. ok' tests.log diff --git a/README.md b/README.md index def0b1f..c808b12 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,17 @@ differs between them. cd examples/openkal && mcpp run ``` +Above openkal two things differ from a host build. Both follow from the layer +beneath and not from this library: + +- On Windows the socket interface is the POSIX one, because the C library is + musl, so the Windows `ROOT` certificate store is not read. Set `SSL_CERT_FILE` + to a PEM bundle of trusted roots; without one every HTTPS connection is + refused, and `statusText` says so. +- `connectTimeoutMs` does not bound the TCP connect. `kal_net_connect` has no + form that begins a connection and reports its outcome later, so the connect + completes or fails before it returns. + ## 使用 mcpp 构建 ### 添加依赖 diff --git a/examples/openkal/src/main.cpp b/examples/openkal/src/main.cpp index 6a63d8b..decd56d 100644 --- a/examples/openkal/src/main.cpp +++ b/examples/openkal/src/main.cpp @@ -8,7 +8,8 @@ // // Run it with `mcpp run` from this directory. It needs the network; when there // is none it says so and exits 0, because a CI job without egress should report -// "not run" rather than "broken". +// "not run" rather than "broken". A certificate the client refused is not an +// absence of network, and exits 1. import mcpplibs.tinyhttps; import std; @@ -40,6 +41,21 @@ int main() { auto response = client.send(request); if (response.statusCode == 0) { + // A certificate the client refused is a verdict about this stack, not + // an absence of network, and is reported as one. Read as "no network", + // a CA bundle this stack cannot find would pass here unnoticed. + constexpr std::string_view refusals[] = { + "certificate verification failed", + "no CA certificate bundle", + "cannot parse the CA certificate bundle", + "TLS handshake failed", + }; + for (auto refusal : refusals) { + if (response.statusText.starts_with(refusal)) { + std::println("TLS refused: {}", response.statusText); + return 1; + } + } std::println("no network ({}) — the build and the parsers are still verified", response.statusText); return 0; diff --git a/tests/test_ca_store.cpp b/tests/test_ca_store.cpp new file mode 100644 index 0000000..38e2dd7 --- /dev/null +++ b/tests/test_ca_store.cpp @@ -0,0 +1,42 @@ +// The trust store the platform supplies is found with nothing configured, and a +// public certificate chain verifies against it. +// +// On a Windows Sockets build the store is the system's ROOT store, read through +// the Win32 API; elsewhere it is a bundle file at one of the known locations. +// Either way a default-constructed client verifies a public server. A host that +// cannot be reached is a skip, so a runner without egress reports "not run"; a +// certificate the client refused is a failure. +#include + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; + +TEST(SystemStore, IsFoundWithNothingConfigured) { + if (const char* file = std::getenv("SSL_CERT_FILE"); file && *file) { + GTEST_SKIP() << "SSL_CERT_FILE is set, so the system store is not consulted"; + } + const auto pem = https::load_ca_certs(); + EXPECT_FALSE(pem.empty()); + EXPECT_NE(pem.find("-----BEGIN CERTIFICATE-----"), std::string::npos); +} + +TEST(SystemStore, APublicChainVerifiesAgainstIt) { + https::Socket::platform_init(); + https::HttpClientConfig config; + config.connectTimeoutMs = 15000; + config.readTimeoutMs = 20000; + https::HttpClient client{config}; + + https::HttpRequest request; + request.url = "https://one.one.one.one/"; + const auto response = client.send(request); + + if (response.statusCode == 0 && response.statusText == "Connection failed") { + GTEST_SKIP() << "one.one.one.one is not reachable from this runner"; + } + EXPECT_NE(response.statusCode, 0) << response.statusText; +} From ea4dd96282c921b8972c77648280b6c923097408 Mon Sep 17 00:00:00 2001 From: SPeak Date: Thu, 1 Oct 2026 20:02:29 +0800 Subject: [PATCH 08/15] tls: link bcrypt where link.exe would not read the dependency's request The msvc row of the new Windows job stopped at the link: entropy_poll.obj : error LNK2019: unresolved external symbol BCryptGenRandom referenced in function mbedtls_platform_entropy_poll mbedTLS's package names the library as `-lbcrypt`, which clang reads and link.exe ignores. The llvm row, the same sources under the same ABI, linked and verified a public chain against the ROOT store. The request is now also made with `#pragma comment(lib, ...)`, as socket.cppm makes it for ws2_32 and ca_bundle.cppm for crypt32. Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- src/tls.cppm | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/tls.cppm b/src/tls.cppm index 5937cae..667f87a 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -7,6 +7,14 @@ module; #include #include +// mbedTLS draws its entropy from BCryptGenRandom on Windows. Its package asks +// for the library as `-lbcrypt`, which link.exe does not read, so the request is +// made here in the form every linker for this target reads, as socket.cppm does +// for ws2_32. +#if defined(_WIN32) && !defined(TINYHTTPS_POSIX_SOCKETS) +#pragma comment(lib, "bcrypt.lib") +#endif + export module mcpplibs.tinyhttps:tls; import :socket; From d48aba114064e8bccb99738374bbcd4214c71256 Mon Sep 17 00:00:00 2001 From: SPeak Date: Thu, 1 Oct 2026 20:06:49 +0800 Subject: [PATCH 09/15] 0.3.3 --- the proxy hop is verified like the target hop, and says why it was refused What the certificate fix and the proxy branch need of each other beyond the text the merge resolved: * TlsSocket::fail() closes the transport with drop_transport(), so a handshake that fails inside a tunnel closes the session to the proxy as well, not only the descriptor a tunnelled session does not use. * connect_over(lower) clears the previous failure reason, as connect() and connect_over(Socket&&) do. * A TLS connection to an https:// proxy that fails reports the session's reason after the hop, `proxy: could not open a TLS connection to : certificate verification failed: ...`. Without it the refusal #21 exists to explain reached the caller with the reason dropped. test_proxy gains the case: an https proxy whose certificate has no trusted issuer is refused with the reason, and receives no CONNECT. It fails without the proxy.cppm change. The version is 0.3.3 rather than 0.4.0. mcpp reads `tinyhttps = "0.3.0"` as ^0.3.0, which stops below 0.4.0, so only a patch release reaches the consumers that need the verification fix without editing a manifest. CHANGELOG states the fail-closed behaviour change first. Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- CHANGELOG.md | 23 +++++++++++++++- README.md | 4 +-- mcpp.toml | 2 +- src/proxy.cppm | 3 +++ src/tls.cppm | 5 +++- tests/test_proxy.cpp | 64 ++++++++++++++++++++++++++++++++++++++++++++ xmake.lua | 2 +- 7 files changed, 97 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 026770c..09a8f7d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,14 @@ # Changelog -## Unreleased +## 0.3.3 + +**Behaviour change, and the reason for this release:** `verifySsl = true` (the +default) now verifies the server certificate, and with no CA bundle to verify +against a connection fails instead of going ahead unverified +([#20](https://github.com/mcpplibs/tinyhttps/issues/20)). It is a patch release +so that every dependency written `tinyhttps = "0.3.x"` receives it. + +### Certificate verification `verifySsl = true` now verifies the server certificate. The handshake used to complete with any certificate (a self-signed one, one for another host, an @@ -20,6 +28,13 @@ result afterwards. certificate store that are usable for TLS servers are read. That code is built only for Windows Sockets targets, not for the musl one. MinGW builds need `-lcrypt32`, as they already need `-lws2_32`. +* `examples/openkal` exits 1 when the certificate is refused; it used to read + every failure as an absence of network. CI builds and runs the Windows store + reader on windows-2022 under msvc and llvm (`tests/test_ca_store.cpp`). +* MSVC builds link `bcrypt`, from which mbedTLS draws entropy; its package asks + for it as `-lbcrypt`, which link.exe does not read. + +### Proxies Proxy URLs carry credentials, and a proxy that refuses the tunnel says why. @@ -58,6 +73,12 @@ HTTP. And every refusal, whatever the proxy said, reached the caller as in `statusText` the same way as for an HTTP proxy. The default port is 1080. SOCKS4 is not supported and, like any unknown scheme, is an error. * `Socket::resolve_address` returns the address a name resolves to as raw bytes. +* The connection to an `https://` proxy is verified like the connection to the + target, and a refusal names both the hop and the reason, for example + `proxy: could not open a TLS connection to proxy:443: certificate + verification failed: ...`. The CONNECT, and any credentials, are sent only + after the proxy's certificate has verified. +* A failed handshake inside a tunnel closes the tunnel's session as well. ## 0.3.2 diff --git a/README.md b/README.md index 8e75009..732e63d 100644 --- a/README.md +++ b/README.md @@ -159,14 +159,14 @@ beneath and not from this library: ### 添加依赖 ```bash -mcpp add tinyhttps@0.3.2 +mcpp add tinyhttps@0.3.3 ``` 或在 `mcpp.toml` 中手动添加： ```toml [dependencies] -tinyhttps = "0.3.2" +tinyhttps = "0.3.3" ``` ### 构建 diff --git a/mcpp.toml b/mcpp.toml index 5ee04a8..e40daad 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "tinyhttps" -version = "0.3.2" +version = "0.3.3" description = "Minimal C++23 HTTP/HTTPS client with SSE streaming support" license = "Apache-2.0" repo = "https://github.com/mcpplibs/tinyhttps" diff --git a/src/proxy.cppm b/src/proxy.cppm index c2061c9..e7c5380 100644 --- a/src/proxy.cppm +++ b/src/proxy.cppm @@ -367,7 +367,10 @@ export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, } else if (proxy.scheme == "https") { auto tls = std::make_unique(); if (!tls->connect(proxy.host.c_str(), proxy.port, timeoutMs, verifySsl)) { + // The session says why when the TCP connection was up, which is + // where a proxy certificate that does not verify is refused. tunnel.error = "proxy: could not open a TLS connection to " + where; + if (!tls->error().empty()) tunnel.error += ": " + tls->error(); return tunnel; } tunnel.error = http_connect(*tls, proxy, targetHost, targetPort, timeoutMs); diff --git a/src/tls.cppm b/src/tls.cppm index 18f8bc3..34c8e55 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -173,6 +173,7 @@ public: // inside it, then this session to the target inside the tunnel. Takes // ownership of `lower`, which must already be past its CONNECT. bool connect_over(std::unique_ptr lower, const char* host, bool verifySsl) { + error_.clear(); lower_ = std::move(lower); return setup_tls(host, verifySsl); } @@ -277,10 +278,12 @@ private: std::unique_ptr state_; std::string error_; + // The session beneath this one, when there is one, is closed as well: a + // failed handshake to the target leaves nothing of the tunnel open. bool fail(std::string message) { error_ = std::move(message); state_.reset(); - socket_.close(); + drop_transport(); return false; } diff --git a/tests/test_proxy.cpp b/tests/test_proxy.cpp index 332f02f..ca1d92f 100644 --- a/tests/test_proxy.cpp +++ b/tests/test_proxy.cpp @@ -9,6 +9,8 @@ #include "proxy_test_server.hpp" #include "tls_test_server.hpp" +#include + import mcpplibs.tinyhttps; import std; @@ -324,6 +326,44 @@ tls_test::Server tls_connect_proxy(Seen& seen, int targetPort, std::string expec } // namespace +namespace { + +// Makes one certificate the only CA bundle the client reads, for the lifetime of +// the object, and restores what was there before. +class TrustOnly { +public: + explicit TrustOnly(std::string_view pem) + : path_(std::filesystem::temp_directory_path() + / ("tinyhttps-proxy-trust-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + ".pem")) { + if (const char* old = std::getenv("SSL_CERT_FILE")) { hadOld_ = true; old_ = old; } + std::ofstream(path_, std::ios::binary) << pem; + set(path_.string().c_str()); + } + ~TrustOnly() { + set(hadOld_ ? old_.c_str() : ""); + std::error_code ec; + std::filesystem::remove(path_, ec); + } + TrustOnly(const TrustOnly&) = delete; + TrustOnly& operator=(const TrustOnly&) = delete; + +private: + static void set(const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s("SSL_CERT_FILE", value); +#else + ::setenv("SSL_CERT_FILE", value, 1); +#endif + } + std::filesystem::path path_; + std::string old_; + bool hadOld_ { false }; +}; + +} // namespace + TEST(ProxyUrl, AnHttpsProxyDefaultsToPort443) { auto p = https::parse_proxy_url("https://proxy.example"); EXPECT_EQ(p.scheme, "https"); @@ -347,6 +387,30 @@ TEST_F(ProxyTest, ARequestGoesThroughAnHttpsProxy) { EXPECT_TRUE(heads[0].starts_with("CONNECT 127.0.0.1:" + std::to_string(target.port()) + " HTTP/1.1\r\n")); } +// The proxy hop is verified like the target hop, and its refusal names the hop +// and the reason. Only the wrong-name certificate is trusted, so the proxy's own +// certificate has no issuer the client knows; nothing, the CONNECT included, +// reaches a peer that did not verify. +TEST_F(ProxyTest, AnHttpsProxyWhoseCertificateDoesNotVerifyIsRefusedWithTheReason) { + auto target = hello_server(); + Seen seen; + auto proxy = tls_connect_proxy(seen, target.port()); + ASSERT_FALSE(proxy.failed()); + TrustOnly bundle(tls_test::kWrongNameCertPem); + + auto cfg = proxied("https://127.0.0.1:" + std::to_string(proxy.port())); + cfg.verifySsl = true; + https::HttpClient client(cfg); + auto resp = client.send(get(target.url("/"))); + + EXPECT_EQ(resp.statusCode, 0); + EXPECT_TRUE(resp.statusText.starts_with("proxy: could not open a TLS connection to 127.0.0.1:")) + << resp.statusText; + EXPECT_NE(resp.statusText.find("certificate verification failed"), std::string::npos) + << resp.statusText; + EXPECT_TRUE(seen.get().empty()); +} + TEST_F(ProxyTest, CredentialsGoToAnHttpsProxyInsideItsTlsSession) { auto target = hello_server(); Seen seen; diff --git a/xmake.lua b/xmake.lua index 36a9d5a..fd21acc 100644 --- a/xmake.lua +++ b/xmake.lua @@ -7,7 +7,7 @@ add_requires("gtest 1.15.2", { optional = true }) target("tinyhttps") set_kind("static") - set_version("0.3.0") + set_version("0.3.3") add_files("src/*.cppm", { public = true, install = true }) add_packages("mbedtls", { public = true }) set_policy("build.c++.modules", true) From 156edebbb138cf07dc8e0a473a4d909dc43d70d4 Mon Sep 17 00:00:00 2001 From: Cloud_Yun Date: Fri, 2 Oct 2026 05:28:26 +0900 Subject: [PATCH 10/15] 0.3.4 --- a request in flight can be cancelled with a std::stop_token (#23) send, send_stream and download_to_file take an optional std::stop_token as their last argument. Stopping it from another thread ends the call within about 50 ms and closes the connection; HttpResponse::cancelled and DownloadToFileResult::cancelled say so. A cancelled request is not retried and its redirect is not followed. Without a token nothing changes. With a token every wait on the socket is a poll in slices of at most 50 ms, the connect is checked before each address, and bio_recv waits before recv, so the TLS handshake and an HTTP, https or SOCKS5 proxy's exchange are covered. The token belongs to its call: it is set on the socket at the bottom of the connection (beneath an https:// proxy's tunnel, where the waits are) and taken off when the connection goes back to the pool. Where the C library is musl the TLS random generator is seeded from getentropy; above openkal on x86_64-windows-musl mbedTLS's /dev/urandom does not exist and every handshake failed. CI: Linux under gcc 16 and llvm 22, macOS arm64, the hermetic tests on Windows (msvc, llvm), and examples/openkal on x86_64-linux-gnu, x86_64-linux-musl, aarch64-linux-musl and x86_64-windows-musl. mcpp 2026.9.30.2. Design and plan: .agents/docs/design-cancel-stop-token.md, .agents/docs/exec-plan-0.3.4-cancel.md. --- .agents/docs/design-cancel-stop-token.md | 216 ++++++ .agents/docs/exec-plan-0.3.4-cancel.md | 101 +++ .github/workflows/ci.yml | 197 +++++- CHANGELOG.md | 59 ++ README.md | 45 +- examples/openkal/mcpp.toml | 10 +- examples/openkal/src/main.cpp | 73 ++ mcpp.toml | 7 +- src/http.cppm | 143 +++- src/proxy.cppm | 5 +- src/socket.cppm | 39 +- src/tls.cppm | 50 ++ tests/test_cancel.cpp | 812 +++++++++++++++++++++++ tests/test_pool.cpp | 4 +- tests/tls_test_server.hpp | 18 + 15 files changed, 1716 insertions(+), 63 deletions(-) create mode 100644 .agents/docs/design-cancel-stop-token.md create mode 100644 .agents/docs/exec-plan-0.3.4-cancel.md create mode 100644 tests/test_cancel.cpp diff --git a/.agents/docs/design-cancel-stop-token.md b/.agents/docs/design-cancel-stop-token.md new file mode 100644 index 0000000..671a1fa --- /dev/null +++ b/.agents/docs/design-cancel-stop-token.md @@ -0,0 +1,216 @@ +# 设计方案：请求取消（`std::stop_token`），基于 PR #23 + +| | | +|---|---| +| 基线 | `origin/master @ f461940`（0.3.3） | +| 输入 | [PR #23](https://github.com/mcpplibs/tinyhttps/pull/23) `feat/cancel @ 8996984`（作者 yspbwx2010，`maintainerCanModify: true`） | +| 落地方式 | 在 PR #23 上追加 commit，不另开 PR | +| 文档日期 | 2026-10-02 | + +**标注约定**：✅ 已核验源码 · 🔁 已在本地复现或跑通 · ❓ 推断，未实测 + +--- + +## 0. TL;DR + +- **应该进库。** 库外做不到：socket 是 `HttpClient` 的私有状态，`HttpClient` 本身又不是线程安全的。库里也已经有半个先例：`download_to_file` 的 `isCancelled` 只在 body 的块与块之间检查。 +- **PR #23 的机制是对的，保留：** 等待只发生在 `Socket::wait_fd`，有 token 时按 50 ms 分片 `poll`；`bio_recv` 读之前先等。**它缺的是 token 的生命周期规则**，H1 就是这个缺口造成的 bug：经 https 代理复用连接时，一个 POST 会被执行两次。 +- **架构收敛成三条不变式：** token 只放在传输链最底层的 `Socket` 上；token 只在调用期间挂着，连接回池时摘掉；不传 token 时行为和 0.3.3 完全一样。 +- **改动量：** 在 PR 基础上，`src/` 约 +15 行（必需部分 +7 行），另加测试和 CI。必需部分已经做成原型跑通（§7）。 +- **CI 目前对这个功能只有一个配置有效：** Linux + gcc 16。Windows job 不编译 `test_cancel`；Linux libc++（llvm 20/22）连链接都过不了；没有 macOS job。 + +### 决策（2026-10-02 均按建议采纳，执行见 [exec-plan-0.3.4-cancel.md](exec-plan-0.3.4-cancel.md)） + +| # | 决策 | 建议 | +|---|---|---| +| D1 | `download_to_file` 是否同时加 `std::stop_token` 和 `DownloadToFileResult::cancelled` | **加**。理由见 §3.3 | +| D2 | 删掉 PR 给 `proxy_connect` 加的 `stop` 参数 | **删**。它是遗留的包装函数，只有 `proxy_tunnel` 内部需要 token | +| D3 | CI 新增 Linux llvm@22 job、Windows 跑 hermetic 测试 | **加**。macOS 视 mcpp 支持情况定（❓） | +| D4 | 写操作阻塞（对端不读）仍不可取消 | **本 PR 不做**，列为已知限制。后续单独做非阻塞发送 | + +--- + +## 1. 该不该进库 + +**结论：进。** + +| 论点 | 说明 | +|---|---| +| 库外无解 | 调用方拿不到 fd。唯一的绕法是把 `send` 丢到 detached 线程里等 `readTimeoutMs`。这期间 `HttpClient` 不能再用（不是线程安全的，`http.cppm` 里 `HttpClient` 的注释写明了），连接也一直占着，服务器还在继续生成响应 | +| 已有先例 | `download_to_file(..., isCancelled)` ✅ 说明库已经承认取消是它的职责。但 `isCancelled` 只在 body 块之间检查，等响应头、TLS 握手这些阶段照样卡死，正是 PR 要解决的问题 | +| 标准词汇 | `std::stop_token` 是 C++20 标准类型，和 `std::jthread` 天然配合，今后也能对接 `std::execution` 的 stop token。不需要自造回调或传 `atomic*` | +| 成本可控 | 不传 token 时零开销（I3）。传了 token 时，每个进行中的调用每秒多 20 次 `poll` 唤醒 | + +**反方意见及回应：**“tiny 库不该长功能”。这里不是加功能，而是补全已有的取消语义（`isCancelled` 覆盖不全），并让三个入口一致。这和 #15 合并三个入口、只保留一份读取逻辑是同一个思路。 + +--- + +## 2. 架构：三条不变式 + +``` +HttpClient::send / send_stream / download_to_file ← 唯一接收 token 的地方 + │ perform_exchange: sock->set_stop(stop) ← 挂上（每次调用、每次重试） + │ PooledConnection::keep(): set_stop({}) ← 回池时摘下；drop() 直接销毁 + ▼ +TlsSocket::set_stop ──转发──► lower_（https 代理隧道）──转发──► Socket + │ 自己不存 token + ▼ +Socket::stop_ ── wait_fd()：分片 poll，片与片之间检查 stop_requested() + ── bio_recv：有 token 时先 wait_readable(-1) 再 recv + ── connect：每个地址、每次 DNS 回退前检查 +``` + +| 不变式 | 内容 | 违反时的后果 | +|---|---|---| +| **I1 只存在底层** | token 只存在于传输链最底层的 `Socket`。`TlsSocket` 不存 token，只往下转发（不管下面是 `socket_` 还是 `lower_`） | **H1** 🔁：https 代理下 `set_stop` 只设了用不到的 `socket_`，复用的隧道一直带着第一个请求的 token | +| **I2 只在调用期间存在** | 进入 exchange 时挂上，`keep()` 把连接还回池时摘掉。**连接池里的连接永远不带 token** | **M1**：直连情况下现在不会出错（下次请求会覆盖），但只要以后加一个在挂 token 之前等待池中连接的路径（比如空闲探活），就会踩到和 H1 一样的坑 | +| **I3 没有 token 时行为不变** | `stop_possible()==false` 时 `wait_fd` 就是原来的单次 `poll_fd`，`bio_recv` 不预先等待 | 已满足 ✅ | + +**为什么 token 必须作为状态存在 `Socket` 上，而不是作为参数一路传下去：** mbedtls 的 BIO 回调只有一个 `void* ctx`，在这里就是 `Socket*` ✅。握手和 TLS record 的后半段都是从 `bio_recv` 里阻塞的，没有别的通道能把 token 送进去。既然状态没法避免，就用 I1 和 I2 限定它在哪里、活多久。 + +### 考虑过、没有采用的方案 + +| 方案 | 不采用的原因 | +|---|---| +| `HttpClient::cancel()` 成员函数 | 需要从别的线程访问连接池，而 `HttpClient` 不是线程安全的。token 是不加锁就能跨线程传递的唯一渠道 | +| `std::stop_callback` + `shutdown(fd)` | 优点：没有轮询，还能覆盖阻塞的写（D4）。缺点：回调在发起 stop 的线程上执行，会和调用线程里 `drop()` 等所有关闭 fd 的路径竞争，必须加锁；另外 connect 进行中被 `shutdown` 时，macOS、Windows、openkal 上的语义都没有验证 ❓。**留作解决 D4 时的候选方案** | +| 第二个 fd（wake-up pipe）加 `poll` | Windows 没有 pipe；PR 作者说 openkal 上两个 fd 的 `poll` 会忙等（❓，未独立核实） | + +--- + +## 3. 在 PR #23 基础上：保留、修改、删除 + +### 3.1 保留（✅ 已逐项核验） + +- `Socket::wait_fd` 的分片 `poll`（50 ms，固定值，不做成配置项）；`bio_recv` 的预等待；connect 前、每个地址、DNS 回退前的 stop 检查。 +- `perform_exchange` 的 `fail()`：stop 优先于真实错误，并在这里 drop 连接；重试循环开头检查 stop；取消后不跟重定向。 +- `HttpResponse::cancelled`，以及它在响应头到达前后的两种形态（见 §4）。 +- `proxy_tunnel` 尾部的默认参数 `stop`（内部需要）。 +- `Socket` 的 move 会带走 `stop_`；`close()` 不清 `stop_`，所以 `set_stop` 必须在 `connect` 之前调用，现在的顺序是对的 ✅。 + +### 3.2 修改 + +| 项 | 改动 | 行数 | 状态 | +|---|---|---|---| +| **H1** | `TlsSocket::set_stop` 先转发给 `lower_`，再设 `socket_`（I1） | +3 | 🔁 原型通过 | +| **M1** | `PooledConnection::keep()` 把池中连接的 token 设为 `{}`（I2）。`drop()` 直接销毁连接，不用处理 | +5 | 🔁 原型通过 | +| **M2** | `test_cancel.cpp` 里 `import std` 的位置按标准库区分（§6.2） | +6 | 🔁 llvm 22 和 gcc 16 都通过 | +| 文档 | CHANGELOG 的 “Existing callers are unaffected” 改为：普通调用不受影响，但取 `&HttpClient::send` 成员函数指针的代码需要调整 | — | — | + +### 3.3 D1：`download_to_file` 加 token（建议） + +三个入口共用 `perform_exchange` 和 `read_body` ✅。PR 只改了其中两个入口，`download_to_file` 仍然只有 `isCancelled`，又回到了 #15 那张“修复只落到部分入口”的矩阵。具体改法： + +- 签名加尾部默认参数：`download_to_file(url, dest, onProgress = nullptr, isCancelled = nullptr, std::stop_token stop = {})`。 +- `DownloadToFileResult::cancelled`：`isCancelled` 或 token 任一触发都置为 true。`error` 文本保持现有的 `"cancelled"` 不变，原来匹配文本的代码不受影响。 +- 实现：`perform_exchange(..., stop)`；body 的 sink 里检查 stop；重定向递归时把 stop 传下去。预计约 +15 行。 +- `isCancelled` 保留，不标记废弃。它是轮询式谓词，和 token 不冲突。 + +### 3.4 D2：删除（建议） + +- `proxy_connect(..., std::stop_token)`：这是只返回 `Socket` 的遗留包装函数，库内部不用它。加参数只会扩大导出面（YAGNI）。 + +--- + +## 4. 语义契约（README 和 CHANGELOG 要写清楚） + +| 情形 | `statusCode` / `statusText` | `bodyComplete` / `bodyError` | `cancelled` | 连接 | +|---|---|---|---|---| +| 调用开始前 token 已经 stop | `0` / `"Cancelled"` | false / `"Cancelled"` | true | 不发起任何连接 | +| 响应头到达前取消（connect、代理、握手、等响应头） | `0` / `"Cancelled"` | false / `"Cancelled"` | true | drop | +| 响应头到达后取消（读 body、流式回调间隙） | 服务器给的值 | false / `"cancelled"` | true | drop | +| token 从未 stop | 和 0.3.3 一样 | 和 0.3.3 一样 | false | 和 0.3.3 一样 | + +其他规则： + +- `ok()` 不看 `cancelled`，就像它也不看 `bodyComplete`。取消的契约是那个 bool，不是文本。 +- 取消后不做 stale-connection 重试，也不跟重定向。 +- 真实失败和 stop 同时发生时，报告为取消。 +- 响应时间约 50 ms（一个分片）。 +- `close_notify` 只在握手完成之后的取消中发送。 + +**已知限制：**`getaddrinfo` 和 `socks5://` 对目标地址的解析不可中断；对端不读导致的阻塞写不可中断（socket 在 connect 之后被设回阻塞模式，`socket.cppm:257` ✅，见 D4）；openkal 上 connect 是同步完成的。 + +--- + +## 5. 测试 + +PR 已有 13 个用例，覆盖各阶段的首次建连 ✅。**缺口在“复用连接”这一维度**，H1 就漏在这里。补成下面这个矩阵： + +| 新增用例 | 覆盖 | 状态 | +|---|---|---| +| 经 https 代理复用：第一个请求结束后它的 token 被 stop，第二个 POST 在目标服务器上**只执行一次**，代理只被连一次 | H1 / I1 | 🔁 修复前 3 次请求、2 次代理连接；修复后通过 | +| 经 https 代理复用：第二个请求自己的 token 在 500 ms 内生效 | H1 / I1 | 🔁 修复前要等满 3000 ms 读超时 | +| 同样的场景走 http:// 代理和直连（对照组） | I1 | 🔁 通过 | +| README 的 jthread 写法：请求完成后 jthread 析构（会 `request_stop`），下一个请求复用连接并成功 | I2 | 🔁（https 代理那一组就是这个写法） | +| `download_to_file`：等响应头时取消、读 body 时取消；`cancelled==true`；目标文件的处理和现有 `isCancelled` 一致 | D1 | 待写 | +| SOCKS5 代理在等握手应答时取消 | 补齐代理种类 | 待写（PR 只覆盖了 http 和 https 代理） | + +时间断言使用 `kPromptMs = 500`。Windows 和 macOS 的 runner 比较慢，如果出现偶发失败，再按平台放宽，不要预先放宽。 + +--- + +## 6. CI 覆盖 + +### 6.1 现状 ✅ + +| job | 工具链 | 跑什么 | 对本功能的覆盖 | +|---|---|---|---| +| linux mcpp | gcc 16.1.0（mcpp 首次运行时引导安装的默认工具链） | 全部 `mcpp test` | ✅ 唯一有效的覆盖 | +| openkal | llvm + openkal libc++ | `examples/openkal` 的 `mcpp run` | 只编译库，不跑 `test_cancel` | +| windows msvc / llvm@20.1.7 | MSVC STL | **只跑 `mcpp test test_ca_store`** | ❌ 根本不编译 `test_cancel` | +| linux libc++ | — | — | ❌ 而且 `test_cancel` 在 llvm 20/22 上链接失败（§6.2） | +| macOS | — | — | ❌ 但 README 声称支持 macOS | + +### 6.2 libc++ 链接问题（M2）🔁 + +- 在 llvm 20.1.7 和 22.1.8（libc++）上，先 `#include ` 再 `import std;`，然后调用 `stop_source::request_stop()`，链接时报 `undefined hidden symbol __atomic_unique_lock<...>::__set_locked_bit`。llvm 23.1.0 和 gcc 16 正常。这是工具链 bug，不是本库的问题。 +- 仓库测试文件的惯例是先 include 后 import，因为 gcc 必须这样：反过来在 gcc 16 上会报 `redefinition of std::byte` 等错误 🔁。 +- 解决办法：在 `test_cancel.cpp` 里按标准库区分顺序（已验证 llvm 22 和 gcc 16 都能 16/16 通过）： + +```cpp +// libc++ (≤22) fails to link stop_source::request_stop() when +// was included textually before `import std`; libstdc++ needs the opposite. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif +``` + +- llvm 20 在 Linux 上两种顺序都编不过（import 在前时报 `requires clause differs`）🔁，显式实例化 `__atomic_unique_lock` 也不行 🔁，因此 **libc++ 只把 22 纳入 CI 矩阵**（Linux 和 macOS 都用 llvm@22.1.8；Windows 上的 llvm@20.1.7 用的是 MSVC STL，不受影响）。 +- README 的 jthread 示例对“include 和 import std 混用、用 libc++ ≤22”的使用者同样会链接失败，需要在 README 里加一行说明。 + +### 6.3 目标 + +| 改动 | 内容 | 状态 | +|---|---|---| +| Linux job 改成矩阵 `[gcc@16.1.0, llvm@22.1.8]`，都跑全部 `mcpp test` | 覆盖 libc++。llvm 22 也是 mcpp 本地常见的默认工具链 | 🔁 本地 llvm 22 + M2 修复通过 | +| Windows 两个 job 追加 `mcpp test test_cancel`，最好再加 `test_framing`、`test_pool`、`test_proxy` | 覆盖 `WSAPoll` 分片和 Windows 上 connect 的等待 | ❓ 测试服务器有 `_WIN32` 分支（`proxy_test_server.hpp:28`），但从没在 Windows 上编译过，先试，编不过再移植 | +| 新增 macOS job（macos-14），跑 hermetic 测试 | PR 作者明确说没在 macOS 上跑过；macOS 上 `poll` 对进行中的 connect 的语义需要实测 | ❓ 要先确认 mcpp 的 macOS 工具链可用。如果不可用，就作为 follow-up 单独开 issue，不阻塞本 PR | +| openkal | 不变。库被编进 example，签名的编译已经覆盖到 | — | + +--- + +## 7. 落地：在 PR #23 上追加的 commit + +原型补丁（H1 + M1 + M2 + 3 个复现测试）已经在 PR 分支上跑通：`test_cancel` 16/16、`test_pool`、`test_proxy`、`test_framing` 通过（gcc 16），`test_cancel` 在 llvm 22 上也通过。之前那一轮（H1 + 复现测试）的全量 `mcpp test` 是 8/8 🔁。 + +| # | commit | 依赖 | +|---|---|---| +| 1 | `fix(tls): a stop token reaches the session a tunnelled one runs over`：H1，以及 https/http/直连三组复用测试 | — | +| 2 | `fix(http): a pooled connection carries no stop token`：M1，以及 jthread 析构场景的测试 | 1 | +| 3 | `test(cancel): import std before the includes under libc++`：M2 | — | +| 4 | `feat(download): download_to_file takes a stop token`：D1，以及 `DownloadToFileResult::cancelled` 和测试 | 2，D1 批准后 | +| 5 | `refactor(proxy): proxy_connect does not take a stop token`：D2 | D2 批准后 | +| 6 | `ci: linux on llvm 22 as well; windows runs the hermetic tests`（再加 macOS job，如果可行）：D3 | 3 | +| 7 | `docs: cancellation contract, download_to_file, libc++ note` | 4、5 | + +顺序说明：1 和 2 是正确性修复，最先落地；3 必须在 6 之前，否则新加的 llvm job 第一次跑就会失败；6 失败时（比如 Windows 测试服务器不可移植），可以把 Windows 部分拆成 follow-up，不阻塞合并。 diff --git a/.agents/docs/exec-plan-0.3.4-cancel.md b/.agents/docs/exec-plan-0.3.4-cancel.md new file mode 100644 index 0000000..390e20f --- /dev/null +++ b/.agents/docs/exec-plan-0.3.4-cancel.md @@ -0,0 +1,101 @@ +# 执行计划 —— 0.3.4：在途请求可取消，三个入口一致，CI 覆盖库支持的全部平台 + +| | | +|---|---| +| 输入 | [`design-cancel-stop-token.md`](design-cancel-stop-token.md)：D1–D4 均按建议采纳 | +| 基线 | `origin/master @ f461940`（0.3.3） | +| 落地 | 全部做在 PR #23 上（`yspbwx2010:feat/cancel`，`maintainerCanModify: true`），squash 合入 | +| 目标版本 | **0.3.4** | +| 跨仓库 | `mcpplibs/tinyhttps` → gitcode `mcpp-res/tinyhttps`（CN 镜像）→ `mcpplibs/mcpp-index` | + +--- + +## 0. 为什么是 0.3.4 + +- 只新增，不改已有行为：`send`、`send_stream`、`download_to_file` 都只加了尾部默认参数；`HttpResponse` 和 `DownloadToFileResult` 各在末尾追加一个字段；不传 token 时逐字节不变（不变式 I3）。 +- 依赖写成 `tinyhttps = "0.3.x"` 的项目会自动解析到它，这是 0.3.3 立下的规则：让现有依赖不改清单就能收到。mcpp-index 的 `tests/examples/tinyhttps` 写的是 `0.3.1`，也会解析到它。 +- 唯一的源码不兼容点是对 `&HttpClient::send` 取成员函数指针（参数个数变了）。在 CHANGELOG 里点名说明。 + +## 1. 八个视角 + +| 视角 | 要求 | 落在哪 | +|---|---|---| +| **架构** | token 存在哪里、存活多久要有规则，不能靠每个调用点自己记得 | I1：只存在传输链最底层的 `Socket`，`TlsSocket` 只往下转发。I2：只在一次调用期间挂着，`PooledConnection::keep()` 把连接还回池时摘下 | +| **稳定性** | 取消不能引发重复执行、fd 泄漏或卡死 | 取消后不重试、不跟重定向；连接 drop；fd 计数测试；复用连接测试覆盖直连、http 代理、https 代理三种 | +| **优雅简洁** | 只有一套机制：在 `wait_fd` 里分片 `poll` | 不引入 pipe、shutdown 或回调；不增加配置项；`proxy_connect` 不加参数 | +| **用户体验** | 用 `std::jthread` 就能直接取消；结果用 bool 判断，不用匹配文本 | README 示例；两个结果结构都有 `cancelled`；README 说明 libc++ ≤22 的工具链问题 | +| **兼容性** | 升级不需要改代码 | §0；`HttpResponse{204, "No Content", {}, {}}` 这种聚合初始化仍然能用（mcpp-index 的测试断言了这一点） | +| **跨平台** | Linux（gcc、libc++）、macOS、Windows（msvc、clang）、openkal（linux-gnu、windows-musl）都要实测 | §4 CI 矩阵；openkal 示例增加一段不联网的取消检查 | +| **一致性** | 三个入口取消语义相同 | `download_to_file` 加 token（D1）；`isCancelled` 保留 | +| **无感升级** | 不传 token 时行为和 0.3.3 一样；不需要改依赖版本号 | I3；0.3.4 是补丁版本 | + +## 2. 任务依赖图 + +``` +T0 计划（本文）+ 设计文档纳入 PR + │ + ├─ T1 src：I1（tls.cppm set_stop 转发给 lower_） + │ └─ T2 src：I2（http.cppm PooledConnection::keep 时摘下 token） + │ └─ T3 src：D1 download_to_file(…, stop) + DownloadToFileResult::cancelled + │ └─ T4 src：D2 proxy_connect 去掉 stop 参数；源码注释 + │ + ├─ T5 测试（可与 T1–T4 并行编写，依赖它们才能通过） + │ a. 复用连接矩阵：直连 / http 代理 / https 代理 × {旧 token 已 stop, 新 token 生效} + │ b. SOCKS5 等待握手应答时取消 + │ c. download_to_file：等响应头时取消、读 body 时取消、已 stop 的 token + │ d. libc++ 下 import std 的位置（M2） + │ + ├─ T6 openkal：examples/openkal 增加不联网的取消检查（本地监听 + 握手中取消） + │ 运行时版本跟 mcpp-index 的 pins.toml 对齐；增加 windows-musl（Linux 上构建，windows-2022 上原生运行；本地用 wine） + │ + ├─ T7 CI（依赖 T5d、T6） + │ linux {gcc@16.1.0, llvm@22.1.8} 跑全部测试 · macos-15 llvm@22.1.8 跑全部测试 + │ windows {msvc, llvm@20.1.7} 跑不联网的测试 · openkal {x86_64-linux-gnu, x86_64-windows-musl} + │ + ├─ T8 文档：README、CHANGELOG 0.3.4、mcpp.toml version、README 安装行 + │ + └─ T9 合入与发版（依赖全部） + CI 全绿 → 自审 → squash 合入 → tag 0.3.4 + GitHub release + → 本地 gtc 上传 gitcode mcpp-res/tinyhttps 的 0.3.4 tarball（核对 sha256） + → mcpp-index PR：pkgs/t/tinyhttps.lua 加 0.3.4（三个平台）， + tests/examples/tinyhttps 升到 0.3.4 并加取消的 API 检查 + → 用已发布的 index 实测：新建项目解析到 0.3.4，GLOBAL 和 CN 两个源都验证 +``` + +**关键路径**：T1 → T2 → T5 → T7 → T9。T3/T4、T6、T8 可以和关键路径并行。 + +## 3. 测试清单（test_cancel） + +| 用例 | 覆盖 | +|---|---| +| PR 原有 13 个 | 首次建连各阶段、不重试、重定向、fd 泄漏、connect、http/https 代理 | +| `APooledTunnelCarriesNoTokenFromTheLastCall`，参数为直连 / http / https | I1 + I2，H1 的回归测试（一个 POST 只能执行一次） | +| `TheSecondCallsTokenReachesAReusedTunnel`，参数为直连 / http / https | I1 | +| `ASocks5ProxyThatNeverAnswersIsAbandoned` | 代理种类补齐 | +| `ADownloadWaitingForTheHeadersIsAbandoned` / `…ForTheBody…` / `…AlreadyStopped…` | D1 | + +## 4. CI 矩阵（目标） + +| job | 工具链 | 跑什么 | +|---|---|---| +| linux | gcc@16.1.0 / llvm@22.1.8 | 全部 `mcpp test`（`test_download`、`test_resolver` 需联网）+ 模板冒烟 | +| macos-15 | llvm@22.1.8 | 全部 `mcpp test` | +| windows-2022 | msvc / llvm@20.1.7 | `test_ca_store`，以及不联网的 `test_framing`、`test_pool`、`test_proxy`、`test_cancel`、`test_tls_verify` | +| openkal | llvm@22.1.8 + openkal-llvm-runtime | `examples/openkal` 在 x86_64-linux-gnu 上原生跑、x86_64-windows-musl 在 Linux 上构建、在 windows-2022 上原生跑（CI 装 wine 要 4–30 多分钟，且不稳定）| + +某个平台上测试编不过时，**修测试，不删平台**。确实做不到的，单独开 issue 记下原因，并在 job 里写清楚。 + +## 5. 过程中发现并修复 + +- **x86_64-windows-musl 上 TLS 从来没通过。** mbedTLS 只有识别到 glibc 才用 `getrandom`，否则读 `/dev/urandom`，而 openkal-windows 没有这个文件。在 musl 平台上改用 musl 的 `getentropy`（`cfg(c-abi = "musl")` → `TINYHTTPS_GETENTROPY`）🔁。 +- **Windows 目标没有名字解析**（openkal 没有解析器接口），已登记 [openkal-musl#46](https://github.com/mcpplibs/openkal-musl/issues/46)，README 已注明。 +- **`test_pool` 在 Windows 上编不过**（`` 的 `max` 宏），因为以前从没在 Windows 上编译过；已修复。 + +## 6. 验收 + +- [ ] PR #23 的 CI 全绿（上表全部 job） +- [ ] 自审：H1、M1、M2、D1、D2 全部落地；PR 描述更新 +- [ ] squash 合入；tag `0.3.4`；GitHub release +- [ ] gitcode 上的 `tinyhttps-0.3.4.tar.gz` 与 GitHub archive 的 sha256 一致 +- [ ] mcpp-index 合入；`validate` 和 `openkal-compat`（tinyhttps 这一项）通过 +- [ ] 新建项目用已发布的 index 解析到 0.3.4，GLOBAL 和 CN 两个源都能构建并运行 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 724b69e..0b2e4ac 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -31,15 +31,30 @@ on: workflow_dispatch: env: - MCPP_VERSION: 2026.9.28.3 + # 2026.9.30.2 and not earlier: 2026.9.28.3 refused x86_64-windows-musl + # ("cannot be built on this host") before the dependency graph that supplies + # its system side, openkal-llvm-runtime, had been resolved. + MCPP_VERSION: 2026.9.30.2 XLINGS_VERSION: v2026.8.17.2 XLINGS_NON_INTERACTIVE: '1' jobs: + # ── Linux, under both standard libraries ───────────────────────────────────── + # + # WHY TWO TOOLCHAINS. The job used to run whatever mcpp installed by default on + # a fresh runner, which is gcc and libstdc++, and nothing ran the tests under + # libc++. libc++ is the standard library of every other job below — macOS and + # openkal — and of mcpp's own default where a developer has chosen llvm; the + # cancellation tests met a libc++ defect that libstdc++ does not have + # (tests/test_cancel.cpp, at the top), and no job here could have seen it. build: - name: build + test (linux x86_64, mcpp) + name: build + test (linux x86_64, ${{ matrix.toolchain }}) runs-on: ubuntu-latest timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + toolchain: ['gcc@16.1.0', 'llvm@22.1.8'] steps: - uses: actions/checkout@v4 @@ -67,21 +82,28 @@ jobs: mcpp self config --mirror GLOBAL # The toolchain mcpp bootstraps is decided by the version pinned above, so - # that version is the whole of the key. + # that version and the toolchain are the whole of the key. - name: Cache mcpp sandbox uses: actions/cache@v4 with: path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry - key: mcpp-sandbox-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }} + key: mcpp-sandbox-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-${{ matrix.toolchain }} + + - name: Select the toolchain + run: | + spec='${{ matrix.toolchain }}' + mcpp toolchain install "${spec%@*}" "${spec#*@}" + mcpp toolchain default "$spec" + mcpp toolchain list - name: Build with mcpp run: mcpp build - # Most of these are hermetic: `test_framing` needs nothing, and `test_pool` - # scripts its own TLS server on 127.0.0.1. The eight in `test_download` - # and `test_resolver` reach httpbin.org and one.one.one.one; a network - # failure among them is distinguishable in the log by the endpoint it - # names. + # Most of these are hermetic: `test_framing` needs nothing, and `test_pool`, + # `test_proxy` and `test_cancel` script their own servers on 127.0.0.1. The + # eight in `test_download` and `test_resolver` reach httpbin.org and + # one.one.one.one; a network failure among them is distinguishable in the + # log by the endpoint it names. - name: Run tests run: mcpp test @@ -95,6 +117,52 @@ jobs: - name: Smoke-test the project templates run: bash tools/template_smoke.sh + # ── macOS ──────────────────────────────────────────────────────────────────── + # + # THE README NAMES macOS, AND UNTIL THIS JOB NOTHING RAN THERE. The + # differences are real and in this library's own code: SO_NOSIGPIPE rather + # than MSG_NOSIGNAL (src/platform.cppm), /dev/urandom as mbedTLS's entropy, + # the system CA bundle's location, and poll(2) on a socket whose connect is in + # progress, which the cancellation slices wait on. + macos: + name: build + test (macos arm64, llvm@22.1.8) + runs-on: macos-15 + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + + - name: Install xlings + run: | + curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 \ + https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh \ + | bash -s "$XLINGS_VERSION" + echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" + + - name: Install mcpp + run: | + xlings update + xlings install "mcpp@$MCPP_VERSION" -y -g + mcpp --version + mcpp self config --mirror GLOBAL + + - name: Cache mcpp sandbox + uses: actions/cache@v4 + with: + path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry + key: mcpp-sandbox-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-llvm@22.1.8 + + - name: Select the toolchain + run: | + mcpp toolchain install llvm 22.1.8 + mcpp toolchain default llvm@22.1.8 + mcpp toolchain list + + - name: Build with mcpp + run: mcpp build + + - name: Run tests + run: mcpp test + # ── The same sources, on a different kernel ABI ────────────────────────────── # # WHY THIS IS A SEPARATE JOB AND NOT A STEP. It resolves a different toolchain @@ -109,10 +177,24 @@ jobs: # openkal-musl defines the first and not the second. A `#ifdef` that is wrong # about that compiles cleanly here and fails there, which is exactly how # 5e7d66f reached master. + # + # FOUR TARGETS, THE ONES mcpp-index MEASURES tinyhttps ON (its + # tests/openkal/pins.toml). aarch64-linux-musl runs under qemu, through the + # runner examples/openkal names. x86_64-windows-musl is built here and run on + # Windows itself by the job after this one: wine took anywhere from four to + # more than thirty minutes to install on these runners, and Windows is the + # system the program is for. The Windows target is where every handshake used + # to fail for want of entropy (src/tls.cppm); it has no name resolution, so its + # network step reports "no network" and the cancellation check, which needs + # none, is what runs. openkal: - name: build + run (linux x86_64, above openkal) - runs-on: ubuntu-latest + name: build + run (${{ matrix.target }}, above openkal) + runs-on: ubuntu-24.04 timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + target: ['x86_64-linux-gnu', 'x86_64-linux-musl', 'x86_64-windows-musl', 'aarch64-linux-musl'] steps: - uses: actions/checkout@v4 @@ -130,32 +212,101 @@ jobs: mcpp --version mcpp self config --mirror GLOBAL + - name: Install the runner + if: matrix.target == 'aarch64-linux-musl' + timeout-minutes: 15 + env: + DEBIAN_FRONTEND: noninteractive + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq --no-install-recommends qemu-user > /dev/null + qemu-aarch64 --version | head -1 + - name: Cache mcpp sandbox uses: actions/cache@v4 with: path: ~/.xlings/data/xpkgs/xim-x-mcpp/${{ env.MCPP_VERSION }}/registry - key: mcpp-openkal-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }} + key: mcpp-openkal-${{ runner.os }}-mcpp${{ env.MCPP_VERSION }}-${{ matrix.target }} # The example builds the library from this checkout (`path = "../.."`) and # takes the stack beneath it from the index, so what is tested is this # commit against the published openkal packages. # - # It makes one HTTPS request and reports "no network" rather than failing - # when there is none, so a runner without egress reports "not run" instead - # of "broken". A certificate the client refused is a failure, not an - # absence of network. The build, the link and the framing parsers are - # checked either way. + # The toolchain is named, as mcpp-index's measurement names it, and + # installed beforehand: on a fresh runner x86_64-windows-musl was refused + # as "cannot be built on this host" while llvm was not yet installed, though + # the Linux targets, which install it on demand, built. + # + # It cancels a handshake against a local listener, which needs no network, + # and then makes one HTTPS request; it reports "no network" rather than + # failing when there is none, so a runner without egress reports "not run" + # instead of "broken". A certificate the client refused is a failure, not + # an absence of network. The build, the link, the framing parsers and the + # cancellation are checked either way. + - name: Install llvm + run: mcpp toolchain install llvm 22.1.8 + - name: Build and run above openkal + if: matrix.target != 'x86_64-windows-musl' working-directory: examples/openkal - run: mcpp run + run: | + set -o pipefail + mcpp run --toolchain llvm@22.1.8 --target '${{ matrix.target }}' 2>&1 | tee run.log + grep -q '^cancellation: ok' run.log - # ── The Windows ROOT certificate store ─────────────────────────────────────── + # A static PE executable that imports only system DLLs, so it runs on any + # Windows as it is. + - name: Build for Windows above openkal + if: matrix.target == 'x86_64-windows-musl' + working-directory: examples/openkal + run: | + mcpp build --toolchain llvm@22.1.8 --target x86_64-windows-musl + mkdir -p "$RUNNER_TEMP/smoke" + cp target/x86_64-windows-musl/*/bin/smoke.exe "$RUNNER_TEMP/smoke/" + + - uses: actions/upload-artifact@v4 + if: matrix.target == 'x86_64-windows-musl' + with: + name: smoke-x86_64-windows-musl + path: ${{ runner.temp }}/smoke/smoke.exe + if-no-files-found: error + + openkal-windows: + name: run (x86_64-windows-musl, above openkal, on Windows) + needs: openkal + runs-on: windows-2022 + timeout-minutes: 15 + defaults: + run: + shell: bash + steps: + - uses: actions/download-artifact@v4 + with: + name: smoke-x86_64-windows-musl + + # Above openkal the Windows certificate store is not read (README, + # "Platforms"), so the program is given a bundle: Git for Windows's, copied + # beside it, because openkal resolves a relative name against the working + # directory and names no absolute path. + - name: Run above openkal + run: | + set -o pipefail + cp "/c/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt" ca-bundle.crt + SSL_CERT_FILE=ca-bundle.crt ./smoke.exe 2>&1 | tee run.log + grep -q '^cancellation: ok' run.log + + # ── Windows ────────────────────────────────────────────────────────────────── # # WHY THIS JOB EXISTS. Windows keeps no bundle file, so on a Windows Sockets # build `load_ca_certs` reads the system's ROOT store through the Win32 API # (src/ca_bundle.cppm). That code is compiled on no other platform, and a job # elsewhere can show neither that it builds nor that the roots it returns # verify a real certificate chain. `test_ca_store` asserts both. + # + # AND THE HERMETIC TESTS RUN HERE TOO. They script their own servers on + # 127.0.0.1, and Windows Sockets is where `poll` is `WSAPoll`, a connect in + # progress is reported differently, and the cancellation slices were never + # run before this job ran them. windows: name: build + test (windows x86_64, ${{ matrix.toolchain }}) runs-on: windows-2022 @@ -197,3 +348,11 @@ jobs: run: | mcpp test test_ca_store 2>&1 | tee tests.log grep -q '^test_ca_store \.\.\. ok' tests.log + + - name: The hermetic tests + run: | + set -o pipefail + for t in test_framing test_tls_verify test_pool test_proxy test_cancel; do + mcpp test "$t" 2>&1 | tee "$t.log" + grep -q "^$t \.\.\. ok" "$t.log" + done diff --git a/CHANGELOG.md b/CHANGELOG.md index 09a8f7d..64f3182 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,64 @@ # Changelog +## 0.3.4 + +A request in flight can be abandoned from another thread. Everything is added +and nothing changes without a token, so it is a patch release that every +dependency written `tinyhttps = "0.3.x"` receives. + +### Cancellation + +`send`, `send_stream` and `download_to_file` take an optional `std::stop_token` +as their last argument ([#23](https://github.com/mcpplibs/tinyhttps/pull/23)). + +* When the token is stopped the call returns within about 50 ms, and the + connection is closed instead of being returned to the pool. It covers the + connect, the exchange with an HTTP, https or SOCKS5 proxy, the TLS handshake, + waiting for the response head, reading the body and the gaps between + streaming callbacks. +* `HttpResponse::cancelled` says so. Before the status line arrives the response + is `statusCode` 0 with `statusText` and `bodyError` `Cancelled`; after it, + `statusCode` is the server's and `bodyError` is `cancelled`. `ok()` does not + look at `cancelled`, as it does not look at `bodyComplete`. +* `DownloadToFileResult::cancelled` is the same for a download, set by the token + or by `isCancelled`, with `error` `Cancelled` or `cancelled` as above. A + destination that failed first is reported as `writeFailed` instead. +* A cancelled request is not retried on a new connection, and its redirect is + not followed. +* A token belongs to its call. It is set on the socket at the bottom of the + connection (through an https:// proxy, the one beneath the tunnel) and taken + off when the connection goes back to the pool, so a token stopped after its + call returned — `std::jthread` stops its own in its destructor — does not + reach the next call on that connection. +* Not covered: name resolution (`getaddrinfo`, and the lookup `socks5://` does + for the target), and a write that is blocked because the server is not + reading. +* `proxy_tunnel` takes the token as a trailing default argument. `Socket` and + `TlsSocket` gain `set_stop`, and `Socket` gains `stop_possible`. +* A call through a pointer to `HttpClient::send`, `send_stream` or + `download_to_file` needs the new parameter in the pointer's type; a call by + name compiles unchanged. +* libc++ 20 and 22 fail to link `std::stop_source::request_stop()` in a file + that includes a standard header such as `` before `import std;`. + Import first in such a file; libc++ 23 and libstdc++ are not affected. + +### openkal + +* Where the C library is musl, the TLS random generator is seeded from musl's + `getentropy` rather than from mbedTLS's reading of `/dev/urandom`. Above + openkal on x86_64-windows-musl that file does not exist, and every handshake + failed with "CTR_DRBG - The entropy source failed". +* `examples/openkal` follows mcpp-index's openkal measurement: runtime 0.15.2, + and a cancellation check against a local listener that needs no network. CI + runs it on x86_64-linux-gnu, x86_64-linux-musl, aarch64-linux-musl (qemu) and + x86_64-windows-musl, which is built on Linux and run on Windows itself. + +### CI + +* Linux runs every test under gcc 16 and under llvm 22 with libc++. +* macOS (arm64, llvm 22) runs every test. Nothing ran on macOS before. +* Windows (msvc and llvm 20) runs the hermetic tests as well as `test_ca_store`. + ## 0.3.3 **Behaviour change, and the reason for this release:** `verifySsl = true` (the diff --git a/README.md b/README.md index 732e63d..cc00774 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,35 @@ else { /* resp.body is all of it */ } `ok()` deliberately does not consult `bodyComplete`, so existing `if (res.ok())` means exactly what it did before. +### Cancelling a request + +Pass a `std::stop_token` to `send`, `send_stream` or `download_to_file` and stop +its source from another thread. The call returns within about 50 ms with +`cancelled` set, and the connection is closed rather than pooled. + +```cpp +HttpResponse response; +std::jthread worker([&](std::stop_token stop) { response = client.send(request, stop); }); +// elsewhere: +worker.request_stop(); +``` + +If the status line had not arrived, `statusCode` is 0 and `statusText` is +`Cancelled`; otherwise they are the server's and `bodyError` is `cancelled`. +As with `bodyComplete`, `ok()` does not look at `cancelled`. A cancelled request +is not retried and its redirect is not followed. `download_to_file` reports the +same in `DownloadToFileResult::cancelled` and `error`, whether the token or its +`isCancelled` callback asked. + +It covers connecting, a proxy's handshake, the TLS handshake, waiting for the +response and reading it. Name resolution and a write blocked on a server that is +not reading cannot be interrupted. Without a token nothing changes. + +With libc++ 20 or 22, a file that includes a standard header such as `` +and then `import std;` fails to link `std::stop_source::request_stop()` (an +inline helper of libc++'s is never emitted). Put `import std;` before the +includes in that file, or use libc++ 23 or libstdc++. + ### Configuration | field | default | what it decides | @@ -152,21 +181,31 @@ beneath and not from this library: refused, and `statusText` says so. - `connectTimeoutMs` does not bound the TCP connect. `kal_net_connect` has no form that begins a connection and reports its outcome later, so the connect - completes or fails before it returns. + completes or fails before it returns. A stop token is checked before the + connect, not during it. +- On Windows there is no name resolution: openkal has no resolver interface, + and musl's reads `/etc/resolv.conf`, which Windows does not have + ([openkal-musl#46](https://github.com/mcpplibs/openkal-musl/issues/46)). A URL + with an address in it connects. + +CI runs `examples/openkal` for x86_64-linux-gnu, x86_64-linux-musl, +aarch64-linux-musl (under qemu) and x86_64-windows-musl (built on Linux, run on +Windows). Locally, +`mcpp run --toolchain llvm@22.1.8 --target x86_64-windows-musl` runs it under wine. ## 使用 mcpp 构建 ### 添加依赖 ```bash -mcpp add tinyhttps@0.3.3 +mcpp add tinyhttps@0.3.4 ``` 或在 `mcpp.toml` 中手动添加： ```toml [dependencies] -tinyhttps = "0.3.3" +tinyhttps = "0.3.4" ``` ### 构建 diff --git a/examples/openkal/mcpp.toml b/examples/openkal/mcpp.toml index f8d89c8..66dc732 100644 --- a/examples/openkal/mcpp.toml +++ b/examples/openkal/mcpp.toml @@ -11,7 +11,7 @@ version = "0.1.0" # sources either way. That is the claim this directory exists to check. [dependencies] tinyhttps = { path = "../.." } -openkal-llvm-runtime = "0.9.1" +openkal-llvm-runtime = "0.15.2" [targets.smoke] kind = "bin" @@ -22,3 +22,11 @@ main = "src/main.cpp" # dependency above. [toolchain] default = "llvm@22.1.8" + +# The programs of the other targets this example is run for, as mcpp-index's +# openkal measurement runs them (tests/openkal/pins.toml there). +[target.x86_64-windows-musl] +runner = ["wine"] + +[target.aarch64-linux-musl] +runner = ["qemu-aarch64"] diff --git a/examples/openkal/src/main.cpp b/examples/openkal/src/main.cpp index decd56d..543439d 100644 --- a/examples/openkal/src/main.cpp +++ b/examples/openkal/src/main.cpp @@ -10,11 +10,82 @@ // is none it says so and exits 0, because a CI job without egress should report // "not run" rather than "broken". A certificate the client refused is not an // absence of network, and exits 1. +// The listener below is made with the C library's sockets, which openkal-musl +// provides on every target this example runs on. C headers only: they declare +// nothing `import std` also declares. +#include +#include +#include +#include + import mcpplibs.tinyhttps; import std; namespace https = mcpplibs::tinyhttps; +namespace { + +// CANCELLATION ABOVE openkal, WITH NO NETWORK. +// +// A listener that never accepts: the connection completes in the backlog, the +// client sends its ClientHello and waits for an answer that is not coming. That +// wait is a `poll` in slices of 50 ms, and above openkal each slice is a bounded +// read (`kal_timeout_read`) rather than a readiness query, which is the part of +// the stack this checks. The read timeout is 30 s, so returning promptly can +// only be the token's doing. +bool cancellation_works() { + const int listener = ::socket(AF_INET, SOCK_STREAM, 0); + if (listener < 0) { std::println("cancellation: no socket"); return false; } + sockaddr_in addr {}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = 0; + socklen_t len = sizeof addr; + if (::bind(listener, reinterpret_cast(&addr), sizeof addr) != 0 + || ::listen(listener, 4) != 0 + || ::getsockname(listener, reinterpret_cast(&addr), &len) != 0) { + std::println("cancellation: no listener"); + ::close(listener); + return false; + } + const std::string url = "https://127.0.0.1:" + std::to_string(ntohs(addr.sin_port)) + "/"; + + https::HttpClientConfig config; + config.verifySsl = false; + config.connectTimeoutMs = 5000; + config.readTimeoutMs = 30000; + https::HttpClient client(config); + https::HttpRequest request; + request.method = https::Method::GET; + request.url = url; + + // A token stopped before the call: nothing is sent. + std::stop_source early; + early.request_stop(); + auto before = client.send(request, early.get_token()); + + // A token stopped while the handshake waits. + std::stop_source source; + std::jthread stopper([&] { + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + source.request_stop(); + }); + const auto start = std::chrono::steady_clock::now(); + auto during = client.send(request, source.get_token()); + const auto ms = std::chrono::duration_cast( + std::chrono::steady_clock::now() - start).count(); + ::close(listener); + + const bool ok = before.cancelled && before.statusCode == 0 + && during.cancelled && during.statusCode == 0 + && ms >= 150 && ms < 2000; + std::println("cancellation: {} (stopped at 200 ms, returned after {} ms)", + ok ? "ok" : "WRONG", ms); + return ok; +} + +} // namespace + int main() { https::Socket::platform_init(); @@ -30,6 +101,8 @@ int main() { } std::println("framing parsers: ok"); + if (!cancellation_works()) return 1; + https::HttpClientConfig config; config.connectTimeoutMs = 15000; config.readTimeoutMs = 20000; diff --git a/mcpp.toml b/mcpp.toml index e40daad..ca4e023 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "tinyhttps" -version = "0.3.3" +version = "0.3.4" description = "Minimal C++23 HTTP/HTTPS client with SSE streaming support" license = "Apache-2.0" repo = "https://github.com/mcpplibs/tinyhttps" @@ -18,6 +18,7 @@ gtest = "1.15.2" # The socket interface follows the C library rather than the operating system # (src/platform.cppm). Where the C library is musl, including musl above openkal # on Windows, it is the POSIX interface; the platform's own headers are not part -# of that graph. +# of that graph. The entropy source follows it too: musl's `getentropy`, rather +# than mbedTLS's reading of /dev/urandom (src/tls.cppm). [target.'cfg(c-abi = "musl")'.build] -cxxflags = ["-DTINYHTTPS_POSIX_SOCKETS"] +cxxflags = ["-DTINYHTTPS_POSIX_SOCKETS", "-DTINYHTTPS_GETENTROPY"] diff --git a/src/http.cppm b/src/http.cppm index 7794e89..6b83628 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -58,6 +58,11 @@ export struct HttpResponse { // "Invalid chunk size: zz" where the server had said "OK". std::string bodyError; + // True when the caller's stop token ended the request. Then `bodyComplete` + // is false, and the status line is the server's if it had arrived and + // 0 / "Cancelled" if not. + bool cancelled { false }; + bool ok() const { return statusCode >= 200 && statusCode < 300; } }; @@ -144,6 +149,12 @@ export struct DownloadToFileResult { // failure downstream of a truncated file otherwise blames the wrong party. bool writeFailed { false }; + // True when the caller abandoned the transfer: `isCancelled` returned true + // or the stop token was stopped. `error` is then "Cancelled" if no status + // line had arrived and "cancelled" if one had, as for `HttpResponse`. A + // destination that failed first is reported as that instead. + bool cancelled { false }; + bool ok() const { return statusCode >= 200 && statusCode < 300 && error.empty(); } }; @@ -837,7 +848,16 @@ public: // Call only where the body was read to the end its framing declared and the // response did not ask for the connection to close. - void keep() noexcept { armed_ = false; } + // + // A connection in the pool carries no stop token. The call that used it + // set its own (`perform_exchange`), and the caller may stop that token + // after this call has returned — a `std::jthread` does so in its + // destructor — which must not reach whatever takes the connection next. + void keep() noexcept { + if (!armed_) return; + armed_ = false; + if (auto it = pool_.find(key_); it != pool_.end()) it->second.set_stop({}); + } // Drop now rather than at the end of the scope. Idempotent, and a no-op // after `keep()`. @@ -880,14 +900,21 @@ public: HttpClient(HttpClient&&) = default; HttpClient& operator=(HttpClient&&) = default; - HttpResponse send(const HttpRequest& request) { - return send_impl(request, 0); + // `stop` can be stopped from another thread to abandon the request: the + // call returns within about 50 ms with `cancelled` set, and the connection + // is closed rather than pooled. It cannot interrupt name resolution or a + // write blocked because the server is not reading. + HttpResponse send(const HttpRequest& request, std::stop_token stop = {}) { + return send_impl(request, 0, stop); } // Streaming SSE request — reads the response body incrementally, feeding // chunks through SseParser to the caller's callback. The callback receives // each SseEvent and returns true to continue or false to stop. - HttpResponse send_stream(const HttpRequest& request, SseCallbackFn callback) { + // `stop` is as for send(). + HttpResponse send_stream(const HttpRequest& request, SseCallbackFn callback, + std::stop_token stop = {}) { + if (stop.stop_requested()) return cancelled_response(); HttpResponse response; auto parsed = parse_url(request.url); @@ -901,12 +928,13 @@ public: PooledConnection guard(pool_, poolKey); auto exchange = perform_exchange(parsed, poolKey, - build_request(request, parsed), guard); + build_request(request, parsed), guard, stop); if (!exchange.error.empty()) { response.statusCode = 0; response.statusText = exchange.error; response.bodyComplete = false; response.bodyError = exchange.error; + response.cancelled = exchange.cancelled; return response; } @@ -921,6 +949,7 @@ public: SseParser parser; auto sink = [&](std::string_view data) -> bool { + if (stop.stop_requested()) return false; if (captureBody) { append_within_limit(response.body, data, stream_error_body_limit); } @@ -932,21 +961,23 @@ public: finish_body(response, exchange, request.method, /*maxBytes=*/std::numeric_limits::max(), - sink, guard); + sink, guard, stop); return response; } // Download URL to file with streaming progress. // Follows redirects. Calls onProgress periodically during body read. // isCancelled is checked after each block — return true to abort. + // `stop` is as for send(), and also ends the wait for the response head. DownloadToFileResult download_to_file( const std::string& url, const std::filesystem::path& destFile, DownloadProgressFn onProgress = nullptr, - std::function isCancelled = nullptr) + std::function isCancelled = nullptr, + std::stop_token stop = {}) { return download_to_file_impl(url, destFile, std::move(onProgress), - std::move(isCancelled), 0); + std::move(isCancelled), 0, stop); } HttpClientConfig& config() { return config_; } @@ -959,8 +990,18 @@ private: TlsSocket* sock { nullptr }; ResponseHead head; std::string error; + bool cancelled { false }; }; + static HttpResponse cancelled_response() { + HttpResponse response; + response.statusText = "Cancelled"; + response.bodyComplete = false; + response.bodyError = "Cancelled"; + response.cancelled = true; + return response; + } + std::string build_request(const HttpRequest& request, const ParsedUrl& parsed) const { std::string reqStr; reqStr += method_to_string(request.method); @@ -999,11 +1040,12 @@ private: // False on failure, with `error` set when there is more to say than that the // connection failed: a proxy that refused the tunnel says so in its own words. - bool open_connection(TlsSocket& sock, const ParsedUrl& parsed, std::string& error) { + bool open_connection(TlsSocket& sock, const ParsedUrl& parsed, std::string& error, + std::stop_token stop) { if (config_.proxy.has_value()) { auto tunnel = proxy_tunnel(parse_proxy_url(config_.proxy.value()), parsed.host, parsed.port, - config_.connectTimeoutMs, config_.verifySsl); + config_.connectTimeoutMs, config_.verifySsl, stop); if (!tunnel.ok()) { error = std::move(tunnel.error); return false; @@ -1028,8 +1070,17 @@ private: // request, so sending it again on a fresh connection is safe — once, and // only when no response byte has arrived. See `retryOnStaleConnection`. Exchange perform_exchange(const ParsedUrl& parsed, const std::string& poolKey, - const std::string& reqStr, PooledConnection& guard) { + const std::string& reqStr, PooledConnection& guard, + std::stop_token stop = {}) { + // Drops the connection; a stop request outranks whatever went wrong. + auto fail = [&](std::string message) -> Exchange { + guard.drop(); + if (stop.stop_requested()) return { nullptr, {}, "Cancelled", true }; + return { nullptr, {}, std::move(message) }; + }; + for (int attempt = 0; attempt < 2; ++attempt) { + if (stop.stop_requested()) return fail(""); bool reused = false; TlsSocket* sock = nullptr; @@ -1041,15 +1092,17 @@ private: if (it != pool_.end()) pool_.erase(it); auto [inserted, ok] = pool_.emplace(poolKey, TlsSocket{}); sock = &inserted->second; + } + sock->set_stop(stop); + if (!reused) { std::string openError; - if (!open_connection(*sock, parsed, openError)) { + if (!open_connection(*sock, parsed, openError, stop)) { // The proxy's refusal, else the TLS session's reason, // else the TCP connection failed and there is no more to say. std::string why = std::move(openError); if (why.empty()) why = sock->error(); if (why.empty()) why = "Connection failed"; - guard.drop(); - return { nullptr, {}, std::move(why) }; + return fail(std::move(why)); } } @@ -1065,8 +1118,7 @@ private: // connection therefore yields exactly one execution. if (!write_all(*sock, reqStr, config_.readTimeoutMs)) { if (mayRetry) { guard.reset(); continue; } - guard.drop(); - return { nullptr, {}, "Write failed" }; + return fail("Write failed"); } auto head = read_response_head(*sock, config_.readTimeoutMs); @@ -1076,18 +1128,16 @@ private: // arrived the server has seen the request, and repeating it could // repeat its effect. if (mayRetry && !head.error().sawBytes) { guard.reset(); continue; } - guard.drop(); - return { nullptr, {}, head.error().message }; + return fail(head.error().message); } - guard.drop(); - return { nullptr, {}, "No response" }; + return fail("No response"); } // Reads the body into `sink` and settles the pool guard by what the read // found. The single place that decides whether a connection is reusable. void finish_body(HttpResponse& response, Exchange& exchange, Method method, std::int64_t maxBytes, const BodySink& sink, - PooledConnection& guard) { + PooledConnection& guard, std::stop_token stop = {}) { if (!response_has_body(method, exchange.head.statusCode)) { if (config_.keepAlive && !exchange.head.connectionClose) guard.keep(); return; @@ -1103,6 +1153,10 @@ private: // Not an error, but bytes are still owed on the socket. response.bodyComplete = false; response.bodyError = "stopped by callback"; + if (stop.stop_requested()) { + response.bodyError = "cancelled"; + response.cancelled = true; + } break; case BodyEnd::ClosedByPeer: // The body ended where it said it would; the connection did too. @@ -1110,11 +1164,17 @@ private: case BodyEnd::Truncated: response.bodyComplete = false; response.bodyError = outcome.error; + if (stop.stop_requested()) { + response.bodyError = "cancelled"; + response.cancelled = true; + } break; } } - HttpResponse send_impl(const HttpRequest& request, int redirectCount) { + HttpResponse send_impl(const HttpRequest& request, int redirectCount, + std::stop_token stop) { + if (stop.stop_requested()) return cancelled_response(); HttpResponse response; auto parsed = parse_url(request.url); @@ -1128,12 +1188,13 @@ private: PooledConnection guard(pool_, poolKey); auto exchange = perform_exchange(parsed, poolKey, - build_request(request, parsed), guard); + build_request(request, parsed), guard, stop); if (!exchange.error.empty()) { response.statusCode = 0; response.statusText = exchange.error; response.bodyComplete = false; response.bodyError = exchange.error; + response.cancelled = exchange.cancelled; return response; } @@ -1157,10 +1218,10 @@ private: response.body.append(data); return true; }, - guard); + guard, stop); // Follow 3xx redirects if configured. - if (config_.maxRedirects > 0 && + if (config_.maxRedirects > 0 && !response.cancelled && response.statusCode >= 300 && response.statusCode < 400 && redirectCount < config_.maxRedirects) { std::string location = find_header(response.headers, "Location"); @@ -1181,7 +1242,7 @@ private: // connection into the pool under this same key, and a guard // still armed would delete it when this scope ends. guard.drop(); - return send_impl(redirectReq, redirectCount + 1); + return send_impl(redirectReq, redirectCount + 1, stop); } } @@ -1193,9 +1254,15 @@ private: const std::filesystem::path& destFile, DownloadProgressFn onProgress, std::function isCancelled, - int redirectCount) + int redirectCount, + std::stop_token stop) { DownloadToFileResult result; + if (stop.stop_requested()) { + result.error = "Cancelled"; + result.cancelled = true; + return result; + } auto parsed = parse_url(url); if (parsed.scheme != "https") { @@ -1212,9 +1279,10 @@ private: request.headers = { {"User-Agent", "tinyhttps/1.0"}, {"Accept", "*/*"} }; auto exchange = perform_exchange(parsed, poolKey, - build_request(request, parsed), guard); + build_request(request, parsed), guard, stop); if (!exchange.error.empty()) { result.error = exchange.error; + result.cancelled = exchange.cancelled; return result; } @@ -1239,6 +1307,11 @@ private: } else { guard.drop(); } + if (stop.stop_requested()) { + result.error = "cancelled"; + result.cancelled = true; + return result; + } if (location.starts_with("/")) { location = parsed.scheme + "://" + parsed.host + @@ -1246,7 +1319,8 @@ private: location; } return download_to_file_impl(location, destFile, std::move(onProgress), - std::move(isCancelled), redirectCount + 1); + std::move(isCancelled), redirectCount + 1, + stop); } } @@ -1339,7 +1413,10 @@ private: written += size; if (onProgress) onProgress(totalBytes, written); - if (isCancelled && isCancelled()) { cancelled = true; return false; } + if ((isCancelled && isCancelled()) || stop.stop_requested()) { + cancelled = true; + return false; + } return true; }); @@ -1357,9 +1434,12 @@ private: result.error = cancelled ? "cancelled" : "stopped"; break; case BodyEnd::Truncated: - result.error = outcome.error; + // A wait that the stop token ended reads as a truncated body. + cancelled = stop.stop_requested(); + result.error = cancelled ? "cancelled" : outcome.error; break; } + result.cancelled = cancelled; // The destination's failure outranks whatever the switch recorded: a // stop that the file asked for is not a cancellation, and a body that @@ -1367,6 +1447,7 @@ private: if (writeFailed) { result.error = write_failure(); result.writeFailed = true; + result.cancelled = false; } return result; } diff --git a/src/proxy.cppm b/src/proxy.cppm index e7c5380..9951291 100644 --- a/src/proxy.cppm +++ b/src/proxy.cppm @@ -351,8 +351,10 @@ export struct ProxyTunnel { // setting that governs the connection to the target. export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, std::string_view targetHost, int targetPort, - int timeoutMs, bool verifySsl = true) { + int timeoutMs, bool verifySsl = true, + std::stop_token stop = {}) { ProxyTunnel tunnel; + tunnel.socket.set_stop(stop); const std::string where = proxy.host + ":" + std::to_string(proxy.port); if (proxy.scheme == "http" || proxy.scheme == "socks5" || proxy.scheme == "socks5h") { @@ -366,6 +368,7 @@ export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, if (!tunnel.error.empty()) tunnel.socket.close(); } else if (proxy.scheme == "https") { auto tls = std::make_unique(); + tls->set_stop(stop); if (!tls->connect(proxy.host.c_str(), proxy.port, timeoutMs, verifySsl)) { // The session says why when the TCP connection was up, which is // where a proxy certificate that does not verify is refused. diff --git a/src/socket.cppm b/src/socket.cppm index c794f91..6dbee5c 100644 --- a/src/socket.cppm +++ b/src/socket.cppm @@ -52,7 +52,7 @@ public: // Move constructor Socket(Socket&& other) noexcept - : fd_(other.fd_) { + : fd_(other.fd_), stop_(std::move(other.stop_)) { other.fd_ = INVALID_SOCKET_FD; } @@ -61,6 +61,7 @@ public: if (this != &other) { close(); fd_ = other.fd_; + stop_ = std::move(other.stop_); other.fd_ = INVALID_SOCKET_FD; } return *this; @@ -70,6 +71,12 @@ public: return fd_ != INVALID_SOCKET_FD; } + // Once the token is stopped, every wait on this socket (connect included) + // ends within 50 ms and reports "not ready". + void set_stop(std::stop_token stop) { stop_ = std::move(stop); } + + [[nodiscard]] bool stop_possible() const { return stop_.stop_possible(); } + bool connect(const char* host, int port, int timeoutMs) { // Close existing connection if any if (is_valid()) { @@ -82,6 +89,7 @@ public: // musl-static build can't — its nameservers live in $PREFIX/etc/resolv.conf // which libc never reads — so fall back to a manual DNS query there. auto try_resolved = [&](const char* node, bool numeric) -> bool { + if (stop_.stop_requested()) return false; struct addrinfo hints{}; hints.ai_family = AF_UNSPEC; hints.ai_socktype = SOCK_STREAM; @@ -103,6 +111,7 @@ public: // Fall back to a manual DNS query when libc can't resolve (Termux: // nameservers live in $PREFIX/etc/resolv.conf, which libc ignores). auto try_manual = [&]() -> bool { + if (stop_.stop_requested()) return false; // DNS must be snappy: a UDP query to a working resolver answers // in well under a second. Cap it hard (independent of the much // larger connect timeout) so an intermittently-dropped packet to @@ -173,6 +182,7 @@ public: // Connect to the first reachable address in a resolved list. bool connect_addrinfo(struct addrinfo* result, int timeoutMs) { for (auto* rp = result; rp != nullptr; rp = rp->ai_next) { + if (stop_.stop_requested()) return false; SocketHandle fd = ::socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol); if (fd == INVALID_SOCKET_FD) { continue; @@ -232,7 +242,7 @@ public: if (errno == EINPROGRESS) { #endif // Wait for connection with timeout - if (poll_fd(fd, timeoutMs, false)) { + if (wait_fd(fd, timeoutMs, false)) { int err = 0; socklen_t len = sizeof(err); if (::getsockopt(fd, SOL_SOCKET, SO_ERROR, reinterpret_cast(&err), &len) == 0 && err == 0) { @@ -297,12 +307,12 @@ public: bool wait_readable(int timeoutMs) { if (!is_valid()) return false; - return poll_fd(fd_, timeoutMs, true); + return wait_fd(fd_, timeoutMs, true); } bool wait_writable(int timeoutMs) { if (!is_valid()) return false; - return poll_fd(fd_, timeoutMs, false); + return wait_fd(fd_, timeoutMs, false); } [[nodiscard]] SocketHandle native_handle() const { @@ -331,6 +341,27 @@ public: private: SocketHandle fd_ = INVALID_SOCKET_FD; + std::stop_token stop_; + + // poll_fd in slices while a token is attached. A negative timeout waits + // without limit. Not std::min: defines a `min` macro. + bool wait_fd(SocketHandle fd, int timeoutMs, bool forRead) const { + if (!stop_.stop_possible()) return poll_fd(fd, timeoutMs, forRead); + constexpr int slice = 50; + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(timeoutMs); + while (!stop_.stop_requested()) { + int wait = slice; + if (timeoutMs >= 0) { + const auto left = std::chrono::duration_cast( + deadline - std::chrono::steady_clock::now()).count(); + if (left < slice) wait = left > 0 ? static_cast(left) : 0; + } + if (poll_fd(fd, wait, forRead)) return true; + if (timeoutMs >= 0 && std::chrono::steady_clock::now() >= deadline) return false; + } + return false; + } static bool set_non_blocking(SocketHandle fd, bool nonBlocking) { #ifdef TINYHTTPS_WINSOCK diff --git a/src/tls.cppm b/src/tls.cppm index 34c8e55..8f16494 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -15,6 +15,21 @@ module; #pragma comment(lib, "bcrypt.lib") #endif +// WHERE THE C LIBRARY IS musl, THE ENTROPY IS THE C LIBRARY'S. +// +// mbedTLS's own platform source asks the kernel through `getrandom` only where +// it recognises glibc (`entropy_poll.c`: `__linux__ && __GLIBC__`); with musl it +// opens `/dev/urandom`. Above openkal that file is not a thing the C library can +// promise — openkal reaches entropy through its own `openkal.random` interface, +// and openkal-musl answers `getrandom` with it — so on x86_64-windows-musl every +// handshake failed before it began, with "CTR_DRBG - The entropy source failed". +// musl's `getentropy` is `getrandom` underneath on every system it runs on, +// which is the source mbedTLS would have chosen had it known. The macro comes +// from mcpp.toml's `cfg(c-abi = "musl")`, because musl defines none of its own. +#ifdef TINYHTTPS_GETENTROPY +#include +#endif + export module mcpplibs.tinyhttps:tls; import :socket; @@ -79,6 +94,10 @@ static int bio_send(void* ctx, const unsigned char* buf, size_t len) { // complete and correct. static int bio_recv(void* ctx, unsigned char* buf, size_t len) { auto* sock = static_cast(ctx); + // With a stop token, wait here so the recv below cannot block past a stop. + if (sock->stop_possible() && !sock->wait_readable(-1)) { + return MBEDTLS_ERR_NET_RECV_FAILED; + } int ret = sock->read(reinterpret_cast(buf), static_cast(len)); if (ret < 0) { return MBEDTLS_ERR_NET_RECV_FAILED; @@ -86,6 +105,19 @@ static int bio_recv(void* ctx, unsigned char* buf, size_t len) { return ret; // 0 means end of stream; mbedtls turns it into SSL_CONN_EOF } +#ifdef TINYHTTPS_GETENTROPY +// `getentropy` gives at most 256 bytes a call. +static int c_library_entropy(void*, unsigned char* out, size_t len) { + while (len > 0) { + const size_t n = len < 256 ? len : 256; + if (::getentropy(out, n) != 0) return MBEDTLS_ERR_ENTROPY_SOURCE_FAILED; + out += n; + len -= n; + } + return 0; +} +#endif + static std::string mbedtls_message(int ret) { char buf[200]; mbedtls_strerror(ret, buf, sizeof buf); @@ -160,6 +192,20 @@ public: // empty if it failed earlier or has not failed. [[nodiscard]] const std::string& error() const { return error_; } + // Call before connect(); it covers the handshake and every later wait. + // + // THE TOKEN BELONGS TO THE SOCKET AT THE BOTTOM, AND ONLY THERE. Every wait + // ends in a `Socket` — this session's own, or, inside an https:// proxy's + // tunnel, the one beneath `lower_` — so that is where it is set. Setting + // only `socket_` left a reused tunnel holding the token of the call that + // opened it: a later call could not be cancelled, and once that first + // token was stopped every wait failed at once and the stale-connection + // retry sent a POST that had already arrived a second time. + void set_stop(std::stop_token stop) { + if (lower_) lower_->set_stop(stop); + socket_.set_stop(std::move(stop)); + } + // Connect over an already-established Socket (e.g. a proxy tunnel). // Takes ownership of the socket and performs TLS handshake on top of it. bool connect_over(Socket&& socket, const char* host, bool verifySsl) { @@ -329,7 +375,11 @@ private: state_ = std::make_unique(); int ret = mbedtls_ctr_drbg_seed( +#ifdef TINYHTTPS_GETENTROPY + &state_->ctr_drbg, c_library_entropy, nullptr, +#else &state_->ctr_drbg, mbedtls_entropy_func, &state_->entropy, +#endif nullptr, 0); if (ret != 0) return fail(mbedtls_message(ret)); diff --git a/tests/test_cancel.cpp b/tests/test_cancel.cpp new file mode 100644 index 0000000..9d677bc --- /dev/null +++ b/tests/test_cancel.cpp @@ -0,0 +1,812 @@ +// Abandoning a request from another thread through a std::stop_token. +// +// Every wait here is far longer than the test should take (readTimeoutMs is +// 30 s), so a request that returns promptly with `cancelled` set can only have +// been ended by the token, never by a timeout. + +// WHERE `import std` GOES DEPENDS ON THE STANDARD LIBRARY, AND ONLY HERE. +// +// Every other test file includes its headers and then imports std, which is the +// order libstdc++ needs: the other way round, gcc 16 rejects the headers as +// redefinitions of what the module already declared. This file is the one that +// calls `std::stop_source::request_stop()`, and libc++ 20 and 22 compile that +// call, after a textual or (gtest and the test +// servers include both), into a reference to an inline helper that nothing then +// emits: the link fails on `__atomic_unique_lock<...>::__set_locked_bit`. With +// the import first it links, and libc++ 23 links either way. Measured with the +// toolchains mcpp installs; see .agents/docs/design-cancel-stop-token.md §6.2. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif + +namespace https = mcpplibs::tinyhttps; + +namespace { + +using Clock = std::chrono::steady_clock; + +// Well above the 50 ms slice and still far below any timeout in play. +constexpr long long kPromptMs = 500; + +https::HttpClientConfig test_config(int readTimeoutMs = 30000) { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = readTimeoutMs; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::GET; + req.url = url; + return req; +} + +long long ms_between(Clock::time_point from, Clock::time_point to) { + return std::chrono::duration_cast(to - from).count(); +} + +// Requests a stop `delay` after `ready` first holds (at once when it is not +// given) and remembers when it did, so a test can measure from the stop to the +// return of the call. Waiting for the server to have seen the request keeps a +// slow machine from stopping a request that was never sent. +class Stopper { +public: + explicit Stopper(std::chrono::milliseconds delay, + std::function ready = {}) + : thread_([this, delay, ready = std::move(ready)] { + for (int i = 0; ready && !ready() && i < 1000 && !quit_.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + for (auto end = Clock::now() + delay; + Clock::now() < end && !quit_.load();) { + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + if (quit_.load()) return; + stoppedAt_.store(Clock::now().time_since_epoch().count()); + source.request_stop(); + }) {} + // A stop that has not come yet is abandoned, so a long delay costs nothing + // when the call it was meant to end has already returned. + ~Stopper() { + quit_ = true; + if (thread_.joinable()) thread_.join(); + } + + long long since_stop_ms(Clock::time_point returned) { + thread_.join(); + return ms_between(Clock::time_point(Clock::duration(stoppedAt_.load())), returned); + } + + std::stop_source source; + +private: + std::atomic stoppedAt_ { 0 }; + std::atomic quit_ { false }; + std::thread thread_; +}; + +// A TCP listener that never speaks TLS: it accepts, reads and discards, so a +// client's handshake waits for a ServerHello that is not coming. +class SilentListener { +public: + SilentListener() { + mbedtls_net_init(&listener_); + if (mbedtls_net_bind(&listener_, "127.0.0.1", "0", MBEDTLS_NET_PROTO_TCP) != 0) { + failed_ = true; + return; + } + sockaddr_in addr {}; + socklen_t len = sizeof addr; + if (::getsockname(listener_.fd, reinterpret_cast(&addr), &len) != 0) { + failed_ = true; + return; + } + port_ = ntohs(addr.sin_port); + thread_ = std::thread([this] { run(); }); + } + + ~SilentListener() { + release(); + if (thread_.joinable()) thread_.join(); + mbedtls_net_free(&listener_); + } + + [[nodiscard]] bool failed() const { return failed_; } + [[nodiscard]] std::string url() const { + return "https://127.0.0.1:" + std::to_string(port_) + "/"; + } + [[nodiscard]] bool peer_closed() const { return peerClosed_.load(); } + [[nodiscard]] bool received() const { return received_.load(); } + + // Lets go of the accepted connection and stops listening. + void release() { released_.store(true); } + +private: + void run() { + while (!released_.load()) { + const int ready = mbedtls_net_poll(&listener_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) return; + if (ready == 0) continue; + mbedtls_net_context client {}; + mbedtls_net_init(&client); + if (mbedtls_net_accept(&listener_, &client, nullptr, 0, nullptr) != 0) { + mbedtls_net_free(&client); + continue; + } + unsigned char buf[512]; + while (!released_.load()) { + const int r = mbedtls_net_poll(&client, MBEDTLS_NET_POLL_READ, 50); + if (r < 0) break; + if (r == 0) continue; + if (mbedtls_net_recv(&client, buf, sizeof buf) <= 0) { + peerClosed_.store(true); + break; + } + received_.store(true); + } + mbedtls_net_free(&client); + return; + } + } + + mbedtls_net_context listener_ {}; + int port_ { 0 }; + bool failed_ { false }; + std::atomic released_ { false }; + std::atomic peerClosed_ { false }; + std::atomic received_ { false }; + std::thread thread_; +}; + +// Waits up to `limit` for `predicate`, which is how a test learns that the server +// saw the connection end. +bool becomes_true(const std::function& predicate, + std::chrono::milliseconds limit = std::chrono::seconds(2)) { + const auto deadline = Clock::now() + limit; + while (!predicate() && Clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + return predicate(); +} + +class CancelTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +} // namespace + +// The case that motivates all of this: the request is written and the server +// has not answered. The server also records that the connection ended, which is +// what makes it stop working on a request nobody is waiting for. +TEST_F(CancelTest, ARequestWaitingForTheHeadersIsAbandoned) { + std::atomic peerClosed { false }; + tls_test::Server server([&](tls_test::Conn& conn, int) { + peerClosed = conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(200), [&] { return server.requests() >= 1; }); + + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + const auto returned = Clock::now(); + const auto latency = stopper.since_stop_ms(returned); + std::cout << "[ latency ] headers: " << latency << " ms after the stop" << std::endl; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_FALSE(res.bodyComplete); + EXPECT_LT(latency, kPromptMs); + EXPECT_TRUE(becomes_true([&] { return peerClosed.load(); })) + << "the connection was not closed"; +} + +TEST_F(CancelTest, AStreamWaitingForTheNextChunkIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n"); + conn.write("b\r\ndata: one\n\n\r\n"); + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + std::atomic events { 0 }; + Stopper stopper(std::chrono::milliseconds(100), [&] { return events.load() > 0; }); + + auto res = client.send_stream(get(server.url("/")), + [&](const https::SseEvent&) { ++events; return true; }, + stopper.source.get_token()); + const auto latency = stopper.since_stop_ms(Clock::now()); + std::cout << "[ latency ] stream: " << latency << " ms after the stop" << std::endl; + + EXPECT_EQ(res.statusCode, 200); + EXPECT_TRUE(res.cancelled); + EXPECT_FALSE(res.bodyComplete); + EXPECT_EQ(res.bodyError, "cancelled"); + EXPECT_EQ(events.load(), 1); + EXPECT_LT(latency, kPromptMs); +} + +// The handshake has no timeout of its own, so a server that accepts and says +// nothing holds the call for good unless the token ends it. The watchdog keeps +// a regression from hanging the suite: it lets go of the connection after 5 s. +TEST_F(CancelTest, AHandshakeThatNeverAnswersIsAbandoned) { + SilentListener listener; + ASSERT_FALSE(listener.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(200), [&] { return listener.received(); }); + + std::atomic done { false }; + std::atomic watchdogFired { false }; + std::thread watchdog([&] { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + if (!done.load()) { watchdogFired = true; listener.release(); } + }); + + auto res = client.send(get(listener.url()), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + watchdog.join(); + const auto latency = stopper.since_stop_ms(returned); + std::cout << "[ latency ] handshake: " << latency << " ms after the stop" << std::endl; + + EXPECT_FALSE(watchdogFired.load()) << "the handshake was not interrupted"; + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(latency, kPromptMs); + EXPECT_TRUE(becomes_true([&] { return listener.peer_closed(); })) + << "the connection was not closed"; +} + +TEST_F(CancelTest, ATokenAlreadyStoppedSendsNothing) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("hello")); + return true; + }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::HttpClient client(test_config()); + + auto res = client.send(get(server.url("/")), source.get_token()); + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + + auto streamed = client.send_stream(get(server.url("/")), + [](const https::SseEvent&) { return true; }, + source.get_token()); + EXPECT_TRUE(streamed.cancelled); + EXPECT_EQ(server.accepts(), 0); +} + +// The slices must not shorten or lengthen the timeout a token-carrying call +// already had. The token is stopped after 5 s, so a timeout that never comes +// fails the test rather than hanging the suite. +TEST_F(CancelTest, ATokenThatIsNeverStoppedLeavesTheTimeoutAlone) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config(/*readTimeoutMs=*/300)); + Stopper backstop(std::chrono::seconds(5)); + + const auto started = Clock::now(); + auto res = client.send(get(server.url("/")), backstop.source.get_token()); + const auto elapsed = ms_between(started, Clock::now()); + + EXPECT_FALSE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "No response"); + EXPECT_GT(elapsed, 250); + EXPECT_LT(elapsed, 1500); +} + +// A cancelled connection has an unknown amount of the response still on it, so +// it must not be handed to the next request. +TEST_F(CancelTest, ACancelledConnectionIsNotReused) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.wait_peer_close(); + return false; + } + conn.write(tls_test::ok_response("second")); + return true; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto first = client.send(get(server.url("/")), stopper.source.get_token()); + EXPECT_TRUE(first.cancelled); + + auto second = client.send(get(server.url("/"))); + EXPECT_EQ(second.statusCode, 200) << "status text was: " << second.statusText; + EXPECT_EQ(second.body, "second"); + EXPECT_FALSE(second.cancelled); + EXPECT_EQ(server.accepts(), 2); +} + +// A pooled connection that fails before any response byte is normally retried +// on a new one. A request the caller has abandoned must not be sent again. +TEST_F(CancelTest, ACancelledRequestIsNotRetriedOnAFreshConnection) { + tls_test::Server server([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write(tls_test::ok_response("first")); + return true; + } + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + EXPECT_EQ(client.send(get(server.url("/"))).statusCode, 200); + + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 2; }); + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + EXPECT_TRUE(res.cancelled); + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(server.requests(), 2); + EXPECT_EQ(server.accepts(), 1); +} + +namespace { + +int open_fds() { + std::error_code ec; + for (const char* dir : { "/proc/self/fd", "/dev/fd" }) { + if (!std::filesystem::exists(dir, ec)) continue; + int n = 0; + for (auto it = std::filesystem::directory_iterator(dir, ec); + !ec && it != std::filesystem::directory_iterator(); it.increment(ec)) { + ++n; + } + return n; + } + return -1; +} + +void cancel_stuck_requests(int times) { + std::atomic closed { 0 }; + tls_test::Server server([&](tls_test::Conn& conn, int) { + if (conn.wait_peer_close()) ++closed; + return false; + }); + https::HttpClient client(test_config()); + for (int i = 0; i < times; ++i) { + const int seen = server.requests(); + Stopper stopper(std::chrono::milliseconds(50), [&] { return server.requests() > seen; }); + client.send(get(server.url("/")), stopper.source.get_token()); + } + // The server's side of each connection is done before its descriptors are + // counted as gone. + for (int i = 0; i < 200 && closed.load() < times; ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + server.stop(); +} + +} // namespace + +TEST_F(CancelTest, CancelledRequestsDoNotLeakDescriptors) { + if (open_fds() < 0) GTEST_SKIP() << "no /proc/self/fd or /dev/fd"; + + cancel_stuck_requests(1); // anything the first request opens and keeps + const int before = open_fds(); + cancel_stuck_requests(20); + EXPECT_EQ(open_fds(), before); +} + +// A redirect whose body is still arriving when the stop comes has to be reported +// as it is, not followed: the next request would be one the caller abandoned. +TEST_F(CancelTest, ARedirectIsNotFollowedOnceTheRequestIsCancelled) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 302 Found\r\nLocation: /next\r\nContent-Length: 100\r\n\r\npartial"); + conn.park(); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto res = client.send(get(server.url("/")), stopper.source.get_token()); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 302); + EXPECT_EQ(res.bodyError, "cancelled"); + EXPECT_EQ(server.requests(), 1); +} + +// Connecting is a wait too, and nothing may be sent to an address once the stop +// has been asked for. The listener only has to exist: the kernel completes the +// connection for it, so an attempt shows as an accept. +TEST_F(CancelTest, AConnectAfterTheStopSendsNoSyn) { + tls_test::Server server([](tls_test::Conn&, int) { return false; }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::Socket sock; + sock.set_stop(source.get_token()); + + EXPECT_FALSE(sock.connect("127.0.0.1", server.port(), 4000)); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + EXPECT_EQ(server.accepts(), 0); +} + +// ── through a proxy ────────────────────────────────────────────────────────── + +namespace { + +https::HttpClientConfig proxied_config(const std::string& proxyUrl) { + auto cfg = test_config(); + cfg.proxy = proxyUrl; + return cfg; +} + +// Holds a connection for up to 5 s or until `done`, so a test whose cancellation +// fails sees the connection dropped (and an assertion fail) rather than a hang. +void hold(const std::atomic& done) { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +} + +} // namespace + +TEST_F(CancelTest, AProxyThatNeverAnswersConnectIsAbandoned) { + std::atomic sawConnect { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + sawConnect = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("http://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawConnect.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "Cancelled"); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +TEST_F(CancelTest, AnHttpsProxyThatNeverCompletesItsHandshakeIsAbandoned) { + std::atomic sawHello { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_some(); + sawHello = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("https://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawHello.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// The session to the target runs inside the one to the proxy, and the stop has +// to reach the proxy's session for the inner handshake to be interruptible. +TEST_F(CancelTest, AHandshakeInsideAnHttpsProxyTunnelIsAbandoned) { + std::atomic tunnelOpen { false }; + tls_test::Server proxy([&](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + tunnelOpen = true; + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("https://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(200), [&] { return tunnelOpen.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// ── a reused connection carries the current call's token ───────────────────── +// +// The token is set on a connection for one call and taken off when it goes back +// to the pool. Through an https:// proxy the waits happen in the session beneath +// the one the pool holds, and it was that session's token, set by the call that +// opened the tunnel, that every later call waited on. + +namespace { + +enum class Route { Direct, HttpProxy, HttpsProxy }; + +std::string route_name(const ::testing::TestParamInfo& info) { + switch (info.param) { + case Route::Direct: return "Direct"; + case Route::HttpProxy: return "HttpProxy"; + case Route::HttpsProxy: return "HttpsProxy"; + } + return "Unknown"; +} + +// The proxy in front of a target, when the route has one, and the client +// configuration that reaches the target through it. +class Path { +public: + Path(Route route, int targetPort) { + if (route == Route::HttpProxy) { + http_ = std::make_unique([targetPort](proxy_test::Peer& peer) { + if (peer.read_head().empty()) return; + peer.write("HTTP/1.1 200 Connection established\r\n\r\n"); + peer.relay_to(targetPort); + }); + } else if (route == Route::HttpsProxy) { + https_ = std::make_unique([targetPort](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.relay_to(targetPort); + return false; + }); + } + } + + [[nodiscard]] bool failed() const { + return (http_ && http_->failed()) || (https_ && https_->failed()); + } + + [[nodiscard]] https::HttpClientConfig config() const { + auto cfg = test_config(); + if (http_) cfg.proxy = "http://127.0.0.1:" + std::to_string(http_->port()); + if (https_) cfg.proxy = "https://127.0.0.1:" + std::to_string(https_->port()); + return cfg; + } + + // Connections the proxy accepted; the target's count when there is none. + [[nodiscard]] int proxy_accepts(const tls_test::Server& target) const { + if (http_) return http_->accepts(); + if (https_) return https_->accepts(); + return target.accepts(); + } + +private: + std::unique_ptr http_; + std::unique_ptr https_; +}; + +https::HttpRequest post(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::POST; + req.url = url; + req.body = "{}"; + return req; +} + +class ReuseTest : public CancelTest, public ::testing::WithParamInterface {}; + +} // namespace + +// THE README'S OWN PATTERN. A jthread requests a stop in its destructor, after +// the call it ran has returned and its connection is back in the pool. That stop +// must not reach the next call: through an https:// proxy it made every wait on +// the tunnel fail at once, and the stale-connection retry then sent a POST the +// target had already received a second time. +TEST_P(ReuseTest, AStopAfterTheCallReturnedDoesNotReachTheNextCall) { + tls_test::Server target([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("done")); + return true; + }); + ASSERT_FALSE(target.failed()); + Path path(GetParam(), target.port()); + ASSERT_FALSE(path.failed()); + + https::HttpClient client(path.config()); + { + std::atomic returned { false }; + std::jthread worker([&](std::stop_token stop) { + auto first = client.send(post(target.url("/first")), stop); + EXPECT_EQ(first.statusCode, 200) << first.statusText; + EXPECT_FALSE(first.cancelled); + returned = true; + }); + while (!returned.load()) std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } // ~jthread: request_stop(), then join + + auto second = client.send(post(target.url("/second"))); + EXPECT_EQ(second.statusCode, 200) << second.statusText; + EXPECT_FALSE(second.cancelled); + + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(target.requests(), 2) << "the second POST reached the target more than once"; + EXPECT_EQ(target.accepts(), 1) << "the pooled connection was not reused"; + EXPECT_EQ(path.proxy_accepts(target), 1); +} + +// The other half: the second call's own token has to reach the connection it +// reuses, whatever the first call's token is doing. +TEST_P(ReuseTest, TheSecondCallsTokenReachesTheReusedConnection) { + tls_test::Server target([](tls_test::Conn& conn, int index) { + if (index == 0) { + conn.write(tls_test::ok_response("first")); + return true; + } + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(target.failed()); + Path path(GetParam(), target.port()); + ASSERT_FALSE(path.failed()); + + https::HttpClient client(path.config()); + std::stop_source first; // never stopped, and alive throughout + EXPECT_EQ(client.send(get(target.url("/a")), first.get_token()).statusCode, 200); + + Stopper stopper(std::chrono::milliseconds(100), [&] { return target.requests() >= 2; }); + auto res = client.send(get(target.url("/b")), stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled) << res.statusText; + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); + EXPECT_EQ(target.accepts(), 1) << "the second call did not reuse the connection"; +} + +INSTANTIATE_TEST_SUITE_P(Routes, ReuseTest, + ::testing::Values(Route::Direct, Route::HttpProxy, Route::HttpsProxy), + route_name); + +TEST_F(CancelTest, ASocks5ProxyThatNeverAnswersTheGreetingIsAbandoned) { + std::atomic sawGreeting { false }; + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_some(); + sawGreeting = true; + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + https::HttpClient client(proxied_config("socks5://127.0.0.1:" + std::to_string(proxy.port()))); + Stopper stopper(std::chrono::milliseconds(100), [&] { return sawGreeting.load(); }); + auto res = client.send(get("https://127.0.0.1:9/"), stopper.source.get_token()); + const auto returned = Clock::now(); + done = true; + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +// ── download_to_file ───────────────────────────────────────────────────────── + +namespace { + +std::filesystem::path scratch_file(std::string_view name) { + auto path = std::filesystem::temp_directory_path() + / ("tinyhttps-cancel-" + std::string(name)); + std::error_code ec; + std::filesystem::remove(path, ec); + return path; +} + +} // namespace + +TEST_F(CancelTest, ADownloadWaitingForTheHeadersIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("head"); + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return server.requests() >= 1; }); + auto res = client.download_to_file(server.url("/file"), file, nullptr, nullptr, + stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.error, "Cancelled"); + EXPECT_FALSE(res.ok()); + EXPECT_FALSE(std::filesystem::exists(file)) << "no file before a status line"; + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); +} + +TEST_F(CancelTest, ADownloadWaitingForTheRestOfTheBodyIsAbandoned) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 100000\r\n\r\n"); + conn.write(std::string(1000, 'x')); + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("body"); + std::atomic written { 0 }; + https::HttpClient client(test_config()); + Stopper stopper(std::chrono::milliseconds(100), [&] { return written.load() >= 1000; }); + auto res = client.download_to_file(server.url("/file"), file, + [&](std::int64_t, std::int64_t now) { written = now; }, + nullptr, stopper.source.get_token()); + const auto returned = Clock::now(); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.statusCode, 200); + EXPECT_EQ(res.error, "cancelled"); + EXPECT_FALSE(res.writeFailed); + EXPECT_EQ(res.bytesWritten, 1000); + EXPECT_LT(stopper.since_stop_ms(returned), kPromptMs); + + std::error_code ec; + std::filesystem::remove(file, ec); +} + +TEST_F(CancelTest, ADownloadWithATokenAlreadyStoppedSendsNothing) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("never")); + return true; + }); + ASSERT_FALSE(server.failed()); + + std::stop_source source; + source.request_stop(); + https::HttpClient client(test_config()); + auto res = client.download_to_file(server.url("/file"), scratch_file("none"), + nullptr, nullptr, source.get_token()); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.error, "Cancelled"); + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + EXPECT_EQ(server.accepts(), 0); +} + +// `isCancelled` and the token are two ways to say the same thing, and the result +// says it the same way for both. +TEST_F(CancelTest, IsCancelledSetsCancelledAsTheTokenDoes) { + tls_test::Server server([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 OK\r\nContent-Length: 100000\r\n\r\n"); + conn.write(std::string(1000, 'x')); + conn.wait_peer_close(); + return false; + }); + ASSERT_FALSE(server.failed()); + + const auto file = scratch_file("predicate"); + https::HttpClient client(test_config()); + auto res = client.download_to_file(server.url("/file"), file, nullptr, + [] { return true; }); + + EXPECT_TRUE(res.cancelled); + EXPECT_EQ(res.error, "cancelled"); + EXPECT_EQ(res.statusCode, 200); + + std::error_code ec; + std::filesystem::remove(file, ec); +} diff --git a/tests/test_pool.cpp b/tests/test_pool.cpp index eb91eaa..995b795 100644 --- a/tests/test_pool.cpp +++ b/tests/test_pool.cpp @@ -190,7 +190,9 @@ TEST_F(PoolTest, AContentLengthPastThirtyTwoBitsIsNotSilentlyZero) { // the width of the field and nothing else. // (`ABodyLargerThanTheConfiguredLimitIsRefused` covers the cap.) auto cfg = test_config(); - cfg.maxResponseBodyBytes = std::numeric_limits::max(); + // Parenthesised: , which the test servers include on Windows, + // defines a `max` macro. + cfg.maxResponseBodyBytes = (std::numeric_limits::max)(); https::HttpClient client(cfg); auto first = client.send(get(server.url("/one"))); diff --git a/tests/tls_test_server.hpp b/tests/tls_test_server.hpp index d25bc2a..8017262 100644 --- a/tests/tls_test_server.hpp +++ b/tests/tls_test_server.hpp @@ -183,6 +183,10 @@ class Conn { // is stopped or after `limit`. void park(std::chrono::milliseconds limit = std::chrono::seconds(20)); + // Waits for the client to end the connection, discarding anything it sends. + // False when `limit` passes or the server is stopped first. + bool wait_peer_close(std::chrono::milliseconds limit = std::chrono::seconds(20)); + // Ends the connection at the TLS layer, then at the transport. void shutdown() { if (closed_) return; @@ -503,6 +507,20 @@ inline void Conn::relay_to(int port, std::chrono::milliseconds gather) { mbedtls_net_free(&target); } +inline bool Conn::wait_peer_close(std::chrono::milliseconds limit) { + const auto deadline = std::chrono::steady_clock::now() + limit; + unsigned char buf[256]; + while (!server_.stopping() && std::chrono::steady_clock::now() < deadline) { + const int ready = mbedtls_net_poll(&net_, MBEDTLS_NET_POLL_READ, 50); + if (ready < 0) return true; + if (ready == 0) continue; + int ret = mbedtls_ssl_read(&ssl_, buf, sizeof buf); + if (ret == MBEDTLS_ERR_SSL_WANT_READ || ret == MBEDTLS_ERR_SSL_WANT_WRITE) continue; + if (ret <= 0) return true; + } + return false; +} + inline void Conn::park(std::chrono::milliseconds limit) { const auto deadline = std::chrono::steady_clock::now() + limit; while (!server_.stopping() && std::chrono::steady_clock::now() < deadline) { From 2cc7310037e05eaa075721afd157cc22b42384aa Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Fri, 2 Oct 2026 04:51:04 +0800 Subject: [PATCH 11/15] CHANGELOG: 0.3.4 reaches a caret range, not a bare version; record the release in the plan A bare version is exact in mcpp, and "0.3.x" is not a form it accepts; measured against the published index, "^0.3.1" resolves to 0.3.4 under both mirrors and "0.3.1" stays at 0.3.1. --- .agents/docs/exec-plan-0.3.4-cancel.md | 14 +++++++------- CHANGELOG.md | 5 +++-- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/.agents/docs/exec-plan-0.3.4-cancel.md b/.agents/docs/exec-plan-0.3.4-cancel.md index 390e20f..b065c49 100644 --- a/.agents/docs/exec-plan-0.3.4-cancel.md +++ b/.agents/docs/exec-plan-0.3.4-cancel.md @@ -13,7 +13,7 @@ ## 0. 为什么是 0.3.4 - 只新增，不改已有行为：`send`、`send_stream`、`download_to_file` 都只加了尾部默认参数；`HttpResponse` 和 `DownloadToFileResult` 各在末尾追加一个字段；不传 token 时逐字节不变（不变式 I3）。 -- 依赖写成 `tinyhttps = "0.3.x"` 的项目会自动解析到它，这是 0.3.3 立下的规则：让现有依赖不改清单就能收到。mcpp-index 的 `tests/examples/tinyhttps` 写的是 `0.3.1`，也会解析到它。 +- 依赖写成 caret 范围 `tinyhttps = "^0.3.0"` 的项目会自动解析到它。**更正（发布后实测）：** mcpp 里裸写的 `"0.3.1"` 是精确版本，不会解析到 0.3.4；`"0.3.x"` 不是 mcpp 的合法写法。0.3.3 的发布说明里也沿用了这个说法。已用已发布的 index 实测：`"^0.3.1"` 在 GLOBAL 和 CN 两个源下都解析到 0.3.4。 - 唯一的源码不兼容点是对 `&HttpClient::send` 取成员函数指针（参数个数变了）。在 CHANGELOG 里点名说明。 ## 1. 八个视角 @@ -93,9 +93,9 @@ T0 计划（本文）+ 设计文档纳入 PR ## 6. 验收 -- [ ] PR #23 的 CI 全绿（上表全部 job） -- [ ] 自审：H1、M1、M2、D1、D2 全部落地；PR 描述更新 -- [ ] squash 合入；tag `0.3.4`；GitHub release -- [ ] gitcode 上的 `tinyhttps-0.3.4.tar.gz` 与 GitHub archive 的 sha256 一致 -- [ ] mcpp-index 合入；`validate` 和 `openkal-compat`（tinyhttps 这一项）通过 -- [ ] 新建项目用已发布的 index 解析到 0.3.4，GLOBAL 和 CN 两个源都能构建并运行 +- [x] PR #23 的 CI 全绿（10 个 job，`10e28b0`） +- [x] 自审：H1、M1、M2、D1、D2 全部落地；PR 描述更新 +- [x] squash 合入（`156edeb`）；tag `0.3.4`；GitHub release +- [x] gitcode 上的 `tinyhttps-0.3.4.tar.gz` 与 GitHub archive 的 sha256 一致（`26e0e496…41e9`） +- [x] mcpp-index 合入（mcpplibs/mcpp-index#504）；`validate` 和 `openkal-compat`（tinyhttps 这一项，四个 target 均为 runs）通过 +- [x] 新建项目用已发布的 index 解析到 0.3.4（`"^0.3.1"`），GLOBAL 和 CN 两个源都能构建并运行，并对 httpbin 做了真实的进行中取消（301 ms） diff --git a/CHANGELOG.md b/CHANGELOG.md index 64f3182..3f276a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,8 +3,9 @@ ## 0.3.4 A request in flight can be abandoned from another thread. Everything is added -and nothing changes without a token, so it is a patch release that every -dependency written `tinyhttps = "0.3.x"` receives. +and nothing changes without a token, so it is a patch release: a dependency +written as a caret range, `tinyhttps = "^0.3.0"`, resolves to it. A bare version +such as `"0.3.3"` is exact in mcpp and stays where it is. ### Cancellation From 9d4315e8bda910e04530ec6bba3f24cc79c779e8 Mon Sep 17 00:00:00 2001 From: Cloud_Yun Date: Sat, 10 Oct 2026 03:07:24 +0900 Subject: [PATCH 12/15] Bound the TLS handshake with connectTimeoutMs (#25) After the TCP connect the socket is blocking, and bio_recv called recv without waiting unless a stop token was attached, so a server or an https:// proxy that accepted the connection and never sent its ServerHello held send(), send_stream() and download_to_file() for good. setup_tls sets a deadline of connectTimeoutMs on the socket at the bottom for the duration of the handshake; bio_recv waits against it (and against the token, if any), and a handshake that runs out of time fails with "TLS handshake timed out". It covers the target, an https:// proxy, and the target inside an https:// proxy's tunnel. A wait that a signal ends early is waited again; the deadline is hit only when the clock has passed it. TlsSocket::connect_over takes a trailing handshakeTimeoutMs (-1, the default, waits without limit); TlsSocket::connect bounds the handshake with the timeout it already took. Socket gains set_deadline, deadline_hit and wait_before_recv. CI: the openkal job on Windows finds Git for Windows's CA bundle instead of naming mingw64/ (Git 2.56 moved it under ucrt64/), and the Windows job runs test_handshake_timeout. Co-authored-by: Sunrisepeak --- .github/workflows/ci.yml | 20 ++- CHANGELOG.md | 23 +++ README.md | 2 +- src/http.cppm | 4 +- src/socket.cppm | 43 ++++- src/tls.cppm | 63 +++++-- tests/test_handshake_timeout.cpp | 289 +++++++++++++++++++++++++++++++ 7 files changed, 420 insertions(+), 24 deletions(-) create mode 100644 tests/test_handshake_timeout.cpp diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0b2e4ac..260f3ab 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -288,10 +288,26 @@ jobs: # "Platforms"), so the program is given a bundle: Git for Windows's, copied # beside it, because openkal resolves a relative name against the working # directory and names no absolute path. + # + # Git is asked where its bundle is, and the install searched after that, + # rather than the directory named: the runner image that brought Git for + # Windows 2.56 has no `mingw64/etc/ssl/certs/ca-bundle.crt`, and the job + # failed there before the program ran. - name: Run above openkal run: | set -o pipefail - cp "/c/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt" ca-bundle.crt + git --version + ca=$(git config --system --get http.sslcainfo || true) + if [ -n "$ca" ]; then ca=$(cygpath -u "$ca"); fi + if [ ! -f "$ca" ]; then + ca=$(find "/c/Program Files/Git" -path '*/ssl/certs/ca-bundle.crt' -print -quit) + fi + if [ ! -f "$ca" ]; then + echo "::error::no CA bundle in the Git for Windows install" + exit 1 + fi + echo "CA bundle: $ca" + cp "$ca" ca-bundle.crt SSL_CERT_FILE=ca-bundle.crt ./smoke.exe 2>&1 | tee run.log grep -q '^cancellation: ok' run.log @@ -352,7 +368,7 @@ jobs: - name: The hermetic tests run: | set -o pipefail - for t in test_framing test_tls_verify test_pool test_proxy test_cancel; do + for t in test_framing test_tls_verify test_pool test_proxy test_cancel test_handshake_timeout; do mcpp test "$t" 2>&1 | tee "$t.log" grep -q "^$t \.\.\. ok" "$t.log" done diff --git a/CHANGELOG.md b/CHANGELOG.md index 3f276a1..0342d10 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,28 @@ # Changelog +## Unreleased + +The TLS handshake is bounded by `connectTimeoutMs`. It had no limit: after the +TCP connection was up, a server (or an https:// proxy) that accepted it and never +sent its ServerHello held `send`, `send_stream` and `download_to_file` for good +unless the caller passed a stop token. + +* The handshake gives up once `connectTimeoutMs` has passed since it began, and + `statusText` is `TLS handshake timed out`. This covers the handshake with the + target, with an https:// proxy, and with the target inside an https:// proxy's + tunnel. The limit is on the handshake as a whole, so a peer that sends a byte + at a time does not hold it open. +* Behaviour change: a handshake slower than `connectTimeoutMs` (10 s by default) + now fails where it used to complete. Each step of setting up a connection (the + TCP connect, a proxy's reply, a handshake) has `connectTimeoutMs` to itself, so + the whole can take several times that. +* `TlsSocket::connect` bounds the handshake with the `timeoutMs` it already + took for the TCP connect, so a caller of it directly sees the change too. +* `TlsSocket::connect_over` takes a trailing `handshakeTimeoutMs`, `-1` (the + default) for no limit; a call by name compiles unchanged, a pointer to it needs + the new parameter in its type. `Socket` gains `set_deadline`, `deadline_hit` + and `wait_before_recv`. + ## 0.3.4 A request in flight can be abandoned from another thread. Everything is added diff --git a/README.md b/README.md index cc00774..d225c79 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ includes in that file, or use libc++ 23 or libstdc++. | field | default | what it decides | | --- | --- | --- | -| `connectTimeoutMs` | 10000 | TCP connect | +| `connectTimeoutMs` | 10000 | setting up the connection: the TCP connect, each wait for a proxy's reply, and the TLS handshake, with the proxy and with the target (the handshake as a whole, not each read in it) | | `readTimeoutMs` | 60000 | any single read, and the total wait on a blocked write | | `verifySsl` | true | verify the server certificate against the CA bundle (`SSL_CERT_FILE`, else the Windows `ROOT` certificate store in Windows Sockets builds, else a system location); the connection fails if the certificate is not trusted, has expired or is for another host, or if no bundle is found | | `keepAlive` | true | reuse connections between requests | diff --git a/src/http.cppm b/src/http.cppm index 6b83628..441ba12 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -1052,10 +1052,10 @@ private: } if (tunnel.proxyTls) { return sock.connect_over(std::move(tunnel.proxyTls), parsed.host.c_str(), - config_.verifySsl); + config_.verifySsl, config_.connectTimeoutMs); } return sock.connect_over(std::move(tunnel.socket), parsed.host.c_str(), - config_.verifySsl); + config_.verifySsl, config_.connectTimeoutMs); } return sock.connect(parsed.host.c_str(), parsed.port, config_.connectTimeoutMs, config_.verifySsl); diff --git a/src/socket.cppm b/src/socket.cppm index 6dbee5c..de56075 100644 --- a/src/socket.cppm +++ b/src/socket.cppm @@ -52,7 +52,8 @@ public: // Move constructor Socket(Socket&& other) noexcept - : fd_(other.fd_), stop_(std::move(other.stop_)) { + : fd_(other.fd_), stop_(std::move(other.stop_)), deadline_(other.deadline_) + , deadline_hit_(other.deadline_hit_) { other.fd_ = INVALID_SOCKET_FD; } @@ -62,6 +63,8 @@ public: close(); fd_ = other.fd_; stop_ = std::move(other.stop_); + deadline_ = other.deadline_; + deadline_hit_ = other.deadline_hit_; other.fd_ = INVALID_SOCKET_FD; } return *this; @@ -77,6 +80,42 @@ public: [[nodiscard]] bool stop_possible() const { return stop_.stop_possible(); } + // Bounds the TLS handshake: until it is cleared, `wait_before_recv` gives up + // once this time has passed. Cleared once the handshake is over. + void set_deadline(std::optional deadline) { + deadline_ = deadline; + deadline_hit_ = false; + } + + // True once `wait_before_recv` gave up because of the deadline. + [[nodiscard]] bool deadline_hit() const { return deadline_hit_; } + + // What `bio_recv` calls ahead of a `recv` that would otherwise block. True + // when a `recv` now will not block. With neither a token nor a deadline it + // answers true without waiting, so the plain `recv` is as it was. + // + // A wait that ends before the deadline is waited again for what is left: a + // signal ends `poll` with EINTR even under SA_RESTART, which the blocking + // `recv` this wait stands in front of would have restarted. The deadline is + // hit only when the clock says so. + bool wait_before_recv() { + if (!stop_.stop_possible() && !deadline_) return true; + for (;;) { + int wait = -1; + if (deadline_) { + const auto left = std::chrono::ceil( + *deadline_ - std::chrono::steady_clock::now()).count(); + wait = left > 0 ? static_cast(left) : 0; + } + if (wait_readable(wait)) return true; + if (stop_.stop_requested() || !deadline_) return false; + if (std::chrono::steady_clock::now() >= *deadline_) { + deadline_hit_ = true; + return false; + } + } + } + bool connect(const char* host, int port, int timeoutMs) { // Close existing connection if any if (is_valid()) { @@ -342,6 +381,8 @@ public: private: SocketHandle fd_ = INVALID_SOCKET_FD; std::stop_token stop_; + std::optional deadline_; + bool deadline_hit_ = false; // poll_fd in slices while a token is attached. A negative timeout waits // without limit. Not std::min: defines a `min` macro. diff --git a/src/tls.cppm b/src/tls.cppm index 8f16494..faf5bba 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -94,8 +94,9 @@ static int bio_send(void* ctx, const unsigned char* buf, size_t len) { // complete and correct. static int bio_recv(void* ctx, unsigned char* buf, size_t len) { auto* sock = static_cast(ctx); - // With a stop token, wait here so the recv below cannot block past a stop. - if (sock->stop_possible() && !sock->wait_readable(-1)) { + // With a stop token or a handshake deadline, wait here so the recv below + // cannot block past a stop or the deadline. + if (!sock->wait_before_recv()) { return MBEDTLS_ERR_NET_RECV_FAILED; } int ret = sock->read(reinterpret_cast(buf), static_cast(len)); @@ -208,20 +209,24 @@ public: // Connect over an already-established Socket (e.g. a proxy tunnel). // Takes ownership of the socket and performs TLS handshake on top of it. - bool connect_over(Socket&& socket, const char* host, bool verifySsl) { + // The handshake gives up after `handshakeTimeoutMs`; a negative value waits + // without limit. + bool connect_over(Socket&& socket, const char* host, bool verifySsl, + int handshakeTimeoutMs = -1) { error_.clear(); socket_ = std::move(socket); - return setup_tls(host, verifySsl); + return setup_tls(host, verifySsl, handshakeTimeoutMs); } // Run the handshake inside another TLS session, which is how a client // reaches a target through an https:// proxy: TLS to the proxy, CONNECT // inside it, then this session to the target inside the tunnel. Takes // ownership of `lower`, which must already be past its CONNECT. - bool connect_over(std::unique_ptr lower, const char* host, bool verifySsl) { + bool connect_over(std::unique_ptr lower, const char* host, bool verifySsl, + int handshakeTimeoutMs = -1) { error_.clear(); lower_ = std::move(lower); - return setup_tls(host, verifySsl); + return setup_tls(host, verifySsl, handshakeTimeoutMs); } bool connect(const char* host, int port, int timeoutMs, bool verifySsl) { @@ -231,7 +236,7 @@ public: return false; } - return setup_tls(host, verifySsl); + return setup_tls(host, verifySsl, timeoutMs); } // The read that says which of the four things happened. Prefer it over @@ -371,7 +376,16 @@ private: socket_.close(); } - bool setup_tls(const char* host, bool verifySsl) { + // As `set_stop`, the deadline belongs to the socket at the bottom, which + // inside a tunnel is the one beneath `lower_`. + void set_deadline(std::optional deadline) { + if (lower_) lower_->set_deadline(deadline); + else socket_.set_deadline(deadline); + } + + bool deadline_hit() const { return lower_ ? lower_->deadline_hit() : socket_.deadline_hit(); } + + bool setup_tls(const char* host, bool verifySsl, int handshakeTimeoutMs) { state_ = std::make_unique(); int ret = mbedtls_ctr_drbg_seed( @@ -429,17 +443,30 @@ private: // Set BIO callbacks using our Socket, or the session beneath this one bind_bio(); - // Perform TLS handshake - while ((ret = mbedtls_ssl_handshake(&state_->ssl)) != 0) { - if (ret != MBEDTLS_ERR_SSL_WANT_READ && ret != MBEDTLS_ERR_SSL_WANT_WRITE) { - if (ret == MBEDTLS_ERR_X509_CERT_VERIFY_FAILED) { - char info[512] = {}; - mbedtls_x509_crt_verify_info(info, sizeof info, "", - mbedtls_ssl_get_verify_result(&state_->ssl)); - return fail("certificate verification failed: " + one_line(info)); - } - return fail("TLS handshake failed: " + mbedtls_message(ret)); + // Perform TLS handshake. The socket is blocking, so `bio_recv` waits + // against the deadline before it reads, and the deadline is taken off + // again before the connection is used. + std::optional deadline; + if (handshakeTimeoutMs >= 0) { + deadline = std::chrono::steady_clock::now() + + std::chrono::milliseconds(handshakeTimeoutMs); + } + set_deadline(deadline); + while ((ret = mbedtls_ssl_handshake(&state_->ssl)) == MBEDTLS_ERR_SSL_WANT_READ + || ret == MBEDTLS_ERR_SSL_WANT_WRITE) {} + const bool timedOut = deadline_hit(); + set_deadline({}); + if (ret != 0) { + if (ret == MBEDTLS_ERR_X509_CERT_VERIFY_FAILED) { + char info[512] = {}; + mbedtls_x509_crt_verify_info(info, sizeof info, "", + mbedtls_ssl_get_verify_result(&state_->ssl)); + return fail("certificate verification failed: " + one_line(info)); + } + if (ret == MBEDTLS_ERR_NET_RECV_FAILED && timedOut) { + return fail("TLS handshake timed out"); } + return fail("TLS handshake failed: " + mbedtls_message(ret)); } return true; diff --git a/tests/test_handshake_timeout.cpp b/tests/test_handshake_timeout.cpp new file mode 100644 index 0000000..a825612 --- /dev/null +++ b/tests/test_handshake_timeout.cpp @@ -0,0 +1,289 @@ +// The TLS handshake is bounded by connectTimeoutMs. +// +// Each case gives the client a peer that completes the TCP connection and then +// never answers the ClientHello, with no stop token to end the wait. The peer +// holds the connection for at most 5 s and then drops it, so a handshake that +// has no deadline fails the case (by the message and by the time it took) +// instead of hanging the suite. + +// As in test_cancel.cpp, `import std` goes first with libc++ because of the +// stop token in one of the cases, and after the includes otherwise. +#include +#ifdef _LIBCPP_VERSION +import std; +#endif +#include +#include "proxy_test_server.hpp" +#include "tls_test_server.hpp" +#ifndef _WIN32 +#include +#include +#endif + +import mcpplibs.tinyhttps; +#ifndef _LIBCPP_VERSION +import std; +#endif + +namespace https = mcpplibs::tinyhttps; + +namespace { + +using Clock = std::chrono::steady_clock; + +// Far below the 5 s the peers hold on for, far above the 300 ms timeout. +constexpr long long kPromptMs = 3000; + +https::HttpClientConfig test_config() { + https::HttpClientConfig cfg; + cfg.verifySsl = false; // the test certificate authenticates nothing + cfg.connectTimeoutMs = 300; + cfg.readTimeoutMs = 30000; + return cfg; +} + +https::HttpRequest get(const std::string& url) { + https::HttpRequest req; + req.method = https::Method::GET; + req.url = url; + return req; +} + +long long since_ms(Clock::time_point start) { + return std::chrono::duration_cast(Clock::now() - start).count(); +} + +// Holds a connection for up to 5 s or until `done`. +void hold(const std::atomic& done) { + for (int i = 0; i < 500 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +} + +// A TCP listener that never speaks TLS: it reads the ClientHello and says nothing. +class Silent { +public: + Silent() : server_([this](proxy_test::Peer& peer) { + peer.read_some(); + hold(done_); + }) {} + ~Silent() { done_ = true; } + + [[nodiscard]] bool failed() const { return server_.failed(); } + [[nodiscard]] std::string url() const { + return "https://127.0.0.1:" + std::to_string(server_.port()) + "/"; + } + [[nodiscard]] std::string proxy_url(const char* scheme) const { + return std::string(scheme) + "://127.0.0.1:" + std::to_string(server_.port()); + } + +private: + std::atomic done_ { false }; + proxy_test::Server server_; +}; + +class HandshakeTimeoutTest : public ::testing::Test { +protected: + void SetUp() override { https::Socket::platform_init(); } +}; + +} // namespace + +TEST_F(HandshakeTimeoutTest, SendGivesUpOnAHandshakeThatNeverAnswers) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send(get(peer.url())); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, SendStreamGivesUpOnAHandshakeThatNeverAnswers) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send_stream(get(peer.url()), [](const https::SseEvent&) { return true; }); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A stop token that is never stopped takes the waits in slices; the deadline +// has to hold there as well. +TEST_F(HandshakeTimeoutTest, TheDeadlineHoldsWithAStopTokenAttached) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + https::HttpClient client(test_config()); + std::stop_source never; + const auto start = Clock::now(); + auto res = client.send(get(peer.url()), never.get_token()); + + EXPECT_FALSE(res.cancelled); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, TheHandshakeWithAnHttpsProxyIsBounded) { + Silent proxy; + ASSERT_FALSE(proxy.failed()); + + auto cfg = test_config(); + cfg.proxy = proxy.proxy_url("https"); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_NE(res.statusText.find("proxy: could not open a TLS connection"), std::string::npos) + << res.statusText; + EXPECT_NE(res.statusText.find("TLS handshake timed out"), std::string::npos) + << res.statusText; + EXPECT_LT(since_ms(start), kPromptMs); +} + +// The session to the target runs inside the one to the proxy, which answered +// the CONNECT and is now silent. +TEST_F(HandshakeTimeoutTest, TheHandshakeInsideAnHttpsProxyTunnelIsBounded) { + tls_test::Server proxy([](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(proxy.failed()); + + // The proxy's own handshake is real and has to fit in the timeout too. + auto cfg = test_config(); + cfg.connectTimeoutMs = 1000; + cfg.proxy = "https://127.0.0.1:" + std::to_string(proxy.port()); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +TEST_F(HandshakeTimeoutTest, TheHandshakeInsideAnHttpProxyTunnelIsBounded) { + std::atomic done { false }; + proxy_test::Server proxy([&](proxy_test::Peer& peer) { + peer.read_head(); + peer.write("HTTP/1.1 200 Connection established\r\n\r\n"); + peer.read_some(); + hold(done); + }); + ASSERT_FALSE(proxy.failed()); + + auto cfg = test_config(); + cfg.proxy = "http://127.0.0.1:" + std::to_string(proxy.port()); + https::HttpClient client(cfg); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:9/")); + done = true; + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A peer that sends a byte at a time keeps every read short, so only a limit on +// the handshake as a whole ends it: the record header promises 64 bytes and the +// peer sends fewer than that, one per 100 ms. +TEST_F(HandshakeTimeoutTest, ABytePerReadDoesNotHoldTheHandshakeOpen) { + std::atomic done { false }; + proxy_test::Server peerServer([&](proxy_test::Peer& peer) { + peer.read_some(); + if (!peer.write(std::string("\x16\x03\x03\x00\x40", 5))) return; + for (int i = 0; i < 50 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + if (!peer.write("\x01")) return; + } + }); + ASSERT_FALSE(peerServer.failed()); + + https::HttpClient client(test_config()); + const auto start = Clock::now(); + auto res = client.send(get("https://127.0.0.1:" + std::to_string(peerServer.port()) + "/")); + done = true; + + EXPECT_EQ(res.statusCode, 0); + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_LT(since_ms(start), kPromptMs); +} + +// A signal whose handler was installed with SA_RESTART restarts a blocking +// `recv`, so it never ended a handshake before the deadline existed. The wait +// that the deadline puts ahead of each `recv` is a `poll`, which a signal ends +// with EINTR whatever the flag; that has to be waited out, not taken for the +// deadline. The signals go to this thread, the one in the handshake. +#ifndef _WIN32 +namespace { +void ignore_signal(int) {} +} + +TEST_F(HandshakeTimeoutTest, ASignalDuringTheHandshakeIsNotATimeout) { + Silent peer; + ASSERT_FALSE(peer.failed()); + + struct sigaction action {}, previous {}; + action.sa_handler = ignore_signal; + action.sa_flags = SA_RESTART; + sigemptyset(&action.sa_mask); + ASSERT_EQ(sigaction(SIGUSR1, &action, &previous), 0); + + auto cfg = test_config(); + cfg.connectTimeoutMs = 1000; + https::HttpClient client(cfg); + const pthread_t self = pthread_self(); + std::atomic done { false }; + std::thread signaller([&] { + for (int i = 0; i < 5 && !done.load(); ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + if (!done.load()) pthread_kill(self, SIGUSR1); + } + }); + const auto start = Clock::now(); + auto res = client.send(get(peer.url())); + const auto took = since_ms(start); + done = true; + signaller.join(); + sigaction(SIGUSR1, &previous, nullptr); + + EXPECT_EQ(res.statusText, "TLS handshake timed out"); + EXPECT_GE(took, 900) << "the handshake ended at a signal, not at the deadline"; + EXPECT_LT(took, kPromptMs); +} +#endif + +// connectTimeoutMs is for establishing the connection. Once the handshake is +// done a read waits as long as it always did: here 1600 ms of silence after a +// 1000 ms timeout, and then the bytes arrive. The timeout leaves room for a +// real handshake on a slow machine. +TEST_F(HandshakeTimeoutTest, TheDeadlineIsLiftedOnceTheHandshakeIsDone) { + tls_test::Server server([](tls_test::Conn& conn, int) { + std::this_thread::sleep_for(std::chrono::milliseconds(1600)); + conn.write("x"); + conn.wait_peer_close(std::chrono::seconds(5)); + return false; + }); + ASSERT_FALSE(server.failed()); + + https::TlsSocket tls; + ASSERT_TRUE(tls.connect("127.0.0.1", server.port(), 1000, /*verifySsl=*/false)) << tls.error(); + const std::string request = "GET / HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"; + ASSERT_EQ(tls.write(request.data(), static_cast(request.size())), + static_cast(request.size())); + char byte = 0; + auto r = tls.read_some(&byte, 1); + + EXPECT_EQ(r.status, https::ReadStatus::Data); + EXPECT_EQ(byte, 'x'); +} From a0f774939e7b105cca54cd3eddd6ae91647212f1 Mon Sep 17 00:00:00 2001 From: Cloud_Yun Date: Sat, 10 Oct 2026 03:14:23 +0900 Subject: [PATCH 13/15] Add HttpClientConfig::extraCaFile to trust extra CA certificates (#24) SSL_CERT_FILE replaces the default store, so trusting a private CA or a TLS-inspecting proxy meant building a combined bundle by hand. The new field names a PEM file whose certificates are added to the default store. It applies to the target connection and to the connection to an https:// proxy. A file that cannot be read or holds no certificate fails the connection and the message names the path; it is not skipped. proxy_tunnel takes the path as a trailing default argument and TlsSocket gains set_extra_ca_file. With the field empty nothing changes. --- .github/workflows/ci.yml | 2 +- CHANGELOG.md | 15 +++ README.md | 16 +++ src/http.cppm | 11 +- src/proxy.cppm | 8 +- src/tls.cppm | 27 ++++- tests/extra_ca_certs.hpp | 161 ++++++++++++++++++++++++++ tests/test_extra_ca.cpp | 243 +++++++++++++++++++++++++++++++++++++++ 8 files changed, 477 insertions(+), 6 deletions(-) create mode 100644 tests/extra_ca_certs.hpp create mode 100644 tests/test_extra_ca.cpp diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 260f3ab..1da4c4c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -368,7 +368,7 @@ jobs: - name: The hermetic tests run: | set -o pipefail - for t in test_framing test_tls_verify test_pool test_proxy test_cancel test_handshake_timeout; do + for t in test_framing test_tls_verify test_pool test_proxy test_cancel test_handshake_timeout test_extra_ca; do mcpp test "$t" 2>&1 | tee "$t.log" grep -q "^$t \.\.\. ok" "$t.log" done diff --git a/CHANGELOG.md b/CHANGELOG.md index 0342d10..33a4a0d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## Unreleased +### TLS handshake time limit + The TLS handshake is bounded by `connectTimeoutMs`. It had no limit: after the TCP connection was up, a server (or an https:// proxy) that accepted it and never sent its ServerHello held `send`, `send_stream` and `download_to_file` for good @@ -23,6 +25,19 @@ unless the caller passed a stop token. the new parameter in its type. `Socket` gains `set_deadline`, `deadline_hit` and `wait_before_recv`. +### Extra CA certificates + +`HttpClientConfig::extraCaFile` is a path to a PEM file whose certificates are +trusted in addition to the default store, which `SSL_CERT_FILE` can only replace. +It is for a private CA or a proxy that re-signs TLS. + +* It applies wherever a trust store is built: the target, and the connection to + an `https://` proxy. +* A file that cannot be read or holds no certificate fails the connection, and + `statusText` names the file. +* `proxy_tunnel` takes the path as a trailing default argument, and `TlsSocket` + gains `set_extra_ca_file`. Nothing changes while the field is empty. + ## 0.3.4 A request in flight can be abandoned from another thread. Everything is added diff --git a/README.md b/README.md index d225c79..94aba0c 100644 --- a/README.md +++ b/README.md @@ -86,8 +86,24 @@ includes in that file, or use libc++ 23 or libstdc++. | `maxRedirects` | 10 | 0 disables redirect following | | `maxResponseBodyBytes` | 64 MiB | the most `send()` will hold in memory; does not bound `download_to_file` or `send_stream` | | `retryOnStaleConnection` | true | resend once when a pooled connection turns out to have been closed while idle | +| `extraCaFile` | empty | path of a PEM file of CA certificates to trust in addition to the default store, see below | | `proxy` | none | proxy URL, see below | +### Extra CA certificates + +A private CA, or a proxy that re-signs TLS, is not in the default store, and +`SSL_CERT_FILE` replaces that store instead of adding to it. Set `extraCaFile` +to a PEM file and its certificates are trusted as well as the default ones: + +```cpp +cfg.extraCaFile = "/etc/corp/root-ca.pem"; +``` + +It applies to the connection to an `https://` proxy as well as to the target. If +the file cannot be read or holds no certificate, the connection fails and +`statusText` names the file. The file adds to a default store and does not stand +in for one: with none found, the connection fails as described under `verifySsl`. + ### Proxies Set `proxy` to a URL and every request is tunnelled through it with `CONNECT`: diff --git a/src/http.cppm b/src/http.cppm index 441ba12..5cbdc73 100644 --- a/src/http.cppm +++ b/src/http.cppm @@ -107,6 +107,13 @@ export struct HttpClientConfig { // endpoints, and restricting it there would make the retry inapplicable to // nearly every request it exists to rescue. Set false to opt out. bool retryOnStaleConnection { true }; + + // A PEM file of CA certificates to trust in addition to the default store + // (`SSL_CERT_FILE`, or else the system roots), for a private CA or a proxy that + // re-signs TLS. It applies to the connection to an https:// proxy as well. + // A file that cannot be read or holds no certificate fails the connection, + // with the path in `statusText`. Empty means none. + std::string extraCaFile; }; // Progress callback for streaming downloads: (totalBytes, downloadedBytes) @@ -1042,10 +1049,12 @@ private: // connection failed: a proxy that refused the tunnel says so in its own words. bool open_connection(TlsSocket& sock, const ParsedUrl& parsed, std::string& error, std::stop_token stop) { + sock.set_extra_ca_file(config_.extraCaFile); if (config_.proxy.has_value()) { auto tunnel = proxy_tunnel(parse_proxy_url(config_.proxy.value()), parsed.host, parsed.port, - config_.connectTimeoutMs, config_.verifySsl, stop); + config_.connectTimeoutMs, config_.verifySsl, stop, + config_.extraCaFile); if (!tunnel.ok()) { error = std::move(tunnel.error); return false; diff --git a/src/proxy.cppm b/src/proxy.cppm index 9951291..bd51d92 100644 --- a/src/proxy.cppm +++ b/src/proxy.cppm @@ -347,12 +347,13 @@ export struct ProxyTunnel { } }; -// `verifySsl` is for the connection to an https:// proxy, and is the same -// setting that governs the connection to the target. +// `verifySsl` and `extraCaFile` are for the connection to an https:// proxy, and +// are the same settings that govern the connection to the target. export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, std::string_view targetHost, int targetPort, int timeoutMs, bool verifySsl = true, - std::stop_token stop = {}) { + std::stop_token stop = {}, + std::string_view extraCaFile = {}) { ProxyTunnel tunnel; tunnel.socket.set_stop(stop); const std::string where = proxy.host + ":" + std::to_string(proxy.port); @@ -369,6 +370,7 @@ export ProxyTunnel proxy_tunnel(const ProxyConfig& proxy, } else if (proxy.scheme == "https") { auto tls = std::make_unique(); tls->set_stop(stop); + tls->set_extra_ca_file(std::string(extraCaFile)); if (!tls->connect(proxy.host.c_str(), proxy.port, timeoutMs, verifySsl)) { // The session says why when the TCP connection was up, which is // where a proxy certificate that does not verify is refused. diff --git a/src/tls.cppm b/src/tls.cppm index faf5bba..112e633 100644 --- a/src/tls.cppm +++ b/src/tls.cppm @@ -166,7 +166,8 @@ public: : socket_(std::move(other.socket_)) , lower_(std::move(other.lower_)) , state_(std::move(other.state_)) - , error_(std::move(other.error_)) { + , error_(std::move(other.error_)) + , extraCaFile_(std::move(other.extraCaFile_)) { // Re-bind BIO to point to our socket_ (not the moved-from one) bind_bio(); } @@ -179,6 +180,7 @@ public: lower_ = std::move(other.lower_); state_ = std::move(other.state_); error_ = std::move(other.error_); + extraCaFile_ = std::move(other.extraCaFile_); // Re-bind BIO to point to our socket_ bind_bio(); } @@ -207,6 +209,10 @@ public: socket_.set_stop(std::move(stop)); } + // A PEM file of roots to trust in addition to the default store. Call before + // connect(); setup fails if the file cannot be read or holds no certificate. + void set_extra_ca_file(std::string path) { extraCaFile_ = std::move(path); } + // Connect over an already-established Socket (e.g. a proxy tunnel). // Takes ownership of the socket and performs TLS handshake on top of it. // The handshake gives up after `handshakeTimeoutMs`; a negative value waits @@ -328,6 +334,7 @@ private: std::unique_ptr lower_; std::unique_ptr state_; std::string error_; + std::string extraCaFile_; // The session beneath this one, when there is one, is closed as well: a // failed handshake to the target leaves nothing of the tunnel open. @@ -425,6 +432,24 @@ private: if (ret < 0) { return fail("cannot parse the CA certificate bundle: " + mbedtls_message(ret)); } + } + // Roots the caller adds to the above. A file that cannot be used is an + // error, not something to skip: the caller named it because a server + // needs it, and skipping would surface later as a verification failure + // that does not mention the file. + if (!extraCaFile_.empty()) { + std::ifstream in(extraCaFile_, std::ios::binary); + if (!in) return fail("cannot read extra CA file '" + extraCaFile_ + "'"); + std::string extra((std::istreambuf_iterator(in)), {}); + ret = mbedtls_x509_crt_parse( + &state_->ca_cert, + reinterpret_cast(extra.c_str()), extra.size() + 1); + if (ret < 0) { + return fail("cannot parse extra CA file '" + extraCaFile_ + "': " + + mbedtls_message(ret)); + } + } + if (!ca_pem.empty() || !extraCaFile_.empty()) { mbedtls_ssl_conf_ca_chain(&state_->conf, &state_->ca_cert, nullptr); } diff --git a/tests/extra_ca_certs.hpp b/tests/extra_ca_certs.hpp new file mode 100644 index 0000000..bfd8229 --- /dev/null +++ b/tests/extra_ca_certs.hpp @@ -0,0 +1,161 @@ +// Two private CAs, each with one server certificate (CN=localhost, SAN localhost +// and 127.0.0.1), for test_extra_ca.cpp. Valid for a century. The keys are here +// on purpose: they authenticate nothing and must never be used for anything. +// +// Made with openssl: a self-signed CA with basicConstraints CA:TRUE and +// keyUsage keyCertSign, and a leaf signed by it whose extensions are +// basicConstraints CA:FALSE, extendedKeyUsage serverAuth and +// subjectAltName DNS:localhost,IP:127.0.0.1. +#pragma once + +namespace extra_ca_test { + +inline constexpr const char* kCa1Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDLzCCAhegAwIBAgIUG2Zap6LiJwyHlkOt1Ko7G28l+DcwDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMTAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3Qg +Q0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALCFaWIMBSe8OolK +5ExxlX3LbNgg3XlOLxa604zHI0nLWFhKJalkticwwMaX4gcJQdETCXADq+V8qj48 +014c3uUC7xlb+INULhe2ocThipOSwhh4E1Vyv8ywNhgVEw8r+Lgl2ExaSIGzcx4O +2tWuOvzQHhe6JG8G5YRWVk5ydimhFGY6VdGzUPfHAID+3JEqr6RUeCct4Ehx2zbg +mkn7ZBu0PHtP2fjrc5KnVwXWDH91MbSfJ1WQvPq5kPbSpeeLoWvDznzpUjg+9aTB +Dbyxu+/5YVFy5jWWEpgOKakyKnfsQ6XTqzz6dvogAjjh3UWTHe01osluXqY8tjls +gapw+N8CAwEAAaNjMGEwHQYDVR0OBBYEFP3PSVqEY/jqpWJmAzftRY3zFZPtMB8G +A1UdIwQYMBaAFP3PSVqEY/jqpWJmAzftRY3zFZPtMA8GA1UdEwEB/wQFMAMBAf8w +DgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQAhMIwH6LSKiywbxaFK +RlUIqgThCIP9uUOZ0m/6jjgMwSJhU0IZ4COCewFVPEGepeqhNjDXHD9Y0EHbxDi5 +cCsk2fSz9xfsjBniI+xETlxGLoCrNDAmNEfPv8mT9UnNKG7KqLghQiHpMbx47895 +wTbr1OldFAlU8usR9pOigRc6j5cmf9b1Tu/piUTFsy5xoANPt/HRg3o/w0mfR8c5 +bHQ3LaqcQO+uBpKAy9guHpszKosOmbTX6AB5JBSyo70u3jx0PHgQuOJTEeKP1PBJ +51EyNShztp7bn+Rz0OLsMk2h6AMns7HB2m5gEIq/ELlaWDYruqiYFmGpqcm7deVd +v6DH +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf1Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDTzCCAjegAwIBAgIUO+WtedkpuiIwKSiJ20nYcEG4pNQwDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMTAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAj3gXKIHuAZ2h0DeMqxZX2ImgZJan +arOOw8vKg8yqyfgAeEtenmH42Hv4DAPhHVycq9DyXutiXVxQ+v1y7ABfG9RG2Z0u +Ueen2TqNDz5F6CrlUeG/Fgk3OVOWVeglcJpd8vqiVzrgRRtc4+uREGlu2YPusgSX +rbVL+6VARdEfypluaStOt1kJkQlLUxa5A7OpsXKIDLRnOKSHCyx1CCJNy3jwPmqz +vng6M/FYns/73TxqSZ8Wi7CRdCpNWjlwfJ0zunjeYdrGUchgVpbyWedexZzKwLMO +NFnjY20qPmN0ZIcXSg+IfZJcUKSD6t+kWZAOtcFwrO6xi7SWFQwNyskW3wIDAQAB +o4GMMIGJMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF +BwMBMBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATAdBgNVHQ4EFgQUNtDg1vH4 +jNeYzFRvdHWYhhdO6WEwHwYDVR0jBBgwFoAU/c9JWoRj+OqlYmYDN+1FjfMVk+0w +DQYJKoZIhvcNAQELBQADggEBAEUwKddb7KxhgZ5a41u78wRANsfkXHJEXX6l2IWL +cOAE9PU2LFj4muABzp7g6WA5rOF5Dt34E3PSB8EQIMTY6SBA9V6Y80/8uBlsF+Er +Labg6ABrvswpGPn7sc65m8uDUYdG5jm0P2lure9mV8kdiYURk0/eOIpdCp8CcuFx +iG1LsAMz9h0WAgbkfIzigDGx9TkvqTZNSfzvvObgGmeRfo++pLMEjifug9NBRHuN +dMiQ4NBvRv+HvJHGSAD1fpm8QvEiMk7spf+PyZbJZP34YZlbJfZJXvBxnTnJY6rD +amz7EBggI4kRiDGG5vxgyH2bYe9uzw01jb02QJyUvJqvOv4= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf1KeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCPeBcoge4BnaHQ +N4yrFlfYiaBklqdqs47Dy8qDzKrJ+AB4S16eYfjYe/gMA+EdXJyr0PJe62JdXFD6 +/XLsAF8b1EbZnS5R56fZOo0PPkXoKuVR4b8WCTc5U5ZV6CVwml3y+qJXOuBFG1zj +65EQaW7Zg+6yBJettUv7pUBF0R/KmW5pK063WQmRCUtTFrkDs6mxcogMtGc4pIcL +LHUIIk3LePA+arO+eDoz8Viez/vdPGpJnxaLsJF0Kk1aOXB8nTO6eN5h2sZRyGBW +lvJZ517FnMrAsw40WeNjbSo+Y3RkhxdKD4h9klxQpIPq36RZkA61wXCs7rGLtJYV +DA3KyRbfAgMBAAECggEAI/sGM86CFj2mO0XzLWUpGtFJPHDcA9KMV2Zgn7VQj952 +n7SnQiln80MD7l96sZZsZFq+rjdyL9/bH1c43x8oux39ts3ItBnY6o3Ymp4Pgi5f +3K008x3Uj7f62Aikexz7uthIpn/JhwtZvH9znh4TAtInptWoBJSXPcEv4VKVGoYI +HglTL1/TvIkyo2hwdLBqOuUF5PioHktG8MZH2ILqFJTfB5bWmOPXDMAEuZD9pkQK +LfyD9AntTRKVxRO/sXjHy27IXSKbDoZq4kqqLkxk0BGJpf4BnxxjZUWoqbQ7MZHs +MS26/fcpMWSJygYT8vMkAQYrGjrEkKQSvABFTmkegQKBgQDASz/4vpa+qPf9soX8 +n9+XXSPBeyNeNKpl2p0b+7h8skRIdi/gmttSiC9g/L2fuI2rpoikii1qxQ1XN+wA +LOcOrL5p/RJdGKAyu8n8cRSGFkhkOO5s2zLsSdXUHesVqWc8keI37DoJkySYEsiC +WI0MZNNK01cUFttD+nEqxgCicQKBgQC+/+3u+aWF1rumheoUV9pRk65F0jJRnTAQ +bfiRE7jWhs/LSYpJ1EOgc0OYaRnJPwKYuck4/Z7K3beNKPH3jPJ+89UvfRjHKE/3 +o89yACWZATh2XMjvZUUUmvn+Om+yMBxWhy1vyTJU+XhFVSIROofqg5xH3F3hMC6P +WIyJctVWTwKBgDUdE+BrFEw5c7Y5d1Td++5dJaEtmAlPVrmndrnh/4VB4CZ7rqHF +ZEsZnqyVYvMZENiWuStplz3ki9jJc4Bg5rlg8X9pDYd1Y9pfkF0QBvE6emhkaUbc +DObDRjK+yM5E6mUKjoeP0kAOyo7OsV9/ggYoW0xRr722yn2uUU/VPRuhAoGBALu/ +TX5cTdWa98g/2JH4rbUEqwkdDrNlBQsRXoYVUdsml3mnp1TdX0CILQjTPzKwSngg +H+lyfLXDbhtHBVqmZPRf8M4GdOXH/ZtanZ7dABg/t+W5XRbdgCM6F8VMYeFRI3n2 +1LYOmvgmZZAZmCxBUbs3dda5ilrCBvKHQ1YU2BpHAoGAGZVbkJ0lnCoXf3tyMBCZ +MliDj7Q6fSGd22ZAoYy/XGLmZUMi/ZK+PkdzVwT6CDwJY07vBDwUIj5qBHlmaHGS +qC7o3bmhdtC4R+eN4Gb6mdKB3wCdrRhx2HTBGDf4aoPwGtUw+kUdlve1L+Rwsdfc +zCIifQbNtFiGohWrskdamio= +-----END PRIVATE KEY----- +)PEM"; + +inline constexpr const char* kCa2Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDLzCCAhegAwIBAgIUe6yBH1jY6Ogs0HBiaUEsnbZlB54wDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMjAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3Qg +Q0EgMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJXfKHI6jLsG/yLn +UqcDb+PBuw4+C6eaYUwgJNJ5VNubXZpjg1JJxqBqOYFKJ69J5OULqBD2A3DWCAKh +vh6FOHeSdXlvwnZQN0fLwtnYcmL90XJaDrUwaFn/NrceqsgBo+pNXHcVb6OYwxQr +Np9/3kta6vbWp9cC8D6vCkmp2ehrz2UfBo4m6oh1Re2ATEBn0H2OD41LKyQslpF4 +CFkm2rIdkTsqlGtdoBkE+JyBX+eoZAlnRBRC5NxJDLEtDlxoPlbVRP/YgPeazv1g +tMgUvBhmC5AROU6oEpiXiGxas/3m3av/21shsq0iz+b+1QOF5gt9MdOEm2mxrxfF +hkf0pH0CAwEAAaNjMGEwHQYDVR0OBBYEFMV3Z2Qg+uzCT8vmKJV84SBhwzQQMB8G +A1UdIwQYMBaAFMV3Z2Qg+uzCT8vmKJV84SBhwzQQMA8GA1UdEwEB/wQFMAMBAf8w +DgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQBmve5C71LtGh24Sq3Q +kNsEROTQYwjbI1W1tEC5C/bqeUFLl4YxyzIHyGFiNSZ7aaaOSRrMEY2aCG2f7am0 +Rckygp6fFaTTAfWnsgH7fVxvBEwywNlLDaAU/co20cnFa1MfUO1MfstF56lqllgX +NHE/PAOCV/ZcO3DAmJ1lWbsJcXQKplbNlZIeEBwqJs2i3xpiJHxhckg95wyq2vnX +qtCBiGlK7LBawlHLAbYEXf4LTDAJvb5OFfb8vIJN5pn7PKvMhhMtp4i0r0nDgLkR +nP25ABxH8uw8KV7rCzp6i0LDSMRAU6pbs/C8kvMSv8BmF2FFK4OX4+29aIPH/utP +x/4p +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf2Pem = R"PEM(-----BEGIN CERTIFICATE----- +MIIDTzCCAjegAwIBAgIUBQEev9nx9U9d1N63ZONvDn/2u9owDQYJKoZIhvcNAQEL +BQAwHjEcMBoGA1UEAwwTdGlueWh0dHBzIHRlc3QgQ0EgMjAgFw0yNjEwMDkwODQ3 +MzNaGA8yMTI2MDkxNTA4NDczM1owFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4piCGItiClSpVhm3Rw2h/OTfgyE4 +vrEXqkEcHB7Oq4z23KeluS9r6eM/6i0ezRUTCI71Q6UBQ/n7mQUPb5ePHZygVqL+ +QEYGiO6xH2q3kB3oDzNixaPYUCg0u7WLCbv4E6E8ngqcpMSBTxTH/Be3AB1j0I9a +AH3xmVbY1E2V79uYhRVc7cQ0mNfXkj1K4G0AaPKuiwwVKnE7/z/BwA4vqWjH9jsV +s1JFhjPpydBY73VWEVE0Ou1ntBNoKGilsVJBs4lf4P8pu9hBS8H0PCAk784kVY8n +0Y1G2xU3fSash6HjljjKtWg+mudbu2edVG8pkv13JYxJNNIUCXLvRH6m4QIDAQAB +o4GMMIGJMAkGA1UdEwQCMAAwCwYDVR0PBAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF +BwMBMBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATAdBgNVHQ4EFgQUSfHOwhMo +5rrgK8aDsQ4K7lvGJ5IwHwYDVR0jBBgwFoAUxXdnZCD67MJPy+YolXzhIGHDNBAw +DQYJKoZIhvcNAQELBQADggEBAIQ2kQBTG7wSFAa/fGup6cph+pV2ZVikoQUN0C9U +29uJaGALXNhx5RUDduPZP0i1Q9chYh55HcCT/CtbVlkO3X3BwLmgvqJel0O0nDnR +bIA90Ccs0wOYBY1bkWqcjqALJlddjeeerYdV3PulAp5uklf5ZWxgxl6XSucRW1fH +wvtBbQ7TirbYos0nZqAF4pxpMq4KLfWa7eAVMThL6ImiQyOYZsblMxZqAOPh8RIU +lmZG8/79aHwWyQdCC7p91dD5bCtOwrn1VZvTgmrzp8+TQnzqGuKAlUZEJXp/OF+a +qana9iMkkKQyE7y+DGZz09xKrvlh8l7kxJMES6ynN/XjITg= +-----END CERTIFICATE----- +)PEM"; + +inline constexpr const char* kLeaf2KeyPem = R"PEM(-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDimIIYi2IKVKlW +GbdHDaH85N+DITi+sReqQRwcHs6rjPbcp6W5L2vp4z/qLR7NFRMIjvVDpQFD+fuZ +BQ9vl48dnKBWov5ARgaI7rEfareQHegPM2LFo9hQKDS7tYsJu/gToTyeCpykxIFP +FMf8F7cAHWPQj1oAffGZVtjUTZXv25iFFVztxDSY19eSPUrgbQBo8q6LDBUqcTv/ +P8HADi+paMf2OxWzUkWGM+nJ0FjvdVYRUTQ67We0E2goaKWxUkGziV/g/ym72EFL +wfQ8ICTvziRVjyfRjUbbFTd9JqyHoeOWOMq1aD6a51u7Z51UbymS/XcljEk00hQJ +cu9EfqbhAgMBAAECggEAFmUu1HiwKae47cjQX64mAK3pNldZ+aTcmyfgvJnB4Nwn +iOLr4uvgLUdXyZAIj4nh7fJdJMCh6aNDHdpNboDK9QkGItHssZMKMc3vhRShzFVc +WmZAgJqjNjbnEVdNxNpGWEBTEqanWTSPuC5sXorEfDcrZpFIeqx37KkT+JWreD9r +DEsHVL51wGwD7KC06qv/IUbQXrfUtwJouof0X3iF/1bSmBTi3gVvyRpBaECRAZab +bOktlfKP/9FquA7KegY0HaaByXC073lj7pl3H9AmnYihXfTHSnxRMLq31IhpWzIN +Bbx/+vaVWgt/lWdedAVg8wVqlen4PSTstKYpq9WrHQKBgQD0SeekTgwjHfixaGm8 +7BoYHrszO32ua7BPAi8zXjbJi2SwFqyLlsw52FaYMMFxWRCarhtOKzUR0AnfB+Tp +HUsNlg5IOzeBXz26Care6Ymmx0jc1ZTrNMNoSTRMhZ4Cq16pD/mXSFNRQaBxP1Uy +ZTKzbz/UMZ6dbw+wY0lRp5gABQKBgQDtdXZHyPWns5OY34IYkO/cX3FY7K6BqqQz +8ciULN1v8bK32J8NXuuHRpbl+gJQ9ZwSzTyUBEW2EJArSmiVQBWLfITOv+Ubb34a +7RmiMO7ectqDsydpXyNr5GVvmJSN7ua/Bgse71ZEER+5T+Wk1IFrmc+dgWD7p1wX +DPoRGVruLQKBgGxA++extmsxcu2bKZBQyFd79tUDPfS3QBAjFaubyoeRv/LQeXH6 +R1Iy7lzO7ko+f78r/Gmtd/0GxfkR1H0BQCsrHfUgWL556C6y7geOUjOYCE/kTfPV +E+r41bY7WAQy0OX2cFJ7+H//pwvIVwbYFZmyaQStYG6/sqIlvW3z6MXJAoGAFNv1 +qPapRbMLqykvJ3NL7sLflR0tcyyHQhziLsoGJbhIp8e6qRUGgZqc1NTycSQyMqCe +yq3ZZdXspDDkIQBBJECq9fJ7cL6JiFn9n3bG9PmH4nvRYXs/BFJK6tOvcQxkbrFc +Nrp1jIMok0HGfYVBXS9kwoRD/bdE6q10Jh5hE30CgYEA1cUVunt7NYrY2NV+jF+1 +kp9Imom/F0ao4wowMmrcJTm1zgw4BehYaRoE7BVOxoq4Rc7SBgSlvb8aoy7EXuu7 +eieh8GE4WeiFGcDtCdpS82s/3IOeHwHo9I7wFa16r1jy/ooygVPM3RB8q7KOmfP+ +Hmwo4XQ7djlELruiEfglIt4= +-----END PRIVATE KEY----- +)PEM"; + +} // namespace extra_ca_test diff --git a/tests/test_extra_ca.cpp b/tests/test_extra_ca.cpp new file mode 100644 index 0000000..f86875c --- /dev/null +++ b/tests/test_extra_ca.cpp @@ -0,0 +1,243 @@ +// `HttpClientConfig::extraCaFile` adds roots to the default trust store. +// +// The default store is whatever SSL_CERT_FILE names, so each test writes the +// bundles it needs to temp files and the machine's own store never enters into +// it. The servers present leaves signed by two private CAs (extra_ca_certs.hpp); +// nothing leaves 127.0.0.1. +#include +#include "extra_ca_certs.hpp" +#include "tls_test_server.hpp" + +#include + +import mcpplibs.tinyhttps; +import std; + +namespace https = mcpplibs::tinyhttps; +namespace pki = extra_ca_test; + +namespace { + +void set_env(const char* name, const char* value) { +#ifdef TINYHTTPS_WINSOCK + _putenv_s(name, value); +#else + ::setenv(name, value, 1); +#endif +} + +class ExtraCaTest : public ::testing::Test { +protected: + void SetUp() override { + https::Socket::platform_init(); + const char* old = std::getenv("SSL_CERT_FILE"); + hadOld_ = old != nullptr; + if (hadOld_) old_ = old; + } + + void TearDown() override { + set_env("SSL_CERT_FILE", hadOld_ ? old_.c_str() : ""); + std::error_code ec; + for (auto& path : files_) std::filesystem::remove(path, ec); + } + + // Writes `content` to a new temp file and returns its path. + std::string write_file(std::string_view content) { + auto path = std::filesystem::temp_directory_path() + / ("tinyhttps-extra-ca-" + + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + + "-" + std::to_string(files_.size()) + ".pem"); + std::ofstream(path, std::ios::binary) << content; + files_.push_back(path); + return path.string(); + } + + // Makes `content` the whole default store. + void default_store(std::string_view content) { + set_env("SSL_CERT_FILE", write_file(content).c_str()); + } + + static https::HttpClientConfig config(std::string extraCaFile = {}) { + https::HttpClientConfig cfg; // verifySsl is true by default + cfg.connectTimeoutMs = 4000; + cfg.readTimeoutMs = 2000; + cfg.extraCaFile = std::move(extraCaFile); + return cfg; + } + + static https::HttpResponse get(const https::HttpClientConfig& cfg, const std::string& url) { + https::HttpClient client(cfg); + https::HttpRequest req; + req.url = url; + return client.send(req); + } + +private: + std::vector files_; + std::string old_; + bool hadOld_ { false }; +}; + +bool answer_ok(tls_test::Conn& conn, int) { + conn.write(tls_test::ok_response("ok")); + return true; +} + +tls_test::Server server_of_ca1() { + return tls_test::Server(answer_ok, pki::kLeaf1Pem, pki::kLeaf1KeyPem); +} + +tls_test::Server server_of_ca2() { + return tls_test::Server(answer_ok, pki::kLeaf2Pem, pki::kLeaf2KeyPem); +} + +} // namespace + +TEST_F(ExtraCaTest, APrivateCaIsNotTrustedWithoutTheFile) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(tls_test::kCertPem); // a store that does not contain CA 1 + + auto r = get(config(), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("not correctly signed by the trusted CA"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, APrivateCaIsTrustedWithTheFile) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(tls_test::kCertPem); + + auto r = get(config(write_file(pki::kCa1Pem)), server.url("/")); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + EXPECT_EQ(r.body, "ok"); +} + +// The file adds to the default store; it does not replace it. +TEST_F(ExtraCaTest, TheDefaultStoreStaysInEffect) { + auto fromDefault = server_of_ca1(); + auto fromExtra = server_of_ca2(); + auto neither = tls_test::Server(answer_ok); // self-signed, in no store + ASSERT_FALSE(fromDefault.failed()); + ASSERT_FALSE(fromExtra.failed()); + ASSERT_FALSE(neither.failed()); + default_store(pki::kCa1Pem); + auto cfg = config(write_file(pki::kCa2Pem)); + + EXPECT_EQ(get(cfg, fromDefault.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, fromExtra.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, neither.url("/")).statusCode, 0); +} + +TEST_F(ExtraCaTest, AFileHoldingSeveralCertificatesTrustsAllOfThem) { + auto a = server_of_ca1(); + auto b = server_of_ca2(); + ASSERT_FALSE(a.failed()); + ASSERT_FALSE(b.failed()); + default_store(tls_test::kCertPem); + auto cfg = config(write_file(std::string(pki::kCa1Pem) + pki::kCa2Pem)); + + EXPECT_EQ(get(cfg, a.url("/")).statusCode, 200); + EXPECT_EQ(get(cfg, b.url("/")).statusCode, 200); +} + +// Both hops of a request through an https:// proxy use the file: the proxy's +// certificate comes from CA 2, which only the file names, and the target's from +// CA 1, which only the default store names. +TEST_F(ExtraCaTest, TheHopToAnHttpsProxyUsesTheFileToo) { + auto target = server_of_ca1(); + ASSERT_FALSE(target.failed()); + const int targetPort = target.port(); + auto proxy = tls_test::Server([targetPort](tls_test::Conn& conn, int) { + conn.write("HTTP/1.1 200 Connection established\r\n\r\n"); + conn.relay_to(targetPort); + return false; + }, pki::kLeaf2Pem, pki::kLeaf2KeyPem); + ASSERT_FALSE(proxy.failed()); + default_store(pki::kCa1Pem); + + auto cfg = config(write_file(pki::kCa2Pem)); + cfg.proxy = "https://127.0.0.1:" + std::to_string(proxy.port()); + auto r = get(cfg, target.url("/")); + EXPECT_EQ(r.statusCode, 200) << r.statusText; + + cfg.extraCaFile.clear(); + r = get(cfg, target.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_TRUE(r.statusText.starts_with("proxy: could not open a TLS connection to 127.0.0.1:")) + << r.statusText; + EXPECT_NE(r.statusText.find("certificate verification failed"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileThatDoesNotExistIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); // the server would verify without the file + const auto missing = (std::filesystem::temp_directory_path() / "tinyhttps-no-such-ca.pem").string(); + + auto r = get(config(missing), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("cannot read extra CA file '" + missing + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileThatIsNotPemIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("this is not a certificate\n"); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AFileWhoseCertificateCannotBeParsedIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("-----BEGIN CERTIFICATE-----\nAAAA\n-----END CERTIFICATE-----\n"); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +TEST_F(ExtraCaTest, AnEmptyFileIsReportedWithItsPath) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file(""); + + auto r = get(config(path), server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +// The file is checked whether or not the server would be verified. +TEST_F(ExtraCaTest, AFileThatCannotBeUsedFailsEvenWithoutVerification) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + const auto path = write_file("not a certificate\n"); + + auto cfg = config(path); + cfg.verifySsl = false; + auto r = get(cfg, server.url("/")); + EXPECT_EQ(r.statusCode, 0); + EXPECT_NE(r.statusText.find("extra CA file '" + path + "'"), std::string::npos) + << r.statusText; +} + +// SSL_CERT_FILE keeps its meaning when the field is unset. +TEST_F(ExtraCaTest, WithoutTheFieldNothingChanges) { + auto server = server_of_ca1(); + ASSERT_FALSE(server.failed()); + default_store(pki::kCa1Pem); + EXPECT_EQ(get(config(), server.url("/")).statusCode, 200); +} From 6153cdd162d0b87651795b40b9ef54452888ce05 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 10 Oct 2026 03:14:55 +0900 Subject: [PATCH 14/15] 0.3.5 --- the TLS handshake has a time limit, and extra CA certificates can be trusted The handshake is bounded by connectTimeoutMs (#25), and HttpClientConfig::extraCaFile adds a PEM file of roots to the default store (#24). A patch release: "^0.3.1" resolves to it. The plan, with the review of both PRs and the CI failure that Git for Windows 2.56 caused, is in .agents/docs/exec-plan-0.3.5.md. --- .agents/docs/exec-plan-0.3.5.md | 169 ++++++++++++++++++++++++++++++++ CHANGELOG.md | 21 +++- README.md | 4 +- mcpp.toml | 2 +- 4 files changed, 190 insertions(+), 6 deletions(-) create mode 100644 .agents/docs/exec-plan-0.3.5.md diff --git a/.agents/docs/exec-plan-0.3.5.md b/.agents/docs/exec-plan-0.3.5.md new file mode 100644 index 0000000..20080fd --- /dev/null +++ b/.agents/docs/exec-plan-0.3.5.md @@ -0,0 +1,169 @@ +# 执行计划 —— 0.3.5：额外信任的 CA，TLS 握手有时限，Windows CI 恢复 + +| | | +|---|---| +| 输入 | [PR #24](https://github.com/mcpplibs/tinyhttps/pull/24) `feat/extra-ca-certs @ bd76c9e` · [PR #25](https://github.com/mcpplibs/tinyhttps/pull/25) `fix/tls-handshake-timeout @ cfdab98`（作者都是 yspbwx2010，`maintainerCanModify: true`） | +| 基线 | `origin/master @ 2cc7310`（0.3.4） | +| 落地 | 两个 PR 各自 squash 合入；对 #25 的修正直接追加在 #25 上（已推送 `97a0657`） | +| 目标版本 | **0.3.5**（D1–D4 按建议采纳） | +| 跨仓库 | `mcpplibs/tinyhttps` → gitcode `mcpp-res/tinyhttps`（CN 镜像）→ `mcpplibs/mcpp-index` | +| 文档日期 | 2026-10-10 | + +**标注约定**：✅ 已核验源码 · 🔁 已在本地复现或跑通 · ❓ 推断，未实测 + +--- + +## 0. TL;DR + +- **两个 PR 的方向都对，设计也符合本项目的约定**：只新增、尾部默认参数、新字段放最后、错误写进 `statusText`、状态只放在传输链最底层的 `Socket`（沿用 0.3.4 的 I1）、测试不联网（hermetic）。和业界通行做法也一致（§2）。 +- **#24 可以直接合**，不需要追加 commit。剩下几处小问题不影响合入（§3.3）。 +- **#25 有一个回归，已修复并推到 PR 上** 🔁：handshake 的等待改成了 `poll`，信号会让它以 EINTR 返回，即使 handler 装的是 SA_RESTART 也一样。结果是装了信号 handler 的进程（profiler、定时器）握手会被误报成 “TLS handshake timed out”。修复前新加的用例在 100 ms 就失败了（时限是 1000 ms）。 +- **Windows CI 红是 runner 镜像的变化，和两个 PR 都无关** ✅：win22/20261004.326 带的是 Git for Windows 2.56，原来硬编码的 `mingw64/etc/ssl/certs/ca-bundle.crt` 不存在了。修复已随 #25 一起推送：向 git 查询 bundle 的路径，查不到再在安装目录里搜索。 +- **合入顺序是 #25 → #24**：#25 带着 CI 修复，先合入后 master 才能变绿；#24 再同步 master，同时解决 CHANGELOG 的冲突。 + +### 待你决定 + +| # | 决策 | 建议 | +|---|---|---| +| D1 | 版本号用 0.3.5 还是 0.4.0 | **0.3.5**。#25 严格说改变了行为（握手超过 `connectTimeoutMs` 会失败），不符合 0.3.4 “补丁版本只新增”的约定。但它修的是一个永久卡死的问题；而且 0.4.0 不会被 `^0.3.x` 解析到，最需要这个修复的正是已经在用 `^0.3.x` 的项目。在 CHANGELOG 里用 “Behaviour change” 单列说明 | +| D2 | `connectTimeoutMs` 继续按阶段各自计时，还是改成整个建连过程共用一个 deadline | **0.3.5 保持按阶段计时**（和现有的代理应答等待一致，PR 已在文档里写明）。共用 deadline 和 curl 的语义一样，更符合直觉，留到 0.4 统一做（§4） | +| D3 | #24 是否同时提供内存中的 PEM（如 `extraCaPem`） | **这次不做**，作为后续单独的一项。路径方式已经覆盖代理、企业 CA 这些主要场景 | +| D4 | `Socket::set_deadline / deadline_hit / wait_before_recv` 都是 public（因为整个类是 export 的） | **接受**。`set_stop / stop_possible` 已经是这样，跨 partition 用 friend 在各家编译器上的风险更大；注释已写明它们是给 TLS 层用的 | + +--- + +## 1. CI 失败分析（run 37911364113 / job 113757995241） + +| 项 | 内容 | +|---|---| +| 现象 | `run (x86_64-windows-musl, above openkal, on Windows)` 运行 4 秒后失败：`cp: cannot stat '/c/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt'` ✅ | +| 根因 | runner 镜像从 win22/20260927.320 换成了 win22/20261004.326，Git for Windows 也从 2.55 升到 2.56 ✅。2.56 把环境从 MINGW64 换成了 UCRT64，bundle 现在在 `/c/Program Files/Git/ucrt64/etc/ssl/certs/ca-bundle.crt`（新 CI 日志里打印出来的路径）✅ | +| 影响 | 所有分支的这个 job 都会失败，在 smoke 程序运行之前就挂了。和 #24、#25 的代码都无关 | +| 修复 | 已推送到 #25 的 `bf99dbd`：先用 `git config --system http.sslcainfo` 查路径（经 `cygpath` 转换），查不到就在 `/c/Program Files/Git` 下搜索 `*/ssl/certs/ca-bundle.crt`，还找不到就报 `::error::` 并给出明确原因。同时打印 `git --version` 和最终用到的路径，方便下次镜像再变时定位 | +| 验证 | #25 的 CI run 37968516327：10 个 job 全绿，这个 job 用 `git config` 查到了 ucrt64 下的路径；smoke 输出 `cancellation: ok (stopped at 200 ms, returned after 261 ms)` ✅ | +| 顺带发现 | 这个 job 里 `framing parsers: ok` 和 `cancellation: ok` 之间有约 39 s 的空档，10-05 master 的运行里也有，所以不是这两个 PR 引入的。空档出现在计时开始之前，原因没有查 ❓，列入 §4 | + +--- + +## 2. 设计评审 + +### 2.1 是否符合本项目的约定 + +| 约定（来源） | #24 | #25 | +|---|---|---| +| 只新增：尾部默认参数、新字段放最后（0.3.4 §0） | ✅ `extraCaFile` 放在最后；`proxy_tunnel` 加了尾部默认参数 | ✅ `connect_over` 加了尾部默认参数；CHANGELOG 点名说明了成员函数指针的不兼容 | +| 状态只放最底层 `Socket`，`TlsSocket` 只往下转发（0.3.4 I1） | 不涉及 | ✅ `set_deadline` / `deadline_hit` 都经 `lower_` 往下转发，https 代理隧道的情况有测试覆盖 | +| 状态只在调用期间存在（0.3.4 I2） | 不涉及 | ✅ 握手结束就清掉 deadline，有测试覆盖 | +| 不配置时行为不变（0.3.4 I3） | ✅ 字段为空时一行代码都不走 | ⚠️ 默认的 10 s 会生效，这是有意的行为变化（见 D1） | +| 失败原因写进 `statusText`，能直接看懂 | ✅ 错误信息里带文件路径 | ✅ 显示 `TLS handshake timed out`；代理那一跳还会带上代理地址 | +| 测试不联网，能在 Windows 上跑 | ✅ 已加进 Windows 的 hermetic 列表 | ⚠️ **原 PR 没有加**，现已补上（`bf99dbd`） | +| 不引入新的环境变量或全局状态 | ✅ 只增加一个字段 | ✅ 不新增配置项 | + +### 2.2 语义和业界做法对照 + +**#24 `extraCaFile`：在默认 store 之外再追加信任的 CA** + +| 实现 | 语义 | +|---|---| +| Node `NODE_EXTRA_CA_CERTS` | **追加**，和 #24 完全一样，名字也对应 | +| reqwest `add_root_certificate` | **追加**到内置根证书 | +| curl `--cacert` / `CURLOPT_CAINFO`、Python requests `verify=path`、Go `tls.Config.RootCAs` | **替换**（本库对应的是 `SSL_CERT_FILE`） | + +结论：库里原本只有“替换”的方式（`SSL_CERT_FILE`），#24 补上了“追加”，这是业界常见的另一半。文件不可用时直接失败、不跳过，这个判断对：跳过的话，错误最后会变成一个不提文件的证书校验失败。 + +**#25 握手时限并入 `connectTimeoutMs`** + +| 实现 | 握手的时限 | +|---|---| +| curl `CURLOPT_CONNECTTIMEOUT` | 包含 TLS 握手，并且整个建连阶段共用一个时限 | +| Go `http.Transport.TLSHandshakeTimeout` | 单独一项，`DefaultTransport` 默认 10 s | +| #25 | 包含握手，但**每个阶段各自计时** | + +结论：并入 `connectTimeoutMs`（不新增配置项）和 curl 一致，默认 10 s 也和 Go 的默认值相同。差别在于各阶段各自计时，最坏情况下总时间是 N × timeout。PR 和 README 都已写明（见 D2）。 + +### 2.3 API 好不好用 + +| 点 | 评价 | +|---|---| +| `cfg.extraCaFile = "/etc/corp/root-ca.pem";` 一行就能用 | 好。代理那一跳自动生效，不用再配置一次 | +| 只接受文件路径 | 证书打包进程序、或者从配置中心拿到的场景要先落盘再用（D3） | +| 只能指定一个文件 | 和 Node 一样，可以把多个证书拼成一个文件，够用 | +| `verifySsl=false` 时文件读不到也会失败 | 有意设计（所有机器上报同样的错），README 里再补一句说明会更好 | +| 握手超时不需要任何改动就生效 | 好，解决的是默认配置下的卡死问题 | +| `connect_over(..., handshakeTimeoutMs = -1)` 和 `connect(..., timeoutMs, ...)` 的默认值不对称 | 为了兼容只能这样：`connect_over` 原本就不带超时 | + +--- + +## 3. 问题清单与处理 + +### 3.1 #25（已推送：`cfdab98..97a0657`） + +| # | 级别 | 问题 | 处理 | +|---|---|---|---| +| H1 | 高 | EINTR 被当成 deadline 到期。`wait_before_recv` 只要等待失败就设置 `deadline_hit_`，不看时钟。没有 token 时只调用一次 `poll`，信号一来就返回。master 上走的是阻塞的 `recv`，在 SA_RESTART 下会自动重启，所以这是回归 🔁 | `3013b98`：提前返回就按剩余时间继续等，只有时钟确实到了才算 deadline 到期。新用例 `ASignalDuringTheHandshakeIsNotATimeout`：时限 1000 ms，每 100 ms 用 `pthread_kill` 给握手线程发一次 SIGUSR1。修复前 100 ms 就失败，修复后通过 🔁 | +| M1 | 中 | `test_handshake_timeout` 没进 Windows 的 hermetic 列表，`WSAPoll` 路径没人测 | `bf99dbd` | +| M2 | 中 | Windows openkal job 的 bundle 路径（§1） | `bf99dbd` | +| L1 | 低 | CHANGELOG 没提导出的 `TlsSocket::connect` 现在也会限制握手时间 | `97a0657` | +| L2 | 低 | 移动构造和移动赋值复制了 `deadline_`，但没复制 `deadline_hit_` | `3013b98` | +| — | 已知限制 | 握手期间的 `send` 是阻塞的；`readTimeoutMs` 管不到阻塞的写（README 里的说法和实际不符） | 不在本版本处理，单列一项（§4） | + +本地结果（Linux, LLVM 22）🔁：`test_handshake_timeout` 9/9、`test_cancel` 24、`test_proxy` 43、`test_pool` 27、`test_tls_verify` 6、`test_framing` 17，全部通过。 + +### 3.2 #24：可以直接合入，不追加 commit + +本地结果 🔁：`test_extra_ca` 11/11，其余测试不受影响。#24 和 #25 合在一起后，全部测试也都通过（`test_extra_ca` + `test_handshake_timeout` 一起跑过）。 + +### 3.3 #24 的小问题（合入后视情况再处理，不阻塞本版本） + +- 在 Linux 上，路径是目录时 `ifstream` 也能打开，错误信息变成 “cannot parse”，不是 “cannot read”。加一个 `is_regular_file` 判断就能修。 +- 部分证书解析失败（`ret > 0`）不会报错，和默认 bundle 的处理一致，README 可以补一句。 +- 每建一个新连接都要重新读、重新解析一次文件。默认 bundle 也是这样，暂时不缓存。 + +--- + +## 4. 0.3.5 之后（不在本版本范围内） + +| 项 | 说明 | +|---|---| +| 写操作不阻塞 | `bio_send` / `write_all` 在阻塞 socket 上永远拿不到 WANT_WRITE，所以 `readTimeoutMs` 管不到阻塞的写，握手期间的写也不受 deadline 约束。这是 0.3.4 D4 留下的问题 | +| 整个建连共用一个 deadline（D2） | DNS、TCP（每个地址）、代理应答、两次握手共用一个 `connectTimeoutMs`。行为有变化，适合 0.4 | +| `extraCaPem`（D3） | 直接从内存加载 PEM | +| openkal Windows smoke 的 39 s 空档 | 见 §1，master 上已有，原因待查 | +| `poll_fd` 统一处理 EINTR | TCP connect 和代理应答的等待也有同样的问题，H1 只修了握手。可以在 `poll_fd` 一处统一修 | + +--- + +## 5. 任务依赖图 + +``` +T0 计划（本文）← 你在这里审 + │ + ├─ T1 #25 追加 commit（已推送 97a0657）→ CI 全绿 ✅（run 37968516327） + │ + ├─ T2 更新 #25 的 PR 描述：Not covered 里关于 EINTR 的说法改成“已修复”；补充 CI 的变化 + │ + ├─ T3 squash 合入 #25（依赖 T1、T2） + │ + ├─ T4 #24 同步 master：合并两边 CHANGELOG 的 “Unreleased” 段 → CI 全绿 → squash 合入 + │ + ├─ T5 发版 commit：CHANGELOG 的 Unreleased → 0.3.5；mcpp.toml version；README 安装行（mcpp add / 依赖示例） + │ + └─ T6 发布（依赖全部） + tag 0.3.5 + GitHub release + → gitcode mcpp-res/tinyhttps 上传 0.3.5 tarball（核对和 GitHub archive 的 sha256 一致） + → mcpp-index PR：pkgs/t/tinyhttps.lua 加 0.3.5（GLOBAL / CN，三个平台）， + tests/examples/tinyhttps 升到 0.3.5，加一条 `extraCaFile` 的 API 检查 + → 用已发布的 index 实测：新建项目写 "^0.3.1" 能解析到 0.3.5，GLOBAL 和 CN 两个源都验证 +``` + +**关键路径**：T1（CI）→ T3 → T4 → T5 → T6。 + +## 6. 验收 + +- [x] #24、#25 以及两者合并后，hermetic 测试在本地全部通过 🔁 +- [x] H1 能复现，修复后用例通过 🔁；已推送到 #25 +- [x] #25 CI 全绿（run 37968516327，10/10），包括 Windows openkal 和 Windows hermetic（`test_handshake_timeout`） +- [x] #25 squash 合入（`9d4315e`）；#24 同步 master（合并时 CHANGELOG 和 Windows 测试列表都有冲突）后 CI 10/10 全绿，squash 合入（`a0f7749`） +- [ ] 版本号按 D1 确定；tag、GitHub release +- [ ] gitcode tarball 的 sha256 一致 +- [ ] mcpp-index 合入；`"^0.3.1"` 在两个源下都解析到 0.3.5 diff --git a/CHANGELOG.md b/CHANGELOG.md index 33a4a0d..fcd7dcf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,18 @@ # Changelog -## Unreleased +## 0.3.5 + +A handshake that never completes no longer holds a request for good, and a +private CA can be trusted alongside the default store. It is a patch release, so +a dependency written as a caret range, `tinyhttps = "^0.3.1"`, resolves to it; +the one change in behaviour, a handshake slower than `connectTimeoutMs` now +failing, is the end of a hang rather than a new failure mode, and is described +below. ### TLS handshake time limit -The TLS handshake is bounded by `connectTimeoutMs`. It had no limit: after the +[#25](https://github.com/mcpplibs/tinyhttps/pull/25). The TLS handshake is +bounded by `connectTimeoutMs`. It had no limit: after the TCP connection was up, a server (or an https:// proxy) that accepted it and never sent its ServerHello held `send`, `send_stream` and `download_to_file` for good unless the caller passed a stop token. @@ -27,7 +35,7 @@ unless the caller passed a stop token. ### Extra CA certificates -`HttpClientConfig::extraCaFile` is a path to a PEM file whose certificates are +[#24](https://github.com/mcpplibs/tinyhttps/pull/24). `HttpClientConfig::extraCaFile` is a path to a PEM file whose certificates are trusted in addition to the default store, which `SSL_CERT_FILE` can only replace. It is for a private CA or a proxy that re-signs TLS. @@ -38,6 +46,13 @@ It is for a private CA or a proxy that re-signs TLS. * `proxy_tunnel` takes the path as a trailing default argument, and `TlsSocket` gains `set_extra_ca_file`. Nothing changes while the field is empty. +### CI + +* The openkal job on Windows asks Git for Windows where its CA bundle is; 2.56 + moved it from `mingw64/` to `ucrt64/`, and the job failed before it ran. +* The Windows job runs `test_handshake_timeout` and `test_extra_ca` with the + other hermetic tests. + ## 0.3.4 A request in flight can be abandoned from another thread. Everything is added diff --git a/README.md b/README.md index 94aba0c..6ee907d 100644 --- a/README.md +++ b/README.md @@ -214,14 +214,14 @@ Windows). Locally, ### 添加依赖 ```bash -mcpp add tinyhttps@0.3.4 +mcpp add tinyhttps@0.3.5 ``` 或在 `mcpp.toml` 中手动添加： ```toml [dependencies] -tinyhttps = "0.3.4" +tinyhttps = "0.3.5" ``` ### 构建 diff --git a/mcpp.toml b/mcpp.toml index ca4e023..911b9b3 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "tinyhttps" -version = "0.3.4" +version = "0.3.5" description = "Minimal C++23 HTTP/HTTPS client with SSE streaming support" license = "Apache-2.0" repo = "https://github.com/mcpplibs/tinyhttps" From a312f639929018beb58a4997b884f8e93822b6e7 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 10 Oct 2026 03:33:42 +0900 Subject: [PATCH 15/15] exec-plan-0.3.5: record the release, the index entry and the end-to-end check --- .agents/docs/exec-plan-0.3.5.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.agents/docs/exec-plan-0.3.5.md b/.agents/docs/exec-plan-0.3.5.md index 20080fd..e903164 100644 --- a/.agents/docs/exec-plan-0.3.5.md +++ b/.agents/docs/exec-plan-0.3.5.md @@ -164,6 +164,8 @@ T0 计划（本文）← 你在这里审 - [x] H1 能复现，修复后用例通过 🔁；已推送到 #25 - [x] #25 CI 全绿（run 37968516327，10/10），包括 Windows openkal 和 Windows hermetic（`test_handshake_timeout`） - [x] #25 squash 合入（`9d4315e`）；#24 同步 master（合并时 CHANGELOG 和 Windows 测试列表都有冲突）后 CI 10/10 全绿，squash 合入（`a0f7749`） -- [ ] 版本号按 D1 确定；tag、GitHub release -- [ ] gitcode tarball 的 sha256 一致 -- [ ] mcpp-index 合入；`"^0.3.1"` 在两个源下都解析到 0.3.5 +- [x] 版本号 0.3.5（D1）；发版 commit `6153cdd` 的 master CI 10/10 全绿；tag `0.3.5`，GitHub release +- [x] gitcode `mcpp-res/tinyhttps` 0.3.5 的 tarball 和 GitHub archive 的 sha256 一致（`6f75f9e6…58a3`） +- [x] mcpp-index 合入（mcpplibs/mcpp-index#517，`bf7e87c`）：PR 的全部检查通过，包括 openkal measure 和 linux/macos/windows workspace；合入后 Publish Index Artifact、validate、deploy-site 也都通过 +- [x] 真实验证：用已发布的 index 新建项目，依赖写 `"^0.3.1"`，在 GLOBAL 和 CN 两个源下（都清了缓存）都解析到 0.3.5。真实 HTTPS 请求返回 200；握手不回应的本地对端在 `connectTimeoutMs=500` 下约 520 ms 报 `TLS handshake timed out`；`extraCaFile` 文件不存在时，报错里带路径；指向系统 bundle 时请求正常返回 200 +- 注：mcpp 在一个源失败时会自动换另一个源，所以“切到某个源跑通”证明不了用的是哪个源。两个 URL 内容一致，是直接下载后比对 sha256 得出的