Repository navigation
Connection process crashes on a queued body chunk for a canceled request #594
Description
ConnectionProcess buffers request body data it cannot send yet in request_stream_queue.
When a request's stream ends, the ref is removed from state.requests and from the Mint
conn struct, but any queued chunks are left behind. The next time the queue is drained,
Mint.HTTP2.get_window_size(conn, {:request, ref}) is called on the stale ref.
That raises, killing the connection process.
This can happen when a request with a body larger than the HTTP/2 send window finished due to a timeout, or was reset by the server.
Reproduction
Two unary calls against a peer that accepts the connection and then does not respond.
The first call leaves a chunk queued, then times out.
The second call drains the queue, killing the connection process.
Mix.install([
{:grpc, github: "elixir-grpc/grpc", sparse: "grpc"},
{:mint, "~> 1.9"}
])
defmodule Msg do
use Protobuf, syntax: :proto3
field(:text, 1, type: :string)
def new(text), do: struct(Msg, text: text)
end
defmodule Echo.Service do
use GRPC.Service, name: "echo.Echo"
rpc(:Send, Msg, Msg, %{})
end
defmodule Echo.Stub do
use GRPC.Stub, service: Echo.Service
end
{:ok, listen} = :gen_tcp.listen(0, [:binary, active: false, reuseaddr: true])
{:ok, port} = :inet.port(listen)
spawn_link(fn ->
{:ok, socket} = :gen_tcp.accept(listen)
:ok = :inet.setopts(socket, active: true)
Process.sleep(:infinity)
end)
{:ok, channel} = GRPC.Stub.connect("localhost:#{port}", adapter: GRPC.Client.Adapters.Mint)
IO.puts("SENDING FIRST MESSAGE")
{:error, _} = Echo.Stub.send(channel, Msg.new(:binary.copy("x", 100_000)), timeout: 200)
IO.puts("SENDING SECOND MESSAGE")
{:error, _} = Echo.Stub.send(channel, Msg.new("ping"), timeout: 200)Both calls return {:error, deadline exceeded} as expected, the crash is only visible
in the logs.
SENDING FIRST MESSAGE
SENDING SECOND MESSAGE
[error] GenServer #PID<0.215.0> terminating
** (ArgumentError) request with request reference #Reference<...> was not found
(mint 1.10.0) lib/mint/http2.ex:880: Mint.HTTP2.get_window_size/2
(grpc 1.0.5) connection_process.ex:410: ConnectionProcess.get_window_size/2
(grpc 1.0.5) connection_process.ex:223: ConnectionProcess.handle_continue/2
Last message: {:continue, :process_request_stream_queue}
[warning] gRPC connection localhost:54018 went down: {%ArgumentError{...}}
[warning] No healthy channels available for localhost:54018
Activity