Skip to content

LatencyService.wait_connected_instance: one caller's cancellation cancels the shared Future for all callers and poisons it #11

Open

Description

@TrApY

Summary

In LatencyService.wait_connected_instance (lib/clients/metaapi/latency_service.py, 29.1.1), every caller for the same account awaits the same asyncio.Future, stored in self._wait_connect_promises[account_id].

  • Shared cancellation. If any single caller is cancelled, for example an RPC wrapped in asyncio.wait_for(...) that times out, the cancellation propagates into the shared Future, and every other caller waiting for that account receives a CancelledError that is not its own.
  • The Future is poisoned. When the account connects, on_connected calls set_result(True) on the already cancelled Future. That raises InvalidStateError, which is logged and swallowed, and the Future is not removed from the dict. From then on, every wait_connected_instance call for that account that finds no active instance awaits the cancelled Future and fails immediately with CancelledError.

Since CancelledError is a BaseException, it slips through except Exception handlers in user code, so jobs that only expected an RPC failure are aborted instead.

Reproduction (no network)

import asyncio
from metaapi_cloud_sdk.clients.metaapi.latency_service import LatencyService

async def main():
    svc = LatencyService.__new__(LatencyService)
    svc._wait_connect_promises = {}
    svc.get_active_account_instances = lambda account_id: []

    a = asyncio.create_task(svc.wait_connected_instance("acc"))
    b = asyncio.create_task(svc.wait_connected_instance("acc"))
    await asyncio.sleep(0)
    a.cancel()                      # e.g. a caller's wait_for timed out
    await asyncio.gather(a, b, return_exceptions=True)
    print(b.cancelled())            # True: b was cancelled by a's timeout
    print(svc._wait_connect_promises["acc"].cancelled())  # True: poisoned for later callers

asyncio.run(main())

Suggested fix

  • In wait_connected_instance, replace a Future that is already done() with a new one, and have each caller await asyncio.shield(future). Cancelling one caller then only cancels its own wait.
  • In on_connected, only call set_result if the Future is not done(), and always remove it from the dict.

We are running this as a monkeypatch in production, and it removed the spurious CancelledErrors.

Related: #10 (tasks left running after MetaApi.close()).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions