Skip to content

Feature Request: Add NNG (Nanomsg Next Gen) Broker Support #602

Open
Open
Feature Request: Add NNG (Nanomsg Next Gen) Broker Support#602
Assignees
s3rius
Labels
enhancementNew feature or request

Description

@alexted
alexted
opened on Mar 22, 2026

Is your feature request related to a problem?

Currently, taskiq provides support for a ZeroMQ broker. However, the existing ZMQ implementation comes with certain architectural limitations, specifically regarding seamless asynchronous integration, dynamic worker discovery, and the complexity of managing socket lifecycles when nodes randomly disconnect or scale.

For high-performance, brokerless distributed systems, developers often need a lighter, more modern alternative to handle messaging topologies without the boilerplate and fragility of raw ZMQ socket management inside asyncio loops.

Describe the solution you'd like

I would like to propose adding NNG (Nanomsg Next Generation) broker support to taskiq, potentially utilizing Python bindings like pynng.

NNG is a modern, lightweight rewrite of Nanomsg that addresses many of the scalability and state-handling issues found in older ZMQ architectures. A dedicated NNG broker would provide:

  • Clean, native asyncio integration out-of-the-box.
  • Decoupled dialers and listeners, meaning far more robust reconnection patterns when workers crash or restart.
  • A clean, lightweight environment for fast inter-process and network communications, ideal for modern microservice architectures.

Describe alternatives you've considered

  1. Using the existing ZeroMQ broker: This currently feels restrictive due to the boilerplate required to safely manage central proxy topologies and the lack of robust, automated reconnect handling in complex asynchronous contexts.
  2. Writing a custom broker from scratch: While possible, NNG is a standard, highly efficient protocol. Having official or community-supported integration in the taskiq ecosystem would greatly benefit developers building fast, lightweight asynchronous services.

Additional context

Adding an NNG backend (e.g., taskiq-nng) would be a huge win for developers looking for reliability and performance in their message queues without deploying heavy infrastructure like Redis, Kafka or NATS, keeping the technology stack lean and efficient.

Which component would this affect?

Broker

Activity

  1. alexted
    assigned
    s3rius
    on Mar 22, 2026
  2. alexted
    added
    enhancementNew feature or request
    on Mar 22, 2026

  3. s3rius commented on Apr 11, 2026

    @s3rius
    s3rius
    on Apr 11, 2026
    Member

    Hello, @alexted and thanks for your interest in the project. I actually appreciate that you showed me this broker. I've never heard of it, and after trying I should admit, that I enjoyed how easy it was to integrate and how cool the technology is.

    Thank you for highlighting it. However, it seems like you have much more experience with this technology than I do. Can you please take a few minutes of your time to review my pull request that I made at #607. I'd be very glad if you could. In case if you cannot, I'll just merge it after few days.

  4. alexted commented on Apr 19, 2026

    @alexted
    alexted
    on Apr 19, 2026
    Author

    Thank you for being so open to this! I'm glad you liked the technology. I’ve left a few comments on the PR.
    Thanks for all your work!

  5. alexted commented on Apr 26, 2026

    @alexted
    alexted
    on Apr 26, 2026
    Author

    Here is a more detailed architectural overview to better illustrate the design and execution flow.
    See #612.

    Architecture Overview

                          ┌─────────────────────────────────────────────┐
                      │                Taskiq app                   │
                      │  producer / scheduler / result consumer     │
                      └───────────────────────┬─────────────────────┘
                                              │
                                              │ 1) submit task
                                              ▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│                                         NNG HUB                                              │
│                                                                                              │
│   ┌─────────────────────────────── Control Plane ─────────────────────────────────────────┐  │
│   │                                                                                       │  │
│   │   register(worker)  heartbeat(worker)  drain(worker)  unregister(worker)              │  │
│   │          │                │                │                 │                        │  │
│   │          ├────────────────┴────────────────┴─────────────────┤                        │  │
│   │          ▼                                                   ▼                        │  │
│   │   worker registry + lease table + inflight counters + routing metadata                │  │
│   │                                                                                       │  │
│   │   routing policy: least_loaded / p2c / round_robin / affinity                         │  │
│   │   lease management: lease_id, ack validation, timeout, requeue                        │  │
│   │   backpressure: queue limits, admission control                                       │  │
│   └───────────────────────────────────────────────────────────────────────────────────────┘  │
│                                                                                              │
│   ┌─────────────────────────────── Data Plane ────────────────────────────────────────────┐  │
│   │                                                                                       │  │
│   │   task ingress ──► enqueue ──► dispatch ──► per-worker delivery sockets ──► task data │  │
│   │                                        │                                              │  │
│   │                                        └──► ack/nack path back to control plane       │  │
│   └───────────────────────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
            │                               │                                  │
            │ 2) leased task                │ 2) leased task                   │ 2) leased task
            ▼                               ▼                                  ▼
     ┌──────────────┐                ┌──────────────┐                   ┌──────────────┐
     │   Worker A   │                │   Worker B   │                   │   Worker C   │
     │ pull / recv  │                │ pull / recv  │                   │ pull / recv  │
     │ process task │                │ process task │                   │ process task │
     │ ack / nack   │                │ ack / nack   │                   │ ack / nack   │
     │ heartbeat    │                │ heartbeat    │                   │ heartbeat    │
     └──────────────┘                └──────────────┘                   └──────────────┘

    Execution Flow (Single Task)

    Taskiq app -> HUB: submit(task)
HUB -> routing policy: choose worker
HUB -> worker: deliver(task, lease_id)

Worker -> HUB: ack(task_id, lease_id)   [success]
        OR
Worker -> HUB: nack(task_id, lease_id)  [exception]

HUB (timeout): lease expired -> requeue(task)
HUB (no heartbeat): worker marked dead -> requeue(task)

    Key Design Principles

    • Separation of concerns: control plane (coordination, lifecycle, routing) vs data plane (task delivery).
    • Explicit worker lifecycle: registration, heartbeat-based liveness, draining, and clean shutdown.
    • Lease-based delivery model: enables at-least-once semantics and protects against late/duplicate acknowledgements.
    • Centralized scheduling: routing decisions are made by the hub (not delegated to socket patterns).
    • Failure-aware execution: timeouts, worker death, and negative acknowledgements all lead to safe requeue.
    • Backpressure handling: queue limits enforce admission control under load.

    This should give a clearer mental model of how the system operates beyond just the code-level changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions