Default APS ack on no-reply frames causes false delivery failures and duplicate commands for sleepy end devices behind routers #1908

Found while investigating #1891. The full analysis, including how each radio stack times its APS acks, is in #1891 (comment).

By default, zigpy requests an APS ack for every frame that expects no ZCL response, and it retries a failed send twice, each time as a new frame. For sleepy end devices behind a router, the coordinator can report a missing APS ack while the frame is still queued at the device's parent. zigpy then sees a false failure, and its retries can make the device receive and act on the same command up to three times.

Details

• Which frames ask for an APS ack: ControllerApplication.request sets TransmitOptions.ACK when ask_for_ack isn't given and expect_reply is false (application.py#L1148-L1153). Cluster.reply() defaults to expect_reply=False (zcl/__init__.py#L871), so Default Responses, other ZCL responses, OTA responses and the unsolicited Image Notify all ask for one.
• Why the ack comes too late: the APS ack comes from the end device itself, after it has polled its parent for the frame. The parent holds the frame for about 7.68 s, but with Koenkk's Z-Stack firmware and with EmberZNet under bellows, the coordinator waits much less for the ack:
  • Koenkk's Z-Stack coordinator firmware waits 1 s for the ack after each of three transmissions (about 3 s in total; the value in effect comes from NV, see DataConfirm timeout (8 s) for routers can be shorter than Z-Stack's APS retry window zigpy-znp#283).
  • EmberZNet, as bellows configures it, waits 1.6 s after each of three transmissions (about 5 s), because bellows clears EmberZNet's extended timeout for acked sends.
  • If the device polls between that point and the end of the parent's hold, the failure is false. The coordinator's own sleepy children are mostly covered by the stacks themselves (apart from a bellows host timeout, see Message-sent timeouts (3 s / 8 s) are shorter than EmberZNet's APS retry windows bellows#764); the gap is devices behind a router.
• Retries become duplicates: Device.request retries a failed attempt twice, unless it failed with a parsing error (DEFAULT_REQUEST_RETRIES = 2, device.py#L68). Each retry is a new frame with a new APS counter, so the device's APS duplicate rejection can't drop it, while earlier copies may still be queued at the parent. The ZCL TSN stays the same, but the ZCL spec defines it for matching responses to requests and defines no duplicate rejection on it.
• extended_timeout covers more than sleepy devices: it's set for every end device, including ones that keep their radio on, and for devices without a node descriptor (device.py#L924-L927). For always-on end devices, this only slows down failure detection: zigpy's reply timeout goes from 5 s to 28 s, zigpy-znp's confirm wait from 8 s to 30 s, and bellows' send wait from 3 s to 8 s.

For comparison

zigbee-herdsman never requests an APS ack on Z-Stack: its dataRequest sends every unicast without the ack option. On EmberZNet, for ZCL frames, it requests one only when a specific or default response is expected. It sends the Image Notify as a single request (sendPolicy: "immediate"), without an APS ack, and then waits up to 60 s for the device's Query Next Image.

Possible changes

• Send the Image Notify (at least) without an APS ack request. With the fix proposed in OTA: notify() treats Image Notify delivery failure as fatal, blocking OTA for pure sleepy end devices #1891 (branch zigpy-bot/ota-notify-non-fatal, not merged yet), what matters is whether the device sends its Query Next Image, not the notify's delivery status. On bellows, the first attempt would then lose EmberZNet's own APS retries. Without an APS ack, zigpy only retries if the send itself raises an error (e.g. a MAC/NWK-level failure or a host-side timeout), and those retries force route discovery, which bellows sends with APS retries again (unless source routing is enabled).
• After an APS ack timeout to an end device, don't retry with a new frame, to avoid duplicate execution. The trade-off is losing a frame the parent really dropped. This needs the radio libraries to report an APS ack timeout in a way zigpy can tell apart: today, DeliveryError.status is the radio's raw status code, and it may not separate an ack timeout from MAC failures on every stack.
• Optionally, only set extended_timeout for end devices that don't keep their radio on (NodeDescriptor.is_receiver_on_when_idle).