Extend post-OTA retry budget for slow-rebooting devices - #1818

Proposed change

After Device.update_firmware() finishes a successful OTA, zigpy probes the device with ota.read_attributes([current_file_version]) before running image_notify and reinterview(). The probe is cheap and serves as a liveness check — if it fails, we skip the heavier shadow-discovery in reinterview().

The probe is wrapped in OTA_RETRY_DECORATOR (zigpy/device.py:57-60). The previous setting was tries=4, delay=10 — a ~40s window. That's too short for slow-rebooting devices on marginal meshes: when the probe exhausts, the DeliveryError propagates out of update_firmware() and skips both image_notify and reinterview(). The device recovers on its own via the legacy _initialize() path on rejoin, but ZHA never re-emits DeviceFullInitEvent, quirks aren't re-applied, and entities aren't rebuilt.

This PR bumps the retry budget so the probe survives realistic reboot windows:

AFTER_OTA_ATTR_READ_DELAY = 10
OTA_RETRY_DECORATOR = zigpy.util.retryable_request(
    tries=10, delay=AFTER_OTA_ATTR_READ_DELAY
)

~10× ~10s ≈ 100s of pure sleep + per-call radio time. Comfortably covers the observed 102s case with headroom.

Real-world evidence

Hue Go LLC020 (router, 0x7F36) — failure case with old tries=4:

04:02:21.133  OTA upgrade progress 100.0000%
04:02:50.971  read_attributes attempt 1 (extended_timeout=False)
04:03:04.018  read_attributes attempt 2 — DELIVERY_FAILED
04:03:21.764  read_attributes attempt 3 — DELIVERY_FAILED
              decorator exhausted, exception propagates
04:03:41.671  legacy _initialize() kicks in (tsn=84 on Basic cluster)
04:04:03.311  Device_annce arrives (+102s after 100%)

The probe gave up ~21s before the device finished rebooting. image_notify and reinterview() were skipped.

Frient WISZB-131 (end device, 0xAE74) — happy path on new mechanism:

02:45:56.490  OTA upgrade progress 100.0000%
02:45:56.601  ota_query_cache_cleared
02:46:06.603  Tries remaining: 4 → first read_attempt (extended_timeout=True)
02:46:11.217  Device_annce arrives (+14.7s, fast reboot)
02:46:47.673  Rebuilding device 0xAE74 after reinterview

End-device OTA does happen and works fine on fast reboots — but slow reboots aren't router-only.

Per-device-type budget analysis

Bellows uses MESSAGE_SEND_TIMEOUT_MAINS = 3 and MESSAGE_SEND_TIMEOUT_BATTERY = 8 per send_packet. application.request retries internally up to 3× per call. Worst case per failed read_attributes:

End-device "wall time on a dead device" is roughly 2× a router's, not the 6-7× I initially estimated — bellows uses tighter timeouts than I assumed. Other radio drivers (znp, deconz, zboss) may differ but should be in the same ballpark.

Probe-density and end devices

End-device probes are sent with extended_timeout=True, which routes them through the parent's indirect transmission buffer. The parent holds the message until the device polls (~8s window per bellows' MESSAGE_SEND_TIMEOUT_BATTERY). So briefly-awake end devices are caught at the radio layer, not by probe density at our level. Tightening delay below the indirect-transmission window doesn't improve coverage. This is why the retry delay is kept at 10s rather than something larger — there's no end-device-specific reason to spread probes further apart.

Alternatives considered

• Wrap step 4 (read_attributes) in try/except and always run reinterview(). Considered, but reinterview()'s shadow _discover() uses retryable_request(tries=2, delay=0.5) — under a second per ZDO request. It won't bridge a 100s reboot gap either; it'll just fail and emit device_reinterview_failure. The read_attributes probe serves as a cheap liveness check before committing to ~5-10 ZDO requests of shadow discovery. Keeping that role and just giving it a realistic budget is the minimal fix.
• Switch to a Device_annce-based readiness signal. Larger structural change, beyond the scope of a retry-budget fix.
• Hard asyncio.timeout() cap around step 4. Would bound end-device wall time, but adds a structural layer this PR avoids. Worth revisiting if 5-6 min waits on truly-dead end devices become a real complaint.
• Decouple AFTER_OTA_ATTR_READ_DELAY from the retry delay. Tried this in an earlier draft (initial-pause vs. inter-retry-backoff are conceptually different). Reverted because (a) the only consumer of "retry delay" is this decorator, so a second named constant is dead weight, and (b) keeping both at 10s gives one number to reason about with no measurable downside.

Risk / behavior change

• Wall time on a successful OTA where the device returns within the old window: unchanged.
• Wall time on a successful OTA where the device returns between 40-180s: previously skipped image_notify/reinterview() and required a rejoin to recover (~no real wall savings, just silently broken state); now waits up to ~3 min and runs the full post-OTA flow.
• Wall time on a truly dead OTA target: ~3 min (router) / ~5.5 min (end device) before update_firmware() raises. This is the only regression — and it's failure-only.
• No API change. No new dependencies.