Use uvloop as base event loop by rlippmann · Pull Request #93173 · home-assistant/core

rlippmann · 2023-05-16T17:01:26Z

Breaking change

uvloop is a faster implementation of the standard asyncio loop, this change implements HA's main event loop using uvloop as opposed to the standard asyncio loop

Type of change

Dependency upgrade
Bugfix (non-breaking change which fixes an issue)
New integration (thank you!)
New feature (which adds functionality to an existing integration)
Deprecation (breaking change to happen in the future)
Breaking change (fix/feature causing existing functionality to break)
Code quality improvements to existing code or addition of tests

Additional information

See this link for benchmarks before and after uvloop implementation.

This PR fixes or closes issue: fixes #
This PR is related to issue:
Link to documentation pull request:
Architecture discussion: Use uvloop as main HA loop architecture#904

Checklist

The code change is tested and works locally.
Local tests pass. Your PR cannot be merged unless tests pass
There is no commented out code in this PR.
I have followed the development checklist
I have followed the perfect PR recommendations
The code has been formatted using Black (black --fast homeassistant tests)
Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

The manifest file has all fields filled out correctly.
Updated and included derived files by running: python3 -m script.hassfest.
New or updated dependencies have been added to requirements_all.txt.
Updated by running python3 -m script.gen_requirements_all.
For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

I have reviewed two other open pull requests in this repository.

frenck · 2023-05-16T17:40:39Z

Home Assistant had uvloop available in the past, and it was removed in 2019 (Home Assistant 0.104).

Considering the part this touches, this definitely needs an architectural discussion first.

Please make sure to create an architectural proposal and discussion in our architecture repository first.

../Frenck

rlippmann · 2023-05-16T18:03:41Z

Home Assistant had uvloop available in the past, and it was removed in 2019 (Home Assistant 0.104).

Considering the part this touches, this definitely needs an architectural discussion first.

Please make sure to create an architectural proposal and discussion in our architecture repository first.

../Frenck

Yeah, I had a feeling something like this was required....

Done.

Proposal link

bdraco · 2023-05-16T18:21:46Z

2023-05-16 13:20:57.229 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "uvloop/handles/udp.pyx", line 212, in uvloop.loop.UDPTransport._send
  File "uvloop/dns.pyx", line 119, in uvloop.loop.__convert_pyaddr_to_sockaddr
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bdraco/home-assistant/homeassistant/components/unifiprotect/discovery.py", line 33, in _async_discovery
    async_trigger_discovery(hass, await async_discover_devices())
  File "/Users/bdraco/home-assistant/homeassistant/components/unifiprotect/discovery.py", line 53, in async_discover_devices
    devices = await scanner.async_scan()
  File "/Users/bdraco/home-assistant/venv/lib/python3.10/site-packages/unifi_discovery/__init__.py", line 504, in async_scan
    await self._async_run_scan(
  File "/Users/bdraco/home-assistant/venv/lib/python3.10/site-packages/unifi_discovery/__init__.py", line 382, in _async_run_scan
    transport.sendto(UBNT_REQUEST_PAYLOAD, destination)
  File "uvloop/handles/udp.pyx", line 305, in uvloop.loop.UDPTransport.sendto
  File "uvloop/handles/udp.pyx", line 217, in uvloop.loop.UDPTransport._send
ValueError: ('<broadcast>', 10001): socket family mismatch or a DNS lookup is required

Found one incompatibility at least right away (easy to fix)

bdraco · 2023-05-16T18:31:10Z

The following packages should need to be updated to support uvloop or uvloop updated to support <broadcast>

~~I believe its deprecated though so updating the packages might make more sense.~~

./lib/python3.10/site-packages/jellyfin_apiclient_python/connection_manager.py:        MULTI_GROUP = ("<broadcast>", 7359)
./lib/python3.10/site-packages/jaraco/net/wake.py:    s.sendto(magic_packet, ('<broadcast>', 9))
./lib/python3.10/site-packages/elkm1_lib/discovery.py:    BROADCAST_ADDRESS = "<broadcast>"
./lib/python3.10/site-packages/pycomfoconnect/bridge.py:            udpsocket.sendto(b"\x0a\x00", ('<broadcast>', Bridge.PORT))
./lib/python3.10/site-packages/miio/miioprotocol.py:            addr = "<broadcast>"
./lib/python3.10/site-packages/discovery30303/__init__.py:    BROADCAST_ADDRESS = "<broadcast>"
./lib/python3.10/site-packages/nmb/NetBIOSProtocol.py:        # We don't use the transport.write method directly as it keeps raising DeprecationWarning for ip='<broadcast>'
./lib/python3.10/site-packages/nmb/NetBIOSProtocol.py:            ip = '<broadcast>'
./lib/python3.10/site-packages/nmb/NetBIOS.py:            ip = '<broadcast>'
./lib/python3.10/site-packages/flux_led/scanner.py:    BROADCAST_ADDRESS = "<broadcast>"
./lib/python3.10/site-packages/aiosenseme/discovery.py:            self.transport.sendto(data, ("<broadcast>", PORT))
./lib/python3.10/site-packages/pyric/pyw.py:     set nic's ip4 netmask (ifconfig <card.dev> broadcast <broadcast>
./lib/python3.10/site-packages/unifi_discovery/__init__.py:            address = "<broadcast>"
./lib/python3.10/site-packages/roonapi/discovery.py:            sock.sendto(msg, ("<broadcast>", SOOD_PORT))
./lib/python3.10/site-packages/pybravia/client.py:            sock.sendto(packet, ("<broadcast>", 9))
./lib/python3.10/site-packages/roombapy/discovery.py:    udp_address = "<broadcast>"

bdraco · 2023-05-16T18:41:35Z

The performance improvements for sockets that read frequently (ie bluetooth, zeroconf, etc) is quite stunning so it would be nice if we could get this working but the <broadcast> issue is definitely a blocker here.

rlippmann · 2023-05-16T19:11:35Z

The performance improvements for sockets that read frequently (ie bluetooth, zeroconf, etc) is quite stunning so it would be nice if we could get this working but the <broadcast> issue is definitely a blocker here.

Override sendto in HassEventLoop? It’s been many years since I’ve done C socket programming…

Would multicast also be affected?

rlippmann · 2023-05-16T22:44:22Z

@bdraco does changing a “” to “255.255.255.255” fix things?

bdraco · 2023-05-17T00:12:09Z

The string is actually <broadcast>. I think you can replace it with 255.255.255.255

MartinHjelmare · 2023-05-17T00:15:38Z

+        self._handle_cancellable_timer(timer, *args)
+        return timer
+
+    def create_future(self) -> asyncio.Future[Any]:


There are quite many places in the project where we call asyncio.Future directly instead of using loop.create_future which is recommended to cater for custom event loops. This may need to be solved in this PR, or a separate prerequisite PR, too.

I thought set_event_loop_policy() takes care of that?

The only reason I overrode the method was to periodically prune the HassJob cancellable timers which I need to manually keep track of.

I figured that was the best place to do it since it won’t effect the event loop too much and HA creates lots of tasks.

Why do you need to prune the timers?

Edit: you need to access loop._scheduled

I don't see that there is a way to access loop._timers in uvloop.

It would be nicer if there was a public method to do that as it would avoid the complexity here. I don't know how willing they would be to add something like that

bdraco/uvloop@5f8e84e

If you could get something like that merged to uvloop you could avoid the subclassing

Also I really don’t like that we need to access _scheduled in our current code base so if there is another option to refactor HA to avoid it that would also be a welcome change (if I had a suggestion to do that I think we already would have )

There is a private attribute that acts similarly to _scheduled in the uvloop code. I just don’t know how to access it from Python.

Why do you need to prune the timers?

Edit: you need to access loop._scheduled

Yeah, if I knew how to do that, all of this method overriding would be unnecessary. I’m working around the fact I can’t access it by just keeping a list of cancellable timers.

I don’t know how to access an attribute coded in C in Python.

Also I really don’t like that we need to access _scheduled in our current code base so if there is another option to refactor HA to avoid it that would also be a welcome change (if I had a suggestion to do that I think we already would have )

Add it to the Hass object?

bdraco/uvloop@5f8e84e

If you could get something like that merged to uvloop you could avoid the subclassing

Well, I originally tried to access _timers directly, but I couldn't figure out how to as it is in C.

rlippmann · 2023-05-17T10:31:40Z

Ok, as far as I see it, the issues are:

broadcast. I think I can just override sendto and test if address == "<broadcast>" and change the address to 255.255.255.255 and optionally do a setsockopt()
overriding create_task/future/etc. I think I can get around this by making loop._scheduled a WeakSet of TimerHandles. These will eliminate the need for pruning expired/cancelled timers
possibly moving the list of the cancellable timers from HassEventLoop._schedued to the hass object to be called on shutdown. These can also be a WeakSet to keep list of timers from constantly growing until shutdown.
instead of 2 and 3, somehow access uvloop's version of loop._scheduled. I'm not even sure this is possible.
abstract linux sockets -- I really don't know exactly what the issue is here. Perhaps someone can explain it to me in more detail?

Thoughts and/or suggestions and/or other issues?

rlippmann · 2023-05-17T21:45:23Z

I cleaned up the HassEventLoop._scheduled to just use a WeakSet and not bother to prune.

I'm at a loss as to what to do about broadcast, but I've logged an issue against uvloop.

MartinHjelmare · 2023-05-18T05:15:18Z

The branch is tainted with unrelated old commits. Please rebase and remove those or start over in a fresh branch and PR.

rlippmann · 2023-07-13T15:40:05Z

So, I’ve logged an issue against uvloop and broadcast, but haven’t even gotten a comment on it.

Also, I’ve noticed that uvloop drags in cython/gcc/etc instead of just downloading a wheel. I’m guessing we probably don’t want that either. Logged an issue, and still, silence.

bdraco · 2023-07-13T18:46:59Z

So, I’ve logged an issue against uvloop and broadcast, but haven’t even gotten a comment on it.

Also, I’ve noticed that uvloop drags in cython/gcc/etc instead of just downloading a wheel. I’m guessing we probably don’t want that either. Logged an issue, and still, silence.

We build our own wheels so this likely isn't a problem unless wheels fail to build manually

bdraco · 2023-07-13T20:29:14Z

selector overhead is at the top of profiles all the time so I started working through that in a series of cpython PRs python/cpython#106555

Hopefully we will get uvloop one day, but in the mean time there are still some wins that can be made on the cpython side

frenck · 2023-10-06T08:09:07Z

Because there hasn't been any activity on this PR for quite some time now, I've decided to close it for being stale.

Feel free to re-open this PR when you are ready to pick up work on it again 👍

../Frenck

rlippmann requested a review from a team as a code owner May 16, 2023 17:01

home-assistant Bot added cla-signed code-quality core labels May 16, 2023

frenck marked this pull request as draft May 16, 2023 17:40

bdraco added the breaking-change label May 16, 2023

MartinHjelmare reviewed May 17, 2023

View reviewed changes

rlippmann added 12 commits May 18, 2023 05:30

first pass at uvloop implementation

2f8f5ce

rework event loop

f75b03b

make sure not to add cancellable timers twice

2947d4f

override create_future to prune cancellable timers

5a1a13c

check len(args) > 0 in _handle_cancellable_timers

80a5b41

override create_task to call _prune_cancellable_timers

68faa30

add cancel_cancellable_timers

789ba91

add call_later

d4904df

remove paramspec from call_at/call_later

539f4de

add missing *

2e12a91

fixes to call_at/call_later

eb73b69

fix _cancellable_timers set size change error

21e3678

rlippmann and others added 9 commits May 18, 2023 05:30

fix _cancel_cancellable_timers

e541c02

fix loop initialization

e4632d2

fix prune

f7bb3ee

add uvloop to requirements.txt

7918b31

added uvloop to requirements.txt

28d67bf

change HassEventLoopPolicy.loop_name to uvloop

1732f32

add uvloop to requirements.txt

b3b8f8f

fix requirements

bea7ff6

make HassEventLoop._scheduled a WeakSet

75a98be

rlippmann force-pushed the uvloop branch from 2e3a04e to 75a98be Compare May 18, 2023 09:32

frenck closed this Oct 6, 2023

github-actions Bot locked and limited conversation to collaborators Oct 7, 2023

rlippmann deleted the uvloop branch January 13, 2026 09:36

Uh oh!

Conversation

rlippmann commented May 16, 2023 • edited by MartinHjelmare Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Breaking change

Type of change

Additional information

Checklist

Uh oh!

frenck commented May 16, 2023

Uh oh!

rlippmann commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bdraco commented May 16, 2023

Uh oh!

bdraco commented May 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bdraco commented May 16, 2023

Uh oh!

rlippmann commented May 16, 2023

Uh oh!

rlippmann commented May 16, 2023

Uh oh!

bdraco commented May 17, 2023

Uh oh!

MartinHjelmare May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlippmann May 17, 2023

Choose a reason for hiding this comment

Uh oh!

bdraco May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdraco May 17, 2023

Choose a reason for hiding this comment

Uh oh!

bdraco May 17, 2023

Choose a reason for hiding this comment

Uh oh!

bdraco May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlippmann May 17, 2023

Choose a reason for hiding this comment

Uh oh!

rlippmann May 17, 2023

Choose a reason for hiding this comment

Uh oh!

rlippmann May 17, 2023

Choose a reason for hiding this comment

Uh oh!

rlippmann May 17, 2023

Choose a reason for hiding this comment

Uh oh!

rlippmann commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rlippmann commented May 17, 2023

Uh oh!

MartinHjelmare commented May 18, 2023

Uh oh!

rlippmann commented Jul 13, 2023

Uh oh!

bdraco commented Jul 13, 2023

Uh oh!

bdraco commented Jul 13, 2023

Uh oh!

frenck commented Oct 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

rlippmann commented May 16, 2023 •

edited by MartinHjelmare

Loading

rlippmann commented May 16, 2023 •

edited

Loading

bdraco commented May 16, 2023 •

edited

Loading

MartinHjelmare May 17, 2023 •

edited

Loading

bdraco May 17, 2023 •

edited

Loading

bdraco May 17, 2023 •

edited

Loading

rlippmann commented May 17, 2023 •

edited

Loading