Repository navigation
Pub / Sub Subscriber: CPU Usage eventually spikes to 100% #4600
Description
This is a distillation of this report from many other issues. Large thanks to @anorth2 and @dmontag for reporting and helping refine the issue.
Issues that have been partially resolved (except for the CPU spike) are being collapsed into this one:
- PubSub: Subscriber Client stops acking messages after several seconds #4274
- PubSub with Python 3.5 spawns threads that don't die, eventually bringing the process to consume all CPU available #4563
- Python PubSub consumer high cpu usage after approx 1 hour #3965
- (more to come?)
Core issue: there is a spinlock bug in gRPC (present in all recent versions 1.6.x, 1.7.x, 1.8.1). A "bandaid" fix exists but has not been merged (as of noon Pacific on December 15, 2017).
Update: The fix was rolled back, but will be rolled forward (grpc/grpc#13918).
Potential workaround: compile grpcio from source (e.g.) while including the bandaid fix or just use the 64-bit manylinux wheel that I already created (I may be open to creating Mac OS X wheels, not sure about Windows)
Activity
dhermes commentedon Dec 15, 2017
@anorth2 Would you mind trying with the patched
grpcio==1.7.4.dev1? (This is a follow-up to #4274 (comment))anorth2 commentedon Dec 15, 2017
Testing it out now, I'll edit this comment once I have it deployed.
@dhermes Having issues with the wheel, getting this:
grpcio-1.7.4.dev1-cp36-cp36m-manylinux1_x86_64.whl is not a supported wheel on this platform.on apython:3-alpinebase image. Any suggestions? (We don't do a lot of whl compiling here)dhermes commentedon Dec 15, 2017
@anorth2
pip install grpcio....whl)manylinuxwheels are not compatible withalpinedhermes commentedon Dec 20, 2017
Fixed by #4642.
arindamchoudhury commentedon Dec 28, 2017
Hi @dhermes ,
I am still having problem with high CPU usage. After around one hour CPU usage is 100%.
google-api-core (0.1.3)
google-auth (1.2.1)
google-cloud-pubsub (0.30.1)
googleapis-common-protos (1.5.3)
grpc-google-iam-v1 (0.11.4)
grpcio (1.8.2)
I am using
Python 3.6.4 (default, Dec 21 2017, 01:35:12)
[GCC 4.9.2] on linux
dhermes commentedon Jan 2, 2018
@arindamchoudhury Can you confirm that your Python shell is in the same environment where those packages are installed? Also, could you share some code so I might reproduce the CPU spike?
dhermes commentedon Jan 2, 2018
I just now have run this with
google-cloud-pubsub==0.30.1andgrpcio==1.8.2and see CPU spikes at 100%. Will investigate soon.arindamchoudhury commentedon Jan 2, 2018
Hi @dhermes ,
If I install
grpciousing:it works fine.
The publish also creates high cpu uses.
dhermes commentedon Jan 2, 2018
The
--no-binaryremark is very good to know! Thanks.Re-running `no-messages-too` with `--no-binary`.
12 remaining items
zburgermeiszter commentedon Jan 17, 2018
Is there a known old version that is not affected by this bug or alternatively a Dockerfile with a workaround?
mehrdada commentedon Jan 17, 2018
@explicitcall @dhermes Please note that this is a limitation enforced by PyPI: we simply have no way to use
epoll_create1becausemanylinux1glibc does not support it. This is not easily fixable by a gRPC release. I am trying to useepoll_createfollowed byfcntlto enable some level of epoll support (PR #14041)--we will have to wait and see if it works well because there are other limitations like eventfd that will remain. Note that even then the so called "source/binary" difference will exist, because you would be building on two separate platforms, one with much older glibc, so to get the "best" Python gRPC it's ideal to build from source.That said, it is my understanding that the CPU issue was resolved (or at least a resolution was tried) in 1.8.4.
dhermes commentedon Jan 17, 2018
@mehrdada The old
glibcshould be unrelated to the spinlock bug. If it's not, then that means some code paths still have a bug.As for the fix that works with
epoll, it was rolled back in grpc/grpc#13898 but then was rolled forward in grpc/grpc#13933. I'm not sure if that has been released for1.8.4.@jonparrott Is it worth discussing alternate hosting for Ubuntu wheels (and other Linux platforms)?
theacodes commentedon Jan 17, 2018
That experience is really bad, but it's worth us discussing some alternatives here, as the sentence
so to get the "best" Python gRPC it's ideal to build from source.is horrifying.mehrdada commentedon Jan 17, 2018
@dhermes Agreed. To be clear, I was addressing the source/binary distinction. My understanding is that a bug fix was attempted in 1.8.4 (including poll code path). Are you still encountering the same bug on 1.8.4? If yes, please ping the relevant issue on the gRPC tracker again.
mehrdada commentedon Jan 17, 2018
@jonparrott We are actively trying to improve the situation on the binary side (grpc/grpc#14041), but as far as PyPI packages are concerned, some of those limitations are imposed by PyPI. Perhaps the Python community should work on a
manylinux2specification?theacodes commentedon Jan 17, 2018
Moving this to an internal thread for now. :) Please check email?
tseaver commentedon Jan 18, 2018
Shared binaries are like the waltzing bear: it's not how well he waltzes, but that he waltzes at all. We shouldn't be surprised that the
manylinuxstrategy isn't perfect, as that was a known problem with it at its adoption.theacodes commentedon Jan 25, 2018
According to @mehrdada and team, this appears to be resolved by grpcio 1.8.4. Closing, but if it reappears we can ofc re-open.
mehrdada commentedon Jan 25, 2018
Yes, please file an issue on gRPC issue tracker (or reopen the one) with new repro if you encountered this again. FYI, since it is relevant to the previous discussions on this thread, we modified the epoll1 IO manager code path in core to rely on epoll_create followed by fcntl when compiled on manylinux1 (instead of epoll_create1), so by default our binary packages should have epoll support enabled by default from 1.9.0rc1, so that should be helpful too.
[change Pub/Sub Client Libraries to Pub/Sub API]