High CPU load when bpython is in use

Experiencing a high python process CPU load despite not running any python script.
The issue persists as long as bpython is in use, quitting bpython resolves the high CPU load.
Running python or ipython does not cause the same issue.

MacOS Catalina 10.15.4
python 3.7.7: /Users/username/opt/anaconda3/bin/python
bpython 0.19

Apologies in advance if this issue has been addressed or the solution is trivial.
Thanks!

Hm do you have a particularly large number libraries installed? bpython walks through installed packages at startup in order to provide import completion, and it could be that it's taking a long time to do this properly, or it could be that there's an issue where this is taking longer for some other reason.

Thanks for the follow-up!
I've created a new environment with a lesser number of packages (23 packages) to run bpython on and left it idle for about 10 minutes.
CPU load averages at about 50% with occasional spikes to >100%.
Running on zsh on iTerm2 but similar results using Terminal as well.
The following is the result:

Experiencing a similar problem on Fedora 32, package installed through DNF.

If I start it from a near-blank venv, the CPU usage is normalized, but running bpython3 with system-installed libs produces the same symptoms (created a brand-new user for this to make sure only the system Python libs were getting pulled in, no user libs).

Helpful to hear about the difference between venv and system-installed libs. My guess is still the import discovery code, I'd love to know what all those system calls are.

I've looked into the systems calls that bpython is making, and found something quite curious...

write(1, "\33[36m>>> \33[39m", 14>>> )      = 14
write(1, "\33[K", 3)                    = 3
write(1, "\0337\33[1000001;1H\33D", 16
) = 16
)            = 93[58;1H", 9
write(1, "\33[36mWelcome to bpython! Press <"..., 250Welcome to bpython! Press <F1> for help.                                                                                                                                                       )                = 7                             ) = 250
write(1, "\33[?12l\33[?25h", 12)        = 12
stat(".", {st_mode=S_IFDIR|0710, st_size=8192, ...}) = 0
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 7
fstat(7, {st_mode=S_IFDIR|0710, st_size=8192, ...}) = 0
getdents64(7, /* 145 entries */, 32768) = 5144
getdents64(7, /* 0 entries */, 32768)   = 0
close(7)                                = 0
stat(".", {st_mode=S_IFDIR|0710, st_size=8192, ...}) = 0
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 7
fstat(7, {st_mode=S_IFDIR|0710, st_size=8192, ...}) = 0
getdents64(7, /* 145 entries */, 32768) = 5144
getdents64(7, /* 0 entries */, 32768)   = 0
close(7)                                = 0
stat("./Desktop", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
stat("./Desktop", {st_mode=S_IFDIR|0755, st_size=48, ...}) = 0
...

For some reason, bpython is looking through my entire home folder recursively, and then gets into an infinite loop due to a symlink pointing to the root of my file system...

If I start bpython from a folder that's not my home directory (or any folder where it won't end up hitting that symlink for that matter), the CPU usage problem goes away. Though the question still stands, what's bpython doing recursing through the cwd and all it's subdirectories?

@thomasballinger

It indeed has to be related to import discovery though I don't think it should do that before even trying an import @thomasballinger ?

I think we do, in idle() in cli.py. What a good bug! I heard about this kind of thing when I was at Dropbox, I think we're hitting the simplest possible version: it's a symlink, we should probably just not follow symlinks?

@vista-

@vista- what bpython is doing worming its way through cwd and all subdirectories (hopefully not all) is looking for python modules that could be imported. If it's checking places that aren't on the import path or couldn't possibly be imported, then that's a bug. It should only be necessary to check a subdirectories of a directory if the directory has a __init__.py file - does your home directly happen to have one of those?

Not following symlinks would be a pretty easy thing to do (we could theoretically make it a setting if some people want to) or we could see if we cycle in the paths and stop there?

@thomasballinger

@thomasballinger does your home directly happen to have one of those?

Unfortunately, I'm a python beginner and am not too sure how I can validate whether bpython is checking places on the import path.

As I understand, I have __init__.pyi file at the following path:
/Users/<username>/opt/anaconda3/lib/python3.7/site-packages/yarl/__init__.pyi

Maybe @vista- may provide better information on this.

Not following symlinks would be a pretty easy thing to do (we could theoretically make it a setting if some people want to) or we could see if we cycle in the paths and stop there?

We might need to handle mount points too. But handling symlinks as a start would definitely help. I'll see if I can come up with something.

I found that if I run bpython in a blank directory, it does not cause high cpu, and if I run bpython in my HOME dir, it causes high cpu. In Macbook, I use tool dtruss to track the bpython pid when cpu is 99%, I found bpython is scanning my home dir which has many files.

In my case, I think that some symlinks in node_modules cause a infinite loop, because the high cpu happend only if node_modules is in the subdirectories of current dir that I run bpython.

log:
sigaction(0x2, 0x7FFEE6C7DF18, 0x7FFEE6C7DF50) = 0 0
select(0x6, 0x7FFEE6C71D00, 0x7FFEE6C71D80, 0x7FFEE6C71E00, 0x7FFEE6C71E80) = 0 0
sigaction(0x2, 0x7FFEE6C7DF18, 0x7FFEE6C7DF50) = 0 0
stat64("./codes/infra_2g/webapp/node_modules/escope/node_modules/es6-weak-map/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_\0", 0x7FFEE6C76AE0, 0x0) = 0 0
stat64("./codes/infra_2g/webapp/node_modules/escope/node_modules/es6-weak-map/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_\0", 0x7FFEE6C766C0, 0x0) = 0 0
stat64("./codes/infra_2g/webapp/node_modules/escope/node_modules/es6-weak-map/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_\0", 0x7FFEE6C76AE0, 0x0) = 0 0
stat64("./codes/infra_2g/webapp/node_modules/escope/node_modules/es6-weak-map/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_\0", 0x7FFEE6C766C0, 0x0) = 0 0
stat64("./codes/infra_2g/webapp/node_modules/escope/node_modules/es6-weak-map/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_\0", 0x7FFEE6C76C20, 0x0) = 0 0
open_nocancel("./codes/infra_2g/webapp/node_modules/escope/node_modules/es6-weak-map/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_\0", 0x1100004, 0x0) = 7 node_modules/es5-ext/node_modules/es6-symbol/node_modules/es5-ext/node_0
fstatfs64(0x7, 0x7FFEE6C765E8, 0x0) = 0 0
getdirentries64(0x7, 0x7F926AAB5000, 0x2000) = 128 0
close_nocancel(0x7) = 0 0

@pyang30

@pyang30 interesting! Based on those repeated filenames I'm guessing there are symlinks in there, so this the fix being worked on should address this. I'm surprised that the search goes into these in the first place though, I wonder why they look like Python packages. Are there init.py files in these folders?

@thomasballinger

@thomasballinger no init.py or init.py file in these folders

Thanks, I must not quite understand how this search works.

Prevent symbolic link loops in import completion

Navigation Menu

Repository navigation

High CPU load when bpython is in use #806

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

High CPU load when bpython is in use #806

Description

Activity

thomasballinger commented on Apr 19, 2020

BryanJian commented on Apr 19, 2020

vista- commented on May 25, 2020

thomasballinger commented on May 25, 2020

vista- commented on May 25, 2020

supakeen commented on May 25, 2020

thomasballinger commented on May 25, 2020

thomasballinger commented on May 25, 2020

supakeen commented on May 26, 2020

BryanJian commented on Jun 22, 2020

sebastinas commented on Jun 22, 2020

pyang30 commented on Jun 28, 2020

thomasballinger commented on Jul 1, 2020

pyang30 commented on Jul 1, 2020

thomasballinger commented on Jul 1, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions