This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

First of all, thanks for showing interest in this plugin. I am evaluating Vim and also looking at this plugin - fast & automatic refactoring is very important.

I have a question: can a Vim plugin not use traditional Python parallelization, like suprocesses (since I suspect this is CPU-bound, so threading would be subject to the Python GIL)?

It seems even the NeoVim docs says vim.async_call will block for long-running computation.

Pardon my ignorance; I have no experience with Vim plugins, but I'd love to hear your thoughts.

From my experience with refactoring in a fairly large Python project, python-mode and rope is always fast enough once the project is indexed. I never had a refactoring operation takes more than 2-3 seconds to plan and complete a project-wide refactoring, and autocompletion is pretty much always instant.

rope has two kind of indexing operations, the initial indexing can take a while but subsequent reindex on the same vim session is generally quite fast. The problem with the re-index operation is that currently they run on the foreground thread and would block you from doing anything until the indexing operation finishes. What this PR does is to put these indexing operations in a background thread so you can continue browsing and editing the file.

In smaller projects, the initial indexing time is generally quite negligible, but in larger projects it can takes a bit of time. For a sense of scale, scanning Django source code, which weighs at about 120kLOC (minus its own tests suite) of Python code takes about 4 seconds, and pandas which weighs at 400kLOC (with its test suite) of Python code is about 12 seconds (i7-6500U CPU @ 2.50GHz, m.2 SSD).

Generally, the initial indexing operation is only an issue if you start a new Vim instance for each edit you do (unlike most IDEs, Vim starts and stops instantly, so many Vim users are used to this style of editing), and python-mode has to do a full-scan of your project when you just started Vim, since a file might be changed outside of Vim without updating rope index. If you are working on a big project, you'd generally want to keep a long-running edit session. That way, the time needed for the initial indexing is usually not going to be a practical issue, as subsequent re-index usually takes less than half a second or so even in large projects.

can a Vim plugin not use traditional Python parallelization, like suprocesses (since I suspect this is CPU-bound, so threading would be subject to the Python GIL)?

I am not aware of anything that would prevent Python from using threading or multiprocessing in either NeoVim or Vim. However, AFAIK, rope is only single-threaded, so you'll have to re-engineer rope to somehow split up its code analysis logic and deal with merging the results. That needs to happen at rope level, so python-mode wouldn't be the place to have that discussion. That said, I don't think parallelizing rope is going to be really necessary, even for large projects such as those I mentioned above, rope's refactoring is already fast enough to usually not be the bottleneck when refactoring.

It seems even the NeoVim docs says vim.async_call will block for long-running computation.

That shouldn't really be an issue. async_call is used to run a callback in the main foreground thread, which should never need to do anything more than updating the UI. The bulk of the indexing work is done by the background thread.

since I suspect this is CPU-bound, so threading would be subject to the Python GIL?

The background thread in Python is a real OS thread, so the OS will pre-empt the background thread and schedule its with the frontend as necessary. GIL would only be an issue if you try to parallelize the code analysis itself, and that's a rope issue.

Wow, thank you for the awesome explanation! I had no idea the indexing is so fast; I am excited to try it out as soon as I get the chance! Hats off to you, sir!

Just be careful that you don't try to edit a Python file with your home directory (or root or other such large, non project directories) as your current working directory while python-mode is enabled. It will try to scan your entire home directory and that might lock up Vim for a very long while if you have lots of files at home.