This repository was archived by the owner on Sep 9, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Sep 9, 2026. It is now read-only.
refactor: count number of documents using hnswlib #1759
Description
Data storage in HnswDocumentIndex works in the following way:
- Vectors are stored on disk using
hnswlib. - All other types of data are saved in an SQLITE database.
One of the operations we frequently perform is determining the total number of documents (num_docs()). However, the only way to get number of documents from SQLITE is by scanning the entire table. Even though we've made efforts to reduce the number of times we use this functionality (#1729), it's still a time-consuming process.
For better performance, let's do the following: instead of scanning the SQLITE table, we can use hnswlib's get_current_count function to quickly get the number of documents in the index.
But there's a potential issue with this approach. What if documents don't have associated vectors? get_current_count would return 0.
We have two potential solutions:
- Notify/Warn users about this behavior and return 0.
- Use to the older method of counting using the SQL table if vector-less documents are detected.
Activity
shobhit9957 commentedon Aug 24, 2023
/attempt #1759
I would like to solve this issue. Can I get some more help.
JoanFM commentedon Aug 24, 2023
Hey @shobhit9957 ,
Sure, the main refactoring to be done is to change the code here:
to something that does not rely on the
_get_num_docs_sqlitebut from a new private method_get_num_docs_hnswshobhit9957 commentedon Aug 24, 2023
so I should replace the _get_num_docs_sqlite to _get_num_docs_hnsw is that correct?
I'm just a beginner, to this open-source-community. Just trying my hands out there...please help me out.
JoanFM commentedon Aug 24, 2023
Yes, this would be the right approach
shobhit9957 commentedon Aug 24, 2023
Thanks! will do the PR Soon.
shobhit9957 commentedon Aug 24, 2023
Hey Joan. Submitted the PR, please check if there are any other issues or mistakes I've done in the PR, I would be happy to solve my mistakes and submit the PR again Joan. Thanks!
shobhit9957 commentedon Aug 24, 2023
def num_docs(self) -> int:
"""
Get the number of documents.
"""
if self._num_docs == 0:
# Replace the old method with the new one
self._num_docs = self._get_num_docs_hnsw()
return self._num_docs
def _get_num_docs_hnsw(self) -> int:
"""
Get the number of documents using the HNSW method.
This method should return the count of documents using the new technique.
"""
# Implement your logic here to get the count using the HNSW method
# For example, you might access some data or perform calculations
# that help you quickly determine the number of documents.
# Then return the count.
return calculated_num_docs
is this correct?
munish0838 commentedon Oct 22, 2023
Hi, is this issue fixed or needs contribution yet.
Would like to contribute
JoanFM commentedon Oct 22, 2023
Hey @munish0838, there is still work to be done
munish0838 commentedon Oct 24, 2023
I would like to start working on this issue
JoanFM commentedon Oct 24, 2023
be my guest. I believe there are parts of the plan that were not applied yet.
Kulraj69 commentedon Oct 3, 2025
Hi Joan, Is this issue still open? I would like to contribute.