Skip to content

PYTHON-5671: Use fine-grained locks in the connection pool to reduce contention#2804

Open
sophiayangDB wants to merge 7 commits into
mongodb:masterfrom
sophiayangDB:PYTHON-5671
Open

PYTHON-5671: Use fine-grained locks in the connection pool to reduce contention#2804
sophiayangDB wants to merge 7 commits into
mongodb:masterfrom
sophiayangDB:PYTHON-5671

Conversation

@sophiayangDB
Copy link
Copy Markdown

@sophiayangDB sophiayangDB commented May 15, 2026

PYTHON-5671

Changes in this PR

Based on investigation in PYTHON-5007, using fine-grained locks improved thread efficiency by about 20% in a small read scenario. This PR seeks to implement the suggestions.

Previously, code in pool.py was protected by the sole lock self.lock and its two conditionals. Now, there are six individual locks that are the sole protectors of the following:

  • self._conns_lock
    Protects self.conns
  • self._active_contexts_lock
    Protects self.active_contexts as well as the subsequent updates to self.__pinned_sockets, self.active_sockets, next_connection_id, self.ncursors, and self.ntxns
  • self.size_cond
    Protects self.requests and self.gen
  • self._max_connecting_cond
    Protects self._pending
  • self._operation_count_lock
    Protects self.operation_count
  • self.lock
    Protects self.state

Other strategies used to mitigate lock contention include:

  • Moving lines that do not require a lock to be performed before lock acquisition
  • Use of temporary, local variables to avoid nested lock contention
  • Removing lock acquisition from update_is_writable() as its sole caller already has the lock
  • Copying self.active_contexts to update its references when additional additions/deletions to the set should not be operated upon or their absence does not modify behavior, avoids errors that arise if calling on the original and it's length/size changes

Test Plan

Tests already exist in main/test/test_pooling.py and test/asynchronous/test_pooling.py. Since this was only a code refactor, no additional tests were added.

Checklist

Checklist for Author

  • [yes] Did you update the changelog (if necessary)?
  • [yes] Is there test coverage?
  • [n/a] Is any followup work tracked in a JIRA ticket? If so, add link(s).

Checklist for Reviewer

  • Does the title of the PR reference a JIRA Ticket?
  • Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
  • Is all relevant documentation (README or docstring) updated?

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 15, 2026

Codecov Report

❌ Patch coverage is 89.75904% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.95%. Comparing base (552b7bf) to head (36738bd).

Files with missing lines Patch % Lines
pymongo/synchronous/pool.py 89.15% 5 Missing and 4 partials ⚠️
pymongo/asynchronous/pool.py 90.36% 4 Missing and 4 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2804   +/-   ##
=======================================
  Coverage   87.95%   87.95%           
=======================================
  Files         141      141           
  Lines       24407    24451   +44     
  Branches     4176     4182    +6     
=======================================
+ Hits        21466    21505   +39     
- Misses       2047     2051    +4     
- Partials      894      895    +1     
Flag Coverage Δ
auth-aws-rhel8-test-auth-aws-rapid-web-identity-python3.14-cov 35.00% <27.71%> (-0.01%) ⬇️
auth-aws-win64-test-auth-aws-rapid-web-identity-python3.14-cov 35.03% <27.71%> (+0.01%) ⬆️
auth-enterprise-macos-test-standard-auth-latest-python3.11-auth-ssl-sharded-cluster-cov 43.79% <52.40%> (+0.03%) ⬆️
auth-enterprise-rhel8-test-standard-auth-latest-python3.11-auth-ssl-sharded-cluster-cov 43.79% <52.40%> (+0.03%) ⬆️
auth-enterprise-win64-test-standard-auth-latest-python3.11-auth-ssl-sharded-cluster-cov 43.83% <52.40%> (+0.05%) ⬆️
auth-oidc-local-ubuntu-22-test-auth-oidc-default 48.65% <60.84%> (+0.03%) ⬆️
compression-snappy-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.36% <63.25%> (-0.01%) ⬇️
compression-snappy-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.52% <62.65%> (+<0.01%) ⬆️
compression-snappy-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.10% <62.65%> (+0.01%) ⬆️
compression-snappy-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 59.02% <62.65%> (+0.02%) ⬆️
compression-zlib-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.35% <63.25%> (+<0.01%) ⬆️
compression-zlib-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.52% <62.65%> (+0.01%) ⬆️
compression-zlib-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.10% <62.65%> (+<0.01%) ⬆️
compression-zlib-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 59.01% <62.65%> (-0.01%) ⬇️
compression-zstd-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.38% <64.45%> (+0.01%) ⬆️
compression-zstd-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.52% <62.65%> (+0.04%) ⬆️
compression-zstd-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.10% <62.65%> (+<0.01%) ⬆️
compression-zstd-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 59.03% <62.65%> (+0.01%) ⬆️
compression-zstd-ubuntu-22-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.99% <62.65%> (+0.01%) ⬆️
coverage-report-coverage-report 87.91% <89.75%> (+5.46%) ⬆️
disable-test-commands-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.36% <63.25%> (+0.01%) ⬆️
disable-test-commands-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.52% <62.65%> (+0.01%) ⬆️
disable-test-commands-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.11% <62.65%> (+0.01%) ⬆️
disable-test-commands-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 59.01% <62.65%> (+<0.01%) ⬆️
encryption-crypt_shared-macos-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 53.10% <61.44%> (+0.01%) ⬆️
encryption-crypt_shared-macos-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.82% <61.44%> (+0.01%) ⬆️
encryption-crypt_shared-macos-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.60% <63.25%> (+<0.01%) ⬆️
encryption-crypt_shared-rhel8-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 53.09% <61.44%> (+0.01%) ⬆️
encryption-crypt_shared-rhel8-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.67% <61.44%> (+<0.01%) ⬆️
encryption-crypt_shared-rhel8-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.60% <63.25%> (+0.02%) ⬆️
encryption-crypt_shared-win64-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 52.98% <61.44%> (+0.02%) ⬆️
encryption-crypt_shared-win64-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.79% <61.44%> (+0.08%) ⬆️
encryption-crypt_shared-win64-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.65% <63.25%> (-0.02%) ⬇️
encryption-macos-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 53.09% <61.44%> (+0.01%) ⬆️
encryption-macos-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.79% <61.44%> (+0.01%) ⬆️
encryption-macos-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.58% <63.25%> (+0.01%) ⬆️
encryption-pyopenssl-rhel8-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 53.77% <61.44%> (+0.01%) ⬆️
encryption-pyopenssl-rhel8-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 55.36% <61.44%> (+0.02%) ⬆️
encryption-pyopenssl-rhel8-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 55.30% <63.25%> (+0.03%) ⬆️
encryption-rhel8-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 53.10% <61.44%> (+0.02%) ⬆️
encryption-rhel8-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.67% <61.44%> (+0.01%) ⬆️
encryption-rhel8-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.60% <63.25%> (+0.01%) ⬆️
encryption-win64-test-non-standard-latest-python3.13-noauth-nossl-standalone-cov 52.98% <61.44%> (-0.01%) ⬇️
encryption-win64-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 54.72% <61.44%> (+0.02%) ⬆️
encryption-win64-test-non-standard-latest-python3.14t-noauth-ssl-replica-set-cov 54.71% <63.25%> (+0.10%) ⬆️
load-balancer-test-non-standard-latest-python3.14-auth-ssl-sharded-cluster-cov 48.43% <66.26%> (+0.02%) ⬆️
mongodb-latest-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 61.49% <63.25%> (+0.02%) ⬆️
mongodb-latest-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 59.37% <63.25%> (+0.01%) ⬆️
mongodb-latest-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 59.47% <39.75%> (-0.06%) ⬇️
mongodb-latest-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 59.56% <39.75%> (-0.05%) ⬇️
mongodb-latest-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 61.41% <63.25%> (+<0.01%) ⬆️
mongodb-rapid-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 61.48% <63.25%> (+<0.01%) ⬆️
mongodb-rapid-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 59.37% <64.45%> (+0.03%) ⬆️
mongodb-rapid-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 59.46% <39.75%> (-0.05%) ⬇️
mongodb-rapid-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 59.55% <39.75%> (-0.05%) ⬇️
mongodb-rapid-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 61.43% <64.45%> (+0.02%) ⬆️
mongodb-v4.2-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 57.19% <59.63%> (+0.01%) ⬆️
mongodb-v4.2-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 55.63% <59.63%> (+<0.01%) ⬆️
mongodb-v4.2-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.03% <38.55%> (-0.04%) ⬇️
mongodb-v4.2-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.13% <38.55%> (-0.04%) ⬇️
mongodb-v4.2-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 57.37% <59.63%> (+0.01%) ⬆️
mongodb-v4.4-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 59.58% <63.25%> (+0.01%) ⬆️
mongodb-v4.4-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 57.60% <62.04%> (+0.03%) ⬆️
mongodb-v4.4-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.54% <39.75%> (-0.04%) ⬇️
mongodb-v4.4-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.61% <39.75%> (-0.04%) ⬇️
mongodb-v4.4-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 59.37% <63.25%> (+<0.01%) ⬆️
mongodb-v5.0-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 59.77% <63.25%> (+0.01%) ⬆️
mongodb-v5.0-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 57.76% <63.25%> (+0.01%) ⬆️
mongodb-v5.0-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.73% <39.75%> (-0.04%) ⬇️
mongodb-v5.0-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.85% <39.75%> (-0.04%) ⬇️
mongodb-v5.0-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 59.62% <63.25%> (+0.02%) ⬆️
mongodb-v6.0-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 59.79% <63.25%> (+0.01%) ⬆️
mongodb-v6.0-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 57.77% <63.25%> (+0.02%) ⬆️
mongodb-v6.0-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.75% <39.75%> (-0.04%) ⬇️
mongodb-v6.0-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.90% <39.75%> (-0.04%) ⬇️
mongodb-v6.0-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 59.75% <63.25%> (-0.01%) ⬇️
mongodb-v7.0-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 59.82% <63.25%> (+0.01%) ⬆️
mongodb-v7.0-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 57.76% <63.25%> (+0.01%) ⬆️
mongodb-v7.0-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 57.80% <39.75%> (-0.04%) ⬇️
mongodb-v7.0-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 57.89% <39.75%> (-0.04%) ⬇️
mongodb-v7.0-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 59.75% <63.25%> (+0.02%) ⬆️
mongodb-v8.0-test-server-version-python3.10-async-auth-ssl-sharded-cluster-min-deps-cov 61.49% <63.25%> (+0.01%) ⬆️
mongodb-v8.0-test-server-version-python3.10-async-noauth-nossl-standalone-min-deps-cov 59.35% <63.25%> (+<0.01%) ⬆️
mongodb-v8.0-test-server-version-python3.10-sync-auth-ssl-sharded-cluster-min-deps-cov 59.46% <39.75%> (-0.05%) ⬇️
mongodb-v8.0-test-server-version-python3.10-sync-noauth-nossl-replica-set-min-deps-cov 59.55% <39.75%> (-0.05%) ⬇️
mongodb-v8.0-test-server-version-python3.11-async-noauth-nossl-replica-set-cov 61.44% <65.06%> (+0.02%) ⬆️
no-c-ext-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 60.58% <64.45%> (+0.04%) ⬆️
no-c-ext-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 62.73% <62.65%> (+0.05%) ⬆️
no-c-ext-rhel8-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 62.31% <62.65%> (+<0.01%) ⬆️
no-c-ext-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 60.21% <62.65%> (+0.01%) ⬆️
ocsp-rhel8-test-ocsp-ecdsa-valid-cert-server-staples-latest-python3.14-cov 34.20% <24.09%> (-0.01%) ⬇️
ocsp-rhel8-test-ocsp-rsa-valid-cert-server-staples-latest-python3.14-cov 34.20% <24.09%> (-0.01%) ⬇️
pyopenssl-macos-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.76% <63.85%> (+0.04%) ⬆️
pyopenssl-rhel8-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.82% <63.85%> (+0.10%) ⬆️
pyopenssl-win64-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.69% <62.65%> (-0.03%) ⬇️
stable-api-accept-v2-rhel8-auth-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.36% <63.25%> (+0.02%) ⬆️
stable-api-accept-v2-rhel8-auth-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 59.03% <63.85%> (+0.03%) ⬆️
stable-api-require-v1-rhel8-auth-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.33% <63.25%> (+0.02%) ⬆️
stable-api-require-v1-rhel8-auth-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 60.96% <62.65%> (+<0.01%) ⬆️
stable-api-require-v1-rhel8-auth-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.98% <62.65%> (+0.01%) ⬆️
storage-inmemory-rhel8-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.37% <63.25%> (+0.03%) ⬆️
storage-inmemory-rhel8-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 59.01% <62.65%> (+0.01%) ⬆️
test-macos-arm64-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.34% <64.45%> (-0.02%) ⬇️
test-macos-arm64-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.51% <63.85%> (+0.01%) ⬆️
test-macos-arm64-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.09% <62.65%> (+<0.01%) ⬆️
test-macos-arm64-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.99% <62.65%> (-0.01%) ⬇️
test-macos-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.36% <64.45%> (+0.01%) ⬆️
test-macos-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.53% <63.85%> (+0.04%) ⬆️
test-macos-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.10% <62.65%> (+<0.01%) ⬆️
test-macos-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 59.00% <63.85%> (+<0.01%) ⬆️
test-numpy-macos-arm64-test-numpy-python3.14-python3.14-cov 32.58% <15.06%> (-0.05%) ⬇️
test-numpy-macos-test-numpy-python3.14-python3.14-cov 32.58% <15.06%> (-0.03%) ⬇️
test-numpy-rhel8-test-numpy-python3.14-python3.14-cov 32.58% <15.06%> (-0.04%) ⬇️
test-numpy-win32-test-numpy-python3.14-python3.14-cov 32.56% <15.06%> (-0.03%) ⬇️
test-numpy-win64-test-numpy-python3.14-python3.14-cov 32.56% <15.06%> (-0.02%) ⬇️
test-win32-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.24% <63.25%> (-0.05%) ⬇️
test-win32-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.44% <62.65%> (+<0.01%) ⬆️
test-win32-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.08% <62.65%> (+<0.01%) ⬆️
test-win32-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.88% <62.65%> (+0.01%) ⬆️
test-win64-test-standard-latest-python3.11-async-noauth-nossl-standalone-cov 59.22% <63.25%> (+0.01%) ⬆️
test-win64-test-standard-latest-python3.12-async-noauth-ssl-replica-set-cov 61.45% <62.65%> (+0.01%) ⬆️
test-win64-test-standard-latest-python3.13-async-auth-ssl-sharded-cluster-cov 61.08% <62.65%> (+<0.01%) ⬆️
test-win64-test-standard-latest-python3.14-async-noauth-nossl-standalone-cov 58.88% <62.65%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sophiayangDB sophiayangDB marked this pull request as ready for review May 15, 2026 19:26
@sophiayangDB sophiayangDB requested a review from a team as a code owner May 15, 2026 19:26
Comment thread doc/contributors.rst
- Steven Silvester (blink1073)
- Noah Stapp (NoahStapp)
- Cal Jacobson (cj81499)
- Sophia Yang (sophiayangDB)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay!! first pymongo change!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants