Skip to content
This repository was archived by the owner on Dec 10, 2025. It is now read-only.
This repository was archived by the owner on Dec 10, 2025. It is now read-only.

Memory leak when scanning large table #32

@clusterfudge

Description

@clusterfudge

I'm currently using bigtable for archival of tsdb data. When attempting to restore, I find that I have to frequently release the table.scan() cursor, or my memory usage skyrockets (will consume 32gb within an hour). With checkpointing in place and new cursors, I am not seeing this issue. Is this working as intended?

I've created 2 gists to demonstrate the behavior, both of which assume they're running on a GCE instance with local instance credentials granted the necessary BT roles.
https://gist.github.com/clusterfudge/3fe40f7f55e8a2a80c65dbd8e8d6764e - implements checkpointing, has a reasonable and stable memory footprint

https://gist.github.com/clusterfudge/3a2e6b484d24070b6f43fd3fd989e229 - simple single call to scan, will consume all memory available, stopped only by the heat death of the universe.

Metadata

Metadata

Assignees

Labels

🚨This issue needs some love.priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions