Repository navigation
Fix misleading hint for iterparse #93618
Description
Documentation
From docs:
Because it’s so flexible, XMLPullParser can be inconvenient to use for simpler use-cases.
If you don’t mind your application blocking on reading XML data
but would still like to have incremental parsing capabilities, take a look at iterparse().
It can be useful when you’re reading a large XML document and don’t want to hold it wholly in memory.
The last sentence is wrong. iterparse eventually loads the entire file to the memory, because iterparse forms XML tree incrementally, i.e., the problem of huge memory consumption cannot be solved by using iterparse without some extra code. If a person wants to process a large file with small memory cost, then they must at least repeatedly clean root element from children (it depends on XML structure). Therefore, I suggest to remove the last sentence in docs.
The code below deletes a root child once it is completed, then processes and removes it from the memory (if nothing more references to it ofc). This allows to process 7GB XML with with a memory usage up to 10MB (in case of great number of root children).
parser = XMLPullParser(['start', 'end']) # can be replaced with iterparse as well
root = None
with open(file) as f:
for line in f:
parser.feed(line)
for event, obj in parser.read_events():
match event:
case 'start':
if root is None: root = obj
case 'end':
if len(root) > 0 and obj == root[0]:
del root[0]
# process obj
parser.close()Linked PRs
- gh-93618: Document the memory usage of the incremental parsers #156763
- [3.15] gh-93618: Document the memory usage of the incremental parsers (GH-156763) #157419
- [3.14] gh-93618: Document the memory usage of the incremental parsers (GH-156763) #157420
- [3.13] gh-93618: Document the memory usage of the incremental parsers (GH-156763) #157421
Activity
c-viswanath commentedon Jun 12, 2022
Hello @Prometheus3375 , I'm new to contributing in Open-Source and want to solve this doc issue as my first issue, could you please guide me through the process of doing the same?
Prometheus3375 commentedon Jun 12, 2022
@c-viswanath AFAIK, the issue must be acknowledged by a contributor or a maintainer, and then you can create a PR with fixes.
Prometheus3375 commentedon Jun 12, 2022
I rephrased initial comment here, now it should be more comprehensive.
serhiy-storchaka commentedon Sep 1, 2026
PR #156763 replaces the misleading sentence.
@Prometheus3375 is right: the tree is only built incrementally, it is not freed incrementally, so the whole document is eventually held in memory. Rather than only removing the sentence, the section now explains this and gives two examples for common cases: clearing large processed elements, and removing the processed children of an element which has many of them. It also mentions that a custom target of
XMLParserdoes not build a tree at all.This should also answer gh-102055.
LeonxLJX commentedon Sep 3, 2026
serhiy-storchaka commentedon Sep 3, 2026
@LeonxLJX Please, read the issue before posting such comments. There is already a PR, and the comment above mentions it.
Also please avoid posting meaningless LLM generated comments.
gh-93618: Document the memory usage of the incremental parsers (GH-15…
[3.14] gh-93618: Document the memory usage of the incremental parsers (…
[3.13] gh-93618: Document the memory usage of the incremental parsers (…
pythongh-93618: Document the memory usage of the incremental parsers (p…
[3.15] gh-93618: Document the memory usage of the incremental parsers (…