I'd like to parse a very large XML file in a memory-efficient way. iterparse seems to be designed for this, but doing it safely is FAR less intuitive than one would expect.

Bug report

Suppose I have an XML file that looks like this:

<root>
  <section1>
    <uninteresting>...</uninteresting>
    <uninteresting>...</uninteresting>
    <uninteresting>...</uninteresting>
    <uninteresting>...</uninteresting>
    <!-- ... huge number of these uninteresting tags ... -->
  </section1>
  <section2>
    <interesting>...</interesting>
    <interesting>...</interesting>
    <interesting>...</interesting>
  </section2>
</root>

I want to parse only the data in the interesting tags. One would think you could write code that looks something like this:

    for event,elem in ET.iterparse(xml_fh, events=("end",), parser=ET.XMLParser()):
        if elem.tag == "interesting":
            process_interesting(elem)

Unfortunately, this causes a huge memory spike where every tag in the entire XML document is held in memory until the end of the loop, at which point it is all freed. I suspect there are circular references between objects somewhere along the line, causing the garbage collector to have a hard time freeing things, but I haven't dug that deep.

I've read in some ancient lxml posts that one needs to run .clear() on the elements. One might think this will fix the issue:

    for event,elem in ET.iterparse(xml_fh, events=("end",), parser=ET.XMLParser()):
        if elem.tag == "interesting":
            process_interesting(elem)
        elem.clear()

Well, it sorta does, but it will break your parsing. If you run the clear method on a tag that is a parent of interesting, then you lose all of your interesting tags. Instead, this helps keep the memory down in my specific case:

    for event,elem in ET.iterparse(xml_fh, events=("end",), parser=ET.XMLParser()):
        if elem.tag == "interesting":
            process_interesting(elem)
            elem.clear()
        elif elem.tag == "uninteresting":
            elem.clear()

This works in my case only because uninteresting tags make up the bulk of the file, with a ton of sub-elements under each. Other tags are of course kept in memory until the loop finishes, but in my case, those aren't big enough to cause me trouble. A more complete solution might be:

    for event,elem in ET.iterparse(xml_fh, events=("end",), parser=ET.XMLParser()):
        if elem.tag == "interesting":
            process_interesting(elem)
            elem.clear()
        elif elem.tag not in ('section2', ... ): # all parents of interesting tags
            elem.clear()

In the end, I don't think this is a behavior a reasonable developer would expect. People turn to iterparse to keep their memory usage down, but without lots of weird explicit memory-manipulating clear() calls, it doesn't do that.

BTW, I'm tracking memory usage during the parsing loop by using simple calls to psutils using helper functions like this:

def current_memory():
    p = psutil.Process(os.getpid())
    mem_info = p.memory_info()
    mb = 1024*1024.0
    return (mem_info.rss/mb,mem_info.data/mb)

def current_memory_usage():
    return "real: %dM, data: %dM" % current_memory()

Your environment

Ubuntu 22.04
python3 from Ubuntu, package version: 3.10.6-1~22.04