Repository navigation
Make ElementTree event handling more modular to allow custom targets for the non-blocking parser #63102
Description
| BPO | 18902 |
|---|---|
| Nosy | @jcea, @scoder, @jkloth |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2013-09-02.00:03:41.532>
labels = ['type-feature', 'library']
title = 'Make ElementTree event handling more modular to allow custom targets for the non-blocking parser'
updated_at = <Date 2013-09-28.17:43:11.005>
user = 'https://bugs.python.org/elibendersky'bugs.python.org fields:
activity = <Date 2013-09-28.17:43:11.005>
actor = 'scoder'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-09-02.00:03:41.532>
creator = 'eli.bendersky'
dependencies = []
files = []
hgrepos = []
issue_num = 18902
keywords = []
message_count = 8.0
messages = ['196758', '196830', '196832', '197552', '197590', '197606', '198526', '198537']
nosy_count = 4.0
nosy_names = ['jcea', 'scoder', 'jkloth', 'eli.bendersky']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18902'
versions = ['Python 3.5']
Activity
elibendersky commentedon Sep 2, 2013
As issue bpo-17741 shows in detail, there's a number of implementation problems in the current ET that make a fully non-blocking parser more difficult to implement than it should be.
The main problem is that XMLParser._setevents relies on internal implementation details of the default TreeBuilder to do some of its tasks.
This has to be decoupled from implementation details completely. An XMLParser already has a non-blocking input side (feed/close) and a callback "push" API to the target. The API has to be fully expanded to include start-ns/end-ns and any other events that are required to allow custom tree builders as targets. This would make it possible to layer XMLPullParser on top of the stock XMLParser coupled with a special target that collects "events" from the callback calls.
scoder commentedon Sep 3, 2013
Given that we have an XMLPullParser now, I think we should not clutter the API with more classes that do not add anything more than making other classes redundant. So, -1 on adding a special target that collects events. Anyone who wants to use that feature can just use the XMLPullParser.
In the Python implementation of that parser, custom targets can already be allowed right now. Only the C implementation prevents it currently (AFAICT) as it has its own built-in TreeBuilder target.
Therefore, +1 for the general cleanup to properly allow arbitrary targets as backend.
Also, +1 for allowing start-ns and end-ns event callbacks on parser targets, although that's a different feature entirely.
scoder commentedon Sep 3, 2013
(fixing subject to properly hit bug filters)
[-]Make ET event handling more modular to allow custom targets for the non-blocking parser[/-][+]Make ElementTree event handling more modular to allow custom targets for the non-blocking parser[/+]scoder commentedon Sep 13, 2013
The way the XMLPullParser is implemented in lxml.etree now is that it simply inherits from XMLParser. This would also make sense for ElementTree, even before supporting arbitrary targets. The patch in ticket bpo-18990 makes this simple to do.
For reference, here is the implementation in lxml.
Iterparse:
https://github.com/lxml/lxml/blob/master/src/lxml/iterparse.pxi
XMLPullParser:
https://github.com/lxml/lxml/blob/d9f7cd8d12a27cafc4d65c6e280ea36156e3b837/src/lxml/parser.pxi#L1357
SAX based parser that collects events and/or maps parser callbacks to callbacks on the target object:
https://github.com/lxml/lxml/blob/master/src/lxml/saxparser.pxi
scoder commentedon Sep 13, 2013
Actually, I take that back. I can't see a use case for this feature, and it doesn't really fit with the notion of fully qualified tag names in ElementTree (which itself is a great feature). I'm -0.7 on adding this.
scoder commentedon Sep 13, 2013
I created a separate ticket bpo-19010 for the inheritance issue.
scoder commentedon Sep 28, 2013
Closing bpo-18990 defines the API of the new XMLPullParser that 3.4 will ship with, so this ticket becomes an enhancement for future versions.
scoder commentedon Sep 28, 2013
Copying a relevant comment by Eli from http://bugs.python.org/issue18990#msg198145 and replying inline.
"""
The way the APIs are currently defined, XMLParser and XMLPullParser are different animals. XMLParser can be considered to only have one "front" in the API - feed() and close(). You feed() until the document is done and then you close() and get the parsed tree. There's no other way to get the parsed tree (unless you use a custom builder, I guess).
On the other hand XMLPullParser has two clear "fronts" - an input front with feed() and close() and an output front with read_events(). For XMLPullParser, close() is just an input signal. The canonical way to get output from XMLPullParser is read_events(). close() has no better reason to return output than feed(). When we decided to change the method names (recall that Antoine's originals were completely different), we perhaps forgot this detail.
"""
No, we didn't.
"""
Even though XMLPullParser's method is named close(), it's *not* like XMLParser's close(). If someone is using XMLPullParser for its close() he's likely using the class incorrectly.
Just as an example: consider that in a lot of use cases the programmer will want to discard parts of the tree that's parsed iteratively (similarly to the main use case of iterparse()), because the XML itself is too huge. It's a convenient streaming API, in other words. Now, if the reader discards parts of the tree (by deleting subtrees), then returning the root from close() becomes even more meaningless, because it's no longer the root and we have no idea what it actually is.
"""
Let me repeat that this was already the case before the new class was added and that it's a feature. If the target decides to discard parts of the tree, or not build a tree at all and (say) instead count elements and return their total number on close(), then that's what the user asked for by selecting that target.
Let's agree to disagree on your conclusions, but I still can't see any advantages of making the separation between the two classes. The way I see it, making XMLPullParser inherit from XMLParser makes it very easy to explain what the difference is: the read_events() method, i.e. the additional way to receive the parse events that the combination of parser and target generate. Essentially, it's the target that does all the work here and the parser only collects the results and presents them to the user. Thus my intention to keep the parser as "stupid" as it looks from the user's side, instead of adding something new right next to it.
That being said, if ElementTree keeps them separate and decides to *never* return anything from XMLPullParser.close(), then that's sufficiently compatible with lxml.etree, so I won't object to it. lxml has a long history of extending what's there in order to make it easier to use.
As long as we can find a way to keep both libraries compatible for users, I think we should be able to both move forward.
Merge remote-tracking branch 'upstream/main' into pythongh-63102-pull…
Merge branch 'main' into pythongh-63102-pull-target
gh-63102: Support custom targets in the pull parser (GH-156888)
serhiy-storchaka commentedon Sep 13, 2026
#156888 is merged:
XMLPullParseranditerparse()accept a custom target, and report what its methods return.#157439 adds
XMLPullTarget, a target which reports elements without building a tree, andXMLPullParser.expand(), which builds the subtree of a reported element on demand. Together they give whatxml.dom.pulldomgives withexpandNode(), but withElementobjects and about 8 times faster.pythongh-63102: Support custom targets in the pull parser (pythonGH-1…