Safer data serialization with marshal module

Feature or enhancement

The main purpose of the marshal module -- serialization of precompiled module code objects. This requires support of code objects, strings for names, and other primitive Python types and simple collection types referred by the code object.

It allows to use it as more data generic serialization tool -- more limited than pickle, but less limited than JSON. The marshal module supports different versions of the format and backward compatible with all earlier versions. But only if the data does not contain code objects. The format of the code objects changed with every Python version, and this is not reflected in marshal format version. Loading marshal data created in different Python version has undefined behavior if the data contains a code object.

I propose to add a keyword-only parameter allow_code with default value True in marshal functions. Specifying allow_code=False forbid saving and loading code objects. It allows to be safer when you load external data and to guarantee that the output can be safely loaded in other Python.

Linked PRs

gh-113626: Add allow_code parameter in marshal functions #113648

pythongh-113626: Add allow_code parameter in marshal functions

@bedevere-bot

is this issue solved now? @bedevere-bot

I appreciate that in your PR you are not describing marshal as "safe" or fit for any particular purpose. ie: the big Warning in https://docs.python.org/3/library/marshal.html always applies.

While I don't mind this feature, I wonder what the point is? Who is expected to use this and why?

We cannot make promises about the marshal data format safety and capabilities. It was never designed for such use.

ie: Anywhere someone should not use pickle, they should also not use marshal, even with allow_code=False. Given that, in what scenarios do you see this feature actually being used?

The marshal code hasn't been written with security in mind, nor do we fuzz test it. Nor do we do meaningful interoperability testing across versions. Because that hasn't been its purpose.

It is still not safe because you can still create a malicious marshal data that has exponentially large complexity of loading, and the unmarshaller is perhaps vulnerable to stack overflow. But you need to construct it specially, not using the marshal module.

marshal is one of builtin data formats, it is often faster and more compact than pickle and JSON. It is safe if used with trusted input. It is used as a data format. If it was unsupported, we would not bother with versioning and backward compatibility. But if a code object is accidentally leaked in the marshalled data, unmarshalling it in other Python version can crash or corrupt the Python interpreter. This option will protect you from this kind of errors.

An example of malicious marshal data:

t = ()
for i in range(100):
    t = (t, t)

k = (1, 2)
import marshal
data = marshal.dumps([k, t, {k: t}])
# replace the reference to k as the dict key with the reference to t
data = data.replace(b'r\0', b'r\3')
x = marshal.loads(data)  # hangs

Thanks, that makes sense. I'm fine with adding this. I think our existing marshal documentation warns people well enough about its caveats.

gh-113626: Add allow_code parameter in marshal functions (GH-113648)

pythongh-113626: Add allow_code parameter in marshal functions (pytho…

Navigation Menu

Repository navigation

Safer data serialization with marshal module #113626

Feature or enhancement

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Safer data serialization with marshal module #113626

Description

Feature or enhancement

Linked PRs

Activity

vaishnavi192 commented on Jan 6, 2024

gpshead commented on Jan 11, 2024

serhiy-storchaka commented on Jan 12, 2024

serhiy-storchaka commented on Jan 12, 2024

gpshead commented on Jan 13, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions