default for tuples , time.struct_time, collections.counter, collections.ordereddict and collections.defaultdic ?

Hi, I'm developing serializejson, a python library for fast serialization and deserialization of python objects in JSON designed as a safe, interoperable and human-readable drop-in replacement for the Python pickle package.

I use python-rapidjson in backend because it is the only one that is fast enough and has the sufficient API for serializejson needs.
Sadly python-rapidjson doesn't call default() method for tuples, time.struct_time, collections.counter, collections.ordereddict and collections.defaultdic. Python-rapidjson serialize them as a list or dictionary without any possibility to catch them with default().

Did I miss something? Would it be possible to add an option to call default for these objects?

Thanks !

Baptiste

Did I miss something?

That seems correct: any iterable value is serialized as a JSON array.

Would it be possible to add an option to call default for these objects?

Yes, but that's tricky: the default() function, when specified, is called for unrecognized values, and is expected to return something else that the library handles, simply recursing thru the serialization machinery. So, if we add an option to not recognize iterables, and thus passing them to the default(), what would it return?

hi
thanks for your response .
default() would return dictionnaries with "__class__" and "__init__" keys.
serializejson 0.1.0 is already able to load desired objects from such dictionnaries :

from serializejson import loads
print(loads('{"__class__":"tuple","__init__":[1,2]}'))
print(loads('{"__class__":"time.struct_time","__init__":[2020,11,28,10,10,58,5,333,0]}'))
print(loads('{"__class__":"collections.Counter","__init__":{"a":1,"b":2,"c":3}}'))
print(loads('{"__class__":"collections.OrderedDict","__init__":{"a":1,"b":2,"c":3}}'))
print(loads('{"__class__":"collections.defaultdict","__init__":[null,{"a":1,"b":2,"c":3}]}'))

The simplejson library is too slow but call default for all this objects (struct_time,Counter, OrderedDict, defaultdict) and for tuples if the option tuple_as_array is set False.

I may try adding a similar option.

New iterable_mode serialization option

The mentioned commit is just a partial fix, more work is needed to handle dict subclasses.
Please give it a try, and tell me if I'm on the right track.

Your iterable_mode works ! I can now serialize tuples and time.struct_time ! Thanks !

Great! I will need to investigate how to handle in the same way also the other types you mentioned.

New mapping_mode serialization option

The new mapping_mode should fulfill your requested feature, can you try that as well please?
BTW, I tried with simplejson and could not figure out how to make it call default() for the dict subclasses, how did you do that?

Perfect ! it works ! I can now serialize all types of objects and replace pickle by serializejson in all my codes !
Thank you very much :)

Forgot to mention that in current master I slightly changed the iterable_mode option: to get the behaviour above you wanna use ONLY_LISTS.

Released as v1.0.

Navigation Menu

Repository navigation

default for tuples , time.struct_time, collections.counter, collections.ordereddict and collections.defaultdic ? #149

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

default for tuples , time.struct_time, collections.counter, collections.ordereddict and collections.defaultdic ? #149

Description

Activity

lelit commented on Nov 27, 2020

SmartAudioTools commented on Nov 28, 2020

lelit commented on Nov 28, 2020

lelit commented on Nov 28, 2020

SmartAudioTools commented on Nov 28, 2020

lelit commented on Nov 29, 2020

lelit commented on Nov 29, 2020

SmartAudioTools commented on Nov 29, 2020

lelit commented on Dec 9, 2020

lelit commented on Dec 13, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions