Skip to content

Benchmark: Might be worth including hyperjson and orjson #130

Closed
Closed
Benchmark: Might be worth including hyperjson and orjson#130

Description

@hartwork
hartwork
opened on Feb 20, 2020

Hi!

The benchmark at https://python-rapidjson.readthedocs.io/en/latest/benchmarks.html does not include the potentially most interesting contender — orjson — now that ujson and yajl are dead. Are there plans benchmarking against orjson?

Best, Sebastian

Activity

  1. lelit commented on Feb 20, 2020

    @lelit
    lelit
    on Feb 20, 2020
    Contributor

    Yes, it could be of some interest. Any chance you can offer a PR?

  2. hartwork commented on Feb 20, 2020

    @hartwork
    hartwork
    on Feb 20, 2020
    Author

    I think I might but the CI started failing today and a green CI would be a better base for a pull request to be merged in a green state. Any idea what macOS Travis is doing wrong?

  3. hartwork commented on Feb 20, 2020

    @hartwork
    hartwork
    on Feb 20, 2020
    Author

    PS: Regarding the benchmarks, I was trying PYTHONPATH=. make benchmarks-tables now but benchmarks/tablize.py will fail with KeyError: 'ujson' if I don't have ujson around. Getting ujson to run with recent Python 3 seems tricky, I first hit ultrajson/ultrajson#271 with 1.35 and then ultrajson/ultrajson#277 with its Git master:

    # python --version
Python 3.7.6

# pip install ujson==1.35
# python -c 'import ujson'
[..]
ImportError: /tmp/tmp.P6fdeuYf1C/python-rapidjson/py37/lib/python3.7/site-packages/ujson.cpython-37m-x86_64-linux-gnu.so: undefined symbol: Buffer_AppendShortHexUnchecked

# pip install git+https://github.com/esnme/ultrajson.git
# python -c 'import ujson'
[..]
ImportError: /tmp/tmp.P6fdeuYf1C/python-rapidjson/py37/lib/python3.7/site-packages/ujson.cpython-37m-x86_64-linux-gnu.so: undefined symbol: dconv_d2s_init

    Do you have an easy way of getting ujson to work?

  4. lelit commented on Feb 21, 2020

    @lelit
    lelit
    on Feb 21, 2020
    Contributor

    Any idea what macOS Travis is doing wrong?

    I saw the failure, but I could not investigate further. I won't be able to switch focus to RJ until next week (reasonably).

  5. lelit commented on Feb 21, 2020

    @lelit
    lelit
    on Feb 21, 2020
    Contributor

    Do you have an easy way of getting ujson to work?

    If that's the current state of affair, I'd consider removing it from the benchmarks...

  6. hartwork commented on Feb 21, 2020

    @hartwork
    hartwork
    on Feb 21, 2020
    Author

    Any idea what macOS Travis is doing wrong?

    I saw the failure, but I could not investigate further. I won't be able to switch focus to RJ until next week (reasonably).

    I'm happy to get back to this when you're back.

    Do you have an easy way of getting ujson to work?

    If that's the current state of affair, I'd consider removing it from the benchmarks...

    I checked for ujson-based alternatives now. This is what I found:

    • fast-json 0.3.2 of 2018-12 — fails to import with py37
    • nujson 1.35.2 of 2019-12 — fails to import with py37
    • ujson-bedframe 1.33.0 of 2014-01 — fails to import with py37
    • ujsongp 1.35.1 of 2017-11 — single surrogates dropped silently, no issue tracker upstream
    • ujson-ia 2.1.1 of 2019-05 — fails to import with py37
    • ujson-segfault 2.1.57 of 2019-02 — works (single surrogates rejected), no issue tracker upstream
    • ujson-x 1.37 of 2019-03 — fails to import with py37
    • ujson-x-legacy 1.35.1 of 2019-03 — fails to import with py37

  7. hartwork commented on Feb 21, 2020

    @hartwork
    hartwork
    on Feb 21, 2020
    Author

    PS: hyperjson would be about as interesting to add as orjson.

  8. hartwork
    changed the title [-]Benchmark: Might be worth including orjson[/-] [+]Benchmark: Might be worth including hyperjson and orjson[/+] on Feb 21, 2020
  9. hartwork
    added a commit that references this issue on Feb 21, 2020

    slides: Add hyperjson and ujson-segfault to JSON decoders

    f5e0bce

  10. lelit commented on Feb 22, 2020

    @lelit
    lelit
    on Feb 22, 2020
    Contributor

    I think I might but the CI started failing today and a green CI would be a better base for a pull request to be merged in a green state.

    It seems I cured the problem, and now CI is green again!

  11. hartwork commented on Feb 23, 2020

    @hartwork
    hartwork
    on Feb 23, 2020
    Author

    Thanks for that!

  12. oleng commented on Feb 23, 2020

    @oleng
    oleng
    on Feb 23, 2020

    There's simdjson as well, written in C++ although I'm not sure about how the Python wrappers works as standard library replacement.

  13. hartwork commented on Feb 25, 2020

    @hartwork
    hartwork
    on Feb 25, 2020
    Author

    @oleng The problem (and feature) with simdjson is that that it needs a CPU with AVX2 instructions, e.g. you cannot run the full benchmark on a Lenovo X220 laptop with an i5-2520M CPU:

    # python -c 'import simdjson'
Illegal instruction

    At least that's how current pysimdjson wheels behave and pip install --no-binary pysimdjson{,} fails.

    So then only people with AVX2 supporting CPUs can update python-rapidjson benchmarks. I'm not sure that's ideal.

  14. oleng commented on Feb 25, 2020

    @oleng
    oleng
    on Feb 25, 2020

    Obviously I haven't mull around this for long, but that sounds to me more like it depends on the goal of releasing benchmark comparison than the amount of people with CPUs supporting AVX2, given that the latter has been around for 6+ years. Maybe if the goal is to provide complete picture of what's available now then it can be included, only with a caveat note?

    Based on my own personal experience, I think people most likely seek/find the benchmark comparison because they want to know which external module/library can offer more/most performance for parsing JSONs because either it would help with increasing frequency of processing small/normal sized JSONs, or if not then they're dealing with non trivial amount of data in their JSON files. As anecdote, I noticed quite a bit of SO questions indicated they need ways to process gigabytes of JSON files with pandas, and pandas aren't memory efficient so loading big JSON into memory isn't possible/preferrable, but process/stream it in chunks is also tricky.

  15. lelit
    added a commit that references this issue on Feb 29, 2020

    Add orjson and hyperjson to the benchmarks and renew comparison tables

    a5ec68f

  16. lelit commented on Feb 29, 2020

    @lelit
    lelit
    on Feb 29, 2020
    Contributor

    orjson is impressive indeed!

  17. hartwork commented on Feb 29, 2020

    @hartwork
    hartwork
    on Feb 29, 2020
    Author

    … if we ignore the pain of installing it for a moment 😃

    hyperjson and orjson have been added by a5ec68f now so I'll close this ticket as fixed.

    PS: The updated benchmarks-tables.rst can be seen at
    https://github.com/python-rapidjson/python-rapidjson/blob/616b91c84021f1a4ddd53e5d691fef6ff23178f7/docs/benchmarks-tables.rst .

  18. hartwork
    closed this as completedon Feb 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions