Skip to content

Possible memory leak in Validator  #126

Description

@wcloete

Thank you for a really great project!

Unfortunately I have observed that when validating invalid events the rapidjson.Validator leaks a little memory. Here is some sample code to reproduce it (tested on Python 3.7.3, MacOS Mojave, with python-rapidjson 0.8.0).

import rapidjson
import gc
import tracemalloc

schema = """{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "required": ["id", "name"],
    "type": "object",
    "properties": {
        "id": {"type": "integer"},
        "name": {"type": "string"}
    }
}""".encode("utf-8")

obj = """{
    "id": 50
}""".encode("utf-8")

validate = rapidjson.Validator(schema)
snapshots = []

# start the test
tracemalloc.start()
for _ in range(2):
    for _ in range(500000):
        try:
            validate(obj)
        except rapidjson.ValidationError:
            pass
    # garbage collect before taking snapshot to exclude memory that is 
    gc.collect()
    snapshots.append(tracemalloc.take_snapshot())

# report
for stat in snapshots[1].compare_to(snapshots[0], "traceback")[:10]:
    print(stat)

This returns something like:

minimal_memory_test.py:27: size=123 MiB (+61.5 MiB), count=1999982 (+1000000), average=64 B
/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tracemalloc.py:397: size=600 B (+600 B), count=4 (+4), average=150 B
/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tracemalloc.py:534: size=488 B (+488 B), count=2 (+2), average=244 B
/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tracemalloc.py:291: size=112 B (+112 B), count=2 (+2), average=56 B
/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tracemalloc.py:532: size=128 B (+64 B), count=2 (+1), average=64 B
minimal_memory_test.py:32: size=464 B (+32 B), count=2 (+1), average=232 B
minimal_memory_test.py:31: size=59 B (+0 B), count=1 (+0), average=59 B
minimal_memory_test.py:24: size=48 B (+0 B), count=1 (+0), average=48 B
minimal_memory_test.py:25: size=28 B (+0 B), count=1 (+0), average=28 B

Note the ~64B leak for the line validate(obj).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions