Skip to content

object_hook memory leak #115

Description

@mklokocka

OS: MacOS High Sierra
Python version: 3.7.1
Rapidjson version: 0.6.3

It seems I am facing a memory leak problem when using rapidjson.loads and utilizing an object hook. Here is an example code:

import rapidjson
from datetime import timedelta
import tracemalloc

tracemalloc.start()

def object_hook(td):
    if '__td__' in td:
        return timedelta(td['__td__'])
    else:
        return td

def parse_json():
    data = []
    for i in range(1, 100):
        data.append({"name": f"a{i}", "timestamp": timedelta(seconds=i)})

    a = rapidjson.dumps(
        data,
        datetime_mode=rapidjson.DM_ISO8601,
        default=lambda td: {"__td__": td.total_seconds()} if isinstance(td, timedelta) else td,
    )

    data = rapidjson.loads(
        a,
        datetime_mode=rapidjson.DM_ISO8601,
        object_hook=object_hook
    )
    return data

print("before")
if tracemalloc.is_tracing():
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    print([str(x) for x in top_stats[:10]])

for _ in range(1000):
    parse_json()

print("after")
if tracemalloc.is_tracing():
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    print([str(x) for x in top_stats[:10]])

After running this I get: 'memleaktest.py:9: size=3867 KiB, count=99000, average=40 B' as the first line of the tracemalloc output. I tried using objgraph as well, but it did not show any Python objects, so it seems like a C++ malloc issue here.

I am using rapidjson in production and my, fairly simple, API is slowly eating memory and this is the prime suspect (getting a tracemalloc pointer to the line basically the same as the one here).

Is this working as intended? Do you need any more information to pinpoint the problem?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions