Skip to content

unix timestamp #61

Closed
Closed
unix timestamp#61

Description

@alka2
alka2
opened on Feb 22, 2017

Rapidjson supports ISO8601 format for datetime object in dump. Possible to add support for unix timestamp?
(can be handled with "default" parameter in dump but it is nice to have it built-in).

Activity

  1. sontek commented on Feb 22, 2017

    @sontek
    sontek
    on Feb 22, 2017
    Contributor

    You want to use the datetime_mode argument.

    python
Python 3.5.0 (default, Sep 23 2015, 04:42:00)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.72)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import rapidjson
>>> from datetime import datetime
>>> rapidjson.dumps(datetime.now(), datetime_mode=1)
'"2017-02-22T11:00:28.885223"'
>>>

    There are also constants for this:

    https://github.com/python-rapidjson/python-rapidjson/blob/master/python-rapidjson/rapidjson.cpp#L33-L38

  2. alka2 commented on Feb 22, 2017

    @alka2
    alka2
    on Feb 22, 2017
    Author

    That's only ISO8601 format.

    Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 12:22:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rapidjson
>>> from datetime import datetime
>>> from calendar import timegm
>>> def encode_datetime(obj):
...     if isinstance(obj, datetime):
...             return int(timegm(obj.timetuple()))  # returns unix timestamp, UTC time
...     raise TypeError(repr(obj) + " is not JSON serializable")
...
>>> rapidjson.dumps(datetime.now(), default=encode_datetime)
'1487766521'
>>>

    possible to have this built-in?
    (ujson is returning unix timestamp by default).

  3. lelit commented on Feb 22, 2017

    @lelit
    lelit
    on Feb 22, 2017
    Contributor

    While it'd be quite easy to add another flag for that, the fact that it could work only for serialization makes me somewhat unhappy...
    BTW, isn't that equivalent to obj.timestamp()?

  4. alka2 commented on Feb 22, 2017

    @alka2
    alka2
    on Feb 22, 2017
    Author

    obj.timestamp() doesn't return UTC time. Rapidjson is already handling ISO8601 format. I think it would be useful to add support for unix timestamp.
    We are trying to switch from ujson to rapidjson and this is the main problem that we are encountering.

  5. kenrobbins commented on Feb 23, 2017

    @kenrobbins
    kenrobbins
    on Feb 23, 2017
    Member

    @s-alikarimi It's useful to you as a built-in just for the speed factor, I assume?

    BTW, I'm curious on your reasons for moving from ujson to rapidjson.

  6. alka2 commented on Feb 23, 2017

    @alka2
    alka2
    on Feb 23, 2017
    Author

    yes. we are processing millions of jsons at a time.
    ujson doesn't support the Decimal objects properly especially when more than 9 precisions are needed.

    Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 12:22:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import simplejson
>>> import ujson
>>> import rapidjson
>>> from decimal import Decimal
>>> simplejson.dumps({"c": 173688.15}, use_decimal=True)
'{"c": 173688.15}'
>>> rapidjson.dumps({"c": 173688.15}, use_decimal=True)
'{"c":173688.15}'
>>> ujson.dumps({"c": 173688.15}, double_precision=15)
'{"c":173688.149999999994179}'
>>> a = {"p": Decimal('45.123456789012349').quantize(Decimal('0.000000000000001'))}
>>> simplejson.dumps(a, use_decimal=True)
'{"p": 45.123456789012349}'
>>> rapidjson.dumps(a, use_decimal=True)
'{"p":45.123456789012349}'
>>> ujson.dumps(a, double_precision=15)
'{"p":45.123456789012351}'
>>>

  7. lelit commented on Feb 23, 2017

    @lelit
    lelit
    on Feb 23, 2017
    Contributor

    I will try to make some experiment on this, next week. Do you think it would require a "combination" of flags, to be able to obtain either the "floating point" value like standard lib .timestamp() or the "integer" value like you asked above, and of course native/UTC like current DATETIME_MODE?

  8. lelit commented on Feb 23, 2017

    @lelit
    lelit
    on Feb 23, 2017
    Contributor

    What should happen to datetime.date and datetime.time instances?

  9. tamjd1 commented on Feb 23, 2017

    @tamjd1
    tamjd1
    on Feb 23, 2017

    I'm also using rapidjson and am interested in this feature. Looking forward to how this plays out.

  10. alka2 commented on Feb 23, 2017

    @alka2
    alka2
    on Feb 23, 2017
    Author

    I think having both flags is good. In our case we need integer (ujson returns integer as well) but to be more precise having the same output as .timestamp() will be useful. For the DATETIME_MODE UTC should be enough.
    Not sure about datetime.date and datetime.time.

  11. tamjd1 commented on Feb 23, 2017

    @tamjd1
    tamjd1
    on Feb 23, 2017

    @lelit, for datetime.date and datetime.time instances, I don't think we need to do anything. They are already formatted. (Also, I don't think anything can be done for datetime.time instances because there isn't enough info to create a unix timestamp from just time).

    I think those can be left with their default formatting.

    The reason we need formatting options for .timetuple() instance is that it returns an object and not a formatted string.

  12. lelit commented on Feb 23, 2017

    @lelit
    lelit
    on Feb 23, 2017
    Contributor

    Yes, I agree that this "timestamp" serialization of datetimes does not apply to dates and times.

    What I can't see right now is a way to tell rapidjson that I want datetimes serialized as timestamps and dates as their ISO8601 representation. In other words, currently I can say

    >>> import rapidjson as rj
>>> import datetime
>>> now = datetime.datetime.now()
>>> today = datetime.date.today()
>>> rj.dumps([now, today], datetime_mode=rj.DATETIME_MODE_ISO8601)
'["2017-02-23T21:37:50.913678","2017-02-23"]'

    but how can I get the first item formatted as a timestamp instead, while treating the second in the same way? Maybe change those DATETIME_XXX constants to be bit flags instead of their current consecutive integer values, so that the last statement would read

    >>> rj.dumps([now, today], datetime_mode=rj.DATETIME_MODE_ISO8601 + rj.DATETIME_MODE_FLOAT_TIMESTAMP)
'[1487882270.913678,"2017-02-23"]'

    ?

  13. tamjd1 commented on Feb 23, 2017

    @tamjd1
    tamjd1
    on Feb 23, 2017

    That's a good point. Maybe in that case it would be better to format all datetime instances in the same way, for the sake of simplicity and consistency.
    @s-alikarimi what do you think?

  14. lelit commented on Feb 23, 2017

    @lelit
    lelit
    on Feb 23, 2017
    Contributor

    Sorry for being thick, but what do you mean with “format all datetime instances in the same way”?

  15. alka2 commented on Feb 24, 2017

    @alka2
    alka2
    on Feb 24, 2017
    Author

    I think @tamjd1 means something like this:

    import rapidjson as rj
>>> import datetime
>>> now = datetime.datetime.now()
>>> today = datetime.date.today()
>>> rj.dumps([now, today], datetime_mode=rj.DATETIME_MODE_ISO8601)
'["2017-02-23T21:37:50.913678","2017-02-23"]'
>>> rj.dumps([now, today], datetime_mode=rj.DATETIME_MODE_FLOAT_TIMESTAMP)
'[1487907904.471632,1487826000.0]'

    That said convert the today date to datetime object and return its unix timestamp.

  16. 13 remaining items

  17. alka2 commented on Feb 27, 2017

    @alka2
    alka2
    on Feb 27, 2017
    Author

    @lelit, for the timestamp and timegm comparison:

    Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 12:22:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rapidjson
>>> from datetime import datetime
>>> from calendar import timegm
>>> def encode_datetime_timegm(obj):
...     if isinstance(obj, datetime):
...             return int(timegm(obj.timetuple()))
...     raise TypeError(repr(obj) + " is not JSON serializable")
...
>>> def encode_datetime_timestamp(obj):
...     if isinstance(obj, datetime):
...             return int(obj.timestamp())
...     raise TypeError(repr(obj) + " is not JSON serializable")
...
>>> rapidjson.dumps(datetime(2017, 2, 27, 10, 20), default=encode_datetime_timegm)
'1488190800'  # https://www.epochconverter.com/ GMT: Mon, 27 Feb 2017 10:20:00 GMT
>>> rapidjson.dumps(datetime(2017, 2, 27, 10, 20), default=encode_datetime_timestamp)
'1488208800'  # https://www.epochconverter.com/ GMT: Mon, 27 Feb 2017 15:20:00 GMT

    In this case my datetime object is a UTC time and it needs to return 2017-02-27 10:20:00.
    timegm() returns 2017-02-27 10:20:00 while timestamp() returns 2017-02-27 15:20:00.

  18. lelit commented on Feb 27, 2017

    @lelit
    lelit
    on Feb 27, 2017
    Contributor

    Uhm, I'm still confused by your assertions 😈

    What I meant is that the timestamp() method on datetimes does convert the value to GMT before computing the offset, consider:

    >>> import datetime
>>> dt = datetime.datetime(2017, 2, 27, 10, 20)
>>> dt.timestamp()
1488187200.0
>>> dtutc = datetime.datetime(2017, 2, 27, 10, 20, 0, 0, datetime.timezone.utc)
>>> dtutc.timestamp()
1488190800.0

    AFAIU, this tells me a different story: the dt value is actually a naïve datetime, that is a value without a timezone, while the latter, dtutc, is clearly a timezone aware datetime, explicitly set to UTC; to compute the timestamp, Python assumes that the former is in local timezone and thus it shifts it to UTC (because by definition the timestamp is an offset from GMT, also explained by the specific section “What is epoch time?” on that www.epochconverter.com page you referenced), while for the second it knows it already is in that timezone, no shift is needed and it simply computes the offset.

    So, it seems to me that your case is different: when you say, at the end, ”my datetime object is a UTC time”, you actually mean that you know that's an UTC time, but Python does not know that, being given a naïve value. On the contrary, the timegm() call assumes that its argument is a GMT time-tuple. In fact, trying your case using explicitly tz-aware value, i.e. moving the UTC datetime above in your timezone, this is what happens:

    >>> inyourtz = dtutc.astimezone(datetime.timezone(datetime.timedelta(hours=-5)))
>>> inyourtz
datetime.datetime(2017, 2, 27, 5, 20, tzinfo=datetime.timezone(datetime.timedelta(-1, 68400)))
>>> inyourtz.isoformat()
'2017-02-27T05:20:00-05:00'
>>> inyourtz.timestamp()
1488190800.0
>>> inyourtz.timestamp() == dtutc.timestamp()
True

    Back to RJ, we have

    >>> rapidjson.dumps([dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_ISO8601)
'["2017-02-27T10:20:00+00:00","2017-02-27T05:20:00-05:00"]'
>>> rapidjson.dumps([dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_ISO8601+rapidjson.DATETIME_MODE_UTC)
'["2017-02-27T10:20:00+00:00","2017-02-27T10:20:00+00:00"]'
>>> rapidjson.dumps([dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_UNIX_TIME)
'[1488190800.0,1488172800.0]'
>>> rapidjson.dumps([dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_UNIX_TIME+rapidjson.DATETIME_MODE_UTC)
'[1488190800.0,1488190800.0]'

    and that seems reasonable to me.

    So, maybe what you really need is another flag, telling RJ what you know, that is that it should assume that the naïve values you have are effectively in UTC, not in your local timezone, so that it shall compute a timegm()-equivalent value.

    Does this make sense to you?

  19. lelit commented on Feb 27, 2017

    @lelit
    lelit
    on Feb 27, 2017
    Contributor

    One further consideration:

    >>> rapidjson.dumps([dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_ISO8601+rapidjson.DATETIME_MODE_IGNORE_TZ)
'["2017-02-27T10:20:00","2017-02-27T05:20:00"]'
>>> rapidjson.dumps([dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_UNIX_TIME+rapidjson.DATETIME_MODE_IGNORE_TZ)
'[1488190800.0,1488190800.0]'

    Maybe we could (ab)use the otherwise strange combination of IGNORE_TZ and UTC flags to mean the above:

    >>> rapidjson.dumps([dt,dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_ISO8601+rapidjson.DATETIME_MODE_IGNORE_TZ)
'["2017-02-27T10:20:00","2017-02-27T10:20:00","2017-02-27T05:20:00"]'
>>> rapidjson.dumps([dt,dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_ISO8601+rapidjson.DATETIME_MODE_IGNORE_TZ+rapidjson.DATETIME_MODE_UTC)
'["2017-02-27T10:20:00","2017-02-27T10:20:00","2017-02-27T05:20:00"]'
>>> rapidjson.dumps([dt,dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_UNIX_TIME+rapidjson.DATETIME_MODE_IGNORE_TZ)
'[1488187200.0,1488190800.0,1488190800.0]'
>>> rapidjson.dumps([dt,dtutc, inyourtz], datetime_mode=rapidjson.DATETIME_MODE_UNIX_TIME+rapidjson.DATETIME_MODE_IGNORE_TZ+rapidjson.DATETIME_MODE_UTC)
'[1488190800.0,1488190800.0,1488190800.0]'

    NB the result of the last statement above is made up, it's not currently implemented!

  20. alka2 commented on Feb 28, 2017

    @alka2
    alka2
    on Feb 28, 2017
    Author

    Oh, got what you mean.
    I found the problem. The object that I have is coming from Postgres database using psycopg2 library. As you know we are using ujson to dump these objects. Here is the result:

    >>> import datetime
>>> import ujson
>>> import rapidjson
>>> import psycopg2
>>> def encode_timestamp(obj):
...     if isinstance(obj, datetime.datetime):
...         return int(obj.timestamp())
...     raise TypeError(repr(obj) + " is not JSON serializable")
...
>>> rapidjson.dumps(datetime.datetime(2017, 2, 27, 10, 20, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)), default=encode_timestamp)
'1488190800'
>>> ujson.dumps(datetime.datetime(2017, 2, 27, 10, 20, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)))
'1488190800'
>>> rapidjson.dumps(datetime.datetime(2017, 2, 27, 10, 20, 0), default=encode_timestamp)
'1488208800'
>>> ujson.dumps(datetime.datetime(2017, 2, 27, 10, 20, 0))
'1488190800'
>>>

    If there is without timezone object, ujson treats as UTC time!
    I think your last combination will tell RJ the same thing? Right? but not sure if you need to add this. You may need to follow timestamp to be consistent with this standard library.

  21. alka2 commented on Feb 28, 2017

    @alka2
    alka2
    on Feb 28, 2017
    Author

    I cloned the branch https://github.com/lelit/python-rapidjson/tree/unix-timestamps and built it but seems the DATETIME_MODE_UNIX_TIME is not available.

    >>> rapidjson.dumps(datetime.datetime(2016, 2, 28, 2, 11, 15), datetime_mode=rapidjson.DATETIME_MODE_UNIX_TIME)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'rapidjson' has no attribute 'DATETIME_MODE_UNIX_TIME'

  22. lelit commented on Feb 28, 2017

    @lelit
    lelit
    on Feb 28, 2017
    Contributor

    Are you sure you checked out and recompiled the right branch?

    That constant is definitely there.

  23. alka2 commented on Feb 28, 2017

    @alka2
    alka2
    on Feb 28, 2017
    Author

    My bad :(. Compiled the wrong branch.
    Thanks

  24. lelit commented on Feb 28, 2017

    @lelit
    lelit
    on Feb 28, 2017
    Contributor

    No problem at all! Let me know how it goes!

  25. alka2 commented on Feb 28, 2017

    @alka2
    alka2
    on Feb 28, 2017
    Author

    It went well. The test matches with the current stuff that I have.

  26. lelit
    added a commit that references this issue on Mar 1, 2017

    Rename DATETIME_MODE_UTC to DATETIME_MODE_SHIFT_TO_UTC and add DATETI…

    c93a7c7

  27. lelit commented on Mar 1, 2017

    @lelit
    lelit
    on Mar 1, 2017
    Contributor

    @s-alikarimi, see these tests, hopefully the new DATETIME_MODE_NAIVE_IS_UTC option is what you asked... let me know whenever you can try that branch again.

  28. lelit commented on Mar 2, 2017

    @lelit
    lelit
    on Mar 2, 2017
    Contributor

    Assuming some positive feedback on this issue, I will propose the inclusion of the related changes.

    As said, due to some sloppyness on my part, they are built on top of other unrelated changes, in particular a new number_mode option (supersedes PR #50) and a new global defaults feature. If there is any objection I will rebase them in a different branch, otherwise they will land together in a future 0.1.0.

  29. lelit
    added a commit that references this issue on Mar 7, 2017

    Handle “POSIX time” serialization, targeting issue python-rapidjson#61

    6a6c290

  30. lelit commented on Jul 3, 2017

    @lelit
    lelit
    on Jul 3, 2017
    Contributor

    The new feature has been merged in the master branch.

  31. lelit
    closed this as completedon Jul 3, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions