Skip to content

Commit 6a6c290

Browse files
lelitlelit
committed
Handle “POSIX time” serialization, targeting issue python-rapidjson#61
The datetime_mode argument is now a bitset, composed by different DATETIME_MODE_XXX values: for example the previous DATETIME_MODE_ISO8601_UTC argument is now DATETIME_MODE_ISO8601|DATETIME_MODE_SHIFT_TO_UTC. A new DATETIME_MODE_UNIX_TIME mode will serialize datetimes, dates and times as plain numbers, either floats or ints (controlled by DATETIME_MODE_ONLY_SECONDS) for datetimes it's the same as executing their timestamp() method. A new option DATETIME_MODE_NAIVE_IS_UTC, combinable with either DATETIME_MODE_ISO8601 and DATETIME_MODE_UNIX_TIME, forces the module to assume that naïve values (those without an explicit timezone) are actually in UTC.
1 parent acce989 commit 6a6c290

3 files changed

Lines changed: 523 additions & 177 deletions

File tree

‎docs/api.rst‎

Lines changed: 187 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -18,24 +18,49 @@
1818

1919
.. data:: DATETIME_MODE_NONE
2020

21-
This is the default ``datetime_mode``: *neither* :class:`datetime` *nor*
22-
:class:`date` instances are recognized by :func:`dumps` and :func:`loads`.
21+
This is the default ``datetime_mode``: *neither* :class:`date` *nor*
22+
:class:`datetime` *nor* :class:`time` instances are recognized by
23+
:func:`dumps` and :func:`loads`.
2324

2425
.. data:: DATETIME_MODE_ISO8601
2526

2627
In this ``datetime_mode`` mode :func:`dumps` and :func:`loads` handle
27-
:class:`datetime` *and* :class:`date` instances representing those values
28-
using the `ISO 8601`_ format.
28+
:class:`date`, :class:`datetime` *and* :class:`date` instances representing
29+
those values using the `ISO 8601`_ format.
2930

30-
.. data:: DATETIME_MODE_ISO8601_IGNORE_TZ
31+
.. data:: DATETIME_MODE_UNIX_TIME
3132

32-
This is like :data:`DATETIME_MODE_ISO8601` except that the value's timezone
33-
is ignored.
33+
This mode tells RapidJSON to serialize :class:`date`, :class:`datetime` and
34+
:class:`time` as *numeric timestamps*: for the formers it is exactly the
35+
result of their :meth:`timestamp` method, that is as the number of seconds
36+
passed since ``EPOCH``; ``date`` instances are serialized as the
37+
corresponding ``datetime`` instance with all the `time` slots set to 0;
38+
``time`` instances are serialized as the number of seconds since midnight.
3439

35-
.. data:: DATETIME_MODE_ISO8601_UTC
40+
Since this is obviously *irreversible*, this flag is usable **only** for
41+
:func:`dumps`: an error is raised when passed to :func:`loads`.
3642

37-
This is like :data:`DATETIME_MODE_ISO8601` except that the times are always
38-
*shifted* to the UTC_ timezone.
43+
.. data:: DATETIME_MODE_ONLY_SECONDS
44+
45+
This is usable in combination with :data:`DATETIME_MODE_UNIX_TIME` so that
46+
an integer representation is used, ignoring *microseconds*.
47+
48+
.. data:: DATETIME_MODE_IGNORE_TZ
49+
50+
This can be used combined with :data:`DATETIME_MODE_ISO8601` or
51+
:data:`DATETIME_MODE_UNIX_TIME`, to ignore the value's timezones.
52+
53+
.. data:: DATETIME_MODE_NAIVE_IS_UTC
54+
55+
This can be used combined with :data:`DATETIME_MODE_ISO8601` or
56+
:data:`DATETIME_MODE_UNIX_TIME`, to tell RapidJSON that the naïve values
57+
(that is, without an explicit timezone) are already in UTC_ timezone.
58+
59+
.. data:: DATETIME_MODE_SHIFT_TO_UTC
60+
61+
This can be used combined with :data:`DATETIME_MODE_ISO8601` or
62+
:data:`DATETIME_MODE_UNIX_TIME`, to always *shift* values the UTC_
63+
timezone.
3964

4065
.. data:: UUID_MODE_NONE
4166

@@ -58,9 +83,12 @@
5883

5984
.. testsetup::
6085

61-
from rapidjson import (dumps, loads, DATETIME_MODE_NONE, DATETIME_MODE_ISO8601,
62-
DATETIME_MODE_ISO8601_IGNORE_TZ, DATETIME_MODE_ISO8601_UTC,
63-
UUID_MODE_NONE, UUID_MODE_CANONICAL, UUID_MODE_HEX)
86+
from rapidjson import (dumps, loads,
87+
DATETIME_MODE_NONE, DATETIME_MODE_ISO8601,
88+
DATETIME_MODE_UNIX_TIME, DATETIME_MODE_ONLY_SECONDS,
89+
DATETIME_MODE_IGNORE_TZ, DATETIME_MODE_NAIVE_IS_UTC,
90+
DATETIME_MODE_SHIFT_TO_UTC,
91+
UUID_MODE_NONE, UUID_MODE_CANONICAL, UUID_MODE_HEX)
6492

6593
.. function:: dumps(obj, skipkeys=False, ensure_ascii=True, allow_nan=True, indent=None, \
6694
default=None, sort_keys=False, use_decimal=False, \
@@ -77,8 +105,8 @@
77105
alphabetically
78106
:param bool use_decimal: whether :class:`Decimal` should be handled
79107
:param int max_recursion_depth: maximum depth for nested structures
80-
:param int datetime_mode: how should :class:`datetime` and :class:`date`
81-
instances be handled
108+
:param int datetime_mode: how should :class:`datetime`, :class:`time` and
109+
:class:`date` instances be handled
82110
:param int uuid_mode: how should :class:`UUID` instances be handled
83111
:returns: A Python :class:`str` instance.
84112

@@ -219,48 +247,101 @@
219247
File "<stdin>", line 1, in <module>
220248
OverflowError: Max recursion depth reached
221249

222-
By default :class:`date` and :class:`datetime` instances are not
223-
serializable. When `datetime_mode` is set to :data:`DATETIME_MODE_ISO8601`
250+
By default :class:`date`, :class:`datetime` and :class:`time` instances are
251+
not serializable. When `datetime_mode` is set to :data:`DATETIME_MODE_ISO8601`
224252
those values are serialized using the common `ISO 8601`_ format:
225253

226254
.. doctest::
227255

228-
>>> from datetime import date, datetime
229-
>>> today = date.today()
230-
>>> right_now = datetime.now()
231-
>>> dumps({'date': today, 'timestamp': right_now})
256+
>>> from datetime import datetime
257+
>>> right_now = datetime(2016, 8, 28, 13, 14, 52, 277256)
258+
>>> date = right_now.date()
259+
>>> time = right_now.time()
260+
>>> dumps({'date': date, 'time': time, 'timestamp': right_now})
232261
Traceback (most recent call last):
233262
File "<stdin>", line 1, in <module>
234263
TypeError: datetime(…) is not JSON serializable
235-
>>> dumps({'a date': today, 'a timestamp': right_now},
236-
... datetime_mode=DATETIME_MODE_ISO8601) # doctest: +SKIP
237-
'{"timestamp":"2016-08-28T13:14:52.277256","date":"2016-08-28"}'
264+
>>> dumps(['date', date, 'time', time, 'timestamp', right_now],
265+
... datetime_mode=DATETIME_MODE_ISO8601)
266+
'["date","2016-08-28","time","13:14:52.277256","timestamp","2016-08-28T13:14:52.277256"]'
238267

239-
Another mode is :data:`DATETIME_MODE_ISO8601_UTC`, that *shifts* all
240-
timestamps to the UTC_ timezone before serializing them:
268+
The `right_now` value is a naïve datetime (because it does not carry the
269+
timezone information) and is normally assumed to be in the local timezone,
270+
whatever your system thinks it is. When you instead *know* that your value,
271+
even being naïve are actually in the UTC_ timezone, you can use the
272+
:data:`DATETIME_MODE_NAIVE_IS_UTC` flag to inform RapidJSON about that:
273+
274+
.. doctest::
275+
276+
>>> mode = DATETIME_MODE_ISO8601 | DATETIME_MODE_NAIVE_IS_UTC
277+
>>> dumps(['time', time, 'timestamp', right_now], datetime_mode=mode)
278+
'["time","13:14:52.277256+00:00","timestamp","2016-08-28T13:14:52.277256+00:00"]'
279+
280+
A variant is :data:`DATETIME_MODE_SHIFT_TO_UTC`, that *shifts* all datetime
281+
values to the UTC_ timezone before serializing them:
241282

242283
.. doctest::
243284

244285
>>> from datetime import timedelta, timezone
245286
>>> here = timezone(timedelta(hours=2))
246-
>>> now = datetime.now(here)
247-
>>> dumps(now)
248-
Traceback (most recent call last):
249-
File "<stdin>", line 1, in <module>
250-
TypeError: datetime.datetime(…) is not JSON serializable
251-
>>> dumps(now, datetime_mode=DATETIME_MODE_ISO8601) # doctest: +SKIP
287+
>>> now = datetime(2016, 8, 28, 20, 31, 11, 84418, here)
288+
>>> dumps(now, datetime_mode=DATETIME_MODE_ISO8601)
252289
'"2016-08-28T20:31:11.084418+02:00"'
253-
>>> dumps(now, datetime_mode=DATETIME_MODE_ISO8601_UTC) # doctest: +SKIP
290+
>>> mode = DATETIME_MODE_ISO8601 | DATETIME_MODE_SHIFT_TO_UTC
291+
>>> dumps(now, datetime_mode=mode)
254292
'"2016-08-28T18:31:11.084418+00:00"'
255293

256-
With :data:`DATETIME_MODE_ISO8601_IGNORE_TZ` the timezone, if present, is
257-
simply omitted:
294+
With :data:`DATETIME_MODE_IGNORE_TZ` the timezone, if present, is simply
295+
omitted:
258296

259297
.. doctest::
260298

261-
>>> dumps(now, datetime_mode=DATETIME_MODE_ISO8601_IGNORE_TZ) # doctest: +SKIP
299+
>>> mode = DATETIME_MODE_ISO8601 | DATETIME_MODE_IGNORE_TZ
300+
>>> dumps(now, datetime_mode=mode)
262301
'"2016-08-28T20:31:11.084418"'
263302

303+
Another :ref:`one-way only <no-unix-time-loads>` alternative format is
304+
`Unix time`_: with :data:`DATETIME_MODE_UNIX_TIME` :class:`date`,
305+
:class:`datetime` and :class:`time` instances are serialized as a number of
306+
seconds, respectively since the ``EPOCH`` for the first two kinds and since
307+
midnight for the latter:
308+
309+
.. doctest::
310+
311+
>>> mode = DATETIME_MODE_UNIX_TIME
312+
>>> dumps([now, now.date(), now.time()], datetime_mode=mode)
313+
'[1472409071.084418,1472335200.0,73871.084418]'
314+
>>> unixtime = float(dumps(now, datetime_mode=mode))
315+
>>> datetime.fromtimestamp(unixtime, here) == now
316+
True
317+
318+
Combining it with the :data:`DATETIME_MODE_ONLY_SECONDS` will produce
319+
integer values instead, dropping *microseconds*:
320+
321+
.. doctest::
322+
323+
>>> mode = DATETIME_MODE_UNIX_TIME | DATETIME_MODE_ONLY_SECONDS
324+
>>> dumps([now, now.date(), now.time()], datetime_mode=mode)
325+
'[1472409071,1472335200,73871]'
326+
327+
It can be used combined with :data:`DATETIME_MODE_SHIFT_TO_UTC` to obtain the
328+
timestamp of the corresponding UTC_ time:
329+
330+
>>> mode = DATETIME_MODE_UNIX_TIME | DATETIME_MODE_SHIFT_TO_UTC
331+
>>> dumps(now, datetime_mode=mode)
332+
'1472409071.084418'
333+
334+
As above, when you know that your values are in the UTC_ timezone, you can use the
335+
:data:`DATETIME_MODE_NAIVE_IS_UTC` flag to get the right result:
336+
337+
.. doctest::
338+
339+
>>> a_long_time_ago = datetime(1968, 3, 18, 9, 10, 0, 0)
340+
>>> mode = DATETIME_MODE_UNIX_TIME | DATETIME_MODE_NAIVE_IS_UTC
341+
>>> dumps([a_long_time_ago, a_long_time_ago.date(), a_long_time_ago.time()],
342+
... datetime_mode=mode)
343+
'[-56472600.0,-56505600.0,33000.0]'
344+
264345
Likewise, to handle :class:`UUID` instances there are two modes that can be
265346
specified with the `uuid_mode` argument, that will use the string
266347
representation of their values:
@@ -289,8 +370,8 @@
289370
:param bool use_decimal: whether :class:`Decimal` should be used for float
290371
values
291372
:param bool allow_nan: whether ``NaN`` values are recognized
292-
:param int datetime_mode: how should :class:`datetime` and :class:`date`
293-
instances be handled
373+
:param int datetime_mode: how should :class:`date`, :class:`datetime` and
374+
:class:`time` instances be handled
294375
:param int uuid_mode: how should :class:`UUID` instances be handled
295376
:returns: An equivalent Python object.
296377

@@ -341,8 +422,8 @@
341422
ValueError: … Out of range float values are not JSON compliant
342423

343424
With `datetime_mode` you can enable recognition of string literals
344-
containing an `ISO 8601`_ representation as either :class:`date` or
345-
:class:`datetime` instances:
425+
containing an `ISO 8601`_ representation as either :class:`date`,
426+
:class:`datetime` or :class:`time` instances:
346427

347428
.. doctest::
348429

@@ -351,14 +432,73 @@
351432
>>> loads('"2016-01-02T01:02:03+01:00"',
352433
... datetime_mode=DATETIME_MODE_ISO8601)
353434
datetime.datetime(2016, 1, 2, 1, 2, 3, tzinfo=...delta(0, 3600)))
354-
>>> loads('"2016-01-02T01:02:03+01:00"',
355-
... datetime_mode=DATETIME_MODE_ISO8601_UTC)
356-
datetime.datetime(2016, 1, 2, 0, 2, 3, tzinfo=...utc)
357-
>>> loads('"2016-01-02T01:02:03+01:00"',
358-
... datetime_mode=DATETIME_MODE_ISO8601_IGNORE_TZ)
359-
datetime.datetime(2016, 1, 2, 1, 2, 3)
360435
>>> loads('"2016-01-02"', datetime_mode=DATETIME_MODE_ISO8601)
361436
datetime.date(2016, 1, 2)
437+
>>> loads('"01:02:03+01:00"',
438+
... datetime_mode=DATETIME_MODE_ISO8601)
439+
datetime.time(1, 2, 3, tzinfo=...delta(0, 3600)))
440+
441+
It can be combined with :data:`DATETIME_MODE_SHIFT_TO_UTC` to *always*
442+
obtain values in the UTC_ timezone:
443+
444+
.. doctest::
445+
446+
>>> mode = DATETIME_MODE_ISO8601 | DATETIME_MODE_SHIFT_TO_UTC
447+
>>> loads('"2016-01-02T01:02:03+01:00"', datetime_mode=mode)
448+
datetime.datetime(2016, 1, 2, 0, 2, 3, tzinfo=...utc)
449+
450+
.. note::
451+
452+
This option is somewhat limited when the value is a non-naïve time literal
453+
because negative values cannot be represented by the underlying Python
454+
type, so it cannot adapt such values reliably:
455+
456+
.. doctest::
457+
458+
>>> mode = DATETIME_MODE_ISO8601 | DATETIME_MODE_SHIFT_TO_UTC
459+
>>> loads('"00:01:02+00:00"', datetime_mode=mode)
460+
datetime.time(0, 1, 2, tzinfo=...utc)
461+
>>> loads('"00:01:02+01:00"', datetime_mode=mode)
462+
Traceback (most recent call last):
463+
...
464+
ValueError: ... Time literal cannot be shifted to UTC: 00:01:02+01:00
465+
466+
If you combine it with :data:`DATETIME_MODE_NAIVE_IS_UTC` then all values
467+
without a timezone will be assumed to be relative to UTC_:
468+
469+
.. doctest::
470+
471+
>>> mode = DATETIME_MODE_ISO8601 | DATETIME_MODE_NAIVE_IS_UTC
472+
>>> loads('"2016-01-02T01:02:03"', datetime_mode=mode)
473+
datetime.datetime(2016, 1, 2, 1, 2, 3, tzinfo=...utc)
474+
>>> loads('"2016-01-02T01:02:03+01:00"', datetime_mode=mode)
475+
datetime.datetime(2016, 1, 2, 1, 2, 3, tzinfo=...delta(0, 3600)))
476+
>>> loads('"01:02:03"', datetime_mode=mode)
477+
datetime.time(1, 2, 3, tzinfo=...utc)
478+
479+
Yet another combination is with :data:`DATETIME_MODE_IGNORE_TZ` to ignore
480+
the timezone and obtain naïve values:
481+
482+
.. doctest::
483+
484+
>>> mode = DATETIME_MODE_ISO8601 | DATETIME_MODE_IGNORE_TZ
485+
>>> loads('"2016-01-02T01:02:03+01:00"', datetime_mode=mode)
486+
datetime.datetime(2016, 1, 2, 1, 2, 3)
487+
>>> loads('"01:02:03+01:00"', datetime_mode=mode)
488+
datetime.time(1, 2, 3)
489+
490+
.. _no-unix-time-loads:
491+
492+
The :data:`DATETIME_MODE_UNIX_TIME` cannot be used here, because there
493+
isn't a reasonable heuristic to disambiguate between plain numbers and
494+
timestamps:
495+
496+
.. doctest::
497+
498+
>>> loads('[1,2,3]', datetime_mode=DATETIME_MODE_UNIX_TIME)
499+
Traceback (most recent call last):
500+
File "<stdin>", line 1, in <module>
501+
ValueError: Invalid datetime_mode, can deserialize only from ISO8601
362502

363503
With `uuid_mode` you can enable recognition of string literals containing
364504
two different representations of :class:`UUID` values:
@@ -384,3 +524,4 @@
384524
.. _ISO 8601: https://en.wikipedia.org/wiki/ISO_8601
385525
.. _RapidJSON: https://github.com/miloyip/rapidjson
386526
.. _UTC: https://en.wikipedia.org/wiki/Coordinated_Universal_Time
527+
.. _Unix time: https://en.wikipedia.org/wiki/Unix_time

0 commit comments

Comments
 (0)