Skip to content

add second-pass type inference step to TypeEngine, DateTime #7537

@cake-monotone

Description

@cake-monotone

Describe the bug

When I built a query with offset-aware datetime object (timezone datetime), it failed with DBAPIError

I think it's for two reasons.

  • sqlalchemy considers offset-aware-datetime's sqltype to be DateTime(). It should be DateTime(timezone=True).
  • asyncpg doesn't allow that offset-aware-datetime input to naive-datetime placeholder.

More simply, sqlalchemy try to build following query. but asyncpg doesn't allow

# asyncpg
stmt = await conn.prepare('SELECT $1::timestamp')
await stmt.fetchval(offset_aware_datetime)

It should be

# asyncpg
stmt = await conn.prepare('SELECT $1::timestamp with time zone')
await stmt.fetchval(offset_aware_datetime)

To Reproduce

import asyncio
import datetime

import sqlalchemy
from sqlalchemy.ext.asyncio import create_async_engine

async def main():
    offset_aware_datetime = datetime.datetime.now(tz=datetime.timezone.utc)

    db_engine = create_async_engine('postgresql+asyncpg://postgres:example@localhost:5432/postgres')

    async with db_engine.connect() as conn:
        # ERROR!!
        result = (
            await conn.execute(
                sqlalchemy.select(
                    sqlalchemy.literal(offset_aware_datetime)
                )
            )
        ).scalar()

asyncio.run(main())

Error

Traceback (most recent call last):
  File "asyncpg/protocol/prepared_stmt.pyx", line 168, in asyncpg.protocol.protocol.PreparedStatementState._encode_bind_msg
  File "asyncpg/protocol/codecs/base.pyx", line 206, in asyncpg.protocol.protocol.Codec.encode
  File "asyncpg/protocol/codecs/base.pyx", line 111, in asyncpg.protocol.protocol.Codec.encode_scalar
  File "asyncpg/pgproto/./codecs/datetime.pyx", line 152, in asyncpg.pgproto.pgproto.timestamp_encode
TypeError: can't subtract offset-naive and offset-aware datetimes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 420, in _prepare_and_execute
    self._rows = await prepared_stmt.fetch(*parameters)
  File "/home/cake/.local/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
    data = await self.__bind_execute(args, 0, timeout)
  File "/home/cake/.local/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
    data, status, _ = await self.__do_execute(
  File "/home/cake/.local/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
    return await executor(protocol)
  File "asyncpg/protocol/protocol.pyx", line 183, in bind_execute
  File "asyncpg/protocol/prepared_stmt.pyx", line 197, in asyncpg.protocol.protocol.PreparedStatementState._encode_bind_msg
asyncpg.exceptions.DataError: invalid input for query argument $1: datetime.datetime(2022, 1, 5, 14, 42, 49... (can't subtract offset-naive and offset-aware datetimes)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 457, in execute
    self._adapt_connection.await_(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
    return current.driver.switch(awaitable)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
    value = await result
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 432, in _prepare_and_execute
    self._handle_exception(error)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 367, in _handle_exception
    self._adapt_connection._handle_exception(error)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 660, in _handle_exception
    raise translated_error from error
sqlalchemy.dialects.postgresql.asyncpg.Error: <class 'asyncpg.exceptions.DataError'>: invalid input for query argument $1: datetime.datetime(2022, 1, 5, 14, 42, 49... (can't subtract offset-naive and offset-aware datetimes)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "t.py", line 24, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "t.py", line 15, in main
    await conn.execute(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/engine.py", line 457, in execute
    result = await greenlet_spawn(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 134, in greenlet_spawn
    result = context.throw(*sys.exc_info())
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
    self._handle_dbapi_exception(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
    util.raise_(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 457, in execute
    self._adapt_connection.await_(
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
    return current.driver.switch(awaitable)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
    value = await result
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 432, in _prepare_and_execute
    self._handle_exception(error)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 367, in _handle_exception
    self._adapt_connection._handle_exception(error)
  File "/home/cake/.local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 660, in _handle_exception
    raise translated_error from error
sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.DataError'>: invalid input for query argument $1: datetime.datetime(2022, 1, 5, 14, 42, 49... (can't subtract offset-naive and offset-aware datetimes)
[SQL: SELECT %s AS anon_1]
[parameters: (datetime.datetime(2022, 1, 5, 14, 42, 49, 972486, tzinfo=datetime.timezone.utc),)]
(Background on this error at: https://sqlalche.me/e/14/dbapi)

Versions

  • OS: Ubuntu
  • Python: 3.8.10
  • SQLAlchemy: 1.4.29
  • Database: postgresql
  • DBAPI (eg: psycopg, cx_oracle, mysqlclient): asyncpg

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdatatypesthings to do with database types, like VARCHAR and others

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions