populate_existing=True makes unloaded previously loaded relations
#13221
Replies: 4 comments 18 replies
-
|
i didnt look deeply yet but if you're using populate_existing that's sort of what it's going to do. are you trying to use populate_existing and have it surgically avoid a particular relationship? im not sure it works that way. you might want to use session.expire() in a surgical way then do a re-load, or maybe use session.refresh() similarly. |
Beta Was this translation helpful? Give feedback.
-
It's may be that column are handled differently here.
So you want to provide possibly nested loaders too? because otherwise it's just a lazy load like what you would get using awaitable_attrs |
Beta Was this translation helpful? Give feedback.
-
|
I eventually figured out why this was happening, but it ultimately feels odd. I'm sure most people expect loaded relationships to be preserved in the session, not thrown away due to technical details like a weak-reference. top = await sess.get(Top, '1', options=[joinedload(Top.right), joinedload(Top.left)])
# save references to relationships so that they remain in the session
# after they are unloaded from the top-orm.
left, right = top.left, top.right
await sess.get(Top, '1', options=[load_only(Top.id)], populate_existing=True)
# relationships are unloaded now
print(inspect(top).unloaded) # {'right', 'left'}
# but available, since they are in session
print(top.left, top.right) |
Beta Was this translation helpful? Give feedback.
-
Yes, that's the idea. Another case: the relation contains a couple of meta fields and one large jsonb; in some cases only the meta data is needed, and in another - jsonb. |
Beta Was this translation helpful? Give feedback.
-
I'm not sure, it depends on what Overall I think it's more similar to the second one than to the first one at the moment, and it seems like more in line with the expectation? note that maybe some more docs may make sense note that if you keep a reference to the data, you are not really changing how the get works, it is working since a lazy-load is firing but there is no sql needed and you have |
Beta Was this translation helpful? Give feedback.
-
|
@zzzeek what's your though on this? |
Beta Was this translation helpful? Give feedback.
-
In any case, such a mechanic is also required. |
Beta Was this translation helpful? Give feedback.
-
|
maybe, but it may be very complex do implement for arbitrarily nested relationship paths. |
Beta Was this translation helpful? Give feedback.
-
it should raise an exception only if sql-required and it raises, just in case - |
Beta Was this translation helpful? Give feedback.
-
I think something like this is quite enough, at least to start with: # save references to relationships in a temp-dict so that they remain in the session
# after they are unloaded from the top-orm
loaded_relations: dict[str, Base] = get_loaded_relations_map(top)
await sess.get(Top, '1', options=[load_only(Top.id)], populate_existing=True)
# now get loaded relations back in top
for rel in inspect(top).unloaded:
if rel in loaded_relations:
getattr(top, rel) |
Beta Was this translation helpful? Give feedback.
-
|
In your code sure, but that's not something that would be added in SQLAlchemy. For example this doesn't work with lazy=raise |
Beta Was this translation helpful? Give feedback.
-
|
Here's an example that highlights the problem more clearly: from sqlalchemy import Column, String, ForeignKey, select
from sqlalchemy.orm import declarative_base, relationship, sessionmaker, joinedload, Mapped, load_only
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
Base = declarative_base()
class Top(Base):
__tablename__ = 'top'
id = Column(String(), primary_key=True)
right: Mapped['Right'] = relationship(lambda: Right, uselist=False, lazy='raise_on_sql')
left: Mapped['Left'] = relationship(lambda: Left, uselist=False, lazy='raise_on_sql')
class Right(Base):
__tablename__ = 'right'
id = Column(String(), ForeignKey(Top.id), primary_key=True,)
class Left(Base):
__tablename__ = 'left'
id = Column(String(), ForeignKey(Top.id), primary_key=True,)
down: Mapped['Left'] = relationship(lambda: LeftDown, uselist=False, lazy='raise_on_sql')
class LeftDown(Base):
__tablename__ = 'left_down'
id = Column(String(), ForeignKey(Left.id), primary_key=True,)
async_engine = create_async_engine("sqlite+aiosqlite://", pool_pre_ping=True)
AsyncSessionFactory = sessionmaker(
bind=async_engine, class_=AsyncSession, expire_on_commit=True, autoflush=False
)
async def create_tables() -> None:
async with async_engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
async def test():
# create related records
async with AsyncSessionFactory() as sess:
top = Top(id='1')
right = Right(id=top.id)
left = Left(id=top.id)
down = LeftDown(id=top.id)
sess.add_all([top, right, left, down])
await sess.commit()
# start business logic
async with AsyncSessionFactory() as sess:
sess: AsyncSession
# suppose we loaded Left-entity with Left.down-relation in one place of the app.
left = await sess.get(Left, '1', options=[joinedload(Left.down)])
print(left.down)
# in another place we loaded Top-entity with left-relation.
top = await sess.get(Top, '1', options=[joinedload(Top.right), joinedload(Top.left)], populate_existing=True)
# now we returned in the first place
assert left.down # bang! relation is no longer available ... wtf?
async def main():
await create_tables()
await test()
import asyncio
asyncio.run(main()) |
Beta Was this translation helpful? Give feedback.
-
|
your use case here is uncommon, given that this was not requested before. Can't you decide the loader options before loading the object? |
Beta Was this translation helpful? Give feedback.
-
I agree, I met this bug/feature literally yesterday ;)
How?:
|
Beta Was this translation helpful? Give feedback.
-
Please note that in synchronous mode this is almost impossible to catch (with lazy='select'). |
Beta Was this translation helpful? Give feedback.
-
in this use case I guess it's not that easy, in previous examples it may be possible to know what's needed beforehand, or maybe reverse the whole thing, so instead of loading via relationships, the related objectes can be loaded directly filtering using https://docs.sqlalchemy.org/en/20/orm/queryguide/api.html#sqlalchemy.orm.with_parent. Looking again to the example I think this is working as expected, sqlalchemy is loading what you are telling it to, populating existing objects with the new data coming from the db. But I don't know if that's the "spirit" of populate existing or not when it was added. let's wait @zzzeek for this. overall here the best options is to either use two different sessions or ensure that you finish processing the first object before tacking the second one |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
SQLAlchemy 2.0.48
Hello, guys!
I came to you with another gift ;)
Beta Was this translation helpful? Give feedback.
All reactions