Skip to content

aggressive adaption in column properties #8168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zzzeek opened this issue Jun 23, 2022 Discussed in #8167 · 14 comments
Closed

aggressive adaption in column properties #8168

zzzeek opened this issue Jun 23, 2022 Discussed in #8167 · 14 comments
Labels
bug Something isn't working hard orm hard ORM bugs mike needs to look at inheritance issues to do wtih ORM inheritance, a particularly tricky area orm
Milestone

Comments

@zzzeek
Copy link
Member

zzzeek commented Jun 23, 2022

Discussed in #8167

from sqlalchemy import Column, Integer, String, ForeignKey, select, func
from sqlalchemy.orm import declarative_base, relationship, column_property, Session, aliased


Base = declarative_base()

class Customer(Base):
    __tablename__ = 'customer'
    id = Column(Integer, primary_key=True)
    type = Column(String(20))

    __mapper_args__ = {
        'polymorphic_on':'type',
        'polymorphic_identity':'customer'
    }

class Store(Customer):
    __tablename__ = "store"
    id = Column(Integer, ForeignKey('customer.id'), primary_key=True)
    retailer_id = Column(Integer, ForeignKey('retailer.id'))
    retailer = relationship(
        "Retailer",
        back_populates="stores",
        lazy="noload",
        foreign_keys=[retailer_id]
    )

    __mapper_args__ = {
        'polymorphic_identity':'store'
    }


# works
sa = Store.__table__.alias()

# FromClause polymorphic adapter gets it, even though it has no ORM
# annotations
sa = Store.__table__

class Retailer(Customer):
    __tablename__ = "retailer"
    id = Column(Integer, ForeignKey('customer.id'), primary_key=True)
    stores = relationship(
        "Store",
        back_populates="retailer",
        foreign_keys=[Store.retailer_id],
        lazy="noload"
    )
    store_count = column_property(
        select(func.count(sa.c.id)).
        where(sa.c.id==id).
        correlate_except(sa).
        scalar_subquery()
    )

    __mapper_args__ = {
        'polymorphic_identity':'retailer'
    }



print(select(Retailer))

the query is:

SELECT (SELECT count(customer.id) AS count_1 
WHERE customer.id = retailer.id) AS anon_1, retailer.id, customer.id AS id_1, customer.type 
FROM customer JOIN retailer ON customer.id = retailer.id

that is even though we are using an unannotated Store.__table__ an adapter is still hitting it.

this patch overselects for this case, meaning, it breaks hundreds of inheritance tests, but this is where it goes wrong:

diff --git a/lib/sqlalchemy/orm/context.py b/lib/sqlalchemy/orm/context.py
index a468244e9a..29eb08c7d4 100644
--- a/lib/sqlalchemy/orm/context.py
+++ b/lib/sqlalchemy/orm/context.py
@@ -460,7 +460,10 @@ class ORMCompileState(CompileState):
             for mp in ext_info.mapper.iterate_to_root():
                 self._mapper_loads_polymorphically_with(
                     mp,
-                    sql_util.ColumnAdapter(selectable, mp._equivalent_columns),
+                    sql_util.ColumnAdapter(selectable, mp._equivalent_columns,
+                         # "_orm_adapt" or "parententity" or whatever
+                        exclude_fn=lambda elem: "parententity" not in elem._annotations if isinstance(elem, sql.ColumnElement) else False
+                    ),
                 )
 
     def _mapper_loads_polymorphically_with(self, mapper, adapter):

to make that work we'd have to figure out all the other columns that are coming in there unannotated that we do want to be adapted and figure out how to split them out. it looks like it's just PolyUnions and PolyAliasedJoins so maybe it's not that far reaching.

@zzzeek zzzeek added bug Something isn't working orm inheritance issues to do wtih ORM inheritance, a particularly tricky area labels Jun 23, 2022
@zzzeek zzzeek added this to the 2.0 milestone Jun 23, 2022
@zzzeek
Copy link
Member Author

zzzeek commented Jun 23, 2022

then again all the columns that are mapped dont have annotations in them with a classical mapping... will have to look at this to see what can be done

@zzzeek
Copy link
Member Author

zzzeek commented Jun 23, 2022

more to the point, the column_property() is against Store.id and this mapper is Retailer. So...why is Store.id in the translate here? this patch adjusts that, but again breaks those same polymorphic rel tests, but this is more appropriate in any case so maybe find a way for this to change behavior based on the with polymorphic

diff --git a/lib/sqlalchemy/orm/mapper.py b/lib/sqlalchemy/orm/mapper.py
index 0241a31232..2e0fd2e78b 100644
--- a/lib/sqlalchemy/orm/mapper.py
+++ b/lib/sqlalchemy/orm/mapper.py
@@ -2839,7 +2839,7 @@ class Mapper(
                 else:
                     result[binary.right] = {binary.left}
 
-        for mapper in self.base_mapper.self_and_descendants:
+        for mapper in self.iterate_to_root():
             if mapper.inherit_condition is not None:
                 visitors.traverse(
                     mapper.inherit_condition, {}, {"binary": visit_binary}

@zzzeek
Copy link
Member Author

zzzeek commented Jun 23, 2022

like if the root mapper is with poly on a UNION of all the tables, then we're out again. kind of a weird problem

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the main branch:

WIP: try to limit wpoly adapt in column_property https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3947

@zzzeek zzzeek modified the milestones: 2.0, 2.0beta Sep 17, 2022
@zzzeek
Copy link
Member Author

zzzeek commented Dec 4, 2022

the test case here was fixed by #8456. Need to determine more closely

@zzzeek
Copy link
Member Author

zzzeek commented Dec 4, 2022

indeed, #8456 lists this as a regression in 1.4, the test case also succeeds on 1.3 with the requisite API calling styles.

@zzzeek
Copy link
Member Author

zzzeek commented Dec 4, 2022

the user's test in #8167 for their original POC still fails in the way they mention, if they call like this:

print(Retailer.store_count.expression)

however that's the col prop without anything to correlate towards, this does work

print(select(Retailer.store_count).select_from(Retailer))

so that seems OK.

@zzzeek
Copy link
Member Author

zzzeek commented Dec 4, 2022

The original case they have "works" but not exactly because Store is the join from Customer to Store, so they get this:

SELECT (SELECT count(store.id) AS count_1 
FROM customer JOIN store ON customer.id = store.id 
WHERE store.retailer_id = retailer.id) AS anon_1 
FROM customer JOIN retailer ON customer.id = retailer.id

whereas using just the table gets this:

SELECT (SELECT count(store.id) AS count_1 
FROM store 
WHERE store.retailer_id = retailer.id) AS anon_1 
FROM customer JOIN retailer ON customer.id = retailer.id

those are still the correct SQL, in the first case the JOIN is expected.

EDIT: now I am not getting this result at all. so not sure.

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the main branch:

Add tests for issue #8168 https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4264

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the rel_1_4 branch:

Add tests for issue #8168 https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4265

@zzzeek
Copy link
Member Author

zzzeek commented Dec 4, 2022

OK, the issue still exists if we use polymorphic_load="inline". exists fully, they have to use an alias of the table to work around. Still happens with the base table.

this bug is still opened, then, though is not that critical. will commit the test cases.

@sqla-tester
Copy link
Collaborator

Mike Bayer referenced this issue:

Add tests for issue #8168 https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4264

@sqla-tester
Copy link
Collaborator

Mike Bayer referenced this issue:

Add tests for issue #8168 https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4265

@zzzeek zzzeek modified the milestones: 2.0 final, 1.4.x Dec 4, 2022
@zzzeek zzzeek added the hard orm hard ORM bugs mike needs to look at label Dec 4, 2022
sqlalchemy-bot pushed a commit that referenced this issue Dec 5, 2022
The issue in #8168 was improved, but not completely fixed,
by #8456.

This includes some small changes to ORM context that
are a prerequisite for getting ORM adaptation to be
better.   Have these in 2.0.0b4 so that we have at
least a better starting point.

References: #8168
Change-Id: I51dbe333b156048836d074fbba1d850f9eb67fd2
@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the main branch:

disable polymorphic adaption in most cases https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4275

@zzzeek zzzeek modified the milestones: 1.4.x, 2.0beta5 Dec 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working hard orm hard ORM bugs mike needs to look at inheritance issues to do wtih ORM inheritance, a particularly tricky area orm
Projects
None yet
Development

No branches or pull requests

2 participants