Skip to content

defaults with decimal places in them are reported differently on MariaDB, need to add decimal point for reflection #5744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Redysz opened this issue Dec 3, 2020 · 6 comments
Labels
bug Something isn't working mariadb mysql question issue where a "fix" on the SQLAlchemy side is unlikely, hence more of a usage question reflection reflection of tables, columns, constraints, defaults, sequences, views, everything else
Milestone

Comments

@Redysz
Copy link

Redysz commented Dec 3, 2020

Suppose I have table main_table with columns:

id (AUTO_INCREMENT), first (int), value1 (double with default=0), value2 (double)

Using code

engine = sqlalchemy.create_engine(db_connect)
session = Session(engine)
base = automap_base()
base.prepare(engine, reflect=True)
table = getattr(base.classes, fact_table)
kwargs = {'first': 200016, 'value2': 1.0}
entry = table(**kwargs)
session.add(entry)
session.commit()

I got an error {"error": 500, "message": "(pymysql.err.IntegrityError) (1048, \"Column 'value1' cannot be null\")

It happens when value1 doesn't accept null value.

I'd like to ignore this and add entry after all value1 is not obligatory (has default).

Versions.

  • OS: Windows 10
  • Python: 3.8
  • SQLAlchemy: 1.3.20
  • Database: 10.5.5

Edit:
MariaDB 10.5.5

SHOW CREATE TABLE main_table;:

CREATE TABLE `main_table` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `first` int(11) NOT NULL,
  `value1` double(22,6) NOT NULL DEFAULT 0.000000,
  `value2` double(22,6) NOT NULL,
  PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=200000 DEFAULT CHARSET=utf8

db_connect = 'mysql+pymysql://[...]'
(fact_table is just a string with "main_table" inside)

@Redysz Redysz added the requires triage New issue that requires categorization label Dec 3, 2020
@zzzeek zzzeek added question issue where a "fix" on the SQLAlchemy side is unlikely, hence more of a usage question and removed requires triage New issue that requires categorization labels Dec 3, 2020
@zzzeek
Copy link
Member

zzzeek commented Dec 3, 2020

hi there -

let me illustrate a proper MCVE for the behavior you describe. note also that the issue template requests that you tell us what kind of database you're using; "10.5.5" seems to be a version number and does not tell us what database this is.

Below is an illustration of the table given and your test script; it works without error as automap reflects that the "value1" column has a server default on it, so you will note it's not included in the INSERT. Please run this test case and feel free to modify it, including database in use, to illustrate how you are getting your results. thanks!

from sqlalchemy import create_engine
from sqlalchemy import table
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session


engine = create_engine("sqlite://", echo=True)

with engine.begin() as conn:
    conn.execute(
        """
        CREATE TABLE fact_table (
                id INTEGER PRIMARY KEY,
                first INTEGER,
                value1 DOUBLE DEFAULT 0,
                value2 DOUBLE
        )
"""
    )


session = Session(engine)
base = automap_base()
base.prepare(engine, reflect=True)
table = getattr(base.classes, "fact_table")
kwargs = {"first": 200016, "value2": 1.0}
entry = table(**kwargs)
session.add(entry)
session.commit()

output, after reflection queries are run:

2020-12-03 08:24:36,341 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2020-12-03 08:24:36,341 INFO sqlalchemy.engine.base.Engine INSERT INTO fact_table (first, value2) VALUES (?, ?)
2020-12-03 08:24:36,341 INFO sqlalchemy.engine.base.Engine (200016, 1.0)
2020-12-03 08:24:36,342 INFO sqlalchemy.engine.base.Engine COMMIT

@zzzeek zzzeek added awaiting info waiting for the submitter to give more information cant reproduce labels Dec 3, 2020
@Redysz
Copy link
Author

Redysz commented Dec 3, 2020

Hello @zzzeek , thank you for your reply. I added information about differences between our codes. Check it now and tell me if I should provide something more.

@zzzeek
Copy link
Member

zzzeek commented Dec 3, 2020

hey there -

just for future reference, the database you're using is MariaDB as the issue does not exist on MySQL, these are important details that help us to isolate the issue.

@zzzeek zzzeek changed the title Default value doesn't work in sqlalchemy automap defaults with decimal places in them are reported differently on MariaDB, need to add decimal point for reflection Dec 3, 2020
@zzzeek zzzeek added bug Something isn't working mariadb mysql reflection reflection of tables, columns, constraints, defaults, sequences, views, everything else and removed awaiting info waiting for the submitter to give more information cant reproduce labels Dec 3, 2020
@zzzeek zzzeek added this to the 1.3.x milestone Dec 3, 2020
@zzzeek
Copy link
Member

zzzeek commented Dec 3, 2020

for workaround you will need to provide a partial model with this column present:

base = automap_base()

class main_table(base):
    __tablename__ = "main_table"

    id = Column(Integer, primary_key=True)
    value1 = Column(Numeric, server_default="0.000")

base.prepare(engine, reflect=True)
table = main_table
kwargs = {"first": 200016, "value2": 1.0}
entry = table(**kwargs)
session.add(entry)
session.commit()

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the master branch:

Reflect decimal points in MariaDB non-quoted numeric defaults https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/2397

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the rel_1_3 branch:

Reflect decimal points in MariaDB non-quoted numeric defaults https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/2398

sqlalchemy-bot pushed a commit that referenced this issue Dec 3, 2020
Fixed issue where reflecting a server default on MariaDB only that
contained a decimal point in the value would fail to be reflected
correctly, leading towards a reflected table that lacked any server
default.

Fixes: #5744
Change-Id: Ifc5960928685a906558ba84ed6f59eecb3b1c358
(cherry picked from commit 7fb41cf)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working mariadb mysql question issue where a "fix" on the SQLAlchemy side is unlikely, hence more of a usage question reflection reflection of tables, columns, constraints, defaults, sequences, views, everything else
Projects
None yet
Development

No branches or pull requests

3 participants