Skip to content

maridab client libs also report utf8mb3 in some way , need to handle it #7136

Closed
@zzzeek

Description

@zzzeek
Member

continuing from #7115 it seems likely that the new Mariadb 10.6 client is also returniung "utf8mb3" for client encoding , which will break for SQLAlchemy prior to the #7115 commit because we try to use that encoding at the Python level. we see this via failure on CI:

[jenkins@41dceb4d72b3 bce7ae67]$ .tox/py39-cext-mysql/bin/python -m pytest \
\         --dburi 'mysql+mysqldb://scott:tiger@mariadb105/test?charset=utf8mb4' \
s         test/dialect/mysql/test_dialect.py::DialectTest::test_special_encodings
============================= test session starts ==============================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/.tox/py39-cext-mysql/bin/python
cachedir: .pytest_cache
rootdir: /home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67, configfile: setup.cfg
plugins: forked-1.3.0, xdist-2.4.0
collected 1 item                                                               

test/dialect/mysql/test_dialect.py::DialectTest::test_special_encodings FAILED [100%]

=================================== FAILURES ===================================
______________________ DialectTest.test_special_encodings ______________________
Traceback (most recent call last):
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/dialect/mysql/test_dialect.py", line 260, in test_special_encodings
    eq_(conn.dialect._connection_charset, enc)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/testing/assertions.py", line 244, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: 'utf8mb3' != 'utf8'
assert 'utf8mb3' == 'utf8'
  - utf8
  + utf8mb3
=========================== short test summary info ============================
FAILED test/dialect/mysql/test_dialect.py::DialectTest::test_special_encodings
============================== 1 failed in 0.09s ===============================

on CI, mariadb client is installed from maridb's RPMs:

# rpm -qa | grep -i mariadb
MariaDB-client-10.6.4-1.el8.x86_64
MariaDB-shared-10.6.4-1.el8.x86_64
MariaDB-devel-10.6.4-1.el8.x86_64
MariaDB-common-10.6.4-1.el8.x86_64

code that breaks with "utf8mb3" is at

_encoding_compat = {
as this lookup will fail. also need to update that test.

Activity

added this to the 1.4.x milestone on Oct 2, 2021
zzzeek

zzzeek commented on Oct 2, 2021

@zzzeek
MemberAuthor

on 1.4.25 on CI, stack trace:

test/dialect/mysql/test_dialect.py::DialectTest::test_special_encodings FAILED [100%]

=================================== FAILURES ===================================
______________________ DialectTest.test_special_encodings ______________________
Traceback (most recent call last):
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/dialect/mysql/test_dialect.py", line 259, in test_special_encodings
    conn = eng.connect()
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/engine/base.py", line 3194, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/engine/base.py", line 3273, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/engine/base.py", line 3240, in _wrap_pool_connect
    return fn()
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/pool/base.py", line 310, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/pool/base.py", line 868, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/pool/base.py", line 476, in checkout
    rec = pool._do_get()
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/pool/impl.py", line 146, in _do_get
    self._dec_overflow()
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/pool/base.py", line 256, in _create_connection
    return _ConnectionRecord(self)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/pool/base.py", line 371, in __init__
    self.__connect()
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/pool/base.py", line 677, in __connect
    pool.dispatch.connect.for_modify(
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/event/attr.py", line 329, in _exec_w_sync_on_first_run
    self(*args, **kw)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/event/attr.py", line 343, in __call__
    fn(*args, **kw)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/util/langhelpers.py", line 1691, in go
    return once_fn(*arg, **kw)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/engine/create.py", line 686, in first_connect
    dialect.initialize(c)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/dialects/mysql/base.py", line 2954, in initialize
    self._detect_sql_mode(connection)
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/dialects/mysql/base.py", line 3411, in _detect_sql_mode
    self._sql_mode = row[1] or ""
  File "/home/jenkins/workspace/sqlalchemy_gerrit/bce7ae67/test/../lib/sqlalchemy/dialects/mysql/base.py", line 3519, in __getitem__
    return item.decode(self.charset)
LookupError: unknown encoding: utf8mb3
=========================== short test summary info ============================
FAILED test/dialect/mysql/test_dialect.py::DialectTest::test_special_encodings
============================== 1 failed in 0.09s ===============================

we should likely backport to 1.3 unfortunately

sqla-tester

sqla-tester commented on Oct 2, 2021

@sqla-tester
Collaborator

Mike Bayer has proposed a fix for this issue in the master branch:

support utf8mb3 char encoding fully for mysqlclient, others https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3125

sqla-tester

sqla-tester commented on Oct 2, 2021

@sqla-tester
Collaborator

Mike Bayer has proposed a fix for this issue in the rel_1_3 branch:

support utf8mb3 char encoding fully for mysqlclient, others https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/3126

added a commit that references this issue on Oct 2, 2021
68cb964
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @zzzeek@sqla-tester

        Issue actions

          maridab client libs also report utf8mb3 in some way , need to handle it · Issue #7136 · sqlalchemy/sqlalchemy