Skip to content

Can't recover after restarting MySQL #4945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yoch opened this issue Oct 24, 2019 · 6 comments
Closed

Can't recover after restarting MySQL #4945

yoch opened this issue Oct 24, 2019 · 6 comments
Labels
bug Something isn't working connection pool external driver issues the issue involves a misbehavior on the part of the DBAPI itself, probably not SQLAlchemy mysql
Milestone

Comments

@yoch
Copy link

yoch commented Oct 24, 2019

Hi,

I use the lastest stable version of SQLAlchemy (1.3.10) in a production system, and lately I noticed that restarting the MySQL server cause big troubles in our application. SQLAlchemy cannot recover until we restart the WSGI application.

This seems to be a regression, because I can't remember having this issue earlier.

Additional informations

We are using the pymysql driver. The server version is Mariadb 10.2.27

Here is the errlog:

[Thu Oct 24 13:17:49.853596 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203] Traceback (most recent call last):
[Thu Oct 24 13:17:49.853663 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
[Thu Oct 24 13:17:49.853683 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     return fn()
[Thu Oct 24 13:17:49.853697 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
[Thu Oct 24 13:17:49.853712 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     return _ConnectionFairy._checkout(self)
[Thu Oct 24 13:17:49.853727 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/pool/base.py", line 791, in _checkout
[Thu Oct 24 13:17:49.853742 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     result = pool._dialect.do_ping(fairy.connection)
[Thu Oct 24 13:17:49.853756 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 138, in do_ping
[Thu Oct 24 13:17:49.853771 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     dbapi_connection.ping(False)
[Thu Oct 24 13:17:49.853805 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 547, in ping
[Thu Oct 24 13:17:49.853819 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     self._read_ok_packet()
[Thu Oct 24 13:17:49.853833 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 394, in _read_ok_packet
[Thu Oct 24 13:17:49.853847 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     pkt = self._read_packet()
[Thu Oct 24 13:17:49.853860 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 684, in _read_packet
[Thu Oct 24 13:17:49.853874 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     packet.check_error()
[Thu Oct 24 13:17:49.853887 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/protocol.py", line 220, in check_error
[Thu Oct 24 13:17:49.853905 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     err.raise_mysql_exception(self._data)
[Thu Oct 24 13:17:49.853919 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
[Thu Oct 24 13:17:49.853933 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     raise errorclass(errno, errval)
[Thu Oct 24 13:17:49.853946 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203] pymysql.err.InternalError: (1927, 'Connection was killed')
[Thu Oct 24 13:17:49.853960 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203] 
[Thu Oct 24 13:17:49.853973 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203] The above exception was the direct cause of the following exception:
[Thu Oct 24 13:17:49.853986 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203] 
[Thu Oct 24 13:17:49.853999 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203] Traceback (most recent call last):
[Thu Oct 24 13:17:49.854013 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/bottle.py", line 862, in _handle
[Thu Oct 24 13:17:49.854027 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     return route.call(**args)
[Thu Oct 24 13:17:49.854040 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/bottle.py", line 1740, in wrapper
[Thu Oct 24 13:17:49.854053 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     rv = callback(*a, **ka)
[Thu Oct 24 13:17:49.854067 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/bottle_sqlalchemy.py", line 156, in wrapper
[Thu Oct 24 13:17:49.854080 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     rv = callback(*args, **kwargs)
[Thu Oct 24 13:17:49.854094 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/galien-api/api/session_plugin.py", line 94, in wrapper
[Thu Oct 24 13:17:49.854108 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     with self.db.connect() as conn:
[Thu Oct 24 13:17:49.854121 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2206, in connect
[Thu Oct 24 13:17:49.854142 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     return self._connection_cls(self, **kwargs)
[Thu Oct 24 13:17:49.854155 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 103, in __init__
[Thu Oct 24 13:17:49.854169 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     else engine.raw_connection()
[Thu Oct 24 13:17:49.854182 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2306, in raw_connection
[Thu Oct 24 13:17:49.854195 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     self.pool.unique_connection, _connection
[Thu Oct 24 13:17:49.854209 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect
[Thu Oct 24 13:17:49.854222 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     e, dialect, self
[Thu Oct 24 13:17:49.854240 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1544, in _handle_dbapi_exception_noconnection
[Thu Oct 24 13:17:49.854254 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     util.raise_from_cause(sqlalchemy_exception, exc_info)
[Thu Oct 24 13:17:49.854291 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
[Thu Oct 24 13:17:49.854312 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     reraise(type(exception), exception, tb=exc_tb, cause=cause)
[Thu Oct 24 13:17:49.854330 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
[Thu Oct 24 13:17:49.854348 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     raise value.with_traceback(tb)
[Thu Oct 24 13:17:49.854365 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
[Thu Oct 24 13:17:49.854384 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     return fn()
[Thu Oct 24 13:17:49.854400 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
[Thu Oct 24 13:17:49.854414 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     return _ConnectionFairy._checkout(self)
[Thu Oct 24 13:17:49.854427 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/pool/base.py", line 791, in _checkout
[Thu Oct 24 13:17:49.854441 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     result = pool._dialect.do_ping(fairy.connection)
[Thu Oct 24 13:17:49.854454 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 138, in do_ping
[Thu Oct 24 13:17:49.854468 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     dbapi_connection.ping(False)
[Thu Oct 24 13:17:49.854481 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 547, in ping
[Thu Oct 24 13:17:49.854494 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     self._read_ok_packet()
[Thu Oct 24 13:17:49.854514 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 394, in _read_ok_packet
[Thu Oct 24 13:17:49.854528 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     pkt = self._read_packet()
[Thu Oct 24 13:17:49.854541 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 684, in _read_packet
[Thu Oct 24 13:17:49.854555 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     packet.check_error()
[Thu Oct 24 13:17:49.854568 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/protocol.py", line 220, in check_error
[Thu Oct 24 13:17:49.854582 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     err.raise_mysql_exception(self._data)
[Thu Oct 24 13:17:49.854602 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]   File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
[Thu Oct 24 13:17:49.854616 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203]     raise errorclass(errno, errval)
[Thu Oct 24 13:17:49.854629 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203] sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1927, 'Connection was killed')
[Thu Oct 24 13:17:49.854643 2019] [wsgi:error] [pid 29865:tid 140251745441536] [remote 78.250.16.86:58203] (Background on this error at: http://sqlalche.me/e/2j85)
[Thu Oct 24 13:17:54.095035 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204] Traceback (most recent call last):
[Thu Oct 24 13:17:54.095373 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
[Thu Oct 24 13:17:54.095521 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     return fn()
[Thu Oct 24 13:17:54.095675 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
[Thu Oct 24 13:17:54.095809 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     return _ConnectionFairy._checkout(self)
[Thu Oct 24 13:17:54.095928 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/pool/base.py", line 791, in _checkout
[Thu Oct 24 13:17:54.096081 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     result = pool._dialect.do_ping(fairy.connection)
[Thu Oct 24 13:17:54.096213 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 138, in do_ping
[Thu Oct 24 13:17:54.096335 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     dbapi_connection.ping(False)
[Thu Oct 24 13:17:54.096475 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 547, in ping
[Thu Oct 24 13:17:54.096600 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     self._read_ok_packet()
[Thu Oct 24 13:17:54.096707 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 394, in _read_ok_packet
[Thu Oct 24 13:17:54.096795 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     pkt = self._read_packet()
[Thu Oct 24 13:17:54.096898 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 684, in _read_packet
[Thu Oct 24 13:17:54.096983 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     packet.check_error()
[Thu Oct 24 13:17:54.097066 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/protocol.py", line 220, in check_error
[Thu Oct 24 13:17:54.097150 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     err.raise_mysql_exception(self._data)
[Thu Oct 24 13:17:54.097233 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
[Thu Oct 24 13:17:54.097316 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     raise errorclass(errno, errval)
[Thu Oct 24 13:17:54.097400 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204] pymysql.err.InternalError: (1927, 'Connection was killed')
[Thu Oct 24 13:17:54.097483 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204] 
[Thu Oct 24 13:17:54.097565 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204] The above exception was the direct cause of the following exception:
[Thu Oct 24 13:17:54.097648 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204] 
[Thu Oct 24 13:17:54.097729 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204] Traceback (most recent call last):
[Thu Oct 24 13:17:54.097811 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/bottle.py", line 862, in _handle
[Thu Oct 24 13:17:54.097895 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     return route.call(**args)
[Thu Oct 24 13:17:54.097978 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/bottle.py", line 1740, in wrapper
[Thu Oct 24 13:17:54.098061 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     rv = callback(*a, **ka)
[Thu Oct 24 13:17:54.098143 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/bottle_sqlalchemy.py", line 156, in wrapper
[Thu Oct 24 13:17:54.098228 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     rv = callback(*args, **kwargs)
[Thu Oct 24 13:17:54.098310 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/galien-api/api/session_plugin.py", line 94, in wrapper
[Thu Oct 24 13:17:54.098394 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     with self.db.connect() as conn:
[Thu Oct 24 13:17:54.098476 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2206, in connect
[Thu Oct 24 13:17:54.098560 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     return self._connection_cls(self, **kwargs)
[Thu Oct 24 13:17:54.098642 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 103, in __init__
[Thu Oct 24 13:17:54.098726 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     else engine.raw_connection()
[Thu Oct 24 13:17:54.098808 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2306, in raw_connection
[Thu Oct 24 13:17:54.098893 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     self.pool.unique_connection, _connection
[Thu Oct 24 13:17:54.098985 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect
[Thu Oct 24 13:17:54.099071 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     e, dialect, self
[Thu Oct 24 13:17:54.099153 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1544, in _handle_dbapi_exception_noconnection
[Thu Oct 24 13:17:54.099241 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     util.raise_from_cause(sqlalchemy_exception, exc_info)
[Thu Oct 24 13:17:54.099334 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
[Thu Oct 24 13:17:54.099419 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     reraise(type(exception), exception, tb=exc_tb, cause=cause)
[Thu Oct 24 13:17:54.099502 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
[Thu Oct 24 13:17:54.099586 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     raise value.with_traceback(tb)
[Thu Oct 24 13:17:54.100739 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 2275, in _wrap_pool_connect
[Thu Oct 24 13:17:54.100777 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     return fn()
[Thu Oct 24 13:17:54.100792 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
[Thu Oct 24 13:17:54.100808 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     return _ConnectionFairy._checkout(self)
[Thu Oct 24 13:17:54.100821 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/pool/base.py", line 791, in _checkout
[Thu Oct 24 13:17:54.100835 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     result = pool._dialect.do_ping(fairy.connection)
[Thu Oct 24 13:17:54.100849 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/home/admgalien/.local/lib/python3.5/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 138, in do_ping
[Thu Oct 24 13:17:54.100862 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     dbapi_connection.ping(False)
[Thu Oct 24 13:17:54.100876 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 547, in ping
[Thu Oct 24 13:17:54.100890 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     self._read_ok_packet()
[Thu Oct 24 13:17:54.100903 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 394, in _read_ok_packet
[Thu Oct 24 13:17:54.100917 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     pkt = self._read_packet()
[Thu Oct 24 13:17:54.100958 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 684, in _read_packet
[Thu Oct 24 13:17:54.100973 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     packet.check_error()
[Thu Oct 24 13:17:54.101122 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/protocol.py", line 220, in check_error
[Thu Oct 24 13:17:54.101152 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     err.raise_mysql_exception(self._data)
[Thu Oct 24 13:17:54.101167 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]   File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
[Thu Oct 24 13:17:54.101181 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204]     raise errorclass(errno, errval)
[Thu Oct 24 13:17:54.101194 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204] sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1927, 'Connection was killed')
[Thu Oct 24 13:17:54.101208 2019] [wsgi:error] [pid 29865:tid 140251812648704] [remote 78.250.16.86:58204] (Background on this error at: http://sqlalche.me/e/2j85)
@zzzeek zzzeek added question issue where a "fix" on the SQLAlchemy side is unlikely, hence more of a usage question external driver issues the issue involves a misbehavior on the part of the DBAPI itself, probably not SQLAlchemy mysql labels Oct 24, 2019
@zzzeek
Copy link
Member

zzzeek commented Oct 24, 2019

This is a bug in the PyMySQL driver because it is an InternalError. There are issues that seem to discuss this but it's not clear what behavior PyMySQL has right now, there is PyMySQL/PyMySQL#526 as well as sidorares/node-mysql2#510 , and it's not clear if you are seeing a new behavior is this is a regression in a newer PyMySQL or a fxed bug in an older version, since I don't know what PyMySQL version you are on.

SQLAlchemy detects a "disconnect" based on the type of error raised and the logic for pymysql does not currently expect an InternalError because the driver is not supposed to raise those. nothing has changed in SQLAlchemy so there is no regression here, unless you can provide a reproducing test case that reproduces the error on the given version and does not reproduce it on a specific former version, with no changes in anything else.

advice here would be to restore the previous version of PyMySQL before your upgrade and if it is an older version, to report a regression at https://github.com/PyMySQL/PyMySQL/issues/

@yoch
Copy link
Author

yoch commented Oct 24, 2019

We use pymysql version 0.9.3 (Dec 18, 2018).

It's a little strange, because no fresh upgrade were done on the driver. I'll try to investigate.

@zzzeek
Copy link
Member

zzzeek commented Oct 24, 2019

if PyMySQL devs confirm this is going to stay as InternalError (which would be very unusual), we could add this to the list of "disconnect" exceptions, however, they should fix this on their end.

You can work around this by providing a disconnect handler for this specific error, if you are really in a bind, see the example at https://docs.sqlalchemy.org/en/13/core/events.html?highlight=handle_error#sqlalchemy.events.ConnectionEvents.handle_error . basically is_disconnect indicates that the connection pool will refresh itself.

Or, you can use NullPool to disable connection pooling, or set pool_recycle to a very low number of seconds: https://docs.sqlalchemy.org/en/13/core/pooling.html#pool-setting-recycle

@zzzeek zzzeek added bug Something isn't working and removed question issue where a "fix" on the SQLAlchemy side is unlikely, hence more of a usage question labels Nov 11, 2019
@zzzeek zzzeek added this to the 1.3.xx milestone Nov 11, 2019
@zzzeek
Copy link
Member

zzzeek commented Nov 11, 2019

there are already some buggy errors that we are testing for in both pymysql and mysqldb so will add this.

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the master branch:

Detect PyMySQL connection was killed https://gerrit.sqlalchemy.org/1575

@sqla-tester
Copy link
Collaborator

Mike Bayer has proposed a fix for this issue in the rel_1_3 branch:

Detect PyMySQL connection was killed https://gerrit.sqlalchemy.org/1576

sqlalchemy-bot pushed a commit that referenced this issue Nov 11, 2019
Added "Connection was killed" message interpreted from the base
pymysql.Error class in order to detect closed connection, based on reports
that this message is arriving via a pymysql.InternalError() object which
indicates pymysql is not handling it correctly.

Change-Id: If6bbe0eb5993e1996c0c5de752eebaf7446cf93e
References: PyMySQL/PyMySQL#816
Fixes: #4945
(cherry picked from commit 31a66b5)
@zzzeek zzzeek modified the milestones: 1.3.xx, 1.3.x Dec 18, 2019
sumau pushed a commit to sumau/sqlalchemy that referenced this issue Dec 29, 2019
Added "Connection was killed" message interpreted from the base
pymysql.Error class in order to detect closed connection, based on reports
that this message is arriving via a pymysql.InternalError() object which
indicates pymysql is not handling it correctly.

Change-Id: If6bbe0eb5993e1996c0c5de752eebaf7446cf93e
References: PyMySQL/PyMySQL#816
Fixes: sqlalchemy#4945
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working connection pool external driver issues the issue involves a misbehavior on the part of the DBAPI itself, probably not SQLAlchemy mysql
Projects
None yet
Development

No branches or pull requests

3 participants