Skip to content

fix logging to truncate large textual/binary values #2837

Closed
@sqlalchemy-bot

Description

@sqlalchemy-bot
Collaborator

Migrated issue, originally created by Anonymous

(original reporter: jonathan) The FAQ on logging

 http://docs.sqlalchemy.org/en/rel_0_8/faq.html?highlight=logging#how-do-i-configure-logging

sent me to the Engine logging docs:

   http://docs.sqlalchemy.org/en/rel_0_8/core/engines.html#dbengine-logging

I wanted to suggest this addition to the docs :


Sometimes you might want to disable existing query logging for certain
statements, such as when you are inserting a BLOB or large TEXT value.

An easy way to accomplish this is to wrap your query in a pair of statements
that toggle the disabled attribute of the logger.

In the example below, note that the logger needed to disable already-enabled
query logging is sqlalchemy.engine.base.Engine , and not the
sqlalchemy.engine logger mentioned above.

import logging
logging.getLogger('sqlalchemy.engine.base.Engine').disabled = True
... execute query ...
logging.getLogger('sqlalchemy.engine.base.Engine').disabled = False    

Activity

sqlalchemy-bot

sqlalchemy-bot commented on Oct 11, 2013

@sqlalchemy-bot
CollaboratorAuthor

Michael Bayer (@zzzeek) wrote:

why don't we fix logging not to log out the full expanse of long binary/text fields ? we've already made similar enhancements long ago regarding long parameter sets.

the doc suggestion is a little bit of a very specific recipe.

sqlalchemy-bot

sqlalchemy-bot commented on Oct 11, 2013

@sqlalchemy-bot
CollaboratorAuthor

Michael Bayer (@zzzeek) wrote:

though for FAQ, its OK sure. though logging should work better I think.

sqlalchemy-bot

sqlalchemy-bot commented on Oct 11, 2013

@sqlalchemy-bot
CollaboratorAuthor

Anonymous wrote:

(original author: jonathan) Limiting the size of these fields logging is a much better idea.

It took me a while to dig out this general approach, and it seemed like a natural "need" by the current system... and also not in-line with the current docs. ( I did not expect the logger path to be different like that, though it does make sense ).

My dev box uses SqlAlchemy as the message broker for Celery Resizing (it's handled by Redis on Production ). All of the sudden during some tests, I'm starting to see a 1.9MB jpeg converted into b64 and piped into an INSERT statement on sqlalchemy. Not fun.

sqlalchemy-bot

sqlalchemy-bot commented on Oct 12, 2013

@sqlalchemy-bot
CollaboratorAuthor

Changes by Michael Bayer (@zzzeek):

  • removed labels: documentation, easy
  • added labels: engine
  • set milestone to "0.8.xx"
  • changed title from "Logging FAQ suggestion" to "fix logging to truncate large textual/binary value"
sqlalchemy-bot

sqlalchemy-bot commented on Dec 30, 2013

@sqlalchemy-bot
CollaboratorAuthor

Changes by Michael Bayer (@zzzeek):

  • changed milestone from "0.8.xx" to "0.9.xx"
sqlalchemy-bot

sqlalchemy-bot commented on Apr 14, 2014

@sqlalchemy-bot
CollaboratorAuthor

Changes by Michael Bayer (@zzzeek):

  • edited description
sqlalchemy-bot

sqlalchemy-bot commented on Apr 14, 2014

@sqlalchemy-bot
CollaboratorAuthor

Changes by Michael Bayer (@zzzeek):

  • changed milestone from "0.9.xx" to "sprintable"
sqlalchemy-bot

sqlalchemy-bot commented on Jul 5, 2014

@sqlalchemy-bot
CollaboratorAuthor

Changes by Michael Bayer (@zzzeek):

  • changed milestone from "sprintable" to "0.9.xx"
sqlalchemy-bot

sqlalchemy-bot commented on Aug 25, 2014

@sqlalchemy-bot
CollaboratorAuthor

Changes by Michael Bayer (@zzzeek):

  • changed milestone from "0.9.xx" to "1.0.xx"
sqlalchemy-bot

sqlalchemy-bot commented on Apr 27, 2015

@sqlalchemy-bot
CollaboratorAuthor

Changes by Michael Bayer (@zzzeek):

  • changed milestone from "1.0.xx" to "1.1"
sqlalchemy-bot

sqlalchemy-bot commented on Feb 17, 2016

@sqlalchemy-bot
CollaboratorAuthor

Michael Bayer (@zzzeek) wrote:

  • All string formatting of bound parameter sets and result rows for
    logging, exception, and repr() purposes now truncate very large
    scalar values within each collection, including an
    "N characters truncated"
    notation, similar to how the display for large multiple-parameter sets
    are themselves truncated.
    fixes fix logging to truncate large textual/binary values #2837

591e0cf

sqlalchemy-bot

sqlalchemy-bot commented on Feb 17, 2016

@sqlalchemy-bot
CollaboratorAuthor

Changes by Michael Bayer (@zzzeek):

  • changed status to closed
added
engineengines, connections, transactions, isolation levels, execution options
on Nov 27, 2018

1 remaining item

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    engineengines, connections, transactions, isolation levels, execution optionsfeature

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @sqlalchemy-bot

        Issue actions

          fix logging to truncate large textual/binary values · Issue #2837 · sqlalchemy/sqlalchemy