[elbe-devel] [PATCH v2 17/27] log: filter out ansi sequences for the soap logger

Fri Sep 25 08:40:53 CEST 2020

On Thu, Sep 24, 2020 at 06:45:59PM +0200, Bastian Germann wrote:
> Am 24.09.20 um 16:56 schrieb Torben Hohn:
> > spyne wsgi chokes on ansi color sequences in strings, because lxml
> > does not like them.
> > 
> > in the log below, the W: lines are color coded yellow.
> > 
> > -------------------------------------------------------------------------------------------------------------------------------------------------
> > [CMD] pbuilder --create --configfile "/var/cache/elbe/6e569880-0c95-4463-9c11-ff358ab73d3c/pbuilderrc" --aptconfdir "/var/cache/elbe/6e569880-0c95-4463-9c11-ff358ab73d3c/aptconfdir" --debootstrapopts --include="git gnupg"
> > W: /root/.pbuilderrc does not exist
> > W: cgroups are not available on the host, not using them.
> > ERROR:spyne.util.coopmt:Exception in coroutine
> > ERROR:spyne.util.coopmt:All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
> > Traceback (most recent call last):
> >   File "/usr/lib/python3/dist-packages/spyne/util/coopmt.py", line 51, in start
> >     next(ret)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 856, in _get_members_etree
> >     sub_name)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 525, in to_parent
> >     return handler(ctx, cls, inst, parent, ns, *args, **kwargs)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 679, in modelbase_to_parent
> >     elt.text = self.to_unicode(cls, inst)
> >   File "src/lxml/etree.pyx", line 1018, in lxml.etree._Element.text.__set__
> >   File "src/lxml/apihelpers.pxi", line 710, in lxml.etree._setNodeText
> >   File "src/lxml/apihelpers.pxi", line 698, in lxml.etree._createTextNode
> >   File "src/lxml/apihelpers.pxi", line 1493, in lxml.etree._utf8
> > ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
> > ERROR:spyne.server.wsgi:All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
> > Traceback (most recent call last):
> >   File "/usr/lib/python3/dist-packages/spyne/server/wsgi.py", line 452, in handle_rpc
> >     self.get_out_string(p_ctx)
> >   File "/usr/lib/python3/dist-packages/spyne/server/_base.py", line 142, in get_out_string_pull
> >     ret = ctx.out_protocol.serialize(ctx, message=ProtocolBase.RESPONSE)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/soap/soap11.py", line 334, in serialize
> >     out_body_doc, body_message_class.get_namespace())
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 525, in to_parent
> >     return handler(ctx, cls, inst, parent, ns, *args, **kwargs)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 890, in complex_to_parent
> >     add_type)
> >   File "/usr/lib/python3/dist-packages/spyne/util/coopmt.py", line 51, in start
> >     next(ret)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 745, in gen_members_parent
> >     ret = self._get_members_etree(ctx, cls, inst, elt)
> >   File "/usr/lib/python3/dist-packages/spyne/util/coopmt.py", line 51, in start
> >     next(ret)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 856, in _get_members_etree
> >     sub_name)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 525, in to_parent
> >     return handler(ctx, cls, inst, parent, ns, *args, **kwargs)
> >   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 679, in modelbase_to_parent
> >     elt.text = self.to_unicode(cls, inst)
> >   File "src/lxml/etree.pyx", line 1018, in lxml.etree._Element.text.__set__
> >   File "src/lxml/apihelpers.pxi", line 710, in lxml.etree._setNodeText
> >   File "src/lxml/apihelpers.pxi", line 698, in lxml.etree._createTextNode
> >   File "src/lxml/apihelpers.pxi", line 1493, in lxml.etree._utf8
> > ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
> > -------------------------------------------------------------------------------------------------------------------------------------------------
> > 
> > Filter out ANSI sequences in the async_logging code, before passing them through
> > the soap interface.
> > 
> > Maybe we should do it a bit later, to catch all occurences.
> > 
> > Signed-off-by: Torben Hohn <torben.hohn at linutronix.de>
> > ---
> >  elbepack/log.py | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/elbepack/log.py b/elbepack/log.py
> > index 22f747dab..db1fb770b 100644
> > --- a/elbepack/log.py
> > +++ b/elbepack/log.py
> > @@ -8,6 +8,8 @@ import collections
> >  import logging
> >  import os
> >  import threading
> > +import re
> > +
> >  from contextlib import contextmanager
> >  
> >  root = logging.getLogger()
> > @@ -263,7 +265,12 @@ class AsyncLogging(object):
> >  
> >              # Log the line now for echo back
> >              if cnt:
> > -                self.stream.info("\n".join(self.lines[-cnt:]))
> > +                logbuf = "\n".join(self.lines[-cnt:])
> > +
> > +                # filter out ansi sequences.
> > +                logbuf = re.sub('\u001b\[.*?[@-~]', '', logbuf)
> > +
> > +                self.stream.info(logbuf)
> >  
> >              # Keep rest for next line buffering
> >              rest = buf[j:]
> > 
> The better way would be configuring the logging to not color the output.
> Anyway:
> Acked-by: Bastian Germann <bage at linutronix.de>

i have thought about this.
But we would need to do this for all command invocations.
Or we would have to setup TERM env to "something" ?
And we would still have to hope for all invoked commands to obey this.

And also see that later commit with CTRL-H. I doubt that we can get
mkfs.ext4 to stop emitting that....

See:
https://github.com/tytso/e2fsprogs/blob/master/lib/ext2fs/progress.c#L52
we could turn it off. but i am not sure if that works for all
incarnations of mkfs.<fs>

Another alternative would be to use a binary transport method.
http://spyne.io/docs/2.10/reference/model/binary.html#spyne.model.binary.ByteArray

But IIRC that is not compatible with suds, and it just boils down to a
base64 transport like our files stuff.

> _______________________________________________
> elbe-devel mailing list
> elbe-devel at linutronix.de
> https://lists.linutronix.de/mailman/listinfo/elbe-devel

-- 
Torben Hohn
Linutronix GmbH | Bahnhofstrasse 3 | D-88690 Uhldingen-Mühlhofen
Phone: +49 7556 25 999 18; Fax.: +49 7556 25 999 99

Hinweise zum Datenschutz finden Sie hier (Informations on data privacy 
can be found here): https://linutronix.de/kontakt/Datenschutz.php

Linutronix GmbH | Firmensitz (Registered Office): Uhldingen-Mühlhofen | 
Registergericht (Registration Court): Amtsgericht Freiburg i.Br., HRB700 
806 | Geschäftsführer (Managing Directors): Heinz Egger, Thomas Gleixner