[elbe-devel] [PATCH v2 17/27] log: filter out ansi sequences for the soap logger

[Bastian Germann]

Am 24.09.20 um 16:56 schrieb Torben Hohn:
> spyne wsgi chokes on ansi color sequences in strings, because lxml
> does not like them.
> 
> in the log below, the W: lines are color coded yellow.
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------
> [CMD] pbuilder --create --configfile "/var/cache/elbe/6e569880-0c95-4463-9c11-ff358ab73d3c/pbuilderrc" --aptconfdir "/var/cache/elbe/6e569880-0c95-4463-9c11-ff358ab73d3c/aptconfdir" --debootstrapopts --include="git gnupg"
> W: /root/.pbuilderrc does not exist
> W: cgroups are not available on the host, not using them.
> ERROR:spyne.util.coopmt:Exception in coroutine
> ERROR:spyne.util.coopmt:All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/spyne/util/coopmt.py", line 51, in start
>     next(ret)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 856, in _get_members_etree
>     sub_name)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 525, in to_parent
>     return handler(ctx, cls, inst, parent, ns, *args, **kwargs)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 679, in modelbase_to_parent
>     elt.text = self.to_unicode(cls, inst)
>   File "src/lxml/etree.pyx", line 1018, in lxml.etree._Element.text.__set__
>   File "src/lxml/apihelpers.pxi", line 710, in lxml.etree._setNodeText
>   File "src/lxml/apihelpers.pxi", line 698, in lxml.etree._createTextNode
>   File "src/lxml/apihelpers.pxi", line 1493, in lxml.etree._utf8
> ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
> ERROR:spyne.server.wsgi:All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/spyne/server/wsgi.py", line 452, in handle_rpc
>     self.get_out_string(p_ctx)
>   File "/usr/lib/python3/dist-packages/spyne/server/_base.py", line 142, in get_out_string_pull
>     ret = ctx.out_protocol.serialize(ctx, message=ProtocolBase.RESPONSE)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/soap/soap11.py", line 334, in serialize
>     out_body_doc, body_message_class.get_namespace())
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 525, in to_parent
>     return handler(ctx, cls, inst, parent, ns, *args, **kwargs)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 890, in complex_to_parent
>     add_type)
>   File "/usr/lib/python3/dist-packages/spyne/util/coopmt.py", line 51, in start
>     next(ret)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 745, in gen_members_parent
>     ret = self._get_members_etree(ctx, cls, inst, elt)
>   File "/usr/lib/python3/dist-packages/spyne/util/coopmt.py", line 51, in start
>     next(ret)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 856, in _get_members_etree
>     sub_name)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 525, in to_parent
>     return handler(ctx, cls, inst, parent, ns, *args, **kwargs)
>   File "/usr/lib/python3/dist-packages/spyne/protocol/xml.py", line 679, in modelbase_to_parent
>     elt.text = self.to_unicode(cls, inst)
>   File "src/lxml/etree.pyx", line 1018, in lxml.etree._Element.text.__set__
>   File "src/lxml/apihelpers.pxi", line 710, in lxml.etree._setNodeText
>   File "src/lxml/apihelpers.pxi", line 698, in lxml.etree._createTextNode
>   File "src/lxml/apihelpers.pxi", line 1493, in lxml.etree._utf8
> ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
> -------------------------------------------------------------------------------------------------------------------------------------------------
> 
> Filter out ANSI sequences in the async_logging code, before passing them through
> the soap interface.
> 
> Maybe we should do it a bit later, to catch all occurences.
> 
> Signed-off-by: Torben Hohn <torben.hohn at linutronix.de>
> ---
>  elbepack/log.py | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/elbepack/log.py b/elbepack/log.py
> index 22f747dab..db1fb770b 100644
> --- a/elbepack/log.py
> +++ b/elbepack/log.py
> @@ -8,6 +8,8 @@ import collections
>  import logging
>  import os
>  import threading
> +import re
> +
>  from contextlib import contextmanager
>  
>  root = logging.getLogger()
> @@ -263,7 +265,12 @@ class AsyncLogging(object):
>  
>              # Log the line now for echo back
>              if cnt:
> -                self.stream.info("\n".join(self.lines[-cnt:]))
> +                logbuf = "\n".join(self.lines[-cnt:])
> +
> +                # filter out ansi sequences.
> +                logbuf = re.sub('\u001b\[.*?[@-~]', '', logbuf)
> +
> +                self.stream.info(logbuf)
>  
>              # Keep rest for next line buffering
>              rest = buf[j:]
> 
The better way would be configuring the logging to not color the output.
Anyway:
Acked-by: Bastian Germann <bage at linutronix.de>