[elbe-devel] [PATCH v2] soapclient: workaround socket exception

[Torben Hohn]

On Fri, Sep 21, 2018 at 10:23:58AM +0200, Manuel Traut wrote:
> maybe the reset occurs, because the initvm is to busy to answer the SOAP
> request in time. Try to ignore this issue by retrying getting the same
> part of the log.
> 
> 17:32:00 2018-08-30 15:29:09 - INFO     - | Traceback (most recent call last):
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "./elbe", line 55, in <module>
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     cmdmod.run_command(sys.argv[2:])
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/home/elbe/elbepack/commands/control.py", line 161, in run_command
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     action.execute(control, opt, args[1:])
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/home/elbe/elbepack/soapclient.py", line 571, in execute
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     busy = client.service.get_project_busy(builddir, part)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 566, in __call__
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     return client.invoke(args, kwargs)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 705, in invoke
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     result = self.send(soapenv)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 747, in send
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     reply = self.options.transport.send(request)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/https.py", line 66, in send
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     return HttpTransport.send(self, request)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 80, in send
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     fp = self.u2open(u2request)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 127, in u2open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     return url.open(u2request, timeout=tm)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 429, in open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     response = self._open(req, data)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 447, in _open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     '_open', req)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     result = func(*args)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     return self.do_open(httplib.HTTPConnection, req)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     r = h.getresponse(buffering=True)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     response.begin()
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 438, in begin
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     version, status, reason = self._read_status()
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     line = self.fp.readline(_MAXLINE + 1)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/socket.py", line 480, in readline
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     data = self._sock.recv(self._rbufsize)
> 17:32:00 2018-08-30 15:29:09 - INFO     - | socket.error: [Errno 104] Connection reset by peer
> 17:32:00 2018-08-30 15:29:09 - INFO     - | WARNING: exit code 1 from a shell command.
> 17:32:00 2018-08-30 15:29:09 - INFO     - | ERROR: Function failed: do_rootfs (log file is located at /home/build/tmp-glibc/work/cortexa9hf-neon-oe-linux-gnueabi/image1/1.0-r0/temp/log.do_rootfs.10761)
> 
> Signed-off-by: Manuel Traut <manut at linutronix.de>
> ---
> 
> changes since v1:
> 
>   * limit retry on socket.errno == 104
>   * added TODO in source
>   * print execption to stderr instead of stdout
> 
>  elbepack/soapclient.py | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/elbepack/soapclient.py b/elbepack/soapclient.py
> index 25ba4090..5a8ef7f1 100644
> --- a/elbepack/soapclient.py
> +++ b/elbepack/soapclient.py
> @@ -585,7 +585,21 @@ class WaitProjectBusyAction(ClientAction):
>          part = 1
>  
>          while True:
> -            busy = client.service.get_project_busy(builddir, part)
> +            try:
> +                busy = client.service.get_project_busy(builddir, part)
> +            # TODO the root cause of this problem is unclear. To enable a
> +            # get more information print the exception and retry to see if
> +            # the connection problem is just a temporary problem. This
> +            # code should be reworked as soon as it's clear what is going on
> +            # here
> +            except socket.error as e:
> +                print(e.message, file=sys.stderr)
> +                if e.errno == 104:
> +                    break

if e.errno != 104:
	raise

this is what i meant.


> +                print("socket error during wait busy occured, retry..",
> +                      file=sys.stderr)
> +                continue
> +
>              if busy == 'FINISH':
>                  break
>              else:
> -- 
> 2.19.0
> 

-- 
Torben Hohn
Linutronix GmbH | Bahnhofstrasse 3 | D-88690 Uhldingen-Mühlhofen
Phone: +49 7556 25 999 18; Fax.: +49 7556 25 999 99

Hinweise zum Datenschutz finden Sie hier (Informations on data privacy 
can be found here): https://linutronix.de/kontakt/Datenschutz.php

Linutronix GmbH | Firmensitz (Registered Office): Uhldingen-Mühlhofen | 
Registergericht (Registration Court): Amtsgericht Freiburg i.Br., HRB700 
806 | Geschäftsführer (Managing Directors): Heinz Egger, Thomas Gleixner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.linutronix.de/pipermail/elbe-devel/attachments/20180921/d035635c/attachment-0001.sig>