[elbe-devel] [PATCH 4/4] soapclient: workaround socket exception

[Torben Hohn]

On Fri, Sep 14, 2018 at 01:53:05PM +0200, Manuel Traut wrote:
> maybe the reset occurs, because the initvm is to busy to answer the SOAP
> request in time. Try to ignore this issue by retrying getting the same
> part of the log.
> 
> 17:32:00 2018-08-30 15:29:09 - INFO     - | Traceback (most recent call last):
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "./elbe", line 55, in <module>
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     cmdmod.run_command(sys.argv[2:])
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/home/elbe/elbepack/commands/control.py", line 161, in run_command
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     action.execute(control, opt, args[1:])
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/home/elbe/elbepack/soapclient.py", line 571, in execute
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     busy = client.service.get_project_busy(builddir, part)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 566, in __call__
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     return client.invoke(args, kwargs)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 705, in invoke
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     result = self.send(soapenv)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 747, in send
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     reply = self.options.transport.send(request)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/https.py", line 66, in send
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     return HttpTransport.send(self, request)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 80, in send
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     fp = self.u2open(u2request)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 127, in u2open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     return url.open(u2request, timeout=tm)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 429, in open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     response = self._open(req, data)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 447, in _open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     '_open', req)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     result = func(*args)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     return self.do_open(httplib.HTTPConnection, req)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     r = h.getresponse(buffering=True)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     response.begin()
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 438, in begin
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     version, status, reason = self._read_status()
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     line = self.fp.readline(_MAXLINE + 1)
> 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/socket.py", line 480, in readline
> 17:32:00 2018-08-30 15:29:09 - INFO     - |     data = self._sock.recv(self._rbufsize)
> 17:32:00 2018-08-30 15:29:09 - INFO     - | socket.error: [Errno 104] Connection reset by peer
> 17:32:00 2018-08-30 15:29:09 - INFO     - | WARNING: exit code 1 from a shell command.
> 17:32:00 2018-08-30 15:29:09 - INFO     - | ERROR: Function failed: do_rootfs (log file is located at /home/build/tmp-glibc/work/cortexa9hf-neon-oe-linux-gnueabi/image1/1.0-r0/temp/log.do_rootfs.10761)
> 
> Signed-off-by: Manuel Traut <manut at linutronix.de>
> ---
>  elbepack/soapclient.py | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/elbepack/soapclient.py b/elbepack/soapclient.py
> index 25ba4090..953bc1f4 100644
> --- a/elbepack/soapclient.py
> +++ b/elbepack/soapclient.py
> @@ -585,7 +585,14 @@ class WaitProjectBusyAction(ClientAction):
>          part = 1
>  
>          while True:
> -            busy = client.service.get_project_busy(builddir, part)
> +            try:
> +                busy = client.service.get_project_busy(builddir, part)
> +            except socket.error as e:
> +                print("socket error during wait busy occured, retry..",
> +                      file=sys.stderr)
> +                print(e.message)
> +                continue

can we limit the retry to socket.errno == 104 ?
because this would yield an endless loop, if the initvm goes away, or
even more bonkers.

this would pose even more problems during testing.
also not happy with no retry count at this point.


> +
>              if busy == 'FINISH':
>                  break
>              else:
> -- 
> 2.19.0.rc2
> 

-- 
Torben Hohn
Linutronix GmbH | Bahnhofstrasse 3 | D-88690 Uhldingen-Mühlhofen
Phone: +49 7556 25 999 18; Fax.: +49 7556 25 999 99

Hinweise zum Datenschutz finden Sie hier (Informations on data privacy 
can be found here): https://linutronix.de/kontakt/Datenschutz.php

Linutronix GmbH | Firmensitz (Registered Office): Uhldingen-Mühlhofen | 
Registergericht (Registration Court): Amtsgericht Freiburg i.Br., HRB700 
806 | Geschäftsführer (Managing Directors): Heinz Egger, Thomas Gleixner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.linutronix.de/pipermail/elbe-devel/attachments/20180920/1ee484e5/attachment.sig>