[elbe-devel] [PATCH 4/4] soapclient: workaround socket exception

Thu Sep 20 11:09:00 CEST 2018

On Thu, Sep 20, 2018 at 10:52:17AM +0200, Torben Hohn wrote:
> On Fri, Sep 14, 2018 at 01:53:05PM +0200, Manuel Traut wrote:
> > maybe the reset occurs, because the initvm is to busy to answer the SOAP
> > request in time. Try to ignore this issue by retrying getting the same
> > part of the log.
> > 
> > 17:32:00 2018-08-30 15:29:09 - INFO     - | Traceback (most recent call last):
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "./elbe", line 55, in <module>
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     cmdmod.run_command(sys.argv[2:])
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/home/elbe/elbepack/commands/control.py", line 161, in run_command
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     action.execute(control, opt, args[1:])
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/home/elbe/elbepack/soapclient.py", line 571, in execute
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     busy = client.service.get_project_busy(builddir, part)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 566, in __call__
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     return client.invoke(args, kwargs)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 705, in invoke
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     result = self.send(soapenv)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 747, in send
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     reply = self.options.transport.send(request)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/https.py", line 66, in send
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     return HttpTransport.send(self, request)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 80, in send
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     fp = self.u2open(u2request)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 127, in u2open
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     return url.open(u2request, timeout=tm)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 429, in open
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     response = self._open(req, data)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 447, in _open
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     '_open', req)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     result = func(*args)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     return self.do_open(httplib.HTTPConnection, req)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     r = h.getresponse(buffering=True)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     response.begin()
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 438, in begin
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     version, status, reason = self._read_status()
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     line = self.fp.readline(_MAXLINE + 1)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/socket.py", line 480, in readline
> > 17:32:00 2018-08-30 15:29:09 - INFO     - |     data = self._sock.recv(self._rbufsize)
> > 17:32:00 2018-08-30 15:29:09 - INFO     - | socket.error: [Errno 104] Connection reset by peer
> > 17:32:00 2018-08-30 15:29:09 - INFO     - | WARNING: exit code 1 from a shell command.
> > 17:32:00 2018-08-30 15:29:09 - INFO     - | ERROR: Function failed: do_rootfs (log file is located at /home/build/tmp-glibc/work/cortexa9hf-neon-oe-linux-gnueabi/image1/1.0-r0/temp/log.do_rootfs.10761)
> > 
> > Signed-off-by: Manuel Traut <manut at linutronix.de>
> > ---
> >  elbepack/soapclient.py | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/elbepack/soapclient.py b/elbepack/soapclient.py
> > index 25ba4090..953bc1f4 100644
> > --- a/elbepack/soapclient.py
> > +++ b/elbepack/soapclient.py
> > @@ -585,7 +585,14 @@ class WaitProjectBusyAction(ClientAction):
> >          part = 1
> >  
> >          while True:
> > -            busy = client.service.get_project_busy(builddir, part)
> > +            try:
> > +                busy = client.service.get_project_busy(builddir, part)
> > +            except socket.error as e:
> > +                print("socket error during wait busy occured, retry..",
> > +                      file=sys.stderr)
> > +                print(e.message)
> > +                continue
> 
> can we limit the retry to socket.errno == 104 ?
> because this would yield an endless loop, if the initvm goes away, or
> even more bonkers.
>
> this would pose even more problems during testing.
>
> also not happy with no retry count at this point.

You could easily quit by pressing CTRL+C

I hope there will be a better solution before becoming elbe-stable.
Would it be OK, to add a TODO marker in the code, that this is more or less
for debugging and should be improved, as soon as we know more detailed what's
going on?

> 
> > +
> >              if busy == 'FINISH':
> >                  break
> >              else:
> > -- 
> > 2.19.0.rc2
> > 
> 
> -- 
> Torben Hohn
> Linutronix GmbH | Bahnhofstrasse 3 | D-88690 Uhldingen-Mühlhofen
> Phone: +49 7556 25 999 18; Fax.: +49 7556 25 999 99
> 
> Hinweise zum Datenschutz finden Sie hier (Informations on data privacy 
> can be found here): https://linutronix.de/kontakt/Datenschutz.php
> 
> Linutronix GmbH | Firmensitz (Registered Office): Uhldingen-Mühlhofen | 
> Registergericht (Registration Court): Amtsgericht Freiburg i.Br., HRB700 
> 806 | Geschäftsführer (Managing Directors): Heinz Egger, Thomas Gleixner