[elbe-devel] [PATCH v2] soapclient: workaround socket exception

[Manuel Traut]

maybe the reset occurs, because the initvm is to busy to answer the SOAP
request in time. Try to ignore this issue by retrying getting the same
part of the log.

17:32:00 2018-08-30 15:29:09 - INFO     - | Traceback (most recent call last):
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "./elbe", line 55, in <module>
17:32:00 2018-08-30 15:29:09 - INFO     - |     cmdmod.run_command(sys.argv[2:])
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/home/elbe/elbepack/commands/control.py", line 161, in run_command
17:32:00 2018-08-30 15:29:09 - INFO     - |     action.execute(control, opt, args[1:])
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/home/elbe/elbepack/soapclient.py", line 571, in execute
17:32:00 2018-08-30 15:29:09 - INFO     - |     busy = client.service.get_project_busy(builddir, part)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 566, in __call__
17:32:00 2018-08-30 15:29:09 - INFO     - |     return client.invoke(args, kwargs)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 705, in invoke
17:32:00 2018-08-30 15:29:09 - INFO     - |     result = self.send(soapenv)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 747, in send
17:32:00 2018-08-30 15:29:09 - INFO     - |     reply = self.options.transport.send(request)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/https.py", line 66, in send
17:32:00 2018-08-30 15:29:09 - INFO     - |     return HttpTransport.send(self, request)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 80, in send
17:32:00 2018-08-30 15:29:09 - INFO     - |     fp = self.u2open(u2request)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 127, in u2open
17:32:00 2018-08-30 15:29:09 - INFO     - |     return url.open(u2request, timeout=tm)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 429, in open
17:32:00 2018-08-30 15:29:09 - INFO     - |     response = self._open(req, data)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 447, in _open
17:32:00 2018-08-30 15:29:09 - INFO     - |     '_open', req)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
17:32:00 2018-08-30 15:29:09 - INFO     - |     result = func(*args)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
17:32:00 2018-08-30 15:29:09 - INFO     - |     return self.do_open(httplib.HTTPConnection, req)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
17:32:00 2018-08-30 15:29:09 - INFO     - |     r = h.getresponse(buffering=True)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
17:32:00 2018-08-30 15:29:09 - INFO     - |     response.begin()
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 438, in begin
17:32:00 2018-08-30 15:29:09 - INFO     - |     version, status, reason = self._read_status()
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
17:32:00 2018-08-30 15:29:09 - INFO     - |     line = self.fp.readline(_MAXLINE + 1)
17:32:00 2018-08-30 15:29:09 - INFO     - |   File "/usr/lib/python2.7/socket.py", line 480, in readline
17:32:00 2018-08-30 15:29:09 - INFO     - |     data = self._sock.recv(self._rbufsize)
17:32:00 2018-08-30 15:29:09 - INFO     - | socket.error: [Errno 104] Connection reset by peer
17:32:00 2018-08-30 15:29:09 - INFO     - | WARNING: exit code 1 from a shell command.
17:32:00 2018-08-30 15:29:09 - INFO     - | ERROR: Function failed: do_rootfs (log file is located at /home/build/tmp-glibc/work/cortexa9hf-neon-oe-linux-gnueabi/image1/1.0-r0/temp/log.do_rootfs.10761)

Signed-off-by: Manuel Traut <manut at linutronix.de>
---

changes since v1:

  * limit retry on socket.errno == 104
  * added TODO in source
  * print execption to stderr instead of stdout

 elbepack/soapclient.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/elbepack/soapclient.py b/elbepack/soapclient.py
index 25ba4090..5a8ef7f1 100644
--- a/elbepack/soapclient.py
+++ b/elbepack/soapclient.py
@@ -585,7 +585,21 @@ class WaitProjectBusyAction(ClientAction):
         part = 1
 
         while True:
-            busy = client.service.get_project_busy(builddir, part)
+            try:
+                busy = client.service.get_project_busy(builddir, part)
+            # TODO the root cause of this problem is unclear. To enable a
+            # get more information print the exception and retry to see if
+            # the connection problem is just a temporary problem. This
+            # code should be reworked as soon as it's clear what is going on
+            # here
+            except socket.error as e:
+                print(e.message, file=sys.stderr)
+                if e.errno == 104:
+                    break
+                print("socket error during wait busy occured, retry..",
+                      file=sys.stderr)
+                continue
+
             if busy == 'FINISH':
                 break
             else:
-- 
2.19.0