[minicoredumper] [External] Debugging corrupt tar files (invalid sparse archive member)

Tue Feb 7 08:20:07 CET 2023

Hello John,

Thank you for your swift reply,

Unfortunately, my mail client refuses to indent your mail, so my answer will become a bit messy....

> The tar files should never be invalid.

Too bad this doesn't match my experience 😉 I checked whether I had the latest version of GNU tar, tried to play with the sparse version numbers and other options a bit, but no success... Since minicoredumper also compresses the file for us and we are able to decompress the file, we don't see any other way our tar archive could get corrupted.

> Can you create a crashing application that reproduces the problem? If yes, can I have such an application so I can reproduce it?

I wish I could. Our application is quite complex (We do real time image processing on the latest Intel I7 with 64 GB of RAM and Nvidia 30/40 series GPU for cuda), requires external servers in order to launch, and we have no clue what the preconditions are for the issue, so I don't know where to start cutting.

> If you can reproduce it, it would also be useful if I had access to the core file produced without minicoredumper. Although a crashing example would be preferred.

Sending you a core dump should not be a problem. I'll send that to you personally, I prefer not to have it on this mailinglist.

> And finally, does it help to force the kernel to wait for the dumping process:

> sysctl -w kernel.core_pipe_limit=0x7fffffff

I will add that option to our settings. It will take a while before you get feedback on this one. (Since we don't know how to reproduce it, it only happens at customer sites, which are all around the world, and it takes a while to release and get new software versions there)

> Some tar creation code was modified in 2.0.2. Do you have these problems with 2.0.1?

We had similar experiences with 2.0.1, which is the first version we used.

In your other mail you ask for errors in the log file. I attached it below this email. I don't see any weird errors, only the very last line "libminicoredumper: no dump data version found" triggers me.

Kind regards,

Jos Hulzink

----------------------------

minicoredumper: argv: /sbin/minicoredumper 715 0 0 11 1675674349 XXXXXXXX GuiVideoRequest"

minicoredumper: comm: GuiVideoRequest"

minicoredumper: exe: XXXXXXXXXXXXXXXXXXXXX"

minicoredumper: recept: /etc/minicoredumper/generic.recept.json"

kernel: [429294.701932][  T812] traps: GuiVideoRequest[812] general protection fault ip:82552d sp:7f552fddf8b0 error:0 in XXXXXXXXX[400000+1b4a000]"

minicoredumper: dump: auxv phdrs: 560 bytes @ 0x400040"

minicoredumper: dump: auxv dyns: 608 bytes @ 0x21f1ac8"

minicoredumper: dump: auxv r_debug: 40 bytes @ 0x7f5597122118"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55971222e0"

minicoredumper: dump: auxv link_map name (): 1 bytes @ 0x7f55971228a0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55971228b0"

minicoredumper: dump: auxv link_map name (linux-vdso.so.1): 16 bytes @ 0x7ffd6a728330"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55970ed070"

minicoredumper: dump: auxv link_map name (/usr/lib/libbpf.so.1): 21 bytes @ 0x7f55970ed050"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55970ed580"

minicoredumper: dump: auxv link_map name (/usr/lib/libxdp.so.1): 21 bytes @ 0x7f55970ed560"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55970eda90"

minicoredumper: dump: auxv link_map name (/usr/lib/libudev.so.1): 22 bytes @ 0x7f55970eda70"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55970edfb0"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libc10.so): 35 bytes @ 0x7f55970edf80"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55970ee4d0"

minicoredumper: dump: auxv link_map name (/usr/lib/libcurl.so.4): 22 bytes @ 0x7f55970ee4b0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55970ee9f0"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libtorch_cpu.so): 41 bytes @ 0x7f55970ee9c0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f558d4a4000"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libtorch_cuda.so): 42 bytes @ 0x7f55970eeef0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f558d4a4540"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcudart.so.11.0): 43 bytes @ 0x7f558d4a4510"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f558d4a4a70"

minicoredumper: dump: auxv link_map name (/usr/lib/libnvidia-ml.so.1): 27 bytes @ 0x7f558d4a4a50"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f558d4a4f90"

minicoredumper: dump: auxv link_map name (/usr/lib/libncursesw.so.6): 26 bytes @ 0x7f558d4a4f70"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f558d4a54b0"

minicoredumper: dump: auxv link_map name (/usr/lib/libstdc++.so.6): 24 bytes @ 0x7f558d4a5490"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f558d4a59c0"

minicoredumper: dump: auxv link_map name (/usr/lib/libc.so.6): 19 bytes @ 0x7f558d4a59a0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f5573761000"

minicoredumper: dump: auxv link_map name (/usr/lib/libm.so.6): 19 bytes @ 0x7f558d4a5eb0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f5573761510"

minicoredumper: dump: auxv link_map name (/usr/lib/libgcc_s.so.1): 23 bytes @ 0x7f55737614f0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f5573761a20"

minicoredumper: dump: auxv link_map name (/usr/lib/libelf.so.1): 21 bytes @ 0x7f5573761a00"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f5573761f30"

minicoredumper: dump: auxv link_map name (/usr/lib/libz.so.1): 19 bytes @ 0x7f5573761f10"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f5597121ad0"

minicoredumper: dump: auxv link_map name (/lib64/ld-linux-x86-64.so.2): 28 bytes @ 0x400270"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55737625f0"

minicoredumper: dump: auxv link_map name (/usr/lib/libnuma.so.1): 22 bytes @ 0x7f55737625d0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f5573762b00"

minicoredumper: dump: auxv link_map name (/usr/lib/libgomp.so.1): 22 bytes @ 0x7f5573762ae0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b30b0"

minicoredumper: dump: auxv link_map name (/usr/lib/libssl.so.3): 21 bytes @ 0x7f55736b3090"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b35c0"

minicoredumper: dump: auxv link_map name (/usr/lib/libcrypto.so.3): 24 bytes @ 0x7f55736b35a0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b3ad0"

minicoredumper: dump: auxv link_map name (/usr/lib/libzstd.so.1): 22 bytes @ 0x7f55736b3ab0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b4330"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcupti.so.11.8): 42 bytes @ 0x7f55736b4300"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b4980"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libc10_cuda.so): 40 bytes @ 0x7f55736b4950"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b1000"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcusparse.so.11): 43 bytes @ 0x7f55736b4e80"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b1540"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcurand.so.10): 41 bytes @ 0x7f55736b1510"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b1a70"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcusolver.so.11): 43 bytes @ 0x7f55736b1a40"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b1fb0"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libnvToolsExt.so.1): 44 bytes @ 0x7f55736b1f80"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b24f0"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcufft.so.10): 40 bytes @ 0x7f55736b24c0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b2a20"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcublas.so.11): 41 bytes @ 0x7f55736b29f0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736af000"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcudnn.so.8): 39 bytes @ 0x7f55736b2f20"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736af640"

minicoredumper: dump: auxv link_map name (/usr/lib/libdl.so.2): 20 bytes @ 0x7f55736af620"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736afb50"

minicoredumper: dump: auxv link_map name (/usr/lib/libpthread.so.0): 25 bytes @ 0x7f55736afb30"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b0060"

minicoredumper: dump: auxv link_map name (/usr/lib/librt.so.1): 20 bytes @ 0x7f55736b0040"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736b0900"

minicoredumper: dump: auxv link_map name (/usr/lib/libutil.so.1): 22 bytes @ 0x7f55736b08e0"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x7f55736a81a0"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libcublasLt.so.11): 43 bytes @ 0x7f55736a8170"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x340f6a0"

minicoredumper: dump: auxv link_map name (/usr/lib/libcuda.so.1): 22 bytes @ 0x340f630"

minicoredumper: dump: auxv link_map: 40 bytes @ 0x3431c50"

minicoredumper: dump: auxv link_map name (/Ellips/Lib/libtorch/lib/libnvrtc.so): 37 bytes @ 0x3431c20"

minicoredumper: first thread: 812"

minicoredumper: dump: stack[812]: 132944 bytes @ 0x7f552fddf8b0"

minicoredumper: dump: pthread data: 8 bytes @ 0x7f55721d0858"

minicoredumper: dump: pthread data: 8 bytes @ 0x7f55971220d0"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5570ff3000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5570ff3000"

minicoredumper: dump: pthread data: 8 bytes @ 0x7f55971220c0"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502a9b000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502a9b000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502adc000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502adc000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502b1d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502b1d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502b5e000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502b5e000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502b9f000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502b9f000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502be0000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502be0000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502c21000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502c21000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502c62000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502c62000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502ca3000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502ca3000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502ce4000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502ce4000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502d25000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502d25000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502d66000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5502d66000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5509b28000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5509b28000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f552fdff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f552fdff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55033f3000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55033f3000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f553007c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f553007c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55300bd000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55300bd000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55300fe000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55300fe000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55301ff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55301ff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5535f7d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5535f7d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5535fbe000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5535fbe000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5535fff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5535fff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5546f3c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5546f3c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5546f7d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5546f7d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5546fbe000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5546fbe000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5546fff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5546fff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f554727c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f554727c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55472bd000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55472bd000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55472fe000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55472fe000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55596fb000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55596fb000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f555973c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f555973c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f555977d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f555977d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55473ff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55473ff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55597be000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55597be000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55597ff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55597ff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f556013c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f556013c000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f556017d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f556017d000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55601be000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55601be000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55601ff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f55601ff000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5570f4a000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5570f4a000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5570f8b000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5570f8b000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5570fcc000"

minicoredumper: dump: pthread data: 2368 bytes @ 0x7f5570fcc000"

minicoredumper: dump: robust mutex head: 24 bytes @ 0x7f5570ff32e0"

minicoredumper: dump: robust mutex: 8 bytes @ 0x7f5570ff32e0"

minicoredumper: dump: [vdso]: 4096 bytes @ 0x7ffd6a728000"

minicoredumper: libminicoredumper: no dump data version found"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linutronix.de/pipermail/minicoredumper/attachments/20230207/927d4f97/attachment-0001.htm>