Daemon segfaults when running under systemd and is stopped #2

When running under systemd using the following config:

[Unit]
Description=cryptic net
Requires=network.target
After=network.target

[Service]
Restart=always
RestartSec=1s
User=root
ExecStart=/usr/local/bin/cryptic-net daemon -c /usr/local/etc/cryptic-net/daemon.yml

[Install]
WantedBy=multi-user.target

More often than not when doing sudo systemctl stop cryptic-net the whole process segfaults, rather than cleaning up nicely. I have never seen this when running the process directly in a shell, and I suspect it has something to do with pmux's signal handling.

Jan 17 20:19:16 mediocre-desktop systemd[1]: Stopping cryptic net...
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: fatal error: fault
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: [signal SIGBUS: bus error code=0x2 addr=0x8ec519 pc=0x8ec519]
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: goroutine 19 [running]:
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: fatal error:
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: panic during panic
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: [signal SIGBUS: bus error code=0x2 addr=0xb65330 pc=0x40342e]
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: runtime stack:
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: fatal error:
Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: stack trace unavailable

Jan 17 20:19:17 mediocre-desktop systemd-coredump[1987291]: [🡕] Process 1141885 (dnsmasq) of user 0 dumped core.

                                                            Module linux-vdso.so.1 with build-id eb929fd31207126092e15885ffcc3ef5d93bc02e
                                                            Stack trace of thread 1141885:
                                                            #0  0x000000000041f340 n/a (n/a + 0x0)
                                                            ELF object binary architecture: AMD x86-64

Jan 17 20:19:17 mediocre-desktop systemd-coredump[1987293]: [🡕] Process 1141883 (garage) of user 0 dumped core.

                                                            Module linux-vdso.so.1 with build-id eb929fd31207126092e15885ffcc3ef5d93bc02e
                                                            Stack trace of thread 1141883:
                                                            #0  0x00007f7075b40240 n/a (n/a + 0x0)
                                                            ELF object binary architecture: AMD x86-64

Jan 17 20:19:17 mediocre-desktop systemd-coredump[1987292]: [🡕] Process 1141884 (garage) of user 0 dumped core.

                                                            Module linux-vdso.so.1 with build-id eb929fd31207126092e15885ffcc3ef5d93bc02e
                                                            Stack trace of thread 1141884:
                                                            #0  0x00007f90e4b40240 n/a (n/a + 0x0)
                                                            ELF object binary architecture: AMD x86-64

Jan 17 20:19:17 mediocre-desktop systemd[1]: cryptic-net.service: Failed with result 'exit-code'.

Notice that all sub-processes except nebula also show a coredump in systemd. So the problem could alternatively be with nebula.

When running under systemd using the following config: ``` [Unit] Description=cryptic net Requires=network.target After=network.target [Service] Restart=always RestartSec=1s User=root ExecStart=/usr/local/bin/cryptic-net daemon -c /usr/local/etc/cryptic-net/daemon.yml [Install] WantedBy=multi-user.target ``` More often than not when doing `sudo systemctl stop cryptic-net` the whole process segfaults, rather than cleaning up nicely. I have never seen this when running the process directly in a shell, and I suspect it has something to do with pmux's signal handling. ``` Jan 17 20:19:16 mediocre-desktop systemd[1]: Stopping cryptic net... Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: fatal error: fault Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: [signal SIGBUS: bus error code=0x2 addr=0x8ec519 pc=0x8ec519] Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: goroutine 19 [running]: Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: fatal error: Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: panic during panic Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: [signal SIGBUS: bus error code=0x2 addr=0xb65330 pc=0x40342e] Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: runtime stack: Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: fatal error: Jan 17 20:19:16 mediocre-desktop cryptic-net[1141853]: stack trace unavailable Jan 17 20:19:17 mediocre-desktop systemd-coredump[1987291]: [🡕] Process 1141885 (dnsmasq) of user 0 dumped core. Module linux-vdso.so.1 with build-id eb929fd31207126092e15885ffcc3ef5d93bc02e Stack trace of thread 1141885: #0 0x000000000041f340 n/a (n/a + 0x0) ELF object binary architecture: AMD x86-64 Jan 17 20:19:17 mediocre-desktop systemd-coredump[1987293]: [🡕] Process 1141883 (garage) of user 0 dumped core. Module linux-vdso.so.1 with build-id eb929fd31207126092e15885ffcc3ef5d93bc02e Stack trace of thread 1141883: #0 0x00007f7075b40240 n/a (n/a + 0x0) ELF object binary architecture: AMD x86-64 Jan 17 20:19:17 mediocre-desktop systemd-coredump[1987292]: [🡕] Process 1141884 (garage) of user 0 dumped core. Module linux-vdso.so.1 with build-id eb929fd31207126092e15885ffcc3ef5d93bc02e Stack trace of thread 1141884: #0 0x00007f90e4b40240 n/a (n/a + 0x0) ELF object binary architecture: AMD x86-64 Jan 17 20:19:17 mediocre-desktop systemd[1]: cryptic-net.service: Failed with result 'exit-code'. ``` Notice that all sub-processes _except_ nebula also show a coredump in systemd. So the problem could alternatively be with nebula.