Murmur does not shut down when SIGTERM #1644

Closed
opened 2026-02-20 21:09:10 -05:00 by deekerman · 9 comments
Owner

Originally created by @Silmerias on GitHub (Sep 30, 2019).

murmur-static_x86-1.3.0
Debian stretch
4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux

When sent SIGTERM by systemd we see murmur closing, but the process will never shut down before systemd sigterm timeout kill it with SIGKILL.

2019-10-01 00:05:06.968 Caught SIGTERM, exiting
2019-10-01 00:05:06.968 Killing running servers

During the process the CPU is 100% overheated and strace show murmur trying to poll at 24000 messages/seconds.

poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])

Originally created by @Silmerias on GitHub (Sep 30, 2019). murmur-static_x86-1.3.0 Debian stretch 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux When sent SIGTERM by systemd we see murmur closing, but the process will never shut down before systemd sigterm timeout kill it with SIGKILL. > <C>2019-10-01 00:05:06.968 Caught SIGTERM, exiting > <W>2019-10-01 00:05:06.968 Killing running servers During the process the CPU is 100% overheated and strace show murmur trying to poll at 24000 messages/seconds. >poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) >poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) >poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) >poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) >poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) >poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) >poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])
deekerman 2026-02-20 21:09:10 -05:00
  • closed this issue
  • added the
    bug
    server
    labels
Author
Owner

@davidebeatrici commented on GitHub (Sep 30, 2019):

Could you attach GDB to Murmur and check the backtrace after SIGTERM is sent, please?

@davidebeatrici commented on GitHub (Sep 30, 2019): Could you attach GDB to Murmur and check the backtrace after `SIGTERM` is sent, please?
Author
Owner

@Silmerias commented on GitHub (Sep 30, 2019):

Attaching to process 16043
[New LWP 16044]
[New LWP 16045]
[New LWP 16046]
[New LWP 16047]
0xf778fc89 in __kernel_vsyscall ()
(gdb) backtrace
#0 0xf778fc89 in __kernel_vsyscall ()
#1 0x00a7b693 in ?? ()
#2 0xf66e4d09 in ?? ()
#3 0xf66e4f00 in ?? ()
#4 0xf66e500d in ?? ()
#5 0xf66e532b in ?? ()
#6 0xf66e563b in ?? ()
#7 0xf66eb331 in ?? ()
#8 0xf66eb5da in ?? ()
#9 0xf66e9c54 in ?? ()
#10 0xf66e9d75 in ?? ()
#11 0xf6718a5c in ?? ()
#12 0xf6fa2042 in ?? ()
#13 0x080b010a in ?? ()
#14 0x0808375c in ?? ()
#15 0x0808dfb1 in ?? ()
#16 0x0810bc39 in ?? ()
#17 0x0807c314 in ?? ()
#18 0x009c5edc in ?? ()
#19 0x0805f1a1 in ?? ()

@Silmerias commented on GitHub (Sep 30, 2019): > Attaching to process 16043 > [New LWP 16044] > [New LWP 16045] > [New LWP 16046] > [New LWP 16047] > 0xf778fc89 in __kernel_vsyscall () > (gdb) backtrace > #0 0xf778fc89 in __kernel_vsyscall () > #1 0x00a7b693 in ?? () > #2 0xf66e4d09 in ?? () > #3 0xf66e4f00 in ?? () > #4 0xf66e500d in ?? () > #5 0xf66e532b in ?? () > #6 0xf66e563b in ?? () > #7 0xf66eb331 in ?? () > #8 0xf66eb5da in ?? () > #9 0xf66e9c54 in ?? () > #10 0xf66e9d75 in ?? () > #11 0xf6718a5c in ?? () > #12 0xf6fa2042 in ?? () > #13 0x080b010a in ?? () > #14 0x0808375c in ?? () > #15 0x0808dfb1 in ?? () > #16 0x0810bc39 in ?? () > #17 0x0807c314 in ?? () > #18 0x009c5edc in ?? () > #19 0x0805f1a1 in ?? ()
Author
Owner

@mkrsn commented on GitHub (Mar 16, 2021):

Same problem here. Any News on that?

version: 1.3.4.x86_static
Linux 5.10.23-gentoo #1 SMP Mon Mar 15 12:07:21 CET 2021 x86_64 GNU/Linux

After a SIGTERM:

...
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
...

fd12 is the file descriptor to the MariaDB Socket.

backtrace

Attaching to process 9207
[New LWP 9209]
[New LWP 9210]
[New LWP 9211]
[New LWP 9212]
[New LWP 9220]
0xf7f68549 in __kernel_vsyscall ()
(gdb) backtrace
#0  0xf7f68549 in __kernel_vsyscall ()
#1  0x00a7b693 in ?? ()
#2  0xf6ebbd09 in ?? ()
#3  0xf6ebbf00 in ?? ()
#4  0xf6ebc00d in ?? ()
#5  0xf6ebc32b in ?? ()
#6  0xf6ebc63b in ?? ()
#7  0xf6ec2331 in ?? ()
#8  0xf6ec25da in ?? ()
#9  0xf6ec0c54 in ?? ()
#10 0xf6ec0d75 in ?? ()
#11 0xf6eefa5c in ?? ()
#12 0xf7779042 in ?? ()
#13 0x080d231a in ?? ()
#14 0x08083a8c in ?? ()
#15 0x08173dcb in ?? ()
#16 0x08173e26 in ?? ()
#17 0x081bcb8c in ?? ()
#18 0xf7398c69 in ?? ()
#19 0xf73998aa in ?? ()
#20 0x081b92a3 in ?? ()
#21 0x0810f08d in ?? ()
#22 0x0807c490 in ?? ()
#23 0x009c5edc in ?? ()
#24 0x0805f201 in ?? ()
@mkrsn commented on GitHub (Mar 16, 2021): Same problem here. Any News on that? version: 1.3.4.x86_static Linux 5.10.23-gentoo #1 SMP Mon Mar 15 12:07:21 CET 2021 x86_64 GNU/Linux After a SIGTERM: ``` ... poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) ... ``` fd12 is the file descriptor to the MariaDB Socket. backtrace ``` Attaching to process 9207 [New LWP 9209] [New LWP 9210] [New LWP 9211] [New LWP 9212] [New LWP 9220] 0xf7f68549 in __kernel_vsyscall () (gdb) backtrace #0 0xf7f68549 in __kernel_vsyscall () #1 0x00a7b693 in ?? () #2 0xf6ebbd09 in ?? () #3 0xf6ebbf00 in ?? () #4 0xf6ebc00d in ?? () #5 0xf6ebc32b in ?? () #6 0xf6ebc63b in ?? () #7 0xf6ec2331 in ?? () #8 0xf6ec25da in ?? () #9 0xf6ec0c54 in ?? () #10 0xf6ec0d75 in ?? () #11 0xf6eefa5c in ?? () #12 0xf7779042 in ?? () #13 0x080d231a in ?? () #14 0x08083a8c in ?? () #15 0x08173dcb in ?? () #16 0x08173e26 in ?? () #17 0x081bcb8c in ?? () #18 0xf7398c69 in ?? () #19 0xf73998aa in ?? () #20 0x081b92a3 in ?? () #21 0x0810f08d in ?? () #22 0x0807c490 in ?? () #23 0x009c5edc in ?? () #24 0x0805f201 in ?? () ```
Author
Owner

@Krzmbrzl commented on GitHub (Mar 17, 2021):

Afaik nobody was able to track the issue down yet. In fact I am not even sure somebody found the time to really try yet 🤔

@Krzmbrzl commented on GitHub (Mar 17, 2021): Afaik nobody was able to track the issue down yet. In fact I am not even sure somebody found the time to really try yet :thinking:
Author
Owner

@Hartmnt commented on GitHub (Mar 7, 2025):

I can not reproduce this using recent versions of mumble-server, nor murmur 1.3.4
Both tested as a service with systemd v256 using systemctl stop murmur
Am I missing something here or was this a systemd or configuration problem?

@Hartmnt commented on GitHub (Mar 7, 2025): I can not reproduce this using recent versions of mumble-server, nor murmur 1.3.4 Both tested as a service with systemd v256 using ``systemctl stop murmur`` Am I missing something here or was this a systemd or configuration problem?
Author
Owner

@mkrsn commented on GitHub (Mar 7, 2025):

This issue is now ~6 years old - maybe it got fixed unintentionally in the meanwhile.

Some months after reporting the issue i switched away from the static binary. Since then I compile it by myself - problem solved for me.

As far as I can tell: it was a problem in the binary. I have never used systemd and have not changed the configuration, only the binary.

@mkrsn commented on GitHub (Mar 7, 2025): This issue is now ~6 years old - maybe it got fixed unintentionally in the meanwhile. Some months after reporting the issue i switched away from the static binary. Since then I compile it by myself - problem solved for me. As far as I can tell: it was a problem in the binary. I have never used systemd and have not changed the configuration, only the binary.
Author
Owner

@Hartmnt commented on GitHub (Mar 7, 2025):

@mkrsn Interesting, I have tried as systemd service and as plain binary. I even downloaded the old 1.3.4 static binary from GitHub. Under no circumstances I could reproduce the described behavior... At least not with a relative recent distro. I'll close this then.

@Hartmnt commented on GitHub (Mar 7, 2025): @mkrsn Interesting, I have tried as systemd service and as plain binary. I even downloaded the old 1.3.4 static binary from GitHub. Under no circumstances I could reproduce the described behavior... At least not with a relative recent distro. I'll close this then.
Author
Owner

@mkrsn commented on GitHub (Mar 7, 2025):

If it is helpful for you, i can check the binary. See if I can still reproduce the issue.

@mkrsn commented on GitHub (Mar 7, 2025): If it is helpful for you, i can check the binary. See if I can still reproduce the issue.
Author
Owner

@Hartmnt commented on GitHub (Mar 7, 2025):

@mkrsn Mhh, we don't support the 1.3 series anymore anyway, so no need to bother if 1.5.x works.

But I have to wonder what went wrong at what point :D

@Hartmnt commented on GitHub (Mar 7, 2025): @mkrsn Mhh, we don't support the 1.3 series anymore anyway, so no need to bother if 1.5.x works. But I have to wonder what went wrong at what point :D
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/mumble-mumble-voip#1644
No description provided.