starred/pikvm-pikvm
1
0
Fork 0
mirror of https://github.com/pikvm/pikvm.git synced 2026-03-02 18:16:56 -05:00
Code Issues 133 Projects Packages Wiki Activity

linux-firmware-2025... is broken, use linux-firmware-pikvm-2024... #1044

New issue
Closed
opened 2026-02-20 14:10:18 -05:00 by deekerman · 9 comments
deekerman commented 2026-02-20 14:10:18 -05:00
Owner
Copy link

Originally created by @vbooka1 on GitHub (Jun 6, 2025).

Originally assigned to: @mdevaev on GitHub.

Describe the bug
After the last update my piKVM randomly hangs some time after each reboot. The embedded WiFi network disappears, SSH closes connection, HTTPS hangs. Strangely HTTP works and this makes me think that something was broken with the encryption code in the kernel.

SSH:

$ telnet 192.168.111.222 22
Trying 192.168.111.222...
Connected to 192.168.111.222.
Escape character is '^]'.
Connection closed by foreign host.

HTTPS:

$ curl -v 192.168.111.222 -L
*   Trying 192.168.111.222:80...
* Connected to 192.168.111.222 (192.168.111.222) port 80
> GET / HTTP/1.1
> Host: 192.168.111.222
> User-Agent: curl/8.6.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.27.5
< Date: Fri, 06 Jun 2025 04:18:30 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://192.168.111.222/
< 
* Ignoring the response-body
* Connection #0 to host 192.168.111.222 left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://192.168.111.222/'
*   Trying 192.168.111.222:443...
* Connected to 192.168.111.222 (192.168.111.222) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

(nothing happens here)

strangely HTTP works:

> curl -v 192.168.111.222
*   Trying 192.168.111.222:80...
* Connected to 192.168.111.222 (192.168.111.222) port 80
> GET / HTTP/1.1
> Host: 192.168.111.222
> User-Agent: curl/8.6.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.27.5
< Date: Fri, 06 Jun 2025 04:18:28 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://192.168.111.222/
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.27.5</center>
</body>
</html>

To Reproduce
Steps to reproduce the behavior, like:

  1. run pacman -Syu (or perhaps pikvm-update, I usually update with pacman)
  2. reboot
  3. connect to embedded WiFi
  4. wait until the WiFi randomly disconnects.

Expected behavior
I do not expect the above behavior to happen.

Screenshots
not applicable

Desktop (please complete the following information):
not applicable

PiKVM info:

  • Raspberry Pi board version [e.g. RPi 4]
    raspberry Pi 4 model B, 4GB
  • PiKVM platform [e.g. v2-hdmi]
    pikvm-v3-hat
  • Video capture type [e.g. CSI bridge]
    not applicable
  • KVMD version: pacman -Q | grep kvmd
    kvmd 4.82-1
    kvmd-fan 0.32-1
    kvmd-platform-v3-hdmi-rpi4 4.82-1
    kvmd-webterm 0.50-1
  • uStreamer version: pacman -Q | grep ustreamer
    ustreamer 6.37-1
  • Linux kernel: uname -a
    Linux pikvm 6.6.45-13-rpi #1 SMP Tue May 27 17:16:41 UTC 2025 armv7l GNU/Linux

Additional context
I have just realized that I may use a standard Arch repo not a piKVM repo, how do I verify?

# cat /etc/pacman.d/mirrorlist        
Server = http://mirror.archlinuxarm.org/$arch/$repo
Originally created by @vbooka1 on GitHub (Jun 6, 2025). Originally assigned to: @mdevaev on GitHub. **Describe the bug** After the last update my piKVM randomly hangs some time after each reboot. The embedded WiFi network disappears, SSH closes connection, HTTPS hangs. Strangely HTTP works and this makes me think that something was broken with the encryption code in the kernel. SSH: ``` $ telnet 192.168.111.222 22 Trying 192.168.111.222... Connected to 192.168.111.222. Escape character is '^]'. Connection closed by foreign host. ``` HTTPS: ``` $ curl -v 192.168.111.222 -L * Trying 192.168.111.222:80... * Connected to 192.168.111.222 (192.168.111.222) port 80 > GET / HTTP/1.1 > Host: 192.168.111.222 > User-Agent: curl/8.6.0 > Accept: */* > < HTTP/1.1 301 Moved Permanently < Server: nginx/1.27.5 < Date: Fri, 06 Jun 2025 04:18:30 GMT < Content-Type: text/html < Content-Length: 169 < Connection: keep-alive < Location: https://192.168.111.222/ < * Ignoring the response-body * Connection #0 to host 192.168.111.222 left intact * Clear auth, redirects to port from 80 to 443 * Issue another request to this URL: 'https://192.168.111.222/' * Trying 192.168.111.222:443... * Connected to 192.168.111.222 (192.168.111.222) port 443 * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): (nothing happens here) ``` strangely HTTP works: ``` > curl -v 192.168.111.222 * Trying 192.168.111.222:80... * Connected to 192.168.111.222 (192.168.111.222) port 80 > GET / HTTP/1.1 > Host: 192.168.111.222 > User-Agent: curl/8.6.0 > Accept: */* > < HTTP/1.1 301 Moved Permanently < Server: nginx/1.27.5 < Date: Fri, 06 Jun 2025 04:18:28 GMT < Content-Type: text/html < Content-Length: 169 < Connection: keep-alive < Location: https://192.168.111.222/ < <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>nginx/1.27.5</center> </body> </html> ``` **To Reproduce** Steps to reproduce the behavior, like: 1. run `pacman -Syu` (or perhaps `pikvm-update`, I usually update with `pacman`) 2. reboot 3. connect to embedded WiFi 4. wait until the WiFi randomly disconnects. **Expected behavior** I do not expect the above behavior to happen. **Screenshots** not applicable **Desktop (please complete the following information):** not applicable **PiKVM info:** - Raspberry Pi board version [e.g. RPi 4] raspberry Pi 4 model B, 4GB - PiKVM platform [e.g. v2-hdmi] pikvm-v3-hat - Video capture type [e.g. CSI bridge] not applicable - KVMD version: `pacman -Q | grep kvmd` kvmd 4.82-1 kvmd-fan 0.32-1 kvmd-platform-v3-hdmi-rpi4 4.82-1 kvmd-webterm 0.50-1 - uStreamer version: `pacman -Q | grep ustreamer` ustreamer 6.37-1 - Linux kernel: `uname -a` Linux pikvm 6.6.45-13-rpi #1 SMP Tue May 27 17:16:41 UTC 2025 armv7l GNU/Linux **Additional context** I have just realized that I may use a standard Arch repo not a piKVM repo, how do I verify? ``` # cat /etc/pacman.d/mirrorlist Server = http://mirror.archlinuxarm.org/$arch/$repo ```
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@vbooka1 commented on GitHub (Jun 6, 2025):

I have changed the journald storage to Persistent, hope it will show what happens before the hang. Anything else I coud do to debug this?

@vbooka1 commented on GitHub (Jun 6, 2025): I have changed the journald storage to Persistent, hope it will show what happens before the hang. Anything else I coud do to debug this?
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@mdevaev commented on GitHub (Jun 6, 2025):

Please try to reflash the service if possible, update and check again.

Best regards,
Max Devaev

Пт, 6 июня 2025 г. в 07:57, vbooka1 @.***>:

vbooka1 left a comment (pikvm/pikvm#1539)
https://github.com/pikvm/pikvm/issues/1539#issuecomment-2948090167

I have changed the journald storage to Persistent, hope it will show what
happens before the hang. Anything else I coud do to debug this?

—
Reply to this email directly, view it on GitHub
https://github.com/pikvm/pikvm/issues/1539#issuecomment-2948090167, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AADFUAYIYXKFJ24D3D2RXO33CENVPAVCNFSM6AAAAAB6XDGQDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNBYGA4TAMJWG4
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

@mdevaev commented on GitHub (Jun 6, 2025): Please try to reflash the service if possible, update and check again. Best regards, Max Devaev Пт, 6 июня 2025 г. в 07:57, vbooka1 ***@***.***>: > *vbooka1* left a comment (pikvm/pikvm#1539) > <https://github.com/pikvm/pikvm/issues/1539#issuecomment-2948090167> > > I have changed the journald storage to Persistent, hope it will show what > happens before the hang. Anything else I coud do to debug this? > > — > Reply to this email directly, view it on GitHub > <https://github.com/pikvm/pikvm/issues/1539#issuecomment-2948090167>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADFUAYIYXKFJ24D3D2RXO33CENVPAVCNFSM6AAAAAB6XDGQDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNBYGA4TAMJWG4> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> >
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@vbooka1 commented on GitHub (Jun 6, 2025):

Please try to reflash the service if possible, update and check again.

Best regards,
Max Devaev

Пт, 6 июня 2025 г. в 07:57, vbooka1 @.***>:
…

Hi, you mean reflash the whole SD card?
Please tell which image from https://files.pikvm.org/images/ I should use, is it v3-hdmi-rpi4?

@vbooka1 commented on GitHub (Jun 6, 2025): > Please try to reflash the service if possible, update and check again. > > Best regards, > Max Devaev > > > Пт, 6 июня 2025 г. в 07:57, vbooka1 ***@***.***>: > […](#) Hi, you mean reflash the whole SD card? Please tell which image from https://files.pikvm.org/images/ I should use, is it v3-hdmi-rpi4?
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@vbooka1 commented on GitHub (Jun 6, 2025):

I have found another problem: now ping -c 1 has just 10ms timeout so on a slow mobile network (I use a USB modem to have piKVM always online) it always fails:

[root@pikvm ~]# ping -c 1 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

[root@pikvm ~]# echo $?
1

strace shows this:

setitimer(ITIMER_REAL, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=0, tv_usec=10000}}, NULL) = 0 <0.000042>

If I explicitly tell ping to wait for at least 1 second then ping works well:

[root@pikvm ~]# ping -c 1 1.1.1.1 -W 1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=35.6 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 35.581/35.581/35.581/0.000 ms
[root@pikvm ~]# echo $?
0

I can't recall for sure but I think the ping timeout was higher before the update. And I believe it should be higher, not just 10ms. Should I open a separate bug report for ping issue?

@vbooka1 commented on GitHub (Jun 6, 2025): I have found another problem: now `ping -c 1` has just 10ms timeout so on a slow mobile network (I use a USB modem to have piKVM always online) it always fails: ``` [root@pikvm ~]# ping -c 1 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. --- 1.1.1.1 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [root@pikvm ~]# echo $? 1 ``` strace shows this: ``` setitimer(ITIMER_REAL, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=0, tv_usec=10000}}, NULL) = 0 <0.000042> ``` If I explicitly tell ping to wait for at least 1 second then ping works well: ``` [root@pikvm ~]# ping -c 1 1.1.1.1 -W 1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=35.6 ms --- 1.1.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 35.581/35.581/35.581/0.000 ms [root@pikvm ~]# echo $? 0 ``` I can't recall for sure but I think the ping timeout was higher before the update. And I believe it should be higher, not just 10ms. Should I open a separate bug report for ping issue?
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@mdevaev commented on GitHub (Jun 6, 2025):

Hi, you mean reflash the whole SD card?

Yes. If you're using PiKVM V3 with 32-bit kernel now, you need this image: https://files.pikvm.org/images/v3-hdmi-rpi4/arm/v3-hdmi-rpi4-box-latest.img.xz

Should I open a separate bug report for ping issue?

Let's check the kernel for now. I think this is some kind of local problem you have. If the error with the kernel is reproduced, then we will deal with it further.

@mdevaev commented on GitHub (Jun 6, 2025): > Hi, you mean reflash the whole SD card? Yes. If you're using PiKVM V3 with 32-bit kernel now, you need this image: https://files.pikvm.org/images/v3-hdmi-rpi4/arm/v3-hdmi-rpi4-box-latest.img.xz > Should I open a separate bug report for ping issue? Let's check the kernel for now. I think this is some kind of local problem you have. If the error with the kernel is reproduced, then we will deal with it further.
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@mdevaev commented on GitHub (Jun 7, 2025):

Sup?

@mdevaev commented on GitHub (Jun 7, 2025): Sup?
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@mdevaev commented on GitHub (Jun 9, 2025):

No feedback and we can't reproduce it on our side. I can recommend you to reflash the OS just to be sure.

@mdevaev commented on GitHub (Jun 9, 2025): No feedback and we can't reproduce it on our side. I can recommend you to reflash the OS just to be sure.
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@vbooka1 commented on GitHub (Jul 8, 2025):

Sorry I needed some time because piKVM also works as a gateway and router of my home lab so I needed to configure some things to be able to safely remove piKVM from the network for a whole day.

I've reflashed the microSD card and spotted a difference in partitioning. Before reflashing it was like this:

Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk0p1             1   524287   524287  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        524288 12500000 11975713  5.7G 83 Linux
/dev/mmcblk0p3      12500001 62333951 49833951 23.8G 83 Linux

after reflash:

Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk0p1             1   524287   524287  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        524288  1048575   524288  256M 83 Linux
/dev/mmcblk0p3       1048576 13631487 12582912    6G 83 Linux
/dev/mmcblk0p4      13631488 17825791  4194304    2G 83 Linux

I think it is normal because the OS booted without issues. But the very first pacman -Syu failed with:

error: failed to commit transaction (conflicting files)
v4l-utils: /usr/bin/edid-decode exists in filesystem (owned by edid-decode)
v4l-utils: /usr/share/man/man1/edid-decode.1.gz exists in filesystem (owned by edid-decode)
Errors occurred, no packages were upgraded.

so I've had to remove the package pacman -R edid-decode, and then the system updated without any issues.

I haven't experienced any random hangs during the day so the issue is resolved and it definitely was not with the kernel. Especially considering that the current kernel version is also 6.6.45-13-rpi as before the full reflash. So the problem was somewhere else.

I think that the problem was with the kernel firmware package because before the reflash I had some kernel errors (oops or taints - I don't remember for sure) in the dmesg every reboot and there are no kernel errors after the reflash. Unfortunately I haven't saved any kernel logs or package versions prior to reflash, but I think that I have installed the "linux-firmware" package from standard Arch repo (linux-firmware-2025...), while the flashed image has "linux-firmware-pikvm" from piKVM repo (linux-firmware-pikvm-20240809.59460076-3) and this might have been the core of this issue.
I recall that the error in the dmesg was about brcm-something and I've thought it was about the WiFi, but I did not pay much attention to these messages because WiFi worked well.

The ping issue is also fixed, its hardcoded timeout is 10 seconds now instead of 10 milliseconds prior to reflash:

setitimer(ITIMER_REAL, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=10, tv_usec=0}}, NULL) = 0 <0.000023>

The fixed version is: ping from iputils 20250605

@vbooka1 commented on GitHub (Jul 8, 2025): Sorry I needed some time because piKVM also works as a gateway and router of my home lab so I needed to configure some things to be able to safely remove piKVM from the network for a whole day. I've reflashed the microSD card and spotted a difference in partitioning. Before reflashing it was like this: ``` Device Boot Start End Sectors Size Id Type /dev/mmcblk0p1 1 524287 524287 256M c W95 FAT32 (LBA) /dev/mmcblk0p2 524288 12500000 11975713 5.7G 83 Linux /dev/mmcblk0p3 12500001 62333951 49833951 23.8G 83 Linux ``` after reflash: ``` Device Boot Start End Sectors Size Id Type /dev/mmcblk0p1 1 524287 524287 256M c W95 FAT32 (LBA) /dev/mmcblk0p2 524288 1048575 524288 256M 83 Linux /dev/mmcblk0p3 1048576 13631487 12582912 6G 83 Linux /dev/mmcblk0p4 13631488 17825791 4194304 2G 83 Linux ``` I think it is normal because the OS booted without issues. But the very first `pacman -Syu` failed with: ``` error: failed to commit transaction (conflicting files) v4l-utils: /usr/bin/edid-decode exists in filesystem (owned by edid-decode) v4l-utils: /usr/share/man/man1/edid-decode.1.gz exists in filesystem (owned by edid-decode) Errors occurred, no packages were upgraded. ``` so I've had to remove the package `pacman -R edid-decode`, and then the system updated without any issues. I haven't experienced any random hangs during the day so the issue is resolved and it definitely was not with the kernel. Especially considering that the current kernel version is also `6.6.45-13-rpi` as before the full reflash. So the problem was somewhere else. I think that the problem was with the kernel firmware package because before the reflash I had some kernel errors (oops or taints - I don't remember for sure) in the `dmesg` every reboot and there are no kernel errors after the reflash. Unfortunately I haven't saved any kernel logs or package versions prior to reflash, but I think that I have installed the "linux-firmware" package from standard Arch repo (`linux-firmware-2025...`), while the flashed image has "linux-firmware-pikvm" from piKVM repo (`linux-firmware-pikvm-20240809.59460076-3`) and this might have been the core of this issue. I recall that the error in the `dmesg` was about `brcm`-something and I've thought it was about the WiFi, but I did not pay much attention to these messages because WiFi worked well. The `ping` issue is also fixed, its hardcoded timeout is 10 seconds now instead of 10 milliseconds prior to reflash: ``` setitimer(ITIMER_REAL, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=10, tv_usec=0}}, NULL) = 0 <0.000023> ``` The fixed version is: `ping from iputils 20250605`
deekerman commented 2026-02-20 14:10:21 -05:00
Author
Owner
Copy link

@mdevaev commented on GitHub (Jul 15, 2025):

@vbooka1 Glad to hear it. Replace the memory card just in case.

@mdevaev commented on GitHub (Jul 15, 2025): @vbooka1 Glad to hear it. Replace the memory card just in case.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
gh-pages
No results found.
Labels
Clear labels   
component:documentation

   
help wanted

   
resolution:delayed

   
resolution:duplicate

   
resolution:fixed

   
resolution:invalid

   
resolution:rejected

   
resolution:wontfix

   
success story

   
type:bug

   
type:bug

   
type:feature

   
type:question

   
type:question
No labels
component:documentation
help wanted
resolution:delayed
resolution:duplicate
resolution:fixed
resolution:invalid
resolution:rejected
resolution:wontfix
success story
type:bug
type:bug
type:feature
type:question
type:question
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/pikvm-pikvm#1044
No description provided.