mirror of
https://github.com/pikvm/pikvm.git
synced 2026-03-02 18:16:56 -05:00
Intermittent black screen in WebRTC when connected through an external network after some time #710
Labels
No labels
component:documentation
help wanted
resolution:delayed
resolution:duplicate
resolution:fixed
resolution:invalid
resolution:rejected
resolution:wontfix
success story
type:bug
type:bug
type:feature
type:question
type:question
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/pikvm-pikvm#710
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @plia7 on GitHub (Aug 11, 2023).
Originally assigned to: @mdevaev on GitHub.
Describe the bug
Intermittent black screen in WebRTC when connected through an external network after some time.
To Reproduce
Steps to reproduce the behavior, like:
Expected behavior
Being able to stay connected and use it for few hours without any interruptions assuming stable and fast internet connection on both ends.
Actual behavior
Intermittent black screen in WebRTC starts appearing after some time (could be after minutes or hours) where it works fine until that point, then prior to the black screen it might appear like the mouse movement is slower to react (although no observed internet quality reduction on both ends). Then when the black screen appears for few seconds or minutes, it shows like an orange sd memory card icon (all other icons don't have a color), and in the window caption it shows 0 fps:
These are my runtime settings:
When opening the log, it shows:
"23-08-09 21:39:00 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [WARN] [497280146060660] Waiting for candidates-done callback... (slow gathering, are you using STUN or TURN for Janus too, instead of just for users? Consider enabling full-trickle instead)
[2023-08-09 21:39:01 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [WARN] [497280146060660] Failed to add some remote candidates (added 0, expected 1)"
As well more detailed log for when the issue happens:
[2023-08-11 001109 kvmd.service] --.txt
After sometime it seem to restore the screen, but the mouse movement and general feel is that the connection is not as stable, and it keeps going back to black screen again after few seconds so it's not very useable at this point.
Note: If I switch to mjpeg, it seem to show the screen then, but doesn't seem like the mouse/input is very reactive in that mode. Trying to click stop stream and start stream while in webrtc doesn't seem to help.
Possible workaround although not always consistent
It seems like if I open an incognito browser for same pikvm or the second pikvm non-incognito web page for the same host pc it might work better for some time and the black screen issue will be gone, but after some time, it starts creeping in into the incognito or second pikvm session again so I have to keep altering between pikvm 1, incognito and pikvm 2 to "escape" the issue but this is not an ideal scenario and I rather not have any interruptions to begin with, just like when I was using it in my local network without this issue.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Windows 11
Google Chrome Version 114.0.5735.199 (Official Build) (64-bit)
or
Microsoft Edge Version 115.0.1901.183 (Official build) (64-bit)
PiKVM info:
PiKVM v3 Pre-Assembled
pacman -Q | grep kvmdkvmd 3.236-1
kvmd-fan 0.26-1
kvmd-oled 0.24-1
kvmd-platform-v3-hdmi-rpi4 3.238-1
kvmd-webterm 0.43-1
pacman -Q | grep ustreamerustreamer 5.41-1
uname -aLinux pikvm 5.15.68-3-rpi-ARCH #1 SMP Mon Oct 31 20:56:54 MSK 2022 armv7l GNU/Linux
Additional context
This is the same system setup I used to have in my local network with the connecting pc and host pc - and I never encountered this issue when everything was in the local network. I only started to encounter this issue when the connecting pc is on an external network (I haven't changed any settings from that "local network working setup").
Here is the main "trademark" of the issue, when it blinks it shows this prompt:
"H.264 - 1920x1080 / Invalid PeerConnection / 0 fps dynamic"
As you can see the sd memory card icon is still showing in green color (not yellow)
@mdevaev commented on GitHub (Aug 11, 2023):
Please update to the latest version and try to reproduce it again.
@plia7 commented on GitHub (Aug 12, 2023):
@mdevaev I updated to kvmd-platform-v3-hdmi-rpi4 3.244-1.
I'll keep you posted if I still encounter this issue with the latest version.
Thank you.
@plia7 commented on GitHub (Aug 14, 2023):
@mdevaev The black screen blinking continues in the latest version. I wouldn't say it's as rare as you would expect, but happens more frequently like every x amount of minutes, although it doesn't seem to last long this time around, it's still a frustrating experience overall.
Here are the logs:
logs 8.14.2023.txt
It happened like 2-3 times in the last 45 minutes session when I was connected. Although it doesn't seem to persist as long like before.
The logs mention things like (which I also mentioned above originally):
[2023-08-14 10:31:59 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [ERR] [ice.c:janus_ice_cb_nice_recv:3078] [1829142975794202] SRTCP unprotect error: srtp_err_status_replay_fail (len=140-->140)
And:
[2023-08-14 10:07:16 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [WARN] [7136366473408474] Waiting for candidates-done callback... (slow gathering, are you using STUN or TURN for Janus too, instead of just for users? Consider enabling full-trickle instead)
[2023-08-14 10:07:17 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [WARN] [7136366473408474] Failed to add some remote candidates (added 0, expected 1)
Internet is stable and fast on both connecting pc (using wifi) and host pc (host pc is using wired ethernet and as well as the pikvm):
Speed test in connecting pc:
Speed test in host pc:
Thank you.
@plia7 commented on GitHub (Aug 14, 2023):
@mdevaev Just had some strong encounter with the issue, it went to became slow again in conjunction with the black screen:
And the black screen seem to persist longer this time around which makes it quite unusable and unreliable when it happens and keeps happening during this session unless I go incognito with the same pikvm or start another pikvm session, here are the logs:
logs 8.14.2023 take 2.txt
Internet is still stable and fast on both connecting pc (using wifi) and host pc (host pc is using wired ethernet and as well as the pikvm):
Speed test in connecting pc:
Speed test in host pc:
Note: It seems like once a session gets a black screen, it seem to be "infected" with it at that point, it never recovers from it and keeps going back to it so it will keep altering to black screen or slowness even if the screen shows up again and might work for some time.
As you can see, there is no substantial slowdown in the quality or speed of the internet (speedtest screenshots).
Please let me know if you need to teamviewer to my machine again, so I could show you the problem if you can't replicate it on your side
Thank you.
@plia7 commented on GitHub (Aug 28, 2023):
Hi @mdevaev,
It's been 2 weeks since the issue was reported.
Are there any progress updates on the fix to the issue? Were you able to reproduce it? Or study the mentioned logs below?
For reference, here are all the latest js logs (including verbose) when the issue occurs which you said you will analyze that you requested that I sent you previously in private (after I got latest version of
kvmd-platform-v3-hdmi-rpi4 3.250-1):
message.txt
"destroying session" "got a slowlink event on session"
that's from 15:29 blink:
and this is from 15:24 blink:
just happened now at "14:01":
message (1).txt
2-3 blinks in a row:
message (2).txt
something interesting that this error comes up when the blink comes:
[15:49:59.733] LOG/INFO -- Stream [Janus]: Got a remote stream changes: MediaStream {id: 'janus', active: true, onaddtrack: null, onremovetrack: null, onactive: null, …} [MediaStreamTrack]
janus.js:1046 Destroying session 5747201551488948 (unload=false)
janus.js:1943 Remote track flowing again: Event {isTrusted: true, type: 'unmute', target: MediaStreamTrack, currentTarget: MediaStreamTrack, eventPhase: 2, …}
janus.js:1952 TypeError: Cannot read properties of null (reading 'addTrack')
at event.track.onunmute (janus.js:1949:28)
it keeps happening now in the session
like 6-7 times in a span of 5-10 mins
also another interesting thing is if I have both pikvms cloned on the same host pc (showing the same screen), one might blink, while the other might not, so I could still be able to do something on the second one, while the first one goes into a black screen and I can't do much, although first one has audio, second doesn't but I don't think it's related as the second one could blink too at some point, sometimes they even go into a black screen at the same time
message (3).txt
It happened in the log above 5 times in a short time span.
Please let me know if you need any additional information or have any questions.
We can have a TeamViewer session to my machine when the issue happens.
Thank you.
@mdevaev commented on GitHub (Aug 28, 2023):
Note: https://github.com/meetecho/janus-gateway/blob/0.x/html/janus.js#L1949
@plia7 commented on GitHub (Aug 28, 2023):
@mdevaev As discussed, here are the triangles you requested:
Thank you.
@Majestic7979 commented on GitHub (Sep 13, 2023):
I need to add myself to being affected by this but in my case I'm in a completely internal network.
I require webrtc/h264 because I need audio - I get audio notifications when there is a notification of new emails/messages coming through from coworkers so I know I need to open the tab to answer. This cannot be done on mjpeg. mjpeg works perfectly but I require the audio which only works on h264 mode. However, I keep getting these black screens and the latency is horrendous, I move the mouse and it does not immediately respond. On mjpeg there is a slight delay for cursor movement but it is reliable and usable. On h264 unusable. Please prioritize this, because not being able to see the screen of the remote device breaks the main use case of PiKVM. mjpeg is not an option due to the lack of audio.
@Majestic7979 commented on GitHub (Sep 13, 2023):
Whenever the screen goes black I get that addtrack error:
@Majestic7979 commented on GitHub (Sep 13, 2023):
Just wanna say changing the polling rate of the mouse to 10ms instead of the default 100ms greatly improved the pointed accuracy and speed. Consider changing the default to 10ms... and I tried to use webrtc in Firefox and have not had any black screens so far so it seems like you might need to investigate how it's working in Chrome, which is the most popular. For now I am fine using Firefox to get what I need to do my job but I am risking getting told off at work and losing my job because Chrome is the only allowed browser.
@plia7 commented on GitHub (Oct 1, 2023):
Weird, I never experienced it in an internal network, are you using wired ethernet cable or wifi? The problem is better for me when I'm using it from a different location, so it might depend on location, distance, ISP, etc.
@plia7 commented on GitHub (Oct 1, 2023):
Mine is already at 10ms. How using the browser is affecting the host pc?
@Majestic7979 commented on GitHub (Oct 2, 2023):
I don't know why I was having h264 dropouts in Chrome but not in Firefox. I also enabled the USB ethernet with ncm driver in the overrides.yaml and I'm serving wired ethernet via USB to the remote device through the PiKVM and this eliminated my issues. So I think it was a connectivity problem. Probably packet loss which Firefox is somehow handling better than Chrome.
@plia7 commented on GitHub (Oct 2, 2023):
The problem is you can't do keyboard shortcuts in firefox when in full screen unlike chrome.
@plia7 commented on GitHub (Oct 19, 2023):
Hi @mdevaev So I actually started to experience the issue again in other location, no internet speed loss in both connecting or host pc... So is it Tailscale again? and what's interesting that in Firefox there are less black screen moments than there are in google chrome... Is it something that can be improved so there is less black screen in chrome like in firefox for when this issue happens? And again, only from an external network and through h264 when this issue happens - I wonder if there is a way to clean the tailscale pipe (or reset it) so the issue stops... As it could work fine for weeks, months, and in one time, somehow have this massive black screen issue? Besides tailscale are there any other supported vpn alternatives?
Thanks.
@mdevaev commented on GitHub (Oct 25, 2023):
I've reported about js error: https://github.com/meetecho/janus-gateway/issues/3283
@plia7 commented on GitHub (Oct 26, 2023):
Thank you, here is the main "trademark" of the issue, when it blinks it shows this prompt:
"H.264 - 1920x1080 / Invalid PeerConnection / 0 fps dynamic"
As you can see the sd memory card icon is still showing in green color (not yellow)
@mdevaev commented on GitHub (Dec 14, 2023):
@plia7 @Majestic7979 I've released small update for this, just a workaround. I don't expect this to significantly improve the situation, but at least it should provide reconnection in case of a bug with unmute. My cursory research shows that this is some kind of bug in Chrome. Since the Janus upstream doesn't know what's going on, I'll try to fix it myself, somehow.
Please update PiKVM OS.
@Majestic7979 commented on GitHub (Dec 14, 2023):
Thanks @mdevaev you're a really nice guy and a responsive developer. I installed. I'll test and report back.
@mdevaev commented on GitHub (Dec 14, 2023):
@plia7 commented on GitHub (Dec 14, 2023):
Thanks, I'll update at my earliest convenience (although I'm not in the condition to test it at the moment).
@Majestic7979 commented on GitHub (Dec 15, 2023):
@mdevaev whatever you did, has cured the issue for me. Thanks again for your hard work on PiKVM. I wanna know if there is any way to improve the latency on h264? When I move the mouse cursor it takes about a second for it to catch up with the grey dot. Is this just how it's always going to be, or are there any optimizations I can do to make the cursor arrow follow the grey dot more responsively? I am also a relatively fast typist and when I am editing configuration files it also takes about a second for a character to appear after I've typed it. Sometimes the Shift key gets stuck and whatever I type is capitalized, and if I hit the dot key . I get a > instead... Shift randomly stays activated (verified through virtual keyboard) when I hold Shift down to capitalize an acronym for example. And sometimes I'll get repeated characters, for example instead of lemon I will get leeeeeeemon.
@mdevaev commented on GitHub (Dec 15, 2023):
Ur welcome. The latency when working through a VPN is an unavoidable evil. You can try reducing the mouse polling rate in the system menu and make h264 gop 0. The latency depends on the network conditions, at the moment everything that I could optimize for my part has already been done. I have it at 200ms between continents.
@plia7 commented on GitHub (Dec 16, 2023):
@mdevaev I updated to kvmd-platform-v3-hdmi-rpi4 3.288-1. I'll try to test when I get a chance and report back. Thanks.
@Majestic7979 commented on GitHub (Dec 18, 2023):
@mdevaev unfortunately the problem has returned.
@mdevaev commented on GitHub (Dec 18, 2023):
Well, that only fixed part of it. I will investigate further.
@Majestic7979 commented on GitHub (Dec 18, 2023):
Thank you
PS. Although it returned, after closing the tab and reopening - I had to do this a few times - it is stable now. So it's better than before, but not completely fixed.
@tgrrr commented on GitHub (Dec 20, 2023):
Hi,
I have the identical problem. It improved slightly after the update. However, still persists.
Same issue:
Desktop (please complete the following information):
Before Update:
PiKVM info:
After update:
PiKVM info:
[2023-12-18 04:48:26 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [252830446426366] Creating ICE agent (ICE Full mode, controlling)
[2023-12-18 04:48:27 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [WARN] [252830446426366] Waiting for candidates-done callback... (slow gathering, are you using STUN or TURN for Janus too, instead of just for users? Consider enabling full-trickle instead)
[2023-12-18 04:48:28 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [WARN] [252830446426366] Failed to add some remote candidates (added 0, expected 1)
[2023-12-18 04:48:29 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [252830446426366] The DTLS handshake has been completed
[2023-12-18 04:48:29 kvmd-janus.service] --- kvmd.apps.janus.runner INFO --- => [WARN] [252830446426366] Failed to add some remote candidates (added 0, expected 1)
[2023-12-18 04:48:29 kvmd.service] --- kvmd.apps.kvmd.streamer INFO --- => -- INFO [62753.927 stream] -- H264: Requested keyframe by a sink client
[2023-12-18 04:48:29 kvmd.service] --- kvmd.apps.kvmd.streamer INFO --- => -- INFO [62754.019 stream] -- H264: Requested keyframe by a sink client
[2023-12-18 04:48:29 kvmd.service] --- kvmd.apps.kvmd.streamer INFO --- => -- INFO [62754.628 stream] -- H264: Requested keyframe by a sink client
[2023-12-18 04:48:30 kvmd.service] --- kvmd.apps.kvmd.streamer INFO --- => -- INFO [62754.730 stream] -- H264: Requested keyframe by a sink client
[2023-12-18 04:48:30 kvmd.service] --- kvmd.apps.kvmd.streamer INFO --- => -- INFO [62755.297 stream] -- H264: Requested keyframe by a sink client
[2023-12-18 04:48:30 kvmd.service] --- kvmd.apps.kvmd.streamer INFO --- => -- INFO [62755.391 stream] -- H264: Requested keyframe by a sink client
PiKVM settings
@shamsevan commented on GitHub (Jan 23, 2024):
@mdevaev Hi ,
I am experiencing same issue frequently in chrome !
while trying to Firefox , the issue seems bit less but it is supper slow there.
The main frustating part for firefox is , it doesn't accept the keyboard shortcuts , which is making it pretty difficult to run and execute tasks. please give me a solution
@blackandcold commented on GitHub (Jan 24, 2024):
Have the same problem - strange it happens in Vivaldi (Chrome) and NOT in Firefox
@mdevaev commented on GitHub (Jan 24, 2024):
Is anyone who has this problem playing constantly ready to provide me with a TeamViewer session to their computer so that I can look at it?
@plia7 commented on GitHub (Jan 31, 2024):
Hi @mdevaev,
@shamsevan says the issue happens 10 times in an hour (every 5-6 minutes), he's on Discord and says he could let you remote to his machine to investigate (he uses the latest os version with the partial fix that you already did for this issue).
Thank you.
@Majestic7979 commented on GitHub (Feb 8, 2024):
@mdevaev I found an interesting thing that might help you. I'm using the latest build of PiKVM as of writing this (ran your curl command) so you know which version. Basically, until there is some audio playback the screen will keep cutting out. But the moment there's some audio playback, it stops and stays stable. I can reproduce this every single time. How I do is I logon to Windows while it's in JPEG mode, then I switch to WebRTC and it starts cutting out. I open the Windows volume slider (which makes a sound when you move and release it) and as soon as I hear the Windows make a sound, I get a stable connection. So it seems like until it has played some sound the screen cuts out!! I use WebRTC for the audio capability over HDMI too, which JPEG mode does not support. Please note that when I swap to WebRTC I have to open the PiKVM menu and move the volume to 100%. I am talking about the Windows volume slider just to make some sound. As long as the remote machine sends a sound the issue will stop. So that narrows it down further for you to investigate.
@mdevaev commented on GitHub (Feb 8, 2024):
Do I understand correctly that even if the sound is turned on in PiKVM, this is not enough, and you need the host to start playing the sound itself?
@Majestic7979 commented on GitHub (Feb 8, 2024):
That's what I was thinking until today but for some reason it was glitching again. I waited a few days just to make sure before making that comment, but today I had issues again. It does seem related to the audio. I'll be monitoring some more to see if it was just "luck" or if there really is a link to audio causing issues.
@plia7 commented on GitHub (Feb 8, 2024):
@Majestic7979 @blackandcold @tgrrr Just a quick update: Per the developer, after inspecting @shamsevan machine who can reproduce the issue every 6 minutes, this might require hiring of WebRTC team specialists (Meetecho) to further investigate the issue and find a resolution for it, so please stay tuned.
I'm hopeful we can have a remedy soon as we continue to struggle with the issue for the last half a year.
Thank you all for your cooperation and patience. And big thanks to @mdevaev who is willing to escalate this to the next level.
@RSATom commented on GitHub (Feb 19, 2024):
@mdevaev Can you please clarify, setup you are using to access streams from Janus server doesn't use TURN server at all, right?
@mdevaev commented on GitHub (Feb 19, 2024):
@RSATom yes, only STUN.
@RSATom commented on GitHub (Feb 20, 2024):
Ok, did you see something like "ICE state changed to failed" in JS console right before issue?
@mdevaev commented on GitHub (Feb 20, 2024):
@RSATom this is a first option. The second is many "remote track muted", see above.
@RSATom commented on GitHub (Feb 20, 2024):
I think there are multiple issues here. So it's required to split them and only after try to fix.
So, first of all it's required to understand how to simplify reproducing this issue. And I think it should be simplest part - if there is firewall present - one can just disable UDP packets passing through firewall and wait ICE disconnected (~1 second), ICE failed (~2 seconds).
Then browser client will try to restore connection. And there are 2 ways to restore WebRTC connection with Janus: full reconnect and ICE renegotiation. Since you are using custom plugin I can't say if it supports the second one (please clarify).
Other thing it's required to take into account - full reconnect with Janus can be tricky, especially if you don't detach from Janus plugin on Peer Connection drop. Below can mean exactly this case:
Maybe all above already known to you, please tell me. I just need to understand what exactly you already tried.
@mdevaev commented on GitHub (Feb 20, 2024):
No, it's not supported: https://github.com/pikvm/ustreamer/blob/master/janus/src/plugin.c
Although I wrote this, I have a fairly superficial idea of the internal structure of Janus. If there are any problems, I completely terminate the connection with Janus and re-establish it:
https://github.com/pikvm/kvmd/blob/master/web/share/js/kvm/stream_janus.js
Perhaps this is really due to packet loss, but users say that this problem either does not reproduce in Firefox or occurs much less frequently than in Chrome.
If it's convenient for you, let's get in touch via PM at discord: https://discord.gg/bpmXfz5 (mdevaev there)
@Majestic7979 commented on GitHub (Feb 20, 2024):
I can have the connection for hours without interruption on Firefox... this is only happening in Chrome for me.
@RandomJerk commented on GitHub (Feb 23, 2024):
I can confirm that Firefox works better and does not see black screen interruptions. The issue happens on Chrome and Chromium based browsers
@plia7 commented on GitHub (Feb 27, 2024):
Yes, maybe Firefox works a little better, but it doesn't support keyboard shortcuts , which is making it pretty difficult to run and execute tasks so it's not really a suitable workaround.
@RSATom commented on GitHub (Feb 28, 2024):
did anybody look into
chrome://webrtc-internals?@plia7 commented on GitHub (Feb 28, 2024):
I believe @mdevaev looked in mine few months ago when I had the issue. Or possibly he looked again recently in @shamsevan machine.
@RSATom commented on GitHub (Feb 28, 2024):
I just got an Idea it can be broken by some reason h264 video stream (i.e. ICE connection is fine)... And working Firefox maybe confirms it... Firefox and Chrome may have different codecs implementation inside.
@plia7 commented on GitHub (Feb 28, 2024):
Just did a little google search for the issue:
Similar issue from 2020 but not resolved:
Invalid peer connection after network change in WebRTC media chat
https://stackoverflow.com/questions/63326570/invalid-peer-connection-after-network-change-in-webrtc-media-chat
Another similar one (but maybe resolved?):
WEBRTC Relay Video to Third Peer (Video is Black With H264)
https://stackoverflow.com/questions/65079999/webrtc-relay-video-to-third-peer-video-is-black-with-h264
Where someone commented there:
"1
there is a known issue with this kind of forwarding. See this bug: https://issues.chromium.org/issues/40150839 –
Philipp Hancke
Dec 1, 2020 at 6:31
"
And someone also commented:
"Quick update if anyone ever lands on this: the issue is caused by a hardware acceleration bug in chrome. You can actually toggle off hardware acceleration and have it work just fine. It's not fixed as writing this. –
David Bell
Jan 12, 2021 at 23:29"
The status of the issue: https://issues.chromium.org/issues/40150839
Appears to be fixed. Maybe it's a similar issue or a variation of it?
Will turning off "hardware acceleration" (flag?) or "use graphics acceleration when available" setting (or maybe some other chrome flag or setting) could help prove it's related?
Some other possible related open tickets there:
Race: ICE restart before end-of-candidates suppresses second end-of-candidate event
https://issues.chromium.org/issues/40229069
Someone said "single connection fiddle with early restart.
Firefox behaves as expected."
And
Fuchsia: Incoming WebRTC video stream is flashing black when HW decoded
https://issues.chromium.org/issues/40901638
@mdevaev @RSATom Do you think some of these issues could be related or same issue/root cause and/or should we submit a new ticket there and let them figure it out?
Thank you.
@RSATom commented on GitHub (Feb 29, 2024):
@plia7 everything is possible. But to say more it's required to know if there is
failedstate happens for ICE before black screen appear. I'm still waiting my HDMI-to-CSI device to start debugging issue myself. I hope it will arrive on this weekend...@Majestic7979 commented on GitHub (Feb 29, 2024):
Just to be clear I never intended my comment to mean that using Firefox is a solution, definitely not, for the same reason. I need Chrome...
@RSATom commented on GitHub (Feb 29, 2024):
Firefox is slowly dying, so yes, it's not a solution in any case...
@Majestic7979 commented on GitHub (Feb 29, 2024):
Based on comment above I turned off H/w video decoding in chrome flags, and it makes no difference. I also noticed that in Firefox my mouse movements are more fluid, more responsive. The delay appears to be much less, so if I move the mouse from one corner of the screen to the opposite corner, I can see my mouse moving faster in Firefox, in Chrome there is a terrible delay. So Chrome is unusable in its own way, but it's the browser I need to use sometimes due to customer restrictions in their IT.
@plia7 commented on GitHub (Feb 29, 2024):
Did you also try to toggle off "use graphics acceleration when available" chrome setting?
I wonder why the mouse movement and usability appears to be slower in chrome compared to firefox. Is it only happening when the black screen starts showing up? Because I believe up to that point the mouse movement is ok. It looks like I did mention it in the ticket description: "After sometime it seem to restore the screen, but the mouse movement and general feel is that the connection is not as stable, and it keeps going back to black screen again after few seconds so it's not very useable at this point."
Thanks.
@plia7 commented on GitHub (Feb 29, 2024):
Looking at the logs on this ticket, I don't see ICE failed state, but it's maybe happening on the first time which causes it to go into this black screen loop (and logs typically gathered after that), so yes, maybe it's good to have full log for it, to make sure 100% if it happens or not so then a proper ticket can be submitted on the chromium bug report website (if it actually turns out not to be pikvm's fault in this case).
Thank you.
@RSATom commented on GitHub (Mar 3, 2024):
Finally I've got my HDMI-2-CSI device, did some debugging, and found with current implementation
failedICE state never happens. I think the reason is related to handlingdisconnectedICE state by browser andjanus.js:Important note about
disconnectedICE state: the name of this state can be confusing but it's not final state. If packets will start flow again - state will switch toconnectedstate again. And only if there will be no packets for too long amount time state will be switched tofailedstate. And it's final state.muteevent to track. Andjanus.jshandles it by scheduling track remove from internal Janus.js structures (but it doesn't mean track disappears from RTCPeerConnection, it's onlyjanus.jsinternal logic)unmute, after timeoutjanus.jsremoves track and sendsremotestreamevent. And this is the point wherestream_janus.jsgets confusing onremotestream without video tracks. I.e. it's intended behavior, and there is no reason to disconnect from janus at this point. The right time to start reconnect logic isfailedICE state (or even betterfailedstate of RTCPeerConnection.connectionstate (but obviously it can keep black screen for longer period of time)Unfortunately all above has nothing to do with "black screen" issue. When I enabled UDP packets flow on my test environment - reconnect to Janus happened successfully and I was able to see incoming video...
@plia7 commented on GitHub (Mar 3, 2024):
Ok so what does this all mean? Do you need the conditions to reproduce the issue to analyze it? Is it possible for you to connect with one of the pikvm users who reported the issue/have the issue and can reproduce it such as through a TeamViewer session to try to analyze the issue on their end (which is what @mdevaev tried to do but couldn't determine the root cause)? Or what is it that you need that will help you to get to the root cause of the issue? Do we need to send HDMI-2-CSI device to one of the users who have the issue?
Keep in mind it's a very specific set of conditions to reproduce the issue that includes being on an external network and connecting to tailscale (preferably the pivkm should have a long distance compared to your location like another country or even another continent is better which typically would result in a ~300ms+ ping when you ping the tailscale ip of the pikvm) , so I'm not sure if you are or can test all of this locally with a physical pikvm and without actually connecting to a remote pikvm through tailscale.
So either you need to connect to a remote pikvm (on an external network) through tailscale and see if you could get the intermittent black screen at some point and then start analyzing it with the HDMI-2-CSI device (if possible) or maybe some of the other members in this post have more suggestions or could help with providing you with the conditions needed to reproduce the issue/analyze it when it happens.
Thanks.
@RSATom commented on GitHub (Mar 3, 2024):
@plia7 I just shared my findings, nothing more at this moment. HDMI-2-CSI - is device all pikvm users already have actually. It's where you put you HDMI cable :)
Also, I've found another one possible issue and working with @mdevaev on it.
@plia7 commented on GitHub (Mar 3, 2024):
@mdevaev reached out to me and asked if someone currently using the external network with tailscale and is able to reproduce the issue - @RandomJerk @blackandcold @tgrrr @shamsevan who commented on this post or anyone else who read this - Are you available to assist through a TeamViewer/Discord session so the issue can be inspected/further analyzed by @mdevaev and @RSATom?
Thank you.
@RSATom commented on GitHub (Mar 3, 2024):
it would be great if someone could provide me remote access to pikvm with reproducible issue. I don't need real server, I just need some video stream. It can be even some tv stick attached to pikvm...
@plia7 commented on GitHub (Mar 3, 2024):
@RSATom @mdevaev I found 10 people who have the issue, we created a group in Discord. Please come and join us.
@Majestic7979 commented on GitHub (Mar 4, 2024):
I feel I need to just remind all reading at this stage that I do not use tailscale. I am accessing the PiKVM from within the internal network... so I'm on 192.168.1.20 and the PiKVM is 192.168.1.21, for example. I get the issue without any VPN or tunneling.
@Majestic7979 commented on GitHub (Mar 4, 2024):
Are you able to send me your email address? I will get in touch.
@RSATom commented on GitHub (Mar 4, 2024):
@Majestic7979 rsatom_at_gmail_dot_com
@RSATom commented on GitHub (Mar 4, 2024):
Are 192.168.1.20 and 192.168.1.21 connected to the same Ethernet switch?
@Majestic7979 commented on GitHub (Mar 5, 2024):
Thanks, you can remove your email now, I've noted it. I'll be writing soon.
I'll be bringing the PiKVM to my house, where I have it connected to my UniFi network. It will be connected to a UniFi Flex Mini, that will connect to a USW-24-PoE, and that in turn connected to a UniFi Dream Machine. But it's all the same network within the router configuration.
@RSATom commented on GitHub (Mar 5, 2024):
👍 but there is no need to do it. It shown on my github profile anyway 😃
and there is no any WiFi in between, right?
@Majestic7979 commented on GitHub (Mar 5, 2024):
I have an Access Point of course, only for the wireless devices. Not for the PiKVM and not for the PC I'll be using because they can be wired and that's how I prefer it.
@Majestic7979 commented on GitHub (Mar 5, 2024):
I emailed you now, @RSATom
@Majestic7979 commented on GitHub (Mar 6, 2024):
As a means of updating everyone I'm writing here. @RSATom logged in to my PC via zoom yesterday and showed me how to capture the data he needed to analyze the issue. He now has all the necessary logs, so hopefully some sort of fix can come out of it. I want to thank him for helping and being patient.
@RSATom commented on GitHub (Mar 6, 2024):
@Majestic7979 unfortunately I don't have good news for you. According to logs your browser just didn't get UDP packets periodically. So it's network (more likely) or pikvm side (less likely) issue. Did you use VPN yesterday when did issue reproducing?
@Majestic7979 commented on GitHub (Mar 6, 2024):
@RSATom No VPN. Oh :( But it's weird that it doesn't happen on Firefox, only on Chrome... so while it may be the case that network can play a role in this issue, the way it's handled is different. Isn't there a way to code that janus server to stop cutting out? What if it just waits for more UDP packets? I don't know how it works so of course it's just me speculating...
@RSATom commented on GitHub (Mar 6, 2024):
Yes, it's confusing me too...
And UDP packets are sending always while peer connection is active, without waiting something...
@Majestic7979 commented on GitHub (Mar 6, 2024):
Right, I thought so, just like a IP CCTV camera works. But in this case the connection is being cut out. I can see when it goes black on the console that it says the peer connection is down. So on chrome something is happening to break the connection... Maybe this is happening in Firefox too, however it doesn't cause any black screens. On Firefox occasionally I get the hiccup on the video where it goes slower but it never completely goes black and cuts out.
@RSATom commented on GitHub (Mar 6, 2024):
can you please try run
ping -t ip(if browser is running on windows) from host where browser is running to pikvm (or router in front of it)?also, if Firefox experiencing the same issue you should be able to see
slowlinkevents in JS console.@RSATom commented on GitHub (Mar 6, 2024):
if it uses RTSP protocol - yes, very similar, but RTSP can send data over TCP in some cases.
@Majestic7979 commented on GitHub (Mar 6, 2024):
I am filtering for "slowlink" in the Firefox console. I'll come back to you soon saying if there was something or not.
@RSATom commented on GitHub (Mar 6, 2024):
it's better keep ping running for long period of time and check output it when you'll get black screen issue.
@Majestic7979 commented on GitHub (Mar 6, 2024):
I understand that PiKVM wants UDP for the lower latency. Is it possible to implement also TCP option (user can configure in override.yaml) for the janus server to see if that could resolve? At a price of higher latency but more reliability... It's a mystery isn't it?
@Majestic7979 commented on GitHub (Mar 6, 2024):
Ah I see. No problem, I will keep it running.
@Majestic7979 commented on GitHub (Mar 6, 2024):
Got the slowlink on Firefox now:
And the ping:
I will continue to run the ping if you need much more info than that. The black screen happens the moment I start using Chrome... it's almost always instant issue.
@RSATom commented on GitHub (Mar 6, 2024):
@mdevaev if I understand correctly jpeg mode uses TCP, right?
@Majestic7979 if I remember correctly Janus doesn't have TCP support. But it's still possible to get it with TURN server.
@Majestic7979 commented on GitHub (Mar 6, 2024):
I'll have to leave the technical details to you cleverer boys :)
@RSATom commented on GitHub (Mar 6, 2024):
hm... there is no packets loss with ping...
@Majestic7979 commented on GitHub (Mar 6, 2024):
Also, I just had the black screen again after just switching to the tab on Chrome and the ping is always less than 1ms... so the network side seems healthy in terms of latency.
@Majestic7979 commented on GitHub (Mar 6, 2024):
I hope I don't sound like an asshole, it's not my intention but if there is a technical way to try the TCP could we do that? Just maybe to rule out that solution? I can see that you asked Maxim, hopefully he can let us know 🤞🏻
@RSATom commented on GitHub (Mar 6, 2024):
Did you see something like picture freezes in Firefox? it's possible it just don't switch to black screen on
videotag detach from WebRTC stream. But i'm not sure...@Majestic7979 commented on GitHub (Mar 6, 2024):
On Firefox I sometimes see slowness in the picture, for example when I move the mouse cursor it will have a delay between the cursor moving and the final position of the blue dot. But never ever black screen.
This is how I have it set up on Firefox:
@Majestic7979 commented on GitHub (Mar 6, 2024):
One thing that I noticed is that on Firefox I have only received 2 slowlinks so far... On Chrome the issue happens pretty much every few seconds...
@RSATom commented on GitHub (Mar 6, 2024):
Don't worry, everything is ok.
I see Janus supports TURN server directly
So maybe simplest way to try it is find/install/buy TURN server and add it to config.
@RSATom commented on GitHub (Mar 6, 2024):
Hm... It's very strange and I can't explain it right now....
@Majestic7979 commented on GitHub (Mar 6, 2024):
If you can send me instructions I'll follow, to ensure I'm not doing something I think is correct but is totally wrong 😂
@Majestic7979 commented on GitHub (Mar 6, 2024):
Also:
@RSATom commented on GitHub (Mar 6, 2024):
@Majestic7979 you can try https://xirsys.com/
If I remember correctly they provided free accounts some way.
And it's ok to contact me if you'll need assistance to configure your pikvm...
@Majestic7979 commented on GitHub (Mar 6, 2024):
I understand you're sleeping at this time so I'll check back later.
I installed "CoTURN" in docker making sure the relevant ports are exposed, I am ready to test and see what happens. How to proceed to add the TURN server I'm running locally? I understand it may not be functional but I want to know how to set up the PiKVM at least. I'll test and see if I can make it work. If not, I appreciate the open invite to get your assistance :)
@Majestic7979 commented on GitHub (Mar 6, 2024):
Another question - if both devices are in the same network/subnet, why do I need a TURN server at all!?
I mentioned in the past that I am using this:
https://docs.pikvm.org/usb_ethernet/
So the device connected to the PiKVM is receiving networking from the PiKVM via USB.
The network subnet will therefore be different. Is this what may be causing the problem? I have used USB networking because it eliminates one more cable and the bandwidth is acceptable over USB since the server in question is for monitoring purposes, it does not do any heavy data transfers.
@Majestic7979 commented on GitHub (Mar 6, 2024):
Just some food for thought... Why is Janus the preferred solution instead of something like go2rtc? Could testing with a different webrtc server maybe offer a solution? Only trying to look at a way of solving this that may be obvious but not yet thought about...
@mdevaev commented on GitHub (Mar 6, 2024):
Switching to another rtc gateway will lead to new problems that did not exist with janus.
@RSATom commented on GitHub (Mar 7, 2024):
@Majestic7979 you can find Janus config on pikvm at
/etc/kvmd/janus/janus.jcfgto configure TURN you should add related lines under the
natsection Unfortunately I never used Janus with NAT configured that way myself (usually it used on client side), so can't say how/if it will work.@Majestic7979 commented on GitHub (Mar 7, 2024):
I'm gonna sit and wait for you guys to explore solutions for these reasons:
Thanks guys, I'll just be patient and let you work on this. I hope you can find a solution for all of us your fans and users sooner rather than later... Wishing you all good luck! I will be watching in case I can provide further info but otherwise just taking myself out of the game, let me know if you need anything from me :-D
@Majestic7979 commented on GitHub (Mar 7, 2024):
Some extra info for you @RSATom, you are aware that I was using the Ethernet over USB, so the subnet was not the same as the main network where I'm accessing from the browser. When I connected the device to WiFi (to test on the same subnet) the black screens in Chrome completely stopped. So yes, this appears to be a NAT issue. But I run something called Frigate NVR in my home lab, this is for security cameras. Frigate uses an implementation go2rtc for WebRTC streaming. If I access from my 5G phone completely bypassing my subnet, it works flawless for streaming UDP cameras (I mean flawless in the sense that the cameras connect normally and stream normally, but PiKVM is not a camera, so it could be different).
I'd be interested to test your implementation of Gstreamer to see if this issue goes away. I really wanted to try your potential solution of TURN server but I don't think I can configure CoTURN in my home lab behind my reverse proxy in a correct manner so I don't want to introduce another issue in the mix... Trying to keep it as simple as possible to help you :)
@RSATom commented on GitHub (Mar 7, 2024):
@Majestic7979 I'm a little bit confused with your network configuration... So let me ask you some questions:
@Majestic7979 commented on GitHub (Mar 7, 2024):
Ok I'll clarify, I'll try to provide the detail how it makes sense.
I have Device I want to manage, this device has HDMI output plugged into input of PiKVM so I can see the device screen. I have USB-C connected from the device to the PiKVM so I can do mouse and keyboard from PiKVM in the browser. I have only 1 ethernet cable plugged into the PiKVM. I think I confused you. I did not use a USB adaptor to provide ethernet to the PiKVM, I plugged the ethernet cable directly into the PiKVM and I am using instructions from here to pass through the ethernet to the device via the PiKVM (because I only have 1 ethernet cable in this location): https://docs.pikvm.org/usb_ethernet/
This device was originally connected via WiFi for simplicity, because wiring an extra cable to the device was going to have higher costs. I could not fit an extra switch. I have a VPN into that device so I can monitor it remotely via SSH. Because the WiFi is meshed (Ubiquiti UniFi) any brief roaming events from one AP to another due to signal variations would cause the VPN to disconnect and break the tunnel. So I had to drop WiFi. A solution I found was to use the Ethernet over USB on the PiKVM, so I could use the single ethernet cable to have the PiKVM on the wired ethernet, and then share the connection over USB (https://docs.pikvm.org/usb_ethernet/).
If you follow all the steps on the above instructions, you can successfully provide ethernet via USB with internet access, but the PiKVM will be on Subnet A and the connection provided to the device will be on Subnet B (because the instructions themselves say it can't be on the same subnet of the PiKVM).
Therefore as it's not on the same subnet, it's doing NAT (as I understand). But all devices are managed in the same network (as in the same router). Maybe a way to think of it is to pretend that the PiKVM is on VLAN 1, the device is on VLAN 2.
Does this all make sense to you so far?
So what have I done: I disabled the OTG Ethernet over USB on the Override.yaml inside /etc/kvmd and I reconnected the device to WiFi like it was before. And the black screens are totally gone. Because now the PiKVM and the device are on the same subnet (or VLAN 1). On the same subnet the issue does not occur. On different subnets/networks the black screen occurs. The problem I have now is that if I use WiFi, I get VPN disconnections that break the tunnel. If I use USB over Ethernet through the PiKVM to share with the device, I get black screens. The physical solution is to pull an extra cable (I can't add a switch). But that still does not fix the issue with the PiKVM not handling this type of situation, which now make sense when people are saying it happens on Tailscale because it's doing NAT as well. So basically, the workaround of pulling a new cable won't fix the issue for everyone else.
@Majestic7979 commented on GitHub (Mar 7, 2024):
Here's a silly drawing if it helps to clarify
@RSATom commented on GitHub (Mar 7, 2024):
@Majestic7979 Now I think I understood... did you try to touch your USB to Ethernet device after some period of time? is it hot?
@Majestic7979 commented on GitHub (Mar 7, 2024):
There's no USB to ethernet device :-P The PiKVM has an ethernet cable coming directly from the router plugged into the ethernet port. If you open the pikvm document I linked (https://docs.pikvm.org/usb_ethernet/) you can see there is a way to make the PiKVM provide a virtual ethernet to the device, just like it provides a virtual keyboard and mouse... So the PiKVM is is just passing through the ethernet (sharing it) with the device as a virtual ethernet via USB the same way the keyboard and mouse is provided to the device virtually via USB... the device gets a virtual ethernet connection... There is no physical USB to ethernet device anywhere in this process, the PiKVM is providing keyboard, mouse and ethernet devices virtually over the USB connection...
@RSATom commented on GitHub (Mar 7, 2024):
Ah... sorry, got it now.
@Majestic7979 commented on GitHub (Mar 7, 2024):
If you have a PiKVM using ethernet please follow the instructions and you will see that the device connected to the PiKVM will get a new ethernet interface, but it's coming via USB from the PiKVM :-)
@Majestic7979 commented on GitHub (Mar 7, 2024):
But that virtual ethernet must be (according to the instructions) on a different subnet...
@RSATom commented on GitHub (Mar 7, 2024):
Got it. Thanks! I need some time to understand how it works internally to understand why it breaks pikvm....
@Majestic7979 commented on GitHub (Mar 7, 2024):
No problem! Here is how this setup looks in Override.yaml inside /etc/kvmd:
@Majestic7979 commented on GitHub (Mar 7, 2024):
Just to clarify the above, you can see that the virtual ethernet is assigning to the device an IP in subnet 10.0.1.x whereas the browser is running on a computer on the subnet 10.0.0.x
@RSATom commented on GitHub (Mar 7, 2024):
@Majestic7979 i.e. computer with browser is running on the same subnet where pikvm connected physically with ethernet cable, right?
@RSATom commented on GitHub (Mar 7, 2024):
@plia7 are you using https://docs.pikvm.org/usb_ethernet/ on your pikvm?
@plia7 commented on GitHub (Mar 7, 2024):
@RSATom No, I'm using the physical ethernet port and it works fine. Should we move @Majestic7979 thread/comments to a new ticket given it's only caused by his usb virtual ethernet (and it works fine for him otherwise)? Since it's not really related at this point to the original issue?
Suggestion: Use his existing ticket https://github.com/pikvm/pikvm/issues/1170 and rename the subject to "janus.js on Google Chrome black screen, or streaming cuts off periodically when using usb virtual ethernet in local network"
@RSATom The issue seem like the same symptoms but looks like two different root causes given the setup difference to get these symptoms. Hopefully resolving one of them should shed some light on the other one.
Thanks.
@Majestic7979 commented on GitHub (Mar 7, 2024):
Yes correct. And to clarify if I am using the passthrough ethernet as explained above then the remote device gets a different subnet from the PiKVM virtual ethernet adaptor. But the PiKVM itself is on the same subnet as the device used to run Chrome.
@Majestic7979 commented on GitHub (Mar 7, 2024):
I am not against your idea, I hope it does not mean my issue gets less attention... Because this does seem related to a network route problem/NAT problem in both instances... whatever is decided I just ask that my issue is not forgotten, thank you 🙏🏻
@plia7 commented on GitHub (Mar 7, 2024):
Let's let @RSATom decide if it's related or not, in both cases I suppose it should work, so if my ticket gets more attention and helps to resolve yours and it leads at the end to some enlightenment with regards to the original issue I don't mind for you to keep using this thread as we both benefit 😀
The other thing is it seem like your issue is more easily and consistently reproducible, so hopefully someone else could set up your setup and see if they can reproduce this issue or not. Because I suppose at some point, whoever created this feature of usb virtual ethernet (including adding it to the wiki) was able to test and confirm that's it's working including @mdevaev so I wonder why it doesn't work for you in webrtc, or was it never tested in webrtc? 🤔
@Majestic7979 Frankly I don't mind volunteering to see if I can setup the same configuration (if it's not too difficult) as you and see if I can reproduce it. Are you able to join us in the Discord group?
@Majestic7979 commented on GitHub (Mar 7, 2024):
It's literally just using the above link to enable the option... you can just copy my code for the override.yaml and add to yours, provided your override.yaml does not have extra CD/Flash drives you should have enough "lanes" to have keyboard, mouse and usb ethernet. Then plug the USB cable into the device you want to control, and you'll see it will get a new ethernet connection. Just disable the wifi on that device and any other ethernet connections, leaving only the pikvm virtual ethernet over usb to provide the connectivity. Then go to a separate device to run the chrome browser and try webrtc/h264 making sure that the audio slider is 100%. The device connected to the PiKVM is running Windows 10. The device I use for Chrome is running Windows 11. That's basically the setup. This issue seems to occur only when your browser device and the device being controlled are on separate subnets.
To clarify the network:
Device used with Chrome (to view KVM): 10.0.0.0
PiKVM itself: 10.0.0.0
Device connected to PiKVM and getting virtual ethernet from PiKVM over USB: 10.0.1.0
@plia7 commented on GitHub (Mar 7, 2024):
How can you see the subnets and how can you control them so they are on different subnets?
How do you access the pikvm from another device, what url do you type and how do you find what's that url?
@RSATom commented on GitHub (Mar 7, 2024):
@Majestic7979 I have an idea about possible reason of your issue... Can you please tell what Raspberry Pi version your pikvm device based on?
@Majestic7979 commented on GitHub (Mar 7, 2024):
I will try to explain.
My router: 10.0.0.1
My PiKVM: 10.0.0.225
My PC (with Chrome): 10.0.0.16
Remote device (which the PiKVM controls): 10.0.1.3
Router gives IP to my PC running Chrome. Router gives IP to the PiKVM. Both of these devices are therefore on the same subnet (10.0.0.x).
PiKVM is connected to the Remote device via USB, that is how we all get keyboard and mouse. You can also enable a ethernet passtrough, if you do this it will create a USB-to-Ethernet bridge internally in the PiKVM, this can be enabled by the stanza in override.yaml that I posted above, specifically this part:
The bridge gives the Remote device a ethernet connection, with a subnet of 10.0.1.x, therefore the IP is 10.0.1.3 normally.
On the documentation the example is:
But it does not matter. You only have to make sure the subnet is different. I cannot do this:
because it is the same subnet of my router. The documentation makes it clear that this cannot happen. So I changed mine from 10.65.0.0 to 10.0.1.0. It does not matter what it is, as long as it's not the same as the subnet being used by the PiKVM from the router (10.0.0.0).
You should read the https://docs.pikvm.org/usb_ethernet/ in full to understand this process.
@Majestic7979 commented on GitHub (Mar 7, 2024):
My PiKVM is a Raspberry Pi 4 with the official V3 HAT, specifically it is this one here: https://shop.pimoroni.com/products/pikvm-full-kit?variant=40006251479123
This kit contains the steel case, the PiKVM V3 HAT and a Raspberry Pi 4 2GB RAM.
@RSATom commented on GitHub (Mar 7, 2024):
@Majestic7979 do you have your server connected to pikvm with usb 2.0 or 3.0?
@Majestic7979 commented on GitHub (Mar 7, 2024):
It will be whatever the USB cable coming out of the PiKVM is... the server can support USB 3 because it's a USB-C/Thunderbolt port. The Windows networking on the server is showing this info for the USB ethernet:
I don't know how the implementation was done in the PiKVM for this feature so I don't know if USB speed makes a difference but we all know USB 2.0 is capped at 480Mbps so seems like the speed is USB 2.0, as we know there's keyboard and mouse also being transmitted over that connection. As you have coding experience are you able to locate and check the PiKVM source code to see how it's programmed? Maybe just to confirm if there is a hard coded limit... because if the usb ethernet was coded to only provide a max of USB 2 speed then that's all I'll ever get.
Having said the above stuff I don't know if the port from the PiKVM V3 hat is providing USB3 or USB2 (I'm talking about the port near the HDMI and ATX ports on the V3 HAT), which is USB-C and provides the emulated keyboard and mouse to the server, and also the usb ethernet if enabled. All the functions are provided by that one single connection.
@blackandcold commented on GitHub (Mar 7, 2024):
Hi guys, I do not have much time (job, child, new home) to help but reading your findings.
I saw the drops only in Chromium based browsers, namely Chrome and Vivaldi. Safari works flawless and Firefox also very well. Even after Mac sleep the KVM instantly reconnect the session.
Quality settings do not change
No USB dongle and 3 PiKVMs running with the same behaviour in the same network or remote over Tailscale.
Most certainly there is some UDP shananigans in the Chrome network code causing the image to instantly go black.
Bitrate and framerate most of the time show activity / traffic.
Mouse input is forwarded and when having two sessions open (Vivaldi, Safari) I can watch the mouse move on the Server in the Safari session.
My own subnet:
I have PiKVM <=> 1GBit unmanaged Switch <=Wifi AP=> M1 Mac
or PiKVM <=> 1GBit built in Switch in the AP <=> PC (EndeavourOS or Windows)
Remote:
or PiKVM <=> Tailscale <=> PC / Mac
The problem seems independent of network topology, OS and IP subnets.
Probably leading janus to terminate the stream:
All PiKVMs are V3 HAT, Pi 4 4G with HDMI to CSI-2 board (Geekworm KVM-A3 Kit for Raspberry Pi 4 on Amazon).
I am a developer so hit me with janus debugging and stack traces. I'd change the JS to some containing code, setting some breakpoints and inspecting the stack before the termination.
Left Safari, right Vivaldi :)
https://github.com/pikvm/pikvm/assets/141038/16d1976c-73e5-474e-be4d-fa40031b9469
@plia7 commented on GitHub (Mar 7, 2024):
Great info, I hope this helps PiKVM dev team to progress and finally resolve this issue.
In the meanwhile, I also posted this issue on Chromium Bugs Tracker website:
https://issues.chromium.org/issues/328677967
And Janus github issues website:
https://github.com/meetecho/janus-gateway/issues/3341
I encourage all participants to contribute more info on these tickets.
Thank you.
@RSATom commented on GitHub (Mar 7, 2024):
@blackandcold is it possible for you to create WebRTC-Internals dump when issue is happens?
How long it takes to get issue after connect to pikvm from Chrome?
@Majestic7979 commented on GitHub (Mar 8, 2024):
I have a good feeling about this... thanks guys!
@plia7 commented on GitHub (Mar 8, 2024):
@RSATom @mdevaev The replies on the respective tickets, in janus:
"Iminiero • 5h Member "noticed that janus.js (on developer tools) keeps saying that ICE disconnected" - Then this is the cause. Not an issue in Janus, IMHO."
In Chromium:
"As per comment#1, issue seems out of scope for TE to triage which requires pikvm from an external network using tailscale installed on the external network connecting machine, hence adding TE-NeedsTriageHelp and requesting respective dev team to take a look and provide further inputs. Thanks..!!"
I guess we'll need to add some substance to these tickets that shows it's either janus.js or chromium issue and not a pikvm issue.
@RSATom Were you able to collect the debugging info from the discord war room group or they not able to reproduce on demand like Shams or not the same issue as "invalid peer connection" but different error? In that case, we might need some help with other users who have the exact issue described in this ticket.
Thank you.
@RSATom commented on GitHub (Mar 8, 2024):
@mdevaev I found why Firefox behaves better:
i.e. it doesn't generate
mutedevent for track indisconnectedstate. In that case with current implementation pikvm starts reconnect to Janus and it givesblack screenissue...@Majestic7979 commented on GitHub (Mar 10, 2024):
@RSATom is this an easy fix? If so do you have an estimate how long long it could take to fix it?
@mdevaev commented on GitHub (Mar 10, 2024):
@Majestic7979 It's not so easy to fix. First, we need to understand which behavior is correct.
@RSATom commented on GitHub (Mar 10, 2024):
@Majestic7979 for you case I think the problem is in USB throughput through linux core. And since UDP is treated as unreliable it's the first thing linux core drops in that case.
@plia7 commented on GitHub (Mar 10, 2024):
@RSATom @mdevaev This is what janus.js dev said about this: lminiero commented 2 minutes ago
"If you're reacting to muted events by removing the video element (or the video stream from the element), just stop doing that and Chrome will behave like Firefox." https://github.com/meetecho/janus-gateway/issues/3341#issuecomment-1987341707
I also asked Chromium devs (still waiting for a response): https://issues.chromium.org/issues/328677967#comment4
@mdevaev As a short term solution (until we understand the correct behavior for the long term solution), maybe we can add this as an experimental option/flag (i.e. dontDestroyJanusDuringMuteEventFlag) in the pikvm ui (disabled/false by default), where if it's enabled, it will not destroy janus/cause black screen versus will destroy it, so people can try both options and compare to see which works better for them and if it helps to reduce the black screen (specifically it will be used by those (who)/(when they) experience the black screen)? Just an idea...
This is the proposed additional condition to add in bold (assuming this code fires during a mute event):
if (!has_video && __isOnline() && !dontDestroyJanusDuringMuteEventFlag) {
__destroyJanus();
}
Thank you.
@RSATom commented on GitHub (Mar 10, 2024):
The problem is right now pikvm reconnect logic relies on current behavior (and btw, it looks like reconnect is broken in Firefox), and it's required to find other trigger to start Peer Connection reconnect.
https://janus.discourse.group/t/the-latest-chrome-doesnt-assign-failed-to-iceconnectionstate/910/2
@mdevaev commented on GitHub (Mar 11, 2024):
I have built the patches into a release, please update pikvm and check if the problem remains.
@RSATom commented on GitHub (Mar 11, 2024):
please keep in mind, mentioned patch will not improve your network... But you should see less amount of black screens (now it will be picture freezes mostly)
@Majestic7979 commented on GitHub (Mar 11, 2024):
Without a doubt whatever you did in the patch has made a difference. Now I see very brief black flashing at most, with no full loss of video. Only the mouse is still very laggy, but I can sorta live with that. You guys f&*^%ing blow my mind!!!!!!! I am so impressed by how fast you react to issues. God bless you guys for being awesome like that. It's a bit of a pity that the USB solution can't handle everything fully. I wonder if there is a way I can set up the quality sliders to reduce the video and audio bandwidth therefore improving the connection further? I noticed that there's no way to configure the audio bitrate and quality, for example... Could we have that added please? In theory if it's the bandwidth issue then making the video and audio less detailed could improve the mouse lag and dropped UDP frames?
@RSATom commented on GitHub (Mar 11, 2024):
@Majestic7979 you can try reduce max fps (in pikvm settings) and screen resolution (on your server) - it should help a little bit.
@mdevaev do you know, if it's possible to do the same passthrough etherenet connection but with USB 3.0 on RPi 4 ?
@Majestic7979 one more idea (but I didn't try it myself, and which maybe will require some work/time on configuration) - you can try to put USB Ethernet adapter to your RPi, configure your pikvm as router, and maybe it will give you additional throughput...
@Majestic7979 commented on GitHub (Mar 11, 2024):
I reduced the sliders and it's slightly better for sure. If it was possible to do via USB3 I would be happy to test and let you know the result.
As for configuring the PiKVM itself as a router, I'm afraid this will mess up the complicated networking situation even further? How does that work? So my ethernet cable goes into the PiKVM and then...? I am sorry, I'm not understanding what you mean by using it as a router... could you explain more please?
@mdevaev commented on GitHub (Mar 11, 2024):
@Majestic7979 The audio is compressed and consumes very little traffic. You can reduce the H.264 bitrate to a minimum, maybe this will help.
@RSATom USB 3.0 is not supported by USB core hardware on Pi, so it's not possible at all.
@RSATom commented on GitHub (Mar 11, 2024):
@Majestic7979 your pikvm already acts as a router - it routes traffic from virtual Ethernet adapter to physical ethernet adapter. So the only difference is use physical usb to ethernet adapter attached to USB on RPi...
Oops...
@Majestic7979 commented on GitHub (Mar 11, 2024):
Don't worry guys, I've been working for an hour now without any irritating black screen issues, so fingers crossed for good luck that it has resolved most of the issue. Guess what, after setting up like this:
My mouse is pretty much in sync, very little lag... Pull my legs I can't believe it haha
🍻🍻🍻🍻 Sending a box of virtual beer to the best guys on Github 😎
@Majestic7979 commented on GitHub (Mar 11, 2024):
Is it possible to set up the volume to be 100% as default? Every time I open a browser window I have to go into the menu and drag the slider from 0 to 100.. not an issue but would be amazing if it was possible to set the default volume as 100% :-)
@RSATom commented on GitHub (Mar 11, 2024):
@Majestic7979 it's better increase gop to max - it actually reduces traffic amount....
also, FYI https://forums.raspberrypi.com/viewtopic.php?t=317326
@Majestic7979 commented on GitHub (Mar 11, 2024):
Thanks I increased it per your suggestion. I thought reducing the GOP would give more responsiveness as it's having to compute new frame for every frame..
Anyway this is perfect now, I have to accept that it's as good as it gets for now. Only thing that remains is setting the volume to a default of 100% so I don't have to ever open that menu again LOL. I have a remaining issue with the keyboard but I'll open a separate issue later for that.
@mdevaev commented on GitHub (Mar 14, 2024):
Has everyone's problem been solved now?
@blackandcold commented on GitHub (Mar 14, 2024):
Chrome works now. Vivaldi showing that is recives data, just not showing the image. Web Sockets open.
I don't know why Vivaldi does not show the content of the element but this is a different problem.
Thanks!
Edit: yeah found the evil, Vivaldi did/does block autoplay on PiKVM (probably by default), so the metadata is updated in the JS but the stream is just not shown. Hope this helps someone.
@mdevaev commented on GitHub (Mar 16, 2024):
Perfect. So I'm finally closing it.
@RSATom thanks again for your research and patches.