mirror of
https://github.com/AdguardTeam/AdGuardHome.git
synced 2026-03-04 00:01:12 -05:00
DOH requests hang after a network reconnection #2822
Labels
No labels
P1: Critical
P2: High
P3: Medium
P4: Low
UI
bug
cannot reproduce
compatibility
dependencies
docker
documentation
duplicate
enhancement
enhancement
external libs
feature request
good first issue
help wanted
infrastructure
invalid
localization
needs investigation
performance
potential-duplicate
question
recurrent
research
snap
waiting for data
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/AdGuardHome#2822
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @spiralshapeturtle on GitHub (Jun 1, 2021).
Originally assigned to: @ainar-g on GitHub.
0.106.3
docker
Adguard -> router -> ISP
OPNsense
Intel
Ubuntu running with Docker containers
Expected Behavior
adguard should always keeps processing DNS requests.
Actual Behavior
Since the latest updates, AdGuard stops processing DNS requests. Even in the same vlan (excluding the router) the software stops listing to DNS requests. Logs keep empty, webgui and ping still works (no network issue)
Screenshots
PRTG polling my DNS setup, also pihole is deployed in the same setup an never show an alarm. It seems for now only related tot adguard.
Additional Information
sudo docker run -d --name adguard --hostname adguard --net=hostnetwork --ip=172.16.1.253 --cap-add=NET_ADMIN --cap-add=CAP_NET_RAW --cap-add=CAP_NET_BIND_SERVICE -e TZ=Europe/Amsterdam -v /home/rodeho/docker/adguard/workdir:/opt/adguardhome/work -v /home/rodeho/docker/adguard/confdir:/opt/adguardhome/ --restart unless-stopped adguard/adguardhome@ainar-g commented on GitHub (Jun 1, 2021):
Hi. Is this really the full command that you use to run the container? Because it doesn't have any volumes attached, so any restart probably removes all previous configuration. See the quick start example on our Docker Hub page.
@spiralshapeturtle commented on GitHub (Jun 1, 2021):
https://github.com/AdguardTeam/AdGuardHome/issues/2657
could be related?
No thats not the full docker command, I removed the volumes due to privacy reasons:
-v /home/foobar/docker/adguard/workdir:/opt/adguardhome/work -v /home/foobar/docker/adguard/confdir:/opt/adguardhome/@spiralshapeturtle commented on GitHub (Jun 1, 2021):
I might thing that when the internet upload is fully congested, the problem occurs faster. Fully utilized like 100% I mean.
@ainar-g commented on GitHub (Jun 1, 2021):
I see. It will be hard to understand what is going on without the logs. Can you configure AGH to make verbose logs and also configure it to write logs to a permanent location using
log_fileand either post the log here as an attachment or send it to devteam@adguard.com with the subject line “AdGuard Home Issue 3217”? Thanks.@spiralshapeturtle commented on GitHub (Jun 1, 2021):
sure will do
@spiralshapeturtle commented on GitHub (Jun 1, 2021):
This is the issue, it's a dual ISP network pihole and windows DNS does not report this issue. It seems almost impossible there is no internet. Let me show you.
Switching to the docker container itself, Adguard to be specific.
Seems like a software issue regarding https DNS upstreams?
upstream config
@ameshkov commented on GitHub (Jun 1, 2021):
Client.Timeout exceeded while awaiting headersmeans that the TCP connection timed out.Does the issue go away with time or is it necessary to restart the container?
When this happens, could you please record a tcpdump?
tcpdump dst 9.9.9.11 -w out.pcapAlso, it'd be interesting to see if you have any issues when you use a plain DNS upstream.
@spiralshapeturtle commented on GitHub (Jun 2, 2021):
Could you guide me where you hide tcpdump in the docker? Any way to run bash from the docker host, i.e. like you have SSH-ed into the docker?
@ameshkov commented on GitHub (Jun 2, 2021):
@rodeho you don't need to run it inside the container, try running it on your host machine should be just okay.
@hplee0120 commented on GitHub (Jun 2, 2021):
I installed it on the OpenWrt router and usually ran normally. But every time Wan is reconnected, ADH does not resolve DNS, the device can't open some web pages, and then check the logs to confirm that no device requests DNS have been received in the ADH background. This phenomenon returns to normal work after turning off the ADH and reopening the operation.
@hplee0120 commented on GitHub (Jun 3, 2021):
What might be about the logs? Looks like it's a connectivity issue for DoH? But you actually need to close the program and restart it to get back to normal.
https://github.com/AdguardTeam/AdGuardHome/issues/3138
2021/06/03 05:37:06.267815 [info] Couldn't get version check json from https://static.adguard.com/adguardhome/release/version.json: *fmt.wrapError updater: HTTP GET https://static.adguard.com/adguardhome/release/version.json: Get "https://static.adguard.com/adguardhome/release/version.json": all upstreams failed to exchange request, cause: couldn't do a GET request to 'https://doh.pub:443/dns-query', cause: Get "https://doh.pub:443/dns-query?dns=M0EBAAABAAAAAAAABnN0YXRpYwdhZGd1YXJkA2NvbQAAAQAB": net/http: request canceled (Client.Timeout exceeded while awaiting headers) (hidden: couldn't do a GET request to 'https://dns.alidns.com:443/dns-query', cause: Get "https://dns.alidns.com:443/dns-query?dns=M0EBAAABAAAAAAAABnN0YXRpYwdhZGd1YXJkA2NvbQAAAQAB": net/http: request canceled (Client.Timeout exceeded while awaiting headers))
@spiralshapeturtle commented on GitHub (Jun 3, 2021):
@ameshkov there are 3 tickets complaining about ADH stop processing requests when it detects an internet down situation. All my other DNS servers is run keeps running normal, no issue. Only ADH with DOH is having issue, whit a client I can browse normal during the "internet down alarms" from ADH.
It really has something to do with the TCP implementation of ADH.
@ameshkov commented on GitHub (Jun 3, 2021):
Okay, I have one idea. Our DOH client has a limit of 1 active connection to the remote host. Maybe, in your case when the network change occurs, the connection is not closed and hangs in some semi-open state. This would block further requests.
Let's try lifting that connections limit and see if it helps.
@EugeneOne1 commented on GitHub (Jun 3, 2021):
@rodeho, the possible fix is implemented in the latest
edgebuild (since084564e). Could you please check, if it solves the problem for you?@spiralshapeturtle commented on GitHub (Jun 3, 2021):
Cool let me check. Could you guide me how to run this edge build into docker?
@EugeneOne1 commented on GitHub (Jun 3, 2021):
Running development builds for Docker is described here.
@spiralshapeturtle commented on GitHub (Jun 3, 2021):
And up and running:v0.107.0-a.62+084564e6
@hplee0120 commented on GitHub (Jun 3, 2021):
After testing, it seems that the problem only exists in DoH. The above problem does not occur with DoT or ordinary port dns.
How to use the repaired version, waiting for 107.b2 release?
@EugeneOne1 commented on GitHub (Jun 4, 2021):
@hplee0120, we've got no information about how you installed your AdGuard Home. Could you please provide this info?
@hplee0120 commented on GitHub (Jun 4, 2021):
I added ADH luci when compiling OpenWrt. Use binary files at https://github.com/AdguardTeam/AdGuardHome/releases.
@spiralshapeturtle commented on GitHub (Jun 5, 2021):
I don't think its fully solved yet. Its very fragile, when the TCP connection drops is running into issues. Could the auto recovery of the TCP session be optimized?@spiralshapeturtle commented on GitHub (Jun 6, 2021):
Hi there,
Did some further testing, when the internet line is OK everything seems to be working well. When I change a FW rule, assume it will hit the current TCP table of the router (OPNsense) the issue kicks in.
The session or the re-connect implementation seems not be able to reconnect in a proper way, and the outage remains for minutes <10. After a minute or what, it resolves by itself.
@spiralshapeturtle commented on GitHub (Jun 7, 2021):
Did some testing with 106.3 the stable one, with normal DNS resolving to unbound within the same network. It also drops when there was a minimal glitch in the internet connectivity.
The interesting part is that I monitor 4 DNS servers:
Adguard edge
Adguard stable
Windows
Pihole
And only the AdGuard servers does have issues, all the other DNS servers are not metioned once in the logs...in the past week. Always working fine, it has to be something in the AGH IP stack...
@ainar-g commented on GitHub (Jun 7, 2021):
@rodeho, thank you for the information! We're still investigating the issues. Just as a side note, what is the value of
dns.max_goroutinesin your configuration file? Does the situation improve if you set it to0?The error you see on the stable version are caused by the fact that
edgemigrates its configuration version, and the stable releases don't know about the new versions. In your case, you should be able to simply replaceschema_version: 11withschema_version: 10in your configuration file when you switch to the release version.@spiralshapeturtle commented on GitHub (Jun 7, 2021):
Hi, the
dns.max_goroutinesis not configured on the stable one, lets configure it on the edge, running a load of servers in the homelab. So there should be a higher load of DNS than regulair.@spiralshapeturtle commented on GitHub (Jun 7, 2021):
Another log from the stable one, before stop serving clients.
07-06 14:15+/- crash
@spiralshapeturtle commented on GitHub (Jun 7, 2021):
@ainar-g commented on GitHub (Jun 7, 2021):
What exactly do you mean when you say that “
dns.max_goroutinesis not configured”? Did you set it to0?Also, the
context cancellederror still means that AGH can't reach the remote DNS server. What happens when you run:From the machine with AGH?
@spiralshapeturtle commented on GitHub (Jun 7, 2021):
I'm testing two machine's
106.3 call it stable
107.edge call it edge
The
dns.max_goroutineschange to 0 on the edge server does improve the situation, first impression the issue is less noticeable. What does this command do? What I mention tot say: thedns.max_goroutinesis not available on the stable server?Curl is not available from within the docker I used wget to perform a baseline on the stable server. Its from a normal situation, to perform baseline testing and compare the outcome later from the faulty situation.
@spiralshapeturtle commented on GitHub (Jun 8, 2021):
The edge server v0.107.0-a.65+4700dea9 just went down, all other DNS servers, pihole, windows, unbound keeps functional during this issue.
``` ```@ameshkov commented on GitHub (Jun 8, 2021):
Are you using encrypted upstreams on these servers or are they using plain DNS upstreams?
Did you save that tcpdump? I'd like to take a look at it. Regarding TCP session recovery, no, it cannot be optimized or changed in any way.
To be honest, I doubt we're dealing with an AGH bug here.
There're three options:
In order to eliminate all these issues one by one:
@spiralshapeturtle commented on GitHub (Jun 8, 2021):
Hi @ameshkov
I greatly appreciate your cooperation. It's a complex setup. I have 5 DNS servers running at home, in my home lab. Of these 5 servers, 2 are adguard, 1 pihole, 1 router and 1x windows DNS. I monitor this entire environment with PRTG, I get alerts if one of these DNS servers no longer responds to requests.
Of these 5 DNS servers, only 2 give alarms; the two AGH servers. That makes it extra remarkable to me? AGH and Phiole both run IntelNUC -> ESXi hypervisor -> Ubuntu VM -> Docker container Pihole/AdGuard.
Everything is identical in terms of underlying network, all dockers are connected with macvlan in Docker. Using an own real IP of the host's network.
#1 that could be, I can force the replication of the issue with sending bulk traffic in my upload ISP. It still confuses me why 3 out of 5 (internal) dns servers remains functional, and only AGH stops serving clients?
#2 Based on the netwerk setup above I won't exclude the IP stack of Golang. Had the same issue with Adguard 106.3 with UDP resolving to the internal router unbound UDP_53. Also ended up in AGH stop processing DNS request.
#3point taken please note that also the OPENdns was also in the list of my DoH config. That poin'ts me to #2 IPstack issues of Golang.
Will try to increase the monitoring, it pols every 30seconds. And after 3 fails I receive a push alert, try to reduce that to have more time to start collecting logs.
@ameshkov commented on GitHub (Jun 8, 2021):
If you were using the same upstreams with the same protocol on all 5 that would be confusing indeed, but that's not the case.
Note that if you have "browsing security" enabled, AGH internally uses AdGuard DNS DOH upstream. If you want to do a clean test with UDP, it'd be better to disable both parental control and browsing security.
@spiralshapeturtle commented on GitHub (Jun 8, 2021):
OK point taken all DNS servers are now pointing to unbound (the router).
Yeah saw that in the logs, lets first monitor all servers with the same uplink and only uplink as unbound. If the issue still persists the next step is to disable the parental/browsingsecurity features, so no DOH is being used.
@hplee0120 commented on GitHub (Jun 8, 2021):
My test results show that this condition exists on the DoH upstream server and there is no problem with DoT or normal port DNS.
@spiralshapeturtle commented on GitHub (Jun 12, 2021):
Okey time for an update. You might know that docker supports macvlan, a solution to provide a docker container a real IP. Super easy for DNS servers;-) to allow this nested-mac design you need to enable promiscuous mode on ESXi. Then all packets seems to be replicated within the same port group (virtual switch port) when you sniff on the host. In that same switch port also the NFS and iSCSI bulk sessions are hosted, so a lot of traffic is flowing by the Golang nic.
My first thoughts are still the same. The IP stack of Golang is not as stable as the one from other DNS servers like PiHole. I moved the docker vlan to a new vlan, which makes the network cleaner. Lets monitor again.
@ameshkov commented on GitHub (Jun 13, 2021):
@hplee0120 have you tried the edge channel build?
@rodeho
Neither golang nor dnsmasq (pi-hole) use their own TCP/IP stack. They all rely on OS sockets. The difference that I see here is that in one case you're using plain DNS over UDP -- no connection state, simple to handle any networking issues just by re-sending the same query again. At the same time, you're comparing it to DNS over HTTPS which is quite different and more vulnerable to any network-related issues.
I still think that we should first make sure that there're no issues when you use plain DNS over UDP upstream in AGH. When it's confirmed, the next thing to check would be a DOT or DNS-over-TCP upstream just to see if there's any difference.
Note that in both cases you need to disable "browsing security" and "parental control" in AGH since they rely on using DOH internally.
@spiralshapeturtle commented on GitHub (Jun 14, 2021):
@ameshkov
What I did was the following, I moved the docker guest (ubuntu) to the docker lan, which keeps the network more silent. Normal network behavior should listen only to their own MAC or a broadcast message, right? The monitoring tool alerts me that the server went (both) down again at the same time. They were still running with the original configuration, because i wanted to capture a full outage with logs and wireshark. I have a set, how could I share these with you in a privacy safe way?
adguard1 recovered (stable build)
adguard 2 stil in crashed situation (edge build)
I fired some manual cli commands let me show you some logs:
The recovered one
You are completely right about the next test with all the DOH options disabled. The one thing I'm questioning myself: what is the benefits of testing further? This seems to be an adguard code issue, the OS is unable to call the DOH servers, but on CLI I'm able to?
@spiralshapeturtle commented on GitHub (Jun 14, 2021):
Next test implemented, all options off. Remain to use DoH upstream providers:
@ameshkov commented on GitHub (Jun 15, 2021):
You could send them directly to me to
am@adguard.com.@rodeho I see you're getting
wget: error getting response: Connection reset by peerondockerstable, so there was still a networking issue.The benefit is that we'll be able to narrow this down. I don't see what's causing this issue, all of my guesses from above are still possible.
@spiralshapeturtle commented on GitHub (Jun 16, 2021):
No no netwerk issue, connection reset by peer is the other side. There is a connection but SSL seems to be failing, could that lead to the root cause? Will show you some other dockers in the same network segment and their output.
@ameshkov commented on GitHub (Jun 21, 2021):
Renaming the issue and merging #3270 here.
I tend to think the problem is specific to DOH and may be caused by a bug in golang http2 implementation. The problem is that we still cannot reproduce this.
We probably can add some workaround (re-create the HTTP client on every timeout error for instance), but I'd like to figure out what's really causing this first.
@spiralshapeturtle commented on GitHub (Jun 21, 2021):
@ameshkov thanks, is there anyway we can validate if its specific to the DOH servers of adguard cloud, or specific to DOH upstream servers?
When trying to simulate in your lab, try to add some packet loss. Seems to be more frequent when the upload is heavy loaded.
@ameshkov commented on GitHub (Jun 21, 2021):
@rodeho IMO, it does not seem to be specific to particular DOH servers, rather to golang's http client in general (which we use internally). If we're not able to reproduce it internally, we'll push another build to the edge build that should in theory solve this by re-creating the http client on timeout errors.
@fizsef commented on GitHub (Jun 21, 2021):
I have observed this problem twice, but when I specifically wanted to reproduce it in order to post the log here, this time there were no problems and everything worked fine. Although the settings were the same. I use a DoH server https://family.cloudflare-dns.com/dns-query
@spiralshapeturtle commented on GitHub (Jun 21, 2021):
Just to inform you, I moved to a local Unbound forwarder. Therefore 95% of my network using port 53 with UDP. Only some clients 5% where specifically configure with extra checking for parental control, and they failed an hour ago. Analyzed the logs only the logs DOH checks failed, which also means that the rest of the IP stack remains functional for at least UDP packets.
request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)@ainar-g commented on GitHub (Jun 22, 2021):
@rodeho, @hplee0120, @fizsef, we've pushed some fixes that may improve the situation to the
edgechannel in revision90a85362. Could any of you check if the situation is better now? Thanks!@hplee0120 commented on GitHub (Jun 25, 2021):
From my personal use, after downloading the latest version of edge, the problem seems to have been solved.!
@spiralshapeturtle commented on GitHub (Jun 26, 2021):
I upgraded to this edge build within my test setup, my primary DNS moved to PiHole during this debugging period AGH was causing to much outages.
@fizsef commented on GitHub (Jun 27, 2021):
and how to install, where to download, this corrected version?
@ainar-g commented on GitHub (Jun 28, 2021):
@fizsef, there are instructions here in our readme.
@spiralshapeturtle commented on GitHub (Jun 30, 2021):
Last night my stable build went down, and the edge build remains up. Seems to be fixed! Thanks