mirror of
https://github.com/AdguardTeam/AdGuardHome.git
synced 2026-03-04 00:01:12 -05:00
AGH stops processing uncached DNS lookups #3357
Labels
No labels
P1: Critical
P2: High
P3: Medium
P4: Low
UI
bug
cannot reproduce
compatibility
dependencies
docker
documentation
duplicate
enhancement
enhancement
external libs
feature request
good first issue
help wanted
infrastructure
invalid
localization
needs investigation
performance
potential-duplicate
question
recurrent
research
snap
waiting for data
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/AdGuardHome#3357
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @thedannywahl on GitHub (Jan 6, 2022).
Issue Details
Expected Behavior
AGH should process all valid DNS requests
Actual Behavior
After the AGH service is started manually (
sudo ./AdGuardHome -s start) or at system start it will eventually stop processing DNS requests for uncached entries.This started occurring after I upgraded from 0.107.0 to 0.107.2 (skipping 0.107.1). Around the same time I started noticing that this would occur I also installed a number of system upgrades including systemd and the kernel (see apt log below).
When it stops processing there is no spike in system load, CPU utilization, or memory. Other services running on this server continue to function normally.
Manually stopping and starting the service or restarting the system resolve the error for a period until it starts happening again.
The error does not seem to be time-based but rather utilization based (
nlookups), though I only have anecdotal evidence of this.Additionally I only have minimal proof that this only affects uncached entries but these are the steps I've done to test it:
chevrolet.com) in a normal browser windowchevrolet.comand it will load.ford.com)I have attached the verbose log from the server here from boot to service restart, however it doesn't appear to capture anything interesting.
Screenshots
Screenshot:
Additional Information
AGH verbose version
Channel: release
Go version: go1.16.12
Build time: 2021-12-29T19:55:46Z+0000
GOOS: linux
GOARCH: amd64
Race: false
Dependencies:
github.com/AdguardTeam/dnsproxy@v0.40.4-0.20211229192924-90569a17b881 (sum: h1:nPMyvZUn1uSa5xVPSjJpZCDkn3+kLnNR7B3XTKk9tKs=)
github.com/AdguardTeam/golibs@v0.10.3 (sum: h1:FBgk17zf35ESVWQKIqEUiqqB2bDaCBC8X5vMU760yB4=)
github.com/AdguardTeam/urlfilter@v0.15.1 (sum: h1:dP6S7J6eFAk8MN4IDpUq2fZoBo8K8fmc6pXpxNIv84M=)
github.com/NYTimes/gziphandler@v1.1.1 (sum: h1:ZUDjpQae29j0ryrS0u/B8HZfJBtBQHjqw2rQ2cqUQ3I=)
github.com/aead/chacha20@v0.0.0-20180709150244-8b13a72661da (sum: h1:KjTM2ks9d14ZYCvmHS9iAKVt9AyzRSqNU1qabPih5BY=)
github.com/aead/poly1305@v0.0.0-20180717145839-3fee0db0b635 (sum: h1:52m0LGchQBBVqJRyYYufQuIbVqRawmubW3OFGqK1ekw=)
github.com/ameshkov/dnscrypt/v2@v2.2.3 (sum: h1:X9UP5AHtwp46Ji+sGFfF/1Is6OPI/SjxLqhKpx0P5UI=)
github.com/ameshkov/dnsstamps@v1.0.3 (sum: h1:Srzik+J9mivH1alRACTbys2xOxs0lRH9qnTA7Y1OYVo=)
github.com/beefsack/go-rate@v0.0.0-20200827232406-6cde80facd47 (sum: h1:M57m0xQqZIhx7CEJgeLSvRFKEK1RjzRuIXiA3HfYU7g=)
github.com/cheekybits/genny@v1.0.0 (sum: h1:uGGa4nei+j20rOSeDeP5Of12XVm7TGUd4dJA9RDitfE=)
github.com/digineo/go-ipset/v2@v2.2.1 (sum: h1:k6skY+0fMqeUjjeWO/m5OuWPSZUAn7AucHMnQ1MX77g=)
github.com/fsnotify/fsnotify@v1.5.1 (sum: h1:mZcQUHVQUQWoPXXtuf9yuEXKudkV2sx1E06UadKWpgI=)
github.com/go-ping/ping@v0.0.0-20210506233800-ff8be3320020 (sum: h1:mdi6AbCEoKCA1xKCmp7UtRB5fvGFlP92PvlhxgdvXEw=)
github.com/google/go-cmp@v0.5.5 (sum: h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=)
github.com/google/gopacket@v1.1.19 (sum: h1:ves8RnFZPGiFnTS0uPQStjwru6uO6h+nlr9j6fL7kF8=)
github.com/google/renameio@v1.0.1 (sum: h1:Lh/jXZmvZxb0BBeSY5VKEfidcbcbenKjZFzM/q0fSeU=)
github.com/insomniacslk/dhcp@v0.0.0-20210310193751-cfd4d47082c2 (sum: h1:NpTIlXznCStsY88jU+Gh1Dy5dt/jYV4z4uU8h2TUOt4=)
github.com/josharian/native@v0.0.0-20200817173448-b6b71def0850 (sum: h1:uhL5Gw7BINiiPAo24A2sxkcDI0Jt/sqp1v5xQCniEFA=)
github.com/kardianos/service@v1.2.0 (sum: h1:bGuZ/epo3vrt8IPC7mnKQolqFeYJb7Cs8Rk4PSOBB/g=)
github.com/lucas-clemente/quic-go@v0.24.0 (sum: h1:ToR7SIIEdrgOhgVTHvPgdVRJfgVy+N0wQAagH7L4d5g=)
github.com/marten-seemann/qtls-go1-16@v0.1.4 (sum: h1:xbHbOGGhrenVtII6Co8akhLEdrawwB2iHl5yhJRpnco=)
github.com/mdlayher/ethernet@v0.0.0-20190606142754-0394541c37b7 (sum: h1:lez6TS6aAau+8wXUP3G9I3TGlmPFEq2CTxBaRqY6AGE=)
github.com/mdlayher/netlink@v1.4.0 (sum: h1:n3ARR+Fm0dDv37dj5wSWZXDKcy+U0zwcXS3zKMnSiT0=)
github.com/mdlayher/raw@v0.0.0-20210412142147-51b895745faf (sum: h1:InctQoB89TIkmgIFQeIL4KXNvWc1iebQXdZggqPSwL8=)
github.com/miekg/dns@v1.1.45 (sum: h1:g5fRIhm9nx7g8osrAvgb16QJfmyMsyOCb+J7LSv+Qzk=)
github.com/patrickmn/go-cache@v2.1.0+incompatible (sum: h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=)
github.com/pkg/errors@v0.9.1 (sum: h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=)
github.com/satori/go.uuid@v1.2.0 (sum: h1:0uYX9dsZ2yD7q2RtLRtPSdGDWzjeM3TbMJP9utgA0ww=)
github.com/ti-mo/netfilter@v0.4.0 (sum: h1:rTN1nBYULDmMfDeBHZpKuNKX/bWEXQUhe02a/10orzg=)
github.com/u-root/u-root@v7.0.0+incompatible (sum: h1:u+KSS04pSxJGI5E7WE4Bs9+Zd75QjFv+REkjy/aoAc8=)
go.etcd.io/bbolt@v1.3.6 (sum: h1:/ecaJf0sk1l4l6V4awd65v2C3ILy7MSj+s/x1ADCIMU=)
golang.org/x/crypto@v0.0.0-20211215153901-e495a2d5b3d3 (sum: h1:0es+/5331RGQPcXlMfP+WrnIIS6dNnNRe0WB02W0F4M=)
golang.org/x/net@v0.0.0-20211216030914-fe4d6282115f (sum: h1:hEYJvxw1lSnWIl8X9ofsYMklzaDs90JI2az5YMd4fPM=)
golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c (sum: h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=)
golang.org/x/sys@v0.0.0-20211216021012-1d35b9e2eb4e (sum: h1:fLOSk5Q00efkSvAm+4xcoXD+RRmLmmulPn5I3Y9F2EM=)
golang.org/x/text@v0.3.7 (sum: h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=)
gopkg.in/natefinch/lumberjack.v2@v2.0.0 (sum: h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=)
gopkg.in/yaml.v2@v2.4.0 (sum: h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=)
howett.net/plist@v0.0.0-20201203080718-1454fab16a06 (sum: h1:QDxUo/w2COstK1wIBYpzQlHX/NqaQTcf9jyz347nI58=)
Verbose CPU version
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Genuine Intel(R) CPU U2300 @ 1.20GHz
verbose OS info
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
verbose system info
Static hostname: plex
Icon name: computer-desktop
Chassis: desktop
Machine ID: 97ff6784cf834e2093ec0329f069d558
Boot ID: 93ef0583297049bcb538bcf0cbbfa719
Operating System: Ubuntu 20.04.3 LTS
Kernel: Linux 5.4.0-92-generic
Architecture: x86-64
OS Update (apt) log
Start-Date: 2021-12-18 06:50:10
Commandline: /usr/bin/unattended-upgrade
Upgrade: linux-firmware:amd64 (1.187.20, 1.187.23)
End-Date: 2021-12-18 06:52:45
Start-Date: 2022-01-05 03:46:44
Commandline: apt-get upgrade
Requested-By: plex (1000)
Upgrade: libsystemd0:amd64 (245.4-4ubuntu3.13, 245.4-4ubuntu3.14), udev:amd64 (245.4-4ubuntu3.13, 245.4-4ubuntu3.14), libudev1:amd64 (245.4-4ubuntu3.13, 245.4-4ubuntu3.14), systemd-timesyncd:amd64 (245.4-4ubuntu3.13, 245.4-4ubuntu3.14), python3-commandnotfound:amd64 (20.04.4, 20.04.5), systemd-sysv:amd64 (245.4-4ubuntu3.13, 245.4-4ubuntu3.14), libpam-systemd:amd64 (245.4-4ubuntu3.13, 245.4-4ubuntu3.14), systemd:amd64 (245.4-4ubuntu3.13, 245.4-4ubuntu3.14), libnss-systemd:amd64 (245.4-4ubuntu3.13, 245.4-4ubuntu3.14), command-not-found:amd64 (20.04.4, 20.04.5), linux-firmware:amd64 (1.187.23, 1.187.24)
End-Date: 2022-01-05 03:50:47
Start-Date: 2022-01-05 04:00:35
Commandline: apt-get --with-new-pkgs upgrade
Requested-By: plex (1000)
Install: linux-image-5.4.0-92-generic:amd64 (5.4.0-92.103, automatic), linux-modules-extra-5.4.0-92-generic:amd64 (5.4.0-92.103, automatic), linux-headers-5.4.0-92:amd64 (5.4.0-92.103, automatic), linux-headers-5.4.0-92-generic:amd64 (5.4.0-92.103, automatic), linux-modules-5.4.0-92-generic:amd64 (5.4.0-92.103, automatic)
Upgrade: linux-headers-generic:amd64 (5.4.0.91.95, 5.4.0.92.96), linux-image-generic:amd64 (5.4.0.91.95, 5.4.0.92.96), linux-generic:amd64 (5.4.0.91.95, 5.4.0.92.96)
End-Date: 2022-01-05 04:02:27
Start-Date: 2022-01-05 06:09:57
Commandline: /usr/bin/unattended-upgrade
Remove: linux-image-5.4.0-90-generic:amd64 (5.4.0-90.101), linux-modules-extra-5.4.0-90-generic:amd64 (5.4.0-90.101), linux-modules-5.4.0-90-generic:amd64 (5.4.0-90.101)
End-Date: 2022-01-05 06:10:05
Start-Date: 2022-01-05 06:10:12
Commandline: /usr/bin/unattended-upgrade
Remove: linux-headers-5.4.0-90-generic:amd64 (5.4.0-90.101), linux-headers-5.4.0-90:amd64 (5.4.0-90.101)
End-Date: 2022-01-05 06:10:17
DNS settings
tls://1.1.1.1 https://dns.cloudflare.com/dns-query mode: load balanced bootstrap DNS: 1.1.1.1 EDNS: Enabled DNSSEC: Enabled Blocking mode: Default Cache size: 4194304 Optimistic caching: Enabledaghlog.txt
@SiNONiMiTY commented on GitHub (Jan 7, 2022):
INITIAL POST
Bumping this up, AGH would randomly "hang" after serving DNS requests for a short time. Restarting the service fixes it, until it happens again. Using v0.107.2.
UPDATE 2022-01-10
When the following lines shows up, DNS resolution stops and the service needs to be restarted.
UPDATE 2022-01-11
I reverted to v0.107.0, this issue is not happening.
@ainar-g commented on GitHub (Jan 20, 2022):
Hello, and apologies for the late response! We cannot reproduce the issue, unfortunately. Can you enable verbose logging and look, what kind of logs are printed when this happens? Or send the logs to devteam@adguard.com? Thanks!
@Birbber commented on GitHub (Sep 1, 2022):
@thedannywahl Hi! Is this issue still relevant?
@chaelli commented on GitHub (Oct 24, 2022):
Yes, I have the same issue (see #1777 )
@0fbcb238c0 commented on GitHub (Nov 9, 2022):
I also have this issue, as well as not being able to reproduce. I have no issues regarding freezes or stops of request processing though.
Excerpt of log is attached.
agh_git.log
@Churator commented on GitHub (Nov 29, 2022):
I have same thing, I guess.
packing udp packet: dns: buffer size too small
0.107.19
@nigam-sa commented on GitHub (Jan 19, 2023):
Looks like this isn't fixed yet. I am running adgaurd current version: 4.7.6 on
Home Assistant 2023.1.5
Supervisor 2022.12.1
Operating System 9.4
Checked and net.core.rmem_max is set to 4194304
I have't noticed any freezing etc but below in the logs.
2023/01/19 07:14:09.827890 [error] unpacking udp packet: dns: buffer size too small
2023/01/19 07:14:12.762781 [error] unpacking udp packet: dns: buffer size too small
2023/01/19 08:06:10.624769 [error] unpacking udp packet: dns: buffer size too small
2023/01/19 08:06:13.639789 [error] unpacking udp packet: dns: buffer size too small
@Churator commented on GitHub (Jan 19, 2023):
Well, no new release, so it makes sense :)
@JulAlx commented on GitHub (Jan 23, 2023):
Same issue here. Have increased the UDP buffer with no effect:
cat /proc/sys/net/core/rmem_max 2500000@nigam-sa commented on GitHub (Jan 23, 2023):
i have upgraded to recently released version 4.8.0 and my net.core.rmem_max is set to 4194304. Still I am having the same errors in logs everyday.
@frenck - Is it possible for you to look at this please?
@MilesTEG1 commented on GitHub (Feb 7, 2023):
I've got the same errors in log.
Is there any way to correct it ?
Here the command's result some launched :
@imisic commented on GitHub (Feb 18, 2023):
Got the same errors in log
v0.107.22
AdGuard Home addon for Home Assistant v4.8.0
[error] unpacking udp packet: dns: buffer size too small
@Churator commented on GitHub (Feb 21, 2023):
What's going on, why no one check/fix this ?
Still stuck on .18
@TheWiresharkGuy commented on GitHub (Jul 24, 2023):
Same error with v0.107.34
[error] dnsproxy: unpacking udp packet: dns: buffer size too small@OlliZabal commented on GitHub (Aug 16, 2023):
Same error with v0.107.36 running as Docker installation on a Synology NAS
[error] dnsproxy: unpacking udp packet: dns: buffer size too small@ainar-g commented on GitHub (Aug 16, 2023):
We still cannot reproduce the
dns: buffer size too smallerror. It comes from thegithub.com/miekg/dnslibrary, so it is either a malformed packet or perhaps there is a bug in the library. Either way, it shouldn't interfere with the normal functioning of AGH. If you are sure that it does, please send the verbose logs of the AGH run that stops processing after these errors.@thecode commented on GitHub (Aug 16, 2023):
I can reproduce it easily but indeed did not notice functional impact, however since AGH is running on my router (OpenWRT), these logs are spamming the syslog and make it unusable.
I am wondering if I can provide additional data to help pinpoint the problem, I can run tcpdump directly on the router and log all packets if it helps, another alternative is to disable this log message.
Is there any issue for this upstream (
github.com/miekg/dns)?Thanks
@ainar-g commented on GitHub (Aug 16, 2023):
We can log this exact error with
[debug]level, I think. I feel, though, that if it really is a bad query, users should probably know.If you could get the dump of the offending request, that'd be great.
Also, are these coming from the same client? Perhaps it's a known issue with that device?
There are a few roughly matching, but without the exact data it's hard to tell.
@szhu25 commented on GitHub (Sep 12, 2023):
@ainar-g Is there any log I can provide to help investigate this issue? I'm currently on v0.107.38 and the issue is severe that I got SERVFAIL (on my network clients) every 20 - 30 minutes...
My AdGuardHome.log are filled with
My usage is typically 280000+ queries in 24 hours, on Raspberry Pi 400. Query method is a mix of plain DNS & DoH (through Nginx reverse proxy with HTTP/3). Pure home use.
@ainar-g commented on GitHub (Sep 12, 2023):
@szhu25, all of the steps I've outlined in the previous comment are still valid, in addition to collecting verbose logs, especially of the exact moment the issue occurs.
@szhu25 commented on GitHub (Sep 12, 2023):
Sounds good! I'll go home,collect logging for 48 hours and see where that goes! Thanks
@privacyguy123 commented on GitHub (Oct 17, 2023):
I get this on asuswrt-merlin, interesting that another user with the same problems is on a openwrt router and the people who can't produce it aren't on routers perhaps? Something in the router firmware?
I don't use a QUIC DNS server in my entries.
@szhu25 commented on GitHub (Oct 17, 2023):
I think it might be due to some weird limitations on ARM-based CPUs. I also have external AdGuard Home resolvers on Intel & AMD CPUs and haven't experienced this issue on those servers. This is a speculation and I have no way to prove it.
@privacyguy123 commented on GitHub (Oct 17, 2023):
Has to be something along those lines ...
@OlliZabal commented on GitHub (Oct 20, 2023):
I dont't think that this issue is limited to ARM-based CPUs as in my case I'm running AGH as Docker installation on a Synology DS720plus. This unit has an Intel Celeron J4125 CPU.
FYI: I do use TLS and HTTPS DNS server in my entries.
@ferferga commented on GitHub (Nov 21, 2023):
I have an Intel Xeon E5620 as well and this issue happens consistently when the number of requests increases. Been rocking these insane numbers and still happens (when the number is higher, the time it takes to appear is longer though).
200000000
200000000
As @OlliZabal, I could reproduce this using https and tls upstreams, but also QUIC and h3. I set
use_http3_upstreams: falseto avoid HTTPS being upgraded to HTTPS/3, but I still get this.However, in a Raspberry Pi 4, I never saw that line logged 😉, so definitely not processor-related.
I'm still trying to collect useful logs for this.
@ferferga commented on GitHub (Nov 21, 2023):
I discovered an interesting thing: so far, it looks like the issue is caused by 2 devices, one is S21 FE and the other is S20 FE. Consistently...
@thecode commented on GitHub (Nov 21, 2023):
This is interesting since I also suspected this is caused buy S20, I notice the errors mostly when S20 device is connected to the network, however I am not sure this is the only problem as I did see errors also when the S20 is not connected
@crazy-matt commented on GitHub (Feb 10, 2024):
Same here, noticed this error right after an S10 joining the network:
AdGuardHome 1.7.4 release
Asuswrt-Merlin firmware
@overwatch3560 commented on GitHub (Apr 1, 2024):
@crazy-matt is this still an issue with the most recent version?
@crazy-matt commented on GitHub (Apr 3, 2024):
Hello,
I've installed the new version yesterday on an Asuswrt merlin router.
Got this:
[error] dnsproxy: unpacking udp packet: dns: buffer size too small
Coming twice in sequence of 2 traces with the same timestamp. Didn't have time to match it to a particular query. Will feedback in when have time.
@ghost commented on GitHub (Apr 3, 2024):
Try also enabling verbose logging which may pinpoint the issue a bit better.
@thecode commented on GitHub (Apr 16, 2024):
I would not close it just as people are starting to identify the root cause, it is now limited to a packet sent by Samsung smartphones and I am still trying to capture this
@sla-te commented on GitHub (May 3, 2024):
+1 (and I have already tweaked the hell out of sysctl.conf, cant imaging anything more I could raise)
@ferferga commented on GitHub (May 3, 2024):
@sla-te Do you have a Samsung device like we mentioned before?
@sla-te commented on GitHub (May 3, 2024):
No, its a RPI4 2GB with openwrt and adguardhome
@ferferga commented on GitHub (May 3, 2024):
@sla-te The server it's not relevant, we're asking for the client (in this case Samsung phones)
@robinjoo1 commented on GitHub (Jun 28, 2024):
hey i own a s24+ and recently i have been seeing this
@thecode commented on GitHub (Jun 28, 2024):
I suggest providing new data at https://github.com/AdguardTeam/AdGuardHome/issues/6893 which replaces this issue (since this one is closed)
@yuxuan0107 commented on GitHub (Oct 4, 2024):
我将使用中文和英文两种方式尝试回答这个问题
首先这个问题我也出现很久了大概从我多年前使用ADG开始，今天查阅错误日志，发现大量同样的错误，通过排查发现了问题的出现原因，目前错误已经消失，大家可以参考一下【ADG=AdGardHome】
如果你在网络出口网关的地方将DNS劫持到你的ADG，那么ADG将会出现两个错误，一个是BAD RDNS，另外一个是dns: buffer size too small，这两个问题是同步出现的而且都是出现在由网关劫持dns到ADG的时候，如果将网关或其他局域网内劫持DNS的功能关掉，这个问题将会解决，目前我通过这样的方式已经成功解决了这个错误，大家可以尝试一下
I will try to answer this question in both Chinese and English
First of all, I have had this problem for a long time. I started using ADG many years ago. Today, I checked the error log and found a large number of the same errors. Through investigation, I found the cause of the problem. Now the error has disappeared. You can refer to [ADG=AdGardHome]
If you hijack DNS to your ADG at the point where the network exits the gateway, two errors will appear in ADG. One is BAD RDNS, and the other is dns: buffer size too small. These two problems occur simultaneously and both occur when the gateway hijacks dns to ADG. If you turn off the function of hijacking DNS in the gateway or other local area network, this problem will be solved. So far, I have successfully solved this error in this way. You can try it
@sla-te commented on GitHub (Oct 6, 2024):
I am indeed getting the same error precisely after the DHCP server provides the handshake to my Samsung phone
Not sure how to make the Samsung phone not try and hijack the DNS tbf :). Even more interesting would be, what is happening there and why does it cause ADG to throw?
@yuxuan0107 commented on GitHub (Oct 10, 2024):
As I understand it, I suggest you check the network traffic of this Samsung device at the gateway to see if there are requests from other DNS servers.