-
-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rust-doh closes connection with no return code on malformed request and upstream timeout #20
Comments
The proxy never returns a 502 response code. But it enforces timeouts, so that clients don't wait forever. There is also a maximum number of simultaneous connections (512, by default). When one of these limits is reached, the proxy will close the connection, probably leading to a 502 response code generated by Nginx. Try increasing these limits to see if it helps. But then you also need to make sure that you are not going to hit system limits, such as the file descriptor limits. I also run the DoH servers behind Nginx, but I don't log. |
I've a slightly better understanding now but further digging is needed and help is welcome. :) nginx configLet's start with some excerpts from the nginx config to better understand the logs:
two types of HTTP 502 log eventsWe observed two type of log entries related to HTTP 502: a)
same for the instance on port 3001. They occur rarely but consistently. b)
They occurred even less frequently than (a) but when they happen they came at a high number at once. https://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_addr explains the difference of the
conclusion about type (b) log eventsThat means during type (b) events, nginx considered all two backends unavailable. So by increasing Now we need to find out more about type (a) log events. According to nginx, doh-proxy actually returned HTTP 502. HTTP 502 consistent across instancesThe following logs suggest that the error is consistent across the two instances of doh-proxy:
|
I assume I understand the issue better now. This is expected:
This is unexpected (no response at all):
What do you think about returning For the upstream timeout case that currently also results in a closed connection: What do you think about returning This would be nice so we could tell these cases apart in the error stats. |
de99e6a returns a Returning something on a timeout seems to be too complicated for my knowledge of Rust and Tokio. The timeout is on the socket itself, and once it fires, the HTTP layer is not accessible any more. Something I may revisit once async I/O in Rust stabilizes and becomes a bit more friendly. |
Hi,
our setup:
nginx -> rust-doh -> unbound
In a small fraction (~1 out of 1000 queries) nginx returns the HTTP 502 response code which suggests that rust-doh had an issue with the request and returned something unexpected.
We never see HTTP 502 response codes when using
https://github.com/facebookexperimental/doh-proxy
Since apparently there are no rust-doh logs (#11) it is hard to further debug this.
version: 0.1.15
The text was updated successfully, but these errors were encountered: