Hi all,
I’m troubleshooting an issue that I’m not sure is on the client side or the server side.
I have an API endpoint that appears to work correctly with HTTP/1.1 and HTTP/2 requests (e.g. via Postman HTTP/1.1, via Insomnia with HTTP/2 and HTTP/1.1, via curl with HTTP/2 and HTTP/1.1)
These same requests – URL, body, content-type, headers, auth, etc. – fail using Apidog with HTTP/2. They work again as soon as I toggle HTTP/2 support off in Apidog.
In looking at the debug logs in Istio when requests come into my Kubernetes cluster, all these failed requests seem to have one thing in common: the “:authority” pseudo-header has a repeated value. If our hostname is “xyz.com”, the authority header is “xyz.com,xyz.com”. This appears to only happen on these failed requests.
What I’m trying to understand is – is Apidog calculating this header and sending it? Is there any way I can see exactly what’s being sent in this way from Apidog so that I can rule it out? I’m trying to understand where the issue lies or whether something is misconfigured in k8s/istio, but it works in so many situations that I’m wondering if it might be a client-side issue.
Are you using the desktop client software or the web version of the app? If it’s the web version, are you using Cloud Agent or Browser extension as Agent?
Karan great question, thanks! Sorry; should have included that. I’m using the Windows desktop client.
Could you please try using the Cloud Agent or Browser extension from the web version to make the request? If possible, this will help determine if the issue is specific to the desktop app which might causing the client-side issue…
Karan very interesting – running from the Apidog web site with HTTP/2 enabled is working as expected.
I noticed that SAP’s NetWeaver app server client is also having a similar issue to Apidog when making requests to this API.
Could someone share the names of the HTTP library/libraries you’re using with Datadog? If I can determine that SAP NetWeaver is using the same one, I might be able to zero in on an upstream fix.
Karan To be clear - I ran this by going to the Apidog site. I didn’t see the options to select an agent, but I’ll look for those and try to complete the steps as you mentioned just to be sure.
Our team has tested multiple HTTP/2 services using the desktop client, including some API from Apple. And we haven’t encountered any issues or identified any bugs. The browser relies on browser-based request services, while the desktop client is based on a custom Node.js library that supports HTTP/2 and has took extensive testing. But it is not an open-source implementation.
Can you provide a ping/pong type of API endpoint? We will only make a small number of requests for the purpose of troubleshooting, with no other actions involved.
This will greatly help us in quickly identifying and testing compatibility. Our team places great importance on addressing user feedback.
Karan thanks – that’s good information to have. A ping-pong endpoint isn’t a bad idea; I need to write & deploy it and I’ll get back to you.
My theory is that it may be a gray area in the HTTP/2 spec, where perhaps Istio/envoy are being stricter than they should or haven’t anticipated this. But those tools are also massively popular and I assume something like this would have surfaced long before now.
Furthermore, is your API service compatible with HTTP/1.1? It might be working in the browser because protocol downgrading is automatically negotiated.
There is further progress, and I found on Stackoverflow that “:authority” is actually strictly not allowed to appear multiple times or with multiple hosts. We used network capture software and found that the packets sent by our desktop client are normal and do not have multiple hosts. You can see the screenshot below.
Thank you for confirming! I figured out what’s happening, and this fits with it perfectly.
SAP BTP Kyma (where we host our app) uses Istio, which uses the Envoy proxy. This appears to be a bug in Envoy.
On requests that have both the :authority and the host defined, it should keep the authority as-is. Instead, it concatenates the authority to include the host value.
My bug report is here and it looks like Envoy does have some work in progress to fix it: Some HTTP/2 requests fail with duplicate value in envoy authority request pseudo-header · Issue #31118 · envoyproxy/envoy · GitHub
Given the popularity of Envoy and Istio, I’m mostly surprised this hasn’t been a more visible issue.
In the inteirm, I’m not sure really what to do. I suppose the only thing to do would be to ask clients that send both host and :authority to only send one, but I understand why that probably isn’t practical.
Karan something that would be very helpful and that isn’t immediately obvious to me – can you confirm: for HTTP/2 requests on the Windows app, does Apidog send both the :authority and the Host field? And in the authority, is that a single value, or is it a comma-separated list?
I don’t have access to the code or the ability to see this on my own as far as I can tell.
The screenshot above already shows that both exist, and not in the form of a list
Karan sorry, I completely missed that above.
I was able to confirm that this was indeed the bug. And by “bug”, I mean: a client doing something that is technically allowed (populating both those headers with the same value) but is not required and is perhaps unexpected by some servers such as Envoy. The true bug is on Envoy’s side for not entirely adhering to the HTTP/2 spec and making an assumption.
As of now, Apidog HTTP/2 requests will fail with a 404 error against any web service that uses the Envoy proxy. SAP NetWeaver client had the same issue. We were able to fix the issue in our SAP environment by “tricking” the NetWeaver client. We leave the hostname out of our request URL, which makes the host field blank. But the :authority pseudo-header is populated. This means that only one of the headers is populated, which avoids the envoy bug and allows the request to succeed.
Apidog may want to consider adopting logic that only populates only one of the “:authority” or “Host” fields when those values are the same, for the sake of making it easier on servers while still adhering to the spec just fine. But, I’ve taken enough of your time and I’ll leave it here. Thank you for your responses!
Thank you for this compatibility suggestion. I will check with our development engineer to see if we can make it compatible easily, as Envoy proxy is indeed a commonly used service.
