Spring Cloud Gateway Connection header
I've recently been working a lot with Spring Cloud Gateway. It's been (mostly) a pleasant experience, but I do encounter some annoying quirks from time to time. This is a story about one of them.
A few days ago I was debugging an issue at work when a backend microservice developer that my API Gateway proxies to asked me to try using keep-alive connections. The request sounded reasonable enough, adding headers to downstream request is easy in Cloud Gateway. I already have few headers that I'm adding.
Hold my beer. I got this.
me, going into this
I configure my Spring Gateway routes programmatically, so adding headers to request looks something like this:
builder.routes() .route(r -> r.path("/foo/bars") .filters(f -> f.setRequestHeader(HttpHeaders.CONNECTION, "keep-alive")) .uri(backendUrl)) .build();
I already had my route set up, so all I needed to add was the
setRequestHeader line. So far so good.
Signs of trouble
Next up, I updated tests to check for the new header. This is where I detected the problem. I use WireMock to simulate backend services in tests. Checking requests that API Gateway sends downstream is straightforward:
verify(getRequestedFor(urlEqualTo("/foo/bars")) .withHeader("Connection", equalTo("keep-alive")) );
And the test failed. Here's what WireMock told me:
No requests exactly matched. Most similar request was: expected:< GET /foo/bars Connection: keep-alive > but was:< GET /foo/bars
Basically, requests were going through, the route was properly sending them to mocked backend service, but
Connection header was missing.
Having a test that fails consistently was useful, because it allowed me to debug the issue. I put my first breakpoint inside
SetRequestHeaderGatewayFilterFactory class. That's where
GatewayFilter that sets headers is implemented. I ran my test and everything looked good.
Connection header was added to mutated request, and mutated exchange was passed on to filter chain.
Next up, I decided to look into
NettyRoutingFilter. That's where Spring Cloud Gateway makes HTTP requests to backend services. I put a break point at the start of
filter method and inspected the
exchange parameter. My
Connection header was there. I proceeded to read the rest of the method, and found this line:
HttpHeaders filtered = filterRequest(getHeadersFilters(), exchange);
Turns out there's a set of
HttpHeadersFilters that operate on HTTP headers and can exclude some of the ones previously set by
GatewayFilters. In my particular case the culprit was
RemoveHopByHopHeadersFilter. As its name suggests, its function is to remove headers that are only relevant for requests between client and API Gateway, and not intended to be proxied to backend services. In this particular case, I wanted to retain the
Connection header. Fortunately
RemoveHopByHopHeadersFilter can be configured with external configuration.
The solution was to add the following:
spring.cloud.gateway.filter.remove-hop-by-hop.headers: - transfer-encoding - te - trailer - proxy-authorization - proxy-authenticate - x-application-context - upgrade
application.yaml config file. With that
RemoveHopByHopHeadersFilter no longer removed the
Connection header. Tests passed, I deployed API Gateway and backend services handled themselves better.
I've created a demo project that illustrates this issue and the solution. You can find it on GitHub. Feel free to play around with it, and if you find additional issues, or better solutions, please let me know about them at: