Spring Cloud Gateway Connection header

I've recently been working a lot with Spring Cloud Gateway. It's been (mostly) a pleasant experience, but I do encounter some annoying quirks from time to time. This is a story about one of them.

Use-case

A few days ago I was debugging an issue at work when a backend microservice developer that my API Gateway proxies to asked me to try using keep-alive connections. The request sounded reasonable enough, adding headers to downstream request is easy in Cloud Gateway. I already have few headers that I'm adding.

Hold my beer. I got this.

me, going into this

setRequestHeader

I configure my Spring Gateway routes programmatically, so adding headers to request looks something like this:

builder.routes()
  .route(r -> r.path("/foo/bars")
    .filters(f -> f.setRequestHeader(HttpHeaders.CONNECTION, "keep-alive"))
    .uri(backendUrl))
  .build();

I already had my route set up, so all I needed to add was the setRequestHeader line. So far so good.

Signs of trouble

Next up, I updated tests to check for the new header. This is where I detected the problem. I use WireMock to simulate backend services in tests. Checking requests that API Gateway sends downstream is straightforward:

verify(getRequestedFor(urlEqualTo("/foo/bars"))
  .withHeader("Connection", equalTo("keep-alive"))
);

And the test failed. Here's what WireMock told me:

No requests exactly matched. Most similar request was:  expected:<
GET
/foo/bars

Connection: keep-alive
> but was:<
GET
/foo/bars

Basically, requests were going through, the route was properly sending them to mocked backend service, but Connection header was missing.

Debugging

Having a test that fails consistently was useful, because it allowed me to debug the issue. I put my first breakpoint inside SetRequestHeaderGatewayFilterFactory class. That's where GatewayFilter that sets headers is implemented. I ran my test and everything looked good. Connection header was added to mutated request, and mutated exchange was passed on to filter chain.

Next up, I decided to look into NettyRoutingFilter. That's where Spring Cloud Gateway makes HTTP requests to backend services. I put a break point at the start of filter method and inspected the exchange parameter. My Connection header was there. I proceeded to read the rest of the method, and found this line:

HttpHeaders filtered = filterRequest(getHeadersFilters(), exchange);

Turns out there's a set of HttpHeadersFilters that operate on HTTP headers and can exclude some of the ones previously set by GatewayFilters. In my particular case the culprit was RemoveHopByHopHeadersFilter. As its name suggests, its function is to remove headers that are only relevant for requests between client and API Gateway, and not intended to be proxied to backend services. In this particular case, I wanted to retain the Connection header. Fortunately RemoveHopByHopHeadersFilter can be configured with external configuration.

Solution

The solution was to add the following:

spring.cloud.gateway.filter.remove-hop-by-hop.headers:
    - transfer-encoding
    - te
    - trailer
    - proxy-authorization
    - proxy-authenticate
    - x-application-context
    - upgrade

to application.yaml config file. With that RemoveHopByHopHeadersFilter no longer removed the Connection header. Tests passed, I deployed API Gateway and backend services handled themselves better.

Example

I've created a demo project that illustrates this issue and the solution. You can find it on GitHub. Feel free to play around with it, and if you find additional issues, or better solutions, please let me know about them at: