HTTP upgrades

Upgrades can be enabled or disabled on a per-route basis. Any per-route enabling/disabling automatically overrides HttpConnectionManager configuration as laid out below, but custom filter chains can only be configured on a per-HttpConnectionManager basis.

Note that the statistics for upgrades are all bundled together so WebSocket and other upgrades are tracked by stats such as downstream_cx_upgrades_total and downstream_cx_upgrades_active

While HTTP/2 and HTTP/3 support for WebSockets is off by default, Envoy does support tunneling WebSockets over HTTP/2 and above for deployments that prefer a uniform HTTP/2+ mesh throughout; this enables, for example, a deployment of the form:

[Client] —- HTTP/1.1 —- [Front Envoy] —- HTTP/2 —- [Sidecar Envoy —- H1 —- App]

In this case, if a client is for example using WebSocket, we want the Websocket to arrive at the upstream server functionally intact, which means it needs to traverse the HTTP/2+ hop.

This is accomplished for HTTP/2 via Extended CONNECT (RFC8441) support, turned on by setting true at the second layer Envoy. For HTTP/3 there is parallel support configured by the alpha option allow_extended_connect as there is no formal RFC yet.

The WebSocket request will be transformed into an HTTP/2+ CONNECT stream, with :protocol header indicating the original upgrade, traverse the HTTP/2+ hop, and be downgraded back into an HTTP/1 WebSocket Upgrade. This same Upgrade-CONNECT-Upgrade transformation will be performed on any HTTP/2+ hop, with the documented flaw that the HTTP/1.1 method is always assumed to be GET. Non-WebSocket upgrades are allowed to use any valid HTTP method (i.e. POST) and the current upgrade/downgrade mechanism will drop the original method and transform the Upgrade request to a GET method on the final Envoy-Upstream hop.

Note that the HTTP/2+ upgrade path has very strict HTTP/1.1 compliance, so will not proxy WebSocket upgrade requests or responses with bodies.

Envoy CONNECT support is off by default (Envoy will send an internally generated 403 in response to CONNECT requests). CONNECT support can be enabled via the upgrade options described above, setting the upgrade value to the special keyword “CONNECT”.

While for HTTP/2 and above, CONNECT request may have a path, in general and for HTTP/1.1 CONNECT requests do not have a path, and can only be matched using a . Please also note that when doing non-wildcard domain matching for CONNECT requests, the CONNECT target is matched rather than the Host/Authority header. You may need to include the port (e.g. “hostname:port”) to successfully match.

Envoy can handle CONNECT in one of two ways, either proxying the CONNECT headers through as if they were any other request, and letting the upstream terminate the CONNECT request, or by terminating the CONNECT request, and forwarding the payload as raw TCP data. When CONNECT upgrade configuration is set up, the default behavior is to proxy the CONNECT request, treating it like any other request using the upgrade path. If termination is desired, this can be accomplished by setting connect_config If it that message is present for CONNECT requests, the router filter will strip the request headers, and forward the HTTP payload upstream. On receipt of initial TCP data from upstream, the router will synthesize 200 response headers, and then forward the TCP data as the HTTP response body.

Warning

For an example of proxying connect, please see For an example of terminating connect, please see configs/terminate_http1_connect.yaml and

Note that for CONNECT-over-tls, Envoy can not currently be configured to do the CONNECT request in the clear and encrypt previously unencrypted payload in one hop. To send CONNECT in plaintext and encrypt the payload, one must first forward the HTTP payload over an “upstream” TLS loopback connection to encrypt it, then have a TCP listener take the encrypted payload and send the CONNECT upstream.

Envoy also has support for tunneling raw TCP over HTTP CONNECT or HTTP POST requests. Find below some usage scenarios.

HTTP/2+ CONNECT can be used to proxy multiplexed TCP over pre-warmed secure connections and amortize the cost of any TLS handshake. An example set up proxying SMTP would look something like this:

[SMTP Upstream] — raw SMTP — [L2 Envoy] — SMTP tunneled over HTTP/2 CONNECT — [L1 Envoy] — raw SMTP — [Client]

HTTP/1.1 CONNECT can be used to have TCP client connecting to its own destination passing through an HTTP proxy server (e.g. corporate proxy not supporting HTTP/2):

[HTTP Server] — raw HTTP — [L2 Envoy] — HTTP tunneled over HTTP/1.1 CONNECT — [L1 Envoy] — raw HTTP — [HTTP Client]

Note that when using HTTP/1 CONNECT you will end up having a TCP connection between L1 and L2 Envoy for each TCP client connection, it is preferable to use HTTP/2 or above when you have the choice.

HTTP POST can also be used to proxy multiplexed TCP when intermediate proxies that don’t support CONNECT. An example set up proxying HTTP would look something like this:

[TCP Server] — raw TCP — [L2 Envoy] — TCP tunneled over HTTP/2 or HTTP/1.1 POST — [Intermidate Proxies] — HTTP/2 or HTTP/1.1 POST — [L1 Envoy] — raw TCP — [TCP Client]

Examples of such a set up can be found in the Envoy example config directory For HTTP/1.1 CONNECT run and bazel-bin/source/exe/envoy-static --config-path configs/terminate_http1_connect.yaml. For HTTP/2 CONNECT run and bazel-bin/source/exe/envoy-static --config-path configs/terminate_http2_connect.yaml. For HTTP/2 POST run and bazel-bin/source/exe/envoy-static --config-path configs/terminate_http2_post.yaml.

In all cases you will be running a first Envoy listening for TCP traffic on port 10000 and encapsulating it in an HTTP CONNECT or HTTP POST request, and a second one listening on 10001, stripping the CONNECT headers (not needed for POST request), and forwarding the original TCP upstream, in this case to google.com.

Envoy waits for the HTTP tunnel to be established (i.e. a successful response to the CONNECT request is received), before start streaming the downstream TCP data to the upstream.