Configuring Envoy as a Forward Proxy with AWS API Gateway IAM Authorization

[njeddi]

Hello,

We are working on a use case where we want to configure Envoy as a forward proxy to intercept all egress traffic with external authentication using AWS API Gateway (IAM authorization). However, when attempting to use the External Authorization (ExtAuthz) filter, we encounter a 403 error without any logs being traced in API Gateway’s CloudWatch logs.

Here’s the flow we are trying to achieve:

1- Incoming request (to access an external URL)
2- Envoy (acting as the forward proxy)
3- External Authorization (ExtAuthz) with AWS API Gateway IAM Authorizer
4- If authentication is successful, the request is forwarded to the internet.

Could you confirm if Envoy is capable of handling this scenario? Additionally, if anyone has experience with this configuration, could you please share any example setups or guidance?

Thank you for your help!

Best regards,

[phlax]

cc @suniltheta @mattklein123 @nbaws for aws expertise
cc @esmet @tyxia @ggreenway for ext_authz

[nbaws]

@njeddi Yes, envoy is capable of handling this scenario.
I'm not familiar with configuring ext_authz, but a simple configuration for API gateway is as follows:

1. Create API gateway HTTP API
2. Create Route to a hello world style Lambda
3. Launch an EC2 with an instance profile, containing an inline policy such as the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "execute-api:Invoke"
            ],
            "Resource": [
                "<your api arn and path>"
            ]
        }
    ]
}

4. Launch envoy on the same EC2 with a configuration similar to the following (based on https://www.envoyproxy.io/docs/envoy/latest/_downloads/f29dcefd0a508a4dbcf821d5e7e6614d/aws-request-signing-filter.yaml ) replacing with the hostname of your api gateway url.

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: AUTO
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: app
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: versioned-cluster
          http_filters:
          - name: envoy.filters.http.aws_request_signing
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigning
              service_name: execute-api
              region: ap-southeast-2
              host_rewrite: <your api gateway hostname>
              match_excluded_headers:
              - prefix: x-envoy
              - prefix: x-forwarded
              - exact: x-amzn-trace-id
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
  - name: versioned-cluster
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: versioned-cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: <your api gateway hostname>
                port_value: 443
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        sni: <your api gateway hostname>
        common_tls_context:
          validation_context:
            trusted_ca: {filename: /etc/ssl/certs/ca-certificates.crt}

5. Test you can reach your API with curl directly ( curl api-gateway-url/route )
6. Test you can reach your API with curl via envoy ( curl localhost/route )
7. Attach IAM authorizer to your API
8. Test you now cannot reach your API with curl directly ( curl api-gateway-url/route )
9. Test you can reach your API with curl via envoy ( curl localhost/route )

If you are receiving Forbidden messages at step 9, you need to check that your IAM policy is correct, it is attached to your instance profile correctly, and that your hostname rewrite ( host_rewrite section ) is correct. SigV4 Authentication requires the hostname in the sent request to be accurate for authentication to succeed.
You will also be able to troubleshoot this on the envoy side by running envoy with flag -l trace and looking through the AWS debug logs.