compressor: expose generic compressor filter to users (envoyproxy#10553)

Currently the generic HTTP compressor filter isn't exposed to users
even though it's used internally by `envoy.filters.http.gzip` and can be
used by external filter extensions.

Expose the compressor's config API to users. For example the filter
can be configured as follows:

    ...
    filter_chains:
      filters:
      - name: envoy.http_connection_manager
        config:
          http_filters:
          - name: envoy.filters.http.compressor
            config:
              disable_on_etag_header: true
              content_length: 100
              content_type:
                - text/html
                - application/json
              compressor_library:
                name: envoy.filters.http.compressor.gzip
                config:
                  memory_level: 3
                  window_bits: 10
                  compression_level: best
                  compression_strategy: rle
    ...

Multiple compressor filters using different compressor libraries,
e.g. gzip and brotli, can be stacked in one filter chain.

Signed-off-by: Dmitry Rozhkov <[email protected]>

CODEOWNERS

@@ -6,6 +6,9 @@
 /api/ @envoyproxy/api-shepherds
 # access loggers
 /*/extensions/access_loggers/common @auni53 @zuercher
+# compression extensions
+/*/extensions/compression/common/compressor @rojkov @junr03
+/*/extensions/compression/gzip/compressor @rojkov @junr03
 # csrf extension
 /*/extensions/filters/http/csrf @dschaller @mattklein123
 # original_src http filter extension
@@ -22,6 +25,7 @@ extensions/filters/common/original_src @snowp @klarose
 /*/extensions/filters/network/thrift_proxy @zuercher @brian-pane
 # compressor used by http compression filters
 /*/extensions/filters/http/common/compressor @gsagula @rojkov @dio
+/*/extensions/filters/http/compressor @rojkov @dio
 # jwt_authn http filter extension
 /*/extensions/filters/http/jwt_authn @qiwzhang @lizan
 # grpc_http1_reverse_bridge http filter extension

api/BUILD

@@ -161,6 +161,7 @@ proto_library(
         "//envoy/extensions/common/dynamic_forward_proxy/v3:pkg",
         "//envoy/extensions/common/ratelimit/v3:pkg",
         "//envoy/extensions/common/tap/v3:pkg",
+        "//envoy/extensions/compression/gzip/compressor/v3:pkg",
         "//envoy/extensions/filters/common/fault/v3:pkg",
         "//envoy/extensions/filters/http/adaptive_concurrency/v3:pkg",
         "//envoy/extensions/filters/http/aws_lambda/v3:pkg",

api/envoy/extensions/compression/gzip/compressor/v3/BUILD

@@ -0,0 +1,9 @@
+# DO NOT EDIT. This file is generated by tools/proto_sync.py.
+
+load("@envoy_api//bazel:api_build_system.bzl", "api_proto_package")
+
+licenses(["notice"])  # Apache 2
+
+api_proto_package(
+    deps = ["@com_github_cncf_udpa//udpa/annotations:pkg"],
+)

api/envoy/extensions/compression/gzip/compressor/v3/gzip.proto

@@ -0,0 +1,79 @@
+syntax = "proto3";
+
+package envoy.extensions.compression.gzip.compressor.v3;
+
+import "google/protobuf/wrappers.proto";
+
+import "udpa/annotations/status.proto";
+import "udpa/annotations/versioning.proto";
+import "validate/validate.proto";
+
+option java_package = "io.envoyproxy.envoy.extensions.compression.gzip.compressor.v3";
+option java_outer_classname = "GzipProto";
+option java_multiple_files = true;
+option (udpa.annotations.file_status).package_version_status = ACTIVE;
+
+// [#protodoc-title: Gzip]
+// [#extension: envoy.compression.gzip.compressor]
+
+// [#next-free-field: 6]
+message Gzip {
+  // All the values of this enumeration translate directly to zlib's compression strategies.
+  // For more information about each strategy, please refer to zlib manual.
+  enum CompressionStrategy {
+    DEFAULT_STRATEGY = 0;
+    FILTERED = 1;
+    HUFFMAN_ONLY = 2;
+    RLE = 3;
+    FIXED = 4;
+  }
+
+  enum CompressionLevel {
+    option allow_alias = true;
+
+    DEFAULT_COMPRESSION = 0;
+    BEST_SPEED = 1;
+    COMPRESSION_LEVEL_1 = 1;
+    COMPRESSION_LEVEL_2 = 2;
+    COMPRESSION_LEVEL_3 = 3;
+    COMPRESSION_LEVEL_4 = 4;
+    COMPRESSION_LEVEL_5 = 5;
+    COMPRESSION_LEVEL_6 = 6;
+    COMPRESSION_LEVEL_7 = 7;
+    COMPRESSION_LEVEL_8 = 8;
+    COMPRESSION_LEVEL_9 = 9;
+    BEST_COMPRESSION = 9;
+  }
+
+  // Value from 1 to 9 that controls the amount of internal memory used by zlib. Higher values
+  // use more memory, but are faster and produce better compression results. The default value is 5.
+  google.protobuf.UInt32Value memory_level = 1 [(validate.rules).uint32 = {lte: 9 gte: 1}];
+
+  // A value used for selecting the zlib compression level. This setting will affect speed and
+  // amount of compression applied to the content. "BEST_COMPRESSION" provides higher compression
+  // at the cost of higher latency and is equal to "COMPRESSION_LEVEL_9". "BEST_SPEED" provides
+  // lower compression with minimum impact on response time, the same as "COMPRESSION_LEVEL_1".
+  // "DEFAULT_COMPRESSION" provides an optimal result between speed and compression. According
+  // to zlib's manual this level gives the same result as "COMPRESSION_LEVEL_6".
+  // This field will be set to "DEFAULT_COMPRESSION" if not specified.
+  CompressionLevel compression_level = 2 [(validate.rules).enum = {defined_only: true}];
+
+  // A value used for selecting the zlib compression strategy which is directly related to the
+  // characteristics of the content. Most of the time "DEFAULT_STRATEGY" will be the best choice,
+  // which is also the default value for the parameter, though there are situations when
+  // changing this parameter might produce better results. For example, run-length encoding (RLE)
+  // is typically used when the content is known for having sequences which same data occurs many
+  // consecutive times. For more information about each strategy, please refer to zlib manual.
+  CompressionStrategy compression_strategy = 3 [(validate.rules).enum = {defined_only: true}];
+
+  // Value from 9 to 15 that represents the base two logarithmic of the compressor's window size.
+  // Larger window results in better compression at the expense of memory usage. The default is 12
+  // which will produce a 4096 bytes window. For more details about this parameter, please refer to
+  // zlib manual > deflateInit2.
+  google.protobuf.UInt32Value window_bits = 4 [(validate.rules).uint32 = {lte: 15 gte: 9}];
+
+  // Value for Zlib's next output buffer. If not set, defaults to 4096.
+  // See https://www.zlib.net/manual.html for more details. Also see
+  // https://github.com/envoyproxy/envoy/issues/8448 for context on this filter's performance.
+  google.protobuf.UInt32Value chunk_size = 5 [(validate.rules).uint32 = {lte: 65536 gte: 4096}];
+}

api/envoy/extensions/filters/http/compressor/v3/compressor.proto

@@ -3,20 +3,25 @@ syntax = "proto3";
 package envoy.extensions.filters.http.compressor.v3;
 
 import "envoy/config/core/v3/base.proto";
+import "envoy/config/core/v3/extension.proto";
 
+import "google/protobuf/any.proto";
 import "google/protobuf/wrappers.proto";
 
 import "udpa/annotations/status.proto";
 import "udpa/annotations/versioning.proto";
+import "validate/validate.proto";
 
 option java_package = "io.envoyproxy.envoy.extensions.filters.http.compressor.v3";
 option java_outer_classname = "CompressorProto";
 option java_multiple_files = true;
 option (udpa.annotations.file_status).package_version_status = ACTIVE;
 
 // [#protodoc-title: Compressor]
+// Compressor :ref:`configuration overview <config_http_filters_compressor>`.
+// [#extension: envoy.filters.http.compressor]
 
-// [#next-free-field: 6]
+// [#next-free-field: 7]
 message Compressor {
   option (udpa.annotations.versioning).previous_message_type =
       "envoy.config.filter.http.compressor.v2.Compressor";
@@ -46,4 +51,10 @@ message Compressor {
   // Runtime flag that controls whether the filter is enabled or not. If set to false, the
   // filter will operate as a pass-through filter. If not specified, defaults to enabled.
   config.core.v3.RuntimeFeatureFlag runtime_enabled = 5;
+
+  // A compressor library to use for compression. Currently only
+  // :ref:`envoy.filters.http.compressor.gzip<envoy_api_msg_extensions.compression.gzip.compressor.v3.Gzip>`
+  // is included in Envoy.
+  // This field is ignored if used in the context of the gzip http-filter, but is mandatory otherwise.
+  config.core.v3.TypedExtensionConfig compressor_library = 6;
 }

api/versioning/BUILD

@@ -44,6 +44,7 @@ proto_library(
         "//envoy/extensions/common/dynamic_forward_proxy/v3:pkg",
         "//envoy/extensions/common/ratelimit/v3:pkg",
         "//envoy/extensions/common/tap/v3:pkg",
+        "//envoy/extensions/compression/gzip/compressor/v3:pkg",
         "//envoy/extensions/filters/common/fault/v3:pkg",
         "//envoy/extensions/filters/http/adaptive_concurrency/v3:pkg",
         "//envoy/extensions/filters/http/aws_lambda/v3:pkg",

docs/root/api-v3/config/compression/compression.rst

@@ -0,0 +1,8 @@
+Compression
+===========
+
+.. toctree::
+  :glob:
+  :maxdepth: 2
+
+  ../../extensions/compression/gzip/*/v3/*

docs/root/api-v3/config/config.rst

@@ -12,6 +12,7 @@ Extensions
   transport_socket/transport_socket
   resource_monitor/resource_monitor
   common/common
+  compression/compression
   cluster/cluster
   grpc_credential/grpc_credential
   retry/retry

docs/root/configuration/http/http_filters/compressor_filter.rst

@@ -0,0 +1,108 @@
+.. _config_http_filters_compressor:
+
+Compressor
+==========
+Compressor is an HTTP filter which enables Envoy to compress dispatched data
+from an upstream service upon client request. Compression is useful in
+situations when bandwidth is scarce and large payloads can be effectively compressed
+at the expense of higher CPU load or offloading it to a compression accelerator.
+
+.. note::
+
+ This filter deprecates the :ref:`HTTP Gzip filter <config_http_filters_gzip>`.
+
+Configuration
+-------------
+* :ref:`v3 API reference <envoy_v3_api_msg_extensions.filters.http.compressor.v3.Compressor>`
+* This filter should be configured with the name *envoy.filters.http.compressor*.
+
+How it works
+------------
+When compressor filter is enabled, request and response headers are inspected to
+determine whether or not the content should be compressed. The content is
+compressed and then sent to the client with the appropriate headers, if
+response and request allow.
+
+Currently the filter supports :ref:`gzip compression <envoy_v3_api_msg_extensions.compression.gzip.compressor.v3.Gzip>`
+only. Other compression libraries can be supported as extensions.
+
+An example configuration of the filter may look like the following:
+
+.. code-block:: yaml
+
+    http_filters:
+    - name: compressor
+      typed_config:
+        "@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
+        disable_on_etag_header: true
+        content_length: 100
+        content_type:
+          - text/html
+          - application/json
+        compressor_library:
+          name: text_optimized
+          typed_config:
+            "@type": type.googleapis.com/envoy.extensions.filters.http.compressor.gzip.v3.Gzip
+            memory_level: 3
+            window_bits: 10
+            compression_level: best
+            compression_strategy: default_strategy
+
+By *default* compression will be *skipped* when:
+
+- A request does NOT contain *accept-encoding* header.
+- A request includes *accept-encoding* header, but it does not contain "gzip" or "\*".
+- A request includes *accept-encoding* with "gzip" or "\*" with the weight "q=0". Note
+  that the "gzip" will have a higher weight then "\*". For example, if *accept-encoding*
+  is "gzip;q=0,\*;q=1", the filter will not compress. But if the header is set to
+  "\*;q=0,gzip;q=1", the filter will compress.
+- A request whose *accept-encoding* header includes any encoding type with a higher
+  weight than "gzip"'s given the corresponding compression filter is present in the chain.
+- A response contains a *content-encoding* header.
+- A response contains a *cache-control* header whose value includes "no-transform".
+- A response contains a *transfer-encoding* header whose value includes "gzip".
+- A response does not contain a *content-type* value that matches one of the selected
+  mime-types, which default to *application/javascript*, *application/json*,
+  *application/xhtml+xml*, *image/svg+xml*, *text/css*, *text/html*, *text/plain*,
+  *text/xml*.
+- Neither *content-length* nor *transfer-encoding* headers are present in
+  the response.
+- Response size is smaller than 30 bytes (only applicable when *transfer-encoding*
+  is not chunked).
+
+Please note that in case the filter is configured to use a compression library extension
+other than gzip it looks for content encoding in the *accept-encoding* header provided by
+the extension.
+
+When compression is *applied*:
+
+- The *content-length* is removed from response headers.
+- Response headers contain "*transfer-encoding: chunked*" and do not contain
+  "*content-encoding*" header.
+- The "*vary: accept-encoding*" header is inserted on every response.
+
+.. _compressor-statistics:
+
+Statistics
+----------
+
+Every configured Compressor filter has statistics rooted at
+<stat_prefix>.compressor.<compressor_library.name>.<compressor_library_stat_prefix>.*
+with the following:
+
+.. csv-table::
+  :header: Name, Type, Description
+  :widths: 1, 1, 2
+
+  compressed, Counter, Number of requests compressed.
+  not_compressed, Counter, Number of requests not compressed.
+  no_accept_header, Counter, Number of requests with no accept header sent.
+  header_identity, Counter, Number of requests sent with "identity" set as the *accept-encoding*.
+  header_compressor_used, Counter, Number of requests sent with "gzip" set as the *accept-encoding*.
+  header_compressor_overshadowed, Counter, Number of requests skipped by this filter instance because they were handled by another filter in the same filter chain.
+  header_wildcard, Counter, Number of requests sent with "\*" set as the *accept-encoding*.
+  header_not_valid, Counter, Number of requests sent with a not valid *accept-encoding* header (aka "q=0" or an unsupported encoding type).
+  total_uncompressed_bytes, Counter, The total uncompressed bytes of all the requests that were marked for compression.
+  total_compressed_bytes, Counter, The total compressed bytes of all the requests that were marked for compression.
+  content_length_too_small, Counter, Number of requests that accepted gzip encoding but did not compress because the payload was too small.
+  not_compressed_etag, Counter, Number of requests that were not compressed due to the etag header. *disable_on_etag_header* must be turned on for this to happen.

docs/root/configuration/http/http_filters/gzip_filter.rst

@@ -1,5 +1,10 @@
 .. _config_http_filters_gzip:
 
+.. warning::
+
+  This filter has been deprecated in favor the
+  :ref:`HTTP Compressor filter <config_http_filters_compressor>`.
+
 Gzip
 ====
 Gzip is an HTTP filter which enables Envoy to compress dispatched data

docs/root/configuration/http/http_filters/http_filters.rst

@@ -10,6 +10,7 @@ HTTP filters
   aws_lambda_filter
   aws_request_signing_filter
   buffer_filter
+  compressor_filter
   cors_filter
   csrf_filter
   dynamic_forward_proxy_filter

docs/root/version_history/current.rst

@@ -7,6 +7,7 @@ Changes
 * access loggers: added GRPC_STATUS operator on logging format.
 * access loggers: applied existing buffer limits to the non-google gRPC access logs, as well as :ref:`stats <config_access_log_stats>` for logged / dropped logs.
 * access loggers: extened specifier for FilterStateFormatter to output :ref:`unstructured log string <config_access_log_format_filter_state>`.
+* compressor: generic :ref:`compressor <config_http_filters_compressor>` filter exposed to users.
 * config: added :ref:`version_text <config_cluster_manager_cds>` stat that reflects xDS version.
 * dynamic forward proxy: added :ref:`SNI based dynamic forward proxy <config_network_filters_sni_dynamic_forward_proxy>` support.
 * fault: added support for controlling the percentage of requests that abort, delay and response rate limits faults
@@ -53,6 +54,8 @@ Deprecated
 * Tracing provider configuration as part of :ref:`bootstrap config <envoy_v3_api_field_config.bootstrap.v3.Bootstrap.tracing>`
   has been deprecated in favor of configuration as part of :ref:`HTTP connection manager
   <envoy_v3_api_field_extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.Tracing.provider>`.
+* The :ref:`HTTP Gzip filter <config_http_filters_gzip>` has been deprecated in favor of
+  :ref:`Compressor <config_http_filters_compressor>`.
 * The * :ref:`GoogleRE2.max_program_size<envoy_v3_api_field_type.matcher.v3.RegexMatcher.GoogleRE2.max_program_size>`
   field is now deprecated. Management servers are expected to validate regexp program sizes
   instead of expecting the client to do it.