Status Reporting for VirtualHostOptions in Kube GW (solo-io#9391)

* use status syncer and factory to propagate proxy reports

* add gwv2 status syncer

* regen and pr comments

* add filterProxiesByControllerName, regen

* propagate reports async to status syncer

* propagate reports async to status syncer

* minor fixes

* use uuid for proxy in status syncer

* revert uuid, proxy pointer works with status syncer

* add e2e test

* fix import, remove files

* passing tests

* changelog

* changelog, regen

* move changelog

* fix assertion

* fix port forward

* check fault injection from curl container instead

* use curl namespace

* remove test framework generics, add clientset

* improve comment

* skip istio cli tests

* pr feedback e2e test

* clients and curl

* stdin -> stdout

* use ephemeral container

* threadsafe buf

* clean up diff

* pr feedback

* fix compile issue

* add changelog, remove istio skip

* goimports

* metav1

* initial route option e2e test

* changelog

* remove extra changelog

* switch to CurlFromEphemeralPod

* clean up diff

* use EventuallyResourceAccepted

* pr feedback

* add vhostopt status

* wip, working unit test

* wip, status matcher

* fix status assertion

* add status and tests to vhopts

* found where the resource client wasn't wired up to the status reporter right

* status syncer refactor

* fix some compile issues

* fix status syncer tests

* ch

* pr feedback, fix status syncer test, delete extra ch

* kube e2e tests for vho status

* add status matcher

* fix some compile issues

* fix linter

* remove extra changelog, revert yq Makefile changes

* mod tidy

* getters

* add changelog

* revert some oopsies

* pr feedback

* remove extra added getter

* cleanup

* Adding changelog file to new location

* Deleting changelog file from old location

* fix ginkgo v2 import

* return properly from resource client creation

* rename status matcher types

* don't assign numOpts, use comparison directly

---------

Co-authored-by: Scott Weiss <[email protected]>
Co-authored-by: npolshakova <[email protected]>
Co-authored-by: soloio-bulldozer[bot] <48420018+soloio-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Sam Heilbron <[email protected]>
Co-authored-by: changelog-bot <changelog-bot>

changelog/v1.17.0-beta25/vhopts-statuses.yaml

@@ -0,0 +1,6 @@
+changelog:
+- type: NEW_FEATURE
+  issueLink: https://github.com/solo-io/solo-projects/issues/6044
+  resolvesIssue: true
+  description: >-
+    Add VirtualHostOptions status tracking for Kubernetes Gateways

devel/architecture/external-options.md

@@ -0,0 +1,38 @@
+# External Options (VirtualHostOption and RouteOption)
+
+## Overview
+Gloo Edge supports decoupling certain resources from the definitions of options related to the resource. Currently there are two objects which support delegating options definition in this way:
+- A VirtualHost (exists on a VirtualService) can be configured with the separate resource [VirtualHostOption](./projects/gateway/api/v1/external_options.proto).
+- A Route (exists on a VirtualHost within a VirtualService or on a RouteTable) can be configured with the separate resource [RouteOption](./projects/gateway/api/v1/external_options.proto).
+
+The Kubernetes CRDs for these resources are generated by solo-kit, NOT skv2.
+
+The options that are defined directly on the resource, if non-nil, will always take precedence over the delegated options resources.
+
+## Classic Gateway
+<!-- TODO -->
+
+## Kubernetes Gateway
+RouteOption and VirtualHostOption resources are honored in the Kubernetes Gateway integration via plugins. These are associated with the related resources in slightly different ways, and the Statuses reported for them have subtleties as well.
+
+### RouteOptions
+
+#### Overview 
+RouteOption resources are associated with the Kubernetes Gateway API resource [HTTPRoute](https://gateway-api.sigs.k8s.io/api-types/httproute/). They may be specified by the HTTPRoute as an extensionRef or on the targetRef field of the RouteOption resource. This "policy attachment" pattern is experimental as of this writing but widely adopted. More details on policy attachment can be found in [GEP-713](https://gateway-api.sigs.k8s.io/geps/gep-713/).
+
+### VirtualHostOptions
+
+#### Overview 
+VirtualHostOption resources are associated with the Kubernetes Gateway API resource [Gateway](https://gateway-api.sigs.k8s.io/api-types/httproute/). They may be specified on the targetRef field of the VirtualHostOption resource. Unlike RouteOption, VirtualHostOption resources support the field `sectionName`, which if specified will (at the time of this writing) indicate which [Kubernetes Gateway API Listener](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1.Listener) the resource should target. If `sectionName` is not specified, all Listeners on the targeted Gateway will be considered associated with the resource. This "policy attachment" pattern is experimental as of this writing but widely adopted. More details on policy attachment can be found in [GEP-713](https://gateway-api.sigs.k8s.io/geps/gep-713/).
+
+It is important to note that since `VirtualHost` is an Envoy primitive, and transitively a Classic Gateway API, without an analogous resource in the Kubernetes Gateway API spec, users of the latter have no direct interaction with `VirtualHost`s. Instead, a Kubernetes Gateway API Listener is translated to a gloov1.Listener, and `VirtualHost`s on that gloov1.Listener are automatically assigned based on what filter chains were created during Listener translation.
+
+#### Policy selection
+In order to keep the original implementation simple, we currently only support one `VirtualHostOption` resource being attached to any given Kubernetes Gateway API Listener. Because an arbitrary number of `VirtualHostOption` resources may be attached to any given Kubernetes Gateway API Gateway with or without `sectionName`, we need some sort of prioritization. This priority order is as follows:
+1. (Highest priority) Oldest `VirtualHostOption` targeting the Listener with `sectionName`
+2. Newer `VirtualHostOption`s targeting the Listener with `sectionName`
+3. Oldest `VirtualHostOption` targeting the Gateway on which the Listener resides, but without `sectionName` defined
+4. (Lowest priority) Newer `VirtualHostOption`s targeting the Gateway on which the Listener resides, but without `sectionName` defined
+
+#### Status
+To provide some insight into which options are applied to a given Kubernetes Gateway API Listener, warnings are logged in the `Status` of the `VirtualHostOption` resource if it would have been attached to a Listener, but was not due to being lower priority than another valid `VirtualHostOption` resource. 

projects/gateway2/controller/start.go

@@ -57,6 +57,9 @@ type StartConfig struct {
 	// RouteOptionClient is the client used for retrieving RouteOption objects within the RouteOptionsPlugin
 	// NOTE: We may be able to move this entirely to the RouteOptionsPlugin
 	RouteOptionClient gatewayv1.RouteOptionClient
+	// VirtualHostOptionClient is the client used for retrieving VirtualHostOption objects within the VirtualHostOptionsPlugin
+	// NOTE: We may be able to move this entirely to the VirtualHostOptionsPlugin
+	VirtualHostOptionClient gatewayv1.VirtualHostOptionClient
 	// StatusReporter is used within any StatusPlugins that must persist a GE-classic style status
 	StatusReporter reporter.StatusReporter
 
@@ -102,11 +105,12 @@ func Start(ctx context.Context, cfg StartConfig) error {
 	inputChannels := proxy_syncer.NewGatewayInputChannels()
 
 	k8sGwExtensions, err := cfg.ExtensionsFactory(ctx, extensions.K8sGatewayExtensionsFactoryParameters{
-		Mgr:               mgr,
-		AuthConfigClient:  cfg.AuthConfigClient,
-		RouteOptionClient: cfg.RouteOptionClient,
-		StatusReporter:    cfg.StatusReporter,
-		KickXds:           inputChannels.Kick,
+		Mgr:                     mgr,
+		RouteOptionClient:       cfg.RouteOptionClient,
+		VirtualHostOptionClient: cfg.VirtualHostOptionClient,
+		StatusReporter:          cfg.StatusReporter,
+		KickXds:                 inputChannels.Kick,
+		AuthConfigClient:        cfg.AuthConfigClient,
 	})
 	if err != nil {
 		setupLog.Error(err, "unable to create k8s gw extensions")

projects/gateway2/extensions/extensions.go

@@ -25,11 +25,12 @@ type K8sGatewayExtensions interface {
 
 // K8sGatewayExtensionsFactoryParameters contains the parameters required to start Gloo K8s Gateway Extensions (including Translator Plugins)
 type K8sGatewayExtensionsFactoryParameters struct {
-	Mgr               controllerruntime.Manager
-	AuthConfigClient  v1.AuthConfigClient
-	RouteOptionClient gatewayv1.RouteOptionClient
-	StatusReporter    reporter.StatusReporter
-	KickXds           func(ctx context.Context)
+	Mgr                     controllerruntime.Manager
+	AuthConfigClient        v1.AuthConfigClient
+	RouteOptionClient       gatewayv1.RouteOptionClient
+	VirtualHostOptionClient gatewayv1.VirtualHostOptionClient
+	StatusReporter          reporter.StatusReporter
+	KickXds                 func(ctx context.Context)
 }
 
 // K8sGatewayExtensionsFactory returns an extensions.K8sGatewayExtensions
@@ -44,16 +45,18 @@ func NewK8sGatewayExtensions(
 	params K8sGatewayExtensionsFactoryParameters,
 ) (K8sGatewayExtensions, error) {
 	return &k8sGatewayExtensions{
-		params.Mgr,
-		params.RouteOptionClient,
-		params.StatusReporter,
+		mgr:                     params.Mgr,
+		routeOptionClient:       params.RouteOptionClient,
+		virtualHostOptionClient: params.VirtualHostOptionClient,
+		statusReporter:          params.StatusReporter,
 	}, nil
 }
 
 type k8sGatewayExtensions struct {
-	mgr               controllerruntime.Manager
-	routeOptionClient gatewayv1.RouteOptionClient
-	statusReporter    reporter.StatusReporter
+	mgr                     controllerruntime.Manager
+	routeOptionClient       gatewayv1.RouteOptionClient
+	virtualHostOptionClient gatewayv1.VirtualHostOptionClient
+	statusReporter          reporter.StatusReporter
 }
 
 // CreatePluginRegistry returns the PluginRegistry
@@ -66,6 +69,7 @@ func (e *k8sGatewayExtensions) CreatePluginRegistry(_ context.Context) registry.
 		queries,
 		e.mgr.GetClient(),
 		e.routeOptionClient,
+		e.virtualHostOptionClient,
 		e.statusReporter,
 	)
 	return registry.NewPluginRegistry(plugins)

projects/gateway2/translator/listenerutils/utils.go

@@ -0,0 +1,28 @@
+package listenerutils
+
+import (
+	v1 "github.com/solo-io/gloo/projects/gloo/pkg/api/v1"
+	"github.com/solo-io/solo-kit/pkg/api/v1/resources/core"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+)
+
+func AppendSourceToListener(listener *v1.Listener, source client.Object) {
+	meta := listener.GetMetadataStatic()
+	if meta == nil {
+		meta = &v1.SourceMetadata{}
+	}
+	sources := meta.GetSources()
+	sources = append(sources, &v1.SourceMetadata_SourceRef{
+		ResourceRef: &core.ResourceRef{
+			Name:      source.GetName(),
+			Namespace: source.GetNamespace(),
+		},
+		ResourceKind:       source.GetObjectKind().GroupVersionKind().Kind,
+		ObservedGeneration: source.GetGeneration(),
+	})
+	listener.OpaqueMetadata = &v1.Listener_MetadataStatic{
+		MetadataStatic: &v1.SourceMetadata{
+			Sources: sources,
+		},
+	}
+}

projects/gateway2/translator/plugins/registry/plugin_registry.go

@@ -78,14 +78,15 @@ func BuildPlugins(
 	queries gwquery.GatewayQueries,
 	client client.Client,
 	routeOptionClient gatewayv1.RouteOptionClient,
+	vhostOptionClient gatewayv1.VirtualHostOptionClient,
 	statusReporter reporter.StatusReporter,
 ) []plugins.Plugin {
 	return []plugins.Plugin{
 		headermodifier.NewPlugin(),
 		mirror.NewPlugin(queries),
 		redirect.NewPlugin(),
 		routeoptions.NewPlugin(queries, client, routeOptionClient, statusReporter),
-		virtualhostoptions.NewPlugin(queries, client),
+		virtualhostoptions.NewPlugin(queries, client, vhostOptionClient, statusReporter),
 		urlrewrite.NewPlugin(),
 	}
 }