feat: support port-based routing for Gateway API routes#2703
feat: support port-based routing for Gateway API routes#2703johannes-engler-mw wants to merge 20 commits intoapache:masterfrom
Conversation
This change implements support for port-based routing in HTTPRoute and GRPCRoute by leveraging the 'server_port' variable in APISIX. When a route targets a specific Gateway listener (via 'sectionName' or matching), the translator now adds a condition to ensure the route only matches traffic on that specific port. Key changes: - Update translator to add 'server_port' matching variables to routes. - Enhance controller to track all matching listeners for a route. - Update E2E test framework to support multi-port APISIX and in-cluster testing. - Add comprehensive E2E tests for port-based routing scenarios.
Reflect the new port-based routing capability in Gateway API documentation. - Updated concepts/gateway-api.md to mark listener port as partially supported. - Updated reference/example.md to explain that port is used for matching.
This fixes a linting error flagged by errcheck where the return value of fmt.Sscanf was being ignored.
This ensures the alpine/curl image is present locally before attempting to load it into the Kind cluster, preventing build failures in CI.
- Add comprehensive unit tests for addServerPortVars function - Fix two-port test to use expectedPortList + ElementsMatch for non-deterministic map ordering - Consolidate redundant single-port tests (removed HTTP/HTTPS duplicates) - Fix image tag mismatch: alpine/curl:flattened -> alpine/curl:latest - Fix comment typo: port 9100 -> port 9081 Co-Authored-By: Claude <noreply@anthropic.com>
|
Hi there, While reviewing the IssueThe Example from acceptStatus := ResourceStatus{
status: true,
msg: "Route is accepted",
}
// ...
for _, gateway := range gateways {
if err := ProcessGatewayProxy(...); err != nil {
acceptStatus.status = false // Overwrites status for ALL parents
acceptStatus.msg = err.Error()
}
}Affected ControllersThis pattern exists in all 5 route controllers:
Impact
Proposed FixTrack status per-gateway using a map: gatewayStatuses := make(map[string]ResourceStatus)
for _, gateway := range gateways {
if err := ProcessGatewayProxy(...); err != nil {
gatewayStatuses[gateway.Gateway.Name] = ResourceStatus{status: false, msg: err.Error()}
} else {
gatewayStatuses[gateway.Gateway.Name] = ResourceStatus{status: true, msg: "Route is accepted"}
}
}
// Use per-gateway status when setting parent status conditionsQuestionHow would you like to proceed?
Happy to submit a fix either way. |
|
Hi again, I just realized this PR probably adds a breaking change: The Change My implementation now does exactly this: The Risk This is technically "correct" per the spec, but for users relying on the old "first match only" behavior, this could inadvertently expose internal services to public ports upon upgrade. Proposal
I'm leaning towards option 1 for safety, or at least a very prominent warning. Let me know your preference and I can update the PR accordingly. |
|
Anyone got time to review my PR and answer my questions? |
|
Any update? |
|
@ronething @AlinsRan Can someone of you please have a look? |
Baoyuantop
left a comment
There was a problem hiding this comment.
- addServerPortVars should only be injected when a route explicitly specifies a sectionName (or has multiple listeners with different ports), not for all routes.
- GRPCRoute translator lacks the addServerPortVars call, resulting in incomplete functionality.
- inject server_port vars only for explicit sectionName or multi-port listeners - add server_port parity for GRPCRoute translation - use matched listeners for HTTPRoute hostname filtering and avoid zero-value listener fallback - add translator/controller unit tests and extend HTTPRoute/GRPCRoute e2e coverage
|
@Baoyuantop Thanks for the feedback, I have added your suggestions. |
|
From a Gateway API perspective, Mapping
This works well in a shared-instance model (one APISIX instance serving multiple listeners), but may introduce constraints if we later want to:
To balance practicality and future extensibility, I suggest making this behavior configurable (e.g., via a feature gate or controller configuration option). For example:
This would allow:
|
|
Thanks @AlinsRan for the feedback on making this behavior configurable. I've evaluated two approaches and would like to propose a mode enum configuration rather than a simple boolean toggle. Proposed:
|
| Mode | Behavior |
|---|---|
auto (default) |
Inject server_port vars when route explicitly targets a listener (sectionName or port) or when multiple listener ports are matched. This is the current behavior and maintains backward compatibility. |
explicit |
Inject server_port vars only when route explicitly targets a listener. No heuristic injection for multiple ports. |
off |
Never inject server_port vars. Listener ports are treated as infrastructure-level metadata only. |
Why enum over boolean?
-
Gateway API semantics: The spec distinguishes between implicit attachment (route attaches to gateway, all listeners) and explicit targeting
(route specifiessectionNameorport). A boolean conflates these —explicitmode preserves this distinction whileoffprovides the pure
infrastructure semantics. -
Avoids future breaking changes: If we ship a boolean and later need the
explicitbehavior, we'd have to break the config schema or add a
second field. -
Minimal additional complexity: The difference is essentially a
switchstatement, a type definition, and a validation check. The config
wiring is nearly identical to a boolean approach.
Behavior matrix
| Scenario | auto |
explicit |
off |
|---|---|---|---|
Explicit sectionName targeting |
inject | inject | no injection (warning) |
Explicit port targeting |
inject | inject | no injection (warning) |
| Multiple listener ports, no explicit targeting | inject | no injection | no injection |
| Single listener port, no explicit targeting | no injection | no injection | no injection |
Config example
listener_port_match_mode: auto # Options: auto, explicit, off. Default: auto.The default auto maintains full backward compatibility. Users who need infrastructure-only semantics can set off, and those who want only explicit intent-based injection can use explicit.
Are you comfortable with this approach, or would you prefer the simpler boolean toggle (enable_listener_port_match: true/false)?
BR,
Johannes
…tRefs When an HTTPRoute/GRPCRoute has multiple parentRefs pointing at the same Gateway (e.g. two sectionNames, or one with and one without), the listeners collected from each RouteParentRefContext can overlap. The controllers were blindly appending them all, producing duplicate entries in tctx.Listeners and consequently duplicate hostnames in GRPC route translation. Add appendUniqueListeners helper that deduplicates by listener Name (unique within a Gateway per Gateway API spec) and use it in both httproute_controller and grpcroute_controller instead of plain append.
Very good. |
Extract backend-processing block into a new helper method translateBackendsToUpstreams to bring TranslateHTTPRoute's cyclomatic complexity from 31 down to 9, fixing the gocyclo lint failure.
Introduces a configurable listener_port_match_mode setting (auto/explicit/off) that controls when server_port route vars are injected from Gateway listener ports, allowing operators to tune port-based routing behaviour without redeploying. - auto (default): inject when parentRefs explicitly target a listener via sectionName or port, or when multiple listener ports need disambiguation; preserves existing behaviour - explicit: inject only on explicit listener targeting (sectionName/port) - off: never inject server_port vars Also fixes hasExplicitListenerTarget to skip non-Gateway parentRefs (e.g. GAMMA Service mesh refs), preventing unintended server_port injection in mixed Gateway + Service parentRef routes.
|
Hi @AlinsRan, i have implemented the feature as discussed. Please review the implementation. |
|
Please resolve the code conflict. |
Keep the refactored translateBackendsToUpstreams call from feat/2639 instead of the inline backend-processing block from origin/master.
|
@Baoyuantop Done! |
|
I have set a higher timeout of 30s for the failing e2e test. Please rerun the workflow. |
|
I have fixed the linting issue, please rerun the workflow. |
There was a problem hiding this comment.
Pull request overview
Implements port-based routing for Gateway API HTTPRoute/GRPCRoute by injecting APISIX server_port route vars based on matched Gateway listener ports, with a new controller configuration switch to control injection behavior.
Changes:
- Add
listener_port_match_modeconfig (auto/explicit/off) and propagate it through manager → provider → translator. - Track all matched Gateway listeners per ParentRef and inject
server_portvars into generated APISIX routes where needed. - Expand E2E scaffolding/manifests and add E2E + unit tests covering multi-listener port routing scenarios.
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/e2e/scaffold/scaffold.go | Adds a helper to create httpexpect clients against arbitrary APISIX service ports. |
| test/e2e/scaffold/k8s.go | Adds in-cluster curl runner for testing server_port behavior without port-forward artifacts. |
| test/e2e/scaffold/grpc.go | Adds gRPC request helper that can target specific service ports via port-forward tunnel. |
| test/e2e/gatewayapi/httproute.go | Adds comprehensive E2E coverage for HTTPRoute sectionName/port-based listener targeting across multiple ports. |
| test/e2e/gatewayapi/grpcroute.go | Adds E2E coverage for GRPCRoute sectionName/port-based listener targeting across multiple ports. |
| test/e2e/framework/manifests/ingress.yaml | Makes ADC exec timeout configurable via E2E_EXEC_ADC_TIMEOUT. |
| test/e2e/framework/manifests/apisix.yaml | Configures APISIX to listen on multiple HTTP ports and exposes the extra service port for E2E. |
| internal/provider/options.go | Adds provider option for ListenerPortMatchMode. |
| internal/provider/apisix/provider.go | Passes port-match mode into translator initialization. |
| internal/manager/run.go | Wires controller config ListenerPortMatchMode into provider options. |
| internal/controller/utils_hostname_test.go | Adds unit tests for listener/hostname selection helpers used by Gateway API reconciliation. |
| internal/controller/utils.go | Tracks multiple matched listeners per ParentRef and updates hostname helper logic accordingly. |
| internal/controller/httproute_controller.go | Populates translation context with all matched listeners for HTTPRoute reconciliation. |
| internal/controller/grpcroute_controller.go | Populates translation context with all matched listeners for GRPCRoute reconciliation. |
| internal/controller/context.go | Extends RouteParentRefContext to carry multiple matched listeners. |
| internal/controller/config/types.go | Introduces ListenerPortMatchMode type and constants. |
| internal/controller/config/config_test.go | Adds validation/defaulting tests for listener_port_match_mode. |
| internal/controller/config/config.go | Defaults/validates listener_port_match_mode. |
| internal/adc/translator/translator.go | Adds mode normalization and decision logic for whether to inject server_port vars. |
| internal/adc/translator/httproute_test.go | Adds unit tests verifying server_port injection behavior for HTTPRoute across modes. |
| internal/adc/translator/httproute.go | Adds server_port vars injection and refactors backend→upstream translation into a helper. |
| internal/adc/translator/grpcroute_test.go | Adds unit tests verifying server_port injection behavior for GRPCRoute across modes. |
| internal/adc/translator/grpcroute.go | Adds server_port vars injection for GRPCRoute. |
| internal/adc/translator/annotations_test.go | Updates translator initialization in tests and adds unit tests for server_port var helpers. |
| docs/en/latest/reference/example.md | Updates docs to reflect listener port routing behavior and config knob. |
| docs/en/latest/reference/configuration-file.md | Documents listener_port_match_mode. |
| docs/en/latest/concepts/gateway-api.md | Updates Gateway API support matrix entry for listener port behavior. |
| config/samples/config.yaml | Adds sample listener_port_match_mode configuration. |
| Makefile | Ensures alpine/curl image is available for in-cluster curl E2E helper. |
| .github/workflows/benchmark-test.yml | Increases ADC timeout in benchmark workflow via E2E_EXEC_ADC_TIMEOUT. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| BaseURL: u.String(), | ||
| Client: &http.Client{ | ||
| Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}, | ||
| Timeout: 3 * time.Second, |
There was a problem hiding this comment.
NewAPISIXClientForPort creates an httpexpect client without the redirect policy used by the other APISIX clients (they set CheckRedirect to http.ErrUseLastResponse). This means tests using this helper may silently follow redirects while other helpers don’t, leading to inconsistent assertions. Consider reusing the same http.Client settings (Transport + CheckRedirect) as NewAPISIXClient/NewAPISIXClientOnTCPPort for parity.
| Timeout: 3 * time.Second, | |
| Timeout: 3 * time.Second, | |
| CheckRedirect: func(req *http.Request, via []*http.Request) error { | |
| return http.ErrUseLastResponse | |
| }, |
internal/adc/translator/httproute.go
Outdated
|
|
||
| t.AttachBackendTrafficPolicyToUpstream(backend.BackendRef, tctx.BackendTrafficPolicies, upstream) | ||
| upstream.Nodes = upNodes | ||
| upstream.Scheme = appProtocolToUpstreamScheme(protocol) |
There was a problem hiding this comment.
In translateBackendsToUpstreams, the upstream scheme is being set unconditionally from the Service/AppProtocol (upstream.Scheme = appProtocolToUpstreamScheme(protocol)). This overwrites any scheme that may have been set by BackendTrafficPolicy (AttachBackendTrafficPolicyToUpstream sets upstream.Scheme), changing behavior compared to the previous logic which only filled scheme when it was empty. Preserve policy-configured scheme by only setting scheme when it’s not already set (or use cmp.Or/empty-check).
| upstream.Scheme = appProtocolToUpstreamScheme(protocol) | |
| if upstream.Scheme == "" { | |
| upstream.Scheme = appProtocolToUpstreamScheme(protocol) | |
| } |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
internal/controller/utils.go
Outdated
| // appendUniqueListeners appends listeners to the target slice, skipping any | ||
| // listener whose Name already exists in the target. Listener names are unique | ||
| // within a Gateway per the Gateway API specification. | ||
| func appendUniqueListeners(target []gatewayv1.Listener, source ...gatewayv1.Listener) []gatewayv1.Listener { | ||
| seen := make(map[gatewayv1.SectionName]struct{}, len(target)) | ||
| for _, l := range target { | ||
| seen[l.Name] = struct{}{} | ||
| } | ||
| for _, l := range source { | ||
| if _, exists := seen[l.Name]; !exists { | ||
| seen[l.Name] = struct{}{} | ||
| target = append(target, l) | ||
| } | ||
| } | ||
| return target |
There was a problem hiding this comment.
appendUniqueListeners de-dupes solely by listener Name. Listener names are only guaranteed unique within a single Gateway, but a Route can reference multiple Gateways via multiple parentRefs; two different Gateways can both have a listener named "http" on different ports. In that case the current de-dupe will drop one listener and its port, causing incorrect server_port var injection (missing ports) and potentially wrong routing. Consider either (a) not de-duping listeners at all when building tctx.Listeners (since downstream logic already de-dupes ports via a map), or (b) de-dupe on a composite key that can’t collide across Gateways (e.g., gateway namespace/name + listener name, or at least listener name+port+protocol).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 30 changed files in this pull request and generated 7 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Create new tunnel for custom port | ||
| serviceName := s.dataplaneService.Name | ||
| tunnel := k8s.NewTunnel(s.kubectlOptions, k8s.ResourceTypeService, serviceName, 0, port) |
There was a problem hiding this comment.
The docstring says it 'uses existing tunnels if available, otherwise creates a new one', but custom ports always create a brand-new tunnel and there’s no caching/reuse on subsequent calls. Either adjust the comment to reflect the actual behavior, or add a simple per-port tunnel cache on the Scaffold to avoid repeated port-forward setup in tests that call this helper multiple times.
|
Hi @johannes-engler-mw, please fix the failed CI |
|
Done, please retry CI. |
|
Fixed the failing e2e tests, it wasnt a regression but more an test config problem. Please rerun CI. |
Type of change:
What this PR does / why we need it:
This change implements support for port-based routing in HTTPRoute and GRPCRoute by leveraging the 'server_port' variable in APISIX. When a route targets a specific Gateway listener (via 'sectionName' or matching), the translator now adds a condition to ensure the route only matches traffic on that specific port.
Key changes:
It fixes #2639.
Pre-submission checklist: