Issue handling loopback traffic for LLM proxy

[nimsara66]

Overview

An LLM proxy consumes an admin-created LLM provider through Envoy using loopback. When an LLM provider is deployed without a VHost configuration, the LLM proxy can access it via the internal loopback IP (127.0.0.1).

However, when the provider is deployed with a VHost configuration, one of the following is required:

• The proxy must invoke the provider with the correct Host header, or
• The router container must have appropriate hostname mappings in /etc/hosts.

With the current implementation, all listeners are configured with auto_host_rewrite enabled [1][2]. As a result, Envoy replaces the incoming Host header with the hostname of the upstream selected by the cluster manager. This behavior prevents correct routing to VHost-based providers.

To address this, a configuration option in api.yaml is required to disable host rewrite either at the API level or at the operation level.

API Level

apiVersion: gateway.api-platform.wso2.com/v1alpha1
kind: RestApi
metadata:
  name: weather-api-v1.0
spec:
  displayName: Weather-API
  version: v1.0
  context: /weather/$version
  disableHostRewrite: true
  upstream:
    main:
      url: http://sample-backend:5000/api/v2
  ...

Operational Level

apiVersion: gateway.api-platform.wso2.com/v1alpha1
kind: RestApi
metadata:
  name: weather-api-v1.0
spec:
  displayName: Weather-API
  version: v1.0
  context: /weather/$version
  upstream:
    main:
      url: http://sample-backend:5000/api/v2
  operations:
    - method: GET
      path: /{country_code}/{city}
      disableHostRewrite: true
    - method: GET
      path: /alerts/active

[1] -

api-platform/gateway/gateway-controller/pkg/xds/translator.go

Line 836 in bb8a758

HostRewriteSpecifier: &route.RouteAction_AutoHostRewrite{

[2] - https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto

[nimsara66]

Proposal 2

Introduce a hostRewrite property at the upstream level to control how the Host header is handled when routing requests to backend services.
By default, the hostRewrite property is set to auto, which rewrites the Host header to match the upstream cluster host.

This scheme is extensible when we introduce operational level endpoints in the future.

apiVersion: gateway.api-platform.wso2.com/v1alpha1
kind: RestApi
metadata:
  name: weather-api-v1.0
spec:
  displayName: Weather-API
  version: v1.0
  context: /weather/$version
  upstream:
    main:
      url: http://sample-backend:5000/api/v2
      hostRewrite: manual
  operations:
    - method: GET
      path: /{country_code}/{city}
    - method: GET
      path: /alerts/active