help request: ApisixRoute CRDs not syncing to APISIX data plane; routes work only if created via UI/Admin API

[Sandeep-Raj-49]

Description

Details:

• ApisixRoute CRD resources are correctly created in Kubernetes and have status Accepted.

• ApisixPluginConfig CRDs (referenced by routes) exist and show as synced.

• The APISIX Ingress Controller logs show no errors but do not reflect sync of these CRDs into APISIX routes.

• Manual creation of equivalent routes in the UI/admin API immediately makes routes functional.

• The APISIX core version is 2.12.0 (installed via Helm chart).

• The APISIX ingress controller version is 2.0.0-rc4.

CLUSTER: Amazon EKS.

• Ingress controller and APISIX are deployed in the same namespace ( apache-apisix )
• Tried adjusting resource sync intervals and namespaces without success.
• I have verified network access, RBAC, and DNS; the admin API key is correct.

Environment

• APISIX version (run apisix version): 3.14.0
• Operating system (run uname -a): Linux apisix-58ff4c9fb6-6m9km 6.12.40 change: added doc of how to load plugin. apisix#1 SMP PREEMPT_DYNAMIC Mon Sep  8 23:26:50 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.27.1.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info)::)
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Baoyuantop]

First, we should avoid using the admin API or UI simultaneously with ingress, as this may cause synchronization problems. You can refer to https://apisix.apache.org/docs/ingress-controller/upgrade-guide/#controller-only-configuration-source

Could you please provide the ApisixRoute configuration file and related configuration files? We can try to reproduce this error.

[bzp2010]

APISIX Ingress Controller (AIC) 2.x do not support APISIX versions below 3.2 and recommend using at least APISIX 3.7 or higher (for existing APISIX users). For new users, we always recommend starting with the latest version available at the time of installation.

We will not provide support for APISIX versions released several years ago on AIC 2.x, encompassing not only AIC 2.0 but also future AIC versions, as this is unsustainable for us.
Furthermore, we will not deliver any new features or fixes to AIC 1.x and APISIX 2.x.

If you insist on using APISIX 2.x, please utilise the legacy AIC 1.x branch and resolve any issues encountered yourself. We strongly recommend upgrading to the latest version within APISIX 3.x to benefit from the diverse new features and optimisations we provide.

[Sandeep-Raj-49]

To clarify my environment and versions (which I had misstated previously):

APISIX Core Version:

• App Version: 3.14.1
• Helm Chart Version: 2.12.2

APISIX Ingress Controller Version:

• App Version: 2.0.0-rc4 (release candidate)
• Helm Chart Version: 1.0.5

Given that my APISIX core is a recent 3.x stable version (3.14.1), and the ingress controller is the upcoming 2.x series RC, I understand that this pairing is within the recommended support range. However, I am encountering synchronization issues where CRD-based routes are not synced into APISIX unless created manually via the dashboard or Admin API.

[bzp2010]

OK, please provide your Ingress Controller's ConfigMap here.

[Sandeep-Raj-49]

apiVersion: v1
data:
  config.yaml: |
    log_level: debug
    controller_name: apisix.apache.org/apisix-ingress-controller
    leader_election_id: apisix-ingress-controller-leader
    leader_election:
      leaseDuration: 15s
      renewDeadline: 10s
      retryPeriod: 2s
      disable: false
    metrics_addr: :8080
    enable_http2: false
    probe_addr: :8081
    secure_metrics: false
    exec_adc_timeout: 15s
    provider:
      type: apisix
      sync_period: 1m
      init_sync_delay: 20m
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: apisix
    meta.helm.sh/release-namespace: apache-apisix
  creationTimestamp: "2025-11-19T08:51:45Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: apisix-ingress-config
  namespace: apache-apisix

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: apisix-admin-healthz
  namespace: apache-apisix
spec:
  ingressClassName: apisix
  http:
    - name: healthz
      match:
        paths:
          - /healthz
      backends:
        - serviceName: apisix-admin
          servicePort: 9180
      plugins:
        - name: proxy-rewrite
          enable: true
          config:
            regex_uri:
              - "/healthz"
              - "/v1/healthcheck"

kubectl logs --tail=20 apisix-ingress-controller-6f4856784f-z5wcq -n apache-apisix
Defaulted container "manager" out of: manager, adc-server
2025-11-19T09:06:47Z    debug   client/client.go:168    syncing all resources
2025-11-19T09:06:47Z    warn    client/client.go:173    no GatewayProxy configs provided
2025-11-19T09:06:47Z    debug   apisix/provider.go:273  handled ADC execution errors    {"status_record": null, "status_update": {}}
2025-11-19T09:06:51Z    debug   controller/apisixroute_controller.go:356        try to get apisixupstream with the same name as the backend service {"Service": "apache-apisix/apisix-admin"}
2025-11-19T09:06:51Z    debug   controller/apisixroute_controller.go:358        no ApisixUpstream with the same name as the backend service found   {"Service": "apache-apisix/apisix-admin", "error": "ApisixUpstream.apisix.apache.org \"apisix-admin\" not found"}
2025-11-19T09:06:51Z    debug   apisix/provider.go:85   updating object {"object": {"kind":"ApisixRoute","apiVersion":"apisix.apache.org/v2","metadata":{"name":"apisix-admin-healthz","namespace":"apache-apisix","uid":"6682e793-57ac-4591-ba3b-2150b1f4c2ca","resourceVersion":"908400596","generation":4,"creationTimestamp":"2025-11-19T08:55:46Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apisix.apache.org/v2\",\"kind\":\"ApisixRoute\",\"metadata\":{\"annotations\":{},\"name\":\"apisix-admin-healthz\",\"namespace\":\"apache-apisix\"},\"spec\":{\"http\":[{\"backends\":[{\"serviceName\":\"apisix-admin\",\"servicePort\":9180}],\"match\":{\"paths\":[\"/healthz\"]},\"name\":\"healthz\",\"plugins\":[{\"config\":{\"regex_uri\":[\"/healthz\",\"/v1/healthcheck\"]},\"enable\":true,\"name\":\"proxy-rewrite\"}]}],\"ingressClassName\":\"apisix\"}}\n"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2025-11-19T09:02:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:conditions":{}}},"subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2025-11-19T09:06:51Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:http":{},"f:ingressClassName":{}}}}]},"spec":{"ingressClassName":"apisix","http":[{"name":"healthz","match":{"paths":["/healthz"]},"backends":[{"serviceName":"apisix-admin","servicePort":9180,"weight":null}],"websocket":false,"plugins":[{"name":"proxy-rewrite","enable":true,"config":{"regex_uri":["/healthz","/v1/healthcheck"]},"secretRef":""}],"authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{},"ldapAuth":{}}}]},"status":{"conditions":[{"type":"Accepted","status":"True","observedGeneration":3,"lastTransitionTime":"2025-11-19T09:02:12Z","reason":"Accepted","message":""}]}}}
2025-11-19T09:06:51.424Z        DEBUG   status.updater  status/updater.go:143   received a status update        {"namespace": "apache-apisix", "name": "apisix-admin-healthz", "kind": "ApisixRoute"}
2025-11-19T09:06:51.425Z        INFO    status.updater  status/updater.go:122   updating status {"name": "apisix-admin-healthz", "namespace": "apache-apisix", "kind": "ApisixRoute"}

[bzp2010]

What information does the status in ApisixRoute now record? Your log shows that it has already been accepted.

[Sandeep-Raj-49]

It shows as "True" but the route doesn't work. If I create the same route via admin API / UI, the route wont be visibile when I do kubectl get apisixroute but the route works and the service is accessible.

[bzp2010]

It shows as "True" but the route doesn't work.

2025-11-19T09:06:47Z    warn    client/client.go:173    no GatewayProxy configs provided

This indicates that no Gateway Proxy Custom Resource has been configured. Please verify this.

This requires using the Helm installation with the --set ingress-controller.gatewayProxy.createDefault=true flag to create the default GatewayProxy, or creating it manually.

Subsequently, your IngressClass must reference this GatewayProxy, which is a specific requirement differing from 1.x.

Please verify that your environment has been correctly configured with the above.

Use the helm template command to check what the K8s manifest should look like. ref: https://apisix.apache.org/docs/ingress-controller/install/

If I create the same route via admin API / UI, the route wont be visibile when I do kubectl get apisixroute

There has never been a reverse sync feature, and we shall never implement such functionality.

Reverse sync: Allows an externally created resource to be inspected by AIC from APISIX Admin API for creation as a Kubernetes Cluster Resource (CR).

I seriously advise against this practice. Not only does it undermine the constraints of declarative configuration, but it also places your business at risk. Should logical conflicts arise between resources configured via AIC and those configured through the Admin API or Dashboard, the gateway's forwarding behaviour becomes unpredictable.