nebula-sync fails to start - error 401

[hspindel]

What happened?

nebula-sync is deployed using portainer under debian 13.

Starting the stack results in:
2025-08-18T22:50:32-07:00 FTL Sync failed error="authenticate: http://pve-pihole6.sci1.com/api/auth: unexpected status code: 401"

I have verified that my passwords are correct. pihole is running and I can access its web page. I tried enclosing the pihole specs in quotes, but got a different error about path segment in URL cannot contain colon. Where do I look next?

Expected Behaviour

nebula-sync should start up.

Nebula-sync Version

0.11.0

Runtime

docker

Operating System

debian 13 trixie

CPU Architecture

x86

Configuration

services:
  nebula-sync:
    image: ghcr.io/lovelaze/nebula-sync:latest
    container_name: nebula-sync
    environment:
    - PRIMARY=http://pve-pihole6.sci1.com|redacted
    - REPLICAS=http://pihole-synology.sci1.com|redacted
    - FULL_SYNC=true
    - RUN_GRAVITY=true
    - CRON=0 * * * *
    - TZ=America/Los_Angeles
    - CLIENT_SKIP_TLS_VERIFICATION=true

Relevant log output

Registries
Notifications

Community Edition
Containers>nebula-sync>Logs
Container logs
Log viewer settings
Auto-refresh logs
Wrap lines
Display timestamps
Fetch
Search
Lines
Actions

      

2025-08-18T23:00:29-07:00 INF Starting nebula-sync v0.11.0

2025-08-18T23:00:29-07:00 INF Running sync mode=full replicas=1

2025-08-18T23:00:29-07:00 INF Authenticating clients...

2025-08-18T23:00:30-07:00 INF Invalidating sessions...

2025-08-18T23:00:30-07:00 WRN Failed to invalidate session for target: http://pve-pihole6.sci1.com

2025-08-18T23:00:32-07:00 FTL Sync failed error="authenticate: http://pve-pihole6.sci1.com/api/auth: unexpected status code: 401"

2025-08-18T23:00:32-07:00 WRN Failed to invalidate session for target: http://pihole-synology.sci1.com

Pi-hole Version

6.1.4

[hardwareadictos]

Same here

[JasonPyMe]

Add me to that list. I am getting the same output. I turned on debug logs as well.

2025-08-31T20:44:25-07:00 INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:35 > Running sync mode=selective replicas=1
2025-08-31T20:44:25-07:00 INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:52 > Authenticating clients...
2025-08-31T20:44:25-07:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:66 > PostAuth client=http://172.19.88.3
2025-08-31T20:44:25-07:00 INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:69 > Invalidating sessions...
2025-08-31T20:44:25-07:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://172.19.88.3
2025-08-31T20:44:25-07:00 WRN github.com/lovelaze/nebula-sync/internal/sync/sync.go:71 > Failed to invalidate session for target: http://172.19.88.3
2025-08-31T20:44:25-07:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://172.19.88.2
2025-08-31T20:44:25-07:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(1): http://172.19.88.2: invalid sid found
2025-08-31T20:44:45-07:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://172.19.88.2
2025-08-31T20:44:45-07:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(2): http://172.19.88.2: invalid sid found
2025-08-31T20:45:05-07:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://172.19.88.2
2025-08-31T20:45:05-07:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(3): http://172.19.88.2: invalid sid found
2025-08-31T20:45:05-07:00 WRN github.com/lovelaze/nebula-sync/internal/sync/sync.go:78 > Failed to invalidate session for target: http://172.19.88.2
2025-08-31T20:45:05-07:00 FTL github.com/lovelaze/nebula-sync/cmd/run.go:25 > Sync failed error="authenticate: http://172.19.88.3/api/auth: unexpected status code: 401"
2025-08-31T20:45:06-07:00 INF github.com/lovelaze/nebula-sync/internal/service/service.go:66 > Starting nebula-sync v0.11.0
2025-08-31T20:45:06-07:00 DBG github.com/lovelaze/nebula-sync/internal/api/server.go:38 > Starting http server

[marcisaninja]

same

[JasonPyMe]

I was able to get it working by putting everything into the compose file. I am using Portainer stacks and I typically use the environment variable options to make changes/update easy.

Here is my exact copy/paste from Portainer. I have just changed the IPs and passwords.

services:
nebula-sync:
image: ghcr.io/lovelaze/nebula-sync:latest
container_name: nebula-sync
environment:
- NS_DEBUG=true
- PRIMARY=https://1.1.1.1|pass1234
- REPLICAS=https://2.2.2.2|pass1234
- FULL_SYNC=false
- RUN_GRAVITY=false
- CRON=*/15 * * * *
- CLIENT_SKIP_TLS_VERIFICATION=true
- TZ=America/NYC
- SYNC_CONFIG_DNS=true
- SYNC_CONFIG_DHCP=false
- SYNC_CONFIG_NTP=false
- SYNC_CONFIG_RESOLVER=false
- SYNC_CONFIG_DATABASE=false
- SYNC_CONFIG_MISC=false
- SYNC_CONFIG_DEBUG=false
- SYNC_GRAVITY_DHCP_LEASES=false
- SYNC_GRAVITY_GROUP=false
- SYNC_GRAVITY_AD_LIST=true
- SYNC_GRAVITY_AD_LIST_BY_GROUP=true
- SYNC_GRAVITY_DOMAIN_LIST=true
- SYNC_GRAVITY_DOMAIN_LIST_BY_GROUP=true
- SYNC_GRAVITY_CLIENT=false
- SYNC_GRAVITY_CLIENT_BY_GROUP=false

[rodrigotrevelin]

The same is valid for the new version 0.11.1

nebula-sync | 2025-09-12T12:44:05-03:00 INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:35 > Running sync mode=selective replicas=1 nebula-sync | 2025-09-12T12:44:05-03:00 INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:52 > Authenticating clients... nebula-sync | 2025-09-12T12:44:05-03:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:66 > PostAuth client=http://192.168.2.10 nebula-sync | 2025-09-12T12:44:06-03:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:66 > PostAuth client=http://192.168.2.20 nebula-sync | 2025-09-12T12:44:07-03:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(1): http://192.168.2.20/api/auth: unexpected status code: 401 nebula-sync | 2025-09-12T12:44:08-03:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:66 > PostAuth client=http://192.168.2.20 nebula-sync | 2025-09-12T12:44:09-03:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(2): http://192.168.2.20/api/auth: unexpected status code: 401 nebula-sync | 2025-09-12T12:44:10-03:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:66 > PostAuth client=http://192.168.2.20 nebula-sync | 2025-09-12T12:44:12-03:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(3): http://192.168.2.20/api/auth: unexpected status code: 401 nebula-sync | 2025-09-12T12:44:12-03:00 INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:69 > Invalidating sessions... nebula-sync | 2025-09-12T12:44:12-03:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://192.168.2.10 nebula-sync | 2025-09-12T12:44:12-03:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://192.168.2.20 nebula-sync | 2025-09-12T12:44:12-03:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(1): http://192.168.2.20: invalid sid found nebula-sync | 2025-09-12T12:44:13-03:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://192.168.2.20 nebula-sync | 2025-09-12T12:44:13-03:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(2): http://192.168.2.20: invalid sid found nebula-sync | 2025-09-12T12:44:14-03:00 DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://192.168.2.20 nebula-sync | 2025-09-12T12:44:14-03:00 DBG github.com/lovelaze/nebula-sync/internal/sync/retry/retry.go:37 > Retrying(3): http://192.168.2.20: invalid sid found nebula-sync | 2025-09-12T12:44:14-03:00 WRN github.com/lovelaze/nebula-sync/internal/sync/sync.go:78 > Failed to invalidate session for target: http://192.168.2.20 nebula-sync | 2025-09-12T12:44:14-03:00 FTL github.com/lovelaze/nebula-sync/cmd/run.go:25 > Sync failed error="authenticate: http://192.168.2.20/api/auth: unexpected status code: 401" nebula-sync exited with code 1

Testing the endpoint on HTTPie is OK

{ "session": { "valid": true, "totp": false, "sid": "F3VAwGqFEPzHAKQvCnDm6Q=", "csrf": "A/YzKi7aO+D4TdKT2EdgGA=", "validity": 1800, "message": "password correct" }, "took": 1.3202383518218994 }

Even with full config on compose wont work

nebula-sync: image: ghcr.io/lovelaze/nebula-sync:latest container_name: nebula-sync depends_on: - pihole environment: TZ: ${TZ} PRIMARY: ${NEBULA_PRIMARY_SYNC} REPLICAS: ${NEBULA_REPLICA_SYNC} FULL_SYNC: false CRON: ${NEBULA_CRON} CLIENT_SKIP_TLS_VERIFICATION: true CLIENT_TIMEOUT_SECONDS: 60 NS_DEBUG: true SYNC_CONFIG_DNS: true SYNC_CONFIG_DHCP: false SYNC_CONFIG_NTP: false SYNC_CONFIG_RESOLVER: false SYNC_CONFIG_DATABASE: false SYNC_CONFIG_MISC: false SYNC_CONFIG_DEBUG: false SYNC_GRAVITY_DHCP_LEASES: false SYNC_GRAVITY_GROUP: false SYNC_GRAVITY_AD_LIST: true SYNC_GRAVITY_AD_LIST_BY_GROUP: true SYNC_GRAVITY_DOMAIN_LIST: true SYNC_GRAVITY_DOMAIN_LIST_BY_GROUP: true SYNC_GRAVITY_CLIENT: false SYNC_GRAVITY_CLIENT_BY_GROUP: false

[lovelaze]

I'm pretty sure this is caused by the envs not being set correctly; either because of the variables not being escape in the configurations or because they contain strange characters.

You can see what nebula-sync actually uses as password by setting NS_DEBUG=true and looking at the logs, then comparing that to your expected value.

[rodrigotrevelin]

I'm pretty sure this is caused by the envs not being set correctly; either because of the variables not being escape in the configurations or because they contain strange characters.

You can see what nebula-sync actually uses as password by setting NS_DEBUG=true and looking at the logs, then comparing that to your expected value.

Yes, for some reason the .env for REPLICAS was not set on execution, but exists on .env file, both instances have the same password, so the strange characters theory is not the problem, after setting direct on compose file REPLICAS: http://192.168.0.20|PassWord start working.

After renaming env var, checking config with docker compose config, the env var won't work, only with the value direct on compose file

`
nebula-sync:
image: ghcr.io/lovelaze/nebula-sync:latest
container_name: nebula-sync
environment:
TZ: ${TZ}
PRIMARY: ${NEBULA_PRIMARY_SYNC}
REPLICAS: ${NEBULA_REPLICA_SYNC} <--- don't work
REPLICAS: http://192.168.0.20|PssWoRd <--- Only works with this line
FULL_SYNC: ${NEBULA_FULL_SYNC}
CRON: ${NEBULA_CRON}
CLIENT_SKIP_TLS_VERIFICATION: true
CLIENT_TIMEOUT_SECONDS: 60
NS_DEBUG: true

`

[derritter88]

For me it is also not working - even without not using environemnt variables.

[derritter88]

After some debugging I saw that my 2nd Pi-hole was having a faulty SQLite db. Deleting and repairing it solved the problem for me.

[ehjr5u]

Having the same issue as well. Someone did mention this problem from #104