Docker v29: DOCKER-FORWARD chain orphaned after update — container outbound traffic blocked when FORWARD policy is DROP

[ccarpinteri]

Summary

After updating Docker to v29 using synology-docker, the DOCKER-FORWARD chain exists but has 0 references from the built-in FORWARD chain. On Synology NAS boxes where synoiptables sets FORWARD -P DROP, this completely blocks container outbound traffic. On boxes where it leaves FORWARD -P ACCEPT, everything appears to work, but the chain is still orphaned.

Environment

• Synology DSM: 7.3.2
• Kernel: Linux 4.4.302+
• Docker version: 29.4.1 (build 6c91b92, installed via synology-docker)
• Reproduced on: Two independent Synology NAS boxes (different hardware, different network, same DSM + Docker version)

Observed iptables state after update

Chain FORWARD (policy DROP)
(empty — no rules)

Chain DOCKER-FORWARD (0 references)
DOCKER-CT  all  --  0.0.0.0/0   0.0.0.0/0
DOCKER-INTERNAL  all  --  0.0.0.0/0   0.0.0.0/0
DOCKER-BRIDGE  all  --  0.0.0.0/0   0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0   0.0.0.0/0

DOCKER-FORWARD is fully populated but unreachable — no jump rule exists in FORWARD to reach it.

Root cause

Docker v25+ changed its iptables architecture. Previously, Docker inserted rules directly into the built-in FORWARD chain on every container start (self-healing if wiped). In v25+, Docker creates a dedicated DOCKER-FORWARD chain and adds the jump rule FORWARD -j DOCKER-FORWARD once at daemon startup only. If Synology's synoiptables runs after the daemon starts and reinitialises the FORWARD chain without preserving Docker's jump rule, the rule is permanently gone until the daemon restarts.

Impact

• Boxes with FORWARD -P DROP: all container outbound/inter-container traffic blocked immediately after update
• Boxes with FORWARD -P ACCEPT: silently orphaned chain — traffic works via the default ACCEPT policy, but Docker's intended iptables filtering (DOCKER-CT, DOCKER-INTERNAL, etc.) is bypassed entirely

Workaround

Idempotent cron entry added to /etc/crontab:

* * * * *  root  /path/to/fix-iptables-forward.sh

Where fix-iptables-forward.sh contains:

#!/bin/sh
/sbin/iptables -C FORWARD -j DOCKER-FORWARD 2>/dev/null || /sbin/iptables -I FORWARD 1 -j DOCKER-FORWARD

This re-adds the jump rule within 60 seconds if it is ever wiped.

Suggestion

It may be worth adding this check to fix_ipforward.sh or switch_forward.sh, or as a new post-install step — particularly since this appears to be a systematic issue with Docker v29 on any Synology box where synoiptables manages the FORWARD chain.

[bslatyer]

Hey @ccarpinteri,

Noting the above, you've reproduced this issue on two different units (albeit with the same DSM and Docker versions).

Can I ask what the hardware of each of these was, and if the other box was running a kernel older or newer than 4.4.302?

I'm asking because I can't reproduce this on a 5.x unit.

Cheers.

[telnetdoogie]

switch_forward.sh exists for switching between the default FORWARD rule that exists in start-stop-status, however it's older, and was only ever experimental; it was designed to be way to toggle the iptables modifications that happen inside the synology start-stop-status script between the default synology behavior and an entry / entries more fitting for an updated docker.

It might be worth evaluating whether bringing that script up to date with modern docker iptables rules - and running it manually - works.

The forward rules have always mysteriously failed on some machines and worked on others. I was never able to find common themes where they worked and did not. When they don't work (or are overridden by DSM or another service somewhere) there is no container communication. Some users resort to adding startup scripts that sleep until all services have started, and then updating iptables with the correct forward rules, or using a cron job like the above.

If indeed the above steps work, it might indicate that this toggle might work since start-stop-status is always executed when ContainerManager starts, however previous experience often shows this setting is overridden by something else in the startup sequence, perhaps.

@bslatyer it's possible that line 73 of syno_docker_update.sh should also be modified to match the new behavior... of course, this would break updates to older versions also.

Others have stated that the issue of the forward override exists if the synology firewall is not enabled (even if running with empty rules), so changes to the update script may still be nullified.

[bslatyer]

@telnetdoogie, copy that.

I'm happy to fire up an older 4.x unit and see if we can make some changes to "fix" the problem.

[ccarpinteri]

Hi @bslatyer, hardware is DS718+. I too weren't able to figure out why they were disappearing after adding, hence the inelegant cron which is certainly a hack. Both servers running 4.4.302+ kernel.

[telnetdoogie]

@ccarpinteri I was running a 218+ which on the software / DSM side is very similar to the 718...
Of all the issues that have happened with this docker update, this one (disappearing / changing forward rule) is the most elusive... I traced back every service loaded and added logging... I just never figured it out.
I added add_logging_to_start_script.sh to trace the insertion of the forward rule as well

[bslatyer]

Hey folks,

Sorry about the long wait.

I wasn't entirely able to reproduce this on several 4.x and 5.x boxes.

All the firewalls are off as they aren't exposed to the WAN.

@ccarpinteri - I couldn't see 0.0.0.0/0, all the boxes I've tested on show DST/SRC anywhere, so would it be safe to assume you're running IPv4 only?

Alongside that, are these containers all on the bridge network rather than on the host?

@telnetdoogie - I'm happy to change L73 in a separate branch for testing.

synology-docker/syno_docker_update.sh

Line 73 in b9e51c6

readonly SYNO_DOCKER_SCRIPT_FORWARDING=" # Added by docker update\n iptables -P FORWARD ACCEPT"

That way, whatever changes need to be made, @ccarpinteri can follow along and provide feedback.

However, when you are referring to updates to older versions also, - are you referring to this script or something else (like Docker or DSM)?

In any case, let me know ;)

[telnetdoogie]

I was referring to older versions of docker... so if the FORWARD entry changes to something like

iptables -C FORWARD -j DOCKER-FORWARD then if another version of Docker's forward chain is just called DOCKER then it wouldn't work... but I just checked and it looks like all of them from 25 onward use DOCKER-FORWARD, so I think it's reasonable.

And... based on what this user is reporting (which is the exact same that other users having this issue have reported) whatever iptables rules are put into this start script get overridden anyways for those few users. It'd probably be smart to bring the forward rule up to date with docker's latest forward chain (and fix fix_ipforward.sh to match this user's forward rule fix) - however it likely won't "fix" it for users where this happens. It would, however, respect the intended iptables of docker..

[bslatyer]

@telnetdoogie sounds like a plan ;)

[bslatyer]

Folks, just a quick update. I should hopefully have something in the fixes-66 branch later today/evening.

[bslatyer]

@ccarpinteri - do you want to give the changes in fixes-66 branch a go?

If it works, great!

If not, back to the drawing board.

Ta.

[ccarpinteri]

Thanks @bslatyer, I'm hoping to give this a go in the next 3-4 days.