investigation of firmware reproducibility

[maurerle]

Most parts of the firmware should be reproducible.

As gluon uses the autoupdater for updates, it is of interest to be able to create images with the same sha256 checksum from a given release tag for downstream updates.

This issue is to document what already works, and what does not.

One can investigate the following three types here:

1. building on an existing build chain on the same host
2. building on a fresh build chain (make clean) on the same host
3. building on a fresh build chain on a different host

How-To

Openwrt uses the openwrt/key-build folder for the opkg signing material. this must be persisted between builds of course.
A fixed GLUON_VERSION version must be used, so that the build does not differ when running on a different date.

For Openwrt, some strings must be fixed throughout building of the buildchain as well:

Using a fixed VERSION is possible using a string in a file called version in the openwrt root.
Using a fixed SOURCE_DATE_EPOCH is possible using a string in a file called version.date in the openwrt root.

cp ~/.key-build/* .
echo r30000 > version
echo 1764720000 > version.date

1. Existing build chain

Except for the following devices, the sha256sum is the same:

mikrotik-routerboard-map-lite
mikrotik-routerboard-wapr-2nd
mikrotik-routerboard-951ui-2nd-hap
mikrotik-routerboard-wap-ac-t2
mikrotik-hap-ac2
mikrotik-sxtsq-5-ac-rbsxtsqg-5acd
mikrotik-discg-5acd
gl.inet-gl-mt2500 (filogic)
gl.inet-gl-mt3000 (filogic)
genexis-pulse-ex400 (mt7621)
wavlink-ws-wn572hp3-4g (mt7621)

Quite interesting.

diff <(xxd foo1.bin) <(xxd foo2.bin)

For the EX400, only a few bytes in the kernel (0.ubi) were changed:

diff of xxd output for genexis-pulse-ex400

129c129
< 00000800: 3118 1006 92c0 2e24 8703 0000 0000 0000  1......$........
---
> 00000800: 3118 1006 763c e2a9 8703 0000 0000 0000  1...v<..........
135,136c135,136
< 00000860: 0000 0000 0000 0000 00ca 9a3b 4e4a aff6  ...........;NJ..
< 00000870: 3cbb 402b 9f79 c0e8 9f55 e16c 0000 0000  <.@+.y...U.l....
---
> 00000860: 0000 0000 0000 0000 00ca 9a3b a443 b832  ...........;.C.2
> 00000870: 61ac 404b ba1c 1290 5638 44c0 0000 0000  a.@K....V8D.....
267386c267386
< 00414790: 641f 12c7 a0ff ffff 3118 1006 a87b bcf1  d.......1....{..
---
> 00414790: 641f 12c7 a0ff ffff 3118 1006 e6e5 aab5  d.......1.......
267390,267391c267390,267391
< 004147d0: 083f 5869 0000 0000 083f 5869 0000 0000  .?Xi.....?Xi....
< 004147e0: 083f 5869 0000 0000 0000 0000 0000 0000  .?Xi............
---
> 004147d0: e8ed 5669 0000 0000 e8ed 5669 0000 0000  ..Vi......Vi....
> 004147e0: e8ed 5669 0000 0000 0000 0000 0000 0000  ..Vi............
267651c267651
< 00415820: 0fff ffff ffff ffff 3118 1006 5a98 7e9d  ........1...Z.~.
---
> 00415820: 0fff ffff ffff ffff 3118 1006 2c36 0d7b  ........1...,6.{
267655,267656c267655,267656
< 00415860: e8ed 5669 0000 0000 083f 5869 0000 0000  ..Vi.....?Xi....
< 00415870: eb3e 5869 0000 0000 0000 0000 0000 0000  .>Xi............
---
> 00415860: b33d 5469 0000 0000 e8ed 5669 0000 0000  .=Ti......Vi....
> 00415870: cced 5669 0000 0000 0000 0000 0000 0000  ..Vi............
267665c267665
< 00415900: 6474 6200 ffff ffff 3118 1006 23ce 8402  dtb.....1...#...
---
> 00415900: 6474 6200 ffff ffff 3118 1006 6d50 9246  dtb.....1...mP.F
267669,267670c267669,267670
< 00415940: 083f 5869 0000 0000 083f 5869 0000 0000  .?Xi.....?Xi....
< 00415950: 083f 5869 0000 0000 0000 0000 0000 0000  .?Xi............
---
> 00415940: e8ed 5669 0000 0000 e8ed 5669 0000 0000  ..Vi......Vi....
> 00415950: e8ed 5669 0000 0000 0000 0000 0000 0000  ..Vi............
267679c267679
< 004159e0: 626f 6f74 00ff ffff 3118 1006 1b6b a231  boot....1....k.1
---
> 004159e0: 626f 6f74 00ff ffff 3118 1006 55f5 b475  boot....1...U..u
267683,267684c267683,267684
< 00415a20: 083f 5869 0000 0000 083f 5869 0000 0000  .?Xi.....?Xi....
< 00415a30: 083f 5869 0000 0000 0000 0000 0000 0000  .?Xi............
---
> 00415a20: e8ed 5669 0000 0000 e8ed 5669 0000 0000  ..Vi......Vi....
> 00415a30: e8ed 5669 0000 0000 0000 0000 0000 0000  ..Vi............
1168036c1168036
< 011d2a30: 9a0a a6a1 0100 0000 0000 023c 0000 0164  ...........<...d
---
> 011d2a30: 7eec da83 0100 0000 0000 023c 0000 0164  ~..........<...d
1168063,1168069c1168063,1168069
< 011d2be0: 7457 6f6c 4758 4754 6c73 7657 4171 776f  tWolGXGTlsvWAqwo
< 011d2bf0: 6573 4f66 7856 3333 4a6b 6e68 3659 686d  esOfxV33Jknh6Yhm
< 011d2c00: 4431 656f 486e 6e48 4c67 4b4a 5956 3170  D1eoHnnHLgKJYV1p
< 011d2c10: 4e63 6879 4763 4f54 636a 4c4b 7867 506a  NchyGcOTcjLKxgPj
< 011d2c20: 366a 3235 557a 6c37 5474 5a47 6249 5179  6j25Uzl7TtZGbIQy
< 011d2c30: 557a 624b 6862 2f45 752f 6667 773d 0a00  UzbKhb/Eu/fgw=..
< 011d2c40: 4657 7830 23fe f6de 0000 0000 0000 0214  FWx0#...........
---
> 011d2be0: 7457 6f6c 4758 4754 624d 7275 776d 6546  tWolGXGTbMruwmeF
> 011d2bf0: 594d 7867 5477 6646 5959 4e69 375a 4f78  YMxgTwfFYYNi7ZOx
> 011d2c00: 4a5a 5642 4434 3238 6b73 686c 4647 5368  JZVBD428kshlFGSh
> 011d2c10: 6670 5534 7754 6159 6e4a 624b 304e 656f  fpU4wTaYnJbK0Neo
> 011d2c20: 724e 4862 4769 4b45 5577 5a41 6c79 6c44  rNHbGiKEUwZAlylD
> 011d2c30: 5749 6778 504f 2f43 6670 5851 633d 0a00  WIgxPO/CfpXQc=..
> 011d2c40: 4657 7830 1fe7 7d22 0000 0000 0000 0214  FWx0..}"........

2. fresh build-chain

The sha256sum between two fresh builds is similar to 1. if version and version.date exists in the openwrt build folder.

analysis of firmware differences

Extracting the kernel and root from a sysupgrade bin and further extracting the squashfs using binwalk -e gives more information on the differences.
Diffing the bytes of _root-latest.extracted/squashfs-root/bin/ash with _root-deployed.extracted/squashfs-root/bin/ash shows that two strings differ in the ash binary:

BusyBox v1.36.1 (2025-12-02 00:25:42 UTC) includes the build date

and GCC: (OpenWrt GCC 13.3.0 r29091+15-b023a06cfb includes the commit id, of the applied gluon patches to the OpenWrt build tree. As this commit id changes when reapplying, this is probably hard to fix.
Luckily, one can adjust this with a version file in the openwrt build folder:

https://github.com/openwrt/openwrt/blob/93e9e67ee9647355c21209e8aa217d941fd812b0/scripts/getver.sh#L61

it seems that the used libgcc vs libgcc1 depends is set differently when building all targets, vs building a single target..?

diff --color -r _root_new.extracted/squashfs-root/usr/lib/opkg/info/libc.control _root_reallynew.extracted/squashfs-root/usr/lib/opkg/info/libc.control
3c3
< Depends: libgcc
---
> Depends: libgcc1
diff --color -r _root_new.extracted/squashfs-root/usr/lib/opkg/status _root_reallynew.extracted/squashfs-root/usr/lib/opkg/status
53c53
< Depends: libgcc
---
> Depends: libgcc1

Example manifest between two builds:

stable-1.txt

stable-2.txt

diffoscope: https://try.diffoscope.org/hrpzhfvktcpa.html

ax3600

The xiaomi-ax3600 has a difference in qca-ssdk.ko which includes a timestamp of the build.
Upstream issue is created here: openwrt/qca-ssdk#5

diff <(xxd _root_new.extracted/squashfs-root/lib/modules/6.6.119/qca-ssdk.ko) <(xxd _root_old.extracted/squashfs-root/lib/modules/6.6.119/qca-ssdk.ko)
47663c47663
< 000ba2e0: 3034 2d30 343a 3335 3a34 3600 0000 0000  04-04:35:46.....
---
> 000ba2e0: 3033 2d31 393a 3339 3a33 3900 0000 0000  03-19:39:39.....

3. different host

TODO