refactor(ipip-0499): restructure Motivation section

lidel · lidel · commit 62d3caef0f8e · 2026-01-28T17:27:37.000+01:00
- move HAMT sharding history to Motivation, add HAMT switch comparison to profiles
- rename Divergence section to Divergences across ecosystem
- merge Observed differences into balanced-packed, promote subsections to h3
diff --git a/src/ipips/ipip-0499.md b/src/ipips/ipip-0499.md
@@ -79,7 +79,7 @@ The following [UnixFS](https://specs.ipfs.tech/unixfs/) parameters were identifi
    - `trickle`: builds a tree optimized for on-the-fly one-time streaming, where data can be consumed before the entire file is available. Useful for logs and other append-only data structures where random access is not important.
 1. UnixFS DAG width (max number of links per `File` node)
 1. [HAMTDirectory](https://specs.ipfs.tech/unixfs/#dag-pb-hamtdirectory) fanout: the branching factor at each level of the HAMT tree (e.g., 256 leaves).
-1. [HAMTDirectory threshold](https://specs.ipfs.tech/unixfs/#when-to-use-hamt-sharding): max `Directory` size before converting to `HAMTDirectory`, based on `PBNode.Links` count or estimated serialized [dag-pb](https://ipld.io/specs/codecs/dag-pb/spec/) size:
+1. [HAMTDirectory threshold](https://specs.ipfs.tech/unixfs/#when-to-use-hamt-sharding): max `Directory` size before converting to `HAMTDirectory`, based on `PBNode.Links` count or estimated serialized [dag-pb](https://ipld.io/specs/codecs/dag-pb/spec/) size. See [Historical inconsistency in HAMT sharding](#historical-inconsistency-in-hamt-sharding) below.
    - `links-count`: `PBNode.Links` length (child count). Simple but ignores varying entry sizes.
    - `links-bytes`: sum of `PBNode.Links[].Name` and `PBNode.Links[].Hash` byte lengths. Underestimates actual size by ignoring UnixFS Data, Tsize, and protobuf overhead.
    - `block-bytes`: full serialized dag-pb node size. Most accurate, accounts for varint `Tsize` and optional metadata such as `mode` or `mtime`.
@@ -92,7 +92,7 @@ The following [UnixFS](https://specs.ipfs.tech/unixfs/) parameters were identifi
 1. [Mode](https://specs.ipfs.tech/unixfs/#mode-field): optional POSIX file permissions.
 1. [Mtime](https://specs.ipfs.tech/unixfs/#mtime-field): optional modification timestamp.
 
-#### Balanced DAG layout variants
+### Balanced DAG layout variants
 
 The `balanced` DAG layout has implementation variants that affect CID determinism for large files. CID mismatches have been [observed](https://discuss.ipfs.tech/t/should-we-profile-cids/18507/41) and [investigated](https://discuss.ipfs.tech/t/should-we-profile-cids/18507/44) when comparing [kubo][] and [Singularity][singularity] outputs for files exceeding 1 GiB. This IPIP introduces the name `balanced-packed` to distinguish Singularity's variant from the original `balanced` layout.
 
@@ -115,16 +115,47 @@ Name introduced by this IPIP for [Singularity][singularity]'s variant. Groups pr
 - Optimized for batch processing of pre-chunked data in CAR files
 - Reference: [`singularity/pack/packutil/util.go`](https://github.com/data-preservation-programs/singularity/blob/v0.6.0-RC4/pack/packutil/util.go) `AssembleFileFromLinks()`
 
-##### Observed differences
+According to [Singularity issue #525](https://github.com/data-preservation-programs/singularity/issues/525), "in Singularity's DAG, the last leaf node is not at the same distance from the root as the others." This structural difference causes CID mismatches for files larger than `chunk_size * dag_width` (e.g., >1 GiB with 1 MiB chunks and 1024 links per node), even when all other parameters match.
 
-According to [Singularity issue #525](https://github.com/data-preservation-programs/singularity/issues/525):
-> "In Singularity's DAG, the last leaf node is not at the same distance from the root as the others."
+### Historical inconsistency in HAMT sharding
 
-This structural difference causes CID mismatches for files larger than `chunk_size * dag_width` (e.g., >1 GiB with 1 MiB chunks and 1024 links per node), even when all other parameters match.
+The IPFS ecosystem was never fully consistent in HAMT directory sharding behavior. This section documents the implementation history to explain why standardization through profiles is necessary.
 
-### Divergence in current implementations
+Timeline of Go implementation changes:
 
-We analyzed the default settings across the most popular UnixFS implementations in the ecosystem. The table below documents the divergence that prevents deterministic CID generation today:
+- 2017-03: [kubo#3042](https://github.com/ipfs/kubo/pull/3042) introduced HAMT sharding with a global `Experimental.ShardingEnabled` flag. When enabled, all directories were sharded regardless of size. This is why historical snapshots like `/ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze` (Wikipedia) have HAMTDirectory nodes even for parent directories with few entries.
+
+- 2021-05: [go-unixfs#91](https://github.com/ipfs/go-unixfs/pull/91) introduced `HAMTShardingSize` threshold for automatic sharding based on estimated directory size, using `links-bytes` estimation. This was part of the work tracked in [kubo#8106](https://github.com/ipfs/kubo/issues/8106).
+
+- 2021-11: [go-unixfs#94](https://github.com/ipfs/go-unixfs/pull/94) added size-based unsharding (switching from HAMT back to basic directory), completing bidirectional automatic sharding with `>=` comparison. The [go-unixfs](https://github.com/ipfs/go-unixfs) repository has since been archived; its code now lives in [boxo](https://github.com/ipfs/boxo).
+
+- 2021-12: [go-ipfs v0.11.0](https://github.com/ipfs/kubo/releases/tag/v0.11.0) (now Kubo) shipped with automatic HAMT autosharding, deprecating the global `Experimental.ShardingEnabled` flag.
+
+- 2023-03: boxo created via [Über Migration](https://github.com/ipfs/boxo/commit/038bdd29), inheriting the `>=` comparison behavior from go-unixfs.
+
+- 2026-01: [boxo#1088](https://github.com/ipfs/boxo/pull/1088) fixed threshold comparison from `>=` to `>`, aligning with JS implementation and documentation. Shipped in Kubo 0.40.
+
+Timeline of JavaScript implementation changes:
+
+- 2017-03: [js-ipfs-unixfs#14](https://github.com/ipfs/js-ipfs-unixfs/pull/14) added HAMT data types to UnixFS protobuf definitions.
+
+- 2018-12: [js-ipfs#1734](https://github.com/ipfs/js-ipfs/pull/1734) added HAMT sharding support to MFS, using entry count threshold (`shardSplitThreshold`, default 1000 entries).
+
+- 2019-01: [js-ipfs v0.34.0](https://github.com/ipfs/js-ipfs/releases/tag/v0.34.0) shipped with HAMT support in MFS. The threshold was based on entry count, not size, which differed from Go's size-based approach.
+
+- 2022-10: [Helia](https://github.com/ipfs/helia) created as the successor to js-ipfs.
+
+- 2023-02: [js-ipfs-unixfs#171](https://github.com/ipfs/js-ipfs-unixfs/pull/171) changed from entry count to DAGNode size threshold (`shardSplitThresholdBytes`, default 256 KiB), aligning with Go implementation. Uses `>` comparison. This was tracked in [js-ipfs-unixfs#149](https://github.com/ipfs/js-ipfs-unixfs/issues/149). The [js-ipfs-unixfs](https://github.com/ipfs/js-ipfs-unixfs) library remains active and is used by Helia.
+
+- 2023-05: [js-ipfs](https://github.com/ipfs/js-ipfs) archived; Helia became the recommended JS implementation.
+
+The JavaScript implementation in Helia uses `size > threshold` (strictly greater than) in [`is-over-shard-threshold.ts`](https://github.com/ipfs/helia/blob/main/packages/unixfs/src/commands/utils/is-over-shard-threshold.ts), consistent with Go after the 2026 fix.
+
+These inconsistencies between Go and JS implementations over the years, combined with differing threshold methods (entry count vs size) and comparison operators (`>=` vs `>`), meant cross-implementation CID determinism for large directories was never reliably achievable. The `unixfs-v1-2025` profile addresses this by standardizing on `block-bytes` estimation and explicit `>` comparison.
+
+### Divergences across ecosystem
+
+We analyzed the default settings across the most popular UnixFS implementations in the ecosystem. The table below documents the divergences that prevent deterministic CID generation today:
 
 | Parameter                     | [kubo][] (CIDv0)         | [helia][]            | [storacha][]       | [kubo][] (CIDv1)              | [singularity][]                     | [dasl][]     |
 | ----------------------------- | ------------------------ | -------------------- | ------------------ | ----------------------------- | ----------------------------------- | ------------ |
@@ -137,6 +168,7 @@ We analyzed the default settings across the most popular UnixFS implementations
 | DAG width (children per node) | 174                      | 1024                 | 1024               | 174                           | 1024                                | N/A          |
 | HAMTDirectory fanout          | 256 blocks               | 256 blocks           | 256 blocks         | 256 blocks                    | 256 blocks (boxo)                   | N/A          |
 | HAMTDirectory threshold       | 256KiB (links-bytes)     | 256KiB (links-bytes) | 1000 (links-count) | 256KiB (links-bytes)          | 256KiB (links-bytes) (boxo)         | N/A          |
+| HAMT switch comparison        | >=                       | >                    | >                  | >=                            | >= (boxo)                           | N/A          |
 | Leaves                        | dag-pb                   | raw                  | raw                | raw                           | raw                                 | N/A          |
 | Empty directories             | included                 | included             | excluded           | included                      | included                            | N/A          |
 | Hidden entities               | excluded (opt-in)        | excluded (opt-in)    | excluded (opt-in)  | excluded (opt-in)             | included (rclone)                   | N/A          |
@@ -174,6 +206,7 @@ Based on the research above, we define **`unixfs-v1-2025`** as an opinionated pr
 | DAG width (children per node) | 1024                 |
 | HAMTDirectory fanout          | 256 blocks           |
 | HAMTDirectory threshold       | 256KiB (block-bytes) |
+| HAMT switch comparison        | >                    |
 | Leaves                        | raw                  |
 | Empty directories             | included (opt-out)   |
 | Hidden entities               | excluded (opt-in)    |
@@ -185,6 +218,8 @@ Based on the research above, we define **`unixfs-v1-2025`** as an opinionated pr
 
 This profile documents the default UnixFS DAG construction parameters used by Kubo through version 0.39 when producing CIDv0. It is provided for users who depend on CIDv0 identifiers generated by Kubo and need to reproduce them with other implementations, or verify content against existing CIDv0 references. The year 2015 in the name indicates that the majority of these parameters were picked a decade ago, when the initial go-ipfs alpha software was implemented, and these defaults were never contested since then.
 
+Note: this profile is a best-effort approximation of historical behavior. It produces deterministic CIDs for files and smaller directories. However, as documented in [Historical inconsistency in HAMT sharding](#historical-inconsistency-in-hamt-sharding), there is a risk of divergence when directories exceed the HAMT sharding threshold, due to differences in threshold comparison operators and estimation methods across software versions. In such cases, the only recourse is to identify which version of software originally created the content and manually adjust import parameters to match those historic settings.
+
 | Parameter                     | `unixfs-v0-2015`     |
 | ----------------------------- | -------------------- |
 | CID version                   | CIDv0                |
@@ -195,13 +230,59 @@ This profile documents the default UnixFS DAG construction parameters used by Ku
 | DAG width (children per node) | 174                  |
 | HAMTDirectory fanout          | 256 blocks           |
 | HAMTDirectory threshold       | 256KiB (links-bytes) |
+| HAMT switch comparison        | >                    |
 | Leaves                        | dag-pb               |
 | Empty directories             | included             |
 | Hidden entities               | excluded (opt-in)    |
 | Symlinks                      | preserved            |
 | Mode (permissions)            | excluded (opt-in)    |
 | Mtime (modification time)     | excluded (opt-in)    |
 
+## Design rationale
+
+### Why `block-bytes` estimation for `unixfs-v1-2025`
+
+The `unixfs-v1-2025` profile uses `block-bytes` instead of `links-bytes` for HAMT threshold estimation because `links-bytes` has fundamental accuracy problems that undermine CID determinism.
+
+What `links-bytes` ignores:
+
+- `Tsize` field: the cumulative size of child sub-DAGs stored in each link. This varint-encoded field can add 1-10 bytes per link.
+- Protobuf overhead: length prefixes, field tags, and varint encoding that wrap each link and the overall message structure.
+- `mode` field: optional POSIX file permissions. When present, adds a varint to the serialized size.
+- `mtime` field: optional modification timestamp. When present, adds an embedded message with seconds (varint) and optional nanoseconds (fixed32).
+
+Problems caused by underestimation:
+
+1. Non-deterministic threshold crossing: a directory estimated at 250 KiB by `links-bytes` might actually serialize to 270 KiB. If another implementation using accurate estimation sees the true size exceeds the threshold, it converts to HAMT, producing a different CID for identical content.
+
+2. Block size limit risks: near the 1 MiB or 2 MiB block size limits used by various transports, underestimation can produce blocks that exceed limits, causing failures or requiring implementation-specific workarounds.
+
+Why `links-bytes` exists in `unixfs-v0-2015`:
+
+The legacy profile documents historical behavior. The `links-bytes` estimation was the original implementation in early go-ipfs, chosen for simplicity. Since many existing CIDv0 DAGs were created with this estimation, the `unixfs-v0-2015` profile preserves this behavior for users who need to reproduce legacy CIDs.
+
+### Why `>` (strictly greater than) for HAMT threshold
+
+The HAMT threshold comparison uses `>` rather than `>=`. A directory with estimated size exactly equal to the threshold (262144 bytes) remains a basic directory; only when size exceeds the threshold does it convert to HAMT.
+
+Rationale:
+
+1. Consistency with other profile limits: all threshold-like values in the UnixFS profile use the same `>` pattern, making the limits represent the maximum allowed value (inclusive). Conversion to a more complex structure happens when the count exceeds the limit:
+
+   - HAMT size threshold: `estimatedSize > HAMTShardingSize` - directory converts to HAMT when size exceeds 256 KiB
+   - Directory link count: `linkCount > maxLinks` - directory converts to HAMT when link count exceeds MaxLinks
+   - File DAG width: `childCount > maxLinks` - file node creates new tree level when child count exceeds FileDAGWidth
+
+   This consistency means implementers only need to understand one rule: limit values are the maximum allowed, conversion happens only when exceeding.
+
+2. Implementation alignment: the Helia (JavaScript) implementation uses `size > threshold`. Kubo/boxo documentation also specified `>`, but the actual Go implementation used `>=` until [boxo#1088](https://github.com/ipfs/boxo/pull/1088) fixed it. This divergence between documentation and implementation is another example of why `links-bytes` never achieved true cross-implementation determinism, and why the `unixfs-v1-2025` profile with `block-bytes` provides an opportunity to establish a proper standard.
+
+3. Threshold semantics: the threshold value represents the maximum allowed size for a basic directory, not the minimum size for HAMT. A directory at exactly the threshold is still within the allowed range for basic representation.
+
+4. Simpler representation preferred: at the exact boundary, basic directory is simpler (single flat node vs HAMT tree). When both representations are valid, preferring the simpler one reduces DAG complexity.
+
+5. Deterministic boundary behavior: edge cases are where CID mismatches most likely occur. Explicitly specifying that the threshold value stays basic eliminates ambiguity.
+
 ### User benefit
 
 Profiles provide key advantages for working with content-addressed data:
