v0.7.0-RC1

Changelog - v0.7.0

Migration Guide

From v0.5.x / v0.6.x

Database Migration

The database schema has changed significantly. Run the following after upgrading:

singularity admin init

This will auto-migrate the schema. Key changes:

• Foreign keys changed from CASCADE to SET NULL for performance
• New indexes added for job and preparation cleanup

admin init is generally cheap to run if schema is healthy so it's advisable to automate as a pre-start script in orchestration, however the initial migration for 0.7.0 may need some time due to index rebuilds so it is advisable to halt other services and run admin init standalone in this one case.

Breaking: MongoDB Removed

MongoDB backend import is no longer supported. If you were using MongoDB (singularity v1):

1. Export your data from v1
2. Upgrade to v0.6.0-RC2 and import to PostgreSQL/YugabyteDB (MySQL strongly discouraged, sqlite only for very small deployments)
3. Then upgrade to v0.7.0

Sequence Reset (PostgreSQL/YugabyteDB)

Automatic: singularity admin init now detects and fixes stale sequences automatically. This handles the common case of importing data with explicit IDs (e.g., from MySQL backup).

Manual: If needed, a standalone script is available at scripts/fix-sequences-after-import.sql.

Piece Type Classification

Automatic: singularity admin init now infers piece_type for CAR files that predate this column, classifying them as data (contains file content) or dag (directory metadata only). This is a no-op if all pieces are already labeled.

Manual: A standalone script is available at scripts/infer_piece_type.sql for inspection or manual runs.

Curio Compatibility (DAG Pieces)

If DAG pieces were rejected by Curio, you will need to regenerate them:

1. Run singularity admin init to classify existing pieces
2. Delete the rejected DAG pieces using singularity prep delete-piece
3. Re-run singularity prep start-daggen on affected preparations to create Curio-compatible pieces

---

Features

• prep delete-piece command - Delete specific pieces from a preparation (#602)
• Small piece padding - Pad very small pieces with literal zeroes for curio compatibility (#595)

Bug Fixes

• S3 car file renames - Fix car file renames on S3-type remotes (#598)
• Remote storage padding - Use local tmpfile for padding small pieces on remote/non-appendable storage (#597)
• Deadlocks resolved - Fix remaining deadlocks in prep deletion and job claiming (#596)
• Download flags - Harmonize download flags across commands (#599, closes #460)
• FK re-scanning - Avoid re-scanning known valid foreign keys during cleanup (#601)
• docker-compose env var - Fix singularity_init container environment variable (#526)
• prep add-piece CLI - Fix parser bug treating positional args as subcommands; auto-lookup existing pieces by CID for piece consolidation workflows (#607)

Database & Performance

• Nullable FKs for massive deletes - Use nullable foreign keys and pointers for fast bulk deletion; orphaned objects cleaned up via reaper task in batches (#600)
• Indexes for FK cleanup - Add indexes on jobs/preps foreign keys for faster operations
• SKIP LOCKED for job claiming - Prevent deadlocks during concurrent job acquisition
• Deferred association loading - Load associations after claiming to reduce lock contention

Infrastructure

• MongoDB removed - Remove legacy MongoDB support (#588)
• Devcontainer CI - Add devcontainer-based CI workflow with PostgreSQL and MySQL (#586)
• Distroless runner - Use distroless container image for smaller footprint (#592)
• Dependency upgrades - Significantly update gorm, boxo, rclone, libp2p, and security packages (#591, #589)
• Swagger fixes - Remove hardcoded host, dedupe storage type fields, bump go-swagger to v0.33.1 (#605, #606, #608)

Breaking Changes

• MongoDB backend no longer supported

---

PRs Included

PR	Title
#608	bump go-swagger to v0.33.1
#607	fix add-piece to lookup existing pieces by CID
#606	dedupe fields in storage types codegen
#605	remove hardcoded @host in swagger
#602	feat: add prep delete-piece command
#601	hotfix: avoid re-scanning known valid FKs
#600	give in and use nullable FKs/pointers for massive deletes
#599	harmonize download flags, closes #460
#598	fix: correctly rename cars on S3 type remotes
#597	hotfix: use local tmpfile for padding small pieces
#596	fix remaining deadlocks + prep deletion
#595	Pad very small pieces with literal zeroes
#592	update runner image
#591	improve lotus API test survivability
#589	chore/update
#588	remove legacy mongodb
#586	Devcontainer-based CI flow
#526	fix docker-compose.yml singularity_init container env var
#519	chore: bump version to v0.6.0-RC3
#516	Fix: restore version.json to v0.6.0-RC2

v0.6.0-RC4

What's Changed

• Fix: restore version.json to v0.6.0-RC2 [skip ci] by @Sankara-Jefferson in #516
• chore: bump version to v0.6.0-RC3 by @Sankara-Jefferson in #519
• Devcontainer-based CI flow by @parkan in #586
• chore/update by @parkan in #589
• improve lotus API test survivability by @parkan in #591
• update runner image by @parkan in #592
• Pad very small pieces with literal zeroes by @parkan in #595
• fix remaining deadlocks + prep deletion by @parkan in #596
• harmonize download flags, closes #460 by @parkan in #599
• fix: correctly rename cars on S3 type remotes by @parkan in #598
• hotfix: use local tmpfile for padding small pieces if storage is remote/does not allow appends by @parkan in #597
• give in and use nullable FKs/pointers for massive deletes by @parkan in #600
• hotfix: avoid re-scanning known valid FKs by @parkan in #601
• feat: add prep delete-piece command by @parkan in #602
• remove legacy mongodb by @parkan in #588

Full Changelog: v0.6.0-RC2...v0.6.0-RC4

v0.6.0-RC2

🚀 Release Candidate: v0.6.0-R2

This RC introduces support for --min-piece-size, allowing more control over padding behavior for small DAG and data segments, which often led to excessive padding and inefficiencies in deal making and retrieval.

✨ Highlights

• Add --min-piece-size CLI option (default: 2MiB)
• Pieces below this size are padded to min, not the full --piece-size
• Improved chunker behavior for small DAGs and remainder CARs
• Metadata now includes pieceType (data vs. DAG)

🧪 Intended for Testing

• Especially relevant for Curio compatibility
• Allows deal size minimization without breaking existing workflows
• Final v0.6.0 may include additional patches if needed

📎 Related PRs

• #479: Feat/allow small pieces
• #467: Linter/tooling prep for release

🙏 Contributors

Thanks to @parkan and others for implementation.

v0.5.17-RC1

Full Changelog: v0.5.17...v0.5.17-RC1

v0.5.17

What's Changed

• update database schema diagram by @parkan in #447
• Update go-libp2p to v0.39.1 by @parkan in #453

Full Changelog: v0.5.16...v0.5.17

v0.5.16

What's Changed

• Update .goreleaser.yaml for 2.x by @parkan in #444

Full Changelog: v0.5.15...v0.5.16

v0.5.15

What's Changed

• Fix tests by @gammazero in #436
• Update dependencies by @gammazero in #438
• [DRAFT] enable union backend by @parkan in #439
• Additional commits required to make the tests pass after #439 by @ianconsolata in #441

New Contributors

• @parkan made their first contribution in #439
• @ianconsolata made their first contribution in #441

Full Changelog: v0.5.14...v0.5.15

v0.5.14

What's Changed

• Disable metrics reporting by default by @xinaxu in #434

Full Changelog: v0.5.13...v0.5.14

v0.5.13

What's Changed

• Update doc about singularity download server by @xinaxu in #430
• Allow unknown remote object size by @xinaxu in #431

Full Changelog: v0.5.12...v0.5.13

v0.5.12

What's Changed

• Ignore error for making identical deals by @xinaxu in #427
• Add flags for enable-piece and enable piece metadata by @xinaxu in #428

Full Changelog: v0.5.11...v0.5.12