[WIP] Extract ZarrCore as a separate library#252
Draft
asinghvi17 wants to merge 11 commits intomasterfrom
Draft
Conversation
Minimal core package with Project.toml (UUID, deps on JSON, DiskArrays,
OffsetArrays, DateTimes64, Dates) and module entry point defining
ZarrFormat{V}, DV constant, and AbstractCodecPipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- chunkkeyencoding.jl: AbstractChunkKeyEncoding, ChunkKeyEncoding
- MaxLengthStrings.jl: MaxLengthString type
- Compressors: abstract Compressor, compressortypes registry,
NoCompressor, default_compressor(), zcompress/zuncompress fallbacks
- Codecs: V3Codecs module with V3Codec{In,Out} abstract type,
v3_codec_parsers registry, codec_to_dict dispatch, BytesCodec,
TransposeCodec
- Filters: all pure-Julia filters (Fletcher32, Shuffle, Delta,
Quantize, FixedScaleOffset, VLen)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- metadata.jl: MetadataV2, type string system, fill value encoding/decoding, default_compressor() in constructors - metadata3.jl: MetadataV3 with registry-based codec parsing via parse_v3_codec/v3_codec_parsers (replaces hardcoded if/elseif), codec_to_dict dispatch for serialization (replaces isa checks), compressor_to_v3_bytes_codecs dispatch - pipeline.jl: V2Pipeline, V3Pipeline, pipeline_encode/pipeline_decode! Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Storage.jl: AbstractStore, storageregexlist, metadata I/O, SequentialRead/ConcurrentRead strategies - DirectoryStore, DictStore, ConsolidatedStore (core stores) - ZArray.jl: ZArray type, readblock!/writeblock!, zcreate/zzeros/zopen with default_compressor() - ZGroup.jl: ZGroup, zgroup, zopen, consolidate_metadata (without HTTP.serve/writezip) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Exports: ZArray, ZGroup, zopen, zzeros, zcreate, storagesize, storageratio, zinfo, DirectoryStore, DictStore, ConsolidatedStore, zgroup - Smoke tests: NoCompressor roundtrip, DirectoryStore, DictStore, V3 uncompressed roundtrip, default_compressor verification Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- src/Zarr.jl: imports from ZarrCore, re-exports public API, includes compressor/codec/store files, overrides default_compressor() to BloscCompressor via __init__, adds Codecs.V3Codecs wrapper module for backward-compatible module paths - src/Codecs/V3/compression_codecs.jl: GzipV3Codec, BloscV3Codec, ZstdV3Codec, CRC32cV3Codec, ShardingCodec with full infrastructure, all registering into ZarrCore.Codecs.V3Codecs.v3_codec_parsers - Compressor files: registration moved to Zarr.__init__ - Storage files: minor fixes for ZarrCore compatibility Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Add ZarrCore as path dependency via [sources.ZarrCore] - Remove DataStructures from [deps] and [compat] (unused) - Add ZarrCore to test/Project.toml with source path Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Remove 21 source files from src/ that now live in lib/ZarrCore/src/. Zarr.jl no longer includes these directly. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add ZarrCore test command and package structure documentation describing the core/wrapper split, default_compressor override pattern, and v3_codec_parsers registry. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- 2026-03-20-zarrcore-extraction-design.md: architectural decisions - 2026-03-20-zarrcore-extraction.md: 22-task implementation plan Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
asinghvi17
changed the title
Julia LTS (1.10) doesn't support [sources] in Project.toml for local path dependencies. Add explicit Pkg.develop step to resolve ZarrCore from lib/ZarrCore/ before the build step. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Pull Request Test Coverage Report for Build 23455574299Details
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #186.
The idea of this pr is to extract a core library ZarrCore.jl so that it can be (a) used in Zarr and (b) potentially depended on by Zarrs.jl also. This way,
Below follows a more detailed changelog. I'm still testing this and not quite there with the API yet. But it seems to work so far.
v3 arrays
Detailed description
What's in ZarrCore
What stays in Zarr.jl
Key design decisions
through Julia precompilation