Add a Jpeg2000 compressor

[juntyr]

Closes #16

[juntyr]

@treigerm Could we merge this before #15?

[treigerm]

Thanks for this @juntyr ! I'm currently trying to test this out. I initially got an error that dtype i32 isn't supported so I changed encode_dtype="int32" to encode_dtype="int16". That made the error disappear but the codec is now either extremely slow or stuck in an infinite loop somewhere: I ran uv run scripts/compress.py --include-dataset "cams-nitrogen-dioxide-tiny" --include-compressor "jpeg2000" on this branch and it has been running for 20 minutes now (the other compressors usually finish in less than a minute). Any idea what could be going on? In general, I wouldn't expect JPEG to be overly expensive.

Also happy to wait to merge #15 until this one is ready! I will get back to #15 tomorrow and try to adjust it based on what we discussed in the meeting.

[juntyr]

Thanks for this @juntyr ! I'm currently trying to test this out. I initially got an error that dtype i32 isn't supported so I changed encode_dtype="int32" to encode_dtype="int16".

Whoops, that's my fault. I'll release a patch release

That made the error disappear but the codec is now either extremely slow or stuck in an infinite loop somewhere: I ran uv run scripts/compress.py --include-dataset "cams-nitrogen-dioxide-tiny" --include-compressor "jpeg2000" on this branch and it has been running for 20 minutes now (the other compressors usually finish in less than a minute). Any idea what could be going on? In general, I wouldn't expect JPEG to be overly expensive.

What are the dimensions of the tiny CAMS dataset? I just tested the codec in the web browser, and it can encode 100x100x100 in <1sec.

Also happy to wait to merge #15 until this one is ready! I will get back to #15 tomorrow and try to adjust it based on what we discussed in the meeting.

We'll merge whichever one is ready first, let's see if we can get this one working

[treigerm]

What are the dimensions of the tiny CAMS dataset? I just tested the codec in the web browser, and it can encode 100x100x100 in <1sec.

The tiny CAMS data is (time=4, vertical=4, lat=241, lon=480) so it is roughly the same order of magnitude. At the very least the size difference wouldn't explain the difference in runtime.

[juntyr]

So ... the problem was a deadlock. I'm still trying to wrap my head around why, but I know where. I'll try to fix the deadlock next week, for now we can just compress chunks sequentially

[treigerm]

Thanks for the fix @juntyr ! I now tested it on the other data sources as well and it seems to work. Just left a question about the SZ3 version update but feel free to merge!

[juntyr]

@treigerm Could you check that the issue is fixed (after a uv sync) with the new numcodecs-wasm version? Feel free to squash and merge if it is

[treigerm]

Can confirm the issue is fixed, thansk @juntyr !