Occasional CRC Errors When Streaming Data into Zip using AsyncZipDeflate

[Masty88]

Occasional CRC Errors When Streaming Data into Zip using AsyncZipDeflate

Context
I am using fflate to fetch a list of 3D geographic files in various formats along with orthophotos in JPEG from Amazon S3. When retrieving the files, I use response.body.getReader() to stream the data into a ZIP folder.

Issue
When using AsyncZipDeflate or ZipDeflate (even with compression level set to 0), I encounter CRC errors intermittently - sometimes immediately, other times sporadically (about one in every two attempts). However, if I use the array buffer directly without streaming, or if I use ZipPassThrough for streaming, it works flawlessly 100% of the time.

Steps to Reproduce
Fetch a list of files from Amazon S3.
Stream the data into a ZIP folder using AsyncZipDeflate or ZipDeflate.
Occasionally encounter CRC errors in the resulting ZIP file.
Expected Behavior
The ZIP file should be created without CRC errors, similar to when using ZipPassThrough or directly passing the array buffer.

Actual Behavior
CRC errors occur intermittently when using AsyncZipDeflate or ZipDeflate for streaming data into a ZIP folder.

Additional Information
The files being fetched are 3D geographic files in various formats along with JPEG orthophotos.
The issue seems to be specific to the streaming process with AsyncZipDeflate or ZipDeflate.
StackBlitz Reproduction
I have created a StackBlitz project to demonstrate the issue: [Link to StackBlitz Project]

https://js-7tnzqy.stackblitz.io
https://stackblitz.com/edit/js-7tnzqy?file=download.js

import { Zip, AsyncZipDeflate } from 'fflate';

async function downloadAndCompress(urlsToDownload) {
  console.log(urlsToDownload);
  let chunks = [];

  const zipFile = new Zip();

  const zipCompletionPromise = new Promise((resolve) => {
    zipFile.ondata = (err, dat, final) => {
      if (err) {
        throw err; // or handle error as you see fit
      }
      chunks.push(dat);
      if (final) {
        const blob = new Blob(chunks, { type: 'application/zip' });
        const url = URL.createObjectURL(blob);
        resolve(url);
      }
    };
  });

  const downloadAndStreamToZip = async (url) => {
    const response = await fetch(url);
    const fileName = url.split('/').pop();
    const fileStream = new AsyncZipDeflate(fileName, { level: 4 });
    zipFile.add(fileStream);

    const reader = response.body.getReader();
    let done = false;
    while (!done) {
      const { done: chunkDone, value } = await reader.read();
      done = chunkDone;
      if (value) {
        fileStream.push(new Uint8Array(value), done);
      }
    }

    return new Promise((resolve) => {
      if (done) {
        fileStream.push(new Uint8Array(0), true);
        resolve();
      }
    });
  };

  await Promise.all(urlsToDownload.map((url) => downloadAndStreamToZip(url)));

  zipFile.end();

  return await zipCompletionPromise;
}

export { downloadAndCompress };

[noseworthy]

@Masty88 did you ever find fix or root cause? I'm seeing a similar thing unfortunately.

[noseworthy]

Sorry, just saw the issue! Ignore me: #194