-
Notifications
You must be signed in to change notification settings - Fork 16
Open
Description
For a long time, I had observed that fetching large containers was slow (drops to 2MB/s).
However, it is only through git-annex, direct download from the website has acceptable bandwidth (~50MB/s).
Puzzlingly, testing with wget with the annex object URL and with the path URL gives very different transfer rates (interrupted but just to show an example):
$ wget datasets.datalad.org/repronim/containers/.git/annex/objects/gG/32/MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif/MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif
--2025-12-29 21:32:44-- http://datasets.datalad.org/repronim/containers/.git/annex/objects/gG/32/MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif/MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif
Resolving datasets.datalad.org (datasets.datalad.org)... 129.170.233.11
Connecting to datasets.datalad.org (datasets.datalad.org)|129.170.233.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3238199296 (3.0G)
Saving to: ‘MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif’
-b77879e14b03c4f13ffa0138b37f 0%[ ] 14.94M 1.61MB/s eta 22m 19s^C
bpinsard@elm:~$ wget datasets.datalad.org/repronim/containers/images/bids/bids-nibabies--25.2.1.sif
--2025-12-29 21:33:24-- http://datasets.datalad.org/repronim/containers/images/bids/bids-nibabies--25.2.1.sif
Resolving datasets.datalad.org (datasets.datalad.org)... 129.170.233.11
Connecting to datasets.datalad.org (datasets.datalad.org)|129.170.233.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3238199296 (3.0G)
Saving to: ‘bids-nibabies--25.2.1.sif’
bids-nibabies--25.2.1.sif 35%[================> ] 1.07G 44.7MB/s eta 46s
Not sure if that's only the case for me.
Could it be a webserver config issue?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels