Skip to content

slow download through git-annex from datasets.datalad.org #155

@bpinsard

Description

@bpinsard

For a long time, I had observed that fetching large containers was slow (drops to 2MB/s).
However, it is only through git-annex, direct download from the website has acceptable bandwidth (~50MB/s).

Puzzlingly, testing with wget with the annex object URL and with the path URL gives very different transfer rates (interrupted but just to show an example):

$ wget datasets.datalad.org/repronim/containers/.git/annex/objects/gG/32/MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif/MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif
--2025-12-29 21:32:44--  http://datasets.datalad.org/repronim/containers/.git/annex/objects/gG/32/MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif/MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif
Resolving datasets.datalad.org (datasets.datalad.org)... 129.170.233.11
Connecting to datasets.datalad.org (datasets.datalad.org)|129.170.233.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3238199296 (3.0G)
Saving to: ‘MD5E-s3238199296--b77879e14b03c4f13ffa0138b37f2014.1.sif’

-b77879e14b03c4f13ffa0138b37f   0%[                                                 ]  14.94M  1.61MB/s    eta 22m 19s^C
bpinsard@elm:~$ wget datasets.datalad.org/repronim/containers/images/bids/bids-nibabies--25.2.1.sif
--2025-12-29 21:33:24--  http://datasets.datalad.org/repronim/containers/images/bids/bids-nibabies--25.2.1.sif
Resolving datasets.datalad.org (datasets.datalad.org)... 129.170.233.11
Connecting to datasets.datalad.org (datasets.datalad.org)|129.170.233.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3238199296 (3.0G)
Saving to: ‘bids-nibabies--25.2.1.sif’

bids-nibabies--25.2.1.sif      35%[================>                                ]   1.07G  44.7MB/s    eta 46s

Not sure if that's only the case for me.
Could it be a webserver config issue?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions