Test the performance of alternative methods for S3 file download in R, for possible GDAL configuration tuning or further investigation of implementations

[ctoney]

This issue is to move the discussion thread in #883 into its own issue topic. Thanks to @pepijn-devries for reporting the original issue (a bug now fixed in dev), and following up on performance questions with tests for comparing some alternative methods (copied below from that original issue thread).

gdalraster::vsi_copy_file() wraps VSICopyFile() in the GDAL Virtual System Interface API (cpl_vsi.h, GDAL >= 3.7 for VSICopyFile()).

gdalraster::VSIFile is a class interface to a GDAL VSIVirtualHandle which abstracts C binary file I/O for GDAL Virtual File Systems.

The R package paws vendors botocore which provides paws.storage::s3_download_file().

Moved from #883:

I've written a test were I've downloaded the file using 3 different approaches:

1. vsi_copy_file
2. read and write using VSIFile
3. using the paws package

I've repeatedly (5 times) downloaded the same file with the three methods. If recorded the duration of each download and summarised the the results. Method 3 is about 25% faster than method 1. Method 2 also seems to be a bit faster than method 1.

Anyway, some stuff I have to think about. 25% doesn't seem to be a huge difference, but it can be when you need to download a lot of large files 😉. If you have any suggestions to boost the performance of gdalraster I'm all 👂👂

library(gdalraster)
library(CopernicusDataspace)
library(tidyr)
library(dplyr)

uri <- "s3://eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff"

vsi_f <- gsub("s3://", "/vsis3/", uri)

set_config_option("AWS_REGION", "us-east-1")
set_config_option("AWS_ACCESS_KEY_ID", dse_s3_key())
set_config_option("AWS_SECRET_ACCESS_KEY", dse_s3_secret())
set_config_option("AWS_VIRTUAL_HOSTING", "FALSE")
set_config_option("AWS_S3_ENDPOINT", "eodata.dataspace.copernicus.eu")
target <- file.path(tempdir(), "temp.tiff")

n_repeat <- 5
result <- data.frame(vsi_copy       = rep(NA, n_repeat),
                     vsi_read_write = rep(NA, n_repeat),
                     paws           = rep(NA, n_repeat))

for (i in seq_len(n_repeat)) {
  timing1 <- system.time({
    vsi_copy_file(vsi_f, target, TRUE)
  })
  unlink(target)
  result$vsi_copy[[i]] <- timing1[["elapsed"]]
  
  ## Let's try a 10 Mb buffer
  buffer_size <- 10*1024*1024
  timing2 <- system.time({
    vsi_fl <- new(VSIFile, vsi_f)
    con_out <- file(target, "wb")
    while (!vsi_fl$eof()) {
      buffer <- vsi_fl$read(buffer_size)
      writeBin(buffer, con_out)
    }
    vsi_fl$close()
    close(con_out)
  })
  result$vsi_read_write[[i]] <- timing2[["elapsed"]]
  
  timing3 <- system.time({
    fn <- dse_s3_download(uri, tempdir())
  })
  unlink(file.path(tempdir(), fn))
  result$paws[[i]] <- timing3[["elapsed"]]
  
}

result

## # Numbers are duration in seconds
##
##   vsi_copy vsi_read_write   paws
## 1   245.56         214.27 174.80
## 2   206.56         137.31 136.73
## 3   152.70         176.37 168.51
## 4   317.28         239.13 198.46
## 5   190.39         221.09 159.53

result |>
  pivot_longer(everything()) |>
  group_by(name) |>
  summarise(
    mean_duration = mean(value),
    sd = sd(value),
    n = n()
  )

## # A tibble: 3 × 4
##   name           mean_duration    sd     n
##   <chr>                  <dbl> <dbl> <int>
## 1 paws                    168.  22.5     5
## 2 vsi_copy                222.  62.6     5
## 3 vsi_read_write          198.  40.7     5

[pepijn-devries]

The R package paws vendors botocore which provides paws.storage::s3_download_file(). (Note: I believe that is what dse_s3_download() is using. Please correct me if that is not the case and I'll edit here.)

Yes dse_s3_download() uses paws.storage::s3_download_file().

[ctoney]

I think we'll want to consider the /vsis3_streaming/ file system handler (https://gdal.org/en/latest/user/virtual_file_systems.html#vsis3-streaming-aws-s3-files-streaming). It should provide a more efficient implementation in the case of sequentially reading the entire file content.

One way to do that is substitute the /vsis3_streaming/ prefix in place of /vsis3/.

However, there is also a currently undocumented configuration option, VSIS3_COPYFILE_USE_STREAMING_SOURCE, which would implement that when calling VSICopyFile() on a /vsis3/ source:
https://github.com/OSGeo/gdal/blob/fbce9c0a225795efd8f7ee553e29476367ed65f4/port/cpl_vsil_s3.cpp#L3608-L3612

We could use that with vsi_copy_file() keeping the /vsis3/ prefix as-is:

set_config_option("VSIS3_COPYFILE_USE_STREAMING_SOURCE", "YES")

vsi_f <- "/vsis3/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff"

system.time(vsi_copy_file(vsi_f, f_test, show_progress = TRUE))

I haven't run your test set with that change yet, but for one file download as above I saw a meaningful improvement in the transfer time. It should be worth checking whether that holds up under replication, and I'll try to run the test set a little later on.

[ctoney]

Correction on the comment above referencing port/cpl_vsil_s3.cpp:
That code does in fact set the default toVSIS3_COPYFILE_USE_STREAMING_SOURCE=YES, so (afaict) the behavior of VSICopyFile() on /vsis3/ is to use a streaming source by default (i.e., use /vsis3_streaming/ for a given /vsis3/ source). In case that fails, it will fall back to a non-streaming implementation for /vsis3/.

(The first time I copied the file with explicitly set VSIS3_COPYFILE_USE_STREAMING_SOURCE=YES, the transfer was ~33% faster than a previous copy without the setting. That seemed possibly outside of typical network variation, but it did not replicate, and then the code behavior as described above was further checked.)

When running the original test code above from @pepijn-devries, I see the same pattern as those results for 5 repeats: the vsi_read_write method on 10 MB chunks was consistently faster than the vsi_copy method. In my case, the average difference was greater (41% faster for vsi_read_write compared to the vsi_copy method), but the variability in timings was greater for vsi_read_write instead of vsi_copy.

With CPL_DEBUG enabled (set_config_option("CPL_DEBUG", "ON")), the output for the vsi_copy method seems to show reading in range requests(?) of size 3 MB, which looks like the following (is this expected output for the S3 streaming implementation?)

...
GDAL DEBUG: S3: Downloading 37699584-40845311 (https://eodata.dataspace.copernicus.eu/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff)...
GDAL DEBUG: S3: Got response_code=206
...
...

The output for the vsi_read_write method looks like the following, with chunk size of 10 MB as expected (buffer_size <- 10*1024*1024 set in the test code):

...
GDAL DEBUG: S3: Downloading 52428800-62914559 (https://eodata.dataspace.copernicus.eu/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff)...
GDAL DEBUG: S3: Got response_code=206
...
...

It seems like tuning on the chunk size is a possibility, but I'm skeptical how productive that would be since it may not generalize well.

[pepijn-devries]

It looks like the vsi_copy function uses the same strategy as the read-write implementation: it reads an writes the file in chunks. Both approaches use a range request, meaning that they target a specific chunk in the file on the server. The response status 206 is just telling you that the range request was successful. As for each request, the server needs to be contacted, and the correct chunk located, you would expect some overhead. This is probably why the strategy with larger chunks is faster. The reason that you see more variability in the read-write approach is perhaps because it is implemented in R. I'm curious if the speeds becomes more predictable when implemented in c++.

I agree that you cannot provide a generalised solution for all servers. However, using common sense, I think using larger yet feasible sized chunks is generally faster than using smaller chunks.

[pepijn-devries]

Maybe, it can easily be fixed by letting the user specify the chunk size and setting it as environment variable. See also OSGeo/gdal#8499

There is the variable VSIS3_CHUNK_SIZE. Same for other VSI systems.

[ctoney]

The streaming implementation should involve multithreading with the byte-range requests in concurrent connections (IIUC?). From OSGeo/gdal#13817 (comment):

indeed a bit surprising. As /vsicurl_streaming/ involves multithreading in its implementation, it might be that there's some inefficiency if the bandwidth of the network transfer is very high. Although I find it a bit surprising.

That's why I expected it to be more comparable to boto-based file copy. There should also be a difference in that respect between vsi_read_write and vsi_copy in the tests, with the latter using the multithreaded streaming implementation (or at least expected to, since VSICopyFile() should switch automatically to /vsis3_streaming/ when given a /vsis3/ source).

Good point about increasing CPL_VSIL_CURL_CHUNK_SIZE and CPL_VSIL_CURL_CACHE_SIZE. That would be good to test and I'll include it once I have a chance to experiment more. I'm unsure if it has an effect when /vsis3_streaming/ is in use.

It sounds like VSIS3_CHUNK_SIZE would only affect upload.

VSIS3_CHUNK_SIZE=: Defaults to 50. Set the chunk size for multipart uploads.

Another thing I will test, but haven't had a chance yet, is whether using a /vsis3_streaming/ prefix instead of /vsis3/ may have an effect with the vsi_read_write method.

[ctoney]

The performance issue with GDAL VSI is potentially due to the /vsicurl_streaming/ implementation not usable with this test file for some reason.

The download fails with a GDAL error when I give a /vsis3_streaming/ prefix explicitly for the test file. Using the configuration options as above in the original test, but giving the /vsis3_streaming/ prefix instead:

uri <- "s3://eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff"

vsi_stream_f <- gsub("s3://", "/vsis3_streaming/", uri)

system.time(vsi_copy_file(vsi_stream_f, target))
#> GDAL FAILURE 3: Copying of /vsis3_streaming/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff to /tmp/RtmpPE5rH6/temp.tiff failed: error while reading source file

However, for a different file in a publicly accessible S3 bucket, vsi_copy_file() is ~25% faster when the /vsis3_streaming/ prefix is given explicitly compared with /vsis3/ (in my limited tests so far using the following code). In a new R session, so that the default configuration options are restored:

code

library(gdalraster)

set_config_option("AWS_REGION", "us-west-2")
set_config_option("AWS_NO_SIGN_REQUEST", "YES")

bucket <- "sentinel-cogs"
path <- "sentinel-s2-l2a-cogs/10/U/CV/2025/6/S2A_10UCV_20250615_0_L2A/B04.tif"
vsi_f <- file.path("/vsis3", bucket, path)

(f_size <- vsi_stat_size(vsi_f))

vsi_curl_clear_cache()

target <- file.path(tempdir(), "temp.tif")

system.time(vsi_copy_file(vsi_f, target))

(vsi_stat_size(target) == f_size)
unlink(target)
vsi_curl_clear_cache()

vsi_stream_f <- file.path("/vsis3_streaming", bucket, path)

system.time(vsi_copy_file(vsi_stream_f, target))

(vsi_stat_size(target) == f_size)
unlink(target)
vsi_curl_clear_cache()

@pepijn-devries, could you possibly test for confirmation?

Mainly, do you see the same GDAL error from vsi_copy_file() but using the /vsis3_streaming/ prefix on your original test file that requires authentication?

It would be interesting to see your test results, running instead on a file in a public S3 bucket and giving /vsis3_streaming/ prefix for the vsi_copy method. The code below has changes to do that using paws directly for anonymous access (or can dse_s3_download() still be used with no-sign-request?).

vsi streaming test

library(gdalraster)
library(paws)
library(tidyr)
library(dplyr)

set_config_option("AWS_REGION", "us-west-2")
set_config_option("AWS_NO_SIGN_REQUEST", "YES")

s3 <- paws::s3(
  config = list(
    credentials = list(
      creds = list(
        access_key_id = "",
        secret_access_key = "",
        session_token = ""
      ),
      anonymous = TRUE
    ),
    region = "us-west-2"
  )
)

bucket <- "sentinel-cogs"
path <- "sentinel-s2-l2a-cogs/10/U/CV/2025/6/S2A_10UCV_20250615_0_L2A/B04.tif"
vsi_f <- file.path("/vsis3", bucket, path)
uri <- gsub("/vsis3/", "s3://", vsi_f)
vsi_stream_f <- file.path("/vsis3_streaming", bucket, path)

target <- file.path(tempdir(), "temp.tif")

n_repeat <- 5
result <- data.frame(vsi_copy       = rep(NA, n_repeat),
                     vsi_read_write = rep(NA, n_repeat),
                     paws           = rep(NA, n_repeat))

for (i in seq_len(n_repeat)) {
  timing1 <- system.time({
    vsi_copy_file(vsi_stream_f, target)
  })
  unlink(target)
  result$vsi_copy[[i]] <- timing1[["elapsed"]]
  
  ## Let's try a 10 Mb buffer
  buffer_size <- 10*1024*1024
  timing2 <- system.time({
    vsi_fl <- new(VSIFile, vsi_f)
    con_out <- file(target, "wb")
    while (!vsi_fl$eof()) {
      buffer <- vsi_fl$read(buffer_size)
      writeBin(buffer, con_out)
    }
    vsi_fl$close()
    close(con_out)
  })
  unlink(target)
  result$vsi_read_write[[i]] <- timing2[["elapsed"]]
  
  timing3 <- system.time(
    s3$download_file(Bucket = bucket, Key = path, Filename = target)
  )
  unlink(target)
  result$paws[[i]] <- timing3[["elapsed"]]
}

result

result |>
  pivot_longer(everything()) |>
  group_by(name) |>
  summarise(
    mean_duration = mean(value),
    sd = sd(value),
    n = n()
  )

[pepijn-devries]

From memory, I think vsis3_streaming did not work for me as well on that particular file, but I will check again. From what I can tell from documentation vsis3_streaming is generally faster than vsis3. I'm kind of busy with work stuff. But I will test the performance of your suggestions over the weekend. I'll get back to you on this ASAP...

[pepijn-devries]

Just repeated my earlier test using /vsis3_streaming/ instead of /vsis3/ It doesn't seem to affect speed much.

> ## # A tibble: 3 × 4
> ##   name           mean_duration    sd     n
> ##   <chr>                  <dbl> <dbl> <int>
> ## 1 paws                    168.  22.5     5
> ## 2 vsi_copy                222.  62.6     5
> ## 3 vsi_read_write          198.  40.7     5

If I have some time tomorrow, I will try some tests by changing some of the suggested environment variables...

[ctoney]

@pepijn-devries, thanks, could you just confirm... in my last comment above I had:

uri <- "s3://eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff"

vsi_stream_f <- gsub("s3://", "/vsis3_streaming/", uri)

system.time(vsi_copy_file(vsi_stream_f, target))
#> GDAL FAILURE 3: Copying of /vsis3_streaming/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff to /tmp/RtmpPE5rH6/temp.tiff failed: error while reading source file

I'm not able to use /vsi_streaming/ on that file at all. It always fails with the GDAL error. I've tried a few different ways including GDAL CLI outside of R. I'm using the credentials obtained in the Copernicus dashboard, from the link you gave previously (https://documentation.dataspace.copernicus.eu/APIs/S3.html - under Generate secrets). It does not fail when I use the /vsis3/ prefix, but the transfer is relatively slow in that case.

Are your credentials potentially different, or does the use of dse_s3_download() affect credentials in any way?

If you were able to test with a file in a public bucket, as in the code for "vsi streaming test" above, that could possibly be illuminating for whether the VSI issue may be specific to the Copernicus bucket or our configuration for it.

[pepijn-devries]

Your code above didn't work for me in a clean session. I've rewritten it as a reprex. I've also added the required config options explicitly. It also needs dse_s3_key() and dse_s3_secret() to return valid strings. As you can see the code below works for me.

library(gdalraster)
#> GDAL 3.11.0 (released 2025-05-06), GEOS 3.13.1, PROJ 9.6.0
library(CopernicusDataspace)
uri <- "s3://eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2024/11/25/S1A_IW_GRDH_1SDV_20241125T055820_20241125T055845_056707_06F55C_12F9_COG.SAFE/measurement/s1a-iw-grd-vh-20241125t055820-20241125t055845-056707-06f55c-002-cog.tiff"

set_config_option("AWS_REGION", "us-east-1")
set_config_option("AWS_ACCESS_KEY_ID", dse_s3_key())
set_config_option("AWS_SECRET_ACCESS_KEY", dse_s3_secret())
set_config_option("AWS_VIRTUAL_HOSTING", "FALSE")
set_config_option("AWS_S3_ENDPOINT", "eodata.dataspace.copernicus.eu")

vsi_stream_f <- gsub("s3://", "/vsis3_streaming/", uri)
target <- tempfile(fileext = ".tif")

system.time(vsi_copy_file(vsi_stream_f, target))
#>    user  system elapsed 
#>    0.34    0.53  110.50

^{Created on 2026-02-09 with reprex v2.1.1}

For completeness sake, I'm using:

gdalraster::gdal_version()
#> [1] "GDAL 3.11.0 \"Eganville\", released 2025/05/06"
#> [2] "3110000"                                       
#> [3] "20250506"                                      
#> [4] "3.11.0"

^{Created on 2026-02-09 with reprex v2.1.1}

[ctoney]

That's showing 110 seconds which is half the previously reported download time for vsi_copy_file, and would be quite a bit faster than any of three methods in the previous results.

[pepijn-devries]

That's showing 110 seconds which is half the previously reported download time for vsi_copy_file, and would be quite a bit faster than any of three methods in the previous results.

Yes, but I didn't apply replication in the last case, and I'm sharing my bandwidth with the rest of the family. So I cannot guarantee that it was performed under similar conditions 😉. My comment from 13 hours ago show the results of the same test I performed earlier (with repetition), with the only change being that I used /vsis3_streaming/ instead of /vsis3/. So even if the streaming variant is faster, the methods (read/write, vsi copy, paws) show a considerable difference.