Skip to content

Enable AWS client to copy objects >5GB in size #563

@james-rms

Description

@james-rms

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I'm trying to copy objects within a bucket as part of my data ingestion workflow. However, when thse objects exceed 5GB in size, this fails with an error. There's a built-in limit in the S3 API for the maximum size of a copy operation.

Describe the solution you'd like

I've outlined the solution i'd prefer in #561 - essentially the copy method should detect large copies and transparently switch to a multipart workflow.

Describe alternatives you've considered

Alternatively, we could also expose this API as part of the multipart-upload API.

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions