S3: Multipart upload fails with invalid request signature

[HenryS1]

Hello

After updating pekko connectors to 1.3.0 and the aws sdk to 2.42.2 I get the following error when I try to run a multipart upload.

java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Previously I was on pekko connectors 1.2.0 and aws sdk 2.29.52.

I noticed this happening in application code but it also happens if I use the following snippet extracted from S3ITTest.scala in this repository, where I supply my own bucket name and client.

      val fileContent = (0 to 1000000).mkString
      val createMultipartUploadResponse = client
        .createMultipartUpload(
          CreateMultipartUploadRequest.builder().bucket(bucketName).key("bar").contentType("text/plain").build())
        .join()

      val p1 = client
        .uploadPart(
          UploadPartRequest
            .builder()
            .bucket(bucketName)
            .key("bar")
            .partNumber(1)
            .uploadId(createMultipartUploadResponse.uploadId())
            .build(),
          AsyncRequestBody.fromString(fileContent))
        .join
      val p2 = client
        .uploadPart(
          UploadPartRequest
            .builder()
            .bucket(bucketName)
            .key("bar")
            .partNumber(2)
            .uploadId(createMultipartUploadResponse.uploadId())
            .build(),
          AsyncRequestBody.fromString(fileContent))
        .join

      client
        .completeMultipartUpload(
          CompleteMultipartUploadRequest
            .builder()
            .bucket(bucketName)
            .key("bar")
            .uploadId(createMultipartUploadResponse.uploadId())
            .multipartUpload(
              CompletedMultipartUpload
                .builder()
                .parts(CompletedPart.builder().partNumber(1).eTag(p1.eTag()).build(),
                  CompletedPart.builder().partNumber(2).eTag(p2.eTag()).build())
                .build())
            .build())
        .join

      val result = client
        .getObject(GetObjectRequest.builder().bucket(bucketName).key("bar").build(),
          AsyncResponseTransformer.toBytes[GetObjectResponse]())
        .join

The client is constructed like this

    S3AsyncClient.builder
      .credentialsProvider(DefaultCredentialsProvider.builder().build())
      .region(region)
      .httpClient(PekkoHttpClient.builder().withActorSystem(system).build())
      .build()

If I use the netty client instead of the pekko client the request works, but I want to use the pekko client.

[pjfanning]

@HenryS1 I am not an AWS expert. Can you confirm if you think that Pekko-Connectors S3 connector has a regression in the 1.3.0 release?
If so, maybe it is related to #1299.
That PR introduced an allow list for headers and maybe, we are now stripping headers because some required headers are not included in the allow list.

[pjfanning]

@Luka-J9 @mdedetrich any chance that #1245 changes could be causing this?

They were merged to 1.3.0 release with #1299

[HenryS1]

@pjfanning At first I thought it might be a regression in pekko-aws-spi-pekko-http, but I've just tried building pekko-connectors on version 1.2.0 using version 2.42.2 of the aws sdk and I still get the error when I run the request. I believe there was a change in AWS data integrity checks between my previous aws sdk version 2.29.52 and 2.42.2 which could be related to this, but I'm confused about why the signature error would happen.

As far as I can tell the signature for the payload is calculated by the SDK and the headers are directly converted from a java representation to a scala representation. Looking at the source it doesn't seem that the pekko client for aws spi uses the allowed-headers list from the PR that you mentioned.

It seems that it is something going wrong in the pekko-aws-spi-pekko-http PekkoHttpClient, but I don't know what the problem is.

[pjfanning]

@HenryS1 are you using the S3 connector as well or just using the pekko-aws-spi-pekko-http lib?

[HenryS1]

@pjfanning I'm using the SQS pekko connector and for S3 I'm just using pekko-aws-spi-pekko-http with the AWS sdk.

[pjfanning]

@HenryS1 would you be interested in trying to build aws-spi-pekko-http with the change in #1546 to see if that helps?

[HenryS1]

@pjfanning ok, I'll try it out and let you know.

[HenryS1]

@pjfanning the changes on that branch have fixed the problem. Thanks so much for getting onto it so quickly!

[pjfanning]

The AWS change that we suspect may have required this change is in

https://github.com/aws/aws-sdk-java-v2/releases/tag/2.41.17 (PR 6607)