reading file twice (in PutObject)

[crusader-mike]

Apparently AWS SDK needs to calculate some sort of signature before sending bytes over wire (in PutObject/etc) -- related stream gets read and then rewound to start and read again. Is there any way to avoid this double "read file from disk" besides using streambuf that keeps entire file in memory?

[JonathanHenson]

The default ctor args for s3 turn the payload hashing off of you are using tls. Also, setting the content length field will prevent the seek operations to figure it out. By controlling both of these settings, the file will only be read once.

…

Sent from my iPhone

On Oct 30, 2017, at 9:27 AM, crusader-mike ***@***.***> wrote: Apparently AWS SDK needs to calculate some sort of signature before sending bytes over wire (in PutObject/etc) -- related stream gets read and then rewound to start and read again. Is there any way to avoid this double "read file from disk" besides using streambuf that keeps entire file in memory? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[crusader-mike]

Huh... I am using https and reading file (in my own implementation of streambuf) only in underflow() -- according to your criteria SDK shouldn't scan my file twice. I'll double check tonight

[crusader-mike]

After examining my (relatively recent) SDK sources I came to a conclusion that it is impossible to switch off "payload signing" without overriding request class methods -- all requests get payload signed if <request-class>::SignBody() returns true. Similar flag passed by S3Client into aws signer class gets used only with payload-less requests -- i.e. not used at all. The only way to avoid payload being signed is to derive from requess-class, override SignBody() and return false.

I dug in this a bit more and found that Amazon S3 (as of now) allows unsigned payloads, but you can configure your bucket to deny such requests. Is it a prelude to total ban on unsigned payloads service wide?

In any case -- it would be very nice to have a switch in S3Client (or library-wide?) that would forcefully disable/enable payload signing (regardless of HTTP/HTTPS flag). Please :)

Until then I believe I have only these options:

1. live with 'double read'
2. load entire file into memory -- not end of the world, since I use multipart upload for large files anyway
3. override SignBody() of each request I use and hope users don't use HTTP urls in my program

Sounds like option number 2 is the least of all evil right now.

[marcomagdy]

This is a bug. We need to fix it.

[crusader-mike]

Thank you. Would be nice to be able to skip payload signature in HTTP case too...

[crusader-mike]

A note about option N2 -- multipart upload is limited to 10'000 chunks. Which means for files bigger than 50GB chunk size will start growing and at some point they won't fit into memory (especially if you upload them in parallel). I.e. you'll have to limit memory buffer and you'll end up reading entire file twice.

[wps132230]

Hi, you can avoid reading file twice by skipping payload signature with the latest update(ver 1.3.1) now:

auto s3Client = Aws::MakeShared<S3Client>(ALLOCATION_TAG, credentialsProvider, config, AWSAuthV4Signer::PayloadSigningPolicy::Never /*signPayloads*/, true /*useVirtualAddressing*/);

in which AWSAuthV4Signer::PayloadSigningPolicy::Never will overrule SignBody(), even though it's true, then the payload will never be signed.

Please consult this commit for more details: b9ccf4d

I am closing this issue, if you encounter any further problem, please reopen it.

[crusader-mike]

Thanks, Pushen!