The files in an object store are not visible until the write has been completed. In-progress writes are simply saved to a local file/cached in RAM and only uploaded. at the end of a write operation. If a process terminated unexpectedly, or failed to call the close() method on an output stream, the pending data will have been lost.

Again, this is due to the fact that the data is cached locally until the close() operation. The S3A filesystem cannot be used as a store of data if it is required that the data is persisted durably after every Syncable.hflush() or Syncable.hsync() call. This includes resilient logging, HBase-style journaling and the like. The standard strategy here is to save to HDFS and then copy to S3.

The application has tried to call either the Syncable.hsync() or Syncable.hflush() methods on an S3A output stream. This has been rejected because the connector isn’t saving any data at all. The Syncable API, especially the hsync() call, are critical for applications such as HBase to safely persist data.

When configured to do so, the S3A connector throws an UnsupportedOperationException when these API calls are made, because the API guarantees absolutely cannot be met: nothing is being flushed or saved.

These recommendations apply to all filesystems.

For consistency with other filesystems, S3A output streams do not by default reject the Syncable calls -instead they print a warning of its use.

The count of invocations of the two APIs are collected in the S3A filesystem Statistics/IOStatistics and so their use can be monitored.

To switch the S3A connector to rejecting all use of hsync() or hflush() calls, set the option fs.s3a.downgrade.syncable.exceptions to false.

<property>
  <name>fs.s3a.downgrade.syncable.exceptions</name>
  <value>false</value>
</property>
Regardless of the setting, the Syncable API calls do not work. Telling the store to not downgrade the calls is a way to 1. Prevent applications which require Syncable to work from being deployed against S3. 2. Identify applications which are making the calls even though they don’t need to. These applications can then be fixed -something which may take time.
Put differently: it is safest to disable downgrading syncable exceptions. However, enabling the downgrade stops applications unintentionally using the API from breaking.
Tip: try turning it on in staging environments to see what breaks.

S3 is not a filesystem. The S3A connector mimics file and directory rename by

When performing file operations, the user may run into an issue where the KMS key arn is invalid.

Reading an unencrypted file would fail when read through CSE enabled client.

java.lang.SecurityException: Instruction file not found for S3 object with bucket name: ap-south-cse, key: unencryptedData.txt
CSE enabled client should read encrypted data only.

KMS key ID is required for CSE-KMS to encrypt data, not providing one leads to failure.

2021-07-07 11:33:04,550 WARN fs.FileSystem: Failed to initialize filesystem
s3a://ap-south-cse/: java.lang.IllegalArgumentException: CSE-KMS
method requires KMS key ID. Use fs.s3a.encryption.key property to set it.
-ls: CSE-KMS method requires KMS key ID. Use fs.s3a.encryption.key property to
 set it.
set fs.s3a.encryption.key=<KMS_KEY_ID> generated through AWS console.

KMS key ID used to PUT(encrypt) the data, must be the one used to GET the data.

cat: open s3a://ap-south-cse/encryptedData.txt at 0 on
s3a://ap-south-cse/encryptedData.txt:
software.amazon.awssdk.services.kms.model.IncorrectKeyException: The key ID in the
request does not identify a CMK that can perform this operation. (Service: AWSKMS;
Status Code: 400; ErrorCode: IncorrectKeyException;
Request ID: da21aa8a-f00d-467c-94a0-32b627d32bc0; Proxy: null):IncorrectKeyException:
The key ID in the request does not identify a CMK that can perform this
operation. (Service: AWSKMS ; Status Code: 400; Error Code: IncorrectKeyException;
Request ID: da21aa8a-f00d-467c-94a0-32b627d32bc0; Proxy: null)
Use the same KMS key ID used to upload data to download and read it as well.

Using a KMS key ID from a different region than the bucket used to store data would lead to failure while uploading.

mkdir: PUT 0-byte object  on testmkdir:
software.amazon.awssdk.services.kms.model.NotFoundException: Key
'arn:aws:kms:ap-south-1:152813717728:key/<KMS_KEY_ID>'
does not exist (Service: AWSKMS; Status Code: 400; Error Code: NotFoundException;
Request ID: 279db85d-864d-4a38-9acd-d892adb504c0; Proxy: null):NotFoundException:
Key 'arn:aws:kms:ap-south-1:152813717728:key/<KMS_KEY_ID>'
does not exist(Service: AWSKMS; Status Code: 400; Error Code: NotFoundException;
Request ID: 279db85d-864d-4a38-9acd-d892adb504c0; Proxy: null)
While generating the KMS Key ID make sure to generate it in the same region as your bucket.

If Range get is not supported for a CSE algorithm or is disabled:

java.lang.SecurityException: Unable to perform range get request: Range get support has been disabled. See https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html
Range gets must be enabled for CSE to work.

The S3 Encryption Client is configured to support range get requests. This warning would be shown everytime S3-CSE is used.

2021-07-14 12:54:09,525 [main] WARN  s3.AmazonS3EncryptionClientV2
(AmazonS3EncryptionClientV2.java:warnOnRangeGetsEnabled(401)) - The S3
Encryption Client is configured to support range get requests. Range gets do
not provide authenticated encryption properties even when used with an
authenticated mode (AES-GCM). See https://docs.aws.amazon.com/general/latest
/gr/aws_sdk_cryptography.html
We can Ignore this warning since, range gets must be enabled for S3-CSE to get data.

The S3 Encryption Client is configured to read encrypted data with legacy encryption modes through the CryptoMode setting, and we would see this warning for all S3-CSE request.

2021-07-14 12:54:09,519 [main] WARN  s3.AmazonS3EncryptionClientV2
(AmazonS3EncryptionClientV2.java:warnOnLegacyCryptoMode(409)) - The S3
Encryption Client is configured to read encrypted data with legacy
encryption modes through the CryptoMode setting. If you don't have objects
encrypted with these legacy modes, you should disable support for them to
enhance security. See https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html
We can ignore this, since this CryptoMode setting(CryptoMode.AuthenticatedEncryption) is required for range gets to work.

If you generated an Asymmetric CMK from AWS console then CSE-KMS won’t be able to generate unique data key for encryption.

Caused by: software.amazon.awssdk.services.kms.mode.InvalidKeyUsageException:
You cannot generate a data key with an asymmetric CMK
(Service: AWSKMS; Status Code: 400; Error Code: InvalidKeyUsageException; Request ID: 93609c15-e490-4035-8390-f4396f0d90bf; Proxy: null)
Generate a Symmetric Key in the same region as your S3 storage for CSE-KMS to work.

If the value in fs.s3a.encryption.key property, does not exist /valid in AWS KMS CMK(Customer managed keys), then this error would be seen.

Caused by: software.amazon.awssdk.services.kms.model.NotFoundException: Invalid keyId abc
(Service: AWSKMS; Status Code: 400; Error Code: NotFoundException; Request ID:
 9d53552a-3d1b-47c8-984c-9a599d5c2391; Proxy: null)
Check if fs.s3a.encryption.key is set correctly and matches the same on AWS console.

User doesn’t have authorization to the specific AWS KMS Key ID.

Caused by: software.amazon.awssdk.services.kms.model.KmsException:
User: arn:aws:iam::152813717728:user/<user> is not authorized to perform:
 kms:GenerateDataKey on resource: <key_ID>
(Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException;
  Request ID: 4ded9f1f-b245-4213-87fc-16cba7a1c4b9; Proxy: null)
The user trying to use the KMS Key ID should have the right permissions to access (encrypt/decrypt) using the AWS KMS Key used via fs.s3a.encryption.key. If not, then add permission(or IAM role) in “Key users” section by selecting the AWS-KMS CMK Key on AWS console.

Number of parts in multipart upload exceeded

org.apache.hadoop.fs.PathIOException: `test/testMultiPartUploadFailure': Number of parts in multipart upload exceeded. Current part count = X, Part count limit = Y
    at org.apache.hadoop.fs.s3a.WriteOperationHelper.newUploadPartRequest(WriteOperationHelper.java:432)
    at org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.uploadBlockAsync(S3ABlockOutputStream.java:627)
    at org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.access$000(S3ABlockOutputStream.java:532)
    at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.uploadCurrentBlock(S3ABlockOutputStream.java:316)
    at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.write(S3ABlockOutputStream.java:301)
This is a known issue where upload fails if number of parts is more than 10000 (specified by aws SDK). You can configure fs.s3a.multipart.size to reduce the number of parts.

The bucket does not exist.

org.apache.hadoop.fs.s3a.UnknownStoreException: `s3a://random-bucket-7d9217b0-b426-4344-82ea-25d6cbb316f1/':
        Bucket does not exist: software.amazon.awssdk.services.s3.model.NoSuchBucketException: null
    (Service: S3, Status Code: 404, Request ID: RD254TC8EVDV98AK,
    Extended Request ID: 49F5CO1IKavFsz+VBecf2uwZeNVar3InHkdIrONvAK5yQ73gqZ1hFoAEMo8/x5wRNe3OXO3aebvZkev2bS81kw==)
    (Service: S3, Status Code: 404, Request ID: RD254TC8EVDV98AK): null
    (Service: S3, Status Code: 404, Request ID: RD254TC8EVDV98AK, Extended Request ID: 49F5CO1IKavFsz+VBecf2uwZeNVar3InHkdIrONvAK5yQ73gqZ1hFoAEMo8/x5wRNe3OXO3aebvZkev2bS81kw==)
    (Service: S3, Status Code: 404, Request ID: RD254TC8EVDV98AK)
Check the URI is correct, and that the bucket actually exists.




    

If using a third-party store, verify that you’ve configured the client to talk to the specific server in fs.s3a.endpoint. Forgetting to update this value and asking the AWS S3 endpoint for a bucket is not an unusual occurrence.
This can surface during filesystem API calls if the bucket is deleted while you are using it, -or the startup check for bucket existence has been disabled by setting fs.s3a.bucket.probe to 0.

The option fs.s3a.metadatastore.impl or the per-bucket version has a value of org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore

Something has happened to the data as it was uploaded.

Caused by: org.apache.hadoop.fs.s3a.AWSClientIOException: saving output on dest/_task_tmp.-ext-10000/_tmp.000000_0:
    com.amazonaws.AmazonClientException: Unable to verify integrity of data upload.
    Client calculated content hash (contentMD5: L75PalQk0CIhTp04MStVOA== in base 64)
    didn't match hash (etag: 37ace01f2c383d6b9b3490933c83bb0f in hex) calculated by Amazon S3.
    You may need to delete the data stored in Amazon S3.
    (metadata.contentMD5: L75PalQk0CIhTp04MStVOA==, md5DigestStream: null,
    bucketName: ext2, key: dest/_task_tmp.-ext-10000/_tmp.000000_0):
  at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:144)
  at org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:121)
  at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
  at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
  at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.close(HiveIgnoreKeyTextOutputFormat.java:99)
  at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:190)
  ... 22 more
Caused by: com.amazonaws.AmazonClientException: Unable to verify integrity of data upload.
  Client calculated content hash (contentMD5: L75PalQk0CIhTp04MStVOA== in base 64)
  didn't match hash (etag: 37ace01f2c383d6b9b3490933c83bb0f in hex) calculated by Amazon S3.
  You may need to delete the data stored in Amazon S3.
  (metadata.contentMD5: L75PalQk0CIhTp04MStVOA==, md5DigestStream: null,
  bucketName: ext2, key: dest/_task_tmp.-ext-10000/_tmp.000000_0)
  at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1492)
  at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131)
  at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123)
  at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139)
  at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47)
  ... 4 more
As it uploads data to S3, the AWS SDK builds up an MD5 checksum of what was PUT/POSTed. When S3 returns the checksum of the uploaded data, that is compared with the local checksum. If there is a mismatch, this error is reported.
The uploaded data is already on S3 and will stay there, though if this happens during a multipart upload, it may not be visible (but still billed: clean up your multipart uploads via the hadoop s3guard uploads command).
Possible causes for this
A (possibly transient) network problem, including hardware faults.
A proxy server is doing bad things to the data.
Some signing problem, especially with third-party S3-compatible object stores.
This is a very, very rare occurrence.

Reads work, but writes, even mkdir, fail:

This has been seen with third party stores.

If the store is configured to require content-MD5 headers with data uploaded: disable it.

If the store requires the use of the v2 signing algorithm, know that it is unsupported on this release.

A multipart upload was trying to complete, but failed as there was no upload with that ID.

java.io.FileNotFoundException: Completing multi-part upload on fork-5/test/multipart/1c397ca6-9dfb-4ac1-9cf7-db666673246b:
 S3Exception: The specified upload does not exist.
  The upload ID may be invalid, or the upload may have been aborted or completed.
   (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload;
This can happen when all outstanding uploads have been aborted, including the active ones.
If the bucket has a lifecycle policy of deleting multipart uploads, make sure that the expiry time of the deletion is greater than that required for all open writes to complete the write, and for all jobs using the S3A committers to commit their work.

The pool of https client connections and/or IO threads have been used up, and none are being freed.

The AWS SDK and the Apache S3 components can be configured to log at more detail, as can S3A itself.

The S3A client can ba configured to retry those operations which are considered retryable. That can be because they are idempotent, or because the failure happened before the request was processed by S3.

The number of retries and interval between each retry can be configured:

<property>
  <name>fs.s3a.retry.limit</name>
  <value>7</value>
  <description>
    Number of times to retry any repeatable S3 client request on failure,
    excluding throttling requests.
  </description>
</property>
<property>
  <name>fs.s3a.retry.interval</name>
  <value>500ms</value>
  <description>
    Initial retry interval when retrying operations for any reason other
    than S3 throttle errors.
  </description>
</property>
Not all failures are retried in the S3A Code -though some may be retried within the AWS SDK. Specifically excluded are those considered unrecoverable:
Low-level networking: UnknownHostException, NoRouteToHostException.
302 redirects.
Missing resources, 404/FileNotFoundException.
HTTP 416 response/EOFException. This can surface if the length of a file changes while another client is reading it.
Failures during execution or result processing of non-idempotent operations where it is considered likely that the operation has already taken place.
In future, others may be added to this list.
When one of these failures arises in the S3/S3A client, the retry mechanism is bypassed and the operation will fail.
Warning: the S3A client considers DELETE, PUT and COPY operations to be idempotent, and will retry them on failure. These are only really idempotent if no other client is attempting to manipulate the same objects, such as: renaming() the directory tree or uploading files to the same location. Please don’t do that. Given that the emulated directory rename and delete operations are not atomic, even without retries, multiple S3 clients working with the same paths can interfere with each other

It is possible to configure a global timeout for AWS service calls using following property:

<property>
  <name>fs.s3a.connection.request.timeout</name>
  <value>5m</value>
  <description>
    Time out on HTTP requests to the AWS service; 0 means no timeout.
    Measured in seconds; the usual time suffixes are all supported
    Important: this is the maximum duration of any AWS service call,
    including upload and copy operations. If non-zero, it must be larger
    than the time to upload multi-megabyte blocks to S3 from the client,
    and to rename many-GB files. Use with care.
    Values that are larger than Integer.MAX_VALUE milliseconds are
    converged to Integer.MAX_VALUE milliseconds
  </description>
</property>
If this value is configured too low, user may encounter SdkClientExceptions due to many requests timing-out.
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request:
  Request did not complete before the request timeout configuration.:
  Unable to execute HTTP request: Request did not complete before the request timeout configuration.
  at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:205)
  at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:112)
  at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:315)
  at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
  at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:311)
When this happens, try to set fs.s3a.connection.request.timeout to a larger value or disable it completely by setting it to 0.

There are some switches which can be set to enable/disable features and assist in isolating problems and at least make them “go away”.