Implementing S3 cross-region replication within the same account

In this recipe, we will learn to implement cross-region replication with S3 buckets. If cross-region replication is enabled for a bucket, the data in a bucket is asynchronously copied to a bucket in another region. Cross-region replication provides better durability for data and aids disaster recovery. Replicating data may be also done for compliance and better latency.

Getting ready

We need a working AWS account with the following resources configured:

• A user with administrator permission for S3 for a source bucket's account. I will be calling the user awssecadmin.
• Create two buckets, one each in two regions, with versioning enabled. I will be using the awsseccookbook bucket in the us-east-1 (N. Virginia) region and the awsseccookbookmumbai bucket in ap-south-1 (Mumbai).

How to do it...

We can enable cross-region replication from the S3 console as follows:

1. Go to the Management tab of your bucket and click on Replication.
2. Click on Add rule to add a rule for replication. Select Entire bucket. Use the defaults for the other options and click Next:

3. In the next screen, select the Destination bucket. Leave the other options as-is and click Next:

4. In the the Configure options screen, ask S3 to create the required IAM role, name your rule (by selecting the relevant option), and then click Next:

5. In the next screen, review the rule and click Save.
6. Upload an object to the source bucket and verify whether the object is replicated in the destination bucket. Also, verify that the value of the Replication Status field of the object in the object's Overview tab is COMPLETE once replication is completed.

The autogenerated role's permissions policy document and trust policy document are available with code files as replication-permissions-policy.json and assume-role-policy.json.

How it works...

In this recipe, we enabled cross-region replication within the same account. We replicated the entire bucket. We can also specify a subset of objects using a prefix or tags. We did not change the storage class of replicated objects in this recipe even though we can.

Here are the prerequisites for cross-region replication:

1. Source and destination buckets must be version-enabled and should be in different regions.
2. Replication can be done only to a single destination bucket.
3. S3 should have permission to replicate to a destination bucket.

We asked S3 to create the required role for replication. The autogenerated role has a permissions policy with s3:Get* and s3:ListBucket permissions on the source bucket, and s3:ReplicateObject, s3:ReplicateDelete, s3:ReplicateTags, and s3:GetObjectVersionTagging permissions on the destination bucket.

There's more...

The steps to enable cross-region replication from the CLI can be summarized as follows:

1. Create a role that can be assumed by S3, with a permissions policy with the s3:Get* and s3:ListBucket actions for the source bucket and objects, and the s3:ReplicateObject, s3:ReplicateDelete, s3:ReplicateTags, and s3:GetObjectVersionTagging actions for the destination bucket objects.
2. Create (or update) a replication configuration for the bucket using the aws s3api put-bucket-replication command providing a replication configuration JSON.

Complete CLI commands and policy JSON files are available with the code files.

Let's quickly go through some more details related to S3 cross-region replication:

• Cross-region replication is done via SSL.
• Only objects that were added after enabling cross-region replication are replicated.
• If the source bucket owner does not have read object or read ACL permission, objects are not replicated.
• By default, the source object's ACLs are replicated. However, changing ownership to the destination bucket owner can be configured.
• Objects with SSE-C encryption are not currently replicated.
• To replicate objects with SSE-KMS encryption, we need to provide one or more KMS keys as required for S3 to decrypt the objects. KMS requests related to S3 in the source and destination regions can cause us to exceed the KMS request limit for our account. We can request an increase in our KMS request limit from AWS.
• Since replication happens asynchronously, it might take some time (even up to hours for larger objects) to replicate.
• Sub-resource changes are not currently replicated. For example, automated life cycle management rules are not replicated. However, we can configure a change in the current storage class of the object during replication.
• We cannot replicate from a replica bucket.
• Deleting a version in the source bucket does not delete the version in the destination bucket. This adds additional protection to data. A delete marker was replicated with the old schema if DeleteMarkerReplication is enabled. However, the new schema does not support delete marker replication, which would prevent any delete actions from replicating.

See also

• You can read more about S3 cross-region replication here: https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.html.