Video Notes

Despite deleting a large amount of files on one of my Amazon S3 buckets, I noticed the bucket size was still sitting high at 230gb.

I checked the size over a period of time, because I had read that sometimes the metrics is delayed 24-48 hours, but even after several days, the size was not dropping.

Upon digging into it, I discovered I had versioning enabled for this bucket so even though I had deleted many files, older versions of those files were still hanging around.

As evidence, I ran the following aws s3api list-objects-v2 command to list objects currently in the bucket:

aws s3api list-objects-v2 \
  --bucket YOUR_BUCKET \
  --output json \
  --query "Contents[].Key" | jq length

This yielded 1683.

I then ran the following command to list noncurrent versions of objects currently in the bucket:

aws s3api list-object-versions \
  --bucket YOUR_BUCKET \
  --query "length(Versions[?IsLatest==\`false\`])"

This yielded a much higher number: 155545.

So there was my problem: Even though I had cleaaned up the bucket removing many files, S3’s versioning system still had noncurrent versions of those files hanging around.

To address this I had to complete the following steps:

1. Disable versioning
2. Create a lifecycle rule to delete existing noncurrent versions

Step 1) Disable Versioning

Do this under Properties > Bucket Versioning.

Note that this only applies to new objects added to this bucket, so the next step is necessary to delete any existing noncurrent versions.

Step 2) Create Lifecycle Rule

Under Management, create a new Lifecycle rule with the following details:

• Name: expire-noncurrent-versions
• ✓ Apply to all objects in the bucket
• ✓ Permanently delete noncurrent versions of objects
• ✓ Delete expired object delete markers or incomplete multipart uploads
• Days after objects become noncurrent: 1

Step 3) Wait

The lifecycle rules run once every 24 hours, and the first time this rule runs it will mark concurrent items as expired, then the second time it runs it will delete those expired items. Because of this, you’ll need to wait 48-36 hours for everything to process.

Appendix: aws s3api

Here’s a quick-start guide to getting started with aws s3api:

1) Install AWS CLI

Follow the instructions here to install the AWS CLI...

Verify:

aws --version

Configure credentials

The simplest setup:

aws configure

It will ask for:

• AWS Access Key ID
• AWS Secret Access Key
• Default region (example: us-east-1)
• Default output format (json is a good default)

This writes to:

• ~/.aws/credentials
• ~/.aws/config

If you use SSO:

aws configure sso
aws sso login

Confirm you’re authenticated:

aws sts get-caller-identity

Quick starter commands

List buckets:

aws s3api list-buckets

List objects in a bucket:

aws s3api list-objects-v2 --bucket YOUR_BUCKET --max-keys 10

Check a specific object’s metadata:

aws s3api head-object --bucket YOUR_BUCKET --key path/to/file.csv