Video Notes

After deleting a large number of files from one of my Amazon S3 buckets, I expected the storage size to drop significantly. Instead, the bucket was still sitting at ~230GB.

I checked the size over a period of time, because I had read that sometimes the metrics is delayed 24-48 hours, but even after several days, the size was not dropping.

After digging deeper, I discovered that bucket versioning was enabled. This meant that even though I had deleted many files, S3 was still retaining older versions of those objects. Deleting the current version does not automatically remove previous versions.

To confirm this, I ran the following aws s3api list-objects-v2 command to list the current objects in the bucket:

aws s3api list-objects-v2 \
  --bucket YOUR_BUCKET \
  --output json \
  --query "Contents[].Key" | jq length

This returned 1,683, so only 1,683 current objects were present.

Next, I checked how many noncurrent versions were still stored:

aws s3api list-object-versions \
  --bucket YOUR_BUCKET \
  --query "length(Versions[?IsLatest==\`false\`])"

This yielded a much higher number: 155545.

So there was the issue: Even though I had cleaned up the bucket, S3 was still storing 155,545 noncurrent object versions.

The Fix

To resolve this, I needed to:

1. Disable versioning
2. Create a lifecycle rule to delete existing noncurrent versions

Step 1) Disable Versioning

Navigate to Properties > Bucket Versioning and disable versioning.

Important: Disabling versioning only affects new objects going forward. It does not remove existing noncurrent versions. That’s why the next step is necessary.

Step 2) Create a Lifecycle Rule

Under Management, create a new Lifecycle rule with the following details:

• Name: expire-noncurrent-versions
• ✓ Apply to all objects in the bucket
• ✓ Permanently delete noncurrent versions of objects
• ✓ Delete expired object delete markers or incomplete multipart uploads
• Days after objects become noncurrent: 1

This rule instructs S3 to automatically remove older versions of objects.

Step 3) Wait

Lifecycle rules run once every 24 hours.

On the first run, objects are marked as expired. On the second run, expired objects are deleted.

Because of this two-step process, you should expect to wait approximately 36–48 hours for storage usage to fully drop.

After the lifecycle rule ran, my bucket size finally decreased.

Appendix: Getting Started with aws s3api

If you’re new to aws s3api, here’s a quick setup guide.

1) Install AWS CLI

Follow the instructions here to install the AWS CLI...

Verify installation:

aws --version

Configure credentials

Run:

aws configure

It will ask for:

• AWS Access Key ID
• AWS Secret Access Key
• Default region (example: us-east-1)
• Default output format (json is a good default)

This writes to:

• ~/.aws/credentials
• ~/.aws/config

If you use SSO:

aws configure sso
aws sso login

Confirm you’re authenticated:

aws sts get-caller-identity

Quick starter commands

List buckets:

aws s3api list-buckets

List objects in a bucket:

aws s3api list-objects-v2 --bucket YOUR_BUCKET --max-keys 10

Check a specific object’s metadata:

aws s3api head-object --bucket YOUR_BUCKET --key path/to/file.csv