Object Storage

What is it? #

Object storage keeps files as objects in buckets, retrieved by key over HTTP. S3, Cloud Storage, Azure Blob and compatible services all work this way.

The reason not to use your server's disk is simple: the disk belongs to one machine. Add a second server and half your uploads are missing. Replace the machine and they are gone.

Object storage is effectively unlimited, durable, and accessible from anywhere, which makes servers disposable again.

It is not a filesystem. There are no real directories, you cannot edit part of an object, and listing is slower than you expect. Objects are written and read whole.

Think of it like this #

A left-luggage service rather than a locker in your own office. You hand over an item and get a ticket; you can collect it from any branch, and it survives your office moving.

Keeping it in your own office works until you have two offices, or you move.

Simple example #

Users upload profile photos and export reports. Storing them on the application server breaks the moment you run two servers, and every deployment risks losing them.

Code #

BAD                                  GOOD

browser ──▶ app server               browser ──presigned PUT──▶ object storage
              │ writes to disk                      │
              ▼                                     ▼
        local /uploads folder              app stores only the key
        (lost on redeploy,                 CDN serves reads
         invisible to other servers)

import boto3
from botocore.config import Config

s3 = boto3.client("s3", config=Config(signature_version="s3v4"))
BUCKET = "app-user-uploads"


def upload_url(user_id: int, filename: str, content_type: str) -> dict:
    """Give the browser a short-lived URL so the file never touches our server."""
    key = f"avatars/{user_id}/{filename}"
    url = s3.generate_presigned_url(
        "put_object",
        Params={
            "Bucket": BUCKET,
            "Key": key,
            "ContentType": content_type,
        },
        ExpiresIn=300,                      # 5 minutes
    )
    return {"upload_url": url, "key": key}


def download_url(key: str) -> str:
    """Short-lived read URL for private objects."""
    return s3.generate_presigned_url(
        "get_object", Params={"Bucket": BUCKET, "Key": key}, ExpiresIn=600
    )

Practices that matter

private by default      buckets public by accident are a classic data leak
presigned URLs          uploads and downloads bypass your servers entirely
content-addressed keys  avatars/7/a1b2c3.jpg — new content, new key, cacheable
lifecycle rules         move old objects to cheaper storage, delete temporary ones
versioning              recover from an accidental overwrite or delete
CDN in front            reads served from the edge, not from the bucket
validate on the server  check type and size before issuing the presigned URL

How it works #

A presigned URL is a normal URL carrying a signature that grants one specific operation on one specific key for a limited time. The browser uploads directly to storage, so the file never passes through your application.

That removes a real bottleneck. A hundred simultaneous 50 MB uploads would otherwise consume your application's bandwidth, memory and worker processes.

Your application still controls access, because it decides whether to issue the URL. Validation — file type, size limit, whether this user may upload here — happens before the URL is created.

The key acts as a path but is really just a string. Including a hash of the content means a changed file gets a new key, which makes long cache lifetimes safe, exactly as with versioned static assets.

Lifecycle rules handle cost. Temporary exports can be deleted after seven days; old files can move to colder, cheaper storage automatically.

Versioning protects against overwrite and deletion mistakes, and it is worth enabling on anything irreplaceable.

Putting a CDN in front means reads are served from the edge, which is both faster and cheaper than serving every request from the bucket.

Real-world use #

Every application that accepts uploads uses object storage: profile images, documents, video, backups, generated reports, static site assets and data lake files.

It is also what makes servers disposable. With no local state, a server can be destroyed and replaced freely, which is the foundation of autoscaling and immutable deployments.

The common security failure is a public bucket. It has caused many well-publicised data leaks, and the fix is to keep buckets private and use presigned URLs for everything.

Cost has a shape worth knowing: storage is cheap, requests are cheap, and egress bandwidth is often the expensive part — which is another reason to put a CDN in front.

For processing, the usual pattern is upload to storage, then enqueue a job that reads the object, processes it, and writes the result back under a new key.

Common mistakes #

• Writing uploads to the application server’s local disk.
• Leaving a bucket publicly readable when it holds user data.
• Proxying large uploads and downloads through your application.
• Predictable keys for private files, so one user can guess another’s.
• No lifecycle rules, so temporary files accumulate indefinitely.

Practice #

Design the upload flow for user documents: which validation happens on your server, what the presigned URL grants and for how long, what the key looks like, and how downloads are authorised. Then list two lifecycle rules you would set.