Design a File Upload System

What is it? #

A file upload system accepts files from users, stores them safely, processes them, and serves them back.

The naive version — post the file to your application, which writes it to disk — fails on several fronts at once: it does not scale, it loses files on redeploy, and it exposes you to malicious uploads.

The workable design has three separations: the upload goes directly to object storage, processing happens asynchronously, and serving happens through a CDN.

Security is not optional here. File upload is one of the most commonly exploited features in web applications.

Think of it like this #

A parcel drop-off point rather than handing every package to the shop assistant.

The assistant records that a parcel arrived and what it should contain, but the parcel itself goes straight into the secure store. Checking and sorting happens afterwards, not while the customer waits at the counter.

Simple example #

Users upload profile images, documents and videos up to 2 GB. Images need thumbnails, videos need transcoding, and everything needs virus scanning before it is served.

Code #

1. The flow

  client                  your API                 object storage         workers
    │  request upload       │                           │                    │
    ├──────────────────────▶│ validate type/size        │                    │
    │◀── presigned URL ─────┤ create pending record     │                    │
    │                       │                           │                    │
    ├── PUT file (direct) ──────────────────────────────▶│                    │
    │                       │◀── storage event / client confirm               │
    │                       ├── enqueue processing ─────────────────────────▶ │
    │                       │                           │◀── write outputs ───┤
    │◀── ready (webhook / poll) ────────────────────────────────────────────  │

# 2. Issue a presigned upload after validating the request
ALLOWED = {"image/jpeg": 10_000_000, "image/png": 10_000_000,
           "application/pdf": 25_000_000, "video/mp4": 2_000_000_000}


def request_upload(user, filename: str, content_type: str, size: int) -> dict:
    if content_type not in ALLOWED:
        raise ValueError("unsupported file type")
    if size > ALLOWED[content_type]:
        raise ValueError("file too large")
    if not quota.has_space(user.id, size):
        raise ValueError("storage quota exceeded")

    upload_id = new_id()
    key = f"uploads/{user.id}/{upload_id}/{safe_name(filename)}"

    uploads.insert(id=upload_id, user_id=user.id, key=key,
                   status="pending", declared_type=content_type, size=size)

    url = storage.presign_put(key, content_type=content_type,
                              max_size=size, expires_in=900)
    return {"upload_id": upload_id, "url": url}


# 3. Verify after the upload, never trust what the client declared
def on_upload_complete(upload_id: str):
    record = uploads.get(upload_id)
    head = storage.head(record.key)

    if head.size != record.size:
        return reject(record, "size mismatch")

    sniffed = detect_type_from_content(storage.read_first_bytes(record.key, 4096))
    if sniffed != record.declared_type:          # a .jpg that is actually a script
        return reject(record, "content type mismatch")

    uploads.update(upload_id, status="scanning")
    processing_queue.enqueue("scan_and_process", {"upload_id": upload_id})

4. Security rules that matter

validate content, not the extension    sniff the actual bytes
never execute uploaded files           serve from a separate domain, no scripts
strip metadata from images             EXIF contains GPS coordinates
random storage keys                    do not let users guess each other's files
scan for malware before serving        quarantine until the scan passes
set Content-Disposition                force download for risky types
limit size and rate                    per file, per user, per day

5. Large files

resumable uploads     split into chunks; the client retries only failed chunks
multipart to storage  object storage supports this natively
progress              the client tracks chunk completion
expiry                clean up abandoned multipart uploads after 24 hours

How it works #

Validation happens twice, and both matter. Before the upload, you check the declared type and size and issue a narrowly scoped URL. After the upload, you verify what actually arrived, because a client can declare anything.

Content sniffing is the important check. A file named photo.jpg with a image/jpeg header can contain anything, and extension-based validation has been bypassed this way for decades.

The presigned URL is limited by key, content type, size and expiry, so it cannot be reused to upload something else somewhere else.

The pending record exists from the start, so an upload that never completes is visible and cleanable rather than becoming an orphaned object.

Processing is queued. Thumbnailing, transcoding and scanning take seconds to minutes and must not block the user or hold a web worker.

Serving from a separate domain is a real security control. If uploaded content is served from your main domain and a user uploads an HTML file, any script in it runs with access to your site's cookies.

Resumable uploads matter on mobile networks. Without them, a 2 GB upload that fails at 95% starts again from zero.

Real-world use #

Every product with uploads converges on this shape, because proxying files through the application is a bottleneck you hit early.

Image pipelines usually generate several sizes on upload and serve them through a CDN with long cache lifetimes and content-hashed keys.

Video is a separate discipline: transcoding to multiple bitrates, generating streaming manifests, and handling jobs that take longer than the original video.

Storage cost management is ongoing: lifecycle rules for old files, deduplication by content hash, and deciding what happens to a deleted user's files.

The security list is not theoretical. Unrestricted file upload appears regularly in vulnerability reports, and the consequences range from stored cross-site scripting to remote code execution.

Common mistakes #

• Proxying uploads through your application, creating a bandwidth and memory bottleneck.
• Trusting the declared content type or the file extension.
• Serving user uploads from the main application domain.
• Predictable storage keys, letting users access each other’s files.
• No cleanup for abandoned or incomplete uploads.

Practice #

Design the upload flow for user documents shared within a team. Specify the validation before and after upload, the storage key format, how access is authorised on download, and two lifecycle rules. Then describe what happens when a virus scan fails.