01 / LUDER / FILE STORAGE ENGINE
Every file, verified before it is stored.
CHUNKED UPLOAD · RESUME-SAFE · TIKA DETECT · CLAMAV SCAN · AES-CBC · STREAMING
LUDER is a file storage engine built in Java and Spring Boot. It receives files in pieces, tracks every piece, assembles them, proves what each file really is, scans it, encrypts it at rest, and streams it back decrypted on demand.
- CHUNKED UPLOAD
- RESUME-SAFE
- TIKA DETECT
- CLAMAV SCAN
- AES-CBC
- STREAMING READ
100
MAX CHUNKS · declared per upload session
- 3
- DEFENSE LAYERS · detect · scan · encrypt
- AES-CBC
- AT REST · fresh 16-byte IV per file
- 4H
- SWEEPER · orphan sessions swept
02 / The pipeline
Five guarantees, before a byte is stored.
From chunk merge to encrypted final store — every file passes the same chain, in the same order, on the way in.
- 01
ASSEMBLE
FileChannel.transferTo merges chunks in index order
- 02
VERIFY
Tika detects the real content type · whitelist of nine MIME types
- 03
SCAN
ClamAV clamscan — exit 0 admits, exit 1 deletes
- 04
ENCRYPT
AES/CBC/PKCS5Padding with a fresh IV · plaintext deleted
- 05
STORE
metadata → Postgres · ciphertext → final store
- POST
/api/uploads/start
declare fileName · totalChunks · contentType → session + Redis init
- POST
/api/uploads/{uploadId}/chunk/{index}
multipart chunk → validated, written to the temp store
- GET
/api/uploads/{uploadId}/status
receivedChunks / totalChunks · served from the Redis cache
- POST
/api/uploads/{uploadId}/complete
gate received == total → assemble · verify · scan · encrypt
- GET
/api/files
paged file metadata · one-hour Redis cache
- GET
/api/files/download/{fileName}
stream + decrypt AES-CBC · attachment response
03 / System map
The engine, opened up.
Twelve nodes across the storage boundary. Select a node to trace how it connects to the rest of the engine.
04 / Security
Three walls between upload and store.
No uploaded byte is stored the way it arrived. LUDER proves, scans, and encrypts every file before it lands in the final store.
DETECT
Apache Tika
The content type is detected from the assembled bytes — not from whatever the client claimed — then matched against an allowlist of nine MIME types. Anything outside is rejected and deleted.
9 allowed MIME types
SCAN
ClamAV clamscan
Every assembled file is scanned by ClamAV before it is stored. Exit code 0 admits the file; exit code 1 deletes it as a threat; any other exit aborts the session.
exit 0 clean · exit 1 infected
ENCRYPT
AES/CBC/PKCS5Padding
Files are encrypted at rest with a fresh 16-byte IV written ahead of the ciphertext, and a key persisted out-of-band as a Base64 file. The plaintext is deleted after encryption.
AES-CBC · 16-byte IV · 8KB stream
MIME allowlist
09 types
- image/jpeg
- image/png
- image/gif
- video/mp4
- video/webm
- video/quicktime
- application/pdf
- application/json
- text/plain
05 / Storage
Two stores, tracked to the piece.
Chunks stage in a per-session temp directory; only an assembled, verified, encrypted file reaches the final store.
STAGING
Two stores, one intent
Chunks land in a per-session temp directory; only a fully assembled, verified, encrypted file reaches the final store. A session never writes final files incrementally.
RESUME
Redis tracks every piece
Received chunk indices live in a Redis set with a two-hour TTL, and progress in a matching hash. An interrupted upload can resume from exactly where it stopped.
SWEEP
Orphans do not linger
A scheduled task runs every four hours, cancels sessions idle over 110 minutes with no live Redis progress, and sweeps their temp directories.
RETRIEVAL
Streamed back, not loaded
Downloads open a streaming read that decrypts AES-CBC in 8KB buffers as the response streams — the full file is never held in memory.
disk layout · ./uploads
two-store- STAGING
- ./uploads/temp/{uploadId}/{index}
- FINAL
- ./uploads/fin/{uploadId}_{fileName}.enc
- · every 4h · cancels sessions idle over 110 minutes · sweeps temp dirs
- · streaming decrypt · 8KB buffer · attachment response
06 / Verification
Proven against the edge cases.
A test suite that chases the failure modes of a chunked file engine.
- SPLITchunk acceptancebounds 0 ≤ index < totalChunks · non-empty · no duplicate index16/16
- RESUMEinterrupted uploadmissed chunks re-sent · status reflects the received set · no double writes16/16
- TEMP-IOstaging I/Ochunks land per session · assembly reads in index order12/12
- ASSEMBLYchunk mergetransferTo merges in order · temp dir swept after completion6/6
- COMPLETE GATEsession completionreceived == total required · incomplete sessions rejected · state flips to COMPLETED8/8
- THREATmalware rejectionclamscan exit 1 → file deleted · session aborted · nothing stored6/6
07 / Foundations
Built as an engine, not a demo.
The decisions underneath a chunked storage pipeline.
CHUNKING
Resumable chunked upload
Files arrive as 1–100 client-declared chunks, each validated against the session before it is written to the temp store.
STATE
Session state in Redis
Progress and received indices live in Redis with a two-hour TTL — the engine knows exactly what is missing at any moment.
DEFENSE
Verify, scan, encrypt
Every file is detected by Tika, scanned by ClamAV, and encrypted with AES-CBC before it is committed to storage.
RETRIEVAL
Streamed, decrypted reads
Downloads decrypt in 8KB streaming buffers with the original MIME type restored and a Content-Disposition attachment.
08 / Technology
Everything the engine runs on.
The stack the storage pipeline actually runs on.
- Stack
- JAVA
- SPRING BOOT
- POSTGRESQL
- REDIS
- TIKA
- CLAMAV
- APACHE MAVEN
- JPA · HIBERNATE
- Layering
CLIENT → API → SERVICE → REDIS · POSTGRES · DISK
- Year
- 2025
- Role
- Backend Engineer
09 / Proof & open source
The engine, in the open.
The repository is the storage engine itself — controllers, services, the chunk pipeline, and the gates that guard it.