Obnam → blog → 2026 → Obnam: upload chunks

Goal

  • Last time I ran out of time and didn't upload chunks with file content data. Today, I'll add that.

  • I'll also upload the SQLite database that has the metadata for files in a backup.

Plan

  • When backing up a file, upload its content as a chunk.

  • Add a type for splitting file content into chunks using the simplest possible chunking method, but with an API that allows other ways to split data into chunks. Change the part that backs up a file to use the new type.

  • Change the part that updates the backup files metadata database to chunk an upload the database file.

Notes

  • The code from last time still builds and tests pass. Good.

  • When I started implementing this, I decided to implement a rudimentary chunk splitter first. This will give me more feedback on the API. The first splitter method is just "put everything into one chunk".

  • To exercise the API, I'll add a command obnam split list, which reads a file and lists the chunks a split method contains. I don't wand to shove this under obnam chunk, at least for now.

    Did that. The command may become more useful when I start experimenting with different ways to split files into chunks. It'd help find good ways for specific data, if nothing else.

  • As I'm implementing this, I see a lot of opportunity for tidying up the codebase by introducing helpful abstractions. Looking forward to that refactoring. I have near-duplicate code all over the place. This has been fine as I've explored what the software needs to do, but now that things are coalescing, it's time to tidy up.

  • I'm happily adding more things to tidy up, too.

  • Implemented uploading of file data chunks and files db. I won't make automated tests today, as I have no time, but my manual testing indicates things work. It's a little limited by there being no way to list backups, or restoring one. That's for later.

  • There's no deduplication, either. Also for later.

  • Merged today's changes.

Summary

  • Obnam can now back up data, including the database of metadata for files. This is a big step, even if there's no way to restore the data yet.

Support?

If you'd like to fund Obnam development, see my funding page. My high level goal is described on the architecture page. What is most important about backup software to you?