Goal

• My goal for today is to fix things so that I can make a backup and restore it successfully. For today I care about the files and their content, but not other file metadata than the name.

Plan

• Add subcommand obnam backup show to output a JSON form of the backup chunk and its files database. The JSON should contain the list of chunks for each file.

• Change obnam backup add to store the list of chunks in the files database.

• Change obnam restore to restore file contents.

Notes

• Added scaffolding for new subcommand. The approach I've taken needs a bit of boilerplate, but I've not figured out a better way, yet. Unless I use macros, and I'm not sure I want that. Macros hide a lot of detail, and I don't want to hide this stuff.

• The guts of the new command were easy enough. I copy pasted most of the code from the restore command, with small tweaks. This is ugly, but I'll refactor later. As I found last time, I'm due some big refactoring anyway.

• In order to store a list of chunks with the contents of a file I'm today choosing to go with the simplest possibility: another field in the files table. I'll store a list of chunk IDs as JSON in a TEXT column.

• Later, I will want to do something smarter for this. Lists of chunks are going be many, and I'll want to design a way to store them efficiently. If two files share many chunks, and have some identical sequences of chunks, how do I store that well? I don't want to only rely on chunk deduplication and compression. Designing this will take a lot of thinking, more than I want to spend time on today.

• Changing the database schema is also a little tedious. I understand why ORMs are popular, but for Obnam, I really, deeply care about the details of the schema, so I want precise control over it. Tediouisly hand-written it is.

• Restoring file content was one more small step, but with copy-pasting.

• Merged these changes.

• That went faster than expected, so I have time to start working on the big refactor. I have a lot of so called technical debt in Obnam. For example, I've been lax at maintaining the test suite. Further, the subplot needs a complete rethink. The issue list is too long. But I'll start changing the UnlockedClient type to make the code for the backup and restore commands be ergonomic to write and have little code duplication.

• Something that stands out when I look at the code is that handling chunk keys for a client is currently messy. The software architecture allows, correctly, any number of chunk encryption keys, but there's no supporting infrastructure for managing which key to use for which chunk. I'll skip that problem, for today, but I'll introduce into UnlockedClient managing a cipher engine for each chunk key. The type already manages the engine for the client chunk. There is also a method to create the cipher engine, but it always creates a new one every time, which is wasteful. I expect there will eventually be a need for many chunk keys, and thus engines.

• Hmm, this ends up in the borrow checker pointing out ownership problems. Solving them will need deeper thinking, so I'll instead introduce a new type to cache the cipher engines. Actually, no, I'll punt on this until there's an actual need to manage many engines.

• A simple refactoring win is to add backup chunk methods to ClientRepository. Except this, too, turned out to not be simple. A backup chunk is not stored as is, it's encrypted and the result is stored as a data chunk. This makes everything complicated. I'm sure it's the wrong approach, but I'm running out of time for today.

Summary

• Backup and restore work, for what I want right now. File metadata is not backed up or restored, but that'll come later. The backup is very inconvenient to use, but fixing that will happen later.

• More worrying, I failed to make even small refactoring of the code to reduce code duplication and make common operations more ergonomic. I'll need to work on that next time.

Support?

If you'd like to fund Obnam development, see my funding page. My high level goal is described on the architecture page. What is most important about backup software to you?