Goal

• Today I'll start the journey towards backing up sets of files and restoring them. It will be a long journey, but I have hopefully built a bicycle to make it nicer than having to walk all the way.

• Specifically, today I want to create a "files database" to contain all the files and their metadata, and the IDs of chunks for their content. The file content will be backed up as chunks. The files database will not itself be backed up yet.

• I'm going to take small steps to avoid having to spend multiple weeks to complete a step. This is important for me to keep the project fun.

• Performance is not a goal today. Neither are incremental backups, where only new or modified files are backed up.

Plan

I intend to implement the following commands today:

• obnam backup start

  • start a new files database
  • there can only be one at a time

• obnam backup add FILE

  • back up a file and add it to the current files database
  • if it's a regular file, back up its contents as a data chunk
  • if it's a directory, do NOT add any files in the directory, only the directory itself
  • add basic filesystem metadata to the files database: fully qualified pathname, mode bits

• obnam backup list

  • list contents of current files database

• obnam backup finish

  • remove the current files database

Notes

• I am intentionally not implementing a restore today, unless I have plenty of extra time.

• The files database will not itself be backed up. This means today will not result in a useful backup system. However, it'll be an important step towards that. Backing up the files database will require a new kind of chunk, one that represents a backup. Adding a new chunk type is likely to be a lot of work, so I'll do that another time.

• I'll be using SQLite for the files database rather than inventing my own file format. I've experimented with SQLite before and it's a good match for this purpose. I'll use the rusqlite crate for this. I've used it before and been happy with it.

• The "current files database" will be stored in the "project state directory", in XDG base directory parlance. This is ~/.local/state/obnam by default, but can be overridden with the usual XDG environment variables.

• I need a way to represent a file in the database. At this stage it will need to store the fully qualified pathname, file system metadata, and list of chunks. I was going to use std::fs::Metadata, but the modified method is fallible, and it'll be awkward to handle errors all the time. So I'll define my own type that'll have only the metadata I need. That'll also make it easier to serialize for the database.

• Running out of time, skipping the file data chunk uploading.

• Otherwise, pretty straightforward today thanks to constrained goal setting.

• Even though this is not functional software, I'll merge the changes.

Summary

• I have the beginning of making a backup, with merely the actual backup content missing. I'll finish that next time.

Support?

If you'd like to fund Obnam development, see my funding page. My high level goal is described on the architecture page. What is most important about backup software to you?