Goal

Today I want to start addressing the mess of how chunks are implemented in Obnam. They're not entirely consistent, and using them is hard.

Plan

Write down what I actually want from chunks, and sketch Rust types to support that. Then start implementing.

Notes

I'm going to first think about chunks from first principles. This is going to repeat much of what I've written previously, but writing it all again will help me think clearly.
So far I've implemented chunks mostly from the bottom up, albeit with a long term top-down vision of what they should do. I've also developed Obnam in sessions of about three hours at a time, once a week. This has meant that I've thought of fairly small parts of the problem, from narrow viewpoints, at a time. I've copy-pasted code freely to avoid having to think hard about abstractions and types.

None of this is bad, as such, but it has resulted in much code that's similar in many places. I've learned when trying to refactor this that while copy-pasted code is similar, it's also different enough to be hard to abstract well. There's unexpected internal dependencies in some places that are hard to capture in an abstraction meant to be generic, but easy to use.

I care about how easy to use, i.e., ergonomic, code is to use, because hard-to-use code is more likely to be buggy. For security sensitive code, such as anything to do with chunks in Obnam, buggy means insecure. Thus, I'm willing to take some time to rethink how chunks are implemented, to build a strong foundation for higher layers of a backup program.
The high-level architectural vision of chunks is fine, I think: they are atomic units of data, encrypted, with a unique random identifier and an arbitrary string label.
There are several kinds of chunks. At this stage, I know of the following:
- data chunk containing unstructured data, such as contents of a file
- client chunk containing metadata about a backup client and the list of chunk encryption keys for that client
- backup chunk containing metadata about one backup run
- credential chunk containing the encryption key for the client chunk, encrypted in some way
Of these, all but the credential chunks can be encrypted and serialized in the same way. I've chosen to use GCM-SIV AEAD encryption, which I'm still happy with. This is a symmetric encryption that also verifies during decryption that the data hasn't been modified. The encrypted message may contain "additional data" that is not encrypted, which I use for chunk metadata. This allows the client to easily verify that what it gets from the repository has the right metadata.
Credential chunks have to be encrypted in another way. They contain the key to encrypt the client chunk, which I call the client key. The client key is encrypted in some other way, such as using OpenPGP. This means the client chunk is structurally different from all other chunks.
My goal is to allow Obnam users to use a variety of different kinds of credential encryption methods. This will allow more flexibility to set up access to one's backups, balancing security and convenience in a way that suits each person's situation.
To make the chunk API in Rust be convenient to use, I want all kinds of chunks to be used in the same way. If variations are necessary, I want to keep that minimal.
The on-disk format for encrypted chunks needs to flexible to change, and robust against corruption, without compromising on security. So far, I'm thinking as follows:
- There is a magic cookie header to identify the version and encoding of the rest of the chunk. This enables evolving the chunk format over time.
- In the first version, the rest is encoded using the postcard crate. That's a very efficient format that should be stable in the long term.
- The chunk metadata is included outside the encrypted message, so that the backup repository can extract it. The metadata is treated as suspicious by the client until it has decrypted the chunk, and thus authenticated the metadata.
- The encryption method is encoded so that Obnam can use the correct method to decrypt. Technically, this leaks some information, because it means an attacker can more easily identify credential chunks. To avoid that, Obnam would need to try every decryption method is has, and that's quite inefficient. Further, many credential encryption methods produce structures that can be identified anyway, e.g., OpenPGP packets. I am, at least for now, willing to take this compromise.

Based on the above, a Rust type for encrypted chunks (apart from the magic cookie) might look like this:

struct EncryptedChunk {
  metadata: Suspicious(Metadata),
  payload: EncryptedPayload,
}

struct Suspicious<T> {
  value: T,
}

enum EncryptedPayload {
  Credential(Vec<u8>),
  Other(Vec<u8>),
}

Here, Suspicious is a type to mark data as suspicious. It will make it a little easier to avoid using such data wrongly.
The encrypted payload captures the distinction between credentials versus other chunks. In principle, I could encode the type of credential as well, e.g., OpenPGP with software keys versus hardware backed keys, but I'm not sure that's worth it. It'd still miss the information of what key to use, and that's best not encoded in a way an attacker can get at. I'll instead make the client try every credential key and method it has. That's less than optimally efficient, but it only needs to be done once per run, to open the client key. I think that's a reasonable compromise.
I could make the EncryptedPayload enum encode a more detailed payload type: is it a client chunk or a data chunk? However, as that isn't needed for the decryption, it would leak information unnecessarily. Instead, I'll encode that in an enum that isn't exposed in the encrypted data:
```
enum Payload {
  UncompressedData(Vec<u8>),
  DeflatedData(Vec<u8>),
  Client(Client),
  Backup(Backup),
}
```
This will get more variants for different kinds of compression. The postcard crate will handle that just fine.

For methods for handling encrypted chunks, I'm starting with the following:

impl EncryptedPayload {
  fn credential(metadata: Metadata, ciphertext: Vec<u8>) -> Self {}
  fn other(metadata: Metadata, ciphertext: Vec<u8>) -> Self {}
  fn suspicious_label(&self) -> &Label {}
  fn ciphertext(&self) -> &[u8] {}
}

Creating an encrypted chunk requires first encrypting the payload, and that will happen differently based on the kind of chunk. Likewise for decrypting. When decrypting the payload you get a value of a type representing the value. There will be one value for credential, and the Payload value above. It would probably be good to have methods for each type of expected type so that the caller doesn't need to match on an enum and check they got what they expected.
```
impl EncryptedPayload {
  fn decrypt_credential(&self, method: &CredentialMethod) -> Result<Credential, ChunkError> {}
  fn decrypt_data(&self, engine: &Engine) -> Result<Vec<u8>, ChunkError> {}
  fn decrypt_client(&self, engine: &Engine) -> Result<Client, ChunkError> {}
  fn decrypt_backup(&self, engine: &Engine) -> Result<Backup, ChunkError> {}
}
```

Implementation

My implementation plan is to start a new branch and a new module, obnam::encrypted_chunk, and implement things there. I'll do this TDD style and try to test everything extensively in unit tests. Once that works, I'll branch off that again and start converting the rest of the codebase to use the new module. This means implementing bottom up.
This also means that as I need to make changes to types like Credential that are used by other parts of the codebase, I'll soft-fork them into my module. This way I don't break anything outside that module.
I'll happily use types I've already implemented, though, when they don't need changes.
I skimped on the Suyspicious type, at least for now. Also not introducing a type for ciphertext, for now.
I'm going to be serializing with postcard both the unencrypted payload and the full encrypted chunk. From some informal benchmarking, postcard is fast enough that this doesn't matter. So I'm not going to worry about that.
OK, implemented uncompressed data chunks, from payload to encrypted chunks. It's very quickly evident that there's many steps to do a round trip. I'll need to think about how to simplify this without compromising on what can be done.
Thought about this while having lunch. I think the best route will be to add methods to EncryptedChunk rather than having caller construct EncrryptedPayload directly. However, I will first add credentials, to make sure I have that working.
Credential is one of the types I need to soft-fork. The new version of the type will need to store the client key in plaintext, not encrypted like the old version. I could manage without a credential type at all, but I'll keep it, at least for now.
Since we're not using AEAD for credential chunks, there's no way to authenticate that the client key is valid. But if it isn't, it can't decrypt the client chunk, so that'll do, I think.
I changed my mind, I'll drop the soft-forked Credential type and just use Key for client key instead.
Moved the new Backup type for backup metadata to its own module. Nothing uses yet, so it doesn't break anything.
OK, time to start converting the codebase to use the new encrypted chunk type. Started a new branch for this.
The Metadata type, and its friends, is in the chunk.rs module, for no good reason. Moved to its own module.
Ran out of time today, will continue tomorrow.
It's tomorrow.
The supposedly helpful ClientRepository is entirely riddled with the old chunk type. I'm not sure how helpful it actually is. It has methods for managing different types of chunks, and that's no longer as helpful. If I change the methods to use the new encrypted chunk type, most of the methods become useless. What remains is other methods, especially client_key to find the client key from credential chunks, and open_client to decrypt the client chunk.
I dropped the now-pointless helper methods and fixed the rest of the module.
Fixed the rest of the code base to use new encrypted chunk. Tests pass.
Started a rebase branch (so it's easy to undo any mistakes I make while rebasing), to tidy up the history of the changes. I tend to commit very often, but there's no point in making a branch with tens of tiny commits.
Merged.

Summary

After an intense day and a half of weekend hacking, I have quite significantly simplified how chunks and their encryption are handled.

Support?

If you'd like to fund Obnam development, see my funding page. My high level goal is described on the architecture page. What is most important about backup software to you?