Obnam → blog → 2021 → 04 → 25 → Iteration planning: April 25—May 9
  1. Assessment of the iteration that has ended
  2. Discussion
  3. Goals
  4. Commitments for this iteration
  5. Meeting participants

Assessment of the iteration that has ended

The goal for the previous iteration was:

The main goal of this iteration is to add at least rudimentary encryption of chunks, before they're uploaded to the server, and decryption and validation after they're downloaded. This should use the encryption keys stored by obnam init. This is done only if the client configuration says encryption is turned on, to allow an opt-in approach to encryption for now. Later on, encryption won't be optional.

At the same time, work needs to start on using more concurrency in the client, and that means that Lars needs to learn more about Rust, specifically async Rust.

Additionally, some smaller issues will be worked on, to tackle the back log of open issues.

That goal was not reached. Lars didn't get obnam#28 (iterators instead of large vectors), obnam#109 (release), or obnam#110 (chunk encryption) done, partly due to life and work keeping him busy, and partly because attempting to do obnam#28 took up almost all the time he'd allocated for Obnam. The issues Lars did work on have outstanding merge requests, because of the multi-day review period process being used. Further, Lars learned a bit about async Rust and has a plan for concurrency in the client.

Those issues will be carried to the next iteration.

Discussion

Concurrency in the client

Lars experimented with async Rust, using tokio, on a branch of the Summain program. His conclusion is that async will work well for those parts of programs that do I/O or other blocking operations that allow switching between async tasks. Such tasks are executed in tokio's worker threads. For CPU intensive tasks, tokio's blocking tasks, executed in its blocking threads, work fine.

Blocking tasks are necessary to use all the available CPU: tokio only schedules normal tasks in a co-operative way, limiting the amount of CPU cycles used for CPU intensive computation without inhibiting progress of other tasks. Blocking tasks run in their own threads, allowing more CPU to be used without interfering with normal async task.

Lars was able to make a version of Summain where all the metadata gathering of files is done in async tasks, but the checksum computations are done in blocking tasks. This allowed all the CPU on the test host used for checksum computation. The resulting program was reasonably clear and obvious, a definite plus.

For Obnam, Lars proposes the following: the Obnam client will execute in four phases:

  1. Download the latest generation from server. Until this is done, nothing can be decided about whether a file needs to be backed up or not. If there are no generations (initial backup), invent a dummy, empty generation to be used instead.

  2. Scan the file system to files to be backed up, excluding based on client configuration (e.g., CACHEDIR.TAG), and checking against the previous generation. Insert metadata of any not-excluded files into a new nascent generation. For each inserted file, mark it as unchanged, changed, or new.

  3. Scan the nascent generation for regular files that are changed or new. For each such file, split it into chunks, compute the checksum for the chunk, and look up the checksum on the server or upload the chunk and insert the chunk id into the nascent generation.

  4. Upload the nascent generation to the server.

Of these, phases 1 and 4 are synchronous, in that phase 1 must finish before 2 can start, and phase 3 must finish before 4 can start; however, otherwise they can use async Rust. Phases 2 and 3 are fully async, and can be interleaved. The checksum computation part of phase 3 is a blocking task, to allow all the available CPU used for it.

Iteration length

This was a one-week iteration. In principle it went OK, but Lars failed to get much of the issues resolved he'd taken on. A longer iteration would allow more flexibility, but can be problematic in other ways.

A bigger issue is that if a change needs three days of review, by default, a one-week iteration means all the work would need to be done very early in the iteration, or else not enough time to get changes reviewed.

For now, Lars suggests going back to two-week iterations.

This iteration

This iteration covers April 25 though May 9. To allow participation and feedback of the meeting plan, the meeting won't be closed until Tuesday, April 27.

Goals

Goal for 1.0 (not changed this iteration)

The goal for version 1.0 is for Obnam to be an utterly boring backup solution for Linux command line users. It should just work, be performant, secure, and well-documented.

It is not a goal for version 1.0 to have been ported to other operating systems, but if there are volunteers to do that, and to commit to supporting their port, ports will be welcome.

Other user interfaces is likely to happen only after 1.0.

The server component will support multiple clients in a way that doesn’t let them see each other’s data. It is not a goal for clients to be able to share data, even if the clients trust each other.

Goal for the next few iterations (not changed for this iteration)

The goal for next few iterations is to have Obnam support encryption well. This will involve having a documented threat model, which has been reviewed by all stakeholders participating in the project, and Obnam defending against all the modeled threats.

Goal for the iteration that is starting

The main goal of this iteration is to add at least rudimentary encryption of chunks, before they’re uploaded to the server, and decryption and validation after they’re downloaded. This should use the encryption keys stored by obnam init. This is done only if the client configuration says encryption is turned on, to allow an opt-in approach to encryption for now. Later on, encryption won’t be optional.

Additionally, work will continue on using iterators instead of potentially enormous vectors when querying the SQLite database.

Commitments for this iteration

New milestone 9 represents this iteration on GitLab.

This is a two-week iteration. For Lars, that means a time budget of 8 hours. Lars is committed to resolving the following issues:

That is a total of 8 hours, rough estimate. Additionally, finishing off the merge requests from the April 18—25 iteration.

Meeting participants

  • Lars Wirzenius