Annoucing Asuran - A deduplicating archiver for next generation technologies

announcements
1

I've been working on an archiver that is intended to combine, in my opinion, some of the best features of borg and restic.

The functionality is currently split into a library for interacting with archives, libasuran, and a CLI binary for using them, asuran.

This archiver does deduplication, encryption, compression, data verification, and (ideally) supports backing up to untrusted storage, including cloud storage (though currently it only supports a local storage backend).

It is still in very early stages of development, and barely has a working binary, but I'm sharing it in case there is interest and to try and get some constructive feedback.

Short term roadmap:

  • Fully implement the repository specification on the project website, tentatively stabilizing the on disk format
  • Implementing proper sparse file support
  • Implementing, at a bare minimum, an SFTP backend
  • Import and export tars and other archive formats to/from the repository in an intelligent way (i.e. opening the archives and storing their component files directly)
  • Multithreading of store and retrieve operations

Medium term roadmap:

  • Opt in delta compression of chunks that are not themselves duplicates, but are adjacent to duplicates
  • Cloud storage backends
  • Repository compaction (currently version only supports append-only)
  • Auto compression (perform test compression on the first part of a file and only compress it if the test results in a reasonable compression ratio)
  • Support for data specific slicers (i.e. slicing a disk image by filesystem allocation unit rather than in a content defined manner)

Long term roadmap:

  • Server/client mode for better remote storage peformance
  • GUI/WebUI front end

Current major limitations:

  • In flux on disk format, does not implement the spec properly
  • CLI utility is currently dumb as a box of rocks and can't access most of the features of libasuran

Please take a look at the readme and internals documents to get a better idea of what asuran does and how it does it, though please do note that the Internals document is more of a specification than a description of how the current version works.

You can find the gitlab repository for libasuran here, and the one for asuran here

2 Likes
2

Regarding cloud storage, I'm the author of the backblaze-b2 crate, and I've long been intending to start up work on it again when async await is stabilized.

1 Like
3

Is there plans to be able to do append-only backups (or something like that) while still be able to prune them from time to time?

To prevent intruders from deleting the backups while still being able to prune (compact).

4

The repository backend is actually fundamentally an append-only structure.

Compaction (which is actually not supported currently) is a independent operation that requires re-writing the modified segments without the deleted data.

This can currently be enforced on linux by setting chattr +a (with the exception of the manifest and index files, but these files only represent a performance improvement, and the entire contents of the repository are recoverable without them), but as with any archive not writing to fundamentally WORM storage, this doesn't prevent an attacker with the ability to set filesystem permissions from deleting archives or segment files. On untrusted storage asuran can, at best, be tamper-evident.

The flatfile backend (still an unpublished work in progress) is WORM friendly if that isn't a strong enough guarantee for you.

5

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.