Github每日精选（第41期）：跨端数据同步加密工具restic
restic
restic 是一个快速、安全、高效的备份工具，有的时候我们在做服务端的时候，有些文件我们需要进行另外的保存，可能是因为这是重要的文件需要进行安全的备份，也可能是因为文件的私密性。
在安全方面上restic 做得不错，但是他就失去了像网盘那样的便利性，这也是restic 为啥只在一部分人里面流行而已。
github的地址在这里。
restic 有几个优势：
- 密码登陆和数据加密
- 版本管理，像git一样，他的仓库可以还原到最初的版本，我们需要哪个版本都可以通过像git那样还原回来这就最大的方便了。
- 开源免费，跨平台可用。
安装
在 Debian 上，可以从官方仓库安装的软件包restic ，apt-get：
$ apt-get install restic
使用
在下面的使用过程中，如果你使用过git的代码管理的话，你会发现，restic 跟git很像，甚至是一样的，一样的命令。
创建仓库
初始化一个仓库，按提示输入密码，这个密码打死也不能告诉别人，但是你也不能自己忘掉，忘掉的话这个仓库基本是毁了。
restic init --repo /tmp/backup
提示我们输入两次代码：
enter password for new repository:
enter password again:
created restic repository 25d29858c9 at /tmp/backup
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
告诉我们，这个密码很重要，不要丢了。
备份文件
这样我们就有一个仓库/tmp/backup了，有了仓库以后，我们就可以把想备份的文件放到这个仓库来了。
restic --repo /tmp/backup backup manim/
这里我们要备份当前文档下的manim 文件夹。提示我们输入密码：
enter password for repository:
repository 25d29858 opened successfully, password is correct
created new cache in /root/.cache/restic
no parent snapshot found, will read all files
Files: 207 new, 0 changed, 0 unmodified
Dirs: 57 new, 0 changed, 0 unmodified
Added to the repo: 75.624 MiB
processed 207 files, 75.530 MiB in 0:01
snapshot 5cc831ec saved
简单的列出了我们这个文件夹里的相关信息，还有我们备份的版本号为5cc831ec 。
我们在备份另外一个压缩文件夹。
restic --repo /tmp/backup backup gotty_linux_amd64.tar.gz
显示的信息如下：
enter password for repository:
repository 25d29858 opened successfully, password is correct
no parent snapshot found, will read all files
Files: 1 new, 0 changed, 0 unmodified
Dirs: 0 new, 0 changed, 0 unmodified
Added to the repo: 2.815 MiB
processed 1 files, 2.814 MiB in 0:00
snapshot 37349667 saved
查看仓库版本
有了多次的备份以后，生成了不同的版本号，之前我们备份了两次，来看看这个两次的版本号：
restic -r /tmp/backup snapshots
显示的信息如下：
enter password for repository:
repository 25d29858 opened successfully, password is correct
ID Time Host Tags Paths
-----------------------------------------------------------------------------------------
5cc831ec 2022-08-16 16:41:16 lqp /home/lqp/manim
37349667 2022-08-16 16:44:07 lqp /home/lqp/gotty_linux_amd64.tar.gz
-----------------------------------------------------------------------------------------
2 snapshots
版本还原
要恢复数据出来使用 restore 命令：
restic -r /tmp/backup restore 37349667 --target resticback
这样就把版本号37349667 恢复到resticback 的文件夹了。
enter password for repository:
repository 25d29858 opened successfully, password is correct
restoring <Snapshot 37349667 of [/home/lqp/gotty_linux_amd64.tar.gz] at 2022-08-16 16:44:07.066697148 +0800 CST by root@lqp> to resticback
root@lqp:/home/lqp# cd resticback/
SFTP 备份还原
本地备份可能还不过瘾，来个 SFTP 备份，为了通过 SFTP 备份数据，您必须首先使用 SSH 设置服务器并让它知道您的公钥。无密码登录很重要，因为如果服务器提示输入凭据，则无法进行自动备份。
配置服务器后，可以通过更改init命令中的 URL 方案来简单地设置 SFTP 存储库：
$ restic -r sftp:user@host:/srv/restic-repo init
enter password for new repository:
enter password again:
created restic repository f1c6108821 at sftp:user@host:/srv/restic-repo
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
源码分析
restic采用的是纯go语言编写，对于go语言比较感兴趣的，还有想了解Snapshot原理的可以参考：
package restic
import (
"context"
"fmt"
"os/user"
"path/filepath"
"sync"
"time"
"golang.org/x/sync/errgroup"
"github.com/restic/restic/internal/debug"
)
// Snapshot is the state of a resource at one point in time.
type Snapshot struct {
Time time.Time `json:"time"`
Parent *ID `json:"parent,omitempty"`
Tree *ID `json:"tree"`
Paths []string `json:"paths"`
Hostname string `json:"hostname,omitempty"`
Username string `json:"username,omitempty"`
UID uint32 `json:"uid,omitempty"`
GID uint32 `json:"gid,omitempty"`
Excludes []string `json:"excludes,omitempty"`
Tags []string `json:"tags,omitempty"`
Original *ID `json:"original,omitempty"`
id *ID // plaintext ID, used during restore
}
// NewSnapshot returns an initialized snapshot struct for the current user and
// time.
func NewSnapshot(paths []string, tags []string, hostname string, time time.Time) (*Snapshot, error) {
absPaths := make([]string, 0, len(paths))
for _, path := range paths {
p, err := filepath.Abs(path)
if err == nil {
absPaths = append(absPaths, p)
} else {
absPaths = append(absPaths, path)
}
}
sn := &Snapshot{
Paths: absPaths,
Time: time,
Tags: tags,
Hostname: hostname,
}
err := sn.fillUserInfo()
if err != nil {
return nil, err
}
return sn, nil
}
// LoadSnapshot loads the snapshot with the id and returns it.
func LoadSnapshot(ctx context.Context, loader LoaderUnpacked, id ID) (*Snapshot, error) {
sn := &Snapshot{id: &id}
err := LoadJSONUnpacked(ctx, loader, SnapshotFile, id, sn)
if err != nil {
return nil, err
}
return sn, nil
}
// SaveSnapshot saves the snapshot sn and returns its ID.
func SaveSnapshot(ctx context.Context, repo SaverUnpacked, sn *Snapshot) (ID, error) {
return SaveJSONUnpacked(ctx, repo, SnapshotFile, sn)
}
// ForAllSnapshots reads all snapshots in parallel and calls the
// given function. It is guaranteed that the function is not run concurrently.
// If the called function returns an error, this function is cancelled and
// also returns this error.
// If a snapshot ID is in excludeIDs, it will be ignored.
func ForAllSnapshots(ctx context.Context, be Lister, loader LoaderUnpacked, excludeIDs IDSet, fn func(ID, *Snapshot, error) error) error {
var m sync.Mutex
// track spawned goroutines using wg, create a new context which is
// cancelled as soon as an error occurs.
wg, ctx := errgroup.WithContext(ctx)
ch := make(chan ID)
// send list of snapshot files through ch, which is closed afterwards
wg.Go(func() error {
defer close(ch)
return be.List(ctx, SnapshotFile, func(fi FileInfo) error {
id, err := ParseID(fi.Name)
if err != nil {
debug.Log("unable to parse %v as an ID", fi.Name)
return nil
}
if excludeIDs.Has(id) {
return nil
}
select {
case <-ctx.Done():
return nil
case ch <- id:
}
return nil
})
})
// a worker receives an snapshot ID from ch, loads the snapshot
// and runs fn with id, the snapshot and the error
worker := func() error {
for id := range ch {
debug.Log("load snapshot %v", id)
sn, err := LoadSnapshot(ctx, loader, id)
m.Lock()
err = fn(id, sn, err)
m.Unlock()
if err != nil {
return err
}
}
return nil
}
// For most snapshots decoding is nearly for free, thus just assume were only limited by IO
for i := 0; i < int(loader.Connections()); i++ {
wg.Go(worker)
}
return wg.Wait()
}
func (sn Snapshot) String() string {
return fmt.Sprintf("" ,
sn.id.Str(), sn.Paths, sn.Time, sn.Username, sn.Hostname)
}
// ID returns the snapshot's ID.
func (sn Snapshot) ID() *ID {
return sn.id
}
func (sn *Snapshot) fillUserInfo() error {
usr, err := user.Current()
if err != nil {
return nil
}
sn.Username = usr.Username
// set userid and groupid
sn.UID, sn.GID, err = uidGidInt(*usr)
return err
}
// AddTags adds the given tags to the snapshots tags, preventing duplicates.
// It returns true if any changes were made.
func (sn *Snapshot) AddTags(addTags []string) (changed bool) {
nextTag:
for _, add := range addTags {
for _, tag := range sn.Tags {
if tag == add {
continue nextTag
}
}
sn.Tags = append(sn.Tags, add)
changed = true
}
return
}
// RemoveTags removes the given tags from the snapshots tags and
// returns true if any changes were made.
func (sn *Snapshot) RemoveTags(removeTags []string) (changed bool) {
for _, remove := range removeTags {
for i, tag := range sn.Tags {
if tag == remove {
// https://github.com/golang/go/wiki/SliceTricks
sn.Tags[i] = sn.Tags[len(sn.Tags)-1]
sn.Tags[len(sn.Tags)-1] = ""
sn.Tags = sn.Tags[:len(sn.Tags)-1]
changed = true
break
}
}
}
return
}
func (sn *Snapshot) hasTag(tag string) bool {
for _, snTag := range sn.Tags {
if tag == snTag {
return true
}
}
return false
}
// HasTags returns true if the snapshot has all the tags in l.
func (sn *Snapshot) HasTags(l []string) bool {
for _, tag := range l {
if tag == "" && len(sn.Tags) == 0 {
return true
}
if !sn.hasTag(tag) {
return false
}
}
return true
}
// HasTagList returns true if either
// - the snapshot satisfies at least one TagList, so there is a TagList in l
// for which all tags are included in sn, or
// - l is empty
func (sn *Snapshot) HasTagList(l []TagList) bool {
debug.Log("testing snapshot with tags %v against list: %v", sn.Tags, l)
if len(l) == 0 {
return true
}
for _, tags := range l {
if sn.HasTags(tags) {
debug.Log(" snapshot satisfies %v %v", tags, l)
return true
}
}
return false
}
func (sn *Snapshot) hasPath(path string) bool {
for _, snPath := range sn.Paths {
if path == snPath {
return true
}
}
return false
}
// HasPaths returns true if the snapshot has all of the paths.
func (sn *Snapshot) HasPaths(paths []string) bool {
for _, path := range paths {
if !sn.hasPath(path) {
return false
}
}
return true
}
// HasHostname returns true if either
// - the snapshot hostname is in the list of the given hostnames, or
// - the list of given hostnames is empty
func (sn *Snapshot) HasHostname(hostnames []string) bool {
if len(hostnames) == 0 {
return true
}
for _, hostname := range hostnames {
if sn.Hostname == hostname {
return true
}
}
return false
}
// Snapshots is a list of snapshots.
type Snapshots []*Snapshot
// Len returns the number of snapshots in sn.
func (sn Snapshots) Len() int {
return len(sn)
}
// Less returns true iff the ith snapshot has been made after the jth.
func (sn Snapshots) Less(i, j int) bool {
return sn[i].Time.After(sn[j].Time)
}
// Swap exchanges the two snapshots.
func (sn Snapshots) Swap(i, j int) {
sn[i], sn[j] = sn[j], sn[i]
}