New to Stash? Please start here.
This tutorial will show you how to backup a Kubernetes deployment using Stash in offline mode. By default, stash takes backup in online mode where sidecar container is added to take periodic backups and check backups. But sometimes you need to ensure that source data is not being modified while taking the backup, that means running backup while keeping workload pod stopped. In such case, you can run the backup in offline mode. To do this you need to specify spec.type: offline in Restic CRD.
At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using Minikube. Now, install Stash in your cluster following the steps here.
In this tutorial, we are going to backup the /source/data folder of a busybox pod into a local backend. First, deploy the following busybox Deployment in your cluster. Here we are using a git repository as a source volume for demonstration purpose.
apiVersion: apps/v1beta1
kind: Deployment
metadata:
labels:
app: stash-demo
name: stash-demo
namespace: default
spec:
replicas: 1
template:
metadata:
labels:
app: stash-demo
name: busybox
spec:
containers:
- command:
- sleep
- "3600"
image: busybox
imagePullPolicy: IfNotPresent
name: busybox
volumeMounts:
- mountPath: /source/data
name: source-data
restartPolicy: Always
volumes:
- gitRepo:
repository: https://github.com/appscode/stash-data.git
name: source-data
$ kubectl apply -f ./docs/examples/tutorial/busybox.yaml
deployment "stash-demo" created
Run the following command to confirm that busybox pods are running.
$ kubectl get pods -l app=stash-demo
NAME READY STATUS RESTARTS AGE
stash-demo-788ffcf9c6-p5kxc 1/1 Running 0 12s
Now, create a Secret that contains the key RESTIC_PASSWORD. This will be used as the password for your restic repository.
$ kubectl create secret generic stash-demo --from-literal=RESTIC_PASSWORD=changeit
secret "stash-demo" created
You can check that the secret was created like this:
$ kubectl get secret stash-demo -o yaml
apiVersion: v1
data:
RESTIC_PASSWORD: Y2hhbmdlaXQ=
kind: Secret
metadata:
creationTimestamp: 2017-12-04T05:24:22Z
name: stash-demo
namespace: default
resourceVersion: "22328"
selfLink: /api/v1/namespaces/default/secrets/stash-demo
uid: 62aa8ef8-d8b3-11e7-be92-0800277f19c0
type: Opaque
Now, create a Restic CRD with selectors matching the labels of the busybox Deployment and spec.type: offline.
$ kubectl apply -f ./docs/examples/tutorial/restic_offline.yaml
restic "stash-demo" created
apiVersion: stash.appscode.com/v1alpha1
kind: Restic
metadata:
name: stash-demo
namespace: default
spec:
selector:
matchLabels:
app: stash-demo
type: offline
fileGroups:
- path: /source/data
retentionPolicyName: 'keep-last-5'
backend:
local:
mountPath: /safe/data
hostPath:
path: /data/stash-test/restic-repo
storageSecretName: stash-demo
schedule: '@every 5m'
volumeMounts:
- mountPath: /source/data
name: source-data
retentionPolicies:
- name: 'keep-last-5'
keepLast: 5
prune: true
When a Restic is created with spec.type: offline, stash operator adds an init-container instead of sidecar container to target workload pods. The init-container takes backup once. If the backup is successfully completed, then it creates a job to perform restic check and exits. The app container starts only after the init-container exits without any error. This ensures that the app container is not running while taking backup.
Stash operator also creates a cron-job that deletes the workload pods according to the spec.schedule. Thus the workload pods get restarted periodically and allow the init-container to take backup.
$ kubectl get pods -l app=stash-demo -w
NAME READY STATUS RESTARTS AGE
stash-demo-788ffcf9c6-p5kxc 1/1 Terminating 0 1m
stash-demo-7b4f6877dc-nhrz9 0/1 Init:0/1 0 4s
stash-demo-7b4f6877dc-nhrz9 1/1 Running 0 32s
$ kubectl get deployment stash-demo -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "2"
restic.appscode.com/last-applied-configuration: |
{"kind":"Restic","apiVersion":"stash.appscode.com/v1alpha1","metadata":{"name":"stash-demo","namespace":"default","selfLink":"/apis/stash.appscode.com/v1alpha1/namespaces/default/restics/stash-demo","uid":"c55d5918-d8da-11e7-be92-0800277f19c0","resourceVersion":"57719","creationTimestamp":"2017-12-04T10:06:18Z"},"spec":{"selector":{"matchLabels":{"app":"stash-demo"}},"fileGroups":[{"path":"/source/data","retentionPolicyName":"keep-last-5"}],"backend":{"storageSecretName":"stash-demo","local":{"volumeSource":{"hostPath":{"path":"/data/stash-test/restic-repo"}},"path":"/safe/data"}},"schedule":"@every 5m","volumeMounts":[{"name":"source-data","mountPath":"/source/data"}],"resources":{},"retentionPolicies":[{"name":"keep-last-5","keepLast":5,"prune":true}],"type":"offline"},"status":{}}
restic.appscode.com/tag: canary
creationTimestamp: 2017-12-04T10:04:11Z
generation: 2
labels:
app: stash-demo
name: stash-demo
namespace: default
resourceVersion: "57824"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/stash-demo
uid: 798d4e60-d8da-11e7-be92-0800277f19c0
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 2
selector:
matchLabels:
app: stash-demo
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: stash-demo
name: busybox
spec:
containers:
- command:
- sleep
- "3600"
image: busybox
imagePullPolicy: IfNotPresent
name: busybox
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /source/data
name: source-data
dnsPolicy: ClusterFirst
initContainers:
- args:
- backup
- --restic-name=stash-demo
- --workload-kind=Deployment
- --workload-name=stash-demo
- --image-tag=canary
- --enable-rbac=true
env:
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: appscode/stash:0.7.0
imagePullPolicy: IfNotPresent
name: stash
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp
name: stash-scratchdir
- mountPath: /etc/stash
name: stash-podinfo
- mountPath: /source/data
name: source-data
readOnly: true
- mountPath: /safe/data
name: stash-local
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- gitRepo:
repository: https://github.com/appscode/stash-data.git
name: source-data
- emptyDir: {}
name: stash-scratchdir
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
name: stash-podinfo
- hostPath:
path: /data/stash-test/restic-repo
type: ""
name: stash-local
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2017-12-04T10:06:26Z
lastUpdateTime: 2017-12-04T10:06:26Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: 2017-12-04T10:04:11Z
lastUpdateTime: 2017-12-04T10:06:26Z
message: ReplicaSet "stash-demo-7b4f6877dc" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 2
readyReplicas: 1
replicas: 1
updatedReplicas: 1
If everything goes well, A Repository crd with name deployment.stash-demo will be created for the respective repository in the specified backend. Verify that, Repository is created successfully using this command,
$ kubectl get repository deployment.stash-demo
NAME AGE
deployment.stash-demo 1m
Restic will take backup of the volume periodically with a 1-minute interval. You can verify that backup is taking successfully by,
$ kubectl get snapshots -l repository=deployment.stash-demo
NAME AGE
deployment.stash-demo-c1014ca6 10s
Here, deployment.stash-demo-c1014ca6 represents the name of the successful backup Snapshot taken by Stash in deployment.stash-demo repository.
Stash operator also creates a cron job to periodically delete workload pods according to spec.schedule. Please note that Kubernetes cron jobs do not support timezone.
kubectl get cronjob
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
stash-kubectl-cron-stash-demo @every 5m False 0 <none>
Note that offline backup is not supported for workload kind Deployment, Replicaset and ReplicationController with replicas > 1.
To cleanup the Kubernetes resources created by this tutorial, run:
$ kubectl delete deployment stash-demo
$ kubectl delete secret stash-demo
$ kubectl delete restic stash-demo
$ kubectl delete repository deployment.stash-demo
If you would like to uninstall Stash operator, please follow the steps here.