Synology GitLab Backup and Restore

Synology and GitLab logos


GitLab has built-in backup and restore capability, but it is not immediately apparent how to use backup and restore on the Synology NAS installation of GitLab.  The following instructions may also be applicable for other Docker based GitLab installations.

This post will explain how to protect your data in the GitLab repository.  This is protecting against a catastrophic RAID failure, and also accidentally corrupting/destroying the repository.  Please ensure you have backups stored off-site, and ideally multiple versions of the backup data.

With any backup/restore procedures it is important that you are able to test/validate the ability to recover your data before you need to do so.

TIP: Check out my quick fix if you get the dreaded GitLab 502 error.

GitLab Backup

Backup is easy to configure - when you know the right commands.  Refer to the GitLab documentation for all the details.

There are two phases to the backup:
  1. A scheduled task to create a local backup of the GitLab repository.
  2. Then a standard HyperBackup backup to external media or cloud storage.
Open the Synology DSM Control Panel, and go to the Task Scheduler:
Task Scheduler
Create a new Scheduled Task to run a User-defined script:
Create a scheduled task
Under General Settings, give the task a name (e.g. GitLab Backup), then set the user to root.
Configure an appropriate backup schedule.  Note you may backup GitLab more frequently than you perform your off-site backup.
In the Task Settings, enter your email notification preferences.
The User-defined script must be entered as follows.  Put the command on one line with spaces where shown:
docker exec -u git -i synology_gitlab /home/git/gitlab/bin/rake gitlab:backup:create
Save changes to the new task.

Select the task in the list:
Select the GitLab backup task
Then click on the Run button to test the backup.

Depending on your NAS configuration you may be able to view the backup within the DSM File Station - under docker/gitlab/backups:
File Station GitLab backup path
Alternatively login to the NAS using ssh (or telnet).  My backup path is /volume1/docker/gitlab/backups/, since Docker is installed to a storage volume called Volume 1.

Confirm the backup tar file has been created.  You may want to inspect the contents:
# tar -vtf /volume1/docker/gitlab/backups/1516284266_2018_01_18_9.4.4_gitlab_backup.tar
drwx------ git/git           0 2018-01-19 01:04 repositories/
...
drwxr-xr-x git/git           0 2018-01-19 01:04 db/
-rw------- git/git       26184 2018-01-19 01:04 db/database.sql.gz
-rw------- git/git       27509 2018-01-19 01:04 uploads.tar.gz
-rw------- git/git         146 2018-01-19 01:04 builds.tar.gz
-rw------- git/git         146 2018-01-19 01:04 artifacts.tar.gz
-rw------- git/git         147 2018-01-19 01:04 pages.tar.gz
-rw------- git/git         146 2018-01-19 01:04 lfs.tar.gz
-rw-r--r-- git/git         156 2018-01-19 01:04 backup_information.yml
Now that the GitLab backup is working and you have identified the backup path, you can modify your off-site backup to include this directory.

For example in HyperBackup, I've included the GitLab backups folder in my backup task.  No other GitLab folders are required, but you may wish to backup the ssh folders - otherwise client keys may need to be regenerated.
HyperBackup gitlab/backups folder

Automatic deletion of old backups

Now that backups are running periodically, you will start to fill your disk with multiple copies of the GitLab backup.  What you need is an automatic mechanism to remove old backups.

Open the Synology Package Center, and stop the GitLab package:
Synology Package Center, Stop GitLab

Then open the Docker application, and click on Container:
Docker show Containers

Select/highlight the synology_gitlab container, then click on Edit:
Docker Edit synology_gitlab

Switch to the Environment tab:
Docker Container edit, switch to Environment tab

Click the plus button to add a new environment variable:
Docker Environment, add new environment variable

Then enter the name of the environment variable as GITLAB_BACKUP_EXPIRY, and set the value to the number of seconds backups should be retained for:
Docker Environment, add GITLAB_BACKUP_EXPIRY environment variable

I've used the value of 1209600 which is a period of 14 days.

Click Apply to save the changes.

In the Synology Package Center, restart the GitLab package:
Synology Package Center, Run GitLab

Every time a backup runs any files older than the configured expiry time will be automatically removed from local storage.

GitLab Restore

Please read through these instructions entirely before proceeding.

GitLab Package Installation

Firstly you need to have the Synology GitLab package installed on your NAS, and GitLab must be running.

Important! you must be running exactly the same version as when the backup was performed.  Otherwise the restore will immediately fail with the error message:
Your current GitLab version (n.n.n) differs from the GitLab version in the backup!
Whenever the GitLab package is upgraded, I recommend you download and preserve the package (spk) file from the Synology website - just in case.  You probably should also perform a manual backup before performing a GitLab upgrade.

You can identify the version of GitLab from the backup file - for example this backup file is from version 9.4.4, and matches my installed package version of 9.4.4-0050:
1516284266_2018_01_18_9.4.4_gitlab_backup.tar
TIP: If you uninstall or reinstall the GitLab package the existing backups directory will be removed.  Be careful to preserve your backup files beforehand.

Backup recovery from off-site storage

If the local copy of the GitLab backup is no longer available, then you will have to restore from your off-site backup.

Make sure you restore the backup file into the docker/gitlab/backups directory.

GitLab restore

Taking note of the prefix on the backup file (e.g. 1516284266_2018_01_18_9.4.4), initiate the restore command - for example:
# chmod go+r 1516284266_2018_01_18_9.4.4_gitlab_backup.tar
# docker exec -u git -it synology_gitlab /home/git/gitlab/bin/rake gitlab:backup:restore BACKUP=1516284266_2018_01_18_9.4.4

Unpacking backup ... done

Before restoring the database we recommend removing all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.
Enter "yes" when prompted:
Do you want to continue (yes/no)? yes
Removing all tables. Press `Ctrl-C` within 5 seconds to abort
Cleaning the database ...
done
Restoring database ...
Restoring MySQL database gitlab ... [DONE]
done
Restoring repositories ...
 * group1/project1 ... [DONE]
 * group2/project2 ... [DONE]
...
Put GitLab hooks in repositories dirs [DONE]
done
Restoring uploads ...
done
Restoring builds ...
done
Restoring artifacts ...
done
Restoring pages ...
done
Restoring lfs objects ...
done
This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.

Do you want to continue (yes/no)? yes
.
Deleting tmp directories ... done
done
done
done
done
done
done
done

Clear the GitLab cache

# docker exec -u git -i synology_gitlab /home/git/gitlab/bin/rake cache:clear
At this point you may need to stop and restart the GitLab package, however in my testing it doesn't appear to be necessary.

Verification

Login to GitLab using the web interface and check the state of the repository.

Verify any Git clients can access the GitLab server.  You may need to recreate ssh keys if this information was not recovered.

Comments

Pong Mok said…
thank you so much, the backup solution work greats in my NAS, is it possible to make the script to keep last 10 backup only???

docker exec -u git -i synology_gitlab /home/git/gitlab/bin/rake gitlab:backup:create
Mark Stead said…
@Pong Mok: Yes you're correct, I hadn't got around to implementing backup expiry. In my case the backups are relatively small, but I had accrued 77 copies.

I've updated the instructions to include setting the GITLAB_BACKUP_EXPIRY environment variable to enable automatic deletion of backups.
Bin Peng said…
Hi, Thanks for this good blog.

I try to follow your instructions but got following error message, and restore failed.

Restoring MySQL database gitlab ... ERROR 1193 (HY000) at line 5: Unknown system variable 'statement_timeout'

Do you have any idea about this issue? Thanks again.
Mark Stead said…
@Bill Peng: I'm not really sure.

A Google search indicates this occurs when exporting from Postgres and importing to MySQL - such as here:
https://forums.mysql.com/read.php?83,550898,550898

However that doesn't make sense in this situation. On the other hand they have introduced two separate versions of MariaDB (10 and 5.5). Perhaps the backup was taken with a different DB version.

I would assume the SET statement is not particularly important - so long as it skips it and completes the restore successfully. If necessary you may be able to edit the dump file by hand to remove the SET statement.

Popular posts from this blog

Resolving FOSCAM connection dropouts

Building an automatic chicken door opener

Installing Adobe Acrobat Pro 8 on Windows 7 x64