Synology GitLab Backup and Restore

Synology and GitLab logos


GitLab has built-in backup and restore capability, but it is not immediately apparent how to use backup and restore on the Synology NAS installation of GitLab.  The following instructions may also be applicable for other Docker based GitLab installations.

This post will explain how to protect your data in the GitLab repository.  This is protecting against a catastrophic RAID failure, and also accidentally corrupting/destroying the repository.  Please ensure you have backups stored off-site, and ideally multiple versions of the backup data.

With any backup/restore procedures it is important that you are able to test/validate the ability to recover your data before you need to do so.

TIP: Check out my quick fix if you get the dreaded GitLab 502 error.

GitLab Backup

Backup is easy to configure - when you know the right commands.  Refer to the GitLab documentation for all the details.

There are two phases to the backup:
  1. A scheduled task to create a local backup of the GitLab repository.
  2. Then a standard HyperBackup backup to external media or cloud storage.
Open the Synology DSM Control Panel, and go to the Task Scheduler:
Task Scheduler
Create a new Scheduled Task to run a User-defined script:
Create a scheduled task
Under General Settings, give the task a name (e.g. GitLab Backup), then set the user to root.
Configure an appropriate backup schedule.  Note you may backup GitLab more frequently than you perform your off-site backup.
In the Task Settings, enter your email notification preferences.
The User-defined script must be entered as follows.  Put the command on one line with spaces where shown:
docker exec -u git -i synology_gitlab /home/git/gitlab/bin/rake gitlab:backup:create
Save changes to the new task.

Select the task in the list:
Select the GitLab backup task
Then click on the Run button to test the backup.

Depending on your NAS configuration you may be able to view the backup within the DSM File Station - under docker/gitlab/backups:
File Station GitLab backup path
Alternatively login to the NAS using ssh (or telnet).  My backup path is /volume1/docker/gitlab/backups/, since Docker is installed to a storage volume called Volume 1.

Confirm the backup tar file has been created.  You may want to inspect the contents:
# tar -vtf /volume1/docker/gitlab/backups/1516284266_2018_01_18_9.4.4_gitlab_backup.tar
drwx------ git/git           0 2018-01-19 01:04 repositories/
...
drwxr-xr-x git/git           0 2018-01-19 01:04 db/
-rw------- git/git       26184 2018-01-19 01:04 db/database.sql.gz
-rw------- git/git       27509 2018-01-19 01:04 uploads.tar.gz
-rw------- git/git         146 2018-01-19 01:04 builds.tar.gz
-rw------- git/git         146 2018-01-19 01:04 artifacts.tar.gz
-rw------- git/git         147 2018-01-19 01:04 pages.tar.gz
-rw------- git/git         146 2018-01-19 01:04 lfs.tar.gz
-rw-r--r-- git/git         156 2018-01-19 01:04 backup_information.yml
Now that the GitLab backup is working and you have identified the backup path, you can modify your off-site backup to include this directory.

For example in HyperBackup, I've included the GitLab backups folder in my backup task.  No other GitLab folders are required, but you may wish to backup the ssh folders - otherwise client keys may need to be regenerated.
HyperBackup gitlab/backups folder

Automatic deletion of old backups

Now that backups are running periodically, you will start to fill your disk with multiple copies of the GitLab backup.  What you need is an automatic mechanism to remove old backups.

Open the Synology Package Center, and stop the GitLab package:
Synology Package Center, Stop GitLab

Then open the Docker application, and click on Container:
Docker show Containers

Select/highlight the synology_gitlab container, then click on Edit:
Docker Edit synology_gitlab

Switch to the Environment tab:
Docker Container edit, switch to Environment tab

Click the plus button to add a new environment variable:
Docker Environment, add new environment variable

Then enter the name of the environment variable as GITLAB_BACKUP_EXPIRY, and set the value to the number of seconds backups should be retained for:
Docker Environment, add GITLAB_BACKUP_EXPIRY environment variable

I've used the value of 1209600 which is a period of 14 days.

Click Apply to save the changes.

In the Synology Package Center, restart the GitLab package:
Synology Package Center, Run GitLab

Every time a backup runs any files older than the configured expiry time will be automatically removed from local storage.

GitLab Restore

Please read through these instructions entirely before proceeding.

GitLab Package Installation

Firstly you need to have the Synology GitLab package installed on your NAS, and GitLab must be running.

Important! you must be running exactly the same version as when the backup was performed.  Otherwise the restore will immediately fail with the error message:
Your current GitLab version (n.n.n) differs from the GitLab version in the backup!
Whenever the GitLab package is upgraded, I recommend you download and preserve the package (spk) file from the Synology website - just in case.  You probably should also perform a manual backup before performing a GitLab upgrade.

You can identify the version of GitLab from the backup file - for example this backup file is from version 9.4.4, and matches my installed package version of 9.4.4-0050:
1516284266_2018_01_18_9.4.4_gitlab_backup.tar
TIP: If you uninstall or reinstall the GitLab package the existing backups directory will be removed.  Be careful to preserve your backup files beforehand.

Backup recovery from off-site storage

If the local copy of the GitLab backup is no longer available, then you will have to restore from your off-site backup.

Make sure you restore the backup file into the docker/gitlab/backups directory.

GitLab restore

Taking note of the prefix on the backup file (e.g. 1516284266_2018_01_18_9.4.4), initiate the restore command - for example:
# chmod go+r 1516284266_2018_01_18_9.4.4_gitlab_backup.tar
# docker exec -u git -it synology_gitlab /home/git/gitlab/bin/rake gitlab:backup:restore BACKUP=1516284266_2018_01_18_9.4.4

Unpacking backup ... done

Before restoring the database we recommend removing all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.
Enter "yes" when prompted:
Do you want to continue (yes/no)? yes
Removing all tables. Press `Ctrl-C` within 5 seconds to abort
Cleaning the database ...
done
Restoring database ...
Restoring MySQL database gitlab ... [DONE]
done
Restoring repositories ...
 * group1/project1 ... [DONE]
 * group2/project2 ... [DONE]
...
Put GitLab hooks in repositories dirs [DONE]
done
Restoring uploads ...
done
Restoring builds ...
done
Restoring artifacts ...
done
Restoring pages ...
done
Restoring lfs objects ...
done
This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.

Do you want to continue (yes/no)? yes
.
Deleting tmp directories ... done
done
done
done
done
done
done
done

Clear the GitLab cache

# docker exec -u git -i synology_gitlab /home/git/gitlab/bin/rake cache:clear
At this point you may need to stop and restart the GitLab package, however in my testing it doesn't appear to be necessary.

Verification

Login to GitLab using the web interface and check the state of the repository.

Verify any Git clients can access the GitLab server.  You may need to recreate ssh keys if this information was not recovered.

Comments

Unknown said…
thank you so much, the backup solution work greats in my NAS, is it possible to make the script to keep last 10 backup only???

docker exec -u git -i synology_gitlab /home/git/gitlab/bin/rake gitlab:backup:create
25 February 2018 at 21:06
Mark Stead said…
@Pong Mok: Yes you're correct, I hadn't got around to implementing backup expiry. In my case the backups are relatively small, but I had accrued 77 copies.

I've updated the instructions to include setting the GITLAB_BACKUP_EXPIRY environment variable to enable automatic deletion of backups.
28 February 2018 at 00:05
Unknown said…
Hi, Thanks for this good blog.

I try to follow your instructions but got following error message, and restore failed.

Restoring MySQL database gitlab ... ERROR 1193 (HY000) at line 5: Unknown system variable 'statement_timeout'

Do you have any idea about this issue? Thanks again.
27 August 2018 at 22:26
Mark Stead said…
@Bill Peng: I'm not really sure.

A Google search indicates this occurs when exporting from Postgres and importing to MySQL - such as here:
https://forums.mysql.com/read.php?83,550898,550898

However that doesn't make sense in this situation. On the other hand they have introduced two separate versions of MariaDB (10 and 5.5). Perhaps the backup was taken with a different DB version.

I would assume the SET statement is not particularly important - so long as it skips it and completes the restore successfully. If necessary you may be able to edit the dump file by hand to remove the SET statement.
30 August 2018 at 00:07
Nicolas Goudard said…
Hi. Thanks for this great job. I have problem. I am on DSM6. The command "docker exec -u git -i synology_gitlab /home/git/gitlab/bin/rake gitlab:backup:create" does not work for me because the folder does not exits. On the official repository : https://github.com/jboxberger/synology-gitlab
they say
sudo /usr/local/bin/docker exec -it synology_gitlab bash -c "sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1"
In command line it is working but not in the task manager
In the task manager I try it with admin account but it does not work. I try with root :
/usr/local/bin/docker exec -it synology_gitlab bash -c "sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1"
sudo su -c '/usr/local/bin/docker exec -it synology_gitlab bash -c "sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1"'
How can I do ?
Thanks in advance
13 August 2019 at 15:43
Nicolas Goudard said…
I find the answer after many hours. not have to put "-it" option for interactive tty because it does not work in cron . Only put in cron this line
sudo su -c '/usr/local/bin/docker exec synology_gitlab bash -c "sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1"'
13 August 2019 at 23:01
Mark Stead said…
@Nicholas Goudard: Thanks for your feedback - I'm glad you got it going. Hopefully your cron command is helpful to others experiencing the same problem.
14 August 2019 at 13:38
Powie said…
Great Great Great! That's what I'm looking for!
13 February 2021 at 21:45
Unknown said…

Hi Mark,

Does this backup work for gitlab installation on server/physical host too ? or is it only for gitlab installation on NAS/Synology ?
5 October 2021 at 05:48
Mark Stead said…
@Unknown: The back mechanism is a standard part of GitLab. However, those UI elements for scheduling backup and configuring Docker are specific to Synology.

If you translate the paths to your environment, then you can try running the commands.

Also you should refer to the documentation provided by GitLab.
https://docs.gitlab.com/ee/raketasks/backup_restore.html
5 October 2021 at 09:01
Post a Comment

Popular posts from this blog

Resolving FOSCAM connection dropouts

Image
I bought a FOSCAM FI8908W wireless IP webcam off eBay recently. It has lots of features for the price. However, I've had problems with being unable to connect to the camera after a period of time (an hour or so). Power cycling the camera would fix the problem. I've tried upgrading to the latest firmware (version 11.14.2.17 is supposed to fix WiFi disconnection problems), but it did not fix my problem. Problem analysis The problem is not related to my browser or the wireless network. I've tried disabling DHCP (router based static IP). It turns out the problem is that the webcam simply ceases to respond to ARP  (Address Resolution Protocol) requests (once in this broken state). ARP is a mechanism to translate between IP addresses and device MAC addresses. It underpins all IP communications over both Ethernet and WiFi, since at the lowest level it relies on exchange of packets between device MAC addresses. The ARP protocol allows a network device to discover t...
Read more

Building an automatic chicken door opener

Image
We recently got chickens and found them to be noisy in the morning because they wanted to be let out of their coop.  Of course, we just wanted to sleep in.  What we needed was an automatic door opener. Here's our 3 hens (Edna, Dora and the bantam Dotty).  Click for more photos. Here's the chicken coop, with the addition of an automatic door opener.  The chicken coop was purchased from Pets Station . There are some (expensive) automatic chicken doors that are available for purchase, but I decided to design and make my own.  You are welcome to copy or adapt the design to suit your own requirements.
Read more

TVersity media serving to the Astone AP-300

Image
TVersity is a digital media server that can share media over your home network using UPnP/DNLA. This is kind-of like Windows file sharing/SMB, except that a digital media server can be a lot smarter.  You get access to traditional media such as picture, audio or movie files, like you would through network file sharing.  TVersity though can take things further, supporting: On-the-fly transcoding of media into a format suitable for your media player, Downloading content from the Internet, Streaming media as it is being downloaded or transcoded, Tagging of media to present a smarter content view instead of just a traditional directory structure, Improved network performance compared with normal network file sharing.  I'm able to stream 1080p video rather than having to copy it to the internal HDD My media player is a Astone AP-300 and is similar to many others in that it has network connectivity, HD video support, and of-course acts as a UPnP Media Player/Render...
Read more