Rsync and Buffalo Linkstation Quad Pro LS-QVL/R5
Latest Update: 2011.05.11 – All bugs are fixed and Deltacopy now works as intended.
The whole concept of having a NAS in the house is to serve as a central repository for all the files so that they are always available on demand. However, as the no. of PCs in the house grows (it is not uncommon anymore for any one individual to now own both a desktop and a notebook plus an array of computing devices such as tablets, etc), it is not surprising to find that maintaining files on the various computing devices suddenly become such a chore not to mention about backup recovery strategies which are often non-existent in a home computing environment.
It is important to note that having a NAS is not an end to having a reliable backup. A NAS is never designed as a backup medium. Having said this, a NAS is extremely handy to consolidate the data from all PCs for backup purposes. Backing up data takes discipline. The problem is how do we achieve this with the most minimal effort. We all have better things to do than sitting in front of a PC tagging files for backup.
This article describes the steps I used to automatically bring all my windows PCs in sync with my buffalo linkstation quad pro and doing this is pretty easy, thanks to the wonderful folks who brought us Rsync!
Now why use Rsync and not other software (such as the Novastor Backup that comes with the Buffalo Linkstation)? Well, most backup software uses a proprietory format and you can’t read the backup volume without using the software itself. There are exceptions of course. A good backup software that I can recommend is Cobian Backup. It is freeware.
So why Rsync? The answer to this may be found here – Rsync, from Wikipedia.
Perhaps the best strength of Rsync is that it is able to do file incremental backup on a “block” level. Assuming you have a huge data file (eg. an outlook email archive) – say 1GB in size – that you want to be backup to the NAS everyday and this file contains “live” data, meaning that you have new data everyday. With Rsync, the first backup will take a longer time as Rsync will copy the entire 1GB worth of data to the NAS. But for subsequent runs, Rsync will only copy the “bytes” or “megabytes” that changed since the last Rsync backup and this therefore make Rsync extremely fast.
There are data backup software that does incremental backup but if they don’t do incremental backup on a “block” level, the program will simply copy the entire 1GB worth of data file to the NAS on each run. Not a very intelligent way to backup in my opinion. I suppose at the end of the day, it all boils down to personal preference and we should all use whatever software we are comfortable with. So if you are still inclined to use Rsync to backup your files to the NAS, read on….
The Linkstation Quad Pro already comes with Rsync pre-installed and enabled (to my pleasant surprise). There is absolutely nothing we need to do on the NAS and this is true even if the NAS has not been rooted (i.e. it is a stock unit). I have verified this using another Linkstation Quad Pro belonging to a friend. So there is really no excuse not to use Rsync immediately as all the background works (the server backend and the the tough part) have already been done by Buffalo!
Rsync is provided as a command line tool and most people (including myself) will freak out when confronted with this command line interface. A GUI to Rsync is greatly needed and there are many of such GUIs available on the net. I am a fan of open source software and freeware and so I will only list several of them here. There are obviously a lot more alternatives than what I am mentioning here. Google is your friend.
- Deltacopy – this is my personal favorite and the user interface is pretty good and intuitive.
- Nasbackup – this is another alternative but I never quite like the interface.
- rsync (for windows) – this is for die-hard fans who are not afraid to tweak stuff on the PCs.
I am a pretty lazy person and so I will just use deltacopy. Download deltacopy here. I prefer the zip version without the installer as I prefer to install the thing myself. This is a left over habit from my early OS/2 days. Side note – those who know OS/2 would probably be as old (or older) as I am now 🙂
Edit: The following is a very good article on setting up Deltacopy. A recommended read – http://forum.synology.com/wiki/index.php/Backup_to_the_Synology_Server_using_DeltaCopy_with_SSH
Unzip deltacopy to a directory. I chose c:\Program Files\Deltacopy. We don’t need the server component on the workstation PC and so we will only need to concentrate on “DeltaC.exe” which is the client side of things.
Before we start, it is to be noted that Deltacopy has a bit of an issue with unicode filenames. If you do not use unicode filenames, then you are all good to go (I hope). I need the correct handling of unicode filenames and I am required to fix this up at this stage.
Just for information only, the following are the kind of errors I get when I use the stock binaries:
- Unicode support is broken. All filename and folder with unicode characters do not get processed correctly.
- Errors such as the following (where XXXXXX.YYY represent the filenames of files being processed) – rsync: readdir(“/cygdrive/XXXXXXXXXXXX.YYY”): File exists (17)
FATAL I/O ERROR: dying to avoid a –delete-during issue with a pre-3.0.7 receiver.
rsync error: requested action not supported (code 4) at /home/lapo/package/rsync-3.0.8-1/src/rsync-3.0.8/flist.c(1799) [sender=3.0.8]
Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.Rsync.exe returned an error. Will try again. This is retry number 1 of 5
sending incremental file list
- As the files are being backup to the NAS volume, I am unable to access the files on windows as it always return a Access Error when doing so. Appears that I lose the read rights to the files.
So yes, I am pretty unhappy when the above happens.
The reason the unicode support is broken in the stock Deltacopy’s setup files is because the set of Cygwin DLLs that deltacopy comes bundled with lack that kind of native support (Deltacopy is afterall just a GUI for the Rsync command line tool which was born in the linux world and adapted to work in the wicked windows world via the Cwgwin library and so the performance is entirely dependent on how up to date the cwgwin libraries are). To resolve this (and I am still experimenting if the patch is truly stable – it works okay so far), Deltacopy’s developer recommends that a new set of Cwgwin library be downloaded and replaced those ones in the original directory. Read more about this here and here. The latest version of Rsync v3.0.8 works for me but not without some more modifications to Deltacopy.
The following references also contain good information how to fix the problem:
- http://digitaldj.net/2011/04/07/rsync-3-0-8-windows-and-chown/ – thanks to this post, rsync is working correctly on my system.
In a nutshell, these are the steps that are required (for my case):
1. Go to C:\Program Files\Deltacopy directory and create a directory called “Backup”. Copy the following files from the deltacopy directory to the “Backup” directory – chmod.exe, cygcrypto-0.9.8.dll, cyggcc_s-1.dll, cygiconv-2.dll, cygintl-8.dll, cygpopt-0.dll, cygwin1.dll, cygz.dll, rsync.exe, ssh.exe. This is just to make sure that we can easily revert to the original deltacopy binaries should whatever we do in step 2 fails.
2. Download the latest cygwin setup file from http://www.cygwin.com/. The latest release I use is cygwin 1.7.9-1 (rsync v3.0.8 comes with this release). The file to download is setup.exe. Install this but make sure that the following files (chmod.exe, cygcrypto-0.9.8.dll, cyggcc_s-1.dll, cygiconv-2.dll, cygintl-8.dll, cygpopt-0.dll, cygwin1.dll, cygz.dll, rsync.exe, ssh.exe) are installed. This is a pretty tedious process. So I have made available the above files in a zip file for download here. Please note that the extension has been renamed to pdf as I am not able to upload zip file in wordpress. Rename to zip and extract accordingly.
3. Unzip the zip file and copy all the files to C:\Program Files\Deltacopy. Overwrite whatever is inside the Deltacopy folder. This will effectively replace all the cygwin support files that come bundled with Deltacopy and update rsync to the latest version v3.0.8.
For whatever reasons, the above link is no longer working. I have now provided all the binaries via my dropbox folder here -> http://dl.dropbox.com/u/9061380/Deltacopy.zip. I hope the link is now working.
4. In C:\Program Files\Deltacopy directory, create a sub directory called “etc”. Inside this directory, use notepad to create a text file called fstab (note there is no file extension .txt, just fstab). Put the following in this text file. Put everything in 1 line only. Save the file.
none /cygdrive cygdrive binary,posix=0,user,noacl 0 0
5. Next, in C:\Program Files\Deltacopy directory, edit the file “deltacd.conf” using notepad. Edit as required to make sure that this file reads as follows:
use chroot = false
strict modes = false
fake super = yes
path = /cygdrive/c/Backup
comment = Backup Drive
read only = false
Save the file.
We are now good to go.
The above fix works, so I thought so…..
Edit 2011.05.11 – I thought the backup works flawlessly until I tried to delete a file using windows. It turns out that all the files backup using Deltacopy are write protected. They read okay but I am not able to delete the files. Not too impress so far….so I decided to investigate further. Knowing that Deltacopy is merely just a GUI to the Rsync command, my gut feeling is that there must be an error in the scripting of the Rsync command line. I was right.
The default run command in Deltacopy (you can right-mouse-click the profile name and select “Display Run Command” to see this) has the following parameters and it goes something like this:
rsync.exe -v -rlt -z –chmod=a=rw,Da+x –delete “/cygdrive/C/Program Files/Deltacopy/Testing/” “192.168.1.3::disk4_Archive4/Testing/” <- This does not work for me!
This tells Rsync to copy the folder called “Testing” in c:\program files\deltacopy directory to a folder called “Testing” in the NAS.
The default “a=rw” parameter should really work in this case as this effectively sets the “user”, “group” and “others” permissions for all the files copied to “read” and “write”. It should work but unfortunately it does not in my case.
It took me a fair while (after some hair pulling exercise) to realize that the correct setup should be as follows:
1. First unclick the tickbox for “Change permissions on server to read/write”
2. Then populate the “Additional Parameters” field to read this “-avr –chmod=a+rwx,g+rwx,o+rwx“.
The revised run command is now
rsync.exe -v -rlt -z –delete -avr –chmod=a+rwx,g+rwx,o+rwx “/cygdrive/C/Program Files/Deltacopy/Testing/” “192.168.1.3::disk4_Archive4/Testing/”
The correct setting for Deltacopy should look like the following:
Now I can declare the setup is truly working now……..I hope 🙂
Side Note: For information only
Oki Software of Japan has a replacement Cygwin library also known to resolve the UTF-8 unicode issues. More information is available here http://www.oki-osk.jp/esc/utf8-cygwin/ and here http://www.oki-osk.jp/esc/utf8-cygwin/download.html. The last page is in Japanese. The google translated version is here.
I have downloaded this Cygwin library file from Oki but I do not see the need to use this version as the new version from v1.7.9-1 (see above) appears to do just fine for me. If really required, this library can be downloaded here: http://www.oki-osk.jp/esc/utf8-cygwin/cygwin1-dll-20-11-18.tar.bz2. This is a tar ball. Open this file using winrar or any other file utilities and extract the file “cygwin1.dll” to replace the one that comes with Deltacopy. Be sure to backup the original version first.
So I suggest you only try this method if the method I outlined earlier does not work.
Launch DeltaC.exe and this is what we will see:
First we need to create a new profile. Click the 2nd button on the toolbar and create a new profile as follows:
Profile name – Use something logical such as “workstation 1”, “compaq notebook”, etc – A name that associate itself with the local PC deltacopy is running from. In this example my profile name is “Enterprise” as I am running deltacopy on my local desktop workstation and I named this after the Starship “Enterprise” from the startrek series….and yes. I am a trekkie too 😀
Server IP/Host Name – put the IP address of the buffalo linkstation here. In my case this is 192.168.1.3.
Virtual Directory Name – Click the small button next to the input field and this will prompt deltacopy to read the share names available on the linkstation and populate the pulldown menu using these information accordingly. So obviously you will have to dictate first where the backups will go on the linkstation, which drive, which share, etc. If this has not been setup correctly, do so now using the web admin of the linkstation.
Assuming the linkstation’s shares have all been setup correctly, clicking the pull down menu will bring up all the shares available. In my case, I select to save the backup to this share called “disk1_Archive1”.
So if you are backing up the folder “d:\My Documents” to the share called “disk1_Archive1”, the physical path on the NAS will be \Archive1\My Documents.
Now press “Test Connection” button. If the settings are all correct, a “Connection Successful” dialog box will pop up like so:
Otherwise you will get a message similar to this:
Assuming the settings are all correct, now press the “Add Profile” button. The screen will now become:
Now all we have to do is to define what files/folders we need to backup. Select accordingly. After the files/folders have been defined, you can further fine tune the kind of files to copy and the target path you desired.
Go to the “Options” tab. I leave all the settings intact as they are pretty much spot on, as far as i am concerned. If you are paranoid about security, you can enable “Connect via SSH” but I am not sure if this works on a stock linkstation box without root access. I am not paranoid and so i did not select the connection via SSH.
Personally I think this setting is only meaningful if you have the linkstation on a remote site, say in your secret underground base somewhere and you want to make sure that the connections used are as secured as possible. I am not affiliated with this kind of organizations and so I never bother.
Now click the “Authentication” tab. Again, if you are connecting to a remote site, it is imperative that the remote server must first authenticate this connection using your username and password. I did not do this and if you need this feature, you will need to experiment with the settings and google for answers if things do not work.
Now click the “Modify Schedule” button. This will be the screen that pops up.
This part is a little tricky and there appears to be a tiny bug between deltacopy and windows 7 (at least on my system) and the issue is that Windows Scheduler is somehow unable to start the deltacopy’s Rsync process correctly. I am not sure that the root problem is…
The remedy to this issue is to make sure that the run task is scripted with the full path name to Deltacopy.
Under the run command in the above screen, add “DeltaC.exe” to the path name like the following:
Click the “Schedule” tab.
Press the “New” button to define the schedule when the backup will occur.
Obviously if the intention is to sync the workstation to the NAS, then rsync will need to run pretty frequently. Don’t set deltacopy to run once each week and expecting your files will be fully in sync all the time. It will not happen obviously.
I suggest for a start, run rsync on a daily basis. Set the time to a period where you know you will not be using the workstation. Click the “Advanced” button to further fine tune the settings, especially if you want rsync to run more than once per day. Everything should be self-explanatory at this point. I leave mine at the default settings.
You can obviously many more schedule in this page. Just click “New” and define away.
This is my settings at the moment. Again, all pretty much self-explanatory.
Click “Apply” and a pop up dialog box will come up asking for password.
Put the desired passwords, click “OK” and then “Apply”.
Click the “Settings” tab and there are more settings to be tweaked. I leave everything as it is.
There is just one thing here which I find it interesting….I have not experiment with it yet but there is a potential here to further automate the process so that the workstation can be in a standby mode at night and wake up only at a certain time to start the backup/sync process and then go back to sleep afterwards – purely to satisfy the desire to save a couple of watthours per day. Don’t we all?
So there we go. Deltacopy is setup and ready to roll.
The first backup to the NAS will obviously be a long process as Rysnc has to copy all the files to the NAS. But the subsequent process will be quite speedy as Rsync only copy whatever files/directories that are changed since the last backup.
Deltacopy has an extremely handy feature to send an email to let us know that the backup process has completed successfully (or to let us know there is an error) and to give us a comprehensive log of what files have been copied. This is setup really easily. Click the third icon on the toolbar. Fill up all the settings accordingly. The settings should pretty much be the same as the email notification feature found on the NAS’s webadmin module.
Press the “Test Email” button to check that the settings are all correct. Amend as required.
Personally I think it is a good idea to check that Windows actually register Deltacopy’s Rsync tasklist correctly. This is where I think using linux is so much more straight forward here. You can just list all the cron tasks using the command line tool. In windows, this is not so straightforward (at least to me).
Go to windows 7’s control panel. Click on “System and Security”, and then “Administrative Tools” and then “Task Scheduler”. Alternatively, type the word task into search bar. Windows Task Scheduler will launch.
If the setup is correct, we will be able to find the task name listed as follows (note the highlighted line).
Now it is time to test the actual backup process.
Right click on the new profile just created. Select “Run Now”. For commandline buff, click on the “Display Run Command” will let us see the parameters Rysnc is set to run with. Deltacopy will start to backup the selected folders to the NAS. Give it time to copy the files over especially if you have a slow network and if the folder is particularly huge. If things are working, you will be able to see the files on the NAS at the end of the process.
Now observe the shares on the NAS over the next few days to see if deltacopy actually copy files to the NAS at the scheduled time. The above setup works well for me and I hope this works for you too. Happy “NAS”ing.