Cloud synchronization performance tests of various Linux clients

I already tried using Nextcloud as a backup solution which will sync my complete home directory into a Nextcloud instance. This is very practical if you want to have access to your files when you don’t have your laptop with you or when it gets lost. Of course there are some drawbacks with this approach: Remote files are unencrypted (so that the Nextcloud file explorer can see them), accidental deletions results in data loss on both sides and lack of file versioning or snapshots (depends on your Nextcloud configuration).
An other issue is the performance in case of syncing many small files. This is a problem in case of the Nextcloud Desktop client because it creates a separate http connection (WebDav) for every single file. I created a bug report on Github to start a discussion for finding a better and faster approach.

In this post I’m going to compare the performance of some Linux file synchronization clients and further describe how to automate the benchmarking.

The test results

First of all, here are my test results. This graph shows the time in seconds needed to synchronize ~8MB of local-to-remote and ~1MB of remote-to-local files (972 single files):

Since my connection speed is not the bottleneck in this test, Nextcloud will need up to 8 minutes to index and up-/download the files. Considering the very small file size in total, this is not a very good result. I’m not sure what technique Dropbox is using but I guess it’s a proprietary, more enhanced protocol. Rclone is a really interesting tool which supports various cloud services. One limitation is that rclone does not support bidirectional file synchronization yet, so it is not really comparable to the other clients.

Preparations

In this part I’ll sketch my setup I used for the benchmark. It will run in a container on Archlinux using systemd-nspawn and as a user called test. In the next steps, we’ll install the AUR handler pacaur und the syncing desktop clients.

pacman -S arch-install-scripts
btrfs subvol create /var/lib/container/archlinux-base
mkdir /etc/systemd/nspawn
pacstrap /var/lib/container/archlinux-base base base-devel
systemctl enable --now systemd-networkd systemd-resolved
systemd-nspawn -nD /var/lib/machines/archlinux-nextcloudcli --template=/var/lib/container/archlinux-base
systemctl start systemd-nspawn@archlinux-nextcloudcli
machinectl shell root@archlinux-nextcloudcli /bin/bash -c "systemctl enable --now systemd-networkd systemd-resolved"
machinectl shell root@archlinux-nextcloudcli
useradd -m test
passwd test
vim /etc/sudoers # add user test
su test
cd
curl "https://aur.archlinux.org/cgit/aur.git/snapshot/cower.tar.gz" | tar xz -C .
curl "https://aur.archlinux.org/cgit/aur.git/snapshot/pacaur.tar.gz" | tar xz -C .
cd cower
makepkg -si --skipinteg
cd ../pacaur
makepkg -si
pacaur -S nextcloud-client rclone dropbox-cli

In our test scenario we’ll sync the openssh source code to the server and also test the bi-directional synchronisation performance by putting the mosh source code into the Nextcloud instance.

su test
cd
wget https://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.7p1.tar.gz
wget https://github.com/mobile-shell/mosh/archive/mosh-1.3.2.tar.gz
tar -xvf openssh-7.7p1.tar.gz -C sync
tar -xvf mosh-1.3.2.tar.gz -C sync_remote
du -hs sync/openssh-7.7p1 sync_remote/mosh-mosh-1.3.2
# 8.3M	sync/openssh-7.7p1
# 1.3M	sync_remote/mosh-mosh-1.3.2
find sync/openssh-7.7p1 sync_remote/mosh-mosh-1.3.2 | wc -l
# 972
nextcloudcmd -h -n -u test -p test123 sync_remote https://nextcloud.project-insanity.org/remote.php/webdav/

After these steps we’ll have the local folder sync and on the remote target the folder mosh-mosh-1.3.2. Further we have Nextcloud Desktop 2.3.3 (latest stable version), rclone and Dropbox installed.

Starting the tests

Following command starts the synchronisation with Nextcloud Client 2.3.3 (stable):

time nextcloudcmd -h -n -u test -p test123 /home/test/sync https://nextcloud.project-insanity.org/remote.php/webdav/

The prepended time command will return the running time of the command.
In the next step we’ll install Nextcloud Desktop 2.5.0 Beta 2 (which was the latest development version at this time), remove all sync databases and rerun the test:

pacaur -S nextcloud-desktop-git
rm -r sync/._sync* sync/mosh-mosh-1.3.2
time nextcloudcmd -h -n -u test -p test123 /home/test/sync https://nextcloud.project-insanity.org/remote.php/webdav/

Starting the test with rclone is different. You have to setup your Nextcloud instance with the configuration wizard.

rclone config
rm -r sync/._sync* sync/mosh-mosh-1.3.2
time rclone copy /home/test/sync remote:

In case of Dropbox, I had to copy the files into the synced directory and simultaneously copy the remote files to a local folder, to trigger the process. The filestatus command will show when it is finished.

dropbox-cli filestatus
date && cp -r sync/openssh-7.7p1 Dropbox/ && cp -r Dropbox/mosh-mosh-1.3.2 /tmp/
date

Conclusion

This very small and limited test shows that commercial alternatives to Nextcloud such as Dropbox will offer faster and better desktop tools for file synchronization. It is not clear if Nextcloud developers will address this issue any time soon and to my knowledge there are no alternative clients with better performance. Let me know if you know other file synchronization tools for Linux which I can cover in future tests.