Ditch FTP and Switch to rsync

rsync isn’t new. It’s been around since the late 1990s. So why are we talking about this in 2021?!?

If we’re still using SFTP for transfers to some hosts, it’s likely you may be too. But you may have also heard of rsync. Let’s all sync up and go on this journey together.

 

FTP: It works

Combined, WebDevStudios (WDS) and Maintainn have several customers which use a wide variety of hosts. SFTP works on all hosts, so it’s easy to go with that. But having used rsync for deploys on projects over 15 years ago, I understand the benefits. Eventually, the time came to see if we can adopt using it for future clients and projects.

WDS likes WPEngine. Why not use their git push functionality? As much as I like WPEngine’s git push (it’s fantastic!), we need something that is adaptable across all of our clients.

Does my host have rsync? How does it work?

If your host supports SSH, it supports rsync.

WPEngine added SSH support in 2018, so it was just a matter of time before we switched to rsync deploys for all of our hosts. Here’s how it works:

1. rsync establishes a connection to the remote host (usually via SSH) and runs another rsync receiver process at the destination.
2. The sender and receiver processes compare what files have changed.
3. What has changed gets updated on the remote host.

Only transferring what has changed makes rsync deploys an order of magnitude faster than traditional SFTP deploys where everything is uploaded. rsync also has the ability to compress files during transfer, making it even more efficient.

Configuration

We use a continuous integration service called Buddy for our deploys. It’s awesome because the interface is super easy to use and set up, but you can use rsync with anything from Jenkins to just running the shell command on your computer to upload files. rsync requires the following arguments, but it supports many options and can grow to be quite complex:

rsync [options] src dest

We typically run rsync as:

rsync -avz --delete /path/to/local/directory/ user@host:/path/to/remote/folder/

Before getting into the options, it’s worth noting that rsync is picky about the trailing slash on the source folder. If you omit it from the src folder in the above example, rsync will make a folder called directory inside /path/to/remote/folder/. That might not be what you want. I just always add trailing slashes to both the source and destination folder paths.

The options we’re using are:

• -a Archive mode (a good combination of options to start with if you’re not an rsync master)
• -v Verbose (shows files as they’re being transferred)
• -z Zip (compress files during transfer)
• --delete Delete (delete extraneous files at the destination)

Deleting files on the destination is where you’ll want to exercise caution. It’s a necessary evil for us because if a plugin or theme file gets renamed, we want the old file to go away. To control what gets deleted and what doesn’t, we’ll use an exclude list.

Excludes

Excludes control both what will get transferred from the source, as well as what shouldn’t be messed with at the destination. When you use it in concert with --delete, you’re making sure you don’t nuke something on the server that you want to keep.

There are probably some assets that you don’t need to transfer to your host. Skipping your .git folder with your project’s revision history database will save a bunch of time. You also probably don’t want to upload any node_modules folders, but you do want to upload the vendor folders.

You for sure don’t want rsync to delete your uploads folder; that would be bad. Add an --exclude option for each thing you want to exclude. Your rsync command will start looking like:

rsync -avz --delete 
  --exclude=.git*  
  --exclude=node_modules/ 
  --exclude=/wp-content/uploads/ 
  /path/to/local/wordpress/ user@host:/path/to/remote/wordpress/

The command is starting to get long and there are probably more things you want to exclude like wp-content/upgrade, WPEngine’s mu-plugin suite, etc. Alternatively you can put your excludes in a file (one pattern per line) and reference that instead:

vi rsync-exclude.txt

.git*
node_modules/
/wp-content/uploads/
/wp-content/upgrade/
/wp-content/debug.log
/wp-content/advanced-cache.php
/wp-content/object-cache.php

Then you can reference that file with the --exclude-from option:

rsync -avz --delete --exclude-from=rsync-exclude.txt /path/to/local/wordpress/ user@host:/path/to/remote/wordpress/

Keep in mind that exclude patterns beginning with a slash (/) are at the relative root of the sync folder, not the absolute root of the host. Also, the leading slash instructs rsync to exclude something in a specific location relative to the sync folder. By omitting the slash, rsync will exclude it everywhere.

For instance, the pattern .git* will exclude all .git files and folders (including .github, .gitignore, .gitattributes) throughout the project tree. The pattern /.git/ will just exclude the .git folder in the root of your project.

Extra work, but worth it

Using rsync instead of FTP does require a little extra brain power during setup than SFTP. Always test your setup on a staging site(!) where any mistakes won’t have catastrophic effects.

When using rsync, double-check your command at least 10 times.

—Aubrey Portwood, Senior Backend Engineer

As we refine working rsync setups, we apply those refinements to our deployment template so the next project to migrate to rsync deploys has the latest and greatest. Deploys go from minutes to seconds… well worth the effort!