Zimbra Open Source backup tools

http://stdout.no/zimbra-open-source-backup-strategy-and-scripts/

Zimbra Open Source is a great way to get a complete self hosted mail solution at zero cost, provided you have a server and don’t mind a bit of tinkering.

One feature missing from the open source version is a system for backup and restore of user accounts. It does have several utilities and features built in that can be put to use, but you have to script it yourself, and there are some quirks and gotchas.

This post documents what I learned about Zimbra’s utilities while writing my own scripts.

I use Zimbra 8, but this should apply to Zimbra 7 as well.

All commands described below can be run as root on your Zimbra server unless otherwise specified, but it is assumed that you have your PATH set up as follows:

export PATH=$PATH:/opt/zimbra/bin

Backing up an account

Thankfully, the most important data is easy to back up.

This stores the account user@domain.com to a file:

zmmailbox -z -m user@domain.com getRestURL '/?fmt=tgz' > user@domain.com.tgz

The file user@domain.com.tgz will contain:

  • Mail
  • Contacts
  • Calendars
  • Briefcase
  • Tasks
  • Searches
  • Tags
  • Folders

All subfolders are included, except Junk and Trash. There is no way to include these in the big dump, but they can be exported separately:

zmmailbox -z -m user@domain.com getRestURL '/Junk?fmt=tgz' > user@domain.com-junk.tgz
zmmailbox -z -m user@domain.com getRestURL '/Trash?fmt=tgz' > user@domain.com-trash.tgz

If you open any one of these tgz files you will find that messages and files are stored in their original format, so they can even be “recovered” manually by just poking around in the archive. Nice!

Note also the use of the tgz format. Many scripts and examples you find online are outdated and recommend using zip, but Zimbra only includes metadata in .meta files when using tgz. You need this metadata for features like tags and searches, for message flags, to allow seamless reassembly of large directories that have been split, and more. Always use tgz.

It may take some time to create a backup like this. On my reasonably fast test server with nothing else going on it took 2 minutes to backup an account with a few thousand messages, resulting in a 500 MB archive.

Restoring an account

Given tgz files created as above, a full restore of an account is also quite simple.

The account you restore to has to exist, and you have to decide how to handle existing content. Given the flexible options available it will likely work just fine to restore into an existing account.

There are four modes of restoring, of which three are relevant:

  • skip
    • Restore deleted items
    • Ignore existing items completely
  • modify
    • Restore deleted items
    • Update existing items to match backup (unread flags etc.)
  • reset
    • Delete all contents of the account
    • Restore the backup into the now empty account

If all you need is to restore deleted items, then skip should be safe to use on an existing account, with a minimum of disruption for the user.

On the other hand, make sure you know what you’re doing before using reset, as there is no undo.

Once you have chosen a mode, a restore is performed like this:

zmmailbox -z -m user@domain.com postRestURL "/?fmt=tgz&resolve=skip" user@domain.com.tgz

(Change resolve=skip to resolve=modify or resolve=reset, as required.)

More info on these options can be found on the Zimbra blog.

Partial restore

What if you only want to restore the user’s contacts, or some other specific folder?

This can be done by unpacking the tgz archive and creating a new one containing only the folder you want to restore.

Unfortunately, Zimbra is very picky when it comes to the metadata in the tgz archive, so simply untarring and creating a new tarball manually will produce unpredictable results, often silently ignoring parts of the backup when restoring, or mixing up metadata.

Here is a Python script that can extract parts of a user backup with all metadata intact, allowing restores to work as expected:

#!/usr/bin/env python
#
# backupextract.py
#
# extract matching parts of a Zimbra user backup in tgz format

import re
import sys
import tarfile

if len(sys.argv) != 4:
    sys.exit("Usage: %s <srcfile.tgz> <dstfile.tgz> <regex>" % sys.argv[0])

try:
    src = tarfile.open(sys.argv[1])
    dst = tarfile.open(sys.argv[2], 'w:gz')
except Exception, e:
    sys.exit("Error: %s" % e)

for f in src.getmembers():
    m = re.search(sys.argv[3], f.name)
    if m:
        dst.addfile(f, src.extractfile(f))

dst.close()
src.close()

If you have a full user backup but want to restore only the contacts folder, first use this script like this (the last parameter is a regex):

backupextract.py user@domain.com.tgz contacts.tgz '^Contacts/'

Then verify that you got what you expected:

tar tvzf contacts.tgz

Finally perform the restore like normal:

zmmailbox -z -m user@domain.com postRestURL "/?fmt=tgz&resolve=skip" contacts.tgz

NOTE! A warning is in order here. This restore will work just fine with skip or modify, but if you run it with reset you will soon find your entire account completely empty except for your contacts.

To use reset safely when restoring a single folder, make sure to also include the folder in the postRestURL argument, like so:

zmmailbox -z -m user@domain.com postRestURL "/Contacts?fmt=tgz&resolve=reset" contacts.tgz

This will delete all contents of the Contacts folder and replace it with the contents of the backup.

You might think (as I did) that including the folder name in the restore command will allow you to restore only that folder from a full backup without having to mess with extracting parts of backup archives. Unfortunately, this is not the case. Instead Zimbra will happily start creating your entire folder structure below your Contacts folder, causing a bit of a mess. Go figure.

Backing up user preferences

Settings and preferences are easy to back up, but not so easy to restore.

This will grab most of it for a specific account:

zmprov getAccount user@domain.com > user@domain.com-settings.txt

Both administrative settings and preferences the user is allowed to change are included, such as:

  • Aliases
  • Filters
  • Preferences
  • Enabled features
  • Policies

Not all of the settings you get in this output can be configured from the command line, but most can. How to do this is well documented, and is usually a matter of something like this:

zmprov modifyAccount user@domain.com zimbraPrefMailFlashIcon TRUE

Good sources of information include Zimbra’s own zmprov reference and section on bulk provisioning.

It’s not exactly automated, but my own restore procedure is typically something like this:

  1. Retrieve output from zmprov getAccount from backup
  2. Run zmprov getAccount on current account
  3. Put old and new output through sort
  4. Do a diff and manually correct the discrepancies

Since I don’t have to do this too often, this is good enough.

Backing up LDAP

The LDAP database holds all user accounts, including passwords in the form of hashes, as well as distribution lists. Most preferences can be found in LDAP as well, although zmprov is the preferred way of accessing it. The LDAP database should definitely be backed up.

Zimbra comes with its own slapcat variant called zmslapcat.

This backs up the main LDAP database with users, passwords and distribution lists:

su zimbra -c "/opt/zimbra/libexec/zmslapcat /tmp"

The backup will be stored here:

/tmp/ldap.bak

To get everything, make a separate backup with the -c (config) option, even if this means we get some data twice:

su zimbra -c "/opt/zimbra/libexec/zmslapcat -c /tmp"

This backup ends up here:

/tmp/ldap-config.bak

Restore is essentially a standard OpenLDAP restore, which means deleting the entire LDAP database and loading it from backup.

See also:

Other considerations

You still want to supplement any scripts you make with a regular file backup of at least /etc and /opt on your server.

There is also the matter of the rest of your Zimbra server configuration. Much of it is included in the LDAP export, but in that form it is rather difficult to parse and restore, so a simple script based on zmprov would probably be better. A starting point might be to search for all getAll* calls in the zmprov reference.

Links and references

Useful information from Zimbra:

Other Zimbra backup tools and scripts: