IMAP Sync

Sync email via IMAP from host1/domain1 to a subfolder on host2/domain2 via a cron/timer. Can be reversed as well, just update Patterns to exclude the subfolders from being cross-replicated (looped).

Sync email via IMAP from host1/domain1 to a local Maildir folder via cront/timer. This can be used to restore a backup to either host1/domain1 or host2/domain2 as well as by a local MUA (mutt, e.g.).

Note: an argument can be made it's more effecient to sync host1/domain1 to a local Maildir first, then sync the local Maildir to host2/domain2. This however comes at a cost of risk - what if the local drive is full, fails or has an error? One runs the risk of using a possible corrupt local datastore to sync to host2/domain2 when the intent is to have multiple verbatim backups of host1/domain1. Choose wisely.

Preparation

Documentation:

https://isync.sourceforge.io/mbsync.html

Debian package:

Install the isync package: apt-get update && apt-get install isync

Passwords for IMAP must be left on disk in plain text:

Generate "app passwords" at the email providers, host1 can be READ only
Keep ${HOME}/.secure contents on encrypted volume unlocked manually

The mbsync program keeps it's transient index files in ${HOME}/.mbsync/ with one per IMAP folder; these are used to keep track of what it's already synced. Should something break it may be necessary to delete one of these files to force a resync.

By design, mbsync will not delete a destination folder if it's not empty first; this means if you delete a folder and all emails on the source in one step, a sync will break with an error/warning. Instead, delete all emails in the folder first, sync those deletions, then delete the empty folder on the source and sync again. See: https://sourceforge.net/p/isync/mailman/isync-devel/thread/f278216b-f1db-32be-fef2-ccaeea912524%40ojkastl.de/#msg37237271

Crontabs

Simple crontabs to run the scripts, 30 minutes apart:

0 */6 * * * /home/USER/bin/hasync.sh
30 */6 * * * /home/USER/bin/halocal.sh

Config

Main config for the mbsync program:

${HOME}/.mbsyncrc

# Source
IMAPAccount imap-src-account
Host imap.host1.com
Port 993
User user1
PassCmd "cat /home/USER/.secure/psrc"
SSLType IMAPS
SystemCertificates yes
PipeLineDepth 1
#CertificateFile /etc/ssl/certs/ca-certificates.crt

# Dest
IMAPAccount imap-dest-account
Host imap.host2.com
Port 993
User user2
PassCmd "cat /home/USER/.secure/pdst"
SSLType IMAPS
SystemCertificates yes
PipeLineDepth 1
#CertificateFile /etc/ssl/certs/ca-certificates.crt

# Local
MaildirStore local-dest
Path /home/USER/Maildir/
AltMap no
Subfolders Verbatim

# Source map
IMAPStore imap-src
Account imap-src-account

# Dest map
IMAPStore imap-dest
Account imap-dest-account

# Transfer options
Channel hasync
Far :imap-src:
Near :imap-dest:HASync/
Sync Pull
Create Near
Remove Near
Expunge Near
Patterns *
CopyArrivalDate yes

# Local options
Channel dmlocal
Far :imap-src:
Near :local-dest:
Sync Pull
Create Near
Remove Near
Expunge Near
Patterns *
CopyArrivalDate yes

Scripts

These scripts leverage https://healthchecks.io to alert on failure; replace XXXXX with the UUID of your monitor URL.

${HOME}/bin/hasync.sh

#!/bin/bash

# vars
LOGDIR="${HOME}/log"
TIMESTAMP=$(date +%Y-%m-%d_%H%M)
LOGFILE="${LOGDIR}/hasync_${TIMESTAMP}.log"
HCPING="https://hc-ping.com/XXXXXXXXXXXXXXXXXXXXXXXXX"

# preflight
if [[ ! -d "${LOGDIR}" ]]; then
  mkdir -p "${LOGDIR}"
fi

# sync
echo -e "\nBEGIN $(date +%Y-%m-%d_%H%M)\n" >> "${LOGFILE}"
/usr/bin/mbsync -c ${HOME}/.mbsyncrc -V hasync 1>>"${LOGFILE}" 2>&1
EC=$?
echo -e "\nEC: ${EC}" >> "${LOGFILE}"
echo -e "\nEND $(date +%Y-%m-%d_%H%M)\n" >> "${LOGFILE}"

# report
if [[ $EC -eq 0 ]]; then
  curl -fsS -m 10 --retry 5 -o /dev/null "${HCPING}"
  find "${LOGDIR}" -type f -name hasync\* -mtime +30 -delete
fi

exit $EC

${HOME}/bin/halocal.sh

#!/bin/bash

# vars
LOGDIR="${HOME}/log"
TIMESTAMP=$(date +%Y-%m-%d_%H%M)
LOGFILE="${LOGDIR}/halocal_${TIMESTAMP}.log"
HCPING="https://hc-ping.com/XXXXXXXXXXXXXXXXXXXXXXXXX"

# preflight
if [[ ! -d "${LOGDIR}" ]]; then
  mkdir -p "${LOGDIR}"
fi

# sync
echo -e "\nBEGIN $(date +%Y-%m-%d_%H%M)\n" >> "${LOGFILE}"
/usr/bin/mbsync -c ${HOME}/.mbsyncrc -V halocal 1>>"${LOGFILE}" 2>&1
EC=$?
echo -e "\nEC: ${EC}" >> "${LOGFILE}"
echo -e "\nEND $(date +%Y-%m-%d_%H%M)\n" >> "${LOGFILE}"

# report
if [[ $EC -eq 0 ]]; then
  curl -fsS -m 10 --retry 5 -o /dev/null "${HCPING}"
  find "${LOGDIR}" -type f -name halocal\* -mtime +30 -delete
fi

exit $EC

Index

SPDX-License-Identifier: CC-BY-SA-4.0
SPDX-License-Identifier: MIT