User:Imz/research on HOW TO backup an IMAP account

From ALT Linux Wiki
< User:Imz
Revision as of 17:32, 17 November 2009 by Imz (talk | contribs) (→‎Partial transfers (only updates) of the mailboxes: Added subsection for more requirements/criteria/constraints (incorrect-UID-proofness & speed))

(Better and worse ways to backup an IMAP account; a research of the available options.)

I want to maintain a backup of an "IMAP account" (I mean the collection of mailboxes/folders accessed via IMAP by a single user on a remote server), with history.

So I'm assessing the variants of doing this, choosing the optimal combination of individual decisions (regarding the subtasks of this greater task) and tools.

Subtasks and goals/desirable requirements

Tracking the history

I want to be able to revert to a certain state of the mailboxes in the past, or to inspect the past states, because I do not want to hold the trust that no valuable message gets accidentally deleted.

-- I'll probably use git to store the historical changes since it's a familiar (for me) and robust tool.

The format for storing the mailboxes

And for git to handle and show the history in the most sensible way, it's very desirable to use a format which uses one file per one message.

-- Maildir is such well-known format. Moreover[???], each message is not allowed to change its content in IMAP or Maildir: meta-information is stored in the filenames -- this will make the history tracking (the git way) especially lucid.

Contra: Maildir might not be supported by all the software.

Alternatives:

  • the non-"1 file per 1 msg" formats (mbox, etc.) -- not desirable;
  • the [??? mix] format, but although it can use multiple files per mailbox, it arbitrarily groups several messages into one file [1], which is not nice for tracking the history in the most lucid way with git.

Transferring the mailboxes from the IMAP server

Tools that could do that (listed in [2] and found through other Google searches):

Transferring the whole account (many folders at once, with one command)

...
movemail
(no)
mailutil transfer
yes
imapsync
(?)
offlineimap
yes
...

Being able to write to the preferred format -- Maildir

...
movemail
(yes)
mailutil transfer
yes, with a patch -- there are several variants of a Maildir patch for the c-client library:
imapsync
(indirectly yes) (a feature request to do it in a cleaner way)
offlineimap
yes, it's even the only supported format (also writing to IMAP is supported)
...

Partial transfers (only updates) of the mailboxes

Obviously, mainatining the local backup copy of a remote IMAP mailbox with a certain utility, I want that the second and further calls to this utility only transfer the new and updated messages.

...
movemail
no: In my case, "in reality, "movemail" creates second copies of the already downloaded messages. <...> "movemail" has an option with a semantics close to what I want; <...> So, in principle, a mechanism analoguous to UIDL is present in IMAP, and is even more effecient than its POP counterpart, so "movemail" could use it to download only new messages. But in practice, this is not so."
(The mechanism meant is unique ID (UID[3]) of a message in a folder on an IMAP server.)
"Perhaps, storing UIDs is even a problem of the design of the Maildir format, and that makes it more difficult to implement what I wanted" -- [4] -- "But there still seem to exist tools which work with Maildirs and that do track UIDs: http://isync.sourceforge.net/ ; another one: http://mailsync.sourceforge.net/ , but this one tracks Message-ID fields instead" in order to allow partial updating transfers.
mailutil transfer
no: "it also duplicates the messages <...>, but that simply seems to conform to how it was intended to work."
imapsync
(yes, by means other than UID)
offlineimap
(yes, by means of UID)
...

Viability under incorrect UID support on the server =

...
movemail
N/A
mailutil transfer
N/A
imapsync
yes, it doesn't use UIDs.
offlineimap
? (probably, not quite viable w.r.t. partial updates: it will re-download the folder with the new UIDs (all the messages, if the UIDs on the server get reset))
...

Speed =

...
movemail
quite slow (since partial updates are not available)
mailutil transfer
quite slow (since partial updates are not available)
imapsync
?
offlineimap
said to be quite fast
...

Saving keywords, flags

...
movemail
(?)
mailutil transfer
yes (there is a command option for this)
imapsync
(?)
offlineimap
(?)
...

Making the new server to serve the old keywords

uw-imap
(???)
dovecot
uses a certain means to strore the extra keywords in Maildirs[5][6], so if one has saved them, one could prepare them for dovecot.

Saving UIDs (to seamlessly switch to a backup IMAP server)

...
movemail
no, as follows from my experiments with movemail+Maildir
mailutil transfer
no, because of no support for UIDs in the Maildir driver available through the Chappa's patch. But possibly yes with other patches? This must be tested.
imapsync
no, because IMAP doesn't support setting of UIDs AFAIU (Remember that imapsync works via IMAP on both ends: it reads via IMAP and writes via IMAP.)
offlineimap
(?)
...
http://isync.sourceforge.net/
(yes[7])
http://mailsync.sourceforge.net/
(no, "this one tracks Message-ID fields

instead"[8])

...

Making the new server to serve the old UIDs

uw-imap
the Maildir patch (by Chappa) won't do this, because it doesn't at all try to preserve the UIDs between sessions, it doesn't store them [9]
dovecot
has a certain method to store UIDs [10][11]; so if one has the UIDs, one can translate the list of the UIDs into the format used by dovecot.
courier-imap
also stores the UIDs in a special own file[12][13], similarly to doevcot (but with formal differences).
uw-imap with another Maildir patch[14]
handles UIDs, stores them in the filename before the flags (extends the Maildir standard format), so in principle could be also a solution if I want a server to serve the backup copy.
uw-imap with yet another Maildir patch[15]
also stores the UIDs in the filenames, but I guess differently from the previous patch, and the author considers switching to a format compatibe with courier-imap.

Restoring the content on the primary IMAP server

It must be impossible with preservation of UIDs if the access is only through IMAP, but otherwise one can--of course--use the following tools for this task: ...

Conclusion

???