The IMAP Protocol

A quick overview of IMAP

IMAP stands for Internet Message Access Protocol. It is a method of accessing electronic mail or bulletin board messages that are kept on a (possibly shared) mail server. In other words, it permits a "client" email program to access remote message stores as if they were local. For example, email stored on an IMAP server can be manipulated from a desktop computer at home, a workstation at the office, and a notebook computer while traveling, without the need to transfer messages or files back and forth between these computers.

IMAP's ability to access messages (both new and saved) from more than one computer has become extremely important as reliance on electronic messaging and use of multiple computers increase, but this functionality cannot be taken for granted: the widely used Post Office Protocol (POP) works best when one has only a single computer, since it was designed to support "offline" message access, wherein messages are downloaded and then deleted from the mail server. This mode of access is not compatible with access from multiple computers since it tends to sprinkle messages across all of the computers used for mail access. Thus, unless all of those machines share a common file system, the offline mode of access that POP was designed to support effectively ties the user to one computer for message storage and manipulation.

Key goals for IMAP include:

  1. Be fully compatible with Internet messaging standards, e.g. MIME.
  2. Allow message access and management from more than one computer.
  3. Allow access without reliance on less efficient file access protocols.
  4. Provide support for "online", "offline", and "disconnected" access modes
  5. Support for concurrent access to shared mailboxes
  6. Client software needs no knowledge about the server's file store format.

The protocol includes operations for creating, deleting, and renaming mailboxes; checking for new messages; permanently removing messages; setting and clearing flags; server-based RFC-822 and MIME parsing (so clients don't need to), and searching; and selective fetching of message attributes, texts, and portions thereof for efficiency.

IMAP was originally developed in 1986 at Stanford University. However, it did not command the attention of mainstream email vendors until a decade later, and it is still not as well-known as earlier and less-capable alternatives such as POP, though that is rapidly changing. With webmail products becoming increasingly popular, the use of the IMAP protocol is increasing quite rapidly.

IMAP Commands

Once an IMAP session is established, all communication between the client and server takes place in the form of commands sent by the client and responses returned by the server. Like POP3, commands and responses are sent as strings of ASCII text and terminated with a ?CRLF? sequence, making them compatible with the way data is sent using the Telnet Protocol. However, IMAP does a few things quite differently than POP and many other TCP/IP application protocols.

The first interesting thing about IMAP commands is that most are not abbreviated into codes of three or four letters?they are spelled out in full. So, where POP3 has a STAT command, the one in IMAP is called STATUS. Commands are normally shown in upper case, as I do in this Guide, but are in fact case-insensitive.

IMAP also uses an interesting system of command tagging to explicitly match client commands with certain server responses. Each time a client sends a command, it prefixes it with a tag that is unique for the particular session. The tags are usually short strings with a monotonically increasing number in them; the examples in the IMAP standards have the first command tagged ?a0001?, then the second ?a0002? and so on. That said, as long as each command is uniquely labelled, it doesn't matter what tagging scheme is used. When the server needs to send a response that is specific to a command, it tags the reply with the appropriate command tag. Not all replies are tagged, however.

The standard doesn't state explicitly why this tagging scheme is needed, but I believe it is probably related to IMAP's multiple command feature. IMAP clients are allowed to send a sequence of commands to the server to be processed, rather than only sending commands one at a time. This can improve performance when certain commands would take a long time to complete. The only restriction is that the commands must be independent enough that the result of executing them all would be the same regardless of the order in which they were processed. For example, sending a command to read a particular entity in combination with a command to store a value into the same entity is not allowed.

Example Commands

Server Capability

To check the capabilities of a server, issue the following two commands:
a001 CAPABILITY
a002 logout

This would look like:

[]:[8:48am]:[/home/rnejdl] > telnet mail 143
Trying 66.13.175.242...
Connected to tethys.ringofsaturn.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LITERAL+ SASL-IR LOGIN-REFERRALS AUTH=LOGIN] tethys.r
ingofsaturn.com IMAP4rev1 2004.350 at Sun, 8 Aug 2004 13:51:21 -0500 (CDT)
a001 CAPABILITY
* CAPABILITY IMAP4REV1 LITERAL+ IDLE NAMESPACE MAILBOX-REFERRALS BINARY UNSELECT
 SCAN SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND SASL-IR LOGIN-REF
ERRALS AUTH=LOGIN
a001 OK CAPABILITY completed
a002 logout
* BYE tethys.ringofsaturn.com IMAP4rev1 server terminating connection
a002 OK LOGOUT completed
Connection closed by foreign host.
Testing Logins

Assume your username is smith, with password blegga, and your mailserver is mserver.mynet.com. First we connect to the imap server, which is on port 143, as follows:

[]:[8:49am]:[/home/rnejdl] > telnet mserver 143
Trying 192.168.1.2...
Connected to mserver.mynet.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=LOGIN] mserver.mynet.co
m IMAP4rev1 2001.315rh at Sat, 30 Aug 2003 11:08:06 -0700 (PDT)

Then we login (obviously, use your account and password, not "smith" and "blegga"!):

a01 login smith blegga
RESPONSE: a01 OK User logged in
Listing all Folders

The list command will list all available folders. This list could be quite large, so be careful. On my mail server, all mail is stored under the Mail directory:

a02 list "Mail" "*"
* LIST (\NoSelect) "/" Mail
* LIST (\NoInferiors \UnMarked) "/" Mail/dc214
* LIST (\NoInferiors \Marked) "/" Mail/Trash
* LIST (\NoInferiors \UnMarked) "/" Mail/Snort
* LIST (\NoInferiors \Marked) "/" Mail/Sent
* LIST (\NoInferiors \Marked) "/" Mail/DDoW
* LIST (\NoInferiors \UnMarked) "/" Mail/CERT
* LIST (\NoInferiors \UnMarked) "/" Mail/Drafts
* LIST (\NoSelect) "/" Mail/inbox
* LIST (\NoSelect) "/" Mail/inbox/new
* LIST (\NoSelect) "/" Mail/inbox/cur
* LIST (\NoSelect) "/" Mail/inbox/tmp
* LIST (\NoInferiors \UnMarked) "/" Mail/.inbox.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.CERT.index.ids
* LIST (\NoInferiors \UnMarked) "/" Mail/.CERT.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.dc214.index.ids
* LIST (\NoInferiors \UnMarked) "/" Mail/.dc214.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.DDoW.index.ids
* LIST (\NoInferiors \UnMarked) "/" Mail/.DDoW.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.Drafts.index.ids
* LIST (\NoInferiors \UnMarked) "/" Mail/.Drafts.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.Sent.index.ids
* LIST (\NoInferiors \UnMarked) "/" Mail/.Sent.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.Snort.index.ids
* LIST (\NoInferiors \UnMarked) "/" Mail/.Snort.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.Trash.index.ids
* LIST (\NoInferiors \UnMarked) "/" Mail/.Trash.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.inbox.index.ids
* LIST (\NoSelect) "/" Mail/outbox
* LIST (\NoSelect) "/" Mail/outbox/new
* LIST (\NoSelect) "/" Mail/outbox/cur
* LIST (\NoInferiors \UnMarked) "/" Mail/outbox/cur/1089121854.99444_1.tethys:2,S
* LIST (\NoInferiors \UnMarked) "/" Mail/outbox/cur/1090216501.56228_21.tethys:2,S
* LIST (\NoSelect) "/" Mail/outbox/tmp
* LIST (\NoInferiors \UnMarked) "/" Mail/.outbox.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.outbox.index.ids
* LIST (\NoSelect) "/" Mail/sent-mail
* LIST (\NoSelect) "/" Mail/sent-mail/new
* LIST (\NoSelect) "/" Mail/sent-mail/cur
* LIST (\NoSelect) "/" Mail/sent-mail/tmp
* LIST (\NoInferiors \UnMarked) "/" Mail/.sent-mail.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.sent-mail.index.ids
* LIST (\NoSelect) "/" Mail/trash
* LIST (\NoSelect) "/" Mail/trash/new
* LIST (\NoSelect) "/" Mail/trash/cur
* LIST (\NoSelect) "/" Mail/trash/tmp
* LIST (\NoInferiors \UnMarked) "/" Mail/.trash.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.trash.index.ids
* LIST (\NoSelect) "/" Mail/drafts
* LIST (\NoSelect) "/" Mail/drafts/new
* LIST (\NoSelect) "/" Mail/drafts/cur
* LIST (\NoSelect) "/" Mail/drafts/tmp
* LIST (\NoInferiors \UnMarked) "/" Mail/.drafts.index
* LIST (\NoInferiors \UnMarked) "/" Mail/.drafts.index.ids
* LIST (\NoInferiors \UnMarked) "/" Mail/FreeBSD
a02 OK LIST complete
Testing Mailbox Speed

I've had some issues with the speed of my webmail system that uses University of Washington's IMAP server. As part of testing and troubleshooting, I wanted to see if this was an issue with my IMAP server. So, after logging into port 143 as above, I want to see how long it takes for the server to return from a SELECT command, which is used by my webmail to generate an index of the INBOX.

After logging, in SELECT the mailbox.

a03 SELECT INBOX
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
* 1242 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1062186210]
* OK [UIDNEXT 1246]
a03 OK [READ-WRITE] Completed

Note that this is testing to see how long it takes your INBOX to load. If you want to test some other mailbox, say "mail/hugebox", type this now: a03 SELECT mail/hugebox

Fetching a message

The FETCH command allows you to fetch a whole mailbox or parts of a message. Since I love examples rather than lengthy explanations, I've included 3 different uses of FETCH.

Here's an example of me pulling a FETCH of the FLAGS for all messages from INBOX:

a04 FETCH 1:* FLAGS
* 1 FETCH (FLAGS (\Seen))
* 2 FETCH (FLAGS (\Seen))
* 3 FETCH (FLAGS (\Seen))
* 4 FETCH (FLAGS (\Seen))
* 5 FETCH (FLAGS (\Seen))
* 6 FETCH (FLAGS (\Seen \Answered))
* 7 FETCH (FLAGS (\Seen))
* 8 FETCH (FLAGS (\Seen))
* 9 FETCH (FLAGS (\Seen))
* 10 FETCH (FLAGS (\Seen))
* 11 FETCH (FLAGS (\Seen))
* 12 FETCH (FLAGS (\Seen))
* 13 FETCH (FLAGS (\Seen))
* 14 FETCH (FLAGS (\Seen))
* 15 FETCH (FLAGS (\Seen \Answered))
* 16 FETCH (FLAGS (\Seen))
* 17 FETCH (FLAGS (\Seen))
* 18 FETCH (FLAGS (\Seen \Answered))
* 19 FETCH (FLAGS (\Seen))
* 20 FETCH (FLAGS (\Seen))
* 21 FETCH (FLAGS (\Seen \Answered))
* 22 FETCH (FLAGS (\Seen))
* 23 FETCH (FLAGS (\Seen \Answered))
* 24 FETCH (FLAGS (\Seen))
* 25 FETCH (FLAGS (\Seen))
* 26 FETCH (FLAGS (\Seen))
* 27 FETCH (FLAGS (\Seen \Answered))
* 28 FETCH (FLAGS (\Seen))
* 29 FETCH (FLAGS (\Seen \Answered))
* 30 FETCH (FLAGS (\Seen))
* 31 FETCH (FLAGS (\Seen \Answered))
* 32 FETCH (FLAGS (\Seen \Answered))
* 33 FETCH (FLAGS (\Seen \Answered))
* 34 FETCH (FLAGS (\Seen \Answered))
* 35 FETCH (FLAGS (\Seen))
* 36 FETCH (FLAGS (\Seen))
* 37 FETCH (FLAGS (\Seen))
* 38 FETCH (FLAGS (\Seen))
* 39 FETCH (FLAGS (\Seen))
* 40 FETCH (FLAGS (\Seen))
* 41 FETCH (FLAGS (\Seen))
* 42 FETCH (FLAGS (\Seen))
* 43 FETCH (FLAGS (\Seen \Answered))
* 44 FETCH (FLAGS (\Seen))
* 45 FETCH (FLAGS (\Seen))
* 46 FETCH (FLAGS (\Seen \Answered))
* 47 FETCH (FLAGS (\Seen))
* 48 FETCH (FLAGS (\Seen))
* 49 FETCH (FLAGS (\Seen))
* 50 FETCH (FLAGS (\Seen \Answered))
* 51 FETCH (FLAGS (\Seen))
* 52 FETCH (FLAGS (\Seen))
* 53 FETCH (FLAGS (\Seen))
* 54 FETCH (FLAGS (\Seen))
* 55 FETCH (FLAGS (\Seen))
* 56 FETCH (FLAGS (\Seen \Answered))
* 57 FETCH (FLAGS (\Seen))
* 58 FETCH (FLAGS (\Seen \Answered))
* 59 FETCH (FLAGS (\Seen))
* 60 FETCH (FLAGS (\Seen \Answered))
* 61 FETCH (FLAGS (\Seen))
* 62 FETCH (FLAGS (\Seen))
* 63 FETCH (FLAGS (\Seen \Answered))
a04 OK FETCH completed

I am pulling the first message from my inbox here and full refers to full headers. One thing to note is that the first message is the last one chonologically.

a05 FETCH 1 full
* 1 FETCH (FLAGS (\Seen) INTERNALDATE "15-Mar-2000 13:10:14 -0500" RFC822.SIZE 1
553 ENVELOPE ("Wed, 15 Mar 2000 13:10:11 -0600" "Perl Stuff" (("Rusty Nejdl" NIL
 "rnejdl" "verio.net")) (("Rusty Nejdl" NIL "rnejdl" "verio.net")) (("Rusty Nejd
l" NIL "rnejdl" "verio.net")) ((NIL NIL "rnejdl" "verio.net")) ((NIL NIL "ttodd"
 "verio.net")) NIL NIL "<20000315131011.A5381@tethys.ringofsaturn.com>") BODY ("
TEXT" "PLAIN" ("CHARSET" "us-ascii") NIL NIL "7BIT" 359 9))
a05 OK FETCH completed

Finally, let's get the text of the message:

a06 FETCH 1 body[text]
* 1 FETCH (BODY[TEXT] {359}
List of commands:
http://www.spu.edu/help/tech/basic-perl/functions.html
Good into class:
http://www.uga.edu/~ucns/wsg/unix/perl/course/introduction.html
Relational Database overview:
http://wdvl.internet.com/Authoring/DB/Intro/client_server_databases.html
--
Rusty Nejdl 
"If it ain't broke, it doesn't have enough features yet."
)
a06 OK FETCH completed
Logout

Easily enough, just type:

a07 logout
* BYE LOGOUT received
a07 OK Completed
Connection Dropped

If you see connections dropped like the following, then another program is checking IMAP mail at the same time. This will cause problems with both mail clients since IMAP doesn't handle this case nicely.

a04 FET* BYE Lost mailbox lock
Connection closed by foreign host.

Further Reading

This is probably the most thorought and least exampled site I found on the IMAP protocol. Lots of information and such that will nicely compliment what I wrote above, which is largely example based:

TCP/IP Internet Message Access Protocol (IMAP/IMAP4)