SMTP AUTH mechanisms

SMTP Authentication is a scheme used to authenticate a sender as valid before allowing them to connect and relay email through your email server. SMTP Authentication is advertised by the SMTP Authentication server, requires a client to authenticate, while finally both parties have to mutually accept and support the chosen authentication procedure. Originally invented as a Host-to-Host protocol, with SMTP Authentication, a User has to identify itself and after successful authentication, reception/transmission of his/her emails is granted.

There are three authentication mechanisms widely used for SMTP Authentication: Login, Plain, and Cram-MD5.

AUTH LOGIN

The most common 'AUTH LOGIN' mechanism looks like this

S: 220 esmtp.example.com ESMTP
C: ehlo client.example.com
S: 250-esmtp.example.com
S: 250-PIPELINING
S: 250-8BITMIME
S: 250-SIZE 255555555
S: 250 AUTH LOGIN PLAIN CRAM-MD5
C: auth login
S: 334 VXNlcm5hbWU6
C: avlsdkfj
S: 334 UGFzc3dvcmQ6
C: lkajsdfvlj
S: 535 authentication failed (#5.7.1)

From all the ESMTP Authentication mechanisms the offered, the client selects 'auth login'. The ESMTP server issues then a '334 VXNlcm5hbWU6' where 'VXNlcm5hbWU6' is a BASE64 encoded string 'Username:'. The client provides the BASE64 encoded user name and the sever responses with the request for the 'Password:' ('334 UGFzc3dvcmQ6'). In the sample above, random input is given and the server finally rejects the authentication request.

AUTH PLAIN

According to IANA's documentation, the PLAIN Authentication is defined in RFC 2245 "Anonymous SASL Mechanism". However, a more useful explanation of the PLAIN Authentication can be found in RFC 2595 "Using TLS with IMAP, POP3 and ACAP" (chapter 6):

"The mechanism consists of a single message from the client to the server. The client sends the authorization identity (identity to login as), followed by a US-ASCII NULL character, followed by the authentication identity (identity whose password will be used), followed by a US-ASCII NULL character, followed by the clear-text password. The client may leave the authorization identity empty to indicate that it is the same as the authentication identity."

In other words, the correct form of the AUTH PLAIN value is 'authid\0userid\0passwd' where '\0' is the null byte.

Some ESMTP AUTH PLAIN implementations don't follow that procedure completely. We see that in the trace using Netscape's 4.8 MUA connecting to a modified Qmail 1.03 to do PLAIN authentication:

C: ehlo client.example.com
S: 220-esmtp.example.com
C: AUTH PLAIN dGVzdAB0ZXN0AHRlc3RwYXNz
S: 235 ok, go ahead (#2.0.0)
C: RCPT TO:<....>

In this sample, the user name was 'test' and the password 'testpass'. Here, the Netscape client immediately blasts the authentication information to the server (including the artificial authorization identity 'test') without waiting for the server to announce his SMTP Auth capabilites.

AUTH CRAM-MD5

While for AUTH PLAIN and LOGIN clear user names and password are transmitted, things go significantly more secure with the CRAM-MD5 authentication mechanism. As already mentioned in it's name, CRAM-MD5 combines a Challenge/Response mechanism to exchange information and a (cryptographic) Message Digest 5 algorithm to encrypt important information.

I use an example based on a posting of Markus Stumpf to the Qmail mailing list. A typical ESMTP AUTH CRAM-MD5 dialog starts like this:

S: 220 popmail.space.net ESMTP
C: ehlo client.example.com
S: 250-popmail.space.net
S: 250-PIPELINING
S: 250-8BITMIME
S: 250-SIZE 0
S: 250 AUTH CRAM-MD5
C: auth cram-md5
S: 334 PDI0NjA5LjEwNDc5MTQwNDZAcG9wbWFpbC5TcGFjZS5OZXQ+

Unlike AUTH LOGIN, the server's response is now a one-time BASE64 encoded 'challenge'. The challenge 'PDI0NjA5LjEwNDc5MTQwNDZAcG9wbWFpbC5TcGFjZS5OZXQ+' translates to '<24609.1047914046@popmail.Space.Net>'. The leading and trailing brackets ('<', '>') are mandatory, as well the portion of the challenge which provides the hostname after the '@'. '24609.1047914046' is a random string, typically build from the 'pid' and the current time stamp to make that challenge unique.

While the user name is transmitted in clear text (but of course BASE64 encoded), the servers's challenge is used by the client to generate a 'digest' from the challenge and the password (which is commonly called 'secret' or 'shared secret' in this context) employing the following MD5 hashing algorithm:

digest = MD5(('secret' XOR opad), MD5(('secret' XOR ipad), challenge))

If both the ESMTP server and the client 'share' the same challenge and secret, the user may now be authenticated successfully by means of the transmitted and BASE 64 encoded 'user name' and 'digest'.

AUTH parameter as part of the 'MAIL FROM:' command

According to RFC 2554, authentication information can optionally provided as ESMTP AUTH parameter with a single value in the 'MAIL FROM:' command. The ESMTP AUTH parameter has to be used in the following way:

C: MAIL FROM:<e=mc2@example.com> AUTH=e+3Dmc2@example.com
S: 250 OK

Here, the AUTH value has to be encoded inside an "xtext" as described in RFC 1891 "SMTP Service Extension for Delivery Status Notifications". RFC 2554 discusses the use of the optional AUTH parameter to the 'MAIL FROM:' command in the context of a "trusted environment to communicate the authentication of individual messages". It actually requires the proliferation of the AUTH information to another MTA (Mail Transfer Agent; eg. email gateway) as AUTH parameter when relaying the message to any server which supports the AUTH extension. In case the authentication is to weak, the Server should set 'AUTH=<>' as parameter to the 'MAIL FROM:' command.

I am not aware, that any MUA implementation using the latter scheme however, some MTA (eg. Postfix) support it.

Qmail 1.03, and in particular qmail-smtpd has no understanding of any parameters in the 'MAIL FROM:' command; it lacks a qualified ESMTP support in that respect. This holds in addition for the ESMTP 'SIZE' announcement (RFC1870), which was partially recovered by Chris Harris' SIZE extension.
My current SMTP-Authentication patch for qmail-smtpd introduces a complete and extensible 'MAIL FROM:' parameter parser and treats the provided AUTH parameter as $TCPREMOTEINFO.

Authentication State

As outlined, RFC 2554 allows two distinct usages of the ESMTP AUTH extension:

1. AUTH parameter exchange as part of the SMTP dialog (as shown above).
2. AUTH as ESMTP parameter in the 'MAIL FROM:' command.

Clearly, this has a significant impact on the authentication state itself. The first approach is actually equivalent with an authenticated SMTP session, while the second is effectively the authentication of the provided 'MAIL FROM:' sender and serves as 'informational' data. Unfortunately, RFC 2554 does not give any hints what an "authenticated" state really means. There is a common sense, that an authenticated user is allowed for unrestricted relaying.

In case the authentication information is transmitted as extension to the 'MAIL FROM:' command, one may treat that equivalently with having an additional 'tcpremoteinfo' - usually provided by means of the 'ident' protocol.

Authentication Aborts

The Client may cancel the authentication request, sending simply a '*' to the server. The server must reject the AUTH procedure and replying the SMTP protocol error '501'. However, the server has to cache the authentication method in order to preserve the state.

Authentication Failures

The server may reject the AUTH request by the client with the following response codes:

534 Authentication mechanism is to weak

538 Encryption required for requested authentication mechanism

454 Temporary authentication failure

530 Authentication required

After a failed ESMTP request, the server has to reset it's state tables and the client may either provide the correct information, or may chose a different authentication mechanism, or may go on in un-authenticated state.

Authentication proliferation

In general, SMTP Authentication allows a one-hop User-to-MTA authentication. An interesting case is to discuss Authentication proliferation. Let's first define what we are talking about:

Typically, a User receives emails by means of the protocols POP3 or IMAP4. For sending, a useful approach would be, that the User - the email originator - sets up an email client (ie. Outlook) for SMTP Authentication and first connects to the Principal-MTA. Here, the user-id and password is stored; which is typically the same as the one used for the POP3/IMAP4 account. In this case, the Principal-MTA acts as SMTP-Relay. Now, we have User-to-MTA Authentication.

What shall this be good for? We have seen, that SMTP Authentication serves mainly to allow unrestricted relaying. With an End-to-End authentication, two additional aims could be achieved:

1. The authenticity of the message itself (the content of the email) can be guaranteed,
2. The uniqueness and authenticity of the email's originator (the provided Mail From: <Return-Path>) can be ensured.

The latter is a requirement for the first, since it enables to reject emails with forged/spoofed "Return-Path" addresses.

In order to maintain an authentication chain for the User's MUA, not only the user-id and password has to be proliferated, but rather in addition the "Return-Path" address. In this respect, the Mail From: <Return-Path> acts as authorization information.
Ironically, this concept was already introduced for the AUTH PLAIN authentication scheme (as discussed above) and later dropped. Unfortunately, with today's SMTP Authentication, an Authentication proliferation is not possible without changing the standard.
Today, we see a huge activity to demand authentication in email traffic, in order to reduce the spam load. As outlined, ensuring authentication for emails is to weak to reduce spam; additionally, qualified authorization information has to be included.

Authentication information in the email "Received:" header [RFC 3848]

One - actually inadequate - attempt in this direction is to add authentication information into the email header, which is required by RFC 3848. The standard SMTP Authentication patches for qmail-smtpd incude the authenticated user equivalent to the tcpremoteinfo in the Received header:

Received: from xdsl-81-173-228-159.netcologne.de (HELO mail.fehnet.net) (erwin@fehcom.net@81.173.228.159)
by hamburg134 with SMTP; 23 Jan 2005 11:53:28 -0000

Though the information erwin@fehcom.net@81.173.228.159 is rather precise, it lacks the knowledge, how it is derived. RFC 3848 requires a different notation, which is incorporated in my most recent SMTP authentication patches for qmail:

Received: from xdsl-81-173-228-159.netcologne.de (HELO mail.fehnet.net) (erwin@fehcom.net@81.173.228.159)
by hamburg134 with ESMTPA; 23 Jan 2005 13:32:13 -0000

The keyword EMSTPA denotes "ESMTP Authentication" and thus the information presented can be clearly interpreted. However, the quality of this information can not be trusted, if it does not originate from the last receiving host.
Some Anti-Spam programs, like SpamAssassin begin to use this information including it in the spam-weight calculation of the message. As pointed out by Dary C.W. O'Shea (Committer of the Apache SpamAssassin) the "trust boundary extension", which deals with the interpretation of the email header, works in a top-down approach, in order to verify the integrity of the presented information.
Since any email header can be forged easily, additional checks for each SMTP connection have to be facilitated, in order to minimize any potential forgery. Thus, the basic problem remains to derive trust-worth information from a per-se un-trusty environment.

Mail SUBMISSION [RFC 4409]

While the standard SMTP port 25 is used for unrestricted email reception, in particular DSL and cable providers would like to setup their MTAs for their customers on a different port and requiring ESMTP Authentication. According to RFC 4409, the mail submission port defaults to 587. A MTA listening on that port will demand a successful SMTP authentication prior of accepting the 'MAIL FROM:' command.; otherwise an error is issued:

530 Authorization required (#5.7.1)