SMTP was designed as a Host-to-Host protocol with no authentication in mind. In fact, the protocol lacks any kind of credentials which are important today. Many criticism has been raised regarding this fact, but let's try to straighten out the basics.
In general -- regarding email -- it is possible to target anybody. Once you've got an email address as recipient, you are subject of solicitated an unsolicitated emails. You need to control it. However, regarding the sender two scenarios exist:
- The sender is permitted to send emails from the origin domain. This permission may be granted by individual authentication. The credentials need to be checked by the sending MTA.
- The sender is granted permission to send to any domain. This may be called authorization. The credentials can be checked by the receiving MTA.
The permission to send by user is covered by SMTP Authentication. The permission to send per domain can an verified against the sender's IP address of the sending host by means of available SPF records on in the DNS or by means of cryptographic signatures in the email, known as DKIM again depending and DNS domain public keys deposited in the DNS.
|SMTP Authentication||... to send emails by (authenticated) user outside the origin domain||allow unrestricted relaying||RFC 4954||5.6, 5.9|
|X.509 Client Certificate||... to send emails by (trusted) user outside the origin domain||allow unrestricted relaying||[None; proprietary]||5.5, 5.9|
|Sending Domain Authorization (SPF)||... to receive emails from specific IP addresses granted by DNS SPF records||prevent email fraud||RFC 4408||5.10|
|DomainKeys Identified Mails (DKIM)||... to receive emails for individual cryptographically signed mails verified by the origin domain's public key issued in the DNS||prevent email fraud and qualify sending MTA||RFC 6376||[not covered here]|
s/qmail supports both SMTP Authentication and SPF Authorization together with X.509 client certificates for ESMTP(S) and QMTPS.
DKIM is possible with some extensions originally forseen for qmail:
- For generating signed emails, you need to replace qmail-remote by a wrapper.
- For verifying signed emails, Russ Nelson's patch for qmail-queue will do the job.
Note: The DKIM mechanism depends on the framework libdomainkey hosted on SourceForge which is not maintained any more.
Every 'Internet citizen' has to be addressed by a valid email address. An email address consists of a local part - the 'mailbox' name and the domain part - telling to which domain the email has to be forwarded to: typically sender@origindomain.
The 'domain part' is subject of the DNS. A valid SMTP recipient domain needs to have a SMTP host to be responsible and to be able to receive emails for this domain.
The 'local part' simply tells, that the user has a valid 'account' for this domain.
First: Sending an email with SMTP envelope address <sender@origindomain> implies, that originator sender posses a reachable mailbox at origindomain -- which might be subject of a test.
Second: Sending an email to <recipient@targetdomain> will only be successful if recipient has a mailbox at targetdomain -- which might be verified in the SMTP dialogue.
The situation can be modelled in the following way:
Thus, we can identify three scopes -- or domains -- to testify the required qualifications:
- The sending MTA has to testify the credentials of sender. Details are discussed in this chapter.
- Within the Message Transfer System (MTS) the MTAs have to obligation to check whether both sending and receiving MTA do have qualified entries in the DNS In general, different qualification (and anti-qualification - like RBLs) schemes do exist. SPF (Sender Policy Framework) is covered in this chapter.
- Receiving an email for recipient@targetdomain will only be possible if recipient posses a Mailbox (or has at least an Alias entry).
- The sender of the email is responsible to be reachable via <sender@origindomain>. Once this is verified, he has the credentials to send this email.
- The sending host has the responsibility to verify the existence of the mailbox sender at origindomain prior of sending an email to <recipient@targetdomain>.
- In consequence, once an email is send from <sender@origindomain> to <recipient@targetdomain>, we assume in the current SMTP model that this case is valid: Sender (Originator) and Recipient are Internet citizens.
- In order for <sender@origindomain> to send an email to <recipient@targetdomain> the sending host may require additional qualifications:
- Based on the IP-address of the sender.
- Based on some additional qualification procedure like POP-before-SMTP.
- Based on SMTP-Authentication (this chapter).
- Based on X.509 client certificates (this chapter).
Thus, regarding the sending of email, we have two situations:
- Domain qualified email sender (ie. Web Shops' sending IP).
- User qualified email sender (based on userid/password or X.509 certs).
The question for the recipient of this email is:
Can we differentiate between those sources?
The answer is: No. We neither can trust any information based on (2.) nor should discriminate emails from source (1.).
An identity is a subject possessing distinguishable attributes; at least a (user)name to address the subject and a particular attribute, the password. The name is the public part of the identity while the password is used to authenticate the identity among others. We call this a 'One-factor' Authentication.
In this scheme, both the name (Userid) and the password are static and don't depend on the (current) authentication. If, however the current timestamp of the authentication process is used as salt (among other information, like the IP address), one may construct a One-Time Password (OTP) which is used and valid only for this authentication request.
The identity which wants to get authenticated is called the Supplicant; while the system, which has knowledge about the (set of) identities is the Identity Provider (IdP). There might be a third party involved, the Identity Broker which is typically available in terms of a Shibboleth service.
Other schemes depend not only on knowledge but rather include some additional 'hardware' which needs to be uniquely issued by the IdP: a token, an USB stick, a smartcard ... If both, this hardware piece needs to be available during authentication as well together with the password we consider this a Two-factor Authentication.
The Identity Provider (IdP) is a system able to store the attributes
of an identity persistently and allowing a lookup. Typically, the IdP
can be considered as Authenticator, a server.
The IdP not only holds the user database, but additionally is able to support different authentication mechanisms.
One important security issue for the IdP is, to protect the user database (and her in particular the passwords) thus an untrusted third party, even in case the user data are proliferated by mistake, is not able to reconstruct the users's passwords. The Unix shadow password and crypt facility are perfect means for those, since the allow (in the running) Unix system access only to a root-owned process and protect the passwords by individual salting.
Other IdPs may use a standard SQL database-access or employ LDAP lookup as back-end.
During qmail's development, several authentication modules have been realized as Pluggable Authentication Module, a PAM. D.J. Bernsten has provided a versatile authentication framework, the checkpassword interface.
Within s/qmail, the qmail-authuser program is available, supporting a file-based authentication, providing access to the Unix user database, and including a call to Vopmail's vcheckpw and VmailMgr's checkvpw in one go.
Thus, qmail-authuser is sqmail's swiss knife for authentication purposes, superseding the older mechanisms; though they still can be used. It supports user, login, plain (which require a 'plaintext' password), and Challenge/Response (CR) authentication; however this depends on the back-end (the IdP) as well. qmail-authuser can be used for (E)SMTP and POP3 authentication.
qmail-authuser depends on a 'tokenized' password scheme in the user database which follows some ideas already outlined for the 'Recipient' extension. Let's explain that by example:
|username(@domain)||password||Plaintext or CRAM/MD5 evaluation|
|username(@domain)||%||hashed password (MD5, SHA1, SHA256)||Plaintext only evaluation|
|username(@domain) || * (any user)||?||(not required)||Plaintext only against Unix crypt|
|username(@domain) || * (any user)||!||(not required)||Plaintext only using Unix crypt + return environment settings for username|
|username(@domain) || @domain || @ (any user/any domain)||+||(not required)||Plaintext or C/R evaluation; calling checkvpw as secondary PAM|
|username(@domain) || @domain || @ (any user/any domain)||&||(not required)||Plaintext, calling vchkpw as secondary PAM|
|username(@domain) || @domain || @ (any user/any domain)||=||(not required)||using Dovecot as IdP wrapper/backend for plaintext and C/R passwords|
The 'token' is part of the password and removed prior of authentication and is used to define the authentication mechanism attached to particular userid. The userid may be a single username or may be constructed as username@domain where both parts are considered independently. The settings are tried in most significant order of the userid and in fall-thru mode for all methods. Thus, you can mix 'local' users with virtual domain users for the different virtual domain handlers.
qmail-authuser password evaluation and storages methods haven enhanced to support SHA1 and SHA256 alongside with MD5 hashes. Given, complex and unique passwords are used, this allows to protect the user's individual password even in case the server is compromised.
This mechanism can (and will) be expanded for salted passwords as well, but requires the maintenance of a cdb and a bookkeeping UI module.
qmail-authuser copes now with Dovecot's authentication provisioning system directly, which requires to invoke a local domain socket within Dovecot; typically within the file 10-master.conf:
The location and path of the domain socket can be chosen arbitrarily; however it is necessary to tell qmail-authuser where to find it:
qmail-authuser will now call the doveadm module with the argument auth test -a and 'feeding' this socket with the provided authentication data given on FD 3. The additional argument 'true' is necessary for calls of qmail-authuser.
Note: Potentially, the following configuration is required in the file /etc/pam.d/dovecot:
What does this mean?
- First, it means to generate X.509 certificates + private keyfiles for individual users, to distribute those to them and to include those to their Mail User Agent (MUA). Of course, the MUA needs to support it. However even very old clients like will do. One exception is the Blackberry's mail client, where this is reserved to be used with an Blackberry Enterprise Server; a stupid idea.
- Second, one has to decide whether those X.509 certs are public or private certs.
Private X.509 certs are preferable and can be generated with OpenSSL easily. One import issue is, to include the email address of the cert's owner into the Distinguished Name (DN) as attribute.
- Those client X.509 certs need to be signed by a local CA. The local root cert, need to be included for verification purpose in case of s/qmail to sslserver.
- The server has to be advised to request client certificates.
qmail-smtpd and qmail-qmtpd recognize the presence of the X.509 client cert and will accept the MUA as relayclient. Further, the validated X.509 client cert and it's DN will be included in the Received header of the email.
Note 1: qmail-smtpd can require the match of the email address in the cert's DN with the used SMTP originator address.
Note 2: SMTP Authentication needs to be enforced by means of the environment variable SMTPAUTH (see below).
Using sslserver to allow X.509 client cert based authentication the following steps are necessary:
- For each User a X.509 client certificate (+ keyfile in PEM format) needs to generated which is digitally signed by a (local) CA. Those X.509 certs are only useful in your down administrative domain and are solely forseen for email authentication purpose. You may alternatively issue more generic X.509 client certs; however since the (private) keyfile to always bound to a MUA, this is not without risk. In addition, the keyfile can be password protected to prevent abuse.
- The signing CA X.509 root Cert (PEM format) has to included in sslserver key repository: CCAFILE=path_to_CA_signing_root_file(s). It is possible to provide several CA root Certs for different domains/customers (in one common file).
- sslserver has to be advised to request a client cert per connection. The (global) option sslserver -m can be used, or alternatively the environment variable CCAVERIFY="" is provided in sslserver cdb.
qmail-smtpd includes the authentication information in the the Received: header and displays the provided Distinguished Name of the user identity. The DN should include the email address of the sender.
In general, there is no connection between the DN and the ESMTP 'Return-Path' address, but this can enforced. For this purpose, the environment variable LOCALMFCHECK="?" can be set; typically as provided in the cdb. Now, only ESMTPA connections with matching conditions are granted.
In case, the ESMTPA authentication was accepted or rejected, qmail-smtpd displays the DN as user identification in the logs following the token "?~". Further, the authentication method is given as cert.
s/qmail includes a rich set of ESMTP Authentication capabilities for qmail-smtpd together with qmail-authuser. For some background, have a look at my SMTP Authentication tutorial; however, this may not be up to date.
Currently, the ESMTP auth mechanisms
- plain, and
are supported; but depend on the IdP.
The environment variable SMTPAUTH can be used to specify the type of SMTP Authentication. Your choices are:
|""||Left blank to allow Authentication types "PLAIN" and "LOGIN"|
|"+cram"||Add "CRAM-MD5" support|
|"cram"||Just (secure) "CRAM-MD5" support, no other types offered|
|"!"||Enforcing SMTP Auth (of type "LOGIN" or "PLAIN")|
|"!cram"||Enforcing SMTP Auth of type "CRAM-MD5"|
|"!+cram"||Enforcing SMTP Auth of type "LOGIN", "PLAIN", or "CRAM-MD5"|
|"-"||Disabling SMTP Auth (for a particular connection)|
Unlike the Submission feature, these settings may be realized in the tcprules cdb per connection.
qmail-smtpd provides logging of accepted and failed authentication attempts.
The authentication mechanism together with the username is provided in the logs. Sample:
qmail-smtpd: pid 1167 Accept::AUTH::plain P:ESMTPSA S:220.127.116.11:example.com H:[127.0.0.1] F:firstname.lastname@example.org T:email@example.com ?~ 'userid'
Authentication for Mailbox users is a 'must' and without authentication access to a Mailbox is not permitted. Along side with qmail-pop3d, qmail-popup is required to provide the userid, password, and the environmental settings.
|POP3AUTH||Meaning||"apop"||Enforcing Auth of type "APOP"|
|"+apop"||Enabling Auth of type "APOP"|
Note: For qmail-popup no particular environment variables are required to enable authentication.
qmail-qmtpd supports authentication solely based on X.509 client certificates employing sslserver. The procedure follows the one discussed in section 5.5, except that the validation of the email address against the provided QMTP originator address is not provided and no specific logging takes place.
- As ESMTP client submitting emails from a specific domain. For this purpose, the control file authusers can be employed.
- As ESMTP deliverer for emails to a relayhost as specified by means of the legacy (but now extended) smptroutes control file.
In a setting, where s/qmail has to serve multiple domains, you can set up qmail-remote to respect to act according to the domain's preferences:
Scenario 1 (control/authusers):
Based on the entire SMTP sending address, the domain part; or perhaps in general, one can specify the ESMTP username, the password, and hostname and port to connect to for ESMTP authentication.
However, if s/qmail depends on some remore relayhosts irrespectively of the sender, the following setup will be beneficial:
Scenario 2 (control/smtproutes):
Based on destination, qmail-remote can be advised to send emails on behalf of a (commonly known) username and password to a specific relayhost on a given port.
Now, the control file domaincerts may be employed to provide domain based X.509 authentication. This requires to include a X.509 + keyfile for a specific domain.
qmail-remote domaincerts provides the following capabilities:
- The PEM encoded X.509 certificate is given per sending domain (syntax):
- If domain equals '* this certificate is used as default for all connections.
- The file certificate may include the private key, thus keyfile can be omitted.
- The private key can be protected with a password.
Note: It is certainly a good idea to change the permission of this file only to be readable by qmail-remote running as user qmailr.
Starting with release 3.2 s/qmail supports the Sender Policy Framework SPF. SPF is mainly a mechanism to attach authorization to particular MTAs; their IP address respectively. Based on Jana Saout's code, s/qmail incorporates a native SPF parser with IPv6 capabilities.
SPF DNS records are common these days for the 'big players' in the Internet, like Gmail and others. However, the may possess a certain level of complexity to parse including text-semantic analysis and delegation to other SPF domain providers.
In order get used to SPF, spfquery may be used to see, how the SPF information retrieval is working. In general, an email supplier (using ESMTP) publishes for this domain a specific DNS TXT record which provides authorization for (mainly) IP addresses of MTAs of the domain example.com. This domain name is taken from the 'Mail From:' SMTP envelope address (in case of bounce, the EHLO/HELO string). The idea is taken from Hadmuth Danisch's RMX proposal, which was sacked in the IETF letter ballot and replaced with the much more complex SPF model.
Within my SPF implementation the entire information retrieval path is provided as output from spfquery.
Here is a (complicated) sample for a FAIL result:
Here, the 'M' denotes the SPF mechanism, 'I' means include. All presented IP addresses are checked against the sending MTA's one. CIDR evaluation is supported.
Once SPF records are published into the DNS, one may specify, if a particular MTA
- is authorized (PASS '+'),
- should be rejected (FAIL '-'),
- is ambiguous (SOTFTAIL '~'), or
- is unknown (NEUTRAL '?')
according to the SPF policy for sending email from example.com as provided as SMTP Return-Path envelope address.
s/qmail provides a complex SPF record parser and supporting all 'mechanisms' as defined in RFC 4408.
SPF records requires intensive DNS lookups and this is done at the stage 'Rcpt to:' by qmail-smtpd, thus other anti-Spam means will work before.
Advise: Use a local DNSCache to speed up the lookup and reduce the load of any potential recursive DNS Resolver.
qmail-smtpd provides three tweaks for the SPF setup:
- The environment variable SPF is used as generic switch for the SPF evaluation. s/qmail follows Jana Saout's principals, as shown in the following table. However, no specific control file for qmail-smtpd is provided.
- A local policy (as last resort) can be specified in spflocalrules. The former spfguess is not supported anymore.
- More important, in case of an SPF Fail, the return code of
can be customized to allow a site specific explanation. The default expression build in is:
This explanation uses the typical SPF macros, which are expanded at run-time.
SPF values to be specified for individual connections:
|SPF||Meaning||"0"||Never do SPF lookups, don't create Received-SPF headers|
|"1"||Only create Received-SPF headers, never block|
|"2"||Use temporary errors when you have DNS lookup problems|
|"3"||Reject mails when SPF resolves to fail (deny)|
|"4"||Reject mails when SPF resolves to softfail|
|"4"||Reject mails when SPF resolves to neutral|
|"6"||Reject mails when SPF does not resolve to pass|
SPF values 1 to 3 are 'save' and can be used in a standard environment. Setting SPF="1" raises the possibility to allow a Mail Delivery Agent (MDA) to inspect the Received-SPF header for spam and fraud detection.
Note: In case auf email authentication, no SPF evaluation is used.
qmail-smtpd generates a standard Received-SPF header, in case
- the provided SPF environment value is given and not null,
- the domain to lookup provides SPF records to be evaluated.
Thus, the Received-SPF header is only present in case SPF information has been looked up successfully or a particular DNS failure hase been recognized. A standard PASS situation looks like this:
The implementation is able to include all cases of SPF results ('+' pass, '-' fail, '~' softfail, '?' neutral, 'o' none, 't' temperror, 'e' permerror); however depends on a feed of SPF variables.
The (global) stralloc variable spfinfo is populated as set of concatenated individual values in the following manner:
- identity carries the information of the retrieval source (i.e. recipient address, Helo greeting ...).
- clientip is the IP of the sending host.
- helo is the Helo greeting of the sender; if this is omitted, qmail-smtpd uses the received E/HELO string.
- envelopefrom is the Return-Path from the envelope.
- receiver is the recipient MTA.
- problem includes a description of encountered errors.
- mechanism is the purported evaluation method.
- result is the result of the evaluation and is expressed in the standard tokens (i.e. '+', '-' ...).
Note 1: Except for result any of these variables may be empty/unsupplied.
Note 2: The provided information follows the guidelines in RFC 6376.
qmail-smtpd displays the evaluated SPF information under two distinct conditions:
- PASS: In case, the SPF information matches the received criterions, irrespectively of the SPF settings (SPF > 0) the email is labeled als SPF accepted and additional receiving information is added.
- FAIL: In case SPF is setup to reject failed SPF evaluation, a Fail is indicated in the logs.