12. Logging
s/qmail follows qmail attitude to use entirely Unix file discriptors for logging. Thus, it does not possess any network capabilities to feed a remote SYSLOG server, rather additional programs are required to do so.
The purpose of logging is to provide:
- Usage information - in particular for qmail-send,
- Abuse information - regarding SPAM and viruses for mainly qmail-smtpd,
- Capacity information - given the setup of the servers.
From here, you can get
- statistics on the general behavior of s/qmail regarding on medium- (qmail-mrtg) and long-time (newanalyse) informations
- and perhaps an alerting invoking a proprietory analysis.
However,
- never, any proprietory information like passwords are disclosed and logged.
s/qmail uses different logging schemes:
- qmail-send feeds the queue whether invoked by qmail-local or qmail-remote and uses the standard qmail format.
- The servers qmail-smtpd and qmail-pop3d use SPAMCONTROLS's logging message format.
- The servers qmail-qmtpd and qmail-qmqpd act 'logging-free', thus any logging is provided by means of tcpserver and sslserver only.
Quick links:
- 12.1 s/qmail logging strategies
- 12.2 qmail-send
- 12.3 qmail-smtpd
- 12.4 qmail-qmtpd
- 12.5 qmail-qmqpd
- 12.6 qmail-pop3d
- 12.7 qmail-mrtg
- 12.8 Analysis qmail-send log files
- 12.9 Newanalyse
12.1 s/qmail logging strategies
If you run a MTA, it is your responsibility to care for its performance.
You also need to care about successful and unsuccessful delivieries. Both on an statistical manner but probably more importent using some kind of on-line monitoring and alarming.
s/qmail's log help you to do this task, but given a busy system you need to setup your own procedures. DJB's multilog is still the best choice. systemds journald is ok for small traffic but IMHO not targeted for application logging.
12.2 qmail-send
The qmail-send's logging uses the following tokens:
- msg 28857444 providing the node-id.
- delivery 1356 telling the delivery number since start of the qmail-send instance.
- qp is the pid of the delivery process.
- uid ist the userid triggering the delivery.
The qmail-send log also provides the delivery channel and the number of used/possible delivery instances:
- local
- remote
A service generic multilog run script is provided. Use the respective qmail-smtpd run script for feeding the log.
Note: Given the many qmail-remote and qmail-local instances running concurrently and the monotone written log by means of multilog, mail transactions don't usually appear contiguous, but rather are splited over many lines and perhaps several qmail-send log files.
12.3 qmail-smtpd
Normally, qmail-smtpd doesn't log anything. Within s/qmail, qmail-smtpd logs some accepted and some (important) rejected SMTP session attempts.
- Format: <server>: pid PID Action::Type::Condition: Information
In order to track a complete SMTP transaction (including tcpserver/sslserver + rblsmtpd) the log line includes the PID.
Caveats: Regarding ssslserver and its capabilities, not always the same PID is referenced in the logs for the same transaction.
Action | Type | Condition | Explanation |
Reject | AUTH | missing | AUTHentication missing |
Reject | AUTH | type | AUTHentication of 'type' rejected |
Reject | Auth | Meth | AUTHentication Method rejected |
Accept | AUTH | type | AUTHentication of 'type' accepted |
Reject | DATA | Invalid_Size | DATA exceeds sizelimit |
Reject | DATA | Bad_MIME | DATA includes BASE 64 MIME type listed in badmimetypes |
Reject | DATA | Bad_Loader | DATA includes BASE64 loader type listed in badmimetypes |
Reject | DATA | Virus_Infected | DATA includes virus infected message (<scanner> | 'AV scanner') |
Reject | DATA | Spam_Message | DATA includes an identified Spam message |
Reject | ORIG | Bad_Mailfrom | ORIG is in badmailfrom |
Reject | ORIG | DNS_MF | Domain part of ORIG has no DNS MX RR |
Reject | ORIG | Failed_Auth | ORIG tried SMTP Authentication; but failed |
Reject | ORIG | Require_Auth | SMTP Authentication required; but not granted |
Reject | ORIG | Invalid_Sender | ORIG not allowed to send |
Reject | ORIG | Missing_Auth | SMTP Authentication required, but not granted |
Reject | ORIG | SPF | ORIG was rejected due to failed SPF permissions |
Accept | ORIG | Local_Sender | ORIG was identified as local sender address |
Accept | ORIG | Relay_Mailfrom | ORIG was accepted als Relaymailfrom |
Reject | RCPT | Bad_Rcptto | RCPT is in badrcptto |
Reject | RCPT | Toomany_Rcptto | Too many RCPTs |
Reject | RCPT | Failed_Rcptto | RCPT could not acceptd as per recipients/cdb |
Accept | RCPT | Recipients_Cdb | RCPT was accepted as per recipients/cdb |
Accept | RCPT | Recipients_Pam | RCPT was accepted as per recipients/pam plug-in |
Accept | RCPT | Recipients_Wild | RCPT was accepted as per recipients/wildlisting |
Accept | RCPT | Rcpthosts_Rcptto | CPT was accepted as per rcpthosts/morercpthosts |
Reject | SNDR | Bad_Helo | SNDR's HELO is in the badhelo |
Reject | SNDR | DNS_HELO | SNDR's HELO has no DNS A RR |
Reject | SNDR | Invalid_Relay | SNDR's tries relaying; but not allowd |
Accept | SNDR | Relay_Client | SNDR was identified as relay client |
Reject | TLS | missing | TLS connection could not be established |
Accept | SPF | Recipients_Cdb | ORIG was authorized and RCPT accepted as per recipients/cdb |
Accept | SPF | Recipients_Pam | ORIG was authorized and RCPT accepted as per recipients/pam plug-in |
Accept | SPF | Recipients_Wild | ORIG was authorized and RCPT was accepted as per recipients/wildlisting |
Accept | SPF | Rcpthosts_Rcptto | ORIG was authorized and RCPT was accepted as per rcpthosts/morercpthosts |
Reject | SPF | Fail | ORIG authorization failed per SPF |
Deferred | SNDR | Grey_Listed | Greylisted sender based on IP/Hostname/Mail From/Rcpt To |
- SNDR (S) corresponds to the sending MTA
- ORIG (F) is the "MAIL From: <Return-Path>"
- RCPT (T) is the "RCPT To: <Forwarding-Path>"
- DATA is the Message (content)
- SPF is Sender Policy Framework
- TLS denotes Transport Layer Security
The Information is typically constructed from the SMTP envelope like:
- S:IP:FQDN P:Protocol H:Helo F:Mailfrom T:Rcptto
This scheme is easy extendable to other successful/deferred SMTP sessions.
Sample for a successful qmail-smtpd session:
Sample for a rejected qmail-smtpd session:
Note 1: Again here, the entire mail transactions may not be found in one contiguous block but rather dispersed; however ordered in monotonically in time.
Note 2: Though the PID for a single transactions is preserved, given sslserver the last statement always shows the parent PID.
Note 3: Here, and in the other logging samples, sslserver is invoked for logging with the argument sslserver -V displaying the mod_ssl variables while showing the TLS Cipher Suite.
12.4 qmail-qmtpd
Starting with s/qmail 4.3 qmail-qmtpd does a minimal logging while acknowleding the received email. Together with sslserver the following log entries may be given in case of a successful transaction:
qmail-qmtpd will do a logging in the following failure cases:
- qmail-qmtpd: client does not talk QMTP
- qmail-qmtpd: Unacceptable sender (#5.1.7)
- qmail-qmtpd: Sorry, that message size exceeds my databytes limit (#5.3.4)
- qmail-qmtpd: Sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)
With s/qmail 4.3 some minimal logging is added:
- qmtpd: pid XXX done.
- qmtpd: pid XXX relayclient done.
12.5 qmail-qmqpd
Currently, qmail-qmqpd does not do any logging. Use tcpserver and its logging capabilities.
12.6 qmail-pop3d
In addition for POP3 services the qmail-smtpd scheme is used; but now logging takes place on FD 5.
qmail-pop3d does not do any logging. Use this run script to generate a logging of the Auth information provided by qmail-popup.
Sample for an accepted qmail-pop3d session:
12.7 qmail-mrtg
qmail-mrtg is used to evaluate the log activity information provided by
- qmail-send - depending on qmail-mrtg.send.cfg
- qmail-smtpd - depending on qmail-mrtg.smtpd.cfg and
- qmail-pop3d - depending on qmail-mrtg.pop3d.cfg
showing usage and ab-usage of those services while feeding the information to MRTG. Sample for the *.cfg files can be found in the etc installation directory of s/qmail.
Once MRTG is installed, the typical invocation includes the call of MRTG with the respective cfg by means of cron:
Here, the crontab is used to update the taken samples by means of qmail-mrtg following the configurations in the cfg files every four minutes. The default sample time is five minutes. Overlapping entries are detected. The evaluation time can be specified by qmail-mrtg.
Additionally to MRTG, the RDD tool can be used to 'beautify' the results. Customization is entirely free and the additionally a local Network Management tool can be used to further actions.
Note: In order to provide an efficient and most effective evaluation of the log files, their size and and revolution needs to be carefully adjusted with the qmail-mrtg timings. Though qmail-mrtg tries best to reduce system load regarding reading large log-files. The smaller the log-files, the more efficient the analysis becomes.
The MRTG statistics of my server is shown here: fehcom.de
12.8 Analysis qmail-send log files
s/qmail includes the qmailanalog package in in well-suited format. qmailanalog provides statistical and performance informations regarding
- qmail-send in particular for
- qmail-local and
- qmail-remote and
using the logging format of qmail-send.
You get (according to D.J. Bernstein):
- overall: how many messages? recipients? attempts? etc.
- ddist: how soon were 50% of the messages delivered? 90%? 95%? 99%?
- rxdelay: what's the best order of recipients for mailing lists?
- recipients, rhosts: who's getting mail? bytes? messages? attempts?
- successes, failures, deferrals: why? how often? how much delay?
- senders, suids: messages? bytes? load? recipients? attempts? delay?
mainly evaluated by awk and some small programs.
12.9 Newanalyse
Use Newanalyse together with multilog to ensure a persistant logging of s/qmail's activity, analysis and long-haule capture.
The purpose of Newanalyse is twofold:
Calling Newanalyse on daily base you get:
- Statistics on qmail-send, qmail-local, qmail-remote, qmail-smtpd, and qmail-pop3d performance,
- safe backup of these processes, to perhaps answer the question "Was my email to X@Y was received in October 2008?"
However, Newanalyse allows to delete logs after a certain time span in order to comply with privacy regulations, given the logs files are not backuped.