12. Logging
s/qmail follows qmail attitude to use entirely Unix file discriptors for logging. Thus, it does not possess any network capabilities to feed a remote SYSLOG server, rather additional programs are required to do so.
The purpose of logging is to provide:
- Usage information - in particular for qmail-send,
- Abuse information - regarding SPAM and viruses for mainly qmail-smtpd,
- Capacity information - given the setup of the servers.
From here, you can get
- statistics on the general behavior of s/qmail regarding on medium- (qmail-mrtg) and long-time (newanalyse) informations
- and perhaps an alerting invoking a proprietory analysis.
However,
- never, any proprietory information like passwords are disclosed and logged.
s/qmail uses different logging schemes:
- qmail-send feeds the queue whether invoked by qmail-local or qmail-remote and uses the standard qmail format.
- The servers qmail-smtpd and qmail-pop3d use SPAMCONTROLS's logging message format.
- The servers qmail-qmtpd and qmail-qmqpd act 'logging-free', thus any logging is provided by means of tcpserver and sslserver only.
12.1 qmail-send
The qmail-send's logging uses the following tokens:
- msg 28857444 providing the node-id.
- delivery 1356 telling the delivery number since start of the qmail-send instance.
- qp is the pid of the delivery process.
- uid ist the userid triggering the delivery.
The qmail-send log also provides the delivery channel and the number of used/possible delivery instances:
- local
- remote
A service generic multilog run script is provided. Use the respective qmail-smtpd run script for feeding the log.
Note: Given the many qmail-remote and qmail-local instances running concurrently and the monotone written log by means of multilog, mail transactions don't usually appear contiguous, but rather are splited over many lines and perhaps several qmail-send log files.
12.2 qmail-smtpd
Normally, qmail-smtpd doesn't log anything. Within s/qmail, qmail-smtpd logs some accepted and some (important) rejected SMTP session attempts.
- Format: <server>: pid PID Action::Type::Condition: Information
In order to track a complete SMTP transaction (including tcpserver/sslserver + rblsmtpd) the log line includes the PID.
Caveats: Regarding ssslserver and its capabilities, not always the same PID is referenced in the logs for the same transaction.
Action | Type | Condition | Explanation |
Reject | AUTH | missing | AUTHentication missing |
Reject | AUTH | type | AUTHentication of 'type' rejected |
Reject | Auth | Meth | AUTHentication Method rejected |
Accept | AUTH | type | AUTHentication of 'type' accepted |
Reject | DATA | Invalid_Size | DATA exceeds sizelimit |
Reject | DATA | Bad_MIME | DATA includes BASE 64 MIME type listed in badmimetypes |
Reject | DATA | Bad_Loader | DATA includes BASE64 loader type listed in badmimetypes |
Reject | DATA | Virus_Infected | DATA includes virus infected message (<scanner> | 'AV scanner') |
Reject | DATA | Spam_Message | DATA includes an identified Spam message |
Reject | ORIG | Bad_Mailfrom | ORIG is in badmailfrom |
Reject | ORIG | DNS_MF | Domain part of ORIG has no DNS MX RR |
Reject | ORIG | Failed_Auth | ORIG tried SMTP Authentication; but failed |
Reject | ORIG | Require_Auth | SMTP Authentication required; but not granted |
Reject | ORIG | Invalid_Sender | ORIG not allowed to send |
Reject | ORIG | Missing_Auth | SMTP Authentication required, but not granted |
Reject | ORIG | SPF | ORIG was rejected due to failed SPF permissions |
Accept | ORIG | Local_Sender | ORIG was identified as local sender address |
Accept | ORIG | Relay_Mailfrom | ORIG was accepted als Relaymailfrom |
Reject | RCPT | Bad_Rcptto | RCPT is in badrcptto |
Reject | RCPT | Toomany_Rcptto | Too many RCPTs |
Reject | RCPT | Failed_Rcptto | RCPT could not acceptd as per recipients/cdb |
Accept | RCPT | Recipients_Cdb | RCPT was accepted as per recipients/cdb |
Accept | RCPT | Recipients_Pam | RCPT was accepted as per recipients/pam plug-in |
Accept | RCPT | Recipients_Wild | RCPT was accepted as per recipients/wildlisting |
Accept | RCPT | Rcpthosts_Rcptto | CPT was accepted as per rcpthosts/morercpthosts |
Reject | SNDR | Bad_Helo | SNDR's HELO is in the badhelo |
Reject | SNDR | DNS_HELO | SNDR's HELO has no DNS A RR |
Reject | SNDR | Invalid_Relay | SNDR's tries relaying; but not allowd |
Accept | SNDR | Relay_Client | SNDR was identified as relay client |
Reject | TLS | missing | TLS connection could not be established |
Accept | SPF | Recipients_Cdb | ORIG was authorized and RCPT accepted as per recipients/cdb |
Accept | SPF | Recipients_Pam | ORIG was authorized and RCPT accepted as per recipients/pam plug-in |
Accept | SPF | Recipients_Wild | ORIG was authorized and RCPT was accepted as per recipients/wildlisting |
Accept | SPF | Rcpthosts_Rcptto | ORIG was authorized and RCPT was accepted as per rcpthosts/morercpthosts |
Reject | SPF | Fail | ORIG authorization failed per SPF |
Deferred | SNDR | Grey_Listed | Greylisted sender based on IP/Hostname/Mail From/Rcpt To |
- SNDR (S) corresponds to the sending MTA
- ORIG (F) is the "MAIL From: <Return-Path>"
- RCPT (T) is the "RCPT To: <Forwarding-Path>"
- DATA is the Message (content)
- SPF is Sender Policy Framework
- TLS denotes Transport Layer Security
The Information is typically constructed from the SMTP envelope like:
- S:IP:FQDN P:Protocol H:Helo F:Mailfrom T:Rcptto
This scheme is easy extendable to other successful/deferred SMTP sessions.
Sample for a successful qmail-smtpd session:
Sample for a rejected qmail-smtpd session:
Note 1: Again here, the entire mail transactions may not be found in one contiguous block but rather dispersed; however ordered in monotonically in time.
Note 2: Though the PID for a single transactions is preserved, given sslserver the last statement always shows the parent PID.
Note 3: Here, and in the other logging samples, sslserver is invoked for logging with the argument sslserver -V displaying the mod_ssl variables while showing the TLS Cipher Suite.
12.3 qmail-qmtpd
Currently, qmail-qmtpd does not do any logging. Use tcpserver or sslserver and their logging capabilities.
12.4 qmail-qmqpd
Currently, qmail-qmqpd does not do any logging. Use tcpserver and its logging capabilities.
12.5 qmail-pop3d
In addition for POP3 services the qmail-smtpd scheme is used; but now logging takes place on FD 5.
qmail-pop3d does not do any logging. Use this run script to generate a logging of the Auth information provided by qmail-popup.
Sample for an accepted qmail-pop3d session:
12.6 qmail-mrtg
qmail-mrtg is used to evaluate the log activity information provided by
- qmail-send - depending on qmail-mrtg.send.cfg
- qmail-smtpd - depending on qmail-mrtg.smtpd.cfg and
- qmail-pop3d - depending on qmail-mrtg.pop3d.cfg
showing usage and ab-usage of those services while feeding the information to MRTG. Sample for the *.cfg files can be found in the etc installation directory of s/qmail.
Once MRTG is installed, the typical invocation includes the call of MRTG with the respective cfg by means of cron:
Here, the crontab is used to update the taken samples by means of qmail-mrtg following the configurations in the cfg files every four minutes. The default sample time is five minutes. Overlapping entries are detected. The evaluation time can be specified by qmail-mrtg.
Additionally to MRTG, the RDD tool can be used to 'beautify' the results. Customization is entirely free and the additionally a local Network Management tool can be used to further actions.
Note: In order to provide an efficient and most effective evaluation of the log files, their size and and revolution needs to be carefully adjusted with the qmail-mrtg timings. Though qmail-mrtg tries best to reduce system load regarding reading large log-files. The smaller the log-files, the more efficient the analysis becomes.
The MRTG statistics of my server is shown here: fehcom.de
12.7 Analysis qmail-send log files
s/qmail includes the qmailanalog package in in well-suited format. qmailanalog provides statistical and performance informations regarding
- qmail-send in particular for
- qmail-local and
- qmail-remote and
using the logging format of qmail-send.
You get (according to D.J. Bernstein):
- overall: how many messages? recipients? attempts? etc.
- ddist: how soon were 50% of the messages delivered? 90%? 95%? 99%?
- rxdelay: what's the best order of recipients for mailing lists?
- recipients, rhosts: who's getting mail? bytes? messages? attempts?
- successes, failures, deferrals: why? how often? how much delay?
- senders, suids: messages? bytes? load? recipients? attempts? delay?
mainly evaluated by awk and some small programs.
12.8 Newanalyse
Use Newanalyse together with multilog to ensure a persistant logging of s/qmail's activity, analysis and long-haule capture.
The purpose of Newanalyse is twofold:
Calling Newanalyse on daily base you get:
- Statistics on qmail-send, qmail-local, qmail-remote, qmail-smtpd, and qmail-pop3d performance,
- safe backup of these processes, to perhaps answer the question "Was my email to X@Y was received in October 2008?"
However, Newanalyse allows to delete logs after a certain time span in order to comply with privacy regulations, given the logs files are not backuped.