Newanalyse - Analysis of s/qmail (qmail) logs
newanalyse
Newanalyse is a tool to post-process and archive the log information produced by
qmail-send, qmail-smtpd, qmail-qmtpd, and qmail-pop3d
written by multilog to disk.
In addition, even in the archived (qmail-send) log files particular emails
can be easily found accordingly to the sender (originator) and/or the recipient by means of findmail.
Thus, Qmail together with multilog and newanalyse conforms with the current legislative changes
of the European Community requiring archival of the email connection data for a certain period.
For a discussion of the political implications and those concerning the personal rights regarding the
required storage of electronic communication data see:
"Vorratsspeicherung von Verbindungsdaten in der Telekommunikation" (in German).
Requirements
You need to have installed in the first place:
- Daemontools
- The Korn-shell and PERL (for qmFind)
For s/qmail you just need:
- s/qmail since it includes everything you need.
In case you use qmail + perhaps SPAMCONTROL check out:
- Qmail (of course)
- Qmailanalog (not required for version newanalyse 2.x)
Note:
- s/qmail has to 'supervised'
- newanalyse does not require the common (LWQ) 'qmail-ctl' script; though happily co-exists with it
- newanalyse is aligned with the current logging of s/qmail
- The previous version of newanalyse could be used for in conjunctions with SPAMCONTROL
Features
- Analysis of qmail-send logfiles by means of Qmailanalog
- Adaption for particular senders/recipients/conditions
- Counters for Bounces, (discarded) Double-Bounces, and Nullsender messages
- Counters for successful remote and local deliveries
- Counters for local and remote recipients
- Display of top senders and recipients according to number of messages and volume
- Display of any (customized) verbose messages in qmail-send log (for error tracking)
- Analysis of qmail-smtpd in particular for Spamcontrol entries
- Counters for tcpserver, sslserver, and rblsmtpd messages
- Display and analysis of successful Greetdelayed sessions
- Information about accepted and rejected Auth sessions
- Topic-based display of other accepted and rejected SMTP sessions like Spam attempts
- Analysis of qmail-pop3d log files
- Counters for tcpserver and sslserver messages
- Information about accepted and rejected POP3 sessions
- Analysis of qmail-qmtpd
log files - Counters for tcpserver and sslserver messages
- Information about accepted and rejected QMTP/QMTPS sessions
- Logfile processing
- Secure and long-haul archival of logfiles and/or the analysis results
- Customizable purge of old log files after N days
- Simultaneous handling of several qmail-smtpd, qmail-pop3d
and qmail-qmtpd instances
- Support for several multilog serviced log directories
- Setup of scratch directories not to be archieved
- Apart from a generic .newanlyse.profile the following particluar profiles can be used (and customized):
- [qmail-send]: newanalyse.senders
- [qmail-send]: newanalyse.recipients
- [qmail-send]: newanalyse.mtas
- [qmail-send]: newanalyse.failures
- [qmail-send]: newanalyse.deferrales
- [qmail-send]: newanalyse.verbose
- [qmail-smtpd]: newanalyse.smtpmessages
- [qmail-pop3d]: newanalyse.pop3messages
- [qmail-qmtpd]: newanalyse.qmtpmessages
- Reporting
- Reporting of the results by email to the sysadmin(s) (newanalyse.logadmin)
- Verbose output for instant error/failure analysis
- Logfile parsing
- Search in the (archived) log files w.r.t. Sender/Recipients by means of findmail/qmFind
Usage
- newanalyse handels any number of logfiles (at least up to 1023 per call)
- newanalyse is easy customizable and extendable and supports multiple Qmail instances
- newanalyse can be simply called by cron to perform the daily logfile processing; it is not suited for a multilog !processor directive
Typical newanalyse crontab entry:
59 23 * * * /usr/local/bin/newanalyse -DAZK 1>/dev/null 2>&1
Download and Installation
Download
- For s/qmail use: newanalyse 2.0.5 (MD5: 7095500be434f9f47331bd7578b5a082)
- For qmail+Spamcontrol use: newanalyse 1.8.1 (MD5: 3073b1dd48994fbe9d413dc1ab83cb16)
Installation
- Make sure, you meet the installation requirements
- Expand the packed tar-archive at an appropriate location; an installation directory will be created
- Logfiles: Edit conf-pop3dlog, conf-sqmail,
conf-qmailanalog, conf-sendlog, and conf-smtpdlog, conf-pop3dlog and conf-qmtpdlog to your needs - Services: Modify conf-sendsvc, conf-smtpdsvc, and conf-pop3dsvc as well as conf-qmtpdsvc to coincide with your situation. This is required for ALRM-processing of the logfiles only.
- Execute ./install; maybe warnings and errors are displayed - fix them
- The executables newanalyse, findmail, and qmFind are installed at /var/qmail/bin/ (symlinks provided in /usr/local/bin/) man-pages at /usr/loca/sqmail/man/, and the profiles at /var/qmail/etc/
- Adjust to newanalyse profiles to your need; verify settings by means of 'newanalyse -h'
Documentation
- newanalyse and findmail include comprehensive man-pages
- newanalyse and findmail use get-opts style arguments; the current settings are displayed via '-h'