[yaala] Sendmail log parser

[qMax]

------------A3E21F3017530A
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable

From=20module POD:

This is a Parser for sendmail log.

Note:
 Each message in sendmail log forms several records (lines):
 one record with 'from=3D' field, and one or several with 'to=3D' field.

 Parser joins each 'from'-part with 'to'-part by message log id.
 However, if there're several recipients, result will be several
 records for the same message: one per recipient. When message is
 first time countd, datafield 'uniq' is set to 1. This is usefull to
 calculate total count/traffic or traffic by type. But if you count
 total by recipients, using this key (as WHERE uniq=3D=3D'1') will make
 yaala ignore all recipients of a message, but the first.

Config options:
 sendmail_aliases     - aliases file used to resolve (local senders) adress=
es
 sendmail_localdomain - local domain to remove from adresses
 sendmail_localrelay  - IP regexp to determine incoming/outgoing/local traf=
fic, egg '192.168.1.\d+'

Data fields:

 everything found in sendmail log, with:
 timedate is splited to 'date' and 'hour', as usual, year is taken from cur=
rent date.
 rrelay is relay field from 'to'-part

 uniq =3D set to 1 when message first time counted.

 type =3D "I","O","L","R" for incoming, outgoing, local and relay traffic.
        It is determined using fields 'mailer' and 'relay'.
        (Thus, only applied to actually sent/recieved messages)

Aggregations: size, count, nrcpts

TODO:
 - Properly handle multiple aliases.
 - Split non-local multiple recipients
 - resolve hosts in relay fields.

--=20
 qMax
------------A3E21F3017530A
Content-Type: application/octet-stream; name="Sendmail.pm"
Content-transfer-encoding: base64
Content-Disposition: attachment; filename="Sendmail.pm"

cGFja2FnZSBZYWFsYTo6UGFyc2VyOwoKPWhlYWQxIFlhYWxhOjpQYXJzZXI6OlNlbmRtYWls
CgpQYXJzZXIgZm9yIHNlbmRtYWlsIGxvZy4KCk5vdGU6CgpFYWNoIG1lc3NhZ2UgaW4gc2Vu
ZG1haWwgbG9nIGZvcm1zIHNldmVyYWwgcmVjb3JkcyAobGluZXMpOgpvbmUgcmVjb3JkIHdp
dGggJ2Zyb209JyBmaWVsZCwgYW5kIG9uZSBvciBzZXZlcmFsIHdpdGggJ3RvPScgZmllbGQu
CgpQYXJzZXIgam9pbnMgZWFjaCAnZnJvbSctcGFydCB3aXRoICd0byctcGFydCBieSBtZXNz
YWdlIGxvZyBpZC4KSG93ZXZlciwgaWYgdGhlcmUncmUgc2V2ZXJhbCByZWNpcGllbnRzLCBy
ZXN1bHQgd2lsbCBiZSBzZXZlcmFsIHJlY29yZHMgZm9yIHRoZSBzYW1lIG1lc3NhZ2U6Cm9u
ZSBwZXIgcmVjaXBpZW50LgpXaGVuIG1lc3NhZ2UgaXMgZmlyc3QgdGltZSBjb3VudGQsIGRh
dGFmaWVsZCAndW5pcScgaXMgc2V0IHRvIDEuClRoaXMgaXMgdXNlZnVsbCB0byBjYWxjdWxh
dGUgdG90YWwgY291bnQvdHJhZmZpYyBvciB0cmFmZmljIGJ5IHR5cGUuCkJ1dCBpZiB5b3Ug
Y291bnQgdG90YWwgYnkgcmVjaXBpZW50cywgdXNpbmcgdGhpcyBrZXkgKGFzIFdIRVJFIHVu
aXE9PScxJykKd2lsbCBtYWtlIHlhYWxhIGlnbm9yZSBhbGwgcmVjaXBpZW50cyBvZiBhIG1l
c3NhZ2UsIGJ1dCB0aGUgZmlyc3QuCgo9Y3V0Cgo9aGVhZDIgQ29uZmlnIG9wdGlvbnMKCiBz
ZW5kbWFpbF9hbGlhc2VzICAgICAtIGFsaWFzZXMgZmlsZSB1c2VkIHRvIHJlc29sdmUgYWRy
ZXNzZXMKIHNlbmRtYWlsX2xvY2FsZG9tYWluIC0gbG9jYWwgZG9tYWluIHRvIHJlbW92ZSBm
cm9tIGFkcmVzc2VzCiBzZW5kbWFpbF9sb2NhbHJlbGF5ICAtIElQIHJlZ2V4cCB0byBkZXRl
cm1pbmUgaW5jb21pbmcvb3V0Z29pbmcvbG9jYWwgdHJhZmZpYywgZWdnICcxOTIuMTY4LjEu
XGQrJwoKPWN1dAoKPWhlYWQyIERhdGEgZmllbGRzCgogZXZlcnl0aGluZyBmb3VuZCBpbiBz
ZW5kbWFpbCBsb2csIHdpdGg6CiB0aW1lZGF0ZSBpcyBzcGxpdGVkIHRvICdkYXRlJyBhbmQg
J2hvdXInLCBhcyB1c3VhbCwgeWVhciBpcyB0YWtlbiBmcm9tIGN1cnJlbnQgZGF0ZS4KIHJy
ZWxheSBpcyByZWxheSBmaWVsZCBmcm9tICd0byctcGFydAoKIHVuaXEgPSBzZXQgdG8gMSB3
aGVuIG1lc3NhZ2UgZmlyc3QgdGltZSBjb3VudGVkLgoKIHR5cGUgPSAiSSIsIk8iLCJMIiwi
UiIgZm9yIGluY29taW5nLCBvdXRnb2luZywgbG9jYWwgYW5kIHJlbGF5IHRyYWZmaWMuCiBJ
dCBpcyBkZXRlcm1pbmVkIHVzaW5nIGZpZWxkcyAnbWFpbGVyJyBhbmQgJ3JlbGF5Jy4KIChU
aHVzLCBvbmx5IGFwcGxpZWQgdG8gc2VudC9yZWNpZXZlZCBtZXNzYWdlcykKCkFnZ3JlZ2F0
aW9uczogc2l6ZSwgY291bnQsIG5yY3B0cwoKPWhlYWQyIFRPRE8KCiAtIFByb3Blcmx5IHJl
c29sdmUgbXVsdGlwbGUgYWxpYXNlcy4KIC0gU3BsaXQgbm9uLWxvY2FsIG11bHRpcGxlIHJl
Y2lwaWVudHMKCj1jdXQKCnVzZSBzdHJpY3Q7CnVzZSB3YXJuaW5nczsKdXNlIHZhcnMgcXco
JURBVEFGSUVMRFMpOwoKdXNlIEV4cG9ydGVyOwp1c2UgWWFhbGE6OlBhcnNlcjo6V2Vic2Vy
dmVyVG9vbHMgcXcoJU1PTlRIX05VTUJFUlMpOwp1c2UgWWFhbGE6OkRhdGE6OlBlcnNpc3Rl
bnQgcXcjaW5pdCM7CnVzZSBZYWFsYTo6Q29uZmlnIHF3I2dldF9jb25maWcjOwoKQFlhYWxh
OjpQYXJzZXI6OkVYUE9SVF9PSyA9IHF3KHBhcnNlIGV4dHJhICVEQVRBRklFTERTKTsKQFlh
YWxhOjpQYXJzZXI6OklTQSA9ICgnRXhwb3J0ZXInKTsKCm91ciAkRVhUUkEgPSBpbml0ICgn
JEVYVFJBJywgJ2hhc2gnKTsKCm15ICVDT1VOVEVEID0gKCk7CgppZiAoIWRlZmluZWQgKCRF
WFRSQS0+eyd0b3RhbGNvdW50J30pKQl7ICRFWFRSQS0+eyd0b3RhbGNvdW50J30gPSB7ST0+
MCwgTz0+MCwgTD0+MH07IH0KaWYgKCFkZWZpbmVkICgkRVhUUkEtPnsndG90YWxhbW91bnQn
fSkpCXsgJEVYVFJBLT57J3RvdGFsYW1vdW50J30gPSB7ST0+MCwgTz0+MCwgTD0+MH07IH0K
aWYgKCFkZWZpbmVkICgkRVhUUkEtPnsnc3RhcnQnfSAgKSkJeyAkRVhUUkEtPnsnc3RhcnQn
fSA9IHVuZGVmOyB9CmlmICghZGVmaW5lZCAoJEVYVFJBLT57J2VuZCd9ICkpCXsgJEVYVFJB
LT57J2VuZCd9ID0gdW5kZWY7IH0KCiVEQVRBRklFTERTID0gKAoJIyBsb2cgbWVzc2FnZSBp
ZAoJaWQJPT4gJ2tleScsCgkjICdmcm9tJyBwYXJ0Cglmcm9tCT0+ICdrZXknLAoJc2l6ZQk9
PiAnYWdnOmJ5dGVzJywKCWNsYXNzCT0+ICdrZXknLAoJbnJjcHRzCT0+ICdhZ2cnLAoJbXNn
aWQJPT4gJ2tleScsCglib2R5dHlwZT0+ICdrZXknLAoJcHJvdG8gICA9PiAna2V5JywKCWRh
ZW1vbiAgPT4gJ2tleScsCglyZWxheSAgID0+ICdrZXknLAoJIyAndG8nIHBhcnQKCXRvCT0+
ICdrZXknLAoJZGVsYXkgICA9PiAna2V5JywKCXhkZWxheSAgPT4gJ2tleScsCgltYWlsZXIg
ID0+ICdrZXknLAoJcHJpCT0+ICdrZXknLAoJZHNuCT0+ICdrZXknLAoJc3RhdAk9PiAna2V5
JywKCXJyZWxheSAgPT4gJ2tleScsCgkjIGFkZGl0aW9uYWwKCWRhdGUJPT4gJ2tleScsCglo
b3VyCT0+ICdrZXknLAoJdW5pcQk9PiAna2V5JywKCXR5cGUJPT4gJ2tleScsCgljb3VudAk9
PiAnYWdnJwopOwoKIyBUaGlzIG5lZWRzIHRvIGJlIGRvbmUgYXQgcnVudGltZSwgc2luY2Ug
RGF0YSB1c2VzIFNldHVwIHdoaWNoIHJlbGllcyBvbgojICVEQVRBRklFTERTIHRvIGJlIGRl
ZmluZWQgIC1vY3RvCnJlcXVpcmUgWWFhbGE6OkRhdGE6OkNvcmU7CmltcG9ydCBZYWFsYTo6
RGF0YTo6Q29yZSBxdyNzdG9yZSM7CgpteSAkVkVSU0lPTiA9ICckSWQ6IFNlbmRtYWlsLnBt
LHYgMS4wYiQnOwpwcmludCBTVERFUlIgJC8sIF9fRklMRV9fLCAiOiAkVkVSU0lPTiIgaWYg
KCQ6OkRFQlVHKTsKCm91ciAlQUxJQVNFUzsKbXkgJGFsaWFzZmlsZSA9IGdldF9jb25maWco
InNlbmRtYWlsX2FsaWFzZXMiKTsKaWYoICRhbGlhc2ZpbGUgKSB7CiAgICBwcmludCBTVERF
UlIgJC8sIF9fRklMRV9fLCAiOiBMb2FkZWQgYWxpYXNlcyBmcm9tICRhbGlhc2ZpbGUiIGlm
ICgkOjpERUJVRyk7CiAgICBsb2FkX2FsaWFzZXMoJGFsaWFzZmlsZSk7Cn0KCm91ciAkbG9j
YWxkb21haW4gPSBnZXRfY29uZmlnKCJzZW5kbWFpbF9sb2NhbGRvbWFpbiIpOwpvdXIgJGxv
Y2FscmVsYXkgPSBnZXRfY29uZmlnKCJzZW5kbWFpbF9sb2NhbHJlbGF5Iik7CnByaW50IFNU
REVSUiAkLywgX19GSUxFX18sICI6IExvY2FsIHJlbGF5OiAkbG9jYWxyZWxheSIgaWYgKCQ6
OkRFQlVHKTsKJGxvY2FscmVsYXkgPSBxci9cWyRsb2NhbHJlbGF5XF0vOwoKcmV0dXJuICgx
KTsKCm91ciAlUkVDRiA9ICgpOyAjIGFsbCBwZW5kaW5nIGZyb20tcGFydHMKb3VyICVSRUNU
ID0gKCk7ICMgYWxsIHBlbmRpbmcgdG8tcGFydHMKCnN1YiBwYXJzZQp7CglteSAkbGluZSA9
IHNoaWZ0IG9yIHJldHVybiB1bmRlZjsKCWlmICggJGxpbmUgPX4gcy9eKC4uLlxzKlxkKyBc
ZFxkOlxkXGQ6XGRcZCkuKltcdy1dKlxbXGQrXF06ICguezE0fSk6IC8vICkgewoJCW15ICRk
YXRldGltZSA9ICQxOwoJCW15ICRpZCAgICA9ICQyOwoJCWlmKCAkbGluZSA9fiAvXmZyb209
LyApIHsKCQkJbXkgJHJlYyA9IHBhcnNlbGluZSgkbGluZSk7CgkJCSRyZWMtPnsnZnJvbSd9
ID0gcmVzb2x2ZV9hbGlhcygkcmVjLT57J2Zyb20nfSk7CgkJCSRSRUNGeyRpZH0gPSB7IGRh
dGV0aW1lPT4kZGF0ZXRpbWUsICUkcmVjIH07CgkJCWNoZWNrcGFpcigkaWQpOwoJCQl9CgkJ
aWYoICRsaW5lID1+IC9edG89LyApIHsKCQkJbXkgJHJlYyA9IHBhcnNlbGluZSgkbGluZSk7
CgkJCSRyZWMtPnsndG8nfSA9IHJlc29sdmVfYWxpYXMoJHJlYy0+eyd0byd9KTsKCQkJJFJF
Q1R7JGlkfSA9IHsgZGF0ZXRpbWU9PiRkYXRldGltZSwgJSRyZWMgfTsKCQkJY2hlY2twYWly
KCRpZCk7CgkJCX0KCQl9Cn0KCnN1YiBwYXJzZWxpbmUKewoJbXkgJGxpbmUgPSBzaGlmdDsK
CW15ICVyZWM9KCk7Cglmb3JlYWNoIChzcGxpdCgvLFxzKy8sJGxpbmUpKSB7CgkJaWYoIG0v
KC4qPyk9KC4qKS8gKSB7CgkJCSRyZWN7JDF9PSQyIGlmIGV4aXN0cyAkREFUQUZJRUxEU3sk
MX07CgkJCX0KCQl9CglyZXR1cm4gXCVyZWM7Cn0KCnN1YiBjaGVja3BhaXIKewoJbXkgJGlk
ID0gc2hpZnQ7CglyZXR1cm4gdW5sZXNzICggJFJFQ0Z7JGlkfSBhbmQgJFJFQ1R7JGlkfSAp
OwoJJFJFQ1R7JGlkfS0+eydycmVsYXknfSA9ICRSRUNUeyRpZH0tPnsncmVsYXknfTsKCWRl
bGV0ZSAkUkVDVHskaWR9LT57J3JlbGF5J307CglteSAlcmVjID0gKCAleyRSRUNGeyRpZH19
LCAleyRSRUNUeyRpZH19ICk7CglkZWxldGUgJFJFQ1R7JGlkfTsKCgkjcHJpbnQgU1RERVJS
ICJcblJFQzogIixqb2luKCI7ICIsIG1hcCgiJF89Ii4kcmVjeyRffSwga2V5cyAlcmVjKSk7
CgkKCSRyZWN7J2RhdGV0aW1lJ30gPX4gLyhcd1x3XHcpXHMqKFxkKykgKFxkXGQpOlxkXGQ6
XGRcZC87CglteSAoJG1vbnRoLCRkYXksJGhvdXIpID0gKCQxLCQyLCQzKTsKCSRtb250aCA9
ICRNT05USF9OVU1CRVJTeyRtb250aH07CglteSAkeWVhciA9IFtsb2NhbHRpbWUodGltZSld
LT5bNV0rMTkwMDsgIyBjdXJyZW50IHllYXIKCW15ICRkYXRlID0gc3ByaW50ZigiJTA0dS0l
MDJ1LSUwMnUiLCAkeWVhciwgJG1vbnRoLCAkZGF5KTsKCglteSAlY29tYmluZWQgPSAoCgkJ
ZGF0ZQk9PiAkZGF0ZSwKCQlob3VyCT0+ICRob3VyLAoJCWNvdW50CT0+IDEsCgkJdW5pcQk9
PiBleGlzdHMoJENPVU5URUR7JGlkfSkgPyAxIDogMCwKCQklcmVjCgkJKTsKCgkkY29tYmlu
ZWR7J3RvJ30gPX4gcy9cPC9cJmx0Oy9nOwoJJGNvbWJpbmVkeyd0byd9ID1+IHMvXD4vXCZn
dDsvZzsKCSRjb21iaW5lZHsnZnJvbSd9ID1+IHMvXDwvXCZsdDsvZzsKCSRjb21iaW5lZHsn
ZnJvbSd9ID1+IHMvXD4vXCZndDsvZzsKCglteSAkdHlwZT0iKiI7CglpZiggJGxvY2FscmVs
YXkgYW5kICRjb21iaW5lZHsncmVsYXknfSBhbmQgJGNvbWJpbmVkeydtYWlsZXInfSkgewoJ
ICAgICMKCSAgICAjIEw6IGwvbCBPOiBsLy0KCSAgICAjIEk6IC0vbCBSOiAtLy0KCSAgICAj
CgkgICAgJHR5cGUgPQoJCSggJGNvbWJpbmVkeydyZWxheSd9ID1+ICRsb2NhbHJlbGF5ICkg
PyAKCQkgICAgKCAoICRjb21iaW5lZHsnbWFpbGVyJ30gZXEgJ2xvY2FsJyApID8gJ0wnIDog
J08nICkgOgoJCSAgICAoICggJGNvbWJpbmVkeydtYWlsZXInfSBlcSAnbG9jYWwnICkgPyAn
SScgOiAnUicgKSA7CgoJICAgIH0KCQoJI3ByaW50IFNUREVSUiAiXG5Gcm9tOiAiLiRjb21i
aW5lZHsnZnJvbSd9LiJcblRvOiAiLiRjb21iaW5lZHsndG8nfS4gIlxuUmVsYXkvbWFpbGVy
OiAiLiRjb21iaW5lZHsncmVsYXknfS4iLyIuJGNvbWJpbmVkeydtYWlsZXInfS4iIFR5cGU6
ICR0eXBlIjsKCgkkY29tYmluZWR7J3R5cGUnfSA9ICR0eXBlOwoKCXN0b3JlIChcJWNvbWJp
bmVkKTsKCgl1bmxlc3MoICRDT1VOVEVEeyRpZH0gKSB7CgkJJENPVU5URUR7JGlkfSA9IDE7
CgkJJEVYVFJBLT57J3RvdGFsY291bnQnfS0+eyR0eXBlfSsrOwoJCSRFWFRSQS0+eyd0b3Rh
bGFtb3VudCd9LT57JHR5cGV9Kz0kY29tYmluZWR7J3NpemUnfTsKCX0KCglpZiggbm90IGRl
ZmluZWQgJEVYVFJBLT57J3N0YXJ0J30gb3IgJG1vbnRoIDwgJEVYVFJBLT57J3N0YXJ0J30t
PnsnbSd9IG9yICRkYXkgPCAkRVhUUkEtPnsnc3RhcnQnfS0+eydkJ30gKSB7CgkJJEVYVFJB
LT57J3N0YXJ0J30tPnsnbSd9ID0gJG1vbnRoOwoJCSRFWFRSQS0+eydzdGFydCd9LT57J2Qn
fSA9ICRkYXk7CgkJfQoJaWYoIG5vdCBkZWZpbmVkICRFWFRSQS0+eydlbmQnfSBvciAkbW9u
dGggPiAkRVhUUkEtPnsnZW5kJ30tPnsnbSd9IG9yICRkYXkgPiAkRVhUUkEtPnsnZW5kJ30t
PnsnZCd9ICkgewoJCSRFWFRSQS0+eydlbmQnfS0+eydtJ30gPSAkbW9udGg7CgkJJEVYVFJB
LT57J2VuZCd9LT57J2QnfSA9ICRkYXk7CgkJfQp9CgpzdWIgbG9hZF9hbGlhc2VzCnsKCW15
ICRmaWxlID0gc2hpZnQ7CglvcGVuKEYsIjwkZmlsZSIpOwoJd2hpbGUoPEY+KSB7CgkJY2hv
bXAoKTsKCQluZXh0IHVubGVzcyggL14oLiopXHMqOlxzKihbXixdKikkLyApOwoJCSRBTElB
U0VTe2xjKCQxKX09bGMoJDIpOwoJfQoJY2xvc2UoRik7Cn0KCnN1YiByZXNvbHZlX2FsaWFz
CnsKCW15ICgkYWRkcikgPSBAXzsKCSRhZGRyID0gbGMoJGFkZHIpOwoJJGFkZHIgPX4gcy9b
PD5dLy9nOwoJJGFkZHIgPX4gcy9cQCRsb2NhbGRvbWFpbi8vIGlmKCAkbG9jYWxkb21haW4g
KTsKCWlmKCAkQUxJQVNFU3skYWRkcn0gKSB7CgkJJGFkZHIgPSAkQUxJQVNFU3skYWRkcn07
CgkJfQoJcmV0dXJuICRhZGRyOwp9CgpzdWIgZXh0cmEKewoJJDo6RVhUUkEtPnsnMC4g0J7R
gtGH0ZHRgtC90YvQuSDQv9C10YDQuNC+0LQnfSA9IHNwcmludGYgIiUwMmQvJTAyZCAtICUw
MmQvJTAyZCIsCgkJJEVYVFJBLT57J3N0YXJ0J30tPnsnZCd9LAoJCSRFWFRSQS0+eydzdGFy
dCd9LT57J20nfSwKCQkkRVhUUkEtPnsnZW5kJ30tPnsnZCd9LAoJCSRFWFRSQS0+eydlbmQn
fS0+eydtJ307CgkKCWlmICgkRVhUUkEtPnsndG90YWxjb3VudCd9LT57J0knfSkgewoJCSQ6
OkVYVFJBLT57JzEuINCS0YXQvtC00Y/RidC40LUnfSA9IHNwcmludGYgIiUgNXMgLyAlIDEy
cyIsCgkJJEVYVFJBLT57J3RvdGFsY291bnQnfS0+eydJJ30sJEVYVFJBLT57J3RvdGFsYW1v
dW50J30tPnsnSSd9LiIg0LHQsNC50YIiOwoJCX0KCWlmICgkRVhUUkEtPnsndG90YWxjb3Vu
dCd9LT57J08nfSkgewoJCSQ6OkVYVFJBLT57JzIuINCY0YHRhdC+0LTRj9GJ0LjQtSd9ID0g
c3ByaW50ZiAiJSA1cyAvICUgMTJzIiwgCgkJJEVYVFJBLT57J3RvdGFsY291bnQnfS0+eydP
J30sJEVYVFJBLT57J3RvdGFsYW1vdW50J30tPnsnTyd9LiIg0LHQsNC50YIiOwoJCX0KCWlm
ICgkRVhUUkEtPnsndG90YWxjb3VudCd9LT57J0knfSkgewoJCSQ6OkVYVFJBLT57JzMuINCb
0L7QutCw0LvRjNC90YvQtSd9ID0gc3ByaW50ZiAiJSA1cyAvICUgMTJzIiwKCQkkRVhUUkEt
PnsndG90YWxjb3VudCd9LT57J0wnfSwkRVhUUkEtPnsndG90YWxhbW91bnQnfS0+eydMJ30u
IiDQsdCw0LnRgiI7CgkJfQoJaWYgKCRFWFRSQS0+eyd0b3RhbGNvdW50J30tPnsnUid9KSB7
CgkJJDo6RVhUUkEtPnsnNC4g0J/RgNC+0YXQvtC00Y/RidC40LUnfSA9IHNwcmludGYgIiUg
NXMgLyAlIDEycyIsCgkJJEVYVFJBLT57J3RvdGFsY291bnQnfS0+eydSJ30sJEVYVFJBLT57
J3RvdGFsYW1vdW50J30tPnsnUid9LiIg0LHQsNC50YIiOwoJCX0KfQo=
------------A3E21F3017530A--