Mailtraq Log Reporting v2

This forum is for discussing Mailtraq's Scripting features. Get help with Mailtraq scripts, templates and external ActiveX scripting.

Mailtraq Log Reporting v2

Postby jimhill » Tue Dec 11th, 2007 9:10pm

Hi all

In response to Martin Clayton's recent request for a report option to output to \database\mail\pending and both Martin's and Nico van Vliet's problems with smtp timeouts when handling very large logfiles, I've updated my website with v2 [url]http://rdns.org/mailtraq/logs/perl/mtq-log-v2.zip[/url].

I've resolved the smtp timeout by the simple expedient of initiating the smtp connection after the script analysis has completed (which is how it should have been in the first place, of course, but we didn't have rogue ftp clients spewing their failed logins 80,000 times per day - on a single connection! - into Mailtraq's logs back then).

The pending option, enabled in the inifile as per normal, should work seamlessly (thanks to Nico for his testing efforts) but I've had to synthesise quite a lot of information in order to recreate a Received header (optional) which may not be to everyone's liking.

I've added quite a few regex match expressions and I've cleaned up quite a few other bits and pieces at the same time so please let me know if there are any problems with anything in v2. Thanks.
jimhill
Expert User
 
Posts: 337
Joined: Sun Dec 19th, 2004 9:59pm
Location: UK

Re: Mailtraq Log Reporting v2

Postby Martin Clayton » Wed Dec 12th, 2007 8:58am

jimhill wrote:In response to Martin Clayton's recent request for a report option to output to \database\mail\pending

I was thinking more along the lines of bodging your work of art. ;-) While we're on requests though, my main 'want' was to support date-based logfile names but I think you've already done that (thanks). Next, is suppression of incident lists. I've just run v2 pointing at pending with the default search patterns - some of the terms output many thousands of lines - although interesting, I can't cope with the volume (and Thunderbird is slow to load large messages) but I'd quite like the counts. (As it stands, I guess I could substitute .*|nul but I expect I'd be left with blank lines, oh, also the timestamp). Similarly, aggregating unique item-level occurrences would be useful e.g. on smtp blacklists there may be many entries for a small number of addresses. Dropping the timestamps and counting incidence would be useful. Bad_address1: 1, Bad_address2: 1240, etc. I expect that this request - no, let's call it an idea - would require a lot of work.

jimhill wrote:and both Martin's and Nico van Vliet's problems with smtp timeouts when handling very large logfiles,

I think my first timeout was on a relatively normal sized file - about 12Mb I'd guess, with 20 or 30 search terms. At the time, I cut the terms down to 3 rather than going for file output. The 3-term search did fall over on the more recent massive file (~282Mb) but "out of memory" occurred before getting to smtp. I can't be sure what happened in the dead of night but when I logged in and ran at the command line an "out of memory" message was returned.

jimhill wrote:The pending option, enabled in the inifile as per normal, should work seamlessly (thanks to Nico for his testing efforts) but I've had to synthesise quite a lot of information in order to recreate a Received header (optional) which may not be to everyone's liking.

All seems mighty fine to me.
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK

Re: Mailtraq Log Reporting v2

Postby jimhill » Wed Dec 12th, 2007 2:05pm

Martin Clayton wrote:
jimhill wrote:In response to Martin Clayton's recent request for a report option to output to \database\mail\pending
I was thinking more along the lines of bodging your work of art. ;-)
Not really mine, I'm afraid. Remember this?
Code: Select all
# Author:      Originally by Jim Hill in MTQ scripting language
#              Translated to Perl by Chris Hastie
# Date:        December 2000
I have changed it substantially since then, of course, but, in retrospect, Chris Hastie's work was one of my principal motivations for starting to learn perl.
Martin Clayton wrote:While we're on requests though, my main 'want' was to support date-based logfile names but I think you've already done that (thanks).
Yep, it looks trivial now but, at the time, I was quite pleased with my algorithm for that.
Martin Clayton wrote:Next, is suppression of incident lists.
That's an easy one. Push line 67 down to add the following
Code: Select all
    # test for details enabled on this section
    my $sec_detail = 1 if ($ini{$section}{detail} == 2);
    $sec_detail = 1 if (($ini{$section}{detail} == 1) && ($ini{report}{detail}));
    next unless ($sec_detail);
   
    # if there are any matches in this iteration -- this is line 67
    if (@found) {
then you need to add 'detail=1' (enable globally) or 'detail=0' (disable globally) to the [Report] section. Finally, in the remaining sections, add 'detail=0' to suppress detail of that section regardless of the global setting, 'detail=1' to accept the current global setting or 'detail=2' to enable detail for that section regardless of the global setting. I think that covers all bases. If you prefer to wait, I'll update the website with v2.1 when I've tested that minor change locally.
Martin Clayton wrote:Similarly, aggregating unique item-level occurrences would be useful e.g. on smtp blacklists there may be many entries for a small number of addresses. Dropping the timestamps and counting incidence would be useful. Bad_address1: 1, Bad_address2: 1240, etc. I expect that this request - no, let's call it an idea - would require a lot of work.
That's a good idea and yes, it does look to be a lot of work at first sight but leave it with me and I'll see what I can come up with.
jimhill
Expert User
 
Posts: 337
Joined: Sun Dec 19th, 2004 9:59pm
Location: UK

Re: Mailtraq Log Reporting v2

Postby Martin Clayton » Wed Dec 12th, 2007 4:58pm

jimhill wrote:Not really mine, I'm afraid. Remember this?
Code: Select all
# Author:      Originally by Jim Hill in MTQ scripting language
#              Translated to Perl by Chris Hastie
# Date:        December 2000

I think so, just about, but I may be conning myself.

jimhill wrote:Push line 67 down to add the following
Code: Select all
    # test for details enabled on this section
    my $sec_detail = 1 if ($ini{$section}{detail} == 2);
    $sec_detail = 1 if (($ini{$section}{detail} == 1) && ($ini{report}{detail}));
    next unless ($sec_detail);
   
    # if there are any matches in this iteration -- this is line 67
    if (@found) {
then you need to add 'detail=1' (enable globally) or 'detail=0' (disable globally) to the [Report] section.

Great, thanks. I went for the former as the last 'Report' field.

jimhill wrote:Finally, in the remaining sections, add 'detail=0' to suppress detail of that section regardless of the global setting, 'detail=1' to accept the current global setting or 'detail=2' to enable detail for that section regardless of the global setting. I think that covers all bases. If you prefer to wait, I'll update the website with v2.1 when I've tested that minor change locally.

I'm getting...
Code: Select all
Use of uninitialized value in numeric eq (==) at C:\1work\mtq-log\mtq-log.bat line 83.
Use of uninitialized value in numeric eq (==) at C:\1work\mtq-log\mtq-log.bat line 84.
...for any search record that's set with 'detail=0'. (Records with no 'detail' field also get the same. As we're in request territory, I'd prefer the global default to apply unless there's a local override).

jimhill wrote:
Martin Clayton wrote:Similarly, aggregating unique item-level occurrences would be useful
That's a good idea and yes, it does look to be a lot of work at first sight but leave it with me and I'll see what I can come up with.

Best place for it! No rush, of course, but thanks again.
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK


Return to Mailtraq Scripting

Who is online

Users browsing this forum: No registered users and 2 guests

cron