Excommunicado reverse path dns rbl

This forum is for discussing Mailtraq's Scripting features. Get help with Mailtraq scripts, templates and external ActiveX scripting.

Excommunicado reverse path dns rbl

Postby Martin Clayton » Fri Mar 14th, 2014 7:51am

This script uses the excommunicado dns blacklist to thwart hepworth spam. See Martin Brooks' blog for details and note that you might need permission to use his (permanently free) service.

It tests the smtp reverse-path and if a hit is found diverts the message to the mailslot specified in 'CFG.hitMailslot' (line 27). There might be an .ini file for config options one day.

It can also add the domain to the smtp service's Sender Barring. This approach doesn't scale as well as native support for reverse-path dns lookups but at least it stops spam at the border - after accepting at least one message. The script is somewhat experimental but it's working well in production so big-thanks to Martin Brooks.

If you're running Mailtraq with spamassassin set to 'reject during receipt' then you're probably better off implementing the service directly under spamassassin. (I don't run spamassassin but a quick look suggested it was surprisingly difficult to set up).

Code: Select all
<@LANGUAGE=Javascript@>
<%
/*  excommunicado.mtq
    -----------------
   
    Use Excommunicado rhsbl to lookup smtp reverse-path host/domain
    See: https://github.com/antibodyMX/communicado & http://bit.ly/17uEO9Y
   
    On hit: readdress message
            add the offending (sub)domain to Sender Barring
            generate Mailtraq log entry
   
    Mailtraq forum: http://support.mailtraq.com/viewtopic.php?f=15&t=2062
   
    Trigger: Incoming Mail
   
    Version: 0.7  23/03/2014
*/

var CFG = {

    // rhsbl query host
    rhsbl         : "excommunicado.co.uk",
    hitResponse   : "127.0.0.2",

    // ignore senders using local domain names
    ignoreLocal   : true,

    // ignore non-root domains [not implemented]
    // ignoreNonRoot : false,

    // new destination mailslot or address (empty string or null for no re-routing)
    hitMailslot   : "spam-excommunicado",

    // header to show recipients prior to any rewriting (empty string or null for no header)
    hdrRcpts      : "X-MTQ-Excom-Original-Rcpts",

    // add domain to Sender Barring
    bar           : true,
    barType       : 1,               // 1 = Incoming Mail, 2 = SMTP service
    barSmtpPort   : 25,              // service location (port) for SMTP sender barring

    // add entry to Mailtraq log file on hit (00008000 00000BB4 17/03/2014 23:26:51 [CFG.mtqLogText])
    mtqLog        : true,

    // valid tokens $domain [matched host/domain], $barred [true|false], $version (this script),
    //              $header(any-header) & $cfg(any-valid-name)

    mtqLogText    : 'Excommunicado: matched "$domain" [barring; active=$cfg(bar) type=$cfg(barType) update=$barred] mid="$header(Message-ID)"'

}, SCRIPT = {

    version       : "0.7"

}, HOST = {

    // script host environment: local objects to avoid expensive calls

    server       : Server,
    request      : this.server.Request

}, MSG = {

    senderDomain    :   function () {

                            var domain = HOST.server.Toolkit.DomainOfEmail(HOST.request.Sender);

                            // redefine self on first completion for lower cost multiple calls
                            this.senderDomain = function () {
                                return domain;
                            };

                            return domain;

                        },

    getRcptArray    :   function () {

                            // only need to run this once so redefine self on first completion
                            var rcpt_array = HOST.request.Recipients.Text.split("\r\n");

                            rcpt_array.pop();  // remove extraneous 'blank'
                            this.getRcptArray = function () {
                                return rcpt_array;
                            };

                            return rcpt_array;

                        },

    writeRcptHdr    :   function () {

                            var fieldName = CFG.hdrRcpts,
                                fieldBody = this.getRcptArray().join("\r\n  ");

                            if (/[^\!-9;-~]/.test(fieldName) || fieldName.length < 1) {
                                // valid: ascii decimals 33-57 & 59-127 (32=space, 58=colon, ignore 127=delta)
                                LOG.entry(LOG.errFieldName);

                            } else {

                                HOST.request.SetHeader(fieldName, fieldBody);
                            }

                            return;

                        },

    rewriteRcpts    :   function () {

                                var rcptCount = this.getRcptArray().length,
                                    i         = 0;

                                while (i < rcptCount) {
                                    HOST.request.Recipients.Delete(0);
                                    i++;
                                }
                                HOST.request.Recipients.Add(CFG.hitMailslot);

                            return;

                        }

}, SITE = {

    isDomainLocal   :   function (domain) {

                            // this wont necessarily work on a multi-domain build
                            // and doesn't look-for/test the root domain
                            var domainPrimary = HOST.server.Config.LocalHost,
                                domainAliases = HOST.server.Config.LocalHostAliasList;

                            return ((domain === domainPrimary) || (domainAliases.IndexOf(domain) !== -1));

                        },

    getSMTPservice  :   function () {

                            var serviceObj         = null,
                                serviceCount       = HOST.server.Config.ServiceList.Count,
                                serviceFound       = false;

                            // smtp service class is 0 & the port must match the CFG specified location
                            for (var i = 0; i < serviceCount; i++) {
                                serviceObj = HOST.server.Config.ServiceList.GetService(i);
                                if ((serviceObj.Location === '' + CFG.barSmtpPort) && (serviceObj.ServiceClass === 0)) {
                                    serviceFound = true;
                                    break;
                                }
                            }
                            // return null or valid object
                            if (!serviceFound) {
                                LOG.entry(LOG.errNoSmtp);
                                return null;
                            }

                            return serviceObj;

                        },

    barSender       :   function (domain) {

                            var smtp           = (+(CFG.barType) === 2) ? true : false,
                                smtpService    = {},
                                domainWildcard = "*@" + domain,
                                alreadyListed  = false;

                            // get out of here on host/domain without "."
                            if (! /\./.test(domain)) {
                                return;
                            }
                            // get out of here if host/domain contains Mailtraq wildcard characters
                            if (/[*?%~]/.test(domain)) {
                                LOG.entry(LOG.errWildDomain);
                                return;
                            }

                            if (smtp) {

                                smtpService = this.getSMTPservice();
                                if (smtpService === null) {

                                    return;

                                } else {

                                    alreadyListed = (smtpService.BlacklistFrom.IndexOf(domainWildcard) !== -1);
                                    if (!alreadyListed) {
                                        smtpService.BlacklistFrom.Add(domainWildcard);
                                        smtpService.Invalidate();
                                        this.barSenderUpdate = true;
                                    }

                                }

                            } else { // site-wide barring

                                // must use javascript indexOf() not Mailtraq IndexOf()
                                alreadyListed = (HOST.server.Mailtraq.SenderBarring.indexOf(domainWildcard) !== -1);
                                if (!alreadyListed) {
                                    // mildly freaky that direct assignment works while the expected Add() Set() don't
                                    HOST.server.Mailtraq.SenderBarring += domainWildcard;
                                    this.barSenderUpdate = true;
                                }

                            }

                            return ;

                        },

    barSenderUpdate :   false

}, LOOKUP = {

    goAhead         :   function () {

                            // domain/host must contain "."
                            if (! /\./.test(MSG.senderDomain()) ) {
                                return false;
                            }
                            // respect option to ignore local sender domain names
                            if (CFG.ignoreLocal && SITE.isDomainLocal(MSG.senderDomain())) {
                                return false;
                            }

                            return true;

                        },

    hit             :   function () {

                            var response = HOST.server.Toolkit.DnsLookupA(MSG.senderDomain() + "." + CFG.rhsbl);

                            return (response === CFG.hitResponse);

                        }

}, LOG = {

    errNoSmtp       : "Excommunicado: ! error; smtp service not found on port [" + CFG.barSmtpPort + "] (v" + SCRIPT.version + ")",
    errWildDomain   : 'Excommunicado: ! warning; domain "' + MSG.senderDomain() + '" not added to sender barring due to unexpected wildcard in msg "$header(Message-Id)" (v' + SCRIPT.version + ")",
    errFieldName    : 'Excommunicado: ! warning; CFG.hdrRcpts "' + CFG.hdrRcpts + '" specifies an invalid header name (v' + SCRIPT.version + ")",

    entry           :   function (text) {

                            var entry        = text,
                                fieldName    = '',
                                fieldValue   = '',
                                re           = {},
                                i            = 0,
                                outOfControl = 50;

                            if (/\$domain/i.test(entry)) {
                                entry = entry.replace(/\$domain/ig, MSG.senderDomain());
                            }
                            if (/\$barred/i.test(entry)) {
                                entry = entry.replace(/\$barred/ig, SITE.barSenderUpdate);
                            }
                            if (/\$version/i.test(entry)) {
                                entry = entry.replace(/\$version/ig, SCRIPT.version);
                            }
                            // 'dynamic' tokens: $header(foo) and $cfg(foo)
                            // ideally, should probably escape any regex special characters in fieldName
                            while (/\$(header|cfg)\(/i.test(entry) && i < outOfControl) {
                                if (/\$header\(.*?\)/i.test(entry)) {
                                    fieldName = entry.replace(/.*?\$header\((.*?)\).*/i, "$1");
                                    fieldValue = HOST.request.GetHeader(fieldName);
                                    re = new RegExp("\\$header\\(" + fieldName + "\\)", "ig");
                                    entry = entry.replace(re, fieldValue);
                                }
                                if (/\$cfg\(.*?\)/i.test(entry)) {
                                    fieldName = entry.replace(/.*?\$cfg\((.*?)\).*/i, "$1");
                                    fieldValue = CFG[fieldName];
                                    re = new RegExp("\\$cfg\\(" + fieldName + "\\)", "ig");
                                    entry = entry.replace(re, fieldValue);
                                }
                                i++;
                            }

                            // [additional hex params possible with Log()]
                            HOST.server.Mailtraq.Log(entry);

                            return;

                        }

};

if (LOOKUP.goAhead() && LOOKUP.hit()) {
    if (CFG.hdrRcpts) {
        MSG.writeRcptHdr();
    }
    if (CFG.hitMailslot) {
        MSG.rewriteRcpts();
    }
    if (CFG.bar) {
        SITE.barSender(MSG.senderDomain());
    }
    if (CFG.mtqLog) {
       LOG.entry(CFG.mtqLogText);
    }
}

%>

Any questions, this is the place.

EDITs:
14/03/14 : updated script to v0.3
17/03/14 : updated to 0.4
17/03/14 : updated to 0.5
24/03/14 : updated to 0.7, added release notes, corrected note on permission: "need" -> "might need".

Release Notes
Code: Select all
0.3  14/03/2014  Correction to SITE.getSMTPservice()
0.4  17/03/2014  Tidy-up & reorganised lookup conditions (when to skip test)
                 added Invalidate() on smtp Sender Barring - forces service restart
0.5  17/03/2014  Add option to write Mailtraq log entry on hit
0.6  18/03/2014  Add custom log entry & error messages
0.7  23/03/2014  Sender Barring can operate on Incoming Mail or SMTP Barring
                 Added CFG.hdrRcpts name validation
                 Improved MSG.rewriteRcpts()
                 Message re-routing now optional
                 Rejigged logging customisation
Last edited by Martin Clayton on Mon Mar 24th, 2014 11:53am, edited 4 times in total.
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK

Re: Excommunicado reverse path dns rbl

Postby jimhill » Fri Mar 14th, 2014 10:11am

Hi Martin

Nice script but, imo of course, Mailtraq's native scripting language is still tidier and more legible. However, I commend you for taking the time and trouble to learn it all the same.

Your script, btw, references a rhsbl (Right Hand Side Block List) not a dnsbl (Domain Name System Block List). With the recent vast increase in tld (Top Level Domain) and with many more yet to come from icann, the possible number of abusive return-paths is virtually infinite making it unwise, imo, to risk adding them to Mailtraq's blacklist even with the benefit of being able to reject them during delivery.

As you're no doubt well aware, it would be much easier to handle these issues if Mailtraq supported rhsbl queries during the inbound smtp protocol exchange.
jimhill
Expert User
 
Posts: 337
Joined: Sun Dec 19th, 2004 9:59pm
Location: UK

Re: Excommunicado reverse path dns rbl

Postby Martin Clayton » Fri Mar 14th, 2014 11:01am

jimhill wrote:Nice script but, imo of course, Mailtraq's native scripting language is still tidier and more legible.

After the initial culture shock, I've fallen into the habit and I like the aesthetic. It was probably David Crockford's site that got me going. I even managed to communicate with the great man - well, he sent me two words by email. :)
I suspect he'd be horrified if he saw the script and looking now I can see a proper clanger in SITE.getSMTPservice(). I'll update the post with a fix later (0.3).

jimhill wrote:Your script, btw, references a rhsbl (Right Hand Side Block List) not a dnsbl (Domain Name System Block List).

Ah, thanks. I've been struggling with the terminology.

jimhill wrote:... the possible number of abusive return-paths is virtually infinite making it unwise, imo, to risk adding them to Mailtraq's blacklist

Oops, I tend to call it the reverse-path but yup, even excommunicado lists 6,000+ domains already. For now I'm happy to risk it but in the longer run...

jimhill wrote:it would be much easier to handle these issues if Mailtraq supported rhsbl queries

... that's got to be the way to go.
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK

Re: Excommunicado reverse path dns rbl

Postby Martin Clayton » Wed Apr 23rd, 2014 12:47pm

Things have moved on a bit here. Despite frequent and timely excommunicado updates (a great service) I was still seeing a fair number of uce messages get through. To-date, they've all used verp addressing so using greylisting with wildcard negation:

Code: Select all
*
~returns-my_user=my_domain.com@my_domain.com

... only affect Communicado-like traffic and a few minutes is usually enough time for the rhsbl to populate.

In the absence of native support for rhsbl or script triggers during the smtp transaction I decided to use perl to monitor the log for suspicious reverse-paths and perform lookups at excommunicado and, if necessary, whois. On any hit a control message is dropped in pending for a Mailtraq script to update Sender Barring. This means it's possible to reject all known Communicado traffic.

There's no remote pop3 collection here so Incoming Mail, Sender Barring is now purely used for Communicado messages and a bespoke rejection message is set:

Code: Select all
Communicado Ltd., see http://blog.hinterlands.org/2013/10/unwanted-email-from-communicado-ltd


The 'good thing' about defending against zero-day type domains is that Sender Barring can be kept minimal - frequently purging of the barring list wouldn't do any harm (there's a local copy of known bad domains to prevent repeat rhsbl or whois lookups). Ideally, you'd consult the greylist manager before knocking out any entries but there's been no need to manage the list, so far.

The scripts are still experimental - they're doing good work with minimal cost to the innocent - but there are some rough edges.

The largely randomly chosen perl whois module can fall over:
Code: Select all
whois.crsnic.net: A connection attempt failed because the connected party did not properly respond after a per
iod of time, or established connection failed because connected host has failed to respond.: whois.crsnic.net:
43 at C:/Perl/site/lib/Net/Whois/Raw.pm line 281.

Strangely, I've only seen this on win2k (32bit), not win8.1 (64bit) when both run perl 5.16.3. I haven't looked too far but the easiest fix is probably to use a different module.

Wanting to run the script non-locally I decided to use Mailtraq's telnet logging rather than sitting on the log file over the network. For longer term use I'd install the script up as a windows service on the production machine - in which case, it's probably better to use the local log file or greylist manager. Performance and lost buffer issues haven't arisen yet but the timeouts need tweaking and debug logging is excessive when the script loses its connection to the Mailtraq machine or logging service.

If anyone wants to see the scripts post here I'll start a separate topic.

Alternatively, with native support for rhsbl ;) all we'd need is a greylisting entry.
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK

Re: Excommunicado reverse path dns rbl

Postby jimhill » Wed Apr 23rd, 2014 6:46pm

Martin Clayton wrote:Things have moved on a bit here. Despite frequent and timely excommunicado updates (a great service) I was still seeing a fair number of uce messages get through.

There's only one way that I know of to keep control over inbound mail and that's to use custom addresses for each and every external contact. I've done that from the start with rdns.org and my spam folder in agent contains exactly one message since the start of 2014 - a blurb about RankTrader addressed to hostmaster.
Martin Clayton wrote:In the absence of native support for rhsbl or script triggers during the smtp transaction I decided to use perl to monitor the log

The best way to do this, I've found, is to monitor Mailtraq's logfile output. I've had my script running for hours on end without problems.
Code: Select all
# tail.pl

use IO::Handle;
use DateTime::Tiny;
use Fcntl qw(:seek);
autoflush STDOUT;
use strict;
use warnings;

die "\nSyntax: tail drive:\\path\\file\.xtn [1]\n        tail [mtq|dns] [1]\n        1 = display from current end of file\n            else display from start of file\n" unless $ARGV[0];

$SIG{'INT'} = \&control_c;

my $now = DateTime::Tiny -> now;

my $file = $ARGV[0];
if ($file eq "mtq") {
    $file = "\\\\Mtq-mail\\c\\Mailtraq\\log\\".$now -> year."-".sprintf("%02d", $now -> month)."-".sprintf("%02d", $now -> day)."-log.txt";
} elsif ($file eq "dns") {
    $file = "\\\\Sdns\\c\\log\\dns\\".$now -> year.sprintf("%02d", $now -> month).sprintf("%02d", $now -> day).".log";
}

my $start = $ARGV[1];
my $pos = -s $file if $start;
my $test = 1;

while ($test) {
    open LOG, $file or die "Cannot open $file, $!";
    seek LOG, $pos, SEEK_SET if defined $pos;
    seek LOG, $pos, 0 if defined $pos;
    # replace next line only with your code
    print while <LOG>;
    $pos = tell LOG;
    close LOG;
    sleep 1;
}

sub control_c {
  print "\nDropping tail on $file\n";
  undef $test;
  kill 'INT', $$;
}
which you'll need to adjust for your paths and logfilenames. Also, I've indicated where your code would start.

Martin Clayton wrote:The largely randomly chosen perl whois module can fall over:

I think you'll get a better handle on whois if you do it yourself instead of using modules of uncertain provenance. Like most things in networking, it's actually quite easy to do once you've see how it works. First thing you have to realise, though, is that there is no longer guaranteed to be a working whois server for every tld - yes, icann are a bunch of useless paper-pushers. Also, due to tld proliferation, particularly lately, there's no longer a reliable _single_ whois server that can answer all whois queries. So, what you have to do is first ask iana for the correct whois server, then parse its output for that server and then repeat your whois query to the new whois server. Like this ...
Code: Select all
use IO::Socket;
use strict;
use warnings;

my $qry = "rdns.org";

my $socket = IO::Socket::INET -> new(
    Proto => "tcp",
    PeerAddr => "whois.iana.org",
    PeerPort => "43",
    Timeout => 10
);
die "\nCan't connect to \"whois.iana.org\"\n" unless $socket;

print $socket "$qry\015\012";
my @out = <$socket>;
close $socket;

my ($field, $server);
foreach (@out) {
    next unless /^whois: /;
    ($field, $server) = split /\:/, $_;
    $server =~ s/^\s+//;
    last;
}

print "$server\n";
Now repeat the whois query to that whois server using the same methodology as above. I'd be more explicit but I don't know how this would fit into your existing script setup.

Having said all that, there are two other pieces of information you need about the whois service. Firstly, some servers require grossly obscure input syntax for unfathomable reasons (.de does this, I think) so your simple domain query will fail. Secondly, not all tld have a whois service running on port 43 because, I believe, icann didn't make it a requirement in their specifications - some tld run a web service but others provide nothing at all.
Martin Clayton wrote:Alternatively, with native support for rhsbl ;) all we'd need is a greylisting entry.
You never know, it might happen one day. When that day comes, I have a list of useful rhsbl to share.
jimhill
Expert User
 
Posts: 337
Joined: Sun Dec 19th, 2004 9:59pm
Location: UK

Re: Excommunicado reverse path dns rbl

Postby Martin Clayton » Mon Apr 28th, 2014 11:36am

Hi Jim
jimhill wrote:The best way to do this, I've found, is to monitor Mailtraq's logfile output. [...]
Code: Select all
# tail.pl

Great. That looks very useful and has quite a few 'tricks' I've not seen before.

jimhill wrote:I think you'll get a better handle on whois if you do it yourself
Code: Select all
use IO::Socket;

Thanks for the run-down. My initial take on the problem was well out of whack -- I should have realised that all dmz whois lookups were failing and none on the lan; it was a simple firewall issue. The script is performing as expected now and as far as I can see the whois module is fairly well maintained so I'll stick with it unless 'something else' happens.

The plan is now to get the Win32::Daemon running. For a gratuitously quick hit, I'll stick with telnet logging but for the longer run 'tail' looks better...

Cheers,
Martin
User avatar
Martin Clayton
Expert User
 
Posts: 529
Joined: Sat Jan 15th, 2005 8:20am
Location: London, UK


Return to Mailtraq Scripting

Who is online

Users browsing this forum: No registered users and 1 guest