Amavisd

Stages of spam-prevention Integrating amavid-new into Postx The nature of content control amavisd-new Implementations Ba
Controlling mail message ow by content analysis

Ralf Hildebrandt Patrick Ben Koetter
T-System Business Services & state of mind
Open Source Days 2008 Copenhagen, 04. October 2008
Ralf Hildebrandt, Patrick Ben Koetter Controlling mail message ow by content analysis
Stages of spam-prevention Where can spam be controlled? Split anti spam measures SMTP Session Control Content Control Integrating amavid-new into Postx smtpd_proxy_lter content_lter Where is the difference? The nature of content control Risks Requirements amavisd-new History Architecture
Implementations Typical Mailserver Typical Mailgateway A dedicated Scanhost Basic Setup Logging Virus Scanners Spam Scanners Quarantine Notication Housekeeping Quarantine Management Quarantine Notication Quarantine Release Log analysis and statistics
Individualization Maps LDAP Policy Banks Policy Banks Example
Beyond the standard stuff Keeping SpamAssassin up to date Extending SpamAssassins rules Extending ClamAV patterns Pen-Pals p0f - passive operating system ngerprinting DKIM - Domainkeys Identied Mail
Scan before and after queue
e-mail can be scanned before and after queueing

Ralf Hildebrandt, Patrick Ben Koetter Controlling mail message ow by content analysis T-System Business Services & state of mind
Split anti spam measures
Measures are typically split

A day in the life of a mail server
More than 98% of mail is spam.

A not so good idea ...
Content scanning is resource expensive. Dont let the full load hit the content scanner
SMTP Session Control reduces load
Reject early! Use SMTP Session Control to minimize the mail that hits after queue measures
Limitations of Before-Queue-Scanning
In a perfect world content inspection takes place before queuing, but clients may run into a timeout.
SMTP Session Control
Best practices require RFC compliance from your senders require RFC compliance rst kill spam later order restrictions from cheap to expensive allow for exceptions!
smtpd_recipient_restrictions I
Cheap restrictions:
smtpd_recipient_restrictions = reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain, check_client_access cdb:/etc/postfix/mynetworks permit_sasl_authenticated, reject_unauth_destination, reject_unlisted_recipient, ...
These keep your users from sending the worst nonsense.
smtpd_recipient_restrictions II
Cheap restrictions:
... check_helo_access pcre:/etc/postfix/helo_checks.pcre, reject_invalid_helo_hostname reject_non_fqdn_helo_hostname check_recipient_access cdb:/etc/postfix/roleaccount_exceptions, check_sender_access cdb:/etc/postfix/sender_checks check_sender_access pcre:/etc/postfix/sender_checks.pcre ...
Were checking the HELO for obvious signs of forgery and block some senders.
smtpd_recipient_restrictions III
Expensive restrictions:
... check_client_access check_sender_mx_access check_sender_ns_access check_sender_mx_access check_sender_mx_access check_sender_mx_access check_client_access
cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/bogon_networks.cidr, cidr:/etc/postfix/wildcard_mx_records.cidr, cdb:/etc/postfix/mx_accepts_no_bounces cdb:/etc/postfix/client_checks
The checks using cidr:/etc/postfix/drop.cidr check if a sender domains MX or NS record point to networks controlled by spammers. This cidr:/etc/postfix/bogon_networks.cidr check if a sender domains MX record points to private netspace. . . .
smtpd_recipient_restrictions IV
Expensive restrictions:
... reject_unknown_reverse_client_hostname reject_rbl_client zen.spamhaus.org reject_rbl_client bl.spamcop.net reject_rbl_client b.barracudacentral.org check_sender_access cdb:/etc/postfix/rhsbl_exceptions reject_rhsbl_sender dsn.rfc-ignorant.org
Consequences
Postx rejects a lot of connections early content_filter and smtpd_proxy_filter are relieved from a lot of load (90% rejection rate)
header_checks
header_checks check headers only:
/^Received: from .*by nomail\.charite\.de/ REJECT Fake nomail.charite.de Received: Header found /Date:.*([3-9].:..:|2[4-9]:..:|:[6-9].:|:..:6[1-9])/ REJECT Invalid time "${1}" in Date header
mime_header_checks
mime_header_checks check MIME-headers only:

/name=\"(.*)\.(386|bat|chm|cmd|com|do|exe|hta|jse|lnk|msi)\"$/ REJECT Unwanted attachment $1.$2
body_checks
body_checks checks everything thats not a header:

if /^Received: from .*by mail(-ausfall)?\.charite\.de/ !/ $Postfix$ with /i REJECT Fake charite.de Received: Header found, this \ is a bounce for a mail our system did not send! endif
smtpd_proxy_lter
Accept mail from the internet, pass it on to the lter on [127.0.0.1]:10025 Caution Enable XFORWARD this allows amavisd-new to see the actual client instead of just 127.0.0.1.
smtp -o -o -o -o inet n receive_override_options=no_address_mappings smtpd_authorized_xforward_hosts=127.0.0.0/8 smtpd_authorized_xclient_hosts=127.0.0.0/8 smtpd_proxy_filter=[127.0.0.1]:10025 smtpd
Reinjection
Accept the mail back into Postx after amavisd-new is done with it: Caution Disable lter on reentry!
localhost:10026 inet n smtpd -o smtpd_proxy_filter= -o content_filter= -o receive_override_options=no_unknown_recipient_checks -o smtpd_authorized_xforward_hosts=127.0.0.0/8 -o smtpd_authorized_xclient_hosts=127.0.0.0/8 -o mynetworks=127.0.0.0/8 -o smtpd_recipient_restrictions=permit_mynetworks,reject -o syslog_name=reinjection
content_lter
Accept mail from the internet, pass it on to the lter on [127.0.0.1]:10025 Caution Enable XFORWARD this allows amavisd-new to see the actual client instead of just 127.0.0.1.
smtp -o -o -o -o inet n receive_override_options=no_address_mappings smtpd_authorized_xforward_hosts=127.0.0.0/8 smtpd_authorized_xclient_hosts=127.0.0.0/8 content_filter=smtp:[127.0.0.1]:10025 smtpd
Where is the difference?
with smtpd_proxy_filter the mail is processed as it arrives (just in time-scanning) with content_filter the mail is queued rst and scanned later Caution Clients may not be patient enough to wait until your system is willing to accept and process mail.
The nature of content control
Risks
Content contains malware Content may decompress without bounds (compression bombs) Content cannot be scanned (encrypted) Content cannot be categorized
Requirements
Unpack into sandbox Apply limits to size(s) Apply warnings to messages
A brief history
Shell script that pipes mail to a scanner (1997) First Perl program (2000) Perl daemon (2001) Modular Design (2002) Perl daemon, pre-fork, NET::Server (2002 - 2003) by Marc Martinec Add internal Anti Spam methods (2006) Add p0f, pen-pals (2007) Add DKIM (2008)
Internal Process Chain
E-mail is processed in several stages

Decomposing E-Mail
Save resources! Decompose once, analyze multiple times Conguration settings $MAXLEVELS $MAXFILES $MIN_EXPANSION_QUOTA $MAX_EXPANSION_QUOTA
Adding Information
p0f results DKIM signature verication results XFORWARD data
Content Inspection
SpamAssassin is the default Anti-Spam tool Adding dspam is optional ClamAV is the default Anti-Virus tool Internal Methods
Inspect MIME types Check if sufx is black-/whitelisted
Content Classication
CLEAN SPAMMY SPAM VIRUS BANNED BADH OVERSIZED MTA
Actions
Store (bad) message in quarantine Archive any message that ows through the system Reinject message into SMTP transport Notify sender/recipient about the message status D_REJECT a message D_DISCARD a message D_BOUNCE a message D_PASS a message
DKIM Signing
... a domain-level authentication framework for email using public-key cryptography and key server technology to permit a signing domain to assert responsibility for a message, thus protecting message signer identity and the integrity of the messages ... Generate private key and public certicate Provide a per-domain/per-user list of DKIM signature options Add DKIM signatures to outgoing messages
Typical Mailserver
Everything takes place on one machine.

Typical Mailgateway
The gateway examines content. Follow-Up systems only do SMTP session control.
A dedicated Scanhost
Gateways do SMTP session control. Content control is done by a dedicated Content Inspection server.
Logging
syslog or native loglevel from silent to go buy new harddisks
Virus Scanners
primary scanners are/should be daemons secondary scanners are scripts They are called only if primary scanners are not available Both groups allow to dene a chain of scanners @av_scanners dening ClamAV access parameters [ClamAV-clamd, \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.ctl"], qr/\bOK$/, qr/\bFOUND$/, qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ]
Spam Scanners
amavisd-new uses Mail::SpamAssassin as default spam scanner dspam can be used as an alternative amavisd-new only congures/controls a few SpamAssassin parameters Additional SpamAssassin conguration makes SA more efcient Caution amavisd-new does not use spamc or spamd!
Quarantine
Quarantine destinations are send message to a mailbox store message in a le store message in a SQL server
Notication
sender and recipients can be notied about a messages status notications can be enabled on a global, a per-domain or on a per-user basis transport method can be set separately from the regular $forward_method powerful macros and templates customize notication messages
Quarantine Management
amavisd-release releases messages from quarantine Messages can be sent as attachments amavisd-release is not enabled by default Enabling amavisd-release in amavisd.conf $interface_policy{SOCK} = AM.PDP; $policy_bank{AM.PDP} = {protocol=>AM.PDP}; $unix_socketname=/var/amavis/amavisd.sock;
Quarantine Notication
Releasing a message from Quarantine
p@mail:~$ sudo amavisd-release virus-MOnrPcyyy3Wo [sudo] password for p: 250 2.0.0 Ok: queued as A974B820007
Viewing a released message
Log analysis and statistics
Know your enemy! mailgraph overall picture at a glance amavisd-nanny gives you info about what the processes are doing amavisd-agent gives SNMP-style statistics amavisd-logwatch catch trends, document daily work manual inspection inspect manually
amavisd-nanny
mailgraph
mailgraph II
amavisd-agent
amavisd-logwatch
Individualization I
Problem amavisd-new carries lots of legacy code! amavisd-new started as a single user script multi user environment requires individual settings newer groups of parameters wrap around older groups like layers of onion skin Map Types Single setting Lists Regular Expression
Individualization II
Layers of onion skin # Check for viruses @bypass_virus_checks_maps = ( \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re ); # Check for spam @bypass_spam_checks_maps = ( \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re );
Maps
amavisd-new has a map for nearly everything ... Example
@local_domains_maps @mynetworks_maps @newvirus_admin_maps @banned_filename_maps @spam_quarantine_bysender_to_maps @spam_tag_level_maps @spam_tag2_level_maps @spam_tag3_level_maps @spam_kill_level_maps @spam_modifies_subj_maps @spam_subject_tag_maps @spam_subject_tag2_maps @spam_subject_tag3_maps @spam_dsn_cutoff_level_maps @spam_dsn_cutoff_level_bysender_maps @spam_crediblefrom_dsn_cutoff_level_maps @spam_crediblefrom_dsn_cutoff_level_bysender_maps @spam_quarantine_cutoff_level_maps @whitelist_sender_maps @blacklist_sender_maps @score_sender_maps @author_to_policy_bank_maps @signer_reputation_maps @message_size_limit_maps @debug_sender_maps @bypass_virus_checks_maps @bypass_spam_checks_maps @bypass_banned_checks_maps @bypass_header_checks_maps @viruses_that_fake_sender_maps @virus_name_to_spam_score_maps ...
Maps
Uses for maps domains create a list of domains amavisd is responsible for exclusion exclude someone from a check exception deliver mail even if it is banned
Maps: Domains
Postx and amavisd-new share knowledge @local_domains_maps = ( \@local_domains_acl, read_hash(/etc/postfix/virtual_mailbox_domains );
Maps: Exclusion
Exclude a user from delivery @bypass_spam_checks_maps = ( ["user\@example.com"] );
Maps: Domains
Postx and amavisd-new share knowledge @banned_files_lovers_maps = ( ["user\@example.com"], );
LDAP I
Benets Enterprise ready Single conguration backend Simplify user administration Fast read access
LDAP II
Enabling and conguring LDAP lookups in amavisd-new
# Enable LDAP $enable_ldap = 1; # LDAP access $default_ldap = { hostname => localhost, port => 389, version => 3, timeout => 120, tls => 0, base => ou=people,dc=example,dc=com, scope => sub, query_filter => (&(objectClass=amavisAccount)(mail=%m)), bind_dn => cn=amavis,ou=services,dc=example,dc=com, bind_password => secret, };
amavisd-new LDAP Schema I
amavisd-new LDAP Schema II

LDAP User Object using amavisAccount
dn: uniqueidentifier=patrick@example.com,ou=people,dc=example,dc=com objectClass: organizationalPerson objectClass: person objectClass: top objectClass: PostfixBookMailAccount objectClass: extensibleObject objectClass: amavisAccount amavisBannedAdmin: postmaster@example.com amavisLocal: TRUE cn: Patrick Koetter givenName: Koetter mail: patrick@example.com mailAlias: abuse@example.com mailAlias: postmaster@example.com mailEnabled: TRUE mailGidNumber: 2000 mailHomeDirectory: /srv/mail/example.com/patrick mailQuota: 10240 mailStorageDirectory: maildir:/srv/mail/example.com/patrick/Maildir mailUidNumber: 2000 sn: Patrick uniqueIdentifier: patrick@example.com userPassword:: ye6Xu9th==
Why use Policy Banks?
Run one instance of amavisd-new Offer different behaviours (policies) Use port and/or client IP to identify policy bank
How Policy Banks work
An Example Policy Bank I

A policy bank ALT
$policy_bank{ALT} = { log_level => 3, syslog_ident => alt-amavis, syslog_facility => LOCAL3, inet_acl => [qw( 10.0.1.14 )], final_spam_destiny => D_PASS, final_bad_header_destiny => D_PASS, forward_method => smtp:*:*, notify_method => smtp:[127.0.0.1]:10025, virus_admin_maps => "abuse\@alt\.example\.com", spam_lovers_maps => [ @spam_lovers_maps, [qw( abuse@example.com )] ], ... };
An Example Policy Bank II
Assign policy to port 10028 $interface_policy{10028} = ALT;
Beyond the standard stuff
Keeping SpamAssassin up to date
Spammers use SpamAssassin to optimize their spam Patterns start to get old the day you download them Use sa-update to update your SpamAssassin rules sa-update creates a new directory SpamAssassin rst loads the packaged rules and then the updated ones
Extending SpamAssassins rules
The standard ruleset has a narrow score corridor Rules Emporium adds more weight to scores Rules Emporium adds spezialized rules Use sa-update to install and update these rules
Extending ClamAV patterns
ClamAV is just a content inspection engine Additional rule patterns add additional functionality Use http://www.sanesecurity.com/clamav/ for Phishing and Scam Signatures ...
Pen-Pals
Question Is it spam if you start talking about sex and your friend replies? amavisd-new can track who started a discussion apply lower scores to replies in a discussion that had been started by you
p0f - passive operating system ngerprinting
p0f is a passive OS ngerprinting tool identies the OS (type, version, patch level, etc.) on machines that connect to your box (SYN mode)
A p0f ngerprint in a message
p0f-analyzer.pl
amavisd-new queries p0f via the p0f-analyzer.pl script passes the OS information as a header eld to SpamAssassin SpamAssassin rules apply scores depending on the OS detected Invocation of p0f # Query p0f-analyzer.pl $os_fingerprint_method = p0f:*:2345;
Some statistics collected from Marcs logs in February 2006: p0f OS guess ham : spam ----------------------------Windows-XP 0.7 % : 99.3 % Windows-2000 5.8 % : 94.2 % UNKNOWN 16.5 % : 83.5 % Linux 58.8 % : 41.2 % Unix 80.3 % : 19.7 % (Unix+Linux 66.5 % : 33.5 %) ham mail with score below 3 spam score above 6
Scores in SAs local.cf

describe __L_P0F_EXISTS header __L_P0F_EXISTS describe L_P0F_WXP header L_P0F_WXP score L_P0F_WXP describe L_P0F_W header L_P0F_W score L_P0F_W describe L_P0F_UNKN header L_P0F_UNKN score L_P0F_UNKN describe L_P0F_Unix header L_P0F_Unix score L_P0F_Unix A header field X-Amavis-OS-Fingerprint does exist exists:X-Amavis-OS-Fingerprint Remote system is truly a Windows XP, not Windows 2000 X-Amavis-OS-Fingerprint =~ /\AWindows XP(?![^(]*\b2000 SP)/m 2.3 Remote system is some Windows variant, except Win. XP X-Amavis-OS-Fingerprint =~ /\AWindows(?! XP)/m 1.3 P0f was unable to determine remote OS type X-Amavis-OS-Fingerprint =~ /\AUNKNOWN/m 0.8 Remote system is running Unix, not Linux X-Amavis-OS-Fingerprint =~ /\A((Free|Open|Net)BSD|Solaris| \ HP-UX|Tru64|AIX)/m -1.0
describe L_P0F_Unix Remote system is running Linux header L_P0F_Linux X-Amavis-OS-Fingerprint =~ /\ALinux/m score L_P0F_Linux -0.1
DKIM - Domainkeys Identied Mail

Goal Sign messages using a digital identity Establish a good sender reputation Use sender reputation to tell good from neutral or bad messages DKIM is a standard (RFC 4871). It describes a framework to cryptographically sign messages cryptographically verify messages
How DKIM works ...
How to apply DKIM signatures in amavisd-new ...

Version You need amavisd-new 2.6 or later to implement DKIM Cryptographic keys Create a pair of cryptographic keys:
# amavisd genrsa /path/to/keylocation/selector
DKIM conguration example

$enable_dkim_verification = 1; $enable_dkim_signing = 1; # signing domain selector dkim_key ( example.org, abc, );
private key
options
/var/db/dkim/a.key.pem
How DKIM works ...
How to verify DKIM signatures in SpamAssassin ...
Loading the plugin in v320.pre

loadplugin Mail::SpamAssassin::Plugin::DKIM
Setting DKIM rules in local.cf

score DKIM_VERIFIED -1.3 score DKIM_POLICY_TESTING 0 score USER_IN_DKIM_WHITELIST -4.0 whitelist_from_dkim *@state-of-mind.de state-of-mind.de whitelist_from_dkim *@intl.paypal.com paypal.com
Questions?
Ralf Hildebrandt ralf@python.org Patrick Ben Koetter patrick@python.org Slides http://postfix.state-of-mind.de/ amavisd.pdf

Amavisd

Caricato da

Informazioni sul documento

Descrizione originale:

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Amavisd

Caricato da

Copyright:

Formati disponibili

Stages of spam-prevention Integrating amavid-new into Postx The nature of content control amavisd-new Implementations Ba

Controlling mail message ow by content analysis

T-System Business Services & state of mind

Open Source Days 2008 Copenhagen, 04. October 2008

T-System Business Services & state of mind

Individualization Maps LDAP Policy Banks Policy Banks Example

T-System Business Services & state of mind

Scan before and after queue

e-mail can be scanned before and after queueing

Split anti spam measures

Measures are typically split

A day in the life of a mail server

More than 98% of mail is spam.

A not so good idea ...

SMTP Session Control reduces load

SMTP Session Control

T-System Business Services & state of mind

These keep your users from sending the worst nonsense.

T-System Business Services & state of mind

T-System Business Services & state of mind

cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/drop.cidr cidr:/etc/postfix/bogon_networks.cidr, cidr:/etc/postfix/wildcard_mx_records.cidr, cdb:/etc/postfix/mx_accepts_no_bounces cdb:/etc/postfix/client_checks

T-System Business Services & state of mind

T-System Business Services & state of mind

header_checks check headers only:

T-System Business Services & state of mind

mime_header_checks check MIME-headers only:

T-System Business Services & state of mind

body_checks checks everything thats not a header:

T-System Business Services & state of mind

T-System Business Services & state of mind

T-System Business Services & state of mind

T-System Business Services & state of mind

Where is the difference?

T-System Business Services & state of mind

The nature of content control

T-System Business Services & state of mind

T-System Business Services & state of mind

Unpack into sandbox Apply limits to size(s) Apply warnings to messages

T-System Business Services & state of mind

T-System Business Services & state of mind

Internal Process Chain

E-mail is processed in several stages

T-System Business Services & state of mind

p0f results DKIM signature verication results XFORWARD data

T-System Business Services & state of mind

T-System Business Services & state of mind

CLEAN SPAMMY SPAM VIRUS BANNED BADH OVERSIZED MTA

T-System Business Services & state of mind

T-System Business Services & state of mind

T-System Business Services & state of mind

Everything takes place on one machine.

syslog or native loglevel from silent to go buy new harddisks

T-System Business Services & state of mind

T-System Business Services & state of mind

T-System Business Services & state of mind

T-System Business Services & state of mind

T-System Business Services & state of mind

T-System Business Services & state of mind

Releasing a message from Quarantine

T-System Business Services & state of mind

Viewing a released message

T-System Business Services & state of mind