Configuring an Inbound and Outbound SMTP Relay

to Complement ISA 2004 Firewall Protection for

Exchange Servers

Configuring an Inbound and Outbound SMTP Relay to Complement

ISA 2004 Firewall Protection for Exchange Servers

By Thomas W Shinder M.D.

I’m a big proponent of the SMTP relay concept. A properly configured SMTP relay can protect your
Exchange Server by preventing untrusted SMTP servers on the Internet from directly
communicating with your Exchange server. An SMTP relay doesn’t require a significant amount of
system resources and you can install the IIS SMTP service without incurring the resource or
security overhead you would have if you installed the IIS W3SVC (World Wide Web service).

An SMTP relay is a machine that can accept incoming and outgoing SMTP messages and forward
them to their appropriate location. The machine is an SMTP relay because it is not the endpoint of
the SMTP message. You can think of the SMTP relay as an SMTP router. The SMTP relay routes
SMTP messages to the appropriate location. No messages are stored on the SMTP relay as it is not
authoritative for any SMTP mail domains.

Most people who implement an SMTP relay think of using it for incoming mail. The SMTP relay
machine accepts mail only for the domains you host. For example, if you host an e-mail domain
named maildomain.com, then the relay accepts messages for that domain and routes them to the
appropriate location. The appropriate location may be an Exchange Server on the internal network,
or it may be another SMTP relay that is responsible for spam whacking and virus checking. Mail
destined to domains you do not host is dropped by the SMTP relay. This helps to prevent spammers
from using your Exchange Server to relay spam to other mail domains on the Internet.
The SMTP relay can also handle also relay outbound mail from your organization. For example, if a
user on your network sends mail to microsoft.com, the Exchange Server on your internal network
must be able to forward that message to the mail server responsible for the microsoft.com mail.
The Exchange Server on the Internal network can forward the mail directly to the microsoft.com
SMTP server by resolving the MX domain name of the microsoft.com mail server, or the Exchange
Server can forward the SMTP message to your outbound SMTP relay and then the SMTP relay
becomes responsible for relaying the message to the microsoft.com SMTP server.

One of the major advantages of the outbound SMTP relay is that it can also be configured as a
message screener that can remove attachments. Many prevalent viruses and worms include
attachments and flood the Internet with bogus mail with viral payload. The outbound SMTP relay
can strip the attachments or block the messages from leaving your network. You can use a full
featured e-mail antivirus application like GFI’s MailSecurity or you can use the built-in ISA Server
2004 SMTP Message Screener.

The SMTP relay can be configured on a separate IIS 6.0 server or on the ISA Server 2004 firewall
machine itself. The advantage of putting the SMTP relay on a separate machine is that if the SMTP
relay machine is attacked by a DoS attack, only that machine is adversely affected. In contrast, if
the ISA Server 2004 firewall is subjected to an e-mail DoS attack (such as a DDoS spam attack),
there is the potential of bringing down the firewall by filling up the log files. However, even with
that possibility in mind, if you keep close eye on your available disk space using an SNMP
application (or any other method you choose), putting the SMTP relay on the ISA Server 2004
firewall is a viable alternative.

The challenge in this entire scheme is configuring the SMTP service on the SMTP relay machine to
support both inbound and outbound relay. There are several requirements for our SMTP relay:

• Inbound SMTP relay is allowed only for domains you host

• Outbound SMTP relay is allowed to any domain
• The machine must not be an open SMTP relay that can be leveraged by criminal
spammers
• Remote users can forward mail to domains you do not host by authenticating with the
SMTP relay machine
Inbound SMTP Relay is Allowed Only to Domains You Host

The SMTP relay machine should only accept mail for relay to domains you host. For example, if you
host the domain maildomain.com, the SMTP relay should only accept mail for that domain and
reject mail for any other domain. (At least, this should be the case for unauthenticated
connections). The reason for this is that if external SMTP servers and clients can send mail to other
domains through your SMTP relay, then those machines will be able to relay spam through your
SMTP server. You want to reject mail sent to the SMTP relay for domains you do not host.

This is a critical configuration. I’ve talked to a lot of ISA Server 2004 and Exchange administrator’s
who have been put on dreaded RBLs because they did not correctly configure their SMTP relays to
prevent unauthenticated relay. If you’ve been victimized by one or more of these RBLs, you know
that you might as well try to come out a winner in an IRS audit. For this reason I highly recommend
that you do not publish SMTP relays to the Internet until you have a good understanding on how to
configure the relay to prevent spammers from abusing it.

The IIS SMTP service allows you to create remote domains and then configure those remote
domains to relay mail to a server on your internal network. The SMTP relay is configured with
remote domains for the domains you host and relay mail to your Exchange Server or to another
SMTP relay on your internal network. The default IIS and Exchange SMTP server configuration
blocks mail relay while allowing relay for mail addressed to the remote domains you configure.

You can see that the IIS SMTP service blocks relay by default by viewing the Properties dialog box
of the SMTP server, as seen in the figure below. The option Only the list below is selected and
there are no entries in the Computers list on the Relay Restrictions dialog box. Notice that I
have unchecked the Allow all computers which successfully authenticate to relay,
regardless of the list above checkbox. I removed this option because I do not need to use this
SMTP relay for outbound relay for authenticated users on external networks. However, you may
wish to enable this feature for your remote users. I’ll talk more about this option later in this article.

The figure below shows a list of remote domains.

Each domain is configured to relay to a specific host on the internal network. An example for the
shinder.net domain appears in the figure below. Note that there is a checkmark in the All
incoming mail to be relayed to this domain checkbox. This must be enabled so that mail
addressed to users in this domain can be relayed. Otherwise, the mail would stop at the SMTP relay
machine.

Outbound SMTP Relay is Allowed to Any Domain

If you choose to take advantage of outbound SMTP relay, you need to configure the SMTP relay
computer to allow outbound relay to any to any mail domain. You must allow relay to any mail
domain because you cannot predict what domains your internal network users are going to send
mail to. You can configure the Exchange Server with a list of domains to which you do not want
users to send mail to if you want to create an outbound block list. But in general, the SMTP relay
machine should be able to send outbound mail to any domain.

One important function the outbound SMTP relay must carry out is resolving names of Internet mail
domains included in the outbound SMTP messages. For example, when a user sends an outbound
message to user@microsoft.com, the outbound SMTP relay must be able to resolve the
microsoft.com e-mail domain to the IP address of a server that can accept incoming mail for the
microsoft.com domain. This requires that the SMTP relay have outbound access to the DNS (UDP)
query and DNS (TCP) zone transfer protocols. The SMTP relay will use these protocols to query DNS
servers when resolving the MX domain name for the destination mail server.

You can test the SMTP relay’s ability to query an SMTP server by using the nslookup command. Just
do this:

1. Open a command prompt and enter nslookup and press ENTER.

2. At the nslookup command prompt, enter set type=MX and press ENTER.
3. Enter the domain name that you want to test. In this example we’ll test
microsoft.com. Enter microsoft.com. (including the trailing period) and press
ENTER.
4. You’ll see that Microsoft has a number of mail servers that accept mail for their
domain.
5. You can exit the nslookup prompt by entering exit at the prompt and pressing
ENTER.
The Machine Must Not Be an Open SMTP Relay for Spammers

As I mentioned earlier, it’s absolutely critical that you do not allow spammers to relay through your
SMTP relay machine. This is the most common reason for getting placed on an RBL. RBLs are pretty
bad, as they try to act like law enforcement agencies without doing the prerequisite research to
determine the details of the crime or criminal. RBL "censors" place SMTP servers on their lists
without checking with the mail administrator of the domain. It’s the classic paradox where the
criminal spammers aren’t affected while hapless and harried e-mail server administrators, who have
no criminal intent at all, are victimized by RBL perpetrators.

OK, so much for my rant regarding RBLs. The bottom line is that you must prevent your SMTP relay
from being used by spammers. The default configuration on Microsoft Exchange and Microsoft IIS
5.0 and 6.0 SMTP servers prevents relay from unauthenticated users. However, if you’re not
familiar with the configuration, you might check or uncheck something that could leave you open. If
you’re not sure if you’re running a properly configured SMTP relay, you can use a nice utility
provided by the good folks at www.zonedit.com to test your configuration.

For example, suppose I want to check if my SMTP relay to www.tacteam.net is configured as an

open relay. First, I go to http://www.zoneedit.com/smtp.html and enter the information you see in
the figure below. I’ve entered the name of the SMTP relay in the Enter the host name or IP
address of the e-mail server text box (this is a name that resolves to an IP address on the
external interface of the ISA Server 2004 firewall that publishes this SMTP relay). I enter a random
user name in the Enter From: e-mail address text box. It doesn’t matter what name you enter
here, unless you have configured the relay to block access from certain addresses or mail domains.
In the Enter To: e-mail address text box, I’ve entered victim@aol.com. Since I don’t host mail
for the aol.com domain, my SMTP relay should never accept mail for that domain. This is how we
test for open relay. If the SMTP server rejects the message, its configured correctly. If it accepts
the message, its misconfigured and you’ll end up on some dreaded RBL.

Click the Begin Test button. You should see output similar to what appears in the figure below.
Notice the line 550 5.7.1 Unable to relay for victim@aol.com. This indicates that the SMTP
relay machine is correctly configured and spammers will not be able to send mail to domains you do
not host through the SMTP relay machine.
You can see the SMTP log entries for this connection attempt if you have enabled logging on the
SMTP virtual server. The 550 code is the SMTP code for Requested action not taken: mailbox
unavailable. The reason for the unavailability is that relay to the aol.com is not allowed.

2004-05-24 14:45:29 64.21.143.22 edit.dnsvr.com SMTPSVC1 FILEWAY 192.168.1.21 0 HELO -

+edit.dnsvr.com 250 0 34 19 0 SMTP - - - -
2004-05-24 14:45:29 64.21.143.22 edit.dnsvr.com SMTPSVC1 FILEWAY 192.168.1.21 0 MAIL -
+FROM:<tshinder@isaserver.org> 250 0 47 34 0 SMTP - - - -
2004-05-24 14:45:29 64.21.143.22 edit.dnsvr.com SMTPSVC1 FILEWAY 192.168.1.21 0 RCPT -
+TO:<victim@aol.com> 550 0 46 24 0 SMTP - - - -
2004-05-24 14:45:29 64.21.143.22 edit.dnsvr.com SMTPSVC1 FILEWAY 192.168.1.21 0 QUIT -
edit.dnsvr.com 0 691 46 24 150 SMTP - - - -

One thing I see a often is people commenting that they’ve configured their SMTP servers to prevent
relay but they are still receiving spam. Correctly configuring the SMTP relay will not prevent spam
from being sent to domains you host. It only prevents spammers from using your SMTP relay to
send spam to mail domains that you do not host.

Allowing Relay for Authenticated Users

While it’s always good to block spammers from relaying to external domains through your SMTP
relay server, you may want to allow your users the ability to relay to external domains through that
same SMTP server. This can be accomplished by requiring your users to authenticate with the SMTP
server. Users can then send mail to the corporate domain, or to external domains after they
successfully authenticate.

The figure below shows what we can accomplish by allowing relay for authenticated users:

1. When a spammer tries to send spam from his spam server through your SMTP relay,
the relay attempt fails because the SMTP relay will not forward mail for domains you
do not host. In this example, the spammer is trying to relay spam to the aol.com
domain through the SMTP relay. The relay blocks the messages.
2. When a corporate user tries to send mail to a user at bob@aol.com, the user will
first authenticate with the SMTP relay via his SMTP client application’s configuration
interface. After the user authenticates, he sends an e-mail message to
bob@aol.com. The SMTP relay allows the mail to be relayed to the aol.com domain
because the user logged on successfully before attempting to send the mail.
3. Another user wants to send mail from his SMTP client application to a user at
ceo@maildomain.com. You host the mail domain maildomain.com on your
Exchange Server. The SMTP relay is configured with a remote domain that forwards
messages destined to the maildomain.com domain to the Exchange Server on the
 internal network. Note that this user does not need to authenticate with the SMTP
relay in order to send mail to the maildomain.com domain because you allow
anonymous inbound mail to be sent to domains that you host.
Configuring Inbound and Outbound Relay on an IIS 6.0 SMTP Server
with GFI MailEssentials

That’s all there is to it. Well, almost. As network and firewall administrators, you know that the
devil is always in the details. Next week (after I return from TechEd in San Diego) I’ll provide you
the step by step details on how to configure the inbound and outbound SMTP relay on a separate
IIS 6.0 machine. We will also go over the details of the ISA Server 2004 firewall configuration that
publishes the SMTP relay, allows the SMTP relay outbound access to the SMTP and DNS protocols,
and how to install and configure GFI’s MailEssentials to prevent spam from clogging the Exchange
Server.

The figure below shows the sample network that I’ll use when we go over the details.

The steps involved with getting this set up include:

• Installing the SMTP service on the SMTP relay

• Requesting a certificate for the SMTP service on the SMTP relay – this is used to allow
secure connections for authenticated users
• Configuring the SMTP service for inbound secure/authenticated and anonymous relay
• Configuring the SMTP service to resolve Internet mail domain names
• Configuring the Exchange Server to use the SMTP relay for outbound relay
• Configuring the ISA Server 2004 Firewall to allow the SMTP relay outbound access to the
DNS (TCP and UDP) protocols
• Configuring the ISA Server 2004 firewall to allow the SMTP relay outbound access to the
SMTP protocol
• Creating an SMTP Server Publishing Rule that allows inbound connections to the SMTP
relay on the Internal network
• Configuring Public DNS entries to support the SMTP Server Publishing Rule
• Configuring authoritative DNS server entries with your domain Registrar – this is only
required if you decide to host your own DNS server

I’ll see you next week for the detailed step by step. If you have any questions about the setup
before then, come on by the Security Cabana at TechEd 2004 and I’ll answer them there!

I hope you enjoyed this article and found something in it that you can apply to your own network.
If you have any questions on anything I discussed in this article, head on over to
http://forums.isaserver.org/ultimatebb.cgi?ubb=get_topic;f=23;t=000058 and post a message. I’ll
be informed of your post and will answer your questions ASAP. Thanks! –Tom

If you would like us to email you when Tom Shinder releases another article on ISAserver.org,
subscribe to our 'Real-Time Article Update' by clicking here. Please note that we do NOT sell or rent
the email addresses belonging to our subscribers; we respect your privacy

About Thomas Shinder

Dr. Thomas W. Shinder is an MCSE, MCP+I, and MCT. He has worked as a technology
trainer and consultant in the Dallas-Ft. Worth metro area, assisting in development
and implementation of IP-based communications strategies for major firms such as
Xerox, Lucent and FINA.

Click here for Thomas Shinder's section.

Check out these recent articles by Thomas Shinder

• Mar 07, 2006, ISA Firewall Quick Tip: Blocking MSN Messenger Access through the
ISA Firewall while Enabling Access to Some Users
• Feb 28, 2006, Using a Commercial Web Site Certificate to Publish Outlook Web Access
(OWA) Part 4
• Feb 21, 2006, Using a Commercial Web Site Certificate to Publish Outlook Web Access
(OWA) Part 3
• Feb 14, 2006, Using a Commercial Web Site Certificate to Publish Outlook Web Access
(OWA) Part 2
• Feb 13, 2006, Steve Moffat's ISAServer.bm Blog Site Now Online

Click here for more articles by Thomas Shinder.

Get new article updates in your Inbox

Get all the ISA Server articles, tutorials and guides delivered directly to your mailbox as and when
they are released on ISAserver.org. Choose between receiving instant updates with the Real-Time
Article Update, or a monthly summary with the Monthly Article Update below.

Monthly Newsletter

Monthly Article Update

Real-Time Article Update

Subscribe
Email address:*

* Please note that we do NOT sell or rent the email addresses belonging to our subscribers; we respect your
privacy!

Featured Links*
Monitor and control ISA Server users in real time with GFI WebMonitor. Free 30-day eval!
Tunnel Windows applications on remote desktops - eliminate the hassle of local application
deployment with 2X ApplicationServer for Windows Terminal Services.
Free website security scan! - Check if your website is hackable - with a FREE web vulnerability
audit by Acunetix!
Learn to find & fix SQL injection, Cross site scripting, Google hacking & other web
vulnerabilities at the Web site security center
VeriSign SSL Certificates enable secure sockets layer encryption via https for secure e-
commerce.
• Monitor and control ISA Server users in real time with GFI WebMonitor. Free 30-day eval!
• Tunnel Windows applications on remote desktops - eliminate the hassle of local application
deployment with 2X ApplicationServer for Windows Terminal Services.
• Free website security scan! - Check if your website is hackable - with a FREE web
vulnerability audit by Acunetix!
• Learn to find & fix SQL injection, Cross site scripting, Google hacking & other web
vulnerabilities at the Web site security center
• VeriSign SSL Certificates enable secure sockets layer encryption via https for secure e-
commerce.

Receive Real-Time & Monthly ISAserver.org Join our Email Discussion List!
article updates in your mailbox! Discuss your ISA Server issues
Enter your email below! with other ISA Server experts through
email.

Click here to join!

Articles | Authors | Blogs | Books | Certification | Discussion List | Events | FAQs | Gaming |
Hardware | Links | Message Boards | Newsletter | | Shinder Section | Software
About Us : Email Us : Product Submission Form : Advertising Information
ISAserver.org is in no way affiliated with Microsoft Corp. *Links are sponsored by advertisers.
Copyright © 2006 TechGenix Ltd. All rights reserved. Please read our online privacy statement.