All About LAMP Create web applications with Linux, Apache, MySQL, PHP and other open source technologies

Overview

Web servers with web applications like forums or blogs often need to send out emails to users. These emails are often legitimate and useful to users, however email providers often incorrectly classify them as spam because of the amount of spam emails found in the internet today.

This article describes what you can do, as the system administor / post master of your mail servers, to make sure emails sent from your mail servers go straight to the inboxes of the recipients instead of landing in their spam/junk folder. Because of the bad guys, good guys have to do a lot more to prove their innocence.

Context

Solution described in this article has been tested in a Fedora / Postfix setup. Most of the steps are not specific to this setting and can be applied on other environments. Note that these are only possible actions one can *attempt* to prevent emails to be marked as spam. It is ultimately up to the recipient mail servers to decide whether emails are spam or not.

This article assumes reader to have a good understanding of DNS, Unix and web server-related terms and how they work in high level. Most changes described in this article require system-administration / root access to the server.

Steps

1. Make sure Sender Username and Server Look Legitimate

If you just configured your server to send email, chances are that your out-going emails have words like “apache”, “localhost”, “nobody”, “localdomain” all over the place in their mail headers. This is often a sign of improper setup and therefore such mails will be marked as spam by spam filters.

To get rid of them, check the following configurations:

• /etc/hosts This file needs to be set up properly so that the “hostname” command returns the public hostname instead of “localhost”, and “hostname -f” returns the fully-qualified domain name, like “foo.fooworld.com”.
• /etc/httpd/conf/httpd.conf Check the ServerAdmin and SererName of this Apache config file and ensure that it is a valid email address and a fully-qualified domain name.
• /etc/mail/trusted-users If you need to send emails under a privileged user like apache as some other users such as no-reply@yourdomain.com, modify this file and add the username of the privileged user.
• /etc/php.ini If you are using php to send emails, you can enforce a different sender identify by adding -f’sender_address@domain.com’ in the sendmail_path argument.
• Use postfix instead of sendmail if you are sending emails from apache and want to ensure that the word “apache” is not in the headers of your out-going emails. This is required because in sendmail, even if a “-f” option is used to enforce a particular sender identity, the text “apache@localhost” will still appear in the first Receive mail header.
• The mail header fields “From”, “Reply-To” and “Return-Path” should be the same and be a valid email address, and will not reply with a bounce message if an email is sent to it.

2. Setup Reverse DNS

The usual DNS records allow everyone to lookup “foo.foobar.com” and find “123.124.125.126″. Reverse DNS does the opposite: It allows you to look up the IP address using the domain name. If the domain name and IP address can be used to look up each other, the sender is more likly to be what it claims it is. Therefore this check is employed by spam filter as well.

Note that Reverse DNS setting is not controlled by the domain name server of the domain in question. Using the example above, the DNS server of “foo.foobar.com” cannot control its Reverse DNS settings. It is the responsibility of the ISP that maintains 123.124.125.126 to set up its Reverse DNS mapping.

To verify that it has been set up, use “nslookup <domain name>” to check for its IP and “nslookup <ip address>” to check for its domain name. They should find each other if DNS and Reverse DNS have been set properly.

3. Add SPF Record

SPF record is a TXT record in the domain server, like the other A / CNAME / MX type DNS records, that describes the domain. Spam filter gets the SPF record from DNS server and check if the sender mail server has been allowed or disallowed to send mails.

SPF implementation is free and simple. You can create a SPF record manually if you understand its specification, or you can go to one of the following websites to generate it after answering some questions about your mail server settings.

http://old.openspf.org/wizard.html
http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/default.aspx

4. IP Blacklist Check

Some of the bad guys might have used *your* IP address before to send out spam emails, and now any emails sent from your machine are classified as spam. A website like the one below lets you check if your machine falls into such category, and if it does, lets you request that your IP be taken out from the list.

http://www.mxtoolbox.com/blacklists.aspx

5. Ensure Mail-Sending Server is listed on MX record

The MX record in the DNS entries tells the whole world which machine should process emails for that domain. If the web server that sends outgoing emails is not listed on one of the MX records, spam filters may think that mails were sent without proper authorisation and therefore should be marked as spam.

As an example, let say you are sending emails from foo.foobar.com. A lookup of MX record of foobar.com shows that the mail server is mail.foobar.com instead. This could be a problem for spam-filter. Configure foo.foobar.com as a proper mail server, point to it using MX record and use it to send and receive emails for foobar.com.

6. Implement DKIM (Domain Keys Identified Mail)

DKIM is similar to SPF. The sender claims that it is from foobar.com and the recipient checks the DNS records of foobar.com to see if the sender is indeed from there. The difference is that, for DKIM, sender has to present a “signature” instead of just using its IP to prove its identity.

Refer to this article for implementation details of DKIM.

Gmail in particular appear to put emails in inbox only after DKIM has been implemented.

7. Contact Individual Providers

The recipient mail servers may still decide that your emails are spam, even after you have done the whole lot above.

For example, official support from Microsoft stated that “… IPs that have little to no history of sending email to Hotmail are more likely to be targeted by SmartScreen…”. SmartScreen is one of their spam-filtering tools that filter emails based on machine learning algorithms. With such assumption made by this program, emails from new servers will inevitably go to the junk folder of hotmail addresses.

Advise from official Microsoft Support for this issue is that “Following all the recommendations (from their postmaster policy guidelines)… will really make a big difference. Have a nice day!”. There is not much you can do. However, your out-going emails may become non-junk after a few weeks of SmartScreen learning about your server IP, given that your emails are not labelled as junk by your recipients and emails are sent consistently.

As a last resort, you can try to contact the free email providers directly using the web pages they provide:

• http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html Yahoo Mail support page for marked-as-spam problem
• https://support.msn.com/eform.aspx?productKey=edfsmsbl&ct=eformts Microsoft support page for marked-as-spam problem

Errors / Configuration Problems

Mail source from Gmail shows “Received-SPF: neutral”

This is a sign that your SPF record has not been implemented properly. If it has been done correctly, the header should say “Received-SPF: pass” instead. Have you specified the correct IP? Have the DNS records been updated properly?

Mail source from Gmail shows “dkim=neutral (body hash did not verify)”

A work-around for this problem is provided in http://allaboutlamp.com/2009/09/setup-dkim-for-postfix-in-fedora-using-dkim-milter/.

Reference

• http://dnsstuff.com A website that generates comprehensive report for your DNS records and tells you how “healthy” they are. Make use of the 7-day trial period!
• http://www.mxtoolbox.com/blacklists.aspx A website to check if the IP address of your mail-sending machine has been blacklisted by spam filters.
• http://postmaster.live.com/Troubleshooting.aspx Microsoft’s help for those trying to send emails to hotmail users but marked as spam.