Web servers with web applications like forums or blogs often need to send out emails to users. These emails are often legitimate and useful to users, however email providers often incorrectly classify them as spam because of the amount of spam emails found in the internet today.

This article describes what you can do, as the system administor / post master of your mail servers, to make sure emails sent from your mail servers go straight to the inboxes of the recipients instead of landing in their spam/junk folder. Because of the bad guys, good guys have to do a lot more to prove their innocence.

Context

Solution described in this article has been tested in a Fedora / Postfix setup. Most of the steps are not specific to this setting and can be applied on other environments. Note that these are only possible actions one can *attempt* to prevent emails to be marked as spam. It is ultimately up to the recipient mail servers to decide whether emails are spam or not.

This article assumes reader to have a good understanding of DNS, Unix and web server-related terms and how they work in high level. Most changes described in this article require system-administration / root access to the server.

Steps

1. Make sure Sender Username and Server Look Legitimate

If you just configured your server to send email, chances are that your out-going emails have words like “apache”, “localhost”, “nobody”, “localdomain” all over the place in their mail headers. This is often a sign of improper setup and therefore such mails will be marked as spam by spam filters.

To get rid of them, check the following configurations:

• /etc/hosts This file needs to be set up properly so that the “hostname” command returns the public hostname instead of “localhost”, and “hostname -f” returns the fully-qualified domain name, like “foo.fooworld.com”.
• /etc/httpd/conf/httpd.conf Check the ServerAdmin and SererName of this Apache config file and ensure that it is a valid email address and a fully-qualified domain name.
• /etc/mail/trusted-users If you need to send emails under a privileged user like apache as some other users such as no-reply@yourdomain.com, modify this file and add the username of the privileged user.
• /etc/php.ini If you are using php to send emails, you can enforce a different sender identify by adding -f’sender_address@domain.com’ in the sendmail_path argument.
• Use postfix instead of sendmail if you are sending emails from apache and want to ensure that the word “apache” is not in the headers of your out-going emails. This is required because in sendmail, even if a “-f” option is used to enforce a particular sender identity, the text “apache@localhost” will still appear in the first Receive mail header.
• The mail header fields “From”, “Reply-To” and “Return-Path” should be the same and be a valid email address, and will not reply with a bounce message if an email is sent to it.

2. Setup Reverse DNS

The usual DNS records allow everyone to lookup “foo.foobar.com” and find “123.124.125.126″. Reverse DNS does the opposite: It allows you to look up the IP address using the domain name. If the domain name and IP address can be used to look up each other, the sender is more likly to be what it claims it is. Therefore this check is employed by spam filter as well.

Note that Reverse DNS setting is not controlled by the domain name server of the domain in question. Using the example above, the DNS server of “foo.foobar.com” cannot control its Reverse DNS settings. It is the responsibility of the ISP that maintains 123.124.125.126 to set up its Reverse DNS mapping.

To verify that it has been set up, use “nslookup <domain name>” to check for its IP and “nslookup <ip address>” to check for its domain name. They should find each other if DNS and Reverse DNS have been set properly.

3. Add SPF Record

SPF record is a TXT record in the domain server, like the other A / CNAME / MX type DNS records, that describes the domain. Spam filter gets the SPF record from DNS server and check if the sender mail server has been allowed or disallowed to send mails.

SPF implementation is free and simple. You can create a SPF record manually if you understand its specification, or you can go to one of the following websites to generate it after answering some questions about your mail server settings.

http://old.openspf.org/wizard.html
http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/default.aspx

4. IP Blacklist Check

Some of the bad guys might have used *your* IP address before to send out spam emails, and now any emails sent from your machine are classified as spam. A website like the one below lets you check if your machine falls into such category, and if it does, lets you request that your IP be taken out from the list.

http://www.mxtoolbox.com/blacklists.aspx

5. Ensure Mail-Sending Server is listed on MX record

The MX record in the DNS entries tells the whole world which machine should process emails for that domain. If the web server that sends outgoing emails is not listed on one of the MX records, spam filters may think that mails were sent without proper authorisation and therefore should be marked as spam.

As an example, let say you are sending emails from foo.foobar.com. A lookup of MX record of foobar.com shows that the mail server is mail.foobar.com instead. This could be a problem for spam-filter. Configure foo.foobar.com as a proper mail server, point to it using MX record and use it to send and receive emails for foobar.com.

6. Implement DKIM (Domain Keys Identified Mail)

DKIM is similar to SPF. The sender claims that it is from foobar.com and the recipient checks the DNS records of foobar.com to see if the sender is indeed from there. The difference is that, for DKIM, sender has to present a “signature” instead of just using its IP to prove its identity.

Refer to this article for implementation details of DKIM.

Gmail in particular appear to put emails in inbox only after DKIM has been implemented.

7. Contact Individual Providers

The recipient mail servers may still decide that your emails are spam, even after you have done the whole lot above.

For example, official support from Microsoft stated that “… IPs that have little to no history of sending email to Hotmail are more likely to be targeted by SmartScreen…”. SmartScreen is one of their spam-filtering tools that filter emails based on machine learning algorithms. With such assumption made by this program, emails from new servers will inevitably go to the junk folder of hotmail addresses.

Advise from official Microsoft Support for this issue is that “Following all the recommendations (from their postmaster policy guidelines)… will really make a big difference. Have a nice day!”. There is not much you can do. However, your out-going emails may become non-junk after a few weeks of SmartScreen learning about your server IP, given that your emails are not labelled as junk by your recipients and emails are sent consistently.

As a last resort, you can try to contact the free email providers directly using the web pages they provide:

• http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html Yahoo Mail support page for marked-as-spam problem
• https://support.msn.com/eform.aspx?productKey=edfsmsbl&ct=eformts Microsoft support page for marked-as-spam problem

Errors / Configuration Problems

Mail source from Gmail shows “Received-SPF: neutral”

This is a sign that your SPF record has not been implemented properly. If it has been done correctly, the header should say “Received-SPF: pass” instead. Have you specified the correct IP? Have the DNS records been updated properly?

Mail source from Gmail shows “dkim=neutral (body hash did not verify)”

A work-around for this problem is provided in http://allaboutlamp.com/2009/09/setup-dkim-for-postfix-in-fedora-using-dkim-milter/.

Reference

• http://dnsstuff.com A website that generates comprehensive report for your DNS records and tells you how “healthy” they are. Make use of the 7-day trial period!
• http://www.mxtoolbox.com/blacklists.aspx A website to check if the IP address of your mail-sending machine has been blacklisted by spam filters.
• http://postmaster.live.com/Troubleshooting.aspx Microsoft’s help for those trying to send emails to hotmail users but marked as spam.

DKIM is a technology that allows mail senders to attach a signature in outgoing mails, so that recipients can check the signatures against DNS records of the sender to see if mail is indeed sent from there.

Because free email providers like gmail make use of DKIM to determine if sender is sending spam, i.e. “if your domain does not implement DKIM, you are a spammer”, it is important to implement DKIM for your mail servers to avoid your legitimate out-going emails be classified as spam.

Context

This article describes how to implement DKIM for Postfix in Fedora using the dkim-milter open-source module. This solution has been tested with Fedora 10.

DKIM allows a domain to be associated with multiple “signatures”. Each signature is identified by its “selector”. In this example, we are going to create only one signature with its selector named “default”.

Steps

1. Generate a private key

openssl genrsa -out default.private 1024

A “default.private” key file will be generated. It will be moved to a specific location later.

2. Generate a public key for this private key

openssl rsa -in default.private -pubout -out default.public -outform PEM

A file with filename “default.public” will be generated with content like

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDVqyBW3CvurzAJrWvw/rbiMVL2
6lytBkhIrgEBWjGWEjjhM6mQpQWLq9VR46xlL4OT6UdVtO8QOMEVI23LN0fwtrPc
/auwHC2U9joUWTWVjOMZWEywOHwATGevh9TApt2hQJkWjMy/xmCIqBs9VZIweRlf
VFqc9WEu6VamGe9C3QIDAQAB
-----END PUBLIC KEY----

It will be used to create a DNS TXT record. See next step.

3. Create a DNS record of type TXT

Modify DNS records and add a record of type TXT:

TXT record name
default._domainkey

TXT record value
v=DKIM1; g=*; k=rsa; p=<content of default.public>

Note that the prefix “—–BEGIN PUBLIC KEY—–” and suffix “—–END PUBLIC KEY—-” should not be put in the TXT record value.

This DNS record will be retrieved by mail receivers who want to verify emails with DKIM signatures. The record name “default._domainkey” tells verifier that the “selector” of this signature is “default”, therefore if you are changing selector name to something else, make sure you change all of them consistently.

4. Install dkim-milter in Fedora

Run the following as root to install the dkim-milter pacakge.

yum install dkim-milter

5. Enable dkim-milter to run on start-up

Make sure dkim-milter service will run on start-up by running this command:

chkconfig --level 3 dkim-milter on

Note that your server may use a different “runlevel”. You can check “/etc/inittab” to see which run level you are on.

6. Move private key to appropriate location

As root, copy the private key to the location specified by the “keylist” (refer to next step) and make sure it is readable by dkim-milter:

mkdir /etc/dkim-milter/
mv default.private /etc/dkim-milter/default
chown dkim-milter.dkim-milter /etc/dkim-milter/default

Make sure the filename of private key file matches the “selector” name specified in the DNS record.

7. Add an entry to the keylist for dkim-milter to read

Add the following line to /etc/mail/dkim-milter/keys/keylist. Replace <domain.com> with your domain name.

*:<domain.com>:/etc/dkim-milter/default

8. Configure postfix to use dkim-milter

Add the following lines to /etc/postfix/main.cf to ask postfix to use dkim-milter.

smtpd_milters = unix:/var/run/dkim-milter/dkim-milter.sock
non_smtpd_milters = unix:/var/run/dkim-milter/dkim-milter.sock

9. Start dkim-milter and restart postfix

Start dkim-milter service and restart postfix using the following commands. Or restart the server.

service dkim-milter start
service postfix restart

10. Change file permissions of the Mail Filter Socket file

Change file permissions of the “Mail Filter Socket” file and its parent directory to allow postfix to write to it:

chmod 755 /var/run/dkim-milter
chmod 777 /var/run/dkim-milter/dkim-milter.sock

Changing dkim-milter.sock permission unfortunately is required EVERYTIME after dkim-milter servuce is restarted. This is because dkim-milter resets the file to mode 755 that postfix cannot read.

For Fedora, one way to have this done automatically is to add the chmod command in /etc/rc.d/rc.local, so that it will be run on start-up everytime.

Errors / Configuration Problems

Gmail header says “dkim=neutral (body hash did not verify)”

If you are sending emails using mail() function in php, these out-going emails will not be verified by Gmail, for some unknown reason. Opening the mail source in Gmail shows that there is a line “dkim=neutral (body hash did not verify)”, hinting that the key in DKIM signature does not match the public key from DNS. Only Gmail knows why.

You may also notice that sending emails directly from the command-line in your server will be DKIM-verified by Gmail, resulting in a “dkim=pass” in the mail source headers. Therefore, a work-around for this problem is to modify your php scripts so that it will do a system call to a shell script, which will indirectly send your email out. This way, Gmail will verify your email and put it to the inbox of recipient instead of marking it as spam.

Gmail header says “dkim=neutral (no key)”

It means that gmail couldn’t find a matching key to verify your signature.

So, how does gmail (or any other dkim verifier) find a matching key? It relies on the “selector name” from the email header. In the email header there should be something like “DKIM-Signature: … s=some_selector_name; t=1254783208;…”, here the “some_selector_name” is the selector name. Dkim-milter specified that as your selector name as instructed by /etc/mail/dkim-milter/keys/keylist file.

Now if you look at the DNS record for your domain, it needs to have a DKIM public key record (TXT type DNS record) with the name “some_selector_name._domainkey” and value being the matching public key. If the name of the record is not “some_selector_name._domainkey”, the verifier will not be able to use the public key, because the selector name does not match.

You might have followed an example from internet to set your selector name to “domain.com_default.key.pem” and the DNS record name to “default._domainkey”, which are not matching each other and therefore getting this no-key message.

Reference

• http://www.howtoforge.com/postfix-dkim-with-dkim-milter-centos5.1 Instructions similar to this article, but for CentOS instead of Fedora.
• http://testing.dkim.org/reflector.html Free testing to check if your DKIM implementation works.
• Config files related to dkim-milter in Fedora:

  Socket file that Postfix connects to
  /var/run/dkim-milter/dkim-milter.sock

  Key List file
  /etc/mail/dkim-milter/keys/keylist

  Actual private key (can be anywhere as specified by “Key List”)
  /etc/dkim-milter/<key filename>

  Config file
  /etc/mail/dkim-milter/dkim-filter.conf

  Another config file but does not appear to be read by the service
  /etc/sysconfig/dkim-milter

  Postfix config that specifies where dkim-milter socket is:
  /etc/postfix/main.cf