CLI Mail with Postfix and Fetchmail
After a few attempts at getting this to work across a couple of different platforms, both on a VPS and on a residential account, I felt it prudent to write a wrap up review of the hurdles I went through that would hopefully provide guidance to others so they don’t have to go through the 20 web articles that only cover part of the overall process. For this article I’m just going to cover getting this done at home, even though it’s more complicated having to skate around all the blockages and BS some ISP’s put on residential accounts. If you’re one who isn’t behind a controlling, micromanaging, BS ISP then your settings might end up being a little different to get the process to fly but the details here will probably either work or get you really close.
First and foremost, everything that is getting done is completely legit from a security perspective. If you’re reading this tutorial with the mindset that you’re going to create your own mail server so you can hide yourself on the internet from responsibility for the purpose of cyber stalking or bullying then:
DON'T
Many people will wonder why bother with learning email from the command line when there are web clients and desktop clients to handle this work for them. Well, because not every environment offers web access to necessary resources no a graphical interface, yet messaging still needs to occur. People who spend a lot of time working in server environments that do not have graphical engines need to learn how to do email this way. It's also the way email started out for those who are kind of nostalgic about tech and want to play with how things were done "back in the day". Everyone has their own reason for doing things, it's not my place to judge as long as it's not causing harm to others.
It’s expected that Postfix is at least installed but not necessarily configured yet. This tutorial covers the necessary settings to use GMail as our transport. It is also recommended that a person have their own private domain that they can work with as an address since there are certain aspects to Postfix that will require a "personal" touch to fly. After all, this is an "actual" mail MTA/MDA service we are setting up even if it has to use a third party as a crutch to work. There are many aspects to this configuration that will be co-dependent on other factors but, when tied together it all works.
First off, a legit email address is necessary that can be used for authentication purposes. Because of the strict security policies now in place for email authentication including SSL and TLS, this needs to be something that can be verified. The easiest way to do this is just create a GMail account that will be used for authentication purposes. Other accounts will work, as long as it's a legit account. When I say legit, I mean an email account tied to a real domain or company, not some fabricated address with the intent for spamming or BS activities on the web, so for this tutorial we'll use:
transport@gmail.com (replace transport with whatever you picked out).
Remember that this account is just for authentication and pushing the message around. The account will also need to have pop enabled as it's that protocol that we will be working with to fetch our mail. The reason we 're going to use pop and not Imap is because we simply want to download the mails to the machine, we don't need the advanced message synchronizing that Imap provides.
Second, we are going to need another email account based on our domain. This username and domain are going to have to cross reference identification on our computer and Postfix installation. I'll explain how all this works as we progress, then it will make more sense so that I don't jump around throwing just a bunch of do this and do that out there. For this example we're going to use:
user@customdomain.com
Our "user" should be set to the same value as the user account on the computer where Fetchmail is going to dump emails to. If the computer user account and email account don't jive together, Google will throw a nice error when pop tries to download the emails that basically says "The username is BS and the account is bogus so your request to download emails is prohibited". They'll phrase it in a nicer way but, essentially that's what it means. Think of it as protection against some a-hole trying to download your emails without proper creds.
So for example, your account on your computer is:
john@computer:~$
Then the email account should be:
john@customdomain.com
Once these two email addresses are created through whatever services one uses, we can get to the meat of things which is the Postfix configuration settings. Besides explaining what I did to make this work, I'll also throw down the links I used for guidance. They're actually really good articles and I hope they come in useful.
Using Postfix in the way we are here only relies on a couple of files which really aren't that bad to set up. For starters on this part, we want to make sure we have our hosts file set up. Edit this file and add a line like this:
127.0.0.1 localhost
127.0.1.1 user <------- this should correlate to the user name of the domain account.
We also need to make sure Postfix can verify the user account from the aliases file which essentially cross references to mailboxes. Edit the aliases file to look similar to this:
# See man 5 aliases for format
postmaster: root
user: user
** Make sure to run the newaliases command after adding a new mapping
Now that those two things are out of the way, the ssl portion of the configuration needs to be done as GMail won't do anything for us without having ssl set up. Remember, everyting is under strict security so it's important to follow all the rules if anything is going to work.
Inside the Postfix directory, create a new file to hold the credentials of the email address that GMail will be using to verify that the email being sent from your computer is legit and uses the proper GMail protocols to use their mail networks. Call the file whatever you want but, the very first line of the file should read like this:
[smtp.gmail.com]:587 transport@gmail.com:password <-- \/
** this should be the password to the account we created to verify ourselves to Google **
Save this file and cut back the authority on it since it has private information in it.
$ chown root:root <path/to/file>
$ chmod 600 <path/to/file>
Once this is done, remember to run postmap on the file so that Postfix knows to read it as part of it's configuration
$ postmap <path/to/file>
Next thing to do is verfy that ssl is installed and useful (more than likely it is but, we'll check anyway). Run this next line from a prompt and there should be a return of a lot of checks on Google's end then a ready prompt. When it gets to that point, just [ctrl]+[c] to get out.
$ openssl s_client -connect pop.gmail.com:995
If the last line from the output doesn't look like below, you're going to have to chase down that issue before anything else will fly.
+OK Gpop ready for requests from ....
Once certs are ready to go and all the above files and prep has been established, we can proceed to configuring the Postfix main.cf file. Out of habit and best practices, fist make a backup of the original file you can fall back on in case all your settings get out of whack and you need to start over. Trust me, it happens to us all and it sucks to try and chase down an original file to know where the baseline is so just do it.
$ cp main.cf main_original.cf
Now lets take a look at this file and plug in the values that we need.
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_host_lookup = native <-- this tells postfix to get the data from hosts
mydomain = customdomain.com <-- your domain name
myorigin = $mydomain
myhostname = <whatever> <-- this is just the name of your computer
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = localhost.localdomain, localhost, customdomain.com <-- last item is necessary
relayhost = [smtp.gmail.com]:587 <-- this is needed for pop to work on Google
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/<password_file>
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_use_tls = yes
After all this has been set then make sure to reset Postfix by running:
$ postfix reload
Test everything by sending some emails between various accounts or have some friends help you out. At this point, you should be able to send to anywhere then receive via a web interface to the transport@gmail.com account or any dektop client for checking mail. If any errors are thrown, Google search them and also be sure to run tail -f /var/logs/mail.log to check and see what kinds of errors show up in there. Use keywords from the logs to do various Google searches.
Fetching mail can be a little fussy but fairly straightforward as long as everything matches and Google doesn't detect anything that seems like BS activity. After installing Fetchmail, create the fetchmailrc file into your home folder:
$ vim .fetchmailrc
Once this is created, open, and ready for editing, add the following lines to it as instructions to Google pop engine to check the transport account for any mail that might be in the inbox and fetch them down to the specified user account for reading from your CLI.
set postmaster "user" <-- specifies the user account to look for on your computer
set daemon 600 <-- the number of seconds to check for mail, don't do more than 5 min intervals
poll pop.gmail.com proto POP3
user 'transport@gmail.com' <-- this specifies the address we set as our transporting address
there with password '<password>' <-- the password to the transport account
is "user" here <-- looks for the username on the computer
options <-- security options
ssl
sslcertck
sslcertpath /etc/ssl/certs
Save the file after these setting and change the permissions. Run the following command as a test and it should download whatever is in the inbox on the email account as well as stuff that is already read.
$ fetchmail -d0 -vk pop.gmail.com
Check your mail log file for any errors and if any settings are wrong then Google will send a reply email with hints about what needs to get fixed for Fetchmail to work. If it works, it will instantly dump everything into Postfix then depending on what reader you have installed you will see all the emails listed there. Once you have it working, I suggest two things:
Make a copy of main.cf like: $ cp main.cf main_it_works.cf
Make a copy of .fetchmailrc like: $ cp .fetchmailrc .fetchmail_working_config
This way, if anything ever gets hosed you can simply copy the backup into a working condition.
The final stage is to activate Fetchmail to run as a regular cron and check your account every X seconds you specified so run:
$ fetchmail
That is pretty much it. I tried to keep things as generic as possible since I have no idea what kind of system this might be executed on and some configuration or commands can be specific. Getting Postfix running on a VPS is actually easier as Fetchmail isn't required and the third party stuff can be eliminated since dancing around BS ISP restrictions isn't necessary. The only catch can be making sure reverse DNS is established and working for the domain as well as changing the IP addresses in the regisrar DNS to point to the new mail server. For the type of install we did here though, it's a lot of work for something trivial like email but, the experience is worth it and if you enjoy the nostalgia of CLI mail as well as learning how to do configuration, give it a whirl and when it works just sit back and smile at your accomplishment.
References:
http://www.stevejenkins.com/blog/2013/06/howto-get-around-comcast-port-25-block-with-a-postfix-server/
http://www.axllent.org/docs/view/gmail-pop3-with-fetchmail/
http://www.postfix.org/BASIC_CONFIGURATION_README.html#myorigin
https://support.google.com/mail/troubleshooter/1668960?hl=en
