from Hacker News

A serverless email server on AWS using S3 and SES

by mzehrer on 1/4/20, 9:55 AM with 260 comments

• by arno1 on 1/4/20, 12:03 PM

  > This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages.

  Interesting approach, though I solved this frustration by the use of a Docker and kept "my data is mine" + "no vendor lock-in" + "I control all the gears" approach. (Though, it's not perfect since VPS is ran by "someone" else.. but that place where you run this stack can be easily changed at your convenience). Simple docker-compose.yml with 3 images and voila.

  This AWS S3 SES setup looks far more complex than what I did using only 3 docker images: the postfix (for smtp), dovecot (for imap), opendkim (for email sigining & verification). It's really easy to fire-up a VPS with a single click nowadays.

  If someone is interested in the images I am using:

  - https://git.nixaid.com/arno/postfix

  - https://git.nixaid.com/arno/dovecot

  - https://git.nixaid.com/arno/opendkim

  Then you just feed the images with the right configs (main.cf, master.cf, .., dovecot.conf, opendkim.conf).

  It's also possible to template the configs and make the variable-based configs. Make things scale friendly. I am also using Terraform to automate the server deployment/DNS record updates so it is easy to get from 0 to 100.

  The only drawback is that you are the one to maintain the OS/SW upgrades, security, etc.. but that's something I really want to do by myself instead of relying on someone else :-)

• by giu on 1/4/20, 4:07 PM

  Just a friendly reminder, since I've worked with SES in the past: Don't forget about bounces when using SES [0].

  From [0]:

  > If your bounce rate is 5% or greater, we'll place your account under review.

  To sum it up, try to keep track of bounced e-mails by using the SES notifications [1].

  [0] https://docs.aws.amazon.com/ses/latest/DeveloperGuide/e-faq....

  [1] https://docs.aws.amazon.com/ses/latest/DeveloperGuide/monito...

• by abjKT26nO8 on 1/4/20, 11:34 AM

  I don't know anything about serverless --- to this day I fail to understand what this word is even supposed to mean. And the deployment diagram[1] sure looks complicated to me. I think I prefer old-school servers.

  [1]: <https://raw.githubusercontent.com/0x4447/0x4447-product-s3-e...

• by primitivesuave on 1/4/20, 11:10 AM

  Thanks for putting this together and documenting it so well. I’ve had to build this solution twice now, and far less elegantly.

  The S3 PUT charges caught me off guard the first time (receiving lots of marketing/spam email will cost $1/1000 emails). I ended up putting small emails up to 400 kB in dynamoDB and only using S3 for large emails and attachments, which could be a means of cost reduction in this solution as well.

• by Cyph0n on 1/4/20, 4:23 PM

  I’ve been working on a small side project that involves processing incoming email. In particular, it’s an app that needs to do something for each email it receives from (hopefully paying!) users.

  I am not interested in storing user mail, so SES is just too costly, at least according to a quick worst-case calculation.

  That leaves me with two options:

  1. Self-hosted Postfix

  2. Mail service like Mailgun

  With (1), there is no need to worry about overages, but scaling the mail server might be challenging.

  The advantage of (2) over SES is that you are only charged a flat fee for each email, regardless of size. Emails are then automatically deleted after some period of time. Scaling up and down is easy.

  For now, I am using Mailgun, but I am writing the mail processing daemon in a way that will make it easy to transition to Postfix, if needed.

  Also, I decided to write the mail processing backend in Rust, so I’ve been learning the language as I go!

• by coding123 on 1/4/20, 6:37 PM

  Seems like a crazy amount of architecture. Does AWS keep all this stuff organized in some way, or will my personal experiments in Lamba accidentally break this because it's all merged together?

  Say I've installed this.

  I now want to write my own lamba service to handle contact form POSTs or something. Then I decide to delete it, but I accidentally delete one of these crazy email things. What happens?

• by theqult on 1/5/20, 10:35 AM

  >his day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages.

  https://mailinabox.email/

• by anonu on 1/4/20, 12:56 PM

  Now someone just needs to create a serverless (aka client side or browser only) Gmail like interface you can host on S3.

  And the shackles will be broken...

• by insomniacity on 1/4/20, 1:32 PM

  I'm missing something - how are people reading this in an email client if it doesn't have IMAP support?

• by bsder on 1/5/20, 2:38 AM

  Please don't use '+' for special purposes in email addresses without making it changeable (I recommend '_' instead).

  Yes, it is "nominally" accepted--in reality there are too many website that "validate" email addresses and barf on '+'.

• by newman8r on 1/4/20, 8:03 PM

  If you only need to read the emails from S3, take a look at this project https://github.com/mewa/s3abird

• by lowdose on 1/4/20, 11:17 AM

  Yes this is legit! Got it to work on a throwaway domain under 30 minutes.

• by whatsmyusername on 1/4/20, 4:38 PM

  You can get receive at any address on a domain forwarded to whatever email you want if you're registered with Monicker.

  I imagine most other registrars offer the same thing.

• by glandium on 1/5/20, 4:34 AM

  Note that SES only retries delivery for a fixed 840 minutes (not configurable), which is an annoyingly too short amount of time.

• by Boulth on 1/4/20, 12:11 PM

  Wow, this is cool! JSON structures resemble JMAP. I wonder what'd be the effort to add JMAP endpoint to this?

• by coder1977 on 1/4/20, 11:54 AM

  How does this compare to setting up something like sendy (sendy.co) on lightsail and connecting to SES?

• by raoulbhatia on 1/4/20, 11:15 PM

  ... and I thought about a similar thing (Email server using Lambdas) just yesterday ... ^^

• by z3t4 on 1/4/20, 11:27 AM

  What about email signing?

• by eivindga on 1/5/20, 12:09 AM

  Cool project! I'll make sure to test it someday.

  Thanks for sharing!