Mailing Lists

Why?

Even though mailing lists have been around since ARPANET, they remain a viable way for groups to communicate. Modern web applications contain dynamic user groups and should leverage email to facilitate communication within them.

Many web applications already send email notifications, and the best applications support replying to that email:

Github allows developers to discuss issues and pull-requests
Craigslist allows buyers and sellers to communicate anonymously
Tender (not Tinder) allows support to address customer issues
Basecamp allows all sorts of project discussion

If your web application sends emails, you should handle the reply button and often the best user-experience encourages its use.

The Implementation

Only the largest companies run email servers, while others send and receive emails using third-party services. We'll use ActionMailer to send emails and the griddler gem to receive emails through MailChimp's Mandrill service.

Model Design

Your application probably already separates users into groups, whether by admins, accounts, trial users, or long-term customers, but for this tutorial we'll use a more general design.

Users have memberships to groups. Group members generate discussions by sending messages. Five models total, here's the database schema:

# db/migrations/20150221052903_create_group_discussions.rb

create_table :users do |t|
  t.string :name
  t.string :email
  t.timestamps
end

create_table :groups do |t|
  t.string   :name
  t.string   :email
  t.datetime :digest_last_sent_at
  t.timestamps
end

create_table :memberships do |t|
  t.integer :user_id
  t.integer :group_id
  t.boolean :receives_every_message, default: false
  t.boolean :receives_digest,        default: false
  t.string  :token,                  null: false
  t.timestamps
end

create_table :discussions do |t|
  t.integer :group_id
  t.string  :email
  t.string  :subject
  t.timestamps
end

create_table :messages do |t|
  t.integer :discussion_id
  t.integer :from_id
  t.text    :content
  t.timestamps
end

User

A user has a name and an email address that can send messages to groups they have a membership in.

class User < ActiveRecord::Base
  has_many :memberships
  has_many :groups,        through: :memberships
  has_many :sent_messages, foreign_key: 'from_id'

  validates :name,  presence: true, format: {with: %r(^[\w\ ]+$)}
  validates :email, presence: true, uniqueness: true
end

Group

Groups have an email address and users that are allowed to create discussions by emailing it.

class Group < ActiveRecord::Base
  has_many :memberships, dependent: :destroy
  has_many :users,       through: :memberships
  has_many :discussions, dependent: :destroy
  has_many :messages,    through: :discussions

  validates :name,  presence: true, uniqueness: true
  validates :email, presence: true, uniqueness: true
end

Membership

Memberships give users access to groups and also indicate what emails the user would like to receive from the group.

class Membership < ActiveRecord::Base
  belongs_to :user
  belongs_to :group

  validates :user_id,  presence: true, uniqueness: {scope: :group_id, message: 'is already a member of this group'}
  validates :group_id, presence: true
  validates :token,    presence: true

  before_validation :generate_token, on: :create

  private

  # tokens uniquely identify a membership for
  # the purposes of unsubscribing through an email's link
  def generate_token
    loop do
      self.token = SecureRandom.hex(64)
      break if Membership.where(token: token).empty?
    end
  end
end

Discussion

A discussion is a collection of messages within a group. Users add messages to the discussion by emailing it.

class Discussion < ActiveRecord::Base
  belongs_to :group

  has_many :messages, dependent: :destroy

  validates :email,   presence: true
  validates :subject, presence: true

  before_validation :generate_unique_email, on: :create

  private

  # if the group's email is admins@your-domain.com,
  # its discussion emails are admins-:unique-hex:@your-domain.com
  def generate_unique_email
    loop do
      self.email = group.email.sub('@', "-#{SecureRandom.hex(32)}@")
      break if Discussion.where(email: email).empty?
    end if group
  end
end

Message

A message is text sent by a user inside a group discussion.

class Message < ActiveRecord::Base
  belongs_to :discussion
  belongs_to :from, class_name: 'User'

  validates :from_id, presence: true
  validates :content, presence: true
end

Griddler Setup

The griddler gem smooths the process of receiving emails from third-party services such as Mandrill, SendGrid, Mailgun, and Postmark. I prefer Mandrill's simple but powerful interface, and MailChimp being my neighbor in Atlanta doesn't hurt.

First, add griddler and griddler's mandrill adapter to your Gemfile and run bundle install

# Gemfile

gem 'griddler'
gem 'griddler-mandrill'

Next, add the routes Mandrill will use to communicate to your app through griddler.

# config/routes.rb

# verifies during initial setup
get '/mandrill', to: proc { [200, {}, ["OK"]] }

# indicates a single received email
post '/mandrill', to: 'griddler/emails#create'

Finally, configure griddler to send received emails to the background job queue.

# config/initializers/griddler.rb

class Griddler::EmailProcessor
  def initialize(email)
    @email = email
  end

  def process
    ReceiveEmailJob.perform_later({
      'from'    => @email.from,
      'to'      => @email.to,
      'subject' => @email.subject,
      'body'    => @email.raw_body,
    })
  end
end

Griddler.configure do |config|
  config.email_service = :mandrill
  config.processor_class = Griddler::EmailProcessor
end

For more information on griddler, please refer to thoughtbot's blog post and the github repository.

Background Jobs

Our mailing list logic lives in background jobs and is actually rather simple:

Ignore emails if the sender doesn't belong to the group
If addressed to a group, create a new discussion and the initial message
If addressed to a discussion, create a new message
If neither a group or discussion is found, ignore it
Forward created messages to others in the group

class ReceiveEmailJob < ActiveJob::Base
  queue_as :default

  def perform(email)
    @from = User.where(email: email['from']['email']).first
    return unless @from # unknown sender

    @subject = email['subject']
    @body = email['body']

    email['to'].each do |to|
      try_group(to['email']) || try_discussion(to['email'])
    end
  end

  private

  def try_group(email)
    group = Group.where(email: email).first
    return unless allow_messages_to?(group)

    discussion = group.discussions.new(subject: @subject)
    message = discussion.messages.new(from: @from, content: @body)

    forward(message) if discussion.save
  end

  def try_discussion(email)
    discussion = Discussion.where(email: email).first
    group = discussion.group if discussion
    return unless allow_messages_to?(group)

    message = discussion.messages.new(from: @from, content: @body)

    forward(message) if message.save
  end

  def allow_messages_to?(group)
    group && group.memberships.where(user_id: @from).any?
  end

  def forward(message)
    ForwardMessageJob.perform_now(message)
  end
end

class ForwardMessageJob < ActiveJob::Base
  queue_as :default

  def perform(message)
    @message = message

    memberships.each do |membership|
      # spawn a new job for each email in case any fail to send
      GroupsMailer.new_message(membership, message).deliver_later
    end
  end

  private

  def memberships
    @message.group.memberships.
      where(receives_every_message: true).
      where('user_id != ?', @message.from)
  end
end

Outgoing emails follow the Action Mailer Basics, though notice the "from name" is the sender but the "from email" is the discussion, ensuring replies are handled properly by our application.

# app/mailers/groups_mailer.rb
class GroupsMailer < ApplicationMailer
  def new_message(membership, message)
    @membership = member
    @message = message
    @discussion = @message.discussion
    @group = @discussion.group

    mail({
      to: @membership.user.email,
      from: %("#{@message.from.name}" <#{@discussion.email}>),
      subject: "[#{@group.name}] #{@discussion.subject}"
    })
  end
end

<!-- app/views/groups_mailer/new_message.html.erb -->
<%= simple_format @message.content %>

Mandrill Setup

Your top-level domain is probably already using an email service like gmail, so it's best to establish a subdomain like app.your-domain.com for sending and receiving emails programmatically. Follow Mandrill's documentation to setup your account:

The end result should be:

Sending domain app.your-domain.com marked DKIM valid and SPF valid
Inbound domain app.your-domain.com marked MX valid with a verified route *@app.your-domain.com with a webhook URL of https://your-app.com/mandrill

Your production environment is now setup to run mailing lists from *@app.your-domain.com

Rounding the Edges

A few more features before we have a proper, user-friendly mailing list:

Future Considerations

Now we've wrapped up the basic mailing list functionality, though there's plenty more to think about:

How are groups managed?
How do users enable the daily digest?
Should we support attachments?
Can users be deleted?
Can non-members view an archive?
Can users create messages inside the archive?
How can we ensure only Mandrill can post received emails?

All these questions can be approached using standard Ruby on Rails MVC, and none are especially difficult with the existing design.

Wrap-Up

Here's what we made:

Groups of users can have discussions via email
Daily digests are sent to members that don't want to see every message individually
Unsubscribe links allow members to stop receiving a group's emails
A web archive exposes a group's complete discussion history

Essentially we have the most important parts of Google Groups, but the real possibilities come to light when you think beyond generic groups and messages. You can send and receive emails in your existing application:

How are your users grouped?
How can email help these groups communicate?
What could have an email address?
What would be convenient for your users to post from their inbox?

Work the reply button into your application's workflow. no-reply@your-app.com is no longer a viable option. At the very least use please-reply@your-app.com and forward it to an intern.