A few days ago, my colleague Dan looked at how email headers can tell you a lot about where a message came from, the path it took to an inbox, and even whether it actually was sent by who it claims to be from.

But the information contained in headers really is just a start. There’s more that can be measured about the journey of an email—and how well it’s working at the job you’re giving it.

Understanding Product Email Performance

Let’s look at a typical SaaS application email. What’s the basic reason your app is sending an email? It’s because your user has done something, and you need them to take an action to complete the task.

Account signup is a common example:

  1. A user signs up for an account
  2. You need the user to confirm her or his email address to complete the signup

Sounds pretty straightforward, right? Certainly, making sure that process works is essential to keeping your user happy (and not churning). Whether you’re a product manager or on the developer and devops side, making sure emails like this are working is an important part of measuring and managing your product’s overall performance.

Behind the scenes, though, the path that email takes is not as simple as it might seem. Most teams building SaaS products don’t have visibility into what happens once an email like this account confirmation message is sent.

With this article, I’m beginning a series that looks at the life of an account confirmation email, the stages it passes through on its way to the user’s inbox, and how data can be used to understand its performance at each stage.

Generating and Submitting a Product Email

The first step in our message’s journey is the generation and initial submission of the message. Depending on your application, this could happen in one of several ways:

  • Your app generates a fully-formed email and then submits it via Simple Mail Transfer Protocol (SMTP) to an external mail server for transmission.
  • Your app generates the content of a message and uses a simple mail sending function with a local SMTP server on your application host.
  • Or you can use an API provided by a dedicated email delivery service (in shorthand, an email API). This approach is the most scalable, as it allows your app to submit just the relevant data like name and address, while the specialized service handles message generation and delivery.

In any case, the email message is submitted to a Message Transfer Agent (MTA)—i.e., a mail server—that queues the message and takes responsibility for its delivery to the next hop in your message’s journey.

Why Latency Is a Key Metric for Product Emails

Measuring performance at this stage focuses on two metrics, injection count and latency.

The injection count is how many messages you submitted (injected) to the mail server for a given period of time. By itself this count won’t mean much, but it’s a good baseline against which to measure other data.

On the other hand, latency is a critical metric; when you send messages such as signup confirmations, your user’s experience depends on the timely delivery of the message. That’s especially important for something like an account confirmation email—the longer the message is delayed, the lower the likelihood the user will complete the registration and use your product.

If your message assembly and submission system has latency, you may want to review your code to see if it can be made more efficient, in addition, using an email API service can go a long ways towards improving performance, as it offloads the message generation from your own systems.

Typically, an app will only need to gather the recipient name, email, and other small amounts of personalization data and submit it to the API. The service then takes care of assembling and personalizing the messages and queuing them for delivery.

How to Measure Email Delivery Latency

Another component of email latency is how long it takes for your mail server (MTA) to deliver the message to its next stop, typically the inbound mail server operated by your user’s ISP. (Gmail or iCloud are typical examples.)

At this stage, some factors are out of your control, but latency here is a good indication of the overall quality of your email infrastructure. It’s a function of your infrastructure’s technical architecture and performance as well as its ability to manage delivery to ISPs. Both of these areas are where a high-performing email delivery service typically exhibits a major advantage over in-house email systems.

If your systems are slow at processing messages, or at managing queues, or are simply overworked, you’ll see latency of a minute or more. Efficient systems should be operating with a latency of less than ten seconds on average. There will always be some variation, as is common with all complex systems, but it’s the long-term trends that should concern you.

When you see latency numbers that are consistently high, you need to look at the underlying infrastructure: are there too many systems in the middle of your sending? Even your message generation process could be at fault: pickup directory systems can be relatively straightforward to build, but they are often based on scheduled pickups, which can introduce a long wait for a message submitted just after the last pass of the reader.

Latency after the first attempt is an indication of your reputation with the ISPs. Often an ISP will reject your message on the first attempt as a form of greylisting, refusing the first attempt at delivery to see if your MTA makes a second attempt (many spam bots will try once and move on). This causes the message to be queued and deferred temporarily for a subsequent attempt. The longer it takes to deliver a multi-attempt message, the more likely your sending reputation needs improvement.

How to Measure Latency by Hand

If you don’t have analytics in a dashboard for latency data, you’ll have to gather it yourself. Setting up scheduled jobs that send messages through your infrastructure to dedicated mailboxes that you maintain on the major ISPs, then retrieve the messages and check the timestamps of the first Received header and the Date header to determine the latency (note in the example below the time zones must be accounted for when calculating latency).

Email latency is a very important part of product email performance, but it’s just the start. Stay tuned for our next installment: understanding message delivery and what happens when a product email is delayed or never arrives.