How we improved email deliverability from 43.7% spam to 1.4% after a spammer exploited our SaaS

Last month spammers found a hole in our app that let them send spam emails. Our customers use Savio to track feature requests. When those requests get deployed, our customers send a "close the loop" email so requesters know the feature they wanted is now live.

Note: Savio helps B2B SaaS Customer Success, Product, and Sales teams organize and prioritize product feedback and feature requests. Learn more about Savio here.

Spammers exploited this feature to send about 20,000 unsolicited emails. Fortunately there was a bit of friction if you're trying to use Savio to spam; it wasn't as easy as pasting in a list of email addresses and hitting send so the spammers weren't able to do major damage. Plus, we noticed the problem within an hour or so and were able to close the exploit quickly.

Here's our daily email sent volume (the attack happened on August 27):

We only had 4 spam complaints. But we also had 3744 bounces:

Doesn't sound terrible, but it had some bad downstream implications for our email deliverability...

Problem: Our Customers' "Close the Loop" Emails were being marked as spam

We ran a few tests to see if email deliverability was impacted and didn't notice anything bad. But we got an email from a customer who was seeing her "Close the Loop" emails end up in spam:

So it was time to dig in and see what was going on.

Diagnosing our deliverability problem

There were two things we did to understand what might be causing the deliverability problem:

1. We verified we had legit SPF, DMARC, and DKIM DNS records

2. We used GlockApps to test deliverability (thanks to my pal @Collin for the tip)

1. Verifying DNS records

We used MailTester to verify our SPF and DKIM records, and MXToolbox's DMarc Tool to verify we had properly added our deliverability-related DNS records. These records help email services like Gmail know whether emails sent from your domain are legit or not. All came back good, so we didn't have to mess around with those.

2. Run deliverability test with GlockApps

I'd never heard of GlockApps before Collin suggested trying them. It's a fantastic service. It works by:

1. Giving you a bunch of email addresses at different providers to send your email to.

2. Analyzing whether those emails end up in the inbox or spam folder

3. Showing you the results

4. Giving you steps to take to fix things

On the first test we ran with Glock, we sent our "Close the Loop" email to 73 GlockApps-provided email addresses at Gmail, Yahoo, Hotmail, Fastmail, GSuite, AOL, and a few other providers.

Here were the high-level results:

43.7% of our emails ended up in the spam folder - NOT good.

When we looked at the heuristics Glock ran, things didn't look too bad...

... but even so, all the emails we sent to AOL, Gmail, GSuite, and Yahoo addresses ended up in spam 😱

Yahoo:

Gmail and GSuite:

Aol:

Yikes.

Fixing our deliverability problem

Luckily Glock gives you some ideas about how to improve deliverability. Specifically:

• They show you if you're on any email blacklists

• They analyze your email's content and tell you if there are issues you can fix

• They provide a list of action steps you can take to improve deliverability

• They also provided two articles - one of which included advice that moved the dial most for us!

1. Getting off email Blacklists

Leadfuze says:

An email Blacklist is a real-time database that uses criteria to determine if an IP is sending email it considers to be SPAM. There are several blacklists… Each list [has] a unique way of accepting inbound mail and determining if email is considered SPAM. They can all impact deliverability for your emails.

We had ended up on two blacklists: SORBS and JustSpam. It's not entirely clear whether our spammer put us on either list. That's because:

1. We are using a shared Mailgun server to send email. Others using the same box (and therefore same IP address) would impact our sending. Being on the same box as a spammer would impact deliverability for everybody who sent email via that IP.

2. SORBS showed multiple spam incidents, none of which seemed to be ours (which occurred on August 27 2020):

Getting off of the SORBS blacklist required us to sign up for an account and write up a ticket explaining what happened. I got a response quickly:

I was hard-pressed to come up with a good reply to that - the clear answer seemed that we should use a different IP address to send emails.

Getting off of JustSpam.org's list was a lot easier: one click and an hour later our server's IP was off:

Ultimately the solution to an IP address with reputation problems is moving servers. Not hard to do but we didn't want to try that just yet.

2. Fixing our content issues

We had three quick fixes here:

1. We didn't have a TITLE tag inside our HTML email's HEAD section. That was a quick fix.

2. We weren't including a text version of our emails alongside the HTML version, so we added that. Google apparently regards emails with no text version as spammier than those with both text and HTML.

3. We included a "Sent with Savio" link in the email footer. I removed that just to make the email as low-risk as possible.

3. Other action steps

Glock provided a list of other action steps we could take:

Many seemed like they didn't apply (use fewer exclamation marks in your emails!), weren't helpful (Google Postmaster literally told us to come back later after we signed up), or would take days / weeks to test and didn't stand out as things that were likely to work.

More resources (and the eventual fix!)

Glock also pointed us to two other articles, one which ended up containing the secret to fixing our deliverability issue:

1. How this guy decreased his spam rate from 35.2% to 2.8%. Spoiler: he sent emails from his spammy domain to friends, had those friends reply, and continued the conversation for a couple weeks. Google weighs replies and engagement very highly when determining whether an email or sender is spammy. In my work with Predictable Revenue I know that warming up email accounts like this is hugely important. But since our account had already been operating for nearly two years, this didn't seem like low-hanging fruit.

2. How to Find and Fix Email Deliverability Issues. This article was gold. It covered:

3. email content (ours was good)
4. server config like DKIM, SPF, etc (also good)
5. IP address reputation (not great, but fixable by switching boxes)
6. and email and domain reputation (unexplored)

Seemed like it was worth exploring email and domain reputation in a little bit more depth.

Testing email and Domain Reputation

Glock described how to test whether email providers saw your email address as spammy:

Use GlockApps to test different sender addresses on the same domain. Remember to keep other variables the same. If the same email (from the same IP and domain) is delivered to more Inboxes with a different sender email address, the problem could be in your old email address.

That seemed like a super easy experiment to run. So we changed our sending email address from notifications@savio.io to email@savio.io.

Here were the results:

Incredibly, the percentage of emails marked as spam had dropped from 43.7% to 1.4%!

To be fair, this test also included the fixes I mentioned above:

• The HTML email included a TITLE tag

• The email also included a text version

• I removed the "Sent with Savio" link from the email footer

But my guess is that changing the sending address is the major factor that solved the spam problem.

Compare the two tests side by side:

What next

There are still a few steps we can take to ensure deliverability stays high.

1. Think like an attacker when building features that customers can use to contact others. Given previous businesses we worked on we really should have spotted this before it went to production. We've now added an "Possible Exploits and Mitigation" section to our requirements document. This is to prompt us to identify potential major exploits worth devoting dev resources to when building a new feature.

2. Move email sending to a dedicated IP address. This would ensure we wouldn't be impacted by the spammy neighbour problem.

3. Isolate "Close the Loop" emails to a different sending subdomain. We provide a template, but ultimately our customers write the emails that get sent on our domain. Isolating these emails to their own subdomain would ensure that and reputation issues would not affect email sent from the rest of the app or from the email accounts we use to conduct business.

Last Updated: April 17, 2023

Want more articles like this?

Product Leaders from Slack, Zapier, and Appcues read our newsletter to delight customers, lower churn, and grow revenue.

Send me more articles like this

Unsubscribe anytime.