Case Studies – Mailtrap

How The Software House Approaches Email Development

veljko.ristic — Tue, 14 Jun 2022 11:35:39 +0000

5 projects

where Mailtrap is involved

500+ messages

sent on a monthly basis

54 users

running email tests with Mailtrap

The company

The Software House (TSH) is a popular software development company based in Poland. They deliver a wide array of services, such as business analysis, web & mobile development, UI/UX design, DevOps, and quality assurance.

Their portfolio of successfully implemented projects includes real estate & marketplace platforms, an employee benefits system, a ticket sales CMS, language learning & business accelerator platforms, as well as a magazine subscription system.

End-to-end testing is a major area of TSH’s expertise, and they have built their own automated, open-source testing framework.

Kakunin is a Protractor extension that enables the easy writing of complex testing scenarios with minimum coding skills. Its goal is to empower people with little to no technical background to create automated tests from day one and eliminate the steep learning curve.

HQ

Gliwice, Poland

Industry

Software Development

Key Features Used

The Challenge

Most of the solutions developed by TSH rely on email communication with end-users. Each sent message – be it an account confirmation, password recovery, product update, or payment reminder – requires extensive testing prior to deployment.

TSH used to set up their own dedicated SMTP servers for testing if emails were sent, validating links, and running other essential tests. This approach, however, proved to be both time-consuming and expensive. Furthermore, the environment had to be regularly maintained which increased engineer workload.

TSH also tried to set up dozens of disposable email addresses for testing purposes. It somewhat did the job but this was, once again, time-consuming and couldn’t provide the appropriate level of security. Dummy inboxes were, after all, publicly available.

Kakunin also required a reliable, automated solution for testing emails. Building an entire infrastructure again wasn’t an option. Using common email services would inevitably lead to some false negatives. TSH couldn’t control all aspects of a third-party inbox nor access it via API.

I love the concept of capturing emails for all the test cases in one place. If we used some other tools, we would have to be logged in to 10 or more inboxes to keep track of each message. Thanks to Mailtrap, we can verify them much quicker. All in one place, it’s a ‘dream for testers!

Marcin Basiakowski

Head of QA at TSH

The Solution

Eventually, The Software House switched to Mailtrap as a dedicated email testing solution. They set it up to capture all emails going out from the development and QA environments. Engineers and testers check that emails are sent to the correct users and that no duplicates are created. They preview emails and check their responsiveness.

In Mailtrap, they created separate, team-wise accessible inboxes for different environments (for example, UAT, staging, local, etc.) and client projects. Emails don’t get mixed up and it’s easy to jump between environments or check emails sent from a particular project.

TSH also integrated Mailtrap with the user acceptance testing servers. Test emails are forwarded to regular inboxes but are still available in Mailtrap. This way, all the messages are kept in one place which is highly appreciated by testing teams.

For Kakunin, accessing Mailtrap via its UI wasn’t an option. CI and automated test environments require automated solutions, and that’s precisely what TSH found in the Mailtrap Email Testing API. They built a Mailtrap adaptor in the Kakunin framework for handling user inputs.

Users of Kakunin write their testing commands in the human-readable Gherkin language as well as JavaScript. A layer of abstraction translates these inputs into specific features of the Mailtrap Email Testing API using the dedicated adaptor.

Mailtrap is super easy and convenient to use. Our devs and QA onboarded in one afternoon. We love ready-to-run configurations for various frameworks that make our job much faster. You just copy & paste, and it simply works.

Adam Polak

VP of Technology at TSH

Why Mailtrap?

TSH engineers and testers told us that they were impressed with the speed of the setup. In 10 minutes, they had a testing inbox configured and integrated with their app. As the test emails kept flowing, they could view any of them without the fear that old messages would be lost.

It was no different for Mailtrap Email Testing API. Our interviewees at TSH told us that it was very straightforward and quick to plug into their framework.

They also appreciate having individual inboxes for each environment and project rather than pushing all emails into one folder. This makes it easy to navigate and quickly find the right messages. They can rapidly run new iterations and view each message in a Mailtrap inbox just seconds later.

Marcin Basiakowski

Head of QA at TSH

How Calendly Streamlines Email Testing with Mailtrap

Oleksii Ianchuk — Tue, 14 Jun 2022 11:34:36 +0000

40M

emails sent every month

40+

developers, QAs, and PdMs involved in email testing

Since 2017

using Mailtrap

What’s Calendly

Calendly is arguably the world’s most popular tool for scheduling meetings. It’s used by over 10 million customers worldwide, with companies such as eBay, Dropbox, and Twilio among their clients.

Calendly connects to your calendar and takes charge of your availability. It coordinates appointment scheduling and keeps all parties in sync through timely reminders. What’s more, it gives you a straightforward overview of upcoming meetings and enables seamless team collaboration.

Email is an essential part of any appointment scheduling, and it’s no different for Calendly. Since 2017, they have been using Mailtrap to test all their transactional emails.

HQ

Atlanta, Georgia, USA

Industry

Business Applications

Key Features Used

Try Mailtrap Now

The Challenge

The Calendly platform sends out around 40 million emails during an average month. These include meeting invitations, reminders, “thank you” notes, account confirmations, password resets, etc. What’s more, customers can customize their email templates, which creates many variables to test before a release.

At any one time, around 15 developers are involved in building and improving email-related features. A similar number of QAs jump in next, running extensive tests of each new build. One mutual inbox for capturing all their test emails wouldn’t work at such a scale. Instead, each tester would need to set up a separate, personal inbox, or ideally, a few of them to begin with.

At the rapid pace that Calendly is growing every year, this would mean an extensive amount of manual work to set things up and organize the daily work of different teams. What’s more, the emails they send are opened in virtually any email client.

As such, Calendly needed a reliable and accessible way of testing the emails inside and out, validating their HTML/CSS rules, identifying spam threats, etc. Essentially, they needed a way to automate their email testing and make it scalable.

Mailtrap streamlines our testing because everything is concentrated in the same area. I jump between different environments, but I still always know where to look for my test emails.

Michele Rawas

QA Engineer at Calendly

The Result

Calendly integrated Mailtrap for all their email testing. Each engineering environment features a dedicated Mailtrap inbox. As engineers iterate on their emails, each subsequent message lands instantly in a respective Mailtrap inbox.

The most common scenario for engineers is to test whether emails are triggered in the first place and subsequently arrive into their Mailtrap inbox. In some cases, they have to dig deeper – i.e., check a message for HTML/CSS errors, test responsiveness, keep an eye on a spam score, etc.

When all tests pass, an engineer updates a PR, and a new review app is sprung up. Automation using Mailtrap API assigns an available Mailtrap inbox from an over 100 QA inbox pool on the Calendly account. A review app is automatically provisioned with the SMTP credentials of the respective inbox.

A QA engineer opens a Jira ticket and finds a connected PR. There, they see the details of a deployed branch and a respective Mailtrap inbox. As they test the workflows, emails such as account confirmations or meeting reminders are delivered into a test mailbox, one by one.

They pay special attention to new emails and those that were updated. In Mailtrap, it’s easy to tell them apart and compare different versions of the same message. QAs complete the acceptance tests, and if everything works as expected, the build proceeds to staging.

For staging, Calendly sets up one more testing mailbox within Mailtrap. Here, QAs test the fixes of the issues they have previously received from support. They simulate the actions of an end-user and monitor if all emails are sent as expected. If everything goes well, the feature is passed for later release on the Calendly platform.

What I like the most about Mailtrap is its simplicity. It’s very easy to set up. We use API to automate tests, and the API experience has been amazing.

Rob Wilson

Engineering Manager at Calendly

Why Mailtrap

When we talked to Calendly about their use case, each of our interviewees emphasized the convenience of having the entire email testing in one place through shared inboxes accessible for any team member. They also highlighted that Mailtrap makes it easy to find any email in question, compare its versions, and navigate around different environments.

They love having the complete testing toolset at their disposal and being able to verify different aspects of an email right from their Mailtrap inbox.

They also appreciate Mailtrap’s simplicity and the support for automation, provisioning review servers, and automating end-to-end scenarios. It is easy to get up and running, and the whole experience is extremely straightforward.

Finally, they enjoy the peace of mind they have knowing that the emails are not actually sent and will never land in customers’ inboxes. They can therefore work more efficiently and test anything in their apps without risks.

Mailtrap has made it so much simpler for me to test because it’s an organized folder of all of the review apps that we have. It makes it so easy for me to find precisely the emails I need.

Raaziq Whyte

QA Engineer at Calendly

How Kinsta’s Developers, Designers, and QAs Collaborate On Emails

veljko.ristic — Fri, 03 Jun 2022 13:46:00 +0000

40M

emails sent every month

40+

developers, QAs, and PdMs involved in email testing

Since 2017

using Mailtrap

What’s Kinsta

Kinsta provides premium WordPress hosting services to companies such as Skillcrush, MariaDB, and TripAdvisor, to name just a few.

With their MyKinsta platform, the company intends to reinvent WordPress management. It’s like a control center for your entire website. A single place to track the performance, fight security threats, and monitor comprehensive analytics.

All of this, on an ultrafast infrastructure, with real-time uptime monitoring, and the best in class support.

HQ

West Hollywood, CA

Industry

Cloud Infrastructure

Key Features Used

Try Mailtrap Now

The Challenge

Thousands of email notifications are dispatched to users every single day from MyKinsta. At such scale, the team cannot afford a broken link, a mismatched dynamic text, not even a typo.

Before they started using Mailtrap, the development team would simply log outgoing emails into the console, and validate them this way. With the sequence of unit tests accompanying each release, the testing needs were pretty much covered.

In 2019, Kinsta started sending HTML emails. They also began to grow rapidly and release a number of new, email-related features, dispatching more and more emails every week.

Therefore, it was no longer enough to just print emails in the console. Kinsta needed to see real messages and improve on them, with the dev, QA, and also design teams involved.

We’re easily talking about multiple engineering days that we’ve saved up because of Mailtrap. It helped us really streamline the email development.

Andras Gerencser

Director of Development at Kinsta

Mailtrap Implementation

Kinsta has many different features in development at any given time. Each is built and tested in isolated environments and many of them have independent email features.

Developers found it very easy to integrate Mailtrap into their environments. Each person would create a separate Mailtrap account to test emails from their project. They would capture every message into their inbox, and iterate on it until the code was ready for review.

They also onboarded the design and QA teams and provided each person a separate Mailtrap account. Finally, they grouped all users into the Kinsta organization on Mailtrap and set up a shared inbox for all.

Shared inbox is so invaluable for our work. I don’t know where we would be with email testing if we didn’t have this feature.

Viktoria Korodi

QA Team Lead at Kinsta

Email Testing Flow at Kinsta

A developer receives a Zeplin file with a base template and a mock of the content for the future they develop.

Their role is to implement the email design in their project, insert all dynamic content (recipient’s name, website, etc.), and test the email template.

When they’re happy with the outcome, an automated system creates an isolated testing environment, connected to the shared Mailtrap inbox. A test email is sent.

At this stage, the QA and design teams get involved. They verify the new template and discuss it on a PR page if needed. Then, everyone can chip in and share their opinion.

Finally, when a template is tested inside and out, it’s queued for deployment and implemented on MyKinsta when the epic is ready for release.

Why Mailtrap is the Best Fit for Kinsta?

The main thing Kinsta emphasizes is the simplicity of Mailtrap. They find it very easy and straightforward, with little to no learning curve. As a result, they had no issues onboarding the entire team.

The ability to share inboxes and their content across teams is really valuable for them as well.

An alternative would be generating some dummy Gmail accounts that devs would send test emails into. However, this takes a lot of time, and it requires sharing credentials and tracking different folders within Gmail (including spam), which can make it easy to miss some of the critical emails. That’s why a single shared inbox within Mailtrap is certainly more convenient.

Developer experience with Mailtrap is really, really good. We’re saving up a lot of time just because we can easily collaborate

Kristof Dombi

Head of Development at Kinsta