Customer story
At AllTrails, automated testing enhances collaboration between engineering and QA
AllTrails is home to the largest collection of digital trail guides in the world. The company was founded on the idea that we’re all made better by spending time in nature. Today, the company continues to be driven by the desire to share the outdoors with as many people in as many places as possible—and to do so responsibly and respectfully.
Tests
Runs
Tests Run
Hours Saved
While an engineer is still in the same headspace of writing a feature, if they see a bug in the code that they've written, they're able to immediately fix that before it even gets to our manual QA team.
The Challenge
Manual testing creates a heavy workload for a lean workforce
AllTrails, a mobile app for outdoor recreation, had a nuanced strategy for testing updates before they shipped. “Before every release, we would rely on other cross functional teams to run a regression suite on the app,” said James Graham, CTO of AllTrails. ”This could take anywhere from three to four days.”
But the company was growing quickly and needed to ship more often and at a regular cadence. Taking cross functional teams away from their primary work to manually test was not scalable.
“We wanted to ship much more quickly than we’d been shipping in the past,” said Adam Blumberg, QA Manager at AllTrails. “We didn’t want to wait two, three, four weeks to get things out the door.”
The Solution
QA team codes its own tests and triages with engineers in real time
Graham and Blumberg set out to find automated end-to-end testing for their mobile app. They knew they didn’t want the technical debt that came with engineers having to work on scripts, so they chose to partner with Waldo.
Waldo’s user-friendly interface removed the burden of having to write and maintain scripts from the engineering team—the QA team can now create tests themselves. “Waldo’s interface makes it really easy for our QA team to edit, test, and modify each test,” Graham said. “It's a more seamless experience than managing and maintaining scripts.”
“Waldo is great for allowing people who aren't software engineers to jump right in and code a test quickly,” Blumberg added.
Now, instead of pulling people away from customer support for manual testing, the QA team can run a test suite on its PR process and code cycle nightly. These tests include login, signup, navigating to a trail page, and looking at a map—critical paths that cannot be broken. Engineers immediately receive the results from the tests QA ran and are able to resolve any issues Waldo may have found.
By taking advantage of the ‘Rules’ feature in Waldo, the QA team can also implement a smaller subset of tests while the engineers run through them simultaneously. “That means the engineers are able to work on the issues before they reach the manual QA team—as opposed to being notified later, when the engineer has moved on to something else,” Blumberg said. “Once you've moved on, it's hard to put yourself back in the headspace of that code, and the bug doesn't immediately get fixed.”
The Result
With Waldo’s seal of approval, issues don’t fall through the cracks
About 60% of AllTrails’s QA on the app is now covered by Waldo. That’s time that goes back to the QA team to spend on ad hoc testing and working on the next release cycle. The QA team can work with confidence knowing that Waldo will find bugs they didn't catch in manual tests.
“We know that Waldo will run a regression suite every day and find any bugs or gaps in the UI that may change from the previous version of the app,” Graham said. “The customer experience is what we're validating.”
By taking time off the manual QA team’s plate and allowing engineers to respond to issues in real time, AllTrails has gained the velocity it needs to ship the app on time.
“It gives everybody high confidence when we have that Waldo seal of approval,” Blumberg said.
We're able to ship a lot quicker in our release cycle. The amount of coverage that we have on the mobile app, it allows us to take those cases off of our manual team's plate.
