With Mobile Growing, Testing Is in Spotlight at Gilt Groupe, Isaacs Tells Developers

Mobile is growing at Gilt Groupe, so the company is codifying and automating some quality assurance (QA) and sanity testing to ensure that all changes made in mobile work well for the consumer. Matt Isaacs of Gilt Groupe explained this and more to mobile developers at a Mobile Application Developers of New Jersey meetup in Hoboken Sept. 11, 2013.

Organizer Gary Rattray was out of town, so the meeting was run by Mark Annett with the help of other group members. Some 25 mobile developers turned up to hear Isaacs speak.

For those who are unaware, Gilt Groupe (New York) is an e-commerce site that runs flash sales on luxury goods. There are 14 mobile developers at Gilt, and much of the QA is outsourced.

“We are getting pretty close now to almost half of Gilt’s revenue on mobile,” Isaacs told the group, adding, “Our overall goal is to make the best shopping experience possible on mobile.”

“Any issues that slip through QA end up giving us poor reviews, and any … frustrated customer means lost revenue for us,” Isaacs pointed out. Users are really unforgiving, he added.

As mobile grew in importance, the company realized it had to put greater emphasis on sophisticated, systematic testing of its mobile app to make sure customers could easily complete sales transactions. If automation was applied to testing, the QA department could better focus on finding issues, Isaacs said.

In addition to focusing on features and bug fixes, “We want our developers to get involved with testing — not just visually testing the code but actually writing tests for features and getting involved in automation of functional testing,” said Isaacs.

He explained that the mobile team doesn’t make continuous builds; rather, it produces nightly releases of the apps for internal use and testing. Full, continuous integration will come later, once functional tests have stabilized and developers are more comfortable with the process, he said.

Every time the continuous integration system creates a build, a suite of sanity tests is run and a report generated. The intent is to free QA from having to do sanity testing manually, he added.

QA can then do stress testing and look at features in a different way, possibly devising ways Gilt can be doing things a bit better, Isaacs said. The real benefit is that QA can find the issues before the users do, he added.

Isaacs explained that the Web team has already figured out testing, so the mobile team isn’t out to reinvent the wheel, though there are some differences.

Gilt Groupe is really focusing on functional UI tests, he said. The test framework the firm started working with is KIF (Keep it Functional), which is maintained by Square. KIF is pretty heavy on the use of undocumented accessibility application programming interfaces (APIs), however. Apple can do away with any of the stuff in there at any time, he said.

Gilt Groupe had a hard time integrating KIF with continuous integration, Isaacs added, so the company started looking at alternatives.

“We’ve been looking at Appium, which translates Selenium calls into UI automation commands,” he said. It also supports testing on Android and mobile Web. “The community is really friendly, and they like input,” he added.

Gilt is a Scala shop, and “we have a ton of Scala infrastructure. Since Appium is language-independent, you can use Scala,” Isaacs said. “We are also getting a lot more involvement from the rest of Gilt, and they can come in and help us with testing because they know this framework,” he said.

Right now the testing Gilt does creates reports that are guidelines for the QA department to look at. Down the line, the firm would like to eliminate manual sanity tests all together, Isaacs said. After fixes are made, the company reruns tests to ensure that the issue has truly been eliminated.

In response to an audience member’s question, Isaacs said both KIF and Appium are open-source.

Mobile developers interested in more information on Gilt’s use of Appium should check out the blog post Automating Mobile Testing at Gilt and the slide presentation attached to it.