Gamifying product testing – learn how Cask did it

Priyanka Nambiar is a Project Manager at Cask where she works with the engineering team simplifying their software development processes and lifecycle. Prior to Cask, she did business development and program management with eBay and Magento Inc.

Last week at Cask was especially abuzz with activity. Why, you ask?

Over the last couple of months, the engineering team at Cask had been focusing on building a brand new version of our flagship product, the Cask Data Application Platform (CDAP) version 3.5, which will include many new engaging and useful features like enterprise-grade security capabilities, new features in Cask Hydrator, data usage analytics, and much more.

Customarily, we dogfood our own software releases by testing and running internal hackathons for the entire engineering team after feature implementation is complete. This happens before every GA release at Cask, and CDAP 3.5 is no different.

Only this time, we decided to add a twist! This blog post gives you a run-down of our model for the special  event, and why it was such a success!

The Idea

The theme for the 3.5 hackathon was gamifying the process of finding bugs – #BugSmash. Our goal was to find and fix the maximum number of bugs, deliver a strong stable release, and bring out the fun and competitive spirit in everyone! We modeled our #BugSmash after Tour de France:

  • Winning team decided per day based on highest points tally as of 5pm
  • The overall winner was based on combined points for all the days.

Setting it up

Every summer at Cask is particularly lively because our engineering interns join us! For this Hackathon, the interns played a huge role..

We starting with planning for a week-long event starting Monday, July 27th. Next, we divided up our engineering team into groups of 4-5 members, each led by a fearless intern!! Each team lead got to pick a name for their team, and we decided to follow the on-going Pokemon Go trend and use Pokemon names!

Slide1
The next step was to establish some rules for the #BugSmash:

  • The entire engineering team at Cask wore their testing hats for the week
  • Each competing team was assigned an area in CDAP to focus on breaking
  • Team lead was responsible for making sure the documented test plan was executed and bugs were filed and fixed
  • The testing ride started Monday 07/25 at 11am (kickoff)
  • The daily points tally was conducted at 5pm, with a daily winner announced
  • The ride ended on Friday 07/29 at 7pm
  • Grand prizes for the teams with the maximum number of cumulative points on the final day!

Our scoring system was designed to incentivize finding and fixing bugs, writing integration tests for corner cases, and to foster collaboration between the teams. At the same time, we also needed to discourage reporting invalid and duplicate issues, using negative points. In addition, each day had surprise rules – like, bonus points for the team with first 5 issues filed, for the highest testing coverage etc. The combination of fixed and flexible runs made each day different and exciting.

The Race

The winner of both Day 1 and Day 2 was team Magikarp by quite a small margin. They got bonus points for fixing and resolving the maximum number of JIRAs, as well as overall good test case coverage!

The winner for Day 3 was Team Ditto for the most number of JIRAs reported + high test coverage!

Day 4 and Day 5 were won by Team Lucario and Gengar respectively, and this came as a quite a big surprise with their performance improving dramatically!

image2
Check out what Kashif Nazir, engineering intern from Stanford University and team lead for Magikarp, had to say about the team’s success on Day 1 and Day 2:

 

The results and the outcome

The overall winner of this Cask #BugSmash was team Ditto, led by Shenggu Lu, intern from Carnegie Mellon – with steady progress on a daily basis and a cumulative total of 158 points!

Team Snorlax led by Riwaz Poudyal, joining us from Williams College, came a close second with 130 cumulative points, and also had the maximum number of JIRAs (55) filed!

Team Magikarp led by Kashif was only team to top daily leaderboard twice!  

Team Lucario (led by Chris Lu of Bowdoin College) and Team Gengar (by Iraida Ermakova of UC, Berkeley) displayed great collaboration in running tests on some complex aspects of CDAP and taking wins on day 4 and day 5 respectively.

Screen Shot 2016-08-02 at 12.10.33 AM
Now for some statistics. Over a span of five competitive days, we reported 262 issues and resolved 143, pushing our overall manual test coverage to over 65%. This definitely sets new records at Cask for the number of issues filed in 5 days, and gives us a lot of confidence with the stability of this upcoming GA release of CDAP 3.5.

 

Untitled drawing (7)
Screen Shot 2016-08-03 at 12.00.04 AM

Conclusion

I am very happy to share that BugSmash turned out to be a huge success, bringing the team closer while finding and fixing issues. Working at Cask has always been fun and enjoyable, and events like these add immensely to the quality of worklife at Cask. There’s no wonder we’ve been named a Great Place To Work 🙂

Let us know what you think of our approach to gamifying product testing, and whether you plan to implement something similar with your own teams? Tweet us @CaskData or write to us!

And finally, if you are an engineer looking to solve interesting problems and working with a super fun team – join us! Cask is always on the hunt! 🙂

 

The team leads out for dinner to celebrate their BugSmash success!

 

<< Return to Cask Blog