Blog

×

Moving from TestNG to Cucumber

By April 25, 2017

This is a guest post by Monika Khandelwal, Member of the Technical Staff at Qubole.

Qubole offers enterprise grade, cloud-native big data analytics platform and we’re always making improvements to ensure we provide the highest quality of service to our customers.

In this blog post, we’ve outlined our motivation to switch from TestNG to Cucumber. This move turned out to be great even though Cucumber is generally considered to be a specification tool and not a test framework. In addition, because the majority of the analyses & workflows processed by our customers are complex and domain driven in nature, the need to use a collaborative test tool was of utmost importance.

Challenges with TestNG

For UI testing, initially we were using Selenium TestNG framework which was running almost 250-300 test cases as part of our Functional Test Suite. We were using all of the available benefits of TestNG like grouping, ordering, parallel execution etc.

At a minor scale, when the product had limited features with fewer test cases and a small team, TestNG was easy to maintain. But as the product and the team continued to grow, we began to experience communication gaps. Some of the frequent questions being asked were :

  1. What tests are we running?
  2. How will we find missing scenarios?
  3. How much is the test coverage?
  4. Can anyone add/fix tests?
  5. How can we debug/analyze the tests?
  6. How do users read the reports?

Reports generally contained stack traces which were hard to decipher by all the team members, especially those from a non-technical background. Because of this we wanted to add a degree of usability across the board.

Migrating to Cucumber

In his blog, Aslak Hellesøy writes that Cucumber was born out of idea of automating acceptance tests, functional requirements and software documentation into one format.

Cucumber uses a business vocabulary, which is well understood by anyone on the team including end users, stakeholders and the engineering team. It increases the readability of reports by giving a clear picture of what is being executed and what is missing.

Technical details

We could reuse our Selenium WebDriver framework following a Page Object Model design pattern. Additional work was to eliminate all TestNG related files and introduce the Cucumber-specific Feature and Step Definition files.

Untitled2
Here is an example of the transformation from TestNG to Cucumber

Below is a TestNG test case to run a hive query and verify the result:

@Test(groups = { "analyze.hive" }, priority = 1)
public void test_RunHiveQuery() throws Exception
{
    homePage.goToAnalyze().compose().selectHiveCommand().hiveStatementOrPath("Query Path").enterHivePath(AnalyzeV2DataProvider.generateCommandDetails(CommandTypes.HIVE, QueryType.valueOf(dataType), "").get("query")).runQuery().waitForQueryToComplete(CommandStatus.SUCCESS).assertForEmptyResultQuery();
}

Using Cucumber, we transformed the same test case into a business use case, which everybody could understand. Our test cases became use cases written in plain English as shown below:

#1 Run a Hive Query and verify various Parameters
Scenario:
Given User is on Analyze Page
When User enters path "HIVE_QUERY_PATH_PRODUCING_EMPTY_RESULTS" and run
Then Verify results are empty

Cucumber scenarios are not an executable form of tests. To make it work, Step Definition contains the supporting code. It acts as the translator between the feature file and the base code which can be written as follows:

public class HiveSteps
{
    CommonResourcesSetup commonDeclaration = CommonResourcesSetup.getInstance();

 [email protected]("^User is on Analyze Page$")
    public void navigateToAnalyzePage() throws Throwable
    {
        commonDeclaration.homePage.goToAnalyze();
    }

 [email protected]("^User enters path \"([^\"]*)\" and run$")
    public void selectHiveQueryPath(String dataType) throws Exception
    {
        commonDeclaration.analyzeCommon.compose().selectHiveCommand().hiveStatementOrPath("Query Path").enterHivePath(AnalyzeV2DataProvider.generateCommandDetails(CommandTypes.HIVE, QueryType.valueOf(dataType), "").get("query")).runQuery().waitForQueryToComplete(CommandStatus.SUCCESS);
    }

 [email protected]("^Verify results are empty$")
    public void verifyResultsAreEmpty() throws Exception
    {
        commonDeclaration.analyzeCommon.assertForEmptyResultQuery();
    }
}

The JUnit runner class is used to trigger the tests. It points to the feature file to be picked and executed.

@RunWith(Cucumber.class)
@CucumberOptions(format = { "pretty", "html:target/cucumber", "json:target/cucumber-analyze.json" }, glue = { "commonsteps", "analyzev2" }, features = "src/test/resources/Features/AnalyzeV2", tags = { "[email protected]", "[email protected]", "[email protected]" })
public class AnalyzeTest
{
}

By default, Cucumber provides a lot of built-in functionality like background and hooks to better handle and manage your test suite. Similar to groups in TestNG, we can group a set of tests under a tag in Cucumber and choose to either include or exclude them during execution.

End to End Test

We created a feature file called endtoend.feature and documented around 20 scenarios, which incorporate frequently used workflows and how a new user will use the product. For any new hire, executing those Scenarios manually acts as a product walkthrough. We also use it as basis for our release sign off.

Contribution

Cucumber has the ability to support more than 60 spoken languages. As a user-friendly tool, it has encouraged developers to contribute to the feature file, review the scenarios and identify missing tests.

Reporting

For reporting, we have installed the Cucumber Reports Plugin into our jenkins job and added the job to a nightly run. It publishes a neat tabular as well as graphical report, which is evaluated across the team and stakeholders. Below is an example of a passed test case:

Data Analytics with Business Value

In case of failure, it attaches a screenshot link and adds the URL at failure as shown below:

Data Analytics with Value

In Summary

Incorporating Cucumber in UI functional tests has increased the overall business value as we get a broader picture of how our customers use the product. Feature files are written soon after the requirements are frozen, thus business stakeholders can give feedback at an early stage.

Test code maintenance is low and distributed, because it is easy for anyone to add a new feature file or update the existing one. After all, we shouldn’t forget “quality is everyone’s responsibility!”

 

Share our Post

Comments are closed.