Headed to BDD? Is Cucumber a cool way to go? Let’s explore.
A software development organization may decide to move its development process to Behaviour Driven Development (BDD) because of 2 main reasons:
1. There is a need to make automated acceptance tests reviewable for clients or users and the non-technical members of the team like project managers, project managers, etc.
This is done to ensure that development is always according to the acceptance criteria and acceptance criteria are always according to the people who drive the business requirements.
2. Another gap that a BDD framework fulfils is it allows testers to get involved in automated testing without much programming language skills.
BDD tool, Cucumber has a frontend that is written in natural language and reduces the dependencies on developers for the construction of an automated framework.
Testsigma is an automated testing tool that further reduces the dependency on developers by removing the programming language’s extensive backend.
With the help of these tools, developers can now focus on feature development better and need not worry about automating the tests.
Introduction to BDD
Anyone who has worked in a software development company in any capacity will know that they follow a Software Development Life Cycle(SDLC) for the development of their software.
There are many ways to go about software development, the agile methodology being the most famous of them all. The different phases in an SDLC, broadly, look like below:
If we look at the SDLC diagram closely, there is a phase with the name, “Requirement Analysis”. During this phase, business analysts, product managers, and project managers work on understanding and gathering the business requirements and putting them in a document which is usually called “Business Requirements document” or “Software Requirements Specification”.
These specifications include all the important details about how the software should look, behave and perform.
The Business Requirements document acts as an input for the next phase which is the Design phase. During this phase, the business requirements from the document are converted to user stories/scenarios and wireframing related to the functionality is done.
These wireframes become the input for real implementation of the scenarios, functionalities, and features.
When the implementation starts, the user stories become the basis for development. And when testing starts, these user stories are translated to acceptance tests. But there is a gap between user stories and test cases.
BDD bridges that gap by mapping the user stories to automated test cases. Thus, the business requirements are bound to the implementation.
Here, we will discuss the most popular BDD framework, Cucumber in detail and then discuss some comparable features of the automated testing tool, Testsigma.
Introduction to Cucumber
Cucumber is the most widely used BDD framework. A fully functional cucumber framework has below components:
1. A frontend that has user stories written in a natural language that is easy to understand for business analysts and other stakeholders. The language is called Gherkin.
2. A backend that has the automation done in a supported programming language like Ruby, JAVA, .Net.
3. A mapping between the frontend and the backend to execute the automated tests via the frontend.
Introduction to Testsigma
Testsigma, a tool for automated testing that has natural language interface and makes it easier for everyone to automate the test cases. A full-functional Testsigma framework has the below components.
1. A natural English language frontend.
2. A mapping to webpage elements in the backend.
Before choosing what tool to use for automation for a project or an organization. There are a few features that are given the most importance.
Let’s compare these features in detail below:
1. Test Case Creation:
For the purpose of this document, we will talk about automation on the flight booking example site http://travel.testsigma.com
Test Steps that will be automated:
1. Go to http://travel.testsigma.com
2. Select “Multi-city”
3. Select “From Destination” as “LA-US”
4. Select “To Destination” as “NY-US”
5. Select “Depart Date” as 1 Mar 2020
6. Select “Return Date” as 7 Mar 2020
7. Select “Passengers & class” as “2 adults” and “Premium Economy”
8. Click on Search
9. Verify that 2 results appear with values from as LA and destination as NY
Test case creation for Cucumber
We will use Cucumber with JAVA, Selenium, and Junit. Because the test cases need to be executed on a website, the use of Selenium is essential. We need a testing framework for proper organization of test cases and reports, for this example, we have used Junit.
The code for the automated scenario is hosted here.
When automated, below is how the frontend looks in Gherkin, this is called a feature file where test scenarios verifying the acceptance criteria are created:
Gherkin has few keywords and the main ones are “Given”, “Then” and “When”. Usually, test cases can be automated with these keywords followed by natural language text. The natural language text is like a string that is passed to the backend and values from it are extracted in the form of variables.
Below is how the backend(JAVA + Selenium + Junit) code looks:
The Gherkin text can be written by testers as soon as the acceptance criteria are available after the Requirements Phase and then the backend code can be written by the developers as soon as the developed code is ready for testing.
For running these tests, a test runner also needs to be written which looks like:
The advantages of the above approach are:
1. Testers don’t need any programming language skills to write the frontend, they just need to know Gherkin.
2. Because the feature file is written in the English language, it is easy to understand and review for the rest of the stakeholders in the team.
3. The testers can write the feature files during the design and development phase while developers are busy coding.
The disadvantages of this approach are:
1. The code development for the backend step-definitions files is heavily dependent on the developers. Sometimes, a developer might be too busy to squeeze out time for test case automation and the activity of writing the feature files may not prove to be useful.
2. Another team member with the knowledge of the programming languages could be hired but the need for interaction with developers to understand the functionalities and code still remains and this also means need of extra resources for the new Individual.
Test case creation in Testsigma
Testsigma is quite similar to Cucumber if the frontend is concerned. Below is how the frontend for automated test cases in Testsigma looks like:
The ones starting with [+] are step-groups, below is how a typical step group looks:
The words in blue are direct values that are passed to a test case. The words in green are variables that need to be assigned value to execute the test case.
The best part about Testsigma is that these steps can be defined before any website is implemented or is functional. After the website becomes functional the only task is to assign these variables respective values.
These variables are elements on a page and the values are assigned by using the object spy feature in Testsigma which will be discussed in detail below.
2. Test Case Maintenance
BDD (Behaviour Driven Development) started with a focus on making requirements and user stories readable for reviews. And for helping the developers write code that passes the test cases written via BDD.
This process is effective when the requirements are fixed and the clients/users are available to review the user scenarios frequently.
But, in this fast-paced world, the software development process is fast too and there may be scenarios where the client may revisit the requirements while the implementation is midway.
For such scenarios, BDD may become tough to maintain because it means changes in the frontend as well as in the backend.
Testsigma, with its minimum number of components that need to be changed and tester-friendliness, has multiple avenues for easy test case maintenance as mentioned below:
1. Object Repository – In Testsigma, whenever a test case corresponding to a requirement is implemented, the UI identifiers or objects on the webpage are identified and mapped to the variables in the frontend.
For eg. Our example test case where a user searches for a flight from one place to another looks like below:
In the above image, the sign [+] denotes a ‘step group’. These lines have more detailed steps under them that can be expanded as below:
The objects in green above are called UI Identifiers and are stored in one place.
These UI identifiers can be assigned multiple properties and values according to their attributes and position on a webpage. The assignment can be done manually or via the Testsigma Chrome extension.
These methods are mentioned in detail below:
i. Manually create/edit UI identifier: The properties of the UI identifier can be opened in a user-friendly UI as in the screenshot above. For the UI identifier, an already existing attribute, operator or value of an element can be edited. Also, a new attribute can be added or removed.
ii. Create/Update UI identifier using Chrome extension: Testsigma has a chrome extension that can be installed easily and then can be used to select and copy attributes of the needed element.
The process is easy and involves below steps:
1. Open the UI identifier that needs to be changed
2. Go to the tab “Using Testsigma Chrome extension”
3. Open another tab and go to the web page that has the element that needs to be identified
4. A green outline will appear on the selected element and a click on it will store the information of the element in the UI identifier object in Testsigma.
2. Maintenance through AI
Testsigma has an AI core that automatically heals test cases when there is a minor change in a web page element that was in use. The AI looks for any other unchanged attribute for the element and uses that attribute instead.
Testsigma also displays the list of affected test cases for a particular UI identifier to ease debugging any change in any UI identifier:
3. Test Case Management
When an application starts being built, the test cases may be few but as it progresses the number of test cases increase. It is essential to have a proper test case management support for an organized and structured test case execution and maintenance.
Test Case Management in Cucumber
Test Cases in Cucumber can be organized in the form of features. A *.feature file that is written in the Gherkin format can accommodate all test cases for that feature. Multiple feature files can be created accordingly. For better visual management, there are IDEs available like Cucumber Studio.
Below is how multiple features can be organized in a folder:
Test Case Management in Testsigma
Testsigma has the built-in capability to create requirements. Related test cases can be added under them. Below is how it appears in the UI:
Testsigma also assigns multiple properties to a test case at the time of creation which can be changed as and when needed. The properties are as below:
Search filters can be used to view test cases according to properties assigned.
4. Object Spy
For an automated testing tool that is used for website automation, object spy or object recognition is an important feature. Correct object recognition can make the test case pass while a small mistake can make a developer spend a lot of time debugging the issue.
Thus, if there is any tool that justifies the investment for test automation, it is the ‘object spy’ / ’object recognition’ feature.
Because Selenium IDE is no longer supported, developers have to use developer tools that come built-in with browsers like chrome to extract information about XPath and other details on a website. The same is practised for Cucumber.
Testsigma has an in-built object recognition tool that can be used while UI identifier objects are created.
There are 2 ways of creating the UI identifiers:
1. Using the Chrome Extension according to the tutorial here . This way, advanced knowledge of CSS and XPath is not needed and objects can be automatically created via the chrome extension.
2. Manually, UI identifiers can be created according to the tutorial here if the creator has knowledge of CSS and XPATH. Here, multiple attributes for an element can be mentioned easily.
5. Reporting
Test case creation and execution are core components of the test automation process but they can only be successful if the test cases are executed frequently for bug detection and reported in a user-friendly way.
Reporting in Cucumber
Cucumber can be integrated with most of the popular reporting tools in the market and the reports can be generated. It is the configuration of the reports that need to be taken care of.
Below is how we configured the reporting tool in the cucumber test runner class. To make this work, the required libraries need to be downloaded and configured in the path too.
Reporting in Testsigma
Testsigma has support for exhaustive reporting to make debugging easy. The Test results are shown for pass/fail. There are screenshots for failed steps and there is a video compiled for the executed test case for the scenarios where the user wants to know what happened during the test executions. There are logs attached for any error logs too.
The screenshots and videos save a lot of time for the developers and testers. The screenshots and videos can really prove helpful in scenarios where the error does not help pinpoint the problem and this happens a lot in the case of Selenium as well as Cucumber.
6. Integration with CI/CD
Cucumber, as well as Testsigma, integrate with most of the CI/CD tools without any hassles.
7. Learning Curve
Cucumber and Testsigma have a lot of similarities – the test cases are created in a natural language being the main one. The largest difference between them is that the backend in Cucumber requires expertise in programming languages to be able to make it work while Testsigma does not require any programming language skills.
The only skills that may be required for advanced debugging will be CSS and XPATH which is again taken care of via the chrome extension. Thus, Testsigma eliminates the overhead of developers for developing the backend code using programming languages and gives the testers control of the frontend as well as the backend for test automation.
Thus, the learning curve for a person who has no knowledge of a programming language is steep for cucumber as compared to a low learning curve for Testsigma.
8. Setup
Below are the few components that need to be set up before the test cases can be created and executed(For our setup on Eclipse):
1. Eclipse IDE
2. Selenium integration plugin for IDE
3. Cucumber plugin for IDE
4. Java-related configuration for IDE
5. Reporting tools configuration for cucumber
For Testsigma, because it is hosted on the cloud, no setup is needed. Just login and you are ready to start the test case execution.
9. Test case Execution and Performance
If there are too many test cases and execution is getting slow, for cucumber, either the test framework’s support could be used for parallel execution or the machine’s capacity may need to be increased.
Testsigma has the inbuilt capacity for parallel execution of test cases that can be executed on multiple devices at the same time. Also, being hosted on the cloud, there is no problem of scaling up or down according to the number of test cases that need to be executed at one time
10. Supported Applications
Cucumber is an ideal BDD tool for web applications only. Testsigma can be used for automation on iOS, Android, APIs as well as web applications.
11. Customer Support
Cucumber being a widely used tool, there is enough documentation and help available on forums, premium support is available as well.
Testsigma has dedicated support for any problems encountered. They are gradually building up their presence in forums.
12. Ease of Reviews
As also discussed in the test case creation process, the ease of review via the cucumber tool is the same as that in Testsigma.
Because of Testsigma’s low learning-curve, project managers and clients can also contribute to creating the test cases if they want to, without any programming language expertise.
13. Tracking/Traceability
Because cucumber has most of the work done at the backend, the information related to the last changes done to the files and the executions can only be stored in the files are hosted on a source code repository that has built-in support for storing this information. Cucumber in itself has no provision of storing this information.
Testsigma being a testing specific tool keeps all the tracking information for all test cases.
Check out Testsigma’s simplified test automation approach!
