TESTOMAT Project

In software development, reliability and agility often seem like opposing forces. This issue is aggravated by the increasing complexity of software and the constant drive towards faster release cycles for maximum market impact. Manual testing cannot meet these challenges, yet many safety-critical and hardware-oriented companies believed that test automation was irrelevant to them back in 2017. SMEs, meanwhile, missed out due to a lack of awareness or investments in automated agile testing. To strike the perfect balance between quality and speed, a new approach to automation was needed.

The goal of ITEA’s TESTOMAT Project, which stands for ‘The Next Level of Test Automation’, was to allow software teams to increase development speed without sacrificing quality. To achieve this goal, the project advanced the state of the art in test automation for software teams, moving towards a more agile and automated development process. The ‘next level’ differs per organisation and depends on factors such as sector, size and practices. Many automation-related variables are unknown to the organisation, so the TESTOMAT Project focused on providing software teams with personalised roadmaps for improving their automated testing in a cost-efficient manner. Through a Test Automation Maturity Survey, completed by 151 respondents in 101 organisations and 25 countries, the consortium of 34 partners from Finland, Germany, the Netherlands, Spain, Sweden and Türkiye was able to gather the status of test automation and key issues worldwide. On this basis, a Test Automation Improvement Model (TAIM) was developed to define measurable steps for improvement.

Success and diversity in test automation

Success in the TESTOMAT Project was measured in terms of test effectiveness, test optimisation and quality and standards in testing:

• Effectiveness refers to the ability to find bugs and maintain the test suites.
• Test optimisation means selecting only relevant test cases from the millions created through automation. One element of this is clone detection and similarity analysis, which enable the maintenance of much larger suites.
• Quality and standards address non-functional aspects such as security and performance testing. Here, the TESTOMAT Project worked to bring automation to standards that previously focused on manual testing and also contributed to standard extensions and to the release of a completely new standard: UML Testing Profile 2.0.

To demonstrate the project’s versatility, its ten use-cases can broadly be grouped into the following categories: trains, planes, telecoms, forestry, machines, robotics, banking, wind turbine maintenance with drones and port integration technologies.

Customised nature of test automation

The TESTOMAT Project’s great strength is its customised nature, which guarantees promising results regardless of the level of automation already found within a company. For those using fully manual testing, an average improvement of 60-80% can be gained in their time to market. Such improvements are highly variable due to enormous differences across companies.

To mention an example, Ponsse, one of the world’s largest manufacturers of cut-to-length forest machines, renewed its test automation systems during the project and improved its unit testing and requirements coverage by 64%. Another advantage for Ponsse was that the execution of tests was 25% faster with test automation and they could execute tests automatically outside of office hours. Nowadays, Ponsse’s simulators indeed run test automation outside of office hours, saving about 500 hours/week of functional testing work for testers. Alongside, Ponsse has been able to reduce defects that could have ended up in the customer’s product by 20%. Finally, the time from development to testing with actual machinery has been reduced, which has also reduced the production time as a consequence. Ponsse has taken a clear step towards test-driven development as their acceptance criteria for business requirements are now linked to test automation cases. In early 2023, Ponsse even took it up a level and started an end-to-end testing project in which they began combining test automation platforms, thereby connecting the platform created for testing their machines and the platform for their digital services. In this way, they can automatically validate the data all the way from the machine end to the digital service endpoint. All in all, the TESTOMAT Project was a great starting point for Ponsse’s test automation and they have plans to expand these capabilities further. In the near future, Ponsse will invest in simulators that will run test automation 24/7 and won’t be shared with manual testing.

Another example is Spanish SME Prodevelop, which managed a 220% increase in test efficiency. Thanks to the project, Prodevelop has increased the type and number of tests, many of which have been automated, thereby allowing them to test more features of their application in less time, increasing the quality of their products and reducing the effort spent on testing. At the other end of the spectrum, Ericsson already carried out completely automatic testing but still saw a 29% improvement in product quality. This significantly lowered development costs and increased the speed to delivery to customers.

Likewise, Saab increased their number of product users by 440%, leading to higher quality in earlier development phases. For already-automated companies, 15-30% fewer faults are predicted. These achievements, resulting in less bugs in software, are fundamentally important as nowadays people totally rely on and trust, for example, the use of software tools in healthcare, scientific experiments and many more domains. The tools therefore also need to be tested to make sure that a strong foundation exists.

“The immaturity of mutation testing in practical, industrial settings has held back its wider adoption. Through the TESTOMAT Project, the tooling and methodology of mutation testing has matured enough that Saab is able to use it for large-scale embedded systems consisting of hundreds of distributed components totalling millions of lines of code. The improvements in how to use mutation testing have significantly helped to reduce the cost of using mutation testing, which is a significant contributor to why mutation testing is now, through the TESTOMAT Project, usable by a wider audience for large-scale systems such as the Gripen E project at Saab. The application of the technique has, at Saab, resulted in a significant overall quality improvement of not only the source code and test suite but also the requirements, test specification and developers' understanding of how their programs work. The ripple effects of the quality improvement have led to less issues being found when performing formal activities as required by standards, which obviously lead to a shorter time to market.”

- Joakim Brännström, software architect for Gripen-E / Saab Group

Another success of the TESTOMAT Project can be viewed in terms of its influence on the software community as a whole. Software is becoming increasingly complex, but widespread dissemination of the project’s results can counter this. Within the project’s duration, the TESTOMAT Project had already released 80 publications with over 146 presentations and demonstrations, created 24 workshops and established 23 university courses on test automation. 22 industry-academia transfers have been reported, as well as 27 new employees within the consortium. A key focus is on long-term promotion through YouTube videos, with 420 subscribers and over 55,600 views. In addition, a book on AI testing has been published by a PhD student who participated in the TESTOMAT Project – and was recruited by Ericsson afterwards – and another book will be released by Springer in the future.

Subsequent large-scale uptake of the project’s innovations will have enormous knock-on effects in society. One example should be increased safety in transportation, as the project clearly demonstrates the importance of safety-critical techniques for even the biggest companies like Saab and Bombardier (now Alstrom); thanks to increased use of automation in simulation testing, many more scenarios have increased in reliability. In addition, the TESTOMAT Project also impacted the automotive industry in Germany, as well as port testing and drone maintenance.

Another aspect is quality of life as society relies on quality software, from automatic maintenance to better patient treatment through healthcare optimisation. Finally, optimised manufacturing and mobility means better resource management and fewer emissions, helping to meet the grand challenge of the energy transition. Through these combined factors, the TESTOMAT Project is likely to touch every corner of our daily lives.

In summary, the TESTOMAT Project has changed the minds of hardcore manual testers in everyday companies and automated testing is now an accepted practice in almost all companies that produce software, making the thought of automation of other aspects, like feeding it into digitalisation, a natural way to move forward. This has paved the way to the ‘AI/generative’ automation era.