TESTOMAT Project

Project results

Through a Test Automation Maturity Survey, completed by 151 respondents in 101 organisations and 25 countries, the consortium of 34 partners from Finland, Germany, the Netherlands, Spain, Sweden and Türkiye was able to gather the status of test automation and key issues worldwide. On this basis, a Test Automation Improvement Model (TAIM) was developed to define measurable steps for improvement.

Success in the TESTOMAT Project was measured in terms of test effectiveness, test optimisation and quality and standards in testing. To demonstrate the project’s versatility, its ten use cases can broadly be grouped into the following categories: trains, planes, telecoms, forestry, machines, robotics, banking, wind turbine maintenance with drones and port integration technologies.

Exploitation

The TESTOMAT Project's great strength is its customised nature, which guarantees promising results regardless of the level of automation already found within a company. Ponsse, one of the world’s largest manufacturers of cutto- length forest machines, renewed its test automation systems during the project and improved its unit testing and requirements coverage by 64%. In addition, it reduced the time needed for testing and saved many hours of testing work for testers. Finally, the time from development to testing with actual machinery has been reduced, which has also reduced the production time as a consequence.

Likewise, Ericsson also significantly lowered development costs and increased the speed of delivery to customers. And through the TESTOMAT Project, the tooling and methodology of mutation testing has matured enough that Saab is able to use it for large-scale embedded systems consisting of hundreds of distributed components totalling millions of lines of code. The application of the technique has, at Saab, resulted in a significant overall quality improvement of not only the source code and test suite but also the requirements, test specification and developers' understanding of how their programs work. The ripple effects of the quality improvement have led to fewer issues being found when performing formal activities as required by standards, leading to a shorter time to market. Another example is Spanish SME Prodevelop. Thanks to the project, Prodevelop has increased the type and number of tests, many of which have been automated, thereby allowing the company to test more features of its application in less time, increasing the quality of its products and reducing the effort spent on testing.

Subsequent large-scale uptake of the project’s innovations will have enormous knock-on effects in society. You can think of increased safety in transportation; thanks to increased use of automation in simulation testing, reliability has increased in many more scenarios. Another aspect is quality of life as society relies on quality software, from automatic maintenance to better patient treatment through healthcare optimisation. Finally, optimised manufacturing and mobility means better resource management and fewer emissions, helping to meet the grand challenge of the energy transition.

Sensing & Control Systems (S&C), a Spanish SME, has integrated the SPEAR results into its proprietary commercial platform, enControl™, extending it from smart homes to the industrial sector and incorporating novel functionalities and services created during project development for the management of ovens and other intensive energy-consuming machineries in related businesses.

In summary, the TESTOMAT Project has changed the minds of hardcore manual testers in everyday companies and automated testing is now an accepted practice in almost all companies that produce software.