ITEA is the Eureka Cluster on software innovation
ITEA is the Eureka Cluster on software innovation

ITEA Impact story

TESTOMAT Project Smart Engineering

Customised test automation, saving time and improving quality

Alt Alt Alt Alt Alt Alt

Impact highlights

  • The TESTOMAT Project's great strength is its customised nature, which guarantees promising results regardless of the level of automation already found within a company. For those using fully manual testing, an average improvement of 60-80% can be gained in their time to market.
  • For already-automated companies, 15-30% fewer faults are predicted thanks to the TESTOMAT Project. These achievements, resulting in fewer bugs in software, are fundamentally important as nowadays people totally rely on and trust, for example, the use of software tools in healthcare, scientific experiments and many more domains.
  • An advantage for the Finnish industry partner Ponsse was that the execution of tests was 25% faster with test automation and it was able to execute tests automatically outside of office hours. Nowadays, Ponsse’s simulators run test automation outside of office hours, saving about 500 hours/week of functional testing work for testers. Alongside, Ponsse has been able to reduce defects that could have ended up in the customer’s product by 20%.
  • Ericsson already carried out completely automatic testing but still saw a 29% improvement in product quality thanks to the TESTOMAT Project.
  • Likewise, Saab increased its number of product users by 440%, leading to higher quality in earlier development phases.
  • Spanish SME Prodevelop managed a 220% increase in test efficiency using to the TESTOMAT Project outcomes.
  • Within the project's duration, the TESTOMAT Project had already established 23 university courses on test automation, 22 industry-academia transfers were reported, as well as 27 new employees within the consortium.
  • A book on AI testing has been published by a PhD student who participated in the TESTOMAT Project – and was recruited by Ericsson afterwards – and another book will be released by Springer in the future.

In software development, reliability and agility often seem like opposing forces. This issue is aggravated by the increasing complexity of software and the constant drive towards faster release cycles for maximum market impact. Manual testing cannot meet these challenges, yet many safety-critical and hardware-oriented companies believed that test automation was irrelevant to them back in 2017. SMEs, meanwhile, missed out due to a lack of awareness or investments in automated agile testing. To strike the perfect balance between quality and speed, a new approach to automation was needed.

The goal of ITEA's TESTOMAT Project, which stands for 'The Next Level of Test Automation' and gathered 34 partners from six countries, was to allow software teams to increase development speed without sacrificing quality. To achieve this goal, the project advanced the state of the art in test automation for software teams, moving towards a more agile and automated development process. The ‘next level’ differs per organisation and depends on factors such as sector, size and practices. Many automation-related variables are unknown to the organisation, so the project focused on providing software teams with personalised roadmaps for improving their automated testing in a cost-efficient manner.

Project results

Through a Test Automation Maturity Survey, completed by 151 respondents in 101 organisations and 25 countries, the consortium of 34 partners from Finland, Germany, the Netherlands, Spain, Sweden and Türkiye was able to gather the status of test automation and key issues worldwide. On this basis, a Test Automation Improvement Model (TAIM) was developed to define measurable steps for improvement.

Success in the TESTOMAT Project was measured in terms of test effectiveness, test optimisation and quality and standards in testing. To demonstrate the project’s versatility, its ten use cases can broadly be grouped into the following categories: trains, planes, telecoms, forestry, machines, robotics, banking, wind turbine maintenance with drones and port integration technologies.


The TESTOMAT Project's great strength is its customised nature, which guarantees promising results regardless of the level of automation already found within a company. Ponsse, one of the world’s largest manufacturers of cutto- length forest machines, renewed its test automation systems during the project and improved its unit testing and requirements coverage by 64%. In addition, it reduced the time needed for testing and saved many hours of testing work for testers. Finally, the time from development to testing with actual machinery has been reduced, which has also reduced the production time as a consequence.

Likewise, Ericsson also significantly lowered development costs and increased the speed of delivery to customers. And through the TESTOMAT Project, the tooling and methodology of mutation testing has matured enough that Saab is able to use it for large-scale embedded systems consisting of hundreds of distributed components totalling millions of lines of code. The application of the technique has, at Saab, resulted in a significant overall quality improvement of not only the source code and test suite but also the requirements, test specification and developers' understanding of how their programs work. The ripple effects of the quality improvement have led to fewer issues being found when performing formal activities as required by standards, leading to a shorter time to market. Another example is Spanish SME Prodevelop. Thanks to the project, Prodevelop has increased the type and number of tests, many of which have been automated, thereby allowing the company to test more features of its application in less time, increasing the quality of its products and reducing the effort spent on testing.

Subsequent large-scale uptake of the project’s innovations will have enormous knock-on effects in society. You can think of increased safety in transportation; thanks to increased use of automation in simulation testing, reliability has increased in many more scenarios. Another aspect is quality of life as society relies on quality software, from automatic maintenance to better patient treatment through healthcare optimisation. Finally, optimised manufacturing and mobility means better resource management and fewer emissions, helping to meet the grand challenge of the energy transition.

Sensing & Control Systems (S&C), a Spanish SME, has integrated the SPEAR results into its proprietary commercial platform, enControl™, extending it from smart homes to the industrial sector and incorporating novel functionalities and services created during project development for the management of ovens and other intensive energy-consuming machineries in related businesses.

In summary, the TESTOMAT Project has changed the minds of hardcore manual testers in everyday companies and automated testing is now an accepted practice in almost all companies that produce software.

Date updated: 27-11-2023

More information

Download TESTOMAT Project Impact story

Related projects



Ericsson (Sweden)
Ponsse Plc (Finland)
Prodevelop (Spain)
Saab AB (Sweden)