ITEA Impact story
ATAC
Developing automated testing for complex real-time systems
Impact highlights
- The ATAC results enabled Barco to test more product variants in a shorter time as the in product-line regression testing has now been largely automated. This has given amongst others a boost to the ClickShare product portfolio, the main contributor to the Corporate segment, which is a growing part of the Enterprise division representing approximately 30% of Barco sales in 2017.
- The production testing platform introduced as a result of the ATAC project has been deployed across Bittium’s complete product portfolio cutting the required test development effort by 70% for new products. This results in hundreds of k€ in cost savings for every new product compared to the situation prior to ATAC.
- ATAC has given Maximatecc the opportunity to boost and develop their simulation platform SimTec. They now have more customers using it and many more developers in total directly affecting the number of licences sold. The number of active licences increased by more than 300%, from 30 to over 100.
- The ATAC results helped Bombardier Transportation reduce by 80% the verification effort for software parts with a safety impact, which comprise a large part of the systems to be verified. Overall, the organisation has been able to raise its standards through its involvement in this project.
- Test automation realised in ATAC enabled Ericsson to easily save 80% execution time and to reach a much faster turnaround in projects, cutting delivery times by months.
- The ATAC results led to the creation of a start-up company geared to further exploring the test tooling prototype created in the project. With the help of the Mälardalen University business incubator (IdéLab), Compratio AB was officially launched in December 2015.
Ever more functionality is demanded by end-customers from the software-intensive systems they use. At the
same time their expectations with respect to the correct operation, safety and security of these systems
have become higher than ever. Severe system failures can lead to significant damage or even loss of life,
while successful cyber intrusions can destroy their reputation. As such, breaches in both safety and
security can have a significant adverse impact on business and reputation. Due to the dramatic increase in
the complexity of the software itself, the intricate interaction modes between the software and the external
world, and the sheer magnitude of the customisability of the software, these systems have become
increasingly difficult to develop and verify by traditional development processes and testing methods. The
ATAC project took on the gauntlet to resolve such challenges by researching, evaluating and rolling out a
number of methodologies, associated processes and tools to efficiently and automatically verify complex and
highly configurable software-intensive systems.
Project results
ATAC’s overarching goal was always to push the functional, safety and security requirements coverage envelope while minimising the number of test cases required to achieve the necessary degree of coverage and confidence in the software, hence greatly reducing the required testing effort for both the fully automated and remaining manual test cases. To ensure that the researched solutions and tools solve real industry needs and to maximise their potential application in other settings, all R&I activities were anchored around a jointly defined set of industrial case studies in different application domains. During the project, a flexible and heterogeneous portfolio of different tools and methodologies was developed to cover the needs of these different application domains as it was soon recognised that one-size-fits all approach would be less than optimal.
Exploitation
During ATAC, Barco’s Collaborative Media Team developed a framework that provided better support for tracing system requirements down to detailed requirements, linking them with corresponding test-cases on all levels of the V-model. This framework reduced the software verification and validation effort by 20% and has since been rolled out company-wide thanks to this success.
Bittium Wireless developed an automated VoIP network test tool in ATAC and introduced a production testing platform for a Bittium product family, both of which Bittium has continued to develop after the project’s end in 2014. Today, the automated VoIP network test tool is used in Bittium as a standard testing framework in all VoIP based products and the company automated more than 90% of its functional testing for VoIP based products resulting in significant verification cost savings.
Bombardier’s prototype tool developed in ATAC minimised the number of test vectors required to demonstrate sufficient code coverage, massively decreasing the time needed by testers to demonstrate these coverage objectives. In addition, the tool gave a figure on the achieved code coverage by testing when the automatically tool-generated test vectors were used. Maximatecc collaborated closely with Bombardier and ATAC has made it possible to develop their products and services for the simulation of embedded systems and applications, especially their SimTecc simulation platform. The flexibility of SimTecc is that it can be applied both at component level for module testing and at application level for testing complete machines and vehicles, e.g. a complete train.
The ATAC results have enabled Valmet Automation to run over 1500 automated tests in less than one hour, where manual testing would take days and be much more error-prone. In ATAC, Ericsson automated millions of test cases that are executed daily in different test systems, and test code is now delivered together with software code and executed with every change to the system.
ATAC has been a key ingredient in the formation and establishment of the Software Testing Laboratory (STL) research group at Mälardalen University, one of the largest of its kind in Sweden. Kaunas University of Technology founded one of the largest mobile application testing laboratories in the Baltic States.
More information
Download ATAC impact storyRelated projects
ATACOrganisations
ALSTOM Rail Sweden AB (Sweden)Barco N.V. (Belgium)
CrossControl (Sweden)
Ericsson (Sweden)
Kaunas University of Technology (Lithuania)
Mälardalen University (Sweden)