ATAC

Impressive reduction of test effort

The heterogeneous portfolio of different tools and methodologies has had a remarkable impact for the ATAC project partners. For example, Barco, the project coordinator and

traditionally a hardware-oriented company, was faced with higher costs due to lower quality and higher expenses to test and fix issues in its transition to mixed hardware/software offerings. During the ATAC project, its Collaborative Media Team developed a framework that provided better support for tracing system requirements down to detailed requirements and linking them with corresponding test cases on all levels of the V-model. This framework reduced the software verification and validation effort by 20% and has since been rolled out company-wide thanks to this success. Within the ClickShare product range for example, more product variants can now be tested in a shorter time as the in product-line regression testing has now been largely automated. This has given a boost to ClickShare, the main contributor to the corporate segment, which is a growing part of the Enterprise division representing approximately 30% of Barco sales in 2017.

Bittium Wireless (formerly Elektrobit Wireless Communications) developed an automated VoIP network test tool in ATAC and introduced a production testing platform for a Bittium product family, both of which Bittium has continued to develop after the project’s end in 2014. Today, the automated VoIP network test tool is used in Bittium as a standard testing framework in all VoIP-based products and the company automated more than 90% of its functional testing for VoIP-based products resulting in significant verification cost savings. The production testing platform, introduced as a result of ATAC project, has been deployed across Bittium’s complete product portfolio cutting the required test development effort by 70% for new products. This results in hundreds of thousands of euros in cost savings for every new product compared to the situation prior to ATAC.

Maximatecc (formerly CrossControl), a developer of rugged hardware and software for mobile machinery in construction, mining, cargo, transportation and other industries, collaborated closely with Bombardier and increased the number of active licences by more than 300%, from 30 to over 100. Ken Lindfors, Global Software Platform Manager at Maximatecc explains: “ATAC has made it possible to develop our products and services for the simulation of embedded systems and applications. Especially, ATAC has given us the opportunity to boost and develop our simulation platform SimTecc. We now have more customers using it and in total also a lot more developers directly affecting the number of licenses sold. The flexibility of SimTecc is that it can be applied both at component leve for module testing and at application level for testing complete machines and vehicles, e.g. a complete train. In addition to the development of the SimTecc product itself, ATAC has also generated more application development services for us, where we help customers to design their simulation and test systems and also to integrate SimTecc with their test system.”

The ATAC results helped Bombardier Transportation reduce the verification effort for software parts with a safety impact, which comprise a large part of the systems to be verified, by 80%. The prototype tool developed in ATAC minimised the number of test vectors required to demonstrate sufficient code coverage, massively decreasing the time needed by testers to demonstrate these coverage objectives. In addition, the tool gave a figure on the achieved code coverage by testing when the automatically tool-generated test vectors were used. “The ATAC project is a good example,” says Bombardier consultant, Stig Larsson, “of where active collaboration between the partners through industrial case studies, sharing experiences and best practices really helped generate the fast exploitation initiatives launched during the project, of which Bombardier Transportation has benefited from the transfer and dissemination of the initial research results in Sweden. Overall, I would say that the organisation has been able to raise its standards through its involvement in this project.”

In addition, the ATAC results have enabled Valmet Automation, leading global developer and supplier of technologies, automation and services for the pulp, paper and energy industries to run over 1500 automated tests in less than one hour, where manual testing would take days and be much more error-prone.

On an academic level, Mälardalen University (MDH) successfully bridged the "valley of death" by refining the results from academic test automation research, conducted in a well-controlled academic environment where many day-to-day industry constraints are not considered, into practical results that can be adopted by industry. By going downstream to the actual problems that the partners were encountering, concrete solutions were targeted, and achieved. The ATAC results even led to the creation of a start-up company geared to further exploring the test tooling prototype created in the project. With the help of the MDH University business incubator (IdéLab), Compratio AB was officially launched in December 2015. In 2017, Vinnova granted €30k as part of their Innovative Start-ups Call (phase 1) to further explore usage of the Compratio tool/method in a non-regulated domain and, in 2018, undertook consultancy work at ABB in the area of test analysis and process improvements.

The ATAC project is a good example of where active collaboration really helped generate fast exploitation during the project

Furthermore, ATAC has been instrumental in securing substantial financial support for followup industrial collaboration and research at MDH. In addition, the ATAC project has been a key ingredient in the formation and establishment of the Software Testing Laboratory (STL) research group at MDH, one of the largest of its kind in Sweden. Expanding dissemination, the results of ATAC have been used in several courses at MDH like the successful development of Master-level online courses on industrial software testing for people active in industry within PROMPT, a Swedish educational initiative in cooperation with several academic parties and a number of leading Swedish industrial companies and organisations.

Ericsson and MDH set up a follow-up ITEA project, the TESTOMAT project, which brings together 36 partners from 6 countries, and will ultimately result in a Test Automation Improvement Model that will define key improvement areas in test automation, with the focus on measurable improvement steps. In the ATAC project, Ericsson automated millions of test cases that are executed daily in different test systems, and test code is now delivered together with software code and executed with every change to the system. This can easily save 80% execution time and enables a much faster turnaround in the projects, cutting delivery times by months. However, automating the test case is not enough in itself as the cost of the test is still very high. Therefore, reducing overlapping test cases and being more adaptive in what is tested is important. The TESTOMAT project aims to find new and different approaches in the test suite and better ways to identify how and what to improve in the different areas of test automation.

Finally, Kaunas University of Technology (KTU) further collaborates with Singleton Labs in the area of software test automation research and the use of innovative testing methods in its Software Engineering Master study programme in which several Master theses have been defended. KTU also founded one of the largest mobile application testing laboratories in the Baltic States.

These testimonials clearly highlight that the participants were very successful in translating the technological innovations resulting from the ATAC project into tangible business advantages and results. These technological innovations were facilitated by both national and international collaborations amongst the partners clearly highlighting the importance of international collaboration frameworks like ITEA.