High-quality, fast automated testing
ATAC Innovation report
The increasing risk of software defects remaining undetected in software-intensive systems and causing severe system failures must be mitigated, with quality assurance being performed earlier, more frequently, and in a more automated fashion than in traditional development processes. The ITEA 2 ATAC project addressed the problem of the automated testing of complex and highly configurable software-intensive systems with the aim of enabling European industry to maintain its leading role in engineering in this area.
The ATAC consortium of 15 partners from industry and research jointly defined a set of industrial case studies in different application domains as the driver for the development of methods and tools geared to creating solutions for the automated testing of complex, highly configurable software-intensive systems. The case studies allowed these methods and tools to be continuously evaluated so that the applicability and effectiveness of the solutions in a complex, industrial environment could be ensured. To safeguard that the impact on research and industry was high, the project results were packaged into a set of expressive demonstrators for use, for instance, at international symposia and industry exhibitions. Throughout the project, the support provided by the Public Authorities was vital. Their support made the difference since it enabled the companies to work on a topic, which is not really at the core of their activities.
Improved methods and tools for test automation
The development of automated testing for the effective and efficient quality assurance of complex and highly configurable systems re-used existing mature techniques and provided a systematic and tool-supported quality assurance process. One of the focal areas was the development of domain-specific languages (DSL) to support high-level modelling of test behaviour and test configurations. Other focal areas included better evaluation of and improvement to existing test suites and their quality as well as testing in the context of complex systems development where components are provided by multiple parties in parallel and need to be continuously integrated. In all, twelve tools were developed, from commercial and open source to a freely available automated testing framework.
Since many of the partners in the ATAC consortium developed building blocks for the verification and validation of software intensive systems, a major goal was to improve the interoperability of testing tools and enable the forming of tool chains to meet the needs of the industry. The integration of the tools developed in the project into the environments and processes of the case study providers enabled the techniques and tools to be transferred into practical application. The dissemination of the project results, both within the partners organisation and through external channels will lead to increased productivity, reduction in costs and work needed for testing and maintenance, and faster time-to-market with better quality of software.
Transfer of information
One of the major goals of ATAC concerned the collection and investigation of test automation techniques and methods, best practices, tooling, modelling guidelines, domain specific languages for testing and testing patterns. The results are available in an online repository in the form of tool evaluation reports and scientific papers and contain the results and measurements of their application in the industrial case studies from a large variety of domains. The publicly available and well published content of the repository will make the adoption of test automation easier for the European software industry and is available for anyone interested in testing complex and highly-configurable systems. Furthermore, a book will be written, led by Ericsson and VTT, about the results of this project, and will include guidelines for other companies.
Achievements
The approach taken by this ITEA 2 project was quite remarkable in that rather than look for a homogenous solution – one size to fit all, as it were – the recognition of the fact that the different sizes (industry needs) made it evident that a heterogeneous solution would be required. So this new and innovative type of research approach can be seen as state-of-the-art: no one single solution, but different guidelines and tools adaptable to specific needs and markets. One of the key objectives of ATAC was to bridge the gap between the results of test automation achieved in the well controlled academic environment where a lot of the day-to-day industry constraints do not apply and potential application in the industry. By going downstream to the actual problems that the partners were encountering, concrete solutions were targeted, and achieved.
The knowledge transfer between academia and industry was an important achievement and the interaction between the two worlds in a kind of symbiotic relationship throughout the project enabled the rapid and effective transfer of information. A real highlight of the iterative approach in this relationship can be found in the substantial number and quality of university Master and PhD programmes that were quick to incorporate the work of the project while all the industrial partners realised an improvement in their level of test automation and test quality due to the input from their academic counterparts. In addition, the number of scientific publications was astoundingly high.
The partners have benefited in a variety of ways from the results of the project. The list is extensive but a few of the examples of the concrete results achieved in this highly downstream project approach can be cited here. Barco, the project coordinator and traditionally a hardware oriented company was hampered by the cost of poor quality in its transition to a mixed hardware/software enterprise. It was able to use the results to develop a companywide common framework that has generated a 20% reduction in software verification and validation effort. Elektrobit reduced the costs of VoIP network functional testing by 30% and through the introduction of a production testing platform within different EB Wireless Business Areas, managed a 40% reduction of investments for production testing costs per new product. Maximatecc, a developer of rugged hardware and software for mobile machinery in construction, mining, cargo, transportation and other industries, collaborated closely with Bombardier and increased the number of active licences from 30 to over 100 (>300%).
This impressive list of achievements, which includes the defining and processing of eleven use cases with requirements, gap-analysis and validation along with the preparation of the many new tools for test automation, will make a fundamental contribution to strengthening European industry and give a substantial thrust of innovation and quality assurance within the associated industries. ATAC has indeed got test automation well and truly on the agenda of European industry.