XIVT

Modern software-intensive systems are rarely built from the ground up for each customer. Instead, they evolve as variants of an existing product. Over time, this can lead to an overwhelming number of configurations that must be tested. For example, a device or application with just 30 configurable features, such as security options, communication protocols, or hardware add-ons, can theoretically produce over one billion different product versions. Testing all these variants manually or with traditional tools is simply impossible and, at the start of the ITEA project XIVT (eXellence In Variant Testing), no broadly accepted industrial solution existed to handle this complexity efficiently.

The XIVT project, bringing together 21 partners from 5 countries, tackled this challenge head-on. Its goal was to give companies in sectors like telecommunications, railways, automotive, and industrial manufacturing a competitive edge by shortening their feedback cycles and improving development efficiency. To achieve this, XIVT developed 28 specialised methods and tools for testing highly configurable, variant-rich embedded systems.

The project also delivered an integrated, configurable, and extendable toolchain designed to support high-end quality assurance. This enables a shift towards systems that are more adaptable, flexible, and even autonomous.

The end result is a set of cost-effective solutions applicable across five major industries, helping organisations manage complexity, improve reliability, and accelerate innovation.

End-to-end testing platform and toolchain

XIVT’s main technological results include new ways to analyse requirements, understand how they relate to each other, and decide which features or test cases should be prioritised. The project also produced certification guidelines for product families. All these elements are combined in an end-to-end testing platform and toolchain that can integrate tools designed for specific applications.

Regarding specific tools, a number of successes stand out. For example:

• VARA, a reuse recommendation tool, based on natural language processing (NLP) and machine learning developed in the scope of XIVT, that can automatically analyse existing projects and recommend implemented features that can be candidates for reuse to realise new products.
• SEAFOX is the first combinatorial testing tool designed to automatically generate and select efficient test suites for industrial control software.
• ReForm is a new algorithm and tool that helps convert written requirements into a more structured form using NLP and machine learning.
• Recorder Tester improves test coverage for reusable C/C++ code with many build options, using static code analysis and advanced scaling techniques.
• QATS is a tool for the assessment of test suite quality developed at Fraunhofer FOKUS to detect design errors in product models based on mutation operators and domain models.

These solutions have been used to build demonstrators that show that the project’s methods can be applied successfully in real-world settings.

Thanks to the outcomes of the XIVT project, the time needed to create and run test cases can drop by up to 95%.

Proven industry benefits with faster development and higher test coverage

For industrial end-users, the main benefits include more effective product line engineering, which means companies can develop different product versions more efficiently by reusing shared components, and faster development cycles that make it easier to address new customer needs and deliver quicker feedback.

For the four use-cases, strong results have been achieved:

• VARAapplication and evaluation in the development processes of Alstom (formerly Bombardier Transportation) for railway propulsion and control systems have already shown promising results as the manual process of reuse analysis can now be done automatically in a matter of minutes instead of a couple of weeks for complex systems, while still achieving a prediction accuracy of 82%.
• SEAFOX reaches 85% code coverage with combinatorial test suites generated in just a couple of minutes of analysis.
• ReForm dentifies requirement elements with about 90% accuracy and has been successfully applied at Bombardier Transportation in a propulsion control use case, enabling automated derivation of formal requirement models from natural-language specifications and significantly reducing manual effort while improving consistency, traceability, and verification efficiency.
• Recorder Tester improves test coverage for product variants from under 5% to over 80%.
• QATS can reach mutation scores of 83-100% on selected operators and examples. This increases the efficiency of testing large product lines via representative sampling.

Overall, engineers using these solutions can reduce their effort on analysing requirements and prioritising features by 10-40%. The time needed to create and run test cases can drop by up to 95%. The project also developed a methodology that helps companies combine different tools in a coordinated way, making it easier to understand how each tool can support their own processes and helping bridge the gap between technical results and commercial application.

Industrial adoption and market expension

Companies are already integrating the results into real products. For example, WinTrust added the new methods to its InnSpect tool, which now reduces the number of test cases required for complex, multi-variant systems by 30-70%. One client at Cable TV industry, with multi-variant versions of the same core system (set-top-box, smartTV and mobile) experienced a decrease of 40% in testing investment due to reduction of combinations.

Working closely with the project partners across Europe, mobileLIVE from Canada was able to accelerate its innovation, refine its test orchestration frameworks and validate its technology across different sectors, including retail, logistics, telecommunication and the Internet of Things (IoT). As a result of this collaboration, mobileLIVE successfully developed a market-ready enterprise edition of its platform and expanded into new international markets. The company unlocked new revenue streams, created highly-skilled jobs in Canada, and positioned itself at the forefront of intelligent testing solutions for complex, software-driven systems.

In Germany, EKS InTec GmbH and FFT Produktionssysteme GmbH & Co. KG successfully transferred XIVT results into their engineering practice. EKS InTec integrated them into solutions such as RF::CAT and Sketcher 3D, enabling automated testing of PLC step-chain variants and significantly reducing manual effort in virtual commissioning, thereby strengthening its service portfolio and market position in engineering and commissioning projects. FFT Produktionssysteme GmbH & Co. KG applied XIVT tools such as RF::CAT, the XIVT Plant Designer, and feature-model-based configuration to real industrial production systems, enabling automated configuration and testing of robot systems with feature models covering more than 200 components. This allowed errors to be detected much earlier in the engineering process, reduced rework at later project stages, and improved efficiency and competitiveness in customer-specific, variant-rich production projects.

In Sweden, the XIVT project significantly strengthened Addiva’s ability to develop and maintain complex enterprise products for the railway industry. This includes advanced systems such as AddTrack (for diagnostics, troubleshooting, root-cause analysis and AI-driven deviation detection) and ELSA, which provides lifecycle cost analysis, reliability assessments, advanced data processing, and real-time monitoring. Through XIVT, Addiva improved overall product quality by reducing requirement deviations and increasing test coverage by 18%. The project enabled more efficient development workflows and more robust, predictable deliveries. XIVT has therefore been a key driver in enhancing Addiva’s innovation capacity and competitiveness within intelligent, data-driven railway solutions.

Several elements of the toolchain, including plug-in modules, have also been released as open source and are already being used by companies, such as Alstom and ABB.

The project’s communication efforts have been highly successful, with 31 conferences and workshops, 4 journal papers, 10 bachelor's and 15 master's theses, and a series of podcasts, available at https://www.xivt.org/, that have been downloaded more than 2,000 times. This visibility has led to recognition across multiple fields, including an award at the 2021 Requirements Engineering for Software Quality Conference and first place in the Automated Software Testing Industrial Competition 2021.

Follow-up initiatives for wider adoption of next-generation variant testing

To ensure long-term impact, all partners have contributed to standardisation efforts for homologation and certification, resulting in recommendations intended for assessors and safety engineers worldwide.

The XIVT project has proven to be a strong foundation for successful follow-up initiatives with many of the same industrial and research partners, most notably SmartDelta and GENIUS. Building on XIVT results, key tools such as VARA+ and ReForm have been systematically extended to address the growing complexity and variability of modern, version-rich software systems, enabling continuous quality assessment across product variants and evolutions. While SmartDelta focused on advanced analytics for quality-aware change management, GENIUS further elevates these concepts through the cutting-edge application of generative AI and large language models to core software engineering and software quality processes. This continuity of research and partnership has delivered clear business impact: it has reduced manual effort in quality assurance, improved early detection of quality risks, accelerated development and verification cycles, and enabled industrial partners to sustainably manage complexity and innovation in long-living software products.

In short, strong results are expected as the project continues to build an ecosystem that supports the wider adoption of next-generation variant testing.