FAMILIAR project enables real-time hand gesture recognition in augmented reality

The ITEA project FAMILIAR project has successfully integrated its core technologies of XR (cross reality) and FedML (federated machine learning), marking a significant step in integrating cutting-edge technologies into real-world applications. Through this technology composition, interaction within cyber-physical environments can be enhanced.

The recently achieved milestone of the project is the successful development of its Minimum Viable Demonstrator (MVD). This demonstrator showcases a real-time hand gesture recognition application within an XR-environment. The MVD serves as proof of concept for the project’s core technology, demonstrating its ability to enhance user interaction and real-time data processing in AR applications. Utilising FedML-based image data processing, the system's nodes collaboratively improve a global machine learning model utilising swarm intelligence. This results in a learning effectiveness higher than for stand-alone models across the federated network while ensuring data privacy.

The impact of the federated architecture on system performance and effectiveness in machine learning-powered cross-reality applications comes from swarm intelligence. It reduces the amount of pre-training needed as the network size increases. For small networks, the effectiveness improves quickly, but for large networks, the gain levels off due to overhead and redundancy. When applied, the FAMILIAR system boosts the effectiveness of machine learning-powered cross-reality applications with zero-trust principles, enabling better cross-application learning that scales with market size.

This breakthrough has significant implications for industries relying on precise human-machine interaction. Real-time gesture recognition in AR can streamline workflows, improve efficiency, and enhance user experiences in fields such as manufacturing, automotive, and engineering. By applying privacy-preserving federated learning techniques, the project also addresses data security concerns, making its technology applicable to sensitive and high-stakes environments.

Building on this success, the FAMILIAR project now shifts focus to industry-specific implementations. These include an immersive driving data simulation add-on, setup assistance for 3D printing industrial robots, and extended failure prediction in FEM (Finite Element Method) simulation software. By applying its adaptable core technology to these real-world challenges, the project aims to further validate its approaches and demonstrate their impact across multiple sectors.

More information: