About the project

Concept

In this project, we intend to advance the state-of-the-art on four levels, each requiring solutions to a variety of technical challenges. The four conceptual levels are:

Sensing – This level focuses on the various sensors and sensor combinations needed to properly perform the required human monitoring tasks. It will develop and adopt remote sensors (radar, lidar, camera), and wearables (for physical and mental state, as well as movement) to provide the components to build a multi-sensor monitoring system capable to perform the required monitoring tasks, Sensors may also be combined with actuators to implement interaction with the users.

Multi-sensor intelligence – On this level, the data obtained from multiple sensing units will be combined to provide information on the monitored subjects. It focusses on the data-fusion and decision-making algorithms needed to combine the sensor data streams and provide high-level cues to the next (application) level. As sensor data processing is increasingly done with data-driven ML algorithms, the development will include provisions for (distributed) training as well as near real-time processing. The algorithms can be improved by taking into account novel data obtained during the deployment of the systems. Methods to include this new data for continuous incremental learning avoiding training with malicious data, and self-explainable methods will be studied as well.

Distributed platform – The platform is responsible for providing secure and privacy-preserving communication between the distributed components including the sensors, hubs, edge devices and the cloud. The computations needed for the recognition algorithms will be distributed over the available resources in the platform. The data processing and decision-making pipeline will also be distributed, needing new methods to divide the intelligent algorithm in relocatable chunks. Furthermore, learning in a distributed environment (federated learning) will be facilitated in the platform. The platform will mostly be based on existing frameworks but adapted and enhanced to enable the distributed intelligence.

Human health & safety assessment – The human monitoring functionality enables a variety of applications, which in itself will go beyond the state-of-the-art. In all cases, information on the person will be collected to form a comprehensive model including – as needed – the physical state, mental state, position, trajectory and intentions. A history can also be maintained allowing for detecting trends and improving predictions. This “digital twin” will be the basis for proactive functionality for the various domains. In health, the focus will be on ensuring the continuous monitoring with full coverage of visited areas, while in automotive and robotics domains the focus will be on combining sensors to improve human positioning, trajectory projection, comprehending intentions as well as monitoring mental state.

Technical focus

The DistriMuSe project faces several technical challenges, and with our work during the course of the project, we intend to advance the state-of-the-art in the following topics:

Seamless spatial and temporal coverage. Deploying multiple sensors will allow for continuous monitoring over wide areas. Spatial coverage will be achieved by a distributed set of sensors, each covering parts of the scene, for example to monitor elderly in multiple rooms or to reveal pedestrians in traffic who may otherwise remain unnoticed. Temporal coverage may involve several different sensors each optimised to monitor the person during a particular activity (e.g., sleeping, exercise), but yielding similar physiological parameters seamlessly contributing to continuous monitoring. Furthermore, accuracy in difficult and varying observational conditions can be improved by combining multiple sensing modalities complementing each other in e.g., changing weather or lighting conditions.

Distribution of processing resources. Operating a system of multiple sensors or sensor systems, distributed over space and time, requires innovative methods, platforms and tools. Furthermore, the processing and decision-making based on the vast amount of real-time data produced by the sensors will demand balanced strategies to split the intelligent algorithmic computations over the available resources, from the edge to the cloud. The DistriMuSe project will also tackle additional complexity issues introduced when sensor modules are added to, or removed from, the ensemble ad hoc during operation, and accommodate sensing quality degradation (soft-fail), for example due to poor observational conditions.

Multi-sensor fusion. Available sensor data can be fused to obtain the required information on human activities, intentions, and health, or provide necessary information on the environment and positioning of objects (like cars and robots) in the co-inhabited space. This sensing and fusion process is by no means trivial, and needs development on the sensors, their data processing, and the fusion algorithms. ML (Machine Learning)-based methods can be used to provide the necessary information and assist the decision-making process.

Trustable and transparent machine learning. Processing complex sensor information with Machine Learning algorithms which often must make decisions involving humans based on their interpretation of the observational data inevitably leads to the issue of the explainability and reproducibility of these results. In DistriMuSe, we will improve trust in ML-based decision-making by favouring explainable AI methods and providing tools to monitor data and model quality and integrity.

Privacy, reliability and security. The frameworks envisioned for DistriMuSe will process sensitive personal data obtained from monitoring and perform real-time decision-making concerning people’s health and safety. Therefore, it is of paramount importance that these systems be secure, reliable and resilient in all circumstances. Privacy preserving as well as data security mechanisms must be in place, and the framework must cope with partial failure of its components. Monitoring of people raises ethics concerns which need to be addressed carefully for each application, seeking benefit for the stakeholders while avoiding privacy invasion.

Objectives and results

This project will research the following objectives and produce associated results to facilitate the development and adoption of distributed multi-sensor systems for human health and safety:

Optimised innovative sensors and sensor systems for advanced human observation, measurement, and interaction – As a result, the project will produce advanced remote sensors (e.g. radar, lidar) and wearable sensors (e.g. blood pressure, sweat, motion) for the monitoring of physiological signals, behaviour and positioning.
ML-based data analytics, fusion and decision-making methodologies for (semi) real-time multi-sensor information processing – The result will be multi-sensor systems providing temporal and spatial continuity, as well as improved accuracy and robustness through multimodal fusion. Resulting ML models will support transparency and distributed deployment, some supporting transfer and federated learning practices.
Supporting distribution of computation algorithms across available resources from edge to cloud – This will result in methodologies supporting the distribution of ML pipelines over available resources and design for distributed and continuous learning. Platforms will be built that support secure and privacy preserving operation while allowing for transparent communication, distribution of computations and near real-time execution.
Building a comprehensive understanding of human physical health, mental state, behaviour, intentions, and safety risks – This will produce proof of concept applications utilising the digital twin paradigm to capture the mental and physical state of the monitored persons, as well as their position, trajectory and intention and inferred level of health or safety.
Validating the technologies in human-centric use cases related to health and wellbeing monitoring and support; ensuring traffic safety especially for VRUs; safe interaction with robots and automated systems – A number of demonstrators will be built and evaluated for the health and wellbeing, mobility and robotics use cases, allowing assessment of feasibility, performance, user acceptance and use case specific requirements. Pilots will be performed both in laboratory environments, as well as real life settings with human subjects.

In health and wellbeing, we will develop sensor systems which permit unobtrusive monitoring of a set of selected physiological parameters in a reliable and inconspicuous way, with minimum disturbance to the individual’s daily routines. In healthcare, 24/7 monitoring of both healthy individuals or patients suffering from cognitive issues will provide a plethora of data usable for detecting health risks, improving and optimising disease management, and providing proactive and preventive care. Achieving optimal spatial and temporal monitoring coverage will require a combination of wearables and sensor strategically placed in the environment, and the seamless operation and handover between coexisting systems (e.g., from a bed sensor to a wearable sensor in the sole of the foot to a sensor in the environment). This envisioned continuous monitoring functionality will help to provide in-place care, allowing elderly people to stay and/or recover at their own homes if desired, alleviating the burden on care facilities and increasing the sense of independence and well-being of users. A reduction on health-care costs and an increase of healthy life years of Europe’s ageing population is to be expected from our developments in this area.

Mobility: Future traffic will feature a mix of participants with very different levels of autonomy and abilities to sense and communicate. While cars increase their support of automated driving, relying on an ever-growing number of onboard sensors (to which future generations will add communication and cooperation with other automated surrounding vehicles), vulnerable road users (VRUs) such as pedestrians and cyclists, are typically not equipped with sensors and communication means. It is our intention to focus on sensor devices which reliably detect the presence, location, and intention of these road users, and transmit this information to the multi-sensor environment formed by the collaborating vehicles and road-side sensor units, so VRUs can be better protected. Meanwhile also driver monitoring will still be necessary to ensure safe traffic, assessing their ability to operate the vehicle at all times, to ensure they can take over control in those delicate situations which automated systems cannot handle yet. Additionally, driver monitoring systems can help to detect whether drivers are missing out on crucial traffic situations (e.g., unnoticed VRUs) and direct their attention to them.

Robotics: In some ways, the factory floor use case poses a similar situation to the traffic scenario just described, being environments shared by robots and human workers, and where potentially dangerous interactions between them can occur. Current safety measurements consist mostly in fencing off robots, or stopping their operation altogether if human presence is detected nearby. Future collaborative environments will consist of robots working along with humans in shared areas, being conscious of the presence, location and intentions of humans, and adjusting their movements and speed to prevent harm. Robots will thus need to be equipped with multiple sensors continuously scanning the environment and fusing algorithms to provide a reliable world view on humans in the environment and their safety perimeter.

Market and impact

Each of these application domains has their own value chain including European players, who can significantly improve their business potential by means of the innovations developed in this project. Thus, the potential economic impact of the DistriMuSe project consists both of significant savings on societal services, as well as growth of related businesses.

In healthcare, the ageing population in Europe requires a paradigm shift from reactive to proactive diagnosis and treatments, which will be greatly facilitated by technology which supports monitoring at home. Regular patient monitoring will help to detect health issues early, call medical assistance or plan rehabilitation, making better use of the scarce resources in healthcare, and saving the patients from unnecessary displacements to doctor offices. We expect that the market of home monitoring solutions will expand rapidly in the near future, aiming to facilitate ageing-in-place. Empowerment of individuals to take care of their own health will also be targeted by sleep monitoring systems for use at home. They analyse the quality of sleep and identify any sleep-related diseases before they affect an individual’s daily living quality or work efficiency. The cost of current laboratory-based sleep studies is high, leaving sleep-related problems undiagnosed and causing the loss of billions of euros in decreased work efficiency and sick leaves. While home-based sleep monitoring solutions exist already, based on wearables and bed-sensor (under mattress) solutions, none can claim clinical accuracy, so the solutions proposed in our project would open a new market segment. Similar solutions apply to gait analysis for early detection of cognitive deterioration issues.

Mobility: Driver monitoring systems (DMS) will be required in all new vehicles in Europe from 2024 onward, and advanced driver monitoring solutions will be a differentiator in the market, with car companies likely to invest heavily in this area in the near future. But the greatest benefit is to be expected from safer traffic which lead directly to saving lives and reducing healthcare costs. While drivers will be better protected than ever, VRUs such as pedestrians and bikers remain at the mercy of vigilant drivers and increasingly automated vehicles. Therefore, it is to be expected that the next focus on improving traffic safety will be on integrating DMS with Advanced Driver Assistance Systems (ADAS) and automated driving functions to properly detect dangerous situations and collaborate with other sensing systems (by means of vehicle communication with the environment – V2X) to protect also VRUs from harm. The market will soon follow this trend.

Robotics: On the factory floor, automation has long ago taken over human labour for the simpler, most repetitive tasks, with humans remaining at a safe distance from robots to avoid injury. The next breakthrough will be intelligent collaborating robots (so-called “cobots”), which interact naturally with human workers sharing the same space, and permit tackling more complex tasks. In other domains this trend is already visible, for example in office work, where intelligent assistants such as ChatGPT perform tasks previously done by human workers only. The increase in efficiency will be an important driver for investments in this area, and the market is expected to expand accordingly in the coming years.

Solving the challenges of current monitoring solutions and realising distributed multi-sensor solutions to monitor people’s health, location and intentions will permit the DistriMuSe project to create safer environments for people in traffic and in the factory, as well as contribute to improve the health and quality of life of the general population in the comfort of their own homes.

Concept

Technical focus

Conceptual architecture

Objectives and results

Domains

Market and impact