The overall conception of the CRC EmpkinS aims to establish the field of empatho-kinaesthetic sensor technology, to research its scientific basis and to expand this area as core expertise and flagship of the medical technology research location Erlangen in terms of breadth, excellence and visibility. Specifically, this takes place in three sub-goals:
- The aim of the CRC is to provide sensor technologies and completely new qualities and quantities of human motion data. Based on these data, we expect from the research in EmpkinS groundbreaking findings in the field of biomechanical, neuromotor, psychomotor and (patho-) physiological models and their interaction mechanisms.
- EmpkinS has set itself the long-term goal of using innovative sensor / medical technology to open up new diagnostic and therapeutic options for medicine and psychology. The CRC thus has a clear sensor and medical technology focus, which is weighted more heavily, especially in the first few years of the twelve-year overall concept. With the increasing degree of maturity of the researched EmpkinS technologies, the field of medicine and psychology becomes increasingly important. In funding period 1, the research focus is on primary sensor technology, model development, analysis methods and the fundamental proof of the functionality of the EmpkinS approach. Funding period 2 will then be dedicated to the miniaturization and improved integration of sensors, among other things with the aim of improving their mobility and ubiquitous applicability. When creating models, neurological aspects and more complex clinical pictures are now being considered more intensively. Funding period 3 should also include clinical testing in the home area of patients and further developments of sensor technology in order to finally and sustainably implement the EmpkinS vision.
- In addition, it is our goal to work on the ethical and social issues for research on EmpkinS and its future applications, as well as to develop specific orientation markers for future social design (governance strategy).
Project Area A - Contactless sensors for recording motion parameters and vital functions
Project area A researches different wave and radio-based sensor technologies for the remote detection of motion parameters of the human body. The targeted new sensor concepts achieve a precision, measuring rate, dynamics and fine-grained resolution in all dimensions (6D pose, 3D body surface and 3D speed with the respective time course of all values) that are at least one order of magnitude better than the current state of the art are.
Coherently phase-sensitive, wave-based sensors such as radar, laser or coherent radio location systems are particularly suitable for measuring both macroscopic and microscopic movement processes remotely with maximum precision, as they are able to evaluate Doppler and micro-Doppler signal phases.
The EmpkinS sensor technologies are selected on the basis that as many different motion parameters as possible can be recorded with the highest possible quality. Since a single sensor cannot optimally record all conceivable parameters equally, complementary sensor technologies are being researched. They cover different types of detection areas (e.g. the entire body shell (sub project A01) or areas of interest such as the face, neck or arm, leg or chest area (A03, A04). Moreover different orders of magnitude of movement (e.g. movements of the limbs (A02) or microfasciculations (A05) e.g. on the face) and / or different areas of application (e.g. measuring vital parameters or the mobility of limbs) are recorded. Depending on the required input variables of the biomechanical neuro- and psychomotor models or depending on the diagnostic question, the different sensor technologies are used individually or in combination.
A01 Multimodal 3D Body Imaging Camera
A02 Holographic 6D Wireless Locating and Motion Tracking
A03 Highly Integrated Localizable Wireless EMG Transponder
A04 Microwave Interferometer for Cardiovascular and Respiratory Monitoring
A05 Electro-optic Microstructure- and Micromotion-Sensor
Project Area B - Sensor signal / data processing and transfer
Project Area B researches innovative methods and solutions for efficient sensor signal processing and transfer. All parts of the processing chain such as the sampling of the high-dimensional, continuous, physical signals, the efficient source coding of the scanned sensor values, the energy-saving mobile sensor data transfer and the visualization are addressed.
Specifically, the compressive sampling of 6D radar data is being researched in sub project B03 in order to shorten the recording time of empatho-kinaesthetic radar sensors. For this purpose, the common sparseness of the location and motion data is used and the radar test signals are optimized and adapted with regard to the compressive scanning. In B01 an efficient source coding method for multimodal body shell camera data is explored, which, similar to the mp3 format for audio data, enables efficient compressed transmission and storage of the human body shell data including their motion. For this purpose, the spatial correlation of the depth information and the temporal coherence of the movement should be used in order to reduce the data volume by at least two orders of magnitude. Innovative protocols and algorithms for energy-efficient reference signal and electromyography (EMG) sensor data transmission are researched in B02. In order to meet the application-related, stringent requirements for energy efficiency, the joint design of reference and data signals as well as the integration of concepts for local energy generation are developed. Finally, B04 explores the visualization of motion sequences based on a biomechanical model. Derived states of the biomechanical model make the sensor signals easier to interpret for medical professionals, whereby visualizations should also be generated for patients that support communication. Important research aspects are the spatial registration and deformation of the multimodal sensor data and the biomechanical model as well as the integrated representation of these components.
B01 Coding of Multimodal Body Hull Data
B02 Protocols and Algorithms for Energy-efficient Reference Signal and EMG Sensor Data Transmission Integrating Local Energy Harvesting
B03 Compressive Sensing for Empatho-kinaesthetic Radar Sensors
First, a synthetic data model is created. In cooperation with the sub projects A01, A04, D04 andD05, this model will be validated and adapted, and replaced by real measurement data as soon as they are available. Based on this, various methods for improving compressive sensing will be researched in order to reduce the measurement time of image-based radar technology in EmpkinS: distributed compressive sensing, adaptive compressive sensing and radar test signals optimised for compressive sensing.
B04 Visualization of Motion Sequences Based on a Biomechanical Model
In sub project B04, innovative methods for the visualisation of movement sequences that were recorded with various empathokinesthetic sensors and/or for which a biomechanical simulation was carried out are being researched. The focus is on visualisations and the extraction of features that support medical interpretation and diagnosis.
Project Area C - Biomechanical modeling and condition monitoring
Project Area C explores approaches to processing, modeling and interpretation of biomechanical data. This is an integral part of EmpkinS in order to obtain meaningful movement information on the basis of the large number of available measurement data, which originate from innovative measurement principles and therefore entail completely new requirements, and thus to be able to carry out further analyzes.
In detail, musculoskeletal human models are improved in C01 by personalization using machine learning approaches. This will help to distinguish individual differences in motion characteristics from those that are caused, for example, by neurological or psychological conditions in general. In C02, musculoskeletal human models are used to clean up / filter the measurement data by relating them to the kinematic / dynamic motion possibilities of the human musculoskeletal system. The aim of C03 is then to use the (improved) models and (adjusted) measurement data to investigate a new model of postural control of walking. This model uses measurements of the empatho-kinaesthetic sensory system and a sensorimotor-enhanced musculoskeletal human model to characterize the components of dynamic balance control and to advance research into balance regulation. Finally, in C04, the analysis and prediction of biomechanical simulation models are improved by integrating empatho-kinaesthetic sensor data. This is achieved by integrating measurement data (position, orientation, speed, acceleration, force, muscle activity) into the mathematical formulation of a motion process as an optimal control problem. The selection of the model approaches is based on the requirement to research and provide the highest possible quality analysis options for the consideration of EmpkinS.
C01 Machine Learning for Personalization of Musculoskeletal Models, Movement Analysis, and Movement Predictions
The extent to which a neural network can be used to effectively personalise gait simulations using motion data is explored. We first investigate the influence of body parameters on gait simulation. An initial version of the personalisation is trained with simulated motion data, since ground truth data is known for this purpose. We then explore gradient-free methods to fit the network for experimental motion data. The resulting network is validated with magnetic resonance imaging, electromyography and intra-body variables.
C02 Filtering of Multimodal Motion Capture Data through Individualized Musculoskeletal Human Models
C03 Investigation of Postural Control using Sensomotorically Extended Musculoskeletal Human Models
A novel postural control model of walking is explored to characterise the components of dynamic balance control. For this purpose, clinically annotated gait movements are used as input data and muscle actuated multi-body models are extended by a sensorimotor level. Neuromotor and control model parameters of (patho-)physiological movement are identified with the help of machine learning methods. Technical and clinical validation of the models will be performed. New EmpkinS measurement techniques are to be transferred to the developed models as soon as possible.
C04 Analysis of Degenerative Motion Impairments through Integration of Empatho-kinaesthetic Sensor Data in Biomechanical Human Models
The focus of the work programme is on the best possible integration of empathokinesthetic sensor data into biomechanical models. Specifically, degenerative movement restrictions of the hand are recorded by EmpkinS and reference sensor technology and the data are optimally integrated into the mathematical formulation of the optimal control problem depending on data type, measurement frequency and fuzziness, etc. The aim of the project is to develop a model of the degenerative hand movement. Objective biomarkers of healthy or impaired movement function are identified through movement tracking and prediction.
Project Area D - Physiological and behavioural modeling and condition monitoring
In project area D, sensor technologies and empatho-kinaesthetic procedures are researched on various diseases and internal states. Important aspects in this project area are the research into medical and psychological body function models for the transformation of the sensor-based EmpkinS measured variables into clinically relevant parameters and the research into new forms of diagnosis and therapy that are based on these parameters and the EmpkinS body function models. The overriding goal is to assign a medical or psychological relevance to the sensory parameters. In order to achieve this, the sub projects focus on different (patho-) physiological states. These states are characteristic of the disease models examined and include prototypically disturbed body functions, whereby the applicability of the different methods is to be demonstrated.
The aspects considered in the project area are limited, fine motor hand function in rheumatic diseases (D01), facial expressions, posture and movement in depressed patients (D02) and, in the case of D03, stress-associated, pathophysiological changes in the skin, cardiopulmonary function, facial expressions and in general body movement as well as the detection of microfasciculation as a stress reaction. Furthermore, gross motor skills and cardiovascular changes are addressed in D04 as a measure of sleep and movement disorders in Parkinson’s disease, whileD05 is devoted to the investigation of motor skills and cardiopulmonary function for the monitoring of palliative patients.
In this way, the perspective transferability of the EmpkinS methodology is evaluated using a broad spectrum of body functions selected as an example: from fine motor skills to gross motor skills and cardiovascular function to the evaluation of psychological functions such as depression and stress reactions. At the same time, different areas of application of medicine are being researched, from diagnostics (D01, D03–D05) to intervention (D02, D04) to prognosis, care and supply support (D04, D05). In the sub projects it is demonstrated that the empatho-kinaesthetic parameters can depict functional disorders and are therefore a measure of (patho-) physiological processes.
D01 Hand Motion Patterns Derived from Empatho-kinaesthetic Sensor Data as a Diagnostic Parameter for Disease Activity in Patients with Rheumatic Diseases
Under standardised conditions, a comprehensive data set on the current disease status of patients (N = 150) with rheumatoid arthritis (RA) and psoriatic arthritis (PsA) will be collected in D01 in combination with a comprehensive clinical test battery of hand function as well as comparative data from a healthy cohort (N = 75). The parallel acquisition of hand function using state-of-the-art motion capture sensor technology provides a critical basis for the experimental evaluation and integrated data interpretation of the novel sensor data (sub projects A01, A02 and A03) acquired during the acquisition of hand function using EmpkinS.
D02 Empatho-kinaesthetic Sensor Technology for Biofeedback in Depressed Patients
The aim of the D02 project is the establishment of empathokinesthetic sensor technology and methods of machine learning as a means for the automatic detection and modification of depression-associated facial expressions, posture, and movement. The aim is to clarify to what extent, with the help of kinesthetic-related modifications influence depressogenic information processing and/or depressive symptoms. First, we will record facial expressions, body posture, and movement relevant to depression with the help of currently available technologies (e.g., RGB and depth cameras, wired EMG, established emotion recognition software) and use them as input parameters for new machine learning models to automatically detect depression-associated affect expressions. Secondly, a fully automated biofeedback paradigm is to be implemented and validated using the project results available up to that point. More ways of real-time feedback of depression-relevant kinaesthesia are investigated. Thirdly, we will research possibilities of mobile use of the biofeedback approach developed up to then.
D03 Contact-free Measurement of Stress, its Determinants and Consequences
The aim of this sub project is to make stress measurable in a contact-free way using empathokinesthetic sensor technology. Due to its effects on human health and performance, research into stress has a high priority, but is hampered by costly and mostly invasive traditional recording methods. In D03, empathokinesthetic sensory modalities are being researched in laboratory experimentally induced, acute stress situations, which make it possible to make stress measurable through contactless recording of macro- and micro-movements in collaboration with A02, A04 and A05.
D04 Sensor-based Movement and Sleep Analysis in Parkinson’s Disease
In D04, innovative, non-contact EmpkinS sensor technology using machine learning algorithms and multimodal reference diagnostics is evaluated using the example of Parkinson’s-associated sleep disorder patterns. For this purpose, body function parameters of sleep are technically validated with wearable sensor technology and non-contact EmpkinS sensor technology in comparison to classical poly-somnography and correlated to clinical scales. In an algorithmic approach, multiparametric sleep parameters and sleep patterns are then evalulated in correlation to movement, cardiovascular and sleep phase regulation disorders.
D05 Empatho-kinaesthetic Assessment and Pattern Recognition of Movement as Biomarkers for Health Status, Wellbeing and Prognosis of Palliative Patients
For the first time,D05 is investigating movement as a biomarker, i.e. objectifiable, measurable and quantifiable parameters that have diagnostic and predictive significance for the state of health, well-being and prognosis of palliative patients. These biomarkers are examined with patients in the laboratory and living lab of the palliative care unit. Contactless sensor technology allows scientific access to the last phase of life for the first time. Similarly,D05 is researching the sociological challenges of medical technology innovations in palliative care as a prototype for all health care sectors.
Project Area E - Ethics in and by Design
Project area E researches the ethical, legal and social issues in the context of EmpkinS and develops orientation marks for its future societal design. In the sense of an “ethics in and by design” approach, the aim is not only to critically accompany technological development, but also to sound out and implement specific design and decision paths. When researching the normative core concepts, the focus is on the concepts of human dignity, informational self-determination, justice and solidarity in times of big data and research and applications driven by algorithms. Furthermore, it is examined which social attitudes and values are affected by EmpkinS and what effects this has on the social evaluation of EmpkinS as well as the weighting of normative criteria.
E Ethics in and by Design: Ethical and Social Challenges of Empathokinaesthetic Sensor Technologies
In E, a continuous ethical monitoring of the EmpkinS research will take place. The identified questions will be theoretically taken up by means of a normative analysis and examined with regard to their effects and consequences on normative conditions of implementation. Qualitative-empirical studies will be used to sharpen the normative work with a view to the concrete challenges and to establish a communication platform for the purpose of broad social participation in the CRC. The foundations for a multi-dimensional perspective approach to responsible governance in dealing with EmpkinS are also being laid.