GAPs

Empathokinesthetic Sensors for Sleep Phase Analysis (“EmpkinS-Bed”)

GAP I focuses on the research of empathokinesthetic sensor technology (A01, A04, B03) for measuring medically relevant parameters in bed related to sleep disorders associated with Parkinson’s disease (D04) and bedridden palliative patients (D05). Parameters relevant to neuro and psychomotor modeling are correlated with sleep disorders and treatment needs of bedridden patients. Fine and coarse motor movement patterns as well as autonomic regulatory parameters are captured and quantified. Internal causes of sleep disorders are differentiated and objectified depending on the disease and treatment. The adaptation and validation of the synthetic data model developed in B03 for this measurement configuration is done using A01/A04 radar sensor data. The technical validation of the sensor data and the evaluation methodology developed in GAP I is performed using established reference sensors (inertial, position, body temperature, heart activity, respiration, EKG, EEG, EMG, pulse oximetry sensors, polysomnography, (infrared) cameras), and the clinical validation is done in correlation with psycho and biometric scales. In addition, an “EmpkinS-Bed” is designed and implemented. Measurement arrangements, signal processing, and segmentation methods for deriving relevant motion parameters from the radar signals are researched, and machine learning methods (including the acquisition of necessary datasets) are designed and implemented to capture relevant neuro and psychomotor parameters.

Capture and Analysis of Hand Function Tests through Empathokinesthetic Sensors

GAP II involves recording motion data and experimentally evaluating different hardware stages of the multimodal body shell camera implemented in A01 through studies involving subjects with and without rheumatic diseases. The measurements are conducted in collaboration with C04 and B04 within D01, which serves as the lead application for evaluating the body shell camera in funding period 1. The demonstrator aims to capture the movement parameters of the human hand, a spatially limited but medically highly relevant body part, as well as larger body regions initially in coarse resolution, for markerless assessment of posture and gait. Based on these applications and measurements, the system concept will be researched, optimized, and the measurement system will be designed. This includes exploring suitable measurement parameters for each sensor (infrared camera and radar from A01, marker-based optical tracking from C04) and research methods to ensure comparability between different sensor modalities. Visualization of motion data will be explored in conjunction with B04.

Construction, evaluation, and optimization of the experimental system

In this GAP, during funding period 1, A02, A03, and B02 will collaborate to build an experimental system for radio localization and wireless sensor data transmission. The system will consist of twelve EBs (Empkins Beacons) and three BSs (Base Stations), each with eight receiving channels. Various sophisticated EB versions will be available during the first funding period.

Initially, A02 and A03 will jointly create an EB using discrete components to explore the novel localization algorithm developed in A02. This EB will be available for evaluating the localization algorithms towards the end of the first year. By the end of year 2, A03, in collaboration with B02, will have developed an energy-efficient EB with additional sensor functionalities and local energy generation in a suitable form factor. At the end of funding period 1, the 61 GHz transceiver will be replaced by a highly integrated version, optimized by A03.

Based on the protocols designed in B02, the baseband signal processing blocks will be selected and implemented by A02, A03, and B02 in the BSs and EBs. Modulation and coding techniques will be determined based on the required uplink and downlink data rates, and appropriate reference signals will be evaluated. The experimental system’s performance will be jointly investigated by A02, A03, and B02 in terms of developed protocols and algorithms for resource allocation and radio localization. The optimization will include data rates achieved, reliability, relative and absolute localization accuracy, locally generated energy, and EBs’ energy consumption.

In addition to the first version of the EB, A02 will be responsible for the base stations, each consisting of eight coherently receiving channels with a local oscillator optimized for frequency stability. The individual transmission channels in the base station will need to be separated through Frequency Division Multiplexing for each receiving antenna. The necessary bandpasses should be efficiently operated in parallel in digital form. Due to the desired measurement rate of up to 10 kHz and the high number of channels and antennas, a large amount of data is processed in the BSs, requiring efficient processing. The data will be transmitted to a central PC via Ethernet, where the algorithm from AP 2 takes over the position and orientation determination.

Experimental Evaluation of Radio Localization and EMG-Empatho-Kinesthetic Sensor Technology

This GAP focuses on the collection of motion data and the experimental evaluation of the radio localization and wireless sensor technology implemented in A02, A03, and B02. The measurements on humans will be conducted within C01 for personalizing musculoskeletal human models and within D01 for capturing movement patterns of the extremities in arthritis. The involvement of patients/healthy subjects will be carried out as part of the ethical approvals of C01 and D01.

A02, in collaboration with A03 and B02, will provide the implemented and pre-evaluated experimental system for these experiments. E will be involved in the planning of the experimental design. The data processing will be conducted in A02 and A03. The analysis of the processed data and integration with gold standard measurement techniques will be carried out in A02, A03, B02, C01, and D01, according to the specific research questions in each field. The storage and transfer of data will follow the EmpkinS-FDM standards developed in A02 in collaboration with the FDM working group.

Direct feedback from C01 and D01 will allow for optimizing critical parameters, both in hardware and software, specifically for the individual measurement requirements. For example, in the localization algorithm, different assumptions regarding speeds, accelerations, and angles of the wearable devices (EBs) will be made to improve localization accuracy, or the positioning of the base stations will be chosen to avoid impairments due to shadowing or dilution of precision. Finally, a comparison will be made with the state of the art in the analysis of human movements using suitable metrics, and the results will be provided as a reference to A01.

Experimental Study for the Analysis and Validation of EMG Data Acquisition

In this GAP, together with A05, A03, and D02, an experimental study will be conducted to investigate and validate methods for capturing EMG data in the context of capturing affect-relevant facial and bodily movements. Given that these movements are based on muscle innervations and that pharmacological manipulation of the “corrugator supercilii muscle” (musculus corrugator supercilii) can lead to clinically significant effects in depressed patients, the focus will initially be on this muscle. A clinically and technologically informed set of requirements will be specified. In the first step, muscle activity during the experiment conducted in D02 will be captured using currently available mobile and stationary EMG measurement methods to establish benchmarks for the methods developed in EmpkinS. In the second step, the activity of the corrugator muscle will be captured using both the surface EMG investigated in A03 and the electro-optical microstructure and micro-movement sensor (EO-MBS) investigated in A05. A joint analysis and evaluation of the obtained data will address the question of to what extent EMG signal components can be detected contactlessly by the EO-MBS and the relationship between the two sensor information and currently available technologies. In addition, sensitivity, dynamics, and visualization of the techniques developed in A03/05 will be further optimized based on feedback from D02 to generate meaningful indicators of depression-relevant kinesthetic movements from diagnostically relevant signal components. Finally, it will be investigated to what extent the radio localization system explored in A02/A03 can be used specifically to capture consent and rejection responses to depression-relevant stimuli, which are reflected in subtle body movements (e.g., nodding vs. shaking the head). In close collaboration with D02, the surface EMG from AP 3 will be evaluated specifically in terms of practicality, user-friendliness, and validity of the measurement compared to a wired standard technology. The evaluation will include aspects such as the arrangement and number of electrodes and possible placements of the sensor nodes on the face. The study with participants will also provide insights into the energy management of the wearable devices (EBs), including the number and duration of measurements.

Sensor Fusion for Compensation of Motion Artifacts

The electro-optical micro-movement sensor (EO-MBS) in A05 and the microwave interferometric sensor in A04 observe small movements of the skin surface caused by muscle fasciculations, heartbeat, and respiration. They not only capture the desired measurement quantities but also large body movements and the measurement quantities of the other sensor, leading to motion artifacts and reduced detection accuracy. Due to the similar problem situations in A04 and A05, this joint AP explores sensor fusion with high-resolution data from the body envelope measurements in A01 and A02 to compensate for the large body movements expected in the measurement scenarios of projects D02-D05 and significantly reduce interference. Furthermore, there will be mutual data exchange to quantify and compensate for mutual interferences. The simulation models of both sensors will be extended to include sensor data from A01 and A02 to derive boundary conditions for successful motion compensation. Additionally, the models from C02 and C03 will be used for plausibility matching of large body movements. To realize the compensation, a transformation between coordinate systems and precise synchronization of individual sensors are necessary to calculate suitable synthetic EO-MBS and MIS signal components of large body movements for compensation and to analyze and evaluate possible distortions of the desired biomarkers caused by large body movements. For A04, it is important to consider that large body movements can often be accompanied by a significant change in the radar cross-section. Therefore, in addition to the actual motion compensation, estimating and compensating for distortions of the biomarker signal will be intensively researched.

Detection of stress through fusion of microstructure and micro-movement sensors in different populations

GAP VII explores stress as an empathic entity in different populations regarding common and different effects on micro-movements and sweating. Based on the cardiorespiratory microwave interferometry sensor and the electro-optical microstructure and micro-movement sensor, A04, A05, D03, and D05 investigate kinetic parameters for stress and validate them against the gold standard. GAP VII allows differentiation of (a) which micro-movements and sweating processes are suitable as target variables, (b) which empathokinesthetic effects are population-specific and which are generalizable, (c) which technical and algorithmic designs best compensate for population-specific disturbances, and (d) how sensor fusion enhances robustness and performance. GAP VII initially investigates measurement scenarios for the induction of acute stress, which are suitable for the use of early patterns of the functional demonstrators of the MISs and EO-MBS (A04 and A05), and makes necessary adjustments to the acute stress protocols. The goal is to establish a common study protocol based on clinical-physiological principles (D03 and D05), which allows for comparison between different collectives. Based on this, A04 and A05 design a measurement setup for the standardized setting in D03 and D05, which should enable the capture of biomarkers from initially sitting or standing individuals at distances of up to 1 meter, later 2.5 meters. Compensation mechanisms for GAP VII are derived from GAP VI. The biometric gold standard validation is performed for MISs using EKG and for EO-MBSs regarding sweating by measuring electrodermal activity (EDA). Population-specific psychometric validation is conducted by comparing collectives and self-reports of patients and healthy subjects. Additionally, data storage, workflow control, logging, and synchronization with experiment-specific additional sensors, including those from A01 and A02, need to be addressed. The fusion of data from the two sensor methods is explored in comparison to single-sensor analysis, and validation as well as its robustness regarding population-specific bias are investigated.

Specifically, within the framework of this GAP, A04 will design the measurement setup using the designs from AP 1 and in coordination with A05, which can be adapted to new deployment scenarios during the project’s progress (see above for distance specification and dynamics as well as the table of subproject objectives) and will serve as the basis for data acquisition. In the subsequent subtask of A04, the data preprocessing for the analysis of tertiary information will be the focus. Spectrograms of the measurement data are considered promising, but pure time-domain, frequency-domain, and hybrid methods will also be investigated. To achieve the required high temporal and spatial resolution, disturbances in post-processing need to be suppressed to the maximum extent, which will be explored in this subtask based on an analysis of the datasets and incorporating the results from AP 1 and GAP VI. Finally, in close cooperation with the research partners from D03 and D05 and in collaboration with A05, the characteristic changes in the measurement signals obtained in the experiments will be correlated and interpreted with respect to the temporal course and experimental conditions. A suitable form of presentation needs to be found for evaluating and commenting on the information by medical professionals.

Evaluation of Multimodal Body Hull Data Coding for Capturing Affective States

This joint work package involves the development and evaluation of psychovisual components for lossy encoding of body hull data. In the first step, the encoding methods developed in B01 will be applied to the body hull data collected in AP 2. Since depression-associated symptoms can already be expressed through minimal changes in facial expression, body posture, and movement, these data are particularly suitable for defining the limits of lossy source encoding and improving the encoding methods based on the insights gained. Similar to the mp3 encoding method, where psychoacoustic phenomena are exploited to achieve high compression rates, the encoding methods used in B01 can benefit from psychovisual insights through close collaboration with D02 to achieve a significant reduction in data rate without or with hardly noticeable quality degradation. Furthermore, a demonstrator (taking into account preliminary results from B04) will be implemented to visualize the influence of the developed source encoding methods and different compression levels on the reconstruction of the body hull data (also compared to other subprojects). Facial expression is particularly suitable for this demonstration, as the human eye can perceive even the smallest changes in facial expression and body posture, making reconstruction errors immediately noticeable.

In close collaboration with D02, the goal is to investigate how the expertise from perceptual psychology as well as audio/video coding in regard to lossy encoding can be transferred to the body hull data collected in the CRC. Initially, heuristics for video-based body envelope data from D02 will be defined and subsequently evaluated. The results of this evaluation will be applied in AP 3 and AP 4 of B01 to further investigate specific sink properties for multimodal body hull data. Depending on the body region considered (face vs. hand vs. body extremities) and the clinical application, these properties may need to be dynamically adjusted to achieve an optimal operating point. The design of data storage and interfaces will be closely coordinated with the DFG working group.

Data acquisition and conceptualization of human models

The data acquisition of all C projects will be coordinated in a GAP, with each subproject conducting its own data acquisition. The synergistic effects resulting from the GAP contribute to making the subproject-specific data acquisition more efficient and establishing interfaces or compatibility between the datasets. In a bimonthly Jour fixe, the involved individuals will exchange information about the progress and planning of data acquisition. Aspects beyond data acquisition, such as the use of data for the conceptualization and personalization of biomechanical human models, as well as in the formulation of mathematical models and their numerical approximation, will also be coordinated within this framework. The research data generated in project area C will be incorporated into the central research data management of EmpkinS.

Sensor-based Capture and Evaluation of Hand Function

Step 1: When conceptualizing and conducting hand function tests, the following questions arise: (1) Which movements should be performed? (2) Which parameters should be evaluated? (3) How can they be measured? In addition to the reference poses necessary for model calibration, different simple and complex movements will be captured using optical marker tracking and EMG measurements: the pose and posture of the hand during the measurement of isometric hand strength (hand dynamometer, Lafayette Instrument, Lafayette, IN, USA), the lifting of specific objects (balls, cylinders of different sizes), and the MPUT. Within the scope of this GAP, it is necessary to plan which parameters should be analyzed. The previously collected pilot dataset serves as a guidance. Furthermore, additional relevant hand movements should be selected to design a measurement protocol and the entire measurement setup accordingly. Potential target parameters could include: hand movement patterns (joint angles, movement velocities, joint forces – using marker-based motion analysis) and muscle activity of the fingers and hand muscles (using EMG).

Step 2: The healthy and impaired movements captured in step 1 will be evaluated and analyzed regarding the planned parameters. After extracting (post-processing) the relevant parameters from the measurement data, correlations with the health condition or disease activity will be investigated to answer the following research questions: (1) Which parameters are suitable as biomarkers for impaired movement due to rheumatoid arthritis and psoriatic arthritis? (2) How can the impairment be represented in the mathematical models?

Step 3: Prospective data collection will further validate the technology and obtain initial data on the influence of therapy on the sensor-based capture of hand function. To assess the relevance of individual modeling aspects, measurements will be repeated on the same study participants over time (after year 1). Depending on disease parameters, the impairments in the biomechanical model will be adjusted, and the predicted (restricted) movement from the simulation model will be compared to the measured movement.