Subproject A01

Subproject A01

Multimodal 3D Body Imaging Camera

The subproject A01 focuses on the innovation of a multimodal sensor concept for precise and contactless detection of the body surface and the velocity vectors of all points on this surface. In the first phase of the project, the focus is on the tracking of hand movements, which in the second step is to be transferred to whole-body applications. The primary research question deals with the integration of an RGB-D camera with an imaging radar, overcoming challenges such as spatial calibration, temporal synchronization, analysis of reflectance behavior based on sensor characteristics, and the development of a coherent strategy for data fusion of both sensors. On the radar front, the project includes the development of new signal shapes and signal processing algorithms. These innovations should facilitate fast data acquisition allowing for velocity measurements and simultaneous high-resolution radar imaging. On the other side, the reconstruction of hand shapes and the estimation of hand posture using RGB-D data are researched. In addition, it is researched how velocity information from the imaging radar can improve RGB-D-based hand tracking.

 

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

Contacts

Prof. Dr.-Ing. Marc Stamminger

Principal Investigator

Prof. Dr.-Ing. Martin Vossiek

Principal Investigator

Tobias Schwab, M. Sc.

Doctoral Candidate

Vanessa Wirth, M. Sc.

Doctoral Candidate

Lukas Engel, M. Sc.

Associated Doctoral Candidate

Dr.-Ing. Ingrid Ullmann, Akad. Rat

Postdoc

 

 

Additional Information

  • Coppers B., Heinrich S., Tascilar K., Phutane U., Kleyer A., Simon D., Bräunig J., Penner J., Vossiek M., Schönau V., Bayat S., Schett G., Leyendecker S., Liphardt AM.:
    Sensor-assessed grasping time as a biomarker of functional impairment in rheumatoid arthritis
    Scientific Reports 15 (2025), Art.Nr.: 6018, ISSN: 2045-2322. DOI: 10.1038/s41598-025-90295-7
  • Engel L., Müller J., Rendon EJF., Dorschky E., Krauß D., Ullmann I., Eskofier B., Vossiek M.:
    Advanced Millimeter Wave Radar-Based Human Pose Estimation Enabled by a Deep Learning Neural Network Trained With Optical Motion Capture Ground Truth Data
    IEEE Journal of Microwaves (2025), S. 1-15, ISSN: 2692-8388. DOI: 10.1109/JMW.2025.3535525
  • Hofmann N., Wirth V., Bräunig J., Ullmann I., Vossiek M., Weyrich T., Stamminger M.:
    Inverse Rendering of Near-Field mmWave MIMO Radar for Material Reconstruction
    IEEE Journal of Microwaves (2025), S. 1-17, ISSN: 2692-8388. DOI: 10.1109/JMW.2025.3535077
  • Wirth V. et al.:
    Automatic Spatial Calibration of Near-Field MIMO Radar With Respect to Optical Depth Sensors
    (2025).
    DOI: 10.48550/arXiv.2403.10981
  • Bräunig J., Heinrich S., Coppers B., Kammel C., Wirth V., Stamminger M., Leyendecker S., Liphardt AM., Ullmann I., Vossiek M.:
    A Radar-Based Concept for Simultaneous High-Resolution Imaging and Pixel-Wise Velocity Analysis for Tracking Human Motion.
    IEEE Journal of Microwaves 4 (2024), S. 639-652. ISSN: 2692-8388. DOI: 10.1109/JMW.2024.3453570
  • Eskofier B., Vossiek M.:
    Invisible Sensing: Radar-based Biomonitoring.
    IEEE Open Journal of Engineering in Medicine and Biology (2024), S. 1-2. ISSN: 2644-1276. DOI: 10.1109/OJEMB.2024.3409086
  • Fröhlich AC., Mejdani D., Engel L., Bräunig J., Kammel C., Vossiek M., Ullmann I.:
    A Millimeter-Wave MIMO Radar Network for Human Activity Recognition and Fall Detection.
    IEEE Radar Conference (Denver, 2024). DOI: 10.1109/RadarConf2458775.2024.10548702
  • Gambietz M., Dröge A., Schüßler C., Stahlke M., Wirth V., Miehling J., Koelewijn A.:
    Unobtrusive Gait Reconstructions using Radar-based Optimal Control Simulations.
    Asilomar Conference on Signals, Systems, and Computers (Pacific Grove, 2024)
  • Krauß D., Engel L., Ott T., Bräunig J., Richer R., Gambietz M., Albrecht N., Hille EM., Ullmann I., Braun M., Dabrock P., Kölpin A., Koelewijn A., Eskofier B., Vossiek M.:
    A Review and Tutorial on Machine Learning- Enabled Radar-Based Biomedical Monitoring.
    IEEE Open Journal of Engineering in Medicine and Biology (2024), S. 1-22. ISSN: 2644-1276. DOI: 10.1109/OJEMB.2024.3397208
  • Mejdani D., Bräunig J., Grießhammer S., Krauß D., Steigleder T., Engel L., Jukic J., Rozhdestvenskaya A., Winkler J., Eskofier B., Ostgathe C., Vossiek M.:
    Radar-Based Tremor Quantification Using Deep Learning for Improved Parkinson’s and Palliative Care Assessment.
    IEEE Transactions on Radar Systems 2 (2024), S. 1174-1185. ISSN: 2832-7357. DOI: 10.1109/TRS.2024.3494473
  • Müller J., Krauß D., Engel L., Vossiek M., Eskofier B., Dorschky E.:
    End-to-End Learning for Human Pose Estimation from Raw Millimeter Wave Radar Data.
    Asilomar Conference on Signals, Systems, and Computers (Pacific Grove, 2024)
  • Öhl N., Steigleder T., Hille EM., Braun M., Weisser A., Muehlensiepen F., Vossiek M., Eskofier B., Kölpin A., Forbes CC., Ostgathe C., Heckel M.:
    Soziale Implikationen von Gesundheitstechnologien in der Palliativversorgung – Entwicklung eines konzeptionellen Rahmenwerks
    Zeitschrift für Palliativmedizin 25 (2024), ISSN: 1615-2921. DOI: 10.1055/s-0044-1788351
  • Öhl N., Weisser A., Ott T., Grießhammer S., Yip J., Klinger I., Kölpin A., Miehling J., Bräuning J., Dabrock P., Steigleder T., Ostgathe C., Heckel M.:
    Mit Bürgerinnen und Bürgern über Forschung und technische Innovationen in der Palliativversorgung ins Gespräch kommen – Pilotierung eines ELSI-Cafés
    Zeitschrift für Palliativmedizin 25 (2024), ISSN: 1615-2921. DOI: 10.1055/s-0044-1788353
  • Rückert R., Ullmann I., Vossiek M., Kaup A., Herglotz C.:
    Data Compression for Close-Range Radar Imaging.
    IEEE Transactions on Radar Systems 1 (2024), S. 421-433. ISSN: 2832-7357. DOI: 10.1109/TRS.2024.3387288
  • Schüßler C., Zhang W., Bräunig J., Hoffmann M., Stelzig M., Vossiek M.:
    Radar-Based Recognition of Static Hand Gestures in American Sign Language.
    IEEE Radar Conference (Denver, 2024)
  • Bräunig J., Mejdani D., Krauß D., Grießhammer S., Richer R., Schüßler C., Yip J., Steigleder T., Ostgathe C., Eskofier B., Vossiek M.:
    Radar-based Recognition of Activities of Daily Living in the Palliative Care Context Using Deep Learning.
    IEEE EMBS International Conference on Biomedical and Health Informatics (New York City, 2023). DOI: 10.1109/BHI58575.2023.10313506
  • Bräunig J., Schüßler C., Wirth V., Stamminger M., Ullmann I., Vossiek M.:
    A Realistic Radar Ray Tracing Simulator for Hand Pose Imaging.
    EuRAD (Berlin, 2023)
  • Bräunig J., Wirth V., Kammel C., Schüßler C., Ullmann I., Stamminger M., Vossiek M.:
    An Ultra-Efficient Approach for High-Resolution MIMO Radar Imaging of Human Hand Poses.
    IEEE Transactions on Radar Systems (2023), S. 1-13. ISSN: 2832-7357. DOI: 10.1109/TRS.2023.3309574
  • Guendel RG., Ullmann I., Fioranelli F., Yarovoy A.:
    Continuous People Crowd Monitoring defined as a Regression Problem using Radar Networks.
    European Radar Conference (Berlin, 2023)
  • Schüßler C., Hoffmann M., Wirth V., Eskofier B., Weyrich T., Stamminger M., Vossiek M.:
    Achieving Efficient and Realistic Full-Radar Simulations and Automatic Data Annotation by Exploiting Ray Meta Data from a Radar Ray Tracing Simulator.
    IEEE Radar Conference (San Antonio, 2023). DOI: 10.1109/RadarConf2351548.2023.10149641
  • Wirth V., Liphardt AM., Coppers B., Bräunig J., Heinrich S., Leyendecker S., Kleyer A., Schett G., Vossiek M., Egger B., Stamminger M.:
    Markerless RGB-D Hand Pose Estimation for Activity Monitoring of Musculoskeletal Diseases.
    IEEE-EMBS International Conference on Biomedical and Health Informatics (Pittsburgh, 2023)
  • Wirth V., Liphardt AM., Coppers B., Bräunig J., Heinrich S., Leyendecker S., Kleyer A., Schett G., Vossiek M., Egger B., Stamminger M.:
    ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty.
    International Conference on Computer Vision Workshops (Paris, 2023). Proceedings of the IEEE/CVF International Conference on Computer Vision
  • Ullmann I., Guendel R., Kruse NC., Fioranelli F., Yarovoy A.:
    A Survey on Radar-Based Continuous Human Activity Recognition.
    IEEE Journal of Microwaves (2023), S. 1-13. ISSN: 2692-8388. DOI: 10.1109/JMW.2023.3264494
  • Ullmann I., Guendel RG., Kruse NC., Fioranelli F., Yarovoy A.:
    Radar-Based Continuous Human Activity Recognition with Multi-Label Classification.
    IEEE SENSORS (Wien, 2023)
  • Vossiek M., Haberberger N., Krabbe L., Hehn M., Carlowitz C., Stelzig M.:
    A Tutorial on the Sequential Sampling Impulse Radar Concept and Selected Applications.
    IEEE Journal of Microwaves 3 (2023), S. 523-539. ISSN: 2692-8388. DOI: 10.1109/JMW.2022.3228724
  • Coppers B., Heinrich S., Phutane U., Berisha D., Tascilar K., Kleyer A., Simon D., Bräunig J., Penner J., Vossiek M., Schönau J., Bayat S., Schett G., Leyendecker S., Liphardt AM.:
    Feasibility of using optoelectronic measurement of hand movement for characterizing hand function in rheumatoid arthritis.
    75th European Alliance of Associations for Rheumatology, European Congress of Rheumatology (Copenhagen, 2022). DOI: 10.1136/annrheumdis-2022-eular.1741
  • Coppers B., Heinrich S., Phutane U., Berisha D., Tascilar K., Kleyer A., Simon D., Bräunig J., Penner J., Vossiek M., Schönau V., Bayat S., Schett G., Leyendecker S., Liphardt AM.:
    Evaluation of marker-based motion capturing to charactherize basic hand movements in rheumatic patients.
    27th Congress of the European Society of Biomechanics (Porto, 2022)
  • Coppers B., Phutane U., Berisha D., Tascilar K., Kleyer A., Simon D., Bräunig J., Penner J., Vossiek M., Schönau V., Heinrich S., Bayat S., Schett G., Leyendecker S., Liphardt AM.:
    Establishment and evaluation of a sensor-based method to assess hand function in patients with rheumatoid arthritis.
    World Biomechanics Congress 2022 (Taipei (online), 2022)
  • Richer R., Koch V., Küderle A., Müller V., Wirth V., Stamminger M., Rohleder N., Eskofier B.:
    Detection of Acute Psychosocial Stress from Body Movements using Machine Learning. 79th Annual Scientific Meeting of the American Psychosomatic Society (Long Beach, 2022).
    Lippincott Williams & Wilkins (Hrsg.): Psychosomatic Medicine 2022
  • Rohleder N., Richer R., Koch V., Küderle A., Müller V., Wirth V., Stamminger M., Eskofier B.:
    Effect of Acute Psychosocial Stress on Body Movements.
    79th Annual Scientific Meeting of the American Psychosomatic Society (Long Beach, 2022). Lippincott Williams & Wilkins (Hrsg.): Psychosomatic Medicine 2022
  • Körner G., Hoffmann M., Stief P., Zhang M., Rückert R., Herglotz C., Kaup A., Vossiek M.:
    Applicability and Performance of Standard Compression Methods for Efficient Data Transmission and Storage in Radar Networks.
    IEEE Journal of Microwaves 2 (2021), S. 78-96. ISSN: 2692-8388
    DOI: 10.1109/JMW.2021.3119781
  • Phutane U., Liphardt AM., Bräunig J., Penner J., Klebl M., Vossiek M., Kleyer A., Schett G., Leyendecker S.:
    Evaluation of Optical and Radar Based Motion Capturing Technologies for Characterizing Hand Movement in Rheumatoid Arthritis — A Pilot Study.
    Sensors 21 (2021), Art.Nr.: 1208. ISSN: 1424-8220. DOI: 10.3390/s21041208
  • Root K., Ullmann I., Gmehling S., Vossiek M.:
    Close range ISAR imaging with target velocity determination for security screening applications.
    Emerging Imaging and Sensing Technologies for Security and Defence VI SPIE Security + Defence (Online, 2021). DOI: 10.1117/12.2602072