698.INTELLIGENT CONTROL OF KUKA ROBOTIC SYSTEMS BASED ON AI-DRIVEN HUMAN MOTION TRACKING
DOI:
https://doi.org/10.55302/MESJ25432101kKeywords:
KUKA manipulator, computer vision, AI-driven control, deep learning, real-time controlAbstract
Industrial robot control, often restricted by proprietary systems like KUKA Robot Language, presents a challenge for advanced scientific research and intuitive programming. This paper introduces a novel, low-cost framework for Intelligent Control of KUKA Robotic Systems using AI-Driven Human Motion Tracking to facili-tate kinesthetic teaching. The system integrates the MediaPipe Hands Deep Learning model for real-time 3D hand landmark tracking with a custom PyOpenShowVar/KUKAVARPROXY control middleware, enabling soft real-time command transmission to a KUKA KR 16-2. The framework achieved 97.3% accuracy for discrete gesture commands and used a Direct Landmark Differencing approach to provide intuitive, simultaneous control over 3D joint-space movement. While exhibiting 200ms soft real-time overhead, the performance is highly suitable for path teaching and signif-icantly lowers the technical barrier for human-robot collaboration and flexible manufacturing.References
Sanfilippo, F., Hatledal, L. I., Zhang, H., Fago, M., Pet-tersen, K. Y. (2015): Controlling KUKA industrial robots: Flexible communication interface JOpenShowVar. IEEE Robotics & Automation Magazine, 22 (4): 96–109.
Elshatarat, H. L., Baniyounis, M., Biesenbach, R. (2016, March): Design and implementation of a RoBO-2L MAT¬LAB toolbox for a motion control of a robotic manipulator. In: 13th International Multi-Conference on Systems, Sig¬nals & Devices (SSD), pp. 89–95, IEEE.
Mehner, M., Matkovic, N., Muehlbeier, E., Mayer, D., Fleischer, J., Verl, A. (2023, September): Evaluation of external control of KUKA industrial robots for laboratory and prototype environments. In: ISR Europe 2023; 56th International Symposium on Robotics, pp. 278–284. VDE.
Bilancia, P., Schmidt, J., Raffaeli, R., Peruzzini, M., Pellic¬ciari, M. (2023): An overview of industrial robots control and programming approaches. Applied Sciences, 13 (4), 2582.
Abdeetedal, M., Kermani, M. R. (2019): An open-source integration platform for multiple peripheral modules with KUKA robots. CIRP Journal of Manufacturing Science and Technology, 27, 46–55.
Shang-Liang, C., Li-Wu, H. (2021): Using deep learning technology to realize the automatic control program of robot arm based on hand gesture recognition. International Journal of Engineering and Technology Innovation, 11(4), 241.
} De Smedt, Q. (2017): Dynamic hand gesture recognition – From traditional handcrafted to recent deep learning approaches (Doctoral dissertation, Université de Lille 1, Sciences et Technologies; CRIStAL UMR 9189).
Akgun, B., Cakmak, M., Yoo, J. W., Thomaz, A. L. (2012, March): Trajectories and keyframes for kinesthetic teach¬ing: A human-robot interaction perspective. In: Proceed¬ings of the Seventh Annual ACM/IEEE International Confer¬ence on Human-Robot Interaction, pp. 391–398.
Foteinos, K., Cani, J., Linardakis, M., Radoglou-Gram-matikis, P., Argyriou, V., Sarigiannidis, P., Varlamis, I., Pa¬padopoul¬os, G. T. (2025): Visual hand gesture recogni¬tion with deep learning: a comprehensive review of meth¬ods, datasets, challenges and future research direc¬tions. DOI: https://doi.org/10.48550/arXiv.2507.04465
