Robust Visual Tracking for Robotic Systems

The RobVis project aims at research and development vision driven real-world robotic system with main focus on robustness and stability. The goal is to develop general purpose vision system which can be easily integrated into various robotic environments or safe human robot interaction.

Research focus

Visual Tracking

Research and Development of Visual Tracking Systems
Benchmarking
Identifying causes of possible degradation in performance
Developing methods to boost robustness and stability
General purpose Interface
Integration with Robotic Systems

Robotics

Low level control of Industrial Robots
Visual servoing
Visual servoing with uncalibrated systems under torque mode
Safe human robot interaction

Videos

*Robust and real-time 3D multiple human tracking in environment with drastic changes in lighting

Robust and real-time 3D multiple human tracking with targets similar in appearance and moving very close to each other

Position based multiple human visual servoing

Self Steering studio using face tracking and robotic camera system

Interaction of humans with virtual object in a virtual environment using human tracking

People

Suraj Nair, M.Sc.
Emmanuel Dean? , PhD.

Acknowledgement

Publications

[1]	Suraj Nair, Nikhil Somani, Artur Grunau, Emmanuel Dean-Leon, and Alois Knoll. Image processing units on ultra-low-cost embedded hardware: Algorithmic optimizations for real-time performance. Journal of Signal Processing Systems, pages 1-17, 8 2017. [ DOI \| .bib ]
[2]	Yadong Xu, Wentong Cai, David Eckhoff, Suraj Nair, and Alois Knoll. A graph partitioning algorithm for parallel agent-based road traffic simulation. In 2017 Annual ACM Conference on SIGSIM Principles of Advanced Discrete Simulation (SIGSIM-PADS '17), Singapore, may 2017. [ DOI \| .bib ]
[3]	Jordan Ivanchev, Daniel Zehe, Suraj Nair, and Alois Knoll. Fast identification of critical roads by neural networks using system optimum assignment information. In Intelligent Transportation Systems (ITSC), 2017 IEEE 20th International Conference on. IEEE, 2017. [ .bib ]
[4]	Luis Fernando D'Haro, Andreea I. Niculescu, Caixia Cai, Suraj Nair, Rafael E. Banchs, Alois Knoll, and Li Haizhou. An integrated framework for multimodal human-robot interaction. In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), dec 2017. ACCEPTED. [ .bib ]
[5]	Nikhil Somani, Yaadhav Raaj, Suraj Nair, and Alois Knoll. Adapting the search subspace of a particle filter using geometric constraints. Technical Report TUM-I1762, 2017. [ .bib \| .pdf ]
[6]	Rahul Gujarathi, Suraj Nair, Daniel Zehe, Goran Marinkovic, Jiajian Xiao, Pranjal Swarup, Yude Lee, Alois Knoll, Fritz Frenkler, Clemens Scharfen, Marianne Eisl, Christopher Lindinger, and Roland Haring. Tumcreate - interactive virtual research lab. Technical Report TUM-I1763, 2017. [ .bib ]
[7]	Yaadhav Raaj, Suraj Nair, and Alois Knoll. Precise measurement of cargo boxes for gantry robot palletization in large scale workspaces using low-cost rgb-d sensors. In 13th Asian Conference on Computer Vision (ACCV2016), 2016. [ .bib ]
[8]	Lavika Goel, Rashi Khandelwal, Eloy Retamino, Suraj Nair, and Alois Knoll. Application of a hybrid relation extraction framework for intelligent natural language processing. Intelligent Systems Technologies and Applications, Springer, 2016. [ .bib ]
[9]	Jordan Ivanchev, Daniel Zehe, Vaisagh Viswanathan, Suraj Nair, and Alois Knoll. Bisos: Backwards incremental system optimum search algorithm for fast socially optimal traffic assignment. In 19th IEEE International Conference on Intelligent Transportation Systems, ITSC 2016, Rio de Janeiro, Brazil, November 1-4, 2016, pages 2137-2142, 2016. [ DOI \| .bib ]
[10]	Luis Fernando D Haro, Andreea Niculescu, Aravindkumar Vijayalingam, Marco Gutierrez, Suraj Nair, and Rafael Banchs. The path to success: Failures in real robots (fine-r). In FinE-R 2015 The path to success: Failures in Real Robots, IROS2015, Oct 2015. [ .bib \| .pdf ]
[11]	M. A. Gutierrez, S. Nair, R. E. Banchs, L. F. D. Enriquez, A. I. Niculescu, and A. Vijayalingam. Multi-robot collaborative platforms for humanitarian relief actions. In Humanitarian Technology Conference (R10-HTC), 2015 IEEE Region 10, pages 1-6, Dec 2015. [ DOI \| .bib ]
[12]	E. Retamino, S. Nair, A. Vijayalingam, and A. Knoll. Architecture and representation for handling dialogues in human-robot interactions. In 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pages 676-679, Dec 2015. [ DOI \| .bib ]
[13]	M.A. Nasseri, P. Gschirr, D. Eberts, M. Eder, S. Nair, K. Kobuch, M. Maier, D. Zapp, C.P. Lohmann, and A. Knoll. Virtual fixture control of a hybrid parallel-serial robot for assisting ophthalmic surgery: an experimental study. In Proceeding of: 5th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (Biorob), August 2014. [ .bib \| .pdf ]
[14]	Morteza Hashemi Farzaneh, Suraj Nair, M.A Nasseri, and Alois Knoll. Reducing communication-related complexity in heterogeneous networked medical systems considering non-functional requirements. In 16th International Conference on Advanced Communication Technology (ICACT), pages 547-552, Feb 2014. [ DOI \| .bib \| .pdf ]
[15]	Caixia Cai, Nikhil Somani, Suraj Nair, Dario Mendoza, and Alois Knoll. Uncalibrated Stereo Visual Servoing for Manipulators using Virtual Impedance Control. In the 13th International Conference on Control, Automation, Robotics and Vision (ICARCV 2014), Dec. 2014. [ DOI \| .bib \| .pdf ]
[16]	Nasseri M. Ali, Maier M, Zapp D, Nair S, Eder M, Kobuch K, Chris P. Lohmann, and Alois Knoll. Robot for assisting ophthalmic surgery. In Proceedings of the International Society for Medical Innovation and Technology(iSMIT). iSMIT, September 2013. [ .bib ]
[17]	S. Nair, M. Ali. Nasseri, M. Eder, C. P. Lohmann, and A. Knoll. Embedded middleware and hard real-time based architecture for robot assisted ophthalmic surgery. In The Hamlyn Symposium on Medical Robotics, 2013. [ .bib ]
[18]	M.A. Nasseri, M. Eder, S. Nair, E.C. Dean, M. Maier, D. Zapp, C.P. Lohmann, and A. Knoll. The introduction of a new robot for assistance in ophthalmic surgery. In Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE, pages 5682-5685, 2013. [ DOI \| .bib \| .pdf ]
[19]	M.A. Nasseri, M. Eder, D. Eberts, S. Nair, M. Maier, D. Zapp, C.P. Lohmann, and A. Knoll. Kinematics and dynamics analysis of a hybrid parallel-serial micromanipulator designed for biomedical applications. In Advanced Intelligent Mechatronics (AIM), 2013 IEEE/ASME International Conference on, pages 293-299, 2013. [ DOI \| .bib \| .pdf ]
[20]	E.C. Dean-León, S. Nair, and A. Knoll. User friendly matlab-toolbox for symbolic robot dynamic modeling used for control design. In IEEE International Conference on Robotics and Biomimetics ROBIO-2012, 2012. [ .bib \| .pdf ]
[21]	Suraj Nair, Emmanuel Dean-Leon, and Alois Knoll. Real-time 3D multiple human tracking with robustness enhancement through machine learning. In International Conference on Computer Vision Theory and Applications, 2012. [ .bib \| .pdf ]
[22]	M. Ali Nasseri, Emmanuel Dean-Leon, Suraj Nair, Martin Eder, Mathias Maier, C. P. Lohmann, and Alois Knoll. Clinical motion tracking and motion analysis during ophthalmic surgery using electromagnetic tracking system. In Proceedings of the 5th International Conference on BioMedical Engineering and Informatics (BMEI 2012), pages 1006-1010. IEEE Press, 2012. [ .bib \| .pdf ]
[23]	Suraj Nair, Emmanuel Dean, and Alois Knoll. 3d position based multiple human servoing by low-level-control of 6 dof industrial robot. In Proceedings of the 2011 IEEE International Conference on Robotics and Biomimetics. IEEE, 2011. [ .bib \| .pdf ]
[24]	Giorgio Panin, Thorsten Roder, Erwin Roth, Claus Lenz, Suraj Nair, Sebastian Klose, Thomas Muller, Thomas Friedlhuber, Martin Wojtczyk, and Alois Knoll. The opentl library for model-based visual tracking (demo session). In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco (CA), June 2010. [ .bib ]
[25]	Suraj Nair and Alois Knoll. Robust tracking of humans by intelligent multi-modal fusion of visual cues using machine learning. Technical report, Technische Universitaet Muenchen, 2010. [ .bib \| .pdf ]
[26]	Martin Wojtczyk, Giorgio Panin, Thorsten Röder, Claus Lenz, Suraj Nair, Rüdiger Heidemann, Chetan Goudar, and Alois Knoll. Teaching and implementing autonomous robotic lab walkthroughs in a biotech laboratory through model-based visual tracking. In IS&T / SPIE Electronic Imaging, Intelligent Robots and Computer Vision XXVII: Algorithms and Techniques: Autonomous Robotic Systems and Applications, San Jose, CA, USA, 2010. [ DOI \| .bib \| .pdf ]
[27]	Suraj Nair, Thorsten Röder, Giorgio Panin, and Alois Knoll. Visual servoing of presenters in augmented virtual reality tv studios. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, October 18-22, 2010, Taipei, Taiwan, 2010. [ .bib \| .pdf ]
[28]	Suraj Nair, Giorgio Panin, Thorsten Röder, Thomas Friedelhuber, and Alois Knoll. A distributed and scalable person tracking system for robotic visual servoing with 8 dof in virtual reality tv studio automation. In Proceedings of the 6th International Symposium on Mechatronics and its Applications (ISMA09). IEEE, 2009. [ .bib \| .pdf ]
[29]	Martin Wojtczyk, Giorgio Panin, Claus Lenz, Thorsten Röder, Suraj Nair, Erwin Roth, Rüdiger Heidemann, Klaus Joeris, Chun Zhang, Mark Burnett, Tom Monica, and Alois Knoll. A vision based human robot interface for robotic walkthroughs in a biotech laboratory. In François Michaud, Matthias Scheutz, Pamela Hinds, and Brian Scassellati, editors, HRI '09: Proceedings of the 4th ACM/IEEE international conference on Human Robot Interaction, pages 309-310, La Jolla, CA, USA, 2009. ACM. [ DOI \| .bib \| .pdf ]
[30]	Claus Lenz, Suraj Nair, Markus Rickert, Alois Knoll, Wolfgang Rösel, Jürgen Gast, and Frank Wallhoff. Joint-action for humans and industrial robots for assembly tasks. In Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication, pages 130-135, Munich, Germany, 2008. [ .bib \| .pdf ]
[31]	Suraj Nair, Giorgio Panin, Martin Wojtczyk, Claus Lenz, Thomas Friedelhuber, and Alois Knoll. A multi-camera person tracking system for robotic applications in virtual reality tv studio. In Proceedings of the 17th IEEE/RSJ International Conference on Intelligent Robots and Systems 2008. IEEE, 2008. [ .bib \| .pdf ]
[32]	Giorgio Panin, Claus Lenz, Suraj Nair, Erwin Roth, Martin Wojtczyk, Thomas Friedlhuber, and Alois Knoll. A unifying software architecture for model-based visual tracking. In IS&T/SPIE 20th Annual Symposium of Electronic Imaging, San Jose, CA, 2008. [ .bib \| .pdf ]
[33]	Giorgio Panin, Erwin Roth, Thorsten Röder, Suraj Nair, Claus Lenz, Martin Wojtczyk, Thomas Friedlhuber, and Alois Knoll. ITrackU: An integrated framework for image-based tracking and understanding. In Proceedings of the International Workshop on Cognition for Technical Systems, Munich, Germany, 2008. [ .bib \| .pdf ]
[34]	Martin Wojtczyk, Kushal Abhyankar, Suraj Nair, and Alois Knoll. A computer vision based approach for cell tracking to increase throughput in visual drug discovery. In Natural Product Discovery and Production II, Whistler B.C., Canada, 2008. [ .bib \| .pdf ]
[35]	Suraj Nair, Rui Guerreiro, and Gerard De Haan. Real-time gaze tracking using low cost devices. Technical report, Philips Research Laboratories, Eindhoven, The Netherlands, (Company Confidential until 2010), 2007. [ .bib ]
[36]	Suraj Nair and Sami Ben Check. Reconfigurable computing using fpga. Technical report, JTH, Jönköping Tekniska Högskolan, Sweden, 2006. [ .bib ]

[1]	Suraj Nair, Nikhil Somani, Artur Grunau, Emmanuel Dean-Leon, and Alois Knoll. Image processing units on ultra-low-cost embedded hardware: Algorithmic optimizations for real-time performance. Journal of Signal Processing Systems, pages 1-17, 8 2017. [ DOI \| .bib ]
[2]	Caixia Cai, Emmanuel Dean-Leon, Nikhil Somani, and Alois Knoll. 6D Image-based Visual Servoing for Robot Manipulators with Uncalibrated Stereo Cameras. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sep. 2014. [ DOI \| .bib \| .pdf ]
[3]	Caixia Cai, Emmanuel Dean-Leon, Dario Mendoza, Nikhil Somani, and Alois Knoll. Uncalibrated 3D Stereo Image-based Dynamic Visual Servoing for Robot Manipulators. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), November 2013. [ DOI \| .bib \| .pdf ]
[4]	Caixia Cai, Emmanuel Dean-Leon, Nikhil Somani, and Alois Knoll. 3D Image-based Dynamic Visual Servoing with Uncalibrated Stereo Cameras. In The 44th Internation Symposium on Robotics (ISR), October 2013. [ DOI \| .bib \| .pdf ]
[5]	Nikhil Somani, Emmanuel Dean-Leon, Caixia Cai, and Alois Knoll. Scene Perception and Recognition for Human-Robot Co-Operation. In ICIAP'13 workshop: First International Workshop on Assistive Computer Vision and Robotics (ACVR 2013). The 17th International Conference on Image Analysis and Processing., September 2013. [ .bib \| .pdf ]
[6]	Nikhil Somani, Emmanuel Dean-Leon, Caixia Cai, and Alois Knoll. Perception and Reasoning for Scene Understanding in Human-Robot Interaction Scenarios. In CAIP'13 workshop: Recognition and Action for Scene understanding (REACTS 2013). The 15th International Conference of Computer Analysis of Images and Patterns., August 2013. [ .bib \| .pdf ]
[7]	Nikhil Somani, Emmanuel Dean-Leon, Caixia Cai, and Alois Knoll. Scene Perception and Recognition in industrial environments for Human-Robot Interaction. In 9th International Symposium on Visual Computing (ISVC'13). Springer, July 2013. [ DOI \| .bib \| .pdf ]
[8]	Vincent T. van Hees, Lukas Gorzelniak, Emmanuel Dean-Leon, Martin Eder, Marcelo Pias, Salman Taherian, Ulf Ekelund, Frida Renström, Paul W. Franks, Alexander Horsch, and Søren Brage. Separating movement and gravity components in an acceleration signal and implications for the assessment of human daily physical activity. PLoS ONE, 2013. [ DOI \| .bib \| .pdf ]
[9]	M.A. Nasseri, M. Eder, S. Nair, E.C. Dean, M. Maier, D. Zapp, C.P. Lohmann, and A. Knoll. The introduction of a new robot for assistance in ophthalmic surgery. In Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE, pages 5682-5685, 2013. [ DOI \| .bib \| .pdf ]
[10]	S. Can, B. Jensen, E. Dean-Leon, C. Staub, A. Knoll, A. Fiolka, A. Schneider, A. Meining, and H. Feussner. Kinematics, control and workspace analysis of a bowden wire actuated manipulator for minimally invasive single-port surgery. In IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 2012. [ .bib \| .pdf ]
[11]	E.C. Dean-León, S. Nair, and A. Knoll. User friendly matlab-toolbox for symbolic robot dynamic modeling used for control design. In IEEE International Conference on Robotics and Biomimetics ROBIO-2012, 2012. [ .bib \| .pdf ]
[12]	Suraj Nair, Emmanuel Dean-Leon, and Alois Knoll. Real-time 3D multiple human tracking with robustness enhancement through machine learning. In International Conference on Computer Vision Theory and Applications, 2012. [ .bib \| .pdf ]
[13]	M. Ali Nasseri, Emmanuel Dean-Leon, Suraj Nair, Martin Eder, Mathias Maier, C. P. Lohmann, and Alois Knoll. Clinical motion tracking and motion analysis during ophthalmic surgery using electromagnetic tracking system. In Proceedings of the 5th International Conference on BioMedical Engineering and Informatics (BMEI 2012), pages 1006-1010. IEEE Press, 2012. [ .bib \| .pdf ]
[14]	Suraj Nair, Emmanuel Dean, and Alois Knoll. 3d position based multiple human servoing by low-level-control of 6 dof industrial robot. In Proceedings of the 2011 IEEE International Conference on Robotics and Biomimetics. IEEE, 2011. [ .bib \| .pdf ]
[15]	J.A. Flores-Campos, E.C. Dean-León, and C. Palacios-Montufar. Robot motion planning for multiple moving objects interception with constant acceleration. In VII Congreso Internacional sobre Innovación y Desarrollo Tecnológico CIINDET, 2009. [ .bib \| .pdf ]
[16]	E.C. Dean-León, V. Parra-Vega, and A. Espinosa-Romero. Visual servoing for constrained planar robots subject to complexfriction. IEEE/ASME Transactions on Mechatronics, 11(4), 2006. [ DOI \| .bib \| .pdf ]
[17]	E.C. Dean-León, V. Parra-Vega, and A. Espinosa-Romero. Global uncalibrated visual servoing for constrained robots working on uncalibrated environments. In Proc. of the IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS), 2006. [ DOI \| .bib \| .pdf ]
[18]	Vicente Parra-Vega and Emmanuel Dean-Leon. Orthogonalization principle for dynamic visual servoing of constrained robot manipulators. In Sadao Kawamura and Mikhail Svinin, editors, Advances in Robot Control, pages 83-105. Springer Berlin Heidelberg, 2006. [ DOI \| .bib \| .pdf ]
[19]	V. Parra-Vega, R. García-Rodriguez, E.C. Dean-León, and F. Ruiz-Sanchez. An adaptive neural network controller for visual tracking of constrained robot manipulators. American Control Conference (ACC), 2005. [ DOI \| .bib \| .pdf ]
[20]	L. García, E.C. Dean-León, V. Parra-Vega, and A. Espinosa-Romero. Experimental results of image-based adaptive visual servoing of 2d robots under jacobian and dynamic friction uncertainties. In Proc. of the 6th International Symposium on Computational Intelligence in Robotics and Automation (CIRA), Jun 2005. [ .bib \| .pdf ]
[21]	E.C. Dean-León, V. Parra-Vega, A. Espinosa-Romero, and L. García. Adaptive visual servoing for constrained robots under jacobian, jointdynamic and contact viscous friction uncertainties. In Proc.of the 16th International Federation of Automatic Control: WORLD CONGRESS(IFAC), jul 2005. [ DOI \| .bib \| .pdf ]
[22]	E.C. Dean-León, V. Parra-Vega, A. Espinosa-Romero, and L. García. Visual servoing for constrained robots: A new complete theoreticalframework and its experimental validation. In Proc. of the IEEE/ASMEInternational Conference on Advanced Intelligent Mechatronics (AIM), Jul 2005. [ DOI \| .bib \| .pdf ]
[23]	E.C. Dean-León, V. Parra-Vega, and A. Espinosa-Romero. Experimental study on image-based position: Force adaptive visual servoing for constrained robots under jacobian and dynamic friction uncertainties. In Proc. of the IEEE International Conference on Mechatronicsand Automation (ICMA), jul 2005. [ DOI \| .bib \| .pdf ]
[24]	E.C. Dean-León, V. Parra-Vega, NewAuthor3, and L. García. Uncalibrated image-based position-force adaptive visual servoing for constrained robots under dynamic friction uncertainties. In Proc.of the IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), aug 2005. [ DOI \| .bib \| .pdf ]
[25]	E.C. Dean-León, V. Parra-Vega, and L. García. Experimental study on a fast tracking and smooth constrained motion controller for robot arms. Congreso Anual de la Asociación de México de ControlAutomático (AMCA), Oct 2005. [ .bib \| .pdf ]
[26]	E.C. Dean-León, V. Parra-Vega, and A. Espinosa-Romero. Visual servoing for constrained robots in an unknown environment: A new complete theoretical framework and its experimental validation. In Proc. of the 36thInternational Symposium on Robotics (ISR), Nov 2005. [ .bib \| .pdf ]
[27]	E.C. Dean-León, V. Parra-Vega, A. Espinosa-Romero, and J.D Fierro-Rojas. Dynamical image-based pid uncalibrated visual servoing with fixed camera for tracking of planar robots with a heuristical predictor. In Proc.of the 2nd IEEE International Conference on Industrial Informatics, INDIN 04, 2004. [ DOI \| .bib \| .pdf ]
[28]	E.C. Dean-León, V. Parra-Vega, and A. Espinosa-Romero. Image-based visual/force/position control under parametric uncertainties of camera and robot arm. In Proc. of the International Conference on Electrical and Electronics Engineering and X Conference on Electrical Engineering(ICEEE/CIE), 2004. [ DOI \| .bib \| .pdf ]
[29]	J.D Fierro-Rojas, E.C. Dean-León, V. Parra-Vega, and A. Espinosa-Romero. Adaptive predictor-based sliding pid uncalibrated visual servoing with uncertain jacobian for dynamic tracking of robots. In Proc. of the 4th International Conference on Advanced Mechatronics (ICAM), 2004. [ .bib \| .pdf ]
[30]	M. Sandoval, A. Espinosa-Romero, E.C. Dean-León, and L.A. Muñoz. Control visual de un sistema de extracción para hornos giratorios:servo control visual. Congreso Mexicano de Robótica, RoboticsMexican Association (COMROB), 2004. [ .bib \| .pdf ]
[31]	E.C. Dean-León, Heberth Sira-Ramirez, and Vicente Parra-Vega. Algebraic on-line identifier for visual servoing of robot manipulators. InternationalSymposium on Robotics and Automation (ISRA), 2002. [ .bib \| .pdf ]

ResearchRobVis