OpenDR in Short
What is the project about?
The aim of OpenDR is to develop a modular, open and non-proprietary toolkit for core robotic functionalities by harnessing deep learning to provide advanced perception and cognition capabilities, meeting in this way the general requirements of robotics applications in the applications areas of healthcare, agri-food and agile production. The term toolkit in OpenDR refers to a set of deep learning software functions, packages and utilities used to help roboticists to develop and test a robotic application that incorporates deep learning. OpenDR will provide the means to link the robotics applications to software libraries (deep learning frameworks, e.g., Tensorflow) and to link it with the operating environment (ROS). OpenDR focuses on the AI and Cognition core technology in order to provide tools that make robotic systems cognitive, giving them the ability to a) interact with people and environments by developing deep learning methods for human centric (WP3) and environment (WP4) active perception and cognition, b) learn and categorise by developing deep learning tools for training and inference in common robotics settings (WP3 and WP4), and c) make decisions and derive knowledge by developing deep learning tools for cognitive robot action and decision making (WP5). As a result, the developed OpenDR toolkit will also enable cooperative human-robot interaction as well as the development of cognitive mechatronics where sensing and actuation are closely coupled with cognitive systems thus contributing to another two core technologies beyond AI and Cognition. OpenDR will develop, train, deploy and evaluate deep learning models that improve the technical capabilities of the core technologies beyond the current state of the art. It will enable a greater range of robotics applications that can be demonstrated at TRL 3 and above, thus lowering the technical barriers within the prioritised application areas. OpenDR aims to an easily adopted methodology to adapt the provided tools in order to solve any robotics task without restricting it to any specific application.
How are we going to achieve this?
The core objective of OpenDR is to provide real-time (as defined by the specific use case requirements), lightweight (to be deployed in use-case specified embedded CPUs/GPUs like the NVIDIA TX1/2 and Xavier) and ultralightweight (to be deployed in embedded architectures like Raspberry Pi, FPGAs, microcontrollers, etc.) deep learning architectures for performing inference and control in various robotics tasks.
State of the art deep learning architectures will be considered for raw sensor signal (speech, image, video, Lidar, GPS, etc.) analysis that will be trained using supervised, semi-supervised, unsupervised and reinforcement learning both in real data and simulation in order to provide easy to train and deploy models for robotics. A library with a collection of ROS nodes and the necessary tools for training and deployment will be developed that will enable any ROS-based robotic architecture to integrate them for improving its technical capabilities. OpenDR will also build and share open datasets, as well as appropriate simulation environments, for training and evaluating the developed tools on selected robotic applications. Three real use-cases on agri-food, healthcare and agile production will be developed in OpenDR to demonstrate the functionalities of the toolkit.
The objectives of OpenDR are:
To provide a modular, open and non-proprietary toolkit for core robotic functionalities enabled by lightweight deep learning
To leverage AI and Cognition in robotics: from perception to action
To propose a co-integration of simulation and learning methodology for deep learning in robotics and demonstrate the potential of OpenDR in three prioritized application area
To establish strong links to robotics Digital Innovation Hubs
Why are we doing this?
Recent advances in deep learning led to a number of spectacular applications, ranging from self-driving cars, and robots that outperform humans in various tasks, to building algorithms that can understand and answer questions by fusing information from many modalities. The application of deep learning in robotics leads to very specific learning, reasoning and embodiment problems and research questions that are typically not addressed by the computer vision and machine learning communities. Moreover, the available deep learning open frameworks are not easily employed in robotics since they have a steep learning curve and radically different methodology (end-to-end learning, from sensing to acting, etc.) than conventional robotics and thus, relatively few robotics laboratories are including deep learning methodologies to their research. Furthermore, even though deep learning was indeed able to successfully tackle these challenging problems and open an exciting new field of novel and high-impact applications, it comes with an additional significant drawback. Deep learning models are becoming more and more complex, with some recent state-of-the-art techniques using more than 1000 neural layers (e.g., ResNets), requiring vast amounts of computational power and energy both for training and inference purposes. These requirements are becoming especially limiting in many robotics applications, where significant energy and computational power constraints exist, restricting the speed and the accuracy of the deployed models. Additionally, the already available deep learning software modules are implemented in order to be deployed to large and expensive Graphics Processing Units (GPUs) and they rarely perform in real-time even for low-resolution input. Thus, they are usually inappropriate for deployment on embedded systems such as those usually found on robotics systems. The need for an open deep learning toolkit that contains easy to train and deploy real-time, lightweight, Robot Operating System (ROS) compliant deep learning models for robotics is evident.