COMBAHO: COMe BAck HOme system for enhancing autonomy of people with acquired brain injury and dependent on their integration into society

This project has been funded by the Ministry of Economy and Competitiveness of the Government of Spain within the State Program of R+D+i Oriented to the Challenges of Society and co-financed with European Feder funds. The grant number is TIN2016-76515-R and its complete title: RETOGAR: RETORNO AL HOGAR: SISTEMA DE MEJORA DE LA AUTONOMÍA DE PERSONAS CON DAÑO CEREBRAL ADQUIRIDO Y DEPENDIENTES EN SU INTEGRACIÓN EN LA SOCIEDAD.

Principal Investigators: Jose Garcia-Rodriguez and Miguel Cazorla

Reference: TIN2016-76515-R

Funded by: Ministerio de Economía y Competitividad

Duration: 2016-2019

Funding: 89.000 €

The care of dependent persons, whether due to illness, accident, disability or aging, is one of the priority lines of research in developed countries today. This attention, in addition to being helpful and companionable, is being considered to be even therapeutic. On the other hand, it is intended that such care is in the home of the person, with the aim of minimizing the cost of therapies. The rehabilitation of patients will be complete when their integration into society is achieved, either in the family environment or in a working and socialising environment.

To solve this challenge, the main scientific objective of this project is to promote the health and well-being of society through the design, development and evaluation of an assistant for people with acquired or dependent brain damage to help them face the challenges posed by their illness in their full social integration. This assistant has a patient’s home aspect based on the design and use of an intelligent monitoring and active learning environment, and an autonomous social robot for assistance and interactive stimulation at home. On the other hand, it is contemplated to assist patients also in external environments, in case of disorientation or complex situations. This implies the integration of existing technologies, as well as providing new solutions to a variety of technological challenges that these types of systems have been matched. In addition, an experimental evaluation carried out by clinical professionals is also proposed, which will assess the effectiveness of the system in improving the quality of life of dependent persons. Both the autonomy and the positive cognitive-affective status of the patient will be evaluated. In order to achieve the general objective proposed, it is necessary to address certain scientific-technological challenges that are broken down into the following specific objectives: i) to develop an intelligent system for monitoring the robust environment, allowing the location and tracking, in an accurate manner, of the individuals present in a scenario; ii) to develop a robust system for location and navigation, and for the recognition and manipulation of small 3D objects on board the robot. iii) To design a personalized assistant to help the patient in situations of memory failure, lack of orientation, motor difficulties, reduced visibility due to hemiplegia and other situations. This system will be trained with common indoor and outdoor behaviors and scenarios. iv) Provide the assistant with natural interaction skills using innovative natural language processing techniques combined with visual attention and in-depth learning. v) to carry out a design of the care and rehabilitation scenario, and to identify metrics, piloting and final evaluation of the system developed with real scenarios and patients; and finally vi) to disseminate and disseminate the results obtained to the scientific community and to companies and associations related to the sector.

The expected results are diverse. At the scientific-technical level, significant advances are expected in the different technologies to be developed and in terms of social impact, the aim is to improve the quality of life of patients. As for the economic impact, it is hoped that a functional system can be obtained and its possible commercialization will increase the transfer of technology to society.

Goals

For each objective we indicate the results obtained and their degree of achievement:

Design an intelligent environment monitoring system, based on low-cost 3D visual sensors, that allows you to accurately locate and track the environment in your home.
Develop a robust system for locating and navigating, and recognizing and manipulating small 3D objects on board the robot using deep learning techniques.
To design a personalized assistant at home based on in-depth learning techniques specialized in temporal sequences to help the patient in situations of cognitive failure.
Develop an outdoor patient assistance system based on wearable vision sensors and location information.
Carry out the integration of the different systems and a design of care scenarios and identify metrics, piloting and final evaluation of the system developed with indoor and outdoor scenarios and real patients.
To coordinate the proposed objectives and disseminate the results of the project.

Team

José García Rodríguez (PI)
Miguel Cazorla (PI)
Diego Viejo
Javier Montoyo
Sergio Orts Escolano
José María Cañas (Universidad Rey Juan Carlos)
Francisco Martín (Universidad Rey Juan Carlos)
Eugenio Aguirre (Universidad de Granada)
Miguel García Silvente (Universidad de Granada)

Publications

2018

Garcia-Garcia, Alberto and Orts-Escolano, Sergio and Oprea, Sergiu and Villena-Martinez, Victor and Martinez-Gonzalez, Pablo and Garcia-Rodriguez, Jose. A Survey on Deep Learning Techniques for Image and Video Semantic Segmentation. Applied Soft Computing (2018).
[ BibTeX ]

@article{Garcia2018,
  author = {Garcia-Garcia, Alberto and Orts-Escolano, Sergio and Oprea, Sergiu and Villena-Martinez, Victor and Martinez-Gonzalez, Pablo and Garcia-Rodriguez, Jose},
  title = {A Survey on Deep Learning Techniques for Image and Video Semantic Segmentation},
  journal = {Applied Soft Computing},
  year = {2018},
  volume = {Accepted}
}

Rangel, Jose Carlos and Cazorla, Miguel and García-Varea, Ismael and Romero-González, Cristina and Martínez-Gómez, Jesús. Automatic Semantic Maps Generation from Lexical Annotations. Autonomous robots (2018).
[ BibTeX ]

@article{Rangel2018b,
  author = {Rangel, Jose Carlos and Cazorla, Miguel and García-Varea, Ismael and Romero-González, Cristina and Martínez-Gómez, Jesús},
  title = {Automatic Semantic Maps Generation from Lexical Annotations},
  journal = {Autonomous robots},
  year = {2018},
  volume = {Accepted}
}

Navarrete-Sanchez, Javier and Viejo, Diego and Cazorla, Miguel. Compression and Registration of 3D Point Clouds Using GMMs. Pattern Recognition Letters (2018).
[ BibTeX ]

@article{Navarrete2018,
  author = {Navarrete-Sanchez, Javier and Viejo, Diego and Cazorla, Miguel},
  title = {Compression and Registration of 3D Point Clouds Using GMMs},
  journal = {Pattern Recognition Letters},
  year = {2018},
  volume = {110},
  pages = {8-15}
}

Cruz, Edmanuel and Rangel, Jose Carlos and Gómez-Donoso, Francisco and Garcia-Rodriguez, Jose and Cazorla, Miguel. Finding the place: how to train and use convolutional neural networks for a dynamically learning robot. 2018 International Joint Conference on Neural Networks (IJCNN) (2018).
[ BibTeX ]

@inproceedings{Cruz2018b,
  author = {Cruz, Edmanuel and Rangel, Jose Carlos and Gómez-Donoso, Francisco and Garcia-Rodriguez, Jose and Cazorla, Miguel},
  title = {Finding the place: how to train and use convolutional neural networks for a dynamically learning robot},
  booktitle = {2018 International Joint Conference on Neural Networks (IJCNN)},
  year = {2018}
}

Dominguez, Alejandro and Orts-Escolano, Sergio and Cazorla, Miguel. A New Dataset and Performance Evaluation of a Region-based CNN for Urban Object Detection. 2018 International Joint Conference on Neural Networks (IJCNN) (2018).
[ BibTeX ]

@inproceedings{Dominguez2018,
  author = {Dominguez, Alejandro and Orts-Escolano, Sergio and Cazorla, Miguel},
  title = {A New Dataset and Performance Evaluation of a Region-based CNN for Urban Object Detection},
  booktitle = {2018 International Joint Conference on Neural Networks (IJCNN)},
  year = {2018}
}

Cruz, Edmanuel and Orts-Escolano, Sergio and Gómez-Donoso, Francisco and Rizo, Carlos and Rangel, Jose Carlos and Higinio Mora, and Miguel Cazorla. An augmented reality application for improving shopping experience in large retail stores. Virtual Reality (2018).
[ BibTeX ]

@article{Cruz2018,
  author = {Cruz, Edmanuel and Orts-Escolano, Sergio and Gómez-Donoso, Francisco and Rizo, Carlos and Rangel, Jose Carlos and Higinio Mora, and Miguel Cazorla},
  title = {An augmented reality application for improving shopping experience in large retail stores},
  journal = {Virtual Reality},
  year = {2018},
  volume = {Accepted}
}

Ortiz, J.C. Rangel and Gomez, J. Martinez and Gonzalez, C. Romero and Varea, I. Garcia and Cazorla, M.. Semi-supervised 3D Object Recognition through CNN Labeling. Applied Soft Computing (2018).
[ BibTeX ]

@article{Rangel2018,
  author = {Ortiz, J.C. Rangel and Gomez, J. Martinez and Gonzalez, C. Romero and Varea, I. Garcia and Cazorla, M.},
  title = {Semi-supervised 3D Object Recognition through CNN Labeling},
  journal = {Applied Soft Computing},
  year = {2018},
  volume = {65},
  pages = {603-613},
  month = apr
}

Vargas, John Alejandro Castro and Garcia, Alberto Garcia and Oprea, Sergiu and Escolano, Sergio Orts and Rodriguez, Jose Garcia. Object Recognition Pipeline: Grasping in Domestic Environments. Advancements in Computer Vision and Image Processing (2018).
[ BibTeX ]

@incollection{Vargas2018,
  title = {Object Recognition Pipeline: Grasping in Domestic Environments},
  author = {Vargas, John Alejandro Castro and Garcia, Alberto Garcia and Oprea, Sergiu and Escolano, Sergio Orts and Rodriguez, Jose Garcia},
  booktitle = {Advancements in Computer Vision and Image Processing},
  pages = {18--33},
  year = {2018},
  publisher = {IGI Global}
}

2017

Ortiz, Jose Carlos Rangel. Scene Understanding for Mobile Robots exploiting Deep Learning Techniques. (2017).
[ BibTeX ]

@phdthesis{RangelTesis2017,
  title = {Scene Understanding for Mobile Robots exploiting Deep Learning Techniques},
  school = {University of Alicante},
  author = {Ortiz, Jose Carlos Rangel},
  year = {2017}
}

Dominguez-Sanchez, Alejandro and Cazorla, Miguel and Orts-Escolano, Sergio. Pedestrian movement direction recognition using convolutional neural networks. IEEE Transactions on Intelligent Transportation Systems (2017).
[ BibTeX ]

@article{Dominguez2017,
  author = {Dominguez-Sanchez, Alejandro and Cazorla, Miguel and Orts-Escolano, Sergio},
  title = {Pedestrian movement direction recognition using convolutional neural networks},
  journal = {IEEE Transactions on Intelligent Transportation Systems},
  year = {2017},
  volume = {18},
  number = {12},
  pages = {3540--3548}
}

Garcia-Garcia, Alberto and Garcia-Rodriguez, Jose and Orts-Escolano, Sergio and Oprea, Sergiu and Gomez-Donoso, Francisco and Cazorla, Miguel. A study of the effect of noise and occlusion on the accuracy of convolutional neural networks applied to 3D object recognition. Computer Vision and Image Understanding (2017).
[ BibTeX ]

@article{Garcia2017,
  title = {A study of the effect of noise and occlusion on the accuracy of convolutional neural networks applied to 3D object recognition},
  author = {Garcia-Garcia, Alberto and Garcia-Rodriguez, Jose and Orts-Escolano, Sergio and Oprea, Sergiu and Gomez-Donoso, Francisco and Cazorla, Miguel},
  journal = {Computer Vision and Image Understanding},
  volume = {164},
  pages = {124--134},
  year = {2017},
  publisher = {Elsevier}
}

Gomez-Donoso, Francisco and Orts-Escolano, Sergio and Garcia-Garcia, Alberto and Garcia-Rodriguez, Jose and Castro-Vargas, John Alejandro and Ovidiu-Oprea, Sergiu and Cazorla, Miguel. A robotic platform for customized and interactive rehabilitation of persons with disabilities. Pattern Recognition Letters (2017).
[ BibTeX ]

@article{Gomez2017,
  title = {A robotic platform for customized and interactive rehabilitation of persons with disabilities},
  author = {Gomez-Donoso, Francisco and Orts-Escolano, Sergio and Garcia-Garcia, Alberto and Garcia-Rodriguez, Jose and Castro-Vargas, John Alejandro and Ovidiu-Oprea, Sergiu and Cazorla, Miguel},
  journal = {Pattern Recognition Letters},
  volume = {99},
  pages = {105--113},
  year = {2017},
  publisher = {Elsevier}
}

Abellan-Abenza, Javier and Garcia-Garcia, Alberto and Oprea, Sergiu and Ivorra-Piqueres, David and Garcia-Rodriguez, Jose. Classifying Behaviours in Videos with Recurrent Neural Networks. International Journal of Computer Vision and Image Processing (IJCVIP) (2017).
[ BibTeX ]

@article{Abellan2017,
  title = {Classifying Behaviours in Videos with Recurrent Neural Networks},
  author = {Abellan-Abenza, Javier and Garcia-Garcia, Alberto and Oprea, Sergiu and Ivorra-Piqueres, David and Garcia-Rodriguez, Jose},
  journal = {International Journal of Computer Vision and Image Processing (IJCVIP)},
  volume = {7},
  number = {4},
  pages = {1--15},
  year = {2017},
  publisher = {IGI Global}
}

Hernandez, Mauricio Andres Zamora and Marin, Eldon Caldwell and Garcia-Rodriguez, Jose and Azorin-Lopez, Jorge and Cazorla, Miguel. Automatic Learning Improves Human-Robot Interaction in Productive Environments: A Review. International Journal of Computer Vision and Image Processing (IJCVIP) (2017).
[ BibTeX ]

@article{Hernandez2017,
  title = {Automatic Learning Improves Human-Robot Interaction in Productive Environments: A Review},
  author = {Hernandez, Mauricio Andres Zamora and Marin, Eldon Caldwell and Garcia-Rodriguez, Jose and Azorin-Lopez, Jorge and Cazorla, Miguel},
  journal = {International Journal of Computer Vision and Image Processing (IJCVIP)},
  volume = {7},
  number = {3},
  pages = {65--75},
  year = {2017},
  publisher = {IGI Global}
}

Escalona, Félix and Rodrı́guez, Ángel and Gomez-Donoso, Francisco and Martinez-Gomez, Jesus and Cazorla, Miguel and others. 3D object detection with deep learning. (2017).
[ BibTeX ]

@article{Escalona2017,
  title = {3D object detection with deep learning},
  author = {Escalona, F{\'e}lix and Rodr{\'\i}guez, {\'A}ngel and Gomez-Donoso, Francisco and Martinez-Gomez, Jesus and Cazorla, Miguel and others},
  year = {2017},
  publisher = {Red de Agentes F{\'\i}sicos}
}

Gomez-Donoso, F and Garcia-Garcia, A and Garcia-Rodriguez, J and Orts-Escolano, S and Cazorla, M. Lonchanet: A sliced-based cnn architecture for real-time 3d object recognition. Neural Networks (IJCNN), 2017 International Joint Conference on (2017).
[ BibTeX ]

@inproceedings{Gomez2018,
  title = {Lonchanet: A sliced-based cnn architecture for real-time 3d object recognition},
  author = {Gomez-Donoso, F and Garcia-Garcia, A and Garcia-Rodriguez, J and Orts-Escolano, S and Cazorla, M},
  booktitle = {Neural Networks (IJCNN), 2017 International Joint Conference on},
  pages = {412--418},
  year = {2017},
  organization = {IEEE}
}

Oprea, S and Garcia-Garcia, A and Garcia-Rodriguez, J and Orts-Escolano, S and Cazorla, M. A recurrent neural network based Schaeffer gesture recognition system. Neural Networks (IJCNN), 2017 International Joint Conference on (2017).
[ BibTeX ]

@inproceedings{Oprea2018,
  title = {A recurrent neural network based Schaeffer gesture recognition system},
  author = {Oprea, S and Garcia-Garcia, A and Garcia-Rodriguez, J and Orts-Escolano, S and Cazorla, M},
  booktitle = {Neural Networks (IJCNN), 2017 International Joint Conference on},
  pages = {425--431},
  year = {2017},
  organization = {IEEE}
}

Dominguez-Sanchez, Alex and Orts-Escolano, Sergio and Cazorla, Miguel. Pedestrian Direction Recognition using Convolutional Neural Networks. 14th International Work-Conference on Artificial Neural Networks (2017).
[ BibTeX ]

@inproceedings{Dominguez2019,
  author = {Dominguez-Sanchez, Alex and Orts-Escolano, Sergio and Cazorla, Miguel},
  booktitle = {14th International Work-Conference on Artificial Neural Networks},
  title = {Pedestrian Direction Recognition using Convolutional Neural Networks},
  year = {2017},
  month = jun
}

Zamora, Mauricio and Caldwell, Eldon and Garcia-Rodriguez, Jose and Azorin-Lopez, Jorge and Cazorla, Miguel. Machine learning improves human-robot interaction in productive environments: A review. 14th International Work-Conference on Artificial Neural Networks (2017).
[ BibTeX ]

@inproceedings{Zamora2017,
  author = {Zamora, Mauricio and Caldwell, Eldon and Garcia-Rodriguez, Jose and Azorin-Lopez, Jorge and Cazorla, Miguel},
  booktitle = {14th International Work-Conference on Artificial Neural Networks},
  title = {Machine learning improves human-robot interaction in productive environments: A review},
  year = {2017},
  month = jun
}

Salvador, Jaime and Ruiz, Zoila and Garcia-Rodriguez, Jose. Big Data Infrastructure: A Survey. 14th International Work-Conference on Artificial Neural Networks (2017).
[ BibTeX ]

@inproceedings{Salvador2017,
  author = {Salvador, Jaime and Ruiz, Zoila and Garcia-Rodriguez, Jose},
  booktitle = {14th International Work-Conference on Artificial Neural Networks},
  title = {Big Data Infrastructure: A Survey},
  year = {2017},
  month = jun
}

Zoila Ruiz, Jaime Salvador and Garcia-Rodriguez, Jose. A Survey of Machine Learning Methods for Big Data. 14th International Work-Conference on Artificial Neural Networks (2017).
[ BibTeX ]

@inproceedings{Ruiz2017,
  author = {Zoila Ruiz, Jaime Salvador and Garcia-Rodriguez, Jose},
  booktitle = {14th International Work-Conference on Artificial Neural Networks},
  title = {A Survey of Machine Learning Methods for Big Data},
  year = {2017},
  month = jun
}

Project

COMBAHO: COMe BAck HOme system for enhancing autonomy of people with acquired brain injury and dependent on their integration into society