Author: Dr Hristijan Gjoreski STSM Period: 2020-01-15 – 2020-02-08
Hosting institution: Jožef Stefan Institute
From ITC: Yes
PURPOSE OF THE STSM
The overall purpose of this STSM was, first to exchange knowledge in the specific Ambient intelligence (AmI) and elderly healthcare domain, then to conduct a research study in the particular domain, and finally to lay the foundations for future collaborations in the domain.
During the stay, I was able to conduct research at the Jozef Stefan Institute (JSI) in the domains of human activity recognition and fall detection with wearable sensors. I investigated how activities and falls can be detected by wearable inertial sensors using machine learning. The conducted research was a good basis for our future work, which includes writing at least 2 conference papers, and potentially 1 journal paper.
Beside the research conducted, during the visit we exchanged knowledge, ideas and we already have plans for future collaborations. I believe that this STSM is beneficial for the AmI and elderly healthcare field in general, since it allowed me to share my expertise in machine learning and wearable sensors analysis with the JSI group – which is highly specialized in this area.
DESCRIPTION OF WORK CARRIED OUT DURING THE STSMS
The description of the work is divides into 2 subsections according to the task: activity recognition and fall detection.
First, I was introduced to the related projects and the previous work of the group at JSI in the domain of human activity recognition. The idea was to study the datasets and the algorithms that were already available at the Jozef Stefan Institute, and to use them in order to develop algorithms beyond state of the art.
We decided that transfer learning is understudied area in the activity recognition domain, and that I can work on this. First, I found related datasets, and then I was working on the learning pipeline that will allow us to transfer knowledge and features from one dataset to another.
First, I did a literature research on the latest publications on transfer learning with time series data, wearable sensors data, and similar. As the most promising approach was with Convolutional Neural Networks (CNNs) because they are able to automatically extract features (knowledge) and the they can be use on another domain/dataset.
I was working with CNNs, in particular deep multimodal Spectro-temporal ResNet (Multi-ResNet), which has already been proved successful for activity recognition in previous study . I analyzed four activity recognition datasets: Skoda, OPORTUNITY, PAMAP and JSI-FOS. Then, I analyzed the number of transferred CNN blocks with respect to the size of the target-adaptation data.
Fall Detection and Prevention
First, state of the art analysis was performed, in the are of fall detection and prevention with wearable sensors. The analysis showed that there are quite a few systems that are based on wearable inertial sensors, and that the successfully detect the human fall. On the other hand, there are only few studies that deal with the fall prevention , i.e., to estimate the risk for falling few days in advance, and this way to prevent the fall from happening. Therefore, I decided to focus on the fall prevention task.
The search for related studies showed that there is a study that claims that the fractal dimension of the movement of the person, can help in the estimation of the risk of fall . The higher the fractal dimension in relative way (day to day comparison), higher the risk of fall. On the other hand, there is no study that calculates the fractal dimension from wearable sensors and IMUs. Thus, we decided to develop a ML algorithm that will calculate the fractal dimension from wearable IMUs. To do this, I designed a data collection protocol, so that I can collect the data, so that I can develop the algorithm.
 M. Gjoreski et al. (2019). Classical and deep learning methods for recognizing human activities and modes of transportation with smartphone sensors. Information Fusion, Under review.
 Kearns et al. Path tortuosity in everyday movements of elderly persons increases fall prediction beyond knowledge of fall history, medication use, and standardized gait and balance assessments. Journal of the American Medical Directors Association, 2012.
DESCRIPTION OF THE MAIN RESULTS OBTAINED
The experiments performed for the activity recognition study on the four datasets, can be seen in Figure 1. It shows the F1-score for each Target (T) – Source (S) dataset combination. The first column represents the number of instances in the adaptation subset. N represents the number of CNN blocks transferred from the S model to the T model. B shows the F1-score of the domain-specific baseline model. The results show that transfer learning using small adaptation subsets is more useful when the target domain contains a small number of different activities. Furthermore, the similarity between the domains, participating in the transfer learning scenario, seems to play a role in its success. The experiments use very little data, i.e., starting from 20 instances (near 1 minute of data) to 2087 instances (near 75 minutes of data). As a consequence, the overall accuracy varies a lot. In future, the plan is to test additional end-to-end DL architectures and to compare their performance. Also, multiple datasets can be used as a source, and this way to learn more general model.
Fall Detection and Prevention
The work performed for the human fall prevention and analysis, is still in its first phase. We were able to study the state of the art, then to define the data collection protocol (the activity scenario, the sensor equipment). Currently we are working on the technical implementation of the software that will collect the data and synchronize them. Smartwatches, smartphones and dedicated inertial sensors (accelerometers, gyroscopes, magnetometers) placed on different body locations will be used (left and right hand, trouser pockets, ankles, etc.). Afterwards, we will work on the development of the algorithm that will calculate the fall risk factor, by calculating the fractal dimension from the wearable sensors data.
As mentioned before, we are in the process of preparing 2 conference papers, one for elderly fall prevention, and the second one for human activity recognition with transfer learning. The potential venues for which we aim are:
– The ACM international joint conference on pervasive and ubiquitous computing (Ubicomp)
– International Symposium on Wearable Computing – ISWC
– International Conference on Activity and Behavior Computing
Potentially, one of the works can be further improved and submitted to a journal. We plan to further improve the algorithms and finish the studies, and potentially start working on other joint publications in this domain.