상세정보

First-person interaction recognition is a challenging task due to unstable video conditions from a camera wearer’s movement. For human interaction recognition from a first-person viewpoint, this paper proposes the three-stream fusion network with two main parts: three-stream architecture and three-stream correlation fusion. The three-stream architecture captures characteristics of the target appearance, target motion, and camera ego-motion. The three-stream correlation fusion combines three feature maps of each stream to consider correlations between the target appearance, target motion, and camera ego-motion. The fused feature vector is robust to the camera movement and compensates for the noise of camera ego-motion. Short-term intervals are modeled with the fused feature vector, and the LSTM considers the temporal dynamics of videos. We evaluated the proposed method on two public benchmark datasets to show the effectiveness of our approach. In the experiments, we showed that the proposed fusion method successfully generated a discriminative feature vector, and our network outperformed all competing activity recognition methods in first-person videos where a lot of camera ego-motion occurs.

1. Introduction 1
2. Related Work 4
3. Three-Stream Fusion Network 7
3.1. Three-Stream Architecture 7
3.2. Three-Stream Correlation Fusion 9
3.3. LSTM for Classification  12
4. Experiments 14
4.1. Datasets 14
4.2. Implementation Details 16
4.3. Performance Evaluations 18
4.4. Three-Stream Correlation Fusion Evaluations 22
5. Conclusion for Research 26