File Download
There are no files associated with this item.
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1007/s11042-023-17788-3
- Scopus: eid_2-s2.0-85179301800
- Find via
Supplementary
-
Citations:
- Scopus: 0
- Appears in Collections:
Article: VT-BPAN: vision transformer-based bilinear pooling and attention network fusion of RGB and skeleton features for human action recognition
Title | VT-BPAN: vision transformer-based bilinear pooling and attention network fusion of RGB and skeleton features for human action recognition |
---|---|
Authors | |
Keywords | Feature fusion Microsoft kinect camera Multi-head pooling attention Self-attention Vision transformer |
Issue Date | 11-Dec-2023 |
Publisher | Springer |
Citation | Multimedia Tools and Applications, 2023 How to Cite? |
Abstract | Recent generation Microsoft Kinect Camera captures a series of multimodal signals that provide RGB video, depth sequences, and skeleton information, thus it becomes an option to achieve enhanced human action recognition performance by fusing different data modalities. However, most existing fusion methods simply fuse different features, which ignores the underlying semantics between different models, leading to a lack of accuracy. In addition, there exists a large amount of background noise. In this work, we propose a Vision Transformer-based Bilinear Pooling and Attention Network (VT-BPAN) fusion mechanism for human action recognition. This work improves the recognition accuracy in the following ways: 1) An effective two-stream feature pooling and fusion mechanism is proposed. The RGB frames and skeleton are fused to enhance the spatio-temporal feature representation. 2) A spatial lightweight multiscale vision Transformer is proposed, which can reduce the cost of computing. The framework is evaluated based on three widely used video action datasets, and the proposed approach performs a more comparable performance with the state-of-the-art methods. |
Persistent Identifier | http://hdl.handle.net/10722/345581 |
ISSN | 2023 Impact Factor: 3.0 2023 SCImago Journal Rankings: 0.801 |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Sun, Yaohui | - |
dc.contributor.author | Xu, Weiyao | - |
dc.contributor.author | Yu, Xiaoyi | - |
dc.contributor.author | Gao, Ju | - |
dc.date.accessioned | 2024-08-27T09:09:48Z | - |
dc.date.available | 2024-08-27T09:09:48Z | - |
dc.date.issued | 2023-12-11 | - |
dc.identifier.citation | Multimedia Tools and Applications, 2023 | - |
dc.identifier.issn | 1380-7501 | - |
dc.identifier.uri | http://hdl.handle.net/10722/345581 | - |
dc.description.abstract | <p>Recent generation Microsoft Kinect Camera captures a series of multimodal signals that provide RGB video, depth sequences, and skeleton information, thus it becomes an option to achieve enhanced human action recognition performance by fusing different data modalities. However, most existing fusion methods simply fuse different features, which ignores the underlying semantics between different models, leading to a lack of accuracy. In addition, there exists a large amount of background noise. In this work, we propose a Vision Transformer-based Bilinear Pooling and Attention Network (VT-BPAN) fusion mechanism for human action recognition. This work improves the recognition accuracy in the following ways: 1) An effective two-stream feature pooling and fusion mechanism is proposed. The RGB frames and skeleton are fused to enhance the spatio-temporal feature representation. 2) A spatial lightweight multiscale vision Transformer is proposed, which can reduce the cost of computing. The framework is evaluated based on three widely used video action datasets, and the proposed approach performs a more comparable performance with the state-of-the-art methods.</p> | - |
dc.language | eng | - |
dc.publisher | Springer | - |
dc.relation.ispartof | Multimedia Tools and Applications | - |
dc.subject | Feature fusion | - |
dc.subject | Microsoft kinect camera | - |
dc.subject | Multi-head pooling attention | - |
dc.subject | Self-attention | - |
dc.subject | Vision transformer | - |
dc.title | VT-BPAN: vision transformer-based bilinear pooling and attention network fusion of RGB and skeleton features for human action recognition | - |
dc.type | Article | - |
dc.identifier.doi | 10.1007/s11042-023-17788-3 | - |
dc.identifier.scopus | eid_2-s2.0-85179301800 | - |
dc.identifier.eissn | 1573-7721 | - |
dc.identifier.issnl | 1380-7501 | - |