Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition

We introduce SynSE, a novel syntactically guided generative approach for\nZero-Shot Learning (ZSL). Our end-to-end approach learns progressively refined\ngenerative embedding spaces constrained within and across the involved\nmodalities (visual, language). The inter-modal constraints are defined between\naction sequence embedding and embeddings of Parts of Speech (PoS) tagged words\nin the corresponding action description. We deploy SynSE for the task of\nskeleton-based action sequence recognition. Our design choices enable SynSE to\ngeneralize compositionally, i.e., recognize sequences whose action descriptions\ncontain words not encountered during training. We also extend our approach to\nthe more challenging Generalized Zero-Shot Learning (GZSL) problem via a\nconfidence-based gating mechanism. We are the first to present zero-shot\nskeleton action recognition results on the large-scale NTU-60 and NTU-120\nskeleton action datasets with multiple splits. Our results demonstrate SynSE's\nstate of the art performance in both ZSL and GZSL settings compared to strong\nbaselines on the NTU-60 and NTU-120 datasets. The code and pretrained models\nare available at https://github.com/skelemoa/synse-zsl\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC