TY - JOUR
T1 - Discriminative video pattern search for efficient action detection
AU - Yuan, Junsong
AU - Liu, Zicheng
AU - Wu, Ying
N1 - Funding Information:
This work was supported in part by the Nanyang Assistant Professorship to Dr. Junsong Yuan, US National Science Foundation grant IIS-0347877, IIS-0916607, and the US Army Research Laboratory and the US Army Research Office under grant ARO W911NF-08-1-0504. The authors thank Dr. Yan Ke, Dr. Cha Zhang, and Dr. Zhengyou Zhang for helpful discussions, and Liangliang Cao for the help on the experiments of the TRECVID data set.
PY - 2011
Y1 - 2011
N2 - Actions are spatiotemporal patterns. Similar to the sliding window-based object detection, action detection finds the reoccurrences of such spatiotemporal patterns through pattern matching, by handling cluttered and dynamic backgrounds and other types of action variations. We address two critical issues in pattern matching-based action detection: 1) the intrapattern variations in actions, and 2) the computational efficiency in performing action pattern search in cluttered scenes. First, we propose a discriminative pattern matching criterion for action classification, called naive Bayes mutual information maximization (NBMIM). Each action is characterized by a collection of spatiotemporal invariant features and we match it with an action class by measuring the mutual information between them. Based on this matching criterion, action detection is to localize a subvolume in the volumetric video space that has the maximum mutual information toward a specific action class. A novel spatiotemporal branch-and-bound (STBB) search algorithm is designed to efficiently find the optimal solution. Our proposed action detection method does not rely on the results of human detection, tracking, or background subtraction. It can handle action variations such as performing speed and style variations as well as scale changes well. It is also insensitive to dynamic and cluttered backgrounds and even to partial occlusions. The cross-data set experiments on action detection, including KTH, CMU action data sets, and another new MSR action data set, demonstrate the effectiveness and efficiency of the proposed multiclass multiple-instance action detection method.
AB - Actions are spatiotemporal patterns. Similar to the sliding window-based object detection, action detection finds the reoccurrences of such spatiotemporal patterns through pattern matching, by handling cluttered and dynamic backgrounds and other types of action variations. We address two critical issues in pattern matching-based action detection: 1) the intrapattern variations in actions, and 2) the computational efficiency in performing action pattern search in cluttered scenes. First, we propose a discriminative pattern matching criterion for action classification, called naive Bayes mutual information maximization (NBMIM). Each action is characterized by a collection of spatiotemporal invariant features and we match it with an action class by measuring the mutual information between them. Based on this matching criterion, action detection is to localize a subvolume in the volumetric video space that has the maximum mutual information toward a specific action class. A novel spatiotemporal branch-and-bound (STBB) search algorithm is designed to efficiently find the optimal solution. Our proposed action detection method does not rely on the results of human detection, tracking, or background subtraction. It can handle action variations such as performing speed and style variations as well as scale changes well. It is also insensitive to dynamic and cluttered backgrounds and even to partial occlusions. The cross-data set experiments on action detection, including KTH, CMU action data sets, and another new MSR action data set, demonstrate the effectiveness and efficiency of the proposed multiclass multiple-instance action detection method.
KW - Video pattern search
KW - action detection
KW - spatiotemporal branch-and-bound search
UR - http://www.scopus.com/inward/record.url?scp=80051863221&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=80051863221&partnerID=8YFLogxK
U2 - 10.1109/TPAMI.2011.38
DO - 10.1109/TPAMI.2011.38
M3 - Article
C2 - 21339530
AN - SCOPUS:80051863221
VL - 33
SP - 1000
EP - 9999
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
JF - IEEE Transactions on Pattern Analysis and Machine Intelligence
SN - 0162-8828
IS - 9
M1 - 5719621
ER -