Developing Advanced Learning and Instruction Technology Through HCI




This project focuses on developing a new instructional technology by utilizing an innovative computer vision based instruction system. In this system, an advanced graphical scene generation software and a face analysis technology are developed for interacting with computer in order to enhance the teaching and learning experience.


Visual teaching and learning – the use of graphics, images, and animations to enable and enhance teaching and learning – is one important strategy that we have employed. Recent developments in computer graphics, multimedia, and human computer interaction technologies have opened new opportunities for educators to engage students in science, engineering, and math. This project is to develop tools for computer-based virtual avatars and human behavior synthesis and analysis, and generate graphical scene through an HCI setting.


A synthesized graphical avatar is used to interact with a user. It is intended to understand the user’s expressions, eye gazes, head pose, and speech accordingly. The system (so-called iTutor) is composed of 3D face model synthesis, facial expression recognition, pose and gaze estimation, speech recognition, etc.   Here is the illustration of the system for proof-of-concept.


Figure 1: Example of iTutor system based on human-computer interaction



In addition, a so-called iDemo system is developed for scene and object composition and visualization from speech or text, and gesture based interaction for control of visualization. Such a tool allows a structure described by a user to be visualized, potentially facilitating the learning and training process. A prototypical system is illustrated in the following figure. 





Figure 2: Example of iDemo system for graphical scene visualization





1.      M. Reale, P. Liu, L. Yin, and S. Canavan, Art Critic: Multisignal Vision and Speech Interaction System in a Gaming Context, IEEE Transactions on System, Man, and Cybernetics – Part B, vol. 43, No. 6, p1546-1559, Dec. 2013 

2.      X. Zhang, L. Yin, J. Cohn, S. Canavan, M. Reale, A.Horowitz, P. Liu, and G. Girard, BP4D-Spontaneous: A high resolution spontaneous 3D dynamic facial expression database, Image and Vision Computing, 32 (2014), pp.692-706

3.      P. Liu and L. Yin, “Spontaneous Facial Expression Analysis Based on Temperature Changes and Head Motions”, The 11th IEEE International Conference on Automatic Face and Gesture Recognition (FG15), 2015

4.      X. Zhang, U. Ciftci, and L. Yin, Mouth Gesture based Emotion Awareness and Interaction in Virtual Reality, ACM SIGGRAPH (poster program), Aug., 2015

5.      K. Hu and L. Yin, “Multi-scale topological features for hand posture representation and analysis”, 14th IEEE International Conference on Computer Vision (ICCV), December 2013.

6.      P. Liu, M. Reale, and L. Yin, “Saliency-guided 3D head pose estimation on 3D expression models”, 15th ACM International Conference on Multimodal Interaction (ICMI), December 2013.

7.      K. Hu and L. Yin, “Multiple Feature Representations from Multi-Layer Geometric Shape for Hand Gesture Analysis”,  The 11th IEEE International Conference on Automatic Face and Gesture Recognition (FG15), 2015



We would also like to thank the support from the SUNY IITG, NSF, and NYSTAR.    


Copyright @ GAIC lab, SUNY at Binghamton 2015