Developing
Advanced Learning and Instruction Technology Through
HCI
DESCRIPTION
This project focuses on developing a new instructional
technology by utilizing an innovative computer vision based instruction system.
In this system, an advanced graphical scene generation
software and a face analysis technology are developed for interacting with
computer in order to enhance the teaching and learning experience.
Visual teaching and learning – the use
of graphics, images, and animations to enable and enhance teaching and learning
– is one important strategy that we have employed. Recent developments in
computer graphics, multimedia, and human computer interaction technologies have
opened new opportunities for educators to engage students in science,
engineering, and math. This project is to develop tools for computer-based
virtual avatars and human behavior synthesis and analysis, and generate
graphical scene through an HCI setting.
A synthesized graphical
avatar is used to interact with a user. It is intended to understand the user’s
expressions, eye gazes, head pose, and speech accordingly. The system
(so-called iTutor) is composed of 3D face model
synthesis, facial expression recognition, pose and gaze estimation, speech
recognition, etc. Here is the
illustration of the system for proof-of-concept.
Figure 1: Example of iTutor system
based on human-computer interaction
In addition, a so-called iDemo
system is developed for scene and object composition and visualization from
speech or text, and gesture based interaction for control of visualization.
Such a tool allows a structure described by a user to be visualized,
potentially facilitating the learning and training process. A prototypical
system is illustrated in the following figure.
Figure 2: Example of iDemo system
for graphical scene visualization
PUBLICATIONS:
1. M. Reale, P. Liu, L. Yin, and S.
Canavan, Art Critic: Multisignal Vision and Speech Interaction
System in a Gaming Context, IEEE
Transactions on System, Man, and Cybernetics – Part B, vol. 43, No. 6,
p1546-1559, Dec. 2013
2.
X. Zhang, L. Yin, J. Cohn, S. Canavan, M. Reale, A.Horowitz, P. Liu, and G.
Girard, BP4D-Spontaneous: A high resolution
spontaneous 3D dynamic facial expression database, Image and Vision Computing, 32 (2014), pp.692-706
3. P. Liu and L. Yin, “Spontaneous Facial Expression Analysis
Based on Temperature Changes and Head Motions”, The 11th IEEE International Conference on Automatic Face and
Gesture Recognition (FG15), 2015
4. X. Zhang, U. Ciftci,
and L. Yin, Mouth Gesture based Emotion Awareness
and Interaction in Virtual Reality, ACM
SIGGRAPH (poster program), Aug., 2015
5. K. Hu and L. Yin, “Multi-scale topological features for hand
posture representation and analysis”, 14th
IEEE International Conference on Computer Vision (ICCV), December 2013.
6. P. Liu, M. Reale, and L. Yin, “Saliency-guided 3D head pose estimation on 3D expression
models”, 15th ACM
International Conference on Multimodal Interaction (ICMI), December 2013.
7. K. Hu and L. Yin, “Multiple Feature Representations from
Multi-Layer Geometric Shape for Hand Gesture Analysis”, The 11th
IEEE International Conference on Automatic Face and Gesture Recognition (FG15),
2015
Acknowledgement:
We would also like to
thank the support from the SUNY IITG, NSF, and NYSTAR.
Copyright
@ GAIC lab, SUNY at