The Computer Vision, Perception and Image Analysis (CVPIA) laboratory has been conducting interdisciplinary research in: (i) Artificial Intelligence, Computer Vision and Deep learning, (ii) Assistive Technology, iii) Data Analytics for Biomedicine, (iv) Cognitive Engineering to understand brain connectome and neural disorders and (v) Human-Machine interaction. The common underlying theme is: (i) semantic integration and mining of heterogeneous data, (ii) robust modeling of all possible types of signals (text, speech, images, video, time series and gene expressions), and (v) develop apps & systems to enhance the quality of life of people with disability. Summary of effort:

  • Non-verbal Communication. Understanding and robust modeling of non-verbal communications is one of the central theme of research at the CVPIA laboratory.  We have introduced the idea of “co-analysis of signal and sense” using prosodic relationship between verbal and non-verbal modalities and developed sophisticated methods for the co-analyses of multimodal articulations. This work helped us to obtain a deeper understanding of (a) how the nucleus of an utterance and a visual prosody interact to render the intent of the utterance, and (b) how the synchronization with other modalities affects the production of multimodal co-articulation. These results made it possible to: (a) analyze user’s behavioral expressions that arise while interacting with autonomous agents and to classify such input into categorical emotions as well as epistemic state of mind, (b) test and modify theories to develop affect-enabled assistive and educational technologies.


  • Assistive Technology. CVPIA is leading the project Blind Ambition – an interdisciplinary research and outreach effort pioneering assistive solutions for people with disability.  This research refocuses our understanding of non-verbal communications that spawned a new avenue for developing emotion-enabled systems for assistive solutions (i.e., Expression/Expression+, EmoAssist, i-FEPS, i-MAPS and RMAP). We have adopted the fusion of participatory design, assistive design thinking in developing such systems. Designing an adaptive assistive technology solution requires an understanding of the user’s need as set by their disability and their ability to perform the system-aided task with minimal cognitive effort. To address the challenge, I introduced the notion of assistive thinking – a seamless blend of participatory design, system & design thinking, cognitive ability-demand gaps and collaborative sense making to design assistive solutions.




  • Big Data Analytics. CVPIA lab in collaboration with Physician (Drs. Latif, Zand) are developing algorithms and software tools for text analytics, data stratification, finding networks of semantic associations among different concepts, effective and flexible visualization of information at various levels of granularity and interactive Web services for diverse users in Bioinformatics and Biomedicine.
  • Cognitive Engineering and Science. In collaboration with Drs. Bidelman and Bashivan, We are focusing on developed methods to characterize human WM capacity through both focal analyses as well as functional neural connectivity. We have also developed a novel data structure that preserves the spatial-spectral-temporal information, removes artifacts and learns representation from EEG signals and have been developing data-driven models and network descriptions of cognitive events (i.e., cognitive load, seizure onset zone and 4-D affective space). This is a paradigm shift towards understanding the neural basis of cognitive capacity limits and it offers a critical insight to the network dynamics of human brain connectome.
  • Dialogue Enabled Multimodal Interfaces. Dr. Yeasin was integral part of an interdisciplinary team that developed a number of dialogue-enabled devices, based on natural, multimodal interfaces have the potential of making a variety of information technology tools accessible to a broad spectrum of people. The Co-analysis of modalities to capture the complementary information was key to success of designing robust multimodal systems. A number of speech-gesture enabled porotypes and commercial systems developed over the years during 2000 – 2003. Key lessons learned are: (i) 80% of users had successful interaction, (ii) System running according to specs. 95% of the time, (iii) Users are hard to please and (iv) Multimodal HCI Systems = Large scale software integration.