Data Visualization Research & Interface Design


Creating a functioning space where humans can begin to interact naturally with computers. 


The Cognitive and Immersive Systems Lab (CISL) is a

multi-year collaboration program between Rensselaer Polytechnic Institute and IBM Research to lead the frontier of research and development in immersive cognitive systems. CISL promotes a culture of multidisciplinary research across science, engineering, art, communications, architecture, and business. Our mission is to create scientific breakthroughs and technical innovations that augment group intelligence in real world scenarios.


Data Visualization Research

Interface Design

Narrative Generation

Use Case Studies

Date   May - December 2017

Role   Data Visualization Designer

Project   IBM Watson's Room

Team   Cognitive & Immersive Systems Lab


The Interaction Loop for Human-Scale Group-Computers Interaction

When considering human-computer interaction, whether in an office with laptops, in a car with a mobile phone, or on a couch with a tablet, an individual user is most often the sole focus of that interaction. However, many situations involve groups of people working together, especially in critical situations involving complex problems. These people often have diverse backgrounds and domain expertise. We can imagine scenarios in corporate board rooms where groups of executives are tasked with allocating capital, or in health centers where teams of doctors determine patient treatment plans.

In most collaborative situations, people communicate using their personal devices and they may share information on a large screen or multiple screens to disseminate information to the group. However, the ability of the group to directly interact with one another’s information is limited by restricted access to personal devices. When additional information is required and not available, that information must be researched and gathered, analysis must be performed and reports must be created for the next meeting.

Given these limited affordance, decisions in these situations are too often made without full knowledge of or insight to pertinent data. Such issues must be resolved to significantly improve the quality and productivity of group decision making. Even with increasingly intelligent computers and data-analytical algorithms, there is inherent difficulty in linking them directly into the group’s wisdom in the context of discussions or decision making in real-time.

The Situation Room


Imagine being in a room to ask questions of one of the world's most powerful computers. An artificial intelligence containing more information than the largest library, it can recognize you, hear you, see what you are pointing at, and even notice if you might be perplexed or inattentive. It knows all of your earlier work and might even anticipate your questions.

Such a room is now being created at Rensselaer Polytechnic Institute under a partnership with technology giant IBM and its supercomputer Watson, which first gained worldwide attention in 2011 when it beat humans in the TV show Jeopardy.

The Situations Room serves as a bridge between groups of individuals working on a task, and the supercomputing services (our Society of Cognitive Agents) those individuals use to make critical decisions. The Situations Room integrates immersive environments—to allow human-scale interaction—and cognitive computing—a digital analysis of the context of group activities, or “mind-of-room.” As an initial step towards understand group dynamics, the interaction between humans and the Situations Room is approached as a long-term multi-round conversation loop with multiple interaction modalities.

We aim to create a Situations Room that allows increasingly powerful computers to interact naturally and directly with group discussions and augment group intelligence in real-time:

  • Summarize meetings automatically

  • Bring real-time analysis of context information to the discussion when person-directed decision making is required

  • Facilitate discussion when people have biases or misunderstand one another
    Interject into discussions as appropriate;

  • Generate new hypothesis

  • Use analogies to create links between seemingly uncorrelated data-sets.

Area 1

Recognize interpersonal communication, identify conversations between humans and computers, and organize this information as input for Area 2.

The idea is to utilize advanced cameras, microphones, and sensors as the eyes and ears the Situations Room. Those elements would capture humans locations, movements, head position, eye focus, and body movements, speech, physical objects, digital objects, and voice in the room. At this level, cognitive computing technologies are applied to room input channels to recognize human speech, gestures, attention, emotion, grouping, and interpersonal interactions.


Eye Movement & Gaze Tracking


Head & Facial Emotion Tracking


Multiple Human Tracking


Hand Gestures Control & Recognition


Voice Recognition & Accent Correction


Multiple Human Tracking - Indoor


Multiple Human Tracking - Outdoor

Within Studio 2, sensors detect human activity, such as a change in the position of an occupant of the room, speech, gesture, and head movement. Absent the CISL architecture, each of the cognitive technologies acts in solitude, responding to a specific activity detected by a single type of sensor and provided to the computer for interpretation. A sensor provides an input, and the computer provides an output. The interaction between human and machine is based on a single action with a finite duration.

Area 2

Interpret and understand group activity and its context, then find relevant information from outside the room which might facilitate the discussion.This step uses the mind of the Situations Room. Cognitive computing technologies are applied to interpret and understand the meaning of human activity, based on input from Area 1.

Based on information derived from the people in the room, cognitive computing will be leveraged behind the scenes to determine additional context not explicitly included in the discussion taking place between participants in the room. With both the context from the conversation in the room and the external context brought to the conversation, cognitive systems will be able to reason and learn from both contexts, find suggestions to present to the participants, and facilitate productive discussions.

The room will be able to address the following questions within the discussion:

  • What is the focus of the discussion? What are its topics and knowledge domains?

  • What types of discussion activities are taking place? Validation, exploration, decisions, or task prioritizations?

  • What are the interpersonal dynamics between discussion participants? What are their relationships and trust levels?

Area 3

Present and communicate the analysis and cognitive viewpoints to discussion participants through creative multimodal narratives. Humans must understand what computational processes are gathered by the computers to analyze context information. What is in the data and why did the computers derive its conclusion(s)?


The Group Meeting Re-imagined: Designing Intelligent Rooms

Facial Recognition &

Auto Check-In

Facial Recognition and auto check-in for group meeting and many other purposes.


Group Interaction

Immersive Smart Room that interacts naturally with the users. Users mix seamlessly with tech.

Natural Control

Natural Interaction with the smart room using voice command, hand gestures, and facial expressions.

Auto Saved Sessions

Everything that was discussed in the meeting will be auto saved in text form to access anytime.


The most memorable way for humans to explain such insight to others is through storytelling, often in a multimodal fashion, with creative analogies, summaries, and references. Research in creative interactive multimodal storytelling will help create multimodal narratives to translate insights and discovery from large amounts of data into advanced visualization and sonification in an immersive environment and allow interaction by humans to further explore the information and thus enable human-in-the-loop analysis. 

Engaging Presentation

Larger scale presentation that allows more users to interact and discuss a new idea.

Virtual Experience

Experience life-size scale design work. Giving the option for a cognitive walk-through before it is actually built.

A group of four people walk into a room and the leader says, “Watson, bring me the last working session.” The computer recognizes and greets the group, then retrieves the materials used in the last meeting and displays them on three large screens. Settling down to work, the leader approaches one screen, and swipes his hands apart to zoom into the information on display. The participants interact with the room through computers that can understand their speech, and sensors that detect their position, record their roles and observe their attention. When the topic of discussion shifts from one screen to another, but one participant remains focused on the previous point, the computer asks a question: “What are you thinking?”



Generative Narrative Agent


Cognitive & Immersive Boardroom


Restaurant Game


Cyber-Enabled Learning

A Virtual Space for Children to Meet and Practice Chinese

This project aims at creating an online space for children to meet virtually and practice Mandarin Chinese. The project will provide a platform for children to talk with each other and communicate using gestures and body languages. This project will investigate how the use of media affects children's communication behaviors and how media can be designed to allow children feeling co-existence with other remote users.

Interactive Storytelling for Foreign Language Learning

Storytelling is an interactive process. Though the audience cannot affect how the story develops, a good storyteller observes the listener's responses and adjusts the emphasis of his/her storytelling accordingly. This project is aimed at creating an automated AI agent for interactive storytelling.

Argument Interactive Narratives with Cognitive Robots

This project aims at making the interaction with virtual characters more natural and more intuitive by combing cognitive robots with digital avatars. Primitive movements of the robots can be used to help express emotions, illustrate locations, and even suggest the social relationships among characters. 

The game features a scene in which players take on the role of a restaurant customer who will converse with a waiter-bot and place orders in a restaurant. The art style will evoke the experience of casual dining at a warmly-lit Chinese restaurant.

Environment art will be displayed on three large screens which are reconfigurable. Food Presentation screen is shown on the table display.


Narrative Generation

Designing Dialog Flow using Diagrams  


Broader Content Experience

The Rensselaer Mandarin Project enables students to improve functional understanding, pronunciation and vocabulary in Mandarin Chinese through authentic speaking situations in a virtual visit to China. Students navigate using speech, gestures, and combinations thereof to navigate an immersive, mixed reality, stylized realism game experience through intelligent interaction with AI agents, immersive technologies, and game mechanics. 

Multi-Participant Experience


The environment was developed in a black box theater equipped with a human-scale 360 Panoramic Display (14’h, 20’r), arrays of markerless skeletal tracking sensors, and speakers for spatialized audio.

Face - to - Face Group Immersion

Existing work in the field suggests that Interactive environments for language learning provide significant benefits to learning a second language when compared to traditional teaching methods. They both improve learning outcomes, and facilitate positive learning attitudes in the classroom. They enable improved understanding and retention of formulaic expressions, persuasive talk, awareness of audience,and collaborative communication as a result of digital immersion inauthentic speaking situations. At the same time, many of existing technologies utilize small displays, and seldom leverage Artificial Intelligence (AI).

Virtual Reality & Mixed Reality

Giving user the experience of presence – feeling “being there” -- is a basic design goal for most virtual environments. The experience of “presence” is affected by many factors. This projects investigates how task and interaction design affect the user's experience of social and physical presence.


Narrative Agent Interface Design_Page_02