To gauge insights on how people natually interact with objects and environments and deal with interruptions and constraints during daily activities, we chose 56 most representative locations to observe and photograph peoples' behaviors. (For private and semi-private locations, we used online resources.)

For the purpose of data analysis, we spent two months designing and revising the labeling framework to streamline the process of image annotation. Our labeling framework is far more complicated than sole object identifying. By measuring the relationship and status of variables of interest, the framework incorporates three situational factors: context, modality, and social acceptability (Xia, 2022).

Using React.js and Firebase, we built our own data encoding system. We filled in a list of fields under different categories [Location, Spectators, Modalities, Demographics, and Posture] that describe critical information in the image.
Our aspiration is that beyond internal use, any researchers/encoders can use our UI and documentations to feed data to the library.

Before a full investment in implementing gesture commands and UIs in our VR environment, we conducted a few rounds of user elicitation and Wizard of Oz testing to gather insights on the intuitiveness, naturalness, and functionablitiy of the interactive system that we have designed.

To accomodate the difference in degrees of freedom and dexterity across modalities, we storyboarded and designed a compensation strategy that converts traditional UIs and keyboard into “flippable” sub-UIs for easier manipulation.

Using Unity, we based our testing scenario and storyline at a museum setting. The avatar's physical availability can be preset or changed by the user's action decision, such as grabbing a heavy briefcase or sitting on a chair. For now, we focus on commands performed by hands, feet, and head.