Anthropomorphic AI: A Toolkit for Authoring Intelligent Virtual Agents in Extended Reality

Anthropomorphic AI is an open-source research toolkit for authoring and interacting with embodied Intelligent Virtual Agents (IVAs) in extended reality (XR). The toolkit, authored by Ke Li and last updated on 2026-04-20, provides rich multimodal capabilities including speech, gaze, gestures, facial expressions, and vision. It was evaluated through four use case demonstrations and two pilot evaluations in immersive VR.

Use Cases

• Authoring embodied IVAs with customizable nonverbal cues based on the toolkit's support for speech, gaze, gestures, and facial expressions.
• Integrating different foundation models and speech-to-text/text-to-speech methods for IVA interaction based on the system's modular architecture.
• Developing natural human-IVA interactions in XR based on integrated features like proximity detection and trajectory-based action recognition.
• Creating vision-based multimodal prompts for IVAs based on the toolkit's vision capabilities mentioned in the description.

Strengths

• The toolkit is described as open-source and flexible, allowing for customization of agent behavior.
• It integrates multiple rich multimodal capabilities, including speech, gaze, gestures, facial expressions, and vision.
• Evaluation included four use case demonstrations and two pilot studies, suggesting practical validation.

Limitations

• The dataset is a 77.2 KB PDF file, indicating it is documentation or a paper, not a primary data corpus.
• Row count and column-level documentation are absent; data content and structure must be inferred from the document.
• Freshness should be verified as the last update date is 2026-04-20, which is in the future relative to the current date.

Provenance

Source

Ke Li via figshare

Collection Method

Toolkit development and evaluation, method details are described in the accompanying document.

Freshness

Last updated 2026-04-20 09:51:04