May 2023 - Aug 2023
CyLab, Carnegie Mellon University
Spatial Computing Researcher
Designing immersive interfaces for real-time human-computer interaction in spatial environments
Project Overview
Working with Professor Anthony Rowe at CyLab, I contributed to cutting-edge research in real-time human pose estimation and tracking. This project served as the foundation for my journey into Computer Vision and Human-Computer Interaction (HCI).
My work focused on creating an immersive, real-time VR/AR experience by integrating depth cameras, LiDARs, 3D modeling, and game engines. The system streamed a live representation of a workspace environment, including digital twins of robotic arms and human subjects, allowing users to remotely interact with the environment as if physically present.
This research explored the future of remote collaboration, teleoperation, and spatial computing interfaces, with applications in fields ranging from remote surgery to hazardous environment operations.
My Role
- •Human Pose Detection & Tracking
- •3D Environment Visualization
- •VR/AR Interface Design
- •Real-time Data Processing
Technologies Used
The Challenge
Creating immersive spatial computing interfaces presents several significant technical and design challenges:
- 1Real-time Performance: Processing and transmitting 3D data with minimal latency is crucial for immersive experiences
- 2Accurate Tracking: Human pose estimation must be precise enough for meaningful interaction
- 3Intuitive Interfaces: Users need to understand how to interact with virtual objects naturally
- 4System Integration: Multiple hardware and software components must work together seamlessly
Technical Approach
Capturing Human Movement in Real-time
I developed a system that uses depth cameras and computer vision algorithms to detect and track human subjects in real-time. Key components included:
- •Multi-camera calibration for comprehensive spatial coverage
- •Skeletal tracking with 25 joint points for detailed pose estimation
- •Temporal filtering to reduce jitter and improve tracking stability
- •Occlusion handling to maintain tracking when body parts are hidden
Creating Digital Twins of Physical Spaces
I created a system to generate and update digital representations of physical environments in real-time:
- •LiDAR-based 3D scanning to create accurate spatial maps
- •Dynamic object tracking to update the position of movable items
- •Texture mapping from RGB cameras to enhance visual fidelity
- •Optimized mesh generation for real-time rendering performance
Creating Intuitive Spatial Interactions
I designed the interface that allowed users to interact with the virtual environment:
- •Gesture recognition for natural interaction with virtual objects
- •Spatial UI elements that integrate with the 3D environment
- •Visual feedback systems to indicate system state and interaction possibilities
- •Multi-user presence representation for collaborative scenarios
User Experience Design
The final spatial computing interface integrated intuitive gesture controls, real-time feedback systems, and multi-user presence to create an immersive environment for remote collaboration and interaction.
Gesture Recognition
I designed a set of intuitive hand gestures for manipulating virtual objects, calibrated to be easily learned yet distinct enough to prevent accidental activation.
Visual Feedback
I implemented a comprehensive visual feedback system using color, animation, and spatial cues to indicate interactive elements and system status.
Multi-User Presence
I created avatar representations for remote collaborators with real-time pose updates and proximity-based interaction capabilities.
Results & Impact
The spatial computing interface demonstrated significant potential for remote operation and collaboration scenarios. Key achievements included:
- 1Low Latency Interaction: Achieved end-to-end latency of under 100ms, creating a responsive feel
- 2Accurate Pose Tracking: Skeletal tracking with average joint position error of less than 2cm
- 3Intuitive Control: 90% success rate for first-time users completing a series of manipulation tasks
- 4Scalable Architecture: System successfully tested with up to 4 simultaneous remote users
End-to-end system latency
First-time user task success rate
Average joint position error
Simultaneous remote users
"This spatial computing interface represents a significant advancement in remote collaboration technology. The system's ability to create an immersive, responsive environment opens up new possibilities for applications in hazardous environments, remote surgery, and distributed team collaboration."
Prof. Anthony Rowe
Professor, CyLab
Potential Applications
Remote Surgery
Enabling surgeons to perform procedures from a distance with precise control and spatial awareness.
Hazardous Environment Operations
Allowing operators to control robots in dangerous locations while maintaining situational awareness.
Distributed Team Collaboration
Creating shared virtual workspaces for teams to collaborate on physical or digital projects remotely.
Lessons Learned
- •Latency Management: Even small delays can significantly impact user experience in spatial computing. Optimizing every part of the pipeline is essential.
- •Intuitive Gestures: Natural gestures that map to physical world interactions are more quickly adopted than abstract control schemes.
- •Sensor Fusion: Combining data from multiple sensors provides more robust tracking but requires careful calibration and synchronization.
- •User Comfort: Extended use of VR interfaces requires careful attention to ergonomics and cognitive load to prevent fatigue.
Future Work
- •Haptic Feedback: Integrating force feedback gloves to provide tactile sensation when interacting with virtual objects.
- •AI-Assisted Interaction: Implementing intelligent systems to predict user intent and assist with complex manipulations.
- •Mobile Deployment: Adapting the system for use with lightweight, mobile VR/AR headsets for field operations.
- •Cross-Platform Support: Extending the interface to work across different VR/AR hardware platforms.
Explore My Other CMU Projects
Check out my work with Xu Lab and the Abstract Lab at Carnegie Mellon.