The Challenge
Interactive Realtime Experience
Designed for Zero Waste Day at the United Nations Headquarters in New York, this project was conceived as a real-time, interactive visual experience driven by visitor movement. The main objective was to move beyond a passive visual installation and create a system where physical presence directly shaped the narrative.
Visitor position, movement, and body data were captured in real time through on-site sensors and translated into inputs that directly influenced digital characters and scene behavior. As visitors entered defined zones, characters were introduced into the scene and the environment transitioned seamlessly into an immersive animated world.
The entire visual environment was built in Unreal Engine and designed to operate fully in real time. Character animations, environmental effects, and scene transitions were generated dynamically rather than relying on pre-rendered sequences, allowing the experience to continuously adapt to live visitor data.
The resulting system was structured as an interactive setup that responds to visitor data in real time, dynamically transforming the flow of the experience as it unfolds.
Equipment
Unreal Engine
Real-time environment setup, character animation systems, environmental effects, and interactive scene controlAzure Kinect Sensors (2 units)
High-precision depth sensing, skeletal tracking, and body tracking for capturing visitor position and movement dataCustom real-time data pipeline
Low-latency integration of depth and skeletal data from Azure Kinect into the Unreal Engine environmentNVIDIA RTX 5090 GPU–based workstation systems
Optimized hardware infrastructure for high-resolution visuals, complex shaders, and stable real-time performanceReal-time system control and synchronization architecture
Centralized management of sensor inputs, animation triggers, and scene logic
Credits
Real-Time Content Design & Production: Sense Department
