Summary

The IoT x MR application emerged as a pioneering endeavor during the Holohacks event in San Francisco, orchestrated by the Microsoft Hololens Team. This proof of concept app was conceived with the ambition to usher in a 3D spatial User Interface (UI) that facilitates interactions across diverse platforms—namely an Android Music Player app and an Arduino-controlled fan and light—through gaze and gesture control enabled by Hololens. This innovative endeavor underscores the potential of creating a nuanced interaction paradigm among devices that, albeit uncorrelated in the real world, can forge a unique interactive nexus in the Mixed Reality (MR) realm. The app further enthralls users with a Zen mode, offering an immersive MR experience as users interact with virtual objects intertwined with the real-world ambiance. The project was engineered by Ozyphus and designed by Ian Sterling.

HoloLens Concept Offers Mixed Reality Smart Home Controls | Digital Trends

Demo

Hololens

https://www.youtube.com/watch?v=vwew8QrInLY

Magic Leap

https://www.youtube.com/watch?v=oSHcLMY1oNE

Technical Highlights:

1. Development Environment:
   • The application was crafted using Unity and was subsequently deployed onto Hololens through Visual Studio, establishing a robust development and operational environment.
2. Networking:
   • The Universal Windows Platform (UWP) networking packages were employed to ensure seamless communication and interactions across platforms.
3. User Interaction:
   • Hololens' intrinsic gaze and gesture control mechanisms were harnessed to enable intuitive user interactions within the 3D spatial UI.
4. Scripting and Programming:
   • The interactivity scripts crucial for Hololens operations were meticulously programmed in C# to ensure precise and responsive user interactions.
5. Cross-Platform Server Implementation:
   • A TCP server, essential for communication, was implemented in Java for the Android Music Player app and in Arduino sketch for the Arduino-controlled fan and light, each equipped with the requisite functionalities for the envisioned interactions.
6. Network Configuration:
   • A unified access point was established to interlink all devices, ensuring coherent and synchronized operations across the network.
7. 3D Spatial UI and Zen Portal Placement:
   • The Hololens coordinate system was leveraged to meticulously position the 3D spatial UI and the Zen portal within the MR environment.
8. TCP Communication:
   • Upon any UI interaction, a corresponding TCP request is dispatched to the pertinent server, ensuring accurate and timely responses to user inputs.
9. Zen Mode Activation:
   • The user's positional relation to the portal serves as the criterion to toggle the Zen mode, enriching the user experience with a seamless transition between standard and Zen modes.

Conclusion