This example displays the basic usage of configuring 360° video sources. WebXR and Native Cardboard mode support is added with special WebXR events for debugging.

Introduction

360° feature has been developed and improved over many years keeping up with changing browser apis , requirements and IOS problems introduced.

This feature has been updated to accomodate the new WebXR api. WebXR is now the new standard for WebVR which Chrome currently supports. The new orientation sensor api available in Chrome browsers in Android has been added.

The rendering of the video makes use of WebGL in browsers that support it, utilising the graphics card available. NVidia, Radeon and Integrated GPU's will render better. Any Html5 video will be supported including Mpeg Dash and HLS Native and HLS.JS.

Supports Occulus headsets, Vive, Valve Index, mixed reality headsets and Apple Vision headset.

Cubemap Video Support

According to the investigations and work at Facebook https://code.facebook.com/posts/1638767863078802/under-the-hood-building-360-video/. Cubemap encoded videos help reduce wasted pixels and therefore bandwidth. The videos frames are tiled into a 3 x 2 formation.

Due to their discoveries equirectangle videos have wasted pixels around the edges and duplicated pixels where it wraps. Cubemaps have defined edges and therefore no redundant pixels are in the video. These are space and performance saving optimisations. Facebook have an Ffmpeg encoder filter called Transform360. Which will help transform Equirectangle sources to Cubemap video sources. The demo was produced using this filter.

Fisheye Camera Support

There is support for both Ricoh Theta dual 360 Fisheye and 180 degree Fisheye support. 180 degree Fisheye cameras like security cameras and Go Pro, angles to lock rotation and position may vary. A configuration is required for the location of the camera.

360 Video Controls

Various user controls are supported for controlling the video camera view. Desktop rotation controls include keyboard key codes, mouse drag and pointer lock controls for rotating and panning, mouse wheel for zooming. An external player api is provide to also control rotation using scripting.

Mobile rotation controls include touch for rotation when not in fullscreen mode. In non VR fullscreen mode mobile orientation apis are used for controlling rotation with the deivce.

In VR headsets mode on Mobile and in WebXR enabled browsers on desktop. WebXR apis are used to control rotation using orientation apis of the VR Oculus or Samsung Gear headsets or the mobile device. On mobile in WebXR cardboard mode it will fallback to the device's orientation.

VR Controllers are supported to control a virtual video controlbar. When connected the Oculus, Quest, Vive and Index gaming controllers will be enabled as a laser pointer.

Thumbstick on VR Controllers can be used to rotate the headset viewport without head turning.

In Quest and Apple Vision headsets with hand tracking support. Hand tracking VR controllers can be used for VR control.

See VR Controller Documentation for how the controllers work with the VR video controlbar.

WebXR Support

VR support for VR headsets like Oculus Rift, Oculus Quest, Apple Vision, Vive, Samsung Gear VR, Google DayDream View, Microsoft Mixed Reality and Google Cardboard is possible with a VR stereo view of the video and autonomous rotating, panning and zooming control using native WebXR/WebVR support if enabled in the browser.

VR headset support is possible with WebXR browser API's found in Chrome, Firefox and Edge browsers. This relies on the Oculus Runtime for Windows, therefore OSX support has been dropped for now. These API's provide controls and status from the headsets which can then be used for controlling the video camera view.

Microsoft Mixed Reality with the Mixed Reality Simulator and supported headsets. Provides WebVR support with the Microsoft Edge browser.

WebXR is available on Android Chrome, Firefox and Samsung Gear VR Browser, this is now enabled by default.

Daydream View requires the Daydream application installed. And Chrome browser with WebVR enabled on Android.

VR Video Controls and subtitles text support is available in WebVR presentation mode.The video controls will display when clicking the VR display. And a reticle can be used for gaze control. Subtitles text will rotate with head tracking.

A flat 2D video Cinema screen mode in VR is supported. When launching from GearVR, Cardoard, or VR headets Occulus, Vive and Windows Mixed Reality. The video will display into a virtual Cinema screen. When subtitles are configured they will also be displayed over the Cinema screen.

Apple Vision Support

Apple vision in WebXR is supported. Work arounds have been implemented for video playback due to a bug with visionOS. visionOS does not have WebXR enabled by default and requires to be enabled in Safari. Enable WebXR.

Vision OS uses a special transient pointer WebXR input source which requires holding a pinch to keep the controller active. Pinching toggles the control bar and a second pinch makes selections.

Video in WebXR has a color gamma bug which should be eventually resolved in an update.

visionOS Safari makes itself known as an Ipad platform making it hard to determine what is a mobile device for mobile controls. A fix is in place to detect for visionOS to not turn on orientation controls or ask for permissions for orientation.

Mobile Support

360 video support is possible on Android with Firefox or Chrome or Safari on IOS.

On Ipad html fullscreen and VR picture in picture is supported.

Screen orientation lock is supported on Android and may be supported in IOS when enabled.

IPhone Support

IPhone does not support native html fullscreen. A Hack has been implemented to show VR rendering as a fullscreen video. Because no html fullscreen is supported orientation landscape lock while in fullscreen is not supported.

The fullscreen video of the VR rendering follows device orientation controls while in fullscreen.

IPhone does not support WebXR and there isn't a polyfill that works with Iphone yet.

Ambisonics Support

Experimental support for 4 channel Ambisonics audio tracks in video is possible through an Ambisonics to Binaural audio decoder. Browsers firstly require mutli channel. The audio requires to be authored according to Google Spatial Media specification.

Amibsonics supports 2nd and 3rd order audio streams. The panning of the camera will pan the audio in headphones.

Producing 360 Videos

360° video can be produced using special surround cameras like ones supplied by https://www.360rize.com/.

The camera sources are required to be stitched together after into a panoromic flat spherical picture using special VR 360° software. Software like http://www.video-stitch.com/.

Live 360 is possible using a device like a Teradeck Sphere. This will publish to Wowza streaming servers which can then be packetised into Mpeg Dash and HLS.

Video Capture Support

Using the Snapshot Plugin, support is available to capture VR Video rendering.

VR Picture in picture support

Not normally supported by default. Features have been implemented to support picture in picture toggling of the VR rendering. This is a unique feature requiring hacks to show VR rendering as picture and picture. In Safari such hacks can break at any stage requiring work arounds when needed.

WebXR Headset Offsets

There is a bug with WebXR where the camera will start 180 degrees from the default position. An offset on the Y rotation is applied to look to the front of camera.

With a config xrTiltOffset, the X rotation offset will match the tilt up and down rotation of the default camera. Useful for sitting in a chair.

Features

• 360° video support in any Html5 supported format including Mp4, Webm, Ogv, Mpeg Dash and HLS.
• Efficient , bandwidth and space saving Cubemap video support.
• 180 degree video rendering support. This will render in a half sphere with head turn rotation.
• Various user control functions for the video rotation including keyboard, mouse, tablet orientation and movements, tablet touch, external api controls and a movement pad area. In WebXR/WebVR mode orientation controls are provided by the headsets.
• XR headset support for Oculus Rift, Vive when used with a XR API enabled browser like Chrome and Firefox nightly.
• Standalone XR headset support for Oculus Quest and Apple Vision Pro with WebXR flags enabled in the browser.
• WebVR support for Microsoft Mixed Reality Simulator and supported headsets using Microsoft Edge browser.
• Support for Samsung Gear VR is through the Samsung Internet or Oculus VR Browser which also supports WebVR after being enabled.
• Google Daydream View support is available with Chrome for Android with the Daydream app installed.
• Cardboard mode support for non WebVR browsers is available using a customised reduced version of WebXR Polyfill. This is optional and loaded on demand.
• Full WebXR display events provided.
• Skeleton Cordova application supplied for IOS and Android based apps. Cordova WebView apps are required to work around IOS Iphone inline video limitations.
• Support for IOS "pinned to Home Screen" standalone WebView apps for inline playback support on Iphone.
• Pseudo WebVR support for Cardboard uses mobile orientation controls in VR mode or uses WebVR controls if enabled in Chrome and Firefox on Android.
• Stereoscopic preview mode for desktop viewing.

• VR headset toggle button to go into VR fullscreen / VR display presentation mode and back to normal mode.

• When in WebVR headset presentation mode. A status splash page will be displayed.

• Mobile landscape orientation screen lock support for browsers that support it in VR mode.
• Android and IOS wakelock support to prevent the screen going to sleep in VR mode.
• Orbital navigation dock area for not only controlling on the video display itself with a mouse.
• Hacks required for mouse controls to prevent the player toggling the video.
• proxy sources for Html5 non cross origin standards compliant browsers like Safari, IE and Android default browser.
• Dock button for switching to the proxy source of the VR video.
• Stereoscopic side by side video support for 2D picture.
• Playlist support between non VR and standard videos to switch between rendering views.
• Pseudo ratio scaling of the WebGL rendered video. VR mode will scale to fit the screen.
• Works with any Html5 based streaming including Mpeg Dash and HLS.JS.
• Ambisonics surround audio support for rotating with the video.
• Mouse pointer lock controls features to rotate the camera with a mouse pointer rather than mouse drag controls.
• HLS support and fixes for Safari.
• Flat 2D video in VR Cinema screen mode support for VR headsets and Cardboard.
• Subtitle text overlay support for Cinema screen and 360 mode.
• XR Video controlbar support in WebVR presentation mode.
• XR Controller support to interact with the virtual controlbar.
• Dual 360 Fisheye and Single 180 Fisheye support.
• IOS 13 orientation api permissions check on user gesture.
• IOS 13 fullscreen and cardboard orientation.
• Using the Snapshot Plugin, support is available to capture VR Video rendering.
• Picture in picture toggling support for VR rendering. Works on Desktop, Android and Ipad.
• WebXR Headset offset support.
• WebXR viewport rotation transform using the grip controller thumbstick.
• Vision OS transient pointer XR input source support.
• Hack provided for Iphone native VR rendering fullscreen support. This is a unique feature to show VR in native fullscreen and easily breaks so work arounds are required when needed.
• Provide a websocket proxy logger for Apple Vision to work around web inspect bugs.
• Fixed visionOS detection to not turn on mobile controls when going fullscreen.

Known Issues:

• visionOS Safari makes itself known as an Ipad platform making it hard to determine what is a mobile device for mobile controls. A fix is in place to detect for visionOS to not turn on orientation controls or ask for permissions for orientation.
• Windows Mixed Reality headsets produce a graphics bug fault when launching. A message may be displayed for the user to launch again.
• IOS 13 requires a gesture to accept orientation api permissions. This happens on first user gesture.
• IOS before 13 lacks html5 video fullscreen api. A pseudo fullscreen in normal and Cardboard VR mode has been provided.
• IOS lacks screen orientation lock api when going fullscreen or pseudo VR Cardboard mode
• IOS lacks WebXR/WebVR api and requires the Cardboard polyfill to be enabled.
• Multi channel audio support works on most browsers apart from IOS. Therefore IOS requires a stereo audio source instead.
• Safari and IE 11 does not support pointer lock browser api's therefore the pointer lock controls will not display.
• HLS rendering in IOS has been fixed in IOS 11.3 and above.
• Bug fixed with HLS rendering on IOS due to dimensions not reported on metadata. This might be unstable still.
• Iphone does not support native fullscreen so a hack has been implemented for VR fullscreen. It keeps breaking due to webkit restrictions requiring work arounds where needed.
• Vision OS has a gamma color bug for video when in WebXR immersion mode. It should resolve itself on an update.
• Safari now reports to support webm but needs to be filtered for WebGL rendering which it doesn't support.
• Vision OS 1.1 currently has a web inspect bug. A websocket proxy logger is provided for debug and development.

More Information:

• https://code.facebook.com/posts/1638767863078802/under-the-hood-building-360-video/
• https://www.360rize.com/
• http://www.video-stitch.com/
• http://www.panocam3d.com/software.html
• Ionic Framework
• Apache Cordova
• Samsung Gear VR
• Oculus Rift
• HTC Vive
• Mozilla VR
• Google Spatial Media
• Omnitone
• Windows Mixed Reality