Poor performance with osgXR

[opalomaresg]

Hi!
I have been trying to use osgXR with Quest3 on Windows, to adapt an old parachuting simulator application based on openSceneGraph with Oculus Rift to the new Quest 3 headset, but I encountered several problems that I was trying to fix:

• Low performance of the application with osgXR (tipically ~18 FPS, or [30-60 FPS] in best case), which causes the image in the headset to have some delay compared to the movement of the headset.
• The mirror image representation in the monitor is not updated when the image is displayed in the headset.
• (...)
  I know that this project is still in development and there are still issues to resolve, but I don't know if you are also experiencing these problems. I tried to address some of these issues while adding the capability to show the passthrough image from the cameras of the Quest 3 headset.

On the other hand, I tried to modify the osgOculus library code on which my simulator was based to incorporate the openXR API and thus access the passthrough camera images and integrate them into the virtual scene. Although I have dedicated little time to it, the result has been quite good (good application performance ~72 Hz, correct representation of the mirror image on the monitor window, integration of the passthrough image into the virtual scene, etc.).
Since you have shared this project with me and I see that you are still working on it, I am willing to share my solution in case it might be useful to you.

Greetings!

[amalon]

Thanks for your comments here. I don't have a Quest 3 headset, but I have experienced some performance issues that I haven't resolved to my satisfaction, so I'm all ears if you have any ideas or can provide any insight where they may be stemming from when used with quest3.

Currently the rendering of the VR view and the desktop view are linked together which I know can cause undesired delays when VSync is enabled, and I know it can also be tricky to convince OSG to pipeline the frames properly.

Currently I'm not using pass-through for anything. I presume your solution uses some openxr extension to get the pass through camera image in order to compose the VR view on top of it?

[opalomaresg]

That's right, I use the 'XR_FB_passthrough' extension to incorporate the camera images because I was interested in it for my simulator. I don't actually get an object or a texture with those images to manipulate them, but I do get an XrPassthroughLayerFB that I can use to integrate it into my scene through the layer composition. My next step is to use the depth map image (obtained from 'XR_META_environment_depth' extension) to perform an image fusion based on the distance of real-world objects to the camera of the headset.
In addition, my simulator does not use the Quest 3 controllers, so the handling of these inputs is disabled.
I don't have my solution published anywhere, but if you're interested I can send it to you.

[amalon]

Yes, I'd be interested if you're able to share it with me. Thanks

[opalomaresg]

Hello James. Here I send you a wetransfer link for the ZIP that contains my osgOculus_openXR project (and takes up 37 Mb), which contains the integration of osgOculus (https://github.com/bjornblissing/osgoculusviewer/tree/master/src) with openXR (https://github.com/meta-quest/Meta-OpenXR-SDK) to be able to see the virtual scene based on OSG with a good refresh rate, good representation of the mirror image in the monitor window, and including an integration of the passthrough image to the virtual scene (maybe you have to disable (comment out) the passthrough part if you use a headset that doesn't have it to prevent an error from occurring during execution. I haven't tried it with any other headset other than the Quest 3 yet). CODIGO_osgOculus_openXR.zip: https://we.tl/t-cqwAS3UNQe There is a lot commented out code from the previous version that could be removed. I also haven't bothered to manage state changes in the headset, which could cause openXR function calls to fail at runtime. I've included the openSceneGraph libraries (many of which surely are not necessary) and the SCRCPY.exe application within COTS, which allows me to display the mirror image of the headset, including the passthrough image. The Visual Studio solution is compiled with VS_2019. If you have a different compiler, you'll need to open it with VS and select the platform toolset you have installed for it to compile and work. If you encounter any problems, let me know. Best regards, Óscar Palomares