Raycasting and Editor Picking

[iamjbq]

I am developing the Jolt integration with O3DE engine, and I have been stuck on the Raycasting portion for a while now. I still am having trouble with the RayCast example in the Architecture doc, whether from exhaustion or maybe just looking at it too long.

In-editor, the engine body class calls RayCast on it's attached engine shape class for any non-static bodies every time the mouse moves over them. This is for Editor mouse picking functionality and the in-params are world space coordinates (there is a dedicated RayCastLocal function that isn't used here). All bodies in-editor are added to the PhysicsSystem but are not activated as we want to be able to query shapes but not have an active simulation.

I've tried both options in the code block below, NarrowPhase and Shape->CastRay and both return no hit. I'm wondering what the correct approach here is? Do I convert the function parameters to local coordinates first and then perform the cast, or can I get a TransformedShape from the body in world coordinates and call CastRay using the parameters as they are?

 AzPhysics::SceneQueryHit Shape::RayCast(const AzPhysics::RayCastRequest& worldSpaceRequest,
        [[maybe_unused]] const AZ::Transform& worldTransform)
    {
        if (m_joltShape)
        {
            if (const bool shouldCollide = worldSpaceRequest.m_collisionGroup.GetMask() & m_collisionLayer.GetMask();
            !shouldCollide)
            {
                return AzPhysics::SceneQueryHit();
            }
            JPH::RayCastResult result;
            
            // Ray for NarrowPhase query
            JPH::RRayCast inRay(
                JoltMathConvert(worldSpaceRequest.m_start),
                JoltMathConvert(worldSpaceRequest.m_direction * worldSpaceRequest.m_distance)
            );
            // Ray for shape
            JPH::RayCast inRay(
                JoltMathConvert(worldSpaceRequest.m_start),
                JoltMathConvert(worldSpaceRequest.m_direction * worldSpaceRequest.m_distance)
            );
            
            inRay.mOrigin -= JoltMathConvert(worldTransform.GetTranslation()); // Conversion in question
            
            // NarrowPhase option
            bool hadHit = m_attachedSystem->GetNarrowPhaseQuery().CastRay(inRay, result);
            
            // Shape option
            bool hadHit = m_joltShape->CastRay(inRay, JPH::SubShapeIDCreator(), result);

            if (hadHit)
            {
                // Do stuff...
            }
        }
    }

Thanks in advance for your time.

[jrouwe]

For NarrowPhase I think you need to do something like:

AzPhysics::SceneQueryHit Shape::RayCast(const AzPhysics::RayCastRequest& worldSpaceRequest,
        [[maybe_unused]] const AZ::Transform& worldTransform)
    {
        if (m_joltShape)
        {
            if (const bool shouldCollide = worldSpaceRequest.m_collisionGroup.GetMask() & m_collisionLayer.GetMask();
            !shouldCollide)
            {
                return AzPhysics::SceneQueryHit();
            }
            JPH::RayCastResult result;
            
            // Ray for NarrowPhase query
            JPH::RRayCast inRay(
                JoltMathConvert(worldSpaceRequest.m_start),
                JoltMathConvert(worldSpaceRequest.m_direction * worldSpaceRequest.m_distance)
            );
            
            // NarrowPhase option
            bool hadHit = m_attachedSystem->GetNarrowPhaseQuery().CastRay(inRay, result);
   
            if (hadHit)
            {
                // Do stuff...
            }
        }
    }

For a shape this would become something like:

AzPhysics::SceneQueryHit Shape::RayCast(const AzPhysics::RayCastRequest& worldSpaceRequest,
        [[maybe_unused]] const AZ::Transform& worldTransform)
    {
        if (m_joltShape)
        {
            if (const bool shouldCollide = worldSpaceRequest.m_collisionGroup.GetMask() & m_collisionLayer.GetMask();
            !shouldCollide)
            {
                return AzPhysics::SceneQueryHit();
            }
            JPH::RayCastResult result;
            
            // Ray for shape
            JPH::RayCast inRay(
                JoltMathConvert(worldSpaceRequest.m_start),
                JoltMathConvert(worldSpaceRequest.m_direction * worldSpaceRequest.m_distance)
            );
            
            // Need center of mass transform, NOT world transform!
            JPH::Mat44 center_of_mass = JoltMathConvert(worldTransform).PreTranslated(mShape->GetCenterOfMass()));

            // Take the ray to center of mass space for the shape
            inRay = inRay.Transformed(center_of_mass.InversedRotationTranslation());
            
            // Shape option
            bool hadHit = m_joltShape->CastRay(inRay, JPH::SubShapeIDCreator(), result);

            if (hadHit)
            {
                // Do stuff...
            }
        }
    }