Feature Request: Add updateMediaStream functionality to dynamically update media source during an active stream

[leporel]

Description:

Could you please add functionality to dynamically update the media source during an active stream? This could be implemented in setMediaStream or like as a new method updateMediaStream.

I’ve created a rough implementation (see example below), but I’m not a JavaScript developer, and I’m concerned about potential issues:

1. Updating the video source with different constraints (e.g., resolution changes). Do we need to create a new offer or perform additional steps to update the parameters?
2. Removing and re-adding tracks (e.g., audioSender.replaceTrack(null)). After removing a track, adding a new one fails because the sender is no longer found.

Here’s my example code for video track replacement:

async function replaceVideoTrackOnFly(newVideoTrack) {
    if (!ovenkit || !isStreaming) return;
    
    try {
        // Find the current video sender in the peer connection
        const videoSender = ovenkit.peerConnection.getSenders().find(sender => 
            sender.track && sender.track.kind === 'video'
        );
        
        if (videoSender) {
            // Replace the track
            await videoSender.replaceTrack(newVideoTrack);
            
            // Also update the input stream that OvenKit keeps track of
            const oldTrack = ovenkit.inputStream.getVideoTracks()[0];
            if (oldTrack) {
                ovenkit.inputStream.removeTrack(oldTrack);
            }
            ovenkit.inputStream.addTrack(newVideoTrack);
            
            console.log('Video track replaced successfully');
            return true;
        } else {
            console.error('No video sender found in peer connection');
            return false;
        }
    } catch (err) {
        console.error('Error replacing video track:', err);
        return false;
    }
}

This seems to work, but I’m unsure about the technical nuances. Could you provide guidance or implement a robust solution for this use case? Thank you!

[SangwonOh]

@leporel Interesting! Some questions.

• What is the purpose to replace media source?
• Replacing media source means we need to handle both of video and audio. Am I correct?
• Does It needs seemless replacing?

[leporel]

• What is the purpose to replace media source?

When screen share or window share used for media source, the stream is running and user wants change screen to another screen either window (like it implemented in discord)

• Replacing media source means we need to handle both of video and audio. Am I correct?

Yes. The code is the same for the audio track, it works. The only issue is if the stream was started without audio, and as a result, the peerConnection is missing an audio track.
However, for audio, I don't think this feature would be as useful as it is for switching captured windows on the fly.

• Does It needs seemless replacing?

Sorry, I don't understand this question.

PS: found similar needs OvenMediaLabs/OvenMediaEngine#1712 which explains this purpose

[SangwonOh]

@leporel

Updating the video source with different constraints (e.g., resolution changes). Do we need to create a new offer or perform additional steps to update the parameters?

Here is documentaion. Here's the documentation. The renegotiation conditions seem a bit unpredictable. If the replaceTrack call fails, then renegotiation is needed. In that case, it seems like the connection needs to be reestablished from scratch.

• https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/replaceTrack#things_that_require_negotiation

Removing and re-adding tracks (e.g., audioSender.replaceTrack(null)). After removing a track, adding a new one fails because the sender is no longer found.

We can use the RTCRtpTransceiver of peerconnection to find missing audio sender.

• https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpTransceiver

rough impementation

function switchStream() {

  ovenLivekit.getDisplayMedia().then(async function (stream) {

    const audioTransceiver = ovenLivekit.peerConnection.getTransceivers().find(function (transceiver) {
      return transceiver.receiver.track.kind === 'audio';
    });

    console.log('audioTransceiver', audioTransceiver);

    const videoTransceiver = ovenLivekit.peerConnection.getTransceivers().find(function (transceiver) {
      return transceiver.receiver.track.kind === 'video';
    });

    console.log('videoTransceiver', videoTransceiver);

    const newAudioTrack = stream.getAudioTracks()[0];
    await audioTransceiver.sender.replaceTrack(newAudioTrack);

    const newVideoTrack = stream.getVideoTracks()[0];
    await videoTransceiver.sender.replaceTrack(newVideoTrack);
  });
}