Custom Video Source and Renderer

Introduction

The Agora SDK uses default audio and video modules for capturing and rendering in real-time communications.

However, these default modules might not meet your development requirements, such as in the following scenarios:

• Your app has its own video module.
• You want to use a non-camera source, such as recorded screen data.
• You need to process the captured video with a pre-processing library for functions such as image enhancement.
• You need flexible device resource allocation to avoid conflicts with other services.

This article describes how to use the Agora Native SDK to customize the video source and renderer, including the following methods:

• Push: Call setExternalVideoSource to set the video source. You need to use your own capture module to capture the video. Call pushVideoFrame to send the captured video to the SDK. You can also create a custom rendering module to render the captured video frames for local preview.
  • If you are using the Push method and you need to show local preview, you need to render the captured data with a custom renderer. The SDK does not support rendering the captured data locally.
  • Switching in the channel from custom video capture by Push to SDK capture is not supported. To switch the video source directly, you must use the custom video capture by mediaIO. See How can I switch from custom video capture to SDK capture.
• mediaIO: Call setVideoSource to set the video source. Call consumeRawVideoFrame to read video frames from your capture module and send them to the SDK. You can also create a custom rendering module to render the captured video frames for local preview.

Sample project

Refer to the following sample projects on GitHub to learn how to customize the video source and renderer in your project:

• Custom video capture (Push)
• Custom video capture (mediaIO)

Implementation (Push)

Before customizing the video source or renderer, ensure that you have implemented the basic real-time communication functions in your project. For details, see Start a Video Call or Start Interactive Video Streaming.

Refer to the following steps to customize the video source and renderer:

1. Before calling joinChannel, call setExternalVideoSource to set the custom video source.
2. Implement the custom video capture module.
3. (Optional) Implement the custom video rendering module.
4. Start the capture module. Call consumeRawVideoFrame to read the captured video frames and send them to the SDK. The SDK automatically renders the video frames.

API call sequence

Refer to the following diagram to customize the video source in your project.

Data transfer

The following diagram shows the video frame transfer in the Push method:

• Develop the capture module using the platform's native libraries, such as DirectShow, to get video frames from the capturing device.
• (Optional) Develop the rendering module using the platform's native libraries, such as Direct3D, to render video frames. You must implement the data transfer between the capture module and the rendering module yourself.
• Call setExternalVideoSource to set the video source.
• Call pushVideoFrame to send the captured video frames.

Sample code

Refer to the following steps to custom video source and renderer:

1. Call the setExternalVideoSource method before joinChannel to set the video source.

BOOL CAgoraCaptureVideoDlg::EnableExtendVideoCapture(BOOL bEnable)
{
    // Create an AutoPtr instance using the IMediaEngine as template
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    // Refer to AgoraBase.h in the SDK for the implementation of AutoPtr
    // The AutoPtr instance calls queryInterface and gets a pointer to the IMediaEngine instance via IID.
    // The AutoPtr instance accesses the pointer to the IMediaEngine instance and calls registerAudioFrameObserver via IMediaEngine.
        // m_rtcEngine is an instance of the IRtcEngine object
    mediaEngine.queryInterface(m_rtcEngine, agora::AGORA_IID_MEDIA_ENGINE);
    int nRet = 0;
    if (mediaEngine.get() == NULL)
        return FALSE;
    if (bEnable) {
        // Set the custom video source
        nRet = mediaEngine->setExternalVideoSource(true, false);
    }
    else {
        // Disable the custom video source
        nRet = mediaEngine->setExternalVideoSource(false, false);
    }
    return nRet == 0 ? TRUE : FALSE;
}

2. Configure your own capture module. Use ExternalVideoFrame to set video attributes, and call setVideoEncoderConfiguration to set encoding settings. Start the capture module.

// Set the encoding configurations and start capturing
void CAgoraCaptureVideoDlg::EnableCaputre(BOOL bEnable)
{
    if (bEnable == (BOOL)!m_extenalCaptureVideo)return;
    // Get device types from the UI
    int nIndex = m_cmbVideoType.GetCurSel();
    if (bEnable)
    {
        // Select capture device. m_agVideoCaptureDevice is an instance from the self-capture module.
        m_agVideoCaptureDevice.SelectMediaCap(nIndex == -1 ? 0 : nIndex);
        // Create VIDEOINFOHEADER instance, which is from the Windows Media Device Manager SDK in MFC
        VIDEOINFOHEADER videoInfo;
        // Create VideoEncoderConfiguration instance
        VideoEncoderConfiguration config;
        // Create capture filter
        m_agVideoCaptureDevice.CreateCaptureFilter();
        // Get the selected capture device
        m_agVideoCaptureDevice.GetCurrentVideoCap(&videoInfo);
        // The following parameters can be acquired from the bmiHeader parameter of videoInfo. 
        // bmiHeader belongs to Win32 _BITMAPINFOHEADER and includes color and dimension information of the image bitmap.
        // Set the config parameter of setVideoEncoderConfiguration
        config.dimensions.width = videoInfo.bmiHeader.biWidth;
        config.dimensions.height = videoInfo.bmiHeader.biHeight;
        // m_videoFrame is an instance of ExternalVideoFrame in IVideoFrame. The following code sets external video frame attributes.
        // The following parameters are from the bmiHeader parameter in videoInfo. 
        // bmiHeader belongs to Win32 _BITMAPINFOHEADER and includes color and dimension information of the image bitmap.
        m_videoFrame.stride = videoInfo.bmiHeader.biWidth;
        m_videoFrame.height = videoInfo.bmiHeader.biHeight;
        // You need to specify the following parameters
        m_videoFrame.rotation = 0;
        m_videoFrame.cropBottom = 0;
        m_videoFrame.cropLeft = 0;
        m_videoFrame.cropRight = 0;
        m_videoFrame.cropTop = 0;
        m_videoFrame.format = agora::media::ExternalVideoFrame::VIDEO_PIXEL_I420;
        m_videoFrame.type = agora::media::ExternalVideoFrame::VIDEO_BUFFER_TYPE::VIDEO_BUFFER_RAW_DATA;
        // Set fps, which equals to 10000000 divided by the average time of each frame in 100 nanoseconds
        m_fps = (int)(10000000ll / videoInfo.AvgTimePerFrame);
        // Set video encoder configurations
        m_rtcEngine->setVideoEncoderConfiguration(config);
        // Initialize the rendering module
        m_d3dRender.Init(m_localVideoWnd.GetSafeHwnd(),
            videoInfo.bmiHeader.biWidth, videoInfo.bmiHeader.biHeight, true);
        // Start video capture
        m_agVideoCaptureDevice.Start();
    }
    else {
        // Stop video capture
        m_agVideoCaptureDevice.Stop();
        // Remove video capture filter
        m_agVideoCaptureDevice.RemoveCaptureFilter();
        if (m_rtcEngine)
        {
            m_rtcEngine->stopPreview();
            m_d3dRender.Close();
        }
    }
}

3. After the user joins the channel, the PushVideoFrameThread is triggered. The thread reads the video frames returned by the capture module, starts the rendering module, and calls pushVideoFrame to push external frames to the SDK.

// This thread is triggered when the local user joins the channel
void CAgoraCaptureVideoDlg::PushVideoFrameThread(CAgoraCaptureVideoDlg * self)
{
   // Create an AutoPtr instance using the IMediaEngine as template
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    // The AutoPtr instance calls queryInterface and gets a pointer to the IMediaEngine instance via IID. 
    // The AutoPtr instance accesses the pointer to the IMediaEngine instance and calls pushVideoFrame via IMediaEngine.
    mediaEngine.queryInterface(self->m_rtcEngine, agora::AGORA_IID_MEDIA_ENGINE);
    // Start local preview
    self -> m_rtcEngine->startPreview();
    // If the self-capture is enabled in the UI and the local user has joined the channel
    while (self->m_extenalCaptureVideo && self->m_joinChannel)
    { 
        // If the video frame is YUV 4:2;0, read the buffer, set timestamp, 
        // use the rendering module (d3d) to render video frames, and call pushAudioFrame to push video frames to the SDK
        if (self->m_videoFrame.format == agora::media::ExternalVideoFrame::VIDEO_PIXEL_I420) {
            int bufSize = self->m_videoFrame.stride * self->m_videoFrame.height * 3 / 2;
            int timestamp = GetTickCount();
            // If there are available video frames from the capture module, set the timestamp to current time
            if (CAgVideoBuffer::GetInstance()->readBuffer(self->m_buffer, bufSize, timestamp)) {
                self->m_videoFrame.timestamp = timestamp;
            }
            else
            {
                // If there are no available video frames from the capture module, 
                // sleep for 1 millisecond, skip the steps below, and continue with the loop
                Sleep(1);
                continue;
            }
            // Assign the buffer to the buffer parameter of m_videoFrame (ExternalVideoFrame) for pushVideoFrame
            self->m_videoFrame.buffer = self->m_buffer;
            // Use the rendering module to render the buffer and send to local preview window
            self->m_d3dRender.Render((char*)self->m_buffer);
            // Call pushVideoFrame to push the captured video frames to the SDK
            mediaEngine->pushVideoFrame(&self->m_videoFrame);
            // Sleep for the number of milliseconds that each frame appears
            Sleep(1000 / self->m_fps);
        }
        else {
            return;
        }
    }
}

API reference

• setExternalVideoSource
• pushVideoFrame

Implementation (mediaIO)

Before customizing the video source or renderer, ensure that you have implemented the basic real-time communication functions in your project. For details, see Start a Video Call or Start Interactive Video Streaming.

Refer to the following steps to customize the video source and renderer:

1. Implement IVideoSource and IVideoFrameConsumer. Implement the following code logic in the callbacks:

   • After receiving the onInitialize callback, save the IVideoFrameConsumer instance.
   • After receiving the onStart callback, call consumeRawVideoFrame to read the video frames from the capture module and send the video frames to the SDK.
   • After receiving the onStop callback, stop sending video frames from IVideoFrameConsumer to the SDK.

2. Call setVideoSource to set the video source outside the SDK.

3. Implement the custom video capture module.

4. (Optional) Implement the custom video rendering module.

5. Start the capture module. Call consumeRawVideoFrame to read the captured video frames and send them to the SDK.

API call sequence

Refer to the following diagram to customize the video source in your project.

Data transfer

The following diagram shows the video frame transfer in the MediaIO method:

• Develop the capture module using the platform's native libraries, such as DirectShow, to get video frames from the capturing device.
• (Optional) Develop the rendering module using the platform's native libraries, such as Direct3D, to render video frames. You must implement the data transfer between the capture module and the rendering module yourself.
• Call setVideoSource to set the video source.
• Call consumeRawVideoFrame to send captured video frames to the SDK.

Sample code

Refer to the following steps to custom video source:

1. Implement IVideoSource and IVideoFrameConsumer. Create a ThreadRun thread to consume video frames from the video source. The thread is triggered in the onStart callback and calls consumeRawVideoFrame to read video frames from the capture module to the SDK.

class CAgoraVideoSource :public IVideoSource {

    // Override the onInitialize callback
    virtual bool onInitialize(IVideoFrameConsumer *consumer) override
    {
        std::lock_guard<std::mutex> m(m_mutex);
        // Get IVideoFrameConsumer from the callback
        m_videoConsumer = consumer;
        OutputDebugString(_T("onInitialize\n"));
        return true;
    }

    // Override the onDispose callback
    virtual void onDispose() override
    {
        OutputDebugString(_T("onDispose\n"));
        Stop();
    }

    // Override the onStart callback
    virtual bool onStart() override
    {
        OutputDebugString(_T("onStart\n"));
        // Start the ThreadRun thread
        CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)ThreadRun, this, 0, NULL);
        return true;
    }


    // The thread reads video frames from the capture module and send the video frames to the SDK
    static void ThreadRun(CAgoraVideoSource* self)
    {
        // Continue the loop as long as onStop is triggered
        while (!self->m_isExit)
        {
            // Set bufSize parameters, get the current timestamp, and read the buffer
            int bufSize = self->m_width * self->m_height * 3 / 2;
            int timestamp = GetTickCount();
            // If no buffer is available, the thread sleeps for 1 millisecond, skips the following steps, and continue the loop
           if (!CAgVideoBuffer::GetInstance()->readBuffer(self->m_buffer, bufSize, timestamp)) {
                Sleep(1);
                continue;
            }
            // If onInitialize is triggered, run the following code
            if (self->m_videoConsumer)
            {
                self->m_mutex.lock();
                // Consume raw data frames. The video type in getBufferType and consumeRawVideoFrame must be consistent.
                self->m_videoConsumer->consumeRawVideoFrame(self->m_buffer, ExternalVideoFrame::VIDEO_PIXEL_I420,
                    self->m_width, self->m_height, self->m_rotation, timestamp);
                self->m_mutex.unlock();
                // Sleep for the number of milliseconds each video frame appears
                Sleep(1000 / self->m_fps);
            }
        }
    }

    // Override the onStop method
    virtual void onStop() override
    {
        OutputDebugString(_T("onStop\n"));
        Stop();
    }

    // Override the getBufferType to return VIDEO_PIXEL_I420. 
    // The video type in getBufferType and consumeRawVideoFrame must be consistent.
    virtual agora::media::ExternalVideoFrame::VIDEO_PIXEL_FORMAT getBufferType() override
    {
        return ExternalVideoFrame::VIDEO_PIXEL_I420;
    }

    // Override to return VIDEO_CAPTURE_CAMERA
    virtual VIDEO_CAPTURE_TYPE getVideoCaptureType() override
    {
        return VIDEO_CAPTURE_CAMERA;
    }


    // Override to return CONTENT_HINT_DETAILS
    virtual VideoContentHint getVideoContentHint() override
    {
        return CONTENT_HINT_DETAILS;
    }


public:
    // Constructor
    CAgoraVideoSource()
    {
        m_buffer = new BYTE[1920 * 1080 * 4 * 4];
    }

    // Destructor
    ~CAgoraVideoSource()
    {
        delete m_buffer;
    }

    // Stop the thread
    void Stop()
    {
        std::lock_guard<std::mutex> m(m_mutex);
        m_isExit = true;
        m_videoConsumer = nullptr;
    }

    // Set parameters
    void SetParameters(bool isExit, int width, int height, int rotation,int fps)
    {
        std::lock_guard<std::mutex> m(m_mutex);
        m_isExit = isExit;
        m_width = width;
        m_height = height;
        m_rotation = rotation;
        m_fps = fps;
    }

private:
    // IVideoFrameConsumer object
    IVideoFrameConsumer * m_videoConsumer;
    bool m_isExit;
    BYTE * m_buffer;
    int m_width;
    int m_height;
    int m_rotation;
    int m_fps;
    std::mutex m_mutex;
};


// Video source object
CAgoraVideoSource m_videoSouce;

2. Create and initialize the video capture object.

// Create and initialize the video capture object
m_agVideoCaptureDevice.Create();

3. Call setVideoSource to set the video source. After calling setVideoSource, the onStart callback triggers the ThreadRun thread, which calls consumeRawVideoFrame to read video frames from the capture module and send them to the SDK. The thread waits if no video frames are available.

// Set video source
BOOL CAgoraMediaIOVideoCaptureDlg::EnableExtendVideoCapture(BOOL bEnable)
{
    bool bRet = true;
    if (mediaEngine.get() == NULL)
        return FALSE;
    if (bEnable) {
        // Set video source
        bRet = m_rtcEngine->setVideoSource(&m_videoSouce);
    }
    return bRet ? TRUE : FALSE;
}

4. Configure and run the capture module.

// Start video capturing
void CAgoraMediaIOVideoCaptureDlg::EnableCapture(BOOL bEnable)
{
    if (bEnable == (BOOL)!m_extenalCaptureVideo)return;

    int nIndex = m_cmbVideoType.GetCurSel();
    if (bEnable)
    {
        // Select video capture device per the UI selection. m_agVideoCaptureDevice is an instance from the capture module.
        m_agVideoCaptureDevice.SelectMediaCap(nIndex == -1 ? 0 : nIndex);
        // Create VIDEOINFOHEADER instance, which is from the Windows Media Device Manager SDK in MFC
        VIDEOINFOHEADER videoInfo;
        // Create a VideoEncoderConfiguration instance
        VideoEncoderConfiguration config;
        // Create capture filter
        m_agVideoCaptureDevice.CreateCaptureFilter();
        // Get the selected video capture device
        m_agVideoCaptureDevice.GetCurrentVideoCap(&videoInfo);
        // Set the config parameter of setVideoEncoderConfiguration
        // The following parameters are from the bmiHeader parameter in videoInfo. 
        // bmiHeader belongs to Win32 _BITMAPINFOHEADER and includes color and dimension information of the image bitmap.
        config.dimensions.width = videoInfo.bmiHeader.biWidth;
        config.dimensions.height = videoInfo.bmiHeader.biHeight;
        // Set parameters for the captured video. See CAgoraVideoSource for the implementation of SetParameters.
        m_videoSouce.SetParameters(false, videoInfo.bmiHeader.biWidth,
            videoInfo.bmiHeader.biHeight, 0, (int)(10000000ll / videoInfo.AvgTimePerFrame));
        // Set video encoding configurations
        m_rtcEngine->setVideoEncoderConfiguration(config);
        // Start video capture
        m_agVideoCaptureDevice.Start();
        // Start local preview
        m_rtcEngine->startPreview();
    }
    else {
        m_videoSouce.Stop();
        // Stop video capture
        m_agVideoCaptureDevice.Stop();
        // Remove video capture module
        m_agVideoCaptureDevice.RemoveCaptureFilter();
        if (m_rtcEngine)
        {
            m_rtcEngine->stopPreview();
        }
    }
}

API reference

• IVideoSource
• IVideoFrameConsumer
• setVideoSource
• consumeRawVideoFrame

Considerations

• Ensure the accuracy and efficiency of the audio and video data processing in the callback methods to avoid any possible crashes.
• Set the audio data to RAW_AUDIO_FRAME_OP_MODE_READ_WRITE if you want to read, write, and manipulate the data.
• Use raw data methods to customize the video renderer. If you do not want the SDK to render the video frame, do not call the setupLocalVideo method. Ensure compatibility on the Windows platform when customizing the video renderer.
• Customizing the video source and renderer requires you to manage video data recording and playback on your own.
  • When customizing the video source, you need to capture and process the video data on your own.
  • When customizing the video renderer, you need to process and render the video data on your own.