在ffplay中获得绿屏:使用Live555通过RTP流将桌面(DirectX表面)作为H264视频流


9

我正在尝试使用Windows10上的Live555和Windows Media Foundation的硬件编码器通过RTP流将桌面(NV12格式的DirectX表面)流化为H264视频,并期望它可以由ffplay(ffmpeg 4.2)呈现。但是只有如下所示的绿屏,

在此处输入图片说明

在此处输入图片说明

在此处输入图片说明

在此处输入图片说明

我提到了MFWebCamToRTP mediafoundation-sample使用硬件MFT编码DirectX表面,以实现live555的FramedSource并将输入源更改为DirectX表面而不是webCam。

这是我的Live555的doGetNextFrame回调实现的摘录,以从directX表面提供输入样本:

virtual void doGetNextFrame()
{
    if (!_isInitialised)
    {
        if (!initialise()) {
            printf("Video device initialisation failed, stopping.");
            return;
        }
        else {
            _isInitialised = true;
        }
    }

    //if (!isCurrentlyAwaitingData()) return;

    DWORD processOutputStatus = 0;
    HRESULT mftProcessOutput = S_OK;
    MFT_OUTPUT_STREAM_INFO StreamInfo;
    IMFMediaBuffer *pBuffer = NULL;
    IMFSample *mftOutSample = NULL;
    DWORD mftOutFlags;
    bool frameSent = false;
    bool bTimeout = false;

    // Create sample
    CComPtr<IMFSample> videoSample = NULL;

    // Create buffer
    CComPtr<IMFMediaBuffer> inputBuffer;
    // Get next event
    CComPtr<IMFMediaEvent> event;
    HRESULT hr = eventGen->GetEvent(0, &event);
    CHECK_HR(hr, "Failed to get next event");

    MediaEventType eventType;
    hr = event->GetType(&eventType);
    CHECK_HR(hr, "Failed to get event type");


    switch (eventType)
    {
    case METransformNeedInput:
        {
            hr = MFCreateDXGISurfaceBuffer(__uuidof(ID3D11Texture2D), surface, 0, FALSE, &inputBuffer);
            CHECK_HR(hr, "Failed to create IMFMediaBuffer");

            hr = MFCreateSample(&videoSample);
            CHECK_HR(hr, "Failed to create IMFSample");
            hr = videoSample->AddBuffer(inputBuffer);
            CHECK_HR(hr, "Failed to add buffer to IMFSample");

            if (videoSample)
            {
                _frameCount++;

                CHECK_HR(videoSample->SetSampleTime(mTimeStamp), "Error setting the video sample time.\n");
                CHECK_HR(videoSample->SetSampleDuration(VIDEO_FRAME_DURATION), "Error getting video sample duration.\n");

                // Pass the video sample to the H.264 transform.

                hr = _pTransform->ProcessInput(inputStreamID, videoSample, 0);
                CHECK_HR(hr, "The resampler H264 ProcessInput call failed.\n");

                mTimeStamp += VIDEO_FRAME_DURATION;
            }
        }

        break;

    case METransformHaveOutput:

        {
            CHECK_HR(_pTransform->GetOutputStatus(&mftOutFlags), "H264 MFT GetOutputStatus failed.\n");

            if (mftOutFlags == MFT_OUTPUT_STATUS_SAMPLE_READY)
            {
                MFT_OUTPUT_DATA_BUFFER _outputDataBuffer;
                memset(&_outputDataBuffer, 0, sizeof _outputDataBuffer);
                _outputDataBuffer.dwStreamID = outputStreamID;
                _outputDataBuffer.dwStatus = 0;
                _outputDataBuffer.pEvents = NULL;
                _outputDataBuffer.pSample = nullptr;

                mftProcessOutput = _pTransform->ProcessOutput(0, 1, &_outputDataBuffer, &processOutputStatus);

                if (mftProcessOutput != MF_E_TRANSFORM_NEED_MORE_INPUT)
                {
                    if (_outputDataBuffer.pSample) {

                        //CHECK_HR(_outputDataBuffer.pSample->SetSampleTime(mTimeStamp), "Error setting MFT sample time.\n");
                        //CHECK_HR(_outputDataBuffer.pSample->SetSampleDuration(VIDEO_FRAME_DURATION), "Error setting MFT sample duration.\n");

                        IMFMediaBuffer *buf = NULL;
                        DWORD bufLength;
                        CHECK_HR(_outputDataBuffer.pSample->ConvertToContiguousBuffer(&buf), "ConvertToContiguousBuffer failed.\n");
                        CHECK_HR(buf->GetCurrentLength(&bufLength), "Get buffer length failed.\n");
                        BYTE * rawBuffer = NULL;

                        fFrameSize = bufLength;
                        fDurationInMicroseconds = 0;
                        gettimeofday(&fPresentationTime, NULL);

                        buf->Lock(&rawBuffer, NULL, NULL);
                        memmove(fTo, rawBuffer, fFrameSize);

                        FramedSource::afterGetting(this);

                        buf->Unlock();
                        SafeRelease(&buf);

                        frameSent = true;
                        _lastSendAt = GetTickCount();

                        _outputDataBuffer.pSample->Release();
                    }

                    if (_outputDataBuffer.pEvents)
                        _outputDataBuffer.pEvents->Release();
                }

                //SafeRelease(&pBuffer);
                //SafeRelease(&mftOutSample);

                break;
            }
        }

        break;
    }

    if (!frameSent)
    {
        envir().taskScheduler().triggerEvent(eventTriggerId, this);
    }

    return;

done:

    printf("MediaFoundationH264LiveSource doGetNextFrame failed.\n");
    envir().taskScheduler().triggerEvent(eventTriggerId, this);
}

初始化方法:

bool initialise()
{
    HRESULT hr;
    D3D11_TEXTURE2D_DESC desc = { 0 };

    HDESK CurrentDesktop = nullptr;
    CurrentDesktop = OpenInputDesktop(0, FALSE, GENERIC_ALL);
    if (!CurrentDesktop)
    {
        // We do not have access to the desktop so request a retry
        return false;
    }

    // Attach desktop to this thread
    bool DesktopAttached = SetThreadDesktop(CurrentDesktop) != 0;
    CloseDesktop(CurrentDesktop);
    CurrentDesktop = nullptr;
    if (!DesktopAttached)
    {
        printf("SetThreadDesktop failed\n");
    }

    UINT32 activateCount = 0;

    // h264 output
    MFT_REGISTER_TYPE_INFO info = { MFMediaType_Video, MFVideoFormat_H264 };

    UINT32 flags =
        MFT_ENUM_FLAG_HARDWARE |
        MFT_ENUM_FLAG_SORTANDFILTER;

    // ------------------------------------------------------------------------
    // Initialize D3D11
    // ------------------------------------------------------------------------

    // Driver types supported
    D3D_DRIVER_TYPE DriverTypes[] =
    {
        D3D_DRIVER_TYPE_HARDWARE,
        D3D_DRIVER_TYPE_WARP,
        D3D_DRIVER_TYPE_REFERENCE,
    };
    UINT NumDriverTypes = ARRAYSIZE(DriverTypes);

    // Feature levels supported
    D3D_FEATURE_LEVEL FeatureLevels[] =
    {
        D3D_FEATURE_LEVEL_11_0,
        D3D_FEATURE_LEVEL_10_1,
        D3D_FEATURE_LEVEL_10_0,
        D3D_FEATURE_LEVEL_9_1
    };
    UINT NumFeatureLevels = ARRAYSIZE(FeatureLevels);

    D3D_FEATURE_LEVEL FeatureLevel;

    // Create device
    for (UINT DriverTypeIndex = 0; DriverTypeIndex < NumDriverTypes; ++DriverTypeIndex)
    {
        hr = D3D11CreateDevice(nullptr, DriverTypes[DriverTypeIndex], nullptr,
            D3D11_CREATE_DEVICE_VIDEO_SUPPORT,
            FeatureLevels, NumFeatureLevels, D3D11_SDK_VERSION, &device, &FeatureLevel, &context);
        if (SUCCEEDED(hr))
        {
            // Device creation success, no need to loop anymore
            break;
        }
    }

    CHECK_HR(hr, "Failed to create device");

    // Create device manager
    UINT resetToken;
    hr = MFCreateDXGIDeviceManager(&resetToken, &deviceManager);
    CHECK_HR(hr, "Failed to create DXGIDeviceManager");

    hr = deviceManager->ResetDevice(device, resetToken);
    CHECK_HR(hr, "Failed to assign D3D device to device manager");


    // ------------------------------------------------------------------------
    // Create surface
    // ------------------------------------------------------------------------
    desc.Format = DXGI_FORMAT_NV12;
    desc.Width = surfaceWidth;
    desc.Height = surfaceHeight;
    desc.MipLevels = 1;
    desc.ArraySize = 1;
    desc.SampleDesc.Count = 1;

    hr = device->CreateTexture2D(&desc, NULL, &surface);
    CHECK_HR(hr, "Could not create surface");

    hr = MFTEnumEx(
        MFT_CATEGORY_VIDEO_ENCODER,
        flags,
        NULL,
        &info,
        &activateRaw,
        &activateCount
    );
    CHECK_HR(hr, "Failed to enumerate MFTs");

    CHECK(activateCount, "No MFTs found");

    // Choose the first available encoder
    activate = activateRaw[0];

    for (UINT32 i = 0; i < activateCount; i++)
        activateRaw[i]->Release();

    // Activate
    hr = activate->ActivateObject(IID_PPV_ARGS(&_pTransform));
    CHECK_HR(hr, "Failed to activate MFT");

    // Get attributes
    hr = _pTransform->GetAttributes(&attributes);
    CHECK_HR(hr, "Failed to get MFT attributes");

    // Unlock the transform for async use and get event generator
    hr = attributes->SetUINT32(MF_TRANSFORM_ASYNC_UNLOCK, TRUE);
    CHECK_HR(hr, "Failed to unlock MFT");

    eventGen = _pTransform;
    CHECK(eventGen, "Failed to QI for event generator");

    // Get stream IDs (expect 1 input and 1 output stream)
    hr = _pTransform->GetStreamIDs(1, &inputStreamID, 1, &outputStreamID);
    if (hr == E_NOTIMPL)
    {
        inputStreamID = 0;
        outputStreamID = 0;
        hr = S_OK;
    }
    CHECK_HR(hr, "Failed to get stream IDs");

     // ------------------------------------------------------------------------
    // Configure hardware encoder MFT
   // ------------------------------------------------------------------------
    CHECK_HR(_pTransform->ProcessMessage(MFT_MESSAGE_SET_D3D_MANAGER, reinterpret_cast<ULONG_PTR>(deviceManager.p)), "Failed to set device manager.\n");

    // Set low latency hint
    hr = attributes->SetUINT32(MF_LOW_LATENCY, TRUE);
    CHECK_HR(hr, "Failed to set MF_LOW_LATENCY");

    hr = MFCreateMediaType(&outputType);
    CHECK_HR(hr, "Failed to create media type");

    hr = outputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
    CHECK_HR(hr, "Failed to set MF_MT_MAJOR_TYPE on H264 output media type");

    hr = outputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264);
    CHECK_HR(hr, "Failed to set MF_MT_SUBTYPE on H264 output media type");

    hr = outputType->SetUINT32(MF_MT_AVG_BITRATE, TARGET_AVERAGE_BIT_RATE);
    CHECK_HR(hr, "Failed to set average bit rate on H264 output media type");

    hr = MFSetAttributeSize(outputType, MF_MT_FRAME_SIZE, desc.Width, desc.Height);
    CHECK_HR(hr, "Failed to set frame size on H264 MFT out type");

    hr = MFSetAttributeRatio(outputType, MF_MT_FRAME_RATE, TARGET_FRAME_RATE, 1);
    CHECK_HR(hr, "Failed to set frame rate on H264 MFT out type");

    hr = outputType->SetUINT32(MF_MT_INTERLACE_MODE, 2);
    CHECK_HR(hr, "Failed to set MF_MT_INTERLACE_MODE on H.264 encoder MFT");

    hr = outputType->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, TRUE);
    CHECK_HR(hr, "Failed to set MF_MT_ALL_SAMPLES_INDEPENDENT on H.264 encoder MFT");

    hr = _pTransform->SetOutputType(outputStreamID, outputType, 0);
    CHECK_HR(hr, "Failed to set output media type on H.264 encoder MFT");

    hr = MFCreateMediaType(&inputType);
    CHECK_HR(hr, "Failed to create media type");

    for (DWORD i = 0;; i++)
    {
        inputType = nullptr;
        hr = _pTransform->GetInputAvailableType(inputStreamID, i, &inputType);
        CHECK_HR(hr, "Failed to get input type");

        hr = inputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
        CHECK_HR(hr, "Failed to set MF_MT_MAJOR_TYPE on H264 MFT input type");

        hr = inputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12);
        CHECK_HR(hr, "Failed to set MF_MT_SUBTYPE on H264 MFT input type");

        hr = MFSetAttributeSize(inputType, MF_MT_FRAME_SIZE, desc.Width, desc.Height);
        CHECK_HR(hr, "Failed to set MF_MT_FRAME_SIZE on H264 MFT input type");

        hr = MFSetAttributeRatio(inputType, MF_MT_FRAME_RATE, TARGET_FRAME_RATE, 1);
        CHECK_HR(hr, "Failed to set MF_MT_FRAME_RATE on H264 MFT input type");

        hr = _pTransform->SetInputType(inputStreamID, inputType, 0);
        CHECK_HR(hr, "Failed to set input type");

        break;
    }

    CheckHardwareSupport();

    CHECK_HR(_pTransform->ProcessMessage(MFT_MESSAGE_COMMAND_FLUSH, NULL), "Failed to process FLUSH command on H.264 MFT.\n");
    CHECK_HR(_pTransform->ProcessMessage(MFT_MESSAGE_NOTIFY_BEGIN_STREAMING, NULL), "Failed to process BEGIN_STREAMING command on H.264 MFT.\n");
    CHECK_HR(_pTransform->ProcessMessage(MFT_MESSAGE_NOTIFY_START_OF_STREAM, NULL), "Failed to process START_OF_STREAM command on H.264 MFT.\n");

    return true;

done:

    printf("MediaFoundationH264LiveSource initialisation failed.\n");
    return false;
}


    HRESULT CheckHardwareSupport()
    {
        IMFAttributes *attributes;
        HRESULT hr = _pTransform->GetAttributes(&attributes);
        UINT32 dxva = 0;

        if (SUCCEEDED(hr))
        {
            hr = attributes->GetUINT32(MF_SA_D3D11_AWARE, &dxva);
        }

        if (SUCCEEDED(hr))
        {
            hr = attributes->SetUINT32(CODECAPI_AVDecVideoAcceleration_H264, TRUE);
        }

#if defined(CODECAPI_AVLowLatencyMode) // Win8 only

        hr = _pTransform->QueryInterface(IID_PPV_ARGS(&mpCodecAPI));

        if (SUCCEEDED(hr))
        {
            VARIANT var = { 0 };

            // FIXME: encoder only
            var.vt = VT_UI4;
            var.ulVal = 0;

            hr = mpCodecAPI->SetValue(&CODECAPI_AVEncMPVDefaultBPictureCount, &var);

            var.vt = VT_BOOL;
            var.boolVal = VARIANT_TRUE;
            hr = mpCodecAPI->SetValue(&CODECAPI_AVEncCommonLowLatency, &var);
            hr = mpCodecAPI->SetValue(&CODECAPI_AVEncCommonRealTime, &var);

            hr = attributes->SetUINT32(CODECAPI_AVLowLatencyMode, TRUE);

            if (SUCCEEDED(hr))
            {
                var.vt = VT_UI4;
                var.ulVal = eAVEncCommonRateControlMode_Quality;
                hr = mpCodecAPI->SetValue(&CODECAPI_AVEncCommonRateControlMode, &var);

                // This property controls the quality level when the encoder is not using a constrained bit rate. The AVEncCommonRateControlMode property determines whether the bit rate is constrained.
                VARIANT quality;
                InitVariantFromUInt32(50, &quality);
                hr = mpCodecAPI->SetValue(&CODECAPI_AVEncCommonQuality, &quality);
            }
        }
#endif

        return hr;
    }

ffplay命令:

ffplay -protocol_whitelist file,udp,rtp -i test.sdp -x 800 -y 600 -profile:v baseline

SDP:

v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
t=0 0
c=IN IP4 127.0.0.1
m=video 1234 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1

我不知道我要缺少什么,我已经将它修复了近一个星期而没有任何进展,并且尝试了几乎所有可能的方法。而且,用于将DirectX表面编码为视频的在线资源非常有限。

任何帮助,将不胜感激。


1
我认为您错误地期望在METransformNeedInput之后再次调用doGetNextFrame。也许您应该在其中循环,直到获得有效的ProcessOutput调用为止。
VuVirt

hr = event-> GetType(&eventType); switch(eventType){....} if(!frameSent){envir()。taskScheduler()。triggerEvent(eventTriggerId,this); 上面的两个代码块很好地处理了ProcessInput,直到我们从编码器获得输出为止。我已验证相同。@VuVirt
拉姆

那么,当frameSent为true时会发生什么呢?在这种情况下,您是否触发新事件?之后,您将有一个“ return”语句。
VuVirt

@VuVirt由底层的live555库自动循环调用。根据switch语句中的事件,可替代地调用“ ProcessInput”和“ ProcessOutput”。我从ProcessOut得到了连续的数据流,但不仅能够查看它。我确定我正确设置了采样时间和持续时间。
拉姆

1
您可能需要检查是否从ProcessOutput收到MF_E_TRANSFORM_STREAM_CHANGE并相应地处理格式更改。
VuVirt

Answers:


6

它比看起来难。

如果要在使用过程中使用编码器,请直接调用IMFTransform接口,将RGB帧转换为NV12。如果要获得良好的性能,则应在GPU上执行。可能与像素着色器有关,渲染2帧,将全尺寸的一帧渲染到具有亮度的DXGI_FORMAT_R8_UNORM渲染目标中,将一半尺寸的颜色渲染到DXGI_FORMAT_R8G8_UNORM目标中,并写入两个像素着色器以产生NV12值。两个渲染目标都可以渲染到具有相同NV12纹理的2个平面中,但仅从Windows 8开始。

其他方法是使用接收器编写器。它可以同时托管多个MFT,因此您可以在VRAM中提供RGB纹理,接收器写入器将首先通过一个MFT将它们转换为NV12(这可能是GPU驱动程序实现的专有硬件,就像编码器一样),然后传递给编码器MFT。编码成mp4文件相对容易,使用MFCreateSinkWriterFromURL API创建编写器。从接收器写入器中获取原始样本要困难得多,但是,您必须实现自定义媒体接收器,视频流的自定义流接收器,并调用MFCreateSinkWriterFromMediaSink创建写入器。

还有更多。

无论采用哪种编码方法,都无法重用帧纹理。从DD获得的每一帧,都应创建一个新纹理并将其传递给MF。

视频编码器期望帧速率恒定。DD并不能为您提供此功能,而是会在屏幕上每次发生更改时为您提供一个框架。如果您有游戏显示器,则可以为144 FPS;如果唯一的变化是闪烁的光标,则可以为2 FPS。理想情况下,您应该以视频媒体类型中指定的恒定帧速率向MF提交帧。

如果您想流式传输到网络,则经常需要提供参数集。除非你使用英特尔硬件H265编码器,是来自英特尔没有意见破碎,MF为您提供了数据MF_MT_MPEG_SEQUENCE_HEADER属性的媒体类型,通过IMFMediaTypeHandler接口调用SetCurrentMediaType。您可以实现该接口以获取通知。您只有在开始编码后才能获得该数据。那就是如果您使用接收器编写器,那么对于IMFTransform方法而言,它更容易实现,您应该MF_E_TRANSFORM_STREAM_CHANGEProcessOutput方法中获取代码,然后调用GetOutputAvailableType以获取具有该魔术Blob的更新的媒体类型。


您的意思是即使设备使用D3D11_CREATE_DEVICE_VIDEO_SUPPORT和表面描述符初始化为DXGI_FORMAT_NV12并在转换中设置MFT_MESSAGE_SET_D3D_MANAGER时,DirectX(台式机复制)也不会传送NV12格式的帧吗?我也认为我们必须将RGB缓冲区显式转换为NV12或任何受支持的输入格式(大多数是YUV的变体)或使用SinkWriter。但是,这个人可以通过我的方法本身以某种方式实现这一目标。stackoverflow.com/questions/43432670/…–
拉姆


1
@Ram Desktop复制始终以DXGI_FORMAT_B8G8R8A8_UNORM格式提供RGB帧。H264和h265编码器MFT仅支持NV12并耦合其他怪异的编码器。有人必须convert依。您使用桌面复制;您已经不支持Windows 7。使用接收器编写器。我非常确定,这些将RGB转换为NV12的nVidia / Intel的硬件MFT比像素着色器ALU的能效更高,它们可能纯粹在硬件中实现。
很快

你是对的。颜色转换必须明确进行。github.com/GPUOpen-LibrariesAndSDKs/AMF/issues/92。我正在朝这个方向前进。
拉姆(Ram)

1
@Ram应该能用,我之前做过。当DD因为没有更新而拒绝提供新帧时,可以通过再次向编码器提交相同的纹理来节省大量VRAM。仅当DD为Ya具有新帧时才创建新纹理。但是,用于检测何时应该提交帧以及等待多长时间的代码并不简单。我已经使用QueryPerformanceCounter来测量时间,并使用最近几帧中的某种滚动平均值来确定我应该捕获还是应该睡觉。顺便说一句,正确的睡眠方式是IDXGIOutput :: WaitForVBlank方法。
即将

1

由于ffplay抱怨流参数,因此我认为它无法接收SPS / PPS。您尚未在硬编码SDP中设置它们-请参阅RFC-3984并查找sprop-parameter-sets。RFC中的一个示例:

m =视频49170 RTP / AVP 98
a = rtpmap:98 H264 / 90000
a = fmtp:98 profile-level-id = 42A01E; prop-parameter-sets = Z0IACpZTBYmI,aMljiA ==

我强烈认为ffplaySDP中会期望这些。我不记得如何从Media Foundation编码器中获取SPS / PPS,但是样本有效载荷中是否包含,您需要通过查找适当的NAL单元来提取它们,或者Google如何从中提取出额外的数据。编码器- 我获得的第一看起来很有希望。


这是有道理的。我也对SPS / PPS有怀疑。我还没有验证。感谢您将我定向到MSDN线程,这给了我一些希望。
拉姆(Ram)

@Ram有一个很好的提示,示例有效负载中包含SPS / PPS,所以我先检查一下。
鲁道夫斯·邦杜利斯

是的,我明白。当我尝试通过Mpeg4MediaSink将样本写入文件时,我掌握了一些直接从媒体基础编码器中检索和解析SPS / PPS的知识。我将朝这个方向前进。
拉姆(Ram)

1

很快就会为您提供解决问题的所有必需的东西。

您需要做的第一件事是在DXGI_FORMAT_B8G8R8A8_UNORM和MFVideoFormat_NV12之间进行格式转换:

格式转换

格式转换信息

我认为最好使用着色器进行格式转换,因为所有纹理都将保留在GPU中(以提高性能)。

这是您需要做的第一步。您将有其他人来改善您的程序。


1
2x4图像在NV12中需要12个字节,而不是24:8个亮度值,但是彩色图像是1x2像素的两倍小,因此该2x4图像的颜色信息总共仅4个字节,U和2个字节V的2个字节
。–即将

是的,您是对的,我省略了降采样到NV12格式的4.2.0。我将尝试制作一个更合适的图表。
mofo77,19年
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.