kmansoft/srtc

srtc - a "simple" WebRTC library

This is srtc, a "simple" WebRTC library (publish side is done and working quite well, subscribe is in progress, works but needs more time to mature).

Features:

• Depends on OpenSSL (or BoringSSL) or Mbed TLS only, nothing else (libsrtp is only used for tests)
• Portable code in "conservative" C++: language level is C++ 17, and no exceptions or RTTI
• Only one worker thread per PeerConnection
• Video codecs: VP8, VP9, H264 (any profile id), H265, AV1
• Audo codec: Opus
• SDP offer generation and SDP response parsing
• ICE / STUN negotiation, DTLS negotiation, SRTP and SRTCP
• IPv4 and IPv6
• Data channels (both sending and receiving)
• Multiple data streams (SDP "media lines")
• Sender Reports, Receiver Reports
• RRTR and DLRR for RTT estimates (both publish and subscribe)
• TWCC extension for bandwidth estimate (both publish and subscribe)
• Abs-capture-time extension (both send and receive)
• Tested with Pion and Amazon IVS (Interactive Video Service)
• Works on Linux, Android, MacOS, Windows, and should work on iOS too.

State of publish

• Retransmits of packets reported lost by the receiver, uses RTX if supported.
• Video simulcast (sending multiple layers at different resolutions) including the Google VLA extension and RFC 8851.
• Basic bandwidth estimation using the TWCC extension and probing.
• Pacing.

State of subscribe

• Sends nacks, supports RTX if negotiated.
• Sends PLI (key frame requests).
• Sends receiver reports.
• Sends TWCC reports if negotiated in the SDP.
• The jitter buffer is fixed size (for now), based on RTT estimates from the ICE exchange while connecting.
• RTP timestamps and NTP timestamps from sender reports are reported to the application, so media synchronization should be handled there.

API design

Efficient. Can be used for publishing media from a server side system i.e. where you want to run multiple (hundreds or perhaps thousands) of simultaneous WebRTC sessions on a single computer so Google's implementation is not a good choice due to its hunger for threads.

Has a command line tool / sample code to publish video, which has already seen some use.

Has a command line tool / sample code to subscribe to audio and/or video, which can save media to files.

Media encoding / decoding and presentation are deliberately out of scope of this library. For publishing, the application needs to provide encoded media samples. For subscribing, the application receives encoded media samples which it needs to decode and present.

The srtc library does handle packetization of media frames into RTP packets when publishing and the reconstruction of media frames from RTP packets when subscribing (a jitter buffer).

The API is deliberately not compatible with Google's, but the concepts are similar. The Google WebRTC library is inteded for browsers, and therefore its API has to match the API defined for JavaScript and cannot be changed. I decided that it's not necessary to follow the JavaScript API.

Basic use from C++

Create a PeerConnection, ask it to create an SDP offer, send it to a WHIP / WHEP server using your favorite HTTP library, then set the SDP answer on the PeerConnection. This will initiate a network connection, whose state will be emitted via the connection state callback.

Once the peer is connected, you can start publishing audio and video samples using these methods:

• setVideoCodecSpecificData
• publishVideoFrame
• publishAudioFrame

You will need to provide a Track for the above methods. The list of negotiated tracks can be obtained from the SDP answer after parsing.

For simulcast, the flow is:

• Configure your layers when generating the offer
• setVideoCodecSpecificData
• publishVideoFrame

The SDP answer will have a Track per layer you requested. Use one of these tracks to address the layer.

For subscribing, use the setSubscribeEncodedFrameListener method to receive encoded frames as they come out of the jitter buffer.

The peer connection will maintain connectivity using STUN probe requests if no media is flowing and will attempt to re-establish connectivity as needed. If the re-connection fails, so will the overall connection state.

A command line tool for publishing

Tested on Linux, MacOS, Windows with Pion and Amazon IVS.

If not using the provided .h264 files, you can convert an .mp4 to raw H.264 with FFMPEG. Make sure to use the baseline profile.

ffmpeg -i /path/to/a.mp4 -c:v libx264 -profile:v baseline -level 3.0 \
    -preset medium -an -f h264 out.h264

Build the project using CMake.

cmake . -B build

The crypto library is OpenSSL by default, use -DSRTC_CRYPTO_LIB=MBEDTLS for Mbed TLS (3.4 or newer 3.x, built with MBEDTLS_SSL_DTLS_SRTP). Linux and macOS use the system's library, and on Windows the build downloads prebuilt binaries. To use a library you've built yourself, pass its install prefix:

cmake . -B build -DSRTC_CRYPTO_LIB=MBEDTLS -DSRTC_MBEDTLS_ROOT=/path/to/mbedtls/install
cmake . -B build -DSRTC_CRYPTO_LIB=OPENSSL -DSRTC_OPENSSL_ROOT=/path/to/openssl/install

Change into the build directory and run:

cmake --build .

Change back to the root (cd ..) and run the command line demo. Use --help to see arguments.

./build/srtc_publish[.exe] --help

Should output:

Usage: ./build/srtc_publish [options]
Options:
  -f, --file <path>    Path to a H264/H265/WEBM (webm) file (default: sintel.h264)
  -u, --url <url>      WHIP server URL (default: http://localhost:8080/whip)
  -t, --token <token>  WHIP authorization token
  -l, --loop           Loop the file
  -v, --verbose        Verbose logging from the srtc library
  -q, --quiet          Suppress progress reporting
  -s, --sdp            Print SDP offer and answer
  -i, --info           Print input file info
  -d, --drop           Drop some packets at random (test NACK and RTX handling)
  -b, --bwe            Enable TWCC congestion control for bandwidth estimation
  -h, --help           Show this help message

To broadcast to Amazon IVS:

./build/srtc_publish -f /path/to/out.h264 -u https://global.whip.live-video.net -t [YOUR STAGE TOKEN]

Testing with Pion

Open a new terminal window, change the directory to pion-webrtc-examples-whip-whep and execute run.sh or go run . to start the Pion WebRTC server.

Open a new web browser window to http://localhost:8080, you will see a web page with controls for publishing and subscribing. Click "Subscribe", you should see "Checking" / "Connected" in the status area below and there should be a progress wheel over the video area.

Now switch back to the terminal window where you built srtc and run <your-cmake-dir>/srtc_publish[.exe], making sure the current directory is the srtc directory. This will load a video file and send it to Pion using WHIP.

Switch back to the browser, after a second or two (keyframe delay) you should see the video being sent by srtc.

Using a VP8 input file

First please run the Pion WebRTC server like this to use VP8 (by default it uses H264):

./run.sh -codec vp8

To send video to Pion, run the publish sample like this:

./build/srtc_publish[.exe] -f sintel-vp8.webm

Using a VP9 input file

First please run the Pion WebRTC server like this to use VP9 (by default it uses H264):

./run.sh -codec vp9

To send video to Pion, run the publish sample like this:

./build/srtc_publish[.exe] -f sintel-vp9.webm

Using an H265 input file

First please run the Pion WebRTC server like this to use H265 (by default it uses H264):

./run.sh -codec h265

To send video to Pion, run the publish sample like this:

./build/srtc_publish[.exe] -f sintel.h265

Using an AV1 input file

First please run the Pion WebRTC server like this to use AV1 (by default it uses H264):

./run.sh -codec av1

To send video to Pion, run the publish sample like this:

./build/srtc_publish[.exe] -f sintel-av1.webm

A command line tool for subscribing

./build/srtc_subscribe[.exe] --help

Should output:

Usage: srtc_subscribe [options]
Options:
  -u, --url <url>      WHEP server URL (default: http://localhost:8080/whep)
  -t, --token <token>  WHEP authorization token
  -v, --verbose        Verbose logging from the srtc library
  -q, --quiet          Suppress progress reporting
  -s, --sdp            Print SDP offer and answer
  --oa <filename>      Save audio to a file (ogg format for opus)
  --ov <filename>      Save video to a file (h264 or webm format)
  -d, --drop           Drop some packets at random (test NACK and RTX handling)
  -h, --help           Show this help message

The subscribe tool handles Ctrl+C and SIGTERM and terminates gracefully, flushing and closing the output files.

Note that *.h264 files have no frame rate information, and so may play "very fast" depending on your video player. If using VLC, you can use the below option to adjust playback speed:

      --rate <float>             Playback speed

Running in VP8 mode

Run the Pion server like this, just like for publishing, and use the web page to publish media.

./run.sh -codec vp8

And then subscribe like this:

./build/srtc_subscribe[.exe] --ov output.webm

The resulting webm file will not contain any audio, just video - if you'd like to capture audio as well, please add --oa output.ogg.

Running in VP9 mode

Run the Pion server like this, just like for publishing, and use the web page to publish media.

./run.sh -codec vp9

And then subscribe like this:

./build/srtc_subscribe[.exe] --ov output.webm

The resulting webm file will not contain any audio, just video - if you'd like to capture audio as well, please add --oa output.ogg.

Running in H265 mode

Run the Pion server like this, just like for publishing, and use the web page to publish media.

./run.sh -codec h265

And then subscribe like this:

./build/srtc_subscribe[.exe] --ov output.h265

Running in AV1 mode

Run the Pion server like this, just like for publishing, and use the web page to publish media.

./run.sh -codec av1

And then subscribe like this:

./build/srtc_subscribe[.exe] --ov output.webm

The resulting webm file will not contain any audio, just video - if you'd like to capture audio as well, please add --oa output.ogg.

An Android demo / sample

There is an Android demo:

https://github.com/kmansoft/srtc-android-demo

Note that the code in this library is not Android specific, only the demo app is.

This demo also captures the camera and microphone and publishes them as H264 and Opus to Pion or Amazon IVS.

For the interface between Android code and the srtc library, please see jni_peer_connection.h / .cpp in that project.

A MacOS demo / sample

There is a MacOS demo:

https://github.com/kmansoft/srtc-macos-demo

Note that the code in this library is not MacOS specific, only the demo app is.

This demo also captures the camera and microphone and publishes them as H264 and Opus to Pion or Amazon IVS.

The interface between Swift code of the app and C++ code in srtc is in the srtc-mac subdirectory.

Disclamier

I work for Amazon IVS (Interactive Video Service).

This library is my side project.

Future plans

• Replace Cisco's SRTP library with my new code using OpenSSL / BoringSSL directly. This is done.

• Implement support for Simulcast (multiple video layers on the same peer connection). This is done.

• Support Google's Transport Wide Congestion Control. This is mostly done and will continue to improve.

• Windows port. This is done.

• Releases. This is done.

• Subscribing. This is done, although some improvements are still possible.

• Support for more codecs. Done: VP8, VP9, H264, H265, AV1.

• Support for multiple media lines. This is done.