webrtc: support publishing and reading H265 tracks (#4003)

2024-12-27 17:12:21 +00:00 · 2024-12-02 23:55:54 +01:00 · 2024-12-02 23:55:54 +01:00 · 72a8b3ca8a
commit 72a8b3ca8a
parent 235fd27dce
7 changed files with 151 additions and 43 deletions
--- a/README.md
+++ b/README.md
@ -22,13 +22,13 @@ Live streams can be published to the server with:
 |--------|--------|------------|------------|
 |[SRT clients](#srt-clients)||H265, H264, MPEG-4 Video (H263, Xvid), MPEG-1/2 Video|Opus, MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), AC-3|
 |[SRT cameras and servers](#srt-cameras-and-servers)||H265, H264, MPEG-4 Video (H263, Xvid), MPEG-1/2 Video|Opus, MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), AC-3|
-|[WebRTC clients](#webrtc-clients)|WHIP|AV1, VP9, VP8, H265, H264|Opus, G722, G711 (PCMA, PCMU)|
-|[WebRTC servers](#webrtc-servers)|WHEP|AV1, VP9, VP8, H265, H264|Opus, G722, G711 (PCMA, PCMU)|
+|[WebRTC clients](#webrtc-clients)|WHIP|AV1, VP9, VP8, [H265](#supported-codecs), H264|Opus, G722, G711 (PCMA, PCMU)|
+|[WebRTC servers](#webrtc-servers)|WHEP|AV1, VP9, VP8, [H265](#supported-codecs), H264|Opus, G722, G711 (PCMA, PCMU)|
 |[RTSP clients](#rtsp-clients)|UDP, TCP, RTSPS|AV1, VP9, VP8, H265, H264, MPEG-4 Video (H263, Xvid), MPEG-1/2 Video, M-JPEG and any RTP-compatible codec|Opus, MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), AC-3, G726, G722, G711 (PCMA, PCMU), LPCM and any RTP-compatible codec|
 |[RTSP cameras and servers](#rtsp-cameras-and-servers)|UDP, UDP-Multicast, TCP, RTSPS|AV1, VP9, VP8, H265, H264, MPEG-4 Video (H263, Xvid), MPEG-1/2 Video, M-JPEG and any RTP-compatible codec|Opus, MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), AC-3, G726, G722, G711 (PCMA, PCMU), LPCM and any RTP-compatible codec|
 |[RTMP clients](#rtmp-clients)|RTMP, RTMPS, Enhanced RTMP|AV1, VP9, H265, H264|MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), G711 (PCMA, PCMU), LPCM|
 |[RTMP cameras and servers](#rtmp-cameras-and-servers)|RTMP, RTMPS, Enhanced RTMP|AV1, VP9, H265, H264|MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), G711 (PCMA, PCMU), LPCM|
-|[HLS cameras and servers](#hls-cameras-and-servers)|Low-Latency HLS, MP4-based HLS, legacy HLS|AV1, VP9, H265, H264|Opus, MPEG-4 Audio (AAC)|
+|[HLS cameras and servers](#hls-cameras-and-servers)|Low-Latency HLS, MP4-based HLS, legacy HLS|AV1, VP9, [H265](#supported-codecs-1), H264|Opus, MPEG-4 Audio (AAC)|
 |[UDP/MPEG-TS](#udpmpeg-ts)|Unicast, broadcast, multicast|H265, H264, MPEG-4 Video (H263, Xvid), MPEG-1/2 Video|Opus, MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), AC-3|
 |[Raspberry Pi Cameras](#raspberry-pi-cameras)||H264||

@ -37,10 +37,10 @@ Live streams can be read from the server with:
 |protocol|variants|video codecs|audio codecs|
 |--------|--------|------------|------------|
 |[SRT](#srt)||H265, H264, MPEG-4 Video (H263, Xvid), MPEG-1/2 Video|Opus, MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), AC-3|
-|[WebRTC](#webrtc)|WHEP|AV1, VP9, VP8, H264|Opus, G722, G711 (PCMA, PCMU)|
+|[WebRTC](#webrtc)|WHEP|AV1, VP9, VP8, [H265](#supported-codecs), H264|Opus, G722, G711 (PCMA, PCMU)|
 |[RTSP](#rtsp)|UDP, UDP-Multicast, TCP, RTSPS|AV1, VP9, VP8, H265, H264, MPEG-4 Video (H263, Xvid), MPEG-1/2 Video, M-JPEG and any RTP-compatible codec|Opus, MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3), AC-3, G726, G722, G711 (PCMA, PCMU), LPCM and any RTP-compatible codec|
 |[RTMP](#rtmp)|RTMP, RTMPS, Enhanced RTMP|H264|MPEG-4 Audio (AAC), MPEG-1/2 Audio (MP3)|
-|[HLS](#hls)|Low-Latency HLS, MP4-based HLS, legacy HLS|AV1, VP9, H265, H264|Opus, MPEG-4 Audio (AAC)|
+|[HLS](#hls)|Low-Latency HLS, MP4-based HLS, legacy HLS|AV1, VP9, [H265](#supported-codecs-1), H264|Opus, MPEG-4 Audio (AAC)|

 Live streams be recorded and played back with:

@ -138,6 +138,9 @@ _rtsp-simple-server_ has been rebranded as _MediaMTX_. The reason is pretty obvi
  * [WebRTC-specific features](#webrtc-specific-features)
    * [Authenticating with WHIP/WHEP](#authenticating-with-whipwhep)
    * [Solving WebRTC connectivity issues](#solving-webrtc-connectivity-issues)
+    * [Supported-codecs](#supported-codecs)
+  * [HLS-specific features](#hls-specific-features)
+    * [Supported codecs](#supported-codecs-1)
  * [RTSP-specific features](#rtsp-specific-features)
    * [Transport protocols](#transport-protocols)
    * [Encryption](#encryption)
@ -1180,19 +1183,6 @@ and can also be accessed without using the browsers, by software that supports t
 http://localhost:8888/mystream/index.m3u8
 ```

-Although the server can produce HLS with a variety of video and audio codecs (that are listed at the beginning of the README), not all browsers can read all codecs.
-
-You can check what codecs your browser can read by [using this tool](https://jsfiddle.net/g1qyf4ea).
-
-If you want to support most browsers, you can to re-encode the stream by using the H264 and AAC codecs, for instance by using FFmpeg:
-
-```sh
-ffmpeg -i rtsp://original-source \
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -b:v 600k \
-c:a aac -b:a 160k \
-f rtsp rtsp://localhost:8554/mystream
-```
-
 Known clients that can read with HLS are [FFmpeg](#ffmpeg-1), [GStreamer](#gstreamer-1), [VLC](#vlc) and [web browsers](#web-browsers-1).

 ##### LL-HLS
@ -2196,6 +2186,49 @@ webrtcICEServers2:
  clientOnly: true
 ```

+#### Supported codecs
+
+The server can ingest and broadcast with WebRTC a wide variety of video and audio codecs (that are listed at the beginning of the README), but not all browsers can publish and read all codecs due to internal limitations that cannot be overcome by this or any other server.
+
+In particular, reading and publishing H265 tracks with WebRTC was not possible until some time ago due to the lack of browser support. The situation recently improved and can be described as following:
+
+* Safari on iOS and macOS fully supports publishing and reading H265 tracks
+* Chrome on Windows supports publishing and reading H265 tracks when a GPU is present and when the browser is launched with the following flags:
+
+  ```
+  chrome.exe --enable-features=PlatformHEVCEncoderSupport,WebRtcAllowH265Receive,WebRtcAllowH265Send --force-fieldtrials=WebRTC-Video-H26xPacketBuffer/Enabled
+  ```
+
+  We are expecting these flags to become redundant in the future and the feature to be turned on by default.
+
+You can check what codecs your browser can publish or read with WebRTC by [using this tool](https://jsfiddle.net/v24s8q1f/).
+
+If you want to support most browsers, you can to re-encode the stream by using H264 and Opus codecs, for instance by using FFmpeg:
+
+```sh
+ffmpeg -i rtsp://original-source \
+-c:v libx264 -pix_fmt yuv420p -preset ultrafast -b:v 600k \
+-c:a libopus -b:a 64K -async 50 \
+-f rtsp rtsp://localhost:8554/mystream
+```
+
+### HLS-specific features
+
+#### Supported codecs
+
+The server can produce HLS streams with a variety of video and audio codecs (that are listed at the beginning of the README), but not all browsers can read all codecs due to internal limitations that cannot be overcome by this or any other server.
+
+You can check what codecs your browser can read with HLS by [using this tool](https://jsfiddle.net/tjcyv5aw/).
+
+If you want to support most browsers, you can to re-encode the stream by using H264 and AAC codecs, for instance by using FFmpeg:
+
+```sh
+ffmpeg -i rtsp://original-source \
+-c:v libx264 -pix_fmt yuv420p -preset ultrafast -b:v 600k \
+-c:a aac -b:a 160k \
+-f rtsp rtsp://localhost:8554/mystream
+```
+
 ### RTSP-specific features

 #### Transport protocols
@ -2393,6 +2426,11 @@ All the code in this repository is released under the [MIT License](LICENSE). Co
 |[Enhanced RTMP v1](https://veovera.org/docs/enhanced/enhanced-rtmp-v1.pdf)|RTMP|
 |[Action Message Format](https://rtmp.veriskope.com/pdf/amf0-file-format-specification.pdf)|RTMP|
 |[WebRTC: Real-Time Communication in Browsers](https://www.w3.org/TR/webrtc/)|WebRTC|
+|[RFC8835, Transports for WebRTC](https://datatracker.ietf.org/doc/html/rfc8835)|WebRTC|
+|[RFC7742, WebRTC Video Processing and Codec Requirements](https://datatracker.ietf.org/doc/html/rfc7742)|WebRTC|
+|[RFC7847, WebRTC Audio Codec and Processing Requirements](https://datatracker.ietf.org/doc/html/rfc7874)|WebRTC|
+|[RFC7875, Additional WebRTC Audio Codecs for Interoperability](https://datatracker.ietf.org/doc/html/rfc7875)|WebRTC|
+|[H.265 Profile for WebRTC](https://datatracker.ietf.org/doc/draft-ietf-avtcore-hevc-webrtc/)|WebRTC|
 |[WebRTC HTTP Ingestion Protocol (WHIP)](https://datatracker.ietf.org/doc/draft-ietf-wish-whip/)|WebRTC|
 |[WebRTC HTTP Egress Protocol (WHEP)](https://datatracker.ietf.org/doc/draft-murillo-whep/)|WebRTC|
 |[The SRT Protocol](https://haivision.github.io/srt-rfc/draft-sharabayko-srt.html)|SRT|
--- a/internal/protocols/webrtc/from_stream.go
+++ b/internal/protocols/webrtc/from_stream.go
@ -8,6 +8,7 @@ import (
 	"github.com/bluenviron/gortsplib/v4/pkg/format"
 	"github.com/bluenviron/gortsplib/v4/pkg/format/rtpav1"
 	"github.com/bluenviron/gortsplib/v4/pkg/format/rtph264"
+	"github.com/bluenviron/gortsplib/v4/pkg/format/rtph265"
 	"github.com/bluenviron/gortsplib/v4/pkg/format/rtplpcm"
 	"github.com/bluenviron/gortsplib/v4/pkg/format/rtpvp8"
 	"github.com/bluenviron/gortsplib/v4/pkg/format/rtpvp9"
@ -23,7 +24,8 @@ const (
 )

 var errNoSupportedCodecsFrom = errors.New(
-	"the stream doesn't contain any supported codec, which are currently AV1, VP9, VP8, H264, Opus, G722, G711, LPCM")
+	"the stream doesn't contain any supported codec, which are currently " +
+		"AV1, VP9, VP8, H265, H264, Opus, G722, G711, LPCM")

 func uint16Ptr(v uint16) *uint16 {
 	return &v
@ -189,10 +191,69 @@ func setupVideoTrack(
 		return vp8Format, nil
 	}

+	var h265Format *format.H265
+	media = stream.Desc().FindFormat(&h265Format)
+
+	if h265Format != nil { //nolint:dupl
+		track := &OutgoingTrack{
+			Caps: webrtc.RTPCodecCapability{
+				MimeType:    webrtc.MimeTypeH265,
+				ClockRate:   90000,
+				SDPFmtpLine: "level-id=93;profile-id=1;tier-flag=0;tx-mode=SRST",
+			},
+		}
+		pc.OutgoingTracks = append(pc.OutgoingTracks, track)
+
+		encoder := &rtph265.Encoder{
+			PayloadType:    96,
+			PayloadMaxSize: webrtcPayloadMaxSize,
+		}
+		err := encoder.Init()
+		if err != nil {
+			return nil, err
+		}
+
+		firstReceived := false
+		var lastPTS int64
+
+		stream.AddReader(
+			reader,
+			media,
+			h265Format,
+			func(u unit.Unit) error {
+				tunit := u.(*unit.H265)
+
+				if tunit.AU == nil {
+					return nil
+				}
+
+				if !firstReceived {
+					firstReceived = true
+				} else if tunit.PTS < lastPTS {
+					return fmt.Errorf("WebRTC doesn't support H265 streams with B-frames")
+				}
+				lastPTS = tunit.PTS
+
+				packets, err := encoder.Encode(tunit.AU)
+				if err != nil {
+					return nil //nolint:nilerr
+				}
+
+				for _, pkt := range packets {
+					pkt.Timestamp += tunit.RTPPackets[0].Timestamp
+					track.WriteRTP(pkt) //nolint:errcheck
+				}
+
+				return nil
+			})
+
+		return h265Format, nil
+	}
+
 	var h264Format *format.H264
 	media = stream.Desc().FindFormat(&h264Format)

-	if h264Format != nil {
+	if h264Format != nil { //nolint:dupl
 		track := &OutgoingTrack{
 			Caps: webrtc.RTPCodecCapability{
 				MimeType:    webrtc.MimeTypeH264,
--- a/internal/protocols/webrtc/from_stream_test.go
+++ b/internal/protocols/webrtc/from_stream_test.go
@ -18,7 +18,7 @@ func TestFromStreamNoSupportedCodecs(t *testing.T) {
 		1460,
 		&description.Session{Medias: []*description.Media{{
 			Type:    description.MediaTypeVideo,
-			Formats: []format.Format{&format.H265{}},
+			Formats: []format.Format{&format.MJPEG{}},
 		}}},
 		true,
 		test.NilLogger,
@ -44,7 +44,7 @@ func TestFromStreamSkipUnsupportedTracks(t *testing.T) {
 			},
 			{
 				Type:    description.MediaTypeVideo,
-				Formats: []format.Format{&format.H265{}},
+				Formats: []format.Format{&format.MJPEG{}},
 			},
 		}},
 		true,
@ -57,7 +57,7 @@ func TestFromStreamSkipUnsupportedTracks(t *testing.T) {
 	l := test.Logger(func(l logger.Level, format string, args ...interface{}) {
 		require.Equal(t, logger.Warn, l)
 		if n == 0 {
-			require.Equal(t, "skipping track 2 (H265)", fmt.Sprintf(format, args...))
+			require.Equal(t, "skipping track 2 (M-JPEG)", fmt.Sprintf(format, args...))
 		}
 		n++
 	})
@ -73,9 +73,6 @@ func TestFromStreamSkipUnsupportedTracks(t *testing.T) {

 func TestFromStream(t *testing.T) {
 	for _, ca := range toFromStreamCases {
-		if ca.in == nil {
-			continue
-		}
 		t.Run(ca.name, func(t *testing.T) {
 			stream, err := stream.New(
 				512,
--- a/internal/protocols/webrtc/incoming_track.go
+++ b/internal/protocols/webrtc/incoming_track.go
@ -75,16 +75,17 @@ var incomingVideoCodecs = []webrtc.RTPCodecParameters{
 	},
 	{
 		RTPCodecCapability: webrtc.RTPCodecCapability{
-			MimeType:  webrtc.MimeTypeH265,
-			ClockRate: 90000,
+			MimeType:    webrtc.MimeTypeH265,
+			ClockRate:   90000,
+			SDPFmtpLine: "level-id=93;profile-id=2;tier-flag=0;tx-mode=SRST",
 		},
 		PayloadType: 103,
 	},
 	{
 		RTPCodecCapability: webrtc.RTPCodecCapability{
-			MimeType:    webrtc.MimeTypeH264,
+			MimeType:    webrtc.MimeTypeH265,
 			ClockRate:   90000,
-			SDPFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f",
+			SDPFmtpLine: "level-id=93;profile-id=1;tier-flag=0;tx-mode=SRST",
 		},
 		PayloadType: 104,
 	},
@ -92,10 +93,18 @@ var incomingVideoCodecs = []webrtc.RTPCodecParameters{
 		RTPCodecCapability: webrtc.RTPCodecCapability{
 			MimeType:    webrtc.MimeTypeH264,
 			ClockRate:   90000,
-			SDPFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f",
+			SDPFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f",
 		},
 		PayloadType: 105,
 	},
+	{
+		RTPCodecCapability: webrtc.RTPCodecCapability{
+			MimeType:    webrtc.MimeTypeH264,
+			ClockRate:   90000,
+			SDPFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f",
+		},
+		PayloadType: 106,
+	},
 }

 var incomingAudioCodecs = []webrtc.RTPCodecParameters{
--- a/internal/protocols/webrtc/to_stream_test.go
+++ b/internal/protocols/webrtc/to_stream_test.go
@ -72,10 +72,13 @@ var toFromStreamCases = []struct {
 	},
 	{
 		"h265",
-		nil,
+		&format.H265{
+			PayloadTyp: 96,
+		},
 		webrtc.RTPCodecCapability{
-			MimeType:  "video/H265",
-			ClockRate: 90000,
+			MimeType:    "video/H265",
+			ClockRate:   90000,
+			SDPFmtpLine: "level-id=93;profile-id=1;tier-flag=0;tx-mode=SRST",
 		},
 		&format.H265{
 			PayloadTyp: 96,
--- a/internal/servers/webrtc/publish_index.html
+++ b/internal/servers/webrtc/publish_index.html
@ -312,7 +312,7 @@ const populateCodecs = () => {
    .then((desc) => {
      const sdp = desc.sdp.toLowerCase();

-      for (const codec of ['av1/90000', 'vp9/90000', 'vp8/90000', 'h264/90000']) {
+      for (const codec of ['av1/90000', 'vp9/90000', 'vp8/90000', 'h264/90000', 'h265/90000']) {
        if (sdp.includes(codec)) {
          const opt = document.createElement('option');
          opt.value = codec;
--- a/internal/servers/webrtc/reader.js
+++ b/internal/servers/webrtc/reader.js
@ -4,16 +4,16 @@

  const supportsNonAdvertisedCodec = (codec, fmtp) => (
    new Promise((resolve) => {
-      const payloadType = 118;
+      const payloadType = 118; // TODO: dynamic
      const pc = new RTCPeerConnection({ iceServers: [] });
-      pc.addTransceiver('audio', { direction: 'recvonly' });
+      const mediaType = 'audio';
+      pc.addTransceiver(mediaType, { direction: 'recvonly' });
      pc.createOffer()
        .then((offer) => {
          if (offer.sdp.includes(' ' + codec)) { // codec is advertised, there's no need to add it manually
-            resolve(false);
-            return;
+            throw new Error('already present');
          }
-          const sections = offer.sdp.split('m=audio');
+          const sections = offer.sdp.split(`m=${mediaType}`);
          const lines = sections[1].split('\r\n');
          lines[0] += ` ${payloadType}`;
          lines.splice(lines.length - 1, 0, `a=rtpmap:${payloadType} ${codec}`);
@ -21,7 +21,7 @@
            lines.splice(lines.length - 1, 0, `a=fmtp:${payloadType} ${fmtp}`);
          }
          sections[1] = lines.join('\r\n');
-          offer.sdp = sections.join('m=audio');
+          offer.sdp = sections.join(`m=${mediaType}`);
          return pc.setLocalDescription(offer);
        })
        .then(() => {
@ -32,7 +32,7 @@
            + 's=-\r\n'
            + 't=0 0\r\n'
            + 'a=fingerprint:sha-256 0D:9F:78:15:42:B5:4B:E6:E2:94:3E:5B:37:78:E1:4B:54:59:A3:36:3A:E5:05:EB:27:EE:8F:D2:2D:41:29:25\r\n'
-            + `m=audio 9 UDP/TLS/RTP/SAVPF ${payloadType}` + '\r\n'
+            + `m=${mediaType} 9 UDP/TLS/RTP/SAVPF ${payloadType}` + '\r\n'
            + 'c=IN IP4 0.0.0.0\r\n'
            + 'a=ice-pwd:7c3bf4770007e7432ee4ea4d697db675\r\n'
            + 'a=ice-ufrag:29e036dc\r\n'
@ -331,7 +331,7 @@
      return Promise.all([
        ['pcma/8000/2'],
        ['multiopus/48000/6', 'channel_mapping=0,4,1,2,3,5;num_streams=4;coupled_streams=2'],
-        ['L16/48000/2']
+        ['L16/48000/2'],
      ]
        .map((c) => supportsNonAdvertisedCodec(c[0], c[1]).then((r) => (r) ? c[0] : false)))
        .then((c) => c.filter((e) => e !== false))