media/mojo

This folder contains mojo interfaces, clients and implementations that extend the core "media" target to support most out-of-process use cases, including Media Player, Metrics (WatchTime), etc.

Currently the “media” target does not depend on mojo, so that other applications can use the “media” target without having to pull in mojo dependency.

[TOC]

Media Player

Media Components

Media Player (WebMediaPlayer) supports HTML5 <video> playback in Chromium. Internally, it depends on many media components to perform some specific tasks, e.g. media renderer, audio decoder, video decoder, and content decryption module (CDM). A CDM is required for a media renderer, audio decoder or video decoder to handle protected content. See more details in the general media documentation.

While most of HTML5 media player stack and Encrypted Media Extensions (EME) stack live in the sandboxed render process (e.g. for security reasons), there are some cases where some media components must live in a different process. For example:

• A hardware-based media renderer, where all audio/video decoding and rendering happens in hardware, which is not accessible in the sandboxed render process.
• A hardware based video decoder, where the hardware decoding libraries are not accessible in the sandboxed render process.
• On Android, a media component depends on Android Java API, which is not accessible in the sandboxed render process.
• A CDM contains third-party code and should run in its own sandboxed process.

Here we provide a generic framework to support most out-of-process (OOP) media component use cases in Chromium.

Media Player Mojo Interfaces

We use mojom interfaces as the transport layer of each media component to support hosting them remotely. These interfaces are called media player mojo interfaces. They are very similar to their C++ counterparts:

• media::Renderer -> media::mojom::Renderer
• media::AudioDecoder -> media::mojom::AudioDecoder
• media::VideoDecoder -> media::mojom::VideoDecoder
• media::ContentDecryptionModule -> media::mojom::ContentDecryptionModule
• media::Decryptor -> media::mojom::Decryptor

Enable Remote Media Components

Standard clients and implementations of these interfaces are provided. For example, for media::Renderer, MojoRenderer implements media::Renderer, and forwards calls to a media::mojom::RendererPtr. MojoRendererService implements media::mojom::Renderer, and can host any media::Renderer implementation.

Remote media components can be easily enabled and seamlessly integrated in the current media pipeline. Simply set the gn argument mojo_media_services to specify which remote media components you want to enable. For example, with the following gn arguments, the media pipeline will enable MojoRenderer and MojoCdm:

enable_mojo_media = true
mojo_media_services = ["renderer", "cdm"]

Note that you must set enable_mojo_media first. Also, some remote media components are also controlled by run time features, e.g. media::kMojoVideoDecoder.

Media Mojo Interface Factory

media::mojom::InterfaceFactory has factory methods like CreateRenderer(), CreateCdm() etc. It is used to request media player mojo interfaces.

In the render process, each RenderFrameImpl has a media::mojom::InterfaceFactoryPtr which is used to request all media player mojo interfaces for that frame from the browser process. In the browser process, each RenderFrameHostImpl owns a MediaInterfaceProxy, which implements media::mojom::InterfaceFactory.

MediaInterfaceProxy is a central hub for handling media player mojo interface requests. By default it will forward all the requests to the MediaService. But it also has the flexibility to handle some special or more complicated use cases. For example, on desktop platforms, when library CDM is enabled, the media::mojom::ContentDecryptionModule request will be forwarded to the CdmService running in its own CDM (utility) process. For another example, on Android, the media::mojom::Renderer request is handled in the RenderFrameHostImpl context directly by creating MediaPlayerRenderer in the browser process, even though the MediaService is configured to run in the GPU process.

Note that media::mojom::InterfaceFactory interface is reused in the communication between MediaInterfaceProxy and MediaService (see below).

MediaService

The MediaService is a mojo service_manager::Service that provides media player mojo interface implementations. It comes with some nice benefits.

Flexible Process Model

Different platforms or products have different requirements on where the remote media components should run. For example, a hardware decoder typically should run in the GPU process. The ServiceManagerContext provides the ability to run a service_manager::Service in-process (browser), out-of-process (utility) or in the GPU process. Therefore, by using a MediaService, it’s very easy to support hosting remote media components interfaces in most common Chromium process types (Browser/Utility/GPU). This can by set using the gn argument mojo_media_host, e.g.

mojo_media_host = "browser" or “gpu” or “utility”

MediaService is registered in ServiceManagerContext using kMediaServiceName. mojo_media_host is checked to decide in which process the service is registered to run.

Connects Different Media Components

Some remote media components depend on other components to work. For example, a Renderer, an audio decoder or a video decoder needs a CDM to be able to handle encrypted streams. Typically there's a SetCdm() call to connect the renderer or decoder with the CDM. If, for example, a Renderer interface and a CDM interface are hosted separately, then it will be hard to implement the SetCdm() call. It would require an object or entity that are aware of both sides to be able to connect them. MediaService handles this internally, and is actually serving as such an object or entity, so you don’t have to reinvent the wheel. See more details below.

Customization through MojoMediaClient

MediaService provides everything needed to host an OOP media component, but it doesn’t provide the media component itself. It’s up to the client of MediaService to provide the concrete media component implementations.

The MojoMediaClient interface provides a way for MediaService clients to provide concrete media components’ implementations. When MediaService is created, a MojoMediaClient must be passed in so that MediaService knows how to create the media components.

For example, ChromeCast uses MediaService to host a media Renderer and a CDM in the browser process, and it provides the CastRenderer and CastCdm through CastMojoMediaClient, a MojoMediaClient implementation. Note that this overriding mechanism is not implemented everywhere. It’s trivial to add the support and we’ll only add it when we need it.

Site Isolation

In Blink, both media element and EME MediaKeys belong to a WebLocalFrame. In Chromium, this translates to media player and CDM belonging to a RenderFrame. In the render process, this is clear. However, when hosting all remote media components in a single MediaService (service manager only supports one service instance per process), the Frame boundary could get fuzzy. This will be especially dangerous for media components that interact with each other. For example, a Renderer from foo.com lives in the same MediaService instance as a CDM from bar.net. It would be wrong if the bar.net CDM is set on the foo.com Renderer to handle decryption.

To prevent this from happening, we introduce an additional layer to simulate the RenderFrame boundary. A MediaService hosts multiple InterfaceFactory (one per RenderFrame), and each InterfaceFactory creates and manages media components it creates.

For this reason, media::mojom::InterfaceFactory interface is reused in the communication between MediaInterfaceProxy and MediaService.

Note: there are plans to split apart the responsibilities of media::mojom::InterfaceFactory to make it clear which interfaces are used where.

Support Other Clients

MediaService, as a service_manager::Service, can be used by clients other than the media player in the render process. For example, we could have another (mojo) service that handles audio data and wants to play it in a media Renderer. Since MediaService is a mojo service, it’s very convenient for any other mojo service to connect to it through a service_manager::mojom::Connector and use the remote media Renderer it hosts.

CdmService

Although MediaService supports media::mojom::CDM, in some cases (e.g. library CDM on desktop) the remote CDM needs to run in its own process, typically for security reasons. CdmService is introduced to handle this. It also implements service_manager::Service, and is registered in ServiceManagerContext using kCdmServiceName. Currently it’s always registered to run in the utility process (with CDM sandbox type). CdmService also has additional support on library CDM, e.g. loading the library CDM etc. Note that CdmService only supports media::mojom::CDM and does NOT support other media player mojo interfaces.

Mojo CDM and Mojo Decryptor

Mojo CDM is special among all media player mojo interfaces because it is needed by all local/remote media components to handle encrypted buffers:

1. Local media components like DecryptingDemuxerStream, DecryptingAudioDecoder and DecryptingVideoDecoder.
2. Remote media components hosted in MediaService, e.g. by MojoRendererService, MojoAudioDecoderService and MojoVideoDecoderService.
3. Legacy remote media components like AndroidVideoDecodeAccelerator.

At the JavaScript layer, the media player and MediaKeys are connected via setMediaKeys(). This is implemented by SetCdm() in the render process.

A media component can use a CDM in two ways.

Using a Decryptor (via CdmContext)

Some CDM provides a Decryptor implementation, which supports decrypting methods directly, e.g. Decrypt(), DecryptAndDecode() etc. Both the AesDecryptor and library CDM support the Decryptor interface.

In the case of a remote CDM, e.g. hosted by MojoCdmService in MediaService or CdmService, if the remote CDM supports the Decryptor interface, the MojoCdm will also support the Decryptor interface, implemented by MojoDecryptor, which set up a new message pipe to forward all Decryptor calls to the Decryptor in the remote CDM.

Using CdmContext

In some cases the media component is set to work with a specific CDM. For example, on Android, MediaCodec-based decoders (e.g. MediaCodecAudioDecoder and AndroidVideoDecodeAccelerator) can only use MediaDrm-based CDM via MediaCryptoContext. The media component and the CDM must live in the same process because the interaction of these two are typically happening deep at the OS level. In theory, they can both live in the render process. But in practice, typically both the CDM and the media component are hosted by the MediaService in a remote (e.g. GPU) process.

To be able to attach a remote CDM with a remote media component, each InterfaceFactoryImpl instance (corresponding to one RenderFrame) in the MediaService maintains a MojoCdmServiceContext that keeps track of all remote CDMs created for the RenderFrame. Each remote CDM is assigned a unique CDM ID, which is sent back to the MojoCdm in the render process. In the render process, when SetCdm() is called, the CDM ID is passed to the local media component (e.g. MojoRenderer), which is forwarded the remote media component (e.g. MojoRendererService). The remote media component will talk to MojoCdmServiceContext to get the CdmContext associated with the CDM ID, and complete the connection.

Secure Auxiliary Services

Media components often need other services to work. In the render process, the local media components get services from content layer through the MediaClient interface. In MediaService and CdmService, remote media components get services from the through secure auxiliary services.

Note that as a service_manager::Service, MediaService and CdmService can always connect to other service_manager::Service hosted by the service_manager through the Connector interface. However, these are generic services that doesn’t belong to any individual RenderFrame, or even user profile.

Some services do require RenderFrame or user profile identity, e.g. file system. Since media components all belong to a given RenderFrame, we must maintain the frame identity when accessing these services for security reasons. These services are called secure auxiliary services. FrameServiceBase is a base class for all secure auxiliary services to help manage the lifetime of these services (e.g. to handle navigation).

In MediaInterfaceProxy, when we request media::mojom::InterfaceFactory in the MediaService or CdmService, we call GetFrameServices() to configure which secure auxiliary services are exposed to the remote components over the separate service_manager::mojom::InterfaceProvider.

Currently only the remote CDM needs secure auxiliary services. This is a list of currently supported services:

• OutputProtection: to check output protection status
• PlatformVerification: to check whether the platform is secure
• CdmFileIO: for the CDM to store persistent data
• ProvisionFetcher: for Android MediaDrm device provisioning
• CdmProxy: (in progress)

Adoption

Android

• MediaService in the GPU process (registered in GpuServiceFactory with GpuMojoMediaClient)
• MojoCdm + MediaDrmBridge (CDM)
• MediaDrmBridge uses mojo ProvisionFetcher service for CDM provisioning
• MojoAudioDecoder + MediaCodecAudioDecoder
• MojoVideoDecoder + MediaCodecVideoDecoder (in progress)
• HLS support:
  • MojoRenderer + MediaPlayerRenderer
  • NOT using MediaService. Instead, MojoRendererService is hosted by RenderFrameHostImpl/MediaInterfaceProxy in the browser process directly.

ChromeCast

• MediaService in the Browser process (registered in CastContentBrowserClient with CastMojoMediaClient)
• MojoRenderer + CastRenderer
• MojoCdm + CastCdm

Desktop (ChromeOS/Linux/Mac/Windows)

• CdmService
  • CdmService in the utility process (registered in UtilityServiceFactory with ContentCdmServiceClient)
  • MojoCdm + CdmAdapter + Library CDM implementation
  • CdmAdapter uses various secure auxiliary services
• MediaService (in progress)
  • MediaService in the GPU process (registered in GpuServiceFactory with GpuMojoMediaClient)
  • MojoVideoDecoder + hardware video decoders such as D3D11VideoDecoder
  • Provides CdmProxy to the CdmService

Other Services

TODO(xhwang): Add documentation on other mojo services, e.g. remoting, etc.