Skip to content

membraneframework/membrane_core

Repository files navigation

Membrane Framework

Hex.pm API Docs CircleCI

Membrane is a versatile multimedia streaming & processing framework. You can use it to build a media server of your need, that can:

  • stream via WebRTC, RTSP, RTMP, HLS, HTTP and other protocols,
  • transcode, mix and apply custom processing of video & audio,
  • accept and generate / record to MP4, MKV, FLV and other containers,
  • handle dynamically connecting and disconnecting streams,
  • seamlessly scale and recover from errors,
  • do whatever you imagine if you implement it yourself :D Membrane makes it easy to plug in your code at almost any point of processing.

The abbreviations above don't ring any bells? Visit membrane.stream/learn and let Membrane introduce you to the multimedia world!

If you have questions or need consulting, we're for you at our Discord, forum, GitHub discussions, X (Twitter) and via e-mail.

You can also follow Membrane on X (Twitter) or join our Discord to be up to date and get involved in the community.

Membrane is maintained by Software Mansion.

Quick start

Mix.install([
  :membrane_hackney_plugin,
  :membrane_mp3_mad_plugin,
  :membrane_portaudio_plugin,
])

defmodule MyPipeline do
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, mp3_url) do
    spec =
      child(%Membrane.Hackney.Source{
        location: mp3_url, hackney_opts: [follow_redirect: true]
      })
      |> child(Membrane.MP3.MAD.Decoder)
      |> child(Membrane.PortAudio.Sink)

    {[spec: spec], %{}}
  end
end

mp3_url = "https://raw.githubusercontent.com/membraneframework/membrane_demo/master/simple_pipeline/sample.mp3"

Membrane.Pipeline.start_link(MyPipeline, mp3_url)

This is an Elixir snippet, that streams an mp3 via HTTP and plays it on your speaker. Here's how to run it:

  • Option 1: Click the button below:

    Run in Livebook

    It will install Livebook, an interactive notebook similar to Jupyter, and it'll open the snippet in there for you. Then just click the 'run' button in there.

  • Option 2: If you don't want to use Livebook, you can install Elixir, type iex to run the interactive shell and paste the snippet there.

After that, you should hear music playing on your speaker 🎉

To learn step-by-step what exactly happens here, follow this tutorial.

Learning

The best place to learn Membrane is the membrane.stream/learn website and the membrane_demo repository. Try them out, then hack something exciting!

Structure of the framework

The most basic media processing entities of Membrane are Elements. An element might be able, for example, to mux incoming audio and video streams into MP4, or play raw audio using your sound card. You can create elements yourself, or choose from the ones provided by the framework.

Elements can be organized into a pipeline - a sequence of linked elements that perform a specific task. For example, a pipeline might receive an incoming RTSP stream from a webcam and convert it to an HLS stream, or act as a selective forwarding unit (SFU) to implement your own videoconferencing room. The Quick start section above shows how to create a simple pipeline.

Membrane packages

To embrace modularity, Membrane is delivered to you in multiple packages, including plugins, formats, core and standalone libraries. The complete list of all the Membrane packages maintained by the Membrane team is available here.

Plugins

Plugins provide elements that you can use in your pipeline. Each plugin lives in a membrane_X_plugin repository, where X can be a protocol, codec, container or functionality, for example membrane_opus_plugin. Plugins wrapping a tool or library are named membrane_X_LIBRARYNAME_plugin or just membrane_LIBRARYNAME_plugin, like membrane_mp3_mad_plugin. Plugins are published on hex.pm, for example hex.pm/packages/membrane_opus_plugin and docs are at hexdocs, like hexdocs.pm/membrane_opus_plugin. Some plugins require native libraries installed in your OS. Those requirements, along with usage examples are outlined in each plugin's readme.

Formats

Apart from plugins, Membrane has stream formats, which live in membrane_X_format repositories, where X is usually a codec or container, for example mebrane_opus_format. Stream formats are published the same way as packages and are used by elements to define what kind of stream can be sent or received. They also provide utility functions to deal with a given codec/container.

Core

The API for creating pipelines (and custom elements too) is provided by membrane_core. To install it, add the following line to your deps in mix.exs and run mix deps.get

{:membrane_core, "~> 1.1"}

Standalone libraries

Last but not least, Membrane provides tools and libraries that can be used standalone and don't depend on the membrane_core, for example video_compositor, ex_sdp or unifex.

Goals

The main goals of Membrane are:

  • To make work with multimedia a more pleasant experience than it is now.
  • To provide a welcoming ecosystem for learning multimedia development.
  • To power resilient, maintainable and scalable systems.

Elixir language

We chose Elixir for Membrane because it's a modern, high-level, easy-to-learn language, that lets us rapidly develop media solutions. Elixir's main selling points are built-in parallelism and fault-tolerance features, so we can build scalable systems that are self-healing. It also runs on the battle-tested BEAM VM, that's been widely used and actively developed since the '80s. When we need the performance of a low-level language, we delegate to Rust or C.

If you don't know Elixir, try this tutorial - it shouldn't take long and you'll know more than enough to get started with Membrane.

Contributing

We welcome everyone to contribute to Membrane. Here are some ways to contribute:

  • Spread the word about Membrane! Even though multimedia are present everywhere today, media dev is still quite niche. Let it be no longer!
  • Create learning materials. We try our best but can cover only a limited number of Membrane use cases.
  • Improve docs. We know it's not the most exciting part, but if you had a hard time understanding the docs, you're the best person to fix them ;)
  • Contribute code - plugins, features and bug fixes. It's best to contact us before, so we can provide our help & assistance, and agree on important matters. For details see the contribution guide.

Support and questions

If you have any questions regarding Membrane Framework or need consulting, feel free to contact us via Discord, forum, GitHub discussions, X (Twitter) or e-mail.

All packages

General

Package Description Links
membrane_sdk Full power of Membrane in a single package Hex.pm Docs
membrane_core The core of the Membrane Framework, advanced multimedia processing framework Hex.pm Docs
membrane_rtc_engine [Maintainer: fishjam-dev] Customizable Real-time Communication Engine/SFU library focused on WebRTC. Hex.pm Docs
kino_membrane Utilities for introspecting Membrane pipelines in Livebook Hex.pm Docs
docker_membrane [Labs] A docker image based on Ubuntu, with Erlang, Elixir and libraries necessary to test and run the Membrane Framework.
membrane_demo Examples of using the Membrane Framework
membrane_tutorials Repository which contains text and assets used in Membrane Framework tutorials.

Plugins

General purpose

Package Description Links
membrane_file_plugin Membrane plugin for reading and writing to files Hex.pm Docs
membrane_hackney_plugin HTTP sink and source based on Hackney Hex.pm Docs
membrane_scissors_plugin Element for cutting off parts of the stream Hex.pm Docs
membrane_tee_plugin Membrane plugin for splitting data from a single input to multiple outputs Hex.pm Docs
membrane_funnel_plugin Membrane plugin for merging multiple input streams into a single output Hex.pm Docs
membrane_realtimer_plugin Membrane element limiting playback speed to realtime, according to buffers' timestamps Hex.pm Docs
membrane_stream_plugin Plugin for recording the entire stream sent through Membrane pads into a binary format and replaying it Hex.pm Docs
membrane_fake_plugin Fake Membrane sinks that drop incoming data Hex.pm Docs
membrane_pcap_plugin [Labs] Membrane PCAP source, capable of reading captured packets in pcap format
membrane_live_framerate_converter_plugin [Maintainer: kim-company] Membrane.Filter that drops or duplicates frames to match a target framerate. Designed for realtime applications
membrane_template_plugin Template for Membrane Elements

Streaming protocols

Package Description Links
membrane_webrtc_plugin Plugin for streaming via WebRTC Hex.pm Docs
membrane_rtmp_plugin RTMP server & client Hex.pm Docs
membrane_http_adaptive_stream_plugin Plugin generating manifests for HLS Hex.pm Docs
membrane_ice_plugin [Maintainer: fishjam-dev] Membrane plugin for ICE protocol Hex.pm Docs
membrane_udp_plugin Membrane plugin for sending and receiving UDP streams Hex.pm Docs
membrane_tcp_plugin Membrane plugin for sending and receiving TCP streams Hex.pm Docs
membrane_rtp_plugin Membrane bins and elements for sending and receiving RTP/SRTP and RTCP/SRTCP streams Hex.pm Docs
membrane_rtp_h264_plugin Membrane RTP payloader and depayloader for H264 Hex.pm Docs
membrane_rtp_vp8_plugin Membrane elements for payloading and depayloading VP8 into RTP Hex.pm Docs
membrane_rtp_vp9_plugin [Labs] Membrane elements for payloading and depayloading VP9 into RTP
membrane_rtp_mpegaudio_plugin Membrane RTP MPEG Audio depayloader Hex.pm Docs
membrane_rtp_opus_plugin Membrane RTP payloader and depayloader for OPUS audio Hex.pm Docs
membrane_rtp_g711_plugin [Maintainer: fishjam-dev] Membrane RTP payloader and depayloader for G711 audio Hex.pm Docs
membrane_quic_plugin [Maintainer: mickel8]
membrane_mpeg_ts_plugin [Maintainer: kim-company] Membrane.Filter that demuxes MPEG-TS streams
membrane_hls_plugin [Maintainer: kim-company] Plugin providing a Membrane.HLS.Source element for HTTP Live Streaming (HLS).

Containers

Package Description Links
membrane_mp4_plugin Utilities for MP4 container parsing and serialization and elements for muxing the stream to CMAF Hex.pm Docs
membrane_matroska_plugin Matroska muxer and demuxer Hex.pm Docs
membrane_flv_plugin Muxer and demuxer elements for FLV format Hex.pm Docs
membrane_ivf_plugin Plugin for converting video stream into IVF format Hex.pm Docs
membrane_ogg_plugin Plugin for depayloading an Ogg file into an Opus stream Hex.pm Docs

Audio codecs

Package Description Links
membrane_aac_plugin AAC parser and complementary elements for AAC codec Hex.pm Docs
membrane_aac_fdk_plugin Membrane AAC decoder and encoder based on FDK library Hex.pm Docs
membrane_flac_plugin Parser for files in FLAC bitstream format Hex.pm Docs
membrane_mp3_lame_plugin Membrane MP3 encoder based on Lame Hex.pm Docs
membrane_mp3_mad_plugin Membrane MP3 decoder based on MAD. Hex.pm Docs
membrane_opus_plugin Membrane Opus encoder and decoder Hex.pm Docs
membrane_wav_plugin Plugin providing elements handling audio in WAV file format. Hex.pm Docs
membrane_g711_plugin [Maintainer: fishjam-dev] Membrane G.711 decoder, encoder and parser Hex.pm Docs

Video codecs

Package Description Links
membrane_h26x_plugin Membrane h264 and h265 parsers Hex.pm Docs
membrane_h264_ffmpeg_plugin Membrane H264 decoder and encoder based on FFmpeg and x264 Hex.pm Docs
membrane_vpx_plugin Membrane plugin for decoding and encoding VP8 and VP9 streams Hex.pm Docs
membrane_abr_transcoder_plugin ABR (adaptive bitrate) transcoder, that accepts an h.264 video and outputs multiple variants of it with different qualities. Hex.pm Docs
membrane_h265_ffmpeg_plugin [Maintainer: gBillal] Membrane H265 decoder and encoder based on FFmpeg and x265 Hex.pm Docs
elixir-turbojpeg [Maintainer: BinaryNoggin] libjpeg-turbo bindings for Elixir
membrane_subtitle_mixer_plugin [Maintainer: kim-company] Membrane.Filter that uses CEA708 to merge subtitles directly in H264 packets.

Raw audio & video

Package Description Links
membrane_generator_plugin Video and audio samples generator Hex.pm Docs

Raw audio

Package Description Links
membrane_raw_audio_parser_plugin Membrane element for parsing raw audio Hex.pm Docs
membrane_portaudio_plugin Raw audio retriever and player based on PortAudio Hex.pm Docs
membrane_audio_mix_plugin Plugin providing an element mixing raw audio frames. Hex.pm Docs
membrane_audio_filler_plugin Element for filling missing buffers in audio stream Hex.pm Docs
membrane_ffmpeg_swresample_plugin Plugin performing audio conversion, resampling and channel mixing, using SWResample module of FFmpeg library Hex.pm Docs
membrane_audiometer_plugin Elements for measuring the level of the audio stream Hex.pm Docs

Raw video

Package Description Links
membrane_raw_video_parser_plugin Membrane plugin for parsing raw video streams Hex.pm Docs
membrane_video_merger_plugin Membrane raw video cutter, merger and cut & merge bin Hex.pm Docs
membrane_live_compositor_plugin Membrane plugin for video and audio mixing/composing Hex.pm Docs
membrane_camera_capture_plugin A set of elements allowing for capturing local media such as camera or microphone Hex.pm Docs
membrane_rpicam_plugin Membrane rpicam plugin Hex.pm Docs
membrane_framerate_converter_plugin Element for converting frame rate of raw video stream Hex.pm Docs
membrane_sdl_plugin Membrane video player based on SDL Hex.pm Docs
membrane_overlay_plugin Filter for applying overlay image or text on top of video Hex.pm Docs
membrane_ffmpeg_swscale_plugin Plugin providing an element scaling raw video frames, using SWScale module of FFmpeg library. Hex.pm Docs
membrane_ffmpeg_video_filter_plugin FFmpeg-based video filters Hex.pm Docs
membrane_video_mixer_plugin [Maintainer: kim-company] Membrane.Filter that mixes a variable number of input videos into one output using ffmpeg filters Hex.pm

External APIs

Package Description Links
membrane_aws_plugin [Maintainer: fishjam-dev]
membrane_agora_plugin Membrane Sink for Agora Server Gateway Hex.pm Docs
membrane_element_gcloud_speech_to_text Membrane plugin providing speech recognition via Google Cloud Speech-to-Text API Hex.pm Docs
membrane_element_ibm_speech_to_text Membrane plugin providing speech recognition via IBM Cloud Speech-to-Text service Hex.pm Docs
membrane_s3_plugin [Maintainer: YuzuTen] Membrane framework plugin to support S3 sources/destinations Hex.pm Docs
membrane_transcription [Maintainer: lawik] Prototype transcription for Membrane

Formats

Package Description Links
membrane_rtp_format Real-time Transport Protocol format for Membrane Framework Hex.pm Docs
membrane_cmaf_format Membrane description for Common Media Application Format Hex.pm Docs
membrane_matroska_format Matroska Membrane format Hex.pm Docs
membrane_mp4_format MPEG-4 container Membrane format Hex.pm Docs
membrane_raw_audio_format Raw audio format definition for the Membrane Multimedia Framework Hex.pm Docs
membrane_raw_video_format Membrane Multimedia Framework: Raw video format definition Hex.pm Docs
membrane_aac_format Advanced Audio Codec Membrane format Hex.pm Docs
membrane_opus_format Opus audio format definition for Membrane Framework Hex.pm Docs
membrane_flac_format FLAC audio format description for Membrane Framework Hex.pm Docs
membrane_mpegaudio_format MPEG audio format definition for Membrane Framework Hex.pm Docs
membrane_h264_format Membrane Multimedia Framework: H264 video format definition Hex.pm Docs
membrane_vp8_format VP8 Membrane format Hex.pm Docs
membrane_vp9_format VP9 Membrane format Hex.pm Docs
membrane_g711_format [Maintainer: fishjam-dev] Membrane Multimedia Framework: G711 audio format definition Hex.pm Docs

Standalone media libs

Package Description Links
ex_webrtc [Maintainer: elixir-webrtc] An Elixir implementation of the W3C WebRTC API Hex.pm Docs
ex_sdp Parser and serializer for Session Description Protocol Hex.pm Docs
ex_libnice Libnice-based Interactive Connectivity Establishment (ICE) protocol support for Elixir Hex.pm Docs
ex_libsrtp Elixir bindings for libsrtp Hex.pm Docs
ex_m3u8 Elixir package for serializing and deserializing M3U8 manifests. Hex.pm Docs
membrane_rtsp RTSP client for Elixir Hex.pm Docs
membrane_ffmpeg_generator [Labs] FFmpeg video and audio generator for tests, benchmarks and demos. Hex.pm Docs

Utils

Package Description Links
unifex Tool for generating interfaces between native C code and Elixir Hex.pm Docs
bundlex Multiplatform app bundler tool for Elixir Hex.pm Docs
beamchmark Elixir tool for benchmarking EVM performance Hex.pm Docs
bunch A bunch of helper functions, intended to make life easier Hex.pm Docs
bunch_native Native part of the Bunch package Hex.pm Docs
shmex Elixir bindings for shared memory Hex.pm Docs
membrane_timestamp_queue Queue that aligns streams from multiple sources basing on timestamps Hex.pm Docs
membrane_common_c Membrane Multimedia Framework: Common C Routines Hex.pm Docs
membrane_telemetry_metrics Membrane tool for generating metrics Hex.pm Docs
membrane_opentelemetry [Labs] Utilities for using OpenTelemetry with Membrane Hex.pm Docs
membrane_precompiled_dependency_provider Provides URLs for precompiled dependencies used by Membrane plugins. Hex.pm Docs

Authors

Membrane Framework is created by Software Mansion.

Since 2012 Software Mansion is a software agency with experience in building web and mobile apps as well as complex multimedia solutions. We are Core React Native Contributors and experts in live streaming and broadcasting technologies. We can help you build your next dream product – Hire us.

Copyright 2018, Software Mansion

Software Mansion

Licensed under the Apache License, Version 2.0