JavaScript video editor, encoder, and streamer - version 5.0.6

Rendering

At the lowest level, FFmpeg is used server-side through the fluent-ffmpeg library to render a Mash into files of various types. To run FFmpeg commands Movie Masher wraps an instance from this library with an instance of the Command class, that itself is wrapped by a RunningCommand instance which allows the FFmpeg process to be monitored and potentially stopped.

At the highest level, the RenderingServer receives a request to render a MashObject into one or more output files, represented by CommandOutputs. It validates the request and passes it to a new RenderingProcess instance. This creates a specific type of RenderingOutput instance for each CommandOutput provided. Each of these is responsible for converting the Mash into a corresponding RenderingDescription object. The RenderingProcess then converts each of them to a CommandDescription that it creates a RunningCommand with:

RunningCommand Command RenderingOutput RenderingProcess fluent-ffmpeg FFmpeg RenderingServer

Hence, the RenderingProcess interface greatly simplifies generating multiple output files from a MashObject, DefinitionObjects, and CommandOutputs. The underlying RunningCommand interface can be used directly to render output, though any inputs specified must be locally sourced. The base-level Command interface can be used when monitoring isn't needed, or when synchronous results are required.

Descriptions

Movie Masher supports five RenderingOutput interfaces that convert a Mash into the RenderingDescription that will render its specific output:

Each RenderingOutput will return a different RenderingDescription for the same Mash content. For instance, one returned by an AudioOutput will only describe a range of its audible content while one from an ImageOutput will only describe its visible content at a single point in time. A VideoOutput will return one describing a range of both, with the visible content potentially broken up into smaller chunks. To accommodate all these cases, the RenderingDescription can contain:

The CommandDescription object structure closely matches FFmpeg's command line options and as such can contain:

  • multiple CommandInputs, each describing a raw media source file plus related input options like start time or duration
  • a single CommandOutput object describing the output file's AV codecs, bitrates, and dimensions as well as its file extension and format plus related output options
  • multiple GraphFilters, each describing an FFmpeg filter to apply (AKA a filtergraph)

Mash Conversion

To generate an appropriate RenderingDescription, a RenderingOutput will have its Mash create a new FilterGraphs instance specifically tailored to its requirements for audible/visible content and time range. This simple interface is also used by the Composition class to cache assets and render previews within the browser. Here it's relied on it to provide:

  • a single FilterGraph instance, describing audible content
  • multiple FilterGraph instances, describing visible content

Each FilterGraph describes a section of the Mash that can be conveniently cached and rendered together, including all Clips from all relevant and populated Tracks. It provides a powerful and flexible interface from which the remaining RenderingDescription data is drawn:

  • multiple CommandInput objects, describing any input files
  • mutiple GraphFilter objects, describing any filters to apply

Each FilterGraph also contains a set of FilterChains, one per Track, from which the data above is constructed. Each FilterChain describes a Clip and any associated Merger, Scaler, and Effects as a collection of GraphFilters and GraphFiles. A FilterGraph essentially combines the GraphFilters from all FilterChains and converts a subset of its GraphFiles, as CommandInputs.

A single Clip can result in multiple GraphFiles which may or may not be converted to CommandInputs. For instance, a Theme that utilizes the DrawTextFilter will include a GraphFile for its referenced FontDefinition, but also one for its text content. Neither will be converted because FFmpeg's underlying drawtext filter expects paths to these files to be specified as option values.

File Caching

To assure that all files in a RenderingDescription are available locally, a RenderingOutput will retrieve a LoadPromise from its Mash to cache them. In some cases, it will retrieve one directly from its Preloader to load specific GraphFiles from the FilterGraphs which are required to determine output duration or dimensions.

A GraphFile ultimately describes a file on disk that is made available to FFmpeg during Command execution. Typically this is the raw asset associated with Video, Image, or Audio clips but other resources are also supported. FFmpeg requires files to be specified either as a CommandInput or GraphFilter option value.

The NodePreloader handles all the complexity of caching each GraphFile locally and correctly providing its file path to GraphFilters. Caching a GraphFile triggers different processing, depending on its type property which can be either a GraphFileType or LoadType.

In the simplest case, the type property is a GraphFileType which implies the file property will be the actual file content. This is simply saved to the cacheDirectory of the RenderingOutput as, for instance, a TEXT, SVG or PNG file. Binary files must be Base64 encoded.

In cases where the type property is a LoadType, the file property will be a relative or absolute URL. The NodePreloader instance is configured by the RenderingProcess instance to download absolute URLs, but resolve relative ones to a local directory. Typically, the RenderingServer specifies this as the user's upload directory.