diff --git a/doc/filter_design.txt b/doc/filter_design.txt index 9e3eb3df6c..6faa1b30f1 100644 --- a/doc/filter_design.txt +++ b/doc/filter_design.txt @@ -33,7 +33,160 @@ Format negotiation Buffer references ownership and permissions =========================================== - TODO + Principle + --------- + + Audio and video data are voluminous; the buffer and buffer reference + mechanism is intended to avoid, as much as possible, expensive copies of + that data while still allowing the filters to produce correct results. + + The data is stored in buffers represented by AVFilterBuffer structures. + They must not be accessed directly, but through references stored in + AVFilterBufferRef structures. Several references can point to the + same buffer; the buffer is automatically deallocated once all + corresponding references have been destroyed. + + The characteristics of the data (resolution, sample rate, etc.) are + stored in the reference; different references for the same buffer can + show different characteristics. In particular, a video reference can + point to only a part of a video buffer. + + A reference is usually obtained as input to the start_frame or + filter_samples method or requested using the ff_get_video_buffer or + ff_get_audio_buffer functions. A new reference on an existing buffer can + be created with the avfilter_ref_buffer. A reference is destroyed using + the avfilter_unref_bufferp function. + + Reference ownership + ------------------- + + At any time, a reference “belongs” to a particular piece of code, + usually a filter. With a few caveats that will be explained below, only + that piece of code is allowed to access it. It is also responsible for + destroying it, although this is sometimes done automatically (see the + section on link reference fields). + + Here are the (fairly obvious) rules for reference ownership: + + * A reference received by the start_frame or filter_samples method + belong to the corresponding filter. + + Special exception: for video references: the reference may be used + internally for automatic copying and must not be destroyed before + end_frame; it can be given away to ff_start_frame. + + * A reference passed to ff_start_frame or ff_filter_samples is given + away and must no longer be used. + + * A reference created with avfilter_ref_buffer belongs to the code that + created it. + + * A reference obtained with ff_get_video_buffer of ff_get_audio_buffer + belongs to the code that requested it. + + * A reference given as return value by the get_video_buffer or + get_audio_buffer method is given away and must no longer be used. + + Link reference fields + --------------------- + + The AVFilterLink structure has a few AVFilterBufferRef fields. Here are + the rules to handle them: + + * cur_buf is set before the start_frame and filter_samples methods to + the same reference given as argument to the methods and belongs to the + destination filter of the link. If it has not been cleared after + end_frame or filter_samples, libavfilter will automatically destroy + the reference; therefore, any filter that needs to keep the reference + for longer must set cur_buf to NULL. + + * out_buf belongs to the source filter of the link and can be used to + store a reference to the buffer that has been sent to the destination. + If it is not NULL after end_frame or filter_samples, libavfilter will + automatically destroy the reference. + + If a video input pad does not have a start_frame method, the default + method will request a buffer on the first output of the filter, store + the reference in out_buf and push a second reference to the output. + + * src_buf, cur_buf_copy and partial_buf are used by libavfilter + internally and must not be accessed by filters. + + Reference permissions + --------------------- + + The AVFilterBufferRef structure has a perms field that describes what + the code that owns the reference is allowed to do to the buffer data. + Different references for the same buffer can have different permissions. + + For video filters, the permissions only apply to the parts of the buffer + that have already been covered by the draw_slice method. + + The value is a binary OR of the following constants: + + * AV_PERM_READ: the owner can read the buffer data; this is essentially + always true and is there for self-documentation. + + * AV_PERM_WRITE: the owner can modify the buffer data. + + * AV_PERM_PRESERVE: the owner can rely on the fact that the buffer data + will not be modified by previous filters. + + * AV_PERM_REUSE: the owner can output the buffer several times, without + modifying the data in between. + + * AV_PERM_REUSE2: the owner can output the buffer several times and + modify the data in between (useless without the WRITE permissions). + + * AV_PERM_ALIGN: the owner can access the data using fast operations + that require data alignment. + + The READ, WRITE and PRESERVE permissions are about sharing the same + buffer between several filters to avoid expensive copies without them + doing conflicting changes on the data. + + The REUSE and REUSE2 permissions are about special memory for direct + rendering. For example a buffer directly allocated in video memory must + not modified once it is displayed on screen, or it will cause tearing; + it will therefore not have the REUSE2 permission. + + The ALIGN permission is about extracting part of the buffer, for + copy-less padding or cropping for example. + + + References received on input pads are guaranteed to have all the + permissions stated in the min_perms field and none of the permissions + stated in the rej_perms. + + References obtained by ff_get_video_buffer and ff_get_video_buffer are + guaranteed to have at least all the permissions requested as argument. + + References created by avfilter_ref_buffer have the same permissions as + the original reference minus the ones explicitly masked; the mask is + usually ~0 to keep the same permissions. + + Filters should remove permissions on reference they give to output + whenever necessary. It can be automatically done by setting the + rej_perms field on the output pad. + + Here are a few guidelines corresponding to common situations: + + * Filters that modify and forward their frame (like drawtext) need the + WRITE permission. + + * Filters that read their input to produce a new frame on output (like + scale) need the READ permission on input and and must request a buffer + with the WRITE permission. + + * Filters that intend to keep a reference after the filtering process + is finished (after end_frame or filter_samples returns) must have the + PRESERVE permission on it and remove the WRITE permission if they + create a new reference to give it away. + + * Filters that intend to modify a reference they have kept after the end + of the filtering process need the REUSE2 permission and must remove + the PRESERVE permission if they create a new reference to give it + away. Frame scheduling