Timeline stretching implementation of audio scrubbing

[timkrief]

Hi,

Right now, consumers like SDL2 or RtAudio support turning audio scrubbing on or off, but they don't offer a way to choose the type of audio scrubbing we want to use.

From my experience, the type of audio scrubbing currently used is Cut-and-paste audio scrubbing. This type of audio scrubbing is producing really annoying audio artifacts when trying to listen a media at a speed different than 1, for instance 2 times playback speed. This makes anything but short listening sessions unbearable. This use case (playing multimedia content at speed higher than one for long periods) is frequent in video editing and is used more and more in video consumption nowadays.

It is a wide spread feature in modern video players, a seemingly ubiquitous one. So much so that users can't spot that the feature is missing and just think that there is a bug. People are used to turn playback speed to 2 and have clean audio playback on services like youtube, podcast players, video editors etc.

I might have missed something, and there could be a way to have timeline stretching audio scrubbing using MLT right now, but I didn't find anything.

It's a much needed feature for a modern multimedia processing framework, and I think it should be considered as a priority since its addition could benefit MLT and its users greatly.

Thanks and have a nice day!

[timkrief]

Hi, @ddennedy what does the "invalid" label means? Do I need to add more information or details?

[ddennedy]

This is a feature request and not a bug report. See
https://github.com/mltframework/mlt/blob/master/.github/ISSUE_TEMPLATE.md

GitHub broke their own feature by no longer showing that by default.

[timkrief]

Oh ok, sorry for not seeing this I'm used to issue board being used for feature request. And thanks for converting this into a discussion.

[timkrief]

(Could we remove the invalid label from the discussion?)

[timkrief]

Looking at current implementation of audio scrubbing it seems like it is just getting the audio at current frame and doing no modification. I know that MLT is capable of time stretching with pitch compensation. Would there be a way to leverage this capability for playback?

[timkrief]

So far I found how to easily test audio scrubbing using melt. You can use melt -consumer sdl2 a_file.mkv for instance and then press 7 on the keyboard to have the playback go at 2 times the speed using audio scrubbing. I had a hard time finding this piece of information, had to dig through the code, melt.c etc, so I write this here so that other people could find it more easily.

EDIT: I had to look through code because I did not understand that the command line prompt of melt are shortcuts for playback speed options.

EDIT2: Corrected the syntax

[ddennedy]

melt displays this at the command line by default.

+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
|1=-10| |2= -5| |3= -2| |4= -1| |5=  0| |6=  1| |7=  2| |8=  5| |9= 10|
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
+---------------------------------------------------------------------+
|               H = back 1 minute,  L = forward 1 minute              |
|                 h = previous frame,  l = next frame                 |
|           g = start of clip, j = next clip, k = previous clip       |
|                0 = restart, q = quit, space = play                  |
+---------------------------------------------------------------------+

Also, the sdl2 consumer is implied and -consumer:sdl2 is the wrong syntax. It should be -consumer sdl2. It sounds whiny when you write "had to dig through the code" when things are already documented and shown, but you clearly did not read them. But it is also fine to read the source code.

[timkrief]

Oh I'm so sorry I did not know that those were playback speed shortcuts

[bmatherly]

Would there be a way to leverage this capability for playback?

Here is how someone could implement Resampling or Timeline Stretching using some code that already exists:

1. Wrap the producer in an mlt_chain
2. Attach the time_remap link to the chain
3. Modify the sdl_consumer to be able to pass scrubbing commands to the time_remap link (perhaps by adding API functions or by attaching properties to frames)
4. Modify the time_remap link to detect the scrubbing commands and recalculate the time remapping parameters

Maybe the modified time_remap link could be encapsulated in a special link called "link_scrub".

I do not volunteer to work on this since I do not see it as a particular priority. I do think it would be good to modify the existing copy-paste scrubbing method so that it can go slower than real time. That would involve changing the speed parameter from an integer to floating point.

[ddennedy]

Krita implemented something like this using a custom producer that it uses to load all audio files in #961. It was closed because I did not like an app-specific producer, and I explained how an app can register a custom service at runtime. The custom producer basically accepts a speed parameter and adjusts the frame's frequency. Here is its current implementation:
https://github.com/KDE/krita/blob/master/libs/ui/KisMLTProducerKrita.cpp

Now, that creates a user experience where the pitch changes with the speed, which is sometimes desirable but not always. I suspect that is about 50/50. One can attach a rubberband filter to compensate for the pitch.

[luzpaz]

Can this discussion be transferred to the issue queue as a feature request ?

[bmatherly]

No. Issues are only for bugs. This discussion was converted from an issue to a discussion for that reason.

[luzpaz]

Thanks for the clarrification. Do users or devs know to look for feature requests in the discussion queue ?

[bmatherly]

I don't know. I guess, like for any other open source project, they would need to search around a little. There aren't a lot of places to look.

[luzpaz]

Are you open to labeling discussions like this 'feature request' or 'enhancement' ? There is the option to do that.

[ddennedy]

This is in the Ideas category. That is sufficient.