Image source from the-scientist.com article.

My Discord bot is keep evolving as some bright ideas penetrate my head in the brain dead time around 2 A.M. It told me...

"Are you not tired of those shady 'youtube to mp3 downloader' website? Why not just make your own lmao".

And so I did.

The idea is simple and there are libraries for it. Simply just

Parse the youtube link
Download the video
Convert the video to audio
Save the audio file and serve.
???
Profit

I managed to implement my own using ytdl-core and fluent-ffmpeg library. But there is a problem. A big problem actually. Let's go back to the steps I've listed before (minus the last 2 one).

Parse the youtube link : Need some validation. But ytdl-core library already have function to tackle it.
Download the video : You're going to write a video file into your file system. This is slow due to I/O operations.
Convert the video to audio : You're going to use ffmpeg that's installed in your system to convert the video file to audio file. This is VERY slow due to using external program and the encoding process that's slow by nature.
Save the audio file and serve. Actually not a big deal. Just remember to delete after expiration time.

So, the naive approach going to cost you twice the process that takes a lot of time. Not mentioning the need to write the video file that costs a lot of megabytes and I/O cycles. The side effect if you're running on local and using SSD is the consumed write bytes quota. Surely you mostly don't need to worry about this if you're using cloud services.

But the main problem is, we need to wait for a process to finish then do the next. This is because we need a complete data in order to process the next.

Or is it?

Introducing, streams...

Streams

You've probably heard this many times before. Data streams. Streams of data. Or whatever it may be called. The characteristics for this is you're dealing with byte sequences which mostly consist of raw data. But it's consistent as it flows continuously, hence the name.

The core of almost every single I/O operations will at least involve streams. Even if it's transported in blocks, the mechanism behind will still involve streams in the low-level sense. And you've probably been using streams without realizing it. Let's see several types of streams in Node.js. I'll just skim through this.

Readable

Readable streams, like the name, is a type of stream that can only read data. Mostly used for reading a file, reading data, reading stuffs, read, read, and read. You can make one by using fs library.

const readStream = fs.createReadStream('path to file to read')

Use this to read data from files.

Writable

Like readable stream, but it's writable (duh). Commonly used to write into new files. The declaration just like the above, but with createWriteStream as the function name. If the file doesn't exist, then it will create new file with the path given.

Duplex

It has both the characteristics of Readable and Writable stream. The difference is it uses different buffer for each modes. It will have one buffer for reading operations, and another buffer for writing operations. Commonly used for communication between two actors, like in TCP connection.

Transform

It's like duplex stream, but the input and the output is somewhat related. You can think it as a stream used to modify or transform data as it flows, hence the name. Commonly used for compression or data transformation. Gzip compression uses this, and we are going to use this for our problem.

That was a pretty short and simple explanation of streams, but if you want to read in depth explanation, you can consult the official documentation or this article

Now, let's talk about the other terminology we're going to use

Pipes

Pipe is the medium the data stream going to flow. Just like the water pipes in real world. But the way it used in computer programming is to actually connect between two streams. Think the stream objects as the endpoints, like the faucet or the-opposite-of-faucets-that-sucks-water. And pipes are the mechanism to connect between these two endpoints.

The mechanism to connect between two streams is called piping. This is also the base of the term pipelining, as you're basically "piping" the output of a process as the input for the next process. You can do so many things with piping, especially when connecting two operations.

In Node.js, any streams will have pipe() function attached to it, that accepts a stream object as the argument. Doing this will stream the data from the source stream into the destination stream, with the exception you can't pipe into Readable stream, which is why Writable stream doesn't have pipe() function.

Duplex and Transform, which implements both, can do this.

Lots of ways to use pipes, and you can start learning it from this article. (it's not that I don't want to write more stuffs, it just feels very redundant to re-write something you've "took" from other source, so I'll just give link to the said article to further increase their viewcount").

Now, lets do this with our app.

ytdl-core and fluent-ffmpeg

As I've mentioned before, we're going to download a video using ytdl-core, then use fluent-ffmpeg to convert it into mp3. fluent-ffmpeg is just a wrapper for creating commands to ffmpeg installed in your system. There is already npm package that implements fully functional ffmpeg inside Node.js (made using WebAssembly thus also browser compatible, WOW!), but it's still not stable the last time I've heard, so we're going to use the conventional method.

First, download the video file. I'm using ytdl-core with audio only setting to exclude the video. It still make quite a big file though.

import ytdl from 'ytdl-core';

const video = ytdl('https://www.youtube.com/watch?v=ft-HOsTzQQ4', {
  filter: 'audioonly',
})
.on('end', () => {
  console.log('ytdl finished fetching file');
});

I've attached 'end' event handler to notify when my download finishes. It's optional.

Next, I'm setting up ffmpeg command. Remember, this is just a command wrapper, and will not do the job until you explicitly tell it by invoking run() method. I declared a Writeable stream to tell where the output file to be

import ffmpeg from 'fluent-ffmpeg';
import fs from 'fs';

const ffmpegPipe = fs.createWriteStream('pipetest.mp3');

const ffmpegCommand = ffmpeg().format('mp3')
  .audioCodec('libmp3lame')
  .output(ffmpegPipe)

Now, let's connect these two. I've said before that this kind of operation, which involves data modifying inbetween, is better suited for Transform stream. There is a well suited stream which is another version of Transform stream called PassThrough stream. This stream simple passes the input to the output. Now let's make it and use it with our two operations.

import stream from 'stream';

const passStream = new stream.PassThrough();

video.pipe(passStream);
ffmpegCommand.input(passStream).run();

The end code should looks like this.

import ytdl from 'ytdl-core';
import ffmpeg from 'fluent-ffmpeg';
import stream from 'stream';
import fs from 'fs';

const ffmpegPipe = fs.createWriteStream('pipetest.mp3');
const passStream = new stream.PassThrough();

const video = ytdl('https://www.youtube.com/watch?v=ft-HOsTzQQ4', {
  filter: 'audioonly',
})
.on('end', () => {
  console.log('ytdl finished fetching file');
});

const ffmpegCommand = ffmpeg().format('mp3')
  .audioCodec('libmp3lame')
  .output(ffmpegPipe);

video.pipe(passStream);
ffmpegCommand.input(passStream).run();

By this, while you're downloading your video, the bytes downloaded will be instantly passed into ffmpeg for encoding, saving you time to wait until the whole video being written into the file system.

This definitely saves you time for I/O operation, file management like deleting the unused video file after conversion, and definitely saves time. And that's only the slight glimpse of what can you do with streams.

Another usage is to use/serve big files, like 2GB or 4GB. Traditional method without streams will going to load this entire file into your memory, losing time since it takes time to load gigabytes of file (even on SSD), and definitely will eat your RAM along the way. Streams will save the day for this workload.

And I think that's it. It's been a nice exploration as I finally manage to actually understand and use the streams defined in Node.js. I believe this knowledge is also applicable in other programming languages, so it's being a nice thing to know.

I think that's all, see ya!

Log-0x16 - Node.js PassThrough Stream with ytdl-core and ffmpeg