Another try was to later re-interpret the data with numpy, but this leads to an image which seems not to contain the right width and height:

 v2: np.ndarray = frames["v:2"]
 data = v2.reshape(-1).view(np.uint16).reshape([-1, 576, 640, 1])  # 640 doesn't work, it has to be 960

@cansik -

First of all, here is my test code

import ffmpegio as ff

input_path = "sandbox/test_cansik.mkv"
streams = ["v:0", "v:1", "v:2"]

sinfo = ff.probe.video_streams_basic(input_path)

# print(ff.media.read(input_path, t=1, map=streams))
with ff.open(str(input_path), "rvv", map=streams, blocksize=1) as fin:
    F = fin.read()

    print(list(F.keys()))
    for i, s in enumerate(streams):
        print(s, F[s].shape, (1, sinfo[i]["width"], sinfo[i]["height"], 3))
        print(s, F[s].dtype, sinfo[i]["pix_fmt"])

this prints

['v:0', 'v:1', 'v:2']
v:0 (1, 1080, 1920, 3) (1, 1920, 1080, 3)
v:0 uint8 yuvj422p
v:1 (1, 576, 640, 3) (1, 640, 576, 3)
v:1 uint8 gray16be
v:2 (1, 576, 640, 3) (1, 640, 576, 3)
v:2 uint8 gray16be

and your issue is that the output frame dtype is not correct for the v:1 and v:2. Am I correct?

There is a way but with a caveat that there is a bug that needs to be fixed first (or rather current incompatibility with FFmpeg). The nature of this generic media reader beast is that the output video format defaults to RGB24. In other words, it appends -pix_fmt rgb24 when ffmpegio calls FFmpeg.

In FFmpeg, what you'd do to override this is to specify per output stream: -pix_fmt:1 gray16le and -pix_fmt:2 gray16le (using the little endian format which is native to x86's). This translates to ffmpegio as

pix_fmts = {"pix_fmt:v:0": "rgb24", "pix_fmt:v:1": "gray16le", "pix_fmt:v:2": "gray16le"}
with ff.open(str(input_path), "rvv", map=streams, blocksize=1, **pix_fmts) as fin:
    ...

However, the said bug currently appends -pix_fmt rgb24 to the end of the FFmpeg command line, overriding per-stream specification because of how ffmpegio orders its FFmpeg option arguments... Anyway, a long story short, I need to fix this bug for you.

Meanwhile, if you'd like to hack your way through, look into Lib/site-packages/ffmpegio_core/configure.py and find Line 623-4 (in finalize_media_read_opts()):

    if "pix_fmt" not in options:
        options["pix_fmt"] = "rgb24"

You can comment out these 2 lines while providing the above pix_fmts keyword arguments, you should be able to get it to work. Meanwhile, I'll try to push the fix out over the weekend.

Oh wow, thanks for your very detailed analysis of the problem, and yes, that is exactly the issue I encounter. Looking forward to a fix 🙌🏻

If you don't find the time, maybe I could try working on a PR. But I think it would take way longer, since I am not into the codebase.

@tikuma-lsuhsc As a workaround, I found out that it is possible to provide a default pix_fmt, as well as the channel specific ones, which seems to work as well (instead of commenting out the check):

with ffmpegio.open(str(input_path), "rvv", map=streams, blocksize=1, pix_fmt="rgb24", **pix_fmts) as fin:
    ...

Nice find! And that's interesting. I guess dict is aware of the orders keys are assigned. Good news. That'll make our life a bit easier :)

Just FYI, I created this generic multi-stream reader just out of curiosity without any specific use case for myself (for now). If you encounter any issues, feel free to post those issues on Issues board and I'll try to address'em as quickly as possible.

Yes, stream_loop_in=-1 seems to work as expected. This is already a good workaround for me, because this prevents the iterator to be invalid.

Regarding the issue, in the example MKV I use for testing, there are 450 frames in the RGB stream (when using readall() on the AviMediaReader object). The loop always stops at 199, which seems to me to be an arbitrary number after the stream is closed and the iterator is no longer valid. The sleep just makes it happen sooner.

I will read a bit more about the blocksize. 1 seems to be an odd number, but it was necessary to set it because for me it controls how many frame sets (n-frames per stream) are returned. Here for example blocksize=2:

The loop always stops at 199

If it's always 199, it's not arbitrary, no doubt. I'll look at your test vid tonight, but are the framerates different on the latter 2 streams? I see you only got 1 frame from them while getting 3 on the first. It could be a bug exposed by the heterogeneous framerates. (something I never tested)

it was necessary to set it because for me controls how many frame sets (n-frames per stream)

OK. Your description above sounded like you changed to 1 to make the loop work.

The test vid will be a great help. I'll keep you posted.

The frame rate is 30/1 on all streams, but it is possible for the streams to be out of sync because they come from different sensors. There are 452 frames for the infrared and depth streams (the two monocular channels). Maybe it's just a really bad example, because usually it makes sense to synchronise them anyway before writing them into a recording file. If you need any further information, just leave a comment.

Oh, doh, I misread the array dimension. Yes, they all have 2 frames each... This should be enough information to debug the issue. Thanks for the report!

Just release v0.10.0 with the fix. It also has an enhancement which you may want to take advantage of:

You can now check the validity (not EOF) of the ffmpeg stream with its __bool__ type caster:

import time

import ffmpegio as ff

input_path = "test.mkv"
streams = ["v:0", "v:1", "v:2"]

i = 0
with ff.open(str(input_path), "rvv", map=streams, blocksize=1) as fin:
    while fin:
        F = fin.read()
        print(f"{i} = {list(F.keys())}")
        time.sleep(0.1)

        i += 1

Also, I fixed the bug in __next__ so you can also do this:

with ff.open(str(input_path), "rvv", map=streams, blocksize=1) as fin:
    for F in fin:
        print(f"{i} = {list(F.keys())}")
        time.sleep(0.1)

        i += 1

and it'll terminate properly.

@tikuma-lsuhsc Thanks, good point about the licence. I must have overlooked it as my template uses the MIT licence by default. I have updated it and will release a new version. Just out of curiosity, what is the reason for releasing ffmpegio under GPL-v2 and not LGPL or something less permissive?

The BGR2RGB conversion is needed, maybe it's because the Kinect stores the frames in bgr24. But I will have a look at it later, thanks for mentioning it.

A test video can be found here: https://github.com/microsoft/Azure-Kinect-Samples/blob/master/body-tracking-samples/sample_recordings/test.mkv

The BGR2RGB conversion is needed, maybe it's because the Kinect stores the frames in bgr24.

I may know why that's happening (though I don't know how I overlooked it) and if so it is an bug in ffmpegio. Let me check on it.

Nope, it (RGB) is working as intended. Here is what the first frame looks

This is plotted by

plt.imshow(F['v:0'][0,...])

So F['v:0'][0,:,:,0] channel must be red.

I think matplotlib works with RGB and opencv by default uses BGR. I can't test it right now, but I think this could be the reason why I do the swap.

If that's the case, maybe cv2.COLOR_RGB2BGR is the right macro to use? Furthermore, you can pass the keyword argument **{'pix_fmt:v:0': 'bgr24'} to FFmpeg to avoid the need of the conversion.