Libcamera Is Becoming An Increasingly Viable Open-Source Camera Support Implementation

[phoronix]

Phoronix: Libcamera Is Becoming An Increasingly Viable Open-Source Camera Support Implementation

With more cameras moving their image processing operations from micro-controllers to the CPU to save on manufacturing, libcamera was started last year to serve as an open-source camera support library across Linux / Android / ChromeOS for supporting these modern cameras...

http://www.phoronix.com/scan.php?page=news_item&px=Libcamera-ELC-E-2019

[uid313]

But can it do...?

• Autofocus
• HDR
• Wide-angle lens
• Bokeh
• 4K
• 8K
• HFR (60 fps)
• Slow motion (1000 fps)
• Capture to RAW format
• Capture video to AV1 format
• Capture video to H.265 format
• Capture image to JPEG 2000
• Capture image to JPEG XR
• Capture image to HEIF
• Optical image stabilization
• Digital image stabilization
• Augmented reality (AR)
• Artificial intelligence (AI) and machine learning (ML)

[kpedersen]

But can it do...?

If it could, then libcamera would be an incorrect name.

But of course it provides a raw pixel buffer (like OpenCV) that *can* enable smart people to do those things.

[schmidtbag]

Depending how this performs with MJPEG streams, this could've been really useful to me a year ago. Too late now, though.

But can it do...?

• Autofocus
• Wide-angle lens
• Bokeh
• 4K
• 8K
• HFR (60 fps)
• Slow motion (1000 fps)
• Capture to RAW format
• Optical image stabilization

Those are up to the hardware capabilities, not necessarily the library. But, assuming the camera is capable of such things, I would hope the library can tap into them.
Though honestly, I don't know of many USB cameras that support such things.

• HDR
• Capture video to AV1 format
• Capture video to H.265 format
• Capture image to JPEG 2000
• Capture image to JPEG XR
• Capture image to HEIF
• Augmented reality (AR)
• Artificial intelligence (AI) and machine learning (ML)
• Digital image stabilization

libcamera shouldn't be responsible for any of that. Those are up to 3rd party libraries. Digital image stabilization could also be done by the camera itself, where applicable.

To my understanding, the point of libcamera is to just simply talk to the camera (which also implies fetching frames from it), and is independent of V4L2.

[Ardje]

But can it do...?

• Capture video to AV1 format
• Capture video to H.265 format
• Capture image to JPEG 2000
• Capture image to JPEG XR
• Capture image to HEIF

I assume at least those belong to libcamera as you can see from the diagram, as it interfaces with v4l2 and encoding is typically done with a v4l2 m2m device.
Although that can be done already with gst. Actually when I look at it better, it is done using gst, with the libcamera plugin controlling the camera.

[Mathias]

[...]
Those are up to the hardware capabilities, not necessarily the library.

I think Autofocus should somehow be supported. I'd like to be able to specify the region of interest. Or even manually setting the focus depth.
Also I doubt any serious 1000fps camera could simply stream the data with libcamera or similar. That usually only works in a discontinuous way which would need support by libcamera - if that is in the scope of libcamera.
Also the camera may have a motion sensor build in, which would allow easier digital image stabilization. That would again need some support by libcamera. If it was a separate data stream handled by a different library it would be hard to get correct timing information.

Apart from those points, I agree with everything you said. (including that "could have been useful to me")

[schmidtbag]

I think Autofocus should somehow be supported. I'd like to be able to specify the region of interest. Or even manually setting the focus depth.

Yes, I definitely agree with this. But the camera's ability to autofocus is up to the hardware itself, not the library.

Also I doubt any serious 1000fps camera could simply stream the data with libcamera or similar. That usually only works in a discontinuous way which would need support by libcamera - if that is in the scope of libcamera.

libcamera isn't based on v4l2 so it's very possible it could do so given the other protocols are correct. The tricky part really comes down to where you're going to find a 1000fps camera that can attach directly to a PC.

Also the camera may have a motion sensor build in, which would allow easier digital image stabilization. That would again need some support by libcamera. If it was a separate data stream handled by a different library it would be hard to get correct timing information.

Digital image stabilization is nothing more than just dynamically cropping the video to minimize shaking. You don't even need a motion sensor to do it (though, that does help). If the camera itself supports the feature, then I think it's reasonable that libcamera would be totally unaware of it when receiving the actual video signal.
If the camera doesn't support image stabilization, it's up to libraries (or gimbals or whatever) like OpenCV to do all the work; libcamera itself shouldn't be responsible for doing the actual image stabilization, but, it should be the thing that can trigger it on or off if the feature is available.

[cybertraveler]

It's my understanding that many camera drivers live in Linux and are exposed by v4l[2]. If that's the case, then surely it's a bit messy also having, effectively, camera drivers in libcamera also.

Not complaining, just it seems a little odd to me. Maybe I'm missing something.

[fthiery]

A full talk about libcamera was made available at the last gstreamer conference

https://gstconf.ubicast.tv/videos/the-first-stable-libcamera-release-a-call-for-public-api-review