Etnaviv NPU Open-Source Driver Now Twice As Fast For Image Classification Workloads

[phoronix]

Phoronix: Etnaviv NPU Open-Source Driver Now Twice As Fast For Image Classification Workloads

Tomeu Vizoso has been leading the effort for supporting Vivante's NPU IP within the Etnaviv driver that began as a reverse-engineered driver for Vivante graphics. The Vivante NPU architecture ends up being close to the graphics cores and Vizoso has been making good progress for enabling the NPUs on this open-source stack. The latest achievement is image classification workloads now running about twice as fast as previously...

https://www.phoronix.com/news/Etnaviv-NPU-2x-Fast

[LoveRPi]

Libre Computer is sponsoring this work and Alta SBC will be available in 2 weeks. This work is initially targeted for Frigate in Home Assistant but more optimizations and models will be supported.

[schmidtbag]

What Linux software actually uses stuff like this?

[timofonic]

What Linux software actually uses stuff like this?

I want to know it!

[Quackdoc]

What Linux software actually uses stuff like this?

Lots of things can use these AI accelerators that people don't realize, Noise removal, STT and TTS (something very important to me personally as I have RSI issues.) Object tracking (good for auto focus) and hundreds of more applications. Not just things like stable diffusion.

[Radtraveller]

Can you use something like this, or this, to identify an object in the image, “classify” the image, assign that classification to image identifier and find other images that have the same “object” while allowing for variations in size and or position of the object in the image?
Ie, this is a picture of a swing.
store it.
pull up all stored images classified as including a swing.
regardless of any “title” for the image in the header?
or maybe replace “swing” with picture of “grandma’s face.”?
That one I’d really like as I have hundreds of pictures with my father in images, but trying to find them all..

[Quackdoc]

Can you use something like this, or this, to identify an object in the image, “classify” the image, assign that classification to image identifier and find other images that have the same “object” while allowing for variations in size and or position of the object in the image?
Ie, this is a picture of a swing.
store it.
pull up all stored images classified as including a swing.
regardless of any “title” for the image in the header?
or maybe replace “swing” with picture of “grandma’s face.”?
That one I’d really like as I have hundreds of pictures with my father in images, but trying to find them all..

Yes! that is explicitly something that an NPU can accelerate. however it is worth noting that a lot of these chips will need specific work done into them. There are some python tools that you can run on your desktop for this, but Im not sure of any specifically

[pong]

Well that's pretty complicated.
Identify objects based on matching to some degree a category corresponding to a large amount of pre-trained labeled examples, yes, a model can do that.

Pull up stored images matching a certain classifier? Well more specifically given a single image a model could indicate the strength of matching of a classifier to that
image. One can store a database of classifiers that correspond strongly to each image, or one can go through one by one and check the classifications.

Here's some examples of various things:

https://github.com/facebookresearch/segment-anything

https://github.com/openai/CLIP

https://github.com/mlfoundations/open_clip

Replace "swing" with a particular face image? Well that's not specifically something I know of pre-existing models that could do it as an atomic operation
but basically I think if you break it down to say "segment the image and identify where there may be a swing" wrt. pixels / bounding box of sub-pixels, then
given a polygonal area, inpaint (inpainting is a good keyword for further research) a given image into that area (scale, whatever), then there's sort of a nebulous possibility to "meld" this big chunk of the image that just got replaced with an inpainted "stamp" of another image and "heal" the discontinuities by creating some kind of synthetic image that "blends" the two disjoint images together so that it looks more "natural" according to either the source image or some description prompt guiding what the blend of the composite image should possibly be like -- but that's out of the realm of operating on the fidelity of the source images and is more in the land of similarity synthesis so the results can be varied and not all necessarily "plausible" or "artistic".

https://comfyanonymous.github.io/ComfyUI_examples/inpaint/

https://github.com/Stability-AI/stablediffusion#image-inpainting-with-stable-diffusion

You can kind of see some guided image synthesis and transformation things to give an idea of some workflows / models that do such:

https://comfyanonymous.github.io/ComfyUI_examples/img2img/

https://comfyanonymous.github.io/ComfyUI_examples/area_composition/

https://comfyanonymous.github.io/ComfyUI_examples/unclip/

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features

https://github.com/NVIDIA/pix2pixHD

https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/

https://github.com/runwayml/stable-diffusion#inpainting-with-stable-diffusion

https://github.com/TencentARC/VQFR

https://github.com/joelmoniz/DepthNets

https://github.com/Mikubill/sd-webui-controlnet

https://github.com/Stability-AI/stablediffusion

I'm pretty sure there are a bunch of face detection / replacement specific flows / models,
and also several "image healing", "object removal", "object insertion", "inpainting" ones so various different approaches / objectices.

Detecting faces in general is very broadly / deeply addressed so that's easy to find where faces may be in pictures.
Detecting a particular person's face would I think involve training a model to recognize that particular person's face given dozens or hundreds or more of known examples then being able to say "grandma" with some good probability.
Or another approach is to detect any image with faces in it then run a facial feature similarity model on that image vs. characteristics extracted from just one or a few good pictures of grandma and then it might be able to say 80% or whatever similar match and flag those images.

Can you use something like this, or this, to identify an object in the image, “classify” the image, assign that classification to image identifier and find other images that have the same “object” while allowing for variations in size and or position of the object in the image?
Ie, this is a picture of a swing.
store it.
pull up all stored images classified as including a swing.
regardless of any “title” for the image in the header?
or maybe replace “swing” with picture of “grandma’s face.”?
That one I’d really like as I have hundreds of pictures with my father in images, but trying to find them all..