FPGA Vision Toolkits

[infinitenothing]

I'm looking to process images with very low latency. I don't know if I have time to ship it off to the CPU and back to make a decision and react. I see that NI has a toolkit https://www.ni.com/en-us/support/documentation/supplemental/18/ni-vision-fpga-function-support-and-compatibility.html

It seems a bit limited though. For example, I'd like a bit more from the particle analysis toolkit. Things like perimeter and the Max Feret Diameter.

Does anyone have any suggestions? I'd rather buy than build.

[Gribo]

If you are open to non NI hardware, two kits come to mind, but good luck finding stock: Xilinx Kria and Nvidia Jetson. 

 

[infinitenothing]

I have a feeling that's not going to be fast enough. My images are relatively small maybe 60x60 x maybe 10 cameras but I need really low latency (sub millisecond). I'm planning on bringing the sensors/pixels into an ADC directly so I don't have to worry about any interfaces slowing me down. I'm hoping the processing is relatively simple (eg take a threshold, measure perimeter). My plan A is a FlexRIO (eg 5751B) or an Scope like the 5170.

Have you used the Kria or Jetson platforms? Is my application in range?

Edited March 8, 2022 by infinitenothing

[ShaunR]

I'm not sure you will find an off-the-shelf solution to this. Sub-millisec means a frame rate over 1000fps before you get to any processing.

I'm not promoting this as a solution but you may find some useful information in this little project.

Opens Source IMX219 Camera MIPI CSI-2 Receiver Verilog HDL Lattice FPGA MachXO3 Raspberry PI Camera

mipi_csi_receiver_FPGA

I'd also be interested in hearing more about your ADC solution if you can flesh it out more thoroughly.

[infinitenothing]

NI has a great start to a solution for me with the FPGA toolkit I mentioned. I was hoping there were some other options. Maybe I can drop a request in the third party toolkit developer exchange.

You can think of my ADC solution as a bit like a frame grabber. Basically, I'm going to grab the voltages off the pixel hardware one at a time. So imagine like 25MSPS and then I reshape that into 60x60 arrays so I can get 7k frames per second. Ideally, most of my processing can be done in parallel (pipelined) with the acquisition (imagine applying a threshold to each pixel as they come off the ADC).

Actually, I have the acquisition going fine right now. I can grab images. I was just complaining because the mipi interface would be something I have to implement at this point. It's not a deal breaker (unless there's an inherent high worst case latency limit on mipi and then it might be) but the interface isn't doing me any favors.

I am more focused on the processing on the FPGA—I don't want to reinvent but will do if I have to.

Edited March 8, 2022 by infinitenothing

[ShaunR]

1 hour ago, infinitenothing said:

Actually, I have the acquisition going fine right now. I can grab images. I was just complaining because the mipi interface would be something I have to implement at this point.

Then the hard work is done. The FPGA in that project was to implement the MIPI receiver. If your camera is giving you the FPS you need, then it's just post-processing you are interested in?

1 hour ago, infinitenothing said:

Ideally, most of my processing can be done in parallel (pipelined) with the acquisition (imagine applying a threshold to each pixel as they come off the ADC).

You are already in the analogue domain so why not just use comparators for that? They are cheap and have can have nanosecond propagation delays. Unless, of course, the image acquisition ADC is not as bare-bones as I am envisaging.

Where I'm going with this, though, is to do as much as possible in hardware but I can see the attractiveness of a general purpose FPGA. I would ordinarily suggest the so called "Smart Camera's" which are excellent bang-for-buck with these kinds of requirements and have a lot of the features that LabVIEW has but your sub msec requirement prohibits most of those.

[infinitenothing]

Yes, mostly just post processing.

Comparitors and other analog solutions are clever but I'm expecting a use case of needing more than one comparitor each with different user defined thresholds. I also have a use case of calculating things like summing the pixels over the whole image which I guess could be analog but an FPGA gives us maximum flexibility to change that up as the project evolves. I also want to take the binary images, perform combinational logic between pixels in the same location across different images, and then use particle analysis to pull out features (I mentioned Feret diameter above)