Create some benchmarks

[timholy]

I think it may be getting close to time to promote JuliaImages a bit more heavily. I still have a few things I want to do first wrt cleaning the code base (getting a release of Images.jl out that is compatible with ImageCore 0.9 is high on the list, as is the abs2 issue). But it will take some time to assemble good benchmarks, and I think it's best to do that out in the open so that others can pitch in. @mkitti, you'll note from the README that I'd love to have Fiji benchmarks, is that by any chance something you'd be able and interested to contribute?

FWIW, here's where we are on my machine today:

Note the log-scaling. The "generic" tests (top) are expressed per-voxel, the "specialized" tests (below) are absolute.

Together with OpenCV, we pretty much dominate already (in one case by more than three orders of magnitude), but there are a few exceptions that merit investigation. (That Gaussian blur algorithm in OpenCV is worth emulating...) Missing points generally represent unsupported operations; the "specialized" for OpenCV is still a WIP but they support very little 3d so a lot of points will be missing anyway.

This also doesn't capture some of our other advantages, like scaling to big data. I've been playing a bit with dask but it seems like a pretty far cry from what we offer. scikit-image has a habit of immediately converting everything to a numpy array.

[timholy]

Time for a fun update. This does rely on some stuff that either hasn't been released yet or is still local on my machine:

and one that now combines both kinds of benchmarking:

OpenCV's performance advantage has, on average, essentially been erased, and I believe there is more to come.

[Tokazama]

I'm assuming things are going to get better as LoopVectorization becomes easier to incorporate?

[timholy]

Yep. The RGB filtering benchmarks (blur and gradient, aka "big kernels" and "small kernels" respectively) still aren't using LoopVectorization. It looks like OpenCV struggles a bit there too, and I think it's possible that a really good implementation may ultimately beat them almost across the board. Even the grayscale ones could use more work, as my thread utilization is hovering at around 67%, but I haven't taken a close look. But beating Matlab & skimage by an order of magnitude on a lot of benchmarks is already pretty satisfying.

[timholy]

Really want ImageJ benchmarks too, as that's kind of the standard in much of biology.

[Tokazama]

Would be nice to have ITK benchmarks too, but that's pretty specialized and the last time I used ITK I nearly ripped out my hair.

[timholy]

I checked a decade or so ago when I was still using Matlab and looking to use something faster, and to my surprise Matlab blew it away on most things I tried. That said, it has been a decade...I agree, ITK would be quite nice if at all possible.

[mkitti]

I am looking into the ImgLib2 benchmarks (ImageJ2 / FIJI). Another technology to benchmark against is the OpenCL based CLIJ2 which is available as an ImageJ plugin as well as for other platforms (Python, MATLAB) . For ITK, perhaps SimpleITK https://simpleitk.org/ might be appropriate.