Cocoa Application with custom Core Image filter 3: Testing the kernel with QuartzComposer

In the last post I wrote a convolution kernel for Core Image. But these kind of things need to be tested. Now, we could package the filter inside an Image Unit (which I'll do in a future post) but that would mean we would have to recompile and install the filter each time we wanted to make a change. It would be much better if there was an interactive environment that could be used to test any changes and to allow us to debug the filter code.

Luckily, Apple has provided just such an environment in QuartzComposer. If you are not familiar, Quartz Composer is an amazing application that provides a visual programming environment that allows the creation of all kinds of visualizations. If you've seen the RSS screensaver. then you've seen a QuartzComposer composition. If you've seen a preview of Time Machine, the swirling galaxy in the background is a QuartzComposer composition. I'm not going to explore this amazing piece of software in depth here, I suggest you look at some of these websites for more information: Quartz Composer Journal or www.quartzcompositions.com or boinx or Quartonian.

As a basic introduction, a composition is defined by dragging patches onto the workspace and connecting them together graphically. Patches contain ports which represent parameters which are passed between the patches. The results are displayed in real time in the viewer window, so you get immediate feedback as you change things. As we'll see later, Quartz composer compositions can be embedded in a QCView in your Cocoa apps, and controlled via Cocoa Bindings.

I've provided a link to my test.qtz composition which consists of four patches. The convolution patch is built by dragging a core image kernel patch onto the editor and then copying the text of the kernel into the patch. You can use the inspector to change the input parameters to adjust the coefficients and test out the composition.


This is the editor window, showing the composition.


Here's the inspector for the Convolution patch showing where you put the kernel code.


This is the viewer, showing the input image filtered by the edge detection filter defined in the Input Parameters to the Convolution filter.


If you have an iSight camera or other video source, replace the Image Importer with a video input, and you will see the convolution filter applied to your video stream in real time. That's just extremely cool.

Here's the editor window with a Video Input instead or the Image Importer.
And here's a view of my living room with edge detection. Notice the frame rate of nearly 60 frames per second on my Mac Book Pro.

Using this basic composition, with some modifications, you should be able to test out any core image kernel you can come up with. Next time, I'll build this filter into an image unit that can be used from any application that uses Core Image filters.

Cocoa application with custom Core Image filter 1: What is image convolution?

Before I delve into development of my Core Image kernel, I think I would like to give a quick description of image convolution. There are plenty of resources on the internet that already describe the mathematics of this procedure in great detail, such as Image Processing Fundamentals - Convolution-based Operations. A Google search will reveal a wealth of information.

While it's easy to get lost in the mathematics, performing an image convolution filter is a relatively simple operation. Basically you apply a multiplier to a matrix of pixels surrounding each destination pixel and add them all together. The result of this operation is the new value of the pixel at the center. For the purposes of these articles, we will be sticking with a simple 3x3 matrix, but there is no reason why you can't perform this type of filtering with larger matrices.

Here are some examples of 3x3 convolution applied to an image:

This first example shows an edge detection algorithm

This second one shows a sharpness kernel.
I find the easiest way to think about this is that the coefficients provide the weight that that particular pixel contributes to the final result. If you are interested in interactively exploring this concept, I recommend this site: Molecular Expressions Microscopy Primer: Digital Image Processing - Convolution Kernels - Interactive Java Tutorial. It's got some great interactive tutorials.

In the next post I'll build the Core Image kernel using the OpenGL shading language.

Building a Cocoa application using a custom Core Image filter (introduction)

A couple of months ago, some members of my team were working on a facial recognition project for one of our customers. One thing they were trying to do was to determine if filtering the images in various ways could improve the recognition algorithm.

They were implementing their system in Java, but I immediately thought that Core Image would make their job much faster, since we could offload the image processing to the graphics cards on our Mac Pros rather than writing the algorithms in Java.

If you are not familiar, Core Image is Apple's image processing framework which can provide very high performance image processing by offloading the work of image processing to the graphics card. Core Image includes a huge number of image filters (just check out the list on the Apple web page).

The test system applied a sequence of image convolution filters to images with varying coefficients and then submitted the filtered images to the recognition system where they would be scored. We iterated over different combinations of filters to find a way to improve the scoring of the algorithm in general.

I was surprised to discover that Apple did not include a convolution filter by default. Convolution is the "jack of all trades" of image processing algorithms and can be used for all sorts of things like edge detection, sharpening, blurring, noise reduction, and so on, depending on the coefficients used.

This meant that I would have to develop my own custom image unit that performed convolution. This seemed like it might be a difficult task, but it turned out to be much easier than I expected. Along the way I learned a lot more about Quartz Composer, programming using the Open GL shading language, Cocoa Bindings, and creating custom image units on Mac OS X.

Over the next several posts I'll step through the development process to a working sample Cocoa application that allows you to apply convolution to any image.