Showing posts with label quartz composer. Show all posts
Showing posts with label quartz composer. Show all posts

Mar 28, 2008

Possible Interface Builder bug with QCView in Leopard


A friend of mine was trying to embed a Quartz Composer composition into a Cocoa app, and he was having problems getting it to work properly. The composition responded to mouse events and it worked just fine within QC, but for some reason not within a QCView. I took a look at the project and he had done everything right, as far as I could tell; the "ForwardAllEvents" checkbox was checked. 

So, I decided to perform a little experiment. I added a view controller to the application with the QCView as an outlet.

Lo and behold, the eventForwardingMask was set to 1, not NSAnyEventMask as it should have been. The solution then presented itself: Set this value correctly in the -awakeFromNib and now the application worked properly
Here's the header file:

#import <cocoa/cocoa.h>
#import <quartzcomposer/qcview.h>

@interface MyViewController : NSObject {
IBOutlet QCView *myView;
}

@end

And here's the implementation:

#import "MyViewController.h"

@implementation MyViewController

- (void)awakeFromNib
{
NSLog(@"%d", [myView eventForwardingMask]);
[myView setEventForwardingMask:NSAnyEventMask];
NSLog(@"%d", [myView eventForwardingMask]);
}

@end

I'm going to file a bugreporter bug right away.


Apr 29, 2007

Cocoa Application with custom Core Image filter 6: Embedding a Quartz Composer composition directly in an app.

In the last episode, we built a Cocoa application to filter an image using Convolution. This application followed a more traditional path to developing a Cocoa application. We took an image from our view, handed it to the model to be processed and sent it back to the view for display. With the advent of Quartz Composer, this application can be implemented in an entirely different way that opens up all kinds of interesting possibilities for interactive graphics applications. I'm sure all of you have seen that Quartz Compositions can respond to external inputs. Quartz Composer can take input from keyboard and mouse, spotlight image searches, folders of images, and even RSS feeds. Apple's RSS screensaver is a prime example of a Quartz Composition that uses RSS input.

An other way of feeding input and getting output from a composition is by publishing input and output parameters from your compositions. In the most simple sense, these will allow Quartz Composer itself to present a simple UI to allow you to interact with your composition. For example, in my test.qtz composition, I publish the inputImage and all the coefficients so that QC can provide a simple user interface.

More interesting than this, however, is the use of a composition embedded within a QCView and accessing its parameters using Cocoa bindings technology and the QCPatchController. I've provided a link to this project so you can see how I did it: Convolver.zip

The first thing you need to do to use these in your projects is to add the Quartz Composer palette to Interface Builder. To do this:

Open Interface Builder Preferences:

IB preferences showing palettes tab

Click the Add... button:

selecting the QuartzComposer palette

You will get this new palette in Interface Builder.

the Quartz Composer palette

Next, we need to copy the test.qtz composition into the bundle for our application. It doesn't really matter where you put it. For our purposes, we just need to make sure that it is packaged with the app when it ships. After that we need to open the composition and make sure that we publish all the inputs that we care about so that the QCPatchController can access them. In order to do this, you have to right-click on whatever patches in you composition have external inputs or outputs and check off each one you wish to publish. As far as I've been able to tell, there are no shortcuts to this, you have to select each one individually.

Publishing inputs from the compostion

Whatever you name the inputs or outputs at this point is the key that will be used to access that parameter later.

Now that we have a properly set up composition, we have to modify the original project to use the QCView and QCPatchController instead of old-school target/action. First thing to do is to replace the outputImage NSImageView on the right of the main window of the app with a QCView. Then, drag a QCPatchController into the nib file. I set the erase color of the QCView to transparent to avoid the large black square on the right side of the application when it starts.

Next., we'll go ahead and set up the bindings for the project. The QCView's patch binding is bound to the patch key on the QCPatchController.


We set the value binding of the NSImageView to the patch of the QCPatchController with the src.value key path.


Finally, each cell of the NSMatrix gets bound to the correct value in the composition:


The final piece of this setup is to load the composition into the QCPatchController


Now, you can actually test the working application from within IB. If you choose File->Test Interface and drop an image on the left image view you will see the convolved output image displayed on the right. Of course, this is exploring the barest minimum of the possibly utility of Quartz Compositions and QCView.

This has greatly reduced the amount of Objective-C code we need to write to build this application. The entire Convolver class that converted the image to a CIImage, called the filter and reconverted the result back into an NSImage is not needed any more. I still want to be able to select File->Open... to drop down a sheet and open the image, so that's really the only thing I need the ConvolutionController to do at this point. There turns out to be one small catch that I'll point out when I get to it.

Here's the new header for the ConvolverController class:

/* ConvolverController */

#import <Cocoa/Cocoa.h>
#import <QuartzComposer/QuartzComposer.h>
#import <QuartzCore/QuartzCore.h>/span>


@interface ConvolverController : NSObject
{
IBOutlet QCView *resultImage;
IBOutlet NSImageView *sourceImage;
IBOutlet NSWindow *window;
}
- (IBAction)openImage:(id)sender;
@end
Some of you may look at this and wonder why I need a pointer to the resultImage any more since the bindings should take care of it. That's where the catch comes in, as you'll see below.

This is the source code for the class:

#import “ConvolverController.h“


@implementation ConvolverController

- (IBAction)openImage:(id)sender
{
NSOpenPanel *panel = [NSOpenPanel openPanel];
[panel beginSheetForDirectory: nil
file:nil
types:
[NSImage imageFileTypes]

modalForWindow: window
modalDelegate:self
didEndSelector:
@selector(openPanelDidEnd:returnCode:contextInfo:)

contextInfo:nil];
}

- (void)openPanelDidEnd:(NSOpenPanel *)panel
returnCode:(int)returnCode
contextInfo:(void *)contextInfo{

NSArray *files = [panel filenames];
NSString *filename = [files objectAtIndex:0];
NSImage *image = [[[NSImage alloc]
initByReferencingFile:filename]
autorelease];

[sourceImage setImage: image];
[resultImage setValue: image forInputKey: @“src“];
}
@end

This is normal sheet handling code, just like in the previous project, but a little simpler. The catch comes in when I have to call [resultImage setValue: image forInputKey: @“src“] even though you might think that the bindings for NSImageView should automatically be updated, they are not. Apparently it's an already filed bug and this one line of code provides a simple workaround.

So, that ends the journey of building a Cocoa app with a custom Core Image filter. Hope it was useful to you. This was a very simple example, there's so much more that can be done with these amazing technologies. Until next time, happy coding!

Mar 10, 2007

Cocoa Application with custom Core Image filter 3: Testing the kernel with QuartzComposer

In the last post I wrote a convolution kernel for Core Image. But these kind of things need to be tested. Now, we could package the filter inside an Image Unit (which I'll do in a future post) but that would mean we would have to recompile and install the filter each time we wanted to make a change. It would be much better if there was an interactive environment that could be used to test any changes and to allow us to debug the filter code.

Luckily, Apple has provided just such an environment in QuartzComposer. If you are not familiar, Quartz Composer is an amazing application that provides a visual programming environment that allows the creation of all kinds of visualizations. If you've seen the RSS screensaver. then you've seen a QuartzComposer composition. If you've seen a preview of Time Machine, the swirling galaxy in the background is a QuartzComposer composition. I'm not going to explore this amazing piece of software in depth here, I suggest you look at some of these websites for more information: Quartz Composer Journal or www.quartzcompositions.com or boinx or Quartonian.

As a basic introduction, a composition is defined by dragging patches onto the workspace and connecting them together graphically. Patches contain ports which represent parameters which are passed between the patches. The results are displayed in real time in the viewer window, so you get immediate feedback as you change things. As we'll see later, Quartz composer compositions can be embedded in a QCView in your Cocoa apps, and controlled via Cocoa Bindings.

I've provided a link to my test.qtz composition which consists of four patches. The convolution patch is built by dragging a core image kernel patch onto the editor and then copying the text of the kernel into the patch. You can use the inspector to change the input parameters to adjust the coefficients and test out the composition.


This is the editor window, showing the composition.


Here's the inspector for the Convolution patch showing where you put the kernel code.


This is the viewer, showing the input image filtered by the edge detection filter defined in the Input Parameters to the Convolution filter.


If you have an iSight camera or other video source, replace the Image Importer with a video input, and you will see the convolution filter applied to your video stream in real time. That's just extremely cool.

Here's the editor window with a Video Input instead or the Image Importer.
And here's a view of my living room with edge detection. Notice the frame rate of nearly 60 frames per second on my Mac Book Pro.

Using this basic composition, with some modifications, you should be able to test out any core image kernel you can come up with. Next time, I'll build this filter into an image unit that can be used from any application that uses Core Image filters.

Feb 27, 2007

Building a Cocoa application using a custom Core Image filter (introduction)

A couple of months ago, some members of my team were working on a facial recognition project for one of our customers. One thing they were trying to do was to determine if filtering the images in various ways could improve the recognition algorithm.

They were implementing their system in Java, but I immediately thought that Core Image would make their job much faster, since we could offload the image processing to the graphics cards on our Mac Pros rather than writing the algorithms in Java.

If you are not familiar, Core Image is Apple's image processing framework which can provide very high performance image processing by offloading the work of image processing to the graphics card. Core Image includes a huge number of image filters (just check out the list on the Apple web page).

The test system applied a sequence of image convolution filters to images with varying coefficients and then submitted the filtered images to the recognition system where they would be scored. We iterated over different combinations of filters to find a way to improve the scoring of the algorithm in general.

I was surprised to discover that Apple did not include a convolution filter by default. Convolution is the "jack of all trades" of image processing algorithms and can be used for all sorts of things like edge detection, sharpening, blurring, noise reduction, and so on, depending on the coefficients used.

This meant that I would have to develop my own custom image unit that performed convolution. This seemed like it might be a difficult task, but it turned out to be much easier than I expected. Along the way I learned a lot more about Quartz Composer, programming using the Open GL shading language, Cocoa Bindings, and creating custom image units on Mac OS X.

Over the next several posts I'll step through the development process to a working sample Cocoa application that allows you to apply convolution to any image.