Frankly, the API and SDK for developing with Unity and the Perc, are well-written and comprehensive all things considered. I do not know about other teams, but from my perspective, this is far from the wild west of experimental programming back in the 8 bit and 16 bit Amiga days of copper list hacking and trial and error pioneering stuff.
That’s not to say that the spirit of those early experimental years isn’t here – it very much is! intel has managed to both inspire me and also give me pause for thought. I went into this challenge without any hope really – The Other Brothers is currently in crunch time for launch, and I have pitiful little time to devote to this challenge.
But I found time today and you know what? it made me smile. It also taught me that Perceptual Computing is NOT a fad. It’s not a here today, gone tomorrow passing fad. I’m going to go out on a limb and stake my reputation on saying that it is the future of all computing to come. Using it with a touch screen display like you get on the Ultrabooks really hits home how useful it will become.
I can see myself lazily swiping the air just to tab between apps – perhaps to check Skype messages – I could pinch gesture and throw skype into a corner of the screen and dock it there, all the while talking and the perc device would just be recording and translating this document. It would work WITH humans, not against them. If you think about it, gesturing is a natural human behavioural trait – the mouse is not. Like it or loathe it, perceptual computing is the future.
Right now it’s the wild west and I feel priveleged to be part of this wonderful challenge and it has been the only thing in recent weeks that has truely been a breath of fresh air. In fact I am now an evangelist of the technology – if people cannot see the raw potential here then perhaps I have reason to despair!
Intel are doing the right thing here, they are courting developers, and doing masses of R&D, holding challenges and more in order to bring perceptual computing under the spotlight. Like all good new things, this may take a while before it becomes an everyday part of our computing lives. But once upon a time, touch-screens were a novelty, and then they were seen as essential for mobile. Now you will find touch screens on high end Ultrabooks in your local computer store. What was once seen as a gimmick is now useful enough for everyday tasks.
Onto the code… reading the basic instructions seems straightforward enough -
1. Copy 3 files into your unity plugins folder. All unity programmers know this, right – done!
2. Create a cube and render gestures to a texture, with a simple piece of code, which we’ll be mapping onto the cube, just to see. Classic prototype stuff here, folks!
using UnityEngine;
public class DrawCube: MonoBehaviour
{
private Texture2D texture;
private PXCUPipeline.Mode mode=PXCUPipeline.Mode.GESTURE;
private PXCUPipeline pp;
void Start()
{
pp = new PXCUPipeline();
pp.Init(mode);
int width, height;
pp.QueryLabelMapSize(out width, out height);
texture = new Texture2D(width,height,TextureFormat.ARGB32,false);
renderer.material.mainTexture = texture;
}
void OnDisable()
{
pp.Close();
}
void Update()
{
if (!pp.AcquireFrame(false)) return;
if (pp.QueryLabelMapAsImage(texture)) texture.Apply();
pp.ReleaseFrame();
}
}
…And it does not work. That’s right – the unity example PDF is out of date, and QueryLabelMapSize interface with dll seems to imply that it wants an array of 2 ints instead. Easily fixed, but it will catch some programmers unawares. Remember, this is an SDK that is still in development. The following code fixes these issues and borrows some extras:
using UnityEngine;
using System;
using System.Runtime.InteropServices;
public class Main:MonoBehaviour
{
private Texture2D tex;
private PXCUPipeline.Mode mode = PXCUPipeline.Mode.GESTURE;
private PXCUPipeline pp;
private int[] size = new int[2]{0,0};
//=======================================================================
void Start()
{
pp = new PXCUPipeline();
if (!pp.Init(mode))
{
print("Unable to initialize the PXCUPipeline");
return;
}
if (pp.QueryLabelMapSize(size))
print("LabelMap: width=" + size[0] + ", height=" + size[1]);
else if (pp.QueryRGBSize(size))
print("RGB: width="+size[0]+", height="+size[1]);
if (size[0]>0) {
tex = new Texture2D (size[0], size[1], TextureFormat.ARGB32, false);
renderer.material.mainTexture = tex;
}
}
//=======================================================================
void OnDisable()
{
pp.Close();
pp.Dispose();
}
//=======================================================================
void Update ()
{
if (!pp.AcquireFrame(false)) return;
Process();
pp.ReleaseFrame();
}
//=======================================================================
void Process()
{
if (pp.QueryLabelMapAsImage(tex)) tex.Apply();
}
//=======================================================================
}
…So what does everything do? Well most of it is just C# setup stuff, the important part is:
pp = new PXCUPipeline();
This creates a new object called pp because that’ll stand for PXCU Pipeline, and we’ll be able to send commands to the system. For this example, only gestures are initialised, and that’s done by setting the mode with:
private PXCUPipeline.Mode mode = PXCUPipeline.Mode.GESTURE;
With the command:
pp.Init(mode)
So far, so good. Nothing scary here, but this example is pretty simple. The main update loop will terminate early if pp hasn’t aquired a frame yet, otherwise it will grab an image from the camera.
At the end of grabbing an image, it lets the system know that it’s ok to continue with its business using:
pp.ReleaseFrame();
Hitting play gives me a darkened blobby thing on screen! wait- thats ME! I do feel I need a haircut. It’s at this point I toy with the concept of wanting to make a virtual hairdressing app. I can only imagine how dazzling I would look with golden flowing locks.
But, back to business! this app is about pottery, and manipulating clay. It never goes as far as say, zbrush, because it’s designed for anyone with no know-how to be able to engage with perceptual technology, and get used to it.
In a way, it is these apps we are doing – pottery, or other similar creativity apps, that will ease computer users into thinking about gesture commands and what they do with their hands. I am quite sure one day, Windows will understand when an Italian user is feeling rather annoyed with the paperclip!
Back to work, so my first job is to be able to get sane depth, and sane numbers that can actually be used. Usually to do this you want to transform your gestures into normalised space, that is, the dimensions must be between 0 and 1, and this means we can scale that back up to resolution, or in 3D space just fine.
To calculate normalised coordinates, you divide the number you’re getting with the largest it can be, so if we’re looking at greyscale depth, the largest colour would be 255, so we divide by that. Or 256, depending on your viewpoint.
So 0 to 255 becomes 0 to 1. Now we can transform this coordinate however we like, into any other space, without worries if it will be too big, or too small, or warped.
—-
The next post will be a bumper one – Giuseppe will be video blogging from GDC, there will be a healthy serving of code-talk from Rob followed up with a hands-on (finally) look at this project we’re calling ‘Pot’.
We will be talking on Intel’s Theater stage, so please do come and participate! We’re on stage on Thursday 28th March at 15:30 and in the booth on the 27th and 29th - follow us on Twitter to be notified of the exact times and places closer to the event.
In the mean time, here is some actual footage of the scene running on the Ultrabook itself to whet your appetite:
Originally posted on Intel.com
