Frankly, the API and SDK for developing with Unity and the Perc, are well-written and comprehensive all things considered. I do not know about other teams, but from my perspective, this is far from the wild west of experimental programming back in the 8 bit and 16 bit Amiga days of copper list hacking and trial and error pioneering stuff.

That’s not to say that the spirit of those early experimental years isn’t here – it very much is! intel has managed to both inspire me and also give me pause for thought. I went into this challenge without any hope really – The Other Brothers is currently in crunch time for launch, and I have pitiful little time to devote to this challenge.

But I found time today and you know what? it made me smile. It also taught me that Perceptual Computing is NOT a fad. It’s not a here today, gone tomorrow passing fad. I’m going to go out on a limb and stake my reputation on saying that it is the future of all computing to come. Using it with a touch screen display like you get on the Ultrabooks really hits home how useful it will become.

I can see myself lazily swiping the air just to tab between apps – perhaps to check Skype messages – I could pinch gesture and throw skype into a corner of the screen and dock it there, all the while talking and the perc device would just be recording and translating this document. It would work WITH humans, not against them. If you think about it, gesturing is a natural human behavioural trait – the mouse is not. Like it or loathe it, perceptual computing is the future.

Right now it’s the wild west and I feel priveleged to be part of this wonderful challenge and it has been the only thing in recent weeks that has truely been a breath of fresh air. In fact I am now an evangelist of the technology – if people cannot see the raw potential here then perhaps I have reason to despair!

Intel are doing the right thing here, they are courting developers, and doing masses of R&D, holding challenges and more in order to bring perceptual computing under the spotlight. Like all good new things, this may take a while before it becomes an everyday part of our computing lives. But once upon a time, touch-screens were a novelty, and then they were seen as essential for mobile. Now you will find touch screens on high end Ultrabooks in your local computer store. What was once seen as a gimmick is now useful enough for everyday tasks.

Onto the code… reading the basic instructions seems straightforward enough -

1. Copy 3 files into your unity plugins folder. All unity programmers know this, right – done!

2. Create a cube and render gestures to a texture, with a simple piece of code, which we’ll be mapping onto the cube, just to see. Classic prototype stuff here, folks!

using UnityEngine;
public class DrawCube: MonoBehaviour
{
private Texture2D texture;
private PXCUPipeline.Mode mode=PXCUPipeline.Mode.GESTURE;
private PXCUPipeline pp;

void Start()
{
pp = new PXCUPipeline();
pp.Init(mode);
int width, height;
pp.QueryLabelMapSize(out width, out height);
texture = new Texture2D(width,height,TextureFormat.ARGB32,false);
renderer.material.mainTexture = texture;
}

void OnDisable()
{
pp.Close();
}

void Update()
{
if (!pp.AcquireFrame(false)) return;
if (pp.QueryLabelMapAsImage(texture)) texture.Apply();
pp.ReleaseFrame();
}
}

…And it does not work. That’s right – the unity example PDF is out of date, and QueryLabelMapSize interface with dll seems to imply that it wants an array of 2 ints instead. Easily fixed, but it will catch some programmers unawares. Remember, this is an SDK that is still in development. The following code fixes these issues and borrows some extras:

using UnityEngine;
using System;
using System.Runtime.InteropServices;

public class Main:MonoBehaviour
{
private Texture2D tex;
private PXCUPipeline.Mode mode = PXCUPipeline.Mode.GESTURE;

private PXCUPipeline pp;
private int[] size = new int[2]{0,0};

//=======================================================================

void Start()
{
pp = new PXCUPipeline();
if (!pp.Init(mode))
{
print("Unable to initialize the PXCUPipeline");
return;
}

if (pp.QueryLabelMapSize(size))
print("LabelMap: width=" + size[0] + ", height=" + size[1]);
else if (pp.QueryRGBSize(size))
print("RGB: width="+size[0]+", height="+size[1]);

if (size[0]>0) {
tex = new Texture2D (size[0], size[1], TextureFormat.ARGB32, false);
renderer.material.mainTexture = tex;
}
}

//=======================================================================

void OnDisable()
{
pp.Close();
pp.Dispose();
}

//=======================================================================

void Update ()
{
if (!pp.AcquireFrame(false)) return;

Process();

pp.ReleaseFrame();
}

//=======================================================================

void Process()
{

if (pp.QueryLabelMapAsImage(tex)) tex.Apply();
}

//=======================================================================

}

…So what does everything do? Well most of it is just C# setup stuff, the important part is:

pp = new PXCUPipeline();

This creates a new object called pp because that’ll stand for PXCU Pipeline, and we’ll be able to send commands to the system. For this example, only gestures are initialised, and that’s done by setting the mode with:

private PXCUPipeline.Mode mode = PXCUPipeline.Mode.GESTURE;

With the command:

pp.Init(mode)

So far, so good. Nothing scary here, but this example is pretty simple. The main update loop will terminate early if pp hasn’t aquired a frame yet, otherwise it will grab an image from the camera.

At the end of grabbing an image, it lets the system know that it’s ok to continue with its business using:

pp.ReleaseFrame();

Hitting play gives me a darkened blobby thing on screen! wait- thats ME! I do feel I need a haircut. It’s at this point I toy with the concept of wanting to make a virtual hairdressing app. I can only imagine how dazzling I would look with golden flowing locks.

But, back to business! this app is about pottery, and manipulating clay. It never goes as far as say, zbrush, because it’s designed for anyone with no know-how to be able to engage with perceptual technology, and get used to it.

In a way, it is these apps we are doing – pottery, or other similar creativity apps, that will ease computer users into thinking about gesture commands and what they do with their hands. I am quite sure one day, Windows will understand when an Italian user is feeling rather annoyed with the paperclip!

Back to work, so my first job is to be able to get sane depth, and sane numbers that can actually be used. Usually to do this you want to transform your gestures into normalised space, that is, the dimensions must be between 0 and 1, and this means we can scale that back up to resolution, or in 3D space just fine.

To calculate normalised coordinates, you divide the number you’re getting with the largest it can be, so if we’re looking at greyscale depth, the largest colour would be 255, so we divide by that. Or 256, depending on your viewpoint.

So 0 to 255 becomes 0 to 1. Now we can transform this coordinate however we like, into any other space, without worries if it will be too big, or too small, or warped.

—-

The next post will be a bumper one – Giuseppe will be video blogging from GDC, there will be a healthy serving of code-talk from Rob followed up with a  hands-on (finally) look at this project we’re calling ‘Pot’.

We will be talking on Intel’s Theater stage, so please do come and participate! We’re on stage on Thursday 28^th March at 15:30 and in the booth on the 27^th and 29^th - follow us on Twitter to be notified of the exact times and places closer to the event.

In the mean time, here is some actual footage of the scene running on the Ultrabook itself to whet your appetite:

Originally posted on Intel.com

So what is Simian Tech: Mode 7?

It’s one part of our proprietary engine called Simian Tech, and specifically what gives The Other Brothers its distinct rendering style.

Built on top of the standard unity rendering pipeline is a dedicated set of tools, shaders and custom mesh code we’ve created that perfectly emulate 16 bit style and DOS style rendering. This technology enables us to colour cycle, flash, make things a “virtual resolution”, add realtime dithering and realtime resolution normalization so that you get a real retro feel from it as opposed to just drawing it all. It includes vignetting, plus screen “burn” effects to emulate an old monitor display as well, however that is kept to a minimum for The Other Brothers in order not to detract from the fine artwork.

Weeks of optimisation have gone into this with hand crafted shaders – none of that rubbish unity default prefab nonsense, this is coded from the ground up for speed.

We can confirm ipad1, ipad2, and the new ipad as well as iPhone 4 and 4S as running at a cool 60fps so far – this is with mode7 and vignetting, burn etc…

And most Droid owners will be really happy with 60fps!

3GS has yet to be tested but if it’s not 60fps on them it will at least never drop below 30fps. It was a nightmare to pull off because whenever I moaned about overdraw / fill rate to them they would stick their fingers in their ears and cry “la la la la la la we can’t hear you! la la la la” and happily just built levels with masses of waste overdraw – all in the name of art!

SIGH! but anyway after losing hair and sanity I think Simian’s pulled off a fine little engine to drive TOB with and the gamers will appreciate the fluid framerates and fast responsiveness.

We made a couple of design decisions such as jumping being on auto repeat (hold it down and it will jump as soon as you land again) – to minimise slips, or fingers not connecting properly with the screen (perils of a touch screen device) and much care has gone into making it feel responsive with good reason… sometimes you get a murderous amount of enemies trying to take you down – they are almost like mini bosses in some ways!

There will be a sexy new video coming very early after the weekend which I hope you guys will be pleased to view.

Thanks for reading!

A.I

Artificial Intelligence, or Acute Idiocity? There’s a lot of fun to be had making AI and it’s actually very easy for an experienced programmer like me to invent AI that is foolproof and able to dodge your every attack while ensuring it can kill you. At Simian Squared, I’ve spent many late nights devising the perfect blend of AI, and observing my pet cats… although to be honest, I think they were observing me!

But AI is an interesting subject. Make them too smart and it becomes tiring…. make it too stupid and it becomes predictable. Instead for games, the best AI is “entertaining” AI. It must always behave interesting and make you play a little bit different for each creature, it is how you make the game or break the game.

I actually felt that the most important part for the AI in The Other Brothers would be to make the AI force the player to Play Different. Play Different is a motto of mine when it comes to playability and game mechanics. I love to make the player use the same controls, but react and act differently in response to situations the game throws up.

As you can see in the above screenshot, it shows ratty, he has a red box surrounding him which allows the level designers to change his ‘Aggro Range’ – the range in which he will get extremely annoyed with you!

The yellow line indicates the patrol route. Both are adjustable, so you can have rats which patrol a very long way, or rats that can’t see very far. The designer is able to change the tint of any creature as well as change these properties. Best of all, every property in the AI is animatable…! So without the programmer being present, it’s possible for an artist or level designer to create a virtually unlimited amount of behaviour from a simple set of behaviours. I try to discourage them though. A level designer out of control is a terrifying thing for a programmer to witness!

Each creature in TOB has 3 difficulty levels the designer can pick from. There’s stupid, average and smart. This is adjusted in the code on a per-creature basis (it’s a lot of work!) but it’s worth it because you don’t want every creature to behave the same. For example stupid dogs can’t spot you creeping up behind them, but average dogs can. Smart dogs may even avoid your stomp attacks from time to time…

Controlled Random

It’s important to have a “controlled random fury” going on with the numbers. In my AI, the system is constantly fed inputs with random variation. The player might be close, but it has a little random fudge to it as the enemy might not judge if you’re perfectly close enough to have a nibble on. Likewise this is present for their movement behaviour.

Sometimes random numbers fight random numbers which are fed in as a fudge for a normal sensory input, which makes it feel like it is making decisions (it actually is) but sometimes it will not always be a good decision.

It was important to slow down how often enemies make decisions. This is known as the time it takes to change your mind. This also has a little random edge to it as well. Nothing is completely random at all, it is merely taking a cold logical true or false, and giving it shades of grey for exciting variation.

Emotion and Logic

Each creature has a boredom level as well. If you keep diddling around it will get bored of your pranks and wander back off… maybe. Don’t be surprised if a smart one pretends to lose interest and walk off…

After all what is more fun to fight: spock or the hulk? Spock would grip you or distract you (yawn) but the Hulk has random fury and will smash the place up, you could slip away or Hulk could get distracted by a cheeseburger. You just don’t know.

Above all, the AI in TOB is very much interested in giving the player a good time rather than being infallable. It’s entertaining AI, not stressful AI. And every single creature is different.

Stay Pixelated

-Rob