Kinect Slide Tutorial

Part of my Kinect Portfolio presentation had slides controlled by gestures. This short tutorial explains how I made it work. As I said in the other post, I used a Gesture Library to help with the gesture recognition, Kinect SDK Dynamic Time Warping (DTW) Gesture Recognition. This is a great project that is available on CodePlex and uses a cool algorithm called Dynamic Time Warping which is pretty cool for pattern recognition such as gestures or voice, among others. If you’re curious like me and want to know how this algorithm works, take a look at this link.

But onto the slide tutorial. I started by taking the core files to handle the gesture recognition and plug it in with my project. After a couple of little changes mostly to call an event handler when a gesture is recognized and to work with the final Kinect SDK 1.0., I used the original program to generate the gestures file which contains the information needed to detect gestures. I actually recorded two different gestures for each “SlideLeft” and “SlideRight” gesture just to make sure that different gestures done by different people would be caught by the algorithm.

The next step was to figure out how to animate the slides. I didn’t have experience with WPF animations but I arrived at this solution which works but might not be the best.

VisualStates and their transitions

I created 3 visual states to handle the slide left and slide right: previous (VisualState 1), current (VisualState 2) and next (VisualState 3). There are 2 visual transitions (left and right) between each 2 states and every time we change from one visual state to another we update the “next” target with new slides.

To automatize the transitions I decided to create a SlideManager that starts by reading all the “slides” (Grid elements) under a parent element and initializing the visual states by adding the event handlers. I created a simple structure to keep track of slides and transitions between slides because initially I wanted to include vertical transitions by ended up not doing them.

// Initialize slides.
for (int i = 0; i < mainSlideContainer.Children.Count; i++)
{
	Grid slide = mainSlideContainer.Children[i] as Grid;
	if(slide == null)
		continue;

	Slide s = new Slide()
				            {
				                Name = slide.Name,
				                GUIElement = slide,
				            };

	m_slides.Add(s);
}

for (int i = 0; i < m_slides.Count; i++)
{
	Slide nSlide = null;
	Slide pSlide = null;

	if (i < m_slides.Count - 1)
		nSlide = m_slides[i + 1];
 	if (i > 0)
		pSlide = m_slides[i];

	Slide s = m_slides[i];
	s.Transitions = new Dictionary();
	if(nSlide != null)
		s.Transitions.Add(Transition.RightToLeft, nSlide);
	if (pSlide != null)
		s.Transitions.Add(Transition.LeftToRight, pSlide);
}

There are two other important methods that change slides: MoveToNextSlide() and MoveToPrevSlide(). Each method checks if the current slide has the transition we want to perform, updates the current slide and visual state attribute members and calls the method that actually performs the transition.

public void MoveToNextSlide()
{
	if(m_inTransition)
		return;

	Slide nextSlide;
	if (m_curSlide.Transitions.TryGetValue(Transition.RightToLeft, out nextSlide))
	{
		m_prevSlideIndex = m_curSlideIndex;

		m_curSlideIndex = Math.Min(m_curSlideIndex + 1, m_slides.Count - 1);
		m_curSlide = m_slides[m_curSlideIndex];
		m_curVSIndex = (m_curVSIndex + 1) % m_visualStates.Count;
		m_curVS = m_visualStates[m_curVSIndex];

		m_inTransition = true;
		ExtendedVisualStateManager.GoToElementState(m_vsManagerContainer, m_curVS.Name, true);
	}
}

Another part of the code that is worth mentioning is the two event handlers at the end of this manager: Storyboard_Completed and VSGCurrentStateChanged.

private void Storyboard_Completed(object sender, EventArgs e)
{
	m_slides[m_curSlideIndex].GUIElement.Opacity = 1;
	m_slides[m_curSlideIndex].GUIElement.Visibility = Visibility.Visible;

	if (m_prevSlideIndex != m_curSlideIndex)
	{
		m_slides[m_prevSlideIndex].GUIElement.Opacity = 0;
		m_slides[m_prevSlideIndex].GUIElement.Visibility = Visibility.Collapsed;
	}
}

The visual state transitions do the slide movement and the storyboard controls the opacity and visibility of the element in that state. Whenever we change from one state to the other we have to reset the opacity and visibility properties so that the next time we do the transitions the new slides have the correct properties.

private void VSGCurrentStateChanged(object sender, VisualStateChangedEventArgs e)
{
	VisualState nvs = m_curVS;

	// update the next visual state time line (only if necessary)
	if (m_curSlideIndex < m_slides.Count - 1)
	{
		Timeline tl = FindObjectByName(nvs.Storyboard.Children, nvs.Name + "_next" + TLOpacName);
		if (tl != null)
			tl.SetValue(Storyboard.TargetNameProperty, m_slides[m_curSlideIndex + 1].Name);

		tl = FindObjectByName(nvs.Storyboard.Children, nvs.Name + "_next" + TLVisName);
		if (tl != null)
			tl.SetValue(Storyboard.TargetNameProperty, m_slides[m_curSlideIndex + 1].Name);
	}

	m_inTransition = false;
}

The same goes for the storyboard target name property, whenever we change to a new state we have to update the next slide. I also used a flag to prevent it from starting a new transition while we’re still doing the animation from the current transition.

The KinectManager is a simple class to handle the Kinect’s initialization that I adapted from the samples. It’s far from complete as it is but it serves well for this tutorial.

You may ask yourself why go through all this trouble when there is PowerPoint and Kinect applications to do the slide animation controls? Firstly, because it is fun to explore and learn new things and secondly because there are parts of the complete interactive portfolio that weren’t so easy to do in PowerPoint such as the particle system or the pong game.

Note: The transitions between texts were programmed in a similar way but instead of using 3 Visual States I only used 2, one for display the text and the other and to hide it, then it’s just a matter of setting the right text when you do the transitions.

Download
Kinect Slide Tutorial (requires: .Net4, Kinect SDK 1.0, VS2010 and up)

More movies as code

I must admit that that Movies as Code site got me addicted Smile Here are a few other entries of mine:

Back To The Future

svn merge -r 1985:1955 .

Indiana Jones and the Raiders of the Lost Ark

void RetrieveGoldenIdol()
{
    while(true)
    {
        float dif = bag.Weight - indy.CalculateApproxWeight(idol);
        // 200g of margin should be enough.
        if(dif < -100) bag.AddSand(rand.Next(100, -dif));
        else if(dif > 100) bag.RemoveSand(rand.Next(100, dif));
        else break;
    }
    Swap(bag, idol);
    DeployTraps();
    indy.Run();
}

Jumper

void Jump()
{
    if(Seen("place"))
        goto place;
    return;
place: ;
}

These ones weren’t posted because they were similar to others on the site:

Inception

function goDeeper() {
    if(kick) {
        if(wakeFromKick)
            return;
        else
            Limbo();
    }
    goDeeper();
}

Stargate

void Stargate()
{
earth:
    goto newPlanet;

    // travelling

newPlanet:
    Explore();
    KillGoaulds();

    goto earth;
}

Back To The Future as Code

David Amador twitted about this cool site Movies as Code and I decided to make a simple entry for Back To The Future as C++. Here is my entry: http://moviesascode.net/adventure/back-to-the-future-2/

The code doesn’t appear correctly on that site so here it is the full version:

#include <future>
#include <iostream>

int main()
{
  auto future = std::async([] { return "Future"; });
  std::cout << "Back To The " << future.get() << std::endl;
  return 0;
}

For those who didn’t get it: C++11 added a new feature that is used in the sample called “future” that will hold the result of an async function somewhere in the future. It was just to play around with the name of that feature 🙂

Windows 8, Visual Studio 11 and CLI Args Made Easy 11

I decided to try the new Windows 8 consumer preview and Visual Studio 11 Beta to see how the new systems behave and to check the new features. Windows 8 may surprise some (as it did to me) at the beginning but I personally like most of the changes. VS 11 also as a new look and lots of new features but I’ve only started to scratch the surface of it. The first thing that I tried on VS11 was to see if CLIArgsMadeEasy2010 worked and it didn’t so I decided that this would be my first project in VS11.

It was actually easier than I thought it would be as it just needed to be recompiled in VS11. So here it is the new version of CLIArgs, CLI Args Made Easy 11. It has all the same features of CLIArgsMadeEasy2010 v1.2 but I’ll keep it beta for now while the VS11 is in beta. Grab the new version here.

Interactive Portfolio Update

I just posted an updated version of my Interactive Portfolio using Kinect. It was updated to work correctly with the latest Kinect SDK 1.0. The application is now installed via ClickOnce which checks if you have the software requirements installed to run the app. It took a bit to figure out while ClickOnce wasn’t including some required files in the installer but it appears that there is a problem when mixing XNA and WPF projects.

You can grab the project here.

Kinect and WP7 interaction

By the same time I did my portfolio I also did another experiment that’s in this post. A few days ago I saw the KinectoPhone project and thought it would be cool to adapt it to another project I was developing with Francisco Campelo. This project allows a desktop application using Kinect to communicate with another application on the Windows Phone 7 synching its state between the two apps by communicating through a server.

I changed a bit the network code and plugged in with other code I had done for the particles and here’s the result:

Kinect and WP7 interaction

The red particles on the big screen are from Kinect’s skeleton and the blue particles are sent from the WP7 through touch screen input. On the WP7 screen, the blue particles are from Kinect’s skeleton and the red/white ones from the user’s input.

This could get many improvements, network and feature wise but I’ll leave that to some other time. I have many ideas but the time is short so I’ll be moving to another one soon.

THANKS TO

  • My parents for helping me out testing and recording the video.
  • Anabela Faria for the video editing.
  • Francisco Campelo for the original idea which led me to the Kinect version.

CLIArgsMadeEasy2010 Update

I just updated my CLIArgsMadeEasy2010 Visual Studio add-in. It fixes a bug that prevented it from working correctly on solutions that included projects inside project folders.

I’ve also decided to update it with a small feature that saves a lot of time. If you’re like me and have a few add-ins installed, you might have a context menu like mine that fits the whole screen at 1080 and it becomes painful to find the “set as starting project” option. So CLIArgsMadeEasy2010 v1.2 now includes a neat combo box with the current startup project and all the other projects which allow you pick another project as the startup project very easily by choosing it from this combo box.

If you don’t use any of the features you can hide them by clicking in the arrow beside the add-in toolbar box.

Grab the add-in here.

Interactive portfolio using Kinect

I like to explore different technologies that relate with computers and that’s why I own a Lego Mindstorms NXT . More recently I bought a Kinect to transform some ideas into real applications.

I’m currently looking for job in software development in the Toronto (Canada) area and it occurred to me that I could present my portfolio in a fun and entertaining way. So I decided to create my first Kinect application to be this interactive portfolio. Although I have some years of experience in business applications, portals, intranets, etc., I decided to compile only my experience in game development and interactive 3d projects.

If you don’t own a Kinect you can watch the video below to see how it works. I encourage you to skip the video in case you want to try the application or you’ll ruin the experience.

Interactive portfolio

The code uses the following technologies: C#, .NET, XNA and WPF. I also used two open source libraries to help with the gestures and the XNA integration with WPF.

DOWNLOAD

Software requirements:

Kinect application:

Note: Please make sure you install the software requirements or you won’t be able to install the application.

THANKS TO

  • My parents for helping me out recording the video.
  • Anabela Faria for the video editing.
  • Paulo Silva for the animated silhoutte.

A Walk in the Dark

Bruno Vidal and Paulo Silva started working on “A Walk in the Dark” while they were working with me at ZPX. I saw this project evolve from the very beginning and was always eager to test new builds and levels they were working on. In the meanwhile they left ZPX and continued their work on AWITD.

374769_282116308502179_229708283742982_796597_2146777246_n          AWalkInTheDark_09

Since then, the project had massive improvements in every direction, gameplay, art, music, etc.. Recently I was asked to help in the project and I gladly accepted to join the team in that effort. Unfortunately the time available isn’t much but I hope to help wherever I can.

Announcement trailer

 

Flying Turtle Software is the name of the company behind A Walk in the Dark. The game is supposed to come out in 2012 for the PC.

Jumping Jack for WP7 – GameDev Fun Facts

Last night was time for another XNA Pizza Night event at Microsoft Portugal (Tagus Park). Once again ZPX presented a “new” game. “New” between quotes because the original game is from 1983 and it was older than 90-95% of the people in the audience. The game is none other than Jumping Jack for ZX Spectrum.

I played this game for the first time almost 27 years ago when I had just made 5. In fact, this was the first game I ever played on a computer and it’s probably because of it I’m here making games.

I want to dedicate this game to the memory of Nuno Sabino, my brother. It was him who received the ZX Spectrum 48k as a Christmas present and it was him who got this great game and so many others that influenced me and made me want to program and make games since then. Thanks for everything bro.

jumping_jack_wp7

Let’s get into the development fun facts:

  • It took me about two weeks working after my daily job to get this working for the demo. This would take less time if I worked on it regularly everyday, I would say 1 week. I don’t know how much time it took Albert Ball to program the original version but my guess is that it took him a little longer. He had  to program it in Z80 assembly and didn’t have help from tools we have available today, specially for debugging, it must have been a challenge at that time.
  • Although we had the Spectrum 48k, the original version of the game ran on the Spectrum 16k, i.e., it ran in less than 16k of memory. The actual version for WP7 takes about 10-16 MB, more than 1000 times larger than the original version. I didn’t and won’t make an effort at this time to make the game as smaller as the original version and I probably couldn’t do it for the WP7 but I think I could do it for the PC version if I had some time. Maybe someday, who knows.
  • The game took around 23 seconds to display a splash screen and 55 seconds to load the game making that 1 minute and 18 seconds total to be able to play it. The WP7 version takes about takes 3 seconds to load (and that’s because it displays the splash screen for 3 seconds otherwise the splash screen would pop in and out too quickly.)
  • At this moment this version has about 2000 lines of code. Once again I don’t know the exact number of lines for the original version but I think it will probably be more than 2k lines of Z80.
  • I started by using Farseer Physics but during development I realized that doing the “physics” by hand would be easier than tweaking Farseer Physics. In this demo I’m still using Farseer doing raycasts to see if the player can jump or falls and the debug to draw the lines “Fixtures”. I will probably end up removing Farseer completely in the final version.
  • The graphics and sounds are exact replicas of the original game. I took screenshots from the game playing on a Spectrum emulator and edited them to adjust the transparencies. I was going to record the sounds from the emulator but Zalo had already done that and I used the sounds from his 3d version.
  • The last thing I want to talk about comes from a Spectrum limitation in the graphics display. Spectrum machines had a 256×192 resolution and 15 available colors (3-bit palette that makes 8 colors and another additional bit that says if a color is bright or dark but since one of those colors was black, that didn’t have the bright version making the 15 colors.) The way the information was drawn on the screen was in blocks of 8×8 pixels known as attribute blocks. Only two colors could be used in each of these blocks and if a program tried to display 3 colors in any attribute block one of the colors would change to match one of the other colors, this was known as attribute clash. In Jumping Jack this problem was evident in various situations, one of them was when Jack crashed against a line/wall as seen in the following image:

jumping_jack_wp7_headcrash

Jack was rendered in red due to this limitation and I decided to keep it rendered in red as I thought it was funnier and made sense to show the shocking moment.

And that’s it, Jumping Jack for Windows Phone 7 will be released eventually (soon I hope), maybe by other name.