Kinect Slide Tutorial

Part of my Kinect Portfolio presentation had slides controlled by gestures. This short tutorial explains how I made it work. As I said in the other post, I used a Gesture Library to help with the gesture recognition, Kinect SDK Dynamic Time Warping (DTW) Gesture Recognition. This is a great project that is available on CodePlex and uses a cool algorithm called Dynamic Time Warping which is pretty cool for pattern recognition such as gestures or voice, among others. If you’re curious like me and want to know how this algorithm works, take a look at this link.

But onto the slide tutorial. I started by taking the core files to handle the gesture recognition and plug it in with my project. After a couple of little changes mostly to call an event handler when a gesture is recognized and to work with the final Kinect SDK 1.0., I used the original program to generate the gestures file which contains the information needed to detect gestures. I actually recorded two different gestures for each “SlideLeft” and “SlideRight” gesture just to make sure that different gestures done by different people would be caught by the algorithm.

The next step was to figure out how to animate the slides. I didn’t have experience with WPF animations but I arrived at this solution which works but might not be the best.

VisualStates and their transitions

I created 3 visual states to handle the slide left and slide right: previous (VisualState 1), current (VisualState 2) and next (VisualState 3). There are 2 visual transitions (left and right) between each 2 states and every time we change from one visual state to another we update the “next” target with new slides.

To automatize the transitions I decided to create a SlideManager that starts by reading all the “slides” (Grid elements) under a parent element and initializing the visual states by adding the event handlers. I created a simple structure to keep track of slides and transitions between slides because initially I wanted to include vertical transitions by ended up not doing them.

// Initialize slides.
for (int i = 0; i < mainSlideContainer.Children.Count; i++)
{
	Grid slide = mainSlideContainer.Children[i] as Grid;
	if(slide == null)
		continue;

	Slide s = new Slide()
				            {
				                Name = slide.Name,
				                GUIElement = slide,
				            };

	m_slides.Add(s);
}

for (int i = 0; i < m_slides.Count; i++)
{
	Slide nSlide = null;
	Slide pSlide = null;

	if (i < m_slides.Count - 1)
		nSlide = m_slides[i + 1];
 	if (i > 0)
		pSlide = m_slides[i];

	Slide s = m_slides[i];
	s.Transitions = new Dictionary();
	if(nSlide != null)
		s.Transitions.Add(Transition.RightToLeft, nSlide);
	if (pSlide != null)
		s.Transitions.Add(Transition.LeftToRight, pSlide);
}

There are two other important methods that change slides: MoveToNextSlide() and MoveToPrevSlide(). Each method checks if the current slide has the transition we want to perform, updates the current slide and visual state attribute members and calls the method that actually performs the transition.

public void MoveToNextSlide()
{
	if(m_inTransition)
		return;

	Slide nextSlide;
	if (m_curSlide.Transitions.TryGetValue(Transition.RightToLeft, out nextSlide))
	{
		m_prevSlideIndex = m_curSlideIndex;

		m_curSlideIndex = Math.Min(m_curSlideIndex + 1, m_slides.Count - 1);
		m_curSlide = m_slides[m_curSlideIndex];
		m_curVSIndex = (m_curVSIndex + 1) % m_visualStates.Count;
		m_curVS = m_visualStates[m_curVSIndex];

		m_inTransition = true;
		ExtendedVisualStateManager.GoToElementState(m_vsManagerContainer, m_curVS.Name, true);
	}
}

Another part of the code that is worth mentioning is the two event handlers at the end of this manager: Storyboard_Completed and VSGCurrentStateChanged.

private void Storyboard_Completed(object sender, EventArgs e)
{
	m_slides[m_curSlideIndex].GUIElement.Opacity = 1;
	m_slides[m_curSlideIndex].GUIElement.Visibility = Visibility.Visible;

	if (m_prevSlideIndex != m_curSlideIndex)
	{
		m_slides[m_prevSlideIndex].GUIElement.Opacity = 0;
		m_slides[m_prevSlideIndex].GUIElement.Visibility = Visibility.Collapsed;
	}
}

The visual state transitions do the slide movement and the storyboard controls the opacity and visibility of the element in that state. Whenever we change from one state to the other we have to reset the opacity and visibility properties so that the next time we do the transitions the new slides have the correct properties.

private void VSGCurrentStateChanged(object sender, VisualStateChangedEventArgs e)
{
	VisualState nvs = m_curVS;

	// update the next visual state time line (only if necessary)
	if (m_curSlideIndex < m_slides.Count - 1)
	{
		Timeline tl = FindObjectByName(nvs.Storyboard.Children, nvs.Name + "_next" + TLOpacName);
		if (tl != null)
			tl.SetValue(Storyboard.TargetNameProperty, m_slides[m_curSlideIndex + 1].Name);

		tl = FindObjectByName(nvs.Storyboard.Children, nvs.Name + "_next" + TLVisName);
		if (tl != null)
			tl.SetValue(Storyboard.TargetNameProperty, m_slides[m_curSlideIndex + 1].Name);
	}

	m_inTransition = false;
}

The same goes for the storyboard target name property, whenever we change to a new state we have to update the next slide. I also used a flag to prevent it from starting a new transition while we’re still doing the animation from the current transition.

The KinectManager is a simple class to handle the Kinect’s initialization that I adapted from the samples. It’s far from complete as it is but it serves well for this tutorial.

You may ask yourself why go through all this trouble when there is PowerPoint and Kinect applications to do the slide animation controls? Firstly, because it is fun to explore and learn new things and secondly because there are parts of the complete interactive portfolio that weren’t so easy to do in PowerPoint such as the particle system or the pong game.

Note: The transitions between texts were programmed in a similar way but instead of using 3 Visual States I only used 2, one for display the text and the other and to hide it, then it’s just a matter of setting the right text when you do the transitions.

Download
Kinect Slide Tutorial (requires: .Net4, Kinect SDK 1.0, VS2010 and up)

Tagged , , , , . Bookmark the permalink.

11 Responses to Kinect Slide Tutorial

  1. Barney says:

    I could not make it work. I connect it to Kinect. When I push F5 in Visual Studio, presentation showed up. I can change frames with left or right button. However, gestures could not work. 🙁 Can you help me?

    • n0n4m3 says:

      I recorded two different gestures so that the system could detect more gestures. Apparently you must be doing a gesture that the system can not recognize.
      Also check that the Kinect sensor is seeing you (at least from your waist to the head) so it can view you do the gestures.

      • Barney says:

        I checked the gesture.txt and there are six different gestures. However, no one is working. I am sure my Kinect device is up because it is working with Kinect SDK Samples. The only problem may be the references. Because when I open the project Microsoft.Expression.Effects and Microsoft.Expression.Interactions dlls have some problems. I upload them from your Interactive portfolio dlls. I hope they are working fine because project is starting and water images showed up. Moreover, your Interactive portfolio does not recognize me when I wave my hand. It is giving the error like Kinect does not recognize you when I get so close to Kinect. What do you think the problem is?
        Thanks.

        • n0n4m3 says:

          Sorry for the delay in my answer.

          Yes, there are 6 gestures because I used the same file from the portfolio which had:
          2 – left hand swipe right
          2 – right hand swipe left
          1 – wave right hand
          1 – wave left hand

          But in the tutorial only the swipes are being processed in the code.
          If you have yellow question marks on the references then it could be a problem.

          If you can’t make it work try the Kinect DTW project which allows you to record your own gestures.

    • jilani says:

      Hi, Im also having the same problem as u had. let me know how u solved it.
      Thanks.

    • n0n4m3 says:

      Hi Barney,
      I noticed that there was a problem with the sample, I uploaded the new version. It should work now.

      I’m sorry about that.

  2. n0n4m3 says:

    You’re welcome! 🙂
    Let me know if you solved your problems.

  3. jilani says:

    Hi n0n4m3,
    im new to the kinect sdk and i happened to look at your slide tutorial. I tried to run it but when i make gestures nothing happens. i dont have anything problems with references. if possible can i have a look at your portfolio demo sourcecode.
    Thanks.

    • jilani says:

      hi, all day i tried to debug and couldn’t able to find where i may be going wrong in ur demo. please let me know if ur demo works in ur pc and works with gestures. just want to know where im going wrong or the source code is not completely finished.
      Thanks

      • n0n4m3 says:

        Hi Jilani,
        I just tested here on my PC and saw that there was a problem hooking up the kinect events, I uploaded the new version. Does it work correctly now?

        Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *