How to Integrate Intel® Perceptual Computing SDK with Cocos2D-x

Downloads

How to Integrate Intel® Perceptual Computing SDK with Cocos2D-x [PDF 482KB]

Introduction

In this article, we will explain the project we worked on as part of the Intel® Perceptual Computing Challenge Brazil, where we managed to achieve 7th place. Our project was Badaboom, a rhythm game set in the Dinosaur Era where the player controls a caveman, named Obo, by hitting bongos at the right time. If you’re curious to see the game in action, check out our video of Badaboom:

​

To begin, you’ll need to understand a bit about Cocos2D-X, an open-source game engine that is widely used to create games for iPhone* and Android*. The good thing about Cocos2D-X is that it is cross-platform and thus is used to create apps for Windows* Phone, Windows 8, Win32*, Linux*, Mac*, and almost any platform you can think of. For more information, go to www.cocos2dx.org.

We will be using the C++ version of the SDK (Version 9302) as well as the Cocos2D-X v2.2 (specifically the Win32 build with Visual Studio* 2012). Following the default pattern of Cocos2D, we will create a wrapper that receives and processes the data from the Creative* Interactive Gesture Camera and interprets it as “touch” for our game.

Setting the environment

To start, you’ll need to create a simple Cocos2D project. We will not cover this subject as it is not the focus of our article. If you need more information, you can find it on the Cocos2D wiki (www.cocos2dx.org/wiki).

To keep it simple, execute the Python* script to create a new project in the “tools” folder of Cocos2d-x and open the Visual Studio project. Now we will add the Intel Perceptual Computing SDK to the project.

To handle the SDK’s input, we will create a singleton class named CameraManager. This class starts the camera, updates the cycle, and adds two images to the screen that represent the position of the hands on the game windows.

CameraManager is a singleton class that is derived from UtilPipeline and imports the “util_pipeline.h” file. Here, we need to reconfigure some of the Visual Studio project properties. Figure 1 shows how to add the additional include directories for the Intel Perceptual Computing SDK.

   $(PCSDK_DIR)/include
   $(PCSDK_DIR)/sample/common/include
   $(PCSDK_DIR)/sample/common/res

Figure 1. Additional include directories

You must also include the following paths to the additional library directories:

   $(PCSDK_DIR)/lib/$(PlatformName)
   $(PCSDK_DIR)/sample/common/lib/$(PlatformName)/$(PlatformToolset)

Figure 2. Additional Library Directories

Add the following dependencies in the input section:

   libpxc_d.lib
   libpxcutils_d.lib

Figure 3. Additional Dependencies

Now we are ready to work on our CameraManager!

Start Coding!

First we need to make the class a singleton. In other words, the class needs to be accessible from anywhere in the code with the same instance (singleton classes have only one instance). For this, you can use a method:

CameraManager* CameraManager::getInstance(void)
{
    if (!s_Instance)
    {
        s_Instance = new CameraManager();
    }

    return s_Instance;
}

After that, we’ll build a constructor, a method that starts the camera:

CameraManager::CameraManager(void)
{
	if (!this->IsImageFrame()){
		this->EnableGesture();

		if (!this->Init()){
			CCLOG("Init Failed");
		}
	}

	this->hand1sprite = NULL;
	this->hand2sprite = NULL;

	hasClickedHand1 = false;
	hasClickedHand2 = false;

	this->inputAreas = CCArray::createWithCapacity(50);
	this->inputAreas->retain();
}

Many of the commands initialize variables that handle sprites which symbolize the users’ hands and get input as they close their hands. The next step is processing the data that comes from the camera.

void CameraManager::processGestures(PXCGesture *gesture){
	
	PXCGesture::Gesture gestures[2]={0};

	gesture->QueryGestureData(0,PXCGesture::GeoNode::LABEL_BODY_HAND_PRIMARY,0,&gestures[0]);
	gesture->QueryGestureData(0,PXCGesture::GeoNode::LABEL_BODY_HAND_SECONDARY,0,&gestures[1]);

	
	CCEGLView* eglView = CCEGLView::sharedOpenGLView();
	switch (gestures[0].label)
	{
	case (PXCGesture::Gesture::LABEL_POSE_THUMB_DOWN):
		CCDirector::sharedDirector()->end();
		break;
	case (PXCGesture::Gesture::LABEL_NAV_SWIPE_LEFT):
		CCDirector::sharedDirector()->popScene();
		break;
	}
}

To be clear, it is in this method that you can also add switch cases to understand voice commands and to implement more gesture handlers. Following this, we must process this information and display it in the CCLayer (Cocos2D sprite layer).

bool CameraManager::Start(CCNode* parent){
	this->parent = parent;

	if (this->hand1sprite!=NULL
    &&  this->hand1sprite->getParent()!=NULL){
		this->hand1sprite->removeFromParentAndCleanup(true);
		this->hand2sprite->removeFromParentAndCleanup(true);
	}

	this->hand1sprite = CCSprite::create("/Images/hand.png");
	this->hand1sprite->setOpacity(150);
    //To make it out of screen
	this->hand1sprite->setPosition(ccp(-1000,-1000));
	this->hand1Pos = ccp(-1000,-1000);
	
	this->hand2sprite = CCSprite::create("/Images/hand.png");
	this->hand2sprite->setFlipX(true);
	this->hand2sprite->setOpacity(150);
	this->hand2sprite->setPosition(ccp(-1000,-1000));
	this->hand2Pos = ccp(-1000,-1000);

	parent->addChild(this->hand1sprite, 1000);
	parent->addChild(this->hand2sprite, 1000);
	
	this->inputAreas->removeAllObjects();
	return true;
}

This method should be called each time a new frame is placed on the screen (most of the time into the onEnter callback). It will automatically remove the hand sprites from the previous parent and add them to the new CCLayer.

Now that our hand sprites have been added to the CCLayer we are able to handle their position by calling the follow method on the update cycle of the CCLayer (which is scheduled by the call: “this->scheduleUpdate();”). The update method is as follows:

void CameraManager::update(float dt){

	if (!this->AcquireFrame(true)) return;

	PXCGesture *gesture=this->QueryGesture();
	
	this->processGestures(gesture);

	PXCGesture::GeoNode nodes[2][1]={0};
	
    gesture->> QueryNodeData(0,PXCGesture::GeoNode::LABEL_BODY_HAND_PRIMARY,1,nodes[0]);
	gesture-> QueryNodeData(0,PXCGesture::GeoNode::LABEL_BODY_HAND_SECONDARY,1,nodes[1]);

	CCSize _screenSize = CCDirector::sharedDirector()->getWinSize();

	
	if (nodes[0][0].openness<20 && !this->hand1Close){
		this->hand1sprite->removeFromParentAndCleanup(true);
		this->hand1sprite = CCSprite::create("/Images/hand_close.png");
		this->hand1sprite->setOpacity(150);
		this->parent->addChild(hand1sprite);
		this->hand1Close = true;
	} else if (nodes[0][0].openness>30 && this->hand1Close) {
		this->hand1sprite->removeFromParentAndCleanup(true);
		this->hand1sprite = CCSprite::create("/Images/hand.png");
		this->hand1sprite->setOpacity(150);
		this->parent->addChild(hand1sprite);
		this->hand1Close = false;
	}
	
	if (nodes[1][0].openness<20 && !this->hand2Close){
		this->hand2sprite->removeFromParentAndCleanup(true);
		this->hand2sprite = CCSprite::create("/Images/hand_close.png");
		this->hand2sprite->setFlipX(true);
		this->hand2sprite->setOpacity(150);
		this->parent->addChild(hand2sprite);
		this->hand2Close = true;
	} else if (nodes[1][0].openness>30 && this->hand2Close) {
		this->hand2sprite->removeFromParentAndCleanup(true);
		this->hand2sprite = CCSprite::create("/Images/hand.png");
		this->hand2sprite->setFlipX(true);
		this->hand2sprite->setOpacity(150);
		this->parent->addChild(hand2sprite);
		this->hand2Close = false;
	}

	this->hand1Pos = ccp(_screenSize.width*1.5-nodes[0][0].positionImage.x*(_screenSize.width*HAND_PRECISION/320) + 100,
						 _screenSize.height*1.5-nodes[0][0].positionImage.y*(_screenSize.height*HAND_PRECISION/240));
	this->hand2Pos = ccp(_screenSize.width*1.5-nodes[1][0].positionImage.x*(_screenSize.width*HAND_PRECISION/320) - 100,
						 _screenSize.height*1.5-nodes[1][0].positionImage.y*(_screenSize.height*HAND_PRECISION/240));

	if (!hand1sprite->getParent() || !hand2sprite->getParent()){
		return;
	}
	this->hand1sprite->setPosition(this->hand1Pos);
	this->hand2sprite->setPosition(this->hand2Pos);

	
    CCObject* it = NULL;
	CCARRAY_FOREACH(this->inputAreas, it)
	{
		InputAreaObject* area = dynamic_cast<InputAreaObject*>(it);
		this->checkActionArea(area->objPos, area->radius, area->sender, area->method);
	}
			
	this->ReleaseFrame();

}

This code not only handles the position of the sprite, it also sets a different sprite (hand_close.png) if the camera detects that the hand is less than 20% open. In addition to this, there is simple logic to create hand precision, which makes the user input more sensitive and easier to get the edges of the screen. We do this because the Perceptual Camera is not that precise on the edges, and the position of the sprites commonly get crazy when we approach the edge.

Now it is indispensable that we add some ways to handle the input (a closed hand is considered a touch). We need to write a method called “checkActionArea” (called in the update method) and register the actionArea.

void CameraManager::checkActionArea(CCPoint objPos, float radius, CCObject* sender, SEL_CallFuncO methodToCall){

	if (sender==NULL)
		sender = this->parent;

	float distanceTargetToHand = ccpDistance(this->hand1Pos, objPos);
	if (distanceTargetToHand<radius){
		if (this->hand1Close&& !hasClickedHand1){

			this->parent->runAction(CCCallFuncO::create(this->parent, methodToCall, sender));
			hasClickedHand1 = true;
		}
	}
	
	if (!this->hand1Close){
		hasClickedHand1 = false;
	} //TODO: repeat for hand2
}

Follow the method registerActionArea() for the registration of areas:

void CameraManager::registerActionArea(CCPoint objPos, float radius, cocos2d::SEL_CallFuncO methodToCall){

	InputAreaObject* newInputArea = new InputAreaObject(objPos, radius, methodToCall);
	this->inputAreas->addObject(newInputArea);
}

Now it is easy to add the Intel Perceptual Computing SDK to your Cocos2D game!!! Just run:

CameraManager::getInstance()->Start(this);

When entering the Layer, register the objects and methods to be called:

CameraManager::getInstance()->registerActionArea(btn_exit->getPosition(), 150, callfuncO_selector(LevelSelectionScene::backClicked));

About us!

Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
Copyright © 2013 Intel Corporation. All rights reserved.
*Other names and brands may be claimed as the property of others.