Coding for Gestures and Flicks on Tablet PC running Vista
LINK: http://www.therobotgeek.net/articles/gestureblocks.aspxFor this article, I wanted to demonstrate how to access gestures and flicks using C# code. I'll talk about accessing flicks using the tablet API and accessing gestures using the N-Trig native libraries. Surprisingly I couldn't find much out there that goes into any kind of depth in coding for either of these topics. So it seemed like a good opportunity to present some examples. I've included code for a C# game that uses the gestures for control. I hope you find it fun and useful.
Even though they are called "flicks", a flick is really a single touch gesture and an N-Trig gesture is a double touch gesture. All the examples run on Windows Vista Ultimate 32bit. Windows Vista will only detect up to two fingers on the touch surface. However, Windows 7 promises to allow more than two finger detection. I'll have to try that next. HP is already shipping hardware that is multi-touch enabled on their TouchSmart PCs. The Dell Latitude XT and XT2 tablet PCs are also using the N-Trig hardware for multi-touch.
Code to Detect If Running on a Tablet PC
I found some simple code to determine if your app is running on a tablet PC. It is a native call to GetSystemMetrics which is in User32.dll and you pass in SM_TABLETPC. I've included this code in the Flicks class. You may want your app to auto-detect if it's running on a tablet to determine which features to enable.
[DllImport("User32.dll", CharSet = CharSet.Auto)]
public static extern int GetSystemMetrics(int nIndex);
public static bool IsTabletPC()
{
int nResult = GetSystemMetrics(SM_TABLETPC);
if (nResult != 0)
return true;
return false;
}
Accessing Flicks (Single Touch Gestures) or UniGesture
If you've never heard of flicks before, they are single touch gestures that are triggered based on the speed of the touch across the tablet surface. You can find more detail here. It currently gives you eight directions that can be programmed at the OS level for use by applications.
I used the tabflicks.h file that I found in the folder - "C:\Program Files\Microsoft SDKs\Windows\v6.0A\Include" This header file is included with the Windows SDK. It contains all the flick structures and macro definitions needed to access flicks. I created a class library project called FlickLib that contains the C# structures, DllImports and code to store and process the data. I also wrote a test WinForm application that shows the usage of the library.
97
98 /// <summary>
99 /// Override the WndProc method to capture the tablet flick messages
100 /// </summary>
101 /// <param name="msg"></param>
102 protected override void WndProc(ref Message msg)
103 {
104 try
105 {
106 if (msg.Msg == Flicks.WM_TABLET_FLICK)
107 {
108 Flick flick = Flicks.ProcessMessage(msg);
109 if (flick != null)
110 {
111 this.txtOutput.Text = string.Format("Point=({0},{1}) {2}\r\n",
112 flick.Point.X,
113 flick.Point.Y,
114 flick.Data.Direction);
115
116 // Set to zero to tell WndProc that message has been handled
117 msg.Result = IntPtr.Zero;
118 }
119 }
120
121 base.WndProc(ref msg);
122 }
123 catch (Exception ex)
124 {
125 this.txtOutput.Text += string.Format("{0}\r\n", ex.Message);
126 }
127 }
128
I override the WndProc method of the WinForm to gain access to the WM_TABLET_FLICK messages sent to the window. The flick data is encoded in the WParam and the LParam of the windows message. I discovered that even though the params are IntPtr type, they are not actually pointers but are the real integer values to be used. I kept getting an Access Violation exception when I was trying to marshal the IntPtr to a structure. The solution was non-obvious. I couldn't find that documented anywhere. But it works correctly now so this is my contribution to the web for anyone trying to solve the same problem.
It is important to note that on line # 117, I set msg.Result = IntPtr.Zero. This tells the base.WndProc method that the message has been handled so ignore it. This is important because the underlying pen flicks can be assigned to actions such as copy and paste. If the message is not flagged as handled, the app will attempt to perform the action.
WinForm Flicks Test C# source code WinFlickTest.zipAccessing Double Touch Gestures or DuoGestures
The N-Trig hardware on the HP TouchSmart tablet provides the ability to detect two (or more) touches on the surface. This allows the tablet to respond on two finger gestures. You can see the current DuoGestures in use from N-Trig here.
I've created a class library project called GestureLib which wraps the native libraries from N-Trig so that they can be called using C#. Obviously this will only run on tablets that have the N-Trig hardware. Currently there is the HP TouchSmart tx2 and the Dell Latitude XT & XT2 laptops. I'll be testing and updating code for the HP TouchSmart IQ505 panel that uses the NextWindow hardware. If someone has any other tablets, touch panels and/or UMPC devices, I'd be happy to work together to create a version for that as well.
private Gestures _myGestures = new Gestures();
private int NTR_WM_GESTURE = 0;
/// <summary>
/// Override OnLoad method to Connect to N-Trig hardware and
/// Register which gestures you want to receive messages for and
/// Register to receive the Window messages in this WinForm
/// </summary>
/// <param name="eventArgs"></param>
protected override void OnLoad(EventArgs eventArgs)
{
try
{
bool bIsConnected = this._myGestures.Connect();
if (bIsConnected == true)
{
this.txtOutput.Text += "Connected to DuoSense\r\n";
TNtrGestures regGestures = new TNtrGestures();
regGestures.ReceiveRotate = true;
regGestures.UseUserRotateSettings = true;
regGestures.ReceiveFingersDoubleTap = true;
regGestures.UseUserFingersDoubleTapSettings = true;
regGestures.ReceiveZoom = true;
regGestures.UseUserZoomSettings = true;
regGestures.ReceiveScroll = true;
regGestures.UseUserScrollSettings = true;
bool bRegister = this._myGestures.RegisterGestures(this.Handle, regGestures);
if (bRegister == true)
{
this.NTR_WM_GESTURE = Gestures.RegisterGestureWinMessage();
if (this.NTR_WM_GESTURE != 0)
this.txtOutput.Text += string.Format("NTR_WM_GESTURE = {0}\r\n", this.NTR_WM_GESTURE);
}
}
else
this.txtOutput.Text = "Error connecting to DuoSense\r\n";
}
catch (Exception ex)
{
this.txtOutput.Text += string.Format("{0}\r\n", ex.Message);
}
}
/// <summary>
/// Override OnClosing method to ensure that we Disconnect from N-Trig hardware
/// </summary>
/// <param name="cancelEventArgs"></param>
protected override void OnClosing(CancelEventArgs cancelEventArgs)
{
this._myGestures.Disconnect();
base.OnClosing(cancelEventArgs);
}
Once you look at the code you will note that you have to connect to the N-Trig hardware at the start of the app. I put that call in the OnLoad method. Once you have a handle to the hardware, then you will use that throughout the lifetime of the app. It is stored in the Gestures class. With the handle, you will register which gesture messages you want the window to receive. Once registered, you will start to see NTR_WM_GESTURE message showing up in the WndProc method. I created a ProcessMessage method in the Gestures class that will determine which gesutre type it is and extract the data for the particular gesture.
One thing to note is if you look at the DllImports for the NtrigISV.dll native library you will see that there is name mangling with the method calls. I had to use Dependency Walker to determine the actual names of the methods. So be warned that when N-Trig releases a new version of their DLL that this code will probably break.
N-Trig Gestures WinForm Test source code WinGestureTest.zipProblems with Marshalling Data from C++ to C#
I had an issue where I was only getting the Zoom messages. My friend, Ben Gavin, helped me solve the problem. It had to do with marshalling the data into the struct. The original C++ struct looks like this:
typedef struct _TNtrGestures
{
bool ReceiveZoom;
bool UseUserZoomSettings;
bool ReceiveScroll;
bool UseUserScrollSettings;
bool ReceiveFingersDoubleTap;
bool UseUserFingersDoubleTapSettings;
bool ReceiveRotate;
bool UseUserRotateSettings;
} TNtrGestures;
For my original port, I just did a LayoutKind.Sequential and set everything as a bool like below.
[StructLayout(LayoutKind.Sequential)]
public struct TNtrGestures
{
public bool ReceiveZoom;
public bool UseUserZoomSettings;
public bool ReceiveScroll;
public bool UseUserScrollSettings;
public bool ReceiveFingersDoubleTap;
public bool UseUserFingersDoubleTapSettings;
public bool ReceiveRotate;
public bool UseUserRotateSettings;
}
Trouble is that bool in C++ is one byte where as a bool in C# is an Int32 or four bytes. Now if the C++ struct was labeled with a BOOL, then it would have been four bytes. So the new declaration of the struct in C# looks like this:
[StructLayout(LayoutKind.Explicit)]
public struct TNtrGestures
{
[FieldOffset(0)]public bool ReceiveZoom;
[FieldOffset(1)]public bool UseUserZoomSettings;
[FieldOffset(2)]public bool ReceiveScroll;
[FieldOffset(3)]public bool UseUserScrollSettings;
[FieldOffset(4)]public bool ReceiveFingersDoubleTap;
[FieldOffset(5)]public bool UseUserFingersDoubleTapSettings;
[FieldOffset(6)]public bool ReceiveRotate;
[FieldOffset(7)]public bool UseUserRotateSettings;
}
Once I used the FieldOffsetAttribute to set each bool to one byte each, the Rotate messages started to work correctly. Thanks Ben...
WPF Gestures Example Code
WindowInteropHelper winHelper = new WindowInteropHelper(this);
HwndSource hwnd = HwndSource.FromHwnd(winHelper.Handle);
hwnd.AddHook(new HwndSourceHook(this.MessageProc));
Using the same GestureLib class library, I also created an app that shows how to capture the gestures using a WPF application. That was an interesting exercise because I had never gotten a Windows handle from a WPF app. I discovered the WinInteropHelper class that gets the underlying Windows handle. I also had to hook the MessageProc handler to get the messages. The WndProc method is not available to override in WPF.
WPF Gestures Test source code WpfGestureTest.zipGesture Blocks - C# Game using Gestures and Flicks
Just creating test apps is boring and I wanted to create a game example that uses the flicks and gestures for control. I chose to build a simple game that shows off some of the possibilities for touch control. I call the game Gesture Blocks. It will give you a good starting point on using gestures in games. Watch the video to see the gestures in action.
Gesture Blocks Game source code WinGestureBlocks.zipConclusion
I hope the code I provided was useful and helpful. I will try to update the examples as issues are resolved. I'm also going to work on some other game examples that continue to show what can be done with gestures. The next project will be to load Windows 7 onto the tablet and write some code for more than DuoGestures.