SUBSIM Radio Room Forums - [TEC] Mouse input

For some time now, I was wondering if I am able to do some modding for SH5. Since currently my time capacity (nevermind any artistic talent, or rather lack of it) does not allow me to work on something of considerable size, I've decided to start small, with tools.

In one of his posts in NewUIs thread, TheDarkWraith stated that mouse input in SH5 is a mess, and I totally agree with that statement. The whole UI scripting is cumbersome and documentation is, well, lacking. Nevertheless, I always wanted to have in-game tools such as attack disc or RAOBF behave just as they did in, say, SH3: i.e. so that I could click my mouse and rotate those rings to my hearts content. And to be honest, my middle finger just got tired of rotating the mouse wheel when I'm using NewUIs :) Not to mention that my setup periodically lags, and from what I can tell (or at least suspect), this causes TDW's mouse polling thread to go 'out of sync' with what I actually see on screen. This brought me some problems in, for example, placing supermarks on the nav map.

Interestingly enough, API provided for UI elements allows adding drag/resize handles for items, but these are just not enough for situations when I would have a complex object, such as RAOBF. Here, I want to share what I was able to find out.

So, I started to dig around and see how exactly point-click-drag can be implemented. Well, basics are obvious: I just need to periodically poll mouse state with that 'animation' timer whilch I'm sure is familiar to most modders. Thus I can emulate mouse motion events and handle dragging. So, I've quickly coded a small class (in C#) that can help with this task. It's called InputCanvas, and it's only purpose is to emulate mouse motion events for FocusMenuItemWrapper, the main UI element class that can receive mouse input (non-focusable elements receive only mouse-in and mouse-out events). Incidentally, it also forwards mouse press/release/click events, just for ease of coding Python scripts.

But then things get trickier. Say I want to implement front side of the attack disc, with all 4 elements draggable by mouse. This means that I'll have 4 overlaying images (not counting outer relative bearing scale, which is static), for which I need to implement rotation via dragging. And of course these images are not just rectangles, they are mostly circles with customary pointers attached to them. But by default, game treats the whole image as an input area, therefore I need some way to determine which image I actually click: in case of attack disc, images are positioned 'on top' of one another, and though some areas of top-level image are transparent, they would still receive mouse input (though I visually click on the image 'below'). Well, everybody saw the attack disc :). I knew custom shapes for UI elements could be done, since even stock game provides round and triangle-shaped buttons (though I must say, for a while I was afraid they are hard-coded), I just needed to find out how.

While I was familiarizing myself with API, for several times I encountered a mention of 'touchpads'. As it turned out, you can specify the shape of input area for MenuItemWrapper by adding those touchpads. Once at least one touchpad is added, the game stops treating UI element as a rectangle, and processes input only in allowed area.
There are two main types of touchpads. One allows input (MenuItemWrapper.InsideTestModes.Inside) and one denies it (MenuItemWrapper.InsideTestModes.NotInside). As far as I can tell, you can add any number of touchpads to the item, and game will combine them all to determine if mouse input is actually allowed for a particular area of the item.
There are three possible 'shapes' for a touchpad: an ellipse (ideal for circular dials and buttons), a triangle (e.g. TAI maximize/minimize button) and an arbitrary polygon (which I think game just splits up to multiple triangles). I made a quick test with ellipses (attached one to each of attack disc parts), and my attack disc suddenly started to respond to dragging its elements. It was first 'hurray' for me. Though before that, I had a small issue in determining what do parameters for AddTouchPadEllipse() actually mean, but more on that later. Anyway, some more quick tests quickly showed me that I can actually define any custom shape for the input area: it's only a matter of determining the coordinates for touchpads.

And so I patiently calculated all the coordinates for each pointer on the attack disc, added a corresponding touchpad polygon, started my game and... well, was surprised. For some reason, game refused to react on my attempts to drag a pointer. And the Attack Course pointer started to behave very strange when I rotated it (when it was horizontal I had a hard time finding a place which allowed me to drag it). Something was definetly going on. Luckily, I found out what rather quickly.

Documentation states that changing Rotation property for MenuItemWrapper (i.e. rotating it) also rotates all touchpads. What it doesn't say is that rotations are confined to item's bounding box. If it is rectangular, polygons start to shrink or extend when rotated. And several images from my attack disc (I use the one from TheDarkWraith's NewUIs) are indeed rectangular. Well, I thought OK, how about I put each disc in a separate control group, make these groups square and rotate them instead. Unfortunately, the game refused me this attempt, stating that it cannot rotate items without a texture (and control groups are just that). So I had to resize all the textures to make them square, recalculate my touchpads and try again. And it worked: all the discs rotated correctly. This was the second 'hurray'. The only issue I have with this is that I had to make images larger, therefore now they consume more VRAM. Though I don't know if it can become a significant problem, considering the overall ratio of UI to game world texture usage (and anyway, square images are only needed if you intend to rotate them).

I promised to tell about parameters for funcions that add touchpads. Here goes:

For all functions, IfInside parameter sets the type of the touchpad: including (allows input) or excluding (denies input)

Code:

AddTouchPadEllipse( IfInside, CenterXPercent, CenterYPercent, WidthPercent, HeightPercent )

Coordinates are in the range [0,1] (hence the 'Percent') suffix. Width and height are ellipse radiuses. But there is a quirk with them. Initially, my thought process was like this: if the full range of coordinates is 1, then to make an incircle for the square image, I'd have to place center at (0.5, 0.5) and make 'width' and 'height' equal to half image width, thus, also (0.5, 0.5). Well, I was wrong. It turns out, that the game considers maximum values for width and height parameters as halves of actual image width and height. Thus, for 200 pixels-wide image (incircle should have a radius of 100 pixels), WidthPercent 0.5 would mean 0.5*0.5*200 = 50! Therefore, to make an actual incircle, parameters should be:

CenterXPercent = 0.5
CenterYPercent = 0.5
WidthPercent = 1.0
HeightPercent = 1.0

This is important to remember when calculating actual size for this type of touchpad.

Code:

AddTouchPadTriangle( IfInside, x0, y0, x1, y1, x2, y2 )

AddTouchPadPolygon( IfInside, Single[] coords )

These two are pretty straightforward: you just specify a list of 2D vertices (x,y pairs), where each coordinate is in range [0,1]. One thing to note is that 0 for x is left, 0 for y is top, no matter what offsets you specified while placing an image on page (i.e. these are local coordinates).

One last thing to note about touchpads: AFAICT, they can only be added, but not removed, and they are not removed when page's script is unloaded. This is a bit unfortunate, but I personally can live with that.

To summarize: while somewhat problematic, it's actually not that hard to implement custom-shaped widgets. With taking several quirks I mentioned into account, it becomes only a matter of calculating actual touchpad areas. This is my second step, as I'm now starting to work on application that will allow to manually "draw" touchpad areas for textures and save them into text file, so that it can be read inside script during initialization. Another approach would be to use some form of edge detection and vectorization to calculate the input areas automagically, but that's beyond my abilities for now.

If anyone is interested, I've uploaded my InputCanvas class along with sample Python module. Maybe it'll help someone in creating another awesome mod. Simply place the dll into SH5 installation folder and use it like this:

Code:

import clr

clr.AddReference("RadCapTools")

from RadCapTools import InputCanvas

The documentation for InputCanvas is included in the archive.

Download here

I've recorded a short video to illustrate what all the above rant means.
Here's the link. I hope whoever would watch would also excuse my English :)

Cheers :salute:

Okay, for all UI modders who want to utilize the touchpads functionality right now, here's the how-to. Beware it's a bit technical!
Also note, the how-to does not use the sample python script provided in the download, for that wait for a further example.

Not really a step, just a note. Be prepared to do some calculations (as you don't want to wait till I finish my calculator application :D)
If you're creating/editing a rotatable control (attack disc, RAOBF, 3 bearings tool, etc.), make sure that it's texture is square. Having non-square textures will screw up your touchpads when you rotate it. Example:

http://img406.imageshack.us/img406/7348/imagerect.png http://img839.imageshack.us/img839/9153/imagesquare.png

On the left is the original rectangular image, on the right is the proper square one. Notice how I left the image itself centered.
Make sure that you synchronize your changes to image sizes with SH5 pages configuration (i.e. open up menu editor and enter proper values)
For each image, define a set of areas you want (or don't want) to click on. Remember there are three basic shapes to choose from (ellipse/circle, triangle and polygon).

Let's take the "Vorhalt" (lead angle) pointer from the attack disc. In this example, I'll mark the areas that I want to handle input with green, and those I don't want to handle input with red.
Note that while I use the same image that I used when created sample code, the numbers I provide here may differ slightly since I redid the whole procedure again.

This is the area I want to 'listen' to my mouse:

http://img6.imageshack.us/img6/1019/imageinput.png

I need to split up this area into simple shapes. I immediately see a circle:

http://img716.imageshack.us/img716/2667/imagecircle.png

Yes, I know that it covers the space where mouse should not be handled, but bear with me on that.
Now, I look at the pointer. It's a trapezoid:

http://img546.imageshack.us/img546/1...agepointer.png

Ok, so now I have defined two areas where I want my mouse input. Now for the areas where I don't want it. I want to keep everything as simple as possible, so:

I don't want my mouse handled in the lower half of the image:

http://img841.imageshack.us/img841/9...erectangle.png

I don't want my mouse handled in these two areas:

http://img840.imageshack.us/img840/6...etriangles.png

Note that I actually cheat, since there's also a small circle in center which also should not receive mouse input:

http://img163.imageshack.us/img163/3...mallcircle.png

But I won't bother with it, since I know that when I combine all images, it will always be covered by another object (the attack course pointer).

Using your image editor (I use GIMP) or any other application that allows you to see mouse coordinates, measure all these areas. For circles, we'll need center and radius (two radiuses for ellipses).
For anything else, we'll need corner points (vertices).

The whole image is 556x556 pixels. For the circle, it's center will be in the middle of image (278x278), and I measured it's radius to be ~151-152 pixels (I used GIMP's measure tool).
For pointer, I just found the coordinates of it's corners (yep, pointed my mouse at them and took GIMP's feedback). Here's an illustration of what I got:

http://img525.imageshack.us/img525/5...agecorners.png

Next, I traversed this polygon and wrote down the coordinates of these points as I encountered them:

http://img72.imageshack.us/img72/891...straversed.png

Next, I did the same measurements for the rectangle an the triangles. Here's what I got in the end:

Circle: center = 278,278, radius = 152
Pointer: 250,123 -> 270,0 -> 286,0 -> 305,123
Rectangle: 0,278 -> 556,278 -> 556,556 -> 0,556
Left Triangle: 278,278 -> 110,278 -> 137,212
Right Triangle: 278,278 -> 454,278 -> 418,212

Okay, you've wrote down all those coordinates. Now you need to convert them into relative coordinates in the range [0,1]. Those familiar with vector graphics know what I mean, for everyone else here's what to do:
take each point, and divide X coordinate by image width and Y coordinate by image height. For my pointer, I get the following:

250,123 -> 270,0 -> 278,0 -> 305,123 divide each X by image width (556) and each Y by image height (also 556):

0.449,0.221 -> 0.485,0 -> 0.514,0 -> 0.548,0.221

You don't need to take all the significant digits, in fact 2 or 3 will be quite enough. Repeat this for each polygon you've got.

Now, with circles... stop. If you haven't read my original post, you don't know that circles are funky. For center, divide the coordinates as usual by image size. But for radius, divide by half image size!
E.g. for my circle I will divide center coordinates by 556, but radius by 278. If you've got ellipse and not a circle, you'll need to divide X radius by half width, and Y radius by half height.

My numbers after the calculations:
Circle: center = 0.5,0.5, radius = 0.546
Pointer: 0.449,0.221 -> 0.485,0 -> 0.514,0 -> 0.548,0.221
Rectangle: 0,0.5 -> 1,0.5 -> 1,1 -> 0,1
Left Triangle: 0.5,0.5 -> 0.197,0.5 -> 0.246,0.381
Right Triangle: 0.5,0.5 -> 0.816,0.5 -> 0.751,0.381

Tedious? Well, it's a lot more in writing than in doing. And anyway, wait for my calculator app. You'll just draw all those areas instead! :)

I must say it would have been a LOT easier if we could just create a mask texture that would show exactly what parts of image are processing input (and I have an itching suspicion that stock game dials do just that).
Code

In your page script, import the InputCanvas class:

Code:

import clr clr.AddReference("RadCapTools") from RadCapTools import InputCanvas

Add a global for InputCanvas so you can access it from different functions:

Code:

BearingLeadPointer = None

In your InitializeScript() function, add the touchpad areas to your items:

Code:

def InitializeScript(): if Page_Test1_AttackDisc_Front_BearingLeadAngle.Touchpads.Length == 0: # Adding areas that process mouse input (MenuItemWrapper.InsideTestModes.Inside) # Circle: Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadEllipse( MenuItemWrapper.InsideTestModes.Inside, 0.5, 0.5, 0.546, 0.546 ) # Pointer: Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadPolygon( MenuItemWrapper.InsideTestModes.Inside, 0.449,0.221, 0.485,0.0, 0.514,0.0, 0.548,0.221 ) # Adding areas that don't process mouse input (MenuItemWrapper.InsideTestModes.NotInside) # Rectangle: Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadPolygon( MenuItemWrapper.InsideTestModes.NotInside, 0.0,0.5, 1.0,0.5, 1.0,1.0, 0.0,1.0 ) # Left Triangle: Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadTriangle( MenuItemWrapper.InsideTestModes.NotInside, 0.5,0.5, 0.197,0.5, 0.246,0.381 ) # Right Triangle: Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadTriangle( MenuItemWrapper.InsideTestModes.NotInside, 0.5,0.5, 0.816,0.5, 0.751,0.381 )

Here, one at a time, I add all the areas that I've previously calculated, specifying whether I want mouse input in them or not.

Why is that Page_Test1_AttackDisc_Front_BearingLeadAngle.Touch pads.Length == 0 check there? Well, the touchpads may only be added, not removed. At least I haven't found a way to do that. But InitializeScript() function will be called more
than once during gameplay. This means that (a) we need some check to only add touchpads once and (b) if you make changes to your touchpads during testing, you'll have to reload the game. Oh well.

Now, in that same function, create an InputCanvas and attach some event handlers to it:

Code:

# InitializeScript() continued global BearingLeadPointer BearingLeadPointer = InputCanvas(Page_Test1_AttackDisc_Front_BearingLeadAngle) # BearingLeadPointer.LostFocus += This_LostFocus BearingLeadPointer.MouseLeftButtonDown += This_MouseLeftButtonDown BearingLeadPointer.MouseLeftButtonUp += This_MouseLeftButtonUp BearingLeadPointer.MouseMoved += This_MouseMoved

Here, I've basically created my mouse polling object and told it what functions it should call when certain events happen. Now for those who haven't seen the video I've posted, I'll repeat that you can track mouse position for the control even when mouse is actually outside it. If TheDarkWraith is reading this, I'm sure he's laughing at this point, because programmatically, there's no restriction whatsoever if the mouse is inside or not, we can just poll it at any given time. What I mean is for our purposes, we need some way to determine whether mouse position is still relevant for our UI item. That's why there are two separate types of events for stock API objects: mouse in/out events and focus events. An item gains focus whenever mouse is moved over it (so MouseIn and GotFocus event arrive in pair). Now if you press and hold mouse item inside an item and drag mouse out of it, it'll still have mouse focus, though it already got a MouseOut event. That's why I add a LostFocus handler here: I want to know when to actually stop waiting for mouse input (i.e stop dragging). The InputCanvas class basically forwards events form FocusMenuItemWrappers it handles, it's usefulness is in that MouseMoved event which is not present for the stock API objects.

In UnloadScript() function, add code to dispose of InputCanvas:

Code:

def UnloadScript(): global BearingLeadPointer if BearingLeadPointer: BearingLeadPointer.Dispose() BearingLeadPointer = None

The only thing left is to define those event handler functions. Here's an example with comments:

Code:

Dragging = False # This will tell us whether we drag our pointer def This_LostFocus( sender ): global Dragging Dragging = False # When our pointer lost input focus, stop dragging def This_MouseLeftButtonDown( sender, point ): global Dragging Dragging = True # When mouse button is pressed, start dragging def This_MouseLeftButtonUp( sender, point ): global Dragging Dragging = False # When mouse button is released, stop dragging def This_MouseMoved( sender, newPos, oldPos ): global Dragging if not Dragging: return # we're not dragging, so we don't want to do anything # Below is simple code to apply rotation based on mouse motion # It's the same code found in sample python module # Note that it uses mouse displacement to calculate rotation angle # You can easily use just current mouse position and calculate # absolute rotation angle # We need to convert coordinates since InputCanvas gives them in screen space # previous mouse position relative to center v0 = sender.screenToClient( oldPos, MenuItemWrapper.LocationPresets.Center ) # current mouse position relative to center v1 = sender.screenToClient( newPos, MenuItemWrapper.LocationPresets.Center ) # rotate one of the discs delta = atan2(v1.Y, v1.X) - atan2(v0.Y, v0.X) # atan2 gives proper sign, so just add the delta to current rotation sender.Rotation = sender.Rotation + delta

That's pretty much it.

Now you may ask, then what the hell this AttackDisc.py in sample downloads is for? Well, this how-to I built from scratch. And that script I extracted from my actual testing sandbox. It's basically just a more
sophisticated version of code presented in this how-to, and it handles the full attack disc. If you look carefully at that code, you'll find that it's pretty similar to the one presented here, it only differs in function signatures (since AttackDisc is a class there) and in amount of InputCanvas objects.

I put in the sample just what I exctracted into one neat class so that I don't clutter my own functions with it. In fact,
I already figured out how to actually use it as a separate script file (i.e. scripts/menu/radcapricorn/AttackDisk.py) without creating any dummy pages, but that's another topic.
If I get permission from TheDarkWraith to post changes to his scripts, I'll show you how you can use that sample python module to actually control attack disc in TDW's UI. I've actually already tested that code with his UI (since I already have touchpad areas calculated for that attack disc) and I must say it works perfectly.

For any other objects (RAOBFs, 3-bearing tools and whatnot) you'll have to wait until I've finished my calculator app: I'm done with manual calculations :)

As a side note, you don't need to use my InputCanvas if you just want to create shaped buttons. Touchpads are enough for that. See how triangle buttons are done in stock Page Default Hud.py, for example.
You can use the same approach to make round buttons, hexagonal ones, you name it. There are limitations, of course, but I'll post them once I've tracked down at least all major ones.

Wow, that took a lot more time than I expected. But that's not all. Stay tuned :)

Have fun!