[TEC] Mouse input

[radcapricorn]

For some time now, I was wondering if I am able to do some modding for SH5. Since currently my time capacity (nevermind any artistic talent, or rather lack of it) does not allow me to work on something of considerable size, I've decided to start small, with tools.

In one of his posts in NewUIs thread, TheDarkWraith stated that mouse input in SH5 is a mess, and I totally agree with that statement. The whole UI scripting is cumbersome and documentation is, well, lacking. Nevertheless, I always wanted to have in-game tools such as attack disc or RAOBF behave just as they did in, say, SH3: i.e. so that I could click my mouse and rotate those rings to my hearts content. And to be honest, my middle finger just got tired of rotating the mouse wheel when I'm using NewUIs Not to mention that my setup periodically lags, and from what I can tell (or at least suspect), this causes TDW's mouse polling thread to go 'out of sync' with what I actually see on screen. This brought me some problems in, for example, placing supermarks on the nav map.

Interestingly enough, API provided for UI elements allows adding drag/resize handles for items, but these are just not enough for situations when I would have a complex object, such as RAOBF. Here, I want to share what I was able to find out.

So, I started to dig around and see how exactly point-click-drag can be implemented. Well, basics are obvious: I just need to periodically poll mouse state with that 'animation' timer whilch I'm sure is familiar to most modders. Thus I can emulate mouse motion events and handle dragging. So, I've quickly coded a small class (in C#) that can help with this task. It's called InputCanvas, and it's only purpose is to emulate mouse motion events for FocusMenuItemWrapper, the main UI element class that can receive mouse input (non-focusable elements receive only mouse-in and mouse-out events). Incidentally, it also forwards mouse press/release/click events, just for ease of coding Python scripts.

But then things get trickier. Say I want to implement front side of the attack disc, with all 4 elements draggable by mouse. This means that I'll have 4 overlaying images (not counting outer relative bearing scale, which is static), for which I need to implement rotation via dragging. And of course these images are not just rectangles, they are mostly circles with customary pointers attached to them. But by default, game treats the whole image as an input area, therefore I need some way to determine which image I actually click: in case of attack disc, images are positioned 'on top' of one another, and though some areas of top-level image are transparent, they would still receive mouse input (though I visually click on the image 'below'). Well, everybody saw the attack disc . I knew custom shapes for UI elements could be done, since even stock game provides round and triangle-shaped buttons (though I must say, for a while I was afraid they are hard-coded), I just needed to find out how.

While I was familiarizing myself with API, for several times I encountered a mention of 'touchpads'. As it turned out, you can specify the shape of input area for MenuItemWrapper by adding those touchpads. Once at least one touchpad is added, the game stops treating UI element as a rectangle, and processes input only in allowed area.
There are two main types of touchpads. One allows input (MenuItemWrapper.InsideTestModes.Inside) and one denies it (MenuItemWrapper.InsideTestModes.NotInside). As far as I can tell, you can add any number of touchpads to the item, and game will combine them all to determine if mouse input is actually allowed for a particular area of the item.
There are three possible 'shapes' for a touchpad: an ellipse (ideal for circular dials and buttons), a triangle (e.g. TAI maximize/minimize button) and an arbitrary polygon (which I think game just splits up to multiple triangles). I made a quick test with ellipses (attached one to each of attack disc parts), and my attack disc suddenly started to respond to dragging its elements. It was first 'hurray' for me. Though before that, I had a small issue in determining what do parameters for AddTouchPadEllipse() actually mean, but more on that later. Anyway, some more quick tests quickly showed me that I can actually define any custom shape for the input area: it's only a matter of determining the coordinates for touchpads.

And so I patiently calculated all the coordinates for each pointer on the attack disc, added a corresponding touchpad polygon, started my game and... well, was surprised. For some reason, game refused to react on my attempts to drag a pointer. And the Attack Course pointer started to behave very strange when I rotated it (when it was horizontal I had a hard time finding a place which allowed me to drag it). Something was definetly going on. Luckily, I found out what rather quickly.

Documentation states that changing Rotation property for MenuItemWrapper (i.e. rotating it) also rotates all touchpads. What it doesn't say is that rotations are confined to item's bounding box. If it is rectangular, polygons start to shrink or extend when rotated. And several images from my attack disc (I use the one from TheDarkWraith's NewUIs) are indeed rectangular. Well, I thought OK, how about I put each disc in a separate control group, make these groups square and rotate them instead. Unfortunately, the game refused me this attempt, stating that it cannot rotate items without a texture (and control groups are just that). So I had to resize all the textures to make them square, recalculate my touchpads and try again. And it worked: all the discs rotated correctly. This was the second 'hurray'. The only issue I have with this is that I had to make images larger, therefore now they consume more VRAM. Though I don't know if it can become a significant problem, considering the overall ratio of UI to game world texture usage (and anyway, square images are only needed if you intend to rotate them).

I promised to tell about parameters for funcions that add touchpads. Here goes:

For all functions, IfInside parameter sets the type of the touchpad: including (allows input) or excluding (denies input)

AddTouchPadEllipse( IfInside, CenterXPercent, CenterYPercent, WidthPercent, HeightPercent )

Coordinates are in the range [0,1] (hence the 'Percent') suffix. Width and height are ellipse radiuses. But there is a quirk with them. Initially, my thought process was like this: if the full range of coordinates is 1, then to make an incircle for the square image, I'd have to place center at (0.5, 0.5) and make 'width' and 'height' equal to half image width, thus, also (0.5, 0.5). Well, I was wrong. It turns out, that the game considers maximum values for width and height parameters as halves of actual image width and height. Thus, for 200 pixels-wide image (incircle should have a radius of 100 pixels), WidthPercent 0.5 would mean 0.5*0.5*200 = 50! Therefore, to make an actual incircle, parameters should be:

CenterXPercent = 0.5
CenterYPercent = 0.5
WidthPercent = 1.0
HeightPercent = 1.0

This is important to remember when calculating actual size for this type of touchpad.

AddTouchPadTriangle( IfInside, x0, y0, x1, y1, x2, y2 )
AddTouchPadPolygon( IfInside, Single[] coords )

These two are pretty straightforward: you just specify a list of 2D vertices (x,y pairs), where each coordinate is in range [0,1]. One thing to note is that 0 for x is left, 0 for y is top, no matter what offsets you specified while placing an image on page (i.e. these are local coordinates).

One last thing to note about touchpads: AFAICT, they can only be added, but not removed, and they are not removed when page's script is unloaded. This is a bit unfortunate, but I personally can live with that.

To summarize: while somewhat problematic, it's actually not that hard to implement custom-shaped widgets. With taking several quirks I mentioned into account, it becomes only a matter of calculating actual touchpad areas. This is my second step, as I'm now starting to work on application that will allow to manually "draw" touchpad areas for textures and save them into text file, so that it can be read inside script during initialization. Another approach would be to use some form of edge detection and vectorization to calculate the input areas automagically, but that's beyond my abilities for now.

If anyone is interested, I've uploaded my InputCanvas class along with sample Python module. Maybe it'll help someone in creating another awesome mod. Simply place the dll into SH5 installation folder and use it like this:

import clr
clr.AddReference("RadCapTools")
from RadCapTools import InputCanvas

The documentation for InputCanvas is included in the archive.

Download here

I've recorded a short video to illustrate what all the above rant means.
. I hope whoever would watch would also excuse my English


Cheers

[radcapricorn]

Okay, for all UI modders who want to utilize the touchpads functionality right now, here's the how-to. Beware it's a bit technical!
Also note, the how-to does not use the sample python script provided in the download, for that wait for a further example.

1. Not really a step, just a note. Be prepared to do some calculations (as you don't want to wait till I finish my calculator application )
2. If you're creating/editing a rotatable control (attack disc, RAOBF, 3 bearings tool, etc.), make sure that it's texture is square. Having non-square textures will screw up your touchpads when you rotate it. Example:
   
   
   
   
   On the left is the original rectangular image, on the right is the proper square one. Notice how I left the image itself centered.
3. Make sure that you synchronize your changes to image sizes with SH5 pages configuration (i.e. open up menu editor and enter proper values)
4. For each image, define a set of areas you want (or don't want) to click on. Remember there are three basic shapes to choose from (ellipse/circle, triangle and polygon).
   
   Let's take the "Vorhalt" (lead angle) pointer from the attack disc. In this example, I'll mark the areas that I want to handle input with green, and those I don't want to handle input with red.
   Note that while I use the same image that I used when created sample code, the numbers I provide here may differ slightly since I redid the whole procedure again.
   
   This is the area I want to 'listen' to my mouse:
   
   
   
   
   I need to split up this area into simple shapes. I immediately see a circle:
   
   
   
   
   Yes, I know that it covers the space where mouse should not be handled, but bear with me on that.
   Now, I look at the pointer. It's a trapezoid:
   
   
   
   
   Ok, so now I have defined two areas where I want my mouse input. Now for the areas where I don't want it. I want to keep everything as simple as possible, so:
   
   I don't want my mouse handled in the lower half of the image:
   
   
   
   
   I don't want my mouse handled in these two areas:
   
   
   
   
   Note that I actually cheat, since there's also a small circle in center which also should not receive mouse input:
   
   
   
   
   But I won't bother with it, since I know that when I combine all images, it will always be covered by another object (the attack course pointer).
   
   Using your image editor (I use GIMP) or any other application that allows you to see mouse coordinates, measure all these areas. For circles, we'll need center and radius (two radiuses for ellipses).
   For anything else, we'll need corner points (vertices).
   
   The whole image is 556x556 pixels. For the circle, it's center will be in the middle of image (278x278), and I measured it's radius to be ~151-152 pixels (I used GIMP's measure tool).
   For pointer, I just found the coordinates of it's corners (yep, pointed my mouse at them and took GIMP's feedback). Here's an illustration of what I got:
   
   
   
   
   Next, I traversed this polygon and wrote down the coordinates of these points as I encountered them:
   
   
   
   
   Next, I did the same measurements for the rectangle an the triangles. Here's what I got in the end:
   
   Circle: center = 278,278, radius = 152
   Pointer: 250,123 -> 270,0 -> 286,0 -> 305,123
   Rectangle: 0,278 -> 556,278 -> 556,556 -> 0,556
   Left Triangle: 278,278 -> 110,278 -> 137,212
   Right Triangle: 278,278 -> 454,278 -> 418,212
   
   Okay, you've wrote down all those coordinates. Now you need to convert them into relative coordinates in the range [0,1]. Those familiar with vector graphics know what I mean, for everyone else here's what to do:
   take each point, and divide X coordinate by image width and Y coordinate by image height. For my pointer, I get the following:
   
   250,123 -> 270,0 -> 278,0 -> 305,123 divide each X by image width (556) and each Y by image height (also 556):
   
   0.449,0.221 -> 0.485,0 -> 0.514,0 -> 0.548,0.221
   
   You don't need to take all the significant digits, in fact 2 or 3 will be quite enough. Repeat this for each polygon you've got.
   
   Now, with circles... stop. If you haven't read my original post, you don't know that circles are funky. For center, divide the coordinates as usual by image size. But for radius, divide by half image size!
   E.g. for my circle I will divide center coordinates by 556, but radius by 278. If you've got ellipse and not a circle, you'll need to divide X radius by half width, and Y radius by half height.
   
   My numbers after the calculations:
   Circle: center = 0.5,0.5, radius = 0.546
   Pointer: 0.449,0.221 -> 0.485,0 -> 0.514,0 -> 0.548,0.221
   Rectangle: 0,0.5 -> 1,0.5 -> 1,1 -> 0,1
   Left Triangle: 0.5,0.5 -> 0.197,0.5 -> 0.246,0.381
   Right Triangle: 0.5,0.5 -> 0.816,0.5 -> 0.751,0.381
   
   Tedious? Well, it's a lot more in writing than in doing. And anyway, wait for my calculator app. You'll just draw all those areas instead!
   
   I must say it would have been a LOT easier if we could just create a mask texture that would show exactly what parts of image are processing input (and I have an itching suspicion that stock game dials do just that).
5. Code
   
   In your page script, import the InputCanvas class:
   
   

   import clr
   clr.AddReference("RadCapTools")
   from RadCapTools import InputCanvas

   Add a global for InputCanvas so you can access it from different functions:
   
   

   BearingLeadPointer = None

   In your InitializeScript() function, add the touchpad areas to your items:
   
   

   def InitializeScript():
       if Page_Test1_AttackDisc_Front_BearingLeadAngle.Touchpads.Length == 0:
           # Adding areas that process mouse input (MenuItemWrapper.InsideTestModes.Inside)
           # Circle:
           Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadEllipse( MenuItemWrapper.InsideTestModes.Inside, 0.5, 0.5, 0.546, 0.546 )
           # Pointer:
           Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadPolygon( MenuItemWrapper.InsideTestModes.Inside, 0.449,0.221, 0.485,0.0, 0.514,0.0, 0.548,0.221 )
           # Adding areas that don't process mouse input (MenuItemWrapper.InsideTestModes.NotInside)
           # Rectangle:
           Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadPolygon( MenuItemWrapper.InsideTestModes.NotInside, 0.0,0.5, 1.0,0.5, 1.0,1.0, 0.0,1.0 )
           # Left Triangle:
           Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadTriangle( MenuItemWrapper.InsideTestModes.NotInside, 0.5,0.5, 0.197,0.5, 0.246,0.381 )
           # Right Triangle:
           Page_Test1_AttackDisc_Front_BearingLeadAngle.AddTouchPadTriangle( MenuItemWrapper.InsideTestModes.NotInside, 0.5,0.5, 0.816,0.5, 0.751,0.381 )

   Here, one at a time, I add all the areas that I've previously calculated, specifying whether I want mouse input in them or not.
   
   Why is that Page_Test1_AttackDisc_Front_BearingLeadAngle.Touch pads.Length == 0 check there? Well, the touchpads may only be added, not removed. At least I haven't found a way to do that. But InitializeScript() function will be called more
   than once during gameplay. This means that (a) we need some check to only add touchpads once and (b) if you make changes to your touchpads during testing, you'll have to reload the game. Oh well.
   
   Now, in that same function, create an InputCanvas and attach some event handlers to it:
   
   

       # InitializeScript() continued
       global BearingLeadPointer
       BearingLeadPointer = InputCanvas(Page_Test1_AttackDisc_Front_BearingLeadAngle)
       #
       BearingLeadPointer.LostFocus += This_LostFocus
       BearingLeadPointer.MouseLeftButtonDown += This_MouseLeftButtonDown
       BearingLeadPointer.MouseLeftButtonUp += This_MouseLeftButtonUp
       BearingLeadPointer.MouseMoved += This_MouseMoved

   Here, I've basically created my mouse polling object and told it what functions it should call when certain events happen. Now for those who haven't seen the video I've posted, I'll repeat that you can track mouse position for the control even when mouse is actually outside it. If TheDarkWraith is reading this, I'm sure he's laughing at this point, because programmatically, there's no restriction whatsoever if the mouse is inside or not, we can just poll it at any given time. What I mean is for our purposes, we need some way to determine whether mouse position is still relevant for our UI item. That's why there are two separate types of events for stock API objects: mouse in/out events and focus events. An item gains focus whenever mouse is moved over it (so MouseIn and GotFocus event arrive in pair). Now if you press and hold mouse item inside an item and drag mouse out of it, it'll still have mouse focus, though it already got a MouseOut event. That's why I add a LostFocus handler here: I want to know when to actually stop waiting for mouse input (i.e stop dragging). The InputCanvas class basically forwards events form FocusMenuItemWrappers it handles, it's usefulness is in that MouseMoved event which is not present for the stock API objects.
   
   In UnloadScript() function, add code to dispose of InputCanvas:
   
   

   def UnloadScript():
       global BearingLeadPointer
       if BearingLeadPointer:
           BearingLeadPointer.Dispose()
       BearingLeadPointer = None

   The only thing left is to define those event handler functions. Here's an example with comments:
   
   

   Dragging = False # This will tell us whether we drag our pointer
   
   def This_LostFocus( sender ):
       global Dragging
       Dragging = False # When our pointer lost input focus, stop dragging
       
   def This_MouseLeftButtonDown( sender, point ):
       global Dragging
       Dragging = True # When mouse button is pressed, start dragging
       
   def This_MouseLeftButtonUp( sender, point ):
       global Dragging
       Dragging = False # When mouse button is released, stop dragging
       
   def This_MouseMoved( sender, newPos, oldPos ):
       global Dragging
       if not Dragging:
           return # we're not dragging, so we don't want to do anything
       
       # Below is simple code to apply rotation based on mouse motion
       # It's the same code found in sample python module
       # Note that it uses mouse displacement to calculate rotation angle
       # You can easily use just current mouse position and calculate
       # absolute rotation angle
   
       # We need to convert coordinates since InputCanvas gives them in screen space    
       
       # previous mouse position relative to center
       v0 = sender.screenToClient( oldPos, MenuItemWrapper.LocationPresets.Center )
       # current mouse position relative to center
       v1 = sender.screenToClient( newPos, MenuItemWrapper.LocationPresets.Center )
       
       # rotate one of the discs
       delta = atan2(v1.Y, v1.X) - atan2(v0.Y, v0.X)
       # atan2 gives proper sign, so just add the delta to current rotation
       sender.Rotation = sender.Rotation + delta

   That's pretty much it.

Now you may ask, then what the hell this AttackDisc.py in sample downloads is for? Well, this how-to I built from scratch. And that script I extracted from my actual testing sandbox. It's basically just a more
sophisticated version of code presented in this how-to, and it handles the full attack disc. If you look carefully at that code, you'll find that it's pretty similar to the one presented here, it only differs in function signatures (since AttackDisc is a class there) and in amount of InputCanvas objects.

I put in the sample just what I exctracted into one neat class so that I don't clutter my own functions with it. In fact,
I already figured out how to actually use it as a separate script file (i.e. scripts/menu/radcapricorn/AttackDisk.py) without creating any dummy pages, but that's another topic.
If I get permission from TheDarkWraith to post changes to his scripts, I'll show you how you can use that sample python module to actually control attack disc in TDW's UI. I've actually already tested that code with his UI (since I already have touchpad areas calculated for that attack disc) and I must say it works perfectly.

For any other objects (RAOBFs, 3-bearing tools and whatnot) you'll have to wait until I've finished my calculator app: I'm done with manual calculations

As a side note, you don't need to use my InputCanvas if you just want to create shaped buttons. Touchpads are enough for that. See how triangle buttons are done in stock Page Default Hud.py, for example.
You can use the same approach to make round buttons, hexagonal ones, you name it. There are limitations, of course, but I'll post them once I've tracked down at least all major ones.

Wow, that took a lot more time than I expected. But that's not all. Stay tuned

Have fun!

[reaper7]

This is amazing, well done - this is one thing I solely missed making my UI .
Wii add this to my UI Ui-boat V5.

Ps The RAOBF are from my Ui-Boat V5 and also HAHD U-Boot_HAHD
Nice to see it working the way I always dreamed it to work

[radcapricorn]

Thanks, reaper7! I was beginning to think nobody's interested

I still have to insist that this RAOBF I took from CSP MaGui by dr. Jones I've taken it to make that example video just for my own sake: it's basically two images (there's more in total, but actual tool is just that), so there wasn't a lot input areas to calculate.

See the outer ring: it has this metallic rod thing at 8 o'clock, and a plastic 'cover' with red mark at 1 o'clock. These actually (I guess) should block the input to the inner ring (they are a part of the outer ring after all), but as of now such fancy stuff is not possible to do without adding (dummy) items, as I encountered some issues with overlapping input areas inside one UI item (I'll investigate a bit more and update the first post accordingly). IIRC, in your mod the plastic markers are in a separate image, so this should not pose any problems at all.

PS.
I beleive I've also answered your PM, but right now my 'sent items' box is empty. I only guess it's due to ongoing maintenance, so I'll wait and see if it pops up.

[Trevally.]

Hi radcapricorn

I have just watched your vid for the mouse control and it looks great

With the download, if I am using TDWs UI and install the .dll in Sh5 main dir and the .py script in the script/menu file - will i be able to control the RAOBF and attack disc as you did in the vid

[radcapricorn]

Thanks, Trevally!

Originally Posted by Trevally

With the download, if I am using TDWs UI and install the .dll in Sh5 main dir and the .py script in the script/menu file - will i be able to control the RAOBF and attack disc as you did in the vid

Not per se, no. This is not that automated The sample .py script is just that - sample (though the touchpad areas defined for the attack disc there are for the attack disc from TDW's UI). I've extracted it from my UI test page script. You'll need to actually add some code to TDW's scripts to use the functionality. One way is to copy/paste that code into TDW's Page TDC.py (IIRC the Attack Disc in TDW's UI is located at TDC page) and add proper initialization/cleanup to InitializeScript()/UnloadScript() functions.
For example, here's how I do it in my test page (simplified code off the top of my head):

# File scripts/menu/pages/Page_Test1.py

# The following is the contents of the sample script

import clr
clr.AddReference("RadCapTools")
from RadCapTools import InputCanvas

class AttackDisc( object ):
# The definition of the attack disc from the sample script follows
# ...

# The contents of sample script end here

# global variable to store the attack disc input handler
g_AttackDisc = None

def InitializeScript():

    compassRose = Page_Test1_AttackDisc_Front_CompassRose
    bearingLead = Page_Test1_AttackDisc_Front_BearingLeadAngle
    targetDisc = Page_Test1_AttackDisc_Front_TargetDisc
    attackCourse = Page_Test1_AttackDisc_Front_AttackCourse

    # Create the attack disc input handler
    global g_AttackDisc
    g_AttackDisc = AttackDisc(compassRose, bearingLead, targetDisc, attackCourse)

# Some more code skipped...

def UnloadScript():
    # Destroy the attack disc input handler
    global g_AttackDisc
    if g_AttackDisc:
        g_AttackDisc.dispose()
    g_AttackDisc = None

That'd roughly be the modification to TDW's scripts/menu/pages/Page TDC.py.
Now, those Page_Test1_AttackDisc_Front_* names are for my UI page. For TDW's, you'll have to look at data/menu/pages/Page TDC.ini to figure out proper names.

I think I may have to write a more detailed instructions on what to do, but that'll be later once I'm at my SH5 rig and have an actual UI mod files to use for illustration.

Oh, yeah, and a follow-up:
In the first post, I mentioned the issues with rotation if the textures are not square. Some original attack disc textures from TDW's UI are not square so I had to edit them accordingly (this required resizing them in GIMP and making changes in SH5 Menu Editor). Anyway, excpect an update with proper instructions in an hour or so.

[Trevally.]

Thanks radcapricorn

[radcapricorn]

Post #2 is updated with a small how-to.

[Trevally.]

Originally Posted by radcapricorn

That's pretty much it.

great tutorial

If I get permission from TheDarkWraith to post changes to his scripts, I'll show you how you can use that sample python module to actually control attack disc in TDW's UI. I've actually already tested that code with his UI (since I already have touchpad areas calculated for that attack disc) and I must say it works perfectly.

For any other objects (RAOBFs, 3-bearing tools and whatnot) you'll have to wait until I've finished my calculator app: I'm done with manual calculations

Fingers crossed

Wow, that took a lot more time than I expected. But that's not all. Stay tuned

[radcapricorn]

Originally Posted by Trevally

great tutorial

Thanks!

Originally Posted by radcapricorn

If I get permission from TheDarkWraith to post changes to his scripts, I'll show you how you can use that sample python module to actually control attack disc in TDW's UI.

I have one silly off-topic question. I've sent a PM to TheDarkWraith, but as with previous PM to reaper7, I cannot see it in 'Sent Items' box. Is it supposed to be like that or something's just currently not working?

[volodya61]

Originally Posted by radcapricorn

..I have one silly off-topic question. I've sent a PM to TheDarkWraith, but as with previous PM to reaper7, I cannot see it in 'Sent Items' box. Is it supposed to be like that or something's just currently not working?

You should enable "Save a copy of sent messages in my Sent Items folder by default" in your "Edit options".

[radcapricorn]

Silly me!

[reaper7]

Thanks to radcapricorn I now have a fully working Attack Disc in my next release of Ui-Boat

[radcapricorn]

You are welcome!