In regards to resolution with HiResFix, edit the d3d9.cfg file's ResX and ResY parameters to set the resolution you need.
As for F9-F11 keys, HiResFix indeed swallows them for some reason, although you can still access those stations using Alt+F9-F11 keys
PS. On my PC (Win7) in order to use HiResFix I had to create a custom 1400x1050 resolution in nVidia control panel (just as you did) and switch to it every time I launched the game. Prior to doing so, I was always getting that "Failed to initialize 3D engine" message (the infamous 59Hz issue).