Well, like I said, I'm not familiar with nVidia but SH3 uses DirectX rather than OpenGL as a rendering engine.
So, within your GFX Panel, there should be separate options/settings for both engines. Look for the DirectX section and see if there is anything that 'jumps' out at you. It's unlikely but maybe possible and is certainly worth checking.
Failing that, I would suggest that you try to find another 8800 user to see if they have ever experienced this. Could be driver related but I'm sure you would notice other things wrong if it was.
I still can't think what might be limiting the actual resolution when the game enters 3d rendering (i.e ingame). The res fix is working, otherwise the menu's wouldn't have changed, so there must be something limiting the 3D/game render part.
