Hi raz, Welcome to Subsim.
Unfortunately, SH3 and the 8000 series (specifically the 8800 series) Nvidia cards do not play well together with SH3 on all occasions.
Though I cannot speak for the high resolution settings you are using, you can experiment using different drivers available here:
http://www.nvidia.com