Just as a line of reference, I'm running a GF5600Ultra 256mb card, P4 3.2, a Hyundai L90D 19" LCD, and 2 sets of 2 matched 512mb sticks of PC3200 DDR RAM for total of 2 gigs. I'm also using the 16km mod but not GW as of yet. Also my SHIII is patched to 1.4( as all should be), 62 mods installed, SHIII Commander installed and running my career.
This computer is about 18 months old but not showing any graphics problems like Renegade and Kiwi are having. Using 16km or 8km should not affect things. It's
not the game but the settings you are using for the hardware you are running.
The resolution I'm running for all my games is 1152x864 and 32 bit color quality. Baseline, has a handle on it with Anisotropic Filtering and Anti Aliasing. Check your cards and see if they're on or off. My AF and AA are Application-controlled for both. Image is set to Quality and Verticle Sync is also Application-controlled. My max res is 1280 by 1024.
As a side note, it might help if you have problems to post your DX dialog, as that will show where your problems are. And it's proof whether it's the game or hardware.