-First of all your video card has enough memory to run the game.
-How much system RAM do you have?
-Make sure you have the latest drivers for both your video and sound cards.
- most games are optimized for 256 (or true) colours,and SH3 would be no exception

- ideally set your resolution to 1024x768,unless you have a widescreen monitor,in which case you can play in higher resolutions,but you will need to install a special mod for this called 'res fix mod'
- if you are having performance issues with the game,first of all set your video card for performance.
- the effects called anti aliasing and anisotropic filtering are advanced features of most if not all video cards,and they can severely impact performance in games,especially games that dont fully support the feature.Anti aliasing reduces the jagged edges you sometimes see on the edge of graphics making them look smoother and clearer.Anisotropic filtering allows graphics to be rendered in great detail in the distance instead of only being rendered in the foreground making games look sharper and more detailed.Turning this off for example would make distant objects look blurred due to the video card not rendering those textures.
I would not recommend AA in SH3 especially with GWX3 installed,as this is a major memory hog and your frame rates would really suffer.If you want to play around with AF first start at x2 and see how your fps go,and increase until they drop to unacceptable levels.Ideally though you want both turned off to maximize your performance.These features are really only designed for video cards with 512MB + memory,and probably dual core CPUs.
Hope some of this helps,and feel free to ask anything else you dont understand.