I assume your Windows is 64 Bit, else you cannot make use of all your RAM and would be limited to some 3.7 GB.
Okay, here is what I recommend to do.
Start with a clean install.
Run the
Venetubo config tweaker (LINK). It lets you download a tweaked config once you entered the data, you repalce the default config with this one. The config is hidden at C:\Documents and Settings\Application Data\(Your Windows User Name)\Microsoft\FSX. Or type in fsx.cfg in the searchg box. You need to set show hidden folders to on.
You install an external frame limiter and fix frames to 30. You starts FSX from here only.
Important: you then must set FSX internal frame limiter to "unlimited".
You check that nVidia driver is set to have FSX controll the settings of Anti-Alias and Ansitropic Filtering.
You hardware is faster than mine, and so you should be able tpo achieve better results than I do. And I can acchieve vry good results already with more complex software setups than just a default FSX.
Then try to work with the settings below. These are what I am using. I fly the PNDG737NGX, use Aerosoft airport sceneries, use ASE and REX2, TrackIR, pedals, Throttle and Stick.
Traffic is a major frame hog. Especially the generic ground traffic costs much, but makes no difference for you if you fly airliners. Start with having all traffic at zero. (I have airport traffic at zero, too, only because I exclusively fly from addon airports whose own traffic could conflict with the default generic airport traffic by FSX. If you do not have addon airports, you can use airport traffic, and can also experiment with reducing the ground scenery settings if needed - but you should not need to do that with your rig).
Oh, before I forget: cancel DX10, if you have activated it. FSX never was optimized for that, it slows down many systems, and produces graphical artifacts. Sorry, that'S how it is. FSX is no DX10 prorgram.
