AA & AF balancing woes

Galanti · 10-04-07, 06:43 AM

Argh, I just can't seem to find the right balance between image quality and perofrmance with AA and AF. For the longest time I ran 4x of each, but ever since the ROW mod, my subs external wires seem jaggy and shimmery. I've tried all the rivatuner settings I can think of for my 7950GT (using drivers only a couple of versions old) and 8x or 16x AA will do the job, but at a huge price in performance.

What's everyone else using, and do you find the wires/cables/stays on ships and your sub are awfully jaggy at all but point-blank range?

AVGWarhawk · 10-04-07, 08:37 AM

Almost sounds like the AA is not being forced. Try one thing, remove ROW and see if the smoothness is back. If it is the same then you need to look elsewhere for the nonforcing of AA.

Galanti · 10-04-07, 08:52 AM

It's definately being forced, i've tried it on and off and the difference is glaring.

I guess my real issue is the wires on the ships, they're pretty jaggy and shimmery at the default zoom level in free camera, no matter what I do in Rivatuner.

It looks as though it probably has to do with LOD, I've got it maxed in game, and still have to zoom right up the sub or ship to get solid wires.
My thought is that perhaps the new camera in ROW has changed the field of view to a point where LOD becomes impacted.

Beh, it's definately not a showstopper, nor a reason not to use ROW, I'll disable the camera tonight and see if there's a change. I've also noticed AA in Nvidia cards doesn't seem to look nearly as good as ATI AA.

AVGWarhawk · 10-04-07, 09:02 AM

I run just 2x on my X1300. As far as LOD is that not just for land and land objects?

Galanti · 10-04-07, 11:48 AM

Ah, that would explain it, I wonder if there is a hidden tweak to boost the LOD of your sub.

How are your wires and cables in game with 2x AA?

AVGWarhawk · 10-04-07, 12:31 PM

Between no AA and 2x there is a very noticable difference in jaggie lines and wires. I know right off the bat if the AA is not working so it is that noticable. If I go to 4x my FPS will drop too much. Thing here is I'm using an ATI with the ATI tool tray. In way we are comparing apples and oranges when it comes to cards and specific settings. Need to get some of the guys with Nvidia cards in here.

Galanti · 10-04-07, 01:21 PM

Quote:

Originally Posted by AVGWarhawk

Thing here is I'm using an ATI with the ATI tool tray. In way we are comparing apples and oranges when it comes to cards and specific settings. Need to get some of the guys with Nvidia cards in here.

Yeah, on my old 9800pro there was a world of difference between 4x AA and 4x AA in my 7950GT.

THE_MASK · 10-04-07, 02:50 PM

What resolution are you running at .

Galanti · 10-04-07, 05:36 PM

1280x1024, shouldn't be much of a jaggy problem at all on a 17" LCD, but there is.

minsc_tdp · 10-04-07, 06:30 PM

I use an NVidia 8800GTS. This has bothered me since the AA support was put in and I've looked very closely at it. I have a very sensitive eye for AA quality and jaggies.

I use nHancer to experiment with three kinds of AA modes: Multisample, Coverage Sample, and SuperSample.

Multisample works with most games and is what the NV control panel uses, if that's all you use to set your modes. In SH4, this results in decent quality up close but faraway objects, namely the ship lines strung across the ships, and to a slightly lesser extent the ships edges, look like hell. It seems that at a certain distance the AA pretty much just turns off. Coverage Sample isn't much help. Even if you crank it to Multi 8xQ or Coverage 16x or 16xQ it doesn't really help except when you are up close.

SuperSampling completely solves the quality problem, but unless you're running an overclocked Core2 at 4 GHz+ and at least two Crossfire ATI cards (also overclocked heavily), you're going to get crashes and/or 5-10 FPS (assuming you were getting 60+ with Multisample 4x!) To say it is slow is the understatement of the century. For my Nvidia card, this mode doesn't work well with most games - it looks better than everything as it generates every frame at 4x the resolution and then resizes it down, which is hands down the best quality AA method. But it's too slow to be usable and unstable and unsupported on Nvidia cards.

I get the feeling if someone were to dabble with the textures used on the ships lines and edges, that perhaps this could be improved. Beyond that, someone would need to look at whether perhaps the AA is literally not being applied at all to ships at a certain distance. This could result in the single largest visual improvement in this game since ROW.

minsc_tdp · 10-04-07, 06:45 PM

Hmm... on close inspection my suspicion is correct. There is a hard distance where the ship quality drastically changes. I'm not good at estimating distance, so here's two pics in external camera mode. Note these are both in the SAME game session - the only difference is that one is a brief UP keystroke closer to the ship than the other. There is a huge difference:

Most noticably the ships lines change, almost becoming invisible and this might seem bad but it's actually because the AA is kicking in and they're getting blended, and since they're so thin it's normal for them to be hard to see. But when you see this in motion, it's a sharp and obvious change from bad to good when you cross the line. It's pretty clear the AA just turns off for distant objects. Modders... you know this stuff way better than me. Are there two sets of ship textures, one for far away and one for up close? Perhaps the far off ones could be replaced en masse with the high-res ones, which would put a higher load on the system but it would allow AA to work, even if only for higher-end systems.

All that is easier to see if you crank up the gamma. On further inspection there seems to be four distinctly different detail levels, which are easier to see if you move away from the ship upward with the camera looking down on the deck. There could be more than four. It would be great if someone could figure out how to replace those lower detail versions with the highest detail version so it just uses it all the time!

Update: I just tried all the AA compatibility modes in nHancer and it had no effect. The textures are just switching, and low res crap can't be AA'd to look better when it starts out as crap in the first place. I'd kill for a solution to this!

Galanti · 10-04-07, 07:58 PM

Wow, that's some good digging there, minsc. Maybe what we're hunting for is a LOD distance parameter somewhere in a config, or cameras.dat.

I'm getting somewhere, I disabled the ROW camera and went back to stock. Jaggies on my boat in free cam were noticealby better among the stays and safety rails. I have no idea why, may have something to do with diffent FOV values or zoom distances?

But you're right, I've tried all the AA modes and settings, and while 95% of the image quality is excellant, those damned jaggies and shimmering still get to me.

minsc_tdp · 10-04-07, 08:14 PM

I found a fix!

This has just made the game twenty times better looking! You spend so much time looking at ships at a long distance trying to ID them, now that is really a lot easier. I can't believe I didn't try this earlier.

That Super - Transparency AA checkbox is the key. You should be able to use any AA mode you want though I'd recommend Multi 4x (as I'm using above). Some small enhancement can be had with Coverage Sample but it's minor and seems noticably slower at times. As mentioned before, I don't think Supersampling is possible with anything but the highest of highest end ATI hardware, and it's probably slow there too.

nHancer is free, available at nhancer.com. First, click the Add button to create a new profile. Then add the SH4.exe to the box below called Executables with the other Add button off to the right. Now make sure that profile is highlighted and the Profile radio button is selected at the top. Select the options I've selected on the right panel. One more I have selected that you can't see is under the Optimizations tab, Texture filtering = High Quality. I'm quite sure the Super - Transparency AA is the key to it all and nothing else matters, other than making sure you have some kind of AA mode selected at the top, even Multi 2x is enough but 4x is a lot better.

Here's a different (but similar) ship, the one from the Torp Training school mission, with those settings closer and closer:

The higher resolution texture kicks in on the second to last one, but you can hardly tell, unlike the images up above.

You can even alt-tab out of the game and change this without restarting to see an immediate comparison. Here's another with virtually identical pics, from the torp training mission right at the start:

Note how this does not make any difference when you are close enough to the ship:

I can barely tell the difference, because it's past the line where the AA kicks in regardless.

Galanti · 10-05-07, 06:49 AM

I've been using supersampled transparency for weeks, with no apparent effect. Maybe it's because I'm at 1280x1024? The Nhancer site claims this mode only works in 1024x768 and lower.

What I did do was use the Silent 3D editor to change the field of view/perspective for freecam back to stock values in the Camera mod part of ROW and it's definately improved the wires. I definately prefer the stock camera freecam perspective over ROW's, although the rest of ROW's camera mod is absolutely brilliant.

minsc_tdp · 10-05-07, 03:00 PM

Quote:

Originally Posted by Galanti

I've been using supersampled transparency for weeks, with no apparent effect. Maybe it's because I'm at 1280x1024? The Nhancer site claims this mode only works in 1024x768 and lower.

What I did do was use the Silent 3D editor to change the field of view/perspective for freecam back to stock values in the Camera mod part of ROW and it's definately improved the wires. I definately prefer the stock camera freecam perspective over ROW's, although the rest of ROW's camera mod is absolutely brilliant.

That's probably it, the resolution.

If you can explain more about the camera edit you did, I don't understand fully exactly what you're talking about. Perhaps editing the FOV altered the calculated distance and so it triggered the higher res versions of the objects earlier or something. Try 1024x768 just to see if supersampled AA works at all like I describe above.

10-04-07, 06:43 AM	#1
Galanti Captain Join Date: Apr 2004 Location: Ottawa Posts: 536 Downloads: 99 Uploads: 0	AA & AF balancing woes Argh, I just can't seem to find the right balance between image quality and perofrmance with AA and AF. For the longest time I ran 4x of each, but ever since the ROW mod, my subs external wires seem jaggy and shimmery. I've tried all the rivatuner settings I can think of for my 7950GT (using drivers only a couple of versions old) and 8x or 16x AA will do the job, but at a huge price in performance. What's everyone else using, and do you find the wires/cables/stays on ships and your sub are awfully jaggy at all but point-blank range?

10-04-07, 08:37 AM	#2
AVGWarhawk Lucky Jack Join Date: Jun 2005 Location: In a 1954 Buick. Posts: 27,343 Downloads: 90 Uploads: 0	Almost sounds like the AA is not being forced. Try one thing, remove ROW and see if the smoothness is back. If it is the same then you need to look elsewhere for the nonforcing of AA. __________________ “You're painfully alive in a drugged and dying culture.” ― Richard Yates, Revolutionary Road

10-04-07, 09:02 AM	#4
AVGWarhawk Lucky Jack Join Date: Jun 2005 Location: In a 1954 Buick. Posts: 27,343 Downloads: 90 Uploads: 0	I run just 2x on my X1300. As far as LOD is that not just for land and land objects? __________________ “You're painfully alive in a drugged and dying culture.” ― Richard Yates, Revolutionary Road

10-04-07, 12:31 PM	#6
AVGWarhawk Lucky Jack Join Date: Jun 2005 Location: In a 1954 Buick. Posts: 27,343 Downloads: 90 Uploads: 0	Between no AA and 2x there is a very noticable difference in jaggie lines and wires. I know right off the bat if the AA is not working so it is that noticable. If I go to 4x my FPS will drop too much. Thing here is I'm using an ATI with the ATI tool tray. In way we are comparing apples and oranges when it comes to cards and specific settings. Need to get some of the guys with Nvidia cards in here. __________________ “You're painfully alive in a drugged and dying culture.” ― Richard Yates, Revolutionary Road

10-04-07, 06:30 PM	#10
minsc_tdp XO Join Date: Mar 2007 Posts: 411 Downloads: 1 Uploads: 0	I use an NVidia 8800GTS. This has bothered me since the AA support was put in and I've looked very closely at it. I have a very sensitive eye for AA quality and jaggies. I use nHancer to experiment with three kinds of AA modes: Multisample, Coverage Sample, and SuperSample. Multisample works with most games and is what the NV control panel uses, if that's all you use to set your modes. In SH4, this results in decent quality up close but faraway objects, namely the ship lines strung across the ships, and to a slightly lesser extent the ships edges, look like hell. It seems that at a certain distance the AA pretty much just turns off. Coverage Sample isn't much help. Even if you crank it to Multi 8xQ or Coverage 16x or 16xQ it doesn't really help except when you are up close. SuperSampling completely solves the quality problem, but unless you're running an overclocked Core2 at 4 GHz+ and at least two Crossfire ATI cards (also overclocked heavily), you're going to get crashes and/or 5-10 FPS (assuming you were getting 60+ with Multisample 4x!) To say it is slow is the understatement of the century. For my Nvidia card, this mode doesn't work well with most games - it looks better than everything as it generates every frame at 4x the resolution and then resizes it down, which is hands down the best quality AA method. But it's too slow to be usable and unstable and unsupported on Nvidia cards. I get the feeling if someone were to dabble with the textures used on the ships lines and edges, that perhaps this could be improved. Beyond that, someone would need to look at whether perhaps the AA is literally not being applied at all to ships at a certain distance. This could result in the single largest visual improvement in this game since ROW. __________________ June 29 2016 - shSpeech v2.55 - Voice Command for Silent Hunter 4! View Thread Last edited by minsc_tdp; 10-04-07 at 07:48 PM.

10-04-07, 08:52 AM	#3
Galanti Captain Join Date: Apr 2004 Location: Ottawa Posts: 536 Downloads: 99 Uploads: 0	It's definately being forced, i've tried it on and off and the difference is glaring. I guess my real issue is the wires on the ships, they're pretty jaggy and shimmery at the default zoom level in free camera, no matter what I do in Rivatuner. It looks as though it probably has to do with LOD, I've got it maxed in game, and still have to zoom right up the sub or ship to get solid wires. My thought is that perhaps the new camera in ROW has changed the field of view to a point where LOD becomes impacted. Beh, it's definately not a showstopper, nor a reason not to use ROW, I'll disable the camera tonight and see if there's a change. I've also noticed AA in Nvidia cards doesn't seem to look nearly as good as ATI AA.

10-04-07, 11:48 AM	#5
Galanti Captain Join Date: Apr 2004 Location: Ottawa Posts: 536 Downloads: 99 Uploads: 0	Ah, that would explain it, I wonder if there is a hidden tweak to boost the LOD of your sub. How are your wires and cables in game with 2x AA?

10-04-07, 02:50 PM	#8
THE_MASK Ace of the deep . Join Date: Jan 2006 Posts: 9,226 Downloads: 901 Uploads: 73	What resolution are you running at .

10-04-07, 05:36 PM	#9
Galanti Captain Join Date: Apr 2004 Location: Ottawa Posts: 536 Downloads: 99 Uploads: 0	1280x1024, shouldn't be much of a jaggy problem at all on a 17" LCD, but there is.

10-04-07, 06:45 PM	#11
minsc_tdp XO Join Date: Mar 2007 Posts: 411 Downloads: 1 Uploads: 0	Hmm... on close inspection my suspicion is correct. There is a hard distance where the ship quality drastically changes. I'm not good at estimating distance, so here's two pics in external camera mode. Note these are both in the SAME game session - the only difference is that one is a brief UP keystroke closer to the ship than the other. There is a huge difference: Most noticably the ships lines change, almost becoming invisible and this might seem bad but it's actually because the AA is kicking in and they're getting blended, and since they're so thin it's normal for them to be hard to see. But when you see this in motion, it's a sharp and obvious change from bad to good when you cross the line. It's pretty clear the AA just turns off for distant objects. Modders... you know this stuff way better than me. Are there two sets of ship textures, one for far away and one for up close? Perhaps the far off ones could be replaced en masse with the high-res ones, which would put a higher load on the system but it would allow AA to work, even if only for higher-end systems. All that is easier to see if you crank up the gamma. On further inspection there seems to be four distinctly different detail levels, which are easier to see if you move away from the ship upward with the camera looking down on the deck. There could be more than four. It would be great if someone could figure out how to replace those lower detail versions with the highest detail version so it just uses it all the time! Update: I just tried all the AA compatibility modes in nHancer and it had no effect. The textures are just switching, and low res crap can't be AA'd to look better when it starts out as crap in the first place. I'd kill for a solution to this! __________________ June 29 2016 - shSpeech v2.55 - Voice Command for Silent Hunter 4! View Thread Last edited by minsc_tdp; 10-04-07 at 07:50 PM.

10-04-07, 07:58 PM	#12
Galanti Captain Join Date: Apr 2004 Location: Ottawa Posts: 536 Downloads: 99 Uploads: 0	Wow, that's some good digging there, minsc. Maybe what we're hunting for is a LOD distance parameter somewhere in a config, or cameras.dat. I'm getting somewhere, I disabled the ROW camera and went back to stock. Jaggies on my boat in free cam were noticealby better among the stays and safety rails. I have no idea why, may have something to do with diffent FOV values or zoom distances? But you're right, I've tried all the AA modes and settings, and while 95% of the image quality is excellant, those damned jaggies and shimmering still get to me.

10-04-07, 08:14 PM	#13
minsc_tdp XO Join Date: Mar 2007 Posts: 411 Downloads: 1 Uploads: 0	I found a fix! This has just made the game twenty times better looking! You spend so much time looking at ships at a long distance trying to ID them, now that is really a lot easier. I can't believe I didn't try this earlier. That Super - Transparency AA checkbox is the key. You should be able to use any AA mode you want though I'd recommend Multi 4x (as I'm using above). Some small enhancement can be had with Coverage Sample but it's minor and seems noticably slower at times. As mentioned before, I don't think Supersampling is possible with anything but the highest of highest end ATI hardware, and it's probably slow there too. nHancer is free, available at nhancer.com. First, click the Add button to create a new profile. Then add the SH4.exe to the box below called Executables with the other Add button off to the right. Now make sure that profile is highlighted and the Profile radio button is selected at the top. Select the options I've selected on the right panel. One more I have selected that you can't see is under the Optimizations tab, Texture filtering = High Quality. I'm quite sure the Super - Transparency AA is the key to it all and nothing else matters, other than making sure you have some kind of AA mode selected at the top, even Multi 2x is enough but 4x is a lot better. Here's a different (but similar) ship, the one from the Torp Training school mission, with those settings closer and closer: The higher resolution texture kicks in on the second to last one, but you can hardly tell, unlike the images up above. You can even alt-tab out of the game and change this without restarting to see an immediate comparison. Here's another with virtually identical pics, from the torp training mission right at the start: Note how this does not make any difference when you are close enough to the ship: I can barely tell the difference, because it's past the line where the AA kicks in regardless. __________________ June 29 2016 - shSpeech v2.55 - Voice Command for Silent Hunter 4! View Thread Last edited by minsc_tdp; 10-04-07 at 08:28 PM.

10-05-07, 06:49 AM	#14
Galanti Captain Join Date: Apr 2004 Location: Ottawa Posts: 536 Downloads: 99 Uploads: 0	I've been using supersampled transparency for weeks, with no apparent effect. Maybe it's because I'm at 1280x1024? The Nhancer site claims this mode only works in 1024x768 and lower. What I did do was use the Silent 3D editor to change the field of view/perspective for freecam back to stock values in the Camera mod part of ROW and it's definately improved the wires. I definately prefer the stock camera freecam perspective over ROW's, although the rest of ROW's camera mod is absolutely brilliant.