AFAIK, stock game contact colors show IFF. Some mods change this - GWX and NYGM do. Don't know if realism settings can change it. I've never played stock.
I don't use the stealthmeter, so I've never seen what you describe. I'd guess that someone is catching a glimpse of you through the mist, then losing the contact as they move out of range. At TC512, you don't ever notice them. If enemy locks on and starts to approach you, hopefully your lookouts will spot him, report, and drop you back to TC1 in time to react. You can set maximum TC for various circumstances at Data/Cfg/Main.cfg or with SH3 Commander. I strongly recommend using SH3 Commander, even with stock SH3. It's not a mod, but a front-end program which launches SH3. It has a lot of nice features which add to game immersion.