That's odd that you couldn't get them to engage any further out than 9km.
I made similar tests for my own purposes a while back with only the following changes to the stock 1.3 game.
AI_Visual.dat > MaxRange increased to 40000 (was 9500)
Sim.cfg > MaxFireRange increased to 40000 (was I think 6000)
Scene.dat > View distance increased to 40km
Scene.dat > Earth Radius increased to ~ double whatever the stock value was
In the test mission I had two Yamatos and two Iowas engaging at ~ 33km. If anything the view distance does infact seem to dictate the engagement range regardless of what the sensor and weapon ranges are set at. And I'm not so sure how well a 40km view distance would sit with most PC's.
