When using imperial measurement units:
the dial for range (top right of the screen) uses yards (as it says on the dial). The height of the enemy ship is measured in feet.
(with 1.2 patch) if you place the mouse over the displayed range in position keeper interface, a tooltip will apear telling you again that the range is measured in yards. So the range is always displayed in yards, as far as the TDC is concerned.
About the ruler-tool in the navigation map: it uses a single digit after the decimal separator so when the distance is 1.04 NM it would display 1.0 and when the distance is 1.06 NM it would display 1.1. The value of the nautical mile is an average, 1853.9948 meters. The value of the actual nautical mile is linked to the length of one degree of longitude which varies with latitude, but this is not easy to deal with, so an average is used.