Sink it. If the wreck icon is red and you gain renown, you've brought down a hostile. Otherwise, you've *******ed up.
If they turn their spotlights on you, there are two possibilities:
- It's from a hostile nation, or
- It's neutral, but it considers you a "renegade", read "hostile", as you've attacked a ship from the same nation less than 24 hours ago.
In general, do not engage targets until you're sure of their nationality, unless you simply can't get a good look at them. Even in enemy ports there'll be neutrals, and the game's not corteous enough to distinguish between neutrals that are off-limits and neutrals you're allowed to attack.