I follow a strict policy:
If it fires at me,
It dies by me.
Also I don't attack unarmed merchants below 1000-3000 tonnes unless its a bad day or their part of a convoy and have escorts
Like Capt. Teach I'd never attack a hospital ship the enemys wounded deserve their rest same as ours.
I run a quick check of all unconfirmed targets:
1. Check the flag, if unable to check flag go to step 2
2. Check ship type
3. Check to see if armed, if not then skip step 4
4. Move to "Blindspot" where they can't hit you(If impossible then not worth the risk as this is a warship/battleship)
5. Surface, man deck gun and fire a shot/SS over the ship, but don't hit it.
6. wait for it to react, if it lights you up its most likly hostile.
7. If armed check to see if it is trying to aim at you.
If it was unarmed surface go over their check the flag from short range.
I run this check every time I cann't see the flag and also never try this or try to attack a warship if you don't know who it belongs too.
Happy Hunting