Sometimes the only way to identify a flag (if its a single merchant) is to simply get them all riled up by fireing a deck gun shell at em.
If you simply cant identify the flag outside of minimal torpedo range, simply forget about the ship.
you dont gotta take out everything you see.