I will attack destroyers if there's a high likelyhood of hitting it.
They almost always sink after just 1 torp, and it's much more satisfiying sinking 'real' combatants than merchants
In most of my careers I have an average of 1 in 3 ships sunk being some sort of warship.
I will almost always take on a lone DD but never bother with them if they are in a TF or Convoy.