Nevermind, the problem fixed itself:
The destroyers decided to go near a port and just park there. All I had to do was sneak up on them from the side. Even when the first one was hit the second didn't even move away or even seem to notice. I hope that wasn't a glitch or problem with AI.
