As a u-boat captain I hunt merchant shipping. Killing a destroyer isn't going to help the war effort as much as sinking a tanker or cargo ship full of war materials.
Those harbor escorts do tend to travel in odd angles at odd times and change speed constantly. I'm not sure how you'd compensate for that other than getting really close in.