That depends on where those far awy contacts are heading...
If they sort of intercept your course at some point it's worth keeping an eye out at the approximate time you're set to intercept, maybe have a quick dive and listen, who knows what you might find...
I've bagged myself 2 King George V Battelships by doing exactly that, which is an extreme example but you shouldn't let tonnage get away from you, ever, at all, even if it's only two medium ships, always better to take those out than to take your chances with a bunch of hyped-on-coffee convoy escorts