Historically 57mm armour piercing shot from the Tse-tse Mosquito was necessary. The pressure hull was at least 18.5mm of steel, so in practice (taking into account angle, range and water in the way) it was proof against all machine guns and even 20mm cannon. The conning tower was at least as well protected. See the career of U441
https://uboat.net/boats/u441.htm
The weak points were the crew manning the flak guns - a quad 20mm needs a whole bunch of loaders - and the saddle ballast tanks (which were relatively small and not essential for safe operation).
An RP-3 solid shot rocket could go into the water then straight through the pressure hull and even, allegedly, out the other side. The pressure bulkheads within the boat were not very strong, so any sizeable hole would prevent the boat from diving and could sink it outright.