Do you have any crew skilled in torpedo loading? if not i recommend you invest in some, also an officer skilled in this skill makes things a wonder to behold, i spent 500 renown on an officer skilled in gunnery, engines and torpedo's well worth the investment the experience level of your crew doing said job also makes a difference I think and how tired they are etc.
The green bar above each compartment tells you how efficiently its running try and max that bar out to get things running nicely
with realistic reloading times on I can have a tube reloaded in about 10 minutes real time.
thats using my uber officer + 3 crew skilled in reloads + any extra random seamen i might if my crew is a bit tired at the end of a patrol/engagement