If your using any of the big mods ie: GW or NYGM then its all to do with flooding the different compartments in the ship....try hitting below the waterline along the length of the ship...you should here a cheer from your crew when the ship is about to sink/the ship icon should change also