You know, (not to dampen your spirit) it might have been the ramming that just tipped the destroyer over the edge, causing it to blow. I must say though, those 2 hits you did get must have been pretty devastating to lower the old hitpoints.
I just know that when I first started playing on stock SH3, a destroyer rammed me and the damn thing sunk, lol. I was out of torps so I used the sub as one...