This is my understanding (admittedly limited):

Every ship/sub/plane has defined damage zones. These zones have different sizes, hit points, armor values, and 'floatability' values. They also have other factors, like 'critical chance' defined. Damage to these zones can result in flooding, fire, explosion, and smoke. Critical hits can occur. These are not guaranteed to destroy, but only to result in enhanced damage.

Weapons have defined damage points and armor penetration values. Damage is variable, so to will be the results of your hits.

The point being, there are a lot of variables to all this, and one cannot expect consistent results.

Watching ships sink in some tests I ran, it was clear that at a certain point in time, the game will 'write off' the ship, and it will start going down. I don't know how the game decides when this will happen. This would be nice to know.

Can't argue with that.

From my experience in RFB, the merchants seem to sink often enough, but like in the picture, they usually sink on an even keel. RFB made changes in the zones to change this, but they still have a tendency to stay on an even keel. There doesn't seem to be any modeling of free surface effects, or the like.

Yes, they will be unstable sometimes, but it seems they rather tend to go down too neatly. At least, that's my experience.