Kings stand on e1 and 8. If opponent figures threaten d1 or f1, d8 or f8 (these are the fields the king would "jump over"), casteling is not possible as long as that threat exists, since the king is at all times prohibited to move so that he exposes himself to check - and he would need to pass through these checks.
Verboten!
No problem with the Rooks on a1 or a8, if an opposing piece threatens b1 or b8, castleing still is possible.