If they were in Iranian waters, and armed, the iranians have a lot of options.
What would the UK do if an Iranian Kilo surfaced outside the limit, and sent in an inflatable boat?
What would the US do in the same situation? Off Galveston, or Bangor....

I'm referring to an escalation once the West finds out Iran would tolerate things in their waters. First a helicopter, then a frigate, then a cruiser. Certainly this kind of "give inch, take yard" mentality is certainly what many in the West would think of Iran. It is hardly unreasonable that they harbor similar thoughts about the West.

Besides, if you assume that the rubber boats were in Iranian waters, then it is a breach of the law. Laws that, let's face it, were nominally drawn collectively by many nations but in fact are already greatly tilted towards the advantage of maritime powers (read, mostly the West). Is stealing 5 dollars not a crime just because it is 5 dollars?

What is the purpose and mission of the inflatable boat? If it came out of a Kilo they probably have a mission different from routine inspection of merchant vessels & information gathering from crews. They are probably special forces operatives on a special mission. Are they headed to a merchant vessel? Or to land? If the latter, intercept, fire warning shots untill they surrender. Be prepared to stop them at land. If the former, let them board the ship where they'll be stuck and ready to be arrested. If at any point you believe them to be carrying a nuclear artifact, shoot to incapacitate the crew. Keep track of the Kilo at all times, ready to sink it at any moment. Don't take any chances. They've gone through the trouble of getting a Diesel submarine all the way across the Atlantic and we were never able to discover it untill they surfaced. They certainly have an important objective, unless they have come to desert (if they're deserting they will probably be carrying white-flags). And we certainly need to improve our intelligence & submarine detection capabilities to prevent another Diesel boat from being able to leave Iran undetected, sail around Africa or the Mediterranean undetected and finally cross the Atlantic ocean undetected and reach our shores undetected.

The same thing applies to Iran. Today we get a little closer to the border, tomorrow we cross it, the day after tomorrow we routinely travel beyond our waters and finally one day we attack Iraq/Kuwait. Why is Iran exempted from any of the demands you make to everybody else?