doesn't offer any rules for how far a ship can "see"; however, the DMG does have some default rules about visibility outdoors. According to these rules, characters can normally see about 2 miles out from ground (or sea) level, and 40 miles out if looking from a vantage point such as a hill or mountain "or are otherwise able to look down on the area around them from a height", though rain or fog can significantly reduce visibility (down to 1 mile in heavy rain, or merely a couple hundred feet in fog).
Applied directly, that would suggest that the crew of a ship in clear weather should be able to see about 2 miles (land, not nautical) to the horizon from the deck or 40 miles to the horizon from the crow's nest atop a mast. These rules don't offer any kind of middle ground between those extremes, and a viewing distance of 40 miles is not realistic for an actual ship, on which the crow's nest will be at best a few tens of meters above sea level - but it is all that the rules have to say about how far characters on a ship might be able to see.
Luckily, the formula for estimating horizon distance based on vantage height is simple and, if you assume the planet you're on has earthlike dimensions, there are online calculators which will do it for you. The world of Oerth from the Greyhawk campaign setting (in which Ghosts of Saltmarsh is set) is almost exactly Earth-sized, according to the 2e sourcebook The Adventure Begins (p.9):
Careful mathematical measurements and magical divinations reveal that the circumference of Oerth is 25,200 miles. Thus, the diameter of Oerth is about 8,021.5 miles...
The actual Earth's circumference is is about 24,900 miles, so the values determined by an Earth-based calculator will be near as makes no difference to Oerth. Popular alternative campaign setting the Forgotten Realms' default planet, (Abeir-)Toril, hasn't been specified with such precision so far as I can find, but the the 2e A Grand Tour of the Realms (p.4) does state it is "Earth-sized", so Earth-based vision measurements should also be valid there.
So, using such a calculator, we can easily figure out some more realistic vision distances for your ships. Using the heights of masts given for sample ships in Ghosts of Saltmarsh Appendix A (and assuming, possibly inaccurately, that the given mast height is measured from sea level rather than the deck), we get the following vision distances:
Galley. One 120ft mast. 13.4 miles to horizon.
Keelboat. One 10ft mast. 3.9 miles to horizon.
Longship. One 20ft mast. 5.5 miles to horizon.
Sailing ship or warship. Three 80ft masts. 11 miles to horizon.
Larger ships with taller masts have an advantage in being able to spot other ships from further away. Of course, larger ships should also be easier to spot from a distance as their masts will be visible at much greater distance than the body of the ship... but observers from the top of a mast on a large ship would be able to see the body of a smaller vessel while only the tip of their mast is over the horizon, so would almost certainly spot the other ship first and could probably skirt it without being noticed.
In any event, ships that are close enough to meaningfully interact with each other would be able to see each other from their decks - barring unusual weather such as extremely heavy fog, in which case the vision range is up to the DM's determination of the weather. Once you know your height of eye you simply plug that into the following formula:
1.17 times the square root of your height of eye = Distance to the horizon in nautical miles
For example, if your height of eye is 9 feet above the surface of the water, the formula would be:
1.17 times the square root of 9 = Distance to the horizon in nautical miles.
1.17 * 3 = 3.51 nautical miles
If you want to calculate the distance at which an object becomes visible, you must know your height of eye and the height of the object. You then do the same calculation for your distance to the horizon and the object’s distance to the horizon and add the distances together. For example:
You have the same height of eye of 9 feet so your distance to the horizon is still 3.51 nautical miles. You’re approaching a port that has a lighthouse that is shown on your chart to have a height of 81 feet. Using the same formula you would find that 1.17 times the square root of 81 (1.17 * 9) = 10.53 nautical miles (the light house can be seen 10.53 nautical miles over the horizon)
By adding the two together: 3.51 + 10.53 = 14.04 nautical miles, you should be able to see the lighthouse when you are 14.04 nautical miles away.
SEE....SIMPLE 