Very true, a lot of sci fi writers have run through various scenarios.
For example, assume a highly advanced race had developed space travel 100,000 years ago, visited earth and then disappeared 95,000 years ago, there would likely be no proof they ever came and we would probably never know about them.
100,000 years is a blink of an eye in our 5 billion year old universe.
__________________
|