Providing the bike hasn't been abused, it's all in your head. I'm 64 yo, 245#, 6'6" in height and ride an '84 Peugeot that I bought new. It is steel-framed (Carbolite) and I have no worries about it.
Jon