That's actually not true. Paleoclimatologists use cores from glaciers and ice packs to get information about atmospheric gases, carbon loads, and net precipitation that goes back tens of thousands of years. More information comes from pollen and particles embedded in the ice cores and in lake sediments. The pollen tells you what kind of plants were dominant, and since plants have very specific climate requirements, that gives an indication of the minimum/maximum temperatures. Other particles such as ash and volcanic dust can show specific events that affected regional climates. Then there are tree rings, which give year by year indications of good or bad growing conditions. All of these layers, ice, sediments, and tree rings, can be cross-referenced with great precision, which makes it possible to distinguish regional climate changes from global ones.

In addition, we have written records from Europe and Scandinavia (probably elsewhere, but my field was medieval European history, so this is what I know) that go back far more than 100 years. People were not measuring temperature with the precision we can now, but they sure as heck paid attention to the weather year to year. Good crops and bad, floods and droughts - those records go back 2000 years. They come in the form of stories and tax records, but the information is there, and now historians are correlating the written record with the above mentioned paleoclimatology approaches.