Go to Buckhead and Little Five Points. L5P has a place called Outback bikes which is pretty cool. A while back I bought socks that said "Police" on the cuf, and "got donuts" on the bottom. the best tatoo artist in ATL is in L5P as well.
Def. go to the Atlanta Gardens. Admission is 5 or 10 dollars and you can get lost in all the splendor.
Buckhead also has a number of bike shops, mostly catering to the elite roadie crowd, meaning they have geat used parts bins.