If you're not planning to be a professional frame builder, owning a lathe and mill is optional. You'll get better results with less effort if you own the "big iron" tools, but a hobbyist can get by with a hacksaw and some files.

The big things to figure out are:

1) How do you plan to join the frame together? Lugs? Fillet brazing? TIG-welding?

2) How are you going to miter tubes? Hacksaw and files? Jig and a drill press? Mill? Lathe?

3) What sort of jig are you going to use to align the frame before you join the tubes together? If you're going to buy a pre-made jig (ex: Anvil) you'll need less tooling that if you're planning to build something from scratch.