I've polished aluminum many times with sand paper with glorious results. Don't worry about sanding the detail off, it won't happen unless you use 80 grit and a power tool. Polishing rusty steel will take longer than aluminum.
Start with 400 wet/dry (use it wet) to get the rust areas down to good metal, then switch to 800. It's going to take some elbow grease. Then finish up on the bench buffer with polishing compound.