What a great topic! There's lots of great advice here.
I, too, am a late learner. My husband taught me to ride a bike when I was 26. Like steve57, I started out practicing in the dark, scooting around our neighborhood on my bike around midnight because I would have been mortified to have the neighborhood kids observing my beginner's efforts. I didn't think to take the pedals off as others have suggested, but I did wear shin guards.
My big mistake was getting a bike that was just a little too big for my comfort. I am short, so it was difficult to find a decent bike in my size and my DH prefers over-sized bikes and hoped I would share that preference. Big mistake. I was shaky on that bike around other cyclists and around any kind of traffic. I had a very scary accident with a car (totally not the driver's fault) that ended with my head on the ground inches from her stopped-just-in-time tire. That shook my confidence and eroded nearly all my progress. DH and I got a tandem after that, and that's how we rode for years until he started having problems with his wrists, exacerbated by the heavy steering of the tandem.
I miss riding with my husband, who is still very much a bike enthusiast, so at age 51, I recently decided to give cycling another go. This time around, I decided to go for a small bike so that I would feel absolutely comfortable about being able to put my feet down on the ground without toppling. Because I wanted a bike with a small frame that would let me "grow into it" as I improved my skills and developed my confidence, I decided on a folding bike. Last week, we purchased a used Brompton. As a "starter bike," it may not be everyone's cup of tea, but you just need to think about what you'll feel comfortable with - then consider borrowing, renting, or buying used if possible.
Learning has been a little different this time around. We live in a rural area with a driveway big enough to practice in, which I have been doing once or twice a day for the last four days. I lowered the seat and scooted around, pushing with my feet and practicing my balance. Today, I managed to keep my balance and pedal up and down the driveway.
The Brompton has been GREAT for learning (re-learning) to ride! Plus I like that my DH can ride the same bike simply by adjusting the seat height. He likes it so well, that he's already shopping for one for himself.
Pardon my long first post, but it's just so good to know that I'm not alone in learning to ride a bike as an adult. A big thumbs up to all of you who are pushing past embarrassment, fear, whatever, and making an effort to learn!