How will you feel if you don't try? Better or worse than if you try and fail?

As for speed, your's is just fine. There will be slower people. And it isn't a race, it's a ride.
If you want to spend less time on the road, just be sure to spend less time stopped at rest areas and feed zones.

Ride within yourself and try not to get get caught up in fast groups.

So you get tired. Do your day, take a nap, go to bed early.

As you can see, I'm leaning toward DO IT. You'll forget the pain really fast, but you'll remember the ride forever.