Knee pain after 3 miles is not a good sign. Make sure you shift often and choose gears that fit the sweet spot - fast cadence but not too fast (no so fast that you feel like a hamster in a wheel), pushing hard enough that your legs feel like they're working, but not working too hard - an effort level that feels like you could do it all day.

If you get this right - get into a good groove - you really can ride all day. Back in the day, when I got my first adult bike - a 10 speed - I started with rides up to 10 miles and thought they were long. Then I started riding longer and longer, and discovered the real limiting factor was my bottom. After 25 miles I still felt great, but my bottom was starting to hurt. After 40 miles I still felt great, except my butt was numb and very painful.

In the ensuing decades I've done a lot of research, bought a lot of saddles, and mostly solved this. Now I can ride a century and not suffer too terribly (although I always suffer when I ride 100 miles - I don't know how those double-century people do it!) I take multi-week tours where I'm riding 40-60 miles, day after day (that's what really tests your butt comfort!) and don't suffer too terribly. (I get off the bike when my butt starts to hurt. A 15-minute break does wonders!)

Keep riding, work on shifting, cadence, and effort level, and you'll be riding much longer distances comfortably before you know it!