Like CaptainCool, I also have a non-mapping Garmin Etrex.

GPS Pros:
No calibration.
Speed, distance, avg speed all available.
Backlight.
Can get you back to where you started, regardless of how lost you are.
Tracks can be downloaded into computer for analysis. (Requires additional software)

GPS Cons:
May require several minutes to acquire satellites.
May require several awkward button presses to get data you want.
No odometer function. (It does have a trip odo).
No auto start/stop function based on wheel motion.

If I'm going somewhere unfamiliar, I'll use the GPS. Otherwise, the Cateye on my roadie is easier. However, the computer on my commuter bike has recently failed, and I'm going to use the GPS instead of replacing it.