The biggest issue with shock and vibration is probably flexing of the circuit board, which leads to fatigue and broken solder joins, broken traces, etc. This is usually more of an issue with larger boards. That's why big boards aren't just mounted on the edges, but have standoffs all throughout the area of the board. A smaller board is going to be stiffer and lighter, so it's less of an issue. Still, I've seen some high end electronics (e.g. pro SLR cameras) where everything is mounted on shock absorbing materials. There's also the issue of connectors shaking loose, such as the connection to the display.

The biggest problem I've had with portable electronics is the batteries shaking loose and the unit shutting off. This is usually with AA or AAA batteries. Never had an issue with a built-in battery.

P.S. I second the recommendation for at least a ziplock bag to protect the phone from the elements. Rain, sweat, dust, etc will take their toll. I have a Polar F11 HRM that has logged pretty close to 10K miles on my handlebars. It still works great, but the glass is noticeably pitted