I have a three year old and a six year old. I ride to work every day. On nice days I bring the six year old to school on my way to work with a tag-a-long.

Unless your commute is extremely dangerous (e.g. through high violent crime areas), just keep calm and carry on. And find somebody else to eat lunch with.