I'm usually slower riding home, but I attribute it to several factors:
-- There's more elevation gain on my return route.
-- In the summer, it is substantially hotter -- usually 20-25 degrees.
-- More traffic, so greater likelihood of getting stopped at traffic lights.

Sometimes I ride faster going home or just as fast. Those are the days when I'm trying to beat a thunderstorm heading my way. Amazing how that will motivate you.