I own all 3 sizes. The one that I tend to grab the most often is 1/4". Very few things in my life require big sockets or big torque. My most frequent use is the car, followed by stuff in the house such as getting into hard-to-reach locations in appliances. I use a lot of the 1/4" handle, 1/4" socket, and a philips screwdriver bit in it.

Before committing to a size, I suggest checking the availability of hex and torx sockets, and a torque handle.