Biggest suggestion is just more miles. It's only 4 miles, so you'll really need to be hammering to get a decent workout, I think.

Also, you don't have to dress in lycra, but if you're wearing baggy clothes, they will slow you down as much as anything.