A workshop handcleaner can dissolve oil pretty well and wont harm clothing. You can then wash the stuff out with water.
Wax-based lubes wont mess up your trousers.