There wasn't an alignment screw in there to begin with? I bought my CC as a complete a few years back and it had them in there.
Anyhow, there's a long spring-loaded screw that belongs in there to make the task of wheel alignment easy after changing a flat. Due to semi-horizontal drops, you need to make sure your wheel is in the same place to keep the brake pads lined up properly on the rim when reinstalling the wheel.
Spares of those alignment screws are about $2.00; talk to your local shop.