One race last season, I decided to race "up" a category, and entered the 3/4 instead of the 4's. When I showed up, I was surrounded by guys who race 2's on the road, including a couple of very strong ones. One of them was racing on a Schwinn Madison singlespeed with 25mm road slicks. He passed me some time in the first lap, and I never saw him again. As long as conditions stay dry, you'll be fine. It's about the engine, not the equipment.
The only caveat is that in cross, you need the right tires. Slicks are fine for dry, but he'd have been going nowhere fast in the mud.