It can make a difference but most of the benefits are not seen until you're traveling at a high speed (23-25mph+). This is due to two factors:

1. The stiffness of aluminum is enough that a low wattage will not flex it very much (pedaling at 200 watts on aluminum vs carbon will result in almost no loss of wattage to flex). Only at higher wattage is the flex enough to cause a noticeable loss

2. Aerodynamic drag increases exponentially. For example, going 20mph takes 239 watts while 15mph requires 110 watts (for the default rider stats seen here: http://kreuzotter.de/english/espeed.htm). That's 129 extra watts to increase 5 mph. But increasing another 5mph to 25mph, requires 446 watts (again for that "default" rider). That's 207 extra watts! So a more aero frame will make a greater advantage at higher speeds.