This is how I see it, in order of priority:

Weight - yes, certainly, but I don't think that was the question here (see OP: "similar bicycles").
[BTW: Not all Peugeots are whippy-as-hell and not all whippy-as-hell bikes are Peugeots. None of my Peugeots is "whippy as hell". All made of Reynolds tubing though, no Vitus (maybe you should be more careful with inductive reasoning - the one you think of ATM doesn't have to be a valid representative for the whole crowd!).]

* Geometry - very important for the feel. Me myself I don't like too steep fork angles, but they are very nice for curvy courses; while a very steep frame with the wrong (core leather) saddle *can* take all the fun from biking (been there).
You simply have to get to know what you personally like most - try out as many different geometries as you can and you will see they can feel impressively different!
Frame geometry has many possible outcomes and many possible results in feel. It matters more than it seems (read everything that Mr. Merckx sen. said about this, very enlightening.)
[and BTW: touring geometry frames don't exhibit a "lack of stiffness". My advice: simply choose the right frame for the right cause, take a touring bike for touring and you will see these aren't "whippy-as-hell" nor "lacking of stiffness" but very well fit to the needs of the situation. And you will *want* a smart damping frame on lousy roads, you will soon learn to appreciate how well cleverly built touring bikes can smooth out uneven surfaces. It's a feature, not a fault!]

* Tires - tires will only "slow you down" if they have too much rolling resistance, e.g. if they get too few air, have too much "profile" etc. But type of tyres (tubulars vs. clinchers) can make a HUGE difference in feel, width of tyres as well!
... in conjunction with ...
* Wheels - yes, important, they WILL particularly slow you down or speed you up. There are differences even between aluminum rims (in weight, that is) and it doesn't matter at all if you're a good rider or not. Lighter wheels mean quicker acceleration. There *can* be a huge difference between a good vintage tubular wheelset and a common vintage clincher wheelset or a modern clincher wheelset (if you don't take the recent "ridiculous" price range). Wheel performance depends on weight of the rims, type of the spokes and built/builder (there's a reason why some very light wheelsets are expensive: they will be quicker in bringing you up the hill or up to the front).

Stiffness - not too important for the general feel, 'xept you have a *very* bad frame (or it's shot). Frame flex can be distracting if you need to sprint (do you?). Cheaper steel frames even tend to be stiffer than expensive ones (more of that cheap rigid steel, you'd call it "lack of responsiveness") - does it mean they are "better"?
BTW: there isn't such a thing as "pedaling power absorbed by the frame". Yes, there is such a thing as frame flexibility and many racers don't like it (tourers sometimes do, see above!), and it's not of much use if it's around the bottom bracket shell, but even then energy will never be able to "go through the roof" or else (we call it "physics")!


So all in all it comes down to
1. frame material and geometry and
2. wheels / tires combination
(or the other way round, depending on how much the pairs you compare differ [and people practically always judge by the pair comparison method])
... with ...
3. saddle and/or bars: can make a huge difference (but not for the situation that the OP described as "comparable in style and components".


The ever-lasting-top-question. Read CR forum, hear fans of the italian stuff talk.
It seems that sports cycling could be a very interesting field of research for psychologists.
Why?
[psycho mode on]
Many cyclists seem to *love* intuitive arguments (that doesn't mean they aren't able to come to a good, problem related judgement) and love to get lost in virtual or imagined preferences for equipment (shimano vs. campagnolo, the suntour problem, carbon, master builders - you name it ... ).
At the same time many cyclists seem to be *very* prone to prestige motivation in their consumer behaviour (no, not the pros. They just ride what they [can] get). It seems that at least some if not many (but not all) cyclists love to show off.
[psycho mode off]

From my POV there's no real reason to prefer a handmade "XYZ" (put any of the highly regarded frame builders here) frame over a handmade frame from some of the big names workshops (Gitane, Raleigh, Peugeot, Benotto, Merckx, Colnago). The main advantage of a top builder should be that he's knowledgable and flexible enough to be able (and hopefully willing) to find special solutions for most possible real applications/rider's problems, is precise in his work and in that way you're sure you won't be disappointed.
And certainly you'll have to pay for that. And from here the psychological dynamic begins to work (expensive - prestigeous - the rest of the crowd will be attracted like flies ... ).
Most of it seems to be pure psychology, playing "her" game with the guys ...

IMHO the difference in real usability is hardly worth mentioning. "Master" builder frames made of the same material with the same intent of use / roughly the same geometry and built by someone who does that every day (which was the case for handmade steel frames from big factories in the 1970s & 1980s) will have virtually the same riding characteristics.
And to make the whole thing worse: I know of examples where the variance in the build quality of different frames of one and the same builder was bigger than the build variance between this builder and the usual quality of a good "big player" workshop bike (we're talking about frames at least in the Reynolds 531 or Columbus SL league, are we?). There are even "worse" Masis out there, or at least such with some superficiality concerning the precision of workmanship.
A vast part of the magic of the "frame builder legends" seems to mostly depend on psychology, as I said, a variant of selective perception.

And, finally, remember: the majority of the existing Colnagos and Cinellis (and even Masis) weren't built by the men themselves (e.g. Colnagos were made in big quantities, some even brazed by smaller builders of the region with spare capacities)!

-----------------------------------

Sorry for the mass of characters, I just had to say it .
(Ahh, feeling better now .)