I don't know th X-Gen derailleur you are talking about, but amongst Shimano front derailleurs, the amount of cable pulled is different between road and mountain derailleurs. Don't ask me why... On the other hand, mountain and road rear derailleurs are compatible.
Friction shifting is definitely an easy solution. Either install a downtube or bar-end left shifter (any chance you could buy the left shifter off someone who upgraded from 8 to 9 or 9 to 10 speeds?), or move to Campagnolo Ergo ($$).
Other solutions, depending on the exact nature of your problem.
– If you moved your chainline closer to the frame, are you sure your derailleur is able to go close enough to the frame? Check to make sure your derailleur isn't bumping onto the seattube or its internal mechanisms.
– Install a
N-Gear Jump Stop. That way, you can set the inner limit screw a little bit further so the cable moves more inboard. The Jump Stop basically prevents you from overthrowing the chain.
– Grind your Tiagra derailleur. Many people seem to have good success by grinding the Tiagra derailleur cage so it's shaped more like your outer chainring.