Sure, depending on the selection they have. Some bike shops only have a handful of second hand bikes in stock, others have 20-30+ at any given time. Also depends on the turnover they have. Some shops will sell a lot of volume second hand, and if you don't find what you need, come back in 2 weeks and they will have many new ones in.

Going second hand will definitely require more patience to find a bike that is your size, fits your needs and your budget, but you will probably end up getting a better deal (higher quality for the money probably versus buying new entry level).