On aluminum bars, any flaw can one day make them fail. Aluminum doesn't fail slowly like steel, it seems fine and then it's in half.
Steel on the other hand doesn't depend entirely on it's shape for structure and I'm sure you can drill at will. I'd keep it a decent distance from the stem though...