Will the 2500 make my light brighter on a fresh charge or just last longer?
Right, the terminal voltage determines the brightness -assuming the battery is charged. So increasing the capacity by 500 mA has no effect on brightness.
However as others have mentioned, different battery types deliver different voltages -even if they are the same cell size. Whether are not your light will dim as the voltage drops has to do with the particular circuitry of your light.