Rollers that I am familiar with have a fixed distance between the two rear rollers. The smaller wheel would drop down between them more, resulting in higher forces on the rear tire since it is more "wedged" between the rollers. Unless rollers have a separate drag unit (fan, magnetic, etc.), most of the resistance comes from tire deformation LINK. Thus unless you drilled new holes for one of the rear rollers, the resistance might be relatively high. Even then, I suspect smaller diameter wheels just by themselves give more resistance on a rollers.

Second, balancing on rollers is largely dependent on spinning the front wheel at reasonable speed for the gyroscopic effect (except for those exceptional cyclists who can even track-stand on rollers). The smaller diameter wheels would have less of this (unless you put a weighted strip of lead or something inside the rim instead of rim tape to increase the wheel's moment of inertia). Thus balancing would be even more difficult.

Undoubtedly it could be done, but it would not be easy. Large diameter rollers (4.5") would helpful.