Yeah, that's what I decided to go with. "My experience with him relevant to your position is limited, but this is how I knew him...In our interactions during his time at_______he seemed professional, polite, and intellectually curious. If you want to discuss further or have additional comments, please feel free to contact me."
I'd hope that, in the degree(s) he has apparently gotten since then (needed to be eligible for this position), he has found someone who can provide him a much stronger reference. If not, I'm sure the director of personnel can figure out what that means. I've had several people apply to work for me recently and all their references are as old as I am to this person with nothing in between (even though they've been to 1 or more schools/jobs in that time), and I took that as not a very positive sign. If I still had this former student's contact info, I'd let him know that personally.
Apparently, I liked him well enough 4 years ago, so I tried to make the letter as positive as I could given our limited interaction and complete lack of relevance to the position he's applying for.