I think one of the mistakes often made when judging potential is to use one's own priorities and viewpoint. If you think, "I wonder what I could achieve if I had that person's abilities", you're not judging their potential accurately; it's what they will achieve with their abilities that is important.

I see no problem with looking at potential, so long as you try to understand someone's motivations as well as abilities. However, being able to do that is a difficult skill to acquire, and that's one area where real managers differentiate themselves from those with "manager" in the job title.