When we view people with our dichotomous thinking, we often thinking of them as either perfect or horrible (this idea is known as splitting). While it can feel good to be thought of as perfect, it is also anxiety provoking and setting others up for failure.
A perfect example is clergy. As a society, we often view clergy as perfect (or supposed to be perfect). A perfect example of this expectation of perfection is when people make mistakes. We've had several big-name pastors "fall" in the last couple of years.
We're never shocked when an actor has an affair (or even a politician), but a pastor? What makes them so special that they are free from temptation and sin? Really, they're more susceptible to such problems because of how the church community uses and abuses pastors.
So we hold this view of perfection, then the person doesn't meet that expectation, and so they move to the opposite extreme. This is not healthy, helpful, or biblical. Jonathan Brink wrote a nice post regarding Ted Haggard and the shock people have that he could start a new church. He also has nice, but sad joke about the situation.
But seriously. Why can't he start and led a church? He should not have been kicked out of his last church, in my opinion. He's human. He's made mistakes. That's what would make him a GOOD pastor! But then again, he moved into the area of sexual sin, the black hold of evil...