Why is Mr. Knightley confident that Mr. Elton won't choose Harriet in Emma?

Mr. Knightley tells Emma quite forcefully that whatever she might think, Mr. Elton will not marry Harriet. First, he notes that Harriet is the illegitimate daughter of nobody-knows-who. Being illegitimate is a stain in this society that a man of Mr. Elton's stature is unlikely to want to bear. Further, as Mr. Knightley points out, Harriet has no dowry and there is no sign that her family, whoever they are, wish to elevate her status above that of a border at Mrs. Goddard's School.
Mr. Knightley also informs Emma that he sees Mr. Elton in all-male company, where men talk in a different way than they do around women. From conversations in that setting, Mr. Knightley is clear that Mr. Elton has his eyes on marrying money and plans to marry a woman with a good dowry, which is, in fact, what he does do.
As Mr. Knightley states:

"Elton is a very good sort of man, and a very respectable vicar of Highbury, but not at all likely to make an imprudent match. He knows the value of a good income as well as any body. Elton may talk sentimentally, but he will act rationally. He is as well acquainted with his own claims, as you can be with Harriet's. He knows that he is a very handsome young man, and a great favourite wherever he goes; and from his general way of talking in unreserved moments, when there are only men present, I am convinced that he does not mean to throw himself away. I have heard him speak with great animation of a large family of young ladies that his sisters are intimate with, who have all twenty thousand pounds apiece."