Why does James Baldwin tell his nephew that his grandfather was defeated long before he died in "My Dungeon Shook"?

According to James Baldwin, his grandfather was defeated long before he died because he believed what white people said about him was true.
Other people's perceptions of his grandfather tore him down and changed who he was. Baldwin says that perhaps this belief is why his grandfather was a holy man. But he cautions his nephew that neither he nor his father are holy in any way. He says they're the kind of people who moved to the "cities of destruction," as Frazier called them.
Baldwin mentions his grandfather in order to caution his nephew against being like him. He warns him of how white people will think of him. To avoid the same destruction, his nephew cannot believe or become what they say he is. He says that society decided that, because he was black, he was a worthless human being whose ambitions must be limited.
He tells his nephew that, instead, he has to be his own man and decide what kind of person he wants to be, without outside interference. He tells him:

Take no one's word for anything, including mine—but trust your experience.