Is it true that the poem "In Memory of W. B. Yeats" represents Yeats as a man as well as an absolutely unique and irreplaceable poet?

This is an interesting question. What is really being asked here is whether Auden's poem is concerned with Yeats himself as a human being, or if it is simply about what his death meant to poetry, the loss of a literary giant.
Certainly, Yeats is celebrated as a writer—he is called "the poet" by Auden in the first part of the poem, as if this were his defining characteristic. In traditional elegiac style, Auden follows a three-part format: lament, then praise or admiration, and then consolation. But in the second section, Auden does take the time to praise and admire Yeats's humanity as well as his gift. He describes Yeats as having been "silly like us," a phrase which is celebrating not Yeats's poetry or his "madness," but his personal qualities. He also speaks of Yeats's homeland, Ireland, and how it stimulated ("hurt") him into writing. Yeats is not simply a vessel born to produce poetry, but a man with emotions from which that poetry was born, and a man with a personality outside of his poetry, too.
In the third part of the elegy, Auden returns again to impersonal language, addressing the dead Yeats directly as "poet." In this consolatory section, Auden seems to have moved beyond discussing Yeats specifically at all—instead, he is expressing his views about poetry itself. For the most part, then, this poem is far more about Yeats as a poet, and about poetry, than it is about Yeats as a man. However, especially in the middle section, we do see elements of Yeats's humanity.