Are the poems of George Herbert on Affliction controversial?

They are not controversial. In fact, it was rather typical, in Herbert's time, for authors to write about spiritual journeys that included moments of backsliding into sin, doubting one's faith, and so on. Therefore, for the speaker of Herbert's poems to admit to having such difficulty in sticking to the spiritual life that used to bring him such joy and happiness is neither a unique nor a surprising position. In "Affliction (I)," the speaker enjoys a satisfying, if somewhat superficial, spiritual life full of blessings and "pleasures" which was "no place for grief or fear." Life is good—"There was no month but May"—but as the speaker ages and begins to experience his own physical decline, as well as the decline and deaths of his friends, his breath is "tune[d] . . . to groans." Suddenly, the speaker feels as though he has no anchor, that he is carried off by each figurative "wind" in his life. He feels a sense of betrayal, as though God has abandoned him. In the end, however, the speaker continues to return and hope that God will not continue to forget him; he vows to remain "meek" and "stout" in God's service. Ultimately, then, the poem is still quite pious, as the speaker accepts God's place in his life, whatever that place is.
Further, in "Affliction (II)," the speaker acknowledges that the death Christ suffered on the cross is more than all of the pain the speaker faces in his life. He feels that he must lay all his happiness and his sadness at God's feet, giving thanks for both. In "Affliction (III)," the speaker acknowledges that God's breath "gave [him] both life and shape" and that the grief he feels is "guide[d] and govern[ed]" by God, that it will be used to help him. In "Affliction (IV)," the speaker once again laments the painful circumstances of his life; however, he expresses his belief that "those powers, which work for grief" will, he says to God,

Enter thy pay,And day by dayLabour thy praise, and my relief;With care and courage building me,Till I reach heav'n, and much more, thee.

In other words, then, the speaker carries on this expectation that God will afflict him with suffering for some purpose—perhaps to prepare his soul for heaven.
Ultimately, then, we see that the speaker retains his faith despite what happens during his life, despite what afflictions he must bear. He finds a way to credit God for his happiness, and he assumes that God will use the sadness and pain that he feels for some greater purpose. He accepts God's workings in his life and tries to remain grateful, even when it is difficult. Thus, these poems are saved from controversy, as they remain firmly faithful to God.