What is the speaker of the poem upset about in the fourth, fifth and sixth stanzas? Why?

The speaker, in "To a Mouse," is contemplating the catastrophe which has befallen the mouse with the destruction of its nest in the field, which he has inadvertently caused while plowing. He reflects upon the amount of time and effort the animal must have taken in building the nest, and the suddenness with which all of it has been destroyed, leaving the mouse exposed to the elements as winter is approaching.
Despite the humorous tone of the poem, which has made it such a favourite of readers for generations, the speaker feels genuine sorrow for the mouse, whose predicament is emblematic of the vulnerability of any living being. In stanzas four through six he is saying by implication that we as humans are the same as a small rodent, because our planning and work can as well be destroyed in an instant.
The final stanza, however, embodies the deeper meaning of the poem: that man is even less fortunate than mouse. It is not only his vulnerability to outside forces that creates the tragedy of human life: man, as the most intelligent creature, is even more victimized by his memory of the past and his anxiety about the future.

But och! I backward cast my e'e
On prospects drear,
And forward, though I canna see,
I guess and fear!