Why does the speaker say, "Stop here, or gently pass"? To whom is this instruction given in The Solitary Reaper?

There is no definitive answer to this question. It is a testament to the poem's complexity and ambiguity that we're not absolutely sure whether the speaker is addressing himself, the reader, or both. On the face of it, it would seem that the speaker is thinking aloud. Having just come across the solitary reaper, he has reached a moment of decision: should he quietly continue on with his journey, or stop and listen to the reaper as she sings her song?
Later on in the stanza we have our answer; the speaker has decided to stay. Not only that, but he implores us to follow suit; we too must listen to the reaper's song:

O listen! for the Vale profound
Is overflowing with the sound.

Once again, we see the ambiguity involved here. The speaker slips imperceptibly between addressing himself and the reader. It's as if having discovered such a wonderful sight—and sound—he wants the whole world to know about it.
https://www.poetryfoundation.org/poems/45554/the-solitary-reaper