In line 5 of "Porphyria's Lover," how does the speaker feel? Why do you think this is?

In the fifth line of the poem, "I listened with heart fit to break," the speaker seems to be feeling something like despair or near hopelessness. Perhaps he is afraid that Porphyria, his lover, will not show up as a result of the rain or for some darker reason (for example, she later reveals that though she loves the speaker, there are some other responsibilities or obligations that prevent her from being able to commit to a relationship with him).
The speaker listens to the storm and wind outside as he awaits the arrival of Porphyria, and he does not seem optimistic that she will arrive. Then, when she does, he all but ignores her until she reveals her shoulders and lets down her long, blond hair. Finally, when she tells the speaker that she loves him but cannot stay with him, he strangles her with her own hair in order to keep her by his side.