Do you think the speaker feels guilty about what he has done?

I think the only feeling the narrator really registers is that they feel "Happy and proud" to know that Porphyria loves them (I use the gender-nonspecific "them" since we do not know for certain that the narrator it is male; they could be female). It is, after all, her love that the narrator seeks to sustain, and that is why the narrator strangles her. The narrator describes Porphyria as having blue eyes that "Laughed [...] without a stain" when they unwind the hair from her neck. The narrator describes her head as "smiling" and "rosy" and "So glad" that it can now have what it wanted: to stay with the narrator, whom Porphyria claimed to love before she was murdered by them. In the end, the narrator says that the pair of them,

"[...] sit together now, And all night long we have not stirred, And yet God has not said a word!"

The narrator seems to attribute God's lack of intervention as a silently condoning their behavior, as though, if God really objected to Porphyria's murder, God would have stopped it or would now punish the narrator. Therefore, the narrator does not seem to feel guilty; rather, they feel satisfied that they've found a way to possess and keep Porphyria forever.