What genre is "The Yellow Wallpaper"?

Genre is never something that can be cleanly simplified to a single classification—a story or novel can and usually does fit multiple genres, all at the same time. "The Yellow Wallpaper" is no different.
First, I think it's safe to classify it as a work of epistolary fiction. Epistolary fiction is fiction which is told through a sequence of documents, usually produced by characters within the story (to give an example, one of the most famous works of epistolary fiction is Bram Stoker's Dracula). Given that "The Yellow Wallpaper" is told as a sequence of diary entries written by its main character, it qualifies as such.
In addition, given its thematic content, we can classify this short story as both a work of feminist literature and a work of psychological fiction. These are just a few examples of classifications which we can apply.

When this story was first published, people weren't really sure how to interpret it, and many read it as a work of Gothic horror (with strange or supernatural happenings, scary settings, and dark characters). Most people did not understand that Gilman meant to criticize the medical establishment and the common practice of failing to take women's illnesses seriously or consult the patients themselves in their treatment. The doctor referenced in the text, Weir Mitchell, was a real-life doctor who created the "Rest Cure" for women who suffered from some nervous disorder or "hysteria." This was a sort of catch-all diagnosis for women with depression or other mental illnesses. In fact, Mitchell once treated Gilman for what we would now probably call post-partum depression, the same ailment from which the narrator of this story seems to suffer. Now that most readers are savvy enough to understand what Gilman was doing—she was certainly ahead of her time in terms of her feminist criticisms of society and medicine in general—we can more accurately term this work one of literary fiction.

"The Yellow Wallpaper" could be classified as psychological fiction, social realism, short fiction, or women's literature.