"The Murders in the Rue Morgue” is a tale reliant on realism rather than supernatural, as a means to confront the unknown. How did this same approach in "The Fall of the House of Usher" lead to the creation of detective fiction—a subgenre of the mystery story which focuses on observation and analysis of evidence?

It’s true that the menace in “The Murders in the Rue Morgue” is one of heightened naturalism and isn’t supernatural in origin, however exotic or unreal. A Poe story more commonly cited as the birth of the investigative detective tale, “The Gold-Bug,” features a resolution of apparently unrelated phenomena as revealed through mathematics. In all cases, including that of “The Fall of the House of Usher,” the author (and thus narrator) means to pull back the veil of initial perception to allow us to better grasp aspects of enigmatic human behavior.
"The Fall of the House of Usher" is well grounded in the Gothic, a genre which consists of highly ritualistic markers—the gnarled landscape, various ruined edifices, meteorological violence, and so on. But it is a mystery story in essence, an investigation (if vague in terms of its resolution). Does human corruption, in some literal way, taint all that it touches? Is some supernatural agent at work in the pollution of the tarn or the plague upon the house? The narrator is compelled to apply logic to evidently irrational or erratic events as he moves through them.
In this case, the setting itself, which should evoke stability, seems to have taken on a different character.

. . . how familiar was all this. I still wondered to find how unfamiliar were the fancies which ordinary images were stirring up.

The environment is oppressive and unwholesome.

An air of stern, deep, and irredeemable gloom hung over and pervaded all.

This funk can be seen to radiate outward from Roderick Usher himself: he who “suffered from an acuteness of the senses,” and, from a psychological standpoint, a co-morbidity (multiplicity of diseases). Poe draws a parallel between the compromised material organism that is the house and the psychosomatic sufferings of Roderick's racked body and naked soul. To a great degree, the mystery here is existential, or metaphysical. We are shown the house’s physical dissolution. Does that mirror Roderick’s numerous pathologies? Can that be?
So, within the categorization of these as detective stories, it could be argued that Poe’s narrators, in a Gothic framework and with an emphasis on deduction, must rely on their analytic powers to plumb the insolvable mysteries of the human soul. Their investigations concern that which motivates our organic existences—and, by extension, our place within a hostile cosmos.
https://www.britannica.com/topic/The-Murders-in-the-Rue-Morgue