Why did the Jeffersonian world view (that of the yeoman farmer as the bedrock of American democracy and the American economy) disappear by the mid-nineteenth century?

Jefferson's worldview did not weather the industrial revolution and rapidly changing United States. The idea of a farming republic could not survive a rapidly growing international economy. While agriculture was important to the young country, dynamic trading networks and growing metropolitan centers quickly dominated. Most students of United States history are aware of the movement of populations from rural landscapes to increasingly densely populated urban centers. As such, power flowed to cities with populations.
But it was not simply that industry pushed out farming—rather, values changed. Value and pride no longer resided on the plantation, and urban denizens were seen as valuable, trendy, and powerful. European fashion flourished, as did foodways and language patterns. So, as social capital moved to cities, the power dynamics of this country changed, reducing the influence of farming. It became untenable for farmers to be powerful, because the urban class was on the rise. Cosmopolitanism and travel were increasingly valued. Being the lord of one's plantation could not compare to trans-Atlantic travel, mercantile economies, and cross-cultural interaction. Ultimately, Jefferson's vision was unable to endure these demographic and social capital changes.

I would suggest the answer to this question lies at least partly in the fact that Jefferson's worldview was formed before the Industrial Revolution began.
In 1782, when Jefferson wrote his Notes on the State of Virginia, it was still possible to believe that a major country's economy could be based primarily on agriculture. Moreover, Jefferson was an idealist, a man whose thinking was not always realistic. This perhaps accounts for his inability to see the contradictions in his own life and his worldview. In the Notes he denounced the institution of slavery in the strongest terms, yet he never took any action to further the cause of abolition: he freed only a handful of enslaved people in his long life, and did not take the action—as George Washington did, for example—of liberating all of his enslaved people in his will. During the French Revolution, he refused to believe that abuses were being committed by the revolutionary government and even became angry at William Short for reporting to him from Paris on the facts. He went massively into debt, leaving it to his daughter to pay off his creditors after his death by auctioning off his estate and his enslaved people. While all of this may be seen as personal failure, it relates to his lack of realism concerning the economic basis for modern society.
By the beginning of the nineteenth century, the Industrial Revolution was invalidating the ideal of yeoman farming that Jefferson cherished. The South began to fall behind the industrialized North, where manufacturing created wealth and eventually created a huge disparity in economic power between the states on opposite sides of the Mason–Dixon line. The fact of Jefferson's own enormous prestige and stated ideals may have actually impeded the progress of the South, both in terms of the slavery issue and the South's lagging behind in technological advancement. By 1850, the agrarian basis of the South was essentially obsolete, a relic of the past.