Who takes away the spirit of the man's life in "The Man With the Hoe"?

Edward Markham's "The Man With the Hoe" is a poem inspired by Jean-Francois Millet's painting"L'homme à la houe." In both the poem and the painting, the viewer/reader is shown a man who is looks decrepit. In the poem, after four lines of description of the man, who holds "[t]he emptiness of ages in his face," Markham asks

Who made him dead to rapture and despair,
A thing that grieves not and that never hopes,
Stolid and stunned, a brother to an ox?
Who loosened and let down this brutal jaw?
Whose was the hang that slanted back this brow?
Whose breath blew out the light within this brain?

Markham, too, questions who is responsible for this man's condition. To leave a person so downtrodden, so broken down, is inhumane. Someone needs to be held accountable for the way in which this man has been treated. However, there is no one person who can be blamed for the man's condition. Instead, the blame lies with the workings of society in general. Markham goes on to state that the man is "[s]lave to the wheel of labor" and blames "masters, lords, and rulers in all lands" for the man's condition. All that has happened to the man is the result of "the immemorial infamies/ Perfidious wrongs, immedicable woes" created by unfair labor systems. Until those in charge begin to treat their workers fairly, the problem will persist, and people will continue to be worn down simply by trying to make a living.