How was Jean Valjean mistreated in Les Miserables?

Jean Valjean is a very dynamic character, and he changes and grows throughout the story.
At the beginning, he is released from prison after serving nineteen years. He was jailed for stealing a loaf of bread in order to feed his starving family. Additional years were added to his sentence because he tried to escape. A character like Javert would say this is not mistreatment, as Valjean is simply being punished for his wrongdoings. Readers might have more empathy and argue that the law is too strict and therefore it is mistreatment—he only tried to escape in order to take care of his family.
When he is released, he is ostracized from society. He ends up with the Bishop because no one else will give him a job, food, or shelter. We can see this as mistreatment from society because even though he is an ex-con, his time in prison was his punishment, so why does he continue to be punished by society?
Although most people mistreat him, the Bishop does not; the Bishop even forgives Valjean for robbing him. The Bishop's kindness encourages Valjean to turn over a new leaf and live a more moral life.