Why did Sancho Panza not want to go with Don Quixote at first in Don Quixote de la Mancha?

Although Sancho Panza becomes a steadfast companion, he is initially hesitant to accompany Don Quixote on his adventures. Prior to asking Sancho to become his squire, Don Quixote had already returned from his first rather disastrous chivalrous outing, which only lasted a matter of days.
Don Quixote had gone no farther than a nearby inn, where he managed to mistake prostitutes for princesses, be "knighted" by the innkeeper, and get in a fight with other guests. Upon realizing he would need money for his journey, he headed back home, but not before picking a battle with a group of merchants and being soundly beaten by their mule-driver. A laborer from Don Quixote's village found him face-down on the road and helped him home, where his family put him to bed.
Already Don Quixote's adventures have been fraught with blunders, and his family and friends think him mad. When word gets around the village, who would want to let him venture forth again, let alone go with him?
Miguel de Cervantes does not explicitly offer an explanation as to any protests Sancho might have made, but he does note in chapter 7 that the peasant needed some convincing to join Don Quixote:

Don Quixote worked upon a farm labourer, a neighbour of his, an honest man (if indeed that title can be given to him who is poor), but with very little wit in his pate. In a word, he so talked him over, and with such persuasions and promises, that the poor clown made up his mind to sally forth with him and serve him as esquire. Don Quixote, among other things, told him he ought to be ready to go with him gladly, because any moment an adventure might occur that might win an island in the twinkling of an eye and leave him governor of it. On these and the like promises Sancho Panza (for so the labourer was called) left wife and children, and engaged himself as esquire to his neighbour.

We know that, in the early stages of their adventures together, Sancho seems to find Don Quixote and his ways bewildering, but he comes to respect the man and share in some of his chivalric ideals and fantasies. So it might be that, at first, Sancho was hesitant to embark on what might sound like a rather silly adventure with the local eccentric.
We are also told that Sancho is poor, so to suddenly leave his job and family would be foolish, even for a man described as one with "little wit." But when his master promises riches, Sancho Panza is quickly convinced. Thus, we can conclude that Sancho's original motivation to become squire to Don Quixote is dependent upon money.