Why does Antipholus of Syracuse think he cannot have a good time in Ephesus?

Antipholus of Syracuse (a city on the Italian island of Sicily) and his servant, Dromio, also of Syracuse, have recently arrived in Ephesus (a Greek city on the western coast of present-day Turkey).
Unbeknownst to Antipholus of Syracuse, his twin brother, also named Antipholus, lives in Ephesus, and Antipholus of Syracuse is constantly mistaken for his brother, Antipholus of Ephesus.
To complicate matters further, the twin brother of Dromio of Syracuse also lives in Ephesus, and he, too, is constantly mistaken for his own twin brother.
Shakespeare cleverly expanded on the ancient Roman comedy Menaechmi, by Plautus, which is the basis for The Comedy of Errors, to include two sets of twins: the Antipholuses and the Dromios. Menaechmi has only one set of twins: the brothers named Menaechmi.
Antipholus of Syracuse's first merry mix-up involves Dromio of Ephesus, whom Antipholus of Syracuse believes to be Dromio of Syracuse, and who Antipholus of Syracuse also believes has stolen all his money.
Dromio of Ephesus also keeps referring to "my mistress," but Dromio of Syracuse has no wife, so this confuses Antipholus of Syracuse.

ANTIPHOLUS OF SYRACUSE. Thy mistress' marks? what mistress, slave, hast thou?
DROMIO OF EPHESUS. Your worship's wife, my mistress at the Phoenix; She that doth fast till you come home to dinner, And prays that you will hie you home to dinner.

This leads Antipholus of Syracuse to remark on the perils of life in the city and his reasons for wanting to leave Ephesus as soon as possible.

ANTIPHOLUS OF SYRACUSE. Upon my life, by some device or other The villain is o'er-raught of all my money. They say this town is full of cozenage, As, nimble jugglers that deceive the eye, Dark-working sorcerers that change the mind, Soul-killing witches that deform the body, Disguised cheaters, prating mountebanks, And many such-like liberties of sin: If it prove so, I will be gone the sooner. I'll to the Centaur, to go seek this slave: I greatly fear my money is not safe.

A little later in the play, after more merry mix-ups—including Antipholus of Syracuse falling in love with Luciana—Antipholus of Syracuse once again considers leaving Ephesus.

ANTIPHOLUS OF SYRACUSE: Go hie thee presently, post to the road: An if the wind blow any way from shore, I will not harbour in this town to-night: If any bark put forth, come to the mart, Where I will walk till thou return to me. If every one knows us and we know none, 'Tis time, I think, to trudge, pack and be gone. . . .
There's none but witches do inhabit here; And therefore 'tis high time that I were hence.

Nevertheless, Antipholus of Syracuse does have some nice things to say about Ephesus.

ANTIPHOLUS OF SYRACUSE: There's not a man I meet but doth salute me As if I were their well-acquainted friend; And every one doth call me by my name. Some tender money to me; some invite me; Some other give me thanks for kindnesses; Some offer me commodities to buy: Even now a tailor call'd me in his shop And show'd me silks that he had bought for me, And therewithal took measure of my body.

In time, all of the merry mix-ups are sorted out, the brothers Antipholus are reunited with their father, Aegeon (who is also reunited with his wife, Aemilia), and all ends happily.

DROMIO OF EPHESUS. We came into the world like brother and brother; And now let's go hand in hand, not one before another.