What reason does Holden offer for being sorry about his expulsion from Pencey?

Holden never expresses specific regret about leaving Pencey. Holden tends to despise people and places that don't suit his disposition but eventually grows to miss them. For example, see the last paragraph of the book in which Holden admits that he misses two of his former classmates that he was constantly irritated by.
When his sister, Phoebe, asks why he leaves, he cites that the whole place was full of "phonies," though at the same time he makes concessions for the few folks he admires like his history teacher, Mr. Spencer. Before leaving Pencey, Mr. Spencer asks him if he has any concerns about leaving to which Holden replies:

Oh, I have a few qualms, all right. Sure . . . but not too many. Not yet, anyway. I guess it hasn't really hit me yet. It takes things a while to hit me. All I'm doing right now is thinking about going home Wednesday. I'm a moron.

However, a reader can detect regret in Holden's tone once he realizes he's leaving. He obviously feels bad that it will sadden his parents, and he's not especially thrilled to let down Mr. Spencer, though he speaks indifferently to him before he leaves.