Why does Crooks say he was only "foolin’" about wanting to be part of the dream?

In chapter 4 of Steinbeck's Of Mice and Men, Lennie wanders into the barn and can't help but see the light from Crooks's room. George and most of the other men have gone into town for the evening and Lennie is feeling a bit lonely. At first, Crooks is cranky and tells Lennie to get out, but he gives in and allows Lennie to come in. He complains to Lennie that he is not allowed in the bunkhouse because he is black which Lennie doesn't really understand. Soon enough, Candy also appears at the door of Crooks's room. He's looking for Lennie because he has figured a way they can make money on the rabbits which Lennie desperately wants to "tend." The conversation turns to the farm which George, Lennie and Candy hope to buy.
Although skeptical at first, Crooks warms to the idea of going off and living at such a place. He hesitatingly asks Candy,

" . . . If you . . . guys would want a hand to work for nothing—just his keep, why I'd come an' lend a hand. I ain't so crippled I can't work like a son-of-a-bitch if I want to."

At this point, Crooks seems totally serious about joining the men, but, just as he is finished speaking, Curley's wife breaks into the conversation. The three men instantly recoil from her presence but this doesn't dissuade her from entering the room. She is, of course, looking for Curley, who has also gone into town. She comments about how all of the "weak ones" have been left behind which draws protests from old Candy. Eventually, Crooks grows weary of her complaints and demands that she leave. Her response demonstrates the destructive racism of the time:

"You know what I can do to you . . . I could get you strung up on a tree so easy it ain't even funny."

After that exchange, Crooks "retired into the terrible protective dignity of the negro." It reminds him, once again, that he is a black man among white people. He is not even remotely considered equal to people such as Curley's wife. At the end of the chapter, Crooks tells Candy he was "jus' foolin'" when he offered to go to the farm to "lend a hand." He reverts back to his alienation and isolation even though Candy and Lennie are probably the least racist men on the ranch.