How did Wesley Moore’s old neighborhood manage to follow him in The Other Wes Moore?

It follows him in the sense that it holds him back, making him less confident than he ought to be about furthering his education. Although Wes may no longer physically be in Baltimore, his mind still remains there to a considerable extent. As such, he finds it difficult to get rid of a debilitating mindset that tells him that higher education isn't really for people like him.
We can see this mentality at work when Wes contemplates applying for Johns Hopkins. As well as being worried over his relatively low SAT scores, Wes is discouraged from applying because he doesn't know anyone from the neighborhood in which he grew up who actually went there. He's also concerned that college will be full of people who don't look or sound anything like him; he's sure that he'll be out of place there.
This is the neighborhood speaking. Despite no longer living there, Wes has internalized its prevailing attitudes concerning education to such an extent that simply entertaining the prospect of applying to Johns Hopkins is incredibly scary.