Why are the house and neighborhood empty?

To understand why the house and neighborhood are empty, we need to understand the context of the story. Bowen has set the story in London during the Second World War. From history, we know that London was heavily bombed by the Nazis during 1940 and 1941. The aerial bombs, the Nazi's weapon of choice, were so heavy and frequent that the term "Blitz"--based on the German word for lightning - was coined to describe the campaign. As a result, many Londoners fled the city and moved to the countryside, a place where they could not be reached by Nazi bombs.
This idea of escaping the Blitz is supported through the following quote:

In this house the years piled up, her children were born, and they all lived till they were driven out by the bombs of the next war.

From the story, we learn that Mrs. Drover and her family were right to leave the city. One of the Nazi bombs has left a number of cracks in the structure of the house. This suggests that the Blitz is still in full swing, making it unsafe for the family to return to London.