Why was Snape the Half-Blood Prince? What was the theory behind him?

Snape used "The Half Blood Prince" as a pseudonym. He is the Half Blood Prince first and foremost because he chose it as a name. The meaning behind the name is twofold. First, he was the child of a muggle, Tobias Snape, and a witch, Eileen Prince. Thus his bloodline half magical and half muggle. Second, his the name refers to his mother's maiden name, Prince.
Throughout the book, Harry's theory is that Snape is working for Voldemort and is a threat to Hogwarts. This theory builds on Harry's dislike and distrust of Snape in previous books. The Half Blood Prince provides conflicting information on Snape's loyalties. Dumbledore assures Harry that Snape is on their side, but Snape kills Dumbledore. Even then, Snape protects Harry and Draco, suggesting he has complicated or conflicting motivations.
In Harry Potter and the Deathly Hallows, we gain more context for Snape's actions, learning that Dumbledore was mortally ill and arenged his murder with Snape, but as The Half Blood Prince closes, Snape appears as a dark and confusing character

Snape calls himself the Half-Blood Prince because he is the child of a witch and a muggle. His mother, the witch, is Eileen Prince, and his father, the muggle, is Tobias Snape. Hermione discovers this.
Harry Potter's theory about Snape in this volume is that Snape and Draco are working in league with Voldemort, which makes Harry alarmed that Snape is his Dark Arts teacher. Yet the information he gleans about Snape is confusing, especially as Dumbledore tells Harry he believes Snape is on their side.
Snape kills Dumbledore but also protects Harry from the Death Eaters. Harry finds himself surprised as he learns of Snape's history. He notes the similarities between Voldemort's ancestry and Snape's. As this volume ends, we realize it will take another volume to sort out Snape's loyalties and agenda. In the end, we will find out that, because of Harry's mother, Snape was on Harry's side. However, Snape is, throughout the series, a dark, confusing, and complex character.