In the book We Were Liars, what was Grandad's only failure?

The novel We Were Liars by E. Lockhart follows the families of the daughters of Harris Sinclair, a wealthy businessman who is a very controlling and is an overbearing father and grandfather. The novel discusses the inheritance he leaves for his family members, as well as how he takes care of them despite his controlling and domineering behavior towards them.
At one point, Granddad professes that his one failure is related to his lineage. His one great failure was never having a son and instead having three daughters. He wishes to have an heir and someone to carry on the family name, mainly due to his pride. He has no issue giving his inheritance through trust funds to his children and grandchildren, but he yearns for someone who would carry on his legacy and take over his work, as well as his wealth.