What is the significance of Hamlet being a Renaissance man trapped in a traditional world in terms of his failure to complete his task of revenge?

This is an interesting question. The traditional culture Hamlet faces is a revenge culture. Once he has proven the ghost really is his father and has told him the truth about being murdered, this traditional culture dictates that Hamlet will avenge his father's death by killing his killer.
This revenge culture saturates the play. Fortinbras is marching on Denmark in revenge for Hamlet's father, the senior Hamlet, having killed his father, also named Fortinbras. Likewise, Laertes wants to kill Hamlet for killing his father, Polonius.
Hamlet, however, is a Renaissance man in the sense of being well-rounded and well-educated, but also in the sense of living in the Christian culture of the Renaissance, in which forgiveness is supposed to predominate over vengeance.
Hamlet's Denmark is clearly conflicted: it is Christian and yet contains such remnants of a pre-Christian culture as honor killing. While Fortinbras and Laertes don't question the ethics of such vengeance, Hamlet, often called the first modern hero, is deeply disturbed about this. He knows a big part of his culture demands he act to avenge his father's death; he also knows he is deeply reluctant to continue a cycle of killing and counter-killing that could go on endlessly. He questions, for example, whether it makes sense for Fortinbras to be willing to sacrifice so many soldiers' lives in order to gain vengeance by taking a few feet of land from Denmark. Notably, Hamlet hesitates and hesitates about killing Claudius.
Hamlet would like a way out of the vise his culture places him in, but he ultimately feels he has no choice but to kill his stepfather.