Why should Banquo have known to protect himself and his family from Macbeth?

Banquo shows great caution long before his untimely demise. He suspects Macbeth of having murdered Duncan, and knowing his friend's ambition, not to mention the details of the Weird Sisters' prophecy, he has every reason to be wary. It's notable that in act 2, scene 1, even before Macbeth's actually killed Duncan, Banquo's already feeling quite antsy, keeping his sword close to him as if expecting to use it at any moment.
And yet Banquo doesn't flee from Macbeth, as so many other Scottish nobles do. Instead, he stays quiet and keeps up the facade of friendship with the tyrant king. Perhaps he feels that it would be better for himself and his family if he were to keep his friends close, but his enemies closer. If that is indeed Banquo's way of thinking, then it proves to be a catastrophic miscalculation that leads to his death. He was under no illusions about the true nature of Macbeth or his despotic rule, and so should've known that the best thing to do, both for himself and his family, was to leave Scotland as soon as possible and join up with those dissident nobles determined to overthrow Macbeth.