Could bad weather defer the pilgrims' journey?

Yes it could, most definitely. Roads in Medieval Europe were absolutely appalling, significantly worse that those built by the Romans. Journeys that would nowadays take a matter of hours could take weeks. As the roads were in such dreadful condition to begin with, bad weather could make them even worse. All it would take was a sudden shower of rain for horses, oxcarts, and carriages to end up stuck in the mud, thus delaying their journeys.
Although the pilgrimage to Canterbury in the Tales takes place in April, during the first burst of spring, the ever-changeable English weather means that a torrential downpour is never far away. All the pilgrims are riding on horseback and so they don't have much protection from the elements. Inevitably, bad weather would mean delays; not just because of the shocking road conditions, but because upper-class pilgrims like the Prioress would feel it beneath their dignity to plow on with the journey, all drenched and bedraggled. Instead, they'd probably rock up at the nearest inn or hostelry—such as the one from where the pilgrims set out, The Tabard—and wait until the weather improved.
This would add considerable time to the journey, but the likes of the Prioress look upon religious pilgrimages primarily as an opportunity to see and be seen, to show off their wealth and social status. So far better to arrive late and in style than get there as soon as possible, but looking for all the world like a drowned rat.