Under what circumstances might fair trade actually cause harm?

Though a perfectly admirable idea in principle, fair trade can often have a damaging impact in practice. The whole purpose behind fair trade is to pay producers in the developing world a fair price—that is to say, a price above the market rate—for their products. The idea of fair trade is that, instead of exploiting Third World producers, as is often the case, Western businesses will treat them as equal partners.
The downside here is that artificially higher prices will inevitably be passed on to the consumer. The consumer will then be less likely to buy the relevant product than if the free market decided its price in the usual way. The end result will be that producers in the developing world will sell much less of their product, possibly resulting in their being considerably worse off.
In turn, the higher price of fair trade products in the stores will encourage domestic competitors to make inroads into the relevant market by offering similar products of markedly inferior quality. In an economic downturn, when money is tight, consumers may feel that it's worth it just to pay less, even if they know they're getting an inferior product. In such a marketplace, fair trade goods—premium goods at premium prices—are unlikely to get a look-in, with damaging consequences for producers in the developing world.