In darkness and loneliness they are strongest

And in any case that is not their way. In dark and loneliness they are strongest; they will not openly attack a house where there are lights and many people–not until they are desperate, not while all the long leagues of Eriador still lie before us. But their power is in terror, and already some in Bree are in their clutch. They will drive these wretches to some evil work: Ferny, and some of the strangers, and, maybe, the gatekeeper too.

(Strider explaining to Frodo why the black riders will not assault the village of Bree and the Prancing Pony where they stay)