The nation's efforts to straddle the fault line between moderate and militant Islam offer a cautionary tale for the post-9/11 world.