Perhaps the most serious difference lies in the areas of freedom and justice. Libertarians occasionally talk as though justice is the most important thing which human society ought to strive towards yet almost unfailingly insist that the measure of an act is the amount of freedom it allows. The problem is that freedom ultimately is a means rather than an end, and while it is easy to think that the society with the most freedom may (and the word may cannot be stressed enough) produce the most justice, that surely cannot be a given. A hedonistic society such as what the Western world sometimes appears hell bent for leather determined to make of itself surely welcomes a great (one is tempted to say an absurd) amount of freedom among the people. Do libertarians sufficiently address what that must mean?
Seeing as they are only seriously opposed to areas of direct and physical violence against persons, it is easy to say no. This isn't to say they oppose any and all nonviolent yet repugnant actions; we presume they are against libel and slander, for example. But do they take it far enough? Are there areas where, even though there may appear a lack of real violence against the person or society, there is a nonphysical violence which may yet hurt individuals and nations?
The conservative says yes. When society allows repugnant actions to be legal then it had begun teaching individuals and nations that the immoral is in fact moral. It begins to tear the fabric of civil society into little more than strands, each strand being each person, each person being free to do what they want, outside of overt violence, of course, without censure. When that happens, how long may it be before the strands cannot support even themselves? Likely as not, the moment they become too individual, too little concerned with the larger society. What will happen then? The strands that get together by whatever means will dominate the rest. That probably won't mean much for freedom, let alone justice.
Without a sense of true justice, physically non violent yet unjust actions will be seen as mere aspects of personal freedom. If it doesn't harm anyone directly and immediately it must, in the libertarian view, be allowed.
Such is a shallow definition of justice. But when a creed is based on mere freedom rather than on actual right and wrong, what can we expect in the long run but decay? If the libertarians wish to be taken seriously then they must accept that freedom isn't the end all be all of human actions. Until that happens, they will always be on the margin of political society. Or, worse, they will lead all of society to its destruction.