The very best villains are women and men that we can relate to in some way, even if they are rotten to the core. Better Bad Guys have a grain of sympathy to them, just enough to make you see things from their point of view. Something made them that way--so--what was it? Answer that question and you've got yourself a villain worthy of any hero or heroine you create.
Opponents are more than archetypes and tropes. These are the people that your Good Guys could have been, if things had gone wrong for them. Yes, it's true: some people are just born rebellious and stubborn to a fault. A few of us really are bad, from the start. We're not ashamed of it, either.
And then, there are the rest of them. All those once-decent folks who tried to play by the rules, be the good guy and all that goes with honesty and hard work. Something went wrong. Most of the time, that tragedy--whatever it was--did not start with them. They really didn't do it, but they were made to pay for it...whatever "it" was. You've heard the saying that evil begets evil, so why not concede that injustice breeds...well, you know. Law-and-order can't be the only philosophy that respects an eye for an eye, so why not get your pound of flesh because you deserve it?
The most sincere villains were driven to it, against their will. They may have even fought hard to preserve their ethics and dignity. The scariest Bad Guy I can think is the person you coulda been. That unlucky soul who made the same choices you did, with one difference. They didn't succeed. They lost the battles you won, never got the lucky breaks that came your way. You've passed them in a hallway or on the street, never knowing their pain, misery, or suffering. That's why they are so...
You get the idea. Villains are often so hard to defeat because they have to out-think the Good Guys. Anyone who got framed for a crime they didn't commit, was robbed when they least expected it, or was betrayed by those whom they trusted; they never saw it coming. Having learned their lesson, so to speak: they now plan ahead in ways that most heroes do not tend to consider.
After all that hype, you're asking me: why do they lose? One reason. In spite of their insights, most villains never do quite "get it," they don't change or adapt enough to avoid the pitfalls of arrogance or anger. Somehow, vindication is just never enough for those who end up dead. When all else fails, greed clouds their judgment. When all is said and done, they really did sew the seeds of their own destruction.