I'm a man in my 60s. Looking back on my romantic life, I was always the guy women spent time with when their husband or boyfriend wasn't paying attention to them or while they waited for the right guy (status, power, money) to show up. I'm good-looking, but I realize from reading you that I never had enough "mate value," never mastering the essence of Cialdini's "scarcity principle." I'm a retired teacher. For 20-some years, I taught kids who had severe behaviorial problems. While parents, grandparents, and school personnel appreciated what I did, it didn't hold much long-term interest for women. My wife left me for someone with much higher "mate value." I keep thinking that all of this could have been avoided if I had only chosen a profession with high-end "mate value."
You were never going to be the guy for those women who pictured themselves spending lazy summer afternoons in Martha's Vineyard (as opposed to Martha's Laundromat).
However, your having a middlin'-bucks job instead of a megabucks one probably wasn't the root of your mate retention issues. It turns out that there's more to mate value than money and a "high-end" job. In fact, evolutionary psychologist David Buss did a massive cross-cultural survey looking at what men and women want in a partner, and kindness topped the list for each. (Yes, kindness—which was pretty much your job description.) Intelligence was another list-topper.
What wasn't on the lists at all? A partner who's a pushover—always available, never any pesky boundaries. Accordingly, you mention psychologist Robert Cialdini, whose "scarcity principle" I've referenced. Basically, we value—and want—what is out of reach and seems hard to get, not what seems hard to get rid of.
That "hard" truth might seem like reason for you to keep looking back with despair. However, within it is actually a message of hope—an action plan. The reality is, you're in a better position than ever to land and keep a woman.
As I often explain, there are sex differences in what men and women prioritize in a partner, with men valuing looks far more than women do. (Youth and beauty are evolution's version of a billboard advertising health and fertility.) This means that women's mate value is higher when they're, say, in their early 20s. And that's why 22-year-old guys find 22-year-old women seriously hot—as do the grandpas of those 22-year-old dudes.
Meanwhile, a 22-year-old guy is essentially gum under the tire of a 32-year-old guy's Mercedes. Because women prioritize status and resources in a male partner, men's mate value tends to increase as they get older and more accomplished. Cruelly, women's mate value declines with age. On a more positive note, what also tends to go is the notion some younger women have that massive character flaws can be outweighed by a massive mansion. Women with a thing for bad boys may likewise come to see the excitement in a man who pays the bills the boring way—through online banking instead of online identity theft.
In short, there are plenty of women who'd want a guy like you—providing you mend your ways. Going back to that "scarcity principle," what needs to become scarce is your willingness to be a convenient option instead of a priority. Though this has been your default state—for decades—it doesn't have to remain that way. As I explain in my new "science-help" book, "Unf*ckology: A Field Guide to Living with Guts and Confidence," "your feelings are not the boss of you. It's not what you feel; it's what you do."
In fact, by repeatedly acting assertively, you'll actually rewire your brain. This isn't to say the old rollover kitty behaviors go away. Unfortunately, there's no giant neural eraser that comes around once a week like the trash guys the city sends to your neighborhood. What happens is that you transform your default behavior—how you behave when you react automatically—to acting like a man instead of like the male friend who braids women's hair while they're waiting for the guy they are having sex with.
As for the practical steps to becoming the new bold you: Figure out what seems fair and right, and then say "no" to everything outside that box. (Generosity is good. Generosity that knows no bounds is a ladyboner killer.) Assert yourself even when you're scared to do it. Sure, you'll feel uncomfortable, especially the first few times. However, you should slowly begin to do better with the ladies—and maybe even find love, despite it being clear that the only fur you'd ever get a woman would come with the rest of the hamster or the cat.