Amnesty Int’l Calls for U.S. Arms Embargo Against Israel
Thursday, February 26, 2009 5:24 PM
By: Nicole Jansezian Article Font Size
Amnesty International has called on the Obama administration to immediately suspend weapons sales to Israel after the human rights organization said it found that most of the weapons Israel used in the Gaza Strip were manufactured in the United States.
“To a large extent, Israel's military offensive in Gaza was carried out with weapons, munitions and military equipment supplied by the USA and paid for with U.S. taxpayers’ money,” said Malcolm Smart, Amnesty International's director for the Middle East. “The Obama administration should immediately suspend U.S. military aid to Israel.”
Israeli officials say the report robs Israel of its right to self defense.
Amnesty’s report, “Fueling Conflict: Foreign Arms Supplies to Israel/Gaza,” also calls on the United Nations Security Council to “impose immediately a comprehensive ... arms embargo on Israel, Hamas and other Palestinian armed groups.”
The group accused both Israel and Hamas of war crimes in its 38-page report, but focused heavily on Israeli actions and American weapons used in the Gaza Strip. Amnesty researchers found that weapons fragments in school playgrounds, hospitals and in homes were mostly made in America, the report said.
In a statement, Israel's Foreign Ministry slammed the report calling it a biased version of events during and leading up to Operation Cast Lead, which left 1,300 Palestinians dead. The report neglected to mention Hamas’ use of civilians as human shields, the statement said.
“Hamas openly and in an organized fashion uses women and children to protect military targets, and booby-trap homes and public buildings,” the foreign ministry said. “The IDF (Israel Defense Forces) never intentionally targeted civilians.”
The report also called for halting arms sales to Hamas, but admitted that “Hamas and other Palestinian armed groups have smuggled small arms, light weapons, rockets and rocket components into Gaza, using tunnels from Egypt into Gaza; this weaponry has been acquired from clandestine sources.”
Hamas’ weapons, funded by Tehran and other Arab regimes, are illegally supplied through tunnels connecting the Strip to Egypt.
NGO Monitor, a watchdog group of non-governmental organization, said, “Amnesty’s attempt to equate the transfer weapons to Israel for legitimate defense, with clandestinely smuggled arms to a terrorist organization, is defamatory, immoral and absurd.”
NGO Monitor Director Gerald Steinberg said the report was “clearly part of a campaign to deprive Israel of the means to defend itself.”
But the report could gain traction in the U.S. After a visit to the region last week, Rep. Brian Baird, D-Wash., said he planned to recommend the U.S. reassess its military support for the Jewish state. Baird, who visited Gaza with Rep. Keith Ellison, D-Minn., said he was troubled by the American origin of Israeli weaponry.
“We need to use every pressure available to make these needed changes happen,” he said. “If our colleagues had seen what we have seen, I think their understanding of the situation would be significantly impacted. They would care about what happened to the Palestinians.”
Danny Reisner, a legal advisor to the Israel Defense Forces, countered accusations that Israel committed war crimes and noted that Israel suspended the war every day to allow humanitarian aid trucks into Gaza.
“People are complaining that war crimes are being committed - and I agree,” he said. “War crimes are being committed by Hamas.
“They are waging war. Therefore we can respond within the realm of war. The other side does two things fundamental to the problem: they don’t abide by the rules and they don’t identify themselves as combatants.”
Hamas is considered a terrorist organization by the United States.
The Amnesty report questioned a 10-year agreement ending in 2017 in which the U.S. would provide $30 billion in military aid to Israel. Israeli defense officials said they were concerned, especially ahead of U.S. Middle East envoy George Mitchell’s arrival in Israel today, that President Barack Obama will cut military aid to Israel.
“Mitchell is a known opponent of the outposts and the settlements,” a senior defense official told The Jerusalem Post. “The Americans may try to use the military aid as a way of pressuring the new government into dismantling outposts and freezing construction in settlements.”
Meanwhile, the British government is being sued by pro-Palestinian groups for selling weapons to Israel. The suit, filed this week, accuses the government of flagrantly breaching international law by continuing British export of arms to Israel. The Foreign Office said that the British government “monitors the situation in Israel with care in considering applications for arms export licenses.”
This case calls for suspension of arms to Israel and for the European Union to suspend a preferential trading agreement with the Jewish state. It also calls for the arrest, on the basis of war crimes, of Israeli agents visiting England. Lawyers for Al Haq, a Palestinian charity representing 30 families in the case, said more legal action is planned against Israel.
© 2009 Newsmax. All rights reserved.
Thursday, February 26, 2009
Jörg Schilling star
http://cdrecord.berlios.de/old/private/index.html
seems a smart bastard all around
seems a smart bastard all around
Wednesday, February 25, 2009
democrat communists try to silence radio dissent with 'fairness'
Radio Insiders, Congressmen Aim to Stop Fairness Doctrine
Wednesday, February 25, 2009 12:17 PM
By: Jim Meyers Article Font Size
Moves are afoot to head off any Democratic efforts to reinstate the so-called Fairness Doctrine and stifle conservative talk radio.
A group of radio insiders has formed the Free Radio Coalition to fight the reinstatement, Radio America President James Roberts said on Tuesday.
Radio America talk show host and former San Diego Mayor Roger Hedgecock will chair the coalition.
“The reinstatement of the misnamed Fairness Doctrine would constitute a massive assault on our cherished First Amendment rights and should be of concern to all Americans, regardless of their political or religious persuasion,” Hedgecock said.
Group members want to hold a conference of talk show hosts and religious broadcasters in Washington to plan strategy, the Washington Times reported.
They also plan to prepare expert testimony in case the Federal Communications Commission or congressional committees hold hearings on the Doctrine.
A Barack Obama spokesman recently said the president opposes reinstatement of the Fairness Doctrine. But a number of Democratic senators, including Debbie Stabenow of Michigan, Tom Harkin of Iowa, Charles Schumer of New York and Dick Durbin of Illinois have expressed support for such a move.
Originally instituted in 1949 by the FCC, the Fairness Doctrine required broadcasters using the public airwaves to give equal time to opposing political views. The FCC repealed the measure in 1987.
Since talk radio is overwhelmingly dominated by conservative hosts, and liberal talk radio draws few listeners, the “equal time” provision would likely force many radio stations to pull popular conservative hosts from the air rather than air low-rated liberal hosts.
Rep. Mike Pence, an Indiana Republican who has guest-hosted on Laura Ingraham’s conservative radio talk show, is seeking to bar the FCC from using taxpayer funds to enforce the Doctrine.
He has drafted an amendment to the $410 billion omnibus spending bill that would block the FCC from resuming the policy, The Hill newspaper reported.
“Bringing back the Fairness Doctrine today would amount to government control over political views expressed on the public airwaves,” said Pence, chairman of the House Republican Conference.
“The American people cherish freedom, especially freedom of speech and of the press.”
In 2007, the House voted overwhelmingly to prevent the FCC from enforcing the Doctrine with taxpayer funds.
But Pence is concerned that on March 6, when the stopgap measure funding the government expires, Democrats could seek to put the policy back in place.
“While a permanent ban is ideal, in the short term the Pence amendment would reassure freedom-loving Americans that the national asset of talk radio would remain free from censorship for the next year,” Pence said in a statement.
Last week Newsmax reported that Sen. Jim DeMint announced that he would force a vote on a bill preventing the FCC from reinstating the Doctrine.
The South Carolina Republican said the bill, the Broadcaster Freedom Act, would be offered as an amendment to the D.C. Voting Rights bill.
© 2009 Newsmax. All rights reserved.
Wednesday, February 25, 2009 12:17 PM
By: Jim Meyers Article Font Size
Moves are afoot to head off any Democratic efforts to reinstate the so-called Fairness Doctrine and stifle conservative talk radio.
A group of radio insiders has formed the Free Radio Coalition to fight the reinstatement, Radio America President James Roberts said on Tuesday.
Radio America talk show host and former San Diego Mayor Roger Hedgecock will chair the coalition.
“The reinstatement of the misnamed Fairness Doctrine would constitute a massive assault on our cherished First Amendment rights and should be of concern to all Americans, regardless of their political or religious persuasion,” Hedgecock said.
Group members want to hold a conference of talk show hosts and religious broadcasters in Washington to plan strategy, the Washington Times reported.
They also plan to prepare expert testimony in case the Federal Communications Commission or congressional committees hold hearings on the Doctrine.
A Barack Obama spokesman recently said the president opposes reinstatement of the Fairness Doctrine. But a number of Democratic senators, including Debbie Stabenow of Michigan, Tom Harkin of Iowa, Charles Schumer of New York and Dick Durbin of Illinois have expressed support for such a move.
Originally instituted in 1949 by the FCC, the Fairness Doctrine required broadcasters using the public airwaves to give equal time to opposing political views. The FCC repealed the measure in 1987.
Since talk radio is overwhelmingly dominated by conservative hosts, and liberal talk radio draws few listeners, the “equal time” provision would likely force many radio stations to pull popular conservative hosts from the air rather than air low-rated liberal hosts.
Rep. Mike Pence, an Indiana Republican who has guest-hosted on Laura Ingraham’s conservative radio talk show, is seeking to bar the FCC from using taxpayer funds to enforce the Doctrine.
He has drafted an amendment to the $410 billion omnibus spending bill that would block the FCC from resuming the policy, The Hill newspaper reported.
“Bringing back the Fairness Doctrine today would amount to government control over political views expressed on the public airwaves,” said Pence, chairman of the House Republican Conference.
“The American people cherish freedom, especially freedom of speech and of the press.”
In 2007, the House voted overwhelmingly to prevent the FCC from enforcing the Doctrine with taxpayer funds.
But Pence is concerned that on March 6, when the stopgap measure funding the government expires, Democrats could seek to put the policy back in place.
“While a permanent ban is ideal, in the short term the Pence amendment would reassure freedom-loving Americans that the national asset of talk radio would remain free from censorship for the next year,” Pence said in a statement.
Last week Newsmax reported that Sen. Jim DeMint announced that he would force a vote on a bill preventing the FCC from reinstating the Doctrine.
The South Carolina Republican said the bill, the Broadcaster Freedom Act, would be offered as an amendment to the D.C. Voting Rights bill.
© 2009 Newsmax. All rights reserved.
ann coulter is awesome
WHY WE DON'T CELEBRATE 'HISTORIANS DAY'
February 18, 2009
Being gracious winners, this week, liberals howled with delight at George Bush for coming in seventh-to-last in a historians' ranking of the presidents from best to worst.
This was pretty shocking. Most liberals can't even name seven U.S. presidents.
Being ranked one of the worst presidents by "historians" is like being called "anti-American" by the Nation magazine. And by "historian," I mean a former member of the Weather Underground, who is subsidized by the taxpayer to engage in left-wing political activism in a cushy university job.
So congratulations, George Bush! Whenever history professors rank you as one of the "worst" presidents, it's a good bet you were one of America's greatest.
Six months after America's all-time greatest president left office in 1989, historians ranked him as only a middling president. (I would rank George Washington as America's greatest president, but he only had to defeat what was then the world's greatest military power with a ragtag group of irregulars and some squirrel guns, whereas Ronald Reagan had to defeat liberals.)
At the time, historian Arthur M. Schlesinger Jr. dismissed Reagan as "a nice, old uncle, who comes in and all the kids are glad to see him. He sits around telling stories, and they're all fond of him, but they don't take him too seriously" -- and then Schlesinger fell asleep in his soup.
Even liberal historian Richard Reeves blanched at Reagan's low ranking in 1989, saying, "I was no fan of Reagan, but I think I know a leader when I see one."
Reagan changed the country, Reeves said, and some would say "he changed the world, making communism irrelevant and the globe safe for the new imperialism of free-market capitalism." In Reeves' most inspiring line, he says Reagan "was a man of conservative principle and he damned near destroyed American liberalism."
By 1996 things hadn't gotten much better for Reagan in the historians' view. A poll of historians placed Reagan 26th of 42 presidents -- below George H.W. Bush, his boob of a vice president who raised taxes and ended Republican hegemony under Reagan. Four of the 32 historians called Reagan a "failure."
I guess it depends on your definition of "failure." To me a failure is someone who aspired to be a legitimate scholar but ends up as an obscure lecturer at Colorado College.
Speaking of which, Colorado College political scientist Thomas Cronin explained Reagan's low ranking, saying Reagan "was insensitive to women's rights, civil rights, oblivious to what was going on in his own Administration -- the procurement scandal, HUD, Iran-Contra."
Soon after he took office, President Reagan famously hung a portrait of President Calvin Coolidge in the Cabinet Room -- another (Republican) president considered a failure by historians.
Coolidge cut taxes, didn't get the country in any wars, cut the national debt almost in half, and presided over a calm, scandal-free administration, a period of peace, 17.5 percent growth in the gross national product, low inflation (.4 percent) and low unemployment (3.6 percent).
Unlike some recent presidents with Islamic middle names, he didn't run around comparing himself to Lincoln constantly.
Arthur Schlesinger Jr. ridiculed President Calvin Coolidge as a hayseed who slept too much and took decisive action only once in his life. Schlesinger never tired of pointing out that Coolidge slept 11 hours a day, as if hours of sleep is the true measure of presidential greatness.
Perhaps Schlesinger's venom toward Coolidge was meant as penance for his once mistakenly admitting that Eisenhower was a good president -- another hated (Republican) president among historians.
Under President Dwight Eisenhower the gross national product grew by over 25 percent and inflation averaged 1.4 percent. George Meany, then AFL-CIO president, said that the American worker had "never had it so good." Like Coolidge and Reagan, Eisenhower was enormously popular with the American people.
In a poll of "leading scholars" taken soon after Eisenhower left office, he was named one of the 10 worst presidents. The distinguished scholars -- none of whose names anyone remembers today -- called him dumb, dismissing the five-star general who smashed the Nazi war machine as "Old Bubble Head." As Patton said, these "bilious bastards ... don't know anything more about real battle than they do about fornicating."
It's as if geologists took a poll and announced their opinion that gold was heavier than lead.
Reagan and Eisenhower have recently started to move up in the presidential rankings -- for the same reason George Washington is always ranked one of the best. Historians ought to detest Washington, but his exclusion from the top ranks of these pompous historian polls would expose the absurdity of their rankings.
Putting preposterously overrated presidents like John F. Kennedy or FDR in the same category as Reagan or Washington is like a teenage girl ranking the Jonas Brothers with the Rolling Stones and the Beatles as the three greatest bands of all time.
Liberals may call him a "war criminal," but historians have inadvertently paid Bush a great tribute this week by ranking him as a "below average" president. I can only dream that, someday, no-name, left-wing historians will rank me as one of the all-time worst columnists.
COPYRIGHT 2009 ANN COULTER
DISTRIBUTED BY UNIVERSAL PRESS SYNDICATE
1130 Walnut, Kansas City, MO 64106
February 18, 2009
Being gracious winners, this week, liberals howled with delight at George Bush for coming in seventh-to-last in a historians' ranking of the presidents from best to worst.
This was pretty shocking. Most liberals can't even name seven U.S. presidents.
Being ranked one of the worst presidents by "historians" is like being called "anti-American" by the Nation magazine. And by "historian," I mean a former member of the Weather Underground, who is subsidized by the taxpayer to engage in left-wing political activism in a cushy university job.
So congratulations, George Bush! Whenever history professors rank you as one of the "worst" presidents, it's a good bet you were one of America's greatest.
Six months after America's all-time greatest president left office in 1989, historians ranked him as only a middling president. (I would rank George Washington as America's greatest president, but he only had to defeat what was then the world's greatest military power with a ragtag group of irregulars and some squirrel guns, whereas Ronald Reagan had to defeat liberals.)
At the time, historian Arthur M. Schlesinger Jr. dismissed Reagan as "a nice, old uncle, who comes in and all the kids are glad to see him. He sits around telling stories, and they're all fond of him, but they don't take him too seriously" -- and then Schlesinger fell asleep in his soup.
Even liberal historian Richard Reeves blanched at Reagan's low ranking in 1989, saying, "I was no fan of Reagan, but I think I know a leader when I see one."
Reagan changed the country, Reeves said, and some would say "he changed the world, making communism irrelevant and the globe safe for the new imperialism of free-market capitalism." In Reeves' most inspiring line, he says Reagan "was a man of conservative principle and he damned near destroyed American liberalism."
By 1996 things hadn't gotten much better for Reagan in the historians' view. A poll of historians placed Reagan 26th of 42 presidents -- below George H.W. Bush, his boob of a vice president who raised taxes and ended Republican hegemony under Reagan. Four of the 32 historians called Reagan a "failure."
I guess it depends on your definition of "failure." To me a failure is someone who aspired to be a legitimate scholar but ends up as an obscure lecturer at Colorado College.
Speaking of which, Colorado College political scientist Thomas Cronin explained Reagan's low ranking, saying Reagan "was insensitive to women's rights, civil rights, oblivious to what was going on in his own Administration -- the procurement scandal, HUD, Iran-Contra."
Soon after he took office, President Reagan famously hung a portrait of President Calvin Coolidge in the Cabinet Room -- another (Republican) president considered a failure by historians.
Coolidge cut taxes, didn't get the country in any wars, cut the national debt almost in half, and presided over a calm, scandal-free administration, a period of peace, 17.5 percent growth in the gross national product, low inflation (.4 percent) and low unemployment (3.6 percent).
Unlike some recent presidents with Islamic middle names, he didn't run around comparing himself to Lincoln constantly.
Arthur Schlesinger Jr. ridiculed President Calvin Coolidge as a hayseed who slept too much and took decisive action only once in his life. Schlesinger never tired of pointing out that Coolidge slept 11 hours a day, as if hours of sleep is the true measure of presidential greatness.
Perhaps Schlesinger's venom toward Coolidge was meant as penance for his once mistakenly admitting that Eisenhower was a good president -- another hated (Republican) president among historians.
Under President Dwight Eisenhower the gross national product grew by over 25 percent and inflation averaged 1.4 percent. George Meany, then AFL-CIO president, said that the American worker had "never had it so good." Like Coolidge and Reagan, Eisenhower was enormously popular with the American people.
In a poll of "leading scholars" taken soon after Eisenhower left office, he was named one of the 10 worst presidents. The distinguished scholars -- none of whose names anyone remembers today -- called him dumb, dismissing the five-star general who smashed the Nazi war machine as "Old Bubble Head." As Patton said, these "bilious bastards ... don't know anything more about real battle than they do about fornicating."
It's as if geologists took a poll and announced their opinion that gold was heavier than lead.
Reagan and Eisenhower have recently started to move up in the presidential rankings -- for the same reason George Washington is always ranked one of the best. Historians ought to detest Washington, but his exclusion from the top ranks of these pompous historian polls would expose the absurdity of their rankings.
Putting preposterously overrated presidents like John F. Kennedy or FDR in the same category as Reagan or Washington is like a teenage girl ranking the Jonas Brothers with the Rolling Stones and the Beatles as the three greatest bands of all time.
Liberals may call him a "war criminal," but historians have inadvertently paid Bush a great tribute this week by ranking him as a "below average" president. I can only dream that, someday, no-name, left-wing historians will rank me as one of the all-time worst columnists.
COPYRIGHT 2009 ANN COULTER
DISTRIBUTED BY UNIVERSAL PRESS SYNDICATE
1130 Walnut, Kansas City, MO 64106
self esteem by bartmann
http://www.businessweek.com/magazine/content/07_18/b4032066.htm?chan=search
MARKETING
I Was A Loser, Baby, So Why Don't You Pay Me?
Bill Bartmann milks his losses on the motivational circuit
"The secret of success is elegantly simple, like the law of gravity. This is how the world works: You need to be willing to take risks. You don't because you are afraid of failure; you are worried about what people will think. And that's because of low self-esteem. To succeed, you have to raise your self-esteem."
This is Bill Bartmann's come-on, his preamble, the most concise explanation he can offer for why he became a billionaire and those listening to him haven't even come close. On a Saturday in September, a couple dozen people, mostly entrepreneurs, of their own volition or at the urging of frustrated spouses and business partners, have paid $49 to attend a daylong motivational seminar with Bartmann at the Reno-Sparks Convention Center in Nevada.
None are familiar with his name. Bartmann, 58, made (and lost) his fortune in the debt collection business, and both his rise and fall brought him to national attention, if fleetingly. He was well-known in the financial world and in the business schools that studied his then-pioneering approaches to raising money; in Tulsa, where he was among the biggest employers; and in certain Las Vegas casinos, where he was a regular at the blackjack tables. But few outside those circles would have followed his tumultuous career.
"Why am I an authority? Because I've lived it. I've seen both sides of self-esteem. I went from poverty and welfare to being the 25th-wealthiest person in America. I went from being a high-school dropout to becoming a lawyer. I went from working at a traveling carnival to being named National Entrepreneur of the Year. I went from being a million dollars in debt to a billion dollars to the good. I learned how to change. I went through a significant, some say remarkable, transformation."
Bill Bartmann's career has been one of seized opportunities: oil in the 1980s, consumer debt in the 1990s, and now, in the age of Oprah, inspiration. Motivation, in all its guises, has become a $10 billion industry even as its very premise--that its advice can change lives--remains unproven. Companies spend close to $3 billion a year to hire people, among them former executives, sports coaches, and survivors of all kinds, to inspire and train employees; books ranging from The 7 Habits of Highly Effective People to The Millionaire Next Door are on BusinessWeek's list of longtime best-sellers.
Richard, one of the main characters in the recent indie film hit Little Miss Sunshine, is a struggling motivational speaker. Michael Arndt, who wrote the movie's screenplay, which won this year's Academy Award, says: "Richard thinks he has a big idea that is going to transform his life. That's a very American fantasy." The success of motivational gurus such as Tony Robbins, Stephen R. Covey, and Zig Ziglar, all of whom have become multimillion-dollar brands, has helped promote the notion that selling inspiration can be a career choice. "If this becomes what I think it can, I'm back," Bartmann says.
The tradition of teaching success goes back to Benjamin Franklin's Poor Richard's Almanack and runs through Dale Carnegie's How to Win Friends & Influence People to Robert J. Ringer's Looking Out for #1. The motivation business is sustained by a peculiarly American belief in reinvention combined with our nearly perpetual feelings of dissatisfaction. The big ideas, the supposed secrets of achievement, reflect and influence the cultural zeitgeist. Today those who have overcome adversity--they may be victims of disease, accidents, or their own daring or recklessness--have captured the national imagination. Such as: the guy who chopped off his arm to save his life (Aron Ralston); the woman who treated herself for breast cancer in the South Pole (Jerri Nielsen); the prison guy who became a top chef (Jeff Henderson).
In business, it is adversity of your own making that resonates today; learning from failure is an animating force for everything from innovation to ethics. "Redemption is big," says Jonathan Black, author of Yes You Can! Behind the Hype and Hustle of the Motivation Biz. "But you have to admit you've failed. Otherwise people think you're covering up." One notable player on the failure circuit is Scott Waddle, commander of the nuclear submarine that accidentally sank a Japanese fishing boat in 2001, killing nine. Stripped of his command and retired, Waddle now earns $10,000 a talk, the most popular of which is "Failure Is Not Final."
By anyone's estimation, Bill Bartmann's life has been one of head-swelling highs and depressing lows. Bartmann made his first fortune running a company that manufactured oil pipes; the collapse of crude prices in 1985 left him without any customers and a million dollars in debt. A year later he had started Commercial Financial Services Inc. with the notion that he might be able to squeeze money out of debtors everyone else had given up on.
In 1997, Bartmann and his wife, Kathy, who together owned 80% of CFS, made Forbes' list of the 400 wealthiest Americans, a distinction Bill had long coveted. The magazine estimated the couple to be worth about $1.1 billion. "We had lots of arguments about getting rich," says Kathy. "I don't care if I'm on any lists, if anyone knows me." Bill worked in an office with a statue of Don Quixote, traveled with a security detail, didn't join the country club as was expected, and generally annoyed the Tulsa business community with his swagger.
HITTING BOTTOM
By then CFS had become the world's largest holder of bad consumer debt; unlike its competitors it owned the debt outright and was trying to collect on some $14.5 billion. A few skeptics wondered at the company's rapid growth; by many accounts its revenues had doubled every year since 1994. When Goldman Sachs Group Inc. (GS ) proposed taking the company public the summer of 1998, CFS was reckoned to be worth $3 billion.
Four months later, credit agencies received an anonymous one-page letter accusing CFS of shady dealings. Bartmann would eventually be indicted on 57 federal counts of fraud, conspiracy, and money laundering related to an alleged scheme to make the company's collection rate appear higher than it was. After an 89-day trial in 2003, he was acquitted of all the charges against him (his partner, Jay L. Jones, pleaded guilty to conspiracy and served 3 1/2 years of a five-year prison term). It took Bartmann until 2006 to contend with various civil suits, and during that time he declared bankruptcy. He and Kathy had to give up 19 acres of their 20-acre estate. They lived off credit cards, borrowing from their two daughters when they had to.
Then Bartmann had his next big idea. As a businessman he had used the story of his early life to put lenders at ease, motivate employees, and in general establish his credibility as someone who can prosper in any circumstances. His setbacks began feeling like an affirmation of sorts: The wild arcs of his life offered valuable lessons. "I think I am the message, the answer," he says. "I'm selling the one thing everyone needs: self-confidence."
Successful motivational speakers require a measure of arrogance: They must believe they can compel an audience, essentially, to do their bidding. Bartmann's charisma is more closely related to cunning than magnetism. A 2001 New Yorker profile described him as having "an uncanny gift for making those who work for him feel that they share in his powers." A former senior executive at CFS, Wayne Learned, knows all about that. He actually invested in another doomed Bartmann enterprise after the CFS collapse, and depicts his former boss this way: "Persuasive isn't the right word to describe him. You do it to yourself as much as anything.... He'll call when he wants something and then he sucks you dry. You're like a raisin. When you work for him, you're his raisin."
What so mesmerizes people seems awkwardly exaggerated onstage in Reno. Bartmann's physical presence can be intimidating: He has a compact body, a rough voice, and an almost feral energy. His hands are those of someone who spent his adolescent years brawling. He wears a tight-fitting shirt and dark suit. He uses the language of the bill collector; he retains an outsider's posture.
SELLING, MODESTLY
If anyone understands the value of seeming authentic, it is Bill Bartmann. "Dogs know when you are afraid," he says. "People know when you're b.s.-ing them. You have to respect that."
Hence that afternoon's onstage pitch: "I'm awkward about this. I haven't set you up for the kill," he says. "I don't want to be a snake-oil salesman. I'm in the motivational, self-help industry. It's an industry that ranks right there with car salesmen. I don't know if I'm any better, any different, but I do know a few things.... Have I done a good job of un-selling?"
Bartmann carries around a laminated card in his pocket. On it is typed the following goal: touch 10 million people in five years. "It's a giant, crazy, ludicrous number," he says. "But if I touch enough people, the money should follow."
Bartmann started out using a classic motivational business model: sell tickets cheaply, give them away if necessary, and push packages of DVDs and books (his is titled Billionaire Secrets to Success). In Reno, he sold several thousand dollars' worth. And he added 31 names to those already on his spreadsheet: nine million, nine hundred and eighty thousand to go.
Though there may be as many as 15,000 people trying to make a living as professional speakers, it is quite possible for the ones who catch on eventually to make a couple of million dollars a year. Bartmann, never one to shy from the grandiose, wants to make $100 million. So in the weeks after Reno he began holding free two-hour seminars around the country as a way to advertise a yearlong mentoring program, conducted mostly over the phone and by e-mail at a cost of $397 a month.
It hasn't escaped his attention that personal coaching has become a billion-dollar business itself. By December, he had visited seven cities and said he signed up 200 students. "If I don't make $100 million a year from now, I'll be disappointed," he said then. He had other aims, too: "At 25 cities, I'll get a little plane. At 50 cities, a bigger one. I miss my G-IV. The plane is my measure of success. If I touch enough people to get a plane, I'll be a financial success."
Bartmann's problem, though, was that success hinged on an admission of failure, which initially he was disinclined to make. That part of his story, he believed, couldn't so neatly be explained and wasn't appropriate at most of his seminars. "I don't amplify the CFS loss. It isn't inspirational," he said in September. "I say 'self-made billionaire.' I don't use the present tense. I know they can Google me and find out everything."
Then in March, Bartmann found himself in Hollywood, where the idea of him hosting a "Dr. Failure" talk show somehow, wondrously, came up. By that time Bartmann was ready to tell his whole story. "I talk about CFS first.... There is no more compelling story. Kathy said I should quit acting like I'm ashamed. It makes everyone else's problems seem smaller. It gives them permission to tell me about their problems." He came up with a new pitch: "To do for failure what Betty Ford did for alcoholism and Susan Komen did for breast cancer."
Bartmann has also made a decision that, in part, reflects his wish to be admired as well as recognized: All profits from his mentor program, for which he now charges $797 a month, and his appearances at Get Motivated! Tours (where he is one of dozens alongside George Foreman and Zig Ziglar) and Learning Annex Real Estate and Wealth Expos (where he is one of dozens alongside Donald Trump and Tony Robbins) will go into a foundation called Bill's Brigade. The foundation's aim is to gather 70,000 high school kids in the Texas Stadium in Dallas for a self-esteem revival meeting.
Bartmann hopes executive coaching, which will be his sole source of income, will pull in $100,000 a day. "I will do enough consulting that we will live well. But the G-IV can't happen, and I'm good with that. It would be a shame to make success a priority. There will be fewer zeroes behind my name, but it won't be bad. I've moved from success to significance."
MARKETING
I Was A Loser, Baby, So Why Don't You Pay Me?
Bill Bartmann milks his losses on the motivational circuit
"The secret of success is elegantly simple, like the law of gravity. This is how the world works: You need to be willing to take risks. You don't because you are afraid of failure; you are worried about what people will think. And that's because of low self-esteem. To succeed, you have to raise your self-esteem."
This is Bill Bartmann's come-on, his preamble, the most concise explanation he can offer for why he became a billionaire and those listening to him haven't even come close. On a Saturday in September, a couple dozen people, mostly entrepreneurs, of their own volition or at the urging of frustrated spouses and business partners, have paid $49 to attend a daylong motivational seminar with Bartmann at the Reno-Sparks Convention Center in Nevada.
None are familiar with his name. Bartmann, 58, made (and lost) his fortune in the debt collection business, and both his rise and fall brought him to national attention, if fleetingly. He was well-known in the financial world and in the business schools that studied his then-pioneering approaches to raising money; in Tulsa, where he was among the biggest employers; and in certain Las Vegas casinos, where he was a regular at the blackjack tables. But few outside those circles would have followed his tumultuous career.
"Why am I an authority? Because I've lived it. I've seen both sides of self-esteem. I went from poverty and welfare to being the 25th-wealthiest person in America. I went from being a high-school dropout to becoming a lawyer. I went from working at a traveling carnival to being named National Entrepreneur of the Year. I went from being a million dollars in debt to a billion dollars to the good. I learned how to change. I went through a significant, some say remarkable, transformation."
Bill Bartmann's career has been one of seized opportunities: oil in the 1980s, consumer debt in the 1990s, and now, in the age of Oprah, inspiration. Motivation, in all its guises, has become a $10 billion industry even as its very premise--that its advice can change lives--remains unproven. Companies spend close to $3 billion a year to hire people, among them former executives, sports coaches, and survivors of all kinds, to inspire and train employees; books ranging from The 7 Habits of Highly Effective People to The Millionaire Next Door are on BusinessWeek's list of longtime best-sellers.
Richard, one of the main characters in the recent indie film hit Little Miss Sunshine, is a struggling motivational speaker. Michael Arndt, who wrote the movie's screenplay, which won this year's Academy Award, says: "Richard thinks he has a big idea that is going to transform his life. That's a very American fantasy." The success of motivational gurus such as Tony Robbins, Stephen R. Covey, and Zig Ziglar, all of whom have become multimillion-dollar brands, has helped promote the notion that selling inspiration can be a career choice. "If this becomes what I think it can, I'm back," Bartmann says.
The tradition of teaching success goes back to Benjamin Franklin's Poor Richard's Almanack and runs through Dale Carnegie's How to Win Friends & Influence People to Robert J. Ringer's Looking Out for #1. The motivation business is sustained by a peculiarly American belief in reinvention combined with our nearly perpetual feelings of dissatisfaction. The big ideas, the supposed secrets of achievement, reflect and influence the cultural zeitgeist. Today those who have overcome adversity--they may be victims of disease, accidents, or their own daring or recklessness--have captured the national imagination. Such as: the guy who chopped off his arm to save his life (Aron Ralston); the woman who treated herself for breast cancer in the South Pole (Jerri Nielsen); the prison guy who became a top chef (Jeff Henderson).
In business, it is adversity of your own making that resonates today; learning from failure is an animating force for everything from innovation to ethics. "Redemption is big," says Jonathan Black, author of Yes You Can! Behind the Hype and Hustle of the Motivation Biz. "But you have to admit you've failed. Otherwise people think you're covering up." One notable player on the failure circuit is Scott Waddle, commander of the nuclear submarine that accidentally sank a Japanese fishing boat in 2001, killing nine. Stripped of his command and retired, Waddle now earns $10,000 a talk, the most popular of which is "Failure Is Not Final."
By anyone's estimation, Bill Bartmann's life has been one of head-swelling highs and depressing lows. Bartmann made his first fortune running a company that manufactured oil pipes; the collapse of crude prices in 1985 left him without any customers and a million dollars in debt. A year later he had started Commercial Financial Services Inc. with the notion that he might be able to squeeze money out of debtors everyone else had given up on.
In 1997, Bartmann and his wife, Kathy, who together owned 80% of CFS, made Forbes' list of the 400 wealthiest Americans, a distinction Bill had long coveted. The magazine estimated the couple to be worth about $1.1 billion. "We had lots of arguments about getting rich," says Kathy. "I don't care if I'm on any lists, if anyone knows me." Bill worked in an office with a statue of Don Quixote, traveled with a security detail, didn't join the country club as was expected, and generally annoyed the Tulsa business community with his swagger.
HITTING BOTTOM
By then CFS had become the world's largest holder of bad consumer debt; unlike its competitors it owned the debt outright and was trying to collect on some $14.5 billion. A few skeptics wondered at the company's rapid growth; by many accounts its revenues had doubled every year since 1994. When Goldman Sachs Group Inc. (GS ) proposed taking the company public the summer of 1998, CFS was reckoned to be worth $3 billion.
Four months later, credit agencies received an anonymous one-page letter accusing CFS of shady dealings. Bartmann would eventually be indicted on 57 federal counts of fraud, conspiracy, and money laundering related to an alleged scheme to make the company's collection rate appear higher than it was. After an 89-day trial in 2003, he was acquitted of all the charges against him (his partner, Jay L. Jones, pleaded guilty to conspiracy and served 3 1/2 years of a five-year prison term). It took Bartmann until 2006 to contend with various civil suits, and during that time he declared bankruptcy. He and Kathy had to give up 19 acres of their 20-acre estate. They lived off credit cards, borrowing from their two daughters when they had to.
Then Bartmann had his next big idea. As a businessman he had used the story of his early life to put lenders at ease, motivate employees, and in general establish his credibility as someone who can prosper in any circumstances. His setbacks began feeling like an affirmation of sorts: The wild arcs of his life offered valuable lessons. "I think I am the message, the answer," he says. "I'm selling the one thing everyone needs: self-confidence."
Successful motivational speakers require a measure of arrogance: They must believe they can compel an audience, essentially, to do their bidding. Bartmann's charisma is more closely related to cunning than magnetism. A 2001 New Yorker profile described him as having "an uncanny gift for making those who work for him feel that they share in his powers." A former senior executive at CFS, Wayne Learned, knows all about that. He actually invested in another doomed Bartmann enterprise after the CFS collapse, and depicts his former boss this way: "Persuasive isn't the right word to describe him. You do it to yourself as much as anything.... He'll call when he wants something and then he sucks you dry. You're like a raisin. When you work for him, you're his raisin."
What so mesmerizes people seems awkwardly exaggerated onstage in Reno. Bartmann's physical presence can be intimidating: He has a compact body, a rough voice, and an almost feral energy. His hands are those of someone who spent his adolescent years brawling. He wears a tight-fitting shirt and dark suit. He uses the language of the bill collector; he retains an outsider's posture.
SELLING, MODESTLY
If anyone understands the value of seeming authentic, it is Bill Bartmann. "Dogs know when you are afraid," he says. "People know when you're b.s.-ing them. You have to respect that."
Hence that afternoon's onstage pitch: "I'm awkward about this. I haven't set you up for the kill," he says. "I don't want to be a snake-oil salesman. I'm in the motivational, self-help industry. It's an industry that ranks right there with car salesmen. I don't know if I'm any better, any different, but I do know a few things.... Have I done a good job of un-selling?"
Bartmann carries around a laminated card in his pocket. On it is typed the following goal: touch 10 million people in five years. "It's a giant, crazy, ludicrous number," he says. "But if I touch enough people, the money should follow."
Bartmann started out using a classic motivational business model: sell tickets cheaply, give them away if necessary, and push packages of DVDs and books (his is titled Billionaire Secrets to Success). In Reno, he sold several thousand dollars' worth. And he added 31 names to those already on his spreadsheet: nine million, nine hundred and eighty thousand to go.
Though there may be as many as 15,000 people trying to make a living as professional speakers, it is quite possible for the ones who catch on eventually to make a couple of million dollars a year. Bartmann, never one to shy from the grandiose, wants to make $100 million. So in the weeks after Reno he began holding free two-hour seminars around the country as a way to advertise a yearlong mentoring program, conducted mostly over the phone and by e-mail at a cost of $397 a month.
It hasn't escaped his attention that personal coaching has become a billion-dollar business itself. By December, he had visited seven cities and said he signed up 200 students. "If I don't make $100 million a year from now, I'll be disappointed," he said then. He had other aims, too: "At 25 cities, I'll get a little plane. At 50 cities, a bigger one. I miss my G-IV. The plane is my measure of success. If I touch enough people to get a plane, I'll be a financial success."
Bartmann's problem, though, was that success hinged on an admission of failure, which initially he was disinclined to make. That part of his story, he believed, couldn't so neatly be explained and wasn't appropriate at most of his seminars. "I don't amplify the CFS loss. It isn't inspirational," he said in September. "I say 'self-made billionaire.' I don't use the present tense. I know they can Google me and find out everything."
Then in March, Bartmann found himself in Hollywood, where the idea of him hosting a "Dr. Failure" talk show somehow, wondrously, came up. By that time Bartmann was ready to tell his whole story. "I talk about CFS first.... There is no more compelling story. Kathy said I should quit acting like I'm ashamed. It makes everyone else's problems seem smaller. It gives them permission to tell me about their problems." He came up with a new pitch: "To do for failure what Betty Ford did for alcoholism and Susan Komen did for breast cancer."
Bartmann has also made a decision that, in part, reflects his wish to be admired as well as recognized: All profits from his mentor program, for which he now charges $797 a month, and his appearances at Get Motivated! Tours (where he is one of dozens alongside George Foreman and Zig Ziglar) and Learning Annex Real Estate and Wealth Expos (where he is one of dozens alongside Donald Trump and Tony Robbins) will go into a foundation called Bill's Brigade. The foundation's aim is to gather 70,000 high school kids in the Texas Stadium in Dallas for a self-esteem revival meeting.
Bartmann hopes executive coaching, which will be his sole source of income, will pull in $100,000 a day. "I will do enough consulting that we will live well. But the G-IV can't happen, and I'm good with that. It would be a shame to make success a priority. There will be fewer zeroes behind my name, but it won't be bad. I've moved from success to significance."
Bill Bartmann
http://money.cnn.com/magazines/fortune/fortune_archive/1999/10/25/267811/index.htm
News | Markets | Technology | Personal Finance | Small Business | CNN.com
Subscribe To Fortune Magazine
Magazine Customer Service
RSS
Newsletters
Video
Home
Fortune 500
Technology
Investing
Management
Rankings
How To Lose A Billion Bill Bartmann amassed a huge personal fortune building a wildly profitable company. Too bad it was all on paper.
By Jerry Useem
October 25, 1999
(FORTUNE Magazine) – Only a year ago Bill Bartmann was one of the richest people in America, 80% owner of an insanely profitable company that bankers valued at as much as $6 billion. He traveled with a coterie of armed bodyguards, entered a Las Vegas arena in a sedan chair to wrestle Hulk Hogan, became one of the nation's top political donors, and even had a state holiday named after him. "I have enough money," Bartmann boasted in 1997, "that if I set it all on fire, I'd be dead before it went out."
There was just one problem with that boast: His wealth was all on paper. And now it's all gone. More from Fortune
The gravity-defying debt problem
Tracking the bear: How bad could it get?
Murdoch's daughter snubs dad
FORTUNE 500
Current Issue
Subscribe to Fortune
Bartmann's privately held debt-collection company, Commercial Financial Services, imploded spectacularly late last year, turning one of America's largest personal fortunes into one of the briefest. Whether or not the allegations of fraud that brought it down are true remains unclear. But at a time when Internet entrepreneurs are amassing paper wealth based on stock appreciation at a pace never before seen, the astonishing reversal of fortune comes as a chilling reminder that paper wealth is just that: paper, apt to vanish even faster than it appeared.
The first thing that visitors to Bartmann's 20-acre estate in Tulsa see is an imposing brick wall, ten feet high and more than a half-mile long. It was built to keep intruders out, but look closely and you'll notice that the spiffy new guardhouse stands empty, its huge cast-iron gates permanently ajar. Inside, the defrocked billionaire acknowledges that few people want to come in anymore. "My fortunes are obviously not what they used to be at one time," he says, lighting a cigarette inside his airy mansion and taking a long drag. Over his shoulder is a plastic model of his beloved Gulfstream jet, sold off some months ago. A statue of Bartmann dressed as Julius Caesar stands, Ozymandias-like, in the corner.
The real Bill Bartmann is a shrimpy, tightly coiled 50-year-old who singlehandedly put a seedy and inefficient industry on the road to respectability. Charismatic, combative, egotistical, both feared and revered by his troops, he built CFS into a formidable debt-collection empire. Yet all the while, he spurned any chance to sell even some of his stock.
Not that he was unaware of the danger this presented. "It's ephemeral," he told me in 1997. "It isn't real until it's in the bank." At the time, Bartmann had been talking about selling a 20% stake in the company to Goldman Sachs for $600 million, which would have given Bartmann all the hard cash he wanted. "Bill Bartmann wouldn't mind having a coffee can [full of money] buried in the backyard," he reasoned then.
But today we're standing in Bill Bartmann's yard, and while there's a driving range and a very attractive pond with a fountain, the coffee can is conspicuously absent. What happened? Bartmann says that he walked away from the Goldman discussions because he thought the numbers were too low. "I remember turning that down with righteous indignation," he says, pausing to turn the absurdity of the statement over in his mind. "If I could go back and undo one decision, that probably would be a good one."
Others would do just as well. In 1997, Norwest Bank chief Dick Kovacevich met with Bartmann to talk about acquiring CFS outright, says Bartmann. (Kovacevich, now CEO of Wells Fargo, confirms the meeting but declines to comment on what was discussed; Goldman also declines to comment.) The offer, according to Bartmann: $400 million in cash, plus an estimated $1.2 billion payout over time. But again Bartmann passed. Later, plans for an initial public offering were scotched.
Most inexplicable of all was Bartmann's decision to take $50 million he had managed to put in the bank (through dividends and his $2-million-a-year salary) and use it to buy...more CFS stock. This he purchased from Jay Jones, co-founder and 20% owner of CFS, who, according to Bartmann, had long been pleading for a way to get liquid. (Jones, through his lawyer, declined to comment.) What was Bartmann thinking? Besides doing a favor for his friend Jay, Bartmann claims, he was beguiled by the prospect of acquiring more CFS stock at a relatively cheap price.
"My fallacy," he says, "is I believed CFS was going to last a long time."
Funny thing is, Bartmann has been here before. After dropping out of high school and leaving home in Iowa at age 14, he moved into a barn and joined a gang called the Manor Boys, apparently destined for a life of poverty and punching people. But he unexpectedly got his act together, moved to Oklahoma, and started an oil equipment company that, at its height, employed 70 people. He grew rich. And then poor again: When oil prices sank in 1985, the company abruptly folded, leaving Bartmann $1 million in debt.
Yet it was from this underwater position that Bartmann began an even more spectacular ascent. His big idea came, oddly enough, from the collection agents who had begun menacing him to pay up. With $13,000 wheedled from lenders, he bought a package of bad loans from a local failed bank for 2 cents on the dollar and began calling the debtors from his kitchen table. He was able to collect nearly 10 cents on the dollar, for a 400% return.
By the mid-1990s, Commercial Financial Services was the largest debt-collection company in the land and still growing at a dizzying pace, eventually filling 50 floors in Tulsa's CityPlex Towers. It bought and collected on more than 50% of America's delinquent credit card debt, opened operations in Europe, and took steps to begin issuing its own credit cards. Through an aggressive use of technology and a collecting approach that could best be described as niceness (debtors were called "customers," not "deadbeats"), the firm produced eye-popping profits; its net margins in 1995, for instance, were 65%.
But Bartmann's boldest stroke of all was on Wall Street, where he pioneered the now-common practice of securitizing bad loans. By bundling together thousands of credit card debts and selling them as bonds, CFS got access to huge pools of cash that could be used to purchase even huger amounts of bad debt. The move effectively put the company on steroids, and it carried risks; should CFS prove unable to collect debts at the pace it promised bondholders, its ratings could sink and the cash flow dry up.
None of this seemed to temper Bartmann's messianic faith in CFS and himself. Associates say his drive to push the company's market value to new heights--and thereby rise on the roster of America's wealthiest--verged on obsession. "There's only one scorecard," Bartmann told a lieutenant, "and that's money."
Bartmann certainly was racking up points on his personal scorecard. In September 1997, BT Alex. Brown estimated CFS to be worth between $5.1 billion and $5.8 billion, placing Bartmann's personal worth somewhere north of $4 billion. Meanwhile, Bartmann was inducted into the American Academy of Achievement, alongside such luminaries as Michael Dell and filmmaker James Cameron. (At the ceremony, he persuaded former CIA chief James Woolsey to join the CFS board.) And per order of Governor Frank Keating, April 30, 1998, was Bill Bartmann Day in Oklahoma.
By this time Bartmann had hired former Secret Service agents to tail him everywhere, including, at times, to the bathroom. He'd also become known for outsized displays of generosity, flying thousands of employees to retreats in the Caribbean and Las Vegas. At one Vegas gathering, Bartmann was carried into the Thomas & Mack arena on a sedan chair dressed as Julius Caesar while 6,000 employees and guests looked on. Legionnaires blew horns at his arrival, and dancing girls threw flowers in his path. "Twelve years from now there will be 30,000 CFS people!" Bartmann bellowed from the ring after Hulk Hogan appeared to wrestle him.
In some eyes the Caesar costume fit Bartmann all too well. "This man is a tyrant," says one ex-employee. "I don't know how many times he called me a dumb bastard SOB." That wasn't the worst of it. On April 21, 1998, Bartmann herded 600 or so employees into a room and accused them of cheating. In a profanity-strewn philippic that likened the employees to Nazis and Klansmen, Bartmann suggested taking a machete to every third person in the room or else placing red dots on their foreheads to "blow [their] fucking brains out," according to a suit filed by 20 of the employees.
Six months later Bartmann's reign was over. And it ended with breathtaking speed. On Sept. 30, 1998, the major bond-rating agencies received an anonymous letter suggesting that CFS, to hide shortfalls in collections from bondholders, had resorted to covertly selling off loan inventory and reporting those sales as collections. Worse still, the buyer of much of that inventory was a shadowy entity called Dimat Corp., which had ties to Jay Jones, Bartmann's 20% partner.
Had Bartmann's $50 million payment to Jones actually been part of a desperate scheme to funnel money through Dimat and prop up a faltering business model? Bartmann insists not. As he tells the story, Jones--stripped of his operational duties by Bartmann and presumably humiliated--had formed Dimat without Bartmann's knowledge to purchase and resell bad loans on the side. It was only after the allegations hit, Bartmann says, that he stormed into Jones' office and demanded to know whether Jones was connected to Dimat. When Jones didn't deny it, Bartmann began to cry.
Many who know Bartmann find the maudlin tale of betrayal hard to believe (if only because it seems lifted wholesale from The Godfather, with Jones playing the role of Fredo). "If a paper clip moves at CFS," one CFS executive told me, "Bill knows about it."
Eyes dry and bravado intact, Bartmann flew to New York sans lawyer to tell a room full of hostile bondholders that he was temporarily stepping down as CEO to let PriceWaterhouse and a law firm conduct investigations. But the death spiral had begun. Within days every rating agency had downgraded or suspended its ratings of CFS bonds, leaving $1.5 billion twisting in the wind. The company's bank lenders responded by cutting off credit. Fearful they'd be left holding the bag, unsecured creditors threatened to file an involuntary bankruptcy petition, causing CFS to file Chapter 11 on Dec. 11. The sudden free fall, says Bartmann, was like a "bobsled to hell." Then, in April, he crashed his motorcycle during a parade, breaking his leg in three places.
Save for a skeleton crew, the last of CFS's 3,900 employees were laid off this month. Whether it was impropriety or just the appearance thereof that felled the company is a question that will be hashed out in the courtroom. NationsBank has sued the company for fraud, and CFS is demanding that Bartmann repay $17 million he borrowed from the company. But there's one thing all parties seem able to agree on: Bartmann came away from the mess nearly broke. "Whatever Bill was doing," says Mike Zarrilli, his former contact at Chase, "it doesn't appear he was doing it to put short-term cash in his pocket."
Staring at a copy of the anonymous letter--the piece of paper that ruined all that paper wealth--Bartmann offers an epitaph that's at once a defense and a self-indictment. "I didn't grab the loot and scoot," he says. "I gave it to Mr. Jones to buy more of a worthless corporation."
News | Markets | Technology | Personal Finance | Small Business | CNN.com
Subscribe To Fortune Magazine
Magazine Customer Service
RSS
Newsletters
Video
Home
Fortune 500
Technology
Investing
Management
Rankings
How To Lose A Billion Bill Bartmann amassed a huge personal fortune building a wildly profitable company. Too bad it was all on paper.
By Jerry Useem
October 25, 1999
(FORTUNE Magazine) – Only a year ago Bill Bartmann was one of the richest people in America, 80% owner of an insanely profitable company that bankers valued at as much as $6 billion. He traveled with a coterie of armed bodyguards, entered a Las Vegas arena in a sedan chair to wrestle Hulk Hogan, became one of the nation's top political donors, and even had a state holiday named after him. "I have enough money," Bartmann boasted in 1997, "that if I set it all on fire, I'd be dead before it went out."
There was just one problem with that boast: His wealth was all on paper. And now it's all gone. More from Fortune
The gravity-defying debt problem
Tracking the bear: How bad could it get?
Murdoch's daughter snubs dad
FORTUNE 500
Current Issue
Subscribe to Fortune
Bartmann's privately held debt-collection company, Commercial Financial Services, imploded spectacularly late last year, turning one of America's largest personal fortunes into one of the briefest. Whether or not the allegations of fraud that brought it down are true remains unclear. But at a time when Internet entrepreneurs are amassing paper wealth based on stock appreciation at a pace never before seen, the astonishing reversal of fortune comes as a chilling reminder that paper wealth is just that: paper, apt to vanish even faster than it appeared.
The first thing that visitors to Bartmann's 20-acre estate in Tulsa see is an imposing brick wall, ten feet high and more than a half-mile long. It was built to keep intruders out, but look closely and you'll notice that the spiffy new guardhouse stands empty, its huge cast-iron gates permanently ajar. Inside, the defrocked billionaire acknowledges that few people want to come in anymore. "My fortunes are obviously not what they used to be at one time," he says, lighting a cigarette inside his airy mansion and taking a long drag. Over his shoulder is a plastic model of his beloved Gulfstream jet, sold off some months ago. A statue of Bartmann dressed as Julius Caesar stands, Ozymandias-like, in the corner.
The real Bill Bartmann is a shrimpy, tightly coiled 50-year-old who singlehandedly put a seedy and inefficient industry on the road to respectability. Charismatic, combative, egotistical, both feared and revered by his troops, he built CFS into a formidable debt-collection empire. Yet all the while, he spurned any chance to sell even some of his stock.
Not that he was unaware of the danger this presented. "It's ephemeral," he told me in 1997. "It isn't real until it's in the bank." At the time, Bartmann had been talking about selling a 20% stake in the company to Goldman Sachs for $600 million, which would have given Bartmann all the hard cash he wanted. "Bill Bartmann wouldn't mind having a coffee can [full of money] buried in the backyard," he reasoned then.
But today we're standing in Bill Bartmann's yard, and while there's a driving range and a very attractive pond with a fountain, the coffee can is conspicuously absent. What happened? Bartmann says that he walked away from the Goldman discussions because he thought the numbers were too low. "I remember turning that down with righteous indignation," he says, pausing to turn the absurdity of the statement over in his mind. "If I could go back and undo one decision, that probably would be a good one."
Others would do just as well. In 1997, Norwest Bank chief Dick Kovacevich met with Bartmann to talk about acquiring CFS outright, says Bartmann. (Kovacevich, now CEO of Wells Fargo, confirms the meeting but declines to comment on what was discussed; Goldman also declines to comment.) The offer, according to Bartmann: $400 million in cash, plus an estimated $1.2 billion payout over time. But again Bartmann passed. Later, plans for an initial public offering were scotched.
Most inexplicable of all was Bartmann's decision to take $50 million he had managed to put in the bank (through dividends and his $2-million-a-year salary) and use it to buy...more CFS stock. This he purchased from Jay Jones, co-founder and 20% owner of CFS, who, according to Bartmann, had long been pleading for a way to get liquid. (Jones, through his lawyer, declined to comment.) What was Bartmann thinking? Besides doing a favor for his friend Jay, Bartmann claims, he was beguiled by the prospect of acquiring more CFS stock at a relatively cheap price.
"My fallacy," he says, "is I believed CFS was going to last a long time."
Funny thing is, Bartmann has been here before. After dropping out of high school and leaving home in Iowa at age 14, he moved into a barn and joined a gang called the Manor Boys, apparently destined for a life of poverty and punching people. But he unexpectedly got his act together, moved to Oklahoma, and started an oil equipment company that, at its height, employed 70 people. He grew rich. And then poor again: When oil prices sank in 1985, the company abruptly folded, leaving Bartmann $1 million in debt.
Yet it was from this underwater position that Bartmann began an even more spectacular ascent. His big idea came, oddly enough, from the collection agents who had begun menacing him to pay up. With $13,000 wheedled from lenders, he bought a package of bad loans from a local failed bank for 2 cents on the dollar and began calling the debtors from his kitchen table. He was able to collect nearly 10 cents on the dollar, for a 400% return.
By the mid-1990s, Commercial Financial Services was the largest debt-collection company in the land and still growing at a dizzying pace, eventually filling 50 floors in Tulsa's CityPlex Towers. It bought and collected on more than 50% of America's delinquent credit card debt, opened operations in Europe, and took steps to begin issuing its own credit cards. Through an aggressive use of technology and a collecting approach that could best be described as niceness (debtors were called "customers," not "deadbeats"), the firm produced eye-popping profits; its net margins in 1995, for instance, were 65%.
But Bartmann's boldest stroke of all was on Wall Street, where he pioneered the now-common practice of securitizing bad loans. By bundling together thousands of credit card debts and selling them as bonds, CFS got access to huge pools of cash that could be used to purchase even huger amounts of bad debt. The move effectively put the company on steroids, and it carried risks; should CFS prove unable to collect debts at the pace it promised bondholders, its ratings could sink and the cash flow dry up.
None of this seemed to temper Bartmann's messianic faith in CFS and himself. Associates say his drive to push the company's market value to new heights--and thereby rise on the roster of America's wealthiest--verged on obsession. "There's only one scorecard," Bartmann told a lieutenant, "and that's money."
Bartmann certainly was racking up points on his personal scorecard. In September 1997, BT Alex. Brown estimated CFS to be worth between $5.1 billion and $5.8 billion, placing Bartmann's personal worth somewhere north of $4 billion. Meanwhile, Bartmann was inducted into the American Academy of Achievement, alongside such luminaries as Michael Dell and filmmaker James Cameron. (At the ceremony, he persuaded former CIA chief James Woolsey to join the CFS board.) And per order of Governor Frank Keating, April 30, 1998, was Bill Bartmann Day in Oklahoma.
By this time Bartmann had hired former Secret Service agents to tail him everywhere, including, at times, to the bathroom. He'd also become known for outsized displays of generosity, flying thousands of employees to retreats in the Caribbean and Las Vegas. At one Vegas gathering, Bartmann was carried into the Thomas & Mack arena on a sedan chair dressed as Julius Caesar while 6,000 employees and guests looked on. Legionnaires blew horns at his arrival, and dancing girls threw flowers in his path. "Twelve years from now there will be 30,000 CFS people!" Bartmann bellowed from the ring after Hulk Hogan appeared to wrestle him.
In some eyes the Caesar costume fit Bartmann all too well. "This man is a tyrant," says one ex-employee. "I don't know how many times he called me a dumb bastard SOB." That wasn't the worst of it. On April 21, 1998, Bartmann herded 600 or so employees into a room and accused them of cheating. In a profanity-strewn philippic that likened the employees to Nazis and Klansmen, Bartmann suggested taking a machete to every third person in the room or else placing red dots on their foreheads to "blow [their] fucking brains out," according to a suit filed by 20 of the employees.
Six months later Bartmann's reign was over. And it ended with breathtaking speed. On Sept. 30, 1998, the major bond-rating agencies received an anonymous letter suggesting that CFS, to hide shortfalls in collections from bondholders, had resorted to covertly selling off loan inventory and reporting those sales as collections. Worse still, the buyer of much of that inventory was a shadowy entity called Dimat Corp., which had ties to Jay Jones, Bartmann's 20% partner.
Had Bartmann's $50 million payment to Jones actually been part of a desperate scheme to funnel money through Dimat and prop up a faltering business model? Bartmann insists not. As he tells the story, Jones--stripped of his operational duties by Bartmann and presumably humiliated--had formed Dimat without Bartmann's knowledge to purchase and resell bad loans on the side. It was only after the allegations hit, Bartmann says, that he stormed into Jones' office and demanded to know whether Jones was connected to Dimat. When Jones didn't deny it, Bartmann began to cry.
Many who know Bartmann find the maudlin tale of betrayal hard to believe (if only because it seems lifted wholesale from The Godfather, with Jones playing the role of Fredo). "If a paper clip moves at CFS," one CFS executive told me, "Bill knows about it."
Eyes dry and bravado intact, Bartmann flew to New York sans lawyer to tell a room full of hostile bondholders that he was temporarily stepping down as CEO to let PriceWaterhouse and a law firm conduct investigations. But the death spiral had begun. Within days every rating agency had downgraded or suspended its ratings of CFS bonds, leaving $1.5 billion twisting in the wind. The company's bank lenders responded by cutting off credit. Fearful they'd be left holding the bag, unsecured creditors threatened to file an involuntary bankruptcy petition, causing CFS to file Chapter 11 on Dec. 11. The sudden free fall, says Bartmann, was like a "bobsled to hell." Then, in April, he crashed his motorcycle during a parade, breaking his leg in three places.
Save for a skeleton crew, the last of CFS's 3,900 employees were laid off this month. Whether it was impropriety or just the appearance thereof that felled the company is a question that will be hashed out in the courtroom. NationsBank has sued the company for fraud, and CFS is demanding that Bartmann repay $17 million he borrowed from the company. But there's one thing all parties seem able to agree on: Bartmann came away from the mess nearly broke. "Whatever Bill was doing," says Mike Zarrilli, his former contact at Chase, "it doesn't appear he was doing it to put short-term cash in his pocket."
Staring at a copy of the anonymous letter--the piece of paper that ruined all that paper wealth--Bartmann offers an epitaph that's at once a defense and a self-indictment. "I didn't grab the loot and scoot," he says. "I gave it to Mr. Jones to buy more of a worthless corporation."
Tuesday, February 24, 2009
OBJECT PREVALENCE
http://www.advogato.org/article/398.html
OBJECT PREVALENCE
Posted 23 Dec 2001 at 03:46 UTC by KlausWuestefeld
Transparent Persistence, Fault-Tolerance and Load-Balancing for Java Systems.
Orders of magnitude FASTER and SIMPLER than a traditional DBMS. No pre or post-processing required, no weird proprietary VM required, no base-class inheritance or clumsy interface definition required: just PLAIN JAVA CODE.
How is this possible?
Question: RAM is getting cheaper every day. Researchers are announcing major breakthroughs in memory technology. Even today, servers with multi-gigabyte RAM are commonplace. For many systems, it's already feasible to keep all business objects in RAM. Why can't I simply do that and forget all the database hassle?
Answer: You can, actually.
Are you crazy? What if there's a system crash?
To avoid losing data, every night your system server saves a snapshot of all business objects to a file using plain object serialization.
What about the changes occurred since the last snapshot was taken? Won't the system lose those in a crash?
No.
How come?
All commands received from the system's clients are converted into serializable objects by the server. Before being applied to the business objects, each command is serialized and written to a log file. During crash recovery, first, the system retrieves its last saved state from the snapshot file. Then, it reads the commands from the log files created since the snapshot was taken. These commands are simply applied to the business objects exactly as if they had just come from the system's clients. The system is then back in the state it was just before the crash and is ready to run.
Does that mean my business objects have to be deterministic?
Yes. They must always produce the same state given the same commands.
Doesn't the system have to stop or enter read-only mode in order to produce a consistent snapshot?
No. That is a fundamental problem with transparent or orthogonal persistence projects like PJama (http://www.dcs.gla.ac.uk/pjava/) but it can be solved simply by having all system commands queued and routed through a single place. This enables the system to have a replica of the business logic on another virtual machine. All commands applied to the "hot" system are also read by the replica and applied in the exact same order. At backup time, the replica stops reading the commands and its snapshot is safely taken. After that, the replica continues reading the command queue and gets back in sync with the "hot" system.
Doesn't that replica give me fault-tolerance as a bonus?
Yes it does. I have mentioned one but you can have several replicas. If the "hot" system crashes, any other replica can be elected and take over. Of course, you must be able to afford a machine for every replica you want.
Does this whole scheme have a name?
Yes. It is called system prevalence. It encompasses transparent persistence, fault-tolerance and load-balancing.
If all my objects stay in RAM, will I be able to use SQL-based tools to query my objects' attributes?
No. You will be able to use object-based tools. The good news is you will no longer be breaking your objects' encapsulation.
What about transactions? Don't I need transactions?
No. The prevalence design gives you all transactional properties without the need for explicit transaction semantics in your code.
How is that?
DBMSs tend to support only a few basic operations: INSERT, UPDATE and DELETE, for example. Because of this limitation, you must use transaction semantics (begin - commit) to delimit the operations in every business transaction for the benefit of your DBMS. In the prevalent design, every transaction is represented as a serializable object which is atomically written to the queue (a simple log file) and processed by the system. An object, or object graph, is enough to encapsulate the complexity of any business transaction.
What about business rules involving dates and time? Won't all those replicas get out of sync?
No. If you ask the use-case gurus, they will tell you: "The clock is an external actor to the system.". This means that clock ticks are commands to the business objects and are sequentially applied to all replicas, just like all other commands.
Is object prevalence faster than using a database?
The objects are always in RAM, already in their native form. No disk access or data marshalling is required. No persistence hooks placed by preprocessors or postprocessors are required in your code. No "isDirty" flag. No restrictions. You can use whatever algorithms and data-structures your language can support. Things don't get much faster than that.
Besides being deterministic and serializable, what are the coding standards or restrictions my business classes have to obey?
None whatsoever. To issue commands to your business objects, though, each command must be represented as a serializable object. Typically, you will have one class for each use-case in your system.
How scalable is object prevalence?
The persistence processes run completely in parallel with the business logic. While one command is being processed by the system, the next one is already being written to the log. Multiple log files can be used to increase throughput. The periodic writing of the snapshot file by the replica does not disturb the "hot" system in the slightest. Of course, tests must be carried out to determine the actual scalability of any given implementation but, in most cases, overall system scalability is bound by the scalability of the business classes themselves.
Can't I use all those replicas to speed things up?
All replicas have to process all commands issued to the system. There is no great performance gain, therefore, in adding replicas to command-intensive systems. In query-intensive systems such as most Web applications, on the other hand, every new replica will boost the system because queries are transparently balanced between all available replicas. To enable that, though, just like your commands, each query to your business logic must also be represented as a serializable object.
Isn't representing every system query as a serializable object a real pain?
That's only necessary if you want transparent load-balancing, mind you. Besides, the queries for most distributed applications arrive in a serializable form anyway. Take Web applications for example: aren't HTTP request strings serializable already?
Does prevalence only work in Java?
No. You can use any language for which you are able to find or build a serialization mechanism. In languages where you can directly access the system's memory and if the business objects are held in a specific memory segment, you can also write that segment out to the snapshot file instead of using serialization.
Is there a Java implementation I can use?
Yes. You will find Prevayler - The Open-Source Prevalence Layer, an example application and more information at http://www.prevayler.org. It does not yet implement automatic load-balancing but it does implement transparent business object persistence and replication is in the oven.
Is Prevayler reliable?
Prevayler's robustness comes from its simplicity. It is orders of magnitude simpler than the simplest RDBMS. Although I wouldn't use Prevayler to control a nuclear plant just yet, its open-source license ensures the whole of the software developing community the ability to scrutinize, optimize and extend Prevayler. The real questions you should bear in mind are: "How robust is my Java Virtual Machine?" and "How robust is my own code?". Remember: you will no longer be writing feeble client code. You will now have the means to actually write server code. It's the way object orientation was intended all along; but it's certainly not for wimps.
You said Prevayler is open-source software. Do you mean it's free?
That's right. It's licensed under the Lesser General Public License.
But what if I'm emotionally attached to my database?
For many applications, prevalence is a much faster, much cheaper and much simpler way of preserving your objects for future generations. Of course, there will be all sorts of excuses to hang on to "ye olde database", but at least now there is an option.
---------------------------------------------------------------------
ABOUT THE AUTHOR
KlausWuestefeld enjoys writing good software and helping other people do the same. He has been doing so for 17 years now. He can be contacted at klaus@objective.com.br.
---------------------------------------------------------------------
"PREVAYLER" and "OPEN-SOURCE PREVALENCE LAYER" are trademarks of Klaus Wuestefeld.
Copyright (C) 2001 Klaus Wuestefeld.
Unmodified, verbatim copies of this text including this copyright notice can be freely made.
Interesting but..., posted 23 Dec 2001 at 07:08 UTC by ncm »
There are still quite a few things we really need transactions for:
When you make the first of a series of changes to objects in the database, you typically break one or more database invariants until you get the last change entered. Other processes looking at the database had better either wait, or had better see the state it had before you started. To get much concurrency, you need to snapshot the state before the first change.
If you get halfway through a series of changes and crash, the system had better come back up without the changes you made, because you're not going to be equipped to continue where you left off.
If you get halfway through a series of changes and discover some condition that keeps you from finishing, you had better be able to just drop the changes and pick up with the original snapshot.
If N processes make a series of conflicting changes concurrently, (N-1) of them had better be told that their changes have failed, and that they must try again.
There's a reason that databases are written by career professionals. A simple object database can be really useful, but that doesn't make it a substitute for the real thing. That's part of why so many "object database" companies failed some ten years back.
Transactions, posted 23 Dec 2001 at 09:03 UTC by Pseudonym »
Actually, transactions are not so important in the external interface of an OODBMS. In an RDBMS, a manipulation typically involves several SQL statements (e.g. insert, update, remove) each of which can act on only one table at a time. So if a transaction needs to manipulate more than one table, you need to ensure that the set of statements is atomic by issuing a transaction.
In an OODBMS, where manipulation methods can operate on more than one class, the need is reduced somewhat. Internally, you can just queue up the command logs until the method is complete, then write them out together. Then the problem becomes entirely one of synchronisation. It's not quite ACID, but it'll do for most business applications.
You (ncm) are right, however, in that this solution, while no doubt excellent for many purposes (e.g. if you're happy with the robustness and performance of MySQL, you'll probably be happy with this, too), won't scale to many critical applications. For example, it would be quite hard to handle replication in any sane manner.
Klaus, as a matter of interest, how did you manage to get Java to force flushing to disk?
Scalability ?, posted 23 Dec 2001 at 12:03 UTC by jneves »
Is it just me, or prevayler isn't useful in anything else but a uniprocessor machine as is? You process requests one at a time, which means that there can't be 2 different requests being processed at the same time in different processors. And when you have several replicas you have to have some coordination between all replicas to insure the order of the requests. Or am I missing something here ?
Interesting but problematic, posted 23 Dec 2001 at 12:59 UTC by dannu »
Thank you (Klaus Wuestefeld) for your nice write-up. I mostly agree to the points made by the other repliers.
Let me discuss/ask some further points:
distributed systems? if a systen-prevalence-deployed application contacts other services resp. other servers you have a synchronization problem. How do you handle that? I guess, that you end up doing a 2PC-like synchronization between your prevalence servers.
fine grained tx-model versus all or nothing? with big BOs -systems there are actually lots of small transactions. the system prevalence paradigma doesn't give you a fine grained application side control, or does it? Note though that you can adapt (extended) 2PC-transactions to efficiently work RAM-based while retaining persistent storage properties (by using RAM-based subtransactions and files or RDBMS in the root transaction).
scalabity? having all "commands queued and routed through a single place" doesn't scale very well. consider one of these big 64 processor multigigabyte machines using a gigabit card: you wouldn't want all requests to be serialized through a single bottleneck which involves IO. With fine grained distributed transactions you don't need this "single place" or even a single server. I appreciate the "do it in background" approach, though, as an advance to requiring requests to be queued while saving the state. It's quite neccesary for 24/7 systems.
In my oppinion the complexity of 2PC-systems comes from shortcomings of the commercial products (BEA WLE, Websphere, Oracle etc.). They impose big clumsy quite old fashioned development schemes where the developer is restricted and has to keep track of many conditions. This partly stems from the pain with underspecified and often incorrectly implemented XA-interfaces. (e.g. writing multithreaded programs with XA-adapters from the main RDBMs is a desaster).
I think that system prevalence would help implementing web applications which are located on single systems. It is a simple enough paradigm to be used and understood by companies which often fail or are very slow with 2pc-transaction systems. Handling of error conditions (pointed out by ncm) might still be a big problem.
just my 2 (soon to be) eurocent and best wishes!
holger
I'll be back..., posted 23 Dec 2001 at 14:08 UTC by KlausWuestefeld »
THANKS A LOT for the FEEDBACK!
This is the first forum outside of my working group to actually get the idea and give me some positive feedback.
I am just leaving on a trip right now (my wife is calling me ;) for Christmas and will be back on wednesday. Then, I will address all concerns: ACID properties, error-condition recovery, scalability, the works...
Just a note on scalability and concurrency to think about over Christmas: Suppose we have a subscriber management system that receives a file from a bank with 100000 (one-hundred-thousand) payment records. A prevalent server running on a regular desktop machine can handle a command/transaction for this in less than a millisecond and be ready for the next command.
Merry Christmas! See you soon.
testing, debugging, integration, and data migration, posted 23 Dec 2001 at 19:33 UTC by jrobbins »
I used to be a professional SmallTalk programmer, I also was a professional Lisp programmer. Both of those languages use the concept of a saved memory image as part of their normal development environment.
The simplicy of "just saving the system state" is a double-edged sword. The downside is that it is often hard to specify a particular system state that you might want to use for testing or debugging. If you ever get an object into a "bad state", it can be very hard to find out how it got into that state. In contrast, the impedence mismatch between OO systems and RDBMSs provides a natural boundary and conceptual bottleneck for testing and debugging. It is realtively easy to compare 2 database dumps to see what is different, or to populate the database with test data, or to see which INSERT statement introduced a particular row into the database. You could have test data consisting of a long set of commands, but that "algebraic" approach to testing does not scale well, and allows defects in mutators to mask defects in accessors.
One thing that I learned while trying to actually sell ST-80 systems to other divisions in a large company is that IS organizations see a standard RDBMS as an integration point. If your system uses an RDMBS, they can plan capacity on a shared database machine: they can generate ad-hoc reports, they can use standard tools for disk backups and such on the database machine only. Also, in the event that your system eventaully dies (is no longer maintained, or the license is not extended or whatever) they will at least have the data in a format that they can get out of your system's tables and into some other system.
Lastly, upgrades were always a pain in image-based tools. Very incremental changes (like adding an instance variable to a class) can be handled by the serialization system. Any reoganization beyond that would require custom coding. In contrast, you can do small and mid-sized reorganizations a lot easier in SQL.
Why bother with disk at all?, posted 23 Dec 2001 at 20:23 UTC by egnor »
I'll take the opposite tack for variety:
If you're going this far, why bother with a disk at all? Just attach a battery to your RAM. If you want reliability, keep replicas. If a replica is lost, "clone" another one by freezing its message queue and copying the frozen image; the two clones can then "catch up" with the queued messages in parallel.
Copy-on-write VM tricks may soften the need to entirely freeze a replica during checkpointing.
I suspect the points raised in most of the comments can be fixed. (After all, suppose we were looking the other way. Compared to modern programming languages, databases and middleware systems have lots of horrible misfeatures, starting with bad syntax and ending with fundamentally broken models of (non-)encapsulation and (non-)reuse and (non-)genericity; the complaints in the other direction seem relatively trivial by comparison. How can any self-respecting software engineer stand to use today's RDBMS systems without feeling dirty all over?)
jrobbins's notes are the most interesting. It's worth noting that these are basically software engineering problems having to do with how to maintain long-running systems, not issues with the physical architecture proposed here. Is an RDBMS the best way to solve those software engineering problems? It's hard to believe. Are these problems worth solving for other domains? You betcha. I'd love to be able to upgrade my applications without restarting them. (Thanks to Debian, I can mostly upgrade my operating system without restarting it -- something users of e.g. Windows may have difficulty imagining.)
Relational algebra and persistence are both supposed to be simple, posted 24 Dec 2001 at 03:36 UTC by tk »
Data persistence is definitely not a new idea. In fact, if I remember correctly, persistent storage (ferrite cores) actually predate volatile storage. I guess it somehow faded away, only to emerge recently under the guise of persistent OSs such as EROS, persistent architectures such as Prevayler which we now discuss, and so on.
It's hard to see how relational algebra or persistence compare with each other. After all, relational algebra was supposed to be simple anyway -- data are nothing more than just lots of mathematical relations, right? We now know however that this `simple' idea is fraught with practical problems.
Will the same happen for persistence? Maybe, or maybe not. As jrobbins mentioned, changing the `shape' of objects is a problem, and there are probably many other problems.
XML Serialization, posted 24 Dec 2001 at 11:54 UTC by CryoBob »
I might be taking a bit of a simplistic view on the subject but couldn't alot of the issues raised by jrobbins relating to testing and having data in a useful format if a system is retired; be addressed by XML serialization. If we are going to be able to serialize all the commands and business objects why not have an option or feature to dump this information to a XML file. Then when tracking states you could do a dump at each command and compare the XML output to see where things are going wrong.
XML serialization also has the advantage of being self describing rather than in a group of tables in binary format on a database server. I mean what happens if your RDBMS company goes bust and you can't get at the data because of a licence timeout for example...
Obviously XML serialization will implement another overhead to the system, but if implemented correctly you could serialize in binary format to boost performance, and then you should you need to restore the state for investigative/testing/export purposes load the objects through an Object to XML parsing engine and look at the output.
Processing speed doesn't increase consistency, posted 25 Dec 2001 at 15:01 UTC by baueran »
Yes, you are right: in RAM a desktop machine may be able to process your 100.000 records in less than a second or something (I don't think that's representative for anything though), but I do not think that makes the system necessarily more consistent or bullet-proof. What happens if you (or any of your client applications) run into a deadlock within a millisecond? How consistent will the rest of the system and data be without an ACID paradigm to rely on? Correct me, if I'm just not getting the point, but I believe such an issue is not addressed in this approach.
trademarks?, posted 25 Dec 2001 at 18:45 UTC by dalke »
Minor point, but '"PREVAYLER" and "OPEN-SOURCE PREVALENCE LAYER" are trademarks of Klaus Wuestefeld'? I'm curious on trademarking a few things of my own, so I checked the USPTO. Neither marks are listed. Given the email address of ".br", are they only trademarked in Brazil?
I'm Back, posted 26 Dec 2001 at 18:59 UTC by KlausWuestefeld »
I agree that the Prevayler implementation, as it is today, is robust, fast and scalable enough for most applications.
In the company where I work there are 7 people working on two projects using Prevayler to be released in January. I am also glad to help any other early Prevayler adopters.
I would like to share some thoughts, though, on the use of prevalence "in the large" to make sure that we are not missing out on some very interesting possibilities.
First, I will give a few very quick, specific and UNJUSTIFIED answers, and then, in a separate comment, I will give a more complete explanation in an attempt to clarify all concerns so far...
Re: Interesting but..., posted 26 Dec 2001 at 19:10 UTC by KlausWuestefeld »
There are still quite a few things we really need transactions for: -- ncm
I apologize. Prevayler does have transactions.
Although a prevalent system can define transactions (commands) and provide them for a client to use, there is NO TRANSACTION SCHEME the client can use to arbitrarily define new TYPES of transactions (new atomic sets of business operations) whenever it fancies. The last thing we need is another transaction scheme allowing clients to bring business logic into their own hands.
I realize the article is confusing in this respect. I have corrected the "oficial" version of the article to make this clear.
When you make the first of a series of changes to objects in the database, you typically break one or more database invariants until you get the last change entered. Other processes looking at the database had better either wait, or had better see the state it had before you started.
Yes. In the prevalence scheme, the other processes shall wait.
To get much concurrency, you need to snapshot the state before the first change.
Hmmm. What if the waiting time for each transaction is only a few microseconds? (I shall explain...)
If you get halfway through a series of changes and crash, the system had better come back up without the changes you made, because you're not going to be equipped to continue where you left off.
Yes. The article already covers this well, though. Are there any doubts?
If you get halfway through a series of changes and discover some condition that keeps you from finishing, you had better be able to just drop the changes and pick up with the original snapshot.
"You" (the system server, I presume) will never be halfway through a series of changes and discover some condition that keeps "you" from finishing. (I shall explain...)
If N processes make a series of conflicting changes concurrently, (N-1) of them had better be told that their changes have failed, and that they must try again.
There are no concurrent changes in a prevalent scheme. All changes are sequenced.
There's a reason that databases are written by career professionals.
Yes. Databases are way too complex. ;)
A simple object database can be really useful, but that doesn't make it a substitute for the real thing. That's part of why so many "object database" companies failed some ten years back.
Prevalence is a persistence scheme, and, like OODBMSs, Prevayler will guarantee a logically crash-free object space for your business objects. Prevayler is not an object database manager, as I see it, though. It does not provide any sort of language for data storage or retrieval (ODBMSs normally provide some OQLish thing). Database managers are also worried, among other things, about how they will store chunks of data from RAM to disk and how they will retrieve those chunks later. When you have enough RAM for all your system data, you need no longer worry about that.
When you have enough RAM (the prevalence hypothesis) and a crash-free object space, many database career professionals' assumptions no longer hold.
Interesting but ... one has to free one's mind. New possibilities are waiting.
Re: MySQL Comparison, posted 26 Dec 2001 at 19:14 UTC by KlausWuestefeld »
(e.g. if you're happy with the robustness and performance of MySQL, you'll probably be happy with this, too)
Of course you will be happy! Prevayler is much more robust* and much faster** than MySQL. ;)
* Robustness, as I understand it, is related to failure. The less failures something presents, the more robust it is - as simple as that. Prevayler's robustness is bounded by the robustness of the VM and its serialization algorithm. Prevayler is so simple (564 lines including comments, javadoc and blank lines) you could probably write a formal proof for it. ** I have tried both but please don't take my word. Try them out too.
"Since Prevayler is also simpler to use, what is the advantage of MySQL?" Some people like SQL and the relational model. MySQL is a relational database manager with an SQL interface. Prevayler is not.
Re: Java Flushing to Disk, posted 26 Dec 2001 at 19:17 UTC by KlausWuestefeld »
Klaus, as a matter of interest, how did you manage to get Java to force flushing to disk?
FileOutputStream.getFD().sync()
Re: Interesting but problematic, posted 26 Dec 2001 at 19:24 UTC by KlausWuestefeld »
Thank you (Klaus Wuestefeld) for your nice write-up.
You are welcome.
Let me discuss/ask some further points: distributed systems? if a systen-prevalence-deployed application contacts other services resp. other servers you have a synchronization problem. How do you handle that? I guess, that you end up doing a 2PC- like synchronization between your prevalence servers.
I didn't understand the question very well.
Fine grained tx-model versus all or nothing? with big BOs - systems there are actually lots of small transactions. the system prevalence paradigma doesn't give you a fine grained application side control, or does it?
No it doesn't. I believe that to be inefficient and unnecessary. Maybe we could discuss an example where you think it might be necessary.
Note though that you can adapt (extended) 2PC-transactions to efficiently work RAM-based while retaining persistent storage properties (by using RAM-based subtransactions and files or RDBMS in the root transaction).
Yes. I know. Three years ago, I wrote an object-relational persistence layer for Java that had nested transactions in RAM and an optional* RDBMS in the root transaction.
* You could run everything in RAM if you wanted. That was good for presentations, developing without database configuration hassle and running test scripts very fast.
Scalabity? having all "commands queued and routed through a single place" doesn't scale very well. We should better consider one of these big 64 processor multigigabyte machines using a gigabit card: you wouldn't want all requests to be serialized through a single bottleneck which involves IO.
Make sure you let the people using ORACLE (and its redo log files) know about that. ;)
With fine grained distributed transactions you don't need this "single place" or even a single server.
Sounds interesting. Could you elaborate and give an example?
I appreciate the "do it in background" approach, though, as an advance to requiring requests to be queued while saving the state. It's quite neccesary for 24/7 systems.
Was it clear to you, from the article, that your prevalent system DOES NOT have to stop in order to save its state?
Re: testing, debugging, integration, and data migration, posted 26 Dec 2001 at 19:31 UTC by KlausWuestefeld »
I used to be a professional SmallTalk programmer, ...
Me too, for 5 years. :)
The simplicy of "just saving the system state" is a double-edged sword. The downside is that it is often hard to specify a particular system state that you might want to use for testing or debugging. If you ever get an object into a "bad state", it can be very hard to find out how it got into that state.
In the prevalent scheme, with some daily system snapshots, you can retrieve the system's state before it "got bad"; and with the command logs you can actually replay your commands one-by-one until you get to the rotten one. Of course, I am supposing you have a decent "object encapsulation breaker" FOR DEBUGGING PURPOSES ONLY.
I know there aren't many of those around (compared to SQL-based tools) but that is more of a cultural problem, I believe. As you say, people are used to rows and columns. They like to break their systems' encapsulation with SQL tools and, at the same time, they like to complain: "Where are all the benefits object orientation has promised us?". ;)
What can you do? I expect things like Prevayler to gradually break this vicious circle.
Lastly, upgrades were always a pain in image-based tools. Very incremental changes (like adding an instance variable to a class) can be handled by the serialization system. Any reoganization beyond that would require custom coding. In contrast, you can do small and mid- sized reorganizations a lot easier in SQL.
Me and my team would always do our migrations in Smalltalk (I wrote an object-relational persistence layer for Smalltalk 6 years ago). We would only use SQL or PL as a last resort and for performance reasons. With all your objects in RAM, that is a different story... ;)
Re: Why bother with disk at all?, posted 26 Dec 2001 at 19:33 UTC by KlausWuestefeld »
You can certainly go for RAM all the way and have several replicas, if you can afford it.
I could not agree more with egnor.
Just a comment on the "Copy-on-write VM tricks" to "soften the need to entirely freeze a replica during checkpointing.": It is a bit complicated dealing with executing threads because your memory might never be in a consistent state at any given moment in time. The orthogonal persistence guys (like the guys mentioned in the article) have not figured how to solve this problem.
With prevalence, the problem simply doesn't exist.
Re: XML Serialization, posted 26 Dec 2001 at 19:34 UTC by KlausWuestefeld »
There is a colleague of mine fiddling with several XML-serialization libraries because he wants to include that in Prevayler.
Re: Processing speed doesn't increase consistency, posted 26 Dec 2001 at 19:37 UTC by KlausWuestefeld »
The point about speed is that, if every transaction is extremely fast, you do not have to handle concurrent transactions. That makes life MUCH easier. I am not only talking about sheer RAM processing speed increase, mind you. I am talking about a design change. I shall explain it in one of the following comments.
The ACID properties do remain.
Re: Trademarks, posted 26 Dec 2001 at 19:38 UTC by KlausWuestefeld »
"PREVAYLER" and "OPEN-SOURCE PREVALENCE LAYER" are trademarks of Klaus Wuestefeld in the same way that "Linux" is a trademark of Linus Torvalds.
They are not REGISTERED trademarks though. Much like a copyright, you do not have to register it to be entitled to a trademark.
Of course, the suits will always tell you that it is better to register.
Serialization Throughput Test, posted 26 Dec 2001 at 19:47 UTC by KlausWuestefeld »
How fast does serialization run on your machine?
import java.io.*;
public class SerializationThroughput {
static public void main(String[] args) {
try {
FileOutputStream fos = new FileOutputStream(new File ("tmp.tmp"));
ObjectOutputStream oos = new ObjectOutputStream(fos);
Thread.sleep(5000); //Wait for any disk activity to stop.
long t0 = System.currentTimeMillis();
int max = 10000;
int i = 0;
while (i++ < max) {
oos.writeObject(new Integer(i));
oos.reset();
oos.flush();
fos.getFD().sync(); //Forces flushing to disk. :)
}
System.out.println("This machine can serialize " + max * 1000 / (System.currentTimeMillis() - t0) + " Integers per second.");
} catch (Exception e) {
e.printStackTrace();
}
}
}
My 450MHz K6II running windows98 with a 3 year old IDE hard drive gives me the following result: "This machine can serialize 576 Integers per second."
Does anyone give me more? :)
PREVALENCE IN THE LARGE, posted 26 Dec 2001 at 20:11 UTC by KlausWuestefeld »
OK, here we go:
I shall leave automatic load-balancing aside for now and concentrate on the concerns we already have.
Atomicity and Crash-Recovery
This is already covered in the article.
Consistency and Error-Conditions
Every command is executed on its own. The business system must either check for inconsistencies before it starts executing any command or be able to undo whatever changes were done if it runs into an inconsistency. In my designs I prefer the first approach. The demo application included with Prevayler has good examples.
Isolation
While a client is preparing a command to be executed, no other client can see what that command is all about.
Durability
The snapshots and command logs guarantee your persistence. If you use replicas, as described in the article, your system shall not only persist, it shall prevail.
Scalability and Performance
Suppose we have a multi-threaded system in which all threads do all of the three following things:
1) Client stuff - Waiting for an HTTP request; Waiting for an RMI request; Reading a file; Preparing a command to be executed; Writing a file; Generating HTML; Painting a GUI screen; etc...
2) Prevayler stuff - Logging a command to a file. (This is the only thing Prevayler does on the hot system during execution. The snapshot is taken by the replica and has no impact here.)
3) Business stuff - Processing a command; Evaluating a query.
For simplicity, Prevayler's implementation, today, will synchronize "Logging a command" and "Processing a command" in a single go. That is not necessary though. The only conditions we have to meet are:
- All commands are logged.
- All commands are executed after they are logged.
- All commands are executed in the same order as they are logged.
Using two producer-consumer queues would already alleviate that a little. The main problems, though, are still:
- It might take a long time to serialize certain large commands and Prevayler doesn't serialize and log more than one command at a time.
- The business system cannot process more than one command at a time.
The first problem is easy to solve. 4096 (or more) "slave" log files could be used to serialize and log up to 4096 (or more) SIMULTANEOUS COMMANDS. There must only be a "master" log file indicating in which "slave" log file the last command was serialized (it is not even necessary that the first command that started being logged be the first one to finish). In terms of scalability and throughput, this is as much as you can get even in an RDBMS like ORACLE because of its redo log files.
Take a look at the "Serialization Throughput Test" above, to see how well your machine would do as a "master logger". :)
All these performance enhancements are already scheduled for future Prevayler releases. If anyone is considering using Prevayler on a project for a system that actually needs them already, I will be glad to implement them sooner (or integrate someone else's implementation) and help out on the project design.
All other thread activities, including query evaluation, mind you, can already be processed in parallel. So, you can have as many processors as your VM, OS and hardware will support.
On to the second problem: "The business system cannot process more than one command at a time.".
To overcome that, then, we will establish a simple rule: "The business system cannot take more than a few MICROSECONDS to run any single command."
"Oh no! I knew it! This guy is crazy!", some might think, "How can I possibly process 100000 payment records in only a few microseconds?".
For 99% of your commands, like changing a person's name, you check for inconsistencies (invalid name, duplicate name, etc), and then you just execute it normally. With your objects in RAM, that will only take a few microseconds anyway.
For 1% of your commands (the hairy ones), like processing a batch payment with 100000 payments, lazy evaluation is the key: your system simply doesn't process the command. Instead, it just keeps the command in the "batch payments" list for future evaluation.
The command will be processed bit-by-bit whenever a query is evaluated regarding that command. It is important to note that, while the client is building the command, the command is internally preparing its structure to be kept in the system without further processing. Remeber: a prevalent command is much more than an atomic set of operations. It is a full-fledged object and can be responsible for much of the system's business intelligence! The batch payment command, for example, would keep all payment records internally in a HashMap with contract id as the key.
Suppose you then query the payment status of any given contract. The contract will see "When was the last time I updated my payment status?". It will then look at the "batch payments" list (there are two or three batch payments a month): "Were there any batch payments since my last update?". If there were, the contract updates itself accordingly (one HashMap lookup per batch). Then, the contract simply returns its payment status. This all takes only a few microsecond too.
You could have a query, though, that actually depends on the processing of ALL the payments (e.g. "Total Monthly Revenue"). In this case, the query AND ONLY THIS QUERY will take about 2 seconds* to execute. All the rest of the system continues working at full speed and with full availability.
*Today, my company has an ORACLE based billing system running on big solaris boxes that takes 62.5 machine hours to process 100000 payment records. We estimate that doing it all in RAM would take no more than 2 seconds (on my desktop machine, mind you).
Are there any more doubts or are all your systems already prevalent? ;)
Re: Trademarks, posted 26 Dec 2001 at 20:14 UTC by dalke »
They are not REGISTERED trademarks though. Much like a copyright, you do not have to register it to be entitled to a trademark.
Ahh, thank you. The USPTO link for that is: http://www.uspto.gov/web/offices/tac/tmfaq.htm#Basic001.
Do I need to register my trademark? No..
Also, What are the benefits of federal trademark registration?
Constructive notice nationwide of the trademark owner's claim.
Evidence of ownership of the trademark.
Jurisdiction of federal courts may be invoked.
Registration can be used as a basis for obtaining registration in foreign countries.
Registration may be filed with U.S. Customs Service to prevent importation of infringing foreign goods.
"PREVAYLER" and "OPEN-SOURCE PREVALENCE LAYER" are trademarks of Klaus Wuestefeld in the same way that "Linux" is a trademark of Linus Torvalds.
Umm, except that Linus owns the registered trademark on Linux, serial number 74560867 at uspto.gov. There was a big hoorah about this some five years ago when someone other than Linus registered the term for himself. Some of the links about the topic are mentioned at http://www.linux10.org/history/ .
Of course, the suits will always tell you that it is better to register.
Most "suits" would say that if you have the $325/10 years and don't want to go through the hassle of defending your mark if your work becomes popular, then it's worth it.
Thoughts about Prevayler, posted 27 Dec 2001 at 02:39 UTC by Gandhi »
I'm using prevayler at a beta system I'm developing, and I think the main problem when you expose this kind of system is that you don't have studies saying it's right or not.
Of course a lot of people thought about this before Klaus, but anyone really made a serious study about what are the more commom actions (procedures) perfomed for each category of application?.
What is the best application category for prevayler?.
Anybody knows what is the REAL consystency of the systems at the market?.
Don't you think inconsystency at 99% of the cases are just result of bad code at the top layer? Can't we just make a fault-tolerant system and keep the system working, no matter how bad coder is the guy?.
New java implementation (1.3 and 1.4) has news classes that allows high speed messaging pipes between applications. Can you imagine a better use to these pipes?.
I agree that XML serialization is a good thing, mainly for debugging purposes and it's atomicity, but how can you compress it? And if you compress, why keep it as XML?.
I think that just a better serialization scheme should do the trick, with compression, cryptography, and a hierarquical system that could allow easily XML translation. Externalize methods do the job. Any volunteer?.
One easy question. Is it a framework? Is there a planned plugin structure? Everything will be done through interfaces? No register classes or similar approaches?.
[]s, gandhi.
Prevayler Plug-ins, posted 29 Dec 2001 at 04:06 UTC by KlausWuestefeld »
One easy question. Is it a framework?
Not at present.
Is there a planned plugin structure?
No. Can there be a plugin structure in the future? Yes.
There is no design trait in Prevayler based on predictions for the future. Prevayler's design, at any point in time, will be the simplest design that we can achieve and that satisfies all CURRENT requirements. The goal is anticlimactic simplicity.
Don't worry. Thanks to simplicity, the day you write the first plug-in for Prevayler, we will easily find a way to "plug it in". The day you write your third Prevayler plug-in, there will certainly be a "plug-in structure" in place.
That is the beauty of open-source and that is the beauty of simple design.
To Be Continued..., posted 29 Dec 2001 at 04:16 UTC by KlausWuestefeld »
Anyone interested in knowing more about prevalence or in further discussing the subject (but not necessarily having Advogato certification) take a look at the Prevayler Forum.
See you there, Klaus.
orthogonal persistense, posted 29 Dec 2001 at 15:44 UTC by jerry »
Askemos has a simillar take on persistense. Just not "all in memory" but "allways saved to file" - after each transaction in any of your objects.
Serialization Throughput for Larger Objects, posted 30 Dec 2001 at 18:08 UTC by Ward »
I generalized the throughput test to write records of various size. For small records the time is dominated by the flush; for large ones, transfer time. I found the knee of this classic curve to be at about 300 Integers (3k bytes) on a Windows platform and 100 Integers on a Linux. All but one machine I tested showed other behaviour that I cannot explain. I've written a short note with graphs and the revised test source code.
Fine, what about Garbage Collection?, posted 3 Jan 2002 at 00:10 UTC by jonabbey »
I designed and implemented a RAM-based, transactional database in Java years ago for Ganymede, and I can attest that keeping everything in memory works splendidly. Add a transaction log for recovery, and you're cooking with gas.
At least, that is, for reasonably small datasets. The big open question for Ganymede, and for any memory-resident Java database systems, is how big a cost does Garbage Collection become when you scale up? Using the operating system's native VM subsystem to handle disk paging works fine, but when the Garbage Collector has to sweep through everything periodically in order to clean up garbage, that sweep has presumably to do a good bit of paging to take care of things.
Do you have any insight into how serious a problem this is? Ganymede works fantastically well for us at the scale we need it to, but I've always imagined (but not tested) that putting a gigabyte of directory data into it would probably not work so terribly well.
Re: Garbage Collection (Raising the Bar), posted 3 Jan 2002 at 01:34 UTC by KlausWuestefeld »
I ran a few tests creating huge arrays of Integers and serializing them to stress the limits of some VMs. Everytime we increased the size of the array to a point where the system started paging, we simply had to abort the test after a few hours because we couldn't stand waiting any longer. 55 million was the max we reached without paging, running on an HP-UX machine (Thanks to the guys at HP/PortoAlegre/Brazil).
The prevalence hypothesis, though, is that you have enough RAM for all your data so, even when the garbage collector kicks in, your system shouldn't have to page to disk.
Even if you have enough RAM, the garbage collector can be a nuisance in many large systems and a real show-stopper for time-sensitive critical systems. I am not an expert but it seems that most VMs use a mix of generational garbage collection and traditional mark-and-sweep. I really would like to see some three-colouring going on anytime soon (if you know of anything about this please post here).
A very popular VM's heap size won't even reach 1GB. (It will allow you to set the parameter but will shamelessly ignore it if it is above a certain limit). It seems that VMs like that one are targeted only at feeble client code.
I believe that projects using Prevayler will actually raise the bar for VM robustness, heap size and garbage collection performance.
New Advogato Features
FOAF updates: Trust rankings are now exported, making the data available to other users and websites. An external FOAF URI has been added, allowing users to link to an additional FOAF file.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!
OBJECT PREVALENCE
Posted 23 Dec 2001 at 03:46 UTC by KlausWuestefeld
Transparent Persistence, Fault-Tolerance and Load-Balancing for Java Systems.
Orders of magnitude FASTER and SIMPLER than a traditional DBMS. No pre or post-processing required, no weird proprietary VM required, no base-class inheritance or clumsy interface definition required: just PLAIN JAVA CODE.
How is this possible?
Question: RAM is getting cheaper every day. Researchers are announcing major breakthroughs in memory technology. Even today, servers with multi-gigabyte RAM are commonplace. For many systems, it's already feasible to keep all business objects in RAM. Why can't I simply do that and forget all the database hassle?
Answer: You can, actually.
Are you crazy? What if there's a system crash?
To avoid losing data, every night your system server saves a snapshot of all business objects to a file using plain object serialization.
What about the changes occurred since the last snapshot was taken? Won't the system lose those in a crash?
No.
How come?
All commands received from the system's clients are converted into serializable objects by the server. Before being applied to the business objects, each command is serialized and written to a log file. During crash recovery, first, the system retrieves its last saved state from the snapshot file. Then, it reads the commands from the log files created since the snapshot was taken. These commands are simply applied to the business objects exactly as if they had just come from the system's clients. The system is then back in the state it was just before the crash and is ready to run.
Does that mean my business objects have to be deterministic?
Yes. They must always produce the same state given the same commands.
Doesn't the system have to stop or enter read-only mode in order to produce a consistent snapshot?
No. That is a fundamental problem with transparent or orthogonal persistence projects like PJama (http://www.dcs.gla.ac.uk/pjava/) but it can be solved simply by having all system commands queued and routed through a single place. This enables the system to have a replica of the business logic on another virtual machine. All commands applied to the "hot" system are also read by the replica and applied in the exact same order. At backup time, the replica stops reading the commands and its snapshot is safely taken. After that, the replica continues reading the command queue and gets back in sync with the "hot" system.
Doesn't that replica give me fault-tolerance as a bonus?
Yes it does. I have mentioned one but you can have several replicas. If the "hot" system crashes, any other replica can be elected and take over. Of course, you must be able to afford a machine for every replica you want.
Does this whole scheme have a name?
Yes. It is called system prevalence. It encompasses transparent persistence, fault-tolerance and load-balancing.
If all my objects stay in RAM, will I be able to use SQL-based tools to query my objects' attributes?
No. You will be able to use object-based tools. The good news is you will no longer be breaking your objects' encapsulation.
What about transactions? Don't I need transactions?
No. The prevalence design gives you all transactional properties without the need for explicit transaction semantics in your code.
How is that?
DBMSs tend to support only a few basic operations: INSERT, UPDATE and DELETE, for example. Because of this limitation, you must use transaction semantics (begin - commit) to delimit the operations in every business transaction for the benefit of your DBMS. In the prevalent design, every transaction is represented as a serializable object which is atomically written to the queue (a simple log file) and processed by the system. An object, or object graph, is enough to encapsulate the complexity of any business transaction.
What about business rules involving dates and time? Won't all those replicas get out of sync?
No. If you ask the use-case gurus, they will tell you: "The clock is an external actor to the system.". This means that clock ticks are commands to the business objects and are sequentially applied to all replicas, just like all other commands.
Is object prevalence faster than using a database?
The objects are always in RAM, already in their native form. No disk access or data marshalling is required. No persistence hooks placed by preprocessors or postprocessors are required in your code. No "isDirty" flag. No restrictions. You can use whatever algorithms and data-structures your language can support. Things don't get much faster than that.
Besides being deterministic and serializable, what are the coding standards or restrictions my business classes have to obey?
None whatsoever. To issue commands to your business objects, though, each command must be represented as a serializable object. Typically, you will have one class for each use-case in your system.
How scalable is object prevalence?
The persistence processes run completely in parallel with the business logic. While one command is being processed by the system, the next one is already being written to the log. Multiple log files can be used to increase throughput. The periodic writing of the snapshot file by the replica does not disturb the "hot" system in the slightest. Of course, tests must be carried out to determine the actual scalability of any given implementation but, in most cases, overall system scalability is bound by the scalability of the business classes themselves.
Can't I use all those replicas to speed things up?
All replicas have to process all commands issued to the system. There is no great performance gain, therefore, in adding replicas to command-intensive systems. In query-intensive systems such as most Web applications, on the other hand, every new replica will boost the system because queries are transparently balanced between all available replicas. To enable that, though, just like your commands, each query to your business logic must also be represented as a serializable object.
Isn't representing every system query as a serializable object a real pain?
That's only necessary if you want transparent load-balancing, mind you. Besides, the queries for most distributed applications arrive in a serializable form anyway. Take Web applications for example: aren't HTTP request strings serializable already?
Does prevalence only work in Java?
No. You can use any language for which you are able to find or build a serialization mechanism. In languages where you can directly access the system's memory and if the business objects are held in a specific memory segment, you can also write that segment out to the snapshot file instead of using serialization.
Is there a Java implementation I can use?
Yes. You will find Prevayler - The Open-Source Prevalence Layer, an example application and more information at http://www.prevayler.org. It does not yet implement automatic load-balancing but it does implement transparent business object persistence and replication is in the oven.
Is Prevayler reliable?
Prevayler's robustness comes from its simplicity. It is orders of magnitude simpler than the simplest RDBMS. Although I wouldn't use Prevayler to control a nuclear plant just yet, its open-source license ensures the whole of the software developing community the ability to scrutinize, optimize and extend Prevayler. The real questions you should bear in mind are: "How robust is my Java Virtual Machine?" and "How robust is my own code?". Remember: you will no longer be writing feeble client code. You will now have the means to actually write server code. It's the way object orientation was intended all along; but it's certainly not for wimps.
You said Prevayler is open-source software. Do you mean it's free?
That's right. It's licensed under the Lesser General Public License.
But what if I'm emotionally attached to my database?
For many applications, prevalence is a much faster, much cheaper and much simpler way of preserving your objects for future generations. Of course, there will be all sorts of excuses to hang on to "ye olde database", but at least now there is an option.
---------------------------------------------------------------------
ABOUT THE AUTHOR
KlausWuestefeld enjoys writing good software and helping other people do the same. He has been doing so for 17 years now. He can be contacted at klaus@objective.com.br.
---------------------------------------------------------------------
"PREVAYLER" and "OPEN-SOURCE PREVALENCE LAYER" are trademarks of Klaus Wuestefeld.
Copyright (C) 2001 Klaus Wuestefeld.
Unmodified, verbatim copies of this text including this copyright notice can be freely made.
Interesting but..., posted 23 Dec 2001 at 07:08 UTC by ncm »
There are still quite a few things we really need transactions for:
When you make the first of a series of changes to objects in the database, you typically break one or more database invariants until you get the last change entered. Other processes looking at the database had better either wait, or had better see the state it had before you started. To get much concurrency, you need to snapshot the state before the first change.
If you get halfway through a series of changes and crash, the system had better come back up without the changes you made, because you're not going to be equipped to continue where you left off.
If you get halfway through a series of changes and discover some condition that keeps you from finishing, you had better be able to just drop the changes and pick up with the original snapshot.
If N processes make a series of conflicting changes concurrently, (N-1) of them had better be told that their changes have failed, and that they must try again.
There's a reason that databases are written by career professionals. A simple object database can be really useful, but that doesn't make it a substitute for the real thing. That's part of why so many "object database" companies failed some ten years back.
Transactions, posted 23 Dec 2001 at 09:03 UTC by Pseudonym »
Actually, transactions are not so important in the external interface of an OODBMS. In an RDBMS, a manipulation typically involves several SQL statements (e.g. insert, update, remove) each of which can act on only one table at a time. So if a transaction needs to manipulate more than one table, you need to ensure that the set of statements is atomic by issuing a transaction.
In an OODBMS, where manipulation methods can operate on more than one class, the need is reduced somewhat. Internally, you can just queue up the command logs until the method is complete, then write them out together. Then the problem becomes entirely one of synchronisation. It's not quite ACID, but it'll do for most business applications.
You (ncm) are right, however, in that this solution, while no doubt excellent for many purposes (e.g. if you're happy with the robustness and performance of MySQL, you'll probably be happy with this, too), won't scale to many critical applications. For example, it would be quite hard to handle replication in any sane manner.
Klaus, as a matter of interest, how did you manage to get Java to force flushing to disk?
Scalability ?, posted 23 Dec 2001 at 12:03 UTC by jneves »
Is it just me, or prevayler isn't useful in anything else but a uniprocessor machine as is? You process requests one at a time, which means that there can't be 2 different requests being processed at the same time in different processors. And when you have several replicas you have to have some coordination between all replicas to insure the order of the requests. Or am I missing something here ?
Interesting but problematic, posted 23 Dec 2001 at 12:59 UTC by dannu »
Thank you (Klaus Wuestefeld) for your nice write-up. I mostly agree to the points made by the other repliers.
Let me discuss/ask some further points:
distributed systems? if a systen-prevalence-deployed application contacts other services resp. other servers you have a synchronization problem. How do you handle that? I guess, that you end up doing a 2PC-like synchronization between your prevalence servers.
fine grained tx-model versus all or nothing? with big BOs -systems there are actually lots of small transactions. the system prevalence paradigma doesn't give you a fine grained application side control, or does it? Note though that you can adapt (extended) 2PC-transactions to efficiently work RAM-based while retaining persistent storage properties (by using RAM-based subtransactions and files or RDBMS in the root transaction).
scalabity? having all "commands queued and routed through a single place" doesn't scale very well. consider one of these big 64 processor multigigabyte machines using a gigabit card: you wouldn't want all requests to be serialized through a single bottleneck which involves IO. With fine grained distributed transactions you don't need this "single place" or even a single server. I appreciate the "do it in background" approach, though, as an advance to requiring requests to be queued while saving the state. It's quite neccesary for 24/7 systems.
In my oppinion the complexity of 2PC-systems comes from shortcomings of the commercial products (BEA WLE, Websphere, Oracle etc.). They impose big clumsy quite old fashioned development schemes where the developer is restricted and has to keep track of many conditions. This partly stems from the pain with underspecified and often incorrectly implemented XA-interfaces. (e.g. writing multithreaded programs with XA-adapters from the main RDBMs is a desaster).
I think that system prevalence would help implementing web applications which are located on single systems. It is a simple enough paradigm to be used and understood by companies which often fail or are very slow with 2pc-transaction systems. Handling of error conditions (pointed out by ncm) might still be a big problem.
just my 2 (soon to be) eurocent and best wishes!
holger
I'll be back..., posted 23 Dec 2001 at 14:08 UTC by KlausWuestefeld »
THANKS A LOT for the FEEDBACK!
This is the first forum outside of my working group to actually get the idea and give me some positive feedback.
I am just leaving on a trip right now (my wife is calling me ;) for Christmas and will be back on wednesday. Then, I will address all concerns: ACID properties, error-condition recovery, scalability, the works...
Just a note on scalability and concurrency to think about over Christmas: Suppose we have a subscriber management system that receives a file from a bank with 100000 (one-hundred-thousand) payment records. A prevalent server running on a regular desktop machine can handle a command/transaction for this in less than a millisecond and be ready for the next command.
Merry Christmas! See you soon.
testing, debugging, integration, and data migration, posted 23 Dec 2001 at 19:33 UTC by jrobbins »
I used to be a professional SmallTalk programmer, I also was a professional Lisp programmer. Both of those languages use the concept of a saved memory image as part of their normal development environment.
The simplicy of "just saving the system state" is a double-edged sword. The downside is that it is often hard to specify a particular system state that you might want to use for testing or debugging. If you ever get an object into a "bad state", it can be very hard to find out how it got into that state. In contrast, the impedence mismatch between OO systems and RDBMSs provides a natural boundary and conceptual bottleneck for testing and debugging. It is realtively easy to compare 2 database dumps to see what is different, or to populate the database with test data, or to see which INSERT statement introduced a particular row into the database. You could have test data consisting of a long set of commands, but that "algebraic" approach to testing does not scale well, and allows defects in mutators to mask defects in accessors.
One thing that I learned while trying to actually sell ST-80 systems to other divisions in a large company is that IS organizations see a standard RDBMS as an integration point. If your system uses an RDMBS, they can plan capacity on a shared database machine: they can generate ad-hoc reports, they can use standard tools for disk backups and such on the database machine only. Also, in the event that your system eventaully dies (is no longer maintained, or the license is not extended or whatever) they will at least have the data in a format that they can get out of your system's tables and into some other system.
Lastly, upgrades were always a pain in image-based tools. Very incremental changes (like adding an instance variable to a class) can be handled by the serialization system. Any reoganization beyond that would require custom coding. In contrast, you can do small and mid-sized reorganizations a lot easier in SQL.
Why bother with disk at all?, posted 23 Dec 2001 at 20:23 UTC by egnor »
I'll take the opposite tack for variety:
If you're going this far, why bother with a disk at all? Just attach a battery to your RAM. If you want reliability, keep replicas. If a replica is lost, "clone" another one by freezing its message queue and copying the frozen image; the two clones can then "catch up" with the queued messages in parallel.
Copy-on-write VM tricks may soften the need to entirely freeze a replica during checkpointing.
I suspect the points raised in most of the comments can be fixed. (After all, suppose we were looking the other way. Compared to modern programming languages, databases and middleware systems have lots of horrible misfeatures, starting with bad syntax and ending with fundamentally broken models of (non-)encapsulation and (non-)reuse and (non-)genericity; the complaints in the other direction seem relatively trivial by comparison. How can any self-respecting software engineer stand to use today's RDBMS systems without feeling dirty all over?)
jrobbins's notes are the most interesting. It's worth noting that these are basically software engineering problems having to do with how to maintain long-running systems, not issues with the physical architecture proposed here. Is an RDBMS the best way to solve those software engineering problems? It's hard to believe. Are these problems worth solving for other domains? You betcha. I'd love to be able to upgrade my applications without restarting them. (Thanks to Debian, I can mostly upgrade my operating system without restarting it -- something users of e.g. Windows may have difficulty imagining.)
Relational algebra and persistence are both supposed to be simple, posted 24 Dec 2001 at 03:36 UTC by tk »
Data persistence is definitely not a new idea. In fact, if I remember correctly, persistent storage (ferrite cores) actually predate volatile storage. I guess it somehow faded away, only to emerge recently under the guise of persistent OSs such as EROS, persistent architectures such as Prevayler which we now discuss, and so on.
It's hard to see how relational algebra or persistence compare with each other. After all, relational algebra was supposed to be simple anyway -- data are nothing more than just lots of mathematical relations, right? We now know however that this `simple' idea is fraught with practical problems.
Will the same happen for persistence? Maybe, or maybe not. As jrobbins mentioned, changing the `shape' of objects is a problem, and there are probably many other problems.
XML Serialization, posted 24 Dec 2001 at 11:54 UTC by CryoBob »
I might be taking a bit of a simplistic view on the subject but couldn't alot of the issues raised by jrobbins relating to testing and having data in a useful format if a system is retired; be addressed by XML serialization. If we are going to be able to serialize all the commands and business objects why not have an option or feature to dump this information to a XML file. Then when tracking states you could do a dump at each command and compare the XML output to see where things are going wrong.
XML serialization also has the advantage of being self describing rather than in a group of tables in binary format on a database server. I mean what happens if your RDBMS company goes bust and you can't get at the data because of a licence timeout for example...
Obviously XML serialization will implement another overhead to the system, but if implemented correctly you could serialize in binary format to boost performance, and then you should you need to restore the state for investigative/testing/export purposes load the objects through an Object to XML parsing engine and look at the output.
Processing speed doesn't increase consistency, posted 25 Dec 2001 at 15:01 UTC by baueran »
Yes, you are right: in RAM a desktop machine may be able to process your 100.000 records in less than a second or something (I don't think that's representative for anything though), but I do not think that makes the system necessarily more consistent or bullet-proof. What happens if you (or any of your client applications) run into a deadlock within a millisecond? How consistent will the rest of the system and data be without an ACID paradigm to rely on? Correct me, if I'm just not getting the point, but I believe such an issue is not addressed in this approach.
trademarks?, posted 25 Dec 2001 at 18:45 UTC by dalke »
Minor point, but '"PREVAYLER" and "OPEN-SOURCE PREVALENCE LAYER" are trademarks of Klaus Wuestefeld'? I'm curious on trademarking a few things of my own, so I checked the USPTO. Neither marks are listed. Given the email address of ".br", are they only trademarked in Brazil?
I'm Back, posted 26 Dec 2001 at 18:59 UTC by KlausWuestefeld »
I agree that the Prevayler implementation, as it is today, is robust, fast and scalable enough for most applications.
In the company where I work there are 7 people working on two projects using Prevayler to be released in January. I am also glad to help any other early Prevayler adopters.
I would like to share some thoughts, though, on the use of prevalence "in the large" to make sure that we are not missing out on some very interesting possibilities.
First, I will give a few very quick, specific and UNJUSTIFIED answers, and then, in a separate comment, I will give a more complete explanation in an attempt to clarify all concerns so far...
Re: Interesting but..., posted 26 Dec 2001 at 19:10 UTC by KlausWuestefeld »
There are still quite a few things we really need transactions for: -- ncm
I apologize. Prevayler does have transactions.
Although a prevalent system can define transactions (commands) and provide them for a client to use, there is NO TRANSACTION SCHEME the client can use to arbitrarily define new TYPES of transactions (new atomic sets of business operations) whenever it fancies. The last thing we need is another transaction scheme allowing clients to bring business logic into their own hands.
I realize the article is confusing in this respect. I have corrected the "oficial" version of the article to make this clear.
When you make the first of a series of changes to objects in the database, you typically break one or more database invariants until you get the last change entered. Other processes looking at the database had better either wait, or had better see the state it had before you started.
Yes. In the prevalence scheme, the other processes shall wait.
To get much concurrency, you need to snapshot the state before the first change.
Hmmm. What if the waiting time for each transaction is only a few microseconds? (I shall explain...)
If you get halfway through a series of changes and crash, the system had better come back up without the changes you made, because you're not going to be equipped to continue where you left off.
Yes. The article already covers this well, though. Are there any doubts?
If you get halfway through a series of changes and discover some condition that keeps you from finishing, you had better be able to just drop the changes and pick up with the original snapshot.
"You" (the system server, I presume) will never be halfway through a series of changes and discover some condition that keeps "you" from finishing. (I shall explain...)
If N processes make a series of conflicting changes concurrently, (N-1) of them had better be told that their changes have failed, and that they must try again.
There are no concurrent changes in a prevalent scheme. All changes are sequenced.
There's a reason that databases are written by career professionals.
Yes. Databases are way too complex. ;)
A simple object database can be really useful, but that doesn't make it a substitute for the real thing. That's part of why so many "object database" companies failed some ten years back.
Prevalence is a persistence scheme, and, like OODBMSs, Prevayler will guarantee a logically crash-free object space for your business objects. Prevayler is not an object database manager, as I see it, though. It does not provide any sort of language for data storage or retrieval (ODBMSs normally provide some OQLish thing). Database managers are also worried, among other things, about how they will store chunks of data from RAM to disk and how they will retrieve those chunks later. When you have enough RAM for all your system data, you need no longer worry about that.
When you have enough RAM (the prevalence hypothesis) and a crash-free object space, many database career professionals' assumptions no longer hold.
Interesting but ... one has to free one's mind. New possibilities are waiting.
Re: MySQL Comparison, posted 26 Dec 2001 at 19:14 UTC by KlausWuestefeld »
(e.g. if you're happy with the robustness and performance of MySQL, you'll probably be happy with this, too)
Of course you will be happy! Prevayler is much more robust* and much faster** than MySQL. ;)
* Robustness, as I understand it, is related to failure. The less failures something presents, the more robust it is - as simple as that. Prevayler's robustness is bounded by the robustness of the VM and its serialization algorithm. Prevayler is so simple (564 lines including comments, javadoc and blank lines) you could probably write a formal proof for it. ** I have tried both but please don't take my word. Try them out too.
"Since Prevayler is also simpler to use, what is the advantage of MySQL?" Some people like SQL and the relational model. MySQL is a relational database manager with an SQL interface. Prevayler is not.
Re: Java Flushing to Disk, posted 26 Dec 2001 at 19:17 UTC by KlausWuestefeld »
Klaus, as a matter of interest, how did you manage to get Java to force flushing to disk?
FileOutputStream.getFD().sync()
Re: Interesting but problematic, posted 26 Dec 2001 at 19:24 UTC by KlausWuestefeld »
Thank you (Klaus Wuestefeld) for your nice write-up.
You are welcome.
Let me discuss/ask some further points: distributed systems? if a systen-prevalence-deployed application contacts other services resp. other servers you have a synchronization problem. How do you handle that? I guess, that you end up doing a 2PC- like synchronization between your prevalence servers.
I didn't understand the question very well.
Fine grained tx-model versus all or nothing? with big BOs - systems there are actually lots of small transactions. the system prevalence paradigma doesn't give you a fine grained application side control, or does it?
No it doesn't. I believe that to be inefficient and unnecessary. Maybe we could discuss an example where you think it might be necessary.
Note though that you can adapt (extended) 2PC-transactions to efficiently work RAM-based while retaining persistent storage properties (by using RAM-based subtransactions and files or RDBMS in the root transaction).
Yes. I know. Three years ago, I wrote an object-relational persistence layer for Java that had nested transactions in RAM and an optional* RDBMS in the root transaction.
* You could run everything in RAM if you wanted. That was good for presentations, developing without database configuration hassle and running test scripts very fast.
Scalabity? having all "commands queued and routed through a single place" doesn't scale very well. We should better consider one of these big 64 processor multigigabyte machines using a gigabit card: you wouldn't want all requests to be serialized through a single bottleneck which involves IO.
Make sure you let the people using ORACLE (and its redo log files) know about that. ;)
With fine grained distributed transactions you don't need this "single place" or even a single server.
Sounds interesting. Could you elaborate and give an example?
I appreciate the "do it in background" approach, though, as an advance to requiring requests to be queued while saving the state. It's quite neccesary for 24/7 systems.
Was it clear to you, from the article, that your prevalent system DOES NOT have to stop in order to save its state?
Re: testing, debugging, integration, and data migration, posted 26 Dec 2001 at 19:31 UTC by KlausWuestefeld »
I used to be a professional SmallTalk programmer, ...
Me too, for 5 years. :)
The simplicy of "just saving the system state" is a double-edged sword. The downside is that it is often hard to specify a particular system state that you might want to use for testing or debugging. If you ever get an object into a "bad state", it can be very hard to find out how it got into that state.
In the prevalent scheme, with some daily system snapshots, you can retrieve the system's state before it "got bad"; and with the command logs you can actually replay your commands one-by-one until you get to the rotten one. Of course, I am supposing you have a decent "object encapsulation breaker" FOR DEBUGGING PURPOSES ONLY.
I know there aren't many of those around (compared to SQL-based tools) but that is more of a cultural problem, I believe. As you say, people are used to rows and columns. They like to break their systems' encapsulation with SQL tools and, at the same time, they like to complain: "Where are all the benefits object orientation has promised us?". ;)
What can you do? I expect things like Prevayler to gradually break this vicious circle.
Lastly, upgrades were always a pain in image-based tools. Very incremental changes (like adding an instance variable to a class) can be handled by the serialization system. Any reoganization beyond that would require custom coding. In contrast, you can do small and mid- sized reorganizations a lot easier in SQL.
Me and my team would always do our migrations in Smalltalk (I wrote an object-relational persistence layer for Smalltalk 6 years ago). We would only use SQL or PL as a last resort and for performance reasons. With all your objects in RAM, that is a different story... ;)
Re: Why bother with disk at all?, posted 26 Dec 2001 at 19:33 UTC by KlausWuestefeld »
You can certainly go for RAM all the way and have several replicas, if you can afford it.
I could not agree more with egnor.
Just a comment on the "Copy-on-write VM tricks" to "soften the need to entirely freeze a replica during checkpointing.": It is a bit complicated dealing with executing threads because your memory might never be in a consistent state at any given moment in time. The orthogonal persistence guys (like the guys mentioned in the article) have not figured how to solve this problem.
With prevalence, the problem simply doesn't exist.
Re: XML Serialization, posted 26 Dec 2001 at 19:34 UTC by KlausWuestefeld »
There is a colleague of mine fiddling with several XML-serialization libraries because he wants to include that in Prevayler.
Re: Processing speed doesn't increase consistency, posted 26 Dec 2001 at 19:37 UTC by KlausWuestefeld »
The point about speed is that, if every transaction is extremely fast, you do not have to handle concurrent transactions. That makes life MUCH easier. I am not only talking about sheer RAM processing speed increase, mind you. I am talking about a design change. I shall explain it in one of the following comments.
The ACID properties do remain.
Re: Trademarks, posted 26 Dec 2001 at 19:38 UTC by KlausWuestefeld »
"PREVAYLER" and "OPEN-SOURCE PREVALENCE LAYER" are trademarks of Klaus Wuestefeld in the same way that "Linux" is a trademark of Linus Torvalds.
They are not REGISTERED trademarks though. Much like a copyright, you do not have to register it to be entitled to a trademark.
Of course, the suits will always tell you that it is better to register.
Serialization Throughput Test, posted 26 Dec 2001 at 19:47 UTC by KlausWuestefeld »
How fast does serialization run on your machine?
import java.io.*;
public class SerializationThroughput {
static public void main(String[] args) {
try {
FileOutputStream fos = new FileOutputStream(new File ("tmp.tmp"));
ObjectOutputStream oos = new ObjectOutputStream(fos);
Thread.sleep(5000); //Wait for any disk activity to stop.
long t0 = System.currentTimeMillis();
int max = 10000;
int i = 0;
while (i++ < max) {
oos.writeObject(new Integer(i));
oos.reset();
oos.flush();
fos.getFD().sync(); //Forces flushing to disk. :)
}
System.out.println("This machine can serialize " + max * 1000 / (System.currentTimeMillis() - t0) + " Integers per second.");
} catch (Exception e) {
e.printStackTrace();
}
}
}
My 450MHz K6II running windows98 with a 3 year old IDE hard drive gives me the following result: "This machine can serialize 576 Integers per second."
Does anyone give me more? :)
PREVALENCE IN THE LARGE, posted 26 Dec 2001 at 20:11 UTC by KlausWuestefeld »
OK, here we go:
I shall leave automatic load-balancing aside for now and concentrate on the concerns we already have.
Atomicity and Crash-Recovery
This is already covered in the article.
Consistency and Error-Conditions
Every command is executed on its own. The business system must either check for inconsistencies before it starts executing any command or be able to undo whatever changes were done if it runs into an inconsistency. In my designs I prefer the first approach. The demo application included with Prevayler has good examples.
Isolation
While a client is preparing a command to be executed, no other client can see what that command is all about.
Durability
The snapshots and command logs guarantee your persistence. If you use replicas, as described in the article, your system shall not only persist, it shall prevail.
Scalability and Performance
Suppose we have a multi-threaded system in which all threads do all of the three following things:
1) Client stuff - Waiting for an HTTP request; Waiting for an RMI request; Reading a file; Preparing a command to be executed; Writing a file; Generating HTML; Painting a GUI screen; etc...
2) Prevayler stuff - Logging a command to a file. (This is the only thing Prevayler does on the hot system during execution. The snapshot is taken by the replica and has no impact here.)
3) Business stuff - Processing a command; Evaluating a query.
For simplicity, Prevayler's implementation, today, will synchronize "Logging a command" and "Processing a command" in a single go. That is not necessary though. The only conditions we have to meet are:
- All commands are logged.
- All commands are executed after they are logged.
- All commands are executed in the same order as they are logged.
Using two producer-consumer queues would already alleviate that a little. The main problems, though, are still:
- It might take a long time to serialize certain large commands and Prevayler doesn't serialize and log more than one command at a time.
- The business system cannot process more than one command at a time.
The first problem is easy to solve. 4096 (or more) "slave" log files could be used to serialize and log up to 4096 (or more) SIMULTANEOUS COMMANDS. There must only be a "master" log file indicating in which "slave" log file the last command was serialized (it is not even necessary that the first command that started being logged be the first one to finish). In terms of scalability and throughput, this is as much as you can get even in an RDBMS like ORACLE because of its redo log files.
Take a look at the "Serialization Throughput Test" above, to see how well your machine would do as a "master logger". :)
All these performance enhancements are already scheduled for future Prevayler releases. If anyone is considering using Prevayler on a project for a system that actually needs them already, I will be glad to implement them sooner (or integrate someone else's implementation) and help out on the project design.
All other thread activities, including query evaluation, mind you, can already be processed in parallel. So, you can have as many processors as your VM, OS and hardware will support.
On to the second problem: "The business system cannot process more than one command at a time.".
To overcome that, then, we will establish a simple rule: "The business system cannot take more than a few MICROSECONDS to run any single command."
"Oh no! I knew it! This guy is crazy!", some might think, "How can I possibly process 100000 payment records in only a few microseconds?".
For 99% of your commands, like changing a person's name, you check for inconsistencies (invalid name, duplicate name, etc), and then you just execute it normally. With your objects in RAM, that will only take a few microseconds anyway.
For 1% of your commands (the hairy ones), like processing a batch payment with 100000 payments, lazy evaluation is the key: your system simply doesn't process the command. Instead, it just keeps the command in the "batch payments" list for future evaluation.
The command will be processed bit-by-bit whenever a query is evaluated regarding that command. It is important to note that, while the client is building the command, the command is internally preparing its structure to be kept in the system without further processing. Remeber: a prevalent command is much more than an atomic set of operations. It is a full-fledged object and can be responsible for much of the system's business intelligence! The batch payment command, for example, would keep all payment records internally in a HashMap with contract id as the key.
Suppose you then query the payment status of any given contract. The contract will see "When was the last time I updated my payment status?". It will then look at the "batch payments" list (there are two or three batch payments a month): "Were there any batch payments since my last update?". If there were, the contract updates itself accordingly (one HashMap lookup per batch). Then, the contract simply returns its payment status. This all takes only a few microsecond too.
You could have a query, though, that actually depends on the processing of ALL the payments (e.g. "Total Monthly Revenue"). In this case, the query AND ONLY THIS QUERY will take about 2 seconds* to execute. All the rest of the system continues working at full speed and with full availability.
*Today, my company has an ORACLE based billing system running on big solaris boxes that takes 62.5 machine hours to process 100000 payment records. We estimate that doing it all in RAM would take no more than 2 seconds (on my desktop machine, mind you).
Are there any more doubts or are all your systems already prevalent? ;)
Re: Trademarks, posted 26 Dec 2001 at 20:14 UTC by dalke »
They are not REGISTERED trademarks though. Much like a copyright, you do not have to register it to be entitled to a trademark.
Ahh, thank you. The USPTO link for that is: http://www.uspto.gov/web/offices/tac/tmfaq.htm#Basic001.
Do I need to register my trademark? No..
Also, What are the benefits of federal trademark registration?
Constructive notice nationwide of the trademark owner's claim.
Evidence of ownership of the trademark.
Jurisdiction of federal courts may be invoked.
Registration can be used as a basis for obtaining registration in foreign countries.
Registration may be filed with U.S. Customs Service to prevent importation of infringing foreign goods.
"PREVAYLER" and "OPEN-SOURCE PREVALENCE LAYER" are trademarks of Klaus Wuestefeld in the same way that "Linux" is a trademark of Linus Torvalds.
Umm, except that Linus owns the registered trademark on Linux, serial number 74560867 at uspto.gov. There was a big hoorah about this some five years ago when someone other than Linus registered the term for himself. Some of the links about the topic are mentioned at http://www.linux10.org/history/ .
Of course, the suits will always tell you that it is better to register.
Most "suits" would say that if you have the $325/10 years and don't want to go through the hassle of defending your mark if your work becomes popular, then it's worth it.
Thoughts about Prevayler, posted 27 Dec 2001 at 02:39 UTC by Gandhi »
I'm using prevayler at a beta system I'm developing, and I think the main problem when you expose this kind of system is that you don't have studies saying it's right or not.
Of course a lot of people thought about this before Klaus, but anyone really made a serious study about what are the more commom actions (procedures) perfomed for each category of application?.
What is the best application category for prevayler?.
Anybody knows what is the REAL consystency of the systems at the market?.
Don't you think inconsystency at 99% of the cases are just result of bad code at the top layer? Can't we just make a fault-tolerant system and keep the system working, no matter how bad coder is the guy?.
New java implementation (1.3 and 1.4) has news classes that allows high speed messaging pipes between applications. Can you imagine a better use to these pipes?.
I agree that XML serialization is a good thing, mainly for debugging purposes and it's atomicity, but how can you compress it? And if you compress, why keep it as XML?.
I think that just a better serialization scheme should do the trick, with compression, cryptography, and a hierarquical system that could allow easily XML translation. Externalize methods do the job. Any volunteer?.
One easy question. Is it a framework? Is there a planned plugin structure? Everything will be done through interfaces? No register classes or similar approaches?.
[]s, gandhi.
Prevayler Plug-ins, posted 29 Dec 2001 at 04:06 UTC by KlausWuestefeld »
One easy question. Is it a framework?
Not at present.
Is there a planned plugin structure?
No. Can there be a plugin structure in the future? Yes.
There is no design trait in Prevayler based on predictions for the future. Prevayler's design, at any point in time, will be the simplest design that we can achieve and that satisfies all CURRENT requirements. The goal is anticlimactic simplicity.
Don't worry. Thanks to simplicity, the day you write the first plug-in for Prevayler, we will easily find a way to "plug it in". The day you write your third Prevayler plug-in, there will certainly be a "plug-in structure" in place.
That is the beauty of open-source and that is the beauty of simple design.
To Be Continued..., posted 29 Dec 2001 at 04:16 UTC by KlausWuestefeld »
Anyone interested in knowing more about prevalence or in further discussing the subject (but not necessarily having Advogato certification) take a look at the Prevayler Forum.
See you there, Klaus.
orthogonal persistense, posted 29 Dec 2001 at 15:44 UTC by jerry »
Askemos has a simillar take on persistense. Just not "all in memory" but "allways saved to file" - after each transaction in any of your objects.
Serialization Throughput for Larger Objects, posted 30 Dec 2001 at 18:08 UTC by Ward »
I generalized the throughput test to write records of various size. For small records the time is dominated by the flush; for large ones, transfer time. I found the knee of this classic curve to be at about 300 Integers (3k bytes) on a Windows platform and 100 Integers on a Linux. All but one machine I tested showed other behaviour that I cannot explain. I've written a short note with graphs and the revised test source code.
Fine, what about Garbage Collection?, posted 3 Jan 2002 at 00:10 UTC by jonabbey »
I designed and implemented a RAM-based, transactional database in Java years ago for Ganymede, and I can attest that keeping everything in memory works splendidly. Add a transaction log for recovery, and you're cooking with gas.
At least, that is, for reasonably small datasets. The big open question for Ganymede, and for any memory-resident Java database systems, is how big a cost does Garbage Collection become when you scale up? Using the operating system's native VM subsystem to handle disk paging works fine, but when the Garbage Collector has to sweep through everything periodically in order to clean up garbage, that sweep has presumably to do a good bit of paging to take care of things.
Do you have any insight into how serious a problem this is? Ganymede works fantastically well for us at the scale we need it to, but I've always imagined (but not tested) that putting a gigabyte of directory data into it would probably not work so terribly well.
Re: Garbage Collection (Raising the Bar), posted 3 Jan 2002 at 01:34 UTC by KlausWuestefeld »
I ran a few tests creating huge arrays of Integers and serializing them to stress the limits of some VMs. Everytime we increased the size of the array to a point where the system started paging, we simply had to abort the test after a few hours because we couldn't stand waiting any longer. 55 million was the max we reached without paging, running on an HP-UX machine (Thanks to the guys at HP/PortoAlegre/Brazil).
The prevalence hypothesis, though, is that you have enough RAM for all your data so, even when the garbage collector kicks in, your system shouldn't have to page to disk.
Even if you have enough RAM, the garbage collector can be a nuisance in many large systems and a real show-stopper for time-sensitive critical systems. I am not an expert but it seems that most VMs use a mix of generational garbage collection and traditional mark-and-sweep. I really would like to see some three-colouring going on anytime soon (if you know of anything about this please post here).
A very popular VM's heap size won't even reach 1GB. (It will allow you to set the parameter but will shamelessly ignore it if it is above a certain limit). It seems that VMs like that one are targeted only at feeble client code.
I believe that projects using Prevayler will actually raise the bar for VM robustness, heap size and garbage collection performance.
New Advogato Features
FOAF updates: Trust rankings are now exported, making the data available to other users and websites. An external FOAF URI has been added, allowing users to link to an additional FOAF file.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!
Subscribe to:
Posts (Atom)
