Clay - I apologize for the length of my comment, but as I read the article the column on Sunday, my thoughts went here: If Bill Clinton supported data mining.. and if the media put that right up on the table so that everyone understood that this is a solid and necessary debate, then would not this biased and partisan spin on the issue become an actual worthwhile national discussion, where the partisanship might cease, and we accomplish something worthwhile?

Just following the much heralded trashing of John Poindexter's Total Information Awareness (TIA), by the media, former President Clinton spoke at UC Davis, CA. In a speech, in which he contradicted himself (politically) at every turn, he presented the obvious - data mining is a necessity, in order to prevent attacks. Now, every time he made a point he immediately (and often, very awkwardly) attempted to appease his base by saying, but we have to protect our civil liberties. A close read, shows why Clinton was never able to accomplish much in protecting the US. Excerpt from Bill Clinton's Nov. 17, 2002, speech (my bold, and my comments, in (  )):  

The most important thing is that the director of homeland security should be able to compel the cooperation and sharing of intelligence across all agencies. (we wish you would have done it just a tad sooner). ..

..The next thing that's important is that we have to have modern information technology. A lot of people are worried, and I'm concerned, that we would give up too many civil liberties or profile people because of their ethnic group or their religion, in the effort to secure America against terror. I can tell you one of the most important things you can do is just to make sure the government has the information on all of us that every mass mailing company in the country already has. [Applause.]

Now, you're laughing, but let me give you a very specific example. About three days after September the 11th, I got a call from my best childhood friend, we've been best friends since we were 9 years old. He works for the biggest mass mailing in America, which happens to be located in my home state of Arkansas.

He said, "Bill, you've got to help me. We've got four FBI agents here and we've already found five of these al-Quaida terrorists in our computers." I said, "Well, that's good, isn't it?" He said, "Well, sort of good, but the government has none of this." And so he said the man who is the head of the company wanted me to come in and try to get together with the government on this. And so about a week later I went down, and this is what I saw in their computers. Because the most important question is, if the government had this information - and we tripled funding for terrorism for the FBI in 1995 and '96 and nothing was done to modernize the computers. (is not Bill Clinton supporting the concept of data mining here?)

So you say, "So what if they found them after the fact, couldn't they have told beforehand?" I'll give you two examples. Mohammad Atta, the ringleader of 9/11, had 12 addresses in the computers of this company. Under Mohammad Atta, Mohammad J. Atta, Jay Atta with his middle initial spelled out. Now if a person has been in our country for about a year, and they have 12 places to live, they're either really rich or up to no good. And it shouldn't be too hard to figure out which.

Another one of the terrorists who flew a plane into one of the World Trade Center towers had - listen to this - 30 credit cards with $250,000 in debt outstanding and a consolidated payment schedule of $9,800 a month. Now if someone's been here a year and they've got 30 credit cards and a quarter million dollars in debt, they're either really rich or up to no good. And it shouldn't be too hard to figure out which.

Now, this information is already available on all of us. We're in lots of computers that say how many credit cards we have, what our debt is, whether we pay our light bill, whether we pay our water bill. All that kind of information. And so before we constrict the civil liberties of Americans further they ought to make darn sure that our government has this information and the people that know how to look at it look at it, since that's all out there anyway. (Excuse me - "before we constirct the civil liberties, we should do it? - Is that what he actually said?) That's a genie you cannot put back in the bottle, and if that information had been carefully studied and screened and acted upon, we might have been able to prevent September the 11th...

[..]

So we should have homeland security. What should our goal be? Our goal should be to stop big, bad things from happening.

Well, let's be clear about what data mining is. I create and maintain databases, and data mining is a normal function. But it has become "magic" jargon, and needs to be de-mythologized.

If you have a database filled with information, you can always look for trends. If you're selling bookbags, for instance, you look to see when the most number of bookbags are sold. That's how you know to re-stock your stores' inventory for maximum efficiency. For most items, you already have a general idea of what you're looking for. Data mining is what you do when you don't know what you're looking for. You don't query the database to discover anything specific ... you just tell the database to find trends, whatever they may be. Of course, most of the trends will confirm your expectations anyway, but data mining finds trends no matter what your expectations are. You're literally looking for trends without prejudice.  

If you research the phone records of suspected terrorists, and you look for the word "explosive," sooner or later the terrorists will learn not to use the word "explosive" very often. So, suppose they start using code. They say "happiness" instead. Using normal techiques, you may never know what to ask the database to look for. But if you use data mining, the technology will discover the code on its own, without having to look for any words specifically.

Data mining discourages you from pursuing data that fits your agenda, because data mining is deliberately researching without expectations. That's why all these "snooping" warnings are just crap -- they're trying to scare the public. Data mining is the opposite of political snooping. Of course, if you have a corrupt snooper, it won't matter ... just don't blame that corruption on the technique.