Spam Filtering and the Plague of False Positives

Jeff Schwartz recently received an inquiry from a senior executive at a major wireless telephone carrier, asking if he could provide some product information about the software his company produces.

“So I e-mailed it,” Schwartz told TechNewsWorld. But it never arrived. “I sent it two more times,” he said. Still, the message didn’t make it through.

“Finally, I printed it out and snail-mailed it to him,” said Schwartz. The culprit was a spam filter.

“Since my e-mail was about spam and contained lots of trigger words, the filter deleted it,” said Schwartz, who is president of NextGen Development, the maker of an antispam software application called GoodbyeSpam.

ISPs Aggressive

Executives and professionals are becoming increasingly negative about the issue of false positives — the unfortunate consequence of using spam filters. When a filter incorrectly identifies a noncommercial e-mail message as spam, it is treated in much the same way as if it were a vile piece of e-mail from a pornographer, a Viagra pitchman or a low-cost mortgage refinancier.

A study released recently by consultancy Return Path, called the E-Mail Blocking and Filtering Report, concluded that 17 percent of permission-based e-mail messages are not being delivered to in-boxes by top ISPs, a statistic that is up from just 2 percent in a similar study done in the fourth quarter of 2002.

“In an effort to protect subscribers from the onslaught of spam, ISPs are inadvertently tagging opt-in e-mail as spam, resulting in them being deposited in bulk or spam folders, or not being delivered at all,” Remy Taylor, a director of marketing at Return Path, told TechNewsWorld. “Some companies saw 46 percent of e-mail sent to their customers not get delivered.”

Return Path — using a delivery monitoring service — eyed the deliverability of 10,000 e-mail marketing campaigns earlier this year across 12 major ISPs representing 60 percent of the domains of large companies’ mailing lists. “Non-deliverability numbers are rising,” said Taylor. But the problem goes beyond even mass online marketing.

Business Communication

Routine business communications — e-mails sent from personalized contact lists on people’s PCs — are being filtered out by corporate blacklists and shunted aside if they come from domains that have been home to problematic messages.

According to several recent studies, up to 15 percent of routine e-mail communications simply are not delivered.

“I publish an electronic newsletter called the Apogee that helps readers build more valuable and profitable companies,” Clifford R. Kurtzman, CEO of Adastro, told TechNewsWorld. “The problem I have is with the [whitelist] spam filters that require the sender of the message to go to a special page and enter in a code before the mail will go through.

“When I have 75,000 subscribers, if even a small fraction of my subscribers start using such software, it creates a nightmare for me to have to manually deal with each recipient using the filter to get past their block and get them the newsletter they signed up for,” Kurtzman said.

Not Going To Take It

“I predict that within the next 24 months, we’ll hear about a user suing their ISP because the ISP’s spam filter censored their e-mail and deleted an important message which caused the user to lose money,” said Schwartz.

Some ISPs are overreacting to complaints from customers about false positives, according to marketing lobbyists and some analysts.

“As one ISP [owner] told me, he is not even filtering anymore,” Michael Herrick, president and CEO of Matter Form Media, a developer of antispam software, told TechNewsWorld. “If there is one false positive, his phone rings off the hook.”

Technology companies are joining the debate, developing solutions that, they say, will generate fewer false positives.

“What people resent the most is having the IT department or ISP determine what is — and what is not — spam,” said Herrick. “No one else has the right to open your regular mail. It should be no different with e-mail.”

New Technologies

The new technologies that are coming on the market — such as SpamFire for Windows — are targeted toward PC users, not network honchos. “Wherever the filtering takes place, it should be with the end user,” said Herrick.

Other new antispam technologies are gaining traction in the market. Emeryville, California-based Sendmail this summer started shipping its antispam software, Sendmail Advanced Anti-Spam Filter, which can be customized down to the user level but deployed at the Internet gateway.

Meanwhile, San Francisco-based Webwasher USA this summer debuted a new antispam filter that employs several statistical algorithms for smaller companies. The filter analyzes the sender, the message and the subject simultaneously to determine whether a piece of e-mail is spam.

Newport Beach, California-based SpamSoap recently debuted a new technology that conducts 22,000 tests — in three seconds — on each e-mail before calling it spam. The technology checks for inefficient routing. For example, a message that originates in the United States but goes through Brazil, China or Germany before arriving in a user’s in-box is a hallmark of spam.

Some observers note, however, that no spam filtering software is perfect, and that all users ultimately wind up wasting a lot of time searching through their bulk mail folders for false positives.

Credit Report?

ISPs are talking about developing some sort of technological ratings system, like a credit report, for message senders. The idea is to move away from the concept of good versus bad filtering toward establishing secure, trusted online relationships.

But some Internet users like the spam status quo — and figure that everybody else is making too big of a deal about the rising tide of e-mail.

“I look at spam as a form of entertainment,” said Dan Seidman, a leading sales coach and founder of SalesAutopsy.com.