SmiteCRC - Spam detection system

Overview - What is SmiteCRC
Use of SmiteCRC output (For Users)
Configuration Options
Using SmiteCRC with SurgeMail
Download SmiteCRC

Overview - What is SmiteCRC

SmiteCRC is to spam what a virus scanner is to viruses, it scans each incoming message and marks it as 'spam' with special headers. As the administrator you can then decide to reject the messages that are 'definitely' spam or more likely just let your users know that these special headers exist, they will then use their email clients filtering rules to store messages that have the 'spam' header in a seperate folder. Which they can quickly scan once a day for 'real' email messages, and then delete. In this way users who want spam still get it, but the tedious sorting of spam and non spam is made enormously quicker.

How does it work? In brief, it uses two systems, a set of rules that correctly identify most spam messages by content, and a database of fuzzy CRC's that let it correctly find messages that other people have reported as spam even if they are slightly different on your system. Together the two systems give a very accurate indication if a message is spam.

Following is a technical description of how it works, but if you just want to implement it jump to here:

The SmiteCRC client has four primary functions -

• Perform calculations on incoming messages (These values can be used by SmiteCRC to find duplicates).
• Run a rule filter over the e-mail to look for elements common to spam, and assign a spam weighting.
• Download regular updates from the server for both messages reported as spam, and filters.
• Check messages at POP time to establish if they have been reported as spam after delivery.

Use of SmiteCRC output (For Users)

SmiteCRC adds two headers to messages that are of use for content filtering, along with a CRC header describing the contents of the message.

    X-Smite-CRC: A$3eHhzQ$EjWgRQ$1VD816z#1q4s9KO$63ZKjj#f12aeF$1lUn0u1#e5Eqn2$r3TqJ7
    X-Smite-Match: 1
    X-SpamDetect: ***: 3.581000 Suspect source,Asks you to click below,A word in all caps repeated on the line

X-Smite-CRC is added to the message when it is accepted for delivered, and is a summary of the contents of the message. If the recipient of the message considers the message to be spam, and submits it to smitespam.com, this header will be used to find matching messages.

X-Smite-Match indicates a match with a sufficiently high weighting has been found for the message in the SmiteCRC database. This header may be applied during delivery, or at POP time. Messages with this header should be considered spam, and this header can be used for filtering in e-mail clients. If the recipient believes this is not spam, the X-Smite-CRC header should be submitted to smitespam.com as non-spam as some legitimate subscription e-mail may be submitted as spam by users.

X-SpamDetect gives the results of the spam filter. A score above 5 would most likely indicate the message is spam. Users should not filter on the presence of this header but on the contents. To simplify filtering, searching for a score of 4 or more can simply be done by searching for "****" in the X-SpamDetect header.

Client	How to filter using SmiteSpam headers	Submitting to SmiteCRC
Eudora 5.1	Menu - Tools | Filters | New... Type 'X-Smite-Match' into the Header field and choose "appears". For the action, select "Transfer To" and select an appropriate destination folder for spam. Add a rule for 'X-SpamDetect' in the same way, but select "contains", and along with the contents "*****"	Forward to SpamBot. Or, manually submit.
Netscape 4.x	Menu - Edit | Message Filters Click 'New' If the headers you want to filter against aren't in the drop-down list, click the Advanced button and add them to the list of custom headers. Filtering on the contents of headers is not possible, so it is recommended only X-Smite-Match is used with this client.	Forward to SpamBot Or, Manually submit (Menu - View | Source)
Outlook Express		Forward as attachment to SpamBot Or Manually submit (Ctrl+F3 for source)

Configuration options

All configuration options for SmiteCRC are placed in smitecrc.ini. This file must be located in the directory in which SmiteCRC is run. In general you will not need to modify any settings, but must specify SMITE_EMAIL and SMITE_PASS.

At a minimum, your smitecrc.ini should look like this -

smite_email user@example.com
smite_pass N1234

The details must be for an update account. A normal account with smitespam.com allows for submission of spam only. Update accounts are provided for all SurgeMail customers automatically, see details below.

Setting	Description	Default
UPDATE_INT	Update interval from smitespam.com in minutes.	60
MSG_SZ	Maximum message size to scan in bytes. Requires restart.	1048576 (1MB)
SMITE_EMAIL	E-mail address to login to smitespam.com.
SMITE_PASS	Password for smitespam.com.
MATCH_MIN	Minimum number of CRC values to match.	5
SCORE_MIN	Minimum score from matching CRC's.	8
RATIO_MIN	Minimum ratio of spam reports to non-spam reports.	0
COUNT_MIN	Minimum number of spam reports to be spam.	0
TRUST_ISSPAM	Ignore the score, trust smitespam.com when it says an item is spam.	TRUE
DEBUG	Log exrtra debugging information.	FALSE
UPDATE_START	Update database if required BEFORE accepting requests. Not recommended.	FALSE
LOG_SIZE	Log size in bytes.	1048576 (1MB)
SKIP_FILTER	Don't run the filter.	FALSE
FILTER_UPGRADE	Allow filter upgrades from smitespam.com.	TRUE
FILTER_MAX	Maximum portion of message to send to the filter in bytes.	10000

The settings SMITE_EMAIL, and SMITE_PASS are required. All other settings are optional. With the exception of changes requiring a restart of SmiteCRC, new configuration settings will be automatically applied when the file modification date changes. Changes to scoring settings are not recommended except for testing purposes or on the advice of support staff to address specific issues.

Each setting must be placed on a line by itself, containing the setting name, a single space, and the value you wish to use.

Using SmiteCRC with SurgeMail

NOTE: SmiteCRC is now included with SurgeMail, and should be activated via the 'SmiteSpam' page in the SurgeMail Web interface rather than by following these instructions.

Step-by-step instructions...
1) Download SmiteCRC.
2) Extract the SmiteCRC distribution in to your SurgeMail directory. Windows users can use their favourite unzipping application. On Unix-based systems (Including MacOSX), use the following commands -

uncompress smitecrc_*

tar xf smitecrc_*

3) Configure SmiteCRC -

Enter values for SMITE_EMAIL, and SMITE_PASS in smitecrc.ini. These specify your update account and password for SmiteSpam.com. When you register to download SurgeMail, an account for SmiteSpam.com is automatically created using your e-mail address as your login name, and your password is set to your registration code (E.g. Nnnnn).

4) (Optional) Create a 'normal' account at SmiteSpam.com for submitting spam. Use this link to set up an account. This is only required if you wish to submit spam manually (or using SpamBOT). You cannot use your update account to submit spam, nor can you use a normal account with SmiteCRC.

4) Configure SurgeMail as follows -

In SurgeMail, SmiteCRC is implemented as a virus checker. Configuration requires the following settings be added to surgemail.ini

Setting	Description	Typical Value
g_virus_filter	Virus checker path (SmiteCRC)	cmd="d:\surgemail\smitecrc.exe" type=""
g_smite_level	If Smite score is above this, throw it away	"1"= has been reported, or "2" = reported multiple times
g_smite_tag	If true then tag a line if message is in db when read	"true"
g_spam_bounce	If the SpamDetect score is above this bounce the message	"10" = Almost certainly spam

Ensure file permissions give ownership to the 'mail' user. This should be tested on Unix using the command -

   su mail
   ./smitecrc

You should see no response from SmiteCRC. If you see version information, this indicates SmiteCRC does not have permission to write the log to the current directory as the 'mail' user. Enter the command 'quit' to close SmiteCRC.

Configure SpamBot if required.

Check SmiteCRC is working in SurgeMail by looking at the status output. Near the top, you should see a line similar to the following -

    Smite found 840, late 482, rule file 4224, total 5850

In the above example, SmiteCRC has been running for a while, and the statistics indicate the number of messages that have been scanned and matched.

Download SmiteCRC

Version 1.6
FreeBSD 4+ (x86)	FreeBSD 4.0 and later
Linux (x86)	Linux (libc6)
MacOS X	MacOS X
Solaris 7+ (Sparc)	Solaris 7 and later
Win32 (x86)	Microsoft Windows (9x,ME,NT,2000,XP)