Breaking Windows

POPFile 0.21.0 has been released, and with very little trouble I have it up and running in Mac OS 10.3. The big change in this version is no more BerklyDB back end, POPFile now using a SQL database, and the easiest way to do this in OS X is to use SQLite. Thankfully Michael Tsai (SpamSieve author) has a built version of SQLite for Mac OS X. This means you do not have to compile and install it. The schema is available to use something like MySQL, but why bother? Next up you have some perl modules to install:

• DBI
• DBD::SQLite
• Digest::Base
• Digest::MD5
• HTML::Tagset

These you will need to download, compile, and install. DBI needs to be installed before you can install the DBD::SQLite module. More on installing SQLite and these associated perl modules below. There are a couple really cool new features to POPFile, and in my opinion, worth the work to use them. They are:

• The global options for Subject Modification, X-Text-Classification insertion and X-POPFile-Link insertion have been removed and replaced with individual options on a per bucket basis to give greater choice in configuring POPFile.
• The 'unclassified' bucket is now visible in the UI so that you can see how many messages were unclassified, and configure header modification. This also means that unclassified messages are counted in the accuracy statistics; previously they were not counted which could have skewed the accuracy statistics if there were unclassified messages. -- This was something I had asked for, and I assume others did as well.
• The history "page" bar has been simplified so that it uses a fixed amount of screen space, while making navigation easy. Filters and searches on the history are now persistent, for example you can click on the Buckets page and return to the History page without losing your filter or search settings.
• The Buckets page has been modified to only show the 'distinct word' count per bucket and to show the total number of distinct words in the database. Previously we showed two counts with confusing titles: now we show the true number of words in the database, not the "word counts" (which was the number of times each word occurred).
• We've recently seen spam start to use CSS to obscure messages and fool filters like POPFile; in response, this version of POPFile does analysis of CSS in HTML encoded messages. POPFile now correctly uses the SpamAssassin headers to make POPFile more efficient when used in conjunction with SpamAssassin. We now also look at TLDs (Top Level Domains) and store them as pseudowords (most useful for TLDs like .biz). -- My favorite new feature.

A couple warnings from the developer:

• It's possible that you might see a drop in accuracy as your corpus gets trained up on the new anti-spam features. This drop in accuracy will be corrected once you've retrained POPFile a little.
• I'm suggesting that if you have a corpus with GREATER THAN 30,000 unique words (you can figure this out from the Buckets page) please wait for v0.21.1.

I have written about POPFile on Mac OS X in the past, and if you are interested in POPFile, you should read my previous articles:

• POPFile Installed on OS X
• POPFile Update in OS X

Upgrading POPFile This assumes you have POPFile 0.20.0 or 0.20.1 installed in Mac OS X (10.3 recommended). If you have not installed POPFile in the past, please read the two above links. For example I am not discussing creating the StartupItem for POPFile here. You can find that information in the above links. Just a reminder: you need the Developer Tools installed to compile the perl modules. A) Download SQLite, DBI, DBD::SQLite, Digest::Base, Digest::MD5, and HTML::Tagset from the links provided above. Mount the SQLite disk image. B) Open Terminal and type this: sudo cp /Volumes/SQLite-2.8.13/sqlite /sbin/ (type admin password) C) Next you will be installing all the perl modules. These are all installed in the same manner. Remember that if you drag a folder from the finder to Terminal, it will write the path for you. So just type cd [space] and then drag the name of the folder (eg. DBD-SQLite-0.31) into the Terminal. Installing the module goes like this: In the Terminal, cd to the unpacked perl module's folder perl Makefile.PL [return] make [return] make test [return] make install [return] Now you need to install the new version of POPFile. 1) Shutdown POPFile, go to http://localhost:8080/ (or the port number you changed POPFile to) and click “Shutdown POPFile” 2) Backup your current POPFile install. This is at /Library/POPFile 3) Keep the folder /Library/POPFile/corpus handy, you will need to move this to the new version of POPFile. Also keep “popfile.cfg” which holds your settings. 4) Download the new version of POPFile, in this case, popfile-0.21.0.zip 5) Replace the /Library/POPFile folder with the newly downloaded version of POPFile. 6) Be sure to move the corpus folder over to /Library/POPFile and the popfile.cfg file, or you will have no buckets, magnets, or anything for that matter! 7) Open the Terminal and type/paste these two commands: cd /Library/StartupItems/popfile sudo ./popfile start (type admin password) 8) Depending on how big your corpus is, and how many buckets you have, it might take a minute or two to completely startup POPFile. You do not get any feedback on this in the Terminal. At first I thought I did something wrong, but just waited a couple minutes and I could finally load the POPFile web page. Now go to http://localhost:8080/ and you should see the default POPFile History page. 9) I suggest you setup your bucket preferences such as this: Subject Header Modification OFF, X-Text-Classification Header ON, X-POPFile-Link Header ON, Quarantine Message, OFF. Being able to turn these on and off on a bucket by bucket basis is a nice feature, but right now I am not using these individual preferences. The developer, John, even has a page devoted to gathering funds for a 12" PowerBook G4. He can test POPFile on Windows and Linux, but not on Mac OS X. So help him out and donate! I talked to John sometime late last year about an easier install for OS X (there is a very nice installer for Windows) but he has not had time to get to me on the subject. It sure would be nice to have the binaries for OS X all setup in a package. So why do I still use POPFile over SpamSieve? Well, I am not afraid of the command line, and I have used POPFile for a long time (I used it on my PC for sometime before installing it on my Mac). Also, not until recently did SpamSieve support the features I need for Apple Mail. SpamSieve might be easier to install then POPFile, but where is the fun in that? I have all my clients buy SpamSieve and use it. It is easy for me to install and support.