Daniel Einspanjer's journal

Performance of Rhino JS engine and Janino library in Kettle

Tue, 17 Nov 2009 06:26:21 GMT

My friend Roland Bouman made an interesting blog post regarding the performance of a bit of JavaScript for Kettle that he saw on a different blog.

Given the large amounts of data that I am shoving through Kettle every day, I tend to be extremely concerned about performance. Even a small inefficiency can lead to dramatic slowdowns. Hence, when I saw his post, I got to thinking about how I would approach the problem if it were within the realm of the large data sets I work with and hence required extreme optimization.

I didn't have a lot of spare time to dedicate to this experiment, so I opted for a screen-cast instead of a nicely formatted blog post. That said, I think there is a certain benefit in being able to see the work flow of someone who is very comfortable with Kettle.

The screen-cast is currently in Apple QuickTime format. Bleh. I need to get a new Ogg Theora transcoder because the one that I tried to use last time is not happy with me and I didn't have time to fiddle with it.

So, if you use Kettle and are interested in these things, here is the screen-cast. Be warned it is 30 minutes long and probably not extremely exciting to anyone outside of the ETL field.

Kettle string transformation optimization walk-through

If you are familiar with developing plug-ins for Kettle and you'd like to take a look at the User Defined Java Class plug-in I demonstrated at the end of the screen-cast, you can pick it up from the Pentaho SVN plugins repository. Just wear gloves because it has rough edges.
User Defined Java Class plug-in

Advice regarding using Travelex Cash Passport cards for travel

Thu, 24 Sep 2009 20:10:50 GMT

If you are a traveling to Europe and considering getting one of these debit cards to make life easier for you while there, my advice is don't! Figure out which of your credit cards charges the least amount of fees for international usage and just use it.

First, a general complaint: interacting with Travelex is a hassle. Their phone menus are a maze of twisty little passages, all alike. When you finally manage to reach a person, you'll have to give them all the exact same information you entered into the phone menu. Calling them when you are on your trip means calling a long distance number plus additional charges if you must use your cell phone or hotel phone or pay phone. The customer service agents are not rude though. I'll give them that.

Most importantly, the card does not work everywhere that MasterCard is accepted. If you give it to a merchant that it is not compatible, their credit card processing machine might decline the card but the money can still be withheld from you for seven days. If the merchant runs the card through three or four times, each attempt will withhold the funds again. A nice €60 meal can turn into a €180 disaster.

If you get into this situation, don't expect help from anyone. The merchant can't do anything because it was declined on his side. When you call Travelex (paying international long distance or international roaming fees), they will tell you that it isn't their fault and that the merchant did something wrong. Ignore the fact that you had to turn around and use a different MasterCard with the merchant and that one went through just fine. Furthermore, Travelex will tell you that your only option is to wait seven business days and see if the hold disappears. If it does, move on to the next challenge. If it doesn't, then expect the following additional hassle:

request a dispute form from Travelex
wait for it to be delivered by mail
fill it out and send them copies of all the decline receipts and a letter from the merchant stating that they did not receive the money that is disputed and that you have already paid them through other means
wait. After pulling teeth with one customer service agent, I was able to get the reassurance that it should definitely take less than a year to resolve. Likely just a few months.

Once you have the funds back on your card, it is likely you might be in the same boat as me and your trip to Europe is now over! So now, how do you get the money back? You have a few options:

Withdraw it from an ATM. You will pay:
- €1.75 ATM fee converted to USD at a worse than market rate
- $x dollars to the owner of the ATM for using a foreign ATM card
- Whatever remainder you can't take out through the ATM will be taken by Travelex after 12 months
Close the card at a Travelex branch. You will pay:
- $20 administration fee
- Worse than market conversion rate (note the fine print that the "Currency Return Guarantee" doesn't apply to the money on the card!)
- Another arbitrary fee/commission that varies from branch to branch and (I suspect) the mood of the employee
Leave the funds on the card until your next trip. You will pay:
- €2.30 Monthly inactivity fee each month after a year.
- The unavailability of the money for the duration (Travelex will happily take advantage of the money while you aren't using it though!)

Ubuntu screen-profiles customization

Tue, 08 Sep 2009 14:16:12 GMT

I recently loaded Ubuntu server 9.04 onto a new machine and encountered Ubuntu's screen-profiles.
In general, I like it. I had one problem and one customization that I wanted to share:

I use Mac OS X's Terminal.app to connect to my remote machines, and by default, it has custom mappings for F1 through F4. I have no idea what those keybindings mean, but they prevent screen-profiles's keybindings from working. It took a little fiddling to figure out how to fix them. Basically, you need to:

Open up the preferences dialog for Terminal.app
Go to the Settings pane
Click on the Keyboard tab button
Edit the action for each of the F1 through F4 keys
When editing, click the "delete one character" button twice to erase the characters currently in there (leave the \033 escape)
Type the following characters: [ 1 1 ~ 11 is F1, 12 is F2, 13 is F3, 14 is F4
The new entries should look just like the F5 through F8 actions.

Once I was able to use the F2 F3 and F4 keys, I decided that they weren't that useful to me. I prefer to use a combination of screen regions and windows. The window commands are very easy for me, but I've always found the split, focus, and remove keybindings to be uncomfortable so I figured those would be great commands to map to F2 F3 and F4. Here is how I did that:

sudo cp /usr/share/screen-profiles/keybindings/common /usr/share/screen-profiles/keybindings/regions
sudo vi /usr/share/screen-profiles/keybindings/regions
replace the first four entries with the new entries below
save and close the file
In screen, hit F9 to bring up the menu
Select the option for "Change keybinding set
Select the new "regions" entry
Hit F5 to reload your screen-profile and pick up the new keybindings.

register n "^aS^a^i^a^c^aA"                                 #     | Goes with the F2 definition
bindkey -k k2 process n                                 # F2  | Create new region and window (and name it)
bindkey -k k3 focus                                     # F3  | Next region
bindkey -k k4 remove                                    # F4  | Remove region

Shell script analytics

Thu, 30 Jul 2009 05:55:16 GMT

I just made a rather lengthy post on the Mozilla blog of data about shell script analytics. I'll try hard not to cross post stuff like this too often, but I thought I'd allow myself the spam this time around because using Bash and AWK to do things like this really is an important part of who I am personally as a geek in addition to what I do for Mozilla. :)

I've always thought my job was fun. Now I hear it is sexy too!

Fri, 29 May 2009 15:10:39 GMT

I just finished reading this lovely little post from the company dataspora titled The Three Sexy Skills of Data Geeks.

By far, my favorite quote was, "A good data munger excels at turning coffee into regular expressions and parsers". That certainly describes me to a tee. :)

I've always found each of these the three facets of working with data fascinating. One of the comments mentioned that decision making was an important missing trait. I could go either way there. I feel it is good to be able to tell a compelling story with the data that helps others to understand it, and then those people take the understanding you imparted to them and make decisions based on it.

It is incredibly hard to find a person who is skilled in just one or two of these facets. When you find the data geek who has all three, then you count yourself lucky. Expecting someone who has that caliber of devotion to data to also be capable of making decisions like a CEO is a bit unrealistic in my opinion.

Anyway, the article is a good, quick read. It also quite nicely summarizes the major passions in my professional life right now.

Interesting crowd-sourced solution sites

Wed, 27 May 2009 14:49:42 GMT

A good friend of mine runs the site bug.gd (and it's more professional pseudonym, errorhelp.com). This service provides something that is slightly missing from the typical Google search for an error to find a solution. It allows you to enter the full text of the error message or stack trace instead of just a couple of keywords, and it provides rich community feedback on solutions. You can even tip people for their solutions through tipjoy.com integration.

I recently came across two other nice sites created by a different company that provide a similar and complimentary service:
stackoverflow.com - A site dedicated to crowd-sourcing answers to programming questions
serverfault.com - A site dedicated to crowd-sourcing answers to system administration questions

I think it is very helpful to have a list of these sites that you can go to post a question and hopefully get an answer that will even be moderated by the community to help you determine the value of the answer. This is something that typically takes a lot longer if you search for a forum or mailing list site and post there. While it is less immediate than IRC, the moderation and ability to leave a question and get an answer "soon" are nice features you are less likely to see in IRC (although I've always gotten great results from #java, #sql, #mysql, and #bash).

As you can tell, I'm a big fan of crowd-sourcing. I have run a couple of contests on 99designs.com and have been incredibly pleased with the results that came out of that community of freelance graphic designers.

Check these places out and see if they can help you or if you can help them!

Ways to visualize and share data

Fri, 17 Apr 2009 22:40:59 GMT

Mozilla needs to be able to provide useful extracts of data such as download trends, etc. and allow the community to perform their own analysis on them, so I'm always keeping a lookout for useful tools to further that goal.

When Tony Wright posted the blog entry Just How Important is the Valley? Let’s Look at some Data on April 17th 2009, he was kind enough to publish the data set (it needs an attribution / license though) and the data looked interesting so I thought I'd spend a little time playing with it using some tools that I've been keeping my eye on.

First, I slurped the table into DabbleDB, a website that is very well suited to messing with this type of data (i.e. sourced from the web, might need a bit of cleanup, etc.). You can view and edit the data I imported to DabbleDB here: Acquired Startups Data

DabbleDB does a great job at allowing a user to sort, filter, group, and modify data using a simple interface, but it does not have a large array of visualizations. For that, we head over here to the IBM AlphaWorks lab's project, Many Eyes Wikified. I created a quick wiki dashboard for throwing together a few visualizations: Acquired Startups Visualizations

This was just a quick break from real work I've been doing, so I spent less than an hour on this. I only took about 20 minutes with DabbleDB: importing the data, cleaning the dollar values, then creating two new views that group the data by country or by state for visualization. Then I moved over to Many Eyes and played with a few visualizations to try to find some interesting views of the data and threw them into the dashboard and two sub pages.

Being able to quickly extract, transform, and visualize this data is the big win for DabbleDB and Many Eyes in my opinion. With both applications having open licensing of the data and collaboration as a key focus, they are tools that I hope to be able to take advantage of at Mozilla soon.

Counting unique visitors in SQL

Mon, 30 Mar 2009 16:17:52 GMT

A lot of web metrics solutions out there like NetTracker or Omniture allow you to perform analysis on the number of unique visitors over time. This is a pretty important metric to a lot of companies, and I recently needed to perform such an analysis, but it was on data stored in a SQL database rather than in one of these proprietary solution's data-stores.

Doing any sort of distinct counting on a large volume of data in SQL can be very costly, both in terms of storage of the raw data (since you can't aggregate it), and in query performance since there are relatively few optimizations that can be performed on the table or the query.

Fortunately for me, our data warehouse is stored in Vertica, and while the queries weren't blindingly fast, I was able to get the analysis done in a very reasonable time frame.

I was dealing with a week worth of traffic (about 80m requests per day), and one of the biggest challenges I had was how to determine what constituted a "unique visitor" (in this case, it is actually more of a unique ping or requestor since there isn't really a person involved).

I didn't have a cookie that I could use, so that left me with the less desirable course of using a combination of IP address and User Agent string. The problem with this is that the solution will under count one class of requests, and over count a different class. Here are the details:

1. If a request comes from a host that receives its public IP address via DHCP (e.g. a cable modem or DSL) and that service provider has their DHCP configured to force a change of IP addresses when the host renews its lease, then when the IP address changes, the requestor will be considered "new". e.g. HostX makes a request on Monday with IP 1.2.3.4 and a request Tuesday with IP 1.2.3.4. Then, on Wednesday, their IP address changes. Later Wednesday, HostX makes a request with its new IP 2.3.4.5. When we perform analysis on this week, we will see one distinct requestor on Monday and Tuesday, but a new requestor on Wednesday. In the worst case, if a new host, HostY is assigned IP 1.2.3.4 which HostX used to have and HostY also makes a request using the same OS version, we will mistakenly believe HostY on Wednesday through Saturday is the same distinct requestor as HostX from Monday and Tuesday.

2. If several hosts are on the same LAN network (e.g. an office), then the public IP address will likely be the same for each of those hosts. I use the partial user agent string to help mitigate this problem. I am pulling the OS and locale details out of the user agent string and using that in addition to the IP address to determine uniqueness. Unfortunately, there are a lot of machines running Windows XP with en-US, so this is only partially helpful. Any host with the same IP + OSversion + locale will be treated as a single distinct requestor in this analysis.

3. When I worked on this IP+UA strategy, I originally tested using the full UA (user agent) string which includes the browser version number. This might make sense for many other websites, but unfortunately, what I saw in the test cases that I used was that we would "forget" the distinctness whenver the browser is upgraded, or in some cases, even when certain plugins or extensions are installed (ones that modify the UA) [I'm glaring at you, MegaUpload!].

So, with this strategy in place, I ran the numbers and while I could see an unfortunate amount of under counting (i.e. multiple requests being counted as the same distinct requestor when they likely should have been separate), it was as good as I was going to get.

The last thing I needed to do was to write a SQL statement that added up the number of distinct requestors grouped by the number of days in the week that requests were made. Here is the SQL I wrote to do that. This was just my first stab at it, I got my answer, and it didn't take more than a few minutes, so I left it at that. I'd still be interested in hearing if anyone else has a better way. :)


SELECT (d15 + d16 + d17 + d18 + d19 + d20 + d21) AS RequestsPerWeek
, COUNT(*) AS NumDistinctRequestors
FROM (
    SELECT
      MAX(CASE WHEN d.date = '2009-03-15' THEN 1 ELSE 0 END) AS d15
    , MAX(CASE WHEN d.date = '2009-03-16' THEN 1 ELSE 0 END) AS d16
    , MAX(CASE WHEN d.date = '2009-03-17' THEN 1 ELSE 0 END) AS d17
    , MAX(CASE WHEN d.date = '2009-03-18' THEN 1 ELSE 0 END) AS d18
    , MAX(CASE WHEN d.date = '2009-03-19' THEN 1 ELSE 0 END) AS d19
    , MAX(CASE WHEN d.date = '2009-03-20' THEN 1 ELSE 0 END) AS d20
    , MAX(CASE WHEN d.date = '2009-03-21' THEN 1 ELSE 0 END) AS d21
    FROM distinct_requests a1
    JOIN dates d ON a1.utc_date_id = d.date_id
    GROUP BY a1.ip_ua_id
) x
GROUP BY (d15 + d16 + d17 + d18 + d19 + d20 + d21)
ORDER BY (d15 + d16 + d17 + d18 + d19 + d20 + d21)

TinyArro.ws URLs

Mon, 09 Mar 2009 16:17:37 GMT

A friend just released an URL shrinking service that I enjoy: tinyarro.ws (more nifty when written as ➡.ws).
It has a few great features over the current main stream shrinkers:

1. Cool/fun URLs (e.g. http://➽.ws/囨 for my website)
2. Very short URLs due to Unicode suffixes (great for Twitter!)
3. Preview by default! (no tweak to the URL to remember)
4. Option to enter your own custom suffix (TinyURL now has this, but it was too useful to not mention).
5. A Ubiquity command ›.ws/☺ (eventually to be integrated directly on the site)

Some news about the site:
TinyArro.ws: 10 new unicode domains. Defaulting previews to ON.
Ask HN: Thoughts on TinyArro.ws? Tiniest urls in the world (or your money back)

Willingness to be a little evil

Thu, 12 Feb 2009 19:33:08 GMT

I have been a supporter of Firefox and Mozilla for several years now, and while I don't write patches and fix bugs, a major part of that support is educating people about Mozilla, open source, and user empowerment whenever a conversation about technology allows for it.

I've found that people who use proprietary software and operating systems often fall into two broad categories for rationalizing that choice:
1. They are told to do so by some authority (usually their employeer, sometimes their social tech support person, and in some cases, just because they were told it was the right thing to do by an ad or magazine article).
2. They started using it for some reason (typically reason #1 above) a long time ago and are now just accustomed to it.

I'm sure all this is going to be old news to most people reading this, but I bring it up because of an interesting article I read today.

In the 1960's and early 70's, psychologist Stanley Milgram performed a series of famous experiments that tested the willingness of people to do something they would normally object to on moral grounds when they are in a strictly controlled environment and instructed to do so by an authority figure.

More recently, psychologist Jerry Burger had the opportunity to perform a series of similar experiments. This alternet article describes the story and discusses the findings. As I read the results and Dr. Burger's statements regarding the findings, I started thinking about how easy it is for the people to choose to give up their freedom to a piece of proprietary software for reasons similar to the ones described in these experiments.

In a green field, these people would normally opt for software that provided them with more freedom and in many cases, subjectively better security, but because they are instructed by an authority figure, or because they got started with it a long time ago and just slid deeper and deeper in, those preferences are not enough by themselves to prompt the person to change their behavior.

Now even this thought in and of itself would not be enough to prompt me to blog about this topic. We're still well in the territory where the people who haven't gotten lost in a Wikipedia article about toothbrush hygiene they found when they clicked my first link are saying, "um, DUH!" So here is my point:

At the end of the article, Dr. Burger focuses on an interesting finding of both experiments. When a person is instructed to do something "wrong", they are significantly less likely to do so if they are surrounded by peers who object first.

So when you talk to someone who is sighing about how much they hate product X but they don't have a choice, don't hate on them and don't deride them for not having a backbone, but just tell them and show them how you chose to stand up for your freedom and your security. An example can go a long way toward giving them the courage to listen to that little voice inside saying, "I want something better!"

Bash functions for going up to a directory

Thu, 05 Feb 2009 21:43:45 GMT

Sometimes, if I'm in a really deep directory, I don't want to cd from / nor do I want to cd ../../../..
I just want to either go up 5 directories, or maybe I want to go up to the parent directory "src" when I'm in /home/dre/src/projects/foo/bar/classes/org/apache/blah

This set of Bash functions lets me do that.
The first, up() will change your directory. The second will instead just print the desired directory name. This makes it easy for you to mv a file up higher or something.

If you pass no arguments, it just goes up one directory.
If you pass a numeric argument it will go up that number of directories.
If you pass a string argument, it will look for a parent directory with that name and go up to it.
(Note, there is a small display bug there. If you give it an invalid name, cd reports the "No such file or directory" error, which is good, but it has a bogus path. Since you can't know what path they were actually trying to go to, it should just say "No such parent directory: ${yourbogusname}". I don't have time to figure that out right now though.)

Just put these functions in your ~/.bashrc file and don't forget to source it. ( source ~/.bashrc )


function up()
{
    dir=""
    if [ -z "$1" ]; then
        dir=..
    elif [[ $1 =~ ^[0-9]+$ ]]; then
        x=0
        while [ $x -lt ${1:-1} ]; do
            dir=${dir}../
            x=$(($x+1))
        done
    else
        dir=${PWD%/$1/*}/$1
    fi
    cd "$dir";
}

function upstr()
{
    echo "$(up "$1" && pwd)";
}

All hail Ken Kovash!

Fri, 19 Dec 2008 16:18:13 GMT

It may be showing my ignorance, but I was unaware until recently of the officially recognized day for celebrating the man, the myth, and the math that is Ken Kovash. To think that all the time leading up to this point, I had just been satisfied with the joyous feeling in my heart every day I interacted with him.

Ken can be a harsh task-master some times.
"Daniel, where are my numbers from yesterday?"
"Daniel, why are the funnelcake trends low here and high there? You're data are wrong, go find it and fix it!"
But the pain is worth it when I see him take my crude raw data and masterfully sculpt it into bounteous bevies of tables, raging rivers of trend lines, triumphant towers of bar charts, overwhelming ontologies of pie graphs, and gilt-edged grids of treemaps.

One must weep to behold it.

Performance improvements at the cost of complexity

Tue, 09 Dec 2008 16:09:55 GMT

I discovered something that I feel is a bit of a bug in the Sun Java implementation.

If you pass in a string to the method InetAddress.getByName(), it does a bunch of testing to see if it is a domain name or a literal IP address.
If it is an IPv4 address, it will then use String.split() to split the four parts. String.split() uses regexes to do its work.

That means that if you are querying for hundreds or millions of addresses in a tight loop (as I've been doing), the JVM is spawning and compiling hundreds or millions of regex objects, in addition to a String array and four String objects per call.

So at first, I just worked around it by doing basic substringing instead of splitting. That gave me about 100x performance improvement. But then I realized I was still generating four string objects for every call..

So I came up with this mapping method and it runs about 1000x faster with a near constant minimal memory footprint.

I pre-calculate a multidimensional array of shorts where each element is indexed by the literal character value - 48 of the digits making up the number 0 - 255.

With that array available, at run time, I can do a simple lookup of the short value and then do the math to get the long representation of the IP address. I'm still generating a couple of references and a few intermediate int values, but the JIT optimizer can make quick work of that.

Linked is the test program I created to play with the different methods: InetAddressParse test

Don't listen to bash, it will lie to you!

Fri, 28 Nov 2008 04:16:37 GMT

Remember folks, if you mv a directory, and there is a bash shell currently in that directory, the bash prompt will not update to reflect the new name until you cd out of the directory and then back in.

I just spend way too long making changes and being frustrated because the changes weren't having any effect. I was clearing cashes and restarting applications and monitoring log files.. It wasn't until I happened to do a :pwd in vim while editing the file for the umpteenth time that I finally noticed that the file I had been editing was actually in a backup of the folder that I had just made.

Sheesh.

The best DHTML date range picker I've ever seen

Sat, 25 Oct 2008 01:37:41 GMT

Filament Group's Date Range Picker

It uses jQuery and a JavaScript date parsing library by the name of Date.js. This thing is simply amazing. Some of the reasons I think so:

The developer can configure a start and end date limits based on what is valid for the system (e.g. if you only have data going back to 1999, no sense in letting the user chose a date in 3000 BC)
The developer can configure a set of predefined ranges such as "Last week", "Month to date", "Year to date".
If the developer allows it, the user can use any combination of preconfigured ranges, a single date, an arbitrary range of dates, or they can use the back and forward arrows to roll the current date range forward or back.
It is smooth and crisp, able to be easily themed, and seems pretty extensible/tweakable.

It is still a work in progress (they just released it today), but I think it is still usable. The only downside that I've found so far is that the back and forward arrows in this very first released version can produce some unexpected ranges. They are currently strictly math based, so if you do something like select the current month and then hit the back arrow thinking it will select the previous month, you'll probably get something slightly different since most adjacent months don't have the same number of days.

I'm also pretty sure it has an off by one error in it that I suspect they'll fix shortly. If you select Sunday to Saturday of a week and then scroll backward, the next range is actually Monday to Sunday and the next Tuesday to Monday...

Ignore these nitpicks and go check it out right away if your website needs a date picker though. To get such a fantastic widget in the very first release can only mean that it is going to be the bee's knees after a little public beta testing.

Open Source Hardware

Tue, 21 Oct 2008 16:06:31 GMT

I thought that this article in Slate about Open Source Hardware was a fun read and worth sharing.
There is an interesting similarity in the way that Arduino handles open sourcing of their design but reserves the trademark to preserve brand quality to the Mozilla Firefox trademark.

If you like reading about geeks going against the status quo in their industry and trying to make the world a better place, give the article a read.

Good bye Mountain View

Sat, 11 Oct 2008 05:56:56 GMT

It has been a great two weeks out here in the office. I've gotten to see a lot of people face to face and had some useful meetings about my projects. I just kicked off another round of massive data loads to run over the weekend while I'm out of pocket. Hopefully they will run smoothly and deliver me high quality data.

There are some really exciting things coming up this quarter:

I'll be working on one of the largest data sets yet, our AMO data. We have several really cool mechanisms for visualizing individual extension projects hosted on AMO. The developer has control over whether to make the statistics public or not. As an example, you can take a look at the statistics for Adblock Plus. I'll be working on ways to be able to integrate data across projects so we can get a better understanding of the extension community that means so very much to Mozilla.
I'll hopefully be blogging a little more about the complexities of processing the large amount of data that I have to crunch through.
I'll be making several pieces of my Pentaho Data Integration (Kettle for those of you in the know) ETL scripts available in an open source repository. It will help with the blogging, they might be useful to other people doing similar things, and who knows, maybe some people will even have suggestions for improvements!
Later in the quarter, I'll be working on an exciting new project to take some of the aggregated data that Mozilla has, such as the number of downloads of Firefox for given time periods, and making it available publicly for the community to explore and visualize. At the moment, I'm leaning toward trying to use the Many-Eyes project from IBM AlphaWorks. If anyone has any better ideas, please let me know.

Been a long difficult week

Sat, 30 Aug 2008 02:14:28 GMT

I have wonderful results to show for it though.

I have gotten the second large data source flowing through our metrics system and have the first report hooked up to it.

It's going to be interesting comparing the performance of the two data sources. Both have similar volume, but this second one is in a much cleaner looking star schema as opposed to the extremely denormalized single table format. Vertica handles both of these formats well, so I'm eager to figure out how close the performance is.

Kettle (a.k.a. Pentaho Data Integration) is a real winner here as far as enabling me to develop and maintain these very complex ETL processes. The ETL for the previous data source working against the single table clips along at over 30,000 records per second. This new ETL is a good bit slower, both because of a difference in the file structure of what I'm parsing, and because I have seven dimensions that I am doing foreign key lookups in. There is lots of room for optimization in this ETL too though.

It is somewhat difficult to optimize the throughput of the transformation for a headless server or when running a clustered transformation in Kettle. Pentaho is supposed to be coming out with some new management tools that will hopefully streamline things there.

One of the interesting things I ran into was the fact that because Kettle runs each step in a separate thread and these steps are passing around rows of data as array objects, certain server class hardware can actually perform much slower than desktop class hardware.
A case in point: a very simple transformation that does nothing more than generate several million records of data and pass them through a few steps can run at more than 700,000 records per second on my MacBook Pro with a 2.5 GHz Intel Core 2 Duo processor. The exact same transformation running on a HP blade with dual quad core 2.5 GHz Intel Xeon processors and 16 GB of EEC memory tops out at about 350,000 records per second. Let me tell you, that was pretty depressing to witness! Of course, the saving grace here is that when there is a lot more work to be done than just passing pages of memory around between cores, the server can do a lot more work, faster. That is another thing that I'm hoping some R&D at Pentaho is going to help solve.

A post about personal data

Wed, 20 Aug 2008 21:45:45 GMT

Mitchell Baker, the Chairperson of Mozilla Foundation and Mozilla Corporation recently posted a series of blog entries about data:

Thinking About Data
Framework for discussing “data”
Why focus on data?
Data Relating to People

Data — getting to the point

This discussion is something I've been looking forward to seeing at Mozilla since I started back in March. In the work that I do, I make every effort to safeguard data and make sure that what I process and store can't turn around and bite me later.

One thing that I felt could use a different approach of listing out is the different forms of personal data that people are likely to generate or come across in the web world.

To me, the best way to categorize these types of personal data is with a matrix. I've created one below that has the origin of the data as the X axis and the classification of the data as the Y axis. Inside each cell, I've placed a few examples that I think represent that intersection of data.

I'd encourage anyone interested in this to comment on other origins, classifications, or examples of personal data. The more we have defined, the easier it will be to make sure that our discussions about data don't leave anything out.

I've also saved this document on docs.google.com (Personal data types matrix).
If anyone wishes to collaborate with me on enhancing it, please just let me know in the comments and I'll send you a collaboration invitation.

	Identifying		Characterizing
	Potential¹	Definite	Self	Relationships
Elicited²	Name/Address (partial) IP address	Contact information (comprehensive) SSN E-mail address³ Blog URL Credit card information	Demographics Location Interests Website filters	Friend invitations Friends list Friends watched/followed
Published	Blog posts⁴	PGP key Contact information (comprehensive)	Interests Blog posts⁵ Wishlists	Friends list
Harvested	cookies Personal search terms		Extrapolated interests clickstream in site Web history	People watched/followed

1Multiple pieces of potential identifying information are usually needed to make definite identification or direct contact

2Data may be elicited as a requirement for interaction with the data collector (e.g. IP address required to view a web page or shipping information required for a purchase) or it may be optional (e.g. a blog comment form requesting your URL).

3E-mail address is a definite identification because it immediately allows a person to contact you directly

4Blog posts talking about who you are or where you live are potentially identifying.

5Blog posts talking about topics that interest you or things you do are characterizing.

Mozilla 2008 Summit

Tue, 29 Jul 2008 16:27:21 GMT

I'm in Whistler, B.C. Canada attending the Mozilla 2008 Summit. It is a huge crowd of people. Should be lots of fun. More later.

Finally. It took work to get this apache front-end configured properly

Fri, 11 Jul 2008 20:45:56 GMT

I've spent most of the day trying to get an Apache 2.2 server set up to do both LDAP authentication and AJP proxying to a tomcat back-end.

The trickiest parts were translating changes from Apache 2.0's implementation of auth_ldap.

In 2.0, the following directory directives were needed to do group based authentication:


        AuthType Basic
        AuthName      "Use your ldap username/password"
        AuthLDAPBindDN xxx
        AuthLDAPBindPassword xxx
        AuthLDAPURL ldap://server/o=xxx?xxx
        Require group cn=xxx,ou=groups

However, in 2.2, the syntax changed slightly and the following was what it took for me to get it going:


        AuthType Basic
        AuthName      "Use your ldap username/password"
        #AuthBasicProvider defaults to file so it is required if you aren't loading mod_authn_file
        AuthBasicProvider ldap
        AuthLDAPBindDN xxx
        AuthLDAPBindPassword xxx
        AuthLDAPUrl ldap://server/o=xxx?xxx
        #It is now ldap-group instead of just group
        Require ldap-group cn=xxx, ou=groups

Before I put the AuthBasicProvider ldap directive in place, I was getting an error in the logs:
configuration error: couldn't check user. No user file?: /

It took me a longer time to figure out the ldap-group vs group problem. In the logs, I was seeing incorrect password attempts being logged properly, but if I typed the right password, there was nothing in the log but the authorization dialog was just redisplayed in the browser.
I imagine that maybe if I tweaked some log settings somewhere I'd find that it was possible to see the Require directive failing.

Also note that most auth_ldap examples show putting the directives in a <Directory> section. Well, if you are using mod_proxy or mod_proxy_ajp, there is no directory so you put the auth directives in a <Location /> section instead.

Status update

Thu, 03 Jul 2008 14:16:14 GMT

Released the first alpha of my project. And I've been pretty happy with the results so far. Gotten some good feedback and there is lots more work to be done.

The project is based on Pentaho and uses a Vertica cluster as the DB backend. I've gotten pretty amazing results out of the combination.

I've been spending a lot of time working with two community additions to Pentaho, the Community Build Framework (CBF) and the Community Dashboard Framework (CDF). These two amazing projects are being driven by Pedro Alves, a BI consultant specializing in Pentaho. They have really allowed my project to move along rapidly in the direction I wanted to take it.

The other exciting thing I hope to blog about further in the near future is the choropleth map I managed to implement in Pentaho. It was based on an example from Chris Schmidt. While writing this post, I just discovered that he lives nearby. I think I might have to take him out to lunch as a treat for the help he's given me. :)

I need to look into integrating the Simile Timeplot widget into my Pentaho dashboards. I really need the ability to provide rich annotations for momentary or duration events.

Nyah, I didn't really want to release today.

Tue, 17 Jun 2008 01:28:22 GMT

So I was doing final cleanup and preparation for releasing the first version of my project to a wider audience at work and one of my machines developed a bad case of fingers in ears. By that I mean it stopped listening to its cluster brothers and then it stopped listening to me. The load average kept going higher and higher and while I was able to log in to the machine, frequently, commands I'd issue would just hang and no amount of breaking or kill -9 would help them.

When I gave it up for lost and tried the reboot command, I discovered to my chagrin that it didn't work either.

So, until one of the IT people (who are all swamped with FF3 release stuff) can get around to logging in to the console and killing it, I'm going to have to find other things to occupy myself with. :/

I did come across this interesting comment on linuxquestions.org talking about what can cause a process to disregard kill -9:

What to do when kill -9 doesn't work

Hopefully this will be a thing of the past after Firefox 3 releases...

Tue, 10 Jun 2008 14:24:45 GMT

Firefox Crash « Lolcats ‘n’ Funny Pictures of Cats - I Can Has Cheezburger?

It is so easy to ignore the facts...

Wed, 04 Jun 2008 03:41:33 GMT

Watched this movie about consumerism. It is depressing how deep in the throes of it I am.

The Story of Stuff with Annie Leonard