Daniel Einspanjer's journal

Thoughts on Mozilla Community

Fri, 11 Feb 2011 16:00:08 GMT

Jamie Zawinski's 1999 post on leaving AOL/Netscape/mozilla.org

This bit of history was linked today on Hacker News, and re-reading it touched on some points that I have been carefully considering the past few months, prompting me to write this.

I believe most people would agree that the Mozilla of today is a vastly changed beast from the dinosaur of the past. We are faster, stronger, and we are delivering products and services that are changing the lives of hundreds of millions of people.

Of course, we still have a few of the same problems Jamie mentioned in his post, but we are much more cognizant of those problems and both the employees of Mozilla and the community are hard at work every day to solve those problems.

The piece I wanted to specifically share my thoughts about is community participation. Jamie mentioned in his post that the release of the Netscape source code was a beacon of hope to him. It was putting the control of the web browser into the hands of anyone who cared to step up to the task. By the time that he decided to resign however, he did not feel that particular goal had succeeded. He stated that the project was not adopted by the outside. I might argue the details since it wouldn't have gotten where it was today if there was zero adoption, but the point is still useful.

Further down was the kicker for me. In "Excuse #2" Jamie states the following:

"People only really contribute when they get something out of it. When someone is first beginning to contribute, they especially need to see some kind of payback, some kind of positive reinforcement, right away. For example, if someone were running a web browser, then stopped, added a simple new command to the source, recompiled, and had that same web browser plus their addition, they would be motivated to do this again, and possibly to tackle even larger projects."

Wow, that harmonizes with my feelings today. Everyone who is active in the Mozilla Community today is active because they get something out of it. I believe that the majority of people who are no longer active in the community reached some point where they stopped getting something useful out of it. That might have been because something was too hard for them to do by themselves, or because of a bad interaction with another community member or even an employee. But, for whatever reason, they lost that spark and then we lost them.

The Mozilla employees need to become flint and steel for the community. We need to be constantly tossing sparks out, flashes of light that will ignite fire in the spirits of our community and will catch the attention of people across the globe and draw them in.

Back in December, I met with a few people including Pedro Alves, David Eaves, Asa Dotzler, and Dan Mosedale. We were discussing a project to quantify and provide actionable insight for community participation.

Since that meeting, the Mozilla Metrics team has been working on the first iteration of this project, a dashboard that visualizes commit activity in Mozilla's Mecurial repository. Within a few weeks, we should have this ready to share with people. Future iterations will also draw data from Bugzilla patches and reviews. There are a ton of other datasources that we could consider, but these are definitely enough to get us started. I am also very pleased to see several other teams working on other community focused projects. Obviously Mozilla Drumbeat is the flagship there, but the SUMO and Engagement teams have interesting projects in the works as well.

All this activity centered on engaging our community is thrilling to me. Every person that we inspire, assist, and collaborate with is much more likely to "pay it forward" and do the same for someone else. Community is what strengthens Mozilla. There are many open source projects with communities around them, but Mozilla is in the company of a special few whose goals are truly public benefit driven rather than using open source as a tool to make a great piece of software.

Throughout 2011, I believe we need to spend as much time focusing on ways to understand and collaborate to make our community an army of awesome as we spend on the more straightforward engineering task of making our services and products awesome. This is what Jamie stated inspired him about Netscape's original choice to embrace community and open source, and it is definitely what inspired me to become involved and make Mozilla's mission my own.

Kspice is a useful technology for BI and Hadoop

Tue, 01 Jun 2010 18:13:31 GMT

Having infrastructure that is completely redundant and easy to bring down for Linux kernel upgrades is a wonderful thing, but it is pretty hard and frequently you end up with systems that still require downtime maintenance windows for rebooting.
Here are a few examples I can offer:

Pentaho BI server
ETL processing servers
Data warehouse database servers
Vertica cluster nodes
Hadoop NameNode

What Ksplice does is pretty cool. They take your currently installed Linux kernel and apply a series of in-memory binary patches to it without requiring a reboot to keep you up to date with the latest security patches. You can be patched and secure within hours of a critical update.

Harry Potter meets SCIENCE!

Fri, 28 May 2010 21:38:23 GMT

I enjoyed reading the Harry Potter series, but at times found myself quite despising Harry as a kid with too big of an ego who didn't listen and didn't think.
I really loved the Ender's Game series because, for the most part, Ender was quite the opposite of Harry.

To anyone who can associate these feelings with theirs: Please go read this fanfic. Try to read at least through chapter 5, because the author states that the common consensus is that is when the story hits its stride, and I concur with that sentiment.
Please note that I have only made it through chapter 7 so far so I could be setting you up for disappointment, but given the number of chuckles I've had, I don't think this will be the case.

Harry Potter and the Methods of Rationality

Mozilla has at least four sites in the Google top 1000 sites list

Fri, 28 May 2010 14:20:43 GMT

I just found out about Google's top 1000 websites list from a post on the fseek.me blog.
I expected to find mozilla.com near the top (it's position #10), but when, on a whim, I did a text search for getfirefox, I bumped into getpersonas.com at position #137, that's just plain cool. :)

Update: Here are the ones spotted so far:

10. mozilla.com
74. mozilla.org
137. getpersonas.com
673. mozilla-europe.org

Book Review - Pentaho Data Integration 3.2: Beginner's Guide

Fri, 21 May 2010 22:46:56 GMT

Pentaho Data Integration 3.2: Beginner's Guide

On my plane flight yesterday, I was finally able to crack this book open and I ended up reading it from cover to cover in one sitting!
I highly recommend the book to anyone who has to deal with manipulating and moving data from one place to another on a regular basis. Pentaho Data Integration (PDI or Kettle) is an amazing tool for these types of tasks, and this book is a fantastic way for someone to quickly become familiar with the tool and start producing useful jobs and transformations in it.

The book has a very light and easy tone, and it is filled with lots of practical, real-world examples and screen-shots which make it easy to follow along. It has quiz questions (a couple of which I had to double check my answer on, and *that* is telling you something!). I also enjoyed the "have a go, hero" sections where you are given some simple follow-up tasks that help drill in the topic that was just discussed.

I've ordered a few copies to be kept at the Mozilla headquarters so new members of my team can become familiar with the tool that is so much the lifeblood of our team's products, and I also want people in Mozilla who currently write little scripts to munge data to take a look as this tool makes it very easy to turn these processes into something that is visually documented and very extensible.

The book can be ordered directly from the publisher via the link above, or through Amazon. Check it out!

Microsoft's random browser ballot

Sun, 28 Feb 2010 23:48:13 GMT

When I saw what the page was supposed to do, (first five and last five sets displayed in random order) the first thing I thought was, "Man, I hope they tested the fool out of that or someone will make a fool out of them."

Looks like there wasn't quite enough testing. That said, I don't think that anyone could argue that the bug was damaging. No one browser was ever left out or in the same position.

Doing the Microsoft Shuffle

Dear LazyWeb

Mon, 04 Jan 2010 13:51:28 GMT

I'd like to help the Thunderbird community out by posting a series of quick video demonstrations of how I use Thunderbird + Lightning in my daily work.

In order to do this though, I need an extension/jetpack that can redact, blur, scramble or one-way pad all the user data (i.e. e-mail addresses, message/meeting/task content, etc. Those are the minimum I need to do what I want. I suppose others might want to redact other fields like dates and categories and folder names too.

Is there anyone who thinks they might be able to create such a tool without too much trouble?

SSH magic

Tue, 08 Dec 2009 20:34:33 GMT

I use SSH on a daily basis. Most of the machines I connect to can be accessed in one of two ways:

OpenSSL VPN
SSH to a jumphost then SSH from there to the desired machine

I wanted to share the configuration I use to make that easier.

My Bash Aliases
These aliases make it easy for me to do a few useful things quickly:

SSH to the vpn box with or without SSH compression
Run the omnitty terminal multiplexer to be able to interactively work with a cluster of machines
SSH to a particular machine and resume a screen session with my SSH agent variables fixed so I can connect to other machines with my pubkey properly.

$ cat .bash_aliases 
#!/bin/bash 
alias vpn='ssh vpn' 

# VPN with compression (useful when on cellular modem) 
alias zvpn='ssh -C vpn' 

# omnitty doesn't work well inside screen so this is a separate alias for running it. 
alias omnicluster=' ssh -t vpn "ssh -t cluster01 \"omnitty -W 15 -T 125\""' 

alias h01=' ssh -t vpn "ssh -t cluster01 \"/home/me/bin/grabssh; screen -xRR\""' 
alias h02=' ssh -t vpn "ssh -t cluster02 \"/home/me/bin/grabssh; screen -xRR\""' 
alias h03=' ssh -t vpn "ssh -t cluster03 \"/home/me/bin/grabssh; screen -xRR\""' 
alias h04=' ssh -t vpn "ssh -t cluster04 \"/home/me/bin/grabssh; screen -xRR\""' 
alias h05=' ssh -t vpn "ssh -t cluster05 \"/home/me/bin/grabssh; screen -xRR\""' 
alias h06=' ssh -t vpn "ssh -t cluster06 \"/home/me/bin/grabssh; screen -xRR\""'

My SSH Config
This config sets up several important SSH features:

SSH MasterControl: The master control feature lets you share just one SSH communication connection among multiple SSH sessions to the same server. Since All of my sessions are going through my VPN jumphost, this makes all my sessions a little snapier since they aren't each doing their own encryption etc.
ServerAliveInterval: If I suspend my laptop or otherwise lose connectivity, this option makes sure that my SSH connections terminate rather than hanging for an annoyingly long time.
ForwardAgent: Some of the machines I connect to use pubkey and some of my other machines don't. I can configure which groups of machines should use it.
User: Same as above. On some machines, I need to log in as a different user. Specifying it here means I don't have to remember to type ssh user@host all the time.
HostName: I can give a short easy machine name alias here so I don't have to type the FQDN everywhere else

$ cat config 
Host othercluster*
    ForwardAgent=yes 
    User metrics 

Host vpn 
    ForwardAgent=yes 
    ProxyCommand none 
    # ControlMaster is magic that lets you re-use one SSH connection when you connect to the same machine multiple times. 
    # Since all my connections to the servers I use go through vpn, if I use ControlMaster on vpn, I only have one encrypted tunnel 
    # that all the connections to the different servers use.  This actually makes it feel much snappier to connect and use them remotely. 
    ControlMaster auto 
    ControlPath=~/.ssh/%r@%h:%p 
    HostName my-vpn.domain.net

Host *.domain.net *.domain.com vpn cluster*
    ForwardAgent=yes 
    # Magic so I don't try to use my machine username by default. 
    User otherusername

# ProxyCommands ended up being a bit flaky in combination with ControlMaster so I'm just using raw bash aliases instead now. 
#Host cluster??
#    ProxyCommand ssh -t vpn "ssh cluster%h" 

Host * 
    # ServerAliveInterval makes sure that if I close my laptop or lose my net connection, the SSH session doesn't "hang" but rather returns me to a command prompt. 
    ServerAliveInterval 15 
    IdentityFile=~/.ssh/id_dsa

My grabssh Script
Found this script on Sam Rowe's website. It lets me update my SSH agent environment variables so an existing screen session can still connect to other machines with pubkey authentication.

$ cat grabssh 
#!/bin/bash 
# This magic script helps when using SSH to connect to a preexisting Screen session.  If grabssh is run 
# before the screen session is reconnected, then you can run the generated "fixssh" script inside of Screen and it 
# will update your SSH agent variables so that you can ssh to other machines without a problem. 
SSHVARS="SSH_CLIENT SSH_TTY SSH_AUTH_SOCK SSH_CONNECTION DISPLAY" 

for x in ${SSHVARS} ; do 
    (eval echo $x=\$$x) | sed  's/=/="/ 
                                s/$/"/ 
                                s/^/export /' 
done 1>/home/me/bin/fixssh

Performance of Rhino JS engine and Janino library in Kettle

Tue, 17 Nov 2009 06:26:21 GMT

My friend Roland Bouman made an interesting blog post regarding the performance of a bit of JavaScript for Kettle that he saw on a different blog.

Given the large amounts of data that I am shoving through Kettle every day, I tend to be extremely concerned about performance. Even a small inefficiency can lead to dramatic slowdowns. Hence, when I saw his post, I got to thinking about how I would approach the problem if it were within the realm of the large data sets I work with and hence required extreme optimization.

I didn't have a lot of spare time to dedicate to this experiment, so I opted for a screen-cast instead of a nicely formatted blog post. That said, I think there is a certain benefit in being able to see the work flow of someone who is very comfortable with Kettle.

The screen-cast is currently in Apple QuickTime format. Bleh. I need to get a new Ogg Theora transcoder because the one that I tried to use last time is not happy with me and I didn't have time to fiddle with it.

So, if you use Kettle and are interested in these things, here is the screen-cast. Be warned it is 30 minutes long and probably not extremely exciting to anyone outside of the ETL field.

Kettle string transformation optimization walk-through

If you are familiar with developing plug-ins for Kettle and you'd like to take a look at the User Defined Java Class plug-in I demonstrated at the end of the screen-cast, you can pick it up from the Pentaho SVN plugins repository. Just wear gloves because it has rough edges.
User Defined Java Class plug-in

Advice regarding using Travelex Cash Passport cards for travel

Thu, 24 Sep 2009 20:10:50 GMT

If you are a traveling to Europe and considering getting one of these debit cards to make life easier for you while there, my advice is don't! Figure out which of your credit cards charges the least amount of fees for international usage and just use it.

First, a general complaint: interacting with Travelex is a hassle. Their phone menus are a maze of twisty little passages, all alike. When you finally manage to reach a person, you'll have to give them all the exact same information you entered into the phone menu. Calling them when you are on your trip means calling a long distance number plus additional charges if you must use your cell phone or hotel phone or pay phone. The customer service agents are not rude though. I'll give them that.

Most importantly, the card does not work everywhere that MasterCard is accepted. If you give it to a merchant that it is not compatible, their credit card processing machine might decline the card but the money can still be withheld from you for seven days. If the merchant runs the card through three or four times, each attempt will withhold the funds again. A nice €60 meal can turn into a €180 disaster.

If you get into this situation, don't expect help from anyone. The merchant can't do anything because it was declined on his side. When you call Travelex (paying international long distance or international roaming fees), they will tell you that it isn't their fault and that the merchant did something wrong. Ignore the fact that you had to turn around and use a different MasterCard with the merchant and that one went through just fine. Furthermore, Travelex will tell you that your only option is to wait seven business days and see if the hold disappears. If it does, move on to the next challenge. If it doesn't, then expect the following additional hassle:

request a dispute form from Travelex
wait for it to be delivered by mail
fill it out and send them copies of all the decline receipts and a letter from the merchant stating that they did not receive the money that is disputed and that you have already paid them through other means
wait. After pulling teeth with one customer service agent, I was able to get the reassurance that it should definitely take less than a year to resolve. Likely just a few months.

Once you have the funds back on your card, it is likely you might be in the same boat as me and your trip to Europe is now over! So now, how do you get the money back? You have a few options:

Withdraw it from an ATM. You will pay:
- €1.75 ATM fee converted to USD at a worse than market rate
- $x dollars to the owner of the ATM for using a foreign ATM card
- Whatever remainder you can't take out through the ATM will be taken by Travelex after 12 months
Close the card at a Travelex branch. You will pay:
- $20 administration fee
- Worse than market conversion rate (note the fine print that the "Currency Return Guarantee" doesn't apply to the money on the card!)
- Another arbitrary fee/commission that varies from branch to branch and (I suspect) the mood of the employee
Leave the funds on the card until your next trip. You will pay:
- €2.30 Monthly inactivity fee each month after a year.
- The unavailability of the money for the duration (Travelex will happily take advantage of the money while you aren't using it though!)

Ubuntu screen-profiles customization

Tue, 08 Sep 2009 14:16:12 GMT

I recently loaded Ubuntu server 9.04 onto a new machine and encountered Ubuntu's screen-profiles.
In general, I like it. I had one problem and one customization that I wanted to share:

I use Mac OS X's Terminal.app to connect to my remote machines, and by default, it has custom mappings for F1 through F4. I have no idea what those keybindings mean, but they prevent screen-profiles's keybindings from working. It took a little fiddling to figure out how to fix them. Basically, you need to:

Open up the preferences dialog for Terminal.app
Go to the Settings pane
Click on the Keyboard tab button
Edit the action for each of the F1 through F4 keys
When editing, click the "delete one character" button twice to erase the characters currently in there (leave the \033 escape)
Type the following characters: [ 1 1 ~ 11 is F1, 12 is F2, 13 is F3, 14 is F4
The new entries should look just like the F5 through F8 actions.

Once I was able to use the F2 F3 and F4 keys, I decided that they weren't that useful to me. I prefer to use a combination of screen regions and windows. The window commands are very easy for me, but I've always found the split, focus, and remove keybindings to be uncomfortable so I figured those would be great commands to map to F2 F3 and F4. Here is how I did that:

sudo cp /usr/share/screen-profiles/keybindings/common /usr/share/screen-profiles/keybindings/regions
sudo vi /usr/share/screen-profiles/keybindings/regions
replace the first four entries with the new entries below
save and close the file
In screen, hit F9 to bring up the menu
Select the option for "Change keybinding set
Select the new "regions" entry
Hit F5 to reload your screen-profile and pick up the new keybindings.

register n "^aS^a^i^a^c^aA"                                 #     | Goes with the F2 definition
bindkey -k k2 process n                                 # F2  | Create new region and window (and name it)
bindkey -k k3 focus                                     # F3  | Next region
bindkey -k k4 remove                                    # F4  | Remove region

Shell script analytics

Thu, 30 Jul 2009 05:55:16 GMT

I just made a rather lengthy post on the Mozilla blog of data about shell script analytics. I'll try hard not to cross post stuff like this too often, but I thought I'd allow myself the spam this time around because using Bash and AWK to do things like this really is an important part of who I am personally as a geek in addition to what I do for Mozilla. :)

I've always thought my job was fun. Now I hear it is sexy too!

Fri, 29 May 2009 15:10:39 GMT

I just finished reading this lovely little post from the company dataspora titled The Three Sexy Skills of Data Geeks.

By far, my favorite quote was, "A good data munger excels at turning coffee into regular expressions and parsers". That certainly describes me to a tee. :)

I've always found each of these the three facets of working with data fascinating. One of the comments mentioned that decision making was an important missing trait. I could go either way there. I feel it is good to be able to tell a compelling story with the data that helps others to understand it, and then those people take the understanding you imparted to them and make decisions based on it.

It is incredibly hard to find a person who is skilled in just one or two of these facets. When you find the data geek who has all three, then you count yourself lucky. Expecting someone who has that caliber of devotion to data to also be capable of making decisions like a CEO is a bit unrealistic in my opinion.

Anyway, the article is a good, quick read. It also quite nicely summarizes the major passions in my professional life right now.

Interesting crowd-sourced solution sites

Wed, 27 May 2009 14:49:42 GMT

A good friend of mine runs the site bug.gd (and it's more professional pseudonym, errorhelp.com). This service provides something that is slightly missing from the typical Google search for an error to find a solution. It allows you to enter the full text of the error message or stack trace instead of just a couple of keywords, and it provides rich community feedback on solutions. You can even tip people for their solutions through tipjoy.com integration.

I recently came across two other nice sites created by a different company that provide a similar and complimentary service:
stackoverflow.com - A site dedicated to crowd-sourcing answers to programming questions
serverfault.com - A site dedicated to crowd-sourcing answers to system administration questions

I think it is very helpful to have a list of these sites that you can go to post a question and hopefully get an answer that will even be moderated by the community to help you determine the value of the answer. This is something that typically takes a lot longer if you search for a forum or mailing list site and post there. While it is less immediate than IRC, the moderation and ability to leave a question and get an answer "soon" are nice features you are less likely to see in IRC (although I've always gotten great results from #java, #sql, #mysql, and #bash).

As you can tell, I'm a big fan of crowd-sourcing. I have run a couple of contests on 99designs.com and have been incredibly pleased with the results that came out of that community of freelance graphic designers.

Check these places out and see if they can help you or if you can help them!

Ways to visualize and share data

Fri, 17 Apr 2009 22:40:59 GMT

Mozilla needs to be able to provide useful extracts of data such as download trends, etc. and allow the community to perform their own analysis on them, so I'm always keeping a lookout for useful tools to further that goal.

When Tony Wright posted the blog entry Just How Important is the Valley? Let’s Look at some Data on April 17th 2009, he was kind enough to publish the data set (it needs an attribution / license though) and the data looked interesting so I thought I'd spend a little time playing with it using some tools that I've been keeping my eye on.

First, I slurped the table into DabbleDB, a website that is very well suited to messing with this type of data (i.e. sourced from the web, might need a bit of cleanup, etc.). You can view and edit the data I imported to DabbleDB here: Acquired Startups Data

DabbleDB does a great job at allowing a user to sort, filter, group, and modify data using a simple interface, but it does not have a large array of visualizations. For that, we head over here to the IBM AlphaWorks lab's project, Many Eyes Wikified. I created a quick wiki dashboard for throwing together a few visualizations: Acquired Startups Visualizations

This was just a quick break from real work I've been doing, so I spent less than an hour on this. I only took about 20 minutes with DabbleDB: importing the data, cleaning the dollar values, then creating two new views that group the data by country or by state for visualization. Then I moved over to Many Eyes and played with a few visualizations to try to find some interesting views of the data and threw them into the dashboard and two sub pages.

Being able to quickly extract, transform, and visualize this data is the big win for DabbleDB and Many Eyes in my opinion. With both applications having open licensing of the data and collaboration as a key focus, they are tools that I hope to be able to take advantage of at Mozilla soon.

Counting unique visitors in SQL

Mon, 30 Mar 2009 16:17:52 GMT

A lot of web metrics solutions out there like NetTracker or Omniture allow you to perform analysis on the number of unique visitors over time. This is a pretty important metric to a lot of companies, and I recently needed to perform such an analysis, but it was on data stored in a SQL database rather than in one of these proprietary solution's data-stores.

Doing any sort of distinct counting on a large volume of data in SQL can be very costly, both in terms of storage of the raw data (since you can't aggregate it), and in query performance since there are relatively few optimizations that can be performed on the table or the query.

Fortunately for me, our data warehouse is stored in Vertica, and while the queries weren't blindingly fast, I was able to get the analysis done in a very reasonable time frame.

I was dealing with a week worth of traffic (about 80m requests per day), and one of the biggest challenges I had was how to determine what constituted a "unique visitor" (in this case, it is actually more of a unique ping or requestor since there isn't really a person involved).

I didn't have a cookie that I could use, so that left me with the less desirable course of using a combination of IP address and User Agent string. The problem with this is that the solution will under count one class of requests, and over count a different class. Here are the details:

1. If a request comes from a host that receives its public IP address via DHCP (e.g. a cable modem or DSL) and that service provider has their DHCP configured to force a change of IP addresses when the host renews its lease, then when the IP address changes, the requestor will be considered "new". e.g. HostX makes a request on Monday with IP 1.2.3.4 and a request Tuesday with IP 1.2.3.4. Then, on Wednesday, their IP address changes. Later Wednesday, HostX makes a request with its new IP 2.3.4.5. When we perform analysis on this week, we will see one distinct requestor on Monday and Tuesday, but a new requestor on Wednesday. In the worst case, if a new host, HostY is assigned IP 1.2.3.4 which HostX used to have and HostY also makes a request using the same OS version, we will mistakenly believe HostY on Wednesday through Saturday is the same distinct requestor as HostX from Monday and Tuesday.

2. If several hosts are on the same LAN network (e.g. an office), then the public IP address will likely be the same for each of those hosts. I use the partial user agent string to help mitigate this problem. I am pulling the OS and locale details out of the user agent string and using that in addition to the IP address to determine uniqueness. Unfortunately, there are a lot of machines running Windows XP with en-US, so this is only partially helpful. Any host with the same IP + OSversion + locale will be treated as a single distinct requestor in this analysis.

3. When I worked on this IP+UA strategy, I originally tested using the full UA (user agent) string which includes the browser version number. This might make sense for many other websites, but unfortunately, what I saw in the test cases that I used was that we would "forget" the distinctness whenver the browser is upgraded, or in some cases, even when certain plugins or extensions are installed (ones that modify the UA) [I'm glaring at you, MegaUpload!].

So, with this strategy in place, I ran the numbers and while I could see an unfortunate amount of under counting (i.e. multiple requests being counted as the same distinct requestor when they likely should have been separate), it was as good as I was going to get.

The last thing I needed to do was to write a SQL statement that added up the number of distinct requestors grouped by the number of days in the week that requests were made. Here is the SQL I wrote to do that. This was just my first stab at it, I got my answer, and it didn't take more than a few minutes, so I left it at that. I'd still be interested in hearing if anyone else has a better way. :)


SELECT (d15 + d16 + d17 + d18 + d19 + d20 + d21) AS RequestsPerWeek
, COUNT(*) AS NumDistinctRequestors
FROM (
    SELECT
      MAX(CASE WHEN d.date = '2009-03-15' THEN 1 ELSE 0 END) AS d15
    , MAX(CASE WHEN d.date = '2009-03-16' THEN 1 ELSE 0 END) AS d16
    , MAX(CASE WHEN d.date = '2009-03-17' THEN 1 ELSE 0 END) AS d17
    , MAX(CASE WHEN d.date = '2009-03-18' THEN 1 ELSE 0 END) AS d18
    , MAX(CASE WHEN d.date = '2009-03-19' THEN 1 ELSE 0 END) AS d19
    , MAX(CASE WHEN d.date = '2009-03-20' THEN 1 ELSE 0 END) AS d20
    , MAX(CASE WHEN d.date = '2009-03-21' THEN 1 ELSE 0 END) AS d21
    FROM distinct_requests a1
    JOIN dates d ON a1.utc_date_id = d.date_id
    GROUP BY a1.ip_ua_id
) x
GROUP BY (d15 + d16 + d17 + d18 + d19 + d20 + d21)
ORDER BY (d15 + d16 + d17 + d18 + d19 + d20 + d21)

TinyArro.ws URLs

Mon, 09 Mar 2009 16:17:37 GMT

A friend just released an URL shrinking service that I enjoy: tinyarro.ws (more nifty when written as ➡.ws).
It has a few great features over the current main stream shrinkers:

1. Cool/fun URLs (e.g. http://➽.ws/囨 for my website)
2. Very short URLs due to Unicode suffixes (great for Twitter!)
3. Preview by default! (no tweak to the URL to remember)
4. Option to enter your own custom suffix (TinyURL now has this, but it was too useful to not mention).
5. A Ubiquity command ›.ws/☺ (eventually to be integrated directly on the site)

Some news about the site:
TinyArro.ws: 10 new unicode domains. Defaulting previews to ON.
Ask HN: Thoughts on TinyArro.ws? Tiniest urls in the world (or your money back)

Willingness to be a little evil

Thu, 12 Feb 2009 19:33:08 GMT

I have been a supporter of Firefox and Mozilla for several years now, and while I don't write patches and fix bugs, a major part of that support is educating people about Mozilla, open source, and user empowerment whenever a conversation about technology allows for it.

I've found that people who use proprietary software and operating systems often fall into two broad categories for rationalizing that choice:
1. They are told to do so by some authority (usually their employeer, sometimes their social tech support person, and in some cases, just because they were told it was the right thing to do by an ad or magazine article).
2. They started using it for some reason (typically reason #1 above) a long time ago and are now just accustomed to it.

I'm sure all this is going to be old news to most people reading this, but I bring it up because of an interesting article I read today.

In the 1960's and early 70's, psychologist Stanley Milgram performed a series of famous experiments that tested the willingness of people to do something they would normally object to on moral grounds when they are in a strictly controlled environment and instructed to do so by an authority figure.

More recently, psychologist Jerry Burger had the opportunity to perform a series of similar experiments. This alternet article describes the story and discusses the findings. As I read the results and Dr. Burger's statements regarding the findings, I started thinking about how easy it is for the people to choose to give up their freedom to a piece of proprietary software for reasons similar to the ones described in these experiments.

In a green field, these people would normally opt for software that provided them with more freedom and in many cases, subjectively better security, but because they are instructed by an authority figure, or because they got started with it a long time ago and just slid deeper and deeper in, those preferences are not enough by themselves to prompt the person to change their behavior.

Now even this thought in and of itself would not be enough to prompt me to blog about this topic. We're still well in the territory where the people who haven't gotten lost in a Wikipedia article about toothbrush hygiene they found when they clicked my first link are saying, "um, DUH!" So here is my point:

At the end of the article, Dr. Burger focuses on an interesting finding of both experiments. When a person is instructed to do something "wrong", they are significantly less likely to do so if they are surrounded by peers who object first.

So when you talk to someone who is sighing about how much they hate product X but they don't have a choice, don't hate on them and don't deride them for not having a backbone, but just tell them and show them how you chose to stand up for your freedom and your security. An example can go a long way toward giving them the courage to listen to that little voice inside saying, "I want something better!"

Bash functions for going up to a directory

Thu, 05 Feb 2009 21:43:45 GMT

Sometimes, if I'm in a really deep directory, I don't want to cd from / nor do I want to cd ../../../..
I just want to either go up 5 directories, or maybe I want to go up to the parent directory "src" when I'm in /home/dre/src/projects/foo/bar/classes/org/apache/blah

This set of Bash functions lets me do that.
The first, up() will change your directory. The second will instead just print the desired directory name. This makes it easy for you to mv a file up higher or something.

If you pass no arguments, it just goes up one directory.
If you pass a numeric argument it will go up that number of directories.
If you pass a string argument, it will look for a parent directory with that name and go up to it.
(Note, there is a small display bug there. If you give it an invalid name, cd reports the "No such file or directory" error, which is good, but it has a bogus path. Since you can't know what path they were actually trying to go to, it should just say "No such parent directory: ${yourbogusname}". I don't have time to figure that out right now though.)

Just put these functions in your ~/.bashrc file and don't forget to source it. ( source ~/.bashrc )


function up()
{
    dir=""
    if [ -z "$1" ]; then
        dir=..
    elif [[ $1 =~ ^[0-9]+$ ]]; then
        x=0
        while [ $x -lt ${1:-1} ]; do
            dir=${dir}../
            x=$(($x+1))
        done
    else
        dir=${PWD%/$1/*}/$1
    fi
    cd "$dir";
}

function upstr()
{
    echo "$(up "$1" && pwd)";
}

All hail Ken Kovash!

Fri, 19 Dec 2008 16:18:13 GMT

It may be showing my ignorance, but I was unaware until recently of the officially recognized day for celebrating the man, the myth, and the math that is Ken Kovash. To think that all the time leading up to this point, I had just been satisfied with the joyous feeling in my heart every day I interacted with him.

Ken can be a harsh task-master some times.
"Daniel, where are my numbers from yesterday?"
"Daniel, why are the funnelcake trends low here and high there? You're data are wrong, go find it and fix it!"
But the pain is worth it when I see him take my crude raw data and masterfully sculpt it into bounteous bevies of tables, raging rivers of trend lines, triumphant towers of bar charts, overwhelming ontologies of pie graphs, and gilt-edged grids of treemaps.

One must weep to behold it.

Performance improvements at the cost of complexity

Tue, 09 Dec 2008 16:09:55 GMT

I discovered something that I feel is a bit of a bug in the Sun Java implementation.

If you pass in a string to the method InetAddress.getByName(), it does a bunch of testing to see if it is a domain name or a literal IP address.
If it is an IPv4 address, it will then use String.split() to split the four parts. String.split() uses regexes to do its work.

That means that if you are querying for hundreds or millions of addresses in a tight loop (as I've been doing), the JVM is spawning and compiling hundreds or millions of regex objects, in addition to a String array and four String objects per call.

So at first, I just worked around it by doing basic substringing instead of splitting. That gave me about 100x performance improvement. But then I realized I was still generating four string objects for every call..

So I came up with this mapping method and it runs about 1000x faster with a near constant minimal memory footprint.

I pre-calculate a multidimensional array of shorts where each element is indexed by the literal character value - 48 of the digits making up the number 0 - 255.

With that array available, at run time, I can do a simple lookup of the short value and then do the math to get the long representation of the IP address. I'm still generating a couple of references and a few intermediate int values, but the JIT optimizer can make quick work of that.

Linked is the test program I created to play with the different methods: InetAddressParse test

Don't listen to bash, it will lie to you!

Fri, 28 Nov 2008 04:16:37 GMT

Remember folks, if you mv a directory, and there is a bash shell currently in that directory, the bash prompt will not update to reflect the new name until you cd out of the directory and then back in.

I just spend way too long making changes and being frustrated because the changes weren't having any effect. I was clearing cashes and restarting applications and monitoring log files.. It wasn't until I happened to do a :pwd in vim while editing the file for the umpteenth time that I finally noticed that the file I had been editing was actually in a backup of the folder that I had just made.

Sheesh.

The best DHTML date range picker I've ever seen

Sat, 25 Oct 2008 01:37:41 GMT

Filament Group's Date Range Picker

It uses jQuery and a JavaScript date parsing library by the name of Date.js. This thing is simply amazing. Some of the reasons I think so:

The developer can configure a start and end date limits based on what is valid for the system (e.g. if you only have data going back to 1999, no sense in letting the user chose a date in 3000 BC)
The developer can configure a set of predefined ranges such as "Last week", "Month to date", "Year to date".
If the developer allows it, the user can use any combination of preconfigured ranges, a single date, an arbitrary range of dates, or they can use the back and forward arrows to roll the current date range forward or back.
It is smooth and crisp, able to be easily themed, and seems pretty extensible/tweakable.

It is still a work in progress (they just released it today), but I think it is still usable. The only downside that I've found so far is that the back and forward arrows in this very first released version can produce some unexpected ranges. They are currently strictly math based, so if you do something like select the current month and then hit the back arrow thinking it will select the previous month, you'll probably get something slightly different since most adjacent months don't have the same number of days.

I'm also pretty sure it has an off by one error in it that I suspect they'll fix shortly. If you select Sunday to Saturday of a week and then scroll backward, the next range is actually Monday to Sunday and the next Tuesday to Monday...

Ignore these nitpicks and go check it out right away if your website needs a date picker though. To get such a fantastic widget in the very first release can only mean that it is going to be the bee's knees after a little public beta testing.

Open Source Hardware

Tue, 21 Oct 2008 16:06:31 GMT

I thought that this article in Slate about Open Source Hardware was a fun read and worth sharing.
There is an interesting similarity in the way that Arduino handles open sourcing of their design but reserves the trademark to preserve brand quality to the Mozilla Firefox trademark.

If you like reading about geeks going against the status quo in their industry and trying to make the world a better place, give the article a read.

Good bye Mountain View

Sat, 11 Oct 2008 05:56:56 GMT

It has been a great two weeks out here in the office. I've gotten to see a lot of people face to face and had some useful meetings about my projects. I just kicked off another round of massive data loads to run over the weekend while I'm out of pocket. Hopefully they will run smoothly and deliver me high quality data.

There are some really exciting things coming up this quarter:

I'll be working on one of the largest data sets yet, our AMO data. We have several really cool mechanisms for visualizing individual extension projects hosted on AMO. The developer has control over whether to make the statistics public or not. As an example, you can take a look at the statistics for Adblock Plus. I'll be working on ways to be able to integrate data across projects so we can get a better understanding of the extension community that means so very much to Mozilla.
I'll hopefully be blogging a little more about the complexities of processing the large amount of data that I have to crunch through.
I'll be making several pieces of my Pentaho Data Integration (Kettle for those of you in the know) ETL scripts available in an open source repository. It will help with the blogging, they might be useful to other people doing similar things, and who knows, maybe some people will even have suggestions for improvements!
Later in the quarter, I'll be working on an exciting new project to take some of the aggregated data that Mozilla has, such as the number of downloads of Firefox for given time periods, and making it available publicly for the community to explore and visualize. At the moment, I'm leaning toward trying to use the Many-Eyes project from IBM AlphaWorks. If anyone has any better ideas, please let me know.