The need for open source reporting

Some things are better illustrated with a story. So, let me begin this with what brought me to the world of Open Source Business Intelligence. My story starts on a late Sunday evening in 2001. I was then a student intern for a midsized network security monitoring operation. Here I am, landing a dream job for a college student, a paid internship at a high technology company, and working in one of the most exciting fields in the tech industry—network security. From this single department came some interesting projects and concepts such as Sguil (the Open Source Network Security Analysis front-end for Snort) and SanCP (a network session profiler). Concepts such as Network Security Monitoring (NSM) were being tested and proven over security appliances that promised a silver bullet to all security problems.

In this particular scenario, we are a department dedicated to providing customers with network security monitoring solutions using open source software. This entire NSM philosophy would later be expanded upon and described in The Tao of Network Security (Bejtlich 2004). And yet, here I am, with a broken keyboard, occupying a cubicle in my office where the A/C shuts off on a timer, so it's boiling hot, surrounded by great gadgets and cool blinking lights, flashing screens, and on the cutting edge monitoring of Internet traffic catching bad guys doing bad things. The whole setup is like a scene right out of a spy movie, with a lot of exciting things happening. Such a great opportunity for me, a young college student. Yet, on this particular evening, I was not enjoying my job.

My job here, this evening, was not to monitor the thousands of alerts, nor was it to inspect the exponentially higher amount of network packets in an effort to protect networks from the insidious under doings of the digital underground. Instead, my job this evening was to perform the most dreaded task in the entire operation—the weekly incident report.

Being a company dedicated to open source, we had tons of open source software, proving that the open source paradigm was a workable one. Our workstations were all Red Hat Linux systems configured and customized to help us achieve our mission statement. Our backend servers were all running FreeBSD. Our security console was a custom, in-house developed front-end, built on open source scripting tools (which was the base for what would later become the Sguil project). Our network sensors were all built on Snort and dumped all transactional data into a PostgreSQL data warehouse. Our office productivity suite was an early release of OpenOffice.org. Yet we lacked one important piece of a customer focused service group, a reporting system.

So, what tools did we use to address the area of reporting? We had scripts—lots of tedious, boring, manual scripts that generated lines of ugly text. The scripts took hours to run, mostly due to a lack of proper indexing on the reporting tables as the DBAs refused to listen to us and focused on the transactional databases that housed all our live data. As a result, the scripts were slow and their output was ugly. The scripts may have outputted RTF documents, but no formatting tags were actually used. Part of our job was to go through each of these reports, cut out duplicate lines, change the fonts and font weighting, and manually go through and confirm the accuracy of each of the counts in the summary. This was a time consuming, tedious, and boring task that was highly error prone and, as a result, required a longer validation time. Of course, due to the setup, the scripts were considered part of the Database Administrator and the system developers' tasks, so of course we didn't have access to change anything in the event of an issue. If a query was returning funny results in the reports, all we could see was an output page with no access to the code that generated it.

Now it's the middle of the night, we're tired, the other guy is griping about having to pick up the slack while I am doing the most tedious task of running reports. It is hot, and I'm pouring through hundreds of pages, cutting out duplicates, changing summary numbers to reflect correct counts, and changing the formatting. I am sitting here asking myself the same question I ask myself each week when we run these reports, "Isn't there a better way?".

Fast forward a few years. I have since moved on from BATC into a new role as a senior software developer for Citibank North Americas National Training Department. It's the 2004 Actuate Users Conference in Los Angeles, California. Always on the prowl for innovative reporting technologies since my encounters as BATC, I came to the conference to learn about the different products Actuate had to offer. But the one thing that always stayed at the back of my mind was that nagging desire to find an open source reporting solution. So, as a few days rolled on during the conference, I was getting a little annoyed at one of the product pitches. Therefore, on this particular day's keynote speech, the vice-president of some such department was speaking at a length about all the products that Actuate had to offer. Of course, as I was already starting to zone out, I started doodling in my notebook with little caricatures of him doing sales pitches. But suddenly, as if I had a moment of clarity from a drunken stupor, this VP said the most remarkable thing. He said, "Actuate realizes the importance for an open source reporting product, which is why we have started on an open source initiative with the Eclipse Foundation to make an open source report platform called BIRT".

It was as if he read my mind, and suddenly I couldn't help but listen to my new messiah. Apparently someone out there was listening, and someone had clued in to an untapped market. And thus was my introduction to BIRT.

While my story demonstrates the need for report developers, what about application developers? Well, let me give you a scenario. You work for the MIS/BI group in your company. For years you have leveraged hand coded, database-driven web pages for your reporting. You have come to the conclusion that development takes too long, and decided to leverage your existing J2EE platform because it is time to change your report development habits.

After extensive research, you have come up with a list of requirements. Reports must be accessible online and offer the ability to export to common desktop formats such as Microsoft Office formats and Adobe PDF. You need pagination and navigation as part of your reports. You also would like to take the hand coding of reports out of the loop to make them quicker to develop and deploy. Because you work in a development group, you need the ability to share common reporting elements among your group. You also would like the platform to be flexible and dynamic, and will also need charting capabilities. However, you have no budget at this time for an enterprise reporting platform, so you decide on an open source platform. After some research, you come across BIRT, and decide it is the way to go.