Building a real-time dashboard to visualize server logs

Did you ever wonder how many users visit your web application or how many exceptions your server application raised in the last month? The solution to various similar problems is to simply analyze and visualize the log or access files of the particular application.

In this book, we will develop a real-time dashboard to visualize application logs and the server's system status in the browser. Thus, we will build a custom Google Analytics-like visualization dashboard for our own server (see the following figure). Additionally, we want to add custom animation and user interactions—such as selecting data ranges, zooming, and panning—to the visualization.

Google Analytics dashboard

The dashboard (frontend) will be capable of loading, parsing, and grouping log files of different formats, filtering data based on selected data ranges and displaying, updating, and animating multiple chart types. We will build the visualization library with D3.js and the application with AngularJS.

While developing this application, we will see all the advantages, problems, and best practices of D3.js and AngularJS in action. Also, you will learn to combine both frameworks to one single application.

During most of the chapters, we will focus on loading log data from static resources. In the last chapter, we will implement a simple web server (backend) using Node.js and Express to detect log file changes in real time and push these changes to the dashboard using WebSockets and the Socket.IO library.

Besides implementing all the functionality, you will also learn how to design and test (we will use unit and integration tests) the frontend components with the test runners, Karma and Protractor.