Authors: Lachezar Ribarov, Iskren Nikolov
DynaTrace is a new generation tool for monitoring and profiling of Java and .NET applications. Its patented PurePath technology provides one of the most accurate and detailed view of application behavior. It started as a tool which profiles applications and gives detailed information at a code level during load, and evolved as a product that can be used in continuous development and in production environments.
In the article that follows, we will give some details about the reasons for choosing dynaTrace, the process of adopting the tool as well as some real life examples from the first interactions.
DynaTrace – why and when?
The software solutions that we develop at Experian are composed of multiple tiers that usually include database, backend server and frontend modules. Typically in production environments, our modules integrate with one or more external systems using various communication protocols like RMI, message queues, Web services etc. In some cases the solution may consist of modules written in Java, C++ and .NET. Understanding how such complex system works, finding its potential bottlenecks and pinpointing various configuration or integration issues could be a challenging task. Add the usual time constraints to the picture or maybe the pressure from a customer with production environment issue and you will see why we had to find a tool to ease our lives.
When we started looking for a solution to our problems we had the following criteria:
- The tool should provide visibility of a transaction across all tiers, from the front end browser/client through various server layers and into the backend DB layer
- We were looking at how easy it is to install, configure, capture information, analyze information and subsequently pinpoint a problem.
- The ability to use the tool regardless of the project nature.
- Ability to share the results easily between the members of the team. Best candidate should have good separation of the Collecting profiling information part and Analysis part, as profiling and collecting of results could be done by one team (Delivery, QA) and analyzed by other (Development).
- Non-intrusive installation, i.e. to be pluggable in customer environment without needing special permissions or endangering environment’s integrity.
We evaluated couple of tools, and at the end we chose dynaTrace as it was the one that fit our needs best. We had a Proof of Concept Project with the guys from dynaTrace, which was mainly investigation of one of our products with the tool and we were amazed how quickly we were able to get it installed, running and getting meaningful results.
DynaTrace installation and configuration
We will give a quick explanation of the dynaTrace modules, how they are installed and how quickly you can start gathering information about your application.
The main components of dynaTrace are shown in the picture above. It consists of:
- dynaTrace agents (located in the different product tiers, collecting raw data)
- dynaTrace PurePath collector. Responsible for collecting transactional data sent from different agents. Could reside on the same box as the server
- monitoring Collector – collects environmental information on different machines
- dynaTrace Server – real-time and root-cause analysis of collected data. Since this analysis of data is happening away from the monitored application, the agents do not bring ‘noise’ in the monitored product.
- dynaTrace client – the client is used to connect to the server and get and display the information to the user
Despite of the complex look of this picture, the installation of each component is very easy. In fact, all of them could reside on one box (except the agents, of course) but it is recommended for the collectors to be ‘close’ to the monitored application (e.g. same network).
Installing agents in the monitored product is just copying them on the box and adding additional line in the Java startup command line, as simple as:
Once you have installed everything, started your product and started to use it, dynaTrace will draw a picture of your system similar to the one shown on the next screenshot.
DynaTrace in action
During our first steps with the tool we got a really strange visual representation of the system we were monitoring. Instead of one DB server we were seeing two separate DB servers. From the name of the machine we were able to identify that there was an issue with the configuration. It was just a single incorrect line in one of the configuration files pointing to incorrect DB server with the same credentials. Obviously this is not something that you’ll get in production environment but illustrates how easy it is to spot a problem with a tool that otherwise could take some time to identify.
Another example from the first days of using dynaTrace was discovering that one of the servers was returning errors during execution. You can see that the visual representation of the whole system helps in noticing the problem, where it is occurring – you can see the faulty machine marked in red.
After drilling down deeper it lead us to the root cause, in this case lots of HTTP 404 errors, because of a missing resource on the server. See it on the next screenshot.
The real power of dynaTrace got revealed when we ran some performance tests and during dynaTrace analyses it was visible that in one particular module of the system we spend huge percentage of the time in thread synchronization.
We were able to drill down and to find quickly and easily which are the methods having the biggest impact and started fixing the problem (next screenshot).
Overall, dynaTrace is quite suitable for usage in complex environments with multiple tiers and interactions with external systems. Also, it could greatly help for spotting issues in production environments that are hard for reproducing in a test lab. Its server can be installed in one central location and used by multiple teams in different locations for collecting and analyzing results.