Over the last week or so you will have seen a couple of blog entries from my colleagues on how we are implementing Continuous Performance Management in our existing CI process and how CI and Application Performance Management are drawing closer and closer.
It’s time to start putting some meat on the bones of Continuous Performance Management and introduce HelloCPM and show how CPM and be integrated into an existing Java module using ANT as it’s build mechanism. Our CPM implementation can work with ANT or Maven, but for the sake of this introduction, we will focus on ANT.
Our HelloCPM project has a default build.xml script which is used, in normal circumstances, to execute builds, tests etc. It also has a cpm-build.xml file which contains the CPM build tasks and properties that are required for the CPM process to succeed. We wanted the addition of CPM to an existing build infrastructure to be as non-invasive as possible, hence we have a separate cpm specific build file. In fact, this file imports the primary build file to gain access to source properties, build destinations and other project specific properties.
CPM Preprocessor inspecting source code and generating cpm specific build script
On invoking the cpm-build.xml file, our main project source code is analysed by a cpm preprocessor to identify the project unit tests. Once these unit tests have been identified the preprocessor generates a larger, source code specific build script which contains ANT batchtest tasks which control the execution of our unit tests. These tests are then executed via the JProbe Analysis Engine wrappers: Code Performance, Heap Usage and Code Coverage. Our CPM implementation leverages ANT tasks and functionality already available as part of the JProbe product. It builds on this existing functionality, automates project integration and focuses user effort in tracing real, measurable code defects and elminates or reduces the amount of time consumed by investigating false positives.
CPM comparing build metrics against previous norms
On completion of the cpm build process, the three sets of metrics (Performance, Heap and Coverage) are collected, merged and inspected for deviations from the norm before being committed to the CPM database. This inspection process is important. It allows us to compare the execution profile of the most recently executed build against what we consider to be the norm. Significant deviations from these norms are flagged for further investigation and, depending on the frequency of the deviation, notifications sent to the development and project management teams working on the module in question. We refer to the first occurance of what will later be determined to be a shift in the normal execution profile as an origin point. The identification of these origin points is one of the major capabilities that CPM introduces into a CI process.
Once the data has been committed to the CPM database, it can be displayed and manipulated in a number of different ways. Different levels of users require different viewpoints of on the data. Recognising this, we have created a number of views including a Dashboard view, a Build Summary View with graphing and a Test Case Detail View including graphing.
The Dashboard view, or CPM-At-A-Glance view, is our portal into CPM. Today, we have a portlet that shows current state of health of a particular module(s) grouped by Project. It uses a traffic light system; Green - Healthy, Orange - Module Strike Count 1, Red - Repeated Deviations and notifications are likely to be sent out. We use the term strike count to determine how many repeated deviations can occur in sequence before the team needs to be notified. This is useful to avoid, once off anomalies occuring during a build e.g. CPU spikes due to virus scanning, backups, updates etc.
CPM Dashboard View
The Build Summary View renders graphs for your chosen Module, displaying Execution Time, Heap Object Change, Heap Memory Change and Code Coverage. This graph spans the development lifecycle of the module and highlights individual builds via datapoints, marks last Acknowledged builds and clearly displays the execution profile of the module is an easy to interpret visual manner.
CPM Graphs showing Origin Points
Each build data point on the graphs, on selection, invokes a search bringing the user to a list of Test Cases executed during the build. These Test Cases are ranked by deviation count, instantly bringing potential problem test cases to the attention of the development and management teams.
CPM recognises deviations at a Test Case level
These views, along with the notification system built into the tool mean that unexpected execution profile deviations no longer slip under the radar; the development and project management teams are aware, at all times, of the execution profile of an application under development. Deviation origin points are highlighted and the team is in control; reducing the cost of mistakes as the development cycle progresses.
In the coming weeks, I will work through real examples and show how repeated deviations can be picked up by CPM and how the root cause(s) can be tracked down using the CPM Toolkit UI and JProbe.
Jay
1 Response to “Continuous Performance Management in Practise”