ISV App Analytics: 3 patterns to improve quality, sales, and your roadmap

Application analytics are playing an increasingly important role in DevOps and Application Lifecycle Management more broadly – but ISV-specific use cases for application analytics have not gotten as much attention. ISV use cases – and by extension, the analytics patterns employed to support them – are unique. Three patterns described here are Beta, Trial, and Production builds. Clients and/or prospects using these “product versions” come with different expectations and hold different kinds of value to the ISV – and, as such – each instance of what is essentially the same application should be instrumented differently.

The case for injection

Typically, application instrumentation is implemented via APIs inside the application itself. While this approach offers the greatest control, any change requires a new branch or version of the app itself. With injection – the process of embedding instrumentation post-compile – the advantage is that you are able to introduce wholly different instrumentation patterns without having to rebuild or branch an application's code base.

The following illustration highlights the differences in instrumentation patterns across product version – patterns that we, at PreEmptive, use inside our own products.

Beta and/or Preview

• Measure new key feature discovery and usage 
• Track every exception that occurs throughout the beta cycle 
• Measure impact and satisfaction of new use cases (value versus usage) 
• *PreEmptive also injects “Shelf Life” – custom deactivation behaviors triggered by the end of the beta cycle 

Trial

• License key allowing for tracking individual user activity in the context of the organization they represent (the prospective client) - this is CONNECTED to CRM records after the telemetry is delivered
• Performance and quality metrics that are likely to influence outcome of a successful evaluation through better timed and more effective support calls 
• Feature usage that suggest user-specific requirements – again, increasing the likelihood of a successful evaluation 
• * Preemptive injects “Shelf Life” logic to automatically end evaluations (or extend them) based upon sales cycle 

Production

• Enforce organization’s opt-in policy to ensure privacy and compliance. NO personally identifying information (PII) is collected in the case of PreEmptive’s production instrumentation. 
• Feature usage, default setting, and runtime stack information to influence development roadmap and improve proactive support. 
• Exception and performance metrics to improve service levels. 
• * PreEmptive injects Shelf Life functionality to enforce annual subscription usage. 

The stakeholders and their requirements are often not well understood at the start of a development project (and often change over time). Specifically, sales and line of business management may not know their requirements until the product is closer to release – or after the release when there's greater insight into the sales process. A development team could not use an analytics API even if they had wanted to. …and this is one very strong case for using analytics injection over traditional APIs.

PreEmptive Solutions ISV application analytics examples

Here are recent screen grabs of Dotfuscator CE usage (preview release) inside Visual Studio 2015.

Here is a similar collection of analytics Key Performance Indicators (KPIs) – this time focusing on current user evaluations.

…and lastly, here are a set of representative KPIs tracking production usage of DashO for Java.

If you’re building software for sale – and you’d like to streamline your preview releases, shorten your sales cycles and increase your win rates – and better align your product roadmap with what your existing clients are actually doing – then application analytics should be a part of your business – and – most likely – injection as a means of instrumentation is for you as well.