The Do's and Don't of App Messaging: Measure Performance

I won't recycle Lord Kelvin's words; nor will I say garbage in, garbage out. Even if these principles exemplify why we don't know whether messaging initiatives fail or succeed.

You have crafted messages that are delivered at the right time and with the right intention. Your organization is more than capable of mapping user journeys and the critical steps involved. Your startup now understands the importance of teamwork and collaboration. Was the investment worth it?

Let’s quote Deming – “You can’t manage what you can’t measure.” This quote does not mean we share measure absolutely everything. Most of the gauges in your car don’t mean anything to you; unless something lights up. Your engine’s temperature is irrelevant, until there’s vapor coming out of the engine.

I advocate for measuring what matters, and keeping data for future use. The action of calculating and tracking multiple KPIs is expensive. It is expensive on time, on money and on the capacity to focus. I do not advocate for not measuring anything. What I I am trying to say is that you need to make sure you are measuring for outcomes with reliable KPIS.

There are risks

Measuring performance has three big risks. The first risk is confidently displaying numbers and indicators that inherently wrong. Second risk is optimizing for the indicator. (Goodhart’s Law). The third risk is incentivizing certain behaviors which can make the issue worse (Cobra Effect). These risks are not mutually exclusive.

Indicators can be wrong for two reasons: poor calculations or inaccurate data. For an indicator to be trustworthy both calculations and inputs must be accurate. An example of these issues can be seen with CAC or Cost of Acquiring a Customer. The formula can be wrong if we don’t accurately define what is a Customer. Data can be wrong if we didn’t identify all marketing and sales costs. In my experience most people get this wrong because of ignorance rather than malice.

The implications of optimizing with results is known as Goodhart’s Law. The law states that when a measure becomes a target it ceases to be a good measure. In other words, when we reward performance we can expect optimization to achieve this goal. A contact center example can be average call duration. Agents will optimize for ending the call quickly, rather tan customer satisfaction or actual problem resolution.

The Cobra effect is closely related to Goodhart’s Law. The Cobra Effect is a perverse incentive in which the intention has a negative effect. The Cobra Effect name came to be when there was a cobra outbreak. In order to control the number of cobras a bounty was set for every dead cobra. People started raising cobras until the bounty was removed. All farmed cobras were then released into the wild.

What do Challenges have to do with App Messaging?

Challenges have implications with App Messaging Initiatives. Your organization need to make sure data is accurate and properly identified and segregated. Secondly you need to understand what is the desired impact of the communication approach; and whether the organization is properly aligned on the desired result. Thirdly we want to incentivize for outcomes rather than outputs.

Simple actions for better measurements

Clean Data and Accurate Calculations

There are simple actions to properly calculate your KPIs. The first one is making sure your data is accurate. This entails defining your data, creating data catalogues and some levels of Data Governance. The second element is understanding how you determine and establish your KPIs.

Right Data and the Right Level of Granularity

Data is not only relevant for app messaging. If you are not properly determining your catalogue and cleaning up your source you will have faulty accounting and performance data. Are we capable to identify which part of our Twilio SMS bill was used for marketing, onboarding or transactional push notification? If we can’t trust our Twilio expenses, Can we trust our Cost of Acquiring a Customer? This goes beyond billing, if all messages fall in the same bucket, can we slice and dice deliverability or open rates?

Data can be sanitized inside our system or outside of our system. Inside of our system will rely on your data strategy. Sanitizing data from outside our system can be done by properly setting up accounts. These setup is good not only to track our expenses and metrics, but also for InfoSec and access controls.

A good practice before using any third party solution is to understand the account structure. Identify how billing is managed, and whether you will need to use different API Keys. Adapt your third-party service providers to your domains and value streams. Most services like OneSignal, Twilio, Mandrill will have ways for you to organize your account. A concrete example is Twilio you can have an organization and multiple sub-accounts.

Optimize for Outcomes

Preventing the Cobra Effects is fairly straight forward. We need to optimize for the desired outcome and end result. Outcomes and indicators can and should be defined based on the message intentionality. As defined in a previous post, messages should have an intention. Ben Chestnut said it best: “If you have something interesting to say, send an email. Otherwise don’t send anything. It’s really that simple.

Informational Messages: These types of messages are meant to inform the user. No action or feedback is required. Since action is not required, deliverability and openrate are the metric we want to evaluate.

Actional Messages: These types of messages are meant to inform the user and motivate action. User action and feedback are desired. Deliverability and Open Rate are important to track baseline behavior. However both activational and promotional messages will need additional measurables.

• Activational: Activational messages are meant to elicit action. In order to track action we should design a funnel based on number of users that performed the action after opening the message.

• Promotional: Promotional messages have the goal of conducting a sale. In a purist sense, this is an activational message and we need to track the number of users that performed the action. However a good practice would be to also track the additional revenue generated by sending that specific message.

Conversational Messages: This type of communication is meant to have an ongoing flow of messages around a specific topic. Your app will most likely delegate this to your sales department and customer support departments. These departments will take considerations like average wait times and time to resolution. The scope for app messaging could be openrate and deliverability.

Closing Remarks

Measuring performance is more about creating effective measurements and optimizing for outcomes. If we optimize for open-rates with clickbait titles the customer will open the message, but it won’t increase sales. If we optimize for number of contacts we will saturate our customers email inbox and their lock screens at detriment of the Net Promoter Score.

Companies need to understand that each type of message will require its own indicator. Using irrelevant metrics has an impact on cognitive load and the number of people required to keep track of performance. Simplify and align metrics to moments in your customer’s journey.

As you start sending messages you will likely skip straight to delivery rather than planning. We all work with deadlines, but it is important to consider the quality of our data from day 1. Otherwise we will be able to claim fantastic but faulty KPIs.