Analysis Question Without Answers

Reflections on the difficulty of assigning a value to a product feature

Analytics

Sometimes a question sounds simple but like some fractal of suffering as one gets closer and closer one finds things are underdetermined, too open to subjective interpretation, and no result will serve as an appropriate answer.

Published

February 5, 2023

After a few years where simple questions would sometimes spiral out of control into soul-sucking quagmires I developed a habit of asking the following questions whenever someone asks me a question requiring data analysis:

Given your question, is there a correct answer?

Can we tell what would make an answer wrong?

A “no” is a red flag that the question needs to be refined much further. Refinement is necessary to ensure that we can arrive at an answer that is appropriate for the purpose the question was posed. Sometimes a “no” can be quickly cleared up by talking to the stake holder about what they plan to do with the information1.

This kind of thing can happen in data science projects too, where we’re perhaps less concerned with a specific inferential question and more concerned with some other goal. For example, clustering or segmenting customers is a common task, but if we don’t have a clear business goal then such tasks lose a lot of value as the ambiguities stack up.

Here, let us consider the question of the value of a particular feature within a product. This kind of question gets asked all the time, and at face value one might think it is well defined, but it is not.

How Much Does It Cost?

First, we can more easily answer the question “How much does a feature cost?” This can be determined by looking at cost of goods sold, how much time we spend developing/fixing the feature, a look at infrastructure costs for the feature, and any other costs can be attributed or partially attributed to the feature.

Though no time was specified, I think everyone would assume we want to know the costs today for the feature. We can look at these costs related to a window of time around the present. When we answer this question we acknowledge that our answer is not:

  • How much will it cost next year?
  • How much will it cost if we change our ad-spend?

Everyone seems to understand that these are different questions. Yet people don’t make this distinction when they ask “What is the value of the feature?”

Though I’ve claimed it is clear, there may be an issue with calculating the cost of a feature that uses some resource shared among other features. If we removed the feature then the cost of the resource would be shared among the remaining features. Similarly, if the feature did not exist we might be able to make use of the shared resource without increasing the cost of the resource. Thus, it may be reasonable to assign anywhere between none of the resource cost to all of the resource cost to our feature. This is an accouting question where we must choose one and be open about it going forward.

The Value of a Feature

I hope it’s clear to everyone that there is no single number that makes sense as the value for a feature, and that there’s a wide range of numbers we could use that could be correct. The question of a feature’s value is wrapped up too tight in context for a single number to make sense. So here I want to divide the space of contexts until such a question does have an answer.

Population

The value question depends critically on the population we’re refering to. We could refer to:

  • The population of potential customers (under a particular distribution of ad-spend),
  • The value to the population of customers that don’t use the feature, and
  • The value to the population of customers that do use the feature.

For potential customers we could do experiments randomizing which features have prominence and which features are advertised in ad-spend. Different ad-spending patterns could change the people we reach, and so we must be mindful that any result under one pattern of ad-spend may not transfer to another pattern of ad-spend.

For existing customers we could do two possible expeiriments (at least theoretically). We could take customers that do not use a feature and try to coerce them to use it. These customers don’t use the feature for a reason, either lack or interest or difficulty of use or ignorance of its existence. Another test could be to take a feature away from users that do use it. I don’t think anyone would do this in real life, but it’s interesting to think about customers that try to get their money’s worth by using all features even if they wouldn’t care too much if the target feature were removed. How would they respond to such an experiment?

Context Among Features

Another important aspect when referring to the value of a feature is in what context are we evaluating such a value. We may want to know value based on:

  • The feature on its own (as a stand-alone product),
  • The feature in the presence of other features in the product,
  • The feature in an environment with competing products.

Before smart phones you could buy a GPS with a maps package that had a directory of locations. GPS is a nice feature on its own. A directory of places to visit is also a useful feature. Combined, however, their value is more than the sum of the value of the parts. This is an example of a synergistic effect, as you can now do things like effortlessly find the nearest gas station when traveling. In general, it’s difficult to examine such values via experimentation because clusters of features tend to remain bundled together. In these cases we have an identifiability problem, where we can really only speak to the value of the combined features.

If this weren’t a problem, or if we found some way around this problem, how would we define value for GPS? Would it be for GPS alone? Would it be GPS when the directory is available? Would it be some weighted average of the two? The answer to this question likely depends on the reason we were asked in the first place.

We also have to be mindful of the role of competing products in determining value. Having an antivirus software was critical for Windows users for a long time, then Microsoft released a free product and bundled it with new computers that covered many of the required use cases for users. The value of an anti-virus product befor such a product was released should be different than the value after such a product was released. New external products can also enhance the value of features. The rise of ransomware has increased the value of personal computer back-up solutions.

Let’s look at some ways we could estimate feature values from historical data. Let’s assume \(A\) is a feature like GPS and \(B\) is a feature like directory search.

We could look at the value of users that use \(A\) and subtract the value of users that do not use \(A\). Similarly, you could do this for \(B\). The problem that arises is that the users that use both \(A\) and \(B\) get counted both times, and this will cause our estiamte of the value of \(A\) plus the estimate for the value of \(B\) to be larger than the total value of all customers. This can be awkward depending on the purpose for the original value-of-a-feature ask.

We could get around the correlation by looking at the value of users that use only \(A\) and only \(B\) respectively, but in cases where the correlation of feature usage is not very small the count of such users will be very small. Our estimates of feature value will be very small as well, and will not sum to anything near the total value of our customers.

When the correlation is high we could do the following. Let \(a\) and \(b\) represent the unique contribution of feature \(A\) and feature \(B\) respectively, and let \(y\) represent the synnergistic contribution of both. Users of both products, written \(AB\), will have value

\[ AB = a + b + y \]

where-as users of \(A\) or \(B\) alone will have value

\[ A = a + y \quad\text{and}\quad B = b + y~. \]

We could look at \(AB - B = a\) to get our estimate for the value of the unique contribution of \(A\) alone. Or we could use \((AB - B + A)/2= a + y/2\) and similarly \((AB - A + B) = b + y/2\) to get two estimates for feature value that add to something closer to the value for all customers.

Again, we find ourselves needing to choose an approach that makes sense given how this information will be used.

Value Changes Over Time

I mentioned above that the value for potential customers can change as changes in ad-spend target different cohorts. That’s one way value can change over time. Also, we expect our customers to change over time as well. Thus, another important question is what time period are we wanting to find the value of a feature?

  • The value in the past and the present can be calculated with historical data or experimentation, and though there is some uncertainy to our answer we can usually bound it appropriately.
  • The value in the future requires modeling assumptions and predictions that make it difficult and add a lot of uncertainty to our answer.

Metacognitive Effect

We’ve been looking at somewhat objective measures of value, but a weird thing happens when you ask people how much they value a thing. Perception of value will impact how much a customer is willing to pay for a product (or if they’re willing to refer a product to a friend) and thus drives the true value of the product and features within that product. Yet the information we get from interviews can be very different from signals we get from randomized experiments. This can add another layer of complexity if the goal is to improve customer’s perception of value.

Conclusion

“What is the value of this feature?” is not a simple question. Here, we’ve looked at several axes along which we might answer the question differently. The only solution is to work with stake holders asking the question to identify which answer is most appropriate for them.

Above we assumed that features themselves were identified and well defined, but it’s possible this is not the case. One could decompose GPS into lat/lon location and mapping features. A directory could be decomposed into listing and search features. If you get a question like, “We need to know the value of all features and what are our features” it might be time to update the ol’ LinkedIn.

Footnotes

  1. Sometimes the stakeholder does not understand the purpose well as the request has passed through many hands. These situations, where stake holders may be embarrassed to go to their higher ups and ask for clarification, are the most periolous. People may be more willing to pressure you for a wrong answer than to do the work necessary work to get the right answer and possibly look bad.↩︎