A Toolkit to Better Model Consumer Choice using Unstructured Data

There are many reasons economists study consumers' preferences in a given market, from estimating the impact of a corporate merger to evaluating school choice policy, or for making assortment decisions in online retail.

Thanks to the ubiquity of e-commerce, usable information about products has expanded beyond simple product attributes. For example, instead of having only information about a book’s price, author and genre, an analysis can now include information extracted from cover images and text including titles, plot descriptions, and reviews. Machine learning and advances in AI have made this unstructured information more easily attained and quantified.

This rich additional data provides an opportunity to better understand and model consumers' preferences. Such models are crucial for answering “what if?” questions, or “counterfactuals”. 

To make the information usable in models of consumer choice, the economist must first quantify this qualitative information. 

It’s this transformation from qualitative to quantitative, or unstructured to structured, where things quickly go awry. In increasingly common practice, AI or ML algorithms are leveraged to quantify the unstructured data. The outputs of such algorithms can be viewed as proxies for the product attributes consumers care about. However, the quality of these proxies can vary greatly from one algorithm to another. A poor choice of algorithm, and therefore a poor proxy, can materially bias the counterfactuals, leading to the wrong insights.

In market analysis involving hard-to-quantify data, practitioners need ways of choosing among these different proxies, and correcting the counterfactual analysis for any proxy error.

In a new paper, Yale economist Timothy Christensen and his co-author Giovanni Compiani provide a practical toolkit to solve these problems.

“The intuition for the bias correction is simple: when proxies are imperfect, the model fit will tend to suffer in a way that correlates with the bias in the counterfactual estimator," notes the paper. "Our approach leverages this correlation, mapping the discrepancy between the model and the data into a correction for the counterfactual.”

The authors use the following example: in an experiment, a large group of people chose one novel they wanted to read from among 10 titles after being shown cover images, price, and textual description and reviews. Then they were asked for their second choice if the first was not available. 

It is important for online retailers making pricing and assortment decisions to be able to predict second-choice behavior, which isn’t usually observed. But the experimental data provides a useful laboratory for validating the model’s ability to predict counterfactuals. 

Information about the books, both the straightforward attributes and the fuzzier ones, were encoded and used in a demand model to predict what a person’s second choice would be based on which title they had chosen first; these were then compared with the actual second choices. The bias correction developed by Christensen and Compiani improved the ability of the model to predict second choice behavior from 40% to 70% accuracy.

The importance of correcting biases extends well beyond this example.

“As e-commerce continues to expand and such data play an increasingly central role in driving consumer choices, the need to incorporate these sources into demand estimation will only grow.”