IDS 019

Selecting Cities for A B Experiment


JMP platforms and statistical methods

Distribution:
Histogram, box plots, summary statistics

Graph Builder:
Dot plots, bar chart

Multivariate:
Correlation, matrix plot, parallel coordinate plot

Clustering:
Hierarchical clustering

Objective

Use hierarchical clustering to match cities across multiple characteristics for the purpose of using one as the control and one as the treatment to test a new client services strategy for an investment brokerage company.

Problem statement

The marketing analytics team within a national investment brokerage firm has been asked to assist the client services division in designing an experiment to test a new strategy to help the company gain share of the mid-net-worth household (MNW) client segment. This new offering consists of providing this client segment with access to services and tools at no cost that are currently only being offered to the company’s high-net-worth client segment (e.g., free quarterly meetings with a financial advisor, online planning tools, etc.).

The plan is to run a pilot of the new program in 7-10 test markets (i.e., large U.S. cities) over the course of a year and evaluate its impact on acquisition and retention of this client segment. The primary question the client service division has is how to determine which cities should serve as the test markets and which non-participating cities should be compared to these test markets to compare to quantify the impact of the program. Clearly the amount of client acquisitions and retention in a city at the end of the year will not only be based upon the impact of the program, but also on many different factors and characteristics of that city. In other words, if the results in a city that receives the new offering are compared to the results of another city that didn’t receive the offering, how can those factors and characteristics of each city be controlled for so that it is reasonable to attribute any differences observed in client acquisition and retention to the offering?

There are 20 cities that are candidates to be used in the experiment. Ideally, 10 will be selected to receive the new offering (i.e., test group) and 10 others, paired with each test group, to continue with the current offering (i.e., control group). These 10 pairs are to be selected such that it is reasonable to assume that client acquisition and retention would be similar for each if they did in fact receive the same offering. However, the client services division is amenable to only seven pairs of cities being included in the study to ensure that each city that does receive the new offering in the pilot program has a corresponding city to serve as the control.

Nine different characteristics for each city that could potentially impact the amount of client acquisition and retention that would occur were identified.