Using a backtest to choose Dynamic Pricing software


Dynamic Pricing software takes data and turns it into actionable real-time prices. At Perfect Price, we've learned a lot about how executives assess and decide on using our software. We think this process could work well for evaluating any software that an enterprise is trusting to make decisions on its behalf.

The problem

As a CFO or CMO deciding to present a new decision-making process to your executive peers, you need confidence that it will work. There is inherent risk in adopting any new business method, especially enterprise software. Will the new pricing system outperform your current strategy? Will it throw operations into disarray? This is just as true for, say, an anti-fraud system. Will it prevent as much fraud as the existing method? Will it reject more legitimate customers? Answers to questions like these are a critical step in the decision process.

Vendor risk is real, as well. Many pre-internet vendors promise rapid timelines–but can take years to deliver. PROS, for example, took 7 years to deliver an upgrade for Avis. Hertz has been working on a similar upgrade for at least 4 years, according to annual reports, with nothing to show for it.

A CEO of a public US hotel company told us that when he chose a pre-internet vendor, he had no idea that implementation would take 4 years, cost $20 million, and require a team of 30 new pricing hires to actually use the software. "If their sales guy told me that up front, I never would have signed up," he said.

How do you mitigate decision risk and vendor risk before you have committed to a seven-figure agreement?

Backtests and their side effects

When data scientists at Google, Amazon, Uber and other leading companies try something new, they backtest it. That means they run simulations using the new approach and the old approach, comparing the results. Where did the new method show improvements? Where was it inferior?

This is called a backtest, and it is the only way to have confidence in a new model or solution before deploying it to real customers.

A backtest consists of several phases, and as you'll see momentarily, there are very positive side effects of this process. (This assumes you have already aggregated your data into a data warehouse of some kind–which most large enterprises have already done.)

  • Data Onboarding and Integration
  • Data Analysis
  • AI Training and Modeling
  • Review
  • Presentation


First, data is on-boarded onto the AI platform. Scalable AI platforms require data to be in a particular format in order to process the data through analysis and models.

The data is then analyzed in smaller, one-off batches. Many models can be applied here to detect, for example, trends and outliers. This can guide the way the AI is trained in the next step.

Once the team running the backtest is familiar with the data and is familiar with the business challenges and limitations, as well as objective goals, they can then begin to train the AI. The way the AI is trained is critical in how accurate it will become; this is frequently an ongoing process.

Once the model seems reasonable, costs and other inputs are included. The work is then reviewed by the business team at the company to make sure that the vendor accurately took into account their business needs. For example, can volume decrease for higher profits? Many businesses do not like to lose volume–even if they could be much more profitable at lower volumes.

Finally, the results are presented to key stakeholders. They should be able to really dig in and not only look at aggregations but also spot check. Are these prices that we could put in front of customers? At Perfect Price, we set up a portal that gives full access to this key data, to give stakeholders the ability to assess the data thoroughly and without compromise.

The side effect of a backtest

At this point, the company and vendor have been collaborating for over a month. The positive side effect is that the teams have gotten to know each other, and the company has had the opportunity to understand how the AI will impact its business and what it's like to work with the vendor. There's no other way to build this familiarity short of implementing the software–and usually, that's a huge commitment from both vendor and company. A backtest is an excellent first step.

Three components to a successful backtest

First, you must use real data–as much as possible, balancing additional data against the cost of using it. This ensures the most accurate and useful backtest, lowering the risk that sample bias leads you to the wrong conclusion.

Second, you must spot check the output (prices, in this case) not just the results, so that you ensure that the business needs are met. This requires engagement throughout the business, so all stakeholders' constraints are included. If there are laws prohibiting price-gouging during national disasters, best not price gouge in your backtest. Tuning, or rules, must be implemented to ensure the prices conform to the business reality.

Finally, you must use an appropriate and scientifically valid simulation technique. The AI provides the basis, but a simulation engine is required to get actual results. For most AI applications, the best approach is a Markov Chain Monte Carlo (MCMC) simulation, which essentially simulates customers showing up and buying or not buying, over time. There are other approaches that you might choose over MCMC depending on the circumstances.

MCMC is a good choice because you can include constraints such as limited supply, and it can be done without knowing how many people actually showed up, all you need are the actual transactions of people who purchased (a technique we have perfected here at Perfect Price).

A backtest involves thousands of simulations of individual microsegments as you tune the AI to understand and react to demand patterns within the real world. Each run of a microsegment would run 10 separate times (is the multiverse real? better safe than sorry).

When you combine the three components–real data in significant volume, an AI that has managed to produce realistic and allowable results, and an appropriate and scientifically valid simulation technique–you can have confidence that the backtest is a representation of your business as if it were using AI.

Backtests vs. consulting

Management consultants do a lot of pricing work and can be a great resource. For a company approaching pricing for the first time, a consulting engagement is perhaps the best entry point, as a huge component of success is the human element. This aspect, "change management" in the parlance, is where management consultants excel.

The primary downside of management consulting engagements is sustainability–or rather, the lack thereof. Engagements with top firms such as McKinsey, BCG or Simon Kucher costs $250,000 minimum for a couple of junior consultants over a 4-8 week period. Their output is a very well done PowerPoint deck and supporting documentation (that most clients never read).

If your market will not change again, ever, then perhaps this is sustainable–but most companies considering AI are very large, and things are changing constantly. Whether it's input costs, competitor actions, consumer preferences, supply, the products themselves–all of this change makes a $250,000 PowerPoint deck look more like a luxury than a practical solution.

Backtests, on the other hand, are much more affordable, for two reasons. First, an AI company like ours is still in this to sell SaaS software–not to profit off of a backtest. So there is no margin in our rates–we are just covering our costs. Second, because it leverages an AI platform, it does not take nearly as many billable-hours.

As our VP of Sales once put it, "It's just like a top-tier consulting firm, but maybe not at their rates."

Caveat emptor

By running a backtest, you as a buyer of AI software can reduce the risk of making decisions differently, while getting to know your vendor, its team and technology much better. All of this at a fraction of the cost of engaging management consultants. The process, if done right, can give you direct insight into what the AI is doing, and the ability to ensure that it conforms to your real-world needs.

Unlike back-of-the-envelope calculations, favored by pre-internet vendors, a backtest uses your real data. By engaging in the process, you can expose "AI pretenders"–who really use Excel for analysis (and no, there is no AI in Excel).

Most importantly, a backtest can help you determine if AI software can add value to your business–and if it can, it can give you the foundation upon which to build the business case to get and implement that software.

If you'd like to learn more about whether a backtest is right for you, please get in touch.

Related Posts