Year: 2006

I am selecting a CRM package for a bank. I asked my colleagues how they’d gone about it, and got 8 responses. Every single one of them had the same weighting approach: Take a huge list of criteria, assign weights, score each package, calculate a weighted-average score, pick the highest one.

As I mentioned earlier, I think weighting is a lousy method. (See Errors in multicriteria decision making.) You can’t say “I picked this package because it has X, Y and Z features, which the others don’t.” You can only say, “Oh, overall, it has the highest score…”

The scores and the weights are subjective. You spend ages arguing between a 3 and a 4. You can manipulate them very easily. And you end up having to revise the scores many times to get to the answer you want.

Since I now had an opinion, I put my foot down, and said, “Here’s what we’ll do. Let’s make a list of essential criteria. They will all be YES / NO questions. Any package that doesn’t meet any criteria is knocked off. That’s it.”

This may appear simplistic, but it isn’t. You see, at the end of the day, only a few criteria really matter. Ideally, you just pick these, and compare packages against these. Since you don’t know which these are, you make a bigger list, evaluate them all, and then realise the truth.

Sometimes, you have too many criteria. Then none of the packages make it, and you have to sacrifice some of your criteria.

Sometimes, all of them make it. Then you can choose to enforce more criteria. Or maybe not. If all of them meet your criteria, just pick the cheapest one.

Internally, we were convinced of this approach, and took it to the client. Things were fine for a week. Then, complaints started trickling in.

1. Fear: “I’ve already used weighting in earlier package evaluations. Now, if you use filtering, it’ll make my earlier evaluations look bad…”
2. Uncertainty: “I don’t know… what if there’s no YES / NO answer? What if we need shades of grey?”
3. Doubt: What if it lets everything through? What if it rejects everything?

Uncertainty is the most popular objection.

“What if we need shades of gray?”

I always ask: “Any example?”

“Well, you know… it can come up.”

So I give them an example, and explain how it can be broken in to sub-questions.

“Well, yeah… but just to be on the safe side, could we have a score?”

The exercise is still going on. I haven’t seen a valid concern yet. What’s interesting is, everyone is hesitant about filtering, but no one can defend their objection.