Drupal-WordPress Evaluation, Part 1

Magic Happens Here

Think about what you are trying to do, and evaluate your options based on your own personalized criteria and ranking. Don’t just buy what any one person, or the crowd, is saying is the “best” solution. There is no such thing as a universal elixir for everything that ails everyone, one size does not fit all!  Most of that pitch you’ve probably heard is marketing BS, and a lot of the rest is people who like to hear themselves talk. You have to think about what you (or your company) want to do and how the solution meets your needs. (And I’m sure I’m not the first one who told you that you should eat your vegetables.)

Okay, so how do you do that. And how do you do that in the context of making a choice between Drupal, WordPress, Joomla and The World’s Most Fabulous CMS? Good question.

Generally the best way that I am aware of to do this is with a decision matrix. Depending on who you need to convince, how critical the outcome is, what resources and talents can be brought to bear, creating a decision matrix can be more formal and methodical, or less formal. Either way – you need to balance the effort with the importance of the results. If a wrong decision is going to waste critical dollars and time, and possibly jeopardize your career or your mission, then it needs to be done more carefully and methodically, with results that can be communicated and substantiated. If you are doing it for yourself, or as a joint exercise with the other stakeholders on your project, you can be less formal. Either way, you should save the documentation of your analysis so that you can check the decision as you work the project to completion. That documentation may also prove valuable as you are getting used to your new system, to go back and understand what trade-offs you made and why you made them.

Talk to a process consultant and they will sell you some religion about how you need to do the process. In the end, there is a lot of leeway as long as you are honest within your process. What I’m going to describe is a process that is more methodical, but it is also flexible. I’ll comment on how it can be done in a less formal way as we proceed. The final technique you use is up to you, and whatever in-house process your management team may demand. I was formally trained in two different processes at different times in my career. The first was Kepner Tregoe Decision Analysis. This was taught as a part of the larger Kepner Tregoe problem solving and decision making collection of processes. The second technique is Pugh’s Matrix. This is taught as part of Six Sigma, a family of business management processes originally developed by Motorola. You can use either of those processes in its pure form, or you can use some variant of either, or both. The process I’ll discuss is a little of both, or can be viewed as a variation on either one (except by an orthodox process priest).

A decision matrix is a table with rows and columns. You can use a software spreadsheet, or you could do this by drawing out a table on a piece of paper. (The back of a napkin has been claimed to be a good place to start in certain circumstances.) If you are going to need to present your results, you probably want to use a spreadsheet, sooner or later. If you are going to have a lot of criteria or a lot of alternatives, you want to use a spreadsheet sooner. Use the rows to define criteria and use the columns to define alternatives.

In my case the alternatives will be Drupal and BuddyPress. (I’m using BuddyPress and not WordPress because I’m specifically evaluating the configuration that is going to be used for a community web site.) If I wanted to expand my search I could include Joomla and The World’s Most Fabulous CMS, but it would take more effort to complete those columns on the evaluation, and I’ve already narrowed my choices. In the figure below I’m showing how you should set up the columns of the decision matrix. For the sake of argument I’ve included extra columns for Joomla on the sample in the figure, but I have no intention now of filling in those columns. (You can have as many alternatives as you want. You just need the resources to do the evaluation.)

The first three columns define the evaluation criteria. The first column will contain a text description of the criteria. (For complex criteria you may want a reference document with a longer write-up, maybe a paragraph or an extended bullet-list of items.) The second column will contain a weighting factor defining how important that criteria element is. The third column is a filter that defines the smallest score an alternative must have to stay in contention. Following that are the alternatives. Each alternative has two columns: a score column and a weighted score column.

Scoring System

You can create any kind of scoring system you want. You just need to be consistent.

Pugh, the way I was taught, uses a “better, same, worse” system to measure alternatives against a baseline solution. It is nice because you are always doing one-to-one comparisons against a common baseline as you go through your evaluation. If the alternative is the same as the baseline, it rates a “same” rating. If it is better, it rates a “better” rating, etc. I like to change that to include “much better” and “much worse” as scoring options. You just want to be able to translate the “grades” into numeric scores; they can be positive or negative. In the case of grading from “much worse” to “much better” the scores might be: much worse (-2), worse (-1), same (0), better (+1), much better (+2).  Or you could use a -3, -1, 0, +1, +3 scale, or a -5, -2, 0, +2, +5 scale. Your choice.

If you don’t have a baseline solution you could create a grading system that includes a range from “non-existing” to “excellent” along the following pattern: non-existing (0), barely adequate (1), average (3), good (5), very good (7), meets my dreams (9). Or, you could rate things on a scale of 1 to 100. Again, it’s your choice. Be consistent. See what the final decision makers in your organization might like and what they feel comfortable with.

Criteria

This is the most critical part of the process. If you are honest, you should really do this before you have done any real evaluation of the alternatives, but that probably isn’t realistic. Even if you’ve looked at and played with the alternatives, you can still do a good job by following the method we are talking about here. That’s why you want to follow the process. So you can prove to yourself and your stakeholders that an honest evaluation was conducted. (If something doesn’t work out later, you’ll be able to see where the decision was misguided and you’ll know what to correct. Like fill in some missing criteria elements, or making something mandatory.)

I’m going to spend most of my time in this series defining and talking about criteria. This post is an overview of what we’ll do. We’re going to come up with criteria in groups. If you want, you can just list the groups. For an  evaluation that is very important to your mission, you will want to detail the groups. If it’s very, very important, you probably want to be very detailed. We will list the criteria in rows under the Criteria Description column. Then we will identify a Weight Factor for each criteria. We can also identify a Minimum Score for each criteria that is capable of acting as a “no-go” kill factor to drop an alternative from contention.

Categories

The first thing to do is to find the major criteria categories. These are our requirements for our solution. I like to start by defining two major groups:

1. Functional Requirements
2. Non-functional Requirements

This is a classical split that is well documented in the systems engineering world and you should be able to find a lot of supporting documentation on the web in terms of this categorization. The functional requirements are statements of the functional things the system must do to meet goals. For example: support blogs, offer a forum, support event scheduling, keep up membership rolls. The non-functional requirements tend to be constraints on the system or specify qualitative judgments about how the system goes about delivering its functions. For example: performance and capacity constraints, security, ease of use. We’ll go deeper into this later.

Additional first level groups might include:

• Costs
• Risks

Costs can include acquisition or development costs, operational costs, maintenance costs, etc. Risks may include such things as determining the risk that a particular critical function may not be technically or economically feasible, risks of missing installation dates, or lack of contingency plan in the event of severe problems.

Weight

The weight factor associated with the requirements is a key piece of data. The weight factor determines how much each requirement matters compared to the other requirements. There are many ways to come up with weight factors. You can poll a group of stakeholders and combine their scores using some averaging technique. You can also use scientific methods to derive normalized scoring systems. You can make the process normalized and hierarchical, or you can make it simple and linear. The choice is yours. You may want to start with something simple up front and play with both simple and more advanced methods of combining the results at the end – see if the outcome changes. We’ll more talk about that later too. For now, one simple weighting scheme is this:

• Use a scale of 1 to 10
• If a criteria is a “must have” – if it defines something that is fundamental to the purpose of the web site – give the criteria a 8, 9 or 10 depending on how important it is. These elements represent things that will be expected by your customers (your web site visitors) as part of the site. These same elements should also have minimum score values assigned to them. (You are using multiple values here because all “must have” elements may not be created equal. You could just as easily give all”must have” elements the same value. Again, it’s your choice.)
• If the criteria is a “should have” – something that you’d really like, but if it was omitted it wouldn’t cripple the ability of the web site to fulfill its mission  – give it a 5, 6 or 7.
• If the criteria is a “could have” – something that would add value to the site such that the more of these that you can do the happier your customers will be – give it a 3 or 4.
• If the criteria is a “want to have” – something that at some point should be part of the site, but it could be delayed for one or more releases – give it a 1 or 2.

Use the same scale and evaluation system for both functional and non-functional requirements. Costs and risks might have different measures (typically monetary). In order to perform an honest (un-biased) evaluation, you should be applying these weight factors before you check the alternatives.

For each criteria you determined to be a “must have” element, you should also give a minimum score. The minimum score is determined according to whatever you decided to use as your scoring system. Depending on your need, you might mandate that a new system provide enhancements over a current system on particular criteria; or you might be able to accept some slight loss of function. You might want all “must have” items to simply be present (not non-existing, aka ‘barely adequate’), or you might want one or all to be ‘above average.’ Your choice, it’s your system. At least your sponsors will understand the way the recommendation was made; and, if you are using a spreadsheet, you can quickly re-compute the scores later using a different value. The point to the minimum score is that any alternative that scores below the minimum score on any criteria is ruled out of the running – no matter how high their score may be on other criteria. For any criteria you do not want to have this level of impact on your decision, you should set the minimum score to the lowest score on your scoring system (or set it to -999 or some other low end value).