Drupal-WordPress Evaluation, Part 1

Developing Criteria

In most cases you shouldn’t have a major problem in developing functional requirements. These are the features you want to offer on the site. Since you are going to rate each requirement for how important it is, you can even use feature lists from some of the candidate solutions to help build your list. What you should do though is express the requirements in terms of functionality you want to offer on your site, not in the terms a developer or supplier might use to hype their product. (The same criteria description needs to fit all alternatives. Avoid trade names; use generic terms.) It is also useful to group these things in sub-categories under functional requirements. Providing a detail profile for users is a feature of user management. An automated system to charge membership fees might be under user/member management, or under eCommerce. Private messaging, chat facilities, forums, blogs, etc. are all functions. Each of these elements can be one line item, or they can be broken down into multiple details if that is important to you. You might have a requirement for multiple forums that are used for different purposes: forums tied to product sets, forums tied to private groups, etc. If those are your functional requirements, that is what you should write here. Remember, for each line item, you also need to show how critical it is (weight and minimum score).

Non-functional requirements are harder for most people, but don’t leave them out. Simply using your functional requirements list can sometimes lead you down a golden path to a place you don’t want to be, with a system that can’t be maintained, that costs too much to run, and doesn’t really satisfy the reason you wanted to build it to begin with. A great place to start with non-functional requirements is the ISO 9126 standards. ISO 9126  defines a quality model for software. It has no specifications about what a software product must do (that was your functional requirements). Instead it defines quality characteristics about how things get done. The quality characteristics are defined in groups. You can use the high level groups as your line items, or the ISO 9126 detail elements, or you can extend the groups or detail elements with further criteria. (Make the system work for you.) The high level characteristics (criteria) in ISO 9126 are:

• Functionality – this is an evaluation of how well the required functions are implemented. Do they adhere to standards? Are they secure? Are they accurate? Can the system inter-operate with other systems (e.g., payment processors)?
• Reliability – this is an evaluation of the overall reliability of the system. How frequent are failures due to faults in the software? Can the system run in spite of faults in the software or related components? How difficult is recovery after failure?
• Usability – How easy is the system to use by the end users? Is there a steep learning curve? Is it intuitive? How easy is it to do a designated task in the system (e.g., post a comment, join a group)?
• Efficiency – What are the response times and throughput rates for processing? How efficient is the system in terms of management of resources? Does it scale well?
• Maintainability – How easy is it to change the software? Is there proper documentation for developers? Is there a developer community? How much effort is required to diagnose problems and develop corrective changes?
• Portability – How adaptable is the software? How easily can it be moved to another environment (e.g., from shared hosting to a dedicated server)? Is it easy to install? Does it use industry standard platforms, or does it require special facilities?

You can start with this list and identify your own detail criteria within this framework, or just use the high level elements as your list. Again, the choice is yours. For each criteria, you again need to apply a weight factor that indicates how much the criteria means in your evaluation, and a minimum score if applicable.

Filling The Matrix

Filling the matrix takes time. Some elements will be easy. Some elements will be hard. Some can be done by reviewing “marketing” type materials, by using consultants, or by reviewing what other people have published on the web.

For the functional requirements, this is a place where a resource like cmsmatrix.org or cmsmatch.com can be useful. (Be careful though. Most sites are not unbiased reviews. The details are completed by the product vendors, AND the information is often out of date showing old versions.) For each criteria, for each alternative, you need to generate a score according to your chosen scoring scale. (If you are using a resource like cmsmatrix to generate scores you may want to use scores of “average” for each candidate that performs the required function as a first pass answer. You can fine tune later on a second pass if the need arises.)

For non-functional requirements it is harder, but more important to do your homework. Consultants may be helpful. Google (or your search engine of choice) is probably your friend. Search out online reviews. Let your developers look at the source code, review documentation and take part in discussion forums. In my case some of these scores will be the result of building one or more prototype system extensions.

Once you have a score for an alternative for a criteria element, the weighted score is your weight for that criteria element multiplied by the score for that alternative.  If your weight was 6 and the score was 5, the weighted score is 30 (6 * 5 = 30). If the score is below the minimum you set for the criteria, the alternative is eliminated. (If you are undergoing a costly evaluation process with limited resources and lots of alternatives, you may want to focus on elimination criteria as a way to cut your workload. On the other hand, if you have very few alternatives, you may want to continue your evaluation process and apply the elimination later in your process. If all of your options are eliminated, either you have no solution, you need to find new candidates, or you need to reconsider your criteria.)

Making A Decision

Once the detail part of the matrix is full, we can compute overall scores for the alternatives. There are multiple ways to do this, some more complex than others. The first approach is to simply sum the weighted scores for each alternative. That is a quick and easy test, but may not be the best way to decide. It can be especially bad if you have a lot of criteria in one group that one alternative is well equipped to satisfy, but other alternatives are more well-rounded, including in some key non-functional areas. An alternative with lots of ‘like to haves’ can overwhelm the system and look better than it should.

I like to develop total scores by group, or develop normalized scores for each group, by developing a weighted or non-weighted average for the category. This makes two categories with different counts of detail elements still able to be measured on a consistent basis. (At the group level you can see things like: Solution A was “average” in blogs and “above average” in forums, even though we had only 3 criteria for forums and 20 criteria for blogs.)

Once you have reduced the criteria to a reasonable count you can:

• Compute a total based on major group scores (roll up each sub-group to create totals by group at the next higher levels)
• Computed a weighted total where some categories may be more important than others to your overall goals (Give a percent weighting factor to each sub-group. Higher values to more important groups. Make sure the sum of the weighting factors for a group add up to 100. Multiply each group score by the weighting factor for the group.)
• Create bar charts by alternative or by criteria to show how the alternatives compare or how well the criteria are covered.
• Create a radar-chart showing how the alternatives stack up against each other in terms of being well rounded and meeting the full set of criteria.
• Measure functionality against costs to decide whether you really want to pay that much for what you think you want to implement

Creating these types of analyses is where the spreadsheet will be very helpful.

In the end, you’ll still make a decision based on human factors. But now you should have the basis for an informed, intelligent decision that goes a long ways to getting your project done. If you have a management team, or a stakeholder group, you will have developed the materials you need to have an informed discussion about how the project should go forward. And you’ll all understand along the way, and after the install, what trade-offs you made, and why.

To Be Continued …

As I continue my evaluation I will discuss how I am filling out the criteria part of the decision matrix. I also plan to include some discussion on particular criteria that are of more concern to me, and the research or experimentation I’m doing to arrive at an evaluation.

Good luck on your project! If you need help searching for other opinions and information, click on the resources I’ve identified in the next section. I have no ax to grind. They are simply links I’ve collected on my journey.

Resources

Some resources you may wish to look at in determining criteria or helping to come up with ratings include the following, in no particular order:

• http://poplarware.com/articles/drupal_vs_wordpress
• http://davidbressler.com/2011/01/04/an-investigation-of-buddypress-vs-drupal-commons/
• http://friendly-machine.com/posts/2011/choosing-content-management-system
• http://friendly-machine.com/posts/2011/wordpress-vs-drupal
• Drupal vs BuddyPress as a Social Networking site – the base post here is old, but the dialog still has some useful information
• http://thereisamoduleforthat.com/content/community-practice-network-drupalorg
• http://www.ibm.com/developerworks/ibm/library/i-osource1/ – dated, but informative
• http://wpmu.org/open-source-social-networking-matchup-buddypress-vs-drupal-commons/
• http://wpmu.org/important-factors-to-consider-in-the-wordpress-vs-drupal-debate/
• http://kevinjohngallagher.com/2012/01/wordpress-has-left-the-building/