How to choose between human coders and automated coders

June 3, 2010 | Comments Off

Photo credit: cohdra from morguefile.com

In the right hands, text analytics can turn a nightmare into a dream come true. With the increasing popularity of social media research, companies are regularly collecting thousands, and even millions, of verbatims that require analysis. On the other hand, human coders have been carrying out text analytics for decades now, and in particular, why use automated systems when humans are doing the job so well?

Here are some guidelines to help you decide which method is right for you.

1. Sample sizes – Sample size will likely be the most prominent variable in choosing a method. If you’re working with thousands or millions of verbatims, automated systems are your best friend. On the other hand, databases of several hundred verbatims are best done by hand. Remember, even if an automated system is used on a small dataset, you would still end up reading every verbatim to get a human flavor for the data. If you’re going to read every verbatim, you might as well do the analysis by hand.
2. Number of constructs – If you normally use only a small number of predefined constructs, the human method is works great. Coders can easily remember all the intricacies of the coding scheme if it is strict and well-defined. And of course, it’s fun and interesting to get your hands right in there. But, if the research plan uses coding systems with hundreds or thousands of constructs, it is simply impossible for coders to remember all of them with sufficient within or between-rater reliability. Automated systems can really ease this process.
3. New constructs – Are you open to discovering and implementing any number of new constructs? If you’re open to adding a handful of new constructs, then automated systems won’t make it much easier for you and you will be happy with your standard manual processes. But, if you want to be surprised and see where the data takes you, automated systems can provide that.
4. Timing – This is the business world, after all. Are you in a rush? Are the results required yesterday? Well, if the data is already in a clean, computerized format, an automated system will work nicely. But, if your data consists of 20 sets of handwritten notes, most of which are barely legible, you might prefer the brain power of human coders who can turn scribbles into codes without any intervening translations.
5. Coder reliability – Are you able to train and retain enough reliable coders? If you have a good team of trusty reliable coders, then keep them happy. They are valuable people who should be treated with kid gloves! But, if you’re having trouble finding those gems, an automated system will ensure that a high level of within-rater and between-rater reliability is maintained. It will even eliminate within-pair compromise.

In the end, you must choose the system that works best for you. Whether automated or human, one method with have the pros and cons that suit your specific needs. Choose well!

We're sorry, but comments are closed.

No Responses to  “How to choose between human coders and automated coders”