#MRX MRA IMRO SMR Guidelines #16: Demos and Geos

Thursday, February 17th, 2011

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #16 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.

duboix from morguefile
.
What, if any, methods are used for determining the geography associated with the data?
.
Geography and demographics are a tricky situation for social media research. Unlike survey research where people provide their age, gender, and other personal information, very few people do so in the social media space. (When was the last time you saw someone tweet something like, “I’m a 28 year old female who has a Marketing Diploma and I love poptarts.”)
.
There certainly are methods of determining the geographical and demographic characteristics of people contributing to social media but they aren’t perfect.

• How about if you notice if the website is .ca or .uk or .au? We wish that method worked but the vast majority of social media is generated in .com websites which are visited from people all around the world. And, people don’t refrain from posting on websites just because they notice it’s reflecting a different country. The internet is a wonderfully global population.
• How about IP addresses? This is another option, but again, a less than perfect option. According to my IP address, I’m sitting in the USA right now. But, I’m pretty sure I’m not!
• What about the demos that people do share? Yes, people do share their demographic information. A very, very tiny % of people share one or two pieces of information but certainly nowhere near what researchers are used to, and certainly nowhere near what is required to create a valid generalization outside of those individual people.

Because of these reasons, Conversition uses a variety of methods for understanding the demographics of social media contributors.

.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues
<

#MRX MRA IMRO SMR Guidelines #15: Validation

Tuesday, February 15th, 2011

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #15 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.

clarita from morguefile

If sentiment scoring is provided, what is the process for validating results?
.
We regularly validate both our sentiment analysis and our content analysis results.
.
Sentiment analysis is validated both at a project level (for example, just YOUR data) as well as at an overall level (a random sample across many different brands). Though there are more scientifically accurate methods (i.e., have fun reading Krippendorf!), we have chosen a simpler method because it is easily understood by both novice and experienced research users.

The two important features of our process are 1) the large sample size which ensures we do not receive spuriously high (or low) validation scores associated with a skewed selection of data points, and 2) the blinded nature of the manual scoring which ensures the researcher does not unconsciously create high validation scores.

Our process is as follows:

• Randomly select 1000 verbatims
• Manually score each verbatim as negative, neutral, or positive
• Align the automated scores to the manual scores
• Calculate the percentage of automated and manual scores that match

The process for validating constructs is similar.

• Randomly select 1000 verbatims from a specific construct
• Determine if it does or does not reflect the intended construct
• Calculate the percentage of verbatims that were correctly identified as reflecting the intended construct

At the end of each process, we then identify the weak points and adjust our systems to account for them. For instance, if we discover that “Charlie Brown has been erroneously coded under the color construct, that anomaly is corrected.

It takes a long time to validate data but if you’re going to do it, you might as well do it right.

.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues

#MRX MRA IMRO SMR Guidelines #14: Sentiment Scoring

Saturday, February 12th, 2011

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #14 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.

Alvimann from morguefile
Does the company provide sentiment scoring?

Why yes we do!

Conversition has built a proprietary sentiment scoring system that features many advantages. We’re very proud of its accuracy, particularly in relation to its application in marketing research.

• We score more than 95% of data.
• We do not throw out verbatims that can’t be scored because it turns out they are true neutrals.
• We score verbatims on a continuous scale from extremely negative to extremely positive.
• We score grammatically correct and grammatically incorrect conversations.
• We score emoticons and slang.
• We use an automated system that lets us score millions of conversations every day.
• Our system is carefully designed to meet the specific needs of market researchers.

.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues

#MRX MRA IMRO SMR Guidelines #13: Data Quality

Thursday, February 10th, 2011

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #13 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.

manicmorff from morguefile

What data quality processes are implemented in each stage of the SMR?

Ah, we love data quality! You need a sharp eye to see all of the details that we focus on in our regular data quality processes.

Once data has been collected, we apply rigorous techniques to identify and remove many different types of spam. Keyword loading and unmoderated boards are the worst of the offendors but we vigorously seek them out. That’s how we know that British Petroleum data isn’t Blood Pressure or Basis Point data.

We also apply apply rigorous processes to both our sentiment scoring and content analysis. That’s how we know that Charlie Brown isn’t a color and that ‘what a crock’ isn’t dishes. That’s how we know that “the bomb” is a good thing to say but “ftl” isn’t.

It’s not sufficient for us to use a one method fits all system. We know that doesn’t work. So we have a one method fits all process and hundreds of individualized methods, a unique one for every client. Because we know that an average level of data quality isn’t good enough. You need data quality specific to your job.

Quality is job 1. We believe it and practice it.

.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR

#MRX MRA IMRO SMR Guidelines #10: Qualifications

Wednesday, February 9th, 2011

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #10 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.

hmm360 from morguefile

What is the level of expertise and industry qualifications of social media researchers and/or SMR companies?

Social media research is a relatively new specialization in the industry of market research. It can be difficult for clients to understand the skills and expertise of their provider, most of whom have wildly different backgrounds.

Conversition’s number one skill is that of research. We are researchers who have built a research product. Our experience comes from decades of first hand experience as data collection providers, full service research providers, survey panel providers, and research client service. We have consulted with clients on research design and methodology, written surveys, analyzed surveys, written research reports, and presented reports to clients. We have conducted research on research including survey and incentive satisfaction, online  survey data quality, and parallel studies. We know the intricate details of  sampling, weighting, standards, norms, and scaling. We know research inside and out.

This extensive research background gives us unique insight into the desires and needs of researchers seeking a quantitative solution to the social media space and, as such, we have created a social media research tool that makes sense for researchers.

Conversition was built by researchers for researchers.

.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues

#MRX MRA IMRO SMR Guidelines #12: Data Gathering Process

Tuesday, February 8th, 2011

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #12 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.

clarita from morguefile
What is the process for gathering data?

Conversition specializes in the preparation of social media data for market research purposes. As such, a portion of our data is gathered by a third party which specializes in data collection from the internet. Data is gathered from millions of different websites including popular ones such as Twitter, Facebook, YouTube, and Flickr, as well as very small, personal websites that may only have a few readers.

We, and our vendors, honor all website notifications regarding crawling permissions. If a website has behind the scenes coding that says ‘do not crawl,’  then we do not crawl the website. In fact, special permissions, though not required by the website owners, have been obtained to crawl and gather data from many of the major websites.

In addition, we regularly check our processes by attempting to crawl websites that we know are password protected.

We take our data collection process very seriously and hope you appreciate that.

.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR – The Conversition Commentary
MRA IMRO Guide #2: Datasources of SMR – The Conversition Commentary
MRA IMRO Guide #3: Data Fusion and SMR – The Conversition Commentary
MRA IMRO Guide #4: Reliability of SMR – The Conversition Commentary
MRA IMRO Guide #5: Responsibilities of Social Media Data Users – The Conversition Commentary
MRA IMRO Guide #6: Social Media Research Skills – The Conversition Commentary
MRA IMRO Guide #7: Research Contributor Awareness – The Conversition Commentary
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues

#MRX MRA IMRO SMR Guidelines #11: Research Outputs

Saturday, February 5th, 2011

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #11 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.

mconnors from morguefile
What are the standard data and/or research outputs?

Conversition offers clients two different outputs. First, you may choose to access our specialized website portal which allows you to look at the data live and instantly create your own charts, tables, and EvoClouds. The portal also includes a number of pre-defined reports that allow you to quickly identify and analyze the important results in your dataset.

Second, you may choose to download datapaks which are very similar to survey datasets. After uploading the files to your preferred statistical software, whether it’s SPSS, SAS, SYSTAT,  or QUANVERT, you then have the utimate in flexibility. You can run any assortment of crosstabs, create new variables, or cut the data by your preferred date range.

Though we often consult with our clients on the preparation of their research reports, Conversition does not, as a regular offering, offer full-service research report writing. You are the master of your category.
.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues

#MRX MRA IMRO SMR Guidelines #7: Research Contributor Awareness – The Conversition Commentary

Wednesday, December 22nd, 2010

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #7 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.

slowfoot from morguefile

Are the participants aware that their user-generated content is under observation?

No.

The simple answer is no.

Not all people using social media realize that the comments and opinions they share online are under observation. It might seem counter-intuitive but as researchers of human behaviour, we need to remember that not everyone comes from the same social environment that we have. Given that you’re reading this blog, chances are you are fully immersed in the social media space and know that there is no such thing as private social media. But you’re an exception to the rule.

Human behaviours fall on a normal curve. Some people fall on the end of the curve where they understand that any sense of privacy was lost many years ago and that is probably where you sit. Other people fall on the end of the curve where they believe that the Nigerian prince truly does need financial assistance to cash in his multi-million dollar inheritance. There may be many fewer people on this side of the curve, but that doesn’t mean we can disregard them. As researchers, we are obliged to take care and protect people who may not fully appreciate all the implications of the internet. We do, after all, respect their opinions and so we should also show respect for the conditions under which they shared those opinions.

Most people, however, fall somewhere in between. They accept that the internet is public. They know that many people will read their comments on YouTube. But, they probably haven’t read the Facebook Terms of Service and they probably aren’t aware of which pieces of data are being shared widely. They probably aren’t thinking that when they obtain a password for a chatgroup about dyslexia there may be a researcher in the group who also easily obtained a password. And they certainly aren’t expecting that a completely disrespectful comment they made during extreme frustration would end up in an official report on the desk of the CEO of a global corporation with their name and twitter ID right beside it.

Research contributors are not all aware.  It’s our duty to respect that.

.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users

#MRX MRA IMRO SMR Guidelines #4 Reliability: The Conversition Commentary

Monday, November 8th, 2010

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #4 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
.

deanjenkins from morguefile
.
How reliable are SMR results?
.
Reliability and validity are topics of particular interest to Conversition because they are staples of the market research industry that are not always well understood by market researchers and users. These quality measures apply to all methods of research, including survey and focus group research, and for best results must be considered jointly.

To begin, reliability refers to results which can be replicated across numerous occasions. For instance, if several different people were to conduct the same research, each person would achieve the same results. Or, if the same study was conducted across several time periods, each study would each achieve the same results. No matter who or where or when, every time the study is performed, the results should be the same. (Unless of course you are conducting a pre/post or time series study where you are looking for different results each time.)

Because different survey panels have different incentive and recruitment strategies, high reliability across survey panels is generally not expected nor appropriate. The same follows for social media research. Every vendor has different methods of data collection and treatment, and they each follow more or less stringent standards and processes. But, within each social media research vendor, an acceptable level of reliability over time and among product categories should be achievable.

Now, remember that reliable results do not quality make. One can easily achieve the same wrong results over and over again by using the same bad survey or bad focus group or bad SMR over and over again. This brings us to validity, the significant other of reliability. Validity refers to results which reflect exactly what was intended to be measured.

When we ask people to select their favourite item from a list, we achieve validity when people read the entire list instead of choosing an item at the top of the list. We achieve validity when people tell us which political candidate they honestly plan to vote for instead of giving us the name of most socially desirable candidate. Of course, people are not robots and these validity issues pop up all the time, but, we have learned many research techniques to solve these problems.

Validity in social media research comes down to the treatment of data. Data quality measures must ensure that the right data is being selected for analysis. As such, data for Apple Computers must not include data for apple pie. Similarly, data for British Petroleum (BP) must not include data for Basis Points or Blood Pressure or Boston Pizza.

Data quality practices extend beyond simply gathering the right set of data. They must be applied to other data treatments as well including sentiment analysis and content analysis. Thus, sentiment analysis must distinguish between dope that is smoked illegally and dope that is hip, cool, and totally rad. And, content analysis must distinguish between the orange fruit and the orange color and the Planet Orange charity so lovingly built by the folks at ING.

For all of these purposes, validity can be evaluated with a fairly simple process.

1. Randomly select 1000 records from across different topics, dates, and data sources.
2. Score each record yourself.  a) What brand name does it reflect? b) What sentiment score does it deserve? c) What variable does it reflect?
3. Run the data through a second system whether it be your automated processes or a second person.
4. Match the two sets of results together.
5. Calculate the percentage of results that agree. a) What percentage of the data was actually about the intended brand name? b) What percentage of sentiment scores matched? c) What percentage of variables were correct?

.

The most important component of this validation work is that the two sets of data are scored blindly. In other words, I don’t know how you scored them, and you don’t know how I scored them.

Reliability and validity are essential components of all quality market research methods, including social media research. You can have one without the other, but without both you really have nothing. You need to ask your social media research provider how they address validity and reliability. Are these words essential components of their work? Do they have processes in place? Are those process grounded in solid research standards?

Go forth and inquire. It’s time.
.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR

How to choose between human coders and automated coders

Thursday, June 3rd, 2010

Photo credit: cohdra from morguefile.com

In the right hands, text analytics can turn a nightmare into a dream come true. With the increasing popularity of social media research, companies are regularly collecting thousands, and even millions, of verbatims that require analysis. On the other hand, human coders have been carrying out text analytics for decades now, and in particular, why use automated systems when humans are doing the job so well?

Here are some guidelines to help you decide which method is right for you.

1. Sample sizes – Sample size will likely be the most prominent variable in choosing a method. If you’re working with thousands or millions of verbatims, automated systems are your best friend. On the other hand, databases of several hundred verbatims are best done by hand. Remember, even if an automated system is used on a small dataset, you would still end up reading every verbatim to get a human flavor for the data. If you’re going to read every verbatim, you might as well do the analysis by hand.
2. Number of constructs – If you normally use only a small number of predefined constructs, the human method is works great. Coders can easily remember all the intricacies of the coding scheme if it is strict and well-defined. And of course, it’s fun and interesting to get your hands right in there. But, if the research plan uses coding systems with hundreds or thousands of constructs, it is simply impossible for coders to remember all of them with sufficient within or between-rater reliability. Automated systems can really ease this process.
3. New constructs – Are you open to discovering and implementing any number of new constructs? If you’re open to adding a handful of new constructs, then automated systems won’t make it much easier for you and you will be happy with your standard manual processes. But, if you want to be surprised and see where the data takes you, automated systems can provide that.
4. Timing – This is the business world, after all. Are you in a rush? Are the results required yesterday? Well, if the data is already in a clean, computerized format, an automated system will work nicely. But, if your data consists of 20 sets of handwritten notes, most of which are barely legible, you might prefer the brain power of human coders who can turn scribbles into codes without any intervening translations.
5. Coder reliability – Are you able to train and retain enough reliable coders? If you have a good team of trusty reliable coders, then keep them happy. They are valuable people who should be treated with kid gloves! But, if you’re having trouble finding those gems, an automated system will ensure that a high level of within-rater and between-rater reliability is maintained. It will even eliminate within-pair compromise.

In the end, you must choose the system that works best for you. Whether automated or human, one method with have the pros and cons that suit your specific needs. Choose well!