#MRX MRA IMRO SMR Guidelines #5 Responsibilities: The Conversition Commentary

Thursday, December 16th, 2010

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #5 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
.

Alvimann from morguefile
.
Within businesses and organizations, how will SMR activities be tracked and aggregated, and whose responsibility is it to handle each of those functions?

This seemingly innocuous question has huge punch behind it. It first requires one to distinguish between social media research and other uses of social media data such as customer relationship management and public relations.

First, each use of social media data requires different skills. Those who intend to use it for social media research need to be knowledgeable about and skilled in such things as sampling, weighting, standards, norms, and scaling. These are industry specific skills that are learned over time through education and practice, and cannot just be picked up casually. As such, if you plan to use social media data for research purposes, make sure you have a researcher on hand who can filter out the noise of monitoring and filter in the science of research.

Second, users taking a research point of view towards the data must understand the ethics associated with that use of the data. Remember, as researchers, we have agreed to respect requests for privacy from our research contributors whether they are participating in surveys, focus groups, or social media research. Though the gathering of social media data brings with it links to an individual’s facebook, twitter, or youtube page, this does not give researchers the unequivocal right to contact and engage with people. This perspective is completely different than that taken by the CRM and PR business so there is definitely room for error and misunderstanding here. Fortunately, numerous professional groups are attempting to work through this issue including the MRA IMRO team.

In the end, as long as you remember to involve people with the right skills and solid ethical considerations, you will be on the right path.
.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR

#MRX MRA IMRO SMR Guidelines #4 Reliability: The Conversition Commentary

Monday, November 8th, 2010

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #4 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
.

deanjenkins from morguefile
.
How reliable are SMR results?
.
Reliability and validity are topics of particular interest to Conversition because they are staples of the market research industry that are not always well understood by market researchers and users. These quality measures apply to all methods of research, including survey and focus group research, and for best results must be considered jointly.

To begin, reliability refers to results which can be replicated across numerous occasions. For instance, if several different people were to conduct the same research, each person would achieve the same results. Or, if the same study was conducted across several time periods, each study would each achieve the same results. No matter who or where or when, every time the study is performed, the results should be the same. (Unless of course you are conducting a pre/post or time series study where you are looking for different results each time.)

Because different survey panels have different incentive and recruitment strategies, high reliability across survey panels is generally not expected nor appropriate. The same follows for social media research. Every vendor has different methods of data collection and treatment, and they each follow more or less stringent standards and processes. But, within each social media research vendor, an acceptable level of reliability over time and among product categories should be achievable.

Now, remember that reliable results do not quality make. One can easily achieve the same wrong results over and over again by using the same bad survey or bad focus group or bad SMR over and over again. This brings us to validity, the significant other of reliability. Validity refers to results which reflect exactly what was intended to be measured.

When we ask people to select their favourite item from a list, we achieve validity when people read the entire list instead of choosing an item at the top of the list. We achieve validity when people tell us which political candidate they honestly plan to vote for instead of giving us the name of most socially desirable candidate. Of course, people are not robots and these validity issues pop up all the time, but, we have learned many research techniques to solve these problems.

Validity in social media research comes down to the treatment of data. Data quality measures must ensure that the right data is being selected for analysis. As such, data for Apple Computers must not include data for apple pie. Similarly, data for British Petroleum (BP) must not include data for Basis Points or Blood Pressure or Boston Pizza.

Data quality practices extend beyond simply gathering the right set of data. They must be applied to other data treatments as well including sentiment analysis and content analysis. Thus, sentiment analysis must distinguish between dope that is smoked illegally and dope that is hip, cool, and totally rad. And, content analysis must distinguish between the orange fruit and the orange color and the Planet Orange charity so lovingly built by the folks at ING.

For all of these purposes, validity can be evaluated with a fairly simple process.

1. Randomly select 1000 records from across different topics, dates, and data sources.
2. Score each record yourself.  a) What brand name does it reflect? b) What sentiment score does it deserve? c) What variable does it reflect?
3. Run the data through a second system whether it be your automated processes or a second person.
4. Match the two sets of results together.
5. Calculate the percentage of results that agree. a) What percentage of the data was actually about the intended brand name? b) What percentage of sentiment scores matched? c) What percentage of variables were correct?

.

The most important component of this validation work is that the two sets of data are scored blindly. In other words, I don’t know how you scored them, and you don’t know how I scored them.

Reliability and validity are essential components of all quality market research methods, including social media research. You can have one without the other, but without both you really have nothing. You need to ask your social media research provider how they address validity and reliability. Are these words essential components of their work? Do they have processes in place? Are those process grounded in solid research standards?

Go forth and inquire. It’s time.
.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR

#MRX MRA IMRO SMR Guidelines #3 Data Fusion: The Conversition Commentary

Thursday, November 4th, 2010

MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #3 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
.

jdurham from morguefile
.
How does SMR interact with other forms of traditional and non-traditional research, including online, offline, in-person, and qualitative and quantitative?
.
This is one of our favorite topics at Conversition. We love to show clients how data fusion of results from traditional market research and social media research generates more powerful results. Note that we say fusing research, not replacing research because there is just no replacement for surveys and focus groups.

One of the first questions many clients have is whether social media results are the same as survey results. Given that we are market researchers with years of first hand experience transitioning offline studies to online versions, we completely understand how new methods affect results. But, social media research isn’t intended to duplicate survey research – it builds and broadens the entire set of results.

For those new to social media research, one of the best ways to get a better understanding of SMR is to run both a survey and SMR at the same time. You can take advantage of the specificity that surveys provide, as well as the descriptive narrative and breadth of topics that social market research provides. Pay attention to the types of survey questions that can and can’t be mirrored in the SMR. Pay attention the the types of survey questions that explode into new questions and serendipitous results in the SMR.

Another interesting way to combine the best of both worlds is to use social media research to inform the contents of a survey. When surveys become too long, we must choose something to leave out and long lists are a great place to start. One way to shorten these long lists is to gather thousands of records from SMR, cull out the top brands or flavours or SKUs, and include just the top items in the survey.

Social media research also gives researchers a much broader and more open perspective of how users communicate about a brand. Once the basic usage and attitude information has been gathered, social media research can be used to flush out those results. The most obvious way is to discover how people talk about brands to gather psychographic information – what word choices do they make, do they use slang, do they use correct grammar.

Additionally, social media allows for the discovery of creativity – how do users incorporate photos, videos, and music, into their approach to a brand.

Social media research is also a great way to conduct pre-post research for new products. Where focus groups allow researchers to observe people using brand new products before those products go to market, social media research allows you to evaluate product instantly as they are introduced to the market.

These are just a few of the ways that traditional research and social media research can be incorporated into a multi-mode method of market research. Try it. You’ll like it.
.
.
Related links

MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR

MRA IMRO Social Media Research Guide has arrived!

Wednesday, October 27th, 2010

The MRA has just announced the release of the much anticipated document. With contributions from Annie Pettit and other market research experts, it was a journey of a couple of months to get it to this point and it will continue to evolve in future iterations. You can download the document here.

We highly encourage you to download and review the document, but in the meantime, here are a couple points of particular note:

• Best practices call for researchers to respect the coded crawling terms of every Web site they visit. Where Web sites are coded to indicate that crawling is not permitted, those Web sites should not be crawled even if it is technically possible. Researchers must not join Web sites under the pretense of being a member so that they then have access to crawl a Web site that prohibits such crawling otherwise – this condition holds for both automated and manual crawling. Where researchers do join groups, they must immediately make it explicit that they are there for the purposes of marketing research.
• Validity refers to the degree to which results reflect truth or reality while reliability reflects the degree to which results can be replicated if someone else were to conduct a similar study. Because research suppliers have different methods, standards of quality, and processing rules, research consumers must conduct their own validity and reliability analysis of any potential supplier to ensure the quality of work is sufficient. As with all types of marketing research, the validity and reliability of social media research varies greatly.

We’ d love to hear your thoughts.

IMRO guidelines for research based on social media data

Tuesday, July 20th, 2010

A committee has been formed and brainstorming is underway!

With the realization that research using social media data is quickly gaining in popularity, the IMRO division within the MRA has put together a committee to help define and guide users and providers in this space. Over the next couple of months, a team of 17 or so researchers, including Annie Pettit (@LoveStats) of Conversition Strategies, and led by Jim Longo (@LongoMR) of iTracks, will be putting their heads together to build a short document that will be made publicly available.
.
The committee will address a number of issues including what is social media research, how should users evaluate it, what are the essential components for consideration, as well as other key topics. The goal is not to provide rules that must be followed but rather to ensure users and providers are aware of all of the issues when considering and conducting social media research.
.
Your opinions are important. Share your thoughts, questions and concerns with us by leaving comments here, emailing Annie (LinkIn) or by emailing Jim Longo.