Posts Tagged ‘validity’
Friday, June 3rd, 2011
Paul Golden recently published an article in research live about the controversy between automated and manual sentiment analysis. An excerpt is below while the full article can be read here.
Those who bemoan the impact of email, text messaging and Facebook on our ability to write tend to forget one thing: that the people who have grown up with these tools write far more than any generation before them. They might not have much respect for the conventional rules of writing, but they’re making the language their own, and adapting it to different formats and audiences.
In the process they’re producing vast amounts of data, which marketers are keen to make sense of. That means understanding the meaning and sentiment of text – and numerous tech companies are queuing up to help researchers do this, each with grander claims than the last about their tools’ ability to accurately interpret and categorise millions of online comments. These text analytics tools are adept at picking up brand references from the massive volumes of online communication generated every day. But opinion is widely divided on their ability to determine whether a comment is positive, negative or neither.
….
“these approaches have flaws, but then so do people, says Annie Pettit, head of research at Conversition. “The important thing to remember is that we don’t care if individual messages are coded correctly. We care that the aggregate coding across large numbers of messages validates well. We aren’t trying to prove whether I hate or love Pop-Tarts, we are trying to prove whether a million people hate or love Pop-Tarts. A reading that is 70% accurate is actually a very accurate reading.
If humans can score hundreds of messages and validate at 85%, and computers can score millions of messages and validate at 70%, then computers are a valuable option”
Read the rest of the article here.
Comments Off
Category conversition | Tags: Tags: annie pettit, conversition, erewards, mrx, research now, sentiment analysis, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Tuesday, May 17th, 2011
As a budding field, social media research is a magnet for questions from both experts and novices alike. People are curious about the processes and methodologies used to accomplish the various aspects of the research. Some of the more common questions we field on a regular basis are as follows:
- What sentiment analysis system do you use?
- How do you carry out the text analysis process?
- What is your method for identifying and eliminating spam?
In fact, each of these questions is one and the same. They have nothing to do with sentiment, text analysis, or spam. They have nothing to do with processes or methods or systems. In fact, they have everything to do with validity.
Validity refers to truth. Is the sentiment scored accurately? Is the text analyzed accurately? Is the spam identified accurately? Is the entire process valid? Among all the pieces of the puzzle, this is the one question that must be answered.
Unfortunately, there is no single method that automatically identifies a sentiment analysis, text analysis, or spam detection system as being the most valid one. You simply have to evaluate a large, representative sample of data and determine the answer for yourself. Are your results valid?
.
Also see
The Sharks and Icebergs of Social Media Measurement
5 Ways to Fool An Automated Sentiment Analysis System
A Formula for Perfect Sentiment Analysis
Comments Off
Category conversition | Tags: Tags: #ngmr, annie pettit, content analysis, conversition, focus groups, lovestats, market research, marketresearch, mrx, newmr, sentiment analysis, smr, social media analytics, social media marketing, social media monitoring, social media plan, social media research, social media strategy, surveys, tessie ting, tessietweets, text analysis, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Thursday, February 17th, 2011
MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #16 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
duboix from morguefile
.
What, if any, methods are used for determining the geography associated with the data?
.
Geography and demographics are a tricky situation for social media research. Unlike survey research where people provide their age, gender, and other personal information, very few people do so in the social media space. (When was the last time you saw someone tweet something like, “I’m a 28 year old female who has a Marketing Diploma and I love poptarts.”)
.
There certainly are methods of determining the geographical and demographic characteristics of people contributing to social media but they aren’t perfect.
- How about if you notice if the website is .ca or .uk or .au? We wish that method worked but the vast majority of social media is generated in .com websites which are visited from people all around the world. And, people don’t refrain from posting on websites just because they notice it’s reflecting a different country. The internet is a wonderfully global population.
- How about IP addresses? This is another option, but again, a less than perfect option. According to my IP address, I’m sitting in the USA right now. But, I’m pretty sure I’m not!
- What about the demos that people do share? Yes, people do share their demographic information. A very, very tiny % of people share one or two pieces of information but certainly nowhere near what researchers are used to, and certainly nowhere near what is required to create a valid generalization outside of those individual people.
Because of these reasons, Conversition uses a variety of methods for understanding the demographics of social media contributors.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues
<
Comments Off
Category conversition | Tags: Tags: annie pettit, citation, citing, content analysis, conversition, focus groups, guide, guidelines, imro, lovestats, market research, marketresearch, mra, mrx, navigating, newmr, references, reliability, sentiment analysis, smr, social media analytics, social media marketing, social media monitor, social media monitoring, social media plan, social media research, social media strategy, surveys, tessie ting, tessietweets, text analysis, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Tuesday, February 15th, 2011
MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #15 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
clarita from morguefile
If sentiment scoring is provided, what is the process for validating results?
.
We regularly validate both our sentiment analysis and our content analysis results.
.
Sentiment analysis is validated both at a project level (for example, just YOUR data) as well as at an overall level (a random sample across many different brands). Though there are more scientifically accurate methods (i.e., have fun reading Krippendorf!), we have chosen a simpler method because it is easily understood by both novice and experienced research users.
The two important features of our process are 1) the large sample size which ensures we do not receive spuriously high (or low) validation scores associated with a skewed selection of data points, and 2) the blinded nature of the manual scoring which ensures the researcher does not unconsciously create high validation scores.
Our process is as follows:
- Randomly select 1000 verbatims
- Manually score each verbatim as negative, neutral, or positive
- Align the automated scores to the manual scores
- Calculate the percentage of automated and manual scores that match
The process for validating constructs is similar.
- Randomly select 1000 verbatims from a specific construct
- Determine if it does or does not reflect the intended construct
- Calculate the percentage of verbatims that were correctly identified as reflecting the intended construct
At the end of each process, we then identify the weak points and adjust our systems to account for them. For instance, if we discover that “Charlie Brown has been erroneously coded under the color construct, that anomaly is corrected.
It takes a long time to validate data but if you’re going to do it, you might as well do it right.
.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues
Comments Off
Category conversition | Tags: Tags: annie pettit, citation, citing, content analysis, conversition, focus groups, guide, guidelines, imro, lovestats, market research, marketresearch, mra, mrx, navigating, newmr, references, reliability, sentiment analysis, smr, social media analytics, social media marketing, social media monitoring, social media plan, social media research, social media strategy, surveys, tessie ting, tessietweets, text analysis, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Saturday, February 12th, 2011
MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #14 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
Alvimann from morguefile
Does the company provide sentiment scoring?
Why yes we do!
Conversition has built a proprietary sentiment scoring system that features many advantages. We’re very proud of its accuracy, particularly in relation to its application in marketing research.
- We score more than 95% of data.
- We do not throw out verbatims that can’t be scored because it turns out they are true neutrals.
- We score verbatims on a continuous scale from extremely negative to extremely positive.
- We score grammatically correct and grammatically incorrect conversations.
- We score emoticons and slang.
- We use an automated system that lets us score millions of conversations every day.
- Our system is carefully designed to meet the specific needs of market researchers.
.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues
Comments Off
Category conversition | Tags: Tags: annie pettit, citation, citing, conversition, guide, guidelines, imro, lovestats, market research, mra, mrx, navigating, references, reliability, smr, social media analytics, social media monitor, social media research, tessie ting, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Thursday, February 10th, 2011
MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #13 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
manicmorff from morguefile
What data quality processes are implemented in each stage of the SMR?
Ah, we love data quality! You need a sharp eye to see all of the details that we focus on in our regular data quality processes.
Once data has been collected, we apply rigorous techniques to identify and remove many different types of spam. Keyword loading and unmoderated boards are the worst of the offendors but we vigorously seek them out. That’s how we know that British Petroleum data isn’t Blood Pressure or Basis Point data.
We also apply apply rigorous processes to both our sentiment scoring and content analysis. That’s how we know that Charlie Brown isn’t a color and that ‘what a crock’ isn’t dishes. That’s how we know that “the bomb” is a good thing to say but “ftl” isn’t.
It’s not sufficient for us to use a one method fits all system. We know that doesn’t work. So we have a one method fits all process and hundreds of individualized methods, a unique one for every client. Because we know that an average level of data quality isn’t good enough. You need data quality specific to your job.
Quality is job 1. We believe it and practice it.
.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
Comments Off
Category conversition | Tags: Tags: annie pettit, citation, citing, conversition, guide, guidelines, imro, lovestats, market research, mra, mrx, navigating, references, reliability, smr, social media analytics, social media monitor, social media research, tessie ting, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Wednesday, February 9th, 2011
MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #10 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
hmm360 from morguefile
What is the level of expertise and industry qualifications of social media researchers and/or SMR companies?
Social media research is a relatively new specialization in the industry of market research. It can be difficult for clients to understand the skills and expertise of their provider, most of whom have wildly different backgrounds.
Conversition’s number one skill is that of research. We are researchers who have built a research product. Our experience comes from decades of first hand experience as data collection providers, full service research providers, survey panel providers, and research client service. We have consulted with clients on research design and methodology, written surveys, analyzed surveys, written research reports, and presented reports to clients. We have conducted research on research including survey and incentive satisfaction, online survey data quality, and parallel studies. We know the intricate details of sampling, weighting, standards, norms, and scaling. We know research inside and out.
This extensive research background gives us unique insight into the desires and needs of researchers seeking a quantitative solution to the social media space and, as such, we have created a social media research tool that makes sense for researchers.
Conversition was built by researchers for researchers.
.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues
Comments Off
Category conversition | Tags: Tags: annie pettit, citation, citing, content analysis, conversition, focus groups, guide, guidelines, imro, lovestats, market research, marketresearch, mra, mrx, navigating, newmr, references, reliability, sentiment analysis, smr, social media analytics, social media marketing, social media monitoring, social media plan, social media research, social media strategy, surveys, tessie ting, tessietweets, text analysis, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Tuesday, February 8th, 2011
MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #12 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
clarita from morguefile
What is the process for gathering data?
Conversition specializes in the preparation of social media data for market research purposes. As such, a portion of our data is gathered by a third party which specializes in data collection from the internet. Data is gathered from millions of different websites including popular ones such as Twitter, Facebook, YouTube, and Flickr, as well as very small, personal websites that may only have a few readers.
We, and our vendors, honor all website notifications regarding crawling permissions. If a website has behind the scenes coding that says ‘do not crawl,’ then we do not crawl the website. In fact, special permissions, though not required by the website owners, have been obtained to crawl and gather data from many of the major websites.
In addition, we regularly check our processes by attempting to crawl websites that we know are password protected.
We take our data collection process very seriously and hope you appreciate that.
.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR – The Conversition Commentary
MRA IMRO Guide #2: Datasources of SMR – The Conversition Commentary
MRA IMRO Guide #3: Data Fusion and SMR – The Conversition Commentary
MRA IMRO Guide #4: Reliability of SMR – The Conversition Commentary
MRA IMRO Guide #5: Responsibilities of Social Media Data Users – The Conversition Commentary
MRA IMRO Guide #6: Social Media Research Skills – The Conversition Commentary
MRA IMRO Guide #7: Research Contributor Awareness – The Conversition Commentary
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues
Comments Off
Category conversition | Tags: Tags: annie pettit, citation, citing, content analysis, conversition, focus groups, guide, guidelines, imro, lovestats, market research, marketresearch, mra, mrx, navigating, newmr, references, reliability, sentiment analysis, smr, social media analytics, social media marketing, social media monitor, social media monitoring, social media plan, social media research, social media strategy, surveys, tessie ting, tessietweets, text analysis, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Saturday, February 5th, 2011
MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #11 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
mconnors from morguefile
What are the standard data and/or research outputs?
Conversition offers clients two different outputs. First, you may choose to access our specialized website portal which allows you to look at the data live and instantly create your own charts, tables, and EvoClouds. The portal also includes a number of pre-defined reports that allow you to quickly identify and analyze the important results in your dataset.
Second, you may choose to download datapaks which are very similar to survey datasets. After uploading the files to your preferred statistical software, whether it’s SPSS, SAS, SYSTAT, or QUANVERT, you then have the utimate in flexibility. You can run any assortment of crosstabs, create new variables, or cut the data by your preferred date range.
Though we often consult with our clients on the preparation of their research reports, Conversition does not, as a regular offering, offer full-service research report writing. You are the master of your category.
.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
MRA IMRO Guide #6: Social Media Research Skills
MRA IMRO Guide #7: Research Contributor Awareness
MRA IMRO Guide #8: Citing Reference
MRA IMRO Guide #9: Legal Issues
Comments Off
Category conversition | Tags: Tags: annie pettit, citation, citing, content analysis, conversition, focus groups, guide, guidelines, imro, lovestats, market research, marketresearch, mra, mrx, navigating, newmr, references, reliability, sentiment analysis, smr, social media analytics, social media marketing, social media monitoring, social media plan, social media research, social media strategy, surveys, tessie ting, tessietweets, text analysis, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
Wednesday, December 22nd, 2010
MRA recently released version 1 of the MRA/IMRO Guide to the Top 16 Social Media Research Questions, a tool to help newcomers and vendors communicate with each other about this new datasource and method. Conversition was a key contributor to this document which is now available on the MRA website.
.
This blog is #7 in a series of 16, each one addressing Conversition’s viewpoint on one of the items in the guidelines. We welcome your questions and comments, and look forward to further discussions on this exciting new trend in the market research industry.
slowfoot from morguefile
Are the participants aware that their user-generated content is under observation?
No.
The simple answer is no.
Not all people using social media realize that the comments and opinions they share online are under observation. It might seem counter-intuitive but as researchers of human behaviour, we need to remember that not everyone comes from the same social environment that we have. Given that you’re reading this blog, chances are you are fully immersed in the social media space and know that there is no such thing as private social media. But you’re an exception to the rule.
Human behaviours fall on a normal curve. Some people fall on the end of the curve where they understand that any sense of privacy was lost many years ago and that is probably where you sit. Other people fall on the end of the curve where they believe that the Nigerian prince truly does need financial assistance to cash in his multi-million dollar inheritance. There may be many fewer people on this side of the curve, but that doesn’t mean we can disregard them. As researchers, we are obliged to take care and protect people who may not fully appreciate all the implications of the internet. We do, after all, respect their opinions and so we should also show respect for the conditions under which they shared those opinions.
Most people, however, fall somewhere in between. They accept that the internet is public. They know that many people will read their comments on YouTube. But, they probably haven’t read the Facebook Terms of Service and they probably aren’t aware of which pieces of data are being shared widely. They probably aren’t thinking that when they obtain a password for a chatgroup about dyslexia there may be a researcher in the group who also easily obtained a password. And they certainly aren’t expecting that a completely disrespectful comment they made during extreme frustration would end up in an official report on the desk of the CEO of a global corporation with their name and twitter ID right beside it.
Research contributors are not all aware. It’s our duty to respect that.
.
.
Related links
MRA IMRO Guide #1: Advantages and Disadvantages of SMR
MRA IMRO Guide #2: Datasources of SMR
MRA IMRO Guide #3: Data Fusion and SMR
MRA IMRO Guide #4: Reliability of SMR
MRA IMRO Guide #5: Responsibilities of Social Media Data Users
Comments Off
Category conversition | Tags: Tags: annie pettit, conversition, guide, guidelines, imro, lovestats, market research, mra, mrx, navigating, reliability, smr, social media analytics, social media monitor, social media research, tessie ting, tessietweets, validity,
Social Networks : Technorati, Stumble it!, Digg, de.licio.us, Yahoo, reddit, Blogmarks, Google, Magnolia.
