Sentiment Analysis 101

July 17, 2009 | No Comments

tweetfeel gives you a taste of it, but really, what is sentiment analysis all about?

At it’s most basic level, sentiment analysis involves reviewing messages or conversations and evaluating the writer’s opinion towards the topic. For instance, someone who tweets a message such as “I like Chuck Norris” is telling people they have a positive opinion towards Chuck Norris. On the other hand, someone who writes “Chuck Norris sucks” clearly has a negative opinion. After assembling all of the messages that mention Chuck Norris, one can easily bucket them into messages with positive opinions and messages with negative opinions.

But, the easy part isn’t so easy. First, one needs to determine which sentiments are positive or negative. Obviously, we’re talking automated sentiment analysis so we need some solid indicators for positive opinions such as words like happy, love, or delightful. Solid indicators for negative opinions would be words such as hate, stupid, or ugly. Simply coming up with that list is difficult enough, but some words just aren’t so easy to assign to buckets. For instance, is “Way to go” positive or negative? People often use this phrase in a positive way but in recent years, it has become a very sarcastic remark that one uses in a negative fashion. The written word is full of words and phrases that have contradictory, ambiguous, or sarcastic meanings. Humans can only catch about 85% of those which means it’s pretty much impossible for an automated process to catch all of them either.

Another problem with bucketing messages is that people don’t think linearly. If I say “I love Chuck Norris and football sucks,” it’s clear to people that I’ve messaged two distinct opinions about two distinct topics. Once you start getting into more complicated grammar though, it can become impossible to tell which topic was rated which way. Automated evaluations of the message have a much harder time differentiating the two. It’s a topic of great interest to academics and eventually, we’ll figure it out.

In the end though, it’s not about individual messages. It’s not about me and what I have to say. It doesn’t matter that your uncle Bob is always wrong and that your Aunt Mary doesn’t know who Chuck Norris is. It doesn’t matter that 5% or 10% of the messages are in the wrong bucket. What matters is the collective wisdom, the wisdom that comes from large sample sizes. When you average opinions across hundreds or thousands of people, the final answer is usually the right one.

No Responses to  “Sentiment Analysis 101”