Evaluating online engagement – effect of the long tail?

ByMichele Ide-Smith

Evaluating online engagement – effect of the long tail?

I’ve found some really useful information recently on evaluating online engagement including Dave Brigg’s suggestion for a common framework, Alice Casey’s excellent presentation on evaluating online participation and the Digital Dialogues report. As I’ve been setting up various e-engagement projects at work I have been thinking about methods for both quantitative and qualitative evaluation, in particular the type of metrics and measures we might use for quantitative evaluation.

I’ve also recently been reading what Clay Shirky has to say on the “power law” distribution in social media, based on an article from 2oo3 and his fascinating book Here Comes Everybody. Shirky outlines that the majority of user activity such as blog posts, photo uploads or wiki edits on social media sites  is done by a relatively small percentage of the total number of contributors. So if you measure the amount of activity per person (e.g. blog posts or photo uploads) and represent the results on a graph, you will find a particular distribution known as a power law distribution, or long tail. This phenomena is also sometimes referred to as the 80/20 rule (Pareto principle) where 80% of the activity is done by 20% of the people.

From Wikipedia: An example of a power law graph being used to demonstrate ranking of popularity. To the right is the long tail, to the left are the few that dominate.

From Wikipedia: An example of a power law graph being used to demonstrate ranking of popularity. To the right is the long tail, to the left are the few that dominate.

Shirky refers to power law distributions in social media as an inevitable inequality:

Inequality occurs in large and unconstrained social systems for the same reasons stop-and-go traffic occurs on busy roads, not because it is anyone’s goal, but because it is a reliable property that emerges from the normal functioning of the system.

As local authorities start using more social media to engage citizens, I wonder how we might deal with the inequality of the power law distribution when evaluating online engagement? Here’s why I think it’s important.

Making a business case for online engagement

Local government has traditionally asked citizens for their opinions using surveys, public meetings, panels and focus groups. Many local authorities have dabbled with social media as a communication channel. Some central government departments and local authorities have experimented with social media as a means to engage citizens in a two-way conversation. But true online engagement of this nature requires significant resource to moderate comments, publish timely responses, collate comments and feed them consultation or decision making processes.

For social media to be taken seriously as a tool for engagement, to which resources  are allocated, Senior Managers will need to be convinced of its value. Which is where you need to start planning exactly how to measure the level, depth, quality and success of online engagement. Local authorities are used to reporting against KPIs and therefore familiar with quantitative statistics (often figures speak louder than words). In this post I won’t delve into qualitative evaluation (such as surveys, focus groups) although I am very interested in possible evaluation methods that can be used and transferred from disciplines such as human computer interaction. More on that another time!

Defining evaluation metrics

Drawing on the sources mentioned in the first paragraph of this post, some possible quantitative metrics for measuring online engagement include:

  • web statistics e.g. number of visitors, page views, time on page (although this may be tricky to measure on 3rd party sites like Facebook);
  • number of people who sign up, or become a friend/fan;
  • total number of discussion posts or comments;
  • percentage of participants who added a post or comment;
  • frequency of comments across all participants and averages*;
  • number of participants who post multiple comments about one topic/discussion (e.g. a thread);
  • length of discussions (e.g. number of comments/responses per discussion);
  • number of instances where a comment (or discussion) directly affects a decision or policy.

*If the power law distribution holds true, then it is likely that a small minority of participants in the online engagement exercise will make the most contributions, with most participants making only a single contribution. Is there a danger that online engagement activities may be interpreted as undemocratic, ineffective and unrepresentative, simply because only a small margin of participants are seen to be actively participating?

As someone who is new to planning, facilitating and evaluating online engagement, I am interested to see how the power law phenomena manifests itself and whether views from a relatively small group have an impact on decisions and policy. I am also interested to gauge the impact on Senior Managers’ perception of the value of online engagement.

This content is published under the Attribution-Noncommercial-Share Alike 3.0 Unported license.


About the author

Michele Ide-Smith administrator

Leave a Reply