community metrics at novell
Post on 13-Sep-2014
1.451 views
DESCRIPTION
TRANSCRIPT
![Page 1: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/1.jpg)
Community MetricsThe Novell Approach
Lee Romero
http://blog.leeromero.org
2009 October
![Page 2: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/2.jpg)
Contents
Communities at Novell 3
Setting up the Discussion 6
Membership Metrics 10
Activity Metrics 17
Tying into Performance Mgmt 22
Some Advanced Metrics 25
Additional Metrics 28
Final Words 32
![Page 3: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/3.jpg)
Communities at Novell
![Page 4: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/4.jpg)
4
Community Metrics at NovellA brief overview of the program, part 1
• Novell started a CoP program in 2002/2003 time frame
• The program was primarily focused around the “solutions” and products Novell has marketed and sold
• The overall program responsibility fell to the Enterprise KM group:• KM group established the goals, methodologies and processes
• KM group also identified a set of standard infrastructure tools and supported those
‒ No new significant development / implementation was supported
‒ This was primarily a case of identifying what was in place and providing an a la carte menu for communities to work with
![Page 5: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/5.jpg)
5
Community Metrics at NovellA brief overview of the program, part 2
• The CoP program identified levels of communities that largely guided investment and formal support for individual communities
• There were approximately a dozen “top level” communities over time
• There were also dozens of informal communities supported to a limited extent
• The a la carte menu of tools included (among other things):• Intranet web sites
• Mailing lists
• Enterprise Wiki
• Team spaces (implemented in LiveLink initially)
![Page 6: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/6.jpg)
Setting up my discussion
![Page 7: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/7.jpg)
7
What is a community member?It all started with a simple question
• The rest of this presentation will focus on a set of metrics we developed at Novell that were manageable, scalable and, I think, provided useful insights
• To start with, though, early on in our program, we were faced with a simple question from our community leaders:“How large is my community?”
• Which quickly turned into:“What is a community member?”
![Page 8: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/8.jpg)
8
An AnswerA definition of community membership
• Because of our development limitations, we were faced with trying to track membership using either our existing tools or having some type of manual support
• In looking at our tools, we found that our mailing list infrastructure provided a reasonably good solution:• It already had (list) membership management functions
• People could subscribe / unsubscribe on their own
• We already knew that the lists needed to be aligned to communities
• With that insight and no other realistic option, we decided on a definition:• You are a member of a community if you are subscribed to any mailing list
associated with the community
![Page 9: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/9.jpg)
9
An ImplementationHow we tracked membership
• Our mailing list server provided a simple XML format for members for each list• We implemented a mechanism to sync this member list data from the XML
format into a SQL database
• This, combined with some integration with other databases already available, provided a wide variety of membership reporting
• We later added on a mechanism that captured an “event” for each post to a list in the same SQL database, which led to a variety of activity based reporting
![Page 10: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/10.jpg)
Membership Metrics
![Page 11: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/11.jpg)
11
Basic Metrics
• With the basic structure in place, we found we could provide data to answer many questions:• How big is any given mailing list?
• How big is a community?
• How much change is there in community population over time?
• Various slicing and dicing by different demographics
• The following slides provide examples of many of these
![Page 12: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/12.jpg)
12
Basic MetricsCommunity size
• Simple to query current size• By tracking “join” and “departure” events, it was also straightforward to
provide growth over time
• We could also measure the total overall population of the “community program”
![Page 13: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/13.jpg)
13
Basic MetricsCommunity penetration
• Knowing the employee population size (total or by demographics), we could provide measures of “penetration” of the community program as a whole and by community against different groupings
![Page 14: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/14.jpg)
14
Basic MetricsDemographics
• By combining community membership with HR information, we could provide break downs of membership (both overall program and individual community) on a variety of dimensions
![Page 15: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/15.jpg)
15
Basic MetricsDemographics, continued
![Page 16: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/16.jpg)
16
Basic MetricsOther questions
• Some other (not necessarily actionable) questions we could easily answer included:• How many communities is an employee a member of on average?
• How many communities is a community member a member of on average?
![Page 17: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/17.jpg)
Activity Metrics
![Page 18: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/18.jpg)
18
Activity Metrics
• Once we implemented a means to capture an “event” for each post in each mailing list, we were able to understand community activity (in this one tool) very easily
• We also were also able to answer (admittedly, simplistically) another key question we had been asked:• “How many *active* members do I have in my community?”
• We answered that with:• A member is active if they have posted at least one post during the time
period of interest.
![Page 19: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/19.jpg)
19
Activity Metrics
• Some examples of specific metrics we were able to track
![Page 20: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/20.jpg)
20
Activity Metrics
• We could also gain other potentially useful insights
• Identifying most active members• Potentially useful for identifying SMEs or core team members (or even
community leaders)
• Percent (and number and even individual identification) of lurkers in communities• Useful within a community to know who is there but not “active”
• Useful across communities to know when a community has a significant different rate of “lurkers” compared to other communities
![Page 21: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/21.jpg)
21
Activity Metrics Networks within communities
• Using the activity data to connect people who corresponded, it was even possible to do data mining to get a sense of the network within a community
![Page 22: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/22.jpg)
Performance Management
![Page 23: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/23.jpg)
23
Performance Management
• Working with the HR group, we eventually were able to community involvement with our performance management program
• This was achieved by having community involvement embedded in the “employee self service” and “manager self service” portals
![Page 24: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/24.jpg)
24
Performance Management
• The intent was not that a specific goal was desirable but that this provided a way to initiate conversations between a manager and an employee about involvement, encouraging:• Some thought about which communities were valuable for an employee
• Some reflection on level of involvement and activity
• Development of employees over time
![Page 25: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/25.jpg)
Advanced Metrics
![Page 26: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/26.jpg)
26
Advanced MetricsSome experiments into compound metrics
• Another simple question prompted some digging into other uses of the data we had:• “Why do we need to provide navigation to community sites [on the intranet],
anyway? They don’t get any traffic at all?”
• The resulting analysis attempts to draw a comparison between “web site visits” and community membership / activity• I came to call this measurement “knowledge flow”
• You can find a detailed description at:• http://blog.leeromero.org/2008/11/20/measuring-knowledge-flow-within-a-c
ommunity-of-practice/
• The eventual formula I worked out is:Kc = 2 * Pc * Ac - Pc * Ac
2
Mc
![Page 27: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/27.jpg)
27
Advanced MetricsKnowledge Flow examples
Animations (in four dimensions!) available at:
http://blog.leeromero.org/2008/11/21/visualizing-knowledg-flow-in-a-community/
![Page 28: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/28.jpg)
Additional Metrics
![Page 29: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/29.jpg)
29
Additional Metrics
• On top of these metrics based on a mailing list implementation, the Novell CoP program also used a number of additional metrics
• These included:• Web site usage
• Intellectual asset production
• Specifically called out white papers as well
• “Anecdotes” from community members
• Knowledge share events
• And attendance at same
• For each of these, we provided quarter-to-quarter changes as well
![Page 30: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/30.jpg)
30
Additional MetricsWiki metrics
• Another area we investigated but did not lock in on was use / edits of the corporate wiki
• Novell had (has) a corporate implementation of MediaWiki (in place since about 2003)
• We explored some metrics related to the Wiki for communities:• Establishing a standard Category for each community to use
• Using that Category assignment, we could:
• Track usage over time for pages in the category (it was integrated with Omniture)
• Track edits (“contributions”) over time to pages in that category
• Potentially identify anyone who edits such a page to be a community member
![Page 31: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/31.jpg)
Final Words
![Page 32: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/32.jpg)
32
In Summary
• Beyond all of the details here, a key takeaway is this:• With a very simple definition (of community membership) and a pretty simple
technical approach to implement that definition, Novell was able to gain a wide variety of insights
• Look at the tools you have, make some definitions (even if you know they are not perfect) and start tracking!
• Also, when you are thinking about what metrics you want to track, make sure all your metrics are actionable• How can your metrics be turned into actions to improve your communities?
![Page 33: Community Metrics at Novell](https://reader033.vdocuments.mx/reader033/viewer/2022061106/5413a6b88d7f728a698b4683/html5/thumbnails/33.jpg)
33
Read more
If you would like to read more about the work behind this material, there are extensive write-ups available at:
http://blog.leeromero.org
Specifically, look under “Collaboration / Communities” on the “Posts by Topic” page