eXtension Ask an Expert Task Force:Findings and Decisions

Final ReportJune 3, 2009

Commitee Membership:Kevin Gamble, ChairRick DurhamMike LamburLarry LippkeBen MacNeillTerry MeisenbachBeth RaneyCraig Wood

Ask an Expert Task Force Report

IntroductionSince its beginnings, eXtension has included an Ask an Expert (AaE) function as part of its portfolio of toolsavailable for public information gathering. Concern has been expressed as to the success of AaE over timeand its sustainability for the future. eXtension has focused resources on AaE and as a part of that focus hasdetermined a need to study where AaE is today and where it can go in the future. With the addition of theAsk eXtension widget as a supplement to AaE, especially at the local/county level, AaE from the nationalperspective has garnered even greater attention. Staffing of the AaE system at the national level has beenformally limited to launched Communities of Practice (CoP). At the same time, however, questionsrequiring expertise outside the launched CoPs also come to eXtension. The public knows that theCooperative Extension System (CES) has expertise that extends far beyond that of the publicly launchedCoPs. The growth in questions outside of staffed areas of expertise has created further questions regardingthe sustainability of AaE in the long term.

This Task Force was charged with reviewing the current status of eXtension's AaE system, and to providedirection for its future.

Objectives of project

• Assess the Cooperative Extension System's ability to operate an AaE system.

• Assess what CES faculty and educators think are acceptable levels of performance: time torespond, depth of answers, etc.

• Assess the current performance (human) to date: timeliness, capacity, correctness, coherence,completeness, etc.

• Determine clientele expectations for an AaE system: timeliness, capacity, correctness, coherence,completeness.

• Determine the level of staffing and resources required to operate a fully functioning AaE system.

• Prepare report to the Governing Committee for consideration.

How is the public best served?Cooperative Extension's "stock and trade" for nearly a century has been via a community presence;Extension educators and agents located within a local community as an educational source for new practicesin agriculture, family sciences, youth development, and community development. As more and moreAmericans turn to the Internet for information, they also find communities... people with similar interestswho formally and informally band together in a social context. It stands to reason that CooperativeExtension should consider this online community as an equally important and relevant public with as muchor more value as a public constituent group. In the traditional model, clients come to the Extension office,ask a question, and are answered by an expert supplemented with publications, fact sheets, and relevantadditional educational resources. The new model via eXtension can and should operate with the same robustservice and educational model.

Americans have grown more and more sophisticated in their Internet use habits and are also moredemanding as they seek information from myriad sources online. Previous research and metrics fromeXtension in general show that 83% of clients come to eXtension from search engines (Google inparticular), and that their visits are to specific pages deep within the site. They look at an average 2.1 pagesand spend an average of 1:26 minutes per visit. General data on Internet use show that the public isinteracting with the eXtension Web site much as they do other "destination" sites that are content centric.Increasingly they are visiting a few large sites, and staying longer. The total number of sites they are visitingcontinues to steadly decrease. According to Steve Reubel citing March 2009 data from Nielsen Online1:

After years of erosion it now it appears the destination web era is drawing to a close. This is atrend that digital thinkers like Om Malik have long noted. In fact, the numbers prove it.

In March the average American visited a mere 111 domains and 2,500 web pages, accordingto Nielsen Online. What’s worse, our attention across these pages is highly fragmented. Theaverage time spent per page is a mere 56 seconds. Portals and search engines dominate,capturing approximately 12 of the 75 hours spent online in March. However, people-poweredsites like Wikipedia, Facebook and YouTube are not far behind, snagging nearly 4.5 hours ofour monthly attention.

Current Conditions: a SnapshotNumbers served: Since its inception, the AaE system has received 12,205 questions. In May of 2009 itreceived 1,435. So far, in 2009 the number of questions received has been increasing at 29% per month.Figure 1 depicts the number of questions received and answered over the past year. It also shows the patternas to when experts answer questions, with a significant drop-off in weekend activity and a flurry ofanswering on Mondays. The average time to respond to a question is 70 hours (3 days) with a medianresponse time of 39 hours (1.6 days). Questions are "retired" after being in the AaE system without beinganswered after 10 days. Recipients are sent an apology and asked to contact their local CooperativeExtension offices. On average, 3.5% of the questions received are going unanswered either for non-performance, or because there are no experts in a subject matter available or willing to answer. In addition,11.5% of the questions received are for subject matter areas where no current CoP exists to generateresponses.

Figure 1: Questions asked and answered

How the public comes to Ask an Expert: Overall 83% of the people visiting www.extension.org enterdirectly from one of the major search engines. This is reflected in where people come from when coming tothe Ask an Expert form on the eXtension web site. Eighty-six percent of the people coming to the AaE formon the eXtension site come from a content page deep within the site. They initially arrive via search, don'tfind the specific information they are seeking, and redirect to the AaE to begin asking their questions.

Who is being served: The audience being served is not particularly diverse or representative of the generalpopulation (Figure 2.) A quick glance tells us that the majority of users are female (63%), white (92%), welleducated (36% with bachelors, and another 39% with advanced degrees), older (66% > than age 50), andfinancially better-off (72% reporting incomes > than the median family income).

Figure 2. Demographics of users of the AaE system

Nonetheless, the AaE system is reaching new clientele (Figure 3). Twenty-nine percent of the people askingquestions had never heard of Cooperative Extension prior to using the AaE system. Another 27% wereaware of Cooperative Extension but had never accessed it previously.

Figure 3: Previous Awareness/Use of Cooperative Extension

Who is Answering Questions: The number of Extension faculty and educators answering questions is whatyou would expect from this sort of system. It is a classic Pareto Distribution (Figure 4) where 20% of theindividuals in the AaE system are responding to 80% of the questions. Of the 656 Extension professionalswho have answered questions, 131 of those individuals are providing the bulk of the effort. Less than 1% ofthe estimated 15,000 Extension professionals are currently active in the AaE system.

Figure 4: Who's answering questions?

What Was StudiedFour studies were conducted as a part of this review:

Directors and Administrators: A sample of 15 Directors and Administrators were interviewed in face-to-face meetings and by phone. The questions used in the interviews were validated and reviewed at a meetingof the eXtension Governing Committee in March of 2009.

Public survey: A complexity-based approach was used to examine the various factors that affect users’perceptions of the Ask An Expert service 2. A survey was conducted of the 6,557 unique users of the AaEsystem. The Cognitive Edge SenseMaker™ software was used for the analysis: SenseMaker™ contains arange of analytical and interrogation tools that allow both recall and interpretation of sense-making items.3 Itmakes extensive use of visualizations to allow complex patterns and exceptions to be discovered. The entirepopulation of question askers were sent an email survey. Subsequent follow-up surveys were sent to non-responders. Five hundred and four individuals completed the survey for a response rate of 7.7%. This was aqualitative research design, and 200 responses were needed at the lower range to begin to see meaningfulpatterns in the data4. If sampling were to be done with this population a sample size of 376 would berequired for a probability of .05.5 The research methodology used was a pre-hypothesis narrative-basedapproach6.

Faculty/Educator Focus Groups: Five focus groups were conducted using the Adobe Connect Pro webconferencing system with selected CoPs, highly active AaE responders, and a group of less-than-enthusiasticCoP members, to gain insight to each of the studied questions (see What Was Observed). The pollingfeatures of Connect were used to determine the extent to which the questions identified the significance ofthe issues in the minds of the participants, and to collect qualitative information through discussion.

Log Analysis: A review of logs was conducted, and other data analyzed to get a clear picture of how thepublic was interacting with the www.extension.org site. This was to answer the questions as to whether thepublic was able to find the page and easily enter their questions. Additionally, an analysis of the internal logswas conducted to determine the responders' use of the AaE system as they interacted with it to provideanswers.

What Was ObservedFollowing are the major findings from the four studies:

The public believes that the answers provided by the AaE system are science-based (figure 5). This is aremarkable finding when you consider that 56% of the respondents had no previous experience workingwith Cooperative Extension. In addition, they believed the answers were from a trusted and unbiased source(figure 6). These two strong preferences would indicate that the Cooperative Extension System is perceivedas being true to its mission even with people who had little or no previous familiarity with the system.

Figure 5: Public perception as to the science-based nature of AaE answers.

In addition, they trusted the answers that were provided and believed them to be from an unbiased source.

Figure 6: Public perception of Answers from eXtension as Not trusted to Trusted

What the clientele said:

“I am glad to have my other research confirmed however by a trustworthy source.”

“I got a reply that, although not what I wanted to hear, I trusted”

“I had a very prompt reply, with very accurate information plus direction to a paper.”

"A Google search provided an answer but I needed someone with authority and experience tocorroborate/validate that answer so I could do my calculations."

Clientele are primarily asking questions that they consider to be of personal and economic importance. TheAaE system is not being used to ask trivial questions or to inquire about things considered a curiosity (instark contract to Yahoo! Answers: answers.yahoo.com). The questions were also skewed toward being ofeconomic importance as well. When the responses were filtered to eliminate consumer horticulture, then theeconomic importance of the questions moved even further towards being of significant economic need.

Figure 7: Public description of their question as to importance.

Figure 8: Public description of their question as to economic need.

What the clientele said:

“I was worried about food safety for my family if I planted my vegetable garden as planned.”

“I wanted to know about safety concerns when cleaning up a rat/mouse infested house as itrelated to hanta virus.”

"Squirrels have been eating my house. We tried several suggestions, and the situation isimproving.”

“He gave practical information about how to decide what is best for my family, as opposed tonecessarily doing what's best for the tree. Very helpful."

The one variable that impacts the utility of the answers provided is the rapidity with which an answer isprovided. After two days, people indicated that the usefulness of the answer in helping them to solve theirproblem was diminished (Figures 9 and 10). If the Cooperative Extension System desires to make adifference in people's lives, to provide answers that are timely, and also of practical use, then it simply mustrespond more quickly.

Figures 9 and 10: Public's Perception of Utility by Response Time (What role did the answer play in solving theproblem?)

What the clientele said:

“We were supplied with the information we wanted in an expedient, competent andprofessional manner (which is so unbelievably rare these days)."

“I do know it took four days for me to get an answer and I thought at the time if the answerwas needed was urgent, I would have been in trouble.”

“To be frank, it’s kinda hit or miss and I’m not sold on the 'quick' response promised.”

Timeliness of the response comes into play once again when measuring the public's satisfaction with theservice. Satisfaction was at 80% when they received an answer within a day. Their satisfaction begins todecrease after two days, and drops to just 68% if the answer isn't provided within a week. Satisfaction fallsdramatically when it takes longer than a week for the experts to respond.

Figure 11: Clientele satisfaction and timeliness of response

What the clientele said:

“I didn’t expect an answer so quickly. In the matter of an hour or so..the response was veryhelpful. I am thrilled with this service”

“I will not keep a client waiting a week for a response. My suggestion would be that if you aregoing to provide this question and answer service, you need to respond in a timely fashion(within 2 days)”

These findings are in stark contrast to what faculty and educators determined were realistic response timesgiven other demands of their appointments. From the focus group findings:

Experts should not be expected to respond to questions during weekends and holidays. Suchtime should be subtracted from the time triggering escalation reports. "Office hours" shouldbe published so questioners do not have false expectations on when to expect a response. But,during such office hours, 48 hours seems an appropriate amount of time.

Exacerbating this is the public's desire for answers that are deeper and more detailed than the experts areproviding. They believed that the answers provided leaned toward being too simple, lacked context, andspecificity. Providing more detailed answers is likely to lead to a larger commitment of time in providingmore in-depth and researched answers. Figure 12 shows the tendency for the responses to be considered toosimplistic with the preponderance of responses falling below the mid-point on the Z scale.

Figure 12: Visualization of quality of answer. Red: science-based; Green: trusted; Black: simple/complex

What the clientele said:

“I did not really get an answer...I did ask a fairly long, drawn out and complicated question. Iwas rather disappointed in the short answer.”

"The answer was well-written and included even more information that I had expected.”

“The ‘Expert’ forwarded an article but it did not address the specifics of my question. I feltrather disappointed and ended up doing more research on my own... even re-emailed the‘Expert’ a 2nd time to clarify but she did not return my email.”

“As a person who has been involved in Dairy business, I knew what we should do but I wantedto get a confirmation on what we are going to do, but unfortunately what I got was very simpleconcepts."

Faculty and Staff Focus Group:

Tenure track faculty observed that answering the public's questions is not considered scholarly work, and isnot rewarded. They acknowledged that it is an important component of the Extension mission, but that it isbest valued internally when carried out by individuals in non-tenure leading positions. One strategy is toprovide the opportunity for responding to AaE questions at an appropriate organizational level where it willbe adequately acknowledged and rewarded. This will differ by state Cooperative Extension Systems.

Faculty, educators and staff also indicated that they needed additional training opportunities to improveawareness and to enhance performance of the AaE.

Administrative Interviews

Administrators' perception of faculty and staff involvement in working with AaE was very positive. Therewere few concerns voiced regarding faculty and staff working or serving clientele outside of their assignedgeographic responsibilities. It was often cited as a positive aspect of eXtension, and an area that was morecritical than ever given the current budget situation. Virtually every administrator mentioned eXtension andAaE as a key strategy for dealing with expertise gaps and personnel shortages.

Administrators observed that AaE is not considered scholarly work, but saw value in reaching clientelethrough this technology. There was a general acknowledgment that involvement of faculty and staff in AaEvaries widely between states. Perception and awareness seem to depend on the state's administrative viewsand promotion of AaE and eXtension in general. In states where the administration encourages and supportsinvolvement, AaE is perceived as positive. In those states where encouragement does not occur, there islittle engagement and faculty and staff are reluctant to participate.

What Needs to Happen

As noted earlier, Americans want engagement with online communities. They also want a more interactiveexperience in that engagement process. To be a viable and valuable resource, eXtension must pay attentionto the needs of clientele and work with Cooperative Extension leaders to prepare for the current and futuredemands of AaE.

There is a need for more people at the appropriate level within and outside Cooperative Extension toanswer questions. CoP's primary focus needs to be on building their communities, recruiting new people toanswer questions, and providing professional development opportunities to better prepare experts to answerquestions in an appropriate and timely manner. With the potential to receive questions from many morepeople (orders of magnitude larger), Cooperative Extension is going to need to grow its capacity to serveever increasing audience numbers. Far more than the current group of 131 experts will be needed if the CESis to even begin to service the public's demand for this type of system. Indeed, the system is struggling toperform given the moderate number of questions received currently, and this will only be alleviated bygrowing the number of active experts responding to questions.

Regardless of the information and educational content provided on the public website from launched CoPs,when engaging with the site the public asks questions that fall outside the expertise of the current CoPs.Clientele know that the CES has expertise outside of the areas of the launched CoPs and will ask theirquestions. Indeed, 11.5% of the questions received are in areas where eXtension is not currently staffed torespond. It is not acceptable to not respond to these questions. Each state needs at least one designatedQuestion Wrangler to handle the assignment of questions emanating from their state and to assist inlocating experts to answer incoming questions in areas where there are no current CoPs. Additional expertsin each state need to be identified. Working through Institutional Teams each state should develop a

strategy to have their staff flesh-out their profiles in the eXtension directory application found atpeople.extension.org (People) so that they can more easily be identified to assist in responding to questions.

Performance metrics must match clientele expectations. The results are clear; customer satisfaction andthe utility of the answers provided drop significantly when it takes longer than two days to receive aresponse. To speed responses, the timing of non-performance escalation reports to CoP leaders needs to bereduced from the current 48 hours to 24. Unanswered questions need to be reassigned after 24 hours.Multiple people need to be alerted to questions in their areas of expertise to mitigate the problems caused bythe current system of assigning each question to an individual creating a single-point-of-failure. The soon tobe released version of the AaE system that allows for multiple responses to a question should help toalleviate some of the issues of experts not always being available.

Create AaE only CoPs. Using the profiling capabilities of for CES staff found in People, work toproactively form new CoPs with an initial single focus of answering clientele questions. Provide tailoredprofessional development opportunities to assist them in getting organized, and in the mechanics of usingthe AaE system. Assist in helping to recruiting new members.

Continue to advance the use of widgets to involve more local staff with AaE. It has been noticed thatmany local educators who get involved with answering questions through their own county widgets also getinvolved in answering questions coming in from the www.extension.org site. Provide additional professionaldevelopment opportunities on using the widgets. Work with state level IT staff on the use of the AaEsystem, and on installation and maintenance of widgets.

Provide additional AaE training and professional development to raise expertise across the entire CES.This needs to include not only the mechanics of using the AaE system, but also proper etiquette, answeringstrategies, and awareness concerning clientele expectations.

Continue to focus on online engagement. To engage or not to engage with online audiences is not achoice. We must engage-- our clientele expect it. AaE is the place within eXtension where this is done, andit needs to be done well. It can no longer be viewed as an add-on, but needs to be a top priority. Eventhough faculty get less credit…this is an opportunity for the CES to be relevant in ways heretoforeunimagined.

Eighty-three percent of the people coming to the AaE system are arriving from an Internet search engine. Itis the content of the www.extension.org site which brings them. One concern is the lack of diversity in theclientele using AaE. Consideration needs to be given to creating CoPs that will appeal to a more diverseand younger demographic.

Recognize that AaE is reaching new, online audiences; and this needs to be promoted. The AaErepresents a strategic opportunity for the CES that should not be squandered.

Vision

From the beginning, one of eXtension's promises has been to reach new audiences and to bring new peopleto the CES. Through the eXtension brand study, through the ECOP Marketing Task Force branddevelopment process, and through this AaE task force, it has been learned that Americans who don't know

Cooperative Extension but who ultimately interact with it, are pleased with their experience. They believeCooperative Extension is reliable and trustworthy. They believe the land-grant university connection addscredibility and trustworthiness to the information they find. And, in the case of the studies done with thiseffort, many of those, over half, who are currently using the AaE system are unfamiliar with the CooperativeExtension System.

So, where does Cooperative Extension go from here? From these studies it is known that people value anddemand Ask an Expert type services. Therefore it is important to meet this challenge as an opportunity forfurther engagement with current audiences, and new engagement with people who have yet to discoverCooperative Extension. There is an opportunity to offer a service on the Internet that is both unique and indemand: scientifically-based knowledge from trusted sources. This is a unique niche, and and opportunity tobe seized.

As evidenced in the data above, demand both in volume and in time, can quickly dwarf CooperativeExtension's ability to respond to Ask an Expert questions. Cooperative Extension must look to alternativestrategies including broadening the expertise base to answer questions. It must look at the use of trainedvolunteers; incorporating eXtension's outstanding professional development capabilities and subject matterspecialists to provide training. It must take advantage of social networking strategies available now withWeb 2.0 to bring experts to a more granular level engaging at the community level where CooperativeExtension has done its best work for nearly a century. It must also "grow" its expertise bank through amentor/protege approach where current experts share their knowledge one-on-one to bring new and youngerfaculty and staff into this effort. The "graying" of America will not stop at the doors to America's land-grantuniversities, and thus Cooperative Extension must look toward empowering and enabling new faculty andstaff to build these experts as soon as possible.

It has long been the goal of eXtension to put Cooperative Extension into the homes of more and moreAmericans. The AeE system truly meets that goal. It is an interactive approach; an engaging approach; avalue-added approach. It takes full advantage of CES' reputation for reliability, credibility, and forproviding research-based expertise in a customer-focused manner. That trust, coupled with a responsive,well-trained bank of experts will serve CES well into the future.

References

1. Reubel, Steve. "The End of the Destination Web Era." Micropersuasion,http://www.micropersuasion.com/2009/05/the-end-of-the-destination-web-era.html, May 3, 2009.

2. Elliott, Jane. Using Narrative in Social Research: Qualitative and Quantitative Approaches.Thousand Oaks, California: Sage Publications, 2005.

3. Sense-Maker Suite version 3, http://www.sensemaker-suite.com

4. Lorenz, M. O. "Methods of measuring the concentration of wealth." Publications of the AmericanStatistical Association. 9 (1905) : 209–219.

5. Chelveldave, Michael. (March 24, 2009). Personal conversation.

6. Israel, Glenn. Determining Sample Size: PEOD6. IFAS, University of Florida, 1992.

7. Snowden, David and Mary Boone. Leader's Framework for Decision Making. Harvard BusinessReview (November 2007): 10pp.