*Research Skills for Public AdministrationDL-MPA Residency January 2013

Getting the MaterialsGo to:

http://usc-mpaol-la-residency-spring-2013.wikispaces.com/Residency+Research+Assignment

Go to Resource-> Residency Research Assignment

*

*Agenda

Learning objectivesResearch Skills: From data to evidenceUses of evidenceFrom data to evidenceResearch processes and types of evidenceEvaluating quality of evidenceLibrary ResourcesResearch Exercise (documentary research)Working with / Sharing Information

*Learning objectivesUnderstand how evidence is used in public managementConsider processes & sources of evidencePrimary sourcesSecondary sourcesLibrary resources at USCApply search skills in a case exerciseDevelop a critical perspective on the quality and appropriate types of evidence.Organize results effectively

Uses of evidence in public management

Refinery Explosion

ContainmentContaminantsCommunity ResourcesCauses

Uses of evidence in public managementService ImprovementPerformance Management TechniquesBureaucratic constraintsCultureModel Practices

It is all Knowledge ManagementDo not define yourself as what you know. Define yourself as what you find out.Robert Biller, former dean Price School of Public Policy Public sector organizations are mainly knowledge-intensive organizations, and to exploit their knowledge, effective knowledge sharing among the different departments is required.Willem, A. and M. Buelens (2007). "Knowledge Sharing in Public Sector Organizations." JPART 17(4): 581-606.

*

Uses of evidence in public managementDemonstrating that a problem requires actionDetermining what causes the problem and who is harmedMapping political obstacles and resourcesIdentifying best (model) practices to aid designJustifying the criteria to be used in evaluating optionsPredicting and evaluating what will happen if decision makers take option X (will X lead to a desired Y?)Determining how benefits and/or burdens are distributed among different groups of people

*DefinitionsDataInformationEvidenceFactsData that have meaningInformation that supports an argument, e.g., affects beliefs about the policy issueProducing evidence involves analysis, interpretation and judgment

*Raw Data is of Little Use

CategoryCodeSourceType of Fund2001-022002-032003-042004-052005-062006-07Major Revenue1Alcoholic Beverages Taxes & FeesSpecial Funds000000Major Revenue2Corporation TaxSpecial Funds11001100Major Revenue3Cigarette TaxSpecial Funds981,195940,611970,681981,195970,681963,168Major Revenue4Horse Racing (Pari-mutual License Fees)Special Funds41,32639,99135,59641,32635,59635,257Major Revenue5Estate, Inheritance & Gift TaxSpecial Funds000000Major Revenue6Insurance Gross Premium TaxSpecial Funds1560015600Major Revenue7Trailer Coach License (In-Lieu) FeesSpecial Funds2,3882,3882,3882,3882,3882,388Major Revenue8Motor Vehicle License (In-Lieu) FeesSpecial Funds1,927,3681,885,1482,239,3481,927,3682,239,3482,268,019Major Revenue9Motor Vehicle Fuel Tax (Gasoline)Special Funds2,828,0242,728,1332,842,9532,828,0242,842,9532,851,631Major Revenue10Motor Vehicle Fuel Tax (Diesel)Special Funds467,879474,378550,428467,879550,428580,896Major Revenue11Motor Vehicle RegistrationSpecial Funds1,891,7761,986,4742,811,9011,891,7762,811,9012,849,952Major Revenue12Personal Income TaxSpecial Funds001,318,70101,318,7011,377,388Major Revenue13Retail Sales and Use Tax (Realignment)Special Funds2,228,9782,279,4192,846,1142,228,9782,846,1142,862,308Major Revenue14Retail Sales and Use TaxSpecial Funds232,113204,468354,305232,113354,305950,814Major Revenue15Telecommunications TaxSpecial Funds14,26401,419,68414,2641,419,6841,411,392Minor Renenue20Trial Court RevenuesGeneral Fund000000Minor Renenue21Emergency Telephone Users SurchargeGeneral Fund000000Minor Renenue22California State University FeesGeneral Fund000000Minor Renenue23Income from Pooled Money InvestmentsGeneral Fund354,988189,577447,193354,988447,193533,149Minor Renenue24Income from Surplus Money InvestmentsGeneral Fund4,13137,0754,0214,1314,0215,813Minor Renenue25State Lands RoyaltiesGeneral Fund15,13178,470295,57315,131295,573255,092Minor Renenue26Abandoned PropertyGeneral Fund287,848257,533334,206287,848334,206205,180Minor Renenue27Settlements & JudgmentsGeneral Fund75,771587,96975,77187,96937,251Minor Renenue28All Other Minor RevenueGeneral Fund1,190,88312,337,0921,820,8201,190,8831,820,8201,213,398Minor Renenue20Trial Court RevenuesSpecial Funds000000Minor Renenue21Emergency Telephone Users SurchargeSpecial Funds125,381139,267130,911125,381130,911112,154Minor Renenue22California State University FeesSpecial Funds735,050818,9801,230,748735,0501,230,7480Minor Renenue23Income from Pooled Money InvestmentsSpecial Funds1,1234014331,1234331,594Minor Renenue24Income from Surplus Money InvestmentsSpecial Funds208,249107,086272,937208,249272,937492,426Minor Renenue25State Lands RoyaltiesSpecial Funds27,6985,0004,00027,6984,0000Minor Renenue26Abandoned PropertySpecial Funds000000Minor Renenue27Settlements & JudgmentsSpecial Funds402,010476,0691,645402,0101,6455,824Minor Renenue28All Other Minor RevenueSpecial Funds6,250,6245,589,4097,849,8476,250,6247,849,8477,955,517Transfers & Loans29Transfers & LoansSpecial Funds-824,131-2,446,48813,420-824,13113,420548,974SCO Adjustments30Adjustments to Reconcile to ControllerGeneral Fund1,242001,24200SCO Adjustments30Adjustments to Reconcile to ControllerSpecial Funds000000

*Formatting and Graphing Make Data UnderstandableSource: CA Legislative Analysts Office

*Manipulation and Comparison Form an ArgumentSource: The Tax Foundation

Assembling evidenceThink before you collect! (you probably know more than you realize)Dont believe everything you think! (some of your ordinary knowledge is almost certainly wrong)Iterate between documentary research and expert interviewsSources lead to people lead to sourcesSurvey promising/smart/model practicesGet opponents point of view

*Sources of evidencePrimary SourcesObservationGovernment data collectionCensusBureau of Labor StatisticsCommerce Dept.State Dept. of FinanceSurveys & InterviewsInterviews via phone or in personMailed or emailed Questionnaires Focus GroupsExperimentsSecondary SourcesAcademic Institutions Government AgenciesPrivate Firms and Orgs.Think TanksProfessional Journals, Books, & Newspapers

Primary and secondary sourcesPrimary sources produce data.The analysts job is to develop data into evidenceSkills are taught in 502x, 541, 540, 542Secondary sources often analyze data to produce evidentiary claims. The job of the analyst is to judge the quality of the evidenceThese skill permeate program and are fully integrated in Capstone

Relevance: Does the information contribute to understanding or support a point?Avoid gratuitous background informationValidity: Does a source accurately measure phenomena or relationships? (Internal validity)Can the evidence be generalized beyond the study or case used? (External validity)

Criteria for judging quality

Error is inevitableStatistical Error: Variance introduces uncertainty regarding the evidence. One may express error as a range of estimates (e.g., prison or foster care populations may vary over time)The question is whether there is simply noise (error/variance) as opposed to systematic error or bias

Bias is the real threat to validityStatistical Bias: Statistical bias is variance that occurs in a particular direction. Example: It is highly likely that the US Census undercounts the number of undocumented aliens in the US. Why?

Interpretive Bias: Bias in interpretation occurs when judgment in use of information is consistently and/or unduly influenced by partisan, ideological or other values

Using the LibraryAn invaluable resource

An introductory videoVideo #1

A closer look at resources

USC Library Resources

The web is your friendbut be carefulGoogle Scholar is integrated into libraryConsider the credibility of the source! This may be a clue in judging relevance, validity, and bias.Be meticulous about documenting sources as you go.Dont get caught in corners of the web

Sources tend to cluster

Multiple entries & long walksUse multiple key wordsTax increase v. Revenue EnhancementBe cognizant of policy predispositionsCenter for Budget and Policy Priorities v. Cato InstituteReview works cited and citing works

Gutting A BookThere is far too much to readNeed to extract essence of material quicklyGutting searches for:Main thesis Factual argumentationFocus on:Introduction and ConclusionChapter and Section introductionsTabular and graphical displays of data*

Interviews

Much critical information resides solely in the experience of key playersOrganizational processes/politicsPoorly documented organizational activitiesNew and developing fieldsGetting information fastThis will be and is a critical career skill

Structured v. semi-structured interviewsHighly Structured: Specified and focused questionsOften goal is enumeration or statistical analysisConsistency is valued

Semi-Structured: Dialogue or directed monologueSeek qualitative understanding of complex phenomenaDivergence is allowed

Preparing for semi-structured interviewsAdequate advance preparationPick a purposive sample, eg:Secure veteran or retiree..Novice..Expert Disgruntled employee?Potential supporters or opponentsSchedule the most important interviews later

Purpose, guides, promptsDevelop a few guide questions to direct the discussionForesee prompts (examples, follow-up)But do not be over-directiveThink on your feet!Use a teach me approachAvoid leading questionsPrompt interviewee to be descriptive rather than evaluative

Dos and DontsDo:Provide and respect confidentialityEncourage story telling rather than evaluationBe alert to revealing statements and quotable quotesBe sensitive about body languageWrite up notes immediately followingRead Hammer and Wildavsky (The Open-Ended, Semi-structured Interview: An (Almost) Operational Guide. In Wildavsky, ed., Craftways, 1993) Dont:Misrepresent yourself or your purposesInterrogateAsk leading questionsAffirm or contradict statements by intervieweeWaste interviewees time

Build a knowledge base

Analysis of decisions requires theoretical and empirical comparisonsReform requires knowledgeKnowledge needs to be constructed and given permanenceThe CA Performance Management Ex.CA Best Practices Wiki

Jumping in . . .Research is a craft best learned through experience

Case challenge

Think before you research! Brainstorm the questions that are important to answerCome up with a list of potential sources.Research!Designate a recorder, upload your answers to wiki

*Examples from the determinate sentencing case: *Examples from the determinate sentencing case: *Examples from the determinate sentencing case: *Example (again) from Three Strikes lawsNumber of people in jail on three-strikes charges (site). This is a fact. What do we need to know to give it meaning? Example: What percentage of all people in jail does this represent? How does this number compare to the three-strikes incarceration rate in other states? Is the number increasing or declining over time? What are the demographic characteristics of these people? Data require context to have meaning.Evidence: Evidence is information that is brought to bear to support an argument.*Which of these sources do you think you might use to collect evidence on the term limits proposal?Relevance all academic research is based on models simplifications of the realworld Does a particular model really work in a certain situation

Models of profit maximizing behavior may or may not apply to public sector

Validity Educational testing

Do math and reading test accurately measure learning --- integrity of tests and question whether learning is fully comprised of reading and math.

*We have a fascination with exact numbers point estimates,

4000 illegal guns300000 immigrants

but there is usually much more uncertainty

Look at polls*Who is familiar with Nate Silver??

You have polls

Statistical bias if polls are consistently wrong not reaching some people

Interpretive bias Republicans had almost willfully deceived themselves***