CEGAA-OSI EE Partners WorkshopMarch 09

Urbanus KiokoCentre for Economic Governance and AIDS in Africa

Your Understanding of Research?

Systematic means of solving a problem

Process in which observable verifiable data is systematically

collected in order to describe, design or predict events

Systematic, objective and scientific investigation of a

phenomenon targeting specific aspect for the purposes of

discovering, interpreting facts, principles and theories

The search for knowledge through objective and systematic

methods of generalisation and theory formulation

To inform decision making by producing evidence

Evidence-Based Advocacy Framework

Project definition/over

all goal

What is your desired long-term impact, the outcome of your efforts, the rights you wish to protect or promote, constitutional and legislative obligations, international/regional commitments. What are you hoping to achieve?

Advocacy objectives

SMART objectives to bring about the change needed to achieve your GOAL. (SMART: Specific, Measurable, Achievable, Realistic, Time-bound), (Note that the advocacy strategy should begin to be developed now, but will be finalized when the preliminary research findings are available).

Problem statement

Clearly state what the problem is that you are going to address. Many sources to find topics or issues that can lead to research questions: Personal experience, Articles in professional periodicals, Unpublished research by others. Focus your research so that it is "do-able." Be careful! Don't try to do too much in one study.

Relevance in terms of being a priority problem◦ How large or widespread is the problem◦ Who is affected◦ How severe is the problem

Avoid duplication Feasibility Political acceptability Applicability of possible results and

recommendations Urgency of the results for making a decision Ethical acceptability

Defining the specific research

question/s

What do you want to answer? What principles do you want to test? What data will the stakeholders require for the advocacy messages? Use Lit.Review (throughout process)

The scope of the research

State, private, donor, household expenditure/ allocations? Sources, agencies, providers, services? Inputs/ outputs/ outcomes/ impact? Federal/national and/or state/sub-national? Recurrent and/or development? Sectoral analysis? Programme analysis? Years to be covered? Population focus? Vulnerable group/ gendered perspective? Your framework of analysis – rights-based, sectoral, issue, equity?

Research aim and objectives

To achieve the advocacy goal, identify more realistic aims of the research required to support the advocacy campaign, and develop specific, measurable objectives of the RESEARCH.

The scope of the research-

Literature review

Survey of professional literature that is pertinent to your particular question.

Find what others have done in relation to topic of interest.

Why review◦Prevent duplication◦Assist in refining statement of problems◦Acquaint researcher with methodologies already used◦Assist in justifying a research

What potential sources of data◦Individuals, groups, and organisations◦Published information, unpublished information

Where can information be found◦Local◦National◦International e.g. internet

Key partners & participation

process. Stakeholder involvement/ ownership.

Sources of data? Collectors of data? Users of data! Benefits/ risks. Plan to involve all the stakeholders to ensure ownership and relevance of data collected and research process.

Methodology:a.Study design

Quantitative, qualitative, participatory, costing, economics – micro/macro analysis, budgetary analysis, expenditure tracking techniques, community/Citizen action research – social audits, satisfaction surveys, score cards. Donor surveys, household/ clinic/ district surveys. Focus Gp Discs.

Identification of appropriate indicators

What indicators/ information will you use to show the principles you are trying to measure?

Research design can be thought of as the structure of research -- it is the "glue" that holds all of the elements in a research project together

A design is used to structure the research, to show how all of the major parts of the research project -- the samples or groups, measures, treatments or programs, and methods of assignment -- work together to try to address the central research questions.

Primary vs Secondary

Exploratory vs Conclusive

Quantitative (BMET analysis, pre-coded closed ended questionnaires, surveys) and/or

Qualitative (open ended questions)

qualitative research attempts to capture "the big picture" and see how a multitude of variables work together in the real world.

It examines naturally occurring behavior, so the investigative methods are as non-intrusive as possible

Focus group discussions)

Continuous (panels, tracking studies) vs Ad hoc / once-off (surveys)

Primary vs Secondary

Primary Research

Data generated specifically to address

the problem/ issue

Secondary Research

Data originally generated for another

reason

c. Sampling techniques

Representative sample? Apply formula. Purposive sampling of districts/ households? Give criteria for respondents’ inclusion/ selection. How will you identify and select respondents from the group you are focusing on?

d. Tools Budget analysis, expenditure analysis, literature review, focus groups, workshops, interviews (face-to-face, telephonic/ questionnaires (self-administered?), surveys, roundtable discussions, stakeholder meetings, observation?

b. Key respondents / your sample

population

Link to earlier decision on population/ vulnerable group. Identify key ‘holders’ of information (govt, donor, service users). Regions/ states? Districts? Households? Individuals? Key officials from MoF, MoH, Parliament,, donors, programme managers, relevant civil society groups, NGOs, etc.

The listing of the accessible population from which you'll draw your sample is called the sampling frame

Sampling: process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen

The sample is the group of people who you select to be in your study.

Random sampling-gives each of the units in the population targeted a calculable and non-zero probability of being selectedAssumption is that each unit has equal chance of being

selected SRS: the members of the population (N) of interest

are numbered and a number (n) of them are selected using random numbers without replacing them

Each sample unit can only appear once in the sample

Each pop member has an equal chance of selection

Steps you need to follow in order to achieve a systematic random sample:

1. number the units in the population from 1 to N 2. decide on the n (sample size) that you want or

need k = N/n = the interval size 3. randomly select an integer between 1 to k then take every kth unit N = the number of cases

in the sampling frame ◦ Note: ◦ n = the number of cases in the sample ◦ NCn = the number of combinations (subsets) of n from N

◦ f = n/N = the sampling fraction

Let's assume that we have a population that only has N=100 people in it and that you want to take a sample of n=20.

To use systematic sampling, the population must be listed in a random order.

The sampling fraction would be f = 20/100 = 20%. Interval size, k, is equal to N/n = 100/20 = 5. Now, select a random integer from 1 to 5. In our example, imagine that you chose 4. Now, to select the sample, start with the 4th unit in the

list and take every k-th unit (every 5th, because k=5). You would be sampling units 4, 9, 14, 19, and so on to

100 and you would wind up with 20 units in your sample.

Sometimes called proportional or quota random sampling,

Involves dividing your population into homogeneous subgroups and then taking a simple random sample in each subgroup – so that each sub-group is proportionally represented in the sample – you can then make inferences about these groups in the population.

Objective: Divide the population into non-overlapping groups (i.e., strata) N1, N2, N3, ... Ni, such that N1 + N2 + N3 + ... + Ni = N.

Then do a simple random sample of f = n/N in each strata.

In cluster sampling, we follow these steps: 1.divide population into clusters (usually along

geographic boundaries)

2.randomly sample clusters

3.measure all units within sampled clusters

The selection of clusters can be multi-stage

Involves selecting districts, the primary sampling units, within a region for the sample, and within these sample electoral wards and finally within these a sample of households.

Is a deliberate non-random method sampling, which aims to sample a group of people, with particular characteristic.

The criteria for selection must be justified and relate to the questions you are trying to answer

Can be justified with difficult access to certain populations eg. HIV+ patients attending the ARV clinic

Can be justified based on costs and time eg. Small case study

Is not generalisable!!

e. Sources of data

Budget books, budget reviews, MTEF plans, expenditure reports, Auditor General reports, ministry budgets and annual reports and costing reports, strategic plans, MoF reports, Parliamentary reports especially of budgeting committees, Public Expenditure Reports, health information systems, legislation governing public financing, Reports of the implementing agencies of the strategic plan, central bureau of statistics, household data, census, surveys (economic eg. I&E, health), previous studies, programme evaluations, impact assessments, any costing exercises (against which to measure the adequacy of budget allocations), financial management agencies, donor reports, etc

h. Review process – to ensure

validity, accuracy and

reliability of data

Stakeholder meetings to present preliminary findings, to discuss and influence the report recommendations. Country and external experts to review draft report – experts in health issue, public finance, budget systems etc. Internal review process within the team – sharing of experiences and solutions, editing each other’s reports. Using civil society networks to give critical feedback.

g. Data collection & management

process

Plan for using the tools to collect the information. Means of capturing data – software programmes, research assistants, data capturers.

f. Data collection tools

Questionnaires – closed/open ended, Questionnaires (mailed or in-person) Observations Interviews-personal: interviewer works directly with the respondent, can probe/ask follow up qns

f. Data collection tools-

questionnaire design-some

tips

Question Content1.How well does the qn address the content you are trying to get at. Is the Question Necessary/Useful? Are several questions needed? double-barreled question—need to split e.g. what is your occupation and monthly income?2.Does the qn have the necessary information for the respondent to be able to answer it?3.Specific-too general making information we obtain more difficult to interpret.E.g. How well did you like this workshop? 4.on some scale ranging from "Not At All" to "Extremely Well.“ What does it mean to say you liked the workshop very well? Biased or Loaded question: your own biases and blind-spots may affect the wording e.g. you may be against random street drug testing. What do you see as the drawbacks of random street drug testing? 5.Is the wording too direct?

too direct/disturbing for respondents. How did you feel about this workshop?

Conventions or rules-of-thumb in the survey design.

Start with easy, non-threatening questions, put more difficult, qns near end

Never start a mail questionnaire with an open-ended question

For filter or contingency questions, make a flowchart

Fully brief the team – project manager, executives

Key questions to be answered Objectives and methodologySampling/ selection detailsQuality control proceduresTimelinesCosts for each stageHuman resource requirements

TIP: Take time, keep records. Successful projects are well communicated, planned and every team member is clear about their role

Project planning& management

Select, train &brief field team

Select and train interviewers very carefully- match interviewer profile with

interviewee profile? - sensitive subjects - language/ cultural issues

Document the quality control procedures Train on field research ethics and specific

project requirementsTIP: Quality control all stages of fieldwork every day through the Supervisor network

Carry out a pilot exercise

- check flow of questions, skip patterns,

- language of questions, selection procedure,

length of interview Conduct a pilot FGD

TIP: Use experienced interviewers/ moderators for good quality feedback

Pilot

Data entry,cleaning

and checking

Develop editing rules and checks for all questionnaires - Check frequency distributions - Check against known statistics - Look out for inconsistencies TIP: Apply common sense. ‘Interesting’ data are often wrong

Data analysis

Excel, Quantitative - develop analysis specification (SPSS) - use techniques to help you understand relationships in the data Qualitative (NVIRO) - develop template for verbatim transcripts - common themes and differences TIP: Think through the analysis as you design the instruments “How will I analyse this?”

Data interpretation

Revisit objectives

Triangulate with other sources

Verify prelim findings with key

stakeholders

Compare with previous work

(lit.review)

Contextualise

Find the story

Report writing and

presentation

Plan and structure to tell the story – using the data as supporting evidence Summarise the key findings Keep it short (use appendices) Illustrate with diagrams, charts, illustrative quotationsDissemination workshopPolicy briefs, media etc

TIP: Rehearse presentations, circulate reports internally for comment. The quality of the deliverable is the basis of the policy maker’s/govt’s assessment of your work

• Research does not only inform advocacy, Advocacy also makes demands of research

• Research needs to answer key questions in order to be useful for advocacy purposes

For more information contact:Urbanus KiokoCentre for Economic Governance and AIDS in Africa Email: urbanus@cegaa.orgTel: +254-720-209-100