nform training #4: collecting high quality data...
TRANSCRIPT
nFORM Training #4: Collecting High Quality Data throughout Your Organization:
Equipping Staff to Use nFORM
Grace Roemer MATHEMATICA POLICY RESEARCH
Mathew Stange MATHEMATICA POLICY RESEARCH
Purpose of Today’s Presentation
• To help grantees plan for collecting high quality performance measures data – Performance measurement requirements – Data collection planning and implementation – nFORM and data collection resources – Special data collection scenarios – Answer your questions about data collection
2
Performance Measurement Requirements
• Grantees use nFORM to collect performance measures data – Client characteristics and outcomes • Applicant Characteristics Survey • Entrance Survey • Exit Survey
– Enrollment and participation (service delivery) – Program operations • All performance measures posted on FaMLE Cross-Site website:
www.famlecross-site.info
4
Performance Measures Come from Multiple Sources
What is the source? Who enters data? What is the timing
of data collection? Where are
data stored? Applicant characteristics survey Client Intake
Entrance survey Client First workshop
Exit survey Client Last workshop
Enrollment and participation data Program staff Ongoing
Grantee program operations data Program staff Quarterly
nFORM
Written Data Collection Plan
• Outline each step of the data collection process– Ideally, the data collection plan will track closely to the
nFORM user manual and provide grantees with specificinformation on using nFORM and collecting data fortheir program
• List and describe data collection steps in as muchdetail as possible
• Include optimal timing of each step, responsiblestaff person, and backup
• Summarize roles and responsibilities so staff knowwhich parts of the plan they are in charge of
8
Example Table of Contents for Data Collection Plan
• Overview of data collection
• Administrative functions and nFORM setup
• Enrolling clients • Setting up and
administering surveys – Applicant Characteristics – Entrance – Exit
• Recording services and workshop participation
• Using nFORM to monitordata collection
• Staff roles and responsibilities
• Data collection checklists • Staff training plan • Data collection process
updates
Include Procedures for Maintaining Client Privacy
• Throughout the plan, identify ways to reduce risk, and what to do ifclient privacy is compromised
• Refer to User Manual Module I and Performance Measures andData Collection Logistics document—Appendix A– Keep discussions about clients confidential– Secure computer accounts– Use password protected screen savers– Store paper materials with personally identifiable information
(PII) in locked cabinets and shred when done• Consult your Institutional Review Board (IRB) if applicable
Include a Roles and Responsibilities Table
• Identify roles of each staffmember– Creating nFORM client
records– Connecting devices to the
Internet– Generating passcodes for
surveys– Handing out devices and
passcodes– Confirming completion of
surveys
Demo Grantee Roles and Responsibilities (August 2017) Names Roles Responsibilities Staff 1 Project
Director Work with site to plan data
collection Staff 2 Survey
Administrator Test Internet and nFORM access Set up surveys
Staff 3 Facilitator Run workshop and record workshop attendance in nFORM
Staff 4 Data Manager Run the Survey Completion operational report to confirm
clients are enrolled
11
Document Data Collection Logistics
• Logistics include: – Obtaining enough tablets or computers for clients to complete
surveys – Arranging for separate computers for program staff to generate
survey passcodes – Providing clean earbuds for clients to listen to survey questions – Bookmarking the survey URL on devices clients will use
https://www.famlecross-site.com/survey
12
Document Data Collection Logistics
• Connecting to the Internet in locations where clients willcomplete surveys– Test Internet connections in advance– Make sure there is enough bandwidth to support survey
administration– Determine whether locations have firewalls that prevent
access• Consider using mobile jetpacks if Internet is not available
– Ensure that jetpacks are optimized for the number of devices accessing nFORM and the surveys
– Test the strength of the jetpack signal in the location inwhich they will be used
Wi-Fi
13
Include Checklists for Data Collection Days
• Create a checklist for each stage of the data collection process – Ideally, tailor this to
constraints of location where surveys will be administered
– Include set-up and administration steps
• Checklists help the processrun smoothly and prevent missed steps fromdisrupting data collection
Applicant Characteristics Survey Setup Up to one hour before the class
begins, generate the Applicant Characteristics Survey passcode foreach client
Write the clients’ nFORM client ID numbers and passcodes on indexcards
Turn on the tablets or computers that clients will use to complete the survey
Open the bookmarked survey URL on each device
Plug earbuds into each device
Train Staff on Data Collection Processes
• Training approach should consider size and experience of your team
• Ask staff to study the data collection plan • Talk through the plan as a group
– Answer staff questions – Discuss logistical concerns – Incorporate staff feedback as needed
• Practice the data collection process with staff • Repeat this process regularly to ensure that it meets current needs
and that staff remain familiar with it
15
Create Client Records in nFORM
• Describe what information is needed – Names – Dates of birth – Contact information
• Discuss confidentiality procedures and IRB approval (if applicable)
• Identify eligible clients – No one under age 13 – Client records cannot be deleted
• Once the information is obtained,complete an application form for each eligible client to create nFORM clientrecord and client ID
19
Administer Surveys (1)
• Generate survey passcodes for clients • Introduce the applicable survey to clients • Hand out tablets or computers • Give each client his or her client ID and passcode • Ask client to log into their survey and confirm their name • Answer any client questions during the survey • Ask client to submit their survey when they are finished • Ask clients to bring their device to program staff, or raise their hand
to signal they are done • Confirm that each client submitted the survey
18
Administer Surveys (2)
• Same basic administration steps for each survey• Entrance survey at beginning of first workshop session
occurrence– Plan short activity to occupy clients while others finish
• Exit survey at end of last workshop session– Must be at least 28 days after Entrance Survey– Follow up with clients who have dropped out of the
program
19
Record Service Delivery Information in nFORM
• Staff record information in nFORM on all services provided to clients – Workshop attendance – Individual service contacts
(such as one-on-one case management) – Referrals – Incentives
20
Use nFORM to Monitor Data Collection (1)
• Client profiles• Operational reports• Query tool• Data export• QPR/PPR• Your monitoring approach should consider pace of client
enrollment and level of services provided
21
Use nFORM to Monitor Data Collection (2)
• Client profile includes client-specific reminders• Client Status and Survey Completion operational reports
– Confirm that all clients have completed enrollment• Enrollment is completed by submitting the Applicant
Characteristics web survey• Service delivery information and workshop attendance
cannot be recorded before enrollment date– Identify clients who need to make up surveys
• Series Session Attendance operational report– Ensure that attendance is recorded for each session
• Additional analysis options: Query tool and data export
22
F #HMRFPavingTheWay
Charlie Brown (Client 10 40001033)
Profile WoOcshops I Sessooos
Progam Information rJD
Marybelh Case Manager; Mathew Case Manager; Mathew Site Administrator Enrollment Date 9120/2015
Client Status Active
Slatus Change 9/20/2015 Cltent &.veys Date
Type 5catus Action
Chen.1 lnfomrat1on rJD Appficant Characteristics Survey Complete~ +~foa M
Entrance Survey 1 ncOO'l>lete •it Exit Survey lncOOl>lete Mi·
.... Appllcadon Date 9120/2015
Population Youth
Grantee location Nofthside
SeJVice Enhanced Services Type #Provided Most Recent
Assignment Service Contacts 11/ 100015 Date of Birth 1W30/1973
A 0 Referrals Follow up nffded 11/100015 Not screened for mtirrote partner violence Incentives
~ Contact Information
230 Main St. Workshop Surrmary
Anytown NJ 08888 Session(s) Attended last Session Attended Next Meeting Date : (908) 555-1032 Name I
Healthy Marriage WOlbhop 1 11/04/2015 11/25/2015 Additional Contacts
Healthy Marriage WOlbhop 3 0 11/19/2015 0 No add1t1onal cootact(s) have been added
HMm
Use Client Profile as Case Management To-Do List
la
HMmF #HMRFPavingTheWay HM.nrYllMllA.:&~M--
Gnntee (iimo Grantee ::::J
I~ ~ LJof4 ~ ~I ~ I ! Fm< I Next Iii\ · ~
Client Status Report Detail as of 8/31/2016 Client Grantee I Chent IO Couple IO Client~ Name Chent First Name Enrollment Date I Chent Status Loe<1t1on
10001439 Bird Tweety Duplicate pending
10001701 22715 Mertz Fred Applicant pending enrollment
Center City 10006599 Allen 1im 8.11812016 Duplicate confirmed
Center City 40001036 Chipmunk Alvin the 8.19/2015 Active
Center City 40001019 CO'(ote Wile E Applicant pending enrollment
Center City 40001038 Cricket Jiminy 812212015 Ac tive
Center City 40001003 Duck Daffy Applicant pending enrollment
Center City 40001005 Gorilla Magi Ila Applicant pending enrollment
Center City 40001026 10092 Jetson George 8.17/2015 Active
Center City 40001027 10092 Jetson Jane 12/1312015 Active
Center City 10001866 24355 Johnson Bill Applicant pending enrollment
Center City 10001879 24355 Johnson Pat Applicant pending enrollment
Center City 40001013 Leghorn Foghorn Applicant pending enrollment
Center City 40001047 Martian Marvin the Duplicate pending
Center City 40001040 McGraw Ouickdraw 911512015 Duplicate confirmed
Center City 10001484 22715 Mertz Ethel 412912016 Active
Center City 40001015 Mertz Fred Applicant pending enrollment
Client Status Operational Report
24
St.rt u oit.. ~15t_61_2_0_16 ______ ~1 ii1!!!1 End Date I SJ27J2i0~.6 lii1!!!1
Gr~ntee I o .. mo Gr~nt:ee "' I
~<4 ~ LJ IOI l ~ ~~ ~ I to0% 'V I I I Find I f\=t ----------------------------~
Survey Completion Detail as of 5/6/2016 Survey Completion Date
"" -Adult couple 10001701 Matt Fred Applicant p:mdimg i emmllment
Adult indM:!wl 10001439 Bird Tw~ty Duplicate p:mdiing
C
C
G
C
ent'=ir Oty Adult couple 40001026 Jet=on G=arge· Att:i'.o,: 8/7/Z015 11/ 9/2015
ent=ir Oty Adult oouple 40001027 Jet=on Jane Acti\'<2 12/Ul:!Ol S L2/13/2015
ent=r Oty Adult oouple 10001484 IMelrtt Etlrsl Arti'li''2 4/29/2016 4/29/2016
ent=r City Adult oouple 40001029 Rubble Barney Ai:th·a 9/ t 4/20t 5 11/ 9/2015
HMIRF #HMRFPavingTheWay HM.nrYllMllA.:&~M--
Survey Completion Operational Report
25
Client Status Report Summary as of 8/4/2017 Grantee Total
(excl Duplicates)
Applicant Pending
Enrollment
Active Completed/Graduated
Consent Revoked
Drop out
Demo Grantee 75 10 62 2 0 0
The program enrolled its new cohort of clients. Does this report show any issues?
Ten applicants have not completed
enrollment. Why might this
happen?
Client ID Couple ID Client Last Name
Client First Name
Enrollment Date
Client Status
10011526 Krueger Freddy Applicant pending enrollment
10011924 31167 Lyman Josh Applicant pending enrollment
10001701 22715 Mertz Fred Applicant pending enrollment
40001019 Coyote Wile E Applicant pending enrollment
10013595 Davis Bette Applicant pending enrollment
40001003 Duck Daffy Applicant pending enrollment
40001005 Gorilla Magilla Applicant pending enrollment
Ten applicants have not completed enrollment. How do you find them?
Use Excel to sort the detail section of the
Client Status operational report.
Survey Completion Summary
Clients Enrolled 4/1/2014 - 8/4/2017
Number of Clients who Completed Grantee Population Applicant
Characteristics Survey
Entrance Survey Exit Survey
Demo Grantee All 33 0 0 Adult couple 0 0 0 Adult individual 33 29 0
Youth 0 0 0
Clients completedtheir entrance survey during thefirst workshopsession. Does this report show any issues?
Four clients did not complete their
entrance survey.
Population Client ID Client Last
Name
Client First
Name
CurrentClient Status
ApplicationDate
Applicant Characteristics Survey*
Entrance Survey
Exit Survey
Adult individual 10006599 Allen Tim Active 4/9/2017 4/9/2017
Adult individual 10011872 Bishop Winston Active 3/31/2017 3/31/2017
Adult individual 40001038 Cricket Jiminy Active 3/31/2017 3/31/2017
Adult individual 10008681 Day Jessica Active 3/31/2017 3/31/2017
Adult individual 40001035 Mouse Mickey Active 3/31/2017 3/31/2017 4/10/2017
Adult individual 10008827 Miller Nick Active 4/2/2017 4/2/2017 4/10/2017
Adult individual 40001017 Bird Tweety Active 4/2/2015 4/2/2015 4/10/2017
Four clients did not complete their survey. How do you identify them?
Sort the detail section of Survey Completion operational report to identify those clients.
Authorization(s) by Active User Date of
User Last Name
User First Name
User Type User Name (email) Account Creation
Last Login # Days since Last
Login Activity
User AccountStatus
Case Manager Marybeth General [email protected] 9/17/2015 4/6/2016 485 Disabled
Site Admin Mark Site Administrator [email protected] 9/17/2015 7/21/2017 14 Active
Site Administrator
Grace Site Administrator [email protected] 9/17/2015 8/4/2017 0 Active
Site Admin Gene Site Administrator [email protected] 3/29/2016 8/1/2017 3 Active
Site Administrator
Mathew Site Administrator [email protected] 9/17/2015 8/4/2017 0 Active
General Mathew General [email protected] 9/17/2015 687 Disabled
Site Administrator
Marybeth Site Administrator [email protected] 9/17/2015 9/1/2016 337 Disabled
The grantee has four current staff who use nFORM. Does this report show any issues?
No issues! Site administrators
can disable accounts for staff who no longer work for the program.
Accounts will automatically disable
if inactive for 180 days.
Clients enrolled in the program this grant year. Does the query tool show any issues?
Clients have not engaged in services
quickly.
A case managerleaves nFORM’s All Clients screen openwhen he walks across the room to answer a client’s question about theentrance survey. Are there any data collection issues?
To protect client personally identifiable
information, never leave nFORM open
when not in use.
Caseload Summary as of: 8/4/2017 Staff with Case Management Capability
Staff User Account Type
User Account Active?
Number of Active clients
Total number of clients
Site Admin, Mark Site Administrator
Y 2 4
Site Administrator, Grace Site Administrator
Y 2 3
Site Administrator Gene Site Administrator
N 3 4
Unassigned N/A N/A 19 65 Site Administrator, Mathew Site
Administrator Y 7 9
Site Administrator, Marybeth Site Administrator
N 4 8
Average Caseload 3.7
The programrequires allclients be assigned to a case manager. Does this report show any issues?
19 active clients are not assigned to a case
manager.
Sort the All Clients Search to Find Clients Not Assigned to Case Managers
Client ID . ... Last Name~
First
Name. ...
Service
Assignment. ...
Grantee
Location . ... Case Manager(s)
Application
Date . ...
Client
Status . ...
Ell•M•IM•ll Bunny Bugs Enhanced
Services
Northside 8/412015 Active
E H ll·l·l·ll·! Cricket Jiminy Enhanced
Services
Center
City
812212015 Active
Ell •l•l•MHI Day Jessica Center
City
10114/2016 Active
El l l·l·l·ll·tl Fatale Natasha Control - No
Services
Northside 8/6/2015 Active
Ell•M•ll•til Jetson George Control - No
Services
Center
City
81712015 Active
E IM·l·ll·tl Jetson Jane Control - No
Services
Center
City
81712015 Active
EIM•l•Hill Malfoy Draco Enhanced
Services
Southside 31712016 Active
HMIRF #HMRFPavingTheWay HM.nrYllMllA.:&~M--
Survey Completion Summary
Clients Enrolled 4/1/2014 - 8/4/2017 Number of Clients who Completed
Grantee Population Applicant Characteristics Survey
Entrance Survey Exit Survey
Demo Grantee All 33 0 0 Adult couple 0 0 0
Adult individual 12 0 0
Youth 21 0 0
The grantee set enrollment targets only for Youth. Does this report show any issues?
12 clients were enrolled as Adult
Individuals. The client population can be edited until theclient completes their
entrance survey.
The program provides 3 curricula. What is wrong with how these workshops are set up?
One Curriculum should NOT be
Entered as Multiple Workshops
Special Data Collection Scenarios
• Performance Measures and Data Collection Logistics Document providesguidance on each of these scenarios: – Youth in schools – Incarcerated individuals – Administering surveys without internet access – Languages other than English or Spanish
• Work with a point of contact ahead of data collection to determine how to administer client surveys in special settings – Incorporate into data collection plan – Plan for group administration (if applicable) – Plan to offer make-up surveys to clients if needed
• Never administer Entrance and Exit Surveys on the same day
38
Administering Surveys Without Internet Access
• Paper copies of surveys (English and Spanish) available for very rare instances when Internet access is not available – Requires FPS approval
• Data collection plan should include staff responsibilities and timelines for data collection, data entry, and secure destruction of paper surveys
• Clients can complete survey on their own, or staff can read questions to clients with low literacy and help navigate skip patterns
• Record client ID number and date of administration in the designated place on paper survey
• Staff should input responses into nFORM same day if possible 39
Data Collection for Other Languages
• Plan ahead for clients who speak other languages – Review the survey and translate it in advance, not on
the fly • Grantee staff (or interpreter) can read translated
surveys to clients who speak languages other than English or Spanish • Enter the client’s responses into nFORM same day if
possible
40
Recap: Key Considerations
• Determine data collection logistics wellbefore data collection begins– Make sure to have enough devices for clients and
reliable Internet access– Work through any special data collection scenarios
• Prepare a written data collection plan– Practice the plan and revise as needed– Train staff on the plan– Schedule enough staff to perform data collection
within the time frame available
41
Recap: Privacy
• Client privacy is paramount• Follow procedures to safeguard client
information– Module I of the nFORM User Manual – Data Collection and Logistics Document– Follow your IRB procedures, if applicable
42