audit report on the vera institute of justice contract

Audit Report on the Vera Institute of Justice Contract with the Department of Probation to Operate the Esperanza Program ME07-133A June 26, 2008

THE CITY OF NEW YORK

OFFICE OF THE COMPTROLLER 1 CENTRE STREET

NEW YORK, N.Y. 10007-2341 ─────────────

WILLIAM C. THOMPSON, JR. COMPTROLLER

To the Citizens of the City of New York Ladies and Gentlemen: In accordance with the responsibilities of the Comptroller contained in Chapter 5, § 93, of the New York City Charter, my office has conducted an audit to determine whether the Vera Institute of Justice complied with the provisions of its contract with the New York City Department of Probation (DOP) to operate the Esperanza/Hope (Esperanza) program. Under the contract with DOP, Esperanza provides family-based, intensive-treatment services to juvenile delinquents who would otherwise be placed in a State facility by Family Court and assists DOP implement structural changes within the department. We audit contracts such as this to ensure that entities with City contracts comply with the terms of their agreements. The results of our audit, which are presented in this report, have been discussed with DOP officials and their comments have been considered in preparing this report. Their complete written response is attached to this report. I trust that this report contains information that is of interest to you. If you have any questions concerning this report, please e-mail my audit bureau at [email protected] or telephone my office at 212-669-3747. Very truly yours,

William C. Thompson, Jr. WCT/ec Report: ME07-133A Filed: June 26, 2008

Table of Contents

AUDIT REPORT IN BRIEF ....................................................................................................... 1

Audit Findings and Conclusions................................................................................................. 1

INTRODUCTION......................................................................................................................... 4

Background................................................................................................................................. 4 Objective..................................................................................................................................... 5 Scope and Methodology ............................................................................................................. 6 Discussion of Audit Results........................................................................................................ 7

FINDINGS AND RECOMMENDATIONS ............................................................................... 8

DOP’s Esperanza Contract Lacks Sufficient Incentives............................................................. 8 Recommendations............................................................................................................. 14

Esperanza’s Cost-Savings Estimates Are Questionable ........................................................... 14 Recommendations............................................................................................................. 15

Esperanza’s Performance-Based Claims for Reimbursement Were Generally Supported Adequately, But Support for Its Non-Performance-Based Claims Was Seriously Deficient ........................................................................................................... 16

Recommendations............................................................................................................. 20 ADDENDUM DOP Response

Office of New York City Comptroller William C. Thompson, Jr.

The City of New York Office of the Comptroller

Bureau of Management Audit

Audit Report on the Vera Institute of Justice Contract with the Department of

Probation to Operate the Esperanza Program

ME07-133A

AUDIT REPORT IN BRIEF This audit determined whether the Vera Institute of Justice (Vera) is complying with the provisions of its contract with the Department of Probation (DOP) to operate the Esperanza/Hope (Esperanza) program. The contract, which was signed in January 2003, requires that Esperanza provide family-based, intensive-treatment services to juvenile delinquents who would otherwise be placed in a State facility by Family Court. In addition to providing direct services to delinquent youths, the Esperanza contract calls for the contractor to aid DOP in implementing structural changes within the agency. In Fiscal Year 2007, Esperanza provided direct services to 160 youths and was paid a total of $3,199,263. Audit Findings and Conclusions

The audit revealed that Vera generally complied with the provisions of its contract with DOP to operate the Esperanza program during Fiscal Year 2007. There was evidence that Esperanza’s performance-based claims for reimbursement were generally supported, that field counselors provided direct services to Esperanza clients, and that required technical assistance and reports were provided to DOP.

However, the audit also concluded that there were serious weaknesses in DOP’s

Esperanza contract and in its oversight of contract compliance. DOP’s contract lacks sufficient incentives with regard to direct client services. In addition, DOP’s review of Esperanza’s non-performance-based claims for reimbursement was seriously deficient.

With regard to the contract, only 15 percent of payments are for meeting performance measures regarding direct services to clients; another 13 percent of contract payments relate to other deliverables, such as the provision of technical assistance and reports to DOP. (The remaining 72 percent is to reimburse Esperanza for staff salaries, executive support, and overhead costs.) With regard to performance incentive measures, Esperanza consistently fell short of meeting the target of 54 client enrollments per quarter in Fiscal Year 2007. Although


2

Esperanza exceeded DOP’s target rate of 70 percent for clients avoiding placement in a State facility (placement) within nine months of enrollment, the audit questions whether the 70 percent target is adequate. The audit also questions Esperanza’s methodology for estimating the City cost savings attributable to Esperanza’s placement-avoidance efforts.

Regarding Esperanza’s non-performance-based claims for reimbursement, the audit identified significant deficiencies in DOP’s oversight of contract compliance. DOP does not require supporting documentation from Esperanza for its staff salary, executive-support, or overhead cost claims, nor does it periodically visit Esperanza to review any of this documentation. Although Esperanza generally maintained adequate records on staff salaries and benefits, it was unable to provide a clear accounting for its executive-support costs (of over $80,000) and overhead costs (of over $125,000) and failed to provide any documentation to support its overhead costs. As a result, DOP and Esperanza were unable to demonstrate that these funds were used in an appropriate manner. By failing to properly monitor executive-support and overhead expenses, DOP is increasing the risk that City funds will be misused.

Audit Recommendations To address these issues, the audit recommends, among other things, that DOP:

• Expand the performance-based component of future contracts providing family-based, intensive-supervision alternatives to placement in State facilities.

• Strengthen the placement-avoidance performance measure in the contract.

• Recoup all funds relating to special compensation, executive-support, and overhead

costs in Fiscal Year 2007 for which Esperanza is unable to provide adequate supporting documentation.

The audit also recommends that Esperanza:

• Improve its cost-savings report by (a) adjusting its placement-avoidance savings estimate to account for clients who are sentenced to incarceration by an adult court within three years of exiting the Esperanza program, and (b) subtracting Esperanza’s costs from the placement-avoidance savings estimate.

• Improve support for its cost-savings reports by providing credible evidence for the

discount rate it uses to adjust estimated savings to account for projected future placements.

• Account for and maintain adequate documentation on its special compensation,

executive-support, and overhead expenses.


3

Agency Response In its response, DOP agreed or partially agreed with four recommendations and disagreed with four, but stated that the successor contract will address the audit’s concerns as is consistent with the purpose and mission of the Esperanza program.


4

INTRODUCTION Background

The Department of Probation (DOP) endeavors to promote public safety by monitoring and enforcing conditions of probation. It is required to supply information and recommendations to the courts on both adult and juvenile cases. The department annually serves over 50,000 adult probationers and provides intake, investigation, and probation supervision services for more than 25,000 juveniles each year.

The DOP Juvenile Operations Division identifies, assesses, and addresses individual needs that contribute to criminality among youths. It works with schools, community-based organizations, and others within the justice system, in conjunction with juvenile delinquents and their families, to advance law-abiding behavior.

According to an Independent Budget Office (IBO) report,1 it costs the State and the City

about $75,000 each annually to keep an adjudicated youth (younger than 16) in a confinement facility operated directly by or under contract with the State Office of Children and Family Services (OCFS). State research indicates that youths discharged from OCFS facilities have often had poor outcomes. According to a 1999 State study, within nine months of release, approximately 50 percent of the youths studied had been re-arrested. According to the research, by the 36th month after release, 81 percent of boys and 45 percent of girls had been rearrested.2

The Vera Institute of Justice (Vera) is an independent center for policy and practice with

the goal of making justice systems fairer and more effective through research and innovation. In January 2003, DOP signed a contract with Vera for the Juvenile Home Placement Program.3 This program, later named Esperanza/Hope, began to provide direct services in May 2003. Esperanza is an intensive, family-centered supervision program used to develop interventions to safely reduce the City’s use of incarceration in juvenile delinquency cases. Vera eventually established Esperanza as a not-for-profit corporation. In January 2007, Vera drafted an independence agreement with Esperanza that set a course of action for Esperanza’s eventual independence from Vera.

DOP’s contract with Vera requires that Esperanza provide family-based, intensive-

treatment services to juvenile delinquents who would otherwise be placed in a State facility by Family Court. In addition to providing direct services to delinquent youths, the Esperanza contract calls for the contractor to aid DOP in implementing structural changes within the agency. According to DOP, Esperanza provides: (1) ongoing analysis of juvenile justice system trends in the City; (2) reports and technical assistance supporting efforts to reduce reliance on placements in State facilities; (3) analyses regarding the frequency with which judges follow

1 Alternative to Jail Programs for Juveniles Reduce City Costs, Inside the Budget, Independent Budget Office, July 11, 2006. 2 Bruce Frederick, Ph.D., Factors Contributing to Recidivism Among Youth Placed With The New York State Division For Youth, New York State Division of Criminal Justice Services (1999).

3 The contract was retroactive to July 1, 2002.


5

DOP’s sentence recommendations, and (4) analyses of the demographics of the most serious offenders in the Family Court system. In addition, Esperanza has developed an evaluation instrument called the Probation Assessment Tool (PAT) whereby DOP uses a systematic method of assessing each youth.

Esperanza is reimbursed in large part based on salary and overhead expenses but is to

some extent reimbursed based on client enrollments, clients’ avoidance of placement in a State facility (placement), and the provision of deliverables, such as analyses, reports, technical assistance, and training to DOP.

Esperanza’s goal is to provide direct services to juvenile delinquents4 who enter the

program after the disposition of their Family Court delinquency case, either as part of a conditional discharge or in conjunction with probation. Esperanza provides comprehensive services to youths and their families. This includes an intensive treatment intervention that provides therapeutic services in the home, as well as ongoing case and crisis management, generally over a four to six month period. From the beginning of the young person’s entry into the program, an Esperanza field counselor focuses on engaging the youth and family in the treatment process and helping them to develop goals that build on personal and family strengths and address areas they would like to change.

Active engagement with Esperanza is divided into three phases. In the first phase, an

Esperanza field counselor assesses whether there are any imminent crises in the household that need to be immediately addressed in order to facilitate productive engagement. In addition, goals are set, a treatment plan is developed, and house rules are established. The second phase focuses on creating and assessing positive changes in clients’ lives. In the final phase, the aim is for the youth and family to integrate the positively defined roles, behaviors, and house rules into daily living, and for them to continue with successful interactions as they move toward autonomy as a family unit. According to DOP, Probation Officers (POs) maintain the lead role in assessing youths’ compliance with their conditions of probation and work closely with Esperanza’s counselors as part of the treatment team.

DOP’s current contract with Vera is a $4.8 million contract extension for January 1, 2007, to June 30, 2008. The previous contract with Vera was also for $4.8 million and was in effect from July 1, 2005, to December 31, 2006. Esperanza submits quarterly claims for payment under the contract. In Fiscal Year 2007, Esperanza provided direct services to 160 youths and was paid a total of $3,199,263. Objective

The objective of this audit was to determine whether Vera is complying with the provisions of its contract with DOP to operate the Esperanza program.

4 A juvenile delinquent is a youth between the age of 7 through 15 who commits an act which would be a crime if the youth was an adult. Esperanza can serve youths who are 16 or older if their offense was committed prior to the age of 16.


6

Scope and Methodology The scope period covered by this audit was July 1, 2006 through June 30, 2007 (Fiscal Year 2007). To gain an understanding of DOP policies, procedures, and practices concerning the oversight of its contract with Vera to operate the Esperanza program, we conducted interviews with DOP officials and probation officers in the Juvenile Operations Division. We also interviewed officials from Vera and Esperanza, including Esperanza field supervisors and counselors. Finally, DOP’s and Esperanza’s policies and procedures relating to the program were reviewed. To determine Vera’s compliance with the provisions of its contract with DOP, we reviewed quarterly claims submitted by Esperanza for January through March 2007 and April through June 2007. We obtained a list of all Esperanza cases that were closed during these two quarters. According to the list, there were 80 such cases. The 80 cases were separated into two categories—those clients who successfully completed the Esperanza program, a total of 53, and those who, due to noncompliance, were terminated from the Esperanza program, a total of 27. For the 53 cases that successfully completed the program, we checked the State’s Universal Case Management System (UCMS) to determine whether any of those clients were placed in a State facility during the same periods of time that Esperanza claimed to provide services to them.

The reliability of selected data generated from Esperanza’s Program Management Application (PMA) system was assessed. We randomly selected 30 of the 80 clients whose cases were closed from January 1, 2007 through June 30, 2007. For the 30 cases, we compared client data in PMA (including client name, docket number, enrollment date in Esperanza, and case-closing date) to the information in hard-copy client files to determine whether the information matched. Information in a separate randomly selected sample of 30 case files was compared to the data contained in PMA.

To determine whether there was sufficient evidence that Esperanza field counselors provided direct services to Esperanza clients, we reviewed case notes of contacts and visits made to 21 randomly selected youths of the 80 whose cases were closed by Esperanza during the third and fourth quarters of Fiscal Year 2007. This included 14 clients who successfully completed the program and 7 clients who were terminated from the program.

To determine whether Esperanza’s performance-based claims for January through June

2007 were properly supported, we reviewed client enrollment and placement information, required reports, and other related documentation. To determine whether Esperanza had adequate documentation to support claims for non-performance-based reimbursement, we reviewed selected financial documents for the period of April through June 2007. In addition, we determined whether all individuals receiving paychecks at Esperanza were bona fide employees. On March 14, 2008, Esperanza’s payroll distribution at its office at 636 Broadway in Manhattan was observed.


7

We determined whether Vera’s contract with DOP was registered with the Comptroller’s Office, as required by Chapter 13, §328, of the New York City Charter.

The results of the above tests, while not statistically projected to their respective

populations, provide us with a reasonable basis to assess Esperanza’s compliance with the provisions of its contract.

This audit was conducted in accordance with generally accepted government auditing standards (GAGAS) and included tests of records and other auditing procedures considered necessary. This audit was performed in accordance with the audit responsibilities of the City Comptroller, as set forth in Chapter 5, §93, of the New York City Charter. Discussion of Audit Results The matters in this report were discussed with DOP and Esperanza officials during and at the conclusion of this audit. A preliminary draft report was sent to DOP officials on April 22, 2008, and was discussed at an exit conference held on May 6, 2008. On May 23, 2008, we submitted a draft report to DOP officials with a request for comments. We received a written response from DOP officials on June 9, 2008. DOP agreed or partially agreed with four recommendations and disagreed with four, but stated that “the successor contract will address the audit’s concerns as is consistent with the purpose and mission of the program.”

The full text of the DOP response is included as an addendum to this report.


8

FINDINGS AND RECOMMENDATIONS

Our audit revealed that Vera generally complied with the provisions of its contract with DOP to operate the Esperanza program during Fiscal Year 2007. There was evidence that Esperanza’s performance-based claims for reimbursement were generally supported, that field counselors provided direct services to Esperanza clients, and that required technical assistance and reports were provided to DOP. Finally, our data reliability assessment concluded that Esperanza’s PMA data were generally reliable.

However, the audit also concluded that there are serious weaknesses in DOP’s Esperanza

contract and in its oversight of contract compliance. DOP’s contract lacks sufficient incentives with regard to direct client services. In addition, DOP’s review of Esperanza’s non-performance-based claims for reimbursement was seriously deficient.

With regard to the contract, only 15 percent of payments are for meeting performance measures regarding direct services to clients; another 13 percent of contract payments relate to other deliverables, such as the provision of technical assistance and reports to DOP. (The remaining 72 percent is to reimburse Esperanza for staff salaries, executive support, and overhead costs.) With regard to performance incentive measures, Esperanza consistently fell short of meeting the target of 54 client enrollments per quarter in Fiscal Year 2007. Although Esperanza exceeded DOP’s target rate of 70 percent for clients avoiding placement in a State facility within nine months of enrollment, the audit questions whether the 70 percent target is adequate. The audit also questions Esperanza’s methodology for estimating the City cost savings attributable to Esperanza’s placement-avoidance efforts.

Regarding Esperanza’s non-performance-based claims for reimbursement, the audit identified significant deficiencies in DOP’s oversight of contract compliance. DOP does not require supporting documentation from Esperanza for its staff salary, executive-support, or overhead cost claims, nor does it periodically visit Esperanza to review any of this documentation. Although Esperanza generally maintained adequate records on staff salaries and benefits, it was unable to provide a clear accounting of its executive-support costs (of over $80,000) and overhead costs (of over $125,000), and it failed to provide any documentation to support its overhead costs. As a result, DOP and Esperanza were unable to demonstrate that these funds were used in an appropriate manner. By failing to properly monitor executive-support and overhead expenses, DOP is increasing the risk that City funds will be misused.

These findings are discussed in more detail in the following sections of the report.

DOP’s Esperanza Contract Lacks Sufficient Incentives DOP’s Esperanza contract is primarily a cost-reimbursement contract. Only 28 percent of the claims Esperanza submits for payment directly relate to performance measures or assigned tasks—the remaining 72 percent of the contract amount (about $3.5 million of the $4.8 million


9

budgeted for Esperanza in the 18-month contract) is for fixed costs relating to Esperanza’s personnel and overhead expenses. Including more incentives in the contract could encourage an enhanced level of performance from Esperanza. The Esperanza contract identifies four cost categories: (1) field, screening, and community services staff salaries and benefits, as well as executive-support and overhead costs, (2) production of monthly indicator and quarterly cost-savings reports, (3) achievement of performance measures, and (4) structural changes, technical assistance, and reporting. We consider claims under the first cost category to be non-performance-based claims, and claims under the remaining three categories to be performance-based claims. Table I below lists the four cost categories and the budgeted and percentage amounts for each.

Table I Budgeted and Percentage Amounts For Each Cost Category of Contract

Fiscal Year 2007

Cost Category Minimum Budgeted Amount

Percentage Of

Total Budgeted Amount

Reimbursement of costs for field, screening, and community services staff; executive support; and overhead

$2,329,754

72%

Achievement of performance measures: enrollment of clients with Esperanza and avoidance of placement for nine months after intake.

$480,000 15%

Structural Changes, Technical Assistance, and Reporting

$328,500

10%

Production of monthly indicator and quarterly cost-savings reports

$96,000 3%

Totals $3,234,254*

100%

* Although the minimum budgeted amounts in each cost category add up to $3,234,254, the contract states that the total budget for one year of the 18-month contract is $3,199,263.

As shown in Table I, 72 percent of the total budgeted amount related to salary and overhead costs, while only 15 percent of the budgeted amount related to the achievement of performance


10

measures (relating to enrollment and placement avoidance) and 13 percent to Esperanza’s structural change, technical assistance, and reporting efforts. The Esperanza contract sets a target of 54 client enrollments per quarter. However, Esperanza consistently fell short of this target number, as shown in Table II below.

Table II Comparison of Target and Actual Client Enrollments

Fiscal Year 2007

Period Target

Measure

# of Youths Enrolled by Esperanza

# of Youths Enrolled

Above (Below) Target

Measure

% Above (Below) Target

Measure Jul-Sep 2006 54 36 (18) (33%)

Oct-Dec 2006 54 48 (6) (11%)

Jan-Mar 2007 54 46 (8) (15%)

Apr-Jun 2007 54 30 (24) (44%)

Totals 216 160 (56) (26%) DOP pays Esperanza $100,000 per quarter if it enrolls the target number of 54 youths. Esperanza receives $2,000 for each additional youth that it enrolls above the target number and is penalized $2,000 for each youth it fails to enroll below the target number. DOP penalized Esperanza $112,000 for falling 56 (26%) short of its annual goal of 216 enrollments. However, this penalty represents less than 3.5 percent of the annual budgeted amount of $3.2 million for this contract.

Esperanza and DOP officials argued that in order for the program to be effective, it has to be selective in the youths it accepts. Esperanza endeavors to ensure that it is enrolling the types of clients that best fit the program. In addition to being otherwise bound for a State facility, to be accepted into the Esperanza program a youth must meet the following three criteria: (1) the client cannot be suffering from active psychosis, (2) the client must consent to the program, and (3) the client must have a caregiver willing to participate in the program.

Nevertheless, we believe that Esperanza might be able to improve its enrollment performance. Esperanza’s Project Manager told us that Family Court refers, on average, about 368 youths per quarter to the program. Esperanza accepts about 40 clients per quarter, or about 11 percent of the youths referred to it by Family Court. DOP should consider increasing the


11

percentage of incentive-based performance targets and reducing the high cost-reimbursement component of the contract by which Esperanza receives substantial reimbursement even if it falls far short of targeted client enrollment goals. This step could encourage Esperanza to enhance its efforts to achieve enrollment goals. After the exit conference, DOP officials submitted a written argument stating that many aspects of the enrollment process are beyond Esperanza’s control and that there are very valid reasons for limiting the number of youths it enrolls in the program. Esperanza is concerned that it does not practice “net-widening” by accepting youths who would otherwise be sentenced to probation or who have other viable alternatives to remain in the community. Esperanza is also concerned that it does not accept youths who pose a potential risk to public safety. Furthermore, DOP officials stated that Esperanza does not have control over the number of referrals they receive nor the percentage of referrals that are viable. We acknowledge the obstacles that Esperanza faces in meeting its target enrollment goals. Nevertheless, DOP, in conjunction with Esperanza, set these targets. In fact, DOP has imposed penalties on Esperanza, as allowed under the contract, due to the agency not meeting enrollment goals. In light of the noted obstacles, if the goal of enrolling 54 clients per quarter is unattainable, then the goal could be adjusted. Such an adjustment could be made while simultaneously expanding the performance-based component of the contract.

Another option would be to identify enrollment goals but allow DOP discretion to determine whether a good-faith effort was made to achieve them. For example, DOP could review a sample of rejected cases to determine whether Esperanza’s rejections of potential clients were legitimate. By expanding the performance-based component of the contract, DOP would obtain additional leverage in working with Esperanza on this issue. Of course, if enrollments fall significantly short of anticipated levels, even if for legitimate reasons, a mechanism should be included in the contract to reduce DOP’s compensation to Esperanza to reflect the lower than expected number of clients who would be receiving services. The Mayor’s Office of Contract Services Agency Procurement Indicators report for Fiscal Year 2004 stated that City contracts representing 57 percent of the dollar value of all City contracts awarded during the fiscal year were substantially linked to performance. (A contract is considered to be substantially linked to performance if more than three-fourths of the total contract budget is linked to performance.) As indicated above, about 72 percent of the Esperanza contract is based on inputs and only about 28 percent is based on outcomes and deliverables. The only other performance measure in the Esperanza contract is a target percentage of Esperanza’s clients who avoid placement during the nine-month period following enrollment. DOP set the target at 65 percent during the first two quarters of Fiscal Year 2007 and at 70 percent during the last two quarters of the fiscal year. According to its Fiscal Year 2007 quarterly claims for reimbursement, Esperanza consistently met those standards, with 76 percent of its clients avoiding placement in an OCFS facility within nine months of enrollment. However, it is questionable how rigorous those standards were.


12

DOP and Esperanza point to the 1999 State study mentioned earlier to justify the nine-month placement-avoidance standard in the contract. There are concerns about DOP’s and Esperanza’s reliance on this study. First, the State study is a dated analysis. The study published in 1999 relied on information about youths discharged from State custody between 1991 and 1995. Although DOP argues that there is no more recent study to rely on, considering changes over the years in patterns of juvenile delinquency, arrests, and sentencing, the appropriateness of such a heavy reliance on this dated study is questionable. Second, the standards and populations being discussed are not comparable. The State study refers to youths being rearrested within nine months of discharge from a State facility, while the contract’s performance measure is a different standard that relates to clients being placed in State custody (after a Family Court “fact-finding hearing”) within nine months of enrollment. The State study is silent on how many of those rearrested were placed in a State facility within nine months of discharge. However, even if the State study had included information on the subsequent placement of those rearrested, the results could not be easily applied as a benchmark in the DOP contract because the populations are dissimilar. The State study looked at all juveniles who had been placed in State custody, while Esperanza’s client population includes many youths who had never been placed.

After the exit conference, DOP argued in writing that Esperanza’s client population is

similar to youths placed in State facilities. However, our comparison was between the youths serviced by Esperanza and those discharged from State facilities, as was the case with the population from the 1999 study. In this regard, DOP’s own analysis underscores our point. According to DOP’s analysis, at least 52 percent (and most likely more) of those serviced by Esperanza had never been placed in a State facility, while, of course, all of the youths from the State study had been in placement.5

In addition, the State study states that about 50 percent of youths discharged from a State

facility were rearrested within nine months (and thereby indicates that the remaining 50 percent avoided placement). IBO data suggest that only between 19 and 33 percent of felony arrests of juveniles lead to placement in a State facility.6 Therefore, if only 50 percent of juveniles discharged from State facilities are rearrested within nine months, and if only up to 33 percent of those rearrested are placed in a State facility, then approximately 83.5 percent of those

5 DOP analyzed the prior arrest records for clients enrolled in Esperanza in Fiscal Year 2007 and found the following: 52 percent of clients had no prior arrests, 31 percent had one prior arrest, 12 percent had two prior arrests, and 5 percent had three or more prior arrests. DOP did not provide any analysis with regard to prior placements; however, it should be noted that many arrests do not result in placements. Therefore, it is very likely that the number of Esperanza’s clients who have never been in a State facility includes those who had one or more prior arrests. 6 Data suggest that a high percentage of arrested juveniles avoid placement in a State facility. According to the IBO in The Rising Cost of the City’s Juvenile Justice System (January 2008), the New York Police Department does not track the misdemeanor arrests of juveniles. The IBO reports that major felony arrests of juveniles in the City ranged between 4,000 and 5,000 each year between Fiscal Years 2003 and 2007. The IBO also reports that that there were between 952 and 1,319 placements of City youths in State facilities each year between Fiscal Years 2003 and 2007. These data suggest that between 19 and 33 percent of juvenile major felony arrests resulted in placement in a State facility (and that between 67 and 81 percent avoided placement). The percentage avoiding placement would increase when misdemeanor arrests are considered but likely decrease when only rearrests following discharge from a State facility are considered rather than all arrests.


13

discharged from a State facility avoid placement,7 which is a better result than the 76 percent rate of placement avoidance achieved by Esperanza in Fiscal Year 2007.

Furthermore, even if a client is rearrested during the nine-month period and placed in a

State facility as a result of this rearrest, as long as the placement occurs more than nine months after enrollment, this is considered to be a placement-avoidance success under the contract. In fact, DOP itself argues in writing that the court process in a juvenile delinquency case can be lengthy. DOP states that Esperanza sometimes works with youths who are as old as 18 because of the length of time involved with resolving their delinquency case. As a result, we question the adequacy of the 70 percent target for placement avoidance within nine months of enrollment as a measure of Esperanza’s performance.

On a related matter, there appears to be some confusion as to the amounts that Esperanza should be paid concerning its enrollment and placement-avoidance achievements. DOP appears to be rewarding Esperanza for exceeding its targets and penalizing it for falling short of its targets without regard to the minimum and maximum amounts indicated in the contract. The contract indicates that DOP’s quarterly payments for Esperanza’s enrollment efforts should be between $80,000 and $120,000 and that its quarterly payments for Esperanza’s placement- avoidance efforts should be between $40,000 and $60,000. However, for the fourth quarter of Fiscal Year 2007, DOP paid Esperanza $52,000 for its enrollment efforts ($28,000 below the minimum payment) when it penalized Esperanza $48,000 for falling 24 clients short of its enrollment goal. For the third quarter of Fiscal Year 2007, DOP paid Esperanza $66,000 for its placement-avoidance efforts ($6,000 above the maximum payment) when it rewarded Esperanza $16,000 for exceeding the target percentage by 16 percent.

While the contract language should be clarified, applying the minimum and maximum

amounts indicated by the contract would only exacerbate the problem of there being a high fixed cost built into the contract. If the minimum amounts apply, then an additional $120,000 ($80,000 in enrollment payments and $40,000 in placement-avoidance payments) would be guaranteed to Esperanza under the contract regardless of its performance in meeting enrollment and placement-avoidance goals.

7 The 83.5 percent figure is arrived at by adding the 50 percent of the juveniles discharged from a State facility who were not rearrested (and thereby avoided placement in a State facility) and the 33.5 percent of the juveniles discharged from a State facility who were rearrested but avoided placement. (The 33.5 percent figure represents the 67 percent [who avoided placement] of the 50 percent who were rearrested.)


14

Recommendations

DOP should: 1. Expand the performance-based component of future contracts providing family-

based, intensive-supervision alternatives to placement. DOP Response: “The Department believes that our contract meets and exceeds the requirements for a performance-based contract.” Auditor Comment: We continue to believe that including more incentives in the contract would encourage an enhanced level of performance from Esperanza. 2. Strengthen the placement-avoidance performance measure in the contract. DOP Response: “The Department believes that the measures we use are based on the best and most recent data available.” Auditor Comment: As we point out in this report, we believe that DOP should strengthen this measure by basing it on more current and comparable data, such as the IBO data presented in this report. Accordingly, we urge DOP to reconsider its response to this recommendation. 3. Clarify contract language on payments and penalties related to Esperanza’s

enrollment and placement-avoidance efforts.

DOP Response: “In the successor contract DOP will eliminate the minimum and maximum payments related to Esperanza’s performance measures but retain contract incentives and disincentives.”

Esperanza’s Cost-Savings Estimates Are Questionable

Esperanza estimated that it saved the City $5.4 million during Fiscal Year 2007 by reducing the number of juvenile delinquents placed in a State facility. However, its cost-savings reports and its savings-calculation methodology statement raise significant questions concerning the reliability of this savings estimate.

Esperanza credited itself with saving the City money in Fiscal Year 2007 if a client

whose case was closed during the fiscal year avoided placement in a State facility between the enrollment date and the end of the fiscal year. However, Esperanza informed us that it does not check adult court records or UCMS to determine whether these clients avoided incarceration during this period for an offense committed as an adult. DOP and Esperanza data indicate that 56 (65%) of the 86 youths who both entered and exited the Esperanza program in Fiscal Year


15

2007 were no longer juveniles by the end of the fiscal year.8 Furthermore, although DOP states that Esperanza tracks youths for three years to adjust the cost savings report if any of the youths are eventually placed in a State facility, such an adjustment is incomplete if adult incarcerations are not checked.

Another problem with Esperanza’s cost-savings reports is that Esperanza arrived at the

Fiscal Year 2007 cost-savings estimate by reducing an initial savings estimate by an unsupported discount rate of 19 percent to account for projected future placements (which led to a final savings estimate of $5.4 million). DOP states that the 19 percent discount rate was established as a result of an “internal analysis.” However, although requested, a copy of this analysis was not provided to us (DOP only provided a brief description of this analysis), so we are unable to assess its validity. Finally, assuming its calculations were accurate, the costs of the Esperanza program ($3.2 million in Fiscal Year 2007) were not deducted from the placement-avoidance savings estimate to arrive at a more appropriate estimate of $2.2 million in City cost savings.

Recommendations Esperanza should: 4. Improve its cost-savings report by (a) adjusting its placement-avoidance savings

estimate to account for clients who are sentenced to incarceration by an adult court within three years of exiting the Esperanza program, and (b) subtracting Esperanza’s costs from the placement-avoidance savings estimate.

DOP Response: “Computing cost savings for a program like ours is extremely difficult. As your analysis points out cost savings calculations are very complex with a host of variables. We will, however, work with Esperanza to ensure that the best-cost savings method available is utilized and takes into account net cost savings.”

5. Improve support for its cost-savings reports by providing credible evidence for the

discount rate it uses to adjust estimated savings to account for projected future placements.

DOP Response: “There is no requirement in the contract that a discount rate must be used or applied. We will eliminate it from all future cost savings calculations.”

8 Youths 16 years of age or older are tried as adults in criminal court, unless the case involves an offense committed prior to the age of 16, which would still be handled by Family Court. Youths who are 13 to 15 years of age can be tried as adults in criminal court if they are charged with committing one of 18 serious criminal acts.


16

Esperanza’s Performance-Based Claims for Reimbursement Were Generally Supported Adequately, But Support for Its Non-Performance-Based Claims Was Seriously Deficient

Esperanza’s performance-based claims for reimbursement for January through June 2007 were generally supported adequately. However, support for its non-performance-based claims for April through June 2007 was seriously deficient. Although Esperanza generally maintained adequate records on staff salaries and benefits, its documentation of executive-support and overhead costs was extremely inadequate. Not only did DOP and Esperanza, although requested, not provide a clear accounting of over $80,000 in executive-support costs and over $120,000 in overhead costs, they also did not provide any documentation concerning its overhead costs. As a result, DOP and Esperanza were unable to demonstrate that these funds were used in an appropriate manner. By failing to properly monitor executive-support and overhead expenses, DOP is increasing the risk that City funds will be misused.

As mentioned earlier, the contract has four major cost categories. The following sections

present detailed explanations of each cost category and the supporting documentation provided by Esperanza.

1. Reimbursement of costs for staff salaries and benefits, executive support, and

overhead: Esperanza is reimbursed for field supervisor, field counselor, intake, and community services staff salaries and fringe benefits. In addition, it bills DOP for the cost of executive support, which includes expenses associated with day-to-day support provided by senior and administrative staff of Vera and Esperanza. Furthermore, Esperanza is paid for its overhead expenses for office space, computers, supplies, and insurance.

During a payroll distribution at Esperanza on March 14, 2008, we observed 27 of the 32 Esperanza employees receiving their checks and asked them to display photo identification to us to confirm their identity. We also found and reviewed W-4 federal withholding forms or other official documentation in employee personnel files for the five employees who were not present at the distribution. Based on the results of this test, we were reasonably assured that all 32 individuals were bona fide Esperanza employees. In addition, payroll registers and general ledger entries for April through June 2007 generally provided adequate support for staff salaries and benefits totaling $486,201, the amount included in Esperanza’s claim.

However, there were two questionable special compensation payments noted in the payroll registers for April through June 2007—one in April 2007 for $4,604 and another in June 2007 for $17,308—for which Esperanza has provided no supporting documentation. Although we requested information on these payment on March 5, 2008, DOP did not respond to this request until the exit conference held on May 6, 2008, at which time it explained that these payments represented termination packages for two employees. DOP still has not provided any documentation to support this explanation.


17

In addition, DOP does not require supporting documentation from Esperanza for its staff salary, executive-support, or overhead cost claims, nor does it periodically visit Esperanza to review any of this documentation. Since about 72 percent of DOP’s payments to Esperanza are related to these costs, DOP is remiss in not periodically reviewing Esperanza’s support for these cost claims and denying or seeking recoupment of payments for unsupported claims.

For April through June 2007, Esperanza provided no accounting for executive-support

costs of $84,135 and overhead costs of $125,687 and, although requested, has not provided any documentation concerning its overhead costs. As a result, DOP and Esperanza are unable to demonstrate that these funds were used in an appropriate manner. By failing to properly monitor executive-support and overhead expenses, DOP is increasing the risk that City funds will be misused.

2. Production of monthly Esperanza performance indicators and quarterly cost-savings

reports: Esperanza develops a monthly report on its performance in relation to the goal of reducing reliance on placements in State facilities. It also produces a quarterly report estimating the City’s cost savings associated with Esperanza’s clients avoiding placement in a State facility.

The monthly indicator report uses data available from Esperanza, DOP, and OCFS to present information on the placement of juvenile delinquents. The report looks at four indicators: (1) the flow of youths into the juvenile justice system, (2) the percentage of placement recommendations, (3) the use of Esperanza, and (4) the use of placement in State facilities. In order to document the completion of this task, Esperanza attached the December 2006 through April 2007 indicator reports to its certification statement and charged DOP for five monthly reports. Esperanza also provided quarterly savings reports that identified each client served, indicated whether the client had been placed in a State facility since enrollment, and estimated the City’s share of cost savings associated with those who avoided placement.

3. Achievement of performance measures: This cost category looks at two key

performance measures: the target number of youths enrolled with Esperanza and the clients’ avoidance of placement for nine months after intake. Esperanza is expected to enroll a target number of 54 youths per quarter. It receives $100,000 per quarter for achieving this target number and a $2,000 bonus for any youth it serves in excess of this target number. Conversely, it is penalized $2,000 for each youth it fails to enroll below the target number. For the performance measure relating to clients avoiding placement, Esperanza receives $50,000 per quarter if a target percentage of the youths in the program avoid placement for nine months after enrollment. Esperanza receives an additional $1,000 for each percentage point by which it exceeds this percentage and is penalized $1,000 for each percentage point by which it falls below this target. For the sampled period of January through June 2007, the target percentage was 70 percent.


18

For the enrollment performance measure, Esperanza provided client lists and certification statements claiming that it enrolled 76 clients for the third and fourth quarters of Fiscal Year 2007: 46 for the third quarter and 30 for the fourth quarter. The list was compared to each client’s enrollment date as listed in the PMA system. The dates identified on the enrollment list matched the dates found in the PMA system. Payments of $84,000 for the third quarter and $52,000 for the fourth quarter were consistent with DOP’s explanation of enrollment reimbursement and were generally consistent with the contract. However, as was discussed above, the contract also indicates that a minimum of $80,000 and a maximum of $120,000 would be provided to Esperanza each quarter for its enrollment efforts. For the performance measure relating to clients avoiding placement, Esperanza provided a list of the clients that reached the nine-month mark after enrollment in the program. For the sampled periods, Esperanza’s list indicated that 37 (86%) of the 43 clients who reached the nine-month mark during January through March 2007 and 26 (72%) of the 36 who reached the nine-month mark during April through June 2007 avoided placement. For these 63 clients, docket summaries from UCMS confirmed that they had in fact avoided placement for nine months following enrollment. For January through March 2007, Esperanza, consistent with its placement-avoidance list, claimed for reimbursement that it achieved an 86 percent placement-avoidance rate. DOP’s placement-avoidance payment of $66,000 was consistent with its explanation of placement-avoidance reimbursement and was generally consistent with the contract. However, as was discussed above, the contract also indicates that a minimum of $40,000 and a maximum of $60,000 would be provided to Esperanza each quarter for its placement-avoidance efforts. For April through June 2007, although Esperanza’s list indicated that only 26 of 36 clients who reached the ninth-month mark during this period had avoided placement, Esperanza claimed 27 (75%) of the 36 clients had avoided placement. Esperanza claimed $55,000 for this deliverable. For having 26 clients (72%) of the 36 clients who reached the nine-month mark during this period avoid placement, Esperanza was entitled to receive the basic placement-avoidance payment of $50,000 for meeting the 70 percent target, plus an additional $2,000 for exceeding the target percentage by 2 percent. Instead, Esperanza incorrectly claimed that it had achieved a 75 percent rate of placement avoidance. Esperanza officials stated that the error occurred because they had mistakenly assessed one client’s status in UCMS. DOP claims that it did not ultimately pay the additional $3,000 in this case because of a reimbursement adjustment in the first quarter of Fiscal Year 2008. DOP provided no documentary evidence to support this claim.

4. Structural changes, technical assistance, and reporting: This cost category involves Esperanza’s efforts to monitor structural changes within DOP, provide technical assistance to government agencies, and report on these activities.

Table III, below, details eight tasks for which Esperanza made claims under this cost category and the various supporting documentation Esperanza provided to us for January through June 2007.


19

Table III Documentation Provided by Esperanza to Support Claims Relating to

Esperanza’s Structural Change, Technical Assistance, and Reporting Efforts January through June 2007

Task Supporting Documentation Provided

1. Technical assistance to Probation (includes development of internal capacity for structural reform work)

Violation of probation policy, graduated sanctions manual, reports on client enrollment and program completion rates

2. Continuing adaptation and maintenance of electronic PMA and other databases for utilization in data analysis and management

Email announcing PMA training on 6/20/07, PMA user guide

3. Follow-up and report to stakeholders on program progress

Calendar of meetings held, City Council testimony graphs, analysis of violations of probation, presentation package on NYC juvenile justice system reforms, Mayor’s Management Report statistics

4. Expand and maintain relationships with community-based providers

List of community-based organizations (identified in certification statement) with whom relationships were cultivated during the quarter

5. Technical assistance to produce Citywide juvenile justice indicator reports

NYC juvenile justice monthly indicators for September 2006 - March 2007

6. Tracking, analysis, and reporting on Probation Assessment Tool recommendations; revalidation and recalibration of PAT

PAT Quality Control Assessment; analysis of risk factors for rearrest, Investigation and Report recommendations, and override percentages

7. Convene Board of Trustees Minutes of March 30, 2007 meeting

8. Work on continuing evolution of Exploration of Placement (EOP)/disposition process; technical assistance to expand use/availability of sentencing options; analysis of placement-bound youth, via EOP or newly devised process

Analysis of EOP referrals and clients accepted by Esperanza

Esperanza provided us sufficient documentation to support its claims in this cost category. Although DOP did not require that Esperanza submit all of this documentation to DOP, officials told us that they, through their ongoing working relationship with Esperanza, were well aware of and satisfied with the tasks performed by Esperanza in this cost category. Esperanza’s claims in this cost category did not exceed contract amounts.


20

Recommendations Esperanza should: 6. Account for and maintain adequate documentation on its special compensation,

executive-support, and overhead expenses. DOP Response: “The Department believes that Esperanza properly accounts for and maintains special compensation, executive-support, and overhead expenses in their QuickBooks financial system.” Auditor Comment: During this audit, both DOP and Esperanza failed to demonstrate that these funds were used in an appropriate manner. DOP did not provide any evidence that it required or reviewed any supporting documentation from Esperanza to account for the spending in these areas. DOP should: 7. Recoup all funds relating to special compensation, executive-support, and overhead

costs in Fiscal Year 2007 for which Esperanza is unable to provide adequate supporting documentation.

DOP Response: “The Department finds no evidence of fiscal impropriety and believes that Esperanza’s supporting documentation is satisfactory.” Auditor Comment: Since DOP has provided no evidence to show that it has reviewed Esperanza’s supporting documentation for these costs, DOP’s conclusion that the documentation is satisfactory appears to be based on little more than its trust in Esperanza. As a result, DOP is not meeting its responsibility to vigorously protect City resources from misuse. 8. Periodically review Esperanza’s supporting documentation for its staff salary,

executive-support, and overhead cost claims and deny or seek recoupment of payments for unsupported claims.

DOP Response: “Overhead expenses are calculated based on a federally approved rate of 21 percent. Generally Accepted Accounting Principles do not require that detailed documentation be reviewed to support payment of such a fixed rate expense. We will, however, periodically review supporting documentation for staff salary and executive-support claims and require that supporting documentation be provided.” Auditor Comment: DOP provided no evidence to support its use of a federally approved rate of 21 percent to calculate Esperanza’s overhead expenses. More importantly, DOP’s statement regarding Generally Accepted Accounting Principles (GAAP) does not apply to the situation our recommendation is attempting to correct. As a generalization, GAAP


21

addresses how accounting information is to be reported in financial statements and management’s responsibility for compiling the data to support the information it reports in its financial statements. GAAP does not specifically address management’s responsibility for reviewing the support for the bills it receives and pays during the normal course of business. In the current instance, DOP is responsible for reimbursing Esperanza for invoices Esperanza submits. To ensure that DOP is reimbursing Esperanza only for costs that are adequately supported, we recommended that DOP periodically review Esperanza’s supporting documentation. This recommendation is in accordance with the Comptroller’s Internal Control and Accountability Directive #2 (Procedures for the Audit of Vouchers Submitted Under Cost Reimbursable Contractual Agreements). While we are pleased to note that DOP agrees to periodically review supporting documentation relating to Esperanza’s staff salary and executive-support claims, we are concerned that DOP did not state that it would also periodically review supporting documentation relating to Esperanza’s overhead costs. Accordingly, we urge DOP to reconsider its response to this recommendation.

audit report on the vera institute of justice contract

Documents