eManifest Initiative Evaluation Study Final Report August 3, 2012 Program Evaluation Division Internal Audit and Program Evaluation Directorate

TABLE OF CONTENTS

Executive Summary……………………………………………………………………………………...i 1. Introduction and Context…………………………………………………………………………….1 2. Key Findings – Relevance…………………………………………………………………………...7 3. Key Findings – Performance………………………………………………………………………..9 4. Demonstration of Efficiency and Economy………………………………………………………18 5. Key Conclusions, Recommendations and Management Response ……………………………19

i

[*] An asterisk appears where sensitive information has been removed in accordance with the Access to Information Act and the Privacy Act.

Executive Summary Background The Canada Border Services Agency (CBSA) is responsible for providing integrated border services that support national security and public safety priorities, and facilitate the free flow of legitimate persons and goods, including animals and plants. In fiscal year 2010-2011, the CBSA’s more than 7,200 border services officers (BSOs) processed over 13 million commercial shipments at numerous service locations including licensed warehouses, 117 land border crossings and 27 rail sites.1 eManifest is a key initiative developed to transform and modernize the processing of goods at the land border. It represents the third phase of the Advance Commercial Information (ACI) project that extends ACI requirements already existing in marine and air modes to all highway and rail commercial carriers, importers and freight forwarders. eManifest is intended to ensure that the CBSA obtains advance commercial cargo, conveyance and crew information to facilitate both risk assessment of shipments prior to their arrival at Canada’s borders and their clearance upon arrival. Major components of the eManifest Initiative include: the development of an electronic data interchange (EDI) and communication protocols and processes required to transmit and receive advance commercial and crew information; an enhanced notification system to facilitate electronic communication between the CBSA and the trade community; an Internet portal to facilitate transmission of commercial and crew information; an integrated passage system for use by BSOs; an automated risk assessment system for the highway and rail environments; retrofitting risk assessment processes to the air and marine modes; and business intelligence and data warehousing capacity to facilitate advanced analysis of trade chain trends in order to support continuous improvements to risk assessment processes and rules. As well, the Initiative includes investments in information technology to support eManifest components. The eManifest Initiative is scheduled for completion in 2014. Evaluation Purpose and Scope The evaluation of the eManifest Initiative was identified as a priority in the CBSA Five-Year Program Evaluation Plan for 2011-2012. This evaluation meets a Treasury Board commitment for an evaluation in year three or four of the Initiative. The purpose of the evaluation was to provide an assessment of the relevance and performance of the eManifest Initiative in accordance with the Treasury Board Directive on the Evaluation Function. Research for this evaluation was conducted by the CBSA Program Evaluation Division between January and November 2011. As the eManifest Initiative is not yet complete and not yet mandatory in the highway and rail modes, the evaluation concentrated on

1 Source: Agency Performance Report Q4 2010-2011, July 15, 2011.

ii

significant activities and operational components undertaken or in the early stages of implementation. Specifically:

• Harmonization of data with international standards including U.S. Customs and Border Protection (CBP);

• Compliance monitoring of highway carriers; • Consultations/communications with external stakeholders; • The EDI transmission systems that were made available to highway carriers in October 2010; • The eManifest Portal; and • The Interim Commercial Risk Assessment Hub Pilot (Hub).2

Another accomplishment is the amendments to the Customs Act, which provide the required legislative foundation for eManifest and allow the CBSA to mandate various members of the trade community (carriers, freight forwarders and importers) to submit pre-arrival data electronically for risk assessment purposes. Evaluation Methodology The evaluation incorporated multiple lines of evidence, including a document and literature review, analysis of operational and financial data, telephone surveys of highway carriers and third party service providers,3 and 49 interviews with internal and external stakeholders. Summary of Findings and Recommendations The evaluation found that the eManifest Initiative is aligned with and supports the February 4, 2011 Declaration on a Shared Vision for Perimeter Security and Economic Competitiveness goal of addressing threats at the earliest possible opportunity through the harmonization of trans-border advance data requirements for cargo and conveyance with the United States Customs and Border Protection (US CBP). eManifest will also contribute to meeting the Government of Canada's obligations under the World Customs Organization (WCO) SAFE Framework through the collection of advance electronic information and pre-arrival automated risk assessment. Similarly, eManifest will support the Agency’s mandate of facilitating the flow of legitimate goods. By focusing enforcement resources on shipments of high or unknown risk identified through the automated risk assessment process, there is a decreased likelihood of an examination of legitimate shipments. As a key business transformation initiative for cross-border trade, there is a continued need for the eManifest Initiative as it will automate processes, facilitate trade and enhance security. By promoting a paperless transaction system through the collection and management of electronic data, the eManifest Initiative will transform the way the CBSA works externally with other government departments

2 Hub is an evolution of the Highway Risk Assessment Centre pilot project which was established in May 2009 in the Windsor/St. Clair region to lay the foundation for the development and implementation of a national risk assessment process for the highway mode. 3 Carriers may contract a third party service provider to transmit eManifest data using the EDI. The CBSA makes available on the Internet, a list of service providers that have successfully tested their software and are known to offer ACI Highway software packages and interface connectivity to CBSA software. http://www.cbsa-asfc.gc.ca/eservices/sp-fs/07-eng.html

iii

(OGDs) and its trade clients, and internally. A key benefit to the CBSA will be the receipt of accurate, complete and timely information. The evaluation found that key external stakeholders — commercial carriers, importers, freight forwarders, brokers, warehouse operators and service providers who will be required to comply with eManifest regulations and reporting processes — were highly satisfied with the CBSA’s outreach and communications activities. These stakeholders were engaged through the Border Commercial Consultative Committee (BCCC) and eManifest Stakeholder Partnership Network (eSPN). An eManifest external help desk was implemented to provide a point of contact for external stakeholders with questions concerning eManifest. The Regional Business Information Service conducts outreach activities, and support to EDI and Portal users is provided via the eManifest Technical Support Unit. Overall, industry stakeholders interviewed characterized the consultation process as effective. In addition, they strongly support the eManifest Initiative although there are concerns over delays in its implementation, a concern that is echoed by CBSA management as it impacts the credibility of the Agency. An eManifest Compliance Management Framework (CMF) and a monitoring process are in place. The CMF includes eManifest-specific Administrative Monetary Penalties (AMPs) and features use of a Trade Chain Partner Performance Report Card to provide clients with information concerning instances of non-compliance and to identify areas for improvement. As of November 30th, 2011, the eManifest and Major Projects Directorate (eMPD) Performance Report Cards had been issued to or were being prepared for 30 carriers, and three clients had been contacted directly as a result of high rates of errors or having provided incomplete information. At the time of the evaluation, only a limited number of highway carriers were using EDI to transmit eManifest data. From October 2010 to October 2011, 278 carriers were approved to use EDI to transmit eManifest data. These carriers accounted for just over 5% of the total highway cargo volume and just over 1% of the over 24,000 highway carriers with carrier codes. The eManifest Portal — a communications tool designed primarily for small and medium-sized businesses — became operational in August 2011, and as of November 2011 there were 338 clients using the Portal. A number of industry association representatives noted that some of their members are deferring registration until they are certain that EDI and the Portal are functioning properly. As a result, there is a risk that the CBSA will receive large volumes of applications shortly prior to eManifest becoming mandatory. In light of these findings it is recommended that: Recommendation 1:

Programs Branch monitor EDI and Portal registration and usage levels during the implementation period to: • Facilitate adjustments of communications/outreach activities with the aim of maximizing

early adoption. • Ensure adequate resources and systems capacity are available to process anticipated

application volumes.

iv

***** Of an overall eManifest budget of $415.1 million,4 the Agency had expended $288 million (as of September 2011). Remaining deliverables are critical to the success of eManifest and include developing capacity to process house bills;5 the collection of crew data; automated risk assessment; a passage application for BSOs; and data warehouse and business intelligence tools. The project schedule and costing plan had not been finalized at the time the evaluation research was conducted. Because the project schedule and costing plan had not been finalized at the time the evaluation research was conducted, the evaluation was not able to determine whether the Agency is well-positioned to meet its commitment to fully implement eManifest by 2014. However, a key component of the Internal Audit and Program Evaluation Directorate (IAPED) eManifest assurance strategy includes a review in 2012-2013 of these plans to assess the clarity of project deliverables and accountabilities; the duration and level of effort required to complete project tasks; resource allocation; and processes and tools in place to monitor and report on progress. eMPD interviewees indicated that the Agency lacks a system to track major project costs by project deliverable. As such, it is not possible to accurately calculate costs incurred for specific eManifest project deliverables. The System Under Development Audit - eManifest (2009) recommended that Information, Science and Technology Branch (ISTB)6 “should explore options to acquire and implement an appropriate project management software tool for major IT projects.” In response, a business case was developed to support acquisition of such a tool. [*] CBSA staff has been prepared for the implementation through training (WebEx training for BSOs and superintendents, online training) and engagement by regional coordinators. In addition, support mechanisms have been put in place, and include the eManifest internal help desk, Operational Bulletins eManifest Highway Processing, and a BSO Guide to eManifest Highway Mode. Conference calls with regional commercial managers are used to help facilitate the exchange of information and issues between the regions and National Headquarters (NHQ). eMPD has also developed an Operations Transition Framework (OTF) to identify and address implementation issues and requirements related to training and support, and a Commercial Program Transition Framework (CPTF) to facilitate measures needed to transition commercial programs to eManifest. For example, as part of the CPTF, a network of representatives from impacted program areas was created in the fall of 2011 to identify commercial program policies that will need to be changed as a result of eManifest. To ensure that frontline officer information needs are addressed, the eMPD is developing an eManifest Essentials session that can be delivered in person and reflects the systems BSOs currently use. These sessions will be delivered nationally in fiscal year 2012-2013. Enhancements are also to be incorporated into an improved online learning product for BSOs and superintendents, and the need to train select regional commercial officers and superintendents has been identified by eMPD. Given the transformational nature of eManifest, the evaluation noted that there will be a need to assess eManifest end-state learning requirements and develop an integrated eManifest learning strategy.

4 Figures include GST. Source: Executive Dashboard, September 2011. 5 A Cargo Control Document submitted by a freight forwarder for shipments that have, or will be, deconsolidated from another Cargo Control Document. (Source – eManifest Component Teams Definition List) 6 At the time the recommendation was made (September 2009), the eManifest Initiative was managed by ISTB.

v

In light of these findings it is recommended that: Recommendation 2:

Programs Branch, in consultation with Human Resources Branch, undertakes a comprehensive assessment of eManifest end-state learning requirements and develops an integrated eManifest learning strategy.

*****

In the highway mode, eManifest requires that cargo and conveyance data be transmitted to the CBSA at least one hour prior to arrival at the first point of arrival (FPOA). Concern was expressed by CBSA staff and industry association representatives over the practicality of this requirement in some regions due to close proximity of businesses to ports of entry (POEs). However, compliance monitoring conducted by eMPD on eManifest traffic processed to the end of November 2011 did not find the one-hour time frame to be a significant issue. eManifest provides for a number of transmission options including a portal specifically developed to support small to medium-sized companies. Nevertheless, some CBSA regional management and staff expressed concern that carriers operating in remote areas may find it difficult to transmit cargo and conveyance information electronically. These interviewees also noted that in some instances, carriers may not be able to provide accurate data at the time a shipment is loaded (e.g., the weight or the types of logs being transported). With respect to the latter issue, eManifest includes provisions for the submission of amendments to cargo and conveyance after arrival at the FPOA. In light of these findings it is recommended that: Recommendation 3:

Programs Branch undertakes targeted outreach activities to promote awareness of eManifest reporting options available to commercial program clients operating in remote areas.

*****

1

1. Introduction and Context The Canada Border Services Agency (CBSA) is responsible for providing integrated border services that support national security and public safety priorities and facilitate the free flow of legitimate persons and goods, including animals and plants. In fiscal year 2010-2011, the CBSA’s more than 7,200 border services officers (BSOs) processed over 93 million travellers, 70 million postal and courier shipments, and 13 million commercial shipments at numerous service locations including licensed warehouses, 117 land border crossings, 13 international airports, 27 rail sites, three mail processing centres, major marine ports, marinas and reporting stations.1 In March 2005, Canada, the United States (U.S.) and Mexico2 announced the Security and Prosperity Partnership of North America, a trilateral initiative to enhance the security of North America, its competitiveness, the economic well-being of its citizens and to further position the continent to meet future challenges. eManifest was identified as a high priority at that time given its potential to contribute to enhanced security efforts and the facilitation of legitimate trade. The CBSA was allocated $415.1 million between 2006 and 2012, to design and implement eManifest.3 In 2009, a budget realignment supporting the CBSA’s contribution4 to Canada’s Economic Action Plan led to the eManifest Initiative completion date being delayed until 2014. Overview of the CBSA eManifest Initiative The objective of eManifest is to further modernize and enhance CBSA processing and screening of commercial goods entering Canada. The eManifest Initiative represents the third phase of the Advance Commercial Information (ACI) project and extends existing ACI requirements in marine and air modes of transportation to all highway and rail commercial carriers, importers/brokers, freight forwarders and warehouse operators. eManifest is intended to ensure that the CBSA obtains advance commercial cargo, conveyance and crew information to facilitate risk assessments and effective targeting of high-risk shipments prior to their arrival at Canada’s borders. Major components of the eManifest Initiative include:

• the development of an electronic data interchange (EDI) and communication protocols and processes required to transmit and receive advance commercial and crew information;

• an enhanced notification system to facilitate electronic communication between the CBSA and the trade community;

• an Internet portal to facilitate transmission of commercial and crew information; • an integrated, user-friendly passage system for BSO use; • an automated risk assessment system for the highway and rail environments;

1 Source: Agency Performance Report Q4 2010-2011, July 15, 2011. 2 Source: eManifest Project Charter Addendum (July 2011). Mexico was not in a position to participate in the implementation of the eManifest Initiative. Therefore, all reference to Mexico and specifically their customs agency, Aduanas Mexico, have been removed from planning documents. 3 Figures include GST. Source: Results-based Management and Accountability Framework (RMAF), eManifest Initiative. Final version: December 22, 2009. 4 The temporary rollback of $85 million of eManifest funding is to be repaid over two years starting in 2011-12.

2

• retrofitting new or enhanced risk assessment systems functionalities to the air and marine modes; and

• business intelligence and data warehousing capacity to facilitate advanced analysis of trade chain trends in order to support continuous improvements to risk assessment processes and rules.

eManifest Implementation The eManifest implementation will take 18 months for each client type (highway carrier, rail carrier, freight forwarder and importer/broker), with different start dates. Within the first 12 months, clients incorporate eManifest requirements into their business processes, using eManifest transmission options (EDI5 and eManifest Portal6). After this time frame, the eManifest regulations take effect. However, the CBSA offers a six-month period of informed compliance (i.e., penalties are recorded but no payment is required) and outreach support. Afterwards, the use of either EDI or the eManifest Portal becomes mandatory and clients will be subject to Administrative Monetary Penalties (AMPs) for non-compliance. The eManifest Initiative is being implemented by client type and time frame, as follows:7 Highway carriers:8

• The 18-month implementation timeline for highway carriers to transmit cargo and conveyance data began on November 1, 2011.

Rail carriers: • The 18-month implementation timeline for rail carriers to transmit cargo and conveyance data

will begin in late spring 2012. Freight forwarders:

• The 18-month implementation timeline will begin in 2012 for freight forwarders in all modes of transportation to transmit house bill9/supplementary cargo data.

Importers/brokers:

5 The four communication methods for EDI clients to transmit eManifest data to the CBSA are: • The Customs Internet Gateway (CIG) - To use the CIG option, clients must first become Canada Revenue Agency Public Key Infrastructure (PKI) subscribers for data security and integrity purposes. Additionally, data must be transmitted from a Canadian office as the PKI certificate is only assigned to a device in Canada. • The Value-Added Network (VAN) - A VAN is a public EDI network for companies to input and exchange data. It provides companies dealing with a large number of trading partners with the opportunity to exchange information with one another using a single communication interface. • Direct Connect to the CBSA - The Direct Connect option provides clients with a direct connection to the CBSA's host system without having to go through a third party or the Internet. This communication method requires an initial investment of $25,000 (+/-5,000) to set up and 6,000 (+/- 2,000) in annual maintenance fees. • Transmission via a third party service provider - CBSA-approved third party service providers transmit data on behalf of clients to the CBSA using one or more of the available EDI communication methods. At the time of the evaluation, there were 10 companies that had completed testing their software/service for eManifest. 6 The eManifest Portal is an option developed by the CBSA to allow the trade community to electronically transmit their pre-arrival information through the Internet. 7 Source: CBSA Internet. January 2012. http://www.cbsa-asfc.gc.ca/prog/manif/implementation-eng.html#s2 8 EDI eManifest transmission options were made available to highway carriers for cargo and conveyance information on October 31, 2010 and the eManifest Portal was made available to highway carriers on August 29, 2011. 9 A Cargo Control Document submitted by a freight forwarder for shipments that have, or will be, deconsolidated from another Cargo Control Document. (Source – eManifest Component Teams Definition List)

3

• The 18-month implementation timeline will begin in 2012 for importers in the marine mode to begin transmission of advance trade data.

eManifest Initiative Stakeholders

Programs Branch The eManifest and Major Projects Directorate (eMPD) is responsible for developing and implementing the eManifest Initiative. Three of the Directorate’s four divisions are involved in various aspects of the eManifest Initiative as follows:

• The eManifest Stakeholder Consultations and Implementation Division is responsible for engaging, supporting, training and communicating with internal and external stakeholders. The Division oversees the management and coordination of the eManifest Stakeholder Partnership Network (eSPN) Working Groups. It develops all communication products (including Internet content), and promotes the project across North America through participation at trade events. Also, the Division approves eManifest-related polices.

• The eManifest Business Design and Development Division directs the design, development, validation/testing and implementation of EDI, the eManifest Web portal, a Graphical User Interface (GUI) and Passage Applications (e.g., transponder technology).

• The eManifest Commercial Business Intelligence and Risk Assessment Division is responsible for the design, development and implementation of risk assessment components of eManifest (including those for cargo, secondary cargo, conveyance, importer, driver, crew, passenger and other government department (OGD) data) and the ongoing enhancements to TITAN-Marine/Air and business intelligence tools.

Three other directorates within the Programs Branch support eManifest: • The Pre-Border Programs Directorate is responsible for regulatory updates/amendments and

provides direct support to the eMPD with respect to commercial policy development. The Commercial Unit in the Advance Information and Programs Division of the Pre-Border Programs Directorate acts as a liaison, receiving input from other directorates.

• The Risk Assessment Programs Directorate provides guidance and advice to the eMPD with respect to risk assessment rules and indicators.

• The Border Programs Directorate provides program direction to the field and consults with OGDs and external clients on their needs and requirements with regard to the processing of commercial imports.

Other CBSA Stakeholders The Information, Science and Technology Branch (ISTB) is responsible for developing and maintaining the eManifest systems/solutions, including the EDI, Web portal, GUI and Business Intelligence/Data Warehouse in support of eManifest. The Operations Branch participates in the Director’s Policy Steering Committee. In addition, representatives provide working level input to the eMPD as required. Once eManifest is implemented, the Operations Branch’s role will expand as it is National Headquarter’s (NHQ) link to the regions.

4

The Human Resources Branch’s Training and Learning Directorate assists with the development of training products related to the eManifest Initiative’s implementation. External Stakeholders Highway and rail carriers, importers/brokers, warehouse operators and freight forwarders are the primary private sector stakeholders. They will be required to comply with eManifest regulations and reporting processes once eManifest becomes mandatory. Evaluation Purpose and Scope The evaluation of the eManifest Initiative was identified as a priority in the CBSA Five-Year Program Evaluation Plan for 2011-2012. This evaluation meets a Treasury Board commitment for an evaluation in year three or four of the Initiative. The CBSA's Program Evaluation Division in the Internal Audit and Program Evaluation Directorate (IAPED) carried out the evaluation research between January and November 2011. The purpose of the evaluation was to provide an assessment of the relevance and performance of the eManifest Initiative in accordance with the Treasury Board Directive on the Evaluation Function. The evaluation framework and related evaluation project plan were developed to focus on significant activities and components undertaken to date since the eManifest Initiative is not yet fully developed and not yet mandatory in the highway and rail modes. These included:

• Harmonization of data with international standards; • Compliance monitoring of highway carriers; • Consultations/communications with external stakeholders; • The EDI transmission systems that were made available to highway carriers in October 2010; • The eManifest Portal; and • The Interim Commercial Risk Assessment Hub Pilot (Hub).

Another accomplishment is the amendments to the Customs Act, which provide the required legislative foundation for eManifest and allow the CBSA to mandate various members of the trade community (carriers, freight forwarders and importers) to submit pre-arrival data electronically for risk assessment purposes. Calibration As the largest IT-enabled project undertaken by the CBSA, the eManifest Initiative is subject to significant oversight. To date, this has included two Systems Under Development Audits10 and an independent third party review11. To reduce the reporting burden and limit disruption to the eManifest project team and supporting areas within the Agency, this evaluation draws on and incorporates relevant information and findings of these studies in responding to evaluation issues.

10 Systems Under Development Audit – eManifest Phase 1 (September 2009) focused on the eManifest management framework, Phase 2 (March 2011) examined processes for defining business requirements and managing procurement. Phase 3, planned for completion in fiscal year 2012-2013, will focus on the implementation of new business processes. 11 The objective of the independent review was to assess the completeness and feasibility of the detailed project plan and definition requirements before undertaking major expenditures. Source: Independent Review, CBSA – eManifest, October 2010.

5

An integrated business and IT project schedule and costing for outstanding eManifest components were not available when the evaluation research was conducted. An analysis of this plan will be required to determine whether the Agency is positioned to meet eManifest objectives, and will be conducted separately in 2012-2013 as a key component of the Internal Audit and Program Evaluation Directorate’s assurance strategy for the eManifest Initiative. Evaluation Issues Evaluation issues and questions are outlined in the following table.

Exhibit 1: Evaluation Issues and Questions Evaluation Issue Evaluation Questions

Relevance

Is the eManifest Initiative aligned with the Government of Canada’s priorities and international commitments? Is there a continued need for the eManifest Initiative and is it aligned with Agency priorities?

Performance

To what extent has CBSA staff been prepared for the implementation of the eManifest Initiative? Have the eManifest internal and external stakeholder engagement, communications strategies and support mechanisms been effective? To what extent have eManifest data requirements been harmonized with the US CBP to facilitate trade between the two countries? Has the eManifest compliance regime for highway carriers been developed and implemented? To what extent are clients satisfied with the EDI transmission system and is it functioning as intended? Is the eManifest Portal functioning as planned? Do the eManifest components that have been implemented contribute to more effective and efficient risk assessments?

Efficiency Are resources being used efficiently and is the Initiative adequately resourced to meet its objectives?

Evaluation Methodology The evaluation used multiple sources of information/evidence in its analysis. A brief description of the methodologies used is presented below. Document and Data Review and Analysis The evaluation team reviewed documents and literature provided by the eMPD and other CBSA areas in order to understand the design and context of the Initiative within the CBSA. This review included Treasury Board Submissions and Effective Project Approval documents, the eManifest Project Charter, communications plans, the ACI/eManifest Highway Electronic Commerce Client Requirements Document (ECCRD), compliance monitoring framework as well as implementation and management reports (including the eManifest senior management dashboards), financial and program data, media reports and relevant internal and external correspondence.

6

Key Interviews The evaluation team conducted interviews with key stakeholders to learn about the design and management of the eManifest Initiative and to solicit views and explanations of data and results obtained from other methodologies. Interviews were conducted with management in CBSA NHQ as well as the regions, and with frontline staff (BSOs). In addition, representatives of associations representing the trade community (importers, brokers, freight forwarders, warehouse operators, trucking, railway and logistics industries) participating on the Border Commercial Consultative Committee (BCCC) were also interviewed. In total, 49 individuals were interviewed over the course of the evaluation. (Exhibit 2)

Exhibit 2: Interviews Conducted for the Evaluation

Stakeholder Type Number of Interviewees

CBSA NHQ management and staff 17 CBSA regional management and staff 24 Private sector stakeholders 8 Total 49

Site Visits The evaluation team conducted site visits in the Southern Ontario (Windsor) and Quebec (Montréal and Saint-Bernard-de-Lacolle) regions. The purpose of these site visits was to assess how the eManifest Initiative is being rolled out, how it is managed and delivered in the field and how regional personnel coordinate/communicate activities with key partners and NHQ. Specifically, the evaluation team interviewed staff and managers involved in the delivery of the eManifest Initiative, observed eManifest processes at the border, and attended a third party service provider’s demonstration of the use of EDI to transmit eManifest data. Survey The two types of clients using EDI for the transmission of eManifest data to the CBSA were service providers and highway carriers. Two surveys, one of each client group, were conducted to solicit their satisfaction with the EDI process and the support provided by the CBSA. As of August 2011, the CBSA approved 10 service providers for using EDI to transmit eManifest data to the CBSA on behalf of highway carriers. All 10 of the third party service providers12 (which are used by about 98%of approved highway carriers) were contacted, and nine of these participated. The survey was conducted by phone in August 2011. As such, survey results can be considered representative of the service providers approved to use EDI for eManifest at the time the survey was conducted. As of October 2011, 278 highway carriers were approved for using EDI to transmit eManifest data to the CBSA. The list of highway carriers was only provided to the evaluation team in late November 2011. Consequently, it was only possible to survey a small portion (20) of the highway carriers.

12 Carriers may contract with a third party service provider to transmit eManifest data using the EDI. The CBSA Internet has a list of service providers that have successfully tested their software and are known to offer ACI Highway software packages and interface connectivity to CBSA software. http://www.cbsa-asfc.gc.ca/eservices/sp-fs/07-eng.html

7

Evaluation Research Limitations Due to the low number of highway carriers surveyed, results cannot be generalized to the EDI eManifest highway carrier population. To compensate for this limitation, whenever possible, information collected from interviews with external stakeholders representing the trade industry has been included in the analysis as an additional line of evidence. The tracking of project costs and the level of financial data available were not sufficient to do an analysis of cost-efficiency and cost-effectiveness. However, as noted above, a review will be conducted on the integrated business and IT project schedule and costing for outstanding eManifest components in 2012-2013.

2. Key Findings - Relevance Is the eManifest Initiative aligned with the Government of Canada’s priorities and international commitments? The eManifest Initiative is aligned with and supports the Declaration on a Shared Vision for Perimeter Security and Economic Competitiveness. One of the key components of the Action Plan13 to implement the Shared Vision for Perimeter Security and Economic Competitiveness is for the U.S. and Canada to address threats at the earliest possible opportunity through the harmonization of trans-border advance data requirements for cargo. eManifest data requirements for cargo and conveyance have been harmonized with the US CBP Automated Commercial Environment (ACE) program. eManifest is a key component supporting the Government of Canada’s Single Window Initiative (SWI) — also part of the Action Plan.14 SWI allows companies to electronically submit all import data required by various government departments through one “window” rather than to each department separately. Because the CBSA is the only organization mandated to manage the border, it was designated as the single entry point for the electronic submission of all OGD import data requirements, and the technology and solutions used to implement eManifest components (i.e., Data Acquisition, Web Portal, Authentication Solution, Risk Assessment, Business Intelligence/Data Mining Tools and the Data Warehouse) will be leveraged to facilitate the implementation of SWI.

13 The Action Plan focuses on four key areas: addressing threats at the earliest possible opportunity; facilitating trade, economic growth and jobs; building on successful cross-border law enforcement programs; and enhancing cross-border critical and cyber infrastructure. 14 The main objective of the Single Window Initiative is to avoid duplication by eliminating paper-based reporting and offering 24/7 service for the submission of all OGD data required for risk assessments and admissibility determination recommendations.

8

Once implemented, eManifest will help the Government of Canada meet its obligations under the World Customs Organization (WCO) SAFE Framework.15 Key elements of SAFE related to the eManifest Initiative include the collection and integration of advance electronic information and the use of a common risk management approach to identify potential security threats. Data elements collected under eManifest are aligned with the WCO standards. Is there a continued need for the eManifest Initiative and is it aligned with Agency priorities? As a key business transformation initiative for cross-border trade, there is a continued need for the eManifest Initiative as it will automate processes, facilitate trade and enhance security. By promoting a paperless transaction system through the collection and management of electronic data, the eManifest Initiative will transform the way the CBSA works externally, and with OGDs and its trade clients, and internally. By automating and modernizing existing means, eManifest will make the process at the border easier and faster for trade clients. As the CBSA moves away from the use of paper, the fluidity and ease of the exchange of information will be particularly noticeable for the largest trade clients. eManifest will also enable the CBSA to modernize its operations by integrating and automating border technology and allowing for effective information sharing at the border (e.g., among targeters, BSOs). For instance, eManifest will make it possible to collect advance electronic commercial information from trade chain partners involved in the importation of goods to Canada. This will provide the Agency with more reliable data on all imports, in all four modes of transportation, contributing to the new business model and improving the CBSA’s ability to perform accurate targeting. In the same way, the eManifest Initiative will allow for the use of new technologies such as the Portal, which is the first and only secure, transactional, high-volume system the CBSA currently has. The Data Warehouse will enhance the CBSA’s ability to store and access data. This, in turn, carries great potential for data mining, consolidation and effective exchange of information. The eManifest Initiative will support the Agency’s mandate of facilitating the flow of legitimate goods. With the implementation of eManifest, BSOs will have another tool at their disposal to make clearance and examination decisions. eManifest provides a recommendation as to whether a shipment should be released16 or referred, information that is available before the shipment arrives in Canada. This decreases the likelihood that shipments of legitimate goods will be delayed due to an unnecessary examination. Conversely, it allows the Agency to focus enforcement resources on shipments of high or unknown risk. 15 Canada has been supporting the implementation of the WCO Standards to Secure and Facilitate Global Trade (SAFE) Framework since June 2006 and is one of 66 signatories of the Revised Kyoto Convention, an international agreement that provides a set of comprehensive Customs procedures with respect to risk management and pre-arrival information. 16 Source: eManifest Project Briefing, Director General Projects Steering Committee Meeting May 18, 2011

9

The electronic information and automated processes associated with eManifest enable BSOs to move away from the manual keying of information and reviewing paper documents, thus reducing processing time. In fact, site visits showed and BSOs interviewed for the evaluation indicated that eManifest was easier and faster to process at Primary Inspection Line (PIL) than paper manifests. From a facilitation of trade perspective, the shorter processing time at PIL is an important consideration. Secondary examinations, on the other hand, were said to take more time using eManifest compared to previous systems, as paper manifests contain more information for secondary examination purposes (e.g., invoices may contain more detail), and it takes more time scrolling down the screens than flipping through a number of Pre-arrival Review System (PARS) documents. eManifest has the capacity to notify clients of deficiencies in the accuracy and completeness of data submitted, thus providing clients with an opportunity to correct any data quality issues prior to goods arriving at the port of entry (POE) and reducing the likelihood of delays or penalties. Industry stakeholders interviewed for the evaluation indicated that the eManifest Initiative is perceived as a positive step towards reducing commercial border processing time and paperwork burden, two key longstanding irritants for the trade community.17 Once implemented, the eManifest Initiative will enhance the receipt of accurate, complete and timely information which is needed to identify potential threats. All CBSA management interviewed for the evaluation expressed a need to ensure that sufficient advance commercial information is transmitted in a timely and complete manner to support robust risk assessment and the accurate targeting of potential threats. With the implementation of eManifest, the CBSA will have access to electronic pre-arrival cargo, conveyance, crew, importer and admissibility data for all modes of transportation.

3. Key Findings – Performance This section is structured according to the elements within the scope of the evaluation and covers their implementation and performance to date. To what extent has CBSA staff been prepared for the implementation of the eManifest Initiative? Prior to April 2011, each region had a regional eManifest coordinator. Regional managers interviewed indicated that coordinators had been effective in identifying regional issues and supporting CBSA staff and clients.

17 A study conducted by Donald Alper and Bryant Hammond (Alper and Hammond conducted 46 interviews of private and public stakeholders in the Pacific Northwest border region from February to August 2009) revealed that stakeholders were mainly concerned about delays at the border. Also, Satinder and Pohlman argue that a third of the Small and Medium Enterprises (SMEs) surveyed in their 2010 study reported government paperwork as a factor in their declining imports. In the same way, in a speech to the CAN/AM Border Trade Alliance in Ottawa, the CEO of the Canadian Trucking Alliance stated that “anything that impairs the predictability and reliability of the North American supply chain also impairs the competitiveness of the North American economy.”

10

When NHQ removed the funding for these positions in April 2011, some regions did not find that the remaining support mechanisms always meet regional needs. As a result, in one region, the eManifest regional coordinator position was maintained using regional resources until September 2011. During this period, the coordinator developed and delivered BSO training and continued to conduct client outreach activities with members of the trade community. While regions no longer have resources dedicated to eManifest coordinators, coordination responsibilities have been transferred to existing positions to support implementation. The eMPD identified a number of eManifest learning needs for BSOs and superintendents, and training has been provided. Learning needs included how to review eManifest information using the Accelerated Commercial Release Operations Support System (ACROSS);18 verifying that data meet regulatory requirements; and understanding planned changes in ACROSS that will result from eManifest. To address these needs, the eMPD worked with the CBSA’s Training and Learning Directorate to develop an eManifest WebEx19 session and an online learning (e-learning) product. WebEx training began in October 2010 and as of March 2011, approximately 1,500, or about half of the 3,100 BSOs and superintendents in the highway mode, had participated. 20 The e-learning product was made available online, and since its release in May 2011, 333 participants have completed the e-learning course. As well, learning products for Border Information Service (BIS) officers21 were developed and made available on the CBSA’s Learning Portal. As an overall conclusion, WebEx participants did not find this format to be effective as it did not include a practical hands-on component, nor for adequate opportunity to interact with trainers and other participants.22 Regional staff interviewed for the evaluation noted that, given the amount of time since the initial training was provided, refresher courses should be made available prior to eManifest requirements becoming mandatory for highway carriers on November 1, 2012. To ensure that frontline officer information needs are addressed, the eMPD is developing an eManifest Essentials session that can be delivered in person and reflects the systems BSOs currently use. The eManifest Essentials session was piloted in Windsor in January 2012, and will be delivered nationally in fiscal year 2012-2013.23 Enhancements will also be incorporated into an improved online learning product for BSOs and superintendents. eMPD also plans to include the delivery of training to select BSOs and superintendents working in the commercial stream in each region, who would then become regional subject matter experts. While these activities have been tailored to address immediate needs, given the transformational nature of eManifest, it will be necessary to assess eManifest end-state learning requirements across the Agency and to develop an integrated eManifest learning strategy.

18 For eManifest, two new service options (i.e., a code that identifies the type of documentation to be processed) were added to ACROSS: 984 Highway Cargo, and 976 Highway Conveyance. 19 WebEx software facilitates online meetings, conferencing and video-conferencing. Source: http://www.WebEx.com/. 20 Source: G11 statistics, March 2011. 21 Since these are considered overview sessions, no data on how many had completed the learning products were available. 22 eManifest Highway WebEx Sessions, Participant Feedback, December 2010. 23 Source: eMPD.

11

To better support eManifest implementation in the regions, the eMPD has begun developing an Operations Transition Framework (OTF).24 The OTF will address eManifest implementation issues and requirements related to training and support, operational implementation,25 workload management and communications with BSOs and regional management.26 The framework requires that a working group be established under the leadership of the Regional Commercial Program area from the Southern Ontario Region. Have the eManifest internal and external stakeholder engagement, communications strategies and support mechanisms been effective? eMPD’s engagement of internal stakeholders at NHQ has ensured that program policy issues are addressed prior to eManifest implementation. As the area responsible for providing direct support to the eMPD with respect to commercial policy development, the Pre-Border Programs Directorate identified approximately 45 program and operational issues requiring decision prior to eManifest implementation. To ensure that these issues would be addressed, the eMPD developed a Commercial Program Transition Framework (CPTF) to support the identification of activities27 needed to transition the CBSA’s commercial programs to eManifest. Under the direction of the Directors Policy Steering Committee, a network of representatives from impacted program areas was created in the fall of 2011 to identify commercial program policies and Standard Operating Procedures that will need to be changed as a result of eManifest. The evaluation also noted other examples of eMPD’s engagement of key internal stakeholders. For example, eMPD recently met with the Targeting Policies and Programs Management Division to review systems development business cases (drafted by eMPD) and ensure that eManifest systems and processes are consistent with the Agency’s new targeting business model. Support to internal stakeholders is available through the eManifest internal help desk,28 the online eManifest Highway Mode learning product, Operational Bulletins eManifest Highway Processing,29 and a BSO Guide to eManifest Highway Mode which was first distributed to all regions in November 2010. As well, conference calls with regional managers working in the commercial stream are used to help facilitate the exchange of information between the regions and NHQ.

24 The OTF concept was approved by the Operations Branch Executive Committee in January 2012. 25 An example is ensuring national consistency in the application of eManifest policies. 26 Source: eManifest Transitions Framework, Presentation to VP Operations. February 2012. 27 Activities are to be prioritized based on the eManifest implementation schedule with the initial focus being highway cargo and conveyance. For example, system functionality for highway cargo and conveyance reporting is currently available for use on a voluntary basis. As a result, commercial policies impacted by this currently available functionality should be given priority over those functionalities that will not come in until later builds (i.e. 2014). Other examples of activities include the elimination of service options that are not needed in the ACROSS. Source: eManifest Transitions Framework, Presentation to VP Operations. February 2012. 28 The Internal Help Desk responds to CBSA employees’ enquiries received by e-mail or voice mail. 29 Operational Bulletins: PRG-2010-01 (eManifest Highway Processing); and PRG-2011-12 (Measure to Increase the Use of Electronic Data Interchange for Release Purposes).

12

Industry stakeholders interviewed characterized the consultation process as an effective means to be engaged and indicated that the Agency listens to its trade community partners. This finding supports the conclusion of the eManifest Independent Review that “stakeholder engagement appears to have been very successful.”30 The major fora for engaging the trade community are the eManifest Stakeholder Partnership Network (eSPN) and the Border Commercial Consultative Committee (BCCC). Within the eSPN, a number of working groups were established to address communications, systems design, system outage issues, etc. Since 2007, the CBSA has hosted 73 eSPN meetings to communicate to and consult with highway and rail carriers, freight forwarders, importers and brokers. In 2011, a BCCC Major Project Sub-Committee was established as a forum for strategic engagement of industry stakeholders with respect to all CBSA major projects. Chaired by the Director General of eMPD, membership includes representatives from both the eSPN Communications Working Group and eSPN Design Working Group. Since 2006, the eManifest project team has participated in and made eManifest presentations at 12 meetings of the full BCCC and two meetings of the BCCC Major Projects Sub-Committee. Significant effort has been made to ensure that stakeholders are aware of eManifest requirements. A key objective of the eManifest communications strategy is to “provide industry the opportunity to become familiar with eManifest requirements before they become mandatory.” 31 Since 2008, eManifest fact sheets have been posted on the eManifest section of the CBSA Internet and 15 e-mail communiqués have been sent to eSPN members to advise them on progress and policy decisions. Over the same period, the CBSA attended 19 trade events across Canada and the U.S. and approximately 4,260 clients visited the eManifest booth to speak to a CBSA representative about the Initiative. To date, five eManifest-related news releases have been issued and in April 2011, over 17,000 highway carriers that are registered in the CBSA carrier code database were provided with eManifest information via e-mail and fax. Since this time, the number of registered carriers has increased to over 24,000 and the eMPD indicated that, as part of its Highway Carrier Outreach Plan,32 another transmission of information is planned to highway carriers by e-mail and fax. Since August 2011, 17 “Attention Highway Carriers: eManifest may apply to you” Webinars33 have been delivered to approximately 150 stakeholders comprising of highway carriers and eight eManifest Portal Demonstration Webinars have been delivered to approximately 100 stakeholders interested in using the eManifest Portal. The majority of service providers and highway carriers surveyed for the evaluation were satisfied with the CBSA’s communication efforts. Seven of the nine service providers surveyed indicated that they 30 Source: Independent review CBSA- eManifest Project, October 12, 2010. 31 Source: Communications Strategy version 1.9. February 4th, 2010. 32 The Highway Carrier Outreach Plan includes dissemination of information via e-mail/fax, free Webinars offered on a weekly basis covering general eManifest information as well as an eManifest Portal demonstration; information mailed to trade associations in both Canada and the U.S.; Portal workshops offered in coordination with the Ontario Trucking Association and other trade partners. 33 A Webinar is a live presentation or demonstration offered by the CBSA using a Web conferencing service. The event is hosted by a CBSA representative and followed by a question and answer period.

13

were satisfied with the job being done by the CBSA in communicating requirements of the eManifest Initiative, as were 18 of the 20 carriers contacted. This finding is supported by information gathered through interviews with representatives of key industry associations, the majority of which expressed satisfaction with the level of information and support being provided to their members by CBSA staff. Results of the survey also serve to highlight the important role that service providers play in increasing eManifest awareness amongst highway carriers: nine34 of the 20 highway carriers surveyed stated that they had learned about eManifest through their third-party service providers. Industry stakeholders strongly support the eManifest Initiative, but expressed frustration over delays in the project. BCCC and eSPN members interviewed for the evaluation indicated they fully understood the need to ensure data integrity and support automated pre-arrival risk assessment as a means to improve efficiency at the border. However, they also expressed concern over implementation delays. Trade association representatives interviewed shared that implementation timelines were often pushed back and that details of key milestones were not always available. CBSA interviewees expressed concern that this uncertainty has had a negative impact on the CBSA’s credibility and on stakeholders’ efforts to prepare for eManifest implementation. To what extent have eManifest data requirements been harmonized with the US CBP to facilitate trade between the two countries? eManifest cargo and conveyance data elements have been harmonized with the US CBP Automated Commercial Environment (ACE), but not yet those for crew. One of the objectives of the alignment of eManifest and ACE data requirements is to reduce compliance costs by allowing the trade community to leverage investments already made to comply with U.S. requirements. A majority of the highway carriers and service providers surveyed for this evaluation agreed that not only were the CBSA’s eManifest requirements and those of the US CBP’s ACE program similar, but that prior investment in ACE had made it easier to meet eManifest EDI data requirements and had resulted in savings. Data elements used by the US CBP that have no CBSA equivalent relate to crew data. The CBSA has yet to specify the crew data elements that will be required under eManifest. The eManifest Strategic Deployment View document identifies several challenges for the CBSA pertaining to the collection and use of crew/passenger data, including regulatory and policy constraints. Implementation of crew data and risk assessment is dependent upon changes to CBSA regulations for mandatory advance electronic reporting of crew information and new provisions in the Advance Passenger Information / Personal Name Record (API/PNR) Agreement with the European Union (EU). Negotiations with the EU began

34 One carrier learned about eManifest through CBSA outreach (e.g., stakeholder conferences and/or trade shows), two through the CBSA newsletter and three via the CBSA Web site, two through industry contacts. The remaining respondents did not indicate how they had learned about eManifest.

14

in 2011 and the final agreement is anticipated in 2014.35As a result, timelines related to eManifest crew data acquisition and risk assessment implementation have not been determined. Has the eManifest compliance regime for highway carriers been developed and implemented? A Compliance Management Framework (CMF) for the eManifest Initiative and the eManifest Trade Chain Partner (TCP) Performance Report have been developed and implemented. The CMF includes the use of AMPs as a compliance tool and two new AMPs36 have been proposed and drafted to address non-compliance with eManifest37 requirements. In addition, it has been proposed that an existing AMP be made applicable to eManifest data.38 To assist clients in adapting to eManifest, these AMPs will be zero-rated39 during the six-month period after eManifest has become mandatory. The eManifest TCP Performance Report Card is used as a tool to notify service providers and highway carriers of non-compliance and to point out areas of improvement. Clients receive a Performance Report Card after they have transmitted 30 cargo and conveyance reports. In cases where all 30 transmissions contain errors, the client will also be contacted directly by eMPD to resolve the situation. The report card is currently being sent to clients via encrypted e-mail but will eventually be made available through EDI and the eManifest Web Portal.40 As of November 2011, 138 carriers had been selected for review.41 Fifteen TCP Performance Report Cards had been issued and 15 were being prepared. In addition, the CBSA had contacted three clients directly as a result of high rates of errors or incomplete information in their submissions, representing fewer than 2% of the 171 clients that had transmitted eManifest data at that time. A preliminary compliance monitoring review showed that overall data quality on conveyance reporting was adequate, with no significant problems identified. With respect to cargo information, the compliance rate was about 70% and two areas for improvement were identified. These related to incomplete addresses and insufficient commodity descriptions having been provided by clients.

35 A Crew Working Group has been formed to facilitate identification and collaboration on outstanding policy requirements and business systems requirements related to the collection of crew information. 36 C363 – Failure to provide the prescribed conveyance information within the prescribed time frame and manner; and C364 – Failure to provide the prescribed cargo information within the prescribed time frame and manner. 37 These AMPS will also apply to Bay Plan and Harmonized Risk Scoring Advance Trade Data program requirements. 38 C005 – Person provided information to an officer that is not true, accurate and complete. 39 The monetary penalty applied is $0. 40 Source: Reporting Requirements for eManifest EDI and Compliance Monitoring – Highway and Rail Transmissions version 1.4. (July 23, 2010). 41 The threshold of 30 cases of non-compliance to trigger contact with clients was lowered during the initial period of EDI use.

15

To what extent are clients satisfied with the EDI transmission system and is it functioning as intended? While the initiative is currently functioning as planned, at the time of the evaluation, only a limited number of highway carriers were using EDI to transmit eManifest data. A number of the industry association representatives interviewed stated that some of their members were waiting for the “bugs” to be worked out before adopting eManifest. These interviewees also expressed concern with respect to the CBSA’s capacity to process increased numbers of eManifest applications as the voluntary compliance period draws to a close. As such, they saw it as important for the CBSA to maintain a high level of outreach and communications activities throughout the implementation period to encourage early adoption. While the Technical Support Unit was unable to provide an estimate as to the level of resources required to process each eManifest application, service providers and highway carriers surveyed indicated that the process had taken about a month to complete. The application and testing process also requires that transmissions be error free 95% of the time during five consecutive days of volume testing.42 Given this, it is unclear as to whether the Agency would have sufficient capacity to process a large volume of applications in a timely manner should a significant number of clients choose to delay their applications. Overall, highway carriers and service providers expressed satisfaction with EDI eManifest implementation, including both the application and testing processes. About three quarters of highway carriers and all but one of the service providers surveyed expressed satisfaction with the primary communication method43 they were using. A majority of the service providers were using Direct Connect as their primary method to transmit eManifest data. A majority of the service providers surveyed indicated that EDI eManifest has made it easier for them to meet CBSA data requirements, and that eManifest has contributed to faster processing of shipments at the border. eManifest data requirements are clearly outlined in the ACI/eManifest Highway Electronic Commerce Client Requirements Document (ECCRD) which is provided to all clients. Eight of nine service providers surveyed thought the ECCRD was useful, clear and complete. The effectiveness of the ECCRD is likely a key reason why six of the nine service providers either somewhat or strongly agreed that as a result of eManifest, it had become easier to meet CBSA data reporting requirements. Five of the nine service providers surveyed either somewhat or strongly agreed that their clients’ shipments were entering Canada faster with the eManifest process. This perception was less evident amongst highway carriers contacted, with seven of the 20 either somewhat or strongly agreeing. It

42 Source: ECCRD (chapter 11, the application and testing process), page 33. 43 There are four communication methods for Electronic Data Interchange (EDI) clients to transmit eManifest data to the CBSA including the Customs Internet Gateway (CIG), the Value-Added Network (VAN), Direct Connect to the CBSA and transmission via a third party service provider.

16

should also be noted that six highway carriers either somewhat or strongly disagreed and that seven indicated that they either did not know or did not have an opinion. Site visit locations received an average of three to five eManifest shipments a day. As such, it was too early to determine the impact of eManifest on border clearance from a CBSA perspective. eManifest requires that cargo and conveyance data be transmitted to the CBSA at least one hour prior to arrival at the first point of arrival (FPOA). Regional staff and trade associations noted that in some areas, a considerable proportion of commercial highway traffic originates within less than one hour driving distance of the POE. As such, there is a risk that carriers may be subject to AMPs for failing to provide cargo and conveyance information within the prescribed time frame. CBSA staff interviewed also expressed some concern regarding the time it takes for the CBSA to acknowledge validity of eManifest data transmitted to trade chain partners. The highway carriers and service providers surveyed noted that the average time it takes to receive the CBSA’s acknowledgement of data validity was in the range of 8-15 minutes. CBSA interviewees noted that if an eManifest submission is not accepted, the client must correct any errors, resubmit the information and wait for the acknowledgement that the information has been accepted before the one-hour requirement starts. Delays and the need to track the status of numerous submissions can result in additional costs to the client. In remote areas, this requirement can be challenging for other reasons. For example, the risk that log carriers might have difficulty complying with the eManifest requirements was identified in an issue paper prepared by eMPD in July 2010. Specifically, it was noted that “at the time of the pickup of the wood, the carrier doesn’t know the exact weight of the cargo because the shipment doesn’t get weighed prior to arrival in Canada.”44 Although eManifest includes provisions for the submission of amendments to cargo and conveyance after arrival at the FPOA, CBSA regional interviewees stated that these carriers continue to express concern with respect to their ability to provide these details, and with the availability of Internet access in remote areas. It is important to note that the compliance monitoring conducted by eMPD on eManifest traffic processed to the end of November 2011 did not find the one-hour time frame to be a significant issue for the early adopters.45 Contingency plans for systems outages have been developed and they state that drivers provide hard copy documentation to the FPOA. The ECCRD includes a section on contingency plans46 in the event of a system outage and refers to the System Outage Contingency Plan – Highway Mode document. However, the ECCRD does not indicate where the client may find this document and there are no clear links from the eManifest section of the

44 Issue Paper: eManifest Implications on the Wood Logging Industry, eMPD, July 2010. 45 Source: eManifest Monitoring Update (eMPD), November 30, 2011. 46 In the event of an outage in either the CBSA’s system, the client’s or the service provider’s systems, each party must make all efforts to continue normal communications, and to restore their systems to normal operating condition as soon as is reasonably possible.

17

CBSA Web site to the contingency plan. Instructions for BSOs on dealing with system outages are provided in the BSO Guide to eManifest – Highway Mode.47 Due to the requirement to perform periodic systems maintenance, some system outages are scheduled. Among the carriers and service providers surveyed for this evaluation who had experienced such system outages, all expressed satisfaction with the scheduling of these outages and the amount of advance notice provided. Is the eManifest Portal functioning as planned? An eManifest Portal to help small to medium-sized businesses comply with eManifest requirements was implemented in August 2011. As of November 2011, there were 338 clients using the Portal. The Agency invested considerable time and effort to ensure that Portal functionality and support would address user needs. In spring of 2009, consultations with industry using an early version of the Portal was conducted to obtain client input and feedback on proposed Portal features and support mechanisms48. As a result of these consultations, enhancements were made to the Portal. In addition to transmission of pre-arrival data, Portal features also allow clients to store, query, and verify the status of their shipments. Client support concerns were addressed with the addition of a Frequently Asked Questions section and a Quick Reference Guide to the Portal, and the provision of a toll-free technical assistance line available on a 24/7 basis. Do the eManifest components that have been implemented contribute to more effective and efficient risk assessments? The Interim Commercial Risk Assessment Hub Pilot (Hub) has provided information to support the development of eManifest pre-arrival risk assessment processes for the highway mode. Hub is an evolution of the Highway Risk Assessment Centre pilot project which was established in May 2009 in the Southern Ontario (Windsor) region to lay the foundation for the development and implementation of a national risk assessment process for the highway mode in November 2011. As of October 2011, there were nine targeters from the commercial area performing risk assessments at the Hub in Windsor. The eManifest Risk Assessment and the Commercial Business Intelligence and Data Services (CBIDS) unit within eMPD developed a software application (referred to as the Macro) that provides information to these targeters and allows them to record their reasons for referral. Using the Macro, over 7,308 shipments were risk-assessed between September of 2010 and March 2011 which resulted in 721 referrals.49 The CBIDS unit’s analysis of the referral data was used to identify possible risk

47 The guide states that “in the event of a system outage, carriers must revert to the paper process in place before eManifest implementation.” 48 Participants’ feedback obtained through the completion of a questionnaire as well as a group discussion focused on training, client support, terminology, navigation and design issues. 49 The evaluation did not receive information pertaining to the results of all 721 referrals. However, data for June 2011 indicates that of 78 referrals, 61 were examined and released, nine referrals were waiting to be processed, four cargo control documents were rejected, two referrals were cancelled, one was not examined and one AMP was issued.

18

indicators for the highway mode. As a result, 17 potential indicators were tested and nine had been identified as valid as of August 2011.50

4. Demonstration of Efficiency and Economy Are resources being used efficiently and is the Initiative adequately resourced to meet its objectives? Of an overall eManifest budget of $415.1 million,51 the Agency had expended $288 million52 as of September 2011. Exhibit 3 provides an overview of eManifest Initiative expenditures by fiscal year.

Exhibit 3: eManifest Actual Expenditures by Fiscal Year (thousands)

2006-2007 2007-2008 2008-2009 2009-2010 2010-2011 2011-2012 $10,870 $36,353 $58,245 $61,188 $65,282 n.a.

Source: Comptrollership Branch.

Remaining deliverables are critical to the success of eManifest and include developing capacity to process house bills; the collection of crew data; automated risk assessment; an enhanced passage application for BSOs; and data warehouse and business intelligence tools. The 2010 independent review noted that major increases in spending were planned for the final two years of the eManifest Initiative and cautioned that remaining budget might not be adequate for completion of the business intelligence and risk assessment components. To help address this risk, an eManifest Strategic Deployment View Task Force was created to identify the remaining major builds required to complete the Initiative, and to establish an efficient sequencing for these components.53 This work, which was completed in May 2011, was intended to serve as a basis for the development of an integrated business and project schedule, including costing. In the absence of a comprehensive project schedule and costing plan, the evaluation was not able to assess whether the Agency is well-positioned to fully implement eManifest by 2014. As previously noted, the project schedule and costing plan had not been finalized at the time the evaluation research was conducted and could not be assessed as part of the evaluation. However, a key

50 Source: eMPD. 51 Approved funding of $415.1 million included $19.1 million for GST. 52 Figures include GST. Source: Executive Dashboard, September 2011. 53 Guiding principles for development of the deployment sequence included: deployments of eManifest applications should not eliminate existing functionality of legacy applications; minimize the amount of interim development to existing and new applications; maintain strong dependencies between groups of requirements identified in the sequencing; and ensure that initial build forms a strong basis for subsequent builds aimed at increasing functionality.

19

component of the IAPED eManifest assurance strategy includes a review of these plans to determine whether the CBSA has a clear basis against which the progress of the eManifest Initiative can be assessed. Key elements of the review will include an assessment of the clarity of project deliverables and accountabilities; the duration and level of effort required to complete project tasks; resource allocation; and processes and tools in place to monitor and report on progress. The CBSA lacks the necessary tools to accurately track project costs for the various components of major initiatives. Issues with respect to tracking costs persist and the eMPD is not able to accurately identify costs incurred for specific project deliverables such as EDI and the eManifest Portal. The Independent Review of eManifest noted that “confirmation of costs is needed in light of changed circumstances”, that “it is unclear as to where to use Preliminary Project Approval [PPA], Effective Project Approval [EPA] or adjusted EPA funding” and that there was “inadequate integration of Project Management with financial management.” The System Under Development Audit - eManifest (2009) recommended that ISTB54 “should explore options to acquire and implement an appropriate project management software tool for major IT projects.” In response, a business case was developed to support acquisition of such a tool. [*]

5. Key Conclusions, Recommendations and Management Response

The evaluation found that the eManifest Initiative is aligned with and supports the February 4, 2011 Declaration on a Shared Vision for Perimeter Security and Economic Competitiveness goal of addressing threats at the earliest possible opportunity through the harmonization of trans-border advance data requirements for cargo and conveyance with the United States Customs and Border Protection (US CBP). eManifest will also contribute to meeting the Government of Canada's obligations under the WCO SAFE Framework through the collection of advance electronic information and pre-arrival automated risk assessment. Similarly, eManifest will support the Agency’s mandate of facilitating the flow of legitimate goods. By focusing enforcement resources on shipments of high or unknown risk identified through the automated risk assessment process, there is a decreased likelihood of an examination of legitimate shipments. As a key business transformation initiative for cross-border trade, there is a continued need for the eManifest Initiative as it will automate processes, facilitate trade and enhance security. By promoting a paperless transaction system through the collection and management of electronic data, the eManifest Initiative will transform the way the CBSA works externally with other government departments (OGDs) and its trade clients, and internally. A key benefit to the CBSA will be the receipt of accurate, complete and timely information.

54 At the time the recommendation was made (September 2009), the eManifest Initiative was managed by ISTB.

20

The evaluation found that key external stakeholders — commercial carriers, importers, freight forwarders, brokers, warehouse operators and service providers who will be required to comply with eManifest regulations and reporting processes — were highly satisfied with the CBSA’s outreach and communications activities. These stakeholders were engaged through the Border Commercial Consultative Committee (BCCC) and eManifest Stakeholder Partnership Network (eSPN). An eManifest external help desk was implemented to provide a point of contact for external stakeholders with questions concerning eManifest. The Regional Business Information Service conducts outreach activities, and support to EDI and Portal users is provided via the eManifest Technical Support Unit. Overall, industry stakeholders interviewed characterized the consultation process as effective. In addition, they strongly support the eManifest Initiative although there are concerns over delays in its implementation, a concern that is echoed by CBSA management as it impacts the credibility of the Agency. An eManifest Compliance Management Framework (CMF) and a monitoring process are in place. The CMF includes eManifest-specific Administrative Monetary Penalties (AMPs) and features use of a Trade Chain Partner Performance Report Card to provide clients with information concerning instances of non-compliance and to identify areas for improvement. As of November 30, 2011, eMPD Performance Report Cards had been issued to or were being prepared for 30 carriers, and three clients had been contacted directly as a result of high rates of errors or having provided incomplete information. At the time of the evaluation, only a limited number of highway carriers were using EDI to transmit eManifest data. From October 2010 to October 2011, 278 carriers were approved to use EDI to transmit eManifest data. These carriers accounted for just over 5% of the total highway cargo volume and just over 1% of the over 24,000 highway carriers with carrier codes. The eManifest Portal — a communications tool designed primarily for small and medium-sized businesses — became operational in August 2011 and as of November 2011 there were 338 clients using the Portal. A number of industry association representatives noted that some of their members are deferring registration until they are certain that EDI and the Portal are functioning properly. As a result, there is a risk that the CBSA will receive large volumes of applications shortly prior to eManifest becoming mandatory. In light of these findings it is recommended that: Recommendation 1:

Programs Branch monitor EDI and Portal registration and usage levels during the implementation period to: • Facilitate adjustments of communications/outreach activities with the aim of maximizing

early adoption. • Ensure adequate resources and systems capacity are available to process anticipated

application volumes.

21

Management Response: Programs Branch concurs with the recommendation. Building on the efforts dedicated to communication thus far, the eManifest and Major Projects Directorate will continue to encourage all clients to become early adopters prior to the legislation rendering the requirement mandatory. Specific activities will include ongoing delivery of Webinars and targeted presentations while providing eManifest information via mail and e-mail, directly to all carriers. In an effort to determine the expected registration workload leading up to the mandatory date, the Project team has undertaken a review of existing carriers and determined that of the 24,000 carriers in all modes of transportation, 22,713 are highway carriers. Of these, 15,504 crossed the border with an importer release on file during the 2011-12 fiscal year. An analysis of these 15,504 carriers reveals that approximately 9,500 cross into Canada less than once a week. Based on current adoption patterns, it is anticipated that 4,500 of these carriers will register to use EDI, and that the majority of these will do so through a third-party service provider. The use of service providers significantly reduces CBSA timelines and resources required to process applications, since no systems testing is required. The remaining 10,500 will likely opt to use the eManifest Portal, as these are primarily lower-volume carriers. The eManifest Project team, working with the Technical Support Unit which processes eManifest applications, will be able to manage these volumes. Based on ongoing review and monitoring of applications, the impact of additional demand will be mitigated through the use of overtime, extended hours and sharing of resources between the Project team and the Technical Support Unit. Management Action Plan: Completion Date • Tailor and deliver communications messages and outreach

activities to targeted carriers to ensure that communication materials reinforce the benefits of early adoption, and increase awareness and availability of Webinars.

• Contact all non-registered carriers by e-mail and phone to help

them identify their registration method and also to work through their implementation concerns.

• Monitor both Portal and EDI registration and usage levels on a

monthly basis. Should adoption patterns vary significantly from what was anticipated, a contingency strategy to address potential resource and systems pressures will be developed.

August 2012 August 2012

September 2012

22

*****

Of an overall eManifest budget of $415.1 million,55 the Agency had expended $288 million (as of September 2011). Remaining deliverables are critical to the success of eManifest and include developing capacity to process house bills;56 the collection of crew data; automated risk assessment; a passage application for BSOs; and data warehouse and business intelligence tools. The project schedule and costing plan had not been finalized at the time the evaluation research was conducted. Because the project schedule and costing plan had not been finalized at the time the evaluation research was conducted, the evaluation was not able to determine whether the Agency is well-positioned to meet its commitment to fully implement eManifest by 2014. However, a key component of the Internal Audit and Program Evaluation Division (IAPED) eManifest assurance strategy includes a review in 2012-2013 of these plans to assess the clarity of project deliverables and accountabilities; the duration and level of effort required to complete project tasks; resource allocation; and processes and tools in place to monitor and report on progress. eMPD interviewees indicated that the Agency lacks a system to track major project costs by project deliverable. As such, it is not possible to accurately calculate costs incurred for specific eManifest project deliverables. The System Under Development Audit – eManifest (2009) recommended that ISTB57 “should explore options to acquire and implement an appropriate project management software tool for major IT projects.” In response, a business case was developed to support acquisition of such a tool. [*] CBSA staff has been prepared for the implementation through training (WebEx training for BSOs and superintendents, online training) and engagement by regional coordinators. In addition, support mechanisms have been put in place, and include the eManifest internal help desk, Operational Bulletins eManifest Highway Processing, and a BSO Guide to eManifest Highway Mode. Conference calls with regional managers working in the commercial stream are used to help facilitate the exchange of information and issues between the regions and National Headquarters (NHQ). eMPD has also developed an Operations Transition Framework to identify and address implementation issues, and a Commercial Program Transition Framework (CPTF) to facilitate measures needed to transition commercial programs to eManifest. For example, as part of the CPTF, a network of representatives from impacted program areas was created in the fall of 2011 to identify commercial program policies that will need to be changed as a result of eManifest. To ensure that frontline officer information needs are addressed, the eMPD is developing an eManifest Essentials session that can be delivered in person and reflects the systems BSOs currently use. These sessions will be delivered nationally in fiscal year 2012-2013. Enhancements are also to be incorporated into an improved online learning product for BSOs and superintendents, and the need to train select regional commercial officers and superintendents has been identified by eMPD. Given the

55 Figures include GST. Source: Executive Dashboard, September 2011. 56 A Cargo Control Document submitted by a freight forwarder for shipments that have, or will be, deconsolidated from another Cargo Control Document. (Source – eManifest Component Teams Definition List) 57 At the time the recommendation was made (September 2009), the eManifest Initiative was managed by ISTB.

23

transformational nature of eManifest, the evaluation noted that there will be a need to assess eManifest end-state learning requirements and develop an integrated eManifest learning strategy. In light of these findings it is recommended that: Recommendation 2:

Programs Branch, in consultation with Human Resources Branch, undertakes a comprehensive assessment of eManifest end-state learning requirements and develops an integrated eManifest learning strategy.

Management Response: Programs Branch concurs with the recommendation. Due to the large scope of eManifest, from a systems perspective, the deployment strategy has been developed to gradually deploy the project through a series of implementations to divide the workload into manageable segments. The training approach to date has mirrored this, with activities timed to coincide with the release of new functionalities. For example, training was provided to all BSOs with the implementation of eManifest Highway in October 2010. To ensure that future training is tailored to meet CBSA needs and is aligned to the release of functionalities in the remaining builds, the Programs Branch, in consultation with the Human Resources Branch, will identify eManifest end-state learning requirements and develop a learning strategy to address these needs. Management Action Plan: Completion Date To ensure that future training is tailored to meet CBSA needs, Programs Branch, in consultation with Human Resources Branch, will: • Identify eManifest end-state learning requirements. (Training

needs analysis.) • Develop a learning strategy that is integrated within the overall

eManifest project plan. • Implementation of end-state training module.

June 2013 November 2013 July – December 2014

*****

24

In the highway mode, eManifest requires that cargo and conveyance data be transmitted to the CBSA at least one hour prior to arrival at the first point of arrival (FPOA). Concern was expressed by CBSA staff and industry association representatives over the practicality of this requirement in some regions due to close proximity of businesses to ports of entry (POEs). However, compliance monitoring conducted by eMPD on eManifest traffic processed to the end of November 2011 did not find the one-hour time frame to be a significant issue. eManifest provides for a number of transmission options including a portal specifically developed to support small to medium-sized companies. Nevertheless, some CBSA regional management and staff expressed concern that carriers operating in remote areas may find it difficult to transmit cargo and conveyance information electronically. These interviewees also noted that in some instances, carriers may not be able to provide accurate data at the time a shipment is loaded (e.g., the weight or the types of logs being transported). With respect to the latter issue, eManifest includes provisions for the submission of amendments to cargo and conveyance after arrival at the FPOA. In light of these findings it is recommended that: Recommendation 3:

Programs Branch undertakes targeted outreach activities to promote awareness of eManifest reporting options available to commercial program clients operating in remote areas.

Management Response: Programs Branch concurs with the recommendation. Client focus is a key component of eManifest. The eManifest and Major Projects Directorate continuously works with clients to resolve concerns and assist them in complying with pre-arrival commercial data transmission requirements. eManifest provides clients in remote areas with the option of using EDI transmission methods as well as free access to the eManifest Portal. As well, clients have been made aware, through the design and implementation of the eManifest communication and outreach strategy, of the CSA and FAST programs for low-risk carriers and importers and the benefits of these programs to clients that consistently import the same type of goods. eManifest also features the provision for clients to amend information post-arrival should changes be required. The Project team has visited numerous cities across Canada on an eManifest Road Show, promoting the benefits of eManifest while presenting the compliance options available to our clients. The team has also presented eManifest at many trade shows across Canada and the United States in an effort to promote awareness of reporting options available for all clients including those operating in remote areas. In addition to the efforts put forth to date, eManifest will identify targeted outreach

25

activities needed to promote awareness of eManifest reporting options available to commercial program clients operating in remote areas. Management Action Plan: Completion Date • Identify targeted outreach activities needed to promote

awareness of eManifest reporting options available to commercial program clients operating in remote areas.

• Implement targeted outreach activities.

September 2012 December 2012

*****