operationalizing open-source electronic … with some final upgrades was used in year 3 by some...

Operationalizing Open-Source

Electronic Monitoring Systems in

New England Groundfish Sectors

Year 3 Public Report

November 17, 2016

Marine Monitoring Initiative

Ecotrust Canada

www.ecotrust.ca

http://www.ecotrust.ca/

Operationalizing Open Source EM Systems in New England Groundfish Sectors: Year 3 Public Report Prepared by Ecotrust Canada’s Marine Monitoring Initiative

page 2 of 34

Table of Contents Table of Figures ............................................................................................................................................. 3 Table of Graphs ............................................................................................................................................. 3 Commonly used Acronyms ........................................................................................................................... 4 1 Introduction .......................................................................................................................................... 5

1.1. Project Description and Rationale ................................................................................................ 5

1.2. Year 3 Project Objectives .............................................................................................................. 6

1.3. Year 3 Overview ............................................................................................................................ 7

2 Methods ................................................................................................................................................ 8

2.1 Data Collection ............................................................................................................................. 8

EM System Description ................................................................................................................. 8

EM Data Collection ..................................................................................................................... 12

2.2 Data Handling ............................................................................................................................. 12

2.3 Data Analysis ............................................................................................................................... 13

Sensor Data Analysis ................................................................................................................... 13

Video Analysis ............................................................................................................................. 13

Video Analysis Feedback Loop .................................................................................................... 14

Video Review Methodology ....................................................................................................... 14

Data Comparison ........................................................................................................................ 15

2.4 Captain’s Responsibilities ........................................................................................................... 15

Fish Handling Procedures ........................................................................................................... 16

3 Results ................................................................................................................................................. 16

3.1 Non-video Data Assets................................................................................................................ 17

3.2 Video Assets ................................................................................................................................ 17

3.3 Comparisons between Logbook and ASM/Observer and EM Data ............................................ 17

3.3.1 EM vs. Logbook .................................................................................................................. 17

3.3.2 EM vs. Logbook Data Agreement ...................................................................................... 20

3.3.3 EM vs. Logbook vs. Observer ............................................................................................. 23

3.3.4 Observer Trip Data Agreement.......................................................................................... 24

3.3.6 Multi-year comparisons ..................................................................................................... 26

4 Discussion and Conclusions ................................................................................................................ 28 References .................................................................................................................................................. 34


page 3 of 34

Table of Figures Figure 1: Schematic showing EM system ...................................................................................................... 8

Figure 2: View of the 4 camera views on a trawl vessel (general view camera focus) ............................... 10

Figure 3: View of the 4 camera views on a trawl vessel (species ID camera focus) ................................... 11

Figure 4: View of the camera angle used for species identification on the gillnet vessel .......................... 12

Table of Graphs Graph 1: Discarded weight of ACE species for all reviewed hauls (2015) .................................................. 18

Graph 2: Discarded weight of ACE species for all reviewed hauls (2014) .................................................. 19

Graph 3: Discarded pieces of ACE species for all reviewed hauls (2015) ................................................... 19

Graph 4: Discarded pieces of ACE species for all reviewed hauls (2014) ................................................... 20

Graph 5: Comparison of discarded weight estimates from logbook vs. video data for all reviewed hauls (2015) .......................................................................................................................................................... 21

Graph 6: Comparison of discarded weight estimates from logbook vs. video data for all reviewed hauls by fish species (2015) .................................................................................................................................. 22

Graph 7: Discarded weight of ACE species for all reviewed hauls where an Observer was present (2015) .................................................................................................................................................................... 24

Graph 8: Discarded weight of ACE species for all reviewed hauls where an Observer was present (2014) .................................................................................................................................................................... 24

Graph 9: Comparison of logbook & video discarded weight data for all Observer trips (2015) ................ 25

Graph 10: Comparison of Observer & video discarded pieces for all Observer trips (2015) ..................... 26

Graph 11: Box plot showing the differences between Observer-reported discarded weights and video reviewer-estimated discarded weights by study year ................................................................................ 27

Graph 12: Box plot showing the differences between logbook-reported discarded weights and video reviewer-estimated discarded weights by study year ................................................................................ 27

Graph 13: Box plot showing the differences between logbook-reported discarded weights and Observer-estimated discarded weights by study year ............................................................................................... 28


page 4 of 34

Commonly used Acronyms ACE Annual Catch Entitlement ACLs Annual Catch Limits ASM At-Sea Monitor EM Electronic Monitoring Fps Frames per second FY Fishing Year GMRI Gulf of Maine Research Institute MCCS Maine Coast Community Sector NEFOP Northeast Fishery Observer Program NEFSC Northeast Fisheries Science Center NMFS National Marine Fisheries Service NOAA National Oceanic and Atmospheric Administration TNC The Nature Conservancy


page 5 of 34

1 Introduction 1.1. Project Description and Rationale Ecotrust Canada, Gulf of Maine Research Institute (GMRI), The Nature Conservancy (TNC) and Maine

Coast Community Sector (MCCS) have collaborated to develop and operationalize an innovative and cost-

effective Electronic Monitoring (EM) system as a means of obtaining accurate discard estimates of

allocated groundfish species in the New England groundfish fishery. The overarching goal of the project is

to use EM systems with video cameras to collect datasets that are comparable to those collected by the

At-Sea Monitoring (ASM) program to verify self-reported data from fishermen.

In Year 1, fishing year (FY) 2013, Ecotrust Canada developed EM technology with open source data

collection software for boats using gillnet and trawl gear. During the first year the EM system included all

the necessary equipment for collecting video, vessel tracking, and hydraulic sensor data. Ecotrust Canada

installed this system on two MCCS vessels, one trawler and one gillnetter, and a server was set up at GMRI

for data downloading and analysis. In Year 1 Ecotrust Canada also developed and began testing an

electronic logbook (eLog) to test the feasibility of integrating EM data with haul-by-haul fishing log data

in Year 2.

In Year 2, the EM system was upgraded with a third camera and deployed on seven active groundfish

sector vessels in the MCCS and Sustainable Harvest Sector: five gillnet vessels and two trawl vessels.

Ecotrust Canada’s eLog was also certified by the National Marine Fisheries Service (NMFS) in Year 2 as an

eVTR software and was piloted at the end of the fishing season. The eLog software was tested and revised,

and with some final upgrades was used in Year 3 by some participants for submitting their haul-by-haul

logbook forms.

In Year 3 of the project, FY 2015, project partners collaborated with NMFS to design a project that would

test and develop technical specifications and analytical protocols for the fishery with the intention to have

an operational EM program for participating vessels in FY 2016.

Due to a variety of reasons including vessels changing ports, changing target species and the overall

constraints of low quota allocations, several vessels did not participate in the program in FY 2015. The

Year 3 system was used on 4 vessels – 2 trawler and 2 gillnet – for a total of 60 trips and over 180 hauls.

A fourth camera was added to most vessels to provide a full-deck overview as the three cameras system

was not adequate on all vessels. In Year 3 catch handling protocols improved and further training for

captains was provided. Vessel Monitoring Plans were updated and tighter review and feedback loops were

established in the hopes of better compliance from participants and increased data quality. Additionally,

access to video data and summary data from the video review was provided to NMFS staff at the Fisheries

Science Branch (FSB) to facilitate audits of both the data collection system and the effectiveness of video


page 6 of 34

reviewers. Video review summary data was provided to NMFS staff for comparison with the eLog data

submitted by harvesters.

One expected outcome of this project was to outfit MCCS vessels with EM systems that could meet

regulatory at-sea monitoring requirements if NMFS approved the use of EM for the 2016 fishing year.

While NMFS may only approve the use of EM for certain gears and types of fishing behavior (i.e. single

stock area fishing trips or for day trips) and no approval will come before FY 2016, this is a significant step

towards facilitating the implementation of EM in this region. Further, as there are currently only a few

operational EM providers operating in North America, Ecotrust Canada endeavors to provide fishermen

with an effective and affordable option for monitoring service provision that draws on local fisheries

expertise and technical capacity through this pilot work and the partnerships involved. A second, longer-

term outcome is that the software and hardware developed for the MCCS vessels can be expanded in the

future for other vessels in the groundfish fleet, both for different fishing operations and for the varying

species catch composition in the broad stocks areas within the fishery.

EM systems, either as a replacement for human observers or in conjunction with them, could offer a way

to reduce the costs associated with ASM without compromising data quality or integrity. Cost savings are

dependent on regulatory requirements, the appropriateness of existing EM technology for those

requirements and overall EM program design. Additionally, EM opens up multiple opportunities to

increase the use of fishery-dependent data for science and management.

1.2. Year 3 Project Objectives 1. To develop a cost-effective EM system using open source technology.

2. To continue to develop onboard catch handling protocols as well as a collaborative review and

analytical process with NMFS using voluntarily collected data.

3. To use the results from FY 2015 to finalize proof of concept for an operational EM program for

participating vessels in FY 2016.

4. To test open source EM technology on up to nine active sector vessels of two different gear types,

with an option to add a tenth vessel later in the season (July/August).

5. To validate fishermen-reported estimates of species, weights and counts of discarded ACE

managed groundfish species and monkfish with data collected by EM systems and with those

collected by the NMFS NEFOP and ASM programs.

6. To test the use of full eLog integration into the EM system for haul-by-haul reporting from

fishermen.

7. To further the work commissioned by NMFS in EM weight estimation and species identification

by applying length to weight relationships in an operational environment.

8. To transfer knowledge through partnerships and to increase the capacity for the support of EM

programs in New England.

9. To collaborate with NMFS on assimilating EM data into current catch monitoring tools.


page 7 of 34

10. To investigate and attempt to alleviate variables for discrepancies between discard data sources

identified in FY 2014.

1.3. Year 3 Overview Two trawl vessels and two gillnet vessels collected data from June to December in FY 2015. All video

downloads were first screened to assess usability. Feedback letters were provided to GMRI and the

harvester outlining any data collection issues that may have come up during a trip to prevent recurrence.

These issues fell into three categories:

1. System issue – a hardware or software issue meant that data was not collected properly

2. User issue – a breach of catch handling protocol, a failure to wipe camera lenses, or a person

blocking a camera view meant that not all collected data could be fully reviewed for discards

3. Random issues – these include glare from sun making species identification or length estimates

difficult or impossible

Next, random hauls were selected from usable video and analyzed to determine the species and weight

of discarded Annual Catch Entitlement (ACE) species. For both trawl and gillnet vessels, length/weight

ratios were used to determine weight, and the total estimated weights of all discarded species for those

hauls were compared to fisherman’s log and, where applicable, to Observer collected data. Originally, the

review level for this pilot was 10% of all1 hauls from each trip, with a minimum of 1 haul per trip being

randomly selected and reviewed within three weeks of the data capture. If more than 1 haul was required

to be fully reviewed in order to reach the 10% review level, additional hauls were selected and reviewed.

When trips had ASM coverage, hauls with Observer present were selected in order to be able to compare

against that data.

By the end of Year 3, the project increased review rate to almost 100% of all useable video from FY 2015

vessels in order to create a more robust dataset for internal data comparisons and for the FSB audit of EM

software and video analysis processes. In some instances, 100% of hauls on a trip were reviewed allowing

NMFS to perform more robust analyses between summary data from video reviews and eLog.

In Year 3 the eLog software also went live, allowing vessels to use it to record and submit logbook data to

NMFS. In an effort to make the software as user friendly as possible while still meeting all federal

regulations regarding data collection and transmission, software upgrades were deployed throughout the

season. Any software bugs were fixed in-season and NMFS technicians were included in these

communications around these releases to ensure that data was submitted correctly.

1 If some hauls in a trip had ‘unusable’ video data and they could not be reviewed, they were taken out of the pool of hauls that could be randomly selected for review but they were still counted in the total hauls made in that trip, i.e. if 30 hauls were made in a trip but only 20 hauls had useable video, 3 hauls randomly selected from the 20 needed to be reviewed to meet our 10%, not just 2 hauls.


page 8 of 34

2 Methods 2.1 Data Collection

EM System Description In Year 3 EM systems were used throughout the season on 2 gillnetters and 2 trawlers. During Year 3 the

EM systems’ hardware components were:

Central control box

o Removable data drive

o Uninterrupted power supply (UPS)

Keyboard & monitor

GPS

Hydraulic pressure sensor

Three or four digital (IP) cameras

One powered IP switch

(See Figure 1)

Figure 1: Schematic showing EM system


page 9 of 34

Due to video storage concerns in Year 2, systems were upgraded to record only fishing activity at high

frame rates; all other activity was recorded at reduced frame rates2. In Year 3 this was further customized

by vessel, with high frame rates being run for a specific amount of time according to that vessel’s catch

handling procedures. This resulted in minimizing the amount of data needed to be stored on land-based

servers while still collecting all video needed for review at a high frame rate. In addition, high frame rate

video data takes up a lot of room on a hard drive. By reducing the amount of high frame rate video being

recorded, the capacity of a vessel’s hard drive was expanded. Though this was not a necessity given the

quick turnaround times on data retrievals in Year 3, it may prove important in the future when trying to

minimize the economic impact of hard drive retrievals.

In Year 3, camera 1 was positioned to capture an overview of deck activity, allowing for a view of unsorted

catch coming onboard, catch processing, and discard handling. Camera 2 focused on discards with a pilot

capture rate of up to 15fps allowing for multiple views of fish. Camera 3 captured an outboard view of

catch that dropped out of fishing gear or was handled and discarded at the rail without being brought

aboard. Camera 4 captured any blind spots not visible by other cameras to ensure all catch handling

happened in view of a camera for review. Not all vessels required a fourth camera.

On the trawl vessels, camera 2 recorded the measuring strip; captain or crew would pass each ACE discard

over the strip before throwing the discard overboard. The other three cameras were focused on the entire

deck and the discard area, respectively. The Ecotrust software allows for all 4 camera feeds to be viewed

at once and the reviewer can select which camera view is the main display (Figures 2 & 3).

On gillnet vessels, the measuring strip was adhered to the sorting tray, so the camera was focused on that

area (Figure 4). As with trawlers, the other cameras on gillnet vessels were focused on the entire deck and

the discard area. Each individual vessel was set up to have cameras in optimal locations to record fishing

activity and fish handling and these specifications were captured in individual vessel monitoring plans.

2 The higher the frame rate the better the resolution of the video, higher resolution is required for species identification but not necessary for non-fishing deck activities.


page 10 of 34

Figure 2: View of the 4 camera views on a trawl vessel (general view camera focus)


page 11 of 34

Figure 3: View of the 4 camera views on a trawl vessel (species ID camera focus)


page 12 of 34

Figure 4: View of the camera angle used for species identification on the gillnet vessel

EM Data Collection In Year 3, EM data was collected in a similar manner to Years 1 and 2 but with slight changes to low and

high frame rate capture, and according to the following software configuration:

Data was logged from the GPS and hydraulic pressure sensor every second while system was

powered.

Video recording triggered when the vessel leaves port, high-frame rate capture triggered when

hydraulics thresholds reached fishing activity levels.

The video from all cameras was captured at 15 frames per second.

Project logbook data was captured by participating Skippers on a haul-by-haul basis using either

the EM system’s integrated eLog or via a paper log.

Observer data was delivered to Ecotrust Canada by sector manager every 4-6 weeks upon request

and was manually entered into the EM database for comparison.

2.2 Data Handling EM system hard drives store up to 4 weeks of data.

Hard drives were retrieved on as close to a weekly basis as possible (fishing trip dependent).

The data from each hard drive was copied, archived and then uploaded to a server and accessed

securely over the Internet for analysis.


page 13 of 34

When logbook data was not entered directly into the eLog or collected by ASM/NEFOP programs

it was converted into an appropriate format for analysis.

At the end of the project, all raw project data will be destroyed except for the raw data released

by the project participants (e.g. sample video clips approved by the vessel owner) as per contracts

with participants.

2.3 Data Analysis

Sensor Data Analysis 100% of the sensor data was automatically analysed to identify hauls.

Hauls identified in the EM data were automatically aligned with hauls reported in the logbook

and, when available, hauls identified by the ASM/NEFOP programs.

An analyst verified the data alignment.

Video Analysis 10% of all hauls were required to be reviewed in Year 3 (with a minimum of one haul per trip).

53% of the gillnet hauls and 43% of the trawl hauls were actually reviewed, for a total of over 180

hauls reviewed by Ecotrust Canada video analysts.

o For each haul reviewed the following was recorded:

vessel name

length per fish of discarded Annual Catch Entitlement (ACE) managed species

video date

video name

any events that occur within the haul – these events are categorized as either

“Crew”, “Vessel” or “EM” and capture any additional information regarding what

is happening during the haul

timestamp of when fishing occurred on video, from GPS and includes location

data – this timestamp is also applied to every measured fish and every noted

event.

video quality

Fish lengths were converted to weights using accepted weight to length ratios by species.

Of the 60 plus trips reviewed by Ecotrust Canada, 73% of the gillnet trips and 96% of the trawl

trips were audited by FSB to check the quality of both EM system data capture and video reviewer

data analysis3. 100% of data from all reviewed hauls were submitted to NMFS for the following

haul level comparisons between EM and logbook and the ASM/Observer program when available:

3 If a captain did not follow all catch handling protocols or system maintenance and data could not be accurately collected for all hauls reviewed in a trip then an FSB audit was not possible resulting in audit levels below 100%.


page 14 of 34

the date, time and location of each haul (and when possible each set), incidental takes, and

discarded monkfish and ACE managed species weights.

Video Analysis Feedback Loop In general, the Ecotrust Canada tech team reviewed downloaded data and technical summaries, and

provided feedback to GMRI staff on technical issues to convey to participating captains within seven days

of upload of each batch of video footage. A more detailed analysis of species IDs and discard weight

estimates with a comparison to captain’s vessel trip reports, and ASM data if applicable, were to be

provided to each participating fisherman no later than three weeks from the upload date of each batch

of data. This was not always possible due to a number of outside factors, including high frequency of video

uploads during the busy season, but any urgent issues were discussed in the feedback letters and were

dealt with promptly. A charts feature was incorporated into the data analysis software to allow for quicker

and easier delivery of detailed analysis to participants by local project partners, and for real-time viewing

of season long comparisons by vessel, gear-type and/or season (please refer to Graph 1 in the results

section for an example of a graph generated by this feature).

Video Review Methodology Video reviewers identified individual fish to species (or to species-group when video did not allow for

species level identification) and estimated lengths of individual fish. In Year 3, all video analysts were

experienced NEFOP Observers and At-sea Monitors who were currently working in the New England

groundfish fishery. Species identification was done as closely as possible in accordance to their NEFOP

and ASM training and FSB auditing of video reviewers was done in FY 2015 to verify species IDs. Fish

lengths were estimated by using a measuring strip marked in centimeters that was adhered to the tray

(gillnet) or the rail of the boat (trawl). If measuring strips were unusable due to video quality (image was

fuzzy or catch-handling prevented proper view of measuring strip) or strip quality (the centimeter

demarcations had worn off or the strip was not to standard), then a hand-held measuring tool specifically

calibrated to the vessel and camera was used by video reviewers to measure the video image of the fish

and estimate length.

Reviewer lengths were converted to weights using NEFSC length-weight relationships from Wigley et al.

(2003). Those fish identified only to a species group could not be included in the weight comparison as it

was not possible to determine which specific species weight-length relationship to apply. This

methodology is consistent with the work commissioned by NMFS in EM weight estimation and species

identification (Pria et al. 2012) and allows the testing of this methodology in an operational instead of

experimental setting. When fish could not be identified to species their discarded weight was captured as

0.0 and allowed for better piece count comparisons but does create some discrepancies in logbook v.

video review weight data. For example, a captain records 10 lbs of discarded witch flounder. A video

reviewer watching the same haul has some trouble getting to the species level due to a blur on the camera


page 15 of 34

during part of the haul so they record 7 lbs of witch flounder and 3 lbs of flounder NK (unknown species).

Comparisons between the two sources will show that the captain estimated more witch flounder than the

video reviewer because the 3 lbs of flounder NK cannot be accounted for by weight.

Data Comparison The following data between EM and logbook and the ASM/Observer program was also compared when

available: the date, time and location of each haul and ACE-managed discarded species weights. Project

comparisons also included species counts for the ACE-managed discards and for monkfish. Ecotrust also

did a preliminary comparison of the discard data collected by Ecotrust Canada video analysts and FSB

video analysts which showed excellent alignment. Full results of these audits will be made available to the

public by FSB. .

ACE Regulated Species:

Atlantic cod (Gadus morhua)

Haddock (Melanogrammus aeglefinus)

Pollock (Pollachius virens)

Redfish (Sebastes spp)

White hake (Urophycis tenuis)

American plaice flounder (Hippoglossoides platessoides)

Winter flounder (Pseudopleuronectes americanus)

Witch flounder (Glyptocephalus cynoglossus)

Yellowtail flounder (Limanda ferruginea)

Other fish species involved in project:

Monkfish (Lophius spp.) – included due to historic interest in this species and inclusion in previous studies.

Halibut (Hippoglossus hippoglossus) – included due to it being listed as a priority species for monitoring

by NOAA in the region and because it is part of the NE Multispecies Fisheries Management Plan.

2.4 Captain’s Responsibilities The deck was required to be well lit during fishing activity and operators were required to monitor the

system to make sure that it was functioning during fishing. The system was developed in such a way that

the monitor in the wheelhouse displays all camera outputs and sensor function at the same time for easy

system checking by users. Operators were also responsible for keeping cameras clean and unobstructed,

and were to report any system problems immediately. Since audits are most effective by comparing EM

data and self-reported data on a haul-by-haul basis, operators also recorded either eLogs or paper logs

for each haul for later comparison with EM data.


page 16 of 34

Fish Handling Procedures As per the 2015 Vessel Monitoring Plans, below are the fishing handling procedures:

All discarded fish must be handled in view of a camera.

Discarded ACE-managed species are to be passed along or placed by the measurement strip in

view of the dedicated measurement camera.

All fish to be discarded will be returned to the sea as soon as practicable once data collection has

occurred.

Handle fish in a manner that does not impede camera views. If this is a trawl vessel, ensure that

there is a clear view of each discarded ACE species (no hands) for at least 1 second.

When an observer is on board, follow NMFS EM-observer protocol (should be provided by

observer at beginning of trip or provided by NMFS).

3 Results In Year 1, the focus was on obtaining high quality video that could be used to identify fish to the species

level and would further the work commissioned by NMFS in EM weight estimation and species

identification by applying length to weight relationships in an operational environment. While Year 1

results proved that the video captured could be used to accurately identify fish to species and get length

estimates on individual fishes, the total amount of usable video was too low to make any statistically

sound comparisons between data collected by EM systems, self-reported data from MCCS fishermen, and

data collected by the NMFS Observer and ASM programs.

Year 2 emphasized more detailed video reviews earlier in the fishing season, allowing for faster iterative

correction of video quality problems. As a result, enough hauls were captured on video to make valid

comparisons. Video quality problems that were addressed and corrected for in Year 2 included: ensuring

video from both cameras was present during fishing activity; ensuring a usable scale reference object was

kept in frame; and testing camera angles that were conducive to species identification.

In Year 3 the data collection from EM systems was consistently high, producing almost 100% usable video

data; the results of the FSB audits speak to the quality of data being captured by the EM systems and the

utility of the data analysis software developed for this project. The major cause of data loss in Year 3 came

from poor fish handling techniques by participants during early trips. In general fish handling improved

and data loss decreased as the season progressed. Overall, systems managed to collect optimal data from

hauls with higher than normal error factors.4

4 For well-handled video data, the error factors include the use of a length-weight conversion v. actual weights and the use of species groups for those individual fish that cannot be identified to species. For poorly handled fish the additional errors include factors such as: assuming fish were kept if reviewers could not witness the discard, having to use “off-screen” measuring


page 17 of 34

3.1 Non-video Data Assets Vessel position data, pressure logs, and events of fishing activity were recorded during 44 data collection

periods; these data collection periods captured data from a total of 52 trips from 4 different vessels.

3.2 Video Assets The resolution of technical and workflow issues in Year 1, particularly the implementation of a tight

feedback loop for data quality control, saw a much improved data capture record in Year 2. In Year 3, the

data capture record further improved with the majority of all data loss/unusable captured data coming

from either on-board handling errors (e.g. fish not being passed over measuring strip, not being discarded

in view, or view being blocked by crew or on-board Observers), on-board user error (e.g. system being

turned off before all discarding is finished or not being turned on until part way through a haul) or due to

miscellaneous events (e.g. water on a measuring strip making gradients too blurry to use or bright sun

washing out picture making species identification hard).

In Year 2, the following EM system issues were recurrent throughout the season and resulted in data loss

or delayed video review, in Year 3 these issues were less common and EM system errors resulted in limited

fishing video loss:

Defective or miscalibrated pressure sensor leading to extra video review requirement to discover

fishing activity

Mismatch between logbooks and apparent fishing activity as detected by the control box.

Video camera restarting and truncating videos (software was patched early in the season to

prevent this from occurring through the year)

Incorrect or changed vessel name or hull numbers, leading to miscategorised data sets.

3.3 Comparisons between Logbook and ASM/Observer and EM Data

3.3.1 EM vs. Logbook Graphs 1 – 4 use data from all participating vessels. The graphs are season-wide aggregations of discards

from all video-reviewed hauls. Graphs 1 & 3 are of 2015 data, Graphs 2 & 4 are of 2014 data.

Data Notes: In 2014, data from two hauls of one vessel was excluded because the video data did not

differentiate between discarded and retained species due to fish handling occurring outside of the field

of view. The video was analyzed but not included in the data summary.

devices which add another source of potential error to weight estimates, and increases in the use of broader species groups due to fish being covered by hands or blocked by heads.


page 18 of 34

In 2015, if a fish was seen being discarded on video but a length could not be discerned, the fish was

included in the piece counts but treated as having a weight of zero. This frequently happened with halibut

as captains were recording piece counts and weights of discarded halibut in their logbooks but it was not

always possible for video reviewers to get fish lengths (and therefore weights) for two reasons: 1)

individual fish were often longer than the measuring strip and so reliable length estimates were not

possible; and 2) fish harvesters would often be trying to get live halibut back overboard as quickly as

possible and did not always place them on the measuring strip. Other data discrepancies in 2015 came

from video methodology/data collection protocol issues. With ocean perch for instance is recorded as

‘Redfish’ by fish harvesters but species ID protocols initiated mid-season by FSB require that video

reviewers record ocean perch as ‘Scorpionfish NK’ resulting in incompatible data sets.

Graph 1: Discarded weight of ACE species for all reviewed hauls (2015)


page 19 of 34

Graph 2: Discarded weight of ACE species for all reviewed hauls (2014)

Graph 3: Discarded pieces of ACE species for all reviewed hauls (2015)


page 20 of 34

Graph 4: Discarded pieces of ACE species for all reviewed hauls (2014)

The graph of piece counts offers insight into the sources of error for the graph of discarded weights. In

video review, weights are calculated from the measured length of each fish and assume that the fish is in

good enough condition to go to market. If a fish is damaged or the reviewer is otherwise unable to

determine the length of a fish, the calculated weight for video may be inaccurate or nonexistent. 2015

pollock is a good example: the difference between video and logbook data is proportionally smaller for

piece counts than it is for weight since they are easy to identifying and enumerate but the pollock often

come aboard in very poor condition and are therefore classified as ‘Large Unmarketable Fish’ (LUMF), i.e.

they are barely more than skeletons and the fishermen cannot sell them so they are discarded at-sea. In

this case even if a length can be taken the application of the length/weight conversion will actually result

poor data alignment as the weight will be overestimated since the conversion factor is assuming live fish

in good condition.

3.3.2 EM vs. Logbook Data Agreement The following graphs show the degree to which logbook and video data agreed on what was discarded –

the closer a point is to the diagonal line, the more the sources agreed. Points move away from the diagonal

line as the difference between the two sources increases.

Missing values are graphed as zero, and thus appear along the x and y-axes. These missing values may

result from a video reviewer not seeing a discard or a fisherman not noting a particular discard. They may

also result from misidentification of species.


page 21 of 34

Each data point in these graphs represents the discard of a single species from a single haul. Data points

come from all hauls that were analyzed from all participating boats (gillnetters and trawlers combined) in

the 2015 fishing season.

Graph 5: Comparison of discarded weight estimates from logbook vs. video data for all reviewed hauls

(2015)

Overall, data points remained quite close to the diagonal line, indicating good agreement between data

sources.

Points along the y-axis indicate instances where logbook data recorded discards of ACE species but video

reviewers did not, implying that video reviewers may have been unable to see the fishermen’s act of

discarding – if the fate of the fish was unknown, video reviewers had to assume that the fish was retained.

Points along the x-axis indicate instances where the video reviewer witnessed the discard of a species that

did not appear on fishermen’s logbooks. This can occur when the fisherman and video reviewer identify

a fish as being two different species. One common occurrence of this in 2015 came from the use of Redfish

NK vs. Scorpionfish NK vs. identification to species level by harvesters; if the video reviewer used an ‘NK’

classification and the harvester went to species level for the same fish then there could be no comparison

of those data points. This discrepancy is investigated in graph series 6 below, which shows the full landed

weight dataset broken down by species.


page 22 of 34

American Plaice Atlantic Cod Grey Sole

Haddock

Halibut

Ocean Perch

Pollock

White Hake

Yellowtail

[no landings]

Graph 6: Comparison of discarded weight estimates from logbook vs. video data for all reviewed hauls by

fish species (2015)

As mentioned above there are specific reasons why some species have poorer data correlation than

others. As a recap, halibut often have poor weight comparisons since lengths often cannot be taken for


page 23 of 34

two reasons, either large fish may be out of frame with the measuring strip or harvesters are returning

them to the water without measuring them to increase survival rates. Ocean perch and pollock can have

poor correlation because of classification protocols (use of ‘redfish nk’ by harvesters v. ‘scorpionfish nk’

by vide reviewers for ocean perch and pollock being classified as LUMF by skippers and but being identified

to species by video reviewers and having a length/weight conversion applied when it probably should not

be). Finally the white hake data points along the y-axis are present because a video reviewer cannot

determine if a fish is a white hake without looking at gillrakers and so by default need to classify them as

‘Hake NK’.

Another source of error in this comparison can come from fishing handling errors (fish being discarded

out of frame and therefore assumed kept) or from poor performance relative to haul by haul recording of

ACE managed discards. As further laid out in the discussion section of this report, buy-in from participants

and good performance on fish handling and system operations protocols are essential for quality data

collection from videos.

3.3.3 EM vs. Logbook vs. Observer Not all trips for which video data was reviewed had an Observer on board. This section compares discards

solely for hauls where an Observer was present. To make valid 3-way comparisons between data sources,

unobserved hauls are excluded. As with the harvester v. video reviewer data, there are some species

specific data comparison issues with the Observer data that stem from differences in classification

protocols. In addition to the previously mentioned issues with halibut, pollock and ocean perch weights,

the lower amount of American plaice from the video reviewers comes from the fact that unless both sides

of the fish are seen the video reviewer needs to classify a probably American plaice as a ‘Flounder NK’ and

cannot identify it to species. For these four species and for white hake, the data sets collected at-sea

(harvesters v. Observers) are often more similar to each other than to the video data sets.

The differences seen between data sets was noted early in FY 2015 and changes to the FSB video review

protocols where made throughout the season in order to try and reduce these types of errors. In season

incorporation of these updated review protocols resulted in our 2015 data having much better correlation

between all three data sources than our 2014 data. As explained in more detail in the discussion section,

these results triggered an extensive review and revision of video methodology for FY 2016 and a

collaborative effort between project partners and NOAA to create standard protocols for various fish

species, for LUMFs and for fish longer than measuring strips.


page 24 of 34

Graph 7: Discarded weight of ACE species for all reviewed hauls where an Observer was present (2015)

Graph 8: Discarded weight of ACE species for all reviewed hauls where an Observer was present (2014)

3.3.4 Observer Trip Data Agreement As before, the following graph show the degree to which logbook and video data agreed on what was

caught. Each data point in these graphs represents the discard of a single species from a single haul. The

closer a point is to the diagonal line, the more the sources agreed. Points move away from the diagonal

line as the difference between the two sources increases.


page 25 of 34

Graph 9: Comparison of logbook & video discarded weight data for all Observer trips (2015)

Points above the diagonal line mean that the data source on the x-axis saw more of something than the

y-axis and points below the diagonal line mean that the data source on the y-axis saw more of something

than the data source on the x-axis. For instance, the points above the diagonal line in graph 10represent

the higher abundance of halibut, ocean perch and American plaice recorded by harvesters in their

logbooks, while the points below diagonal line on graph 10 represent the higher abundances of grey sole

and pollock recorded by video reviewers.


page 26 of 34

Graph10: Comparison of Observer & video discarded pieces for all Observer trips (2015)

3.3.6 Multi-year comparisons The following comparisons illustrate both the variances seen between each of the datasets and the

changes in those variances between 2014 and 2015.

A single datapoint in the Observer - video graph below, for example, represents the Observer-reported

value minus the video-reported value for a single species on a single haul. This dataset is limited to the

hauls where all three data sources are available – Observer, video, and logbook.

The graphs are annotated to show which of the two data sources is on which side of the x-axis. In the first

graph, for example, positive values represent cases where the Observer reported a higher value than the

video reviewer, while negative values represent cases where the video-reported value was higher. In some

cases, the minimum and/or maximum value falls outside the bounds of the graph. That value is noted

where the whisker lines exit the graph area.


page 27 of 34

Graph 11: Box plot showing the differences between Observer-reported discarded weights and video

reviewer-estimated discarded weights by study year

Graph12: Box plot showing the differences between logbook-reported discarded weights and video

reviewer-estimated discarded weights by study year


page 28 of 34

Graph13: Box plot showing the differences between logbook-reported discarded weights and Observer-

estimated discarded weights by study year

Graph 11 shows that in 2014, Observers generally recorded larger discarded weights than video reviewers.

In 2015, this difference between sources was substantially less; the box plot is compressed and nearly

centered on the x-axis.

Graph 12 shows similar results – that logbook values were much higher than those from video reviewers

in 2014, but that in 2015 these differences were reduced and centered on the x-axis.

Graph 13 compares the two non-video data sources as a reference point for the other comparisons. The

graph shows that Observer and logbook data did have some variance in 2014, though this difference was

somewhat less than what was seen in the 2014 graphs involving video data. As with the other two graphs,

the 2015 data showed much better correlation, centered near the x-axis.

4 Discussion and Conclusions The Year 3 objectives of this pilot program are listed below, with discussion and conclusions for each.

1. To develop a cost-effective EM system using open source technology.

Currently the 3-4 camera systems as developed by Ecotrust Canada cost $5,000 - $7,500. This includes

$3,300 for the EM system itself and $1,700 - $2,200 for sensors and cameras. Installation costs are $500 -


page 29 of 34

$1,000 depending on vessel size and locations of sensors and cameras. The data collection software is

open source, meaning that other people can view the code and verify, replicate, or improve upon it. The

hope is that by having the data collection software be open source, users from other fisheries and

locations can share improvements for different sensors and configurations. The data analysis and delivery

software is proprietary and is licensed to users. This software is customizable and allows for different

analysis techniques to be tested and different delivery standards to be met.

The cost of data analysis and delivery is dependent on a variety of factors, including coverage and review

levels, pass/fail criteria, and the quality and quantity of data being collected by sensors and video. The

higher the level of review and required detail, the higher the cost of analysis. Ecotrust Canada has worked

hard to determine the most cost-effective balance of data collection and cost, and is continually reducing

delivery cost estimates based on the outcomes of this pilot program. For FY 2016 the estimated per day

costs of Ecotrust Canada EM coverage for gillnet vessels is $540 USD and for trawl vessels is $450 USD.

This day rate is based on the average number of hauls per day on a groundfish trip and the fact that

coverage in FY 2016 requires full trips to be reviewed and not random hauls from all trips. The better a

harvester complies with all fish-handling techniques the lower the cost of video review as it reduces the

need for repeat viewings of individual fish and can eliminate the need to review additional trips based on

any pass/fail criteria.

2. To design a project to test and develop onboard catch handling protocols as well as a collaborative

review and analytical process with NMFS using voluntarily collected data.

In Year 3, data quality was improved though an increase in effective catch handling. The lessons learned

through this project are currently being used to design Vessel Monitoring Plans for the 2016 fishing

season.

Throughout FY 2015 project partners worked with GARFO and FSB staff as video data was reviewed and

delivered to both parties. The audits have allowed us to better highlight critical v. non-critical catch

handling issues as well as work on standard procedures for fish not identified to species or those not seen

discarded. The specifics of the species level changes to the video review methodology are laid out

extensively in the results section. These changes include creating standard protocols for LUMFs, for very

large fish and for fish not identified to the species level.

Finally, the delivery of video review summaries to NMFS has allowed us to work out the technical issues

around data delivery and set the stage for ‘live’ delivery of data in FY 2016.

3. The results of this project in FY 2015 would finalize proof of concept for an operational EM

program for participating vessels in FY 2016.


page 30 of 34

Due to the collaborative nature of this project in FY 2015 and the ongoing working relationship with

regional NOAA staff and the FSB, the EM implementation project has helped create a place for the use of

EM technology in FY 2016. In FY 2015, project partners collaborated extensively with different branches

of NOAA to develop and test various aspects of EM delivery in the northeast region. Specially, participants

gave permission to share data with the FSB in order to have Ecotrust Canada video reviewers’ data

collection audited; video was reviewed by agency staff and the results were compared to the same data

collected by project video analysts as a quality assurance measure. A second collaboration with GARFO

created data collection and delivery specifications and included a testing phase to work out technical

glitches and the syncing of two data streams.

4. To test open source EM technology on up to nine active sector vessels of two different gear types,

with an option to add a tenth vessel later in the season (July/August).

Due to unforeseen circumstances including vessels changing port, changing fisheries and low quota issues,

many vessels did not participate in the program in FY 2015. The project was able to collect data from 2

gillnet vessels and 2 trawl vessels throughout the fishing year. From these 4 vessels enough data was able

to be collected for a robust FSB audit and for statistically significant data comparisons between all data

sets (video collected data, logbook data and Observer collected data).

5. To validate fishermen-reported estimates of species, weights and counts of discarded ACE

managed groundfish species and monkfish using data collected by EM systems and by the NMFS

Observer and ASM programs.

Comparisons between EM collected data and data collected by both NEFOP Observers and at-sea

monitors were made when all data sources existed. On those vessels and trips where fish handling by

harvesters and by Observers/Monitors was good enough for complete video review these comparisons

lead to useful data. There continues to be discrepancies between video analyst data and

Observer/Monitor collected data caused by different levels of fish identified (Observer and Monitors may

go to species on some fish but video analysts only to species group) and in some weight comparisons due

to either conversion error or measuring inaccuracies (often due to bent or curled fish), but these are

minimal and obvious in the data.

If a vessel had poor compliance with fish handling techniques, or if those techniques had not been fully

worked out, then data comparisons and validations could not take place (i.e. there was no way to collect

adequate data from the video and so no haul level comparisons of data sets were made). In these cases

additional training and/or updated fish handling techniques were needed in order to be able to collect

the necessary video data for comparison. These early issues helped highlight the importance of

communication and training with harvesters and the overall need of user support of the system and


page 31 of 34

program for successful implementation. Updated vessel monitoring plans and training resources have

been created to help harvesters in FY 2016.

6. To test the use of eLog, which is fully integrated into the EM system, for haul-by-haul reporting

from fishermen.

The eLog was tested and used throughout FY 2015. There were technical issues stemming from both the

Ecotrust Canada software and from NMFS, but they were mostly resolved in-season. Fisherman input was

essential to making in-season adjustments to the eLog. Although there were complications with recoding

without a proper testing period, the benefits achieved far outweighed any software glitches; Ecotrust

Canada technical staff were able to accommodate suggestions from fishermen like species short lists or

automating certain fields. By the end of the season, fisherman reported the system being much easier to

use and the time it took to enter a trip was only 1/3 of what it took them at the beginning of the season.

Final testing of software at the start of FY 2016 is recommended and will require ongoing collaboration

with NMFS.

7. To transfer knowledge through partnerships and to increase the capacity for the support of EM

programs in New England.

8. To collaborate with NMFS on assimilating EM data into current catch monitoring tools.

The year-long collaboration between project partners and different staff and branches of the NOAA has

helped to assimilate EM data into current catch monitoring tools. As mentioned in other parts of this

report, extensive work was done with FSB to establish video review standards for the most common data

discrepancies. Additionally, a great deal of work was put in by all parties to successfully upload EM

summary files into the API at GARFO with all needed data elements. These two efforts have allowed the

agency to audit Ecotrust Canada’s EM systems and the data being collected and analyzed. It has also

allowed all parties to understand technical adjustments needed for the successful collection and delivery

of EM data during FY 2016.

9. To further the work commissioned by NMFS in EM weight estimation and species identification

by applying length to weight relationships in an operational environment.

10. To investigate and attempt to alleviate variables for discrepancies between discard data sources

identified in FY 2014.

Results from 2015 show a marked increase in the agreement between video review data and logbook data

than in previous years. Some discrepancies still remain, however, which are largely caused by:

Difficulty obtaining accurate measurements of fish under certain circumstances, i.e. due to curled

fish or fish whose length is greater than the measuring strip


page 32 of 34

Seasonal fluctuations in species length:weight ratios

The inability to fully identify fish, resulting in the use of species groups which do not have a

conversion factor such as Fish NK or Flounder NK

Difficulty in obtaining measurements on species that captain and crew have a heightened interest

in getting back into the water

Use of different classifications of fish by different data sources, e.g. harvesters using Redfish NK

for Ocean Perch but video analysts using Scorpionfish NK

Use of LUMFs classification by harvester & Observers but not by video reviewers

Occasional lack of adherence to vessel monitoring plans resulting in the inability to collect quality

EM data

Misidentification of catch

Catch discarded outside of camera view, if all catch is being identified on camera could some of

these disagreements be corrected and/or explained by a secondary comparison, i.e. both data

sources identified species X but video data did not see it being discarded. This would speak to the

need for a behavioral change from operators or a readjusting of a camera instead to implying EM

systems inability to capture good enough data for species identification and/or the video

reviewer’s inability to accurately identify fish on video.

Accuracy of captain’s fish counts or weight estimates

Transcription errors.

Inaccuracy of estimation formulae.

Inaccuracy of length estimation.

In Year 2 comparisons were made between EM data, self-reported logbook data and Observer/Monitor

data, despite Observer/Monitor trips only accounting for 12% of the trips analyzed. Video to be reviewed

was chosen randomly but attention was paid to Observer/Monitor trips so that those comparisons could

be made as often as possible. In order to troubleshoot sources of logbook/EM data disagreement through

potential comparisons of results between studies, the same length to weight relationships that were used

by NMFS were applied to our data.

In FY 2015, discarded weight comparisons between logbook and EM data was high when both data sets

recognized the presence of a species. However, when one data source identified a species and another

did not then R2 values dropped substantially. Data sources not aligning can happen for three main reasons

– 1) the species was not seen at all, 2) the species was not seen being discarded, or 3) a fish was not

identified to same taxonomical level. As a result, some species and vessel/gear combinations exhibited

substantial disagreement in discard counts in logbook and EM video review data. This underlines the

importance of having multiple bases for comparison, such as Observer/Monitoring data.

An important aspect of FY 2016 EM use will be to more clearly identify and understand sources of this

disagreement, while keeping in mind that a control or baseline for the accuracy of the logbook does not


page 33 of 34

exist. By conducting an audit of the video reviewed data in FY 2015, FSB now has a database that highlights

weight discrepancies by species and month to reduce this source of error. Similar to the methodology for

improving video usability in 2014, in 2015 Ecotrust Canada recommended “real time” generation and

review of agreement charts and metrics to identify vessels not only problematic video footage, but any

repeated source of error in general. This software update was completed in FY 2015 and added real value

to the pilot study. As the project grows in FY 2016 an even more robust video review methodology will be

finalized prior to data collection and that standards for recurrent issues will be created to facilitate better

logbook and Observer/Monitoring data validation.

Over the 3-year pilot period, this work has informed the ongoing discussions around EM feasibility, utility,

operability, and cost, and has highlighted the importance of user buy-in and firmly established monitoring

coverage rates when determining the feasibility and costs related to establishing EM programs. This study

has also demonstrated that for the New England groundfish fishery EM can not only be made operational

but is also reliable fisheries data collection tool and is a viable alternative to ASM. This work has shown

that EM technology is capable of identifying individual fish to species, measuring the fish, and assessing

whether they are kept or discarded, and that video analysts can be trained to collect this data and that

the data collection is replicable. Furthermore, the collaborative nature of this pilot meant that not only

were data collection and delivery processes developed but were also audited and tested by FSB and

GARFO. This means that regulators are prepared to receive EM-collected fisheries data in FY 2016 from

New England fishing sectors that wish to use EM as an alternative to ASM.

The success of this program in creating policy change in New England would not have occurred without

the tireless effort of the project staff members at the Gulf of Marine Research Institute, the Nature

Conservancy, the Maine Coast Community Sector and Ecotrust Canada. Nor could any of this have taken

place without the generous funding of the Gordon & Betty Moore Foundation and an anonymous donor,

and without the participation and input from fish harvesters in the New England groundfish fishery. Thank

you all for your contribution to this work.

For information regarding this work or to make questions or comments about this report please contact

Amanda Barney, General Manager of the Marine Monitoring Initiative at Ecotrust Canada via email at

[email protected]

mailto:[email protected]


page 34 of 34

References Pria, M.J., McElderry, H., Fedoruk, A. and McVeigh, R. 2012. Estimating Weight and Identifying Species

through Electronic Monitoring (EM): A Preliminary Comparison of Electronic and Observer-Based

Reporting. Interim Technical Review prepared for the Fisheries Sampling Branch of the Northeast Science

Center by Archipelago Marine Research Ltd., Victoria, BC Canada. 37 p.

Wigley, S.E., McBride, H.M., and McHugh, N.J. 2003. Length-Weight Relationships for 74 Fish Species

Collected during NEFSC Research Vessel Bottom Trawl Surveys, 1992-99. NOAA Technical Memorandum

NMFS-NE-171. 26 pp.

operationalizing open-source electronic … with some final upgrades was used in year 3 by some...

Documents