EQ

MGa

b

a

ARRA

KSODCBE

1

tp1

D(((

h1

Ecological Indicators 45 (2014) 677–686

Contents lists available at ScienceDirect

Ecological Indicators

j o ur na l ho me page: www.elsev ier .com/ locate /eco l ind

nvironmental sustainability indicators for cash-crop farms inuebec, Canada: A participatory approach

arie-Noëlle Thiviergea, Diane Parenta, Valérie Bélangera, Denis A. Angersb,uy Allarda, Doris Pellerina, Anne Vanassea,∗

Faculté des sciences de l’agriculture et de l’alimentation, Université Laval, Québec, QC, Canada G1V 0A6Soils and Crops Research and Development Centre, Agriculture and Agri-Food Canada, 2560 Hochelaga Boulevard, Québec, QC, Canada G1V 2J3

r t i c l e i n f o

rticle history:eceived 17 June 2013eceived in revised form 22 April 2014ccepted 11 May 2014

eywords:ustainable agriculturen-farm assessmentecision-aid toolropping systemar chartnd-use validation

a b s t r a c t

On-farm environmental assessment, with consideration to the specificity of the farming system and thegeographic zone, can enable farmers to include the environmental aspect in their management decisions.In the province of Quebec, Canada, 45% of the cultivated land is dedicated to grain production and amongthe 13,800 farms that sell grains, 3975 are specialized in this production. Cereal-based systems havetheir own constraints and realities and could benefit from a specific tool to assess their environmen-tal sustainability. The objective of this research was to adapt and further develop a set of indicators ofenvironmental sustainability at the farm level for cash-crop farms of the province of Quebec, in order toprovide a self-assessment and decision-aid tool to farmers. Using a methodology based on focus groupsof experts (researchers, stakeholders, and farmers), several indicators developed for dairy farms wereadapted to cash-crop farms. Then the set of indicators was tested on cash-crop farms across the provincethrough interviews with 31 farmers. The indicators were weighted according to their contribution tofour sub-objectives of environmental sustainability (soil, water, air, and biodiversity conservation). Anew type of chart was designed to help farmers understand and interpret the scores obtained from theset of indicators. Finally, a questionnaire was sent to the 31 farmers for end-use validation. A total of 16indicators emerged from this research. The weighting reveals that, out of a total of 177 points, the indi-cators that contribute the most to environmental sustainability of cash-crop farms are “integrated pestmanagement” (21 points), “crop diversity” (19 points), “riparian buffer strip” (18 points), and “incorpo-

ration of manure into the soil” (16 points). In comparison with a radar chart and a conventional bar chart,a new bar chart revealed to be a better decision aid tool, allowing the majority of farmers to identify thesustainability weaknesses of a fictive farm. However, the graphic design of this chart could be improvedfor easier understanding. The end-use validation confirmed the interest of farmers in this decision-aidtool.

© 2014 Elsevier Ltd. All rights reserved.

. Introduction

Environmental sustainability can be defined as the main-

enance of natural capital, which comprises the resourcesroviding sink and source functions in ecosystems (Goodland,995; Van Cauwenbergh et al., 2007). Many attempts to address

∗ Corresponding author. Tel.: +1 418 656 2131x12262; fax: +1 418 656 7856.E-mail addresses: Marie-Noelle.Thivierge.1@ulaval.ca (M.-N. Thivierge),

iane.Parent@fsaa.ulaval.ca (D. Parent), Valerie.Belanger@fsaa.ulaval.caV. Bélanger), Denis.Angers@agr.gc.ca (D.A. Angers), Guy.Allard@fsaa.ulaval.caG. Allard), Doris.Pellerin@fsaa.ulaval.ca (D. Pellerin), Anne.Vanasse@fsaa.ulaval.caA. Vanasse).

ttp://dx.doi.org/10.1016/j.ecolind.2014.05.024470-160X/© 2014 Elsevier Ltd. All rights reserved.

sustainability have been made since the Rio Earth Summit of 1992,through efforts from several countries to establish indicators formeasuring progress (Rigby et al., 2001). Indicators are variablesthat provide information on other variables that are less available(Gras et al., 1989). They simplify the information (Andersen et al.,2013; Girardin et al., 1999; Mitchell et al., 1995; Rigby et al., 2001;Singh et al., 2012) and serve as a benchmark to make a decision(Gras et al., 1989) or to quantify the degree of compliance withenvironmental objectives (Van der Werf et al., 2007).

In agriculture, on-farm assessment is essential to guide farmers

with their management decisions (Häni et al., 2003; Pacini et al.,2003; Van Cauwenbergh et al., 2007). The use of a set of indi-cators constitutes a holistic approach that takes into account allagricultural practices within the system (Bockstaller et al., 1997).

6 ical Indicators 45 (2014) 677–686

OstiBtovtItEpgmtota2tbi

saCuptoptitM(twaf

aQdiao

2

F

robde

2

bi

Fig. 1. Conceptual framework of the methodology developed to adapt a set of indi-

(AOR, 2002) was also used as a data source. To be easy to imple-

78 M.-N. Thivierge et al. / Ecolog

ne of the first sets of indicators at the farm level was the Farmerustainability index of Taylor et al. (1993), with 33 weighted indica-ors designed for cabbage farmers in Malaysia. Their methodologyncluded a panel of experts as well as interviews with farmers.ockstaller et al. (1997) and Girardin et al. (2000) went one step fur-her by linking their indicators with sustainability sub-objectivesr components. Their AGRO*ECO method for cash-crop farms wasalidated in France and Germany, and the results were presentedo farmers using a radar chart. Since its first edition in 2000, theDEA method (Vilain et al., 2008) moved the focus to the educa-ional aspect of assessing the sustainability at the farm level. TheVAD method (Rey-Valette et al., 2008), inspired by IDEA’s princi-les, improved and documented the methodology to help farmerroups to construct their own set of indicators using a participatoryethodology. The MOTIFS method (Meul et al., 2008) contributed

o the user-friendliness of this kind of tool with an improved versionf the radar chart. Other indicators and methods for on-farm sus-ainability assessment were developed within the European Union,s the Common Agricultural Policy (CAP) reform under Agenda000 made of sustainable development a priority (Commission ofhe European Communities, 1999). Recently, in the province of Que-ec, Canada, Bélanger et al. (2012) developed agri-environmental

ndicators to specifically assess the sustainability of dairy farms.Indicators at the local production level have to reflect site-

pecific characteristics (Sattler et al., 2010), including the climaticnd natural conditions of the site (Commission of the Europeanommunities, 1999), and the particularities of the farming systemnder study (Meul et al., 2008). The climatic and natural conditionsrevailing in Quebec differ from those of Europe, mostly regardinghe length of the growing season, the water regime, and the naturef arable soil. As those factors have a strong influence on croproduction, it appears relevant to offer farmers a tool adapted toheir specific conditions. Moreover, in Quebec, grain production hasncreased by 25% between 1998 and 2007 (BPR, 2008), and 47% ofhe cultivated land is now dedicated to this production (ISQ and

APAQ, 2013). Cash crops in Quebec mostly include grain maizecorn), wheat, oats, barley, canola (colza), and soybeans. Amonghe 13,800 farms that sold grains in 2010, there were 3975 forhich it accounted for more than half of the farm income (ISQ

nd MAPAQ, 2013; Statistics Canada, 2012). Therefore, this specificarming system deserves some attention.

The objective of this research was to adapt and further develop set of farm-level indicators of environmental sustainability foruebec cash-crop farms, in order to provide a self-assessment andecision-aid tool to farmers. Complementary objectives were to

mprove the methodology to allocate weights to such indicators,nd to design a new type of chart leading to a better interpretationf the scores resulting from the sustainability assessment.

. Methodology

The conceptual framework of the methodology is illustrated inig. 1 and will be detailed in Sections 2.1–2.5.

The steps in the construction of indicators are interactive: theesults from one step could lead to some modifications in previ-us ones (Rey-Valette et al., 2008). Those feedbacks are illustratedy the arrows in Fig. 1. Furthermore, this methodology can beescribed as adaptive and iterative (Meul et al., 2009; Rey-Valettet al., 2008).

.1. Adaptation of indicators from dairy farms to cash-crop farms

The original set of indicators from Bélanger et al. (2012) hadeen developed using the Delphi method (Delbecq et al., 1975) to

nquire 25 experts through anonymous individual questionnaires,

cators of environmental sustainability to cash-crop farms of the province of Quebec.The arrows illustrate the many feedbacks, making it an adaptive and iterative pro-cess.

for several rounds of questions. Thereafter, 12 experts (researchers,stakeholders, and farmers) were gathered to discuss the results ina panel, also referred to as a focus group. See Bélanger et al. (2012)for the detailed methodology regarding the Delphi method and thefocus group. This participatory approach is named co-constructionof indicators (Rey-Valette et al., 2008) or bottom-up approach(Fraser et al., 2006; King et al., 2000; Singh et al., 2012). Accord-ing to Rey-Valette et al. (2008), it is important to bring togetherdifferent stakeholders, including farmers, in the process of indi-cator construction. The inputs of farmers, often neglected in suchprocesses, increase the likelihood of the indicators being acceptedby the users (Dalal et al., 1999; Fraser et al., 2006; King et al., 2000).

Thus, to adapt the dairy farm indicators from Bélanger et al.(2012) to the reality and constraints of cash-crop farms, the sametype of methodology based on the consultation with experts waschosen, though with a smaller panel of eight experts (researchers,stakeholders, and farmers). The evaluation criteria described byBélanger et al. (2012) were being sought during the adaptation pro-cess and must be seen as guidelines. Thereby, selected indicatorsshould aim at being: (1) easy to implement, (2) immediately under-standable, (3) reproducible, (4) sensitive to variations, (5) adaptedto the objectives, and (6) relevant for users (see Bélanger et al.,2012, for a detailed description of these evaluation criteria). Thediscussions among experts were recorded for future references.

2.2. Testing of the indicators on cash-crop farms

After a first focus group with the panel of experts, the selectedindicators were tested on 31 cash-crop farms across eight areas ofthe province of Quebec (Table 1). A cash-crop farm can be definedas a farm where cash crops production accounts for 50% or moreof its income (Statistics Canada, 2012). The objectives of thesetests were to validate the calculations for each indicator, verifyif the indicators fulfilled some of the criteria (criteria 1, 3, and 4of Section 2.1), establish their suitability for all cropping systems,and determine whether the questions were understandable to allfarmers. The farms were recruited with the help from several Agri-Environmental Advisory Clubs across the province.

For each farm, a one-to-one interview with the farmer was con-ducted. During this 2-h interview, a questionnaire was filled withthe farmer. The agri-environmental fertilization plan of the farm

ment, on-farm indicators must take advantage of the informationalready available that is credible (Bockstaller et al., 1997; Halberg,1999; Meul et al., 2009; Mitchell et al., 1995; Rigby et al., 2001).Feedbacks from farmers were collected to improve the indicators.

M.-N. Thivierge et al. / Ecological Indicators 45 (2014) 677–686 679

Table 1Characteristics and geographical distribution of the 31 cash-crop farms used for the testing of the indicators.

Geographic areas Number offarms

Average landarea (ha/farm)

Type of production Type of soil tillage

Organicproduction

Conventionalproduction

Conservationtillage only

Conventionaltillage only

Mix of bothtypes

Montérégie-Ouest 6 275.9 2 4 2 0 4Montérégie-Est 6 183.6 2 4 4 0 2Centre du Québec 2 348.2 0 2 1 0 1Estrie 4 317.3 2 2 0 2 2Chaudière-Appalaches 2 41.0 0 2 1 1 0Bas-St-Laurent 5 266.6 0 5 3 0 2Mauricie 2 196.0 1 1 0 0 2

2

iidbTooeepssgboseiwbts

2tsdwovtc

tcmatsspletaac

Saguenay/Lac St-Jean 4 164.3 1

31 231.8 8

.3. Weighting of the indicators

The weight is the measure of the relative importance of an itemn an ensemble (Morris, 1992). Here, the importance of an items seen in regard of environmental sustainability, which can beefined as the maintenance of natural resources that are providedy the ecosystem (Goodland, 1995; Van Cauwenbergh et al., 2007).o facilitate a structured discussion about weighting, the conceptf environmental sustainability was subdivided into four sub-bjectives. These sub-objectives were based on the four challengesnsuring a sustainable agriculture (Lefebvre et al., 2005; Michaudt al., 2006) and are: (1) soil quality conservation (comprisinghysical, chemical, and biological aspects), (2) water quality con-ervation (regarding pollution by fertilizers, pesticides, suspendedolids, and pathogens), (3) air quality conservation (regardingreenhouse gases, ammonia, and pesticides), and (4) abovegroundiodiversity conservation. The aim of these subdivisions is not toversimplify the relationships existing in this ecosystem, but totructure the discussions among experts. According to Lefebvret al. (2005), biodiversity comprises the indigenous species which,f reduced, will disrupt the ecosystem functions. To avoid double-

eighting of some indicators, it was decided that belowgroundiodiversity (microorganisms, fungi, etc.) would be comprised inhe biological aspect of soil quality, rather than as part of biodiver-ity.

Although farmers were consulted for the adaptation (Section.1), testing (Section 2.2), and validation of the indicators (Sec-ions 2.4 and 2.5), they were not convened for the weighting. Thisecond panel gathered seven experts (researchers and stakehol-ers). As reported by King et al. (2000), the bottom-up approach,hich favors the creation of indicators by the users (farmers), is

ften criticized because it implies that scientific knowledge has lessalue than that of farmers. We argue that the weighting of indica-ors should be based on scientific references in order to reflect theontribution of each indicator to environmental sustainability.

The indicators were weighted according to their contributiono each of the sub-objectives. A scale was built from 0, for a nilontribution of the indicator to a sub-objective, to 5 points, for aajor contribution of the indicator to a sub-objective. The points

warded to indicators act as units of sustainability. Care was takeno be consistent both horizontally (between sub-objectives for theame indicator) and vertically (between indicators for the sameub-objective), which required several adjustments (Table 2). Theoints of sustainability of an indicator do not constitute an abso-

ute value, but rather an indication of its relative importance (Rigbyt al., 2001) compared to others in the achievement of environmen-

al sustainability as defined in this study. The total contribution ofn indicator to the overall environmental sustainability was takens the sum of the individual contributions of this indicator to theonservation of soil, water, air, and biodiversity (Table 2). As an

3 2 2 023 13 5 13

implicit premise behind this methodology, the four sub-objectives(soil, water, air, and biodiversity conservation) were assumed ofequal importance in environmental sustainability.

2.4. Global expert validation

According to Girardin et al. (1999), to validate indicators is toverify if they meet the objectives for which they were created. Thiscorresponds to the “accuracy evaluation” described by Meul et al.(2009). As an indicator is by definition a variable that providesinformation on other less accessible variables (Gras et al., 1989),it is often impossible to validate it by comparing with real data(Bockstaller and Girardin, 2003; Girardin et al., 1999; Rigby et al.,2001; Vilain et al., 2008). Furthermore, there is rarely a linear rela-tionship between an indicator and a given measure (Bockstaller andGirardin, 2003; Girardin et al., 1999).

The participation of experts and the constant reference to sci-entific literature were considered to be an a priori validation of theset of indicators, called design validation (Bockstaller and Girardin,2003; Meul et al., 2009). Moreover, after the weighting of the indi-cators by the panel of experts, the final product was sent to all theexperts involved at this point for a global expert validation or outputvalidation (Bockstaller and Girardin, 2003; Rigby et al., 2001; Vilainet al., 2008). All the experts (researchers, stakeholders, and farmers)had the opportunity to provide comments about the calculation orthe weighting, which were used to make small adjustments to theset of indicators. We consider that this validation is still in progress:publishing the results in peer-reviewed scientific journals is part ofit, as independent experts will express their comments (Meul et al.,2009; Vilain et al., 2008). Furthermore, as this set of indicators willbe implemented on farms, more feedbacks and comments fromfarmers and advisors will be taken into consideration.

2.5. End-use validation

The end-use validation, or usefulness test, is a feedback processbetween users (the farmers to whom the indicators are dedicated)and designers of the indicators (Bockstaller and Girardin, 2003;Bockstaller et al., 1997; Girardin et al., 1999; Mitchell et al., 1995). Itcorresponds to the “credibility evaluation” described by Meul et al.(2009). It can be performed through a survey among users (Girardinet al., 1999) to verify their satisfaction with the proposed tool (Vilainet al., 2008), their understanding of the results (Bockstaller andGirardin, 2003), and the usefulness of the indicators as a decision-aid tool (Meul et al., 2008).

Thereby, a questionnaire was sent to the 31 farmers who par-

ticipated in the interviews, and it was filled and returned by 16 ofthem. The questionnaire was designed to inquire about: (i) the per-ceptions of farmers about the proposed approach, (ii) their opinionabout the scores of their farm, (iii) their willingness to use this set

680 M.-N. Thivierge et al. / Ecological Indicators 45 (2014) 677–686

Table 2Example of the weighting of three indicators according to their contribution to environmental sustainability sub-objectives. Scale 0 (nil contribution) to 5 points ofsustainability (major contribution).

Indicator Contribution to environmental sustainability sub-objectives Contribution tosustainability

Soil quality Water quality Air quality Biodiversity Maximum weight ofthe indicator

ossTm

3

3

(f

Banh2aortofit(2DricaDt

cVg(firItcpmtirQbi

Soil organic matter content (#1) 5 3

Soil phosphorus saturation (#2) 1 4

Deep and surface drainage (#3) 4 2

f indicators, and (iv) the most effective chart to illustrate theircores. It allowed the authors to verify if the indicators fulfilledome of the selection criteria (criteria 2, 5 and 6 of Section 2.1).he questionnaire, which could be filled out in 20–30 min, includedultiple-choice questions and spaces for feedbacks.

. Results and discussion

.1. Selected indicators

The adaptation of the dairy farm indicators of Bélanger et al.2012) by the panel of experts led to 16 indicators for cash-croparms, grouped into four components (Table 3).

As mentioned in the methodology, the original indicators fromélanger et al. (2012) had been developed using the Delphi methodnd focus groups. Despite its high value to structure group commu-ication when dealing with complex problems, the Delphi methodas some limitations that must be recognized (Linstone and Turoff,002). In the literature, most of the common pitfalls reported aressociated with unclear objectives (too specific or too vague), a lackf criteria in the selection of experts, bias introduced by the mainesearcher’s influential position, the desire for excessive simplicityhat leads to a reductionist approach of the studied system, the lossf interest from the participants over time, the creation of an arti-cial consensus by ignoring some dissident opinions, and finally,he risk to conclude that consensus always means righteousnessHasson and Keeney, 2011; Keeney et al., 2001; Linstone and Turoff,002; Vernon, 2009). Many of these pitfalls are not unique to theelphi method, but apply to many consensus techniques in action

esearch. Taking appropriate precautions and documenting explic-tly the communication process, as the authors endeavored to do,an reduce the vulnerability of the method to criticism (Linstonend Turoff, 2002; Vernon, 2009). As stated by Vernon (2009), theelphi method “will only be as robust as the researchers’ justifica-

ion for their protocol.”Indicators #4 to #16 (Table 3) are means-based indicators, also

alled action variables or management indicators (Payraudeau andan der Werf, 2005; Von Wirén-Lehr, 2001). These indicators areenerally based on methods or means of practicing agricultureBockstaller et al., 2008), which arise from decisions taken by thearmer (e.g. to incorporate or not the manure into the soil). Its therefore an indirect assessment, or a prediction, of the envi-onmental impact (Bockstaller et al., 2008; Rigby et al., 2001).ndicators #1 to #3 are state indicators, or effect-based indica-ors (Payraudeau and Van der Werf, 2005; Von Wirén-Lehr, 2001),alculated from measures taken directly into the field (e.g. soilhosphorus saturation). They identify the state of the environ-ent at a given time (Gras et al., 1989) but without identifying

he cause–effect relationship (Bockstaller et al., 2008). This set ofndicators being intended as an educational tool, it was considered

elevant to combine means-based and state indicators. Moreover,uebec’s regulation (AOR, 2002) provides requirements coveringoth agricultural practices and state of the environment. Accord-

ng to Heink and Kowarik (2010) and Rey-Valette et al. (2008), it

2 2 120 3 84 0 10

is possible to use different types of indicators within the same set,provided that the method of construction and calculation of eachindicator is specified. In an effort of transparency, the three stateindicators were then identified as such in the component state ofthe soil resource (Table 3).

Among the many differences from the dairy farm indicatorsof Bélanger et al. (2012) is the addition of an indicator referringto energy consumption. On cash-crop farms of Quebec, 52% ofthe energy consumption can be attributed to the use of diesel,mainly for motorized vehicles (AGECO, 2006). The diesel consump-tion is difficult to quantify at the farm level because diesel isused for personal purposes as well as for farm work (Bélangeret al., 2012). Furthermore, the intensity of soil tillage, which islinked to the fuel consumption, is already assessed with indi-cator #4 (Table 3). Hence, diesel consumption was not retainedas a potential indicator. As second in order, the propane gasaccounts for 23% of the total energy consumption of cash-cropfarms and is mainly used to dry maize grains (AGECO, 2006; LaFinancière agricole du Québec, 2010). For this reason, the drying ofmaize was considered relevant as an energy means-based indicator(#8, Table 3).

The other differences from Bélanger et al. (2012) includethe addition of indicators about split nitrogen applications (#11,Table 3), presence of annual legume crops in the rotation (#6C),seed treatment (#7C) and implementation of refuges along withinsect-resistant transgenic crops (#7D), and the removal of the indi-cator about manure storage. Also, major changes have been madein the calculation of indicators #1, 3, 5, 6A, 6B, 12, and 13.

Most of the indicators (Table 3) are expressed with intervalclasses (0–20–40–60–80–100%) rather than a continuous scale(0–100%). This allows some consistency between quantitative andqualitative indicators and avoids putting too much emphasis onthe accuracy of the measurements but rather on the diversity ofthe selected indicators (Rey-Valette et al., 2008).

3.2. Modifications following on-farm testing

The testing on 31 farms led to improvements for some indicatorsin regard with their understanding by farmers. For erosion in slop-ping fields (indicator #13), farmers were first asked to identify areaswith erosion issues by coloring their farm map. Large differenceswere observed in the way farmers responded. Many of them did nothave the expertise to diagnose soil erosion problems. Gomez et al.(1996) faced a similar problem and used cover crops as an indirectassessment of the risk of soil erosion. The objective behind indicator#13 is not to evaluate the farmer’s ability to identify problems of ero-sion, but rather the susceptibility of farmlands to erosion (Joel Aubin,2009, pers. comm.). In such a case, one solution is to ask a successionof yes or no questions to the farmer regarding concrete and objectivekey factors that play a role in erosion (e.g. slopes, plowing, cover

crops, and riparian buffer strip). By aggregating the answers tothese questions, it is possible to estimate more objectively the sus-ceptibility of farmlands to erosion. It was decided to present thesesuccessive yes or no questions in the form of a decision tree (Fig. 2),

M.-N. Thivierge et al. / Ecological Indicators 45 (2014) 677–686 681

Table 3Definition of the 16 indicators for the assessment of environmental sustainability on cash-crop farms in the province of Quebec.

Component Indicator(number)

Sub-indicator Indicator definition Weight

State of soilresource

Soil organicmatter content(#1)

Proportion of the cultivated area with soil organic matter content greater or equal to 3% 12

Soilphosphorussaturation (#2)

Proportion of the cultivated area with soil phosphorus saturation under the maximum levelallowed according to the province of Quebec regulation (<7.6% for clay soils; <13.1% for othersoils)

8

Deep andsurfacedrainage (#3)

Proportion of the cultivated area with deep or surface drainage problems (0% of area withproblems = score of 100%; 1–5% = score of 80%; 6–10% = score of 60%; 11–15% = score of 40%;16–20% = score of 20%; >21% = score of 0%)

10

Croppingpractices

Conservationtillage (#4)

Proportion of the cultivated area under different types of conservation tillage (areas underno-till = score of 100%; ridge-till = score of 80%; other reduced-till = score of 50%; conventionaltill = score of 0%)

9

Cover crops(#5)

Proportion of annual crops area sown with or followed by green manure or cover crop (areaswith green manure incorporated into the soil after winter = score of 100%; with green manureincorporated before winter = score of 70%; with crop remaining during winter (e.g. winterwheat) = score of 40%; without green manure or cover crop = score of 0%)

10

Crop diversity(#6)

Sequence ofcrops (6A)

The main crop rotation: (a) includes 3 different crops and (b) does not include the same annualcrop more than 2 years in a row (a and b = score of 100%; only a or only b = score of 70%; noneof those = score of 0%)

7

Perennial crops(6B)

Proportion of the cultivated area with perennial crops (≥15% of the cultivated area = score of100%; 12–14.9% = score of 80%; 9–11.9% = score of 60%; 6–8.9% = score of 40%; 3–5.9% = score of20%; 0–2.9% = score of 0%)

8

Annual legumecrops(Fabaceae) (6C)

Proportion of annual crops area cultivated with annual legume crops (≥30% of annual cropsarea = score of 100%; 25–29.9% = score of 80%; 20–24.9% = score of 60%; 15–19.9% = score of40%; 10–14.9% = score of 20%; 0–9.9% = score of 0%)

4

Integrated pestmanagement(IPM) (#7)

Herbicide use(7A)

Proportion of the cultivated area without herbicide (score of 100%), with herbicides accordingto an IPM approach (score of 70%), with herbicides without IPM approach (score of 0%)

6

Insecticide andfungicide use(7B)

Proportion of the cultivated area without insecticide and fungicide (score of 100%), withinsecticide and fungicide according to an IPM approach (score of 70%), with insecticide andfungicide without IPM approach (score of 0%)

8

Seed treatment(7C)

Proportion of annual crops area without seed treatment (insecticide and fungicide) 4

Refuges alongwithinsect-resistanttransgeniccrops (7D)

The appropriate refuges are implemented along with an insect-resistant transgenic crop whenrequired (yes = score of 100%; no = 0%)

3

Drying ofmaize (#8)

Ratio of the volume (liters) of propane gas used to dry maize by the total quantity (tons) ofmaize harvested (≤23.9 L/t = score of 100%; 24–27.9 L/t = score of 80%; 28–31.9 L/t = score of60%; 32–35.9 L/t = score of 40%; 36–39.9 L/t = score of 20%; ≥40 L/t = score of 0%)

5

Fertilizationmanage-ment

Phosphorusbalance (#9)

Proportion of the cultivated area where the phosphorus added to the soil does not exceed theneeds of the crop by more than 10 kg P2O5 ha−1

5

Nitrogenbalance (#10)

Proportion of the cultivated area where the nitrogen added to the soil does not exceed theneeds of the crop by more than 10 kg N ha−1

9

Split nitrogenapplications(#11)

Proportion of areas cultivated with maize, spring wheat or colza with split nitrogenapplications

8

Incorporationof manure intothe soil (#12)

Solid manure(12A)

Proportion of total amount of solid manure (≥15% DM) or solid waste fertilizer managedunder different categories (incorporated within 12 h = score of 100%; applied on a growingcrop and incorporated within 12–48 h = score of 80%; applied on a bare soil and incorporatedwithin 12–48 h = score of 60%; applied on a growing crop and non incorporated = score of 40%;applied on a bare soil and non incorporated = score of 0%)

7

Liquid manure(12B)

Proportion of total amount of liquid manure (<15% DM) managed under different categories(incorporated within 3 h = score of 100%; applied on a growing crop and incorporated within3–24 h = score of 80%; applied on a bare soil and incorporated within 3–24 h = score of 60%;applied on a growing crop and non incorporated = score of 40%; applied on a bare soil and nonincorporated = score of 0%)

9

Farmlandmanage-ment

Erosion insloping fields(#13)

Limitation of erosion in sloping fields with preventive practices (see decision tree, Fig. 2) 10

Riparian bufferstrip (#14)

Alongwatercourses(14A)

Respect of a riparian strip of 3 m width, without fertilization, without annual crops andwithout tillage, along all water courses (>2 m2 of flow area) (yes = score of 100%; no = score of0%)

10

Alongagriculturalditches (14B)

Respect of a riparian strip of 1 m width, without fertilization, without annual crops andwithout tillage, along all agricultural ditches (≤2 m2 of flow area) (yes = score of 100%;no = score of 0%)

8

Windbreaks(#15)

Ratio of the total length (m) of windbreaks on the farm by the cultivated area (ha)(>80 m ha−1 = score of 100%; 60–80 m ha−1 = score of 75%; 40–60 m ha−1 = score of 50%;20–40 m ha−1 = score of 25%; <20 m ha−1 = score of 0%)

9

On-farmwoodlot (#16)

Presence of on-farm wood lot of a minimum area of 5 hectares (yes = score of 100%; no = 0%) 8

Maximum score 177

682 M.-N. Thivierge et al. / Ecological In

wq

3

pTtfittiAt

e#e(c(

tcsst

Fttw

Fig. 2. Decision tree for indicator #13, “erosion in sloping fields”.

hich facilitates the aggregation of the answers to the severaluestions.

.3. Weight of the 16 indicators

Fig. 3 illustrates the results of the weighting as established by theanel of experts. The points awarded act as units of sustainability.he contribution of one indicator to the overall environmental sus-ainability is the sum of the contributions of this indicator to theour sub-objectives, each represented by a different color/texturen Fig. 3. The total weight of this set of indicators is 177. Consideringhat the objective is to provide a self-assessment and decision-aidool to farmers, the focus should be on the identification of thendicators needing improvements, rather than on the total score.ggregating indicators to a unique score leads to a loss of informa-

ion (Von Wirén-Lehr, 2001).Some indicators show a very large contribution to the overall

nvironmental sustainability of cash-crop farms, e.g. the indicator7 “integrated pest management” (21 points out of 177), while oth-rs show a lower contribution, e.g. indicator #8 “drying of maize”5 points out of 177) (Fig. 3). The weighting allows the farmer toompare the risks associated with different agricultural practicesRigby et al., 2001).

Fig. 3 also reveals which indicators contribute in a special wayo specific sub-objectives. “Crop diversity” (#6) is the indicator thatontributes the most to soil quality conservation, “riparian buffer

trip” (#14) to water quality, “incorporation of manure into theoil” (#12) to air quality, and “integrated pest management” (#7)o biodiversity conservation.

ig. 3. The 16 indicators, grouped into four components, and their contributiono environmental sustainability after weighting. The different colors/textures showhe contribution of the indicators to the four sub-objectives (biodiversity, air quality,ater quality, and soil quality).

dicators 45 (2014) 677–686

For many reasons, some authors chose not to allocate weight toindicators. However, as highlighted by Rey-Valette et al. (2008),the deliberate choice of not weighting indicators is the equiva-lent to agree that they all have an equal value, which is a form ofweighting. This tool is seeking an educational purpose with farm-ers. According to our research team, to claim that all the indicatorshave the same value in terms of sustainability would be worsethan to attempt allocating weights to them. The consistency of themethodology, coupled with the quality of experts convened, con-tributes to make the weighting as objective as possible. As statedby King et al. (2000), it is the rigor of the focus group that ensuresthe validity of the results. In this type of approach, it is decided toaccept the subjectivity associated with the decision-making pro-cess within the group (Roy, 1992, cited in Bockstaller et al., 1997).The weighting should not be read as the quantitative influence ofan agricultural practice on the ecosystem, but rather as a relativecontribution to sustainability based upon an understanding of howeach agricultural practice will affect the physical, chemical, and bio-logical processes, and then influence the entire system (Rigby et al.,2001).

3.4. Scores of the 31 farms

The farm score for a given indicator is calculated by multiplyingthe result of the farm by the total weight attributed to this indicator(Table 3). For example, for the indicator #8 “drying of maize”, a farmthat uses 30 liters of propane per ton of maize will have a score of3 points (the category 28–31.9 L/t is worth 60% of the total weightof 5 points).

The scores of the 31 farms are shown in Table 4, as well as thescores of the leading group as a reference value. This group includesthe 25% best-performing farms for the overall environmental sus-tainability assessment. Thus, the score of the leading group is theaverage result of those farms for each indicator.

The reference value allows the user to compare his score and toassess its value (Bockstaller et al., 2008; Piorr, 2003). According toHalberg (1999), the best way to interpret results at the farm level isto compare with other farms: farmers commonly compare them-selves to their peers (Gomez et al., 1996). When asked about this inthe end-user validation questionnaire, 15 out of 16 farmers appre-ciated the option to compare their score with the leading groupwhile all of them liked a comparison to the average of all farms.The authors believe that the use of a leading group can raise thestandard, while providing a reference value more stable than anaverage.

It is worth mentioning that the 31 participating farms in thestudy were chosen to depict the main cultivated areas within theprovince of Quebec, as well as the diversity of cropping prac-tices for cash-crop farms. The Agro-Environmental Advisory Clubssuggested farms that are members of their organization and, pre-sumably, the ones that tend to be interested in this kind of project.For this reason, the authors believe that most of the 31 farms were apriori more sustainable than the average cash-crop farm in Quebec.However, this test was not meant to establish the environmentalsustainability of the farms of the province of Quebec, but ratherto verify if the indicators fulfilled the selection criteria, and if theywere suitable for all cropping systems. Descriptive statistics arepresented in Table 4.

For some indicators, even the leading group gets a poor score,as with indicator #5, “cover crops”, where the leading group gets3.8 out of 10 while the average is 2.3. However, this is consistentwith the official data from the province of Quebec, where the areas

declared as green manure barely reach 6% of the cultivated lands(BPR, 2008). This example demonstrates that getting a score higherthan the leading group does not guarantee a sustainable practice(Bockstaller et al., 2008). Experts agreed that the cover-crop

M.-N. Thivierge et al. / Ecological Indicators 45 (2014) 677–686 683

Table 4Results of the testing of the 16 indicators on 31 farms across the province of Quebec.

Indicators Weight Scores of the 31 farms Scores of the leading group

Median Average Standard deviation Median Average

Soil organic matter content (#1) 12.0 12.0 10.6 ±2.76 12.0 11.9Soil phosphorus saturation (#2) 8.0 8.0 7.3 ±1.31 7.9 6.9Deep and surface drainage (#3) 10.0 8.0 7.5 ±3.35 10.0 9.0

Conservation tillage (#4) 9.0 4.8 5.1 ±2.87 4.7 4.6Cover crops (#5) 10.0 2.1 2.3 ±2.28 3.9 3.8Crop diversity (#6) 19.0 11.0 12.4 ±3.52 12.6 13.2Integrated pest management (#7) 21.0 16.4 16.3 ±3.39 19.2 18.7Drying of maize (#8) 5.0 5.0 3.7 ±1.77 5.0 4.4

Phosphorus balance (#9) 5.0 4.7 4.2 ±0.96 4.7 4.4Nitrogen balance (#10) 9.0 8.8 7.8 ±1.76 9.0 8.2Split nitrogen applications (#11) 8.0 8.0 6.6 ±2.83 8.0 8.0Incorporation of manure into the soil (#12) 16.0 11.8 11.0 ±4.06 14.4 13.9

Erosion in sloping fields (#13) 10.0 10.0 8.5 ±2.47 10.0 9.3Riparian buffer strip (#14) 18.0 18.0 15.4 ±5.35 18.0 18.0Windbreaks (#15) 9.0 4.5 4.4 ±3.18 6.8 6.5

pwiitacfr

aitRsattipb(

ctTtcftfib

3

tpittq

of each indicator is written besides its number.This type of chart is renowned for facilitating the comparison

of results (Bockstaller et al., 1997; Gomez et al., 1996; Vilain et al.,2008), for instance, between a farm and a leading group (Fig. 4).

On-farm woodlot (#16) 8.0 8.0

Total 177.0 130.0

ractice is quite applicable but simply not a custom for farmers,ho sometimes lack knowledge on how to integrate those crops

n their rotation. It therefore appeared appropriate to keep thisndicator as it is, so that farmers can realize its importance inhe sustainability of their farm. Similarly, an indicator for whichll the farms have high scores will also be preserved. This is thease for indicators #2 “soil phosphorus saturation” and #16 “on-arm woodlot”, for which we obtain the averages of 7.3 and 7.7espectively, out of 8.

As reported by Singh et al. (2012), some authors use bivari-te or multi-variate analyses to verify the correlation for pairs ofndicators, or between an indicator and the total score. A sensi-ivity analysis can also be performed (Andersen et al., 2013). Likeigby et al. (2001), we found those verifications to be unneces-ary. Indeed, each indicator was deemed relevant by the expertsnd thus carries a message that needs to be delivered in an authen-ic way to farmers. Although a special effort was made to reducehe redundancy of information in the selection and the weighting ofndicators, it is impossible to avoid it completely in a system as com-lex as a farm. In the case of an educational tool, the redundancyetween some indicators may even be conscious and voluntaryVilain et al., 2008).

During the end-use validation, farmers were asked if their scoresorresponded to the idea they had of the environmental situation ofheir own farm. Thirteen out of 15 farmers agreed with their scores.en farmers out of 13 (some farmers did not answer this ques-ion) claimed that if this set of indicators was available, they wouldonsider using it regularly (every 2 or 3 years). These feedbacksrom farmers are crucial, since “a good indicator is an indicatorhat is used” (Rey-Valette et al., 2008). In addition, 13 out of 13armers consider that this tool could help all cash-crop farmers tomprove their agricultural practices in order to move towards aetter environmental sustainability.

.5. Visual presentation

As this set of indicators is intended to be a decision-aid tool,he results should be easily understood by farmers. Charts allowresenting the score for each indicator without aggregating them

nto an overall score (Rigby et al., 2001). The choice of the chartype “is crucial in terms of communication and use” and must beested (Rey-Valette et al., 2008). Hence, in the end-use validationuestionnaire sent to the 31 farmers, different charts (radar chart,

7.7 ±1.44 8.0 8.0

130.8 ±15.36 147.9 148.7

conventional bar chart, and a new bar chart) were presented to thefarmers. For a given graph, the scores of a fictive case-study farmwere shown. Farmers were asked to identify the three indicatorsthat this fictive farm could use to most improve the farm’s environ-mental sustainability. The results for the three types of charts arepresented below, with emphasis on the comparison between theradar chart (Section 3.5.1) and the new bar chart (Section 3.5.2).

3.5.1. Radar chartFig. 4 shows the scores of one fictive farm with a radar chart, the

most common way to express indicators (Bockstaller et al., 1997;Gomez et al., 1996; Rigby et al., 2001; Von Wirén-Lehr, 2001). Eachaxis represents an indicator. The results are expressed in percent-ages. For each indicator, the center is the lowest score (0%), whilethe outer ring corresponds to the highest score (100%). The weight

Fig. 4. Scores of the 16 indicators for a fictive case-study farm (solid line) andcomparison with the leading group (dotted line) using a radar chart.

684 M.-N. Thivierge et al. / Ecological Indicators 45 (2014) 677–686

Table 5Comparison of the three charts proposed to the farmers in the end-use validation questionnaire to illustrate the scores of fictive case-study farms.

Number of answers for this question Radar chart Conventional bar chart New bar chart

Number of farmers able toidentify two out of threeindicators to improve

16 5 3 10

Is this chart easy to 16 12 10 10

I(#(iititciog1tmotttistafcbo

3

dEw

Ft

understand?Which chart do you prefer?

(select only one)15

ndeed, a visual comparison of two values for the same indicatoron the same axis) is convenient. For example, for the indicator5, the case-study farm is more sustainable than the leading group

Fig. 4). Nevertheless, a concern raised by Bockstaller et al. (1997)s that the radar chart makes it extremely difficult to compare onendicator to another (on different axes), and therefore, to identifyhe strengths or weaknesses of the farm. In fact, the weight of eachndicator is not illustrated in this chart, although it is written besidehe axis. For example, in Fig. 4, the first indicator that the farmould benefit from improving seems to be #8, because its values very close to the center of the chart. However, this indicator isnly worth five points. In terms of sustainability, the farm wouldain much more by improving indicator #14, with a possibility of3 points (units of sustainability). The necessity to have in mindhe respective weight of each indicator complicates the decision-

aking process. In the end-use validation questionnaire, only fiveut of 16 farmers (Table 5) correctly identified at least two out ofhe three indicators that would most benefit sustainability. Misin-erpretations related to the comparison of the indicators are suchhat this chart does not show the full potential of a set of weightedndicators used as a decision-aid tool. Although this type of charteems to be the least understood by cash-crop farmers, it is alsohe one that the largest number of them (12 out of 16) identifieds being easy to understand (Table 5). This may be because manyarmers in the province of Quebec are familiar with this type ofhart through financial or management counseling services. It cane noted that with indicators that are not weighted, this problemf comparison does not occur.

.5.2. New bar chart

To help reveal all the potential of a weighted set of indicators as a

ecision-aid tool, another type of chart has been designed (Fig. 5).ach horizontal bar represents one indicator, and the respectiveeights of the indicators are illustrated by the length of each bar.

ig. 5. Scores of the 16 indicators for a fictive case-study farm and comparison withhe leading group using the new bar chart.

6 6 3

Again, the points awarded to indicators act as units of sustainability.The central axis separates the sustainability units already acquired(colored/textured portion of the bar on the right side of the chart)from those that remained to be earned (white portion on the leftside). The indicators that contribute the most to current sustaina-bility of the farm are those with the longest colored/textured barson the right side of the chart, while the ones that the farm wouldbenefit the most to improve are those with the longest white barson the left side. With the improvement of agricultural practices overtime, longer portions of the bars will be found on the right side ofthe chart. The dots represent the scores of the leading group.

In comparison with the radar chart, it is then more obvious that,from an environmental standpoint, the case-study farm would ben-efit more by improving indicator #14 than #8 (Fig. 5). Anotheradvantage of this new bar chart, as opposed to the radar chart,is that it allows the comparison of an endless number of indica-tors without compromising clarity. During the end-use validation,10 out of 16 farmers (Table 5) could correctly identify at least twoout of the three indicators that would most improve environmen-tal sustainability. In addition, 10 out of 14 farmers believe the newbar chart is easy to understand, although it was selected by onlythree farmers as their favorite (Table 5). The authors believe thatthe demonstrated greater understanding allowed the farmers todraw accurate conclusions from their scores with the new bar chart,thus justifying its choice as a self-assessment and decision-aidtool.

3.6. Is self-assessment possible

Despite the fact that the results obtained with the new bar chartare better than with other charts, it is still disappointing that only10 out of 16 farmers reached accurate conclusions (Table 5). First,these results need to be validated with more farmers. Secondly,this raises a concern with the objective of a self-assessment tool.According to Bockstaller et al. (1997) and Von Wirén-Lehr (2001),the scores must be supported by advice. The end-use validationquestionnaire showed that 14 out of 16 farmers would like writtenrecommendations with the charts. These results are consistent withthose of Halberg (1999) and Meul et al. (2009), in which farmersasked for guidance in interpreting the scores.

The authors think that the presence of an advisor for the inter-pretation of the scores could enhance the educational purpose ofsuch a tool. When questioned about this, 12 out of 16 farmerssaid they would appreciate this presence. A farm advisor awareof the reality of the farm may be able to help the farmer to inter-pret its scores and to provide advices and additional information(Meul et al., 2008, 2009) taking into account other factors regardingeconomic and social aspects of sustainability.

Several of the participating farmers mentioned not being com-fortable with computer technology. A strictly online tool wouldrestrict the access to many farmers. Finally, 13 out of 15 farmers

wished not to spend more than 2 h for a sustainability assessment.This corroborates the findings of Rey-Valette et al. (2008). The pres-ence of an advisor could help with the technology as well as withthe duration.

ical In

3

3

cfiochiep

3

oedetta6twrcs

4

dekfacttdsfs

assitd

A

toiRf(

R

A

M.-N. Thivierge et al. / Ecolog

.7. Weaknesses

.7.1. Representativity of the first panel of expertsBecause this study was meant to adapt existing dairy farm indi-

ators to cash-crop farms, only 10 experts were convened for therst focus group. The withdrawal of two of them brought the sizef the discussion panel to eight experts, including two farmers spe-ialized in cash crops from farms certified as organic. Although thisad no influence on the weighting, since farmers did not participate

n that step, the discussions of the first focus group could have beennriched by the presence of one or two farmers from conventionalroduction.

.7.2. Evaluation of indicators regarding the selection criteriaMitchell et al. (1995) and Piorr (2003) mentioned the difficulty

f meeting all the criteria for each indicator. According to Dalalt al. (1999), an evaluation of the indicators in this regard can beone through discussions or constructive debates held between thexperts. In the present study, the selection criteria listed in Sec-ion 2.1 were explained to the experts at the first focus group andhen discussed more deeply after tests on farms (for criteria 1, 3nd 4 of Section 2.1) and end-use validation (for criteria 2, 5 and). However, there was no posteriori evaluation of the criteria. Ahreshold could have been established, as in Bélanger et al. (2012),here indicators that met less than four out of six criteria were

emoved. At least, as highlighted by Mitchell et al. (1995), the indi-ators which do not respond, or only partially, to selection criteriahould be identified as such and revised periodically.

. Conclusions

It is essential to advocate an agricultural development thatoes not harm the rights of future generations to grow (Vilaint al., 2008). The present set of indicators is based upon currentnowledge of sustainability and cropping practices on cash-croparms in the province of Quebec. By the means of focus groupsnd interviews with farmers, 16 indicators have been adapted toash-crop production and weighted according to their contributiono four sub-objectives of environmental sustainability (conserva-ion of soil, water, air, and biodiversity). A new type of chart wasesigned to help farmers interpret their sustainability assessmentcores. Finally, an end-use validation questionnaire was used forarmer feedback. Those results need to be validated at a broadercale.

Precision agriculture and energetic self-sufficiency of farms aremong the elements that should be further explored in subsequenttudies. Moreover, special attention must be paid to score transmis-ion to farmers; it should not be forgotten that a set of indicatorss a communicative tool, and that the participation of farmers inhe construction and validation of sustainability indicators is fun-amental.

cknowledgements

The authors would like to thank all experts who were involved inhis project, as well as all farmers who allowed to test the indicatorsn their farm and who shared their feedbacks. The fruitful discuss-ons with Hayo van der Werf and Joel Aubin, from UMR SAS, INRA,ennes, France, are warmly acknowledged. The authors are grate-

ul to the Fonds de recherche du Québec – Nature et technologiesFQRNT) for financing this research.

eferences

GECO, 2006. Profil de consommation d’énergie à la ferme dans six des principauxsecteurs de production agricole du Québec. Rapport no 1.

dicators 45 (2014) 677–686 685

Agricultural Operations Regulation (AOR), 2002. Environment Quality Act, Q-2,r.26. Minister of Sustainable Development, Environment, Wildlife and Parks,www.mddep.gouv.qc.ca/publications/index.htm (accessed 18.04.14).

Andersen, P.S., Vejre, H., Dalgaard, T., Brandt, J., 2013. An indicator-based methodfor quantifying farm multifunctionality. Ecol. Indic. 25, 166–179.

Bélanger, V., Vanasse, A., Parent, D., Allard, G., Pellerin, D., 2012. Development of agri-environmental indicators to assess dairy farm sustainability in Quebec, EasternCanada. Ecol. Indic. 23, 421–430.

Bockstaller, C., Girardin, P., 2003. How to validate environmental indicators. Agric.Syst. 76, 639–653.

Bockstaller, C., Girardin, P., Van der Werf, H.M.G., 1997. Use of agro-ecological indi-cators for the evaluation of farming systems. Eur. J. Agron. 7, 261–270.

Bockstaller, C., Guichard, L., Makowski, D., Aveline, A., Girardin, P., Plantureux, S.,2008. Agri-environmental indicators to assess cropping and farming systems. Areview. Agron. Sustain. Dev. 28, 139–149.

BPR, 2008. Suivi 2007 du Portrait agroenvironnemental des fermes du Québec.In: Rapport présenté au ministère de l’Agriculture, des Pêcheries et del’Alimentation du Québec, à l’Union des producteurs agricoles et à Agricultureet Agroalimentaire Canada, Ottawa.

Commission of the European Communities, 1999. Directions Toward SustainableAgriculture. Communication from the Commission, no 22. European Commis-sion, Brussels.

Dalal, R.C., Lawrence, P., Walker, J., Shaw, R.J., Lawrence, G., Yule, D., Doughton, J.A.,Bourne, A., Duivenvoorden, L., Choy, S., Moloney, D., Turner, L., King, C., Dale, A.,1999. A framework to monitor sustainability in the grains industry. Aust. J. Exp.Agric. 39, 605–620.

Delbecq, A.L., Van de Ven, A.H., Gustafson, D.H., 1975. Group Techniques for ProgramPlanning: A Guide to Nominal Group and Delphi Processes. Scott, Foresman,Glenview, IL.

Fraser, E.D.G., Dougill, A.J., Mabee, W.E., Reed, M., McAlpine, P., 2006. Bottom upand top down: analysis of participatory processes for sustainability indicatoridentification as a pathway to community empowerment and sustainable envi-ronmental management. J. Environ. Manage. 78, 114–127.

Girardin, P., Bockstaller, C., Van der Werf, H.M.G., 1999. Indicators: tools to eval-uate the environmental impacts of farming systems. J. Sustain. Agric. 13,5–21.

Girardin, P., Bockstaller, C., Van der Werf, H.M.G., 2000. Assessment of potentialimpacts of agricultural practices on the environment: the AGRO*ECO method.Environ. Impact Assess. Rev. 20, 227–239.

Gomez, A.A., Swete Kelly, D.E., Syers, J.K., Coughlan, K.J., 1996. Measuring sustaina-bility of agricultural systems at the farm level. Methods for assessing soil quality.Soil Sci. Soc. Am. J. 49, 401–410.

Goodland, R., 1995. The concept of environmental sustainability. Annu. Rev. Ecol.Syst. 26, 1–24.

Gras, R., Benoit, M., Deffontaines, J.P., Duru, M., Lafarge, M., Langlet, A., Osty, P.L.,1989. Le fait technique en agronomie. In: Activité agricole, Concepts et Méthodesd’Étude. Institut National de Recherche Agronomique, Paris.

Halberg, N., 1999. Indicators of resource use and environmental impact for use in adecision aid for Danish livestock farmers. Agric. Ecosyst. Environ. 76, 17–30.

Häni, F., Braga, F., Stämpfli, A., Keller, T., Fischer, M., Porsche, H., 2003. RISE, a toolfor holistic sustainability assessment at the farm level. Int. Food AgribusinessManage. Rev. 6, 78–90.

Hasson, F., Keeney, S., 2011. Enhancing rigour in the Delphi technique research.Technol. Forecast Soc. 78, 1695–1704.

Heink, U., Kowarik, I., 2010. What are indicators? On the definition of indicators inecology and environmental planning. Ecol. Indic. 10, 584–593.

Institut de la statistique du Québec (ISQ), Ministère de l’Agriculture, desPêcheries et de l’Alimentation du Québec (MAPAQ), 2013. Profil secto-riel de l’industrie bioalimentaire du Québec 2012, www.stat.gouv.qc.ca/statistiques/agriculture/index an.html (accessed 18.04.14).

Keeney, S., Hasson, F., McKenna, H.P., 2001. A critical review of the Delphi techniqueas a research methodology for nursing. Int. J. Nur. Stud. 38, 195–200.

King, C., Gunton, J., Freebairn, D., Coutts, J., Webb, I., 2000. The sustainability indicatorindustry: where to from here? A focus group study to explore the potential offarmer participation in the development of indicators. Aust. J. Exp. Agric. 40,631–642.

La Financière agricole du Québec, 2010. Assurance stabilisation. Coût de production,revenu stabilisé et caractéristiques techniques, 2010, www.fadq.qc.ca (accessed10.06.13).

Lefebvre, A., Eilers, W., Chunn, B., 2005. Environmental Sustainability of CanadianAgriculture: Agri-Environmental Indicator Report Series-Report 2. Agricultureand Agri-Food Canada, Ottawa.

Linstone, A., Turoff, M., 2002. The Delphi Method: Techniques and Applications. NJIT,New Jersey.

Meul, M., Van Passel, S., Nevens, F., Dessein, J., Rogge, E., Mulier, A., Van Hauwer-meiren, A., 2008. MOTIFS: a monitoring tool for integrated farm sustainability.Agron. Sustain. Dev. 28, 321–332.

Meul, M., Nevens, F., Reheul, D., 2009. Validating sustainability indicators: focus onecological aspects of Flemish dairy farms. Ecol. Indic. 9, 284–295.

Michaud, A., Martel, S., Seydoux, S., Beaudin, I., 2006. Évaluation des effets combinésdes principales pratiques de gestion bénéfiques (PGB). In: Initiative Nationale

d’Élaboration de Normes Agroenvironnementales. Institut de recherche et dedéveloppement en agroenvironnement, Québec.

Mitchell, G., May, A., McDonald, A., 1995. PICABUE: a methodological framework forthe development of indicators of sustainable development. Int. J. Sustain. Dev.World 2, 104–123.

6 ical In

M

P

P

P

R

R

S

S

86 M.-N. Thivierge et al. / Ecolog

orris, C. (Ed.), 1992. Academic Press Dictionary of Science and Technology. Aca-demic Press, California.

acini, C., Wossink, A., Giesen, G., Vazzana, C., Huirne, R., 2003. Evaluation of sus-tainability of organic, integrated and conventional farming systems: a farm andfield-scale analysis. Agric. Ecosyst. Environ. 95, 273–288.

ayraudeau, S., Van der Werf, H.M.G., 2005. Environmental impact assessmentfor a farming region: a review of methods. Agric. Ecosyst. Environ. 107,1–19.

iorr, H.-P., 2003. Environmental policy, agri-environmental indicators and land-scape indicators. Agric. Ecosyst. Environ. 98, 17–33.

ey-Valette, H., Clément, O., Aubin, J., Mathé, S., Chia, E., Legendre, M., Caruso, D.,Mikolasek, O., Blancheton, J.-P., Slembrouck, J., Baruthio, A., René, F., Levang,P., Morissens, P., Lazard, J., 2008. Guide to the Co-construction of SustainableDevelopment Indicators in Aquaculture. EVAD, Montpellier.

igby, D., Woodhouse, P., Young, T., Burton, M., 2001. Constructing a farm levelindicator of sustainable agricultural practice. Ecol. Econ. 39, 463–478.

ingh, R.K., Murty, H.R., Gupta, S.K., Dikshit, A.K., 2012. An overview of sustainabilityassessment methodologies. Ecol. Indic. 15, 281–299.

attler, C., Nagel, U.J., Werner, A., Zander, P., 2010. Integrated assessment of agricul-tural production practices to enhance sustainable development in agriculturallandscapes. Ecol. Indic. 10, 49–61.

dicators 45 (2014) 677–686

Statistics Canada, 2012. Statistics on revenues and expenses of farms 2010. Catalogno. 21-208-X. Minister of Industry, Ottawa.

Taylor, D.C., Mohamed, Z.A., Shamsudin, M.N., Mohayidin, M.G., Chiew, E.F.C., 1993.Creating a farmer sustainability index: a Malaysian case study. Am. J. AlternativeAgric. 8, 175–184.

Van Cauwenbergh, N., Biala, K., Bielders, C., Brouckaert, V., Franchois, L., Cidad,V.G., Hermy, M., Mathijs, E., Muys, B., Reijnders, J., Sauvenier, X., Valckx, J.,Vanclooster, M., Van der Veken, B., Wauters, E., Peeters, A., 2007. SAFE – a hierar-chical framework for assessing the sustainability of agricultural systems. Agric.Ecosyst. Environ. 120, 229–242.

Van der Werf, H.M.G., Tzilivakis, J., Lewis, K., Basset-Mens, C., 2007. Environmentalimpacts of farm scenarios according to five assessment methods. Agric. Ecosyst.Environ. 118, 327–338.

Vernon, W., 2009. The Delphi technique: a review. Int. J. Ther. Rehabil. 16 (2),69–76.

Vilain, L., Boisset, K., Girardin, P., Guillaumin, A., Mouchet, C., Viaux, P., Zahm, F.,

2008. La méthode IDEA, indicateurs de durabilité des exploitations agricoles, 3e

édn. Guide d’utilisation. Educagri Éditions, Dijon.Von Wirén-Lehr, S., 2001. Sustainability in agriculture-an evaluation of principal

goal-oriented concepts to close the gap between theory and practice. Agric.Ecosyst. Environ. 84, 115–129.