Revisões Sistemáticas da Literatura&

Construção de Referenciais Teóricos

Carlo BelliniPPGA/UFPB

carlo.bellini@pq.cnpq.br

ReferênciasKitchenham, B. Procedures for performing systematic reviews. Technical Report TR/SE-0401. Keele:

Keele University, 2004.Kitchenham, B. What’s up with software metrics? A preliminary mapping study. Journal of Systems

& Software, 83, 2010.Kitchenham, B.; et al. Systematic literature reviews in software engineering: A tertiary study.

Information & Software Technology, 52, 2010.Kitchenham, B.; et al. Refining the systematic literature review process: Two participant-observer

case studies. Empirical Software Engineering, 15, 2010.Kitchenham, B.; et al. Using mapping studies as the basis for further research: A participant-

observer case study. Information & Software Technology, 53, 2011.---

Bellini, C.G.P.; Pereira, R.C.F.; Becker, J.L. Customer team effectiveness through people traits in information systems development: A compilation of theoretical measures. International Journal of Human Capital & Information Technology Professionals, 3, 2012.

Bellini, C.G.P.; Pereira, R.C.F.; Becker, J.L. Measurement in software engineering: From the roadmap to the crossroads. International Journal of Software Engineering & Knowledge Engineering, 18, 2008.

Bellini, C.G.P.; Isoni Filho, M.M. Limitações digitais: Evidências teóricas preliminares. Análise, 23, 2012.

SILVEIRA, G.; BORENSTEIN, D.; FLOGLIATTO, F.Mass customization: Literature review and research directions.International Journal of Production Economics, 72(1), 2001, 1-13.

Web of Science : 160 citaçõesScopus : 321 citações

• 2005: ensaio teórico de tese

• 2006: capítulo da tese

• 2006: saga IJSEKE SLR Kitchenham

• 2007: Moreno et al. (Enanpad) JGITM 2009

• 2011: dissertações e teses incluem SLR

Aprendizado

• Systematic literature reviews (SLRs) are recommended for aggregating empirical studies and communicating to the professional community.

• SLR is an evidence-based method to produce and reproduce literature compilations of empirical findings, since it is assumed to be unbiased, auditable and repeatable.

• SLR asks a fairly specific question, while a mapping study asks a more general question.

• SLRs and mapping studies are called secondary studies because they aim to aggregate other studies which are referred to as primary studies.

Kitchenham

• Mapping studies tend to stop at categorizing identified papers, while SLRs attempt to aggregate results into a coherent body of knowledge that allows researchers and practitioners to answer specific questions.

• Diferentemente de meta-análises, que têm algum caráter de pesquisa autocontida, SLRs dificilmente representariam uma pesquisa completa. Servem para pesquisas intermediárias.

Kitchenham

• Automated search: A search of digital libraries and electronic indexing systems using search strings aimed at finding candidate primary studies.

• Broad search: A search intended to identify as many candidate primary studies as possible either by searching a large number of journals and conference proceedings manually and/or by automated searches of a several different digital sources including digital libraries and electronic indexing systems. Such a search may include searching references in primary studies and direct appeal to experts in the field.

• Manual search: A search carried out on specific journals and conference proceedings by a researcher who identifies whether the paper is a candidate primary study.

Kitchenham

• Restricted/targeted search: A search that targets a specific set of journals and/or conference proceedings on the assumption that these are the most important and or best quality sources of candidate primary studies.

• Grey literature: Works that have not been published in a source where there is a full peer review process, including technical reports, PhD and masters theses, workshop papers and book chapters.

• Meta-analysis: A SLR where the outcomes of primary studies are aggregated quantitatively.

• Tertiary study: A study that is based on an analysis of secondary study results. In other words, a form of secondary study where the primary studies are in fact secondary studies.

Kitchenham

• Publication bias: The problem that journals and conferences are more likely to accept for publications studies that show a positive effect of some method/procedure than papers that show no effect.

• Quality criteria: A set of concepts (usually in the form of questions) that are used to assess the quality of a primary study.

• Quality data: The answers to the quality criteria for each primary study.

• Quality score: After assigning numerical values to individual quality questions, the total score for a primary study is the sum of the individual numerical values.

Kitchenham

• Etapas metodológicas (Breivold et al., 2011):• contextualizar a revisão• estabelecer critérios de inclusão e exclusão de estudos• procurar publicações relevantes (estudos primários em fontes

primárias)• avaliar a qualidade dos estudos primários• extração e síntese de dados: top-down ou bottom-up

• Nem todas as etapas são relatadas por completo em pesquisas.

• Há casos em que nem todas as etapas precisam ou devem ser descritas, por exemplo quando envolvem informações irrelevantes ou que podem gerar sobrecarga de informações (Bellini et al., 2012).

Método

• SLR é baseada em bom-senso e decisões ad hoc.

• Expressões-chave na busca são um risco, pois

• estabelecem grafias específicas• em bases de dados grandes, resultam fontes de variável

qualidade.

• O que considerar para a seleção dos estudos primários: título, resumo, palavras-chave?

Método

Kitchenham (2010)

• Bellini et al. (2008) perform a more systematic search process.

• They did not explicitly discuss the results of their search and have a narrative style aggregation process that does not show a clear link between individual papers and conclusions.

• They do not confirm how many primary studies contribute to their study but have a total of 122 references.

Bellini et al. (2008)

• We started by searching for the areas of interest related to measurement in software engineering, for this relying mostly on the Guide to the Software Engineering Body of Knowledge.

• We claimed that discussing software measurement should address four key subjects: measurement theory, methods to collect and analyze core measures, the concept of software metrics, and how to identify metrics.

• From this initial set of concepts, leading publications related to such areas were selected (journals, books, and technical reports or guides; conference proceedings were not included in the search, due to papers wherein being probably still under construction) based on the opinion of experts and on known rankings by the academic community.

Bellini et al. (2008)

Bellini et al. (2008)

• The next step was to search the primary studies for works dealing with the concept areas previously identified — whenever applicable, the search started in 1990.

• Main references within each publication were occasionally researched, as well as other publications of the authors whose works were reviewed.

• Aided by electronic search engines, we looked for specific expressions within the articles’ titles.

• A complete search in all articles was also done in order to find similar expressions in the titles, as well as other concepts not included in the original search — that served merely to assemble a minimum set of relevant primary sources.

Bellini et al. (2008)

Bellini et al. (2008)

• Two of the authors were assigned to reading the set of articles after relevance was confirmed from their abstracts and the conclusions.

• After the selection of the primary sources, we followed a bottom-up approach to content analysis for building the categories of interest: the set of categories was developed as the study unfolded from the primary sources, so we devised the whole picture of software measurement only after concluding the readings.

Bellini et al. (2008)

• Inclusion criteria: the review covered all articles published between 2000 and 2006 in five major journals that explicitly mentioned people or software development issues in the articles’ titles. The journals included in the original search were MIS Quarterly, Journal of the AIS, Information Systems Research, Information Systems Journal, and Management Science, all of them consistently ranking among the top journals on the subject of this research.

• There was no exclusion criteria for the papers in the original search, since we assumed that the sources were very reliable.

Bellini et al. (2012)

• Contributions published in other periods and in other sources were also included in the search whenever a seminal work was found in the references of articles included in the original search – for this very reason, exclusion criteria were also not applicable here.

• It was no problem to include sources from different periods in the review, since we were searching for measures available in the most cited IT literature, with no interest in making statistical inferences from the set of primary sources. In other words, we did not perform a meta-analytical research, but a systematic review of the literature.

• Quality assessment of the primary studies was not necessary, since we assumed that the five journals included in the search were very reliable information sources.

Bellini et al. (2012)

• We did not keep track of the number of studies included in the full search. We assumed that if a single work dealing with personal traits was found in any of the selected publications, it was per se an important source for our measurement framework. In other words, it was not important to know how many works mentioned a particular trait, but the trait itself.

• The only possible exclusion criteria would be if a primary study rejected a specific trait found in another primary study – what was not the case in our review.

Bellini et al. (2012)

Data extraction and synthesis:

• All manuscripts that fit the inclusion criteria were read in full by one of the authors. Although this may have introduced some bias, having someone with a general view of the full dataset is beneficial.

• The other two authors validated the extraction of personal traits from the primary studies by means of conceptual validation based on their academic or industry experience.

• Data were codified into an electronic spreadsheet according to five categories of interest: the full reference of the primary source, the name of the original measure (personal trait) as mentioned in the source, the definition of the measure as given in the source, the underlying theory (if applicable), and an arbitrary code to identify each measure.

Bellini et al. (2012)

Bellini & Isoni Filho (2012)

Seleção das fontes primárias (periódicos):

• Orientação geral: TICs, ciência da informação, sociologia

• Periódicos #1, #2, #3 e #4 foram os mais citados nas referências dos artigos de base para o modelo tridimensional de Bellini et al. (2010).

• Periódicos #1, #4, #5 pertencem aos primeiros lugares do ranking da sociedade internacional de tecnologia da informação (AIS).

• Periódicos #2, #3, #7, #8, #9, #10 e #11 apresentam especificidade temática.

• Periódico #6 apresentou edição especial, em 2006, abordando estudos específicos sobre formas de limitações digitais (conforme análise preliminar dos autores).

• Período considerado: 2006 a 2011.

Bellini & Isoni Filho (2012)

• Seleção por palavras-chave em mecanismos de busca do website de cada periódico: “divide”, “exclusion”, “inclusion”, “inequality”, “equality”, “limitation”, “adoption”, “acceptance”, “accessibility”, “use”, “usage”, “usability”, “usable”, e “resistance”.

• Tal estratégia produziria resultados apenas parciais, dado que conceitos importantes (como “cognition”, “behavior” e suas variantes) não foram considerados. Assim, uma segunda estratégia considerou título, resumo e palavras-chaves de todos os artigos publicados entre 2006 e 2011 nas fontes primárias.

• Gerou-se uma lista de artigos potenciais, dos quais seriam selecionados os três mais recentes publicados em cada periódico.

• A lista final de estudos primários resultou em 31 artigos, pois dois periódicos proveram apenas dois artigos cada um.

Bellini & Isoni Filho (2012)

• Dois artigos publicados em The Information Society fogem à regra, já que havia outros mais recentes. Incluíram-se esses dois artigos devido à sua popularidade, pois foram referenciados em estudos subsequentes publicados no mesmo periódico e em outros estudos aqui considerados, de modo que a contribuição daqueles mais antigos foi estabelecer a base para os demais.

• Um dos autores efetuou leitura completa de cada artigo.

• A construção de categorias seguiu uma perspectiva top-down, dado que foram elaboradas previamente pelos autores: forma de limitação digital, escopo, método, extensão, contexto, implicação, causas e soluções.

• A síntese dos dados relacionou diretamente cada artigo a conclusões específicas.

Bellini & Isoni Filho (2012)

Revisões Sistemáticas da Literatura&

Construção de Referenciais Teóricos

Carlo BelliniPPGA/UFPB

carlo.bellini@pq.cnpq.br