The ontological-engineering approach (see Section \ref{sec:}) adopted in this dissertation highlights the importance of evaluation phase to demonstrate the trustworthiness of contributions. In this sense, it is necessary to perform evaluations of ontology OntoGaCLeS and the proposed intelligent theory-aware system for instructional des

According to \citet{mizoguchi2005seiji-thesis}, the better ontology is the more ontological. By ontological, he refers how close to the fundamental conceptualization of knowledge can be explained and how the essential properties of concepts are explicitly represented.

\subsection{Evaluation Methodology}

The evaluation of ISIDGaCLeS (an Intelligent System for Instructional Design of Gamified Collaborative Learning Scenarios) (contributions) will be supported by the CSCL-ERM (Computer Supported Collaborative Learning Evaluand-oriented Responsive Evaluation Model), which is a framework indicated for the evaluation of a wide range of CSCL systems, such as CSCL programs, innovations, instructional resources, teaching strategies, tools, and institutional evaluations. This framework allows us to formalize multiple experiments that assess an evaluand. In this dissertation, the evaluand is the support provided by ISIDGaCLeS to gamify collaborative learning scenarios. This support steams from ontology OntoGaCLeS and cover different authoring activities of instructional design process of gamified CL scenarios, and so, several methods and data source are proposed to select and obtain data and evidences about the compliance to this support during the multi-case study proposed as experiments for this research work.

\subsubsection{Evaluation Framework and Multi-case Studies}

As said before, the evaluation framework CSCL-EREM is focused on an evaluand (what is evaluated), and it allows to define formal experiments whose purpose is to assess the evaluand. This framework was chosen to carry out the evaluation of this dissertation, because the proposed ontological engineering approach is a solution intended to support the systematic instructional design of well-thought-out gamified CL scenarios, in which the learners are motivated and engaged in these scenarios through gamification. In this case, the evaluand is this support to gamify CL scenarios.

Figure \ref{fig:cscl-erem-components} shows a generic representation of CSCL-EREM framework, in which there are a set of facets (perspective, ground, and method). The perspective covers the main goal pursued, and other significant open questions. The ground describes the information about the context and the participants gathered. The method indicates the data gathering techniques and documents that support the conclusions.

\begin{figure}[thbp]\begin{center}% \includegraphics[width=0.85\textwidth]{cscl-erem-components.png}\caption{CSCL-EREM components. Figure taken from \citep{jorrin2009the}}\label{fig:cscl-erem-components}\end{center}\end{figure}

Depending on the particular kind of evaluand, one of the four courses (itineraries) proposed in the CSCL-EREM framework could be used to evaluate a CSCL system. The first course is the evaluation of CSCL programs, innovations, and courses. The second course is the evaluation of CSCL tools. The third course is the evaluation of teaching strategies/learning resources to promote collaboration. Finally, the fourth course is the evaluation of CSCL projects. In this research work, the second and third courses are the adequate itinerary to perform the evaluations of contributions. In order to present and organize the data obtained during the evaluations, the graphical representations and recommendations proposed in the CSCL-EREM framework will also be used.

To assess the evaluand, multiple experiments are defined using the CSCL-EREM framework. These experiments as multicase studies formalize and involve real end-users, like instructional designers, students, and also experts and practitioners of gamification as knowledge authors. The use of different kinds of actors in the experiments is a recurrently practice applied in the evaluation of different CSCL systems \citep{isotani2010, hernandez2007webcollage, bote2008gridcole, GLUE-ref}. The need of involving human will entail an additional difficult for this dissertation, because the opportunities to organize this kind of experiments are normally quite limited.

For this research work, we propose four main experiments and three different collaborative learning situations (see Section \ref{sec:} to obtain more details about these experiments and situations). Two secondary experiments without real practitioners will also be performed in order to replicate the instantiation processes of four main experiments with and without ISIDGaCLeS. Apart from providing evaluation results, These experiments will served to receive useful feedback from educators and students; this feedback was employed to improve the design and development of the proposed architecture. This idea perfectly compatible with the research methodology \citep{gla95}, which is iteratively developed, as it is the implementation of ISIDGaCLeS. The use of proposed experiments is not without problems, insofar as the replicability of these experiments is rather limited, and the results may be biased by participants. However, to minimize these problems, we propose multicase studies as evaluation methods rather than single case study. Finally, several methods and data gathering techniques are combined to collect and analyze the data obtained from the experiments, because the support requirements stakeholders to ensure that requirements satisfy the necessary support for instructional design of gamified CL scenarios, and by extension the evaluand, cover the technological, psychological, and educational fields.

\subsubsection{Evaluation Methods and Data Sources}

To assess the usefulness of ISIDGaCLeS to design well-thought-out gamified CL scenarios, different methods and data sources will be employed focusing on the instructional design process using ISIDGaCLeS rather than students evaluation using the obtained gamified CL scenarios by this tool. Table \ref{tab:stakeholders-requirements-tool} summarized the stakeholders requirements, evaluation methods, and data sources employed to assess each of them.

\begin{table}[thbp] \scriptsize\begin{center}\begin{tabular}{|p{2cm}|c|p{6cm}|p{2cm}|p{2cm}|} \hlineStakeholder & Tag & Requirements (ISIDGaCLeS should) & Evaluation methods & Data sources \\ \hline \hline

\multirow{4}{2cm}{Instructional designers}& RE1 & assist novice and expert users in the instructional design providing structured guidance with recommendations based on best practices and theories of gamification & & \\ \cline{2-5}& RE2 & give good recommendations during the instructional design process of gamified CL scenarios & & \\ \cline{2-5}& RE3 & include proper theoretical justifications in the recommendations that helped designers to identify essential characteristics that compose well-thought-out gamified CL scenarios & & \\ \cline{2-5}\cline{2-5}& RE4 & able to interchange the obtained gamified CL scenarios easily & & \\ \hline

Experts and practitioners of gamification (knowledge authors)& RE5 & able to increase the number of conceptual models and frameworks to gamify CL scenarios & & \\ \hline

\multirow{2}{2cm}{Instructional designers}& RE6 & enable the instructional design of gamified CL scenarios with an attainable effort & & \\ \hline& RE7 & enable the gamification of CL scenarios, increasing the motivation, engagement, and learning outcomes of students & & \\ \hline\end{tabular}\caption{Stakeholders requirements, methods, and data sources employed for evaluate the usefulness of ISIDGaCLeS}\label{tab:stakeholders-requirements-tool}\end{center}\end{table}

A feature analysis method in its case study approach \citep{kitchenham1997desmet} will assess the satisfaction for the requirements RE1 and RE5. The feature analysis is a qualitative evaluation method, in which the features that a system should possess are mapped to the requirements that users expect for a particular situation \citep{kitchenham1997evaluating}. In the case study approach, these features are evaluated by means of case studies with users who will have used the system (ISIDGaCLeS) on a real project. The use of case studies helps to increase confidence in the results, reducing the risk of systematic bias. However, the obtained results by the feature analysis still have a certain degree of subjectivity \citep{ kitchenham1997desmet }. The multicase studies (as experiments) and the implementation of ISIDGaCLeS will support the discussion concerning the requirements RE1-RE5.

Assessment findings about the requirements RE6 and RE7 will be obtained from the experiments (multicase studies), using an adaptations of mixed evaluation method proposed by \citet{martinez2006thesis-davinia, martinez2003thesis-glue}. This method combines quantitative and qualitative data gathering techniques, and it has successfully been applied in the evaluation of systems that promote collaboration \citep{bot2008thesis-carlos, nav2011thesis-carlos}.

The first step in this method is the creation of a set of categories with the different concepts to be analyzed in the multicase study (experiments). For the assessment of requirements (RE6 and RE7), these categories are: the experience of instructional designers with gamification and CL scenarios; the motivation, engagement, and learning outcomes (stages of skill development and knowledge acquisition) of students; the easiness and usefulness of system; the problems found in the use of system; and the suggestions for improvements.

The second step is the collection of data from different quantitative and qualitative sources. We propose the collection of quantitative data using likert scales \citep{lik1932thesis-carlos} passed to participants, and the use of open text questionnaires and interviews to collect qualitative data in forms filled by participants (individual interviews). Focus on students, monitoring reports of participation, including the performed actions, in gamified CL scenarios will be used as quantitative data. Besides, additional data focused on instructional designers will be used as quantitative; these data are the time and complexity to design gamified CL scenarios and to gamify CL scenarios with and without ISIDGaCLeS.

Finally, the third step consists in the triangulation of data obtained from different sources \citep{mar2003thesis-carlos}. In this step, the quantitative data will be use for showing trends, and the qualitative results are used to confirm or reject these trends, as well as to understand them and identify emergent features in particular representative situations \citep{denzin2005tesis-davinia}.

\subsubsection{Collaborative Learning Situations for Assessments of Contributions}

Three authentic collaborative learning situations will be instantiated in four occasions by different participants, these situations will be designed covering various knowledge domains collaborative strategies and durations. Tables \ref{} and \ref{} respectively summarize the three collaborative situations, and the four experiments that will be done for the ISIDGaCLeS evaluation.

Figure \ref{fig:cscl-erem-evaluation} depicts the CSCL-EREM diagram for the multicase studies (experiment); the changes for each CL situations that will be used in the assessment are located in the ground facets, such as general information, stakeholders features, evaluation features, and the date in which will occur the experiment.

\begin{figure}[thbp]\begin{center}\includegraphics[width=0.85\textwidth]{cscl-erem-evaluation.png}\caption{CSCL-EREM representation diagram for the ISIDGaCLeS evaluation in CL scenarios}\label{fig:cscl-erem-evaluation}\end{center}\end{figure}