Read LeAM-D28.pdf text version

Deliverable No: D28

Interactive Concept Mapping

The LeActiveMath Consortium December 2005

Main Authors: Martin Homik (UdS) Erica Melis (DFKI)

Project funded by the European Community under the Sixth Framework Programme for Research and Technological Development

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Project ref.no. Project title

IST-507826 LeActiveMath- Language-Enhanced, User Adaptive, Interactive eLearning for Mathematics

Deliverable status Contractual date of delivery Actual date of delivery Deliverable title Type Status & version Number of pages WP contributing to the deliverable WP/Task responsible Author(s) EC Project Officer Keywords

Restricted Project Month 24 - 2005 December 31st 2005 December 31st Interactive Concept Mapping Prototype final 36 T2.2, T3.3, T3.9, T5.6, T6.1

WP3 Martin Homik, Erica Melis Colin Stewart concept maps, mind maps, interactive, concept map exercises, concept map exercise authoring

c LeActiveMath Consortium 2004

Page 2 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Contents

1 Introduction 1.1 1.2 1.3 1.4 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 5 5 6 6 7 8 8 9 11 12 13 13 15 15 16 20 20 21 21 22 23 25 26 26 28

Requirements and description in DoW . . . . . . . . . . . . . . . Source Material . . . . . . . . . . . . . . . . . . . . . . . . . . . . Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 Knowledge Representation 3 Interactive Concept Mapping 3.1 3.2 3.3 3.4 Existing Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interactivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4 Architecture 4.1 4.2 4.3 4.4 4.5 4.6 4.7 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Assessment Generator . . . . . . . . . . . . . . . . . . . . . . . . Next Step Suggestor . . . . . . . . . . . . . . . . . . . . . . . . . Prefetching and Caching . . . . . . . . . . . . . . . . . . . . . . .

5 Authoring Concept Map Exercises 5.1 5.2 Concept Map Exercises Format . . . . . . . . . . . . . . . . . . . Author Interface . . . . . . . . . . . . . . . . . . . . . . . . . . .

6 Performance 7 Usability Evaluation 7.1 7.2 Experiment Setting . . . . . . . . . . . . . . . . . . . . . . . . . . Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . .

c LeActiveMath Consortium 2004

Page 3 of 36

Deliverable D28 Interactive Concept Mapping 8 Related Work 9 Conclusion

LeActiveMath (IST-507826)

32 33

c LeActiveMath Consortium 2004

Page 4 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

1

Introduction

Concept mapping is a well known cognitive technique for visualising and constructing relationships between different concepts in a diagram [20].

1.1

Motivation

Concept maps are used for many - including educational - purposes. They are known as a means to help humans to structure their knowledge and to construct a structure in their minds. Several tools are available for visualising and creating those maps, e.g., CoolModes [22], CognitiveTools [21, 11], IHMC [18] or BrainBank [19]. As far as we know, those tools give no or very limited feedback. A few other tools can provide feedback, e.g., the simulation software SMISLE [5] and the recent Verified Concept-Mapping tool VCM [3]. In mathematics education, discovering and constructing structure and dependencies is an important aspect for a deep understanding. Therefore, we developed an interactive concept mapping tool, iCMap, integrated it into the adaptive, web-based learning environment for mathematics, LeActiveMath [13], and evaluated it in respect of performance and usability.

1.2

Requirements and description in DoW

In this section, we summarise all requirements stated in the requirement analysis [8] and in the Description of Work [7] and point to sections that elaborate on the accomplishment of each requirement. The requirement 4.20 for the interactive concept mapping tool iCMap as it is stated in the requirement analysis is the "provision of a mind map tool" that supports "the construction of dependencies of concepts in learner's mind" (§3.2) because it "helps students to visualise dependencies and have an individual way of approaching a topic" (§7.2). It requires to check "how students use the tool" (§7.2). In particular, iCMap should intend two usage scenarios: "one for free brainstorming and another for exercises requesting the construction of a map, in which feedback is given" (§3.2,§3.3). Moreover, an investigation is suggested to examine to what degree "students may not understand the intention of the tool and would have to be instructed"(§3.2). The task T3.9 (p. 32) in the Description of Work adheres: "There exist some concept map tools, e.g., MindMap and the concept map in SourceForge (belvedere.sourceforge.net). It is unclear and has to be investigated whether one of these tools can be adapted to work as services for LeActiveMath, or whether the interactive concept map tool has to be implemented from scratch (§3.1, §4.1, c LeActiveMath Consortium 2004 Page 5 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

§4.2). This tool will enable the student to draw diagrams with concept and item nodes, to annotate the nodes with properties, and to connect them via edges that are annotated with ontological relations, e.g., is-definition-of, depends-on (§3.2). The tool will automatically check the correctness of the student's input according to metadata and structures of the domain knowledge representation in the knowledge base (§3.4). This tool is supposed to coach a student in understanding the dependencies and in acquiring a holistic view of the domain" (§7). Finally,the Description of Work [7] describes the deliverable D3.9 (p. 34) as follows: · evaluate existing concept map tools (§3.1) · adapt or build browser-based GUI for tool (§4.1) · adapt or develop evaluation of student input (and possibly multi-modal (§3.3, §3.4) feedback) based on a comparison with information in the knowledge base · integration with LeActiveMath components, e.g., student model and knowledge base (§4.2, §4.3) Our work for iCMap accomplished all those requirements as shown in the remainder of the deliverable.

1.3

Source Material

Part of this deliverable has already been published or submitted. The description of iCMap's user interface , its architecture, and authoring interface is published in [16]. In [15], we report on a qualitative evaluation of iCMap, present how to connect iCMap to LeActiveMath's event framework, and elaborate how to interpret iCMap's events.

1.4

Outline

In §2, we briefly introduce LeActiveMath's knowledge representation and notation which is a prerequisite for the following sections. The concept mapping tool iCMap with its functionalities is introduced in §3 while the architecture is presented in §4. Afterwards, we describe authoring of concept map exercises in §5. A performance and a usability experiment report follows in §6 and in §7. Finally, we discuss related work §8 and conclude with a summary in §9. c LeActiveMath Consortium 2004 Page 6 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

2

Knowledge Representation

LeActiveMath's knowledge representation [14] extends OMDoc [6] - a semantic markup format for mathematical documents. OMDoc in turn has evolved as an extension of the OpenMath 1 standard for mathematical expressions [2]. Mathematical documents are represented by a collection of typed OMDoc learning items annotated with metadata. These metadata annotations define mathematical and educational properties by which, amongst others, learning items can be related to each other. That is, implicitly the encoding by an author implies a domain ontology as well as an educational ontology. Both can be exploited for checking a concept map's relative correctness. Each item belongs to one of three layers: abstract, concept, or satellite layer. The abstract layer contains topics, currently represented by OpenMath symbols2 that are also defined in OpenMath content dictionaries. The concept layer comprises definitions and theorems. Definitions are statements that define a meaning of symbols while theorems typically describe relations between mathematical concepts. Symbols and concept items are the main items of a mathematical ontology. The satellite layer includes example and exercise items which are auxiliary from a domain-point-of-view. Example items illustrate concepts while exercise items are used for training and testing competencies w.r.t. one or more concepts. Symbols, concepts and satellites are stored in LeActiveMath's mathematical database and are accessible via a unique identifier. For a hierarchical mathematical ontology and in order to connect different areas of mathematics, meaningful collections of items have to be represented. The element theory serves the purpose of assembling learning items in mathematical theories. Smaller theories can be assembled to bigger ones via the import mechanisms. Items are linked by relations. In the following, we formally define a set of relations used for interaction and evaluation in iCMap. Let S be the set of symbols, Def the set of definitions, Thm the set of theorems, Ex the set of examples and Exc the set of exercises. Additionally, let C = (Def Thm) be the set of concept items, Satellites = (Ex Exc) the set of satellite items, and U = S C Satellites the universe. Then, a relation that expresses a mathematical dependency between concepts and symbols is defined by domPre C × S 3 . It describes which symbols are mathematically necessary in order to define the actual concept. The for relation links definitions to symbols, concepts to concepts, and satellites to concepts. It is defined by: for (Def × S) (C × C) (Satellites × C). Note that a symbol can have several definitions, e.g., convergence of a function can be defined via - formulae or via sequences. The

http://www.openmath.org/ the abstract layer is sometimes called "abstract concept layer", its members are also called "abstract concepts". For historical reason and in order to prevent confusion, we use the term "symbol" in the following. 3 domPre is an abbreviation for the OMDoc relation type domain prerequisite

2 Because 1 see

c LeActiveMath Consortium 2004

Page 7 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Figure 1: Illustration of LeActiveMath's knowledge representation

counter relation describes the current item as a counter example for a specific concept item, i.e., counter (Satellites × C). The is a relation is a partial order on S with is a S × S that serves to represent mathematical hierarchies and special cases, e.g. a 'function' is a specific 'relation'. Figure 1 illustrates a small piece of the LeActiveMath knowledge representation by a simple concept map. It contains three symbols4 and three definitions. The definition of a 'relation' is linked to the symbol 'relation' by a for relation; the symbols 'right-unique relation' and 'symmetric relation' are specification of the symbol 'relation' referenced by an is a relation; and the definition of an 'equivalence relation' demands the symbol 'symmetric relation' as a domain prerequisite. The definition of a 'strict order' is not linked to any other present item.

3

Interactive Concept Mapping

In this section, we present our evaluation of existing tools and we describe the three phases of interactive concept mapping: interaction, feedback, and verification.

3.1

Existing Tools

Prior to implementation, we evaluated several existing tools. Our requirements were: easy development of iCMap, stability, interaction facilities, and steady development. At that time, we knew only a few concept mapping tools and only a few more could be found in the Internet. For example, IHMC [18] is an excellent open source Java implementation for creation of and navigation through concept

4 Note, due to a better natural language understanding, iCMap's user interface displays the annotation "Concept" for symbols.

c LeActiveMath Consortium 2004

Page 8 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

maps. It is developed by concept mapping researchers and offers conversion to many other formats known in this field. Unfortunately, only the binaries are open source. No programming interface is provided for integration of IHMC. CognitiveTools [21, 11] is another concept mapping tool developed by cognitive science researchers. Due to its commercial licence, it is inadequate for our implementation. Belvedere5 is designed to support problem-based collaborative learning scenarios with concept and evidence models, and provides multiple representational views (tables and graphs) on those models. It is distributed under GPL licence. But the project's activity stopped and is not developed any more. Moreover, it is developed as a standalone application, unsuitable for extensions. CoolModes [22] is a client application that provides a collaborative framework designed to support discussions and cooperative modelling processes in various domains. It allows for a variety of different modelling languages and comes with a general programming interface for defining and plugging in new visual languages. We decided to base iCMap on CoolModes for the following reasons: a) its plugin mechanism allows an easy development of iCMap, b) it is stable and it is tested in many projects, and c) it provides support for collaboration

3.2

Interactivity

A concept map in iCMap contains nodes that represent symbols, concept items, satellite items, and theories (see Figure 2). It also contains edges that correspond to relations and theory imports as described in §2. In addition, an author or a user can define (arbitrary) nodes and edges for a concept map (see §5). As mathematical formulae cannot be rendered in iCMap and long texts are inappropriate for concept maps, iCMap displays merely an item's type and title. iCMap can be employed for both, predefined exercises and free exploration activities. Teachers and authors can design explicit mapping tasks. That is, they can provide a partial concept map and ask a student to complete it by adding nodes and edges from predefined sets of nodes and edge types. Alternatively, the tool can be used independently for exploration and demonstration purposes in LeActiveMath, i.e., to demonstrate or explore structures and dependencies of a domain. In exercises with iCMap, the learner can interact by manipulating (adding, deleting, labelling, replacing) nodes. She can add a node from the palette shown on the right hand side of Figure 2. The nodes available there are dynamically generated from the exercise representation and include nodes for symbols, concept items, satellite items, theories, and templates. Apart from template nodes, each node has an associated meaning that relates to some learning item in the

5 belvedere.sourceforge.net

c LeActiveMath Consortium 2004

Page 9 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Figure 2: iCMap: concept map with feedback and hand-written node

knowledge base. A template node is instantiated as soon as a learner drags a node from the palette or a link from content presented by LeActiveMath and drops it into the template node. In addition to general template nodes, iCMap supports typed template nodes. Only mathematical items of a particular type are allowed to be dropped into the template node. The type specification of a node is encoded in the exercise itself. This eases the verification of user input. The user can also manipulate edges, i.e., add, delete or name edges in a partial map. The choice in the palette is dynamically generated (see Figure 2). An edge can represent relations available in OMDoc as well as (deductively) implied or 'any' relations. OMDoc relations correspond to those described in in §2. Implied relations, such as the equivalence relation, are not explicitly used in OMDoc but their validity can be deduced. For example, if two definitions relate (for) to the same symbol, then they are considered mathematically equivalent. The user indicates an equivalence by drawing an is equivalent edge. Another example is the belongs to edge, which is used to relate theories. Finally, 'any' relations c LeActiveMath Consortium 2004 Page 10 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

are presented by unspecified, no-name edges. They enable the user to express her awareness of a relation between mathematical items, in case she does not know which one. Similar to template nodes, template edges can occur in authored concept maps and are to be instantiated. Nodes and edges can be renamed or annotated with arbitrary relation names. A learner decides which descriptions or annotations are more suitable for her understanding. Since the learning item is attached to a node and each edge has a type, a renaming does not affect the validation. In addition, hand-writing annotations, as depicted in Figure 2, are supported on tablet PCs, i.e., they can be placed anywhere, in particular they can be linked to template nodes. Currently, those nodes are not verified. The learner benefits from iCMap not only by actively constructing a concept map and receiving feedback where possible but also from browsing (in LeActiveMath) the learning item information linked to nodes. Those links are presented by the information button in each node (see Figure 2). A click on the information button opens LeActiveMath's dictionary [9] that displays the node's item information. At any time, the user can request a hint or verify a concept map. iCMap offers global and local verification. The former checks all edges; the latter checks only a selected edge.

3.3

Feedback

There are different kinds of feedback presentation. iCMap presents feedback graphically and textually. The former uses colours for presentation of the verification results. Correctly introduced relations are rendered green; false relations are rendered red. This is called flag feedback [1], because it provides a binary flag. Studies show that flag feedback can dramatically improve the efficiency of learning [1]. Additionally, edges that are evaluated as incorrect relations are decorated with annotations, which are also required, e.g., for colour blind people (see Figure 2). Textual feedback (i.e. hints) is presented upon request by using the context menu of an edge. Among others, explanations can be: · "There is no relation between these nodes." · "This edge has a wrong type." · "This edge has a wrong direction." · "This edge is correct, but subsumes several steps. Please elaborate." In the background, iCMap calculates a set of suggestions and, upon request, selects one suggestion randomly. Some suggestion can be presented in more and more detail as a three-hint sequence (similar to the sequence of pointing c LeActiveMath Consortium 2004 Page 11 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

hint, teaching hint, and bottom-out hint in [23]). For instance, a suggestion to add a missing edge can be specified by presenting the source node, the target node, or the edge type. Other suggestions propose to replace a template node or a template edge, to add a node from the palette, or to add an item from LeActiveMath content presented in a browser. The latter always calculates a reasonable item, i.e., an item that is strongly related in the sense of having many links to items represented by nodes in the workspace and the palette. The rational behind is that a reasonable item connects the most items that are part of the concept map, and hence, contributes mostly to the completion of a concept map. Feedback cannot be generated for all types of nodes and edges. In particular, annotations that are hand-written cannot be evaluated, yet. Still, they can be useful for the learner.

3.4

Verification

iCMap has two verification resources: the knowledge base and the authored information contained in an exercise. The verification includes three subroutines: direct, deductive, and fault-tolerant. Direct verification includes the mere match of the learner's input with the (mathematical and educational) ontologies represented in LeActiveMath's knowledge representation and the match with nodes and edges in an authored exercise. Here, metadata and imports of elements are checked to determine correctness. For example, if the learner draws an edge from node a to node b, then iCMap requests information linked to learning item for a, and it checks if there is any relation represented by the edge that points to the learning item for node b. Indirect verification is performed by deduction. In case the matching fails, e.g. for handling transitive relations, deduction is used for the evaluation. Let s, s1 , s2 , s3 S be symbols, d, d1 , d2 Def definitions, e Ex an example, x, y, z S C any symbol or concept, and t1 , t2 , t3 U theories. Suppose the user draws an edge from a node labelled by `periodic function' referring to a symbol s1 to a node labelled by `relation' referring to a symbol s3 . Then the edge will be evaluated as correct, since (s1 , s2 ) is a and (s2 , s3 ) is a, where s2 is a symbol for `function'. The general transitivity rule for the is a relation is: (s1 , s2 ) is a (s2 , s3 ) is a - (s1 , s3 ) is a Similar transitivity rules apply for the `domain prerequisite' relation (domPre) and for the `belongs to' relation (belongs to) for theories: (x, y) domPre (y, z) domPre - (x, z) domPre t1 t2 t2 t3 - t1 t3 c LeActiveMath Consortium 2004 Page 12 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

The belongs to edges that connect a theory node with another node are checked against the inclusion of elements in a theory. Apart from transitivity, equivalence relations can be deduced by a simple rule: if two definitions are for the same symbol, then they are equivalent: (d1 , s) for (d2 , s) for - d1 d2 The diagnosis is somewhat fault tolerant through allowing correctness modulo some simple deductions. For instance, in some context it could be tolerable to accept the relation (d, s) is a or (e, s) for. The fault tolerance rules are: (d, s1 ) for (s1 , s2 ) is a - (d, s2 ) is a (e, d) for (d, s) for - (e, s) for (e, d) for (d, s1 ) for (s1 , s2 ) is a - (e, s2 ) for

4

Architecture

In this section, we explain design decisions for iCMap's implementation and integration into LeActiveMath's learning environment including its invocation method, its communication, and dissemination of events.

4.1

Implementation

CoolModes and its plug-ins, so-called reference frames, are implemented in Java. A reference frame comprises various Java classes of nodes and edges that logically belong together. All these classes must be part of a Java package that can be located anywhere in the system. CoolModes can scan the file system for new reference frames and integrate them automatically. A ReferenceFrame acts as a controller. It provides initialisation, updating and disposing methods as well as handling of PackagePrefixes to complement the classpath. It may have a Palette that basically provides the graphical user interface for node selections, edge buttons, and control buttons. Figure 3 illustrates iCMap's implementation. The shaded box comprises classes and interfaces defined in CoolModes. To get a working plug-in we need a controller class (CMapReferenceFrame). Following the Model-View-Controller paradigm, we associate our nodes and edges with their respective models. The node model CMapNodeModel class stores the node's learning item identifier, its internationalised title, its type, as well as whether it is a template node. It's corresponding view is in charge of the node's presentation, i.e, visual aspects of the title, the node's background, and the information button including an c LeActiveMath Consortium 2004 Page 13 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Figure 3: iCMap architecture

action listener for invocation of the dictionary tool that displays the full content. In contrast to nodes, edges have in CoolModes no views, only a model. The CMapEdgeModel class maintains the learning items it connects, its type, and whether it is currently validated. The palette (CMapPalette) is used for the construction of the graphical user interface and for the implementation of action listeners enabling the palette to react to user activity, e.g., verify a concept map or make a suggestion. For internationalisation, the palette implements a locale listener and several resource bundles. The first version of iCMap [17] merely consisted of a reference frame and a knowledge base connector for retrieving a learning item's metadata information. In the scope of the EU project LeActiveMath we added further components such as an event manager (see §4.4) for publishing learner action events, an assessment generator (see §4.5) for judging a learner's final performance, a next step suggestor (see §4.6) for providing hints on request, and an authoring interface (see §5). iCMap can be started in two different modes. To run it either in exercise or in authoring mode, a CMapExerciseRefFrame class or a CMapAuthoringRefFrame class respectively is loaded into the system. Both are derived from the common class CMapReferenceFrame. c LeActiveMath Consortium 2004 Page 14 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Figure 4: iCMap communication with LeActiveMath

4.2

Integration

Since CoolModes is a client application LeActiveMath provides iCMap as a Java WebStart application that communicates via synchronous XML-RPC requests and asynchronous events. Upon triggering a link to a concept map exercise, LeActiveMath's content manager creates dynamically a JNLP file (Java Network Launching Protocol). It contains a link to iCMap's resources, links to LeActiveMath's relay services for event publishing and knowledge base access, a user's id, a user's name, and a concept map exercise identifier. The file is sent to the user's client which in turn executes it by downloading iCMap if not already present on the user's host, and by instantiating iCMap with the transmitted parameters. iCMap includes an exercise loader, which retrieves the exercise from the knowledge base and dynamically generates a (partial) concept map as well as a palette comprising a given set of nodes and edge types for representation of mathematical items and relations (see §5 for authoring concept map exercises). Additionally, each node is equipped internally with its item's identifier. Whenever the user validates edges or request item information, affected identifiers are transmitted to LeActiveMath's relay services.

4.3

Communication

Communication between iCMap and LeActiveMath is twofold (see Figure 4). Requests to the content knowledge base are synchronous while client-events c LeActiveMath Consortium 2004 Page 15 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

are issued asynchronously and they are distributed via LeActiveMath's event framework [10]. iCMap acts as an event source, i.e., a component that can publish events. Any component that subscribes to the events published by an event source is called a listener. It receives event messages from the event source. Learner models can be listeners. Any registered listener decides itself which events to analyse. iCMap's verification uses synchronous communication. For each edge to be evaluated, iCMap requests metadata information from the knowledge base on the affected mathematical items wrt. to the nodes connected by the edge. Further metadata is retrieved from authored exercise. On the basis of this data iCMap generates feedback and presents it to the learner (see §3.4).

4.4

Events

LeActiveMath's event framework (see §4.3 and Figure 4) passes events issued by iCMap to any registered listener (such as learner models, tutorial component, etc.) for further analysis and presentation. Each event message comprises descriptive attributes. Attributes common to all LeActiveMath events are: type, timestamp, and source. The type indicates the user action. Depending on the type, the event may carry additional data describing the event. The timestamp indicates when the event took place. The source refers to the class name of the component that produced the event. iCMap produces three event types: exercise started, exercise step, and exercise finished. All types contain additional information, namely the identifier of the exercise the user is currently dealing with. Exercise started and exercise finished event types just publish that the user has started or finished an exercise. The timestamp information in both types enables a listener to calculate the duration of an exercise and present to the user. Below is a typical exercise start event.

<ActivemathEvent type="ExerciseStarted" ts="1138965848405" id="196657" source="org.activemath.events.impl.XmlrpcEventService"> <Date>2006-02-03 12:24:08 CET</Date> <User id="majax"/> <Session id="6B3DD49E2376AD74301274717872FD1F"/> <Item type="private" id="mbase://theory_icmap/private_exercise4"/> <ExerciseStarted/> </ActivemathEvent>

The exercise finish event additional information, namely the success rate of a worked exercise. c LeActiveMath Consortium 2004 Page 16 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

<ActivemathEvent type="ExerciseFinished" ts="1138966205231" id="196658" source="org.activemath.events.impl.XmlrpcEventService"> <Date>$2006-02-03 12:30:05 CET</Date> <User id="majax"/> <Session id="6B3DD49E2376AD74301274717872FD1F"/> <Item type="private" id="mbase://theory_icmap/private_exercise4"/> <ExerciseFinished success="0.6"/> </ActivemathEvent>

An exercise step event reports about any user action in detail. Since exercise step events also have a timestamp, a listener can deduce if a learner worked intensely on an exercise, i.e., if she added or deleted many nodes and edges in respect of the duration. An exercise step event comprises two basic parts: one for action description and one for metadata. The action description's notation is in OpenMath. The metadata block complies to LeActiveMath's metadata as described in [4]. The following piece of code depicts an exercise step event's scaffold.

<ActivemathEvent type="ExerciseStep" ts="1138972328577" id="196661" source="org.activemath.events.impl.XmlrpcEventService"> <Date>2006-02-03 14:12:08 CET</Date> <User id="majax"/> <Session id="6B3DD49E2376AD74301274717872FD1F"/> <Item type="private" id="mbase://theory_icmap/private_exercise4"/> <ExerciseStep> <ExerciseInput> ... </ExerciseInput> <metadata> ... </metadata> </ExerciseStep> </ActivemathEvent>

If a user adds or deletes a node, an exercise event is published reporting the action and the node's item identifier. In the following example, a user's action is encoded as an application of an OpenMath symbol, added, on a node that represents some learning item with a $ID identifier.

<ExerciseInput> <OMOBJ> <OMA> <OMS cd="org.activemath.exercises.cmap" name="added" /> <OMA> <OMS cd="org.activemath.exercises.cmap" name="node" /> <OMSTR>$ID</OMSTR> </OMA>

c LeActiveMath Consortium 2004

Page 17 of 36

Deliverable D28 Interactive Concept Mapping

</OMA> </OMOBJ> </ExerciseInput>

LeActiveMath (IST-507826)

The metadata part contains a competency annotation and lists all knowledge items that are directly involved into the action step.

<metadata> <competency value="tools" subvalue="concept_map" /> <relation type="for"> <ref xref="ID" /> </relation> </metadata>

Note, an event for removing a node differs only in the OpenMath symbol that describes the user action. In this case, it is removed. From this kind of event, a learner model can derive an action pattern and analyse it, e.g., what items has a learner added to the concept map (with or without a connection); what items has she omitted (items she probably does not know very well); what items has she added and then deleted (items she is probably unsure about); in what order has she processed items (probably a decreasing order starting with items she knows pretty well) etc. Adding or deleting an edge causes an exercise step event which carries additional information describing if an edge has been added or removed, the edges type, and the source node's as well as the target node's identifier. A learner model can analyse (in)correct relations and assesses whether a learner has understood a relation between learning items. This event's encoding is very similar to the one for adding a node.

<ExerciseInput> <OMOBJ> <OMA> <OMS cd="org.activemath.exercises.cmap" name="added" /> <OMA> <OMS cd="org.activemath.exercises.cmap" name="edge" /> <OMA> <OMSTR>$ID1</OMSTR> <OMSTR>$ID2</OMSTR> </OMA> </OMA> </OMA> </OMOBJ> </ExerciseInput> <metadata> <competency value="tools" subvalue="concept_map" />

c LeActiveMath Consortium 2004

Page 18 of 36

Deliverable D28 Interactive Concept Mapping

<relation type="for"> <ref xref="$ID1"/> <ref xref="$ID2"/> </relation> </metadata> </ExerciseStep> </ActivemathEvent>

LeActiveMath (IST-507826)

A global validation triggers an exercise step event of type hint with a list of all checked edges. A local validation lists only the checked edge, but additionally inserts the depth of the hint, i.e., '1' for a validated action, '2' for an explained action, and '3' for a corrected action. Here is an example for a local validation.

<OMA> <OMS cd="org.activemath.exercises.cmap" name="validated" /> <OMA> <OMS cd="org.activemath.exercises.cmap" name="edge" /> <OMA> <OMS cd="org.activemath.exercises.cmap" name="foredge" /> <OMA> <OMSTR>$ID1</OMSTR> <OMSTR>$ID2</OMSTR> </OMA> </OMA> </OMA> </OMA> <hint depth="1" /> <relation type="for"> <ref xref="$ID1"/> <ref xref="$ID2"/> </relation>

Suggestion are step events of type hint. In the exercise input part, we only encode the action. In the metadata part, we also provide the depth of the suggestion and the supplied text.

<OMS cd="org.activemath.exercises.user_requests" name="hint" /> <hint type="hint" depth="1" text="There is a template node left. You can replace it./>

Apart from exercise events, iCMap triggers dictionary events when a learner presses a node's information button. Dictionary events carry the item's identifier the user has looked for. c LeActiveMath Consortium 2004 Page 19 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

4.5

Assessment Generator

The assessment generator judges a learner's performance when he finishes an exercise. Currently, we use a simple approximation. iCMap counts all correct correct edges and correlates them with all user actions. A learner who creates a correct concept map and rarely uses hints and verification obtains a high grade. Someone who has deficiencies in a particular domain, and therefore, asks for help is punished by a lower assessment value. The disadvantage of this simplicity is that a learner who just adds a correct edge and finishes the exercise will be assessed unjust, namely with a value of 1.0.

4.6

Next Step Suggestor

The next step suggestor distinguishes between six types of suggestions: important edge, important node, missing edge, missing node, template edge, and template node. For each of the first four types, the suggestor maintains a list. · Important edges and nodes are marked as such by an author and are extracted from the concept map exercise. · In contrast, missing edges and nodes are computed automatically. Missing edges are edges that the learner has not yet drawn in the workspace, and that are correct in respect of the constraints presented in §3.4. Moreover, missing edges connect either nodes in the workspace or nodes in the workspace pointing to nodes in the palette. · A missing node is a node that a learner has not yet inserted into the workspace and which can be connected to at least one node already present in the workspace. Note, this includes that a missing node is not necessarily listed in the palette. It can represent any learning item that is member of the knowledge base. For the latter two suggestion types, the suggestor only counts the number of template nodes and template edges. It does not distinguish between them. The next step suggestor's logic is as follows: first it randomly selects a type. If the type is one of "important edge", "important node", or "missing edge", then it randomly chooses an element from the corresponding list. For this element, a suggestion is generated. There is a special case for "missing node". Each element in the "missing node" list has a priority value which is determined by the number of possible connections between the element and the nodes in the workspace. The suggestor selects the node with highest priority. We call such a node reasonable. For the types "template node" and "template edge", the suggestor only creates a message that indicates the learner to look at a template node or a template edge. c LeActiveMath Consortium 2004 Page 20 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

For all types of suggestions exists a basic rule: never make the same suggestion twice in a row.

4.7

Prefetching and Caching

In a first implementation of iCMap, verification of concept maps caused a large number of XML-RPC requests. Since XML-RPC requests are computationally time-consuming the concept mapping tool was not sufficiently scalable. For performance reasons (see §6), now iCMap incorporates a prefetching and a client-side caching mechanism. One way to reduce execution time, is to fetch information as soon as possible. To do so, iCMap stores an item's relation metadata information locally on the client-side in a list. This data is requested in a background thread when a learner inserts an unconnected node into the partial concept map. Among others, the obtained data comprises all possible (ordered) relations between the new node and the nodes already present in the concept map. Additionally, iCMap tests if there exist any (ordered) relations between nodes of the partial concept map and the new node. Essentially, we create a local snapshot of the knowledge base such that iCMap verifies quickly against the snapshot by testing if a drawn edge is a member of the client-side list. This approach is called prefetching Another method to reduce execution time, is to minimise XML-RPC requests by caching. To prevent repeated requests, a request and its result is stored in a local cache after its first invocation. Prefetching differs from caching by storing merely possible relations between nodes in a partial concept map and omitting relations between nodes in a partial concept map and nodes not yet present. The calculation of suggestions suffers similar problems, and therefore, iCMap applies the same mechanisms for overcoming the bottleneck.

5

Authoring Concept Map Exercises

iCMap's concept mapping exercises are tightly integrated into LeActiveMath's knowledge representation. In fact, they are OMDoc exercise items which can be annotated by metadata. But because the current exercise markup format does not support concept mapping exercises, and because iCMap's exercises do not need the rich facilities of the exercise markup format such as condition or answer map, iCMap's problem statements are formalised by additional XML elements. In the following, we discuss the representation of concept mapping exercises and the authoring. c LeActiveMath Consortium 2004 Page 21 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

5.1

Concept Map Exercises Format

Exercises in iCMap are OMDoc satellites. Each exercise representation starts with an exercise element that carries a unique identifier and a type declaration. An metadata part follows, which contains amongst others, a for relation that links the exercise to one or more concepts. A iCMap exercise comprises three parts: description, evaluation, and layout. The main structure is shown below.

<exercise id="$ID" type="concept_map"> <metadata> <extradata> <relation type="for"> <ref xref="$ID"/> </relation> </extradata> </medadata> ... <ConceptMapDescription> ... </ConceptMapDescription> <ConceptMapEvaluation> ... </ConceptMapEvaluation> <ConceptMapLayout> ... </ConceptMapLayout> </exercise>

The description part describes the setting of CoolMode's palette and workspace. The palette element lists nodes and edge buttons. Nodes carry the MBase identifier of the learning item. Edge buttons require an edge type description, which can represent either an OMDoc edge or an additional edge defined by the author. These type declarations are used in the workspace part and in the evaluation part. The workspace element lists nodes and edges that are to be displayed, when the exercise starts. Each node has a unique identifier, which is local to the actual iCMap exercise and carries a unique knowledge base identifier if and only if it refers to a learning item. Otherwise, the node is a template node that may be specified by a type. Edges describe which nodes have to be connected and the relation they represent. The relation type is restricted to the set of edge types introduces in the palette.

<ConceptMapDescription> <palette> <palettenode mBaseID="$ID"/> <theory mBaseID="$ID"/> <edgebutton type="$EdgeType"/> <edgebutton type="uses"/> </palette> <workspace> <node mBaseID="$ID" id="1"/> <node template="true" id="2"/>

c LeActiveMath Consortium 2004

Page 22 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

<node template="true" type="definition" id="3"/> <edge from="1" to="2" type="$EdgeType"/> <edge from="2" to="3" type="uses"/> </workspace> </ConceptMapDescription>

In the map evaluation part, the author can define new relations, that may not be represented in the knowledge database. Such a possibility was demanded by several teachers. Moreover, she can decide, which edges she considers important. In both cases, these particular edges are described by the learning items the edge connects and by the edge type.

<ConceptMapEvaluation> <addedRelationInstance from="$ID" to="$ID" type="$EdgeType"/> <importantEdge from="$ID" to="$ID" type="$EdgeType"/> </ConceptMapEvaluation>

Finally, in the layout part, node positions of those nodes introduced in the workspace can be assigned.

5.2

Author Interface

The implementation of source code by non-experts is far too error-prone and unintuitive and may reduce an author's motivation and willingness to write new content. Therefore, iCMap offers a visual authoring tool. This is implemented as an authoring mode which extends the user mode by provision of means for maintenance and classification of nodes and edges. Figure 5 exemplifies the creation of a concept map exercise. On the left, an author produces the partial concept map that is initially presented to the user. On the right, the palette's tabbed panel provides a view on the exercise's nodes, edges, and their annotations. The learning items and the theories tab lists learning and theory items that are also to be shown in the user's palette. An author can augment these lists by drag-and-drop learning items from LeActiveMath browser or from the dictionary. In addition to learning items that are already present in the knowledge base, an author is allowed to create and add any new learning item. These items are represented by so-called "non-knowledge base nodes". Consequently, these nodes and their relations cannot be verified against the knowledge base (see §3.4). However, these nodes are part of an exercise and can be checked against its encoding. Depending on the exercise's goal, some nodes are more important than others. These nodes are considered `important' and can be marked by an author. c LeActiveMath Consortium 2004 Page 23 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Figure 5: Authoring concept map exercises

Similarly, any edge in a partial concept map, can be marked as a 'important' edge. Below the tabbed panel, a list of buttons shows what types of edges are allowed in an exercise. An author can assemble any set of edge types. To assure flexibility, an author can introduce new relation types and new relations that do not exist in the knowledge base. For instance, she can define a new relation type similar to for expressing that two learning items are somewhat similar. A 'quadratic function' is a 'continuous function' is another example which is a true mathematical statement but one that is not encoded in the knowledge base. iCMap allows an author to define new relations just by pressing the "New edge button" and by provision of a label. New relations are introduced by drawing an edge from the source to the destination node. Edges that are considered correct are marked as such and if they are also important then they are marked as such, too. This relation is stored in the exercise rather than in the knowledge base. Finally, an author can save the exercise in a file and reload it for revision at a later point in time. It is also possible to retrieve an exercise from the knowledge base by provision of a unique identifier but we cannot transmit a modified c LeActiveMath Consortium 2004 Page 24 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Node

Learning item

Time (ms) without cache with cache 375 180 34 37 46 50 87 40

Factor

1 2 3 4 5 6 7 8

symbol function definition of a function (1) definition of a function (2) example of a function symbol relation symbol right-unique relation theory relations theory functions

1915 1992 1923 1543 1731 1960 1753 1950

5.11 11,07 56.56 41.70 37.63 39.2 20.15 48.75

Table 1: Node creation performance exercise back to the system.

6

Performance

Prefetching and caching (see §4.7) improves iCMap's performance significantly. In our evaluation, the client application was hosted on an Intel(R) Pentium(R) 4 CPU 2.80GHz with 1.5 GB RAM. The LeActiveMath server including its MBase service ran on an Intel(R) Xeon(TM) MP CPU 2.70GHz with 8 GB RAM. Waiting for an edge verification for 1.5s is just bearable, but waiting more than 10s for a concept map verification in the first prototype was not acceptable. Prefetching and caching reduces the verification time to approximately 10ms. The cost for a decreased verification time is an increase in time for adding a node to a partial concept map. Without caching, prefetching costs on average 1.8s for a first node, 2.0s for a fifth node, and 4.2s for a 13th node. The more nodes are present in a partial concept map and the more relations a new node has, the longer prefetching takes. Client-side caching reduces the fetching time by a factor of up to 60. Table 1 lists durations for adding nodes one by one c LeActiveMath Consortium 2004 Page 25 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

into the partial concept map depicted in Figure 2 with and without iCMap's performance enhancements. Even with iCMap enhancements, fetching takes up to 340ms, but this duration is negligible as the transfer is processed in the background in a concurrent thread right after dragging a node into the workspace.

7

Usability Evaluation

In order to test whether iCMap is usable and whether iCMap's feedback stimulates learning and meta-cognition we conducted a first qualitative evaluation. Since meta-cognition is hard to discover from behaviour we used think-aloud protocols and questionnaires.

7.1

Experiment Setting

We conducted evaluative experiments at the University of Saarland with students in a seminar Hand-On Mathematics for Computer Scientists. Eight third and fourth year bachelor students from computer science, mathematics and computer linguistics with basic knowledge on calculus and rudimentary knowledge on LeActiveMath's knowledge representation participated in the experiment. Each student was assisted and tested individually by a tutor6 in a session of about 90-120 minutes. Each session has been recorded on audio as well as on video for later more detailed analysis. A session consisted of three phases: instruction, problem solving without feedback, and problem solving with feedback. In the instruction phase, the students were introduced into LeActiveMath's knowledge representation. On a sheet of paper, the tutor listed the types of LeActiveMath's learning items and its relation types which were likely to occur in the problem solving tasks. In parallel, he sketched a graph whose nodes denoted learning items and edges represented relations interpreting LeActiveMath's knowledge representation as a concept map. During the whole session, the student could glance at the sketch at any time. The first problem solving task required to complete a partial concept map covering the mathematical topics on relations and functions (see left concept map of Figure 6). The students had to inspect the given concept map, to add nodes from iCMap's palette, and to link nodes by edge types offered by the palette. Neither the system nor the tutor provided feedback on the concept map to be constructed, but the tutor hinted where it was necessary to overcome misunderstandings or misinterpretations of the system's functionality. For instance, if the student asked if a node in the concept map and a node in the palette were the same the tutor advised to use a node's information button.

6 first

author of this deliverable

c LeActiveMath Consortium 2004

Page 26 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Figure 6: Partial concept maps for first and second problem solving task

The partial concept map consisted of six nodes: three symbols and three definition nodes. They are connected by one for, one is a, one is equivalent, and two template edges. The palette contains in summary 14 definition nodes, 13 symbol nodes, and 2 assertions nodes. In addition to the the edge types in the partial concept map, the palette offers domPre and any edges. This partial concept map is an erroneous example for which the learner has to find the errors (see [12] for a detailed discussion). Erroneous examples are qualified for testing a student's current knowledge and understanding of a domain as well as her critical thinking abilities. This partial concept map contains three erroneous edges: · The definition of a 'strict order' is no definition for the symbol 'relation'. · A definition is never equivalent to a symbol, in particular, from a mathematical point of view, a `relation' is not equivalent to a `right-unique relation'. · A symbol is never a specification of a definition. Especially, the symbol 'relation' is no specification of the definition of a 'relation' which includes that the symbol 'relation') is a specification of itself. Moreover, the exercise contains a misguidance element, that is, the palette contains a node for the definition of an 'equivalence relation' as well as nodes for the symbols 'symmetric relation' and 'reflexive relation' but no node for the symbol 'transitive relation'. Due to completeness, the latter is necessary in order to connect the definition of an 'equivalence relation' with all its domain prerequisites. By introducing erroneous edges and misguidance elements we intended (1) to provide a starting point and (b) to test whether students would spot and be able to correct the errors or if they would use the partial concept map just as a starting point. Before the session start, the tutor mentioned that the given concept map might be correct. c LeActiveMath Consortium 2004 Page 27 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

In the second problem solving task, iCMap's feedback facilities were enabled. Students could verify their concept maps, ask for error explanations, and ask for next step suggestions. The partial concept map was similar to the concept map in the second phase (see right concept map of Figure 6). It covers the mathematical topics relations, functions, order, sequences, and continuity. The concept map comprises six nodes: one unconnected example node, three symbol nodes, one definition node and one template node of type `definition'. These nodes are linked by is a edges and by template edges. The palette provides 15 symbol nodes, five definition nodes, two assertions, one exercise, and one example node. The set of edges equals that of the first task. This concept map exercise is an erroneous example, too, for which the learner has to find and correct the errors. Erroneous edges are: · A symbol 'relation' is no specification of a symbol 'quadratic function'. · There is no relation between the definition of a 'function' and a symbol 'quadratic function'. The second item is a misguidance. Since, in this phase, the user was allowed to ask for feedback, it was expected that using validation would be one of the first steps. In this case, a learner would notice immediately a false connection. To prevent this, we included erroneous template edges into the partial concept map and since template edges are not evaluated, no feedback is provided. Another misguidance element is hidden in the node palette: the exercise node does not relate to any other concept or satellite in this map. After a session, the tested student filled a questionnaire. On a range from 1 ("I totally agree") to 5 ("I totally disagree") 15 statements on concept maps in general and iCMap in particular had to be rated.

7.2

Results and Discussion

The questionnaires reveal that almost all students were not aware of concept maps as intuitive means for learning in maths and they also were surprised about the complexity of mathematical domain structures, even for basic topics such as relations or functions. All students reported that concept maps support learning and structuring mathematical learning items. They suggested that a different presentation of mathematics, in particular a graphical presentation, can foster memorisation. Statements like "I memorise stuff in pictorial form." or "A picture clarifies many things." supports this hypothesis. As expected, the students performed worse in the first problem solving task due to the lack of feedback. Typical errors were: misinterpretation of an edge's direction and wrong relation types. One of the most frequent errors was the usage of an is a relation between definitions as well as between definitions and c LeActiveMath Consortium 2004 Page 28 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Figure 7: A student concept map without feedback

symbols. Figure 7 depicts a typical concept map created by a student without feedback7 . The student declares the definition for a 'strict partial order' as a specification of a definition for a 'partial order' which in turn is a specification of the symbol 'asymmetric relation'. In principle, the edges themselves are correct, but not the edge types. A correct answer is: the symbol 'strict partial order' is a specification of a symbol 'partial order' whereas the symbol 'reflexive relation' is a domain prerequisite of the definition for a 'partial order'. These typical errors were corrected by the system in the second task and the students adapted their actions. Surprisingly, in the beginning of the second task, all but one students did not request a validation for the initial partial concept map. They started examining the concept map by requesting information on the learning items with the information button as well as by deleting or adding edges. As soon as they were confident that they had corrected the initial concept map they validated it. The student who validated the concept map right in the beginning argued that his confidence in his mathematical knowledge is low, and therefore, he did not believe he did well in the first problem solving task. By validation he intended to understand iCMap's verification feedback. This suggests that he was trying to find a pattern which he might use. In other words, he reflected on his behaviour, assessed his performance, and devised a plan how to proceed.

7 Figure

7 and 8 are reproduced in English

c LeActiveMath Consortium 2004

Page 29 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

All students were able to find the erroneous edges in the second problem task by using the validation and the dictionary, but only students with a strong background in maths identified all errors in the first task. Generally speaking, the validation function fosters self assurance. In the first problem solving task, students were unsure whether their edges were correct or not. They spent more time on examining single edges and requested information on learning items related to the edge quite often. If they were unable to resolve the particular problem confidently, they focused on other parts of the concept map but returned to the original problem regularly. In contrast, when they were allowed to use validation and the validation confirmed the correctness of the concept map, they continued to complete the concept map attentively without returning to correct edges and their nodes. This observation is confirmed by the questionnaires in which students state that validation is very useful and simplifies the overall task. The longer the students worked with iCMap the more they used the validation function regularly as soon as they encountered difficult relations. By regular verification they prevented getting on the wrong track (e.g., by using a wrong edge type) as it was in the first task. Due to a rather small number of participants, we could not observe any evidence for a bias in using a specific validation mode. Both, local and global validation, were used frequently, though sometimes students used the global validation to check their last step, which is essentially a local validation. Students who validated locally also used the error explanation and correction functionality. The system's suggestions received mixed ratings. In the first session, suggestions were provided deterministically, i.e., the same suggestions were provided in the same order. This strategy is cumbersome and frustrates the user. So we changed the strategy to a random mechanism which was more motivating. But still, the students were not enthusiastic about the suggestions, as the system often advised to draw transitive edges. This is because students started to request hints only, when their concept map advanced. There were only a few relevant relations left compared to lots of transitivities. The figure below exemplifies the problem. A suggestion to connect the symbols 'sequence' and 'relation' by an is a edge does not make much sense, if a learner already has connected both symbols by a chain of correct is a edges.

As opposed to this, such a suggestion is reasonable, if one of the listed edges between the symbols 'sequence' and 'relation' is missing. In one experiment, c LeActiveMath Consortium 2004 Page 30 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

Figure 8: A student concept map with feedback

the student wondered about such a suggestion and found out that there was an edge missing in a chain. Instead of inserting the suggested edge, the student added the missing edge. All students stressed they never followed a suggestion blindly. Instead, they examined each hint by comparing it with their understanding of the learning item and by requesting the dictionary information on the learning item. This critical thinking behaviour is also exhibited in the think-aloud protocols and user logs.8 Moreover, most students were convinced of the correctness of their own concept maps and disagreed with some feedback. That is, they considered some relations between learning items valid and indicated this by an 'any' relation or by keeping typed edges, though, they were indicated as incorrect by the system. Figure 8 depicts a student's insistence on the specification a 'quadratic function' is a 'monotone function', which in fact is true but was not encoded in the knowledge base previously. Similarly, she insisted on the pedagogical dependency between the symbols 'injective' and 'function'. The students tried to understand iCMap's feedback and the mathematical structure the feedback bases on. Again, they used the dictionary or tried to find similar, correct pat8 Tests

with school students could reveal different results.

c LeActiveMath Consortium 2004

Page 31 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

terns in other parts of the concept map. If they considered their own arguments stronger, then they kept the edges, otherwise they deleted them. This behaviour suggests that students are able to evaluate the quality of their knowledge and to assess their own thinking in terms of recalling their reasoning path, of deriving a possible explanation for the system's feedback, and of comparing their own explanation with the system's. Some students solved the tasks strategically. Strategies observed in the studies are: spotting errors in the partial map first and continuing to complete the map afterwards; connecting symbols and definitions by a for edge first; connecting symbols by is a edges first; sorting all nodes according to term distance; processing palette nodes one by one; or deleting all edges first, including all erroneous edges. Some students developed a plan which involved the selection of a strategy (or several strategies). Others knew they could apply a strategy, but did not do so, because their concept maps would get too messy. It turned out that the dictionary was the major supporting tool not only as an information tool. Students extracted conceptual as well as structural information. Conceptual information, for instance, helped to identify two equivalent definitions for a common symbol 'function'. Moreover, it helped to understand the difference between an `asymmetric relation' and an `antisymmetric relation'. As consequence, two students were able to connect the definitions for '(strict) order' with the correct symbol 'asymmetric' respectively 'antisymmetric'. But in this case, only one student managed to extract structural information and connected the nodes with the correct relation type. Moreover, the students used the dictionary to look up learning items that were connected by a false edge, instead of using the explanation function for the local error. In summary, we can state iCMap is a useful learning tool. It helped students to visualise and to structure mathematical knowledge. Its interaction, feedback, and verification facilities improved a student's understanding of a mathematical domain.

8

Related Work

To date, we know only one strongly related Australian project. Laurent Cimolino, Juday Kay, and Amanda Miller [3], the Verified Concept Mapper Tool (VCM). While VCM was explicitly designed to capture learner models as a side-effect of the concept mapping tool, iCMap's goal was to provide rich interaction facilities for visualisation, creation, and verification of concept maps, as well as to integrate it into the LeActiveMath learning environment. Another difference between VCM and iCMap is the stimulation of reflection by verification. Instead of decorating false edges by colour or annotation, VCM provides a list of natural language questions. These questions indicate false or missing connections. In fact, one of our students proposed to phrase suggestions in interrogative form, and pointed out that this is more motivating and c LeActiveMath Consortium 2004 Page 32 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

challenging. However, the drawback of this kind of verification is, that the more complex a domain or an exercise becomes as well as the more errors a learner's concept has the more questions are asked. iCMap has not this problem since it provides single suggestions and context-aware (local) error explanations. Since VCM puts a strong emphasis on the open learner model, it also presents the learner model's assumptions about the learner's performance and allows for negotiating. iCMap is not connected to LeActiveMath's learner model. It just publishes events that encapsulate the user's actions. If the learner wants to inspect the system's assumptions, she has to ask the learner model herself. Finally, VCM is not integrated in any larger learning environment, nor is any other concept mapping tool we know of. Hence, it cannot benefit from other tools an environment such as LeActiveMath might provide. For instance, in order to solve a concept mapping task students had to prepare their learning material. In iCMap, students can consult the dictionary during problem solving.

9

Conclusion

We presented the interactive concept mapping tool iCMap that has been developed in LeActiveMath and describes how all requirements are accomplished. We explained iCMap's architecture and design decisions, introduced a markup format for concept map exercises, and characterised the author interface. A performance evaluation proved that our design decisions for prefetching and caching in the context of web-based applications with lots of XML-RPC requests improves the execution time. The analysis of our first evaluative usability experiment has shown that interaction as well as the feedback aspects of iCMap can help students to improve their understanding of a domain and their metacognitive skills, among others, planning, structuring, critical thinking, and reflection. Because of their overall positive experience with iCMap, the students stated that they intend to integrate concept maps into their learning process. The opportunities that arise from iCMap's integration with the learning environment LeActiveMath and its tools (e.g., dictionary, knowledge base) include sophisticated interaction, feedback, and verification facilities as well as a connection to the learner model which can draw conclusions from the student's interaction in iCMap.

c LeActiveMath Consortium 2004

Page 33 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

References

[1] J. R. Anderson, A. T. Corbett, K. R. Koedinger, and R. Pelletier. Cognitive Tutors: Lessons Learned. The Journal of Learning Sciences, 4(2):167­207, 1995. [2] O. Caprotti and A. M. Cohen. Draft of the Open Math Standard. Open Math Consortium, http://www.nag.co.uk/projects/OpenMath/omstd/, 1998. [3] L. Cimolino, J. Kay, and A. Miller. Incremental Student Modelling and Reflection by Verified Concept-Mapping. In Open Learner Modelling Workshop at AIED-2005, Amsterdam, 2005. [4] G. Goguadze, C. Ullrich, E.Melis, J. Siekmann, Ch. Gross, R. Morales. LeActiveMath Structure and Metadata Model. Deliverable D6, LeActiveMath Consortium, 2004. accessible from http://www.leactivemath.org/. [5] W.R. Joolingen and T. Jong. Design and Implementation of SimulationBased Discovery Environments: the SMISLE Solutio n. Journal of Artificial Intelligence and Education, 7:253­277, 1996. [6] M. Kohlhase. OMDoc: Towards an Internet Standard for the Administration, Distribution and Teaching o f Mathematical Knowledge. In Proceedings Artificial Intelligence and Symbolic Computation AISC'2000, 2000. [7] LeActiveMath Partners. Description of Work. Technical report, LeActiveMath Consortium, 2004. accessible from http://www.leactivemath.org/. [8] LeActiveMath Partners. Requirement analysis. Deliverable D5, LeActiveMath Consortium, 2004. accessible from http://www.leactivemath.org/. [9] LeActiveMath Partners. Enhanced Dictioanry. Deliverable D15, LeActiveMath Consortium, 2005. accessible from http://www.leactivemath.org/. [10] P. Libbrecht and S. Winterstein. The Service Architecture in the ACTIVEMATH Learning Environment. In N. Capuano, P. Ritrovato, and F. Murtagh, editors, First International Kaleidoscope Learning Grid SIG Workshop on Distributed e-Learning Environ ments. British Computer Society, 2005. See http://ewic.bcs.org. [11] H. Mandl and F. Fischer. Wissen sichbar machen, chapter Mapping Techniken und Begriffsnetze in Lern- und Kooperationsprozessen. Hogrefe, 2000. [12] E. Melis. Design of Erroneous Examples for ActiveMath. In Artificial Intelligence in Education. Supporting Learning Through Intelligent and Socially Informed Technology. 12th International Conference (AIED 2005). c LeActiveMath Consortium 2004 Page 34 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

[13] E. Melis, E. Andr`s, J. B¨denbender, A. Frischauf, G. Goguadze, P. Libe u brecht, M. Pollet, and C. Ullrich. ActiveMath: A Generic and Adaptive Web-Based Learning Environment. International Journal of Artificial Intelligence in Education, 12(4):385­407, 2001. [14] E. Melis, J. Buedenbender E. Andres, A. Frischauf, G. Goguadse, P. Libbrecht, M. Pollet, and C. Ullrich. Knowledge Representation and Management in ActiveMath. International Journal on Artificial Intelligence and Mathematics, Special Issue on Managem ent of Mathematical Knowledge, 38(1-3):47­64, 2003. [15] E. Melis and M. Homik. Student Modelling Through ActiveMath's Interactive Concept Mapping. International Journal of Artificial Intelligence in Education, 2006. submitted. [16] E. Melis, P. Kaerger, and M. Homik. Interactive Concept Mapping in ActiveMath. In U.Lucke J.M.Haake and D. Tavangarian, editors, 3. Deutsche e-Learning Fachtagung der Gesellschaft f¨r Informatik, DeLFI 2005, pages u 247­258. GI, 2005. [17] J. M¨ller, M. M¨hlenbrock, and N. Pinkwart. Towards using concept mapu u ping for math learning. Learning Technology newsletter, 6(3):13­15, July 2004. accepted. [18] A. J. Ca nas, G. Hill, R. Carff, N. Suri, J. Lott, T. Eskridge, G. G´mez, o M. Arroyo, and R. Carvajal. CmapTools: A Knowledge Modeling and Sharing Environment. In A.J. Ca nas, J.D. Novak, and F.M. Gonz´lez, editors, a Concept Maps: Theory, Methodology, Technology, Proceedings of the First International Conference on Concept Mapping, pages 125­133, Universidad P´blica de Navarra: Pamplona, Spain, 2004. http://cmap.ihmc.us/. u [19] T.W. Nordeng, D. Dicheva, L.M. Garshol, L. Rønningsbakk, and J.R. Meløy. Using Topic Maps for Integrating ePortfolio with eCurriculum. In Proceedings of ePortfolio2005, pages 213­222, Cambridge, UK, 2005. EIfEL. [20] J.D. Novak and D.B. Gowin. Learning How to Learn. Cambridge and NY: Cambridge University Press, 1984. [21] M. Nueckles, J. Gurlitt, T. Pabst, and A. Renkl. Mind Maps & Concept Maps. Beck Juristischer Verlag, 2004. http://www.emindmap.de. [22] N. Pinkwart. A Plug-In Architecture for Graph Based Collaborative Modeling Systems. In H.U. Hoppe, M.F. Verdejo, and J. Kay, editors, Shaping the Futire through Intelligent Technologies. Proceedings of the 11th Conference on Artificial Intelligence in Education, pages 535­536. IOS Press, 2003. c LeActiveMath Consortium 2004 Page 35 of 36

Deliverable D28 Interactive Concept Mapping

LeActiveMath (IST-507826)

[23] K. VanLehn, C. Lynch, K. Schulze, J. A. Shapiro, R. Shelby, L. Taylor, D. Treacy, A. Weinstein, and M. Wintersgill. The Andes Physics Tutoring System: Lessons Learned. International Journal of Artificial Intelligence in Education, 15(3):137­204, 2005.

c LeActiveMath Consortium 2004

Page 36 of 36

Information

36 pages

Find more like this

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

125346


You might also be interested in

BETA