|Citation Details: Earle, A. (2002). Designing for Pedagogical Flexibility - Experiences from the CANDLE Project. Journal of Interactive Media in Education, 2002 (4). ISSN:1365-893X [www-jime.open.ac.uk/2002/4]|
|Print: [PDF] [HTML]
Published: 25 July 2002
|Editor: Martin Oliver (Univ. College London, UK)
Reviewers: Gwyneth Hughes (U. East London, UK), Albert Ip (U. Melbourne, AUS)
Institute of Education
20 Bedford Way
London, WC1H 0AL
Abstract: This paper examines the experience of a group of designers attempting to implement pedagogical flexibility in the design of the CANDLE system. It sketches out how flexibility is emerging as a new design criterion, but warns that the implementation of such flexibility is fraught with conflicts. After foregrounding the myth of pedagogical neutrality in system design, it examines CANDLE's early decision to build a system around a pan-pedagogical framework and the problems inherent in such an undertaking. In particular it reviews issues such as the operationalisation of pedagogical theory, the difficulties of disaggregation of learning resources into separate objects, the epistemological conflicts in the use of static ontologies for domain representation, metadata, meaning and communities of practice, access rights and granularity. It concludes by calling for educational systems designers to consider pedagogy in all its complexity in the process of design and development.
Commentaries: All JIME articles are published with links to a commentaries area, which includes part of the article's original review debate. Readers are invited to make use of this resource, and to add their own commentaries. The authors, reviewers, and anyone else who has 'subscribed' to this article via the website will receive e-mail copies of your postings.
This paper describes some of the challenges faced by the CANDLE project team in implementing pedagogical flexibility in a courseware management and delivery system structured around a learning objects database. After a brief account of the increasing importance of flexibility in educational systems, a working conception of flexibility will be sketched out. A quick overview of conflicts and issues in the provision of this flexibility will then serve to ground the reasoning behind the decision to implement multiple pedagogies in the CANDLE system. The experiences of the designers in implementing this decision will be used to formulate questions which go to the heart of the issue to be addressed here: the difficulties and contradictions inherent in designing an educational system based on different conflicting pedagogical models. Particular attention will be paid to the problems inherent in the disaggregation and de-contextualisation of learning objects and to the epistemological difficulties of using a database to contain learning objects and static ontologies to re-assemble them into learning resources. Finally, the question of how feasible it is to implement multiple pedagogies will open up for further debate the issue of whether the foundational antinomies which set different pedagogical models apart preclude their co-existence in a single educational system.
Flexibility and customisability are increasingly being positioned at the core of educational systems across the university and training sectors, a focus which lies behind the current drive for the establishment metadata standards (EPFL (ARIADNE, 1999; ADL 2001; IMS 2001). To illustrate, one of the Learning Objects Metadata Working Group's stated purpose is: "to enable computer agents to automatically and dynamically compose personalized lessons for an individual learner (LOM 2000). In the same vein, Schatz believes that "we are on the verge of being able to provide learning customized for each specific learner at a specific time, taking into account their learning styles, experience, knowledge and learning goals" (Schatz 2001). Collis establishes flexibility as one of the cornerstones of her 'new didactics for university instruction' (Collis 1998) implemented in the TeleTOP system, whilst Moran and Myringer see the move towards flexible learning as a paradigm shift (Moran and Myringer 1999). In the commercial sector, e-learning providers are increasingly using customisability of courseware as a major selling point: Arthur Anderson employ 'blended learning events' to offer flexibility in their Virtual Learning Network. SmartForce, who stake a claim as the world's largest e-learning company, describe their system as "a next generation e-Learning environment made up of over 20,000 e-Learning objects that can be assembled on-the-fly to create truly personalized, truly collaborative learning experiences based on the wants and needs of the learners." How far this rhetoric of re-usability and customisability is applicable in existing and future educational and training environments is arguable and forms the orienting question for this paper. First, however, it is pertinent to examine the concept of flexibility itself.
Despite the increasing significance of flexibility in the design of educational systems, and the irresistible forces driving it forward, there is as yet no commonly accepted conceptualisation of flexibility. Here I will not undertake the task of constructing such a conceptualisation, but will refer the reader to work which has already been done in this area (Edwards 1997; Edwards, Nicoll et al. 1999) and will limit myself to sketching out the landscape of flexibility, sufficient for the purposes of explaining some of the implementation difficulties involved in CANDLE.
Moran et al note how different terms have been adopted in different contexts, including flexible learning and flexible delivery; they argue that this stems from the fact that converging trends in different countries remain to be developed into a sound theoretical construct. They cite the definition of flexible learning adopted by Mid Sweden University as a working example (Moran and Myringer 1999):
Van den Brande (ibid) defined flexibility as "enabling learners to learn when they want (frequency, timing, duration), how they want (modes of learning), and what they want (that is, learners can define what constitutes learning to them)." Collis et al. went further and identified five dimensions of flexibility in their Telescopia project, which they further broke down into 19 different aspects (Collis, Vingerhoets et al. 1997):
Although in order to claim to be truly flexible, a system needs to integrate all the above, this paper will concentrate on the fourth dimension, flexibility related to instructional approach and resources, as this maps most closely onto the objective of this paper: an examination of the difficulty of designing for pedagogical flexibility.
Despite the ubiquity of flexibility as a goal of educational systems designers, there is not a great deal of literature that looks at the conflicts involved in the provision of flexibility, although the literature there is suggests that this is an issue that requires critical engagement e.g. (Collis, Vingerhoets et al. 1997). Johnston argues that whilst flexibility has achieved many desirable ends, it "has also been instrumental in opening opportunities for other, less desirable, developments in the system" (Johnston 1999). She identifies four sets of issues here. The first set of issues are administrative, and revolve around issues of location, scheduling and employment. Johnston only sees these issues as problems insofar as they have not yet been systematically resolved, and she locates the site of their resolution as the organisation.
The other three sets of issues are the pedagogic, economic and philosophical implications of flexibility. The pedagogic issues will be addressed later in the paper. The economic issues include intellectual imperialism, the "MacDonaldization" of education and access inequalities. The inextricable relativism of post-modernism constitutes the main philosophical issue. Although Johnston views these three sets of issues as less tangible, she argues that they are nonetheless more contentious than the administrative challenges and she calls for them to be addressed at the level of the education system and profession.
It is in an attempt to take a small step in this direction that this paper interrogates the pedagogical issues that have arisen from the implementation of flexibility in CANDLE. It will not, therefore, address the administrative, philosophical or economic challenges. The rationale for this is not that it is logically possible to disentangle the pedagogical from the other issues for they are inextricably linked, but rather because a critical pedagogical gaze unencumbered by the wider, contextual issues is needed to get to the heart of the question of whether it is possible to implement multiple pedagogies in a single system. Accordingly, the thrust of this paper is deliberately theoretical.
CANDLE (Collaborative and Network Distributed Learning Environment) is a Fifth Framework project which set out "to use the Internet to improve the quality and reduce the cost of ICT teaching in Europe by using web and multimedia technology, and to enable co-operation between universities and industry in creating and reusing learning material and improving the quality of delivery" CANDLE Consortium,</AUTHOR></AUTHORS><YEAR>2001</YEAR><TITLE>CANDLE Project Web Site</TITLE><VOLUME>2001</VOLUME><NUMBER>March 2001</NUMBER><URL>http://www.candle.eu.org/web/about/about.htm</URL></MDL></Cite></EndNote> ADDIN EN.CITE <EndNote><Cite><Author>Wiley</Author><Year>2000</Year><RecNum>199</RecNum><MDL><REFERENCE_TYPE>7</REFERENCE_TYPE><AUTHORS><AUTHOR>D. A. Wiley</AUTHOR></AUTHORS><YEAR>2000</YEAR><TITLE>Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy</TITLE><SECONDARY_AUTHORS><SECONDARY_AUTHOR>D.A. Wiley</SECONDARY_AUTHOR></SECONDARY_AUTHORS><SECONDARY_TITLE>The Instructional Use of Learning Objects: Online Version</SECONDARY_TITLE><URL>http://reusability.org/read/chapters/wiley.doc</URL></MDL></Cite></EndNote>.
Metadata, literally "data about data," is descriptive information about a resource. For example, the card catalog in a public library is a collection of metadata. In the case of the card catalog, the metadata are the information stored on the cards about the Author, Title, and Publication Date of the book or resource (recording, etc.) in question. The labels on cans of soup are another example of metadata: they contain a list of Ingredients, the Name of the soup, the Production Facility where the soup was canned, etc. In both the case of the library book and the can of soup, metadata allow you to locate an item very quickly without investigating all the individual items through which you are searching.
Figure 1: Functions of CANDLE system
CANDLE was conceived of as an integrated learning environment into which trainers and teachers (and, in some cases, learners) could deposit and subsequently re-engineer their courseware. This would then provide a large repository of learning objects which could be searched (by tutors, learners and, ultimately, by on-the-fly course creation algorithms) and assembled into courses which would also be delivered through CANDLE. The system would then store the data on student pathways through material etc. and use this information to suggest new ways of working through the learning objects. Finally, evaluation functions were to be built in to the system.
*To make each of the intended functions of the system clearer, one example of courseware, a bank of multiple-choice questions for a first year Introduction to Psychology course, will be used as illustration. The first function of CANDLE is that of courseware re-engineering which involves the decomposition of extant courseware into separate learning objects identifiable through a series of metadata tags. So, the tutor who had developed the bank of multiple-choice questions would need to think about how she would enter this courseware into the system. She would immediately be presented with the problem of deciding whether to enter all the questions into the database as a whole, or to enter them as separate learning objects. Whichever decision she made, she would then need to describe each object with the appropriate metadata tags, indicating to potential searchers the content, form, and pedagogical purpose of each object.
The second function is the search and retrieval of these learning objects and their construction into courses by learners, or by lecturers or tutors (the third function) for delivery to learners. At this stage, another tutor teaching on Introduction to Psychology might search the courseware depository for assessments for first year Psychology courses, to complete a course she was developing. In a more open setting, students might also need to seek out assessments to evaluate their own learning.
The fourth function consists of delivering this learning to the student. So, in this instance, CANDLE might allow students to take their multiple-choice exam online. The fifth and final function revolves around the evaluation of courseware and learning objects. Here the system would assist both tutors and students in evaluating the multiple-choice exam.
As CANDLE is in the first instance an educational system, a decision was made at the proposal stage of the project to position pedagogical issues at the heart of the system design. The question of how to do this is a more difficult issue and can be resolved in a number of ways. The first, and seemingly the simplest way of ensuring pedagogical flexibility is not to include pedagogical considerations in the system design, resulting in pedagogical neutrality ? or so the argument runs. Although this is the default approach taken, including in much of the recent work in the field of learning objects (Wiley 2000), it is based on the fallacy that unless a pedagogical position is made explicit (whether in terms of an individual's beliefs or the functionality of a system), it does not exist. In effect, what is usually touted as pedagogical neutrality by software vendors is merely pedagogical naïvety. Two arguments will be brought into play to defend this point.
First, pedagogical beliefs are not the exclusive remit of educators ? everyone holds what Bruner refers to as folk pedagogies, that is, ideas about teaching and learning which themselves have an impact on the educational strategies they employ (Bruner 1996). Moreover, these beliefs are deeply rooted in the cultural environment which imposes epistemological values on its communities which in turn have pedagogical implications of their own. Even if ? or arguably precisely because - systems designers are not able to articulate their own pedagogical stances, their beliefs will, by default, be articulated into the systems design process. More importantly, by construing pedagogical issues as outside of the remit of systems development, such designers will not be able to recognise the unintended pedagogic effects of other, apparently unrelated design decisions. To borrow from the words of Tenner, they will not be able to recognise technological decisions that "bite back" (Tenner 1997).
The second point revolves around the nature and boundaries of pedagogy. Watkins et al. sketch out how conceptions of pedagogy have developed over the last century (Watkins and Mortimore 1999). They note a shift in the research literature from a focus on the largely unidimensional analysis of teacher's 'style' to current, complex and multi-dimensional models of pedagogy which take into account the teacher, the classroom or other context, content, epistemologies and other elements. According to such models which emphasise the interplay of all the elements, it is not logically possible to disentangle pedagogical issues from technological ones as the latter are a constituent element of the former. In other words, one cannot draw a boundary between pedagogy and technology in the way that a pedagogically naïve approach suggests you can.
In view of the logical impossibility of developing a pedagogically neutral system, what options are open to educational systems designers? One way of ensuring the pedagogical soundness of an application is to take a single, recognised pedagogical model and integrate it into the system design. There are an increasing number of suggestions of how to do this. Leflore gives potential developers sets of guidelines to implement gestalt theory, cognitive theory and constructivist theory in web-based learning materials (Leflore 2000). Jonassen et al offer a six-step framework for designing constructivist learning environments (Jonassen and Rohrer-Murphy 1999). Reigeluth's updated Instructional-Design Theories and Models book offers a wide variety of theories and suggestions on how to implement them (Reigeluth, 1999).
This method has been used in the development of a number of significant educational systems, including the Fifth Dimension materials, designed around cultural-historical activity theory, and Plato which, along with all Integrated Learning Systems is an instantiation of the behaviourist paradigm (Underwood and Brown 1997). The same approach has recently been taken in the development of other systems designed around learning objects Bannan-Ritland</Author><Year>2000</Year><RecNum>201</RecNum><MDL><REFERENCE_TYPE>7</REFERENCE_TYPE><AUTHORS><AUTHOR>Brenda Bannan-Ritland</AUTHOR><AUTHOR>Nada Dabbagh </AUTHOR><AUTHOR>Kate Murphy</AUTHOR></AUTHORS><YEAR>2000</YEAR><TITLE><styles><style face='1'></style></styles>Learning Object Systems as Constructivist Learning Environments: Related Assumptions, Theories and Applications</TITLE><SECONDARY_AUTHORS><SECONDARY_AUTHOR>D.A. Wiley</SECONDARY_AUTHOR></SECONDARY_AUTHORS><SECONDARY_TITLE><styles><style face='2'></style></styles>The Instructional Use of Learning Objects: Online Version</SECONDARY_TITLE><URL>http://reusability.org/read/chapters/</URL></MDL></Cite></EndNote> ADDIN EN.CITE <EndNote><Cite><Author>Norman</Author><Year>1997</Year><RecNum>177</RecNum><MDL><REFERENCE_TYPE>0</REFERENCE_TYPE><AUTHORS><AUTHOR>N. Norman</AUTHOR></AUTHORS><YEAR>1997</YEAR><TITLE>Communication technologies and education: lessons in the potential of innovation.</TITLE><SECONDARY_TITLE>Journal of Advanced Learning Technologies</SECONDARY_TITLE><VOLUME>5</VOLUME><NUMBER>3</NUMBER><PAGES>43-53</PAGES><KEYWORDS><KEYWORD>pedagogy, instructional design, ict</KEYWORD></KEYWORDS></MDL></Cite></EndNote>· Scaffold the learner's increased self-responsibility for learning.
A related, but slightly different approach is taken by the developers of such programmes as STAR Legacy, JASPER and CSILE, who could be described as working from an action research paradigm. Although much of the design rests on a wide literature base, these programmes are built on a considerable, and evolving, research base of their own ? an expensive process in terms of manpower.
CANDLE initially rejected these options for two reasons. The first is that the whole of the CANDLE system is built around a learning objects database; consequently a critical factor in the success of the system will be the number and kind of learning objects that populate the database (Connolly and Thorn 1991). If no one enters any of their courseware into the system, then there will be nothing from which to develop new courses. A scenario where only learning materials which fit a given pedagogical model are allowed into the database will automatically reduce the material available to users. The reverse of the coin is that if instructors use a pedagogical paradigm in their teaching practices which does not match the paradigm legitimised by the system, then CANDLE will not be able to offer any materials of use to them. This is in essence an issue of usability, and abuts on the distance between a user's current practice and the practice required by the system. This distance is of particular importance in an educational system where the pedagogical beliefs of an instructor are one of the key factors affecting the success or otherwise of the implementation of new educational technology (Fullan 1992).
The second reason is more fundamental: as CANDLE is intended for use across the university, corporate and SME sectors throughout the European Union, at a time when the importance of the context of education is increasingly being recognised, it was judged that no single model or set of pedagogical guidelines would effectively fit all learning events. Certainly, there is evidence that although the different countries in Europe are facing similar challenges, there is more divergence than convergence in the responses of their education and training systems (Green, Wolf et al. 1999). Prescriptions are necessarily simplifications and as such cannot do justice to the complexity of the site of implementation of CANDLE. Moreover, pedagogic prescriptions involve value judgements about pedagogy, which the team did not feel able to make, being cognisant of the link between educational theory and the socio-political contexts of that theory (Walkerdine 1998) and the impossibility of developing a single, decontextualised, over-arching definition of quality (Green 1995). The designers therefore made an early decision to design CANDLE around multiple pedagogies, so that it encompasses the traditional, instructivist approaches and the more constructivist and collaborative constituents of the increasingly prevalent student-centred learning environments (Jonassen and Land, 2000), as well as the behaviourist paradigms which underpin the competency-based frameworks which structure much of the SME and corporate training sectors.
Designing for this degree of pedagogical flexibility poses a whole new set of problems. This section will not attempt a full account of the pedagogical design process and all the difficulties encountered but will concentrate on a few of the more crucial problems with a view to illustrating the challenges inherent in the implementation of pedagogical flexibility.
Figure 2: CANDLE and the life-cycle of learning objects (Graphic adapted from original by Anne Gilliland Swetland, in Baca2000, http://www.getty.edu/research/institute/standards/intrometadata/index.html)
Where CANDLE has found solutions, these will be presented, not as a neat resolution of the axiomatic conflicts which set apart the various pedagogical paradigms, but rather as subject to critical review. A discussion of the degree to which it is feasible to engineer such a resolution will be reserved for the concluding paragraphs of the paper.
Before addressing the problems, a fuller description the functions of CANDLE and their mapping onto the development life-cycle of learning objects (Baca 2000) will help to clarify to the reader some of the problems encountered in the design of CANDLE.
As can be seen from the diagram above, CANDLE does not have in-built courseware creation facilities such as html editors or packages such as Perception. Instead, it is anticipated that the actual courseware, such as presentation slides, will be created outside of CANDLE and then imported into the system. Therefore, the first stage of the learning object lifecycle, the creation and multiversioning, will not be part of CANDLE.
The second stage, that of organisation, is one where "objects are automatically or manually organized into the structure of the digital information system and additional metadata for those objects may be created through registration, cataloging, and indexing processes" (Baca ibid). This links with the re-engineering function in CANDLE and is probably the area in which the most intractable of the problems have been encountered, as it constitutes the lynchpin of the whole system: its structure.
Here, courseware that already exists in the computers and files of the tutors is re-engineered. In other words, it is analysed, broken down into its constituent parts and described using metadata. Although some of this metadata tagging can be performed automatically, as is the case with file size and format, the more difficult semantic tagging of elements such as content and pedagogical stance needs to be undertaken by a subject specialist ? in the case of CANDLE the instructor or trainer. Immediately one of the main constraints on the implementation of flexibility becomes apparent: there is a trade-off between the richness of the descriptions provided by the metadata and the amount of time required to tag each learning object.
Although this is a very real problem, it can be considered largely an administrative one. Other, fundamental issues deserve more extended treatment: these include the operationalisation of pedagogical theory in metadata, the implications of building a learning system around a database whose structure is provided by a static ontology, the issue of access rights and the problems of disaggregation and granularity. These issues are identified as fundamental because their solution cannot be easily found by administrative means as the problems themselves are theoretical.
The only real criterion for assessing the usefulness of a metadata schema (a set of metadata descriptors) is the extent to which it assists in the retrieval of objects, whether it be by an instructor searching for courseware, or by a learning algorithm built into the system for delivery of that courseware. From a pedagogical perspective, this will rest on two requirements: first that the metadata should adequately and faithfully represent pedagogical theory, secondly that it should have meaning for users of the system.
The first issue is arguably the more difficult of the two as it requires an operationalisation of the whole of the domain of pedagogical theory. As instructional designers have long known, the operationalisation of even a single, essentially descriptive, pedagogical theory, is far from trivial (Reigeluth 1999). This suggests that the operationalisation of multiple theories emerging from diverse research traditions with very different sets of frames of reference, let alone of the whole domain of pedagogy, is impracticable. This question was raised, albeit in a slightly different form, by the Shared Vocabularies for Representing Pedagogical Knowledge Working Group: "can pedagogical knowledge even be formalized?" The task of developing a pedagogical vocabulary, which they began, is still being carried forward by the IEEE Learning Technology Standards Committee.
A brief reference to the work of Reigeluth, who has pulled together a wide range of pedagogical models in one volume, Instructional-Design Theories and Models (Reigeluth 1999), will demonstrate some of the difficulties involved. Reigeluth summarises the goals, preconditions and values of every model presented. The values include such commitments and issues as:
Concepts such as authenticity, ownership, relevance, perspective, individualisation of learning, nature of the problem, focus on content emerge from these few values.
Operationalising any one of these concepts is a challenge. Moreover, even if a meaningful operationalistion of each concept were possible, there has to be a limit to the number of metadata tags that can be used to describe learning objects - we already have seven possible tags here - drawn from only two pedagogical models. The issue of selecting which concepts to choose as tags inevitably brings one back to the question of which values, and, hence, which models to privilege.
In CANDLE, after a first, unsuccessful attempt at using a literature survey to identify the factors that would constitute the pedagogical metadata, a decision was made to adopt a subset of the fourteen pedagogical dimensions of computer-based education (Reeves 2000), in addition to some of the tags in the IMS Learning Object Meta-data scheme (IMS 2001) as it was judged that these went a long way towards offering a workable operationalisation of the different pedagogical paradigms. Moreover, these dimensions (e.g. teacher role, user activity, flexibility) allow for the matching of materials, instructors and learners: just as Reeves has used the dimensions to characterise two different educational systems, it should be possible, in principle, to similarly describe different pedagogical models. Currently a project at the University of Twente is attempting to build a questionnaire based around Reeve's dimensions to help instructors and learners characterise their teaching and learning styles.
However, it is worth re-stating the first criterion for usefulness of pedagogical metadata: that it "should adequately and faithfully represent pedagogical theory." The very process of selecting metadata tags, and building questionnaires and tools around these automatically circumscribes pedagogy and will thus compromise the adequacy of representation of pedagogical theory.
This is an important point as it turns on the very nature of pedagogical theory and its boundaries - an issue which has already been touched on in the section on the myth of pedagogical neutrality. Becher, in his analysis of the cultures of disciplines has delineated two cognitive dimensions which differentiate different forms of knowledge: hard/soft (restricted/unrestricted) and pure/applied. These can be recruited to highlight the essence of the problem of operationalising pedagogical theory in metadata. He draws a parallel between the hard/soft dimension and Pantin's distinction between the restricted and the unrestricted sciences (Pantin 1968):
Briefly, restricted knowledge has clearly defined boundaries; the problems with which it is concerned tend to be relatively narrow and circumscribed. It focuses on quantitative issues, and tends to have a well-developed theoretical structure embracing causal propositions, generalizable findings and universal laws. It is cumulative, in that new findings tend to be linear developments of the existing state of knowledge? Unrestricted knowledge has the opposite characteristics: unclear boundaries, problems which are broad in scope and loose in definition, a relatively unspecific theoretical structure, a concern with the qualitative and particular, and a reiterative pattern of enquiry." (Becher 1989)
In Becher's terms, education can be considered a soft, applied discipline. Reframing the operationalisation problem in the terms presented above, attempts to fit pedagogy into a metadata schema, although quite properly motivated by a desire for the integration of theory into practice, can be seen as endeavouring to shoe-horn what is essentially an unrestricted/soft, applied domain into a hard form. The theoretical discontinuities inherent in such a process could arguably serve to undermine the validity of any metadata schema thus developed.
One further point deserves attention. Becher notes that one of the characteristics of soft, unrestricted knowledge domains is that they have a "relatively unspecific theoretical structure". Evidence of this is ubiquitous in the field of education, and can be found in this paper by an alert reader. Whereas the 'hard' sciences have a tight handle on what constitutes a hypothesis, a model, or a theory, there is no such widely accepted theoretical structure to organise pedagogical theory. Thus slippage between the usage of the various terms is commonplace and of itself makes the implementation of pedagogical theory problematic. However, attempts to impose such a theoretical model would be addressing the wrong problem, as such theoretical structures emerge both from the nature of the subject matter of the domain itself and from the communities, with their various degrees of coherence, that form around them.
This perhaps explains in part why it soon became apparent that, in practice, although Reeves' fourteen pedagogical dimensions did permit a description of learning objects which had the potential to link teaching and learning styles, pedagogical models and the learning objects from a systems point of view, it did not really work. The different communities constituting the CANDLE community had very different ways of conceiving of learning and teaching, and although Reeves offered them a common language with which to discuss pedagogical issues, it was not necessarily the language that they used in their practice and context. Metadata, meaning and communities of practice
The second issue to consider is whether the metadata chosen has meaning for the users. CANDLE is a system which requires, and is being designed to facilitate, the collaboration of all users, in particular those who are submitting and using courseware. Common ground has been identified as one of the most important factors in establishing collaboration (Baker, Hansen et al. 1999). It is therefore important that the concepts used in CANDLE metadata have a common and accessible meaning for all users. This does not happen automatically. On the contrary, in different sectors, very different conceptions of pedagogy are typically held, so much so that people in the commercial sector often are uncomfortable with conceptualisations framed in pedagogical terminology (Kewell, Hopkinson et al. 1999).
The 14 dimensions of Reeves offered the advantage of not obviously belonging to any particular pedagogical school ? they can be used to describe commercial training materials as easily as university lecture notes. Moreover, unlike most pedagogical paradigms which tend to change over time and as they are adapted by subsequent researchers and authors, they offer a relatively stable conceptualisation of pedagogy. In the CANDLE project, some early, informal experience of introducing instructors from both the commercial and university sectors to Reeves, suggests that it provided a relatively straightforward way of helping people conceptualise the differences between different pedagogies. A reaction of "Finally I understand what pedagogy is all about!" was not uncommon. It is as if the instructors and trainers have been given a language to describe pedagogy.
However, as mentioned above, the construction of common meaning is essentially a social process and, as such, is not unproblematic. How possible, then, is it for metadata tags to embed common meanings in view of the fact that they are used to describe learning objects which are, by definition, de-contextualised? An analysis of different learning resources will illustrate the problem. The educational metadata section of ADL's SCORM document includes a tag entitled 'learning resource', which can take such values as exercise, simulation, questionnaire or exam (ADL 2001). Whilst at first glance these descriptors seem unproblematic, a closer examination shows that even seemingly evident educational concepts change their meaning depending on the contexts in which they are used. For example, the ASTER project found that words such as tutorial and seminar were employed to describe different learning events, depending on the discipline in which they were used. So, for example, in the humanities, the term tutorial is used to describe small group learning events, and seminars slightly larger groups. In physics, however, these larger groups would be described as lectures (ASTER 1999).
According to Gee, who takes a situated cognition perspective, this should come as no surprise. He argues that there "is really no such thing in general as a report, an explanation, an argument, an essay? and so forth", but that such concepts are in fact situated meanings and cannot therefore be disentangled from their contexts. They "are made understandable, not in terms of some generic genre label like article or essay, but in terms of a cultural model of the production of work in the academic fields whose situated instances these are." (Gee 1997) Returning to the example of the multiple choice assessment given above will further underline this point: assessment is an essentially situated practice. Frederiksen & Collins view teaching, and one of it's key components, testing, as a complex system nested within a larger institutional system itself situated in a wider socio-political system (Frederiksen and Collins 1989). Accordingly, the forms of assessment that are legitimated in different socio-political contexts will differ (Broadfoot 1996): whilst multiple-choice questions will be quite acceptable as a summative assessment or 'exam' in one context, they would be rejected in another. The reasons for this stem from institutional beliefs about the purpose of assessment, the kind of learning that it should promote, and hence it's very nature.
Wenger offers another illuminating perspective here. In his work on Communities of Practice he argues that meaning is located in a process of its own negotiation (Wenger 1998). This process in turn involves the interaction of two complementary processes, namely participation and reification. Reification refers to a process of 'objectifying' our experience, and covers activities such as designing, naming and encoding. However, Wenger argues, reification cannot exist apart from participation: "indeed, reification always rests on participation: what is said, represented, or otherwise brought into focus always assumes a history of participation as a context for its interpretation. In turn, participation always organizes itself around reification because it always involves artifacts, words, and concepts that allow it to proceed." An application of these concepts to metadata suggests that tags, as reifications cannot have universal meaning, but can only make good sense within the participatory activities of a community of practice. Indeed, this is one of the reasons that issues of collaboration are being foregrounded in the design of CANDLE.
An alternative solution to this has been suggested by Recker & Wiley who recommend the addition of 'non-authoritative' metadata which they describe as follows: "this form of metadata captures the 'embedding' context of a learning object within instruction. For example, these data elements can describe how a learning object was reused, its juxtaposition to other learning objects, and its usefulness in particular instructional contexts. The metadata can also describe the community of users from which the learning object is derived (Recker and Wiley 2001). Epistemology, pedagogical philosophy and ontologies
This section will change the focus from pedagogical metadata to how domain knowledge (in CANDLE this is the telematics domain) is represented. Reference will be made to three of Reeves' dimensions, Epistemology, Pedagogical Philosophy and User Activity, to highlight the epistemological conflict that has arisen from the use of a database and ontology to structure CANDLE.
In order to establish some coherence in the naming conventions employed in describing the content of learning objects, and to allow for greater flexibility in the on-the-fly delivery of courseware (the utilization phase of the lifecycle) CANDLE is in the process of developing an ontology of telematics. An ontology can be described as a "coherent set of terms that characterizes the conceptual commitments and representational choices for modeling a body of knowledge" (Suthers 1997). Although this definition has moved away from hard objectivism, the academic pedigree of ontologies is firmly rooted in that tradition, signifying as it does, the study of what is. It is also worth highlighting the fact that ontologies are usually built around entities, and that the very act of conceptualising the items that make up CANDLE as learning objects point also point to a positivist philosophical epistemology.
The tool being used to develop the telematics domain ontology is OntoEdit by AIFB, and it is worth examining it to uncover some of the assumptions on which it rests. First of all, the ontology is "constructed and maintained in a collaborative effort of domain experts, end-users and IT specialists" (AIFB, 2000). In the case of CANDLE, this consists of a group of telematics lecturers and an ontology engineer. The main elements of the ontology are the restricted set of terms of the domain lexicon with added relations (see Figure 5 for an example of a relation) and axioms ("used to model sentences that are always true") built onto these. All of these are structured into a concept hierarchy. Once built, an ontology provides a static representation of knowledge.
Figure 3: An example of relations made explicit in OntoEdit.
Here we come up against the first real pedagogical conflict: whilst it is easy to see that an ontology resonates well with the objectivist pole of the epistemological dimension, with its emphasis on the independent existence of knowledge, it is much more difficult to see how it can be reconciled with the more constructivist pole, which eschews the whole notion of knowledge separate from knowers. For the very process of objectifying knowledge in the form of an ontology subtly shifts the epistemological balance in such a way that what is, in essence, a representation of the current beliefs of a specific group of knowers, in this case the telematics experts, takes on a far more positivist hue, which is further enhanced by the essentially rationalist character that is attributed to computer and database technology (Chandler 1990).
CANDLE sought to resolve this by softening the epistemological rigidity inherent in the use of an ontology through the use of two naming conventions. The first is tied to the ontology and replicates its hierarchical structure with a restricted set of nested concept descriptors. The second naming convention is not in fact a convention at all: CANDLE provides two additional metadata tags, keywords and weighting, which together allow system users to break out of the preconceived domain ontology. The keywords are entirely free and 'unconventional' in the most literal sense of the word: the metadata taggers are free to use whatever words they feel most appropriately describe their courseware objects. In addition, they are offered the option of weighting these objects in terms of their relevance to the more conventional ontological descriptors of the first naming convention. The courseware objects can then be organised not only according to a static ontology, but using fuzzy logic semantic search algorithms which in principle embody a much more fluid and dynamic conception of knowledge.
Like most of the solutions in this paper, this is a practical, rather than a logical solution, and as such it is only partial. It does not really solve the problem of providing students with multiple pathways through learning objects, which is essential in a truly flexible system. Whilst fuzzy logic searches might provide routes through material determined by how previous system users (either learners or tutors) have navigated the database, and ontologies might offer the 'accredited' steps to an understanding of a particular domain, until there is a way to link these diverse pathways with learner characteristics, the system will not be truly flexible.
Very closely linked to Reeves' first dimension, Epistemology, are Pedagogical Philosophy and User Activity. Reeves sees the much debated conflict between instructivist and constructivist approaches to teaching and learning as a dimension of pedagogical philosophy. At one pole of the dimension lies a strict instructivist position, which holds that objectives exist apart from the learner, and are drawn from a pre-specified domain of knowledge, typically constructed by acknowledged experts. Once specified, these goals can be structured into learning hierarchies progressing from lower to higher order learning through which learners need to move sequentially. The other pole of the dimension represents a radical constructivist position, where learners construct their own knowledge so that neither content nor learner paths through that content can be pre-specified.
Here too, there is a neat dove-tailing of the domain ontology and instructivism, and a corresponding conflict with constructivism. The softening of the epistemological stance referred to above is not sufficient to allow for the more generative learning activities that are axiomatic to the latter paradigm. To be true to the spirit of the constructivist pole of the Pedagogical Philosophy dimension, all users of the system should be able to contribute to the construction of knowledge, including the ontology. Thus learners, as well as acknowledged experts and tutors, should be able to enter courseware into the database and tag is as they see fit. Herein lies the second pedagogical conflict: how can you ensure the integrity of an ontology which so neatly embodies the instructivist paradigm whilst allowing learners to take part in truly generative activities, which implies that they too should be able to contribute to and construct in some meaningful way the knowledge base (including the ontology) that is represented in the system? One possible solution, which is to allow learners to contribute materials and to use free keywords to tag those materials only goes part of the way to solving the problem: until they are able to change the ontology which itself structures, and arguably constitutes the backbone of the knowledge domain, the activities they engage in will not be truly generative. This is one problem that has not been adequately solved in CANDLE; arguably it is not possible to provide for truly generative user activities in a system structured around a static ontology.
Another aspect of the Pedagogical Philosophy dimension needs to be foregrounded: in an instructivist position, knowledge can be broken down and structured into a hierarchy, where learning typically consists of moving sequentially from the smaller, lower order blocks of material to the higher, more complicated levels of learning. This disaggregation of knowledge into separate parcels accords with the whole rhetoric of courseware re-engineering and tagging, where the smaller the object, the more usable and flexible the system.
Granularity essentially refers to the size of a learning object and is one of the key issues that the metadata tagger needs to confront. How far should she separate her learning materials into their constituent parts? Should she break down a lecture into separate Powerpoint slides or should she insert the course as a complete unit into the database? Again, the answer to this turns on pedagogical issues.
The instructivist pole of pedagogical philosophy can easily accommodate the disaggregation of learning materials into smaller parcels of knowledge ? in effect, it invites it. However, not all pedagogies do. I will take a short excursion into the literature on conceptions of learning in higher education to illustrate this. In particular I would like to refer to Marton & Dallalba's six conceptions of learning (Marton, Dall'Alba et al. 1993). Marton & Dallalba interviewed undergraduates at the Open University, and, based on these interviews, they identified six qualitatively different conceptions of learning, with different impacts on student learning. The lowest three conceptions of learning embraced a conception of learning as first the collection, then the reproduction and finally the use of ready-made pieces of knowledge. As students advanced in their studies, they moved away from this fragmented view of knowledge. Similarly Dowling (Dowling 1998) illustrates how the presentation of knowledge as disaggregated parts contributes to a partial, fragmented conception of any domain which excludes the learner from full participation in the practices of the higher levels of academe. It is appropriate to ask whether the very fact of parcelling knowledge, which lies at the heart of the learning objects project, is contributing to such a partial conception of knowledge.
The way CANDLE has solved this problem is to give control over the issue of granularity to the contributor. This allows for courses to be entered into the database as a whole unit if the instructor feels that the disaggregation of the material is not logical or feasible.
The case of assessment in general and multiple-choice questions in particular can again be brought in here a an example of a situation in which disaggregation is not perhaps feasible and, certainly, runs counter to some pedagogical perspectives. It has long been known that assessment is one of the most powerful drivers of student learning. Broadfoot argues that:
assessment has always been, and will probably continue to be, the single most significant influence on the quality and shape of students' educational experience and hence on their learning. Since the earliest days of mass educational provision, assessment procedures have largely governed the content of the curriculum, the way in which schools are organised, the approach to teaching and the learning priorities of students (Broadfoot 1996)
This being the case, it becomes difficult to argue that the disaggregation of the assessment from the rest of the course is a pedagogically defensible process. Furthermore, Resnick & Resnick, arguing from a situated cognition perspective, voiced powerful criticisms of objective testing, such as the multiple-choice tests under discussion here, because of the assumptions of decomposability and decontextualisation of knowledge inherent in it. Within the learning object project, these are not merely assumptions, but constitute instead central, structuring commitments (Resnick and Resnick 1992) .
There is a related concern about granularity and instructional design in current metadata literature. There appears to be a move away from the conceptualisation of granularity as mainly an issue of the size of a learning object to a more "robust" view of it which considers the manner and extent to which a number of factors which together make up the instructional design process are combined: "?in determining the robust granularity of a learning object, one might ask, "what elements of the model, message, instructional strategy, representation, and media-logic layers are compressed within this learning object?" The larger the count, the larger the grain size of the learning object" (Wiley, Gibbons et al. 2000). This is an important theoretical move as it foregrounds the purposive activity that is instructional design and signals that simplistic views of teaching as the delivery of disaggregated learning objects joined together in some straightforward combination hide the very real complexity of the development of learning materials which in effect takes place in a number of interconnected layers.
Three main themes can be seen as emerging in the above discussion of the experiences of those involved in the design of the CANDLE system. The first is a tension between the privileging of the ontology of a small group of experts in the design of the system necessary to ensure that it offers an accepted, coherent representation of a domain and the generative requirements of the more constructivist pedagogies. The second suggests that the disaggregation of any domain, or indeed of teaching materials used in that domain, into separate pieces does violence both to conceptions of knowledge as essentially articulated and to the marked complexity of the instructional design process. The third, and possibly most important, theme to emerge is that instructional activities are essentially situated, and that de-coupling the elements which make up these activities from each other and from the context in which they occur strips them of meaning.
Although the above points have emerged in the process of designing the CANDLE system, the issues faced will apply, in varying degrees to the development of all educational systems, including libraries, portal sites and automated online courses. If these systems, including, of course those based around the technology of learning objects, are to live up to their much-touted promise, designers will have to address these, and many other pedagogical issues head-on, recognising that the easiest solutions to implement probably involve a simplification and de-contextualisation of pedagogical theory which is not easy to defend. The alternative is that the systems designed will offer simplified, water-down versions of the pedagogical process which quite simply fail.
A few of the more important pedagogical issues that have arisen as a consequence of the decision to implement multiple pedagogies in CANDLE have been briefly sketched out. There are many more which deserve attention, including the provision of multiple pedagogical templates to assist instructors in the creation of courseware from objects in the database, and the development of sequencing algorithms for the on-the-fly delivery of courses, however space does not permit their examination here. Nonetheless, it is hoped that these examples have given some flavour of the very real contradictions arising from the attempts to design a system around conflicting pedagogical models. As mentioned above, the solutions adopted in CANDLE are practical rather than logical, for in many cases the contradictions are so intractable as to obviate logical resolution. Purists might argue that in view of this it is theoretically impossible to implement multiple pedagogies in a single system, a position that deserves some credit.
An alternative perspective on this issue is possible, and probably more constructive. This would argue that like all theories and paradigms, learning theories are not perfect, but should be viewed merely as guidelines (Snelbecker 1999). As Popper notes "All theories are trials; they are tentative hypotheses, tried out to see whether they work; and all experimental corroboration is simply the result of tests undertaken in a critical spirit, in an attempt to find out where our theories err" (Popper 1957). An equally cogent argument can be found in Schön's work on the reflective practitioner (Schön 1987), which will be used as an appropriate way close this paper which set out with the aim of stimulating in some small way a necessary debate on the feasibility of implementation of true pedagogical flexibility.
In the varied topology of professional practice, there is a high, hard ground overlooking a swamp. On the high ground, manageable problems lend themselves to solution through the application of research-based theory and technique. In the swampy lowland, messy, confusing problems defy technical solution. The irony of the situation is that the problems of the high ground tend to be relatively unimportant to individuals or society at large, however great their technical interest may be, while in the swamp lie the problems of greatest human concern. The practitioner must choose. Shall he remain on the high ground where he can solve relatively unimportant problems according to prevailing standards of rigor, or shall he descend to the swamp of important problems and nonrigorous inquiry?
AIFB (2000). OntoEdit, Knowledge Management Group, University of Karlsruhe. Available online at http://ontoserver.aifb.uni-karlsruhe.de/ontoedit/guide/users_guide.html (last visited, March 2002) [cited]
Althauser, R. and J. Matuga (1998). On the pedagogy of electronic instruction. Electronic Collaborators Learner-centred technologies for Literacy, Apprencieship and Discourse. K. K. Curtis Bonk. Mahwah New Jersey, Lawrence Erlbaum Associates. [cited]
ARIADNE (1999). ARIADNE Educational Metadata Recommendation. EPFL (Lausanne, Switzerland), KUL (Leuven, Belgium) and the ARIADNE project. Available online at http://ariadne.unil.ch/Metadata/ariadne_metadata_v3final1.htm (Last visited, March 2002) [cited]
ASTER (1999). Report 2: A survey of current practice in the use of C&IT to support small-group teaching activities in disciplines associated with the Humanities, Physics and Psychology. Available online at http://cti-psy.york.ac.uk/aster/report2.pdf (Last visited, March 2002) [cited]
Baca, M. (2000). Introduction to metadata: pathways to digital information. 2001. Available online at http://www.getty.edu/research/institute/standards/intrometadata/index.html. (Last visited March 2002) [cited] [cited]
Baker, M., T. Hansen, et al. (1999). The Role of Grounding in Collaborative Learning Tasks. Collaborative Learning: Cognitive and computational approaches. P. Dillenbourg. Amsterdam, Pergamon. [cited]
Bannan-Ritland, B., N. Dabbagh, et al. (2000). Learning Object Systems as Constructivist Learning Environments: Related Assumptions, Theories and Applications. The Instructional Use of Learning Objects: Online Version. D. A. Wiley. Available online at http://reusability.org/read/chapters/ (Last accessed, March 2002) [cited] [cited] [cited]
Becher, T. (1989). Academic Tribes and Territories: Intellectual enquiry and the cultures of disciplines. Milton Keynes & Bristol, PA, SHRE and Open University Press. [cited]
Bruner, J. (1996). The Culture of Education. Cambridge, Massachusetts & London, England, Havard University Press. [cited]
Collis, B., J. Vingerhoets, et al. (1997). "Flexibility as a key construct in European training: experiences from the Telescopia Project." British Journal of Educational Technology 28(3): 199-217. [cited] [cited]
Connolly, T. and B. K. Thorn (1991). Discretionary Data bases: Theory, Data, and Implications. Organisations and Communication Technology. J. Fulk and C. Steinfield. Newbury Park, CA, Sage. [cited]
Dowling, P. (1998). Sociology of mathematics education: mathematical myths, pedagogical texts. London, Falmer Press. [cited]
Edwards, R. (1997). Changing Places? Flexibility, Lifelong Learning and a Learning Society. London, Routledge. [cited]
Edwards, R., K. Nicoll, et al. (1999). "Migrating metaphors: the globalization of flexibility in policy." Journal of Education Policy 14(6): 619-630. [cited]
Frederiksen, J. and A. Collins (1989). "A systems approach to educational testing." Educational Researcher 18(9): 27-31. [cited]
Fullan, M. G. (1992). Successful school improvement and the implementation perspective. Successful School Improvement. Buckingham, Open University Press. [cited]
Gee, J. P. (1997). The situated sociocultural mind. Situated Cognition. D. Kirshner and J. A. Whitson. Mahwah, New Jersey, Lawrence Erlbaum. [cited]
Green, A., A. Wolf, et al. (1999). Convergence and Divergence in European Education and Training systems. London, Institute of Education. [cited]
Green, D. (1995). What is Quality in Higher Education? Concepts, Policy and Practice. What is Quality in Higher Education? D. Green. Buckingham, SRHE and Open University Press. [cited]
IMS (2001). IMS Learning Resource Meta-data Information Model . Version 1.1 - Final Specification., IMS Global Learning Consortium, Inc (http://www.imsproject.org/; last visited, March 2002). [cited] [cited]
Jarret, W., Mendes, E., et al. (in preparation). Preliminary Metadata, University College London: Report Number WP 2 - D1 -Int.
Johnston, R. (1999). "Beyond Flexibility: issues and implications for higher education." Higher Education Review 32(1): 55 - 66. [cited]
Jonassen, D. H. and Land, S. M., (2000). Eds. Theoretical Foundations of Learning Environments. Mahwah, New Jersey, Lawrence Erlbarum Associates. [cited]
Jonassen, D. H. and L. Rohrer-Murphy (1999). "Activity Theory as a Framework for Designing Constructivist Learning Environments." Educational Technology: Research and Development 47(1): 61-79. [cited]
Kewell, B., S. Hopkinson, et al. (1999). A new kind of knowledge transfer? A report into the feasibility of embedding Learning Technologies within the SME market for vocational training. Cascade report no. 1, London, University of North. [cited]
Leflore, D. (2000). Theory Supporting Design Guidelines for Web-Based Instruction. Instructional and Cognitive Impacts of Web-Based Education. B. Abbey. Hershey USA and London UK, Idea Group Publishing. [cited]
Marton, F., G. Dall'Alba, et al. (1993). "Conceptions of learning." International Journal of Educational Research 19(3): 277-300. [cited]
Moran, L. and B. Myringer (1999). Flexible learning and university change. Higher Education Through Open and Distance Learning: World review of distance education and open learning. K. Harry. London and New York, Routledge. 157-171. [cited] [cited]
Norman, N. (1997). "Communication technologies and education: lessons in the potential of innovation." Journal of Advanced Learning Technologies 5(3): 43-53. [cited]
Pantin, C. F. A. (1968). The Relations Between the Sciences. Cambridge, Cambridge University Press. [cited]
Popper, K. R. (1957). The Poverty of Historicism. Boston, Beacon Press. [cited]
Recker, M. M. and D. A. Wiley (2001). "A non-authoritative educational metadata ontology for filtering and recommending learning objects." Journal of Interactive Learning Environments: Special edition on metadata).Available online at http://wiley.ed.usu.edu/writings2.pl (Last visited, March 2002) [cited]
Reeves, T. (2000). Evaluating What Really Matters in Computer-Based Education, Dr. Tom Reeves. 2001. Available online at http://www.educationau.edu.au/archives/cp/reeves.htm (Last visited, March 2002) [cited]
Reigeluth, C. (1999). What is Instructional-Design Theory and How is it Changing? Instructional Design Theories and Models. C. Reigeluth. Mahwah, New Jersey, Lawrence Erlbaum Associates. II: 5 - 29. [cited] [cited] [cited]
Resnick, L. B. and D. P. Resnick (1992). Assessing the Thinking Curriculum: New Tools for Educational Reform,. Changing Assessments: Alternative Views of Aptitude Achievement and Instruction. B. Gifford and O. C. M. Boston. Boston, Dordrecht, London, Kluwer Academic Publishers. [cited]
Schatz, S. (2001). An Introduction to Meta Tags and Knowledge Bits, IMS Global Learning Consortium, Inc. Available online at http://www.imsproject.org/feature/kb/knowledgebits.html (Last visited March 2002) [cited]
Schön, D. A. (1987). Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions. San Francisco, CA, Jossey-Bass Inc. [cited]
Snelbecker, G. E. (1999). Some Thoughts About Theories, Perfection and Instruction. Instructional-Design Theories and Models. C. Reigeluth. Mahwah, New Jersey and London, Lawrence Erlbaum. [cited]
Somekh, B. and N. Davies (1991). "Towards a pedagogy for information technology." Curriculum Journal 2(2): 153-170. [cited]
Suthers, D. (1997). Working Group: Shared Vocabularies for Representing Pedagogical Knowledge. 2001. [cited]
Tenner, E. (1997). Why Things Bite Back: Technology and the Revenge of Unintended Consequences. New York, Vintage Books. [cited]
Underwood, J. and J. Brown, (1997). Eds. Integrated learning systems: potential into practice. Oxford, Heinemann. [cited]
Walkerdine, V. (1998). Developmental psychology and the child-centred pedagogy: the insertion of Piaget into early Education. Changing the Subject. Psychology, Social Regulation and Subjectivity. J. Henriques. London, New York, Routledge: 153-202. [cited]
Watkins, C. and P. Mortimore (1999). Pedagogy: What Do We Know? Understanding Pedagogy and its impact on learning. P. Mortimore. London, Paul Chapman Publishing Limited. [cited]
Wenger, E. (1998). Communities of practice: learning, meaning and identity. New York, Cambridge University Press. [cited]
Wiley, D., A. S. Gibbons, et al. (2000). A reformulation of the issue of learning object granularity and its implications for the design of learning objects. 2001. [cited]
Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. The Instructional Use of Learning Objects: Online Version. D. A. Wiley. [cited] [cited]
See Schatz, S. (2001). An Introduction to Meta Tags and Knowledge Bits, IMS Global Learning Consortium, Inc., or Baca, M. (2000). Introduction to metadata: pathways to digital information. 2001 for an introduction, or Greenberg, J., Ed. (2001). Metadata and organizing educational resources on the Internet. New York, Haworth Information Press, for a fuller treatment.
This paragraph was added after a discussion about how assessment would be dealt with in CANDLE.