Well, here I was thinking about how all I ever do was the formal/summative evaluation in my teaching, focusing on the assessment of student learning, when it all hit me (right between the eyes it seemed!) that I do complete much of the evaluation outlined in the readings.
The CIPP model, as I have said recently, now not only seems to be the most relevent, but actually is the most accurate description of what I do! Up until now, I was thinking that it was the model that most represented what I should be doing.
This unit of evaluation seems to have taken the most brain energy to get through, despite having the least number of readings. The limited number has maybe encouraged me to think deeper rather than just take in as much as I can.
In this process, the evaluation of my learning in this course is improving! Very exciting!
Signing off, ready to hand in assignment Discussion 3!
Thursday, May 10, 2007
Tuesday, May 8, 2007
Evaluation of Flexible Programs - Module 4
Having now read through the entire module 4 and through some of the checklists, it is now clear to me the whole process of evaluation and the full extent of the different processes and the procedures that it can inform.
Whilst reading through Eseryel as shown in the previous post, Shufflebeam's CIPP (Conext, Input, Process, Product) model came out as the most appropriate to my context in my mind. It seemed the most appopriate to my situation in catering for online/flexible delivery and the face to face classroom.
But as read through the notes of module 4, the depth of the evaluation becomes more apparent. The 4 areas of evaluation are:
Whilst reading through Eseryel as shown in the previous post, Shufflebeam's CIPP (Conext, Input, Process, Product) model came out as the most appropriate to my context in my mind. It seemed the most appopriate to my situation in catering for online/flexible delivery and the face to face classroom.
But as read through the notes of module 4, the depth of the evaluation becomes more apparent. The 4 areas of evaluation are:
- Context
- Input
- Process and
- Product
However, as I now know, these are not just the different types of evaluation that can take place, but these indicate 4 completely different areas, and line up well (it seems to me) to the ADDIE method of instructional design as each area focuses and infroms the different areas of ID using the ADDIE method.
It certainly seems exhaustive, so will keep reading and thinking about it's implementation in my context.
Monday, May 7, 2007
Approaches to Evaluation of Training
In this article, (found at http://www.ifets.info/journals/5_2/eseryel.html) it outline show evaluation is a complex task, and often not undertaken due to the complexity or lack of experience/skills.
The 4 purposes:
Evaluation of student learning
evaluation of instructional materials
transfer of training
return on investment
Were interesting as it seems that only the first 2 were relevent for the situation that I have chosen.
When looking at the systems approaches, CIPP (1987) is the most useful one in my situation:
Context: obtaining information about the situation to decide on educational needs and to establish program objectives
Input: identifying educational strategies most likely to achieve the desired result
Process: assessing the implementation of the educational program
Product: gathering information regarding the results of the educational intervention to interpret its worth and merit
However, as he suggests, they do not address the collaborative process of evaluation.
Eseryel also suggests that systematic and planned evaluation was generally not found in practice, nor was there a distinction between formative evaluation and summative evaluation. The most common type of evaluation is of the student performance, in the form of assessment (which is a heavy focus in our school and educational setting) and not enough on the reviewing the design of the instruction based on the results of the assessment. Much of the focus is on the learner, not the course design.
In my experience and in discussing this issue with a number of staff, it seems that some staff are aware of inadequacies of the instructional design, but are also equally aware of the weakness of the students, and only use assessment to fromally "prove" that their judgements are correct.
It has also recently arisen the plan to modify the assessment to suit the final outcome, as Eseryel suggests - the bias for internal evaluators (and in this situation - the assessers) the bias may have a very positive effect on course uptake - even at the school level.
One staff member has said to another: "Mark this task easily so that we don't turn off the students from choosing this subject!"
Similarly, evaluation tools are limited, thereaction sheets are adding to the failure of evluation in training scenarios.
My thoughts about the parts of this article worth mentioning.
The 4 purposes:
Evaluation of student learning
evaluation of instructional materials
transfer of training
return on investment
Were interesting as it seems that only the first 2 were relevent for the situation that I have chosen.
When looking at the systems approaches, CIPP (1987) is the most useful one in my situation:
Context: obtaining information about the situation to decide on educational needs and to establish program objectives
Input: identifying educational strategies most likely to achieve the desired result
Process: assessing the implementation of the educational program
Product: gathering information regarding the results of the educational intervention to interpret its worth and merit
However, as he suggests, they do not address the collaborative process of evaluation.
Eseryel also suggests that systematic and planned evaluation was generally not found in practice, nor was there a distinction between formative evaluation and summative evaluation. The most common type of evaluation is of the student performance, in the form of assessment (which is a heavy focus in our school and educational setting) and not enough on the reviewing the design of the instruction based on the results of the assessment. Much of the focus is on the learner, not the course design.
In my experience and in discussing this issue with a number of staff, it seems that some staff are aware of inadequacies of the instructional design, but are also equally aware of the weakness of the students, and only use assessment to fromally "prove" that their judgements are correct.
It has also recently arisen the plan to modify the assessment to suit the final outcome, as Eseryel suggests - the bias for internal evaluators (and in this situation - the assessers) the bias may have a very positive effect on course uptake - even at the school level.
One staff member has said to another: "Mark this task easily so that we don't turn off the students from choosing this subject!"
Similarly, evaluation tools are limited, thereaction sheets are adding to the failure of evluation in training scenarios.
My thoughts about the parts of this article worth mentioning.
Friday, May 4, 2007
Models of Evaluation
I have started Module 4 about Evaluation, and I am a little confused...the 3 models suggested are not clear to me.
Part of the problem I see here, is that I am planning a unit that includes online components, but is not totally online. So there are some distinctions that I need to make, and maybe the simplest is to focus the work of my project/unit of work on the components that will be solely online.
The naturalistic evaluation seems like it would be more difficult to method to utilise in the online environment, whereas it is one of the simplest when working face to face with students.
I will need to think hard about these models and my context that I am applying this course to.
Part of the problem I see here, is that I am planning a unit that includes online components, but is not totally online. So there are some distinctions that I need to make, and maybe the simplest is to focus the work of my project/unit of work on the components that will be solely online.
The naturalistic evaluation seems like it would be more difficult to method to utilise in the online environment, whereas it is one of the simplest when working face to face with students.
I will need to think hard about these models and my context that I am applying this course to.
Where am I at now???
Well I have not read for a little while as I have been stewing over the whole instructional design concept and thinking through all of the articles that I have read so far. I keep coming back to this blog and am appreciating the work that I put in early. (The concept of the blog works well for me, I am keen to continue it!)
I am still struggling with the difference between the design and the development of the course. The assignments have been pushing design, and not Development, but the comments from Shirley (facilitator) suggest that I am blurring the lines here. I need to refocus on design.
My thoughts are that there are 2 key areas for this course:
1. Choosing an instructional design model
2. Choosing a model for sequencing.
I am not confident this is the case, but for my task this seems to suit the situation. My application of this is:
1. Follow the ADDIE model of instructional design (Intulogy)
2. Utilise the Simplifying Conditions Method of sequencing (Reigeluth)
I particularly like in the ADDIE method, the flowchart outline and the detail provided in the clear, but numerous steps. It forces me to think about all of the concepts, and address the relevent ones, with the flowchart keeping the big picture in focus.
The SCM of sequencing, gives a title and structure for a method of sequencing that I have been using regularly in class for certain topics - that has provedn very effective.
My weakness is the assessment and evaluation. I am now looking forward to Module 4 and evaluation.
I am still struggling with the difference between the design and the development of the course. The assignments have been pushing design, and not Development, but the comments from Shirley (facilitator) suggest that I am blurring the lines here. I need to refocus on design.
My thoughts are that there are 2 key areas for this course:
1. Choosing an instructional design model
2. Choosing a model for sequencing.
I am not confident this is the case, but for my task this seems to suit the situation. My application of this is:
1. Follow the ADDIE model of instructional design (Intulogy)
2. Utilise the Simplifying Conditions Method of sequencing (Reigeluth)
I particularly like in the ADDIE method, the flowchart outline and the detail provided in the clear, but numerous steps. It forces me to think about all of the concepts, and address the relevent ones, with the flowchart keeping the big picture in focus.
The SCM of sequencing, gives a title and structure for a method of sequencing that I have been using regularly in class for certain topics - that has provedn very effective.
My weakness is the assessment and evaluation. I am now looking forward to Module 4 and evaluation.
Tuesday, April 24, 2007
The Elaboration Theory: Guidance for Scope and Sequence Decisions" Charles M. Reigeluth
This article has enabled a formalisation of my initial model of instructional design...my model did outline a learning centred model which had little or no formal basis, except through experience in the classroom and success in a variety of situations. Not all of the courses I am involved with are structured in this manner, however, as I progress through the paper by Reigeluth, I find a stronger and stronger link to my experience as "... the elaboration theory was developed to provide ... a holistic approach to sequencing"
Types of Sequencing Strategies - Relationships.
I found it useful to have some direction to the topic of sequence and the models offered, summarised below:
Historical: chronological relationship - E.g. Timelines in history
Procedural: order of performance - E.g. Software development cycle
Hierarchical: the various skills and subskills that comprise a task - E.g. Authoring and Multimedia topics
Simplifying conditions: based upon the degree of complexity of different versions of a complex task - E.g. IPT Intro to Information Systems topic and IST - Digital Media (Video editing) unit
I think the spiral sequencing links well to the Simplifying conditions method and this relates well to my initial model of ID.
Types of Sequencing Strategies - Relationships.
I found it useful to have some direction to the topic of sequence and the models offered, summarised below:
Historical: chronological relationship - E.g. Timelines in history
Procedural: order of performance - E.g. Software development cycle
Hierarchical: the various skills and subskills that comprise a task - E.g. Authoring and Multimedia topics
Simplifying conditions: based upon the degree of complexity of different versions of a complex task - E.g. IPT Intro to Information Systems topic and IST - Digital Media (Video editing) unit
I think the spiral sequencing links well to the Simplifying conditions method and this relates well to my initial model of ID.
Comments from Siemens "Learning Development Cycle: Bridging learning Design and Modern Knowledge needs
Here are some quotes and my thoughts on the Siemens Article:
Quote 1:
"Learning design is primarily about creating guideposts, not describing how to walk on a particular path. The best that a well designed course, workshop, or work-integrated learning
resource can offer is the climate in which a learner can choose to learn."
I agree, you cannot force a learner to learn. However, you can create a positive learning environment (positive in the sense that the students will want to learn through intrinsic motivation to learn the content) and you can create a negative learning environement where stduents will learn because they are required to. The latter is a questionable about whether students learn, or just form an ability to reproduce or know or "learn enough to pass the assessment".
Quote 2:
"Designing courses requires set steps and guidelines for instructors and learners to follow. Learning design, in contrast, is concerned with more than simply creating courses. Instead, the intent is to create the constructs within which learning will occur - networks and ecology."
Again, this comment highlights the differences in learning strategies, through identifying the differences between setting courses and designing instruction.
Quote 3:
"Bridging prior learning with academic standards requires...that learners verify stated learning through a variety of sources and means."
Again, linking this back to my initial definition of Instructional design, this fits as I try to verify students learning through multiple real life scenarios and problems.
Quote 4:
" Learner-centred design focuses on giving the learner the ability to decide what he/she feels is important and relevant."
But how can I do this? How do I cater for the student who is not motivated to learn, or not motivated to learn in this particular course?
Quotes 5:
"Research (neuroscience) is beginning to indicate that the primary learning component of our
brains is pattern recognition, not information processing. ...Replacing the causal model of learning (need highlighted, instructional intervention planned, measurement enacted) with “network phenomenon”:" and
"...distributed representation has a profound implication for pedagogy, as it suggests that learning (and teaching, such as it is) is not a process of communication, but rather, a process of immersion.”
This begs the question of assessment, how does one tie this all in? Immersion is fine, but I struggle with assessment of the immersed topic...have I come at it from the wrong (or different) perspective? Is problem based learning a solution to this? In which case, again, my initial definition is still intact!
The following quote again from this article suggests that this is the case:
"...learning can simply be defined as actuated or actionable knowledge. This definition has two components - knowledge: understanding of an implicit or explicit nature, and actuation: doing
something appropriate (defined as contextually aware) with knowledge."
Quote 6:
Not so much a quote, but a response to Siemens' "Learning Domains" of accretion, transmission, aquisition and emergence...
If I have the concept right, one method of implementing this is to set a project or task or set the outcomes, the students can decide how they are going to demonstrate achievement of this outcome and the teacher provides resources for the students to use. Assessment can be through rubrics. Or is this the domain of accretion only? Should we have a broad spectrum of types of instruction to cover all 4 domains?
I love the idea of the learning network/environment/ecology as an ideal, but this is not always possible...the basics must be taught here...(I see a strong link here to Reigeluth's "Elbaoration Theory")
How do I create a learning ecology for the Robotics unit I am designing? Must think more...
Quote 7:
Conclusion:
"Taking a panoramic view of learning, and accounting for unique facets and domains, equips a designer with numerous approaches and methods. Instead of only transmitting learning, educators begin to create structures and networks that will foster a lifetime of learning and learning skills....The monochromatic world of course design is replaced with a vibrant environment where learning occurs in an integrated ecosystem. Learning is a continuous stream, rather than a dammed up reservoir."
Quote 1:
"Learning design is primarily about creating guideposts, not describing how to walk on a particular path. The best that a well designed course, workshop, or work-integrated learning
resource can offer is the climate in which a learner can choose to learn."
I agree, you cannot force a learner to learn. However, you can create a positive learning environment (positive in the sense that the students will want to learn through intrinsic motivation to learn the content) and you can create a negative learning environement where stduents will learn because they are required to. The latter is a questionable about whether students learn, or just form an ability to reproduce or know or "learn enough to pass the assessment".
Quote 2:
"Designing courses requires set steps and guidelines for instructors and learners to follow. Learning design, in contrast, is concerned with more than simply creating courses. Instead, the intent is to create the constructs within which learning will occur - networks and ecology."
Again, this comment highlights the differences in learning strategies, through identifying the differences between setting courses and designing instruction.
Quote 3:
"Bridging prior learning with academic standards requires...that learners verify stated learning through a variety of sources and means."
Again, linking this back to my initial definition of Instructional design, this fits as I try to verify students learning through multiple real life scenarios and problems.
Quote 4:
" Learner-centred design focuses on giving the learner the ability to decide what he/she feels is important and relevant."
But how can I do this? How do I cater for the student who is not motivated to learn, or not motivated to learn in this particular course?
Quotes 5:
"Research (neuroscience) is beginning to indicate that the primary learning component of our
brains is pattern recognition, not information processing. ...Replacing the causal model of learning (need highlighted, instructional intervention planned, measurement enacted) with “network phenomenon”:" and
"...distributed representation has a profound implication for pedagogy, as it suggests that learning (and teaching, such as it is) is not a process of communication, but rather, a process of immersion.”
This begs the question of assessment, how does one tie this all in? Immersion is fine, but I struggle with assessment of the immersed topic...have I come at it from the wrong (or different) perspective? Is problem based learning a solution to this? In which case, again, my initial definition is still intact!
The following quote again from this article suggests that this is the case:
"...learning can simply be defined as actuated or actionable knowledge. This definition has two components - knowledge: understanding of an implicit or explicit nature, and actuation: doing
something appropriate (defined as contextually aware) with knowledge."
Quote 6:
Not so much a quote, but a response to Siemens' "Learning Domains" of accretion, transmission, aquisition and emergence...
If I have the concept right, one method of implementing this is to set a project or task or set the outcomes, the students can decide how they are going to demonstrate achievement of this outcome and the teacher provides resources for the students to use. Assessment can be through rubrics. Or is this the domain of accretion only? Should we have a broad spectrum of types of instruction to cover all 4 domains?
I love the idea of the learning network/environment/ecology as an ideal, but this is not always possible...the basics must be taught here...(I see a strong link here to Reigeluth's "Elbaoration Theory")
How do I create a learning ecology for the Robotics unit I am designing? Must think more...
Quote 7:
Conclusion:
"Taking a panoramic view of learning, and accounting for unique facets and domains, equips a designer with numerous approaches and methods. Instead of only transmitting learning, educators begin to create structures and networks that will foster a lifetime of learning and learning skills....The monochromatic world of course design is replaced with a vibrant environment where learning occurs in an integrated ecosystem. Learning is a continuous stream, rather than a dammed up reservoir."
Subscribe to:
Comments (Atom)