Research Methods – The Door

STUDY TITLE:

The Impact of a PBL-based Course Redesign on Post-secondary Learners

Scott J. Warren, Ph.D.

Cliff Whitworth, Ph.D.

University of North Texas

CECS 1100 Course for the Quality Enhancement Program

The problem of current undergraduate education

“Many students graduate having accumulated whatever number of courses is required, but still lacking a coherent body of knowledge or any inkling as to how one sort of information might relate to others. And all too often they graduate without knowing how to think logically, write clearly, or speak coherently…(t)he university has given them too little that will be of real value beyond a credential that will help them get their first jobs.”

The Boyer Commission on Educating Undergraduates (1995)

In response to the Boyer Commission report, a major attempt to improve college teaching has come in the form of the Scholarship of Teaching and Learning (SOTL) movement. A resurgence of study regarding the SOTL began with Chickering and Gamson’s article “Seven principles for good practice in undergraduate education” (1987). The principles suggested by these authors for good practice in undergraduate education are:

1. (E)ncourages contact between students and faculty,

2. (D)evelops reciprocity and cooperation among students,

3. (E)ncourages active learning,

4. (G)ives prompt feedback,

5. (E)mphasizes time on task,

6. (C)ommunicates high expectations, and

7. (R)espects diverse talents and ways of learning.
(Chickering & Gamson, 1987) (p. 1)

In addition, Chickering and Gamson suggest that there are five powerful forces that may impact education: (1.) activity, (2.) expectations, (3.) cooperation, (4.) interaction, (5.) diversity, and (6.) responsibility (1987)(p.2). These forces must be coupled with a supportive environment in order for substantial effects to be felt in the educational community. Table A1 includes the attributes of an environment supportive of good teaching practices.

Table A1.

Also, stemming from attempts to engage in research related to his own post-secondary teaching, Randy Bass (1999) suggests that the process of teaching should include the investigation of problems within the context of courses as implicit in both the teaching and research missions of faculty. He also states that teaching should be viewed not as an activity, but a process made up of vision, design, interactions, outcomes, and analysis. As such, this view also values scientific inquiry into ones’ own teaching practices as an integral part of the process of teaching. The questions instructors develop from the process of teaching should inform the direction of research and need not be directly tied to their own disciplines. More recent research regarding SOTL has been conducted by Cottrell and Jones (2003) in the form of case studies and qualitative analyses regarding the following questions:

1. What influenced instructors to implement the scholarship of teaching and learning?

2. What learning outcomes did instructors expect their students to realize?

3. What learning approaches did instructors implement in their course designs?

4. How did instructors assess whether course revisions and teaching practices are making a difference in student learning and development?

5. How did instructors use the assessment results to improve the courses? (p. 171)

Cottrell and Jones also found that there were eight significant influences that caused faculty to implement the scholarship of teaching and learning. These influences may be found in table A2.

Table A2.

Factors influencing the adoption of scholarship and teaching

Influences	N	%
Internal influences
Personal philosophy of teaching	37	79
Frustration with student learning	7	15
Organizational influences
Support from administration	12	26
External influences
CASTL	6	13
Teaching conferences	4	9
Accreditation organizations	2	4

From Cottrell & Jones, 2003, p. 174

While there were a substantial number of influences, the instructor’s personal philosophy of teaching, frustration with student learning, and support from administration were found to have the largest impact on their decision to engage in scholarship of teaching and learning. Another notable finding was that “an analysis of the data revealed that all of the participating instructors realized that the most essential part of the scholarship of teaching and learning is using the assessment results to make deliberate changes in the course to maximize student learning and development” (Cottrell & Jones, 2003)(p.178). It would therefore follow that studying the active engagement of an instructor in the change process of curriculum and teaching methods within an ongoing course would be a logical area for further study.

Related to this finding is Shulman’s (1998) assertion that in order for teaching to become a form of scholarship, it must educate faculty. The scholarship of teaching and learning must be made public, subject to critical review and evaluation, and accessible so that scholars can exchange and build upon previous work. This assertion is central to the usefulness of the SOTL movement and carries with it many implications for future research in the area.

Two areas for future research suggested by Cottrell and Jones are (1.) how do faculty share their findings related to scholarship of teaching and learning and (2.) research may also inquire as to whether scholarship of teaching and learning initiatives are valued and used to make tenure and promotion decisions at public and private universities (Cottrell & Jones, 2003).

Game+Narrative+Distributed Learning

Quote from Neuromancer or other Cyberpunk book

Building on the theory that narrative must be an integral part of learning activities to frame and situate learners with respect to their application of knowledge and skills in future learning activities, this course also integrates the concept that this narrative is most effective when it supports learning activities that have free choice game activities that reveal enabling information and resources that improve learner effectiveness and sense of reward as revealed using emerging and popular online resources that students use as part of their daily lives in a distributed fashion rather than a fully integrated, stand-alone product (i.e. MySpace, blogs, podcasts.)

A larger conception in which the multiple media are hosted all over the place and then reintegrated into a course – i.e. taking links to videos made by the designers and posted in YouTube, music posted to MySpace, Podcasts on different webs sites, multiple blogs, etc. to create game-like structures, learning activities, etc. that better match the reality of the Internet as it evolves. This conception takes advantage of the many free online resources and blends them together to give learners a situated, coherent narrative that carries them through their learning experience, tasks, etc.

One instructional method that shows promise for developing game scenarios for instruction at the post-secondary level is problem-based learning (Albanese & Mitchell, 1993; Diekelmann & Scheckel, 2003; DiPasquale, Mason, & Kolkhorst, 2003; Elder & Paul, 2002; Everett & Zinser, 1998; Keller, 2002; Kolodner, 2002; Willis, 2002; Zembal-Saul, Blumenfeld, & Krajcik, 2000). It has a propensity for improving critical thinking skills (Tiwari & Lai, 2002), improving post-secondary learning experiences by providing authentic tasks (Bonk, Kirkley, Hara, & Denned, 2001), and forces that students engage in story-driven, problem-centered tasks similar to those found in video games (Jonassen, 1999; Jonassen & Hernandez-Serrano, 2002; Scott J. Warren, 2006; S.J. Warren, 2006). Combined with the fact that post-secondary learners tend to be motivated by internal self-esteem, recognition, need for a better quality of life, and self-actualization rather than extrinsic rewards, the learning activities can be packaged in such a way that they mirror real-world tasks while providing additional narrative that has been found to be motivating to learners that require additional feedback, peer support, and motivations that they may not find in more traditional classrooms.

Problem-based learning

Savery and Duffy (1995) suggest a series of instructional design principles based on the PBL vision and orientation. These principles are:

1. Anchor all learning activities to a larger task or problem.

2. Support the learner in developing ownership for the overall problem or task.

3. Design an authentic task.

4. Design the task and the learning environment to reflect the complexity of the environment they should be able to function in at the end of learning.

5. Give the learner ownership of the process used to develop a solution.

6. Design the learning environment to support and challenge the learner's thinking.

7. Encourage testing ideas against alternative views and alternative contexts.

8. Provide opportunity for and support reflection on both the content learned and the learning process. (P.3-6)

When implementing PBL in college courses, the following complimentary suggestions have been made by A. Sandra Willis based on her implementation of PBL in a university psychology course and will guide the design of the course under study:

1. Peer tutoring, used by many PBL advocates may provide advanced (students) opportunities for their own skills-based learning experiences within the framework of an independent study or an undergraduate teaching assistantship.

2. Conversely, students in a PBL course may feel overworked by having to schedule time outside of class to work in groups and by having to complete many written assignments. Plan carefully to provide a significant proportion of class time for group work, especially for the entire group to reach a consensus on answers to the problems.

3. Try to construct problems that are holistic, not divisible, so that all students are exposed to similar content knowledge and opportunities for skills development.

(Willis, 2002)(P.289-290)

Critical Thinking and PBL

From a critical thinking perspective, research has shown that PBL:

“(E)ncourages students to develop thinking skills including the ability to: (i) analyze and synthesize data; (ii) develop hypotheses; (iii) apply deductive reasoning to a problem situation; (iv) draw conclusions after analysis, synthesis and evaluation of new information; (v) synthesize strategies/solutions; and (vi) monitor and evaluate own thinking process.”

(Tiwari & Lai, 2002)(p. 2)

Stemming from existing research evidence, several authors have set out to combine a problem-based learning approach with proven critical thinking strategies originally developed outside of PBL in order to enhance the existing strengths of the approach while providing additional scaffolding for learners new to working in a CLE (Albanese & Mitchell, 1993; Diekelmann & Scheckel, 2003; DiPasquale, Mason, & Kolkhorst, 2003; Elder & Paul, 2002; Everett & Zinser, 1998; Keller, 2002; Kolodner, 2002; Willis, 2002; Zembal-Saul, Blumenfeld, & Krajcik, 2000).

One technique for improving critical thinking has been to require student self assessment so as to encourage metacognition in students in order to revise their thinking and PBL approaches, resulting in improvements in performance (Bransford, Vye, Bateman, Brophy, & Roselli, 2003; Meyerson & Adams, 2003). As a result of this increased metacognition, a problem based learning approach may also stimulate increased transfer as students are able to perform problem solving in a number of new contexts and settings (Lin, Hmelo, & Kinzer, 1999). It is therefore reasonable to expect that including such methods in a post-secondary course would lead to comparable results.

Critical thinking and post-secondary education

Bonk, Ehman, Hixon, & Yamagata-Lynch (2002) conducted research related to the use of web-based tools in stimulating student critical thinking as well as for scaffolding the student learning experience. This article details the findings and methodology of a quantitative research study in which teacher learning is supported by using web-based tools rather than traditional classroom methods such as lecture. This form of self-study is comparable to that found in the SOTL research and was used to revise the course to better meet student needs.

This study provides specific lessons and heuristics related to structuring a course with the intent of taking advantage of the unique properties of online tools, while avoiding common errors made in including an online component in a course. The research also generates several questions related to online instruction and critical thinking strategies as originally outlined by Bonk, Kirkley, Hara, & Denned (2001). What critical thinking strategies are appropriate for use in a PBL based course? Where is there engagement happening in terms of critical thinking? Can online tools be used to implement activities that stimulate critical thinking? Such questions are at the heart of the present inquiry.

Several authors have used these principles in order to design models for the effective development of constructivist learning environments (CLE). The Design of Constructivist Learning Environments and Learning by Design/Design based learning (LBD/DBL) are two relevant, recent models based on these principles and are discussed in the following sections.

Designing constructivist learning environments

David Jonassen outlines a model for the design of a learning environment based on the aforementioned constructivist vision and design principles as given by Savery and Duffy (1996), Cunningham et al (1991), and Jonassen’s (1999; Jonassen & Hernandez-Serrano, 2002; , 1993) own manifesto for constructivist learning environments. This model centers on an ill-structured problem to be solved representing a particular learning goal that requires students to learn domain content and skills in order to solve the problem rather than learning the content out of the context of an authentic problem. Such problems must be at the heart of any designed constructivist learning environment and will therefore be central to the design of the proposed course setting.

An ill-structured problem is defined as one that has:

1. (U)nstated goals and constraints,

2. (M)ultiple solutions, solution paths, or no solutions at all,

3. (M)ultiple criteria for evaluating solutions,

4. (U)ncertainty about which concepts, rules, and principles are necessary for the solution or how they are organized,

5. (N)o general rules or principles for describing or predicting the outcome of most cases, and

6. (A requirement that) learners …make judgments about the problem and to defend their judgments by expressing personal opinions or beliefs.

(Jonassen, 1999)(p.219)

A number of tools are also discussed as supports for students’ knowledge construction in Jonassen’s conception of a CLE. They include (a.) cognitive tools to enhance student cognitive processing, such as visualization tools, (b.) static and dynamic knowledge modeling tools used to build models or simulations of real-world phenomena, (c.) performance support and (d.) information gathering tools for increasing student productivity in repetitive or difficult tasks. A final set of tools are for (e.) conversation and collaboration with the purpose of providing a method or environment for collaboratively constructing socially shared knowledge within the context of the PBL (Jonassen, 1999)(p. 220-230). Such tools must be provided whether designing for first grade students or college seniors and, while they may take many different forms, will be central for scaffolding the problem while allowing students to engage in serious inquiry.

The role of the instructor changes substantially in a CLE. They are no longer a provider of information to be memorized as in traditional learning environments. They take on the role of a modeler of appropriate performance, an articulator of examples of the reasoning and decision making skills they want students to exhibit, a motivating coach there to trigger reflection as they keep an eye on and regulate the learners’ performance, while scaffolding to support the learners as they encounter difficulty with the task or require alternative forms of assessment (Jonassen, 1999; Land & Zembal-Saul, 2003; Lin, Hmelo, & Kinzer, 1999; Vygotsky, 1978). In the college classroom it is especially important that the instructor become a facilitator rather than a giver of knowledge as we prepare students to work and engage in practice in real world settings independent of direct instruction.

Problem based learning in a CLE requires a number of responsibilities on the part of the learner. These include that students be self-directed and take advantage of the great diversity of experience they and others bring to the ill-structured problem and that are open to learning when they experience a need to know or do something. PBL also requires that students engage in problem-centered tasks and that they be motivated by internal self-esteem, recognition, need for a better quality of life, and self-actualization rather than extrinsic rewards (Kolodner, 2002; Lin, Hmelo, & Kinzer, 1999; Liu, 2003; Tiwari & Lai, 2002; Willis, 2002; Yip & Gafarian, 2002). If the instructor interferes in this process by demotivating or interfering with the students knowledge construction process, they disempower them from owning both the inquiry process and the products of that process.

As a result, a PBL-based course has been found to meet the needs of undergraduate students preparing for professional careers, better than lecture-based, instructor directed courses (Ironside, 2003; Lin, Hmelo, & Kinzer, 1999; Norman & Schmidt, 1992; Terehoff, 2002; Zembal-Saul, Blumenfeld, & Krajcik, 2000). The reported satisfaction rates of students engaged in PBL-based courses has commonly been higher than those rates reported by students in traditional learning environments (Ge & Land, 2003). Sandra Harris (2003) outlines an model that incorporates both the metacognitive principles and relevant, authentic tasks that are central to PBL’s common instructional design principles, as outlined by Savery and Duffy (1995), Jonassen (1999), and Kolodner (2002).

Not all the writing about problem-based learning has been positive, especially when it comes to students’ performance on accountability measures. PBL has also been criticized for lack of results on standardized tests due to its focus on practice and lack of work on rote memorization(Yip & Gafarian, 2002). In addition, the extensive amount of time that teaching from a PBL perspective often takes in a constructivist classroom is deemed to make it practical (Dede, 2004). In a similar learning theory, Learning by Design (LBD), Kolodner describes five guidelines for developing learning activities (2002):

1. Foregrounding of skills and practices: Cycles of iteratively applying what one is learning, interpreting and explaining one's solutions, debugging one's knowledge and skills based on those explanations, planning and predicting future use, and then applying what one has just figured out; reflection on use is critical.

2. Practicing: Consistent, continuous, reflective practice of targeted skills in contexts consistent with authentic use; undertaken individually, in small groups, and publicly.

3. Establishing need: Giving students a need to use the skills and practices as well as reasons to want to work together and learn from each other, through challenges that are complex enough to require collaboration and that require the targeted skills.

4. Making recognition of the need to use procedures automatic: Through ritualizing the practice of important packages of skills.

5. Establishing and enforcing expectations: Through establishing and sustaining a culture of rigorous thinking and collaboration through activities that make the need for such practice clear, and then using that culture to promote and support reflective practice. (p.35)

While this model has commonly been used in K-12 science education environments, it has also been applied in post-secondary environments. Stemming from research into the establishment of a curriculum based in Design Based Learning (DBL), it has been suggested by Jacob Perrenet and Ivo Adan (2002) that the following characteristics should be present in the development of a DBL curriculum: “(a.) professionalization, (b.) activation, (c.) co-operation, (d.) creativity, (e.) integration, and (f.) multidisciplinarity.” (p. 192)

Furthermore, Perrenet and Adan’s research also suggests that DBL can:

. to improve the quality of education;

. to increase the level of competence orientation;

. to reinforce the coherence between education and research;

. to strengthen cohesion and coherence within the (course or program);

. to achieve innovation of technical systems.

(Perrenet & Adan, 2002)(p. 192)

It is important to note that the roles of the instructor in DBL and LBD are comparable to those described by Jonassen for constructivist learning environments (CLE), although there may be a more substantial expert designer role in the development of student technology competency needed to respond to problems. These roles include: facilitator, coach, and provider of scaffolding for learners exhibiting needs not met within the initial explanation of the ill-structured design problem.

Further clarification regarding the uses of DBL in classroom and research contexts has been put forth by Barab and Squire (2004)related to learning and cognition issues and an overall charting of the DBL approach to constructivist learning. According to these authors,

“What separates design-based research in the learning sciences from formative evaluation is (a) a constant impulse toward connecting design interventions with existing theory, (b) the fact that design-based may generate new theories (not simply testing existing theories), and (c) that for some research questions the context in which the design-based research is being carried out is the minimal ontology for which the variables can be adequately investigated (implying that we cannot return to the laboratory to further test the theoretical claims).”

(Barab & Squire, 2004)(p.5)

It is from this framework that a serious attempt to study teaching and learning in situ is derived, leading to scholarship and inquiry related to developing theories and methods that stand up to authentic, real-world testing.

PBL and SOTL: A course design model

While not its explicit intent, a study was conducted by Sage (2002) that engages in a similar form of SOTL to Bass’ example. This particular research was designed to qualitatively evaluate the attitudes and conceptions of students during the implementation of a PBL-based an online course as it is taught while also examining explicit teacher practices and curriculum issues. A secondary focus of the study was to capture the learning and teaching experiences and challenges inherent in revising such a course to include problem-based learning. Sage specifically looks at a group of graduate students engaged in a PBL environment, structured online, using an asynchronous collaboration tool for the purpose of learning to integrate the internet into K-12 classroom content. Due to the fact that online courses have become prevalent, whether for distance Masters students or other, traditional students engaged in course work during summer sessions, this study holds special relevance in the current post-secondary environment.

Sage found that many of the problems students had in the course related to the presented constructivist, ill-structured problems rather than the online format of the course. Prior background experience, group learning experiences, individual learner differences, assessment and grading issues, building trust among the students, how to use the online interface, as well as how to design the problem based learning environment all came into play within the context of the study. Each issue generated substantial difficulty for both the learners and instructor, leading to formative correction as feedback from students was generated in the online environment (Sage, 2002).

As Sage suggests, research related to asynchronous, online collaboration tools is still in its infancy and presents an area in which substantial studies could be conducted, whether the tool is used as part of an online-only course, or is part of a face-to-face course in which the tool is used as part of a larger PBL environment. By studying the use of such a tool in this environment as we explore our own teaching practices, there is the belief that studying the appropriate use of PBL tools in the course of inquiry will provide insight into how to better make use of their inherent, specific affordances.

Angeli, Valanides, & Bonk (2003) developed tools for improving critical thinking in the context of online learning by addressing learning and cognition issues related to implementing strategies that were intended to improve student learning. The study focused on a group of undergraduate students who were preparing for careers as teachers and uncovered issues related to implementing an asynchronous, web-based tool for improving student learning. These included technology failures, student apathy, individual learner differences, as well as problems that stemmed from revising the course to embed innovative tools or to repair problems caused by them.

RESEARCH

Pilot Study of CECS 1100.004 [a.k.a The Door] (Spring 2007)

Hypotheses

Quantitative hypotheses

1. The use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources should improve the achievement of learners at a statistically significant level more than those learners in the existing drill and practice-based course.

2. The use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources should improve the level of satisfaction expressed by learners more than those learners in the existing drill and practice-based course at a statistically significant level.

3. The use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources should increase the level and amount of metacognitive reflection activities engaged in by learners more than those learners in the existing drill and practice-based course at a statistically significant level.

Qualitative questions related to Scholarship of Teaching and Learning

4. How does the instructor experience of teaching and revising the curriculum where they implement the elements of SOTL unfold differently than the experience of an instructor in an existing course?

5. How does the instructor report the findings of his SOTL experience (as taken from Cottrell and Jones (2003)?

Qualitative hypotheses related to the course game

1. The experience of learners using the course should be qualitatively different than for learners in the traditional sections as impacted by the game narrative.

2. Learners should have periods of success during the class that are qualitatively different from those students’ successes in the traditional curricula.

3. The kinds and forms of difficulty encountered by learners in the course game should be qualitatively different from the challenges faced by learners in the traditional curriculum.

4. The peer interactions engaged in by learners in the course game should be qualitatively different from peer interactions in the traditional course sections.

5. The struggles in working with peers in the course game should be qualitatively different from those faced by students in the traditional course sections.

Research Questions

Quantitative questions

6. Can the use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources should improve the achievement of learners at a statistically significant level more than those learners in the existing drill and practice-based course.

7. Can the use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources improve the level of engagement shown by learners at a statistically significant level versus learners in the existing drill and practice-based course?

8. Can the use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources improve the level of satisfaction expressed by learners more than those learners in the existing drill and practice-based course at a statistically significant level?

9. Can the use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources increase the level and amount of metacognitive reflection activities engaged in by learners more than those learners in the existing drill and practice-based course at a statistically significant level?

Qualitative questions related to Scholarship of Teaching and Learning

10. How does the instructor experience of teaching and revising the curriculum where they implement the elements of SOTL unfold differently than the experience of an instructor in an existing course?

11. How does the instructor report the findings of his SOTL experience (as taken from Cottrell and Jones (2003)?

Qualitative interview prompts related to the course

1. Tell me about your experience with the class.

2. Tell me about a time when you were successful in learning in the class.

3. Tell me about a time when you struggle to learn in class.

4. Tell me about a time when you worked well with your peers to complete a class task.

5. Tell me about a time when you struggled to work with your peers to complete a class task.

Research methods

As with any new instructional methodology, it is as important to discover the specific answers as to why it was more effective than another method as it is to learn that it is effective at modifying behavior, improving achievement, or engaging learners in reflection associated with improvements in critical thinking. By studying learner responses to questions regarding their experience, researchers learn more about how to recreate effective learning experiences on a global level rather than just a local one and how to modify a treatment to improve its effectiveness when it is employed in future courses. As such, this study will employ both quantitative and qualitative methods.

This research study will employ a quasi-experimental, posttest comparison design to measure the effect of a digital game-based, problem-based curriculum in a hybrid course on student achievement in an introductory computer education class. It is quasi-experimental because students were randomly assigned by their choice of section to be in one class or the other class to three classes that will be randomly assigned to the treatment or comparison group. The posttests will be based on the shared learning objectives of each section under the CECS 1100 course. The independent variable in this design is the type of instruction (hybrid, digital game-based, problem-based learning) and the dependent variable is student achievement on a posttest assessment given to all three sections and covering the same objectives shared across sections falling under this course number.

Participants and Setting

Setting

There will be three settings for this study. The first two comprise the comparison groups; one will be set in the online-only course using WebCT and the SAM 2003 course software while the other will be set in a face-to-face classroom employing the SAM 2003 software. The third (treatment group) will be set in the classroom for 6 total class periods over the course of the semester and in the digital environment Second Life combined with the Moodle course management system combined with online web logs, web sites, and tutorials.

Participants

The participants will be 96 undergraduate students at the University of North Texas enrolled in the CECS 1100 course during the spring of 2007. These students were quasi-randomly selected as they were self-selected to participate in each section with no prior knowledge by the researchers. These participants have no prior knowledge of the instructional style of the course prior to enrolling other than knowing that the comparison section (CECS 1100.030) is a hybrid course which meets only partly in a classroom or that the CECS 1100.020) section meets entirely online.

Quantitative methods

The research in the instance of the quantitative questions will differ dependent on the outcomes sought by the questions. In the instance of the achievement questions, students have been randomly assigned to a condition dependent upon the section of the CECS 1100 course that they signed up to take with no prior knowledge of the research questions and with no influence by the researchers. Two sections engaged in the existing 1.) online and 2.) face-to-face sections will act as the comparison groups. A third section, developed specifically as a hybrid course with six total face-to-face meetings over the course of the semester combines with learning tasks and activities that students will complete online using the open-source Moodle course management software, the no cost Second Life three-dimensional digital space, and numerous web resources developed to support learning housed in several open-source web log spaces. This third section will act as the treatment condition. Each quantitative question will use its own instrument.

Question 1. Can the use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources should improve the achievement of learners at a statistically significant level more than those learners in the existing drill and practice-based course?

In order to address this question, students in the two conditions will engage with either the treatment or comparison curricula over the course of the semester. At two points during the semester (midterm and final), each group will complete two exams based on the same learning objectives shared by each curricula. These objectives can be found in Appendix A. The first part of each examination will consist of knowledge specific questions that ask students about the features and functions of the software used up to the time of the examination to complete the assigned learning tasks The second part of the examination will require students to develop a small product using the software that shows application of their acquired skills within a set time limit. The second exam will cover the knowledge and skills that students were expected to acquire over the course of the entire semester.

Question 2. Can the use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources improve the level of satisfaction expressed by learners more than those learners in the existing drill and practice-based course at a statistically significant level?

This question will be addressed by providing students with a survey of learner satisfaction with the overall course, the means of instruction, means of assessment, and learning activities similar to the means of course quality assessment that will be provided by the College of Education. However, while summative, this survey will be more in-depth and ask specific questions about the delivery of instruction by system, attitude toward instructor, attitude towards instructional style, self-report of instructional style, attitude towards individual learning components and activities, attitude towards peers, as well as general satisfaction with the course.

In this instance, students will take the Constructivist On-Line Learning Environment Survey (COLLES), which is embedded within the Moodle course management software. A separate version was created for the non-treatment classes within an online survey collection tool. This survey will be administered at the conclusion of the course (see Appendix B.)

The COLLES is made up of 24 statements within six scales. Each has been developed to determine the quality of the on-line learning environment:

Relevance How relevant is on-line learning to students' professional

practices?

Reflection Does on-line learning stimulate students' critical reflective

thinking?

Interactivity To what extent do students engage on-line in rich educative

dialogue?

Tutor Support How well do tutors enable students to participate in on-line

learning?

Peer Support Is sensitive and encouraging support provided on-line by

fellow students?

Interpretation Do students and tutors make good sense of each other's

on-line communications?

Question 3. Can the use of a game-driven, problem-based learning curriculum for post-secondary learners that leverages existing and developed distributed learning resources increase the level and amount of metacognitive reflection activities engaged in by learners more than those learners in the existing drill and practice-based course at a statistically significant level?

This question will be addressed by collecting metacognitive reflections developed by students in their online, public web logs that are responses to online learning activities, peer questions, and instructor prompts that related directly to learning activities in the CECS 1100.030 course. The text of these reflections will be analyzed using both Computer Mediated Discourse Analysis (Herring, 2004) and Critical Discourse Analysis (Carspecken, 1996) to examine the level of critical discourse engaged in by learners related to metacognitive reflection. These utterances will be classified by section to determine the degree to which metacognitive reflection and power are expressed by students in the comparison and treatment groups.

Qualitative research

Qualitative questions related to Scholarship of Teaching and Learning.

1. How does the instructor experience of teaching and revising the curriculum where they implement the elements of SOTL unfold differently than the experience of an instructor in an existing course?

2. How does the instructor report the findings of his SOTL experience (as taken from Cottrell and Jones (2003)?

Qualitative questions related to the qualities of the class

6. Tell me about your experience with the class.

7. Tell me about a time when you were successful in learning in the class.

8. Tell me about a time when you struggle to learn in class.

9. Tell me about a time when you worked well with your peers to complete a class task.

10. Tell me about a time when you struggled to work with your peers to complete a class task.

References

Albanese, M. A., & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine, 68(1), 52-81.

Angeli, C., Valanides, N., & Bonk, C. (2003). Communication in a web-based conferencing system: the quality of computer mediated interactions. British Journal of Educational Technology, 34(1), 31-43.

Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. THE JOURNAL OF THE LEARNING SCIENCES, 13(1), 1-14.

Bass, R. (1999). The scholarship of teaching and learning: What's the problem? Inventio, 1(1), 1-9.

Bonk, C., Ehman, L., Hixon, E., & Yamagata-Lynch, L. (2002). The pedagogical TICKIT: Web Conferencing to promote communication and support during teacher professional development. Journal of Technology and Teacher Education, 10(2), 205-233.

Bonk, C., Kirkley, J., Hara, N., & Denned, V. (2001). Finding the Instructor in post-secondary online learning: pedagogical, social, managerial and technological locations. In J. Stephenson (Ed.), Teaching and learning online: Pedagogies for new technologies (pp. 76-97). London: Kogan Page.

Bransford, J., Vye, N., Bateman, H., Brophy, S., & Roselli, B. (2003). Vanderbilt's AMIGO Project: Knowledge of How People Learn enters cyberspace. Retrieved 4/20/2004, 2004, from http://www.vanth.org/mmedia/vanth0103/vanth0103cd/papers/AmigoWFig.pdf

Carspecken, P. F. (1996). Critical ethnography in educational research. New York: Routledge.

Chickering, A., & Gamson, Z. (1987). Seven principles for good practice in undergraduate education. American Association for Higher Education (AAHE) Bulletin, 3-7.

Cottrell, S. A., & Jones, E. A. (2003). Researching the scholarship of teaching and learning: An analysis of current curriculum practices. Innovative Higher Education, 27(3), 169-181.

Cunningham, D. J., Duffy, T. M., & Knuth, R. A. (Eds.). (1991). The textbook of the future. London: Horwood Publishing.

Dede, C. (2004). If Design-Based Research is the Answer, What is the Question? A Commentary on Collins, Joseph, and Bielaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthal, Krajcik, and Soloway in the JLS Special Issue on Design-Based Research. Journal of the Learning Sciences, 13(1), 105-114.

Diekelmann, N., & Scheckel, M. (2003). Teaching students to apply nursing theories and models: Trying something new. Journal of Nursing Education, 42(5), 195-197.

DiPasquale, D., Mason, C., & Kolkhorst, F. (2003). Exercise in inquiry: Critical thinking in an inquiry-based exercise physiology laboratory course. Journal of College Science Teaching, 32(6), 388-393.

Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the Design and Delivery of Instruction. In D. H. Jonassen (Ed.), Handbook of Research for Educational Communications and Technology. New York: Macmillan.

Elder, L., & Paul, R. (2002). Critical Thinking: Teaching students how to study and learn (Part II). Journal of Developmental Education, 26(2), 34-35.

Entwistle, N. (2002). Approaches to studying and levels of understanding: The influences of teaching and assessment. 156-214.

Everett, M., & Zinser, O. (1998). Interdisciplinary social science courses: Using a critical thinking approach. Journal of General Education, 47(3), 225-239.

Ge, X., & Land, S. (2003). Scaffolding students' problem-solving processes in an ill-structured task using question prompts and peer interactions. Educational Technology Research & Development, 51(1), 21-38.

Harris, S. (2003). An andragogical model: Learning through life experiences. Kappa Delta Pi Record, 40(1), 38-41.

Herring, S. C. (2004). Computer-mediated discourse analysis: An approach to researching online behavior. In S. A. Barab, R. Kling & J. H. Gray (Eds.), Designing virtual communities in the service of learning (pp. 338-376). Cambridge, UK: Cambridge University Press.

Ironside, P. (2003). New pedagogies for teaching thinking: The lived experiences of students and teachers enacting narrative pedagogy. Journal of Nursing Education, 42(11), 509-516.

Jonassen, D. H. (1999). Designing constructivist learning environments. In C. M. Reigeluth (Ed.), Instructional Design Theories and Models: A new paradigm of instructional theory (Vol. 2, pp. 215-239). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Jonassen, D. H., & Hernandez-Serrano, J. (2002). Case-based reasoning and instructional design: Using stories to support problem solving. Educational Technology Research & Development, 50(2), 65-77.

Jonassen, D. H., Mayes, T., & McAleese, R. (1993). A manifesto for a constructivist approach to uses of technology in higher education. In T. M. Duffy, J. Lowyck, D. H. Jonassen & T. M. Welsh (Eds.), Designing environments for constructive learning (Vol. F 105, pp. 231-248). Berlin, Germany: Springer-Verlag.

Keller, G. S. (2002). Using problem-based and active learning in an interdisciplinary science course for non-science majors. The Journal of General Education, 51(4), 272-281.

Kolodner, J. (2002). Facilitating the learning of design practices: Lessons learned from an inquiry into science education. Journal of Industrial Teacher Education, 39(3), 9-40.

Land, S., & Zembal-Saul, C. (2003). Scaffolding reflection and articulation of scientific explanation in a data-rich, project-based learning environment: An investigation of progress portfolio. Educational Technology Research & Development, 51(4), 65-84.

Lin, X., Hmelo, C., & Kinzer, C. (1999). Designing technology to support reflection. Educational Technology Research & Development, 47(3), 43-62.

Liu, M. (2003). Enhancing learners' cognitive skills through multimedia design. Interactive Learning Environments, 11(1), 23-39.

Meyerson, P., & Adams, S. W. (2003). Designing a project-based learning experience for an introductory psychology course: A quasi-experiment (Reports - Evaluative). Chicago, IL: American Educational Research Association.

Norman, G. R., & Schmidt, H. G. (1992). The psychological basis of problem based learning: a review of the evidence. Academic Medicine, 67(9), 557-565.

Perrenet, J., & Adan, I. (2002). From mathematical modeling to design based learning: A bridge too far? International Journal of Math Education, Science and Technology, 33(2), 187-197.

Sage, S. M. (2002). The learning and teaching experiences in an online problem-based learning course (Poster). New Orleans, LA: American Education Research Association.

Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its constructivist framework. Educational Technology, 35, 31-38.

Shulman, L. S. (1998). Course anatomy: The dissection and analysis of knowledge through

teaching. In P. Hutchings (Ed.), The course portfolio: How instructors can examine their teaching to advance practice and improve student learning (pp. 5-12). Washington, D.C.: American Association for Higher Education.

Terehoff, I. (2002). Elements of adult learning in teacher professional development. NASSP Bulletin, 86(632), 65-77.

Tiwari, A., & Lai, P. (2002). Promoting nursing students’ critical thinking through

problem-based learning. Paper presented at the 2002 Annual International Conference of the Higher Education Research and Development of Australasia (HERDSA), Perth, Australia.

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.

Warren, S. J. (2006). The Impact of a Multi-user Virtual Environment (MUVE) Teacher Instructional Time, Voluntary Student Writing Practice, and Student Writing Achievement Unpublished Dissertation, Indiana University-Bloomington, Bloomington, Indiana.

Warren, S. J. (2006, April 7-11, 2006). Using a Digital Learning Environment Scaffolds to Improve Student Writing in a PBL-style Instructional Space. Paper presented at the American Educational Research Association Annual Meeting, San Francisco, CA.

Willis, A. S. (2002). Problem-based learning in a general psychology course. The Journal of General Education, 51(4), 281-292.

Yip, W., & Gafarian, A. (2002). Problem-based learning assessment for information systems courses (Reports - Evaluative No. IR 058 611). Brisbane, Australia: International Academy for Information Management.

Zembal-Saul, C., Blumenfeld, P., & Krajcik, J. (2000). Influence of guided cycles of planning, teaching, and reflection on prospective elementary teachers' science content representations. Journal of Research in Science Teaching, 37(4), 318-339.

Appendices

Appendix A.

Skill and Knowledge Objectives for CECS 1100

1.0 CECS 1100 SKILL OBJECTIVES

1.1 Computer architecture and file management

The Learner Will (TLW) use their knowledge of computer architecture

and file management to:

1.11 Backup files and other data

1.2 Hardware

TLW be able to Identify and Use:

1.21 Hardware/Software Overview

1.22 Computer’s CPU

1.23 Memory/storage, I/O

1.24 Peripherals

1.3 Internet

TLW will be able to use the Internet to:

1.31 Develop a Blog reflecting on current trends in technology

1.32 Identify and engage with peers using appropriate etiquette

1.33 Use Search Engines for research and information gathering

1.34 3D environments for navigating course tasks

1.4
Word Processing

TLW be able to use a word processor for the following purposes:

1.41 Compose a document with appropriate typography

1.42 Undo/Redo commands

1.43 Moving and copying features

1.44 Find, replace, go to in a doc

1.45 Formatting: Alignment, indents, tabs, borders and spacing,

columns, page breaks

1.46 Graphics, clip organizer, Word Art

1.47 Bullets, Headers/Footers, Page numbers

1.48 Table of contents, sections, indexes

1.5 Desktop publishing

TLW will be able to design one or more of the following using a

desktop publishing program:

1.51 Flyers, advertisements, press releases

1.52 Invitations and menus

1.53 Publications and newsletters

1.54 Brochures and catalogs

1.6 Collaboration tools

TLW will be able to use collaboration tools to design and edit:

1.61 Correspondence such as business and personal documents

with formatting templates and wizards

1.62 Reports and long documents

1.63 Long documents such as multi-page letters, reports, meeting

documents and schedules/forms

1.64 Mail merge

1.7 Spreadsheets

TLW be able to use a Spreadsheet program to:

1.71 Enter and format data

1.72 Enter and edit formulas

1.73 Setup spreadsheets on multiple pages

1.74 Modify cells/ranges

1.75 Format using copy, move, paste, and alignment tools

1.76 Adjust fonts, borders, patterns, and shading

1.77 Develop charts and graphs

1.78 Use freezing, hiding tools

1.79 Print spreadsheets and charts

1.791 Functions

1.792 Templates

1.8
Statistical and Mathematical Functions

1.81 Spreadsheets/what if

1.82 Statistical functions

1.83 Expressions vs. functions such as If, Lookup

1.84 PMT, FV

1.85 Ledgers

1.86 Payroll, asset records, income

1.87 Budget development

1.88 Analysis, accounting

1.89 Sales, Investment

1.9 Presentation software

TLW use Presentation software tools and features to:

1.91 Modify and choose appropriate views

1.92 Choose, modify and develop layouts and save them as

templates

1.93 Use the clip organizer to embed graphics

1.94 Set up different forms of slide show

1.95 Use auto content and custom Schemes

1.96 Embed multimedia and animation in presentations

1.97 Include appropriate bullets

1.98 Automate processes, include motion paths, timings, and action

buttons to develop electronic presentations

2. 0 Integration

TLW will use Productivity Software tools in an integrated fashion to

develop:

2.1 A basic online portfolio

2.2 A collaborative document that uses multiple software tools such as

spreadsheets and word processors to develop a presentation

2.3 Proofread documents that have multiple authors

2.4 APA citations to support their work

3.0 Knowledge Objectives

3.1 General course

3.11 Course Objectives

3.12 Course Overview

3.13 Project requirements

3.14 Available Resources

3.15 Expectations of participation and behaviors

3.2 Computer architecture/file management including:

3.21 The history of computers

3.22 Components of the desktop and Explorer

3.23 Files and folders/General organization

3.24 Bits/bytes

3.3 Hardware/software
TLW be able to Identify and Understand the Purpose of:

3.31 Hardware/Software

3.32 Computer CPU

3.33 Memory/storage

3.34 I/O

3.35 Peripherals

3.4 Internet knowledge

TLW will understand issues related to Using the Internet such as:

3.41 Security issues

3.42 Privacy concerns

3.43 Copyright

3.44 Etiquette (a.k.a. netiquette)

3.45 TCP/IP setup and functioning

3.46 WWW vs. intranets

3.47 Blogs and other current trends

3.48 Search Engines and boolean language

3.49 URLs/DNS

3.5 TLW be able to identify Word Processing General tools and features

3.6 TLW understand and identify Spreadsheet Basics such as:

3.61 General Tools and features

3.7 TLW understand and locate Presentation software tools and features

including:

3.71 Views

3.72 Layouts

3.73 Templates

3.74 Clip organizer

3.75 Slide show

3.8 TLW understand current Information Technology topics such as:

3.81 Emerging communication tools and techniques

3.82 The role of gaming in learning and training

3.83 The need for and tools that allow for multi-user collaboration

3.84Ethical issues

3.9 TLW understand the purpose of developing an Online Portfolio

Appendix B.

Midterm exam with objectives

Appendix C.

Final exam with objectives

Appendix D.

Constructivist On-Line Learning Environment Survey (COLLES)

Appendix E.

Weekly metacognitive response prompts by week

Appendix F.

General instructor exit interview script

Appendix G.

Treatment instructor exit interview script

Appendix H.

Student formative interview script (semi-structured)

1. Tell me about your experience with the class.

2. Tell me about a time when you were successful in learning in the class.

3. Tell me about a time when you struggle to learn in class.

4. Tell me about a time when you worked well with your peers to complete a class task.

5. Tell me about a time when you struggled to work with your peers to complete a class task.

Appendix I.

Student exit interview script

1. Tell me about your experience with the class.

2. Tell me about a time when you were successful in learning in the class.

3. Tell me about a time when you struggle to learn in class.

4. Tell me about a time when you worked well with your peers to complete a class task.

5. Tell me about a time when you struggled to work with your peers to complete a class task.

6. How did you feel overall about this class? Why?

---