Evaluations
External Evaluation Reports
External Evaluator, Dr. Victor Hernandez-Gantes, Ph.D. (see bio)
Final Evaluation Report: 2009-2014
Highlights (emphasis added)
Overall, given the efficacy of project implementation and completion, quality of research
and development work, and assessment of broader impacts in the field, there was
consistent evidence supporting a representation of exemplary project performance (p. 5). Unlike typical projects resulting in a particular unidimensional
product (e.g., a game), the CyGaMEs project produced an informatics structure that
turned the Selene game into a multi-purpose instrument with broader impacts on teaching
and learning (students and educators as users) and on cyberlearning research (researchers
interested in related topics) (p. 24).
- Further, the project developed an outstanding record of synergistic research and development activities contributing to broader impacts in the areas of videogame development, using embedded assessments, and dissemination of results. (p. 2)
- Overall, development and research activities were on Green Status upon completion of the project. That is, related activities, events, or products were completed on schedule and all milestones anticipated in the project plan were met. (p. 3)
- The quality of research and development activities was characterized by a strong conceptualization capacity leading to the transformation of the Selene game into a robust research instrument, and contributions to the shared understanding of important mediating factors in cyberlearning.
(p. 3)
- In general, research activities reflected broader contributions to the advancement of discovery and understanding of science education through cyberlearning. The quality of research activities was reviewed using two indicators: The research capacity to meet related project goals, and the nature of research outcomes. To that end, the project built upon unique interdisciplinary expertise and collaboration with partners in the field contributing to the quality of the project's research capacity. Further, based on the nature and totality of research results, the project contributed to the shared understanding of important factors in cyberlearning through the use of embedded assessments. In addition, the project produced a reporting system that allows educators monitor player's progress and results, and
project researchers access game data for research purposes. (p. 3)
- This unique collaboration on the conceptualization of the videogame underscored the
original quality of the project featuring Dr. Reese's application of structure mapping
theory and Dr. Wood's knowledge of the formation and evolution of the Earth's moon.
This collaboration also allowed the team to monitor and assess the validity and accurate
representation of science concepts, and verify the reliability of the game concept
throughout the design and development stages. (pp. 17-18)
Conceptualization capacity. This factor aligns with the NSF's indicator of
interdisciplinary intellectual capacity. In the case of this project, the intellectual capacity
behind the conceptualization of the CyGaMEs project stemmed from strong
interdisciplinary and synergistic work of senior staff. The Co-PIs leading the project -
Drs. Debbie Reese and Charles Wood - built upon unique interdisciplinary expertise
bridging research and development in instructional design and content knowledge in
lunar science. Dr. Reese provided the overall leadership and core conceptualization ideas
for the project drawing from prior research and related theoretical underpinnings (Reese,
2009a). Based on the premises of structured mapping and analogical reasoning she
provided the conceptual basis for the development of a videogame-based environment for
learning and assessment of science concepts. Dr. Reese's prior work leading to the
conceptualization of CyGaMEs can be traced through her graduate studies and a series of
publications on the subject (see for example Reese, 2007, 2008b; 2009a; Reese &
Coffield, 2005).
In turn, the focus on lunar formation benefited from the direct contribution of NASA
science through the work of Dr. Wood, who has devoted his life to the study of the moon.
Upon earning a doctorate in Planetary Geology, he spent 10 years as a NASA Space
Scientist at the Johnson Space Center in Houston Texas. He continued his work at other
venues and he is currently the Executive Director of the Center for Educational
Technologies at Wheeling Jesuit University. In his view, many discoveries and
understandings from planetary exploration have not been incorporated into education,
and he seized the opportunity to develop a videogame targeting lunar concepts in
collaboration with Dr. Reese (C. Wood, personal communication, December 13, 2012;
Wood, 2003).
As a result of their collaboration, their conceptual plan was to transform the existing
Selene game into an instrument to generate more reliable data of player performance
while mediating learning of lunar geology. The focus on lunar formation and evolution,
in the content domain of Earth and Space Science, is based on the Moon's unique record
of basic geological processes of planetary birth and evolution. Its proximity allows for
personal observation and makes for real connections during game playing. Further, the
science of lunar formation and evolution underscores the Selene game (i.e., accretion,
differentiation, impact cratering, volcanism, and stratigraphy) and aligns with national
science education standards (D. Reese, personal communication, June 15, 2011; National
Research Council, 2007; Wood, 2003).
-
Based on the nature and totality of research results, the project contributed to the shared
understanding of important mediating factors in cyberlearning through the use of
embedded assessments. In short, the research results stemming from project work yielded
evidence that when playing CyGaMEs, players work hard toward the game goal and
develop an understanding of how lunar geology works. In addition, the project produced
an informatics reporting system allowing the use of the Selene as a research instrument
for access to researches interested in cyberlearning. (p. 22)
- Moreover, the project did a remarkable job producing a robust record of dissemination including numerous publications and conference presentations. (p. 29)
- Overall, the project established an impressive network of partners with several organizations including public schools, universities, research centers, and outreach agencies. (p. 29)
Evaluation Report: 2011-2013
Evaluation Review Session, October 25, 2013
External evaluator Dr. Victor Hernandez-Gantes rated NSF-funded CyGaMEs project “exemplary” in all project areas. He said
In looking at the overall performance of the project, year by year and where you are right now and what is coming out, you have met all what you were supposed to do. There is evidence on each of the goals and beyond. . . . Dissemination and outreach with the addition of the bilingual version; that is just icing on the cake.
He noted that CyGaMEs dissemination is
phenomenal. That what I think at this point for you it is routine, but it is anything but routine. It is truly a team exercise that you are doing over and over again, and every time you do it, of course, you come away with something extra to say. Because, now you’ve been doing it and you find another tweak that you can show – and it gets better and better! Lots of credits to you and your team on that. That has emerged over the past five-six years, and it’s been remarkable. They way now the team has coalesced and I can see that productivity at this last couple of years, it is just phenomenal. It has had a burst of excellence.
A five-minute video excerpting selections from the October 25, 2013, three-hour evaluation session is available below or through YouTube.
Additional highlights include
- Spanish Selene and the NASA Badging award ceremony conducted in the Canary Islands. Hernandez-Gantes translated the TV news coverage of the event, at which students (ages 11-12) who earned the CyGaMEs Selene Lunar Geology Badges said Selene empowered them to learn without a teacher.
- The WJU Data Management System. CyGaMEs collaborated WJU Information Technology Services to build and deploy the Data Management System (http://www.wju.edu/dms ). The system meets federal regulations that data from federally-funded research must be available to the entire research community. The system is available to all WJU researchers, and the data are available to the public.
- The quality of CyGaMEs dissemination and effectiveness of team.
- An overall exemplary rating in all four project areas.
| Project Goals and Objectives |
| Our goal is to make science concepts intuitive through structure mapping. Our approach to applied structure mapping requires an interactive environment and pragmatic constraints. Videogames meet these requirements. Thus, our project objectives are: |
| Objective 1. Transform our existing instructional videogame Selene: A Lunar Construction GaME into a robust research instrument. |
| Objective 2. Collect GaME data representing learning and perceptions, i.e., flow. |
| Objective 3. Develop and refine GaME methods and metrics for assessing motivation, engagement, and learning. |
| Objective 4. Integrate GaME data collection and assessment within a cyber-enabled informatics infrastructure. We apply cognitive science to instructional game design and develop an informatic system incorporating assessment methodologies that measure and report learner growth. |
Session Excerpts: Video and Transcript
Read the comments in an excerpt from the transcript of the evaluation review session (below), or listen to them as spoken by Dr. Hernandez-Gantes and documented by the session video. Click the image below to watch and listen using Flash. If your device does not support Flash, watch using YouTube.
Video Transcript [PDF]
DISSEMINATION AND EFFECTIVENESS OF TEAM
Every time you have something going on, and especially where there are some stakes attached to it, you always, you and your team come up with some way to really enhance the visibility of the project and your team delivers or you find a way for them to put something together that really highlights what the team is about. Every time you show me or send me something, you either put together a poster…something that was specifically designed and prepared for that particular event, that is phenomenal. That what I think at this point for you it is routine, but it is anything but routine. It is truly a team exercise that you are doing over and over again, and every time you do it, of course, you come away with something extra to say. Because, now you’ve been doing it and you find another tweak that you can show – and it gets better and better! Lots of credits to you and your team on that. That has emerged over the past five-six years, and it’s been remarkable. They way now the team has coalesced and I can see that productivity at this last couple of years, it is just phenomenal. It has had a burst of excellence. (2:08)
NASA BADGING
Spanish Selene, NASA Badging Awards, and La Palma TV coverage of “The Best Moon Builders” Award Ceremony Event (see also, news story brief , which includes links to external media and news sources) Dr. Hernandez-Gantes translated the TV news coverage of the La Palma Badges for NASA Activities award ceremony (the news feature broadcast in Spanish). Students (ages 11-12) who earned the CyGaMEs Selene three and seven star Lunar Geology Badges were interviewed. They told the reporter Selene empowered them to learn without a teacher. Other translated components of the broadcast follow.
- The anchor (reporter) is talking about the NASA award as if it were something very special and to make it sound really official.
- The narration was by the television crew, but they were describing what Selene is about and the connection to NASA, and to clarify the role of NASA. They describe Selene as an internationally award winning project. And then they talked about how engaging the game is and how the students master it and they receive the award.
- Out of three thousand players, nine were awarded at their site, and 2 of those were in the top 10 or 20.
- This is really nice. They are talking about the role of engaging videos in authentic learning. They are also talking about the potential of the game to really enhance learning for kids, and the kids [interviewed] are talking about what they are getting out of it. And they are remarking that it is all on their own without anyone looking over their shoulders telling them what to do. Just learning without a teacher.
DATA MANAGEMENT SYSTEM
CyGaMEs collaborated with WJU Information Technology Services to build and deploy the Data Management System(http://www.wju.edu/dms). The system meets federal regulations that data from federally-funded research must be available to the entire research community. The system is available to all WJU researchers. Look at this. Not only is a data management system making the data available from the project, it will be now a university data management system, not only for your project, but for everybody who gets funded. And all of a sudden, here you are adding value to a project that was funded by NSF and extended to a university. That is evidence of an exemplary project.
OVERALL PERFORMANCE OF PROJECT
In looking at the overall performance of the project, year by year and where you are right now and what is coming out, you have met all what you were supposed to do. There is evidence on each of the goals and beyond. So essentially, for each area, I think this is going to emerge again as exemplary on all four areas. Dissemination and outreach with the addition of the bilingual version; that is just icing on the cake. So if anything, I agree with you that at this point your energy should be on articles. I think this is going to emerge again as exemplary on all four areas.
[Erroneously edited out of the video] This is really, really good. In looking at the overall project, in each area it is coming out really well.[The Data Management System] is just another evidence of the things that are emerging out of it. To me, another significant contribution is not just the number of publications, that is just the tip of the iceberg. Under the surface, there is a lot of work to come with up the structure of Selene, to come with structure like you are showing me. The tracking system is another big development the project has to offer.
Transcript reviewed and approved for distribution by CyGaMEs External Evaluator Dr. Victor Hernandez-Gantes, 10-26-2013
External evaluator Dr. Hernandez-Gantes found that "Overall, given project progress, quality of research and development activities, and level of synergistic collaboration and dissemination, the project continues to exhibit exemplary standards of implementation" (p. 3).
Some of the many report highlights:
"The broader impacts of project activities align with contributions to emerging evidence that simulations advance conceptual understanding and can motivate students’ interest in science learning. Project results have also established the value of embedded assessments in understanding how players experience cyberlearning. These results are significant because they align with and contribute to recent calls for conducting research and development about how embedded assessment technologies can be used to engage students in learning and assess what and how they learn" (p. 2).
"Regarding the quality of products designed to improve cyberlearning, the project has generated research evidence supporting the merits of the design and learning experience. Further, the project has continued to develop a consistent record of synergistic research and development activities contributing to broader impacts in the areas of videogame development, use of embedded assessments, and dissemination of results" (p. 10).
"Building upon the strong intellectual capacity supporting project activities, research and development activities have produced important and interesting results. To this end, the dissemination of project information and emerging results has been outstanding. Overall, dissemination activities have included a variety of formats and the project is creating increased interest in the field as evidenced by the number of presentations, invited talks, and emerging publications" (p. 19).
"Based on the review of reports of activities, audit of project documentation, and analysis of project result, it is evident the project earned Green Status through Year 3 of implementation. That is, proposed activities, events, or products are on schedule and all milestones anticipated during this period of the project plan are on track. Overall, at the end of Year 3, it is evident that development, research, and dissemination activities are following well-developed and realistic operational timelines and strategies. In turn, the merits of project activities and products designed to improve cyberlearning are well documented and supported by sound strategies. Further, the project has continued to develop a remarkable record of synergistic activity with potentially significant broader impacts resulting from research and development activities. All in all, given project progress, quality of research and development activities, and level of synergistic collaboration and dissemination, the project continues to exhibit exemplary standards of implementation" (pp. 20-21).
Click here to view or download this report in PDF format.
This report was prepared for the Center for Educational Technologies by Dr. Victor M. Hernandez-Gantes, an associate professor at the University of South Florida who is serving as the external evaluator for the CyGaMEs project. The report finds that “in general, given project progress, quality of research and development activities, and level of synergistic collaboration and dissemination, the project may be characterized as reaching exemplary standards of implementation.”
Click here to view or download this report in PDF format.