Ensuring the integrity of data in virtual immersive assessments

Document Type

Presentation

Publisher

American Educational Research Association (AERA)

Publication Date

4-30-2010

Abstract

Research Goals. We are developing and studying the feasibility of using immersive virtual technology as a platform for assessing middle school students science inquiry for use in school settings as a standardized component of an accountability program. Our research question presented here is: How do we design immersive virtual environments to best support construct-related activities that result in the collection of valid and reliable assessment data?

Theoretical Framework. We are using the Evidence Centered Design framework (Mislevy, Steinberg, & Almond, 2003; Mislevy & Haertel, 2006) to ensure construct validity of the assessments we develop.

Assessment Context and Methods. Using the Unity engine, we are developing three assessments that measure students learning in situ. The assessments are based on real ecosystems and are all measuring identical constructs. In order to ensure that the data we are capturing is valid and that our assessments can be used in schools as part of an accountability program, we have developed a back-end architecture and administration system.

We are conducting cognitive studies and alignment studies to ensure we are measuring what we plan to assess (Quellmalz, Kreikemeier, DeBarger, & Haertel, 2007; Quellmalz, 2007). We will conduct a generalizability study of our measures effectiveness (Baxter and Shavelson, 1994; Shavelson, Baxter, & Pine, 1991).

Data and Results. Here we present the results of our administration system and our process of ensuring the data we collect and report has integrity. Everything students do in this 3-D environment is captured and stored (see figure 1).

Results. The clients connect to an Adobe Flex application utilizing the Unity web plugin. This application is administrated on the server itself through Internet Explorer in a webpage served on the loopback address of the Ethernet card. This application in turn reads and writes data to a local MySQL database through a set of PHP scripts.

In order to manage the use of multiple servers to collect data and to unify those data for analysis, the architecture includes a Repository. This online database application includes the following features:


Registration for servers;

Facilities for managing the upload of tests and logs from servers without record collision;

Central management of students, teachers, assessments, and administrations to prevent ID collision cross servers;

Automatic generation of SQL scripts for updating servers; and

Role-based security.

While, in the current state, synchronization must be undertaken manually between server and repository; the application is designed in such a fashion to support future automation of this task. To our knowledge, this architecture is unique in research on virtual environments.

Significance. There is a growing interest in using immersive virtual environments for assessments. Unless researchers can ensure these assessments have construct validity, are reliable and valid, and ensure integrity of the data collected, scholars will never be able to convince policy makers and psychomatricians why these assessments are better measures of learning than paper and pencil item-based tests.

This document is currently not available here.

Share

COinS