Student portal and test player field trials
Background
The focus of the interface enhancement project was to replace the old flash test player. The challenge was to develop a new player that could replace the old player and modernise the player interface so that it is accessible from a range of devices without impacting students test performance.
Student results obtained using the old player needed to be directly comparable to their results obtained using the new player on a broader range of devices.
For this reason, changes were focused on positional changes of the three major elements of the test player:
- the reading passage or question
- the response
- the question navigation bar.
Previously, the reading passage or question occupied the top of the screen. It now occupies the left side of the screen.
The question used to appear under the reading passage below a screen splitter. It now appears on the right side of the screen.
Finally, the navigation bar had been on the left side of the screen. Now, it is at the bottom of the screen.
e-asTTle test play comparison – downloadable PDF that compares the old student site and test player with the new site and player.
No changes were made to the wording, content of the questions, or the reading passages.
What is the same are the response options for each question, previous student test scores, and the test generation processes used by e-asTTle. Additionally, all reports remain the same.
For this reason, it was expected that students’ performance would be unaffected by the high-level layout changes introduced by the new player.
Field Trialling
To test this expectation, trialling was undertaken using Reading tests with Year 7 students. Two trials were run with approximately 100 students each.
The first trial compared students’ performance on the old and new players. The second trial compared students’ performance using the new player on desktops and tablets.
During these trials, students sat two tests each and the effect of test order was controlled.
Trial 1 - Player Comparability trial
Comparison of Flash test player and the new test player
Ninety-nine Year 7 students sat a different Level 3-4 reading test on each of the test player versions. No significant difference was found between students’ scores across the Flash and HTML versions. Eighty percent of students preferred the HTML player.
Trial 2 - Device Comparability trial
Comparison of the new player used on laptops (Chromebooks) and tablets (iPads)
Ninety-eight Year 7 students sat a different Level 3-4 reading test on each of a Chromebook and an iPad.
No significant difference was found between students’ scores across the laptop and tablet versions. Although the results involved different students, they produced similar test results to the first trial.
Overall, there was no evidence to suggest that the device used or the layout changes had a significant impact on students’ performance.
Why trialling was done using Reading tests and Year 7 students
Intermediate students are the heaviest users of e-asTTle. In addition, Year 7 students are halfway between the youngest and oldest students using e-asTTle.
Reading tests were used because more scrolling is typically required in Reading tests.
It would be expected that if there were a difference in performance due to the reduced scrolling or the layout between tablets and desktops, the difference would be greater in Reading tests.
Usability Testing
Throughout the e-asTTle interface project, four primary schools, four intermediate schools, and two high schools participated in usability testing.
Students sat tests on a range of different devices, including tablets (iPad/Android) and laptop/desktop machines (Mac/Windows/Chromebook). Students’ valubale feedback was incorporated into the new interface design.
Teacher input was sought for the new online administration guidelines and for the new practice test scripts.
In addition to the trials focusing on student performance, user acceptance testing was undertaken on the final version of the test player in Term 4, 2015.
Teachers from three schools administered a test to a group of students. They used the new administration guidelines, practice tests, and the new test player. A range of devices were used during the user acceptance testing. In addition, Reading and Maths tests as well as Custom and Adaptive tests were used.
- Related information