Evaluation

End to End Performance

3handsWe conducted a study of student perception of RSV images under degraded image quality. The degradations included reduction of the available bandwidth for the transport of the images through the network links, and two different latencies between server and client.

Subjects: 1st year Stanford Medical students and their Anatomy TAs

Variables

Resolution: Hand images at multiple resolutions: original resolution (978x1357), 50% resolution (489x679), and 25% resolution (245x339). In addition, we used Adobe Photoshop compression quality to vary the images.

Latency: Images from local or remote servers:  Stanford and Michigan

Bandwidth: Images sets that were bandwidth restricted:  1 Mbps, 2 Mbps, 5 Mbps, 10 Mbps, 15 Mbps, 20 Mbps, 30 Mbps, 40 Mbps, 50 Mbps, 60 Mbps, 70 Mbps

Content: Hand images with an interactive quizzing script

Methodology

Pilot Test
Once we had the 20 variations, we had 8 users from SUMMIT rate the images on image quality and responsiveness.

Rating

Corresponds to

Which means that the degradation

5

Excellent

Is not noticeable

4

Good

Is noticeable, but not annoying

3

Fair

Is annoying

2

Poor

Makes the application very hard to use

1

Bad

Makes the application unusable

Based on their responses, we then selected 10 image sets for the subjects.

Implementation
5 students plus a TA met in the SUMMIT Collab room where we set up 6 individual stations plus a theater style seating in front of the stereo projection screen.  After an initial orientation about the study and the HAVnet grant project, we asked the students to rate ten image sets on a scale of one to five as shown in the above figure.   They were asked to rate based on the image quality (sharpness, ability to differentiate detail, outline of anatomy vs. background, clarity of 3D) and responsiveness (length of response time between user and computer).  One of the researchers then showed each of the images explaining when he started rotating the image and when he stopped.  The rotation was repeated.  Then the subjects were asked to rate the image.

This process was repeated for all 10 image sets.
Stanford 1 Mbps, Full resolution, best quality
Stanford 1 Mbps, 25% resolution, poor quality
Stanford 20 Mbps, Full resolution, best quality
Stanford 70 Mbps, Full resolution, best quality
Michigan 1 Mbps, 25% resolution, poor quality
Michigan 1 Mbps, 50% resolution, poor quality
Michigan 2 Mbps, Full resolution, best quality
Michigan 5 Mbps, 50% resolution, poor quality
Michigan 30 Mbps, Full resolution, best quality
Michigan 30 Mbps, 25% resolution, poor quality

The final component of the study provided us with the opportunity to understand how students would rate the images in a teaching and learning context.  The TA quizzed the students asking them to identify various features of the hand.  The leadership rotated between the TA and the students, giving them a way to rate the responsivness of the application.  In addition by trying to identify various anatomical features, they were able to rate the quality of the images.

Results
The analysis of the data provides some important insights into the understanding of the impact on user perception. An important finding from the analysis is that medical students rated with scores around 4 (considered very good) the scenario where the RSV application works with a network connection set at 5Mbps and with an image set at 50% resolution. This is a striking finding considering that the non-degraded scenario works with a bandwidth around the high 70 Mbps and 100% image resolution.

This opens the opportunity to extend the use of RSV on IEEE 802.11 Wireless Lan environments, where bandwidth is scarce. We also investigated the impact of responsiveness and image quality on the overall user perception. We found that the responsiveness component has a bigger influence on the overall perceived quality of the RSV application.