ICS Resume Posting Service: Usability Testing: Usability Testing 1: Data Collection Form

back to home

Members
Joy Agustin (jagustin@hawaii.edu)
Astrid Tomsic (astrid_tomsic@yahoo.com)

Memos

Plans

Requirements Analysis

Usability Testing

   Interviews



   Usability Testing 1



   Usability Testing 2

Conceptual Design

Prototype

Participant ID: 2 (Alumni)		Session Date: 10/23/2002 Start Time: 18:15 End Time:
Task Description	Number of Errors(1)	Elapsed Time	Participant’s Actions and Comments	Evaluator’s Observations
1. Locate Resume Index	1	116 secs	Don't know where to start. (Reads through items.) Look at icon things to give me some clues. Is it under People? Maybe the About this web site link will tell me where to look. No, nothing about Resumes here. (back to home page.) I guess I'll have to read all the links. Oh, here it is, in the Career Zone.	Did not associate Career Zone category with the Resume Index.
2. Add a Resume	2	313 secs	OK, so where is the link to Add a Resume, not here. Maybe it's at the bottom of the page (scrolls to bottom of the page). No, maybe I missed it at the top (scrolls to top of the page). I guess I'll have to read all of this (reads text at top of page). I have to Login to add my resume (clicks the Login link and types in a user ID). Oh boy, I don't remember my old password! I'll try something…Nope, here's an error telling me I have an Incorrect username and/or password. It's telling me to login but there's also a link for New Student / Alumni. Let me try that. No, I don't have an uhunix account. So, I have to send an email to the web master (clicks on the email link). I wonder how long this is going to take to get a user ID. At point the evaluator provided a valid username and password so that the task could continue. (Clicked the Back button and typed in the username and password). OK, I've got to fill these fields. (Fills in First Name, Last Name, Email, checks Show in People Section, checks Show Resume in Career Zone, types in Resume URL. Clicks Make Changes. Error: Data Missing from the Resume Fields). But I did add in everything! (Clicks the Back button and types in "Masters" for degree and "HTML, Graphics Design" for Skills. Leaves Graduation Month and Year as default zeros. Clicks Make Changes.) Good, the changes went through.	An alumni would not have a uhunix username and password, so one was provided for the purpose of allowing the subject to complete the test.
3. Update Personal Preferences	0	149 secs	I can’t believe there’s no update button. I had to login to add the resume and I can update my preferences from there. (Clicks login link and logs in. Changes email address successfully. Checks Alumni box. Types in “May” then “2000” for graduation date. Types in “ICS Master’s” for degree. Typed in “Java, c++, Networking, HCI” for skills). OK, I’m done. (Clicks Make Changes button and quickly scans changes made).	Graduation Date format was not clear so the participant typed in “May”, not “05” as was expected.
4. Find and view a Resume ("HTML" skills)	2	243 secs	OK, I need to find the Search (clicks on People - Students link on the ICS Home page). Who do I know that has HTML skills? I think Ryan Au does. Let me see his resume (clicks on Ryan Au’s name and email program launches with Ryan Au’s email address in it). This is not what I want! (Closes email window). Where do I go next? Now I see a link to the Online Resumes (clicks on link “…See also our online resumes page…” at the top of the screen. Scrolls down the page, looking at the Skills Column). I’m sure that Ryan Au does have HTML skills but I don’t see them listed here? (Keeps scrolling down and locates Floyd Casem). Here’s one with HTML skills listed. But it doesn’t look like people have all their skills listed.	Got lost in the ICS site and tried to find resume’s in the People – Students category. Not sure why he did this, as he had already viewed the resume list in the previous task. Did not find the Search tool and preferred to scroll down the list of resumes until he located the first one that had the matching skills?
5. Find and view a Resume ("Pascal" and "COBOL" skills)	2	246 secs	I know where to look now. (Clicks on Resume Index). Let me see who has Pascal skills (presses Control-F on the keyboard and types in “pascal”). It’s telling me there’s no one with pascal skills. Let me try the other one (types in “cobol”). None found here. I don’t think this search is working. Let me try the other section again. (Clicks the Back button to go back to the ICS Homepage, then clicks People – Faculty link). Looks like I’m going to have to do this the long way! Let’s see if Martha has COBOL skills. (Clicks on Martha’s name than clicks on link to personal page). Nothing here about her skills and there’s no link to her resume. At this point the evaluator ends the task, as it is obvious that the participant is lost in the site and has resorted to not using the Resume Index at all.	The participant did a search for “pascal” and “COBOL” and even though the search result told him that there were no people with matching skills, he didn’t trust it and went on to look elsewhere in the site. The Resume Index search tool was not used.
6. Find and view a Resume ("SQL" skills and most recent graduate)	0	39 secs	From the ICS Homepage, clicked on Resume Index link. Typed Control-F on the keyboard and typed in “sql”). OK, Floyd Casem graduated in ’96 but there must be people who graduated later than that. (Clicks on the Grad Date column header and sorts the records). That’s better, now I need to search again. (Types Control-F and types in “sql”). Floyd Casem comes up again. Darn, I need to sort in the opposite order, but the link on the column isn’t active anymore! (Clicks on the Grad Date column header a few times). So, that means I have to scroll all the way to the bottom of the page. (Scrolls to bottom of page, types Control-F, types in “sql” and selects to search in Up direction). So, I found Matthew Sharritt, but he hasn’t graduated yet. (Clicks Find Next until he comes to Steven Miyakawa). This is it, Steven Miyakawa. He graduated in May 2002.	The participant was annoyed when he clicked on the Grad Date column header and it worked once but when he wanted to use it again to sort the records in the reverse order the link had become inactive.
7. Find Resume Count	1	81 secs	(Clicks on the Resume Index link from the ICS Homepage). I don’t see any kind of numbers here – no totals of how many resumes are here (reads instructions at top of page). This will have to be done manually again! It would really help if it were clearer how to do things in this Index. I don’t want to have to read the instructions all the time and then I find that what I want to do isn’t even in there! (Types Control-F and types “Java”. Clicks Find Next, Find Next…and counts number of occurrences one by one until no more entries are found). Looks like there are 48 people with Java skills	Participant did not read the question properly and found people with both “Java” and “JavaScript” skills, not only people with “JavaScript” skills. The evaluator did not stop the task, as they were more interested in observing the process used to count the resumes.
8. Remove a Resume	0	343 secs	(Clicks on the Resume Index link from the ICS Homepage. Scrolls down the list of resumes looking for his name). Why am I not listed? Don’t tell me this thing doesn’t add people in real time! (Types Control-F and types in his last name). Oh, forget that, I found it. Now, how do I remove my listing? (Looks at instructions at the top of the page). Well, it tells me how to add a resume but not how to remove it. Maybe I have to click on the row. (Scrolls down to his name and right clicks on the row). No options here to remove my resume. Maybe I have to fill out a form? (Clicks on the Forms link at the top right of the page). Nothing here to do with removing a resume. I can’t think of anything else except to search. (Clicks on the Search link at the top right of the page and types in “remove resume”. Clicks Search button). What a surprise, no results found! I give up! I’m going to send an email to the webmaster and ask them to remove my resume that way. (Clicks on the Career Zone link on the left navigation bar. Gets blank screen). What happened here? Looks like the link is wrong. (Clicks Back button three times to get back to Resume Index. Clicks “email ICS Department” link to send an email).	It appears that it is not easy to find names in the list of resumes, even though they are listed alphabetically by last name. The participant could not even locate his own name and had to resort to using the browser page search tool. The participant was frustrated that he couldn’t find a way to remove his resume from the list and especially, that nothing was mentioned about it in the instructions.

(1) An error or negative critical incident is something that happens while a participant is working that has a significant negative effect on task performance or user satisfaction, and thus on usability of the interface.

Last modified: Mon Dec 09 16:36:12 Hawaiian Standard Time 2002