A lot of members of the group had brought the same kind of question, namely which kind of evaluation method should we use? To answer that question we first decided to have a look at the DECIDE framework (from the third edition). This framework will help us set guidelines for our future evaluations, although we would have gotten more out of it if we had done this before we started this project. One reason for this is that we probably formed this DECIDE framework in line with our results so far unintentionally. The theories regarding the ethical issues when conducting interviews etc was also something that we found very useful but too late to apply now.
A project needs to be evaluated several times in order to receive feedback on the different design stages. It would be naive at this stage to set a goal that the users should be able to use this new system without problems. Instead we set the framework with respect to our current state, which is a clickable prototype that we created with pop-app. From this we would like to answer questions and sub-questions listed below.
DECIDE MODEL
- Setting Goal:
- Evaluate our pop-app
- Asking questions:
- How do you feel about the number of options/buttons?
- What can be improved?
- How is the size of the buttons?
- Are the language options enough?
- What languages do you request?
- Are any languages unnecessary?
- How easy is it to understand the information?
- What did you find difficult?
- Is there any help?
- Is the new app better than the current system?
- In what way?
- Does users miss any information?
- If they do, what information
- Evaluation method
- Let experts evaluate the prototype next exercise using heuristic evaluation.
1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Help users recognize, diagnose, and recover from errors
6. Error prevention
7. Recognition rather than recall
8. Flexibility and efficiency of use
9. Aesthetic and minimalist design
10. Help and documentation
- Practical issues
- We will tell the experts:
- What questions they should ask themselves
- What we want them to have in consideration
- How to record their feedback
- The goal with the evaluation
- The evaluation environment will not be similar to real conditions.
- The evaluation device (tablet) will not be similar to the ticket machine.
- It will be hard for Swedish young people to enter the roles of tourists in Sweden.
- Often people with the same background think alike. Since we are all students and engineers to come, we might think alike and miss some important flaws.
- A practical issue is that during the tomorrows exercise when we are going to get our app evaluated we’re not going to be present and really looking over how the students/experts interact with it. We may therefore compromise in some way where we let other students test it while we overwatch.
- Time will also be a factor that limits our results since we are limited to the assigned exercise session.
- There might be problem using our English interface for the students due to the fact that they are swedes.
- Decide how to deal with ethical issues
We should let the experts know that their personal information is not stored together with their names. If they don't want any information about them then we’ll accept that. We could give them a informed consent form to sign if we feel that it’s necessary.
Another great way to eliminate different kinds of hold-backs or doubts from the participants is to offer them a preview of their transcripts before we make anything official.
- Evaluate, analyze, interpret and present the data (1-5, 5 being highest rated)
When we have collected the data from the evaluation there are several things we need to keep in consideration. Firstly we have the reliability of the evaluation. How sure can we be that our findings are correct? Will we get the same results during separate occasion under the same circumstances? Due to our limited time we will maybe not be able to make a second evaluation and may therefore have to make a qualified assumption. But there are other reliability problems to consider here as well. Maybe our young student as experts will have a hard time to enter the roles of tourists in Sweden which could make the result less reliable. The fact that our app is not similar to a real ticket-machine with all the stress attached could also make our results less reliable?
The second point we need to consider is the validity of our evaluation. Do the required measurements match the ones we specified in the beginning? The third point is a part of the validity, namely ecological validity. We must here consider the fact that the result might have been different if we would have done the evaluation in the subway instead of doing it here at KTH. Biases is another important point. Specially since all our experts we will be using are students and engineers to come, they might think alike and miss some important flaws. When we get their findings we must try not to influence their responses so what we get back is as accurate as possible. Lastly but not least the we have the scope. Here we’ll need to contemplate whether our findings can be generalized or not.
Afterwards we can set grades on a scale from 1 to 5 where 5 is really good and 1 is really bad. We did during the exercise made a guess on how we think it will be.
- Reliability: 3
- Validity: 4
- Ecological validity: 1
- Biases: 3
- Scope: 3
Inga kommentarer:
Skicka en kommentar