Seminar 2 - Individual Post - Viktor Bram

Seminar 2  - Viktor Bram

The further down a project goes the more important evaluation becomes. When you evaluate a product it is important to know when and how to do so. This is something we'll need to keep in mind for our project! One way of doing it is the DECIDE framework which consists of six steps.

1. Determine the goals.

By determining the goals the evaluation process gets easier since the goals helps to determine the scope.

2.Explore the questions.

To reach the goals stated we need to create questions to make it possible to reach them.

3.Choose the evaluation methods

To make the correct choice we must first figure out what best fit our project. Sometimes it’s better to have a combination of methods in order to gather richer data. So my first question for the seminar is which evaluation methods should we use?

4.Identify the practical issues.

Before we conduct a evaluation it’s helpful to know about the the practical issues that exists. Maybe there are some kind of constraints that we need to address. For us a practical issue could be if we choose to let tourists test our pop-app that they might not like us to watch them when they test it.

5. Decide how to deal with ethical issues.

Our participants privacy have to be protected. You can only show personal information that they accept. One nice way of make our participants aware of the whole evaluation is to give them a informed consent form to sign if they accept.

6.Evaluate, analyze, interpret and present the data.

Here we will make decisions about which data is relevant to answer the study questions, how the data will be analyzed and how the findings will be presented. The reliability of our evaluation may differ if it produces the same result on separate occasion under the same circumstances or if it doesn't. Validity is also important which means that we have to check if the evaluation method measures what is should. There is a special validity called ecological validity that checks if the environment in where a evaluation is conducted influences the results. Biases can occur when expert evaluators who are performing a heuristic evaluation are more or less sensitive than the rest to certain kind of design flaws and then the result get distorted. Also interviewers may unconsciously influence responses from participants and then biases occurs. The scope is also important which refers to how much a evaluation study’s findings can be generalized.

Even though the DECIDE framework is divided into several steps it’s important that we know that each item in it is related to the other items in different ways which means that an iterative working process is preferred.

Chapter 15 was about inspections, analytics and models. One big deal of this chapter was about experts. Sometimes users are not easily accessible, or it may be too expensive or hard to involve them that it is prefered that experts provide feedback.

Heuristic evaluation is a usability inspection method in which experts guided by a set of usability principalities known as heuristics, evaluate user-interface elements. How many heuristics and which ones may vary and depend on the goals of the evaluations. I think that this method is really good and something we should use. My second question is: Which heuristics should we use?

It’s important to note that some usability problems are easy to find by all experts while several others just get discovered by a few. Therefore it can be argued that we should involve several evaluators. But since it’s very costly and we don't have the money for that we can hopefully use us of other students of the course as our experts.

There are three steps of heuristic evaluation that we should follow:

1. The briefing section in which we tell the experts what to do.
2. The evaluation period where the experts inspect the product using heuristic for guidance.
3. The debriefing section where they come together and discuss their findings and suggest solutions.

Walkthroughs is an alternative approach for predicting user problems without doing user testing that we could do. Then we have the choice between cognitive walkthroughs which focus more closely on identifying specific user problems at high level of detail and pluralistic walkthrough where each of the evaluators is asked to assume the role of a typical user. Like I stated earlier it's sometimes better to use multiply evaluation methods since it can gathers richer data. Hence my third question: Should we use both heuristic evaluation and walkthroughs?

Analytics is a method for evaluating user traffic through a system. This could be really good for us if we get a large testing pool so we can make correct assumptions. My forth question: Could we use any form of analytics when we evaluate our pop-app and then which and how?

Predictive models use formulas to derive various measures of user performance. GOMS is a good example of a predictive model which attempts to model the knowledge and cognitive processes involved when users interact with systems. I don't think we should use of a predictive model since it is to hard to apply for us.

// Viktor Bram