Design & Research 2 with Birgit Bachler
During our last session with Birgit Bachler we performed some usability tests of our prototypes.
My aim was to find out if the reporting game was understandable and engaging. The reporting minigame does not have a VR function yet and I supposed that this could have created a difference between how I imagined the prototype and how it was perceived by others. Since the testing happened during broad daylight in a classroom, I suspected that users would not have understood that the minigame is meant to be used at night to send a useful report. For this reason I prepared some pictures of the sky to simulate different observations.
I measured these parameters:
- Time spent on tasks
- Number of trials until the participants succeeded in completing the task (if they did)
- Task completion — whether the task was completed in way I intended it
The method used was the “thinking aloud” one. Users should speak while they perform the test and tell the interviewer their thoughts and feelings.
These were the tasks that I gave my participants:
- Play the game without lying or pretending
- Play the game in this situation (picture of clear sky)
- Play the game in this situation (picture of cloudy sky)
- Play the game in this situation (picture of polluted sky)
- Imagine you are seeing a clear sky but you cannot find that constellation
- You already know where the constellation is and want to jump straight to reporting
The questions I asked them at the end were the following:
- Did anything create frustration or confusion (text, icon, something in the flow)?
- How would you explain the main goal of the game to a friend?
- What was your favourite part?
- If you could add a feature, what would it be?
- What do you think about the visual style and aesthetics?
Three people participated in the test, including two students and one professor. I called them P1, P2 and P3. I did not ask them all of the questions, instead I selected different ones every time to keep the test shorter.
The results of the tests are summed up in this table:
These are the pictures I showed my participants for task 2, 3 and 4:
Some other observations are summed up in the following table:
This is how the participants answered the questions:
Did anything create frustration or confusion (text, icon, something in the flow)?
P1: I would say that in the beginning I wasn’t quite sure what I’m supposed to do, but maybe also because I did not read the text fully in the beginning, so I wasn’t quite sure whether you can only report when it’s night. I didn’t quite get that at the moment and was confused because I could not see anything, but I should have seen something. Yeah, then maybe the hint, but I already said that about the northern hemisphere, when you give a hint that something is supposed to be there, that there’s some sort of hint about which direction to look and maybe also how high or maybe if there’s another star that you would recognise that you can take as a reference point or something. Then I think you can make it quicker. Also, in the beginning I didn’t know what this button meant, I would have had to test it out to see what it does and also the exit button didn’t go to the start, but it went back to the star constellation, so maybe from exit I would expect that it would let me exit. I think maybe in the beginning it could be a little more clear what to do and more help, and maybe that the stars could be a little bigger or further apart. The smaller ones are a challenge to click.
P2: A lot of stuff generates confusion, like when I started I didn’t even know how to report. It said “locate this star” and I don’t even know what that is, so it was really frustrating, like what exactly I have to find? So I just randomly tapped here and there and found out. The most frustrating part was that everything is designed for a person who knows about this.
P3: The finger (the tapping on stars feature) – didn’t realise that at the start.
How would you explain the main goal of the game to a friend?
P1: I would say: identify constellations and also report about light pollution in different areas based on what stars are visible with the eye in the sky.
P2: The main goal of this game is to locate the stars and report them.
P3: Reporting to what extent a constellation is visible from a certain location, and perhaps including metadata as well – or did that come later? Yeah, that would be cool, because you’d say, ‘I can’ ’t see it, and then you can say, ‘Why not?’ I reckon one reason is that I’m too daft to find the star, and the other is that it’s too light-polluted, so I think you have to… But since it’s just a game, you can simply track light pollution and constellations.
What was your favourite part?
P1: I think clicking the constellations, like the selection of the stars I could see.
P2: When I can see the clear sky and I’m like “yeah maybe I am doing something” and also I really like that there’s a hint option, it gave me a little bit of information.
P3: I think the writing is a good start, ‘Help me’ and so on.
If you could add a feature, what would it be?
P1: Maybe more constellations? And also I had an app on my phone when I was stargazing and it showed the entire night sky. Every star is supposed to be there. Maybe it could also be nice to have an overview of all the constellations that should be in the sky and if you could kind of see on the phone and then compare it to what you actually see. I think it might also be easier to find a constellation. Like, okay, I found this constellation and if I turn a little more up then I can find more constellations and get a better understanding of what is where in the sky.
P2: I would really like the feature to be really using the current sky, like having a region of it. That could be really cool.
P3: Maybe the step where you have to look for the constellation in the sky or check if it’s visible. It’d also be cool to have a location, just like with those apps, so you know that’d be the next feature – and at the same time you’d be learning about the constellations, so you could pick up on that sort of mythological stuff on the side as well – that mix of astronomy and astrology.
What do you think about the visual style and aesthetics?
P1: It is very nice, very clean, very easy. Also it’s like the theme of the stars, black-and-white. Also, I think the constellation with the outline of the animal and then the stars inside is good because then you also learn a little more about the constellations, because sometimes in the sky if you only have the stars it’s hard to see the animal or whatever that’s supposed to depict. I think the style was very nice.
P2: The night vision, I really liked it. I think it’s a good direction.
P3: Simple and neat.
After analysing the recordings of the tests, I concluded that:
- It was not clear to the participants that you are supposed to play the game at night and tell the game if it is daytime; I could have specified that in the introduction (“this game should be played at night”)
- Sometimes when the stars are not visible, participants would still send a report; I could make a first screen that asks a yes/no question to filter out reports that are not valid
- Constellations are not always a concept known by everyone; I should explain what they are at the beginning
- There were issues with orientation and localisation of the constellation; the VR feature with compass sensor and GPS that other applications use would help with that
- Users tried to tap on the stars on the first screen; it should be clear that they are not tappable or they should be tappable right from the start
- Exit buttons were not working properly; I should check them
- The stars were a challenge to tap on; maybe the tappable area could be bigger or there should be a zoom/swipe feature
- The option to tap on the stars was appreciated; I can keep working with that
- There was some confusion about what was reported at the end; I should specify it
- The black and white theme with the constellation illustrations was appreciated
For the next testing, I need to remind myself to:
- explain the product and what we are doing with it
- ask some introductory questions to understand the participant’s knowledge of the topic
- ask everyone the same questions to compare results better
