by anja.kljajic -

D&R2 – Lo-Fi Prototypes 1/6

Continuing on with my theme of User Interfaces in Video Games, I developed 3 quick and dirty prototypes for the Design & Research 2 course kick-off.

Prototype 1 – The Sticky HUD

This prototype that I will most likely bring to class is the sticky UI kit. I want to see how people who might not have preconceptions (but also those that do) would naturally arrange different UI elements when put on the spot. User expectations are pretty important when it comes to interfaces so that’s why I decided to play around with that idea, where people could tangibly display their expectations and intuitions in a setting where I could also document it.

It consists of:

  • piece of paper with a background
  • 10 pieces of paper with UI elements with tape on the back

Prototype 2 – Menu Journey

This prototype is based on diving in and out of menus and the information hierarchies associated with them. It’s made with folded papers which would be flipped up depending on which button is clicked. The task would, for example, be finding how to change subtitle sizes, as they are often in video settings under accessibility. The confusing part is also that its also sometimes under audio settings so this could serve as sort of cart sorting test as well. Traversing complex tree menus that just keep going in and in can be frustrating.

It consists of:

  • 3 folded pieces of paper with 6 different menus

Prototype 3 – Subtitle Slider

This prototype is about my biggest pet peeve with games, which is subtitles. It’s uses a piece of paper with scribbles to simulate complex backgrounds that acts as the background and a strip with different subtitle styles. I would simulate different situations (e.g. placing the paper further away) and slide the subtitle styles and ask what was written. This could open a discussion about what is necessary and what works in situations like a game where a lot of mental power could be being used.

It consists of:

  • piece of paper that acts as a background with a cutout
  • strip of paper with different subtitles
by anja.kljajic -

ID1 – NIME Article

The paper I chose to read on the NIME archive was:

Bubble Drum-agog-ing: Polyrhythm Games & Other Inter Activities by Jay Alan Jackson

The reason I chose this paper is because it had something to do with games, which was my very open research topic so far. The whole thing is about using big exercise balls as drum kits.

As seen on the picture, this project wanted to re-imagine drum kits capable of input. Regular drums are loud and can damage hearing, but provide a steady exercise value. Rubber drum kits for practicing with input also exist in the form of Guitar Hero or Rock Band.

What the author wanted to achieve was to eliminate the feeling of no feedback and thus no feeling when it comes to hard rubber kits. The data is captured using an accelerometer, microphone and camera inputs, to make it possible to play rhythm games. There are microphones placed closely to the bubble drum and they use Drumagog to replace the drum samples, while replicating the original performance responsively and accurately.

The paper also mentions that both aural and visual feedback are provided, but this is within the games themselves. The game that was developed by the author was a simple flash game “Polynome”. The objective and challenge is for the player to perform polyrhythmic patterns to existing songs, using the drums as controllers. The drum samples are using different elements depending on the song, in order to create unique remixes of rhythm and sound.

Figure 2: Polynome Screenshot

What’s interesting to me is the UI shown, the circles with the lines inside them are a reoccurring motif that is, I assume, meant to be the main indicator of what to do within this game. I’m not very well versed with music theory, but I am well versed with rhythm games, so I would have to wonder what these symbols mean and how this game actually makes things clear to the player. Unfortunately, these aspects aren’t described or analysed in this paper.

Overall, I find the idea fun because I like rhythm games and unique interaction methods acting as controllers. However, I find the paper to be a bit shallow and lacking more technical information. I can’t fully imagine the interaction, how the game would work, or how this entire thing would provide a “rigorous workout” (as stated many times in the text).

References

  • [1] J. A. Jackson, “Bubble Drum-agog-ing: Polyrhythm Games & Other Inter Activities,” in Proc. 12th Int. Conf. on New Interfaces for Musical Expression (NIME), Ann Arbor, MI, USA, May 22, 2012, exhibit.
by anja.kljajic -

User Interfaces in Video Games 10/10

User Interfaces in Video Games – The quest for genre-appropriate and usable game UI

Over the last nine blog posts, I’ve went from the oscilloscope screens of 1958 to visual representations of game UI to the modular, accessible hardware of today. Now I’d like to wrap up my journey with a short reflection.

Final Thoughts

This research diary started with a fairly open topic that I had no idea how to navigate. After my research, I can safely say that I’ve learned a lot about games, their interfaces and the interaction design behind it.

I learned that game UI has a history worth respecting. Going back to the oscilloscope screens of the 50s and the early arcade days of Space Invaders was interesting because I realised that the simple high score was an innovation at the time and was the start of the complex feedback loops we have now.

I learned how to categorise the visual representations of game UI. Breaking down the four types of UI completely changed how I look at game screens. I now see how a UI can either be overlayed on top of a game or be woven directly into the world, with the player character being aware of it.

I learned that style can actually drive usability. Exploring the Aesthetic-Usability Effect showed me that when a game aesthetic, it isn’t just for show. I learned that if a menu feels like it belongs in the game’s world, players are more likely to find it intuitive and engaging, which holds water with my personal gaming journey.

I learned that accessibility is a fundamental responsibility. From my struggle with tiny subtitles to the impact of the QuadStick, I learned that game UI design isn’t just an aesthetic choice but also about inclusion. This coincides with the fact that I’d like games to be enjoyed as many people since they made my life much better.

In the end, through all these learnings and this, to be honest, hard journey I realised that designing a game user interface is a way, way, way, way more complicated and diverse topic than I anticipated. When I picked this topic I was mostly just focusing on the simple thought of “cool, stylish UI that also respects users” and kept a narrow focus on the visual part, the user interface design of it all.

However, through this research diary and through a conversation with the Senior UX/UI Designer at Bongfish, I’ve realised that game user interface designers are responsible for way more than the graphical menus and HUDs. The 60 accessibility options of The Last of Us Part II kind of blew me away with the use haptics, audio cues and difficulty settings.

If I continued with this (now daunting) topic, I’d have to consider narrowing down the research to specific devices (PC, console, mobile or VR/AR etc.). Placing emphasis on just the visuals doesn’t really work for this topic, as evidenced by this extensive journey of many sub-topics, so finding a focus area could be hard. Either way, I’d say it was a valuable journey and I’ve collected some actual knowledge on my newfound love: games.

by anja.kljajic -

User Interfaces in Video Games 9/10

interaction
accessibility

User Interfaces in Video Games – The quest for genre-appropriate and usable game UI

To continue with the accessibility topic of my last blog post, in this one I would like to dive deeper into how this complex interaction of playing video games is for people with disabilities.

Accessibility Pt. 2 – Physical Control

Beyond the screen and graphical user interface, we have to consider the physical interface players control the game with. While reading the Saunders and Novak book Game Development Essentials: Game Interface Design, I was really moved by the story of Robert Florio, a quadriplegic artist. He uses a “mouth stick” to play games like Devil May Cry 3, a fast paced action game with complex combos [1]. It made me realise that an accessible interface isn’t just about ease of use, but it’s also about giving someone control over a world they can’t physically interact with anymore. When a designer adds the option of remappable buttons, they aren’t just making a “setting”, they’re opening doors for people who wouldn’t be able to interact with the product at all otherwise.

The “mouth stick” in question was an early model of the QuadStick, pictured on Figure 1. This is a mouth-operated controller produced by an independent manufacturer. It acts as an “add-on” to existing consoles or PCs, using sip-and-puff sensors to translate breath and lip movements into complex game inputs [2].

Figure 1:
Quadstick
Source: [3]

A major sign that the industry is finally taking this seriously is that console manufacturers are now building these solutions themselves. The PlayStation Access Controller is a modular kit designed specifically to be accessible out of the box. It moves away from the “fixed” shape of a standard controller, allowing players to create a layout that works for their specific hand strength or range of motion. This further emphasises the importance of customisable and remappable inputs in games.

Figure 2: Playstation 5 Access Controller
Source: [4]

This is where one really sees how interacting with games goes beyond the user interface. It’s also about the user experience and overall interaction design. I already mentioned The Last of Us Part II in my last blog post, focusing on the vast variety of subtitle adjustment options. This is just one out of over 60 different accessibility options [5].

Their design philosophy follows a “sensory redundancy” model. This means that if a player can’t see the path, Navigation Assistance uses haptic pings and 3D audio cues to guide them. If a player can’t hear an enemy, Awareness Indicators and Combat Vibration Cues translate sound into visual and tactile data. This really showed me how expansive this theme can get once we look at the broader spectrum of the interaction between the game and the player.

References

by anja.kljajic -

User Interfaces in Video Games 8/10

usability
accessibility

User Interfaces in Video Games – The quest for genre-appropriate and usable game UI

The question proposed in my last blog post is a big can of worms that has many aspects influencing it. One big aspect of interfaces being usable is accessibility, which I took a look at in this blog post.

Accessibility

In my research, I’ve found that many people treat accessibility as a “bonus feature,” but as Saunders and Novak point out in Game Development Essentials: Game Interface Design, it’s a fundamental responsibility. Since there are no strict government regulations for games, it’s up to developers to self-regulate to meet the needs of those with disabilities [1].

In my introductory blog post, I mentioned the frustration of games having subtitles but them being too small to read, often with bad contrast. Subtitles are a perfect example of where game UI often has issues. In many modern AAA games, the text is optimised for someone sitting in front of a high-resolution monitor. But for a console player sitting on a couch 3 meters from a TV, that text becomes unreadable.

I noticed this in many games but want to point out Black Myth: Wukong as an example, pictured on Figure 1. The text is so tiny that even at my monitor I could barely read it, especially on white backgrounds where it lacked contrast in addition to it’s small size. It really dampened my experience because I played the game with the Chinese dub, but this would be an even worse experience for someone who’s, for example, deaf.

Figure 1: Black Myth: Wukong
Source: [2]

To combat this, the choice of typeface is important. Sans Serif fonts (like Arial or Verdana) are preferred for difficult viewing conditions because they don’t have the tiny “cross strokes” (serifs) that can blur together at low resolutions [1]. Simply testing the legibility on different devices and positions during development would already make a huge difference.

A best practice example for dealing with subtitles can be seen in The Last of Us Part II. They provide incredibly adjustable subtitle options where players themselves can massively increase the text size, change the color of the names to identify speakers, and add a dark semi-transparent backing box behind the text. This means that no matter how bright the game world is, the text is still legible.

Figure 1: The Last of Us Part II
Source: [3]

Another aspect to consider is colour-blindness. Around 8% of men (1 in 12) and 0.5% of women (1 in 12) are affected [4]. Considering this data, its vital to never use color as the only way to give information. A health bar shouldn’t just change from green to red; it should also change in length so a color-blind player can still read the state of the game [1]. Likewise, if a game uses only red and green to signal “enemy” versus “friend”, a significant portion of the audience is excluded.

References

by anja.kljajic -

User Interfaces in Video Games 7/10

aesthetic
function
style

User Interfaces in Video Games – The quest for genre-appropriate and usable game UI

So far I’ve introduced some history, UI elements, their visual representations and common game genres. Now I’d like to take a look at the “Should games sacrifice functionality for style and vice versa? Do accessibility options affect the art being made?” question that popped up in my introductory blog post.

Aesthetic vs Function

One of the most difficult parts of game UI design is the “battle” between aesthetics and functionality. In my gaming journey so far, I’ve seen both sides of the coin: games that are beautiful but hard to navigate, and games that are perfectly functional but look like sterile, uninspired and out of place.

“Form follows function” is a famous phrase coined by Louis Sullivan which has been applied to many different types of design that deals with this topic [1]. This means that the way something looks is influenced by what it’s supposed to do. The function of game UI is to communicate states, so it should adapt itself to what users actually need to function within the game. The common questions are “Where am I?”, “How much health do I have?” and “Am I winning?”. These are answered with mini-maps, health bars and scores, all of which have evolved through necessity to communicate status.

So can style actually improve function? Why do I enjoy stylish UI in games if minimalist UI also does the job? Thinking about this led me to the Aesthetic-Usability Effect, which is defined in the book Universal Principles of Design. The Aesthetic-Usability Effect is described as “a phenomenon in which people perceive more aesthetic designs as easier to use than less-aesthetic designs – whether they are or not.” [2] This means that if a player loves the look of a menu, they’re more inclined to keep using it, and thus learn how to use it.

A personal example I’d like to showcase is the difference between the Metal Gear Solid: Peace Walker (2011) and Metal Gear Solid V (2015) staff management menus. In Metal Gear Solid: Peace Walker I learned to navigate the menu thanks to the more simplified information and “military file” aesthetic which fit the game world, being set in the 70s.

Figure 1:
Metal Gear Solid: Peace Walker HD
Source: [3]
Figure 2: Metal Gear Solid V
Source: [4]

Metal Gear Solid V, which was released 4 years later, features a virtually identical menu, but goes for an angled look which is supposed to be a hologram from the device the character is holding in his hand. This takes away real estate for the sake of diegetic immersion. This, however, clutters the UI with more information displayed. I would have been overwhelmed with this menu had I not already “trained” myself with the previous game. I knew which information to ignore and what the actual function of the menu is. The aesthetic is also lost within this blue, minimalist, hologram look which clashes with the fact the game is set in the 80s.

This leads me to believe that style shouldn’t be sacrificed for function or vice versa.

A visual style is first determined for the game experience overall. Then, the information is made to come across in the most immediate and understandable way. Finally, both form a framework for the user interface aesthetics. The visuals shouldn’t drive the function, but they can certainly bend and influence it. – Stieg Hedlund [5]

In my next blog post, I want to dive deeper into the usability aspect of this debate by exploring the topic of Accessibility.

References

  • [1] L. H. Sullivan, The Tall Office Building Artistically Considered. Philadelphia, PA, USA: J. B. Lippincott, 1896.
  • [2] W. Lidwell, K. Holden, and J. Butler, Universal Principles of Design: A Cross-Disciplinary Reference. Gloucester, MA, USA: Rockport Publishers, 2003.
  • [3] Game UI Database, “Metal Gear Solid: Peace Walker HD,” Game UI Database. Accessed: Feb. 06, 2026. [Online.] Available: https://www.gameuidatabase.com/gameData.php?id=530
  • [4] Game UI Database, “Metal Gear Solid V: The Phantom Pain,” Game UI Database. Accessed: Feb. 06, 2026. [Online.] Available: https://www.gameuidatabase.com/gameData.php?id=98
  • [5] K. Saunders and J. Novak, Game Development Essentials: Game Interface Design. Clifton Park, NY, USA: Thomson Delmar Learning, 2007.

by anja.kljajic -

User Interfaces in Video Games 6/10

game genres

User Interfaces in Video Games – The quest for genre-appropriate and usable game UI

In my last post I introduced the concept of diegetic, non-diegetic, spatial and meta interfaces. You may have noticed that some of them were mostly tied to specific genres, and since my thesis would like to explore how interfaces can be genre-appropriate, I thought it would be appropriate to introduce these genres.

Game Genres

The following graph [1] shows the most played game genres in the 2nd quarter of 2025. There’s a wide range of genres from shooters to puzzles and what I find interesting is that the most successful genres would have the most people, which means that the UI probably also caters to the masses.

Graph 1: Share of video gamers worldwide who have played games in select gaming genres in the past 12 months as of 2nd quarter 2025
Source: [1]

Now let’s take a look at some of the most important genres as well as a few visual examples of them to get a good idea of the UI elements and styles.

Action Adventure – Action games encompass an incredibly large arsenal of sub-genres, such as action-adventures, platformers, stealth games etc. and rely on good hand-eye coordination [2]. The UI here can range depending on the sub-genre and focus of the game but do often include health bars, currency and ammo indicators.

Figure 1:
The Legend of Zelda: Breath of the Wild
Source: [3]
Figure 2: Grand Theft Auto V
Source: [4]

Puzzle – Another genre that relies on hand-eye coordination, albeit very different in style. Puzzle games rely on simple logic and have thus seen a big audience within the casual mobile games scene. Brighter colours help with differentiation for quick mental grouping of objects.

Figure 3: Candy Crush Saga
Source: [5]
Figure 4: Tetris Ultimate
Source: [6]

Racing – Racing games rely on mini-maps to mirror navigational devices inside the car. Another important UI element within the HUD is the speed, position and lap indicators. Racing games increasingly use minimal and generic design, especially in other screens such as customisation screens.

Figure 5: Forza Horizon 5
Source: [7]
Figure 6: Need for Speed Heat
Source: [8]

RPG (Role-Playing Game) – Role-playing games came from table-top games, where statuses, inventories and treasure finding are important. They’ve come a long way from the classical parchment style of fantasy games to more daring and modern UI with JRPGs. Inventories and managing them play a big role, which is why designing them in an intuitive way can make or break the immersion.

Figure 7: Baldur’s Gate II
Source: [9]
Figure 8: Persona 5
Source: [10]

Shooter – Shooters have been the most popular genre in the past year, so they have to appeal to the largest player base with their UI as well. The elements that are indicative of shooters are crosshairs (the aim indicator in the middle of the screen to help with accuracy) and the ammo and health gauge. A newer development is the shop screens as games shifted to in-game purchases.

Figure 9: Call of Duty: WWII
Source: [11]
Figure 10: Fortnite
Source: [12]

Simulation – Simulation games let players experience real-world situations. They have a massive range, but the two examples I picked are Microsoft Flight Simulator and The Sims 4. Flight Simulator relies on UI similar to racing games, but on another level. It has way more indicators and even more complex UI to mirror actual planes. Sims 4 has a contrasting bubbly style and famously uses spatial UI to help with customising the environment.

Figure 11: Microsoft Flight Simulator 2024
Source: [13]
Figure 12: Sims 4
Source: [14]

Strategy – Games that require thinking and planning, such as Sid Meir’s Civilization and the Age of Empires series [2]. These games show a very good example of the importance of spatial UI. Dealing with the “god” perspective makes individual pieces of land important to discern using markings and icons. Another UI element I’ve noticed is graphs and charts to show post battle results.

When it comes to the style one can notice that the older game [Figure 12] uses a historical, parchment-like style, while the more modern game [Figure 11] uses a more generic and cleaner style.

Figure 13: Civilization V
Source: [15]
Figure 14: Age of Empires II: Definitive Edition
Source: [16]

References

by anja.kljajic -

User Interfaces in Video Games 5/10

types of game interfaces
visual representations
diegetic

User Interfaces in Video Games – The quest for genre-appropriate and usable game UI

Now that we have a basic understanding of the different types of HUD and UI elements within video games, I’d like to share some interesting considerations about the the functions and different types of game UI.

Types of Visual Representation

Diegetic vs Non-Diegetic UI

During my research the most interesting thing I came across is the idea of diegetic user interfaces. Diegetic is defined as “existing or occurring within the world of a narrative rather than as something external to that world [1]“. This usually refers to cinema with the example of in-universe music, but in the video game context user interfaces can also take on this role. Some UIs feel like they’re slapped on the screen without consideration of the world they’re in, while others barely feel like utilities for the player with how immersive they are.

This four-part framework helps with differentiation and categorisation by asking two questions:

  • Is the UI part of the game world?
  • Is the UI part of the story?
Figure 1: The Four Types of Visual Representations of Interfaces
Source: [2]

Non-Diegetic – User interfaces that AREN’T part of the world and AREN’T part of the story [3]. The player character isn’t aware of the UI and it is only there to aid the player. An example would be Baldur’s Gate 3, with the spell inventory, mini-map and party information.

Figure 4: Baldur’s Gate 3 UI
Source: [4]

Spatial – User interfaces that ARE part of the world and AREN’T part of the story [3]. On the example of the Sims 4, the game characters aren’t aware of the diamond icon above their head. This type of UI typically aids the player when selecting different objects in the world, such as the door in the example below.

Figure 4: Sims 4 UI
Source: [5]

Meta – User interfaces that AREN’T part of the world and ARE part of the story [3]. The go-to example here are blood splatters in shooter games. They reflect the status of the player/player character and utilise the link between the game and player.

Figure 4: Killzone 2 UI
Source: [6]

Diegetic – User interfaces that ARE part of the world and ARE part of the story [3]. The player character is aware of and can interact with these elements. The most famous example of this is Dead Space, where the health bar is integrated into the back of the suit of the player character. There are no other UI elements overlayed on top of the screen for maximum immersion.

Figure 4: Dead Space UI
Source: [7]

References

by anja.kljajic -

User Interfaces in Video Games 4/10

UI elements
interfaces

User Interfaces in Video Games – The quest for genre-appropriate and usable game UI

After getting a bit more familiar with how the need for UI in games emerged through a bit of history, I want to get more theory focused now with some definitions. I’ve already mentioned HUDs in my last post so let’s get it all on paper, as well as clear up any abbreviations that may keep popping up.

UI Elements

Definitions

UI (User Interface) – User interfaces are “part of a computer and its software that people can see, hear, touch, talk to, or otherwise understand or direct” [1]. In the video game context, this refers to all interface elements such as menus and setting screens, but also HUDs [2]. Importantly, this also includes the hardware, such as controllers or the keyboard and mouse.

HUD (Heads-Up Display) – HUDs are the persistent UI elements that appear on screen, usually indicating the status of the player [3]. This is the term that is most widely used within game spaces, but only really refers to the overlay during the actual gameplay, and not the other screens mentioned above.

GUI (Graphical User Interface) – GUIs are user “visual systems that let users interact with digital devices through elements like buttons, icons, and menus” [4]. I had some confusion with this term popping up, as I thought it was also the abbreviation for Game User Interface.

Figure 1: Battlefield HUD
Source: [5]

Game UI Elements

In this section I would like to make a distinction between HUD UI elements and other game screens, since all of this is part of the UI, the difference being that HUDs are active during gameplay. As games have evolved, these elements have became staples of the user interfaces within them.

The following are some visual examples of individual HUD elements as well as game screens (on the example of the 1999 game Metal Gear Solid), providing a short overview of the elements.

HUD Elements

  • Health bar – shows the current life the player has remaining, often abbreviated as HP (Hit Points), which came from the tabletop role-playing game Dungeons & Dragons [6].
    • health
  • Score – shows the current or final score accumulated.
  • Ammunition gauge – shows the number of bullets/projectiles available, important for shooters.
  • Inventory – shows items possessed/equipped, important for role-playing games.
  • Map/Radar – assists player with navigation by providing an on-screen means of navigation with “you are here” indicators.
  • Context-Sensitive Prompt – text or icon that appears when the player is near an object that can be interacted with [7].

Figure 2: Elements of a HUD
Source: Own Production, referenced from [7]

Game Screens

  • Title Screen/Start Screen
  • Pause Screen
  • Options/Settings
  • Save/Load Game
  • Controls
  • Game Over Screen
  • Loading Screen
  • Legal/Copyright
  • Credits
Figure 3: MGS Title Screen
Source: [8]
Figure 4: MGS Pause Screen
Source: [8]
Figure 5: MGS Options
Source: [8]
Figure 6: MGS Load Screen
Source: [8]
Figure 7: MGS Controls
Source: [8]
Figure 8: MGS Game Over
Source: [8]

References

by anja.kljajic -

User Interfaces in Video Games 3/10

game history
interfaces

User Interfaces in Video Games – The quest for genre-appropriate and usable game UI

Last time I answered the question of what the first video games were, namely Tennis for Two and Spacewar!. Today, I’ll be taking a step further and covering the universally known earliest video game Pong, as well as taking a quick look at the progression of games and their interfaces further.

Early Game Interfaces (Arcade and Home Consoles)

Are we getting to the UI yet?

Yes! Games finally became commercially successful and available to the general public outside of science fair and university contexts thanks to the shift away from giant computers that weighted dozens of kilos and cost thousands.

1972 – Pong

Figure 1: Pong on the TV
Source: [1]

Pong was developed by Atari which was a company formed by Nolan Bushnell and Ted Dabney, and programmed by Al Alcorn. It used dials for vertical movement of paddles where players competed to hit the ball back and forth [2]. This is where we notice the first truly UI element of a game, which is the score counter on the top of the screen.

1978 – Space Invaders

Figure 1: Space Invaders Cabinet
Source: [3]

Released 20 years after Tennis For Two, Space Invaders was developed by Tomohiro Nishikado and it marks the rise in popularity of arcade games, with two buttons for moving left and right and one for firing projectiles [4]. In this new age of arcades, one of the very first innovations in game interface design, the high score and the high score screen, were born [5]. The high score was a different motivation compared to the other games mentioned so far which were based on competition between two players. Space Invaders and many popular arcade machines featured single player experiences, where the high score screen would incentivize not only beating your friends but also beating your own score. Arcades were the place where HUDs were born, with more and more permanent UI elements aiding players emerging.

As much as I would love to talk about the entire history and progression, and go all the way from Pac-Man to modern games, I’ll keep it to these two blog posts. They will serve as a sort of starting point for when I dive deeper into the history for my actual thesis.

References