1. Introduction

For digital humanities scholars, breaking down a videogame into its component parts may seem like an obvious strategy for better understanding it. Many of these researchers are players themselves, after all, and the act of play implicitly invites this sort of systematic deconstruction.

As a player engages with a videogame, they accumulate rules knowledge, acquire in-game resources, and develop a tacit understanding of what the game asks of them, slowly inching toward a deepened comprehension of the entire play experience. Only through this accumulation of resources and expertise can they improve their performance within a videogame, eventually completing objectives and perhaps even beating a title in its entirety.

From an academic perspective, this deconstructive learning process serves a purpose other than game mastery. Instead of forging a better understanding of a game through play (as intended by the developer), technically-minded scholars can separate a videogame into its component parts, which can then be arranged and rearranged to facilitate the needs of their research. This process can be as simple as using save files or cheat codes to access parts of a game strategically, a process that often manifests as targeted or repeated playthroughs, or can involve delving into a game’s code and assets in an attempt to peek behind the curtains—scrutinizing the numerous individual elements that make up a videogame title. It is this latter form of analysis that we are interested in exploring within this paper. By accessing game assets directly, rather than solely through play, we believe that scholars have the opportunity to analyze videogames in new ways—not just as a series of set pieces or vignettes pre-determined by the original developer. When broken into parts and viewed as a collection of diverse assets, rather than a homogenous whole, new angles of research become possible through the adoption of both established and emerging forms of quantitative analysis.

This paper discusses one potential avenue for this type of videogame analysis by documenting the creation, functionality, and potential applications of GameSound, a digital humanities database project developed between 2017 and 2018.¹ We developed GameSound as a prototype ludomusicological database, with the intent of providing users with easy access to the music and sound effects present within videogames. By making these audio files accessible through a web-based interface, and supplementing them with technical and contextual data, our hope is that GameSound could be used to facilitate new types of academic research. Using Civilization IV as a case study and HEURIST to build the database, GameSound currently provides access to over 2000 music and sound effect files. This paper begins with a brief overview of quantitative game analysis, both as it is defined for this particular digital humanities project and how it has been used in other projects, specifically Jason Bradshaw’s “BioShock Infinite and Feminist Theory: A Technical Approach.” We then discuss the functionality of the database itself, while also outlining the technical, legal, and theoretical challenges that arose while designing it. Finally, we hypothesize where this research could lead in the future: both for GameSound and other similar projects.

However, we would be remiss to begin these discussions without first touching on the academic field that stands behind GameSound, ludomusicology. An emerging sub-discipline of musicology, ludomusicology focuses on the academic study of the audio present in videogames. Primarily concerned with the direct study of a videogame’s music and sound effects, ludomusicology also interrogates how we study audio within the context of digital software. Since the neologism was coined in 2007, the field has expanded to include music games, fan cultural music practices, live concerts, and the impact that game music has had on other musical genres (Dudley 2018). Ludomusicological scholars may discuss how the idea of diegesis is complicated by the interactivity of videogames (Kamp 2016), the impact of music games such as Taiko: Drum Master and Bloom (Kassabian and Jarman 2016), or the tension between art and entertainment that arises when videogames remix classical music (Gibbons 2016). Like many topics associated with the digital humanities, ludomusicology traverses disciplines: fostering collaborations with computer science, film and media studies, and communications. GameSound is our first foray into this type of research, originally conceived as a digital humanities class assignment at McGill University, and we believe the database has potential academic, professional, and hobbyist applications.

2. Quantitative videogame analysis

We describe GameSound as a tool for enabling new types of quantitative videogame analysis, but we also acknowledge that the term “quantitative” can be vague and requires further elaboration. In the context of GameSound, it reflects an approach that is comprehensive (collecting all of the audio in a given game), measurable (defining collected game audio using quantities and simple identifiers), and statistical (enabling the comparison of data using tables, charts, faceted searches, and visualizations). In contrast to qualitative videogame analysis methods, which commonly rely on written logs constructed through repeated playthroughs (Consalvo and Dutton 2006), quantitative analysis is much more tool-oriented and focuses on parameters that can be measured or counted. Although a somewhat new approach, quantitative videogame analysis can be facilitated by adopting existing digital humanities tools and established methods that have their origins in fields such as literature, history, and philosophy. The practice of isolating and extracting specific words and phrases has enabled new types of distant reading in literature, for example, and many of the methods we use to analyze books can be co-opted for analyzing various aspects of a videogame, from text to media assets.

Jason Bradshaw provided an excellent example of this type of analysis at the Congress of the Social Sciences and Humanities conference in 2018, with his presentation “BioShock Infinite and Feminist Theory: A Technical Approach.” Bradshaw collaborated with fan communities to acquire a complete written script (featuring nearly 250,000 words) from the videogame BioShock: Infinite, which then served as the corpus for his project. Influenced by a close reading of the game by Catlyn Origitano, who analyzed the representation of female protagonist Elizabeth (Origitano 2015), Bradshaw wanted to demonstrate how distant reading could corroborate Origitano’s qualitative analysis. He also had a strong desire to expand traditional distant readings of text to include videogames: “Why stop at the traditional textual mediums historically studied in the humanities? New types of digital analysis can also lend themselves to mediums born of the digital age” (Bradshaw 2018). After acquiring the game’s script, Bradshaw fed the entirety of the text into Voyant Tools—a popular piece of textual analysis software—to determine the frequency of certain words and where they occurred in the game’s timeline. Bradshaw tracked how Elizabeth was referred to throughout the game as well as how she was treated by other characters, resulting in a measured, textual character arc. By paying keen attention to labels such as “baby” or “child,” he analyzed her function in the storyline and how it could be “an allusion to the treatment of women in contemporary society” (Bradshaw 2018). Although by no means an authoritative take on the game’s narrative, the study demonstrates a progressive methodological approach for game studies: one that invites complementary quantitative analysis using methods generally reserved for literary works.

Much like Bradshaw’s work, GameSound also takes a deconstructive approach, but focuses on the sound within a videogame rather the text. Sound was chosen as the focus for the project as we felt that it was one of the more difficult assets of a videogame to isolate—often framed as a secondary element to mechanics, visuals, or narrative—and because of our own experiences in professional sound and music theory. By putting an emphasis on searchability and utilizing a variety of both technical and ludological descriptors, GameSound aims to facilitate access to a videogame’s audio elements without having to rely on playthroughs or secondary resources. Once the heavy lifting of data input and formatting is complete, GameSound users have the freedom to access audio through faceted searches, create custom reports, and even embed content on the web for easy collaboration and knowledge dissemination. This approach allows for a rapid oscillation between different types of analysis, some of which may not have otherwise been available. We have outlined three speculative use cases for the database listed below:

1. Game studies scholars, who may be interested in tracking the prevalence and use of certain sound effects within a videogame (such as gunshots or dialogue snippets) could sift through the database using filters or a keyword search. Being able to point to the quantity of sound effects recorded in a game, as well as how they are triggered, could serve as a valuable point of analysis for speculating on a developer’s priorities when designing a game.

2. As GameSound allows users to quickly survey the parameters of the audio found in a particular game, professional videogame developers may wish to use the database to better guide their own efforts. By sorting, counting, and listening to audio files by type (sound effects, music, etc), developers could use existing games as a blueprint for determining how many audio recordings they need for their own projects, how long each recording should be, and how many variations of the same sound may be required.

3. Sound scholars, perhaps inspired by Jonathan Sterne’s The Death and Life of Digital Audio, could use GameSound to analyze how both data compression and audio compression have been applied to videogame sound. Sterne has discussed the effects of compression through his documentation of the music industry’s loudness wars (Sterne 2006, 345), but little of this research has been carried over to the videogame industry.

Although GameSound primarily focuses on quantitative research opportunities, an additional advantage of the database is its ability to expedite various types of listening exercises. GameSound allows scholars to listen directly to sound effects or music from a videogame, enabling approaches such as Michel Chion’s conception of reduced listening—a mode of listening that focuses on repetition and the removal of visual context (Chion 2012). This type of listening would be difficult, if not entirely impossible, through normal playthroughs where sounds cannot be divorced from their accompanying visuals, are usually layered with a multitude of other audio tracks, and can only be triggered when certain objectives or criteria are met.

3. Constructing and using GameSound

GameSound was created using HEURIST, a free platform for scholars in the digital humanities that enables online database construction, mixed-media assets, and dynamic data visualization. Like many digital humanities tools, HEURIST pursues a certain level of accessibility for its user base:

HEURIST’s research-driven data management system puts the user in charge, allowing them to design, create, manage, analyse and publish their own richly-structured database(s) within hours, through a simple web interface, without the need for programmers or consultants. (Sydney University 2018)

After some experimentation with other tools, we chose HEURIST for this project primarily due to its ease-of-use. While we may be studying a subject that is steeped in code, we are certainly not experienced computer engineers or web developers (nor did we want to place this expectation on our collaborators). Thus, it was necessary to find a database tool that would take care of most of the heavy lifting for us while providing a very shallow learning curve for additional participants. In a way, this embodies one of the key design sensibilities for GameSound: a desire to create a useful resource that, at the same time, is easily accessible for game scholars, ludomusicologists, and independent researchers.

The ultimate goal for the database is to include videogames from different platforms and eras, but the initial dataset focuses entirely on the sounds found within the 2005 computer game Civilization IV. Civilization IV was chosen for the prototype for three main reasons: First, it possesses an incredibly open programming architecture in which developers have enabled transparent access to the game’s assets—making both data extraction and interpretation simpler than in comparable titles. Secondly, Civilization IV was the first computer game to be nominated for (and win) a Grammy, granting it a special place in the history of game studies while affirming a certain level of cultural significance. Finally, the title’s availability across various platforms and marketplaces ensured that we could acquire the game without having to seek out additional hardware. In contrast, videogames that are exclusive to a specific era or console would have presented severe challenges in both acquisition and access. Super NES games, as an example, were released on proprietary cartridges and stored their music in heavily compressed formats that are difficult to access. GameSound currently facilitates access to Civilization IV’s audio in two ways: faceted searches (Figure 1) and reports (Figure 2).

Figure 1

A screenshot of GameSound’s faceted search.

Figure 2

A screenshot of a single database entry.

GameSound’s faceted search functionality allows users to explore Civilization IV’s audio files through a web browser, by activating and deactivating filters presented within a column on the left side of the screen. These filters range from technical parameters (such as file type and sample rate) to ludomusicological ones (such as IEZA classification and sound type). Additionally, users can search for a specific piece of audio by filename, or simply sift through all of the game’s audio in a linear fashion by scrolling through the list-view. Clicking on a single entry in the list-view will bring forth additional information about an audio file, such as file size and duration, and loads an audio player that plays back the selected sound. Some entries also contain a screenshot or video link, both of which document one of the many possible situations in which the sound can be triggered in-game.

Through the use of HEURIST’s back-end tools, GameSound users are able to export customized reports that contain entries determined by parameters that exist within the database. Seen in Figure 3 are two excerpts from an enormous report that contains every single GameSound entry that possesses both a screenshot and a video link. Reports are quite flexible in both content and layout—as this report is embedded on GameSound’s homepage, it has been customized to mimic the font styles from the website’s CSS files. Able to query any type of data that is present in the database, reports are a versatile way to share targeted sets of data with other researchers.

Figure 3

Two entries from a custom GameSound report.

In addition to the database itself, GameSound also exists as a web resource for those who are interested in ludomusicology and quantitative game analysis. The website currently hosts documentation outlining the project’s methods and explains how many aspects of the research were conceived. Similar to how the database provides transparent access to videogame audio, we hope that the website provides insight into how GameSound was developed.

4. Challenges in designing the database

Ludomusicology can be a complicated endeavour for researchers, scholars, and digital humanists. Conventional qualitative methodologies prevalent in ludomusicology, such as analytical play (Summers and Hannigan 2016, 52), can lack the capacity to adequately access and isolate audio or investigate the role of interactivity—the layers “between the operations of a machine and the instructions given to it by an operator” (Burdick 2012, 53). Commercial soundtracks and other official releases of game audio can be somewhat unreliable sources, as composers may alter the recordings from their original presentation (while completely divorcing them from their in-game context). Thus, one of the key challenges in creating a ludomusicological database is gaining access to videogame audio directly without dismissing its role within gameplay. This challenge is exacerbated by the opacity of videogame file structures—much of a game’s audio assets are obfuscated through layers of code, file compression, and technical protection measures. There is a notable lack of academic tools that can penetrate these layers, and most of the existing software used to extract audio and game code falls squarely into the realm of modding or hobbyism. Thus, researchers must seek out games with open file structures (as we did with Civilization IV) or use independently developed tools that have very little documentation or support.

In addition to the technical challenges of acquiring audio and determining its purpose within the framework of the game, one of the biggest hurdles faced when designing GameSound’s initial prototype was deciding which data types we would include in the database. As GameSound was not created with a specific research project in mind—focusing more on potential applications—much of the initial data gathering was speculative in nature. As a result, we experimented with an extremely broad range of technical and ludomusicological data throughout the database’s development. This, admittedly, may have led to some arbitrary decisions regarding data types, but gave us permission to contemplate an enormous variety of potential applications for the database. Code snippets and written descriptors were both strongly considered during the prototyping process before we whittled down the selection to its current state, which focuses more on measurable technical parameters, game media, and ludomusicological identifiers. These data types were selected for practical reasons, as much of their extraction could be automated, as well as their perceived usefulness for digital humanities scholars.

In an effort to better place GameSound within current ludomusicological discourses, instead of developing our own identification system we decided to adopt the IEZA framework—a two-dimensional method for describing sound in computer games. Designed by Sander Huiberts and Richard van Tol at the Utrecht School of the Arts, the IEZA framework (Figure 4) provides an effective vocabulary for audio classification. The vertical dimension in the framework makes a distinction between audio originating from inside the fictional game world (diegetic), such as the footsteps of a game character, and sound coming from outside the fictional game world (non-diegetic), such as the title’s musical score. The horizontal dimension separates sounds that result from direct player action (activity) such as those triggered from clicking buttons within a game’s interface, and from sounds that are ambient (setting) such as atmospheric and music tracks. Four domains are formed across these two planes of comparison: Interface, Effect, Zone and Affect (Huiberts and Van Tol 2008), with the authors providing some key examples of sounds that fall within these domains:

• Effect: sounds rooted in the game world that are triggered directly by player action, such as dialogue, footsteps, and gunshots.

• Zone: sounds rooted in the game world that are not directly triggered by player action, such as rain, wind, or city noise.

• Interface: sounds that exist outside the game’s fictional setting that are triggered directly by player action, such as beeps and clicks emitted by a game’s menu or HUD.

• Affect: sounds that exist outside the game’s fictional setting that are not triggered directly by player action, such as a game’s musical score or ominous drones in a horror game.

Figure 4

IEZA Framework: Sanders Huibert and Richard van Tol.

These categorizations help to establish the importance of interactivity in videogame audio—the simple fact that “the body cannot be removed from the experience of videogame play” (Collins 2013, 3). Players are not just the receiver of a sound signal—as with radio, television, and film—but also the transmitter. In-game actions may trigger dialogue, sound effects, music, and ambient sounds directly (by clicking a button in the interface) or indirectly (through timed events or algorithms). Thus, by adopting the IEZA classification system, we simultaneously acknowledge the uniqueness of game audio while also encapsulating it within a simple set of database parameters. When used in conjunction with the existing technical categories found within the database, these domains provide valuable context to the audio files found within GameSound.

In addition to ludomusicological challenges and database design conundrums, Canadian copyright law poses two intriguing hurdles for GameSound. First, the music and sound effect files in the database are, naturally, the intellectual property of the original game developer and are used without explicit permission or licensing. Although there is a strong argument that GameSound’s acquisition and use of these files falls under the educational aspects of fair dealing—“a user’s right in copyright law permitting use of, or ‘dealing’ with, a copyright protected work without permission or payment of copyright royalties” (Simon Fraser University 2018)—copyright law is difficult for researchers to pin down and academic institutions are often wary of projects that engage with it. Despite its current availability as a public resource, access restrictions may need to be introduced as the database expands in order to mitigate legal risks or to appease university ethics departments. Secondly, the open file structures present in Civilization IV can be considered an outlier in a media industry that is shifting toward tighter control of videogames and their assets. Publisher mandated terms-of-use and security measures, such as End User License Agreements and Digital Rights Management software, often create legal and technical barriers in accessing game data. This not only complicates the idea of fair dealing but creates additional challenges for the research team as they attempt to access and recover game assets that may be encrypted or hidden behind software restrictions.

5. Conclusion

An obvious question for GameSound is: “what are the next steps for the project?” After receiving feedback on the working prototype from scholars at the McGill Music Graduate Students’ Society Symposium and the Congress of the Humanities and Social Sciences, expanding the dataset to include additional videogames seems to be the obvious path forward. We are currently canvassing the game studies community to seek out collaborations with those who may find GamesSound’s unique toolset useful for their own research projects. This expansion is important as, although the development process has been enlightening, a digital humanities project such as this cannot be truly evaluated until it has moved beyond speculative use and has been tested in multiple academic research projects.

As with the prototype, any potential collaboration would likely revolve around the addition of a single videogame. Much like we did with Civilization IV, database updates will focus around the acquisition of audio assets from a game, followed by a sorting process (based around the researcher’s needs as well as the existing categories present in the database). This fresh infusion of data will offer new opportunities for introspection and revision, and the database may be altered or completely rebuilt to include new categories, data types, or search functionality (such as the ability to quickly navigate between titles). Essentially, our goal would be to allow new collaborators to sift through the videogame of their choice with ease, providing them with access to the audio elements of a game without relying on repeated playthroughs or unreliable secondary sources. As with any addition to the database—which, barring copyright concerns, will always be publicly available—additional data will also provide scholars from all across the game studies and digital humanities communities with an opportunity to reflect upon the technical parameters of game audio and to experiment with quantitative game analysis tools and applications. As the database moves toward a multitude of videogames, rather than a single one, we will be presented with further opportunities to measure its value as an ongoing digital humanities project.

GameSound is an iterative work. Beyond the research possibilities that it provides, it is meant to explore the ongoing technical challenges ludomusicologists face, such as data accessibility, intellectual property concerns, and the lack of established standards. Over time, we hope to collaborate with scholars across the world to expand both the scope and functionality of the database, while constantly interrogating its efficacy as a research tool. Just as we hope to learn more about videogames by breaking them down into their component parts, it is our belief that by deconstructing and reconstructing our own work we can unearth valuable insights for scholars across various disciplines.

Competing Interests

The author has no competing interests to declare.

Editorial Contributions

Section/Copy editor, text: Darcy Tamayose, University of Lethbridge Journal Incubator.

Copy editor, bibliography: Shahina Parvin, University of Lethbridge Journal Incubator.