Introduction
We base this work on the premise that images reflect socio-political-historical moments and their underlying power structures in novel ways, often overlooked by analyses that prize written language as the most authentic source of historical knowledge. Too often, as Peter Burke notes, “historians … prefer to deal with texts and political or economic facts, not the deeper levels of experience that images probe” (Burke 2001, 10). We begin with the proposition that political, economic, and cultural facts in the twentieth and twenty-first century are mediated through, and consecrated by, technologically reproduced—and increasingly, technologically generated—imagery. Ours is, as Guy Debord famously argued, a “society of the spectacle,” and so the traditional historian’s “condescension towards images” is becoming increasingly problematic in methodological terms (Debord 1995).
In addition to this, the visual historian’s toolkit has been radically expanded in recent years. Previously, it was difficult—if not impossible—to track and analyze the modern flood of imagery that has shaped our world, beginning in the twentieth century. As Lev Manovich argues, new digital technologies allow for the “computational analysis of cultural data,” which allows scholars to apprehend the true “scale” of modern visual production for the first time (Manovich 2020, 1). Once limited to close readings of a handful of select images, scholars can now recognize previously undetected or overlooked trends using digital methods on large collections of historical images, a development that has been dubbed the visual digital turn (Wevers and Smits 2020). The ability to leverage digital analytics to interpret and study images has defined the subdiscipline of the visual digital humanities (Münster and Terras 2020), in which this work belongs. In this study, we apply cultural analytics techniques to a collection of 3,389 issues of Time magazine, ranging from 1923 to 2014.
In order to grasp and analyze Time’s vast visual archive, we have used a combination of machine learning and human verification to identify large-scale visual trends, which we have supplemented with close readings of individual photographs and advertising images. This hybrid approach allows us to capture the magazine’s complexity at both the macro- and micro-levels. In this, we have been inspired by Arnold and Tilton’s distant viewing method. As they explain, the way a computer views images is characterized by a dual meaning of distance: distance from the way a human eye (and mind) understands visuality—for “computational methods “see” by calculating images as numbers”—and in terms of “scale,” or how “[t]hrough computer vision, interpretation of images can exceed a person’s physical ability to view and remember” (Arnold and Tilton 2023, 11–12). At the same time, distant viewing makes explicit how the computational interpretation of images is reliant upon human understanding. As Arnold and Tilton insist, “distance is not objective. Viewing makes explicit that the computational processing of images is shaped by ways of seeing and practices of looking and, therefore, by a set of social and cultural decisions … computer vision is a technology of communication shaped by people and imbued with cultural and social values” (Arnold and Tilton 2023, 12). We have found that Time is an ideal vehicle for this fruitful approach to visual culture.
Has any object been photographed more often than the human face? Surely not. This is no accident, for no earthly object carries as much inherent human meaning as the face. This phenomenon has deep and universal physiological roots. As Simon Schama explains, the face is the first thing that newborns engage with. The relationship between one’s eyes and another’s face are, Schama argues, the “elementary social wiring” that anchors human cultures everywhere (Schama 2016, chap. 1, Kindle). It’s a relationship that can, however, be “rewired” in different contexts. Modern advertising, as historian Jackson Lears points out, is unthinkable without depictions of sincere human faces establishing trust between seller and buyer. Photographs of faces are also central to modern politics (Lears 1994, 60). For Roland Barthes, “the [political] candidate’s effigy establishes a personal link between electors and the candidate,” in which “not his projects but his motives … as well as mental characteristics” come to the fore as “an entire style of being of which he is simultaneously the product, the example, and the appeal” (Barthes 2012, 181). In the modern era, the facial image becomes a tool that erases politics as a “body of problems and solutions” in favour of personalized imagery (Barthes 2012, 181). Morgan Klaus Scheuerman, Madeleine Pape, and Alex Hanna draw our attention to the use of faces in colonial rule, in which facial features of the colonized were subjected to “practices of measurement and classification” that established racial and gender hierarchies, legitimizing colonialism as a political and economic project (Scheuerman, Pape, and Hanna 2021, 4). They also point out that this physiological-political strategy has been given distressing new life in our technologically driven twenty-first century in the form of “auto-essentialization,” in which “automated facial analysis projects … are reinscrib[ing] … the colonial order by reproducing—and automating—essentialized (and racialized) notions of gender/sex difference” (Scheuerman, Pape, and Hanna 2021, 8). Our attachment to human faces begins in innocence, but before long, it’s pulled this way and that by complex influences that are often far from innocent.
Time magazine, from its inception in 1923, evinced a unique understanding of the political, commercial, and colonial (and post-colonial) power of the human face. Its first issue, for instance, featured an illustration of the Speaker of the U.S. House of Representatives Joseph Cannon—a powerful politician presented in a manner that radiated dignity, eliding the more concrete machinations underlying Cannon’s political authority, in the terms Barthes outlined. Time also broke ground through its innovative mixture of commercial imperatives and political affairs. The two were, in fact, often difficult to pry apart in its pages. The magazine’s literary “house-style”—dubbed “Timestyle” by its critics—was a popular translation of the high-cultural prose its editor, Briton Hadden, and its publisher and editor-in-chief, Henry Luce, venerated at Yale College (now Yale University). They blended this with the rapidly developing practices of American advertising, both written and visual. For Hadden, and particularly Luce, serious journalism and commercial imperatives could be squared in Time, as a uniquely American, eminently modern magazine. Finally, Time had a marked global slant. This is in large part due to Luce’s often overbearing editorial influence. The child of Christian missionaries operating in China, he made sure that his magazine attended to the non-western world, as evidenced by Time’s international editions printed for overseas circulation. This was a sympathy he extended to racialized peoples in the United States—with important qualifications—in a way that made Time unique among American magazines.
In a recent peer-reviewed publication, we examined the representation of female faces in Time from the 1940s to the 1990s, supplementing our data with historically informed close-readings of the magazine’s visual and textual content. We found that the percentage of faces that present as female was closely aligned with larger currents in women’s history and the history of feminism (Jofre, Cole et al. 2020). More specifically, we found that the percentage of female facial images peaked as women became more active in public life and waned in eras characterized by anti-feminist backlash. We also noted a change in tone and content as Time, from the 1970s onwards, began hiring and promoting more women as writers, photographers, and editors (Prendergast and Colvin 1986).
In this paper, we expand upon this approach, examining a wider range of attributes over a longer period of time. In total, we analyzed 8,789 faces from a sample of 100 issues of Time, labelled with characteristics including gender, race, age, and context (advertisement or news feature). We found that the overall percentage of smiling faces increased significantly from 1923 until approximately 1940, after which it remained relatively constant through the rest of our dataset. Not surprisingly, the majority of faces found in advertisements are smiling. Looking at gender, we find that women’s faces are more likely to be found in advertisements than in news features, and women’s faces are more likely to be smiling in both news and advertising contexts. We break down the faces by racial categories and find that by the 1960s, the percentage of African American and Asian American faces increased markedly, due to Time’s engagement with key historical events of the era, particularly the Civil Rights Movement and the U.S. War in Vietnam—events that Time’s leadership was keen to both reflect and influence.
Methods
Description of datasets
Our datasets were published and described in the Journal of Cultural Analytics (Jofre, Berardi, Bennett et al. 2020), and are freely available. Briefly, we extracted 327,322 faces from a collection of 3,389 issues of Time magazine ranging from 1923 to 2014. This collection includes more than half of the issues of Time published within that period, sufficient for a wide-ranging exploration of Time’s visual culture. In one dataset, each of the 327,322 extracted faces is tagged with the year, issue, and page, as well as a machine-generated gender classification. A second dataset, on which we base this paper, provides more detailed classifications of a subset of 8,789 of those faces, derived from a sample of 100 issues, sampled at one July issue per year (91 issues) plus one random issue per decade (9 issues). These detailed classifications were done via crowdsourced human labour, and include (in addition to the date, issue, page, and gender) the following categories: age (adult or child), face context (does the face appear as part of an ad or part of a news story?), image type (photo or illustration), race (according to simplified census categories), and facial expression. Table 1 lists all the attributes classified in this dataset.
List of attributes classified on the subset of 8,789 extracted faces, which constitutes the data examined in this paper.
| Column Name | Possible Value |
| Year | Integer: publication year (1923 – 2014) |
| Month | Integer: publication month (1 – 12) |
| Day | Integer: publication day (1 – 31) |
| Page | Integer: page number (1 – 250) |
| Gender | Categorical: Female, Male, or Unknown |
| Adult | Boolean: Yes (1) or No (0) |
| Face Angle | Categorical: Profile or Forward |
| Image Colour | Boolean: Colour (1) or Monochrome (0) |
| Image Type | Boolean: Photo (1) or Illustration (0) |
| Context | Categorical: Advertisement, Cover Page, or Feature Story |
| Multi-face | Is there more than one face in the image? Boolean: Yes (1) or No (0) |
| Race | Categorical: American Indian, Asian, Black, Pacific Islander, White, or Unknown (where “White” includes Middle Eastern and Latinx) |
| Smiling | Boolean: Yes (1) or No (0) |
Data sampling
We note that although our sample size in this work is relatively small, consisting of 100 issues, sampled at 1 to 2 issues per year, we believe that the 8,789 faces we extracted from our sample comprise a statistically significant set of data that is representative of the larger archive. In previous work, we used machine learning to classify all 327,322 faces from our archive by gender, and we compared the resulting trendline with the gender trendline of the human-classified 8,789 faces; we found that the two trendlines matched up reasonably well (Jofre, Cole et al. 2020). In Figure 1 below, we compare the total number of faces per year in our sample to the number of faces per year in our collection. (It should be noted that our collection of magazines, comprising 3,389 issues of Time magazine ranging from 1923 to 2014, does not contain all the issues of Time magazine published in that year range. A comparison of our collection with the total number of issues is provided in another publication [Jofre, Berardi, Brennan et al. 2020]).
Comparison of the total number of faces per year extracted from our entire collection with the total number of faces per year extracted from our sample.
We chose to sample in the same month and week each year to avoid errors that could come from sampling at different times of the year (a holiday season issue might look different from a summer issue, for example). Choosing the same time every year also avoids differences that could be caused by the time of year. July was an arbitrary choice. We chose the second week of July to avoid 4th of July.
Data collection: Extracting the faces
Our procedure for extracting faces from the archives has been described in detail elsewhere (Jofre, Cole et al. 2020; Jofre, Berardi, Bennett et al. 2020; Jofre, Berardi, Brennan et al. 2020). In brief, we used human labour via Amazon Mechanical Turk (AMT) to collect a training set of data (approximately 4,000 faces); then RetinaNet detector was used to extract faces from the remainder of the archive (Lin et al. 2018). RetinaNet was originally designed to detect objects in a scene, but we modified it to detect faces. It outputs candidate bounding boxes around prospective faces, accompanied by a confidence metric. Candidate faces were filtered using a threshold of 90% confidence to produce the 327,322 faces.
Data collection: Human-generated classification
Our data-collection and verification procedures have been extensively documented and published elsewhere (Jofre, Berardi, Brennan et al. 2020), along with our custom software. We provide a brief description here. To collect the data, a custom interface was deployed via Amazon Mechanical Turk (AMT) to enable human labourers to classify faces according to the categories in Table 1. Each worker was given a randomly selected batch of 25 pages, each with a clearly highlighted face to be categorized. Three of the 25 pages were verification pages with known features, which were used for quality control. Each face was labelled by two distinct human coders, determined at random so that the pairing of coders varied with each image. A proficiency rating was calculated for each coder by considering all images the coder annotated and computing the average number of labels that matched those of other coders. Coders were also asked to label the quality of each image, and prior to calculating the proficiency score, all faces that were tagged as having “Poor” or “Error” image quality by either of the two coders were eliminated. The finalized dataset was created by resolving inconsistencies between the two image coders by selecting the labels from the coder with the highest proficiency rating.
Interpretation methods
To interpret our data, we start with visualizing the data by plotting aggregate percentages per year, as presented in the charts throughout this paper, applying a LOWESS (Locally Weighted Scatterplot Smoothing) technique to reveal the trendlines in the data. We then turn to secondary sources to contextualize the trends revealed by the data and examine relevant issues and articles of Time magazine to verify that the publication’s output fits within our described frameworks, providing supporting evidence for our arguments.
Findings
Facial expressions and context
Contemporary advertising frames the world in a positive light. As historian of advertising Philip Marchand notes, until the 1930s, advertising was caught between two poles—staid descriptions of products and how they were made (“ads depicting the bewhiskered founder and his factory”) and what amounted to scare campaigns, in which potential consumers were frightened by an affliction they had heretofore been unaware of—the classic case being “halitosis,” or “bad breath,” as the problem; Listerine brand mouthwash as its happy solution (Marchand 1985, 24; 18). By the 1930s, a new, more potent advertising approach had taken shape. Instead offering up a bland, lifeless series of commercial facts, or evoking banishment from polite society, advertisers began to shift towards “empathetic depiction of consumer experiences, instead of the product itself” (Marchand 1985, 23). This innovation saw everyday problems wrapped “in the tissue of a dream,” and lifted into the realm of capitalist utopianism (Marchand 1985, 24).
This was also when photographs became more widely used in advertisements, as cameras became both more technologically advanced and cheaper to use. Advertisements could now depict, with novel verisimilitude, people enthralled with what the product or service bestowed upon them, their happy faces in sync with the imperatives of the American economy, soon to be the biggest the world has ever seen. This is consistent with our observation, shown in Figure 2, that faces are more likely to be smiling when they appear in ads as compared to other contexts. Above all else, photographs and illustrations now stressed “situations of fulfillment”—“illustrations of the housewife pushing a vacuum cleaner or otherwise using the product,” followed by “the housewife’s friends blinded by her gleaming floor,” as she used time formerly wasted on domestic matters to take her beaming children “on an outing to pick wildflowers” (Marchand 1985, 24). This “dramatic realism” was designed to make this commercial dream feel real to readers of a magazine like Time, which provided a ready context for such fantasies (Marchand 1985, 24). An important effect of this was, as historian of photography Naomi Rosenblum points out, an erosion over time of the border separating advertising and photojournalism—a phenomenon represented in Time more clearly than any comparable American magazine of the period (Rosenblum 1997, 494–495).
Black – Percentage of faces that are smiling in ads, where 100% = all faces in ad contexts. Grey – Percentage of faces that are smiling in news features, where 100% = all faces in news contexts. Each point indicates the calculated percentage for each year, and the lines represent the LOWESS smoothed data.
The shift in the culture of advertising was accompanied by a shift in the culture of photography itself. The Eastman Kodak company owned many of the trade journals in the field and funded many of those it didn’t, establishing itself as the leading expert on photographic practice, and in that role, it strived to present the photograph’s subject as “happy”—a state of mind represented by smiling—that became an industry standard (Kotchemidova 2005). The effects of this shift can be seen across the cultural spectrum, from ads, to news, to more unexpected places, such as American high school yearbooks (Ginosar et al. 2015). Our data is consistent with this shift in the culture of photography, as we see an increasing proportion of smiling faces between 1923 and 1940, as shown in Figure 3. The data suggests that Time magazine followed (and likely modelled) professional practices in advertising along with social conventions in human representation. While the overall fraction of smiling faces in Time, a serious news magazine, is likely lower than in other contexts such as high school yearbooks, the trendline between 1920 and 1940 illustrates the transition as smiling became normalized in photographs.
Total percentage of faces that are smiling. Each point indicates the total percentage of smiling faces in a given year; the solid line represents the LOWESS smoothed data, and the dotted line is a linear regression fit to the data between 1923 and 1940 (R2 = 0.61).
Observations on gender
A note on gender
While we acknowledge that gender is not binary, this analysis assumes that gender falls into binary male and female categories. We note, however, that our raw data does include an “unknown/other” category for gender, but in a subset of this data that we analyzed, the “unknown” gender category accounted for only about 6% of the faces, and none of these were actually gender non-binary adult individuals: many were not faces at all (errors in the face extraction), many were very small low resolution images that were hard to read, some were non-gendered cartoon illustrations (a face drawn onto an object, for example), and some were infants or small children. So, our sample contains very few individuals that visually fall outside the two main gender categories, which prohibits us from drawing any conclusions about them through quantitative analysis.
Gender and facial expression
We observe, as shown in Figures 4 and 5, that throughout all eras, women are significantly more likely to be depicted as smiling than men. On average, over all years, women are about 1.6 times more likely to be smiling than men. This bias towards smiling women is consistent in multiple print media in the twentieth century (Dodd, Russell, and Jenkins 1999; Morse 1982), and can be understood as the effect of stereotyped gender conventions, where women are expected to be agreeable, even in still photographs. It is significant that this trend appears in Time magazine, a presumably neutral source for news, as opposed to a magazine like Good Housekeeping that transparently perpetuates gendered stereotypes. It is in these “neutral” spaces where the most ingrained biases persist. As late as 2018, in contexts in which gender equality is ostensibly promoted, an overwhelming bias towards images of smiling women remains, where it is presented as a “natural” state of affairs (Pavlidis, Kennelly, and Castro 2020). In our study, as shown in Figure 5, we found that women are more likely than men to be smiling in ads, but more interestingly, we found that women are more likely than men to be smiling in “neutral” news contexts as well.
Percentage of women that are smiling (red), where 100% = all women’s faces, compared to the percentage of men that are smiling (blue), where 100% = all men’s faces. The points are the actual percentage of faces in each year, and the lines are the LOWESS smoothed data.
Top: Percentage of women that are smiling in ads (red), where 100% = all women’s faces in ads, compared to the percentage of men that are smiling in ads (blue), where 100% = all men’s faces in ads.
Bottom: Percentage of women that are smiling in news features (red), where 100% = all women’s faces in news features, compared to the percentage of men that are smiling in news features (blue), where 100% = all men’s faces in news features.
The points are the actual percentage of faces in each year, and the lines are the LOWESS smoothed data.
In her landmark 1949 book The Second Sex, Simone de Beauvoir insisted that women, from birth, are socialized as “Other” to men, serving them not only materially (as unpaid childminders and housekeepers) but culturally: as a “natural” counterpart or foil that foregrounds and celebrates the qualities and behaviours socially consecrated as “Male” (de Beauvoir 1956). Journalism and the advertising industry has served, over the course of the twentieth century (and into the twenty-first), to solidify and reproduce these sentiments through its imagery. This representational bias toward smiling women, particularly in ostensibly “neutral” contexts, reinforces the notion that women are lesser than men, and that they exist to serve male interests, thus subliminally hardening patriarchal social and economic structures.
In Figure 4, we observe some fluctuations in the percentage of women smiling that are not present in the data of men smiling, which could be due to multiple factors. We notice that there is a significant dip in the percentage of smiling women around 1970, when the women’s movement enjoyed an unprecedented surge of influence. As we have shown in our previous work (Jofre, Cole et al. 2020), at this same time, there was a peak in the number of women appearing in the magazine’s news features.
We note that because we are dividing up our 8,789 faces by gender, by context, by expression, and then by year, the data presented in Figure 5 becomes sparse in some years. The zeros depicted on the graphs in Figure 5 are true zeroes, and we provide here an explanation for each of them. In 1930, our sample has a total of 32 faces; 6 of them are women, only one is smiling, and she is in an ad, so there are 0 women smiling in news contexts in 1930 in our sample. In 1962, there are a total of 65 faces in our sample; 55 of them are men, but only 2 of those are in ads, and neither of them is smiling, so there are 0 men smiling in ads in our sample from that year. Of the 10 female faces in the 1962 sample, only 4 of them are smiling, and all 4 of those are in ads, so there are 0 women smiling in news contexts in our sample from that year. Similarly, in 1985, out of a total of 70 faces in our sample, 13 of them are women, only 3 of them are smiling, and all 3 of those are in ads. In our 2014 sample, there are 50 faces, 21 of which are women, but only 1 is in an ad, and she is not smiling. In the 1920s, there are fewer images in the issues, as well as fewer people smiling, so our samples are particularly small in these early years. In 1923, there are only 13 faces in our sample, and there are no faces (neither male nor female) in ads, so these points are normalized to zero. In 1924, there are 18 faces in our sample, only 2 are women, and neither are smiling. Of the 16 faces in our 1925 sample, 3 are women, only 1 is smiling, and she is in a news context. Similarly, of the 19 faces in our 1926 sample, 2 are women, only 1 is smiling and she is in an ad.
Of the 8,789 faces in our sample, 2,300 of them are female. For clarity and transparency, we present, in Figure 6, a direct visualization of all 2,300 female faces in our sample as a tree-map, which visualizes the proportion of women in each context.
An image of all 2,300 female faces in our sample, divided into 4 quadrants according to facial expression and context. Top Left: Smiling in news contexts. Top Right: Smiling in ad contexts. Bottom Left: Not smiling in news contexts. Bottom Right: Not smiling in ad contexts. In other words, the faces in the top row are smiling, the faces in the bottom row are not. The faces in the left column are in news contexts, and the faces in the right column are in ad contexts.
Gender and advertisements
Our data shows that throughout all eras we considered, women are more likely to be found in ads than men. Figure 7 shows that, throughout all eras, the percentage of women (relative to all women) appearing in ads is on average greater than the percentage of men (relative to all men) appearing in ads. Aggregating all years, out of the 2,300 female faces in our sample, 1,358 appear in ads (40%), while out of the 6,489 male faces in our sample, 1,720 appear in ads (26%). In other words, a man’s face is more likely to appear in the context of news than a woman’s face. This finding underlines traditional gender roles, which place women as accessories while the men are the news-makers. John Berger succinctly summarizes the convention with which men and women are depicted since the Renaissance with the phrase “men act while women appear” (Berger 1977, 47).
Percentage of women that are in ads (red), where 100% = all women’s faces, compared to the percentage of men that are in ads (blue), where 100% = all men’s faces. The points are the actual percentage of faces in each year, and the lines are the LOWESS smoothed data.
In his late-1970s work on gender and advertisement, the sociologist Erving Goffman elucidated this socialization process in a world that was, even at that point, becoming more radically “mediatized.” In his previous work, Goffman broke ground by arguing that people make their way in the modern world “dramaturgically,” unconsciously using body language to assume their social position: the pedestrian presents themselves more-or-less passively as not to invite the threat of strangers; the physician assumes a complex set of postures to seem professional to patients. Goffman understood that, in the later twentieth century, “social relations were … organized more around the appearance than the content of things” (Lemert 1998, xxxii). In Gender Advertisements, Goffman turned his attention to how these processes were condensed and intensified in ads—the happy, passive housemaker, for instance. Implicit in Goffman’s analysis was the idea that millions of women (and men) themselves learn how to “act” by aligning themselves with mass mediated gender distinctions and hierarchies—males invariably are presented as “powerful and assertive … utilitarian, controlling, and bold,” while women are manipulated and posed as passive receptacles of the male gaze (Jhally 2009, 6). Images of women signify fragility and submission: they lie down, recline, or “cant” in Goffman’s language; if they are standing, they do so beside taller males, suggesting a diminished status; when men are portrayed, they are shown “instructing” women in some activity or other, who smile or laugh in return, demonstrating approval or appreciation (Goffman 1979, 46; 34). Many advertisements appearing in Time magazine are consistent with this general pattern.
One can find specific examples of advertisements in Time that illustrate the gender roles Goffman identifies. In Figure 8, we provide six specific examples. A woman serves as window dressing in a 1950 ad for Aluminum window screens. A 1965 ad portrays a man in action while a woman lights his cigarette. A 1973 ad for gin depicts a woman serving as an accessory and looking up to a man in a power pose. In a 1984 ad, a man instructs a woman about the merits of a luxury car, while she looks enthralled by the car and the man. In a 1997 automobile advertisement, a man initiates an embrace with a woman, to which she happily submits, her face nearly covered by his. A 2007 ad for vodka reifies a woman into the literal embodiment of a consumable object—in this case, as alcohol in a cocktail glass.
Top row from left to right: A) July 10, 1950, page 46. Ad for Alcoa Aluminum window screens. B) July 16, 1965, page 2. Ad for Viceroy cigarettes. C) July 2, 1973, page 54. Ad for Burnett’s Gin.
Bottom row from left to right: D) July 2, 1984, page 56. Ad for Viceroy cigarettes. E) July 14, 1997, page 81. Ad for Mazda. F) July 2, 2007, page 21. Ad for Three Olives vodka.
In summary, we have observed two things. First, that a woman’s face is more likely to appear in an advertisement than a man’s face. Second, close inspection of the advertising images reveals that many follow the patterns identified by Goffman.
Representation of race
Problematics of categories
Race and ethnicity are very difficult to classify, in part because racial categories are arbitrary for complex cultural and historical reasons. For example, census categories have changed significantly over the past century. The arbitrariness of race is important for us to underscore, as how we count things directly affects the nature of our data (D’Ignazio and Klein 2023, 97).
The faces we examined are labelled according to the 2010 U.S. census simplified categories: White, Asian, Black, American Indian, and Pacific Islander, and “Other or Unknown” (in which “White” encompasses Latin American and Middle Eastern ethnicities). Since the representation of Pacific Islanders and American Indians was very small in our data, we combined these two categories into the “Other” category in our research.
We used the simplified census categories rather than the more extensive classifications because crowdsourcing attempts to classify images by ethnicities that fall within the “White” category (such as Latino or Middle Eastern ethnicities) relied too much on context—including problematic stereotypical depictions. The difficulty we experienced highlights the problems—and arbitrariness—of racial classification. Even within the 2010 simplified categories, racialized faces in this sense occasionally eluded recognition unless they were embedded within a stereotyped setting. (We know this because we compared two versions of our data collection interface: one that showed the face in context and one that showed a context-free cropped image of the face. These findings are discussed in our Digital Humaities Quarterly publication [Jofre, Berardi, Brennan et al. 2020]). The elusiveness and arbitrariness of race classification is a problem that we want to underscore in this paper, and in this research as a whole. Hence, we emphasize that we cannot make sweeping generalizations with this data; we are only trying to identify patterns.
The simplified census categories are very broad and imperfect, but they allowed us to investigate two specific questions. We were specifically interested in faces categorized as “Black” in the interest of tracking Time’s role in American race relations throughout the century, and in faces categorized as “Asian” because Time extensively covered the U.S. interventions in Southeast Asia.
Results
Unsurprisingly, we found that most of the faces depicted in Time magazine were identified as white. Because this data only samples about 1 to 2 issues per year, and because the representation of non-white people has been so low, we do not have enough instances to break down the race data into other categories, such as gender or context, in a statistically significant way. Therefore, the analysis that follows in Figure 9 and in the next two sections limits itself to examining trends in the overall representation of race categories.
Percentage of faces in each year that are identified as non-white, where non-white includes all faces in the Black, Asian, and “Other” category. The points represent the actual percentage in each year of our data; the line is the smoothed LOWESS curve.
Asian faces
The average percentage of Asian faces peaks around 1970. This coincides with the zenith of the US war in Vietnam.
Henry Luce—who by mid-century had begun to see his magazines as political vehicles—made his sense of mission plain in his famous 1941 speech, “The American Century.” In it, he proclaimed: “the fundamental trouble with America has been, and is, that whereas their nation became in the 20th Century the most powerful and the most vital nation in the world, nevertheless Americans were unable to accommodate themselves spiritually and practically to that fact.” Luce went on to insist that the United States has “failed to play [its] part as a world power—a failure which has had disastrous consequences for themselves and for all mankind” (Luce 1999, 165). The “cure”—in Luce’s words—was this: “to accept wholeheartedly our duty and our opportunity as the most powerful and vital nation in the world and in consequence to exert upon the world the full impact of our influence, for such purposes as we see fit and by such means as we see fit” (Luce 1999, 165). From this point forward, Luce did all he could to make sure the journalists in his employ shared in his new, interventionist worldview.
Time Inc., with Time magazine as its flagship publication, quickly took up arms in the Cold War between the United States and the Soviet Union. Luce’s particular concern was the development of a hard line on Asian communism in India, Burma (now Myanmar), China, and Vietnam—one he shared with other powerful, elite Americans, as historian David Ekbladh has noted (Ekbladh 2010, Intro., Kindle). Luce was so resolute that he shifted Time Inc.’s long-standing allegiance from the Republicans to John F. Kennedy’s Democratic Party by the early-1960s due to the latter’s tougher stance on communism (Brinkley 2010, 424).
This did not mean that Time’s journalists simply fell in line, particularly on the issue of America’s war in Vietnam—a event that completely upended “common [American] values and goals, cutting across color, age, gender, religion and class” lines (Ishizuka 2018, 221). While Luce sought to make his magazines—Time above all—into “ideologically reliable vehicles” for the fight against the Southeast Asian insurgents, he met both direct and indirect resistance to these demands from writers, photographers, and editors (Brinkley 2010, 287). In one notable battle, one of Time’s most influential editors, Luce ally Otto Fuerbringer, insisted in 1962 that Time’s press section write a piece attacking American journalists reporting out of Saigon, including those from Time, for supposedly working against American interests. The resulting article claimed that these reporters were misrepresenting the war effort to boost their personal and professional status, and that this led them to be “severely critical of practically everything” (Brinkley, 2010, 447–448). Only after some of Time’s most prominent journalists threatened to quit, and a retraction was issued, did the issue die down. By that point, Time’s newsroom, alongside most mainstream news outlets, had shifted decidedly towards a more critical stance on the war. Luce himself left his position as editor-in-chief in 1964 (Brinkley 2010, 448).
We observe in our data, shown in Figure 10, that the representation of faces identified as Asian peaks around the time of the U.S. war in Vietnam. By 1965, Time’s cover—its “most valuable piece of real estate”—was occupied by none other than the leader of the Viet Cong, Ho Chi Min (Spiker 2018, 377). This visual trend in Time abates with the end of the war in 1975.
Percentage of faces in each year that are identified as Asian. The points represent the actual percentage in each year of our data; the line is the smoothed LOWESS curve.
African American faces
In the case of the Vietnam War, Time’s most powerful editors had to be dragged, kicking and screaming, towards a left-leaning stance. Not so with the other major issue of the era: Civil Rights. Time stood apart from other major American publications in its long-standing support for African American social and economic struggles. The magazine covered lynchings from the 1920s onwards, with no less than 455 articles published on the topic between 1923 and 1960. Life, another major Time Inc. publication, devoted much of its October 3, 1938, issue to an article entitled “Negroes: The U.S. Also Has a Minority Problem,” which included a groundbreaking sequence of photos depicting the grim reality of African American life in the South, which made white supremacy palpable for many who had never seen such imagery. There was another, deeper message embedded in these photographic representations—one that spanned all of Time Inc.’s publications—that the degradation of African Americans equalled the degradation of America itself. White supremacy was, according to Life, a “glaring refutation of the American fetish that all men are created free and equal” (Brinkley 2010, 234).
By the 1950s, Time Inc. was openly supporting desegregation, with Time magazine at the forefront of this effort. The magazine extensively covered the 1956 Brown v. Board of Education Supreme Court decision that made racial segregation in public schools unconstitutional. In addition to this, Time—still tied to the Republicans at this point—threw its considerable cultural weight behind President Dwight D. Eisenhower’s 1957 decision to send federal troops to Central High School in Little Rock, Arkansas, to enforce the Brown decision. Time devoted most of its October 7, 1957, issue to the event, denouncing segregationists in uncharacteristically caustic terms: “An old man turned his dry, grassfire eyes on Central High School, worked his bare gums in pleasure over the time ‘we burned a n***** in ‘27’”; “A fat ex-schoolteacher named Arthur Bickle looked around at the crowd’s hooligans, and chortled his satisfaction” (TIME 1957). In words and especially photographs, Time stood in stark contrast to other publications, “from Paris to Tokyo,” which reported on Little Rock with “undisguised disgust” at what they took to be Black-led mob violence (Booker and Booker 2013, 120). Between 1955 and 1965, Time published over one thousand articles on the civil rights movement, far outpacing any comparable journalistic outlet.
Importantly, we must not conclude that Time was a radical publication, despite its support for African American human and civil rights. It remained resolutely in the centre of American political life, oscillating (slightly) right or (slightly) left, but always bounded by what it considered “the essential unity and the shared values of the American people” (Brinkley 2010, 134). Time’s politics were, in the end, a politics of liberal consensus (Brinkley 2010, 310–311). While it anointed Martin Luther King, Jr. as Man of the Year in 1963, by the mid-1960s, it revealed a deep discomfort with what it considered the “extremism” of some civil rights activists (Brinkley 2010, 421). For instance, Time’s coverage of the race riots in the late-1960s was distinctly unsympathetic. Its 1967 cover story on the Newark riot focused on the violence wrought and the property damage incurred, downplaying the racial injustice that lay at the heart of that violence. Police brutality in the Newark riot was deemphasized in statements such as “The very triviality of the riot’s immediate cause made the Newark outburst particularly terrifying” (TIME 1967). Time abhorred anti-black racism, as long as it didn’t violate what it considered the tenants of the American social contract. We also note that the demographic make-up of the writers and editors at Time magazine began to change in the 1970s, leading to more nuanced perspectives on these issues (Prendergast and Colvin 1986).
As shown in Figure 11, our data tracks with this historical picture as the percentage of faces identified as Black increased steadily throughout the twentieth century before plateauing at an average of about 11% starting in about 1999. According to the 2010 census, 13% of the American population identifies as African American, which means Time’s pages now more accurately represent—at least visually—the racial makeup of the country, a fact we attribute to larger public shifts in attitudes towards race, of which Time can be considered an important marker (United States Census Bureau 2010).
Percentage of faces in each year that are identified as African American. The points represent the actual percentage in each year of our data; the line is the smoothed LOWESS curve. The high point in 2009 is due to the fact that our sampled issue in that year contained a long tribute to the recently deceased entertainer Michael Jackson, featuring photos of him and his family throughout his long career.
Conclusions
Although we’ve shown that a distant view of images in Time magazine tracks well with cultural trends, we warn against reaching conclusions through metadata analysis alone, as contextualization is key to understanding the trends we observe. Our approach combines analytic methods with humanistic contextualization and a close reading of our source. We show that the context and history of Time magazine helps to elucidate discernable patterns in aggregated data.
In summary, we found that smiling becomes on average more prevalent after the 1930s due to changes in attitude toward photography and to changes in the culture of advertising. Faces that are in the context of advertising are more likely to be smiling than faces in other contexts, which is consistent with a culture of advertising that depicts happy alternate realities in order to sell goods and services. Women are more likely to appear in ads than men, which is consistent with the sexist tropes of twentieth-century advertisement. Yet sexist tropes persist beyond advertisements, as we find that women are more likely to be smiling in all contexts, including news features. Turning to the data on representation of racialized faces, we find that the number of Asian faces peaks around 1970, coincident with the pinnacle of Vietnam war coverage, and that the number of African American faces increases steadily throughout the twentieth century, reaching something close to proportional representation around the turn of the century, which is consistent with Time’s evolution towards inclusivity as an institution and in its editorial perspective.
There are yet more connections to be drawn from this data. We have observed, for example, that the proportion of children increases sharply after 1990, and we intend to examine this trend more closely as it coincides with an increase in visibility of children in the public sphere. The human-sourced data presented in this paper can be used to train an algorithm to classify all 327,322 faces extracted from our archive, resulting in a more detailed data set to be examined.
Another important objective of this project is to use our image collection to create an interactive web resource to disseminate our research and findings to the general public, with the goal of encouraging people to explore, contextualize, and understand their own cultural and visual histories. A prototype version of the resource can be viewed at https://magazineproject.org/sandbox/.
Acknowledgements
We acknowledge all our co-authors on previous papers who helped with collecting the data we used here. In particular, we thank Michael Reale (who helped with extracting all the facial images from the archive), Vincent Berardi (who helped develop our verification methods on the human-generated classifications), and Carl Bennett (who helped develop the interface we used to collect the human-generated classification, and who helped classify all the facial images from the archive by gender). We acknowledge SUNY Polytechnic Institute for financial and institutional support.
Competing interests
The authors have no competing interests to declare.
Contributions
Authorial
Author contributions—described using the NISO (National Information Standards Organization) CrediT taxonomy—are ordered by magnitude of contribution, as follows:
Author name and initials:
Ana Jofre (AJ)
Josh Cole (JC)
Authors are listed in descending order by significance of contribution. The corresponding author is Ana Jofre.
Conceptualization: AJ, JC
Data Curation: AJ
Formal Analysis: AJ, JC
Funding Acquisition: AJ
Investigation: AJ, JC
Methodology: AJ, JC
Project Administration: AJ
Resources: AJ
Software: AJ
Supervision: AJ
Validation: AJ
Writing – Original Draft: AJ, JC
Writing – Review & Editing: AJ, JC
Editorial
Senior Editor
Lisa Snyder, The University of California, Los Angeles, United States
Section, Copy, and Layout Editor
A K M Iftekhar Khalid, The Journal Incubator, University of Lethbridge, Canada
Translation Editor
Davide Pafumi, The Journal Incubator, University of Lethbridge, Canada
Production Editor
Christa Avram, The Journal Incubator, University of Lethbridge, Canada
References
Arnold, Taylor, and Lauren Tilton. 2023. Distant Viewing: Computational Exploration of Digital Images. Cambridge, MA: The MIT Press.
Barthes, Roland. 2012. Mythologies. Translated by Annette Lavers. New York: Hall and Wang.
Berger, John. 1977. Ways of Seeing. London: British Broadcasting Corporation and Penguin.
Booker, Simeon, and Carol McCabe Booker. 2013. Shocking the Conscience: A Reporter’s Account of the Civil Rights Movement. University Press of Mississippi.
Brinkley, Alan. 2010. The Publisher: Henry Luce and His American Century. New York: Knopf.
Burke, Peter. 2001. Eyewitnessing: The Use of Images as Historical Evidence. London: Reaktion.
D’Ignazio, Catherine, and Lauren F. Klein. 2023. Data Feminism. MIT Press.
De Beauvoir, Simone. 1956. The Second Sex. Translated by Howard Madison Parshley. London: Jonathan Cape.
Debord, Guy. 1995. The Society of the Spectacle. Translated by Donald Nicholson-Smith. New York: Zone Books.
Dodd, David. K., Brenda L. Russell, and Cynthia Jenkins. 1999. “Smiling in School Yearbook Photos: Gender Differences from Kindergarten to Adulthood.” The Psychological Record, 49(4): 543–553. Accessed March 8, 2024. http://doi.org/10.1007/BF03395325.
Ekbladh, David. 2010. The Great American Mission: Modernization and the Construction of an American World Order. Princeton: Princeton University Press. Kindle.
Ginosar, Shiry, Kate Rakelly, Sarah Sachs, Brian Yin, and Alexei A. Efros. 2015. “A Century of Portraits: A Visual Historical Record of American High School Yearbooks.” In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1–7. Accessed March 8, 2024. http://doi.org/10.1109/ICCVW.2015.87.
Goffman, Erving. 1979. Gender Advertisements. Cambridge, MA: Harvard University Press.
Ishizuka, Karen L. 2018. “Kill That Gook, You Gook: Asian Americans and the Vietnam War.” In The Global 1960s: Convention, Contest, and Counterculture, edited by Tamara Chaplin and Jadwiga E. Pieper Mooney, 217–236. New York: Routledge.
Jhally, Sut. 2009. The Codes of Gender: Identity and Performance in Pop Culture. Northampton, MA: Media Education Foundation.
Jofre, Ana, Josh Cole, Vincent Berardi, Carl Bennett, and Michael Reale. 2020. “What’s in a Face? Gender Representation of Faces in Time, 1940s–1990s.” Journal of Cultural Analytics 5(1). Accessed January 28, 2024. http://doi.org/10.22148/001c.12266.
Jofre, Ana, Vincent Berardi, Carl Bennett, Micheal Reale, and Josh Cole. 2020. “Faces Extracted from Time Magazine 1923–2014.” Journal of Cultural Analytics 5(1). Accessed January 28, 2024. http://doi.org/10.22148/001c.12265.
Jofre, Ana, Vincent Berardi, Kathleen P. J. Brennan, Aisha Cornejo, Carl Bennett, and John Harlan. 2020. “Crowdsourcing Image Extraction and Annotation: Software Development and Case Study.” Digital Humanities Quarterly, 14(2): 000469. Accessed January 28, 2024. http://www.digitalhumanities.org/dhq/vol/14/2/000469/000469.html.
Kotchemidova, Christina. 2005. “Why We Say ‘Cheese’: Producing the Smile in Snapshot Photography.” Critical Studies in Media Communication, 22(1): 2–25. Accessed March 13, 2024. http://doi.org/10.1080/0739318042000331853.
Lears, Jackson. 1994. Fables of Abundance: A Cultural History of Advertising in America. New York: Basic Books.
Lemert, Charles. 1998. “Goffman.” In The Goffman Reader, edited by Charles Lemert and Ann Brannaman, ix–xliii. Malden and Oxford: Blackwell.
Lin, Tsung-Yi, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2018. “Keras RetinaNet.” GitHub Repository. Accessed March 22, 2023. https://github.com/fizyr/keras-retinanet.
Luce, Henry R. 1999. “The American Century.” Diplomatic History 23(2): 159–171. Accessed March 13, 2024. https://www.jstor.org/stable/24913736.
Manovich, Lev. 2020. Cultural Analytics. Cambridge, MA: MIT Press.
Marchand, Roland. 1985. Advertising the American Dream: Making Way for Modernity, 1920–1940. Berkley: University of California Press.
Morse, Claire. 1982. “College Yearbook Pictures: More Females Smile than Males.” The Journal of Psychology 110(1): 3–6. Accessed March 13, 2024. http://doi.org/10.1080/00223980.1982.9915318.
Münster, Sander, and Melissa Terras. 2020. “The Visual Side of Digital Humanities: A Survey on Topics, Researchers, and Epistemic Cultures.” Digital Scholarship in the Humanities 35(2): 366–389. Accessed March 12, 2024. http://doi.org/10.1093/llc/fqz022.
Pavlidis, Adele, Millicent Kennelly, and Laura Rodriguez Castro. 2020. “White Women Smiling? Media Representations of Women at the 2018 Commonwealth Games.” Sociology of Sport Journal 37(1): 36–46. Accessed March 12, 2024. http://doi.org/10.1123/ssj.2018-0144.
Prendergast, Curtis, and Geoffrey Colvin. 1986. The World of Time Inc: The Intimate History of a Changing Enterprise 1960–1980. New York: Atheneum Books.
Rosenblum, Naomi. 1997. A World History of Photography. 3rd ed. New York: Abbeville.
Schama, Simon. 2016. The Face of Britain: The Nation through Its Portraits. London: Penguin. Kindle.
Scheuerman, Klaus, Madeleine Pape, and Alex Hanna. 2021. “Auto-Essentialization: Gender in Automated Facial Analysis as Extended Colonial Project.” Big Data & Society (July–December): 1–15. Accessed March 12, 2024. http://doi.org/10.1177/20539517211053712.
Spiker, Ted. 2018. “The Magazine Cover: The Craft of Identity and Impact.” In The Routledge Handbook of Magazine Research: The Future of the Magazine Form, edited by David Abrahamson and Marcia R. Prior-Miller, 377–392. New York: Routledge.
TIME. 1957. “Quick, Hard & Decisive.” TIME Magazine 70(15): 21–25. Accessed January 28, 2024. https://content.time.com/time/subscriber/article/0,33009,809941,00.html
TIME. 1967. “Sparks & Tinder.” TIME Magazine 90(3): 15–21. Accessed January 28, 2024. https://content.time.com/time/subscriber/article/0,33009,941523,00.html.
United States Census Bureau. 2010. “Modified Race Data 2010.” Accessed January 28, 2024. https://www.census.gov/data/datasets/2010/demo/popest/modified-race-data-2010.html.
Wevers, Melvin, and Thomas Smits. 2020. “The Visual Digital Turn: Using Neural Networks to Study Historical Images.” Digital Scholarship in the Humanities 35(1): 194–207. Accessed March 12, 2024. http://doi.org/10.1093/llc/fqy085.
