Introduction to the Semiotics of Emoji and Digital Stickers

By Lukas R.A. Wilde and Klaus Sachs-Hombach

Every day, billions of emoji are sent via mobile devices and chat programs, messengers, and emails. The worldwide emoji standardization – established in 2010 by the California-based Unicode Consortium – was aimed at overcoming linguistic and cultural barriers through a new digital form of pictograms and ideograms. Certainly, much has been written on the various linguistic functions of emoji ever since (cf. the contributions in GIANNOULIS/WILDE 2020). They intensify, neutralize, or soften the content of linguistic messages and serve as markers of interpersonal relationships and social contexts. To Luke Stark and Kate Crawford (2015: 1), emoji can thus be thought of as »signifiers of affective meaning« doing »›emojional‹ labor« (STARK/CRAWFORD 2015: 4) within economies of attention and affect. Vyvyan Evans (2017) conceptualized constantly changing ›emoji codes ‹ – in contrast to an overarching ›emoji language‹ – in order to emphasize that their meaning can only ever be determined in relation to specific cultural circles and according to different social, gender, or age groups. Marcel Danesi (2019) likewise argued that emoji use corresponds to an episodic and in fact narrative logic. A further technical development is represented by digital stickers, which are offered by various messaging services as further ›translations‹ of individual emoji into unique pictorial expressions (cf. WILDE 2021). Going beyond notions of static codes or fixed grammars, this special issue of IMAGE approaches emoji and digital stickers from the perspective of everyday communication and mediation. It is based on the panel »Emoji and Digital Stickers: Affective Labor and Lifeworld Mediation« held during the 15th World Congress of Semiotics (IASS/AIS), »Semiotics in the Lifeworld«, at the Aristotle University of Thessaloniki (Greece) on August 31, 2022. Apart from some of the presenters, additional contributors have been sought to represent better at least a part of current approaches to emoji research at the intersection of semiotics, linguistics, and media studies. Emoji seem especially suited to such a multi-disciplinary approach: As (partly) pictogrammatic signs, they can be investigated as a (special, quite peculiar) forms of pictoriality just as well as an innovation within digital writing modifying and enhancing our linguistic means of expressions.

The addition of pictorial signs to written texts in order to represent emotions has a long history (cf. for the following WILDE 2020). The first digital emoticon can be traced back to the computer scientist Scott E. Fahlman, who, in 1982, used the character combinations :- ) and :- ( for the first time in a digital discussion forum at Carnegie Mellon University. The employed term »emoticon« is composed of the words »emotion« and »icon« and describes the pictorial representation of facial expressions using regular ASCII characters. ASCII (American Standard Code for Information Interchange) is the (Western) character standard for digital text in instant messaging (IM) services, chats, emails, social networking services (SNS), SMS text messages, and blogs, through which emoticons soon became more widespread. Emoticons have also been in frequent use in Japan since about 1986, where they are called kaomoji (顔文字, literally: ›face characters‹). Exactly when it became possible in Japan to ›translate‹ individually designed digital character images into prefabricated pixel graphics is difficult to determine today. Many of the intervening technical gadgets, innovations, and also dead ends are documented sparingly, and the large number of competing suppliers, devices, and standards already by the end of the 1990s makes the exact history nearly impossible to reconstruct. For a long time, the Japanese telecommunications company NTT Docomo and developer Shigetaka Kurita were thought to have invented emoji in 1998. Then, it was discovered that the market competitor SoftBank had evidently developed a set of 90 emoji one or two years earlier already (cf. BURGE 2019). What is certain: It took until 2006 that Google submitted an official application to the Unicode Consortium in California (Unicode Inc.) to standardize emoji internationally. On 6 February 2009, the Consortium defined for the first time a set of 674 emoji for global use in Unicode Standard 6. One year later, this emoji set was made available to software developers worldwide, so that it could soon be ›re-imported‹ into Japan. The repertoire of emoji motifs has been constantly expanding ever since. So far, no emoji has ever been removed from the set. The admission procedures for new candidates are relatively complex and involve a range of political negotiations (cf. BERARD 2018). It is also no secret that a large proportion of the decision-makers and programmers in the Consortium are white males and that currently, nine of the thirteen institutional members with full voting rights are major US corporations such as Adobe, Apple, Facebook, Google, IBM, Microsoft, and Netflix, which thus end up making the decisions affecting our global communication standards (cf. UNICODE CONSORTIUM 2023).

Figure 1: Different glyphs for three emoji offered by iOS Emoji, Twemoji, and OpenMoji

Most emoji researchers now hold the view that the revolutionary aspect of the small pictorial symbols lies less in their design or meaning than in their technical standardization (cf. ABEL 2020). These processes are based on a combination of two different encoding methods: on the one hand, an internal hexadecimal code with a unique CLDR (Common Locale Data Repository) name, and on the other a so-called ›glyph‹, a ready-made graphical representation that results from reading out that CLDR name. Only the level of hex code and CLDR name is uniformly defined by Unicode, while the actual designs of the individual glyphs depend on the respective platforms. WhatsApp, Twitter, Facebook, etc. accordingly ›interpret‹ the Unicode description in widely divergent ways (cf. fig. 1). The glyphs designed by Apple are currently considered the de facto standard. The website emojipedia.org collects and lists the different variants (glyphs) of each emoji. Because of this medial double coding between hex code/CLDR name and glyph, many established picture-theoretical assumptions seem to be reversed in an interesting way. From art historical discussions we know that pictures can never be fully expressed in analogous sentences or linguistically encodable information. Instead, each work of art is unique and, as an aesthetic perceptual phenomenon, always more than just the type of object it represents. Prototypical photographs or paintings are thus in principle ›untranslatable‹ into other depictions or sign systems without being transformed into different images or different signs. For emoji, by contrast, it is clear that the actual identity of an emoji character consists of the Unicode number and the descriptive CLDR name, for example »burrito« for U+1F32F. This text is then ›translated into an image‹, or rather interpreted by different platforms and operating systems. The glyphs we see thus represent a language-independent interpretation of concepts, which can be rendered in various visual variants.

A further technical development is represented by digital stickers which are specific to individual messengers and platforms. Interestingly, the stickers in the Telegram messenger app (and many others like Signal) could be regarded as advanced translations of emoji or, in crude terms, as ›second-order emoji‹ (cf. fig. 2). Instead of the existing glyphs, high-resolution (and increasingly animated) graphics, which can likewise be uploaded and downloaded in sets, can thus be inserted into messages.

Figure 2: Telegram sticker sets for three existing core emoji

In the preview mode on Android or Apple, which is used to inform users in the background about messages they have received, the stickers are in fact always ›translated back‹ into the emoji to which they are linked. One can therefore speak of a repertoire of facial expressions and gestures itself being ›pictographed‹, i. e. transformed into clearly distinguishable, repeatable units. Interestingly, in many cases, we do not have any linguistic terms that we can use to differentiate these just as finely (cf. WILDE 2020). Although Unicode descriptions exist for these poses and facial expressions, do we really know the difference between a »squinting face with tongue« and a »winking face with tongue« or a »grinning squinting face«? Emoji seem here to form a distinctly structured repertoire of technically stabilized depictions of emotions that we are able to recognize in every conceivable variation. Gala Rebane (2021: 47-49) addressed these communicative affordances, perhaps not surprisingly, with recourse to Jean Baudrillard’s concept of simulacra: When TikTok users challenge each other to create ›emoji faces‹ through highly exaggerated selfie photographs, emoji do not refer back to any actual emotion (or their facial expressions) anymore but generate a medial reality sui generis that has become the point of reference for digital natives.The following six contributions represent different approaches to conceptualizing and analyzing the use of emoji and digital stickers. First, Michael Beißwenger and Steffen Pappert address emoji from a pragmalinguistic perspective. They start out from the two most prevalent, although contradictory assumptions about emoji in popular newspaper articles, namely a) the worry that emoji could be capable of making language obsolete as a means of interpersonal communication (= ›end of cultivated written language‹), as well as b) that the use of emoji threatens the function and expressive power of written communication (= ›language decline‹). Against these assumptions, Beißwenger and Pappert – drawing on authentic examples of private WhatsApp communication from a linguistic corpus –show that emoji within written everyday communication do not make language ›poorer‹. Instead, emoji can take on important functions for securing understanding and shaping interpersonal relationships, the authors point out.

While Beißwenger and Pappert approach the use of emoji from the perspective of linguistics, the following contribution by Deborah Enzmann takes an alternative approach by conceptualizing emoji (or, rather, communication through emoji) with recourse to Charles Sanders Peirce’s semiotics. Introducing a novel semiotic model (developed within her dissertation ENZMANN 2023), she draws especially on Peirce’s ›universal‹ categories of ›firstness‹, ›secondness‹, and ›thirdness‹, tracing important and perhaps unexpected connections to cognitive semiotics and comic book theory. Applying this model to a range of examples from her ›Textmoji‹ case study, Enzmann especially addresses how the recognition of a digital face differs across the degrees of abstraction to be found between earlier emoticons such as :- ) and the contemporary Unicode emoji. Her article thus advocates a more detailed look not only at the linguistic functions of emoji but at their formal-aesthetic properties that can strongly influence the interpretation of digital faces, for instance by regulating intensity and affect.

A third, again contrasting semiotic understanding is then presented by Andrea Ferretti who conceptualizes the emoji code in the tradition of European post-saussurean structuralism. The author reflects especially on the encyclopedic skills that the use and comprehension of emoji require, arguing that – despite their apparent iconicity – they can only be understood through the filter of linguistic, cultural, and socio-pragmatic coordinates. Ferretti especially objects to the commonplace notion that emoji substitute the ›paralinguistic clues‹ of face-to-face communication, such as prosody, gestures, interpersonal space management, and facial expressions. »To recover something from the lost spontaneity of face-to-face communication, emojis could be used to represent how the body reacts beyond the screen, not how it would have reacted if it had been face-to-face« (FERRETTI 2023, this issue). The emoji code, in other words, should be seen to signify a new paralinguistics, native and specifical of digital communication.

While these first three contributions offer different perspectives on emoji in a broader sense, the second half of the present special issue focuses on more specific contexts and cases. First, Christina Margrit Siever investigates how emoji are employed to express the opposite of what they often seem typical for: not for any carefree or playful decoration of messages, but rather for the expression of sincere mourning. Her case study, based on a corpus of 8,351 Twitter tweets, investigates the use of emoji to commemorate so-called ›angel babies‹ (German: »Sternenkinder«), i.e., children who die (shortly) before, during, or after birth. Are there specific emoji for digital mourning and to what extent can they be interpreted in religious or spiritual terms (angels, praying hands, or candles) – or merely signal a specific kind of mediated empathy? Siever distinguishes several functions and meanings that emoji can take on in these contexts, especially with respect to (partial) redundancy or complementarity within the multimodal expressions in general.

From emoji proper, we then turn to the even newer forms of digital expressions addressed above, digital stickers: Michaela Oberwinkler proposes a closer look at various functions of these stickers within communication through the Japanese messenger LINE, based on 764 cases of authentic data provided by Japanese university students. Despite the notion that such stickers could be seen as emoji that are merely larger in size, Oberwinkler shows that they are actually more expressive and fulfill various additional functions, perhaps even performing distinct ›speech acts‹ on their own. Despite these indications, the author warns against generalizations, pointing to many peculiar gender differences as well as to cultural issues that are strongly connected to specifically Japanese habits of communicating.

In the concluding contribution to this special issue, Marcel Lemmes finally traces many ways that media studies can contribute to our understanding of emoji/sticker use. He analyzes digital pictograms on the live streaming platform Twitch and the community chat platform Discord, investigating especially the affordances, constraints, and ›social protocols‹ connected to these specific interfaces and communities. Integrating a cultural semiotics, a situational semiotics, and an intersemiotic analysis of Twitch- and Discord-emoji, Lemmes argues that their potential lies especially in fostering affective and communal interactions within the online communities in which they are interpreted, negotiated, and continuously modified.

We hope that these six contributions to the blooming field of emoji research demonstrate the rich potential a broadly semiotic approach – spanning across or putting into dialogue text-linguistic, picture-theoretical, as well as media-theoretical methods and concepts – can bring to our understanding of digital, affective communication. We would like to thank all our contributors for their efforts in making this special issue possible in such a short span of time since the 15th World Congress of Semiotics, and we hope you will find the subsequent six articles as stimulating and inspiring as we did.

Tübingen & Trondheim, August 2023

References

BURGE, JEREMY: Correcting the Record on the First Emoji Set. In: Emojipedia.org. 08.03.2019. https://blog.emojipedia.org/correcting-the-record-on-the-first-emoji-set/ [August 1, 2023]

BERARD, BETHANY: I Second that Emoji: The Standards, Structures, and Social Production of Emoji. In: First Monday, 9(3), 09.01.2018. https://www.firstmonday.org/ojs/index.php/fm/article/view/9381 [August 1, 2023]

DANESI, MARCEL: Emojis: Langue or Parole? In: Chinese Semiotic Studies, 15(2), 2019, pp. 243-258

ENZMANN, DEBORAH: Emojisierung: Eine historische und semiotische Studie zu Emojis. Salenstein [Niggli] 2023

EVANS, VYVYAN. The Emoji Code: How Smiley Faces, Love Hearts and Thumbs Up are Changing the Way we Communicate. London [Michael O’Mara] 2017

GIANNOULIS, ELENA; LUKAS R.A. WILDE (eds.): Emoticons, Kaomoji, and Emoji: The Transformation of Communication in the Digital Age. New York [Routledge] 2020

REBANE, GALA: Emojis: Digitale Bildkulturen. E-Book-Ausgabe. Berlin [Klaus Wagenbach] 2021

STARK, LUKE; KATE CRAWFORD: The Conservatism of Emoji: Work, Affect, and Communication. In: Social Media + Society, 1(2), 2015, pp. 1-11

UNICODE CONSORTIUM: Members. 2023. https://home.unicode.org/membership/members/ [August 1, 2023]

WILDE, LUKAS R.A.: Digital Pictograms? Emojis between Infantilisation and International Ideography. In: Leopold-Hoesch-Museum und Museum für Neue Kunst Freiburg (ed.): Die Gesellschaft der Zeichen – Piktogramme, Lebenszeichen, Emojis. Cologne [Verlag der Buchhandlung Walter König] 2020, pp. 236-241

WILDE, LUKAS R.A.: Emoji als Piktogrammatik. In: Annette Geiger; Bianca Holtschke (eds.): Piktogrammatik: Grafisches Gestalten als Weltwissen und Bilderordnung. Bielefeld [transcript] 2021, pp. 6-29


About this article

Copyright

This article is distributed under Creative Commons Atrribution 4.0 International (CC BY 4.0). You are free to share and redistribute the material in any medium or format. The licensor cannot revoke these freedoms as long as you follow the license terms. You must however give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. More Information under https://creativecommons.org/licenses/by/4.0/deed.en.

Citation

Lukas R.A. Wilde; Klaus Sachs-Hombach: Introduction to the Semiotics of Emoji and Digital Stickers. In: IMAGE. Zeitschrift für interdisziplinäre Bildwissenschaft (Themenheft: The Semiotics of Emoji and Digital Stickers), Band 38, 19. Jg., (2)2023, S. 151-157

ISSN

1614-0885

DOI

10.1453/1614-0885-2-2023-15747

First published online

Oktober 2023