What do Emojis Stand for? Notes for a Semiotic View on the Digital Relationship between Written and Spoken Communication Practices

By Andrea Ferretti

Abstract

The integration of digital messaging applications into the most diverse practices of everyday life drives a rethinking of the relationship between bodies, orality, and writing within human communication. Communicative cues such as tone of voice, gestures, facial expressions, etc. find in emojis a way of translating themselves through a new kind of informal writing. Under this lens, some significant aspects of the merging of emojis and alphabetic writing can be investigated. This article reflects on the semiotic features of this translation and on the encyclopedic skills that the use and comprehension of emojis require. Special emphasis is placed on the linguistically mediated nature of emojis meaning. Despite their apparent grounding and iconicity, they can only be explained through the filter of linguistic, cultural, and socio-pragmatic coordinates. In short, the paper provides an understanding of emojis as a semiotic code, which contributes to characterizing digital communication. The semiotic code of emojis creates a space for collective reflection and creativity surrounding the expression of emotions, mutual social evaluation, social relationships, etc. Despite the fragmentation and the exploitative dynamics typical of digital environments, the use of this semiotic code will be depicted as an ongoing collective game, concerned with the replacement of individual bodies, even in the most mundane and informal communicative exchanges.

0. A linguistic-semiotic framework for the analysis of communication practices: between praxis and codes

In this paper, emojis semantics is analyzed in the context of a broader conception of human communication, linked to the tradition of European post-saussurean structuralism. The aim of this contribution is to illustrate the role of emojis in everyday communication, addressing their leading function in the transition from orality to writing in everyday communication made possible by new digital media technologies. However, before addressing these issues, it is necessary to clarify some theoretical notions and the relationships between a specific code such as emojis on the one hand and the linguistic-cultural context in which they are embedded on the other. The core of the explanation to the problem of the meaning of emojis lies in the integrated understanding of this code with the other cultural codes (especially the linguistic one) that are simultaneously at work in social communication.

The term ›code‹ is used as a semiotic extension of the way Saussure used the term langue, as a system of signs, about language. First of all, for Saussure (cf. 2011), the langue is not a nomenclature, i.e. it is not the rigid pairing of two already given entities through which one expresses the other. Langue’s units are not simply material signals and psychic or material objects univocally connected to them. On the contrary, the possibility of stipulating explicit nomenclatures lies in the existence of a much more complex and articulate code such as the langue. The langue is a dual system of relations, within which both the values of signifying and signified units emerge by difference. Signifiers and meanings, as semiotically determined and intersubjectively comprehensible, do not pre-exist the relationships within the system but are derived from it. For this reason, it can be said with Hjelmslev (1953) that language ›carves out‹ i.e. makes relevant and articulates not only the phonic substance of expression but also the experiential substance of content.

This interplay of relationships between entities, both on the vertical (signifier/signified) and horizontal (signified/signified; signifier/signified) planes, is said to be ›radically arbitrary‹ in that it derives neither from the logical structure of reality nor from the immediate relationship between the human perceptual apparatus and things. On the contrary, within the bio-physiological constraints of the human species, signifiers and signifies are mutually determined and limited in each other according to a process of sedimentation that coincides with the history of various human cultures and vary across different languages. The synchronic sign ›values‹ at this level do not emerge through an explicit act of individual will, but are the result of a gradual, collective, i.e., diachronic, process. All the various mechanisms of ›bio-cognitive motivation‹ (image-schemata, basic metaphors, embodiment, etc.) in the formation of meanings or of ›iconicity‹ in the relations between signifiers and signifies must be understood within this broader arbitrary cultural-historical process.

The intersubjective validity of signs, i.e., the value of signifies and signifiers, relies on presenting itself to users as a set of ›norms‹ that successfully direct communicative practices and the attainment of practical ends. Languages are born with social organization for social organization’s purpose (cf. VOLOSINOV 1973:12). As explained by Saussure (2011: 78), the foundation of the arbitrary system of language is found in »time« (i.e., traditional-institutional continuity) and the »speaking mass« (i.e., the implicit consensus of the speakers). Language is what constitutes and permeates the entire social organization: from this, it originates and in this, it derives its validity. The langue studied by Saussure then is the set of principles of order that govern the relations between its signs in their different contexts of use and for different practical purposes. Just as there is no single purpose for the use of linguistic signs, no single communicative practice, no single social stand, and no single way of evaluating them in communicative exchange (speaking a language is not just describing its units in the activity of writing a vocabulary or a grammatic handbook), so a code is not a single system of relations, whereby the same signs and their components enter into different relations according to different contexts of use and are always available to enter into new ones. Different practical orientations generate different principles of order according to which signs arrange each other.

This Saussurian approach to the problem of the semiotic code or system overcomes the old distinction between ›semantics‹ and ›pragmatics‹, as well as that between ›linguistic‹ and ›cultural‹, between ›sign‹ and ›praxis‹. This overcoming is well exemplified by the concept of »encyclopedia« developed by Umberto Eco (1975: 99-100). There is no code as a simple system of univocal correspondences that is later integrated into individual communicative contexts by the inferential decisions of individual speakers. On the contrary, different communicative practices, as linguistically constituted, orient from within the different planes of the language system and determine different relational configurations from which speakers derive the appropriate norms of different forms of communication. The language system, Saussure’s langue, is the most complex semiotic system because verbal languages shape and structure the entire socio-cultural world: there is no social practice or relationship that is not constituted without the mediation of language (cf. VOLOSINOV 1973: 14). This is why language has the property of »omniformativity« (DE MAURO 1982: 134): its pervasiveness in the organization of the human world, combined with its capacity to have itself as content, does not allow us to indicate with certainty what language cannot strive to say. Like any other social practice, the use of other semiotic codes is also organized and interpreted through the mediation of language.

This priority role of language in social life distinguishes it from the rest of the codes, leading Roland Barthes (1968: 11) to declare semiotics itself an internal part of linguistics. This position is the starting point of the paper: emojis, considered as a code (a pragmatically oriented and multidimensional sign system) do not have language as their content, but are a content of language, i.e., they are explained and interpreted through the more general linguistic-cultural competence of speakers. To use and understand emojis, we employ the whole of our linguistic knowledge; to study them, we establish in the language a meta-level that has emojis as its object, whereas the opposite is unthinkable.

In written digital communication, while the alphabetic code is an expression of the linguistic code, the emoji code is a pictographic expression of some facial emotions, some bodily gestures, and objects, but it is so only through the mediation of language and its encyclopedic semantics. The mediation of language in the use and understanding of emojis is not a subjective issue, relating to the time the individual writer spends in their use in single communicative instances, but an objective feature of the relationship between semiotic codes and between them and our general social experience. As it imposes itself in the common pre-understanding of experience, so does the language impose itself in the interpretation of emojis. The emoji code receives from language its plasticity and its characteristic as a sign code crossed by different possibilities of use.

If emojis can function as an additional writing system, complementary to the alphabetic one, and allow further levels of meaning, this is not due to their iconic relationship to perception, but, on the contrary, to their being tools for elaborating already linguistically oriented reflections on the role of the body in communicative interaction. First of all, it is necessary to understand their general function in communicative practices: from this, it will be possible to describe their semantic possibilities.

1. Writing our bodies on the screen: Functional identity between emojis and paralinguistic clues

If we think about chatting, using apps like WhatsApp, Telegram, or Messenger, emojis are used mainly to represent in writing the paralinguistic clues that are so characteristic of spoken face-to-face communication. By the term ›paralinguistic clues‹, I mean everything from prosody to gestures, interpersonal space management, and facial expressions.[1] All these elements have an important function in shaping the content of messages, expressing the emotive tone of an overall conversation, the evaluative intention about the subject of discourse, and the meaning of the relationship between speakers. Often, it is precisely through these paralinguistic clues that we give a hint of which »language game« (WITTGENSTEIN 1953: 5) we are playing.

The communicative functions that have relied on paralinguistic clues prior to the advent of emojis are particularly evident in daily communications, related to the needs, contingent tasks, and immediate necessities of daily life. Today, this very kind of exchange is multiplied and displaced more and more in digital writing. For this functional analogy, chats tend to reproduce the typical structures of speech, giving rise to a particular style of writing. This reproduction of orality within practices of writing (a peculiar form of ›secondary orality‹[2]) is evident not only in lexical and syntactic structures but also in the reproduction of what we have called paralinguistic clues. Indeed, the use of emojis highlights the shift from a culture in which the written word retained a connection with more public and formal purposes (linked to large audiences of mixed receivers), to one in which it spills over into the construction of the more private, affective, and mundane sphere of small groups.

However, from a semiotic point of view, the paralinguistic clues typical of face-to-face communication are difficult phenomena to categorize. As Umberto Eco (1975: 19-21) explains, they are situated across the ›lower semiotic threshold‹: they can thus be either signifiers, interpretable and usable according to cultural codes, or unintentional reactions to stimuli triggering bodily responses, i.e. unreflective effects caused in the bodies and minds of communication partners. These can be interpreted by receivers not as ›signs‹, but as ›signals‹, standing with their meaning in a relationship similar to that with which smoke stands with fire. Keeping this in mind, the problem I would like to discuss is the following: how do emojis, with their strange remediation of paralinguistic clues, contribute to shaping how we write in our daily digital communications?

It is important to remember right at the outset that emojis are not something completely new. Vyvyan Evans (2017) shows how emojis need to be interpreted as a mere new step in a long history of punctuation marks.[3] This is a history that follows the development of different technological media for written communication and the different uses of writing that these technologies made possible. Indeed, even in the more formal uses afforded by printing, it was necessary to introduce a system of signs to signify different types of pauses, prosodies, or to mark the presence of the spoken word (direct speech), etc.:

Emoji is to textspeak what non-verbal cues are to spoken interaction. The primary function of Emoji is not to usurp language, but to provide the non-verbal cues essential to effective communication that are otherwise missing from textspeak. Emoji […] fulfills exactly the same functions that non-verbal cues do in face-to-face linguistic interaction (EVANS 2017: Calibre pos. 10.210).

Thus, Evans asserts a functional identity between emojis and paralinguistic clues. Emojis, however, are a broader code than punctuation marks when they generate paralinguistic clues; their success can be explained by the extension of digital writing into areas of interpersonal communication practiced orally and/or face-to-face before the advent of digital media. The closer writing approximates the countless forms of informal everyday communication, the more it needs complex semiotic codes to translate its paralinguistic clues. Following the identification of functions asserted by Evans, my purpose in the following will be to question how these functions are exercised by emojis, seen as a complex and multifaceted semiotic code that refers only in a linguistically and culturally mediated sense to the sphere of paralinguistic clues. This approach also allows us to show how the emoji code produces a peculiar type of semiotic creativity. Finally, it allows us to trace along which conceptual lines some important differences between face-to-face and digitally chat-mediated communication emerge.

2. Emojis as a system of signs

As has often been noted in the available literature, emojis aren’t a plain, immediate, or natural universal language. On the contrary, they are a system of signifiers that could have certain meanings/signifies only within communicative practices of the different communities that use them. Reason for that can not only be found in cultural differences through which emojis can be interpreted in various ways but, much more foundational than that, within their semiotic traits. It is clear that emojis are not simply iconic reproductions of the paralinguistic elements that accompany speech. If we just consider their visual, two-dimensional modality, the limitations of their iconicity become evident: an emoji can’t mean anything just because they look like something else that has some meaning in face-to-face communication. Emojis are thus not a nomenclature for which each signifier corresponds to an unambiguous or self-evident meaning. On the contrary, they are an arbitrary system of signs,[4] open to the most diverse possibilities of cultural uses, i.e., not bound to stimulus-response causal patterns, but open to the interpretive dimension proper to semiotics. The static uniformity of the repertoire of emojis does not imply the uniqueness of their meanings, but, on the contrary, causes them to vary across the linguistic and cultural diversity of the communities (and micro-communities) that use them (up to and including jargon conventions limited to a single chat group). Only when they are considered in relation to one of the actual communities using them, identifiable according to the most diverse linguistic, cultural, and social coordinates, can emojis mean something to someone. The norms governing the use of emojis can thus be imagined as several concentric circles centered around the individual. From the broadest, most general, and indeterminate circle that simply includes the users of a certain cultural area, to narrower ones concerning linguistic and/or national communities, up to highly determined local conventions that may only be valid within small circles of interest groups, friends, or relatives. In particular, the innermost circles may be variously intertwined and overlapping, while being motivated by the norms that are valid in the larger circles.

On the more general level of cultural competence, consider, for example, some emojis that could seem easily understandable, like smiles and laughter: 🙂; 😀; 😆; 😂; 🤣. The construction of a paradigm or »associative relation« (SAUSSURE 2011: 129) based on their degree of intensity would seem obvious, as would the differential traits of their signifiers (based on the position of the mouth, eyes, the presence/absence of tears, and the general spatial orientation) and thus their correspondent »denotative« content (HJELMSLEV 1953: 114). This, however, does not tell us anything about the possibilities of their usage and thus about their actual signified in specific communication practices. In our linguistic and cultural competence, there are encodings of different types of smiles and laughter, and they differ not only with regard to a greater or lesser intensity. Such complexities drive the determination of the semantic relationships within the emoji code off – and thus also the meanings of individual emoji. For example, just think about the cultural differences between ›laughing with your interlocutor‹ and ›laughing at your interlocutor‹. Yet, for both these types of laughter, relying only on the iconic/formal characteristics of signifiers, one could use all these emojis indifferently.

Furthermore, emojis, like every system of signs, are combinable with each other, forming larger syntagms. These syntagms could be structured using ›grammatical rules‹[5] that can only be chosen from the linguistic co-text and socio-pragmatic context. For example, consider this syntagma: 👃💨. It would seem to signify a ›sneeze‹, supposing a nexus of spatial contiguity between the two emojis. It could instead indicate ›sniffing something‹, in a literal or metaphorical sense, if one assumed that the relationship between the two signs is a causal one.

Before I deal with the heart of the topic, however, I would like to briefly summarize my theoretical framework. As already mentioned, emojis are seen as a pictographic and logographic writing system integrated in the broader context of alphabetical writing. Their main purpose is to integrate alphabetic writing with paralinguistic aspects typical of face-to-face communication. To understand how emojis achieve this goal, we must consider that they do not signify directly an iconic reproduction of facial expressions, gestures, body postures, and so on. On the contrary, to understand how the same set of emojis is created, interpreted, and used, we need to look at the many ways in which we can talk and interpret paralinguistic clues through different verbal languages and cultures. Just because the use of emojis is mediated by different languages and cultural norms, we can also find so many inter- and intra-cultural differences in the interpretation and use of emojis (cf. ABEL 2020; KARPINSKA et al. 2020). Now, we need to consider why expressing our communicative attitudes with our body in face-to-face communication is a very different practice than writing in the space of digital interfaces: how does the ›warmth‹ of the former translate into the asynchronous ›coldness‹ of the latter? What change occurs from a semiotic point of view? So, we need to consider the emoji code as:

• an arbitrary sub-system of digital writing; using some pictographic and logographical strategies to create signifiers (cf. GENSINI 2002: 112; DANESI 2017: 7); designed to be used in an integrated way with alphabetic writing; within a broader cultural environment in which writing is systematically bound to linguistic units;
• of the paralinguistic aspects of face-to-face communication (expression of emotions, body language, gestures, prosody, pointing, etc.);
• whose interpretation, both by the users and by those who study it, is mediated
  through a practice of linguistic analysis and reflection.[6]

3. The use of emojis as a cultural-encyclopedic practice

In the next paragraphs, we will see how general semiotic features apply to the emoji code and how they distinguish emojis from the paralinguistic bodily clues that they try to translate into written communication. We will address the following points:

Emojis constitute a code whose uses are, first of all, like those of every other sign optional. We choose to use them with considerations about which emoji could work better in a given situation for a specific purpose and a given addressee. On the contrary, some sort of paralinguistic clues are always present: they are not only signs but also symptoms and spontaneous bodily reactions.

The use of emojis is the result of an arbitrary discretization made on the continuum of paralinguistic phenomena. Paralinguistic reactions emitted by the sender, as individual physical phenomena, are continuous. Furthermore, the iconic motivation of their signifiers does not determine their meaning.

Unlike most paralinguistic clues (which are on a continuum scale of major or minor intentionality), we always use emojis through a process of linguistic mediation and, sometimes, with some strategic communicative purpose. Emojis only simulate bodily spontaneity because they are written signs chosen from a variety of expressive possibilities.

Firstly, it must be remembered how alphabetic writing, too, can give an emotional or evaluative tone to its contents relying on its own resources, without appearing artificial or unnatural. So, the use of emojis, which, as has been noted by Danesi (2017: 96), connotes a positive and friendly attitude toward other communication partners regardless of specific choices, does not imply that their absence automatically means coldness or interpersonal distance. Although today emojis are widespread, they represent just one possibility to add an emotional tone and attitude. Indeed, a friendly tone could emerge through strategies within the alphabetic code and the different sociolinguistic varieties it can express. When we choose to use emojis instead or in addition, we try to exploit the semantic »connotations« (HJELMSLEV 1953: 114-125) that the use of emojis has assumed in our culture and their well-known pragmatic implications (speed, informality, graphic appeal, etc.). But the choices of using emojis are very different choices from the management of the paralinguistic elements of face-to-face communication. These are present and impose themselves independently of any communicative will; the speaker must know how to manage them. A voice indeed always has a certain prosody, a face, a certain expression, and so on, while every chat message can be written without emojis – and yet not lose communicative efficacy necessarily. So, it is impossible to have face-to-face communication without paralinguistic clues, while digital communication can well work without emojis. The absence of emojis does not necessarily mean a formal or serious tone toward the content and/or the receiver because alphabetic writing has its own tone resources. The use of emojis is never a necessity that just imposes itself (as our body does) but remains always a semiotic choice. It is a choice that has its own semantic connotations as well as its own pragmatic aims and benefits.

In semiotic terms, this difference can be explained by referring to the distinction made by Hjelmslev (1953: 50-57) between form, substance, and purport. What we generally call ›paralinguistics-kinesics-proxemics‹ is the result of the interaction of different semiotic systems, whose signifying substance consists of several dimensions of the communicative partners’ bodies. For this reason, these elements are always active at the same time in face-to-face communication. So, paralinguistic semantics consists of a simultaneous, multimodal, and »global« meaning (DE MAURO 1982: 33), in which the different modalities form a coherent sign pattern. It is this pattern that can be integrated with the meaning of the spoken word. However, the speaker’s body is only partially used as an intentional sign by the speaker and interpreted as a sign by his partners. Most of the parts of our body simply do not produce sign dynamics and remain semiotically undefined and continuous. Some parts of the body simply produce stimulus-response mechanisms, which can only later be interpreted by someone else as signals. So, the body only partially gets access to the socio-cultural, shared, and interpretative dimension of signs; in other respects, it is present only as a natural ›fact‹, acting and reacting causally. Our body also continues to express the idiosyncratic dimension of the speaker’s unique and individual characteristics, as well as the universal dimension of the most natural emotional reactions.

Emojis, in contrast, are far removed from any embodied experience: they represent a sign system whose ›substance of expression‹ is a digital graphic system itself. It follows that each emoji is a discrete entity that can be combined linearly with other emojis or with alphabetical signs, according to spatial and temporal relations of the linear sequence. Instead of global, their meaning is the result of signs »articulated or combinatory« (DE MAURO 1982: 39). The result of this combination will refer to only a few paralinguistic aspects, chosen according to criteria of semantic relevance and pragmatic economy. Moreover, emojis, unlike the ›facts‹ that constitute paralinguistic clues in verbal speech, have neither natural nor idiosyncratic features. Emojis lie entirely in the sphere of semiotics and therefore in the cultural and communicative competence of the writers. So, emojis are a sort of ›common or cultural body‹ whose semantic capacity is continually enriched and modified through how they are used by the various communities.

Figure 1: Biden’s sarcastic expression and some possible written ›translation‹ through emojis by the author A.F.

To stress the cultural or »encyclopedic« (ECO 1975 99-100) competence required for the use of emojis let us turn to consider a famous example like the sarcastic line made by Biden during the first debate for the presidential election in 2020: »it’s hard to get any word in with this clown«. If we wanted to translate Biden’s paralinguistic attitude into a WhatsApp message using emojis, there would be an undeterminable number of possibilities related to different logical and cultural criteria. The first two ›translations‹ I have proposed above (cf. fig. 1), for example, try to represent the expressiveness or tone of the speaker. Likewise, the third and fourth use a gesture instead of a facial expression. These are not gestures that Biden made, but they are gestures that, reflecting linguistically, could well portray the meaning of his global attitude. For example, the gesture of raising hands (›I give up‹) combined with the clown emoji can be used to express an attitude such as ›you can’t argue with people like that, I give up/I raise my hands‹. What matters in the choice of emojis is not the likeness to Biden’s face or movement, but the motivation with respect to the semantic content of the speaker’s general attitude. In the third and fourth examples, moreover, Biden’s expression is reinforced by its connection to a possible Trump connotative description. Particularly, in the fourth example, this description is made through a logographic strategy representing an Italian idiomatic expression: ›being mad as a horse‹. The fifth message, in contrast, uses a semantically fuzzy emoji, which could refer as much to his interlocutor (portrayed in a circus performance) as to the overall tone of a communicative situation (in which words are used to play as the juggler plays with his balls). In addition, the last two examples six and seven use the gesture of pointing in different senses; while in the sixth the finger is pointed at who is being talked about, in the seventh the emoji is used to reproduce a gesture easily understood by Italian speakers. Indeed, pointing at one’s temple, accompanying the gesture with a puzzled expression, means that someone is ›out of his mind‹. Of course, these proposed translations make sense only because, in this context, ›clown‹ is broadly synonymous with madman.

According to this example, to understand how emojis could be used, we must relate them not only to objects or words but to the entire encyclopedic ›frames‹ in which they appear (cf. ECO 1979; FILLMORE 1982), i.e. the structured sets of prototypical cultural knowledge. Emojis do not simply represent facial expressions or things, but the cultural, encyclopedic pathways in which those facial expressions, parts of the body, or things appear. They thus signify only in relation to situations, social positions and roles, prejudices, widespread metaphors, idioms, etc. Consequently, it makes no sense to ask for which specific expression or paralinguistic attitudes this or that emoji stands for or what it replaces. Instead, it makes sense to ask what aspects of a certain cultural-linguistic unit can be symbolized, in some respects, through a particular emoji or through a combination of two or more emojis. For example, the ›sardonic laugh‹ is a complex cultural unit, broadly related to the expression of sarcasm, derision, and mocking. All these emojis (🙃; 😏; 😁) stand, to some extent, for the sardonic laugh’s bittersweetness: only the specific communication practices and the aim of the specific exchange can indicate which would work better in what instance. In determining their meaning, a key role derives from norms (shared even by small groups of individuals), from the alphabetic part of the text in which they are embedded and also from the overall tone of the conversation and the relation between the partners. As we could explain with Eco and Fillmore, what motivates the construction, interpretation, and finally the use of emojis is not the objects depicted, but the whole cultural contents (text, pictures, practice, conceptual frames, or scenes, etc.) corresponding to those objects. The entire communication using emojis can be seen as an open, endless game, in which writers collectively explore the possibilities of making sense, in connection with alphabetic writing, encyclopedia, and face-to-face practices, of this system of images.

4. Devices for lying: Against an apocalyptic theory of emojis

Unlike some aspects of expressions that are highly spontaneous, emojis constitute an arbitrary network of signs without any inherent meaning. The signifiers of emojis are of course motivated by cultural aspects of their possible content, but this motivation in no way determines the actual use of emojis and their evolutions: »emoji alone are unreadable« (FREEDMAN 2020: 45). As we have seen, completely unexpected motivations can be continually found. The ›motivated signifiers‹ of emojis are thus also empty: their motivation doesn’t give us a ›rule‹ to follow, but just an indefinite array of signifying possibilities. Only to the extent that they are used routinely, they can obtain a definite meaning that could be more or less widely spread, more or less dependent on the alphabetic code, more or less related to a specific jargon, or even understandable only within very small groups. A specific use of a specific emoji could indeed make sense only in a given case, even only within a single group chat: the very fragmentation of the communicative spaces inherent to digital media drives toward a very wide range of socio-semiotic variations in emojis’ uses.

If emojis can thus carry complex meanings, they can activate long and nuanced pathways in the speakers’ encyclopedias. Unlike our empirical bodies, however, emojis cannot express or reveal anything about the real, embodied emotional mood of the speaker. Unsurprisingly, while we can use emojis to lie as much as we want (according to ECO 1975: 6-7 they are an object of semiotic study for this very reason), we can use our body only to lie up to a certain extent. Given the presence of elements of spontaneity, unreflective reaction, and naturalness, in face-to-face communication one must make an effort to lie, and the chances of success are bound to the ability of the individual speaker to deceive, as well as to the opposing skills of their interlocutor. Indeed, our bodies are only to some extent a sign and not natural objects that, as such, can reveal something about our mood simply by reacting to environmental stimuli. In contrast, with emojis, we reflexively and linguistically work on a common, completely semiotic body. The use of emojis gives us no clue about if someone is using this ›digital body‹ to lie or not. Who can tell the emotional status of the people behind this chat by relying only on written clues?

It is important to highlight how this argument applies not just to emotional or gesture emojis, but also to any emojis representing objects. These emojis could be used indeed to evoke entire cultural frames (cf. SIEVER 2020: 143) exactly like smiley-emojis. For example, in the following message, »come on, let’s have a coffee break🍹⛱️«, the cocktail and the beach umbrella emojis are used to indicate the speaker’s ›summer/beach holiday‹ attitude (as well as a poor will to work). The ›summer holiday‹ frame implies certain ways of presenting yourself, not directly reducible to a certain set of physical traits, but linguistically definable as being relaxed, slow, lazy, and so on. This use of object-related emojis confirms how the encyclopedia and not the iconic relation to face-to-face expressions drive the paralinguistic function of emojis.

We can thus say that emojis contribute to shaping a digital and strategic written orality. From a semiotic-cultural point of view, this communicative practice opens up sophisticated writing habits. Using emojis to signify our emotional attitudes, we embrace a task that, in face-to-face communication, we perform without reflexive efforts and with much more spontaneity. From another point of view, however, the loss of the material and individual body as an expressive resource makes communication much safer and more controllable by the writers. In digital communication we do not have to deal with a body that can betray us, producing communicative clues independent of our will and planning; we are freed from the non-semiotic but idiosyncratic or universal components bound to our material bodies. In the asynchronous times and the screen spaces of digital communication – the spaces we share with our partners are generated by the interfaces of the application employed – we are much freer to plan evaluative and emotional aspects of meaning using the full range of emojis in combination with written language strategies.

Starting from these considerations on the open-ended and indeterminate character of the use of emojis as a semiotic body of digital communication, we can reconsider the arguments put forward by Luke Stark and Kate Crawford, from the point of view of the social sciences, in a 2015 paper. In their text, the authors discuss the role of emojis in platform capitalism and its dynamics of value extraction from the communicative activity of the users who inhabit them. The discussion combines three main arguments:

1. emojis foster sociality on platforms, making an otherwise inhuman way of communicating more acceptable;
2. emojis vehicle their own ideology, related to consumerism, individualism, etc.;
3. emojis are another opportunity for the extraction and quantification of economically valuable data.

This passage from the conclusion of the paper summarizes all the above points:

Emoji, like that original smiley, are prophylactic – they help people in digital environments cope emotionally with the experience of building and maintaining social ties within hierarchical technological platforms and unjust economic systems that operate far outside of their control. Yet the emancipatory potential of emoji is restricted by their industrial and commercial limitations […] Emoji offer us more than just a cute way of humanizing the platforms we inhabit: they also remind us of how informational capital continually seeks to instrumentalize, analyze, monetize, and standardize affect. Emojis are an exuberant form of social expression but they are also just another means to lure consumers to a platform, to extract data from them more efficiently, and to express a normative, consumerist, and predominantly cheery world-view. In this light, emoji should be understood both as a rear-guard action to enable sociality in digital networks and also the means to quantify, measure, signal, and control affective labor, and reinforce existing regimes of inequality and exploitation (STARK/CRAWFORD 2015: 10).

It can thus be pointed out that communication is not simply an end in itself, but is often motivated by the material needs and purposes of individuals as well as the social formations they are a part of. Even simple gossip has its purpose in the regulation of social relations. So, as long as the digital platforms that profit from the most diverse communicative exchanges are integrated into how social needs are satisfied, their success has little to do with the codes made available to users. Emojis may be part of user-friendly interface design concerns, but while they may promote the use of digital platforms, they are far from being a significant cause of it. While, as argued above, using emojis to express a conversation’s mood remains a choice, the chat itself and its channel very often are not.

To address Stark’s and Crawford’s second argument about the emojis’ ›ideology‹ or ›world-view‹, it is helpful to recall the distinction provided by Eco between the two main attitudes that divide the field of mass communication scholars between »apocalyptic and integrated« (ECO 1994: 17-18). While the latter claims the total neutrality of the medium with regard to the message, the media (here understood in a broad sense, as the combination of channel, social practices, relations between the users, and codes) »is its ideology« for the former (ECO 1986: Calibre pos. 16.4). Semiotics can and must warn against these two positions. When Stark and Crawford claim that emojis convey their ideology, as if they automatically connoted the content of each message with their further autonomous meaning, they are not carefully considering the role of emojis in the communicative process, which is the exclusive one of code. Emojis, as a code, are thus »empty forms« (ECO 1986: Calibre pos. 16.23) for both the source and recipient of each communicative instance: having no natural or determined meaning prior to their actual uses, the processes by which they signify are open to indefinite attempts of possible creativity, changes, or subversions. Even if it were possible to show that the choice of the emojis made available to users responds, in the articulations of their various subsets, to some precise ideological aims of the platform owners, this would not alter the fact that this is only one of the indefinite ways in which the more general cultural knowledge allows them to be received, used, and interpreted.

Stark and Crawford acknowledge indeterminacy[7] and semantic openness as distinctive features of the affective and emotional communicative practices in which emojis appear. However, from a semiotic point of view, such indeterminacy is not in itself the proper or ideologically determined meaning of the paralinguistic affectivity expressed through emojis. On the contrary, it is the indefinite possibility of many different determined uses of this picto-logographic writing system. The indeterminacy lies at the level of the Saussurean langue, of the abstract system at its highest level of generality, not at the level of the particular uses or conventions, gradually narrower, of national linguistic communities, of sociolinguistic groups identified according to the most varied coordinates (demographic, professional, local, etc.) down to the individual micro-groups of acquaintances sharing a specific background of knowledge. At this very last level, some uses of emojis can take on a meaning highly determined but also extremely unpredictable based on the superficial iconic characteristics of the signifier.

Referring precisely to this possibility of fluctuation in the use of the emojis between ›plain basic‹ and ›cryptic‹ (linked to the knowledge shared at the edge between only two people), the authors acknowledge that the meaning associated with emojis use »would be difficult for intelligence analysts or machine-learning algorithms to parse« (STARK/CRAWFORD 2015: 6). After all, even the ›plain basic‹ meaning of a ›transparent‹ emoji such as ›❤⛱ ‹ could be both rendered as a loving tone as well as a sarcastic tone in different communicative interactions. It is precisely this irreducible dependence on interpretative choices in the here-and-now and/or within strongly local conventions that make it difficult to predict uniform and predictable uses for advertising exploitation or user profiling (for these purposes, simple alphabetical writing seems probably to be more effective, although posing similar problems). Moreover, as we have seen, even considering emojis in their association with the expression of paralinguistic cues, it is reductive to identify the emojis’ ›substance content ‹ with the expression of emotions. On the contrary, what emojis refer to are the broader cultural contexts/frameworks in which certain prototypical emotional aspects can be found. Concerning the purposes of datafication, one must also always assume the rule that the owner of the platform isn’t in control of how the sender uses emojis to encode its message, nor of how the message is interpreted by the addressee (cf. ECO 1986: Calibre pos. 16.14). While the stimulus compels a certain response, the characteristic of signs is precisely that of the dialogical and unpredictable openness of the interpretative response or non-response (both at the moment of encoding and decoding). What is reducible to data does not coincide with the meaning of a message for the users. To conclude, one can comment on this last passage:

Emoji, too, seem trapped in this bind, a ›normcore system of emotion‹: the generic basic-wear of digital communication, ›a taxonomy of feeling in a grid menu of ideograms‹ […]. Emoji can deaden as well as enliven. For all their creative potentials, emoji were intended to normalize and then capitalize on the collective strength of affect in human social relations online (STARK/CRAWFORD 2015: 4).

A semiotic perspective suggests reconsidering the point at which the problem of exploitation and valorization of digital communicative flows should be placed. As said before, communication is never an end in itself and therefore it is not the sum of its codes and channels that are in themselves practices of exploitation or alienation. On the contrary, it is the economic structure in which the communicative flow is embedded and for which it is aimed that could determine a further level of alienation and value extraction (in some cases hidden from the very semiotic consciousness of those who communicate). In this sense, emojis can at most be considered as one among many affordances to make the digital architecture in which communication is embedded more user-friendly (but this, too, is an empirical question: there might be people for whom emojis and the skills necessary for their use are more an obstacle than an encouragement). Regarding the first sentence of the quote, however, it must be replied that it seems impossible to reduce the communicative practices and paralinguistic contents that can be expressed through emojis made available by a certain interface into a predetermined enclosure.

Concludingly, we can return to the example from which we have started: the meaning of👃💨. It is a good example to illustrate one of the many possible ways of semiotic creativity that emojis have enabled. To recover something from the lost spontaneity of face-to-face communication, emojis could be used to represent how the body reacts beyond the screen, not how it would have reacted if it had been face-to-face. For example, our syntagm could stand for what in internet jargon is called »snoffing«.[8] This word means the reaction of amusement expressed by un uncontrolled exhalation from the nose. This is an unusual reaction in speech, but common when reading funny messages from the screen. So, emoji creativity could be used not just to represent the lost closeness of face-to-face communication and to replace the functions usually carried out by the bodies. They can furthermore signify a new paralinguistics, native and specifical of digital communication practices.

References

ABEL, JONATHAN: Not Everyone 💩s: Or, the Question of Emoji as „Universal“ Expression. In: Elena Giannoulis; Lukas R.A. Wilde (eds.): Emoticons, Kaomoji and Emoji: The Transformation of Communication in the Digital Age. New York [Routledge] 2020, pp. 25-43

ANTONELLI, GIUSEPPE: Scrivere e Digitare. In: Treccani, 2009. https://www.treccani.it/enciclopedia/scrivere-e-digitare_%28XXI-Secolo%29/ [accessed March 2, 2023]

BARTHES, ROLAND: Elements of Semiology. Translated by Annette Lavers and Colin Smith. New York [Hill and Wang] 1967 [1964]

DANESI, MARCEL: The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London [Bloomsbury] 2017

CERRUTI, MASSIMO: Varietà dell’italiano. In: Iannaccaro Gabriele (ed.): La linguistica italiana all’alba del terzo millennio (1997-2010). Roma [Bulzoni] 2013, pp. 91-127

DE MAURO, TULLIO: Introduzione alla semantica. Roma-Bari [Laterza] 1965

DE MAURO, TULLIO: Minisemantica dei linguaggi non verbali e delle lingue. Roma-Bari [Laterza] 1982

ECO, UMBERTO: A Theory of Semiotics. Bloomington [Indiana UP] 1975

ECO, UMBERTO: The Role of the Reader: Explorations in the Semiotics of Text. Bloomington [Indiana UP] 1984 [1979]

ECO, UMBERTO: Towards a Semiological Guerrilla Warfare. Translated by William Weaver. In: Travels in Hyperreality: Essays. New York [Harcourt] 1986 [1967], pp. 135-144

ECO, UMBERTO: Apocalyptic and Integrated Intellectuals: Mass Communications and Theories of Mass Culture. Translated by Jenny Condie. In: Roberts Lumey (ed.): Apocalypse Postponed: Umberto Eco. Bloomington [Indiana UP] 1994 [1964], pp. 17-35

EVANS, VYVYAN. The Emoji Code: How Smiley Faces, Love Hearts and Thumbs Up are Changing the Way we Communicate. London [Michael O’Mara] 2017

FILLMORE, CHARLES: Frame Semantics. In: The Linguistic Society of Korea (ed.): Linguistics in the Morning Calm: Selected Papers from SICOL-1981. Seul [Hanshin Publishing] 1982, pp. 111-137

FREEDMAN, ALISA: Cultural Literacy in the Empire of Emoji Signs: Who Is 😂 ?
In: Elena Giannoulis; Lukas R.A. Wilde (eds.): Emoticons, Kaomoji and Emoji: The Transformation of Communication in the Digital Age. New York [Routledge] 2020, pp. 44-66

GENSINI, STEFANO: Elementi di Semiotica. Roma [Carocci] 2002

HJELMSLEV, LUIS: Prolegomena to a Theory of Language. Translated by Francis J. Whitfield. Madison [Wisconsin UP] 1961 [1953]

KARPINSKA, MARZENA; PAULA KURZAWASKA; ROZANSKA KATARZYNA: Emoticons: Digital Lingua Franca or a Culture-Specific Product Leading to Misunderstanding? In: Elena Giannoulis; Lukas R.A. Wilde (eds.): Emoticons, Kaomoji and Emoji: The Transformation of Communication in the Digital Age. New York [Routledge] 2020, pp. 67-82

POYATOS FERNANDO: Nonverbal Communication across Disciplines, vol. II: Paralanguage, Kinesics, Silence, Personal and Environmental Interaction. Amsterdam-Philadelphia [John Benjamins] 2002

ONG, WALTER: Orality and Literacy: The Technologizing of the Word. London [Routledge] 2002

SAUSSURE, FERDINAND DE: Course in General Linguistics. Translated by Wade Baskin. New York [Columbia UP] 2011 [1916]

SIEVER, CHRISTINA MARGRIT: Iconographetic Communication in Digital Media Emoji in WhatsApp, Twitter, Instagram, Facebook – From a Linguistic Perspective. In: Elena Giannoulis; Lukas R.A. Wilde (eds.): Emoticons, Kaomoji and Emoji: The Transformation of Communication in the Digital Age. New York [Routledge] 2020, pp. 127-147

STARK, LUKE; KATE CRAWFORD: The Conservatism of Emoji: Work, Affect, and Communication. In: Social Media + Society, 1(2), 2015, pp. 1-11

VOLOSINOV, VALENTIN NIKOLAEVICH: Marxism and Philosophy of Language. Translated by Ladislav Matejka and I.R. Titunik. New York [Seminar Press] 1973 [1929]

WITTGENSTEIN, LUDWIG: Philosophical Investigations. Translated by G.E.M. Anscombe. Oxford [Blackwell] 1953

Footnotes

---

About this article

Copyright

This article is distributed under Creative Commons Atrribution 4.0 International (CC BY 4.0). You are free to share and redistribute the material in any medium or format. The licensor cannot revoke these freedoms as long as you follow the license terms. You must however give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. More Information under https://creativecommons.org/licenses/by/4.0/deed.en.

Citation

Andrea Ferretti: What do Emojis Stand for?. Notes for a Semiotic View on the Digital Relationship between Written and Spoken Communication Practices. In: IMAGE. Zeitschrift für interdisziplinäre Bildwissenschaft (Themenheft: The Semiotics of Emoji and Digital Stickers), Band 38, 19. Jg., (2)2023, S. 196-214

ISSN

1614-0885

DOI

10.1453/1614-0885-2-2023-15753

First published online

Oktober/2023