Hypertype
This brief article attempts to give some context to the artistic project, Hypertype which is a generative art piece that was exhibited in London, November 2022 with Verse Works. In a nutshell, Hypertype uses text emotion and sentiment analysis data as its main content. This content is given form through the organisation of words, typographic signs, letters and symbols. It is first and foremost a textual artwork that attempts to draw our attention to a particular use of AI and the concept of computers automating humans.
Hypertype is the result of a long period of personal research on the topic of text sentiment analysis. When I first came across this technology some years ago now, I was intrigued by how a machine was capable of labelling words with sentiment and emotion. The first question to arise was simply; how does it do this? How can a machine analyse text and then make some inference about whether that text is positive, negative and furthermore how do words become classified as being happy, sad, angry, fearful or disgusting? That was the beginning of what became a long and sustained interest in a particular field of artificial intelligence, what is called natural language understanding. I am not a computational linguist and I have no deep understanding of the algorithms used in this area. However, language is a strong undercurrent to a lot of my thinking. It has been ever since my undergraduate days as a student of modern languages where Beckett and Barthes both sparked interest in learning more about our relationship with language. Indeed, language, in its most widely considered sense, has shaped my thinking in many ways, so when I started to tinker with computational language, I became particularly fascinated with this idea of machines producing content on which we humans project concepts and eventually meaning.
Current machine learning models used to extract features from a text and classify words into different emotions depend on neural networking techniques such as recursive neural tensor networks RNTN and LSTM (Long-Short-Term Memory). The particularity of these techniques resides in their capacity to both ‘learn’ and ‘memorise’ relation extraction, i.e. semantic relationships across textual content. This enables the model to retain some context which is quite a leap from earlier models that relied on crude reference dictionaries that simply listed words labelled with emotions and/or sentiment. There are a lot of academic papers out there which delve into the technicalities of how these systems work. It makes for rather dry reading and personally I found it challenging to grasp the intricacies to say the least. While perhaps having some understanding of what these models do can open some doors, I realised quite quickly that I wasn’t all that interested in the concept of the machine operating on mass data with a prediction model. At some point, another, more pertinent question arose from all this: What is an emotion? A human emotion that is.
At the time I was reading quite a lot of Oliver Sacks’ books and while he never really wrote about emotion per se, he somehow pointed me in the direction of work by Robert Sapolsky and Antonio Damasio. These two leading figures are working very much in the field of emotion and they opened up a fascinating field for me to read and learn about. There is a fourth person to add - Dr Lisa Feldman Barrett who led me to an interesting understanding of emotion. What drew me to the work of Dr. Barrett is the fact that she pulls apart the classical view of emotion which states that emotions are somehow hard-coded into our genes, inhabit certain areas of the brain, are universal and hence measurable and quantifiable. Her current research in this field shows that none of these theories is entirely true. She mentions the work of a certain Paul Ekman whose research since the 1960s has fuelled this universal view on emotion. Moreover, Ekman’s research is most widely considered as being behind the founding ideas for the highly contested facial recognition technology that is becoming so prevalent in our societies.
There is this particular quote from Dr. Barrett that makes so much sense to me; “Emotions are not reactions to the world. They are constructions of the world.” Emotions are meaning. We create emotions through experience, association, memory recall, we predict and we categorise. Emotions are based on concepts that are learned and are culturally, socially, as well as personally orientated. She goes on to say that contrary to what we think of emotion, as being some common state that we share across cultures, variation is in fact the norm. I found Dr. Barrett’s work very inspiring and it gave me lots to think about when it came to language too. Emotions are constructs and we essentially create as well as reinforce these with the concepts and language we learn and use on a daily basis. We could say that emotion concepts are body-states/behaviours associated to words and that these words and their meanings are incredibly varied.
What Sapolsky and Damasio added to this understanding of emotion was that emotions were not just tied to the brain, rather that the body had an essential role in how we experience and construct our emotions. Damasio has written extensively on the subject matter and really digs deep into the various mechanisms of feelings, emotions, thoughts and indeed culture. While his book, ‘Descartes' Error’ challenges the traditional ideas about rationality, postulating that our decisions are motivated by our affect and emotions, he takes these theories much further in ‘The Strange Order of Things’, explaining how cultures are made through this complex tissage of emotional states. With these various insights, the computational inference of emotion, by which a machine is claimed to be able to detect and measure human emotion, began to look more and more suspect. Indeed, from my point of view, the so-called science driving the belief of such a capacity clearly did not add up.
At this particular point, Hypertype was not even an intention as an artistic project. In fact, at the time I was working with generative sound and had a loose idea revolving around using text sentiment analysis of media articles - the so-called News basically - as a means for generating sound. I would use the data as raw material to drive oscillators. The project entitled The Drone Machine is an on-going one that has resulted in four main publications so far. Three as concept albums and one which aired on radio 19.12.20 as part of a sonic festival about sleep. For me, sound is the perfect medium for tapping into our emotions and with the Drone Machine I was seeking a meditative experience, one that drew us towards introspection. Hence the following remark ; “What I am trying to achieve is a means for transformation of one media that is manipulative of one’s attention in order to instil dogma, to another that is manipulative of one’s attention in order to encourage introspection.”
Hypertype was a natural progression from this idea but the challenge was to make something visual. Sound has strong links with our emotional states but within the visual field, this is quite different. It was quite clear that I didn’t want to make this into some kind of data visualisation of emotion and I certainly didn’t want to fall into the trap of using colour as some means for communicating emotion either. I gave myself two constraints for Hypertype : The visual should be entirely textual. That is to say that the visual field is made up of text - a font if you like. Secondly, these textual elements should derive from the articles & papers that I read during my research. I wanted the data to serve as material like paint or clay serves a painter or a sculptor and I wanted to lay bare textual elements that could be read but were also at times more suggestive and perhaps ambiguous.
The program I wrote to make Hypertype is relatively simple but relies on a large amount of data that had been prepared beforehand. The program pulls in this data and uses it to place words and typographic form within a given space. So, the compositional elements, that is to say how the text is positioned, the resolution and the colours chosen are in part determined in some way by the data. I do also use quite a bit of randomness in the program. Again, my intention was not a data visualisation - there is no strict analogy to be made between the graphic elements and the data.
There are two main things I really like about the work. Firstly, on a purely formal level there is a visual interaction between the typographic elements. When some letters align perfectly on the grid, the overlap of the words and signs produce the effect of another language. This brings me back to text as a material. There are also moments when the signs accentuate the serif character of the letters, break the geometry or cover completely certain letters. Patterns form which give both rhythm and direction to the overall composition of the work. It brings me back to this idea of something textual and also this idea of patterns as meaning.
The other aspect that I like is that there is a communicative dimension. The text tells us something and yet the text is not a coherent arrangement of sentences, rather they are single words, names, concepts and occasionally phrases. The work orientates us towards a topic at hand - AI - yet there is room for interpretation, possibilities to engage with the work and read into the text to create one’s own narrative. The intention is clear here. Hypertype is purposefully open. There is visual content based on a language system and there is visual context based on a subject matter - computers automating humans.