Creativity

Needs

“I think it's really important, especially if you're stuck, if you don't know what to do next, just start drawing, start making and something will come out.”
— Just van Rossum, Lunch Lectures: Computational Typography (2023)

What other mediums for creative work can we use, and how does this change the way we work?

During his one-font-a-day project, half of Alex Slobzheninov’s time was spent on the creative process. His problem is probably the opposite of a creative block: too many ideas in too little time. When anyone at Dinamo gets stuck, Fabian Harb likes to talk and discuss together, and see if there is another perspective or idea in the group that could be tried out.

Iteration also seems a bit part of a computational designer’s processes, too. Beatriz Lozano likes to revisit old sketches and tweak them, as does designer André Burnier. “I’ve never had a creative block when I’m working”, he says, “But when I’m doing artistic projects, I do have creative blocks often. To force myself out of it, I’ll just grab an old sketch that I’ve done in the past and start playing with it.”

What role can AI take on in such an iterative making process? Tra Giang Nguyen mostly uses new AI for the first stage of the design process. Talia Cotton agrees, “We use new AI technologies for about the first 10% of every few projects.” AI in general seems to be a great tool for iterative thinking processes and brainstorming. Sander Sturing adds, “When I start coding something, I use [AI] more like a brainstorming tool.”

Just van Rossum mentions that with generative AI imagery, the initial wow-factor is high. But after seeing many generated images, the aesthetic quickly becomes less impressive and a little tiresome. Marc Schütz thinks the most interesting part of generative AI is the association of things that are linked semantically because there is some logic or understanding in this model. “Sometimes I wish there was a more direct way to bring a feeling to a design, as if the computer could read my mind.”

Can AI be used as a visual brainstorming tool for the sketching phase, drafting visual ideas on the fly, without too much impacting the visual character of the sketch?

And, what are other ways of creativity that might get us unstuck, where interfacing gets surpassed by a subliminal connection? What does that mean for type design as a practice?

“Intelligent Typography”, Mao et al. (2023)

Adobe Firefly (2023)

Using AI to create visual typographic sketches is a popular approach, although often it seems more intended for a final format/later in the stage. Mao at al. have experimented with using style transfer to apply complex texture and structure to text [Mao et al. 2023]. Adobe launched FireFly the same year: a generative AI model for other Adobe applications. One of its functionalities is adding custom textures to letters with the use of prompts [Adobe 2023]. GenType by Google promises to let you make an alphabet out of any image [Google 2024]. Smaller initiatives like Word.io’s Word Art Generator or AppiePie’s AI Font Generator also apply similar principles of adding visual textures to lettering. , and emotion detection, although the last one may produce unwanted results when using it to interface.

Primer of Visual Literacy

What if the shape of a text was influenced by some subconscious feature instead of textual input, such as the reader’s emotions? There is a whole canon of work that tries to understand the relation of visual shapes to a universal ‘meaning’ or ‘feeling’. One key example is Primer of Visual Literacy by the first woman dean at Boston University, Donis Dondis, in which she described that abstract compositions can have a “meaning”, specifically a universal mood [Lupton and Miller 2008]. An example of this and spoken text is the famous “bouba and kiki” experiment, where people universally link the sound of a word to the same visual shape. A specific text-related project is the work Affective Typography: The Effect of AI-Driven Font Design on Empathetic Story Reading [Shen et al. 2023].

Bouba Kiki

Experiments

In a way, this is the ‘wildcard’ section of the experiments. This section explores a few ways to circumvent creative barriers and discover how generative AI can assist with making in different ways. Specifically, I experimented with alternative input methods for typographic design. The following explored in this section:

• Converting those styles to fonts;
• Changing a text shape based on human emotion.
• Mapping prompts to visual typographic styles;

C.0: Prompt letters

In this experiment, the goal was to find an interesting way to design lettering through prompt and convert it to a vector format. Similar projects like Adobe’s Firefly specifically apply colorful styling effects to lettering. I find these decorative lettering designs quite over the top and not necessary for type sketching (although of course very helpful for letter sketching), so I could imagine converting this to a vector graphic in just black and white instead.

I started off by setting the environment. The experiment uses OpenAI’s Dall-E in a node.js environment. Node.js is a javascript-like framework that creates a back-end and a front-end. Since Dall-E’s API works with an API key, it seemed to make sense to use a back-end that users cannot access to set this information up without releasing my key in the wild.

Dall-E’s model training sources are unknown, and the training used a lot of energy. Calling the model significantly less so. So moving forward, I was stingy with the amount of calls made, as well as the type and amount of images we call. I also made sure not to refer to any artists or designers and block that out as much as possible.

The image should result in something that could be thresholded to SVG, so this should be considered in the prompt. ${text} is the text to be visualized, ${mood} is the feeling the text should convey, and ${shape} is the shape the text should take on. The first prompt I wrote was The word ${text} in ${mood} lettering on a white background. After many iterations of engineering, the prompt had turned into Generate an image of the word ${text} in a stylized lettering that is shaped like ${shape}. Ensure that the entire word is clearly visible and not cropped. The letters should have a ${mood} style. The background should be white and the lettering should be the only element in the image.

Next to that, I also switched from Dall-E 3 to Dall-E 2. I used the smaller resolution that the earlier version offers, so that not too much data is used. For the sketching tool this doesn’t matter much anyway, because the image will be thresholded and converted to a vector graphic.

I also noticed that negative prompts do not work well. So, if you prompt do not include hands it will absolutely include hands. Not bringing up the pink elephant is key, so instead you can describe the lettering should be the only element in the image.

Prompt: The word [Water] in [elegant, curly, script] lettering on a white background.

Prompt: The word [Water] in [elegant, curly, script] lettering on a white background, large whitespace around text.

One other improvement was the change from image to vector, done with Python’s potrace.

The prompt drawing or illustration inadvertently showed the text on a piece of paper as if photographed on a drawing board and include a set of drawing utensils and hands. Perhaps this is some sort of compensation for its former technical challenge to generate hands.

This improved after updating the prompt to: Generate an image of the word '${text}' in a stylized lettering that is shaped like ${shape}. Ensure that the entire word is clearly visible and not cropped. The letters should have a ${mood} style. The background should be white and the lettering should be the only element in the image.

Some words and letters are hard to draw. This does seem to correlate to visual properties of the semantic meaning of the word.

The design that results from the prompt is still volatile. Completely different designs can come out of the same prompt. To reduce the volatility of the design, the prompt had to be refined further. In prompt guides, the following descriptors are advised to include:

This tool worked well to help with sketching ideas for lettering much more than for type design. Perhaps this is because of the training data. I think I would love to use this for lettering ideas, but indeed just in the brainstorming phase of the text. The Dall-E imagery on its own still gives me an ick because it can get close to “AI garbage”, so I could even imagine just displaying the vector file.

C.2: Emotional letters

We can map this to typographic shapes too. ml5.js, a “friendly machine learning library for the web”, can easily detect the emotions neutral, happy, sad, angry, fearful, disgusted, and surprised.

The following arbitrary visual properties were attributed to the emotions of ML5.js, extrapolated from Primer of Visual Literacy’s outline.

In this first example, feelings are attributed to the outline of the letters. This resulted in a quite interesting way of reading, because I became instantly aware of my facial expression.

As an interface, facial expressions are a bit too far removed from your actual thoughts. Although this example is quite direct and on the nose, I can imagine a feature like this can be part of a design rather than a tool. Imagine reading a text that changes according to the emotions it evokes?