As a 2018 Resident Artist at CultureHub, created a prototype for a space that would visually respond to the live reading of poetry and prose. Thanks to the teaching of Matt Romein and help of Oren Shoham, was able to get a basic version working in four days in June, which was then tested by an amazing assortment of poets, writers, and editors.
As text was read aloud, the words would appear on screen, changing the color of the entire 40 foot space based on what word was read (for example, if you read "volcano" the space turned red). While very basic aesthetically, the project opens up new possibilities for how new media artists and writers could collaborate to perform literary work live.
We used Google Cloud's speech-to-text API, Python, Max/MSP, and the NRC Word-Emotion Association Lexicon created by Saif Mohammad and Peter Turney at the National Research Council Canada.
During the second week in November, we ported the project into Unity with the help of Sam Von Ehren, and then game developers Flan Falacci, Dennis Carr, and Milan Koerner-Safratta spent three days developing prototypes for several types of interactive graphics.
GitHub available here.