Identity-Driven Poetry
I have created a standalone website for exploring identity through creative computing. My software translates any identity statement or manifesto by a user into a distribution of diverse and high-quality art, including poetry.
As a PhD researcher, my work focuses on the intersection of generative AI and the arts. I aim to pioneer individualized generative distributions in artistic creation through a combination of theoretical and technological advancements.
My dissertation involves a neurosymbolic analysis of images, folk music, and poetry, identifying gaps in existing theoretical models. This led to the development of a neurosymbolic model that combines empirical data with domain-specific theory. This model introduces a new data structure called a “prototype graph," redefining musical and poetic form as relationships between prototype nodes and their associated spans. This work began with a 2019 ICML paper and was continued in a 2022 NEURIPS paper.
My research in electronic music examines the computational semantics and Turing completeness of Digital Audio Workstations, with a specific focus on Ableton. I have dissected the computational capabilities of DAWs, exploring their potential as complete computational systems and their role in shaping our approach to music creation and analysis.
In addition, I have proposed a novel method for analyzing DAW files, treating them as structured programs requiring a blend of program analysis, digital signal processing, and traditional harmonic and rhythmic analysis. This approach equates DAW files in electronic music to traditional scores in classical compositions.
For more detail on this work, see the following preprints:
To put all my ideas together, I have used my knowledge of both traditional and deep-learning-based theories of poetry and various types of music to translate poetic and musical "identity statements" into AI-interpretable language. This has led to the creation of AI-generated poems and music that demonstrate a more nuanced and diverse range of artistic outputs, effectively creating a unique distribution of poetry or vocal music for each identity statement. For a comprehensive display of this identity-driven paradigm, go to my project webpage.
Going forward, my research aims to establish a new paradigm in the generation of creative output, where individuals can use their unique identities to create art, experiences and productive tools that reflect their individual visions. The integration of theory, technology, and creativity will contribute significantly to the advancement of generative AI and open new avenues for personalized expression.
I have built LLMs from scratch for music generation, including developing a novel positional encoding system to account for the specifics of the domain in how sequences are represented. Some example outputs can be found here.
I have worked extensively with Google DeepMind on developing novel methods to steer expert-guided features of non-text-based transformers using variants of prompt and bias tuning. My work is published in the the HAI-GEN proceedings.
I have developed intensive pipelines, algorithms, and methods to convert ChatGPT output to creative artistry and software across many domains:
I have created a standalone website for exploring identity through creative computing. My software translates any identity statement or manifesto by a user into a distribution of diverse and high-quality art, including poetry.
More at this link.
The field of neurosymbolic computing represents a groundbreaking fusion of neural networks and symbolic reasoning, offering powerful solutions even without large amounts of data. This approach is especially useful in scenarios where data scarcity poses a challenge, providing a robust framework for understanding and generating complex structures.
The versatility of neurosymbolic models lies in their ability to integrate empirical data with structured, logical frameworks. This integration not only advances the field of AI but also opens up new avenues for creative expression and analysis in various domains.
My ongoing research in neurosymbolic computing aims to further refine these models, enhancing their applicability and effectiveness across different fields. The goal is to establish neurosymbolic models as a standard tool in scenarios where data limitations are a significant challenge, thereby broadening the horizons of AI research and its practical applications.
For inquiries or collaboration opportunities, please reach out at halleyy@seas.upenn.edu.
My thesis proposes a groundbreaking neurosymbolic model for musical form, a pivotal innovation in understanding and generating harmonious compositions. This model bridges the gap between music theory and computer science, offering new insights into musical form through the lens of advanced data structures and algorithms.
My dissertation epitomizes my commitment to transforming academic music theory into practical, generative applications. This ambitious endeavor bridges the gap between theoretical concepts and their empirical validation and application.
This facet of my research underscores a novel approach in music theory - using logical and mathematical models as a foundation for music generation, enabling a deeper understanding and exploration of music's emotional and theoretical dimensions.
My collaboration underscored my commitment to making generative music theory accessible and practical for composers.
The outcome of this collaboration highlights the potential of technology to enhance musical creativity, providing composers with new tools to shape their musical ideas in the realm of automatic music generation.
My research delves deep into the capabilities and complexities of Digital Audio Workstations, revealing their immense potential in the realm of music theory and production.
This avenue of research not only highlights the importance of DAWs in modern music theory and production but also opens new pathways for understanding and appreciating genres like hip-hop and beatmaking, traditionally overlooked in academic studies.
My research has made significant strides in improving the sound quality and interpretability of computer-generated music, focusing on the innovative use of DAWs for music generation.
This work not only propels the field of music composition forward but also underscores the untapped potential of DAWs in enhancing the creative capabilities of composers across various genres, particularly in popular music with vocals.
In my dissertation, I embarked on a pioneering journey to reevaluate and redefine traditional Western models of melody structure in folk music. Integrating advanced computational models with a deep understanding of music theory and computer science, I developed a novel approach that challenges both conventional Western music theory models and purely theoretical models in musicology. This work aims to demonstrate the limitations of existing models and advocate for a more nuanced, computationally informed approach to understanding and generating folk music melodies.
The heart of my research is the development of a neurosymbolic model for musical form, introducing the innovative 'prototype graph' data structure.
This prototype graph represents musical form as a network of relationships between 'prototype nodes' (specific musical elements) and the music they represent, as well as the music-theoretic relationships between musical spans and their respective nodes. Understanding these relationships, both generative and classificational, is key to grasping the model's capacity to create and categorize musical structures.
My hypothesis challenged two prevailing models: the traditional "antecedent-consequent" or "period/sentence" models found in Western music theory, and neural models devoid of music-theoretic grounding. I posited that these models were inadequate in capturing the diversity and complexity of forms found in folk music.
A significant aspect of my research involved applying novel harmonic constraints to the generated music, diverging from typical characteristics seen in folk music. This included variations inspired by composers like Debussy and Messiaen, challenging the model's ability to maintain formal coherence while exploring the essence of a folk song's appeal.
In addition to my dissertation work, I have delved into the fascinating world of comparative musicology, using formal logic to uncover previously unobserved parallels and distinct divergences in global musical traditions. This exploration spans a wide array of musical styles, from the serialist melodic contours of Robert Morris to the rhythmic intricacies in the performances of Armenian artist Tigran Hamasyan.
Using first-order logic as a primary tool, my research systematically compares different musical traditions, offering new perspectives on the universal and unique aspects of global music.
The cornerstone of this research is the application of first-order logic to dissect and understand complex musical concepts, allowing for a systematic comparison of different musical traditions and facilitating a nuanced understanding of their underlying structures. For instance, I contrast formal definitions of Western common practice music with aspects of music from other cultures, such as Hindustani gamaka.
In my publication of this work at the Programming Languages and Interactive Entertainment (PLIE) workshop, I demonstrated the usefulness of SMT solvers, an advanced technique in applied logic, in automatically converting formal theories into actual music. These SMT solvers take in first-order logic specifications and output specific “world models” - in this case, pieces of music - that satisfy those specifications. This approach enables a precise analysis of how specific musical features evoke certain cultural contexts or themes.
Digital Audio Workstations like Ableton or Logic represent a fusion of advanced computing and creative music production, offering an unprecedented scope for musical innovation. I began this research by establishing that DAWs are Turing complete, highlighting their immense computational power and versatility. Next, I demonstrated the undecidability of common queires about DAW files in various music-making modes, discovering fundamental boundaries to our ability to understand their capabilities.
I explore the intricate layers of electronic compositions through a comprehensive method combining program analysis, digital signal processing, and traditional harmonic and rhythmic analyses.
The proposed method for analyzing DAW files treats them as complex programs. The combination of program analysis, digital signal processing, and traditional harmonic and rhythmic analyses is pivotal in understanding the creative and technical aspects of DAW-based music.
I built a database of over 2400 DAW programs, then applied program analysis and digital signal processing to extract musical insights from the database. I then began developing a music-theoretic language for discussing DAW-produced music. My hope is that having a language equal in sophistication to Western score notation will inspire more work understanding these popular but understudied genres.
This research venture embarks on an ambitious goal: enabling artists across various mediums to harness technology and theory for creating unique expressions true to their personal visions. It's a journey through the realms of digital art, music, narrative, and poetry.
Artists are empowered to articulate their 'aesthetic identities' through personalized manifestos. These statements serve as a blueprint for AI to generate works that are authentic reflections of the artist's individuality. The ultimate objective is a comprehensive system that fosters a new era of identity-based artistic generation, offering a platform for artists to produce deeply personal and technically sound art.
A pivotal discovery in my research, 'identity-based computing' represents a breakthrough in the fusion of AI and art. This concept involves utilizing 'aesthetic identity statements' to fine-tune AI models like ChatGPT, enabling the creation of art that not only showcases technical excellence but deeply resonates with the artist's personal vision and identity.
By defining their aesthetic identities, artists guide AI in producing works that are true reflections of their unique perspectives. This approach marries the precision and capabilities of AI with the nuanced and subjective realm of artistic creation, opening up new avenues for personalized art generation.
This innovative approach is not just a technological advancement but also a step towards understanding and appreciating the role of personal identity in the realm of art and creativity.
My collaboration with Charles Bernstein, a distinguished poet known for his deep exploration of language and form, marks a pivotal moment in my research. This partnership has birthed a groundbreaking approach where poetic 'identity statements' are translated into AI-interpretable language, leading to the creation of AI-generated poetry.
This innovative method is centered on empowering artists to express their aesthetic identities, allowing AI to produce works that genuinely reflect these individual visions. Bernstein's involvement in this project has been especially insightful. Through his experimentation with different personas and styles, such as embracing Yeats' poetic style or developing unique manifestos, he has effectively showcased the profound influence of identity on creative output.
This journey not only highlights the capabilities of AI in artistic creation but also illuminates the significant role of personal identity in the realm of creative expression.
A key technical achievement of our research is the transformation of each identity statement into a unique generative model distribution. This innovative approach allows us to derive an unlimited array of artistic pieces from a single identity statement, effectively creating a boundless spectrum of artistic expression tailored to individual identities.
Crafting the language of these identity statements requires extensive domain expertise. The challenge lies in translating a wide array of diverse and complex identity statements into art that authentically aligns with each identity. This process necessitates a deep understanding of both the artistic domain and the intricacies of AI technology.
Our empirical findings, particularly in the poetry domain, provide substantial evidence that the generated output closely aligns with the identity statements. This not only validates our approach but also paves the way for further exploration and refinement in other artistic domains.
Looking ahead, my ambition is to forge partnerships with art therapists and music therapists, and to deepen engagement with professional artists across a wide range of disciplines. This research is not just about pushing the boundaries of artistic creation; it also explores the therapeutic potential of AI-enhanced art for personal healing and expression.
The versatility of our developed system allows it to cater to a wide array of artistic and therapeutic needs. From facilitating the philosophical explorations of renowned poets to providing a medium for young individuals facing personal challenges, this technology adapts to the unique requirements of its users. The system’s adaptive quality highlights its capacity to tailor creative expression, meeting the diverse and deeply personal needs of a broad spectrum of individuals.
My ongoing research is dedicated to refining this technology, broadening its influence and utility, and unlocking new dimensions of the human experience through the integration of AI and identity exploration in both artistic and therapeutic contexts.