. : Hogar Collection : .

All material in the Artforum Archive is protected by copyright. Permission to reprint any article from the Artforum archive must be obtained from Artforum Magazine.

Lost in Translation: Sound in the Discourse of Synaesthesia

In the late 1940s, radio engineer-turned-composer Pierre Schaeffer celebrated a defining property of audio recording and radio transmission: the ability to separate sounds from their visible sources. This affirmation cut against the grain of modern thought, for no lesser cultural critics than Martin Heidegger, Theodor Adorno, and Max Horkheimer had assailed these technologies for dulling our auditory sensibility. Schaeffer, however, argued that records and radio triumphantly subvert the hegemony of vision to make possible the experience of "sound as such." In doing so, Schaeffer continued, they revive a neglected form of listening he termed "acousmatic," in deference to the ancient akousmatikoi, disciples of Pythagoras who were made to listen to their master's voice while he was hidden behind a curtain. (1)

Schaeffer's position remains a significant one within the practice of sound art today. Indeed, any sound art worthy of the name affirms something of this effort to restore to sound its ontological and aesthetic value. (Such insistence on the autonomy of sound and its acousmatic experience is manifested most dramatically now in the work of Spanish composer and sound artist Francisco López, who performs behind a shroud, urges his listeners to don blindfolds, and delivers sonic abstractions that thwart recognition of the environmental sounds from which they are generated.) Yet for the most part, contemporary sound artists and their curators have been interested in negotiating the visual, rather than rejecting it wholesale. In fact, the very tension of such negotiation is what animates this uncertain art form operating between music and visual art, medium specificity and a postmedium condition.

This provocative ambiguity becomes particularly evident as one compares institutional presentations of sound art since its coming to prominence in the late 1990s. Exhibitions such as "Volume: Bed of Sound" (P.S. 1 Contemporary Art Center, New York, 2000), "Rooms for Listening" (CCA Wattis Institute for Contemporary Art, San Francisco, 2000), "BitStreams" (Whitney Museum of American Art, New York, 2001), and the 2002 Whitney Biennial, for example, implicitly adopted Schaeffer's paradigm, providing banks of headphones or darkened rooms for the acousmatic delivery of audio. A contrary approach was taken in last year's "Treble" exhibition at SculptureCenter in New York, which foregrounded the use of sound as a tool to link media from drawing to sculpture.

But, a third strategy, surely the most widespread and significant, has come to prominence in a spate of recent shows presenting sound under the banner of "synaesthesia," an aesthetic appropriation of the neurological condition in which stimulation of one sensory modality triggers involuntary sensation in another. Sound artists and curators have long flirted with the "synaesthesia" idea. However, this engagement has lately emerged—problematically, it must be said—as the dominant mode of conceiving conjunctions between the sonic and the visual. At least five exhibitions have been organized under this rubric within the past year: the largely overlapping blockbusters "Visual Music" (Hirshhorn, Washington, DC/moca, Los Angeles) and "Sons & Lumières" (Centre Pompidou, Paris), which reconsidered the history of modernism as a story of crossovers between sight and sound; then three smaller shows assembling new work in this vein, "Synaesthesia: A Neuro-Aesthetics Exhibition," mounted at London's Institute for Contemporary Arts last fall, and "What Sound Does a Color Make?" and "Crossed Circuits," held in New York this past summer at Eyebeam and the Hogar Collection, respectively.

What are we to make of this new fascination with synaesthesia? And what are the stakes for the very conception of sound art? To answer, it's worth noting that the art world's attraction to sensory cross-wirings is in fact part of a more general cultural formation. In contemporary science, for example, freak occurrences of colored hearing or tactile smell—dismissed as fakery for much of the twentieth century—have suddenly become the subjects of a booming industry in the fields of cognitive psychology and neurology. Developments in contemporary technology also promote the idea of synaesthesia. Brain-imaging technologies used to explore the phenomenon are, paradoxically enough, themselves synaesthetic in their psychedelic visual representation of nonvisual sensory phenomena. This quality points to the more general fact that digital technologies offer, if not a union of the senses, then something akin: the intertranslatability of media, the ability to render sound as image, and vice versa. As Friedrich Kittler, who has written extensively on communication technology, puts it: "The general digitization of channels and information erases the differences among individual media. . . . Inside the computers themselves everything becomes a number: quantity without image, sound, or voice. And once optical fiber networks turn formerly distinct data flows into a standardized series of digitized numbers, any medium can be translated into any other." (2)

Finally, the new discourse on synaesthesia must be considered against the background of a broad revaluation of the senses and their traditional hierarchy—particularly the modern supremacy of vision over audition, sight over sound. Since the 1960s, theorists from Marshall McLuhan and Walter Ong to Jacques Attali and Thomas Docherty have forecast this sort of revolution in the sensorium. And while skeptics have treated these claims as purely speculative or merely wishful, there is growing empirical evidence for them. Witness, for example, the recent profusion of historical and anthropological scholarship—collected in volumes such as Hearing History: A Reader and Hearing Cultures: Essays on Sound, Listening, and Modernity (both 2004)—that takes its evidence not primarily from sight, but from sound. (3)

The emergence of sound art as a prominent practice is aligned with this more general revaluation of the senses and, particularly, of hearing. And the curatorial and artistic interest in synaesthesia is surely a strategy for dealing with this emergence of auditory culture. Thus far, this strategy has been complex and ambivalent. On the one hand, the synaesthesia paradigm recognizes and invites sound as an aesthetic element; on the other, the paradigm still privileges the old order, conceiving sound under the hegemony of the visual and thwarting the development of a genuine sound art.

Consider the example of "Visual Music," which was not so much a historical as a genealogical effort to disclose the modernist lineages of current sound-art practice and synaesthesia discourse. The exhibition was subtitled "Synaesthesia in Art and Music Since 1900"; yet, true to its main title, it clearly favored the image. (In contrast, "Sons & Lumières" only tangentially flirted with synaesthesia, rightly subsuming it within a more expansive examination of sound-image relationships in the multimedia art of the twentieth century.) From the "musical" canvases of Wassily Kandinsky and Franti?sek Kupka to the musically inspired but silent projections of Thomas Wilfred and Leo Villareal, "Visual Music" almost solely presented one-way translations from sound into sight. One may reasonably ask why the curators did not complement these selections with sound-centered classics such as Walter Ruttman's film-without-images Weekend, 1930; Luc Ferrari's Presque rien (Almost Nothing), 1970, a sonic portrait of dawn in a Dalmatian fishing village; Derek Jarman's monochrome film, Blue, 1993; or a Janet Cardiff audio walk. Or why the exhibitions "Crossed Circuits" and "What Sound Does a Color Make?" similarly centered on the image, focusing attention on screens, photographs, and drawings that occluded sound by standing in for it, much in the same way that the mute but visible score came, within modernity, to circulate as the musical work itself.

However lamentable, this imbalance of media and sensation is true to the neurological experience of synaesthesia. This condition (and the aesthetic analogy to it) may hold out the ideal of sensuous plenitude and cross-mixing; yet by far its most common expression is the unidirectional visual experience of sound. (Sound provoked by sight is extremely rare. (4) In the aesthetic domain, even when generated to enhance aural experience—for example, the '60s light shows featured in "Visual Music" or, today, the iTunes Visualizer—the visual is almost never a mere supplement to the auditory. As film theorist Christian Metz points out, our syntax and entrenched sensual hierarchy hold us in thrall to a metaphysics according to which sight and touch signify being and presence, while sound—spatially vague, materially elusive, and temporally ephemeral—signifies absence and can only have the status of a secondary "attribute" in relation to a primary visual and tactile "substance." (5) Cinema might in principle be a synaesthetic art, an intersensorial conjunction of sound and image. In practice, however, cinematic sound is almost invariably subservient to the image. So too is it with synaesthetic art more generally. Indeed, the dominance of the visual in synaesthetic art corresponds with the prevailing idea that sound-in-itself is unnatural or inadequate, in need of an anchor in the visible.

This situation was already evident in early experiments with the visualization of sound, most prominently those of Ernst Chladni, an important but, until recently, neglected figure. In 1787, Chladni drew a violin bow along the edge of a metal plate covered with a thin layer of sand. The vibrating plate bounced the granules into symmetrical forms: stars, waves, grids, and labyrinths. Chladni's demonstration made visible and palpable the hitherto elusive and fleeting materiality of sound. Napoleon was so impressed that he put Chladni on his payroll. Friedrich Nietzsche was also intrigued; but he warned of a potential misapprehension of Chladni's results: "One can imagine a man who is totally deaf and has never had a sensation of sound," he wrote. "Perhaps such a person will gaze with astonishment at Chladni's sound figures; perhaps he will discover their causes in the vibrations of the string and will now swear that he must know what men mean by ‘sound.'" (6) Wary of the attempt to reduce sound to sight, Nietzsche insists that the visual and the auditory constitute separate spheres and that the relationship between the two can only ever be a matter of translation or metaphor (in the etymological sense of "carrying over") that bears the traces of an unassimilated remainder. For Nietzsche, the distinction between the metaphorical and the literal is simply that the latter no longer acknowledges the difference that constitutes it, taking itself to be what it represents. Such literalness is a chief characteristic of the aesthetic discourse of synaesthesia today.

As Kittler notes, the ready translatability of digital media encourages this literalness. The fact that all digital material shares a common base—binary code—supports the illusion that sound, image, word, and movement can be made identical and interchangeable. What is forgotten is that they can be made so only via the intermediary of arbitrary mapping formuli decided in advance. This disparity was evident in the sound-image juxtapositions of numerous pieces in "What Sound Does a Color Make?" When sound and image pulse together in Granular-Synthesis's video projection LUX, 2003, for example, it is because they have been programmed to do so, not because of any real correspondence between these sounds and images. The same is true of the correspondence between vocal tones and luminosity in Jim Campbell's Self-Portrait of Paul DeMarinis, 2003, which uses sound to draw a pixelated figure with LEDs, and the relationship between the viewer's movements and the alteration of image and sound in Atau Tanaka's Bondage, 2004, which features a mutating digital photograph projected onto a shoji screen.

Presented almost tangentially in "Visual Music," an excerpt from Oskar Fischinger's film Ornament Sound, circa 1932, provides a very different and much richer model for sound-image translation. The German abstract filmmaker drew bands of jagged and undulating patterns across the optical soundtrack and extending into the visual frame. These forms, read by the projector as both sound and image, produce corresponding bursts of multitextured and variously pitched noise. Sound ceases to be a mere accompaniment to image or suture for visual cuts, but instead collaborates directly with image in the production of a genuine audio-visual experience. Indeed, Fischinger's forms appear as stylized versions of ordinary optical sound bands and thus draw attention to the sound track both visually and aurally.

While scarcely followed in the past seven decades, Fischinger's model has been rejuvenated today in a number of works utilizing audio-visual feedback loops without any intermediary, from Austrian video artist Billy Roisz's __AVVA,__ 2004, to installations such as Carsten Nicolai's Telefunken, 2000, and Scott Arford's Static Room, 2003. Arford, for example, begins by generating a palette of video static: granulated washes, throbbing bands, and pixel fragments. He then runs the video output through the audio input, producing a matching gamut of noise: dense blasts, dirty pulses, and disintegrated drones. The translation is effected not by the conversion of color and sound to a neutral digital substratum, or by the idiosyncratic sensual associations of the artist, but is, rather, effected simply by the routing of the electronic signal and the medium of display. Herein lies the true potential for a sound-art discourse steeped in a multisensory approach. Where indirect and arbitrary digital translation too often attempts to elide the differences between media and sensory modalities, this direct, analog translation does the reverse, intensifying sensory differences and the materiality of the video medium.

Such procedures and results not only revive Fischinger but recapitulate the work of video pioneers Nam June Paik, Steina Vasulka, and Gary Hill—selections by whom were included in "What Sound Does a Color Make?" to the detriment of the more recent work on view. For example, Hill's Full Circle (formerly Ring Modulation), 1978, brilliantly foregrounds both correspondences and differences among sound, text, and image. A video screen divided into three sections presents several ways of rendering the sound of a voice that repeatedly drones an "ahhh" sound held for various durations and wavering in pitch and volume. In the upper left portion of the screen, an oscilloscope image represents the sound as a jittery, rotating ring; below, a close-up image shows a pair of hands bending a metal wire into a circle; above right, the screen presents a distorted full-body view of this wire-bending exercise. True to the theme of circularity, one is never certain which element controls which others. The voice seems to be attempting to form a perfect circle in the oscilloscope image that represents it, but it could also be matching the hand movements that seem to mimic its vocal fluctuations. The malleable wire and quivering ring also reference the vibrating vocal chords; and the title of the piece puns on the verbal connection between the vocal-visual-manual efforts to form circles and the technical process of generating dissonance by multiplying electronic signals (ring modulation).

From a quarter-century's distance, Hill's piece presents an appropriate directive to sound art today and underscores the deficiencies of facile synaesthetic discourses. The best sound works neither reject the visual nor succumb to it, but instead amplify differences among media and sensory modalities, drawing attention to sound as a semiautonomous power. They are complex engagements with the visual that intensify the moment of translation and the movement of metaphor (in Nietzsche's sense of the term). For the silence of the visual can cut two ways. It can stifle or, as John Cage taught us, powerfully disclose sound. Exemplary instances of this second sense are recent installations by Stephen Vitiello, which feature suspended speaker cones that tremble inaudibly. At once mouths and ears, these mobilized membranes draw attention to the kinetic energy of sound, the vibrations that constitute its production and reception.

Or consider Christian Marclay's The Sound of Silence, 1988, a framed photograph of Simon & Garfunkel's 1965 single "The Sounds of Silence"—displaying, in effect, the record as a mute visual object. The piece is a joke, but an epistemologically and ontologically profound one, the humor of which consists in an evident confusion of categories: Photograph, object, and text are absurd because they cannot be what they claim they are. Sound is thereby shown to be of another order, one inadequately represented or even foreclosed by the imaginary domain of the visual and the symbolic domain of the written word. Sound is real, Marclay's work seems to say, something too quickly forgotten by the fantasy of a "union of the senses," which remains a visual fantasy. Genuine sound art today is fostered not by this consensus but by a dis-sensus that gives sound and hearing their due.

Christoph Cox is associate professor of philosophy at Hampshire College and coeditor of Audio Culture: Readings in Modern Music (Continuum, 2004).

1. See Martin Heidegger, "The Turning," in The Question Concerning Technology and Other Essays, trans. William Lovitt (New York: Harper & Row, 1977), 48; Theodor Adorno, Essays on Music, ed. Richard Leppert, trans. Susan H. Gillespie (Berkeley: University of California Press, 2002), 213–317; Max Horkheimer, Dawn and Decline: Notes 1926–1931 and 1950–1969, trans. Michael Shaw (New York: Seabury Press, 1978), 162; and Pierre Schaeffer, "Acousmatics," in Audio Culture: Readings in Modern Music, ed. Christoph Cox and Daniel Warner (New York: Continuum, 2004), 76–81.

2. Friedrich Kittler, Grammophon, Film, Typewriter, trans. and with an introduction by Geoffrey Winthrop-Young and Michael Wutz (Palo Alto, CA: Stanford University Press, 1999), 1–2.

3. Hearing History: A Reader, ed. Mark M. Smith (Athens, GA: University of Georgia Press, 2004), and Hearing Cultures: Essays on Sound, Listening, and Modernity, ed. Veit Erlmann (Oxford and New York: Berg, 2004).

4. See Richard Cytowic, Synesthesia: A Union of the Senses, 2nd edition (Cambridge, MA: MIT Press, 2002), 16. See also Steven Connor, "Intersensoriality," www.bbk.ac.uk/english/skc/intersensoriality/.

5. Christian Metz, "Aural Objects," in Film Sound: Theory and Practice, eds. Elisabeth Weis and John Belton (New York: Columbia University Press, 1985), 154–61.

6. Friedrich Nietzsche, "On Truth and Lies in a Nonmoral Sense," in Philosophy and Truth: Selections from Nietzsche's Notebooks of the Early 1870s, ed. and trans. Daniel Breazeale (Atlantic Highlands, NJ: Humanities Press, 1979), 82.