Friday, June 02, 2017

Upsetting the Synaesthetic - The Arbitrary Function in Visual Music

Proposition: to counter the synaesthetic, and thus create dialectic in visual music an arbitrary function must be introduced.

Of all the senses, it is sight and sound that work most in tandem, so it is perhaps not surprising that moving images and audio are inherently attracted to one another. At the slightest hint of simultaneity the two adhere, a tendency noted with some anxiety by Eisenstein, Pudovkin and Alexandrov in their 1928 ‘Statement’ on sound, written just as a reliable method for reproducing sync was being introduced in cinema. The Russian filmmakers were concerned about the sight of synchronised lips, and the sound of dialogue producing a theatrical self-contained cinema, something they sought to counter by the use of asynchronous combinations of sound and image. Asynchronism is often misunderstood as audio and vision being in some way out of sync, but what was being proposed is a form of juxtaposition, rather than of sound merely reinforcing image.  

The problem of adhesion is if anything more acute in visual music, for synchronising abstract imagery and music doesn’t create a hermetically sealed story world, but rather potentially the suggestion of a literal equation of the two media. Reflecting this, numerous papers on visual music have titles such as 'seeing sound, hearing colour'.

The tendency to equate sound and image, and specifically pitch and colour is as old as visual music itself: from Louis Bertrand Castel’s “ocular harpsichord for the eyes” built in the 1700s (a keyboard above which were 60 small windows, each with different coloured-glass and a small curtain, which as the player depressed the relevant key would open), through to Rimington, Scriabin and Whitney. Such synasethetic combinations claim not just equation but an underpinning ‘natural’ harmonic relationship between visual and musical forms. Arguably the history of visual music has been dogged by a search for such quasi-spiritual correlation.

So why is this problematic?  Firstly an absolute or natural correlation would require consensus on the precise nature of the relationships, and even Castel changed his mind several times during his many of years of working on his ocular harpsichord as to what hue should match with what pitch. From a mathematical perspective, the principles of western musical notation and harmony do no lend themselves to a systematic translation into a colour wheel, and in any case western musical harmony is only one method for organising sound.

Technicalities aside, the core issue is that the quest for absolute synaesthetic correlation is at odds with achieving dialectic tension in the audio-visual relationship. In synaesthesia, it is not so much sensory interplay that is encouraged, as a form of submersion and sublimation, an immersive running together of the senses, thereby reducing the audience’s potential participation in making meaning. Such synaesthetic inclinations have recently been given impetus within contemporary visual music practice, where digital technologies have opened up myriad new ways of combining sound and image. 

If Eisenstein (et al) sought to upset audio-visual adhesion in narrative film by the use of asynchronism, then what is proposed is that in visual music to offset literalness and a tendency towards immersive synaesthesia, another technique,must be deployed, the arbitrary.

The arbitrary does not denote simple randomness, but rather a changing and reciprocal choreography between the audio and the visual, which recognises, and even embraces the potential for adhesion and equation, but that either then foregrounds these relationships as arbitrary rather than essential, or uses arbitrary elements to problematise the correlations and hence offset the synaesthetic.

The Three Methods
There are three principle ways of introducing the arbitrary. Firstly one can allow adhesion to take place but then willfully multiply the number of audio-visual equations. This process is facilitated by digital mapping, so at its simplest, if say an equation of pitch and colour is created in one part of a work, such that as the frequency rises the hue changes, this can be offset elsewhere, by mapping pitch to changes in form, or some other visual parameter. By shifting and changing the mapping, the arbitrary nature of the audio-visual correlation is foregrounded, and revealed, as the audience find themselves making first one equation and then another.

The second arbitrary function involves questioning causality and can be applied to either abstract or representational imagery, but arguably works best with the latter. In narrative cinema on-screen action or activity is typically perceived as having a causal relationship with the sound one hears. A car pulls up, the engine stops, the door opens and closes, footsteps on the gravel, etc. Indeed sound without an accompanying visual source is often used as a way of creating dramatic tension, building to the moment when the two become united. We hear the sound of footsteps approaching a door before it opens to reveal who is coming through.  In visual music one can question or upset this causal relationship, such that the audience asks, is the image or action producing the sound, or is the soundtrack in some way generating or producing the image? Again digital technology allows us to do this in various ways, for example by either using the same data stream or algorithms to manipulate both image and audio, or by scanning the moving image to produce the soundtrack (in a manner not dissimilar to optical sound in film). So it is the action of the frame, rather than the action in the frame that is the causal link.

The third arbitrary approach is looser and closer to asynchronism, as in this model, moving images are not synchronised note for note with music or sounds, but married with syncopated musical rhythms. Each retains its own channel and identity and the arbitrary element comes in the open-ended nature of this correspondence, there is no beat-by-beat or 4/4 dynamics, locking the audio-visual relationship, but rather flashes of momentary adhesion, possibly taking place at different tempi and at different points within the frame. As such there may be many possible moments of adhesion or of equation. 

This third approach requires no special digital technology and can be seen in the films of Len Lye and his use of Cuban dance band rhythms, and Malcolm Le Grice’s Berlin Horse (1970) with its looping imagery counterpointed by by Brian Eno's piano loops.  The choice of image and music is quite particular however, as though some adhesion and equation will happen if one combines just about any music and moving imagery, without careful selection and counterpointing, the effect can be to diminish rather than enhance both audio-visual elements. Whilst digital technology is not required in this third method, it nonetheless can facilitate the making of syncopated moving images and audio. 

The three methods are not exclusive and elements from the three approaches may be combined.

To follow - practice based examples