Proposition: to counter the
synaesthetic, and thus create dialectic in visual music an arbitrary function
must be introduced.
Introduction
Of all the senses, it is
sight and sound that work most in tandem, so it is perhaps not surprising that moving images and audio are inherently attracted to one another. At
the slightest hint of simultaneity the two adhere, a tendency noted with some
anxiety by Eisenstein, Pudovkin and Alexandrov in their 1928 ‘Statement’ on
sound, written just as a reliable method for reproducing sync was being
introduced in cinema. The Russian filmmakers were concerned about the sight of synchronised
lips, and the sound of dialogue producing a theatrical self-contained cinema, something
they sought to counter by the use of asynchronous combinations of sound and
image. Asynchronism is often misunderstood as audio and vision being in some
way out of sync, but what was being proposed is a form of juxtaposition, rather
than of sound merely reinforcing image.
The problem of adhesion is if anything more
acute in visual music, for synchronising abstract imagery and music doesn’t
create a hermetically sealed story world, but rather potentially the suggestion
of a literal equation of the two media. Reflecting this, numerous papers on visual
music have titles such as 'seeing sound, hearing colour'.
The tendency to equate sound
and image, and specifically pitch and colour is as old as visual music itself:
from Louis Bertrand Castel’s “ocular harpsichord for
the eyes” built in the 1700s (a keyboard above which were 60 small windows,
each with different coloured-glass and a small curtain, which as the player
depressed the relevant key would open), through to Rimington, Scriabin and Whitney.
Such synasethetic combinations claim not just equation but an underpinning ‘natural’ harmonic relationship
between visual and musical forms. Arguably the history of visual music has been
dogged by a search for such quasi-spiritual correlation.
So
why is this problematic? Firstly an
absolute or natural correlation would require consensus on the precise nature of the relationships,
and even Castel changed his mind several times during
his many of years of working on his ocular harpsichord as to what hue should
match with what pitch. From a mathematical perspective, the principles of western musical notation
and harmony do no lend themselves to a systematic translation into a colour
wheel, and in any case western musical harmony is only one method for organising
sound.
Technicalities
aside, the core issue is that the quest for absolute synaesthetic correlation is
at odds with achieving dialectic tension in the audio-visual relationship. In
synaesthesia, it is not so much sensory interplay that is encouraged,
as a form of submersion and sublimation, an immersive running together of the senses, thereby reducing
the audience’s potential participation in making meaning. Such synaesthetic inclinations have recently been given impetus within contemporary visual music practice, where digital technologies
have opened up myriad new ways of combining sound and image.
If Eisenstein (et al) sought
to upset audio-visual adhesion in narrative film by the use of asynchronism, then
what is proposed is that in visual music to offset literalness and a tendency
towards immersive synaesthesia, another technique,must be deployed, the arbitrary.
The arbitrary does not
denote simple randomness, but rather a changing and reciprocal choreography
between the audio and the visual, which recognises, and even embraces the
potential for adhesion and equation, but that either then foregrounds these
relationships as arbitrary rather than essential, or uses arbitrary elements to
problematise the correlations and hence offset the synaesthetic.
The Three Methods
There are three principle ways
of introducing the arbitrary. Firstly one can allow adhesion to take place but then willfully multiply the number of audio-visual equations.
This process is facilitated by digital mapping, so at its simplest, if say an equation of pitch and colour is created in one part of a work, such that as the frequency rises the hue changes, this can be offset elsewhere, by mapping pitch to
changes in form, or some other visual parameter. By shifting and changing the
mapping, the arbitrary nature of the audio-visual correlation is foregrounded, and
revealed, as the audience find themselves making first one equation and then
another.
The second arbitrary
function involves questioning causality and can be applied to either abstract
or representational imagery, but arguably works best with the latter. In
narrative cinema on-screen action or activity is
typically perceived as having a causal relationship with the sound one hears. A
car pulls up, the engine stops, the door opens and closes, footsteps on the
gravel, etc. Indeed sound without an accompanying visual source is often used
as a way of creating dramatic tension, building to the moment when the two
become united. We hear the sound of footsteps approaching a door before it
opens to reveal who is coming through. In
visual music one can question or upset this causal relationship, such that the
audience asks, is the image or action producing the sound, or is the soundtrack in
some way generating or producing the image? Again digital technology allows us
to do this in various ways, for example by either using the same data stream or algorithms to manipulate both image
and audio, or by scanning the moving image to produce the
soundtrack (in a manner not dissimilar to optical sound in film). So it is the action of the frame, rather than the action in the frame that is the causal link.
The third arbitrary approach is looser and closer to asynchronism, as in
this model, moving images are not synchronised note for note with music
or sounds, but married with syncopated
musical rhythms. Each retains its own channel and identity and the arbitrary element comes in
the open-ended nature of this correspondence, there is no beat-by-beat or 4/4 dynamics,
locking the audio-visual relationship, but rather flashes of momentary adhesion,
possibly taking place at different tempi and at different points within the frame. As such there may be
many possible moments of adhesion or of equation.
This third approach requires
no special digital technology and can be seen in the films of Len Lye and his
use of Cuban dance band rhythms, and Malcolm Le Grice’s Berlin Horse (1970) with its looping imagery counterpointed by by Brian
Eno's piano loops. The choice of image and music is
quite particular however, as though some adhesion and equation will happen if one combines just about any music and moving imagery, without careful selection and
counterpointing, the effect can be to diminish rather than enhance both
audio-visual elements. Whilst digital technology is not required in this third method, it nonetheless can facilitate the making of syncopated moving images and audio.
The three methods are not exclusive and elements from the three approaches may be combined.
To follow - practice based examples
