Martin Stig Andersen 2008
(The
present text will feature in a PhD thesis supervised by professor Denis Smalley,
City University, London)
In 2005 director Jacob Ballinger invited me to
compose the music for his 35mm short film, Rocketman (2008), in which he intended to fully exploit the storytelling
potential of the relationship between sound and image. In order to fulfil his
ambition it soon became clear that it was necessary to dismiss the traditional
dividing line between music and sound design, and instead to integrate the two
elements in a through-composed sound track. Having worked as a composer within
the field of electroacoustic music for several years, using essentially the
same tools as provided in a standard post production studio, I agreed to
venture into the world of film sound, being responsible for the making of the
entire sound track. In the working process I made several observations
regarding audiovisual relations and their potential in relation to structural
articulation in particular. On this basis the concepts elaborated below
emerged. The ideas are demonstrated using classic, well-known feature films as
case studies, and finally applied to examples from Rocketman, from which they originally stem.
Music versus sound
effects - sound designer qua composer?
Given soundÕs ability to create temporal experiences
unique to that of other media, it seems no accident that it has attained an
important role in film. Where a film montage made out of cuts in time and space
causes a chaotic, fractionated temporal experience, sound can bring to the
image an all-embracing temporal trajectory. Conversely, where a film shot
implies linear time, sound can impose the experience of non-linear time. While
applying to sound in general, in traditional film making such temporal
attributes are mostly exploited by means of a film score accompanying the picture. The reasoning behind this, according to
Michel Chion, is that where other sound ... elements ... are obliged to remain clearly defined in their relation to
diegetic space and to a linear and chronological notion of time, music, on the other hand, enjoys the status of being a little
freer of barriers of time and space (Chion 1990:
81). Chion states that the spatiotemporal quality of music especially applies to what he himself calls pit music[1], with reference to the orchestral pit in the opera house (Chion
1990: 81). In other words, pit music is the traditional music accompaniment to
a film, often performed by an orchestra, residing outside the filmÕs world. Accordingly he
implies that screen music[2], that is music arising from a source within the filmÕs world, may only in some cases attain the status of its aforementioned
counterpart. A classic example occurs when music is heard over a car radio,
linearizing an otherwise nonlinear montage of images showing a character travelling
a long distance. The other sound elements, on
the other hand, the structuring of which Chion considers subordinated to the
spatiotemporal information within the filmÕs
world, could, besides dialogue, be regarded simply as sound effects.
In the context of motion picture production, sound effects denote
all sound elements that do not fall into the categories of music and dialogue.
The recording, processing, editing and mixing of sound effects, including Òon
screenÓ, Foley, and background sound effects, is often managed by a sound
designer whose task, according to Walter Murch who
was the first person to be credited the title in recognition of his
contribution to the film Apocalypse Now (Ford
Coppola 1979), is to take care of the overall treatment of sound in film (R.
Thom 1998: 122). Equally, in general, sound design is considered as an artistic field covering all non-compositional elements of film sound.
While
one can arguably make such a distinction between music and sound effects when
considering films featuring traditional film music, incorporating
electroacoustic music in film challenges this idea. For decades any sound
available, be it instrumental or environmental, has been part of the
electroacoustic composerÕs sound palette, and musical properties, such as space
(spatial articulation) have developed and acquired equal importance to pitch
and rhythm. Furthermore, just like the sound designer, the electroacoustic
composer is concerned with sound recording, editing, and mixing, each
representing essential, and often indistinguishable parts of the compositional
process. Thus, in principle, by means of the electroacoustic medium the
composer has the opportunity to include all sounds relating to a given filmÕs
world in his compositional work, thereby potentially exploiting their temporal
forces in relation to image. However, temporal utilization of sound effects in
film is not new but has been practised for decades by innovative sound
designers. As a brief example, consider the Òtiger sceneÓ in Apocalypse Now in which a high-pitched, sustained sound of insects freezes time and
causes the experience of suspense, an effect usually achieved through the use
of music, and by orchestral means. In such examples, if the categorization of
sound elements into film music and sound effects is to retain any meaning, we
would have to redefine the terms according to soundsÕ correspondence with the
image rather than simply their means of production.
An early example of an entirely electroacoustic
soundtrack, which defies the subdivision between film music and sound effects
based on the differentiation between sound sources or qualities, is found in
the 1956 sci-fi movie Forbidden Planet. Here all
sounds, except the most basic Foley sounds such as foot steps, share the same
ÒelectronicÓ origin, timbre and identity. Nonetheless, we perceive some sounds
as sound effects while others as musical, and often, being the most interesting
case, we experience them as having both functions simultaneously. Our
interpretation of the sounds at any given instant relies, here, entirely on the
level of correspondence between sound and image.
Regarding
film in general, we can identify three main components of correspondence
between sound and image, identity, time and space. Identity concerns the
correspondence between the identity of a sound-object associated with a visual
action, and the sound we instinctively expect to be produced by that action.
More specifically it relates to the objectÕs material and intrinsic[3] spatial properties such as texture
and size respectively. The expectation as to how a given visual action will
sound may be founded on both natural and culturally-related perceptual
experiences. Time-correspondence concerns the
timing between the visual action and the associated sound. In contrast to the
real world, in film the occurrence of a sound-object may be dislocated from the
action producing it. Finally, space-correspondence concerns the match between the space implied by auditory and visual
cues respectively, in terms of extrinsic[4] spatial properties such as room
characteristics and size and the proximity of a given source.
In
the context of this writing, the image not only
refers to the actions taking place within the visual field of the spectator,
defined by the screen, but the entire space of the filmÕs world. For example
when we see a character walking, his feet will often have been cut off by the
edge of the camera frame. However, as spectators we still believe that the
person has feet, and expect that they will make sound when he walks. Another
example occurs when an establishing shot showing
two characters in a room chatting cuts to a close-up shot of one of the characters, to give a more detailed vision of facial
expression. Does that mean that the other character has disappeared? Maybe, but
that would need to be revealed to us later. At the instant where we cut to the close-up
shot we maintain an internal image of the entire
space in which two characters are standing next to each other. While the above
concerns the representation of an objective-external reality, the image may
also, by means of point of view shots, especially if moving, represent an
objective-internal reality, bringing the audience inside the body of a
character so to speak.
Identity, time and space each inhabits a
continuum between close and remote
correspondence. In short, close identity
correspondence is the result of coupling sound with
the imaged representation of its course, as, for example, when the sound of
footsteps, whether real or soundalike, accompanies the image of footsteps. If
we replace the sound of footsteps with string pizzicati, provided these remain
synchronized with the image, we experience a remote identity correspondence. Close time-correspondence occurs
when sound and image are synchronized with each other, as in the example above.
If, by contrast, we locate the sound and visual action separate places in time,
yet retaining the association between the two, a remote time-correspondence occurs. When spaces indicated by
image and sound respectively match each other we identify a close
space-correspondence, while if there is a mismatch
between the two, as there would be if we added a hugh amount of reverberation
to the sound of a car driving through the desert, we would experience a remote
space-correspondence. The level of identity-, time-
and space-correspondence may differ from each other at any given instant, and
each may change dynamically over time.
Accounting
for both identity, time and space, generally the closer the correspondence, the
more that dimension contributes towards making the sound appear as originating
from within the image. When, for example, the sound of footsteps 1) reflects
plausibly the sonic identity of the shoes and the material being walked on, 2)
is synchronized with the movement of the character, and 3) matches the space in
which the character is walking, we identify an overall close correspondence
between sound and image. Conversely the more remote the overall correspondence
between sound and image, the more the sounds are pushed outside the filmÕs
world. Pit music is a good example of sound which lacks correspondence with
image in terms of identity, time and space.
In
the above investigation of the endpoints of the overall correspondence
continuum we discovered a link with the two traditionally divided elements of
film sound, that is, at one extreme, sound effects relating to overall close
correspondence, and, at the other, film music, here considered in the guise of
pit music, resembling overall remote correspondence. The concept of
correspondence will not, though, differentiate between sound effects and screen
music. Imagine, for example, a character plays the piano, say a Beethoven
sonata, until suddenly, for some reason connected to the diegesis of the film,
he bangs his head into the keyboard. During the entire sequence we may identify
close correspondence between sound and image in identity, time and space, and
yet experience the sound functioning as both music and sound effects
respectively. In order to categorize the associated sounds as either music or
sound effects we need to ask whether they are organized according to the image
or vice versa. When sounds seemingly follows a filmic
logic, for example the cluster sound determined by
the piano playerÕs head hitting the keyboard, they function as a sound effects,
whereas if the image, as that of the finger movements of the character playing the
piano, follows a musical logic the sound will
serve as screen music. In the field of overall close correspondence, due to the
implicit Òone-to-one relationshipÓ between sound and image, very limited room
is left for ambiguity between sound effects and music to exist. Accordingly,
attempts to transform gradually the role of sound from one to the other often
yield rather deceptive results in which the spectators merely flip between the
two interpretations.
Delving
into the realm of overall remote correspondence we encounter the symbol
correspondence continuum. Although sound may not correspond to image through
identity, time and space altogether, it may do so in a symbolic fashion. By
means of culturally generated codes sound can take part in the moods of the
images, it can tell us when to be alert, and through the imitation of musical
styles or sound qualities, (e.g. analogue distortion) define the (relative)
temporal setting of the images. Moreover, by means of internal references,
sound is able to contextualize images, and hence to create a network of
relations between otherwise separated elements of a film. As with the above
three dimensions of correspondence, symbol correspondence may vary from close to remote.
In
film sound the field of remote correspondence, ultimately allowing for a
symbolic relationship with the image, has largely been the territory of
traditional instrumental music. Being accustomed to the film genre and its
precursors (theatre, opera) we, the audience, are willing to accept the presence
of the accompanying orchestra although lacking correspondence with the image.
This precondition, uniquely allowing for the operation outside the filmÕs
world, has facilitated the promotion of traditional music as the prime temporal
mediator in film. In the following I will elaborate the above ideas leading to
the discussion of the potentials of other sound elements belonging to the filmÕs world, by maneuvering within the middle
ground of the three-dimensional correspondence continua, to transcend the
traditional division between sound effects and film music, and eventually
adopting temporal functions usually achieved through the use of traditional
music.
Identity
correspondence continuum
Regardless of the level of identity
correspondence, when a sound and a visual action occur at the same time the
spectator will believe that there is a correspondence between the two.
According to Chion, who denoted this phenomenon synchresis (a combination of
the words ÒsynchronismÓ and ÒsynthesisÓ), we may use any sound we might desire
for footsteps. While the level of identity correspondence may not influence our
perception of an audiovisual event as integrated, it will nevertheless affect
our interpretation of the event, from being hyper-realistic to unreal. Sound designers
and Foley artists mostly utilize the property of synchresis in terms of
substituting sound sources with others that produce similar sonic identities.
This includes providing actions which in reality make only little or no sound
with a strong sonic identity, based on natural properties of sound. By being
primarily concerned with close identity correspondence, these examples occupy
the upper part of the continuum, and serve to make audiovisual events seem
plausibly realistic. Lowering the level of identity correspondence, audiovisual
events become more ambiguous and reliant on creative, associative activity on
the part of the spectator. Accordingly, remote identity correspondence is often
utilized to create a dreamy atmosphere, and may evoke the sensation that we
hear what a character is hearing through a blurred perception.
A
factor in locating a sound in the identity correspondence continuum is the
number of Materializing Sound Indices (M.S.I.) involved. According to Chion,
Materializing Sound Indices are the details in sound that supply information
about its concrete materiality and production, and which cause us to
"feel" the material conditions of the soundÕs source. The number of
Materializing Sound Indices provided by a sound often depends on the quality of
the sound recording. For example, a high-quality sound recording carries a
higher number of Materializing Sound Indices compared to dull sound recordings
(Chion 1990: 114). A large number of Materializing Sound Indices will often
cause a sound to tip towards one of the extremes of the identity correspondence
continuum, whereas a lack of Materializing Sound Indices will place the sound
in the midfield of the continuum, potentially allowing the listener to
associate the sound with other sources within the image.
Close
identity correspondence may not necessarily rely directly on real-world
experiences. That is to say that one does not need to have experienced a given
audiovisual event in real-world life in order to accept it as true in the movie
theatre. For example, although some of us have never experienced a rocket
launch in the real world, when such an event is presented to us on television
or film we may perceive a close identity correspondence between image and
sound. Rather than real world experiences, here the inference of close identity
correspondence relies on previous experiences from watching rocket launches on
TV and film. This condition is exactly what allows for the utilization of
audiovisual media to establish fake realities in which the spectators may
(temporarily allow themselves to) believe. By consistently coupling a sound
with a visual event or object, the two of which do not cohere in reality,
provided that the audience has not encountered this event or object outside the
movie theatre (e.g. if nonexistent), it is possible to establish a close
identity correspondence between the two. While, in principle, we may pair any
sound with a fictious object we do, however, remain critical as to what we
accept as representing ÒrealityÓ, and in the validation process our real-world
experiences continue to provide the most important tool. Thus, for the
correspondence between image and sound to be close, in this way suggesting that
the object in question is ÒrealÓ, it needs to reflect real-world physics.
Material versus
spatial identity
With regards to identity correspondence two
sub-components can be identified - material and spatial identity. Being equally
concerned with the correspondence between implications made by image and sound
respectively, material identity applies to substance and texture while spatial
identity concerns size, and position in vertical space. To illustrate the
difference between the two, and to show how each may contribute towards making
a fictive object become ÒrealÓ, it is relevant to take a look at the lightsaber
in Star Wars, whose sound was designed by Ben
Burtt. In terms of material identity the lightsaberÕs continuous, humming sound
can be considered to correspond closely with its visual counterpart, firstly
because the image represents stable, continuous energy[5], and secondly because in real life that energy would most likely be
dependent on electricity. The humming sound, which we recognize from electronic
circuits, reflects exactly those two properties. The concept supporting the
materialization of the lightsaber is further enhanced by the occurrence of
electrical zaps or cross-connection sounds when two lightsabers cross. While
visually the crossing is only marked by a flash at the very instant where two
lightsabers cross, the zapping sound is held throughout the entire period of
contact, thereby indicating a continuous friction.
Central
for the spatial identity correspondence between image and sound is the natural
relationship between source size and fundamental pitch. Although the
fundamental pitch of an electronic hum will not tell us anything about the size
of the source the apparatusÕ ability to resonate at that frequency indeed will.
Drawing a parallel with musical instruments, the fundamental pitch of the
lightsaber (90 hz) lies between that of the french horn and the lowest
produceable pitch of the acoustic guitar. Suggesting a magnitude in the
vicinity of these instruments 90 hz seems slightly weighty but somewhat
appropriate for representing the lightsaber with its 4feet length, including
handle and blade.
Besides
size, frequency may also imply something about the sounding objectÕs position
in vertical space. Generally there is a tendency to associate a glissando
moving upwards in terms of frequency with a
literal upward movement. Consequently, an upward glissando will correspond more
closely than a downward glissando to, for example, the image of a rocket
launch. While this may seem natural, in reality, due to the doppler effect[6] relating to space-correspondence[7], a slight downward glissando is more likely to occur. Trevor
Wishart has suggested that the former association of pitch with height relies
on an environmental metaphor: airborne creatures are dependent on a small body
weight with which follows a small sound-producing organ confined to produce
high pitched sound. With this in mind, creatures with deeper voices are
spontaneously assumed to earthbound (Wishart 1996: 191).
Potentials of identity
correspondence
The rules outlined above, derived from our
common perceptual experiences in the real world, are often followed by film
makers in order to bring credibility to fictitious elements or even to turn
reality into hyperrealism. Examples of the former include the association of
sound to many of the fantasy creatures in JacksonÕs Lord of the Rings films, modeled to have a rational physiology, in this way almost
giving them a potential existence. Here, recordings of animals are transposed
down to match the, often enormous, size of the creatures. However, a paradox
relating to spatial identity occurs when it comes to the audiovisual design of
large scale airborne creatures since in such instances it is evidently not
possible to conform to the rules of magnitude and height simultaneously. Here
the latter, spatial metaphor is generally favored at the expense of magnitude
correspondence, resulting in dinosaur-sized eagles vocalizing the same high
pitches as those residing outside the cinema. Presumably, for the same reason,
the winged beasts in Lord of the Rings, ridden
by ringwraiths, only roar with profound voices, matching those of real world
earth bound creatures, when grounded[8]. A brief example of hyper realism is the conventional association
of sound with a rising, or descending helicopter. Working as sound editor on
CarpenterÕs Escape from New York (1981) David
Lewis Yewdall discovered that the upward glissando caused by jet whine warm-ups
(prior to blade rotation) could be utilized to enhance the impression of the
helicopterÕs movement in vertical space. Played forward or backward the sound
supports the associated image of the rising or descending helicopter
respectively (Yewdall 2003: 213). The most extreme example of hyperrealism
achieved through the use of diegetic sound is found in space movies. Although
we know that sound cannot travel in space, for us who have never been there it
remains just as abstract as any fantasy world represented on film. As our
perceptual reference frame remains confined by life on earth, the absence of
sound may provoke a conspicuous and dramatic effect. Thus for fiction films,
sound in space can be considered as a means of protecting perceptual reality,
allowing the audience to forget about the real world and enter the filmÕs
world. Conversely, in documentary, the lack of sound enhances conceptual
reality, bringing awareness of the real world, which the film aims to describe,
into the conscious mind of the audience.
The
identity correspondence continuum is often used to separate natural from
supernatural objects, or events from each other. Just as sound may contribute
to making unnatural objects or creatures become natural, it may turn the
natural into the supernatural. While the former is obtained through close
identity correspondence, as demonstrated above, the latter calls for a more
remote correspondence. To evoke the sensation that a specific object, event or
a character is supernatural, its associated soundÕs spectral register is often
extended downwards, e.g. by transposition, hence making the source size
indicated by sound exceed that of the associated visual object. If context
excludes the existence of supernatural phenomena the cause of such spatial
disorder is likely to be attributed to the blurred perception or the memory of
a given character. The same goes for examples in which material identities,
otherwise corresponding to image, are distorted.
Film
sound may not only represent the exterior surrounding the characters, or their
subjective perception of it, but also the physical interior of a given
character. In other words, we might identify a close correspondence between a
point of view shot and the sound of breathing, a combination situating the
audience inside the head of the character. This category of sound, which Chion
has designated objective-internal sounds, includes
most biosonic phenomena.
The time-correspondence continuum concerns the
level of timing between sound and visual action. Close time-correspondence
denotes tight synchronicity, while remote time-correspondence reflects an
asynchronous relationship between a sound and its associated image. Between the
two extremes lies loose or random synchronicity. As in the case of identity
correspondence, the higher the level of time-correspondence, the more realistic
the result produced. Chion suggests loose synchronization, occupying the
midfield of the time-correspondence continuum, yields a less naturalistic, more
readily poetic effect (Chion 1990: 65). Finally, remote time-correspondence
will ultimately push the sound outside the filmÕs world.
A
certain level of time correspondence is required for the audience to perceive
an interconnection between image and sound. Owing to the property of
synchresis, any sound, regardless of whether it is remote from image in terms
of identity and space, may function as a link to visual action and vice versa,
thereby establishing an apparent mutual motivation between the two. For
instance, where there is close time correspondence between an action and a
musical cue, sound often serves as an ÒalibiÓ or motivation for visual action
or vice versa. Accordingly, time correspondence provides the key with which the
composer or music editor unnoticeably can access the film.
In
the context of this writing, visual action in
film not only refers to actions within shots but also to cuts between shots. In
situations where the causes of the sounds heard are not visualized explicitly,
synchronizing sound and film cuts (e.g. angle cuts) with each other may be
enough to establish the sensation of close time correspondence. For example, in
LynchÕs Eraserhead (1977), where the
ÒindustrialÓ soundtrack offers very limited correspondence with image in terms
of time (within shots) as well as identity and space, it is simultaneous cuts
in sound and picture which bind sound and image together.
Space-correspondence
continuum
In audio post production the means for operating
within the space-correspondence continuum can be considered to include panning,
sound level mixing, filters (e.g. equalization and crosstalk cancelers) and
artificial reverberation. While the former three properties relate to the
position of a given sound source, in that, for example, sound level and
spectral brightness gradually decrease as the sound source moves away from the
listener, the latter relates to the actual space in which the source is
situated. In film the relations between the spatial makeup of different sounds
associated with image, may to some extent contradict reality without
compromising the perceived realism. Sound designer Walter Murch made a relevant
analogy between the property of spatial sound mixing and photography: Just as
the photographer, by focussing his camera, decides what are the important
elements of a given setting, and hence what the spectator should look at, the
sound mixer may, by way of low-pass filtering and gain-reduction, spatially
soften the sounds associated with specific events, thereby throwing them Òout
of focusÓ, guiding the attention of the spectator towards the central action[9] (Murch 1998: 89).
ÒFocussingÓ
through spatial sound mixing may also be utilized as a means for the
compensation of the low spatial resolution of theater sound projection
resulting from the limited amount of loudspeakers available. For example while
in real world life our binaural hearing system allows us to filter out one
voice from a group of voices according to its carrierÕs spatial position, say
in a pub, when such a sound is recorded and projected over a limited number of
loud speakers we merely hear a pool of voices. In order to compensate for such
reduction, the sound levels of all but the one voice determined to be the
object of our attention may be reduced in level.
While
the above examples, because applying to the upper part of the
space-correspondence continuum (i.e. close space-correspondence), generally
articulate what may be called an objective space,
lowering the level of correspondence may bring about the sensation of what can
be designated subjective or portal space. The former occurs when the
manipulation of sound space conveys to the audience the perceptual focus of a
character or a group of characters, for instance on a threatening sound whose
source is not yet visible. In this case all sounds connected with the image are
reduced in level in order to make room for the approaching sound. Another type
of subjective space is achieved by setting the Òsound focusÓ opposite to that
of the camera. For example, the sounds associated with the visually in-focus
actions of a character in a given situation may be softened compared with the
sounds emitting from the surrounding actions being visually out-of-focus, in
this way indicating that the character is distracted from the situation.
Portal
space, on the other hand, is the result of juxtaposing two spaces essentially
remote from each other, one suggested by image and the other by sound. The
effect indicates that a voice, a character, an object or an entire setting
occupies a space distant from the spatial setting of the film, for instance the
memory or imagination of a character. When applied to a voice, a character or
an object within the image, spatial distancing prevents them from ÒphysicallyÓ
entering the image or, in other words, turns them into phantoms. When applied
to an entire setting it will prevent the audience from ÒenteringÓ it, making
them reside in the more realistic spatial setting of the film, ÒwatchingÓ the
sequence in question from a distance. In most cases, the articulation of portal
space relies upon the use of reverberation as it provides the means of creating
unambiguous mismatches between the space implied by sound and that of the image
unless, of course, the actions take place in a supposedly reverberant space.
Portal space is also the means for separating supernatural beings or events
from apparently ÒrealÓ ones. For example, when Obi-Wan Kenobi, after his
physical death in Star Wars IV, reappears,
however slightly nebulously, in Star Wars V his
voice is coated with reverberation clearly contrasting the exterior location in
which he is situated, thereby suggesting that he is a mere ghost the spirit of
which belongs to another realm (i.e. the force).
In
terms of spatial positioning the voice may, more than any other sound element
associated with image, contradict the image without evoking any sense of
spatial remoteness to the conscious mind of (most of) the audience. This is due
to a convention of cinematic presentation allowing, for example, a soft-spoken
dialogue to be heard Òclose-upÓ although the characters speaking are situated
at a considerable distance from the ÒspectatorsÕ point of viewÓ, or to hear a
telephone receiver as if we were holding it next to our own ear in an otherwise
objective space. When conforming to this convention, dialogue evidently ceases
to have any significance as spatial articulator.
Symbol correspondence
continuum
Although sound may not correspond to image in
terms of identity, time and space all together, it may do so in a symbolic way.
Symbol correspondence bases itself upon codes either culturally derived or generated
within, and thus unique to, a given film. Culture-based symbol
correspondence occurs when a sound fits the mood of
an event or a character, a feature traditionally assigned to film music. In this
respect close symbol correspondence has certain
equivalence to ChionÕs concept of empathic music while remote symbol
correspondence could be considered to embrace
anempathic music, that is, music which exhibits
conspicuous indifference from the situation (Chion
1990: 8). Yet, in certain cases, although being indifferent to the imaged
situation, music may contract a symbolic relationship with more abstract
phenomena associated with the image. Consider, for example, the very moment in
Lynch and FrostÕs Twin Peaks (1990-91) in which
the police unveils the face of dead Laura Palmer. While there is nothing
beautiful about this situation (except Laura Palmer looking like a bride) an
accompanying melody of beauty culminates exactly at this point. What is at play
here is a Òhorror music codeÓ which Robert Spande has denoted Òthe sublimeÓ,
the effect of which is a ÒhauntingÓ evocation of a realm unrepresentable,
namely death (Spande filmsound.org).
Unique
to a specific film, internal symbol correspondence may be designed by the composer or sound designer by deliberately
associating a specific, otherwise remote sound with a character, an object or
event. If subsequently this sound is played in combination with an image in
which the associated gestalt is absent we experience a remote symbol
correspondence. Because of the established association, the sound will,
however, convey the knowledge of the gestalt into the imaged situation,
eventually evoking a sensation of its presence. As the sound in question is not
associated directly with the images through identity, time and space, but
rather commenting from outside the filmÕs world, this knowledge is shared with
the audience but kept secret from the characters within the image. It goes
without saying that the concept of internal symbol correspondence equals that
of WagnerÕs leitmotif or BerliozÕ idŽe fix. Returning to Twin Peaks, the
melodic theme which in its first appearance was associated with the death of
Laura Palmer, later accompanies, for example, the love scenes between Donna and
James, who were close friends to Laura, giving the impression that Laura,
although being dead, is aware of their action. In this example the symbol
correspondence between the imaged situation and tender music is close, while
the internal symbol correspondence, due to the absence of Laura, is remote.
Sound effects poaching
on the territory of film music
Relying primarily on overall remote
correspondence in terms of identity, time and space, structural articulation by
means of symbol correspondence has, not surprisingly, become the territory of
film music. However, by somewhat loosening the correspondence with the image,
other sound materials belonging to the filmÕs world (i.e. sound effects) may
eventually come to carry out equal functions.
A
case study in this respect is Forbidden Planet (1956),
a major Hollywood film of its time, in which Louis and Bebe BarronÕs pioneering
work on the sound effects track literally developed into adopting the role of a
complete musical score[10]. Consider, for example, the association of the muffled, pulsing
sound with the invisible monsters of the ID[11], that is, the monsters manifested in the subconscious of scientist
Dr. Morbius. On the first occurrence, the sound is paired with a point-of-view shot,
too high up to represent the sight of a human being, approaching the spaceship
which has just landed on the planet ÒAltairÓ, where Dr. Morbius resides.
Despite lacking apparent correspondence with the image, the abstract, pulsing
sound is interpreted as the internal sound of a beating heart, hence imagining
that the imaged sight is that of an alien creature. In other words, we identify
a close identity correspondence between the equally objective-internal sound
and image. Later, on the creatureÕs second return, leading to the murder of
Chief Quinn, the pulsing sound is synchronized with the appearance of huge
footprints in the dust. Hence, the previous identity correspondence ceases,
while a close time-correspondence between sound and image is established. The
general operation within the mid-ground of the overall correspondence continuum
allows the pulsing sound to place one foot in diegetic space, and the other in
non-diegetic space, or, in other words, to play as sound effects and music at
the same time, projecting a tense mood on the images. In the following scene,
in which the main character, Skipper, assesses a sculpture cast from the
monsterÕs footprint, the pulsing sound appears for a third time, but without
suggesting the actual presence of the monster. Due to the inherent ambiguity of
the pulsing sound, and its lack of correspondence with the image, it now
attains a purely musical quality. Yet, because of its previous association with
the monster, the pulsing sound corresponds symbolically with the image of the
foot-sculpture, thus emphasizing the connection between the sculpture and the
monster. While having identified an instance of internal symbol correspondence,
a culture-based symbol correspondence, between the foreign, unsettling mood of the
sound and the situation, is also present. As the story develops, the pulsing
sound gradually becomes associated symbolically with the image of Dr. Morbius,
revealing to the audience a possible, otherwise unspoken, relation between him
and the monster. Towards the end, in the process of realizing that the monster
of the ID is the product of the subconscious of Dr. MorbiusÕ mind, Skipper
confronts Morbius with the underlying clues. As Morbius apparently does not
approve of SkipperÕs line of thought, Skipper finally cries out: ÒYour mind
refuses to face the conclusion!Ó At the moment in which the word ÒmindÓ is
pronounced, the pulsing sound breaks the silence, and provides the audience
with the decisive clue as to what the unspoken conclusion is. Underlining the
connection between Dr. Morbius and the monster, in the following moment the
sound is, again, associated with the monster approaching from the outside.
Shifting the soundÕs association with Morbius to the monster, internal symbol
correspondence accordingly shifts from remote to close. However, at the end,
the pulsing sound comes to symbolize closely the unity of Morbius and the
monster, manifested from his subconscious, and when finally the two die
together the sound equally ceases.
The
success of the sound track in seamlessly integrating music and sound effects,
providing the key for composers/sound designers to make strong contributions to
the overall structure of the film, may to a large extent be attributed the
almost exclusive use of purely electronically generated sound. Compared to
real-world, recognizable sounds which we tend to associate with specific
objects, abstract electronic sounds can be associated with a variety of
fictitious sources. Moreover, in situations where overall correspondence between
sound and image is inevitably remote, real-world, recognizable sounds tend to
remain associated with the filmÕs world, yielding rather surrealist results,
while abstract electronic sounds are able to exit entirely from the diegetic
space of the film. For instance, in the opening of Forbidden Planet a continuous electronic soundscape linearizes a sequenze of cuts
between shots of the interior and exterior of the spaceship. When the sound is
associated with the exterior shots, showing the spaceship flying by, an audio
visusal correspondence is established between the rising and falling pitches
and the vertical movement of the spaceship (i.e. spatial identity
correspondence), hence making the audience perceive the soundscape as sound
effects. Contrarily, when associated with images of the spaceshipÕs interior,
with which the electronic soundscape lacks overall correspondence, the
soundscape is pushed outside the filmÕs world, attaining a purely musical
function.
Being biased towards exploiting ÒabstractÓ
electronic sound for its associative independence, and hence its ability to
correspond closely with a number of non-real objects, the science fiction genre
provides, more than any other film genre, a foundation for the integration of music
and sound design. Consider, as another brief example, the equivocal nature of
the muffled soundscape, presumably created by composer Edward Artemiev,
permeating TarkovskyÕs Solaris (1972). When
associated with the image of the planet ÒSolarisÓ a close correspondence
between sound and image in terms of identity and space is articulated, causing
us to believe that the sound emits from the planet. Yet, on most occasions the
soundscape does not correspond with the image in terms of identity, time and
space, but in terms of cultural and internal symbolism, expressing the planetÕs
psychological impact on the characters. As in Forbidden Planet, the soundscapeÕs ability to connect with both the diegetic and
non-diegetic space of the film is exploited in the articulation of SolarisÕ overall narrative structure.
What
is, of course, not exclusive to the science fiction or fantasy genre is the
concept of imagination which easily brings one beyond the boundaries of
reality. While the fantasy genre strives to make an imaginary world real,
imagination, being a part of our everyday lives, may well be reflected in films
without compromising an otherwise established ÒrealityÓ. Remarkable
Ònon-fantasyÓ films blurring the line between ÒrealityÓ and imagination by
means of audiovisual design, include LynchÕs The Elephant Man (1980) with sound design by the director himself, Ford CoppolaÕs
aforementioned Apocalypse Now (1979), and also
his earlier The Conversation (1974), both with
sound design by Walter Murch[12]. In the latter we identify a transition from overall, to partially
remote, correspondence between sound and image, or film music gradually
entering the diegetc space of the film. In the first part of the film the
music, performed on solo piano, acts like a traditional score. However, as the
mind of the main character, surveillance expert Harry Caul, from whose point of
view the story is told, increasingly becomes distorted, preventing him to tell
imagination from reality, the pure piano sound equally gets distorted. Towards
the end, as Harry forces himself to tear apart a small figure of the holy
Maria, suspecting a microphone to be hidden in her, the piano undoubtedly
sounds as if being heard through surveillance equipment similar to that used by
Harry himself[13]. Consequently, a pathway between close and remote correspondence,
or between the diegetic and non-diegetic space of the film, or ultimately
between ÒrealityÓ and imagination, is hinted. In the end, having torn his
entire apartment apart without succeeding in locating the hidden microphone,
Harry is sitting on the floor playing the saxophone over the film music, in
this way binding the two worlds together. While initially suggesting that the
music gradually enters the filmÕs world, a related interpretation would be that
Harry, in playing along with the music otherwise residing in a non-diegetic
space, ultimately exits in the real world.
Before
proceeding to examples from Rocketman, it is
worth noticing, firstly, that in the above three films, overlaps between
picture and sound departments took place. While obviously in the case of The
Elephant Man (the director and sound designer being
the same person), in the latter two films it resulted from Murch also working
as picture editor. Secondly, the films, as well as many of the other films
discussed, date back about three decades. On several occasions sound designer
Randy Thom have implied that this period was the bygone time where film makers
were interested in exploring the potentials of sound in film, or, more specifically,
to include sound as an equal collaborator at the pre-production stage as
opposed to merely regarding it as something to be added in the post-production
process where the structure of the film is already in place[14].
All the way up until the final scene of Rocketman, I have striven to retain a level of correspondence between image
and sound or, in other words, to situate sound within the space of the film.
The approach brought with it a number of constraints as to how to cope with specific
structural issues. For example, where linearization of two subsequent shots
representing two different locations was required, rather than utilizing
traditional music preconditioned to reside entirely outside the filmÕs world, I
would need to search for sounds which would fit in both locations within the
space of the film. For instance, on several occasions a noisy, grainy sound,
able to bond with both the image of the sea and that of a rocket, was used to
linearize two shots in which the respective objects were present. The approach
taken also brought with it a number of new possibilities. For example, it was
possible to create gradual transformations between ÒrealityÓ and imagination,
as demonstrated below.
Example of temporal
linearization using sound effects
The most significant example of temporal
linearization by use of sound effects in Rocketman is found in the Òdressing roomÓ scene (example 1). In terms
of cinematography, two contradictory things happen at the same time. On one
hand, the camera is gradually approaching the main character, Sergej, focussing
on his calmness and complete ignorance as to what is going on around him. On
the other hand, an accelerating montage, that
is, a sequence of shots of increasingly shorter length, builds up an atmosphere
of tension. Subsequently, we arrive at a steady close-up shut of Sergej which, however, soon cuts into what may be interpreted as
a psychological shot, in other words, a shot
where the spectators see what the character sees, or imagines. Thus, during the
course of the scene, a transformation between an objective and a subjective
perspective takes place.
In
the process of creating sound for the scene I realized that Foley and sound effects
tended to support the accelerating montage, making the scene even more hectic,
hence preventing viewers from identifying with the calm Sergej, the reason
being that the sounds associated with the vast amount of actions within the
images inevitably also happen to join with the cuts of the montage. In other
words, close time-correspondence not only occurs between the sounds and their
associated actions within the shots but also between the sounds and the cuts
between shots, in this way accentuating the rhythm of the accelerating montage.
Because of their inability to identify with Sergej, the spectators are inclined
to interpret what was otherwise intended to be a psychological shot as an
objective or establishing shot. In other words, rather than considering the
shot showing an empty window as a product of SergejÕs imagination, viewers may
experience being taken to a different time and space.
Therefore,
in order gradually to transport the spectator from watching Sergej Òfrom the
outsideÓ to take on his point of view it became necessary to go against the
accelerating montage, and instead follow the slow and steady zooming-in on
Sergej. The solution was to add an overall steady rhythm to the scene, derived
from SergejÕs space suit which is brought into the image early during the
scene, and which anyway comes to represent his shield from the outside world.
The rhythmic sound of space suitÕs respirator stabilizes the temporal flow of
the images, and minimizes the tension otherwise created by the accelerating
montage. The change from objective to subjective perspective is achieved by
gradually softening Foley sounds which do not follow the rhythm, while
exaggerating by means of saturation those which do. In this way, besides
supporting the established rhythm, the process reveals SergejÕs ignoring of the
noisy surroundings. Furthermore, the brightness of the reverberation
corresponding to the acoustic properties of the setting decreases as SergejÕs
ears become blocked during dressing up. Finally, when arriving at the close-up
shut, the viewers have now ÒenteredÓ Sergej, and believe that the window shot
is a picture from his imagination.
As
a whole, the scene exemplifies a gradual transformation from overall close to
slightly remote correspondence in terms of identity, space, and to some extent,
time - identity as Foleys and sound effects are
gradually exaggerated, space as brightness
decreases, and time as Foley sounds not fitting
into the overall rhythm are gradually removed. While Foleys and sound effects
evolve into a rather musical sequence, and indeed carry out functions usually
achieved by use of traditional music, they can neither be categorized as pit or
screen music. Not as pit music because the
sounds are easily associated with elements within the images, and not as screen
music as the sounds are organized according to a
filmic logic rather than the other way around.
Just as in the dressing room scene, in terms of
space, the entire first part of the Rocketman
concentrates solely on transitions between the objective and subjective, or in other words
between the objective space of the film, and the main character SergejÕs
perception of it. The sense of realness, evoked by the exclusive use of
objective and subjective space, however, becomes challenged in the scene in
which Sergej fixes the satellite (example 2), when
suddenly a huge amount of reverberation is added to sounds whose sources,
according to the image, inhabit an open space. The resulting mismatch between
the enclosed space implied by the sound, and the open space suggested by the
image, evokes the sensation of portal space. Adding to the foreignness of the
scene in question, a slightly bell-like and musical quality emerges as a result
of the reverberation applied to tiny pling-sounds associated with the image of
neon lights continuously switching on. Ultimately, although corresponding
somewhat closely with the image in terms of identity and time, the associated sound
seems to be framing the image rather than emerging from it, hence temporarily
transcending the diegesis of the film. The ambiguous space correspondence
between sound and image challenges viewersÕ ability to distinguish between
fantasy and reality, almost putting them in the place of the main character
Sergej who is completely absorbed in fixing the satellite, neglecting that he
just broke the contact with mission control, and is in great danger.
Examples of symbol
correspondence
Towards the end of the film, after having
repaired the satellite, Sergej finally awakes from his reverie realizing that
his space ship has left without him, that he is running out of air, and is
going to die (example
3). Experimenting with sound effects and Foley sounds for the scene, for
example that of SergejÕs breathing slowing down, indicating that he is running
out of air, I realized that such sounds yielded a rather conceptual and
deceptive result; a sort of Òemotional vacuumÓ where the spectator would merely
witness the physical death of Sergej. For the first time in the film I chose to
associate a sound lacking overall correspondence with the image in terms of
identity, time and space. Yet, the sound, being the most melodic and beautiful
of the entire film, corresponds with image in two symbolic ways. On one hand, a
cultural symbol correspondence is established between the haunting sound and
the un-representable fate of Sergej (cf. Spande). On the other hand, since the
sound is extracted from and thus refers to the sound which was earlier
associated with Sergej indulging himself with fixing the satellite (i.e. neon
lights switching on) an internal symbol correspondence occurs. Hence, the sound
functions as a hint to the audience as to the very reason why Sergej is
captured in space, in this way emphasizing the tragedy.
Another
example of internal symbol correspondence, exclusively based on the use of
sound effects, is the ever returning muffled sound of the ocean associated with
a variety of images. In the opening of the film, where Sergej is standing in
the kitchen watching the sea through the window (example 4), the
sound is associated with the image of the sea. Over it we hear SergejÕs silent,
internal breathing, suggesting a subjective point of view, and the two sounds
combine to represent SergejÕs peace of mind. In the following long section of
the film Sergej is exposed to stressful situations. He is taken to the space
station, and transported to space where he is to fix the satellite. When a
software error causes the mission to fail, and mission control starts to blame
Sergej, he decides to cut the communication with them, and enter the satellite to
fix it in his own way. As he enters (example 5), the
sea-soundscape from the opening of the film returns. Although it is now
associated with the image of the satelliteÕs interior, the reference symbolizes
that Sergej is regaining his peace of mind. Towards the end of the film, having
completed the mission, Sergej finds himself in total peace symbolized by the
final return of the sea-soundscape (example 6). This
time the sound leads SergejÕs thoughts back to the kitchen where we initially
heard it.
Although
the sea-soundscape is associated with different images, in all three occurrences
a level of overall correspondence is retained, hence situating the sound within
the space of the film. In the first instance (Sergej watching the sea through
the window) a close identity correspondence between the noisy sound and the
image of the sea, as well as a close time correspondence between the gradual
amplitude and spectral changes of the sound and the image of the waves are
evident. However, the dull quality of sound, suggesting it is being heard from
a distance, combined with the rather close up shot of the sea, results in a
slightly remote space correspondence. On the second return (entering the
satellite), the dull quality of the sea-soundscape comes to correspond with the
dark, rather undefined space of the images (i.e. close space correspondence),
while identity correspondence, as we discover no sounding objects within the
images, becomes remote. Time correspondence, however, remains close as
amplitude and spectral changes of the sound follow the movement of objects
within the image. An equal overall audiovisual correspondence is evident on the
sea-soundscapeÕs third return (the completion of the satellite). However, this
time the spectator may experience a close identity correspondence between the
image of distant neon lights switching on and the sea-soundscape, as the latter
resembles that which is earlier associated with similar images.
Manoeuvring
within the midfield of overall correspondence in terms of identity, time and
space possibilities opens up ambiguities in audiovisual relations, thereby
permitting internal symbol correspondence to occur. As in the example above,
such potential may be exploited in the articulation of the overall structure of
a film.
Chion, M. (1994): Audio-Vision. New York: Columbia University Press.
Henriksen, F. E. (2002): ÒSpace in
Electroacoustic Music: Composition, Performance and Perception of Musical
SpaceÓ. Unpublished PHD thesis. City University.
Murch, W. (1998): ÒTouch of SilenceÓ. In
Sider, L., Freeman, D., Sider, F. (Ed.) (2003) Soundscape, The School of
Sound Lectures 1998-2001. London: Wallflower Press.
Spande, S.: ÒThe Three Regimes: A Theory
of Film MusicÓ. Unpublished essay.
http://www.uselessindustries.com/robobo/filmmusic.html
Thom, R. (1998): ÒDesigning a Movie for SoundÓ.
In Sider, L., Freeman, D., Sider, F. (Ed.) (2003) Soundscape, The School of
Sound Lectures 1998-2001. London: Wallflower Press.
Wishart, T. (1996): On Sonic Art. S. Emmerson, ed. Harwood Academic Publishers.
Yewdall, D. L. (2003): Practical Art
of Motion Picture Sound, second edition. Elservier
(USA).
Ballinger,
J./Beer, J. (2008): Rocketman
Carpenter,
J. (1981) Escape From New York
Coppola,
F. F. (1979): Apocalypse Now
Coppola,
F. F. (1974): The Conversation
Jackson,
P. (2002): The Lord of the Rings: The Two Towers
Lynch,
D. (1977): Eraserhead
Lynch
and Frost (1990): Twin Peaks
Lucas,
J. (1980): Star Wars: Episode V - The Empire Strikes Back
Tarkovsky,
A. (1972): Solaris
Wilcox,
F. M. (1956): Forbidden Planet
[1] Pit
music is ... music that accompanies the image
from a nondiegetic position, outside the space and time of the action (Chion 1990: 80)
[2] Screen
music is ... music arising from a source
located directly or indirectly in the space and time of the action, even if
this source is a radio or an off screen musician
(Chion 1990: 80)
[3] The
notion of intrinsic space is based on the perception of internal spatial
components inherent in individual sounds and sound events (Henriksen 2002)
[4] The
notion of extrinsic space ... concerns the sound in space (Henriksen 2002)
[5] Within the
fiction of Star Wars, the lightsaberÕs ÓbladeÓ
is made by a tight loop of highly focused energy
[6] When the
sounding rocket moves away from the
listener in continuously increasing speed, the sound wave reaching the listeners
ears is continuously stretched and consequently slowed down forming a downward
glissando.
[7] Because
being concerned with sound in space, in the
context of this writing the doppler effect adhere to the concept of space-correspondence
[8] When airborne
the winged beasts tend to be associated with the ringwraithsÕ crow-like screams
[9] Murch
originally uses the example to illustrate how music by use of spectral
treatment (i.e. equalizing) can be brought into the background
[10] As the score did not make
use of traditional instruments the original screen credit was changed from
Òelectronic musicÓ to Òelectronic tonalitiesÓ in order to comply with the
American Federation of Musicians
[11] The id is a
Òcauldron full of seething excitationsÓ (Freud 1933, p 73) of raw,
unstructured, impulsive energies... (Mitchell and
Black 1995: 20)
[12] In The Conversation Murch is credited Òsound montageÓ and Òsound re-recordistÓ
[13] According to Murch, who
also carried out the music editing work for The Conversation, the distortion
was generated using a synthesizer
[14] See, for example,
ThomÕs articles Designing A Movie For Sound
(1999) Confessions Of An Occasional Sound Designer (1995), More Confessions Of A Sound Designer (undated)