Sound Transformation - Moon Ha

# What Is Sound Transformation? **Sound transformation** is the process of modifying an existing sound to change some of its characteristics. ```text Existing Sound ↓ Transformation ↓ Modified Sound ``` The original sound remains the starting material. Examples: - Change its pitch - Change its duration - Change its spectrum - Change its amplitude - Change its spatial location - Break it into small pieces - Combine it with another sound # Sound Generation vs. Sound Transformation These are related but different concepts. ## Sound Generation The computer creates a sound from a model or algorithm. ```text Parameters ↓ Synthesis Algorithm ↓ New Sound ``` Examples: - Additive synthesis - FM synthesis - Wavetable synthesis - Physical modeling - Subtractive synthesis ## Sound Transformation The computer starts with an existing sound. ```text Existing Sound ↓ DSP / Transformation ↓ Modified Sound ``` Examples: - Filtering - Pitch shifting - Time stretching - Granulation - Spectral processing - Convolution ``` NOTE: Synthesis creates sound; transformation modifies existing sound. ``` However, the boundary is not always clear. A transformation can become so extreme that the result sounds like an entirely new sound. # Historical Background Sound transformation existed before digital computers. ## Tape Music Composers working with magnetic tape could physically manipulate recorded sounds. Basic techniques included: - Cutting - Splicing - Reversing - Looping - Changing playback speed - Filtering - Layering ```text Recorded Sound ↓ Magnetic Tape ↓ Cut / Reverse / Speed Change / Loop ↓ Transformed Sound ``` # Pierre Schaeffer and Sound Transformation **Pierre Schaeffer** and the tradition of **musique concrète** are important to the history of sound transformation. Instead of beginning with traditional musical notation, composers could begin with **recorded sounds**. ```text Traditional Composition Notes ↓ Performance ↓ Recording Musique Concrète Recorded Sound ↓ Selection ↓ Transformation ↓ Composition ``` **The sound itself becomes the compositional material.** # What Can We Transform? A sound can be understood through several dimensions. ```text SOUND │ ┌────────────────┼────────────────┐ ↓ ↓ ↓ TIME SPECTRUM DYNAMICS │ │ │ ↓ ↓ ↓ Pitch Timbre Loudness │ ↓ SPACE ``` Major transformation dimensions: |Dimension|Examples| |---|---| |Time|Stretch, compress, reverse| |Pitch|Transpose, pitch shift| |Spectrum|Filter, EQ, spectral shift| |Dynamics|Compress, expand, gate| |Timbre|Distort, morph, cross-synthesize| |Space|Pan, rotate, spatialize| # Time-Domain Transformation A digital sound can be represented as a sequence of samples: x[n] where: - x = audio signal - n = sample index The simplest transformations operate directly on these samples. ## Amplitude Scaling We can multiply every sample by a constant: y[n] = Ax[n] where: - A > 1 → louder - 0 < A < 1 → quieter ```text Input ↓ Multiply amplitude ↓ Output ``` This is one of the simplest forms of sound transformation. # Time Reversal A sound can be played backwards. ```text Original A → B → C → D → E Reversed E → D → C → B → A ``` Mathematically: y[n] = x[N−n] where N represents the length of the sound. What happens to the **attack and decay**? For example: ```text Original: Attack → Sustain → Decay Reversed: Decay → Sustain → Attack ``` This can dramatically change the perceived identity of a sound. # Resampling One of the simplest ways to change pitch is to change playback speed. **Resampling changes both pitch and duration.** This becomes a problem when we want to change only one of them. ```text Playback faster ↓ Duration decreases ↓ Pitch increases ``` and: ```text Playback slower ↓ Duration increases ↓ Pitch decreases ``` # Pitch Shifting Suppose we want: ```text Pitch ↑ Duration = same ``` or: ```text Pitch = same Duration ↑ ``` Simple resampling cannot do this. More advanced algorithms are required. Common approaches include: - Time-domain methods - Phase vocoder - Granular processing - Spectral methods # Time Stretching **Time stretching** changes the duration of a sound without intentionally changing its pitch. But time stretching can produce artifacts. Common artifacts include: - Smearing - Metallic sounds - Phasiness - Transient distortion - Loss of rhythmic clarity # Pitch Shift vs. Time Stretch This distinction is fundamental. |Process|Pitch|Duration| |---|---|---| |Resampling faster|↑|↓| |Resampling slower|↓|↑| |Pitch shifting|↑ / ↓|Approximately same| |Time stretching|Approximately same|↑ / ↓| **How can a computer change pitch without changing duration?** This leads us to frequency-domain processing. # Frequency-Domain Representation A sound can be represented in two important ways. ## Time domain Shows amplitude changing over time. ```text Amplitude │ │ /\ /\ │ / \ / \ │_/ \__/ \____ └────────────────────→ Time ``` ## Frequency domain Shows the frequency components contained in the sound. ```text Amplitude │ │ │ │ │ │ │ │ │ │ │ └────────────────────→ Frequency Low Mid High ``` The frequency-domain representation tells us about the **spectral content** of the sound. # Fourier Transform The Fourier transform allows us to analyze a signal in terms of frequency components. Conceptually: x(t) → X(f) ```text Sound waveform ↓ Fourier Transform ↓ Frequency components ``` For digital audio, the **Fast Fourier Transform (FFT)** is commonly used. X[k] = FFT{x[n]} The FFT is extremely important in computer music. # Why Is FFT Important? Once we have a frequency representation, we can manipulate individual frequency components. ```text Audio ↓ FFT ↓ Frequency components ↓ Modify ↓ IFFT ↓ Audio ``` This makes possible: - Spectral filtering - Spectral freezing - Spectral shifting - Spectral morphing - Time stretching - Pitch shifting --- # Filtering A filter changes the balance of frequencies in a sound. Low-Pass Filter: Allows lower frequencies to pass. High-Pass Filter: Allows higher frequencies to pass. Band-Pass Filter: Allows a selected frequency range to pass. # Filter Parameters Important parameters include: ### Cutoff Frequency The frequency at which filtering begins to take effect. ### Resonance Emphasizes frequencies around the cutoff. ### Slope Determines how strongly frequencies outside the pass region are attenuated. Examples: - 6 dB/octave - 12 dB/octave - 24 dB/octave - 48 dB/octave ### Q Controls the sharpness of a filter's frequency response. # Spectral Transformation Instead of thinking of a sound as a waveform, we can think of it as a collection of frequency components. ```text Sound ↓ Spectral Analysis ↓ Frequency Components ↓ Modify ↓ Reconstruct ↓ New Sound ``` Possible transformations: - Remove frequencies - Emphasize frequencies - Shift frequencies - Freeze frequencies - Randomize frequencies - Interpolate between spectra # Phase Vocoder The **phase vocoder** is one of the most important techniques for advanced sound transformation. Basic process: ```text Audio ↓ Windowing ↓ FFT ↓ Magnitude + Phase ↓ Modify ↓ IFFT ↓ Overlap-Add ↓ Audio ``` The phase vocoder allows us to manipulate the temporal evolution of spectral components. Major applications: - Time stretching - Pitch shifting - Spectral processing # Why Does the Phase Vocoder Produce Artifacts? Phase-vocoder processing is not perfect. **An algorithm's artifacts reveal something about how the algorithm represents sound.** Possible artifacts: - Spectral smearing - Loss of transients - Metallic character - Phasiness - Reduced rhythmic clarity # Granular Processing Granular processing treats sound as a collection of very short fragments called **grains**. A grain may typically last from a few milliseconds to hundreds of milliseconds, depending on the technique. ```text Original Sound ──────────────────────────── ↓ | | | | | | | | | | | | | | grains ↓ Rearrange Overlap Repeat Stretch Transpose Randomize ↓ Transformed Sound ``` # Granular Transformation A recorded sound can be broken into grains and reorganized. Parameters include: - Grain duration - Grain density - Grain position - Grain pitch - Grain amplitude - Grain envelope - Grain overlap - Randomization - Spatial position # Granular Synthesis vs. Granular Transformation These terms are related but not identical. ### Granular synthesis Creates a sound using grains. ```text Grains ↓ Synthesis ↓ Sound ``` ### Granular transformation Starts with an existing sound. ```text Existing Sound ↓ Grains ↓ Reorganization ↓ Transformed Sound ``` The boundary between synthesis and transformation can become blurred. # Convolution Convolution is another powerful sound transformation technique. Mathematically: y[n] = x[n] ∗ h[n] where: - x[n] = input sound - h[n] = impulse response - y[n] = output # Impulse Response An **impulse response** describes how a system responds to a short impulse. For example: ```text Impulse ↓ Concert Hall ↓ Reflections + Reverberation ↓ Impulse Response ``` If we convolve another sound with that impulse response: ```text Dry Sound + Impulse Response ↓ Convolution ↓ Sound with Hall Characteristics ``` # Creative Convolution Convolution does not have to be used only for realistic reverb. We can use unusual impulse responses as creative material. ```text Voice + Piano impulse response ↓ Convolution ↓ Hybrid / transformed sound ``` Other possibilities: - Percussion + resonant object - Voice + room - Synthesizer + instrument - Environmental sound + mechanical response # Cross-Synthesis Cross-synthesis combines characteristics of two sounds. ```text Sound A + Sound B ↓ Cross-Synthesis ↓ Hybrid Sound ``` For example: ```text Voice + Synthesizer ↓ Hybrid ``` The voice can provide information about its spectral characteristics while the synthesizer provides the sound source. # Vocoder A vocoder is a classic example of cross-synthesis. ```text VOICE Modulator ↓ Spectral Analysis ↓ Spectral Envelope ↓ Apply to Carrier ↑ SYNTHESIZER Carrier ↓ OUTPUT ``` The result can sound like: - Robotic speech - Talking synthesizers - Synthetic vocals # Nonlinear Sound Transformation So far, many transformations have been approximately **linear**. Nonlinear processing behaves differently. A simple model is: y[n] = f(x[n]) where f is a nonlinear function. Examples: - Distortion - Clipping - Waveshaping - Wavefolding # Waveshaping Waveshaping changes the shape of the waveform. ```text Input waveform /\ / \ ___/ \___ ↓ Nonlinear function ↓ New waveform __ __/ \__ ``` Changing the waveform shape changes its harmonic content. Therefore **waveform transformation can produce timbral transformation.** # Distortion and Harmonics A nonlinear transformation can create frequency components that were not present in the original sound. ```text Original Spectrum │ │ │ │ │ │ │ │ │ └──────────── Nonlinear Processing │ │ │ │ │ │ │ │ │ │ │ │ │ └──────────── ``` New harmonics can appear. This is why distortion changes timbre. --- # Ring Modulation Ring modulation is another important transformation. A simplified equation is: y(t) = x(t)cos(2πfct) where: - x(t) = input signal - f_c = carrier frequency It produces sum and difference frequency components. Possible results: - Metallic sounds - Bell-like sounds - Inharmonic timbres - Robotic sounds # Spectral Morphing We can transform one sound gradually into another. ```text Sound A ↓ Spectral Analysis ↓ MORPH ↑ Spectral Analysis ↑ Sound B ↓ Hybrid Sound ``` For example: ```text Piano ↓ ↓ ↓ Hybrid ↓ ↓ Voice ``` Possible parameters to interpolate: - Magnitude - Frequency - Spectral envelope - Phase # Spectral Freezing Another interesting transformation is **spectral freezing**. A short moment of a sound is analyzed and its spectral information is held. The result can produce: - Sustained tones - Static textures - Drone-like sounds - Spectral clouds A transient sound can become a sustained texture. # Transformation and Sound Identity How much can we transform a sound before we no longer recognize it? What makes a sound recognizable? Possible characteristics include: - Pitch - Spectral envelope - Harmonic structure - Attack - Decay - Temporal pattern - Rhythm - Dynamics - Resonance - Noise characteristics Different transformations preserve different aspects. # Transformation as Deconstruction We can think of sound transformation as taking a sound apart. ```text SOUND OBJECT ↓ ┌────────────┼────────────┐ ↓ ↓ ↓ TIME SPECTRUM DYNAMICS ↓ ↓ ↓ Transform Transform Transform └────────────┼────────────┘ ↓ Reconstruction ↓ NEW SOUND OBJECT ``` The computer allows us to manipulate these dimensions independently—or approximately independently. # A Continuum of Transformation Sound transformation can range from subtle to extreme. ```text Subtle ↓ EQ ↓ Compression ↓ Pitch Shift ↓ Time Stretch ↓ Granulation ↓ Spectral Processing ↓ Cross-Synthesis ↓ Extreme Transformation ↓ New / Unrecognizable Sound ``` The interesting area is often between: ```text Recognizable ↕ Ambiguous ↕ Unrecognizable ``` # Transformation as Composition Sound transformation is not merely a technical process. It can become a compositional method. Instead of asking: > "What instrument should I use?" we can ask: > "What transformation should I apply to this sound?" ```text Sound Object ↓ Transformation ↓ Transformation ↓ Transformation ↓ Transformation ↓ Composition ``` A complete composition can be built from one original sound. # Example: One Sound → Entire Composition Start with: ```text Recording of a Glass ``` Transform it: ```text Glass ↓ Reverse ↓ Pitch Shift ↓ Time Stretch ↓ Granulation ↓ Filtering ↓ Convolution ↓ Spectral Freeze ↓ Layering ``` The final composition may contain: - Percussive sounds - Drones - Textures - Rhythmic patterns - Bass-like sounds - High-frequency textures All derived from one sound object. # Practical Workflow A general computer-music transformation workflow: ```text Record / Import Sound ↓ Analyze ↓ Choose Parameter ↓ Apply Transformation ↓ Listen ↓ Evaluate ↓ Modify Parameters ↓ Compare ↓ Compose ```