TDM 30200: Project 6 - Audio Signal Processing Fundamentals

Let’s start by understanding what an audio signal looks like in the time domain and how to work with it.

Let’s start by loading up the libraries we will use for this project and exploring some audio samples.

We have 8 available emotions, but we will primarily focus on using a subset of the emotions: happy, sad, angry, and neutral for this project. This function will load a sample audio file for each emotion we will use for testing purposes. This function uses the library librosa to load in the .wav files. It will return two parameters: the waveform, y, and the samplerate, sr.

Now let’s load in the audio samples we want to analyze:

The number of samples we see for each file differs because each audio file is a slightly different length and each "sample" corresponds to a specific point in time.

But notice the sample rate - the average number of samples taken per second - is a uniform 48000 Hz across the audio files. Depending on the source of the audio, not all audio files may necessarily have the same sample rate. If you run into that you may want to consider resampling your audio files to a standard sample rate.

Now that we have loaded in the audio samples, let’s visualize these signals in the time domain:

We also plotted the Root Mean Square (RMS) of each emotion. This is a measure of the "average" amplitude of the signal, which is useful for comparing the relative loudness and energy of the different emotions.

Notice there are some similarities in the waveform between certain emotions. Why could that be? What are some differences you see between the different waveforms in the time domain?

So far, we have looked at signals in the time domain and frequency domain separately. But what if we want to see how the frequency content changes over time? This is where spectrograms come in.

A spectrogram is a 2D visualization that shows frequency content over time. It’s created by:

Windows are used in the FFT to calculate the transformation to the frequency domain. To create the windows, we define two parameters: window_size (how wide each window is) and hop_size (how many timesteps are skipped between windows). This can be used to create overlapping windows that help capture continuous features.

Let’s use the 'happy' emotion as an example:

To create the windows, we divide our signal into overlapping segments. Each window contains window_size samples, and consecutive windows overlap by window_size - hop_size samples. This ensures we capture temporal changes while maintaining continuity.

Lets create the windows and see some basic information:

Since our hop_size is half of our window_size, we have a 50% overlap, allowing us to see continuous features between windows. Modifying the window size and hop size will change the amount of overlap, the number of segments and the overall resolution of the spectrogram.

To actually see the windows, let’s plot out the signal and highlight a few windows:

The first graph shows us the windows as they are defined within the original signal. Notice how each window captures a different part of the signal. In the second graph, we see the 50% overlap we talked about earlier which ensures we don’t miss important features that might occur at window boundaries.

We have created the actual window segments so now we can apply a window 'operation' to a segment of the signal to look at how applying a window to the signal affects the frequency analysis. There are a bunch of different windows you can choose from but we will look at the Hann window. The Hann window is essentially like putting a smooth fade-in/fade-out on each segment, so the edges do not create artificial noise in our frequency analysis. We can apply the Hann window to one of our segments like so:

Lets take a look at the original segment compared to the Hann window segment:

Notice how the Hann window smooths out the frequency response and reduces artifacts from the sharp edges of the original segment.

We are seeing a lot of the data in this window around ~10k Hz indicating this information is likely not key to speech intelligibility. It could be due certain sharper language sounds (like fricatives/sibilants), noise, audio sampling or a host of other reasons.

We just tried out the Hann window, but like we said, there are different types of windows with different effects on the frequency response which you may want to use depending on the application.

However, instead of creating the windows, applying the window, and then calculating the FFT ourselves, we can let librosa handle this for us in one step using librosa.stft which defaults to applying the Hann window:

Note that you can pass in parameters to define the window size and hop size, but we will use the defaults which are 2048 samples and 512 samples respectively, so there might be a slight difference compared to what we did manually.

With our signal now in the frequency domain, we can use librosa to create spectrograms. We can do this by using librosa.amplitude_to_db to convert the magnitude values to decibels which is standard for visualizing spectrograms and then using librosa.display.specshow to plot the spectrogram:

The color bar on the right shows levels of decibels which correspond to the intensity of a frequency at a given point in time. This creates a nice way for us to analyze which frequencies are apparent at a given time.

Now that we understand how to analyze signals, let’s learn how to modify them using filters. Filtering is crucial for removing noise, enhancing specific frequencies, and preparing signals for analysis.

Filters can be categorized by their frequency response:

Let’s start by just reloading the happy sample again:

To create the different filter types we can use the Butterworth filter from the signal library.

There are a few key parameters that we are going to set in the Butterworth filter:

The low pass filter blocks all frequencies above a certain threshold which we set to 3000 Hz to preserve the strongest speech energy. This typically captures vowels, but you will lose the higher frequency details like the softer consonants.

The high-pass filter cuts off frequencies below a certain frequency which we set to 300 Hz here. Frequencies below 300 Hz are typically background noise from your room, your mic, or other lower rumbles.

The band-pass filter cuts off frequencies outside of a certain range. For this example we are going to only keep between 300 and 5000 Hz to focus within the speech intelligibility range.

Now we can apply these filters to our signal using signal.filtfilt():

Now we can graph these 3 filters and compare the signal against the original:

What do you notice between the different filters compared to the original? Notice the shape of the signal as well as the amplitudes of each filter - some are different. Why do you think that is?

Now let’s perform the FFT on the filtered signals and compare the spectrogram content before and after filtering:

Here we opted to use the built in librosa function to get the frequency bins and FFT values.

Now lets plot the spectrograms:

Similarly, what do you notice between the different filters compared to the original spectrogram? Notice the shape of the spectrogram as well as the intensity of the frequencies.

Now that we understand some fundamentals, let’s extract meaningful features from our audio signals. These features will be crucial for any following analysis or machine learning tasks.

We will extract several types of features that capture different aspects of the audio signal:

Let’s start with statistical features:

We talked about it earlier but RMS helps to measure the loudness and overall energy of a signal. Zero Crossing Rate (ZCR) is defined as the number of times the signal changes sign (crosses zero) per frame. So a higher ZCR means more high-frequency/noisy/fricative content whereas a lower ZCR means more voiced/tonal content. This would typically be used as an auxiliary feature versus a main feature.

The librosa implementation of ZCR returns the ZCR per frame so we can either plot it over time to compare the dynamics between the emotions or we can also aggregate it to get a single value for each emotion like we did here.

Next let’s move on to extracting some spectral features using librosa:

Spectral centroid is the “center of mass” of the spectrum which is the weighted average frequency, where weights are the magnitudes. So a higher centroid means a "brighter"/more high-frequency energy whereas a lower centroid means a "darker"/more low-frequency energy. It could be useful to give us information about emotions and differentiating voiced vs noisy segments of audio.

Spectral rolloff is the frequency below which some percent of the energy is contained (we used 85% here). So a higher rolloff means more high-frequency energy whereas a lower rolloff means more low-frequency energy. It can give a quick summary of how fast the energy decays with frequency and can be paired with the centroid to give a more robust spectral shape description.

Next, we are going to extract one of the more powerful features for audio classification:

Mel-frequency cepstral coefficients (MFCCs) are a very common and powerful primary feature used in audio applications. These capture the spectral characteristics in a very compact representation. The blog post in Towards Data Science has a some good explanations and good graphics to help explain more about MFCCs. We highly recommend taking a look at it.

Now let’s extract these features for each emotion. Fill in the for loop to populate the features for each emotion sample.

Now that we have our features for each emotion we should do some brief analysis. There are a lot of different ways to visualize and compare the features before using them, but for a quick preview, we can create a small formatted table to compare raw values:

Based on this table, do any of the features immediately jump out at you as not useful to include? There should be at least one which can be explained by what we talked about earlier with symmetry.