Lesson 7 - Audio

Lesson Video:

(Largely based on rbracco's tutorial, big thanks to him for his work on getting this going for us!)

fastai's audio module has been in development for a while by active forum members:

• rbracco
• kevinbird15
• mogwai
• Hiromis
• scart97
• hoftherose
• madhavajay
• Jo

What makes Audio different?

While it is possible to train on raw audio (we simply pass in a 1D tensor of the signal), what is done now is to convert the audio to what is called a spectrogram to train on.

Free Digit Dataset

Essentially the audio version of MNIST, it contains 2,000 recordings from 10 speakers saying each digit 5 times. First, we'll grab the data and use a custom extract function:

tar_extract_at_filename simply extracts at the file name (as the name suggests)

Now we want to grab just the audio files.

We can convert any audio file to a tensor with AudioTensor. Let's try opening a file:

Preparing the dataset

fastai_audio has a AudioConfig class which allows us to prepare different settings for our dataset. Currently it has:

• BasicMelSpectrogram
• BasicMFCC
• BasicSpectrogram
• Voice

We'll be using the Voice module today, as this dataset just contains human voices.

Our configuration will limit options like the frequency range and the sampling rate

We can then make a transform from this configuration to turn raw audio into a workable spectrogram per our settings:

For our example, we'll crop out the original audio file to 1000 ms

Let's build a Pipeline how we'd expect our data to come in

And try visualizing what our newly made data becomes.

First, we'll remove that cropping:

You can see that they're not all the same size here. Let's add that cropping back in:

And now everythign is 128x63

Using the DataBlock API:

• We'll want to use our same transforms we used for the Pipeline
• An appropriate getter
• An appropriate labeller

For our transforms, we'll want the same ones we used before

Our filenames are labelled by the number followed by the name of the individual:

• 4_theo_37.wav
• 2_nicolas_7.wav

And now we can build our DataLoaders

Let's look at a batch

Training

Now that we have our Dataloaders, we need to make a model. We'll make a function that changes a Learner's first layer to accept a 1 channel input (similar to how we did for the Bengali.AI model)

Now we need to grab our number of channels:

Now we can find our learning rate and fit!

Not bad for zero data augmentation! But let's see if augmentation can help us out here!

Data Augmentation

We can use the SpectrogramTransformer class to prepare some transforms for us

Let's take a look at our original settings:

And we'll narrow this down a bit

For our transforms, we'll use:

• RemoveSilence
  • Splits a signal at points of silence more than 2 * pad_ms (default is 20)
• CropSignal
  • Crops a signal by duration and adds padding if needed
• aud2spec
  • Our SpectrogramTransformer with parameters
• MaskTime
  • Wrapper for MaskFre, which applies einsum operations
• MaskFreq
  • SpecAugment Time Masking

Let's look a bit more at the padding CropSignal uses:

There are three different types:

• AudioPadTypes.Zeros: The default, random zeros before and after
• AudioPadType.Repeat: Repeat the signal until proper length (great for coustic scene classification and voice recognition, terrible for speech recognition)
• AudioPadtype.ZerosAfter: This is the default for many other libraries, just pad with zeros until you get the specified length.

Now let's rebuild our DataBlock:

Let's look at some augmented data:

Let's try training again. Also, since we have to keep making an adustment to our model, let's make an audio_learner function similar to cnn_learner:

With the help of some of our data augmentation, we were able to perform a bit higher!

Mel Frequency Cepstral Coefficient (MFCC)

Now let's look at that MFCC option we said earlier. MFCC's are a "linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency" - Wikipedia. But what does that mean?

Let's try it out!

There's a shortcut for replacing the item transforms in a DataBlock:

Now let's build our learner and train again!

Now we can begin to see why choosing your augmentation is important!

MFCC + Delta:

The last transform we'll discuss is the Delta transform:

Local estimate of the derivative of the input data along the selected axis.

This allows multiple-channeled inputs from one signal

Let's try training one more time:

Let's try fitting for a few more: