Athos Damiani | Sound annotation with Shiny and wavesurfer | RStudio (2020)
We observed a huge improvements of Machine Learning tools but the main effort were to help at post annotated dataset step. We still struggle to build a trusty pipeline to make these annotations. The package wavesurfer brings to R users the ability to annotate audio files with ease and reliability, exploring the friendly user interface of Shiny to make this hard and laborious part of the project more joyful and efficient. A 5-minute presentation in our Lightning Talks series
image: thumbnail.jpg
Transcript#
This transcript was generated automatically and may contain errors.
Hello, everyone. Thank you very much for having me here. I'm going to talk about wavesurfer for audio notation in R. A little bit about myself. I'm currently working on my master's on the theme automatic bird species recognition based on their songs. And to tackle this problem, I chose supervised machine learning, of course. And to do so, I needed label data.
One can split the machine learning project workflow in these three parts. Data extraction, data annotation and modeling. And I think that we are pretty well served in terms of tools in the data extraction and modeling. We have Warbler, which is an outstanding package for bioacoustics. You have all the diverse ecosystem. Keras, TensorFlow, everything. But I had a really hard time labeling my data in R. Annotating. So, wavesurfer was an attempt to bridge this gap.
What is wavesurfer?
What is wavesurfer? It's the JavaScript library for making beautiful audio visualizations for the web. And it also has a rich set of plugins to work with. So, it's pretty cool and active in the community.
What I did was just an HTML widget for this library, which has this pipeable API. And I also implemented the plugins. The most important one which would make me choose this library was the annotator or the region plugin, which allows you to draw out some areas on your audio representation and also label it. So, it's perfect for the task, right? This square, this yellow region is an example of the annotation.
Another cool feature is that you can visualize spectrogram instead of the sound waves, which is important for us in this task. One cool thing is that this representation is not available in the original JS. I tweaked it a little bit in this part. It's cool, but dangerous because I'm not very skilled in the JavaScript coding.
The annotator app
But this is the API. What I would like to highlight now is the annotator app, which provides to anyone, to the researchers to start annotating their own audios right away without knowing Shiny and stuff. This requires just two inputs, the path to the wave files folder and the path to where you want to store the annotations.
And it is what it looks like. I'm going to zoom in some key parts. Here is the main part of the app. It's where you're going to do the actual notation with the point and click, resize, drag and drop and stuff. This is the widget itself.
Here's some metadata, where is the folders and some switches like bigger spectrogram or autosaving. This is the standard controllers of any media player. This table is where you can navigate through your wave files and you can select the file you want to annotate and also tells whether the audio was annotated already or not. The green is the annotated audios.
Here's just a feedback of what you're going to start to use it later on.
And this finally is what I call the region suggester. It's a cool feature that will draw for you automatically regions that it thinks is the event you are looking for. So it's pretty make your life easier and your job quicker.
I would like to highlight the fact that you can put it on the cloud and make this laborious part of the job collaborative. So it can scale your job pretty much. And this is it. Thank you very much. This is a work in progress. So reach me out in GitHub and thank you very much.
I would like to highlight the fact that you can put it on the cloud and make this laborious part of the job collaborative. So it can scale your job pretty much.
