Why automated transcription just doesn't cut it for video research

Dave Carruthers - 02/03/2016

Over the past decade the advances in Artificial Intelligence and machine learning have been nothing short of remarkable. From self-driving cars from Google and Tesla to the AI powerhouse that is IBM’s Watson, it genuinely is an exciting time to be alive - especially if you’re a bit of geek like me. I personally can’t wait to see how this technology evolves over the next 5 to 10 years.

That said, as remarkable as the advances are in cognitive computing, some of the most basic tasks such as transcription are still left wanting. Computers just aren’t there yet.  They are close, but sometimes close just isn’t good enough.

Over the past three years the team at Voxpopme has been looking closely at machine transcription. It has been clear since our launch in 2013 that accurate transcription was an essential part of any video platform looking to make video scalable and easy to analyse, both for insight and customer feedback.

When building our MVP we reviewed many speech to text engines, however they all fell short of our expectations. With individual training speech to text accuracy can be improved to upwards of 90%, but we needed a solution that could take a short audio clip - of up to 60 seconds and from a completely new user - and achieve consistent 97%+ accuracy levels. Most of the vendors we reviewed made bold claims, but all delivered an average accuracy below 80% with some as low as 50%.

So Voxpopme decided that there needed to be a different approach. We needed something that was fast, accurate and scalable. We knew crowd sourcing solutions were on the rise and that they provided API driven access to large amounts of human resource, providing an artificial ‘artificial intelligence’.

To illustrate this accuracy issue, let’s take a look at the results below. We took 2 leading solutions in machine transcription, IBM's Watson and Microsoft’s Bing Speech API. We then looked at multiple videos and languages to test the accuracy - below we will show 2 examples from the UK and US.

UK Transcription

 
 

Watson

Accuracy - 77.5%

I think if they do well is that they allow you to buy and sell things the cardigans the shops anymore that quite difficult to go and St countries and you know if you've got textbooks eating teasing walks to finish a degree that is a good place to sell them and get a good buying textbooks you never need new textbooks rarely and is a good place to buy them from as well. I think my issue with eBay and why <missing> don't we do use any more used to buy shoes and clothes of that sometimes but my issue is that it's very difficult to tell if something is authentic or not let us know eBay's fall in them a love like Scotty salads out that it has definitely improved and over the years that I've been cloned before I spent a hundred pounds.<missing> To be official but the promise that it was. And does <missing> nothing<missing> eBay can do about that <missing> pretty <missing> frustrating size like sixteen a break. And now twenty four break so yeah that would be the only thing I would like to see improved is there. In a way that proof of the fish out at sea.

 

Bing

Accuracy - 68.8%

I think way they do well is that they allow you to buy and sell things <missing> he can't He get his shots anymore <missing> That quite difficult to get into that country and you know if you<missing> got textbook She didn't even will be finished a great finish<missing> A good place to sell them and it gonna be buying textbooks You have any <missing> new text books ready like a good place to buy them from is well I think my issue with eBay wanna Take me to eat anymore He's to buy shoes and clothes off at some time with my issue is that certificate kalasam think it'll fenticonazole but it's not a baseball in the Lovelites County salad out that I would have definitely improved and they have the gas but I've been caught before I<missing>spent 100 pound of something that turned out not to be official But the promise that it was on those<missing> missing a big deal but that's pretty frustrating coz I was like16 break and. <missing>


Voxpopme

 Accuracy - 99%

I think what eBay do well is that they allow you to buy and sell things that you can't really get in the shops anymore or they're quite difficult to get in certain countries and, you know if you've got textbooks that you don't use anymore because you've finished your degree, then it's a good place to sell them. And again, if you're buying textbooks, you never need new textbooks really, and it's a good place to buy them from as well. I think my issue with eBay and why I don't really use it anymore - I used to buy shoes and clothes off their sometimes - but my issue is that it's very difficult to tell if something is authentic or not. I know that's not eBay's fault and there are a lot of scammy sellers out there and it has definitely improved over the years, but I have been conned before. I've spent 100 pounds on something that turned out not to be official but the promise that it was and there was nothing that eBay could do about that so it was really frustrating. So I was 16 and broke and now I'm 24 and broke. So yeah, that would be the only thing I'd like to see improved is the proof of officiality.

US Transcription
 
 
 

Watson

Accuracy - 80.9%

The best thing I've ever had and group on objective thing I think is probably one of the ones that I bought is a massage probably massages where like you got these fifty percent off and of the upgrade part was a great massage places I was a. Very happy with it however the best group on I was invited to those offered was a I paid my friend paid forty Bucks. For group on so it was two people. At a dinner and we ended up like spending it was he costed over a hundred twenty Bucks. And we only pay like forty so I that was a great deal.

 

Bing

Accuracy - 82.7%

The best deal I've ever had and Groupon Um I'm trying to think I think is public well The ones that I bought is a massage probably massages would like to get at least 50% off end of the upgrade Parlez a great massage places I was a very happy with it However the best Groupon I was invited to his offer was a yeah I paid my friend paid 40 Bucks um for group Ancylus 2 people um at the dinner and we ended up sex penny It was it cost is over 120 bucks and we only pay like 40 so that was a great deal.

 

Voxpopme 

Accuracy - 99%

The best deal I've ever had on Groupon, I'm trying to think it's probably. Well the ones that I bought is a massage, probably massages where you get at least 50 percent off and they have great parlors, great massage places. I was very happy with it. However the best Groupon I was invited to, I was offered, was, I paid, my friend paid 40 bucks for a Groupon. So it was two people at a dinner and we ended up spending-it costed over 120 bucks, and we only paid like 40. So that was a great deal.

 

Our Conclusion

As you can see, the currently available automated machine transcription is just not accurate enough to be considered an acceptable quality for the types of market research and insight projects we conduct - and that’s before we even introduce things like strong regional accents and background noise.

At Voxpopme we strongly believe that it is far better to provide a very accurate transcription 10 minutes after a video is uploaded, rather than a transcription at below 80% accuracy in 1 minute.

For those who might question the scalability of this solution, using the crowd is also extremely scalable. There are currently close to 10m workers within this ecosystem capable of processing tens of thousands of transcriptions per hour, whilst delivering high levels of accuracy.

We will continue to monitor advances in machine transcription technology closely to find a solution that can return almost instant transcription at 97%+ accuracy, but until then we will definitely continue to leverage the power of the crowd to deliver accurate, fast and scalable video transcriptions to our clients.

Get in touch with us to see for yourself how our crowd sourced human transcriptions deliver better results.

Contact Us

Subscribe by email