For Musicians and Songwriters, Streaming Creates Big Data Challenge

Paying musicians and songwriters when their work is played might not seem like a big data problem but not only is it a big one, it’s a growing one, says Trevor Jackson.

He’s the director of IT infrastructure and architecture at the Society of Composers, Authors and Music Publishers of Canada, a non-profit group that distributes royalty payments to artists when songs are played on the radio, TV or, increasingly, through streaming services.

“We’ve been around since 1925. We’ve been a big data company since before ‘big data’ was a word,” Jackson says.

While the organization used to do it all on paper, it’s gone digital over the past 25 years. Still, Jackson says, it’s a struggle to keep up.

“Traditionally it used to be that the radio and television stations, they would send us what we call their ‘cue sheets,’ or their playlists, we would aggregate that data and determine who should be paid what. With the advent of the digital world, with streaming technology, that’s changed dramatically for us,” Jackson says.

Now, instead of receiving a straightforward song list from a station, SOCAN is getting data on streaming plays from providers like YouTube, Pandora and Spotify.

“Every provider is different,” Jackson says. “When they send us the data, it’s not a standard format … we have to transform that data, put it in the format that we need it to be in, then we have to analyze it to determine who gets what.”

That’s a challenge in and of itself – since music is generally a collaborative effort, there are usually multiple people who get payed every time a song is streamed. Three years ago, SOCAN purchased a new storage solution to deal with the data that was coming in but its needs have already grown beyond that, Jackson says.

“Within those three years, the business and the amount of data that we ingested, it grew exponentially,” he says. YouTube alone sends SOCAN billions of lines of data.

And that slowed all his computer systems down.

“With all these new services coming on-board, there was no way we could continue on that same path,” he says.

To solve the storage problem, SOCAN bought an HP HP 3PAR array that uses all-flash memory.

He says the results have been immediate.

“We’ve seen a huge increase in our throughput,” Jackson says. “It’s allowed us to step back and think about how we work as an organization.”

While many organizations have looked to cloud-based solutions, that wasn’t an option for SOCAN.

“There’s a lot of organizations that have made the move to the cloud, we’re not there yet,” he says.

One of the big concerns for SOCAN is the potential for data to be stored in foreign countries that may have different privacy and surveillance regimes.

“Our members said they don’t want that,” Jackson says.

While other organizations might have dynamic storage needs, SOCAN’s only increase.

“We have to retain data for a long time,” Jackson says.

Still, he won’t rule out a move to the could or to other storage options in the future. And while the changes affecting the music industry are creating new challenges for SOCAN, Jackson says he’s not worried about what the future holds.

“We obviously want to be concerned about any changes that could affect the market that we’re in, but we always try to look at things as an opportunity,” Jackson says. “A threat can easily turn into an opportunity as long as you’re well-aware of it in advance and make the right decision.”