by 
There’s a popular new heavy-metal band on social media, The Frostbite Orckings, whose animated videos are racking up millions of views. Their music is created by an AI model from a startup based in Bingen, Germany, Musical Bits. Founded in late 2021 by Stefan Schmidt, a singer, guitar player, piano player, and composer, who has toured internationally with his metal a-cappella band VAN CANTO while working a day job in Artificial Intelligence. “It was finally the idea to bring two of the things that I love together,” he told me in a conversation last month.
According to Schmidt, Musical Bits’ innovation is based on what they call “Creativity Layers,” which stack the creative parts of composition like arrangement, rhythm, instrumental interplay, harmony, melodies, lyrics, and text, on top of each other, with each layer having its own set of custom built and trained AI models.
“We started with things in the style of hard rock and heavy metal because this is where our domain expertise and that of the team members lies,” said CEO Schmidt of his small team of creative technologists and musicians. “We are also working with external artists that bring domain expertise for other genres or instrumentations, like orchestral arrangements, like EDM stuff, where you cannot just ask one person to do it, but you need the different domain experts from a musician side as well to make a meaningful system.”
Schmidt says the music Musical Bits produces is completely proprietary. “If their model produces something that sounds like another band, they might still get sued for copyright infringement – even if they didn’t train on that. Copyright in music is about how it sounds and the lyrics, not about how the human musician was trained.” Said Dmitry Shapiro, founder and CEO of Youai, and former CTO of MySpace Music. “One could (should) argue that ALL composers have been influenced by others that came before them and their contemporaries, and therefore argue a model trained on publicly available music is no more infringing on copyright than a human musician who has been influenced by listening to the music.”
Microsoft just indemnified its licensed users against any claims their AI has violated anyone’s copyrights. Getty licensed its 34 million image library to Adobe for its Fireflies AI, which integrated into Photoshop. Adobe, too, indemnified its users. While we’ve heard a lot about music models trained on all the world’s music, which may be why both Google and Meta are not releasing their text-to-music generative AI. But artists are out there training their own models, which is what Musical Bits has done.
Stable Diffusion launched a new web-based AI Audio platform, Stable Audio, last week. It, too, was trained with a licensed dataset consisting of over 800,000 audio files containing music, sound effects, and single-instrument stems and text metadata from AudioSparx. Stable Audio starts at $12/month for a subscription that allows users to generate up to 90 seconds of audio that can be used license-free in any medium. The goal of Musical Bits and its business plan is different. Like the rest of the music business, it’s about hits and acts, like Orckings, not underscoring a football montage on YouTube.
Jens De Vos is a European tour manager with vast experience in the music industry. I asked the most basic question: Are the Frostbite Orckings good? Do people really like their music, or is it a gimmick? “The first time I heard that first song, Guardians of Time, I was on tour with a band I worked for. I checked it out with a lot of people around a laptop and to be honest, everyone really loved it. I can still recall it has this great hook. The rest of the tour everyone was humming that melody,” said De Vos. “They could have easily fooled us into believing Frostbite Orckings was a real band.”
“We fed our system with potential output sounds like in our style,” explained Schmidt, “with snare hits and with bass drum hits, with guitar power chords, with single scales, with everything that you need going down to smallest syllables or phonemes of a singer. And then the other thing we trained is how it should be played. So we had a real drummer drumming for 40 hours in a specific style. We had musicians that played chord progressions that are related to a certain emotion or style. And out of all these elements, the system picks first the musical elements, the concepts like chord progressions, arrangement, and how to play drums together with guitars. Then they are painstakingly annotated by the musicians, which is the time-consuming part.”
“What they are doing is very interesting. When sampling first came in, engineers hired musicians to create samples and there was no copyright issue with that,” said legendary music executive and producer Albhy Galuten, who is now Senior Fellow of Technology Initiatives at Intertrust Technologies, a pioneer in digital rights management. “This is similar in that the training set is being created with full permission. The larger question for me is how fans relate to the bands. It is not just about the music; it is also about the personality and the emotional connection to the musicians. I think without that, it will be difficult to move listeners deeply.”
Musical Bits’ The Frostbite Orckings will release the first fully AI-created heavy metal album on December 22nd. Schmidt says it will be promoted just like any other album is promoted. “We do not necessarily want to brand it as ‘AI-generated.’ We think it’s better to create something and then have people surprised by learning how it was made afterward.”
The business plan calls for the company to license components of its software. “So for example, the drum generator would be a product; it would also be a product to generate guitars to drums; it would be a product to generate lyrics, it would be a product to generate a vocal line to lyrics,” Schmidt said. “The complete system where you can create the complete composition currently is what we are using internally and this is the final goal to have this as a product. But it’s not the near-term goal where we can say this is the product in two months, we will take it step by step.” Schmidt envisions a diverse product range, from drum generators to lyric creators and more. “Our ultimate aspiration is a holistic system that births complete compositions and musical worlds. We’re adopting a phased strategy to realize this dream.”
Charlie Fink is the author of the AR-enabled books “Metaverse,” (2017) and “Convergence” (2019). In the early 90s, Fink was EVP & COO of VR pioneer Virtual World Entertainment. He teaches at Chapman University in Orange, CA.
Header image credit: Photo by James Kovin on Unsplash
More from AR Insider…
