Behind the Scenes Video!
Ideation
The main goal of this project was to create something similar to a virtual concert, as seen in Japan with shows held by Crypton Future Media with Hatsune Miku concerts, and then you have concerts held by Hololive with their new AR addition shows. Having those shows as a base reference, I needed to look into the overall outlook of how these projects and video files get made to be displayed on a large screen while also giving off the illusion of a character performing on a stage.
One of the ways I figured out some of the ways that the character seemed to blend in seamlessly into the stage, with the lights and band, was the fact the 3D video was rent with a black background with a pepper ghost illusion, causing the background and character to blend as seen from the attached image. I plan to accomplish something similar and use the pepper ghost illusion for the showcase. The additions to the concert videos include occasional lyrics appearing on the screen, inviting slight audience interaction, making it a fun video and show to watch, with the character moving around on the stage, too.
The overall development of this idea started with the idea of using metahumans as the characters of the performance. However, since I wanted to stick with the anime idea, I found a slightly similar software called VRoid Studio by Pixiv, a Japanese art platform. This software allowed me to build characters similar to metahumans, which helped form my idea for this project.
Vroid Testing Characters
Some of the different styles I was working through were popular-styled girls. Cool types, with me than attempting to create the gyaru Japanese style, until for a while I landed on three characters, one with a pink fox motif where she was shy but has a powerful voice, then a popular girl with a mint bunny motif who had a strong voice, and then finally a cool more masculine looking girl who had a dark blue cat motif with a deep seductive voice. I kept and worked through these ideas, attaching them to my email when asking for dances. The colours worked well with each other whilst each of them still had individuality. After making different mockup versions of each character, I made base posters of them, helping to make up the idea that I was trying to see, this time, I still did not have the whole idea of how the dance would come out like, but knowing I was following the outlook of a virtual concert.
Music
When it came to putting together the ideas and thoughts of how I wanted to do this project, I decided to start with a song choice. Initially, I wanted to work with someone from the music course, but given some of the time planning for some of the things I wanted to do, I was advised to contact a song that had already been made and then work around that song. So with that idea, I switched between 2 songs that I felt drifted towards, Round & Round by Interlunium or Heart Attack! by HeartBEAT.
Moving between the two whilst playing around in Vroid studio and dancing away in my kitchen many nights, I landed on Round & Round, their fun eurobeat sound. I ended up shortening the song to fit the guidelines of the project. When I chose the music, I spoke to my tutor and was advised to contact the song creator for permission. I emailed Caitlyn Myers, the creator of Interlunium, and I was in their discord to see if there were any answers to similar requests. I was making, and to my surprise, I found that they allowed the creation of projects like this if the correct credits were handled, which is what I will give. With this answer, I moved right to get my hands on a performer for my project, my motion capture performer and choreographer
Motion Capture Studio
Moving on to the motion capture creation, I set up a date and time with Connoll and Ryan, and we got a 4-hour time slot to record multiple takes for me to use. Once we arrived and got everything set up, Ryan got in the ping ball-looking morph suit he got in position; with his choreography style, he took much freestyle, which we used a lot to have fun with when it came to his performance, with some of the takes us adding a few extra different moves as such as when it comes to the line in the song “Triple Crown” I asked to see if adding a crown gesture with his hands could be added, once we did a take with it, it allowed us to have much fun with it. Each take we did got better and better, with some coming across as better than others; once we hit the fifth take, it was our first perfect take after shuffling through different things and Connol fixing some of the balls that were moving out of place and missing recordings, we hit four more perfect takes coming to a total of eight takes which were promptly transferred into a ZIP file and sent off to me. The creation process in that studio had the energy of all of us understanding and having fun with this idea and concept of a project.
Importation & Retargeting
Once I received the motion capture files, I moved on to the next step, exporting the Vroid model and placing it into the 3D software. With that, I followed many online tutorials attempting to figure out the process. My first attempt was to bring the model into Maya and use the motion capture data there. However, after looking up the process, I just needed to move the file from Blender to Maya, which was working, but then, the motion capture data fell into an issue, and I chose to move the project right into Unreal.
My overall original idea was to use Unreal for my project, but I was worried about my lack of experience. After bringing the files in, the problems only got worse. I initially thought the plugin used to bring in the Vroid models was outdated. However, after playing around with it and falling into numerous issues with the retargeting of the animation, all I needed to do was look lower on the plugin page to see an up-to-date version. I reimported all the files to retarget my void model into the animation rig. One problem with that is that no in-built IK retargeters fit with the skeletons of both models, meaning I needed to create my own. After looking through different tutorials on what to do with the retargeting for VROID, I found an Italian tutorial that went into detail on any model. With that slight chance, I followed it, matching up each bone on both skeletons with an IK Rig. Then, moving on to an IK Retargiter and matching up each IK rig with the motion capture data of the Round and Round performances, finally hitting movement and parts actually retargeting. Some issues with the legs not moving caused me to realise minor errors I made; after a few fixes, I finally got it retargeted and matched up and moving.
Camera & Render
After getting the model and animation set up and in the scene, I moved to start building up the scene with the lights, camera and overall setting. The first step I did was follow the overall look I saw with one of the Miku performances I saw, where you could see a slight stage with a complete background, meaning when I built up the scene, I needed to wipe what was already prebuilt, I then proceeded to delete everything on the scene till I got a dark world with a platform. I imported the animation and placed him in the centre of the platform, changing the texture of the platform to a completely black texture. Then, I added a few spotlights to have the character get light bouncing off him, giving off the outlook of a concert stage when I render the scene. Once I got the lighting and staging setup, I moved to the primary camera; since I deleted the old character player, I made a new camera asset and played around with the settings, mainly the FOV of the camera, to make the camera very far back but very zoomed-in causing an effect of the character seeming very much in place no matter how much he moved forwards and backwards. Once that was set up and looked decent, I added a few effects to the camera to give the overall outlook of the scene a lot more virtual and encapsulated that feeling.
After setting up the scene, I needed to move on to the next step, which was the rendering of the project. Following a few tutorials on how to turn my built scene to be fully set up and then rendered, each step was a lot simpler than it looked in the old engine, but once I had everything set up, I changed the FPS to the motion captured rendered 120 FPS I then hit render and hoped for the best. After a moment, I saw that it was fully exporting; once finished, I went to the file with the render files needing to be placed together in After Effects. Then, placing each photo together, I finally got it rendered, and the final render was out of Unreal.
Typography
My final idea for this project was to display the effects and text throughout the song using animated typography. A few words or effects/items around the screen would add to the video. To start with this, I went back into After Effects as I had used it before for movements, but once I started playing the video with the music, It was not performing as well as I was hoping, as it was causing significant issues that I could not time up anything which made me overall change to premiere pro. Moving everything to Premiere made me feel freer since I could line everything up better, but I needed to figure out the after-effects more to understand the work process.
Once I had everything set up in Premiere Pro, I moved everything around, including the audio, to line everything to work with the lyrics and effects I were to add. Running up and down the song, I tried to figure out what parts I wanted the song lyrics to appear around and would make some audience interaction work. I knew I wanted the triple crown lyric to appear at some point since when doing the motion capture; I added the had movement in at the last moment. Once I started, I knew I wanted to add the characters saying the group name Aromatique, so working through bit by bit and figuring out how I could animate the lyrics flying by bit by bit, trying out different styles, as the first attempt, I just had them flying across the screen. However, it worked, as I felt like it belonged in PowerPoint, so I saw how the character’s body moved and chose to follow the overall flow of his hands. That overall flow led me into the whole song, where parts I felt needed extra jazz, I brought in different ways of displaying it. Once I had completed the typography, I moved to render the video for the publication to move on to the preparation for the Showcase, where I could show what I wanted to do from the beginning of this project.
