Creating Cinematics with Unreal Engine 3

Overview

This document will serve as a starting point for identifying the steps in creating cinematics using the Unreal Engine content pipeline.

Real-time vs. Pre-rendered

Matinee was created for the purpose of creating real-time cinematics during gameplay. Real time cinematics make use of loaded game assets and manipulate them live to create a scene. When creating real-time cinematics, there must be a balance in the need to push the limits without maxing out resources of the engine. Matinee can also be used to create pre-rendered cinematics so it's a matter of deciding what works best for your situaion - a pre-rendered scene or a real-time scene.

For pre-rendered cinematics, the scene may require higher-res characters, effects, lighting, and set design. Even still, the budget constraints of the resources used by the engine must be kept in mind. In addition to the in-engine tools, pre-rendered cinematics need video editing software and in some cases, additional recording equipment.

As far as the pipeline for creating scenes, both are done the same; but ultimately the resource constraints will drive the need for pre-rendered or real-time scenes.

Unreal pipeline

Be aware that the cinematics pipeline in Unreal, it might be easier to do scene-by-scene, rather than, meaning that each file will be a scene of multiple camera shots, rather than one file per camera shot. There may be some situations where this is needed; but especially in dialog-based scenes, one file would suffice.

In Unreal, but especially when creating real-time cinematics, the pipeline revolves around a scene. So, for a typical 20-shot pre-rendered scene, you'd have to do a scene for each shot (or an Unreal Project for each shot) in a 3D suite such as Maya. In Unreal, you take a whole scene and do camera cuts and such using the cinematics tools in Matinee.

To summarize: in real-time cinematics in Unreal, you cut through cameras while the Actors are moving in real-time, and certain things are being handled in the background (cameras, etc.).

Other considerations

Here are some other things to keep in mind when building cinematic scenes in Unreal:

• Scene Complexity. Factor in all of the necessary props and meshes needed for the scene.
• Environment. Can you contain all of the shots in one level scene; or do you need more?
• Number of characters. If you have 5-8, the scene can be done in real-time; but for crowds, pre-rendered should be used.
• Interaction with the environment. Some things can be done in real-time; but in many cases, a lot may have to be faked. If the scene must be more realistic, opt for pre-rendered scenes.
• Length of scene.
• Others:
  • Lighting Complexity.
  • Effects. Particles - size and detail.
  • Character complexity. In terms of the details, the more detail, the more resources.

Need to be smart with resources in real-time, otherwise, consider pre-rendered.

Preparation

Scenes are often initially designed with outlines, scripts, and storyboards.

From there, content needs can be determined:

• characters
  • consider re-using to avoid creating just for scenes that may never be in the game; how important is it? is it worth it?
• animations
  • anything that isn't part of the usual gameplay set
  • motion capture data and any additional keyframe data
  • get what you need defined to avoid re-shooting, while avoiding any excess data that might never be used
• audio
  • get dialog: voice overs; try to make any edits early on
  • review script; "radio play" - during mo-cap: have characters act based on voice-acted script reading
  • also consider "performance capture" (vs. radio play): capture everything in one shot (voice script, etc.)
• effects
• props

From there, decisions can be made early before full production. Any mistakes made early on are much cheaper during this time. The goal is to avoid excess voice over or mo-cap data, while preventing the need to re-capture or re-shoot.

Pre-visualizations (animatics, etc.) can be done in-engine by moving Skeletal Meshes around in T-poses, in time with dialog. This helps to get a feel of how the scene will be.

Assets

Cinematic scenes are built using assets from animations and audio, as well as building levels for the sets, adding meshes, effects, and lighting.

Engine tools

The following tools in the Unreal Editor suite are primarily used:

• Matinee for keyframing. See Matinee User Guide.
• Kismet for scripting and triggering Matinee scenes. See Kismet User Guide.
• Sound Cues for Music and sounds. See SoundCue Editor User Guide.

Scenes

Scene Layout and Pallettes

Pallettes are text files that contain pre-made Matinee data that just need to be hooked up. Essentially, every object in Kismet is just code and that code can be copied as text. This saves time setting up scenes that use similar setups.

To create a pallette:

1. click on active object, Ctrl-C to copy to clipboard
2. paste as text in an empty text file (see Begin Object)
3. create text documents for all these
   • copy what you need as text and in Kismet use Paste here...

Example:

1. Create a new Matine, which results in a blank scene.
2. Set length to be 1:00 or whatever stand length you would like your palette to be.
3. Add director track, some camera tracks, character skeletal mesh tracks, prop groups, particle effect tracks, blank sound group with empty sound files, lighting, etc. You can organize these groups into subfolders for organization.
4. Data is in place, saved as a palette.
5. Copy the text data, place in Kismet, and the base scene configuration is in place.
6. Save a lot of time using this process to setup your cinematic scenes vs. having to build it from scratch everytime.

Here is an example of a Matinee scene palette: Big_Matinee_Pallette.txt Copy all the code in this file and then paste it into Kismet. A new subproject should appear. Open it. You'll find a new Matinee with some supporting cinematic Kismet functions added in. Open the Matinee object and see the what groups and tracks have already been created for it. Take this and use it! It will save some prep time!

It can also be useful to make Character Pallettes. If for a given set of scenes, there are usually four characters in a scene, then you can set up a pallette to simplify the process in setting them up for each scene. In the empty Matinee you would create them and line them up how you want. Make sure all flags are set – make sure to enable the flag for update SkeletalMesh when not rendered to ensure all animations are drawn when not on camera, physics for movement to interpolate to ensure movement, assigned physics asset, etc. – to be sure the base rules for characters are configured for initial setup. Guns can also be attached, so that they don't have to be attached later. Once set up, you can select all characters at once, copy, paste into text and save as a character palette

Once the Matinee is brought in, as well as the characters, select the characters; and in Kismet, right-click for creating variables. Create all variable hook-ups for the characters, which can then be plugged into the Matinee. When the Matinee is opened, all characters are hooked up for applying animations and making them move.

When doing palettes, it's also a good idea to have one for cameras – either separate, or with the group of characters.

Keep in mind that putting characters together in a palette is best done side-by-side for weapon attachment. You can also place them on top of each other, because all mo-cap data has a (0,0) reference point (zeroed out in the world); it's then translated from that zero-point. If you put in all the animations while there, they will all move accordingly. This avoids repositioning from the zero-point. It's important to realize that if the characters are moved from the original point, you will have to do more work to tweak everything in relation to how it was originally mo-capped. This is especially useful for dialog scenes, to save time. Of course, other times, they may be far apart and this may not be necessary.

Shooting the Scene and Polish

Once everything is set up and in place, the scene will be shot, so to speak. Once the scene is blocked out in the game environment (cameras, etc.) you can then work on the higher details – secondary work such as cleaning up "pan hands" (gripping guns, not clipping through armor, etc.), facial work (no need to make mouth move if not on camera), effects, meshing (more props and atmospheric elements to liven up the scene), and lighting. At the same time, after the effects is sound design pass (explosion, ricochets, music, etc.) – sound cues for the scene.

After sound design (audio, music), you may want to perform a small polish pass (camera tweaks - finesse the key frames, to make it smoother or rougher like "hand held", post process). These touches can go a long way!

Any secondary stuff can just be reimported in.

Optimizations

One thing to understand is the importance of resources and budgets for the scene.

For scenes, loading animations is often done from large packages. The result is a big memory footprint, especially for console. The option is to through a bake and prune process to look at what is being used in Matinee. The goal is take just what is needed – the bare minimum – for a more representative memory footprint.

The bakeandpruneonbuild flag in the Matinee data bubble pulls the necessary animations from the AnimSets and bakes them down into the game level, itself. This is often done only for cooking to consoles and gives Level Designers better idea of level sizes and asset usage.

Sometimes this can be problematic when using in-game animations; if baked and pruned, the data is actually another copy! So, that is an instance where there will be more memory than what is saved. The bake and prune process is mostly beneficial for custom animations that were created for cinematics; but its is very important to cutting out the size of levels vs. removing excess animations.

TIPS and TRICKS

When working with many tools, it's a good idea to always be ready to accommodate change in tools and make adjustments to your workflow. Here are some useful tips and strategies for getting the most out of the latest toolset.

Track Relativity

When you have a movement track it can be relative to initial (default), or relative to world.

Relative to world

wherever you are in the physical world of the 3D space, if you set a keyframe down, it will be right at that spot in the world.

Relative to initial

whenever characters are not in action, they are in the initial position; keyframes are based on where they were initially sitting in the world. This is set by default to avoid messing up the scene and having to do it over.

For example, consider a kitchen layout for a scene. If you were required to move everything in the scene, simply select all of the objects when Matinee is closed and move them; and everything will be where it needs to be. This gives flexibility if something must be moved in the world. Otherwise, all keyframes would have to be re-animated, essentially re-doing the entire scene.

Razoring

Making edits in Matinee can be tricky in that if there is a camera shot that is sitting too long, and you need another camera to come in, in NLE you would just drag it over; but in Matinee, consider all keyframes in the group, otherwise you can mess up everything. For a given camera cut ‐ on that needs to be shortened – an Event track can be placed in the Director Group; Make sure the time bar is where you want it, then put in a keyframe; from there you can then go to the next edit and put the timebar for animations and razor them for a clean cut. Make sure a keyframe is there; put in an in and out for the group, choose Select Keys in Section from the Edit menu which will select all keys in the green section and move them all. You can then move them all, or keyframe them with the arrow keys. This gives a marker to where you're going to move those keypoints to. The Event track is useful for edit points; you can even label it as "cut here" for later.

Audio

Sound in Unreal is in 3D space (5.1).

For audio design in scenes, you can put sound tracks in the director track, as well as anywhere else. If the sound cue is placed in a Director Track, it means it will be FULL STEREO (left, right center). If the sound cue is placed on a character, for example – in the skeletal mesh group – then the sound will originate from the meshes in 3D space. This will make the audio sound like it is coming off to one side, or wherever they are in relation to camera.

Currently, placing the sound cue on a character will base it on origin of the character (zero mark); for motion capture animation, they usually translate far off their origin! The recommended workaround is to use the null cube object in Matinee, which can't be seen in-game. Place the null cube in the scene and attach it to the character's head. Then, create a group attached to the box, and put in a sound track for that box.

Keyboard Shortcuts

Here are some keyboard configurations for more NLE-type manipulation (in Matinee).

1. Note: JKL controls are similar to what is used in Final Cut Pro, etc.); JKL are the positions of your 3 middle fingers.

2. Tip: for placing markers in timeline; a green area in matinee for looping; you can move the timebar that you scrub in Matinne, then press I and scrub to another and press O for the other looping marker.

3. Frames: make sure nothing is selected (click in blank area) to scrub through, use left arrow or right arrow to move one frame at a time throughout a sequence. Select a keyframe, you can use the arrows to move the individual keyframe left or right.

Rendering Out Cinematic Sequences

You can render out a cinematic sequence all from within the editor using the following command to render out each frame.

GameName.exe -dumpmovie -benchmark -fps 30

You may notice that in the first 10-15 frames of the cinematic the textures for the level are not fully loaded. Rather than putting a couple seconds of buffer at the begining of the cinematic to allow enough time for everything to load, you can actually go in and edit a configuration file to turn off texture streaming. Go into your game build Config folder and edit the Engine.ini file. Search for the key bUseTextureStreaming - you'll probably find a few of them. Set the values of these to be false. Save the file and there should be no texture streaming issues.

NOTE: Be sure to make a backup copy of the configuration file for when you're done.

For pre-rendering, you can use the following command for maximum quality:

Game.exe <MapName> -Benchmark -MAXQUALITYMODE -NoTextureStreaming -DumpMovie -FPS=30 -ResX=1280 -ResY=720

Video Capture

While different needs may require unique hardware and software, here's what Epic is currently using in-house:

• Aja IO Board for capturing straight HDMI, which avoids the need for additional video processing
• Final Cut Pro editing on a Mac