Entry 6 : 3D Camera Tracking and Match moving

-

 After the lens distortion is corrected we need to map out the footage and get a 3D representation of it, where we have points on the surfaces of the footage that depict their 3d spacial position, which we can use in Maya to orient our assets.

Camera Tracker node - automatically tracks and generates a point cloud and creates a virtual camera that mimics the real camera which was used to shoot the footage. In the settings tab we can enable visibility of trackers, set no of features, ideally between 500-1000 and enable refine features.

Before we can start tracking we need to input all of the known info in the camera tracker properties, like the presence of distortion in the footage if any, focal length of the lens, camera used (film back preset).

Once the Tracking is done we have a collection of orange markers dotted around the scene, which turn green after solving. The quality and accuracy of the tracking is a measure of how many of the tracking points turned green and how many stayed orange or turned red. 

TIP: If there are any elements in the footage that move independently of the camera motion, such as a person walking through the shot, then we need to Roto and mask out this section throughout the footage to the camera tracker doesnt pick up on that movement.

Reducing Solve error- Why and How ?

Solve error should ideally be kept below 1 for accurate match moving. We can reduce the error rate by:

1. Increasing the Minimum Length, which determines how long( as in no. of frames) a track needs to be alive on screen for it to be factored into the final calculation.

2. Decrease the max Track error and max error.

3. Delete Rejected and unsolved tracks.

Scanline Render node- When connected to a Scene node, the ScanlineRender node renders all the objects connected to that scene from the perspective of the Camera. This node takes the 3D data as input and renders them back into a 2D.

bg link is connected to the backplate footage, Obj/scn is connected to the scene node and camera connected to the camera of the tracker.


Tracker point cloud in 3D view




Establishing Ground plane and setting scale

Select points from the point cloud that represent the ground, ones which have low error rate and longer track length and choose points from different part of the scene at varying depths from the camera.

If you have a reference to the real world dimensions, select 2 points on the footage and select add scale distance. In the properties, enter value in "cm", because Maya uses cm as default measurements.

Pointcloud generator node - can create a dense point cloud using CameraTracker data and to create 3D meshes out of it, which can be exported out to be used in Maya as a reference plane to position assets in 3D.

WriteGeo node- allows us to export the Nuke scene's data such as geometry, lights, cameras, axes and point cloud as an FBX file.

What is HDR and Image based Lighting?

The HDR image is a spherical exr file that has a 360 panoramic view of the whole scene where the footage was originally shot in. We capture this image at varying levels of exposure and combine them in Photoshop to create the final HDR image.

Image based Lighting is a very interesting concept where we use the colour and intensity values of the HDR image itself to light the scene. So in the HDR the sky region produces light as it has very high exposure values and the buildings and other objects in the scene produce very little light and help with creating light color and reflection accordingly.

Maya Workflow

1.Import the 3D geo scene FBX file. Add the undistorted footage as background image to the virtual camera.

2.Create Arnold skydome light and add the HDR image as color to light up the scene.

3.Group all the locators/point clouds in the outliner.

4. Set new Aperture width and height for the Maya camera.

5. Bring in assets. add textures.




Tips of the week: 
1. Reduce Camera visibility of the HDR to 0, and disable backplate visibility for the camera before rendering.
2. Group and rename objects in the outliner and make sure you are using the correct camera before hitting render.
3. While creating the HDR make sure the regions are are meant to cast light (i.e the sky) have a very high exposure value as this is important for Image based lighting.
4. Before exporting the 3D tracker scene from Nuke set scale, else the scene in Maya will be too small to start with.
5. Lock the Virtual camera imported from Nuke to avoid accidentally moving it and screwing up the whole match move



Comments

Post a Comment

Popular posts from this blog

Entry 7: Intro to Maya Render Layers and AOVs

-
Image
 As a 3D Artist I was familiar with the modelling and texturing side of Maya, but I was never exposed to the VFX side of it until now. This week we looked into how to add textures to our assets and how to use the node graphs for the same. Once that is set we learnt about render layers, overriding textures and AOVs. All of the assets in the scene will have aiStandardSurface and the surfaces that receive shadows and reflections will have aiShadowmatte as their materials.  Tips for Materials : 1. Always set color space as RAW for everything except base color. 2. Enable "Alpha is luminance" for everything except base color and normal map. 3. If you have a very shiny CG asset in the scene, add real textures from the footage to the ground and wall planes for the asset to pick up and reflect them as this would help it blend  with the scene. What are Render Layers and why do we use them? Render Layers allows us to separate each element of the render into their own exr image,...
Read more

PTL Production Blog : Entry 2

-
   Colour Correction, Grading and Colour Theory This week’s class was focused on understanding colour properties and how to grade and correct colour in Nuke. One of the key things here is to match the black and white levels of all the elements, the darkest and lightest points of the footage and the corresponding points of the CG and foreground elements should be the same. This can be achieved using the Grade node with which we can set the black and white levels to match. Different levels of correction – Primary Colour correction aims to correct colour at the highest level, it affects the entire final output. Secondary correction is applied to specific elements individually to make them tonally consistent with the rest of the footage. Colour correction is an essential process to make sure all the elements we add match the footage, whereas colour grading is a more artistic choice that aims to bring about a tone to the full final image output, so this layer is added o...
Read more

Final Project Entry 2 : Nuke Workflow

-
Image
The first step was to setup relative paths and to organize everything into appropriate folders. Once I bring in the footage I convert it into an image sequence and apply a primary color grade and correction. Then I imported the 35mm lens distortion grid and ran the grid detection function using the lens Distortion node. Some manual correction was required to detect every point on the grid once that was done the STmap was exported and attached it to the footage. Raw Footage Primary Grading Color Correction The lens distortion caused the bounding box to be expanded by 10 pixels on both axes, which was factored into the reformat node and the undistored footage was exported. Next was to match move and 3d track the footage. This footage was fairly clean there were no issues as the footage only has 2 planes the ground and the wall behind, there are buildings and trees beyond the wall but they get cut as the camera closes into the wall section. So for the majority of the shot its just trackin...
Read more