Camera Tracking is a technique utilized in Virtual Production to pair the movement of a real life camera to that of a digital camera.
PIXERA supports many camera tracking solutions including MoSys, Stype, EzTrack, Trackmen, PSN, SPNet, Blacktrax, FreeD, Optitrak, and many others. For an updated list of what tracking modules are available in PIXERA Control refer to this article. For more on tracking integrations generally refer to this article.
Camera tracking has many places where jitter or delay can be introduced ranging from software operation, network issues, non-optimized content, cabling issues, or system design choices. The following article is not exhaustive, as system designs, project files, hardware types, and so on vary from person to person and company to company. Consider this article a starting place for common troubleshooting problems for camera tracking.
Genlock & Framelock
If camera tracking is introduced to your project several hardware and software pieces of the system running the project will need a unified understanding of timing. Genlock is a solution for ensuring that multiple video sources operate on the same timing reference. Therefore the camera in your setup must be genlocked, the camera tracking solution you're using must be genlocked, and the servers which display tracked content must be genlocked.
Directly genlocking servers is an option wherein each server has a BNC cable plugged into it from a genlock source, ensuring the timing is exactly the same. This approach is simpler and effective, but may not be as cost-effective as combining genlock and framelock together.
Display servers can be a combination of genlocked and framelocked. From Epic Game's documentation they recommend a combination of Genlock and Framelock for Virtual Production applications as it utilizes NVIDIA API's barriers which allows Unreal Engine control over frame presentation on the application level.
If your server stack utilizes a combination of genlock and framelock that means that downstream daisy-chained framelocked servers can be vulnerable to non-optimal performance if an upstream framelocked server goes offline or runs into issues.
In order to have a genlocked server setup a server must have a NVIDIA Quadro or NVIDIA RTX Pro series card and a NVIDIA Quadro sync board.
Studio Camera Parameter Exposure
In addition to PIXERA Control, RX Manager is another resource for troubleshooting. To access and monitor statistics related to the Studio Camera some steps will need to be taken to expose those parameters to RX Manager.
Use with Input from Control or Direct API
In your Studio Camera's inspector the first parameter under the Tracking parameter group is one called Use with Input from Control or Direct API. When this box is un-checked this means the following:
The Studio Camera can be changed and manipulated by hand in the Workspace by the PIXERA operator.
Tracking calculations travel first to the GUI interface and then to PIXERA Control.
Because calculations travel to the GUI first that means that resources are not 100% dedicated to the tracking data calculations. Therefore one should expect possible jitter in camera tracking performance. Instead if Use with Input from Control or Direct API is checked that means the following:
The Studio Camera cannot be changed and manipulated by hand in the Workspace by the PIXERA operator.
Tracking calculations travel directly to the Render Engine, optimizing tracking performance with no jitter.
Create Live Trace in Engine for Parameter
In the Screens tab, select the Studio Camera and look into the Inspector on the right under the Tracking parameter drop-down menu. Within it Create Live Trace in Engine for Parameter should be set to something, for example RotY. The specific item selected in this drop-down isn't what matters so much as selecting anything ensures that timing data is being sent and processed by RX Manager.
RX Manager & Monitoring
RX Manager is a built-in monitoring tool and method to query PIXERA calculations and traffic to a detailed degree. To access RX Manager navigate to Settings (Gear Icon at the top right of PIXERA) > Help > System Info and then click the button called Launch Web UI.
You can also access the RX Manager by typing in the IP address of your Manager or your loopback adapter if it's running locally followed by “:8020”. So what you type in might look something like this:
192.168.10.100:8020
Select this button and it will launch a browser window called RX Manager. After that select the Monitoring tab at the top. From here you can search for some terminology.
Below is a non-exhaustive list of terminology you can search for in RX Manager's Search bar. It is recommended to type these into the Search bar available under the Monitoring tab, since it has a line graph that displays data. The following parameter names are common ones to look for first when troubleshooting tracking.
Dropped Frames/s
This parameter Dropped Frame/s records dropped frames from all of the systems connected to Pixera. This includes not only a Manager machine, but also clients that are connected to the Manager via Presence.
Dropped Frames don't necessarily point to one category of issue, but help to diagnose where snags or jitter could be occurring. It has been observed that if jittering in camera tracking is present, but the Dropped Frames isn't present (value is 0 when you search for Dropped Frames/s) that means there is something wrong with the timing of the tracking system prior to it entering into PIXERA.
If, however, Dropped Frames are present in the Monitoring tab and also jitter is observable on the camera tracking that means there is something wrong with sync (genlock / framelock) or there is some processing overhead with rendering the output occurring.
In the case of rendering, a dropped frame prevents the inner frustum from updating and as a result the output drops a frame and causes jitter. If, for instance, a Manager/Client setup with several servers has this type of problem it may manifest as Dropped Frames across all Machine outputs. This may look like spikes on the line graph under the Monitoring tab.
Trace Update After VSync
Trace Update After VSync represents the tracking data that is received after vsync timing is included. If the target frame rate is 30 FPS that is 33.33333 milliseconds equivalent. For this parameter's line graph it should not return 0 nor 33, because if it did it would indicate either the beginning edge of your frame or the end edge of your frame. Instead a healthy system would have a number float somewhere between these values to indicate that the calculation is happening somewhere just after the start of the frame and before the center of the frame.
In order to ensure that calculation happens somewhere in the beginning/center of the frame you have to apply a delay of your desired millisecond amount upstream of PIXERA in your tracking device. Generally it is recommended to stay in the first third of the frame rather than the center or near the end of the frame. In the above example, the Trace Update After VSync parameter for a 30FPS (33.33333 millisecond) setup, should return a healthy value of somewhere in the 5-10 range.
Trace Update Interval
Trace Update Interval represents the time in between each tracking frame. Sometimes this value floats up and down from where it is supposed to be. If this occurs sometimes shifting the timing upstream of PIXERA such that the timing lands in the middle of the frame is recommended.
Trace Manager Service Latency
Trace Manager Service Latency is the latency that occurs when the Manager machine sends tracking data to the clients. Normally this value is low, but if this value is high it is likely an indication that there is something wrong with your network.
A Manager machine that receives the tracking data then sends it to the Render Engine and to the Clients. It is expected then that the line graph which represents this value is a straight line for client servers that are genlocked and/or framelocked. This value may not necessarily be a consistent value for the Manager machine because the manager isn't necessarily genlocked.
Trace Client Manager Latency
Trace Client Manager Latency is the latency that occurs when the Client machine sends data to the Manager machine.
Rendering Considerations
Sometimes jittering can occur across client servers when tracking data coming from a Manager machine is also rendering an Unreal Engine scene in it's Preview. If in PIXERA Hub the Manager GPU reports that it's usage is particularly high (80% and above is a clear warning sign) it would be helpful to turn off Preview Rendering on the Unreal Engine Resource by selecting it on your Timeline within the Compositing tab, then navigating to the resource's Setup tab. Next find the Render Properties and select Do Not Render in Preview.
Soft Border Width Factor
If you're utilizing a 3D Virtual Production workflow there is a good chance that content is being back-projected by way of the Studio Camera's Background parameter. The Inner Frustum that is being back-projected onto an LED wall has a crisp, hard edge by default, however a soft blended edge is an option as well.
To engage a softer blended edge to the inner frustum navigate to your Studio Camera's inspector and look for the Background parameter group. Within it there is a parameter called Soft Border Width Factor. If your client servers have a discrepancy with how it's rendering the edge of the Inner Frustum (for example, some servers rendering a crisp edge and others rendering a soft edge), stopping Presence on the servers and restarting it will correct this issue.
Unreal Engine Optimizations
Timing in your uProject
If Unreal Engine is in use with your Virtual Production project and also contains animations within it there are some recommended ways to handle timing within the project to ensure good performance. Affix timing animations to either the Event Tick or the Frame Delta of your project and don't attempt to drive it using any other timing method. This is because outputs defined by nDisplay don't start at the same time, but a consistent through-line between servers running Unreal is the Event Tick or Delta Time.
Re-syncing Clusters on PIXERA's Timeline
A way to re-sync a cluster quickly is to move PIXERA's Now Pointer off of the Unreal Engine resource within your Layer and then move it back onto the Resource container. It is recommended to move the Now Pointer off of the tail-end of the resource rather than the front end. The reason for this is because if you move it off of the front end PIXERA will think you want to pre-load the resource again, which will take time and processing overhead to complete.
Sync Frame Latency
If an Unreal Engine project's FPS is under-performing you may need to adjust it's Sync Frame Latency parameter. Unreal Engine has multiple threads it attempts to synchronize to negotiate game calculations against rendering calculations. Sync Frame Latency is a way to adjust the timing of these threads to allow for optimal performance in real-time media applications.
You can find the Sync Frame Latency parameter by selecting your Unreal Engine Project from the Resources tab in your Compositing tab. Then in the inspector on the right hand side locate to the parameter cluster called Options. Within this grouping there is a parameter called Sync Frame Latency. Change it's value to 1.
Normally when this value is set to 2 that means that Unreal's threads are run in parallel with a chance that one thread may ‘wait’ on another to complete it's calculations, resulting in performance drop in the form of jittering or stuttering. By changing it to 1 the threads runs sequentially.
To confirm that performance is better, navigate to the project's inspector and under Custom Command Line Arguments ensure that stat fps is checked on. Next, launch your project using the Run button under the Launch parameter group. When the project launches a FPS readout will appear in the top right corner of the screen.
Studio Camera Delay Settings
When you select a Studio Camera, in the inspector to the right there is a parameter group called Delay which collects a series of different kinds of delays to ensure that all elements being composited together using the Studio Camera maintain consistent visual timing.
If you're utilizing SMPTE 2110 in a 3D Virtual Productionworkflow wherein the Frustum is back-projected onto the LED wall there may be a certain amount of frames of latency to expect. If, for example, there are 3 frames of latency from utilizing SMPTE 2110 and Unreal Engine introduces 2 frames of latency that means that the Delay value for BG Reprojection Tracking Delay (Frames) should be set to 5.
If you move from a 3D Virutal Production workflow to a 2.5D but you're still utilizing SMPTE 2110 that means that the processing overhead of Unreal Engine is no longer in play. You then calculate the Delay value without the 2 frame latency and are left with a value of 3 which only accounts for the latency that SMPTE 2110 introduces.
PIXERA Control
PIXERA Control's camera tracking modules come with a Damping value that can be used to smooth jitter. For this to be useful framelock and genlock must be stable.
Note that the frequency value listed in tracking modules isn't always 100% correct. Using RX Manager via the web UI is the best way to ensure the information you're querying is accurate.
Contact Us
