Troubleshooting

This document collects troubleshooting guidance for the Raptor SDK and the CANV tooling. It first lists common runtime issues and their solutions, then describes how to use Vantor 3D Explorer to visually debug the CANV recordings produced by the refine_canv_poses.py tool and by the SDK's Debug Capture.

Common Issues & Solutions

Initialization Failures

Problem: SDK initialization throws exception

Solutions:

• Ensure map files (.r3db or .3tz) exist and are readable
• Check available RAM/GPU memory (minimum 4GB required, 8GB recommended)
• Verify Vulkan-compatible GPU and drivers
• Use absolute paths for reliability

Poor Quality Results

Problem: Low Confidence (confidence < 0.7) or PoseEstimationError::Failed

Root Causes & Solutions:

1. Initial pose too inaccurate

   • If the initial pose is too far off, the algorithm may fail to find a good match
   • Provide covariance matrix reflecting actual uncertainty

2. Camera calibration mismatch

   • Verify FOV values match actual camera calibration
   • Make sure the image has been undistorted — the camera's intrinsic parameters must be calibrated and compensated for before calling updatePose()

3. Image quality issues

   • Ensure sufficient contrast and texture in image
   • Check for motion blur or focus issues

4. Map Coverage problems

   • Verify 3D map covers your area of interest
   • Check map resolution is sufficient for your altitude

5. Correlation mismatch between image and map

   • If the image contains water bodies and/or sky, the correlation with the map will be poor.
   • Verify the map covers the area with sufficient detail
   • If the terrain has changed since the map was created (e.g. construction, natural disasters), the correlation will also be poor.

Covariance Configuration Issues

Problem: Estimation failures with uncertain initial poses

Solutions:

• Always provide covariance when pose uncertainty is high
• Set variance values to match actual uncertainty (not smaller)
• Monitor success rates and adjust covariance accordingly

Performance Issues

Problem: Processing too slow for real-time applications

Solutions:

• Set maxTime parameter for time-limited processing
• Reduce detailStage1 and detailStage2 values
• Provide appropriate covariance to focus search area

Debugging CANV recordings with 3D Explorer

Vantor 3D Explorer can load a CANV recording, project each frame onto the loaded 3D map, and replay the camera trajectory. Because it renders the same 3D data the SDK matches against, it is the most direct way to visually judge whether a pose is correct: when the pose is right, the video frame projects onto the terrain so that image features line up with their counterparts on the map; when the pose is wrong, the projection is visibly offset or tilted.

Where the CANV recordings come from

Two SDK workflows produce CANV recordings that are useful to inspect this way:

• refine_canv_poses.py writes an output CANV (--output refined.canv) in which each frame carries Raptor's refined position, attitude, metrics.covariance_matrix, metrics.confidence, and an extras.raptor_guide block. Keep the original input recording too — comparing input vs. refined is one of the most useful checks (see below).
• PoseEstimator::startDebug() / stopDebug() records every updatePose() / updatePosition() call made by your own integration into a CANV archive, so you can replay exactly what the SDK saw in production.

Loading a CANV recording

1. Add the 3D map the recording was refined against to the workspace — alignment can only be judged against the same map data. Right-click the workspace, click Add, then click Vivid Terrain Dataset for a 3DSM (.r3db) map or 3D Tiles for a 3D Tiles dataset. Browse to the dataset, click Get layers, and click OK.
2. Right-click the workspace, click Add, then click Canonical Video.
3. In the URL box, browse to the .canv file and click Get layers. A single-stream recording (such as a refine_canv_poses.py output) yields one layer; a Debug Capture recording yields several (see the note below).
4. Double-click the layer to navigate to the recording, then use the timeline playback controls to play it back.

Note: A Debug Capture (startDebug()) recording is self-describing and exposes three streams as separate layers when you click Get layers: camera_initial (the input camera image with the input pose), camera_refined (the same image with Raptor's refined pose), and rendered (the SDK's own rendered view of the map at the input pose). Add whichever layers you need: overlaying camera_refined and comparing it against rendered shows directly how the refined pose lines up with what the SDK matched against, while camera_initial vs. camera_refined shows the correction the refinement applied.

Checking pose alignment against the map

This is the primary sanity check for a refined pose:

1. Select the video overlay and click Align camera to place the 3D view at the exact position and orientation of the current frame's camera — you are now looking "through" the estimated pose.
2. Compare the projected video frame with the underlying 3D terrain:
   • Good pose — buildings, roads, and terrain edges in the video line up with the same structures on the map.
   • Bad pose — the projection is shifted, rotated, or scaled relative to the map. A consistent shift across the whole recording usually points at a systematic input error (e.g. wrong height system, lens FOV, or platform-to-camera mounting), whereas erratic per-frame jumps point at low-confidence frames.
3. Step frame by frame to find exactly which frames are misaligned, then cross-reference those frame indices with the per-frame log printed by refine_canv_poses.py.

Inspecting per-frame CANV metadata

Both the player window (opened by clicking the video layer in the workspace) and the Video Comparison window can show the raw CANV metadata next to the image:

1. Click Set layout and choose one of the table layouts (metadata below / left / right of the video, or in a separate tab) to reveal the metadata table.
2. Open the Canonical Metadata tab. As you scrub the timeline it lists the per-frame values the SDK wrote — position, attitude, lens, metrics.confidence, and metrics.covariance_matrix.

Use this to confirm the values Explorer is drawing match the per-frame log from refine_canv_poses.py (or the extras.raptor_guide block in a Debug Capture recording).

Comparing the input and refined recordings

To see what the SDK actually changed, load both the original recording and the refined.canv and compare them:

1. Select both canonical video layers in the workspace, right-click, then click Compare Videos... to open the Video Comparison window (up to six videos side by side, all overlays shown simultaneously in the 3D view).
2. Click Build Track on each to draw its flight path.
3. Click Covariance to toggle a covariance ellipsoid at the camera position for the current frame (only available when the frame carries metrics.covariance_matrix, which refined CANV frames do). The ellipsoid visualizes the positional uncertainty only — it is derived from the position block of the covariance and does not depict the attitude (orientation) uncertainty. An implausibly large ellipsoid flags a frame whose refined pose should be treated with caution.

Flipping between overlays

To rapidly A/B the two recordings directly in the 3D view, add both layers to a flip group: select both layers in the workspace, right-click, then click Add to flip group. With the workspace in flip-group mode, press the , (comma) / . (period) keys — or the index number shown next to each layer — to flip between them. Because both project onto the same terrain, a shift visible while flipping is exactly the pose change the refinement applied (or, for a Debug Capture recording, the residual between camera_refined and rendered).

Persisting a flight path as a layer

Build Track in the comparison window is transient. To keep a flight path as its own workspace layer, use the Video Path tool: click Tools, then Video toolset > Video Path, drag the canonical video into the Add video dataset box, choose the Flight path / Look at path / Direction visualizations, enter an Output name, and click Start. Generating a path layer for both the input and refined recordings lets you compare the two trajectories on the terrain after the comparison window is closed.

Canonical Video Analysis

The Canonical Video Analysis tool runs a per-frame geometric analysis and flags problematic frames, which maps closely onto the SDK quality concepts:

1. Right-click the video layer and click Canonical Video Analysis.
2. Drag the mesh/3DSM dataset into the Mesh dataset box.
3. Select a frame range on the frame bar and click Process frames.
4. Frames with issues are colored in the frame bar; use the navigation buttons to jump between them.

The reported metrics relate back to the SDK as follows:

Explorer metric	SDK relevance
Mesh Coverage	Mirrors PoseOutput::mapCoverage; low coverage means the view falls outside the map and correlates with low confidence (see Map Coverage problems).
Sky Visible	High sky fraction is a common cause of poor correlation (see Correlation mismatch).
Confidence	The metrics.confidence written per frame by refine_canv_poses.py.
Height Above Ground / Pitch Angle / Footprint	Sanity-check the geometry of the refined pose against the terrain.

Note: Explorer's analysis results are approximate and intended as guidance; treat them as a way to locate suspect frames, then confirm with the SDK's confidence and mapCoverage values.

Plotting per-frame parameters

To spot trends across a recording (e.g. an altitude bias or a drifting attitude), right-click the video dataset and click Plot parameter, then select Yaw, pitch, roll, Field of view, Altitude (EGM2008), or Height above terrain and click Start. Plotting the same parameter for the input and refined recordings side by side highlights exactly which pose component the refinement adjusted.

Further reference

• SDK integration and the Config / EstimationOptions knobs referenced above: Integration Guide
• Confidence and covariance interpretation: Uncertainty
• Producing the CANV recordings: python/tools/README.md and Debug Capture