X4D Overview

X4D tracks 3D objects (bones, implants, etc.) in X-ray images by generating digitally reconstructed radiographs (DRRs) of the objects and matching them to the X-ray images. It requires that the 3D X-ray configuration parameters be defined in the subject file and that the X-ray images be corrected.

To match DRR images to X-ray images, both sets of images are processed first. X-ray images are smoothed with a convolution filter, and then an edge detection algorithm is performed on them. The final processed X-ray image that is used for tracking is the edge detection image times a weighting factor, plus the smoothed intensity image. DRR images are not smoothed, but are input to the edge detection algorithm. The final processed DRR image is the edge detection image time a weighting factor, plus the original DRR image.

X4D has a job batch option, enabling multiple tracking optimizations to be run without user intervention.

Bone Tracking

[+]4D Optimization

4D optimization tracks one or more objects by solving all frames in the tracking range at the same time. It works using a simulated annealing optimization process over two passes. In the first pass, a temporary pose map is created by evaluating the object’s pose map over the tracking range at equal time stamps. The object poses at these times are the control points that the optimizer varies to find the global solution. The spacing between the time stamps is determined by the Frame Spacing (Pass 1) parameter. For example, if the parameter is set to 4 and there are 20 reporting times in the tracking range, there will be 20 / 4 + 1 = 6 poses in the pose map-- a pose at each end of the range and 4 poses equally spaced between them. The algorithm then proceeds to optimize each control point independently of the others, while evaluating the DRR/X-ray matching for the X-ray frames in the neighborhood of the control point. The size of this neighborhood is determined by the Frame Envelope (Pass 1) parameter. For example, if the parameter is set to 2, each time the optimizer changes the pose at one of the time stamps, it will evaluate the DRR/X-ray matching at the closest X-ray frame to that time (in each view), and in the 2 frames on either side of that closest frame. A value means to evaluate only the closest X-ray frame.

In the second pass, a temporary pose map is created by evaluating the optimized pose map from the first pass at equal time stamps. This process is the same as described above, but uses the Frame Spacing (Pass 2) parameter to determine the spacing. The algorithm then optimizes each control point independently, evaluating a neighborhood of frames determined by the Frame Envelope (Pass 2) parameter. For this second pass, the number of simulated annealing iterations per control point is halved, the translation and rotation ranges are divided by four, and the initial temperature is divided by four. Once the second pass is complete, its final pose map is evaluated at the reporting times in the tracking range and those poses are inserted into the object’s pose map.

[+]Matching Xray and DRR Images

The process of matching the DRR images to the X-ray images is sensitive to the image processing parameters in the X-ray/DRR Settings widget. Once the DRR images have been generated for a particular set of bone poses the DRR images and the X-ray images are processed (using an identical method) and then compared to each other. The algorithm used to compare them, called the image metric, can be selected in the Object Tracking widget.

The processing method consists of:

1. performing a Sobel edge detection on the image,
2. thresholding the edge-detection image (which is controlled by the Edge Capping values in the X-ray and DRR Settings widget),
3. multiplying the edge-detection image by a weighting factor (which is controlled by the Edge/Intensity Merge value in the X-ray and DRR Settings widget) and adding it to the original image, and
4. thresholding the merged image

Pixels in the edge image whose values are greater than the Edge Capping maximum are set to zero. Very bright pixels (sharp edges) are usually inorganic objects like EMG electrodes or metal plates or wires. They can be removed from the edge image by lowering the Edge Capping maximum from 100%.

Every pixel in the edge image whose value is above the Edge Capping minimum is set to the Edge Capping minimum. This effectively strengthens weaker edges (those below the Edge Capping minimum) because the entire image is scaled later. For example, if the Edge Capping values are set to 5% and 30%, every pixel in the edge image with an intensity that is greater than 30% of the total image intensity range is set to 0. Every pixel with an intensity that is between 5% and 30% of the intensity range is given an intensity of 5%. The resulting edge image has all pixels between 0 and 5% of the intensity range of the original edge image, with many of the edges of interest set to a value of 5% of the intensity range, making them all equally strong.

After the edge image has been weighted and added to the original image, the result is thresholded using the Image Threshold values. All pixels above the Image Threshold maximum are set to the maximum, and all pixels below the Image Threshold minimum are set to 0. Much of the time these thresholds should be left at 100% and 0%. However, there are times when it is useful to raise the minimum above zero to mask soft tissue regions, and lower the maximum from 100% to remove artificial edges, such as the end of a CT bone that is within the X-ray image.

The success of the bone tracking algorithm is particularly sensitive to the Edge Capping minimum for the X-ray images. A change in this value of 2 or 3 (percent) can make a big difference in the solution found by the optimization. If you find that the optimization is locking into a bad pose with the edge capping minimum set to 10%, try lowering the minimum to 7% or 8% (or even 5%) and try again.

When you start processing a new data set, it can take some trial and error to figure out the set of image processing parameters that produce the best results. It is recommended that you start by tracking the bones (individually) in a single frame, trying different parameter values until the optimization algorithm can lock onto the correct pose for each bone. Then try solving a few adjacent frames to see if the same parameter values work on those.

Motion Capture Data

[+]Working with Motion Capture Data

If you have surface-marker-based motion capture data for an x-ray trial, you can use it to calculate initial poses for the bones you want to track in the trial. To use this data in X4D, you must have C3D files with labelled marker trajectories for the motion trial and its associated static trial, and Visual3D files defining a kinematic model and its motion constraints. In the DSX software, this model is referred to as a mocap model. Creating the model template and the constraint scripts requires a good working knowledge of Visual3D. The model template file (MDH) defines the segments and landmarks in the kinematic model, based on a specific marker set. The build script (V3S) creates the constraints that define the relative motions of the segments in the model. The recalc script (V3S) updates the segment poses to satisfy the constraints. Once the model is defined, and applied to a particular DSX trial, it can be used in three modes:

Marker Mode

The motions of the segments are driven by the surface markers, with constraints enforced. This mode is used to calculate mocap poses in X4D.

Interactive Mode

You can interactively change all of the model's degrees of freedom (the 6 degrees of freedom of the root segment, and the joint angles). Constraints are enforced. This mode is used when you manipulate the bone poses in X4D.

DSX Mode

The motions of the segments are driven by DSX tracking results (transform matrices). Constraints are not enforced. This mode is used when DSX results are viewed and analyzed in Visual3D.

In addition to generating initial poses for the tracking objects, you can use the mocap model to help with manual positioning of the objects. In the Mocap Model widget in X4D, you can turn on the model's joint constraints, making it easier to move all of the objects together as a group, or to move them relative to each other in a realistic manner (e.g., flexing or twisting the vertebrae in a spine model).

Optimization Image Metrics

The algorithm used to compare the X-ray and DRR images during object tracking.

Pixel Difference

Each pixel in the processed DRR image is compared directly to its corresponding pixel in the X-ray image. To compare a DRR pixel to an X-ray pixel, first the absolute value of the difference between them is calculated. If this difference is greater than the Difference Threshold, and the DRR pixel value is greater than Bright Threshold, the difference is squared. This gives a greater weight to DRR pixels that are bright and which do not match well with their corresponding X-ray pixels. The pixel difference is then squared and multiplied by a brightness factor. This factor is 1.0 plus Bright Factor times the DRR pixel value. When Bright Factor is zero, bright DRR pixels are not weighted differently than any others. But when it is greater than zero, the DRR’s brightness is used to weight the error for that pixel. This is a second method of weighting a bright DRR pixel more heavily, without considering its difference with the X-ray pixel (as the first method does). The values for all pixels in each view are then summed to determine the fitness for that view. The sums for the two views are then multiplied to get the overall image correlation value, which the algorithm tries to minimize.

Parameters

Parameters for Pixel Difference Metric View Weighting: This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. It is the relative weight of the two views in the final image metric (0.0 = all view 1, 1.0 = all view 2). 0.5 means to weight each view equally. Constant DRR Scaling: Whether or not to scale the DRRs for each view by a constant value for all iterations of an optimization. If false, each DRR will be scaled so that its maximum value is equal to the DRR image scale value (X-ray/DRR Settings widget). If true, a single scale value will be calculated during the first iteration and be used to scale the DRRs in all subsequent iterations. Bright Factor: This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. The fitness of a pixel is weighted by its intensity in the DRR image. Once the raw fitness of a pixel is calculated (by finding the squared difference between the DRR value and the X-ray value) it is multiplied by the value: 1.0 + bright_factor * DRR_pixel_intensity. Bright Threshold: This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. The raw fitness of a pixel is calculated by taking the squared difference between the DRR value and the X-ray value. The raw fitness is squared again if the DRR pixel's intensity is greater than this value and the difference between it and the X-ray pixel's intensity is greater than Diff Threshold. Diff Threshold: This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. The raw fitness of a pixel is calculated by taking the squared difference between the DRR value and the X-ray value. The raw fitness is squared again if the DRR pixel's intensity is greater than Bright Threshold and the difference between it and the X-ray pixel's intensity is greater than this value. Xray Scale: The processed X-ray image is scaled linearly by this factor so that all pixels are in the range 0 to this value. Drr Scale: The processed DRR image is scaled linearly by this factor so that all pixels are in the range 0 to this value. Note: Most of the time the scale values for the X-ray and DRR images should both be set to 255.0, but occasionally there are times when it is helpful to scale one image more than the other (e.g., when most of the bone edges in a DRR image are too dull because of one bright area that scales to 255.0).

Conditional Entropy

The cross-conditional entropy, H, of the DRR image relative to the X-ray image is calculated and minimized by the optimization algorithm.

Reference: Wang F, Vemuri B, Rao M, Chen Y (2003). Cumulative residual entropy, a new measure of information & its application to image alignment. Proceedings of the Ninth IEEE International Conference on Computer Vision, October, 2003. DOI: 10.1109/ICCV.2003.1238395.

Parameters

Parameters for Conditional Entropy Metric View Weighting: This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. It is the relative weight of the two views in the final image metric (0.0 = all view 1, 1.0 = all view 2). 0.5 means to weight each view equally. Constant DRR Scaling: Whether or not to scale the DRRs for each view by a constant value for all iterations of an optimization. If false, each DRR will be scaled so that its maximum value is equal to the DRR image scale value (X-ray/DRR Settings widget). If true, a single scale value will be calculated during the first iteration and be used to scale the DRRs in all subsequent iterations.

Menus

[+]File Menu

Open Subject... (Ctrl + O)

loads a subject file.

Save Subject (Ctrl + S)

saves the currently loaded subject to its existing subject file.

Save Subject As...

saves the currently loaded subject to a new subject file.

Load Pose Maps...

loads an existing pose map file for each selected object. You will be prompted to browse for each file, which must be saved as a .csv file.

Save Pose Maps

saves the pose map of each selected object in the current trial to a .csv file. This file contains 6 values for each pose: the XYZ translation and the XYZ Euler angles representing the transform from the X-ray lab frame to the object's local coordinate system (i.e., as if you were traveling from the lab frame to the object frame). If a name for the pose map file is not already specified in the subject file, you will be prompted to browse for the file to create. A link to this file will then be added to the subject file.

Save Pose Maps As...

saves the pose map for each selected object in the current trial to a new .csv file. You will be prompted to browse for a new file for each object. A link to this new file will then be added to the subject file.

Save Object Transforms

saves the pose map of each selected object in the current trial to a .txt file. This file contains a 4x4 transform for each pose, expressing the transform from the X-ray lab frame to the object's local coordinate system (i.e., as if you were traveling from the lab frame to the object frame). If a name for the transforms file is not already specified in the subject file, you will be prompted to browse for the file to create. A link to this file will then be added to the subject file.

Save Landmarks

saves the locations in the X-ray lab frame of each selected object's landmarks in the current trial to a .txt file. This file contains a set of landmark coordinates for each pose in the pose map. If a name for the landmarks file is not already specified in the subject file, you will be prompted to browse for the file to create. A link to this file will then be added to the subject file.

Export Poses to CMZ

exports the pose map of each selected object in the current trial to the session's CMZ file.

Exit (Alt + E)

exits the program. The program can also be closed using the X in the top right corner of the program window.

[+]View Menu

Body Reference Frames (Alt + R)

toggles the display of the local coordinate systems of the selected objects in the X-ray windows and the 3D view.

Body Landmarks (Alt + L)

toggles the display of the landmarks of the selected objects in the X-ray windows and the 3D view.

DSX Configuration (Ctrl + Shift + C)

toggles the appearance of the dockable DSX Configuration widget, which lets you select sessions, configurations, trials, and objects.

3D View (Ctrl + Shift + V)

toggles the appearance of the dockable 3D View window, which displays the X-ray configuration, selected objects, and mocap markers.

Output (Ctrl + Shift + O)

toggles the appearance of the dockable Output window, which shows informational and error messages.

Xray/DRR Settings (Ctrl + Shift + X)

toggles the appearance of the dockable Xray/DRR Settings widget, which gives you access to the settings used to process the X-ray and DRR images.

Pose Maps (Ctrl + Shift + A)

toggles the appearance of the dockable Pose Maps widget, which has commands for editing the pose maps.

Object Tracking (Ctrl + Shift + T)

toggles the appearance of the dockable Object Tracking widget, which lets you track objects in the X-ray images.

Image Optimization (Ctrl + Shift + I)

toggles the appearance of the dockable Image Optimization widget, which can be used to optimize the X-ray, DRR, and image metric settings to better track objects in the X-ray images.

Graphs (Ctrl + Shift + G)

toggles the appearance of the dockable Graphs widget, which displays the pose maps of the selected objects.

Mocap Model (Ctrl + Shift + M)

toggles the appearance of the dockable Mocap Model widget, which lets you apply the mocap model to the objects in the current trial.

Optimization Job Queue (Ctrl + Shift + Q)

toggles the appearance of the dockable Optimization Job Queue widget, which displays the currently loaded tracking optimization job queue.

[+]Queue Menu

Load

loads a job queue from file (*.xjq) into memory.

Save

saves the current job queue to file (*.xjq).

Clear

clears the current job queue (no undo!).

Run

runs the currently loaded job queue.

Abort

aborts a currently running job queue.

[+]Pose Maps Menu

This menu provides the same functionality as the Pose Maps widget.

Objects Affected

Specifies the objects that are affected by the actions in this menu. Available options are: a) All Selected For Display and b) All Selected For Optimization.

Add Current Pose to Maps (Alt + A)

adds the current pose of each object specified in the Objects Affected combo box to its pose map.

Clear Current Poses from Maps (Alt + C)

removes the pose for the current time of each object specified in the Objects Affected combo box from its pose map.

Clear Pose Maps

removes all poses from the pose map for each object specified in the Objects Affected combo box.

Smooth Pose Maps (Alt + M)

smooths the poses in the tracking range for each object specified in the Objects Affected combo box, using the cutoff frequency specified in the graph widget.

Fill Pose Maps (Alt + I)

adds a pose for each reporting time in the tracking range that does not already have one to the pose map of each object specified in the Objects Affected combo box.

Clean Pose Maps (Alt + N)

removes all poses that are not at one of the trial's reporting times from the pose map of each object specified in the Objects Affected combo box.

[+]Options Menu

Settings...

opens the Settings dialog window.

Select GPUs... (Alt + G)

opens a dialog window that lets you assign GPUs to the X-ray views.

[+]Help Menu

Help (F1)

opens the X4D wiki page in the default browser.

System Information...

opens a dialog window with information about your computer, including available RAM and graphics card specifications.

About...

displays a dialog with information about X4D, including the installed version number. It also contains the Deactivate button, for deactivating your DSX suite license.

Widgets

[+]DSX Configuration

Subject

This field is automatically populated with the ID of the subject when you load a subject file.

Session

This drop down menu is automatically populated with all of the sessions available for the current subject.

Configuration

This drop down menu is automatically populated with all of the configurations available for the currently selected session.

Trial

This drop down menu is automatically populated with all of the trials available for the currently selected configuration.

Object

This list is automatically populated with all of the objects in the currently selected trial. Each object is enabled if it is trackable, meaning that it has a 3D image file associated with it, for generating DRRs.

[+]X-ray and DRR Settings

The matching of DRR images to X-ray images is sensitive to the image processing parameters in this widget. Once the DRR images have been generated for a particular set of bone poses, the DRR images and the X-ray images are processed (using an identical method), and then compared to each other using the selected image metric algorithm.

The processing method consists of

1. performing a Sobel edge detection on the image,
2. thresholding the edge-detection image
3. multiplying the edge-detection image by a weighting factor and adding it to the original image, and
4. thresholding the merged image.

Parameters

Edge/Intensity Merge This is the value of the weighting factor in step 3. Edge Capping (%) These two values values govern the edge thresholding in step 2. Pixels in the edge image whose values are greater than the edge capping maximum are set to zero. Very bright pixels (sharp edges) are usually inorganic objects like EMG electrodes or metal plates or wires. They can be removed from the edge image by lowering the edge capping maximum from 100. Every pixel in the edge image whose value is above the edge capping minimum is set to the edge capping minimum. This effectively strengthens weaker edges (those below the edge capping minimum) because the entire image is scaled later. Image Thresholds (%) All pixels above the image threshold maximum are set to the maximum, and all pixels below the image threshold minimum are set to 0. Much of the time these thresholds should be left at 100 and 0. However, there are times when it is useful to raise the minimum above zero to mask soft tissue regions, and lower the maximum from 100 to remove artificial edges, such as the end of a CT bone that is within the X-ray image. Also, when tracking metal implants, it may be helpful to make the DRR a solid, bright color by setting both the minimum and maximum thresholds to 0.

[+]Pose Maps

A pose map is the set of poses for a tracked object. Each pose expresses the transform from the X-ray (lab) frame to the object’s anatomical frame (i.e., as if you were traveling from the lab frame to the object frame), and is at one of the trial’s reporting times. Each of the 6 DOFs in a pose map is interpolated (and possibly smoothed) with a GCV (generalized cross-validation) spline. Smoothing of pose map splines is controlled by the Pose Map Cutoff Frequency parameter. Pose maps are stored in CSV files.

The pose map commands described below operate on the objects specified by the Objects Affected combobox at the bottom of the widget. These commands can also be accessed through the Pose Maps Menu.

Add Current Poses to Maps

For each object, add its pose at the current reporting time to its pose map.

Clear Current Poses From Maps

For each object, removes from its pose map the pose at the current reporting time.

Clear Pose Maps

For each object, clears all poses from its pose map.

Smooth Pose Maps

For each object, replaces each pose in the pose map with one calculated by evaluating the splines at the pose time. The resulting splines may still not pass through the smoothed poses because of the nature of GCV cut-off frequency smoothing.

Fill Pose Maps

For each object, adds a pose to its pose map for each reporting time in the tracking range that does not already have one.

Clean Pose Maps

For each object, removes all poses from its pose map that are not at one of the trial's reporting times.

Objects Affected

Lets you choose which objects are affected by the commands described above. All Selected For Display means all objects selected in the DSX Configuration widget. All Selected For Optimization means all objects selected in the Object Tracking widget.

[+]Object Tracking

Optimize / Move (Keyboard)

These check boxes let you specify which objects are allowed to move during the tracking optimization and moved with keyboard commands. DRRs for all of the objects selected in the Object Configuration widget will be generated, but only the objects selected here will be allowed to move. This allows you to perform hierarchical tracking, which can be useful when bones overlap in the X-ray images. For example, you can track the femur, then fix it in its optimized pose while tracking the patella.

Max Iterations

The maximum number of iterations for one pass of the optimization algorithm. For single-frame optimization, this is the number of iterations for one frame (reporting time). For 4D optimization, this is the number of iterations for the first pass. For the second pass of 4D, half of this number is used as the maximum.

Translation Ranges

There is a slider for each of the X, Y, and Z translation ranges. Each one controls the amount that the DOF is allowed to change from its initial value during optimization. X, Y, and Z are the axes of the X-ray lab reference frame (defined by the calibration object).

Rotation Ranges

There is a slider for each of the X, Y, and Z rotation ranges. Each one controls the amount that the DOF is allowed to change from its initial value during optimization. X, Y, and Z are the axes of the X-ray lab reference frame (defined by the calibration object).

Object Tracking Widget

Tracking

The range of reporting times which will be optimized when using the Solve Range, Solve Poses in Range, or Solve Range 4D commands.

Algorithm

The type of optimization algorithm to use. ASA (adaptive simulated annealing) is currently the only recommended algorithm. SPAN is another type of simulated annealing. LBFGSB is a bounded, least-squares algorithm (local optimizer).

Image Metric

The type of image metric algorithm to use when comparing the processed X-ray images to the processed DRR images. Changing the image metric will automatically update the image metric parameters widget. Currently there are two algorithms implemented: Pixel Difference and Conditional Entropy.

Solve Range

Starts the frame-by-frame optimization process. The selected objects’ poses are optimized at each reporting time in the tracking range. It starts at the current time and proceeds towards the earliest, then goes back to the current time and proceeds towards the latest. Although each frame (reporting time) is optimized independently of the others, previously optimized frames will modify the pose map, which could affect the starting poses for subsequent frames. Right-clicking gives the option to add the frame-by-frame optimization to the job queue.

Solve Current Frame

Performs an optimization of the current frame. Right-clicking gives the option to add the current frame optimization to the job queue.

Solve Poses in Range

Performs an optimization of each reporting time in the tracking range that already has a pose specified in the pose map. Right-clicking gives the option to add the pose optimization to the job queue.

Solve Range 4D

Starts the 4D optimization process. The entire tracking range is optimized at the same time. Right-clicking gives the option to add the 4D optimization to the job queue.

Stop

Stops the currently running optimization.

Evaluate

Evaluates the poses of the currently selected bones using the selected image metric. The results are written to the output window.

Poses Saved During Tracking

Sometimes the global minimum of the image fitness function is not the correct pose of the bone being tracked. This is often because the X-ray and DRR image processing parameters are not ideal (see Image Optimization, below). Other times, it is because the bone in the X-ray images is occluded by soft tissue or other bones, or because of inherent differences between X-ray images and DRRs (image resolution, X-ray scatter, CT thresholding, etc.). To account for these other times, the ASA tracking algorithm attempts to save bone poses that result in local minima of the cost function, while still searching for the global minimum. The idea is that if the correct bone pose is not the global minimum, it is likely to be a local minimum somewhere within the search range. The ASA algorithm thus attempts to save a set of local-minimum poses so that you can revisit them once optimization has finished. The Max Poses field specifies how many local minima to save. When the tracking optimization has finished, if the final bone pose is not the correct pose, you can re-apply the saved poses by dragging the slider for that bone. If one of them is correct (or at least better than the final pose), you can use the Add Current Poses to Maps command in the Pose Map widget to add that pose to the map, thus overwriting the final pose. Note: When the ASA algorithm evaluates a particular bone pose during optimization, it cannot determine if it is actually a local minimum or not because it does not compute gradients. It estimates if the pose is a local minimum by examining its cost function value and its proximity to other poses with similar values. Here is an example that details the process. If you perform an ASA optimization of 1000 iterations, with Max Poses set to 20 and Distance (squared) set to 3.0, when the optimization is complete the 1000 bone poses that were evaluated are sorted from lowest to highest cost function value. The pose with the lowest value is marked as the global minimum and added to the list of saved poses. Then the pose with the next lowest value is checked to see how close it is to the first pose. The sum of the squares of the differences in the 6 DOF values is computed and compared to the Distance (squared) value (3.0 in this case). If the distance is less than 3.0, it is assumed that this pose is in the same valley of the solution space as the first pose, so it should not be saved as a local minimum. If it is greater than 3.0, the pose is added to the saved pose list. Then the next pose in the sorted list is checked. If it is not within 3.0 units of any pose already in the saved list, it is added to the saved list. This process proceeds until the saved list contains 20 poses. It will take some trial and error to determine the best value of the Distance (squared) parameter for a particular data set, but a value in the range of 1.0 to 3.0 is reasonable, as the solution spaces are often very bumpy with numerous local minima. The units for the DOF translations are model units (usually mm), and the units for the rotations are degrees.

[+]Image Optimization

Accurate tracking of bones is sensitive to the X-ray and DRR image processing parameters. In some cases it can take a significant amount of trial and error to determine the parameter values that enable the image matching algorithm to "lock in" on the correct bone pose. That is, the parameter values that cause the global minimum of the image fitness function to be at the correct bone pose. There are 15 different parameters, and sometimes changes of 1 or 2% in one of them can change the "optimal" bone pose by a few degrees or millimeters. The Image Optimization widget is a tool for automatically finding optimal values of the parameters. The sliders allow you to specify ranges for each of the 15 parameters in the Xray/DRR Settings widget, and the Max Iterations slider at the top is for specifying how many tracking optimizations to perform with randomly chosen parameter values within the specified ranges. Each tracking optimization proceeds as if you had selected Solve Current Frame in the Object Tracking widget. It optimizes the selected bones, starting at their current poses, and compares the final poses to the initial ones. The smaller this difference, the better the result. Please see How To: Optimize Xray/DRR Settings for a step-by-step guide on using this widget.

[+]Graphs

In the Graphs widget you can display and edit the pose map of one of the selected objects. The graphs show the six DOFs of the object's pose at each reporting time. To display the graphs, select an object from the combo box in the upper left corner of the widget. With the checkboxes above the graphs you can individually control the display of the X, Y, and Z translations and the X, Y, and Z rotations (Euler angles). The order in which the Euler angles are applied can be changed using the Euler combobox at the top. The default order of XYZ is sufficient for most cases, but if you encounter gimbal lock with an object's poses you can try a different order to avoid the problem. You can also select the type of spline used to interpolate the poses (GCV or linear), and the cutoff frequency used to smooth the GCV splines.

The circles on the graphs represent the poses in the object’s pose map. The dashed lines are splines (GCV or linear) fit to the circles (control points) and smoothed with the cutoff frequency specified at the top. The gray region behind the curves represents the tracking range specified in the Object Tracking widget. The vertical green line shows the current reporting time.

The graphs can be moved left, right, up, and down in the window using the Shift key and the left mouse button. The Y axis can be zoomed in and out using the mouse wheel. Pressing 'r' resets the graph axes to display the splines for all of the selected curves in that graph. Pressing Ctrl + 'r' resets the graph axes to display all of the control points of the selected curves.

To move a control point, left-click on it and drag it up or down to a new location (you cannot change the time of a control point). You can select multiple points by pressing Ctrl while left-clicking on the points, or by sweeping a region while pressing the left mouse button. You can then move all selected points together by left-clicking on any of them and dragging up or down. To unselect all control points, left-click away from all points or press Esc.

Note: moving control points in the graphs directly modifies the pose map that is loaded into X4D, and there is no undo feature. It does not, however, modify the pose map file; this can only be done by selecting Save Pose Maps from the File menu. Also, whenever a pose map is modified by any method in X4D, it is saved to a backup file in the same folder as the original pose map file.

[+]Mocap Model

Time Offset

The offset between the start of the x-ray data capture and the surface-marker motion capture. It is specified as the x-ray start time minus the motion capture start time.

Calculate Mocap Poses, Fixed Body, Mocap Seed Spacing

If Fixed Body is set to none, then Calculate Mocap Poses will clear the pose maps for the objects selected in the Configuration widget and create new poses based on the motion capture data. Poses are created for the entire trial, using the value of the Mocap Seed Spacing parameter. For example, if the spacing is 5, then a motion capture pose will be calculated for every 5th reporting time in the trial.

If Fixed Body is set to one of the tracked objects in the trial, then that object's pose map will not be modified by Calculate Mocap Poses. Instead, the pose maps for the other selected objects will be cleared. Then for each pose in the fixed object's pose map, the poses of the other objects will be calculated such that they satisfy the joint constraints relative to the fixed object and the generalized coordinates of the motion capture data.

Note: One strategy for tracking multiple bones in a trial is to first Calculate Mocap Poses with no fixed body. This will position all of the bones using the surface markers in the motion capture data. Then track one of the bones for all frames in which it is trackable, deleting the extra mocap-based poses when you are done. Then Calculate Mocap Poses again, with Fixed Body set to the bone that you just tracked. The other bones will "snap into place" relative to the tracked bone. In other words, it will use the motion capture data to pose all the bones, then move them as a group so that the fixed bone lines up with its tracked pose.

Apply Joint Constraints

When this option is selected, the constraints in the kinematic model specified for the trial are applied to the tracking objects. When you select it, the root object remains in its current pose, and the other objects are moved so as to satisfy the kinematic constraints. You can then move the entire model as a rigid unit by selecting and moving any object in it, and you can move the joints of the model by moving the generalized coordinate sliders below the checkbox.

[+]3D View

The 3D View window displays a 3D scene representing the X-ray configurations and the tracking objects. It is a good way to visualize the relative poses of multiple tracking objects, and the mocap markers can also be displayed around them.

This window also provides another method of moving the objects. To select an object for moving, left-click on it. A trackball will be displayed, centered on and aligned with the object's local coordinate system. This trackball allows you to move the object in the following ways:

3D View

• Press and hold the left mouse button on one of the three circles, then drag the cursor to rotate the object about the axis perpendicular to the circle. Release the left mouse button when you are done.
• Press and hold the left mouse button on one of the three axis lines, then drag the cursor to translate the object along that axis. Release the left mouse button when you are done.
• For free-form rotation about all three axes at once, press and hold the left mouse button inside the trackball but not on any circle or line. Drag the cursor in any direction to rotate the object about an axis perpendicular to the direction of mouse movement. Release the left mouse button when you are done.
• For free-form translation, press Shift and then press and hold the left mouse button inside the trackball but not on any circle or line. Drag the cursor to translate the object in the plane of the screen. Release the left mouse button when you are done.
• Press the Esc key or left-click outside the trackball to deselect the object and remove the trackball.

Note: moving an object with these controls does not move other objects with it, even if Apply Joint Constraints is turned on in the Mocap Model widget.

Click to expand the commands that are available in the 3D window

Show Lab (Alt + L): toggles the display of the X-ray source, X-ray image frame, and projection line. Show Objects (Alt + O): toggles the display of the surface models of the selected objects. Show <view 1> Image: toggles the display of the X-ray + DRR image for the first view. Show <view 2> Image: toggles the display of the X-ray + DRR image for the second view. Show Axes (Alt + A): toggles the display of the X-ray lab reference frame. Show Mocap Markers (Alt + M): toggles the display of the motion capture markers for the current reporting time. Align View with X-Axis: moves the camera to look down the negative X axis. Align View with Y-Axis: moves the camera to look down the negative Y axis. Align View with Z-Axis: moves the camera to look down the negative Z axis. Align View with View 1 Source: moves the camera to the X-ray source in the first view, and points it along the X-ray projection line. Align View with View 2 Source: moves the camera to the X-ray source in the second view, and points it along the X-ray projection line. Context Menu: pressing the right mouse button will display a context menu with some of the functionality described above.

[+]X-ray Windows

The X-ray windows display the X-ray images of each view, as well as the DRRs of all selected objects.

Moving the DRRs

To move an object in one of the 2D X-ray windows, Shift-middle-click on the DRR to select it. A reference frame will be displayed over it, with its origin at the center of the object's CT volume. To translate the object, Shift-left-click on the DRR and drag it to a new location. To rotate the object about the origin of the displayed reference frame, Ctrl-left-click on the DRR and drag the cursor. To deselect the object, press the Esc key or Shift-middle-click in the window away from any of the DRRs. Note 1: Moving objects with the mouse does not modify their pose maps. To save the new poses you must use the Add Current Poses to Maps command in the Pose Map widget. If you do not save the poses, they will be lost when you change the current reporting time. Note 2: If Apply Joint Constraints is turned on in the Mocap Model widget, all of the objects in the trial will move with the one you manipulate directly, even if they are not selected.

Mouse and Keyboard Commands

The following mouse and keyboard commands are available in the 2D windows for interacting with the xray images and DRRs. The commands that move the objects move all objects selected in the DSX Configuration widget, whether or not they have been picked for interaction (with Shift-middle-click). To rotate all selected objects by +0.5 degrees about their local X, Y, and Z axes, press Ctrl and the X, Y, or Z keys. To rotate all selected objects by -0.5 degrees about their local X, Y, and Z axes, press Alt + Ctrl and the X, Y, or Z keys. To translate all selected objects by +0.5 mm along their local X, Y, and Z axes, press Shift and the X, Y, or Z keys. To translate all selected objects by -0.5 mm along their local X, Y, and Z axes, press Alt + Shift and the X, Y, or Z keys. The left, right, up, and down arrow keys translate all selected objects by 0.5 mm in the plane of the X-ray window. Pg Up/Down: moves to the previous/next reporting time. Home/End: moves to the first/last reporting time. < and > rotates the selected objects about an axis perpendicular to the screen. The mouse wheel zooms the X-ray image in and out. Press and hold the left mouse button to change the image contrast (move cursor left/right) and brightness (move cursor up/down). Press and hold the middle mouse button to pan the X-ray image within the window. H toggles the display of on-screen help text. C moves the selected objects in front of the X-ray images. R resets the position of the X-ray image in the window, as well as the brightness and contrast. S toggles the display of the X-ray image (same as Show Xray checkbox). D toggles the displays of the DRRs (same as Show DRR checkbox). P toggles whether the displayed X-ray is the raw image or the processed image (same as Show Processed Xray checkbox). O toggles whether the displayed DRRs are the raw ones or the processed ones (same as Show Processed DRR checkbox). Alt + R toggles the display of the selected objects' local coordinate systems. Alt + L toggles the display of the selected objects' landmarks. Note 1: Moving objects with keyboard commands does not modify their pose maps. To save the new poses you must press the Add Current Poses to Maps button in the Pose Map widget. If you do not save the poses, they will be lost when you change the current reporting time. Note 2: If Apply Joint Constraints is turned on in the Mocap Model widget, all of the objects in the trial will move with the one you manipulate directly, even if they are not selected.

Context Menu

Context menu for 2D view Pressing the right mouse button will display a context menu with the following functionality (shortcut key between brackets): Select Drr: selects the DRR under the cursor for manipulation. Show Xray (S): toggles the display of the X-ray image. Show Processed Xray (P): toggles whether the displayed X-ray is the raw image or the processed image. Show DRR (D): toggles the displays of the DRRs. Show Processed DRR (O): toggles whether the displayed DRRs are the raw ones or the processed ones. Move to First Frame (Home): moves to the first reporting time. Move to Previous frame (PgDown): moves to the previous reporting time. Move to Next Frame (PgUp): moves to the next reporting time. Move to Last Frame (End): moves to the last reporting time. Reset View (R): resets the position of the X-ray image in the window. Center Objects (C): moves the selected objects in front of the X-ray images. Toggle Help (H): toggles the display of on-screen help text. Toggle Image Info (I): toggles the display of some meta data of the currently loaded image.

[+]Optimization Job Queue

Optimization Job Queue

X4D has the option to run multiple tracking optimization jobs without user intervention. To add a job to the job queue, right-click on the appropriate Solve button in the Object Tracking widget and select Add to Queue. A new job is created with the current values of the X-ray/DRR, metric, and tracking parameters, then it is added to the queue. All jobs in a queue must be from the same subject file. As you are adding jobs, you can select any session, configuration, trial, and group of objects to track. When an optimization queue is run, the jobs are run in the order of the queue and for each individual job, X4D will automatically adjust all settings and parameter values to the appropriate values before this job is actually run.

The Optimization Job Queue widget has a toolbar with buttons that have the same functionality as the items in the Queue menu. Below its toolbar, this dialog displays the currently loaded job queue in a tree view, and various items can be expanded or collapsed. Each job can be dragged-and-dropped to change the order of the jobs in the queue. Right-clicking on an item in the tree view opens a menu with the following commands:

Edit: opens a dialog in which the current item can be edited.

Remove: removes the current item [only for entire jobs].

Resize: resizes the column to the width of the current item.

Note 1: When editing a job queue file, it is advised to only change the values of certain attributes. There are currently no built-in checks to prevent the job queue files from becoming corrupted. Be careful to keep doubles as doubles (with decimal point) and integers as integers (without decimal point).

Note 2: Only entire jobs should be removed from the queue. Although the Edit dialog shows a red 'x' to delete an item, it should be used sparsely; removing part of a job could make the job incomplete, and there is currently no way to add individual parts to a job.

[+]Output Window

Shows messages output by certain commands, such as when files are written. Also shows the progress and results of tracking and image optimizations, and the results of image evaluation.

[+]Settings

CPU Threading

With two CUDA-capable graphics cards, object tracking can be made faster when each card is controlled by its own CPU thread. To assist with customer support, C-Motion staff might occasionally ask to disable CPU threading when debugging problems with graphics cards.

X-Ray Opacity

The opacity of the X-ray images in the 2D and 3D windows. Their opacity should be less than 1.0 so that the DRRs can be seen superimposed on them. The default value is 0.5.

Simulated Annealing Temp

The initial temperature for the ASA simulated annealing algorithm. The higher the initial temperature, the more slowly the search range will narrow during the optimization. The default value is 2000.

Color Theme

You can choose between a light and dark color theme for the graphical interface.

[+]Select GPUs

This dialog lets you specify which graphics cards are used for processing the X-ray and DRR images for each view. If you have two CUDA-capable graphics cards, it is recommended that you use one card for each view. If you have two cards and one is significantly faster than the other, it may make DRR generation (and thus object tracking) more efficient by using the faster card for both views. If you are having problems with DRR generation that you suspect might be related to the graphics card[s], you can also turn off the use of the cards by one or both X-ray views. This will make DRR generation very slow, however. You cannot assign more than one graphics card to a view.