X4D Overview: Difference between revisions
Line 91: | Line 91: | ||
<ul> | <ul> | ||
<li> | <li> | ||
'''View Weighting''': | '''View Weighting''': This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. It is the relative weight of the two views in the final image metric (0.0 = all view 1, 1.0 = all view 2). 0.5 means to weight each view equally. | ||
</li> | </li> | ||
<li> | <li> | ||
Constant DRR Scaling | '''Constant DRR Scaling''': Whether or not to scale the DRRs for each view by a constant value for all iterations of an optimization. If false, each DRR will be scaled so that its maximum value is equal to the DRR image scale value (X-ray/DRR Settings widget). If true, a single scale value will be calculated during the first iteration and be used to scale the DRRs in all subsequent iterations. | ||
</li> | </li> | ||
<li> | <li> | ||
Bright Factor | '''Bright Factor''': This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. The fitness of a pixel is weighted by its intensity in the DRR image. Once the raw fitness of a pixel is calculated (by finding the squared difference between the DRR value and the X-ray value) it is multiplied by the value: 1.0 + bright_factor * DRR_pixel_intensity. | ||
</li> | </li> | ||
<li> | <li> | ||
Bright Threshold | '''Bright Threshold''': This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. The raw fitness of a pixel is calculated by taking the squared difference between the DRR value and the X-ray value. The raw fitness is squared again if the DRR pixel's intensity is greater than this value and the difference between it and the X-ray pixel's intensity is greater than ''Diff Threshold''. | ||
</li> | </li> | ||
<li> | <li> | ||
Diff Threshold | '''Diff Threshold''': This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. The raw fitness of a pixel is calculated by taking the squared difference between the DRR value and the X-ray value. The raw fitness is squared again if the DRR pixel's intensity is greater than ''Bright Threshold'' and the difference between it and the X-ray pixel's intensity is greater than this value. | ||
</li> | </li> | ||
<li> | <li> | ||
Xray Scale | '''Xray Scale''': The processed xray image is scaled by this factor before comparison to the DRR image. | ||
</li> | </li> | ||
<li> | <li> | ||
Drr Scale | '''Drr Scale''': The processed DRR image is scaled by this factor before comparison to the xray image. | ||
</li> | </li> | ||
</ul> | </ul> | ||
|} | |||
|} | |} | ||
Line 134: | Line 128: | ||
| | | | ||
[[Image:CE_Parameters.png|200px|right|Parameters for Conditional Entropy Metric]] | [[Image:CE_Parameters.png|200px|right|Parameters for Conditional Entropy Metric]] | ||
<ul> | |||
<li> | |||
'''View Weighting''': This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. It is the relative weight of the two views in the final image metric (0.0 = all view 1, 1.0 = all view 2). 0.5 means to weight each view equally. | |||
</li> | |||
<li> | |||
'''Constant DRR Scaling''': Whether or not to scale the DRRs for each view by a constant value for all iterations of an optimization. If false, each DRR will be scaled so that its maximum value is equal to the DRR image scale value (X-ray/DRR Settings widget). If true, a single scale value will be calculated during the first iteration and be used to scale the DRRs in all subsequent iterations. | |||
</li> | |||
</ul> | |||
|} | |} | ||
|} | |} | ||
Revision as of 13:35, 18 June 2020
Language: | English • français • italiano • português • español |
---|
X4D tracks 3D objects (bones, implants, etc.) in X-ray images by generating digitally reconstructed radiographs (DRRs) of the objects and matching them to the X-ray images. It requires that the 3D X-ray configuration parameters be defined in the subject file and that the X-ray images be corrected.
To match DRR images to X-ray images, both sets of images are processed first. X-ray images are smoothed with a convolution filter, and then an edge detection algorithm is performed on them. The final processed X-ray image that is used for tracking is the edge detection image times a weighting factor, plus the smoothed intensity image. DRR images are not smoothed, but are input to the edge detection algorithm. The final processed DRR image is the edge detection image time a weighting factor, plus the original DRR image.
Tutorials
How To: Get Started Tracking Bones
How To: Optimize Xray/DRR Settings
How To: Calculate Bone Poses from Mocap Data
Bone Tracking
4D Optimization |
---|
4D optimization tracks one or more objects by solving all frames in the tracking range at the same time. It works using a simulated annealing optimization process over two passes. In the first pass, a temporary pose map is created by evaluating the object’s pose map over the tracking range at equal time stamps. The object poses at these times are the control points that the optimizer varies to find the global solution. The spacing between the time stamps is determined by the Frame Spacing (Pass 1) parameter. For example, if the parameter is set to 4 and there are 20 reporting times in the tracking range, there will be 20 / 4 + 1 = 6 poses in the pose map-- a pose at each end of the range and 4 poses equally spaced between them. The algorithm then proceeds to optimize each control point independently of the others, while evaluating the DRR/X-ray matching for the X-ray frames in the neighborhood of the control point. The size of this neighborhood is determined by the Frame Envelope (Pass 1) parameter. For example, if the parameter is set to 2, each time the optimizer changes the pose at one of the time stamps, it will evaluate the DRR/X-ray matching at the closest X-ray frame to that time (in each view), and in the 2 frames on either side of that closest frame. In the second pass, a temporary pose map is created by evaluating the optimized pose map from the first pass at equal time stamps. This process is the same as described above, but uses the Frame Spacing (Pass 2) parameter to determine the spacing. The algorithm then optimizes each control point independently, evaluating a neighborhood of frames determined by the Frame Envelope (Pass 2) parameter. For this second pass, the number of simulated annealing iterations per control point is halved, the translation and rotation ranges are divided by four, and the initial temperature is divided by four. Once the second pass is complete, its final pose map is evaluated at the reporting times in the tracking range and those poses are inserted into the object’s pose map. |
Matching Xray and DRR Images |
---|
The process of matching the DRR images to the X-ray images is sensitive to the image processing parameters in the X-ray/DRR Settings widget. Once the DRR images have been generated for a particular set of bone poses the DRR images and the X-ray images are processed (using an identical method) and then compared to each other. The algorithm used to compare them, called the image metric, can be selected in the Object Tracking widget. The processing method consists of:
Pixels in the edge image whose values are greater than the Edge Capping maximum are set to zero. Very bright pixels (sharp edges) are usually inorganic objects like EMG electrodes or metal plates or wires. They can be removed from the edge image by lowering the Edge Capping maximum from 100. Every pixel in the edge image whose value is above the Edge Capping minimum is set to the Edge Capping minimum. This effectively strengthens weaker edges (those below the Edge Capping minimum) because the entire image is scaled later. For example, if the Edge Capping values are set to 5 and 30, every pixel in the edge image that is greater than 30 is set to 0. Every pixel that is between 5 and 30 is set to 5. The resulting edge image has all pixels between 0 and 5, with many of the edges of interest set to 5, making them all equally strong. After the edge image has been weighted and added to the original image, the result is thresholded using the Image Threshold values. All pixels above the Image Threshold maximum are set to the maximum, and all pixels below the Image Threshold minimum are set to 0. Much of the time these thresholds should be left at 100 and 0. However, there are times when it is useful to raise the minimum above zero to mask soft tissue regions, and lower the maximum from 100 to remove artificial edges, such as the end of a CT bone that is within the X-ray image. Note: the Image Scale Value is used in step 5. The final processed image is scaled linearly so that all pixels are in the range 0 to this scale value. Most of the time the scale values for the X-ray and DRR images should both be set to 255.0, but occasionally there are times when it is helpful to scale one image more than the other (e.g., when most of the bone edges in a DRR image are too dull because of one bright area that scales to 255.0). The success of the bone tracking algorithm is particularly sensitive to the Edge Capping minimum for the X-ray images. A change in this value of 2 or 3 (percent) can make a big difference in the solution found by the optimization. If you find that the optimization is locking into a bad pose with the edge capping minimum set to 10, try lowering the minimum to 7 or 8 (or even 5) and trying again. When you start processing a new data set, it can take some trial and error to figure out the set of image processing parameters that produce the best results. It is recommended that you start by tracking the bones (individually) in a single frame, trying different parameter values until the optimization algorithm can lock onto the correct pose for each bone. Then try solving a few adjacent frames to see if the same parameter values work on those. |
Motion Capture Data
Working with Motion Capture Data |
---|
If you have surface-marker-based motion capture data for an x-ray trial, you can use it to calculate initial poses for the bones you want to track in the trial. To use this data in X4D, you must have C3D files with labelled marker trajectories for the motion trial and its associated static trial, and Visual3D files defining a kinematic model and its motion constraints. In the DSX software, this model is referred to as a mocap model. Creating the model template and the constraint scripts requires a good working knowledge of Visual3D. The model template file (MDH) defines the segments and landmarks in the kinematic model, based on a specific marker set. The build script (V3S) creates the constraints that define the relative motions of the segments in the model. The recalc script (V3S) updates the segment poses to satisfy the constraints. Once the model is defined, and applied to a particular DSX trial, it can be used in three modes:
In addition to generating initial poses for the tracking objects, you can use the mocap model to help with manual positioning of the objects. In the Mocap Model widget in X4D, you can turn on the model's joint constraints, making it easier to move all of the objects together as a group, or to move them relative to each other in a realistic manner (e.g., flexing or twisting the vertebrae in a spine model). |
Optimization Metrics
Conditional Entropy | ||
---|---|---|
|
Menus
File Menu | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
View Menu | ||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
The view menu controls which screen widgets are displayed
|
Options Menu | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Help Menu | ||||||
---|---|---|---|---|---|---|
|
Widgets
DSX Configuration | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
X-ray and DRR Settings |
---|
![]() The matching of DRR images to X-ray images is sensitive to the image processing parameters in this widget. Once the DRR images have been generated for a particular set of bone poses, the DRR images and the X-ray images are processed (using an identical method), and then compared to each other using the selected image metric algorithm.
The parameters that control this process are in the X-ray/DRR Settings widget. Their descriptions can be found here. |
Pose Maps | ||||||||
---|---|---|---|---|---|---|---|---|
A pose map is the set of poses for a tracked object. Each pose expresses the transform from the X-ray (lab) frame to the object’s anatomical frame (i.e., as if you were traveling from the lab frame to the object frame), and is at one of the trial’s reporting times. Each of the 6 DOFs in a pose map is interpolated (and possibly smoothed) with a GCV (generalized cross-validation) spline. Smoothing of pose map splines is controlled by the Pose Map Cutoff Frequency parameter. Pose maps are stored in CSV files.
|
Object Tracking | ||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Image Optimization |
---|
Accurate tracking of bones is sensitive to the X-ray and DRR image processing parameters. In some cases it can take a significant amount of trial and error to determine the parameter values that enable the image matching algorithm to "lock in" on the correct bone pose. That is, the parameter values that cause the global minimum of the image fitness function to be at the correct bone pose. There are 15 different parameters, and sometimes changes of 1 or 2% in one of them can change the "optimal" bone pose by a few degrees or millimeters. The Image Optimization widget is a tool for automatically finding optimal values of the parameters. The sliders allow you to specify ranges for each of the 15 parameters in the Xray/DRR Settings widget, and the Max Iterations slider at the top is for specifying how many tracking optimizations to perform with randomly chosen parameter values within the specified ranges. Each tracking optimization proceeds as if you had selected Solve Current Frame in the Object Tracking widget. It optimizes the selected bones, starting at their current poses, and compares the final poses to the initial ones. The smaller this difference, the better the result. Please see How To: Optimize Xray/DRR Settings for a step-by-step guide on using this widget. |
Graphs |
---|
In the Graphs widget you can display and edit the pose map of one of the selected objects. The graphs show the 6 DOFs of the object's pose at each reporting time. To display the graphs, select an object from the combo box in the upper left corner of the widget. With the checkboxes at the top you can individually control the display of the X, Y, and Z translations and the X, Y, and Z rotations (Euler angles). You can also select the type of spline used to interpolate the poses (GCV or linear), and the cutoff frequency used to smooth the GCV splines.
![]() The circles on the graphs represent the poses in the object’s pose map. The dashed lines are splines (GCV or linear) fit to the circles (control points) and smoothed with the cutoff frequency specified at the top. The gray region behind the curves represents the tracking range specified in the Object Tracking widget. The vertical green line shows the current reporting time. The graphs can be moved left, right, up, and down in the window using the Shift key and the left mouse button. The Y axis can be zoomed in and out using the mouse wheel. Pressing 'r' resets the graph axes to display the splines for all of the selected curves in that graph. Pressing Ctrl + 'r' resets the graph axes to display all of the control points of the selected curves. To move a control point, left-click on it and drag it up or down to a new location (you cannot change the time of a control point). You can select multiple points by pressing Ctrl while left-clicking on the points, or by sweeping a region while pressing the left mouse button. You can then move all selected points together by left-clicking on any of them and dragging up or down. To unselect all control points, left-click away from all points or press Esc. Note: moving control points in the graphs directly modifies the pose map that is loaded into X4D, and there is no undo feature. It does not, however, modify the pose map file; this can only be done by selecting Save Pose Maps from the File menu. Also, whenever a pose map is modified by any method in X4D, it is saved to a backup file in the same folder as the original pose map file. |
Mocap Model | ||||||
---|---|---|---|---|---|---|
|
3D View | ||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
The 3D View window displays a 3D scene representing the X-ray configurations and the tracking objects. The surface models generated from CT data can be manipulated by the user. To do so the model first needs to be selected by left-clicking on it. A trackball consisting of three circles and three lines will be displayed centered in and aligned with the selected object’s local (anatomical) coordinate system. The trackball allows the selected object to be manipulated as follows: ![]()
Note: moving an object this way does not move the other objects with it, even if Apply Joint Constraints is turned on.
|
Xray Windows | ||||||
---|---|---|---|---|---|---|
The Xray windows display the xrays of each view, as well as the DRRs of all selected objects.
|
Output Window |
---|
|