Differences

This shows you the differences between two versions of the page.

--- inspect3d:tutorials:clean_your_data [2024/06/19 12:45] – sgranger
+++ inspect3d:tutorials:clean_your_data [2024/07/12 13:26] (current) – removed sgranger
@@ Line 1: / Line 1: @@
-{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{this tutorial will show you how to use inspect3d as a data cleaning, or quality assurance, tool. you will learn how to check for and correct faulty force assignments from all files at once, thereby removing any artifacts from the data. this is particularly useful for lab managers (or supervisors) who may not be familiar with the raw data, or collaborators that were not involved in the collection process.
-===== data =====
-this tutorial uses overground walking data from four subjects. the subjects walked at three different speeds; slow, normal and fast. the data was analyzed using a pipeline which included an automatic gait event detection command. [[visual3d:documentation:visual3d_signal_types:link_model_based_data_type|link model based items]] (including right and left ground reaction forces) were created using a pipeline. this data is reflective of a typical outcome from batch data analysis in [[visual3d:overview:visual3d_overview|visual3d]].
-these data files can be downloaded using this link: [[https://www.has-motion.com/download/examples/foursubjectswalkingdata.zip|four subjects walking data set]].
-===== loading the library =====
-inspect3d_four_subjects_loaded.png
-. click i3dloadlibrary.png **load library** in the [[inspect3d:documentation:dialogs:toolbar|toolbar]] to open the load library dialog.
-. click **browse** and select the folder where the cmz files are stored.
-. click **load** button to import the data.
-this step selects the path to the data you are using. if you intend to modify the data, e.g., by correcting invalid assignments, then your data must be saved in a directory where you have read and write access. this tutorial uses a folder located on the desktop. the data folder must contain the .cmx file associated with every [[visual3d:documentation:definitions:file_formats:cmz_format_|.cmz]] file, since without both file types inspect3d will be unable to load the files. each .cmz file has multiple .c3d files with the different walking speeds identified using tags.
-note: after the library path is set, files from the selected folder and sub-folders will be loaded and parsed. inspect3d will show a summary of the loaded files and what they contain in the load library dialog. examine this summary to make sure that the correct files have loaded.
-===== defining queries and calculating groups =====
-. click on the i3dgroups.png **query definitions** icon on the [[inspect3d:documentation:dialogs:toolbar|toolbar]] and then click on i3dgroups.png **query definitions** to open the query builder dialog.
-. to create a new query definition, click the actionadd48x48.png green addition button in the empty **queries** list.
-.1. type grf in the **query name** text box in top-right and click **save**.
-inspect3d_creating_query_(four_subjects_walking).png
-. while the grf query is selected, click the actionadd48x48.png green addition button in the empty **conditions** list.
-.1. type r_grf in the **condition name** text box in the top-right.
-.2. there are now three tabs that need to be completed in order to define the sub-group.
-.2.1. **signals**: this tab allows the user to select specific signals according to their names in the [[visual3d:documentation:visual3d_signal_types:data_tree_summary_|visual3d data tree]]. for this tutorial, select the signal: type - link_model_based, folder - original, name - r_grf, and component - z.
-inspect3d_signal_tab_(four_subjects_walking).png
-.2.2. **events**: this tab allows the user to specify the desired event sequence to extract data from. for instance, the right gait cycle could be extracted using the event sequences ron, roff or rhs, rhs. for this tutorial select the ron and roff events and leave the default normalization values of 101 for the number of points and cubic for the spline type.
-inspect3d_events_tab_(four_subjects_walking).png
-.2.3. **refinements**: this tab allows the user to refine their selected signal using two separate and distinct methods: according to the signals contained within the [[visual3d:documentation:definitions:file_formats:cmo_format|cmo files]], or according to tags. for this tutorial, refine using tags by leaving the **use and logic** checkbox unchecked and selecting the slow, normal, and fast tags.
-inspect3d_refinement_tab_(four_subjects_walking).png
-.3. click **save** to create the condition.
-. for this tutorial, define a second condition within the same group to account for the left side. this can be done by modifying the existing sub-group:
-.1. while the grf group is selected, click the actionadd48x48.png green addition button in the **sub-groups** list and type l_grf in the **sub-group name** text box.
-.1.1. in the signals tab, change the name to l_grf and leave the other parameters as they are.
-.1.2. in the events tab, select the event sequence lon and loff.
-.1.3. leave the refinements tab as it was.
-.2. click **save** to create this second condition.
-inspect3d_two_conditions_(four_subjects_walking).png
-you should now see one query (grf) in the **queries** list and two conditions (r_grf and l_grf) in the **conditions** list. at this point in the tutorial these definitions have been created but they have not been applied to the signals in the loaded library. to do this, click on **calculate all queries** (or **calculate selected queries**, since there is only one group in this tutorial).
-once the group is calculated you can close the query builder dialog, it is now time to examine the results visually by plotting them.
-===== visualizing and exploring your data =====
-. with the groups calculated, the specific traces that have been extracted from the loaded library's .c3d files can be explored. the group name defined in the previous step is shown in the **groups** section (see 1). when the groups are clicked, the workspaces associated with the selected group(s) are shown (see 2). the icon beside a workspace indicates its colour in the **queried data** plots. a check mark indicates that the workspace has had none of its traces excluded; an x indicates that at least one trace has been excluded.
-inspect3d_visualizing_four_subjects_walking.png
-. select all of the workspaces either by using the **select all workspaces** check box or by selecting them all in the **workspaces** list. then ensure that the **plot all sequences** check box is checked and click **refresh plot**.
-. when inspect3d tries to plot [[inspect3d:documentation:troubleshooting#data_missing_graph|missing data]], your y-axis will go to -999 999.0 and the axis scale will cause the data to be compressed into lines. to avoid this, open the i3dshowoptions.png **[[inspect3d:documentation:dialogs:load_library_dialogue#plotting_options|options]]** dialog and deselect the **show data not found** checkbox in the **plotting options** section.
-.1. update your plot by clicking **refresh plot**; it should now resemble the graph below.
-inspect3d_queried_data_(four_subjects_walking).png
-. use your cursor to select only lines on the graph that you wish to inspect. click on single traces in order to examine individual curves without the other curves 'polluting' the graph. when a trace is clicked, a tooltip will appear to describe the source of the trace in terms of both inspect3d's groups and workspaces and the underlying .c3d files.
-inspect3d_select_trace_(four_subjects_walking).png
-===== verifying raw data and excluding traces =====
-once the queried traces have been plotted, it is now possible to begin cleaning the data. in this example, in particular, it is evident that some incorrect forces assignments exist.
-. in order to exclude these incorrect assignments, select any lines that may be a result of a 'bad' event. for the obvious 'bad' events, it is possible to click and drag over multiple traces on the plot to select them simultaneously.
-. right-click and navigate to exclude, at the very bottom, and click exclude trace (raw data).
-. verify that your desired traces have been properly excluded.
-.1 when traces are excluded, two notable differences should appear in the queried data subwindow. first, the previously selected, but now excluded data should not be visible on the graph. second, the **workspaces** widget will indicate when traces have been excluded. specifically, an x should replace the original check mark beside each workspace where signal traces have been excluded, two horizontal bars should replace the check mark beside each .c3d file with excluded traces, and a red x should replace the checkmark beside each excluded trace. additionally, the fraction of traces excluded for each workspace is displayed on the right-hand side of the **workspaces** widget: in this case all of the workspaces contained poor quality events. if unintentional exclusion occurred for a subject, you can right click on the subject and select 're-include data'.
-inspect3d_excluded_traces_(four_subjects_walking).png
-. if you want to visualize the data you have excluded alongside the remaining data, go to the [[inspect3d:documentation:dialogs:load_library_dialogue|show options]] dialog and select the "show excluded data" option. you can choose to use a specific line style for excluded data and specify this style in the [[inspect3d:documentation:dialogs:load_library_dialogue#line_style|line styles]] tab.
-inspect3d_excluded_traces_shown_(four_subjects_walking).png
-.
-inspect3d_animate_trace_(four_subjects_walking).png
-traces can also be examined more closely before deciding to include or exclude them.
-.1 select the traces of interest by clicking and dragging on the plot.
-.2 click the i3dshowanimation.png **show animation** button.
-.3 the show animation dialog can display animations for those selected traces with enough data in the .cmz file to permit a recreation. each of the selected traces can be selected from the drop-down box near the top of the dialog. this view can be used to determine what (if any) issue exists in the data and whether this merits exclusions. excluded traces are removed from the drop-down box within the dialog as well as the queried data subwindow.
-. once all of the desired exclusions have been made, the original data is ready to be updated.
-===== updating the original data =====
-.
-inspect3d_update_cmz_(four_subjects_walking).png
-now that the incorrect force assignments have been identified in our data set, it is possible to update the original [[visual3d:documentation:definitions:cmo_library_|cmz]] data by clicking the i3dupdatecmoswithbadevents.png **updatecmzs** item in the application's tool bar. this opens a dialog that provides different options for processing excluded data.
-. the **excluded traces** options allow the user to add a bad event to the excluded traces. this option is helpful if the intention is to [[visual3d:documentation:definitions:tag|tag]] the bad data, re-open the original data, and make manual corrections one trace at a time.
-. alternatively, the **update force assignments** option actually removes the force assignment from the original data. this option should be used if the user does not intend to revisit the original data after the modifications have been made (note: this second option changes the underlying data. re-performing this tutorial with the newly saved data will produce different results).
-. select both **add event to exclude signals** with the default event name "bad" and **update force assignments using excluded query data from:** with "grf". then click **update cmos**.
-. this results in inspect3d opening each .cmz file in the background, adding a bad event for the excluded traces, and removing the force assignments. this process can take some time depending on the .cmz size, but you can further explore your data (or get a coffee) as it is occurs. when the update is complete, the check mark icons will return beside each workspace in the **workspaces** widget and the original data will be modified.
-===== recap =====
-in this tutorial you learned how to use inspect3d as a data cleaning tool for quality assurance purposes. here, the process of loading data, defining queries, excluding data, and updating the original data are described.
-===== navigate tutorials =====
-previous: [[inspect3d:tutorials:load_and_view_data|tutorial 1: load and view data]]
-**inspect3d getting started tutorials**
-next: [[inspect3d:tutorials:perform_principal_component_analysis|tutorial 3: perform principal component analysis]]
-}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}