Batch Command Line Processing Tutorial

In this tutorial, we will guide you through running Sift on the command line. You'll learn how to load a library, execute queries, run PCA (Principal Component Analysis), and use it for outlier detection. After walking through each command step by step, we’ll create a batch script to automate these tasks. We'll also include some logic to further streamline the process. To get started, download the sample data folder. automation_tutorial.zip

Running on Command Line

First, we need to run Sift via the command line. On your keyboard, press Windows Key + R, type “cmd” into the run window, and press Enter to open a command prompt.
Next, navigate to Sift's installation directory.
To launch Sift, call the .exe file and pass the -NoGUI argument
This will open new console where we can interact with Sift
In this console, load a library by entering the command: -LoadLib directory “Path to downloaded library”
Next load and run a predefined query by entering: -LoadQuery file “Path to query definition file”
Select all signals by running -SelectSignals (Ensure all signal groups and work-spaces have been selected)
Run a PCA named “Outlier_PCA” -RunPCA name “Outlier_PCA”
Now, let's perform the first round of outlier detection. Run a Local Outlier Factor (LOF) test with the command: -RunLOF grouping “group” autoExclude This will find any group level outliers and exclude them automatically
To export the results of this LOF test, run: -ExportPCA file “Path to exported txt file” exportFormat “transposed” LOF LOFThrehsold
Next, run another LOF test, this time on workspaces: -RunLOF grouping “workspace” autoExclude
Export the results of this LOF test as well: -ExportPCA file “Path to exported txt file” exportFormat “transposed” LOF LOFThreshold
Finally, save the project to allow for future review in the GUI, if needed: -SaveProject file “Path to saved project”

Now that processing is complete, we have two exports detailing group and workspace-level outliers, along with a saved project file that excludes all detected outliers. This can serve as a baseline for future processing.

Batch Processing

Now that we have a clear understanding of our baseline processing, it's time to automate the workflow!

The Scenario

To effectively automate a process, we need to understand all the details. Let’s create a scenario:
Imagine we're working with a baseball team, and each day at practice, we gather data that needs to be processed. This data is stored in date-labeled folders within a root directory, and inside each folder are three trials containing relevant C3D files.

The Plan

Now that we know what we're dealing with, we can plan how to build our batch script. First, we’ll need to check each folder within the root directory. If a folder doesn’t already contain our exports (such as .txt files and .i3d files), we know that folder still needs to be processed. Below is a batch script that accomplishes this. You’ll need to modify the path in targetDir, as well as the path pointing to Sift.exe, to match your computer's setup. Afterward, save the file as a .bat. To do this, create a .txt file and rename it, changing the extension to .bat.

@echo off
setlocal

:: Set the target directory where the practice data is stored
set "targetDir=C:\Users\shane\OneDrive\Documents\practice"

:: Loop through each folder (%%d) in the target directory
for /D %%d in ("%targetDir%\*") do (
    
    :: Reset the flag to check if an i3d file exists
    set "i3dFileFound="
    
    :: Loop through each .i3d file in the current folder
    for %%f in ("%%d\*.i3d") do (
        
        :: If an i3d file is found, set the flag to true
        if exist "%%f" (
            set "i3dFileFound=true"
        )
    )
    
    :: If no i3d file was found, process the folder
    if not defined i3dFileFound (
        
        :: Set the folder path that needs processing
        set "folderWithoutI3d=%%d"
        
        :: Run Sift with the necessary commands to process the data
        "C:\Program Files (x86)\Sift\Sift.exe" -NoGui -LoadLib directory "%%d" -LoadQuery file "%%d\query_definition.q3d" -selectSignals -RunPCA name "Outlier_PCA" -RunLOF grouping "group" autoExclude -ExportPCA file "%%d\Group Outliers.txt" exportFormat "transposed" lof lofThreshold -RunLOF grouping "workspace" autoExclude -ExportPCA file "%%d\Workspace Outliers.txt" exportFormat "transposed" lof lofThreshold -SaveProject file "%%d\Outliers.i3d" -Exit
    )
)

:: End the script and clean up
endlocal

The Execution

Now that we have our script ready, running the processing is as simple as double-clicking the .bat file. But we can take the automation a step further! By using Windows Task Scheduler, we can set the batch script to run at a specific time each day automatically.

Open the start menu and type Task Scheduler into the search bar
Open Task Scheduler from the search results.
In Task Scheduler, select Create Basic Task from the top-right of the screen.
In the popup dialog, enter a meaningful name (in this case, we used “Baseline Processing”) and press Next.
On the next screen, select Daily and press Next.
Now, choose the time of day you'd like the task to run. We selected 12:00 AM.
Ensure Start a Program is selected on the next screen and press Next.
Browse and select the batch script you created earlier.
On the final screen, review your settings and press Finish.
Lastly, check the list of existing tasks to confirm it was created successfully.