Exporting historical data from ThinkorSwim for external analysis

Join useThinkScript to post your question to a community of 21,000+ developers and traders.

Kory,
Thanks for posting this, but I am struggling a bit. I have followed your directions EXACTLY but I am getting an error as shown below.

D:\tos-data> : The term 'D:\tos-data>' is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
  • D:\tos-data> $data = .\Get-OHLC.ps1 .\StrategyReports_SPX_12620.csv
  • ~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (D:\tos-data>:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException


Any thoughts?

I think it has to do with the ">" after D:\tos-data

thanks,
 
I see the confusion in my post.
The 'D:\tos-data>' is my prompt. Just use this part:
$data = .\Get-OHLC.ps1 .\StrategyReports_SPX_12620.csv
 
yes, you can if you get the calculation of that indicator in your strategy code and add the value to the metadata you add to the string along with open/close/etc data.
 
these are very smart scrips, I tried it and it works, thanks!. just a question, when I try to copy the Thinkscript code and put it on my account (studies->edit studies-> create) instead of using your Opened shared item it does not work on my chart (the script does not show errors but it shows nothing), how can I fix it? (and prevent it from opening real trades with the AddOrder command) Thanks!
 
@star900, not 100% sure what you are asking. This needs to be a Strategy not a Study so you can have AddOrder show the metadata we are exporting from TOS.
 
Thanks, I made a mistake and I put your script in the studies tab instead of in the strategies tab. now it is o.k. thanks again. by the way, is there a way to implement automation in the process of saving the data from Thinkorswim to the desktop? (e.g. every 5 minutes?)
 
Update on 3/5/2020

See my YouTube video with a walk through of how to export historical data and run a PowerShell script to generate a proper csv file. Also includes adding additional data like other indicators to the csv data.

See previous posts for more information as well about exporting historical data.
https://usethinkscript.com/threads/...nkorswim-for-external-analysis.507/post-14606

Video:
Exporting Data With thinkorswim by Kory Gill, @korygill

Strategy Code:

Code:
#
# kg_EveryTickSOHLCP
#
# Strategy to capture every bar OHLC and P, the previous close.
# Useful for exporting data from TOS into a CSV file for further processing.
#
# Author: Kory Gill, @korygill
#
declare upper;
declare once_per_bar;

input startTime = 820; #hint startTime: start time in EST 24-hour time
input endTime = 1600; #hint endTime: end time in EST 24-hour time

def adjStartTime = startTime;# - 1;
def adjEndTime = endTime;# - 1;

def agg = GetAggregationPeriod();

# we use a 1 bar offset to get orders to line up, so adjust for that here
def marketOpen = if agg >= AggregationPeriod.DAY then 1 else if SecondsTillTime(adjEndTime) >= 60 and SecondsFromTime(adjStartTime) >= -60 then 1 else 0;

# get indicator values (note, these are optional and were a demonstration to show how you can do these things..
# comment them out if you just want SOHLCP data using the # symbol on front of line.
# if you do, change that name= line below to just (no leading # commend character of course):
# name="SOHLCP|"+GetSymbol()+"|"+open[-1]+"|"+high[-1]+"|"+low[-1]+"|"+close[-1]+"|"+close);
def macdValue = MACD().Value;
def macdAvg = MACD().Avg;


AddOrder(OrderType.BUY_TO_OPEN,
    marketOpen,
    low,
    1,
    Color.White,
    Color.White,
    name="SOHLCP|"+GetSymbol()+"|"+open[-1]+"|"+high[-1]+"|"+low[-1]+"|"+close[-1]+"|"+close+"|"+macdValue[-1]+"|"+macdAvg[-1]);
AddOrder(OrderType.SELL_TO_CLOSE, marketOpen, high, 1, Color.White, Color.White, name="SellClose");

PowerShell code for Get-SOHLCP.ps1:

Code:
#
# Get-SOHLCP.ps1
#
# Script to convert a TOS ThinkOrSwim kg_EveryTickSOHLCP strategy report csv file to an object and/or proper csv data file.
#
# Author: Kory Gill, @korygill
# Version: 20200304.2300
#
# Examples
# $x = gci 'D:\Database\MarketData\TOS-StrategyReport-Files-SOHLCP\UseThinkScriptDemo\StrategyReport*' | Select -ExpandProperty FullName | D:\Source\Repos\technical-analysis-dev\dev\Get-SOHLCP.ps1 -InformationAction Continue
#
# Get the symbol back:
# $m = 'SOHLCP-(~ES.XCME)(1.21.20 5.00 AM - 1.27.20 1.14 PM).csv' | Select-String -Pattern '\((.*?)\)'  -AllMatches; $m.Matches[0].Groups[1].Value -replace '~', '/'
#
[CmdletBinding()]
param (
    [Parameter(Mandatory=$true, ValueFromPipeline=$true)]
    [string[]]
    $File,

    [switch]
    $ExportAsObject
)

Begin {
    function Convert-CurrencyStringToDecimal ([string]$input)
    {
        ((($input -replace '\$') -replace '[)]') -replace '\(', '-') -replace '[^-0-9.]'
    }

    $global:sohlcpAllData = New-Object System.Collections.Generic.List[PSCustomObject]
}

Process {
    foreach ($f in $File)
    {
        if (-not (Test-Path $f))
        {
            throw "Cannot open file '$f'."
        }

        Write-Information "Processing file: '$f'."

        # read csv file
        $content = Get-Content -Path $f

        # generate filename
        $csvSymbol = ($content[1] -split 'Symbol: ')[1] -replace '/', '~' -replace ':', '.'
        $csvWorkTime = ($content[2] -split 'Work Time: ')[1] -replace '/', '.' -replace ':', '.'
        $outFile = 'SOHLCP-(' + $csvSymbol +')('+ $csvWorkTime + ')'

        # find the lines that contain price information
        $csvdata = $content | ? {$_ -match ";.*;"} | ConvertFrom-Csv -Delimiter ';'

        # filter just the lines with (OHLC on them and make into CSV structure
        $data = $csvData | ? {$_ -match "\(SOHLCP"}

        $sohlcpFileData = New-Object System.Collections.Generic.List[PSCustomObject]

        foreach ($item in $data)
        {
            # capture the OHLC data
            $null = $item.Strategy -match "\(SOHLCP\|(.*)\)"
            $v = $Matches[1] -split '\|'

            $symbol = $v[0]
            $open  = $v[1] | Convert-CurrencyStringToDecimal
            $high  = $v[2] | Convert-CurrencyStringToDecimal
            $low   = $v[3] | Convert-CurrencyStringToDecimal
            $close = $v[4] | Convert-CurrencyStringToDecimal
            $prevClose = $v[5] | Convert-CurrencyStringToDecimal
            $macdValue = $v[6] | Convert-CurrencyStringToDecimal
            $macdAvg = $v[7] | Convert-CurrencyStringToDecimal

            $sohlcpData = [PSCustomObject]@{
                'Symbol' = $symbol
                'DateTime' = ([datetime]::Parse($item.'Date/Time'))
                'Open' = [decimal]$open
                'High' = [decimal]$high
                'Low' = [decimal]$low
                'Close' = [decimal]$close
                'PrevClose' = [decimal]$prevClose
                'macdValue' = [decimal]$macdValue
                'macdAvg' = [decimal]$macdAvg
                }

            # add to our $sohlcpData array
            $null = $sohlcpFileData.Add($sohlcpData)
            $null = $sohlcpAllData.Add($sohlcpData)
        }

        # save to file
        $sohlcpFileData | Export-Csv -Path (Join-Path (Split-Path -Path $f -Parent) ($outFile + '.csv')) -Force -NoTypeInformation -Encoding ASCII
    }
}

End {
    if ($ExportAsObject)
    {
        # helpful message to show caller our output variable
        Write-Information "Out Data $($sohlcpAllData.Count) items (exported as `$sohlcpAllData)"
    }
    else
    {
        # don't show any extraneous output, and just return the data to the pipeline
        return $sohlcpAllData
    }
}
 
Last edited:
It's probably worth calling out like I show in the video, that you can export multiple symbols and data from TOS, then convert them to "SOHLCP" csv data files in batches. This should help everyone doing this across multiple symbols be more efficient.
 
Followed the steps in the Video(thanks for this!), but for some reason I get an empty SOHLCP.csv File. Anyone an Idea what might be the reason for this?
 
Is it possible to download the data from a chart like price, time and all indicator values into a csv or excel file?
 
Yes. Tradestation also does this. I have some indicators that work only in TOS. Rather than trying to program in easy language thought it would be easier if there is a way of downloading into excel and do some analysis.
 
Followed the steps in the Video(thanks for this!), but for some reason I get an empty SOHLCP.csv File. Anyone an Idea what might be the reason for this?
If you've changed the name of the export in Thinkscript the PS script may not be able to find it and it will output a blank csv.
 
Hi korygill, is there a version of the shell script that would work on Mac? I am able to download the CSV file but I am not a Windows user and don't understand the shell script. appreciate any help you can provide. Thanks!
 
@korygill Regarding exporting historical daily data for Futures.

Is it possible to export daily data based only on pre-market hours (6pm est to 930am est) ? Or regular trading hours (930am est to 5pm est) ?
 
@korygill Regarding exporting historical daily data for Futures.

Is it possible to export daily data based only on pre-market hours (6pm est to 930am est) ? Or regular trading hours (930am est to 5pm est) ?

The script already includes a StartTime and EndTime parameter to control what bars to gather data for. That should do what you are asking.
 

Similar threads

Not the exact question you're looking for?

Start a new thread and receive assistance from our community.

87k+ Posts
354 Online
Create Post

Similar threads

Similar threads

The Market Trading Game Changer

Join 2,500+ subscribers inside the useThinkScript VIP Membership Club
  • Exclusive indicators
  • Proven strategies & setups
  • Private Discord community
  • ‘Buy The Dip’ signal alerts
  • Exclusive members-only content
  • Add-ons and resources
  • 1 full year of unlimited support

Frequently Asked Questions

What is useThinkScript?

useThinkScript is the #1 community of stock market investors using indicators and other tools to power their trading strategies. Traders of all skill levels use our forums to learn about scripting and indicators, help each other, and discover new ways to gain an edge in the markets.

How do I get started?

We get it. Our forum can be intimidating, if not overwhelming. With thousands of topics, tens of thousands of posts, our community has created an incredibly deep knowledge base for stock traders. No one can ever exhaust every resource provided on our site.

If you are new, or just looking for guidance, here are some helpful links to get you started.

What are the benefits of VIP Membership?
VIP members get exclusive access to these proven and tested premium indicators: Buy the Dip, Advanced Market Moves 2.0, Take Profit, and Volatility Trading Range. In addition, VIP members get access to over 50 VIP-only custom indicators, add-ons, and strategies, private VIP-only forums, private Discord channel to discuss trades and strategies in real-time, customer support, trade alerts, and much more. Learn all about VIP membership here.
How can I access the premium indicators?
To access the premium indicators, which are plug and play ready, sign up for VIP membership here.
Back
Top