Exporting historical data from ThinkorSwim for external analysis

[Azimuth]

I think this is failing @Azimuth because you are on Windows 7 and that version of PowerShell does not have ::new() for ArrayList. I have a few updated scripts, and plan on updating this post, but I am doubtful this will work on as Windows 7 machine.

@korygill thanks for the reply, what OS are you using?

 

[korygill]

Windows 10, non insiders program, with every update installed.

 

[velocity12345]

Kory,
Thanks for posting this, but I am struggling a bit. I have followed your directions EXACTLY but I am getting an error as shown below.

D:\tos-data> : The term 'D:\tos-data>' is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1

• D:\tos-data> $data = .\Get-OHLC.ps1 .\StrategyReports_SPX_12620.csv
• ~~~~~~~~~~~~

+ CategoryInfo : ObjectNotFound: (D:\tos-data>:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException

Any thoughts?

I think it has to do with the ">" after D:\tos-data

thanks,

 

[korygill]

I see the confusion in my post.
The 'D:\tos-data>' is my prompt. Just use this part:
$data = .\Get-OHLC.ps1 .\StrategyReports_SPX_12620.csv

 

[velocity12345]

Thanks!
I got it to work and really appreciate your help.

 

[IFRPilot541]

Can this be used on other indicators? I am wondering about the AdvancingDecline indicator in particular

 

[korygill]

yes, you can if you get the calculation of that indicator in your strategy code and add the value to the metadata you add to the string along with open/close/etc data.

 

[star900]

these are very smart scrips, I tried it and it works, thanks!. just a question, when I try to copy the Thinkscript code and put it on my account (studies->edit studies-> create) instead of using your Opened shared item it does not work on my chart (the script does not show errors but it shows nothing), how can I fix it? (and prevent it from opening real trades with the AddOrder command) Thanks!

 

[korygill]

@star900, not 100% sure what you are asking. This needs to be a Strategy not a Study so you can have AddOrder show the metadata we are exporting from TOS.

 

[star900]

Thanks, I made a mistake and I put your script in the studies tab instead of in the strategies tab. now it is o.k. thanks again. by the way, is there a way to implement automation in the process of saving the data from Thinkorswim to the desktop? (e.g. every 5 minutes?)

 

[korygill]

Update on 3/5/2020

See my YouTube video with a walk through of how to export historical data and run a PowerShell script to generate a proper csv file. Also includes adding additional data like other indicators to the csv data.

See previous posts for more information as well about exporting historical data.
https://usethinkscript.com/threads/...nkorswim-for-external-analysis.507/post-14606

Video:
Exporting Data With thinkorswim by Kory Gill, @korygill

Strategy Code:

#
# kg_EveryTickSOHLCP
#
# Strategy to capture every bar OHLC and P, the previous close.
# Useful for exporting data from TOS into a CSV file for further processing.
#
# Author: Kory Gill, @korygill
#
declare upper;
declare once_per_bar;

input startTime = 820; #hint startTime: start time in EST 24-hour time
input endTime = 1600; #hint endTime: end time in EST 24-hour time

def adjStartTime = startTime;# - 1;
def adjEndTime = endTime;# - 1;

def agg = GetAggregationPeriod();

# we use a 1 bar offset to get orders to line up, so adjust for that here
def marketOpen = if agg >= AggregationPeriod.DAY then 1 else if SecondsTillTime(adjEndTime) >= 60 and SecondsFromTime(adjStartTime) >= -60 then 1 else 0;

# get indicator values (note, these are optional and were a demonstration to show how you can do these things..
# comment them out if you just want SOHLCP data using the # symbol on front of line.
# if you do, change that name= line below to just (no leading # commend character of course):
# name="SOHLCP|"+GetSymbol()+"|"+open[-1]+"|"+high[-1]+"|"+low[-1]+"|"+close[-1]+"|"+close);
def macdValue = MACD().Value;
def macdAvg = MACD().Avg;


AddOrder(OrderType.BUY_TO_OPEN,
    marketOpen,
    low,
    1,
    Color.White,
    Color.White,
    name="SOHLCP|"+GetSymbol()+"|"+open[-1]+"|"+high[-1]+"|"+low[-1]+"|"+close[-1]+"|"+close+"|"+macdValue[-1]+"|"+macdAvg[-1]);
AddOrder(OrderType.SELL_TO_CLOSE, marketOpen, high, 1, Color.White, Color.White, name="SellClose");

PowerShell code for Get-SOHLCP.ps1:

#
# Get-SOHLCP.ps1
#
# Script to convert a TOS ThinkOrSwim kg_EveryTickSOHLCP strategy report csv file to an object and/or proper csv data file.
#
# Author: Kory Gill, @korygill
# Version: 20200304.2300
#
# Examples
# $x = gci 'D:\Database\MarketData\TOS-StrategyReport-Files-SOHLCP\UseThinkScriptDemo\StrategyReport*' | Select -ExpandProperty FullName | D:\Source\Repos\technical-analysis-dev\dev\Get-SOHLCP.ps1 -InformationAction Continue
#
# Get the symbol back:
# $m = 'SOHLCP-(~ES.XCME)(1.21.20 5.00 AM - 1.27.20 1.14 PM).csv' | Select-String -Pattern '\((.*?)\)'  -AllMatches; $m.Matches[0].Groups[1].Value -replace '~', '/'
#
[CmdletBinding()]
param (
    [Parameter(Mandatory=$true, ValueFromPipeline=$true)]
    [string[]]
    $File,

    [switch]
    $ExportAsObject
)

Begin {
    function Convert-CurrencyStringToDecimal ([string]$input)
    {
        ((($input -replace '\$') -replace '[)]') -replace '\(', '-') -replace '[^-0-9.]'
    }

    $global:sohlcpAllData = New-Object System.Collections.Generic.List[PSCustomObject]
}

Process {
    foreach ($f in $File)
    {
        if (-not (Test-Path $f))
        {
            throw "Cannot open file '$f'."
        }

        Write-Information "Processing file: '$f'."

        # read csv file
        $content = Get-Content -Path $f

        # generate filename
        $csvSymbol = ($content[1] -split 'Symbol: ')[1] -replace '/', '~' -replace ':', '.'
        $csvWorkTime = ($content[2] -split 'Work Time: ')[1] -replace '/', '.' -replace ':', '.'
        $outFile = 'SOHLCP-(' + $csvSymbol +')('+ $csvWorkTime + ')'

        # find the lines that contain price information
        $csvdata = $content | ? {$_ -match ";.*;"} | ConvertFrom-Csv -Delimiter ';'

        # filter just the lines with (OHLC on them and make into CSV structure
        $data = $csvData | ? {$_ -match "\(SOHLCP"}

        $sohlcpFileData = New-Object System.Collections.Generic.List[PSCustomObject]

        foreach ($item in $data)
        {
            # capture the OHLC data
            $null = $item.Strategy -match "\(SOHLCP\|(.*)\)"
            $v = $Matches[1] -split '\|'

            $symbol = $v[0]
            $open  = $v[1] | Convert-CurrencyStringToDecimal
            $high  = $v[2] | Convert-CurrencyStringToDecimal
            $low   = $v[3] | Convert-CurrencyStringToDecimal
            $close = $v[4] | Convert-CurrencyStringToDecimal
            $prevClose = $v[5] | Convert-CurrencyStringToDecimal
            $macdValue = $v[6] | Convert-CurrencyStringToDecimal
            $macdAvg = $v[7] | Convert-CurrencyStringToDecimal

            $sohlcpData = [PSCustomObject]@{
                'Symbol' = $symbol
                'DateTime' = ([datetime]::Parse($item.'Date/Time'))
                'Open' = [decimal]$open
                'High' = [decimal]$high
                'Low' = [decimal]$low
                'Close' = [decimal]$close
                'PrevClose' = [decimal]$prevClose
                'macdValue' = [decimal]$macdValue
                'macdAvg' = [decimal]$macdAvg
                }

            # add to our $sohlcpData array
            $null = $sohlcpFileData.Add($sohlcpData)
            $null = $sohlcpAllData.Add($sohlcpData)
        }

        # save to file
        $sohlcpFileData | Export-Csv -Path (Join-Path (Split-Path -Path $f -Parent) ($outFile + '.csv')) -Force -NoTypeInformation -Encoding ASCII
    }
}

End {
    if ($ExportAsObject)
    {
        # helpful message to show caller our output variable
        Write-Information "Out Data $($sohlcpAllData.Count) items (exported as `$sohlcpAllData)"
    }
    else
    {
        # don't show any extraneous output, and just return the data to the pipeline
        return $sohlcpAllData
    }
}

 

[korygill]

It's probably worth calling out like I show in the video, that you can export multiple symbols and data from TOS, then convert them to "SOHLCP" csv data files in batches. This should help everyone doing this across multiple symbols be more efficient.

 

[TB93_Trader]

Followed the steps in the Video(thanks for this!), but for some reason I get an empty SOHLCP.csv File. Anyone an Idea what might be the reason for this?

 

[PSK]

Is it possible to download the data from a chart like price, time and all indicator values into a csv or excel file?

 

[Topas]

Historical Data is downloadable thru MT4 platform, i am sure NOT about TOS

 

[PSK]

Yes. Tradestation also does this. I have some indicators that work only in TOS. Rather than trying to program in easy language thought it would be easier if there is a way of downloading into excel and do some analysis.

 

[jeffquick]

Followed the steps in the Video(thanks for this!), but for some reason I get an empty SOHLCP.csv File. Anyone an Idea what might be the reason for this?

If you've changed the name of the export in Thinkscript the PS script may not be able to find it and it will output a blank csv.

 

[Yghanian]

Hi korygill, is there a version of the shell script that would work on Mac? I am able to download the CSV file but I am not a Windows user and don't understand the shell script. appreciate any help you can provide. Thanks!

 

[TradeCloserApp]

@korygill Regarding exporting historical daily data for Futures.

Is it possible to export daily data based only on pre-market hours (6pm est to 930am est) ? Or regular trading hours (930am est to 5pm est) ?

 

[korygill]

@korygill Regarding exporting historical daily data for Futures.

Is it possible to export daily data based only on pre-market hours (6pm est to 930am est) ? Or regular trading hours (930am est to 5pm est) ?

The script already includes a StartTime and EndTime parameter to control what bars to gather data for. That should do what you are asking.