Getting Data From An Illiquid Asset

[StoneMan]

I'm working with options data which can be quite illiquid. On certain instruments you may get only a few ticks per day or even zero. Because of this bars based look back strategy will not work due to the chart being incomplete.

My workaround to this is using the time of day to get the data at the right time. I'm reaching out because I am not sure if thinkscript is capable of returning a bar based on the time.

The logic looks something like this:

If data is present in the time slot 1 on the current session get data, if not set relevant values to zero.

If data is present in the time slot 2 ob the current sesaion get data, if not set relevant values to zero.

.
.
.

I also want to know what time slot is currently active in a given session. The trade day is from 930 am to 4pm. That is 13 30 minute sessions over 6.5 hours. This is how I am planning on discritizing the trading day. I need a variable that can return 1-13 corresponding with the proper time slot. That way averages can be calculated no matter when you run the code. After the market closes that value would return 13.

Any insight would be greatly appreciated!

 

[Joshua]

Otherwise, TOS is still wonky right now, I'm still getting no time and sales on several futures contracts, some charts are stuck in place, but the active trader ladder is in motion. Its taking quite awhile to load charts as well. Its also telling me that some optionable symbols have no options, and then later, the chain loads in. Now is probably not the best time for this, I'll check back tomorrow.

 

[StoneMan]

@Joshua My bad, I should have been more specific by saying volume to tic ratio. What I am looking at is ARNC at 11:52:31 19 MAY 23 22 Put. The volume to tic ratio does not correctly the highest level for anything less than 30 minutes, where it returns 797.67. All time periods less than 30 minutes return a value of 10 for the volume to tic ratio at 15:54.

But you are correct that these columns and the way TOS handles this data is super wonky and makes it tough to work with. Maybe the script needs a way to prioritize a time when its given equal values to display at different times, and, due to options lacking liquidity, maybe an if statement for when a time slot returns NaN could could make the script more robust. Just speculating here though.

 

[StoneMan]

@Joshua I wanted to provide photos of some of the bugs I have ran into while running the code to so you can see yourself and hopefully help with debugging.

Maybe this post I made about working with options data and the use of the if IsNaN(....) code blocks in the attached code can be of assistance.

https://usethinkscript.com/threads/...-options-data-on-charts-in-thinkorswim.15242/

I set the scan to 1 minute today and some of the tic time values are returning 15:90 and 15:70

The next few images are for the ARNC discrepancy I was describing. I want to show my screen grabs.

Here is the ARNC TAS I was referring to

Here is the 5 min scan. I ran your original column along with the new one. It looks like there was some problems picking up the LZ 300 volume as well.

Here is the hour scan where the ARNC block is picked up. It still doesn't pick up the LZ volume though.

 

[Joshua]

Good timing. Having a specific option to check it against was a big help, but I am still not one hundred percent sure. Try this and let me know how it goes.

def TickCount;
def VolRatio;
    if GetDay() == GetLastDay() {
        TickCount = Tick_Count;
        VolRatio = Volume / Tick_Count;
    } else {
        TickCount = 0;
        VolRatio = 0;       
    }
;
def HighTick;
def TickTime;
def TickHour;
def TickMinute;
    if TickCount > HighTick[1] {
        HighTick = TickCount;
        TickTime = secondsfromTime(0000);
        TickHour = Floor(TickTime / 60 / 60);
        TickMinute = Floor((TickTime / 60 / 60 - TickHour) * 60);
    } else {
        HighTick = HighTick[1];
        TickTime = TickTime[1];
        TickHour = TickHour[1];
        TickMinute = TickMinute[1];
    }
;
def HighVol;
def VolTime;
def VolHour;
def VolMinute;
def V; def T;
    if VolRatio > HighVol[1] {
        HighVol = VolRatio;
        VolTime = secondsfromTime(0000);
        VolHour = Floor(VolTime / 60 / 60);
        VolMinute = Floor((VolTime / 60 / 60 - VolHour) * 60);
        v = Volume;
        t = Tick_Count;
    } else {
        HighVol = HighVol[1];
        VolTime = VolTime[1];
        VolHour = VolHour[1];
        VolMinute = VolMinute[1];
        v = V[1];
        t = T[1];
    }
;
addLabel(Yes,
    " "
    + asPrice(TickHour)
    + ":" + asPrice(TickMinute)
    + " (" + asPrice(HighTick) + ") - "
    + asPrice(VolHour)
    + ":" + asPrice(VolMinute)
    + " (" + asPrice(Round(HighVol,2)) + ") - "
    + asprice(V) + " / " + asPrice(t)
    + " "
);

There might be one more issue though, rather systemic, dealing with how thinkscript handles the absolute beginning of the existing data once you've introduce prior offsets. Option data is, unfortunately, often right up against the hard-wall.

 

[StoneMan]

@Joshua I see what you mean by think script struggling to handle the absolute beginning of the existing data once you've introduce prior offsets. Your attached image seems to suggest that data exists at one point as zero but not at all at the other point.

I remember I once had an issue in NinjaScript where I was trying to draw some lines using the OnRender method to display data from a list generated from the OnBarUpdate method. The problem was that the OnRender method and the OnBarUpdate updated at different intervals. So the lines could never display properly. To remedy the situation and make the update periods sync up I had the OnBarUpdate method fill the list array with zeros at every point there was no data and had the OnRender method draw everything that wasn't zero. This allowed the polling period to "sync up" between the two methods and have the data properly display.

In our case it appears the "desync" is with the janky data and your logic for dividing up the trading day. Which is correct but not necessarily reflected in the data set.

What I am wondering, and this is just speculation, is if a similar solution can be applied here for time slots where data does not exist and "desyncs". We force those values to be zero, sync up the polling periods, and the rest of our logic now works properly.

For this to work we may need to individually account for all the times the user can set in the scan list column. Also not sure about this and speculating.

I am at work right now and will run the code this evening when I get back and hunt for bugs and report back. Thanks so much for all your help with this!

 

[Joshua]

On option charts, plot statements seem to reject a number of bars, equivalent to the largest prior offset present in the script, at the beginning of the chart. In most other cases, this isn't a problem, unless you're dealing with something like stock that just recently had its IPO, but the data for many illiquid options often starts "today."

It might just be a visual thing with the plot though, chart bubbles appear to work for some weird reason. Perhaps the variables still hold the proper values. But, when you translate that to a column, its very difficult to truly see what is transpiring with only text based output.

If all else fails, I could probably harvest the data by waiting for the end of the chart, and folding backwards from the right edge, rather than tracking from the left edge onwards with prior offsets.

 

[StoneMan]

@Joshua I think folding backwards from the right edge could be a good idea. The fold function would run backwards until the start of the day is reached or no previous data exists if the instrument is that illiquid.

In the case of that extreme illiquidity it may be necessary to count how many times the fold function has ran its loop before there is no data left to analyze. You know how many times it should loop with a complete data set so you can take the difference to determine a proper time slot.

Maybe it would look something like:

Establish the time interval from the users chosen aggregatetion period so you know how many the trading day consists of.

Establish what time you are running the scan at using GetTime so you know what time interval the loop will start at.

Establish the fold count variable so you know how many times the loop gets run.

Determine how many times the loop would need to run with a complete data set for the given time interval at the current GetTime. This can be compared to the fold count variable.

For all data between start day time stamp and end day time stamp fold through time interval until start day time or IsNaN and return highest value of whatever you are looking for.

Compare the high value times to the start and end day time stamps to help determine what the time actually was.

.
.

Doing this time interval related math would probably just be a start to identifying high values and displaying them when given an incomplete data set where barcounts become unreliable.

What I gave is very incomplete as I am getting lost at this point as how to engineer a potential solution given thinkscripts tool set. Hopefully something here can help.

 

[StoneMan]

@Joshua I got around to running the code and their are still some bugs present. Each scan column you see here is running at a 5 minute interval. I would like to focus on the COOP 100 19 MAY 23 45 CALL. The time display between 9:30 and 11:0 is not correct. The two options above it display incorrect time stamps as well. Below is the TAS page for COOP options so you can see when that block was placed. The Whale Metrics and WhaleTic columns are the code you previously gave me so I could compare the results. A noticeable improvement is that the max volume of the blocks seem to be getting picked up correctly though. In the scan and TAS there is also a COOP MAY 50 Call that had the 10k block picked up. The old code did not do this. So it looks like progress is definitely being made and the issue now lies just with the time stamps.

 

[Joshua]

Here it is on a one minute chart.

1 tick was the largest tick count of any period throughout the day. The first occurrence was at 9:30, and with none being greater than that, it simply records 9:30 as the time of the largest. So you get 9:30 (1).

Unless you want the latest occurrence, instead of the first, there is no way to really differentiate them when they're all the same.

The 1000 volume trade occurred on 1 tick, at 11:01 AM, which falls within the 11:00 AM period on a five minute aggregation. 1000 / 1 = 1000, so you get 11:00 (1000) as the time of the highest tick/volume ratio. The 1000 / 1 on the end is the long-form Volume / Tick that makes up the (1000).

Looks like your time and sales is on west coast time, and the column is on east coast time. COOP230519C45 didn't trade premarket, the only trade before the regular 9:30 open occurred the day before at 3:48 PM. 8:01 AM is incorrect.

Its behaving as intended as far as I know. Perhaps I misunderstood what you're actually trying to do in the first place?

 

[StoneMan]

@Joshua you are correct about the timing being out of sync on my charts, a reflection of some recent travel and an oversight on my part. It would make sense this is throwing that timing off.

As you said it appears to be functioning as intended. Thank you for the hard work and bearing with me here.

If you don't mind, could you paste this code as a reply to this thread along with an explanation. I don't want it to look like I'm taking credit for your work:

https://usethinkscript.com/threads/...ain-column-for-thinkorswim.15238/#post-124464

The goal of this column was always to run it alongside the WhaleScore unusual options metric so you would know exactly where to check the time and sales for the unusual activity it picked up. That metric is an incomplete tool without this.

Speaking of that post it looks the community updated the column to include custom color logic for certain thresholds. Maybe this column could be updated to include custom color logic for what are most likely block or sweep trades and their size. Would be a very rapid visual. You could compare the total volume of the biggest block period (high volume to tic count ratio) to sweep period (high tic count) see what had more total volume, then only display the data for the higher volume block/sweep period with appropriate color coding. If those periods happen to be the same, then the volume to tic count ratio has to be above a certain threshold (like 50) to qualify as a block.

If you could make the output of that code a string that displays the total volume of the period described above, then B for block classification, S for sweep classification, and the time it occurred, you would have a very succinct display you could order by size in the options hacker scan. I think this may be the best way to display this information.

Lots of directions you could take using your code as a foundation. Looking forward to seeing what the community does with it. Let me know if those modifications are something you are interested in building. Thanks again for all the help!

 

[StoneMan]

@Joshua you are correct about the timing being out of sync on my charts, a reflection of some recent travel and an oversight on my part. It would make sense this is throwing that timing off.

As you said it appears to be functioning as intended. Thank you for the hard work and bearing with me here.

If you don't mind, could you paste this code as a reply to this thread along with an explanation. I don't want it to look like I'm taking credit for your work:

https://usethinkscript.com/threads/...ain-column-for-thinkorswim.15238/#post-124464

The goal of this column was always to run it alongside the WhaleScore unusual options metric so you would know exactly where to check the time and sales for the unusual activity it picked up. That metric is an incomplete tool without this.

Speaking of that post it looks the community updated the column to include custom color logic for certain thresholds. Maybe this column could be updated to include custom color logic for what are most likely block or sweep trades and their size. Would be a very rapid visual. You could compare the total volume of the biggest block period (high volume to tic count ratio) to sweep period (high tic count) see what had more total volume, then only display the data for the higher volume block/sweep period with appropriate color coding. If those periods happen to be the same, then the volume to tic count ratio has to be above a certain threshold (like 50) to qualify as a block.

If you could make the output of that code a string that displays the total volume of the period described above, then B for block classification, S for sweep classification, and the time it occurred, you would have a very succinct display you could order by size in the options hacker scan. I think this may be the best way to display this information.

Lots of directions you could take using your code as a foundation. Looking forward to seeing what the community does with it. Let me know if those modifications are something you are interested in building. Thanks again for all the help!

Hello @Joshua I revisited this code with a 5 minute scan period and some of the time-based bugs are persisting. See the CZR weekly's that are highlighted near the bottom. Maybe just labeling the start time of the time period will help. Unfortunately, I don't know how to fix the code you provided to make the time display reliable. We may just have to accept this as a limitation of working with options data here.

 

[Joshua]

.CZR230505C46

At 14:33 (14:30 on Five Min) it did 750 volume, over 375 ticks.
That's a 2 to 1 ratio.
375 was the largest tick count on the day.

At 15:31 (15:30 on Five Min) it did 33 volume, in one tick.
That's a 33 to1 ratio.
33 to 1 was the larges Volume/Tick Ratio on the day.

I don't see the issue?

 

[StoneMan]

@Joshua thank you for the explanation I see now that I have been misinterpreting the output. I was looking at those times as some sort of range for some reason and I really don't know why. Quite the brain fart on my part. I apologize for being a bother and making you repeat yourself.

I am still curious if you could compare the total volume of the biggest block period (high volume to tic count ratio), to sweep period (high tic count), see what had more total volume, then only display the data, as that periods volume, for the higher volume block/sweep period. If those periods happen to be the same, then the volume to tic count ratio has to be above a certain threshold (like 50) to qualify as a block.

If you could make the output of that code a string that displays the total volume of the period described above, then B for block classification, S for sweep classification, and the time it occurred, you would have a very succinct display that you could also be ordered by size in the options hacker scan. I think this may be the best way to display this information.

Here is my rough outline of that logic:

If Blockvolume == SweepVolume and BlockVolumeTicRatio >= 50

Addlabel: BlockVolume ' B' 'Time Block occurred';

If Blockvolume == SweepVolume and BlockVolumeTicRatio < 50

Addlabel: SweepVolume ' S' 'Time Sweep occurred';

If Blockvolume > SweepVolume

Addlabel: BlockVolume ' B' 'Time Block occurred';

If Blockvolume < SweepVolume

Addlabel: SweepVolume ' S' 'Time Sweep occurred';

 

[MerryDay]

If you could make the output of that code a string that displays the total volume of the period described above, then B for block classification, S for sweep classification, and the time it occurred, you would have a very succinct display that you could also be ordered by size in the options hacker scan. I think this may be the best way to display this information.

The ToS Options Hacker Module doesn't allow the using of custom studies to scan against options related data.
https://usethinkscript.com/threads/options-scan-hacker-in-thinkorswim.5114/page-3#post-82892
https://usethinkscript.com/threads/options-scan-hacker-in-thinkorswim.5114/page-2#post-81329

 

[StoneMan]

@MerryDay this is not a study I am referenceing this is still just a column. I was just discussing a way to augment the output @Joshua already successfully created. We have screenshots of it being ran in the chat. This script he has made proves we can do more with these column outputs than meets the eye imo. I was just writing this concept up:

To get crazier with this idea, you could calculate how much money was spent at the high tic, and high volume to tic ratio periods by multiplying options volume by price and compare those. That way, price changes from the day would be accounted for when comparing different time periods.

Then, using the GetUnderlyingSymbol() function, you could find how much money was spent on the underlying stock during the time periods of interest by multiplying volume by price.

You are then set up to create a ratio of options dollars spent to underlying dollars spent at a specific time period to see how big of an "impact" the unusual options activity had relative to the liquidity of the underlying. Higher value is more unusual.

This ratio could be displayed, then a B or S for block or sweep then the time it occurred.

Might be neat. The one part I am unsure about is the viability of GetUnderlyingSymbol() for intraday data. I know works for the daily aggregatetion period.

 

[MerryDay]

@MerryDay this is not a study I am referenceing this is still just a column. I was just discussing a way to augment the output @Joshua already successfully created. We have screenshots of it being ran in the chat. This script he has made proves we can do more with these column outputs than meets the eye imo. I was just writing this concept up:

To get crazier with this idea, you could calculate how much money was spent at the high tic, and high volume to tic ratio periods by multiplying options volume by price and compare those. That way, price changes from the day would be accounted for when comparing different time periods.

Then, using the GetUnderlyingSymbol() function, you could find how much money was spent on the underlying stock during the time periods of interest by multiplying volume by price.

You are then set up to create a ratio of options dollars spent to underlying dollars spent at a specific time period to see how big of an "impact" the unusual options activity had relative to the liquidity of the underlying. Higher value is more unusual.

This ratio could be displayed, then a B or S for block or sweep then the time it occurred.

Might be neat. The one part I am unsure about is the viability of GetUnderlyingSymbol() for intraday data. I know works for the daily aggregatetion period.

But you still do not have any custom option data available available in the "options hacker scan".
Still cannot scan.

When you use words like "options hacker scan" in relation to custom scripts in your posts;
we need to post the above caveat.

As it is not possible to use custom scripts in the "options hacker scan".

 

[StoneMan]

Ahhhhhh @MerryDay I understand what you are saying now and will watch how I phrase my posts to prevent confusion. I was just using that terminology with respect to one of the use cases for custom columns. I will be more clear from now on. For example, I will make sure to say something like, "...adding this custom column to an options hacker scan for ranking purposes..."

 

[StoneMan]

I took a stab at those two ideas I described using the code @Joshua provided as a starting point. They are both a bit buggy. The first always returns B and the second returns 0% in the vast majority of cases. The first bit of code will be for the simpler block or sweep volume output. The second is for the more complex block or sweep option to underlying dollar ratio output. In the dollar ratio output I think there may an issue with how I try to sync up the underlying trade time to the unusual options trade time. Not sure though. Any assistance from the wise sages here would be greatly appreciated. I'd love to be able to deploy these alongside the WhaleScore unusual options metric to drill down further into unusual options activity!

def TickCount;
def VolRatio;
if GetDay() == GetLastDay() {
    TickCount = Tick_Count;
    VolRatio = Volume / Tick_Count;
} else {
    TickCount = 0;
    VolRatio = 0;       
}
;
def HighTick;
def TickTime;
def TickHour;
def TickMinute;
if TickCount > HighTick[1] {
    HighTick = TickCount;
    TickTime = secondsfromTime(0000);
    TickHour = Floor(TickTime / 60 / 60);
    TickMinute = Floor((TickTime / 60 / 60 - TickHour) * 60);
} else {
    HighTick = HighTick[1];
    TickTime = TickTime[1];
    TickHour = TickHour[1];
    TickMinute = TickMinute[1];
}
;
def HighVol;
def VolTime;
def VolHour;
def VolMinute;
def V; def T;
if VolRatio > HighVol[1] {
    HighVol = VolRatio;
    VolTime = secondsfromTime(0000);
    VolHour = Floor(VolTime / 60 / 60);
    VolMinute = Floor((VolTime / 60 / 60 - VolHour) * 60);
    v = Volume;
    t = Tick_Count;
} else {
    HighVol = HighVol[1];
    VolTime = VolTime[1];
    VolHour = VolHour[1];
    VolMinute = VolMinute[1];
    v = V[1];
    t = T[1];
}
;
def BlockVolume = v;
def SweepVolume = TickCount * t;
def BlockVolumeTicRatio = BlockVolume / t;

addLabel(yes,
    if (BlockVolume == SweepVolume and BlockVolumeTicRatio >= 50) then
        BlockVolume + " B " + asPrice(VolHour) + ":" + asPrice(VolMinute)
    else if (BlockVolume == SweepVolume and BlockVolumeTicRatio < 50) then
        SweepVolume + " S " + asPrice(TickHour) + ":" + asPrice(TickMinute)
    else if (BlockVolume > SweepVolume) then
        BlockVolume + " B " + asPrice(VolHour) + ":" + asPrice(VolMinute)
    else if (BlockVolume < SweepVolume) then
        SweepVolume + " S " + asPrice(TickHour) + ":" + asPrice(TickMinute)
    else
        ""
);

def optionsPrice = close * 100;

def TickCount;
def VolRatio;
if GetDay() == GetLastDay() {
    TickCount = Tick_Count;
    VolRatio = Volume / Tick_Count;
} else {
    TickCount = 0;
    VolRatio = 0;       
}

def HighTick;
def TickTime;
def TickHour;
def TickMinute;
if TickCount > HighTick[1] {
    HighTick = TickCount;
    TickTime = secondsfromTime(0000);
    TickHour = Floor(TickTime / 60 / 60);
    TickMinute = Floor((TickTime / 60 / 60 - TickHour) * 60);
} else {
    HighTick = HighTick[1];
    TickTime = TickTime[1];
    TickHour = TickHour[1];
    TickMinute = TickMinute[1];
}

def HighVol;
def VolTime;
def VolHour;
def VolMinute;
def V; def T;
if VolRatio > HighVol[1] {
    HighVol = VolRatio;
    VolTime = secondsfromTime(0000);
    VolHour = Floor(VolTime / 60 / 60);
    VolMinute = Floor((VolTime / 60 / 60 - VolHour) * 60);
    v = Volume;
    t = Tick_Count;
} else {
    HighVol = HighVol[1];
    VolTime = VolTime[1];
    VolHour = VolHour[1];
    VolMinute = VolMinute[1];
    v = V[1];
    t = T[1];
}

def BlockVolume = v;
def SweepVolume = TickCount * t;
def BlockVolumeTicRatio = BlockVolume / t;

def underlyingPrice = close(GetUnderlyingSymbol());
def underlyingVolume = volume(GetUnderlyingSymbol());

def isBlockPeriod = VolRatio >= 50;
def isSweepPeriod = TickCount > HighTick[1];

def BlockUnderlyingVolume = if isBlockPeriod then underlyingVolume else 0;
def SweepUnderlyingVolume = if isSweepPeriod then underlyingVolume else 0;

def BlockOptionsMoneySpent = BlockVolume * optionsPrice;
def SweepOptionsMoneySpent = SweepVolume * optionsPrice;
def BlockUnderlyingMoneySpent = BlockUnderlyingVolume * underlyingPrice;
def SweepUnderlyingMoneySpent = SweepUnderlyingVolume * underlyingPrice;

def BlockOptionsToUnderlyingRatio = if BlockUnderlyingMoneySpent != 0 then (BlockOptionsMoneySpent / BlockUnderlyingMoneySpent) * 100 else 0;
def SweepOptionsToUnderlyingRatio = if SweepUnderlyingMoneySpent != 0 then (SweepOptionsMoneySpent / SweepUnderlyingMoneySpent) * 100 else 0;

addLabel(yes,
    if (BlockOptionsMoneySpent > SweepOptionsMoneySpent and isBlockPeriod) then
        asPrice(BlockOptionsToUnderlyingRatio) + "% B " + asPrice(VolHour) + ":" + asPrice(VolMinute)
    else if (BlockOptionsMoneySpent > SweepOptionsMoneySpent and !isBlockPeriod) then
        asPrice(BlockOptionsToUnderlyingRatio) + "% S " + asPrice(VolHour) + ":" + asPrice(VolMinute)
    else if (BlockOptionsMoneySpent < SweepOptionsMoneySpent) then
        asPrice(SweepOptionsToUnderlyingRatio) + "% S " + asPrice(TickHour) + ":" + asPrice(TickMinute)
    else if (BlockOptionsMoneySpent == SweepOptionsMoneySpent) then
        if (isBlockPeriod) then
            asPrice(BlockOptionsToUnderlyingRatio) + "% B " + asPrice(VolHour) + ":" + asPrice(VolMinute)
        else
            asPrice(SweepOptionsToUnderlyingRatio) + "% S " + asPrice(TickHour) + ":" + asPrice(TickMinute)
    else
        ""
);

 

[Joshua]

I kind of lost track of what's going on here. I'll check it out again when I have more time.

 

[StoneMan]

The tldr is that both scripts are a modified version of what you provided.

The first looks at the total volume of the high tick and high volume to tick ratio periods then displays the time, and volume, of the highest volume measurement along with a 'B' for block or 'S' for sweep classification. A volume to tick ratio of 50 is used as the cutoff to determine B or S. The goal is to always show the highest volume period.

The second script is a modification of the first that complicates things further by comparing the amount of money spent at those high tick and high volume to tick ratio periods. Using price is a way to "normalize" for an options changes in value throughout the day. We also find the amount of money spent on the underlying stock at the same times as our high tick and high volume to tick count ratio options activity. This is used to create and display a ratio of options activity money to underlying money spent. The goal being to try and assess the "impact" that options activity may have had relative to the liquidity of underlying. Higher impact being more unusual. The display logic is set up to show that ratio for wherever the highest options dollar spend was.