scaling standard deviation

harrison

New member
Joined
Aug 10, 2017
Messages
20
Basically what I am trying to do is scale standard deviation using a value that is 0<x<1


So for example, if standard deviation (S)= .0001, and y = resulting scaled value using the value between 0<x<1 applied to standard deviation


and x is between the interval [0,.5) then y<.0001$ where y is incrementally getting smaller as x decreases in value.


If x is between the interval (.5,1] then y>.0001 where y is incrementally getting larger as x increases in value.


How would you accomplish this?
 
Basically what I am trying to do is scale standard deviation using a value that is 0<x<1


So for example, if standard deviation (S)= .0001, and y = resulting scaled value using the value between 0<x<1 applied to standard deviation


and x is between the interval [0,.5) then y<.0001$ where y is incrementally getting smaller as x decreases in value.


If x is between the interval (.5,1] then y>.0001 where y is incrementally getting larger as x increases in value.


How would you accomplish this?

I'm not sure what you mean by "scaling standard deviation". What is it that has a standard deviation of .0001, and what do x and y represent?

Can you perhaps give a specific example of what this might look like?

If you hadn't mentioned standard deviation, I could imagine what you might be asking for. You would be asking for a function like this: y = 0.0002x. That rises from 0 to 0.0001 as x increases from 0 to 0.5, and continues rising from 0.0001 to 0.0002 as x increases from 0.5 to 1.

Or maybe a piecewise function that changes slope at x = 0.5.
 
I'm not sure what you mean by "scaling standard deviation". What is it that has a standard deviation of .0001, and what do x and y represent?

Can you perhaps give a specific example of what this might look like?

If you hadn't mentioned standard deviation, I could imagine what you might be asking for. You would be asking for a function like this: y = 0.0002x. That rises from 0 to 0.0001 as x increases from 0 to 0.5, and continues rising from 0.0001 to 0.0002 as x increases from 0.5 to 1.

Or maybe a piecewise function that changes slope at x = 0.5.

What I meant by scaling standard deviation was just the general idea of what i was trying to accomplish by what was described in the post.

Standard deviation is constantly changing as I am applying it to time series.

The value of .0001 was just an example value of standard deviation used to show what I want to accomplish.

Using standard deviation = .0001 as an example

x = hurst exponent thus 0<x<1

y = resulting value after scaling standard deviation by x
 
So, as an example if SD = .0001

x = 0 y = .00001
x = .1, y = .00003
x = .2, y = .00005
x = .3, y = .00007
x = .4, y = .00009
x = .5, y = .00011
x = .6, y = .00013
x = .7, y = .00015
x = .8, y = .00017
x = .9, y = .00019
x = 1, y = .00021

Obviously those do not need to be the exactly values, and of course using Hurst exponent you will have values between (.1,.2), (.2,.3), (.3,.4)....It was just an example/idea of what I am trying to accomplish.

Also, the magnitude the effect that x has on y should be scale-able.

Is there some logical mathematical way I could accomplish something similar to this example?
 
So, as an example if SD = .0001

x = 0 y = .00001
x = .1, y = .00003
x = .2, y = .00005
x = .3, y = .00007
x = .4, y = .00009
x = .5, y = .00011
x = .6, y = .00013
x = .7, y = .00015
x = .8, y = .00017
x = .9, y = .00019
x = 1, y = .00021

Obviously those do not need to be the exactly values, and of course using Hurst exponent you will have values between (.1,.2), (.2,.3), (.3,.4)....It was just an example/idea of what I am trying to accomplish.

Also, the magnitude the effect that x has on y should be scale-able.

Is there some logical mathematical way I could accomplish something similar to this example?

I'm sorry, I still don't understand what you are trying to do.

You have hinted, now, at some context that had not been mentioned before, and it may require knowledge I lack; but I think more explanation is needed to clarify your context and goals.
 
I'm sorry, I still don't understand what you are trying to do.

You have hinted, now, at some context that had not been mentioned before, and it may require knowledge I lack; but I think more explanation is needed to clarify your context and goals.

Did you see the post before the one you quoted?
 
I just meant "scaling standard deviation" in the sense of what I was trying to accomplish. It really means nothing.

Knowledge of the hurst exponent is really not necessary to be honest. That's why i said 0<x<1
 
So, as an example if SD = .0001

x = 0 y = .00001
x = .1, y = .00003
x = .2, y = .00005
x = .3, y = .00007
x = .4, y = .00009
x = .5, y = .00011
x = .6, y = .00013
x = .7, y = .00015
x = .8, y = .00017
x = .9, y = .00019
x = 1, y = .00021

Obviously those do not need to be the exactly values, and of course using Hurst exponent you will have values between (.1,.2), (.2,.3), (.3,.4)....It was just an example/idea of what I am trying to accomplish.

Also, the magnitude the effect that x has on y should be scale-able.

Is there some logical mathematical way I could accomplish something similar to this example?

Let's guess that you just want a formula to produce the values you show here. You have a linear relationship y = f(x), where f(0.5) = 0.00011, and an increase of 0.1 in x leads to an increase of 0.00002 in y. This means the slope of the function is 0.00002/0.1 = 0.0002. Therefore, the function is

f(x) = 0.0002(x - 0.5) + 0.00011

which simplifies to

f(x) = 0.0002x + 0.00001

For example, f(0.7) = 0.0002(0.7) + 0.00001 = 0.00014 + 0.00001 = 0.00015 as required.

To change the value of f(0.5), just change the 0.00011; to change the rate at which y increases, change the 0.0002. And the 0.00001 in the simplified form represents f(0), if you knew that ahead of time.

Is that something like what you want?
 
Let's guess that you just want a formula to produce the values you show here. You have a linear relationship y = f(x), where f(0.5) = 0.00011, and an increase of 0.1 in x leads to an increase of 0.00002 in y. This means the slope of the function is 0.00002/0.1 = 0.0002. Therefore, the function is

f(x) = 0.0002(x - 0.5) + 0.00011

which simplifies to

f(x) = 0.0002x + 0.00001

For example, f(0.7) = 0.0002(0.7) + 0.00001 = 0.00014 + 0.00001 = 0.00015 as required.

To change the value of f(0.5), just change the 0.00011; to change the rate at which y increases, change the 0.0002. And the 0.00001 in the simplified form represents f(0), if you knew that ahead of time.

Is that something like what you want?

Wow I completely missed your response when i checked on Thursday. Sorry for the delay and thank you for your post.

The relationship you are showing is obviously if standard deviation was constant at .0001 to obtain the values I posted (x = 0 y = .00001 , x = .1, y = .00003 , etc) but in my case standard deviation is variable and thus the same proportion of values obtained using f(x) = 0.0002x + 0.00001 would have a much smaller percentage change to standard deviation if SD = 1.

Do you know what I mean? Is there a way to incorporate SD in a way that would be similarly proportional to the slope of line you suggested?
 
Wow I completely missed your response when i checked on Thursday. Sorry for the delay and thank you for your post.

The relationship you are showing is obviously if standard deviation was constant at .0001 to obtain the values I posted (x = 0 y = .00001 , x = .1, y = .00003 , etc) but in my case standard deviation is variable and thus the same proportion of values obtained using f(x) = 0.0002x + 0.00001 would have a much smaller percentage change to standard deviation if SD = 1.

Do you know what I mean? Is there a way to incorporate SD in a way that would be similarly proportional to the slope of line you suggested?

No, I still have no idea what standard deviation has to do with your question; last time you said, "It really means nothing," which is why I decided to ignore it.

The standard deviation of WHAT? Which thing here are you saying is a standard deviation, or has a standard deviation?
 
I just meant "scaling standard deviation" in the sense of what I was trying to accomplish. It really means nothing.

Knowledge of the hurst exponent is really not necessary to be honest. That's why i said 0<x<1
"It really mean nothing" seems to be a very accurate description of your attempt to make clear to others what appears to be a very vague idea in your own mind.
 
"It really mean nothing" seems to be a very accurate description of your attempt to make clear to others what appears to be a very vague idea in your own mind.

Fair enough. I will write a better description of what I am looking for.
 
"It really mean nothing" seems to be a very accurate description of your attempt to make clear to others what appears to be a very vague idea in your own mind.
I wonder if OP was referring to scaling of SD that is done to transform Normal distribution to Z-distribution
 
Ok, here is my attempt at a better explanation. Thank you for the replies!


Let,


s = standard deviation


x = 0<x<1


p = the resulting value that x has on s


Calculating s of a time series, the value will constantly be changing. So s can essentially take on any value.


If 0<x<.5 then p<s and p is getting incrementally smaller based on x as x decreases in value within the interval (0,.5)


If .5<x<1 then p>s and p is getting incrementally larger based on x as x increases in value within the interval (.5,1)


Ofcourse if s is constant then like Dr peterson said we could just create a linear relationship where x would have a fixed change on s.


But, since s is variable this causes some issues as the same % effect that x would have on s would be completely different if s = .01 and then changed to s = 15.


So, if I wanted x to have a fixed % change on s regardless of what s was, how would I do this?
 
Ok, here is my attempt at a better explanation. Thank you for the replies!

Let,

s = standard deviation

x = 0<x<1

p = the resulting value that x has on s

Calculating s of a time series, the value will constantly be changing. So s can essentially take on any value.

If 0<x<.5 then p<s and p is getting incrementally smaller based on x as x decreases in value within the interval (0,.5)

If .5<x<1 then p>s and p is getting incrementally larger based on x as x increases in value within the interval (.5,1)

Of course if s is constant then like Dr peterson said we could just create a linear relationship where x would have a fixed change on s.

But, since s is variable this causes some issues as the same % effect that x would have on s would be completely different if s = .01 and then changed to s = 15.

So, if I wanted x to have a fixed % change on s regardless of what s was, how would I do this?

Let's see if I can fill in the gaps that remain.

You start with some set of data, your time series. You haven't described it, or said what it has to do with the problem -- is x a data value in this time series? Or is x something else that can vary independently?

At any moment, there is some amount of data (perhaps from a window of time), and you find its standard deviation, s.

I have no idea what "the resulting value that x has on s" means, so I can't interpret what p is. I'll hope that becomes apparent eventually.

Somehow the value of x at any moment causes p to have some value, which is less than s if x<0.5, and greater than s if x>0.5. And if x is increasing, p decreases; that is, p is a monotonically increasing function of x: the graph of p versus x slopes upward, though not in generally linear. This function also has to depend on s, since for x=0.5, p=s (or is near s?). So maybe we need to think of p as a function p(s,x)?

It's also unclear whether p is something real that you are trying to determine, or just some arbitrary function you can invent as long as it follows the criteria you've stated. How is it to be used?

I don't know what you mean by "I want x to have a fixed change on s regardless of what s is". That's not English; I suppose you mean "a fixed effect", but that's not clear either. And I thought it was p that depended on s, not s depending on x. Or do you mean that since x is the data, s is (of course) dependent on the values of x (though not just the current one)? But you have no control over how the data affects the standard deviation of that data.

So, no, I still don't get it. Maybe someone else does.

Can you tell us what this is really about, so we can use our broader knowledge to see what is going on? At least say what x and p ARE.
 
Let's see if I can fill in the gaps that remain.

You start with some set of data, your time series. You haven't described it, or said what it has to do with the problem -- is x a data value in this time series? Or is x something else that can vary independently?

At any moment, there is some amount of data (perhaps from a window of time), and you find its standard deviation, s.

I have no idea what "the resulting value that x has on s" means, so I can't interpret what p is. I'll hope that becomes apparent eventually.

Somehow the value of x at any moment causes p to have some value, which is less than s if x<0.5, and greater than s if x>0.5. And if x is increasing, p decreases; that is, p is a monotonically increasing function of x: the graph of p versus x slopes upward, though not in generally linear. This function also has to depend on s, since for x=0.5, p=s (or is near s?). So maybe we need to think of p as a function p(s,x)?

It's also unclear whether p is something real that you are trying to determine, or just some arbitrary function you can invent as long as it follows the criteria you've stated. How is it to be used?

I don't know what you mean by "I want x to have a fixed change on s regardless of what s is". That's not English; I suppose you mean "a fixed effect", but that's not clear either. And I thought it was p that depended on s, not s depending on x. Or do you mean that since x is the data, s is (of course) dependent on the values of x (though not just the current one)? But you have no control over how the data affects the standard deviation of that data.

So, no, I still don't get it. Maybe someone else does.

Can you tell us what this is really about, so we can use our broader knowledge to see what is going on? At least say what x and p ARE.

No, x is not a data value in the time series. x = hurst exponent which is a value within the interval (0,1) (as said before knowledge of the hurst
exponent is not necessary but you wanted to know what x was). The hurst exponent is just a measure of long term memory of a time series. So to answer
your first question, x is something else that can vary independently.


"Somehow the value of x at any moment causes p to have some value, which is less than s if x<0.5, and greater than s if x>0.5. "


Yes exactly.


"And if x is increasing, p decreases; that is, p is a monotonically increasing function of x:"


No, If x<0.5 then p is getting smaller then s as x decreases in value. vice versa if x>.5 then p is getting larger then s as x increases
in value.


p needs to be a function of s and x thus, p(s,x)


p is some abitrary function as long as it follows the criteria I have stated.


Yes, I mean a fixed effect.
 
No, x is not a data value in the time series. x = hurst exponent which is a value within the interval (0,1) (as said before knowledge of the hurst
exponent is not necessary but you wanted to know what x was). The hurst exponent is just a measure of long term memory of a time series. So to answer
your first question, x is something else that can vary independently.

"Somehow the value of x at any moment causes p to have some value, which is less than s if x<0.5, and greater than s if x>0.5. "

Yes exactly.

"And if x is increasing, p decreases; that is, p is a monotonically increasing function of x:"

No, If x<0.5 then p is getting smaller then s as x decreases in value. vice versa if x>.5 then p is getting larger then s as x increases
in value.

p needs to be a function of s and x thus, p(s,x)

p is some arbitrary function as long as it follows the criteria I have stated.

Yes, I mean a fixed effect.

Okay, let's try again.

First, note that if p gets smaller as x decreases, and p gets larger as x increases, then p is monotonically increasing (as a function of x). Am I totally misunderstanding you there?

But as I understand it, you have some time series data coming in (which is off in the distance where I can't see it). From it, you estimate x, the Hurst exponent (usually called H?), which is presumably based on all the data you currently have, so that it can be changing (but wouldn't be expected to change much, I suppose). Presumably also the standard deviation s is obtained from that same data, and likewise can be changing.

Now you want to calculate a quantity p, whose purpose is unknown and is not "real" in the sense that it is definitely determined by anything, but is somewhat arbitrary. The only criteria you have stated for it, I think, are that it must be greater than s for "large" x (x>0.5) and less than s for "small" x, and that it is an increasing function of x.

With no other criteria, it seems to me that my first suggestion, y = 0.0002x, with 0.0001 replaced with s, would do all you want:

p = 2sx

This is an increasing function of x (and of s, if it matters), and when x = 0.5, p = s.

If, as in the example you gave, you don't want p to be 0 when x is 0, you could modify my subsequent suggestion of p = 0.0002x + 0.00001, and use

p = (2x + 0.1)s

Is that all you want? I suppose these could be called "scaling standard deviation", though your emphasis all along has been on how p should relate to x, rather than that p should be proportional to s.
 
Okay, let's try again.

First, note that if p gets smaller as x decreases, and p gets larger as x increases, then p is monotonically increasing (as a function of x). Am I totally misunderstanding you there?

But as I understand it, you have some time series data coming in (which is off in the distance where I can't see it). From it, you estimate x, the Hurst exponent (usually called H?), which is presumably based on all the data you currently have, so that it can be changing (but wouldn't be expected to change much, I suppose). Presumably also the standard deviation s is obtained from that same data, and likewise can be changing.

Now you want to calculate a quantity p, whose purpose is unknown and is not "real" in the sense that it is definitely determined by anything, but is somewhat arbitrary. The only criteria you have stated for it, I think, are that it must be greater than s for "large" x (x>0.5) and less than s for "small" x, and that it is an increasing function of x.

With no other criteria, it seems to me that my first suggestion, y = 0.0002x, with 0.0001 replaced with s, would do all you want:

p = 2sx

This is an increasing function of x (and of s, if it matters), and when x = 0.5, p = s.

If, as in the example you gave, you don't want p to be 0 when x is 0, you could modify my subsequent suggestion of p = 0.0002x + 0.00001, and use

p = (2x + 0.1)s

Is that all you want? I suppose these could be called "scaling standard deviation", though your emphasis all along has been on how p should relate to x, rather than that p should be proportional to s.

"p = (2x + 0.1)s" That is exactly it! Thank you so much!
 
Top