Alright. I hope you'll bear with me, as I am still quite confused as to what you are asking. You say that your textbook teaches that the square root of a fraction is equal to the square root of the numerator over the square root of the denominator. And that's absolutely true. The only time that wouldn't be true is if the numerator and/or the denominator is negative (as the square root of a negative number is undefined). Using that rule, you apply to it to a specific example:
\(\displaystyle \sqrt{\strut \dfrac{64}{4096}\,}\, =\, \dfrac{\sqrt{\strut 64\,}}{\sqrt{\strut 4096\,}}\,=\,\dfrac{8}{64}\,=\,\dfrac{1}{8}\)
But it's after this that I get confused. You ask "What would happen if there was some formula that someone needed to follow to find a result; that math resulted in a fraction that wasn't in its simplest terms; and getting the correct answer required taking it to simplest terms before finding its square [root]?" Now, why I'm confused about this is that it doesn't matter at all if you simplify before or after taking a square root. You'll get the same answer regardless. As another example, let's look at 1296/11664. First, we'll simply to get 1/9.
\(\displaystyle \sqrt{\strut \dfrac{1}{9}\,}\, = \,\dfrac{\sqrt{\strut 1\,}}{\sqrt{\strut 9\,}}\, = \,\dfrac{1}{3}\)
We'll do it again, but take the square root first.
\(\displaystyle \dfrac{\sqrt{\strut 1296\,}}{\sqrt{\strut 11664\,}}\,=\,\dfrac{\sqrt{\strut 1296\,}}{\sqrt{\strut 11664\,}}\,=\,\dfrac{36}{108}\,=\,\dfrac{1}{3}\)
As for your last sentence, "In other words how would we demonstrate with 64/4096, that we want the person doing the work to find (1/8)^(1/2)?" If you want the person to find the square root of 1/8, then you'd need to ask them to take the square root of the square root, or the fourth root, of 64/4096. That is to say:
\(\displaystyle \sqrt{\strut \sqrt{\strut \dfrac{64}{4096}\,}\,}\,=\,\sqrt[4]{\strut \dfrac{64}{4096}\,}\,=\,\sqrt{\strut \dfrac{1}{8}\,}\)
