As I understand entropy: it means - how random components are in an architecture. More random components you use the architecture becomes complex.
Informally, entropy is the
average level of information gained from some possible outcomes.
The core idea of entropy is as follows:
- If a highly likely event occurs, the message carries very little information.
- Whereas, if a highly unlikely event occurs, the message is much more informative
For example, you have some knowledge about which days are NOT going to rain for the next 365 days. This provides a little information about which day it's going to rain exactly, but highly probable. On the other hand, if you know exactly the day that it's going to rain, this is highly informational, but it communicates a very low probability.
All that is to say information value and probability are inversely related. To describe the relationship, we say informational value, [imath]I_i[/imath]:
[math]I_i= \log\left({\frac{1}{f_i}}\right) ; \text{where } f_i = \frac{x_i}{\sum x_i }\text{ the probability of event } i[/math]
The reason we use [imath]\log[/imath] is because when [imath]f_i = 1[/imath], the surprisal of the event has no value [imath]I_i = 0[/imath] and when the probability decreases to 0, [imath]I_i[/imath] increases .
Formally, the entropy measure is defined as the weighted average of all informational values gained:
[math]\text{Entropy} = \sum I_i\cdot f_i= \sum \log\left(\frac{1}{f_i}\right) \cdot f_i[/math][math]\text{ where } f_i \text{ are the weights}[/math]
Understanding it on a fundamental level helps you see how it can be applied in different contexts like the example
@blamocur gave, enterprise architecture in your PowerPoint, decision tree in machine learning, ect...
