Algorithm creates representative set of data

mundek88

New member
Joined
May 23, 2016
Messages
1
I have algorithm to analyze and make it easier to implement in programming language (Python). We have table with data and we want to select only representative part.


It looks like:
ID_PRODUCT | CARDINALITY | SET VARIANCE WITH THIS ELEMENT AND ABOVE
10 ---------------- 110 --------------- 400
11 ---------------- 90 ---------------- 350
12 ---------------- 80 ---------------- 300
... --------------- ... ---------------- ...


* variance is calculated for cardinality column




Algorithm works as follows:
From the top of table iterate over rows and in each loop add new row and count variance for cardinality column. Stop iteratation if variance is equal or less than specified (so, finally we want to produce set of rows with variance bigger than X) and then return created (now representative) set


Question:
This is legacy solution and hard to say for me how we can do it better. Is there any math tool which cut away elements hardly representative? We can not statically based on the cardinality (like: just give rows with cardinality > 50) because the day-to-day can change the order of magnitude.


Thanks in advice!
 
Top