Query Builder Problem - Distribution of Property value buckets are skewed by data, making them unusable
Context
When using the ‘distribution of property’ option in the query builder, the data is automatically allocated into buckets that are defined by splitting the range of the data into five equal parts.
ie if the range of the data is 0-100, the buckets will be something like 0-20, 21-40, 41-60, 61-80, 81-100.
Problem
Outliers in the data will heavily skew the result, making the feature unusable. For example
You have 100 logged events
- 90 of them have a value between 1-10
- 5 have a value of 11-20
- 5 have a value of 100
the buckets will be split from 0-20, 21-40, 41-60, 61-80, 81-100.
You'll then have 95 of your events in bucket 1 (0-20) and 5 in bucket 5 (81-100)
As a result you have no granularity on the bulk of the data which is all collated in 0-20, meaning you can’t extract any learnings
Suggested solutions
- Allow the user to set the bucket sizes, including an outer limit. ie 1”-3, 4-6, 7-9, 10+”
- It would also be helpful in this case to be able to set the number of buckets
- Allow the user to manually not include certain data samples to remove the skew.
- The caveat here is that if you have a lot of data this could take some time...so maybe it could be done as a part of the query
- ie ‘”Don’t include # > 20”