librelist archives

« back to archive

Whence cowbells?

Whence cowbells?

From:
Chad Woolley
Date:
2009-12-31 @ 04:43
What's the algorithm for calculating cowbells?

-- Chad

Re: [fretwar] Whence cowbells?

From:
Zed A. Shaw
Date:
2009-12-31 @ 08:05
On Wed, Dec 30, 2009 at 09:43:13PM -0700, Chad Woolley wrote:
> What's the algorithm for calculating cowbells?

select avg(mean) as rating, avg(sd) as rating_sd, other_id,
player_id, round_id
from statistic, submission where other_id=submission.id and
other_type='submission' 
and round_id = $rnd.id
group by other_id, player_id, round_id order by round_id
desc, rating desc, rating_sd asc;

That's the query.  It's just the average of all the means times 1000 so
it's not a fractional number.

This produces this table:
rating       |     rating_sd     | other_id | player_id | round_id
-------------------+-------------------+----------+-----------+--
1.19444445272287  | 0.498557180166245 |       81 |       181 | 71
1.1363636503617   |  0.54525621732076 |       73 |        39 | 71
1.07575759788354  | 0.495001117388407 |       79 |       199 | 71
1.00000000745058  | 0.542712951699893 |       80 |         1 | 71
0.972222242504358 | 0.431989908218384 |       78 |       120 | 71
0.962962968895833 | 0.494796648621559 |       77 |        40 | 71
0.928571457664172 |  0.49314454694589 |       82 |        20 | 71

If we pop the raw scores from that first one (submission id 81) we get:

>>> (0.333 + 0.5 + 0.666 + 1 + 4 + 0.666)/6
1.1941666666666668

And finally if you mutliply by 1000 you get:

1194 cowbells.

Notice also that the ranking is influenced by the rating_sd, which is
how tightly or widely people's opinions of you range.  So people with a
lower sd get ranked higher as being more consistent.

Keep in mind that I gotta redo round 6 because of the bogus votes from
Ram, which will happen tomorrow when I write round 7 and announce round
6 done on the fret show.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [fretwar] Whence cowbells?

From:
Chad Woolley
Date:
2009-12-31 @ 11:01
On Thu, Dec 31, 2009 at 1:05 AM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> On Wed, Dec 30, 2009 at 09:43:13PM -0700, Chad Woolley wrote:
>> What's the algorithm for calculating cowbells?
>
> select avg(mean) as rating, avg(sd) as rating_sd, other_id,
> player_id, round_id
> from statistic, submission where other_id=submission.id and
> other_type='submission'
> and round_id = $rnd.id
> group by other_id, player_id, round_id order by round_id
> desc, rating desc, rating_sd asc;
>
> That's the query.  It's just the average of all the means times 1000 so
> it's not a fractional number.
>
> This produces this table:
> rating       |     rating_sd     | other_id | player_id | round_id
> -------------------+-------------------+----------+-----------+--
> 1.19444445272287  | 0.498557180166245 |       81 |       181 | 71
> 1.1363636503617   |  0.54525621732076 |       73 |        39 | 71
> 1.07575759788354  | 0.495001117388407 |       79 |       199 | 71
> 1.00000000745058  | 0.542712951699893 |       80 |         1 | 71
> 0.972222242504358 | 0.431989908218384 |       78 |       120 | 71
> 0.962962968895833 | 0.494796648621559 |       77 |        40 | 71
> 0.928571457664172 |  0.49314454694589 |       82 |        20 | 71
>
> If we pop the raw scores from that first one (submission id 81) we get:
>
>>>> (0.333 + 0.5 + 0.666 + 1 + 4 + 0.666)/6
> 1.1941666666666668
>
> And finally if you mutliply by 1000 you get:
>
> 1194 cowbells.

Still confused.  This calculation is for a single round?  If so:

1. How were the raw scores calculated (where did 0.333) come from?
2. How does the standard deviation factor in (above looks like just
the average of the raws)?  How does a lower SD result in a higher
score?
3. What is the standard deviation of?  The raw scores?  If so, how can
there be multiple SDs per round or one SD per raw score (and why would
they be averaged)?

Or, if this calculation isn't for a single round, what is it for?

Sorry if I'm obtuse.  I've always thought I was the inspiration for
your "programmers must learn statistics of I will kill them"
article...

-- Chad

Re: [fretwar] Whence cowbells?

From:
Zed A. Shaw
Date:
2009-12-31 @ 19:34
On Thu, Dec 31, 2009 at 04:01:36AM -0700, Chad Woolley wrote:
> Still confused.  This calculation is for a single round?  If so:

http://zedshaw.com/blog/2009-11-6.html

Explains the math behind it.

> 1. How were the raw scores calculated (where did 0.333) come from?

It's from a rolling statistic of the five qualitative elements and the
overall rating.  It's done so that it can be calculated as each person
submits their rating, rather than having to be done repeatedly over all
the comments each time.  See the blog post.

> 2. How does the standard deviation factor in (above looks like just
> the average of the raws)?  How does a lower SD result in a higher
> score?

Nope, it's the average of the standard deviations for all of the
measurements.  So it's average of accuracy, tone, speed,
overall_rating's standard deviation.

Think of it as a "meta statistic".  Each stat has a set of numeric
elements, which were derived from people voting.  Now if I then take all
the stats of your submssion, and get the mean of each numeric element,
then I've got a normalized meta stat I can use to say where you stand.

In this case, the sd is more like meta-sd.  It's the mean of all your sd
ratings so far, or it could be the mean of the sd of all your ratings
ever.

> 3. What is the standard deviation of?  The raw scores?  If so, how can
> there be multiple SDs per round or one SD per raw score (and why would
> they be averaged)?

There are six stats you can have on a submission.  Each stat has
variables n, sum, mean, sd, min, max.  There's only one set of these for
each submission, and as people vote, these numbers are recalulated using
their vote.

So for each submission that's 6 sd (one for each stat).

The average sd is then what I showed you before in the other email and
in that blog post.  It's just the sum of these std.devs / 6.

> Or, if this calculation isn't for a single round, what is it for?

It's for a single round and just rolls up all your stats into a nice
easy to understand number.

But, it can also be used for the "best player" stats.  Instead of doing
it for one round, just do the average for all the rounds and you've got
a meta-statistic that ranks you among other players.

I could also do it for the whole site, by averaging all the stats I
could see how good all the players are.

> Sorry if I'm obtuse.  I've always thought I was the inspiration for
> your "programmers must learn statistics of I will kill them"
> article...

No problem, but you should read that blog post and read more about mean,
std.dev, and the mean of means, and mean of std.devs and why they're
important.

Hint:  binary votes on accuracy aren't actually normally distributed,
they're logit, but if I average a bunch of them together they become
normally distributed.

-- 
Zed A. Shaw
http://zedshaw.com/