As some of you may already know, I created an "Upvote Estimator" tool on ccmoons.com that tries to estimate the amount of Karma you've earned since the last snapshot after accounting for the modifications from the governance polls.

Now that the round 18 snapshot CSV has been posted, I wanted to see how well the tool did by manually looking at 50 users (across a wide range of earned Karma) and comparing my estimate with the actual Karma earned.

Disclaimers

Before I begin I want to reiterate that there are a lot of reasons why my estimate will never be exact and could be quite inaccurate:

• The admins don't disclose when exactly the snapshot period starts and ends. I guess what these cutoffs are, but I could be up to 1 day off. This means a popular submission you made could easily be excluded from my estimate when it should have been included, or vice-versa
• No one except Reddit knows the formula for Karma. 1 Upvote does not equal 1 Karma
• The admins don't disclose when the cutoff periods are for the 50 comment penalty. Previously my estimator didn't account for this at all, but going forward I will randomly guess what these are too.
• The estimator can only pull the last 1k comments for a user (across all subreddits). The "legacy estimator" on my site can pull more, but is slow and unreliable

These disclosures are listed on the website, but based on the many DMs/comments I received I don't think people read them.

Now the results!

My estimator outputted the following sentence:

Estimated Net Upvotes <X> (Up to <Y> with 30% bonus for holding & voting).

For the following I'll call X the "Lower Estimate" (LES), Y the "Upper Estimate" (UES) and (X+Y)/2 the "Mean Estimate" (MES)

In the plot below, each point represents one of the 50 users I looked at. The blue circles are the MES, and the error bars are made up of the LES and UES.

The black line is a 45-degree line indicating where predicted=actual. If my estimator was perfect all blue circles would fall on the black line.

Predicted Karma vs. Actual Karma

Generally the MES was pretty good!

Too see this I plot distributions of the errors (how much they differed from the actual Karma) for the LES, MES, and UES

Karma Estimate Errors

MES had an average error of +2.8% and a median error of +1.23%

LES had an average error of -11.3% and a median error of -11.3%

UES had an average error of +13.7% and a median error of +12.6%

However (see next section), IMO the average error rates aren't as bad as the above suggests

Diagnosing Errors

In most cases the reasons for large errors were very clear:

• The biggest two overestimations were from users that earned <20 karma. So while the % error was large I wasn't off by much Karma
• The next 5 largest overestimates were for power users who commented between 1562 and 2629 times during the snapshot. As mentioned before I can't really account for the 50 comment penalties, which these users hit quite often

The largest underestimates were because I excluded some popular comments when they should have been included. Again, I don't know exactly when snapshot starts/ends so this is mostly unavoidable.

Summary & Next Steps

Overall I was somewhat pleased by how well the Mean Estimate performed

My big mistake was in the phrasing of the tool when I said "Up to <Y> with 30% bonus for holding & voting. This naturally made people expect that higher amount if they held and voted and lead to some disappointment when the result was lower. My apologies for this!

Going forward my estimate will output the Mean Estimate in addition to a range based on the lower and upper estimates.

Thanks for reading and let me know if you have any suggestions!

TLDR: I created a karma estimation tool at ccmoons.com. It seemed to do alright