[0B]

The accuracy of a sample size is not based on what percent of the population is taken, the math for this is more complex. If you take a true random sampling of 1,000, out of a population of 100,000,000, the confidence level will vary based on the rate and what level of confidence you want.

Let's say you have a total population of 8 billion. You take a random sampling of 1,000 people- this is a fraction of a percent of the total population. From your sample, you end up with 50% of respondents being male, and 50% being female.

I use an online calculator to do this, but based on our sample of 1,000 people, we can have-

99% confidence that between 45.92% and  54.08% of the total population is male.
95% confidence that between 46.90% and 53.10% of the total population is male.
90% confidence that between 47.40% and 52.60% of the total population is male.
50% confidence that between 48.93% and 51.07% of the total population is male.

The confidence level means that if you were to take the survey many times (assuming true random sampling), X% of the confidence intervals would contain your true population. 95% is pretty standard, 99% usually has too wide an interval to be practical, and 90% or lower is just too unreliable.

It's true that this isn't a true random sampling. It's also true that this measures 3 specific online communities. It does NOT measure playerbase.

Now, let's say you have a total population of 10,000. You take a random sampling of 50 people- this is half a percent of the total population. Let's say you end up with 46% of respondents being male, and 54% being female.

Now, these are our levels:

99% confidence that between 27.89% and 64.11% of the population is male.
95% confidence that between 32.22% and 59.78% of the population is male.
90% confidence that between 34.43% and 57.57% of the population is male.
50% confidence that between 41.26% and 50.74% of the population is male.

So, now we can see that with a smaller sample size, the confidence levels are larger, even when the sample size is a larger percentage of the total population. However, the confidence levels are still mathematically accurate. They are wider because it is less precise with the smaller sample size, but a 95% confidence level with the sample of 50 is exactly as likely to contain the true proportion as the 95% confidence level with the sample of 1000.

So, if we find that 8 people out of 77 (Or, 10.39%) say they like something, and we assume a worst-case total population of 60,000, we can apply this to our population on /r/shadowrun, the official forums, and dumpshock:

99% confidence that between 1.44% and 19.34% of the population likes SR 6e.
95% confidence that between 3.58% and 17.20% of the population likes SR 6e.
90% confidence that between 4.67% and 16.11% of the population likes SR 6e.
50% confidence that between 8.05% and 12.73% of the population likes SR 6e.

Is this helpful at all? Is there a better way that I can explain why this works? OpenStax has an intro to stats book that explains confidence intervals more in-depth than I can.

This is true when you have a truly random sample size to survey. But problems of bias come in during execution. And the biggest issue is usually who or where you are polling from.

For example:

If I went to catholic Churches and handed out a poll asking if they supported Abortion, the responses would be overwhelmingly "No." But that result wouldn't reflect the true opinion of a country as Abortion is against the tenets of Catholicism.
If I asked the same question to Twitter, I would get an overwhelming response of "yes". And again this would not reflect the actual opinion of the a country as Twitter is overwhelmingly used by those with a political left leaning, and only fractionally by those of a Right political leaning.

However, If I was to call 10,000 random households and ask the question of "do you support Abortion?", the results would fall much more closely into the national average,and could be a good barometer for a Country. (Again, depending on other factors. Only Conducting the survey on Sundays could again skew the results)

A big one that came up in my Province 20 years ago was Welfare reform. LOTS of polling was done... All of it said "everyone" wanted an increase of the welfare state... Right up until election day.. And the party that made it an election issue was wiped out (and has never recovered).
Turns out, the polling firms had introduced a Bias in their polling. They conducted their polling Monday to Friday from 10am to noon, right when the vast majority of people are at work, and the only people who could answer the surveys were those who were already on Social Assistance, and they, had a natural self interest in seeing the social systems expanded that was not reflected in the rest of the population.

In your case, Its the people that visit the sites.

If the average person bought 6e and encountered NO ISSUES whatsoever with the product, they would not be on these forums, or Reddit or the Facebook page(?).

Thus, they never saw your poll to respond.

You only have to look at the topics that are being posted to see if you are going to have issues with Bias. In the case of Reddit and here (Don't use facebook, so I can check to see.. and Dumpshock has a same posting patterns as here). The overwhelming topic that gets posted are "Help" topics... meaning people that come here, dumpshock, and reddit are looking for answers to issues.

And if they are looking for answers to issues, chances are they completely happy with the product...


To get a truly random sample for your survey, you would need a mailing list from say DTRPG, then email out the questionnaire and compile the results.

At best, you have taken the temperature of those that visit those locations, but possibly not the actual customer base.



Basically, 
you are asking people who have most likely come here seeking help for a problem, if they have a problem with the product. I wonder if that could skew your results any

This is why I have mentioned, ad nauseum, that this measures users on this site, Dumpshock, and reddit. So, this survey is a result of the opinions of people who use those sites. Mailing list from DTRPG won't cover the whole playerbase either, because DTRPG isn't the only place that sells SR.

Heck, that's even in the part you quoted "we can apply this to our population on /r/shadowrun, the official forums, and dumpshock."

I agree that it's not a measure of the customer base- I have also mentioned this ad nauseum.

This is an accurate measurement of the sampled populations. As with any survey (including your mailing list suggestion), we can expect some selection bias because responding to this survey was voluntary. So, this does not contain results from people on those three sites who do not like taking surveys. If there is a significant difference in opinion regarding 6e in the population who likes taking surveys versus the population who does not like taking surveys, this will affect the results. I left the survey running for a week so that I did not exclude people in different time zones or with different work schedules.

~~~
 
I think our difference of opinion then comes from whether or not this is useful. Well, useful to who?

A writer isn't going to care about the opinions of folks who haven't read the book- that makes sense. They've already been paid, so the total sales don't matter.

A publisher will care about people who were potential customers but did not buy the book- they need to identify the reasons why in order to make more sales next time. Whether they read the book or liked it is irrelevant.

There's also the question of how much of the total shadowrun community that these three sites take up. I'm sure folks will throw out "It's a lot!" and "It's not a lot!" with a whole lot of conjecture, so I'll just do that to get that over with.

You can declare it dead whenever you want. However, Catalyst is probably not going to agree with you.

I wish I could weigh in on this topic, but NDA so...

You always gotta tease XD I think we can infer from the insta pic CGL dropped a bit ago that at least one more book is coming

Here's a better question you may be able to answer- do the people you work with at CGL enjoy working on Shadowrun?

Membership on the forums goes like this: Out of the 7498 registered user (current count as of right now [before I start cleaning spammers]).

Between 100-200 people have over 1,000 posts. but only 29 of those have posted in the last 30 days.
There's over 700 people with over 100 posts, but only 75+ have posted in the last 30 days.
Over 1700 members have never posted at all, but I know from looking a logs, they are still active, spending time just reading.

Thank you so much! So, spitballing, potentially 2000 people are "active," or a little over 25% of registered users.

[FastJack]

Thank you so much! So, spitballing, potentially 2000 people are "active," or a little over 25% of registered users.

You and I have very different definitions of "active".

61 forum members with more than 100 posts have been on the boards in the last week. If we look at people with 1 or more posts, it's somewhere between 100-200 (I can't get an exact count due to how the member list search works). There have been 20 new topics and 264 posts in the past week (I'm not sure if it counts spam or not against that, but I know I've deleted at least 50 spam posts in this week). It's pretty much those that are in this thread, plus a couple of outliers. Being generous, I'd put it at about 50 "active" (i.e., participating, not just reading) users. Of the approximately 7500 registered users, over 6700 have not been on the boards in more than six months. You're spitball is way off, and is actually closer to 0.67% of registered users.

Put that with your survey, means you got about 34% of the active users on the boards to answer your survey, or about 0.23% of registered users. Reaver's actually got it pretty right from what I see. Most people come on to ask questions, be they about rules or when the next book is announced, and a small number of people are on here as hardcore fans. I mean, of the top ten all time posters, 3 haven't been on the boards for more than a year, and one of them hasn't been seen on here in almost six years.

[0B]

Thank you so much! So, spitballing, potentially 2000 people are "active," or a little over 25% of registered users.

You and I have very different definitions of "active".

61 forum members with more than 100 posts have been on the boards in the last week. If we look at people with 1 or more posts, it's somewhere between 100-200 (I can't get an exact count due to how the member list search works). There have been 20 new topics and 264 posts in the past week (I'm not sure if it counts spam or not against that, but I know I've deleted at least 50 spam posts in this week). It's pretty much those that are in this thread, plus a couple of outliers. Being generous, I'd put it at about 50 "active" (i.e., participating, not just reading) users. Of the approximately 7500 registered users, over 6700 have not been on the boards in more than six months. You're spitball is way off, and is actually closer to 0.67% of registered users.

Put that with your survey, means you got about 34% of the active users on the boards to answer your survey, or about 0.23% of registered users. Reaver's actually got it pretty right from what I see. Most people come on to ask questions, be they about rules or when the next book is announced, and a small number of people are on here as hardcore fans. I mean, of the top ten all time posters, 3 haven't been on the boards for more than a year, and one of them hasn't been seen on here in almost six years.

Yeah, that's fair to say. I aim high on population because that leads to larger intervals: larger intervals are more likely to contain the "true" value of the population.

When I say "active users," I mean anyone who might've seen the post and clicked on it. I don't ask about their posting habits in the survey, so it'd be wrong to assume that everyone who takes the survey is a regular poster.

If we say that only a third of users across all boards are sampled (60,000 -> 20,000), then the intervals become smaller. More precise, possibly less accurate if we're underestimating the population.

Given a total sampled population of 20,000, we have the following:

99% confidence that between 1.45% and 19.33% of the population likes SR 6e.
95% confidence that between 3.59% and 17.19% of the population likes SR 6e.
90% confidence that between 4.68% and 16.10% of the population likes SR 6e.
50% confidence that between 8.05% and 12.73% of the population likes SR 6e.

This means that there is a smaller margin of error in the survey, but only by about a hundreth of a percent or less.

Again, the proportion of the population that is sampled is not the determinant of how accurate it is. The OpenStax book explains this better than I can, I don't expect you to take me at my word. This isn't a debatable thing, it's how statistics work.

Whether this survey is an accurate sampling of the playerbase at large, or even of the online community, is far more debateable. I do not think it is a good measurement of the playerbase, and I'd say it's a low confidence measurement of the online community.

[penllawen]

Two things I want to highlight/call out/reinforce:

Again, the proportion of the population that is sampled is not the determinant of how accurate it is. The OpenStax book explains this better than I can, I don't expect you to take me at my word. This isn't a debatable thing, it's how statistics work.

This (like so much of statistics beyond the absolute basics) is counter-intuitive, but is completely correct.

And:

99% confidence that between 1.45% and 19.33% of the population likes SR 6e.

Even at these very tight confidence intervals and (IMO) pretty conservative assumptions about the underlying population, this still paints a pretty poor picture for 6e's reputation with Shadowrun players. This is about as forgiving to 6e as you can possibly be in interpreting these results. And it's still bad.

[FastJack]

You guys do remember I have a degree in Mathematics, right? Not Math-Education, but straight up Math...

I'm not disagreeing with the explanations to prove how accurate your survey is, but you're going on and on about the population of the survey when the true discussion on making a qualitative survey accurate is to make sure the sampling is as random as possible. The method you used is convenience sampling and it's the most biased of all types, because you're asking people to respond if they have the time. This leads to the only people responding as those that want to skew the survey to their "side".

[0B]

Convenience sampling bias is a thing. I'd imagine that people who are really interested or really disinterested in the topic are more likely to respond. However... for what purpose? I'd also be curious how you would do a voluntary, true-random survey, while still targeting these specific populations.

Also, I got a new survey out- this one's about your favorite editions

[Michael Chandra]

You guys do remember I have a degree in Mathematics, right? Not Math-Education, but straight up Math...

And there's a metal train that's a mile long
And at the very back end a lightning bolt struck her,
How long 'til it reaches and kills the driver,
Provided that he's a good conductor?

[DigitalZombie]

Convenience sampling bias is a thing. I'd imagine that people who are really interested or really disinterested in the topic are more likely to respond. However... for what purpose? I'd also be curious how you would do a voluntary, true-random survey, while still targeting these specific populations.

Also, I got a new survey out- this one's about your favorite editions

Link isnt workin :/


I also did a poll on 6th ed quite  a while ago.
https://forums.shadowruntabletop.com/index.php?topic=30525.0

The set up is somewhat different. And also the purpose.

[FastJack]

Convenience sampling bias is a thing. I'd imagine that people who are really interested or really disinterested in the topic are more likely to respond. However... for what purpose? I'd also be curious how you would do a voluntary, true-random survey, while still targeting these specific populations.

Also, I got a new survey out- this one's about your favorite editions

The only way to get the random sampling is to either go to GenCon or something similar and survey passerbys, or to take a random sample of your sample (but that requires a lot more responses).

[Lormyr]

Re: Reaver,

I get your point, and I certainly agree that it is a factor in terms of both posters in general, as well as posters who replied to the survey. All I am saying is that to sum it up as you did (paraphrased: only those with problems with the product would come to the forum) is patently false, myself being proof of that.

I wish I could weigh in on this topic, but NDA so...

Didn't you ragequit up thread? Guess that didn't last!

0B, FastJack, Reaver and all you other guys talking hard statistical deep dive, the precision you are taking your analysis too is legit out of my depth, but I appreciate you doing it. Regardless of how representative of the entire fan base / consumer base / whatever the survey accurately represents, what demographic of those bases do you believe are the types that would take the survey? Like are you saying only the most embittered who want to see SR6 crash and burn would bother taking the survey, or only those more invested than a surface level would take the time, ect?

[FastJack]

Didn't you ragequit up thread? Guess that didn't last!

Please stop antagonizing MC or an official warning will come down.

[Lormyr]

Whatever man, do it. I don't see you "come down" on him when he is antagonizing the rest of us.

[FastJack]

Whatever man, do it. I don't see you "come down" on him when he is antagonizing the rest of us.

I'm not reading through every single thread. This one you did right in front of me. And if he comes down, there's this handy little "report to moderator" button in the forum post and I guarantee action will be taken as needed.

And, before you start the "favoritism" argument, I'd point out that MC has had four official warnings in the last 18 months while you've had one.

[Lormyr]

Sure, but I can handle it myself, I don't need to summon daddy to hide behind your skirt.

[FastJack]

Sure, but I can handle it myself, I don't need to summon daddy to hide behind your skirt.

Official warning #2 sent. Care to try again and go for a five day ban?