Reply-to: meyer@newslink.org
Priority: normal
X-mailer: Pegasus Mail for Win32 (v2.41)
Sender: owner-online-news@marketplace.com
Precedence: bulk
Status: RO
X-Status: 

Excerpts from Joe Shea's message at 0:30 on 28 Aug 96:

> As we established the last time we went through this, Eric's
> alleged Top 10 has no validity other than as something he "sampled" from
> visits to his site.

This is my one and only response to this note.

We established no such thing, Joe. 

And frankly, the terms you toss around in your note reveal that you
have a shockingly limited knowledge of how to conduct legitimate
public opinion research. 

How a question is phrased, what sample frame is used and how the
sampling is performed are much more significant than the raw numbers
of responses. But don't trust me on this, Joe. Go read a book like
Precision Journalism. Or go enroll in an introductory course in 
marketing or communications research methods.

Methodologically, our list is based on responses to a question,
buried well within a survey, asking people to identify which one
online publication they read most frequently. 

Asking the question this way makes it much more likely to be a
reliable gauge of actual reading preferences than simply asking
people to vote in a straightforward popularity poll. In both cases,
you have the limitation of self-reporting -- limitations that, as
Paul Klein found with television viewing, can be severe. However, we
manage to avoid a large participation bias in our survey by burying
the question in a questionaire that does not have as its fundamental
purpose a ranking of "favorite" sites. We're talking about behavior,
not favor. And we aren't trying to see who can rally more people to 
go punch a site's name into a ballot. We, by the way, do not even use 
a ballot for this. Everything is done open-ended. So, no, the 
American Reporter didn't get left off our ballot. Even the New York 
Times had to be written in. This is yet another way of addressing 
reliability.

Our survey has been, and continues to be, conducted with
self-selected sampling procedures, just as almost all online surveys
are done. However, we twice have taken the same set of questions to a
controlled sample frame, sampled by scientific interval means, and
have found no statistically significant differences in responses
between the self-selected sample and the interval sample. (We used
SPSS, a standard mainframe statistical analysis tool, for the data
analysis.) By replicating the self-selected survey with controlled
samples, we have taken a methodological step that few if any online
surveys have bothered to take. But it is a very essential step if you
are to hope to increase the survey's validity. 

Our overall number of responses is well in excess of 2,500. But
frankly, Joe, once you get past around 600 in a scientific sample
that becomes largely irrelevant unless you are looking for crosstabs
with subgroups. If you don't believe me, consider this: I'll bet 
someone took some sort of survey a couple of weeks ago in whcih 
20,000 people were asked for whom they would vote for president, and 
Bob Dole won handily. The survey, of course, was conducted at the 
Republican convention. Gallup, meanwhile, is talking to only 600 to 
1,200 people nationwide in making its election predictions. Think 
about it.

Our rankings themselves are a weighted average of rankings for the
past 12 months. This is done to diminish the potential for "flash in
the pan" responses. In many lists, new sites vault to the top for a
month, then are replaced by whatever else is new. We try to avoid
this by weighting the current month most heavily, then applying
steadily decreasing weights to each previous month, spanning back to
12 months ago. 

Are there flaws with our approach? Sure. I could talk about the 
limitations for hours. Even the Pew survey, which is probably the 
best done work regarding online behavior, has flaws. 

But frankly, Joe, it's a hell of a lot more methodologically sound 
than damned near anything else you'll find online. And that's a 
statement I would make regardless of whether American Reporter made 
the Top 10 or did not.
----------------------------------------------------
                  N E W S L I N K
  Online news research, consulting and publishing
----------------------------------------------------
Eric K, Meyer               3,700 media links & more
meyer@newslink.org           http://www.newslink.org
----------------------------------------------------
    Preview acclaimed guide for online publishers
            Write to report@newslink.org
+---------------------------------------------------------------------------+
This message was posted to ONLINE-NEWS. http://www.planetarynews.com/o-n.html

From owner-online-news@marketplace.com Wed Aug 28 17:07:10 1996
Received: from marketplace.com (majordom@marketplace.com [206.168.5.232]) by cnj.digex.net (8.6.12/8.6.12) with ESMTP id RAA18839 ; for ; Wed, 28 Aug 1996 17:07:06 -0400
Received: (from majordom@localhost) by marketplace.com (8.6.12/8.6.12) id IAA08020 for online-news-outgoing; Wed, 28 Aug 1996 08:18:08 -0600
Received: from server.indra.com (server.indra.com [204.144.142.2]) by marketplace.com (8.6.12/8.6.12) with ESMTP id IAA08014 for ; Wed, 28 Aug 1996 08:18:02 -0600
Received: from indra.com by server.indra.com (8.7.4/Spike-8-1.0)
	id IAA18459; Wed, 28 Aug 1996 08:08:11 -0600 (MDT)
Received: from maildeliver1.tiac.net by indra.com (8.7.4/Spike-8-1.0)
	id IAA02408; Wed, 28 Aug 1996 08:08:09 -0600 (MDT)
Received: from mailserver1.tiac.net (mailserver1.tiac.net [199.0.65.232]) by maildeliver1.tiac.net (8.6.12/8.7.4) with ESMTP id KAA29875 for ; Wed, 28 Aug 1996 10:11:48 -0400
Received: from vgiulian (newmpub.tiac.net [206.119.206.92]) by mailserver1.tiac.net (8.6.12/8.7.4) with SMTP id KAA02877 for ; Wed, 28 Aug 1996 10:09:51 -0400
Message-Id: <199608281409.KAA02877@mailserver1.tiac.net>
X-Sender: newmpub@pop.tiac.net
X-Mailer: Windows Eudora Pro Version 2.1.2
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
To: online-news@planetarynews.com