~ Evaluating essays ~
         Petit image    Evaluating
essays
Version April 2006

[Old methods, recent syntoms]    [Negative evaluation rules]    [Essays]

"If you just torture the data long enough, they will confess"

"Never accept results out of hand, always question wether your analysis may have lead you to a faulty solution"

Old methods, recent syntoms

Reading the material you find around the web about evaluation, you will notice that it boils down to some common 'basic' rules. The problem is that, like all generic rules, they are -by a long shot- not valid enough. Therefore, even if they were all valid, the capability to 'feel' if a site is bogus (or not) will still require some 'Fingerspitzengefühl' which amounts to a combination of experience and subtle intuition, to acquire the uncanny insight of accurately perceiving fake constructions and attitudes.
Basically I believe that looking for information on the Internet you will need to use the same critical evaluative skills that you would use when choosing or reading a book, when choosing or using a paper index, when choosing or evaluating a musical score, or when deciding about the value of a database. For this reason you should by all means and aforemost head [Sielaff's] teachings.


Let's have a closer look, evaluating critically the data you'll find is more and more important within the morass of networked data, where the few valuable nuggets are submerged by an incredible amount of junk.
  1. Internal (Intrinsic) clues to the 'credibility' of the target site
    1. Author's own qualifications
      Of paramount importance, yet watch it! You must not just believe what HE HIMSELF says! Stalking and combing techniques may come handy to check his assertions.
    2. Bibliographical precision
      Watch it! Many studies LOOK professional, but scanning the references you'll quickly realise they are not even at ordinary universitary level (besides NOT ALL universities are equal: is the university where the study / author 'dwells' an university, or study institute, with a certified track of competence or is it just "some kind" of university? You would not believe how many universities nowadays are fighting hard to keep at levels any good secundary school had thirty years ago :-(
    3. Style patterns and Language correctness
      Careful, very careful with this on a multilinguistical web!
      Like the previous one, a difficult clue to use. Take for instance a good messageboard, with significative multinational partecipation, like one of these. You may (may underlined) find snippets of incredibly valuable knowledge in pidgin english, full of mistakes, dotted with unscientific references. On the other hand, often enough, you'll find preposterous university-thesis with lengthy wording, perfectly correct, and exact scientific layout, yet with no knowledge value whatsoever.
    4. Update frequency
      Quite important. You may reasonably expect better value and competence on a site that is continuously updated vis-a-vis a site that 'does not move' for months (or years). The usual caveats apply, though: updates must regard CONTENT, not just frills.
    5. Timestamps (chronological indications)
      VERY important: veritatem dies aperit (Time discovers truth). You'll quickly notice how many sites DO NOT DATE their documents, and even go to great lenghts in order to avoid the possibility that you could infer their real publication date. The simple reason for this is that those sites FEAR 'LOOKING' OBSOLETE. Our perception (skewed by propaganda and advertisement) is that 'new' research means 'better' research. This is absolutely not true (the contrary is quite often the case, see [Sielaff's] teachings) but this untruth, being perceived as such, is worshipped on lazy sites ('Readers should not know that this stuff is rather old').
    6. Bias
      Most 'apparently' academic sites avoid polemic and prefer an "equilibrated essay style". This is often due to the 'mimicking' of the scientific style as perceived by the broad public. In the reality valuable scientific research does not avoid confrontation, never. To establish a new theory (not just trivial addendums) means mostly to DESTROY established (and 'cherished' and 'bread-gaining') theories. This is even more true on the Web, where you can "publish and be damned" with an ease that would be impossible in the academic world.
      Yet bias can indeed constitute a clue in 'negative' sense as well: excessive obsession with own theories (taking them very seriously) is a classical sign of boiled air content.

  2. External (also non-intrinsic) clues to the 'credibility' of the target site
    1. HTML Style of the site (VERY careful with this!)
    2. art and kind of sites linking to the target site
    3. art and kind of sites your target site links to

  3. 'Librarian' clues to the 'credibility' of the target site
    1. is there any feedback provision?
    2. how is the indexing of the target site performed?
    3. is the target site easy to use and navigate?
    4. does its content addresses 'the real needs' of his users?

Some of these rules are (at least in part) correct, yet some of them are QUITE obsolete. This being the case -for instance- for rule 1.3 (style and language). this kind of approach does not have much sense on a more and more pidging-english and multi-cultural web, where a very valuable apport can be expressed with a very poor english. You should not care about the presentation, spelling, or grammar of the written work that you evaluate except when the wording is so unclear that the message results unclear, imprecise, or ambiguous.
I am personally unconvinced by rule 1.6: the apparent lack of bias and political correctness of the textes being often just a simple rethorical cover for very partisan positions ("look! It looks really like I'm really neutral and unbiased, my dear friend and future client"). Note that the viewpoint of the site may be explicit -say included in a scope statement- or you may be able to confirm your suspicions only analyzing the point of view that 'transpares' through the contents of the site, which you should by all means always do, scepticism is a very sound attitude when perusing the web.
Rule 2.1, on the other hand, is patently absurd only if it tries to identify a 'positive' style for evaluation purposes, but I personally found that it is after all a useful rule for judging immediately the 'negativeness' of the sites you found, as I will explain below.

But the rules above need to be implemented:
A common problem is due to the many 'hidden' forms and techniques to smuggle advertising inside the pages you will find, which seem definitely more difficult to differentiate - on the net  than in printing, on the streets or in mass media & television channels. The various techniques used for [reality cracking] purposes can come - I believe - very handy in such contexts.

Note that very often the commercial label of a given font is blurred: a growing number of sites may have started out because some people felt that the content belonged on the web, but now many of these sites wish to take advantage of their 'position'. I doubt that you'll be able to find nowadays many "important" sites without any advertisement whatsoever (bar mine :-)
Your own maturity is required: the important thing to pay attention to is whether a site has valuable content or not and whether its presentation or biases make any difference in terms of what you need to get out of it.

Yet another problem is due to the incredible "individual" possibilities and to the half-anonymous character of the media: it is definitely possible on the web for individuals to share working papers or information they have been working on very seriously, but as you can imagine, there is no guarantee that this work has undergone a rigorous review process, in fact it hasn't probably been through the peer review processes that are intrinsic to scholarship (even if those same processes are very often far from being perfect, as anyone that works in an university knows). This opens the doors to all sort of scams: you could easily (with a little phantasy and a little research) put up a beautiful 'University of Cordonshire' and start distributing interesting courses for distance learning MBAs that would have more or less the same chances of success that all the other scam schemes that already exist (have a look at the last dozen pages of each issue of 'The Economist' if you want to see a nice mix of pseudo and bogus "universities" offering MBAs aplenty and targeting ignorant CEOs that need some kind of university papers on-the-fly :-)

Anyway the fact that the work you must evaluate was not examinated and did not went through 'the trade publishing industry' has some disadvantages and not only advantages :-)

Note moreover that the 'free' characteristics of the Web stimulates the experimentation, by all types of publishers, of various forms of parallel publishing with -say- a printed magazine and the supplementary publishing of "some" information on the Internet, mostly holding back some important stuff in order to sell the print publication.
Since in most cases there is somewhere a 'correspondent' database on-line (which requires payment to be accessed), power seekers will probably - ahem - have to learn to find their way to the 'unrestricted' publications as well. It is nevertheless important to realize that most of the results you will get from a search may be (for profit reasons) in their 'crippled' form, and you will have to 'cut deeper' to get the meat, should you fancy to do it.

So, once more, I will not deny that some of the rules listed above are (at least in part) correct, but I would like to add some rules of mine, that I found VERY valuable in order to quickly evaluate results in our more and more commercially infested web.

Negative evaluation rules


Fravia's "negative" evaluation rules are in fieri and such will remain until I find some time to update them...

  1. "Slides" style crap
    More and more sites have adopted a zombizied "slide-presentation" model. You have to click a dozen times to scrap together the (meager) content that could have been presented - and more easily printed - on a single unique page. The point is exactly that: minimum data, maximum spamming.
    Such crap-sites are designed, built and 'ameliorated' on the "visual" assumption that zombies will never print any info anyway, being too busy 'butterflying' from one site to another without construct whatsoever.
    This slide-sliding is -obviously- not only limited to the net. But on the web nowadays many many sites present scarce info in a "crumble" way that bores searchers to death. See also point 7 (Powerpoint) below.

  2. Money traps

  3. link bazaars

  4. Publishing - advertising frenzy

  5. statistical tricks: tables
    Every now and then you can find some tables on the web. There are some rules that apply to ALL tables, no matter what are they specifically related to.
    Have a good look at them: tables are very easy to reverse. As a first rule: the more 'grid lines' and -even more important- the more 'white space' you see and find in those tables the harder it will be to compare data... (see on these same lines also 'the slide problem' described above). This dictates a very handy rule: lotta unused grids & space... hey! Throw that table away!

    Some useful additional 'parameters':
    Are the LABELS (of those tables) clear?
    Are the LABELS (of those tables) complete?
    Are those tables ordered in columns? (a sequence of numbers is easier to follow down a column rather than across a row, duh)
    And, this being the case... are the most important columns EVIDENT or are they buried in the middle of the tables?

  6. statistical tricks: charts & graphs
    When faced with charts, always check the vertical scale to make sure that it has not been expanded to make a small change look large or worse, inverted, to make a fall look like a rise.
    Remember that SCALES are extremely important when analyzing graphs:
    scales that do not start form zero will exagerate modest movements
    logarithmic scales will flatten out every increase.
    The moment you see a scale that do not start from zero or is logarithmic, you have VERY GOOD REASONS to suspect why they did present it so :-)

  7. Powerpoint slides
    See also point 1 about the "crumbles" of information on some sites.
    Powerpoint is THE distinctive tract of idiocy. If someone uses powerpoint, you can be SURE it is not worth remaining on the side (or in the room): go away IMMEDIATELY. Any PowerPoint presentation elevates format over content. It's commercialism stupidity at its worst, turning everything into a sales pitch. The anti-knowledge tool par excellence, loved by idiots all over the world. With powerpoint even the few pre-chewed "ideas" hidden inside the bambinesque noise, are simplified to the point that they become redundant and unclear.

Evaluation Essays

Well, the two older essays should give the reader a strong enough WARNING about possible manipulations trough the web. The first one is a 'slight fake', the second one is totally bogus (but you can immediately discern its bogus-ness only because it was purposedly written in that way: of course all sorts of more subtle alterations would be possible.

Notice also, in the http://www.searchlores.org/nobel/nobelprize.htm essay, how the very LINK to the bogus pages could be effectively obscured.


to be continued
Petit image
Back to evaluate
(c) 2000: [fravia+], all rights reserved