HOW TO EVALUATE A PAPER(From 小木虫)

HOW TO EVALUATE A PAPER？？

Questions:

A thorough understanding and evaluation of a paper involves answering several questions:
a. What questions does the paper address?
b. What are the main conclusions of the paper?
c. What evidence supports those conclusions?
d. Do the data actually support the conclusions?
e. What is the quality of the evidence?
f. Why are the conclusions important?
a. What questions does the paper address?

Before addressing this question, we need to be aware that research in biochemistry and molecular biology can be of several different types:
Type of research Question asked:
Descriptive  What is there? What do we see?
Comparative How does it compare to other organisms? Are our findings general?
Analytical   How does it work? What is the mechanism?
Descriptive research often takes place in the early stages of our understanding of a system. We can't formulate hypotheses about how a system works, or what its interconnections are, until we know what is there. Typical descriptive approaches in molecular biology are DNA sequencing and DNA microarray approaches. In biochemistry, one could regard x-ray crystallography as a descriptive endeavor.
Comparative research often takes place when we are asking how general a
finding is. Is it specific to my particular organism, or is it broadly applicable? A typical comparative approach would be comparing the sequence of a gene from one organism with that from the other organisms in which that gene is found. One example of this is the observation that the actin genes from humans and budding yeast are 89% identical and 96% similar.
Analytical research generally takes place when we know enough to begin formulating hypotheses about how a system works, about how the parts are interconnected, and what the causal connections are. A typical analytical approach would be to devise two (or more) alternative hypotheses about how a system operates. These hypotheses would all be consistent with current knowledge about the system. Ideally, the approach would devise a set of experiments todistinguish among these hypotheses. A classic example is the Meselson-Stahl experiment. Of course, many papers are a combination of these approaches. For instance, researchers might sequence a gene from their model organism; compare its sequence to homologous genes from other organisms; use this comparison to devise a hypothesis for the function of the gene product; and test this hypothesis by making a site-directed change in the
gene and asking how that affects the phenotype of the organism and/or the biochemical
function of the gene product.
Being aware that not all papers have the same approach can orient you towards
recognizing the major questions that a
paper addresses.
What are these questions? In a
well-written paper, as described above, the Introduction generally goes from the general to the specific, eventually framing a question or set of questions. This is a good starting place. In addition, the results of experiments usually
raise additional questions, which the authors may attempt to answer. These questions usually become evident only in the Results section.

b. What are the main conclusions of the paper?
This question can often be answered in a preliminary way by studying the abstract of the paper. Here the authors highlight what they think are the key points. This is not enough, because abstracts often have severe space constraints, but it can serve as a starting point.
Still, you need to read the paper with this question in mind.


c. What evidence supports those conclusions?
Generally, you can get a pretty good idea about this from the Results section. The description of the findings points to the relevant tables and figures. This is easiest when there is one primary experiment to support a
point. However, it is often the case that several different experiments or approaches combine to support a particular conclusion.
For example, the first experiment might have several possible interpretations, and the later ones are designed to distinguish among these.
In the ideal case, the Discussion begins with a section of the form "Three lines of evidence provide support for the conclusion that... First, ...Second,... etc." However, difficulties can arise when the paper is poorly written (
see above). The authors often do not present a concise summary of this type, leaving you to make it yourself. A skeptic might argue that in such cases the logical structure of the argument is weak and is omitted on purpose! In any case, you need to be sure that you understand the
relationship between the data and the conclusions.

d. Do the data actually support the conclusions?
One major advantage of doing this is that it helps you to evaluate whether the conclusion is sound. If we assume for the moment that the data are believable (see next section), it still might be the case that the data do not actually support the conclusion the authors wish to reach. There are at least two different ways this can happen:

i. The logical connection between the data and the interpretation is not sound
ii. There might be other interpretations that might be consistent with the data.

One important aspect to look for is whether the authors take multiple approaches to answering a question. Do they have multiple lines of evidence, from different directions, supporting their conclusions? If there is only one line of evidence, it is more likely that it could be interpreted in a
different way; multiple approaches make the argument more persuasive.
Another thing to look for is implicit or hidden assumptions used by the authors in interpreting their data. This can be hard to do, unless you understand the field thoroughly.


e. What is the quality of that evidence?
This is the hardest question to answer, for novices and experts alike. At the same time, it is one of the most important skills to learn as a
young scientist. It involves a major reorientation from being a
relatively passive consumer of information and ideas to an
active producer and critical evaluator of them. This is not easy and takes years to master.
Beginning scientists often wonder, "Who am I to question these authorities? After all the paper was published in a top journal, so the authors must have a
high standing, and the work must have received a critical review by experts." Unfortunately, that's not always the case. In any case, developing your ability to evaluate evidence is one of the hardest and most important aspects of learning to be a critical scientist and reader. 
How can you evaluate the evidence?
First, you need to understand thoroughly the methods used in the experiments. Often these are described poorly or not at all. The details are often missing, but more importantly the authors usually assume that the reader has a general knowledge of common methods in the field (such as immunoblotting, cloning, genetic methods, or DNase I footprinting). If you lack this knowledge, as discussed above you have to make the extra effort to inform yourself about the basic methodology before you can evaluate
the data.
Sometimes you have to go to the library, or to a lab that has a
lot of back issues of common journals, to trace back the details of the methods if they are important. One new development that eventually will make this much easier is the increasing availability of journals on the Web.
1
Second, you need to know the limitations of the methodology. Every method has limitations, and if the experiments are not done correctly they can't be interpreted. For instance, an immunoblot is not a very quantitative method. Moreover, in a certain range of protein the signal increases (
that is, the signal is at least roughly "linear"), but above a certain amount of
protein the signal no longer increases. Therefore, to use this method
correctly one needs a standard curve that shows that the experimental
lanes are in a linear range. Often, the authors will not show this
standard curve, but they should state that such curves were done. If
you don't see such an assertion, it could of course result from bad
writing, but it might also not have been done. If it wasn't done, a
dark band might mean "there is this much protein or an indefinite amount
more". 

Third, you need to distinguish between what the data show and what the authors say they show. The latter is really an interpretation on the authors' part, though it is generally not stated to be an interpretation. Papers usually state something like "the data in Fig. x show that ...". This is the authors' interpretation of the data. Do you interpret it the same way? You need to look carefully at the data to ensure that they really do show what the
authors say they do. You can only do this effectively if you understand the methods and their limitations.

Fourth, it is often helpful to look at the original journal (or its electronic counterpart) instead of a photocopy. Particularly for half-tone figures such as photos of gels or autoradiograms, the contrast is distorted, usually increased, by photocopying, so that the data are misrepresented.

Fifth, you should ask if the proper controls are present. Controls tell us that nature is behaving the way we expect it to under the conditions of the experiment. If the controls are missing, it is harder to be confident that the results really show what is happening in the experiment. You should try to develop the habit of asking "where are the controls?" and looking for them.

f. Why are the conclusions important?
Do the conclusions make a significant advance in our knowledge? Do they lead to new insights, or even new research directions?
Again, answering these questions requires that you understand the field relatively well.

The content comes from http://emuch.net/html/200605/246755.html