Technology & Innovation 12(3) Abstracts

Return to Technology & Innovation main page>

Technology and Innovation, Vol. 12, pp. 189–195, 2010
1929-8241/10 $90.00 + .00
DOI: 10.3727/194982410X
E-ISSN 1949-825X12895770313952
Copyright © 2010 Cognizant Comm. Corp.
Printed in the USA. All rights reserved

Making Judgments About Grant Proposals: A Brief History of the Merit Review Criteria at the National Science Foundation

Marc Rothenberg

National Science Foundation, Arlington, VA, USA

This is a brief study of the changes in the merit review criteria for proposals submitted to the National Science Foundation (NSF) over its 60-year history. Because far more worthy proposals are received than are fundable, it has been necessary for the NSF to develop review criteria to distinguish among meritorious proposals. For reasons of politics and policy, NSF has had to consider criteria other than simply good science—what are now known as “broader impacts.” This study shows that the general nature of the criteria has not changed over the years. Instead, the NSF has fought a continuing battle to clarify the criteria and persuade the peer communities to use the criteria as set down. The trend from the 1960s has been to reduce the number of criteria, but to broaden the definition of those that remain.

Key words: National Science Foundation; Merit review criteria; Peer review

Address correspondence to Marc Rothenberg, NSF Historian, National Science Foundation, 4201 Wilson Boulevard, Arlington, VA 22230, USA. Tel: (703) 292-7729; E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it


Technology and Innovation, Vol. 12, pp. 197–201, 2010
1929-8241/10 $90.00 + .00
DOI: 10.3727/194982410X12895770313998
E-ISSN 1949-825X
Copyright © 2010 Cognizant Comm. Corp.
Printed in the USA. All rights reserved

Challenges for International Peer Reviewing of Research Proposals

Guntram Bauer

Human Frontier Science Program (HFSP), Strasbourg, France

Recent governmental programs to invest in science and technology as a countermeasure to the economic downturn promise an unparalleled incentive for basic research but may also result in thousands of additional research proposals submitted to national research councils. The unprecedented magnitude of these investments represents a challenge for the organizations that implement them and justifies that international peer review procedures are the preferred method for evaluation of the proposals. This study presents the perspective of the Human Frontier Science Program (HFSP) as an example for the successful conduct of international peer review. The HFSP operates a small funding program to support intercontinental collaboration and training in the basic life sciences. The program funds international, interdisciplinary, and innovative projects by encouraging novel collaborations of scientists from biology and neighboring disciplines. Because of its global reach, international peer review procedures and the rigorous selection of HFSP awards are essential for its success. Diversity at the reviewer level in terms of nationalities and scientific expertise reduces the risk of conflicts of interest and assures fair evaluation of academic track records. During the selection process reviewers should follow clear criteria so as to build up credibility and reputation.

Key words: Peer review; Funding organization; International collaboration; Basic research; Conflict of interest

Address correspondence to Dr. Guntram Bauer, Director of Fellowships, Human Frontier Science Program (HFSP), 12 quai Saint-Jean, 67080 Strasbourg, France. Tel: 00 33 3 88 21 51 28; Fax: 00 33 3 88 32 88 97; E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it


Technology and Innovation, Vol. 12, pp. 203–211, 2010
1929-8241/10 $90.00 + .00
DOI: 10.3727/194982410X12895770314032
E-ISSN 1949-825X
Copyright © 2010 Cognizant Comm. Corp.
Printed in the USA. All rights reserved

The Role of Peer Review in Policy Decisions

A. Alan Moghissi and Michael S. Swetnam

Potomac Institute for Policy Studies, Arlington, VA, USA

There is a universal recognition that science upon which policies are based must be reliable or, as we call it, Best Available Science (BAS). Peer review is acknowledged to be an effective process to ensure that policy decisions are based on BAS. Despite the recognition of the value of peer review there appears to be some confusion on what is peer review and how it is to be used in policy decisions. This article identifies several categories of peer review and their respective applications. These include: 1) independent peer review for publication in scientific journals, 2) independent peer review for evaluating projects submitted for funding by governmental and other agencies, 3) independent peer review of documents prepared by various agencies as a foundation of regulatory and other decisions, 4) independent peer review of competency of individuals and organizations to be given permits or license to perform specific tasks, 5) review of various activities using a process that does not meet requirements of independent peer review, and 6) review within an organization, sometimes referred to as a vertical review. This article describes the foundation of independent peer review, addresses various variations of the currently used processes, and identifies strengths and weaknesses of each process. It also describes the contribution of each type in policy decisions and concludes that a sound policy decision must use science that has been directly or indirectly subjected to independent peer review.

Key words: Peer review; Scientific assessment; Vertical review

Address correspondence to A. Alan Moghissi, Ph.D., President, Institute for Regulatory Science, P. O. Box 7166, Alexandria, VA 22307, USA. Tel: (703) 765 3546; Fax: (703)765 3143; E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it


Technology and Innovation, Vol. 12, pp. 213–224, 2010
1929-8241/10 $90.00 + .00
DOI: 10.3727/194982410X12895770314078
E-ISSN 1949-825X
Copyright © 2010 Cognizant Comm. Corp.
Printed in the USA. All rights reserved

The Use of Societal Impacts Considerations in Grant Proposal Peer Review: A Comparison if Five Models

J. Britt Holbrook

University of North Texas, Denton, TX, USA

Increasing demands on the part of the public for a demonstrable return on their investment in scientific and technical research have led to the widespread introduction of considerations of societal impacts into the peer review processes at public science and technology funding agencies. This answer to the accountability challenge also introduces a peculiar strain on peer review: expertise in particular areas of scientific and technical research is no guarantee of expertise in addressing the societal impacts of proposed research. Presenting preliminary results of a larger study, this article describes five current models of the peer review of grant proposals and shows that different agencies have very different ways of incorporating societal impacts considerations. The article also elucidates a notion of theoretical adequacy, which will be used to determine whether and how some peer review processes are better than others. The objectives of this article are to lay out the description of the agencies  and to offer a preliminary assessment of each model’s theoretical adequacy. The objective of our larger study is to determine the best ways to incorporate societal impacts considerations into the peer review of grant proposals, thus helping funding agencies respond to the demand for demonstrable results.

Key words: Peer review; Societal impacts; Accountability; Expertise

Address correspondence to J. Britt Holbrook, University of North Texas, 1155 Union Circle #310920, Denton, TX 76203-5017, USA. Tel: (940) 565-4048; Fax: (940) 565-4448; E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it


Technology and Innovation, Vol. 12, pp. 225–232, 2010
1929-8241/10 $90.00 + .00
DOI: 10.3727/194982410X12895770314113
E-ISSN 1949-825X
Copyright © 2010 Cognizant Comm. Corp.
Printed in the USA. All rights reserved

Improving Peer Review in the Federal Government

Elmer Yglesias

Science and Technology Policy Institute, Institute for Defense Analyses, Washington, DC, USA

Maintaining the integrity of peer review in the Federal government is vital but there are indications that the system may be overstretched and prone to error. The author discusses various methods that ensure current Federal peer review systems are reliable and valid. The article maps how an error can occur, and what safe guards can be implemented to prevent it. On the basis of this study, the author suggests that the best solution to ensure a review’s integrity may be an Internet-based calibration system.

Key words: Peer review; Federal; Error; Reliability; Validity

Address correspondence to Elmer Yglesias, Science and Technology Policy Institute, Institute for Defense Analyses, 1899 Pennsylvania Avenue NW, Suite 520, Washington, DC 20006, USA. Tel: (202) 419-3726; Fax: (202) 419-3720; E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it or This e-mail address is being protected from spambots. You need JavaScript enabled to view it


Technology and Innovation, Vol. 12, pp. 233–240, 2010
1929-8241/10 $90.00 + .00
DOI: 10.3727/194982410X12895770314159
E-ISSN 1949-825X
Copyright © 2010 Cognizant Comm. Corp.
Printed in the USA. All rights reserved

CDMRP: Fostering Innovation Through Peer Review

E. Melissa Kaime,* Katherine H. Moore,* and Steven F. Goldberg†

*Congressionally Directed Medical Research Programs, Fort Detrick, MD, USA
†Center for Peer Review and Science Management, SRA International, Frederick, MD, USA

The Congressionally Directed Medical Research Programs (CDMRP) is a collection of 18 individual programs that seek to find and fund the best research to eradicate diseases and support the warfighter for the benefit of the American public. In fulfilling its mission, CDMRP emphasizes innovative, high-risk, high-gain research that might otherwise not be funded and projects that forge new collaborations in furtherance of important research objectives. Research proposals (applications) are reviewed using a two-tiered process that includes peer review panels that evaluate scientific merit, innovation, and impact, followed by an external review that makes funding recommendations based on programmatic intent and portfolio balance. At both levels of review, CDMRP’s processes are distinguished by the inclusion of consumer advocates, who are integral to the program’s ability to focus on research that will have an impact on the communities affected by the relevant illness, injury, or disorder. Scientific peer review is executed using a dynamic and flexible process and produces a robust and comprehensive summary statement that serves as the basis for the second tier of review, informs subsequent award negotiations, and provides valuable feedback to all applicants. In combination with a strong commitment to integrity and transparency, CDMRP’s peer review processes support the organization’s mission to fund innovative, high-impact research.

Key words: Peer review; Innovation; Consumer involvement

Address correspondence to E. Melissa Kaime, M.D., Director, Congressionally Directed Medical Research Programs, 1077 Patchel Street, Fort Detrick, MD 21702, USA. Tel: 301-619-7071; Fax: 301-619-7796; E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it


Technology and Innovation, Vol. 12, pp. 241–247, 2010
1929-8241/10 $90.00 + .00
DOI: 10.3727/194982410X12895770314195
E-ISSN 1949-825X
Copyright © 2010 Cognizant Comm. Corp.
Printed in the USA. All rights reserved

Uses and Needs for Peer Review in Army Medical Research

John F. Glenn

U.S. Army Medical Research and Materiel Command, Fort Detrick, MD, USA

The U.S. Army Medical Research and Materiel Command (USAMRMC) manages and executes multiple research programs focused on both military and civilian health problems. As a policy, all Command-sponsored research is subjected to peer review for both science and program alignment, but the details of the peer review process vary according to the objectives of each program. USAMRMC’s core mission research programs conduct product-focused research aimed at delivering new products and information on health matters of military operational importance. As such, these programs have needs for peer review that go beyond ensuring high-quality science; peer review also ensures that programs are directed toward all items on the critical path for product transition, operating efficiently, and moving in a direction that is likely to lead to the accomplishment of their technical objectives. USAMRMC uses both prospective peer review of research proposals and retrospective and prospective peer review of research program subareas to further its core research mission. Peer review is also needed to validate the end products of research, particularly medical information products. While this form of peer review has not been universally employed, greater utilization of peer review can contribute to wider adoption of evidence-based medicine within the defense community.

Key words: Peer review; Research program management; Biomedical research

Address correspondence to Dr. John F. Glenn, Department of the Army, Headquarters USAMRMC, 504 Scott Street, Fort Detrick, MD 21702-5012, USA. Tel: 301-619-7363; Fax: 301-619-1889; E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it


Technology and Innovation, Vol. 12, pp. 249–253, 2010
1929-8241/10 $90.00 + .00
DOI: 10.3727/194982410X12895770314230
E-ISSN 1949-825X
Copyright © 2010 Cognizant Comm. Corp.
Printed in the USA. All rights reserved

Alternatives to Independent Peer Review: A Case Study

Wren Prather-Stroud

Center of Excellence for Hazardous Materials Management, Carlsbad, NM, USA

There is a need for review processes that can assist projects that do not require independent peer review. Termed “technical review,” these processes can be designed for specific projects that require experts to review the project and make recommendations. This article describes an example of the development of this type of review. As a sample case, a process for signing conservation agreements between government agencies and landowners/users is explained. Solutions for preserving habitat and promoting the population of two threatened species, the Lesser Prairie Chicken and the Sand Dune Lizard, are proposed and a process for having experts weigh these solutions and make recommendations to the concerned parties is mapped out. Independent peer review would not be a good fit for these reviews due to short turn-around time and lack of need for finding peers with no conflict of interest. Development of this project-specific process is an example of how to create a technical review process that functions exactly as needed. The process delineated in the article has generated widespread interest. In the first 6 months of the existence of the agreements, 36 parties have signed letters of intent to participate. Both the government agencies involved and the landowners/users agree that the process is fair. There has been insufficient time since implementation of these measures to quantify the effects on the target species population. This straightforward technical review is inexpensive to implement, tailored to accomplish the goals of the participants, and effective. Due to the nature of the project and the high volume of reviews associated with it, the technical review procedure that was developed is a perfect fit.

Key words: Technical review; Threatened species; Conservation; Lesser prairie chicken; Sand dune lizard; Endangered Species Act

Address correspondence to Wren Prather-Stroud, 404 West Riverside Drive, Carlsbad, NM 88220, USA. Tel: 575-302-5153; Fax: 575-885-0776; E-mail: This e-mail address is being protected from spambots. You need JavaScript enabled to view it