Shaké Ketefian1; Isabel Amélia Costa Mendes2
1University of Michigan School of Nursing, Michigan, EUA; 2University of São Paulo at Ribeirão Preto College of Nursing, SP, Brazil
Abstract. Background: Universities worldwide are seeking objective measures for the assessment of their faculties’ research products, to improve their national and international standing. Despite concerns, many have adopted bibliometric measures such as the impact factor (IF) of journals for these purposes. The objective of this study was to explore conditions created within Brazil as a result of national and institutional policies requiring faculty to publish in high IF journals, and the extent to which these facilitated or hindered the development of nursing science. Design: The design was a case study, with the country as the unit of analysis. A key informant for the country was identified to assist in identifying institutions and individuals for participation, and to provide relevant information on the context within the country. Faculty members of senior rank from six of the highly ranked nursing doctoral programs were invited to participate. Methods: A data collection instrument was developed; it has content validity. The study was approved by a human subject review committee. Five respondents provided information. All communication occurred electronically. Results: Respondents confirmed the presence of national and institution policies, but faculty committees, in the main, did not use it. There was general criticism of the policy. The policy led to competition rather than cooperation. Characterization of current published works in the country did not convey high regard for such publications. Conclusions: The policy in the country does not seem to have created an environment conducive to collaboration or interactions with international scholars, nor does it seem to have led to scientific production viewed as noteworthy. This may in part be due to the lack of general acceptance of the policy and/or the opportunities provided by newer regional data bases for indexing publications.
Keywords: Environments for scholarship; Impact factor; Publications; Brazil
Resumo. Introdução: Universidades ao redor do mundo tem procurado medidas objetivas para avaliar seus produtos de pesquisa e para melhorar suas posições nacionais e internacionais. Mesmo sendo fontes de preocupações, muitas tem adotado para este fim medidas bibliométricas como o fator de impacto (FI) dos periódicos. O objetivo deste estudo foi o de explorar as condições criadas no Brasil como resultado das políticas nacionais e institucionais que exigem que os professores publiquem em periódicos com alto FI, e em que grau isto tem facilitado ou impedido o desenvolvimento da ciência em enfermagem. Desenho: O desenho foi o de estudo de caso, com o país como unidade de análise. Um informante-chave foi identificado em nível do país para ajudar a identificar instituições e indivíduos para participação, e para fornecer informações relevantes sobre o contexto do país. Professores-sênior de seis dos programas de doutorado melhor avaliados foram convidados a participar. Métodos: Foi desenvolvido um instrumento para coleta de dados, com validação de conteúdo. O projeto foi aprovado por um comitê de ética para pesquisas envolvendo seres humanos. Cinco respondentes forneceram informações. Toda comunicação foi feita de forma eletrônica. Resultados: Os respondentes confirmaram a presença de políticas nacionais e institucionais mas, no geral, o FI não foi usado por comitês de avaliação dos docentes. A política foi sujeita a críticas gerais, por levar a competição ao invés de cooperação. A caracterização de estudos atualmente publicados no país não transmitiu grande consideração por este tipo de publicação. Conclusões: A política no país não parece ter criado um ambiente que leva à cooperação ou a interações com especialistas internacionais, nem parece ter levado a produções científicas vistas como de destaque. Isso pode ser parcialmente atribuído à falta de aceitação geral da política e/ou às oportunidades dadas por bases regionais mais recentes para a indexação das publicações.
Descritores: Ambientes para pesquisa independente; Fator de impacto; Publicações; Brasil
Introduction
Many higher education institutions around the world are seeking to improve their offerings, their research, and their standing nationally and internationally. This has led to competition and a search for objective measures to assess quality, especially as it relates to the output of the faculties. The development of bibliometric measures such as the impact factor (IF), intended as a measure of a journal’s impact, and citation analyses, which is the number of times a scientific article is cited by others (1), have spurred interest, and are now being used for a variety of purposes.
Many countries are now requiring that their university faculties publish in high IF journals, and have developed various reward systems such as cash bonuses to further spur their faculties in this regard, and are using various bibliometric measures in faculty hiring and promotion decisions (2). Despite caution expressed in the literature about unintended uses of such measures, institutions are using them for making both individual faculty decisions such as in hiring and promotion, as well as for institutional rankings, for determination of research funding to individuals and/or institutions/departments, and for national priority setting (3-4). Some authors have decried this tendency on various grounds.
An important concern in the professions has been that the peer review process does not take into account the social utility of published papers, while for the professions that practice and provide services to the public social relevance is a major concern (5). Another important concern about the mis-use of the IF is that it is being used as a measure of quality of individual articles or of a scholar’s body of work. Researchers investigated the predictive validity of IF scores in the hiring and promotion decisions of social work faculty, and found a low effect, concluding that their findings would not justify using the IF in making decisions about individuals in hiring and promotion (6).
Another team of authors investigated the relationship between quality elements of articles and frequency of citation of articles in four psychiatric journals with IFs ranging from 11.2 to .88 in 2004, covering a 9-year period (7). Quality features such as statistical errors, reporting of sample size, poorly reported research questions and the primary outcome of the study did not relate to citation counts. However, some of these same quality features were related to the visibility and prestige of the journal (in this case represented by two of the four journals with high IFs). The authors concluded that the latter findings were due to detailed author guidelines and rigorous peer review that were characteristic of the high IF journals.
Within
Many countries face similar
situations as Brazil, and yet there have been no studies in nursing that
address the consequences of this state of affairs, such as how the behavior of
scholars is affected, and most importantly, how constraints imposed by national
or institutional policies affect the environment for the development of nursing
science. This study aims to address this vacuum in our understanding with
respect to
This paper is part of a
five-country study involving
Methods
The design is a case study of
The country was selected on the
basis of what was “known” about national and institutional policies.
Selection of institutions and
participants. A key informant was
identified to provide country specific information regarding institutional
rankings and to assist in identifying study participants. The key informant was
a senior academic, holding the rank of professor in a major university, who has
held offices in professional organizations over many years, and has overall
familiarity with nursing programs and nurse academics in the country.
Given the geographic diversity of
the respondents, the most realistic method for collecting data was through a
questionnaire. The individual respondents were viewed as representing the
country, since the information sought was not about themselves but about the
policies of the country and their institutions, their views regarding such
policies and the impact these may have on the scholarly community in nursing.
Given these considerations, five respondents per country were deemed
sufficient.
Procedures. Institutional Review Board approval was obtained
(initial approval date, 10/31/2007, study ID HUM00016714) from the University
of Michigan where the first author is employed. Due to the low risk posed by
the study, the IRB did not require signed consent forms, but required that a
letter with the elements of informed consent be provided to the participants
for information only. Identified individuals were then invited to participate
by the investigator through an invitation letter, providing relevant
information on the study, and the approved consent form for information.
Several reminder letters were sent; within eight weeks five responses were
received. All communication occurred electronically.
Study instrument. A questionnaire was developed for data collection to
ensure that respondents provide information on a consistent set of issues. The
questionnaire sought to understand the extent and purposes for which
institutions and systems in the country made use of impact factor of journals
in which their faculties publish. Further, it sought to explore the issues that
are the focus of this study.
The questionnaire development was
guided by the literature in generating the questions. A draft was reviewed by
four individuals for clarity and relevance of the items to the study objective
and issues to be explored, and revisions were made accordingly; thus, the
questionnaire has content validity. Eight of the questions present a list of
statements as options, five questions require yes/no responses to be checked,
and three questions require narrative answers, for a total of 16 questions.
Table 1 provides a list of questions in the questionnaire.
Table 1 Summary of Items in the
Questionnaire
_____________________________________________________________________________________________________
1. Who uses the IF policy?
2. Purposes for which the IF is
used.
3. How do nurse scholars behave as
a result of policies to publish in high IF journals?
4. What are the hurdles nurse
scholars encounter in publishing in top-tier, internationally recognized
journals?
5. How do scholars who publish in
local/national journals perceive their contributions vs. those who publish
internationally?
6. What are the positives, if any,
of policies to publish in high IF journals?
7. How would you characterize the
current published works in your country?
8. To what extent have there been
efforts in your country to internationalize nursing journals? Check those that
apply.
9. Is there an Impact Factor [IF]
policy in use in your country and institution?
10. Do you or your colleagues
agree with the use of the IF?
11. What do you see as the
disadvantages in the use of the IF from the standpoint of individual faculty,
nursing school/department or the nursing profession?
12. To what extent are university
rankings and ratings in your country dependent on the research productivity of
the collective faculty?
13. Are there top-tier journals
that universities compile in nursing where nursing faculty are urged to
publish?
14. High IF journals have been
criticized in the literature on various grounds. In your view, to what extent
is this the case for high IF journals in nursing?
15. How many nursing journals
published in your country are listed in the Web of Science?
16. Do you agree with efforts
underway by the International Academy of Nursing Editors to increase
nursing journals in the Web of Science?
________________________________________________________________________________________________________
Note: Questions 1-8 correspond to
the items in Table 2.
Data Analysis. The responses to questions providing options were
summarized in table form. Comments and narrative responses were summarized and
presented descriptively. In addition, the country informant was critical in
providing background and context on the country and in providing interpretation
of statements from respondents that could be understood only by knowledge of
the context, setting and country. This approach is in line with the literature
on the nature of case studies where data can legitimately be obtained from
multiple sources (8).
Results
Results are provided first as
description of the context of the country in terms of higher education,
nursing, and respondents’ perceptions about how policies operated; this is
followed by a summary of the questions in which respondents chose the provided
options.
Respondents provided answers to questions
to clarify and better understand the situation in Brazil. Their report is
enhanced by information provided by the country informant regarding higher
education and nursing in Brazil.
Respondents confirmed that journal
IF requirement was in use by an agency of the MoE and by their institutions.
While the criteria used by the MoE are complex and take account of many
variables, faculty publication in high IF journals played an important role in
institutional and program evaluation and rankings, which are published and made
available nationally.
While respondents agreed that the
IF can be an indicator of journal quality and can be useful internationally,
they were critical of the policies on a number of grounds: that the IF measure
was being used for comparisons across disciplines, placing nursing at a
disadvantage; that there was a bias in favor of English-language journals and
few Brazilian nurses could publish in English; that there were many Portuguese
language journals that publish research useful to practicing nurses, but are
not included in the WoS.
Respondents described a system
through which journals were classified in the country – that included both
nursing and non-nursing journals, and that faculty were urged to publish in the
recommended journals.
Respondents were asked about their
view of an initiative under way by the International Academy of Nursing
Editors, aimed at increasing the number of nursing journals in the WoS; the
majority supported the initiative, and said it will increase the visibility and
profile of nursing and its journals.
Table 2 provides a summary of
options chosen on the questions where options were given. Some highlights from
the table are provided below, focusing only on areas of majority
agreement.
Table 2. Summary of Responses to
Selected Items
|
Questions and Options |
Frequency |
|
|
|
|
1. Who uses IF: |
|
|
a) University administrators |
5 |
|
b) School of nursing/department |
5 |
|
c) Government agency |
5 |
|
d) Faculty committees |
2 |
|
|
|
|
2. Purpose for which IF is used: |
|
|
a) As measure of individual’s productivity |
4 |
|
b) As measure of a group’s productivity |
3 |
|
c) As measure of school/dept. quality |
3 |
|
d) To assure high ranking of institution in national and international surveys |
2 |
|
e) As measure of journal quality |
4 |
|
|
|
|
3. How nurse scholars behave as a result of policy requiring publication in high IF journals: |
|
|
a) Scholars compete rather than cooperate |
5 |
|
b) Scholars publish in journals of other countries |
4 |
|
c) They want to publish in high quality journals regardless of their IF |
3 |
|
|
|
|
4. Hurdles encountered in publishing in top tier Journals: |
|
|
a) Insufficient language skills |
5 |
|
b) Not familiar with top tier journals or their guidelines |
4 |
|
c) Topics of interest to researchers do not interest such journals |
5 |
|
d) Those with graduate degrees from overseas are at an advantage |
1 |
|
|
|
|
5. Compare/contrast those who publish locally vs. those who publish internationally: |
|
|
a)Locally published authors are studying important problems in their country |
4 |
|
b)Those who publish internationally add prestige to their institution and country |
5 |
|
c) Country cannot benefit from international publications |
0 |
|
d) Publishing internationally means focus on health problems of those journals rather than own country |
2 |
|
|
Frequency |
|
6. Positive aspects of policy to publish in high IF journals has meant that: |
|
|
a)There is greater methodological rigor in research in country |
2 |
|
b) There is stronger theoretical grounding in published papers |
1 |
|
c) Graduate level training in research has improved in the country |
4 |
|
|
|
|
7. How do you characterize current published works in your country? |
|
|
a) They are trailblazing efforts in theory or methodology |
0 |
|
b) They are responsive to health needs of the country |
2 |
|
c) They are of interest to investigators but not of value to population |
0 |
|
d) They frame practical applications of research for health problems of country |
2 |
|
e) They frame practical applications of research to health problems of other regions |
3 |
|
f) They represent replication of work done elsewhere for their relevance to local needs |
3 |
|
|
|
|
8. Efforts to internationalize journals: |
|
|
a)International members added to journal editorial boards or as assistant/associate editors |
3 |
|
b) International members are added as manuscript reviewers |
3 |
|
c) Researchers from the country serve on editorial boards of journals in other countries |
1 |
|
d) Researchers from country serve as reviewers for journals in other countries |
1 |
|
e) Any of the above steps have strengthened the quality of the journals in the country. |
3 |
|
|
|
Who uses the IF and for what purpose. Respondents were unanimous in their agreement that university administrators, government agencies and school of nursing administrators used the policy, and the majority indicated that the policy was used as a measure of individual’s productivity and journal quality; it was also used to measure departmental quality and the for the productivity of a group of faculty, according to three respondents. However, the policy was not used by faculty committees (readers are referred to Table 2 throughout the presentation of results).
How scholars’ behavior is influenced by the existing policy. The majority indicated the presence of strong competition to publish in high impact journals that got in the way of cooperation across scholars, that such a requirement meant that scholars end up publishing in journals in other countries rather than their own, and at the same time, some indicated that there was interest in publishing in high quality journals regardless of a journal’s IF. Some commented that many nurses did not worry about the policy.
Perceived hurdles. Respondents felt that insufficient English language skills, the perception that the topics scientists investigate were not of interest to top-tier journals, and their lack of familiarity with top-tier journals were important hurdles that get in the way of publishing in international, high IF journals.
Compare/contrast those who publish nationally vs. internationally. Respondents indicated that those who publish locally/nationally are studying important problems of the country, while those who publish in international journals believed that they added to the prestige of their institutions and their country, and two of these individuals felt that these scholars end up focusing on problems of interest to those journals rather than problems of interest to their country.
Positive aspects of policy. The majority felt that research training in the country had improved as a result of the policy, and saw no other positive aspects of the policy.
Characterization of published works in nursing. Three respondents stated that investigators framed the practical application of their work in terms of what might be of interest in other regions of the world, and three stated that the current research in Brazil involved replications of work done elsewhere to determine their relevance and application to local needs.
Efforts to internationalize Brazilian journals. Several steps in the past decade were mentioned in efforts to internationalize journals within the country; these were appointment of international members to journal manuscript review panels, as editorial board members or as assistant/associate editors; it was further felt that these steps have strengthened journal quality. There was scant indication that the reverse may also be occurring, i.e., that Brazilian scholars were serving in various roles for journals of other countries. Additional information was provided on regional data bases, described below.
Discussion and Recommendations
Brazil has clear national policies and expectations on graduate education and research established by the MoE that specify venues for publication as measures of research output. The outcome of the MoE reviews plays an important role in funding decisions for departments and in research funding for faculties. Despite these policies and expectations, the overall responses to questions and additional remarks made by the respondents did not indicate a sense of urgency within the nursing community to comply with the stated policy, indicating that not many Brazilian nurses publish in international venues or high IF journals. Rather, there appeared to be a sense of resignation on nursing’s state of affairs, conveyed through statements such as: “nursing research that is useful to Brazilian nurses are in Portuguese and are not considered in the computation of IF,” “Brazilian nurses do not worry about citation counts,” (a measure of the number of times an article or researcher’s work is cited).
Respondent comments indicated that nursing research is ongoing in the country, published in national venues in Portuguese. It is then unfortunate that such knowledge is unavailable to the wider world beyond Brazil. If published more broadly, these scientific works could create opportunities for dialogue and collaboration with international colleagues and be mutually beneficial and enriching.
However, in characterizing current publications in the country, the responses to question 7 (Table 2, options a, b, d) did not indicate a high regard for current publications. These matters require further exploration, as the data provided here do not enable us to explain these findings. Yet, in not choosing the statement that the research is “of interest to investigators but not of value to the population,” (question 7, option c) it can be inferred that respondents felt that topics chosen for research were of value to the population.
In identifying as a hurdle the statement that the “topics of interest to researchers do not interest such journals,” a dichotomy is suggested. Whereas journals of high quality do in fact look for manuscripts that investigate issues of significance that promise to improve the health of populations; at the same time, they want to publish manuscripts that represent sound research, both theoretically and methodologically.
Many schools of nursing in Brazil have their own journals. Until 2005, these were typically published in Portuguese only, with abstracts in English and Spanish, and occasional full-length articles in English, which suggests some degree of interaction with scholars in other countries. Although the best journals in nursing are affiliated with institutions, this factor does not affect the range of their circulation; rather, as they increasingly make efforts to achieve quality criteria, the journals circulate widely across the Brazilian territory, with some also reaching different countries. This group of journals affiliated with institutions stands out by the joint policy they have adopted to strengthen their publications; this fits into the peculiar context of Brazilian science which, since 1997, has established a policy and method that guarantee values like equity and open access in practice. This system is called SciElo – the Scientific Electronic Library Online, which includes scientific journals meeting eligibility criteria. SciElo converts full editions of journals into electronic form, providing open access and automatically generating reference databases, usage and citation statistics. In 2002, the first Brazilian nursing journal was included in this collection. Recently, this journal broadened its scope and began to publish the full version of all articles in three languages, in order to reach readers in Portuguese-, Spanish-, and English-speaking countries (10).. By doing this, it became a reference for other Brazilian journals aiming for internationalization and compliance with sophisticated assessment criteria, including IF (10-11).
Publishing any journal commands a high degree of faculty time in preparing articles, conducting blind reviews, editing, and the like. Yet, the ultimate rewards that come from such major investment of time and resources are limited to individuals or their institutions, in view of policies and expectations set forth in the country. Instead, institutions can combine resources to improve the production of high quality journals that can be included in electronic libraries like the SciElo, and increase their visibility.
The views of these respondents on the nature of nursing science and scholarship in the country were equivocal, lacking unanimity. Some stated that the current research in the country entailed replications of work done elsewhere to test the relevance for the country; some indicated that investigators framed the practical applications of their work in terms of other regions of the world. Fewer numbers expressed other views. The tenor of these responses contradicts respondent statements elsewhere, where they stated that there was much valuable research being done in the country in the local language that could not be published in international venues, and that nurse scholars were addressing local health needs of the population. This may be due to the fact that Brazilian researchers are now increasingly pressured to publish in high IF journals without a clear appreciation Brazilian authors place on publishing in Brazilian journals, while maintaining their academic, scholarly and service obligations(4). The respondents were from leading graduate programs that face the disadvantage of nursing in comparison with other disciplines in the Brazilian graduate evaluation system. This puts the same pressure to publish in high IF journals on researchers from all knowledge areas, without acknowledging the unique obligations and responsibilities that characterize nursing and other professional disciplines. Considering that this analysis takes place in a transition period, a more in-depth and broad assessment may be needed with a view to a clearer conclusion.
In a recent report the visibility of Latin American nursing research for 46 years (ending with 2005) was analyzed (12). The authors addressed the issues of limited nursing research and lack of the international visibility of the research produced in the region, identifying the limited knowledge and experience in conducting research on the part of the nursing community as one factor. Yet, they identified Brazil as producing a full 31.9 percent of the total number of publications analyzed, and commented on the leadership of Brazilian nursing, especially in the past 20 years, in developing graduate/doctoral education and nursing research. These findings on the leadership of Brazilian nurses are indeed highly encouraging; yet, while acknowledging this, the authors point to the need for greater visibility and integration of Brazilian science with scientific communities globally.
In a recent editorial several Brazilian investigators addressed themselves to the issues and challenges of internationalization of nursing science, drawing on interviews in making recommendations as to strategies. They identified the need for effective collaboration between national and international researchers, dissemination of their research, and strengthening the analysis and discussion of their research findings, indicating that these strategies would make their scholarship more attractive to international audiences (13-14).
The findings in this study seem to dichotomize global knowledge vs. national/local knowledge. This perspective is no longer valid within a globalized environment with a multitude of means and methods of communication and transmission of information and knowledge. Nurse scientists need to endeavor to present their work in a manner that those in other countries relate to and benefit from, even while recognizing that situations and settings vary across countries and some studies may need to be replicated in new settings before their relevance and applicability can become evident.
Three responses to the pursuit of the IF by Asian researchers in the management field were analyzed: the “adopted Western” approach, which entails accepting the pressure to publish in high IF journals and publishing in them; the “Asian” approach, focusing on domestic problems and publishing in local/national journals; and the “integrationist” approach, focusing on local issues and publishing in international journals (15). The current research does not definitively answer the question of which of these approaches have been implicitly adopted by the scholars in Brazil, but these data would suggest that the “Latin American” approach may be favored (the equivalent to the “Asian” approach in the three responses by the above authors).
Conclusion
While a direct relationship between nursing scholarship and the implementation of policies in place in the country on publishing in high IF journals cannot be inferred at this time, the policies in the country do not appear to have created an environment conducive to collaboration or one leading to interactions with scholars in other countries, factors that can promote skill acquisition, capacity building, and greater global visibility of scientific production. This may be due, in part, to the lack of a general acceptance of the policy and also, due to the opportunities available in the region to include their local publications in new, regional data bases. An in-depth study of national scope examining relevant factors might provide additional insights that might be useful to the country.
Limitations
There are several limitations of this study. The first is that some of the questions in the study instrument presented options to check, and thus, respondents were given a “recognition” task rather than a “generation” task. A second limitation has to do with the case study design; thus, the data do not provide the basis for generalizations. Lastly, the use of the English language for the questionnaire may have created comprehension difficulties for some respondents.
References
1. Meho LI. The rise and rise of citation analysis. Physics World 2007, 20:32-6.
2. Monastersky R. The number that’s devouring science. The Chronicle of Higher Education 2005, 52 (8): A12.
3. Campanario JM, Gonzales L, Rodriguez C. Structure of impact factor of academic journals in the field of education and educational psychology. Scientometrics 2006, 69 (1): 37-56.
4. Johnstone MJ. Journal impact factors: Implications for the nursing profession. Int Nurs Rev 2007, 54(1): 35-40.
5. Freshwater D. Impact factors and relevance of research outputs: One step forward, two back? J Psychiatr Ment Health Nurs 2006, 13: 473-4.
6. Holden G, Rosenberg G, Barker K, Onghena P. An assessment of the predictive validity of impact factor scores: Implications for academic employment decisions in social work. Research on Social Work Practice 2006, 16(6): 613-24.
7. Nieminen P, Carpenter J, Rucker G, Schumacher M. The relationship between quality of research and citation frequency. BMC Med Res Methodol 2006, 6: 42-9.
8. Ketefian S, Hanucharurnkul S. Environment for scholarship and journal impact factor in Thailand. Thai Journal of Nursing Research 2009, 13(3):159-66.
9. Woods N F, Catanzaro M. Nursing research: Theory and practice. St. Louis, MO: C.V. Mosby Co., 1988.
10. Mendes IAC, Marziale MHP. The innovative and enterprising course of the Latin American Journal of Nursing and the adoption of an internationalization policy. Rev Lat Am Enfermagem 2007, 15 (2):185-6.
11. Marziale MHP, Mendes IAC. A great achievement: Latin American journal of nursing enters the SciELO collection. Rev Lat Am Enfermagem 2002 10 (1): 1-2.
12. Mendoza-Parra S, Paravic-Klijn T, Muñoz-Muñoz A M, Barriga OA, Jiménez-Contreras E. Visibility of Latin American nursing research (1959-2005). Journal of Nursing Scholarship 2009, 41(1): 54-63.
13. Leite de Barros ALB, Cassuli-Matbeus MC, Rivero de Gutiérrez MG. Internationalization of nursing know-and-do. Acta Paul Enferm 2008, 21(1): v-vi.
14. Cruz I. Peer review: what are the OBJN responsabilities? Online Brazilian Journal of Nursing [serial on the Internet]. 2006 December 16; [Cited 2010 Mar 8]; 5(3):[about ## p.]. Available from: http://www.objnursing.uff.br/index.php/nursing/article/view/745
15. Leong FTL, Leung K. Academic careers in Asia: A cross-cultural analysis. Journal of Vocational Behavior 2004, 64: 346-357.