ORIGINAL ARTICLES
Evaluation of effectiveness of training of nursing professionals: a correlational study
Vera Lucia Mira1, Nadia Nasser Follador1, Cláudia Regina Seraphim Ferrari1, Ligia Fumiko Minami Neves Oliveira1, Jaqueline Alcântara Marcelino Silva1, Patrícia Tavares Santos1
1University of São Paulo
ABSTRACT
Aiming to evaluate the professional training programs, this study wants to evaluate the effectiveness of training about contact precautionary steps, run to nursing professional. This is a correlational design research. The material of analysis is composed by the pre- and post-training learning evaluations (n=248), performed in the University Hospital of the São Paulo University, which were submitted to a descriptive and inferential statistical analysis. This study was approved by the Ethics in Research Committee of the institution. The main results showed that, if compared the pre- and post-training periods, there was a significant rise on the post moment, of the average grade, in 10 out of 14 questions from the evaluation chart. Despite the fact the training was considered effective, it was observed there is a necessity to improve the diagnosing method of educational actions, as well as the evolution of a deeper evaluation methodology that embraces the observation of the results of training in assisting care.
Palabras-clave: Education; Inservice Training; Nursing; Assessment.
INTRODUCTION
Nowadays, people management in health is aimed to professional training and to the development of the human resources focusing on the reflexive and participative education, which is the opposite of the technical-operational training. Within this perspective, to study the learning process in the organizational environment can generate new findings and contribute to changes in practice(1).
The educational process at work transcends the elaboration of Training and Development programs (T&D). In their conception, the real necessities of the institution and individuals must be assessed, prioritizing in clear and well established goals for each program(2). The main objective is to offer in a systematized way the learning of knowledge, abilities and attitudes that improve technical, ethical and political competences, as well as personal ones, which equip the individual to understand and transform the reality(2).
The T&D strategies to induce learning integrate the five main learning concepts or objects: information, instruction, training, development and education. Information is an inducted learning design established by the use of organized content that receive a value attribute. Instruction involves instructive proceedings to transmit knowledge. Training is focused to the improvement of performance to the work field. Development refers to the learning opportunities to the personal development of the worker, while education means a continuous formation of the laborer(3).
The articulation between the learning objects can occur through different combinations, intending to contemplate the simplest and the most complex forms of educational induction, starting from information to education(3).
T&D can be understood as three coordinated subsystems: evaluation of necessities, planning and executing, and training evaluation, which the last can provide systematized information about the training set and the others refer to the training development(4).
The training of health professionals tends to generate horizontal information and knowledge, and to improve the sharing of responsibilities, searching to a continuous improvement of healthcare practices in the whole health network(5).
The improvement in work practices after T&D programs cannot be denied, which is a reflex in individual changes in the professionals, as a rise in knowledge and abilities that may lead to behavioral changes.
Therefore, it is indispensable the need to evaluate the effectiveness and the impact of educational measures in a systematic way.
The educational evaluation and the evaluation of the results of training programs are the less developed aspects in education proposals, and despite the recognized importance of these facts, their effective performance and resource allocation is secondary. This evaluation has a primary role in the improvement of all actions (6).
The training evaluation allows the identification of factors that restrain or contribute to a better performance of the trained people, suiting the training actions to the organizational needs(1).
In this article, the learning evaluation was adopted to verify how much a learner acquired from the participation in an educational action. This study integrates a research project that its final goal is to propose an evaluation methodology to educational programs in the area of health.
The T&D evaluation process can be performed in four levels of evaluation: reaction or satisfaction – participant’s opinion about the learning conditions; apprenticeship – effectiveness of the training related to the acquisition or development of knowledge; behavior – changes generated by the training in participants’ behavior; and results – practical transformation of participant’s daily work.
METHOD
This is a correlational study that tests variables, checking how much the behavior of a variable influences the alteration of another variable(7).
In educational actions, the variables can be measured before, during and after a process, but are not controlled as in experimental researches(1). In this study, the variables of learning evaluation were taken before and after the training through numerical grades.
This study was conducted at the University Hospital of São Paulo University (HU-USP), which is a member of the Brazilian Unified Health System (SUS, in Portuguese), composed by 278 beds for secondary healthcare, about 1,800 staff members, and from those, 708 nurses.
The Educational Guidance Service (SEd, in Portuguese) is responsible for coordination, planning, execution and evaluation of the educational programs developed to the nursing professionals. This service is subordinated to the Nursing Department (DE, in Portuguese) of the HU-USP, subdivided in: Surgical Nursing Division (DEC, in Portuguese), composed by the Clinical Surgery unit, Surgical Center unit, Material Center unit and Day Hospital unit; the Clinical Nursing Division (DECLI, in Portuguese), composed by the Medical Clinic unit and Intensive Care and Hemodialysis unit; the Maternal-Infant Nursing Division (DEMI, in Portuguese), composed by the Adjacent Lodging, the Nursery, Pediatrics, the Obstetric Center and the Infant Intensive Care unit and; the External Patients Division (DEPE, in Portuguese), with the Adult Emergency Room, Infant Emergency Room, Ambulatory, Iconology and Endoscopy.
The analyzed documents were all leaning evaluation sheets (n=248), from the do “Contact Precautionary Care” training (TPC, in Portuguese), performed by SEd nurses, training instructors and by the researchers of this study. The period of data collection was from June to December 2007.
The learning evaluation, created by the training instructors, was composed by theoretical questions to check the specific understanding from the training session, submitted to content appreciation by the nurse of Hospital Infection Control Commission. It was applied the evaluation form, immediately before and after the program, by the instructors that performed the correction of such forms and the storage of the grades in a databank software called Statistical Package for the Social Sciences 17.0 (SPSS©).
The questions applied before and after training were identical, so there would not be any difference in the level of difficulty and to make it possible to compare the development of the trained personnel, through the result showed in their grades.
The data obtained were analyzed through a descriptive and inferential statistics to verify the correlation between the variables. The Shapiro Wilk test was used to check the distribution of variables and guided the choice of the non-parametrical tests used in this study: Wilcoxon and Kruskal Wallis.
To compare the results before and after training and aiming to identify the existence of a significant statistical difference between the two moments, we used the Wilcoxon evaluation. This test involves the measurement of a variable in the individual in two distinct moments; between these moments, the intervention to be evaluated happens so to observe how it will affect the answers; then there is a calculation for each individual, the difference between the initial and final observations(8).
To confirm the existence of a relationship between the variables, this study used the Kruskal-Wallis test, “that uses sample posts from three or more independent populations”(9).
In both statistical tests, a level of significance of 95% was adopted.
This project was approved by the Chamber of Research and by the Ethics in Research Committee from the University Hospital of São Paulo University, under protocol registration 555/55. As it is a prospective research and, understanding that, in the abovementioned evaluation processes – which there is an access to personal information and results of performance evaluations – and in both situations, this could cause expectation and/or fear, and therefore, an authorization to use such results as data for this study was asked, and participants also received the Researcher’s Responsibility Terms, after detailed explanation of such terms.
RESULTS
From the 248 analyzed evaluations, 78 (31.5%) were nurses, 89 (35.9%) were nursing technicians and 81 (32.6%) of nursing assistants; the majority came from DEMI, which is the Division with the highest number of personnel.
Regarding the learning evaluation, table 1 shows that, if compared the moments before and after training, there was a reduction of the number of blank and incorrect answers in the post-training tests, and a rise of correct answers.
On table 2 , we observe the descriptive and in-dispersion statistical values referred to the general grades of learning evaluation in the pre- and post-training moments.
We observe on table 2 that there was a rise on the average, median, mode and minimal grades; the standard deviation also showed some reduction; and the maximum grade was the same. These results suggest an improvement in the performance of the post-tests, if compared to the pre-tests.
To an object appreciation of the relation between the pre- and post-training grades, the average grades were compared, in these two moments, and then applied the non-parametrical test of Wilcoxon, as seen on table 3.
We observed that there was a significant statistical difference from the pre- to the post-training grades in all questions, except the issues 1, 2, 8 and 13, in which the test did not show significant difference. Considering the professional category and the average grades in the pre- and post-training results, there was no significant change in the average grades through the application of the non-parametrical test of Kruskal-Wallis, p=0,244, and in the post-training moments, the non-parametrical test of Kruskal-Wallis showed p=0,104.
The professional categories presented, among each other, significant difference in the pre-training moment (Kruskal-Wallis, p=0,002) and in the post-training moment (Kruskal-Wallis, p=0,034). In the pre- moment, the nursing auxiliary personnel present significant lower average than the nurses (test of Dunn, p< 0,05); in the post-training moment, technicians presented significant higher average than the assistant nurses (test of Dunn, p< 0,05). In the three categories, there was a significant rise from the pre- average to the post- average.
DISCUSSION
The significant rise of the final grade before and after training, it was demonstrated a development in trained people’s understanding, or in other words, the training intervention was successful. The acquisition of knowledge suggests that the content was well elaborated, which is a favorable condition to learning. Besides the content, among the learning promoting aspects, there is interest and participation of the student(2).
Usually, the questions of the tests that referred to new content are essential to evaluate the acquisition of knowledge as it is expected to have a lower score in the pre-training evaluation, and a higher one after. The questions that had a higher level of success in the pre-training and, simultaneously, in the post-training, lead to three different explanations: first, the trainees already had some information about the topic – a positive aspect, as the reinforcement of known information helps to reinforce the content and correlate it to new information; second, the participants were not the correct target audience; third, the instructors did not know how to elaborate the content according to the necessity of the trainees(2).
The data collection and evaluation of the necessities of training (ANT, in Portuguese) are the first and one of the most important steps of planning of an educational activity, that involve the availability and the preparedness of the professionals to recognize their limitations and instructing needs. The ANT is associated to the integration of the teams with the personnel related to the continued educational service. Another format used by ANT is the analysis of institutional quality control indicators, as fallen indexes, ulcers caused by pressure and hospital infections (2).
In practice, however, in the majority of the organizations, the data collection is restricted to the analysis of individual tasks, which lead to untie the educational processes to the reality of the institutions(10).
The good result after the training, identified in this study, is positive. On the other hand, it is vital to analyze with instructors and trainees, question by question, comparing the result of the grades with the proposed objectives and the training content with what is demanded in each question, with will allow the student to evaluate the educational process, considering the whole process of teaching-learning (besides the learning by itself).
Anyhow, and besides being nurses, technicians and assistant nurses with an improvement in their grades in post-training, it is also fundamental to observe deeply this relationship, to adjust the content and the learning strategies to the necessities and the previous level of knowledge of the professionals (2).
Even after proving the effectiveness of training, it is necessary to consider the operational factors and the work environment that influence the educational process, and therefore, they must be included in the evaluation method. The environmental aspects are capable to aid or impede the use, at work, of all that was taught during the training. As some national and international research show, the organizational conditions are understood as the main determinant to the human development(10).
Besides that, the acquisition of knowledge after training was measured in 14 out of 19 studies included in an integrative review about this topic(11), showing that it is, together with the reaction evaluation, the most frequent type used.
Learning is one of the necessary conditions to transfer what was taught to a daily working practice(12) and it is processed as soon as the person appropriates a certain knowledge, which depends on this person’s interest, maturity, environmental conditions, among others, and it materializes in the moment of complete awareness of the learning object, that can be observed in the acquired competencies, or in other terms, the development of new abilities and attitudes founded in a cognitive understanding.
Learning can occur even in adverse situations, although such scenario is not desired. The idea is that we understand the evaluation as a process that demands changes, or in other words, if there is learning without changes in behavior and quality of work, it is important to investigate the reasons to recover the procedural essence of evaluation(2).
The changes in behavior must be derived from the expected performance of all trainees, in accordance to the objective of the trainings, which rarely happens, hindering the identification of the expected changes in the organization(13). It is imperative the learning implies in practical changes, and therefore, investments in action planning and in evaluation processes are necessary.
CONCLUSION
This study permitted to confirm the effectiveness of training to acquire or to enlarge knowledge, due to a significant rise of the variable “grade” on the post-training moment.
The effectiveness evaluation provides important answers about the training, however, this traditional method of evaluation is limited, because it is not possible to evaluate the totality of the project.
Therefore, to guide the construction of an evolution of evaluations of health education, it is necessary to apply other strategies of result and impact analysis of these actions at work, as well as to appreciate the process of teaching. It is essential to investigate the relationship between the intervening variables from learning, aiming to know the predicting variables of the training evaluations.
Due to the low offer of research and publications about the evaluation of T&D programs in the field of health in Brazil, this present study may contribute to affiliate practices in other institutions, as well as in the proposition of new inquiries.
REFERERENCES
1. Pilati R, Borges-Andrade JE. Construção de medidas e delineamentos em avaliação de TD&E. In: Borges-Andrade JE, Abbad GS, Mourão L. Treinamento, desenvolvimento e educação em organizações e trabalho: fundamentos para a gestão de pessoas. Porto Alegre: Artmed; 2006. p. 359-84.
2. Mira, VL. Avaliação de programas de treinamento e desenvolvimento da equipe de enfermagem de dois hospitais do município de São Paulo. Tese [ Livre-docência em enfermagem ]. Escola de Enfermagem da Universidade de São Paulo. São Paulo; 2010.
3. Vargas MRM, Abbad GS. Bases conceituais em treinamento, desenvolvimento e educação – TD&E. In: Borges-Andrade JE, Abbad GS, Mourão L. Treinamento, desenvolvimento e educação em organizações e trabalho: fundamentos para a gestão de pessoas. Porto Alegre: Artmed; 2006. p. 137-58.
4. Borges-Andrade JE. Avaliação integrada e somativa em TD&E. In: Borges-Andrade JE, Abbad GS, Mourão L. Treinamento, desenvolvimento e educação em organizações e trabalho: fundamentos para a gestão de pessoas. Porto Alegre: Artmed; 2006. p. 343-58.
5. Erdmann AL, Mello ASF, Andrade SR, Drago LC. Organization of care practices in the health network. Online Brazilian Journal of Nursing [ serial in the Internet ]. 2011 May 3; [ Cited 2011 oct 20 ] 10(1). Available from: http://www.objnursing.uff.br/index.php/nursing/article/view/3220
6. Organización Panamericana de la Salud. Capacitación del personal de los servicios de salud (OPAS). Quito; 2002. (Proyectos relacionados com los procesos de reforma sectorial, 137).
7. LoBiondo WG, Haber J. Pesquisa em enfermagem: métodos, avaliação crítica e utilização. Rio de Janeiro: Guanabara Koogan; 2001.
8. Menezes RX, Azevedo RS. Bioestatística não paramétrica. In: Massad E, Menezes RX, Silveira PSP, Ortega NRS. Métodos quantitativos em medicina. São Paulo: Manole; 2004.
9. Triola MF. Introdução à estatística. Rio de Janeiro: LTC; 2005.
10. Menezes PPM, Zerbini T. Levantamento de necessidades de treinamento: reflexões atuais. Análise. 2009; 20(2):50-64.
11. Otrenti E. Avaliação de processos educativos formais para profissionais da área da saúde: revisão integrativa de literatura. [ dissertação ]. São Paulo: Escola de Enfermagem, Universidade de São Paulo; 2011.
12. Mira VL, Peduzzi M, Melleiro MM, Tronchin DMR, Prado MFF, Santos PT, et al. Análise do processo de avaliação da aprendizagem de ações educativas de profissionais de enfermagem. Rev Esc Enferm USP. 2011;45 (nº esp):1574-81.
13. Zerbini T, Abbad G. Transferência de treinamento e impacto do treinamento no trabalho: análise crítica de literatura. Rev Psicol Organ Trab. 2010; 10(2):97-111.
Received: 21/10/2011
Approved: 30/06/2012