Оценка и планирование программ (2010-2014). Тематика и политика журнала

The purpose of this review is to analyze the methods of implementing the journal policy on a particular example of a nonRussian scientific journal. We have chosen a Scopus journal and analyzed the issues published within the years 2010-2014. The total number of the analyzed articles is 316 in 24 issues.

As it is stated on the journal website “Evaluation and Program Planning is based on the principle that the techniques and methods of evaluation and planning transcend the boundaries of specific fields and that relevant contributions to these areas come from people representing many different positions, intellectual traditions, and interests.” [1] In order to further the development of evaluation and planning, the journal publishes articles from the private and public sectors in a wide range of areas: organizational development and behavior, training, planning, human resource development, health and mental, social services, mental retardation, corrections, substance abuse, and education. “The primary goals of the journal are to assist evaluators

Evaluation and Program Planning (2010-2014). Themes and the Journal Policy

and planners to improve the practice of their professions, to develop their skills and to improve their knowledge base.” [1].

We analyzed the journal on a yearly basis focusing on the themes and key words of the articles. The data was classified into generalized themes and percentage was calculated for each issue. Then the results were integrated by calculating average percent per years and for the whole period. The results are presented in Tables 1-4.

Themes discussed

We grouped the themes of the articles into the following four clusters: program evaluation methods, health programs, social programs and training programs. The grouping was done with respect of the main focus of the article, e.g. “Gender differences in client–provider relationship as active ingredient in substance abuse treatment” [3]

was referred to ‘social programs’, “A proposed model for the analysis and interpretation of focus groups in evaluation research” [34] is classified as “program evaluation methods”, “Optimizing an immersion ESL curriculum using analytic hierarchy process” is an educational program [9] and “Improving the oral health of residents with intellectual and developmental disabilities: An oral health strategy and pilot study” [24] is a health program. Some articles are focused on more than one theme, e.g. “Evaluation of program performance and expenditures in a report of performance measures (RPM) via a case study of two Florida county tuberculosis programs” [5] is classified both as ‘program evaluation methods’ and ‘health programs’. Therefore, the total of the classification percentage does not equal 100%. The results are presented in Table 1.

Table 1. Themes discussed in the journal

Article themes	2010	2011	2012	2013	2014	Average
Program evaluation methods	43	47	46	25	28	37,8
Health programs	26	44	18	30	19	27,4
Social programs	20	36	19	33	32	28
Training programs	27	25	22	16	22	22.4

N.A. Pavlova, A.D Shcherbov

It is clear from the table that the main focus of the article themes is program evaluation, and this is a natural outcome of the journal policy.

Program evaluation methods vary from fundamental approaches to program evaluation e.g. “Defining the system of care concept and philosophy: To update or not to update?” [2] or “A proposed model for the analysis and interpretation of focus groups in evaluation research” [6] to application of special methods, e.g. “Cybernetics: Apossible solution for the “knowledge gap” between “external” and “internal” in evaluation processes” [5]. Often an evaluation method is applied to a particular range of programs “From policy to practice: A program logic approach to describing the implementation of early intervention services for children with physical disability” [6].

Health programs are focused on general issues like “Challenges and strategies in applying performance measurement to federal public health programs” [5], particular health problems like “Peer-led interventions to reduce HIV risk of youth: A review” [3] and health problems closely related to social problems e.g. “Community health promotion: A framework to facilitate and evaluate supportive social environments for health” [5].

Social programs deal with health issues such as HIV, AIDS, tuberculosis, cancer, drugs etc. e.g. “Peer-led interventions to reduce HIV risk of youth: A review” [3] or “Validating self-reports of illegal drug use to evaluate National Drug Control Policy: A reanalysis and critique” [5], abuse and violence as in “Volunteer and user evaluation of the National Sexual Assault Online Hotline” [8], “An evaluation of a child welfare training program on the commercial sexual exploitation of children (CSEC)” [18]. There is a scope of articles dealing with disabilities “Improving the oral health of residents with intellectual and developmental disabilities: An oral health strategy and pilot study” [25], different social groups “Estrangement factors associated with addiction to alcohol and drugs among homeless youth in three U.S. cities” [5] as well as regional problems “Measuring the performance of police forces in Taiwan using data envelopment analysis” [5].

Training programs mostly deal with professional training e.g. “Research on teacher education programs: Logic model approach” [14], environmental education “Environmental education evaluation: Time to reflect, time for change” [3] and cultural training “Process evaluation of a diversity training program: The value of a mixed method strategy” [10].

We can see that the focus on program evaluation methods slightly decreases, while

Table 2. Key words analysis

The most frequent key words are: system of care, children (mental) health, children welfare, values, (program) evaluation, ecological theory, logic models, assessment, education, immigration, family, abuse, assistance, youth, implementation, validity, competence, environmental programs, outcome, disability, systems thinking, adolescent, collaboration.

“advice may involve methodology, how the evaluation was implemented or conducted, evaluation utilization tactics, or any other wisdom that you think could benefit your colleagues” [1]. Evaluation is based on fundamental scientific knowledge and methodology.

Scientific concepts include many fields, connected with the covered scope, such as philosophy – values, system theory, worldview etc.; medicine – disparities, emotion, disturbance; social sciences and evaluation methodology such as gender, efficiency, accumulation and education, e.g. knowledge gap. We can see that the importance of this group slightly decreases in the time aspect, which might be an indication of the growing practical focus of the journal.

Research methods include different

Evaluation and Program Planning (2010-2014). Themes and the Journal Policy

evaluation methods such as programme evaluation, system evaluation, participatory evaluation etc., models e.g. logic models, validity model, measurement and specific methods such as Delphi method, multidimensional scaling etc.

Process agents are people and social groups involved in the evaluated programs. The percentage of this section in the total amount of keywords is not high, but it is evident. The results are presented in Table 3.

Table 3. Process agents

Process agent	2010	2011	2012	2013	2014	Average
Children	6	0.75	1.7	0.3	0.6	1.87
Young people and teenagers	0.75	-	0.67	0.8	0.6	0.56
Family	1.25	-	-	1.2	-	0.49
Other social groups	2.25	1.75	1.7	1	1	1.54
Agents of training process	0.75	-	-	0.3	1.2	0.45
Agents of evaluation	1	1.25	0.67	0.3	-	0.64
Agents of program implementation	-	0.75	0.3	0.3	0.3	0.33

The most numerous group includes children, young people, teenagers and family (2.92). This shows a certain program focus in the social aspect. Other social groups include homeless, offenders, community, public health professionals, released prisoners etc. training process agents are teacher, students, local learning collaborative etc. Agent of evaluation is mainly evaluator. Agents of program implementation include moderator, mediator, multiple stockholders, lay counselor, consultants etc.

We can see from the table that focus on certain agents is permanent, i.e. these groups can be seen in all time periods studied, e.g. children and other social groups, or mainly permanent, i.e. these groups can be seen in most time periods studied, – see young people and teenagers, agents of program evaluation and agents of program implementation, while other categories are addressed less frequently, these groups can be seen only in some time periods studied, – see agents of training process and family. This may be explained by the changing social focus of the problem studied towards certain programs.

Other factors that influence program implementation and assessment are geography, i.e. geographical area where the program is realized, program management factors, social factors, health factors, ethnic and cultural factors and other specific factors. The percentage of these factors is not much but they feature important aspects of program implementation. The results of analysis are seen in Table 4.

N.A. Pavlova, A.D Shcherbov

Table 4. Factors influencing program implementation and assessment

Factor	2010	2011	2012	2013	2014	Average
Geographical	2	3	3	1.8	1.8	2.3
Management	-	5.3	3.2	1.3	2.6	2.48
Social	10	6	6.5	7	6.2	7.14
Health	5	-	-	3	0.5	1.7
Ethnic and cultural	3	-	-	2	-	1
Other	2	1	0.25	0.5	0.7	0.89

The most numerous group of factors is social factors. They include factors originating in different social groups and phenomena such as drug abuse, police efficiency, immigration, domestic violence, right-wing extremism, social environment, quality of life, housing, prostitution etc. Social factors are part of keywords at all time periods.

Another group are factors connected with social management e.g. urban activities, non-governmental organizations, voluntary sector, EU LEADER etc. another segment includes health e.g. modified therapeutic community, serious mental illness, health equities, health system, AIDS etc. Ethnic and cultural factors include such key words as Alaska native, native American, African American, Hispanic, Latinos, cultural food etc. Other factors in this group feature financial matters such as federal funding, program cost and different other issues such as e.g. Proposition 36, Geographic Information Systems, conflict of interest, emergencies, unidentified circumstances, church etc. this is the least numerous group of factors. Health and ethnic factors are focused on only at specific time periods.

Geographical factors include different locations of program implementation. They can be countries on different continents like Asia: Iran, Singapore, Palestine, Afghanistan etc; Africa: Burkina Faso, Mozambique etc.; North America: Canada United States etc.; South America: Venezuela; Europe: Slovenia, France etc. and regions and groups of countries like e.g. South Africa, Caribbean, Asia-Pacific Region, SubSaharan Africa, developing countries. The wide scope of geographical factors shows the wide range of areas studied worldwide. The importance of geographical factors is permanent in the articles published.

Conclusion

To conclude we can say, that the journal policy is strictly followed. Program evaluation and planning are based on fundamental scientific concepts and the methodology includes both general scientific methods and applied methods for particular areas. The scope of problems analyzed is wide and comprehensive, including the general issues of program evaluating as well as analysis of particular programs in specific areas and geographical locations. There are special issues dedicated to in-depth analysis of specified programs [2; 3; 4; 9; 11; 14].

It interesting to note that there was only one article by a Russian author in the material analyzed “Participatory Training Evaluation Method (PATEM) as a collaborative evaluation capacity building strategy” by Alexey Kuzmin, Volume 35, Issue 4, 2012 [10].

We also would like to note the following articles on educational matters that we find interesting as they study a wide scope of approaches to educational programs: