The evaluation of the quality management system effectiveness at machine-building enterprises

Quality management system (QMS) regulates the work of an enterprise sectors in identifying, providing, and maintaining the production quality at the stage of designing, developing, realizing, and servicing. The purpose of this is to constantly improve the organization effectiveness and efficiency. Evaluation is an essential condition for the functioning and improvement of a quality management system. It can differ by its sector of application and includes such activities as audit (monitoring) and the analysis of the quality management system, together with the organization self-evaluation (sce figure).

The analysis of the quality management system is conducted in order to provide its constant suitability, sufficiency, and effectiveness. It includes the evaluation of improvement capabilities – including policy and goals in the field of quality.

The quality management system analysis and evaluation has been conducted on the basis of product and process monitoring results, internal audits, the evaluation of consumer satisfaction, corrective and preventive actions, as well as suggestions made by workers and process managers.

The most important problem of quality management evaluation is the selection of the most “problematic” processes (from the process quality point of view), ranging them according to the level of “significance” for the organization activity, making decisions about conducting technical-organizational activities and allocation of funds for various purposes. This is why it is necessary to develop a method for evaluating the quality management system effectiveness.

Let’s apply the following method to JSC Krasnoyarsk Refrigerator Plant “Biryusa”. In order to evaluate the effectiveness three main criteria of goal achievement are highlighted: the improvement of the quality management system effectiveness, its compliance with the GOST R ISO 9001 requirements and internal regulations, the increase of the customers’ satisfaction level. To achieve the planned targets each criterion is divided into two parts which include a list of problems, reflecting the accomplishment of planned results (tab. 1).

Types of quality management system evaluations

Table 1

Evaluation sub-criterion	Criterion evaluation questions	Satisfaction evaluation	Percentage of subcriterion accomplishment, %
1.1. Planning quality improvement activities	1. Are the department targets clearly defined?	met	100
	2. Are meetings on quality problems being conducted at all levels?	met
	3. Are quality indicators of manufactured products being planned?	met
1.2. Accomplishing planned and correcting activities in quality improvement	1. The accomplishment of required quality indicators for products	not met	50
	2. What actions are taken by workshop managers to prevent the appearance of defects?	met
	3. Accomplishment of planned activities on time	not met
	4. The effectiveness of correcting / preventing actions taken	not met
	5. What are the worker actions if he notices a product defect?	met
	6. Who supervises corrective activities?	met
2.1. The quality system level of compliance with GOST R ISO 9001–2008 requirements	1.Compliance of the QMS regulations with GOST R ISO 9001–2008 requirements	met	100
	2. Compliance of department targets to the Policy in the field of quality	met
2.2. The quality system level of compliance with internal QMS regulations	1. Managing paperwork according to STP SK 222-4005–2002 requirements	not met	50
	2. Managing documentation according to STP SK 222-4016–2002 requirements	not met
	3. Appropriate qualification of the staff	met
	4. Research and development according to STP SK 222-4004–2002 requirements	met
	5. Supervising irrelevant products according to STP SK 222-4013–2002 requirements	met
3.1. The level of consumer contentment based on conducted surveys	1. Do you regard “Biryusa” refrigerators as reliable?	not met	50
	2. Are you satisfied by the prices of “Biryusa” refrigerators?	met
	3. Do you suppose that purchasing a “Biryusa” refrigerator is a sign of prestige?	not met
	4. Are you satisfied with the range of refrigerators the company has to offer?	met
	5. Are you comfortable with using “Biryusa” refrigerators?	met
	6. Does the outer design of “Biryusa” refrigerators satisfy you?	not met
	7. Does the inner design of “Biryusa” refrigerators satisfy you?	not met
	8. Do the “Biryusa” refrigerators save energy?	met
	9. Do the temperature regimes of the “Biryusa” refrigerators satisfy you?	met
	10. Does the after sales service of the “Biryusa” refrigerators satisfy you?	not met
3.2. The level of consumer contentment based on warranty returns	1. Electronics	not met	50
	2. Compressor defects	not met
	3. Leakage	met
	4. Outer shell	met
	5. Does not fit the doze	met

Questions for evaluating quality management system effectiveness

It is easier to evaluate the quality management system effectiveness by an amount of points gained for each of the target achievement criteria. A 100 point scale can be used for the evaluation. Each of the criteria is given a quality index. When identifying the quality index it is supposed that the sum of all coefficients is equal to 100 %. The quality index – determined for each criterion is divided into sub-criterion, depending on its importance. The highest quality index is defined by the quality management system effectiveness improvement criterion as well as by the increase of the consumer’s contentment. This is important, first of all, because any economical activity of organizations is bent on supplying consumers with quality products in order to meet their needs.

According to the developed method, problems of the first two criteria are determined by the results of conducted monitoring. The third sub-criterion is determined by the level of consumer contentment. The evaluation of this sub-criteria is conducted by the marketing and service department specialists studying the return and refund statistics.

Evaluation is conducted through three parameters:

– the sub-criterion is achieved – 100 % score;

• –    it is achieved partially: If more than 50 % of actions to achieve the subcriteria done, then it is 50 %, but if the number is less than 50 %, the score is 0 %;

• –    if the sub-criterion is not achieved at all – 0 % score.

Then the quality management system effectiveness score is determined.

The common effectiveness score is calculated in the following way:

Б = ΣВj × Пj/100 %, where Вj – is the quality index, Пj – percentage of performed and accomplished subcriteria.

The common score of effectiveness determines the quality management system level (tab. 2).

Table 2

Evaluation of quality management systems

Common grade for effectiveness	The effectiveness level
91–100	Very good
71–90	Good
51–70	Acceptable
Under 50	Poor

According to the developed method an evaluation of quality management system effectiveness at JSC Krasnoyarsk Refrigerator Plant “Biryusa” has shown a number of drawbacks. The results of the quality management system evaluation are shown in tab. 3.

The results of quality management system evaluation at the enterprise according to the proposed method are displayed in tab. 3.

On conducting the effectiveness evaluation a number of drawbacks has been found. For example, the criterion of quality management system effectiveness improvement was 27.5 points. This does not fully satisfy the planned value. At the same time the sub-criteria of scheduled quality improvement activities is 100 % accomplished, because the monitored department targets are defined, regular meetings on the quality management are held and quality indicators of manufactured products planned. The sub-criterion of accomplishing scheduled corrective activities for quality improvement is only 50 % fulfilled.

This was discovered during a screening, which revealed that the quality indicators of manufactured goods do not correspond to the ones established. Activities do not meet their deadlines and the corrective and preventive actions applied are not useful.

The effectiveness level of criterion, the process compliance with the GOST R ISO 9001 requirements, and internal regulations is 12.5 points – this does not meet the planned value. The quality management system regulation corresponds to the GOST R ISO 9001–2008 requirements, the targets of the department to the quality policy [1]. Hence the quality management system complies with the GOST R ISO 9001–2008 requirements by 100 %. The qualification of staff working in the department, research and development, as well as control of irrelevant products is done through the use of normative-technical regulations. This cannot be said about paperwork supervision; hence the sub-criterion of quality management system compliance with internal regulations is only 50 %.

The consumer contentment criterion is only 50 % from the planned value – specialists from the marketing department conducted a survey and specialists from the service department analyzed data about return and warranty. All these studies showed that the effectiveness level is only 20 points. The survey showed that the consumer was not satisfied by the inner and outer designs and the after sales service of “Biryusa” refrigerators. They suppose that these refrigerators are unreliable and do not suppose that the mentioned brand is prestige. The after sales service departments carried out the warranty repair caused by defects in compressors and the electric system.

The results of the quality management system evaluation by using the developed method showed that the total effectiveness is 60 points. This means that quality management system functions appropriately but still does not fulfill all of the GOST R ISO 9001–2008 requirements.

Table 3

The evaluation of the quality management system at JSC Krasnoyarsk Refrigerator Plant “Biryusa”.

Evaluation criteria	Quality index	Evaluation sub-criterion	Subcriterion quality index	Percentage of criterion accomplis hment, %	Effectivene ss score, points
1. Improvement of quality system effectiveness	0.4	Planning quality improvement activities	0.375	100	15
		Accomplishing scheduled and correcting activities in quality improvement	0.625	50	12.5
2. Compliance of the quality system with GOST R ISO 9001 requirements and internal regulations	0.2	The quality system level of compliance with GOST R ISO 9001–2008 requirements	0.25	100	5
		The quality system level of compliance with internal QMS regulations	0.75	50	7.5
3. Increasing the consumer contentment	0.4	The level of consumer contentment based on conducted surveys	0.25	50	5
		The level of consumer contentment based on warranty returns	0.75	50	15
					60

The analysis results showed that the total quality management system effectiveness – in particular, can improve the capability of current activity improvements according to quality system criterions and improve the contentment of the consumer. This value should be increased based on the results of organizational and technical activity evaluation.

Having conducted the evaluation of the quality management system effectiveness applying the developed method, one can identify not only the functioning conditions of the quality management system, but see problematic areas, based on the established evaluation criteria. Using this method of analyzing results of enterprise activity and the functioning of the quality management system, the enterprise management can quickly focus their actions on the planning of improving enterprise activity. The given method of evaluation concentrates on the most important factors influencing the effectiveness of the quality management system provides the development, introduction, and realization of the best solutions. The appliance of the developed method will permit an enterprise in case of well functioning quality management system to reduce significantly the expenditures on quality and the functioning of the company. This will result in a concrete economical effect.