Effective use of lessons learned to conduct the project review for ERP implementation

Автор: Atsushi Taniguchi, Masahiko Onosato

Журнал: International Journal of Information Technology and Computer Science @ijitcs

Статья в выпуске: 5 Vol. 10, 2018 года.

Бесплатный доступ

According to a recent study, it has been said that “lessons learned” is one of the most important and “value added” aspects of the project management lifecycle. However, it has been reported that it is often the most ignored part of finishing a project. Various reasons have been offered for this phenomenon. This article describes the systematic approach to initiate the project review on the specific project identified for requiring the formal quality audit based on the use of project management information system for having the execution date fixed by the independent quality reviewer with the project manager. Then, the project review process is started by retrieving the lessons learned data from the lessons learned repository, which were collected from the previous project reviews for the relevant ERP implementation projects, for the preparation of conducting the project document review and project stakeholder interviews. A case study methodology was applied to the historical lessons learned data of the ERP implementation projects conducted by the solution provider for their customers in the various industries in Japan, which were retrieved for a period of four years from 2014 to 2017 to analyze how the lessons learned collected from the project reviews of the earlier projects were reused in those of the succeeding projects conducted during the period. Use of lessons learned based on the past project review results was found to be effective in focusing on the specific areas projected for improvement during the processes of conducting the project document review and project stakeholder interviews, as well as putting together the practical recommendations for the findings to finalize the results of the project review, which were to be formally presented and submitted to the customer as the results of the quality audit.

Еще

Lessons Learned, Project Review, Quality Audit, Project Management Information System, Independent Quality Reviewer, ERP Implementation, Solution Provider

Короткий адрес: https://sciup.org/15016257

IDR: 15016257 | DOI: 10.5815/ijitcs.2018.05.01

Текст научной статьи Effective use of lessons learned to conduct the project review for ERP implementation

Published Online May 2018 in MECS

In any organization, dealing with lessons learned is a complex issue that involves people, processes and technologies [1]. One of the main challenges that organizations face, specifically project-oriented organizations, is the lack of structure and incentives for organization-wide learning [1, 2]. Since lessons learned are elements of both organizational learning and knowledge management [3, 4], creating, managing, sharing and utilizing knowledge effectively is vital for organizations to take full advantage of the value of knowledge [5].

According to a recent study [6], it has been said that “lessons learned” is one of the most important and “value added” aspects of the project management lifecycle. However, it has been reported that it is often the most ignored part of finishing a project. Various reasons have been offered for this phenomenon. Some actions to prevent the loss of knowledge and experiences are known from the literature. However, only a few firms manage systematically to identify and transfer valuable knowledge from projects to following projects [7]. To date, much of the research and industry focus has been a capturing lessons learned from the projects. However, even if lessons learned are successfully captured, there are still numerous problems to address in terms of their dissemination [8].

This article describes the systematic approach to initiate the project review on the specific project identified for requiring the formal quality audit based on the use of project management information system (PMIS) [9, 10] for having the execution date fixed by the independent quality reviewer with the project manager [11, 12]. Then, the project review process is started by retrieving the lessons learned data from the lessons learned repository [13-16], which were collected from the previous project reviews for the relevant ERP implementation projects, for the preparation of conducting the project document review and stakeholder interviews.

A case study methodology was applied to the historical lessons learned data of the ERP implementation projects [17-24] conducted by a solution provider for their customers in the various industries in Japan, which were retrieved for a period of four years from 2014 to 2017 to analyze how the lessons learned collected from the project reviews of the earlier projects were reused in those of the succeeding projects conducted during the period. The set of projects was determined based on the following criteria that the solution provider is [25]:

• To provide a project manager and project team;
• To be responsible for providing particular results based on contractual agreements;
• To provide advisory services that are mainly relevant to meet customers’ project goals;
• To provide project work with the budget of the

contract that is greater than the threshold value; and

• To have an agreement with the customer for conducting the project reviews (or quality audits) at the selected phases or project post mortem for continuous improvement.

Use of lessons learned based on the past project review results was found to be effective in focusing on the specific areas projected for improvement during the processes of conducting the project document review and key stakeholder interviews, as well as putting together the practical recommendations for the findings to finalize the results of the project review, which were to be formally presented and submitted to the customer as the results of the quality audit.

This article is structured as follows: Section II reviews the works that relate to lessons learned definitions, lessons learned processes, a lessons learned session, commonly used synonyms for lessons learned and their adoption. Section III presents the literature review of PMIS and its current configuration implemented. Use of lessons learned effectively to conduct the project review for the ERP project carried out by the solution provider is presented in Section IV. Results based on the use of lessons learned from the past project reviews are summarized in Section V. Finally, Section VI is composed by the conclusion.

II. Related Works

The Project Management Institute defines the term lessons learned as “the learning gained from the process of performing the project” in the 3rd Edition of PMBOK [26], such as the activities of the project that went well or could be improved [27]. Another definition used by the American, European and Japanese Space Agencies is: “A lesson learned is knowledge or understanding gained by experience.” The experience may be positive, as in the successful test or mission, or negative, as in a mishap or failure [28, 29]. The latest PMBOK 6th Edition defines it in more detail as “the knowledge gained during a project which shows how project events were addressed or should be addressed in the future for the purpose of improving future performance [30].”

The literature on learning organization has described a set of lessons learned processes named as follows: collect, capture, gather, verify, store, share, distribute, disseminate, reuse, and apply [31]. Lessons learned processes have been deployed in commercial, government, and military organizations since the late 1980s to capture, store, disseminate, and share experiential working knowledge [29]. PMBOK 3rd Edition defines a process as a set of interrelated actions and activities performed to achieve a specified set of products, results, or services [26]. The purpose of a lessons learned process is to define the activities required to successfully capture and use lessons learned. The lessons learned process includes five steps: identify, document, analyze, store and retrieve. The following are the details of the five steps [13-16]:

• Step 1: Identify Lessons Learned is to identify comments and recommendations that could be valuable for future projects
• Step 2: Document Lessons Learned is to document and share the findings in the following manner:

o Detailed Report – The detailed lessons learned report consists of the data captured during the lessons learned session o Summary – This is a one-page brief for leadership summarizing the findings and providing recommendations for correcting the findings o Executive Report – This report should present an overview of the lessons learned process and a summary of project strength o Findings – A summary of the issues found during the review process o Recommendations – Recommendation are actions to be taken to correct findings

• Step 3: Analyze lessons Learned is to analyze and organize the lessons learned for application of results
• Step 4: Store Lessons Learned is to store in a repository
• Step 5: Retrieve Lessons Learned is to retrieve for use on current projects

A lessons learned session focuses on identifying project successes and project failures, and includes recommendations to improve future performance on projects [26]. During the project lifecycle, the project team and key stakeholders identify lessons learned concerning the technical, managerial, and process aspects of the project. The lessons learned are compiled, formalized, and stored through the project’s duration [26].

Commonly used synonyms for lessons learned include project assessments, project reviews, project completion audits, postmortems, reviews, appraisals, after-action reviews, debriefings and post-implementation evaluations [7, 32]. The project management literature describes lessons learned as practices that [32]:

• Are quality improvement oriented and help correct process efficiency and effectiveness problems in a timely manner [33]
• Help deliver more successful projects, improve customer satisfaction [33] and help participants learn about successful and unsuccessful practices [34]
• Involve dissemination and sharing functions [34]
• Involve both inter-and intra-project learnings [33] because they assist with externalizing implicit knowledge [7]

By a postmortem, it is meant to be a collective learning activity which can be organized for projects either when they end a phase or are terminated [35]. The main motivation is to reflect on what happened in the project in order to improve future practice – for the individuals that have participated in the project and for the organization as a whole [35].

An audit is structured, independent process used to determine if project activities comply with organizational and project policies, processes, and procedures [30]. A quality audit is usually conducted by a team external to the project, such as the organization’s internal audit department, PMO (Project Management Office), or by an auditor external to the organization. Quality audit objectives may include, but are not limited to [30]:

• Identifying all good and best practices being implemented;
• Identifying all nonconformity, gaps, and shortcomings;
• Sharing good practices introduced or implemented in similar projects in the organization and/or industry;
• Proactively offering assistance in a positive manner to improve the implementation of processes to help raise team productivity; and
• Highlighting the contributions of each audit in the lessons learned repository of the organization.

The ERP implementation methodology [17] used by the solution provider is based on the traditional waterfall model consisting of the four phases. A project review (or quality audit) is conducted by the independent quality reviewer who is external to the project on the project documents against the project review checklist relevant for the target phase or project post mortem along with the interviews of key project stakeholders as shown in Fig. 1.

The major objectives of the project reviews (or quality audits) are as follows:

• Focus on project management, but also assess organizational and technical readiness
• Conduct on-site interviews with key project

stakeholders

• Evaluate project documents
• Uncover project risks and issues that are documented in a set of review reports, providing actionable recommendations for improvement of project management

Delivery roadmap for a typical project review is as follows:

• Initiate:

o Contact project manager
o Gather and review project information

• Plan:

o Conduct review planning meeting o Fix project review schedule in PMIS o Maintain project review checklist o Retrieve relevant lessons learned data

• Execute:

o Prepare

• Prepare for interview

o Conduct

• Study project documents
• Perform interviews
• Apply retrieved lessons learned
• Analyze project documents
• Analyze interviews
• Discuss initial observations

o Complete

• Apply retrieved lessons learned
• Develop findings
• Develop recommendations
• Develop a detailed report
• Develop a summary report
• Present a summary report
• Present a detailed report

• Close:
o Maintain project review checklist
o Maintain lessons learned register
o Archive review results in PMIS
o Store in lessons learned repository

Table 1 shows the description of the project review checklist for the project document review, which covers the checklist items (use cases and elements) for all phases of the project. The checklist consists of the two major methodologies, the project management knowledge areas [30] and ERP (i.e. processes, products and services [36]) implementation methodology since ERP implementation faces many difficulties that cause its failure [18, 37].

Table 2 shows the description of the project review checklist for the project stakeholder interviews, which covers the checklist items (use cases and elements). The checklist consists of the three major methodologies, the project governance, project management knowledge areas and ERP implementation methodology.

Table 3 describes the scope of the project review (or quality audit) based on the methodology in terms of the project governance, project management knowledge areas and ERP implementation methodology.

2	Project Phase 1	\ Project \ ) Phase 2 /	Project \ Phase 3 /	Project Phase 4


	Phase 1	Phase 2	Phase 3	Phase 4
	Project Review	Project Review	Project Review	Project Review

Project Postmortem

Postmortem

Review

Fig.1. Project Review by Phase and Postmortem Review

Table 1. Project Review Checklist for Project Document Review

Methodology	Use Case (Process)	Element (Document Name)	Project Phase 1	Project Phase 2	Project Phase 3	Project Phase 4	Project Postmortem
Project Management Knowledge Areas	Project Integration Management	Project Statement of Work	X	-	-	-	X
		Business Case	X	X	-	-	X
		Organizational Process Assets	-	X	-	-	X
		Project Charter	X	-	-	-	X
		Preliminary Scope Statement	X	-	-	-	-
		Project Management Plan	X	X	X	X	X
		Project Kick-off Presentation	X	-	-	-	X
		Issue Management Procedure	X	-	-	-	X
		Issue Register	X	X	X	X	X
		Requested Changes	-	X	X	X	X
		Lessons Learned Register	X	X	X	X	X
		Stakeholder Register	X	X	X	X	X
	Project Scope Management	Scope Management Plan	X	-	X	X	X
		Project Scope Statement	X	X	X	X	X
		Work Breakdown Structure	X	X	X	X	X
		Work Breakdown Structure Dictionary	X	X	X	X	X
		List of Project Deliverables	-	-	X	X	X
		Accepted Deliverables	-	X	X	X	X
	Project Schedule Management	Schedule Management Plan	X	-	-	-	X
	Project Schedule Management	Project Schedule	X	X	X	X	X
	Project Cost Management	Cost Management Plan	X	-	X	X	X
	Project Cost Management	Cost Baseline	X	X	X	X	X
	Project Quality Management	Quality Management Plan	X	X	X	X	X
	Project Resource Management	Resource Management Plan	X	X	X	X	X
		Project Organizational Charts	X	-	X	X	X
		Roles and Responsibilities	X	X	X	X	X
		Team Performance Assessments	-	-	X	X	X
	Project Communications Management	Communications Management Plan	X	X	X	X	X
		Performance Reports	-	X	X	X	X
		Project Reports	X	X	X	X	X
		Latest Steering Committee Meeting Minutes	X	X	X	X	X
		Latest Project Management Team Meeting Minutes	X	X	X	X	X
	Project Risk Management	Risk Management Plan	X	X	X	X	X
	Project Risk Management	Risk Register	X	X	X	X	X
	Project Procurement Management	Contract Management Plan	X	-	-	-	-
	Project Stakeholder Management	Stakeholder Engagement Plan	X	X	X	X	X
ERP (Processes, Products and Services) Implementati on Methodology	Organizational Change Management (OCM)	OCM Charter	X	-	-	-	X
		OCM Master Plan	-	X	X	X	X
		Stakeholder Analysis	X	-	-	-	X
		Communications Plan	-	-	X	X	X
	Business Process Management	Business Blueprint	-	X	X	X	X
		Functional Specifications - RICEF Objects	-	-	X	X	X
		Development List	-	-	X	X	X
	Technical Solution Management	Future Technical System Landscape	X	-	-	-	X
		Support Expectations	-	X	-	-	-
		System Administration Procedures	-	X	-	-	-
		System Landscape Design (DEV, QA, PRD)	-	-	X	X	X
		Production Support Processes	-	-	X	X	X
	Integrated Solution Management	Development Test Plans	-	-	X	X	X
		Final Test Plan	-	-	X	X	X
		End-User Testing	-	-	X	X	X
	Data Management	Data Migration Strategy	X	-	-	-	X
	Data Management	Data Migration Plan	-	X	-	-	X
	Training	Project Team Training Plan	-	X	-	-	X
		End-User Training Documentation	-	-	X	X	X
		Training Evaluation Results	-	-	X	X	X
		End-User Training Evaluation Summary	-	-	-	-	X
	Production Cutover	Cutover Plan	-	-	X	X	X

Table 4 shows the criteria for evaluation of the audit findings based on the five levels of risk severity, “No Finding”, “Low Risk”, “Medium Risk”, “High Risk” and “Problem”.

Table 5 is a sample of the lessons learned register extracted from one of the review results (i.e. based on a postmortem review) based on the record layout consisting of all the mandatory fields.

Table 6 is a sample of the project review dashboard extracted from one of the review results (i.e. based on a postmortem review) stored in the lessons learned repository. It shows a total of 12 lessons learned consisting of one finding with the severity level, “Problem” and eleven findings with the severity level, “High Risk”.

Table 2. Project Review Checklist for Stakeholder Interviews

Methodology	Use Case	Element (Interview Topic)
Project Governance	Sponsor Interview	Project Governance
Project Management Knowledge Areas	Sponsor Interview	Project Sponsor Role and Involvement
		Project Goals and Objectives
		Project Success Criteria
		Value Realization Strategy
		Project Information Sheet Contents
	Functional Team Interview	Project Management Activities (Risk, Scope, etc.)
	Functional Team Interview	Project Issues
ERP (Processes, Products and Services) Implementation	Functional Team Interview	Functionality Definition
		Functionality Status
		Production Support
		Project Team Training and Knowledge Transfer

Table 3. Project Review Scope

Methodology	Use Case (Process)
Project Governance	Project Governance – Governance
Project Management Knowledge Areas	Project Integration Management
	Project Scope Management
	Project Schedule Management
	Project Cost Management
	Project Communications Management
	Project Resource Management
	Project Quality Management
	Project Risk Management
	Project Procurement Management
	Project Stakeholder Management
ERP (Processes, Products and Services) Implementation Methodology	Solution Readiness – Requirements
	Solution Readiness – Building
	Solution Readiness – Testing
	Business and User Readiness – Organizational Change Management
	Business and User Readiness – End User Training and Documentation
	Data Readiness – Data Readiness
	Technical Infrastructure Readiness – Technical Infrastructure Readiness
	Support Readiness – Production Support and Center of Excellence
	Support Readiness – Knowledge Transfer and Documentation

Table 4. Risk Reporting Levels

Severity	Description	Action Required
No Finding	Topic in good order	No action necessary.
Low Risk	Topic with minor finding	Minimum impact. No management action is required.
Medium Risk	Topic with serious finding	Some disruption may occur. No immediate management action is required. However, continuous risk monitoring has to be initiated and future action may be needed if the situation persists.
High Risk	Topic with critical finding	Unacceptable risk. Major disruption is likely to occur. Priority management attention is required to bring the situation under control.
Problem	Topic with issue	A disruption has already occurred to the project. Immediate management attention is required to bring the situation under control.

Table 5. Lessons Learned Register Sample

Date	2014/8/13
Project ID	PS-05325
Project Name	Project T
Review Period	Postmortem
Method (PMBOK / ERP)	PMBOK
Use Case (Process)	Project Governance – Governance
Element	Accountability (Escalation Procedure)
Lessons Learned (Finding) Headline	Although the issue with delay in creation of the master data by a business unit had been reported week after week, it was never cleared till the end of the project.
Severity	High Risk
Finding	According to the weekly progress report, although the issue with delay in creation of master data by a business unit had been reported week after week, it was never completed due to running out of time based on the comment after all that work could not be completed from lack of man-hours.
Impact	Due to not timely taking effective corrective action for the issue, there is a possibility for the significant impact to occur in the project such as the go-live delay and cost overrun.
Recommendation	By clearly documenting the escalation procedure to define the ultimate accountability, please make sure to be able to timely take effective corrective action for the issue.

Table 6. Project Review Dashboard Sample

Risk Reporting Levels

Methodology	Use Case (Process)	Problem	High Risk	Medium Risk	Low Risk	No Finding
Project Governance	Project Governance – Governance		2
Project Management Knowledge Areas	Project Integration Management		3
	Project Scope Management		2
	Project Schedule Management		1
	Project Cost Management					-
	Project Communications Management					-
	Project Resource Management					-
	Project Quality Management					-
	Project Risk Management		1
	Project Procurement Management					-
	Project Stakeholder Management					-
ERP (Processes, Products and Services) Implementation Methodology	Solution Readiness – Requirements					-
	Solution Readiness – Building					-
	Solution Readiness – Testing		1
	Business and User Readiness – Organizational Change Management		1
	Business and User Readiness – End User Training and Documentation					-
	Data Readiness – Data Readiness	1
	Technical Infrastructure Readiness – Technical Infrastructure Readiness					-
	Support Readiness – Production Support and Center of Excellence					-
	Support Readiness – Knowledge Transfer and Documentation					-

III. Literature Review of PMIS and its Production Configuration

PMIS, which is part of enterprise environmental factors, provides access to information technology (IT) software tools, such as scheduling, cost, and resourcing software tools, work authorization systems, configuration management systems, information collection and distribution systems, as well as interfaces to other online automated systems such as corporate knowledge base repositories. Automated gathering and reporting on KPIs can be part of this system [30]. PMIS provides a wide range of functions directly supporting a complex of a process involving various projects related activities: planning, monitoring, control and others [38]. In the IT industry, Gartner Research estimates that 75% of large IT projects managed with the support of a PMIS will succeed, while 75% of projects without such support will fail [39]. Using PMIS to manage projects, while not sufficient to ensure project success, has thus become a necessity [40]. The most appropriate PMIS configuration defined depends on the project situation [41]. Project situation requirements for PMIS have been identified accordingly to project classification [42] based on the project type, product, size, organization, management, planning approaches and related guidance, as well as project environments and specific requirements, enterprise environment factors and organizational process assets [30]. Definition of the PMIS configuration requirements must include the following information [41] such as data entities or work items used in the project, attributes or data fields of each data entity and processes or workflows related to the data.

The configuration use case elements supported by the PMIS implemented for the use by the solution provider are shown in Table 7. It aims to provide the KPIs, risk registers and reports such as project financials in terms of EVM. This part of the paper is based on the previous study conducted [9] in 2017.

Table 7. PMIS Production Configuration Use Case Elements

Use Case	Elements
Project Management	Project Identification	Key Project Information
	Project Classification	Contract Type (i.e. T&M, FFP), Quality Requirements, Governance
	Project Scope Description	Project Scope
	Management Summary	Status Reporting
	Status Indicators	Overall, Margin, Cost, Accounts Receivable, Schedule, Risks, Issues, Resources, Quality, Scope, Customer Satisfaction, Governance, Value Management
	Key Issues	Top Issues Reporting
	Key Risks	Top Risks Reporting
	Project Financials	Expenses (Bid Baseline / PM Baseline), Revenue (Bid Baseline / PM Baseline), Earned Value Management (EVM)
	Project Milestones	Performance Reporting
	Change Request	Change Request Management
	Issue List	Issue Management
	Risk Register	Risk Management
	Financial Contract	Plan (Man Days)
	WBS	Phases, Schedule, Milestones
	Roles w/ Assigned Tasks	Man Days by Resource
	Resources (Plan vs. Actual)	Budget Monitoring
	Contact List	Project Manager, Quality Manager, Sales
	Authorization	Access Authorization Level
	Accounting	Plan, Actual, Revenue, Expenses, Billing, Backlog
Portfolio Management	Reports	Online Portfolio Report, Change Request Report, Issue and Risk Report, Action Item Report, Financial Contract Report, Consolidated Financial Report, Portfolio Revenue Forecast Report, Solution Scope Report

IV. Use of Lessons Learned Effectively to Conduct Project Review for ERP Implementation Project

The process for applying the lessons learned collected from the previous project reviews to conduct the project review for the ERP implementation project consists of two major processes. One is Prepare for Project Review Leveraging Lessons Learned Repository that is conducted at the beginning of the project or phase by the independent quality reviewer when the project is identified to have an agreement with the customer for conducting the project reviews (or quality audits) at the selected phases or project post mortem for continuous improvement. The other is a process of Conduct Project Review that is conducted by the independent quality reviewer once the project review schedule is fixed in the project review control list maintained in PMIS upon agreement with the project manager for the set of projects described in Section I. PMIS applied to trigger the initiation of the project review systematically during the selected project phases is discussed in detail below.

A. Apply Lessons Learned Process to Conduct Project Review for ERP Implementation

Systematic overview of the use of lessons learned process to conduct the project review that is triggered by the appropriate project initiation information from PMIS can be expressed in IDEF0 (Integration DEFinition level 0) [43, 44] as shown in Fig. 2. This is the top-level context diagram A-0.

It is decomposed to the next level diagram with a systematic framework that consists of two nodes, A1 and A2 as shown in Fig. 3. Node A1 is Prepare for Project Review Leveraging Lessons Learned Repository process that is triggered by the relevant project initiation information from PMIS to be conducted at the beginning of the selected phases of the project. It is specifically positioned in preparation for conducting on-site interviews with key project stakeholders as well as evaluating project documents based on the retrieved lessons learned data collected from the previous project reviews, to influence the phase and project results positively for continuous improvement. Node A2 is a process of Conduct Project Review to be conducted in the selected phases in the project duration. It is positioned to focus on project management, but also assess organizational and technical readiness, and uncover project risks and issues that are documented in a set of review reports, providing actionable recommendations for improvement of project management.

Operating Schedule

Lifecycle Directive

Project Initiation Information from PMIS

Project Renew Control List ___________

Scope Document __________________

Project Information Sheet Template

Lessons Learned Data Retneval _______

Project Review Checklist _______________

Project Review Documents ___________

Lessons Learned Register Template

Apply Lessons Learned to Conduct Project Review for ERP Implementation

Updated Project Renew Control List Retrieved Lessons LeamedData Project Review Result Report _______ Updated Project Information Sheet Updated Project Review Checklist Reviewed Project Review Documents Review Results .Archived in PMIS

Updated Lessons Learned Register

Note 1: Dashed (---)

arrows represent physical items. .All other arrows represent information.

Note 2: PMIS = Project Management Information System

IT Service or Infrastructure

Support

Infrastructure

Standard Commercial Transaction Set

Fig.2. Apply Lessons Learned to Conduct Project Review for ERP Implementation

Lifecycle Directive

Project Initiation

Information from PMIS

Project Review Control List

Scope Document Lessons Learned Data

Retrieval

Project Initiation Information from PMIS

Prepare for Project Review Leveraging Lessons Learned Repository

Updated Project Review Control List

Project Information Sheet Template

Project Review Checklist __________

Project Review Documents _______

Lessons Learned Register Template

Project Information Sheet

Updated Project Review Checklist i Project Review Documents

Review I Results Archived in PMIS

Lessons Learned Register

Updated

Reviewe

Updated

Fig.3. Prepare for Project Review Leveraging Lessons Learned Repository and Conduct Project Review

Project Review Result Report

B. Classify Project Having Project Review Contracted, Fix Project Review Schedule in PMIS and Retrieve Lessons Learned Data

The decomposition of node A1 to 6 activities is shown in Fig. 4. PMIS strategically implemented is effectively used by the independent quality reviewer who does not belong to the organization unit responsible for the project delivery, in searching for the projects classified for the contractual needs of having project reviews conducted at the selected phases of the projects. This process for having the project review conducted by the independent quality reviewer plays the most important role to properly kick off the project review process and get the project review schedule fixed in the project review control list maintained in PMIS based on the agreement with the project manager. Then, the independent project reviewer is to retrieve the lessons learned data collected from the previous project reviews conducted for the ERP projects so that they can be applied to the project review process in preparation for conducting the project document review as well as key project stakeholder interviews.

Fig.4. Classify Project Having Project Review Contracted, Fix Project Review Schedule in PMIS and Retrieve Lessons Learned Data

Below are the major activities required to clarify the project having the project review contracted, fix the project review schedule in PMIS and retrieve the lessons learned data from the lesson learned repository.

• Node A11; Classify Project Having Project Review Contracted: The independent quality reviewer is to check (during the 1^st two weeks of the month) if there is any project in PMIS which is having the project review (or quality audit) contracted and relevant for triggering the initiation of the project review process based on the following criteria that the solution provider is: o To provide a project manager and project team;

o To be responsible for providing particular results based on contractual agreements;

o To provide advisory services that are mainly relevant to meet customers’ project goals;

o To provide project work with the budget of the contract that is greater than the threshold value; and o To have an agreement with the customer for conducting the project reviews (or quality audits) at the selected phases or project post mortem for continuous improvement.

Table 8 shows a snapshot of the project initiation information from PMIS taken in June 2017 for classifying the project having the project review contracted.

• Node A12; Request Project Review Planning Meeting: Once a relevant project is found: o The independent quality reviewer is to send an email to the project manager responsible

for the execution of the project, which is also copied to the delivery manager in charge of the portfolio category, based on the explanation for the need of getting a project review planning meeting conducted before a proposed due date for completion stated on the email.

o The project manager is to send back an hour meeting request with a date specified for having the project review planning meeting conducted.

o The independent quality reviewer is to respond to the meeting invite to have the meeting date finally fixed.

Node A13; Maintain Project Review Schedule in PMIS: The independent quality reviewer is: o To set the preliminary project review schedule in the project review control list maintained in PMIS based on the project schedule stated in the scope document which is the addendum of the contract for the project as shown in Table 9.

Node A14; Agree on Project Review Schedule: The independent quality reviewer is:

o To have an agreement with the project manager in the project review planning meeting for the scope of the project review as well as the dates and duration of the project review in reference to the preliminary project review schedule set in the project review control list which is maintained in PMIS.

Node A15; Update Project Review Schedule in PMIS If Needed: The independent quality reviewer is:

o To update the preliminary project review o schedule maintained in the project review control list of PMIS if the proposed preliminary project review schedule was not acceptable to the project manager due to whatever the reason may be.

• Node A16; Search in Lessons Learned Repository:

Once the project review schedule is agreed and fixed by the project manager:

The independent quality reviewer is to search and retrieve the lessons learned data collected from the previous project reviews for the ERP projects conducted by the solution provider so that they can be applied to the project review process in preparation for conducting the project document review as well as key project stakeholder interviews.

Table 8. Online Portfolio Report for Project Having Project Review Contracted

Project ID	Industry Sector	Project Manager	Project Name	Period	Contract Type	Project Type	Planned Finish
PS-10782	Consumer / Trading	Project Manager 1	Project K	2017 M 05	T&M	Consulting Project	2020/3/31
PS-11634	High Tech	Project Manager 2	Project S	2017 M 05	T&M	Consulting Project	2018/3/30

Table 9. Project Review Control List Maintained Prior to Conducting Project Review

Industry Sector	Project Name	Project Type	Project Phase	Service Name	Planned Finish	Actual Finish	User Status	Severity
Consumer / Trading	Project K	Consulting Project	Phase 2	Project Review	2017/9/8		In Preparation	®
High Tech	Project S	Consulting Project	Phase 3	Project Review	2017/9/26		In Preparation	®

C. Conduct Project Review, Apply Lessons Learned Data, Store in Lessons Learned Repository and Archive Results in PMIS

The decomposition of node A2 to 6 activities is shown in Fig. 5. In an iterative process, the project review (or quality audit) by the independent quality reviewer is to be conducted in the selected phases in the project duration based on the project review scheduled agreed and fixed with the project manager that is maintained in the project review control list of PMIS. It is positioned to focus on project management, but also assess organizational and technical readiness, and uncover project risks and issues that are documented in a set of review reports, providing actionable recommendations for improvement of project management.

Fig.5. Conduct Project Review, Apply Lessons Learned Data, Store in Lessons Learned Repository and Archive Results in PMIS

Below are the steps of major activities required to conduct the project review, apply lessons learned data, store in lessons learned repository and archive results in PMIS.

• Node A21; Identify Project Review Due in Control List: By leveraging the project review control list, which is maintained in PMIS, the

independent quality reviewer is to check the set of relevant projects (based on the criteria set by the solution provider) for triggering the iterative process of conducting the project review (or quality audit) on the 25th of every month. Table 10 shows the set of selected projects classified for having the project reviews at the selected phases contracted for continuous improvement.

Table 10. Project Review Control List with Project Review Due Next Month

Project ID	Industry Sector	Project Name	Project Type	Project Phase	Service Name	Planned Finish	Actual Finish	User Status	Severity
PS-10782	Consumer / Trading	Project K	Consulting Project	Phase 2	Project Review	2017/9/8		In Preparation	®
PS-11634	High Tech	Project S	Consulting Project	Phase 2	Project Review	2017/2/2	2017/2/2	Green	✓
PS-11634	High Tech	Project S	Consulting Project	Phase 3	Project Review	2017/9/26		In Preparation	®
• Node A22; Request Project Review Meetings: o Discuss initial observations Once a relevant project is found: Complete: o The independent quality reviewer is to send o Apply retrieved lessons learned an email to the project manager responsible o Develop findings for the execution of the project, which is o Develop recommendations also copied to the delivery manager in o Develop a detailed report charge of the portfolio category, based on o Develop a summary report the need of having a series of the identified o Present a summary report project stakeholder interviews scheduled o Present a detailed report along with the due date for submission of Criteria for evaluation of the audit findings is based the response stated on the email. on the five levels of risk severity: o The project manager is to send back the o “No Finding” response with the dates and times specified o “Low Risk” for having the key project stakeholder o “Medium Risk” interviews to be conducted by the due date. o “High Risk” • Node A23; R equest Project Review Target ^o ^“P^robl^em” Documents: o The independent quality reviewer is to send The independent quality reviewer is to present the an email to the project manager responsible summary report to the customer sponsor and key project for the execution of the project, which is stakeholders. Also, the independent quality reviewer is to also copied to the delivery manager in submit the detailed report to the customer sponsor as well charge of the portfolio category, based on as the key project stakeholders so that they plan the the need of a set of project documents for corrective actions for the recommendations based on the evaluation based on the project review findings as the results of the review. checklist and project information sheet along with the due date for submission of the • Node A25; Store in Lessons Learned Repository: response stated on the email The independent quality reviewer is to store the o The project manager is to send back the project review result report and updated lessons response with the updated project review learned register maintained based on the findings checklist having all the target documents as the results of the project review in the lessons mapped to each checklist item, target project learned repository. documents for evaluation and updated • Node A26; Archive Results in PMIS: The project information sheet by the due date. independent quality reviewer is to archive the • Node A24; Execute Project Review: The updated project information sheet, project review independent quality reviewer is to conduct the result report, updated project review checklist, project review (or quality audit) based on the reviewed project review documents and updated steps as follows: lessons learned register maintained based on the Prepare: findings as the results of the project review in o Prepare for interview PMIS. Also, the independent quality reviewer is Conduct: to enter the actual completion date for activating o Study project documents the completion flag for the project review in the o Perform interviews project review control list maintained in PMIS as o Apply retrieved lessons learned shown in Table 11. o Analyze project documents o Analyze interviews

Table 11. Project Review Control List Updated upon Completion of Project Review

Project ID	Industry Sector	Project Name	Project Type	Project Phase	Service Name	Planned Finish	Actual Finish	User Status	Severity
PS-10782	Consumer / Trading	Project K	Consulting Project	Phase 2	Project Review	2017/9/8	2017/9/8	Green	У
PS-11634	High Tech	Project S	Consulting Project	Phase 2	Project Review	2017/2/2	2017/2/2	Green	V
PS-11634	High Tech	Project S	Consulting Project	Phase 3	Project Review	2017/9/26	2017/9/26	Green	V

V. Results

There is a total of 102 lessons learned collected from the 17 project reviews performed for the 10 relevant ERP projects conducted by the solution provider in Japan for the period of 4 years from 2014 to 2017 applied to this case study as shown in Table 12.

Phase 3 Project Review for Project C completed on February 27, 2015 has been ranked first among the 17 projects reviews with the 14 lessons learned registered. Postmortem Review for Project T on August 13, 2014 with the 12 lessons learned registered is ranked second. Ranked third is Phase 3 Project Review for Project J on June 28, 2016 with the 9 lessons learned registered.

Table 12. Number of Lessons Learned Used in Case Study

#	Date	Project ID	Project Name	Review Period	Lessons Learned
1	2014/2/12	PS-04406	Project Y	Phase 2	5
2	2014/3/31	PS-02924	Project E	Phase 3	7
3	2014/4/15	PS-03196	Project H	Phase 2	6
4	2014/5/8	PS-04406	Project Y	Phase 3	7
5	2014/6/30	PS-04238	Project N	Phase 3	5
6	2014/7/30	PS-04238	Project N	Phase 4	3
7	2014/8/13	PS-05325	Project T	Postmortem	12
8	2014/11/18	PS-05325	Project T	Phase 3	7
9	2015/2/27	176012765	Project C	Phase 3	14
10	2015/3/10	PS-06790	Project B	Phase 2	3
11	2015/3/17	PS-05325	Project T	Phase 4	5
12	2015/5/27	PS-06790	Project B	Phase 4	3
13	2016/6/28	PS-09862	Project J	Phase 3	9
14	2016/11/22	PS-09862	Project J	Phase 4	2
15	2017/2/1	PS-10717	Project S	Phase 2	5
16	2017/9/7	PS-10782	Project K	Phase 2	5
17	2017/9/25	PS-11634	Project S	Phase 3	4

The reuse of lessons learned is analyzed based on the frequency of use sorted by the element (or topic) and lessons learned headline among the 102 lessons learned data retrieved from the lessons learned repository. Table 13 shows the reuse of the lessons learned of the findings with WBS related topics such as progress control 13 times in total among the 13 respective project reviews or 76% that is the reuse % divided by the total of 17 project reviews conducted during the period. Specifically, the index term of lessons learned with “Some tasks not defined in WBS” was repeatedly reused in the 4 respective project reviews. Also, the index term of lessons learned with “Actual dates not maintained in WBS” was repeatedly reused in the 6 respective project reviews.

Likewise, Table 14 shows the reuse of the lessons learned of the findings with Requested Changes 10 times in total among the 10 respective project reviews (or 59%). Specifically, the index term of lessons learned with “Change request log not created” was repeatedly reused in the 4 respective project reviews. The index term of lessons learned with “Change request log not updated appropriately” was also repeatedly reused in the 4 respective project reviews.

Lastly, Table 15 shows the reuse of the lessons learned of the findings with Risk Management related topics 8 times in total among the 8 respective project reviews (or 47%). The 2 index terms of lessons learned with “Process not functioning in Risk Management” and “Risk Management Plan / process not defined” were repeatedly reused 4 times each in the 4 respective project reviews.

Table 13. Reuse of Lessons Learned of Findings with Progress Control in WBS

#	Date	Project Name	Element	Index Term	Lessons Learned (Finding) Headline
1	2014/2/12	Project Y	WBS	Some tasks not defined	As the work packages for the customer tasks in scope are not created and maintained in the WBS, the progress of the entire tasks cannot be managed against the project scope.
2	2014/6/30	Project N	WBS	Some tasks not defined	There is an item described as quality audit (which is named project review in the WBS) on the SOW. It is also stated to be conducted in Phase 3 and Phase 4 on the project approach document. However, those 2 tasks are not actually created in the WBS.
3	2014/8/13	Project T	Progress Control in the WBS	Actual dates not maintained	There are many tasks in the WBS without having the actual start / end dates entered for managing the progress where the planned start / end dates were overdue in the past based on the date of update.
4	2014/8/13	Project T	WBS	Some tasks not defined	Only a part of the users was included in the project team. However, the user tasks were not defined in the WBS to manage the progress although the % of participation for the project work by the user was decided.
5	2014/11/18	Project T	WBS	Actual dates not maintained	There are 13 uncompleted activities having the planned start / end dates in the past compared to the last date of the WBS update. Some have the delayed actual start dates against the plan and others do not even have the actual dates entered. Some activities which are not started have the planned start dates of 40 days in the past.
6	2015/3/17	Project T	WBS	Some tasks not defined	Schedule for retesting the programs related to the 3 systems planned after January 2015 has not been entered in the WBS yet.
7	2016/6/28	Project J	WBS	Actual dates not maintained	As of the WBS update on 2016/6/17, actual start / end dates of the 62 tasks having the planned end dates before 2016/6/16 were not maintained for managing the progress.
8	2016/6/28	Project J	WBS	Misleading date maintenance	End date of the delayed task, the system basic design document update is now expressed "To be scheduled" after having the end date changed over and over.
9	2016/6/28	Project J	WBS	Actual dates not maintained	Due to lack of the management process of another WRICEF WBS updated on 2016/6/18, the actual start / end dates of the 5 tasks having the planned end dates before 2016/6/17 not maintained for managing the progress.
10	2017/2/1	Project S	WBS	Actual dates not maintained	Actual start / end dates of the activities in the WBS are not properly maintained for managing the progress.
11	2017/2/1	Project S	WBS	Process not defined	Schedule management process for the progress of planned activities in the WBS not defined.
12	2017/9/7	Project K	WBS	Actual dates not maintained	Actual start / end dates of the work packages are not properly maintained for timely managing the progress.
13	2017/9/7	Project K	WBS	Dependencies not managed	Dependencies and relationships of the tasks managed by each team in the WBS are not managed against the entire tasks in the WBS in an integrated manner.

Table 14. Reuse of Lessons Learned of Findings with Requested Changes

#	Date	Project Name	Element	Index Term	Lessons Learned (Finding) Headline
1	2014/4/15	Project H	Requested Changes	Change request log not created	Change request log to maintain all the change requests is not created although the change requests are already existing.
2	2014/5/8	Project Y	Requested Changes	Change request log not created	Change request log to maintain all the change requests is not created although the change requests are existing.
3	Table 15	Project N	Requested Changes	Process not functioning	There is a knowledge transfer task for training the key users on operating 38 updated functions using the production system on the SOW. However, it is treated out of scope without conducting the change request approval process.
4	2014/8/13	Project T	Requested Changes	Change request log not created	Regarding the change control process, although the change request policy definition and the individual change request sheet exist, the change request log to list all the change requests is not created.
5	2014/11/18	Project T	Requested Changes	Process not done timely	Change request log has the change request related requirements as well as the defects registered. Among the 16 defects registered, one high priority item has 37 change request target objects with the estimated workload of 90.6 man day. It implies the requirements of the timely CR approval judgement process.
6	2015/2/27	Project C	Requested Changes	Change request log not created	Definition of the Change Request Control approach, the individual change request sheet and the change request log to list all the change requests are not created.
7	2015/3/10	Project B	Requested Changes	Change request log not updated	Change request log is not maintained despite some approved change requests existing.
8	2015/5/27	Project B	Requested Changes	Change request log not updated	Change request log created based on the previous finding recommendation is found to be not maintained timely.
9	2016/11/22	Project J	Requested Changes	Change request log not updated	Expected completion date and actual completion date of the 15 approved change request items entered in the change control log are not maintained properly for managing the progress.
10	2017/2/1	Project S	Requested Changes	Change request log not updated	Change request log not properly maintained despite some changes existing.

Table 15. Reuse of Lessons Learned of Findings with Risk Management Plan

#	Date	Project Name	Element	Index Term	Lessons Learned (Finding) Headline
1	2014/2/12	Project Y	Risk Management Plan	Process not functioning	Risk management plan defined along with the creation of the risk register is not functioning properly as the risk management process are not conducted by the project teams.
2	2014/6/30	Project N	Risk Management Plan	Process not functioning	Although there was a minimum explanation of the risk management process stated in the risk management plan, the risk register is not created accordingly to the plan and the risks are not managed based on the risk management process defined.
3	2014/8/13	Project T	Risk Management Plan	Plan / process not defined	There is no risk management plan defining the risk management process to be conducted by the project team. Neither the risk register is created.
4	2014/11/18	Project T	Risk Register	Plan / process not defined	Although the risk register is created, there is no risk management plan existing. As the risk management process is not properly functioning, there are some risks initially created in the risk register but not timely maintained for the purpose of effective risk management.
5	2015/2/27	Project C	Risk Planning and Identification	Plan / process not defined	Although the project risk register is created and maintained, there is no risk management plan of the project management plan existing
6	2017/2/1	Project S	Risk Management Plan	Plan / process not defined	Risk management process for handling the risk response plan in the team is not defined.
7	2017/9/7	Project K	Risk Register	Process not functioning	Risk management process is not properly functioning as some items initially registered were not updated for 6 weeks.
8	2017/9/25	Project S	Risk Register	Process not functioning	Risk management process is not functioning as the risk register is not properly managed for some identified case.

Furthermore, there are lessons learned of 13 other topics with the frequency of reuse over 2 times identified in addition to the above mentioned 3 topics of lessons learned most frequently reused in the project reviews conducted during the period.

The breakdown summary list of the reuse of lessons learned over 2 times by topic is shown in Table 16. The topic of lessons learned with Issue Management is ranked fourth and reused 6 times. The 2 topics of lessons learned with Testing Plan as well as Training Document are ranked fifth and reused 5 times each. The 2 topics of lessons learned with Stakeholder Analysis as well as Stakeholder Participation are ranked sixth and reused 4 times each. The 4 topics of lessons learned with Documentation Management, Integrated Schedule Management, Project Management Plan, we well as Quality Check Process are ranked seventh and reused 3 times each. Lastly, the 4 topics of lessons learned with Data Migration Plan, Go-live Checklist, Production Data Migration, as well as Production Support Plan are ranked eighth and reused 2 times each.

Consequently, 75 out of the total of 102 lessons learned identified among the 16 topics were found to be effectively applied or reused to analyze the findings and put together the recommendations for the corrective actions, as the results of the project reviews conducted in the succeeding projects carried out during the period used by this case study.

Table 16. Breakdown Summary of Reuse Frequency over 2 by Topic

#	Reference	Topic (Element)	# of Reuse	% Rank
1	Table 13	WBS	13	76	1
	Table 14	Requested Changes	10	59	2
3	Table 15	Risk Management Plan	8	47	3
4		Issue Management	6	35	4
5		Testing Plan	5	24	5
6		Training Document	5	29	5
7		Stakeholder Analysis	4	24	6
8		Stakeholder Participation	4	24	6
9		Documentation Management	3	18	7
10		Integrated Schedule Management	3	18	7
11		Project Management Plan	3	IN	7
12		Quality Check Process	3	IN	7
13		Data Migration Plan	2	12	8
14		Go-live Checklist	2	12	8
15		Production Data Migration	2	12	8
16		Production Support Plan	2	12	8

VI. Conclusion

As discussed in Section V, the results of the case study indicate that the use of lessons learned based on the past project review results was found to be effective in focusing on the specific areas projected for improvement during the processes of conducting the project document review and key stakeholder interviews, as well as putting together the practical recommendations for the findings to finalize the results of the project review for continuous improvement, which were to be formally presented and submitted to the customer as the results of the quality audit.

Список литературы Effective use of lessons learned to conduct the project review for ERP implementation

M. S. Chaves, et al., “A New Approach to Managing Lessons Learned in PMBOK Process Groups: The Ballistic 2.0 Model,” International Journal of Information Systems and Project Management, Vol. 4, No. 1, 2016, 27-45. DOI: 10.12821/ijispm040102
M. Hobday, “The Project-Based Organization: An Ideal Form for Managing Complex Products and Systems?” Research Policy, 29, 2000, 871-893.
P. M. Carrillo, “Lessons Learned Practices in the Engineering, Procurement and Construction Sector,” Engineering, Construction and Architectural Management, 2005, 12 (3), 236-250.
X. Ferrada, D. Nunez, A. Neyem, A. Serpell, M. Sepulveda, “A Lessons-Learned System for Construction Project Management: A Preliminary Application,” Procedia – Social and Behavioral Sciences, 226, 2016, 302-309. DOI: 10.1016/j.sbspro.2016.06.192
F. O. Omotayo, “Knowledge Management as an Important Tool in Organizational Management: A Review of Literature,” Library Philosophy and Practice (e-journal), 2015, 1238.
L. W. Walker, “Learning Lessons on Lessons Learned,” Paper presented at PMI® Global Congress 2008-North America, Denver, Co., 2008, Newton Square, PA: Project Management Institute.
G. Disterer, “Management of Project Knowledge and Experiences,” Journal of Knowledge Management, Volume 6, Number 5, 2002, 512-520. DOI: 10.1108/13673270210450450
P. Paranagamage, P. M. Carrillo, K. D. Ruikar, et al., “Lessons Learned Practices in the UK Construction Sector: Current Practice and Proposed Improvements,” Engineering Project Organization Journal, 2012, 2 (4), 216-230.
A. Taniguchi, M. Onosato, “Use of Project Management Information System to Initiate the Quality Gate Process for ERP Implementation,” International Journal of Information Technology and Computer Science, 2017, Vol. 9, No. 12, 1-10. DOI: 10.5815/ijitcs.2017.12.01
S. Berzisa, “Application of Project Management Information Systems in Efficiency Improvement of Quality Management System,” in Proceedings of the 10th International Scientific and Practical Conference on Environment. Technology. Resources, 2015, Volume III, 17-21. DOI: http://dx.doi.org/10.17770/etr2015vol3.173
E. Sundqvist, F. Backlund, D. Chroneer, “What Is Project Efficiency and Effectiveness?” Procedia – Social and Behavioral Sciences, 119, 2014, 278-287. DOI: 10.1016/j.sbspro.2014.03.032
H. A. Bilal, S. Amjad, M. Ilyas, “A Comparative Study of Global Software Development Tools Supporting Project Management Activities,” International Journal of Education and Management Engineering, 2017, 6, 32-39. DOI: 10.5815/ijeme.2017.06.04
S. F. Rowe, S. Sikes, “Lessons Learned: Sharing the Knowledge,” Paper presented at PMI® Global Congress 2006-EMEA, Madrid, Spain, May 2006, Newtown Square, PA: Project Management Institute.
S. F. Rowe, S. Sikes, “Lessons Learned: Taking It to the Next Level,” Paper presented at PMI® Global Congress 2006-North America, Seattle, WA, USA, October 2006.
S. F. Rowe, “Lessons Learned: Taking It to the Next Level,” Paper presented at PMI® Global Congress 2007-EMEA, Budapest, Hungary, 2007, Newtown Square, PA: Project Management Institute.
S. F. Rowe, “Lessons Learned: Applying Lessons Learned,” Paper presented at PMI® Global Congress 2008-EMEA, St. Julian’s, Malta, 2008, Newtown Square, PA: Project Management Institute.
H. R. Kerzner, Ph.D., Project Management - Best Practices: Achieving Global Excellence, Third Edition, John Wiley & Sons, Inc., 207-211, 2014.
M. R. J. Qureshi, A. M. Abdulkhalaq, “Increasing ERP Implementation Success Ratio by Focusing on Data Quality & User Participation,” International Journal of Information Engineering and Electronic Business, 2015, 3, 20-25. DOI: 10.5815/ijieeb.2015.03.03
M. Antero, J. Hedman, S. Henningsson, “Sourcing Strategies to Keep up with Competition: The Case of SAP,” International Journal of Information Systems and Project Management, Vol. 2, No. 4, 2014, 61-74. DOI: 10.12821/ijispm020403
M. Haddara, A. Elragal, “ERP Adoption Cost Factors Identification and Classification: A Study in SMEs,” International Journal of Information Systems and Project Management, Vol. 1, No. 2, 2013, 5-21. DOI: 10.12821/ijispm010201
S. S. Zabukovsek, S. Bobek, “TAM-Based External Factors Related to ERP Solutions Acceptance in Organizations,” International Journal of Information Systems and Project Management, Vol. 1, No. 4, 2013, 25-38. DOI: 10.12821/ijispm010402
S. Goyette, L. Cassivi, M. Courchesne, E. Elia, “The ERP Post-Implementation Stage: A Knowledge Transfer Challenge,” International Journal of Information Systems and Project Management, Vol. 3, No. 2, 2015, 5-19. DOI: 10.12821/ijispm030201
T. Munkelt, S. Volker, “ERP Systems: Aspects of Selection, Implementation and Sustainable Operations,” International Journal of Information Systems and Project Management, Vol. 1, No. 2, 2013, 25-39. DOI: 10.12821/ijispm010202
V. Hasheela-Mufeti, K. Smolander, “What Are the Requirements of a Successful ERP Implementation in SMEs? Special Focus on Southern Africa,” International Journal of Information Systems and Project Management, Vol. 5, No. 3, 2017, 5-20. DOI: 10.12821/ijispm050301
Atsushi Taniguchi, Masahiko Onosato, "Effect of Continuous Improvement on the Reporting Quality of Project Management Information System for Project Management Success", International Journal of Information Technology and Computer Science(IJITCS), Vol.10, No.1, pp.1-15, 2018. DOI: 10.5815/ijitcs.2018.01.01
Project Management Institute, A Guide to the Project Management Body of Knowledge (PMBOK® Guide)-Third Edition. PMI, 2004.
E. Teller, “Lessons Learned on Conducting the Strategic Planning Progress for a Nonprofit: What Went Right and What Could Be Improved,” Journal for Nonprofit Management, Vol. 15, 2012, 50-64.
S. Mclntyre, K. Dalkir, Perry Paul, I. C. Kitimbo, Utilizing Evidence-Based Lessons Learned for Enhanced Organizational Innovation and Change, IGI Global, 2015, 1-9.
R. Weber, D. W. Aha, I. Becerra-Fernandez, “Intelligent Lessons Learned Systems,” International Journal of Expert Systems Research & Applications, Vol. 20, No. 1, 2001, 17-34.
Project Management Institute, A Guide to the Project Management Body of Knowledge (PMBOK® Guide)-Sixth Edition. PMI, 2017-09-22.
M. S. Chaves, G. S. Veronese, “Proposal to Manage Lessons Learned in Projects: Web 2.0 Technologies to Promote Innovation,” International Journal of Innovation, Sao Paulo, Vol. 2, No. 1, Jan/Jun 2014, 1-17.
K. Jugdev, “Learning from Lessons Learned: Project Management Research Program,” American Journal of Economics and Business Administration, 4 (1): 13-22, 2012.
T. Kotnour, “A Learning Framework for Project Management,” Project Management Journal, June 1999, 30 (2), 32-38.
J. S. Busby, “An Assessment of Post-Project Reviews,” Project Management Journal, September 1999, 30 (3), 23-29.
T. Dingsoyr, “Postmortem Reviews: Purpose and Approaches in Software Engineering,” Information and Software Technology, 47 (2005), 293-303.
ISO, ISO 9001:2015 – Quality management systems – Requirements. Fifth edition, 2015-09-15.
A. M. Qazi, S. Shahzadi, M. Humayun, “A Comparative Study of Software Inspection Techniques for Quality Perspective,” International Journal of Modern Education and Computer Science, 2016, 10, 9-16. DOI: 10.5815/ijmecs.2016.10.02
S. Berzisa, “The Baseline Configuration of Project Management Information System,” Scientific Journal of Riga Technical University, Computer Science, 2009, Vol. 5, No. 40, 59-65.
M. Light, B. Rosser, S. Hayward, Realizing the benefits of projects and portfolio management, Gartner, Research ID: G00125673, January 4, 2005, 1-31.
L. Raymond, F. Bergeron, “Project Management Information Systems: An Empirical Study of Their Impact on Project Managers and Project Success,” International Journal of Project Management, 26 (2008), 213-220.
S. Berzisa, “Application of Knowledge and Best Practices in Configuration of Project Management Information Systems,” Ph.D. Thesis, Riga Technical University, Latvia, 2012.
S. Berzisa, “Project Management Knowledge Retrieval: Project Classification,” in Environment. Technology. Resources: Proceedings of 8th International Scientific and Practical Conference, June 20-22, 2011, Volume II, Vol. I, 33-39.
B. El-Sharef, K. S. El-Kilany, “Process Modeling and Analysis of a Quality Management System for Higher Education,” in Proceedings of the World Congress on Engineering 2011, Vol. I, WCE 2011, July 6-8, 2011, London, U. K.
L. Tsironis, A. Gentsos, V. Moustakis, “Empowerment the IDEF0 Modeling Language,” International Journal of Business and Management, May 2008, Vol. 3, No. 5, 109-118.

Еще

Статья научная