Stakeholders’ Program Outcomes Evaluation Before and During the Pandemic

Abstract

Stakeholders’ surveys are conducted periodically on the program outcomes evaluation of the business administration program of the Mongolian University of Science and Technology. This research compares survey results before (2017) and during the pandemic (2021) to determine the difference between the groups of stakeholders’ evaluation. The program outcomes evaluation differs across the stakeholder groups – alumni, students, employers, and parents. Overall, students tend to have higher judgment about their professional knowledge and skills and have lower judgment for their computer skills compared to other stakeholders. Employers’ program outcomes evaluation during the pandemic had no statistically significant difference compared to before the pandemic. In contrast, alumni’s evaluation of all program outcomes is statistically significant with a higher value during the pandemic. Students who studied predominantly online evaluated their “professional skills” and “professional practice” outcomes statistically significantly lower than those who had studied mainly in the classroom. These two outcomes are certainly more difficult to achieve in an insufficiently prepared and equipped online learning environment. Parents’ evaluation of “professional practice” and “professional traits and attitude” lowered statistically significantly during the pandemic.

The major factor negatively affecting students’ satisfaction during the pandemic is the difficulty concentrating on online studies due to a significant amount of distraction in the home environment. This can be attributed to the extended characteristics of the Mongolian family, where up to three generations live under the same roof. In addition, online learning in an environment lacking strict lock-down measures appears to foster student employment. On the positive side, the pandemic accelerated the long targeted digital shift of the education process of MUST. However, students, as well as faculty, were required to adapt to online learning conditions at abruptly faster rates. As the pandemic becomes the “The New Normal”, resilience is required from all stakeholders as well as the faculty and staff.

Background

Mongolian University of Science and Technology (MUST) was founded in 1959. In 1998, it pioneered introducing an academic credit hour system in the higher education system of Mongolia. In 2014 Computer Science and Management School of MUST was restructured and renamed as School of Business Administration and Humanities (SBH). Business administration programs at MUST were accredited by ACBSP in 2013 and reaffirmed by ACBSP in 2017. The following bachelors degree business administration programs at MUST are ACBSP accredited (ACBSP, 2021):

  1. Financial Management; 
  2. International Business Management;
  3. Small and Medium Business Management;
  4. Human Resource Management;
  5. Marketing Management;
  6. Information Systems Management.

The university business administration program has defined program outcomes in the following 4 focus areas: 

  1. Professional traits and attitude
  2. Professional knowledge
  3. Professional skills
  4. Professional practice

In addition to the main 4 outcomes, 2 additional outcomes, “foreign language proficiency” and “computer skills and information technology knowledge” are evaluated. Business administration students have the core curriculum in the 1st and 2nd year, and from 3rd-year the courses are specialized according to a chosen major field (Financial Management, International Business Management, Small and Medium Business Management, Human Resource Management, Marketing Management, or Information Systems Management).

Mongolia went into quarantine when the first imported COVID-19 case was confirmed on March10, 2020. All schools and universities were closed and 2019-2020 spring semester courses at MUST were delivered online. It should be noted that students didn’t had any online classes pre-pandemic and all classes were delivered through classroom teaching. Extensive training was organized in spring 2020 for the entire university faculty and staff. MUST has blended the traditional LMS platform with interactive Microsoft Office 365. The Mongolian University of Science and Technology has its own learning management system, Unimis for e-learning, but due to the massive access of all students and professors to the system, some weaknesses of the system were found. The Unimis was not able to handle all requests from teaching faculties therefore, professors heeded to use other software for teaching, such as Zoom, Microsoft Team, Moodle, among others [Uranchimeg Narantsatsral, 2021]. Teachers, as well as students, were all going through learning the online education process. 

The start of the new 2020-2021 academic year was back to normal as the country had no locally transmitted cases. But from November 11, 2021, when the first locally transmitted case was discovered, Mongolia went into total lockdown and MUST shifted back to online mode. Due to extensive vaccination efforts by the Mongolian government in the 2021-2022 academic year, students and faculty were able to combine both online and offline modes of education, depending on the occurrence of infections.

Data and methodology

Stakeholders’ surveys are conducted periodically on the program outcomes evaluation of the business administration program of the Mongolian University of Science and Technology. A stakeholder in an organization is (by definition) any group or individual who can affect or is affected by the achievement of the organization’s objectives (Freeman, 1984). Ensuring that the various stakeholders are both satisfied and receiving positive value should further improve the public opinion and societal-level outcomes of higher education (Hickman & Akdere, 2017). 

The stakeholder groups considered in this research are alumni, students, employers, and parents. 

In the 2017-2018 academic year, alumni and employer survey questionnaires were updated to allow program outcomes evaluations. Student satisfaction was estimated based on the course evaluation questionnaire taken at the end of each semester. The course evaluation questionnaire is more focused on the evaluation of the faculty and the classroom environment for the particular course. Alumni and employer surveys in the 2017-2018 academic year were collected via the Google Form platform, while students’ course evaluation and the parents survey were filled by respondents manually on paper forms. 

In the 2020-2021 academic year, the parents’ and students’ questionnaires were also updated to include questions asking to evaluate program outcomes. In the 2020-2021 academic year, all surveys were collected by Google Form. Evaluation data were collected using the Likert scale and converted to a 0-100 score. Likert, or ordinal variables with five or more categories, can often be used as a continuous variable without any harm to the analysis (Johnson & Creech, 1983).

Table 1. Surveys’ sample size


Stakeholder2017-20182020-2021
1Employers5152
2Alumni260167
3Students918*260
4Parents50103

Note: * students filled in multiple questionnaires for each course they had to take

The survey data from the 2017-2018 academic year represents the pre-pandemic situation, while data from the 2020-2021 academic year represents the situation during the pandemic. 

Survey data were processed using Microsoft Excel and analyzed using IBM SPSS 26 statistical software. Descriptive analysis of before and during the pandemic data was complemented by T-test and ANOVA analysis independent samples to determine the statistical evidence of a difference in program outcome evaluation.

Program outcomes evaluation by stakeholders before and during the pandemic

Employer’s evaluation of program outcomes

Before the pandemic the employers gave the highest scores for “Professional skills” and then for “Professional traits and attitude”. During the pandemic, they gave the highest scores for “Professional traits and attitude” and then “Computer and information technology skills” (Figure 1). Foreign language proficiency has the lowest score, which is an indication that significant improvement is needed in foreign language teaching. 

Figure 1. Employer’s evaluation of program outcomes before and during the pandemic

The independent sample test results showed that the evaluation scores difference before and during the pandemic is statistically insignificant for each outcome (Table 2).

Table 2. Independent samples test results for employer’s
evaluation of program outcomes before and during the pandemic

Levene’s Test for Equality of Variancest-test for Equality of Means

Outcome

Test type
FSig.tdfSig. (2-tailed)
Professional knowledgeEqual variances assumed0.1560.694-0.813950.419
Equal variances not assumed  -0.81294.2730.419
Professional skillsEqual variances assumed0.5100.4770.920960.360

Equal variances not assumed  0.91995.1520.360
Professional practiceEqual variances assumed0.0280.8660.031960.975
Equal variances not assumed  0.03195.2940.975
Professional traits and attitudeEqual variances assumed0.1870.666-0.438960.662
Equal variances not assumed  -0.43895.9570.662
Foreign language proficiencyEqual variances assumed8.3870.005-0.258910.797

Equal variances not assumed  -0.26488.5870.792
Computer and information technology skillsEqual variances assumed0.0850.771-0.646910.520

Alumni’s evaluation of program outcomes

Alumni members’ evaluation increased during the pandemic (2020-2021) compared to evaluation before the pandemic (2017-2018) for each outcome. Alumni scored the highest on “Professional traits and attitude”. The second highest scored outcome was “Computer and information technology skills”, which is, however, scored less by students. Compared to students, alumni evaluated “professional skills” over “professional knowledge”. Both alumni and employers’ evaluation for “foreign language proficiency” is the lowest (Figure 2).

Figure 2. Alumni’s evaluation of program outcomes before and during the pandemic

The independent sample test results showed that the evaluation scores before and during the pandemic are statistically significant for each outcome (Table 3).

Table 3. Independent samples test results for alumni’s evaluation of program outcomes before and during the pandemic

Levene’s Test for Equality of Variancest-test for Equality of Means
OutcomeTest typeFSig.tdfSig. (2-tailed)
Professional knowledgeEqual variances assumed1.902.169-2.847425.005***
Equal variances not assumed

-2.866361.708.004
Professional skills Equal variances assumed4.563.033-1.965425.050
Equal variances not assumed

-2.009379.305.045***
Professional practiceEqual variances assumed.317.574-3.103425.002***
Equal variances not assumed

-3.145369.786.002
Professional traits and attitude Equal variances assumed2.371.124-2.467425.014***
Equal variances not assumed

-2.577401.457.010
Foreign language proficiencyEqual variances assumed.076.783-2.547425.011***
Equal variances not assumed

-2.553356.617.011
Computer and information technology skillsEqual variances assumed5.175.023-2.205425.028
Equal variances not assumed

-2.245375.271.025***

Figure 3. Alumni overall satisfaction by graduation period, 2021

The alumni satisfaction is the highest for 2000-2004 graduates and the lowest for 2010-2014 graduates (the period of restructuring and renaming of the school). Satisfaction fell again for 2020 and later graduates during the pandemic (Figure 3). 

Students’ evaluation of program outcomes

The number of business administration program students who participated in the survey was 260 in 2021. The 2021 questionnaire was modified from 2017 to measure program outcomes. The previous questionnaire aimed to evaluate a particular course and teacher rather than the program. Therefore, we can not compare the survey results of 2017 with the ones from 2021. Fortunately, data from the 2021 survey allows the comparison of students’ evaluation who had studied before the pandemic in a regular classroom environment with those of the students who had studied predominantly online during the pandemic. We assume that students who attended the school for 4 and more years (40% of respondents) have enough experience from before the pandemic with classroom education. On the other side, we assume that 1st, 2nd, and 3rd year students (60% of respondents), who predominantly have been learning online by using Teams and LMS platforms, are lacking experience in traditional classroom education.

Figure 4. Students’ years of study at SBH, 2021

Figure 5. Students’ overall satisfaction level, 2021

Students’ average satisfaction level differs according to their years of study. The 4th year students who took more professional courses compared with 3rd year students have higher satisfaction levels (Figure 5). Students’ evaluation of program outcomes indicated that foreign language proficiency and computer and information technology skills have the lowest scores and require measures to improve. Among the four main outcomes, the professional practice outcome has the lowest score and probably suffered most during online classes (Figure 6).

Comparison of two groups of students revealed that the group 1 students, the majority of which are 3rd-year students who mainly studied online, had a lower evaluation of the program outcomes, except for “professional traits and attitude” (76.2%). The outcome targeted to be achieved mainly in freshmen year, during which they were able to study in classrooms just before the pandemic. Students in group 2 gave “professional skills” and “professional practice” higher scores than for other outcomes.

Figure 6. Students’ evaluation of program outcomes by years of study

The independent sample test results showed that the difference in evaluation between the two groups of students for “professional skills” and “professional practice” outcomes are statistically significant with the lowest scores during the pandemic learning. These two outcomes are certainly more difficult to achieve in a poorly-equipped online learning environment (Table 4). One-third of students do not have access to a laptop or computer so they had to use solely mobile phones during live sessions.

Table 4. Independent samples test results for students’ program outcomes evaluation before and during the pandemic

Levene’s Test for Equality of Variancest-test for Equality of Means
Program outcomeTest typeFSig.tdfSig. (2-tailed)
Professional traits and attitudeEqual variances assumed1.131.2891.286253.200
Equal variances not assumed

1.277213.564.203
Professional knowledgeEqual variances assumed3.209.074-1.896249.059
Equal variances not assumed

-1.934233.746.054
Professional skillsEqual variances assumed3.411.066-2.303231.022***
Equal variances not assumed

-2.341222.513.020
Professional practiceEqual variances assumed.228.634-2.654219.009***
Equal variances not assumed

-2.652207.634.009
Foreign language proficiencyEqual variances assumed2.317.129-1.014249.311
Equal variances not assumed

-.993202.441.322
Computer and information technology skillsEqual variances assumed.122.727-.608247.543
Equal variances not assumed

-.613223.278.540

Note: Group 1 is consists of students who mainly studied online, Group 2 consists of students who mainly studied in classrooms

The survey results indicated that the online learning experience overwhelmed the students and caught them unprepared in terms of device availability and maintenance of a distraction-free environment. The majority of students (64.9%) described their online class experience with the answer choice “There is a lot of distraction at home or work, so I can’t concentrate on my online studies”. For 43%, the online learning required more time and efforts to learn, but 33.8% appraised that the quality of online lessons improved compared to the previous semester (Table 5). 

Table 5. Students’ description of the online classes (multiple choice question)

Description of the situationCount%
There is a lot of distraction at home or work, so I can’t concentrate on my online studies24064.9
Online learning requires more time and efforts 15943.0
The quality of online lessons is improving compared to the previous semester12533.8
After the online lesson session, I  listen to the recording of the lesson left in the Teams12533.8
I do not have a laptop or computer available for online classes, therefore usually use a mobile phone11531.1
Compared to the previous semester, I am learning and getting used to online learning11531.1
When I miss the lesson, I watch/listen to the recording left in the Teams10227.6
I study only by the teacher’s lectures and do not use any additional materials or textbooks10127.3
Studying at home have some advantages, I am in a comfortable home environment and I can focus on my studies8623.2
I only attend live classes during online sessions and don’t listen to recordings again3810.3
Compared to normal conditions, less time and effort are spent on the learning and less strain on me318.4
In addition to teacher lectures, I use additional materials, books, and textbooks for my studies225.9
There is nothing much different from the normal situation before the pandemic154.1

Interestingly, with partial lockdown measures in Mongolia and universities being closed and shifting to online learning, a majority of businesses stayed open. The student employment appears to have increased, as 79% of students indicated that they have either part-time or full-time jobs or had occasional work.

Parent’s evaluation of program outcomes

The questionnaire was also changed in 2021 to measure program outcomes. The questionnaire used in 2017 was aimed to evaluate teachers and the educational environment. Therefore, we can not compare the survey results. Fortunately, data from the 2021 survey allows the comparison of parents’ evaluation whose children had studied before the pandemic in a regular classroom environment with those whose children had studied predominantly online during the pandemic. We assume that students who attended the school for 4 or more years (62% of respondents) have enough experience before the pandemic with classroom education. On the other side, we assume that 1st, 2nd, and 3rd-year students (38% of respondents), who predominantly have been learning online by using Teams and LMS platforms, are lacking experience in traditional classroom education (Figure 7).

Figure 7. Students’ years of study according to their parents

Parents gave the highest scores to professional traits and attitudes and the lowest scores to professional practice. Foreign language proficiency was evaluated with the second lowest scores (Figure 8).

Figure 8. Parent’s evaluation of program outcomes during the pandemic

Independent sample test results indicate that evaluation differences before and during the pandemic of “professional practice” and “professional traits and attitude” are statistically significant.

Table 6. Independent samples test results for parents’ evaluation before and during the pandemic

Levene’s Test for Equality of Variancest-test for Equality of Means
Program outcomeTest typeFSig.tdfSig. (2-tailed)
Professional knowledge Equal variances assumed0.2430.623-1.099960.274
Equal variances not assumed  -1.15390.2570.252
 Professional skillsEqual variances assumed0.8450.360-1.793910.076
Equal variances not assumed  -1.77073.7890.081
 Professional practiceEqual variances assumed2.4220.123-2.462970.016***
Equal variances not assumed  -2.34666.7000.022
Professional traits and attitude Equal variances assumed1.1510.286-2.167960.033***
Equal variances not assumed  -2.24890.7590.027
Foreign language proficiency Equal variances assumed0.7830.379-0.466940.643
Equal variances not assumed  -0.47285.6940.638
Computer and information technology skills Equal variances assumed1.3030.257-1.126940.263
Equal variances not assumed  -1.17692.0210.242

Differences in evaluation between stakeholders

Before the pandemic in 2017, the statistically significant difference between stakeholders is observed for evaluation of “professional practice”, “professional traits and attitude” and “foreign language skills”. There is no difference between stakeholders’ evaluation of “professional knowledge”, “professional skills”, and “computer knowledge and information technology skills” (Table 7). 

Table 7. Comparing evaluation means between different groups of stakeholders, 2017

ANOVA

Sum of SquaresdfMean SquareFSig.
KnowledgeBetween Groups2.2233.7411.0240.381
Within Groups337.769467.723

Total339.992470


Professional skillsBetween Groups2.6303.8771.1130.343
Within Groups364.556463.787

Total367.186466


PracticeBetween Groups26.49238.8319.6520.000***
Within Groups424.499464.915

Total450.991467


Traits and attitudesBetween Groups9.15233.0513.3210.020***
Within Groups423.395461.918

Total432.546464


Foreign language proficiencyBetween Groups16.82635.6095.5100.001***
Within Groups467.2214591.018

Total484.048462


Computer knowledge and information technology skillsBetween Groups5.92931.9761.9690.118
Within Groups459.7184581.004

Total465.647461


During the pandemic in 2021, the statistically significant difference between stakeholders is observed for the evaluation of “traits and attitudes”, “computer knowledge and information technology skills”. For other program outcomes, there is no difference between stakeholders’ evaluations (Table 8). 

Table 8. Comparing evaluation means between different groups of stakeholders, 2021

ANOVA

Sum of SquaresdfMean SquareFSig.
KnowledgeBetween Groups2.6123.8711.2810.280
Within Groups383.456564.680

Total386.069567


Professional skillsBetween Groups4.67831.5592.1190.097
Within Groups409.882557.736

Total414.560560


PracticeBetween Groups1.3583.453.4980.684
Within Groups494.777545.908

Total496.135548


Traits and attitudeBetween Groups37.880312.62715.6720.000***
Within Groups428.625532.806

Total466.506535


Foreign language proficiencyBetween Groups1.4353.478.4870.691
Within Groups549.884560.982

Total551.319563


Computer knowledge and information technology skillsBetween Groups31.831310.61011.3390.000***
Within Groups522.114558.936

Total553.945561


Conclusion

Different stakeholders evaluate the achievement of program outcomes differently. Students tend overestimate their knowledge and skills and underestimate their computer skills. Professional skills and practice evaluation scores dropped during the pandemic. This is certainly an issue to pay attention to and take efforts to improve. Within program outcomes, the lowest evaluation of all stakeholders was for foreign language proficiency. Faculty and administration of MUST should review the foreign language courses and undertake serious measures to improve foreign language education in the business administration program.


Before the pandemic, 2017

During the pandemic, 2021
#EmployersAlumniStudentsParentsEmployersAlumniStudentsParents
1Skills (76.2)Traits (79.6)Knowledge (77.4)Traits (84.4)Traits (76)Traits (84)Traits (76.2)Traits (76.9)
2Traits (74.6) Computer (74.8)Skills (76.6)Knowledge (76.7)Computer (75.6)Computer (79.2)Knowledge (73.2)Knowledge (73.2)
3Computer (74.4)Skills (73.8)Practice (76)Skills (75)Knowledge (73.8)Knowledge (77.8)Skills (70.8)Computer (70.3)
4Practice (72)Knowledge (73)Traits (73.4)Practice (74.8)Skills (73.2)Skills (77.4)Practice (69)Skills (68.6)
5Knowledge (71.2)Practice (65.8)Computer (69.2)Computer (74.4)Practice (72.4)Practice (71.8)Computer (67.6)Language (64.6)
6Language (61.8)Language (57.6)Language (65.4)Language (66.7)Language (62.8)Language (62.8)Language (63)Practice (64.2)

Comparison of stakeholders’ evaluation before and during the pandemic through the independent samples test results revealed the following differences (Table 10):

  1. Employers’ program outcomes evaluation during the pandemic had no statistically significant difference compared to before the pandemic.
  2. In contrast, alumni’s evaluation of all program outcomes is statistically significant with a higher value during the pandemic. However, the satisfaction level of graduates during the pandemic period is lower compared to the satisfaction level of those who graduated before the pandemic.
  3. Students who studied predominantly online evaluated their “professional skills” and “professional practice” outcomes statistically significantly lower than those who had studied mainly in the classroom. These two outcomes are certainly more difficult to achieve in a poorly equipped online learning environment.
  4. Parents’ evaluation of “professional practice” and “professional traits and attitude” lowered statistically significantly during the pandemic.

Table 10. Differences in stakeholders’ evaluation before and during the pandemic

#Program outcomesStakeholder groups




EmployersAlumniStudentsParents
1Professional knowledge
(+)***

2Professional skills
(+)***(-)***
3Professional practice
(+)***(-)***(-)***
4Professional traits and attitude
(+)***
(-)***
5Foreign language proficiency
(+)***

6Computer and information technology skills
(+)***

Before the pandemic in 2017, the statistically significant difference between stakeholders is observed in the evaluation of “professional practice”, “professional traits and attitude” and “foreign language skills”. During the pandemic in 2021, the statistically significant difference between stakeholders is observed in the evaluation of “traits and attitude”, “computer knowledge and information technology skills”. For other program outcomes, there is no difference between stakeholders’ evaluations.

Students experienced some difficulties with the shift to online learning during the pandemic. The main difficulties experienced by students during online learning are as follows:

  • Inability to fully concentrate on the learning process during online sessions due to distractions at home or in their work environment
  • Online learning requires more effort and time
  • Lack of online learning devices such as notebooks and internet connection in rural areas
  • On the positive side, students also acknowledged the following developments:
  • For some students, online learning in the home environment is more comfortable
  • One-third of students noticed that the quality of online lessons improved compared to the previous semester
  • One-third of students listened to the recording of the lesson left in Teams after the online lesson session

The inability to fully concentrate on the learning process during online sessions due to distractions at home or in the work environment partially can be attributed to the extended characteristic of the Mongolian family, where up to three generations live under the same roof. In addition, partial lockdown measures in Mongolia, with universities being closed and shifting to online learning, while the majority of businesses stayed open, led to an increase in student employment, as 79% of students indicated that they have either part-time or full-time jobs or had occasional work.

References

  • ACBSP. (2021, 12 17). Mongolian University of Science and Technology. Retrieved from https://acbspsearch.org/Home/Details?instId=Inst3838
  • Freeman, E. R. (1984). Strategic Management: A Stakeholder Approach. Pitman Publishing Inc.
  • Hickman, L., & Akdere, M. (2017). Stakeholder Theory: Implications for Total Quality Management in Higher Education. International conference on lean six sigma for higher education. Purdue University.
  • Johnson, D. R., & Creech, J. C. (1983). Ordinal measures in multiple indicator models: A simulation study of categorization error. American Sociological Review, 48, 398-407.
  • Uranchimeg, T., & Narantsatsral, D. (2021). E-Learning Evaluation Based on SURE Model: Case of Mongolian University of Science and Technology. In Communications in Computer and Information Science book series (pp. 520-532).