E-services evaluation criteria : the case of Azerbaijan

The development of evaluation and effective public administration mechanisms are crucial for providing effective e-government services. The article explores the international expertise in the field of the e-service evaluation process. The organization and provision of e-services in government entities, as well as the evaluation regarding the organization and functioning of information systems, are carried out by the “ASAN service” system in Azerbaijan. The evaluation of the organization and functioning of e-services is conducted by considering three determined criteria: the level of digitalization and relevance, the level of information openness and accessibility, and the level of convenience of use. The evaluation process of e-services allows for improving the level of services and the development of feedback mechanisms; currently, much attention is attributed to the development of citizen-centered services.


I. Introduction
Achievements in the field of transformation of public administration, the building of a citizen-centered e-government, the provision of e-services, as well as the development of effective administrative mechanisms with the application of ICT in most developed and developing countries, are highly appreciated.The modern economy is characterized by two tendencies comprised in the concept of e-services, such as the shift from goods to services and the rapid development of the information economy and electronic systems (Ostašius and Petravičiūtė 2010).The concept of e-services refers to the organization of ser-vices across various fields in electronic format organized with the application of ICT without any loss of time and transparency.
A growing demand of users for services boosts the development of e-services in order to respond more effectively to the users' needs.E-services must be simple and easily accessible for a wider audience of users; in other words, e-services must be simple and comprehensive as much as possible.In this regard, the evaluation of electronic services is one of the most important issues.By experience, usually, common characteristic problems pertain to such systems, and from this point of view, the development of projects specific to the field of e-services are of no importance.It can be said that it is more reasonable to agree and to cooperate in order to find common solutions to be applied by all interested parties.In this regard, the cooperation and the application of a common e-services model allow for finding more efficient solutions by considering the competencies of the parties that provide the services (Lindgren and Jansson 2013).
During the recent periods of the ICT's rapid development, purposeful initiatives were being carried out toward developing egovernment platforms and the organization of various e-services in different parts of the world.An e-government project has already been adopted in Azerbaijan; a national strategy has been developed and carried out within the framework of the Electronic Azerbaijan Program, which was started in 2003 (E-Azerbaijan 2003(E-Azerbaijan -2012)).The main purpose of the project was the provision of service quality, timeliness, transparency, efficiency, accessibility for citizens based on the "one window" principle in the government and other governmental institutions with the application of the continuously developing ICT.
Nowadays, e-services are developed with the application of new technologies, innovation, standards and learning from the best practices.The evaluations of eservices were conducted with the purpose of contributing to a more qualitative and effective development of e-services and to increase the efficiency of the e-government platform.A rapid growth of the number of new e-services entails their evaluations and multilevel measurements.E-service evaluation models make it possible to measure e-services and their systems in different ways and dimensions.This article explores an international practice regarding the evaluation process of e-services and the experience of Azerbaijan.

II. International practice on the evaluation of e-services
Achievements in the implementation of initiatives related to e-government have turned e-government into a world phenomenon.Governments adopt various strategies and state programs in order to implement an egovernment development policy.Notwithstanding all these efforts, the development of ICT and new services, the requirements posed to the provision of citizen-oriented services etc. are accompanied with challenges and problems at local and regional levels.Some of these are the requirements posed to e-services that change over time, and this necessitates the development of egovernment strategies and models.
The development of e-government models can be divided into 3 arbitrary stages: starting from the Gov.1.0model in the world in 1995, the application of Gov. 2.0 model during 2005-2010 and the application of Gov. 3.0 model since 2015 (E-Gov. concepts. 2008;Meijer et al. 2012;Government 3.0. 2016).
At present, the Gov 2.0 platform is being utilized in order to provide more citizen-oriented e-government services.Across countries, Web 2.0 technologies are used in order to implement and provide government-and citizen-oriented services.It should be noted regarding the Gov 3.0 model that it is proposed as a concept with an incentive of customizing the services in accordance with the desires and needs of each citizen; in other words, it is a Webbased semantic government incentive providing smart services.E-government models differ from each other in accordance with the various characteristics of their stages of development (E-Gov. concepts. 2008;Government 3.0. 2016).
As one of the leading countries -according to the government development index developed by United Nations (UN) -South Korea has adopted an e-government 2020 strategy and aims to provide citizen-centered e-government services with the application of advanced technologies (E-Government Survey 2014;2016;Government 3.0. 2016).According to the experience of South Korea, the Gov 3.0 concept is a citizen-centered government service and its aim is to make human lives more comfortable, create new job positions with open government data and to promote cooperation and communication among governmental entities.By making the government more service-oriented, competent and transparent, Gov 3.0 is oriented toward the improvement of the satisfaction level of citizens (Government 3.0.2016).Ray and Rao (2005) have proposed a method for an accurate evaluation of the efficiency level of services provided as a result of the implementation of the e-government project.The authors proposed to employ the analytical hierarchy approach as a tool for evaluating the provided e-services.
Ostašius and Petravičiūtė (2010) attribute much attention mainly to the modelling of e-government services in their study.This approach is based on a reference model of service-oriented architecture.The model is oriented toward stimulating cooperation among entities participating in the process of the provision of e-services and toward the improvement of the efficiency in their functioning driven by the development of joint services.The proposed model has been tested in the e-services of registering transport vehicles and can be used for developing other services in e-government.The proposed e-service model makes it possible to evaluate the maturity and completeness of an e-service system and compare it with similar e-services (Ostašius and Petravičiūtė 2010).
In their study, Lindgren and Jansson (2013) discuss the definitions used for studying e-service issues related to the employment and development of e-government services within the context of an e-government.The conceptual foundations of e-services are explored, and e-government services are studied in three aspects: service, electronic and public (Lindgren and Jansson 2013).Zaidi et al. (2013) explore various aspects of the provision of e-government services, and an approach is proposed for evaluating the trustworthiness and the efficiency of e-services provided from the citizens' points of view.Existing models for evaluating efficiency, such as SERVQAUAL, E-S-Qual, D&M and others, have been systematically analyzed, and a new conceptual model was proposed (Zaidi et al. 2013).
Butt (2014) reviews the evaluation of a result-oriented e-government by considering citizen interests.Butt's study presents the evaluation of an e-government website as an evaluation of the e-government itself.The literature review states that such studies are usually conducted in order to evaluate the functioning of governmental institutions.The evaluation process of an e-government is carried out in three stages in this research.The first stage discusses the indicators for e-government evaluation.The second stage considers the collection of citizen opinions on each indicator for conducting an experiment.The approach is experimentally tested in a selected government entity at the third stage.As this approach is outcome-oriented, the prepared reports, which are based on the study results, were sent to the interested parties.
The results of the study have shown that the evaluation conducted in a government institution allows for analyzing and critically evaluating the process at a particular level in a governmental institution (Butt 2014).
Hasan ( 2015) has carried out a literature analysis of studies conducted in the e-government sphere.The analysis of studies conducted in a regional context, the perspective of the studies, the method of data analysis and the analysis of the research approaches were carried out from the perspective of research improvement.The outcome of this study is important in terms of exploring e-government services and determining future research directions.The results obtained in the study cover research works related to e-government services, service development, evaluation and mutual communication.The mentioned studies considered this topic from the points of view of advanced and developing countries; empirical data analysis methods have been used (Hasan 2015).
Taherdoost et al. ( 2015) summarized e-services and comprehensively explored their conceptual perspectives.The scientific-theoretical foundations of e-services are studied in terms of conducting future empirical studies, and the e-service users are also divided into categories.
Asad, Khazaei et al. ( 2016) investigate the conceptual foundations of the importance of integration of provided services in implementing e-government; the scholars also conduct a literature analysis.The integration of e-services, together with the respective definitions, problems, analytical and conceptual approaches, are explored.As a result, a model is proposed that enables us to specify the importance of service integration in implementing an e-government (Asad, Khazaei et al. 2016).Malik et al.
(2016) consider the evaluation of citizen satisfaction during the use of e-government services.Citizen satisfaction plays a crucial and decisive role for the systematic use of e-government services and influences the rates of failure or success of e-government projects.The results obtained in this research allow for an evaluation of the level of citizen satisfaction in Pakistan (Malik et al. 2016).
Over time, the development of egovernment models is carried out based on proposals and social innovations; this is done for e-services to be regularly evaluated by citizens and governments and that the provided services can be continuously improved.This process results in the generation of new e-government models.
Considering the European experience, the evaluation of e-services in the UN model is carried out based on a 4-stage (emerging, enhanced, transactional, connected) development model in accordance with the online service index (E-Government Survey 2014; 2016).It is worth mentioning that, although e-services are being evaluated based on various criteria in international practice, and that various local and regional approaches are present, a set of unified evaluation criteria does not exist.

III. Criteria for evaluating e-services in Azerbaijan
Purposeful works are being carried out aiming at the development of e-services in Azerbaijan.Particular requirements are posed to the implementation of each e-service.These requirements can be classified as follows: 1.The facilitation of the use of e-servicesthe provision of the name of a provided service by relevance, an accurate list of documents required for providing the service, the electronic forms of the corresponding documents (application forms and others) in an e-service section of an official internet website and the free use of this section by people without any obstacles; 2. The facilitation of document submission -the provision of the acceptance of scanned copies of applications and required documents in the digital form, the notification of acceptance to a person sending the documents and the review of those documents in accordance with the procedure and time specified by the legislation; 3. User informing -the facilitation of obtaining information regarding the procedure of e-service provision for each user and the presentation of results; 4. The facilication of integration -building communication for mutual information exchanges with other governmental entities regarding for the purpose of eservices provision; 5. Security provision -the adoption of necessary technical security measures in order to make the operations conducted via the e-service section more secure and protect personal information; 6.The presence of administrative regulation -information resources must be developed by a governmental institution that will provide the service, as well as the performance and actions, of all institutions participating in the service delivering.All such actions must be described accurately and publicly open; 7. Payments for e-services -the provision of payment opportunities in the digital format using a real-time regime.8.The use of an e-signature -the authentication of e-service users, the submission of applications and responses and the signing of documents sent in a digital format to central executive authorities.
The organization and provision of eservices in government entities, as well as the procedure of evaluating the organization and functioning of information systems, are carried out on an annual basis based on the principles of lawfulness, objectiveness, transparency and professionalism by the "ASAN service" (State Agency for Public Services).The main purpose of the evaluation is as follows: 1.The organization and provision of e-services as well as the study of the current situation and conducted works regarding the organization and functioning of information systems; 2. The determination of information sources necessary for organizing and providing e-services; 3. Supervision overseeing that the implementation of information systems is carried out in accordance with approved projects, technical norms, standards and requirements; 4. The provision of a mutual integration of information systems across governmental entities; 5. Supervision overseeing the acceleration of the connection process to developed infrastructure for the purpose of information exchange between governmental entities and information systems; 6.The investigation of opportunities of simplifying the procedure of providing state services (with regard to required documents and periods); 7. The determination of information systems that are necessary for the implementation of state services; 8.The acceleration of the process of organizing e-services; 9.The organization and provision of e-services; methodical and operational assistance to governmental entities on the organization and functioning of information systems; 10.The provision of transparency in the operation of governmental entities; 11.The exploration of opportunities for providing modern and flexible administration in this field (State Agency for Public Service).
The evaluation of the level of organization and provision of e-services is carried out by the State Agency in accordance with 3 criteria and sub-criteria aimed at the following directions (State Agency for Public Service): ▫ The level of digitalization and relevance; ▫ The level of information openness and accessibility; ▫ The level of convenience of use.

The level of digitalization and connectedness criterion consists of five sub-criteria
and is evaluated with 0, 1 and 2 points in terms of digitalization, integration of electronic information resources between other governmental entities, the use of digital information systems and other aspects.The sub-criteria of the indicated criteria are as follows: 1.Using digital information systems during the organization and provision of eservices (the acceptance of applications in regard to e-services, processing in governmental entities and the delivery of results); 2. Using internal digital information resources (databases, information-search systems, registries and other information resources) during the provision of e-services; 3. Using digital information resources (databases, information-search systems, registries and other information resources) of other governmental institutions regarding the provision of e-services; 4. The provision of the use of e-signatures while providing e-services; 5.The presentation of the results of eservice provision.
The level of information openness and accessibility criterion consists of the following four sub-criteria: 1.The presence of an administrative regulation of an e-service; 2. The provision of service included in the section of state services in an official internet website of the governmental entity providing the particular service; 3. The presentation of comprehensive, up-to-date and accurate information regarding a service in the "E-registry of State Services"; 4.An availability of opportunities for obtaining comprehensive, up-to-date and accurate information in the official website of a governmental entity regarding the rules of using a particular e-service.
The level of convenience of use criterion consists of the following seven subcriteria, and evaluation is carried out in accordance with corresponding sub-criteria: 1.The opportunity of accepting information required for the use of a service in the digital format; 2. The availability of sample documents (application forms and others) in the digital format in the respectibe section of a website for the state services, and the free use of this section for people without any obstacles; 3. The notification of acceptance to a person who had submitted the documents in the digital format (who had sent the request for informative services); 4. The facilitation of opportunities for obtaining digital information regarding the procedure of e-service provision for users; 5.The facilitation of digital payments in a real-time regime for the provision of an e-service; 6.The possibility of reviewing the documents submitted in the digital format in accordance with the procedure and periods determined by the legislation; 7. The provision of reviewing complaints and suggestions regarding services received via the "ASAN appeal" system (ASAN service).
It must be noted that the evaluation of e-services is carried out in three stages -the submission of requests, the implementation of spot reviews of the corresponding government institutions providing he e-services and the testing of services using the information systems (test use of service).
Figure 1 presents statistical indicators regarding the organization and provision of services by "ASAN service" in 2013-2016: According to the each specified subcriteria and stages, electronic services were assessed with 0, 1 and 2 points by experts, and the total points on percentages are correspondingly evaluated as follows: ▫ 50% and lower -non-satisfactory; ▫ 51% to 70% -satisfactory; ▫ 71% to 90% -good; ▫ 91% 100% -excellent.E-services provided by the corresponding entities of executive authorities have been evaluated in accordance with relevant indicators and 79 services have been added to the list of excellent services.
It must be noted that the evaluation was carried out in accordance with the rules prepared by the State Agency for Public Service.Various institutions, community representatives and experts have been engaged in the preparation of the rules.These rules, which are annually updated, have been prepared in line with recommendations of the European Union, UN and international practices.
As a result of the evaluation of e-services provided by the State Agency of Public Service, the level of the digitalization of services, integration and relevance, convenience of use have been improved, user informing has been organized in a more comprehensive way, the best practice of various institutions has been shared with other institutions.Moreover, specific proposals are submitted to government institutions, oriented toward the improvement of the level of e-services that are provided by those institutions (State Agency for Public Service).
According to the results of the report by the UN (E-Government Survey 2016) on the development of an e-government, implemented for the purpose of measuring the efficiency of e-government in delivering the main economic and social services to citizens, Azerbaijan has advanced 12 steps in the e-government development index and ranks 56 th among 193 countries in 2016 in comparison with 2014.According to the regional ranking, Azerbaijan occupies the 11 th place in the Asian region and the 4 th place among CIS countries after Kazakhstan, Russia and Belarus.Azerbaijan has gained 0.6780 points for its e-participation index in 2016 and occupied the 47 th place among 193 countries (E-Government Survey 2016).These results reflect that the performance of the State Agency for Public Service is highly appreciated, and at present, the international experience in the sphere of providing citizen-oriented services is closely studied for the purpose of improving e-services.

Conclusion
At present, ICT facilities are broadly used in the provision of state services and the implementation of public functions.In this regard, if it is possible to say so, the application of ICT takes the lead in evaluating e-service systems.The evaluation of e-services is practically related to the application and use of the system.There exists a need for a systematic approach for analyzing, monitoring and measuring the effectiveness of e-government services.Therefore, a systematic approach plays an important role in the evaluation of the efficiency of e-services and, at present, much attention is attributed to the study of citizen-centered services.
This article explores the existing international practices in the field of e-services evaluation.The experience of Azerbaijan in the sphere of the provision of e-services is investigated.The article explores the criteria for measuring e-services provided by the "ASAN service" as well as the proposed approaches to the evaluation of the level of organization and the provision of e-services in government institutions.Following the evaluation of e-services, the level of the digitalization of services, the level of integration and relevance, as well as the convenience of the use of services, have been improved; specific proposals have been received regarding the level of e-services and their improvement.
The importance of citizens' opinions in the process of the evaluation of e-government services cannot be denied, as citizens are the main interested party in using eservices.In this regard, the development of feedback mechanisms between citizens and state institutions bears great importance, and the use of feedback indicators during the evaluation of the efficiency of e-services has greater potential for facilitating more effective decision-making and the development of effective solutions.