Customer Experience Index for Evaluating Network Quality for Cloud Applications

Introduction This document introduces a unified Customer Experience Index (CEI) designed to assist cloud vendors in assessing the network quality that mirrors the customer experience of cloud applications when accessed via the public network. The CEI, once quantified, empowers cloud vendors to proactively enhance network services, aiding in network planning and construction. Furthermore, it enables cloud customers to distinguish the service quality of various cloud vendors, allowing them to select cost-effective services tailored to their applications. Cloud vendors and cloud enterprises focus on different network indicators (Key Performance Index) used to anticipate the quality of customer experience regarding various applications(e.g., gaming, audio and video, online stores). However, KPIs only provide implicit information and cannot directly reflect the customers' perceived experience. Moreover, there is no unified evaluation method of customer experience based on common network KPIs in the industry. On the other hand, it is difficult for cloud vendors to directly access application-level Key Quality Index (KQI) data though it may explicitly imply customer experience. As the number of enterprises who deploy the service in the cloud gradually increases, there is growing demand for deriving authentic customer experience from basic network metrics to facilitate network optimizations. A significant gap persists between network KPIs and customer experience. The primary network KPIs accessible to most cloud vendors—network latency, packet-loss rate, and jitter—encompass three categories. Considering multiple dimensions of network quality proves beneficial for end-users. Customers' demands for experience quality vary across different cloud services and are linked to specific KPIs. For instance, those accessing real-time interactive games prioritize network latency; those utilizing video-on-demand services are more concerned with packet-loss rate than latency; and those engaging with cloud storage services consider both latency and packet-loss rate. No single KPI can provide an accurate reflection of the experience for diverse services. Both cloud vendors and customers seek unified evaluation standards for experience quality when accessing cloud services. This document accounts for a range of key network-observable indicators, offering a unified, objective, and comprehensive CEI to help enterprises evaluate customer experience through measurable network KPIs in a reasonable and fair manner. Predominantly based on three network KPIs—network latency, packet-loss rate, and jitter—the CEI aims to thoroughly assess network quality. The allocation of weights to these KPIs within the CEI can be customized to suit different application scenarios.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology CEI: Customer Experience Index KPI: Key Performance Indicator KQI: Key Quality Indicator

Motivation Cloud service providers aim to precisely evaluate the network quality, crucial to the customer experience of their cloud services, and implement targeted improvements to their network infrastructure. Similarly, cloud customers seek a unified and fair scoring standard to guide their selection of superior cloud services. But they currently face some challenges:

Cloud service providers often find it challenging to gather direct feedback on customer experiences.
Although obtaining network monitoring data, such as Key Performance Indicators (KPIs), is relatively straightforward in real-time, a clear correlation between the customer experience's Key Quality Indicators (KQI) and network KPIs remains elusive.
Network quality is primarily assessed using three KPIs: network latency, packet-loss rate, and jitter. Cloud service providers support a wide array of applications, including those sensitive to latency (like gaming applications) and packet loss (such as audio and video applications). This diversity necessitates different KPIs to gauge the experience quality accurately, leading to a fragmented evaluation approach.

Customer Experience Index This document introduces the Customer Experience Index (CEI), a measure reflecting customer experience with cloud services. It enables cloud service providers to swiftly evaluate their service quality through a synthesis of key network metrics.

Observation Customer experience often exhibits distinct zones—sensitive and smooth—based on their response to changes in specific indicators. For instance, in scenarios sensitive to latency, such as cloud gaming, customer satisfaction remains high within an acceptable latency range (smooth zone). However, exceeding a certain latency threshold leads to a sharp decline in experience (sensitive zone). Accordingly, the CEI employs an S-Curve for its calculation, a method prevalent in biostatistics and sociology for modeling ecosystems and urbanization trends. The S-Curve, particularly through the Sigmoid function, effectively maps values to a (0,1) interval, delineating two smooth zones and a sensitive zone, mirroring the nuanced nature of customer experience.

S-curve Example ]]> Therefore, this document designs the following formula to evaluate customer experience for each network KPI: f(x) = (1+e^b)/(1+e^(a*x+b))

x is the value measured by network KPI, a and b are tunable parameters, and f(x) represents S-curve for certain KPI.
Parameter a represents the overall slope of the curve, mainly affecting the range of the central sensitive area.
Parameter b represents the offset and scaling of the curve. The initial smoothing area can be shielded via tuning b, which can express KPIs that immediately enter the sensitive area from the very beginning.

Unified Index Each KPI is represented by a distinct S-curve to ensure independence among the indicators. Specifically, unique S-curves for network latency, packet-loss rate, and jitter are created by assigning specific parameters (a and b), offering tailored indexes for applications sensitive to these different metrics. The comprehensive CEI score is then derived by aggregating these three S-curves, each weighted appropriately: CEI(x, y, z) = w1 * f1(x) + w2 * f2(y) + w3 * f3(z)

x, y, and z respectively indicate values of the three major network KPIs: network latency, packet-loss rate, and jitter.
f1, f2, f3 represent the three individual S-curves.
w1, w2, w3 represent the empirical weights.

Parameter Tuning

Weight Proportion The CEI's flexibility allows for fine-tuning to meet specific application needs by adjusting its weight values (w1, w2, w3), enabling precise adaptation for various application categories. Typically, cloud customers engage in scenarios that are either sensitive to latency—like gaming applications—or to packet loss, such as audio and video streaming. For instance, in latency-sensitive scenarios, the weights for latency, packet-loss rate, and jitter could be adjusted to a ratio of 7:2:1 (w1:w2:w3); whereas for packet-loss-sensitive scenarios, a ratio of 2:7:1 (w1:w2:w3) might be more appropriate. This tailored approach allows the CEI to accurately assess network quality for different types of applications from a specific viewpoint (e.g., a fixed test point) across various cloud vendors.

Parameter a, b The parameters a and b of the CEI formula can be fine-tuned via:

Determine the initial values of parameters a and b by fitting each KPI CEI curve based on a large amount of operational data.
Parameters a and b can be further tuned based on preferences of certain application class. For example, when the packet-loss rate is no higher than 𝑘, it is desired that CEI goes up as the network latency lowers. CEI can set tuning goals according to such preferences and fine-tune parameters a and b.

Security Considerations TBD.

IANA Considerations This document has no IANA actions.