2.4.2 Why Measure Performance?

Policy makers seeking to promote broadband development need mechanisms to ensure that their objectives are being achieved and to identify whether corrections and refinements to policies and programs are needed. In short, they need to measure progress through regular monitoring in order to identify successes and failures. Different countries will adopt broadband strategies with different objectives, which will affect the appropriate indicators to monitor. It is best to build the indicators most appropriate for the selected objectives into the design of the programs from the beginning and to allocate the necessary resources for data collection and analysis from the start. Broadband indicators are also needed for analysis, for example, to examine trends and the link between broadband adoption and social and economic development. They are also important for monitoring license compliance in areas such as coverage and quality. As a result, the specific indicators appropriate for a particular country, the frequency of data collection and reporting, the geographic unit of analysis, and so forth will also differ from country to country. Consequently, the following section provides a range of options rather than a single prescription for countries to consider when looking at measurement issues.

2.4.3 What to Measure?

The broadband indicators likely to be of the most interest to policy makers are availability, demand, quality, and pricing (Figure 2.4). These indicators relate to local retail access rather than to wholesale and backbone markets.*  Although the deployment of national backbones is an important goal of some broadband plans, the indicators to measure developments in these areas have not been identified or defined by the international statistical community, and the data are not widely available. Nevertheless, perusal of plans from some countries can help to identify relevant indicators. For example, India’s proposed broadband plan calls for the construction of a national fiber optic backbone throughout the country. Deployment might be measured by indicators such as the number of localities served by the national fiber optic backbone and kilometers of fiber backbone in the network. See TeleGeography, “India’s National Broadband Policy to Be Sent for Cabinet Approval Shortly,” March 31, 2011, http://www.telegeography.com/products/commsupdate/articles/2011/03/31/indias-national-broadband-policy-to-be-sent-for-cabinet-approval-shortly/. Additional indicators also may be useful for monitoring and analysis, including monetary-based statistics such as broadband revenues. The Partnership on Measuring ICTs for Development, a coalition of intergovernmental agencies, has produced a methodological manual identifying core ICT statistics including several broadband indicators.*  [[AQ: notes 34 and 3 were combined]] The Partnership on Measuring ICTs for Development aims to develop further initiatives regarding the availability and measurement of ICT indicators at the regional and international levels. It provides an open framework for developing a coherent and structured approach to advancing the development of ICT indicators globally, particularly in developing countries. Partners include EUROSTAT, ITU, the OECD, United Nations Conference on Trade and Development, United Nations Educational, Scientific, and Cultural Organization’s Institute for Statistics, the United Nations regional commissions (Economic Commission for Latin America and the Caribbean, Economic and Social Commission for Western Asia, Economic and Social Commission for Asia and the Pacific, and Economic Commission for Africa), United Nations Department of Economic and Social Affairs, and the World Bank. See http://www.itu.int/ITU-D/ict/partnership/index.html. See also UNCTAD (2010). In addition, the ITU has identified and defined other broadband-related statistics. See ITU (2010). This manual provides a useful list of key broadband indicators based on definitions with international consensus.

2.4.3.1 Availability (Supply)

Availability refers to the ability to access wireline and wireless broadband networks and services. Different modes of providing broadband exist; therefore, different indicators of availability are needed for each of the modes. In the case of wireline systems, availability can be measured by the percentage of households passed. This is a conventional measure in the cable industry that can be extended to fiber and DSL as well. The indicator reflects the number of copper (telephone), coaxial (cable television), or fiber optic lines accessible by a premise, regardless of whether users actually subscribe to the broadband service.*  For more on issues related to measuring broadband coverage, see OECD (2009). It may also be useful to distinguish between the type of technology, such as DSL, cable modem, and fiber to the premises (FTTP). This provides an idea of the relative importance of each to broadband development as well as the degree of intermodal competition between technologies. It may also be useful to provide a breakdown of subscriptions by speed ranges and geographic area. These considerations are becoming increasingly important as countries seek to deploy minimum-speed broadband services to unserved and underserved populations.

In the case of wireless, the obvious indicator of availability is signal coverage. This can be measured in terms of population or area. The International Telecommunication Union (ITU) has developed a definition for wireless broadband coverage in the form of 3G or 4G network coverage, although the data are not reported for most countries. Parallel definitions for fixed wireless, satellite, and wireline coverage do not exist within the ITU definitions. However, several countries in the OECD report these data using definitions developed either by national governments or by industry organizations (OECD 2009). They may be adapted by countries wishing to develop comprehensive coverage indicators.

2.4.3.2 Adoption (Demand)

While supply-side indicators give a general idea of high-speed Internet availability, they do not reflect concrete adoption or usage. Measuring the uptake or adoption of wireline and wireless technologies, however, is significantly more difficult than measuring the supply. While coverage measures the theoretical ability to access broadband services, the number of subscribed connections measures actual demand for the service. Subscriptions should be minimally broken down by wireline and wireless broadband and preferably by additional categories to allow for deeper analysis. A growing number of countries are measuring broadband access by households and businesses through surveys typically carried out by the national statistical offices. These demand-side surveys typically include various indicators of use, which can illuminate factors contributing to broadband take-up.

There is no international indicator for the percentage of the population that uses broadband, although some countries conduct surveys to determine the percentage of Internet users. This would be a useful supplementary indicator for monitoring and evaluating broadband markets.

2.4.3.3 Quality

In order to use or fully utilize certain applications, certain performance parameters must be met by the broadband connection. Two of the most important are latency (the amount of time it takes for a packet to travel between sender and receiver) and speed, which can be monitored for both fixed and wireless networks. Other broadband performance metrics include signal quality, availability (“uptime”), complaint ratios, and service activation and restoration times. Technical means exist to measure these aspects at various points in the link between the end user and the server providing the application. Such information is important both to policy makers, who can use it to ensure that the broadband networks and services being supplied are up to industry standards, and to consumers, who can use it to decide which service will provide them with the highest quality. Many consumer complaints hinge on differences between advertised and actual speeds.*  “Average Broadband Speed Is Still Less Than Half Advertised Speed,” Ofcom, March 2, 2011, http://media.ofcom.org.uk/2011/03/02/average-broadband-speed-is-still-less-than-half-advertised-speed/.

In Bahrain, for example, the Telecommunications Regulatory Authority publishes quarterly reports measuring average download and upload speeds and domain name system (DNS) and latency times (Figure 2.7; see Bahrain, Telecommunications Regulatory Authority 2011). In the absence of regular monitoring some regulators publish links on their websites to third-party applications for measuring speed and other aspects of quality.*  For example, the Federal Communications Commission in the United States has a broadband webpage, where consumers can test their speed, latency, and jitter. See http://www.broadband.gov/qualitytest/about/.

Because differences exist in performance inside the ISP domain (the user and the server are within the ISP’s system), the national domain (the user and the server are on different systems, but within the national territory), and the international domain (the user and server are in different countries), measuring performance in each domain yields diagnostic information useful for regulators, operators, and consumers. For example, the Info-communications Development Authority of Singapore establishes different latency parameters depending on whether Internet traffic is national or international (IDA 2011).

2.4.3.4 Pricing

A government that launches a broadband initiative using public resources will want the service to be affordable to the intended beneficiaries. One could argue that prices need not be monitored in the case of purely private supply, where no public resources have been expended. However, when broadband is seen as an essential public utility or where prices are high due to market failure, governments may want to monitor pricing. Concerns about this issue have prompted countries, such as India and the United States, to include “affordable” broadband access as a key factor or goal in their broadband initiatives (United States, FCC n.d., 10; TRAI 2010).

Competitive broadband markets typically have multiple tariffs with varying levels of bandwidth, data download caps, and discounts. This presents methodological challenges in terms of compiling comparative broadband tariff indicators across technologies. Baskets of monthly services are often used as a common measure of price trends that factor in caps and speeds. The key components include the monthly price of broadband service, the corresponding speed, and, if applicable, the cap and prices for exceeding the cap. Capped versus unlimited packages pose comparison problems, but can be mitigated somewhat by comparing price per advertised Mbit/s.

An example comparing wireline and mobile broadband monthly prices for selected economies is shown in Table 2.6 . The example illustrates the various ways of looking at broadband pricing and highlights comparability issues. One notable aspect is the difference between entry-level prices, speeds, and affordability (in terms of price as a percentage of per capita income). For example, although an entry-level fixed broadband package in Turkey costs almost twice as much as in Brazil, the Turkish tariff is a slightly better value since the download speed is twice as fast as in Brazil. Similarly, although the entry-level price for fixed broadband in Brazil is more than twice that in Vietnam, it is much more affordable in Brazil than in Vietnam (although the value of the Vietnamese package is 10 times more).

Mobile broadband pricing is a bit more difficult to compare, since some operators do not guarantee advertised speeds. Instead, prices tend to vary by the volume of data downloaded per month. Nevertheless, as Table 2.6 shows, the price of mobile broadband is the same as or lower than the price of wireline broadband (except in Brazil). It is important to note that mobile broadband is more often capped than wireline and that real mobile broadband speeds are also lower in many cases.

2.4.4 How to Measure?

Broadband indicators affect many parties. Government agencies responsible for broadband policy should consult internationally comparable indicators and identify those most suitable for monitoring and evaluation. Best practice suggests that national regulatory agencies should compile broadband statistics, such as number of subscriptions, solicited from operators. This arises out of their mandate to regulate and monitor the sector. Ideally, policy makers consult and cooperate with national statistical agencies that have the technical skills to produce demand-side statistics through household and enterprise surveys, asking about broadband possession or use of different ICTs by households and businesses (or by individuals). Broadband operators play a key role, both as providers and as consumers of the data.

The entities best positioned to provide supply-side data are the suppliers of the relevant services. It is common for provisions mandating that the reporting of data to the government or the regulatory agency be included in statutes governing the industry or in licenses or concession contracts. Irrespective of legal provisions, the principal challenge will be to ensure regular and timely reporting of the required indicators based on adherence to agreed-upon standard definitions and procedures.

Most governments do not monitor their country’s broadband development in a vacuum. They typically need data from other countries to put their nation’s high-speed market evolution in perspective and benchmark it with that of other countries. Brazil, for instance, compared its broadband penetration and forecast evolution to those of Argentina, Chile, China, Mexico, and Turkey.

Several international sources harmonize and disseminate statistics for different countries. The ITU has been the traditional repository of supply-side data on telecommunications and now on ICTs, including some demand-side data. Similarly, the OECD collects and disseminates a number of broadband indicators for its member countries, as does EUROSTAT, the statistical arm of the European Union. All of these organizations make the data available on dedicated websites (Table 2.7). The Economic Commission for Latin America and the Caribbean recently launched a broadband indicator site for its members.*  ECLAC, “ECLAC Launched Regional Broadband Observatory,” Press Release, May 27, 2011. In addition, several private sector entities publish broadband statistics on mobile broadband subscriptions as well as average download speeds and other quality metrics.*  [[AQ: notes 46 and 47 were combined]] GSM World, “Market Data and Analysis,” http://www.gsmworld.com/newsroom/market-data/market_data_and_analysis.htm. Ookla’s Net Index provides average download speeds for 170 economies; see http://www.netindex.com/download/allcountries/. Akamai compiles performance data for a number of economies. Also see Akamai’s network performance comparison, http://www.akamai.com/html/technology/dataviz2.html.