China’s Internet Service Providers (ISPs) do not like to talk to each other. the vast majority (90% in 2006) of customer complains received by the Ministry of Information Industry is about super slow interconnectivity of the two major ISPs, China Telecom and China Netcom, and people’s workarounds of the interconnectivity problem have side effects, like the recent Internet outrage caused by the Baofeng Media Player.
Here is a little background. The interconnectivity problem arose after the spin-off of China Netcom from China Telecom 2002. Before the split, China Telecom had a monopoly on network bandwidth, and local ISPs could not really compete with China Telecom because they had to rent China Telecom’s bandwidth for non-local services. The government ordered a split of the company after most of local ISPs went out of business, and created the interconnectivity problem.
After the spin-off, the China Telecom network is now split into two subnets, China Telecom in south and China Netcom in north. The two companies employed some anticompetitive tactics, including exclusive service agreements with estate developers and not keeping up with growing demand of interconnectivity between major networks. In 2006, the only bridge between them was a connection point at Beijing, which was often so overwhelmed that it was usually slower than routing though a USA network or switching to China Mobile’s CDMA wireless network.
In fact, the majority of connectivity was done by businesses. With ISPs reluctant to connect each other themselves, the only solution was to connect both networks and double the connectivity expense. At the beginning, the users were directed to ISP-specific addresses like http://chinatelecom.example.com in the China Telecom network, and http://chinanetcom.example.com in the China Netcom network. The problem was, users often didn’t know what their ISP is when they are not visiting Internet at home. Soon ISP-specific domain name service (DNS) server were created to direct visitor to the the address inside the visitor’s network. Now the user can use http://www.example .com and the DNS will decide which address the user will be visiting based on the user’s network. The free, personal financed DNSPod is this kind of DNS service.
Unfortunately, several DNS servers of DNSPod were hit by a distributed denial-of-service (DDOS) attack on the night of May 18. DNSPod’s owner did not think the attack was serious, because he dealt with larger attacks before. However, the DNS server China Telecom shut-off this time was hosting a web site used by the Baofeng Media Player, which manifested the DNS outrage to a much greater magnitude.
There was no explanation why the Baofeng Media Player’s company, a multi-million dollar investment with a self-claimed user base of 200 million , is using a free (*read* no warranty of any kind, agreed not liable to damage before using) DNS service. However, the greater mistake was that the Baofeng Media Player’s advertisement downloading service was programmed to go through each server on its server list until it gets an answer. When the ISP’s local DNS cache pointing to the downed DNSPod server expired, the requests to find servers put almost all DNS servers in China under a DDOS attack . At the attack’s peak, Baofeng Media Player’s DNS requests occupied 40% of China Telecom’s total bandwidth. A DDOS attack targeting a specific domain now transformed into a DDOS attach on almost all DNS servers in China, which slowed down the Internet in China to a crawl.
After a day of network outrage, there are lessons to be learned , from programming errors to doubtful business tactics. One thing is for sure, Baofeng Media Player’s advertisement downloading service is the one to blame for this disaster. However, Baofeng Media Player is not the first software with bugs, nor will it be the last one. China ISPs need to think about how to avoid this problem in the future, the sooner the better.
