DC Lesson: Complaints lead to better service
One might usually *groan* at the sight of a ‘complaint’. But over the years, I have learned that listening to them USUALLY leads to valuable insights that help improve our operations.
Such was the case yesterday. This complaint ‘chain’ started with an email from one of our clients. Apparently, they themselves were told that their website (hosted with BNSHosting.net) was not accessible from their complainant’s IP address block (223.25.0.0/18).
Using ipinfo.io, I ascertained that this was now assigned to a Philippine network : AS56099 belonging to Asian Vision Cable. To troubleshoot, we asked for traceroute information from them. A traceroute shows the hops the packets take from the source (client) to reach us (hosted webservice). As we can see below, the packets stopped at hop #6:
The above shows that the packets — for some unknown reason — never got to the webserver. From our operations side we did a reverse traceroute too:
Strange, why would the packets be dropped at our edge router in Manila? This provided the vital clue, and this lead to discovering this routing entry:
Digging back into our ticketing system (knowledgebase), I found out that a few years back, we got a lot of ‘bad traffic’ from 223.0.0.0/8. It was assigned back then to AS63555 CNBIDCC (China). We had blocked all traffic from that network — since our abuse complaints remained unresolved.
APNIC (Asia Pacific Network Information Center) re-allocated the IP addresses that belonged to the SAME block to Asian Vision Cable. Their subscribers would be using the IP addresses that we had blocked from our network. Their subscribers would not be able to access our hosted clients. The block/filter was still in place. And it would have stayed there forever — had our attention not been called by this ‘complaint’ This is bad for both sides (hosting provider and ISP provider).
The solution was to resolve the routing entry and voila, the traceroute completes!
Now we get a win-win scenario. Both ISP provider and hosting provider are able to provide better service. We thank Asian Visions for bringing this to our attention. Thankfully, it was a happy ending for all three parties!