It is known that there exist Triangle Inequality Violations (TIVs) with respect to network Quality of Service (QoS) metrics such as latency between nodes in the Internet. This motivates the exploitation of QoS-aware routing overlays. To find an optimal overlay route, we would usually need to examine all the possible overlay routes. However, this requires both measuring QoS between all node pairs and investigating all the routes in the full-mesh overlay topology, which poses scalability problem in terms of both measurement cost and route calculation and dissemination cost. We thus propose a method of constructing a QoS overlay network that enables us to find a near optimal route in a cost-effective manner. Our idea is based on the finding that a small number of overlay nodes can provide the optimal routes for a large number of node pairs, which is obtained through measurement data analysis between PlanetLab nodes. Our overlay network has two layers where the upper-layer consists of such small number of overlay nodes that can provide the optimal routes while the lower-layer consists of the other overlay nodes. By allocating such overlay nodes at the upper-layer, we can provide better QoS routes for each node pair with high probability. We construct the overlay network topology where the upper-layer overlay nodes are connected in full-mesh manner while the lower-layer overlay nodes are not connected in full-mesh but only to upper-layer nodes. Through this structure, we can reduce measurement and route calculation costs. Using PlanetLab data, we show that our method can achieve almost the same performance as the optimal solution.