Increasing number of the cloud-based Internet applications demands for efficient resource and cost management. This paper proposes a real-time group auction system for the cloud instance market. The system is designed based on a combinatorial double auction, and its applicability and effectiveness are evaluated in terms of resource efficiency and monetary benefits to auction participants (e.g., cloud users and providers). The proposed auction system assists them to decide when and how providers allocate their resources to which users. Furthermore, we propose a distributed algorithm using a group formation game that determines which users and providers will trade resources by their cooperative decisions. To find how to allocate the resources, the utility optimization problem is formulated as a binary integer programming problem, and the nearly optimal solution is obtained by a heuristic algorithm with quadratic time complexity. In comparison studies, the proposed real-time group auction system with cooperation outperforms an individual auction in terms of the resource efficiency (e.g., the request acceptance rate for users and resource utilization for providers) and monetary benefits (e.g., average payments for users and total profits for providers).