Vehicle routing problem with time windows (VRPTW) is a well-known NP-hard combinatorial optimization problem that is crucial for transportation and logistics systems. Even though the particle swarm optimization (PSO) algorithm is originally designed to solve continuous optimization problems, in this paper, we propose a set-based PSO to solve the discrete combinatorial optimization problem VRPTW (S-PSO-VRPTW). The general method of the S-PSO-VRPTW is to select an optimal subset out of the universal set by the use of the PSO framework. As the VRPTW can be defined as selecting an optimal subgraph out of the complete graph, the problem can be naturally solved by the proposed algorithm. The proposed S-PSO-VRPTW treats the discrete search space as an arc set of the complete graph that is defined by the nodes in the VRPTW and regards the candidate solution as a subset of arcs. Accordingly, the operators in the algorithm are defined on the set instead of the arithmetic operators in the original PSO algorithm. Besides, the process of position updating in the algorithm is constructive, during which the constraints of the VRPTW are considered and a time-oriented, nearest neighbor heuristic is used. A normalization method is introduced to handle the primary and secondary objectives of the VRPTW. The proposed S-PSO-VRPTW is tested on Solomon's benchmarks. Simulation results and comparisons illustrate the effectiveness and efficiency of the algorithm.