Establishing free space optical links between flying vehicles is a technically tough challenge and needs enhancements both in the technology and its features. For unmanned aerial vehicle (UAV) swarms to evolve to effective usage in civil and military applications, a means of high data rate communications need be established between them. The free space optical links due to their huge bandwidth and license free spectrum provide a viable communications solution between the UAV swarm. The swarm UAV scenario due to the continuous motion and changing relative speeds of all its members provide extremely challenging conditions to maintain a line-of-sight free space optical link and this paper reviews the existing related scientific developments and talks about the challenges of utilizing FSO in the swarm UAV environment. Further, this survey paper discusses the utility and challenges of developing free space optical links between HAPs.