With the various technological advances, mobile devices are not just being used as a means to make voice calls; but are being used to accomplish a variety of tasks. Mobile devices are being envisioned to practically accomplish any task which could be done on a computer. This is hurdled by the limited computational resources available with the mobile devices due to their portable size. With the mobile devices being connected to the Internet, leveraging cloud services is being seen as a promising solution to overcome this hurdle. Computationally intensive tasks can be offloaded to the Cloud servers. However, owing to the latency and cost associated with using cloud services, edge devices (termed cloudlets) stationed near the mobile devices are being seen as a prospective alternative to replace/assist the Cloud services. The mobile devices have an easier access to the cloudlets being situated in their vicinity and can offload their task requests to them to be served at a lower cost. This paper considers a network of such connected cloudlets which provide service to the mobile devices in a given area. We address the issue of task assignment in such a scenario (i.e. which cloudlet serves which mobile device) aimed towards improving the quality of service experienced by the mobile devices in terms of minimizing the latency. Through numerical simulations we demonstrate the performance gains of the proposed task assignment scheme showing lower latency as compared to the traditional scheme for task assignment.