The OpenFlow (OF) paradigm embraces third-party development efforts, and therefore suffers from potential trust issue on OF applications (apps). The abuse of such trust could lead to various types of attacks impacting the entire network. In this paper, we propose PermOF, a fine-grained permission system, as the first line of defense, in order to apply minimum privilege on apps. We summarize a set of 18 permissions to be enforced at the API entry of the controller. To accommodate the isolation requirements, we propose a customized isolation mechanism, which achieves comprehensive resource isolation and access control.