A membrane bioreactor (MBR) was continuously operated to investigate mechanisms of fouling caused by the gel layer in this study. Agar was used as a model foulant for gel layer formation, and filtration resistance of gel layers was systematically assessed. The results showed that gel layer possessed unusually high specific filtration resistance (SFR) and high measured porosity as compared with cake layer. Current knowledge cannot explain the contradiction between high filtration resistance and high porosity of gel layer. A new fouling mechanism based on Flory-Huggins theory was then proposed. Filtration resistance of agar gel layer was found to be independent of pH and ionic strength, but linearly increase with gel thickness. The results are accordant with the mechanism deductions. Simulation of the mechanism model showed that the filtration resistance induced by mixing chemical potential variation was comparable to the experimental data of filtration resistance of agar gel layer, indicating that the proposed mechanism is the predominant mechanism responsible for the high filtration resistance of gel layer. The proposed mechanism was further verified from the bound water viewpoint.