This paper presents a new and efficient approach for optimization and implementation of filter banks e.g. velocity channels, orientation channels and scale spaces. The multi layered structure of a filter network enable a powerful decomposition of complex filters into simple filter components and the intermediary results may contribute to several output nodes. Compared to a direct implementation a filter network uses only a fraction of the coefficients to provide the same result. The optimization procedure is recursive and all filters on each level are optimized simultaneously. The individual filters of the network, in general, contain very few non-zero coefficients, but there are are no restrictions on the spatial position of the coefficients, they may e.g. be concentrated on a line or be sparsely scattered. An efficient implementation of a quadrature filter hierarchy for generic purposes using sparse filter components is presented. keywords filter optimization, filter network, sequential convolution, sparse filters, efficient filtering.