Today a growing number of applications that operate in dynamic environments use distributed systems. Often, one or more constrained resources need to be allocated across such systems. In addition, for a number of important applications, the resources to be allocated are discrete quantities and the demand profile is intrinsically location and time dependent (thus pre-computating the resource allocation problem off-line is inadequate). We model such systems using graphs - vertices representing the sites and edges capturing locality - where each vertex is associated with the a pair of (integer) numbers representing the current and the required level of a resource. We seek on-the-fly reallocation of resources that satisfy the demand at each node, if it is feasible to do so, while minimizing a given metric of interest, such as the total distance traveled, maximum disruption time, etc. Due to integrality constraints, one expects such a problem to be NP-hard. However, we show that the matrix representing constraints of the problem satisfies the Total Unimodularity property and hence the problem is solvable in polynomial time. We propose a distributed algorithm that employs only local communications. We characterize the proposed algorithm and by numerical performance evaluation, show that it significantly outperforms known heuristic algorithms.