This work presents the use of multiple sensor modalities in order to perform traffic analysis for health monitoring of transportation infrastructure. In particular, testbeds containing video and seismic sensors giving complementary information about vehicles are described. Computer vision algorithms are used to detect and track the vehicles and extract their properties. This information is combined with the data from seismic sensors for robust classification of vehicles. Experimental results obtained with our testbeds are described.