Efficient bit stream adaptation and resilience to packet losses are two critical requirements in scalable video coding for transmission over packet-lossy networks. These requirements have a greater significance in scalable H.264/AVC video bit streams since missing refinement information in a layer propagates to all lower layers in the prediction hierarchy and causes substantial degradation in video quality. This work proposes an algorithm to accurately estimate the overall distortion of the reconstructed frames due to enhancement layer truncation, drift/error propagation, and error concealment in the scalable H.264/AVC video. This ensures low computational cost since it bypasses highly complex pixel-level motion compensation operations. Simulation results show an accurate distortion estimation at various channel loss rates. The estimate is further integrated into a cross-layer optimization framework for optimized bit extraction and content-aware channel rate allocation. Experimental results demonstrate that precise distortion estimation enables our proposed transmission system to achieve a significantly higher average video PSNR compared to a conventional content independent system.