Chronic stroke lesion segmentation on magnetic resonance imaging scans plays a critical role in helping physicians to determine stroke patient prognosis. We propose a convolutional neural network (CNN) segmentation network - a 3D Cross-hemisphere Neighborhood Difference ConvNet -which utilizes brain symmetry. The main novelty of this network lies on a 3D cross-hemisphere neighborhood difference layer which introduces robustness to position and scale in brain symmetry. Such robustness is important in helping the CNN distinguish between minute hemispheric differences and the asymmetry caused by a lesion. We compared our model with the state-of-the-art method using a chronic stroke lesion segmentation database. Our results demonstrate the effectiveness of the proposed model and the benefit of a CNN that combines the physiologically based information, that is, the brain symmetry property.