We present an architecture of projection-based autoregressive neural network (aNN) where model-reduced adjoint is efficiently produced with the help of an easy-to-use auto-differentiation (AD) tool in deep-learning frameworks. Analogy to reduced-order tangent linear model, a projection-based aNN (POD-aNN) structure is proposed to accelerate the construction of adjoint model based on reduced subspace. The POD-aNN consists of a dimensionality reduction and an intermediate non-linear transition unit which is used to project a state system to a low-dimensional subspace and approximate time-varying evolution of system states in low dimension, respectively. The adjoint model is run in reduced space with negligible computational cost and memory requirement. Once the gradient is obtained in reduced space it is projected back in full space and then the inversion modeling is conducted. Characteristics and performance of the method are illustrated with two sets of inverse modeling experiments in a synthetic 2D fluid flow model with random spatially dependent parameters. It shows that the proposed POD-aNN obtains satisfactory results with significantly reduced computational cost and, therefore, demonstrates promising applicability to practical reservoir models.