基于代理模型和深度强化学习的试井自动解释方法

doi:10.19596/j.cnki.1001-246x.8533

摘要/Abstract

摘要：

基于深度确定性策略梯度(DDPG)算法，提出一种基于深度强化学习的试井曲线自动解释方法。将该方法应用于4种不同试井模型条件下的曲线自动拟合。为提高训练效率，建立基于LSTM神经网络的试井代理模型。通过训练智能体与代理模型交互，最终得到最优曲线拟合策略。结果表明：曲线参数解释的平均相对误差为5.51%，平均计算时间为0.427 s，计算速度相较物理模型提高两个数量级。该方法在案例应用中参数解释平均相对误差为4.32%，证明了方法的可靠性。

关键词: 试井解释, 参数反演, 深度强化学习, 自动拟合

Abstract:

A deep reinforcement learning (DRL) based approach is proposed for automatic interpretation of well-testing curves. Based on a deep deterministic policy gradient (DDPG) algorithm, the proposed DRL approach is successfully applied to automatic matching of four different types of well-testing curves. To improve training efficiency, a surrogate well-testing model based on LSTM neural network was established. With episodic training, through interaction with the surrogate model the agent converged finally to an optimal curve matching policy on different well-testing models. It shows that the average relative error of the curve parameter interpretation is 5.51%. Additionally, the proposed DRL approach has a high calculation speed, and the average computing time is 0.27 seconds. In case study applications, the proposed method achieved an average relative error of 4.32% in parameter interpretation, which shows reliability of the method.

Key words: well testing, parametric inversion, deep reinforcement learning, automatic matching

董鹏, 廖新维. 基于代理模型和深度强化学习的试井自动解释方法[J]. 计算物理, 2023, 40(1): 67-80.

Peng DONG, Xinwei LIAO. An Efficient Approach for Automatic Well-testing Interpretation Based on Surrogate Model and Deep Reinforcement Learning[J]. Chinese Journal of Computational Physics, 2023, 40(1): 67-80.

图/表 20

图1 物理模型示意图(a)垂直裂缝井模型; (b)多段压裂水平井模型; (c)双重介质水平井模型; (d)径向复合水平井模型

Fig.1 Physical model schematics of a fractured well (a) vertical fractured well model; (b) multiple-fractured horizontal well model; (c) dual-porosity horizontal well model; (d) radial composite horizontal well model

表1 试井模型参数范围及分布

Table 1 Parameter range and distribution of well-testing models

参数	最小值	最大值	分布	模型1	模型2	模型3	模型4
C_D	20	2 000	对数均匀分布	√	√	√	√
S	0.01	1	均匀分布	√	√	√	√
L_xf/m	30	300	对数均匀分布	√	√	√	√
C_fD	400	25 000	对数均匀分布	√	√	√	√
n_f	5	25	均匀分布		√	√	√
L_w/m	200	2 000	均匀分布		√	√	√
ω	0.01	0.2	均匀分布			√
λ	10^-8	10^-4	对数均匀分布			√
R_i/m	200	4 000	均匀分布				√
M	2	10	均匀分布				√

图2 代理模型训练曲线

Fig.2 Training curves of the surrogate model

图3 验证集不同模型中各时间点的预测压力及导数相对误差(a)垂直裂缝井模型; (b)多段压裂水平井模型; (c)双重介质水平井模型; (d)径向复合水平井模型

Fig.3 Relative errors of predicted pressure and derivative in different models in the test set at each time point (a) vertical fractured well model; (b) multiple-fractured horizontal well model; (c) dual-porosity horizontal well model; (d) radial composite horizontal well model

表2 物理模型与代理模型计算效率

Table 2 Computational efficiency of physical model and proxy model

试井模型	物理模型计算时间/s	代理模型
试井模型	物理模型计算时间/s	训练时间/min	预测时间/s
垂直裂缝	4.23	7.1	0.018
水平井	7.71	7.5	0.023
双重介质水平井	13.63	8.2	0.021
径向复合水平井	12.87	7.7	0.027

图4 PTA模型和LSTM代理模型的计算成本

Fig.4 Computation cost of PTA model and LSTM surrogate model

图5 代理模型预测结果的可视化

Fig.5 Visualization of surrogate model prediction results

图6 强化学习中智能体与环境的交互作用[24]

Fig.6 Interaction between agent and environment in reinforcement learning

图7 基于代理模型和强化学习的自动曲线匹配过程

Fig.7 Automatic curve matching process based on surrogate model and reinforcement learning

图8 网络的结构(a)评论网络; (b)行动网络

Fig.8 Structure of (a) critic network; (b)actor network

图9 基于DDPG算法的智能体训练过程

Fig.9 Training process of agent based on DDPG algorithm

表3 智能体反演变量及训练时间

Table 3 Inverted variables and agent training time

试井模型	变量	训练时间/min
试井模型	变量	代理模型训练	智能体训练
垂直裂缝	C_D, S, L_xf, C_fD	6.3	67
水平井	C_D, S, L_xf, C_fD	7.5	70
双重介质水平井	L_xf, C_fd, ω, λ	7.1	68
径向复合水平井	L_xf, C_fd, R_i, M	7.6	72

图10 DDPG在自动曲线匹配上的表现

Fig.10 DDPG performance on automatic curve matching

图11 评判网络和行动网络的损失曲线(a)评判网络；(b)行动网络

Fig.11 Loss curves of (a) critic network; (b) actor network

图12 智能体的1 000次曲线拟合的裂缝半长散点图(a)垂直裂缝井模型; (b)多段压裂水平井模型; (c)双重介质水平井模型; (d)径向复合水平井模型

Fig.12 Scatter plots of 1000 times curve matching of agent for fracture half length (a) vertical fractured well model; (b) multiple-fractured horizontal well model; (c) dual-porosity horizontal well model; (d) radial composite horizontal well model

表4 参数反演结果的误差统计

Table 4 Error statistics of parameter inversion results

模型	参数	平均相对误差/%	R²
垂直裂缝	log (C_D)	4.658 4	0.830 2
	S	11.718 4	0.893 9
	log (L_xf)	2.551 5	0.898 4
	log (C_fD)	2.701 8	0.825 7
水平井	log (C_D)	5.112 0	0.849 3
	S	15.223 9	0.834 3
	log (L_xf)	3.709 9	0.892 8
	log (C_fD)	4.600 5	0.885 2
双重介质水平井	log (L_xf)	2.069 5	0.894 8
	log (C_fD)	3.608 4	0.852 1
	ω	6.925 1	0.869 5
	λ	5.680 4	0.898 4
径向复合水平井	log (L_xf)	3.713 4	0.891 9
	log (C_fD)	5.376 5	0.843 7
	R_i	2.976 5	0.896 8
	M	7.454 3	0.829 6

图13 案例1中曲线匹配过程的可视化

Fig.13 Visualization of curve matching process in Case 1

表5 案例1参数反演过程及结果

Table 5 Parameters inversion process and result in Case 1

	log C_D	S	log L_xf	log C_fD
初始参数	1.501	0.066	1.506	2.782
步骤1	2.003	0.240	1.712	2.812
步骤2	2.219	0.191	1.868	3.502
步骤3	2.563	0.161	1.958	3.722
步骤4 (预测结果)	2.815	0.076 8	2.006	3.908
目标参数	2.735	0.0691	2.053	3.857
相对误差/%	2.83	11.12	2.33	1.31

图14 案例2中曲线匹配过程的可视化

Fig.14 Visualization of curve matching process in Case 2

表6 案例2参数反演过程及结果

Table 6 Parameters inversion process and result in Case 2

	log L_xf	log C_fD	ω	log λ
初始参数	1.477	2.602	0.011	-8.854
步骤1	1.602	2.616	0.037	-7.374
步骤2	1.903	2.811	0.074	-7.092
步骤3	1.987	3.389	0.065	-6.431
步骤4 (预测结果)	2.013	3.610	0.054	-6.773
目标参数	1.954	3.699	0.050	-7.000
相对误差/%	3.00	2.40	8.32	3.24

参考文献 35

1	WEI X X , FENG Q H , ZHANG X M , et al. Distribution of remaining oil in water flooding at pore scale: Volume of fluid method[J]. Chinese Journal of Computational Physics, 2021, 38 (5): 573- 584.
2	ZHANG J , CHENG S Q , ZENG Y , et al. Applicability exploration of superposition principle for pressure transient analysis in polymer-flooding system[J]. Chinese Journal of Computational Physics, 2021, 38 (3): 324- 332.
3	严涛. 试井多解性问题研究[J]. 海洋石油, 2008, (3): 46- 50.
4	KHADIVI K , HASSANZADEH H . Role of integration of core data in elimination of nonunique solutions in well testing of heterogeneous formations[J]. Journal of Porous Media, 2021, 24 (1): 25- 40. DOI
5	文必龙, 李艳春. 基于大数据的试井解释参数分析[J]. 计算机应用与软件, 2019, 36 (9): 64- 69.
6	张晓东. 试井曲线计算机自动拟合算法及应用[J]. 西南石油大学学报(自然科学版), 2008, (4): 135- 137.
7	王一, 陈殿远, 韩鑫, 等. 基于信赖域算法的试井曲线自动拟合研究[J]. 重庆科技学院学报(自然科学版), 2019, 21 (4): 47- 50. DOI
8	DASTAN A , HORNE R . Robust well-test interpretation by using nonlinear regression with parameter and data transformations[J]. SPE Journal, 2011, 16 (3): 698- 712. DOI
9	吴明录, 姚军, 王子胜, 等. 利用双种群遗传算法进行数值试井自动拟合[J]. 油气地质与采收率, 2007, (2): 81- 83. DOI
10	王培玺, 张静. 基于分群式粒子群算法的压裂水平井试井曲线自动拟合[J]. 中国石油大学学报(自然科学版), 2012, 36 (2): 136- 140. DOI
11	AWOTUNDE A A . Estimation of well test parameters using global optimization techniques[J]. Journal of Petroleum Science and Engineering, 2015, 125, 269- 277. DOI
12	GOMEZ S, CAMACHO R, VASQUEZ M, et al. Well test characterization of naturally fractured vuggy reservoirs, with a global optimization method[C]. Offshore Technology Conference, 2014.
13	胡修凤. 基于BP神经网络的沁中南煤层气井解吸压力预测及应用[D]. 焦作: 河南理工大学, 2017.
14	ALMARAGHI A M, EL-BANBI A H. Automatic reservoir model identification using artificial neural networks in pressure transient analysis[C]. Society of Petroleum Engineers, 2015.
15	MOOSAVI S R , VAFERI B , WOOD D A . Auto-detection interpretation model for horizontal oil wells using pressure transient responses[J]. Advances in Geo-Energy Research, 2020, 4 (3): 305- 316. DOI
16	VAFERI B , ESLAMLOUEYAN R , AYATOLLAHI S . Automatic recognition of oil reservoir models from well testing data by using multi-layer perceptron networks[J]. Journal of Petroleum Science and Engineering, 2011, 77 (3-4): 254- 262. DOI
17	糜利栋, 顾少华, 薛亮, 等. 基于数据驱动技术的智能试井解释方法——以有水气藏产水气井为例[J]. 天然气工业, 2021, 41 (2): 119- 124.
18	XUE L , GU S , MI L , et al. An automated data-driven pressure transient analysis of water-drive gas reservoir through the coupled machine learning and ensemble Kalman filter method[J]. Journal of Petroleum Science and Engineering, 2022, 208, 109492. DOI
19	李道伦, 刘旭亮, 查文舒, 等. 基于卷积神经网络的径向复合油藏自动试井解释方法[J]. 石油勘探与开发, 2020, 47 (3): 583- 591.
20	LIU X , LI D , YANG J , et al. Automatic well test interpretation based on convolutional neural network for infinite reservoir[J]. Journal of Petroleum Science and Engineering, 2020, 195, 107618. DOI
21	周子琪. 基于深度学习的试井和测井资料解释方法研究[D]. 合肥: 合肥工业大学, 2021.
22	DONG P , CHEN Z , LIAO X , et al. Application of deep learning on well-test interpretation for identifying pressure behavior and characterizing reservoirs[J]. Journal of Petroleum Science and Engineering, 2021, 109264.
23	ZHANG Q , ZHU S . Visual interpretability for deep learning: A survey[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19 (1): 27- 39.
24	SUTTON R S , BARTO A G . Reinforcement learning: An introduction[M]. Cambridge: MIT Press, 2018.
25	LILLICRAP T P , HUNT J J , PRITZEL A , et al. Continuous control with deep reinforcement learning[J]. arXiv Preprint arXiv: 1509.02971, 2015,
26	HOURFAR F , BIDGOLY H J , MOSHIRI B , et al. A reinforcement learning approach for waterflooding optimization in petroleum reservoirs[J]. Engineering Applications of Artificial Intelligence, 2019, 77, 98- 116.
27	HE J , TANG M , HU C , et al. Deep reinforcement learning for generalizable field development optimization[J]. SPE Journal, 2021, 1- 20.
28	GUEVARA J L , PATEL R . Optimization of steam injection in SAGD using reinforcement learning[J]. Journal of Petroleum Science and Engineering, 2021, 108735.
29	ZHANG Y Y , LI W W , CHEN H J . Reduced-order model based on cluster analysis in numerical simulation of low permeability reservoir[J]. Chinese Journal of Computational Physics, 2015, 32 (5): 603- 609.
30	HUANG C , TIAN L , WANG H L , et al. A single well production forecasting model of reservoir based on conditional generative adversarial net[J]. Chinese Journal of Computational Physics, 2022, 39 (4): 465- 478.
31	TANG M , LIU Y , DURLOFSKY L J . A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems[J]. Journal of Computational Physics, 2020, 413, 109456.
32	XIAO C , ZHANG S C , MA X F , et al. Model-reduced autoregressive neural network for parameter inversion[J]. Chinese Journal of Computational Physics, 2022, 39 (5): 564- 578.
33	ZHONG Z , SUN A Y , REN B , et al. A deep-learning-based approach for reservoir production forecast under uncertainty[J]. SPE Journal, 2021, 26 (3): 1314- 1340.
34	KINGMA D P , BA J . Adam: A method for stochastic optimization[J]. arXiv Preprint arXiv: 1412.6980, 2014,
35	MNIH V , KAVUKCUOGLU K , SILVER D , et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529- 533.

[1]	肖聪, 张士诚, 马新仿, 周彤, 侯腾飞. 基于模型降维和递归神经网络的油藏参数反演[J]. 计算物理, 2022, 39(5): 564-578.
[2]	刘福平, 王安玲, 杨长春. 地层渗透参数反演计算[J]. 计算物理, 2013, 30(6): 879-885.
[3]	孙维志, 韩华. 用进化策略方法反演二维弹性波动方程的参数[J]. 计算物理, 2002, 19(6): 544-548.