Reinforcement Learning, from Games to Geologic Interpretation

Source: Deep Learning on Medium

At GSSI (Geo-Steering Solutions Inc.) we develop software that helps ourselves and customers steer horizontal wells through complex subsurface geology. We drill horizontal wells, like the one shown below, in order to maximize reservoir production while minimizing drilling costs and environmental impact.

To give you some scope, realize that the above diagram shows a well that was drilled about 2 miles beneath the earths’s surface and 1 mile long. All by using a tool to essentially listen to the natural gamma signature from the rock. This information is then sent from the drillbit in a form of Morse code, through the drilling fluid. These pulses provide a response curve that is able to measure characteristics of the rock. The geo-steers (all trained geologists), then use our software, Geo-Direct, to help them model the subsurface geology and steer the drillbit through the target rock formation. Certainly, no easy feat, and a task often complicated by poor or lost data. However, the task is 100% data driven and therefore perfect for machine learning.

Enter Machine Learning

Data science and geoscience go hand in hand and have done so for years. I christened my first geoscience machine learning project in 2006 which used Monte Carlo and genetic algorithms to provide analysis of risk and perform back calculations. Through this experience and many others, we at GSSI have been able to quickly build several machine learning and automated processes for post analysis of wells. From using machine learning to visually identifying and recognize wireline logs/curves to predictions of geological tops at 99.999%. We are still pushing to get real-time tops and re-steering capability into our software the end of the year using deep learning and deep reinforcement learning. An example of reinforcement learning being used to play games is shown in the video below:

Deep Reinforcement Learning

The AI technology that is playing the above game rocked the tech world in 2015 and brought to the forefront the power of reinforcement learning. Reinforcement learning is a deeply specialized area of machine learning that is based on control theory and animal behavior. In the example game above, all the algorithm or agent sees is the game’s screen shots. Then through a process of trial and error reinforced by rewards, the agent incrementally learns to play the game better than humans. In fact, the agent is able to develop it’s own winning strategy, as shown in the video, in order to do so.

At GSSI we have already worked with the more cutting edge deep learning architectures, from convolutional neural networks to sequencing recurrent convolutional neural networks. All with the hopes of identifying characteristic markers or features in the data steaming from the rig. While identifying these visual markers on their own is relatively easy, the issue becomes one of context or understanding past events. Often times, a geologist is only able to identify a marker by understanding what type of rock or markers that may have come before it or after it. Keeping in mind that as the well is drilled it is possible for the rig to steer in and out of the targeted rock formation or for the geology itself to fault, dip or slip. With all these factors at play, reinforcement learning makes perfect sense as the optimum and perhaps only solution.

Progress Thus Far

Below is a simple video/gif that shows some of our more recent experimentation. The example shows an out of the box reinforcement learning tool from Unity, called ML-Agents, being applied to picking geological tops. We essentially built a game that challenges geologists to identify geological tops from a strip log. Then we used the same ‘game’ to train an AI to do the same thing. Here are our amazing results, thus far:

Of course we have a lot of work left to do, but the above demo only took about a week to build and customize. With the advent of more and more powerful Machine Learning technologies we are sure these results will only get better.

Distributed AI Microservices

Going forward, GSSI is working to improve it’s general overall experience with machine learning data management and creating a set of geological distributed AI microservices. Creating a collection of unique services allows us to layer and deploy these AI services in various configurations, on different infrastructure and across our cloud service or the customers. Here is a snapshot of the services we are looking to develop within the coming year:

Geo-Direct is our existing software that our geo-steering geologists and customers use 24/7 to steer wells. On top of this we initially plan to release the areas in dark blue within the next 6 months. This will include the Trainer, Well Analyzer and Geo-Agent for tops picks, warnings and re-steering. The Geo-Agent Training module will allow our staff and customers to train deep learning networks on specific basin data, kept in our proprietary well file format and/or using wireline LAS logs. This will allow customers to build their own deep learning models with their own data or purchase general basin models from GSSI. The WellAnalyzer module will allow customers to analyze well steering performance on multiple wells in order to provide insights into optimizing production and directional drilling performance. If you are interested in discovering the capabilities of our other modules please contact GSSI.