A major oil and gas company works in a volatile market, where IoT data floods in at the rate of gigabytes per second – and where up-to-date financial projections, generated by a .NET application, serve as a major competitive advantage. After considering MongoDB and Oracle Exadata, they replaced two NoSQL databases at the core of their analytics architecture – MapR and ElasticSearch – with SingleStore. After a fast implementation, they now run financial forecasts multiple times a day, with much greater performance, with many fewer servers, and at much lower cost than with competitors. The company now also uses SingleStore for a wide range of financial forecasting applications, land contracts analysis, ERP reporting, and more.
Finding, extracting, and delivering oil and gas is one of the most challenging businesses in the world. And it is also, in normal times, one of the most lucrative businesses in the world. But the recent crash in the price of oil, along with a collapse in demand caused by the coronavirus epidemic, have exerted unprecedented pressures on this industry.
But oil and gas has always been a tough business. One of the biggest challenges, and opportunities, is financial. Well-run companies use mountains of data to help them manage their reserves of available capital. With good access to capital, when things look good, these companies invest in turning on existing supply, and exploring for new reserves. When times are tough, they reduce production and hold tight until the next opportunity.
Players which are poorly run, or which are too small to compete effectively, tend to miss out on some of the peaks and get caught short in the valleys, driving them to sell out, or go out of business. The assets of the weaker company are then purchased by one of their better-run competitors, which thereby grows larger, and – to the extent that the company is run well – more resilient.
A large oil and gas company has built an extensive Internet of Things (IoT) deployment across its physical plant of wells, drilling rigs, and pipelines. They also use a wide range of external data feeds, and mix this internally generated and market information to make smart decisions. However, a few years ago, their IT infrastructure was not up to the task.
They replaced the databases at the core of their infrastructure with the SingleStore database platform. The primary use case for this newly powerful engine is to constantly adjust the company’s financial forecasts. Management is in constant touch with everyone from crews in the field, to lenders and analysts in financial capitals, while scrutinizing roughly 100 major variables at a time. With all of this input, they are constantly creating and stress-testing potential new budgets. All of this forms a company-wide risk exercise, with outsize rewards for success, and severe business consequences for failure.
In this case study, we’ll describe in detail the business problem caused by the previous architecture; what alternative solutions were considered; and how SingleStore helped solve the problem. Now, using SingleStore, the company stands out from competitors, stays ahead of the market, and grows, through the best and the worst of times.
This is likely to be of interest to other oil and gas companies; other companies that use IoT in daily operations; companies that need financial analytics and budgeting flexibility, across large and changing data sets; and any company that needs to speed up the flow of information, to meet the current wave of business challenges.
Several years ago, the oil and gas company described here, which is based in Texas, approached SingleStore. This Fortune 500 oil and gas company is publicly held, with tens of billions of dollars in annual revenues and thousands of employees. They actively explore for hydrocarbons – oil and gas – predominantly from shale. Shale sources are notable for being on the expensive side to develop, but are also more easily turned off and on for production than traditional sources of oil and gas. This means they can be actively managed, differently than traditional sources – and elicits the need for constant flows of information to get the best use from the shale resources.
The company uses the data from sensors across the drilling process for real-time well production analysis. They combine this with a wide range of live, always-on, internal and external data feeds to embed themselves in financial and informational data flows. The company can use these massive data flows for a wide range of purposes:
The company uses these data flows to reduce non-productive time (NPT) and cut costs – and also to tap capital markets and take advantage of opportunities. But the company’s ability to bring in and process all of this data was severely handicapped by slow and limited data processing operation.
Because of this, decisions that could have saved the company millions of dollars a week – or made the company even more money – went unmade, as the data was not available when and where it was most needed.
Existing resources were weak at every point:
The business saw that its weakest point was financial forecasting. If they could integrate all this data to generate solid budgets, and test them against different scenarios so as to best manage risk, they could get ahead of aggressive competition.
After careful study, a SingleStore database was chosen, to speed ingest, processing, and the delivery of results, with increased concurrency. The new system was scoped, deployed, and brought into production in just seven months. Costs are down; flexibility, manageability, and profitability are up.
The Business Problem
The company is constantly looking for new customers and negotiating with existing ones. Like any public company, they need to meet detailed financial reporting requirements, and to accurately forecast revenues, costs, and profits and losses – per region, and then roll these budgets up for the company as a whole.
None of this is easy. Shale oil company operations are heavily dependent on prices of conventional oil and gas, as production costs must be below the commodity price for a well to be profitable. These costs include drilling and completion costs, delivery costs, operating expenses, taxes, and more. Companies use decline curves to look at the rate of production decline over the life of a well, and work to determine what the estimated ultimate recovery (EUR) will be for a given well. Getting the economics right – not quarter by quarter, but day by day – is a necessity for these companies, given the impact of changing commodity prices.
Financial forecasting for an oil and gas company
depends on a constantly fluctuating mix of factors.
depends on a constantly fluctuating mix of factors.
When the price of oil is high enough, a producing well can be quite lucrative. Today’s Digital Oilfield uses IoT-enabled devices, sensors, and systems to reduce costs, improve performance, enhance safety, minimize downtime, and drive operational efficiencies. Maintaining extremely complex and expensive equipment, using predictive analytics to do preventive equipment maintenance, and managing the supply chain are all areas that have been improved with IoT.
As a result of the company’s IoT investment, the business is able to manage, and centralize the flow of information from, the myriad of sensors on the drilling rig. Instead of managing each sensor independently, the company can now look holistically at everything from flow rates to temperature and pressure to mud pulse telemetry signals. They are able to profile potential sites extensively and choose which ones to develop. And they can use real-time data to decide when to take rigs online and offline, and when to operate existing wells.
The cost of money – borrowing to fund capital requirements – is another variable. Each region where the company operates wells has its own forecasts and budget. Modeling to support the forecasting and budgeting process for each region is a huge challenge, due to the inherent difficulties of the business, and the sheer number of variables involved (more than 1,000, in many decisions).
Managing and budgeting resources is a critical business requirement. Having the right data infrastructure to power the necessary decision-making processes, and the right people to run the infrastructure, is just as important as having the right equipment and the right people in the field.
With the oil and gas company’s previous infrastructure, they were seeing more than 2x swings in budgetary allocations. That is, the size of opportunities – and the need for funds to pursue them – could easily fluctuate by more than a factor of two during a quarter. The business lost money due to unoptimized decision making, and lost opportunities that they could not see coming up, since the business could not reallocate funds quickly enough. Further, the company could not predict earnings accurately. All of this hurt the company’s financial results.
The Technical Challenge
The company had a wide range of data sources, which overloaded their existing system; growing processing requirements, which were not being met; and ever-increasing analytics demands, both in volume of queries and in number of users. Desired service level agreements (SLAs) for response time were dropping, in the face of stiff competition; SLAs for concurrency were rising, as human analysts, BI tools, financial dashboards, applications, and machine learning models compete for analytics access.
The company used a combination of MapR to accumulate incoming data and ElasticSearch to power analytics. Internal opinions of the system were poor. One member of the assessment team described MapR harshly: “It’s a hole in the ground. You put your stuff in it and light it on fire.” MapR Technologies, the company behind this product, was sold last year to Hewlett-Packard Enterprise (HPE).
Before: MapR and ElasticSearch powered analytics and apps.
ElasticSearch is “a distributed, open source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured,” built on Apache Lucene. ElasticSearch is valuable for use with mixed data, but it’s a NoSQL solution, about which we’ve expressed our opinion before. Relational databases that support SQL tend to be much faster for running analytics, per machine, as well as far more compatible. NewSQL solutions, such as SingleStore, take this efficiency and scale it smoothly across distributed clusters of generic servers.
Faced with the poor results they were experiencing, the company considered four alternative solutions, in addition to SingleStore. One of the alternatives, Oracle, is a relational database solution; the other three are NoSQL solutions. The alternatives to SingleStore were quickly found wanting:
Solving the Problem
The oil and gas company realized they had a big problem, deserving of investment at the company level. So they started by setting out the requirements for a solution:
SingleStore was considered carefully, and was found to not only solve the core business problem, but several additional problems as well. Here are key technical characteristics of the SingleStore solution:
In contrast to the previous use of NoSQL solutions (Elasticsearch and MapR), SingleStore delivered the familiarity of SQL, the ACID capabilities inherent to a relational database management system (RDBMS), and the performance and scalability of a distributed system – an attribute of SingleStore that is generally found in NoSQL solutions, and not generally found in truly SQL-compliant systems, but a key part of what SingleStore delivers.
Price-performance has proven to be excellent. The closest competitor to SingleStore, in terms of capabilities, was Oracle Exadata. But SingleStore consistently excels over Oracle on many attributes, especially price-performance. Many organizations achieve as much as ten times the performance of Oracle at roughly one-fourth the cost.
After: SingleStore powers analytics and apps
The business benefits that the company has gained with the move to SingleStore are substantial, and becoming even more significant with time:
The move to SingleStore has contributed to strong earnings and profitability, as shown in the company’s market capitalization and share price – which consistently rises in good times, and stays ahead of the competition when times are tough.
After the first successful IoT project, this leading oil company has greatly expanded the use of SingleStore across a wide range of applications: additional financial forecasting use cases; land contracts analysis; ERP reporting; and more.
The adoption of SingleStore at this leading oil and gas company led to many breakthroughs; improved performance, a simpler system design, new capabilities, and a new data backbone for a wide range of applications. Today, SingleStore is being considered for ever-broader use within the company.