This month, the S&P 500 officially hit bear-market territory—meaning a fall of 20+ percent from recent highs—and investors everywhere are looking for some way to predict how long the pain could last.
Machine learning startups specializing in “nowcasting” attempt to do just that, by analyzing up-to-the-minute data on everything from shipping costs to the prices of different cuts of beef. In times of economic volatility, investors and executives have often turned to market forecasts, and ML models can offer a way to absorb more information than ever into these analyses.
One example: Complete Intelligence is a ML startup based outside Houston, Texas, that specializes in nowcasting for clients in finance, healthcare, natural resources, and more. We spoke with its founder and CEO, Tony Nash, to get a read on how its ML works and how the startup had to adjust its algorithms due to market uncertainty.
This interview has been edited for length and clarity.
Can you put the idea of nowcasting in your own words—how it’s different from forecasting and the nature of what you do at Complete Intelligence?
So Complete Intelligence is a globally integrated machine learning platform for market finance and planning automation. In short, we’re a machine learning platform for time series data. And nowcasting is using data up to the immediate time period to get a quick snapshot on what the near-term future holds. You can do a nowcast weekly, daily, hourly, or minutely, and the purpose is really just to understand what’s happening in markets or in a company or whatever your outlook is right now
And what sort of data do you use to fuel these predictions?
We use largely publicly available datasets. And we’re using billions of data items in our platform to understand how the world works…Macroeconomic data is probably the least reliable data that we use, so we use it for maybe a directional look, at best, at what’s happening. Currencies data is probably the most accurate data that we use, because currencies trade in such narrow bands. We use commodities data, from widely traded ones like oil and gold, to more obscure ones like molybdenum and some industrial metals. We’re also looking at individual equities and equity industries, and we track things like shipping times for goods—shipping times…are usually pretty good indicators of price rises.
Who are your clients, and how are the nowcasts used in practice?
Our clients range from investors and portfolio managers, to healthcare firms and manufacturing firms, to mining and natural resources firms. So they want to understand what the environment looks like for their, say, investment or even procurement—for example, how the current inflation environment affects the procurement of some part of their supply chain.
In fact, we’re talking to a healthcare company right now, and they want to nowcast over the weekend for some of their key materials. In an investment environment, of course, people would want to understand how, say, expectations and other variables impact the outlook for the near-term future, like, days or a week. People are also using us for continuous budgeting—so revenue, budgeting, expenses, CFOs, and heads of financial planning are using us…to understand the 12- to 18-month outlook of their business, [so they don’t have to have an annual budgeting cycle].
Tell me about how the AI works—which kinds of models you’re using, whether you’re using deep learning, etc.
There are basically three phases to our AI. During the pre-process phase, we collect data and look for anomalies, understand data gaps and how data behaves, classify data, and those sorts of things.
Then we go into a forecasting phase, where we use what’s called an ensemble approach: multiple algorithmic approaches to understand the future scenarios for whatever we’re forecasting. Some of those algorithms are longer-term and fundamentals-based, some of them are shorter-term and technical-based, and some of them are medium-term. And we’re testing every forecast item on every algorithm individually and in a common combinatorial sense. For example, we may forecast an asset like gold using three or four different forecast approaches this month, and then using two forecast approaches next month, depending on how the environment changes
And then we have a post-process that really looks at what we’ve forecasted: Does it look weird? Are there obvious errors in it—for example, negative numbers or that sort of thing? We then circle back if there are issues…We’re retesting and re-weighting the methodologies and algorithms with every forecast that we do.
We’ve had very unique market conditions over the past two years. Since AI is trained on data from the past, how have these conditions affected the technology?
You know, there’s a lag. I would say that in 2020, we lagged the market changes by about six weeks. It took that amount of time for our platform to catch up with the magnitude of change that had happened in the markets. Now, back then, we were not iterating our forecasts more than twice a month. Since then, we’ve started to reiterate our forecasting much more frequently, so that the learning aspect of machine learning can really take place. But we’ve also added daily interval forecasts, so it’s a much higher frequency of forecasting and in smaller intervals, because we can’t rely on, say, monthly intervals as a good input in an environment this volatile.
Complete Intelligence – a fully automated and globally integrated AI platform for smarter cost and revenue planning.
Complete Intelligence provides actionable, accurate, and timely data to make better investment and procurement decisions.
The platform provides an integrated global model to ensure that actions in one market, country, or sector of the economy are reflected elsewhere in markets, industries, and the global economy. International trade, economic indicators, currencies, commodity prices, and equity indices are all factored in to create a proxy of the global economy. Over 1200 industries in more than 100 countries are covered!
Based on interviews with Tony Nash, founder, CEO, and Chief Data Scientist, this brief report introduces Complete Intelligence, one of a growing number of highly innovative companies supported by the Oracle for Startups program. The company, founded in 2019, is already significantly improving the forecasting and budget planning of a variety of large corporations through its advanced AI-driven intelligence platform. The theme for this month is around startups in the energy and utility sector and how they are innovating, changing the competitive landscape, and contributing to sustainability. CX-Create is an independent IT industry analyst and advisory firm, and this report is sponsored by the Oracle for Startups program team.
The business context for Complete Intelligence
Commodity price volatility and a post-pandemic surge in demand drive the need for more timely and accurate forecasting Businesses coming out of lockdown have increased demand for commodities, from energy supply to raw materials for their products. In Europe, benchmark prices for natural gas to power their factories and heat their buildings have risen from €16 megawatt-hour in January 2021 to €88 in October. This, in turn, has sent electricity prices soaring. (Source: Euronews). While some have locked in prices through forward-buying, others have been exposed and seen profit margins plummet, unable to pass on price hikes to their customers.
But it is not just energy prices that are volatile. Semiconductor chip shortages have impacted many industries that depend on them, from automotive to electronic household goods manufacturers, putting a brake on their post-pandemic recoveries despite strengthening demand.
The growing demand for clean and sustainable energy sources and precious metals, like copper and lithium that power batteries have also seen tremendous volatility. As major industrial companies digitally transform their organizations and business models seeking elusive growth, the importance of data and AI are increasingly recognized as fundamental to success.
Forecasting and budgeting needs data science, not spreadsheets The ability to sense change, respond quickly and adapt rapidly relies on a synthesis of massively increased volumes and varieties of data, both from operational and external sources. Data volumes are too complex for manual approaches and spreadsheets and require AI to extract insight and meaning from this complex array of external demand and supply signals. The old industrial-age planning approaches can’t cope. They are too slow, involve armies of accountants and analysts, and political wrestling between departmental heads, and are often based on opinion and inaccurate forecasts leading to erroneous budgeting decisions.
Complete Intelligence provides the accurate evidence base for budgeting and forecasting decisions
When markets are relatively calm and stable, the cycle of annual planning and budgeting makes sense. But amidst continual volatility and dramatic accelerated change, the planning cycle is too slow. It fails to mitigate the risks unfolding at such speed and is impacted by a confluence of so many variables, like extreme weather, scarcity of raw materials, pandemics, and weakened supply chains. An array of intelligent internal and external feedback loops is needed to mitigate risks and optimize resources in pursuit of the company’s goals. This is what Complete Intelligence provides with its integrated and modular intelligence platform.
Key observations
• Complete Intelligence provides the accurate evidence base for budgeting and forecasting decisions • The Complete Intelligence Platform consists of three modules – CI Futures, RevenueFlow and CostFlow • Forecast accuracy has rapidly improved, and error rates are now around 2%, which compares favorably with traditional methods and error rates of 35% or more
Complete Intelligence, the story so far
Tony Nash, founder, CEO, and Chief Data Scientist, is steeped in market intelligence. A former VP of market intelligence firm IHS (now IHS Markit), and The Economist Intelligence Unit, where he was Global Director Consulting and Custom Research. He observed that large international companies he had supported typically followed an annual budgeting cycle based on often inaccurate or opinion-based data. It was not unusual to find large teams of people, sometimes several hundred involved in the process and heavily reliant on gathering data from multiple departments in complicated spreadsheets. The process could last several months, and the variance between forecasts and actuals was often above 35%, which could erode profits or tie up resources unnecessarily.
Trial, error, and persistence As a data scientist familiar with cloud technologies, he developed algorithms to improve forecast accuracy and a complete process from data ingestion to forecasting and testing the results. He started developing the machine learning ML algorithms in 2017 while still consulting in Asia from his base in Singapore. His first iteration failed to produce a level of accuracy that would provide a sufficiently compelling proposition. He wanted to get down to an error rate of no more than 5%-7%. He adopted the ‘ensemble’ approach covering thousands of different scenarios layering external data on commodities such as the copper price with a customer’s actual costs, identified in their general ledger.
Ready for launch late 2019 In 2019, Nash returned from Singapore and set up his company in The Woodlands, near Houston, Texas. He continued his work on the algorithms and developed a commercial product ready to launch in early 2020. And then Covid-19 struck.
Through Covid-19, companies first tried to understand the changing environment, then remained risk-averse until public health, business environment, and supply chains became more stable. This has been a challenge for a cutting-edge machine learning firm like Complete Intelligence. It is only as the environment has begun to stabilize that enterprises have sought new solutions to legacy problems. With that has come a renewed interest in Complete Intelligence and deployment at a large scale.
Solution overview The Complete Intelligence Platform consists of three modules The Complete Intelligence Platform hosted on Oracle Cloud Infrastructure (OCI) consists of three forecasting modules:
•CI Futures – to forecast market trends. Covering over 1,400 industries in more than 100 countries and a database of over 16 billion data points from proprietary and publicly available data. Millions of learning algorithms are used, which factor in the most recent global events.
• RevenueFlow – provides accurate results for demand and forecast sales and revenue projections.
• CostFlow – to enhance product line profitability and improve supply chain and procurement outcomes.
Figure 1. provides a diagram of the Complete Intelligence Platform
Figure 1: Complete Intelligence Platform by Complete Intelligence.
Market data is ingested from multiple trusted data sources like national statistical agencies, multilateral banks, multilateral government bodies, commodities exchanges, bilateral trade bodies and combined with the client’s data from their general ledger. A multi-layer testing and validation process used to ensure the accuracy of the data to be used in any forecast. Third-party data is gathered via internet spiders and APIs.
The platform provides an integrated global model to ensure that actions in one market, country, or sector of the economy are reflected elsewhere in markets, industries, and the global economy. International trade, economic indicators, currencies, commodity prices, and equity indices are all factored in to create a proxy of the global economy.
A comprehensive list of futures, currencies, and market indices is covered and accessed through a highly graphical and easy-to-use interface. Almost 1,000 assets, with historical data from 2010 and forecasts over a one-year horizon, are provided. More assets are being added all the time.
The platform is designed around three attributes: • A globally integrated model • A data-driven process without human intervention in the output • A simple means of interfacing with the platform.
The platform can be connected to existing ERP systems and automatically upload pricing data from the general ledger at a very granular level for each item.
The Complete Intelligence Platform supports a variety of use cases: • Supply Chain & Purchasing Optimization – help lower costs, anticipate risks, and provide input to sourcing strategies. • Sales and market entry strategies – by identifying higher growth markets and optimizing resources • Strategic Financial Planning – identifying growth markets and fine-tuning resource allocations in each market to minimize exposure to currency fluctuations. • Mergers and acquisitions – provide a snapshot of cost structures and projections of future costs and profitability of target acquisitions.
Forecast accuracy has rapidly improved, and error rates are now around 2% Nash’s persistence has resulted in significant levels of forecasting accuracy. A twelve-month forecast now sees error rates around 2%, which gives users considerable confidence compared with traditional methods, where the error rates are often above 35%.
As well as dramatically improving forecast accuracy on markets, revenues, and costs, the onboarding process to going live is a matter of a few weeks. After that, forecasting takes hours, not months.
Current position
Successes to date
While still a relatively new company, Complete Intelligence has already proved its value to several large companies.
• A major petrochemical company wanted to improve its predictive intelligence capability for feedstocks and refined products. They asked Complete Intelligence to examine nine categories across crude oil, gasoline, diesel, natural gas, and gas-to-liquid (GTL) products. Monthly forecast averages are provided by category with extremely low differences from actual results on the order of 3% or less.
• A global furniture company wanted a more explicit link between their sales and revenue planning and their sales teams in China. Complete Intelligence built a sales forecasting model that more clearly identified and utilized market demand drivers and connected these directly to their business. An analytics-based approach to identify the drivers of sales by city and industry. Complete Intelligence built a city and industry-level forecasting tool that determined the company’s growth trajectory and provided recommendations to support the direction and transition of their sales teams. • A global chemicals company needed a better understanding of the trends for costs in their supply chain and a more precise way to manage margin expansion and contraction at the bill of material level. Complete Intelligence was commissioned to forecast factor inputs and currencies for the key categories. The forecasts were calibrated based on the component make-up of the bill of materials. This enabled the client to identify the direction of the materials pricing and the impact on their BOM. Through the process, the client learned how to anticipate cost movements and protect margins.
Current go-to-market model
Complete Intelligence sells directly to large organizations, mainly targeting CFOs and COOs with a broad view of their companies and strategic decisions.
The company also has strategic partnerships with Microsoft and is listed on the Azure Marketplace and with Oracle as part of the Oracle for Startups program and hosted on OCI.
Other partnerships with Bloomberg and Refinitiv allow for exchanging financial and market data and connection to their platforms.
More transparent accuracy reporting so customers can view accuracy/error for every line item
More robust and flexible data visualization for clients to utilize Complete Intelligence forecasts within their visual narratives
More sophisticated data science to account for detailed sentiment and other qualitative factors
Do-it-yourself forecasts for customers to do ad hoc forecasts for any data at any time. This will enable teams within a company to do their own sophisticated, reliable forecasts without waiting on their in-house market analysis or forecasting team with complicated macros and massive spreadsheet workbooks
Embedding Complete Intelligence forecast APIs into ERP and accounting software.
Oracle Cloud Infrastructure and the Oracle for Startups program prove their value to Complete Intelligence When asked what he felt about the relationship with Oracle and the Oracle for Startups program, Nash said, “Oracle Cloud Infrastructure is very flexible and secure. The Oracle for Startups team has been great. Oracle has been the most responsive and helpful of all our partnerships, connecting us to the right people to help with marketing, sales, or technical questions. I really feel that they want us to succeed. I’m a huge advocate of the Oracle for Startups program.’’
CX-Create’s viewpoint The Complete Intelligence Platform addresses a fundamental business need
Providing a global proxy model on markets, commodities, currency fluctuations, and many other aspects and making this easily accessible for business people will significantly improve strategic investment and procurement decisions. The emphasis on accurate and timely data supported by ML models will make it easier for business people to make informed decisions, stripped of personal bias. Digital transformation should lead to a more agile and responsive organization. The more progressive organizations will want highly attuned external signals that are constantly updated, enabling them to de-risk investment decisions and optimize resources for growth. Complete Intelligence provides for that.
Complete Intelligence is mentioned in this article by digital innovation expert Geoffrey Cann. You can find the first and original version of this at https://geoffreycann.com/transforming-capital-projects-using-digital/. We thank Geoffrey for including us in this valuable piece that helps oil and gas companies in modernizing their operations and technologies.
The oil and gas industry spends hundreds of billions each year on new capital projects. An effort by a group of international producers should eventually improve the efficiency of that spend.
DIGITAL CAPITAL
I was contacted recently by a trade association representing about 40% of the global production of oil and gas to discuss the role of digital innovation in upstream capital. Their brief states that while most oil and gas companies have programs in place to progress their internal digitalization agenda, some initiatives need to be tackled at the industry level to unlock value at scale. An example of an efficiency opportunity with industry-level appeal is the digitalization of the supply chain.
While their aim is to focus initially on capital projects, it’s probably safe to assume that the initiative will move to other areas of interest in time.
This post summarizes the survey that I submitted in response to the survey.
Question 1 — Scope of Digitalization
What are the key areas that you think of as being part of a Digitalization agenda?
RESPONSE TO SCOPE OF DIGITALIZATION
Rather than listing off a random set of possible digital technologies to frame the scope of digital, I set out the key elements of my digital framework which also incorporates infrastructure and work processes areas as integral to a digital game plan.
Question 2 — Business Impacts
How do you see Digitalization impacting Major Projects in the Oil and Gas Industry? What are your thoughts on the impact on key Capital Project areas?
RESPONSE TO BUSINESS IMPACTS
Oil and gas capital projects have slipped backwards in terms of productivity gains while most other industry sectors have advanced. At the LNG18 event in Perth in 2016, Shell presented their analysis which shows oil and gas capital has declined in productivity by 25% over the preceding decade whereas most other sectors had gained. The upside for capital is to capture this loss of productivity, and to catch up with other sectors (leading to an outsized gain potential).
Oil and gas spends hundreds of billions per year in capital. The IEA estimates that oil and gas stands to gain a minimum of 20% productivity improvement and 20% cost reduction through digital. The opportunity is in the range of $100B in cost savings, and $100B in capital avoidance. Substantial carbon emissions stand to be avoided. Every aspect of the capital cycle is able to leverage digital tools to capture these savings.
I contributed to a confidential government study in Australia that set out to understand how the competitiveness of their LNG sector could be improved. The modelling showed that a 25% reduction in schedule (from 4 years to 3, for example), would reduce the break even cost of a typical project by $1 per million British thermal units (MMBTU) for 20 years. To give a sense as to what this means, a 9 million ton LNG plant ships 441 trillion BTU per year. Do the math.
CAPITAL STRATEGY
New securitisation technologies (distributed ledger) could be used to transform capital access, and create a new capital asset class. New government crypto currencies (China, EU) may allow for capital market access that avoids US banking system and related sanctions abilities.
RISK ANALYSIS
Advanced ML tools can provide much better predictability to underlying volatile commodity assets (currencies, carbon, hydrocarbons, cement, steel, etc). See company Complete Intelligence. Better predictability to commodity risk can lower project capital costs and improve purchasing strategy.
SCHEDULING AND PROJECT CONTROLS
The industry routinely produces digital twins of operating assets, but how about creating a digital twin model of the schedule? Another possibility is the use of game tools to create the “game” equivalent of a capital project (see Real Serious Games), used for schedule tuning and post build auditing. Cloud computing can help create deeper virtual environments that span entire supply chains, not just one link at a time, so that schedule and carbon impacts can be visible.
ENGINEERING
It’s practically here, but the use of robotic tools to automate routine engineering work is still nascent. Data visualization tools can assist with engineering reviews (see Vizworx) across disciplines and suppliers, provided data is normalized. Open data standards can enable industry cooperation (see OSDU). Deeper virtualisation of teams working across time and location boundaries is enabled by cloud computing, digital twin tools, collaboration systems (zoom, slack). Finally, blockchain tools can be used to capture document versions, protect IP.
CONTRACTING
Some companies already use AI to read/interpret contracts, flag areas for review. Bot technology can then conduct alerts, notifications, payments using blockchain interface (smart contracts).
PROCUREMENT
The industry can leverage entirely new supply models for common procurement (see The IronHub). Blockchain technology can be used to track carbon content and asset provenance throughout the supply chain during sourcing, fabrication, and mobilization.
ON-SITE EXECUTION
There are already examples of robots being used on project sites to facilitate work execution—drones for visual inspections in both aerial and subsea applications. Advanced measurement tools are starting to close the gap between engineering and fabrication (see Glove Systems), which is handy when fabrication is modularised and distributed to multiple global shops. Leading companies create the digital twin of civil site works (see Veerum), allowing for continuous monitoring of site performance, and analytic tools to improve execution, reduce carbon. Safety analytics can identify and predict emerging safety hazards.
DIGITAL COLLABORATION
Large projects will leverage cloud computing to enable single source of truth about capital projects.
WORKFORCE MANAGEMENT
With most workers now carrying one or two supercomputers on their person, industry can now bring valuable data directly to the worker. Two-way collaboration using cameras and audio can connect workers to supervisors, sites to suppliers, builders to engineers. Game tools can be deployed to show individual performance (safety, time on tools) compared to team, shops, fabricators, best teams, best practice (See EZOPS).
MATERIAL MANAGEMENT
Blockchain technology is already in use in supply chains to provide for track and trace of materials in support of warranties, product specifications, certifications (see Finboot) to tighten compliance.
Question 3 — Longer Term Impact
How do you see Digitalization impacting the overall Oil and Gas Industry over the next 10 years?
RESPONSE TO LONGER TERM IMPACT
In my book, I set out the substantial headwinds to the oil and gas industry (decarbonization efforts, capital constraints, talent shortfalls, environmental activism, competitive alternatives for transportation). Digital innovations are the only known solution that addresses these cost, productivity and carbon concerns simultaneously.
Technology companies supplying the industry are already rapidly adopting digital tools to stay competitive. Brownfield assets are going to slowly adopt digital tools because of operating constraints (short outage windows to make change, management of change process). Capital projects have the opportunity to drive change precisely because they are greenfield, and specifically the short duration capital cycles in unconventional areas.
Over the next ten years I expect to see some oil and gas companies distinguishing themselves with new business models that are digitally led. With its substantial spend, oil and gas companies could become one of the leading advanced digital technology industries globally.
Question 4 — Key Drivers for Digital
What do you see as the key drivers and value areas behind a Digitalization program?
RESPONSE TO KEY DRIVERS FOR DIGITAL
There are many drivers for digital innovation, but here are four that are at an industry level.
TALENT.
The industry is at risk of becoming unattractive to talent (the Greta Thunberg effect). People in oil and gas are falling behind in companies that are falling behind in an industry that is falling behind. Digital tools can make junior resources as productive has highly experienced, as well as make the industry more “high tech” and attractive as an employer.
CAPITAL MARKET ACCESS.
Capital markets are shut off to much oil and gas investment. The top 7 largest companies by market cap are all digital (Amazon, Facebook, Alphabet, Apple, Microsoft, Tencent, AliBaba). Oil and gas has shrunk from 15% of NYSE to less than 5%. Apple alone is now larger than the combined oil and gas majors. Capital markets need to hear a thoughtful strategy about how the industry is embracing digital innovations.
CARBON MITIGATION.
The EU Green deal is driving carbon neutrality targets for oil and gas (see BP, Shell, Repsol). Oil companies and their supply chains will be unable to access markets without thoughtful carbon gameplan (track, measure, monitor).
COST AND PRODUCTIVITY.
Oil and gas spends hundreds of billions per year in capital. The IEA estimates that oil and gas stands to gain a minimum of 20% productivity improvement and 20% cost reduction through digital. The opportunity is in the range of $100B in cost savings, and $100B in capital avoidance. Substantial carbon emissions stand to be avoided. Every aspect of the capital cycle is able to leverage digital tools to capture these savings.
Question 5 — Biggest Challenge
What is the biggest challenge at implementing a Digitalization strategy?
RESPONSE TO BIGGEST CHALLENGE
As I see it, digital is not a ‘technology’ opportunity. It is a culture change opportunity. Oil and gas tends to view digital as something to purchase (buy and do digital), rather than as a lever to drive behaviour change (to be digital). Oil and gas companies underinvest in the necessary change management actions to create the conditions for digital success.
There is an inadequate amount of training on the digital basics for the front line workers who need to embrace this unknown technology. A reliance on engineering water fall methods of work instead of agile methods undermines the speed by which digital change can take place. By underinvesting in the user experience side of change, and placing the asset at the center of digital efforts, the industry increases the resistance to technology.
Question 6 — Foundational Capabilities
What foundational capabilities do you feel need to be in place for O&G companies to fully exploit Digitalization?
RESPONSE TO FOUNDATIONAL CAPABILITIES
I cover much of this in my book. For example, IT and OT need to be merged into a single organization. Systems need to be cloud enabled as much as possible. Enterprise solutions (SAP, Maximo) need to be upgraded to their digital versions (so that they do not block other digital efforts). An experimentation capacity to run digital trials must be in place. Funding for digital investments must be in place. Clear expectations for achieving desired outcomes (cost, productivity), must be expressed. Methods for doing work must follow agile principles. Better connections to the digital start up ecosystem should be in place.
Question 7 —Investment Candidates
Have you seen any Digitalization initiatives that should be carried out collectively or would be more effective if adopted in a common way across the industry (including the supply chain)?
RESPONSE TO INVESTMENT CANDIDATES
OSDU is a powerful illustration for enabling sub surface data management and exchange to accelerate the adoption of digital in the upstream. Something like this for capital projects would be valuable. The OOC is demonstrating the power of community of collaboration to drive blockchain-enabled initiatives forward.
CLOSING THOUGHTS
Building assets that last 20 years or more is just the first step in their lifecycle. Digital efforts in Capital Projects should enable must faster and more graceful commissioning and handover. For example, CSA Z662 and PHMSA 192 set out the new materials tracing for linear infrastructure (tubular, pumps, fittings, flanges) which can only be achieved by deploying digital in the capital project. Poor quality data about installed infrastructure destroys up to 40% of value in a transaction (and that data is largely generated and collected during capital spend).
The sooner the industry tackle capital project efficiency the better.
