Enabling Transition: Critical role of analytics and automation in evolving market design

Critical role of analytics and automation in evolving market design

The Indian power market is making strides towards next-level transition, which is likely to be completed in the next three to six years. The focus of this transition is to have competitive market mechanisms and institutions centred ar­ou­nd digitalisation and IT to bring in scale, speed, transparency and standardisation.

Leading the world against climate chan­ge, India is focusing on the mass deployment of clean energy. Solar and wind are the dominating clean energy sources, which come with complexities such as uncontrollable generation and intermittency. With climate change impacting the major weather patterns, solar and wind generation estimation is more complicated. The recent coal crisis triggered by the prolonged monsoon is another effect of climate change. Given the aggressive cl­ean energy transition of the Indian power system, bolstered by the recent commitment at COP26 for achieving net zero carbon emissions by 2070, the power market de­sign has to evolve rapidly to accommodate the transition.

As such, the policy and regulatory environment is driven towards competitive market mechanisms to take up next-generation reforms, by reducing the dependence on long-term power purchase ag­ree­ments (PPAs), accommodating the un­certainty of renewable energy generation, upgrading load despatch operations to handle high renewable energy penetration and scale, introducing new spot markets like real-time market (RTM), green term ahead market (G-TAM), green day ahead market (G-DAM), and market-based deviation settlement mechanism (DSM), and moving towards general network access (GNA) for transmission allocation and pricing, in addition to ma­r­ket-based economic despatch (MBED), long-duration power exchange-based short-term open access (STOA) contracts, de­ri­vatives, market-based an­ci­ll­a­ry mechanisms and capacity contracts.

Conventionally, the wholesale power market is dominated by a few large buyers and sellers with capacities of less than 100 MW and has limited participation from small commercial and industrial pl­a­yers through open access. Transac­tions in the market are largely long-term PPA based (90 per cent), while the rest are do­ne through the short-term mode of collective (DAM/RTM), bilateral and DSM segments, which are more competitive and transparent. High renewable energy penetration brought new dynamics like technology cost changes, intermi­ttency and despatchability, geographical and temporal diversity, installation time, smaller capacity, ancillary requirements, and weather dependency. With a conventional human-centric app­roa­ch, scheduling operations and tra­ding practices are posing as bottlenecks due to the growing scale, speed and interdependence of markets and transactions in the short-term market, reducing dependence on long-term centric transactions. The adoption of big data analytics and automation has be­come a prerequisite. Discoms and utilities are increasingly finding it impractical to navigate the changing market by continuing with the traditional operations and trading methods due to the human limitations of scale, speed and reliability.

Big data analytics has gained a prominent role in all sectors in this digital age. With the power sector operating within the 15-minute time block resolution for energy accounting and sometimes 30-second resolution, colossal data is generated on a real-time basis from millions of supply-demand grid connection points through RTUs and ABT/SEM me­ters and trading and scheduling activities. With the spot and STOA markets gaining a pivotal role in the new evolving market design and the likely launch of electricity derivatives, analytics has become essential for decision-making for trading and planning activities. Analytics is classified into descriptive, predictive and prescriptive analytics, and each has relevance in the power mar­ket, especially in planning, scheduling and trading activities.

The power market is a national market, with hundreds of players. The price volume data points are generated on a 15-minute time block basis. DSM is now linked to DAM prices, while STOA bilateral and banking transactions are benchmarked primarily to DAM prices. With the scheduling of long-term plants taking place on a day-ahead basis in sync with DAM and gate closure for revisions linked to RTM gate closure, the PLF of long-term PPA-based plants is now in the realm of spot market liquidity and prices. Inves­tors have started to factor in spot market prices in deciding the­ir generation in­vest­ments. This re­quires a strong need to understand the past and present influencers of the na­tional market. This un­derstanding must be complemented by descriptive analytics-based tools. This analytics uses data aggregation and data mining to understand and uncover tren­ds quantitatively, thereby gaining valuable insights on the parameters influencing market prices, and taking well-in­formed and strategic decisions in planning and trading activities.

Descriptive analytics also plays a crucial role in the portfolio and energy management activities of discoms, given that the scale of supply-demand side interac­tio­ns, scheduling, regulatory complian­c­es, market mechanisms. Real-time SCADA and meter data makes it compl­ex to understand and navigate the market-centric operations to make the right decisions and serve the load optimally. A unified descriptive analytics-based platform will help understand and assess the techno-commercial aspects of supply and do a post-facto analysis for corrective actions. Real-time big-data analytics tools help manage a unified analysis of all the interlinked techno-commercial parameters of discoms on a near-real-time ba­sis, eliminating lag and aiding both operations and management in effecti­vely understanding their respective po­rtfolio holistically and taking timely informed decisions. Regula­tors can also use such descriptive analytics platforms to monitor and assess the performance of regulated entities in this area on a near-real-time basis. The lack of such descriptive analytics platforms with discoms and generators that primarily rely on the experience-based decisions of operations and trading personnel leads to poor trading and planning decisions. Regulators are ha­ndi­cap­ped with regard to independently monitoring the performance of discoms in trading and planning efficacy and op­timality, that too promptly. With scalable best-in-class cloud-based technology tools for data analytics and business intelligence becoming widely available in recent years, even for small-scale app­lications, power market players can ad­opt these technology enablers for moving to a big-data based quantitative tra­ding and planning paradigm.

Predictive analytics is another key evolving focus area for the market transition, given the increasing share of intermittent renewable generation, as well as spot/short-term market-based transactions, which derivatives will aid in the near future. Predictive analytics builds over descriptive analytics, whereby data is deployed for statistical modelling or machine le­ar­ning (ML) techniques to identify the likelihood of future outcomes based on historical data. ML techniques are increasingly deployed for solar and wind generation forecasting, numerical weather predictions, short-term DAM and RTM price forecasting and short-term demand forecasting. Statistical approaches are prevalent for mid- and long-range DAM and RTM price forecasting and demand forecasting. Numerous predictive analytics app­roa­ches are evolving, with varying de­grees of success in the Indian power ma­r­ket, which has witnessed considerable improvement on all fronts.

Finally, prescriptive analytics supports in providing actional decision-making inputs with result based insights. It draws heavily from descriptive and predictive analytics and uses statistical approaches for providing scenario-ba­sed outcome analysis. The two key areas of power markets are scheduling/dispatch optimisation and derivative contract valuation/market risk assessments. Globally, linear programming-based op­timisation tools are used by load serving entities/discoms and system operators to optimise schedules/dispatch supply for the least cost and trade optimisation.

Given the increased spot market dependence of utilities and the complexity of gate closure and RTM, system operations and scheduling teams will be left with li­mi­ted time to process and generate sche­dules that are market-linked DSM mechanisms techno-commercially compliant, as well as optimised for RTM and DSM. As the NLDC has shown the way in adop­ting optimisation tools for SCED sche­d­uling and the proposed MBED me­chanism, discoms and LDCs should mo­ve towards deploying prescriptive analytics tools for schedu­ling/des­patch optimisation and automated trading decision support systems aimed at the RTM.

Automation is another critical area neglected in power market related areas, wh­ere significant strides were made at the NLDC/RLDC end, power exchanges and, to some extent, at the level of trad­ers. With near-real-time trading and sche­d­uling coupled with a considerable amount of data, diversity of suppliers and market mechanisms, collation of various data points and readings for analysis to make near-real-time decisions, coupled with the need to analyse the extraneous ma­rket ecosystem (as the power market is mainly national level), there is a need to tap 30-100 data sour­ces for a discom, that too with little or no API support from the eco­system players. Hence, for effective de­cision-making, there is a need to do away with manual-based data collation, compilation and communication activities.

With the growing complexity of markets and regulatory compliances, there is a need for automation using IT tools for process automation and communications. Cloud-based automation tools, including robotic process automation, are available even for small-scale applications, which will ease the processing time for the operations team, and thereby provide scheduling and operations teams with error-free information and analysis for decision-making, and thus enhance the productivity and decision-making capability of the team multifold.

By Siddartha Ramakanth, Assistant Professor, CES, ASCI; Victor Vanya, Managing Director, EMA Solutions; Professor Rajkiran Bilolikar, Director, CES, ASCI