March 28, 2023
Mitch O'Neill
There are quite simple and obvious improvements that can be made to reduce barriers to market participation for CER. They’re not sexy and they’re not going to change the world, but if you were to ask the CER industry “does this remove a barrier to participation?” they would say “yes”. Often we can lose sight of these types of improvements and instead focus on large reform packages and fancy trials, but I’d like to highlight some examples of where we can make small efficiency or quality of life gains in the realm of CER and markets.
So here are three boring, incremental improvements that make CER market participation either a bit better, or a bit easier:
Currently we bid in 1MW increments. This was fine in the days where parties were just bidding very big things, 360MW or 361MW… who cares it’s all about the same. When you consider aggregations of small sites, say, 1,000 5kW systems as part of a 5MW aggregation it starts to matter a lot.
Limiting these CER aggregations to bidding in 1MW increments means on average they’ll have about 500kW of extra capacity they’re not allowed to bid. E.g. Sometimes they’ll have 1.9MW they want to bid, but can only bit 1MW, leaving 900kW with nothing to do.
This is bad for individual aggregators. Imagine FCAS is paying out around $100,000/MW/year. For an aggregator, leaving 500kW of capacity unused translates to $50,000/year of value they can’t bid. This is per NEM region too. If they operate in QLD, NSW, VIC and SA that means on average $200,000/year of unrealised revenue. In just one market. For a small startup this could mean the difference between a few more employees or showing investors future potential to secure the next funding round.
It gets worse though if you look at it from an energy system perspective. Say there were 10 aggregators operating in NSW. That means for every 5-minute period, 24 hours a day NSW is missing out on 5MW (10x500kW) of potential capacity it could use. This number only grows as more aggregators enter the market.
Lowering the increment to 0.1MW means that bad effects to aggregators and the market are 10 times less, but keeps bids discretised for ease of use in the NEM Dispatch Engine. This is a pretty good incremental solution.
Back in 2020 the American federal energy regulator did something similar with Order No. 2222 which mandated that regional operators must allow aggregations as low as 0.1MW to operate in energy markets. So for all the ways we’re leading the world in CER integration, we’re weirdly behind in this one.
Let’s talk about Nemweb. Nemweb is the public source of energy market data. Prices, what generators are doing, carbon emissions, ‘reasons’ that generators had to change their bids that is totally re-bidding in good faith and not attempting to manipulate the market, things like that.
This is what one might see if they google Nemweb:
Nice clean website, good callouts to information transparency and assisting the public, it ticks all the boxes.
That though, dear reader, is not Nemweb. This is Nemweb:
It’s the kind of system that clearly was made many years ago and few have dared to touch it since.
It hosts zip files, containing CSV files (or more zip files within the zip files, containing CSV files) of energy data stuffed into multiple tables with poorly labeled columns. So poorly labeled I remember seeing one column with a “MW” suffix, but the documentation made an important callout along the lines of “while it says MW, we actually mean MWh”.
These files go up at pretty irregular times, and your best bet for grabbing data is writing code to continually scrape Nemweb until it sees a change in the HTML with a link to the new file.
On Nemweb the data is put into different directories: Current, which shows the most recent data up to a point, Archive, which shows older data, but not recent data, and the less known colloquially called ArchiveArchive, which shows even older data.
The tricky thing about this is if you wanted to get the last 12 months of data on something it’s likely you’ll have to get the first month from Current, then the next 11 months from Archive, and stitch them together. If you wanted to get 2 years of data it would likely require the stitching together of Current, Archive, and ArchiveArchive data. Additionally you’ll likely have to download far more data than you actually need, which costs you time and AEMO money (for the nerds out there: it looks like Nemweb, which is literally just static files, is hosted on 4 persistent AWS EC2 instances. AWS egress starts at 9c/GB in Australia.).
Many grid operators around the world provide good public energy application programming interfaces (APIs). These allow people to easily query for specific information over specific time periods and get that information back in an automate-able way.
While there are nice solutions out there like opennem, and a bunch of open source libraries on github that help you handle the beast that is Nemweb, I think that it should be a requirement on AEMO to offer base-level energy market data in easily accessible formats for humans and computers. Nemweb is neither.
But how is this a barrier to CER? By making energy market data harder to acquire, understand, and use it inherently makes it harder for innovation to happen in the CER space. Often the table stakes for getting access to this data is having a software developer or two that know the ins and outs of the poorly labeled or poorly placed data throughout Nemweb. Product development, research, monitoring, modelling. All functions that are important to companies and individuals in the CER space and all are currently inhibited far more than they have to be.
I’ve been rambling for a while, and in this section the fixes aren’t as easy or immediate, but to quickly enumerate some of the issues:
Often when we hear “removing barriers to CER” it’s related to big bang reforms or large fancy trials. The main thing I want to communicate here is let’s not put all our “removing barriers to CER” eggs in these baskets. Sometimes reforms can stall, sometimes trials can fail and deliver very little. I think it’s important we don’t lose focus on the low risk, incremental, boring improvements that can be made to improve the efficiency of the market and remove barriers that CER operators are pointing out today.
Thanks for reading.