| Field group | Fields | Complete on all fields | Worst single field |
|---|---|---|---|
| Riparian cover: shading, bank overhang, trailing vegetation | 3 | 84% | 87% |
| Riparian structure: tree, shrub and ground strata | 3 | 48% | 49% |
| Instream cover: macrophytes, algae, detritus, moss | 4 | 84% | 90% |
| Channel form: pool and riffle share, width, depth | 4 | 73% | 80% |
| Substrate: eight size classes | 8 | 83% | 88% |
| Sedimentation rating | 1 | 87% | 87% |
18 What to measure next
Blue Mountains City Council Healthy Waterways — statistical analysis
18.1 What this chapter is
A staging area, and calling it one is better than dressing it up as an argument. It holds four lists that all belong to the same question — what should the next few years of monitoring look like? — and that have no better home yet:
- a riparian and geomorphic rapid assessment, which you have already flagged as a future direction;
- further analyses the existing data would support;
- further questions it could answer that nobody has asked, which Section 18.5 is right to insist is a different kind of list from the one above it;
- what could be bought, with what is and is not on your licences.
Expect this chapter to be split further. The natural cut is between getting data that does not exist yet and working on data you already hold. The rapid assessment and the purchasing section are the first; the two lists of analyses are the second. Two pieces also argue with chapters elsewhere and would read better beside them — the rapid assessment next to chapter 15’s discussion of what the rating leaves out, and the co-located AUSRIVAS round next to chapter 12. They are together here because they are all proposals, not because they form one chapter.
Everything on these lists is additional to the data questions. Those are in chapter 2, and they are the deliverable.
18.2 The data behind this chapter
Each question and request below is set out again in What we need from you, with what it blocks, what an answer is worth and what it would cost you to find, ranked against every other ask in the report.
18.2.1 Questions only you can answer
Do you want to complete flow velocity, discharge, average depth, wetted width and cross-sectional area on every visit — or take them off the sheet altogether?
Flow is probably the most important physical driver of a macroinvertebrate community and it appears nowhere in this report, because those fields are populated on roughly one visit in five. A field filled one visit in five is zero of a dataset plus the cost of collecting it. Either is a defensible answer; the current state is the one that is not.
Refer to it as
dq:flow-fields-complete-or-drop.
18.2.2 What would answer them
Would you add a pick-count and subsample-fraction field to the laboratory sheet from now on, and flag any count that was scaled up from a subsample?
The cheapest high-value change available, and it closes the largest confound in this report for everybody who comes after us. It also fixes a second problem: 24% of counts between 100 and 499 are exact multiples of ten, so a count of 500 does not have anything like the precision of a count of 5, and nothing currently records which is which.
Refer to it as
dq:pick-count-field.
Would you add a per-parameter “below detection limit” tick box to the laboratory sheet, and a < prefix to the value?
The single highest-yield form change on this list. This one box would have prevented 101 mis-stored coliform records and a published trend that turned out to be an artefact, and it would have made the biggest question on the whole master list unnecessary.
Refer to it as
dq:below-detection-tickbox.
Would you replace the free-text officer field with a staff dropdown, on both the field and laboratory sheets — and reconcile the names already in there? There are eight labels for six people.
Near-zero cost. In ten years this is what separates a change in the creeks from a change in who is looking at them, and at the moment there is nothing before 2018 to do that with.
Refer to it as
dq:officer-dropdown.
Would you record the clock time of sampling on every visit — and make sure it survives into the database, which the current one does not?
Diurnal water temperature variation in a shaded headwater stream is 2-5 °C. The one water quality trend that survives everything else in this report is about half a degree Celsius per decade averaged over the year. A slow drift in the hour of sampling would confound it completely and nobody could check, because the time is collected on the sheet and then thrown away on the way in.
Refer to it as
dq:sample-time-through.
Would you keep the free-text field notes and give them more room on the sheet, plus tick boxes for “result estimated”, “plate crowded” and “too many to count”?
The notes are the best thing in the dataset and they cost nothing. They are the only reason the coliform detection limit is now documented at 10 rather than guessed at 1. More room and three tick boxes would make them better still.
Refer to it as
dq:structured-field-notes.
Would you complete all eight substrate classes, and the three vegetation strata, on every visit from now on? Both are already on the sheet.
Fine sediment is the strongest predictor of sensitive rare-family richness in this report, and the silt and clay boxes are the ones most often left blank. The three vegetation strata are recorded on about half of visits. A field that is filled some of the time cannot carry a trend, and neither of these costs anything to fix — they are already printed on the form.
Refer to it as
dq:substrate-classes-complete.
Would Assets backfill Construction Date for the treatment assets — it is populated for 38 of 223 — or confirm that it never will be? And start recording disposals, and enter the treatment assets that are not pits?
The draft WSUD register in TRIM lists about 260 devices with zero dates and names “Construction Date” as an attribute still to be collected, so this is already known internally. The whole evaluation question depends on it being fixed once. “It never will be” is a genuinely useful answer, because it tells us to stop designing around a register that is not coming.
Refer to it as
dq:asset-register-completeness.
Would you write down a protocol for the habitat observations, and set up a photo point at each site?
Ten of the thirteen habitat variables drift within a fixed site, and there is no written protocol behind any of them. Until there is one, recorded shading has to be read as a state variable — good for comparing reaches, useless for comparing years — and a rise in it must not be read as improvement. A photo point makes the next twenty years auditable for the price of a stake in the ground.
Refer to it as
dq:habitat-protocol-photopoints.
At the next probe changeover, would you run two field rounds with both instruments before retiring the old one?
Two days of fieldwork protects the next twenty years of data. The absence of exactly this at the last two changeovers is why turbidity and dissolved oxygen have been abandoned in this report.
Refer to it as
dq:probe-overlap-next-change.
Would it be feasible to record values once, on a tablet at the creek, with each parameter’s plausible range built in — so that a DO% of 7.8 beside a DO of 8.8 mg/L asks a question at the creek rather than twenty years later?
This removes an entire error class rather than a particular error. Every single item in the transcription register is a hand moving between a clipboard and a keyboard, and the post-2020 probe data — which writes its own file — has none of them at all. We do not know what a system change costs you, which is why this is a question rather than a suggestion.
Refer to it as
dq:tablet-form-at-creek.
Would you replace the free-text water_level box with a six-option dropdown — no flow, low, low-moderate, moderate, moderate-high, high?
Nothing is lost retrospectively: the parser recovered 1,003 of 1,005 rows, which is a testament to how consistently people wrote in that box. But the next seventeen years should not need a parser, and the six options are the ones your own field staff already use.
Refer to it as
dq:flow-state-dropdown.
Would you store the imperviousness figure for each sub-catchment in the database, alongside the health results? Neither “DCI” nor “impervious” appears anywhere in either database at present.
Your own sub-catchment classification matrix — the thing that decides whether a creek is reference, slightly disturbed or urban — cannot be reproduced from your own records, because the number it turns on is not in them. We rebuilt it from public data and it took weeks. One column would mean nobody ever has to again, and it would let the tier assignment be audited rather than trusted.
Refer to it as
dq:imperviousness-in-database.
18.3 A riparian and geomorphic rapid assessment
You flag this as a future direction. It is the right direction, and the evidence in this report says what it should measure and what it should not duplicate.
You already record, free, at most water quality visits: riparian shading, bank overhang, trailing vegetation, the three vegetation strata, pool and riffle proportion, macrophyte, algal and detritus cover, a sedimentation rating, channel width and depth, and eight substrate percentages. That is most of a rapid riparian and geomorphic assessment already. The gaps are in consistency, not coverage (Table 18.1).
- Substrate is complete on 83% of records — all eight classes — and silt and clay together, which is what a fine-sediment measure needs, are present on 88%. Chapter 10 identifies fine sediment as the strongest predictor of sensitive rare-family richness, so the missing records are the ones that would matter most.
- The three vegetation strata are recorded on about half of visits. Whatever changed in the form or in practice, half the riparian structure record is absent.
- No protocol is recorded anywhere. These are visual estimates by different officers over 19 years with no stated method, no photographs and no calibration. A systematic drift in how officers estimate cover would appear in the analysis as an environmental gradient, which is chapter 9’s second limitation.
18.3.1 What it should add
Ranked by what the evidence in this report supports.
| Priority | Measure | Rationale | Cost |
|---|---|---|---|
| 1 | A written protocol and photo points for the fields already collected | Chapter 9 shows shading is the strongest reach-scale lever on composition, but the estimates are unmethodised. A one-page protocol, a fixed photo point per site and an annual calibration session would make the existing series analysable. | One day to write; ten minutes per visit. |
| 2 | Complete the eight substrate classes on every visit | Fine sediment is the strongest predictor of sensitive rare-family richness (chapter 10). Complete on 83% of records now. | None; it is already on the form. |
| 3 | Bank erosion and channel stability | Not recorded at all. Geomorphic condition is the one part of the physical picture the current form misses entirely, and it is what a creek naturalisation project changes. | Two minutes per visit using an existing rapid method. |
| 4 | Flow permanence and connectivity | Nothing in either database records whether a reach dries. Chapter 8’s compositional analysis cannot distinguish a creek that lost taxa from one that dried. Of the variables the form does not record, this is the one that would do most to explain a bad rating; the priority order ranks all six rows on the broader question of what the evidence supports. | One field observation plus a desktop estimate. |
| 5 | Riparian weed cover and native canopy species | Distinguishes a shaded creek under native canopy from one shaded by privet. Directly actionable by your bushcare program. | Two minutes per visit. |
| 6 | Large woody debris count | Standard in rapid geomorphic assessment, cheap, and directly manipulable through instream works. | Two minutes per visit. |
18.3.2 How it should be used
Not as a factor in the health rating. Section 15.3.6 shows that adding riparian shading — the strongest of these variables — to the rating reduces its discrimination, because habitat is a site attribute rather than a condition measure, and folding it in makes the rating partly a description of where the creek is rather than of how it is doing.
Use it instead as the third panel of a three-part site report: condition (the macroinvertebrate rating), water quality (trigger-value exceedance, Section 15.3.6.1), and habitat (the rapid assessment). Each scored and reported separately. A creek that rates Fair with good water quality and poor habitat needs different work from one that rates Fair with good habitat and poor water quality, and a single averaged number cannot tell you which you are looking at.
The habitat panel is also the right home for the protection tier, and that placement is R7: Section 19.4 sets out why the tier cannot sit in the condition panel, and chapter 10’s Section 10.6 is the full version of the tier itself.
18.4 Suggested next analyses
Roughly in order of value, and all of them cheap now that the data layer and the covariate layer exist.
- Re-run chapter 16’s six-test protocol on the revised index using 2025–2029 data. Not because the revision is calibrated on its own data — percentile anchors are monotone transforms, so a different calibration window can only move a factor’s discrimination where the ramp clips, and chapter 15 measures what is left of that separately from the change of evaluation window it is confounded with. Re-anchoring on 2015–2019 alone moves the revised index’s AUC advantage by 0.000 judged over 2010–2024 and -0.006 judged over 2020–2024; moving the evaluation window instead is worth -0.139, about 22 times as much (Table 15.11). Self-calibration is the smaller of the two effects the old holdout confounded, not the binding one. The reason is simpler and harder to fix: only 11 of your 17 reference sites carry enough 2010-onward record to enter chapter 13’s discrimination test (Section 12.3), and every discrimination result there rests on those 11. That is the binding constraint and no re-analysis relieves it. More reference creeks would relieve it; so would five more years of data. So, more cheaply, would recovering the other 6, which are excluded by record length and by missing coordinates rather than by absence: getting a usable location for the historic reference sites Section 12.3 names, P7 in particular, is a smaller undertaking than establishing a new reference creek, and it is on no list in this report. One route does not require you to establish any: the state AUSRIVAS reference network covers the same region on a compatible protocol, and the case for reading a local program alongside the national bioassessment framework rather than against its own isolated reference set is made directly by Nichols et al. (2017).
- A co-located AUSRIVAS sampling round. Chapter 12 costs this out in full (Section 12.8) — one field season, and it is the only thing that converts the external archive from a spot check into a test of the reference set. It belongs on this list as well as in that chapter.
- Rebuild the rating anchors from the versioned script and publish the back-cast series. Show the team and the public what the last ten years of ratings look like under the revised system before switching, so the transition is understood rather than discovered.
- A functional-traits analysis of the community change. The tolerant-to-sensitive shift is established; what is not known is whether it is driven by water quality, by flow regime, or by habitat. Feeding group, dispersal ability and desiccation tolerance would separate those explanations. No trait database exists in either database, but published Australian family-level trait sets do.
- A measured imperviousness figure, once the pipe, pit and kerb layers are available — and with it a time series as the network grew, which would let the effect of catchment change on the creeks be estimated directly rather than cross-sectionally.
- Hydrology, now that flow state has been recovered. The free-text
water_levelbox on the site description sheet turned out to hold an ordered flow-state scale recorded on 1,005 visits since 2008 — nearly twice the 558 visits with an average flow velocity — and normalising it (Section 9.3.3) made seventeen years of flow observations analysable. What it showed: flow state is the strongest visit-level predictor of dissolved oxygen in the record (Section 6.7), it moves community composition significantly but by only about 1.7% of within-creek variation (Section 9.6.1), and it does not move the health rating up its scale: there is no monotone response across the six-point flow scale. What it does do is depress the rating on visits where the creek was not flowing — a step, not a slope — and three of the four inputs fall with it (Section 13.5.6). That is an artefact channel rather than a response, and chapter 13 counts it against freedom from artefact, not for responsiveness. It is not the missing driver. What remains genuinely missing is hydrology: antecedent discharge, time since the last flushing flow, and whether a sample sat on a rising or a falling limb. A field officer’s one-word judgement cannot carry any of those. That needs a gauge — yours or a partner’s — and it is the version of this question still worth asking.
18.5 Further questions the existing data could answer
A separate list from the one above, and a different kind of thing. These are questions nobody has asked that the data you already hold could settle. The admission test was that an item must be answerable from data in hand and that a plausible answer would change a decision; anything needing new data is a data question and lives in chapter 2 instead.
They are ordered by value to you, not by statistical interest. The first four are about how the monitoring budget is spent and what gets published; the next four are about whether anything you do to a creek can be shown to work; the rest close narrower gaps.
R/audit/FURTHER-QUESTIONS.md.
| # | Question | Effort | New data? |
|---|---|---|---|
| 1 | Which creeks are recovering, and what do the recoverers have in common? | 1 week | No |
| 2 | Would a defensible family count change anyone’s published rating? | 1 day | No |
| 3 | Does the published site-year rating survive testing at its own grain? | 3 days | No |
| 4 | Where should the next monitoring dollar go? | 3 days | Part-answered in ch 13 |
| 5 | Is the monitoring network the right network? | 2 weeks | Partly |
| 6 | Can a creek’s decline be seen coming? | 1 week | No |
| 7 | Is there a detectable signal from works already done? | 1-2 weeks | Works dates |
| 8 | Does changing the shade on a creek change its community? | 1 week | No |
| 9 | Is the stream warming faster than the air, and if so, where? | 3 days | No |
| 10 | Is imperviousness measured at the right scale, and does it earn its place? | 1 week | No |
| 11 | Is the bug-water quality subset representative? | 2 days | No |
| 12 | Are the rare families going locally extinct, or just going unseen? | 2 weeks | No |
| 13 | Does what Council measures match what the public can see, and what the public values? | 1 week + a survey | Partly |
Four are worth pulling out here (Table 18.3) because they bear directly on decisions this report leaves open.
Question 1 — which creeks are recovering. The report shows a network-wide improvement to about 2014 and a tolerant-to-sensitive shift, but nothing in it fits a per-creek trend, so a rising tide cannot be told from a handful of creeks improving dramatically while the rest sit still. A random-slope model on the health score, per-creek slopes ranked and then regressed on imperviousness, shading, fire history, catchment size and network position, would say which. The chapter 5 audit called this the single most useful extension the existing data would support, and that is still true.
Question 3 — the published rating, at its own grain. Chapter 13 tests reliability per sample and discrimination on ten-year site means. The product you actually publish is a single word for one creek in one year, and nothing in this report evaluates that object: how often does a creek’s published word change from one year to the next when nothing about the creek changed? That is a three-day analysis and it bears directly on chapter 13’s first recommendation.
Question 7 — a signal from works already done. Chapter 17’s position is that nothing in the archive is a properly dated intervention. That is right about proof and too pessimistic about looking. Nobody has scanned the record for step changes at every site with enough samples, controlled the multiplicity, and taken the resulting list of site-and-year candidates to Assets to ask what happened nearby. The scan does not need the dates, so it is not blocked by the data request; and a negative result is worth having, because it says the current design cannot detect the works you build at the scale you build them, which is the strongest possible argument for funding the dedicated before-and-after design Section 17.5 costs out.
Question 12 — extinct or unseen. The rare-family decline is the most consequential conservation finding in the report, and a fall in how often a family is recorded can mean it has gone or that it is being missed. The two have opposite management implications. Separating them needs a multi-season dynamic occupancy model, which is the most technically demanding item on the list, and it has to be run alongside the taxonomic check — an occupancy model will faithfully model a laboratory naming change as a colonisation event.
18.6 What could be bought, and what it would cost
Several external datasets would materially help. The useful finding is not the prices; it is that three of the four are not available to you at all, and two of those three were assumed to be. Know that before anyone builds a funding case around them.
| Product | What it would answer | Position |
|---|---|---|
| Nearmap AI Packs — surface permeability, vegetation, building footprints, roof characteristics | Measured impervious surface and canopy, replacing the modelled imperviousness figure and settling the riparian-shading question | Not on your licence. The endpoint returns HTTP 403, “forbidden: AI Feature API general access”. Licensed separately from imagery and metered by export credits, so a price needs a conversation with Nearmap |
| Nearmap 3D (surface and terrain models, about 15 cm) | Kerb extraction from the ~150 mm step, and canopy height rather than cover | Not available for this LGA at all — zero 3D surveys over Katoomba, Wentworth Falls, Springwood or Glenbrook. There is nothing to price unless Nearmap will add the area to its program |
| Planet Planetary Variables — soil water content, land surface temperature | Antecedent catchment wetness before each sample | Not on the NSW licence. It also would not substitute for a gauge: no discharge, no hydrograph limb, and at 100 m a small headwater catchment is a handful of pixels |
| NSW 1 m LiDAR DEM via ELVIS | Would replace the elevation surface used for catchment delineation | Free to discover, but the files sit in a requester-pays bucket — about 27 GB over the window, on the order of AU$2 in egress. Only about 63% of the window was ever flown, and you already hold 2 m LiDAR contours and a 4 m ALS surface, which is probably the better path and costs nothing |
What your licences do cover, tested rather than assumed:
- Nearmap vertical imagery. Eighteen surveys over Katoomba between January 2010 and November 2025, roughly annual since 2014 with a gap from 2010 to 2014, at 4.4 to 8 cm. That archive is the real asset: it can date any stormwater asset visible from the air to within a year since 2014 — which is chapter 17’s binding constraint — test the canopy trend, and help resolve ambiguous site coordinates. Classifying the tiles yourself is real work, but it is tractable over small buffers.
- Planet, through the NSW Imagery Hub. A whole-of-NSW-government licence rather than a Council contract: four active subscriptions, quotas effectively unlimited, and 1,860 PlanetScope scenes at 70% clear or better over the study area between 2017 and 2026 — including two full years of pre-fire baseline and 221 clear scenes through the recovery year. That is enough to turn “did this catchment burn” into “how far through recovery was it on the day this sample was taken”.
One time-sensitive point. The Planet licence is funded only to the end of FY2026-27 and now runs without dedicated staff support. Imagery already downloaded stays available even if the service stops, so if Planet data is wanted, pull it now rather than after the analysis is written. Nothing else in this report has a deadline attached to it.
Two smaller access requests are worth making internally rather than commercially. Three folders on the GIS share are ACL-denied — Data\Environment, Data\Images\Model and Data\Assets\Building — and they are the most likely homes of the missing stormwater sub-catchment polygons and DEM rasters. And the financial asset register may hold the commissioning dates the operational extract does not expose, which is dq:sqid-commissioning-dates — chapter 17’s binding constraint, and one of the 14 asks chapter 2’s master list rates transformative. It is not the highest-value ask in the report: no single ask is, and Where to start in the preface is the list that answers that question.