Cover Image: Fathomland by Fenella Jacquet + Gabe Sentlinger
Fathom Scientific is committed to making your #SaltyLife even sweeter. We’ve done this by creating an online Salt Dilution Instream Q (SDIQ) processing tool and Rating Curve Segment Editor (RCSE). It is an extension of the Watershed Information Tool (WIT) with HydroMetric powers (WIT-HM), but to the common folk, it’s known as the Salt Portal.
If you are on the Salty Road without access to the SkyMind (aka internet) then consider downloading the Field Portal (FP). The instructions below apply to both systems, albeit the FP does not include the RCSE
This Sounding gives Ye Traveler of Salty Roads the knowledge to navigate the cavernous depths of the maze, battle the dreaded CF.T monster that lurks at the top of the stairs, win the hand of the handsome and beautiful SDIQ (with contractual Uncertainty) and emerge victorious with the treasures of the Golden Rating Curve. Enter… if… Ye… Dare!
Watershed Information Tool- HydroMetric (WIT-HM)
(aka Salt Portal at salt.fathomscientific.com. Contact us for an account)
(Updated Dec 1, 2020)
The Salt Portal is an online post-processing system to easily load, view, process, grade, and manage your QiQuac and T-HRECS-DL (AT) data. It also has a built in Rating Curve Segment Editor (RCSE), which allows fitting and manipulation of a single, multi-stage hydraulically-based rating curve.
- Allows Fathom Scientific Ltd. (FSL) to
spyoversee and assist the QiQuac operator with their measurements and troubleshoot some problems remotely.
- Allows Organizational QA/QC whereby operators can upload SD data and a supervisor can review and approve measurements, and compare to existing rating curves.
- Export data for import into DataBase Management Systems (DBMS) such as Aquarius Time Series or Kisters Wiski.
- Quickly calculate Q and Uncertainty to all recommended guidelines to achieve a BC Resource Inventory Standards Committee (RISC) Standard Operating Protocol (SOP) Best Practices (BP) grade (of which the author was a contributor.)
- Rigorous Uncertainty Analysis as per Sentlinger et al (2018)
- CF.T derivation, expectations, and uncertainty as per Richardson et al (2017)
The Tool is currently in Beta and these instructions will eventually be replaced with a more detailed online guide.
- Login to https://salt.fathomscientific.com using login credentials provided by FSL. It may not work behind a corporate firewall. You can possibly tether your computer to your phone’s data plan if necessary.
- Create a new Project (for example Culliton Creek HPP)
- Create a new Station (for example Culliton Creek DCP)
- If desired, add Station, Visit, or Background photos under the Edit Station view.
- Upload a new measurement by selecting “Upload & Edit SDIQ File”. SDIQ stands for Salt Dilution Instream Q.
- This should take about 5 seconds for normal QiQuac measurements but can take several minutes for larger AT files spanning several days with 5 second data.
- If the spinning bar at the top stops without the page updating with a graph, then there has been an error. Try again and contact FSL if unsuccessful.
The Salt Portal is an interface to an underlying database. The database stores the original data file, and relevant CF.T and SDIQ measurement parameters such as mass of salt injected and Start and End times. CF.T records and SDIQ records are independent. This allows the user to select a historical CF.T measurement to apply to the SDIQ measurement. It also allows several CF.T measurements independent of an SDIQ measurement.
When you first upload an SDIQ file, a new record is created every time Save SDIQ or Save CF.T is clicked. In this way it’s possible to process several measurements from the same file sequentially and quickly, for example when processing an AutoSalt T-HRECS-DL file shown in Figure 2. The flipside of this functionality is that you cannot Save SDIQ, Edit the same SDIQ, and reSave the same SDIQ: a new record will be created. If you click Save SDIQ, then change the Mass, clicking Save SDIQ again will create a new record.
To Edit a record after saving, you must first exit the tool by clicking Back or on the Station name at the top, then clicking the pencil tool beside the record. When entering the tool from the Edit path, every time you click Save SDIQ or Save CF.T it will overwrite the existing record. However! If you enter the the tool from the Edit SDIQ path, the CF.T pointers will not be set properly even though the CF.T used to originally calculate the Discharge will be correct. Clicking Save SDIQ will write over the existing Discharge record, but clicking Save CF.T will create a new CF.T record. Conversely, entering Edit CF.T will overwrite the existing CF.T but Save SDIQ will create a new SDIQ record. Clear as mud? Excellent, read on!
The CF.T is the relationship between [NaCl] and Electrical Conductivity in μS/cm. We use Temperature Compensated EC, or EC.T, or Specific Conductivity, as per the recommendations in Richardson et al. (2017). Following this reference, there are 3 possible CF.T entry methods, each with its own tab in the CF.T tool. Current File (derived in situ), Stored Value (from this station record), or Manual Value. From the abstract of that paper:
The calibration factor can be determined with an uncertainty of less than ± 1% under “best-case” conditions, and the uncertainty may be as high as ± 4% under less favourable conditions. If calibration is not performed, CFT can be estimated from the relation between CFT and background temperature-corrected electrical conductivity (ECBG) with an uncertainty of about ± 2%, or estimated as a set value of 0.486 mg·cm·μS-1·L-1 with an uncertainty of about ± 2.8% for a properly calibrated probe
And from the Draft BC RISC SOP guidelines:
For mass balance methods, the user may derive CFT calibration coefficients either by: i) deriving in situ CFT at time of measurement (preferred); ii) using a site specific, sensor specific CFT derived over multiple measurements; or iii) using the published lab derived constant (Richardson et al. 2017)
Figure 3 shows all CF.T values from Richardson et al.(2017). From this figure, we can see that the average CF.T is around 0.49 mg·cm·μS-1·L-1 with a slight dependence on Background EC.T (1.5% over 500 μS/cm) but this is not a hard rule and the CF.T can be between 0.475 – 0.50 μS/cm over the entire range of Background EC.
The QiQuac and AutoSalt use a default CF.T of 0.486 mg·cm·μS-1·L-1. The expected CF.T is between 0.486 mg·cm·μS-1·L-1 ± 2.8%, which is between 0.474 and 0.500 mg·cm·μS-1·L-1. The Salt Portal will use this value + the small Background EC.T dependence if the R2 falls below 0.98 OR the derived CF.T is <0.4 or >0.6 mg·cm·μS-1·L-1. The CFT report will turn red if these boundary conditions are exceeded and use the Expected CF.T value. Likewise, the user can choose to use the Expected CF.T with the associated uncertainty (around 2.1%) by selecting “Use Expected CF.T”.
- After uploading your file, you will land in the CF.T page, shown in Figure 4. The tool will try to automatically determine the placement of the steps of the calibration. This seems to work best for 5 second data with steps of ~10 μS/cm. You can train the CF.T calibration by using the Box Select tool. If this tool does not properly select the steps, then unselect and reselect the calibration with the Box Select tool. When Auto is unselected, then the steps are evenly divided across the selection.
- Enter the correct CF.T Calibration parameters. We ship QiQuac kits with 5.00 g of NaCl in 1.00L of distilled water. The Salt Standard can be checked with another EC.T probe calibrated around 10 mS/cm; it should be 9.03±0.03 mS/cm. Our standard CF.T method is to inject 1.00ml of this standard solution into 1.00L of River H2O.
- To fine-tune the step selection, use the Adjust Calibration Steps sliders. The checkbox informs the tool to adjust all subsequent steps based on the current slider position. Unselect the Link checkbox to prevent this for a given step. Adjust the Calibration Step Slider Sensitivity to increase range of each step slider.
- Examine the CF.T, the R², and the Calibration plot below the calibration sliders to achieve an acceptable CF.T. The CF.T has an acceptable range for a properly calibrated probe, using temperature compensation to 25°C. This is detailed in Richardson et al. Essentially, it’s 0.486 mg·cm·µS-1·L-1 ±2.8% with a small positive relationship on BG ECT. For higher BG ECT, i.e >200 μS/cm, the CF.T could be >0.50 mg·cm·µS-1·L-1, although not necessarily as it depends on the chemistry. Very low conductivity water, i.e. <50 μS/cm should not have a CF.T exceeding 0.50 mg·cm·µS-1·L-1 . The R² should be 1.00 or greater than 0.99
- Once the CF.T is satisfactory, click Save Calibration to create a new CF.T record.
As per Richardson et al. (2017) it is acceptable to use a site derived CF.T. This may depend on the EC-T calibration of the meter, or the Background EC.T, however. Operator oversight is required when selecting an appropriate CF.T. The uncertainty associated with a stored value is assigned to the selected CF.T and Discharge measurement. To select an appropriate CFT, you can sort the table and select records as appropriate, based on :
- Background ECT
- Most Recent
The statistics associated with your selection appear below the plot showing your selection. Select “Use Average” or “Use Selected” to apply the derived CFT to the current measurement.
If no CF.T available, of if the CF.T is known, it can be entered directly. An uncertainty of 5% is automatically assigned to this method. If a QQ file is used, the value from the QQ is set as the default.
Once the CF.T is chosen, the Discharge can be calculated. The Discharge Calculation page, shown in Figure 5, is divided into five areas:
- The pulse (aka Breakthrough Curve) selection are showing the box select tool in red.
- The Rating Curve plot (note the Rating Curve Tool button in the upper right).
- Discharge Calculation variables.
- 3rd U/S Probe configuration
- Summary Information. Note that the Calibration Results are used to derive a CFT, but the CFT used in the Discharge record is stored separately from the CFT record.
To begin processing a measurement:
- Enter the mass of NaCl used. This will be populated with the Mass from the Summary line in the QQ File.
- Zoom in on the pulse to better see the shape and Background EC.T.
- Select the pulse using the box select tool. You do not have to select the top of the pulse; the tool only chooses the start and end time from the points selected at the extremities. Also it selects the middle of the start and end ranges.
- You can zoom in on the Pre- and Post- background EC.T to look for any slope or evidence of an incomplete pulse. Currently there is no way to fill in a missing tail, but this ability is on the task list. Fine-Tune the Start and End times using the sliders and sensitivity bar. If all the sliders are at the maximum value and you still need to adjust the start and end time, reselect the data using the box select tool.
- If you believe the Post BG_ECT should be lower (missing tail), the tool offers the ability to enter the BG_ECT manually, or copy the Pre- BG ECT. The tool will assign an appropriate error based on the difference between the EC.T values in the selected range and the manually entered BG ECT.
- The Discharge summary on the right should display results with uncertainty. The function will take the standard error, s, about a regression line trained from the pre- and post- BG ECT values and result in a lower uncertainty. If this is left unchecked, s will be the standard deviation of both pre- and post-BGECT together and result in a higher uncertainty.
- For a good measurement, the uncertainty should be less than ±5%. The uncertainty shown is the 95% confidence interval. If it’s larger than ±5%, check the mass uncertainty and also the CF.T uncertainty.
- Click on the Rating Curve Tool to work on the Rating Curve or the Grading Tool to create a group (CH0 and Ch1 for example) and Grade the measurement, and export to Aquarius Time Series or Kisters Wiski.
3rd U/S Probe Tool
Since 2019, QQ’s with firmware with suffix -s have the ability to record and compensate using a 3rd U/S sensor. This can be challenging to properly take into account in the field with the QQ, and is easier to consider in the Salt Portal. The 3rd U/S Probe must be adjusted in 3 dimensions: X or transit time, Y or EC.T Offset, and F or frequency. While the salt wave has no repeating signal, based on Fourier analysis, any signal can be reconstructed from adding component signals of different frequencies. As it turns out, the U/S signal has more higher frequency components than the D/S signal, in general. It depends on the inputs to the mixing reach. If there is a significant confluence with two different ECT patterns, then this will introduce more higher frequency components into the D/S signal. If there are no significant inputs into the mixing reach, then the D/S signal will be smoothed out by the storage elements in the mixing reach. We have hence named this filter function value, the Smoothing Factor.
- The transit time can be calculated in the field with the QQ by pressing the “Injection” button on the 3rd U/S T-HRECS Radio when making the injection OR by adjusting the Pre-BG ECT for CH2. The arrival time is determined by the rise of the ECT as the salt wave passes. This is handled automatically in the QQ.
- The EC.T offset is also handled automatically in the QQ and Salt Portal but lining up the Transit Time adjusted U/S probe signal with the Pre-BG ECT of the D/S channel signal.
- The Smoothing Factor is currently only handled in the Salt Portal. There is no automated method to filter out the higher frequencies in the U/S signal yet.
Rating Curve Segment Editor (RCSE)
The RCSE is an interface and a database based on the hydraulic rating curve taken from Handbook of Hydrology (Maidment, 1992) and represents the hydraulic reaction of a smoothly varying channel with increasing stage. The database has 4 attributes:
- PZF-The Point of Zero Flow, or the stage when no flow occurs
- Coefficient-Is roughly equal to the channel width by a factor of 2.
- Exponent-Is related to the shape of the channel; 1.67 is closer to a rectangular weir, 2.17 is Parabolic, and 2.67 is Triangular.
- Transition-Most rating curves are multi-part with a different equation applying to each regime. Most are 2 part with a transition at approximately the D90 in the hydraulic control. Some can have 3 parts, such as upstream of a bridge, and in extreme cases 4-5 parts.
The hydraulic rating curve equation is
Q = C(h – PZF)^N (1)
Where h is the stage value. We also fit a linear Least Squares (LS) line to the stage-discharge pairs, solving for N and C. PZF must be given though. By default, it is zero, but after the user enters a PZF value in the RCSE, it will use that from the lowest RCC. The RCSE will sort the RCCs by Transition value. For example, if an RC has a transition at 0.5m, and the next has a transition of 10m, it will use the PZF of the first to fit the LS line to.
The LS RC is meant as a guide, but does not often suffice to represent the entire range of an RC. For this reason, the RC Tool allows you to select points to “Use” in the LS RC Fit. The “Copy LS Values” copies the N and C values of the current LS fit to the current RCC.
There are 5 series in the legend.
- Anomaly: These are points the user has classified as Anomalous. They would not be used in the final RC Grade.
- Not Used: Are points currently turned off for RC fitting but still used in the final RC Grade.
- Used: are points used in the current RC fitting and the final RC Grade.
- Stn RC: are the user created station Rating Curve.
- LS RC: is the Least Squares fit to the points currently Used. Push the “Calculate LS RC” button to update this series.
The legends in the plot are interactive. Click on a series to turn its display on or off. This is cool.
Creating a Compound RC with the Select Tool
The RC Tool is a powerful and hydraulically valid Rating Curve development tool.
- To begin, select all points and click “Use” then “Apply”. All points should become blue. If you click “Calculate LS RC” you may get an error due to the bad points included.
- Now we must exclude the bad points. These are points with a stage or Zero, or Q or 0. In this case, I’m going to exclude all the points that are far away from a single RC. I do this by selecting anomalous points with the select tool (use the Shift key for a multi-part selection) and selecting “Anomaly” and then “Apply”, those point will become orangey-grey.
- Next, click “Calculate LS RC.” After a moment, a red LS RC line will appear and should generally follow the points, as in Figure 7. However, it is clear from this figure, that there is a sharp transition. Rollergate is an artificial channel, shown in Figure 8, with steep sides above a stage of 0.55m. This is consistent with a sharp transition at this level.
- To create an RC Segment for the lower portion, select all those points above 0.55 and then select the “Do not use” radio button, then Apply. Next click “Calculate LS RC” The Read LS RC series should pass through just the lower portion of the curve, shown in Figure 9. Adjust the pzf and click “Calculate LS RC” until the C and N values are reasonable (see Rating Curve Segment Editor (RCSE) above for reasonable values).
- When you are satisfied with the RC segment, it’s time to update the RC Segment record. To do so, click on the drop down list called “Select RC Segment” and select the first segment. Enter 0.55 into transition. Click “Copy LS Values” to copy the values from the LS RC into the fields of this segment. The RC Segment record will will be updated. Close the form
- To create the upper RC, click on “Select RC Segment” and “Add new segment”. A generic RCS form will appear. Click Swap On/Off to swap the non-anomalous points from lower to upper. Click “Calculate LS RC”
- Create a new record by clicking “Create new segment” and then select the new segment from the drop down list.
- Select “Copy LS Values.” This final step should leave you with a compound RC. Export your RC to Aquarius or any other database using the Download button from Station View page.
The RCSE is not currently able to convert a stage record to a flow series, but is meant to track how far measured Q values are from an established RC, or help the user define a new RC.
Importing Other Filetypes into the Salt Portal
The Salt Portal only recognizes QiQuac and T-HRECS-DL file types currently. However, it is possible to both pre-process files (such as remove outliers or correct EC.T for a new temperature) and process records from other devices. To do so:
- Open the file in Excel, or other spreadsheet software.
- Manipulate the file if needed, but don’t manipulate people, unless required.
- Open a QQ file you know that you can import (or simply download an existing one using the download button.
- Copy and paste in your EC.T data. Don’t worry about the EC,Temp, header, or measurement (i.e. mass, Q) columns.
- This step is important. When you save as .csv in excel, the current date-time format is saved to text. The date-time format must be YYYY-MM-DD hh:mm:ss in order for WIT-HM to recognize it. You can set it to this by formatting the date time column, format->custom->YYYY-MM-DD hh:mm:ss . It must have seconds shown or it will throw a “Zero time step detected” error and fail to load. Excel by default displays timestamps by minute and when this is saved as a csv, the second stamp is lost.
- Save as .csv.
Note that the current version requires a minimum of 40 non-zero time step records.
You should now be able to import your file into the Salt Portal.
Grading + Grouping Tool
There’s a new tool on the block, and nope, it’s not me. It’s the Grading + Grouping tool. This tool has been a long-time coming and there’s still more work to do. The idea is that each flow measurement is a pair, or group, of individual SDIQs, each with their own uncertainty and a conglomerate uncertainty. The Grouping tool also adds the 3rd U/S probe signal to the group. Here’s how it works.
In Figure 1, we see several measurements for the Aberdeen Project (Site) and the Pretty River Station. Only 2 channel measurements are currently shown.
The Grading Tool reads the SDIQ from each record, along with SDIQ Uncertainty and displays these values in the upper right. The colour scheme is Orange: Grade C, Blue: Grade B, and Green Grade A.
Only the first Group selected will be graded.
The DQ is taken as:
Q Uncert = Max (DQ, Q Uncert. CH0, Q Uncert. CH1)
and assigned a Grade based on :
<7%: Grade A
<15%: Grade B
>15%: Grade C
Only 1 Channel: Grade N
These Grading Thresholds are the default values in the QiQuac and the RISC guidelines here in BC, Canada. Future iterations of the Salt Portal will have user defined thresholds.
The Measurements table contains all those measurements associated with this Project-Site. Click on a measurements check box to plot it.
The colour scheme is always Blue: First series, Red: Second Series, Green: Third Series.
If you’d like to plot a group, you must deselect the measurements already plotted. The first three series selected will always be plotted.
To manually create a group, select all those measurements that belong to it and click “Create Group”. Alternately, click “Find Groups” to select all those measurements that have a start time within 5 mins of one another and create a group. The 5mins can be changed in the settings file.
To Export one or several groups to WISKI: BIBER, select them and click “Export Group” The xml file will be downloaded to the download directory.
To Delete one or several groups, select them and click “Delete Group”. Groups are very ephemral; don’t worry too much about deleting them. They can fairly quickly be generated again, as they are just pointers to the SDIQ measurements.
From time to time, less and less, you will get an error message. Below are a list of the known messages and how to address them.
“Zero Time Step detected”: Check .csv.: In previous versions, the WIT-HM would check to see if the zero time step (no difference in seconds between subsequent records) count exceeded some threshold. We can get normally occurring Zero Time Steps when the button was pressed (an entry in the .csv file made) or when both probes are reporting 1 second measurements. In the current version, WIT-HM checks to ensure that the number of non-zero time steps exceeds 40. This implies you must have at least 40 records in your file and they must have a non-zero (in seconds) time step.
“File not recognized”: Either a) you are using a newer QiQuac and we haven’t updated the Salt Portal to accept these filetypes yet or b) you modified the file in Excel and Excel put in its own characters etc. The best way to fix this is to open a file you know works in the Salt Portal in a simple word processor, like Notepad, Wordpad, or Notepad++, open your revised file, and paste the text into the working template.
“On entry to DLASCL parameter number 4 had an illegal value”: Well, yah, obviously! Of course it did, anyone can see that! Clearly. This is what’s called an unhandled exception, i.e. we didn’t know it would happen and didn’t provide a better error message. It means there is a record with a negative flow for the station and it’s causing the Least Squares RC Fit to fail. You’ll have to delete that record and reprocess it and save it as a positive Q. What is a negative Q anyways? I guess it’s water flowing uphill, like Magnetic Hill in New Brunswick, Canada.
“Can only convert an array of size 1 to a Python scalar “: This is just a random error on our list to fix. It’s a fault of the automated CFT routine. To fix, just copy the first ten rows from the file in question, ideally in Notepad, Wordpad, or Notepad++, and paste them at the top of the record. Don’t worry about the timestamp, the Salt Portal doesn’t seem to care.
“Cannot convert float infinity to integer” Well who can! If I could do that, I’d be a very wealthy man. However, sadly, this is not possible. It’s likely caused by a bunch of empty columns or rows that results when you open the csv file in excel and then save to csv. To fix it, open the file in a text editor and look for empty
Future Bells and Whistles
We have a long wishlist, but we need subscriptions to fund further development. Contact me about your needs. Right now, the Salt Portal is open to all Fathom Customers, but we can’t fund further developments without subscriptions. We’re targettng $1000/year/site, but that is just our opening highball offer. Improvements that have been added are crossed out.
SDIQ and CFT Measurement Tool (Updated Dec 1, 2020)
Link measurements by injection. Right now CH0 and CH1 measurements are treated the same as independent measurements. So no grading is possible within the system. A basic feature would be to read in the QQ summary info like mass of salt. It should be simple, but it turns out it’s not. Batch processing of measurements such that no user intervention is required. All the information is already in the QQ file anyways, but we just need to extract it. This would also be required for AutoSalt.
- Eventually make QiQuac a phone/laptop based app with all the RC data available so the operator can see where the point falls against the RC in realtime, provide feedback on ways to improve the measurement (i.e. reduce the uncertainty or things to check if it’s significantly different from the RC)
- Allow filling in of missing tail or missing data during a measurement, including interpolation to 1 second if 5 second data available.
- Allow filtering of EC.T data such as a 10pt median or average kernel.
Breakdown the components of the Uncertainty so the user can identify and correct high uncertainty values.
- Remember the View settings when editing an SDIQ or CFT.
Allow the user to adjust the BGECT manually.
- Allow user to disregard steps, i.e. reduce or increase the number of steps from the current 4.
Create an option for %Mass Uncertainty instead of Mass Uncertainty in kg.
SDIQ Database Management and Analysis
Allow site, project, and organization database analysis, such as “Plot all CFT measurements from the TM6.23 probe from Jan 1, 2018 to present” Allow sortingand filtering of database table.
- Add components of the SOP such that each station would have all the necessary meta-data for BP designation and future SOP Grade. The meta data would flow from measurement to measurement so no repetition.
Add the following fields for each SDIQ or CFT: Notes, Device, Injection ID, Photos, TimeZone.
- Transfer measurements between sites. This is needed if SDIQ entered into wrong site.
Aquarius Time Series System Integration
- Seamless integration of necessary information with Aquarius Time Series
- Offer field forms for SOP compliance to easily upload to Aquarius Time Series.
Rating Curve Component Editor
Allow user to turn points on and off for a) display b) rating curve fitting.
- Allow multiple Rating Curves with a time range of use.
- Apply RISC (or other) grading of Rating Curve automatically.
- Allow user to click on a point in the RC and be taken to that measurement.
- Change colour of points based on a) Device b) Date:Time c) Other?
Show %Diff from current RC and “Significantly Different” which depends on the uncertainty in the SDIQ. Show all RCs in a table, just like SDIQs and CFTs.
- Play Spotify or Apple music during a measurement.
- Allow the user to change the sound of the Quack or chose from different Quacks.
- Compliment the user on their clothes.
- Make a good americano.
Pike, Robin 2018 “Draft Salt Dilution SOP Guidelines for a BP rating” BC Ministry of Environment Resource Inventory Standards Committee (RISC)
Sentlinger, Gabe, John Fraser and Evan Baddock, 2018 “Salt Dilution Flow Measurement: Automation and Uncertainty” HydroSenSoft, International Symposium and Exhibition on Hydro-Environment Sensors and Software. 26 Feb – 1 Mar 2019, Madrid, Spain.
Richardson, M., Zimmermann, A., Sentlinger, G. and R. D. Moore. 2017 “Uncertainty in the relation between electrical conductivity and salt concentration, with application to dilution gauging via dry salt injection” Confluence: Journal of Watershed Science and Management.