Fathom Scientific is committed to making your #SaltyLife even sweeter. We’ve done this by creating an online Salt Dilution Instream Q (SDIQ) processing tool and Rating Curve Component Editor (RCCE). It is an extension of the Watershed Information Tool (WIT) with HydroMetric powers (WIT-HM), but to the common folk, it’s known as the Salt Portal.
If you are on the Salty Road without access to the SkyMind (aka internet) then consider downloading the Field Portal (FP). The instructions below apply to both systems, albeit the FP does not include the RCCE
This Sounding gives Ye Traveler of Salty Roads the knowledge to navigate the cavernous depths of the maze, battle the dreaded CF.T monster that lurks at the top of the stairs, win the hand of the handsome and beautiful SDIQ (with contractual Uncertainty) and emerge victorious with the treasures of the Golden Rating Curve. Enter… if… Ye… Dare!
Watershed Information Tool- HydroMetric (WIT-HM)
(aka Salt Portal at salt.fathomscientific.com. Contact us for an account)
The Salt Portal is an online post-processing system to easily load, view, process, and manage your QiQuac and T-HRECS-DL (AT) data. It also has a built in Rating Curve Component Editor (RCCE), which allows fitting and manipulation of a single, multi-stage hydraulically-based rating curve.
- Allows Fathom Scientific Ltd. (FSL) to
spyoversee and assist the QiQuac customer with their measurements and troubleshoot some problems remotely.
- Allows Organizational QA/QC whereby operators can upload SD data and a supervisor can review and approve measurements, and compare to existing rating curves.
- Export data for import into DataBase Management Systems (DBMS) such as Aquarius Time Series or Kisters Wiski.
- Quickly calculate Q and Uncertainty to all recommended guidelines to achieve a BC Resource Inventory Standards Committee (RISC) Standard Operating Protocol (SOP) Best Practices (BP) grade, of which the author was a contributor.
- Rigorous Uncertainty Analysis as per Sentlinger et al (2018)
- CF.T derivation, expectations, and uncertainty as per Richardson et al (2017)
The Tool is currently in Beta and these instructions will eventually be replaced with a more detailed online guide.
- Login to https://salt.fathomscientific.com using login credentials provided by FSL. It may not work behind a corporate firewall. You can possibly tether your computer to your phone’s data plan if necessary.
- Create a new Project (for example Culliton Creek HPP)
- Create a new Station (for example Culliton Creek DCP)
- Upload a new measurement by selecting “Upload & Edit SDIQ File”. SDIQ stands for Salt Dilution Instream Q.
- This should take about 5 seconds for normal QiQuac measurements but can take several minutes for larger AT files spanning several days of 5 second data.
- If the spinning bar at the top stops without the page updating with a graph, then there has been an error. Try again and contact FSL if unsuccesful.
The Salt Portal is an interface to an underlying database. The database stores the original data file, and relevant CF.T and SDIQ measurement parameters such as mass of salt injected and Start and End times. CF.T records and SDIQ records are independent. This allows the user to select a historical CF.T measurement to apply to the SDIQ measurement. It also allows several CF.T measurements independent of an SDIQ measurement.
When you first upload an SDIQ file, a new record is created every time Save SDIQ or Save CF.T is clicked. In this way it’s possible to process several measurements from the same file sequentially and quickly, for example when processing an AutoSalt T-HRECS-DL file shown in Figure 2. The flipside of this functionality is that you cannot Save Discharge and edit the same discharge. If you click Save Discharge, then change the Mass, clicking Save Discharge again will create a new record.
To Edit a record after saving, you must first exit the tool by clicking Back or on the Station name at the top, then clicking the pencil tool beside the record. When entering the tool from the Edit path, every time you click Save Discharge or Save CF.T it will overwrite the existing record. However! If you enter the the tool from the Edit Discharge path, the CF.T pointers will not be set properly even though the CF.T used to originally calculate the Discharge will be correct. Clicking Save Discharge will write over the existing Discharge record, but clicking Save CF.T will create a new CF.T record. Conversely, entering Edit CF.T will overwrite the existing CF.T but Save Discharge will create a new Discharge record. Clear as mud? Excellent, continue on!
The CF.T is the relationship between [NaCl] and Electrical Conductivity in μS/cm. We use Temperature Compensated EC, or EC.T, or Specific Conductivity, as per the recommendations in Richardson et al. (2017). Following this reference, there are 3 possible CF.T entry methods, each with its own tab in the CF.T tool. Current File (derived in situ), Stored Value (from this station record), or Manual Value. From the abstract of that paper:
The calibration factor can be determined with an uncertainty of less than ± 1% under “best-case” conditions, and the uncertainty may be as high as ± 4% under less favourable conditions. If calibration is not performed, CFT can be estimated from the relation between CFT and background temperature-corrected electrical conductivity (ECBG) with an uncertainty of about ± 2%, or estimated as a set value of 0.486 mg·cm·μS-1·L-1 with an uncertainty of about ± 2.8% for a properly calibrated probe
And from the Draft BC RISC SOP guidelines:
For mass balance methods, the user may derive CFT calibration coefficients either by: i) deriving in situ CFT at time of measurement (preferred); ii) using a site specific, sensor specific CFT derived over multiple measurements; or iii) using the published lab derived constant (Richardson et al. 2017)
Figure 3 shows all CF.T values from Richardson et al.(2017). From this figure, we can see that the average CF.T is around 0.49mg·cm·μS-1·L-1 with a slight dependence on Background EC.T (1.5% over 500 μS/cm) but this is not a hard rule and the CF.T can be between 0.475 – 0.50 μS/cm over the entire range of Background EC.
- After uploading your file, you will land in the CF.T page, shown in Figure 4. The tool will try to automatically determine the placement of the steps of the calibration. This seems to work best for 5 second data with steps of ~10 uS/cm. You can train the CF.T calibration by using the Box Select tool. If this tool does not properly select the steps, then unselect and reselect the calibration with the Box Select tool. When Auto is unselected, then the steps are evenly divided across the selection.
- Enter the correct CF.T Calibration parameters. We ship QiQuac kits with 5.00 g of NaCl in 1.00L of distilled water. Our standard CF.T method is to inject 1.00ml of this standard solution into 1.00L of River H2O.
- To fine-tune the step selection, use the Adjust Calibration Steps sliders. The checkbox informs the tool to adjust all subsequent steps based on the current slider position. Unselect the Link checkbox to prevent this for a given step. Adjust the Calibration Step Slider Sensitivity to increase range of each step slider.
- Examine the CF.T, the R², and the Calibration plot below the calibration sliders to achieve an acceptable CF.T. The CF.T has an acceptable range for a properly calibrated probe, using temperature compensation to 25°C. This is detailed in Richardson et al. Essentially, it’s 0.486 mg·cm·µS-1·L-1 ±2.8% with a small positive relationship on BG ECT. For higher BG ECT, i.e >200 μS/cm, the CF.T could be >0.50 mg·cm·µS-1·L-1 , although it’s not necessary depending on the chemistry. Very low conductivity water, i.e. <50 μS/cm should not have a CF.T exceeding 0.50 mg·cm·µS-1·L-1 . The R² should be 1.00.
- Once the CF.T is satisfactory, click Save Calibration to create a new CF.T record.
As per Richardson et al. (2017) it is acceptable to use a site derived CF.T. This may depend on the EC-T calibration of the meter, or the Background EC.T, however. Operator oversight is required when selecting an appropriate CF.T. The uncertainty associated with a stored value is assigned to the selected CF.T and Discharge measurement. Eventually this tab will show more statistics such as average and variance, and allow filtering based on device, Background EC.T, and or time of year.
If no CF.T available, of if the CF.T is known, it can be entered directly. An uncertainty of 5% is automatically assigned to this, but eventually the user will be able to enter the associated uncertainty.
Once the CF.T is chosen, the Discharge can be calculated. The Discharge Calculation page, shown in Figure 5, is divided into two areas, the pulse selection on the left and the Rating Curve Component Editor (RCCE) on the Right. Eventually the RCCE will have it’s own page, but for simplicity it resides on the Discharge page.
- Enter the mass of NaCl used.(Eventually we will pull this from the QQ file if set).
- Zoom in on the pulse to better see the shape and Background EC.T.
- Select the pulse using the box select tool. You do not have to select the top of the pulse; the tool only chooses the start and end time from the points selected at the extremities. Also it selects the middle of the start and end ranges.
- You can zoom in on the Pre- and Post- backround EC.T to look for any slope or evidence of an incomplete pulse. Currently there is no way to fill in a missing tail, but this ability is on the task list. Fine-Tune the Start and End times using the sliders and sensitivity bar. If the all the sliders are at the maximum value and you still need to adjust the start and end time, reselect the data using the box select tool.
- If you believe the Post BG_ECT should be lower (missing tail), the tool offers the ability to enter the BG_ECT manually, or copy the Pre- BG ECT. The tool will assign an appropriate error based on the difference between the EC.T values in the selected range and the manually entered BG ECT.
- The Discharge summary on the right should display results with uncertainty. The function will take the average of the pre- and post-pulse standard deviation (s) and result in a lower uncertainty. If this is left unchecked (which is what happens on the QiQuac) s will be the standard deviation of both pre- and post-BGECT together and result in a higher uncertainty.
- For a good measurement, the uncertainty should be less than ±5%. The uncertainty shown is the 95% confidence interval. If it’s larger than ±5%, check the mass uncertainty and also the CF.T uncertainty.
Rating Curve Component Editor (RCCE)
The RCCE is an interface and a database based on the hydraulic rating curve taken from Handbook of Hydrology (Maidment, 1992) and represents the hydraulic reaction of a smoothly varying channel with increasing stage. The database has 4 attributes:
- PZF-The Point of Zero Flow, or the stage when no flow occurs
- Coefficient-Is roughly equal to the channel width by a factor of 2.
- Exponent-Is related to the shape of the channel; 1.67 is closer to a rectangular weir, 2.17 is Parabolic, and 2.67 is Triangular.
- Transition-Most rating curves are multi-part with a different equation applying to each regime. Most are 2 part with a transition at approximately the D90 in the hydraulic control. Some can have 3 parts, such as upstream of a bridge, and in extreme cases 4-5 parts.
The hydraulic rating curve equation is
Q = C(h – PZF)^N (1)
Where h is the stage value. We also fit a linear Least Squares (LS) line to the stage-discharge pairs, solving for N and C. PZF must be given though. By default, it is zero, but after the user enters a PZF value in the RCCE, it will use that from the lowest RCC. The RCCE will sort the RCCs by Transition value. For example, if an RC has a transition at 0.5m, and the next has a transition of 10m, it will use the PZF of the first to fit the LS line to.
The LS RC is meant as a guide, but does not often suffice to represent the entire range of an RC. Future iterations of the RCCE will allow the user to select which points to plot, and which to use in the LS fit, thereby allowing fitting to a range of stage values. The “Copy LS Values” copies the N and C values of the current LS fit to the current RCC.
The legends in each plot are interactive. Click on a series to turn it’s display on or off. This is cool.
The RCCE is not currently able to convert a stage record to a flow series, but is meant to track how far measured Q values are from an established RC, or help the user define a new RC.
Importing Other Filetypes
The Salt Portal only recognizes QiQuac and T-HRECS-DL file types currently. However, it is possible to both pre-process files (such as remove outliers or correct EC.T for a new temperature) and process records from other devices. To do so:
- Open the file in Excel, or other spreadsheet software.
- Manipulate the file if needed, but don’t manipulate people, unless required.
- Open a QQ file you know that you can import (or simply download an existing one using the download button.
- Copy and paste in your EC.T data. Don’t worry about the EC,Temp, header, or measurement (i.e. mass, Q) columns.
- This step is important. When you save as .csv in excel, the current date-time format is saved to text. The date-time format must be YYYY-MM-DD hh:mm:ss in order for WIT-HM to recognize it. You can set it to this by formatting the date time column, format->custom->YYYY-MM-DD hh:mm:ss . It must have seconds shown or it will throw a “Zero time step detected” error and fail to load. Excel by default displays timestamps by minute and when this is saved as a csv, the second stamp is lost.
- Save as .csv.
Note that the current version requires a minimum of 40 non-zero time step records.
You should now be able to import your file into the Salt Portal.
From time to time, less and less, you will get an error message. Below are a list of the know messages and how to address them.
“Zero Time Step detected”: Check .csv.: In previous versions, the WIT-HM would check to see if the zero time step (no difference in seconds between subsequent records) count exceeded some threshold. We can get normally occurring Zero Time Steps when the button was pressed (an entry in the .csv file made) or when both probes are reporting 1 second measurements. In the current version, WIT-HM checks to ensure that the number of non-zero time steps exceeds 40. This implies you must have at least 40 records in your file and they must have a non-zero (in seconds) time step.
“File not recognized”: Either a) you are using a newer QiQuac and we haven’t updated the Salt Portal to accept these filetypes yet or b) you modified the file in Excel and Excel put in its own characters etc. The best way to fix this is to open a file you know works in the Salt Portal in a simple word processor, like Notepad, Wordpad, or Notepad++, open your revised file, and paste the text into the working template.
“On entry to DLASCL parameter number 4 had an illegal value”: Well, yah, obviously! Of course it did, anyone can see that! Clearly. This is what’s called an unhandled exception, i.e. we didn’t know it would happen and didn’t provide a better error message. It means there is a record with a negative flow for the station and it’s causing the Least Squares RC Fit to fail. You’ll have to delete that record and reprocess it and save it as a positive Q. What is a negative Q anyways? I guess it’s water flowing uphill, like Magnetic Hill in New Brunswick, Canada.
“Can only convert an array of size 1 to a Python scalar “: This is just a random error on our list to fix. It’s a fault of the automated CFT routine. To fix, just copy the first ten rows from the file in question, ideally in Notepad, Wordpad, or Notepad++, and paste them at the top of the record. Don’t worry about the timestamp, the Salt Portal doesn’t seem to care.
Future Bells and Whistles
We have a long wishlist, but we need subscriptions to fund further development. Contact me about your needs. Right now, the Salt Portal is open to all Fathom Customers, but we can’t fund further developments without subscriptions. We’re targettng $1000/year/site, but that is just our opening highball offer. Improvements that have been added are crossed out.
SDIQ and CFT Measurement Tool
- Link measurements by injection. Right now CH0 and CH1 measurements are treated the same as independent measurements. So no grading is possible within the system.
- A basic feature would be to read in the QQ summary info like mass of salt. It should be simple, but it turns out it’s not.
- Batch processing of measurements such that no user intervention is required. All the information is already in the QQ file anyways, but we just need to extract it. This would also be required for AutoSalt.
- Eventually make QiQuac a phone/laptop based app with all the RC data available so the operator can see where the point falls against the RC in realtime, provide feedback on ways to improve the measurement (i.e. reduce the uncertainty or things to check if it’s significantly different from the RC)
- Allow filling in of missing tail or missing data during a measurement, including interpolation to 1 second if 5 second data available.
- Allow filtering of EC.T data such as a 10pt median or average kernel.
- Breakdown the components of the Uncertainty so the user can identify and correct high uncertainty values.
- Remember the View settings when editing an SDIQ or CFT.
Allow the user to adjust the BGECT manually.
- Allow user to disregard steps, i.e. reduce or increase the number of steps from the current 4.
Create an option for %Mass Uncertainty instead of Mass Uncertainty in kg.
SDIQ Database Management and Analysis
- Allow site, project, and organization database analysis, such as “Plot all CFT measurements from the TM6.23 probe from Jan 1, 2018 to present”
- Allow sorting and filtering of database table.
- Add components of the SOP such that each station would have all the necessary meta-data for BP designation and future SOP Grade. The meta data would flow from measurement to measurement so no repetition.
- Add the following fields for each SDIQ or CFT: Notes, Device, Injection ID, Photos, TimeZone.
- Transfer measurements between sites. This is needed if SDIQ entered into wrong site.
Aquarius Time Series System Integration
- Seamless integration of necessary information with Aquarius Time Series
- Offer field forms for SOP compliance to easily upload to Aquarius Time Series.
Rating Curve Component Editor
- Allow user to turn points on and off for a) display b) rating curve fitting.
- Allow multiple Rating Curves with a time range of use.
- Apply RISC (or other) grading of Rating Curve automatically.
- Allow user to click on a point in the RC and be taken to that measurement.
- Change colour of points based on a) Device b) Date:Time c) Other?
- Show %Diff from current RC and “Significantly Different” which depends on the uncertainty in the SDIQ.
- Show all RCs in a table, just like SDIQs and CFTs.
- Play Spotify or Apple music during a measurement.
- Allow the user to change the sound of the Quack or chose from different Quacks.
- Compliment the user on their clothes.
- Make a good americano.
Pike, Robin 2018 “Draft Salt Dilution SOP Guidelines for a BP rating” BC Ministry of Environment Resource Inventory Standards Committee (RISC)
Sentlinger, Gabe, John Fraser and Evan Baddock, 2018 “Salt Dilution Flow Measurement: Automation and Uncertainty” HydroSenSoft, International Symposium and Exhibition on Hydro-Environment Sensors and Software. 26 Feb – 1 Mar 2019, Madrid, Spain.
Richardson, M., Zimmermann, A., Sentlinger, G. and R. D. Moore. 2017 “Uncertainty in the relation between electrical conductivity and salt concentration, with application to dilution gauging via dry salt injection” Confluence: Journal of Watershed Science and Management.