![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Over the last two weeks I was able to spend few more hours on my project, and put together sensors data visualizer using Highcharts and Knockout. It produces pretty graphs like this:
:
The UI is pretty simple, and is really more of a concept rather than finished product. It allows selecting dates range, sensor, and chart type (Daily, Hourly etc). Graphing is done using Highcharts, input fields and interaction is handled using Knockout.
In the process of putting it together I had to re-write sensors handling backend. Original code worked, but unfortunately performance was unacceptable - it could take up to 16 seconds to fetch the data and redraw the graph. This was caused by having sensor data in annual logs - each sensor was getting its own annual log, and by the year end it could accumulate quite a lot of data - about 17K records (at 30min interval). Records are pretty small, and overall file size was only 260K or so, but unfortunately for a tiny controller like Arduino this was already way too much.
I investigated where time was spent, and found that out of 16 seconds the lion's share (12 seconds) was spent in reading the file (using fgets), and another 4 seconds in parsing records using sscanf. I could've probably bring it down to 4-5 seconds by reading data manually in bigger chunks (e.g. 512 bytes) instead of fgets, and by writing custom parser code instead of sscanf. But 4-5 seconds is still too much for an interactive UI.
To solve this problem I changed the backend code to use monthly files instead of annual files. I.e. each sensor's data is stored in a separate file, one file per month. Because in most cases user wants to visualize last week or last month, it dramatically reduces the amount of data that needs to be processed. Typically the chart will redraw in just over a second, and on the month boundary it may take up to two seconds - not ideal but usable. The time to redraw the graph is proportional to the dates interval, and for a whole year worth of graph you still need to wait for 16seconds, but it expected to be an edge case.
Another possible enhancement is to change the visualizer to download and consume CSV files directly, parsing it in the javascript in the browser rather than Arduino backend. This would allow further improving redraw time by 2x/3x, because direct file download of the original CSV file works a lot faster - no need to parse it on the weak Arduino CPU, also file download uses 512 bytes buffer which helps. But this is a project for another weekend.
Overall I'm satisfied with the Sensors logging and visualizer mechanism, it appears to be fairly reliable and efficient. I'm using it already for over a month, and it was extremely stable.
Next steps
As the next steps I'm planning to add second sensor (DHT21) to the controller, and most importantly - add remote controllers connected via Xbee 900 RF link. This would allow me to have distributed sensors and irrigation controllers network on my property.
:
The UI is pretty simple, and is really more of a concept rather than finished product. It allows selecting dates range, sensor, and chart type (Daily, Hourly etc). Graphing is done using Highcharts, input fields and interaction is handled using Knockout.
In the process of putting it together I had to re-write sensors handling backend. Original code worked, but unfortunately performance was unacceptable - it could take up to 16 seconds to fetch the data and redraw the graph. This was caused by having sensor data in annual logs - each sensor was getting its own annual log, and by the year end it could accumulate quite a lot of data - about 17K records (at 30min interval). Records are pretty small, and overall file size was only 260K or so, but unfortunately for a tiny controller like Arduino this was already way too much.
I investigated where time was spent, and found that out of 16 seconds the lion's share (12 seconds) was spent in reading the file (using fgets), and another 4 seconds in parsing records using sscanf. I could've probably bring it down to 4-5 seconds by reading data manually in bigger chunks (e.g. 512 bytes) instead of fgets, and by writing custom parser code instead of sscanf. But 4-5 seconds is still too much for an interactive UI.
To solve this problem I changed the backend code to use monthly files instead of annual files. I.e. each sensor's data is stored in a separate file, one file per month. Because in most cases user wants to visualize last week or last month, it dramatically reduces the amount of data that needs to be processed. Typically the chart will redraw in just over a second, and on the month boundary it may take up to two seconds - not ideal but usable. The time to redraw the graph is proportional to the dates interval, and for a whole year worth of graph you still need to wait for 16seconds, but it expected to be an edge case.
Another possible enhancement is to change the visualizer to download and consume CSV files directly, parsing it in the javascript in the browser rather than Arduino backend. This would allow further improving redraw time by 2x/3x, because direct file download of the original CSV file works a lot faster - no need to parse it on the weak Arduino CPU, also file download uses 512 bytes buffer which helps. But this is a project for another weekend.
Overall I'm satisfied with the Sensors logging and visualizer mechanism, it appears to be fairly reliable and efficient. I'm using it already for over a month, and it was extremely stable.
Next steps
As the next steps I'm planning to add second sensor (DHT21) to the controller, and most importantly - add remote controllers connected via Xbee 900 RF link. This would allow me to have distributed sensors and irrigation controllers network on my property.