December 2016

252627282930 31

Style Credit

Expand Cut Tags

No cut tags

November 2nd, 2014

tony_osp: (Default)
Sunday, November 2nd, 2014 04:38 pm
Over the last two weeks I was able to spend few more hours on my project, and put together sensors data visualizer using Highcharts and Knockout. It produces pretty graphs like this:
Sensors data visualizer

The UI is pretty simple, and is really more of a concept rather than finished product. It allows selecting dates range, sensor, and chart type (Daily, Hourly etc). Graphing is done using Highcharts, input fields and interaction is handled using Knockout.

In the process of putting it together I had to re-write sensors handling backend. Original code worked, but unfortunately performance was unacceptable - it could take up to 16 seconds to fetch the data and redraw the graph. This was caused by having sensor data in annual logs - each sensor was getting its own annual log, and by the year end it could accumulate quite a lot of data - about 17K records (at 30min interval). Records are pretty small, and overall file size was only 260K or so, but unfortunately for a tiny controller like Arduino this was already way too much.

I investigated where time was spent, and found that out of 16 seconds the lion's share (12 seconds) was spent in reading the file (using fgets), and another 4 seconds in parsing records using sscanf. I could've probably bring it down to 4-5 seconds by reading data manually in bigger chunks (e.g. 512 bytes) instead of fgets, and by writing custom parser code instead of sscanf. But 4-5 seconds is still too much for an interactive UI.

To solve this problem I changed the backend code to use monthly files instead of annual files. I.e. each sensor's data is stored in a separate file, one file per month. Because in most cases user wants to visualize last week or last month, it dramatically reduces the amount of data that needs to be processed. Typically the chart will redraw in just over a second, and on the month boundary it may take up to two seconds - not ideal but usable. The time to redraw the graph is proportional to the dates interval, and for a whole year worth of graph you still need to wait for 16seconds, but it expected to be an edge case.

Another possible enhancement is to change the visualizer to download and consume CSV files directly, parsing it in the javascript in the browser rather than Arduino backend. This would allow further improving redraw time by 2x/3x, because direct file download of the original CSV file works a lot faster - no need to parse it on the weak Arduino CPU, also file download uses 512 bytes buffer which helps. But this is a project for another weekend.

Overall I'm satisfied with the Sensors logging and visualizer mechanism, it appears to be fairly reliable and efficient. I'm using it already for over a month, and it was extremely stable.

Next steps

As the next steps I'm planning to add second sensor (DHT21) to the controller, and most importantly - add remote controllers connected via Xbee 900 RF link. This would allow me to have distributed sensors and irrigation controllers network on my property.

tony_osp: (Default)
Sunday, November 2nd, 2014 06:52 pm
As I'm working on my Sprinklers/Sensors controller, it is becoming more and more of a SmartHome controller. I just need to add X10 interface to it, to enable interaction with light controls and switches I have around the house to get the controller to that new role.

Unfortunately Arduino hardware is becoming more and more of a problem due to limited resources. RAM is ridiculously low (8K on Mega), CPU is pretty slow (so even parsing few hundreds kilobytes file becomes a challenge), and even simple tasks like driving decent 5"-7" display becomes pretty hard due to overhead.

I was thinking for some time about ways to overcome this problem. On one hand I like (and need) Arduino extensibility, but system resources are just too low. Arduino Due is an option, it is faster and has more RAM - but difference is not really that big. Rasberry Pi or Beagle Bone is another good option - they have way more power, although there are some problems with using sensors and devices that require precise timing - code running under high-level OS like Linux just cannot control IO with good precision.

I also looked at an interesting project called X86Duino, which is an Arduino-compatible board that uses X86 CPU running at 300 MHz. It does not use Linux or another high-level OS, and can provide precise timing, also system resources are way higher than what you could dream of in Arduino land - >20x CPU perf of Arduino, and many orders of magnitude more RAM. Sounds tempting.

But thinking more about it I came across another possible option - I can use a small, inexpensive Tablet as the controller, with Arduino board as the IO extender. There are readily available Android and even Windows mini-tablets with 7" screen for under $100, and this buys you complete system - powerful CPU (at least as good as RPi), 7" touch screen, WiFi, Bluetooth, Audio, camera/sensors and even built-in backup battery. If you simply try to assemble comparable set (e.g. RPi + 7" touch screen + WiFi/Bluetooth) - it will already cost you more than $100.
And to cover IO needs I can use a standard, inexpensive Arduino board - it can handle sensors and devices communication (essentially as an intelligent IO hub), and it can connect to the main device using readily available USB.

Both Android and Windows are good options, Android tablets are slightly cheaper (due to lower specs) but Windows devices have more power. I'm considering both options, but it appears to be a good way to move to a more powerful controller.