OpenLog is a an open source datalogger developed by Nathan Seidle of Spark Fun Electronics.
It consists of an ATmega328 microcontroller interfaced to a SDcard or microSD card. It retains a high degree of compatability with Arduino hardware - and indeed, early prototypes were developed by adding a shield with an SD card socket to an Arduino.
OpenLog is a serial datalogger. It does exactly what it says on the tin, and records all serial data that it sees on it's serial Rx pin. It is compatible with FAT16 and latterly FAT32, and so can work with SD cards up to 2Gbyte and 16Gbyte respectively.
It has a fairly simple set of commands which allow you to read, write and append the files on the card, plus traditional DOS-like commands for changing directory, deleting files etc.
As part of the Arduino Slave Network project, I thought it would be neat to port OpenLog to a suitable serial slave, and have it act as a central fileserver for the rest of the network. As the slaves are somewhat limited in their RAM resource, and cannot hold much data locally, having a central storage facility will be an asset to the system.
In this way, slaves with sensors connected could record perhaps a hour's worth of data and then write it out to the communications network, where it could be loaded into the RAM buffer on the Master or Hub unit - but more importantly stored as a CSV file on the OpenLog slave.
With this in mind the first task was to make an OpenLog and get it working. I have a NuElectronics "Sensor" shield, which has an SDcard socket and a Dallas RTC on board. Most of the I/O lines on the NuElectronics board were directly compatible with those in the OpenLog design, apart for the chip select line, which is on Digital 5, rather than Digital 10 on OpenLog - which used D 5 for running a status LED.
The quick solution was to modify the OpenLog code to move the status LED down to Digital 2, and then set the pin mode of Digital 5 to be an input. I then hooked a wire jumper from Digital 10 to Digital 5 - so that D10 still selected the SD card, and D5 remained a high impedance input.
To test OpenLog, you need a terminal emulator programme. Microsoft no longer include a terminal with their bundled software, so you have to find one that works. I eventually chose Tera Term 4.67 which seems to work well. I had no success with Termite or the Arduino Terminal window - neither of which appear to send CRTL z correctly.
OpenLog Storage Control.
To access data from the Slave RAM, there will have to be a command from the Master which requests the slave to write a block of data. The Master can also be used to create the filenames for the slave data to be stored and put the OpenLog into append mode at the right time to receive the slave data.
Suppose we want to create a file called temperature1.csv to accept temperature data from Slave 1.
The Master will have created this filename using the new command:
new temperature1.txt
Then before the request to the slave for data, it should issue an append command:
append temperature1.txt
Followed by the command to slave 1 to send its RAM contents eg. write out the first 1024 16bit integers from RAM:
w,1,0,1023
The slave will then write these as comma separated integer variables finishing with a ? to inform the Master that it has finished the command process. The Master then needs to send the escape sequence to the OpenLog so that the logging to temperature1.csv is closed.
The escape sequence could always be incorporated into the end of the data issued by the slave - this would then need no intervention from the Master to close the appended file.
Tuesday, September 07, 2010
Monday, September 06, 2010
Applying the Slaves
The Arduino Slave is designed to be simple to make and very low cost. Just an ATmega328 microcontroller, a 16MHz crystal and a reset circuit. In addition a 74AHC125 tristate buffer is used to control access to the serial network and ack as a buffer and line driver. One digital output line is needed to control the tristate buffer - leaving all of the remaining I/O lines for the Slave application.
The primary function of the slaves is for remote distributed monitoring and control tasks. Slaves are used to monitor changes in physical quantities - such as temperature and then perform some related control task. If the role of the slave is merely for monitoring a changing quantity - such as gas meter pulse counting, then the Slave acts as a small part of a multi-channel datalogger.
The Slaves are intended to run semi-autonomously. For example if a Slave is running a room temperature monitoring and control application, it is expected to do this without regular updates from the hub. It will continue to maintain the temperature set point and control the radiator valve accordingly until it receives an update to the set point form the Hub.
RAM resources on the '328 microcontroller are limited to 2K, of which about 1.5K will be allocated as a buffer to allow the slave to record up to 1 hour of data, and then feed this forward to the hub in the form of a CSV file sent across the network. Central to this store and feedforward mechanism working is the implementation of the OpenLog opensource datalogger on the Hub. A slave could take temperature readings every 6 seconds for an hour, package them up into a CSV file and then output them to the network in about 5 seconds at 9600 baud - in the time between taking consecutive temperature readings.
The datastreams will be organised as channels. A channel is a series of sampled readings taken from a time variant physical variable - for example "living room temperature", "electricity instantaneous power", "solar panel power output". A channel can be named, logged and then passed up to the Hub or laptop for subsequent further processing, graphing or display.
The primary function of the slaves is for remote distributed monitoring and control tasks. Slaves are used to monitor changes in physical quantities - such as temperature and then perform some related control task. If the role of the slave is merely for monitoring a changing quantity - such as gas meter pulse counting, then the Slave acts as a small part of a multi-channel datalogger.
The Slaves are intended to run semi-autonomously. For example if a Slave is running a room temperature monitoring and control application, it is expected to do this without regular updates from the hub. It will continue to maintain the temperature set point and control the radiator valve accordingly until it receives an update to the set point form the Hub.
RAM resources on the '328 microcontroller are limited to 2K, of which about 1.5K will be allocated as a buffer to allow the slave to record up to 1 hour of data, and then feed this forward to the hub in the form of a CSV file sent across the network. Central to this store and feedforward mechanism working is the implementation of the OpenLog opensource datalogger on the Hub. A slave could take temperature readings every 6 seconds for an hour, package them up into a CSV file and then output them to the network in about 5 seconds at 9600 baud - in the time between taking consecutive temperature readings.
The datastreams will be organised as channels. A channel is a series of sampled readings taken from a time variant physical variable - for example "living room temperature", "electricity instantaneous power", "solar panel power output". A channel can be named, logged and then passed up to the Hub or laptop for subsequent further processing, graphing or display.
Sunday, September 05, 2010
Towards the Networked Home
In the last few weeks I have been putting together the various tools I will need to make a distributed Home Monitoring Network. Previous posts have discussed most of the basics of this project, here I thought I'd summarise the project and what we are going to do with it over the next few months.
In essence, the network consists of several Arduino Slaves, all connected together on a simple wired serial network and communicating with a Master "Hub" device.
I'm doing this in conjunction with my friends at OpenEnergy with the intention of providing a demonstration system of what can be achieved with low cost hardware and a simple wired network. We are making the most use of open source hardware and software, and by keeping the system simple, we hope that it will be of interest to others. Later on, we might extend it to include some wireless nodes, but for now we want to keep the costs and complexity down.
Central to every node on our network is a small microcontroller board, based on the ATmega328 microcontroller, and firmware produced using the Arduino IDE. Keeping things compatible with Arduino hardware and firmware will ensure that others can try out our methods with minimum of fuss. At a bare minimum, our network could be used to connect an Arduino board to a laptop, so that it can be individually controlled over a few metres of cable by simple serial commands typed into a serial terminal emulator.
To extend the system beyond 1 board, some simple additional hardware is needed to ensure that only one Arduino Slave can access the network at a time. This involves fitting a 74AHC125 quad tristate buffer to each slave, such that only one will drive the network at a time. This device costs about 20p and is easy to add to an existing Arduino. The 74AHC125 and a 4 way screw terminal connector could be fitted to a protoshield board.
I've been programming my bootloader into new ATmega328 chips using another Arduino as a programmer, and in this way it keeps the cost of the Slave down. The basic Slave can be built for less than £5 which means that having several slaves on a network is still inexpensive.
Slaves send and receive data at 9600baud as simple comma separated variable (CSV) commands. The command set is compact, relatively easy to remember and can be typed from a serial terminal programme. The use of a CSV command means that commands can be readily interpreted using simple command interpreter code, and a low overall overhead to the application sketch. Additional commands are easy to write and can be added as the application needs them. Further details of the command set and application sketches will appear in a later post.
So with the basics of a wired network in place, it's time to start applying it to a real household application. In Snowdonia we worked on a monitoring and control system for a small solar pV installation, but at a planning meeting at the start of the weekend we discussed all the applications which could easily be handled with such a monitoring and control network. Here's a few suggestions in a non-exhaustive list for domestic and renewable energy applications:
1. Room by room, central heating controller
2. Gas consumption monitor
3. Whole house electricity monitor
4. Solar water heating controller
5. Solar Photovoltaic controller
6. Wind turbine controller
7. Battery Bank Management System
8. Generator controller
9. CHP controller
10. Greenhouse monitor/plant irrigation system
In essence, the network consists of several Arduino Slaves, all connected together on a simple wired serial network and communicating with a Master "Hub" device.
I'm doing this in conjunction with my friends at OpenEnergy with the intention of providing a demonstration system of what can be achieved with low cost hardware and a simple wired network. We are making the most use of open source hardware and software, and by keeping the system simple, we hope that it will be of interest to others. Later on, we might extend it to include some wireless nodes, but for now we want to keep the costs and complexity down.
Central to every node on our network is a small microcontroller board, based on the ATmega328 microcontroller, and firmware produced using the Arduino IDE. Keeping things compatible with Arduino hardware and firmware will ensure that others can try out our methods with minimum of fuss. At a bare minimum, our network could be used to connect an Arduino board to a laptop, so that it can be individually controlled over a few metres of cable by simple serial commands typed into a serial terminal emulator.
To extend the system beyond 1 board, some simple additional hardware is needed to ensure that only one Arduino Slave can access the network at a time. This involves fitting a 74AHC125 quad tristate buffer to each slave, such that only one will drive the network at a time. This device costs about 20p and is easy to add to an existing Arduino. The 74AHC125 and a 4 way screw terminal connector could be fitted to a protoshield board.
I've been programming my bootloader into new ATmega328 chips using another Arduino as a programmer, and in this way it keeps the cost of the Slave down. The basic Slave can be built for less than £5 which means that having several slaves on a network is still inexpensive.
Slaves send and receive data at 9600baud as simple comma separated variable (CSV) commands. The command set is compact, relatively easy to remember and can be typed from a serial terminal programme. The use of a CSV command means that commands can be readily interpreted using simple command interpreter code, and a low overall overhead to the application sketch. Additional commands are easy to write and can be added as the application needs them. Further details of the command set and application sketches will appear in a later post.
So with the basics of a wired network in place, it's time to start applying it to a real household application. In Snowdonia we worked on a monitoring and control system for a small solar pV installation, but at a planning meeting at the start of the weekend we discussed all the applications which could easily be handled with such a monitoring and control network. Here's a few suggestions in a non-exhaustive list for domestic and renewable energy applications:
1. Room by room, central heating controller
2. Gas consumption monitor
3. Whole house electricity monitor
4. Solar water heating controller
5. Solar Photovoltaic controller
6. Wind turbine controller
7. Battery Bank Management System
8. Generator controller
9. CHP controller
10. Greenhouse monitor/plant irrigation system
Bootloading Arduinos
What I really love about the Arduino is that it is so accessible and comes with all the utilities to make it work. For example, it takes just minutes for a Newcomer to connect up an Arduino and flash a LED. I think this accessability and ease of use across all the main platforms is one of it's key attractions - and something that other vendors of microcontroller kit have failed to appeciate or capitalise upon.
Once you gain more experience of programming, you can choose to move away from the "safety net" offered by the Arduino IDE, and use the GCC compiler and AVRdude tools directly. However for most simple applications there is no problem just using the Arduino flavour of the C language - provided you don't want to do anything too fast or clever.
One of the nice features is the ability to use an Arduino to program the bootloader into a new IC. Whilst pre-loaded chips are available from a number of suppliers - they charge a premium for these - and if you are making a lot of Arduino based devices - it's much cheaper to program your own - using the easy to use "Burn Bootloader" tool that comes as standard with the Arduino IDE. We used this for the first time on the Snowdonia Mega-Hack Session and found it to be quick and reliable.
Using this utility, I programmed up 10 ATmega328 ICs with the Arduino bootloader in under 30 minutes, and for a cost of about £3.30 per chip (includes VAT and postage from CoolComponents). This saved me approximately £2.90 per chip! CoolComponents are located near Clapham Junction Station and have keen prices on ATmega328s and have a very quick turnaround - 24hrs.
To make use of this Arduino ISP facility, you either have to have 2 Arduino boards, or 1 Arduino and a small breadboard. I used the breadboard method as described in the Arduino Tutorial: Using the Arduino as an ISP
In order to make this device, you need to make up a clock crystal and reset circuit for a ATmega328 on a small breadboard and connect up the Vcc and Gnd pins. Then you use the existing Arduino board and connect Digitals 10 - 13 across to your target '328. This brings across MOSI, MISO, SCK and Digital 10 which is used to reset the target processor. You also need a feed of +5V and 0V from the Arduino board to power your target.
David A. Mellis's programming application also uses 3 other port lines for driving LED indicators. Digital 7 flashes to show that programming is underway, Digital 8 will light up an LED if there is an error and Digital 9 is a Heartbeat to show that the ISP application is running. You don't have to fit these LEDs - but if you are doing this for the first time, it's a good confidence booster that all is well.
During programming the Rx and Tx LEDs on the Arduino board flash rapidly as the serial data is sent from the PC. The LED on digital 13 also flashes quite quickly. At the end of the programming session, the target microcontroller is reset and starts to execute a LED flashing routine on digital 13 - which shows up as a faint slow flash on the programming Arduino LED. This confirms that the bootloader has been programmed and that it is running code. This is a neat feature, as when it comes to use these bootloaded ICs in DIY Arduinos, the LED flashing code will still be present to tell you that the processor is running.
The picture above shows my bootloading breadboard - not the tidiest of layouts, but works fine. There are just 6 wires to connect across from the Arduino, or 7 if you want the Error LED. The breadboard has a 16MHz crystal, two 22pF loading capacitors and a 10K resistor on the reset line. If you are actively building DIY Arduino hardware, you will have all these components to hand anyway!
Now that I'm boodloading my own mega328 chips I think that the cost of a DIY Arduino Slave has dropped to about £4.50 a unit.
Saturday, September 04, 2010
Solar Stuff
Micro-inverters from Enphase Energy
A small inverter fitted directly to the back of the solar pV panel - one per panel.
Performs Maximum Power Point Tracking (MPPT) on a panel by panel basis, leading to higher overall system efficiency and compensating for panels that might be partially shaded.
Enphase also do a neat little display which shows you solar panel and room thermostat data. It would certainly be a good idea to be able to monitor all your household energy systems from a common display. These devices communicate via Powerline Carrier (PLC).
Perhaps it would be possible to "roll your own" display using the 2.8" colour touch screen shield from NuElectronics.
A small inverter fitted directly to the back of the solar pV panel - one per panel.
Performs Maximum Power Point Tracking (MPPT) on a panel by panel basis, leading to higher overall system efficiency and compensating for panels that might be partially shaded.
Enphase also do a neat little display which shows you solar panel and room thermostat data. It would certainly be a good idea to be able to monitor all your household energy systems from a common display. These devices communicate via Powerline Carrier (PLC).
Perhaps it would be possible to "roll your own" display using the 2.8" colour touch screen shield from NuElectronics.
Autumn Almanac
It's been a short week what with the Bank Holiday Monday, and my new employer's policy of closing the firm at lunchtime on Fridays. With us rapidly moving into the Autumnal month of September, here's a round-up of some of the things I have been inspired with this week.
New Toys.
I've been tinkering with a new Texas Instruments DSP dev-board at work, which will become part of a dc motor control system. I first used TI DSPs back in the late 1980s with the TMS320 series, and wow have they come on in performance since then.
The dev-board is based around their "Delfino" DSP which is a 32bit floating point DSP capable of 150MHz clock speeds. It has 512K of program space, 68K of RAM and 64 general purpose I/O pins - 18 of which can be PWM for driving power stages. It also has 16, 12 bit ADC inputs, which can be run in two banks of 8, for making simultaneous measurements - such as voltage and current, for calculating instantaneous power.
The Delfino is mounted on a small strip of PCB known as a "control card", which brings out 100 pins in a common footprint. This makes it easy to swap control cards for any of the DSPs in the TI range, and ensure functional pin compatability - a neat idea. It removes the user from the difficulties of handling 176 pin ball grid arrays (BGAs), and it means that you can develop your application on one of the larger parts and then move down to a smaller, more cost effective part, later, when the code is finalised.
Putting the processing elements onto a common footprint is a great way of taming what would otherwise be a difficult technology. The raw technology is grouped into a series of manageable ports or interfaces, and the user is presented with a simplified programming model.
On a simpler level, - this is exactly what the Arduino does. It simplifies the microcontroller's interfaces into a series of easily understood pin functions, and presents an extra programming layer into the code which divorces the new user from the complexities of manipulating I/O at a register level. This makes the technology much more accessible to the new user, but at the expense of a speed reduction in I/O operations - compared to direct I/O register manipulation.
When the User becomes more experienced in programming, then the "stabilisers" can be removed. Learning to programme an Arduino in C will hopefully produce a whole new generation of embedded programmers - who have the confidence to get "down and dirty" to the hardware level. This is something that has perhaps been lacking in recent years, as coders generally develop applications in high level languages on sophisticated platforms such as laptops and mobile phones.
Networked Arduinos
A week after the Snowdonia Mega-Hack session, the dust is starting to settle on the networked Arduino project. Trystan has been developing firmware so that he can use the networked slave as part of his solar photovoltaic monitoring system. This was chosen as an example application as something that would illustrate the capabilities of the network slave as a means of monitoring pV panel volts, amps and watts and controlling a battery charging system.
The low cost serially connected slave would make an ideal candidate for a battery monitoring system (BMS). Using optically isolated serial comms, one slave could be applied to each 12V battery, in a larger battery bank or an electric vehicle battery. The low power requirements of each slave would allow it to be powered from the battery it was monitoring - with its I/O referenced to the 0V terminal of each battery.
Data Logging.
Another neat application which has come to light is the OpenLog Data Logger This is an ATmega328 interfaced to a 2Gbyte SD card. It has a simple serial interface, and it just sits an logs all the serial data it sees on its RxI pin. The source code and schematics are all open source, allowing this application to be easily incorporated into some home-made hardware.
One idea would be to use it to record "both sides" of the data on the Arduino serial network. The slaves could be interrogated at regular intervals for stored data, which they transmit to the network, and the OpenLog stores this data away to flash in a large text file. OpenLog comes with a small but neat collection of commands allowing data to be retrieved, files to be appended and erased. This seems like an easy method to get data from the slaves into permanent storage, from where it can be accessed by laptop, sent via ethernet to the web, or whatever.
2Gbytes is a huge amount of storage. Even sending continuous data at 9600baud, it would take more than 25 days to fill the SD card.
Central Storage and Real Time
The Arduino slaves are lacking in vast amounts of local storage. There is 2K of RAM on the ATmega328 microcontroller, and if just over half of that was available for data storage, it could hold about 600, 16 bit readings. Using a store and feed-forward method, the slave could hold say an hour's worth of instantaneous power readings, taken at 6 second intervals and then send the whole file up to the central storage. At 9600baud, this data transfer would take about 4 seconds - easy enough to do once every hour between taking sensor readings.
The problems of limited storage have been with us for much of the last 60 years of computing. Certainly until the late 1980s, memory was an expensive resource, and computer engineers found ways of connecting many users to the one central mainframe and sharing its memory resources between several users.
This is exactly the same situation we have with our network of distributed Arduino slaves - albeit on a very much smaller scale. We have to find a clever way of sharing central resources, such as file storage, realtime clock and ethernet connection between the slaves. By storing the slave data in RAM and then forwarding it to the central SDcard at regular intervals would provide a sensible compromise to storing data, without over-complicating the slaves with their own additional storage. The feed forward model could also be applied to the Master device - collecting data from the slaves storing in SD card and then forwarding this data to a laptop, or web connected server. This would give the Arduino network a degree of autonomy and not reliant on having a PC permanently switched on to service the data.
The NuElectronics Sensor Shield comes with RTC and SDcard for data storage. With a bit of juggling it should be possible to implement the OpenLog firmware onto this shield. The RTC timestamp could be distributed as a CSV packet across the wired network at regular intervals - so that slaves could synchronise their local time to the realtime.
CurrentCost Channels and Pachube Feeds.
One idea, starting to "bubble to the surface" is the concept of having a number of data channels which can be monitored. This is something which the CurrentCost energy monitors have implemented, where originally they had a separate channel for the energy monitored on each of the three phases of an electricity supply. The LCD display would allow basic channel information to be viewed.
The concept was extended to allow data from Individual Applance Monitors (IAMs), to present the energy used by an appliance such as washing machine or fridge to be given their own channel. The CC128 display unit was given a capacity of 10 channels, with the user being able to step between them using a button on the front panel. This approach allows other sensors to be added later to enhance a sensing network - and with the Current Cost DevBoard, enthusiasts can add their own sensors, linking back to the CC128 via their proprietry wireless link. Thanks to @jtonline and @yellowpark for letting me know about the CurrentCost Technical Blog. -in which Chris Dalby, a developer at CurrentCost introduces some of their technology to the Homecamp/Hacker community.
Incidently - for those interested in wireless networks, JeeLabs has introduced a kit for a wireless JeeNode. Its based on the ubiquitous ATmega328 and a wireless transceiver module. It presents 4 identical "ports" to which various sensors or devices can be attached. It's all open source and based on Arduino technology. At 17.50 Euros it seems good value for money, and the wireless module used is the same one which CurrentCost use, which to the experienced coder should open up some possibilites for integrating the two systems.
So if we consider any source of data, such as electricity instantaneous power, or room temperature, or solar panel output could be expressed as a data channel. The CurrentCost display allows basic monitoring of channels using its restricted LCD, but this could be extended to an application running on an O2 Joggler display or a laptop etc. With an application running on a laptop, iPhone or via the web, would give enhanced monitoring options such as historical graphing, comparing channels or using them for further processing or other applications such as Pachube - where several physical channels are combined together as a Pachube feed.
New Toys.
I've been tinkering with a new Texas Instruments DSP dev-board at work, which will become part of a dc motor control system. I first used TI DSPs back in the late 1980s with the TMS320 series, and wow have they come on in performance since then.
The dev-board is based around their "Delfino" DSP which is a 32bit floating point DSP capable of 150MHz clock speeds. It has 512K of program space, 68K of RAM and 64 general purpose I/O pins - 18 of which can be PWM for driving power stages. It also has 16, 12 bit ADC inputs, which can be run in two banks of 8, for making simultaneous measurements - such as voltage and current, for calculating instantaneous power.
The Delfino is mounted on a small strip of PCB known as a "control card", which brings out 100 pins in a common footprint. This makes it easy to swap control cards for any of the DSPs in the TI range, and ensure functional pin compatability - a neat idea. It removes the user from the difficulties of handling 176 pin ball grid arrays (BGAs), and it means that you can develop your application on one of the larger parts and then move down to a smaller, more cost effective part, later, when the code is finalised.
Putting the processing elements onto a common footprint is a great way of taming what would otherwise be a difficult technology. The raw technology is grouped into a series of manageable ports or interfaces, and the user is presented with a simplified programming model.
On a simpler level, - this is exactly what the Arduino does. It simplifies the microcontroller's interfaces into a series of easily understood pin functions, and presents an extra programming layer into the code which divorces the new user from the complexities of manipulating I/O at a register level. This makes the technology much more accessible to the new user, but at the expense of a speed reduction in I/O operations - compared to direct I/O register manipulation.
When the User becomes more experienced in programming, then the "stabilisers" can be removed. Learning to programme an Arduino in C will hopefully produce a whole new generation of embedded programmers - who have the confidence to get "down and dirty" to the hardware level. This is something that has perhaps been lacking in recent years, as coders generally develop applications in high level languages on sophisticated platforms such as laptops and mobile phones.
Networked Arduinos
A week after the Snowdonia Mega-Hack session, the dust is starting to settle on the networked Arduino project. Trystan has been developing firmware so that he can use the networked slave as part of his solar photovoltaic monitoring system. This was chosen as an example application as something that would illustrate the capabilities of the network slave as a means of monitoring pV panel volts, amps and watts and controlling a battery charging system.
The low cost serially connected slave would make an ideal candidate for a battery monitoring system (BMS). Using optically isolated serial comms, one slave could be applied to each 12V battery, in a larger battery bank or an electric vehicle battery. The low power requirements of each slave would allow it to be powered from the battery it was monitoring - with its I/O referenced to the 0V terminal of each battery.
Data Logging.
Another neat application which has come to light is the OpenLog Data Logger This is an ATmega328 interfaced to a 2Gbyte SD card. It has a simple serial interface, and it just sits an logs all the serial data it sees on its RxI pin. The source code and schematics are all open source, allowing this application to be easily incorporated into some home-made hardware.
One idea would be to use it to record "both sides" of the data on the Arduino serial network. The slaves could be interrogated at regular intervals for stored data, which they transmit to the network, and the OpenLog stores this data away to flash in a large text file. OpenLog comes with a small but neat collection of commands allowing data to be retrieved, files to be appended and erased. This seems like an easy method to get data from the slaves into permanent storage, from where it can be accessed by laptop, sent via ethernet to the web, or whatever.
2Gbytes is a huge amount of storage. Even sending continuous data at 9600baud, it would take more than 25 days to fill the SD card.
Central Storage and Real Time
The Arduino slaves are lacking in vast amounts of local storage. There is 2K of RAM on the ATmega328 microcontroller, and if just over half of that was available for data storage, it could hold about 600, 16 bit readings. Using a store and feed-forward method, the slave could hold say an hour's worth of instantaneous power readings, taken at 6 second intervals and then send the whole file up to the central storage. At 9600baud, this data transfer would take about 4 seconds - easy enough to do once every hour between taking sensor readings.
The problems of limited storage have been with us for much of the last 60 years of computing. Certainly until the late 1980s, memory was an expensive resource, and computer engineers found ways of connecting many users to the one central mainframe and sharing its memory resources between several users.
This is exactly the same situation we have with our network of distributed Arduino slaves - albeit on a very much smaller scale. We have to find a clever way of sharing central resources, such as file storage, realtime clock and ethernet connection between the slaves. By storing the slave data in RAM and then forwarding it to the central SDcard at regular intervals would provide a sensible compromise to storing data, without over-complicating the slaves with their own additional storage. The feed forward model could also be applied to the Master device - collecting data from the slaves storing in SD card and then forwarding this data to a laptop, or web connected server. This would give the Arduino network a degree of autonomy and not reliant on having a PC permanently switched on to service the data.
The NuElectronics Sensor Shield comes with RTC and SDcard for data storage. With a bit of juggling it should be possible to implement the OpenLog firmware onto this shield. The RTC timestamp could be distributed as a CSV packet across the wired network at regular intervals - so that slaves could synchronise their local time to the realtime.
CurrentCost Channels and Pachube Feeds.
One idea, starting to "bubble to the surface" is the concept of having a number of data channels which can be monitored. This is something which the CurrentCost energy monitors have implemented, where originally they had a separate channel for the energy monitored on each of the three phases of an electricity supply. The LCD display would allow basic channel information to be viewed.
The concept was extended to allow data from Individual Applance Monitors (IAMs), to present the energy used by an appliance such as washing machine or fridge to be given their own channel. The CC128 display unit was given a capacity of 10 channels, with the user being able to step between them using a button on the front panel. This approach allows other sensors to be added later to enhance a sensing network - and with the Current Cost DevBoard, enthusiasts can add their own sensors, linking back to the CC128 via their proprietry wireless link. Thanks to @jtonline and @yellowpark for letting me know about the CurrentCost Technical Blog. -in which Chris Dalby, a developer at CurrentCost introduces some of their technology to the Homecamp/Hacker community.
Incidently - for those interested in wireless networks, JeeLabs has introduced a kit for a wireless JeeNode. Its based on the ubiquitous ATmega328 and a wireless transceiver module. It presents 4 identical "ports" to which various sensors or devices can be attached. It's all open source and based on Arduino technology. At 17.50 Euros it seems good value for money, and the wireless module used is the same one which CurrentCost use, which to the experienced coder should open up some possibilites for integrating the two systems.
So if we consider any source of data, such as electricity instantaneous power, or room temperature, or solar panel output could be expressed as a data channel. The CurrentCost display allows basic monitoring of channels using its restricted LCD, but this could be extended to an application running on an O2 Joggler display or a laptop etc. With an application running on a laptop, iPhone or via the web, would give enhanced monitoring options such as historical graphing, comparing channels or using them for further processing or other applications such as Pachube - where several physical channels are combined together as a Pachube feed.
Friday, September 03, 2010
Monitoring your Heating System
A couple of months back, I had some ideas about Smarter Heating Controls - with the weather turning Autumnal, it's time to start thinking about these ideas again.
Here are a few tips and gizmos to help you put together a DIY boiler and heating monitor
Pipe Clip Thermistor Temperature sensors from Rapid Electronics
The Arduino code needed to linearise a thermistor and display it in degerees C
Wireless actuators for thermostatic radiator valves from Conrad Electronics
Low-cost pulse output water flow meter froom SeeedStudio Depot - these are otherwise stupidly expensive in the UK. Easily read using a pulse input on the Arduino.
Here are a few tips and gizmos to help you put together a DIY boiler and heating monitor
Pipe Clip Thermistor Temperature sensors from Rapid Electronics
The Arduino code needed to linearise a thermistor and display it in degerees C
Wireless actuators for thermostatic radiator valves from Conrad Electronics
Low-cost pulse output water flow meter froom SeeedStudio Depot - these are otherwise stupidly expensive in the UK. Easily read using a pulse input on the Arduino.
Wednesday, September 01, 2010
September - Time to Reflect
July and August have been busy months, what with starting a new job in July and developing a whole bunch of Arduino ideas with the guys from openenergymonitor.org, Pachube and the London Hackspace.
I thought I'd summarise the project so far - and think about where it's all heading.
Low Cost, Arduino Compatible Devices.
At the heart of this project is the popular, open source hardware Arduino microcontroller board. An 8-bit microcontroller with clock circuit and programming interface, which can be programmed in a C like language from an IDE which runs on Windows, Linux and Mac platforms. The Arduino and its derivatives have become very popular amongst developers, hackers and other creative communities. Whilst an Arduino costs about £12.50 from a number of vendors, the hardware is trivial and easily built on stripboard or breadboard for about £5 in components. Such was the attraction of having a number of low cost Arduinos, that in early August I considered the notion of DIY Arduinos having seen some examples produced by the FizzPop hackspace in Birmingham.
The Internet of Things.
At the end of July, I attended a MiniBar event in London discussing the Internet of Things. For some time I had an interest in the idea of a number of low cost smart sensors, all connected to the Internet and able to communicate with applications and each other. The Arduino already had an ethernet shield available, so I wondered just how cheap you could make a home-made, web connected Arduino. This led on to the "ether-duino" project - an Arduino compatible web server for £12. Whilst the hardware was relatively easy to build on a breadboard, the ethernet library to control the ethernet controller IC was a bit out of my depth, so I was delighted to receive assistance from Andrew Lindsay and John Crouchley - both of whom had worked with this controller and had some example sketches available.
Low Cost Web Connectivity.
For some time, I had been thinking on how these network connected microcontrollers would communicate with each other, and also with the internet, so by way of experiment I developed a serial command interpreter which allowed Arduino compatible hardware to be controlled with simple serial commands. It was not much of a step on from this to realise that if these serial commands could be passed across the internet or other network then it would be possible to get these network connected devices to talk to each other. Smart sensors could pass data and commands between similar devices across the wired network and the internet. So I came up with a simple method by which the web-enabled Arduinos could pass simple messages between each other using Pachube as a message broking service. This bit of experimental work came to fruition in the third week of August when I finally got it together and succeeded in getting one web enabled Arduino to control a LED on another similarly web connected device.
A Smart Sensor Network
Having succeeded to make a breadboard version, to prove the concept of the web connected Arduino, I worked on a pcb version, done in conjunction with the London Hackspace. These would be inexpensive boards made from conventional components which anyone with basic soldering skills could put together. They would provide the basic circuit to connect to the internet and allow experimentation with internet connectivity for smart sensors. Whilst making the prototype, I realised that one of the components on the board, a tristate buffer, had two spare gates, and these would make an excellent means to extend the communication network beyond the web, and allow several slave sensor boards to be connected to a network radiating out from the web connected Master unit.
Building the Network.
With these thoughts in mind, I travelled up to Snowdonia, for a weekend build session with the lads from openenergy and it was here that we were to share the ideas and develop the hardware and firmware for the network connected Arduinos. We built up a total of 4 slave sensors, two with displays, and proved that we could send data and commands down 300m of network cable. We used one of the network sensors to monitor and control a solar pV installation and lead acid battery monitor.
The Next Steps.
So in just a few weeks we have developed a system of low cost network sensors which can be accessed through the internet and send commands and data to each other. From the outset this project was to be experimental, and make use of readily available low cost components and be easy to build by anyone interested in the Arduino Project.
The techniques that have been used may not be optimal in terms of performance, but they are easy to follow, by anyone wishing to build something similar. Now that the fundamentals have been proven, it is time to make up a batch of low cost pcbs, which will support both the web connected Master, or the simpler network Slave. With the infusion of low cost hardware into the hacker community, plus simple firmware to control it, then hopefully the project will soon gain momentum.
I thought I'd summarise the project so far - and think about where it's all heading.
Low Cost, Arduino Compatible Devices.
At the heart of this project is the popular, open source hardware Arduino microcontroller board. An 8-bit microcontroller with clock circuit and programming interface, which can be programmed in a C like language from an IDE which runs on Windows, Linux and Mac platforms. The Arduino and its derivatives have become very popular amongst developers, hackers and other creative communities. Whilst an Arduino costs about £12.50 from a number of vendors, the hardware is trivial and easily built on stripboard or breadboard for about £5 in components. Such was the attraction of having a number of low cost Arduinos, that in early August I considered the notion of DIY Arduinos having seen some examples produced by the FizzPop hackspace in Birmingham.
The Internet of Things.
At the end of July, I attended a MiniBar event in London discussing the Internet of Things. For some time I had an interest in the idea of a number of low cost smart sensors, all connected to the Internet and able to communicate with applications and each other. The Arduino already had an ethernet shield available, so I wondered just how cheap you could make a home-made, web connected Arduino. This led on to the "ether-duino" project - an Arduino compatible web server for £12. Whilst the hardware was relatively easy to build on a breadboard, the ethernet library to control the ethernet controller IC was a bit out of my depth, so I was delighted to receive assistance from Andrew Lindsay and John Crouchley - both of whom had worked with this controller and had some example sketches available.
Low Cost Web Connectivity.
For some time, I had been thinking on how these network connected microcontrollers would communicate with each other, and also with the internet, so by way of experiment I developed a serial command interpreter which allowed Arduino compatible hardware to be controlled with simple serial commands. It was not much of a step on from this to realise that if these serial commands could be passed across the internet or other network then it would be possible to get these network connected devices to talk to each other. Smart sensors could pass data and commands between similar devices across the wired network and the internet. So I came up with a simple method by which the web-enabled Arduinos could pass simple messages between each other using Pachube as a message broking service. This bit of experimental work came to fruition in the third week of August when I finally got it together and succeeded in getting one web enabled Arduino to control a LED on another similarly web connected device.
A Smart Sensor Network
Having succeeded to make a breadboard version, to prove the concept of the web connected Arduino, I worked on a pcb version, done in conjunction with the London Hackspace. These would be inexpensive boards made from conventional components which anyone with basic soldering skills could put together. They would provide the basic circuit to connect to the internet and allow experimentation with internet connectivity for smart sensors. Whilst making the prototype, I realised that one of the components on the board, a tristate buffer, had two spare gates, and these would make an excellent means to extend the communication network beyond the web, and allow several slave sensor boards to be connected to a network radiating out from the web connected Master unit.
Building the Network.
With these thoughts in mind, I travelled up to Snowdonia, for a weekend build session with the lads from openenergy and it was here that we were to share the ideas and develop the hardware and firmware for the network connected Arduinos. We built up a total of 4 slave sensors, two with displays, and proved that we could send data and commands down 300m of network cable. We used one of the network sensors to monitor and control a solar pV installation and lead acid battery monitor.
The Next Steps.
So in just a few weeks we have developed a system of low cost network sensors which can be accessed through the internet and send commands and data to each other. From the outset this project was to be experimental, and make use of readily available low cost components and be easy to build by anyone interested in the Arduino Project.
The techniques that have been used may not be optimal in terms of performance, but they are easy to follow, by anyone wishing to build something similar. Now that the fundamentals have been proven, it is time to make up a batch of low cost pcbs, which will support both the web connected Master, or the simpler network Slave. With the infusion of low cost hardware into the hacker community, plus simple firmware to control it, then hopefully the project will soon gain momentum.