Monday, December 19, 2011

Robot Mapping and Exploration using Websockets and HTML5

As promissed last week, here's an introduction video to "Robot Mapping and Exploration using Webockets and HTML5" Project. The memory on my phone got full unexpectedly just as I was going to show the map building features of the project. I'll post the rest on a later video.

Sunday, December 11, 2011

NEW PROJECT: Robot Exploration and Mapping using Websockets

For some time now, I have been working on this project but I have not published anything until now. The goal of the project is to use a small mobile robot to explore and map an area. The real cool thing about it is that the map is built in real time and can be viewed using any browser that supports HTML5 and websockets.
HTML5 is the fifth revision of the HTML language for presenting content on the world wide web and, as of November 2011 is still under development. In this project I make extensive use of a new HTML element, introduced with HTML5 called the canvas element. It is basically used to draw images that are then displayed on the browser. Specifically, I use three canvas elements on top of each other with the first displaying the calculated position of the robot, the second displaying the map and the third displaying the occupancy grid.

Also exciting is the use of websockets to link the client browser (real time map builder application) with the server. Wikipedia states that "websocket is a technology providing for bi-directional, full duplex communications channels, over a single Transmission Control Protocol (TCP) socket".

On this project I have adapted a python script that implements draft 76 of the websockets protocol (available only on safari as the time of writing - other reference browsers have support for websockets but are now on a posterior, more robust and secure version. I don't feel like reprogramming my server to work with the new version of chrome so I chose to use safari for windows to still be able to use the older code - hoep this doesn't get auto updated as y copy of chrome did!

The python code above also receives serial data from the robot (sent over wireless radio using an xbee module) and makes it available to the client application. Also running on the same machine, besides this websocket server is a http server also written in python. Both scripts are adapted from scripts freely available on the internet.

I also make heavy use of javascript as the client page uses it to display information real time on the screen.

On the robot, I use pololu robot base with two arduinos, one connected to the xbee shield and the other connected to a custom shield that I built and that basically takes care of all sensor connections and basic electronics to allow them to be connected to the arduino. Both boards exchange data over an i2C link.

Currently I can send data from the mobile robot to the browser, and I can control the robot manually through the browser on a proof of concept application of data moving both ways over a websocket connection.

I will leave you here now, in a week or so, I will post some screenshots of the system in action and perhaps a demo video.

In the meantime, let me know if you have any questions.

Saturday, December 10, 2011

NEW PROJECT: Open Source Internet of Things HW and SW

This is the first post on the subject of our latest project: An open Source HW and SW platform that collects sensor data and sends it to an online server for real-time manipulation by client systems. This is in line with pachube and thingspeak, two such platforms (you may recall I use thingspeak to store data from my weather station and to display sensor data as you can see here).

A general description of the objectives follows (mainly for the hardware, the part I'm mostly involved in):

1. arduino based main console with wired ethernet connection (version 1, planned to go on to wireless later)
2. arduino based sensor modules and control boards
3. sensor modules connect to the main console and require no configuration (plug and play)
4. the protopyte sensor modules will be:
a) temperature sensor
b) open door/ window sensor
c) humidity sensor and 
d) power/ current sensor
e) relay output board (at least one output available on the controller board, probably more)
5. HW as cheap as we can make it, all open source
6. At the end of the project we will have a prototype controller board and a prototype sensor board for each of the sensors on the list above
7. controller and sensor boards software open source
8. case study application: environment control of a server room
9. board configuration through built-in web interface

1. https: secure connection
2. branch of the project to work on tablet data display
3. other general specs later

1. server room environment monitoring and control
2. reef aquarium data storage, display, monitoring and system control
3. weather stations data storage, display, monitoring
4. others by the community

Available time: 6 months starting Dec 2011, to be completed on Jun 2012 with a prototype that can be manufactured.

A Technology Lab project of the University of Saint Joseph in Macau SAR, China.

Tuesday, October 4, 2011

Make a drive mirror on linux

A very brief post to quickly describe what I had to do to have an uncompressed automatic backup of my external hard drive on to another external hard drive of the same capacity.

I have an external USB 1TB hard drive where I store all media files from movies and music to family photos. Once I started having a pretty decent number of family photos I started to think that a backup was definitely in order so I got another 1TB external USB hard drive where I wanted to backup my data to. Here's what I wanted, along with the options that I considered, why they work or not for my application and finally my actual solution for this need:


RAID1 Simple Backup rsync

I do not need the files to be compressed since most of the files on my main HD are media and compressed anyway. I don't want to loose time with compression.


I think it compresses by default


I want to be able to remove anyone of the disks and the data is still available on the local computer.




I want to make incremental backups, i.e. data is only sync'ed not entirely copied over each time.




I'm not really interested in keeping several versions of the same files. What I delete on the main, gets deleted on the backup drive, what gets added to the main disk, gets added to the backup.




Any of the drives can just be unmounted and used on another machine with no extra software.




So, from the table above you can see which of the options was right for me. I ended up using the following command using rsync:

rsync -avh --delete /path/to/localfiles /path/to/externaldrive >>rsyncLog.txt 2>>rsyncErr.txt

This command tells rsync to copy the files in (-a) archive mode, basically keepin
g all the same as the original in terms of symbolic links, devices, attributes, permissions and ownership. It requests an increse in verbosity (-v), i.e. more information is generated by the program (useful for debugging and detecting potential problems and all numbers appear in human-readable format (-h). I additionally pipe the program output to two files, one is a simple log of actions taken and the other contains any potential error messages generated. The --delete option deletes files from the destination in case they have been deleted on the origin.

As I final step I use cron to run the command above everyday at 04:00am. To simplify I use gnome-schedule to add the command and set the trigger time.

It all works well. Hope this helps someone.

Thursday, August 11, 2011

WiFi Wireless Speakers Set

Following the development of the wired speaker project of which I prepared only a demo prototype that included the boards connected externally to a set of computer speakers, I have now ported that project to a wireless version where I replaced the original Ethernet board (the official arduino Ethernet shield) with the Wifly shield from Sparkfun (see below for detailed hardware description). All code is open source and available on

With this setup you can stream files wirelessly from your PC (using the wifly shield from sparkfun) or with a cable (using the official Arduino Ethernet shield) to a set of speakers anywhere you can get a connection to your LAN.

Here's a video showing the operation (as requested, please see more videos and photos of the construction at the bottom of he page):

As a reminder, the original project included the following:

1. server software to stream files to a remote speaker set (originally connected to the network using cables, but now wireless) - this server software is available for download from the google code pages - made with DELPHI, requires VLC 0.8.6 as streaming engine - also available from my google code pages for convenience because it may or may not work with other versions of VLC - note that the software is for PC only - tested on windows 7 and winXP.
Main server Window
2. client software where I can stream files to another PC on the network (also needs VLC on the client computer)

3. Arduino code for both the wired and the wireless version - files are well identified on the google code pages.

4. Extra library for interaction with SD cards at low level but not actually used on the final versions of the project (see description here, download also from the google code pages).

5. The speakers will be auto-detected by the server. You just need to switch them on, start a stream on the server, and the IP of the new speakers will show up on the list automatically.

6. The server supports at least 20 different speakers sets on the network and the selection of which speakers are going to play the stream is done speaker by speaker or by groups of speakers (the server has a configuration area where groups of speakers can be defined).
The manage speakers and groups window on the server.
Hardware you will need: (links below  to not because I particularly endorse them - although I do like them ;-) but because info on their site is plentiful and I did get their mp3 decoder chip and wifly shields:

Arduino Controller Board - tested with the duemilanove, probably works with the UNO, but that needs to be confirmed.

The old version of the mp3 decoder shield from sparkfun. May work with the new one since it uses the same decoder chip, but that needs to be tested.

For the WIRED version, I used the Official Arduino Ethernet Shield.

For the WIRELESS version, I used the Sparkfun Wifly Shield.

As mentioned above, all the code required is available on the pages - including the required version of VLC and the Wifly library version that was used for the project.

Things still not quite right:

1. Improve sound quality. Currently the system allows playback of streams with 32kbps only. Above that the sound is chirping and not good at all. After a lot of tests, I think that perhaps we are limited to that value because of the very modest specs of the hardware used.
2. The server software currently only streams on file at a time, hardly useful if you would like to have the speakers streaming songs ALL DAY LONG!!! Should not require a lot of modifications to have it stream a playlist - code is available, maybe you can do it?

More videos and Photos:
Inside the prototype speakers - The boards are tightly held inside using some foam (also serves as electrical isolation from the speaker surface just below) and compressed slightly by the speaker back cover.

WiFi speakers, completely wireless!! The connections you see between the speakers are for sound signal between the amplifier on one of the speakers and the other one, the sound signal from the decoder board to the amplifier, the power connection between the main speaker and the controller boards I added.
 Thankfully the original speakers had plenty of room to add stuff inside!!

I had to add the filtering capacitor because the main power from the amplifier was extremely noisy and it affected the output too much. You can still hear the noise now, but you really need to be trying. Why that oversized capacitor value? Simple: the only one I had laying around!

Sunday, June 26, 2011

The "Internet of Things"

On an older post I mentioned that I have migrated the output of the sensors on my weather station to thingspeak. is a service that make an API available to you that takes data from your sensors and then makes it available for your (and others if you so desire) use in ways that go from chart building or any other customized way you can think of.

There's a lot of potential in that. For instance, in my case, I used to generate my own charts (this link may not be active at the time you read this) for my weather station. It worked but in a way it was a waste of my time. I spent ma lot of time writing the code to generate the charts(even though I used a graph generating library I still had to prepare the data to be displayed - things like fetching the data from the database, extracting the highest, calculation of averages and things like that). I wanted to concentrate on the hardware, the Arduino programming and the analysis
 of data. For my own applications, this is where came in.

The idea is simple: make it easy for the user to upload sensor data to a server and then offer tools to manipulate that data while at the same time offering easy ways to make the raw data available in the case the user want to actually do something more customized that what can be done using the tools of the website.

To achieve the above, thingspeak developed an open source API that takes your data and makes it available to you later. Like in any other API, you need to follow a protocol (a way to interact with the API to post and retrieve data).

The API interaction is extremely simple; I was up and running in literally 5 minutes. To start, follow this simple steps (I use PHP to interact with the API, there's some other tutorials to teach you how to do the same with other platforms, including how to do it directly from the Arduino):

1. create an account on the website.

2. login and go to the channels tab

3. click the "create new channel" button

4. change the channel name to something that makes sense to you and for others (this will later show up on the list of available channels) - in my case I chose "Weather Station Macau SAR, China".

5. go through the list and input other data as you see fit - you can always go back and edit the information. Click on the "Make Public?" check box to make the data public and therefore giving others the chance to use your data on their own projects.

6. then, add your sensors to the Field list. For example, if you have only one sensor that is a temperature sensor, add "Temperature" to the field 1 text box. Then click "Create Channel".

At the end of the steps above you have created a channel that can be thought of as an opening door to your "region" of the thingspeak server. The next step is to use the API to load data to that "region".

To add data do the following (using PHP):

1. on return to the channels page

2. note the box on the right on the channels page and note the section "Sending Data" and specifically the following line:

3. Make a note of the sequence of characters that make up your API keys and identified above as "YOUR_KEY_HERE"

4. on your PHP code add the following lines:
$post_thingspeak = "".$temp.""; $response = file_get_contents($post_thingspeak); echo "| reply: $response"; Above, $temp is the variable that holds the temperature value. Replace "YOUR_KEY_HERE" by the characters that you noted on step 3.

And that is all you need to post to the API. After these steps are complete, it might be useful for debugging purposes to check your channel feed by going to the "Viewing Data" link on the channels page and checking the raw data being sent to the thinkspeak server. By the way this is also how you would access the data to be used on your own applications, but this is a subject for another post.

Hope this helps to cut the time of using the service using PHP from the 5 minutes it took me to 4 minutes.

Brilliant Manipulated Plastic Balls!

Here's a great project idea, that requires a great deal of work, better to be undertaken as a team. Anyway something of this scale might be a little over kill for most of us, but perhaps something smaller on the same lines could still be quite impressive.

Take a look at the video below and leave some comments in the comments area. The only thing I really don't get about this video is that it shows you the preparation phase (like getting the magnet, getting the electronics inside the balls) but then is quite brief on showing the actual thing working. I would happily take a full 10 minutes or more just looking at the movement of the balls!

INDUCTANCE - Physalia from F5 on Vimeo.

Saturday, June 25, 2011

Weather Station migrated to

I just migrated the output of my weather station to thingspeak is a start to the implementation of an "internet of things", much like the more famous pachube. A fundamental difference is that thingspeak is open source and pachube isn't (I found this article that says pachube is open source, but I can't find the source or anything that talks about the code being open on the pachube website). I'm an open source guy, so I had to go for thingspeak!

The main differences from the previous versions of the weather station software are not many really, basically:

/*Main Changes on this version:
1: Removed connection to LCD (there's a web interface anyway!)
2: Added luminosity sensor
4: Optimized the code a little
5: Cleaned up the code making it easier to read
6: recalculation for rain (5 sec sampling is too fast, 
I never got any rain readings. Increased to 30 seconds)
7: Added a bit more comments to make the sensor setup more clear

The required and revised code has been uploaded to the code page. And checkout the new graphs here.

Friday, May 27, 2011

ubuntu 10.04 guest on VMware with a WIN7 host; using the arduino IDE.

Sometimes I just spend hours and hours trying to do something because I can't believe that it cannot work! I thought there should be quite a number of people running linux as guest on VMware and then running the arduino IDE on it.

I can tell you that it all works very very well, including the exchange of serial data when the sketch is already loaded to the board. All is good. That is until you try to load a sketch to your arduino. When you try to do that you get:

avrdude: stk500_getsync(): not in sync: resp=0x00
avrdude: stk500_disable(): protocol error, expect0x14, resp=0x00

meaning there's exchange of info for a while but then something fails. the rx and tx lights on the board, light up briefly and then it fails.
And after spending a good part of a day going around with it, I finally start to become closer to giving up. Besides all the things I tried (it doesn't matter to list them here, because they obviously did not work!) the following links started to make it clear...
(in the page above, see the small print at the bottom: "Note: I tried to program using Linux on VMware on windows 7… but it was not working.") this is a user that is trying to use avrdude (the program the arduino IDE uses to upload the sketches to the board) under vmware, very similar setup to mine but earlier distro versions of ubuntu. It also does not work. win7 guest on (I think a win7 host). Slightly different problem of the arduino not being recognized on the guest OS, points for some trouble on the comms when using virtual machines to upload programs to the arduino board.

I spent a day on this and so far I have not found a solution for this problem. I will try a bit more because I just like to suffer, but I start to loose hope. Almost for sure, the problem seems to be coming from the communication phase between the board and avr-dude, some kind of reply that takes too long to come back from the board. I will keep looking for a solution, if you know of one, please write something on the comments area.

Thursday, May 19, 2011

Network Speakers Project Complete

I finally finished this project! I was able to:

1. develop very-low-cost hardware capable of receiving a network (TCP) stream and play it. The hardware consists of an arduino controller board, an ethernet shield that had to be slightly modified to work with the mp3 shield (see previous posts describing the connections of these components). Final arduino software is available here (note limit to the playback of 32 kbps bitrate maximum on the hardware client app).

2. develop a windows application written in Delphi that:
a) automatically adds speaker clients as they are added to the network
b) manages the speaker database (allows grouping of speakers, edit names and descriptions, delete and manual add).
c) sends only one stream to the network (to decrease bandwidth usage) but selects speakers or groups of speakers for stream playback.

3. Develop a windows client application that emulates the arduino hardware (in this implementation, this is NOT a general arduino emulator!)

The software described in 2. and 3. can be used without any HW client to stream audio (mp3) content over the network from a server to a client.

All software is available here (the latest versions are the top three on the list for each of the described components).

Saturday, May 7, 2011

New Personal Weather Station Webpage

Now you can access all the data collected by my weather station online, including the old graphs by connecting to my own personal server at this location.

I have been collecting and storing quite reliable weather data for over a year now and the weather station is getting quite a lot of interest. Just FYI, here are the list of upcoming developments I would like to implement:

1. Beautify the reports page.
2. Connect a barometric pressure sensor (I already have it!!)
3. Connect a simple photocell to quantify light levels.
4. Use the barometer to actually predict the weather
5. Allow for searches and graphs for user-selectable periods on the database.
6. Add a simple webcam and broadcast an image of the weather station sensors and their surroundings.

Slowly the improvements will come.

Tuesday, February 22, 2011

SDCARDmodded library

I was looking for raw SDcard code, of the type that does not create, write and/ or append files. I was looking for something simple that would allow me to increase the memory space of my arduino and would provide a simple interface to save bytes to the card.

There are quite a few SD card libraries but they are all way more complex than what I needed, but some days ago I came across the SDCARD library at the arduino forums. I love what carl47 writes in the forum:

It reads and writes directly to sectors on a sd card.
It's use is for mass storage on the arduino platform. 
It's not to store data and then load it(without a special program) to a PC.

It's a dream, exactly what I needed!

It had all I wanted but just a small problem: it defaulted to the basic SPI comms using D10 of the arduino as the SS pin. Furthermore, it used its own implementation of SPI comms. I needed to easily hange the SS pin to whatever Digital pin I wanted and I needed it all the work with the other SPI devices.

Because the other SPI devices are using the SPI library that comes with the arduino IDE, I had to change the SDCARD library to allow me to:

1. use the standard SPI.h lib instead of its own implementation
2. Allow me to use any digital pin

So I changed it to the SDCARDmodded library that you can download here (unzip, place folder on your arduino/libraries folder ans restart the IDE.

library tested with IDE022.

Have fun.

Wednesday, February 9, 2011

Location of Arduino's Build Directory

In case you're wondering, as I was, where does the arduino IDE save the .elf and .hex files and everything that it needs to compile your code, let me tell you it's here (for windows7 and  IDE 0022):


where the NNNNNNNNNNN is a sequence of numbers making up the .tmp directory. If you have a lot of things on the Temp folder, you may want to order it by "date modified" to quickly find your files.

Why is this important?

1. the .cpp.elf file will not show up on searches if it is on a Temp directory (I know, I looked and looked for it)
2. You need those files for things like Atmega emulators (see emulare for an arduino emulator - AtMega328, really - still trying to make the emulator work...)

Hope this helps.

Tuesday, January 25, 2011

Convert old ethernet shield to an SDCard compatible version

I had the need to find some more storage space for a project I'm working on. I really would like to use the SD card that now comes with the new ethernet shield, but I had no time to order one and wait for it to arrive. So I noticed that the old shield actually has a space to put an SDCard but the area is empty. Also, although there are obvious differences, the shields look pretty much the same. All of that led me to believe that there was a way to add an SDcard to the old ethernet shield version and convert it in terms of operation onto the new version. This is a short description about how I went on to do just that.
The old ethernet card. Note the solder points on the top but no SDCard slot in there! Begging for an upgrade!

The new ethernet card version with a microSD card slot installed.

First thing to do is to pull out the schematics for each of these cards and actually look for the differences:

old version
new version

The parts on the schematics that are of interest to us at the moment are the following:
for the old card (click the image to enlarge)

for the new card (click the image to enlarge)

First thing we note is that the DETECT, COM_TERM and WRITE_PROT circuits are the same. Everything else is pretty much different... so we have our first conclusion: thre will be no direct soldering of an SDCard slot on to the old board.

Ok, so it will not look nice and beautiful anymore but perhaps it can still work by simply copying the connections on the new board direct from an SDcard slot to the board headers. First thing to do then is to confirm the microSD card pinout. I simply used this image that I found on google images (from this website, another cool description to get the SDCard to work with an Atmega168):
Because I had a microSD card slot on an old device that I could de-solder out and because we can easily communicate with the card via SPI, I was interested on the bottom picture and the SPI column pinout. Cross checking this information with the new ethernet card schematics we can see that:

SDcard                  -----                Arduino

pin1                      -----     not connected
pin2 (CD/DAT3)*    -----     to the SD_CS signal (i.e. to J2, pin5 corresponds to D4 on the arduino)
pin3 (CMD/DI)*      -----     to the MOSI3 signal (i.e. to D11/ SPI MOSI on the arduino)
pin4 (Vdd)             -----     to the 3.3V feed from the arduino
pin5 (CLK)*           -----     to the SCK signal (i.e. to D13/ SPI SCK on the arduino)
pin6 (Vss)             -----     GND
pin7 (DAT0/DO)     -----     direct to the MISO (D12) signal on the arduino
pin8                     -----     not connected

Above note the 3 items with a * after them. that means that the connection is not direct to the arduino pins but it's fed through a voltage divider. This is simply because the SDCard logic is at 3.3V while the arduino's is at 5V, so we need to convert from one to the other and the voltage divider is an efficient and cheap way of achieving it (just for fun, you can verify through the voltage divider equation Vout = R2/(R2+R1)*Vin, if you make R2 = 2K2, R1 = 1K and Vin = 5V, then Vout = 3.4V which is the levels we need to work at).

So we also know that we need to add these voltage dividers to our external circuit. What confused me for a bit was the direct connection of the MISO line (D12) to the arduino. Originally I assumed I wasn't reading the schematics properly and actually connected a fourth voltage divider for that line. Well, it doesn't work! It has to be direct (any ideas?)

So I went ahead and built all of what I just described on a breadboard. As I also mentioned I pulled a microSD card slot from an old camera I had and soldered leads to each pin and then took the leads to the breadboard and connected the voltage divider and the arduino to it.

My test setup looks like this:
To make it work now without writing all your code from scratch, you need to use the sdfatlib that you can download from here. The better option after building the hardware is to follow this tutorial to get it to work (EDIT: this one is also helpful).

BUUUUUUUUUUUUT things are never this easy and now we come to the other problem (besides overall connections as described) that does not allow us to properly use the card (or any other SPI peripheral for that matter) with the old ethernet card. It all has to do with a hardware bug (may I call it a bug?) on the wiznet chip that comes with the shield. That chip is described as a SPI device but it doesn't do something just quite well: it does not release the SPI bus even if its CS (chip select) pin is telling it to do it. So you can't speak to anything else on the SPI bus... or can't you? Well, you can but you need to make a little change:

Sometime in the past Wiznet published some application notes on the wiznet ethernet chip (you can read that doc here) where it stated that:

"Multiple SPI Slave Usage

Basically, multiple SPI slave usage is the same as single SPI slave usage. One difference between other SPI slave devices compared to the W5100 is that the MISO output is continuously driven in the W5100 whether the /SCS is asserted as high or as low. As well, when the 5100 /SCS is asserted as high when using multiple slaves, other SPI devices cannot be read or written by the SPI master on the SPI BUS simultaneously. These problems will continue unless the recommendations listed below are followed.

- When accessing another device on the SPI BUS rather than the W5100, assert the SEN pin in the W5100 as low first, then access the other devices.
- When accessing the W5100, the SEN pin should be high."

So besides the CS signal, we need to implement another that is basically the inverse of the CS line. This other line is the SEN line as stated in the text above. Hum, but where can we have access to the SEN line? looking at the schematics we can see that that line appears on the PROG jumper, pin2 (see the bottom left corner of the schematics).
So, to control that SEN signal along with the CS signal (already implemented, we need only to choose a free output from the arduino and connect it to PROG jumper, pin2 and therefore to the SEN line. I chose to connect it to D7 and I soldered a jumper wire under the ethernet shield board connecting these two points. Any selection/ deselection of the ethernet chip now becomes:

//enable the ethernet chip on the SPI bus

void EtherSelect(){
  digitalWrite(ETHER_CS, LOW);
  digitalWrite(ETHER_SEL, HIGH);

//disable the ethernet chip on the SPI bus
void EtherDeselect(){
  digitalWrite(ETHER_CS, HIGH);
  digitalWrite(ETHER_SEL, LOW);

where ETHER_CS is pin10 (see schematics) and ETHER_SEL is pin 7 as I described above.

I know that the description above may seem a little confusing but it is very rewarding to have it working without having to buy a new shield.

All the best with your projects!

EDIT (7th May 2011): See my comment for this post for additional information.