Tswarm pt 3 – The code for your node

On my github page you can find first draft of the micropython code for ‘slave’ nodes which are basically responsible only for pulling data from sensor every 5 minutes, and sending it to very simple API I have created with Django + REST framework, and run on Heroku.

Let’s talk more about it…

After flashing your ESP, you can get REPL prompt by connecting node through USB UART interface and using, for example, picocom. Just like this:

sudo picocom /dev/ttyUSB0 -b115200

You may need to experiment with baud rate, but 115200 seemed to work with all my modules. After that you are able to execute python code on your device, and just play with it a little bit.

Under the hood, micropython is using two very important files: boot.py (file content is executed on device boot up) and main.py (ran just after booting finishes) that we can override with standard python file operations (just put your code inside string and write it down – or upload it with software of your choose that works with UART). Remember that it’s not fully featured dev environment, so after opening a file, you have to close it. I find little sense in modifying boot.py. You may want to put some setup code there, but we may just do the same in main.py.

Anyway on tswarm_devices/slaves/ on my github page, you will find tree files:

  • main.py – which covers main loop
  • node.py – which helds all node code
  • config_sample.json – in which you can find sample config for our device

Let’s start with configuration

It’s a little bit overconfigurated. Not everything is used (at this time anyway). It should be pretty self explanatory, but just in case:

"modules": {
  "dht11": {
    "desc": "Temperature and humidity meter",
    "gpio": [2]
  }
 }

Modules part contains dict with all modules connected to this particular board, as well as their configuration. In case above, just gpio (in master node I also have screen module which has some more config)

Netconf part contains wifi config. Use static IP to save energy (it’s more efficient and faster than using DHCP). I’m also providing list of SSID, cause I have few different access points with the same password in my home. Depending on localization, module will try to connect with first one, if fail we will go to second one etc.

Svconf part is responsible for webservice configuration. Every node has it’s own URL to which it posts the data. URL_local is for local testing, used during development and troubleshooting. It also contains some credentials – it’s basically standard HTTP basic authentication. You probably don’t want to use it on unencrypted service, cause it’s worth as much as sending plaintext credentials in your post. Or maybe using ROT26 to protect them ;)

Here be dragons – let’s fianlly use those leds and gpios

In node.py you can see few functions that are responsible for board operation. First we are importing some useful stuff, loading config and configuring our thermal sensor:

import machine
import dht
import ujson
import network
import urequests
import esp

# modules initialization
f = open('config.json', 'r')
config = ujson.loads(f.read())
f.close()

# thermal
d = dht.DHT11(machine.Pin(config['modules']['dht11']['gpio'][0]))

Let’s move on. initialPowersave() function is for well… Saving power. So we are turning all LEDs off. Keep in mind that .low() and .high() states depends on how your board is connected. You may need to play with it a little bit, but it was pretty consistent through all my boards.

rgbLed(color) function is for handling three basic colors, and turning LED off. You have to use three pins to configure the color. I’ve added the table for different colors in previous post. It’s also in code comment.

networkSetup() – again, pretty self explanatory, but few things here. If you want to use your ESP as station instead of AP (default behavior after flashing board with micropython), you have to deactivate AP interface. After that we can pull up our station interface and try to connect with networks from our config. I’m also flashing some RGB LED colors just to know what is happening. Here is some network module documentation, I just want to point out two things: sta.status() == 1 means connecting, sta.status() != 5 means anything else than connected.

getThermalData()  is very simple. It reads DHT11 sensor and returns tuple with temperature and humidity.

uploadData() is function for handling our communication with webservice.

def uploadData():
  headers = {'authorization': ' Basic %s' % config["svconf"]["xauth"], 'content-type': 'application/json'}
  t = getThermalData()
  data = '{"name":"%s","data":"%s %s"}' % (config["svconf"]["login"], t[0], t[1])
  url = config["svconf"]["url"]

  print('webservice address: %s' % url)
  print('data to be sent: %s' % data)

  # PUT the data
  resp = urequests.put(url, data=data, headers=headers)
  if resp.status_code == 202:
    print('SUCCESS PUT: Data uploaded!')
  # if not POST the data
  elif resp.status_code == 405:
    resp = urequests.post(url, data=data, headers=headers)
    if resp.status_code == 202:
      print('SUCCESS POST: Data uploaded!')
    else:
      print('ERROR: Unable to connect with webservice! Code returned %s' % resp.status_code)
   elif resp.status_code == 423:
     return False
 return True

Firs we are setting up HTML headers and some JSON content. Prints are for debugging. Then we are using urequests library to send our data and headers to url from our config. We are sending data to RESTful API, so PUT, POST, GET, DELETE is available. If PUT fails, it means that node is not created in webservice, and we will try to POST the data (well, so maybe it’s not fully restfull design, but you can do it differently if you want). Code 423 it’s just my idea of stopping all nodes and breaking their loops (cause sometimes it’s not easy).

goToSleep(seconds) just goes to sleep for provided time period. ESP is using its own deepsleep function, don’t use the generic one. Keep in mind that you need to hook up GPIO16 with REST pin to allow board to wake up.

Where the magic happens

main.py is executed after every device boot. By every I mean also after waking up from deepsleep. I’ve decided to use main.py as orchestrator instead of putting all code inside it. So it’s very simple:

from node import *

initialPowersave()
wlan = networkSetup()
sleep = uploadData()

# if False, break deepsleep loop
if sleep:
  goToSleep()

first we are turning off all LEDs. Then we are connecting to our network. After that we are trying to upload some data to webservice. If I set ENV variable on my heroku app to return 423 status, sleep will have False value and board won’t go to sleep. If not, and 202 or anything else is returned, we are sleeping for five minutes.

Reason for this 423 status is, that during development I’ve messed something up, and had some problem with breaking this main.py loop before it went to sleep. It was just happening too fast. Ultimately i was able to kill it from REPL, but I’ve decided to have mechanism to stop all nodes by changing one ENV var on Heroku, just to simplify update process.

Summary

I hope this article helps and gives you some directions on how to tackle your board, and force it to do something useful. In the next part I will show the webservice code, and my idea of handling Django configuration on Heroku.

TSwarm pt 2 – ESP12 yellow board hardware and micropython basics

After playing a little bit with all the junk mentioned in previous post, especially ESP12 board itself, I have decided to put together some of my initial findings in single place – just because compiling all of it took some time and a little bit of exploration, and maybe will save you some time while doing own projects.

ESP12 yellow board - pinout

Connecting to UART

Connecting yellow board to UART USB interface I’ve bought was fairly simple. Just use some wires to connect RXD, GND and TXD pins from the board, with corresponding ones on UART. You will find three dedicated pins on board that are best to use.

For power source, I’m just using three AA batteries. It is possible to power this thing from USB through UART interface, but ESP can be picky when i comes to power source, and I’ve red somewhere that it can even burn your USB port. I’ve decided to not try this one, mainly because I don’t trust UART dongle I’ve bought. Batteries are cheap, laptops are not. And it’s generally good idea to work with thing, and test it in target configuration – so my sensor swarm will be powered by IKEA AA batteries during both, development and real-life usage.

Flashing the thing

After connecting board to my laptop, I’ve followed manual on micropython page (make sure that flash jumper is on board). Long story short – you will have to:

  • install esptool
pip install esptool
  • erase flash
esptool.py --port /dev/ttyUSB0 erase_flash
  • grab your micropython version from the download page and write it to board flash (for my yellow boards I need to use -fm dio parameter)
  • esptool.py --port /dev/ttyUSB0 --baud 115200 write_flash --flash_size=detect -fm dio 0 esp8266-20170108-v1.8.7.bin

    Give it some time to flash. After it’s finished, you can remove flash pin, and follow manual from micropython page to get REPL prompt. I won’t be copy-pasting more from the oficial manual manual, so…

Let’s cut to the chase

Here is some general stuff that might be useful when playing with the board, sensor and screen.

  • I have checked few boards of mine, and all of them have following GPIO pins connected to RED LEDS (order from bottom of the image to top): 2, 0, 4, 5, 14, 16
  • RGB led is controlled by three GPIO PINs: 12, 13, 15. Here are the colors:
  • p12 p13 p15
     0   0   0   - off
     1   0   0   - green
     0   1   0   - blue
     0   0   1   - red
     1   1   0   - light-blue
     0   1   1   - violet
     1   0   1   - lime-green (yellowish)
     1   1   1   - white(ish)
  • Using DHT (11 or 22):
  • import dht
    import machine
    
    d = dht.DHT11(machine.Pin(2))
    d.measure()
    d.temperature() # eg. 23 (°C)
    d.humidity() # eg. 41 (% RH)
  • Example use of OLED screen:
  • import ssd1306
    i2c = machine.I2C(-1, machine.Pin(4), machine.Pin(5))
    oled = ssd1306.SSD1306_I2C(128, 64, i2c) # width, height, screen
    
    oled.contrast(contrast=155) #default 255
    
    oled.fill(0) # 1 fill white, 0 fill black = off
    oled.text('temp: ' + str(d.temperature()), 0, 0) # posX, posY
    oled.text('humi: ' + str(d.humidity()), 0, 10)   # posX, posY
    oled.invert(True) # invert screen
    oled.show() # display screen

    Until you call oled.show() nothing will be printed

  • Connecting GPIO16 with REST allows you to use ESP deepsleep function (to be precise, it allows you to wake up from deepsleep). When board wakes it will run all of the scripts that it would during normal powerup (so boot.py and main.py). According to the ESP specs, deepsleep drains around 10uA.
  • import esp
    esp.deepsleep(seconds * 1000000)
  • There is no way to programmatically turn off PW LED (power indicator led), so if you need low power solution (cause this led will drain most of your battery power), just use knife or something sharp – it should pop easily without damaging the board

Summary

For now I’m really impressed by this tiny thing. For me this project is much more fun than playing with RaspberyPi (which is, in my opinion, just overkill in projects like this). I have done some initial coding, mainly for ‘slave’ units that will connect to the network, get temperature and humidity, send this data to dango-based kinda-webservice and then go to sleep for five minutes. Now I’m trying to tackle the security, so I would like to use SSL, do some authentication that will send CRSF token from django to the unit, re-send this token while posting the data. That’s the idea for now.

 

TSwarm pt 1 – How I Learned to Stop Worrying and Love the Aliexpress

[captain obvious mode on]

You can buy a lot of crap on Aliexpress. Seriously – A LOT. But you can also buy stuff you need for fraction of the price you need to pay in your country (well, I guess if you live in China it’s not the case). And if you are a geek – you are in heaven, cause most of the world electronics comes from there anyway.

[captain obvious mode of]

My wife wanted few temperature and humidity meters, cause the old ones was crap and broke down after two years. They say: “Happy wife happy life”, so I’ve ordered five ESP-12E yellow boards, five DHT-11 modules, one 128×64 OLED screen, UART USB connector, few buttons and bunch of wires.

Not sure if this is what she meant, but hell, this will be fun project :D

It came two days ago, nicely packed. So I’ve played with it a little bit – tried to connect it with my laptop, flash it with micropython, play with sensors, leds and screen. Just wanted to check if everything works as I expected and i didn’t break anything in the process.

Idea

The idea (for now) is to create sort of ‘sensor cloud’ that communicates with the central one (with display, buttons and also sensor) which orchestrate the whole thing, displays stuff and also pushes the data online for reading it through our phones (website, or maybe a simple app for android?). I will see how it plays during development, cause i have concerns with security etc. cause I don’t want to add 5 more devices to the insecure IOT cloud that will DDoS someone

Why this hardware?

  • ESP-12E yellow board – it’s cheap, has 4MB flash, all pins nicely available, bunch of leds (one rgb), fotoresistor, power stabilization and battery holder
  • DHT11 module – in general I prefer modules over bare sensors, cause I can focus on software instead of soldering and preparing boards – DHT11 is enough for indoor usage, maybe it’s not as accurate as DHT22, but i don’t really care if temperature is 22C or 21.8C, and we have ready python module for handling it
  • SSD1306 128×64 OLED display module – again, cheap, cool looking, python module ready
  • buttons and wires – just needed this, didn’t really care which one

And maybe the best reason for above – it’s all supported by micropython modules almost out of the box (module names: esp – for board, dht – for sensor, ssd1306 – for screen).

I will be adding more posts under tswarm tag and my projects/tswarm category as the project moves forward. Here is photo of some unpacked modules.

 

Cloudflare and Heroku – free SSL for your domain

In previous entry I have described the process of configuring heroku and cloudflare to work together and serve your app under custom domain, so it’s probably a good idea to read that first.

There is one more great thing about using cloudflare – even in free subscription they will give you SSL encryption. For free. And like in python – batteries included ;)

Enable free SSL

So ho  to turn this on? Go to CRYPTO tab in your cloudflare account, and select ‘Full’ option. In theory you can also use ‘Flex’, but I can’t see reason why, plus it’s causing a lot of trouble with heroku.

SSL settings on Cloudflare

You can scroll down and browse some of the other options. I’m using ‘Automatic HTTPS Rewrites’, just to be sure that I encrypt as much as I can.

Create page rules

After turning on encryption, we should redirect all potential http traffic to https. So let’s go to ‘Page Rules’ tab and do so. Click ‘Create Page Rule’, provide domain name with asterisk eg. http://*abouteverything.pl/* then click ‘Add s Setting’ and choose ‘Always Use HTTPS’.

Page rules for your site

Change DNS target

One final thing we have to do to make all this work – we have to change our DNS target from herokudns to herokuapp. Go to DNS tab, and edit your DNS records. It should point to your app address provided by heroku, so we have to change abouteverything.pl.herokudns.com to abouteverything.herokuapp.com

Changing a DNS target for CNAME record from herokudns to herokuapp

SSL won’t work with yourapp.com.herokudns.com. At least not the free one (correct me if I’m wrong). Of course you can try, but I had number of problems with infinite redirects etc.

Final thoughts

When I first try to configure this whole cloud stuff, I have encountered number of problems. First I’ve turned everything, including free SSL and pointed it to herokudns, and as you may imagine, it didn’t work (redirect loops etc.). When working with free account in cloudflare it’s very irritating, cause it all takes time (first to propagate, then to change some options), but it comes with the territory – after all, we are not paying them a cent.

In general, you should encrypt. Especially when it’s free. Not only google will position you better (just google: ‘google ssl rank’ – there is number of articles about it) but your ISP won’t be able to spy on you (well, they will know that you are on the page, but they don’t know what are you doing in there).

Cloudflare and Heroku – adding custom domain

Root cause

As all of you know, hooking up custom domain with cloud services can be pain it the butt. Long story short, it’s because whole idea of cloud (computing, hosting, whatever) is flexibility pushed to the boundaries. Scaling up and down, dynamic adding and removing resources when necessary, so in the end you are simply unable to point your finger at specific server in server room and say – “Here. This box is running my app.”. And forget about stable IP address. Or better – forget about IP address at all, cause your resources are spread on multiple boxes in multiple locations. At standard, heroku will give you with semi-static url to your app. That’s it.

On the other hand, custom domains and whole DNS system is almost as old as whole Internet concept, and are all about persistence. You want to be able to point your finger and know what is where. You have to set your A or AAAA record to specific IP address. And you can’t point A record to heroku provided subdomain – it’s against the spec, and simply won’t work.

So how to combine those two worlds? Simplest way is to use DNS record called CNAME which is basically an alias to other (canonical) domain/subdomain (ex. provided by heroku for your app). This works just fine with domains like blog.myapp.com, but what when we want our app to be available under myapp.com? Bummer, you can’t – again it’s, against the spec. You can’t use CNAME to resolve apex/root domain.

What to do when DNS spec is as old and outdated as author of this post? Well, just ignore the part that says “you can’t do it” and choose DNS provider that thinks the same. It’s called “CNAME flattening” and it’s awesome and easy to use feature, done automatically by Cloudflare.

Recipe

You will need:

  • Custom domain
  • Heroku account with app running (I was dumb and lost some time because i was trying to point to empty app this placeholder page you see after creating a new one – which seemed to be a good idea at the time, and believe me, it isn’t)
  • Cloudflare account with domain hooked up (just register there, and point your domain to DNS addresses provided by them)

Step 1

Go to your heroku account, select app from dashboard and go to settings. Under “Domains and certificates” you will find a “Add domain” button. Just click it and type your domain name. You will see that new domain will appear with DNS target looking like: myapp.com.herokudns.com

If you prefer to use heroku CLI you can use domains command.

ex.
> heroku domains:add myapp.com  # add domain
> heroku domains                # display domains connected to your app

DNS target is, well, the target for your DNS server obviously.

Custom domain configuration on your Heroku account

Step 2

Go to your Cloudflare account and select DNS button from top menu. In the form for adding new records, select CNAME, type your domain name into Name input, and myapp.com.herokudns.com in Domain name input. It might look a bit confusing, but that’s how the CNAME record works – domain name is the canonical name to which your alias will be resolved.

Correct DNS settings on Cloudflare account

Step 3

Depending on when you have pointed your domain to Cloudflare name servers it may take up to 24 hours to propagate. After some time, you can check if it works, and if you have app running it should.

If you left default heroku page with “There is nothing here, yet”, you will see cloudflare error. It’s because this default page isn’t returning http status 200 (it’s just fancy looking 502 ‘bad gateway’ error), despite it looks like valid html. After you start your app it should be fine.

More reading