Tswarm pt 3 – The code for your node

On my github page you can find first draft of the micropython code for ‘slave’ nodes which are basically responsible only for pulling data from sensor every 5 minutes, and sending it to very simple API I have created with Django + REST framework, and run on Heroku.

Let’s talk more about it…

After flashing your ESP, you can get REPL prompt by connecting node through USB UART interface and using, for example, picocom. Just like this:

sudo picocom /dev/ttyUSB0 -b115200

You may need to experiment with baud rate, but 115200 seemed to work with all my modules. After that you are able to execute python code on your device, and just play with it a little bit.

Under the hood, micropython is using two very important files: boot.py (file content is executed on device boot up) and main.py (ran just after booting finishes) that we can override with standard python file operations (just put your code inside string and write it down – or upload it with software of your choose that works with UART). Remember that it’s not fully featured dev environment, so after opening a file, you have to close it. I find little sense in modifying boot.py. You may want to put some setup code there, but we may just do the same in main.py.

Anyway on tswarm_devices/slaves/ on my github page, you will find tree files:

  • main.py – which covers main loop
  • node.py – which helds all node code
  • config_sample.json – in which you can find sample config for our device

Let’s start with configuration

It’s a little bit overconfigurated. Not everything is used (at this time anyway). It should be pretty self explanatory, but just in case:

"modules": {
  "dht11": {
    "desc": "Temperature and humidity meter",
    "gpio": [2]
  }
 }

Modules part contains dict with all modules connected to this particular board, as well as their configuration. In case above, just gpio (in master node I also have screen module which has some more config)

Netconf part contains wifi config. Use static IP to save energy (it’s more efficient and faster than using DHCP). I’m also providing list of SSID, cause I have few different access points with the same password in my home. Depending on localization, module will try to connect with first one, if fail we will go to second one etc.

Svconf part is responsible for webservice configuration. Every node has it’s own URL to which it posts the data. URL_local is for local testing, used during development and troubleshooting. It also contains some credentials – it’s basically standard HTTP basic authentication. You probably don’t want to use it on unencrypted service, cause it’s worth as much as sending plaintext credentials in your post. Or maybe using ROT26 to protect them ;)

Here be dragons – let’s fianlly use those leds and gpios

In node.py you can see few functions that are responsible for board operation. First we are importing some useful stuff, loading config and configuring our thermal sensor:

import machine
import dht
import ujson
import network
import urequests
import esp

# modules initialization
f = open('config.json', 'r')
config = ujson.loads(f.read())
f.close()

# thermal
d = dht.DHT11(machine.Pin(config['modules']['dht11']['gpio'][0]))

Let’s move on. initialPowersave() function is for well… Saving power. So we are turning all LEDs off. Keep in mind that .low() and .high() states depends on how your board is connected. You may need to play with it a little bit, but it was pretty consistent through all my boards.

rgbLed(color) function is for handling three basic colors, and turning LED off. You have to use three pins to configure the color. I’ve added the table for different colors in previous post. It’s also in code comment.

networkSetup() – again, pretty self explanatory, but few things here. If you want to use your ESP as station instead of AP (default behavior after flashing board with micropython), you have to deactivate AP interface. After that we can pull up our station interface and try to connect with networks from our config. I’m also flashing some RGB LED colors just to know what is happening. Here is some network module documentation, I just want to point out two things: sta.status() == 1 means connecting, sta.status() != 5 means anything else than connected.

getThermalData()  is very simple. It reads DHT11 sensor and returns tuple with temperature and humidity.

uploadData() is function for handling our communication with webservice.

def uploadData():
  headers = {'authorization': ' Basic %s' % config["svconf"]["xauth"], 'content-type': 'application/json'}
  t = getThermalData()
  data = '{"name":"%s","data":"%s %s"}' % (config["svconf"]["login"], t[0], t[1])
  url = config["svconf"]["url"]

  print('webservice address: %s' % url)
  print('data to be sent: %s' % data)

  # PUT the data
  resp = urequests.put(url, data=data, headers=headers)
  if resp.status_code == 202:
    print('SUCCESS PUT: Data uploaded!')
  # if not POST the data
  elif resp.status_code == 405:
    resp = urequests.post(url, data=data, headers=headers)
    if resp.status_code == 202:
      print('SUCCESS POST: Data uploaded!')
    else:
      print('ERROR: Unable to connect with webservice! Code returned %s' % resp.status_code)
   elif resp.status_code == 423:
     return False
 return True

Firs we are setting up HTML headers and some JSON content. Prints are for debugging. Then we are using urequests library to send our data and headers to url from our config. We are sending data to RESTful API, so PUT, POST, GET, DELETE is available. If PUT fails, it means that node is not created in webservice, and we will try to POST the data (well, so maybe it’s not fully restfull design, but you can do it differently if you want). Code 423 it’s just my idea of stopping all nodes and breaking their loops (cause sometimes it’s not easy).

goToSleep(seconds) just goes to sleep for provided time period. ESP is using its own deepsleep function, don’t use the generic one. Keep in mind that you need to hook up GPIO16 with REST pin to allow board to wake up.

Where the magic happens

main.py is executed after every device boot. By every I mean also after waking up from deepsleep. I’ve decided to use main.py as orchestrator instead of putting all code inside it. So it’s very simple:

from node import *

initialPowersave()
wlan = networkSetup()
sleep = uploadData()

# if False, break deepsleep loop
if sleep:
  goToSleep()

first we are turning off all LEDs. Then we are connecting to our network. After that we are trying to upload some data to webservice. If I set ENV variable on my heroku app to return 423 status, sleep will have False value and board won’t go to sleep. If not, and 202 or anything else is returned, we are sleeping for five minutes.

Reason for this 423 status is, that during development I’ve messed something up, and had some problem with breaking this main.py loop before it went to sleep. It was just happening too fast. Ultimately i was able to kill it from REPL, but I’ve decided to have mechanism to stop all nodes by changing one ENV var on Heroku, just to simplify update process.

Summary

I hope this article helps and gives you some directions on how to tackle your board, and force it to do something useful. In the next part I will show the webservice code, and my idea of handling Django configuration on Heroku.

Cloudflare and Heroku – free SSL for your domain

In previous entry I have described the process of configuring heroku and cloudflare to work together and serve your app under custom domain, so it’s probably a good idea to read that first.

There is one more great thing about using cloudflare – even in free subscription they will give you SSL encryption. For free. And like in python – batteries included ;)

Enable free SSL

So ho  to turn this on? Go to CRYPTO tab in your cloudflare account, and select ‘Full’ option. In theory you can also use ‘Flex’, but I can’t see reason why, plus it’s causing a lot of trouble with heroku.

SSL settings on Cloudflare

You can scroll down and browse some of the other options. I’m using ‘Automatic HTTPS Rewrites’, just to be sure that I encrypt as much as I can.

Create page rules

After turning on encryption, we should redirect all potential http traffic to https. So let’s go to ‘Page Rules’ tab and do so. Click ‘Create Page Rule’, provide domain name with asterisk eg. http://*abouteverything.pl/* then click ‘Add s Setting’ and choose ‘Always Use HTTPS’.

Page rules for your site

Change DNS target

One final thing we have to do to make all this work – we have to change our DNS target from herokudns to herokuapp. Go to DNS tab, and edit your DNS records. It should point to your app address provided by heroku, so we have to change abouteverything.pl.herokudns.com to abouteverything.herokuapp.com

Changing a DNS target for CNAME record from herokudns to herokuapp

SSL won’t work with yourapp.com.herokudns.com. At least not the free one (correct me if I’m wrong). Of course you can try, but I had number of problems with infinite redirects etc.

Final thoughts

When I first try to configure this whole cloud stuff, I have encountered number of problems. First I’ve turned everything, including free SSL and pointed it to herokudns, and as you may imagine, it didn’t work (redirect loops etc.). When working with free account in cloudflare it’s very irritating, cause it all takes time (first to propagate, then to change some options), but it comes with the territory – after all, we are not paying them a cent.

In general, you should encrypt. Especially when it’s free. Not only google will position you better (just google: ‘google ssl rank’ – there is number of articles about it) but your ISP won’t be able to spy on you (well, they will know that you are on the page, but they don’t know what are you doing in there).

Cloudflare and Heroku – adding custom domain

Root cause

As all of you know, hooking up custom domain with cloud services can be pain it the butt. Long story short, it’s because whole idea of cloud (computing, hosting, whatever) is flexibility pushed to the boundaries. Scaling up and down, dynamic adding and removing resources when necessary, so in the end you are simply unable to point your finger at specific server in server room and say – “Here. This box is running my app.”. And forget about stable IP address. Or better – forget about IP address at all, cause your resources are spread on multiple boxes in multiple locations. At standard, heroku will give you with semi-static url to your app. That’s it.

On the other hand, custom domains and whole DNS system is almost as old as whole Internet concept, and are all about persistence. You want to be able to point your finger and know what is where. You have to set your A or AAAA record to specific IP address. And you can’t point A record to heroku provided subdomain – it’s against the spec, and simply won’t work.

So how to combine those two worlds? Simplest way is to use DNS record called CNAME which is basically an alias to other (canonical) domain/subdomain (ex. provided by heroku for your app). This works just fine with domains like blog.myapp.com, but what when we want our app to be available under myapp.com? Bummer, you can’t – again it’s, against the spec. You can’t use CNAME to resolve apex/root domain.

What to do when DNS spec is as old and outdated as author of this post? Well, just ignore the part that says “you can’t do it” and choose DNS provider that thinks the same. It’s called “CNAME flattening” and it’s awesome and easy to use feature, done automatically by Cloudflare.

Recipe

You will need:

  • Custom domain
  • Heroku account with app running (I was dumb and lost some time because i was trying to point to empty app this placeholder page you see after creating a new one – which seemed to be a good idea at the time, and believe me, it isn’t)
  • Cloudflare account with domain hooked up (just register there, and point your domain to DNS addresses provided by them)

Step 1

Go to your heroku account, select app from dashboard and go to settings. Under “Domains and certificates” you will find a “Add domain” button. Just click it and type your domain name. You will see that new domain will appear with DNS target looking like: myapp.com.herokudns.com

If you prefer to use heroku CLI you can use domains command.

ex.
> heroku domains:add myapp.com  # add domain
> heroku domains                # display domains connected to your app

DNS target is, well, the target for your DNS server obviously.

Custom domain configuration on your Heroku account

Step 2

Go to your Cloudflare account and select DNS button from top menu. In the form for adding new records, select CNAME, type your domain name into Name input, and myapp.com.herokudns.com in Domain name input. It might look a bit confusing, but that’s how the CNAME record works – domain name is the canonical name to which your alias will be resolved.

Correct DNS settings on Cloudflare account

Step 3

Depending on when you have pointed your domain to Cloudflare name servers it may take up to 24 hours to propagate. After some time, you can check if it works, and if you have app running it should.

If you left default heroku page with “There is nothing here, yet”, you will see cloudflare error. It’s because this default page isn’t returning http status 200 (it’s just fancy looking 502 ‘bad gateway’ error), despite it looks like valid html. After you start your app it should be fine.

More reading