Tuesday, January 17, 2017

Testing the Super-Capacitor

We have included a 1F 5.5V super-capacitor on the new Mesh Extender board.

The idea of this is so that when power is lost, the unit has a couple of seconds to stop writing to the microSD card, and perhaps tidy things up to ensure a clean boot next time.

However, we haven't managed to get it to work properly yet.

The boards have a 20 Ohm resister in-line with the super-cap, so that there is no unmanaged in-rush current when charging the capacitor from empty.

The capacitor takes a few minutes to fully charge to about 4.8V (5V - the voltage of the zener diode that forms part of the power loss detection circuit).

At this point, it should be possible to then remove the power source, and have the unit operate powered solely by the super-capacitor, until its voltage level drops below about 3.3V.

GPIO24 is the line that reads our input-power-available signal (effectively the input power before passing through a zener diode, after which it charges both the super-capacitor, as well as powering the rest of the board.  To test the system, I configured that GPIO line for input, and had a little shell script that sits in a loop reading from it.

root@OpenWrt:/# cd /sys/class/gpio
root@OpenWrt:/sys/class/gpio# echo 24 > export
root@OpenWrt:/sys/class/gpio# cd gpio24
root@OpenWrt:/sys/devices/virtual/gpio/gpio24# while [ true ]; do cat value; don


It reports 1 when there is an input power supply (other than USB, which is another little problem we need to solve), and 0 when there is no input power supply, but is running on the super-capacitor.  At least that is the theory.

However, with the 20 Ohm resister, the voltage on the low-side is only about 2.2V, much too low to be useful.  Thus I tried putting a 10 Ohm resister in parallel, dropping the combined resistance to about 6.7 Ohms.  That still wasn't enough.

Then I tried bridging the resister by using a multi-meter in current measuring mode after having already charged the capacitor.  This gives effectively 0 Ohms, and provides an upper bound of what is possible with this super-capacitor.  With that configuration, things were more promising, as I saw u-boot try to boot, but as can be seen, as soon as it tried using the DDR RAM, it would reset, as presumably the DDR RAM draws too much power for the super-cap to sustain the supply above 3.3V:


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1

*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1

*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


As can be seen above, there is no line with 0, before it hits the u-boot reboot loop, indicating that the board does not operate on the super-capacitor,  even for just a few milliseconds.

We'll have to think of an appropriate solution to this problem, but first we need to understand the situation more completely.  

Is the capacitor unable to provide enough current for the DDR RAM?

Is the capacitor too small?

Also, only one of three super-capacitors seemed to work.  The other two might have been damaged at some point in the process, but having 2/3 faulty makes me think that we shouldn't be relying on them.

Monday, January 16, 2017

Mesh Extender 2.0 second revision prototype boards

The second revision of the Mesh Extender 2.0 (ME2.0) PCBs have arrived.  These were manufactured at almost miraculous speed by RF Design ahead of the Christmas break.  They contain fixes for three of the main problems we encountered with the first revision, fixing problems with both the ethernet and serial level-converter interfaces, as well as the bootstrap pull-up and pull-down on the Domino Core Atheros 9k module.  Here is the new PCBs:

Ethernet - both ethernet ports now work nicely.  One is capable of giga-bit, but sometimes it only connects at 100mbit/sec.  We need to understand what is going on there.

Serial port - the serial port now works without problem, since we have switched to a well-known MAXIM part, instead of the TI level converter, that although in principle capable of level-converting serial lines, turns out to be much too glitchy in our situation.  Other people have had similar problems.

The pull-up registers are now correctly set, although we did still have a problem with one of them apparently not being pulled up hard enough, so we have had to install a single resister to ensure that the Ath9k knows to boot from the 16MB flash, instead of from its internal ROM.  This resister can be seen in the photo below:

We now also have both D-SUB connectors on the same side of the PCB, but having double checked with the 3D-printed case prototype, they are both on the wrong side now, so we will have to fix that in the next revision:

We are currently working on building firmware images for this board, so that we can test other functions, which I'll report on in a separate post.

OpenWRT doesn't see packages from custom feed

Just a quick post to document a problem we encountered with adding a feed to an OpenWRT build process for the Serval Mesh Extender, and how we fixed it.

We followed the process in https://wiki.openwrt.org/doc/devel/feeds to add a new feed, by:

1. Modifying feeds.conf (copying feeds.conf.default, if required)
2. Running ./scripts/feeds update customfeed, with customfeed replaced with the name of the new feed.
3. Running ./scripts/feeds install -p customfeed,  again with customfeed replaced by the name of the new feed.

However, when we did this, and then ran make menuconfig, our new packages were nowhere to be seen.

We solved this by running the following additional command:

./scripts/feeds install -a customfeed

Wednesday, December 7, 2016

More details about the Mesh Extender prototype PCBs

I have been asked if I can share the current design documents for the Mesh Extender prototypes, which of course I am happy to do.  I have also confirmed with RF Design who are our PCB design partners (they also make the nice RFD900 / RFD868 radios that we use), that we can share the current schematics as well.

Please remember that all of this information is provisional, and subject to change.

Any feedback and suggested improvements welcome.

The images below have all been uploaded at 300dpi, so click on the image to see in higher resolution.

Jumpers and headers

The first the silk-screen, showing the various connectors:

Note that the only LEDs on the unit at the moment are on the ethernet connectors, so take care in working out whether the unit has power or not. We may put an inline LED in the power cable to make it easier to see if it is powered up or not.  Adding LEDs to the injection-moulded housing would increase costs a lot, and make it harder to achieve the IP65/66 rating we are aiming for.

Going through the headers and jumers from left to right, 

1. Internet of Things / Mesh of Things Interfaces

O1 is opto-coupled input 1, O2 is opto-coupled input 2, and R is the relay-driver output.
I.e., there are two inputs and one output that have some sort of electrically protected interface for IoT/Mesh of Things type applications.

We have allowed some extra flexibility,  not just for during the development phase, on where they will actually connect, i.e., to the GPIOs on the radio module (R position), or to GPIOs on the Ath9k in the Domino Core module (D position).  For R, O1 and O2 the centre pin goes to the relay or optocoupler, and the jumper connects this to either (or possibly both if you had some strange need to do so) the radio module or Ath9k GPIO.  The final decision on which position we default to on shipped units will occur as we continue the development process.

2. Options for powering the device

VIN is for selecting the input power type as automotive (i.e. DC supply or lead acid battery connection) or solar panel input. 

Note to self #1: I'll have to check with RFDesign, if it possible for us to make this selection available on one of the external connectors, so that you don't have to open the unit to make this change.  We could, for example, lose one of CTS or RTS on the UART, in order to make this selection possible externally

For battery charging, you need to select the battery chemistry internally:
LiPO Sets the output charge voltage for charging a 2s LiPO4 battery (as there is no support on the charger IC for load balancing of the cells this type is not recommended)
LiFePO Sets the output charge voltage for charging a 2s LiFePO4 battery
Lead Sets the output charge voltage for the charging a 12V lead acid battery
We will likely make this default to LiFePO, or possibly Lead, but of course it can be easily changed by opening the unit and moving a jumper.

3. Connectivity and debugging

USB is the breakout header for the micro USB connector, in case people want to connect some other device to it.

JTAG is the programmer port for the Domino module.

By default, you need to have a cable on the DB25 connector, in order to tie the Radio and Ath9k serial ports together.  This is on purpose, as it allows for use of an external radio connected by a special cable, that can just re-route the UART from the Ath9k to the external radio.  This is how we intend to allow easy connection to HF radios, for example.  If for some reason you want to operate the device without a cable on the DB25, but still want to bridge the UART to the radio interface, there are some pads just above the DB25 connector that can be soldered to short-circuit the UART connection, so that it doesn't require a cable to complete the circuit.


The main thing to note here are the pinouts of the DB25 and DB15 connectors.

These are subject to change! 

All six GPIOs of the RFD900 are exposed on the DB25 connector. These will be used by the radio firmware to check which regulatory domain it is in, and thus, what the allowed frequency and power level is.  It will also share this information (in the form of the 6-bit value) with the Ath9k, where it will be used to tell the firmware updater whether it is allowed to accept 3rd-party firmware or not.

As these values are supplied by the cable, it will be possible for research and development purposes to make your own cable that does not tell the firmware updater to block 3rd-party firmware, even if you received the unit with a cable for a regulatory domain where this is normally enforced.

Anyway, the current pinouts:

1. DB25 - Power and Radio connector

Going from left to right on the connector:

1 - GND
14 - GND
2 - RFD_IO2 (A GPIO from the RFD900/868 radio)
15 - RFD_IO6 (A GPIO from the RFD900/868 radio)
3 - RFD_IO3 (A GPIO from the RFD900/868 radio)
16 - RFD_IO1 (A GPIO from the RFD900/868 radio)
4 - RFD_IO4 (A GPIO from the RFD900/868 radio)
17 - RFD_IO5 (A GPIO from the RFD900/868 radio)
5 - Domino_TX_H (UART TX line on Ath9k module. Should ordinarily be connected to pin 18)
18 - RFD_TX (UART TX line on RFD900 module. Should ordinarily be connected to pin 5) 
6 - RFD_RX (UART RX line for the RFD900 module. Should ordinarily be connected to pin 19)
19 - Domino_RX_H (UART RX line on Ath9k module. Should ordinarily be connected to pin 6)
Note to self #2: Check if UART lines are around the right way here, or if we need to move them, to ensure that UART bridging can be achieved with just solder blobs between adjacent pins.
20 - RFD_CTS
8 - Ath9k CTS
21 - Ath9k RTS
Note to self #3: Do we really need CTS/RTS lines, and are they also paired the right way around?
9 - Domino Ath9k GPIO0
22 - Domino Ath9k GPIO1
10 - Battery negative (-) pole
23 - Battery centre pole for two-cell configurations (Lithium batteries only?)
11  -Battery positive (+) pole
24 - Solar/Automotive power GND
12 - Solar/Automotive power positive (+) pole
25 - GND
13 - +5V

The UART connections to the Ath9k go through a TXB0104 level converter, making them 5V tolerant.

The +5V line (on both the DB25 and DB15 connectors) can be either used to supply a modest amount of power to some external device, or to supply power to the board.  Thus a minimal DB25 power connector would simply connect the +5V and GND, e.g., from a USB cable, and short the appropriate pairs of pins to connect the UART of the radio module to the Ath9k, and optionally short pins to GND or +5V as required to encode the appropriate radio regulatory domain.

2. DB15 Utility / IoT / Mesh of Things connector

Again, from left to right.

Note that on the prototype PCBs this connector is on the opposite side of the PCB to the DB25 connector.  This is not how it will be in the production units.  I expect that we will be able to re-route the connector without changing the pinout, but nothing is certain yet.

15 - +3.3V
7 - Opto-isolated input 1
8 - Domino GPIO14 (no protection)
14 - Domino GPIO13 (no protection)
6 - Opto-isolated input 2
10 - Domino GPIO26 (no protection)
13 - GND
5 - Domino GPIO27 (no protection)
2 - GND
12 - Relay P3 (The centre-pole of the integrated 5V relay)
4 - Domino GPIO17 (no protection)
9 - Domino GPIO15 (no protection)
11 - Relay P5 (The normally disconnected pole of the integrated 5V relay)
3 - Domino GPIO16 (no protection)
1 - +5V

It should be stressed that the GPIO lines on this connector have no electrical protection whatever, and connect directly to the Ath9k on the Core Domino module.  We have exposed them to provide the maximum flexibility possible with this device.

Regarding the relay: 

The relay itself is an IMC03, which is rated up to 240V and loads of up to ~60VA, and could thus potentially directly switch low-power loads, however, we would recommend using it only as a staging relay, to something larger externally, e.g., a low-cost automotive relay if you were planning on switching low-voltage equipment.

When the relay is activated (via the XLNA pin from the Ath9k, or the IO2 line from the RFD radio module), then pins 11 and 12 will be connected, or disconnected otherwise.  This facility can be used to drive higher-voltage relays externally, e.g., to actuate controllers of various kinds.  

I'll post more information about the Mesh Extender design itself as time allows.

In the meantime, it would be great if someone were interested in starting creating a page for the Serval Mesh Extender 1.0 to the OpenWRT hardware table, so that we can collect and maintain the appropriate information there, and look at creating a relevant target in the OpenWrt DD build system, as I'd greatly prefer if we can include the Mesh Extender in OpenWRT's supported hardware.

Tuesday, December 6, 2016

The new Mesh Extender Prototypes have arrived!

After DHL claiming that they couldn't deliver the parcel to our address here in Germany for "technical reasons," we finally have the two prototypes of the new Mesh Extender design.

As a reminder, this design is required for our AusAID grant to pilot Serval in the Pacific.  The new board is much easier to manufacture, and is designed to fit in a custom-designed housing designed to meet the IP65 or IP66 standard.  This is all being done on a shoe-string budget, in part because hardware always costs more to design than you expect.

Here is the side that accepts an RFD900 or RFD868 packet UHF radio, if it is being optionally included, which will be the normal situation.  The round thing is a 1F super-cap, so that the unit can shut itself down gracefully if the battery/solar power supply fails.  

You can tell it's a prototype by the yellow tape over one of the ethernet connector's pins, where we hadn't thought too hard about where the antenna cables would end up.  Also, the D-Sub 15 is on the opposite side of the PCB to the D-Sub 25 connector, when they should probably be on the same side. Hopefully these will be the worst issues we have to deal with.

You can also see the internal USB connector for holding a memory stick, or something else if you want.  We actually hope to not use the USB port, because of the myriad problems we have had using USB memory sticks on battery-powered Mesh Extenders.  Basically the power consumption is too high, the performance is horrible, and the memory sticks get easily corrupted, or even die, if they lose power at the wrong time.

Below you can see both sides of the PCB.  Here the two internal ethernet jacks can be seen, which face downwards, so that cables can be routed inside the water-proof case.  We won't by default be providing external routing for those cables, as they are not normally needed in a Mesh Extender.  We figure that people will just drill holes in the cases in convenient locations if they need ethernet.  Not ideal, but given our extremely limited budget, it was the best we could manage.

The D-Sub 25 connector has 5V in, as well as 9-26ishV input, which can come from a car or truck plug, or from a solar panel.  That is, you can connect one of these to a suitable naked solar panel, without needing any interface circuitry.  Similarly, there are three pins for connecting a 2-cell LiFePO4, LiPo or sealed lead-acid battery.  The multi-chemistry battery charger is included in the unit.  

Dealing with EU Directive 2015/53/EU and potential unhelpful FCC rules

The D-Sub 25 also has device identification pins that are intended to allow the unit to work out which countries regulations apply, to control allowed transmission frequencies, and if required by FCC rules or EU directives to prevent the installation of third party firmware, that this can be done without any DRM locks. That is, by building a "research and development cable," the device will always be able to accept 3rd-party firmware, but when used with the default cable for a country that requires it, will only accept firmware updates that we have signed.  

We have talked about this solution to mis-regulation in a few different places, and understand that it isn't perfect, however, given our pragmatic requirements, it seems the least-bad option available to us.  We are always interested to hear people's thoughts on how we can improve on this, and of course people are always free to make their own firmware that follows their own policy, instead of this one.

This issue might become more important in the near future, if low-cost wireless routers start coming firmware locked due to these counter-productive regulations.

Getting down to business

So now I need to start qualifying these boards, so that we can confirm to our design and manufacture friends when it is safe to finalise the design, and manufacture the units we need for our pilot in the pacific.

There are a number of steps that we need to cover here, and that I would welcome assistance with, including:

1. Bringing the units up for the first time

2. Making the microSD slots work

The boards have two microSDs, because we don't know which method will work with the Ath9k chipset. One is connected to an SPI bus, which would be the preferred option, but may require some careful control of that bus, as it may have other devices connected. The other is connected via GPIOs to the Core Domino module, which should be fairly easily supportable by the Linux GPIO SPI/SD card kernel driver, although performance is likely to be poor.  We are very keen to get at least one of these options working, so that we can avoid all the horrors of using USB that I have described previously.

3. Building Serval Mesh and Mesh Extender packages for the Core Domino, and testing.

4. Porting our firmware patches for the RFD900 radio to the European version, the RFD868.

5. Testing the solar and battery controller interfaces.

6. Testing the ethernet interfaces.

7. Building and testing power cables, including the region identification and regulatory satisfaction functions.

8. Adding support for the super-cap to cleanly shut the thing down when power is lost, and characterising the super-caps performance. Added bonus: Abort shut down when power is restored during the shutdown process.

9. General integration testing for the whole Mesh Extender software suite

10. Help us develop the remaining functions required for the AusAID Pacific Pilot.

Anyone who is willing and able to contribute to these efforts would be greatly appreciated.

There is the practical limitation that we have only two prototypes, and that they need to stay with me here in Darmstadt, Germany.

That said, if anyone is interested in buying their own prototypes to assist in the process, I would be happy to facilitate this, however we don't have funds to subsidise this process at the moment, and because they are being built as prototypes, we would expect that they will be relatively expensive.  However, if we get enough orders to get a batch made at once, we can look into that.

Related to that, at this stage it would seem reasonable to expect that the prototype boards will be able to fit into the final custom injection-moulded cases that we are getting designed, to allow use of the prototypes outdoors in the future -- although nothing is certain until it happens.

Tuesday, November 22, 2016

Exploring the legal situation of disaster communications networks

I'm currently based in Darmstadt, Germany for a few months with the nice folks in the Secure Mobile Communications laboratory (SEEMOO) at TU-Darmstadt.

One of the main things that we are looking at while I am here, is to better understand the current legal situation facing networks like Serval.

This is important, because the telecommunications laws in most countries were made before such networks were beginning to be widely considered.  As a result, the particular characteristics and requirements of these networks are typically not accommodated in the telecommunications regulations of most countries (or at least the ones we have looked at so far).

So we are hoping over the next month or so to compare the situation here in Germany with Australia, and perhaps a few other countries, to see what road-block might be there, and perhaps to propose how such legislation could be amended to better facilitate them.

As a result I have spent much of the last week reading through the EU Directive 2014/53/EU (in both English and German), as well as draft legislation to implement it in Germany and Austria.

There are some points of concern that we have, particularly that open-source development using wireless routers and SDRs would effectively be outlawed, if the legislation is not carefully worded.  This would be a very unfortunate outcome, not just for the open-source communities who volunteer their efforts and donate the results of their efforts to the common good, but to society as a whole, who would be denied the benefits of these activities (most wireless routers are based on a version one of the open-source projects, very often OpenWRT) because modifying the software on a router to patch a security fix would also be illegal, and router vendors could be required to lock down firmware on the routers to prevent it being patched, updated or replaced in the first place.

This all echoes some of the same problems that are coming up in the USA, with the FCC's proposed rules to require firmware-lockdown on routers.

As we have seen with the confusion there, including TP-LINK being fined for locking down router firmware by the FCC, the matter is not a simple one, and certainly it is one that there still seems to be an understanding gap that we need to help regulators bridge, so that they can be enabled to enact regulations that protect these important rights, while addressing other legitimate social and political concerns.

Wednesday, August 31, 2016

Bridging Rhizome over HF

I'm currently up at Pacific Endeavour in Brisbane, which is a multi-national defence and civilian humanitarian and disaster response interoperability exercise, attended by, among others, the military of 22 nations around the Pacific basin.

We have been talking with Codan for a while about integrating Serval with their HF radios to allow Rhizome-based services over HF distances.  The goal is to allow MeshMS and our (still being written) Mesh-based social media applications to be carried over HF-distances (kilometres to hundreds of kilometres) between more localised concentrations of Mesh Extenders equiped with Wi-Fi and UHF packet radios.

This is of direct interest for our AusAID/DFAT grant to pilot Serval in the Pacific, where the beneficiary nations typically have small populations on islands separated by tens to hundreds of kilometres.   If we can link distant islands, then we can provide national and regional level communications, subject to the limitations of HF radio bandwidth (typically in the bytes to tens of bytes per second, although hundreds of bytes per second is possible in principle).

As part of the Pacific Endeavour, exercise the folks from Codan, Barrett and the other HF radio vendors are here.  The folks from Codan and Barrett in particular have been super helpful and enthusiastic as I have been explaining to them what we are trying to do, and how, to me at least, it seems to create the potential for new markets for HF radios.

They have kindly provided me with access to radios and documentation that they have with them, so that I can attempt a code sprint to establish two-way packet communications between their radios. That is, my goal is not just to prove that Serval Rhizome can run over HF radio, but that we can do it between what I understand are the two largest vendors in the humanitarian and disaster HF radio space.  Here is my setup with the Barrett radio (the black one on the desk) and the Codan radio (on the floor next to me):

After some false starts with having the correct firmware on one of the radios, I was pleasantly surprised to find that I was able to establish basic packet communications after only about ten hours of work.  Here you can see a packet being received, and a peer being detected by LBARD (the Serval Mesh program for handling slow radio links):

As the messages above show, LBARD knows what brand the radios are.  

This is partly out of necessity, because the serial control interface is quite different on the two radios, but also because we know if we are linking radios produced by the same vendor, that we have the potential to use higher-speed communications options.  

But between vendors, we currently have to use only inter-operable standards, which when it comes to HF radio means the older version of the Automatic Link Establishment protocol.  This is a bit unfortunate, because the interoperable ALE 2G standard is really slow.  The native data rate of ALE 2G is only 375 bits per second to begin with.  There is some protocol overhead, which reduces that somewhat.  Then we discovered that some radios don't support the full ASCII-64 restricted character set specified by the ALE 2G standard, and as their documentation claims. In particular, most of the punctuation characters in the ASCII-64 character set actually cause the message to be truncated on some of the radios.  This means that we can only use hex encoding, which means that we are consuming 2x 6-bit characters to encode each byte, i.e., 12 bits per byte. This reduced bandwidth by a further third.  Then there are a number of other losses relating to slow turn-around from TX to RX and back again, the need to fragment messages into 43 byte pieces to fit into the short ALE 2G AMD message limitations, all of which together reduce the effective bandwidth to perhaps 10 bytes per second.  Fortunately LBARD was designed to handle such slow links.

(We also found that some versions of firmware on some of the radios wouldn't receive ALE AMD messages sent by the opposite vendor, although we think that particular problem has been corrected in the latest version of firmware, so the problem is manageable.)

Naturally, we would much prefer to have higher speed connections, including the ALE 3G standard, that allows for up to 1KB/sec under ideal conditions, and without the need to fragment packets, which further saves on the TX/RX/TX turnaround cycle.  Unfortunately not all HF radios support ALE 3G.  That said, we will add support for radios that have it, and thus need to know if the remote radio also supports it -- thus another reason to know the vendor and model of the remote radio.

Higher-speed HF modems are also available, but are rather expensive.  Thus we are contemplating designing a low-cost HF radio using a cheap Spartan 3 or 6 FPGA for DSP, and attempt to implement something compatible with one of the existing HF modem standards (since they have done the hard work of knowing which modulation schemes are actually sensible on HF), that can be part of a Serval Mesh Extender HF adapter, to allow similar practical performance to the way that we are using the RFD900 UHF packet radios.    The RFD900 is actually much faster natively, but we are using it in an ad-hoc manner to allow many radios to be in range of each other, thus limiting each transmitter to ~0.25 - 1KB/sec.  For HF links, we are using 1:1 direct links, so that the best channel and modulation can be used, and thus we can use a simple TDM or equivalent scheme, that allows us to utilise a channel fully.

After getting everything working on ALE 2G, here is our Codan / Rhizome gateway (running from my laptop), and using this Codan "manpack" radio:

And the Barrett / Rhizome gateway, which I have yet to setup for the demo, will use this radio, a Barrett 2050:

So now I just need to update the version of OSX on the 2nd-hand Mac I just bought down the street, so that I can run this updated version of LBARD on it, and show it all running during the demo session this afternoon.  

But in the end, we have reached a significant milestone in that we have a proof-of-concept of near-real-time Serval communications over hundreds of kilometres, by using HF radios.