Tuesday, March 28, 2017

Teaching the RFD900 about uboot, so that it doesn't interrupt the Mesh Extender boot process

The Serval Mesh Extenders logically consist of a CPU with a serial port, and an RFD900 radio.  Our RFD900 firmware provides a regular heart-beat.

This all means that the RFD900 radio, as soon as it powers on, begins sending heart-beat messages.

This ordinarily wouldn't be a problem, except that the CPU modules we are using have only one serial port, so the serial boot loader is listening on the serial port when the system first powers up, and whenever it resets.

This means that if uboot sees a heartbeat, it will interpret it as the user wishing to interrupt the boot process to interact with the boot loader.  Following Murphey's Law, this is assured to happen on every boot.

We can't get rid of uboot, as it is important to have a way to reflash the devices (especially while we are still developing the software for them).

The remaining solution is to teach the RFD900 to notice when uboot is active.

In theory, this would just take a simple little finite state machine to notice the uboot banner, and then make sure it doesn't send any characters for a while, to allow uboot to boot normally.

There are, however, two complications.

1. We have to stay silent for long enough that we don't interfere with OpenWRT's fail-safe boot mode, which just means staying silent for longer.

and

2. The RFD900 is set to 230400bps for talking to LBARD, while uboot uses the serial port at 115200.

It is this last problem that makes life more interesting, because we can't just watch for the characters of the uboot banner. Instead, we need to look for a character sequence at 230400bps that is what you would see if you send the uboot banner at 115200bps.

As uboot is working at 1/2 the baud rate that the RFD900 is using, every serial bit received from uboot will be interpreted as two bits. Thus each uboot serial character should be able to be reliably received as two characters by the RFD900. We're lucky it isn't the other way around, as then the bits would be only half as wide, and there would be some uncertainty about how the bits would be read.

In the process, I also had to clean up a few other spots where our RFD900 firmware was causing other characters to be emitted immediately on power-up.

So after a few frustrating hours of debugging the finer points, I can now reliably boot a uboot-based system with an RFD900 running our packet radio firmware permanently attached to the serial port.




Wednesday, March 15, 2017

The custom IP67 power cables have arrived!

So, just over ten days ago I wrote that a Chinese manufacturer said that they could manufacture and ship the cables we want within ten days, and that I would order some.  Today they arrived!  Some fun unpacking photos:











First up, I must congratulate the Chinese manufacturer for being able to manufacture and ship our order so quickly.  They really were a pleasure to work with, including when we had a minor problem with the shipping (which wasn't their fault).

Also, as the photos hopefully show, the cables are very solidly built, with much thicker wire than the little ones we were contemplating using previously.  They will be able to easily support large panels and batteries, with charge currents being limited only by the battery charge controller inside the Mesh Extender. The cables simply give the strong impression that they will do the job well, which we will test over coming months.

The only complication we encountered, was that they didn't have stock of the white cable material, so we had to opt for the black PVC cable, which will slightly increase heat loading of the Mesh Extenders.  But given the time frames that we are working to at the moment, we needed to simply be sure that we would have the cables we need, so that we can get the cable head moulding tools made and cables assembled in time.  Critical paths are everything when building hardware!

Sunday, March 5, 2017

Sourcing cables directly from a Chinese manufacturer

I'm still working on the power/radio-regulatory cable for the Mesh Extenders.  Fortunately, this process is drawing towards its natural ending.

My current challenge is to find suitable UV-stable and IP65 or better rated cables.

We have already found some on banggood.com, but they leads are only ~20cm long. I'd much prefer leads approximately 0.5m - 1m long, so that there is ample length for attaching solar panels and battery packs, and general ease of connection.

A bit of hunting on http://alibaba.com yielded an enquiry from Bett Electronic Co. LTD in Shenzhen.

They have exactly the kind of thing we are looking for, and nicely, their cables are thicker (18-AWG instead of 22-AWG and 24-AWG) than the ones we found on BangGood, which means lower resistance.  A further bonus, is that they can manufacture in white, which will reduce heat loading a touch. Their cables are also IP67 certified.  Also very nice.  Here are the pictures that they sent of a sample:




Also, as the manufacturer, their prices are quite good.  Given that we are looking for cables that are 87cm long on average (compared with 20cm), and with the heavier-duty wiring, it is pleasing that the price is only about 50% higher.  They can also manufacture and ship quite quickly, apparently being able to get 100 units to us within about 10 days, including shipping by DHL.

So the only complication is that they don't have any certification of UV performance that I could use to easily determine whether they would be suitable.  However, Leah was very helpful and patient in answering my many, many questions.

What I now know is that the white cables are EPDM with added Titanium Oxide, which should have excellent UV and weathering properties.  The connectors themselves are PVC, without any special UV stabilising material.  However, according to this document, the main problem with PVC under UV, is that it will discolour (not a problem), and will have reduced impact resistance (only a minor problem).

Otherwise, the only minor issue, is that I can't figure out if the PVC is Lead-free or not. However, given the small mass of the connectors, this is not an immediate concern.  Nonetheless, I would love to be able to be sure that there was no Lead in the PVC.

So it looks like we have a good solution, and I think I will place the order with Bett Electronic in the next day or so.

Wednesday, March 1, 2017

Reading and writing the I2C EEPROM connected to an RFD900

As previously described, for the new Mesh Extender we have an I2C EEPROM for encoding regulatory information, e.g., radio permitted frequency and TX power.

Basically, I am building the tools that we will need to program Mesh Extender cables as we build them, and also in the field, to reprogram them as required.

Over the past couple of weeks I have worked on the code for the RFD900+ firmware to be able to read and write the I2C EEPROM, and then to modify our flash900 utility to be able to flash not just an RFD900 radio, but also an I2C EEPROM connected via an RFD900 radio.

I've had it mostly working for a while now, but there have been strange problems with accessing certain parts of the EEPROM, and weird intermittent write problems affecting the whole EEPROM. I tried swapping EEPROMs, to confirm that it wasn't a faulty part.

I spoke to the engineering workshop folks here, who suggested that I might need external pull-up resisters to terminate the I2C bus, which fortunately turned out not to be the case, as this simplifies the cable and PCB design process (one of them would have required some extra resisters).

They also had a nice little logic analyser that could understand I2C.  That proved to be super-helpful.  It took me less than an hour to see that the problem was when back-to-back page reads were occurring and the last bit of the last byte read was a zero.  Basically I was failing to implement the NACK required at the end of a multi-byte read sequence.  

I fixed this by inserting a dummy read with NACK, like this:

$ git show 490025c33a3e821aab05055d627d192bb0d186fc               
commit 490025c33a3e821aab05055d627d192bb0d186fc                   
Author: gardners <paul@servalproject.org>                         
Date:   Thu Mar 2 14:02:22 2017 +1030                             
                                                                  
    properly terminate sequential read operations                 
                                                                  
diff --git a/Firmware/radio/i2c.c b/Firmware/radio/i2c.c          
index 76fe0be..6e9d68f 100644                                     
--- a/Firmware/radio/i2c.c                                        
+++ b/Firmware/radio/i2c.c                                        
@@ -183,6 +183,9 @@ char _eeprom_read_page(unsigned short address)
     }                                                            
   }                                                              
                                                                  
+  // Terminate the sequential read                               
+  i2c_rx(0);                                                     
+                                                                 
   i2c_stop();                                                    
                                                                  
   return 0;                                                      

With that in place, I could suddenly reliably access the whole EEPROM.  In fact, the correct data was always written there, it was just that I couldn't read it.

Following that, I spent a bit of time optimising the EEPROM read and write speed, and also the speed of the flash-rfd900 utility, both for flashing the radio and flashing the EEPROM.

It is now possible to flash an RFD900 radio in about 23 seconds, even when using a USB serial adapter with 16ms latency.  This is a huge improvement on the ~100 - 150 seconds it took previously.

Flashing the I2C EEPROM takes just under 20 seconds, including reading the entire EEPROM once before hand, and once after, to verify that all bytes have been correctly written.

Now I just need to tweak the format of the data that I store in the EEPROM, so that it contains all the information that I need, principally the list of countries a particular cable is suitable for, and increasing the resolution of the frequency parameter from MHz to KHz.

Monday, February 27, 2017

Attaching Serial Numbers

Today I have been researching options for permanently marking Mesh Extenders with serial numbers on the housing, cables and PCBs.  This is complicated by the tropical-maritime operating environment, and that the housings are to be made of poly-carbonate.

Poly-carbonate can't be safely laser-cut or laser-etched, because it gives off toxic and corrosive chlorine gas.

Tropical-maritime environments eat glue for breakfast, and abound in UV radiation to degrade everything.

So far, the best option I have found is to use a laser-etchable UV laminates, e.g.:

https://www.trotec-materials.com/laser-materials/trolase-metallic-plus/lmt-314-203-br-stainl-steel-black-0-8mm.html

With an appropriate glue.

They do supply them with either 3M 467MP adhesive tape, which claims to be good for polycarbonate, and tolerates high heat etc, or a similar TESA adhesive.

Otherwise, you can get the laminates with no adhesive, and then apply your own, such as Dymax 3025 or 3099, which seem like they should be stronger than the 3M or TESA options.

The laminates themselves are quite cheap, less than AU$20 for 30x60cm. Assuming 4x5cm labels, this gives a material cost of only about AU$0.25 per label (plus glue) -- much less than the cost of the housing.  We will likely need one 4x5cm label and two 2x5cm labels, so that we can have safety labels on the insides, incase the one on the outside falls off some how.

For the PCB, we can just have a silk-screen white panel to write the serial number and manufacture date on using a permanent marker, so no great problems there.

For the D-SUB 25-pin cables, we are looking at a nice solution to get those professionally manufactured with a low-pressure over-mould process, for about $14 per cable, plus parts.  I need to have a chat to those folks to see what our options are there for imprinting them with serial numbers during production, or failing that, whether we can laser-etch or otherwise mark those.


Tuesday, February 21, 2017

SPI microSD card on Atheros 9k

For the new Mesh Extenders, we want to use a microSD card for bulk storage, instead of a USB memory stick.  This is in part because USB draws a lot of power just to run the bus (about 10% - 20% of the total power consumption of a Mesh Extender), but also because we have had a lot of reliability problems with USB memory sticks in Mesh Extender prototypes. Basically, they don't like being shut down suddenly.

Our preferred solution was to add a microSD slot to the Ath9k modules we are using (the Core Domino).  However, we could not find any good sources on how to do this. First, we tried using the bit-bashing driver, which we did get working, however it was horribly slow (about 10KB/sec), and would routinely freeze or reset the Ath9k (possibly due to watchdog timeouts?).

The Atheros 9k does have an SPI interface which is compatible with microSD cards, however, after searching for people having used it, I couldn't find anything sensible.  The main problem is that the 16MB FLASH that boots the Mesh Extender is also attached to the SPI bus, and so there is a need to arbitrate between that, and our microSD card.

After I had given up hope and was trying to make the bit-bash driver work, I accidentally found a page that described exactly what I wanted to do, and had some simple patches for OpenWRT CC to implement it.  You just had to assign a GPIO to be the chip-select for the microSD slot, and all would be well.  Unfortunately I had no spare GPIOs.

What I did have, however, was the chip-select line for the 16MB FLASH.  Since I had only two devices, I suspected that I could just invert that signal, and feed it as the chip-select to the microSD slot.  As hoped, this worked just fine.

The total changes were very small:

https://github.com/servalproject/openwrt/commit/1cfb77ccaf4fd6a5ef345ba599534a5ae26114df
https://github.com/servalproject/openwrt/commit/3133e6ed935bc4d7705f72e639519e39cafa711d
https://github.com/servalproject/openwrt/commit/034daf1f4c63c90b079e40da3f8529288ddc1a79

Note that while I didn't have a spare GPIO, this approach still requires that you assign some GPIO to be the chip-select line. I chose a GPIO that was not exposed on the board I was using.  For many low-cost wireless routers, like the TP-LINK WR703N and friends, there are plenty of such blind GPIOs to choose from.  Indeed, this microSD hack should be fairly easily adaptable to the 703N, MR3020, MR3040 and many other cheap Atheros 9331 (or similar Atheros parts) wireless routers, such as the very nice GL-AR150 and related family.

Update: I have spoken to the nice people who make the GL-AR150 and related devices, and they are taking a look at this for evaluation.

Mesh Extender EEPROM-equiped cable work

One of the novel features of the new Mesh Extender, is that the national radio regulatory parameters will not be stored in the main unit, but rather in the power cable, as I have described previously.

This means that we can, the theory at least, have bunches of Mesh Extenders pre-positioned regionally (or even globally), without having to configure them for each country in which they will be used. Instead, we just need to have batches of country-specific power cables for them, or similarly, program sets of power cables for them, and then send those to the relevant country/countries.

It also means that we can deal with unfortunate regulations that require firmware be locked in some countries, by having the firmware lock encoded on the power cable as well.

To implement all this, we are using 2KB I2C EEPROMs, so that we can have quite a bit of information stored, and also re-program the cables from one country to another when the need arises.

We want the RFD900/RFD868 radio to read this information directly when they power up, so that they can set their radio frequency and TX power levels appropriately.  This means it makes sense to have the RFD radio talk directly to the I2C EEPROM, rather than via the Ath9k CPU.  A second reason for doing it this way is that we don't have enough GPIOs left on the Ath9k module we are using.

All this means that I needed to implement an I2C library in the RFD900 firmware, and as mentioned in the previous post, implement the SHA-3 hash function, so that we can validate that the parameters are intact, and have not been corrupted by problems with the I2C EEPROM.

I found some nice reference information about I2C, and apart from some challenges at getting the timing and protocol right, it only took a few days work to get to the point where I can now reliably and quickly read and write the EEPROM.  This meant spending a lot of time staring at oscilloscope displays like this one.




For development, I wired up a little board with one of the EEPROMs to an RFD900, to simulate the wiring for the cable.



One nice little trick that we do to save power consumption, is that the EEPROM is powered directly from one of the GPIOs on the RFD900.  If we turn that GPIO off, then the EEPROM is physically switched off, saving a (very) little bit of power.

For the actual cables, we have the pinout planned such that the EEPROM can be soldered directly onto the relevant pins of the D-SUB 25 connector, to simplify assembly and avoid the cost of a little PCB, something like this:



Bending the pins the right amount was a bit more work than hoped, but with practice (or a little jig) could be done quite quickly. This is mainly because the chip is quite a bit fatter than the pitch between the rows of pins.  The pins are the same pitch as the pins within a row on the connector, however, so that aspect is fine.

This theme of simplifying assembly is, I am finding, a recurring one as I plan how to make something that can actually be manufactured, even in modest quantities.  All manual steps need to minimised so far as possible.  This means actually trying to do things, rather than just designing them.  So half way through writing this post, I wandered downstairs and tried to solder one of these cables together. I succeeded eventually, more or less, but in the process realised that to just fit the chip onto the back of the D-SUB connector, two pins needed to be swapped around.  For now, I have soldered it up with the temporary fix, until we can get this pin change made on the next revision of PCBs (which fortunately haven't been fabricated yet). You can see the resulting mess here:




Here is the whole thing with the nice $1.35 a piece IP65-rated connectors I found recently:


Apart from the pinout problem, it wasn't too hard to assemble, really, so I am encouraged that this could still work.  I won't stop thinking about how I can further optimise the process, however.

What I haven't covered above is how I am going to protect the cable head, so that it is sealed etc.  For that, I have been talking to the plastics manufacturing people who are helping us with the injection-moulded housing design.  They have suggested we could fairly cheaply produce a tool for low-pressure plastic encapsulation of the cable, similar to what is used to produce serial and printer cable ends.  Because it is low pressure, it is possible to use aluminium for the tool, rather than the hard tool steel that is required for the injection-moulded housing.  Because aluminium is so much softer, it is quicker (and thus cheaper) to machine.