Merge pull request #753 from cyberman54/development

v2.4.0

README.md

@@ -95,7 +95,7 @@ Compile time configuration is spread across several files. Before compiling the
 ## platformio.ini
 Edit `platformio_orig.ini` and select desired hardware target in section boards. To add a new board, create an appropriate hardware abstraction layer file in hal subdirectory, and add a pointer to this file in sections board. Copy or rename to `platformio.ini` in the root directory of the project. Now start Platformio. Note: Platformio is looking for `platformio.ini` in the root directory and won't start if it does not find this file.
 
-## src/paxcounter.conf
+## paxcounter.conf
 Edit `src/paxcounter_orig.conf` and tailor settings in this file according to your needs and use case. Please take care of the duty cycle regulations of the LoRaWAN network you're going to use. Copy or rename to `src/paxcounter.conf`.
 
 If your device has a **real time clock** it can be updated bei either LoRaWAN network or GPS time, according to settings *TIME_SYNC_INTERVAL* and *TIME_SYNC_LORAWAN* in `paxcounter.conf`.
@@ -110,7 +110,7 @@ To configure OTAA, leave `#define LORA_ABP` deactivated (commented). To use ABP,
 The file `src/loraconf_sample.h` contains more information about the values to provide.
 
 ## src/ota.conf
-Create file `src/ota.conf` using the template [src/ota.sample.conf](https://github.com/cyberman54/ESP32-Paxcounter/blob/master/src/ota.sample.conf) and enter your WIFI network&key. These settings are used for downloading updates. If you want to push own OTA updates you need a <A HREF="https://bintray.com/JFrog">Bintray account</A>. Enter your Bintray user account data in ota.conf. If you don't need wireless firmware updates just rename ota.sample.conf to ota.conf.
+Create file `src/ota.conf` using the template [src/ota.sample.conf](https://github.com/cyberman54/ESP32-Paxcounter/blob/master/src/ota.sample.conf) and enter your WIFI network&key. These settings are used for downloading updates via WiFi, either from a remote https server, or locally via WebUI. If you want to use a remote server, you need a <A HREF="https://bintray.com/JFrog">Bintray account</A>. Enter your Bintray user account data in ota.conf. If you don't need wireless firmware updates just rename ota.sample.conf to ota.conf.
 
 # Building
 
@@ -124,8 +124,11 @@ The LoPy/LoPy4/FiPy board needs to be set manually. See these
 <A HREF="https://www.thethingsnetwork.org/labs/story/program-your-lopy-from-the-arduino-ide-using-lmic">instructions</A> how to do it. Don't forget to press on board reset button after switching between run and bootloader mode.<p>
 The original Pycom firmware is not needed, so there is no need to update it before flashing Paxcounter. Just flash the compiled paxcounter binary (.elf file) on your LoPy/LoPy4/FiPy. If you later want to go back to the Pycom firmware, download the firmware from Pycom and flash it over.
 	
-- **During runtime, using FOTA via WIFI:**
+- **During runtime, automated using OTA via WIFI:**
-After the ESP32 board is initially flashed and has joined a LoRaWAN network, the firmware can update itself by FOTA. This process is kicked off by sending a remote control command (see below) via LoRaWAN to the board. The board then tries to connect via WIFI to a cloud service (JFrog Bintray), checks for update, and if available downloads the binary and reboots with it. If something goes wrong during this process, the board reboots back to the current version. Prerequisites for FOTA are: 1. You own a Bintray repository, 2. you pushed the update binary to the Bintray repository, 3. internet access via encrypted (WPA2) WIFI is present at the board's site, 4. WIFI credentials were set in ota.conf and initially flashed to the board. Step 2 runs automated, just enter the credentials in ota.conf and set `upload_protocol = custom` in platformio.ini. Then press build and lean back watching platformio doing build and upload.
+After the ESP32 board is initially flashed and has joined a LoRaWAN network, the firmware can update itself by OTA. This process is kicked off by sending a remote control command (see below) via LoRaWAN to the board. The board then tries to connect via WiFi to a cloud service (JFrog Bintray), checks for update, and if available downloads the binary and reboots with it. If something goes wrong during this process, the board reboots back to the current version. Prerequisites for OTA are: 1. You own a Bintray repository, 2. you pushed the update binary to the Bintray repository, 3. internet access via encrypted (WPA2) WiFi is present at the board's site, 4. WiFi credentials were set in ota.conf and initially flashed to the board. Step 2 runs automated, just enter the credentials in ota.conf and set `upload_protocol = custom` in platformio.ini. Then press build and lean back watching platformio doing build and upload.
+
+- **During runtime, manually using OTA via WIFI:**
+If option *BOOTMENU* is defined in `paxcounter.conf`, the ESP32 board will try to connect to a known WiFi access point each time cold starting (after a power cycle or a reset), using the WiFi credentials given in `ota.conf`. Once connected to the WiFi it will fire up a simple webserver, providing a bootstrap menu waiting for a user interaction (pressing "START" button in menu). This process will be aborted by ESP32 hardware watchdog after *BOOTDELAY* seconds, ensuring booting the device to runmode. Once a user interaction in bootstrap menu was detected, the watchdog time will be extended to *BOOTTIMEOUT* seconds. During this time a firmware upload can be performed manually by user, e.g. using a smartphone in tethering mode providing the firmware upload file.
 
 # Legal note
 
@@ -437,6 +440,7 @@ Send for example `8386` as Downlink on Port 2 to get battery status and time/dat
 	2 = reset device to factory settings
 	3 = flush send queues
 	4 = restart device (warmstart)
+	8 = reboot device to maintenance mode (local web server)
 	9 = reboot device to OTA update via Wifi mode
 
 0x0A set LoRaWAN payload send cycle

include/boot.h

@@ -0,0 +1,16 @@
+#ifndef BOOT_H
+#define BOOT_H
+
+#include "globals.h"
+#include "hash.h"
+
+#include <Update.h>
+#include <WiFi.h>
+#include <WiFiClientSecure.h>
+#include <WiFiClient.h>
+#include <WebServer.h>
+#include <ESPmDNS.h>
+
+void start_boot_menu(void);
+
+#endif // BOOT_H

include/globals.h

@@ -58,7 +58,8 @@ enum runmode_t {
   RUNMODE_NORMAL,
   RUNMODE_WAKEUP,
   RUNMODE_UPDATE,
-  RUNMODE_SLEEP
+  RUNMODE_SLEEP,
+  RUNMODE_MAINTENANCE
 };
 
 // Struct holding devices's runtime configuration
@@ -83,7 +84,6 @@ typedef struct __attribute__((packed)) {
   uint8_t macfilter;     // 0=disabled, 1=enabled
   uint8_t rgblum;        // RGB Led luminosity (0..100%)
   uint8_t monitormode;   // 0=disabled, 1=enabled
-  uint8_t runmode;       // 0=normal, 1=update
   uint8_t payloadmask;   // bitswitches for payload data
   uint8_t enscount;      // 0=disabled 1= enabled
 

include/hash.h

@@ -4,6 +4,6 @@
 #include <Arduino.h>
 #include <RokkitHash.h>
 
-uint32_t IRAM_ATTR hash(const char *data, int len);
+uint32_t IRAM_ATTR myhash(const char *data, int len);
 
 #endif

include/main.h

@@ -20,5 +20,6 @@
 #include "lorawan.h"
 #include "timekeeper.h"
 #include "corona.h"
+#include "boot.h"
 
 #endif

include/mqttclient.h

@@ -3,6 +3,7 @@
 
 #include "globals.h"
 #include "rcommand.h"
+#include "hash.h"
 #include <MQTT.h>
 #include <ETH.h>
 #include <mbedtls/base64.h>

platformio_orig.ini

@@ -49,7 +49,7 @@ description = Paxcounter is a device for metering passenger flows in realtime. I
 
 [common]
 ; for release_version use max. 10 chars total, use any decimal format like "a.b.c"
-release_version = 2.3.0
+release_version = 2.4.0
 ; DEBUG LEVEL: For production run set to 0, otherwise device will leak RAM while running!
 ; 0=None, 1=Error, 2=Warn, 3=Info, 4=Debug, 5=Verbose
 debug_level = 3
@@ -57,7 +57,7 @@ extra_scripts = pre:build.py
 otakeyfile = ota.conf
 lorakeyfile = loraconf.h
 lmicconfigfile = lmic_config.h
-platform_espressif32 = espressif32@3.0.0
+platform_espressif32 = espressif32@3.1.0
 monitor_speed = 115200
 upload_speed = 115200 ; set by build.py and taken from hal file
 display_library = ; set by build.py and taken from hal file
@@ -86,8 +86,7 @@ lib_deps_basic =
     jchristensen/Timezone @ ^1.2.4
     makuna/RTC @ ^2.3.5
     spacehuhn/SimpleButton 
-    ;lewisxhe/AXP202X_Library @ ^1.1.2
+    lewisxhe/AXP202X_Library @ ^1.1.3
-    https://github.com/lewisxhe/AXP202X_Library.git
     geeksville/esp32-micro-sdcard @ ^0.1.1
     256dpi/MQTT @ ^2.4.7
 lib_deps_all =

src/boot.cpp

@@ -0,0 +1,199 @@
+#include "boot.h"
+#include "reset.h"
+
+// Local logging tag
+static const char TAG[] = __FILE__;
+
+// start local web server with user interface for maintenance mode
+// used for manually uploading a firmware file via wifi
+
+void start_boot_menu(void) {
+
+  uint8_t mac[6];
+  char clientId[20];
+
+  // hash 6 byte MAC to 4 byte hash
+  esp_eth_get_mac(mac);
+  const uint32_t hashedmac = myhash((const char *)mac, 6);
+  snprintf(clientId, 20, "paxcounter_%08x", hashedmac);
+
+  const char *host = clientId;
+  const char *ssid = WIFI_SSID;
+  const char *password = WIFI_PASS;
+
+  RTC_runmode = RUNMODE_NORMAL;
+
+  hw_timer_t *timer = NULL;
+  timer = timerBegin(2, 80, true);                 // timer 2, div 80, countup
+  timerAttachInterrupt(timer, &esp_restart, true); // callback device reset
+  timerAlarmWrite(timer, BOOTDELAY * 1000000, false); // set time in us
+  timerAlarmEnable(timer);                            // enable interrupt
+
+  WebServer server(80);
+
+  const char *loginMenu =
+      "<form name='loginForm'>"
+      "<table width='20%' bgcolor='A09F9F' align='center'>"
+      "<tr>"
+      "<td colspan=2>"
+      "<center><font size=4><b>Maintenance Menu</b></font></center>"
+      "<br>"
+      "</td>"
+      "<br>"
+      "<br>"
+      "</tr>"
+      "<tr>"
+      "<td><input type='submit' onclick='start(this.form)' "
+      "value='Start'></td>"
+      "</tr>"
+      "</table>"
+      "</form>"
+      "<script>"
+      "function start(form) {window.open('/serverIndex')}"
+      "</script>";
+
+  const char *serverIndex =
+      "<script "
+      "src='https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/"
+      "jquery.min.js'></script>"
+      "<form method='POST' action='#' enctype='multipart/form-data' "
+      "id='upload_form'>"
+      "<input type='file' name='update'>"
+      "<input type='submit' value='Update'>"
+      "</form>"
+      "<div id='prg'>progress: 0%</div>"
+      "<script>"
+      "$('form').submit(function(e){"
+      "e.preventDefault();"
+      "var form = $('#upload_form')[0];"
+      "var data = new FormData(form);"
+      " $.ajax({"
+      "url: '/update',"
+      "type: 'POST',"
+      "data: data,"
+      "contentType: false,"
+      "processData:false,"
+      "xhr: function() {"
+      "var xhr = new window.XMLHttpRequest();"
+      "xhr.upload.addEventListener('progress', function(evt) {"
+      "if (evt.lengthComputable) {"
+      "var per = evt.loaded / evt.total;"
+      "$('#prg').html('progress: ' + Math.round(per*100) + '%');"
+      "}"
+      "}, false);"
+      "return xhr;"
+      "},"
+      "success:function(d, s) {"
+      "console.log('success!')"
+      "},"
+      "error: function (a, b, c) {"
+      "}"
+      "});"
+      "});"
+      "</script>";
+
+  WiFi.disconnect(true);
+  WiFi.config(INADDR_NONE, INADDR_NONE,
+              INADDR_NONE); // call is only a workaround for bug in WiFi class
+  // see https://github.com/espressif/arduino-esp32/issues/806
+  WiFi.setHostname(host);
+  WiFi.mode(WIFI_STA);
+
+  // Connect to WiFi network
+  // workaround applied here to avoid WIFI_AUTH failure
+  // see https://github.com/espressif/arduino-esp32/issues/2501
+  // 1st try
+  WiFi.begin(ssid, password);
+  while (WiFi.status() == WL_DISCONNECTED) {
+    delay(500);
+  }
+  // 2nd try
+  if (WiFi.status() != WL_CONNECTED) {
+    WiFi.begin(ssid, password);
+    while (WiFi.status() != WL_CONNECTED) {
+      delay(500);
+    }
+  }
+
+  MDNS.begin(host);
+
+  server.on("/", HTTP_GET, [&server, &loginMenu]() {
+    server.sendHeader("Connection", "close");
+    server.send(200, "text/html", loginMenu);
+  });
+
+  server.on("/serverIndex", HTTP_GET, [&server, &serverIndex, &timer]() {
+    timerAlarmWrite(timer, BOOTTIMEOUT * 1000000, false);
+    server.sendHeader("Connection", "close");
+    server.send(200, "text/html", serverIndex);
+  });
+
+  server.onNotFound([&server, &loginMenu]() {
+    server.sendHeader("Connection", "close");
+    server.send(200, "text/html", loginMenu);
+  });
+
+  // handling uploading firmware file
+  server.on(
+      "/update", HTTP_POST,
+      [&server]() {
+        server.sendHeader("Connection", "close");
+        server.send(200, "text/plain", (Update.hasError()) ? "FAIL" : "OK");
+        WiFi.disconnect(true);
+        if (!Update.hasError())
+          RTC_runmode = RUNMODE_POWERCYCLE;
+        esp_restart();
+      },
+
+      [&server, &timer]() {
+        bool success = false;
+        HTTPUpload &upload = server.upload();
+
+        switch (upload.status) {
+
+        case UPLOAD_FILE_START:
+          // start file transfer
+          ESP_LOGI(TAG, "Uploading %s", upload.filename.c_str());
+          success = Update.begin();
+          break;
+
+        case UPLOAD_FILE_WRITE:
+          // flashing firmware to ESP
+          success = (Update.write(upload.buf, upload.currentSize) ==
+                     upload.currentSize);
+          break;
+
+        case UPLOAD_FILE_END:
+          success = Update.end(true); // true to set the size to the current
+          if (success)
+            ESP_LOGI(TAG, "Upload finished, %u bytes written",
+                     upload.totalSize);
+          else
+            ESP_LOGE(TAG, "Upload failed, status=%d", upload.status);
+          break;
+
+        case UPLOAD_FILE_ABORTED:
+        default:
+          break;
+
+        } // switch
+
+        if (!success) {
+          ESP_LOGE(TAG, "Error: %s", Update.errorString());
+          WiFi.disconnect(true);
+          esp_restart();
+        }
+      });
+
+  server.begin();
+  MDNS.addService("http", "tcp", 80);
+  ESP_LOGI(TAG,
+           "WiFi connected to '%s', open http://%s.local or http://%s in your "
+           "browser",
+           WiFi.SSID().c_str(), clientId, WiFi.localIP().toString().c_str());
+
+  while (1) {
+    server.handleClient();
+    delay(1);
+  }
+}

src/cyclic.cpp

@@ -13,16 +13,14 @@ Ticker cyclicTimer;
 extern boolean isSDS011Active;
 #endif
 
-void setCyclicIRQ() {
+void setCyclicIRQ() { xTaskNotify(irqHandlerTask, CYCLIC_IRQ, eSetBits); }
-  xTaskNotify(irqHandlerTask, CYCLIC_IRQ, eSetBits);
-}
 
 // do all housekeeping
 void doHousekeeping() {
 
-  // check if update mode trigger switch was set by rcommand
+  // check if update or maintenance mode trigger switch was set by rcommand
-  if (RTC_runmode == RUNMODE_UPDATE)
+  if ((RTC_runmode == RUNMODE_UPDATE) || (RTC_runmode == RUNMODE_MAINTENANCE))
-    do_reset(true);
+    do_reset(true); // warmstart
 
   // heap and task storage debugging
   ESP_LOGD(TAG, "Heap: Free:%d, Min:%d, Size:%d, Alloc:%d, StackHWM:%d",

src/display.cpp

@@ -156,9 +156,10 @@ void dp_init(bool verbose) {
 
       // give user some time to read or take picture
       dp_dump(displaybuf);
+#if !(BOOTMENU)
       delay(8000);
-      dp_contrast(DISPLAYCONTRAST);
+#endif
-      dp_clear();
+     
 #endif // HAS_LORA
 
     } // verbose

src/hash.cpp

@@ -40,6 +40,6 @@
 #undef ROKKIT_ENABLE_8BIT_OPTIMIZATIONS
 #endif
 
-uint32_t IRAM_ATTR hash(const char *data, int len) {
+uint32_t IRAM_ATTR myhash(const char *data, int len) {
   return rokkit(data, len);
 }

src/macsniff.cpp

@@ -141,7 +141,7 @@ uint16_t mac_analyze(MacBuffer_t MacBuffer) {
   // hashed 4 byte MAC
   // to save RAM, we use only lower 2 bytes of hash, since collisions don't
   // matter in our use case
-  hashedmac = hash((const char *)&saltedmac, 4);
+  hashedmac = myhash((const char *)&saltedmac, 4);
 
   auto newmac = macs.insert(hashedmac); // add hashed MAC, if new unique
   bool added =

src/main.cpp

@@ -51,6 +51,7 @@ So don't do it if you do not own a digital oscilloscope.
 -------------------------------------------------------------------------------
 0	displayIRQ -> display refresh -> 40ms (DISPLAYREFRESH_MS)
 1 ppsIRQ -> pps clock irq -> 1sec
+2 watchdog -> used in boot.cpp
 3	MatrixDisplayIRQ -> matrix mux cycle -> 0,5ms (MATRIX_DISPLAY_SCAN_US)
 
 
@@ -290,6 +291,17 @@ void setup() {
     start_ota_update();
 #endif
 
+#if (BOOTMENU)
+  // start local webserver after device powers up or on rcommand request
+  if ((RTC_runmode == RUNMODE_POWERCYCLE) ||
+      (RTC_runmode == RUNMODE_MAINTENANCE))
+    start_boot_menu();
+#else
+  // start local webserver on rcommand request only
+  if (RTC_runmode == RUNMODE_MAINTENANCE)
+    start_boot_menu();
+#endif
+
   // start mac processing task
   ESP_LOGI(TAG, "Starting MAC processor...");
   macQueueInit();
@@ -470,6 +482,8 @@ void setup() {
 
 // display interrupt
 #ifdef HAS_DISPLAY
+  dp_clear();
+  dp_contrast(DISPLAYCONTRAST);
   // https://techtutorialsx.com/2017/10/07/esp32-arduino-timer-interrupts/
   // prescaler 80 -> divides 80 MHz CPU freq to 1 MHz, timer 0, count up
   displayIRQ = timerBegin(0, 80, true);

src/mqttclient.cpp

@@ -48,7 +48,7 @@ int mqtt_connect(const char *my_host, const uint16_t my_port) {
 
   // hash 6 byte MAC to 4 byte hash
   esp_eth_get_mac(mac);
-  const uint32_t hashedmac = hash((const char *)mac, 6);
+  const uint32_t hashedmac = myhash((const char *)mac, 6);
   snprintf(clientId, 20, "paxcounter_%08x", hashedmac);
 
   ESP_LOGI(TAG, "MQTT name is %s", MQTT_CLIENTNAME);

src/paxcounter_orig.conf

@@ -6,8 +6,11 @@
 //
 // Note: After editing, before "build", use "clean" button in PlatformIO!
 
-// Verbose enables additional serial debug output
+// Device options
-#define VERBOSE                         1       // set to 0 to silence the device, for mute use build option
+#define VERBOSE                         1       // set to 0 to silence the device, 1 enables additional debug output
+#define BOOTMENU                        0       // 0 = no bootmenu, 1 = device brings up boot menu before starting application
+#define BOOTDELAY                       30      // time [seconds] while devices waits in boot menue for input
+#define BOOTTIMEOUT                     300     // time [seconds] while devices waits to finish upload a firmware file
 
 // Payload send cycle and encoding
 #define SENDCYCLE                       30      // payload send cycle [seconds/2], 0 .. 255

src/rcommand.cpp

@@ -31,8 +31,12 @@ void set_reset(uint8_t val[]) {
     ESP_LOGI(TAG, "Remote command: restart device warm");
     do_reset(true);
     break;
-  case 9: // reset and ask for software update via Wifi OTA
+  case 8: // reset and start local web server for manual software update
-    ESP_LOGI(TAG, "Remote command: software update via Wifi");
+    ESP_LOGI(TAG, "Remote command: reboot to maintenance mode");
+    RTC_runmode = RUNMODE_MAINTENANCE;
+    break;
+  case 9: // reset and ask OTA server via Wifi for automated software update
+    ESP_LOGI(TAG, "Remote command: reboot to ota update mode");
 #if (USE_OTA)
     // check power status before scheduling ota update
     if (batt_sufficient())

src/reset.cpp

@@ -14,7 +14,8 @@ RTC_DATA_ATTR struct timeval RTC_sleep_start_time;
 RTC_DATA_ATTR unsigned long long RTC_millis = 0;
 timeval sleep_stop_time;
 
-const char *runmode[5] = {"powercycle", "normal", "wakeup", "update", "sleep"};
+const char *runmode[6] = {"powercycle", "normal", "wakeup",
+                          "update",     "sleep",  "maintenance"};
 
 void do_reset(bool warmstart) {
   if (warmstart) {
@@ -45,7 +46,8 @@ void do_after_reset(void) {
     break;
 
   case SW_CPU_RESET: // 0x0c Software reset CPU
-                     // keep previous runmode (could be RUNMODE_UPDATE)
+                     // keep previous runmode
+                     // (i.e. RUNMODE_UPDATE or RUNMODE_MAINTENANCE)
     break;
 
   case DEEPSLEEP_RESET: // 0x05 Deep Sleep reset digital core

src/wifiscan.cpp

@@ -82,7 +82,12 @@ void wifi_sniffer_init(void) {
 
 void switch_wifi_sniffer(uint8_t state) {
   if (state) {
-    // start sniffer
+// start sniffer
+#if (BLECOUNTER)
+    // workaround needed for ESP-IDF v3.3
+    // see https://github.com/espressif/esp-idf/issues/5427
+    esp_wifi_set_ps(WIFI_PS_MIN_MODEM);
+#endif
     esp_wifi_start();
     esp_wifi_set_promiscuous(true);
     esp_wifi_set_channel(WIFI_CHANNEL_MIN, WIFI_SECOND_CHAN_NONE);