From Gesture to Action: Integrating ESP32-S3 Edge AI with ASI Biont via MQTT

Introduction

Edge AI is no longer a buzzword—it's a practical reality for embedded systems. The ESP32-S3, with its integrated neural network accelerator (ESP-DL) and support for TensorFlow Lite Micro, allows you to run machine learning models directly on the microcontroller, without sending raw data to the cloud. But what happens after the model makes a prediction? How do you turn a detected hand gesture, a keyword, or an anomaly into a real-world action—like turning on a light, sending a Telegram alert, or logging data to a spreadsheet?

This is where ASI Biont comes in. ASI Biont is an AI agent that connects to virtually any device—from industrial PLCs to single-board computers—and automates tasks based on your instructions. In this guide, we'll walk through a concrete integration: an ESP32-S3 running a TensorFlow Lite Micro model for gesture recognition, sending its predictions via MQTT to a local broker, and an ASI Biont AI agent that subscribes to those messages, analyzes them, and triggers actions. No cloud dependency, no manual coding of integration logic—just describe what you need, and the AI writes the code.

Why Connect ESP32-S3 to an AI Agent?

The ESP32-S3 is powerful for on-device inference, but it has limited memory and storage for complex decision-making or multi-device orchestration. By connecting it to ASI Biont, you gain:
- Contextual automation: The AI agent can combine sensor data from multiple devices (e.g., gesture + time of day + humidity) before acting.
- Flexible output channels: Send commands to smart plugs, update databases, send emails, or call external APIs—all through the agent's industrial_command tool.
- No-code integration changes: Want to change the action from turning on a light to sending a Slack message? Just ask the AI in natural language.

Connection Architecture

We'll use MQTT as the bridge between the ESP32-S3 and the ASI Biont AI agent. Here's why MQTT is ideal:
- Lightweight and publish/subscribe: Perfect for constrained devices like ESP32.
- Local broker: Keep data on your LAN for low latency and privacy.
- Standard protocol: Works with any MQTT client, including paho-mqtt in ASI Biont's sandbox.

The flow is:
1. ESP32-S3 runs a TFLite Micro model (e.g., gesture classifier).
2. On each inference, it publishes a JSON message to topic esp32/gesture via MQTT.
3. A local broker (Mosquitto) receives the message.
4. ASI Biont's AI agent, via execute_python, runs a Python script that subscribes to the broker, parses the message, and uses industrial_command to trigger an action (e.g., write_register on a Modbus light controller or publish to another topic for a smart plug).

Real-World Scenario: Gesture-Controlled Smart Light

Imagine a workshop where you need hands-free control of a bench light while your hands are dirty. The ESP32-S3 is mounted on the wall, running a gesture recognition model (trained on gestures like "swipe left" for off, "swipe right" for on). When you swipe, the ESP32 publishes {"gesture": "on"} to the MQTT broker. The ASI Biont agent, subscribed to that topic, receives the message and sends an MQTT command to a smart plug (or directly to a relay via Modbus) to toggle the light.

Step 1: ESP32-S3 Code (Arduino / PlatformIO)

The ESP32 uses the TensorFlow Lite Micro library and ESP-DL to run the model. After inference, it connects to the MQTT broker and publishes the result.

#include <WiFi.h>
#include <PubSubClient.h>
#include <TensorFlowLite_ESP32.h>
#include "model.h" // your TFLite model as a byte array

const char* ssid = "your_SSID";
const char* password = "your_PASS";
const char* mqtt_server = "192.168.1.100"; // broker IP
const char* topic = "esp32/gesture";

WiFiClient espClient;
PubSubClient client(espClient);

tflite::MicroInterpreter* interpreter;
static tflite::MicroMutableOpResolver<10> resolver;

void setup() {
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) { delay(500); }
  client.setServer(mqtt_server, 1883);
  client.connect("ESP32Gesture");

  // Initialize TFLite interpreter
  static tflite::MicroErrorReporter error_reporter;
  static constexpr int tensor_arena_size = 16 * 1024;
  static uint8_t tensor_arena[tensor_arena_size];
  interpreter = new tflite::MicroInterpreter(
    g_model, resolver, tensor_arena, tensor_arena_size, &error_reporter);
}

void loop() {
  // Read sensor (e.g., accelerometer) and run inference
  float input[128]; // example 128-element feature vector
  memcpy(interpreter->input(0)->data.f, input, sizeof(input));
  interpreter->Invoke();

  float* output = interpreter->output(0)->data.f;
  int predicted_class = std::distance(output, std::max_element(output, output + 3));

  const char* gestures[] = {"off", "on", "toggle"};
  char payload[64];
  snprintf(payload, sizeof(payload), "{\"gesture\": \"%s\"}", gestures[predicted_class]);

  client.publish(topic, payload);
  delay(1000); // inference every second
}

Step 2: MQTT Broker Setup

Install Mosquitto on a local server or Raspberry Pi:

sudo apt install mosquitto mosquitto-clients
sudo systemctl enable mosquitto

No authentication required for a local network, but you can add username/password for security.

Step 3: ASI Biont AI Agent Integration

Now the magic happens. Instead of writing a Python script yourself, you simply describe the task in the chat with the AI agent. For example:

"Connect to MQTT broker at 192.168.1.100:1883, subscribe to topic 'esp32/gesture'. When a message with gesture 'on' arrives, publish a command to activate a smart plug on topic 'smartplug/light' with payload 'ON'. When gesture is 'off', publish 'OFF'."

The AI agent will generate and execute the following Python script (you can inspect it before running):

import paho.mqtt.client as mqtt
import json

def on_message(client, userdata, msg):
    try:
        data = json.loads(msg.payload)
        gesture = data.get("gesture")
        if gesture == "on":
            userdata["command_client"].publish("smartplug/light", "ON")
            print(f"Published ON to smartplug/light")
        elif gesture == "off":
            userdata["command_client"].publish("smartplug/light", "OFF")
            print(f"Published OFF to smartplug/light")
        else:
            print(f"Unknown gesture: {gesture}")
    except Exception as e:
        print(f"Error: {e}")

# Main subscription client
sub_client = mqtt.Client()
sub_client.user_data_set({"command_client": mqtt.Client()})
sub_client.on_message = on_message
sub_client.connect("192.168.1.100", 1883, 60)
sub_client.subscribe("esp32/gesture")
sub_client.loop_forever()

Note: The sandbox has a 30-second timeout, so for long-running subscriptions, the AI agent may use the industrial_command tool with protocol mqtt:// to subscribe persistently. However, for demonstration or short tasks, the execute_python method works perfectly.

Alternative: Using Hardware Bridge with COM Port

If your ESP32-S3 is connected via USB (e.g., as a serial device), you can use the Hardware Bridge. The user runs bridge.py on their PC, and the AI agent sends commands like serial://COM3?baud=115200 to read serial data. This is useful if you want to send raw inference results over UART instead of MQTT.

# On the PC
python bridge.py --token=YOUR_API_KEY --ports=COM3 --default-baud=115200

Then in ASI Biont chat:

"Read from COM3 at 115200 baud. The device sends JSON lines like {\"gesture\":\"on\"}. When gesture is 'on', write '1' to a Modbus coil at address 0 on 192.168.1.50:502."

The AI generates the integration using industrial_command with protocol modbus:// and serial://.

Why This Approach Wins

Conclusion

The combination of ESP32-S3's on-device ML and ASI Biont's AI agent creates a powerful edge-to-action pipeline. You keep inference local for low latency and privacy, while the AI agent handles the messy world of integration—parsing messages, calling APIs, controlling hardware, and adapting to new requirements on the fly. No need to wait for a dashboard or a custom backend; just describe what you want, and it happens.

Ready to try it yourself? Head over to asibiont.com, create an API key, and start integrating your ESP32-S3 with the AI agent today. Whether it's gesture-controlled lighting, voice-triggered automation, or predictive maintenance, ASI Biont turns your edge AI into real-world action.