Sentinel Vision Platform
Background
The Sentinel Vision Platform or SVP for short, was the final project for my ECE 350 - Digital Logic course. The course built upon what we had learned in ECE 250 - Intro to Computer Architecture in which we had built a very basic pipelined processor in Logisim. ECE 350 differed in terms of depth, breadth, and development format. Logisim is a pick and place visual editor where we manually laid out our gates and logic, whereas in ECE 350, we used structural Verilog to synthesize our processors. Throughout the semester we built a fully pipelined processor step by step: Addition Subtraction → Multiplication Division → Pipelining Jumping Hazards→ etc. Along the way we experimented with IO focused modules in our labs: Keyboard/Controller IO → Screens → speakers → etc. By the end of the course, we were tasked for the final project to use our processor - deployed on a Diligent Nexus A7-100T FPGA - in a meaningful way. Completely open ended and therefore my favorite final project thus far.
Motivation
Choosing a project was much easier than expected. I partnered up with my friend Lu - an all around amazing friend and fellow electronics team member on the FSAE team - and pitched the idea of making a vision guided laser turret.
"Thats exactly what I was thinking!"
There was absolutely no argument on the topic because, after all, who doesn't think laser turrets are cool !?
Platform
Control: Diligent Nexus A7-100T FPGA
Movement: 2x MG996R Servo | pan-tilt configuration
Visual: ESP32-CAM Module
Laser: Qianfabeam Green Laser Module GM532-30
Control & Body
Since we already had a full processor built out, most of our work centered around our project specific implementations. We built out some general custom modules such as:
- dmem mux → data
- imem mux → data
- packet parser → for UART
- pwm generators → servo control
- audio → for cool laser and targeting sounds
- laser gate & watchdog → safe and careful control over laser activation
Visual
We chose the ESP32 camera because it was:
- Easy to power: USB C
- Had many communication profiles: SPY UART WiFi
- Cheap and easy to hookup
Originally, we trained a small custom model to perform all of our object detection and deployed it on the small dev board that comes with the camera. Lu spent a non-trivial amount of time taking pictures of a small toy radio, and uploading them for the model to learn. When we tested this in the lab, the differences in background, mounting surface, and even light (many a times we were there from sunup to sundown) caused for a finicky and ultimately underperforming image model. I spent way too long debugging this and thinking there was an issue with my Verilog code (which there was - unrelated, but more on that later) and only realized the light issue when I moved from a dark workstation to one in front of a window. We decided to scrap that idea and ended up connecting the video feed to my laptop over WiFi, running a small YOLO object detection model, and beaming the coordinates back to the FPGA through a wired connection. This extensive data send process naturally created an interested bug - check the favorite bugs section to hear about it. From there, we were able to use those coordinates to calculate where we had to rotate our laser to, and sent commands to the servos to make that adjustment.
Laser Safety
Obviously one wouldn't want a laser just flying around a lab of students who were all definitely wearing the proper eye protection, so we implemented a set of safety measures against misfires, bugs, and electrical silliness. For our presentation we did have the proper eye protection.
- Software: The watchdog module controls the on signal for the laser. Essentially it waits for a certain number of packets where the camera is centered on the object to make sure it isn't fired if it sees an object it isn't completely confident in or if it pans quickly past something and wants to fire.
- FPGA Hardware: Our second line of defense was a switch on the FPGA board that wouldn't let the laser fire unless the LED was on and the switch was activated.
- Electrical Hardware: Our last line of defense was our buck converter. We stepped down our 5V that was powering the servos to 3.3V to power the laser. On the converter, there was a button that turned it on and let the power through. We generally turned this on when all other systems were green.
Favorite Bugs
Vision Overshoot Oscillation Loop: This bug was probably my favorite because it was a very simple and self introduced bug that when I realized what was happening, I couldn't help but feel disappointed in myself for not catching it earlier. Our original hookup between the cam and the FPGA was through UART and decently fast. When we introduced the laptop for processing, the video feed → laptop over wifi → Yolo model processing → FPGA loop added a ton of delay (relatively). This led to the servo moving past the object because the frames that detected the object hadn't been sent through the pipeline yet. Then it would realized it went too far, spin the other way, past the object again, for the same reason. Eventually it would just oscillated back and forth. This was relatively easily solved with a delay and block to additional capture after first detecting the object.
Video
This was a very informal video just to document the process. I never had the foresight to take a final production level video unfortunately and I never got my hands on the video that was used for grading.