A collaborative computational cooking robot built using computer vision with Raspberry Pi
Web application for control panel and sous chef interfaces
See it in action on YouTube | Read more about the project on DesignSpark
Collaborative computational cooking using computer vision. The system combines human and robot cooking: (b) auto management of heating; (c) intelligent reminders and warnings; (d) screen instructions for the human to execute.
Food image classification is tricky, as food images often have numerous difficult-to-define features and a lot of environmental variation. A new cooking device must tackle these perception problems.
OnionBot introduces two improvements:
- The fixed stove-top camera view ensures a consistent environment
- With general classification, the model must identify characteristics from 1000s of potential classes. Instead, we classify only key events at which actions occur (milestones) for a single recipe. Each model must identify 10s of classes (or less!), dramatically simplifying the perception challenge.
These datasets don’t currently exist, so the OnionBot-UI control panel enables easy creation of labelled datasets of cooking images. Simply click along with each milestone as you cook, and labelled images are automatically uploaded to Google Cloud, where they can be accessed by model training platforms. Training is simplified using Google AutoML; AutoML allows models to be trained for new recipes with just a few clicks, no ML expertise required!
A) Examples from image classes created for the BBC GoodFood tomato sauce recipe. B) The Live Labelling interface makes creating labelled datasets as simple as clicking along with recipe milestones.
- Creates inbalanced datasets when some steps take longer than others. Should take a balanced sample from each recipe milestone.
- AutoML makes training easy, but can get expensive and gives no insight into the model trained
- Improve intuitiveness of recipe following when creating datasets
- Reduce data cleaning required, move towards automatic model training
- Start your local web server of choice (we use Apache)
- Point the server to
onionbot/portal - Start cooking!
Get in touch!