SP-Tools are a set of marchine learning tools that are optimized for low latency and real-time performance. The tools can be used with Sensory Percussion sensors, ordinary drum triggers, or any audio input.
SP-Tools includes low latency onset detection, onset-based descriptor analysis, classification and clustering, corpus analysis and querying*, neural network predictive regression, and a slew of other abstractions that are optimized for drum and percussion sounds.
SP-Tools is built around the FluCoMa Toolkit and requires v1.0 to be installed for this package to work.
Max 8.3 or higher or Live/M4L (Mac/Windows).
FluCoMa v1.0 or higher.
All abstractions work in 64-bit and M1/Universal Binary.
SP-Tools Teaser Video
SP-Tools (alpha v0.1) Video Overview
SP-Tools (alpha v0.2) Video Overview
SP-Tools (alpha v0.3) Video Overview
Corpus-Based Sampler
Metal by the Foot 1/4
v0.3 - SP-Tools v0.3 Video Overview
- added ability to filter corpora by descriptors (baked into
sp.corpusmatchviafiltermessages) - added improved/unified corpus playback with
sp.corpusplayer~ - add realtime analysis abstractions (
sp.realtimeframe~,sp.descriptorsrt~,sp.melbandsrt~,sp.mfccrt~) - added new stereo corpus (corpus_plumbutter.json)
- improved corpus analysis to work with stereo files and files shorter than 100ms as well as adding more comprehensive metadata
- added
sp.corpuslistabstraction for visualizing and playing samples in a corpus in list form - removed old playback abstractions (
sp.corpussimpleplayer~,sp.corpusloudnessplayer~,sp.corpusmelbandplayer~)
v.02 - SP-Tools v0.2 Video Overview
- added "setups" (corpus scaling and neural network prediction/regression)
- added "controllers" (meta-parameters extracted from onset timings and descriptor analysis)
- added four new abstractions (
sp.controllers,sp.speed,sp.setupanalysis,sp.setuptrain~) - added new corpus (corpus_voice.json)
- added
@roundrobinmode tosp.corpusmatch
Depending on your knowledge level with machine learning processes, some of these terms may not make a lot of sense, so here is a short glossary to help you get going.
class: a category or label, or "zone" in Sensory Percussion lingo
classification: the process of defining and labelling classes
cluster: a category or label that is determined by a clustering algorithm
corpus: a pre-analyzed folder of samples
descriptors: analyzed characteristics of a sound
melbands: perceptually-spread frequency bands
MFCCs: a list of numbers that describes complex timbral shapes
onsets: an analyzed attack in audio
regression: interpolating or predicting a new point given training data
sp.classifierdisplay lets you visualize what classes have been matched by sp.classmatch. Can display the typical snare/tom classes as well as an option to visualize kick classes.
sp.classmatch will find the nearest match in a set of pre-trained classes or clusters. The classes can be the default Sensory Percussion classes, auto-generated cluster names, or any arbitrary labels.
sp.classtrain will create a classifier based on incoming class labels and descriptor analysis. The labels can be the default Sensory Percussion labels or any arbitrary input.
sp.clustertrain will create a classifier based on incoming class labels and descriptor analysis. The labels can be the default Sensory Percussion labels or any arbitrary input.
sp.controllers works in conjuntion with sp.descriptosr~/sp.descriptorframe to create several meta-parameters based on loudness and centroid (brightness).
sp.corpusanalysis works in conjunction with sp.folderloop to analyze all the samples in a folder for a variety of descriptors, timeframes, and metadata to be used in sp.corpusmatch. Keeps track of the location of the samples when analyzed.
Analyze all the samples in a folder for a variety of descriptors, timeframes, and metadata. Keeps track of the location of the samples when analyzed.
sp.corpuslist loads and displays the contents of the polybuffer~ at the center of a corpus, allowing the viewing of the selected samples.
sp.corpusmatch works in conjunction with sp.descriptors~ or sp.descriptorsframe to find the nearest match in a pre-analyzed corpus. sp.corpusmatch also houses the required datasets, coll, and polybuffer~.
sp.corpusplayer~ is an all-in-one playback object that allows for mono or stereo playback, optional loudness and spectral compensation, along with various sample playback controls and features.
sp.crossbank~ is a cascade of cross~ filters for spectral compensation. Frequencies are pre-set to adjust the spectrum based on the melband analysis/compensation. It should be used inside a poly~ object.
sp.descriptordisplay plots the incoming realtime descriptors, along with the nearest match on a radar chart for visualizing the differences between the incoming audio and its nearest match.
sp.descriptorframe outputs loudness, centroid, spectral flatness, and pitch along with the derivatives for loudness/centroid/flatness and confidence for pitch.
sp.descriptors~ outputs loudness, centroid, spectral flatness, and pitch along with the derivatives for loudness/centroid/flatness and confidence for pitch.
sp.descriptors~ outputs loudness, centroid, spectral flatness, and pitch along with the derivatives for loudness/centroid/flatness and confidence for pitch.
sp.folderloop is used in conjunction with sp.corpusanalysis to analyze every sample in a folder for the required descriptors and metadata.
sp.melbandframe outputs 40 melbands which can be used for spectral compensation in corpused-based sample playback.
sp.melbands~ outputs 40 melbands which can be used for spectral compensation in corpused-based sample playback
sp.melbands~ outputs 40 melbands which can be used for spectral compensation in corpused-based sample playback
sp.mfcc~ outputs 13 MFCC coefficients (skipping the 0th coefficient) which can be used for classification and clustering. Although abstract they can also be used to control parameters.
sp.mfcc~ outputs 13 MFCC coefficients (skipping the 0th coefficient) which can be used for classification and clustering. Although abstract they can also be used to control parameters.
sp.mfccframe outputs 13 MFCC coefficients (skipping the 0th coefficient) which can be used for classification and clustering. Although abstract they can also be used to control parameters.
sp.onset~ takes audio input and outputs a bang, trigger, and a gate when an onset is detected. The sensitivity is adjustable (0-100%) and a threshold can be set as an absolute noise floor (in dB).
sp.onsetframe~ takes audio input, just like sp.onset~ but instead of outputting just a bang/trigger/gate, it outputs the frame to start descriptor analysis. sp.onsetframe~ is useful when you want to analyze multiple descriptors and want them to all refer to the same exact analysis frame.
sp.playbackcore~ is the underlying poly~ that handles the polyphonic sample playback of matched corpus entries. It's not intended to be used on its own, but rather is the core component of sp.corpusplayer~.
sp.plotter is a utility for visualizing corpora and trained classes.
sp.realtimeframe~ is the counterpart to sp.onsetframe~ where instead of outputting the frame to analyzed based on onset detection, sp.realtimeframe~ spits out a constant stream of frame values to analyze enabling realtime analysis of multiple descriptor types that remain in sync.
sp.setupanalysis is used in conjunction with sp.onsetframe~ to create analyses for multiple descriptors at 256 and 4410 analysis windows. This is later used to improve matching with sp.corpusmatch.
sp.setuptrain creates a setup or overview of of your instrument/sticks/sources. It saves multiple descriptors at multiple time frames and can be used to scale your input to match a corpus, or to improve matching overall.
sp.speed works in conjunction with sp.onset~ to create several parameters based on the time between attacks.