Show HN: I made an audio-to-MIDI model 60x more efficient
kobi.musicYes! Offset and velocity both are provided.
Does the model know when a piece (whole piece, not a section) is 3/4 or 4/4 th and can it give the onsets of all measures? Or all the midi notes just based on time and not organized rhythmically?
Classical Music as an academic concern is very pedantic in the best way, and has the lions share of the research hours and funding compared to any other genre. As a result the best 1:1 approximations of multi-instrument Audio into multi-channel MIDI files is done simply by OCR of the established Gesamtausgabe in each case. Computer based scoring and playback has been solid since Finale/Sibelius/Score came on the market, and democratized at a wide consumer level somewhere between Amiga mod-trackers and Macintosh Quicktime 2.0.
This is probably where I would find it limiting range of sounds can be huge, much more than just 6 sounds (10000+) and not all of them percussive. Worth an eval though, thanks!
Anyway, it would be awesome to be able to do those sorts of transcriptions better and faster - so I'll definitely give this a whirl. Maybe I can dig up the pipeline scripts from processing my licks for that talk and see how dettato slots in.
You might want to implement deduplication for your website so impatient users do not convert the same mp3 multiple times (at multiple times the price) when they click "run audio to midi" and the server takes a bit longer to respond.
The top illustration looks like an Angry Birds character with an ear. Fine for a one time blog post but if you plan to reuse it then you may want to change the design.
As a guitarist who can't play keys well, I'd love to use something like this to help me transcribe string arrangements.
What do you use for the audio engine? I see that ardour, developed for linux, also runs on MacOS, although it is less tested on your platform.
Probably you're well along already, and it may not be the time to discuss architecture, but...
Ecasound, which I use for audio processing, compiles on MacOS. It is a very efficient library for audio processing, however it is limited to internal, LADSPA and LV2 plugins. If you use the JACK audio framework, there are hosts such as Carla that accommodate VST and VST3 plugins, however I can't say how well tested they are on MacOS.
When you include MIDI, automation, etc. there is a huge feature set. Probably you can get away with the minimum necessary for your use case.
Interested to hear whatever you'd like to share.
Cheers and good luck!
I haven't tried anything outside of 12TET, but probably not good (it predicts in 12TET). Alternative tunings should work fine (it re-tunes to a440 for the model).
This is why I believe (AMT, OMR) and MIR-adjacent projects to be so much more meaningful than generative models (e.g. Suno). I'm looking forward to hearing more from kobi music - congrats on the launch!
Unfortunately, Phish live recordings are copyrighted, so it's unlikely you can put them in the training corpus. Demonstrating that you can deconvolute and transcribe multiple different instruments with highly overlapping frequencies and sounds faster than an expert human would be a great demonstrate of the capabilities.
I'm so glad to see a smaller model being so accurate, MuScriptor is especially bad on the Romanian kaval, can't wait to try this in-app. I do it with already separated stems through BS-RoFormer so I'm hoping this can improve accuracy a bit.
And it pretty much nailed it note for note, including drums. Only fault is some mix up of instruments, e.g. electric bass turns into electric guitar in higher registers which is totally excusable.
can you go into more detail here? how did you make synthetic instrument recordings? with virtual instruments or generative AI?
I've got a use-case for this, but I'm wondering how "stuck" in this format this is. By format I mean, basically, python runtime, etc. Would it be possible to extract into something more usable by a consumer, like into a standalone rust module? Can anyone poitn me to how this would be easy?
That said, this is awesome.