This post takes a deep dive into Cantabile's ability to synthesize and send complex MIDI System Exclusive messages from a binding.
The idea for this post came from a question from Cantabile user Dave Dore about sending reverb parameter change messages to a Roland RD-2000 Stage Piano. The solution encompasses a few different areas so I thought it might make an interesting post:
- understanding sys-ex messages
- interpreting MIDI implementation charts
- writing the Cantabile sys-ex expression to generate the correct stream of bytes
Introduction to MIDI Sys-Ex Messages
The MIDI standard defines a number of standard MIDI messages, most of which you've probably heard of: note on, note off, control change, program change, pitch bend etc...
Less commonly used and understood are System Exclusive, aka "Sys-Ex" messages. You can generally think of these as a fallback for anything that doesn't fit into the standard set of messages. While there are some standard defined sys-ex messages, generally they're used by manufacturers of MIDI equipment for custom data that needs to be transmitted and received.
Sys-ex messages are used for things like bank and patch data, control parameters and even firmware updates.
Unlike other MIDI messages that have a fixed length, sys-ex messages are a variable length string of bytes that start with 0xF0, are followed by a series of bytes with values between 0x00 and 0x7f (the "payload") and terminated by a0xF7 byte. They're often represented as a series of hex encoded values like so:
F0 41 10 00 00 75 12 10 00 06 01 00 69 F7
Notice the sys-ex leading and terminating bytes and everything between which is the payload. It's the payload where things get interesting because the format of this is completely up to the device manufacturer and to work out what goes here we need to refer to the device's MIDI implementation chart.
MIDI Implementation Charts
Every device that sends or receives sys-ex messages should have a MIDI implementation chart the explains the format of these messages. For this walkthrough we'll be looking at the Roland RD-2000 - its implementation chart is available here. For other devices refer to the appropriate documentation.
(Although we're specifically looking at the RD2000 here, other Roland devices have very similar specs).
Once you've got the correct MIDI implementation chart it should contain all the information you need to work out what needs to be sent - although often the docs can be a little cryptic and you might need to refer to examples to decipher exactly what's going on.
Let's take a look at what the RD-2000 expects. The sys-ex specification starts on page 8 under the section "System Exclusive Messages" where it says that Data Set 1 (DT1) are the only sys-ex messages... and a little further down it explains the Data Set 1:
(Note that the Roland docs use a H suffix to indicate hex numbers - in this post everything will be assumed hex unless otherwise stated.)
Above we can see the familiar sys-ex start byte F0 on the left, the payload in the middle column labelled ("2nd byte") and the terminator F7 in the third column. The document then proceeds to explain what each byte is if you're interested.
We can now start building up our sys-ex message by copying the above and substituting 10 for dev - a device number which defaults to 10 for the RD2000.
F0 41 10 00 00 75 12...
Next we have the numbers aaH, bbH, ccH and ddH which are the "starting address of the data to be sent". What are these? Well, if we scroll down a little further we come to section 3 the "Parameter Address Map". This rather convoluted section of the document explains the device's address map - which is where in the RD-2000's memory the various configuration parameters are stored. For this example, we'll look at how to set the Program Reverb parameters.
Starting at the top, we can see that the "Program" parameters start at address 10 00 00 00:
Next, we see that the "Program Reverb" parameters start at offset 00 06 00 from the program area:
That gives a base address of 0x10000000 + 0x00000600 = 0x10000600 for the program reverb parameters. To get the final address for a particular parameter we then need to look further down the document for the individual parameter addresses:
Let's say we want to set reverb parameter 2, we need to add 00 06 to our base address to yield a final parameter address of 10 00 06 06 which forms the next bit of our sys-ex message:
F0 41 10 00 00 75 12 10 00 06 06...
Next we need the actual data to be passed for this parameter and that's shown in the same table:
It's not at obvious at first, but the above indicates that we need to send 4 bytes for the data. The 0000 aaaa is a bit pattern indicating a byte with 4 bits of data in the lower 4-bits - ie: a number between 0x00 and 0x0F.
Another way to think of this is we have a 16-bit value we need to assign to the parameter and we take that 16-bit value in aaaabbbbccccddd bit format and repack it as shown above.
For example a 16-bit value of 0x1234 would be sent as 01 02 03 04.
OK, but what 16-bit numbers do we want to send? Take a look at the right hand side of the above table. Notice the (12768 - 52768) with -20000 - +20000 underneath it. That means to set the parameter value to -20000 you need to send the value 12768. In other words you need to add 32768 to the value you want to send before sending it.
Now, you'd think then each parameter must have values ranging from -20,000 to +20,000, but not so fast - after all, we don't even really know what Reverb Parameter 2 is. We need to refer to a second document for that: the RD-2000 Parameter Guide. Scroll down to page 67 and you'll see that the meaning of each reverb parameter depends on the reverb type selected. For Room 1/2, Hall 1/2 and Plate reverbs, the parameter map is as follows:
Reverb Parameter 2 is the Pre-Delay parameter and we can see that it accepts a value from 0 to 100. But, we need to add 32768 to that value before we send it.
So suppose we want to set the Pre-Delay to 50 milliseconds:
50 + 32768 = 32818 (decimal) = 8032 (hex)
and that's the next part of our sys-ex message (after repacking into the low 4-bits of each data byte):
F0 41 10 00 00 75 12 10 00 06 06 08 00 03 02...
Finally the Roland spec says we need a checksum byte which for this example is 0x57 (the spec shows how to calculate this, but Cantabile has a built in function to do it too) and the sys-ex terminator byte F7 which completes the message:
F0 41 10 00 00 75 12 10 00 06 06 08 00 03 02 57 F7
Phew! Congratulations if you followed all of that.
The Roland MIDI implementation chart is complex to figure out but once you understand how it works it's not that hard to adjust for other parameters and settings. Not all devices and MIDI implementations are this hard to work out.
In any case, we now know how to work out what to send... now let's now configure Cantabile to send it.
(Quick side note. You'll notice for the Time parameter above it indicates a value from 0.1 to 10.0. You actually need to multiply the value by 10. For example, to set it to 3.2 seconds send 32).
Cantabile Sys-Ex Expressions
Cantabile Performer has a very flexible sys-ex expression engine that can be used to generate complex MIDI messages like what we're dealing with here.
For this example I'm going to map CC 16 from the on-screen keyboard to send a Sys-Ex event to the RD-2000's Pre-Delay reverb parameter.
You can click on "(no data)" to enter the sys-ex expression which we'll now work out...
Firstly, CC values range from 0 to 127, but the Pre-Delay reverb parameter expects a value from 0 to 100, so lets scale it appropriately, add the 32768 and convert it to an integer. This is the 16-bit value we need to send.
var val16 = int(value * 100 / 127 + 32768)
(value is a built in variable that will contain the sent CC value)
Next we need to format the address and value part of the payload. We do this in a separate variable as we need to calculate the checksum of this later.
var addrAndValue = [x"10 00 06 06",
(val16 >> 12) & 0x0F, // aaaa
(val16 >> 8) & 0x0F, // bbbb
(val16 >> 4) & 0x0f, // cccc
val16 & 0x0f, // dddd
]
To explain the above, (val16 >> 8) means to shift the value 8 bits to the right filling the left with zeros:
aaaabbbbccccdddd >> 8 = 00000000aaaabbbb
The & 0xF means to bitwise and it with the binary value 0000000000001111 which effectively masks out the upper bits just keeping the lower 4.
00000000aaaabbbb & 0000000000001111 = 000000000000bbbb
ie: we’ve extracted the bbbb bits. Similar for the other 4-bit parts.
Finally, we generate the full sys-ex message including the sys-ex lead/terminator bytes and generate the checksum byte with Cantabile's built-in checksum function:
0xF0 x"41 10 00 00 75 12" addrAndValue RolandChecksum(addrAndValue) 0xF7
Putting it altogether:
// Map value range and add 32768 as required by RD2000
var val16 = int(value * 100 / 127 + 32768)
// Format parameter address and value
var addrAndValue = [x"10 00 06 06",
(val16 >> 12) & 0x0F, // aaaa
(val16 >> 8) & 0x0F, // bbbb
(val16 >> 4) & 0x0f, // cccc
val16 & 0x0f, // dddd
]
// Generate message
0xF0 x"41 10 00 00 75 12" addrAndValue RolandChecksum(addrAndValue) 0xF7
And that's it!
A Less Verbose Version
After working through all this and getting it working it became obvious that the above is a bit verbose with all the bit-shifting and masking. So the newest builds of Cantabile now include two new functions to support easier packing of the data values:
RolandPack16()- packs a 16-bit value into 4 bytesRolandPack8()- packs an 8-bit value into 2 bytes.
This simplifies the final sys-ex expression to:
// Map value range and add 32768 as required by RD2000
var val16 = int(value * 100 / 127 + 32768)
// Format parameter address and value
var addrAndValue = [x"10 00 06 06", RolandPack16(val16)]
// Generate message
0xF0 x"41 10 00 00 75 12" addrAndValue RolandChecksum(addrAndValue) 0xF7
Conclusion
This was a long and detailed look at generating a fairly complex MIDI Sys-Ex message but I hope it gives a good idea of what's possible.
Half of the challenge is interpreting and deciphering the MIDI implementation chart for your device. Often there will be examples but usually it'll come down to some trial and error too.
More documentation on Cantabile's Sys-Ex expression engine is available here.