METEOR: Melody-aware Texture-controllable
Symbolic Orchestral Music Generation

| paper | code | demo |

Dinh-Viet-Toan Le, Yi-Hsuan Yang

Abstract. Western music is often characterized by a homophonic texture, in which the musical content can be organized into a melody and an accompaniment. In orchestral music, in particular, the composer can select specific characteristics for each instrument's part within the accompaniment, while also needing to adapt the melody to suit the capabilities of the instruments performing it. In this work, we propose METEOR, a model for Melody-aware Texture-controllable Orchestral music generation. This model performs symbolic multi-track music style transfer with a focus on melodic fidelity. We allow bar- and track-level controllability of the accompaniment with various textural attributes while keeping a homophonic texture. We show that the model can achieve controllability performances similar to strong baselines while greatly improve melodic fidelity.

Style transfer task

Audio samples are synthesized from the generated MIDI files using the Musescore default soundfonts.

Style transfer

The indicated polyphonicity and rhythmicity values are differences with the analyzed value from the reference piece.

Example 1

Reference file: 3814911.mid (SymphonyNet dataset), bar 34

Reference
METEOR + melody
Instruments: Auto
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Auto

Instrumentation

Instruments: Solo piano
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Piano
Instruments: Woodwind quintet
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Flute
Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Trumpet

Bar-level controls

Instruments: Auto
Polyphonicity: +3
Rhythmicity: Kept
Melody: Auto
Instruments: Auto
Polyphonicity: -3
Rhythmicity: Kept
Melody: Auto
Instruments: Auto
Polyphonicity: Kept
Rhythmicity: +3
Melody: Auto
Instruments: Auto
Polyphonicity: Kept
Rhythmicity: -3
Melody: Auto

Instrumentation + Bar-level controls

Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: +3
Melody: Trumpet
Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: -3
Melody: Trumpet
Instruments: Solo Piano
Polyphonicity: Kept
Rhythmicity: +3
Melody: Piano
Instruments: Solo Piano
Polyphonicity: Kept
Rhythmicity: -3
Melody: Piano

Instrumentation + Track-level controls

Instruments: String trio
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Violin
Pitch average: Auto
Pitch diversity:
  • Violin: High (5/12)
  • Viola: High (5/12)
  • Cello: Low (1/12)
Instruments: Piano trio
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Cello then violin
Pitch diversity: Auto
Pitch average:
  • Violin: Low then high (60 then 90/130)
  • Cello: High then low (70 then 40/130)
  • Piano: Medium (60/130)
METEOR no melody

Instrumentation + Track-level controls

Instruments: String trio
Polyphonicity: Kept
Rhythmicity: Kept
Pitch average: Auto
Pitch diversity:
  • Violin: High (5/12)
  • Viola: High (5/12)
  • Cello: Low (1/12)
Instruments: Piano trio
Polyphonicity: Kept
Rhythmicity: Kept
Pitch diversity: Auto
Pitch average:
  • Violin: Low then high (60 then 90/130)
  • Cello: High then low (70 then 40/130)
  • Piano: Medium (60/130)
MuseMorphose Multi-track

Instrumentation + Bar-level controls

Instruments: Solo piano
Polyphonicity: Kept
Rhythmicity: Kept
Instruments: Solo piano
Polyphonicity: +3
Rhythmicity: +3
Instruments: Solo piano
Polyphonicity: -3
Rhythmicity: -3
Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: Kept
Instruments: Classical orchestra
Polyphonicity: +3
Rhythmicity: +3
Instruments: Classical orchestra
Polyphonicity: -3
Rhythmicity: -3
FIGARO
Sample 1
Sample 2
Sample 3

Example 2

Reference file: 268756.mid (SymphonyNet dataset), bar 8

Reference
METEOR + melody
Instruments: Auto
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Auto

Instrumentation

Instruments: Solo piano
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Piano
Instruments: Piano trio
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Violin
Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Flute

Bar-level controls

Instruments: Auto
Polyphonicity: +3
Rhythmicity: Kept
Melody: Auto
Instruments: Auto
Polyphonicity: -3
Rhythmicity: Kept
Melody: Auto
Instruments: Auto
Polyphonicity: Kept
Rhythmicity: +3
Melody: Auto
Instruments: Auto
Polyphonicity: Kept
Rhythmicity: -3
Melody: Auto

Instrumentation + Bar-level controls

Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: +3
Melody: Flute
Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: -3
Melody: Flute
Instruments: Solo Piano
Polyphonicity: Kept
Rhythmicity: +3
Melody: Piano
Instruments: Solo Piano
Polyphonicity: Kept
Rhythmicity: -3
Melody: Piano

Instrumentation + Track-level controls

Instruments: String trio
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Violin
Pitch average: Auto
Pitch diversity:
  • Violin: High (5/12)
  • Viola: High (5/12)
  • Cello: Low (1/12)
Instruments: Piano trio
Polyphonicity: Kept
Rhythmicity: Kept
Melody: Violin then cello
Pitch diversity: Auto
Pitch average:
  • Violin: High then low (60 then 90/130)
  • Cello: Low then high (40 then 70/130)
  • Piano: Medium (60/130)
METEOR no melody

Instrumentation + Track-level controls

Instruments: String trio
Polyphonicity: Kept
Rhythmicity: Kept
Pitch average: Auto
Pitch diversity:
  • Violin: High (5/12)
  • Viola: High (5/12)
  • Cello: Low (1/12)
Instruments: Piano trio
Polyphonicity: Kept
Rhythmicity: Kept
Pitch diversity: Auto
Pitch average:
  • Violin: High then low (60 then 90/130)
  • Cello: Low then high (40 then 70/130)
  • Piano: Medium (60/130)
MuseMorphose Multi-track

Instrumentation + Bar-level controls

Instruments: Solo piano
Polyphonicity: Kept
Rhythmicity: Kept
Instruments: Solo piano
Polyphonicity: +3
Rhythmicity: +3
Instruments: Solo piano
Polyphonicity: -3
Rhythmicity: -3
Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: Kept
Instruments: Classical orchestra
Polyphonicity: +3
Rhythmicity: +3
Instruments: Classical orchestra
Polyphonicity: -3
Rhythmicity: -3
FIGARO
Sample 1
Sample 2
Sample 3

Lead sheet orchestration

Example 1

Auld Lang Syne
Reference
Instruments: Solo piano
Polyphonicity: Kept
Rhythmicity: +3
Melody: Auto
Instruments: Woodwind + Brass
Polyphonicity: Kept
Rhythmicity: +3
Melody: Trumpet
Instruments: Woodwind + Brass
Polyphonicity: Kept
Rhythmicity: +5
Melody: Trumpet
Instruments: String quartet
Polyphonicity: Kept
Rhythmicity: +3
Melody: Violin
Altered harmony
(F chords → Fm chords)
Altered harmony
Instruments: String quartet
Polyphonicity: Kept
Rhythmicity: +3
Melody: Auto

Example 2

Ding Dong Merrily on High
Reference
Instruments: Solo piano
Polyphonicity: Kept
Rhythmicity: +3
Melody: Auto
Instruments: Classical orchestra
Polyphonicity: Kept
Rhythmicity: +3
Melody: Clarinet
Instruments: Wind quintet
Polyphonicity: Kept
Rhythmicity: +3
Melody: Flute
Instruments: Synths
Polyphonicity: Kept
Rhythmicity: +3
Melody: Lead
Instruments: Percussion ensemble
Polyphonicity: Kept
Rhythmicity: +3
Melody: Glockenspiel

Pitch class impact

Reference
Detected melodic instrument: Glockenspiel, then flute

In the following samples, bar-level and track-level controls are unmodified. The instrumentation is automatically chosen and the melodic instrument is enforced. The melodic tracks are represented in black on the pianorolls.

Melodic instrument: Flute
Melodic instrument: Bassoon
Melodic instrument: Trumpet
Melodic instrument: Violin
Melodic instrument: Cello