CPC_Practicals

Workshop 05 - gltf models

Hello and welcome back!

This fifth workshop (in week six) is aimed at helping you understand how we can undertake a few different tasks to useful to us while designing our projects. So, we will:

It’s little bit silly to have a flamingo and a dancing robot, but here’s what we’re going to be building in the first two tasks to show you how to load models and how to drive movements with both pre-defined animations and with real-time sound properties:

Where code is provided, you are expected to write it out yourself rather than copying and pasting it. We know this is tempting, but the point of a university education is for you to learn the necessary skills for your field, and copying and pasting code will not be on any job descriptions.

Task 1 - Loading Models

Right, we’re going to use a starter project again this week. So please fork the project below. it’s basically just our normal template but we’ve added some models for you to work with. Take a look around the project linked below and fork it so that you can start working on it. You can see at the top of index.js that we have already imported our GLTF loader that we’re going to use to load the models in our models folder.

OK, as usual let’s get going by adding some global variables towards the top of our index.js file:

let modelLoaded;
let robot, flamingo;


let loader;
let mixers;

Now, in init() just below where we remove the overlay, let’s start our Tone js instance and initialise our modelLoaded boolean to false. This will be set to true once our models have been loaded and we can then use them in the rest of our code.

Tone.start();
modelLoaded = false;

OK let’s initialise mixers to an array and make a call to loadModels function. The mixers array will be an array that will hold our model’s animations. The main bulk of our code is now going to be in the loadModels function. You will be getting an error now sayin that loadModels() is not defined. So let’s move onto the next step.

mixers = [];
loadModels();

Now, at the very bottom of our index.js file, let’s define our loadModels function to instantiate our new gltfLoader:

function loadModels() 
{
    loader = new GLTFLoader();
}

We’re going to continue working in our loadModels function and define two callbacks that will process our models once they are loaded by the GLTFLoader.

We’re going to give you the first callback for loading models and animations, then we’d like you to have a go at a simpler callback for loading models without animations.

There’s quite a bit of code in this onLoadAnimation function, so comments are provided to explain what is going on:

// this callback handles loading a flamingo GLTF model with animation data
const onLoadAnimation = function (gltf, position) {
    flamingo = gltf.scene.children[0]; // look for the first child of the scene contained in the gltf - this is our flamingo model
    flamingo.scale.multiplyScalar(0.125); // scale our model to make it smaller
    flamingo.position.copy(position); // set the desired position
    
    const animation = gltf.animations[0]; // get animation data from the gltf file and assign it to a varible called animation

    const mixer = new THREE.AnimationMixer(flamingo); //create a new ThreeJS animation mixer and pass our flamingo model to it

    mixers.push(mixer); // add our animation mixer to our mixers array

    const action = mixer.clipAction(animation); // pass the animation data to the animation scheduler in the animation mixer
    action.play(); // start the animation

    scene.add(flamingo); // add our animated flamingo model to our scene
};

Adding this code will not (yet!) display our flamingo - since we haven’t yet added code to import the gltf file… Read on, to find out how to handle static (robot) models, very soon we’ll be able to see the fruit of our labour. 😊

Static robot

Ok, now you’ve seen how to load models and their animations. Can you try and write your own callback called onLoadStatic that loads the static (i.e no animation data) robot GLTF model?:

Be sure you write onLoadStatic as a const function declaration inside loadModels (just like the onLoadAnimation above).

HINT All of the functionality described above, apart from the setting of the modelLoaded flag can be adapted from the onLoadAnimation callback function definition.

Don't worry if this is a bit of a head scratcher. This is what your onLoadStatic function should look like 
const onLoadStatic = function (gltf, position) 
{
        robot = gltf.scene.children[0]; // assign the first child of the scene contained in the gltf file to a variable called robot
        robot.scale.multiplyScalar(1.125);
        robot.position.copy(position); //copy the position passed from the load function call

        modelLoaded = true; // once our model has loaded, set our modelLoaded boolean flag to true
        scene.add(robot); // add our model to the scene
};

Let’s add a couple more const callback functions that will help tell us that the models are loading and if there are any errors do this in loadModels right next to where we have just added onLoadAnimation and onLoadStatic:

  // the loader will report the loading progress to this function
  const onProgress = function () 
  {
    console.log("progress");
  };

  // the loader will send any error messages to this function
  const onError = function (errorMessage) 
  {
    console.log(errorMessage);
  };

And now beneath all of these callback functions, but still in the loadModels function, let’s create some initial positions for our models, then actually invoke the loader.

This code below will set up the animated flamingo loader with all the associated callbacks, but it’s up to you to have a go at the robot:

// desired position of our flamingo
const flamingoPosition = new THREE.Vector3(-7.5, 0, -10); 

// load the GLTF file with all required callback functions
loader.load(
    "models/Flamingo.glb", // specify our file path
    function (gltf) {
      // specify the callback function to call once the model has loaded
      onLoadAnimation(gltf, flamingoPosition);
    },
    onProgress, // specify progress callback
    onError // specify error callback
    );

So, now you’ve made the flamingo load, you need to write code to load the robot model and put it at a position of 0, 0, 0 i.e right in the middle of our scene. You will need a variable named robotPos and should call the onLoadStatic callback.

Hopefully this should have been smooth sailing for you. Do ask for help if you need it.

This is what your robot loading code should look like 
const robotPos = new THREE.Vector3(0, 0, 0);
loader.load(
        "models/robot.gltf", // robot file path
        function (gltf) {
        onLoadStatic(gltf, robotPos); // remember the robot model has no animation
        },
        onProgress, // specify progress callback
        onError     // specify error callback
        );

You may notice that our flamingo is a .glb file and the robot is .gltf. They’re essentially just different ways of formatting model data. Take a look here to see the difference.

This is what our final loadModels function ends up looking like 
function loadModels() 
{
  loader = new GLTFLoader();

  // this function loads a flamingo model with animation data
  // note we're passing in a position parameter
  const onLoadAnimation = function (gltf, position) {
    flamingo = gltf.scene.children[0]; // assign the first child of the scene contained in the gltf file to a variable called flamingo
    flamingo.scale.multiplyScalar(0.125); // scale our model to make it smaller
    flamingo.position.copy(position); //copy the position passed from the load function call

    const animation = gltf.animations[0]; // get the animation

    const mixer = new THREE.AnimationMixer(flamingo); //create a new animation mixer and assign pass our new model to it

    mixers.push(mixer); // add our animation mixer to our mixers array

    const action = mixer.clipAction(animation); // pass the animation to the animation scheduler in the animation mixer
    action.play(); // start the animation scheduling

    scene.add(flamingo); // add our model to our scene
  };

  // this function loads a robot model without animation
  const onLoadStatic = function (gltf, position) {
    robot = gltf.scene.children[0]; // assign the first child of the scene contained in the gltf file to a variable called robot
    robot.scale.multiplyScalar(1.125);
    robot.position.copy(position); //copy the position passed from the load function call

    modelLoaded = true; // once our model has loaded, set our modelLoaded boolean flag to true
    scene.add(robot); // add our model to the scene
  };

  // the loader will report the loading progress to this function
  const onProgress = function () {
    console.log("progress");
  };

  // the loader will send any error messages to this function, and we'll log
  // them to to console
  const onError = function (errorMessage) {
    console.log(errorMessage);
  };

  const flamingoPosition = new THREE.Vector3(-7.5, 0, -10); // create new vector for the position of our flamingo
  loader.load(
    // call the loader's load function
    "models/Flamingo.glb", // specify our file path
    function (gltf) {
      // specify the callback function to call once the model has loaded
      onLoadAnimation(gltf, flamingoPosition);
    },
    onProgress, // specify progress callback
    onError // specify error callback
  );

  const robotPos = new THREE.Vector3(0, 0, 0);
  loader.load(
    // call the loader's load function
    "models/robot.gltf", // specify our file path
    function (gltf) {
      // specify the callback function to call once the model has loaded
      onLoadStatic(gltf, robotPos);
    },
    onProgress, // specify progress callback
    onError // specify error callback
  );
}

Is this thing on?

Right, hopefully you can now see some models in your scene! But please do ask a tutor for some assistance if you can’t see anything!

The only thing is, our flamingo doesn’t seem to be doing anything. So now we need to actually animate it.

We just need to update our animation driving it with our delta variable. Add the following lines INSIDE the time dependent “if statement” in our update function:

for (let i = 0; i < mixers.length; i++) {
      mixers[i].update(delta);
    }
The entire update function should now look like this 
// our update function
function update() 
{
  orbit.update();
  //update stuff in here
  delta += clock.getDelta();

  if (delta > interval) 
  {
    // The draw or time dependent code are here
    // iterate through animation mixers
    for (let i = 0; i < mixers.length; i++) 
    {
      mixers[i].update(delta);
    }
    delta = delta % interval;
  }
}

Now you should be able to see your flamingo flapping in a nice smooth animation!

Task 2 - Making the Robot Dance

Right, we’re not going to show you exactly how to do this bit. We’re starting to take the training wheels off. But here is what needs to happen to make the robot dance back and forth to the beat.

Remember you can always check the code in the example at the very top of this page to see a worked solution - but do have a go at working thru the process of figuring it out first

We need to take the level of our audio signal coming from a Tone.Player and apply it to the z position of our robot model.

In our init() function, we will need to do the following:

Now, in our time controlled code inside our update() function i.e inside if (delta > interval), we will need to do the following:

And now for something completely different

Task 3 - Sphere Synth

Now we’re going to try a new little project project, it’s a sphere synthesiser that plays a note and moves the sphere with the mouse. The note changes when the mouse is moved on the x axis and the volume of the synth is mapped to the y position of the mouse.

For this task, please use the starter sandbox below:

OK, once again, let’s get going by adding some global variables towards the top of our index.js file:

let geometry, material;
let planeGeometry, planeMaterial;
let planet, plane;
let mouseDown;
let raycaster, mouse, intersects;
let synth, synthNotes;
let effect, reverb;

Just below where we remove the overlay in our init() function, let’s initialise some of those variable. First we’ll start tone; then initialise our mouseDown boolean flag to false, then we’ll may a delay effect and a reverb effect.

  Tone.start(); // ensure Tone starts and that audio will be processed
  mouseDown = false; // initialise mouse down to be false
  effect = new Tone.FeedbackDelay().toDestination(); // create a delay effect and connect it to the master output
  reverb = new Tone.Reverb({
    // connect a reverb effect and connect it to the master output
    decay: 2, // decay time of 2 seconds.
    wet: 1.0, // fully wet signal
    preDelay: 0.25 // pre-delay time of 0.25 seconds
  }).toDestination();

The rest of our main audio stuff can now be added directly below. We’re going to create an array with some musical note data; then we’ll make a synth that has a bit of glide. Finally we’ll connect the synth to the effects.

 synthNotes = [
    // create an array with some choice notes in it
    "C2",
    "E2",
    "G2",
    "A2",
    "C3",
    "D3",
    "E3",
    "G3",
    "A3",
    "B3",
    "C4",
    "D4",
    "E4",
    "G4",
    "A4",
    "B4",
    "C5"
  ];

  synth = new Tone.MonoSynth().toDestination(); // create an instance of a monosynth and connect it to the master output
  synth.set({
    // set some default settings
    portamento: 0.1, // a bit of glide
    volume: -10, // reduce the level by 10dB

    oscillator: {
      // set the oscillator type to sawtooth
      type: "sawtooth"
    },

    envelope: {
      // set the envelope settings
      attack: 0.005,
      release: 2.0,
      sustain: 0.5
    }
  });

  synth.connect(effect); //connect the synth to the delay
  synth.connect(reverb); //connect the synth to the reverb

OK great, now, just below where we add the lights to our scene in init() let’s initialise the stuff we need for our mouse interactions. They are a Raycaster, the intersects array and a mouse position vector.

  raycaster = new THREE.Raycaster();
  mouse = new THREE.Vector2();
  intersects = [];

OK our visual aids in this scene are going to be quite simple. We’re going to add a ground plane and a planet that we will use as our synthesiser controller.

  planeGeometry = new THREE.PlaneGeometry(7, 7); 
  planeMaterial = new THREE.MeshPhongMaterial({
    color: 0x919191,
    side: THREE.DoubleSide
  });
  plane = new THREE.Mesh(planeGeometry, planeMaterial);

  plane.position.set(0, -0.5001, 0);
  plane.receiveShadow = true;
  plane.rotation.set(Math.PI / 2, 0, 0); // rotate around the x axis to become flat
  plane.shadowColor = 0xffffff;
  scene.add(plane);
  geometry = new THREE.SphereGeometry(1.5, 32, 32);
  material = new THREE.MeshPhongMaterial({
    color: 0xffffff,
    wireframe: false
  });
  planet = new THREE.Mesh(geometry, material);
  scene.add(planet);
  planet.castShadow = true;

Now, towards the bottom of the init() function, just above where we added our onWindowResize event listener, let’s add some new event listeners. These will listen for mouse events. We’re using the “pointer” terminology here just to avoid any possible conflicts with our OrbitControls object.

 // add our event listeners for pointer as opposed to mouse as this stops conflicts with OrbitControls
  window.addEventListener("pointermove", move, false);
  window.addEventListener("pointerdown", triggerAttack, false);
  window.addEventListener("pointerup", triggerRelease, false);

OK, you should now be seeing an error that the event handler functions don’t exit. Let’s start to define them at at the bottom of our index.js file, below the onWindowResize function.

First of all, let’s handle when the mouse point is down. Just to test, let’s use console.log statements to make sure they’re working:

function triggerAttack(event) 
{
  console.log("down"); 
}

function move(event) 
{
 
}

function triggerRelease() 
{
  console.log("up");
}

Assuming that’s working, let’s put the rest of the code in. For triggerAttack. Again, there’s a fair bit going on so please read the comments for explanations

function triggerAttack(event) {
  console.log("down");
  raycaster.setFromCamera(mouse, camera); // create our ray
  intersects = raycaster.intersectObject(planet); // test whether our ray is intersecting with our planet object

  if (intersects.length > 0) {
    //if there is something in our array
    mouseDown = true; // set mouseDown boolean flag to true
  }

  
  const note =
    synthNotes[
      Math.round((event.clientX / sceneWidth) * (synthNotes.length - 1)) // constrain our mouseX position using Math.round to create an integer index that can be used to pick a note from our note array
    ];

  if (mouseDown) {
    // Make the sphere follow the mouse
    let vector = new THREE.Vector3(mouse.x, mouse.y, 0.5); // create a new 3D vector using our mouse position
    vector.unproject(camera); // project mouse vector into world space using camera's normalised device coordinate space
    let dir = vector.sub(camera.position).normalize(); // Create a direction vector based on subtracting our camera's position from our mouse position
    let distance = -camera.position.z / dir.z; // derive distance from the negative z position of the camera divided by our direction's z position
    let pos = camera.position.clone().add(dir.multiplyScalar(distance)); //create a new position based on adding the direction vector scaled by the distance vector, to the camera's position vector
    planet.position.copy(pos); // copy our new position into the planet's position vector
    synth.triggerAttack(note); // trigger the envelope on the synthesiser
    planet.material.color.setHex(0xff00ff); // change the colour of our planet to purpley pink
  }
}

Our move function is pretty similar to the triggerAttack one.

The main difference now being that instead of triggering the envelope on the synth, we’re updating the note and the volume based on the mouse position

We do this ONLY if the mouse is down, since we don’t want to hear sound when the mouse is moving around before clicking.

function move(event) {
  mouse.x = (event.clientX / sceneWidth) * 2 - 1; // convert our mouse x position to be a value between -1.0 and 1.0
  mouse.y = -(event.clientY / sceneHeight) * 2 + 1; // convert our mouse y position to be a value between -1.0 and 1.0
  const note =
    synthNotes[
      Math.round((event.clientX / sceneWidth) * (synthNotes.length - 1))
    ]; // constrain our mouseX position using Math.round to create an integer index that can be used to pick a note from our note array

  if (mouseDown) {
    // Make the sphere follow the mouse
    let vector = new THREE.Vector3(mouse.x, mouse.y, 0.5); // create a new 3D vector using our mouse position
    vector.unproject(camera); // project mouse vector into world space using camera's normalised device coordinate space
    let dir = vector.sub(camera.position).normalize(); // Create a direction vector based on subtracting our camera's position from our mouse position
    let distance = -camera.position.z / dir.z; // derive distance from the negative z position of the camera divided by our direction's z position
    let pos = camera.position.clone().add(dir.multiplyScalar(distance)); //create a new position based on adding the direction vector scaled by the distance vector, to the camera's position vector
    planet.position.copy(pos); // copy our new position into the planet's position vector
    synth.setNote(note); // update our synthesiser's note
    let volume = THREE.MathUtils.mapLinear(
      // map the y position to volume of the synth. Output range is in decibels
      mouse.y, //input value
      -1.0, // lower input range
      1.0, // upper input range
      -60, // lower output range
      -3 // upper output range
    );
    synth.volume.linearRampTo(volume, 0.01); // set new volume level with a ramp of 0.01 seconds
  }
}

Our final function now simply releases the envelope and returns the planet to the default grey colour.

function triggerRelease() {
  mouseDown = false; // set mouseDown flag to false
  console.log("up");
  synth.triggerRelease(Tone.now()); // trigger the release phase of our synth's envelope
  planet.material.color.setHex(0x919191); // return planet's colour to it grey
}

Stretch goal time!

OK so hopefully now you have made a couple of little fun projects this week! Here are some stretch goals:

That’s it for this week

Nice one for making both little projects! 😀

Back up your work

Super important task: go to file->export to .zip in your codesandbox and download your project as a zip.

Remember, files in the cloud are just files on someone elses computer. It is critical to keep a good backup.