Picking something should be easy, right? We can just click the item, then get feedback from WebGL, right?

No.

WebGL is just a rasterizer, which means it just outputs something to the screen, and that's it.

We need to store a data somewhere, and then access that data. And here is texture can come in handy.

I'm not going to dive deep into explaining what is texture and why it matters, but just think of it as our data container.

Picking Drawings

There are several approach to pick something in OpenGL or WebGL, and for this case, the easiest one to implement (at least for me) is the picking method. It uses the readPixels function in WebGL, where we can... well... read several pixel data from the screen. The data format which returned can be specified, where in this case we're going to use color data. This trick is pretty clever for me, so bear with me.

But before that, if you notice, our triangleData got vertex values that are simply small. It ranges from -1 to 1, as to say it uses the clip space format. To prevent more confusion, we need to convert this into pixel space format, a format where we all are familiar with.

Converting Clip Space to Pixel Space

In main.ts, somewhere before GLObject instantiation and after gl.linkProgram call, put this stuff.

// above is gl.linkProgram
    gl.viewport(0,0, gl.canvas.width, gl.canvas.height)
    const u_resolution = gl.getUniformLocation(shaderProgram, 'u_resolution')
    gl.uniform2f(u_resolution, gl.canvas.width, gl.canvas.height)
// below is GLObject instantiation

In draw-vert.glsl, add a new uniform and turn the shader until it looks like this.

attribute vec2 a_pos;
uniform mat3 u_proj_mat;
uniform vec2 u_resolution;


void main() {
    vec2 position = (u_proj_mat * vec3(a_pos, 1)).xy;
    vec2 zeroToOne = position / u_resolution;
    vec2 zeroToTwo = zeroToOne * 2.0;
    vec2 clipSpace = zeroToTwo - 1.0;
    gl_Position = vec4(clipSpace, 0, 1);
}

Now, after this, the object will disappear. This is because the position of the vertices in triangleData is still in clip space, with values between -1 and 1. Change this into something more realistic, such as

var triangleData = [
    400, 400.0,
    400.0, 200.0,
    200.0, 400.0
]

This will give a pretty good looking red triangle we've always loved.

To show both triangles, just set the position of glObject2 into something big, like (600,400).

glObject2.setPosition(600, 400)

There is an issue regarding the positioning, but it's not really an issue. It has something to do with world coordinates and maths, and we will address that problem later on.

Now, we can start to go with the next step.

Defining our 'picking' shader

Handling the picking mechanism is done through storing the metadata into color channel of each pixel drawn. The logic is as simple as we're going to color each GLObject in different color according to the ID.

Because it deals with coloring stuff, then we're going to work with the fragment shader. Let's define a new file called select-frag.glsl. Here I've defined select-vert.glsl, but it's pretty much just the same with draw-vert.glsl, so you can just skip the vertex shader part.

// select-vert.glsl
attribute vec2 a_pos;
uniform mat3 u_proj_mat;
uniform vec2 u_resolution;


void main() {
    vec2 position = (u_proj_mat * vec3(a_pos, 1)).xy;
    vec2 zeroToOne = position / u_resolution;
    vec2 zeroToTwo = zeroToOne * 2.0;
    vec2 clipSpace = zeroToTwo - 1.0;
    gl_Position = vec4(clipSpace, 0, 1);
}

// select-frag.glsl
precision mediump float;
uniform vec4 u_fragColor;
void main() {
    gl_FragColor = u_fragColor;
}

It's just... the same... Well, not for too long.

Because we're defining another shader, then we need to instantiate this shader also. Just copy the lines to define the shader, and change several parameters.

// after shaderProgram linkProgram

    var selectVert = await fetchShader('select-vert.glsl')
    var selectFrag = await fetchShader('select-frag.glsl')
    
    var selectVertShader = gl.createShader(gl.VERTEX_SHADER)
    gl.shaderSource(selectVertShader, selectVert)
    gl.compileShader(selectVertShader)
    if (!gl.getShaderParameter(selectVertShader, gl.COMPILE_STATUS)) {
        alert('Error when compiling shaders: ' + gl.getShaderInfoLog(selectVertShader))
    }
    var selectFragShader = gl.createShader(gl.FRAGMENT_SHADER)
    gl.shaderSource(selectFragShader, selectFrag)
    gl.compileShader(selectFragShader)
    if (!gl.getShaderParameter(selectFragShader, gl.COMPILE_STATUS)) {
        alert('Error when compiling shaders: ' + gl.getShaderInfoLog(selectFragShader))
    }
    var selectProgram = gl.createProgram()
    gl.attachShader(selectProgram, selectVertShader)
    gl.attachShader(selectProgram, selectFragShader)
    gl.linkProgram(selectProgram)

// before gl.useProgram

Next, define a new method inside GLObject class, call it drawSelect. This method will be invoked for rendering the texture for storing the ID.

// inside GLObject class

    drawSelect(selectProgram: WebGLProgram) {
        const gl = this.gl
        const id = this.id
        gl.useProgram(selectProgram)
        var vertexPos = gl.getAttribLocation(this.shader, 'a_pos')
        var uniformCol = gl.getUniformLocation(this.shader, 'u_fragColor')
        var uniformPos = gl.getUniformLocation(this.shader, 'u_proj_mat')
        gl.uniformMatrix3fv(uniformPos, false, this.projectionMat)
        gl.vertexAttribPointer(
            vertexPos,
            2, // it's 2 dimensional
            gl.FLOAT,
            false,
            0,
            0
        )
        gl.enableVertexAttribArray(vertexPos)
        const uniformId = [
            ((id >> 0) & 0xFF) / 0xFF,
            ((id >> 8) & 0xFF) / 0xFF,
            ((id >> 16) & 0xFF) / 0xFF,
            ((id >> 24) & 0xFF) / 0xFF,
        ]
        gl.uniform4fv(uniformCol, uniformId)
        gl.drawArrays(gl.TRIANGLES, 0, this.va.length/2)
    }

// ...

Inside Renderer class, add new method renderTex, which just the same as render method, but it invokes drawSelect for each objects. It should be crystal clear.

    renderTex(selectProgram: WebGLProgram) {
        for (const obj of this.objectList) {
            obj.drawSelect(selectProgram)
        }
    }

Before we can start using this, we need to capture the mouse movement as to detect the exact position of our cursor.

Adding Event Listener

Our canvas is an HTML element, which means we can add various event listener for interactivity. Here I'll add an event listener to respond to "mousemove" event, which means every single time (or frame) our mouse moved inside the canvas, a callback function will be fired.

We'll define a callback function that process the mouse event into (x, y) coordinate of our cursor inside the canvas. We'll save this information inside a global state variable, let's call this variable appState.

// outside of the main(), on the top of main.ts
let appState = {
    mousePos : {
        x: 0,
        y: 0
    }
}

// inside main(), somewhere...
    canvas.addEventListener('mousemove', (event) => {
        const bound = canvas.getBoundingClientRect()
        const res = {
            x: event.clientX - bound.left,
            y: event.clientY - bound.top
        }
        appState.mousePos = res
    }, false)

Our event listener is done, now let's put everything in.

Render Loop

Our current application will just render everything once, and then doing nothing again after that. This is because we're only calling the render function once, which will make our picking useless because it will not respond to any changes unless explicitly told.

We need to define a render loop, and it's pretty easy. Just make a function that takes a callback function as parameter, and use the one and only requestAnimationFrame function.

// just after GLObject addition call of Renderer
    function render(now: number) {
        renderer.render()
        requestAnimationFrame(render)
    }
    requestAnimationFrame(render)

This will call the stuff recursively, creating an infinite loop of calling that will still able to respond to events.

We'll need to change and make more stuffs.

// defining texture buffer    
    const texBuf = gl.createTexture()
    gl.bindTexture(gl.TEXTURE_2D, texBuf)
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR)
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE)

// defining depth buffer
    const depBuf = gl.createRenderbuffer()
    gl.bindRenderbuffer(gl.RENDERBUFFER, depBuf)
    function setFrameBufferAttatchmentSizes(width: number, height: number) {
        gl.bindTexture(gl.TEXTURE_2D, texBuf)
        gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, width, height, 0, gl.RGBA, gl.UNSIGNED_BYTE, null)
        gl.bindRenderbuffer(gl.RENDERBUFFER, depBuf)
        gl.renderbufferStorage(gl.RENDERBUFFER, gl.DEPTH_COMPONENT16, width, height)
    }
    setFrameBufferAttatchmentSizes(gl.canvas.width, gl.canvas.height)

// defining frame buffer
    const frameBuf = gl.createFramebuffer()
    gl.bindFramebuffer(gl.FRAMEBUFFER, frameBuf)
    const attachmentPoint = gl.COLOR_ATTACHMENT0
    const lvl = 0

// using the texture and depth buffer with frame buffer
    gl.framebufferTexture2D(gl.FRAMEBUFFER, attachmentPoint, gl.TEXTURE_2D, texBuf, lvl)
    gl.framebufferRenderbuffer(gl.FRAMEBUFFER, gl.DEPTH_ATTACHMENT, gl.RENDERBUFFER, depBuf)

Inside the render loop, turn it to this

// ze render loop function
    function render(now: number) {
        gl.clearColor(1,1,1,1)
        gl.clear(gl.COLOR_BUFFER_BIT)
        gl.viewport(0,0, gl.canvas.width, gl.canvas.height)
        // drawing texture
        const frameBuffer = frameBuf
        gl.bindFramebuffer(gl.FRAMEBUFFER, frameBuffer)
        gl.enable(gl.DEPTH_TEST)
        gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT)
        gl.useProgram(selectProgram)
        const resolutionPos = gl.getUniformLocation(selectProgram, 'u_resolution')
        gl.uniform2f(resolutionPos, gl.canvas.width, gl.canvas.height)
        renderer.renderTex(selectProgram)
        // getting the pixel value
        const pixelX = appState.mousePos.x * gl.canvas.width / canvas.clientWidth
        const pixelY = gl.canvas.height - appState.mousePos.y * gl.canvas.height / canvas.clientHeight - 1
        const data = new Uint8Array(4)
        gl.readPixels(pixelX, pixelY, 1,1, gl.RGBA, gl.UNSIGNED_BYTE, data)
        const id = data[0] + (data[1] << 8) + (data[2] << 16) + (data[3] << 24)
        console.log(id)
        gl.bindFramebuffer(gl.FRAMEBUFFER, null)
        // draw the actual objects
        gl.useProgram(shaderProgram)
        renderer.render()
        requestAnimationFrame(render)
    }
    requestAnimationFrame(render)

Here, I'm logging the id of GLObject where the mouse lies.

And it just works! Magical!

And with this, you can add more events and handle it. You can create click event so that the color changes when it got clicked, or even to glow. It's all up to your imagination.

Next, we'll start into creating stuffs. Up until now, the triangles are hard-coded. As we're thriving to make a CAD application, it should support features to create and manipulate existing objects.

Until next time, see ya!

AwooCAD - Selecting Items