Banner image source from Hackernoon. All rights reserved.

I finally made my first step in learning Golang.

I once touched this before but only as a part to solve a challenge. The challenge itself wasn't hard. Just to do a looping without using the "for" keyword, which can easily be tackled using recursion.

But I believe there is more than this. I've wanted to learn Golang ever since I saw a repository in which you can put a "middleman" between your Arknights game client and the server. It was written in Golang and it hooked me ever since. Yeah, also because I'm interested in writing in another language other than JavaScript.

Baby Steps : Golang Basics

The basics are pretty simple. It's somehow resembles C in a weird flavor since there are pointers and reference. It requires importing packages, and third party packages can also be imported from public repositories. I'm pretty curious as there is no such thing as "node_modules" folder in my working directory. It says that you can manage your modules by using Go Modules, but it will be on another day.

So far, what I learned is that main application should have "package main" as the starting line of the file. It simply marks that this is the main package that contains the main function. Here is a simple "Hello, World!" application made with Go.

package main

import "fmt"

func main() {
  fmt.Println("Hello, World!")
}

Run this file using "go run [filename].go", and you will have this file being run then exit. It looks like it was interpreted, but on the background it was compiled and then executed. The compiled binary is located in the "temp" directory (if you're in Windows).

Surely, this is not interesting at all. I'm not learning Go only to make a lame "Hello, World!" application. Let's make something better at least.

Something Better: Web Scraper

This already sounds fancier, but it's actually pretty simple with the usage of existing third-party libraries. In here I'm going to use two libraries: one for doing the web scraping and one for doing the fuzzy searching. I want to make a library that can tell me the currently airing seasonal anime from MyAnimeList.net, and a RSS scraper for giving me the torrent links from nyaa.si

Using third-party library in Go require you to import the package using the import keyword, or adding the package name inside the import construct. Below example gives the same result

import "external-package-here"

import (
  "external-package-here"
)

The latter option is preferred when you use multiple packages, which is already something unavoidable.

Here, I'm going to use these packages for my imports. fmt for text formatting and printing, os for accessing OS related stuffs like command line arguments, strings for string manipulation. The latter will be my external library usage.

import (
    "fmt"
    "os"
    "strings"

    "github.com/gocolly/colly" // Web Scraper
    "github.com/sahilm/fuzzy" // Fuzzy searching
)

When you run a Go application with external libraries, the library will be automatically fetched before compilation. If you're using Visual Studio Code and relying on Intellisense or the language server, then it's preferred to fetch the library beforehand. You can do that using "go get [library name here]" command. As an example, the command for fetching the library for colly above will be

go get "github.com/gocolly/colly"

The libraries fetched will be stored outside your project folder, in separate central repository, so there won't be anything like "go_modules" folder after you do this. You can manage library versions without affecting other projects by specifying the version number inside go.mod file, which works like package.json file if you're a hardcore Node.js developer like me. This article covers this issue in a straightforward and clear sense on why you don't need something like "go_modules" or virtual environment like in Python.

Now let's go for the scraper part.

gocolly : A Web Scraping Library

A quick search in Google showed this library as one of the top result, and has 12.9k stars on GitHub, so you can trust this library. You can visit the repository here.

The usage is simple. Really. You start by declaring the scraper by calling NewCollector() function from the package. This "collector" acts as the crawler and the place where you do all the web scraping stuff.

c := colly.NewCollector()

The way it work is through event handlers. There are many event handlers for each type of events which you can read in the docs, but here I'll be using 4 event handlers: OnRequest, OnHTML, OnXML, OnError.

OnRequest will have a function handler that will be fired for every request made. Here I'm using it to make a simple logger to tell me that the request has been made.

(Apparently something breaks down with the code wrapper when I wrote this code, so I'll continue with images from this part)

OnHTML will have a CSS selector string and a function handler, which will be fired for every instance that matches the CSS selector string. Here I'll just print the text for the matching element.

OnXML will have an XPath string and a function handler, which works the same way with OnHTML. Put attention on the difference, being this using XPath string and the other using CSS selector string. Here I'll get the content for every child it has and adding it to a container for fuzzy searching later on (will be explained later).

OnError will handle any kind of HTTP error, so use this for error handling, logging, and many other stuffs you want with error events.All left to do is to scrape the page by visiting the link. It's a simple as calling the Visit() function with the link as the argument.

c.Visit("https://myanimelist.net/anime/season") // for MAL seasonal anime scraping
c.Visit("https://nyaa.si/?page=rss") // for nyaa.si RSS feed scraping

Now, it's time to work with the data we've fetched. Here I'll focus on the nyaa.si RSS feed data, to retrieve the torrent link based from anime title inputted by user.

Fuzzy searching

Fuzzy string searching is a type of searching using the approximate match of the string, in contrast with exact matching. In exact matching, every single character must be in the same order and has the same characters, living the name "exact matching". This is not very helpful when we don't exactly know, or only know the most but not all part of the word or text.

That's why fuzzy string searching comes in. It searches using the closest string with the query given, so any errors will be tolerated. It's approximate anyway. Most of the time, the search will use Levenshtein Distance, which measures the difference between two strings. But in fuzzy search, many other additional algorithms can be used, like searching with acronym, number-letter exchange, and many others.

I'm using this package with consideration it has the highest star on GitHub after comparing with several other similar packages. I think this will do the job I needed. You can visit the repository here

Regular fuzzy string searching can be done in straightforward manner when you have an array or slice of string. Searching for the closest match only need you to use Find() function with the query string as the first argument and the array or slice for the second argument. As I make my items with a struct defined to it, then it needs another approach.

I defined the struct with the name NyaaItem which contains fields I need. Declaring struct in Go is very similar with declaring struct in C, albeit not the same.

After that, I'm declaring a type which is just a wrapper for array or slices for NyaaItem struct I made before. Then in this new type, I'm adding member functions or method functions that's exclusive on this type only. This is the most basic form of Object Oriented Programming in Golang, although Golang doesn't have support for OOP. Well, it's a close one. These functions are required for the fuzzy library to search for the items according to one of the attribute in the struct. Here I want to search using the title of the item.

And now, it's all good. You can use this type to declare a variable as a container for the items, and use this variable to append data, and to do the search.

With fuzzy, the search using a struct array or slice will going to use FindFrom() function, with signature similar with FindFrom() function.

After this, it's pretty much done, and I can go test it by building it first using "go build [filename].go" command, and execute the compiled binary.

As you can see, I'm searching with "haraku sabou" and the fuzzy search give me my desired results, being entries with "Hataraku Saibou" as the title.

Loading...

There will be awesome contents here

Log-0x15 - Baby Steps in Golang