Everything that goes around the internet is about data transfer, from one point to another point. Either using client-server setting, or peer to peer setting, you're going to throw out some data into a certain computer on earth in order to communicate.

Data transfer is essential to communication. And because it's a form of communication, both sides needs to understand and follow an agreement or guideline. Going without agreement is just the same as trying to converse with Japanese by using Bahasa Indonesia. Nothing gonna get into your head, and neither does your peer. So this is why such an agreement is required.

Communicating through Internet

In today's ~~episode~~ chance, I'm going to talk about communication in the internet. How things just works, and something like that.

You can read lots of resources about how computer communicate each other by using the keyword "OSI Layer" on Google, but I'm just going to pinpoint the two major thing to be discussed here: Transport layer, and TCP and UDP.

Both TCP and UDP are protocols that works in Transport layer, defining protocols/guideline on how data are "transported", which roughly points to the 'Transport layer' term. One big difference between them is the existence of persistent connection, where in TCP it exist, but not for UDP.

But really, if we break down how things works, the 'connection' here is basically talking about how the other side you're communicating with actually 'remembers' you. Ignoring that, both are just using the same concept of sending 'packets' through the wild world of internet, until it reaches the other end.

Packet here is defined as the thing that get sent. For each protocols, there is a designated structure for the packets, and any system that using said protocol should follow the convention. In most cases, packet consist of:

What kind of packet it is
How long the content is
Something to check how legit it is, and
The actual content

You can go wild on how you define the structure of your packets, but make sure your peer also understand what you're trying to say. (I'm trying...)

Attempt to Communicate

Back in the old days where C language is just about everywhere, no Java, no Python, no Javascript, the program for this is written in, well, C. Lots of frustration as you need to make sure you're not going for a segmentation fault. Today, everything is sunshine and rainbow. You can use Javascript, or Python, or any easier-to-understand-by-millenials programming language.

In this chance, I'm going to use Python. Particularly, Python 3.8.5

Defining Packet Structure

Your packet needs to be in the form of byte streams. All because when you define it as a string, you will have bad time when there is null bytes coming in. Either you go for byte streams, or stay with the regular/f-string/b-string/whatever-string, but make sure you handle the data well.

This kind of data is best defined by referencing how C language defines it's data, where in most cases are defined by using the keyword 'struct'. And to achieve such a thing in Python, we're going to use a native library with the same name.

import struct

What this library does is basically converting any data you're feeding it into your preferred format. The format here is about what type of data (integer, char, char[], float, long int, etc...) and how long it is. You can just explore the documentation regarding the formats.

In most cases, the function you're going to use the most is the 'struct.pack()', as it does what it says.

I like to define my data structure by using classes, and here I basically made a class, called Packet. This class essentially going to build the packet by specifying the info for each of the arguments.

I also made class 'PacketType' which just going to act as an enum. Python please add enum, onegai.

import struct


class PacketType():
    DATA = b"\x00"
    ACK = b"\x01"
    FIN = b"\x02"
    FINACK = b"\x03"


class Packet():
    '''Packet Builder
        4 arguments
        - type which needs to be using PacketType for shorthand
        - length : 1-byte integer
        - seq_num : 2-bytes integer
        - data : <length>-bytes array of bytes


        Convert data into byte string or raw array of bytes first
    '''
    def __init__(self, type, length, seq_num, data):
        self.set_type(type)
        self.data_length = length
        self.set_data(data)        
        self.set_length(length)
        self.set_seq_num(seq_num)
        self.set_checksum()
        self.buffer = self.build()


    def set_type(self, _type):
        self.type = struct.pack('c', _type)


    def set_length(self, length: int):
        self.length = struct.pack('c', length.to_bytes(1, byteorder="little"))


    def set_seq_num(self, seq_num: int):
        self.seq_num = struct.pack('2s', seq_num.to_bytes(2, byteorder="big"))


    def set_checksum(self):
        self.checksum = struct.pack('2s', b"\x17\x70")


    def set_data(self, data):
        if (len(data) > self.data_length):
            print('Data segmentation is too big')
        else:
            self.data = struct.pack('{}s'.format(self.data_length), data)
    
    def calc_checksum(self, data):
        return b"\x17\x70\x13"


    def build(self):
        return struct.pack('cc2s2s{}s'.format(self.data_length), self.type, self.length, self.seq_num, self.checksum, self.data)

Things are getting pretty complicated, but bear with me. What I did is just defining several setters that calls 'struct.pack' method, and specify the format and the data. For example, the set_type method here

  def set_type(self, _type):
      self.type = struct.pack('c', _type)

What it does is converting the data on _type into a 1-byte char, which basically 8 bit data. 'c' corresponds to 'char' data type in C, and you can just check the rest in the documentation here

After defining every single things you need for your packet, it's time to build your packet. It's not that hard, just call the same method used earlier again. Here I'm packing it by specifying several data in the format string (which also handles variable-sized data length), and just add the data I'm going to pack as the arguments. Make sure to write the data in order.

    def build(self):            return struct.pack('cc2s2s{}s'.format(self.data_length), self.type, self.length, self.seq_num, self.checksum, self.data)

With this, your packet builder is pretty much done. For the receiving side, you're going to define a packet unwrapper, but it's basically just reversing the process of the packet builder. There is 'struct.unpack' method, as a hint, teehee~

Handling The File Processing

Now, we're defined how our packets gonna transport the data, but we haven't talked about the actual data, yet. In most cases, the data we're talking is a file that exist in the storage. It needs to be read into the memory first before you can actually start using it.

But what if the data is big? Will it cause lags? Stalling?

I'm talking about transferring 2GB files here, which I'm sure not all of you want to let your Python app suddenly use 2GB of your RAM. Heck, loading 2GB file means the needs to read the entire file. As any form of file I/O is relatively very slow (either you're on SSD or HDD), this is not a reliable choice.

So, what to do about this? Any way to prevent the possible lags?

Well, one of the way I can think of (based from several research), is to implement a what-so-called "file piece manager". What it does is just to manage the pieces or parts of the file that's going to be loaded back and forth between the storage and memory.

We're going to go with the simplest one. We're going to think that it will act like a queue, where each items in the queue is the data we retrieved from the packet. A prerequisite on this one is each data that gets into the queue needs to be verified first (through checksum, etc...), and still in order. Handling this will be discussed later.

As I like to define a component as a class, this is how I prototyped the class FileQueue:

import os


class FileQueue():
    def verify_piece(self):
        pass


    def add_piece_to_complete_table(self):
        pass


    def write_file_to_disk(self):
        pass

Importing 'os' here is to access the file I/O capabilities of Python, and it's also native. There could be another native libraries you can use, so this is just a slight hint.

You can go with basically whatever the method you need. The basic idea I have is :

Verify a piece that retrieved : Is it in order? Is it damaged?
If verified, then put in into queue.
Write the head. Then pop.
Repeat

This way, we're going to access the storage for every packet we retrieved. Then write into it.

But this is waaay to slow! File I/O requires bus communication between CPU and the controller, and it's.... slow. Imagine if we have 100000 packets, that means 100000 I/O access. This is where we introduce "buffering".

The idea is simple. We wait for n amount of packets/pieces to fill the queue, then write them all, and then flush the write buffer. Just that. This will make our previous flow be like this:

Verify a piece that retrieved : Is it in order? Is it damaged?
If verified, then put in into queue.
If there is a certain amount of items already in queue, write the them all to the disk.
Flush the items we just wrote
Repeat

Increasing the n value corresponds to the division factor of the total I/O access we need. With n = 2, I/O access is total_packets/2, and so on.

By using this queue system, you can handle a reliable file processing from one source. But if you want to go for multithreading, multiple sources, and other multi-mumbo-jumbo stuff, one alternative is to implement a table that maps each incoming packet into their respective slots. This is pretty complicated, and left for as an exercise for readers.

The Actual Protocol

It's time to talk about how computers actually talk.

I've talked on how it sends a packet to other end, and that's it. By default, it's only about sending packet. But for the actual communication to work, we need a flow control.

Lets say, that Ani is sending a packet to Bram. If there is nothing wrong, then when Bram receives a packet from Ani, Bram sends an acknowledgement packet to Ani (dubbed ACK), that the packet is received and verified. After that, Ani sends the next packet, and so on...

What if there is error? Out of order packets? The actual packet doesn't arrive? The ACK packet doesn't arrive?

This is where we introduce a time window. A time window is basically how long it is for a single packet to finish the entire communication process. This involves the sending from A to B, B verifies, then B sends ACK to A. Failing any of the step result the next step won't happen. And when the sender doesn't get the ACK from B when timeout happens, then it classified as "there is something wrong".

So, by theory, as A, we're going to send a packet to B, then wait. If there is an ACK packet, then the packet is successfully sent. If by the end of the timeout, still no ACK packet, then A is going to retransmit the same packet, until ACK packet is retrieved.

This called "Stop-and-Wait" flow control.

What about the receiver? The error possible by the receiver is out-of-order packet, which only happen if you're going to transmit more than one packet at a time. This is a scenario of using sliding window protocol, which can be also left as an exercise for the readers.

Another error by the receiver is duplicate packets, which happened if the ACK packet doesn't arrive to sender, which triggers the sender to send the same packet the receiver received earlier. Handing this is as simple as discarding the incoming packet, and send the ACK packet back.

So, we're going for the simple Stop-and-Wait protocol. It'll just basically goes like this.

A data is packaged, and sent. Timeout for that packet is set. Packet counter set to current packet number.
While timeout is not reached and packet is not acknowledged
Wait for any incoming packets
If there is any incoming packets, check the packet type, if it's ACK, then break the while loop by setting the packet status to 'acknowledged'
When timeout is reached and packet is still not acknowledged, go back to step 1.
When packet is acknowledged (and timeout is possibly not reached), then increment the packet number. Go back to step 1.
Repeat this until all the file parts / packages are sent, and we received something like "finisher" (FIN-ACK) packet.

The receiver side is not that hard. Most of the time the receiver will just sit there and wait. I think at this point it's pretty straightforward, eh?

Your only job is to handle the data flow between the components of your program, handling the sockets, and another auxiliary stuffs. The reference on using UDP to communicate, using 'socket' library in Python could be read here. I could probably write my own implementation for this case, but on another chance.

Wrapping It Up

I think the communication process is not that hard when we breakdown every single process or components that are involved. Implementing the most basic Stop-and-Wait protocol is one of the simplest way to establish a reliable file transfer, and you should go for that.

Improving the existing system, you could go and try to implement sliding window protocol. FYI, Stop-and-Wait protocol is basically sliding window protocol with window size of one packet. There are various resources on the internet you can check.

Lastly, the examples shown here have room for improvement. This can act as the basic guideline if you want to try to communicate using UDP in Python.

I'm open for questions or comments about this article, so make sure to hit me on DM (in LINE) in case you have any questions and comments.

I'm just too lazy to implement comment section here.

Thanks for reading!

Adios.

Talking about a reliable file transfer using UDP