Building a Downloader in Go

People say Go is well suited for Network applications. With its light Goroutines and huge standard library it can get most of the job done without even needing a third-party library.

Great, let me build a terminal program to download files

Accio

harry-potter-casting-spell-accio-dittany

The summoning charm from Harry Potter seems like a fit name for my downloader. Like when Harry Potter says “Accio Dittany” and out pops the bottle of Dittany potion from Hermione’s bag with an “Undectable Extension Charm”, our version of Accio would do something similar, albeit with URLs.

So running accio url would get you the file from the url.

Simple enough! Let’s start then.

The process

The module net/http gives us this amazing method for making a GET request -

http.Get(url string) (resp *httpResponse, err error)

With this, we can run a GET request to url and we get a response object (of type httpResponse) and a possible error (of type error). We check for any error that might have occured while making the request and then move on to reading the data bytes from httpResponse.Body.

But before reading the body, we would need to create and open a file for writing with the following method. And os module has the capability we need

os.OpenFile(name string, flag int, perm fs.FileMode) (*os.File, error)

we also need os to get the CLI argument for URL being passed into our program by the user while running command accio URL. We need io module because it has defined the io.EOF error value that we need to test to know when we’ve reached the end of reading the response body. With the file opened and the response body ready for reading, we would simply need to loop this following

Copy Data process

Initialize a buffer
Read n bytes from response body into a buffer
If n is non-zero we write n bytes to the file
If error occured is EOF (End of file - indicates response data end)

We break the loop (go to step 6)
Go to step 2 (repeat)
We close the file and the response body

Here’s the actual code

response, err := http.Get(downloadUrl)
// handle error
defer response.Body.Close()

file, err := os.OpenFile(
    "download",
    os.O_WRONLY | os.O_CREATE,
    os.ModePerm,
)
// handle error
defer file.Close()

buffer := make([]byte, 512)
for {
    buffer = buffer[0:512]
    n, err := response.Body.Read(buffer)

    if n > 0 {
        file.Write(buffer[0:n])
    }

    if err != nil {
        if err != io.EOF {
            fmt.Fprintf(
                os.Stderr,
                "download error: %s\n",
                err,
            )
            os.Exit(-1)
        }
        break
    }
}

We open the file with flags as os.O_WRONLY | os.O_CREATE which causes it to open a file for writing and the perm as os.ModePerm so that the opened file gets the same permissions as our executable program accio

We reset the buffer slice’s capacity to 512 each time in the loop to utilize the full capacity and let the Read method of response.Body to fill it up with n bytes.

We then proceed to write only n bytes to the file because that’s how much we read in this iteration from response.Body

If error has occured, we need to stop everything. If the error is not EOF we also log this onto the terminal. Otherwise, we silently break the loop

Showing progress

The next part is to show progress of the download. For this I’m going to use the channel feature of golang.

Basically the download (more specifically the copy of bytes from response body to file) would be happening on another go routine and it will send progress stats over a channel. The main routine can then read messages from this channel to display progress.

Here’s the structure I used for the download progress status message

type DownloadStatus struct {
	Error           error
	IsComplete      bool
	BytesDownloaded int64
}

This will suffice. The copying logic will post a message of type DownloadStatus on the statusChannel channel so that the main routine can display the progress

here’s the updated function to copy data from src to dest while posting updates of the progress

func copyVerbose(dest io.Writer,
                 src io.Reader,
                 statusChannel chan DownloadStatus) {
	buf := make([]byte, COPY_BUFFER_SIZE)
	status := DownloadStatus{}

	for {
		nread, err := src.Read(buf)
		if nread > 0 {
			nwritten, err := dest.Write(buf[0:nread])
			if err != nil {
				status.Error = err
				break
			}
			status.BytesDownloaded += int64(nwritten)
		}
		if err != nil {
			if err == io.EOF {
				status.IsComplete = true
			} else {
				status.Error = err
			}
		}
		statusChannel <- status
	}
}

Showing progress in

Now to show progress on the main routine, we need to do this

create a ticker (which ticks periodically at say 500 ms)
use select to read from ticker and the statusChannel
if read from statusChannel, store it in a local variable
if read from ticker channel, show the progress
go to step 2 and repeat

of course, we would close this repeat loop once we find the status message contains IsComplete as true

here’s the code for that (partial main function)

lastStatus := DownloadStatus{}
ticker:= time.NewTicker(time.Millisecond * 500)
go downloadUrl(url, &DownloadOptions{Filepath: filename}, statusChannel)

done := false
for done != true {
    select {
    case <-ticker.C:
        n, unit := getFormattedSize(lastStatus.BytesDownloaded)
        if lastStatus.IsComplete == true {
            fmt.Printf("\x1B[1K\rcompleted: %.2f %s\n", n, unit)
            done = true
            break
        }
        if lastStatus.Error != nil {
            fmt.Fprintf(
                os.Stderr,
                "download failed: %s\n",
                lastStatus.Error)
            done = true
            break
        }
        fmt.Printf("\x1B[1K\r%.2f %s", n, unit)
    case status := <-statusChannel:
        lastStatus = status
    }
}

The escape sequence in printf (\x1B[1K) deletes the current line and \r moves the cursor to the beginning of the line. See more

This effectively shows us the updated progress by deleting the old progress every 500 ms

Conclusion

We’ve got ourselves a http(s) downloader using Golang’s standard libraries only. The downloader shows progress as it downloads and saves the file with name as found looking at the url (or download if the url doesn’t contain a /resource_name.extension part at the end

There are many more things that can be done here. Some of the basic ones are

Add multi connections to speed up downloads
Add pause / resume capability

Both of them requires the usage of response header Accept-Range which denotes whether the server supports requesting file data by a specific range. The corresponding request header is Range which can be added in the request to let the server know which range of data we need

I’ll pick this up in the next iteration of Accio our very own http downloader.

Full code

https://github.com/riturajborpujari/accio

Accio#

The process#

Copy Data process#

Showing progress#

Showing progress in#

Conclusion#

Full code#

References#