Building a Gopher Server: An Advenure

Published: November 30th, 2021
Updated: April 21th, 2022

I’ve always had a bit of an interest in older technology, from growing up watching tapes and VCDs during the peak years of DVDs, learning BASIC and FORTRAN during my school days, spending most of my adolescence on IRC, to going straight from playing the Super Mario Brothers Trilogy on NES to playing Halo on the original Xbox, I’ve definitely found enjoyment in the older school.

That’s why, when in need of a portfolio piece, I decided on the idea of creating a Gopher server from scratch. For those that don’t know, Gopher, named after the Minnesota state animal, is a pre-HTTP specification for a stateless distributed document search and retrival, originally meant for use at universities for low overhead sharing of information without needing to build a centralized store.

Research Phase

To start off with, I have to, of course, research the Gopher Protocol. Sure I had already read the RFC, but that was nearly 3 decades old, it could be stale or obsolete with further RFCs or such replacing it. Not that I expect much since the dominace of HTTP.

First thing I found was the Wikipedia article on Gophers, then after clicking through the disambiguation, the Wikipedia article on Gopher. It’s actually fairly extensive, detailing the technical details about the protocol and even lists some Client and Server software. Given how recently some of them were updated, it seems I’m not the only one interested in this old protocol. There is also a section about something called “Gopher+” but it states it was never widely adopted so I’ll disregard that for now.

Next thing I found was RFC4266, a standard for a Gopher URI scheme. While interesting, it seems like it is for Gopher clients to implement rather than servers. Oddly, there is another mention of Gopher+ here, but it’s just a reiteration that it’s not widely used. I guess I really don’t need to worry about it.

After that I found… nothing really? There are of course still plenty of articles out there on the subject, but most are framed as a retrospective of a quirky obsolete thing. Which… it might be, considering most of what I can learn about it seems to be from the 00s or earlier.

But still, research phase complete. I didn’t find any reason for the original RFC to be considered stale so that’s what I’ll use for the specification.

Sanity Check

If you’ve had experiences like I’ve had though, you’ll have been burnt by official documenation not being correct, so it’s always good to play around a bit and make sure everything works.

To start I found a gopher client in Ubuntu’s repositories, while I could just interact with my implementation via netcat, testing it against an actual client would probably help me make sure I am actually doing things right. For testing I made a simple server that just sends a static response to any request, since I plan to do the empty request for root listing.

const { createServer } = require('net');

const menu = [
	"0foobar\t/foobar\tlocalhost\t7000",
	"0qinzuk\t/qinzuk\tlocalhost\t7000",
	"1secret folder\t/secret\tlocalhost\t7000"
]

createServer(socket => {
	socket.once('data', data => {
		let message = menu.reduce((accumulator, current) => {
			return accumulator + current + "\r\n";
		}, "");

		message += ".\r\n";
		socket.write(message, () => {
			socket.destroy();
		})
	})
}).listen(7000);

Now to run gopher localhost 7000 and see the results and… huh.

Just to make sure it isn’t programmer error, I’ll try it with netcat and make sure it’s replying with the correct message.

So what’s up with that?

A quick debugging cycle shows that the Ubuntu gopher isn’t sending the expected blank line to the server, but instead a tab and a dollar sign. From this I assume it expects something different from the results I am sending it.

Trying to DuckDuckGo the problem didn’t turn up many results, outside of emailing the maintainer and asking what was up with that, I didn’t really know what to do. That is, until I stumbled upon something that I really should have noticed during my original research phase.

I was trying to find information about a data communication protocol while using it’s competitor, no wonder I was barely finding anything.

Down the Gopherhole

It’s a bit embarassing how quickly I found out that Gopher is definitely still in use nowadays, with multiple Gopher communities and hundreds of servers serving distributed documents to anyone who wants them, including the Gopher+ specification.

It seems that the default gopher for Ubuntu is sending a Gopher+ command to the server, and it doesn’t seem like there is a way to turn that off. It’s a bit of a bummer that the application named after a protocol doesn’t work with said protocol but there isn’t much I can do about that.

Instead I’ve found Gophie which is a GUI gopher client that seems to actually work with the gopher protcol.

For Real Now

So I’ll be honest with you, I’m actually writing this after getting my implementation to a point where it works at a level I’m not completely embarassed by. This section is going to mostly be a cursory explaination of my thinking throughout it.

I haven’t implemented Gopher+ so the Ubuntu gopher doesn’t work with it, but Gophie and the Floodgap proxy do. At the time of writing this, it only serves menus and text files, but that’s all I need from it at the moment. If/when I work more on it, I’ll be more likely to write in media res about it.

The Data

Originally, I was going to flex my SQL muscles by building out a database to hold most of the information that I was going to serve. My idea was two tables, one of documents/entities, and another of attributes (Gopher+) for the former with enums to hold the types. I think this design would be more than serviceable, if a bit boring. Compared to the next couple ideas, it would probably be the better choice if you are going to have a giant gopher server, so maybe the idea will be revisited in the future.

My next idea came courtesy of the static site generator I am using, Jekyll. In Jekyll, the pages/posts are prefixed with something called ‘Front Matter’, this front matter lets you store information to be used by the pages and generator without having it show up directly in the rendered pages. I do really like this idea, it would open up all sorts of options regarding templating and the ilk. However it’s also a bit more complicated than I wanted to get into to start with. I definitely want to revisit this option in the future too.

Ideas aside, for now I am using structuring similar to Jekyll, however I am using simple .JSON files to hold the information about the posts. It’s nothing fancy but it means I don’t have to do much generating/rendering/compiling to get it to work.

The Communication

As Gopher is stateless, handling the sockets and communication is rather simple, no need for much logic. Pretty much just some message cleanup, including removing the newline at the end. I then have the SocketManager propogate it up to an instance of the Gopher class.

The Logic

I handle most of the logic in the Catalogue class. The Catalogue is what parses the data directory for entities and generates the entries and menus that gets sent back to the client. Given the Catalogue holds all the info about what data the server has, I also have it handling lookups based on selectors passed to it by the Gopher class. If found, the Gopher Class gets a result that it can either display to the client or tells it where to find the file requested.

The Result

If you’d like to check my work, I do have it hosted on Github.

I admit I didn’t go into as much detail here as I could have, but considering that as of now, it’s a minimum viable project, I doubt doing so would be very useful for very long.

That said, as minimally viable as it may be, it is usable. In fact you can access a Gopher version of this site!
I used to have it running, but found it was limiting. Maybe I’ll do it again in a later version.

I look forward to working on it more, I think there is so much potential.