score.netfs

Introduction

Aim of this module is to provide an infrastructure for sharing arbitrary files between multiple hosts. The reason we built this module is that we have the need to distribute uploaded files (like images, or PDF files) across an array of web servers and were not content with any other technology for our use case.

Environment

In order to discuss the need for this module, one must first understand the usual server setup of a web application with multiple servers. The following diagram shall provide an overview of a typical scenario:

             HTTP Request
                  |
<clients>         |     HTTP Request
                  |           |
                  .           .
<other layers>    .           .
                  .           .
                  |           |
                  V           V
             +---------+   +---------+
application  |  web01  |   |  web02  |  ...
layer        +---------+   +---------+
                  |
                  +-------------+--
                  V             V
             +--------+   +--------+
database     |  db01  |   |  db02  |  ...
layer        +--------+   +--------+

We have a multitude of web servers handling requests. In order to have a robust setup, where web servers are interchangeable, any web server must be able to handle any HTTP request by the client.

The Problem

The definition of our environment dictates that web01 must be able to serve a file that was uploaded on web02, for instance. This means that we need some algorithm for getting the file, that was uploaded on web02 onto web01’s file system.

There are three different strategies to solving this problem:

• Distribute on arrival: whenever web02 receives a file, it sends the file to all other web servers. This would presume that every web server knows every other web server. But the acual drawback is that one would need another strategy for retrieving missing files: It is well possible that this web server was offline for some time and did not receive an uploaded file.
• Retrieve on demand from peers: Whenever web01 needs a certail file, it fetches that file from somewhere. That somewhere might be any of the other web servers, which would mean that web01 “broadcasts” to all others a query (“Who has file X?”) and downloads from one of those responding to the query. The issue here is that web01 would have to define a timeout and wait until all web servers respond in a defined time period before giving up and deciding that the file does not exist. This can make for a pretty bad user experience, once a peer is getting slow. Another drawback, which will be eliminated by the next strategy, is that it is hard to migrate files: they are, by the nature of this approach, distributed on multiple machines.
• Retrieve on demand from central server: Whenever a web server receives a file, it immediately uploads it to a central server. Any other web server, that needs a file, can ask that central file server for the file and immediately knows, if it exists. There is a single point of failure in this system, the effects of which can be mitigated by components described at a later point.

Our choice is strategy number three, now we need to define the means of the transport, i.e. how do we upload/download files? Here are some options:

• A network share (like nfs or smb) is the first idea, that comes to mind: It provides the convenience of a local file system and shared nature of a centralized server. The problem with these solutions is that all network shares start behaving in a very unfavorable way whenever there are networking errors. We have seen cases, where machines hang for minutes, even if the network share had an outage of a few milliseconds. So, whereas the idea is a great one, the implementations of these shares have some erratic “quirks”.
• The next idea could be ftp, a rock-stable protocol, where all timeouts can be specified in almost all client libraries. In spite of that, it still has two distinct drawbacks for us:
  • performance: Although everything is running on a local network, on pretty good hardware, we want our downloads to be as fast as possible. We espacially want to reduce the time-to-deliver from the web server’s point of view, and the CPU and memory footprint on the central server. Although the hardware is usually pretty good, it should not have to waste its resources on BASE64-encoding and -decoding files.
  • race conditions: We do not want to download half-uploaded files, or create a broken file on the server through concurrent uploads. Unfortunately, these are issues on FTP servers, due to lack of support in the protocol itself.

The Solution

After some research and a lot of time used on testing various protocols, we decided to implement our own simple client/server protocol. The result was the score module called netfs.

Features

Distributed

This modules provides a dedicated proxy component, which acts as a façade to multiple standalone backends. This setup allows one to ignore all backend errors, as long as at least one backend is still functional.

Transactions

Since web projects usually work within transactions that can succeed or fail, the communication implements transactions at its core. Whenever a client connect, an implicit transaction is started. If the client disconnects without committing the transaction (i.e. the web application had an error), all pending operations are thrown away. If the client was successfull and wants to persists all uploads, it just needs to tell the server to do so.

Locking

The protol prevents mulitple writes onto the same file, as well as downloads of incomplete files.

Hash Comparison

All uploaded data is additionally checked for consistency with its SHA512-hash. This will not fix any errors, but detect most erros in the transmission. It also offers a good compromise between transmission speed and data integrity.

On the Wire

The protocol is extremely simple, where the party initiating the TCP connection (=client) also initiates the conversation. The first thing the client sends is a job byte, followed by job-specific data. The server always responds to the request and once it is finished responding, the client can send the next request.

The following list contains all possible requests accepted by the server:

upload

The client sends a file to the server, that can be persisted with a commit request (below):

+----------+
|  1 Byte  |  Job Byte: always "1" for upload requests.
+----------+
|  4 Bytes |  Signed integer: Length of file name. This is the
|          |    byte length of the UTF-8 encoded file name.
+----------+
|  ? Bytes |  File name: The UTF-8 encoded file name.
|    ...   |
+----------+
|  8 Bytes |  Signed long long: Length of the file content.
|          |
+----------+
|  ? Bytes |  File content
|    ...   |
+----------+
| 64 Bytes |  SHA512-Hash of the file content.
|          |
+----------+

The server responds with a single byte to the whole request, even if it encountered errors earlier. The response is either 1 for success, or 2 for error. If the file is already being uploaded by another client, it is also considered an error.

prepare

An optional step that can be performed before a commit command is issued. It causes the server to verify all uploads and issue an error response in case something went wrong. The request consists of a single job byte:

+----------+
|  1 Byte  |  Job Byte: "3" for prepare request.
+----------+

The response is a single byte: 1 for success, anything else in case of an error.

commit

All uploaded files are persisted. Any existing files, that were re-uploaded, will be overwritten. As the request has no payload, it just consists of a job byte:

+----------+
|  1 Byte  |  Job Byte: "4" for commit request.
+----------+

The response is again a single byte: 1 for success, anything else in case of an error.

rollback

Discards all pending uploads in this session. Consists of a single job byte:

+----------+
|  1 Byte  |  Job Byte: "5" for commit request.
+----------+

download

Requests the contents of a file:

+----------+
|  1 Byte  |  Job Byte: "2" for download requests.
+----------+
|  4 Bytes |  Signed integer: Length of file name. This is the
|          |    byte length of the UTF-8 encoded file name.
+----------+
|  ? Bytes |  File name: The UTF-8 encoded file name.
|    ...   |
+----------+

If the file is found, the server responds with the exact same byte sequence, as a client would initiate an upload, including the “job byte”, which in this case is the equivalant “status byte” 1, which indicates success.

Todo

Download operation now also returns last modification time of the file on the server. This needs to either be implemented for the upload operation, or documented here separately.

Starting the Server

Upon installation, this module registers a score.cli command that can be used to manage a netfs server:

$ score netfs serve path/to/folder

Configuration

score.netfs.init(confdict, ctx=None)

Initializes this module acoording to our module initialization guidelines with the following configuration keys:

server [default=localhost:14000]

The server to connect to for all remote operations. Read using the generic score.init.parse_host_port().

The special value None indicates that all remote operations will immediately raise an exception. It is still possible to use higher level functions like put and get (although get will raise an exception if the requested file is not present in the local folder.)

cachedir [default=None]

A local folder that will hold downloaded files. If this value is omitted, the module will create a new temporary folder on demand, that will be used as the local folder for this session.

deltmpcache [default=True]

This option is only relevant if the configuration did not contain a cachedir. If this value is True, any temporary folder that was created for this session—as described for the configuration value cachedir, above—will be removed when the ConfiguredNetfsModule is freed.

The only use case where you might need this configuration is when you want to operate on a temporary folder, but still keep its contents when you are done with this module.

class score.netfs.ConfiguredNetfsModule

This module’s configuration class.

connect()

Connects to the configured server and returns a NetfsConnection.

class score.netfs.NetfsConnection

put(path, file, ctx=None, *, move=True)

Uploads a file with given path to the server and moves it into the cache folder. The file is either a string (denoting a file system path to the file), or a file object.

In case the file parameter was a string, it is possible to keep that original file in place by specifying a falsy value for move.

get(path)

Returns the local path to a file, downloading it from the server, if it does not already exist in the local cache folder.

upload(path, file, ctx=None)

Puts the contents of given file object file with given path onto the server.

You must call commit() to actually persist the upload. If you are using the ctx module, though, you should pass a context object as ctx. This will automatically commit the upload if the transaction was successful.

commit()

Instructs the server to persist all uploaded files, so that other clients can find them.

download(path, file, retry=1)

Downloads the file with given path from the server and writes it into the file object file.