Brandon Morris

Reinforcement Learning: Playing Doom with PyTorch

2018-10-09T00:00:00+00:00

This tutorial is adapted from the one on ViZDoom’s website. Additionally, the code used here is adapted from this tutorial, with substantial modification.

Machine learning allows us to program by example. We can present the algorithm with some data, potentially provide it some feedback, and then glean the results of our system. For image classification, we give the model some images and it learns to identify what object(s) are in that image. Tasks like this where the model only needs to find the “right answer” (i.e. supervised learning) have seen a lot of success, and have huge potential to automate mundane manual tasks. But is that all machine learning can do?

In this post, I’ll introduce some of the ideas fundamental to reinforcement learning, and how it differs from typical supervised learning. We will then examine up close one algorithm for solving reinforcement learning problems, known as Deep Q-learning. Then, we’ll implement Deep Q-learning to teach a neural network how to play a simple game of Doom using the ViZDoom environment and PyTorch.

Slow-motion capture of the reinforcement learning agent shooting a monster in Doom

What is Reinforcement Learning?

Reinforcement learning is a branch of machine learning where we try to teach the model to actually do something. The most famous example of reinforcement learning is the success of DeepMind’s AlphaGo and its variants. Rather than just predicting an answer, AlhpaGo is a reinforcement learning agent that learns to masterfully play the game of Go. It can’t just classify; it needs to sequentially interact with its environment – making moves and receiving its opponent’s moves – in such a way that it will be most likely to achieve its long-term goal of winning the game.

Even though Go is just a board game, programming a competent player is exceedingly difficult. And interestingly, the same framework used to design AlphaGo can be applied to nearly any other domain. This framework is known as the Markov Decision Process (MDP), and it allows us to rigorously and mathematically characterize a system for reinforcement learning (as well as other situations).

The Markov Decision Process

MDPs have several different formulations and variants. However, there are two critical components that are tacitly understood: the agent and the environment. The agent is the person or thing that is actually trying to perform the task. They make the decisions and carry out the actions. The environment is essentially everything else: the world around the agent, the rules of that world, and even other players can be abstracted out to the environment.

The loop for reinforcement learning: The agent perceives the environment and decides an action, which changes the environment.

In addition to the agent and the environment, MDPs are made up of several pieces. First is the set of states, which is just the potential configurations of the agent/environment at a given point in time. For something like a board game, the state is just the current board and whether it’s the agent’s turn or not. Next are the actions: all the things that the agent can actually do. Note that the actions are dependent on the state, since not all actions are valid in every state. The last main piece is the reward function, which tells us how “good” our agent is doing at a task. This is what our reinforcement learning algorithm is going to focus on. Ultimately, we want to train the model to know how to act such that it will maximize the overall reward, also known as the return. The overall reward is calculated as just the sum of the rewards at each step in the process, but maximizing it is difficult, since we may have to make strategic decisions that are initially low reward to boost the final return.

Depending on the situation, the MDP may also include transition probabilities. These tell us how likely we are to transition to a new state $s_2$ if we’re currently in a state $s_1$ and we take some action $a$. However, in complex problems we often don’t know what the transition probabilities will be. In a board game like Go, how can I effectively predict how my opponent will move? Additionally, in domains like Doom, the state space is so large that enumerating the transitions is impractical. So instead we let the reinforcement algorithm learn these transitions as well. Here is how reinforcement learning differs from planning systems: we don’t assume to know the world dynamics, and instead try to learn those dynamics along with good actions.

Aside: You may be wondering why we call this framework a Markov decision process. The Markov property states that we can reason about all future state given only the current state. That is, we only need to know where we are right now, not necessarily how we got here. This is usually the case, and is crucial for reinforcement learning algorithms to be tractable. Even in cases where the history is significant, there are ways we can encode that history into the current state to maintain the Markov property.

Deep Q-Learning

Up to this point, we’ve only described the reinforcement learning problem: given an MDP, we want to figure out good actions that will maximizes the sum of our rewards (i.e. the return). The process of deciding an action from a state is known as a policy, so in other words, we want to learn the best policy for a given task. There are several different algorithms that do this, but one of the most straightforward that we’ll look at here is known as Q-learning.

Before we can discuss how Q-learning actually works, we need some more terminology. Recall that the policy involves selecting an action from a state, and that the return is the sum over all our rewards at each state. Then the value of a state $V_\pi(s)$ is the expected return if we start in state $s$ and follow policy $\pi$. Essentially, $V_\pi$ tries to predict our final score using just the current state and the action-selection process.

We can take this a step further. Instead of taking just a state and trying to predict the final score, we can take the current state and an action and try to predict the return. This is known as the Q-value: $Q_\pi(s, a)$. If our Q-values are accurate, then playing optimally just boils down to picking the action with the highest Q-value in our state. However, since we don’t know the transition probabilities, we have to estimate these Q-values, and try to improve them. This is where Q-learning comes in. Additionally, we can use deep neural networks to approximate the Q-functions, hence Deep Q-Learning.

You may have noticed that Q-functions are inherently recursive. That is, we can decompose the value of a Q-function by putting it in terms of the Q-function in the next state:

\[Q(s_t, a_t) = r_{t+1} + \gamma \cdot \max_a Q(s_{t+1}, a)\]

where $r_{t+1}$ is the reward we got for taking action $a_t$, and $\gamma$ is our discount factor that trades off immediate vs. long-term rewards. All this equation says is that Q-functions build off each other over time, and we can leverage that fact to efficiently estimate them.

To learn the Q-functions, we’ll utilize a deep neural network. The network will take a state as input, and output a vector of Q-values, one for each action. We will train it by presenting it with sets of transitions (a first state, action, reward, and second state). The Q-value for the first state should have a value in the index of the selected action that matches the right-hand side of the above equation. That difference (squared) will be our loss backpropagated through our network.

That’s really all there is to Deep Q-learning. We try to approximate a function that estimates our overall return after taking a particular action is a particular state. We turn this global problem into a much more localized variant by trying to optimize our Q-function estimations over individual transitions. Provided we have enough of these transitions and they are adequately diverse, the Q-function will converge to reasonably correct values that let us derive an optimal policy by repeatedly selecting the maximum action from the Q-values.

Putting it to Practice: ViZDoom

The ViZDoom environment is a fantastic tool for playing with reinforcement learning. It provides a nice programming interface for the classic video game Doom, and was designed with reinforcement learning in mind. It comes with several scenarios out of the box, such as the one we will use that involves shooting a monster across the room. However, these scenarios can actually be custom-built using existing free tools like Doom Builder.

For the sake of brevity, I’m only going to walk through the particularly important parts of the Q-learning implementation. You can see the full script to train and run the ViZDoom agent at this GitHub gist.

First, we’re going to define a class for our replay memory. The replay memory will serve as a bank of the recent transitions (e.g. first state, action taken, second state, and reward). Additionally, we need to keep track as to whether the action terminated the episode, since that will mean there is no second state to process.

class ReplayMemory:
    def __init__(self, capacity):
        channels = 1
        state_shape = (capacity, channels, *resolution)
        self.s1 = torch.zeros(state_shape, dtype=torch.float32).to(device)
        self.s2 = torch.zeros(state_shape, dtype=torch.float32).to(device)
        self.a = torch.zeros(capacity, dtype=torch.long).to(device)
        self.r = torch.zeros(capacity, dtype=torch.float32).to(device)
        self.isterminal = torch.zeros(capacity, dtype=torch.float32).to(device)

        self.capacity = capacity
        self.size = 0
        self.pos = 0

    def add_transition(self, s1, action, s2, isterminal, reward):
        idx = self.pos
        self.s1[idx,0,:,:] = s1
        self.a[idx] = action
        if not isterminal:
            self.s2[idx,0,:,:] = s2
        self.isterminal[idx] = isterminal
        self.r[idx] = reward

        self.pos = (self.pos + 1) % self.capacity
        self.size = min(self.size + 1, self.capacity)

    def get_sample(self, size):
        idx = sample(range(0, self.size), size)
        return (self.s1[idx], self.a[idx], self.s2[idx], self.isterminal[idx],
                self.r[idx])

The replay memory mostly just stores huge batches of the transitions, though we included some useful methods for adding transitions to the memory and gathering a random sampling from the non-zero entries. The replay memory will be critical to training our network, since it allows us to efficiently gather numerous and diverse inputs from the agent’s experience. In fact, the model will only learn from the replay memory directly. As the agent learns during training, it will leverage the Q-network to determine its actions, add its experience to the replay memory, and then update its parameters from a sample of transitions that come from the replay memory.

Next, we’ll build out our actual Q-function model. Recall that we are using a deep neural network to approximate our Q-function. Since ViZDoom will give us raw pixels as our inputs, we’ll leverage a convolutional neural net that can effectively learn the visual features.

class QNet(nn.Module):
    def __init__(self, available_actions_count):
        super(QNet, self).__init__()
        self.conv1 = nn.Conv2d(1, 8, kernel_size=6, stride=3) # 8x9x14
        self.conv2 = nn.Conv2d(8, 8, kernel_size=3, stride=2) # 8x4x6 = 192
        self.fc1 = nn.Linear(192, 128)
        self.fc2 = nn.Linear(128, available_actions_count)

        self.criterion = nn.MSELoss()
        self.optimizer = torch.optim.SGD(self.parameters(), FLAGS.learning_rate)
        self.memory = ReplayMemory(capacity=FLAGS.replay_memory)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = x.view(-1, 192)
        x = F.relu(self.fc1(x))
        return self.fc2(x)

    def get_best_action(self, state):
        q = self(state)
        _, index = torch.max(q, 1)
        return index

    def train_step(self, s1, target_q):
        output = self(s1)
        loss = self.criterion(output, target_q)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss

    def learn_from_memory(self):
        if self.memory.size < FLAGS.batch_size: return
        s1, a, s2, isterminal, r = self.memory.get_sample(FLAGS.batch_size)
        q = self(s2).detach()
        q2, _ = torch.max(q, dim=1)
        target_q = self(s1).detach()
        idxs = (torch.arange(target_q.shape[0]), a)
        target_q[idxs] = r + FLAGS.discount * (1-isterminal) * q2
        self.train_step(s1, target_q)

Here we define the basic architecture and some useful methods for training. Note that the network isn’t particularly large: only 4 layers and not a great deal of parameters at each of those layers. The particular task isn’t very complex, and we’re restricting our inputs to small grayscale images of 30x45 pixels.

Pay particular attention to the second to last line in the learn_from_memory() method. We want the Q-values for s1 to match the recursive equation above (but only at the action that was actually taken during that transition). But updating these indexes to the “true” value, we can take the squared difference as our network loss.

Now that we have our replay memory and model, we can flesh out our training loop method. As I mentioned before, the basic formula is to first experience a transition, then record that transition and learn from the replay memory. Here’s the code below:

def perform_learning_step(epoch, game, model, actions):
    s1 = game_state(game)
    if random() <= find_eps(epoch):
        a = torch.tensor(randint(0, len(actions) - 1)).long()
    else:
        s1 = s1.reshape([1, 1, *resolution])
        a = model.get_best_action(s1.to(device))
    reward = game.make_action(actions[a], frame_repeat)

    if game.is_episode_finished():
        isterminal, s2 = 1., None
    else:
        isterminal = 0.
        s2 = game_state(game)

    model.memory.add_transition(s1, a, s2, isterminal, reward)
    model.learn_from_memory()

Note the action selection process. Initially, our agent has no idea what good actions are. As such, we want it to explore very broadly, so that it can get a diverse range of experience that it can build off of. The find_eps method will determine some exploration rate depending on how far into training we are. As the agent is more and more trained, it will take random actions (i.e. explore) less and more often take the best action available. This is known as an “epsilon-greedy” policy. When we’re done training, or evaluating our model, we will always select the best action and no longer explore.

The rest of the code involves setting up the ViZDoom game, command line flags, and training epoch loops. All of it is pretty standard, and has thus been omitted. You can see and run the full script here. Since the network and inputs are pretty small, you should be able to run this on your personal computer, even if you don’t have a GPU.

I trained this model on my machine for 20 epochs at 2,000 iterations per epoch. Pretty quickly the agent learned a reasonable policy, and the whole thing converged in a little less than 20 minutes. You can see one of the test episodes in gif form below.

Slow-motion capture of the trained agent. The agent overshoots initially, but moves back in front of the monster to get the full reward of the kill.

Additional Resources

Deep reinforcement learning is a burgeoning field with lots of exciting new advancements. This tutorial barely scratches the surface of Deep RL, but should provide you with everything to get started. If you’re interested in learning more, here are several resources that I found particularly interesting and/or useful.

“Reinforcement Learning: An Introduction” by Sutton and Barto This is the textbook on reinforcement learning broadly. A classic in the field, and a free draft is available here.
Reinforcement Learning Crash Course Lectures from David Silver’s course of reinforcement learning. Link
“Deep Reinforcement Learning Doesn’t Work Yet” A critical yet honest appraisal of the current state of reinforcement learning and how it often falls short of our press releases. Link
Deep Reinforcement Learning NIPS 2018 Workshop Collection of talks and papers literally on the cutting edge of Deep RL research (to be published with the conference in December). Link

Building a World-Class CIFAR-10 Model From Scratch

2018-06-30T00:00:00+00:00

In this post, I walk through how to build and train a world-class deep learning image recognition model. Deep learning models tout amazing results in competitions, but it can be difficult to go from a dense, technical research paper to actually working code. Here I take one of those papers, break down the import steps, and translate the words on the page into code you can run and get near state-of-the-art results on a popular image recognition benchmark.

The problem we will be solving is one of the most common in deep learning: image recognition. Here, our model is presented with an image (typically raw pixel values) and is tasked with outputting the object inside that image from a set of possible classes.

The dataset will be using is CIFAR-10, which is one of the most popular datasets in current deep learning research. CIFAR-10 is a collection of 60,000 images, each one containing one of 10 potential classes. These images are tiny: just 32x32 pixels (for reference, an HDTV will have over a thousand pixels in width and height). This means the resulting images are grainy and it’s potentially difficult to determine exactly what’s in them, even for a human. A few examples are depicted below.

Images of a boat and frog from the CIFAR-10 dataset

The training set consists of 50,000 images, and the remaining 10,000 are used for evaluating models. At the time of this writing, the best reported model is 97.69% accurate on the test set. The model we will create here won’t be quite as accurate, but still very impressive.

The architecture we will use is a variation of residual networks known as a wide residual network. We’ll use PyTorch as our deep learning library, and automate some of the data loading and processing with the Fast.ai library. But first, let’s dig into the architecture of ResNets and the particular variant we’re interested in.

The Residual Network

Deep neural networks function as a stack of layers. The input moves from one layer, to the next, with some kind of transformation (e.g. convolution) followed by a non-linear activation function (e.g. ReLU). With the exception of RNNs, this process of pushing inputs directly through the network one layer at a time was standard practice in top-performing deep neural networks.

Then, in 2015, Kaiming He and his colleagues at Microsoft Research introduced the Residual Network architecture. In a residual network (resnet, for short), activations are able to “skip” past layers at certain points and be summed up with the activations of the layers it skipped. These skip connections form what are typically referred to as a residual block. The image below depicts one block in a resnet.

The structure of a resnet block. Inputs are allowed to skip past layers and be summed up with the activations of the layers they skipped.

Architectures built by stacking together residual blocks (i.e. resnets) train much more efficiently and to less error. The original paper explores various depths, and are able to train networks of over 1,200 layers. Before, it was difficult to train networks with just 19 layers. One potential reason resnets allow for deeper networks is because they allow the gradient signal from backpropagation to travel further back up through the network, using the skip connections like a highway to get closer to the input layer. In 2015, a residual network won the ImageNet with 3.57% test error.

The authors explain the intuition (and the name) of the residual block as a recharacterization of the learning process. Consider just a few layers, like those that make up a single residual block. Now, there should be some ideal mapping from the block’s inputs to it’s output. Let’s call this mapping $H(x)$. Typical learning tries to derive this mapping directly: that is, find an $F(x, W)$ similar to our ideal $H(x)$. But we can change this, and instead allow $F$ to approximate the residual, or the difference, between $H(x)$ and $x$. That is,

\[F(x, W) := H(x) - x\]

which is equivalent to

\[H(x) = F(x, W) + x\]

which is the definition of our residual block.

The Wide ResNet

Since their introduction, resnets have become a standard choice for deep learning architectures dealing with computer vision. Several variations of the residual blocks and architectures presented in the original paper have been explored, one of which currently holds the state of the art test accuracy for CIFAR-10.

The variation we are going to implement here is the wide residual network. Here, the authors point out that the depth of resnets was the focal point in their introduction, rather than the width (that is, the number of convolutional filters in the layers). They explore some different kinds of resnet blocks, and show that shallow and wide can be faster and more accurate than the original deep and thin.

Comparison of the different block structures in vanilla and wide resnets. The two on the left are those found in a traditional resnet: a basic block of two thin 3x3 convolutions and a "bottleneck" block. On the right, the wide resnet uses blocks similar to the original basic block, but much wider convolutions (i.e. more filters). There may or may not be dropout between the convolutions to regularize the model.

The Structure of a Wide ResNet

The wide resnet consists of three main architectural components:

An initial convolution. This is done to pull out any high level features and help upsample our initial image from only three channels to a high-dimensional convolutional activation.
A number of “groups”. Each group will consists of a set of $N$ residual blocks. More on this in a moment.
A pooling and linear layer. This will downsample our convolutions and convert them into class predictions.

The real meat of the wide resnet will lie in the groups: that’s where all of our residual blocks will live. The original paper always used three groups in their experiments, but we will write our code to be modular to the number of groups.

Outline of the wide resnet architecture. `conv1` is the initial convolution and `conv2` through `conv4` make up the three groups, each consisting of $N$ blocks. In this case, the blocks are the wide 3x3 basic blocks, where the width is initially 16$\cdot k$ and doubled after each group. Every group after the first also downsamples to reduce the width and height of the convolutional activations.

There are a few considerations that will become key to implementing the blocks in each of our groups:

Each block after the first will downsample the size of the activations. This means that the 32x32 activation block will shrink to 16x16. We’ll do this by setting the stride of the first convolution in the blocks to 2.
Each group will double the number of filters from the previous group.
The first block of each group will need to have a convolution in its shortcut to get it to the right dimensions for the addition operation.

So how wide will these convolutions be? Our initial convolution will turn our three channels into 16. The first group will multiply the number of channels by the widening factor $k$, and every subsequent group will double the width of the convolutions. Essentially, the $i$th group will have $(16 \cdot k)\cdot 2^i$ filters in its convolutions (where $i$ starts from 0).

Implementing the Wide ResNet

Now that the architecture is all settled, it’s time to write some code. I’m going to implement this in PyTorch, with a little help from the fastai library. Fastai is a fantastic library for quickly building high quality models. It’s also really helpful for automating the more mundane aspects of writing deep learning code, like building data loaders and training loops, which is what I’ll use it for here.

The implementation will be done piece by piece: starting with the basic block, then fleshing out the whole network, and finally building our data pipeline and training loop. You can find the complete implementation here.

Note: Some of this code is not going to be as tidy as it could be. In this article, I’m optimizing for understanding, not necessarily style or cleanliness.

The `BasicBlock` Class

Since the majority of the model will consist of basic residual blocks, it makes sense to define a reusable component that we can fill our model with. Fortunately, PyTorch makes this really easy by allowing us to subclass the nn.Module class.

The full implementation of the BasicBlock class can be seen below:

class BasicBlock(nn.Module):
    def __init__(self, inf, outf, stride, drop):
        super().__init__()
        self.bn1 = nn.BatchNorm2d(inf)
        self.conv1 = nn.Conv2d(inf, outf, kernel_size=3, padding=1,
                               stride=stride, bias=False)
        self.drop = nn.Dropout(drop, inplace=True)
        self.bn2 = nn.BatchNorm2d(outf)
        self.conv2 = nn.Conv2d(outf, outf, kernel_size=3, padding=1,
                               stride=1, bias=False)
        if inf == outf:
            self.shortcut = lambda x: x
        else:
            self.shortcut = nn.Sequential(
                    nn.BatchNorm2d(inf), nn.ReLU(inplace=True),
                    nn.Conv2d(inf, outf, 3, padding=1, stride=stride, bias=False))

    def forward(self, x):
        x2 = self.conv1(F.relu(self.bn1(x)))
        x2 = self.drop(x2)
        x2 = self.conv2(F.relu(self.bn2(x2)))
        r = self.shortcut(x)
        return x2.add_(r)

A few things to note:

After the first group, the first block of each group will need to downsample the height and width of the convolutional activation. This can be done by passing in a 2 to the stride parameter when instantiating the first BasicBlock of the group.
With the exception mentioned above, each convolution should preserve the width and height of the convolutional activation. We achieve this by always using a kernel_size of 3 and a padding of 1. Additionally, since we’re using batchnorm, our convolutions don’t need a bias parameter, hence bias=False.
We follow the order of batchnorm -> relu -> convolution. Although the original batchnorm paper used a different order, this has since been shown to be more effective during training.
If this is the first block in a group, we’re going to double the width via our convolutions. In that case, the dimensions won’t match for the shortcut connection, so shortcut will need it’s own convolution (preceeded by batchnorm and relu) to increase it to have width outf. Also, since we only double on the first block in a group, and we may be downsampling then too, we’ll need to use the stride parameter in this convolution as well.

The `WideResNet` Class

Now that we have our BasicBlock implementation, we can flesh out the rest of the wide resnet architecture.

class WideResNet(nn.Module):
    def __init__(self, n_grps, N, k=1, drop=0.3, first_width=16):
        super().__init__()
        layers = [nn.Conv2d(3, first_width, kernel_size=3, padding=1, bias=False)]
        # Double feature depth at each group, after the first
        widths = [first_width]
        for grp in range(n_grps):
            widths.append(first_width*(2**grp)*k)
        for grp in range(n_grps):
            layers += self._make_group(N, widths[grp], widths[grp+1],
                                       (1 if grp == 0 else 2), drop)
        layers += [nn.BatchNorm2d(widths[-1]), nn.ReLU(inplace=True),
                   nn.AdaptiveAvgPool2d(1), Flatten(),
                   nn.Linear(widths[-1], 10)]
        self.features = nn.Sequential(*layers)

    def _make_group(self, N, inf, outf, stride, drop):
        group = list()
        for i in range(N):
            blk = BasicBlock(inf=(inf if i == 0 else outf), outf=outf,
                             stride=(stride if i == 0 else 1), drop=drop)
            group.append(blk)
        return group

    def forward(self, x):
        return self.features(x)

You can see the outline of the architecture in the code. Right after we call the super constructor, we initialize the first convolutional layer (conv1 in the architecture table).

After the initial convolution, we calculate the widths (i.e. number of filters) in each block, creating a list that will become our inf and outf parameters during block construction. Then we construct each group in a for loop. If this is the first group, we use stride=1 since this is the only time we don’t want to decrease the width and height of the convolutional activations. Making a group involves calling our _make_group helper function, which will construct N instances of BasicBlock with the appropriate inf, outf, and stride parameters.

Finally, we average pool our activations, turning each $64 \cdot k$ convolutional activations into a single value, which is input to our last linear layer used for classification.

Data Loading and Training

Our model is locked and loaded, now we just need some data to feed it and a training loop to optimize it. Since this is the least interesting part of building a model, I’m going to rely heavily on the fastai library. Note that for this code to run, the library will need to be importable, which is most simply done by cloning the repository and then symlinking the library directory into the same directory that the model is in.

To start, we’ll set up our data folder and download our dataset via torchvision.datasets. We’ll also convert the dataset to numpy arrays of floating point values, and move the inputs between 0 and 1.

os.makedirs(PATH, exist_ok=True)
trn_ds = CIFAR10(PATH, train=True, download=True)
tst_ds = CIFAR10(PATH, train=False, download=True)
trn = trn_ds.train_data.astype('float32')/255, np.array(trn_ds.train_labels)
tst = tst_ds.test_data.astype('float32')/255, np.array(tst_ds.test_labels)

Now trn and tst are tuples containing our training and test inputs/outputs, respectively. Next we’ll set up our preprocessing transformations using fastai.

sz, bs = 32, 128
stats = (np.array([ 0.4914 ,  0.48216,  0.44653]),
         np.array([ 0.24703,  0.24349,  0.26159]))
aug_tfms = [RandomFlip(), Cutout(1, 16)]
tfms = tfms_from_stats(stats, sz, aug_tfms=aug_tfms, pad=4)

Our inputs will be size 32x32, with batch size 128 (you may need to decrease this depending on your hardware; this is the value used in the original paper). We set up our tfms object to be a list of transformation for our inputs: we normalize based on the known means and standard deviations, take a random crop after padding each size 4 pixels, and randomly flip the image 50% of the time. Additionally, we also use cutout, which will randomly zero out a square in our input image. Here we set cutout to use 1 square of length 16.

Finally, we’ll put everything together by creating a dataset object, instantiating our model, and creating a learner object.

data = ImageClassifierData.from_arrays('data', trn, tst, bs=bs, tfms=tfms)
wrn = WideResNet(n_grps=3, N=4, k=10)
learn = ConvLearner.from_model_data(wrn, data)

Here we’re using a wide resnet with 3 groups, each group has four blocks, and a widening factor of 10. We’ll also let dropout be 0.3, which is the default we picked when we defined the class. This results in a 28-layer network and produced the best results for our dataset.

To train, we will follow the same training procedure outline in the original paper.

lr = 0.01
wds = 5e-4
for i, epochs in enumerate([60, 60, 40, 40]):
    learn.fit(lr, epochs, wds=5e-4 best_save_name=f'wrl-10-28-p{i}')
    lr /= 5

We train for 200 epochs, decreasing the learning rate by a fifth at certain intervals. Fastai will automatically save the best performing model of each phase in our data directory since we set the best_save_name parameter.

In my own tests, this model achieved a final test time accuracy of 95.84%. The current state of the art for CIFAR-10 is about 98% (though they also trained for 9 times as long). Not bad for less than 100 lines of code!

Conclusion

In this post, I walked through implementing the wide residual network. Leveraging PyTorch’s modular API, we were able to construct the model with just a few dozen lines of code. We also were able to skip past the mundane image processing and training loop using the fastai library.

Our final results got us almost 96% accuracy on a rather challenging dataset. We are within 2% of the best that anybody has ever done. While deep learning moves at breakneck speeds, often times papers will present ideas that are fairly straightforward to reimplement yourself. This isn’t always the case, like in some experiments that require absolutely enormous computational power. But in cases like the wide resnet, it can be really fun and extremely rewarding to recreate a paper’s experiments from scratch.

Mastering the Learning Rate to Speed Up Deep Learning

2018-06-24T00:00:00+00:00

Efficiently training deep neural networks can often be an art as much as a science. Industry-grade libraries like PyTorch and TensorFlow have rapidly increased the speed with which efficient deep learning code can be written, but there are still a lot of work required to create a performant model.

Let’s say, for example, you want to build an image classifier model. A convolutional neural network would be the proper approach to utilize deep learning. But how many layers go in your network? How much momentum and weight decay should you use? What’s the best dropout probability?

The reality is that these questions don’t have definitive answers. What works great on one dataset might not work nearly as well on another. There are sensible defaults and good rules of thumb, but finding the best combination is nontrivial. These kinds of decisions are known as hyperparameters: values that are determined prior to actually executing the training algorithm. Figuring out the optimal set of hyperparameters can be one of the most time consuming portions of creating a machine learning model, and that’s particularly true in deep learning.

Difficulties in Finding the Right Hyperparameters

Unlike the parameters inside the model, the hyperparameters are difficult to optimize. While it’s possible to optimize hyperparameters with Bayesian methods, this is almost never done in practice. Instead, the best set of hyperparameters is typically sought through a brute force search.

Part of the difficulty of finding the right hyperparameter values is their complex interplay between each other. One value of weight decay may work well for a particular learning rate and poorly for another. Changing one value impacts many others in ways that are difficult to control.

A tempting, naive method is to set up reasonable steps for each hyperparameter, and loop over a range, trying different values for each one. This is known as grid search, and it’s generally a bad idea for two reasons. First, the model has to be completely retrained for each set of hyperparameters, and the number of sets grows exponentially with the number of hyperparameters. Most of these values will be suboptimal, meaning that we’re wasting a great deal of time and energy unnecessarily retraining out model. The second reason is a little more subtle. Our steps will need to have a reasonable size to reduce the number of times we need to retrain the model, meaning we’re jumping over a decent bit of the search space with each iteration. There’s no reason our particular intervals are likely to contain good values, so it very possible we will entirely skip over good values. In fact, just doing a random search will usually yield better results that stepping across a fixed interval. The picture below depicts this visually.

Somewhat counterintuitively, randomly searching for hyperparameter values can give better results than a systematic grid search. Taken from Random Search for Hyper-Parameter Optimization

Unfortunately, the state of the art in hyperparameter selection is little more than a random search. Most values have sensible defaults, but picking the best possible set can have a significant impact on the model’s final performance. Many machine learning researchers and practitioners develop intuitions about good values and how hyperparameters interact, but it takes a lot of time and practice. However, some recent and exciting research has outlined techniques for finding arguable the most important hyperparameter: the learning rate.

What is the Learning Rate?

Neural network training is typically performed as stochastic optimization. We start out with a random set of network parameters, find out which direction they should move to be improved, then take a step in that direction. This process is known as gradient descent (the stochastic portion comes from the fact that we find our improvement direction on a random subset of the training data). The learning rate determines how big of a step we take in updating the parameters.

# w is our weight, and dw is the derivative
w += -learning_rate * dw

The above parameter update occurs every iteration of the training process (though modern networks almost always use a more sophisticated update that adds extra terms). Without a doubt, the learning rate is the single most important hyperparameter for a deep neural network. If the learning rate is too small, the parameters will only change in tiny ways, and the model will take too long to converge. On the other hand, if the learning rate is too large, the parameters could jump over low spaces of the loss function, and the network may never converge.

3e-4 is the best learning rate for Adam, hands down.
— Andrej Karpathy (@karpathy) November 24, 2016

Picking the learning rate is pretty arbitrary. There are a range of reasonable values, but that range and the optimal value will vary with the architecture and dataset. As Andrej Karpathy joked in a tweet seen above, saying that one learning rate is “the best” is pretty preposterous.

Commonly, the ideal learning rate will change during training. Most world-class deep architectures are trained with a piecewise annealing strategy: train the network for a while with one learning rate, and when the model stops improving, decrease the learning rate by some factor and keep going. Intuitively, this makes some sense: if the model gets to a low spot in the loss space, the steps we take may be too big to keep from jumping across deeper valleys. Decreasing the learning rate allows for a more fine-grained training.

While piecewise annealing works in practice, we’ll soon see that it is suboptimal. There are better ways that we can (1) systematically find appropriate learning rate(s) for our particular problem, and (2) schedule the learning rate to automatically vary for faster training and improved performance.

Cyclical Learning Rates

Picking the perfect learning rate is hard. In fact, it’s probably too hard to find the singular best value. Instead, we can pick a range of learning rates and move through them during training. Kind of surprisingly, this method of cyclical learning rates works pretty well.

Plot of the learning rate during training. Each cycle starts with a high learning rate that decreases during training. In this case, we're using cosine annealing. During training, we'll spend most of our time at least close to the optimal learning rate, and automatically decrease it for better fine-grained training.

Cycling through values for the learning rate during training alleviates two of the problems with picking the learning rate. First, we don’t need to find an exactly perfect value, just a range of potentially good values. If we pick our range well (more on that shortly), we will be close to the optimal value for most of the training cycle, which is much better than randomly searching for the perfect learning rate. Additionally, we no longer need to manually schedule the learning rate to decrease during training, since the cycle does it for us. Just be sure to start the cycle with a high rate, and decrease it to a low rate.

You might be asking yourself why immediately reset the learning rate to a high value instead of allowing it to gradually climb back up. Resetting to a high learning rate gives us the benefit of a warm restart in our optimization and can improve our generalization. Remember that in machine learning, our primary goal is not to create a model that works well on the training data, but has high performance on the test data. This means that we not only want to find a low spot in the loss space during training, but we also want a very wide space. That way, even when our model is presented with new data that moves it around in loss space, it’s still likely to be at very low spot, and hence very accurate. Warm restarts helps us find those low and wide spaces that we’re looking for. Even if we find a cozy low valley with our low learning rate, restarting it to a high value at the start of a new cycle will pop us right out of that space if it’s not wide enough.

There are several ways to tinker with cyclical learning rates that might improve the final results. The length of a cycle is usually about an epoch, but longer cycles are possible. We can even increase the length of the cycle after each cycle. For instance, start with a cycle length of one epoch, then two epochs, then four, and so on. Schedules like this often give good results in part because they spend more time at lower rates during the end of training, allowing the model to hone in on an optimal space of the loss.

Cyclical learning rates allow us to circumvent the difficulty of picking a good learning rate. All we need are approximate bounds, and we can spend the majority of our training time being close to the optimal value, even as that optimal value changes during training. Additionally, we get the added benefit of restarts that will help us find wide areas in the loss space, improving our generalization ability. Now all we need is a method to find the approximate bounds to cycle through.

The LR Range Test

Cyclical learning rates preclude us from needing to find an optimal learning rate, but we still need an upper and lower bound for our cycles. Luckily, we don’t need to resort to the guessing game and random search that plagued or initial hyperparameter search. Instead, the paper that described the cyclical learning rate method also introduced a systematic method for finding good boundaries: the LR Range Test.

The LR Range Test is simple to understand and cheap to execute. Start with your initialized network, and pick a very small learning rate (much smaller than you would ever likely use). As you train, exponentially increase the learning rate. Keep track of the loss function for each value of the learning rate. If you’re in the right range, the loss should drop, then increase as the learning rate gets too high. Below is a graph of the loss value as a function of the learning rate.

Loss as a function of the learning rate for the LR Range Test. Starting from scratch, exponentially increase the learning rate until the loss begins increasing. The point just before the loss starts to increase is the upper bound to use for cyclical learning rate. A good lower bound is one tenth of the upper bound.

Looking at a plot of the loss vs. the learning rate, we can find our boundaries to use for our cycles. The place to look for is the learning rate where the loss stops decreasing: the minimum value. In the graph above it’s roughly $10^{-1}$. This value is probably too high to use as our boundary, which is why the loss stopped decreasing here. We need to go back just a bit to a smaller value for our maximum boundary to use in our cycles. A good one to use here would be $10^{-2}$. At that point the loss is still decreasing with some gusto. We wouldn’t want to pick the value with the steepest slope, since this will be the maximum, and the cycle will only spend a little while at that point. For the minimum, we can use any value that is smaller; typically we can divide the maximum by a factor such as 3 or 10.

Pedal to the Metal: Super-Convergence

Cyclical learning rates work well in practice, but there’s actually a way to take it a step further. The technique was introduced by Leslie Smith again and dubbed super-convergence. This strategy is a modification of the cyclical learning rate, and allows for training to converge substantially faster, hence the name.

To exploit super-convergence, instead of iterating over cycles of the learning rate, we use a single “1cycle” policy. We derive the maximum and minimum learning rate from the LR Range Test as before. Now, we take one long cycle, moving up from the minimum to the maximum, and back down again. Then we continue training and decreasing the learning rate. We also inversely cycle the momentum, going from a high to low, and allowing it to continue increasing. A plot of the learning rate and momentum schedules are shown below.

The 1cycle learning rate and momentum schedule. Following this policy during training leads to very fast train times and the phenomenon known as "super-convergence".

Amazingly, adopting the 1cycle policy permits incredibly fast training. The original authors reported training deep networks on large datasets in a fraction of the epochs required by other training regimes. Recently, fast.ai leveraged super-convergence to train an ImageNet model in less than three hours, and a CIFAR10 model in lest than three minutes.

Conclusion

There are many difficulties in training deep neural networks. The best practitioners have spent a long time cutting their teeth and developing intuitions about the best values for hyperparameters. Fortunately, research has shown us better ways to pick the learning rate than wasting time and computing power fumbling around in the dark. The LR Range Test provides a quick way to find suitable boundaries for the learning rate, which we can cycle through during training to completely avoid having to find an optimal value. This means more time can be spent training more networks, and less time searching for hyperparameters. Additionally, the 1cycle policy lets us train neural nets at breakneck speeds, creating performant models in a fraction of the training time.

Special thanks to the fast.ai course for providing the inspiration and instruction for this blog post.

Breaking Neural Nets with Adversarial Examples

2018-02-11T00:00:00+00:00

Deep learning has asserted itself as the king of machine learning. No other method produced thus far has had such excellent success at machine learning tasks that are increasingly complex. In some cases, deep neural networks trained by backpropagation and stochastic gradient descent (i.e. deep learning) have been able to dramatically outperform humans merely by being presented examples of a particular task. The model cultivates “knowledge” by discerning which signals (called “features”) are significant and how their existence or absence, in conjunction with other signals, contributes to an overall results.

It is beyond doubt that deep neural networks are incredibly sophisticated and versatile machine learning models. They can derive meaning in clever and sometimes unexpected ways, all under the loose guide of human-defined architecture and algorithm. The fact of the matter is that very little human knowledge is explicitly implanted in these performant models: neural networks achieve a bottom-up understanding of their task from the training data. Indeed, understanding how neural networks actually operate is an active area of research that deserves more attention, as Ali Rahimi points out in his NIPS 2017 talk.

But deep neural networks, for all their successes in the recent past, have a significant weakness. It turns out that these very sophisticated models can be fooled into making dramatically incorrect results. Consider the figure below: to any normal human being it seems like the image is simply random noise. And indeed, the image below was created in part by selecting random values for each of the pixels.

Just some harmless random noise; that is, unless you're a neural network.

A deep neural network, trained on the ImageNet dataset of over a million images across one thousand categories, might categorize this image as something innocuous like a prayer rug. Furthermore, it would likely give it a low confidence rating, suggesting that the neural network isn’t exactly sure what the image contains, but it might be a prayer rug.

However, the Inception network pretrained on ImageNet classified this image as a school bus with 93% confidence. Inception is normally about 70% accurate on normal images, so what went wrong? Is this just a fluke? The reality is that no: results like this are consistently reproducible for even the most advanced and accurate deep neural networks, and they form the basis of a persistent weakness in deep learning, known as adversarial examples.

What are adversarial examples?

Adversarial examples lack a formal definition, but they can broadly be considered as inputs that cause otherwise performant machine learning models to produce very inaccurate results. Note that normal inputs can be incorrectly interpreted by a model without necessarily being adversarial.

Some of the most interesting adversarial examples are those that come from otherwise normal (and correctly handled) inputs. Take for instance the two pictures below. On top, the image is perfectly normal, and the Inception model classifies it accurately as school bus with 95% confidence. However, we can manipulate the image ever so slightly by introducing minute perturbations to the image and create the picture on the bottom. The two are almost indistinguishable, but not to our Inception model. The new image is classified as an ostrich with over 98% confidence.

A school bus or an ostrich? On top, the normal image is correctly classified, with high confidence. But, by slightly changing the pixels in the image here and there (bottom), we can trick the model into thinking this is an ostrich with almost complete confidence.

And these kinds of results can be replicated with nearly any image. So while deep neural networks are ostensibly very well equipped to manage normal data, when that input can be manipulate, even minutely, the classifier can be dramatically fooled.

How adversarial examples are created

When they were first presented, “crafting” (the process of taking a normal input and transforming it to become adversarial) was construed as an optimization problem. Given an input and a classifier, find a perturbation (subject to constraints) that maximizes the error when combined to the original input and fed through the neural network. From this loose framework, we can apply known non-convex optimizers to solve the problem, such as L-BFGS which was done in the paper.

While this technique is effective, it is also somewhat slow. L-BFGS can have a hard time converging with deep neural networks, since its a second-order optimization method. In recent years, a number of alternative attack methods have been proposed in the literature, but in this post we will only examine one in detail.

One of the simplest and fastest methods for crafting adversarial examples relies on utilizing the gradient that is crucial for training the network. With a single backwards pass through the network, we can collect all the information necessary to strike a remarkably effective attack. This method is known as the Fast Gradient Sign Method, or FGSM.

An example of crafting and adversarial input using the Fast Gradient Sign Method (FGSM). By backpropagating to the original image, we can determine which direction each pixel should move to increase the error in the prediction. We then take a small step in the sign of that direction, to produce an image that completely fools the target model.

The FGSM attack can be characterized by the following equation:

\[x_{adversary} = x + \epsilon \cdot \text{sign}(\nabla_x J(x, y; \theta))\]

where $\epsilon$ is the attack strength and $J$ is our cost function for training the model. Since $J$ is a function that determines how “wrong” our model is after making a prediction, taking the gradient with respect to our input ($\nabla_x J$) tells us how modifying $x$ will change how correct our model’s prediction is. By taking the sign we only concern ourselves with direction and not magnitude. Then we multiply these gradients by our attack strength $\epsilon$, which is constrained to be small so that the resulting adversarial image is similar to the original input. Finally, we combine the calculated perturbation with the original image by simple addition.

While the math may be somewhat intimidating, the reality is that the FGSM attack is both very simple to program, and extremely efficient to execute. Here’s a simple Python function that calculates adversarial examples for a TensorFlow model:

def fgsm(model):
    grad, = tf.gradients(model.loss, model.x)
    grad_sign = tf.sign(grad)
    adv_x = model.x + FLAGS.eps * grad_sign
    return adv_x

The function takes a classifier, which can be any object that has an input tensor model.x and a loss function model.loss, and returns the adversarial example crafted from the initial input. The process follows the description above almost exactly, and is easy to see how cheap the computation is (the most expensive portion is the backwards pass in tf.gradients()).

FGSM is simple and fast, but is it effective? I trained a simple, three layer neural network on the MNIST dataset of handwritten digits. Even without convolutional layers, I was able to achieve a test accuracy of 97%. Using the above code, the adversarial example accuracy of the network was less than one percent. This was possible with an attack strength of just 0.1, which meant that no pixel was modified by more than 10%. Below is a sample of an adversarial example the above method produced, which my network classified as a 3.

Result of an FGSM attack on an MNIST handwritten digit example. While clearly still a 5, my network (which has normal test accuracy of 97%) thought this image was a 3. Can you see where FGSM manipulated the image?

Adversarial examples outside of image classification

Adversarial examples are typically studied in the domain of computer vision, and typically with the task of object classification (given an image, output what is inside the image). However researchers have demonstrated that this weakness of deep neural networks infects other kinds of applications as well. Reinforcement learning involves teaching “agents” or programs how to perform tasks or play games intelligently. RL algorithms often employ deep neural networks to manage the huge number of possible environment states and action strategies. As such, they can be fooled by manipulating the data that is fed to the agent, causing the agent to take incorrect and disadvantageous actions.

Systems that try to comprehend passages of text can also be fooled. Since text comprehension is less well understood than image recognition, these systems don’t necessarily need that intelligent of manipulations to be fooled. Models that can answer basic questions about a paragraph can be completely thrown off by the addition of a single, irrelevant sentence. The image below depicts one such situation.

By adding a single irrelevant sentence, reading comprehension systems can give very incorrect answers to the contents of a paragraph. Taken from "Adversarial Examples for Evaluating Reading Comprehension Systems" (https://arxiv.org/abs/1707.07328)

Another susceptible application within computer vision is image segmentation. In some cases, it is insufficient to simply describe what is in an image: we need to know where it is in the image (for instance, self-driving cars). Here again, adversarial examples can cause devastating effects, such as drawing pictures of minions in the segmentation map.

Conclusions and implications

Adversarial examples are a humbling weakness in a time where deep learning seems capable of solving all the world’s problems and paving the road for general intelligence. Their existence forces researchers to reevaluate the precise workings of deep neural networks, and impose a new fundamental understanding about the robustness of deep learning.

From a practical standpoint, adversarial examples all but cripple deep learning from being legitimately used in sensitive applications. Indeed, adversarial examples pose extremely serious threats to systems that rely on deep learning. How can anyone responsibly employ facial recognition for authentication, when 3D printed glasses can make you appear like someone else? The same goes for financial trading, self-driving cars, and defense applications.

Although a flood of research has been produced into understanding, exploiting, and defending against adversarial examples, no known technique for properly defending against adversarial examples exists. Even some of the most recently proposed defense methods accepted to ICLR 2018 were able to be bypassed by smarter attacks.

The result is a looming cloud over all progress in deep learning, which resembles a large portion of recent progress in artificial intelligence generally. So far, matter how advanced our models become, they can be always be fooled by relatively cheap method that would never confuse a human. Does a simple defensive scheme exist that will solve the issue? Or are adversarial examples inherent to the current deep learning paradigm, and will persist until more sophisticated learning techniques that go beyond deep learning are discovered?

Dynamic Routing Between Capsules

2017-11-16T00:00:00+00:00

Convolutional neural networks have dominated the computer vision landscape ever since AlexNet won the ImageNet challenge in 2012, and for good reason. Convolutions create a spatial dependency inside the network that functions as an effective prior for image classification and segmentation. Weight sharing reduces the number of parameters, and efficient accelerated implementations are readily available. These days, convolutional neural networks (or “convnets” for short) are the de facto architecture for almost any computer vision task.

However, despite the enormous success of convnets in recent years, the question begs to be asked, “Can we do better?” Are there underlying assumptions built into the fundamentals of convnets that makes them in some ways deficient?

Capsule networks are novel one architecture that attempts to supersede traditional convnets. Geoffrey Hinton, who helped develop the backpropagation algorithm and has pioneered neural networks and deep learning, has talked about capsule networks for some time, but until very recently no work on the idea had been publicly published. Just a few weeks ago, Dynamic Routing Between Capsules by Sara Sabour, Nicholas Frosst and Geoffrey Hinton was made available, explaining what capsule networks are and the details of their functionality. Here, I’ll walk through the paper and give a high level review as to how capsules function, the routing algorithm described in the paper, and the results of using a capsule network for image classification and segmentation.

Why Typical Convnets are Doomed

Before we dive into how capsules solve the problems of convnets, we first need to establish what capsules are trying to solve. After all, modern convnets can be trained for near-perfect accuracy over a million images with a thousand classes, so how bad can they be? While traditional convnets are great at classifying images like ImageNet, they fall short of perfect in some key ways.

Sub-sampling loses precise spatial relationships

Convolutions are great because they create a spatial dependency in our models (see my previous post for a high level overview of convolutions, or these lecture notes for an in-depth explanation of convnets), but they have a key failure. Commonly, a convolutional layer is followed by a (max) pooling layer. The pooling layer sub-samples the extracted features by sliding over patches and pulling out the maximum or average value. This has the benefit of reducing the dimensionality (making it easier for other parts of our network to work with), but also loses precise spatial relationships.

By precise spatial relationships, I mean the exact ways that the extracted features relate to one another. For instance, consider the following image of a kitten:

Just an ordinary, though cute, kitten

A pretrained ResNet50 convnet classifies as a tabby cat, which is obviously correct. Convnets are excellent at detecting specific features within an image, such as the cats ears, nose, eyes, paws, etc. and combining them to form a classification. However, sub-sampling via pooling loses the exact relationship that those features share with each other: e.g. the eyes should be level and the mouth should be underneath them. Consider what happens when I edit some of the spatial relationships, and create a kitten image more in the style of Pablo Picasso:

A slightly less ordinary kitten

When this image is fed to our convnet, we still get a tabby classification with similar confidence. That’s completely incorrect! Any person can immediately tell by looking at the image something isn’t right, but the convnet plugs along as if the two images are almost identical.

Convnets are invariant, not equivariant

Another shortcoming of typical convnets is that the explicitly strive to be invariant to change. By invariant, I mean that the entire classification procedure (the hidden layer activations and the final prediction) are nearly identical to small changes in the input (such as shift, tilt, zoom). This is effective for the classification task, but it ultimately limits our convents. Consider what happens when I flip the previous image of the kitten upside-down:

The view of a kitten if you were hanging from the ceiling

This time, our ResNet thinks our kitten is a guinea pig! The problem is that while convnets are invariant to small changes, they don’t react well to large changes. Even though all the features are still in the image, the lack of spatial knowledge within the convnet means it can’t make head or tail of such a transformation.

Rather than invariance that’s built into traditional convnets by design in pooling layers, what we should really strive for is equivariance: the model will still produce a similar classification, but the internal activations transform along with the image transformations. Instead of ignoring transformations, we should adjust alongside them.

A note on sub-sampling

Before we proceed, I feel it necessary to point out that Hinton identifies these problems with sub-sampling in convnets, not the convolution operation itself. The sub-sampling in typical convnets (usually max-pooling) is largely to blame for these deficiencies in convnets. The convolution operation itself is quite useful, and is even utilized in the capsule networks presented.

Capsules to the Rescue

These kinds of problems (lack of precise spatial knowledge and invariance to transformations) are exactly what capsules try to solve. Most simply, a capsule is just a group of neurons. A typical neural network layer has some number of neurons, each of which is a floating point number. A capsule layer, on the other hand, is a layer that has some number of capsules, each of which is a grouping of floating point neurons. In this work, a capsule is a single vector, though later work utilizes matrices for their capsules.

The key idea is that by grouping neurons into capsules, we can encode more information about the entity (i.e. feature or object) that we’re detecting. This extra information could be size, shape, position, or a host of other things. The framework of capsules leaves this open, and its up to the implementation to define and enforce these encoding principles.

There’s a few things we need to be careful about before we can get started using capsules. First, since capsules contain extra information, we need to be a little more nuanced about how we connect capsule layers and utilize them in our network. Typical convnets only care about the existence of a feature/object, so their layers can be fully connected without problem, but we don’t get that luxury when we start encoding extra properties with capsules. We also have to be smart about how we connect capsules so that we can appropriately manage the dimensionality without having to resort to pooling.

Connecting Capsule Layers: Dynamic Routing by Agreement

The central algorithm presented in the capsules paper that came out recently is one that describes how capsule layers can be connected to one another. The authors chose an algorithm that encourages “routing by agreement”: capsules in an earlier layer that cause a greater output in the subsequent layer should be encouraged to send a greater portion of their output to that capsule in the subsequent layer.

The routing procedure happens for every forward pass through the network, both during testing and training. The image below visually describes the effect of the routing procedure.

A visual description of the before and after of the routing procedure. Arrow widths correspond to the strength of the output. The lines coming out of the capsules in the second layer correspond to the portions of the output that come from the capsule in the previous layer.

Before the routing procedure, every capsule in the earlier layer spreads its output evenly to every capsule in the subsequent layer (initial couplings can be learned like weights, but this isn’t done in the paper). During each iteration of the dynamic routing algorithm, strong outputs from capsules in the subsequent layer are used to encourage capsules in the previous layer to send a greater portion of their output. Note how caps_21 has a large portion of its output influenced by caps_11 (denoted by the thick arrow coming out on top). After the routing procedure, caps_11 sends much more of its output toward caps_21 than any of the other capsules in the second layer.

The mathematical details of this procedure are excellently explained in the original paper, but for brevity I will omit a complete explanation.

CapsNet: A Shallow Network with Deep Results

Now let’s look at the actual capsule network utilized int the paper, known as CapsNet.

The CapsNet architecture

Our input images are the MNIST data set: 28x28 grayscale pictures of handwritten digits (later we’ll see how capsules perform on other, more complex data sets). These get fed into a convolutional layer (256 9x9 filters, stride of 1 and no padding). That layer passes through another set of convolutions to become the PrimaryCaps layer, the first capsule layer. Each capsule inside PrimaryCaps is a size 8 vector. There are 32x6x6 = 1152 of the capsule in the PrimaryCaps layer. In the paper, they construct their capsule architecture such that the length of a capsule (i.e. the value after putting the vector through the Euclidean norm) represents the likelihood an entity exists, and the orientation (i.e. how the values are distributed within a capsule) represents all other spatial properties of the entity.

The PrimaryCaps layer is connected to the DigitCaps layer. This is the only place in the CapsNet architecture where routing takes place. DigitCaps is a layer of 10 capsules (one corresponding to each potential digit), each a size 16 vector. Since the length of the vector corresponds to its existence, all we need to do is norm the capsules in DigitCaps to get our logits, which can be fed into a softmax layer to get prediction probabilities.

Since each digit could appear independently, and in some tests multiple digits, the authors used a custom loss function that penalized each digit independently during training. Additionally, they wanted to ensure that each DigitCaps was learning a good representation within the capsule. To do this they appended a fully connected network to function as a decoder and attempt to reconstruct the original input from the DigitCaps layer.

Reconstructing the original input from the DigitCaps layer

Only the capsule value from the correct label was used to reconstruct (the others were masked out). The reconstruction error was used to regularize CapsNet during training, forcing the DigitCaps layer to learn more about how the digit appeared in the image.

Results

With only three layers, the CapsNet architecture performed remarkably well. The authors report a 0.25% test error rate on MNIST, which is close to state of the art and not possible with a similarly shallow convnet.

They also performed so experiments on a MultiMNIST data set: two images from MNIST overlapping each other by up to 80%. Since CapsNet understands more about size, shape, and position, it should be able to use that knowledge to untangle the overlapping words.

Reconstructing from severely overlapping digits

The image above represents CapsNet reconstructing its predictions twice, one for each digit in the image. The red corresponds to one digit, and green for another (yellow is where they overlap). CapsNet is extremely good at this kind of segmentation task, and the authors suggest is in part because the routing mechanism serves as a form of attention.

CapsNet is also performant on several other data sets. On CIFAR-10, it has a 10.6% error rate (with an ensemble and some minor architecture modifications), which is roughly the same as when convnets were first used on the data set. CapsNet attain 2.7% error on the smallNORB data set, and 4.3% error on a subset of Street View Housing Numbers (SVHN).

Conclusion

Convnets are extremely performant architectures for computer vision tasks. Their resurgence has marked the recent AI renaissance currently unfolding with the advent of deep learning. However, they suffer from some serious flaws that make them unlikely to take us all the way to general intelligence.

Capsules are a novel enhancement that go beyond typical convnets by encoding extra information about detected objects and retain precise spatial relationships by avoiding sub-sampling. The simple capsule architecture presented in this paper, CapsNet, is able to get incredible results considering its small size. Additionally, CapsNet understands more about the images its classifying, like their position and size.

Although CapsNet doesn’t necessarily outperform convnets, they are able to match their accuracy out-of-the-box. This is really promising for the future role of capsules in computer vision. There’s still a huge amount to research to be done into improving capsules and scaling them to larger data sets. As a computer vision researcher, this is an extremely exciting time to be working in the field!

Simplifying Deep Learning Programming with Keras

2017-11-01T00:00:00+00:00

Last post, I gave an introduction into programming a deep neural network with TensorFlow. The model worked quite well (98% accuracy on the test set) with only 150 lines of code, but it was arguably a bit complex.

The problem was we had to really dig into the nitty-gritty details of how we wanted our model to work. But a lot of times, we do not need to deal with that level of detail and the complexity that comes with it. This kind of problem occurs often in software engineering, and it is generally solved with a convenient library.

Keras is such a library: it does a great job of taking the complexity out of building a neural network, so you can focus on the interesting parts of training and utilizing the model. In this post I’ll walk through some of the basics of Keras and we will rebuild our MNIST handwritten-digit classifier in a much simpler program.

Keras: The Model Abstraction

In Keras, the fundamental abstraction is the Model object. We can design, train, and evaluate the Model without necessarily knowing the exact details. In this example, TensorFlow will be the backend that Keras will utilize behind the scenes, but Keras can actually function agnostic of its specific backend and run with TensorFlow, Theano, or CNTK.

There are a few different model types, but the one we will utilize is the Sequential model. The Sequential model view the network architecture as a sequence of layers strung together, one after another. This is exactly the architecture we used in our previous convolutional neural network. With Keras, we can stack our network layers like individual building blocks to create our overall model.

from keras.models import Sequential
model = Sequential()

Adding Layers

To add new layers to a Keras model, we simply call the add() function and pass in the layer we want to use. To recreate our previous convnet, we’ll need main kinds of layers: Dense for our fully connected layers, and Conv2D for our two dimensional convolutional layers. We’ll also need MaxPool2D and Dropout layers to utilize max pooling and dropout. Finally, a Flatten layer will be used to convert between our convolutional and fully connected layers.

from keras.layers import Dense, Conv2D, Dropout, Flatten
model.add(Conv2D(32, (5, 5), padding='same'), input_shape=(28, 28, 1))
model.add(MaxPooling2D(padding='same'))
model.add(Conv2D(64, (5, 5), padding='same'))
model.add(MaxPooling2D(padding='same'))
model.add(Flatten())
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))

There’s a few things we should not here. The first layer we add() needs to take an additional argument: input_shape. This tells Keras the size of the inputs that we will feed into our model (in our case, a 28x28 pixel image for MNIST). For the Conv2D layers, the first argument represents the number of filters, followed by the dimensions of our convolution. The Dense layers take an argument that represents the number of neurons in that layer. We can also specify the activation function we want to use by a keyword argument, as we did here. Alternatively, we could have added an Activation layer.

Compiling and Training the Model

Now that we have defined what our convnet will look like by stacking all of our layers into our model, we can get ready to start training our model on the data set. However, first we need to compile the model. Since Keras serves as a high-level wrapper of other machine learning libraries, it needs to convert our Keras-defined model into a model of our backend. Additionally, we will need to specify some other attributes of our training procedure.

from keras.optimizers import Adadelta
model.compile(optimizer=Adadelta(),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

Here we specify that training will use the Adadelta optimizer, our loss function is defined by the cross entropy of the output (since this is a classification task), and we want to optimize over the accuracy of the model.

Next, we can get our training data ready. Luckily, Keras even has some common data sets built in.

from keras.dataset import mnist
from keras.utils import to_categorical
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Convert to values between 0. and 1.
x_train = x_train.reshape(-1, 28, 28, 1).astype('float32') / 255.
x_test = x_test.reshape(-1, 28, 28, 1).astype('float32') / 255.
y_train = to_categorical(y_train, num_classes)
y_test = to_categorical(y_test, num_classes)

For the inputs, we need to convert the arrays into the right shape, and scale the values between $[0, 1]$. The outputs get converted to binary one-hot vectors by Kera’s to_categorical utility function.

Now we can finally train our model. This is done by the Model’s fit method:

model.fit(x_train, y_train, epochs=10, batch_size=50)

The epochs argument will determine how many passes through the data training will make. The batch_size determines how many samples to train with for each weight update. Keras will output its progress as it works, updating you on which epoch is running, approximately how long it will take, and the current loss in the model.

Evaluating the Results

Once our model is trained, we can see how accurate it is at predicting on novel data. To see how our model stacks up against the test set, use the evaluate method:

loss, accuracy = model.evaluate(x_test, y_test)
print('Accuracy of {:.2f}%'.format(accuracy * 100))

So in just a few lines of Python, we were able to create a high performing MNIST classifier! Using Keras is really straightforward, and allows us to avoid the nitty-gritty details of programming complex deep neural networks. Instead, we can work on other interesting aspects of our models and keep the implementation from hindering our ideas. And when Keras is too high level, we can even use it as a simplified interface to TensorFlow. As a deep learning researcher, Keras takes a lot of the hassle out of programming deep neural networks.

Getting Started with Tensorflow

2017-09-12T00:00:00+00:00

Note: All the code from this post can be found here. This tutorial is adopted from Google’s own TensorFlow tutorial, “Deep MNIST for ML Experts”

My last post talked about deep learning very generally, describing the fundamentals of how deep neural networks work and are used. In this post, we’ll look more concretely at actually building a convolutional neural network to classify handwritten digits from the MNIST data set. Using Google’s popular machine learning library TensorFlow, we’ll have a model that gets over 98% accuracy with about 150 lines of code and 10 minutes of training time on my laptop.

Prerequisites

This tutorial assumes that the reader has a basic familiarity of programming in Python. If you’ve never seen the syntax before, it is pretty easy to pick up. I also assume that you have TensorFlow installed on you machine. For the sake of brevity, much of the fundamentals of deep learning are omitted, but you can learn about those in my previous blog post.

What is TensorFlow?

TensorFlow is a high performance numerical computing library developed by Google. It generally supports any kind of scientific computation, but was developed specifically with machine learning in mind. TensorFlow has lots of packages that make building and running deep and even distributed neural networks much simpler than before. It was open sourced by Google in 2015.

Before TensorFlow’s arrival, many of the machine learning libraries in use were developed by research labs to support their needs. While these libraries were great, they often lacked strong software engineering expertise and failed to meet enterprise scale needs. Thankfully, TensorFlow was developed from the ground up by experts in both of these domains. TensorFlow is the product of choice that Google, a machine learning leader, uses in many of their products.

Another great aspect of TensorFlow is that the models are portable. A TensorFlow program trained to run on a rack of servers can be deployed to execute on a smartphone. As we will discuss in a moment, this is because TensorFlow builds computational graphs that can be stored independently of the program that developed them. The parameters can be trained, and then the model shipped off to run in production. TensorFlow can also utilize specialized hardware like graphics processors without any explicit programming by the end user.

Programming in TensorFlow

TensorFlow was originally developed in C++, but language bindings for Python are the most common way people program their machine learning applications. TensorFlow operates by building a computational graph that can be executed. This differs slightly from typical imperative programming, where each statement is explicitly executed line by line. Instead, TensorFlow has us describe how to make certain calculations, and then when we want, we can evaluate them against a session, feeding in any potential inputs. Let’s look at an example

import tensorflow as tf
sess = tf.Session()

x = tf.placeholder(tf.float32)
y = x * x
z = y + 100

sess.run(z, feed_dict={x:42})

# >>> 1864.0

First, we defined x as a placeholder: this is essentially an input to our computational graph. Then we told TensorFlow how to calculate y and z. Finally, we told TensorFlow to actually calculate the value of z, populating the value for our placeholder x.

This was a pretty simple example, but it illustrates the basic mechanics of programming in TensorFlow. However, let’s add some complexity. TensorFlow derives its name from the tensor: a mathematical generalization of a matrix. Tensors are kind of like arbitrarily high-dimensional arrays, and TensorFlow excels at working with these constructs. For instance, if we have 100 images, each 28 by 28 pixels in size, with three values for the red, green, and blue, we can represent all that data as a single 100x28x28x3 tensor. Let’s try an example of programming with tensors through matrix-vector multiplication.

# Matrix-vector multiplication
A = tf.placeholder(tf.float32,
                   shape=[3, 3])
b = tf.placeholder(tf.float32,
                    shape=[3, 1])
z = tf.matmul(x, y)

sess.run(z, feed_dict={
  A:[[1, 2, 3], [4, 5, 6], [7, 8, 9]],
  b:[[1], [2], [3]]
})

# >>> [[14.],[32.],[50.]]

TensorFlow has lots optimized implementations of common operations like matmul. These are really helpful when building deep neural networks that are blazingly fast.

Before we start building a deep neural network, we need to introduce the idea of TensorFlow variables. Variables are dynamic values that are global to a TensorFlow session. Generally these are used as the parameters in models that are tuned during training. Although we won’t directly manipulate them, the optimization procedure that we utilize will. Before we can start using our variables in a session, though, we will need to execute sess.run(tf.global_variables_initializer()) in our program.

Building an MNIST classifier

Now let’s get started building an MNIST image classifier. The MNIST is a common data set of 28x28 pixel images of handwritten digits. They were scraped from tax forms, then centered on the image.

The goal of our model is to input these images as arrays of pixels and learn how to derive which digit is displayed in the image. This is called classification, since each image has to fall within 10 categories (the number 0 to 9).

To accomplish this feat, we’re going to utilize a deep convolutional neural network. We’ll have a total of 4 layers: the first two convolutional, the last two fully connected. To prevent overfitting, we’ll utilize dropout. And finally, our outputs will be converted into a probability distribution via the softmax functions. It’s not necessary that you fully understand all the details; these are just common practices within deep machine learning.

To get started, we’re going to build some helper methods that will make constructing our model a little less tedious. The code presented in this tutorial will be somewhat out of order, but it should run fine when combine (remember, all of the code in this tutorial can be found here, with some additional features).

# Create some random variable weights
def weights(shape):
  return tf.Variable(tf.truncated_normal(
      shape, stddev=0.1))

# Create a “constant” bias variable
def bias(shape):
  return tf.Variable(tf.constant(
      0.1, shape=shape))

# Hardcode our convolution parameters
def conv2d(x, W):
  return tf.nn.conv2d(x, W,
      strides=[1, 1, 1, 1], padding='SAME')

# Same with our maxpooling
def max_pool(x):
  return tf.nn.max_pool(
      x,
      ksize=[1, 2, 2, 1],
      strides=[1, 2, 2, 1],
      padding='SAME'
    )

The first two methods define the initialization for weights and constants that we’ll use in our model. The last two methods hardcode some of our parameters for our convolutional operations.

Next, we’ll actually define our model. We will encapsulate it as a method so as to keep our main method a bit cleaner. The model will take a tensor input for an argument and return it’s output predictions (as well as the dropout probability used, though that’s not significant).

def cnn(x):
  keep_prob = tf.placeholder(tf.float32)
  imgs = tf.reshape(x, [-1, 28, 28, 1])

  # Layer 1
  W_conv1 = weights([5, 5, 1, 32])
  b_conv1 = bias([32])

  z_conv1 = conv2d(imgs, W_conv1) + b_con1
  h_conv1 = tf.nn.relu(z_conv1)
  h_pool1 = max_pool(h_conv1)

  # Layer 2 (convolutional)
  W_conv2 = weights([5, 5, 32, 64])
  b_conv2 = bias([64])

  z_conv2 = conv2d(h_pool1, W_conv2) + b_conv2
  h_conv2 = tf.nn.relu(z_conv2)
  h_pool2 = max_pool(h_conv2)

  # Layer 3 (fully connected)
  W_fc1 = weights([7*7*64, 1024])
  b_fc1 = bias([1024])

  h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])

  z_fc1 = tf.matmul(h_pool2_flat, W_fc1) + b_fc1
  h_fc1 = tf.nn.relu(z_fc1)
  h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

  # Layer 4 (fully connected, last hidden layer)
  W_fc2 = weights([1024, 10])
  b_fc2 = bias([10])

  z_fc2 = tf.matmul(h_fc1_drop, W_fc2) + b_fc2
  h_fc2 = tf.nn.relu(z_fc1)

  return h_fc2, keep_prob

For each layer, we define our weights and biases (TensorFlow won’t reinitialize these during execution), and perform our operation before moving on to the next layer. For the convolutional layers, this involves applying the actual convolution followed by a maxpooling to decrease the dimensionality. For the fully connected layers, the operation is a matrix vector multiplication with the weights, followed by an ReLU activation (and dropout in the second to last layer). When switching from convolutional to fully connected layers, we needed to reshape our data.

Now that our model has been established, we can program our training procedure. When training a deep neural network, we typically feed in some data, measure the error, and adjust the parameters so as to decrease the error. Our input data and labels will be placeholders, and we can measure the error using TensorFlow’s built-in cross entropy operation on the softmax of our model outputs. Then we can program that our optimization step (i.e. weight adjustment) should use the Adam optimizer, which is an enhancement of stochastic gradient descent. While we’re at it, we’ll also tell TensorFlow how to measure our accuracy.

def main(_):
  mnist = input_data.read_data_sets(
    FLAGS.data_dir, one_hot=True)
  
  x = tf.placeholder(tf.float32,
                     [None, 784])
  y_true = tf.placeholder(tf.float32,
                               [None, 10])
  y_hat, keep_prob = cnn(x)

	# Cross entropy measures the error in our predictions
  cross_entropy = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(
      logits=y_hat, labels=y_true))
  # Once we have error, we can optimize
  training_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

  # Define a “correct prediction” to calc accuracy
  correct_prediction = tf.equal(tf.argmax(y_hat, 1),
      tf.argmax(y_true, 1)
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

Now we can write the actual loop that will run the training procedure.

  # inside main()
  with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(EPOCHS):
      batch = mnist.train.next_batch(MINIBATCH_SIZE)
      train_step.run(feed_dict={
        x:batch[0],
        y_true:batch[1],
        keep_prob:0.5
      })

  test_acc = accuracy.eval(feed_dict={
    x:mnist.test.images,
    Y_true:mnist.test.labels,
    keep_prob:1.0
  })

  print('Test accuracy is {:2f}%'.format(test_acc * 100))

Finally, we can put the finishing touches so that our program will run. Outside of any function, at the bottom of the file write:

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

if __name__ == '__main__':
  EPOCHS = 5000
  MINIBATCH_SIZE = 50

  tf.app.run(main=main)

On my 2013 MacBook Pro, I was able to run this program in about 5-10 minutes, and achieved 98% accuracy. That’s pretty amazing! All of this code, including some enhancements like model saving, can be found at this link.

What is deep learning?

2017-09-09T00:00:00+00:00

Deep learning is a subfield of machine learning that has had remarkable research success in the past decade. Huge numbers of research groups and top software companies are pushing the boundaries of what was previously thought possible through computation with these advancements.

One of my favorite things about deep learning is that, despite the hype and number of PhD’s who work in the field, it is actually relatively simple to get started. Most common laptops are powerful enough to train and run simple deep models. This is in part due to the growing amount of open source machine learning libraries that make building deep models significantly simpler than rolling them by hand. In this post we’ll walk through some of the fundamentals of deep learning and the historical background. This foundations should provide enough context to start digging into deep learning and building your very own models.

Getting started

Deep learning is one specific category within the general field of artificial intelligence. At its core, it leverages learning models that are artificial neural networks with lots of layers. These models are general purpose, and similar architectures can be trained to run a variety of tasks. They “learn” by adjusting parameters, of which deep neural nets can have billions. We can think of these parameters as lots of dials on a big, complicated machine. By discovering the right combination of dial settings, we can configure the machine to properly accomplish a particular task.

Neural networks are layers of groups of neurons that feed into each other. We’ll discuss more about the neurons in a moment, but suffice it to say that they perform a relatively simple mathematical transformation. Every neuron in a layer receives its input from every neuron in the layer before, and sends its output to every neuron in the next layer. There are three different kinds of layers in a typical network: input, hidden, and output. The input layer is the first layer in our network, and it is just the input for the task we’re trying to solve (e.g. each neuron corresponds to a pixel within an image). The output layer is what our model will ultimately result in for a given input. It may just be a single number or, if our model is dealing with multiple categories, it may be multiple outputs that we can combine in some meaningful way. Hidden layers are where all of the interesting things happen. They contain all the parameters in our model that are used during the mathematical transformation. By tuning the parameters in the hidden layer neurons, we train our network. Deep neural networks get their namesake by generally having lots of hidden layers.

A quick history of deep learning

Artificial neural networks were originally theorized in the late 1950’s and early 1960’s, and derive their name from the loose inspiration of how actual neurons within our brain work. Although the concept is almost as old as artificial intelligence itself, neural networks did not receive much attention until the 1980’s. By then, researchers discovered how to increase the size of these networks without dramatically increasing the training complexity. However, they again fell out of vogue due to their inability (at the time) to outperform other machine learning algorithms.

All of this changed in 2006, when deep learning was born. Huge advancements in computer hardware and some algorithmic improvements permitted researchers to build neural networks with huge numbers of layers (i.e. “deep”) and train them to beat other kinds of machine learning models. This marked the start of the era of deep learning, where these models have been adapted and modified to perform amazingly well in a wide variety of complex tasks. Many times, they can even outperform humans.

Going deeper: What’s in a neuron?

Neurons within a deep learning network perform a mathematical transformation of their input that is determined by some parameters, or weights. There’s a lot of variety within the general model, but most models do a linear combination of their weights and the input (easily calculated by a matrix-vector multiplication), followed by some nonlinear “activation function”. The activation function is important to make our models learn nonlinear data, and several kinds exist. In equation form, we can describe what happens within a neuron by

\[h = \sigma(x_1\theta_1 + x_2\theta_2 + \cdots + x_n\theta_n)\]

where $x_n$ is the input, $\theta_n$ are the weights, and $\sigma()$ is the activation functions. The most popular activation functions, called rectified linear unit (ReLU), defined as $\sigma(z) = \text{max}(0, z)$.

Training a neural network: Optimization and backpropagation

By tuning the model parameters, we can teach our neural network to perform a task. But how do we know how to adjust those parameters? This is particularly difficult in deep models that are complex and have parameters on the order of billions.

The way most deep models are trained follows an optimization procedure. When a model produces a result, we can define an error function (also known as a cost function) that measures how incorrect our model is. Then we can define the training procedure as an optimization problem: we want to find the model parameters such that the error on our training set is minimized.

To actually perform this minimization, most deep learning models make use of an optimization algorithm known as stochastic gradient descent (SGD). This algorithm repeatedly approximates the gradient of the error function, and then slightly moves the parameters in the direction that will decrease the error. The gradient can be calculated through a procedure known as backpropagation. Essentially, the gradient gives us an idea of how much blame to assign to ever parameter in our network for a prediction. By calculating the blame for a number of inputs, we can approximate how the parameters are affecting the model’s accuracy generally. Then, we can adjust the parameters such that they will hopefully make our model more accurate. By repeating this enough times, we can train our neural network by tuning all the parameters.

Deep learning variations

Vanilla neural networks can be useful, but we can get even better results if we modify the model to utilize some inherit characteristics of our objective task. For instance, within images, the pixel data often has a high spatial dependence, meaning that pixels close together often work together to make out specific attributes within an image. We can exploit this fact by utilizing a convolutional neural network. Without getting into the details, convolutional neural networks (also known as convnets) replace some of the early layers with neurons that perform a convolutional operation. This combines areas of a picture using math and lets us extract spatial information from our inputs.

Another variation of the typical (or feedforward) neural network involves making use of temporal dependencies, or when the correct output depends on multiple inputs spaced out over time. Temporal dependencies are really common, for instance speech recognition and text comprehension. To exploit these dependencies, models known as recurrent neural networks (RNNs) are typically used. These are neural networks where some of the neurons feed back into previous layers, creating a cycle. When inputs are sequentially fed into the network, part of the calculations will depend on not only what the network currently sees, but what it has previously seen. The mechanisms for when and how to feed back into the network can get pretty advanced, and can even mimic our understanding of how our own brains work, such as the long short-term memory (LSTM) neuron.

Deep learning has also significantly improved other areas of machine learning. A great example of this is within reinforcement learning, where models are “agents” that try to perform a specific task like win a game against an opponent or traverse a robot through a maze. Although the algorithms required to perform reinforcement learning don’t necessitate deep neural networks, they have benefited from the accuracy and generalization abilities of these models. For instance, most of the interesting problems in reinforcement learning have a huge search space, meaning that the number of possible states (e.g. chess positions, robotic sensor data) the agent could be in is enormous and impossible to completely exhaust during training. However, the agent will still need to take actions in these states, even though it has not seen them before. To solve this, we can represent the state space with a deep neural network, which can learn what “kind of” state its in, and approximate it feasibly for the agent. Other calculations the agent performs can be similarly approximated. Because deep networks generalize so well (meaning they give similar outputs for similar, though distinct, inputs), the agent can encounter entirely novel situations and still act appropriately because it has learned what to do in similar states.

Conclusion

Deep learning has moved the field of machine learning and artificial intelligence dramatically forward and into the forefront of our technological future. However, despite all the hype, it turns out that deep learning is really some manageable mathematics and algorithms, refined and improved over a half century or so. Ultimately, the models will keep improving and expanding their capabilities beyond even what we’ve seen so far. Now is an amazing time to jump in on deep learning, to both discover more about the nature of intelligence and leverage our existing knowledge to improve the world we live in.

Clustering with K-Means

2016-12-06T00:00:00+00:00

K-Means is a classic machine learning algorithm for discovering clusters within data sets. It is a form of unsupervised learning, where the algorithm is not privy to the structure of the data, but is in fact trying to learn said structure. My last blog post, concerning linear regression, was an example of supervised learning, where the algorithm attempts to predict values and with a clear right or wrong answer. Here, we will examine the K-Means algorithm that identifies data points that have grouped together, or clustered, within the data set.

Clustering occurs frequently in natural data, and discovering these groupings can be very advantageous to a data scientist. Although searching for clusters is typically an unsupervised task, their existence permits us to seek new information about the data set and attempt to make predictions about novel data. This kind of application is commonly employed by companies seeking to recommend new products or services for their customers. If Netflix knows that people who love movie A also love movie B, then they will likely suggest movie B to you after you view movie A.

The Setup

We will begin with the following scenario: imagine that we collected a survey from a number of participants, asking them to rate their preference of Cola C and Cola P. If we plot these ratings on a two-dimensional coordinate plane, the results might look something like this:

It is clear that the results have clustered around certain areas of the graph. Some people mostly prefer Cola C, others just Cola P, and some do not particularly care for either.

Our goal will be to identify these groups mathematically. We will do this by assuming that each cluster has some “center” point that the data points in that cluster gather around. The K-Means algorithm will give us a way to identify where these centers are for each of the clusters in our data set.

Note that estimating these centers is inherently an unsupervised task. We do not know where the centers are in the data, we want the algorithm to discover them. A supervised task would involve trying to match data to a known structure. Here, we are trying to learn the structure of the data itself.

The Algorithm

The K-Means algorithm is relatively straightforward and simple mathematically. Unlike gradient descent, no partial derivatives are necessary; just basic addition and division.

We begin by assuming there are a specific number of clusters in the data set: $k$. This hyperparameter may need to be tuned if the data is opaque, but in our case a value of 3 is obviously correct.

Each of these assumed clusters must have a center, as discussed earlier. We can initialize these cluster centers randomly. Although incorrect at the outset, their correct positions will be learned as the algorithm progresses.

The algorithm will then proceed as an iteration over two sequential phases, referred to here as assign and adjust.

The assign phase will iterate over ever data point, calculate its distance from each of the current cluster centers, and “assign” itself to the center it is closest to. Here, distance can be measured a number of way (the example below will use the common Euclidean distance). The result will be that every data point will become associated with the cluster center that it best belongs with.

The next phase, adjust, will recalculate the cluster center positions based on which data points were assigned to it in the previous phase. Specifically, the new center position will be the average position of its assigned data points. This average, or “mean”, is where the algorithm receives its name, K-Means, since each of the K clusters will be centered around their average positions.

As these two phases are repeated, the cluster centers will eventually settle in their right position. That is because cluster centers will consistently move towards the data points closest to them, thus making them even closer. Eventually, the centers will no longer move, when they are in their final position, at which point the algorithm can stop.

So long as the number of cluster centers ($k$) matches the number of clusters in the actual data, the algorithm will converge fairly quickly. It should be noted, however, that it is possible for the cluster centers to settle in incorrect positions, even when the value of $k$ is correct. This can occur if clusters are located close to one another, or when the initial values of the cluster centers are initialized poorly. To prevent poor initialization, it is common to run the algorithm multiple times, and compare the results from each execution for consistency.

The Code

For this example, we will use the same data set depicted from our Cola C vs. Cola P survey. We will assume that these data points come as lists, X and Y, corresponding to the $x$ and $y$ values, respectively.

The adjust phase of the algorithm is implemented below. It takes, the X and Y values as parameters, as well as a list of the locations of the centers (each center is a pair, i.e. centers[0] might be (22, 19)). The function will return a list, nearests, which will contain a list of the assigned data points for each center.

def assign(X, Y, centers):
    nearests = [list() for c in centers]
    for (x, y) in zip(X, Y):
        dists = [math.sqrt((x - cx) ** 2 + (y - cy) ** 2)
                 for (cx, cy) in centers]
        index = dists.index(min(dists))
        nearests[index].append((x, y))
    return nearests

The next function, adjust, take a cluster center position and a list of positions assigned to that center, and return the new position for the cluster center. It calculates the new position as the average $x$ and $y$ values of the assigned positions.

def adjust(center, neighbors):
    if len(neighbors) == 0: # Avoid dividing by zero
        return center
    avg_x = sum([n[0] for n in neighbors]) / len(neighbors)
    avg_y = sum([n[1] for n in neighbors]) / len(neighbors)
    return (avg_x, avg_y)

The main function, which will serve as the program’s entry point, will initialize the cluster center positions, and iterate over the assign and adjust functions until the cluster centers no longer move by any significant amount (the value of EPSILON) in any direction.

def main():
    # ...omitting data collection...
    # Initial center points randomly
    centers = [(randint(0, 100), randint(0, 100)) for i in range(3)]

    iterations = 0
    while True:
        # Move each center to the mean position of its assigned points
        neighbors = assign(X_train, Y_train, centers)
        new_centers = [adjust(c, n) for c, n in zip(centers, neighbors)]

        # Stop if we've converged
        if all([abs(n[0] - c[0]) < EPSILON and abs(n[1] - c[1]) < EPSILON
                for (n, c) in zip(centers, new_centers)]):
            break
        else:
            centers = new_centers
            iterations += 1

The following graphs display the algorithm in action. Initially, the cluster centers are placed at random on the graph.

After the first iteration

And after the second iteration, the cluster centers will settle, or converge on their correct positions.

Conclusion

K-Means is an extremely useful algorithm for discovering correlations within a data set. Even better, the algorithm is efficient and simple to implement (though much better implementations exist in open source machine learning libraries).

An Introduction to Machine Learning

2016-11-10T00:00:00+00:00

If you’ve seen any technology news in the past couple of years, it was probably a headline having something to do with computers able to achieve feats previously thought impossible. Driving cars, diagnosing patients, or beating world champions at complex strategy board games, have all be done. All of these tasks are being done with an exploding branch of computer science called artificial intelligence. In this post I’ll look at a subset of AI called machine learning and breakdown one of the simpler algorithms called linear regression.

Machine learning is a category of algorithms that can “learn” from data. In other words, if they’re given a particular task, $T$, machine learning algorithms can improve at that task (to some limit) given experiences, $E$.

Linear Regression

The algorithm that we’ll look at is called linear regression. At its heart, linear regression is a “line-fitting” algorithm: given some data that seems to form some kind of line on a graph, we want to derive an equation that describes that line. With such an equation, we can make predictions about new values that are outside out initial dataset.

To put it in more concrete terms, let’s say we wanted to build a model that could predict house prices in a certain area. If we plotted the size of the the house (in square feet) compared to the price that the house sold for, we would likely see some kind of linear distribution of the data points. Using linear regression, we could find the equation of that line, and then predict how much money we think a new house will sell for given its square footage.

In reality, most problems like this are much more complex than this simple example. For instance, houses can have dozens of factors (or “features”) that contribute to their selling price, like the number of bedrooms, school district, and so on. Despite this complexity in the real world, we can look past the specific and learn the principles that form the foundation of these complicated models.

Case Study

For our example, let’s say we have the following data:

I generated this data myself with a linear equation. I also added some random noise to make it a little bit more realistic. Our goal will be to find a way to predict where a new data point will fall vertically given where it sits horizontally (in other words, the $y$ value given the $x$ value). We can state this goal as such:

Goal: Create a model that can reasonably predict new values based on the existing data

This is a pretty vague goal, and we’ll refine it as we progress.

You may remember from school something called the slope-intercept equation of a line. If you don’t here’s what it looks like:

\[y = b + mx\]

Where $x$ and $y$ are the point coordinates, $b$ is the $y$-intercept (where the line hits the $y$-axis), and $m$ is the slope of the line.

We can rewrite this equation with some different symbols that are commonly used in machine learning like so:

\[h_\theta(x) = \theta_0 + \theta_1x\]

The function $h_\theta$ is called the hypothesis function, and it serves as our predictor. Given an $x$ value, $h_\theta(x)$ will produce what our model thinks $y$ should be.

Using what we already have said about the slope-intercept equation, we can consider how we might be able to achieve our goal. We said that the $\theta_0$ value determines the $y$-intercept of the line, or where it sits vertically. By increasing or decreasing $\theta_0$, we can shift the line up or down, respectively. Additionally, $\theta_1$ determines the slope of the line, or its angle. By modifying $\theta_1$, we can tilt the line more or less. By combining these two mechanisms, shifting and tilting, we can create the line that resembles the data set.

$\theta_0$ and $\theta_1$ serve as the parameters of our model, since their values will directly determine how well our model can predict existing and new values.

(It should be noted that this is a relatively contrived and simple example with only two parameters, but the same principles apply even if we have more dimensions or even higher-order dimensions like $x^2$ or $x^3$.)

Now we can modify our goal slightly:

Goal: Find the values of the parameters, $\theta_0$ and $\theta_1$, that form a line that best “fits” our data.

But what do I mean when I say a line that “fits” the data? If we just look at a plot of our line on top of the data, we can get an intuitive feel of if the line is “good” or not. But what if our data has many dimensions and can’t easily be visualized? And how can we empirically determine which of two lines are better if they both seem pretty close to the data set?

To solve these problems, we’ll need a tool called the cost function. You may also see it called the error or loss function, but it all refers to the same concept. The cost function will let us “score” our model by determining mathematically how close or far it is from the original data. Many different cost functions exist, but most of them follow some scheme of measuring the distance between a model prediction ($h_\theta(x)$) and the actual value ($y$) for each data point.

For our example, we’ll use the sum of squares cost function, which is a common and effective cost function. To calculate the cost, we will loop over every data point, find the difference between the predicted and actual value, square it, then add up the result for all the data points. Mathematically, it looks like

\[J(\theta_0, \theta_1) = \frac{1}{2N}\sum_{n=1}^{N}(h_\theta(x_n) - y_n)^2\]

We’ll also divide the sum by the number of data points, $N$, which is called normalization. It allows our model’s cost to be independent of the number of the number of data points it was calculated against (i.e. adding more data doesn’t necessarily increase the cost). The division by 2 is somewhat arbitrary and not completely necessary. We do it to make some of the math in the future slightly cleaner.

With the cost function, we now have a way to actually evaluate how good (or rather, bad) our model is at predicting values. Ideally, we would want our parameters ($\theta_0$ and $\theta_1$) to be such that the cost function is equal to 0, but that’s often impossible or even undesirable. So we’ll settle with minimizing the cost function to the lowest value we reasonably can, which means finding the parameter values that produce a lower cost than any other set of values. So let’s update our goal to reflect this new idea

Goal: Find the values of the parameters that forms the equation of a line that minimizes our cost function

Mathematically, this is called an optimization problem and can be written as such:

\[\min\limits_{\theta_0, \theta_1}J(\theta_0, \theta_1)\]

Optimization via Gradient Descent

Gradient descent is an algorithm that will allow us to perform the aforementioned optimization. The key insight to gradient descent comes from the shape of our cost function. If we plot out the cost function with some different values, we’ll likely get a “U” shaped curve like below.

This comes from the square term in our sum of squares function. Note that this graph is in two dimensions, where our actual graph of $J(\theta_0, \theta_1)$ would be three-dimensional, but the same idea still applies.

So how can we determine a minimum from this curve? With gradient descent, we’ll pick an arbitrary set of starting values for our initial parameters (usually 0). The cost of those parameters will give us some point on the U curve of the cost function we were just looking at. Since we are aiming for the minimum, we’ll want to adjust our parameters such that the cost function will be less as a result. To figure out how to tweak the parameters to achieve this, we’ll take the derivative of the cost function, and subtract the current value of the parameter by that partial derivative (multiplied by a constant). We’ll then repeat this process over and over again, moving closer and closer to the minimum. Mathematically this can be described as

Repeat until convergence:
for $j = 0\dots m$:
$\theta_j := \theta_j - \alpha \frac{\partial}{\partial\theta_j}J(\theta_0, \theta_1)$

A few things to note about this pseudocode

“Until convergence” is user defined, but usually when the cost function stops decreasing by a significant amount after an iteration of the outer loop
$m$ is the number of parameters in the model
$\alpha$ is the learning rate, which will determine how big or little of changes to the parameters we make per iteration.
Simultaneous update: we’ll have to calculate the new values of the parameters one at a time, but we want to update them all at once. We don’t want to mix new parameter values and old ones during the inner loop.

Intuitively, we can think of gradient descent as if we were trying to walk down a hill into a valley. The partial derivatives will tell us the “gradient”, or which direction we need to move in order to progress to the bottom. Each iteration of the outer loop is like taking a small step. Given enough steps, we’ll eventually make it to the valley floor, or the minimum of our cost function.

A nice debugging feature of gradient descent is that, when everything is working properly, the cost function should always decrease with every iteration. If the cost function stays the same, or only increases my an infinitesimal amount, then we’ve likely converged at the minimum. If it is ever increase, something has probably gone wrong and we should check our code (or adjust our learning rate to be smaller).

There’s one last thing we need to define before we can start implementing this algorithm in code, and that’s the partial derivative of the cost function. If you know multivariate calculus, feel free to try and derive this yourself, but for the sake of brevity I’ll simply give the answer below:

\[\frac{\partial}{\partial\theta_j}J(\theta_0,\theta_1) = \frac{1}{N}\sum_{n=1}^{N}[(h_\theta(x_n) - y_n) * x_{n,j}]\]

Where $x_{n, j}$ is the $j$th feature value on the $n$th training set. Although we didn’t explicitly state it, we can think of each $x$ example as a pair, $(1, x)$, that gets multiplied and summed to the respective parameter, $(\theta_0, \theta_1)$.

Naive Implementation in Python

Math is fun, but what does this all look like in code? For this example, I’m going to skip over the data collection and cleaning, and focus on the interesting parts. I’m going to assume that I have two (Python) lists, X and Y that hold the $x$ and $y$ values of our data set, respectively.

We’ll first break our X and Y lists into two separate sets of lists: X_train, Y_train, X_test, and Y_test. The rule of thumb for this split is about 70% of the examples will go to the training lists, and the other 30% will be reserved for testing.

Before we move any further, I want to stress how critical it is that we split up our data set into training and test sets. The training set will be used for the “learning” portion (determining the values of $\theta_0$ and $\theta_1$), while the tests set will be used to evaluate how good of a model we have once the parameter values are set. The point here is that our ultimate goal is to have a model that’s general: it is effective at predicting new values that it hasn’t seen before. So if we try to test with the same data that we trained with, we have no idea if we’re achieving our goal, since all the test points will have been seen during training. With the segregated sets, we can determine if our model can be effective when it encounters new data.

The first thing we’ll need to do is define our hypothesis function. This is trivial since our simple model only has two parameters.

def hypothesis(theta0, theta1, x):
    return theta0 + (theta1 * x)

Next, we’ll need to define our cost function

def cost(theta0, theta1, X, Y):
    errors = [hypothesis(theta0, theta1, x) - y for (x, y) in zip(X, Y)]
    squared = [i * i for i in errors]
    return 1 / (2 * len(X)) * sum(squared)

Now we’ll define a descend function that will serve as one iteration of the gradient descent loop

def descend(t0, t1, X, Y):
    xy = zip(X, Y)
    N = len(X)

    # Calculate partial wrt theta0
    p0 = [hypothesis(t0, t1, x) - y for (x, y) in xy]
    p0 = (1 / N) * sum(p0)

    # Calculate partial wrt theta1
    p1 = [(hypothesis(t0, t1, x) - y) * x for (x, y) in xy]
    p1 = (1 / N) * sum(p1)

    new_t0 = t0 - LEARNING_RATE * p0
    new_t1 = t1 - LEARNING_RATE * p1
    return (new_t0, new_t1)

Finally, we’ll define our main function that will serve as the driver of the program

def main():
    # ...data generation omitted...
    theta0, theta1 = 0, 0
    old_cost = cost(theta0, theta1, X_train, Y_train)

    while True:
        theta0, theta1 = descend(theta0, theta1,
                                 X_train, Y_train)
        new_cost = cost(theta0, theta1, X_train, Y_train)

        # Check for convergence
        if abs(new_cost - old_cost) > EPSILON:
            break
        else:
            old_cost = new_cost

And that’s all we really need to implement linear regression with gradient descent. However, it may be useful to visualize the results and evaluate how good of a job we did. We’ll create some extra functions that let us do that.

def graph(t0, t1, X, Y):
    # using matplotlib.pyplot
    pyplot.plot(X, Y, color='r.')
    pyplot.plot(X, [hypothesis(t0, t1, x) for x in X])
    pyplot.show()

def r_squared(t0, t1, X, Y):
    Yp = [hypothesis(t0, t1, x) for y in Y]
    u = sum([(yp - yt)**2 for (yp, yt) in zip(Yp, Y)])
    mean = sum(Y) / len(Y_true)
    v = sum([(y - mean)**2 for y in Y])
    return (1 - (u / v))

The r_squared function implements a common statistical equation called $R^2$. The details aren’t important; just know that a score closer to 1.0 is better.

scikit-learn Implementation

As painful as that might have been, luckily other people have done it before and given away their code for free. As a plus, they probably did it better.

scikit-learn (sklearn) is a popular Python library that’s full of handy algorithms for machine learning. Let’s take a look at how we can leverage this awesome resource for our example problem.

from sklearn.linear_model import LinearRegression
from numpy import array

# Transform X and Y to numpy vectors
X_train = array(X_train).reshape(-1, 1)
Y_train = array(Y_train).reshape(-1, 1)

# Perform the actual training
lr = LinearRegression()
lr.fit(X_train, Y_train)

# Predict and evaluate
y_15 = lr.predict(15)
r_squared = lr.score(X_test, Y_test) # must be vectors
params = lr.get_params()

And that’s it: the whole algorithm boils down to essentially two lines. You’ll note that we had to turn our lists into numpy vectors, and that is so that scikit-learn can do some internal optimization with our data to run faster.

Results

When I ran our implementation, I found that a learning rate of about 0.0003 worked well. Gradient descent took about 25,000 iterations to converge, but only about 2-3 seconds. We also got an $R^2$ score of 0.9980. The results are plotted below.

So even though our example was pretty simple, we did a pretty nice job fitting our data. Not bad for less than 50 lines of code!

Conclusion

Although this was a relatively simple example, these principles are foundational and used throughout machine learning. Indeed, machine learning can often seem magical (especially considering its powerful applications), but at the end of the day it boils down to some algorithms leveraging statistics quite nicely.

Here, we looked at one specific example of machine learning: linear regression with gradient descent. But tons of algorithms exists, like support vector machines, logistic regression, and artificial neural networks to name a few. Each has their own appeals and drawbacks that makes them more or less suited to particular problems.

Machine learning, and more broadly, artificial intelligence, can’t solve every problem. But they are an extremely powerful way of tackling problems that often seem impossible to accomplish through computation. We are still discovering new applications and methods for machine learning, and I have no doubt that it will even more of a dramatic impact on our daily lives in the near future.

Docker for Beginners

2016-10-13T00:00:00+00:00

Docker and containerization are all the rage nowadays, but what are they, and what makes them so technologically appealing?

We should first consider the problems that they are trying to solve. Let’s say that you have an application you want to deploy: some code that will have to run on another machine, either your own server or in the mystical cloud. How can you be sure that your app will work after you deploy it? Of course, you have tests, and of course, they are passing before you even consider releasing, so you know that it runs fine on your computer, but that’s not where we’re deploying. On a different computer or in a different environment, there’s no real way for us to ensure that something won’t break somewhere because the configurations don’t perfectly match up.

Now let’s consider a slightly different problem. Perhaps your application has become incredibly popular. Congratulations. But the immense attention its now receiving is causing a serious degredation in the performance. How can you scale to meet your users needs? You could buy a nicer server or upgrade your cloud instance, but that’s pricey and can only get us so far. What we need is an effective way to scale outward, or horizontally.

The Old Solution: Virtual Machines

Virtual machines can be a viable solution to these problems. They serve as full-fledged computers implemented solely in software. Its like having a computer inside your computer. They’re entirely self-contained, which solves the consistency issue: instead of shipping code, you can ship a VM that has your code on it. So long as you deploy with the same machine you test with, you can rest assured that the two environments will remain consistent.

Virtual machines can also aid in terms of scalability, though in a limited fashion. If you design your application appropriately, you can scale by adding new instances of your VM to a pool in production. When a user hits your service, they will utilize only part of your total deployment, leading to a distributed workload. Of course, developing an application for this kind of architecture is no small feat, since distribution introduces problems of consistency and fault-tolerance, to name a few.

So why even bother with containization when we already have virtual machines. A large drawback to VMs is that they are bulky. A typical virtual machine will be gigabytes in size, since they have to exhibit all the characteristics of a legitimate computer. All of that virtualization creates additional overhead, even with “bare-metal” VMs that interact more closely with the underlying hardware.

In addition, VMs aren’t very flexible. Each one has to be configured prior to starting with the exact amount of physical hardware (CPUs, memory, etc.) that it, and it alone, has access to. Virtual machines can’t share resources, and the ammount alloted to them can’t be modified at runtime. So although a physical machine can run multiple VMs, its limited in terms of its total resources.

Containers: The Lean, Mean, Virtual Machine

Containers leverage a lot of the benefits of virtual machines while avoiding their detriments, and they do it in a very clever way. Recall that virtual machines created a significant amount of overhead by recreating every aspect of the physical computer in software. Containers skip this and leverage the underlying kernel of the host, effectively sharing the low level code and resources with the host and other containers. In addition, they also share common binaries and packages. In reality, containers aren’t so much a virtualized computer as they are a shim: the real work is keeping track of the ways that a particular container differs from the actual machine its running on. By only keeping up with the differences, containers are dramatically smaller and faster.

Here’s what I mean concretely: the ISO for Ubuntu 14.04 on I have downloaded is 649 megabytes. If I download the same version as a Docker container image, it’s only 188 megabytes.

Similarly, starting an Ubuntu VM takes around 30 seconds on my machine (Macbook Pro with SSD), even if it just headless. A container, however, will take less than a second.

But with all this sharing going on, doesn’t that negate one of our primary reasons for choosing VMs: that we would have completely isolated environments? Actually, no. Containers maintain all their “differences”, what makes them unique to the host and other containers, constantly. The effect is that they are in fact completely isolated on a logical level. They may share a file or a library under the hood, but its completely transparent to you, the user.

The fact that containers are still isolated while also being lightweight means that we can use them similarly to the way we can use VMs with much less overhead. We can be confident that our apps will operate the way we expect them too because we can literally ship the environment with the app all bundled up in neat containers.

Containers are not VMs

It can be tempting, especially at the outset, to view containers simply as virtual machines, particularly since the analogy is so elegant. However the two differ in some substantial ways; not only technologically, but functionally.

First of all, containers should be ephemeral. They should be logically small and capable of quickly being started and stopped. They should not serve as a location for data that needs to be maintained over time (for that, you need volumes).

Additionally, containers should only run one main process. This is important. Since containers are lightweight, we should use them as such. Since containers are ephemeral, we shouldn’t rely on them sticking around. By breaking up our app into multiple containers, we can acheive greater modularity and scalability, though not without some redesign.

Working with Docker

Enough theory, let’s actually get our hands dirty.

To utilize containers yourself, you’ll need a containerization engine. The most popular one out there is Docker. Instruction for installing on your platform can be found here.

If everything went smoothly, you should be able to run docker info and get some reasonable output.

Once you have Docker running, the real fun begins. Let’s print a message from the Docker whale using cowsay:

$ docker run docker/whalesay cowsay "Hello, docker"

The command should take a few seconds to run fully. If this is your first time running this command, Docker has to pull (download) the image first. An image in Docker is analogous to an snapshot of a VM. It serves as a template from which we build all of our containers off of. However, each container from this image won’t cause the image to be reconstructed, since the container will simply keep track of how it differs from that base image. Concretely, this means that no matter how many containers of an image we spin up, we only need to have one copy of the image on disk.

Images are almost always layered, which is why you probably saw multiple lines downloading after issuing the command. Even these layers can be reused by Docker. So if in the future you use a different, but similar, image, the download time (and disk space) will be decreased.

Docker hosts tons of images for lots of different applications, which can be found at Docker hub. Docker will pull from Dockerhub automatically if it can’t find the image locally. You can download an image with docker pull <<image name>> and search images with docker search <<image name>>

Once the image downloads, the container will start. In our command, we specified that the docker/whalesay image should execute the command cowsay. You should see the following output:

Once the cowsay command ends, the container will promptly stop. However, it will stick around should you want to run it again. You can view your running containers with docker ps and view all containers with docker ps -a.

Docker will automatically generate a silly name for your container, because whimsy is an important aspect of software engineering. To delete the container, you can run docker rm <<container name/id>>

Playing in a Sandbox

Let’s toy around with this some more. If you ran the docker/whalesay container from the previous section, you should have the image saved on your computer. You can verify this with docker images.

Earlier, we ran the cowsay command on this image, but there’s nothing stopping us from running other commands. Try

$ docker run docker/whalesay date

It should print out the current time. That’s nice, but what if we want to keep the container up and run multiple commands? This is called attaching to a container. To do this, we will need two things. We need to tell Docker to give our container a tty interface, so we can issue commands, as well as to read input from stdin (our keyboard, in this case). Additionally, we will need to execute a long-running process that won’t immediately end and kill our container. We can do all this with the following command:

$ docker run -it docker/whalesay bash

Here, the -it flags give us an “interactive” and “tty” run on our container. We also execute bash, which will serve as our long-running process. After executing this command, you should see a different terminal prompt, coming from inside your container.

Feel free to mess around and issue any commands you normally could on an Ubuntu machine. But be warned: any changes you make die along with the container. Do not store data inside a container. Instead you should use volumes to export the data to a more persistent location. When you’re done, you can exit the container by exiting the bash shell: <Ctrl-d>.

Go forth, and Dockerize

There’s a load more to say about Docker and containers in general. Some interesting points that I didn’t get to in this post include (but are not limited to):

Volumes and persistent storage
Building your own images with Dockerfiles
Creating a full-stack app with containers
Networking to and between containers

Security Warning

I would be remiss if I did not specifically clarify a security concern regarding containers: containers are not a securely isolated solution. Although we spoke of the isolation that containers offer, they fundamentally share the kernel with the host operating system, and are therefore not secure in their own right. The common solution is to run Docker from within a (single) virtual machine. Since the VM is securely isolated, the host machine is not at risk.

Brandon Morris

Reinforcement Learning: Playing Doom with PyTorch

What is Reinforcement Learning?

The Markov Decision Process

Deep Q-Learning

Putting it to Practice: ViZDoom

Additional Resources

Building a World-Class CIFAR-10 Model From Scratch

The Residual Network

The Wide ResNet

The Structure of a Wide ResNet

Implementing the Wide ResNet

The BasicBlock Class

The WideResNet Class

Data Loading and Training

Conclusion

Mastering the Learning Rate to Speed Up Deep Learning

Difficulties in Finding the Right Hyperparameters

What is the Learning Rate?

Cyclical Learning Rates

The LR Range Test

Pedal to the Metal: Super-Convergence

Conclusion

Breaking Neural Nets with Adversarial Examples

What are adversarial examples?

How adversarial examples are created

Adversarial examples outside of image classification

Conclusions and implications

Dynamic Routing Between Capsules

Why Typical Convnets are Doomed

Sub-sampling loses precise spatial relationships

Convnets are invariant, not equivariant

A note on sub-sampling

Capsules to the Rescue

Connecting Capsule Layers: Dynamic Routing by Agreement

CapsNet: A Shallow Network with Deep Results

Results

Conclusion

Simplifying Deep Learning Programming with Keras

Keras: The Model Abstraction

Adding Layers

Compiling and Training the Model

Evaluating the Results

Getting Started with Tensorflow

Prerequisites

What is TensorFlow?

Programming in TensorFlow

Building an MNIST classifier

What is deep learning?

Getting started

A quick history of deep learning

Going deeper: What’s in a neuron?

Training a neural network: Optimization and backpropagation

Deep learning variations

Conclusion

Clustering with K-Means

The Setup

The Algorithm

The Code

Conclusion

An Introduction to Machine Learning

Linear Regression

Case Study

Optimization via Gradient Descent

Naive Implementation in Python

scikit-learn Implementation

Results

Conclusion

Docker for Beginners

The Old Solution: Virtual Machines

Containers: The Lean, Mean, Virtual Machine

Containers are not VMs

Working with Docker

Playing in a Sandbox

Go forth, and Dockerize

Security Warning

The `BasicBlock` Class

The `WideResNet` Class