Back to the future (vector representation of future, an approach to analysis of time series)

Today I generated an interesting idea on analysis of time series. Those are frequently used in pattern recognition, mathematical finance and forecastings (weather prediction, sales prediction, number of site visitors etc.).
Apart from different window-based approaches, there are two basic models that by their nature represent the sequential structure in data:

• hidden Markov chain (HMC)
• recurrent neural network (RNN)

Both of the above are actually Markov models and have some internal state (in first case it's probabilities of each of internal states of model, in the case of RNN the state is equal do dumped states of all elements with delay). The notable difference between these models is first is generative, while the second is discriminative.

Some pros and cons of both models I see:

• HMC training is inexpensive, but requires predicted values to be binned (so it doesn't work with continuous variables, even with variables with many possible values)
• RNN requires quite a long training, but has hard times to find stable mapping (since it should be continuous), so requires some regularizations built in its architecture. Training is usually based on prediction of one or several next points in sequence, which is not always adequate.

Apart of setting several next observation in sequence as a target, there is plenty of various targets, like running mean over $n$ next observations. However, I'd prefer machine learning to find out these targets for me. 

Possible approach I came to:

1. train HMC in inverse time, so HMC predicts past, not future
2. it's predictions (probabilities of hidden states) can be treated as vector representation of future, because it is computed on information from future
3. RNN is trained to predict hidden states of HMC
4. Later, some other model is used based on the output of RNN to predict the value of interest. 

So the trick is that we use HMC predictions as some reliable informative target about the future, like this is done is deep learning, for instance.

Theano-based libraries for machine learning

I wrote several times about mathematical vector engine theano and its benefits.

If you're going to dive deep into neural networks, I recommend learning it and using pure theano. However, there are numerous neural network libraries based on theano, let's list some of them:

• Theanets, http://theanets.readthedocs.org/ (pay attention - it is different from theanet, which I haven't found useful)
  theanets is a good option to start,  quite efficient and simple, provides also posssibilities for recurrent neural networks. However, notice, that rprop implementation uses mini-batches, which makes it unstable.
• Keras, http://keras.io/
  so far seems to be very adequate theano library, contains several minibatch-based optimizers and several loss functions, mostly for regression. Authors compare it to Torch.
• Pylearn2, http://deeplearning.net/software/pylearn2/
  this library was written by LISA-lab, authors of theano, however library though being very advanced, itself is terribly complex. Usually it is easier (at lest for me) to write things from the scratch then writing YAML configuration
• others which I consider to be not as mature and partially forgotten:
  lasagne, blocks, crino, deepANN (last one in deprecated), .
• finally, want to note my nano-library (500 lines of code!), which allows using 5 trainers + 6 losses on feedforward networks. Supports weights, and extremely flexible, because it's main paradygm: write an expression. Yes, use theano and just write activation function, blackbox optimization methods will do everything for you. This allows writing amazingly complex activations since you're not more restricted to 'layers' model and fitting of arbitrary functions:
  
  https://github.com/iamfullofspam/hep_ml/blob/master/hep_ml/nnet.py
  
  One more notable thing: it uses scikit-learn interface, so you can use at as a part of, say,  pipeline. Or run AdaBoost over neural networks (which is very fast,  by the way).

Very deep neural network

Link to article: http://arxiv.org/abs/1505.00387

With proposed technique one can build very deep neural networks (up to hundreds of layers). The key principle why this works is very simple: $$ x_{n+1} = x_n + f(x_n), $$ where the second summand is small enough.

There are two points actually:

• First, one uses very many layers, and is are able to approximate all needed functions.
• Second, since the first summand dominates, there is no vanishing gradient problem.

Not sure if this really has some advantages over shallow ANNs, but still an interesting approach.

So, it's a way to train deep network, though doesn't have any attitude to what people usually call 'deep learning', since here we are not trying to establish some new hidden categories.

Code and the importance of vectorization

That awkward moment, when the code written in matlab is easier to read and understand then tons os explanations:
Salakhudinov's code of RBM

This code IMHO is a good argument when you need to explain someone that he really needs to learn at least one language with vectorization, no matter whether it is R, matlab, or numpy or theano in python.

Good list of NN training methods

Dived deeper into the methods of training NNs.
Good yet incomplete list of what people did in this area is given in this article dated 2006.

But there is no my favourite Rprop and it's modifications (IRprop+, IRprop-).

BTW, recently dived deeper into experiments with neural networks, and I decided to improve IRprop by keeping track about not simply moving along each axis, but as well along directions
$w_i + w_j$, $w_i - w_j$, being sure that this should speed up training progress.

Well, it increased the speed for the first time, but very fast it stops decreasing loss function and when it is close to the minimal value, it starts serious oscillations and becomes simply unstable.

Pairwise layer in neural network

In one of previous posts I said that pairwise layer doesn't seem to work...

Well, I was wrong: after checking on higgs-boson dataset from kaggle I found out that this kind of neural network works much better than traditional ones! Hurrah!

Though, much worse then GBDT, but after building AdaBoost over neural network I was able to get comparable (or just the same) quality. The only problem is GBDT trained in minutes, while it took ~24 hours for boosting over NN to train.

Neural Networks

Just to describe one of my experiments with neural networks.

Neural networs initially were developed as simulation of real neurons, first training rules (i.e. Hebb's rule) were 'reproducing' the behaviour we observe in nature.

But I don't expect this aproach to be very fruitful today. I prefer thinking of neural network as of just one of ways to define function (which is usually called activation function).

For instance, one-layer perceptron's activation function may be written down as
$$f(x) = \sigma( a^i \, x_i )$$
following the Einstein rule, I omit the summation over $i$. $a_i$ are weights.

Activation function for two-layer perceptron ($a^i_j$ and $b^j$ are weights):
$$f(x) =  \sigma( b^j \, \sigma( a^i_j \, x_i )) $$

If one operates the vector variables, and $Ab$ is matrix-by-vector dot product, $\sigma x$ denotes elementwise sigmoid function, then activation function can be written down in pretty simple way:
$$f(x) = \sigma b \sigma A x $$

This is how one can define two-layer perceptron in theano, for instance. Three- or four- layer perceptron isn't more complicated really.

But defining function is only the part of the story - what about training of network? 

I'm sure that the most efficient algorithms won't come from neurobiology, but from pure mathematics. And that is how it is done in today's guides to neural networks: you define activation function, define some figure of merit (logloss for instance), and then use your favourite way of optimization.

I hope that soon the activation functions will be inspired by mathematics, though I didn't succeed much n this direction.

One of activation functions I tried is the following:

First layer:

$$y = \sigma A x $$

Second (pairwise) layer:

$$f(x) = \sigma (b^{ij} y_i y_j ) $$

The difference here that we can use now not only activation of neurons, but introduce some pairwise interaction between them. Unfortunately,  I didn't feel much difference between this modification and simple two-layer network.

Thank to theano,  this is very simple to play with different activation functions :)

Theano (python library)

If you want to experiment with neural networks,  and you know python, then my best recommendation is: use theano.

Theano is not about neural networks, really - it is ... hm ... mathematical engine. Something between Matlab and Mathematica.
It is close to Matlab because uses vectorization (the final function will operate vectors).

It is close to Mathematica,  because first you define some expression (function as analytical expression). You can compute (analytical) derivatives, and this is crucial for neural networks, because the main thing you need is derivatives, and nobody wants to spend his time on computing derivatives, specially when some library can do this for you.

After you defined needed function-expressions (activation function and gradient of loss function for neural network), you can compile it with theano (get really a function that can be evaluated for some arguments). The compiled function is vectorized and very fast (though compilation usually takes some time). Your functions can be evaluated on GPU as well.

This gives an ability to define your new neural networks in few lines of code (just by defining activation function). Impressive?

Documentation on theano
Github
Examples (mostly about neural networks, probably the easiest way to start and understand how good theano is)

Even more examples (with less explanation) can be found here.