Back to the future (vector representation of future, an approach to analysis of time series)

Today I generated an interesting idea on analysis of time series. Those are frequently used in pattern recognition, mathematical finance and forecastings (weather prediction, sales prediction, number of site visitors etc.).
Apart from different window-based approaches, there are two basic models that by their nature represent the sequential structure in data:

• hidden Markov chain (HMC)
• recurrent neural network (RNN)

Both of the above are actually Markov models and have some internal state (in first case it's probabilities of each of internal states of model, in the case of RNN the state is equal do dumped states of all elements with delay). The notable difference between these models is first is generative, while the second is discriminative.

Some pros and cons of both models I see:

• HMC training is inexpensive, but requires predicted values to be binned (so it doesn't work with continuous variables, even with variables with many possible values)
• RNN requires quite a long training, but has hard times to find stable mapping (since it should be continuous), so requires some regularizations built in its architecture. Training is usually based on prediction of one or several next points in sequence, which is not always adequate.

Apart of setting several next observation in sequence as a target, there is plenty of various targets, like running mean over $n$ next observations. However, I'd prefer machine learning to find out these targets for me. 

Possible approach I came to:

1. train HMC in inverse time, so HMC predicts past, not future
2. it's predictions (probabilities of hidden states) can be treated as vector representation of future, because it is computed on information from future
3. RNN is trained to predict hidden states of HMC
4. Later, some other model is used based on the output of RNN to predict the value of interest. 

So the trick is that we use HMC predictions as some reliable informative target about the future, like this is done is deep learning, for instance.

Theano-based libraries for machine learning

I wrote several times about mathematical vector engine theano and its benefits.

If you're going to dive deep into neural networks, I recommend learning it and using pure theano. However, there are numerous neural network libraries based on theano, let's list some of them:

• Theanets, http://theanets.readthedocs.org/ (pay attention - it is different from theanet, which I haven't found useful)
  theanets is a good option to start,  quite efficient and simple, provides also posssibilities for recurrent neural networks. However, notice, that rprop implementation uses mini-batches, which makes it unstable.
• Keras, http://keras.io/
  so far seems to be very adequate theano library, contains several minibatch-based optimizers and several loss functions, mostly for regression. Authors compare it to Torch.
• Pylearn2, http://deeplearning.net/software/pylearn2/
  this library was written by LISA-lab, authors of theano, however library though being very advanced, itself is terribly complex. Usually it is easier (at lest for me) to write things from the scratch then writing YAML configuration
• others which I consider to be not as mature and partially forgotten:
  lasagne, blocks, crino, deepANN (last one in deprecated), .
• finally, want to note my nano-library (500 lines of code!), which allows using 5 trainers + 6 losses on feedforward networks. Supports weights, and extremely flexible, because it's main paradygm: write an expression. Yes, use theano and just write activation function, blackbox optimization methods will do everything for you. This allows writing amazingly complex activations since you're not more restricted to 'layers' model and fitting of arbitrary functions:
  
  https://github.com/iamfullofspam/hep_ml/blob/master/hep_ml/nnet.py
  
  One more notable thing: it uses scikit-learn interface, so you can use at as a part of, say,  pipeline. Or run AdaBoost over neural networks (which is very fast,  by the way).

Links on deep learning

Didn't know where to put it, so just for the memory will post it to blog.

• https://charlesmartin14.wordpress.com/2015/03/25/why-does-deep-learning-work/
  There is much fuzz nowadays about why deep learning works at all (there is no any deep theory under today), and I love reading these hypothetical explanations (though I'm absolutely sure all of them are wrong. A good explanation of success should give you new ideas about what new things will work).
  
  In this couple of articles author is arguing that the action of RBM can be derived as an action of renormalization group. BTW, this is not a first physical analogy in neural network. Apart from RBMs which use Gibbs-like distribution, there were explanations of Hopfiled neural networks via spin glasses and derivation of update rule from mean-field theory.
  
  
• http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
  This impressive post was written a year ago and shows recent fresh ideas about deep representations of objects. Namely, I was surprised with how common space of representation for different objects may help in translations.
  
  
• http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
  Finally, a link to a paper where convolutional RBMs were introduced. Using softmax to combine with poll-layer is a good idea.

PS. Found a link of recommended reading for new LISA-lab students. http://www.datakit.cn/blog/2014/09/22/Reading_lists_for_new_LISA_students.html