Multimixture fitting

I was wondering how one can modify Expectation-Maximization procedure for fitting mixtures (well, gaussian mixtures, because it's the only distribution that can be fitted easily) to support really many overlapping summands in mixture.

Randomization probably can be a solution to this problem.

Let me first remind how EM works. There are two steps that are computed iteratively

1. (Expectation) where we compute probability that each particular event belongs to each distribution
2. (Maximization) where given the probabilities we maximize parameters of each distribution.

What if we sample events according to distribution from expectation step? At each stage we will attribute each event to one (in simplest case) component of mixture, or maybe several of them. This kind of randomization should prevent us from 'shrinking' of distribution.

Ok, this again needs time for experiments.

Tensor train

One of probable approaches to build graphical models with categorical variables is tensor decomposition:
http://bayesgroup.ru/wp-content/uploads/2014/05/icml2014_NROV-1.pdf
Notably both tensor decomposition (tensor train format) and method to use it for graphical models were developed at my faculty, though by different people at different chairs.
One more interesting question is interpretation of those hidden variables emerging in the middle.
At this moment I'm thinking over possibility to build this into GB-train, since trained model for each event provides sequence of binary visible variables. In principle, this may be written in quite simple way. Provided, that $x_i$ is boolean variable corresponding to $i$th cut in the tree (in the train, to be more precise).

For instance, one can write partition function as $$ Z = A_1[x_1, y] A_2[x_2, y] \dots A_n[x_n, y] $$ or as $$ Z = A_1[x_1] B_1[y] A_2[x_2] B_2[y] \dots A_n[x_n] B_n[y] $$

In both cases it's quite simple to estimate posterior probability since we have only limited set of options (targets) to be checked. But which one should be preferred and why?

Links on deep learning

Didn't know where to put it, so just for the memory will post it to blog.

• https://charlesmartin14.wordpress.com/2015/03/25/why-does-deep-learning-work/
  There is much fuzz nowadays about why deep learning works at all (there is no any deep theory under today), and I love reading these hypothetical explanations (though I'm absolutely sure all of them are wrong. A good explanation of success should give you new ideas about what new things will work).
  
  In this couple of articles author is arguing that the action of RBM can be derived as an action of renormalization group. BTW, this is not a first physical analogy in neural network. Apart from RBMs which use Gibbs-like distribution, there were explanations of Hopfiled neural networks via spin glasses and derivation of update rule from mean-field theory.
  
  
• http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
  This impressive post was written a year ago and shows recent fresh ideas about deep representations of objects. Namely, I was surprised with how common space of representation for different objects may help in translations.
  
  
• http://ai.stanford.edu/~ang/papers/icml09-ConvolutionalDeepBeliefNetworks.pdf
  Finally, a link to a paper where convolutional RBMs were introduced. Using softmax to combine with poll-layer is a good idea.

PS. Found a link of recommended reading for new LISA-lab students. http://www.datakit.cn/blog/2014/09/22/Reading_lists_for_new_LISA_students.html

Vector representations of categories

After thinking a while over the last point in my list of topics in machine learning I was going to think over...

I finally came to the thought that I'd better use some graphical model, since there is a great majority of possible algorithms without any warranty of their workability.

So, first thing - probably one can use LDA (Latent Dirichlet allocation, wiki) to find good representation for categories. In this case we should use bag-of-words model, and `categories` will stand for words.
Yes, the model of LDA doesn't seem to be appropriate, though I'm sure it will give reasonable representations.

Another graphical model I came with, is much simpler (and probably was developed a long time ago, but I don't know).
It's generative model looks looks this:

1. First, a `topic` of event is generated. That's an n-dimensional vector $t_{event} \in \mathbb{R}^n$ drawn from gaussian distribution.
2. For each category, say, `site_id`, each value of site_id has it's own topic $t_{cat.value}$ of same dimension $n$. The greater dot product $(t_{cat.value}, t_{event})$, the higher probability that this value of category will be chosen. I shall stress here, that I'm not thinking over the arbitrary number of categories, I'm currently interested in the case, where the number and types of categories are fixed.
3. To be more precise, the probability of each value within category to be drawn for the event with $t_{event}$ vector is propostional to $$ p(\text{cat_value} | \text{event}) \sim p(\text{cat.value}) \times e^{(t_{cat.value}, t_{event})} $$ so, to compute final probability one shall use softmax function.

That said, currently the problem I expect to meet is extremely low speed of training when the number of values for some category is very large (say, there are cases when number of categories one shall proceed is greater then 10000)

PS. After writing this I understood that usage of decision trees to generate ids of leaves - new 'categories' + usage of LibFM over these new features was a very interesting and good idea.

Code and the importance of vectorization

That awkward moment, when the code written in matlab is easier to read and understand then tons os explanations:
Salakhudinov's code of RBM

This code IMHO is a good argument when you need to explain someone that he really needs to learn at least one language with vectorization, no matter whether it is R, matlab, or numpy or theano in python.