Tuesday, June 30, 2015
Do you know that convolution operation is implemented in deep learning systems via matrix multiplication?
Here is a detailed description of how it is done by chunking the data
http://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/
Theano library, by the way, provides many possibilities for implementation of convolution, among which CPUCorrMM (this isn't the main way however)
http://deeplearning.net/software/theano/library/tensor/nnet/conv.html
Thursday, June 25, 2015
A Programming Language
(~R∊R∘.×R)/R←1↓ιR
Monday, June 22, 2015
Machine learning used in tracking of COMET (japanese particle physics experiment), part I
From physics courses we know that there are conservation laws, most famous are conservation of energy and momenta, but there are other. For instance, conservation of different leptonic family numbers.
For instance, electronic number is number of electrons + number of electronic neutrinos. For some time these numbers (electronic number, muonic number, tauonic number) were considered to be preserved, but in the late 1960's it was proved, that neutrino oscillations change the leptonic number of system (neutrino can become electronic from muonic, for instance). This observation is called lepton flavour violation. However, in the Standard Model overall leptonic number (which is sum of three named family numbers) is conserved.
Since that time physicists are searching for other processes which change the leptonic number and include charged leptons (electrons, muons, tauons), which could be a key for new physics, however no success in this direction and the results are currently mostly upper limits we can prove for some processes. Such possible processes have the name of CLFV (charged lepton flavour violation).
Details of COMET experiment
COMET is small experiment (which is only in plans at this moment) built to detect one specific decay (muon to electron conversion on nucleus), which is suppressed in the standard model:$$ \mu^{-} + N \to e^{-} + N.$$
$$ \mu^{-} + Al \to e^{-} + Al,$$
while this isn't the only way we can obtain electron from muon
$$ \mu^{-} \to e^{-} + \nu_{\mu} + \overline{\nu}_e.$$
Some other illustrations of COMET experiment
![]() |
| The scheme of planned experiment COMET. Detector (with aluminum target) will be in the end of this pipeline. This is what planned during phase II, but for the first time experiment will be simpler |
| COMET during phase I will be much simpler. |
Data collected from COMET
The main information (apart one from trigger) is the data taken from wires, which are registering when charged particle flies nearby (by adsorbing the particles after ionization) and also it measures the time when this happened.
Another characteristic, which helps in distinguishing electrons, is the radius of trajectory, which is computed (in constant magnetic field) by formula
$$r = \dfrac{p_T}{eB}, $$
so for electron we know the distribution of gyroradius from distribution of energies. Gyroradius is how we actually will reconstruct energy of electron.
![]() |
| Typical signal event in COMET, particle is helixing in magnetic field, leaving energy in drift cambers. |
In orthogonal projection of CyDET, the collected information from wires will look like this:
But the shape of circle with some radius is not the only useful information. During signal hits usually less energy is disposed (which may sound a bit contradictory to what I wrote earlier, but the faster particle flies, the less energy it disposes).
![]() |
| The energy deposited is very strong feature, though leaves significant amount of bck hits, which can spoil the picture, when we have high occupancy. |
Apart from this, we can use the 'hit time' (measured from the moment when trigger detected particle).
PS. There is second part of this post, devoted to machine learning.
Links
1. Official COMET site.2. Detailed presentation about CLFV, COMET (and e2mu).
3. Раритеты микромира, an article in russian about rare processes (CLFV is one of examples)
4. COMET phase-I proposal
5. Repository with algorithms and results of tracking.
Monday, May 4, 2015
Vector representations of categories
After thinking a while over the last point in my list of topics in machine learning I was going to think over...
I finally came to the thought that I'd better use some graphical model, since there is a great majority of possible algorithms without any warranty of their workability.
So, first thing - probably one can use LDA (Latent Dirichlet allocation,
wiki) to find good representation for categories. In this case we should use bag-of-words model, and `categories` will stand for words.
Yes, the model of LDA doesn't seem to be appropriate, though I'm sure it will give reasonable representations.
Another graphical model I came with, is much simpler (and probably was developed a long time ago, but I don't know).
It's generative model looks looks this:
- First, a `topic` of event is generated. That's an n-dimensional vector $t_{event} \in \mathbb{R}^n$ drawn from gaussian distribution.
- For each category, say, `site_id`, each value of site_id has it's own topic $t_{cat.value}$ of same dimension $n$. The greater dot product $(t_{cat.value}, t_{event})$, the higher probability that this value of category will be chosen. I shall stress here, that I'm not thinking over the arbitrary number of categories, I'm currently interested in the case, where the number and types of categories are fixed.
- To be more precise, the probability of each value within category to be drawn for the event with $t_{event}$ vector is propostional to $$ p(\text{cat_value} | \text{event}) \sim p(\text{cat.value}) \times e^{(t_{cat.value}, t_{event})} $$ so, to compute final probability one shall use softmax function.
PS. After writing this I understood that usage of decision trees to generate ids of leaves - new 'categories' + usage of LibFM over these new features was a very interesting and good idea.
Tuesday, January 6, 2015
Benchmarks of speed (Numpy vs all)
However I am much worried about the speed, so decided to collect different benchmarks
numpy vs cython vs weave (numpy is about 2 times slower than others)
(posted in 2011)
http://technicaldiscovery.blogspot.ru/2011/06/speeding-up-python-numpy-cython-and.html
Primarily the post is about numba, the pairwise distances are computed with cython, numpy, numba.
Numba is claimed to be the fastest, around 10 times faster than numpy.
(posted in 2013)
https://jakevdp.github.io/blog/2013/06/15/numba-vs-cython-take-2/
Julia is claimed by its developers to be very fast language.
Well, if that is true, there would be no need in writing easily-vectorizable operation in pure python (yep, I mean they are simply cheating), currently these 'benchmarks' are at the main page
http://julialang.org/
The following post was written in 2011, the problem observed is solution of Laplace equation as usual.
Numpy is ~10 times slower than pure C++ solution (and equal to matlab), weave shows the speed comparable with pure C++ (I don't know anyone using weave now)
http://wiki.scipy.org/PerformancePython
Fresh (2014) benchmark of different python tools, simple vectorized expression A*B-4.1*A > 2.5*B is evaluated with numpy, cython, numba, numexpr, and parakeet (and two latest are the fastest - about 10 times less time than numpy, achieved by using multithreading with two cores)
http://nbviewer.ipython.org/github/rasbt/One-Python-benchmark-per-day/blob/master/ipython_nbs/day7_2_jit_numpy.ipynb
Haven't found any general-purpose theano vs numpy benchmarks, but in the article there is comparison of neural networks and theano is expected to give much better speed than numpy/torch(c++)/matlab, specially it is fast on GPU
http://conference.scipy.org/proceedings/scipy2010/pdfs/bergstra.pdf
One more detailed review of numpy vs cython vs c (held in 2014)
http://notes-on-cython.readthedocs.org/en/latest/std_dev.html
Let me copy-paste the example results (computation of std).
| Method | Time (ms) | Compared to Python | Compared to Numpy |
|---|---|---|---|
| Pure Python | 183 | x1 | x0.03 |
| Numpy | 5.97 | x31 | x1 |
| Naive Cython | 7.76 | x24 | x0.8 |
| Optimised Cython | 2.18 | x84 | x2.7 |
| Cython calling C | 2.22 | x82 | x2.7 |
Simple, but comprehensive comparison of python accelerators was prepared by Jake Vanderplaas and Olivier Grisel:
http://nbviewer.ipython.org/github/ogrisel/notebooks/blob/master/Numba%20Parakeet%20Cython.ipynb
Problem is computation of pairwise distances. The results this time seem to be unbiased:
| Python | 9.51s |
| Naive numpy | 64.7 ms |
| Numba | 6.72ms |
| Cython | 6.57ms |
| Parakeet | 12.3 ms |
| Cython | 6.57 ms |
By the way, I was recently looking for a slow place in my code, and it totally dropped of my mind that computation of residual is long operation:
http://embeddedgurus.com/stack-overflow/2011/02/efficient-c-tip-13-use-the-modulus-operator-with-caution/
Integer division / remainder computation time significaly depends on the bit depth (16 bit operation are ~2 times faster than 32 bit ones, division takes roughly 10 times more time than multiplication).
Surprising: multiplication/addition/binary operations take the same time (compared on numpy 1.9)
Thursday, January 1, 2015
Why using HDF5?
The only thing don't agree with is blaze, I've tried it and it is obviously raw and needs much time to become not even stable but at least really useful.
My current workflow is completely based on IPython, and I'm working much with pandas (which I personally consider as a good example of poor library design).
Nevertheless, I moved recently to HDF, though installing pyTables (which is needed to use hdf with pandas) isn't as straightforward as I expected.
And now I convert all the data to hdf.
- First, this usually results in less (about 2-3 times) space needed to store data (but that depends on the dataset, for some ot them there is no difference between csv and hdf).
- Second, your data now stored in binary format, so all the types are strictly defined - no parsing is needed, no guessing of types
- Thus, read/write operations are orders of times faster
- And no approximations are made for float numbers.







