Deep Learning Papers¶
参考Deep Learning Papers Reading Roadmap来阅读DL相关论文，并做简单记录。
Deep learning(Three Giants' Survey)¶
Deep Learning1是三位大佬的综述，对DL的发展做了一个总结。 看完对DL有了一个更加明确的认识(其实黑盒之前的理解也差不多，只不过这里很明确地指出来了)，那就是Deep Learning主要就是learn representations of data with multiple levels of abstraction.
总的来说，文章指出了传统ML的缺点是： required careful engineering and considerable domain expertise to design a feature extractor. 这就使得传统ML受人工知识和经验等的限制，很难在raw data“不规则”时取得好的效果。为了使得传统ML算法更加高效，便有了kernel methods, 但是实际中发现有时候效果并不好，如Gaussian kernel引入的generic features带来的效果并不能泛化到测试集上。而依赖于Representation Learning的Deep Learning就不受上述限制，其用到的特征是在网络的各个layer中自动学习的。Good features can be learned automatically using a general-purpose learning procedure.(This is a key advantage of deep learning). 所以说，很多DL方法本质上就是做了自动化特征提取的工作： The hidden layers can be seen as distorting the input in a non-linear way so that categories become linearly separable by the last layer.
论文中对Representation Learning和Deep Learning给出了大致定义:
- Representation Learning is a set of methods that allows a machine to be fed with raw data and automatically discover the representations needed for detection or classification.
- Deep Learning methods are representation-learning methods with multiple levels of representation, obtained by composing simple but non-linear modules that each transform the representation at one level(starting with raw input) into a representation at a higher, slightly more abstract level.
论文后面大概讲了下CNN和RNN，CNN中对convolutional layer和pooling layer的讲解很好：The role of the convolutional layer is to detect local conjunctions of features from the previous layer; the role of the pooling layer is merge semantically similar features into one.
最后是关于unsupervised learning的两个比较感兴趣的地方。For smaller data sets, unsupervised pre-training helps to prevent over-fitting, leading to significantly better generalization when the number of labelled examples is small, or in a transfer setting where we have lots of examples for some 'source' tasks but very few for some 'target' tasks. (给出了参考文献，后面打算看一下)。 另外就是文末的一句：Human and animal learning is largely unsupervised: we discover the structure of the world by observing it, not by being told the name of every object.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015. ↩