- talks - lists.cs.princeton.edu

Colloquium Speaker: Tushar Krishna, TODAY at 12:30
by Emily C. Lawrence 07 Nov '19

07 Nov '19

CS Colloquium Speaker Tushar Krishna from Georgia Tech Thursday, November 7 - 12:30pm E-Quad, B205 Host: Margaret Martonosi [ https://www.cs.princeton.edu/events/25888 | https://www.cs.princeton.edu/events/25888 ] Enabling Continuous Learning through Synaptic Plasticity in Hardware Ever since modern computers were invented, the dream of creating artificial intelligence (AI) has captivated humanity. We are fortunate to live in an era when, thanks to deep learning (DL), computer programs have paralleled, and in many cases even surpassed human level accuracy in tasks like visual perception and speech synthesis. However, we are still far away from realizing general-purpose AI. The problem lies in the fact that the development of supervised learning based DL solutions today is mostly open loop. A typical DL model is created by hand-tuning the deep neural network (DNN) topology by a team of experts over multiple iterations, followed by training over petabytes of labeled data. Once trained, the DNN provides high accuracy for the task at hand; if the task changes, however, the DNN model needs to be re-designed and re-trained before it can be deployed. A general-purpose AI system, in contrast, needs to have the ability to constantly interact with the environment and learn by adding and removing connections within the DNN autonomously, just like our brain does. This is known as synaptic plasticity. In this talk we will present our research efforts towards enabling general-purpose AI leveraging plasticity in both the algorithm and hardware. First, we will present GeneSys (MICRO 2018), a HW-SW prototype of a closed loop learning system for continuously evolving the structure and weights of a DNN for the task at hand using genetic algorithms, providing 100-10000x higher performance and energy-efficiency over state-of-the-art embedded and desktop CPU and GPU systems. Next, we will present a DNN accelerator substrate called MAERI (ASPLOS 2018), built using light-weight, non-blocking, reconfigurable interconnects, that supports efficient mapping of regular and irregular DNNs with arbitrary dataflows , providing ~100% utilization of all compute units, resulting in 3X speedup and energy-efficiency over our prior work Eyeriss (ISSCC 2016). Finally, time permitting, we will describe our research in enabling rapid design-space exploration and prototyping of hardware accelerators using our dataflow DSL + cost-model called MAESTRO (MICRO 2019). Bio: Tushar Krishna is an Assistant Professor in the School of Electrical and Computer Engineering at Georgia Tech. He also holds the ON Semiconductor Junior Professorship. He has a Ph.D. in Electrical Engineering and Computer Science from MIT (2014), a M.S.E in Electrical Engineering from Princeton University (2009), and a B.Tech in Electrical Engineering from the Indian Institute of Technology (IIT) Delhi (2007). Before joining Georgia Tech in 2015, Dr. Krishna spent a year as a post-doctoral researcher at Intel, Massachusetts. Dr. Krishna’s research spans computer architecture, interconnection networks, networks-on-chip ( NoC ) and deep learning accelerators - with a focus on optimizing data movement in modern computing systems. Three of his papers have been selected for IEEE Micro’s Top Picks from Computer Architecture, one more received an honorable mention, and two have won best paper awards. He received the National Science Foundation (NSF) CRII award in 2018, and both a Google Faculty Award and a Facebook Faculty Award in 2019. He also received the “Class of 1940 Course Survey Teaching Effectiveness” Award from Georgia Tech in 2018.

1 0

Speaker Ravi Chugh, Friday Nov 8 at 11am
by Emily Lawrence 06 Nov '19

06 Nov '19

Title: Bidirectional Programming with Direct Manipulation Speaker: Ravi Chugh, University of Chicago Date/Time: Friday, November 8th, 11 AM — 12 PM Location: CS 402 Abstract: Traditional text-based programming environments present users with mental hurdles and tedious, inefficient development workflows. In this talk, I'll describe techniques that aim to improve the user interfaces for programming. What new capabilities can programming languages offer to users, and what new interaction mechanisms can be used to expose the new capabilities? We are developing a prototype programming environment, called Sketch-n-Sketch, to explore these questions. Regarding capabilities, we present program synthesis and repair techniques for "bidirectionally" developing and evaluating programs. Regarding interaction mechanisms, we present "direct manipulation" interfaces for interacting with programs and their execution. These systems augment traditional text-based programming with new program development workflows, which we plan to further improve and evaluate in the future. Bio: Ravi Chugh has been Assistant Professor of Computer Science at the University of Chicago since 2014. His research and teaching interests revolve around programming language technology --- type systems, synthesis algorithms, and other program analysis techniques --- particularly as they relate to challenges in software engineering and human-computer interaction. Ravi received an NSF CAREER Award in 2017, and a Neubauer Fellowship from the University of Chicago in 2015 for innovative and effective undergraduate teaching. Ravi obtained his Ph.D. from the University of California, San Diego in 2013. Picture: http://people.cs.uchicago.edu/~rchugh/static/images/ravi-2018-10.jpg

1 0

Colloquium Speaker: Jeff Mogul, Wednesday 11/13 at 12:30pm
by Emily C. Lawrence 06 Nov '19

06 Nov '19

Distinguished CS Colloquium Speaker Jeff Mogul, Google Networking Infrastructure Wednesday, November 13 - 12:30pm Computer Science - Room 105 Host: Kyle Jamieson [ https://www.cs.princeton.edu/events/25892 | https://www.cs.princeton.edu/events/25892 ] Self-Driving Networks: Is it a good idea to mix AI and computer networks? Just as you wouldn't want to be in a car without seat belts, even if it's a self-driving car, you probably don't want your Self-Driving Network ( SelfDN ) to rely completely on control-theory or ML, no matter how sophisticated or formally verified. While autonomous systems promise to solve many problems that afflict networks, just as they promise to solve many problems that afflict our roadways, they are not immune from a multitude of practical problems and unexpected consequences. At Google, we have been moving towards highly-automated networking, and along the way we have learned a lot about the risks involved, and how to build robust networks in spite of the wonders of automation. I will talk about some of the problems that are likely to afflict SelfDNs as they enter the real world. Bio: Jeff Mogul works on fast, cheap, reliable, and flexible networking infrastructure for Google. Until 2013, he was a Fellow at HP Labs, doing research primarily on computer networks and operating systems issues for enterprise and cloud computer systems; previously, he worked at the DEC/Compaq Western Research Lab. He received his PhD from Stanford in 1986, an MS from Stanford in 1980, and an SB from MIT in 1979. He is an ACM Fellow. Jeff is the author or co-author of several Internet Standards; he contributed extensively to the HTTP/1.1 specification. He was an associate editor of Internetworking: Research and Experience, and has been the chair or co-chair of a variety of conferences and workshops, including SIGCOMM, OSDI, NSDI, USENIX, HotOS , and ANCS.

1 0

Reminder: Tushar Krishna, tomorrow at 12:30
by Emily C. Lawrence 06 Nov '19

06 Nov '19

CS Colloquium Speaker Tushar Krishna from Georgia Tech Thursday, November 7 - 12:30pm E-Quad, B205 Host: Margaret Martonosi [ https://www.cs.princeton.edu/events/25888 | https://www.cs.princeton.edu/events/25888 ] Enabling Continuous Learning through Synaptic Plasticity in Hardware Ever since modern computers were invented, the dream of creating artificial intelligence (AI) has captivated humanity. We are fortunate to live in an era when, thanks to deep learning (DL), computer programs have paralleled, and in many cases even surpassed human level accuracy in tasks like visual perception and speech synthesis. However, we are still far away from realizing general-purpose AI. The problem lies in the fact that the development of supervised learning based DL solutions today is mostly open loop. A typical DL model is created by hand-tuning the deep neural network (DNN) topology by a team of experts over multiple iterations, followed by training over petabytes of labeled data. Once trained, the DNN provides high accuracy for the task at hand; if the task changes, however, the DNN model needs to be re-designed and re-trained before it can be deployed. A general-purpose AI system, in contrast, needs to have the ability to constantly interact with the environment and learn by adding and removing connections within the DNN autonomously, just like our brain does. This is known as synaptic plasticity. In this talk we will present our research efforts towards enabling general-purpose AI leveraging plasticity in both the algorithm and hardware. First, we will present GeneSys (MICRO 2018), a HW-SW prototype of a closed loop learning system for continuously evolving the structure and weights of a DNN for the task at hand using genetic algorithms, providing 100-10000x higher performance and energy-efficiency over state-of-the-art embedded and desktop CPU and GPU systems. Next, we will present a DNN accelerator substrate called MAERI (ASPLOS 2018), built using light-weight, non-blocking, reconfigurable interconnects, that supports efficient mapping of regular and irregular DNNs with arbitrary dataflows , providing ~100% utilization of all compute units, resulting in 3X speedup and energy-efficiency over our prior work Eyeriss (ISSCC 2016). Finally, time permitting, we will describe our research in enabling rapid design-space exploration and prototyping of hardware accelerators using our dataflow DSL + cost-model called MAESTRO (MICRO 2019). Bio: Tushar Krishna is an Assistant Professor in the School of Electrical and Computer Engineering at Georgia Tech. He also holds the ON Semiconductor Junior Professorship. He has a Ph.D. in Electrical Engineering and Computer Science from MIT (2014), a M.S.E in Electrical Engineering from Princeton University (2009), and a B.Tech in Electrical Engineering from the Indian Institute of Technology (IIT) Delhi (2007). Before joining Georgia Tech in 2015, Dr. Krishna spent a year as a post-doctoral researcher at Intel, Massachusetts. Dr. Krishna’s research spans computer architecture, interconnection networks, networks-on-chip ( NoC ) and deep learning accelerators - with a focus on optimizing data movement in modern computing systems. Three of his papers have been selected for IEEE Micro’s Top Picks from Computer Architecture, one more received an honorable mention, and two have won best paper awards. He received the National Science Foundation (NSF) CRII award in 2018, and both a Google Faculty Award and a Facebook Faculty Award in 2019. He also received the “Class of 1940 Course Survey Teaching Effectiveness” Award from Georgia Tech in 2018.

1 0

Colloquium Speaker: Tushar Krishna, Thursday 11/7 at 12:30pm
by Emily C. Lawrence 31 Oct '19

31 Oct '19

CS Colloquium Speaker Tushar Krishna from Georgia Tech Thursday, November 7 - 12:30pm E-Quad, B205 Host: Margaret Martonosi [ https://www.cs.princeton.edu/events/25888 | https://www.cs.princeton.edu/events/25888 ] Enabling Continuous Learning through Synaptic Plasticity in Hardware Ever since modern computers were invented, the dream of creating artificial intelligence (AI) has captivated humanity. We are fortunate to live in an era when, thanks to deep learning (DL), computer programs have paralleled, and in many cases even surpassed human level accuracy in tasks like visual perception and speech synthesis. However, we are still far away from realizing general-purpose AI. The problem lies in the fact that the development of supervised learning based DL solutions today is mostly open loop. A typical DL model is created by hand-tuning the deep neural network (DNN) topology by a team of experts over multiple iterations, followed by training over petabytes of labeled data. Once trained, the DNN provides high accuracy for the task at hand; if the task changes, however, the DNN model needs to be re-designed and re-trained before it can be deployed. A general-purpose AI system, in contrast, needs to have the ability to constantly interact with the environment and learn by adding and removing connections within the DNN autonomously, just like our brain does. This is known as synaptic plasticity. In this talk we will present our research efforts towards enabling general-purpose AI leveraging plasticity in both the algorithm and hardware. First, we will present GeneSys (MICRO 2018), a HW-SW prototype of a closed loop learning system for continuously evolving the structure and weights of a DNN for the task at hand using genetic algorithms, providing 100-10000x higher performance and energy-efficiency over state-of-the-art embedded and desktop CPU and GPU systems. Next, we will present a DNN accelerator substrate called MAERI (ASPLOS 2018), built using light-weight, non-blocking, reconfigurable interconnects, that supports efficient mapping of regular and irregular DNNs with arbitrary dataflows , providing ~100% utilization of all compute units, resulting in 3X speedup and energy-efficiency over our prior work Eyeriss (ISSCC 2016). Finally, time permitting, we will describe our research in enabling rapid design-space exploration and prototyping of hardware accelerators using our dataflow DSL + cost-model called MAESTRO (MICRO 2019). Bio: Tushar Krishna is an Assistant Professor in the School of Electrical and Computer Engineering at Georgia Tech. He also holds the ON Semiconductor Junior Professorship. He has a Ph.D. in Electrical Engineering and Computer Science from MIT (2014), a M.S.E in Electrical Engineering from Princeton University (2009), and a B.Tech in Electrical Engineering from the Indian Institute of Technology (IIT) Delhi (2007). Before joining Georgia Tech in 2015, Dr. Krishna spent a year as a post-doctoral researcher at Intel, Massachusetts. Dr. Krishna’s research spans computer architecture, interconnection networks, networks-on-chip ( NoC ) and deep learning accelerators - with a focus on optimizing data movement in modern computing systems. Three of his papers have been selected for IEEE Micro’s Top Picks from Computer Architecture, one more received an honorable mention, and two have won best paper awards. He received the National Science Foundation (NSF) CRII award in 2018, and both a Google Faculty Award and a Facebook Faculty Award in 2019. He also received the “Class of 1940 Course Survey Teaching Effectiveness” Award from Georgia Tech in 2018.

1 0

Jordan Fix will present his FPO "Hardware MultiThreaded Transactions: Enabling Speculative MuliThreaded Pipeline Parallelization For Complex Programs" on Tuesday, 11/12/2019 at 10am in CS 402
by Nicki Mahler 28 Oct '19

28 Oct '19

Jordan Fix will present his FPO "Hardware MultiThreaded Transactions: Enabling Speculative MuliThreaded Pipeline Parallelization For Complex Programs" on Tuesday, 11/12/2019 at 10am in CS 402. The members of his committee are: David August (adviser); Readers: David August and David Callahan (Reservoir Labs); Examiners: Margaret Martonosi, David Wentzlaff (ELE), and Andrew Appel. A copy of his thesis is available upon request. Please email ngotsis(a)cs.princeton if you would like a copy of the thesis. Everyone is invited to attend his talk. The talk abstract follows below: Speculation with transactional memory systems helps programmers and compilers produce profitable thread-level parallel programs. Prior work shows that supporting transactions that can span multiple threads, rather than requiring transactions be contained within a single thread, enables new types of speculative parallelization techniques for both programmers and parallelizing compilers. Unfortunately, software support for multi-threaded transactions (MTXs) comes with significant additional inter-thread communication overhead for speculation validation. This overhead can make otherwise good parallelization unprofitable for programs with sizeable read and write sets. Some programs using these prior software MTXs overcame this problem through significant efforts by expert programmers to minimize these sets and optimize communication, capabilities which compiler technology has been unable to achieve to date. Instead, this dissertation makes speculative parallelization less laborious and more feasible through low-overhead speculation validation, presenting the first complete design, implementation, and evaluation of hardware MTXs. Even with maximal speculation validation of every load and store inside transactions of tens to hundreds of millions of instructions, profitable parallelization of complex programs can be achieved. Across 8 benchmarks, this system achieves a geomean speedup of 104% over sequential execution on a multicore machine with 4 cores, with modest increases in power and energy consumption. This work could be used as a building block to enable more realistic automatic parallelization of complex programs, by providing low-overhead, long-running, resilient transactions that support a diverse set of parallel paradigm options.

1 0

Nayana Prasad Nagendra will present her Pre FPO "A Frontend For All Workloads" on Monday, November 4, 2019 at 11:30am in CS 302
by Nicki Mahler 28 Oct '19

28 Oct '19

Nayana Prasad Nagendra will present her Pre FPO "A Frontend For All Workloads" on Monday, November 4, 2019 at 11:30am in CS 302. The members of her committee are as follows: David August (adviser), Tipp Moseley (Google), David Wentzlaff (ELE), and Sharad Malik (ELE). Everyone is invited to attend her talk. The abstract follows below. Warehouse-Scale Computing applications experience an order of magnitude more front-end stalls than desktop applications because their code footprints are well beyond the capacity of today’s processor structures, including instruction caches and branch target buffers. Current approaches to reducing front-end stalls rely, at least in part, on some subset of processor structures having sufficient capacity. Since working set sizes for important applications will continue to grow faster than processor structures (27% Year-over-Year growth in web-search instruction working set size for example vs. 7% for L1 instruction cache), these approaches are not viable. Instead, we propose Predicted Path Prefetching with Confidence Hedging, a front-end architecture that does not rely on the capacity to prevent front-end stalls by also prefetching the path not predicted when confidence in the path prediction is low. Initial results show many fewer instruction cache misses per kilo instructions with similar prefetch costs compared to existing techniques.

1 0

Colloquium Speaker Sham Kakade TODAY at 12:30pm
by Emily C. Lawrence 24 Oct '19

24 Oct '19

CS Colloquium Speaker Sham Kakade from University of Washington Thursday, October 24 - 12:30pm Friend Center Convocation Room Host: Sanjeev Arora [ https://www.cs.princeton.edu/events/25891 | https://www.cs.princeton.edu/events/25891 ] Representation, Modeling, and Optimization in Reinforcement Learning Reinforcement learning is now the dominant paradigm for how an agent learns to interact with the world. The approach has led to successes ranging across numerous domains, including game playing and robotics, and it holds much promise in new domains, from self-driving cars to interactive medical applications. Some of the central challenges are: -Representational learning: does having a good representation of the environment permit efficient reinforcement learning? -Modeling: should we explicitly build a model of our environment or, alternatively, should we directly learn how to act? -Optimization: in practice, deployed algorithms often use local search heuristics. Can we provably understand when these approaches are effective and provide faster and more robust alternatives? This talk will survey a number of results on these basic questions. Throughout, we will highlight the interplay of theory, algorithm design, and practice. Bio: Sham Kakade is a Washington Research Foundation Data Science Chair, with a joint appointment in the Department of Computer Science and the Department of Statistics at the University of Washington, and is a co-director for the Algorithmic Foundations of Data Science Institute. He works on the mathematical foundations of machine learning and AI. Sham's thesis helped in laying the foundations of the PAC-MDP framework for reinforcement learning. With his collaborators, his additional contributions include: one of the first provably efficient policy search methods, Conservative Policy Iteration, for reinforcement learning; developing the mathematical foundations for the widely used linear bandit models and the Gaussian process bandit models; the tensor and spectral methodologies for provable estimation of latent variable models (applicable to mixture of Gaussians, HMMs, and LDA); the first sharp analysis of the perturbed gradient descent algorithm, along with the design and analysis of numerous other convex and non-convex algorithms. He is the recipient of the IBM Goldberg best paper award (in 2007) for contributions to fast nearest neighbor search and the best paper, INFORMS Revenue Management and Pricing Section Prize (2014). He has been program chair for COLT 2011. Sham was an undergraduate at Caltech, where he studied physics and worked under the guidance of John Preskill in quantum computing. He then completed his Ph.D. in computational neuroscience at the Gatsby Unit at University College London, under the supervision of Peter Dayan. He was a postdoc at the Dept. of Computer Science, University of Pennsylvania , where he broadened his studies to include computational game theory and economics from the guidance of Michael Kearns. Sham has been a Principal Research Scientist at Microsoft Research, New England, an associate professor at the Department of Statistics, Wharton, UPenn , and an assistant professor at the Toyota Technological Institute at Chicago.

1 0

Reminder: Colloquium Speaker Sham Kakade tomorrow at 12:30pm
by Emily C. Lawrence 23 Oct '19

23 Oct '19

CS Colloquium Speaker Sham Kakade from University of Washington Thursday, October 24 - 12:30pm Friend Center Convocation Room Host: Sanjeev Arora [ https://www.cs.princeton.edu/events/25891 | https://www.cs.princeton.edu/events/25891 ] Representation, Modeling, and Optimization in Reinforcement Learning Reinforcement learning is now the dominant paradigm for how an agent learns to interact with the world. The approach has led to successes ranging across numerous domains, including game playing and robotics, and it holds much promise in new domains, from self-driving cars to interactive medical applications. Some of the central challenges are: -Representational learning: does having a good representation of the environment permit efficient reinforcement learning? -Modeling: should we explicitly build a model of our environment or, alternatively, should we directly learn how to act? -Optimization: in practice, deployed algorithms often use local search heuristics. Can we provably understand when these approaches are effective and provide faster and more robust alternatives? This talk will survey a number of results on these basic questions. Throughout, we will highlight the interplay of theory, algorithm design, and practice. Bio: Sham Kakade is a Washington Research Foundation Data Science Chair, with a joint appointment in the Department of Computer Science and the Department of Statistics at the University of Washington, and is a co-director for the Algorithmic Foundations of Data Science Institute. He works on the mathematical foundations of machine learning and AI. Sham's thesis helped in laying the foundations of the PAC-MDP framework for reinforcement learning. With his collaborators, his additional contributions include: one of the first provably efficient policy search methods, Conservative Policy Iteration, for reinforcement learning; developing the mathematical foundations for the widely used linear bandit models and the Gaussian process bandit models; the tensor and spectral methodologies for provable estimation of latent variable models (applicable to mixture of Gaussians, HMMs, and LDA); the first sharp analysis of the perturbed gradient descent algorithm, along with the design and analysis of numerous other convex and non-convex algorithms. He is the recipient of the IBM Goldberg best paper award (in 2007) for contributions to fast nearest neighbor search and the best paper, INFORMS Revenue Management and Pricing Section Prize (2014). He has been program chair for COLT 2011. Sham was an undergraduate at Caltech, where he studied physics and worked under the guidance of John Preskill in quantum computing. He then completed his Ph.D. in computational neuroscience at the Gatsby Unit at University College London, under the supervision of Peter Dayan. He was a postdoc at the Dept. of Computer Science, University of Pennsylvania , where he broadened his studies to include computational game theory and economics from the guidance of Michael Kearns. Sham has been a Principal Research Scientist at Microsoft Research, New England, an associate professor at the Department of Statistics, Wharton, UPenn , and an assistant professor at the Toyota Technological Institute at Chicago.

1 0

Dingli Yu will present his General Exam on Thursday, October 24, 2019 at 9:30am in CS 401.
by Nicki Mahler 22 Oct '19

22 Oct '19

Dingli Yu will present his General Exam on Thursday, October 24, 2019 at 9:30am in CS 401. The members of his committee are as follows: Sanjeev Arora (adviser), Jason D. Lee (ELE), and Jia Deng. Everyone is invited to attend his talk, and those faculty wishing to remain for the oral exam following are welcome to do so. His abstract and reading list follow below. Title: Harnessing the Power of Infinitely Wide Deep Nets Abstract: Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to compute the NTK, as well as its convolutional counterparts (CNTK), appears in Arora et al. (2019a), yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of the corresponding CNN architecture (best figure being around 78%). Here, we show that NTK and CNTK or their enhanced version perform strongly in the following tasks. 1. On a standard testbed of classification/regression tasks from the UCI database, NTK SVM beats the previous gold standard, Random Forests (RF), and also the corresponding finite nets. 2. On VOC07 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al., 2019), replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance. 3. On CIFAR-10 with 10 – 640 training samples, CNTK consistently beats ResNet-34 by 1% - 3%. On the whole CIFAR-10 dataset, CNTK achieves 89% accuracy, matching the performance of AlexNet (Krizhevsky et al., 2012) by incorporating horizontal flip data augmentation, a pre-processing technique proposed by Coates et al. (2011) and a new operation called Local Average Pooling (LAP) which preserves efficient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts. Reading list: 1. Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, 2018 2. Peter L Bartlett, and Shahar Mendelson, Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463–482, 2002. 3. Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint arXiv:1904.11955, 2019. 4. Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.08584, 2019b. 5. Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations, 2019. 6. Simon S Du, Jason D Lee, Haochuan Li, LiweiWang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint arXiv:1811.03804, 2018. 7. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019. 8. Matthew Olson, Abraham Wyner, and Richard Berk. Modern neural networks generalize on small data sets. In Advances in Neural Information Processing Systems, pp. 3619-3628, 2018. 9. Manuel Fernandez-Delgado, Eva Cernadas, Senen Barro, and Dinani Amorim. Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research, 15(1):3133-3181, 2014. 10. Julien Mairal, Piotr Koniusz, Zaid Harchaoui, and Cordelia Schmid. Convolutional kernel networks.In Advances in neural information processing systems, pages 2627–2635, 2014. 11. Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215–223, 2011. 12. Shuxiao Chen, Edgar Dobriban, and Jane H Lee. Invariance reduces variance: Understanding data augmentation in deep learning and beyond. arXiv preprint arXiv:1907.10905, 2019.

1 0