Machine vision relies on images captured by low-level optics and hardware, however, image acquisition and image processing are often considered as disjoint tasks to be solved in isolation. Conventional cameras are designed to capture the most visually pleasing images to human consumers without concern for the post-processing algorithms, and similarly, computer vision algorithms are often expected to perform equally well regardless of the camera used. In this talk, I present my recent research that aims to bridge this gap by learning the imaging stack in an end-to-end fashion. First, I demonstrate this framework by designing a diffractive optic imager for capturing high-dynamic range pictures in a single shot. Next, I introduce an end-to-end camera design system that jointly optimizes compound lenses, image signal processors (ISP), and downstream post-processors with respect to high-level computer vision loss functions. Finally, I present a neural nano-optic imager that captures high-quality RGB images on par with commercial compound lenses but is over 500000 times smaller in size. This state-of-the-art performance was achieved through end-to-end learning of the nano-optic's physical structure in conjunction with a novel, neural feature-based image reconstruction algorithm.