86-375/675 (15-387) Computational Perception

Carnegie Mellon University

Spring 2018

Course Description

The perceptual capabilities of even the simplest biological organisms are far beyond what we can achieve with machines. Whether you look at sensitivity, robustness, or sheer perceptual power, perception in biology just works, and works in complex, ever changing environments, and can pick up the most subtle sensory patterns. Is it the neural hardware? Does biology solve fundamentally different problems? What can we learn from biological systems and human perception?

In this course, we will first study the biological and psychological data of biological perceptual systems, particularly the visual system, in depth, and then apply computational thinking to investigate the principles and mechanisms underlying natural perception. The course will focus primarily on visual perception. You will learn how to reason scientifically and computationally about problems and issues in perception, how to extract the essential computational properties of those abstract ideas, and finally how to convert these into explicit mathematical models and computational algorithms. The course is targeted to neuroscience and psychology students who are interested in learning computational thinking, and computer science and engineering students who are interested in learning more about the neural basis of perception. Prerequisites: First year college calculus, some linear algebra, probability theory and programming experience are desirable.

Course Information

GHC 5th Floor Citadel Teaching Commons (Friday 4:30-5:30)

Instructors	Office (Office hours)	Email (Phone)
Tai Sing Lee (Professor)	Mellon Inst. Rm 115 (Friday 1:30-2:30)	tai@cnbc.cmu.edu (412-268-1060)
Ge (Summer) Huang	Mellon Inst. Rm 115 (Tuesdsy 4-5 p.m)	hgesummer@gmail.com
Shefali Umrania		sumrania@andrew.cmu.edu

Class location and time: Mellon Institute Room 130. Monday/Wednesday 1:30 p.m - 2:50 p.m.
Website: http://www.cnbc.cmu.edu/~tai/cp18.html (course info and readings )
Canvas: Lecture materials and Information would be on Canvas.

Handouts in Blackboard.
Frisby and Stone Seeing: The computational approach to biological vision . MIT Press, 2010 (recommended).
Supplementary Simon Prince Computer Vision: Models, Learning, and Inference . Cambridge University Press , 2012. Downloadable at: http://www.computervisionmodels.com. More relevant to graduate students.

Classroom Etiquette

Please turn OFF your laptop, cell phones or any other electronic devices in the classroom.

Grading Scheme

Evaluation	% of Grade
Assignments	60
Midterm	10
Final Exam	10
class Participation/Quiz	10
Term paper and Presentation	10
Term project	10 for replacing assignments or midterm only
675 Term Project	Required

Grading scheme: A: > 88, B: > 75. C: > 65.

Assignments

6 homework assignments, each involves Matlab (or Python) exercises as well as reading/research assignments.

Term Project

A term paper involves in-depth reading (of a group of 5 papers) on a particular topic in computational perception. Students can work in a team of two, and ultimately produce a term paper that is 6-10 pages long. All term papers will be published on canvas, but the top 5 term papers will be selected for presentation, with 3 extra points of bonus credit.
Term project is optional for undergraduate versions of the course. An undergraduate student can use the OPTIONAL term project (with a max score of 10 points) to replace any of the assignments or exam, except the Final Quiz and the in-class quizzes. It will require a 6 pages written final report and a presentation to the class. Matlab codes and additional output should also be submitted as supplementary materials in a different pdf/doc file and/or matlab zip files.
Term project is required of 675 students. It is anticipated to take 20-30 hours to complete. Students can work on issues of their choice that are not covered in the course, but should discuss project proposals with the faculty instructors in advance for approval.

Examinations

There will be a midterm and a final quiz (10 points each) to test understanding of the materials covered discussed in the lectures and the reading assignments.
There will be 26 points worth of in-class Piazza pool or quiz that keep track of attendance and test you on your understanding of the lecture materials. Each quiz worth 0.5 points and you can earn up to 10 points total.

Late Policy

Each student will have 7 days grace period for late homework. This grace period can be used for one or multiple assignments or the term paper. Use it wisely and do not ask for more. Blackboard submission after the starting of class time is considered late.

Syllabus

Date	Lecture Topic	Relevant Readings	Assignments
	SENSORY CODING
W 1/17	1. Philosophy of computational approach	ch. 1, Marr
M 1/22	2. Vision and retina	ch. 6 (retina), Meister
W 1/24	3. Linear System	ch 3 (receptive fields), Simoncelli	Homework 1
M 1/29	4. Representation	ch 5 (edges features)
W 1/31	5. Frequency analysis	Ch 4 (frequency analysis)
M 2/5	6. Principle components	ch 9 (V1, neurons), Shlens
W 2/7	7. Source separation	Fodiak, Hyvarinen, Olshausen, Sejnowksi	Homework 2
	EARLY PERCEPTUAL INFERENCE
M 2/12	8. Bayesian inference	ch 13 (inference)
W 2/14	9. What and where	Sompolinsky, retina, V1, object detection
M 2/19	10. Lightness and color	ch 16, Land, Horn, Morel
W 2/21	11. Intrinsic images and Retinex	ch 17, Adelson, Weiss, Freeman	Homework 3;
M 2/26	12. Features and segmentaiton	ch 7 (figure/ground)
M 2/28	Midterm
	MID-LEVEL VISION
W 3/05	13. Texture and surfaces	ch 2, Julesz, Simoncelli
W 3/07	15. Perceptual Organization	ch 7	Homework 4
M 3/12	Spring Break
W 3/14	Spring Break
M 3/19	16. Visual Hierarchy	ch 10 (brain maps), Van Essen
W 3/21	17. Depth and 3D	ch 18,19
M 3/26	18. Shape from Shading	Horn, Zucker
W 3/28	19. Style and Contents	Bethge	Homework 5
M 4/02	20. Motion Perception	ch 14,15, Weiss
W 4/04	22. Cue Integration	ch 20, Bank, Ma
	OBJECT AND SCENES
M 4/09	23. Belief and Association	Hinton, Lee, Gilbert, Geisler
W 4/11	24. Context and scenes	Torrelba, Oliva	Homework 6
M 4/16	25. Object recognition	ch 8 (objects), Sinha, LeCun, Hinton
W 4/18	26. Composition and Grammar	ch 11 (complexity) Yuille, Mumford
Th 4/19	Spring Carnival 20/21
M 4/23	27. Attention and routing	ch 22, Hinton, Arathon, Olshausen
W 4/25	28. Review		Homework 6 due
M 4/30	29. Final Quiz		Term paper due
W 5/02	30. Project presentations	Term paper due
M 5/14	FCE deadline
M 5/X	Final Exam (hold)

Supplementary Readings (Relevant to Understanding Lectures)

Part 1: Vision, Perceptual Systems and Philosophy

Overview of the visual system of cats and monkeys, and the philosophy of computational approaches:
Teller, D. (2016) Vision and Visual System 130 MB) Chapter 13, 16 and 17
Marr, D. (1982) Chapter 1: The Philosophy and the Approach Vision 8-38 .
Gollisch and Meister (2010) Eye Smarter than Scientists Believed: Neural Computations in Circuits of the Retina Neuron 65: 150-164
Van Essen, Anderson and Felleman (1992) Information processing in primate visual systems: an integrated approach Science 5043: 419-423.
Fellman and Van Essen ( 1991) Distributed Hierarchical Processing the the Primate Cerebral Cortex Cerebral Cortex 1-47.

Part 2: Neural Codes, Features and Representational Learning

Also cover some bread and butter basic math of computaitonal perception (neuroscience and computer vision) -- Relevant for problem set 1 and 2:
Olshausen and Field (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images Nature 381: 607-609.
Olshausen and Field (2004) Sparse coding of sensory inputs Current Opinions in Neurobiology 14:481-487.
Vinje, W.E., Gallant, J.L. Sparse Coding and Decorrelation in Primary Visual Cortex During Natural Vision, Science Vol. 287, Issue 5456, pp. 1273-1276.
Vinje and Gallant (2002) Natural stimulation of the non-classical receptive field increases information transmission efficiency in V1 J. Neuroscience 22(7): 2904-2915.
Barlow (1954) Possible principles underying the transformation of sensory messages Sensory Communication MIT Press, p217-234.
Barlow (2001) Redundancy reduction revisited Network: Computation in Neural Systems 12: 241-253.
Shlens: PCA tutorial (2003) online
Simoncelli: Geometric review of linear algebra (2017)) online
Hyvarinen and Oja (1999) Indpendent component analysis: tutorial On-line tutorial https://research.ics.aalto.fi/ica/fastica/

Part 3: Lightness and color perception, Retinex

Some classic papers by Land, Horn, Morel, Adelson, Weiss and Freeman (retinex and intrinsice images) -- Relevant for problem set 3
Adelson, Ed, (2000) Lightness Perception and Lightness Illusion The New Cognitive Neuroscience, Gazzaniga ed. MIT Press. (2000).
Land, E, (1977) The retinex theory of color vision Scientific America 1977
Horn, B, (1974) Determininng lightness from an image Computer Graphics and Image Procwssing. 1974
Morel JM, Petro AB, Sbert C. (2010) IEEE Trans Image Process. 19(11) 2825-37.
Tappen, Freeman and Adelson (2005) Recovering intrinsic images from a single image IEEE PAMI. 27(9): 1459-1472.

Part 4: Mid-level vision: Texture, depth and motion Perception

Some classic papers by Simoncelli, Julesz -- Relevant for problem set 4
Weiss, Simoncelli and Adelson (2002) Motion Illusion as optimal percepts Nature Neuroscience 5(6): 598-.
Freeman and Simoncelli (2011) Metamers of the ventral stream. Nature Neuroscience 14(9): 1195-.
Freeman et al. (2013) A functional and perceptual signature of the second visual area in primate Nature Neuroscience 16(7): 974-

Part 5: Visual Hierarchy, Object recognition, Abstract Representations

Papers by Fukushima, LeCun and Hinton, Geman and Rust in invariance and robustness, DiCarlos -- Relevant for problem set 5
S. Geman (1999) Invariance and selectivity in the ventral visual pathway. Journal of Physiology Paris 100: 212-224.
LeCun, Bengio and Hinton (2016) Deep Learning Nature 521: 436-444.
Krizhevsky, I. Sutskever, and G. E. Hinton, (2012) ImageNet Classification with Deep Convolutional Neural Networks.” NIPS online.
M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks, ECCV. online.

Part 6: Generative models, Art, Abstract Representations

Papers by Bethge, Deep generative models, Deep dream and others , -- Relevant for problem set 5
Salakhutdinov (2015) Learning Deep Generative Models Annual REview of Statistics and its applications 2: 361-85
Gatys, L, Ecker, AS, Bethge, Matthias (2018) Texture and art with deep neural networks Trends in Neuroscience Submitted.
Gatys, L, Ecker, AS, Bethge, Matthias (2016) Image style transfer using CNN CVPR (code) https://github.com/leongatys/NeuralImageSynthesis

Part 7: Belief, memories and Association

Papers by Hinton -- Relevant for problem set 6.
Ramalingam, McManus, Li and Gilbert (2013) Top-down modulation of lateral interactions in visual cortex Journal of Neuroscience 33(5): 1773-1789.
Li and Gilbert (2014) Top-down influences on visual processing Nature Review Neuroscience 14: 350.
Yan et al. (2014) Perceptual training continously refines neuronal population codes in primary visual cortex Nature Neuroscience 17(1); 1380.
Hinton and Sejnowski (1984) Learning and relearning in Boltzmann machine Parallel Distributed Processing
Salakhutdinov (2015) Learning deep generative models Annual Review of Statistics and Its Application 2:361-85.
Zhang et al. (2016) Relating functional connectivity in V1 neural circuits and 3D antural scenes using Boltzmann machines Vision Research 120: 121-131.
Shushruth et al. (2012) Strong Recurrent Networks Compute the Orientation Tuning of Surround Modulation in the Primate Primary Visual Cortex. J. Neuroscience 32(1): 308-321.

Part 8: Composition and Grammar

Papers by Zhu and Mumford, Alan Yuille, and Science paper. -- term projects
E. Bienenstock, S. Geman, and D. Potter. Compositionality, MDL Priors, and Object Recognition NIPS Advances in Neural Information Processing Systems 9. 1998.
L. Zhu, C. Lin, H. Huang, Y. Chen, A.L. Yuille. Unsupervised Structure Learning: Hierarchical Recursive Composition, Suspicious Coincidence and Competitive Exclusion. Proceedings of the European Conference on Computer Vision. ECCV. (see also another paper by Zhu and Yuille with Bill Freeman.) 2008.
S.C. Zhu and D. Mumford (2006) A Stochastic Grammar of Images Foundations and Trends in Computer Graphics and Vision 2(4): 259-362.

Part 9: Attention, Eye Movement and Routing

Papers by Hinton on capsule network, particle map-seeking circuit, and Olshausen and Lee and Deco -- attentional saliency models - term projects
Olshausen, Anderson and Van Essen (1995) Mutliscale dynamic routing circuit for forming size- and position-invariant object recognition J. Neuroscience 2:45-62.
David Arathorn (1995) Computation in the Higher Visual Cortices: Map-Seeking Circuit Theory and Application to Machine Vision online. Long version (book) by Stanford University Press as Map-Seeking Circuit.
Sara Sabour, Nicholas Frosst and Geoffrey Hinto (2017) NIPS

Questions or comments: contact Tai Sing Lee
Last modified: Jan 2018, Tai Sing Lee