86-375/675 (15-387) Computational Perception

Grading Scheme 86-675

Grading Scheme 86-375

Assignments

• 5 homework assignments, each involves Matlab (or Python) exercises as well as reading/research assignments.

Term Project

• A term paper involves in-depth reading (of a group of 5 papers) on a particular topic in computational perception. A student must work by himself, and can use this to replace up to 10 points of any assignment or exam grade. For example, it can replace the entire midterm, or it can earn up to 10 points of the 15-point Final Exam. The term paper is limited to 8 pages including reference. All term papers will be published on Canvas, but the top 3 term papers will be selected for presentation, with 3-5 extra points of bonus credit.
• Term project is required for 86-675 and counts for 30 points of a total grade of 130 points. If you are interested in doing a term project, you should switch to 86-675. A term project must involve some computational experiments, using either downloadable softwares or programs you develop. All the codes should be documented and archived in github and submitted together with a term paper (8 pages) materials in a different pdf/doc file and/or matlab zip files. Term project should take about 30 hours to complete. Each student should work on his/her own project. A team project might be approved but the project is expected to be more substantial and ambitious (60 hours of work) and each student should have a well defined task and component in the project and could receive a different grade. A project proposal is due on October 16 with literature research and preliminary results and thus students are encouraged to discuss project ideas with professor from the very beginning of the semester.

Examinations

• There will be a midterm (10 points) and a final exam (15 points) to test understanding of the materials covered or discussed in the lectures and the reading assignments.
• There will be 15 points worth of in-class Piazza pool that keep track of attendance and test you on your understanding of the lecture materials. Each partcipation worths 1 point and you can earn up to 10 points total.

Late Policy

• Each student will have 7 days grace period for late homework. This grace period can be used for one or multiple assignments or the term paper. Use it wisely and you cannot ask for more. CANVAS submission after the starting of class time of due day is considered late by one day.

Syllabus

Supplementary Readings (Relevant to Understanding Lectures)

Part 1: Vision, Perceptual Systems and Philosophy

• Overview of the visual system of cats and monkeys, and the philosophy of computational approaches:
• Teller, D. (2016) Vision and Visual System 130 MB) Chapter 13, 16 and 17
• Marr, D. (1982) Chapter 1: The Philosophy and the Approach Vision 8-38 .
• Gollisch and Meister (2010) Eye Smarter than Scientists Believed: Neural Computations in Circuits of the Retina Neuron 65: 150-164
• Van Essen, Anderson and Felleman (1992) Information processing in primate visual systems: an integrated approach Science 5043: 419-423.
• Fellman and Van Essen ( 1991) Distributed Hierarchical Processing the the Primate Cerebral Cortex Cerebral Cortex 1-47.

Part 2: Neural Codes, Features and Representational Learning

• Also cover some bread and butter basic math of computaitonal perception (neuroscience and computer vision) -- Relevant for problem set 1 and 2:
• Olshausen and Field (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images Nature 381: 607-609.
• Olshausen and Field (2004) Sparse coding of sensory inputs Current Opinions in Neurobiology 14:481-487.
• Vinje, W.E., Gallant, J.L. Sparse Coding and Decorrelation in Primary Visual Cortex During Natural Vision, Science Vol. 287, Issue 5456, pp. 1273-1276.
• Vinje and Gallant (2002) Natural stimulation of the non-classical receptive field increases information transmission efficiency in V1 J. Neuroscience 22(7): 2904-2915.
• Barlow (1954) Possible principles underying the transformation of sensory messages Sensory Communication MIT Press, p217-234.
• Barlow (2001) Redundancy reduction revisited Network: Computation in Neural Systems 12: 241-253.
• Shlens: PCA tutorial (2003) online
• Simoncelli: Geometric review of linear algebra (2017)) online
• Hyvarinen and Oja (1999) Indpendent component analysis: tutorial On-line tutorial https://research.ics.aalto.fi/ica/fastica/

Part 3: Lightness and color perception, Retinex

• Some classic papers by Land, Horn, Morel, Adelson, Weiss and Freeman (retinex and intrinsice images) -- Relevant for problem set 3
• Adelson, Ed, (2000) Lightness Perception and Lightness Illusion The New Cognitive Neuroscience, Gazzaniga ed. MIT Press. (2000).
• Land, E, (1977) The retinex theory of color vision Scientific America 1977
• Horn, B, (1974) Determininng lightness from an image Computer Graphics and Image Procwssing. 1974
• Morel JM, Petro AB, Sbert C. (2010) IEEE Trans Image Process. 19(11) 2825-37.
• Tappen, Freeman and Adelson (2005) Recovering intrinsic images from a single image IEEE PAMI. 27(9): 1459-1472.

Part 4: Mid-level vision: Texture, depth and motion Perception

• Some classic papers by Simoncelli, Julesz -- Relevant for problem set 4
• Weiss, Simoncelli and Adelson (2002) Motion Illusion as optimal percepts Nature Neuroscience 5(6): 598-.
• Freeman and Simoncelli (2011) Metamers of the ventral stream. Nature Neuroscience 14(9): 1195-.
• Freeman et al. (2013) A functional and perceptual signature of the second visual area in primate Nature Neuroscience 16(7): 974-

Part 5: Visual Hierarchy, Object recognition, Abstract Representations

• Papers by Fukushima, LeCun and Hinton, Geman and Rust in invariance and robustness, DiCarlos -- Relevant for problem set 5
• S. Geman (1999) Invariance and selectivity in the ventral visual pathway. Journal of Physiology Paris 100: 212-224.
• LeCun, Bengio and Hinton (2016) Deep Learning Nature 521: 436-444.
• Krizhevsky, I. Sutskever, and G. E. Hinton, (2012) ImageNet Classification with Deep Convolutional Neural Networks.” NIPS online.
• M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks, ECCV. online.

Part 6: Generative models, Art, Abstract Representations

• Papers by Bethge, Deep generative models, Deep dream and others , -- Relevant for problem set 5
• Salakhutdinov (2015) Learning Deep Generative Models Annual REview of Statistics and its applications 2: 361-85
• Gatys, L, Ecker, AS, Bethge, Matthias (2018) Texture and art with deep neural networks Trends in Neuroscience Submitted.
• Gatys, L, Ecker, AS, Bethge, Matthias (2016) Image style transfer using CNN CVPR (code) https://github.com/leongatys/NeuralImageSynthesis

Part 7: Belief, memories and Association

• Papers by Hinton -- Relevant for problem set 6.
• Ramalingam, McManus, Li and Gilbert (2013) Top-down modulation of lateral interactions in visual cortex Journal of Neuroscience 33(5): 1773-1789.
• Li and Gilbert (2014) Top-down influences on visual processing Nature Review Neuroscience 14: 350.
• Yan et al. (2014) Perceptual training continously refines neuronal population codes in primary visual cortex Nature Neuroscience 17(1); 1380.
• Hinton and Sejnowski (1984) Learning and relearning in Boltzmann machine Parallel Distributed Processing
• Salakhutdinov (2015) Learning deep generative models Annual Review of Statistics and Its Application 2:361-85.
• Zhang et al. (2016) Relating functional connectivity in V1 neural circuits and 3D antural scenes using Boltzmann machines Vision Research 120: 121-131.
• Shushruth et al. (2012) Strong Recurrent Networks Compute the Orientation Tuning of Surround Modulation in the Primate Primary Visual Cortex. J. Neuroscience 32(1): 308-321.

Part 8: Composition and Grammar

• Papers by Zhu and Mumford, Alan Yuille, and Science paper. -- term projects
• E. Bienenstock, S. Geman, and D. Potter. Compositionality, MDL Priors, and Object Recognition NIPS Advances in Neural Information Processing Systems 9. 1998.
• L. Zhu, C. Lin, H. Huang, Y. Chen, A.L. Yuille. Unsupervised Structure Learning: Hierarchical Recursive Composition, Suspicious Coincidence and Competitive Exclusion. Proceedings of the European Conference on Computer Vision. ECCV. (see also another paper by Zhu and Yuille with Bill Freeman.) 2008.
• S.C. Zhu and D. Mumford (2006) A Stochastic Grammar of Images Foundations and Trends in Computer Graphics and Vision 2(4): 259-362.

Part 9: Attention, Eye Movement and Routing

• Papers by Hinton on capsule network, particle map-seeking circuit, and Olshausen and Lee and Deco -- attentional saliency models - term projects
• Olshausen, Anderson and Van Essen (1995) Mutliscale dynamic routing circuit for forming size- and position-invariant object recognition J. Neuroscience 2:45-62.
• David Arathorn (1995) Computation in the Higher Visual Cortices: Map-Seeking Circuit Theory and Application to Machine Vision online. Long version (book) by Stanford University Press as Map-Seeking Circuit.
• Sara Sabour, Nicholas Frosst and Geoffrey Hinto (2017) NIPS

Journal Club Reading (September 13) Computations in the Retina

• Some current papers on retina (using connectome, deep learning as well as standard neurophysiology) by Ganguli, Seung and Meister
• Gollisch and Meister (2010) Eye Smarter than Scientists Believed: Neural Computations in Circuits of the Retina Neuron 65: 150-164
• Green, Kim and Seung, "Analogous Convergence of Sustained and Transient Inputs in Parallel On and Off Pathways for Retinal Motion Computation" Cell Reports 14, 1892–1900, March 1, 2016
• Ocko, Lindsey, Ganguli and Deny (2018) The emergence of multiple retinal cell types throughefficient coding of natural movies NIPS
• Maheswaranathan, Kastner, Baccus and Ganguli (218) Infeerring hidden struture in multilayered neural circuits. PLOS Computaitonal Biology 14(8):e1006291
• Maheswaranthan, .... Ganguli and Baccus (2018) Deep learning models reveal internal structure and diverse computations in the retina under natural scenes bioRxiv, June 8, 2018.
• McIntosh, Maheswaranathan, Nayebi, Ganguli and Baccus (2016) Deep Learning Models of the Retinal Response to Natural Scenes NIPS

Journal Club Reading (September 20) Neural Codes: Sparse versus Redundant Codes

• There seems to be rather broad acceptance of the efficient sparse code theory (Olshausen, Jack Gallant and others -- see Par2 literature above). Michael Berry and Bill Bialek at Princeton seems to argue for an alternative code -- redundancy code. This is a debate that worths some consideration. You may also check out this webpage at Princeton: https://www.princeton.edu/~wbialek/rome/lecture3.htm.
• J. Puchalla, E. Schneidman, R. Harris, and M.J. Berry II, “Redundancy in the Population Code of the Retina”, Neuron 46: 493-504 (2005
• G. Tkacik, O. Marre, D. Amodei, W. Bialek, and M.J. Berry II, “Searching for Collective Behavior in Networks of Real Neurons”, PLoS Computational Biology 10(1): e1003408 (2014).
• J.S. Prentice, O. Marre, M.L. Ioffe, A.R. Loback, G. Tkacik, and M.J. Berry II, “Error-Robust Modes of the Retinal Population Code”, PLoS Computational Biology 12(11): e100514 (2016).
• E. Schneidman, M.J. Berry II, R. Segev, and W. Bialek, “Weak Pairwise Correlations Imply Strongly Correlated Network States in a Neural Population”, Nature 440: 1007-1012 (2006).

Journal Club Reading (September 27) Recurrent Network and Neural Control

• We will read a couple exciting papers coming out of Jim DiCarlos lab at MIT.
• Bashivan P, Kar K, DiCarlo JJ. (2019) Neural population control via deep image synthesis. Science. 364(6439)
• Kar K, Kubilius J, Schmidt K, Issa EB, DiCarlo JJ. Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature Neuroscience. 2019. doi: 10.1038/s41593-019-0392-5.

Journal Club Reading (October 4) Predictive Coding

• We will investigate papers on "predictive coding". What is it? How is this different from sparse coding?
• Bastos, A.M., et al. (2012) Canonical microcircuits for predictive coding. Neuron, 2012. 76(4): p. 695-711.
• Rao, R.P. and D.H. Ballard (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 1999. 2(1): p. 79-87.
• Spratling, M.W., A review of predictive coding algorithms. Brain and cognition, 2017. 112: p. 92-97.
• Miller, K.D., Canonical computations of cerebral cortex. Current opinion in neurobiology, 2016. 37: p. 75-84..

Journal Club Reading (October 11) Computational theory on Perception of Beauty

• Schmidhuber Jurgen (inventor of LSTM) proposed that our ability to compress and to predict is related to our perception of beauty. Things that are completely predictable are boring, not predictable are discomforting. There is an apparent sweet spot between predictiblity and surprise that engenders interest and the perception of beauty. Mingchi Hou will present some papers on these ideas.
• Schmidhuber Jurgen. (1997) Low complexity art.
• Schmidhuber Jurgen. (2008) Driven by Compression Progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, and jokes!

Journal Club Reading (October 18) Drawing and Perception

• Erik Ulberger will present an interesting article by Patrick Cavanagh on what we draw or what drawing tells us about the visual brain.
• Bilge Sayim and Patrick Cavangah (2011) What line draws reveal about the visual brain.

Journal Club Reading (October 25) Perceptual Rivelary and V1 circuits

• Rusi Su will present papers on Perceptual Rivlary and the potential V1 circuits underlying it.
• Kovacs, I., Papathomas, T., Yang, M. and Feher, A. (1996). When the brain changes its mind: Interocular grouping during binocular rivalry. Proceedings of the National Academy of Sciences, 93(26), pp.15508-15511.
• Ganna Palagina, Jochen F. Meyer, and Stelios M. Smirnakis (2018) Inhibitory Units: An Organizing Nidus for Feature-Selective SubNetworks in Area V1. J Neuroscience.

Journal Club Reading (November 8) Impact of association and memory on perception

• Hou Mingchi will present a paper by Tom Albright on the role of memory and assocaiton on perception, and the neural evidence for such phenomena.
• Tom Albright. (2012) On the Perception of Probable Things: Neural Substrates of Associative Memory, Imagery, and Perception Neuron 74. 227-245.

Journal Club Reading (Nov 15) Emotion and perception

• Does emotion influence our perception? Julian Cheng presents.
• Niedenthal and Wood (2019) Does emotion influence visual perception? Depends on how you look at it. Cognition and Emotion, 33 (1),77-84.
• Gayet, Paffen, Belopolsky and Theeuwes and Van Der Stigchel (2016) Visual input signaling threat gains preferential access to awareness in a breaking continuous flash suppression paradigm. Cognition 149, 77-83.
• Siegel, Wormwood, Quigley and Barrett. (2018) Seeing what you feel: affect drives visual percpetion of structurally neural faces. Psychological Science 29(4):496-503.
• Phelps, Ling and Carrasco M. (2006) Emotion facilitates perception and potentiates the perceptual benefits of attention. Psychological Science. 17(4): 292-299.

Journal Club Reading (November 22) How Sensorymotor feedback or Action can Affect Our Perception

• Last week, Julian in his presentation mentioned an interesting -- inhibiting a person's facial expression expression by putting a wax on his face seems to disrupt that person's visual perception of facial expression. We would like to explore if there is a link between action and perception. In psychological tradition, the connections between action and perception are not new, for example, in Gibson's theory of affordance.
• Wood, A., Lupyan G., Sherrin, S, Niedenthal, P. Alterating sensorimotor feedback disrupts visual discrimination of facial expressions. Psychonomic Bulletin & Review. Volume 23, Issue 4, pp 1150–1156.
• Gallese Vittorio (2013) Mirror Neurons and the Perception–Action Link. The Oxford Handbook of Cognitive Neuroscience: Volume 2: The Cutting Edges.
• Gibson, J. J. (1979). The ecological approach to visual perception (Boston: Houghton Mifflin)
• Loomis J M, Philbeck J W, (2008). Measuring spatial perception with spatial understanding and action”, in Embodiment, Ego-Space, and Action Eds. R L Klatzky, B MacWhinney, M Behrmann (Cambridge, MA: The MIT Press) pp. 1–44, here: p. 33.
• Philbeck and Witt (2015) Action-Specific Influences on Perception and Post-Perceptual Processes: Present Controversies and Future Directions. Pyschological Bull. 141(6) 1120-1144.
• Witt, Jessica (2011) Action's Effect on Perception. Current Directions in Psychological Science.