How is temporal information about the world learned and how do the
temporal dynamics of the world influence learning?
(Initiative Coordinators: Dan Feldman and Paula Tallal)

Organisms explicitly learn many aspects of the temporal dynamics of the world, from temporal sequences of basic sensory stimuli (e.g., speech sounds), to dynamics of complex events that must be recognized (e.g., gestures, or dynamic features on computer displays). Other temporal features of our world, such as the relative timing of actions and rewards, can have a significant influence on learning, even though they are not explicitly learned. Our current understanding of how temporal information is learned is quite limited. By increasing our understanding of these processes, we hope to provide insights that will improve teaching and learning of dynamic stimuli. In addition, following a successful example with speech learning, we hope to develop teaching/training tools that manipulate the temporal dynamics of the world to facilitate or improve many forms of learning.

Strand 1.1: Learning of Temporal Patterns: How do organisms recognize, learn, and remember temporal patterns of sensory stimuli, including sequences? How are these represented in the brain, and how do they guide perception and behavior?

Strand 1.2: Cross-modal learning: how does timing of multi-modal sensory stimuli contribute to learning to integrate across modalities?

Strand 1.3: How does the relative timing of action, outcome, and reward influence the effectiveness of learning and duration of memory?

What are the temporal dynamics of brain cells, brain systems, and behavior? How do these dynamics change with learning, and how do they influence learning?
(Initiative Coordinators: Andrea Chiba and Isabel Gauthier)

Some important temporal dynamics are intrinsic to the way neurons and brain systems function (e.g., cellular properties of neurons, activity patterns in brain networks measured using EEG). These dynamics can influence how and what the brain learns. Other important temporal dynamics of brain systems supporting perceptual and cognitive systems have been shaped by development and experience. Understanding the temporal dynamics of perception and cognition is fundamental for understanding new perceptual and cognitive learning.

Strand 2.1: What aspects of neuronal ensemble dynamics, measured by EEG, are important for learning? How is ensemble dynamics generated by interactions between neurons? How do the natural spike trains that occur during the ensemble dynamics of learning drive plasticity via STDP rules?

Strand 2.2: What are the temporal dynamics of brain systems and the interaction between different brain systems and how do they constrain learning and memory?

Strand 2.3: How does neuromodulation set intrinsic dynamics for plasticity and learning?

What are the temporal structures for body movements and sampling the environment and how are they learned?
(Initiative Coordinators: Javier Movellan and Emo Todorov)

This initiative seeks to understand how humans learn real-time sensory-motor control laws. Our interest is both the learning of specific motor sequences and the learning of mappings that coordinate perception and action in real time. Are there common principles across domains and can we formulate general mathematical frameworks for sensory motor learning that will help explain long-standing questions in the field.

Strand 3.1: How do we actively sense the dynamically changing environment and learn to use sensory feedback to control and refine behavior?

Strand 3.2: What are some of the basic temporal features of speech and language learning?

Strand 3.3: How do we learn the temporal dynamics of movements, gestures, speech, facial expressions, and the like? Are they optimized for the environment? How well do they generalize to new dynamics environments and objects?

What mechanisms determine the time course of learning itself and what general principles explain the dynamics of learning across multiple scales and domains?
(Initiative Coordinators: Terry Sejnowski, Thomas Palmeri, and Tony Bell)

Learning takes place on multiple time scales, at multiple levels, in a variety of domains, relying upon a diversity of brain systems. Are there general principles about the temporal dynamics of learning across scales, levels, domains, and systems that can be unified into a coherent theoretical framework?

Strand 4.1: Are there general principles about the temporal dynamics of learning that can be unified into a coherent theoretical framework?

Strand 4.2: What factors determine rates of learning and duration of memories?

Strand 4.3: How do animals or people adapt to a changing environment?

Initiative Coordinators: Howard Poizner, Scott Makeig, Javier Movellan, and Emo Todorov.

Creating a new science requires new technologies for measuring and manipulating the dynamics that the brain controls. These include such phenomena as local field potentials, whole-brain activity, muscle activations, limb and body configurations, facial expressions, and student-teacher interactions. The quality and quantity of the resulting data require unique facilities for a large-scale system for storing, synchronizing, sharing, and analyzing that data. Such technological capabilities are beyond the reach of any individual lab and can only be realized in the center mode of funding. These capabilities will enable a number of cross-cutting research collaborations which would otherwise be technologically impossible. Our plan includes the further development of three such infrastructure facilities: Brain Dynamics, Motion Capture, and Data Sharing with others to be added depending upon demand. Given the inherent strength in measuring brain dynamics combined with motion capture, the Brain Dynamics Facility and the Motion Capture Facility are conjoined, offering our Center members the power of combined technology.

The Brain Dynamics Facility enables accurate measurement and analysis of whole-brain activity, by using a novel approach to combining the excellent temporal resolution of EEG along with the advanced data analysis and software tools developed by center participants. This facility is continually being advanced in collaboration with UCSD’s Swartz Center for Computational Neuroscience. These capabilities will be complemented by the Motion Capture Facility, which together with the Brain Dynamics facility will enable simultaneous recording on brain activity and complex motor behavior. This emerging integrated technology holds great promise in terms of understanding the spatio-temporal changes in brain dynamics that underlie the process of learning. The system will be housed in UCSD's Institute for Neural Computing. On the software side we will continue to develop and refine open-source appropriate to the new equipment including new analytic methods appropriate to the study of the temporal dynamics of learning.

The Motion Capture Facility is being developed in collaboration with the Institute for Neural Computation at UCSD. The facility will provide a range of devices for tracking behavior, including hand movements, eye movements, full body movements, facial expressions and inter-personal interactions, as well as to present stimuli that are tightly coupled with the observed behaviors (e.g. via Virtual Reality or mechanically via robotic devices). The goal is to provide researchers with the tools to manipulate time and timing and to investigate its role in learning and in the development of adaptive behavior. The facility will feature state-of-the-art equipment for marker-based motion capture, high-speed video recording, eye tracking, hand tracking, muscle recording. The facility will also be integrated with the Brain Dynamics Facility through the addition of a high-density EEG recording system for measuring brain dynamics. Complementing the hardware facilities, the Motion Capture Facility will provide a suite of software tools for data analysis and simulation, including a system for automated recognition of facial expressions, hand gestures and gaze directions; a system for probabilistic inference of joint angle trajectories, skeletal parameters and marker attachments from noisy data; and a modeling environment for simulation and visualization of musculo-skeletal dynamics.

The Data Sharing Facility is developed in collaboration with Data Intensive Cyber Environments group (DICE) of the School of Information and Library Sciences at the University of North Carolina Chapel Hill, the Advanced Computing Center for Research and Education (ACCRE) at Vanderbilt University, and other members of the TDLC. The TDLC includes laboratories from twelve universities across the United States, Canada, Australia, and the UK. A
significant challenge for collaborations among such geographically distributed scientists is sharing large quantities of data and stimuli quickly and easily. Traditional methods of “data sharing” such as email attachments can only handle small files of a few tens of megabytes. FTP servers are cumbersome for many users, transfer times can be exceedingly slow, and a glitch in an internet connection – which happens far too frequently– can often mean restarting a transfer from scratch. CDs and DVDs of data sent through the mail are no solution when there are hundreds of gigabytes of data to share. Some researchers have resorted to mailing hard disk drives with data, but that approach engenders significant limits on accessing and annotating data in real time and brings significant challenges for keeping data, analyses, and annotations synchronized across multiple laboratories. Sharing neurophysiological data, motion-capture data, fMRI and electrophysiology data, or high-quality images and video demands a system for easy,
efficient, fault-tolerant transfer of hundreds of gigabytes, terabytes, and one day perhaps petabytes, of data on a regular basis. Moreover, collaborators need to be assured that shared data are only seen by those who should see the data, and sharing data collected from humans and animals demands strict access control as dictated by human IRB and animal IACUC protocols and regulations. In addition to being fast, efficient, and fault-tolerant, a distributed data sharing system needs sophisticated access control policies. The Data Sharing Facility will include data sharing, data analysis, and quality control. The data systems will assure that all of the shared data have, among other components, proper IRB approval, traceable informed consent, and authorized privilege control. Sharing will be based on the concept of a datagrid - which provides data "virtualization" and makes it possible to organize distributed files into a logical collection that appears locally accessible. In addition to raw data, researchers can upload results from multiple iterations of data analysis in a variety of file formats. In this way complex data can be analyzed in collaborative fashion, while at the same time providing the access control and version control mechanisms needed to avoid data corruption and desynchronization. Such datagrids have becoming increasingly important in a growing list of scientific disciplines, and the DICE group through its Storage Resource Broker (SRB) and next generation iRODS technology– arguably the software platform of choice for datagrid development – can efficiently provide these functions with a moderate degree of tailoring being required. The second function of the Data Sharing facility will be to provide software tools for administrative management and reporting management, data mining and innovative analysis of large and diverse datasets incorporating sensory stimuli, brain responses, and behavioral responses. We will utilize (and when necessary, develop) unsupervised learning methods for automated discovery of meaningful features and dimensions of the raw data. Additional funding for the development of this collaborative data analysis environment will be sought in the form of a data-net cyberinfrastructure grant that will be submitted in partnership with the DICE group and several other large-scale research ventures.

Initiative Coordinators: Paula Tallal and Terry Sejnowski

In order to facilitate initiation of our research and education initiatives, while maintaining the bulk of our resources for science, the following implementation measures will be taken.

1) K-12 students and teachers: We describe our partnership with Reach for Tomorrow and the Preuss School in Initiative 7 (Diversity). We will reach K-12 educators (teachers and administrators) through lecturing in our corporate partner Jensen Learning Corporation’s “Brain Expo” for teachers. We will volunteer our time so that scholarships for teachers (especially at our partner schools) may attend for free. . We will also reach tens of thousands of K-12 educators annually through our corporate partner Scientific Learning Corporation’s website for educators www.brainconnection.com and electronic newsletter. We will contribute a quarterly column called “Advances from the NSF Science of Learning Centers” that will summarize in lay terms new scientific publications of most relevance to K-12 educators. We will also encourage responses and discussion from K-12 educators to this column as part of our “inreach” program. These outreach and inreach programs will not cost us anything but our time and will provide us with an ongoing mechanism for open and timely dialogue with K-12 educators nationally.

Through the UC COSMOS Summer Program, a workshop will be given to high school students on the temporal dynamics of learning. We intend to do one round of this each year, as it does not cost us anything but our time. Additionally, UCSD’s existing Neuroscience Outreach Team will develop and incorporate a unit on temporal dynamics of the brain for in their outreach to K-12 students and teachers.

The Center Program Representative (Andrew Kovacevic) will facilitate these partnerships and arrange plans to ensure that the programs run smoothly. Center investigators will be responsible for entering all reporting data into the Center administrative database, in order to record and document these activities.

2) Undergraduates: First, throughout the award period, we will continue our strong commitment to directly involving undergraduates at each of our participating Universities in our research labs. Each laboratory will report their undergraduate research trainees and the projects on which they worked. Center investigators will be responsible for ensuring that this information is entered into the Center administrative database. Through the UCSD Educational Advancement Office, Michael Dabney will coordinate matching efforts between Center investigators and incoming students participating in academic enrichment programs. The Summer Undergraduate Science of Learning Institute will be put on hold until year 3.

3) Graduate students and postdocs: Starting in year 3 of the project, we will establish a new “Ph.D. plus” Graduate Program in the Learning Sciences, aimed at students in Cognitive Science, Computer Science, Engineering, Psychology, and Neuroscience. A major objective of this program is to insure that students will work with center faculty to gain a balanced coverage of at least two emphasis areas in our 2 by 2 table: Experimental and computational approaches to understanding and modeling the dynamics of learning in behavior or in the brain. This program will be initiated in year 3, pending the availability of funds.

Starting in year 3 of the project, bridge postdocs will spend a year in one lab of a research network and another year in a lab at another location. This cements relations within the network and transfers techniques and knowledge from one lab to another. The Center Program Representative (Andrew Kovacevic) will be responsible for assisting with travel and housing arrangements to ensure that this is possible.

4) Researchers: In years one and two, we will invite researchers from outside UCSD and our partner institutions to come to the Center to give talks. Planning will be facilitated by Program Representative, Andrew Kovacevic.

5) The public: Our premier mechanism for outreach to the public will be through The Science Network (TSN). In year 1, we will provide seed money for editing equipment, record our all-hands meeting and the lecture content at the Brain Expo for web export to teachers. These efforts will be ramped up as appropriate to the budget increments in later years. Roger Bingham of TSN will coordinate these efforts.

6) Translational activities: We will begin to show translation of our science to education through further development of the “Let’s Face It!” and RUBI projects.

Initiative Coordinator: Gary Cottrell

In terms of including more minority faculty in our Center, we have identified and will continue to identify minority junior faculty from other institutions who we will ask to participate in appropriate network meetings. We have also developed a simple procedure for incorporating these faculty members into our research community:
1) Invite faculty to participate in All Hands Meeting or a Network meeting and give brief talk about their research.
2) Have an initiative or project leader propose a collaboration (funded) to the initiative/project.
3) Initiative/project leader proposes the research collaboration to the Executive Committee.
4) The Executive Committee votes on whether to fund the collaboration.
5) These new faculty collaborators are allowed to submit a proposal budget of their own for the following year.
6) In order to facilitate this process, the Center will reserve a small amount of University matching funds ($30-50k) each year to fund projects immediately if they are of importance (and not have to wait for NSF approval for budget changes).

In terms of graduate students and postdocs, we will use the considerable diversity already existing at Rutgers Newark to attempt to recruit diverse graduate students and postdocs to our Center labs. We do not have enough funds in any one project to pay for a postdoc or graduate student, but we have two minority graduate fellowships promised to us from our Vice Chancellor of Research at UCSD that will be useful for this purpose.

Our plans for recruiting minority graduate students include the Faculty Partners Program, a mechanism intended to help us form relationships with faculty at minority serving institutions. In California, there are a large number of minority serving institutions in the form of the state university system, and in neighboring states, there are a large number of Hispanic serving institutions. We will first concentrate our efforts on these by traveling there and giving talks in order to meet the faculty, for the purpose of inviting them for a two-day visit to our Center. It is through personal relationships that faculty at these institutions will begin to encourage their students to apply to us for graduate school.

Our plans for recruiting minority undergraduates center on two aspects of our program: 1) our partnership with the Preuss school, a 72% underrepresented minority serving charter school on campus, and 2) the Reach for Tomorrow program, which brings inner city and other minority high school students to UCSD (and other campuses) for an intensive one to two week program. Our plans are to work with the new Preuss principal, Scott Barton, a member of our Executive Committee, to expand the number of Preuss students who are either in our labs or mentored by members of our center.

Initiative Coordinators: Isabel Gauthier and Thomas Palmeri