Variational Bayesian Methods for Unsupervised Latent Factor Models of Text and Audio

In this talk, I will discuss variational strategies for fitting two Bayesian models that explain high-dimensional media data in terms of sets of latent factors.

The first model, Latent Dirichlet Allocation (LDA), is a popular model of text corpora that learns to represent documents as mixtures of latent “topic” distributions. We develop an online variational Bayes (VB) algorithm for LDA. Online LDA is based on online stochastic optimization with a natural gradient step, which we show converges to a local optimum of the VB objective function. It can handily analyze massive document collections, including those arriving in a stream. We study the performance of online LDA in several ways, including by fitting a 100-topic topic model to 3.3M articles from Wikipedia in a single pass. We demonstrate that online LDA finds topic models as good as or better than those found with batch VB, and in a fraction of the time.

The second model, Gamma Process Non negative Matrix Factorization (GaP-NMF), is a new Bayesian nonparametric model of audio spectrograms that addresses the problem of latent source discovery and separation in audio recordings. GaP-NMF allows us to discover what sounds (e.g. bass drums, guitar chords, etc.) are present in a recording and to isolate or suppress individual sources. Crucially, this model is able to decide how many latent sources are necessary to model the data. This feature is particularly valuable in this application, since it is impossible to guess a priori how many sounds will appear in a given recording. Although the GaP-NMF model lacks the conditional conjugacy enjoyed by models such as LDA, we are nonetheless able to efficiently fit it to data using a novel variational algorithm.

Bio:

Matthew D. Hoffman is a postdoctoral researcher in the Department of Statistics at Columbia University. He received his Ph.D. in Computer Science from Princeton University in 2010. His research interests are in machine learning, statistical modeling, audio signal processing, content-based music information retrieval, and the associated computational issues.

“Generating Descriptions of Visible Objects”

Abstract:  What do people describe when they look at objects?  Can we model what they say?  (Why does this matter?)  This talk will characterize what makes up a visual description and define some of the methods necessary to automatically generate such language.  Taking this a bit further, I describe an end-to-end prototype system that reads in computer vision output and generates natural language descriptions.  Time permitting, I argue that improving visual descriptions can also improve computer vision, and working on the interaction between the two may lead to advances in both computer vision and natural language generation.  My prototype vision-to-language system, largely developed during the Hopkins summer workshop 2011 in collaboration with vision researchers at Stony Brook and language researchers at U. Maryland, is available at:  http://recognition.cs.stonybrook.edu:8080/~mitchema/midge/

BIO:

Meg Mitchell graduated with a Bachelor’s in Linguistics from Reed College in 2005.  Since then, she has worked at the Center for Spoken Language Understanding at Oregon Health and Science University, helping to automatically diagnose neurological disorders by analyzing the syntactic and phonetic characteristics of spoken language.  She has also received a Master’s in Computational Linguistics from the University of Washington and is working towards a Ph.D in Computing Science at the University of Aberdeen.

“Statistical Modeling and Learning for Machine Translation”

Recent years have seen a flurry of research in translation modeling for statistical machine translation. Widely-used approaches include (1) models based on flat phrase-to-phrase mappings, and (2) models that use syntactic structure. I will present an approach that combines the strengths of these two in a single model. Experiments show that it leads to improved translation quality over state-of-the-art systems. When supervised syntactic parsers are not available, I will show that unsupervised parsers can be substituted with minimal loss in translation quality. I will also discuss the problem of tuning model parameters. An abundance of techniques has been developed by the statistical machine learning community, but the machine translation problem is fundamentally different from other structure prediction tasks in key ways. These differences have caused well-known learning algorithms to lose theoretical guarantees when applied to machine translation. I will propose a novel algorithm that does offer theoretical guarantees and shows empirical advantage over state-of-the-art approaches.

BIO:

Kevin Gimpel is a PhD student in the Language Technologies Institute at Carnegie Mellon University where he is advised by Noah Smith. His research focuses on machine translation, with supporting interests in natural language processing and machine learning. He is also interested in new tasks involving emerging data sources, including social media, movie reviews, and restaurant menus. He interned with the machine translation team at Google during summer 2009 and has been a Sandia National Laboratories Excellence in Science and Technology Fellow since 2010.

“Learning to Efficiently Rank”

Technological advances have led to increases in the types and amounts of data, and there is a great need for developing methods to manage and find relevant information from such data to satisfy user’s information needs. Learning to rank is an emerging discipline at the intersection of machine learning, data mining, and information retrieval. It develops principled machine learning algorithms to construct ranking (i.e., retrieval) models, for finding and ranking relevant information to user queries over large amounts of data. Although learning to rank approaches are capable of learning highly effective ranking functions, they have mostly ignored the important issue of model efficiency (i.e., model speed). Given that efficiency and effectiveness are competing forces that often counteract each other, models that are optimized for effectiveness alone may not meet the strict efficiency requirements when dealing with real-world large-scale data-sets.

My PhD thesis introduces the Learning to Efficiently Rank framework for learning large-scale ranking models that facilitate fast and effective retrieval, by exploiting and optimizing the tradeoffs between model complexity (i.e., speed) and accuracy.  At a basic level, this framework learns ranking models whose speed and accuracy can be explicitly controlled.  I proposed and designed solutions for three problems within this framework: 1) learning large-scale ranking models according to a desired tradeoff between model speed and accuracy; 2) constructing temporally-constrained models capable of returning results under time budgets; 3) breaking through the speed/accuracy tradeoff barrier by developing a novel cascade ranking model, and learning the cascade model structure and parameters with a novel boosting-based learning algorithm. My research extends the conventional effectiveness-centric approach in model learning and takes an efficiency-minded look at building effective retrieval models.  Results show that models learned this way significantly outperform traditional machine-learned models in terms of speed without sacrificing result effectiveness. Moreover, the new models work particularly well when users impose stringent time requirements for ranked retrieval on very large data-sets.

Bio:

Lidan Wang is a PhD candidate in the Computer Science Department at the University of Maryland, College Park. She received her Master’s degree from the Department of Computer Science at the University of Wisconsin, Madison, and Bachelor’s degree from the Department of Computer Science at the University of Florida. Lidan’s research interests lie at the intersection of machine learning, information retrieval, and text and data mining. Lidan’s work focuses on designing large-scale machine learning and information retrieval techniques for learning, mining, and retrieving information at scale. Her PhD dissertation research led to a recent NSF research grant (IIS-1144034), which she co-authored.

Interactive Machine Learning: Combining Learning Strategies with Humans in the Loop

People learn by interacting with their teachers.  Why not machines?  What would it take to develop software that can learn how to solve problems by interacting and collaborating with humans?  This talk will describe my efforts to develop such systems, with the goal of training effective machine learners more quickly and economically.  In particular, I focus on two projects in natural language processing that combine multiple learning strategies: incorporating domain knowledge (taking advice in the form of human-provided rules), active learning (asking “questions” of human annotators), and semi-supervised learning (attempting to “teach itself” by extrapolating what has been learned onto abundant, unlabeled data).  Empirical results from user experiments show that these approaches are superior to their state-of-the-art “passive” learning counterparts.  Interestingly, these experiments provide initial insights into human “teaching” behavior as well, suggesting ways in which human factors can and should be taken into account.  I will also briefly discuss opportunities for interactive learning in other areas, such as supporting online communities, creative work, and biological discovery.

Bio:

Burr Settles is a Postdoctoral Fellow in the Machine Learning Department at Carnegie Mellon University. He received a PhD in Computer Sciences from the University of Wisconsin-Madison in 2008, with additional studies in Linguistics and Biology. His current research focuses on interactive machine learning that resembles a “dialogue” of decision-making and knowledge acquisition between computers and humans, with applications in natural language processing, biology, and social computing. He recently organized workshops at the ICML and NAACL conferences on these topics, and is the author of a popular literature survey on active learning (active-learning.net). He also runs the website FAWM.ORG, prefers sandals to shoes, and plays guitar in the Pittsburgh pop band Delicious Pastries.

Generate-and-Test Models for Alignment and Machine Translation

I discuss translation as an optimization problem subject to three kinds of constraints: lexical, configurational, and constraints enforcing target-language wellformedness. Lexical constraints ensure that the lexical choices in the output are meaning-preserving; configurational constraints ensure that the relationships between source words and phrases (e.g., semantic roles and modifier-headrelationships) are properly transformed in translation; and target-language wellformedness constraints ensure the grammaticality of the output. In terms of the traditional source-channel model of Brown et al. (1993), the “translation model” encodes lexical and configurational constraints and the “language model” encodes target language wellformedness constraints. On the other hand, the constraint-based framework suggests a generate-and-test (discriminative) model of translation in which features sensitive to input and output structures, and the feature weights are trained to maximize the (conditional) likelihood of a corpus of example translations. The specified features represent empirical hypotheses about what variables correlate (but not why) and thus encode domain-specific knowledge that is useful for the problem at hand; the learned weights indicate to what extent these hypotheses are confirmed or refuted.

To verify the usefulness of the feature-based approach, I discuss the performance two models: first, a lexical translation model evaluated by the word alignments it learns. Unlike previous unsupervised alignment models, the new model utilizes features that capture diverse lexical and alignment relationships, including morphological relatedness, orthographic similarity, and conventional co-occurrence statistics. Results from typologically diverse language pairs demonstrate that the generate-and-test model provides substantial performance benefits compared to state-of-the-art generative baselines.  Second, I discuss the results of an end-to-end translation model in which lexical, configurational, and wellformedness constraints are modeled independently. Because of the independence assumptions, the model is substantially more compact than state-of-the-art translation models, but still performs significantly better on languages where source-target word order differences are substantial.

Bio:

Chris Dyer is a postdoctoral researcher in Noah Smith’s lab in the Language Technologies Institute at Carnegie Mellon University. He completed his PhD on statistical machine translation with Philip Resnik at the University of Maryland in 2010. Together with Jimmy Lin, he is author of “Data-Intensive Text Processing with MapReduce”, published by Morgan & Claypool in 2010. Current research interests include machine translation, unsupervised learning, Bayesian techniques, and “big data” problems in NLP.

Applications of weighted finite state transducers in a speech recognition toolkit

The open-source speech recognition toolkit “Kaldi” uses weighted finite state  transducer (WFSTs) for training and decoding, and uses the OpenFst toolkit as a  C++ library.  I will give an informal overview of WFSTs and of the standard AT&T  recipe for WFST based decoding, and will mention some problems (in my opinion)  with the basic recipe and how we addressed them while developing Kaldi.  I will  also describe how to use WFSTs to acheive “exact” lattice generation, in a sense  will be explained.  This is an interesting application of WFSTs because, unlike  most WFST mechanisms, it does not have any obvious non-WFST analog.

Bio:   Daniel Povey received his Bachelor’s (Natural Sciences, 1997), Master’s  (Computer Speech and Language Processing, 1998) and PhD (Engineering, 2003) from  Cambridge University.  He is currently a researcher at Microsoft Research,  Redmond, Washington, USA.  From 2003 to 2008 he worked as a researcher in IBM  Research in Yorktown Heights, NY.  He is best known for his work on  discriminative training for HMM-GMM based speech recognition (i.e. MMI, MPE, and  their feature-space variants).