Automatic speech recognition (ASR) has made fantastic progress in the last decades, and current systems achieve word error rates below 5% on many human-to-machine tasks. This has finally made ASR a viable commercial field. But recognition of natural, fluent, and human-to-human speech is a much harder problem; error rates are at least 20% and often much higher. If Conversational Agents are to become a key way for humans to interact with computers, as I think they will, our ASR systems must be much better at dealing with natural conversational speech. In this talk I summarize our basic research on statistical models of three kinds of knowledge that have limited our ability to deal with natural conversational speech. These include models of word prediction based on LSA, an information retrieval technique, ways to achieve robustness to pronunciation variation (ASR systems are notoriously troubled by people speaking quickly and casually), and ways to figure out whether a human is asking us a question, giving a command, or merely making an irreverent aside about a stubbed toe. I will try to include an brief overview of speech recognition for audience members unfamiliar with the basic algorithms.