Following this thread initiated by @orph and @higgs - opening a new “tutorial” thread for development on English conversational model based on A Neural Conversational Model,Vinyals, O., & Le, Q. (2015) paper.
Using OpenSubtitle corpus (Jörg Tiedemann, 2009), English XML dump (19Gb, 323905 movies, 338M sentences) - consecutive sentences with following properties were extracted:
- first sentence ends with question mark
- second sentence has no question mark
- second sentence follows in the movie the first sentence by less than 20 seconds
The extracted pairs were additionally cleaned-up (detokenized, removal sound sequence like
(BANG), removal of too complex sentences, normalization of quotes), uniq-ed and shuffled.
This gives a 14M sentence corpus, and 10000 sentences for validation available here. Difference with OpenSubtitle corpus mentioned in the paper is that pairs are questions/answers while in their corpus, it contains any sequence of sentences. Note that there is no certitude that answers are actually answering the questions (answer can come from the same the same speaker, or change topic and/or scene in the movie).
What are you doing here, Cao? To pay Mr. and Mrs. Mak a visit. But can we cut to what's important here? There's a Fey Sommers estate out there with your name on it. I assume it was granted? He's been given full access - gold level and beyond. How did you know of Devavrita's return? The entire Universe knows of it my king What do you think it means? l appreciate the show and tell, and I don't want you to take this personally, but I work alone. Who'd have thought it, Venancio? Us, keeping the Count of Albrit from going hungry... when not even twenty years ago... he was the lord and master of Polan, Jerusa and everything. Stay close, understand? Keeper What's toll? I don't know, they said it. Let me guess, Kyle? No, it's Mark actually. Did you notice what car they were driving? It's a nightclub, Hal, not a drive-in. Didn't you? Yes, we did. What? Small and bent of course, deformed, and he hobbles. Got an easier one? Just needed to ask you. He's a nice boy, isn't he? Yes, he's nice. How can you possibly hope to stop me? Well, like the old man said... Jealous? I've got more things on my mind. What a beautiful young voice, how old are you? Twenty-two.