We don't seem to have bigram data when the word includes an apostophe

[tnantais]

I'm wondering if there's an issue in the XMLtoSQLite import code where words with apostrophes are rejected.

[OwenMcGirr]

There should not be but I'll explore.

On Tue, Sep 10, 2013 at 2:20 PM, tnantais notifications@github.com wrote:

I'm wondering if there's an issue in the XMLtoSQLite import code where
words with apostrophes are rejected.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/206
.

Thanks,
Owen McGirr

[tnantais]

I think it's possible that the apostrophe problem in XMLtoSQLite is causing multiple identical bigrams to appear in the database, e.g., I suspect "I'd,think" is going in as "I,think". This is a problem for the prediction algorithm, which expects each bigram to appear once.