Re:

I agree w/ Joscha's caution about discrimination tasks: They =an be often be solved rather well, but in devious ways, =y statistical supervised learning =lgorithms.

The =ttached picture will get our current best statistical methods to tell =s that it sees four to six people.

A really good =ystem might recognize that two move forward and two in the opposite =irection, and that the latter ones have full bags.

l=don't think that there is a system that would tell us that there is =robably a store in the direction the bagless people are =oing.

Most of all, current systems probably won't =igure out that there must be a sniper behind that wall.

People don't stop at matching patterns; they construct a =onceptual world view, and integrate what they see, hear and read into =t. IMHO, this is what an Al challenge needs to be about: use any kind =f information, and integrate it into a deeper, growing and dynamic =nderstanding of the world.

One way of doing that might be to let =t re-tell the story of a movie we show it. We will have to make sure =hat the particular movie is unknown to the system (it is likely going =o be trained on many annotated movies), which could be achieved by ticking one (or rather, several, with different degrees of difficulty) =hat has not aired yet when the submission is made. We might also put =imits on the memory footprint of the system, to make sure that it does =ot memorize existing stuff too literally, but is forced to make =nferences. The whole thing could be a competition, where the =erformance is compared to children of different ages, using a mixed =ury including developmental psychologists, computer scientists and =creenwriters.

Prices could be given to contributions that match the =erformance of a 3yr old, 6yr old, and adult, for a challenge of five =ovies unknown to the submitters.

Suppose you pose a =inguistic discrimination task of some sort -- and a =upervised learning algorithm, trained on a mass of data, =an solve it with 97% accuracy.

I don't think that Bob =erwick knew Deep Speech yet, a system that Andrew Ng built for Baidu =arlier this year. It does not do any Fourier transform on auditory =ata, but only raw convolutional networks. It works on phone audio, =oisy streets, in conference rooms etc., and it matches what it hears to =honemes, which are then mapped to language on a separate layer. =ccording to the publication, it outperforms humans at =his.

Bob =old as that he knows how language itself works, i.e. not the mapping =rom sound to phonemes, but the much more interesting part behind that. =hat is missing, from an Al perspective, is the link between language =nd the simulated inner world in our minds. I think that Al is still =uite a way from recreating our imagination (and Noam is skeptical that =t will ever happen, if I understand him correctly). For a Chomsky =hallenge, perhaps we would need to find a task that requires deep =anguage models without necessarily requiring deep understanding. I am =ure that the linguists have much better ideas than me, of =ourse.

– Joscha