The current reported success of large language models is based on computationally (and environmentally) expensive algorithms and prohibitively large amounts of data that are available for only a few, non-representative languages. This limitation reduces the access to natural language processing technology to a few dominant languages and modalities and leads to the development of systems that are not human-like, with great potential for unfairness and bias.
To reach better, possibly human-like, abilities in neural networks' abstraction and generalisation, we need to develop tasks and data that train the networks to more complex and compositional linguistic abilities. We identify these abilities as the intelligent ability to infer patterns of regularities in unstructured data, generalise from few examples, using abstractions that are valid across possibly very different languages.
We have developed a new set of tasks inspired by IQ intelligence tests. These tasks are developed specifically for language and learn disentangled linguistic representations of underlying linguistic rules of grammar.
These investigations can lead to three beneficial improvements of methods and practices: (i) deep, compositional representations would be learnt, thus reducing needs in data size; (ii) current machine learning methods would be extended to low-resources languages or low-resource modalities and scenarios; (iii) higher-level abstractions would be learnt, avoiding the use of superficial, associative cues that are the cause of bias and potential harm in the representations learned by current artificial linguistic systems.
Date: 21.06.2023 12:30 – 13:30 [online, free, registration needed]
See : https://cui.unige.ch/fr/pin/digital-innovators/di20230621/