Salesforce develops natural language processing model that performs 10 tasks at once

Undeterred, scientists at Salesforce Research, led by chief scientist Richard Socher, took a two-pronged stab at the problem. They developed both a 10-task natural language processing challenge — the Natural Language Decathlon (decaNLP) — and a model that can solve it — the Multitask Question Answering Network (MQAN) — in PyTorch, an open source machine learning library for the Python programming language. Beyond Wikipedia, researchers experienced issues assembling ebooks in each language, which are often used to train NLP models. For Arabic and Urdu, many titles were available as scanned images rather than text format, requiring processing by optical character recognition tools that ranged in accuracy from 70% to 98%. With Chinese ebooks, the optical character tool the researchers used incorrectly added spaces to each new line.

Clinical efficacy of pre-trained large language models through the lens of aphasia

Explore the future of AI on August 5 in San Francisco—join Block, GSK, and SAP at Autonomous Workforces to discover how enterprises are scaling multi-agent systems with real-world results. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.

The problem of underrepresented languages snowballs from data sets to NLP models

The vast majority of NLP tools are developed in English, and even when they gain support for other languages, they often lag behind English with respect to robustness, accuracy, and efficiency, the coauthors assert. In the case of BERT, a state-of-the-art pretraining technique for natural language processing, developers released an English model and subsequently Chinese and multilingual models. But the single-language models retain performance advantages over the multilingual models, with both English and Chinese monolingual models performing 3% better than the combined English-Chinese model. Moreover, when smaller BERT models for teams with restricted computational resources were released, all 24 were in English. The researchers found that the MQAN, when jointly trained on all 10 tests without any task-specific modules or parameters, performed at least as well as 10 MQANs trained on each test separately. Lack of representation at each stage of the pipeline adds to a lack of representation in later stages, the researchers say.

• For Arabic and Urdu, many titles were available as scanned images rather than text format, requiring processing by optical character recognition tools that ranged in accuracy from 70% to 98%.
• They developed both a 10-task natural language processing challenge — the Natural Language Decathlon (decaNLP) — and a model that can solve it — the Multitask Question Answering Network (MQAN) — in PyTorch, an open source machine learning library for the Python programming language.
• We welcome theoretical-applied and applied research, proposing novel computational and/or hardware solutions.
• There’s a document summarization test, a natural language inference test, a sentiment analysis test, a semantic role labeling test, a relation extraction test, a goal-oriented dialog test, a query generation test, and a pronoun resolution test.

The databases are even less representative than they might appear because not all speakers of a language have access to Wikipedia. In the case of Chinese, it’s banned by the Chinese government, so Chinese articles in Wikipedia are more likely to have been contributed by the 40 million Chinese speakers in Taiwan, Hong Kong, Singapore, and overseas. To judge the model’s performance, the researchers normalized the results of each test and added them together to arrive at a number between 0 and 1000 — the decaScore. This Collection is dedicated to the latest research on methodology in the vast field of NLP, which addresses and carries the potential to solve at least one of the many struggles the state-of-the-art NLP approaches face. We welcome theoretical-applied and applied research, proposing novel computational and/or hardware solutions.

• For instance, a script they used to download the Chinese, English, Spanish, Arabic, French, and Farsi corpora from Wikipedia experienced a 0.13% error rate for Farsi and a 0.02% error rate for Chinese but no errors across 5 million English articles.
• Undeterred, scientists at Salesforce Research, led by chief scientist Richard Socher, took a two-pronged stab at the problem.
• With Chinese ebooks, the optical character tool the researchers used incorrectly added spaces to each new line.
• And for the Urdu and Wolof corpora, the script wasn’t compatible because it lacked support for their formats.
• They vary not only by the file size of the corpora and the total number of pages, but along dimensions including the percentage of stubs without content, number of edits, number of admins working in that language, total number of users, and total number of active users.

The AI insights you need to lead

And because the Wolof language doesn’t have a written character set, the team was forced to rely on English, French, and Arabic transcriptions that might have taken stylistic liberties. Socher said the model’s ability to perform well in tasks it hasn’t been trained to do could pave the way for more robust, natural chatbots that are better able to infer meaning from human users’ questions. For instance, a script they used to download the Chinese, English, Spanish, Arabic, French, and Farsi corpora from Wikipedia experienced a 0.13% error rate for Farsi and a 0.02% error rate for Chinese but no errors across 5 million English articles. And for the Urdu and Wolof corpora, the script wasn’t compatible because it lacked support for their formats. Join leaders from Block, GSK, and SAP for an exclusive look at how autonomous agents are reshaping enterprise workflows – from real-time decision-making to end-to-end automation.

As something of a case in point, the multilingual BERT model was trained on the top 100 languages with the largest Wikipedia article databases, but there are substantial differences in the size and quality of the databases when adjusting for the number of speakers. They vary not only by the file size of the corpora and the total number of pages, but along dimensions including the percentage of stubs without content, number of edits, number of admins working in that language, total number of users, and total number of active users. A typical NLP pipeline involves gathering corpora, processing them into text, identifying language elements, training models, and using these models to answer specific questions. The degree to which some languages are underrepresented in data sets is well-recognized, but the ways in which the effect is magnified throughout the NLP toolchain is less discussed, the researchers say. There’s a document summarization test, a natural language inference test, a sentiment analysis test, a semantic role labeling test, a relation extraction test, a goal-oriented dialog test, a query generation test, and a pronoun resolution test.