Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
101010.pl is one of the many independent Mastodon servers you can use to participate in the fediverse.
101010.pl czyli najstarszy polski serwer Mastodon. Posiadamy wpisy do 2048 znaków.

Administered by:

Server stats:

504
active users

101010.pl: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.0-alpha.5

Alek

The white paper for the Pile, a dataset used for many projects (including key AI efforts) is a fascinating read.

It's managed by a mysterious group called the Eye, it includes things like pirated books and subtitles (for pirated movies) and it has a very interesting take on open science, strongly skewed towards research freedom and pragmatism of accessing and using data.

What I want to highlight here is that the dataset also includes a corpus of @europarl_en from over 20 years.

Anyone who follows the EP legislative process knows it's extremely , with little sense that there are collective knowledge management tools that could significantly facilitate the process, make it more transparent and engaging, etc.

So it's interesting to see it end up as a foundational bit for large language models.

I will now be imagining the outputs being just a bit tinged with European political talk.

arxiv.org/abs/2101.00027

arXiv.orgThe Pile: An 800GB Dataset of Diverse Text for Language ModelingRecent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction.
··Web
4boosts·5favorites
ExploreLive feeds
About