Philip John Gorinski<p>Thrilled to announce the Regular Expression Inference Challenge (REIC), with Mojtaba Valizadeh, Ignacio Iacobacci, Martin Berger.</p><p>REI is a supervised machine learning (<a href="https://qoto.org/tags/ML" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ML</span></a>) and program synthesis task, and poses the problem of finding minimal regular expressions from examples: Given two finite sets of strings P and N and a cost function cost(⋅), the task is to generate an expression r that accepts all strings in P and rejects all strings in N, while no other such expression r' exists with cost(r')<cost(r).</p><p>Turns out, this sort of inference seems to be really hard for current DL (<a href="https://qoto.org/tags/llms" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>llms</span></a> ) approaches. Prompting StarChat-beta -- a SOTA large LM for code with 15.5B parameters -- yields extremely low results.<br>Even a fully supervised 300M parameter model, which we call ReGPT, only achieves around 14% precise and minimal expressions.</p><p>Check out our preprint on arXiv: <a href="https://arxiv.org/abs/2308.07899" rel="nofollow noopener" target="_blank"><span class="invisible">https://</span><span class="">arxiv.org/abs/2308.07899</span><span class="invisible"></span></a><br>The challenge is available on CodaLab: <a href="https://codalab.lisn.upsaclay.fr/competitions/15096" rel="nofollow noopener" target="_blank"><span class="invisible">https://</span><span class="ellipsis">codalab.lisn.upsaclay.fr/compe</span><span class="invisible">titions/15096</span></a></p><p>We formally define the problem, and provide training and validation data, as well as starter code for all our baselines.</p><p>We invite researchers anywhere to participate in tackling our challenge.</p><p><a href="https://qoto.org/tags/machinelearning" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>machinelearning</span></a> <a href="https://qoto.org/tags/inference" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>inference</span></a> <a href="https://qoto.org/tags/challenge" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>challenge</span></a> <a href="https://qoto.org/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://qoto.org/tags/ML" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ML</span></a> <a href="https://qoto.org/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>llm</span></a> <a href="https://qoto.org/tags/llms" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>llms</span></a> <a href="https://qoto.org/tags/huawei" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>huawei</span></a></p>