Recent searches

No recent searches

Search options

Only available when logged in.
101010.pl is one of the many independent Mastodon servers you can use to participate in the fediverse.
101010.pl czyli najstarszy polski serwer Mastodon. Posiadamy wpisy do 2048 znaków.

Administered by:

Server stats:

485
active users

101010.pl: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.5.0-alpha.1

#glowdust

0 posts0 participants0 posts today
Chris Gioran 💔<p>New blog post: </p><p>A simple, async page cache built on top of io_uring</p><p><a href="https://radiki.dev/posts/glowdust-page-cache-1/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">radiki.dev/posts/glowdust-page</span><span class="invisible">-cache-1/</span></a></p><p><a href="https://fosstodon.org/tags/glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>glowdust</span></a> <a href="https://fosstodon.org/tags/io_uring" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>io_uring</span></a> <a href="https://fosstodon.org/tags/rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rust</span></a> <a href="https://fosstodon.org/tags/rustlang" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rustlang</span></a></p>
Chris Gioran 💔<p><a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> now comes with a single threaded, request based page cache. This lets me serve pages to multiple transactions, asynchronously, without spending time optimizing a thread safe version.</p><p>Which means I can start actually implementing the stuff I wrote about last week:</p><p><a href="https://radiki.dev/posts/glowdust-tx-implementation-1/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">radiki.dev/posts/glowdust-tx-i</span><span class="invisible">mplementation-1/</span></a></p><p>TL;DR Transaction state is kept in the page cache and it's basically just another file. This causes a lot of interesting behavior to experiment with.</p>
Chris Gioran 💔<p>And just like that, <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> now has a page cache that can actually fault pages in memory.</p><p>From some other place in memory.</p><p>And it's all it can do. Just page faults.</p><p>But hey, progress.</p><p><a href="https://fosstodon.org/tags/database" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>database</span></a></p>
Chris Gioran 💔<p>2500 words, a princess bride meme and my surprise at finishing it.</p><p>That's about 86% of what you need to know about my new blog post. </p><p>The remaining 14% is describing a new architecture for database transaction subsystems. In <a href="https://fosstodon.org/tags/Rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Rust</span></a> , obviously.</p><p>Do let me know what you think - especially about the backpressure design. </p><p><a href="https://radiki.dev/posts/glowdust-tx-implementation-1/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">radiki.dev/posts/glowdust-tx-i</span><span class="invisible">mplementation-1/</span></a></p><p><a href="https://fosstodon.org/tags/rustlang" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rustlang</span></a> <a href="https://fosstodon.org/tags/database" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>database</span></a> <a href="https://fosstodon.org/tags/glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>glowdust</span></a></p>
Chris Gioran 💔<p>Because transaction state is </p><p>1) stored in the page cache and so </p><p>2) can be moved freely between threads,</p><p>transactions can be served from a priority queue from a fixed number of workers.</p><p>Thus, <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> has, by design, the ability to automatically balance CPU and memory use.</p><p>Memory pressure pushes back via I/O to persist overflowing tx state and serve page faults.</p><p>CPU pressure reduces time spent on each tx per worker, which makes tx state accumulate in memory, which releases CPU.</p>
Chris Gioran 💔<p>Update: Initial transaction support has landed in <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> </p><p><a href="https://codeberg.org/glowdust/glowdust/commit/37a0b94f9846611c8b5f0433711537b678c07594" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">codeberg.org/glowdust/glowdust</span><span class="invisible">/commit/37a0b94f9846611c8b5f0433711537b678c07594</span></a></p><p>Pretty large update, and I still have some work to clean up and commit locally. But it's coming together.</p><p>The independence of this code from the language syntax is deliberate. It's all in the direction of having the frontend be pluggable so you can roll your own DBMS, language and all.</p>
Chris Gioran 💔<p>Pro tip: If you want to step through a thread, you can pass it a sync_channel(1) and use write/read calls to block and unblock it.</p><p>You can do this with multiple threads and create a state machine for controlling synchronization and reproducing or checking for race conditions.</p><p>I am currently doing this with <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> and it works pretty well for writing tests for transaction isolation.</p><p><a href="https://fosstodon.org/tags/rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rust</span></a> <a href="https://fosstodon.org/tags/rustlang" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rustlang</span></a></p>
Chris Gioran 💔<p>Maybe I've been thinking about <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> the wrong way.</p><p>Maybe I shouldn't talk about it as a DBMS. Perhaps it makes more sense to talk about it as a database development kit, a DBDK, if you will.</p><p>I'll think about it on my way to vote <a href="https://fosstodon.org/tags/EU" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>EU</span></a> <a href="https://fosstodon.org/tags/Election" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Election</span></a> <a href="https://fosstodon.org/tags/Europe" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Europe</span></a></p>
Chris Gioran 💔<p>So <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> commits stuff from different threads now, NBD.</p><p>About a month ago this seemed impossible. I still can't really believe that I got this far.</p>
Chris Gioran 💔<p><a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> update: Multithreaded transaction execution works, with tx state storage in the page cache. </p><p>What remains is commit(), i.e. make per tx state visible to other transactions.</p><p>I mean, there are a million steps after that, but for now, only commit() remains.</p>
Chris Gioran 💔<p>TheRegister on the 50th anniversary of <a href="https://fosstodon.org/tags/SQL" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SQL</span></a> </p><p><a href="https://www.theregister.com/2024/05/31/fifty_years_of_sql/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">theregister.com/2024/05/31/fif</span><span class="invisible">ty_years_of_sql/</span></a></p><p>50 years is a lot, but for SQL I'll make an exception. It's a fantastically well thought out system.</p><p>But as one of its creator says, it's not enough</p><p><a href="https://www.theregister.com/2024/05/10/sql_cocreator_nosql/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">theregister.com/2024/05/10/sql</span><span class="invisible">_cocreator_nosql/</span></a></p><p>We need to progress beyond, and take advantage of modern computing platforms.</p><p>Have you looked at <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a>, per chance? 😉 </p><p><a href="https://codeberg.org/glowdust/glowdust" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">codeberg.org/glowdust/glowdust</span><span class="invisible"></span></a></p>
Chris Gioran 💔<p>Starting the implementation of multithreaded access to the <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> store.</p><p>Thoughts and prayers can be sent as replies to this toot.</p>
Chris Gioran 💔<p>First pass of performance work done on <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a>|'s key memtables - it no longer deserializes not in use entries.</p><p>The point of optimization work this early was to get tests running fast enough, so I can run the entire test suite as often as I like during development.</p><p>Next up, start bringing transactions in the LS store.</p><p>(and you can read all about the store layout in my latest post: <a href="https://radiki.dev/posts/glowdust-lsm-tree-1/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">radiki.dev/posts/glowdust-lsm-</span><span class="invisible">tree-1/</span></a></p>
Chris Gioran 💔<p>A few years ago I wrote my first blog post - a code walkthrough of the storage format of a new, at the time, database: <a href="https://fosstodon.org/tags/neo4j" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>neo4j</span></a> </p><p>Yesterday my post was about my own storage format - a log structured key value store for <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a>.</p><p>Personal milestone moment, this one. Happy to be sharing it here.</p><p><a href="https://radiki.dev/posts/glowdust-lsm-tree-1/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">radiki.dev/posts/glowdust-lsm-</span><span class="invisible">tree-1/</span></a></p>
Chris Gioran 💔<p>You can't have a database without a store, now, can you?</p><p>Well, you can, actually, but that's not fun.</p><p>So <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> now sports its own native store.</p><p>Check it out - a log structured store for functions, the first step towards a full LSM Tree implementation.</p><p>Written in <a href="https://fosstodon.org/tags/Rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Rust</span></a>, of course</p><p><a href="https://radiki.dev/posts/glowdust-lsm-tree-1/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">radiki.dev/posts/glowdust-lsm-</span><span class="invisible">tree-1/</span></a></p><p><a href="https://fosstodon.org/tags/Rustlang" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Rustlang</span></a> <a href="https://fosstodon.org/tags/database" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>database</span></a></p>
Chris Gioran 💔<p>As the sole developer of <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a>, I need to choose if I'll support platforms without <a href="https://fosstodon.org/tags/io_uring" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>io_uring</span></a> support or drop uring support altogether.</p><p>The small experiment with CSV import showed me that I can't maintain two I/O stacks.</p><p>Plus, I started Glowdust in part because I wanted to play around with uring.</p><p>Hmm. Let's be honest - no one but me cares about Glowdust. And I only run <a href="https://fosstodon.org/tags/Linux" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Linux</span></a>.</p><p>And that's how, in the space of a single toot, I decided to do only io_uring.</p><p>Thank you for following along.</p>
Chris Gioran 💔<p>Log structured storage for <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> is getting faster, but its still slow.</p><p>Still some algorithmic improvements remaining, but I think sooner or later I'll have to deal with the cost of serializing/deserializing with serde.</p><p><a href="https://fosstodon.org/tags/rust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rust</span></a> <a href="https://fosstodon.org/tags/rustlang" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>rustlang</span></a></p>
Chris Gioran 💔<p>The advantage of using a log structured store for <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a> is that transaction state can be maintained without memory allocations - in fact, at the moment, the only allocation is the stack for the bytecode interpreter.</p><p>The disadvantage is that I don't yet know how to code a LS store and it's slow AF.</p><p><a href="https://fosstodon.org/tags/database" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>database</span></a></p>
Chris Gioran 💔<p>How often do you get to design a database transaction API?</p><p>For me, that's twice now.</p><p>Here's my notes on the second time - transaction API design for <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a></p><p><a href="https://radiki.dev/posts/glowdust-tx-design/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">radiki.dev/posts/glowdust-tx-d</span><span class="invisible">esign/</span></a></p>
Chris Gioran 💔<p>Huh, it looks like tomorrow will be the day when <a href="https://fosstodon.org/tags/Glowdust" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Glowdust</span></a>|'s Log Structured native store will have support for multiple pages.</p><p>You know, for the fancy pants use cases that need more than 64k of data.</p>
TrendingLive feeds
About