Kaifeng Zheng
Some notes on Rust, distributed systems, databases, and more.
Zola
2025-04-01T00:00:00+00:00
https://kev1n8.github.io/atom.xml
OpenTelemetry Basics Tour
2025-04-01T00:00:00+00:00
2025-04-01T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/opentelemetry-basics-tour/
<h1 id="opentelemetry-basics-tour">OpenTelemetry Basics Tour</h1>
<h2 id="intro">Intro</h2>
<p>OpenTelemetry is an observability framework and toolkit to <strong>create</strong> and <strong>manage</strong> telemetry data like <strong>traces</strong>, <strong>metrics</strong>, and <strong>logs</strong>.</p>
<h3 id="concepts">Concepts</h3>
<h4 id="traces">Traces</h4>
<p>Composed of spans in a tree structure. A trace represents the end-to-end journey of a request, while a span represents a single operation within that journey.</p>
<p><code>TracerProvider</code> is a factory for <code>Tracer</code>s. A <code>Tracer</code> creates spans with information about what is happening for a given operation.
<code>Trace Exporter</code> sends traces to a consumer, which can be standard output, the OpenTelemetry Collector, or any open source or vendor backend.</p>
<h5 id="spans">Spans</h5>
<p>Fields contained:</p>
<ul>
<li>Name</li>
<li>Parent span ID (empty for root spans)</li>
<li>Start and End Timestamps</li>
<li>Span Context</li>
<li>Attributes — info in KV pairs</li>
<li>Span Events — things happening at a point in time, whose timestamp is meaningful</li>
<li>Span Links — indicate causal relationships between spans in distributed situations</li>
<li>Span Status — indicates whether the tracked operation completed with or without error (<code>Unset</code>, <code>Error</code>), or was explicitly set to <code>Ok</code></li>
</ul>
<p>Kind:</p>
<ul>
<li>Client — remote call such as an outgoing HTTP request or DB call</li>
<li>Server — incoming remote call such as an incoming HTTP request or RPC</li>
<li>Internal — operations that do not cross a process boundary, such as instrumenting a function call</li>
<li>Producer — represents the creation of a job</li>
<li>Consumer — represents the processing of a job, after the Producer span has already ended</li>
</ul>
<h4 id="metrics">Metrics</h4>
<p>A <strong>metric</strong> is a measurement of a service captured at runtime. The moment of capturing a measurement is known as a <strong>metric event</strong>.</p>
<p>Application and request metrics are important indicators of availability and performance.</p>
<p>A <code>MeterProvider</code> is a factory for <code>Meter</code>s. A <code>Meter</code> creates metric instruments, capturing measurements about a service at runtime. <code>Metric Exporter</code>s send metric data to a consumer (similarly to <code>Trace Exporter</code>).</p>
<h5 id="metric-instruments">Metric Instruments</h5>
<p>Fields:</p>
<ul>
<li>Name</li>
<li>Kind</li>
<li>Unit (optional)</li>
<li>Description (optional)</li>
</ul>
<p>Name, unit, and description are chosen by the developer or defined via <strong>semantic conventions</strong> for common ones like request and process metrics.</p>
<p>Kind enums:</p>
<ul>
<li>Counter — a value accumulating over time</li>
<li>Asynchronous Counter — same, but is collected <strong>once</strong> for the export</li>
<li>UpDownCounter — counter that can go down</li>
<li>Asynchronous UpDownCounter</li>
<li>Gauge</li>
<li>Histogram</li>
</ul>
<h5 id="aggregation">Aggregation</h5>
<p>Exact or estimated statistics about metric events during a time window.</p>
<p>e.g.</p>
<ul>
<li>Total bytes read by a service, per protocol type</li>
<li>Total bytes read and bytes per request</li>
<li>Duration of a system call</li>
<li>Request sizes for trend prediction</li>
<li>CPU or memory usage of a process</li>
</ul>
<h4 id="logs">Logs</h4>
<p>A log is a timestamped text record.</p>
<p>Log Appender/Bridge emits logs into an OpenTelemetry <code>LogRecordExporter</code>.</p>
<p><code>LoggerProvider</code> is a factory for <code>Logger</code>s. A <code>Logger</code> creates log records. <code>LogRecordExporter</code>s send log records to a consumer.</p>
<p>Fields:</p>
<ul>
<li>Timestamp — Time when the event occurred</li>
<li>ObservedTimestamp — Time when the event was observed</li>
<li>TraceId — Request trace ID</li>
<li>SpanId — Request span ID</li>
<li>TraceFlags — W3C trace flag</li>
<li>SeverityText — The severity text (also known as log level)</li>
<li>SeverityNumber — Numerical value of the severity</li>
<li>Body — The body of the log record</li>
<li>Resource — Describes the source of the log</li>
<li>InstrumentationScope — Describes the scope that emitted the log</li>
<li>Attributes — Additional information about the event</li>
</ul>
<h3 id="major-components">Major Components</h3>
<ul>
<li>A specification for all components</li>
<li>A standard protocol that defines the shape of telemetry data</li>
<li>Semantic conventions that define a standard naming scheme for common telemetry data types</li>
<li>APIs that define how to generate telemetry data</li>
<li>Language SDKs that implement the specification, APIs, and export of telemetry data</li>
<li>A library ecosystem that implements instrumentation for common libraries and frameworks</li>
<li>Automatic instrumentation components that generate telemetry data without requiring code changes</li>
<li>The OpenTelemetry Collector, a proxy that receives, processes, and exports telemetry data</li>
<li>Other tools</li>
</ul>
Rust WASM: Building Conway's Game of Life
2025-04-01T00:00:00+00:00
2025-04-01T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/rust-wasm-building-conway-s-game-of-life/
<h1 id="rust-wasm-building-conway-s-game-of-life">Rust WASM: Building Conway's Game of Life</h1>
<p>I've heard of wasm for quite a while, and finally decided to try it out.</p>
<p>This blog briefly writes the process of the practice.</p>
<h2 id="prerequisites">Prerequisites</h2>
<p>Should know basic web development stuff, since wasm is run inside a browser (mostly).</p>
<ul>
<li>HTML</li>
<li>CSS</li>
<li>JavaScript</li>
</ul>
<p>There are technically 2 ways of using rust wasm.</p>
<ol>
<li>Build the whole project in Rust, or</li>
<li>Rust the heavy part of the project and use it as a lib.</li>
</ol>
<p>Currently the latter one is preferred.</p>
<h2 id="game-of-life">Game of Life</h2>
<p>The practice is going to focus on implementing Conway's Game of Life in Rust, building it to wasm and use it in JS.</p>
<p>The rules of the game are:</p>
<ol>
<li>Any live cell with fewer than two live neighbours dies, as if by underpopulation.</li>
<li>Any live cell with two or three live neighbours lives on to the next generation.</li>
<li>Any live cell with more than three live neighbours dies, as if by overpopulation.</li>
<li>Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.</li>
</ol>
<p>For more info: https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life</p>
<h2 id="implementation">Implementation</h2>
<h3 id="rust-part">Rust Part</h3>
<p>Rust comes with powerful tools like <code>wasm-pack</code>, <code>wasm-bindgen</code> and more for wasm programming.</p>
<p>We use <code>#[wasm-bindgen]</code> macro to make complier generates wasm-related files (js API file, wasm memory file, ts file and so on) when we use <code>wasm-pack build</code> to build our rust wasm lib.</p>
<p>I'll simply copy the code as follows:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> fixedbitset</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">FixedBitSet</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> wasm_bindgen</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">prelude</span><span style="color: #E06C75;">::*</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[wasm_bindgen]</span></span>
<span class="giallo-l"><span>#[repr(</span><span style="color: #E5C07B;">u8</span><span>)]</span></span>
<span class="giallo-l"><span>#[derive(</span><span style="color: #E5C07B;">Clone</span><span>,</span><span style="color: #E5C07B;"> Copy</span><span>,</span><span style="color: #E5C07B;"> Debug</span><span>,</span><span style="color: #E5C07B;"> PartialEq</span><span>,</span><span style="color: #E5C07B;"> Eq</span><span>)]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> enum</span><span style="color: #E5C07B;"> Cell</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Dead</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Alive</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 1</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[wasm_bindgen]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> Universe</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> width</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> height</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> cells</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> FixedBitSet</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Universe</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> get_index</span><span>(</span><span style="color: #E06C75;">&self</span><span>,</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>,</span><span style="color: #C6CCD7;"> column</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> usize</span><span> {</span></span>
<span class="giallo-l"><span> (</span><span style="color: #C6CCD7;">row</span><span style="color: #E06C75;"> * self.</span><span>width </span><span style="color: #E06C75;">+</span><span style="color: #C6CCD7;"> column</span><span>)</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> usize</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> live_neighbor_count</span><span>(</span><span style="color: #E06C75;">&self</span><span>,</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>,</span><span style="color: #C6CCD7;"> column</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> u8</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> count</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> delta_row</span><span style="color: #E06C75;"> in</span><span> [</span><span style="color: #E06C75;">self.</span><span>height </span><span style="color: #E06C75;">-</span><span style="color: #56B6C2;"> 1</span><span>,</span><span style="color: #56B6C2;"> 0</span><span>,</span><span style="color: #56B6C2;"> 1</span><span>] {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> delta_col</span><span style="color: #E06C75;"> in</span><span> [</span><span style="color: #E06C75;">self.</span><span>width </span><span style="color: #E06C75;">-</span><span style="color: #56B6C2;"> 1</span><span>,</span><span style="color: #56B6C2;"> 0</span><span>,</span><span style="color: #56B6C2;"> 1</span><span>] {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> delta_row</span><span style="color: #E06C75;"> ==</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;"> &&</span><span style="color: #C6CCD7;"> delta_col</span><span style="color: #E06C75;"> ==</span><span style="color: #56B6C2;"> 0</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> continue</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> neighbor_row</span><span style="color: #E06C75;"> =</span><span> (</span><span style="color: #C6CCD7;">row</span><span style="color: #E06C75;"> +</span><span style="color: #C6CCD7;"> delta_row</span><span>)</span><span style="color: #E06C75;"> % self.</span><span>height;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> neighbor_col</span><span style="color: #E06C75;"> =</span><span> (</span><span style="color: #C6CCD7;">column</span><span style="color: #E06C75;"> +</span><span style="color: #C6CCD7;"> delta_col</span><span>)</span><span style="color: #E06C75;"> % self.</span><span>width;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> idx</span><span style="color: #E06C75;"> = self.</span><span style="color: #B57EDC;">get_index</span><span>(</span><span style="color: #C6CCD7;">neighbor_row</span><span>,</span><span style="color: #C6CCD7;"> neighbor_col</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> count</span><span style="color: #E06C75;"> += self.</span><span>cells[</span><span style="color: #C6CCD7;">idx</span><span>]</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> u8</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> count</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[wasm_bindgen]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Universe</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> new</span><span>()</span><span style="color: #E06C75;"> -> Self</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> width</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 64</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> height</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 64</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> sz</span><span style="color: #E06C75;"> =</span><span> (</span><span style="color: #C6CCD7;">width</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> height</span><span>)</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> usize</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> cells</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> FixedBitSet</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">with_capacity</span><span>(</span><span style="color: #C6CCD7;">sz</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..</span><span style="color: #C6CCD7;">sz</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> cells</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set</span><span>(</span><span style="color: #C6CCD7;">i</span><span>,</span><span style="color: #61AFEF;"> js_sys</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">Math</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">random</span><span>() <</span><span style="color: #56B6C2;"> 0</span><span>.</span><span style="color: #56B6C2;">5</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> Self</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> width</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> height</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> cells</span><span>,</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> width</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> u32</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>width</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> height</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> u32</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>height</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> cells</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;"> -> *</span><span style="color: #61AFEF;">const</span><span style="color: #E5C07B;"> usize</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>cells</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_slice</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_ptr</span><span>()</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> tick</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> next</span><span style="color: #E06C75;"> = self.</span><span>cells</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">clone</span><span>();</span></span>
<span class="giallo-l"><span> </span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..self.</span><span>height {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..self.</span><span>width {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> idx</span><span style="color: #E06C75;"> = self.</span><span style="color: #B57EDC;">get_index</span><span>(</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #C6CCD7;"> col</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> cell</span><span style="color: #E06C75;"> = self.</span><span>cells[</span><span style="color: #C6CCD7;">idx</span><span>];</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> live_neigbors</span><span style="color: #E06C75;"> = self.</span><span style="color: #B57EDC;">live_neighbor_count</span><span>(</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #C6CCD7;"> col</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> next</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set</span><span>(</span><span style="color: #C6CCD7;">idx</span><span>,</span><span style="color: #E06C75;"> match</span><span> (</span><span style="color: #C6CCD7;">cell</span><span>,</span><span style="color: #C6CCD7;"> live_neigbors</span><span>) {</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">true</span><span>,</span><span style="color: #C6CCD7;"> x</span><span>)</span><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> <</span><span style="color: #56B6C2;"> 2</span><span style="color: #E06C75;"> =></span><span style="color: #56B6C2;"> false</span><span>,</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">true</span><span>,</span><span style="color: #56B6C2;"> 2</span><span>)</span><span style="color: #E06C75;"> |</span><span> (</span><span style="color: #56B6C2;">true</span><span>,</span><span style="color: #56B6C2;"> 3</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #56B6C2;"> true</span><span>,</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">true</span><span>,</span><span style="color: #C6CCD7;"> x</span><span>)</span><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> ></span><span style="color: #56B6C2;"> 3</span><span style="color: #E06C75;"> =></span><span style="color: #56B6C2;"> false</span><span>,</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">false</span><span>,</span><span style="color: #56B6C2;"> 3</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #56B6C2;"> true</span><span>,</span></span>
<span class="giallo-l"><span> (</span><span style="color: #C6CCD7;">otherwise</span><span>,</span><span style="color: #C6CCD7;"> _</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #C6CCD7;"> otherwise</span><span>,</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>cells </span><span style="color: #E06C75;">=</span><span style="color: #C6CCD7;"> next</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="javascript-part">JavaScript Part</h3>
<p>With the functional lib we built, now all we have to do is to use it to display the animation on the browser.</p>
<p>A basic web app is consist of:</p>
<ul>
<li>HTML: Defining the layout of the website</li>
<li>CSS: Defining the style of the website</li>
<li>JavaScript: Handling any user input and logical changes.</li>
</ul>
<p>For HTML and CSS:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="html"><span class="giallo-l"><span><!</span><span style="color: #E5C07B;">DOCTYPE</span><span style="color: #D19A66;"> html</span><span>></span></span>
<span class="giallo-l"><span><</span><span style="color: #E5C07B;">html</span><span>></span></span>
<span class="giallo-l"><span> <</span><span style="color: #E5C07B;">head</span><span>></span></span>
<span class="giallo-l"><span> <</span><span style="color: #E5C07B;">meta</span><span style="color: #D19A66;"> charset</span><span>="</span><span style="color: #98C379;">utf-8</span><span>"></span></span>
<span class="giallo-l"><span> <</span><span style="color: #E5C07B;">title</span><span>>Hello wasm-pack!</</span><span style="color: #E5C07B;">title</span><span>></span></span>
<span class="giallo-l"><span> </</span><span style="color: #E5C07B;">head</span><span>></span></span>
<span class="giallo-l"><span> <</span><span style="color: #E5C07B;">body</span><span>></span></span>
<span class="giallo-l"><span> <</span><span style="color: #E5C07B;">canvas</span><span style="color: #D19A66;"> id</span><span>="</span><span style="color: #98C379;">game-of-life-canvas</span><span>"></</span><span style="color: #E5C07B;">canvas</span><span>></span></span>
<span class="giallo-l"><span> <</span><span style="color: #E5C07B;">script</span><span style="color: #D19A66;"> src</span><span>="</span><span style="color: #98C379;">./bootstrap.js</span><span>"></</span><span style="color: #E5C07B;">script</span><span>></span></span>
<span class="giallo-l"><span> </</span><span style="color: #E5C07B;">body</span><span>></span></span>
<span class="giallo-l"><span> <</span><span style="color: #E5C07B;">style</span><span>></span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> body</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> position</span><span>:</span><span style="color: #56B6C2;"> absolute</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> top</span><span>:</span><span style="color: #56B6C2;"> 0</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> left</span><span>:</span><span style="color: #56B6C2;"> 0</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> width</span><span>:</span><span style="color: #56B6C2;"> 100</span><span style="color: #E06C75;">%</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> height</span><span>:</span><span style="color: #56B6C2;"> 100</span><span style="color: #E06C75;">%</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> display</span><span>:</span><span style="color: #56B6C2;"> flex</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> flex-direction</span><span>:</span><span style="color: #56B6C2;"> column</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> align-items</span><span>:</span><span style="color: #56B6C2;"> center</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> justify-content</span><span>:</span><span style="color: #56B6C2;"> center</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> </</span><span style="color: #E5C07B;">style</span><span>></span></span>
<span class="giallo-l"><span></</span><span style="color: #E5C07B;">html</span><span>></span></span></code></pre>
<p>For JS:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="javascript"><span class="giallo-l"><span style="color: #E06C75;">import</span><span> {</span><span style="color: #C6CCD7;"> Universe</span><span>,</span><span style="color: #C6CCD7;"> Cell</span><span> }</span><span style="color: #E06C75;"> from</span><span> "</span><span style="color: #98C379;">wasm-game-of-life</span><span>";</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Import the linear memory space of wasm.</span></span>
<span class="giallo-l"><span style="color: #E06C75;">import</span><span> {</span><span style="color: #C6CCD7;"> memory</span><span> }</span><span style="color: #E06C75;"> from</span><span> "</span><span style="color: #98C379;">wasm-game-of-life/wasm_game_of_life_bg.wasm</span><span>";</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> CELL_SIZE</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 5</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // px</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> GRID_COLOR</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">#CCCCCC</span><span>";</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> DEAD_COLOR</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">#FFFFFF</span><span>";</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> ALIVE_COLOR</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">#000000</span><span>";</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Construct the universe, and get its width and height.</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> Universe</span><span>.</span><span style="color: #B57EDC;">new</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> width</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> universe</span><span>.</span><span style="color: #B57EDC;">width</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> height</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> universe</span><span>.</span><span style="color: #B57EDC;">height</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Give the canvas room for all of our cells and a 1px border</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// around each of them.</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> canvas</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> document</span><span>.</span><span style="color: #B57EDC;">getElementById</span><span>("</span><span style="color: #98C379;">game-of-life-canvas</span><span>");</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">canvas</span><span>.</span><span style="color: #C6CCD7;">height</span><span style="color: #E06C75;"> =</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> height</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>;</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">canvas</span><span>.</span><span style="color: #C6CCD7;">width</span><span style="color: #E06C75;"> =</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> width</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> ctx</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> canvas</span><span>.</span><span style="color: #B57EDC;">getContext</span><span>('</span><span style="color: #98C379;">2d</span><span>');</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #B57EDC;"> renderLoop</span><span style="color: #E06C75;"> =</span><span> ()</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span><span>.</span><span style="color: #B57EDC;">tick</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;"> drawGrid</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> drawCells</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;"> requestAnimationFrame</span><span>(</span><span style="color: #C6CCD7;">renderLoop</span><span>);</span></span>
<span class="giallo-l"><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #B57EDC;"> drawGrid</span><span style="color: #E06C75;"> =</span><span> ()</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">beginPath</span><span>();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #C6CCD7;">strokeStyle</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> GRID_COLOR</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Vertical lines.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span> (</span><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>;</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;"> <=</span><span style="color: #C6CCD7;"> width</span><span>;</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;">++</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">moveTo</span><span>(</span><span style="color: #C6CCD7;">i</span><span style="color: #E06C75;"> *</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>,</span><span style="color: #56B6C2;"> 0</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">lineTo</span><span>(</span><span style="color: #C6CCD7;">i</span><span style="color: #E06C75;"> *</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>, (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> height</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Horizontal lines.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span> (</span><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> j</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>;</span><span style="color: #C6CCD7;"> j</span><span style="color: #E06C75;"> <=</span><span style="color: #C6CCD7;"> height</span><span>;</span><span style="color: #C6CCD7;"> j</span><span style="color: #E06C75;">++</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">moveTo</span><span>(</span><span style="color: #56B6C2;">0</span><span>,</span><span style="color: #C6CCD7;"> j</span><span style="color: #E06C75;"> *</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">lineTo</span><span>((</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> width</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>,</span><span style="color: #C6CCD7;"> j</span><span style="color: #E06C75;"> *</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">stroke</span><span>();</span></span>
<span class="giallo-l"><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #B57EDC;"> getIndex</span><span style="color: #E06C75;"> =</span><span> (</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #C6CCD7;"> column</span><span>)</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> width</span><span style="color: #E06C75;"> +</span><span style="color: #C6CCD7;"> column</span><span>;</span></span>
<span class="giallo-l"><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #B57EDC;"> drawCells</span><span style="color: #E06C75;"> =</span><span> ()</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Accessing the buffer of wasm.</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> cellsPtr</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> universe</span><span>.</span><span style="color: #B57EDC;">cells</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> cells</span><span style="color: #E06C75;"> = new</span><span style="color: #B57EDC;"> Uint8Array</span><span>(</span><span style="color: #C6CCD7;">memory</span><span>.</span><span style="color: #C6CCD7;">buffer</span><span>,</span><span style="color: #C6CCD7;"> cellsPtr</span><span>,</span><span style="color: #C6CCD7;"> width</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> height</span><span style="color: #E06C75;"> /</span><span style="color: #56B6C2;"> 8</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">beginPath</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span> (</span><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>;</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;"> <</span><span style="color: #C6CCD7;"> height</span><span>;</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;">++</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span> (</span><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>;</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> <</span><span style="color: #C6CCD7;"> width</span><span>;</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;">++</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> idx</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> getIndex</span><span>(</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #C6CCD7;"> col</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #C6CCD7;">fillStyle</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> cells</span><span>[</span><span style="color: #C6CCD7;">idx</span><span>]</span><span style="color: #E06C75;"> ===</span><span style="color: #C6CCD7;"> Cell</span><span>.</span><span style="color: #C6CCD7;">Dead</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ?</span><span style="color: #C6CCD7;"> DEAD_COLOR</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> :</span><span style="color: #C6CCD7;"> ALIVE_COLOR</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">fillRect</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> *</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;"> *</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> CELL_SIZE</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> CELL_SIZE</span></span>
<span class="giallo-l"><span> );</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span>.</span><span style="color: #B57EDC;">stroke</span><span>();</span></span>
<span class="giallo-l"><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;">drawGrid</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">drawCells</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">requestAnimationFrame</span><span>(</span><span style="color: #C6CCD7;">renderLoop</span><span>);</span></span></code></pre><h2 id="testing">Testing</h2>
<p>Example test.rs:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Only test for target: "wasm-unknown-unknown"</span></span>
<span class="giallo-l"><span>#![cfg(target_arch </span><span style="color: #E06C75;">=</span><span> "</span><span style="color: #98C379;">wasm32</span><span>")]</span></span>
<span class="giallo-l"><span>#![cfg(test)]</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Use wasm_bindgen_test and our [`Universe`];</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> wasm_bindgen_test</span><span style="color: #E06C75;">::*</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> wasm_game_of_life</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Universe</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Configure our test to run in the browser.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">wasm_bindgen_test_configure!</span><span>(</span><span style="color: #C6CCD7;">run_in_browser</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> input_spaceship</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Universe</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Universe</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set_width</span><span>(</span><span style="color: #56B6C2;">6</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set_height</span><span>(</span><span style="color: #56B6C2;">6</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set_cells</span><span>(</span><span style="color: #E06C75;">&</span><span>[(</span><span style="color: #56B6C2;">1</span><span>,</span><span style="color: #56B6C2;">2</span><span>), (</span><span style="color: #56B6C2;">2</span><span>,</span><span style="color: #56B6C2;">3</span><span>), (</span><span style="color: #56B6C2;">3</span><span>,</span><span style="color: #56B6C2;">1</span><span>), (</span><span style="color: #56B6C2;">3</span><span>,</span><span style="color: #56B6C2;">2</span><span>), (</span><span style="color: #56B6C2;">3</span><span>,</span><span style="color: #56B6C2;">3</span><span>)]);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> expected_spaceship</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Universe</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Universe</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set_width</span><span>(</span><span style="color: #56B6C2;">6</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set_height</span><span>(</span><span style="color: #56B6C2;">6</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set_cells</span><span>(</span><span style="color: #E06C75;">&</span><span>[(</span><span style="color: #56B6C2;">2</span><span>,</span><span style="color: #56B6C2;">1</span><span>), (</span><span style="color: #56B6C2;">2</span><span>,</span><span style="color: #56B6C2;">3</span><span>), (</span><span style="color: #56B6C2;">3</span><span>,</span><span style="color: #56B6C2;">2</span><span>), (</span><span style="color: #56B6C2;">3</span><span>,</span><span style="color: #56B6C2;">3</span><span>), (</span><span style="color: #56B6C2;">4</span><span>,</span><span style="color: #56B6C2;">2</span><span>)]);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Similar to #[test]</span></span>
<span class="giallo-l"><span>#[wasm_bindgen_test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> test_tick</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> input_universe</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> input_spaceship</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> expected_universe</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> expected_spaceship</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> input_universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">tick</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">input_universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">get_cells</span><span>(),</span><span style="color: #C6CCD7;"> expected_universe</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">get_cells</span><span>());</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>By setting the test rs file, we can simply <code>wasm-pack test --chrome --headless</code> to test our test suite.</p>
<p><code>--chrome</code> means we prefer to run the test in Google Chrome, so be sure the latest Google Chrome is downloaded.</p>
<p><code>--headless</code> means we prefer to test without opening the browser and viewing.</p>
<h2 id="debugging">Debugging</h2>
<h3 id="log-for-panics">Log for Panics</h3>
<p>If our rust wasm panics for unknown reasons, we would like to at least be aware of the location of panicking.</p>
<p>We could achieve this by using the crate <code>console_error_panic_hook</code>.</p>
<p>Usage is simply:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Place this in the init function.</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// `set_once` ensure `set_hook` is called once.</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">console_error_panic_hook</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">set_once</span><span>();</span></span></code></pre>
<p>Then, if the rust part panics, we'll get messages like (build with <code>wasm-pack build --dev</code>):</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="plain"><span class="giallo-l"><span>panicked at src/lib.rs:65:9:</span></span>
<span class="giallo-l"><span>explicit panic</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>Stack:</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>@</span></span>
<span class="giallo-l"><span>logError@</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[__wbg_new_8a6f238a6ece86ea externref shim]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[console_error_panic_hook::Error::new::h19e56b4cb73b36b1]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[console_error_panic_hook::hook_impl::hbfe233a683db229c]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[console_error_panic_hook::hook::h9ea87a8daf4cfd59]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[core::ops::function::Fn::call::h9d1dc855776bf28e]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[std::panicking::rust_panic_with_hook::h9ea01914b5d92659]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[std::panicking::begin_panic::{{closure}}::h1965b49f537d1c88]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[std::sys::backtrace::__rust_end_short_backtrace::hdeabdd4295ec7969]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[std::panicking::begin_panic::hbacf3f38466eeebe]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[wasm_game_of_life::Universe::new::hc5d33d50da932d85]@[wasm code]</span></span>
<span class="giallo-l"><span>wasm_game_of_life.wasm.wasm-function[universe_new]@[wasm code]</span></span>
<span class="giallo-l"><span>new@</span></span>
<span class="giallo-l"><span>@</span></span></code></pre><h3 id="log-error-messages-to-console">Log Error Messages to Console</h3>
<p>It's useful to log info if something went wrong. How do we log then?</p>
<p>We need another dependency, <code>web-sys</code>, which provides us the <code>console::log</code> API.</p>
<p>Well, simple as that. Then we can write a macro to easy up the coding:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #B57EDC;">macro_rules! log</span><span> {</span></span>
<span class="giallo-l"><span> (</span><span style="color: #E06C75;"> $</span><span>(</span><span style="color: #E06C75;"> $</span><span style="color: #C6CCD7;">t</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;">tt</span><span> )</span><span style="color: #E06C75;">*</span><span> )</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> web_sys</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">console</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">log_1</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #B57EDC;">format!</span><span>(</span><span style="color: #E06C75;"> $</span><span>(</span><span style="color: #E06C75;">$</span><span style="color: #C6CCD7;">t</span><span>)</span><span style="color: #E06C75;">*</span><span> )</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">into</span><span>());</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Log while ticking.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">log!</span><span>(</span></span>
<span class="giallo-l"><span> "</span><span style="color: #98C379;">cell[</span><span>{}</span><span style="color: #98C379;">, </span><span>{}</span><span style="color: #98C379;">] is initially </span><span>{}</span><span style="color: #98C379;"> and has </span><span>{}</span><span style="color: #98C379;"> live neighbors</span><span>",</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> row</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> col</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> cell</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> live_neigbors</span></span>
<span class="giallo-l"><span>);</span></span></code></pre><h3 id="use-a-debugger">Use a Debugger</h3>
<p>Either by placing an <code>debugger</code> in advanced in <code>.js</code> or by clicking in the sources interface of the browser is ok for js.</p>
<p>However, there's currently no mature debugging tool for rust wasm. We may end up stepping into raw wasm instructions rather than the Rust source text we authored.</p>
<h2 id="interaction-between-rust-and-js">Interaction between Rust and JS</h2>
<p>We are going to add 2 interactions:</p>
<ol>
<li>Play and pause button, and</li>
<li>Toggle cell state by clicking</li>
</ol>
<h3 id="add-a-button">Add a Button</h3>
<p>Add a button in <code>.html</code> with <code><button id="play-pause"></button></code>.</p>
<p>In <code>index.js</code>, add the logic code:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="javascript"><span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> playPauseButton</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> document</span><span>.</span><span style="color: #B57EDC;">getElementById</span><span>("</span><span style="color: #98C379;">play-pause</span><span>");</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #B57EDC;"> isPaused</span><span style="color: #E06C75;"> =</span><span> ()</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #C6CCD7;"> animationId</span><span style="color: #E06C75;"> ===</span><span style="color: #56B6C2;"> null</span><span>;</span></span>
<span class="giallo-l"><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #B57EDC;"> play</span><span style="color: #E06C75;"> =</span><span> ()</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> playPauseButton</span><span>.</span><span style="color: #C6CCD7;">textContent</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">⏸</span><span>";</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> renderLoop</span><span>();</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #B57EDC;"> pause</span><span style="color: #E06C75;"> =</span><span> ()</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> playPauseButton</span><span>.</span><span style="color: #C6CCD7;">textContent</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">▶</span><span>";</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> cancelAnimationFrame</span><span>(</span><span style="color: #C6CCD7;">animationId</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> animationId</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> null</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // This would be set in `renderLoop`.</span></span>
<span class="giallo-l"><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;">playPauseButton</span><span>.</span><span style="color: #B57EDC;">addEventListener</span><span>("</span><span style="color: #98C379;">click</span><span>",</span><span style="color: #C6CCD7;"> event</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span> (</span><span style="color: #B57EDC;">isPaused</span><span>()) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> play</span><span>();</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> pause</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>});</span></span></code></pre><h3 id="toggling-cells">Toggling Cells</h3>
<p>First we have to impl the toggle method on the rust side.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Universe</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> toggle_cell</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self</span><span>,</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>,</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> idx</span><span style="color: #E06C75;"> = self.</span><span style="color: #B57EDC;">get_index</span><span>(</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #C6CCD7;"> col</span><span>);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>cells</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set</span><span>(</span><span style="color: #C6CCD7;">idx</span><span>,</span><span style="color: #E06C75;"> !self.</span><span>cells[</span><span style="color: #C6CCD7;">idx</span><span>]);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>Then, we add the toggling logic code in js.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="javascript"><span class="giallo-l"><span style="color: #C6CCD7;">canvas</span><span>.</span><span style="color: #B57EDC;">addEventListener</span><span>("</span><span style="color: #98C379;">click</span><span>",</span><span style="color: #C6CCD7;"> event</span><span style="color: #61AFEF;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> boundingRect</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> canvas</span><span>.</span><span style="color: #B57EDC;">getBoundingClientRect</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> scaleX</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> canvas</span><span>.</span><span style="color: #C6CCD7;">width</span><span style="color: #E06C75;"> /</span><span style="color: #C6CCD7;"> boundingRect</span><span>.</span><span style="color: #C6CCD7;">width</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> scaleY</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> canvas</span><span>.</span><span style="color: #C6CCD7;">height</span><span style="color: #E06C75;"> /</span><span style="color: #C6CCD7;"> boundingRect</span><span>.</span><span style="color: #C6CCD7;">height</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> canvasLeft</span><span style="color: #E06C75;"> =</span><span> (</span><span style="color: #C6CCD7;">event</span><span>.</span><span style="color: #C6CCD7;">clientX</span><span style="color: #E06C75;"> -</span><span style="color: #C6CCD7;"> boundingRect</span><span>.</span><span style="color: #C6CCD7;">left</span><span>)</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> scaleX</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> canvasTop</span><span style="color: #E06C75;"> =</span><span> (</span><span style="color: #C6CCD7;">event</span><span>.</span><span style="color: #C6CCD7;">clientY</span><span style="color: #E06C75;"> -</span><span style="color: #C6CCD7;"> boundingRect</span><span>.</span><span style="color: #C6CCD7;">top</span><span>)</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> scaleY</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> Math</span><span>.</span><span style="color: #B57EDC;">min</span><span>(</span><span style="color: #C6CCD7;">Math</span><span>.</span><span style="color: #B57EDC;">floor</span><span>(</span><span style="color: #C6CCD7;">canvasTop</span><span style="color: #E06C75;"> /</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>)),</span><span style="color: #C6CCD7;"> height</span><span style="color: #E06C75;"> -</span><span style="color: #56B6C2;"> 1</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> Math</span><span>.</span><span style="color: #B57EDC;">min</span><span>(</span><span style="color: #C6CCD7;">Math</span><span>.</span><span style="color: #B57EDC;">floor</span><span>(</span><span style="color: #C6CCD7;">canvasLeft</span><span style="color: #E06C75;"> /</span><span> (</span><span style="color: #C6CCD7;">CELL_SIZE</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span>),</span><span style="color: #C6CCD7;"> width</span><span style="color: #E06C75;"> -</span><span style="color: #56B6C2;"> 1</span><span>));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> universe</span><span>.</span><span style="color: #B57EDC;">toggle_cell</span><span>(</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #C6CCD7;"> col</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;"> drawCells</span><span>();</span></span>
<span class="giallo-l"><span>})</span></span></code></pre><h2 id="time-profiling">Time Profiling</h2>
<h3 id="counting-fps-by-performance-now">Counting FPS by <code>performance.now()</code></h3>
<p>Add dependency <code>web-sys</code> with features <code>["Window", "Performance"]</code> if would like to count fps in Rust.</p>
<p>Or we can simply count it in js:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="javascript"><span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">const</span><span style="color: #C6CCD7;"> fps</span><span style="color: #E06C75;"> = new</span><span style="color: #61AFEF;"> class</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> constructor</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> this</span><span>.</span><span style="color: #C6CCD7;">fps</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> document</span><span>.</span><span style="color: #B57EDC;">getElementById</span><span>("</span><span style="color: #98C379;">fps</span><span>");</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span style="color: #E06C75;"> =</span><span> [];</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> this</span><span>.</span><span style="color: #C6CCD7;">lastFrameTimeStamp</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> performance</span><span>.</span><span style="color: #B57EDC;">now</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;"> render</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> now</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> performance</span><span>.</span><span style="color: #B57EDC;">now</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> delta</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> now</span><span style="color: #E06C75;"> - this</span><span>.</span><span style="color: #C6CCD7;">lastFrameTimeStamp</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> this</span><span>.</span><span style="color: #C6CCD7;">lastFrameTimeStamp</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> now</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> const</span><span style="color: #C6CCD7;"> fps</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 1</span><span style="color: #E06C75;"> /</span><span style="color: #C6CCD7;"> delta</span><span style="color: #E06C75;"> *</span><span style="color: #56B6C2;"> 1000</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span>.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #C6CCD7;">fps</span><span>);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span> (</span><span style="color: #E06C75;">this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span>.</span><span style="color: #C6CCD7;">length</span><span style="color: #E06C75;"> ></span><span style="color: #56B6C2;"> 100</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span>.</span><span style="color: #B57EDC;">shift</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> min</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> Infinity</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> max</span><span style="color: #E06C75;"> = -</span><span style="color: #56B6C2;">Infinity</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> sum</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span> (</span><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>;</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;"> < this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span>.</span><span style="color: #C6CCD7;">length</span><span>;</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;">++</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> sum</span><span style="color: #E06C75;"> += this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span>[</span><span style="color: #C6CCD7;">i</span><span>];</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> min</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> Math</span><span>.</span><span style="color: #B57EDC;">min</span><span>(</span><span style="color: #E06C75;">this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span>[</span><span style="color: #C6CCD7;">i</span><span>],</span><span style="color: #C6CCD7;"> min</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> max</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> Math</span><span>.</span><span style="color: #B57EDC;">max</span><span>(</span><span style="color: #E06C75;">this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span>[</span><span style="color: #C6CCD7;">i</span><span>],</span><span style="color: #C6CCD7;"> max</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> mean</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> sum</span><span style="color: #E06C75;"> / this</span><span>.</span><span style="color: #C6CCD7;">frames</span><span>.</span><span style="color: #C6CCD7;">length</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> this</span><span>.</span><span style="color: #C6CCD7;">fps</span><span>.</span><span style="color: #C6CCD7;">textContent</span><span style="color: #E06C75;"> =</span><span> `</span></span>
<span class="giallo-l"><span style="color: #98C379;">Frames per Second:</span></span>
<span class="giallo-l"><span style="color: #98C379;"> latest = </span><span>${</span><span style="color: #C6CCD7;">Math</span><span>.</span><span style="color: #B57EDC;">round</span><span>(</span><span style="color: #C6CCD7;">fps</span><span>)}</span></span>
<span class="giallo-l"><span style="color: #98C379;">avg of last 100 = </span><span>${</span><span style="color: #C6CCD7;">Math</span><span>.</span><span style="color: #B57EDC;">round</span><span>(</span><span style="color: #C6CCD7;">mean</span><span>)}</span></span>
<span class="giallo-l"><span style="color: #98C379;">min of last 100 = </span><span>${</span><span style="color: #C6CCD7;">Math</span><span>.</span><span style="color: #B57EDC;">round</span><span>(</span><span style="color: #C6CCD7;">min</span><span>)}</span></span>
<span class="giallo-l"><span style="color: #98C379;">max of last 100 = </span><span>${</span><span style="color: #C6CCD7;">Math</span><span>.</span><span style="color: #B57EDC;">round</span><span>(</span><span style="color: #C6CCD7;">max</span><span>)}</span></span>
<span class="giallo-l"><span>`.</span><span style="color: #B57EDC;">trim</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>};</span></span></code></pre>
<p>And call <code>fps.render()</code> in <code>renderLoop</code>.</p>
<h3 id="set-timer-for-universe-tick">Set Timer for <code>Universe::tick</code></h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> Timer</span><span><'</span><span style="color: #E5C07B;">a</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span><'</span><span style="color: #E5C07B;">a</span><span>></span><span style="color: #E5C07B;"> Timer</span><span><'</span><span style="color: #E5C07B;">a</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> new</span><span>(</span><span style="color: #C6CCD7;">name</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>)</span><span style="color: #E06C75;"> -> Self</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> console</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">time_with_label</span><span>(</span><span style="color: #C6CCD7;">name</span><span>);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> Self</span><span> {</span><span style="color: #C6CCD7;"> name</span><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span><'</span><span style="color: #E5C07B;">a</span><span>></span><span style="color: #E5C07B;"> Drop</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> Timer</span><span><'</span><span style="color: #E5C07B;">a</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> drop</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> console</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">time_end_with_label</span><span>(</span><span style="color: #E06C75;">self.</span><span>name);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>By that the timer ticking will show outputs in the js console.</p>
<p>(TODO: Does it brings more info in the recorded frames(waterfalls)?)</p>
<h3 id="enhancing-performance">Enhancing Performance</h3>
<p>With the timer above, we can inspect the time cost of each function calls.</p>
<p>For some reason my bottleneck is different from the project guide, but I'll first enhance the render part to keep the same with the guide anyway, which is to skip <code>fillRect</code> when it's not necessary.</p>
<p>The bottleneck of my implementation is <code>Universe::tick</code>. Let's add more timer in between:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> tick</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self</span><span>,</span><span style="color: #C6CCD7;"> times</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> next</span><span style="color: #E06C75;"> =</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> _timer</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Timer</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">allocate next cells</span><span>");</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>cells</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">clone</span><span>()</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> _</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..</span><span style="color: #C6CCD7;">times</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> _timer</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Timer</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">new_genaration</span><span>");</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> row</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..self.</span><span>height {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..self.</span><span>width {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> idx</span><span style="color: #E06C75;"> = self.</span><span style="color: #B57EDC;">get_index</span><span>(</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #C6CCD7;"> col</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> cell</span><span style="color: #E06C75;"> = self.</span><span>cells[</span><span style="color: #C6CCD7;">idx</span><span>];</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> live_neigbors</span><span style="color: #E06C75;"> = self.</span><span style="color: #B57EDC;">live_neighbor_count</span><span>(</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #C6CCD7;"> col</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> next_cell</span><span style="color: #E06C75;"> = match</span><span> (</span><span style="color: #C6CCD7;">cell</span><span>,</span><span style="color: #C6CCD7;"> live_neigbors</span><span>) {</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">true</span><span>,</span><span style="color: #C6CCD7;"> x</span><span>)</span><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> <</span><span style="color: #56B6C2;"> 2</span><span style="color: #E06C75;"> =></span><span style="color: #56B6C2;"> false</span><span>,</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">true</span><span>,</span><span style="color: #56B6C2;"> 2</span><span>)</span><span style="color: #E06C75;"> |</span><span> (</span><span style="color: #56B6C2;">true</span><span>,</span><span style="color: #56B6C2;"> 3</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #56B6C2;"> true</span><span>,</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">true</span><span>,</span><span style="color: #C6CCD7;"> x</span><span>)</span><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> ></span><span style="color: #56B6C2;"> 3</span><span style="color: #E06C75;"> =></span><span style="color: #56B6C2;"> false</span><span>,</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">false</span><span>,</span><span style="color: #56B6C2;"> 3</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #56B6C2;"> true</span><span>,</span></span>
<span class="giallo-l"><span> (</span><span style="color: #C6CCD7;">otherwise</span><span>,</span><span style="color: #C6CCD7;"> _</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #C6CCD7;"> otherwise</span><span>,</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> next</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set</span><span>(</span><span style="color: #C6CCD7;">idx</span><span>,</span><span style="color: #C6CCD7;"> next_cell</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> _timer</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Timer</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">free old cells</span><span>");</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>cells </span><span style="color: #E06C75;">=</span><span style="color: #C6CCD7;"> next</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>TODO: for some reason I don't see the timer with these labels in the dev tools.</p>
<p>Any way, according to the flamegraph, the most time consuming function call is <code>Universe::live_neighbor_count</code> (check out the orange bar in the pic below).</p>
<p><img src="https://kev1n8.github.io/posts/rust-wasm-building-conway-s-game-of-life/./Task%20Archive/images/wasm%20count%20neighbor%20flamegraph.png" alt="Flamegraph showing the time-consuming function call in Universe::live_neighbor_count" /></p>
<h2 id="shrinking-binary-size">Shrinking Binary Size</h2>
<p>Use <code>wasm-opt</code> tool to shrink code size:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;"># Optimize for size.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">wasm-opt</span><span style="color: #56B6C2;"> -Os -o</span><span style="color: #98C379;"> output.wasm input.wasm</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"># Optimize aggressively for size.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">wasm-opt</span><span style="color: #56B6C2;"> -Oz -o</span><span style="color: #98C379;"> output.wasm input.wasm</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"># Optimize for speed.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">wasm-opt</span><span style="color: #56B6C2;"> -O -o</span><span style="color: #98C379;"> output.wasm input.wasm</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"># Optimize aggressively for speed.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">wasm-opt</span><span style="color: #56B6C2;"> -O3 -o</span><span style="color: #98C379;"> output.wasm input.wasm</span></span></code></pre>
<p>For further optimizations, we could turn our <code>Universe</code> into a static global instance, and so is its cells. Without any allocator, we could even make our lib <code>#![no_std]</code></p>
<p>I'll leave it here, though.</p>
<h2 id="publishing">Publishing</h2>
<p>Publish to npm:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #B57EDC;">wasm-pack</span><span style="color: #98C379;"> login</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">wasm-pack</span><span style="color: #98C379;"> build</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">wasm-pack</span><span style="color: #98C379;"> publish</span></span></code></pre>
Understanding Async Rust: The Future Trait
2025-04-01T00:00:00+00:00
2025-04-01T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/understanding-async-rust-the-future-trait/
<h1 id="understanding-async-rust-the-future-trait">Understanding Async Rust: The Future Trait</h1>
<p>This blog is intended to summarize the key ideas when it comes to async Rust, hence it's not going to deep in every topic. Please refer to more professional papers.</p>
<h2 id="general-picture-of-async-and-await">General Picture of <code>async</code> and <code>await</code></h2>
<p>It's obvious that the biggest difference between sync and async code is that the latter one has these keywords used in function signatures or bodies. However, one may wonder what is that supposed to mean.</p>
<p><code>async</code> is used in front of a <code>fn</code> or a closure. It literally means that the <code>fn</code> or the closure is returning a <code>Future</code>. Meanwhile, <code>await</code> is "called" on a <code>Future</code>. To simply say, it tells the compiler to generate necessary codes for state changing.</p>
<p>So the big picture is, we treat async functions (or async closures) as a mission to be completed for in the future. And it's not going to activate/be invoked unless we call <code>await</code> on that <code>Future</code>. While waiting for the result of that <code>Future</code>, the control is yielded to other tasks that might be ready and waiting to be executed. The "pause" and "resume" of tasks mechanism should thanks to a specific async task runtime.</p>
<p>There are several great tools when it comes to async runtime for Rust, e.g. tokio, async-std, smol. Since tokio is the most used one, we'll focus on it.</p>
<p>A runtime is responsible for the reconsiling of tasks. In tokio, there are several workers, each runs on a thread, polling the futures (we'll explain "polling" later. you may think it's like processing that task). Each worker has a local queue whose items are tasks waiting to be polled. And there also is a global queue first. The task is stealing-based in tokio, which means if a worker has empty queue, it'll try to steal task from other workers' queue if available.</p>
<p>A simple example of usage:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> run_async_task</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> i32</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #56B6C2;"> 0</span><span>;</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[tokio</span><span style="color: #E06C75;">::</span><span>main]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> fut</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> run_async_task</span><span>();</span><span style="color: #5F6672;font-style: italic;"> // Return a future, not run.</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> zero</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> fut</span><span style="color: #E06C75;">.await</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // Here, the control yields and the runtime will start `run_async_task()`</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h2 id="about-future">About <code>Future</code></h2>
<p><code>Future</code> is a trait that indicates its result might be ready in the future. It's similar to promise in JavaScript. Let's look at its definition:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> trait</span><span style="color: #E5C07B;"> Future</span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// The type of value produced on completion.</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> type</span><span style="color: #E5C07B;"> Output</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> poll</span><span>(</span><span style="color: #E06C75;">self:</span><span style="color: #E5C07B;"> Pin</span><span><</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> Self</span><span>>,</span><span style="color: #C6CCD7;"> cx</span><span style="color: #E06C75;">: &</span><span style="color: #61AFEF;">mut</span><span style="color: #E5C07B;"> Context</span><span><'</span><span style="color: #E5C07B;">_</span><span>>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Poll</span><span><</span><span style="color: #E06C75;">Self::</span><span style="color: #E5C07B;">Output</span><span>>;</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h2 id="example">Example</h2>
<p>By the following example, you'll get how futrues work:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> fn_1</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> i32</span><span> {</span></span>
<span class="giallo-l"><span style="color: #56B6C2;"> 1</span><span style="color: #E5C07B;">i32</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> add_2</span><span>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> i32</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> i32</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 2</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> run_add</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> fn_1</span><span>()</span><span style="color: #E06C75;">.await</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> add_2</span><span>(</span><span style="color: #C6CCD7;">a</span><span>)</span><span style="color: #E06C75;">.await</span><span>;</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("{</span><span style="color: #98C379;">b</span><span>}");</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span>#[tokio</span><span style="color: #E06C75;">::</span><span>main]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> run_add</span><span>()</span><span style="color: #E06C75;">.await</span><span>;</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Simplified pseudocode illustrating the state machine for run_add()</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">future</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Future</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">pin</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Pin</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">task</span><span style="color: #E06C75;">::</span><span>{</span><span style="color: #E5C07B;">Context</span><span>,</span><span style="color: #E5C07B;"> Poll</span><span>,</span><span style="color: #E5C07B;"> Waker</span><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Let's assume these async functions exist:</span></span>
<span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> fn_1</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> i32</span><span> {</span></span>
<span class="giallo-l"><span style="color: #56B6C2;"> 1</span><span style="color: #5F6672;font-style: italic;"> // Returns immediately for simplicity.</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> add_2</span><span>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> i32</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> i32</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 2</span><span style="color: #5F6672;font-style: italic;"> // Also returns immediately for simplicity.</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// The state machine states for run_add</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> RunAddState</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Start</span><span>,</span><span style="color: #5F6672;font-style: italic;"> // Before any await.</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> WaitingFn1</span><span> {</span><span style="color: #5F6672;font-style: italic;"> // Waiting for fn_1 to complete.</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> future</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Pin</span><span><</span><span style="color: #E5C07B;">Box</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> Future</span><span><</span><span style="color: #E5C07B;">Output</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> i32</span><span>>>>,</span></span>
<span class="giallo-l"><span> },</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> WaitingAdd2</span><span> {</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> i32</span><span>,</span><span style="color: #5F6672;font-style: italic;"> // Holds the result from fn_1.</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> future</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Pin</span><span><</span><span style="color: #E5C07B;">Box</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> Future</span><span><</span><span style="color: #E5C07B;">Output</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> i32</span><span>>>>,</span></span>
<span class="giallo-l"><span> },</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Finished</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> RunAdd</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> state</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> RunAddState</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Future</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> RunAdd</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> type</span><span style="color: #E5C07B;"> Output</span><span style="color: #E06C75;"> =</span><span> ();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> poll</span><span>(</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self:</span><span style="color: #E5C07B;"> Pin</span><span><</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> Self</span><span>>,</span><span style="color: #C6CCD7;"> cx</span><span style="color: #E06C75;">: &</span><span style="color: #61AFEF;">mut</span><span style="color: #E5C07B;"> Context</span><span><'</span><span style="color: #E5C07B;">_</span><span>>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Poll</span><span><()> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> loop</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> match &</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self.</span><span>state {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> RunAddState</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Start</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // We initiate fn_1() here.</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> future_fn1</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Box</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">pin</span><span>(</span><span style="color: #B57EDC;">fn_1</span><span>());</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Transition state to WaitingFn1 with the future.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>state </span><span style="color: #E06C75;">=</span><span style="color: #E5C07B;"> RunAddState</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">WaitingFn1</span><span> {</span><span style="color: #C6CCD7;"> future</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> future_fn1</span><span> };</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> RunAddState</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">WaitingFn1</span><span> {</span><span style="color: #C6CCD7;"> future</span><span> }</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Poll the inner future from fn_1.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> match</span><span style="color: #C6CCD7;"> future</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_mut</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">poll</span><span>(</span><span style="color: #C6CCD7;">cx</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Poll</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Ready</span><span>(</span><span style="color: #C6CCD7;">a</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // fn_1() is done, we got the result 'a'.</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Now prepare to call add_2(a).</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> future_add2</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Box</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">pin</span><span>(</span><span style="color: #B57EDC;">add_2</span><span>(</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>state </span><span style="color: #E06C75;">=</span><span style="color: #E5C07B;"> RunAddState</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">WaitingAdd2</span><span> {</span><span style="color: #C6CCD7;"> a</span><span>,</span><span style="color: #C6CCD7;"> future</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> future_add2</span><span> };</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Poll</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Pending</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // fn_1() isn't finished yet.</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Yield control back to the executor.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #E5C07B;"> Poll</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Pending</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> RunAddState</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">WaitingAdd2</span><span> {</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> _</span><span>,</span><span style="color: #C6CCD7;"> future</span><span> }</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Poll the inner future from add_2.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> match</span><span style="color: #C6CCD7;"> future</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_mut</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">poll</span><span>(</span><span style="color: #C6CCD7;">cx</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Poll</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Ready</span><span>(</span><span style="color: #C6CCD7;">b</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // add_2 has finished. We can now use the result.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Result: </span><span>{}",</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>state </span><span style="color: #E06C75;">=</span><span style="color: #E5C07B;"> RunAddState</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Finished</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #E5C07B;"> Poll</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Ready</span><span>(());</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Poll</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Pending</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Still waiting for add_2—for now, yield control.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #E5C07B;"> Poll</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Pending</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> RunAddState</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Finished</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> panic!</span><span>("</span><span style="color: #98C379;">Polling after completion</span><span>");</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[tokio</span><span style="color: #E06C75;">::</span><span>main]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">async fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> RunAdd</span><span> {</span><span style="color: #C6CCD7;"> state</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> RunAddState</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Start</span><span> }</span><span style="color: #E06C75;">.await</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // Runtime polls here.</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>Then, the async runtime will start "epolling" (kinda) the futures, until it returns <code>Finished</code>.</p>
<p>ref. https://wiki.cont.run/lowering-async-await-in-rust/</p>
Understanding Rust Macros: Compilation & Declarative Patterns
2025-04-01T00:00:00+00:00
2025-04-01T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/understanding-rust-macros-compilation-declarative-patterns/
<h1 id="understanding-rust-macros-compilation-declarative-patterns">Understanding Rust Macros: Compilation & Declarative Patterns</h1>
<p>Before deciding which type of macro to write, ask yourself the questions below:</p>
<ul>
<li>Is this a basic find/replace (e.g. implementing the same trait for different integer types or tuple lengths) or do I need some sort of conditional logic?</li>
<li>Do I need to track more "state" than a can be achieved with a simple pushdown accumulator?</li>
<li>Is this purely a syntactic operation or do I need to inspect a token's value and do logic with it?</li>
</ul>
<h2 id="compiling-trip">Compiling Trip</h2>
<p>Rust compilation goes through stages below:</p>
<ol>
<li>Tokenization -- keywords</li>
<li>Parsing -- where AST constructed</li>
<li><strong>Macro Expansion</strong></li>
<li>Name Resolution and Type Checking</li>
<li>Intermediate Representation</li>
<li>Optimization</li>
<li>Code Generation -- to LLVM IR</li>
<li>LLVM Optimization</li>
<li>Machine Code Generation</li>
<li>Linking</li>
</ol>
<p><strong>Tokens tree is distinct from AST</strong></p>
<h3 id="macro-in-ast">Macro in AST</h3>
<ol>
<li><code># [ $arg ];</code> e.g. #[derive(Clone)], #[no_mangle], …</li>
<li><code># ! [ $arg ];</code> e.g. #![allow(dead_code)], #![crate_name="blang"], …</li>
<li><code>$name ! $arg;</code> e.g. println!("Hi!"), concat!("a", "b"), …</li>
<li><code>$name ! $arg0 $arg1;</code> e.g. macro_rules! dummy { () => {}; }.</li>
</ol>
<h3 id="expansion">Expansion</h3>
<p>Intuitively, this is done by "expanding" the AST on its <em>syntax extensions</em> invocations <strong>recursively</strong>, and level-up no more than 128(default).</p>
<h3 id="hygiene">Hygiene</h3>
<p>Simply saying, in Rust, macro can't <code>use</code> the identifier ouside or <code>create</code> an identifier then <code>use</code> outside of it.</p>
<h2 id="declarative-macro">Declarative Macro</h2>
<p><code>macro_rules!</code> checks the rules one by one, perform expansion once the <code>matcher</code> matches.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #B57EDC;">macro_rules!</span><span style="color: #E06C75;"> $</span><span style="color: #C6CCD7;">name</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> $</span><span style="color: #C6CCD7;">rule0</span><span> ;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> $</span><span style="color: #C6CCD7;">rule1</span><span> ;</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // …</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> $</span><span style="color: #C6CCD7;">ruleN</span><span> ;</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #E06C75;">$</span><span style="color: #C6CCD7;">rule</span><span style="color: #E06C75;">:</span><span> (</span><span style="color: #E06C75;">$</span><span style="color: #C6CCD7;">matcher</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span><span style="color: #E06C75;">$</span><span style="color: #C6CCD7;">expansion</span><span>}</span></span></code></pre>
<p><code>macro_rules!</code>'s invocation does not expand in AST, the macro is registered internally in the compiler.</p>
<p>Repetition example</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">$</span><span> (</span><span style="color: #E06C75;"> ...</span><span> )</span><span style="color: #C6CCD7;"> sep rep</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;">macro_rules! sample</span><span> {</span></span>
<span class="giallo-l"><span> (</span><span style="color: #E06C75;">$</span><span>(</span><span style="color: #E06C75;">$</span><span style="color: #C6CCD7;">e</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;">ident</span><span>),</span><span style="color: #E06C75;">*</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> $</span><span>(</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // operation</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;">*</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p><code>sep</code> options: <code>,</code>, <code>;</code> or space by default
<code>rep</code> options: <code>*</code>, <code>+</code>, or <code>?</code> which can't be used with <code>sep</code> since at most 1</p>
<h3 id="minutiae">Minutiae</h3>
<p>The following labels are that we can capture from the input:</p>
<ul>
<li>block -- capture a code block like <code>{...}</code></li>
<li>expr -- a lot</li>
<li>ident -- identifiers and keywords</li>
<li>item -- definitions like <code>struct Foo</code> or <code>pub use crate::mod</code></li>
<li>lifetime -- similar to ident but start with <code>'</code></li>
<li>literal -- immediate values</li>
<li>meta -- matches attributes</li>
<li>pat -- any kind of pattern</li>
<li>pat_param -- any pattern but no <code>|</code></li>
<li>path -- a path like <code>crate::mod</code></li>
<li>stmt -- a statement</li>
<li>tt -- a token tree</li>
<li>ty -- a type expression</li>
<li>vis -- visibility like <code>pub(crate)</code></li>
</ul>
<h3 id="scoping">Scoping</h3>
<p><code>#[macro_use]</code>: <code>use</code> all the macro inside the first following mod, applicable for the whole "level".
<code>#[macro_export]</code>: export the macro to outside (of <code>crate</code>), other users may <code>use mod::some_macro;</code>(only applicable for external crate).</p>
<h3 id="todo">TODO</h3>
<p>There are more sections to learn. Maybe learn on practice.</p>
<h2 id="procedural-macro">Procedural Macro</h2>
<h3 id="types">Types</h3>
<ul>
<li><em>function-like</em> proc-macros -- <code>$name ! $arg</code></li>
<li><em>atribute</em> proc-macros -- <code>#[$arg]</code></li>
<li><em>derive</em> proc-macros -- <code>#[derive(...)]</code></li>
</ul>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[proc_macro]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> my_proc_macro</span><span>(</span><span style="color: #C6CCD7;">input</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> TokenStream</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> TokenStream</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> TokenStream</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>()</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[proc_macro_attribute]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> my_attribute</span><span>(</span><span style="color: #C6CCD7;">input</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> TokenStream</span><span>,</span><span style="color: #C6CCD7;"> annotated_item</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> TokenStream</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> TokenStream</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> TokenStream</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>()</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[proc_macro_derive]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> my_derive</span><span>(</span><span style="color: #C6CCD7;">annotatated_item</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> TokenStream</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> TokenStream</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> TokenStream</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>()</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="useful-third-party-tools">Useful Third-party tools</h3>
<ul>
<li>proc-macro2</li>
<li>quote</li>
<li>syn</li>
</ul>
Subtyping & Variance in Rust Lifetime
2024-12-16T00:00:00+00:00
2024-12-16T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/subtyping-variance-in-rust-lifetime/
<h1 id="rustsheng-ming-zhou-qi-subtyping-variance">Rust生命周期 Subtyping & Variance</h1>
<p>Rust 使用生命周期来跟踪引用和所有权，而生命周期检查实现的背后是 <strong>subtyping</strong>（子类型） 和 <strong>variance</strong>（变型）。</p>
<p>Subtyping 和 variance 本是继承中使用的概念，但是 Rust 中并没有继承机制，Rust 中这两概念用在了生命周期检查上。</p>
<h2 id="subtyping">Subtyping</h2>
<p>先看一个例子：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Note: debug expects two parameters with the *same* lifetime</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug</span><span><'</span><span style="color: #E5C07B;">a</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a = </span><span>{</span><span style="color: #98C379;">a:?</span><span>}</span><span style="color: #98C379;"> b = </span><span>{</span><span style="color: #98C379;">b:?</span><span>}");</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span><span style="color: #5F6672;font-style: italic;"> // 'static</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> hello</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">static str</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">hello</span><span>";</span></span>
<span class="giallo-l"><span> {</span><span style="color: #5F6672;font-style: italic;"> // 'world</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> world</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">world</span><span>");</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> world</span><span style="color: #E06C75;"> = &</span><span style="color: #C6CCD7;">world</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // 'world has a shorter lifetime than 'static</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug</span><span>(</span><span style="color: #C6CCD7;">hello</span><span>,</span><span style="color: #C6CCD7;"> world</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>一个<strong>保守</strong>的生命周期实现不会允许这样的行为，因为 <code>'static</code> 并不等于 <code>'world</code>。但是直觉告诉我们，这段代码这样做并没有安全问题。尝试编译这段代码，发现确实是可以正常通过检查的。这样灵活的约束得益于 <strong>subtyping</strong>。</p>
<p>首先我们要知道：</p>
<ul>
<li>一个生命周期定义了代码的一个区域</li>
<li><code>'long <: 'short</code> 表示 <code>'long</code> 是 <code>'short</code> 的 subtype；当且仅当 <code>'long</code> 定义的代码区域完全包括了 <code>'short</code> 定义的区域</li>
</ul>
<p>回到上面的例子，我们可以知道 <code>'static <: 'world</code>。</p>
<p>同时，<code>debug(&'static str, &'world world)</code> 合法，因此我们说 <code>&'static str</code> 是 <code>&'a str</code> 的一个 subtype</p>
<p>可以认为在这里 <code>&'static str</code> 被“降级”为 <code>&'a str</code>，从而匹配了 <code>debug</code> 的函数签名。</p>
<p>也就是说，尽管生命周期不完全匹配，只要参数是函数签名要求的subtype，这个行为就是允许的。</p>
<h2 id="variance">Variance</h2>
<p>Variance 描述了 subtype 属性在不同情况下是否可以保持下去的性质。</p>
<p>给定类型 <code>F<T></code>，其中 <code>T</code> 为 <code>Sub</code> or <code>Super</code>， variance 定义了三种关系：</p>
<ul>
<li>如果 <code>F<Sub></code> 是 <code>F<Super></code> 的 subtype，那么我们说 <code>F</code> 是 co-variant （协变）的；就好像 subtype 属性传递下去了</li>
<li>如果 <code>F<Super></code> 是 <code>F<Sub></code> 的 subtype，那么我们说 <code>F</code> 是 contra-variant （逆变）的；就好像 subtype 属性扭转了；这种情况在实际中非常少出现</li>
<li>其余情况，<code>F</code> 是 in-variant （不变）的；不能应用 subtype 属性</li>
</ul>
<p>各类型和它们的 variances 如下表：</p>
<table><thead><tr><th></th><th style="text-align: center">'a</th><th style="text-align: center">T</th><th style="text-align: center">U</th></tr></thead><tbody>
<tr><td><code>&'a T </code></td><td style="text-align: center">covariant</td><td style="text-align: center">covariant</td><td style="text-align: center"></td></tr>
<tr><td><code>&'a mut T</code></td><td style="text-align: center">covariant</td><td style="text-align: center">invariant</td><td style="text-align: center"></td></tr>
<tr><td><code>Box<T></code></td><td style="text-align: center"></td><td style="text-align: center">covariant</td><td style="text-align: center"></td></tr>
<tr><td><code>Vec<T></code></td><td style="text-align: center"></td><td style="text-align: center">covariant</td><td style="text-align: center"></td></tr>
<tr><td><code>UnsafeCell<T></code></td><td style="text-align: center"></td><td style="text-align: center">invariant</td><td style="text-align: center"></td></tr>
<tr><td><code>Cell<T></code></td><td style="text-align: center"></td><td style="text-align: center">invariant</td><td style="text-align: center"></td></tr>
<tr><td><code>fn(T) -> U</code></td><td style="text-align: center"></td><td style="text-align: center"><strong>contra</strong>variant</td><td style="text-align: center">covariant</td></tr>
<tr><td><code>*const T</code></td><td style="text-align: center"></td><td style="text-align: center">covariant</td><td style="text-align: center"></td></tr>
<tr><td><code>*mut T</code></td><td style="text-align: center"></td><td style="text-align: center">invariant</td><td style="text-align: center"></td></tr>
</tbody></table>
<p>简单来说：</p>
<ul>
<li><code>Vec<T></code> 和其他指针和集合同 <code>Box<T></code></li>
<li><code>Cell<T></code> 和其他具有内部可变性的同 <code>UnsafeCell<T></code></li>
<li><code>UnsafeCell<T></code> 因为它的内部可变性因此同 <code>&mut T</code></li>
<li><code>*const T</code> 同 <code>&T</code></li>
<li><code>*mut T</code> 同 <code>&mut T</code></li>
</ul>
<p>下面结合例子来理解这三种关系。</p>
<h2 id="covariance">Covariance</h2>
<h3 id="a-t-over-a-is-covariant"><code>&'a T</code> over <code>'a</code> is covariant</h3>
<p>If <code>'a <: 'b</code>, then <code>&'a T <: &'b T</code>.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Debug</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> T</span><span>,</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> T</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a = </span><span>{</span><span style="color: #98C379;">a:?</span><span>}</span><span style="color: #98C379;"> b = </span><span>{</span><span style="color: #98C379;">b:?</span><span>}");</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">static</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">hello</span><span>";</span></span>
<span class="giallo-l"><span> {</span><span style="color: #5F6672;font-style: italic;"> // 'b</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> string</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">world</span><span>");</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span style="color: #E06C75;"> = &</span><span style="color: #C6CCD7;">string</span><span>;</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug</span><span>(</span><span style="color: #C6CCD7;">a</span><span>,</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p><code>debug</code> 的参数必须是相同类型 <code>T</code>，在这里类型推断为 <code>&'a str</code>，完整函数签名：</p>
<p><code>fn debug<'a>(a: &'a str, b: &'a str)</code></p>
<p><code>main</code> 中有 <code>a: &'static str</code> 和 <code>b: &'b str</code>，且 <code>'static <: 'b</code>。</p>
<p>因为 <code>&'a T</code> 对 <code>'a</code> 是 covariant 的，所以 <code>&'static str</code> 在这里可以“降级”为 <code>&'b str</code>，被 <code>debug</code> 接受。</p>
<h3 id="a-t-over-t-is-covariant"><code>&'a T</code> over <code>T</code> is covariant</h3>
<p>If <code>t1 <: t2</code>, then <code>&'a t1 <: &'b t2</code>.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug</span><span><'</span><span style="color: #E5C07B;">a</span><span>,</span><span style="color: #E5C07B;"> T</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Debug</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a T</span><span>,</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> T</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a = </span><span>{</span><span style="color: #98C379;">a:?</span><span>}</span><span style="color: #98C379;"> b = </span><span>{</span><span style="color: #98C379;">b:?</span><span>}");</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">static str</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">hello</span><span>";</span></span>
<span class="giallo-l"><span> {</span><span style="color: #5F6672;font-style: italic;"> // 'b</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> string</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">world</span><span>");</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span style="color: #E06C75;"> = &</span><span style="color: #C6CCD7;">string</span><span>;</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>,</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>这次参数变成了 <code>(a: &'a T, b: T)</code>。</p>
<p>我们传入的参数为 <code>(&'b &'static str, &'b str)</code></p>
<p>类型推导过程中，</p>
<p>当 <code>T = &'static str'</code>，编译器判定不能满足生命周期约束，因为 <code>&'b str <: &'static</code> 不成立；</p>
<p>当 <code>T = &'b str</code>，根据规则， <code>'static <: 'b</code> => <code>&'static str <: &'b str</code>。
从而可以将 <code>a: &'a &'static str</code> “降级”为 <code>&'a &'b str</code>，成功匹配了函数签名。</p>
<h3 id="a-mut-t-over-a-is-covariant"><code>&'a mut T</code> over <code>'a</code> is covariant</h3>
<p>If <code>'a <: 'b</code>, then <code>&'a mut T <: &'b mut T</code>.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug</span><span><'</span><span style="color: #E5C07B;">a</span><span>,</span><span style="color: #E5C07B;"> T</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Debug</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a</span><span style="color: #61AFEF;"> mut</span><span style="color: #E5C07B;"> T</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a=</span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">static str</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">hello</span><span>";</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #C6CCD7;"> a</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>参数：<code>(&'static mut &'static str)</code></p>
<p>根据规则： <code>'static <: 'a</code> => <code>&'static mut &'static str <: &'a mut &'static str</code></p>
<p>类型推断 <code>T = &'static str</code>，匹配成功。</p>
<h2 id="invariance">Invariance</h2>
<h3 id="a-mut-t-over-t-is-invariant"><code>&'a mut T'</code> over <code>T</code> is invariant</h3>
<p>Even if <code>t1 <: t2</code>, there's no <code>&'a mut t1 <: &'a mut t2</code>.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug</span><span><'</span><span style="color: #E5C07B;">a</span><span>,</span><span style="color: #E5C07B;"> T</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Debug</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a</span><span style="color: #61AFEF;"> mut</span><span style="color: #E5C07B;"> T</span><span>,</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a T</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a=</span><span>{</span><span style="color: #98C379;">:?</span><span>}</span><span style="color: #98C379;">, b=</span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>,</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">static str</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">hello</span><span>";</span></span>
<span class="giallo-l"><span> {</span><span style="color: #5F6672;font-style: italic;"> // 'b</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> string</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">world</span><span>");</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> string</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_str</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #C6CCD7;"> a</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">b</span><span>);</span><span style="color: #5F6672;font-style: italic;"> // won't compile</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>参数：<code>(&'b mut &'static str, &'b '&'b str)</code></p>
<p>尽管有 <code>&'static str <: &'b str</code>（根据 <code>&'a T</code> over <code>'a</code> is covariant）</p>
<p>我们不能得出 <code>&'b mut &'static str <: &'b mut &'b str</code></p>
<p>因此 <code>T</code> 必须是 <code>&'static str</code> （即不允许“降级”）</p>
<p>而 <code>&'b str <: &'static str</code> 不成立，所以两个参数的 <code>T</code> 无法统一，编译无法通过。</p>
<hr />
<p><strong>TIP</strong>：</p>
<p>下面的代码是可以正常运行的</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Debug</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">: &</span><span style="color: #61AFEF;">mut</span><span style="color: #E5C07B;"> T</span><span>,</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> T</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a=</span><span>{</span><span style="color: #98C379;">:?</span><span>}</span><span style="color: #98C379;">, b=</span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>,</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug_str</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Debug</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">T</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">hello</span><span>";</span><span style="color: #5F6672;font-style: italic;"> // &str, with an implicit lifetime.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug_str</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>);</span></span>
<span class="giallo-l"><span> {</span><span style="color: #5F6672;font-style: italic;"> // 'b</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> string</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">world</span><span>");</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> string</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_str</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #C6CCD7;"> a</span><span>,</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>参数： <code>(&'b mut &'b str, &'b str)</code></p>
<p>这是因为在没有手动指定生命周期的情况下，编译器会根据代码逻辑自动推断生命周期。</p>
<p><code>a</code> 的生命周期被“压缩”到了 <code>'b</code>，使得它能匹配 <code>debug</code> 签名同时不违反 invariance。</p>
<p>而下面的代码：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Debug</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">: &</span><span style="color: #61AFEF;">mut</span><span style="color: #E5C07B;"> T</span><span>,</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> T</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a=</span><span>{</span><span style="color: #98C379;">:?</span><span>}</span><span style="color: #98C379;">, b=</span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>,</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> debug_static</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Debug</span><span>>(</span><span style="color: #C6CCD7;">a</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">static T</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">hello</span><span>";</span><span style="color: #5F6672;font-style: italic;"> // &str, with no explicit lifetime.</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug_static</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>);</span><span style="color: #5F6672;font-style: italic;"> // 1.</span></span>
<span class="giallo-l"><span> {</span><span style="color: #5F6672;font-style: italic;"> // 'b</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> string</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">world</span><span>");</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> string</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_str</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> debug</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #C6CCD7;"> a</span><span>,</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a=</span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>);</span><span style="color: #5F6672;font-style: italic;"> // 2.</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // 1. sets lifetime of `a` to 'static</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // 2. needs lifetime of `a` at least longer than 'b</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>上面的代码是不能编译通过的，因为编译器推断出的生命周期</p>
<ol>
<li><code>'static</code> 以及</li>
<li><code>at least longer than 'b</code></li>
</ol>
<p><code>1.</code> 和 <code>2.</code> 任意存在都会导致违反 <code>&'a mut T</code> 对 <code>T</code> 的 invariance.</p>
<h2 id="contravariance">Contravariance</h2>
<p>最后，也是是唯一一个的 contra-variant。</p>
<p><code>fn(T) -> U</code> over <code>T</code> is contravariant.</p>
<p>If <code>t1 <: t2</code>, then <code>fn(t2) <: fn(t1)</code>.</p>
<p>直接看例子：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #B57EDC;">thread_local!</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #61AFEF;"> static</span><span style="color: #E5C07B;"> StaticVecs</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> RefCell</span><span><</span><span style="color: #E5C07B;">Vec</span><span><</span><span style="color: #E06C75;">&</span><span>'</span><span style="color: #E5C07B;">static str</span><span>>></span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> RefCell</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Vec</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>());</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">/// saves the input given into a thread local `Vec<&'static str>`</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> store</span><span>(</span><span style="color: #C6CCD7;">input</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">static str</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> StaticVecs</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">with_borrow_mut</span><span>(</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;">v</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #C6CCD7;">input</span><span>));</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">/// Calls the function with it's input (must have the same lifetime!)</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> demo</span><span><'</span><span style="color: #E5C07B;">a</span><span>>(</span><span style="color: #C6CCD7;">input</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span><span style="color: #C6CCD7;"> f</span><span style="color: #E06C75;">: fn</span><span>(</span><span style="color: #E06C75;">&</span><span>'</span><span style="color: #E5C07B;">a str</span><span>)) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> f</span><span>(</span><span style="color: #C6CCD7;">input</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> demo</span><span>("</span><span style="color: #98C379;">hello</span><span>",</span><span style="color: #C6CCD7;"> store</span><span>);</span><span style="color: #5F6672;font-style: italic;"> // "hello" is 'static. Can call `store` fine</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> smuggle</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">smuggle</span><span>");</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // `&smuggle` is not static. If we were to call `store` with `&smuggle`,</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // we would have pushed an invalid lifetime into the `StaticVecs`.</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Therefore, `fn(&'static str)` cannot be a subtype of `fn(&'a str)`</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> demo</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">smuggle</span><span>,</span><span style="color: #C6CCD7;"> store</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // use after free...</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> StaticVecs</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">with_borrow</span><span>(</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;">v</span><span style="color: #E06C75;">|</span><span style="color: #B57EDC;"> println!</span><span>("{</span><span style="color: #98C379;">v:?</span><span>}"));</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p><strong>References</strong></p>
<p>https://doc.rust-lang.org/nomicon/subtyping.html
https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)</p>
Extending DataFusion: Custom Query Plans & Optimizer Rules
2024-09-21T00:00:00+00:00
2024-09-21T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/extending-datafusion-custom-query-plans-optimizer-rules/
<h2 id="kuo-zhan-datafusionzhi-zi-ding-yi-ji-hua">扩展Datafusion之自定义计划</h2>
<p>上一篇扩展datafusion介绍了通过自定义的 <code>TableProvider</code>, <code>ExecutionPlan</code> 和 <code>DataSink</code> 来帮助我们达到下面的目的：</p>
<ol>
<li>使用上下文注册 <code>TableProvider</code> 定向 <code>scan</code> 和 <code>Insert_into</code> 执行时的调用的对象（<code>ExecutionPlan</code>）</li>
<li>自定义的 <code>ExecutionPlan</code> 执行从rocksdb读取表数据的操作</li>
<li>自定义的 <code>DataSink</code>，提供 <code>write_all</code> 函数供datafusion的 <code>DataSinkExec</code> 调用，后者在 <code>insert_into</code> 的时候被创建</li>
</ol>
<p>而本文也将从几个要自定义实现的 <code>trait</code> 开始介绍，分别是：</p>
<ul>
<li>
<p><code>QueryPlanner</code> -- 专门用于调用各 <code>Plannner</code> 生成 <code>LogicalPlan</code> 和 <code>ExecutionPlan</code>，本文用于注册自定义的 <code>ExtensionPlanner</code></p>
</li>
<li>
<p><code>OptimizerRule</code> -- 用于自定义一个优化规则，本文实现一个规则将datafusion自带的 <code>DML::DELETE</code> <code>PlanNode</code> 替换为自定义的 <code>DeletePlanNode</code></p>
</li>
<li>
<p><code>UserDefinedLogicalNodeCore</code> -- 用于实现自定 <code>LogicPlanNode</code></p>
</li>
<li>
<p><code>ExtensionPlanner</code> -- 用于根据输入 <code>LogicPlanNode</code> 创建一个 <code>ExecutionPlan</code></p>
</li>
<li>
<p><code>ExecuionPlan</code> -- 类似上一篇，只不过本文将实现 <code>Delete</code> 逻辑</p>
</li>
</ul>
<h2 id="queryplanner">QueryPlanner</h2>
<p>这是 <code>SessionState</code> 的一个成员，我们可以看看如果不为其添加任何扩展，它会如何初始化：</p>
<h3 id="chu-shi-hua-lu-jing-yuan-ma-yi-lan">初始化路径源码一览</h3>
<p>首先是出处，<code>query_planner</code> 作为成员，需要满足 <code>QueryPlanner</code> 动态分发：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> SessionState</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// Responsible for planning `LogicalPlan`s, and `ExecutionPlan`</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> query_planner</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> QueryPlanner</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Send</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Sync</span><span>>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>其次，我们看 <code>SessionStateBuilder</code>，这里省略了其它属性设置 ：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> SessionStateBuilder</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> build</span><span>(</span><span style="color: #E06C75;">self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> SessionState</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #E06C75;"> Self</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> query_planner</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> = self</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> state</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> SessionState</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // 如果没有设定好的，则使用DefaultQueryPlanner</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> query_planner</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> query_planner</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap_or</span><span>(</span><span style="color: #61AFEF;">Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">DefaultQueryPlanner</span><span> {})),</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> state</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>没有规定一定要通过Builder来设置 <code>SessionState</code>，重要的是我们知道了Datafusion提供了默认的 <code>QueryPlanner</code>。</p>
<p>接着，来看这个 <code>DefaultQueryPlanner</code>：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> DefaultQueryPlanner</span><span> {}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[async_trait]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> QueryPlanner</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> DefaultQueryPlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// Given a `LogicalPlan`, create an [`ExecutionPlan`] suitable for execution</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> create_physical_plan</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> logical_plan</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">LogicalPlan</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> session_state</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">SessionState</span><span>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #61AFEF;"> datafusion_common</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Result</span><span><</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> planner</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> DefaultPhysicalPlanner</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">default</span><span>();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> planner</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">create_physical_plan</span><span>(</span><span style="color: #C6CCD7;">logical_plan</span><span>,</span><span style="color: #C6CCD7;"> session_state</span><span>)</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .await</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>可见，datafusion又把初始化交给了 <code>DefaultPhysicalPlanner</code>：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[derive(</span><span style="color: #E5C07B;">Default</span><span>)]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> DefaultPhysicalPlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // 如果我们注册了扩展，则会保存在这里</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> extension_planners</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExtensionPlanner</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Send</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Sync</span><span>>>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[async_trait]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> PhysicalPlanner</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> DefaultPhysicalPlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// Create a physical plan from a logical plan</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> create_physical_plan</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> logical_plan</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">LogicalPlan</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> session_state</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">SessionState</span><span>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // 具体的创建plan流程</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> match self.</span><span style="color: #B57EDC;">handle_explain</span><span>(</span><span style="color: #C6CCD7;">logical_plan</span><span>,</span><span style="color: #C6CCD7;"> session_state</span><span>)</span><span style="color: #E06C75;">.await?</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #C6CCD7;">plan</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #C6CCD7;">plan</span><span>),</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> None</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> plan</span><span style="color: #E06C75;"> = self</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">create_initial_plan</span><span>(</span><span style="color: #C6CCD7;">logical_plan</span><span>,</span><span style="color: #C6CCD7;"> session_state</span><span>)</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .await?</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span style="color: #B57EDC;">optimize_physical_plan</span><span>(</span><span style="color: #C6CCD7;">plan</span><span>,</span><span style="color: #C6CCD7;"> session_state</span><span>,</span><span style="color: #E06C75;"> |</span><span style="color: #C6CCD7;">_</span><span>,</span><span style="color: #C6CCD7;"> _</span><span style="color: #E06C75;">|</span><span> {})</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>到这里我们就知道，<code>QueryPlanner</code> 起到一个根据 <code>LogicalPlan</code> 创建 <code>PhysicalPlan</code> (<code>ExecutionPlan</code>) 的作用，具体的计划创建过程（关于从 <code>LogicalPlanNode</code> 到 <code>ExecutionPlan</code> 可以参考后续 <code>Planner</code> 的定义）就不展开了。</p>
<h3 id="ru-he-zi-ding-yi">如何自定义</h3>
<p>至于自定义的 <code>QueryPlanner</code>， 就是简单添加一个自定义的 <code>Planner</code> 到 <code>DefaultPhysicalPlanner</code> 中去：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> DeleteQueryPlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> table_id</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u64</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span><</span><span style="color: #E5C07B;">DB</span><span>>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[async_trait]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> QueryPlanner</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> DeleteQueryPlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> create_physical_plan</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> logical_plan</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">LogicalPlan</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> session_state</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">SessionState</span><span>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // 这里并非使用默认，而是添加了一个自定义的DeletePlanner</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> planner</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> DefaultPhysicalPlanner</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">with_extension_planners</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span><span style="color: #61AFEF;">Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> DeletePlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> table_id</span><span style="color: #E06C75;">: self.</span><span>table_id,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&self.</span><span>db),</span></span>
<span class="giallo-l"><span> },</span></span>
<span class="giallo-l"><span> )]);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> planner</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">create_physical_plan</span><span>(</span><span style="color: #C6CCD7;">logical_plan</span><span>,</span><span style="color: #C6CCD7;"> session_state</span><span>)</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .await</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>这样一来，后续的sql操作在转换计划的时候，注册了这个 <code>QueryPlanner</code> 的 <code>SessionState</code> 就知道 <code>DeletePlanner</code>，那么在遇到 <code>DeleteLogicalPlanNode</code> 的时候也就知道如何生成相应的 <code>ExecutionPlan</code>，也就支持了 <code>Delete</code> 操作。</p>
<p>那么问题来了，怎么把Datafusion中原有的 <code>DML::Delete</code> 这个 <code>PlanNode</code> 替换成真正的，自定义的 <code>PlanNode</code> 呢？</p>
<h2 id="optimizerrule">OptimizerRule</h2>
<p>顾名思义，这个 <code>trait</code> 可以让我们自定义一个逻辑优化器规则。本文要做的，就是将datafusion解析 <code>DELETE</code> 操作产生的 <code>LogicPlan::DML(DELETE)</code> 给换成我们自己的实现，前者在执行阶段会返回错误：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> DefaultPhysicalPlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> map_logical_node_to_physical</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> node</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">LogicalPlan</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> session_state</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">SessionState</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> children</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> ChildrenContainer</span><span>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> exec_node</span><span style="color: #E06C75;"> = match</span><span style="color: #C6CCD7;"> node</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> LogicalPlan</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Extension</span><span>(</span><span style="color: #E5C07B;">Extension</span><span> {</span><span style="color: #C6CCD7;"> node</span><span> })</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> maybe_plan</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> None</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> children</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> children</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">vec</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> planner</span><span style="color: #E06C75;"> in &self.</span><span>extension_planners {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> maybe_plan</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">is_some</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> break</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> logical_input</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> node</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">inputs</span><span>();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> maybe_plan</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> planner</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">plan_extension</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> node</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_ref</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">logical_input</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">children</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> session_state</span><span>,</span></span>
<span class="giallo-l"><span> )</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .await?</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> plan</span><span style="color: #E06C75;"> = match</span><span style="color: #C6CCD7;"> maybe_plan</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #C6CCD7;">v</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #C6CCD7;">v</span><span>),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _</span><span style="color: #E06C75;"> =></span><span style="color: #B57EDC;"> plan_err!</span><span>("</span><span style="color: #98C379;">No installed planner was able to convert the custom node to an execution plan: </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> node</span><span>)</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;">?</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Ensure the ExecutionPlan's schema matches the</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // declared logical schema to catch and warn about</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // logic errors when creating user defined plans.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if !</span><span style="color: #C6CCD7;">node</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">schema</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">matches_arrow_schema</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">plan</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">schema</span><span>()) {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #B57EDC;"> plan_err!</span><span>(</span></span>
<span class="giallo-l"><span> "</span><span style="color: #98C379;">Extension planner for </span><span>{</span><span style="color: #98C379;">:?</span><span>}</span><span style="color: #98C379;"> created an ExecutionPlan with mismatched schema. \</span></span>
<span class="giallo-l"><span style="color: #98C379;"> LogicalPlan schema: </span><span>{</span><span style="color: #98C379;">:?</span><span>}</span><span style="color: #98C379;">, ExecutionPlan schema: </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> node</span><span>,</span><span style="color: #C6CCD7;"> node</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">schema</span><span>(),</span><span style="color: #C6CCD7;"> plan</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">schema</span><span>()</span></span>
<span class="giallo-l"><span> );</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> plan</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> LogicalPlan</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Dml</span><span>(</span><span style="color: #C6CCD7;">dml</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // DataFusion is a read-only query engine, but also a library, so consumers may implement this</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #B57EDC;"> not_impl_err!</span><span>("</span><span style="color: #98C379;">Unsupported logical plan: Dml(</span><span>{</span><span style="color: #98C379;">0</span><span>}</span><span style="color: #98C379;">)</span><span>",</span><span style="color: #C6CCD7;"> dml</span><span style="color: #E06C75;">.</span><span>op);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>从上面的代码可以看见，如果我们没有用我们自己的实现（也就是 <code>LogicalPlan::Extension</code>）替换 <code>LogicalPlan::Dml(dml)</code>，就会返回错误。而如果通过我们自定义的rule将这个计划结点换成，就可以执行我们自己的实现（对应 <code>DeleteLogicalPlanNode</code>）。</p>
<p>实现 <code>OptimizerRule</code> 关键是实现 <code>re_write</code> 方法，这个方法检查当前结点是否为目标结点（<code>LogicalPlan::Dml(delete)</code>），如果是的话就用自定义结点取代（<code>DeleteLogicalPlanNode</code>），代码如下：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> DeleteReplaceRule</span><span> {}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> OptimizerRule</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> DeleteReplaceRule</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> rewrite</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> plan</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> LogicalPlan</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _config</span><span style="color: #E06C75;">: &dyn</span><span style="color: #E5C07B;"> OptimizerConfig</span><span>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">Transformed</span><span><</span><span style="color: #E5C07B;">LogicalPlan</span><span>>,</span><span style="color: #E5C07B;"> DataFusionError</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #61AFEF;"> let</span><span style="color: #E5C07B;"> LogicalPlan</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Dml</span><span>(</span><span style="color: #E5C07B;">DmlStatement</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> table_name</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> op</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> WriteOp</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Delete</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> input</span><span>,</span><span style="color: #5F6672;font-style: italic;"> // table source</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> output_schema</span><span>,</span><span style="color: #5F6672;font-style: italic;"> // single count</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ..</span></span>
<span class="giallo-l"><span> })</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> plan</span></span>
<span class="giallo-l"><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // input should have only 1 item, and</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // it can only be `LogicalPlan::Filter`, or</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // `LogicalPlan::Scan`?</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> _name</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> table_name</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> _schema</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> input</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">schema</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> _inputt</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> input</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_ref</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #E5C07B;">Transformed</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">yes</span><span>(</span><span style="color: #61AFEF;">LogicalPlan</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Extension</span><span>(</span><span style="color: #E5C07B;">Extension</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> node</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">DeletePlanNode</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> input</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> input</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_ref</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">clone</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> output_schema</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> expr</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> input</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">expressions</span><span>(),</span></span>
<span class="giallo-l"><span> }),</span></span>
<span class="giallo-l"><span> })))</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #E5C07B;">Transformed</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">no</span><span>(</span><span style="color: #C6CCD7;">plan</span><span>))</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h2 id="userdefinedlogicalnodecore">UserDefinedLogicalNodeCore</h2>
<p>直接贴代码：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> DeletePlanNode</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> input</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> LogicalPlan</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> DFSchemaRef</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> expr</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">Expr</span><span>>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> UserDefinedLogicalNodeCore</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> DeletePlanNode</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> with_exprs_and_inputs</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> exprs</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">Expr</span><span>>,</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> mut</span><span style="color: #C6CCD7;"> inputs</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">LogicalPlan</span><span>>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E06C75;">Self</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">inputs</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>(),</span><span style="color: #56B6C2;"> 1</span><span>, "</span><span style="color: #98C379;">input size inconsistent</span><span>");</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #E06C75;">Self</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> input</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> inputs</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">swap_remove</span><span>(</span><span style="color: #56B6C2;">0</span><span>),</span><span style="color: #5F6672;font-style: italic;"> // 假定只有一个子节点</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&self.</span><span>schema),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> expr</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> exprs</span><span>,</span></span>
<span class="giallo-l"><span> })</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>实际运行时，这个函数会被调用很多次，但都没有任何改变。To be found...</p>
<h2 id="extensionplanner">ExtensionPlanner</h2>
<p>重头戏之一，我们在上面已经实现了 <code>QueryPlanner</code>, <code>OptimizerRule</code>, <code>PlanNode</code>，这三者结合已经能够做到下面的事情：</p>
<ol>
<li>为 <code>SessionState</code> 注册 <code>QueryPlanner</code> 和 <code>OptimizerRule</code></li>
<li>sql执行DML之 <code>DELETE</code> 操作的时候，生成的逻辑计划将是自定义的逻辑计划</li>
</ol>
<p>但是，<code>DefaultPhysicalPlanner</code> 并不知道我们自定义的逻辑计划要生成什么样的执行计划。所以，<code>ExtensionPlanner</code> 就要站出来发挥作用了。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> DeletePlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> table_id</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u64</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span><</span><span style="color: #E5C07B;">DB</span><span>>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[async_trait]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> ExtensionPlanner</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> DeletePlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> plan_extension</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _planner</span><span style="color: #E06C75;">: &dyn</span><span style="color: #E5C07B;"> PhysicalPlanner</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> node</span><span style="color: #E06C75;">: &dyn</span><span style="color: #E5C07B;"> UserDefinedLogicalNode</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> logical_inputs</span><span style="color: #E06C75;">: &</span><span>[</span><span style="color: #E06C75;">&</span><span style="color: #E5C07B;">LogicalPlan</span><span>],</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> physical_inputs</span><span style="color: #E06C75;">: &</span><span>[</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>],</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _session_state</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">SessionState</span><span>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">Option</span><span><</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> node</span><span style="color: #E06C75;"> =</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #61AFEF;"> let</span><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #C6CCD7;">delete_node</span><span>)</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> node</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_any</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">downcast_ref</span><span style="color: #E06C75;">::</span><span><</span><span style="color: #E5C07B;">DeletePlanNode</span><span>>() {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">logical_inputs</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>(),</span><span style="color: #56B6C2;"> 1</span><span>);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">physical_inputs</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>(),</span><span style="color: #56B6C2;"> 1</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> output_schema</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #E06C75;">&</span><span>(</span><span style="color: #E06C75;">*</span><span style="color: #C6CCD7;">delete_node</span><span style="color: #E06C75;">.</span><span>schema)));</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> input_schema</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">physical_inputs</span><span>[</span><span style="color: #56B6C2;">0</span><span>]</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">schema</span><span>());</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #E5C07B;">Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">DeleteExec</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>table_id,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self.</span><span>db,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">input_schema</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">output_schema</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">physical_inputs</span><span>[</span><span style="color: #56B6C2;">0</span><span>],</span></span>
<span class="giallo-l"><span> )))</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> None</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #C6CCD7;">node</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">map</span><span>(</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;">exec</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;"> exec</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>))</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>逻辑就是：查看当前逻辑结点是否为“我”负责的，如果是的话则生成对应执行计划，否则返回None，datafusion会自动寻找其它对应的 <code>Planner</code>。</p>
<h2 id="executionplan">ExecutionPlan</h2>
<p>终于回到我们的 <code>KVTable</code> 了，我们在这里实现真正的删除操作，目的是实现从表中（db中）删除对应的行，而这些行会通过 <code>input</code> 过滤进来。每个行有一个唯一标识，即 <code>row_id</code>，删除操作会遍历表的每个列，然后遍历每个要删除的 <code>row_id</code>，这样可以构造出目标 <code>key</code>， <code>t[tid]_c[name]_r[rid]</code>。</p>
<p>实现如下：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// This method takes in the batch that is supposed to be deleted, and</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// delete them from the table of self.table_id, and</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// return rows deleted</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> delete_batch_from_table</span><span>(</span><span style="color: #E06C75;">&self</span><span>,</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> RecordBatch</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">u64</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> target</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">columns</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">last</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span><span style="color: #5F6672;font-style: italic;"> // by design, the `row_id` is the last column</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Iter over the `target` array to get each `row_id` to delete</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> cnt</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span style="color: #E5C07B;">u64</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> target</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> as_string_array</span><span>(</span><span style="color: #C6CCD7;">target</span><span>)</span><span style="color: #E06C75;">?</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> field</span><span style="color: #E06C75;"> in self.</span><span>in_schema</span><span style="color: #E06C75;">.</span><span>fields</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">iter</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> field</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">name</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">eq</span><span>("</span><span style="color: #98C379;">row_id</span><span>") {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> continue</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // break</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> id</span><span style="color: #E06C75;"> in</span><span style="color: #C6CCD7;"> target</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> match</span><span style="color: #C6CCD7;"> id</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> None</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> todo!</span><span>()</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #C6CCD7;">id</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> cnt</span><span style="color: #E06C75;"> +=</span><span style="color: #56B6C2;"> 1</span><span>;</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // TODO: Change type of `row_id` from Utf8 to u64</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> key</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> make_row_key</span><span>(</span><span style="color: #E06C75;">self.</span><span>table_id,</span><span style="color: #C6CCD7;"> name</span><span>,</span><span style="color: #C6CCD7;"> id</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">parse</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>());</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>db</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">delete</span><span>(</span><span style="color: #C6CCD7;">key</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #C6CCD7;">cnt</span><span>)</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>至此，我们的 <code>KVTable</code> 就支持删除操作了。</p>
<h2 id="bonus-ce-shi">BONUS：测试</h2>
<p>同样的，贴一份基础测试，验证删除的有效性：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span> #[tokio</span><span style="color: #E06C75;">::</span><span>test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> basic_delete_test</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">DB</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">open_default</span><span>("</span><span style="color: #98C379;">./tmp</span><span>")</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>());</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Registering the custom planners and rule</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> config</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> SessionConfig</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">with_target_partitions</span><span>(</span><span style="color: #56B6C2;">1</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> runtime</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">RuntimeEnv</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">default</span><span>());</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> state</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> SessionStateBuilder</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>()</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">with_config</span><span>(</span><span style="color: #C6CCD7;">config</span><span>)</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">with_runtime_env</span><span>(</span><span style="color: #C6CCD7;">runtime</span><span>)</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">with_default_features</span><span>()</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">with_query_planner</span><span>(</span><span style="color: #61AFEF;">Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">DeleteQueryPlanner</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> table_id</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 1003</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">db</span><span>),</span></span>
<span class="giallo-l"><span> }))</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">with_optimizer_rule</span><span>(</span><span style="color: #61AFEF;">Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">DeleteReplaceRule</span><span> {}))</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">build</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> ctx</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> SessionContext</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new_with_state</span><span>(</span><span style="color: #C6CCD7;">state</span><span>);</span></span></code></pre>
<p>首先，我们创建一个 <code>SessionState</code> ，并注册自定义的 <code>Rule</code> 和 <code>Planner</code>，并根据这个State创建Datafusion的上下文。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create table for testing and register</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Fields</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">c1</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>)),</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">row_id</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>)),</span></span>
<span class="giallo-l"><span> ])));</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> table_meta</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">KVTableMeta</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> id</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 1003</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">:</span><span> "</span><span style="color: #98C379;">t</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">schema</span><span>),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> highest</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 0</span><span>,</span><span style="color: #5F6672;font-style: italic;"> // default, unknown</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> table_provider</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">KVTable</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">table_meta</span><span>,</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">db</span><span>)));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Since create table is not used here, we have to init the meta of table in db first</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> meta_key</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> table_meta</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">make_key</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> meta_val</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> table_meta</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">make_value</span><span>();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">put</span><span>(</span><span style="color: #C6CCD7;">meta_key</span><span>,</span><span style="color: #C6CCD7;"> meta_val</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ctx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">register_table</span><span>("</span><span style="color: #98C379;">t</span><span>",</span><span style="color: #C6CCD7;"> table_provider</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span></code></pre>
<p>接下来，创建表，并注册 <code>TableProvider</code> 到上下文。因为还没有实现 <code>CREATE</code> DDL，所以这里手动往 <code>db</code> 中添加表的元数据。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Now insert table with init data</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> inserted</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> ctx</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">sql</span><span>("</span><span style="color: #98C379;">INSERT INTO t VALUES('hello', '001'), ('world', '002'), ('!', '003')</span><span>")</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .await</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Should have inserted 3 row</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> bind</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> inserted</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">collect</span><span>()</span><span style="color: #E06C75;">.await.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> bind</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">first</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> t</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> as_uint64_array</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">columns</span><span>()[</span><span style="color: #56B6C2;">0</span><span>])</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">t</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">value</span><span>(</span><span style="color: #56B6C2;">0</span><span>),</span><span style="color: #56B6C2;"> 3</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Now try to delete from the table</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> deleted</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> ctx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">sql</span><span>("</span><span style="color: #98C379;">DELETE FROM t where c1='!'</span><span>")</span><span style="color: #E06C75;">.await.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> _plan</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> deleted</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">logical_plan</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> bind</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> deleted</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">collect</span><span>()</span><span style="color: #E06C75;">.await.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> bind</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">first</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> t</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> as_uint64_array</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">columns</span><span>()[</span><span style="color: #56B6C2;">0</span><span>])</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Should have deleted 1 row</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">t</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">value</span><span>(</span><span style="color: #56B6C2;">0</span><span>),</span><span style="color: #56B6C2;"> 1</span><span>);</span></span></code></pre>
<p>创建完表后，利用之前实现的 <code>insert_into</code> 插入一些数据（这里有一个隐藏的问题，因为 <code>row_id</code> 是显式地在表schema中的，插入时其实并不知道具体的 <code>row_id</code>，目前的方法是随意输入值填充，但是后期的话应该要考虑让这个列对用户来说是透明的），插入后尝试删除行。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Now scan the table and compare</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> expected</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> RecordBatch</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">try_new</span><span>(</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">schema</span><span>),</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">StringArray</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>["</span><span style="color: #98C379;">hello</span><span>", "</span><span style="color: #98C379;">world</span><span>"])),</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">StringArray</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>["</span><span style="color: #98C379;">000001</span><span>", "</span><span style="color: #98C379;">000002</span><span>"])),</span></span>
<span class="giallo-l"><span> ],</span></span>
<span class="giallo-l"><span> )</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> df</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> ctx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">sql</span><span>("</span><span style="color: #98C379;">SELECT c1 FROM t</span><span>")</span><span style="color: #E06C75;">.await.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> bind</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> df</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">collect</span><span>()</span><span style="color: #E06C75;">.await.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> res</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> pretty_format_batches</span><span>(</span><span style="color: #C6CCD7;">bind</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_slice</span><span>())</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("{}",</span><span style="color: #C6CCD7;"> res</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> out</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> bind</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">first</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">out</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">expected</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>删除成功后，执行 <code>scan</code> 操作，查看输出是否和预期相同。</p>
Extending DataFusion: Connecting an External KV Store
2024-09-13T00:00:00+00:00
2024-09-13T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/extending-datafusion-connecting-an-external-kv-store/
<h2 id="kuo-zhan-datafusionzhi-cong-wai-bu-kvcun-chu">扩展Datafusion之从外部KV存储</h2>
<p>作为library，Datafusion在各个层次都提供了Extension trait，实现这些trait可以扩展datafusion的功能，非常方便。本文主要介绍Datafusion提供的<code>TableProvider</code>, <code>ExecutionPlan</code> 和 <code>DataSink</code>三个<code>trait</code>。通过这几个<code>trait</code>我们可以实现让只能在内存(<code>datafusion::datasource::MemTable</code>)中的数据存储到任何我们想要的位置。</p>
<h2 id="tableprovider">TableProvider</h2>
<p><code>TableProvider</code>顾名思义，让一个结构具有提供表相关功能的能力，一系列方法中最核心的就是要实现<code>scan</code>方法（当然，还有<code>insert_into</code>方法）。</p>
<p>一般来说，要让自定义的数据源来<code>TableProvider</code>。在我的实现中，我把KV存储作为<code>TableProvider</code>，使用的是RocksDB，那么相关的结构体就是这样：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[derive(</span><span style="color: #E5C07B;">Debug</span><span>,</span><span style="color: #E5C07B;"> Clone</span><span>)]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> KVTable</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span><</span><span style="color: #E5C07B;">DB</span><span>>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> table_id</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u64</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> meta</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> KVTableMetaRef</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>我的实现比较简陋，只有一些必要的成员。接下来为了实现扫描功能，这个<code>TableProvider</code>需要有一个<code>scan</code>函数（和<code>insert_into</code>函数，如果想要支持写入到KV的话），返回<code>SendableStream</code>供后续异步回调数据。</p>
<p>但是实际在做的时候呢，并不会直接把执行细节放在数据源这里，而是通过构造一个<code>ExecutionPlan</code>来执行。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[async_trait]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> TableProvider</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> KVSource</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;">...</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> scan</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _state</span><span style="color: #E06C75;">: &dyn</span><span style="color: #E5C07B;"> Session</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> projection</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Option</span><span><</span><span style="color: #E06C75;">&</span><span style="color: #E5C07B;">Vec</span><span><</span><span style="color: #E5C07B;">usize</span><span>>>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _filters</span><span style="color: #E06C75;">: &</span><span>[</span><span style="color: #E5C07B;">Expr</span><span>],</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _limit</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Option</span><span><</span><span style="color: #E5C07B;">usize</span><span>></span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span style="color: #B57EDC;">create_scan_physical_plan</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>table_id,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> projection</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span style="color: #B57EDC;">schema</span><span>()</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;">.await</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;">...</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h2 id="executionplan">ExecutionPlan</h2>
<p>一个<code>ExecutionPlan</code>的职责，就是提供一个<code>execute</code>方法，能够返回<code>SendableStream</code>。而这个<code>Stream</code>会回调<code>ExecutionPlan</code>内部的一个方法，这个被回调的方法就是真正的功能实现的地方。以<code>DBTableScanExec</code>为例：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[derive(</span><span style="color: #E5C07B;">Debug</span><span>)]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> DBTableScanExec</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> id</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u64</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span><</span><span style="color: #E5C07B;">DB</span><span>>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> SchemaRef</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> cache</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> PlanProperties</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>我的初步实现按照以下的想法和限制：</p>
<ol>
<li>数据类型只支持String，便于统一<code>put</code>和<code>get</code></li>
<li>key的编码方式是 <code>t[table_id]_c[col_name]_r[row_id]</code></li>
<li>暂不支持<code>null</code></li>
</ol>
<p>那么就可以有下面的<code>read_columns</code>方法：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> read_columns</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">ArrayRef</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> result</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">ArrayRef</span><span>></span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Vec</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> in &self.</span><span>schema</span><span style="color: #E06C75;">.</span><span>fields {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Temporary ugly code</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> start_key</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> format!</span><span>("</span><span style="color: #98C379;">t</span><span>{}</span><span style="color: #98C379;">_c</span><span>{}</span><span style="color: #98C379;">_r000001</span><span>",</span><span style="color: #E06C75;"> self.</span><span>id,</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">name</span><span>());</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> prefix</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> format!</span><span>("</span><span style="color: #98C379;">t</span><span>{}</span><span style="color: #98C379;">_c</span><span>{}",</span><span style="color: #E06C75;"> self.</span><span>id,</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">name</span><span>());</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> iter</span><span style="color: #E06C75;"> = self.</span><span>db</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">iterator</span><span>(</span><span style="color: #61AFEF;">IteratorMode</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">From</span><span>(</span><span style="color: #C6CCD7;">start_key</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_bytes</span><span>(),</span><span style="color: #61AFEF;"> rocksdb</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">Direction</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Forward</span><span>));</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> values</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Vec</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> while</span><span style="color: #61AFEF;"> let</span><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #E5C07B;">Ok</span><span>((</span><span style="color: #C6CCD7;">key</span><span>,</span><span style="color: #C6CCD7;"> value</span><span>)))</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> iter</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">next</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Safety:</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // All keys and values are supposed to be encoded from utf8</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> unsafe</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> key_str</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from_utf8_unchecked</span><span>(</span><span style="color: #C6CCD7;">key</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_vec</span><span>());</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if !</span><span style="color: #C6CCD7;">key_str</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">starts_with</span><span>(</span><span style="color: #C6CCD7;">prefix</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_str</span><span>()) {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> break</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> value_str</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from_utf8_unchecked</span><span>(</span><span style="color: #C6CCD7;">value</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_vec</span><span>());</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> values</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #C6CCD7;">value_str</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> array</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> ArrayRef</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">StringArray</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #C6CCD7;">values</span><span>));</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> result</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #C6CCD7;">array</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> result</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>这样一来，就可以查本<code>DBTableScanExec</code>实例对应<code>table_id</code>和<code>schema</code>的所有数据了。</p>
<h2 id="datasink">DataSink</h2>
<p><code>DataSink</code>的特点是有一个<code>write_all</code>方法，它的参数提供了一个<code>Stream</code>表示数据源，我们要实现的则是将<code>Stream</code>带来的数据写入"Sink"中，可以是各种目标，我这里的实现是将数据放进KV存储引擎中。</p>
<p>首先说明，实现<code>CustomDataSink</code>的目的是让Datafusion的<code>DataSinkExec</code>能够执行自定义的<code>write_all</code>方法。这个流程，举例来说，先在<code>ctx</code>中注册了已经实现了<code>KVTable</code>，然后使用<code>LogicPlanBuilder</code>构建一个简单的<code>insert_into</code> 逻辑计划。这时候<code>KVTable</code>中实现的<code>insert_into</code>就派上用场了，它会返回一个<code>DataSinkExec</code>附带了目标Sink的信息（也就是我们实现的<code>KVSink</code>）。再用<code>ctx</code>生成执行计划，最后调用<code>collect</code>回调计算函数，此时<code>write_all</code>就会在数据准备好之后被调用。</p>
<p>下面来看关键代码：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[async_trait]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> TableProvider</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> KVSource</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;">...</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> insert_into</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _state</span><span style="color: #E06C75;">: &dyn</span><span style="color: #E5C07B;"> Session</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> input</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _overwrite</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> bool</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E06C75;">dyn</span><span style="color: #E5C07B;"> ExecutionPlan</span><span>>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> sink</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">KVTableSink</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E06C75;">self.</span><span>table_id,</span><span style="color: #E06C75;"> &self.</span><span>db));</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #E5C07B;">Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">DataSinkExec</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> input</span><span>,</span><span style="color: #5F6672;font-style: italic;"> // input source informed by the caller of insert_into</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> sink</span><span>,</span><span style="color: #5F6672;font-style: italic;"> // target sink</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>meta</span><span style="color: #E06C75;">.</span><span>schema</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">clone</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> None</span><span>,</span></span>
<span class="giallo-l"><span> )))</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;">...</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>这个函数帮助我们构造执行计划，<code>DataSinkExec</code>会在执行的时候调用<code>write_all</code>，参考下面的<code>DataSinkExec::execute</code>：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> ExecutionPlan</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> DataSinkExec</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;">...</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> execute</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> partition</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> usize</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> context</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span><</span><span style="color: #E5C07B;">TaskContext</span><span>>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">SendableRecordBatchStream</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> partition</span><span style="color: #E06C75;"> !=</span><span style="color: #56B6C2;"> 0</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span style="color: #B57EDC;"> internal_err!</span><span>("</span><span style="color: #98C379;">DataSinkExec can only be called on partition 0!</span><span>");</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> data</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> execute_input_stream</span><span>(</span><span style="color: #5F6672;font-style: italic;"> // Fetch input stream here</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&self.</span><span>input),</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&self.</span><span>sink_schema),</span></span>
<span class="giallo-l"><span style="color: #56B6C2;"> 0</span><span>,</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">context</span><span>),</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;">?</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> count_schema</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&self.</span><span>count_schema);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> sink</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&self.</span><span>sink);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> stream</span><span style="color: #E06C75;"> =</span><span style="color: #61AFEF;"> futures</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">stream</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">once</span><span>(</span><span style="color: #E06C75;">async move</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> sink</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">write_all</span><span>(</span><span style="color: #C6CCD7;">data</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">context</span><span>)</span><span style="color: #E06C75;">.await.</span><span style="color: #B57EDC;">map</span><span>(</span><span style="color: #C6CCD7;">make_count_batch</span><span>)</span><span style="color: #5F6672;font-style: italic;"> // write_all here</span></span>
<span class="giallo-l"><span> })</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">boxed</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #E5C07B;">Box</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">pin</span><span>(</span><span style="color: #61AFEF;">RecordBatchStreamAdapter</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> count_schema</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> stream</span><span>,</span></span>
<span class="giallo-l"><span> )))</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;">...</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>我的实现没有考虑context要求和多个partition的情况，只实现了基本的写入。其中<code>put_batch_into_db</code>的实现细节就不展开了，大致就是将一个<code>cell</code>（不是很高效的value单位）对应一个<code>key</code>（编码方式大概是：<code>t[table_id]_c[col_id]_r[row_id]</code>）存入rocksdb。下面是我的<code>write_all</code>实现：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> write_all</span><span>(</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> data</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> SendableRecordBatchStream</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _context</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">Arc</span><span><</span><span style="color: #E5C07B;">TaskContext</span><span>>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">u64</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> data</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> data</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> cnt</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 0</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #61AFEF;"> let</span><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #C6CCD7;">batch</span><span>)</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> data</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">next</span><span>()</span><span style="color: #E06C75;">.await.</span><span style="color: #B57EDC;">transpose</span><span>()</span><span style="color: #E06C75;">?</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> cnt</span><span style="color: #E06C75;"> = self.</span><span style="color: #B57EDC;">put_batch_into_db</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">batch</span><span>)</span><span style="color: #E06C75;">.await?</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #C6CCD7;">cnt</span><span>)</span></span>
<span class="giallo-l"><span> }</span></span></code></pre><h2 id="bonus-ce-shi">BONUS：测试</h2>
<p>另外补充一下写的几个基本测试，主要是记录一下测试的方法。</p>
<p>首先，是元数据的编解码测试：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span> #[tokio</span><span style="color: #E06C75;">::</span><span>test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> test_meta_encode_decode</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> meta</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> KVTableMeta</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> id</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 1002</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">:</span><span> "</span><span style="color: #98C379;">TableTest</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Fields</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">column1</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>),</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">column2</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>),</span></span>
<span class="giallo-l"><span> ]))),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> highest</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 0</span><span>,</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> key</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> meta</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">make_key</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> val</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> meta</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">make_value</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">key</span><span>, "</span><span style="color: #98C379;">mt1002</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">into_bytes</span><span>());</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">val</span><span>, "</span><span style="color: #98C379;">t1002_TableTest_0_c2_column1_column2</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">into_bytes</span><span>());</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> decode</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> KVTableMeta</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #C6CCD7;">val</span><span>);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">meta</span><span style="color: #E06C75;">.</span><span>id,</span><span style="color: #C6CCD7;"> decode</span><span style="color: #E06C75;">.</span><span>id);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">meta</span><span style="color: #E06C75;">.</span><span>highest,</span><span style="color: #C6CCD7;"> decode</span><span style="color: #E06C75;">.</span><span>highest);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">meta</span><span style="color: #E06C75;">.</span><span>name,</span><span style="color: #C6CCD7;"> decode</span><span style="color: #E06C75;">.</span><span>name);</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>对于<code>DBTableScanExec</code>结构，定义表元数据、直接通过KV的API添加100个元素，然后使用<code>DBTableScanExec</code>执行返回的<code>Stream</code>获取<code>Batch</code>后逐行比对：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span> #[tokio</span><span style="color: #E06C75;">::</span><span>test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> test_read_from_db</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> meta</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">KVTableMeta</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> id</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 101</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">:</span><span> "</span><span style="color: #98C379;">a</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">hello</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>),</span></span>
<span class="giallo-l"><span> ])),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> highest</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 0</span><span>,</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> key_prefix</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">t101_chello_r</span><span>";</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> target</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">hell</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>),</span></span>
<span class="giallo-l"><span> ]));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> DB</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">open_default</span><span>("</span><span style="color: #98C379;">./tmp</span><span>")</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> src</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> KVTable</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">meta</span><span>,</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #C6CCD7;">db</span><span>));</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..</span><span style="color: #56B6C2;">100</span><span style="color: #E5C07B;">i32</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> key</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> format!</span><span>("{</span><span style="color: #98C379;">key_prefix</span><span>}{</span><span style="color: #98C379;">i:0>6</span><span>}")</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">into_bytes</span><span>();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> src</span><span style="color: #E06C75;">.</span><span>db</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">put</span><span>(</span><span style="color: #C6CCD7;">key</span><span>,</span><span style="color: #B57EDC;"> format!</span><span>("{</span><span style="color: #98C379;">i</span><span>}")</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">into_bytes</span><span>())</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> table_exec</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> DBTableScanExec</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">101</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">target</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">src</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> result</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> table_exec</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">execute</span><span>(</span><span style="color: #56B6C2;">0</span><span>,</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">TaskContext</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">default</span><span>()));</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> output</span><span style="color: #E06C75;"> = match</span><span style="color: #C6CCD7;"> result</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #C6CCD7;">out</span><span>)</span><span style="color: #E06C75;"> =></span><span style="color: #C6CCD7;"> out</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Err</span><span>(</span><span style="color: #C6CCD7;">_</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #56B6C2;">9</span><span>,</span><span style="color: #56B6C2;"> 1</span><span>);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> stream</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> output</span><span>;</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Only 1 batch</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #61AFEF;"> let</span><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #C6CCD7;">batch</span><span>)</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> stream</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">next</span><span>()</span><span style="color: #E06C75;">.await</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> match</span><span style="color: #C6CCD7;"> batch</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #C6CCD7;">batch</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">num_rows</span><span>(),</span><span style="color: #56B6C2;"> 100</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">columns</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">get</span><span>(</span><span style="color: #56B6C2;">0</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span> (</span><span style="color: #C6CCD7;">i</span><span>,</span><span style="color: #C6CCD7;"> row</span><span>)</span><span style="color: #E06C75;"> in</span><span style="color: #B57EDC;"> as_string_array</span><span>(</span><span style="color: #C6CCD7;">col</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">iter</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">enumerate</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">row</span><span>,</span><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #B57EDC;">format!</span><span>("{</span><span style="color: #98C379;">i</span><span>}")</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_str</span><span>()))</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // println!("{i} {}", row.unwrap());</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _</span><span style="color: #E06C75;"> =></span><span> {}</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>最后，对<code>KVTableSink</code>进行测试，目的是为了确保数据确实有通过<code>insert_into</code>插入表中：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// Create a `KVTable` with a single column and `insert into` it</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// by `values`, check if the data is inserted</span></span>
<span class="giallo-l"><span> #[tokio</span><span style="color: #E06C75;">::</span><span>test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> test_db_write</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><()> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create a new schema with one field called "a" of type Int32</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span><span style="color: #61AFEF;">Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">a</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>)]));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create a new batch of data to insert into the table</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> RecordBatch</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">try_new</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">clone</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #61AFEF;">Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">StringArray</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>["</span><span style="color: #98C379;">hello</span><span>", "</span><span style="color: #98C379;">world</span><span>", "</span><span style="color: #98C379;">!</span><span>"]))],</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;">?</span><span>;</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Run the experiment and obtain the resulting data in the table</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> resulting_data_in_table</span><span style="color: #E06C75;"> =</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> experiment</span><span>(</span><span style="color: #C6CCD7;">schema</span><span>,</span><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #B57EDC;">vec!</span><span>[</span><span style="color: #C6CCD7;">batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">clone</span><span>()]],</span><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #B57EDC;">vec!</span><span>[</span><span style="color: #C6CCD7;">batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">clone</span><span>()]])</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .await?</span><span>;</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Ensure that the table now contains two batches of data in the same partition</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> in</span><span style="color: #C6CCD7;"> resulting_data_in_table</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">columns</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> arr</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> as_string_array</span><span>(</span><span style="color: #C6CCD7;">col</span><span>);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> arr</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &</span><span style="color: #E5C07B;">StringArray</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>["</span><span style="color: #98C379;">hello</span><span>", "</span><span style="color: #98C379;">world</span><span>", "</span><span style="color: #98C379;">!</span><span>", "</span><span style="color: #98C379;">hello</span><span>", "</span><span style="color: #98C379;">world</span><span>", "</span><span style="color: #98C379;">!</span><span>"]),</span></span>
<span class="giallo-l"><span> )</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // todo: remove test table after this test</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(())</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// This function create a table with `initial_data` to </span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// insert `inserted_data` into and return the final batch of the table.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> async fn</span><span style="color: #B57EDC;"> experiment</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> SchemaRef</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> initial_data</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">Vec</span><span><</span><span style="color: #E5C07B;">RecordBatch</span><span>>>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> inserted_data</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">Vec</span><span><</span><span style="color: #E5C07B;">RecordBatch</span><span>>>,</span></span>
<span class="giallo-l"><span> )</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><</span><span style="color: #E5C07B;">RecordBatch</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> expected_count</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u64</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> inserted_data</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">iter</span><span>()</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">flat_map</span><span>(</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;">batches</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;"> batches</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">iter</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">map</span><span>(</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;">batch</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">num_rows</span><span>()</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> u64</span><span>))</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">sum</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create a new session context</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> session_ctx</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> SessionContext</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>();</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create meta of a table</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> dest_meta</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">KVTableMeta</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> id</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 1002</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">:</span><span> "</span><span style="color: #98C379;">Dest</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Fields</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">a</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>),</span></span>
<span class="giallo-l"><span> ]))),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> highest</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 0</span><span>,</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create KV store</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> DB</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">open_default</span><span>("</span><span style="color: #98C379;">tmp</span><span>")</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> db</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #C6CCD7;">db</span><span>);</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create and register the initial table with the provided schema and data</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> initial_table</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">KVTable</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">try_new</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">dest_meta</span><span>,</span><span style="color: #61AFEF;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">db</span><span>),</span><span style="color: #C6CCD7;"> initial_data</span><span>)</span><span style="color: #E06C75;">.await?</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> session_ctx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">register_table</span><span>("</span><span style="color: #98C379;">Dest</span><span>",</span><span style="color: #C6CCD7;"> initial_table</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">clone</span><span>())</span><span style="color: #E06C75;">?</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create source table meta</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> src_meta</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> KVTableMeta</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> id</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 1001</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">:</span><span> "</span><span style="color: #98C379;">Src</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> schema</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Fields</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">a</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>),</span></span>
<span class="giallo-l"><span> ]))),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> highest</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 0</span><span>,</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> exprs</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #E5C07B;">Expr</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Literal</span><span>(</span><span style="color: #61AFEF;">ScalarValue</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Utf8</span><span>(</span><span style="color: #E5C07B;">Some</span><span>("</span><span style="color: #98C379;">hello</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>())))],</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #E5C07B;">Expr</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Literal</span><span>(</span><span style="color: #61AFEF;">ScalarValue</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Utf8</span><span>(</span><span style="color: #E5C07B;">Some</span><span>("</span><span style="color: #98C379;">world</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>())))],</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #E5C07B;">Expr</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Literal</span><span>(</span><span style="color: #61AFEF;">ScalarValue</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Utf8</span><span>(</span><span style="color: #E5C07B;">Some</span><span>("</span><span style="color: #98C379;">!</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>())))],</span></span>
<span class="giallo-l"><span> ];</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> values_plan</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> LogicalPlanBuilder</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">values</span><span>(</span><span style="color: #C6CCD7;">exprs</span><span>)</span><span style="color: #E06C75;">?</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">project</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span><span style="color: #61AFEF;">Expr</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Column</span><span>("</span><span style="color: #98C379;">column1</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">into</span><span>())</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">alias</span><span>("</span><span style="color: #98C379;">a</span><span>")])</span><span style="color: #E06C75;">?</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">build</span><span>()</span><span style="color: #E06C75;">?</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create an insert plan to insert the source data into the initial table</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> insert_into_table</span><span style="color: #E06C75;"> =</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> LogicalPlanBuilder</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">insert_into</span><span>(</span><span style="color: #C6CCD7;">values_plan</span><span>, "</span><span style="color: #98C379;">Dest</span><span>",</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">schema</span><span>,</span><span style="color: #56B6C2;"> false</span><span>)</span><span style="color: #E06C75;">?.</span><span style="color: #B57EDC;">build</span><span>()</span><span style="color: #E06C75;">?</span><span>;</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Create a physical plan from the insert plan</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> plan</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> session_ctx</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">state</span><span>()</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">create_physical_plan</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">insert_into_table</span><span>)</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .await?</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Execute the physical plan and collect the results</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> res</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> collect</span><span>(</span><span style="color: #C6CCD7;">plan</span><span>,</span><span style="color: #C6CCD7;"> session_ctx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">task_ctx</span><span>())</span><span style="color: #E06C75;">.await?</span><span>;</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #B57EDC;">extract_count</span><span>(</span><span style="color: #C6CCD7;">res</span><span>),</span><span style="color: #C6CCD7;"> expected_count</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> target_schema</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Schema</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Fields</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Field</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>("</span><span style="color: #98C379;">a</span><span>",</span><span style="color: #61AFEF;"> DataType</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Utf8</span><span>,</span><span style="color: #56B6C2;"> false</span><span>),</span></span>
<span class="giallo-l"><span> ])));</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> exec</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> DBTableScanExec</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">1002</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">target_schema</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">initial_table</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> stream</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> exec</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">execute</span><span>(</span><span style="color: #56B6C2;">0</span><span>,</span><span style="color: #C6CCD7;"> session_ctx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">task_ctx</span><span>())</span><span style="color: #E06C75;">?</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #61AFEF;"> let</span><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #C6CCD7;">batch</span><span>)</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> stream</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">next</span><span>()</span><span style="color: #E06C75;">.await.</span><span style="color: #B57EDC;">transpose</span><span>()</span><span style="color: #E06C75;">?</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #C6CCD7;">batch</span><span>)</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Err</span><span>(</span><span style="color: #B57EDC;">exec_datafusion_err!</span><span>("</span><span style="color: #98C379;">unknown err when fetching batch from stream</span><span>"))</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// This function takes out the number from batches, and</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> /// `res` should have only one row and one column.</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> extract_count</span><span>(</span><span style="color: #C6CCD7;">res</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">RecordBatch</span><span>>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> u64</span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">res</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>(),</span><span style="color: #56B6C2;"> 1</span><span>, "</span><span style="color: #98C379;">expected one batch, got </span><span>{}",</span><span style="color: #C6CCD7;"> res</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>());</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;"> = &</span><span style="color: #C6CCD7;">res</span><span>[</span><span style="color: #56B6C2;">0</span><span>];</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">num_columns</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #56B6C2;"> 1</span><span>,</span></span>
<span class="giallo-l"><span> "</span><span style="color: #98C379;">expected 1 column, got </span><span>{}",</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">num_columns</span><span>()</span></span>
<span class="giallo-l"><span> );</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> batch</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">column</span><span>(</span><span style="color: #56B6C2;">0</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_primitive</span><span style="color: #E06C75;">::</span><span><</span><span style="color: #E5C07B;">UInt64Type</span><span>>();</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">col</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>(),</span><span style="color: #56B6C2;"> 1</span><span>, "</span><span style="color: #98C379;">expected 1 row, got </span><span>{}",</span><span style="color: #C6CCD7;"> col</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>());</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> val</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> col</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">iter</span><span>()</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">next</span><span>()</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">expect</span><span>("</span><span style="color: #98C379;">had value</span><span>")</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">expect</span><span>("</span><span style="color: #98C379;">expected non null</span><span>");</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> val</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>完整代码可以参考<a rel="external" href="https://github.com/Kev1n8/drdb/commit/62a552803d070be7db2381351013d021b99f6e1e">这里</a>。</p>
StringView in Apache Arrow & a DataFusion SUBSTR Optimization
2024-09-09T00:00:00+00:00
2024-09-09T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/stringview-in-apache-arrow-a-datafusion-substr-optimization/
<p>Apache Arrow是一个高效的列式数据的内存表示，本文所指Arrow是arrow-rs，也就是Arrow的Rust实现，本文介绍其<code>StringViewArray</code>与<code>StringArray</code>的不同，这也是Rust版本才有的特性。</p>
<h2 id="stringarrayjian-jie">StringArray简介</h2>
<p><code>StringArray</code>是arrow的一种基本列实例、是一列字符串数据，一个<code>StringArray</code>主要有</p>
<ul>
<li>一个<code>nullmap</code>，指示某一行是否为null</li>
<li>一个<code>offset</code>数组，指示某一行的字符串在Buffer上的位移</li>
<li>一个Buffer用来存储实际的UTF8字符串数据</li>
</ul>
<h2 id="yi-ge-stringviewchang-shen-me-yang">一个StringView长什么样？</h2>
<p><code>StringViewArray</code>中同样也有<code>nullmap</code>，但是却不使用简单的<code>offset</code>，而是用一个<code>StringView</code>来指示字符串。首先，一个<code>StringView</code>的大小为16字节，前4个字节用来表示字符串长度。剩余的12字节分两种情况，</p>
<ol>
<li>字符串编码字节数小于12时，该字符串直接内联到存放在剩余的12字节内</li>
<li>字节数大于12时，存放buffer编号、偏移和前缀，各占用4字节。其中，buffer编号即完整字符串数据所存放的，属于该<code>StringViewArray</code>的buffer的编号；偏移即在该buffer中第几个字节开始为该字符串数据；前缀存放了字符串前4个字节的编码数据。</li>
</ol>
<p><img src="/images/Stringview.png" alt="Stringview" />{:height="360px" width="auto"}</p>
<h2 id="stringarray-vs-stringviewarray">StringArray vs StringViewArray</h2>
<p>好处显而易见，如果一个Array中的字符串有大量重复，那么<code>StringView</code>就可以简单指向Buffer中的同一段数据，大大减少内存用量。如下面的示例，对于一个Array：</p>
<p>"Apache StringView"</p>
<p>"Apache DataFusion"</p>
<p>"Apache StringView"</p>
<p>"InfluxDB"</p>
<p><code>StringViewArray</code>能够重复利用此前出现过的字符串，相比<code>StringArray</code>，Buffer减少了17字节（另外这里可以看出在数据量小的情况，因为<code>StringView</code>占用的内存大于一般的<code>OffsetSize</code>，如<code>i32, i64</code>，<code>StringView</code>的好处较难体现）</p>
<p><img src="/images/StringViewDesign.png" alt="StringViewDesign" />{:height="230px" width="auto"}</p>
<h2 id="stringviewarrayru-he-gou-jian">StringViewArray如何构建</h2>
<p>arrow提供了<code>StringViewArrayBuilder</code>来构建一个Array，而关键函数就是<code>append_value(&mut self, value: impl AsRef<T::Native>)</code>。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> append_value</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self</span><span>,</span><span style="color: #C6CCD7;"> value</span><span style="color: #E06C75;">: impl</span><span style="color: #E5C07B;"> AsRef</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Native</span><span>>) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;">: &</span><span>[</span><span style="color: #E5C07B;">u8</span><span>]</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> value</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_ref</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_ref</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> length</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">try_into</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> length</span><span style="color: #E06C75;"> <=</span><span style="color: #56B6C2;"> 12</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> view_buffer</span><span style="color: #E06C75;"> =</span><span> [</span><span style="color: #56B6C2;">0</span><span>;</span><span style="color: #56B6C2;"> 16</span><span>];</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> view_buffer</span><span>[</span><span style="color: #56B6C2;">0</span><span style="color: #E06C75;">..</span><span style="color: #56B6C2;">4</span><span>]</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">copy_from_slice</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">length</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_le_bytes</span><span>());</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> view_buffer</span><span>[</span><span style="color: #56B6C2;">4</span><span style="color: #E06C75;">..</span><span style="color: #56B6C2;">4</span><span style="color: #E06C75;"> +</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>()]</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">copy_from_slice</span><span>(</span><span style="color: #C6CCD7;">v</span><span>);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>views_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">append</span><span>(</span><span style="color: #61AFEF;">u128</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from_le_bytes</span><span>(</span><span style="color: #C6CCD7;">view_buffer</span><span>));</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>null_buffer_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">append_non_null</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Deduplication if:</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // (1) deduplication is enabled.</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // (2) len > 12</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #61AFEF;"> let</span><span style="color: #E5C07B;"> Some</span><span>((</span><span style="color: #61AFEF;">mut</span><span style="color: #C6CCD7;"> ht</span><span>,</span><span style="color: #C6CCD7;"> hasher</span><span>))</span><span style="color: #E06C75;"> = self.</span><span>string_tracker</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">take</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> hash_val</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> hasher</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">hash_one</span><span>(</span><span style="color: #C6CCD7;">v</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> hasher_fn</span><span style="color: #E06C75;"> = |</span><span style="color: #C6CCD7;">v</span><span style="color: #E06C75;">: &</span><span style="color: #C6CCD7;">_</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;"> hasher</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">hash_one</span><span>(</span><span style="color: #C6CCD7;">v</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> entry</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> ht</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">entry</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> hash_val</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> |</span><span style="color: #C6CCD7;">idx</span><span style="color: #E06C75;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> stored_value</span><span style="color: #E06C75;"> = self.</span><span style="color: #B57EDC;">get_value</span><span>(</span><span style="color: #E06C75;">*</span><span style="color: #C6CCD7;">idx</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;"> ==</span><span style="color: #C6CCD7;"> stored_value</span></span>
<span class="giallo-l"><span> },</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> hasher_fn</span><span>,</span></span>
<span class="giallo-l"><span> );</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> match</span><span style="color: #C6CCD7;"> entry</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Entry</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Occupied</span><span>(</span><span style="color: #C6CCD7;">occupied</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // If the string already exists, we will directly use the view</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> idx</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> occupied</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">get</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>views_builder</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">append</span><span>(</span><span style="color: #E06C75;">self.</span><span>views_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_slice</span><span>()[</span><span style="color: #E06C75;">*</span><span style="color: #C6CCD7;">idx</span><span>]);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>null_buffer_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">append_non_null</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>string_tracker </span><span style="color: #E06C75;">=</span><span style="color: #E5C07B;"> Some</span><span>((</span><span style="color: #C6CCD7;">ht</span><span>,</span><span style="color: #C6CCD7;"> hasher</span><span>));</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> return</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Entry</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">Vacant</span><span>(</span><span style="color: #C6CCD7;">vacant</span><span>)</span><span style="color: #E06C75;"> =></span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // o.w. we insert the (string hash -> view index)</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // the idx is current length of views_builder, as we are inserting a new view</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> vacant</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">insert</span><span>(</span><span style="color: #E06C75;">self.</span><span>views_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>());</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>string_tracker </span><span style="color: #E06C75;">=</span><span style="color: #E5C07B;"> Some</span><span>((</span><span style="color: #C6CCD7;">ht</span><span>,</span><span style="color: #C6CCD7;"> hasher</span><span>));</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> required_cap</span><span style="color: #E06C75;"> = self.</span><span>in_progress</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>()</span><span style="color: #E06C75;"> +</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if self.</span><span>in_progress</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">capacity</span><span>() <</span><span style="color: #C6CCD7;"> required_cap</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span style="color: #B57EDC;">flush_in_progress</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> to_reserve</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">max</span><span>(</span><span style="color: #E06C75;">self.</span><span>block_size</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">next_size</span><span>()</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> usize</span><span>);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>in_progress</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">reserve</span><span>(</span><span style="color: #C6CCD7;">to_reserve</span><span>);</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> offset</span><span style="color: #E06C75;"> = self.</span><span>in_progress</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>()</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> u32</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>in_progress</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">extend_from_slice</span><span>(</span><span style="color: #C6CCD7;">v</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> view</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> ByteView</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> length</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> prefix</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from_le_bytes</span><span>(</span><span style="color: #C6CCD7;">v</span><span>[</span><span style="color: #56B6C2;">0</span><span style="color: #E06C75;">..</span><span style="color: #56B6C2;">4</span><span>]</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">try_into</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>()),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> buffer_index</span><span style="color: #E06C75;">: self.</span><span>completed</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>()</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> u32</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> offset</span><span>,</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>views_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">append</span><span>(</span><span style="color: #C6CCD7;">view</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">into</span><span>());</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>null_buffer_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">append_non_null</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>大致逻辑如下：</p>
<ol>
<li>检查value大小，如果小于12，直接内联并制作一个view，使用成员<code>views_builder</code>加入views中</li>
<li>若大于12，则会使用哈希方法检查是否在buffer中已有该字符串，如有则直接根据所在位置制作一个view并添加</li>
<li>若不存在这个字符串，就会尝试添加进buffer，此处如果buffer达到指定大小，则会保存起来作为只读“历史buffer”，以便GC。</li>
</ol>
<p>当然，这是从零构建一个<code>StringViewArray</code>的方法，我们还可以直接通过<code>generic_view_array::new_unchecked</code>方法来不安全地直接构造一个Array，具体见下文。</p>
<h2 id="shi-yong-stringviewyou-hua-substrhan-shu">使用StringView优化SUBSTR函数</h2>
<p><code>substr(str, start, [count])</code>可以帮助我们获得字符串的指定字串，因为<code>substr</code>的结果一定是原字符串的子集，那么如果输入的<code>DataType</code>是<code>StringViewArray</code>的话，我们就可以这样操作：</p>
<ol>
<li>直接将原本的所有buffer拷贝作为新Array的buffer，</li>
<li>然后直接操作每个<code>StringView</code>， 只需要根据通过计算得到的<code>sub_string</code>的index构建view并载入<code>views_buffer</code>中</li>
</ol>
<p>这样做的好处是：</p>
<ul>
<li>使用<code>StringViewArray</code>带来了节约内存的好处</li>
<li>直接复制所有buffer相比通过此前<code>generic_array_builder</code>一次次调用<code>append_value</code>更高效。虽然buffer可能暂时会有无用的部份，但是GC可以后续把无用的部份回收掉。</li>
</ul>
<p>下面列举一个示例函数用于手动收集view和构造<code>nullmap</code>以便在<code>new_unchecked</code>中使用：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #5F6672;font-style: italic;">/// Make a `u128` based on the given substr, start(offset to view.offset), and</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">/// push into to the given buffers</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> make_and_append_view</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> views_buffer</span><span style="color: #E06C75;">: &</span><span style="color: #61AFEF;">mut</span><span style="color: #E5C07B;"> Vec</span><span><</span><span style="color: #E5C07B;">u128</span><span>>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> null_builder</span><span style="color: #E06C75;">: &</span><span style="color: #61AFEF;">mut</span><span style="color: #E5C07B;"> NullBufferBuilder</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> raw</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">u128</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> substr</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> start</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>,</span></span>
<span class="giallo-l"><span>) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> substr_len</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> substr</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> sub_view</span><span style="color: #E06C75;"> = if</span><span style="color: #C6CCD7;"> substr_len</span><span style="color: #E06C75;"> ></span><span style="color: #56B6C2;"> 12</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> view</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> ByteView</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #E06C75;">*</span><span style="color: #C6CCD7;">raw</span><span>);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> make_view</span><span>(</span><span style="color: #C6CCD7;">substr</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_bytes</span><span>(),</span><span style="color: #C6CCD7;"> view</span><span style="color: #E06C75;">.</span><span>buffer_index,</span><span style="color: #C6CCD7;"> view</span><span style="color: #E06C75;">.</span><span>offset </span><span style="color: #E06C75;">+</span><span style="color: #C6CCD7;"> start</span><span>)</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // inline value does not need block id or offset</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> make_view</span><span>(</span><span style="color: #C6CCD7;">substr</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_bytes</span><span>(),</span><span style="color: #56B6C2;"> 0</span><span>,</span><span style="color: #56B6C2;"> 0</span><span>)</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> views_buffer</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #C6CCD7;">sub_view</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> null_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">append_non_null</span><span>();</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// Somewhere else:</span></span>
<span class="giallo-l"><span style="color: #E06C75;">...</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> views_buf</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> ScalarBuffer</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #C6CCD7;">views_buf</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> nulls_buf</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> null_builder</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">finish</span><span>();</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // Safety:</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // (1) The blocks of the given views are all provided</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // (2) Each of the range `view.offset+start..end` of view in views_buf is within</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // the bounds of each of the blocks</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> unsafe</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> array</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> StringViewArray</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new_unchecked</span><span>(</span><span style="color: #5F6672;font-style: italic;"> // 在这里直接构造一个StringViewArray</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> views_buf</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> string_view_array</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">data_buffers</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_vec</span><span>(),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> nulls_buf</span><span>,</span></span>
<span class="giallo-l"><span> );</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(</span><span style="color: #E5C07B;">Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #C6CCD7;">array</span><span>)</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> ArrayRef</span><span>)</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;">...</span></span></code></pre>
<p>完整的优化代码可以参考<a rel="external" href="https://github.com/apache/datafusion/pull/12044">这里</a>。</p>
Distributed System Lab
2024-07-20T00:00:00+00:00
2024-07-20T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/distributed-system-lab/
<h1 id="distributed-system-labs">Distributed System Labs</h1>
<p>Some drafts and notes.</p>
<h2 id="1-mapreduce">#1 MapReduce</h2>
<h3 id="test-mr-sh">test_mr.sh</h3>
<p><code>mrcoordinator .//pg*txt</code></p>
<p>then multiple workers</p>
<p><code>mrworker .//.//mrapps/app.so</code></p>
<p><strong>Workflow</strong>:</p>
<p>master awake, prepare rpc and splits. workers awake, prepare map/reduce functions and start asking for task.</p>
<ol>
<li>master get M map tasks and R reduce tasks ready. picking up workers to assign</li>
<li>map workers read the splits and save the intermediate files, tell the master about the R locations(which files)</li>
<li>master receive task done msg, master picks workers to do reduce task, telling them the location of the R they are going to deal with.</li>
<li>reduce workers use rpc to read target R files from different map workers. once all read, sort them all by inter-key. reduce workers then iterate over the sorted data and do reduce func. the ouput of the func is appended to a final output file of this R partition. done, call master is done</li>
<li>once all done, master</li>
</ol>
<h4 id="conclusion">Conclusion</h4>
<p>So I suppose that this is not that hard since it didn't take much time for me to accomplish the task. To be honest, the reason I'd been stuck for a little while when <code>bash test-mr.sh</code> is because the path of the file is incorrect (changed it to absolute path later ).</p>
<h2 id="2-key-value-server">#2 Key/Value Server</h2>
<p>To implement an in-memory server and clients. It's pretty easy that there's really no much to talk about.</p>
<p><strong>Techniques Used</strong></p>
<ol>
<li>Still, using <strong>RPC</strong> to call remote functions</li>
<li>Using <em>unique Ids</em> to represent each of the clients and each request.</li>
</ol>
<p><strong>2 Things blocked me for a while</strong></p>
<ol>
<li>
<p>A silly mistake - the fact is that <code>Call()</code> is executed only once, and then there's a terrible infinite loop</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="go"><span class="giallo-l"><span style="color: #E06C75;">for</span><span style="color: #C6CCD7;"> ok</span><span style="color: #E06C75;"> :=</span><span style="color: #C6CCD7;"> ck</span><span>.</span><span style="color: #B57EDC;">Call</span><span>("</span><span style="color: #98C379;">Server.Put</span><span>",</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">args</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">reply</span><span>);</span><span style="color: #E06C75;"> !</span><span style="color: #C6CCD7;">ok</span></span></code></pre></li>
<li>
<p>Using <em>Unique Ids</em> to represent <code>values</code> instead of Reqs - bad things would happen when other clients change the record, hence causing duplicate put/append</p>
</li>
</ol>
<h4 id="conclusion-1">Conclusion</h4>
<p>The lab is marked <strong>Easy</strong>, which is true just that I am supposed to finish it in real quick.</p>
<h2 id="3-raft">#3 Raft</h2>
<h3 id="election">Election</h3>
<p>I've drawn a figure listing out the summary of rules, check out:</p>
<p><img src="/images/election_state.png" alt="state_transfer" /></p>
<p>Also, some variables and details as:</p>
<p><img src="/images/election_variable&rules.png" alt="election_variables_rules" /></p>
<h3 id="log">Log</h3>
<p>The rule of log replication can be interpreted as the following graphs:</p>
<p><img src="/images/log_replication_basic.png" alt="log_replication_basic" /></p>
<p>Demonstrating an unreliable test that I'd debugged for a while and Tips:</p>
<p><img src="/images/log_replication_atest.png" alt="log_replication_atest" /></p>
<p>In order to back up faster, I also implement the following technique:</p>
<p><img src="/images/log_replication_quick_backup.png" alt="log_replication_quick_backup" /></p>
<p>By sending <code>XTerm</code>, <code>XIndex</code>, and <code>Len</code> back to the leader, the leader can then quickly decide what <code>nextIndex[target]</code> should be set and get rid of meaningless retries.</p>
<h3 id="persistence">Persistence</h3>
<p><img src="/images/persistence_summary.png" alt="persistence_summary" /></p>
<h3 id="log-compaction">Log Compaction</h3>
<p><img src="/images/log_compaction_procedure.png" alt="log_compaction_procedure" /></p>
<p><img src="/images/log_compaction_points.png" alt="log_compaction_points" /></p>
<h2 id="4-fault-tolerant-k-v-service">#4 Fault-Tolerant K/V Service</h2>
<p>Note: All of the fields in RPC args should be initialized, otherwise the RPC will fail.(on returning(never return))</p>
<p><img src="/images/lab4.png" alt="lab4" /></p>
Summary of 445 Projects
2024-05-10T00:00:00+00:00
2024-05-10T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/summary-of-445-projects/
<h1 id="summary-of-445-projects">Summary of 445 Projects</h1>
<h2 id="project-0">Project 0</h2>
<h3 id="task1">Task1</h3>
<p>首先，任务是实现Trie前缀树的Get、Put、Remove三种操作。和一般的树操作实现不同的是，这个Task要求<strong>Copy on Write</strong>。意思是说，在写操作时，获取Trie根结点的拷贝，然后在副本上面进行写操作，等写操作结束了以后再将它写回原本的Trie根。这么做的一个好处是读可以随时进行，因为写是在副本上的，在写完的那一个瞬间才会以类似”读“的形式写回去。</p>
<h3 id="task2">Task2</h3>
<p>并发控制，主要是两点：</p>
<ol>
<li>高级指针的使用，比如<code>std::unique_lock()</code>搭配互斥锁</li>
<li>资源访问时的上锁逻辑</li>
</ol>
<h3 id="task3">Task3</h3>
<p>主要是调试和检查变量状态</p>
<h3 id="task4">Task4</h3>
<p>两个函数的实现：根据Option对字符串进行大写或小写转换，以及在另一个cpp文件中注册并调用该函数</p>
<h3 id="zong-jie">总结</h3>
<p>Task1是占用时间最长的，主要原因是</p>
<ol>
<li>对现代C++语法不太熟悉</li>
<li>不熟悉COW的实现，还总是想着修改结点，而不是Clone()</li>
</ol>
<ul>
<li>熟悉了C++的现代高级语法，比如<strong>锁</strong>和<strong>高级指针</strong>的使用</li>
</ul>
<h2 id="project-1">Project 1</h2>
<h3 id="task1-1">Task1</h3>
<p>实现LRU-K算法，算是比较经典的题，第一次实现的时候，为了优先实现功能，于是直接给每一个函数都上了锁，等要Leaderboard挑战的时候再看怎么减小锁的粒度。</p>
<p>在本地实现都通过的情况下，Evict()一直过不了。后面发现，我的LRU-K在没有$k_{th}$引用记录的情况下，对overall最佳是采用LatestTimestamp，然而项目要求是Earliest。</p>
<h3 id="task2-1">Task2</h3>
<p>缓冲池管理实现。理解每个函数的要求以后做起来其实是比较轻松的。同样的，为了减少思考，第一次实现直接给整个函数上锁。十分顺利地通过大部份测试，除了<code>FetchPage</code>, <code>DelPage</code>, <code>IsDirty</code>和<code>Unpin</code>。</p>
<p>错误原因：</p>
<ul>
<li><code>FetchPage</code>：从磁盘读取数据的函数用错了，用了<code>strcpy</code>。看了提供的测试代码，应该用<code>memcpy</code>。前者截断了<code>'\0'</code>，而后者原封不动地拷贝内存</li>
<li><code>IsDirty</code>：因为在<code>Unpin</code>直接给page的dirty标志赋值为参数所给值，如果page本来是脏但是参数不脏就会导致错误。正确做法应该是求异或</li>
<li><code>DelPage</code>：一开始只是简单调用了<code>ResetMemory</code>，元数据没有重新初始化，导致错误</li>
</ul>
<p>此外，<code>Evict</code>进行后，要从<code>page_table</code>删除对应键值对，但是顺序错了。在删除后，<code>page_id</code>已经被设置成<code>INVALID</code>的情况下试图删除，造成的结果就是frame看似还在池里。这是最主要的错误，是上述错误的部份原因。</p>
<h3 id="task3-1">Task3</h3>
<p>页守卫的编写。最关键要理解它们的作用：</p>
<ul>
<li>Basic：负责自动Unpin</li>
<li>Read：负责自动释放读锁和Unpin</li>
<li>Write：负责自动释放写锁和Unpin</li>
</ul>
<p>这一节的代码量实际并不高，但是却在这里卡了许久。最后是让代码看上去更“规范”一点的情况下过了测试。猜测是因为一开始调用Drop()和处理要丢弃的项，导致锁可能没有顺利释放，造成其它线程永久等待。</p>
<h3 id="leaderboardtiao-zhan">Leaderboard挑战</h3>
<p>TODO，在通过Leaderboard测试的前提下，官方提供了一个Buffer Pool的Benchmark。根据官方建议和我自己的想法，可能的优化方法有：</p>
<ol>
<li>前面提到的<strong>锁粒度减小</strong>，分别在LRU-K和buffer pool manager优化锁的使用</li>
<li>Benchmark有8个序列扫描Scan线程，8个随机获取Get线程，一共16个线程。在LRU-K算法层添加请求类别这一参数，优化算法使其在Evict的时候能作出<strong>更有效率的决定</strong></li>
<li><strong>并行化I/O操作</strong>，其实就是1带来的好处</li>
</ol>
<p>多线程编程，一定要注意函数内外是否会重复获取互斥锁，不然会死锁；还有就是操作的原子性，有的时候读操作用写锁是必须的，比如在Unpin之前的page检查。</p>
<p>完成情况：</p>
<ol>
<li>
<p>锁粒度减小，一开始简单地以为把每个共享资源保护起来就可以了（用完就释放，导致前后不一致，见<strong>TIP</strong>）。最后的做法是把每个函数切割成两部分，第一部分是涉及到bpm的共享数据结构的部份，需要所有线程共用一把互斥锁保证访问顺序。第二部分是对页面<code>pages_</code>的具体操作，使用了<code>pool_size_</code>个互斥锁，让每个页面得以并行运行。</p>
<p><strong>TIP</strong>：最后看上去过了，后面其实修改了很久，因为多线程出错很随机，最后才终于想明白。突破口是bpm_bench.cpp文件，考虑到这里只涉及到NewPage, FetchPage和UnpinPage，而NewPage只是用来创建。分析思路：首先Fetch读到的page肯定是不能被篡改的，因为篡改的唯一途径是evict，而pin值不为0是不可能的，就是说正在读的page是不可能被篡改的。那么就考虑数据写入的时候数据就是错误的了。造成写入错误的可能有：</p>
<ol>
<li>page_id_错误</li>
<li>page_.data错误</li>
</ol>
<p>此前的<strong>数据写入测试</strong>正常通过，所以判断是page_id_出错。我的代码是两段锁，但是这样保证不了前后读取page_id的一致性。于是我把锁的范围扩大，终于通过测试。从最后到达排位540，末尾水平。</p>
</li>
<li>
<p>尝试了LRU-K添加一个lru链表（最后总体长得有点像倒排索引），位次提升到440。但是对于区分Scan和Get的算法优化，我只尝试了把Scan添加到lru中（LRUReplacer），位次到435。就在写这段话的时候，意识到在BPM层面还可以对Scan只进行以下个性化的Fetch</p>
<ol>
<li>检查是否在pool，在的话不计数</li>
<li>不在的话，也就是history为空，加一个远古时间戳——0</li>
</ol>
<p>上述操作能尽量让本次Scan仿佛没来过，如果遇到了连续的Scan，返回的也会是最近的Scan。</p>
<p>经过完善Fetch优化，提升到了259。题目在Unpin的参数还预留了一个Accesstype，暂时先放下了。</p>
</li>
</ol>
<h3 id="zong-jie-1">总结</h3>
<p>学到的东西：</p>
<ul>
<li>LRU-K的实现，全局时间戳</li>
<li>BPM实现，锁的覆盖范围怎么确定、unique_lock等的使用</li>
<li>Guard的基本原理，RAII技术、右值引用</li>
<li>互斥资源的保护</li>
</ul>
<h2 id="project-2-checkpoint-1">Project 2 Checkpoint#1</h2>
<h3 id="task1-2">Task1</h3>
<p>简单来说，就是完善B+树的page基类、internal page和leaf page的一些函数。前期刚开始做的时候被源码的意图弄得有点糊涂，这几个类其实也是在做后面的Task时逐步完善的。</p>
<h3 id="task2a">Task2a</h3>
<p>完成<code>GetValue</code>和<code>Insert</code>函数，现在能想到的要注意的点有：</p>
<ul>
<li>
<p><code>GetValue</code>： 只需要自上而下遍历树到相应的叶子结点就行了。写的时候被方向guide变量坑到了，一开始没有意识到二分查找的值和guide应该的对应关系。好在后面顺利修复了。关键词：二分查找、guide</p>
</li>
<li>
<p><code>Insert</code>：插入比较简单，只需要应对页的分裂的问题即可。但是做的时候状态不太好，索性照着教材的伪代码敲了，幸运的是没遇到什么bug，很快就通过了测试。中途SplitTest和ScaleTest没过（我想ScaleTest没过应该是Split的问题导致的）。到discord找线索，发现做B+树实现的人比哈希索引的人相对少好多。看到一个对于Split思路的hints，拷贝如下。总之按照这个思路过了一遍之后发现自己的Size调整有点问题，修改之后就过了。关键词：递归</p>
<blockquote>
<p>Make sure you follow this: "However, you should correctly perform split if insertion triggers current number of key/value pairs after insertion equals to max_size".</p>
<p>Leaf page : <strong>on</strong> successful insert, max size = actual size? Split it. In other words, the size of a leaf page is the number of values.</p>
<p>Internal page : <strong>before</strong> insert, max size = current size (aka current number of pointers)? Split it. In other words, the size of an internal page is the number of pointers <strong>including</strong> the next page pointer.</p>
<p>SplitTest is a pretty invasive test that unfortunately tests the implementation details (because it is hard not to). So your code might be right but the autograder might not accept it. Though more commonly, people make some error with MoveHalfTo or similar functions.</p>
</blockquote>
</li>
<li>
<p>补充：SplitTest后面因为更改了Split Leaf的写法（从先插入后split到先split后插，避免溢出），导致特殊情况（LeafMaxSize==2, InternalMaxSize==3的情况下没有处理好分裂操作，0位元素没有正确被剔除）下发生错误；另外，更改了刚刚说的先后顺序后，没有在最开始判断是否是已经存在的元素，导致元素会先被剔除后被判断，呈现的结果就是如下所示，<code>[1,2,3,4,5]</code>基础上再试图插入<code>[1,2]</code>，结果分裂为空：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="cpp"><span class="giallo-l"><span> (</span><span style="color: #56B6C2;">3</span><span>) </span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">2</span><span>) (</span><span style="color: #56B6C2;">4</span><span>,</span><span style="color: #56B6C2;">5</span><span>) </span></span>
<span class="giallo-l"><span> () () (</span><span style="color: #56B6C2;">3</span><span>) (</span><span style="color: #56B6C2;">4</span><span>) (</span><span style="color: #56B6C2;">5</span><span>) </span></span></code></pre></li>
</ul>
<h3 id="zong-jie-2">总结</h3>
<p>这次的Project难度还是挺大的，之前在课上了解的很多的优化方法，一到了实践，随着时间，我脑子里那些的优化理论都变成了最基本功能实现的编程细节。脚踏实地很重要。</p>
<p><strong>补充</strong>：</p>
<p>并发测试因为页的的max_size比较大，因此测出了我的二分查找其实有点问题，会出错的情况是：已有key[1, 2, 3, 4, 6]，想要插入5，此时应该返回4
错误代码：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="cpp"><span class="giallo-l"><span style="color: #61AFEF;">int</span><span> low </span><span style="color: #E06C75;">=</span><span style="color: #56B6C2;"> 0</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> int</span><span> high </span><span style="color: #E06C75;">=</span><span style="color: #C6CCD7;"> page</span><span>-></span><span style="color: #B57EDC;">GetSize</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> while</span><span> (low </span><span style="color: #E06C75;"><</span><span> high) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> int</span><span> mid </span><span style="color: #E06C75;">=</span><span> low </span><span style="color: #E06C75;">+</span><span> (high </span><span style="color: #E06C75;">-</span><span> low)</span><span style="color: #E06C75;"> /</span><span style="color: #56B6C2;"> 2</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span> (</span><span style="color: #C6CCD7;">comparator_</span><span>.</span><span style="color: #B57EDC;">cmp_</span><span>(</span><span style="color: #C6CCD7;">page</span><span>-></span><span style="color: #B57EDC;">KeyAt</span><span>(mid), key)</span><span style="color: #E06C75;"> ==</span><span style="color: #56B6C2;"> 1</span><span>) {</span></span>
<span class="giallo-l"><span> high </span><span style="color: #E06C75;">=</span><span> mid </span><span style="color: #E06C75;">-</span><span style="color: #56B6C2;"> 1</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // 这一步会让返回的pos是第一个“不大于”目标的数</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else if</span><span> (</span><span style="color: #C6CCD7;">comparator_</span><span>.</span><span style="color: #B57EDC;">cmp_</span><span>(</span><span style="color: #C6CCD7;">page</span><span>-></span><span style="color: #B57EDC;">KeyAt</span><span>(mid), key)</span><span style="color: #E06C75;"> == -</span><span style="color: #56B6C2;">1</span><span>) {</span></span>
<span class="giallo-l"><span> low </span><span style="color: #E06C75;">=</span><span> mid </span><span style="color: #E06C75;">+</span><span style="color: #56B6C2;"> 1</span><span>;</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span> low </span><span style="color: #E06C75;">=</span><span> mid;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> break</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span></code></pre>
<p>修正后：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="cpp"><span class="giallo-l"><span style="color: #61AFEF;">int</span><span> low </span><span style="color: #E06C75;">=</span><span style="color: #56B6C2;"> 0</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> int</span><span> high </span><span style="color: #E06C75;">=</span><span style="color: #C6CCD7;"> page</span><span>-></span><span style="color: #B57EDC;">GetSize</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> while</span><span> (low </span><span style="color: #E06C75;"><</span><span> high) {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> int</span><span> mid </span><span style="color: #E06C75;">=</span><span> low </span><span style="color: #E06C75;">+</span><span> (high </span><span style="color: #E06C75;">-</span><span> low)</span><span style="color: #E06C75;"> /</span><span style="color: #56B6C2;"> 2</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span> (</span><span style="color: #C6CCD7;">comparator_</span><span>.</span><span style="color: #B57EDC;">cmp_</span><span>(</span><span style="color: #C6CCD7;">page</span><span>-></span><span style="color: #B57EDC;">KeyAt</span><span>(mid), key)</span><span style="color: #E06C75;"> ==</span><span style="color: #56B6C2;"> 1</span><span>) {</span></span>
<span class="giallo-l"><span> high </span><span style="color: #E06C75;">=</span><span> mid;</span><span style="color: #5F6672;font-style: italic;"> // 收敛更“收敛”一点，这样能确保返回的pos是应该插入的位置</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else if</span><span> (</span><span style="color: #C6CCD7;">comparator_</span><span>.</span><span style="color: #B57EDC;">cmp_</span><span>(</span><span style="color: #C6CCD7;">page</span><span>-></span><span style="color: #B57EDC;">KeyAt</span><span>(mid), key)</span><span style="color: #E06C75;"> == -</span><span style="color: #56B6C2;">1</span><span>) {</span></span>
<span class="giallo-l"><span> low </span><span style="color: #E06C75;">=</span><span> mid </span><span style="color: #E06C75;">+</span><span style="color: #56B6C2;"> 1</span><span>;</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span> low </span><span style="color: #E06C75;">=</span><span> mid;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> break</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span></code></pre><h2 id="project-2-checkpoint-2">Project 2 Checkpoint#2</h2>
<h3 id="task2b">Task2b</h3>
<p>树的删除操作，本次project最难。最开始是想着按照官网提示的Roadmap自己写出伪代码，但是脑子转不太动。后面在YouTube上找到一个B+树删除操作的讲解视频，讲解非常清晰，很快理解了核心思路，最后自己把Delete完成了。同时也是最耗费时间的Task。绝大部分时间都花在调试上，出现错误的点为：</p>
<ul>
<li>根节点特别判断没有做好，具体是：前一个if条件判断要求是根节点，后一个if（不是else if）又没有要求不是根节点，导致根节点被当作普通parent进行merge</li>
</ul>
<p>==完整理解删除的两大操作：<code>Merge</code>和<code>Borrow</code>以后，这个操作从难以理解变为了比较难落实。==</p>
<h3 id="task3-2">Task3</h3>
<p>Iterator实现，和前面的操作比起来过于简单。值得注意的点：</p>
<ul>
<li>End的判定，到底怎么样才算End？</li>
<li>++操作，怎么判断到没到底？到底了怎么办？</li>
</ul>
<h3 id="task4-1">Task4</h3>
<p>利用Project1的PageGuard非常好实现，只要注意写锁和读锁的使用和释放位置就行。</p>
<hr />
<p>但是并发MixTest一直无法通过，涉及到的log都是说对应key不存在。</p>
<p>我自己测试了多达10000个key的插入，得到的树看似非常正常。再试了一下1000个key，发现key的顺序被打乱，！！！原来是我的<code>BorrowFromInternal</code>在向右sibling借的时候，我的实现只满足了<code>max_internal_size</code>为2的情况。</p>
<hr />
<p>最后只剩并发Insert2和Mix1过不了，索性整个Insert函数上锁，结果通过了。于是确定只是锁的问题。</p>
<p>Contention一直过不了。在并发Insert时偶尔发生Segementation fault (具体是在InsertLeaf和InsertParent)，有时发生死锁。</p>
<p><strong>死锁排查过程：</strong></p>
<ol>
<li>
<p>想到死锁可能发生的情况：buffer pool满了而且每个页都pinned，这时候如果请求HeaderPage而且池子里没有HeaderPage的话，就会锁住？不是。</p>
</li>
<li>
<p>找到Segmentation Fault，是由于latch crabbing的时候，条件判断错误了，只因我最开始将InternalPage的Size设计成key的数量，很混乱，总之错了+1+2的问题。现在还差ContentionBenchmark会发生死锁。</p>
</li>
<li>
<p>通过打印大量log，我发现，在两个线程的情况下，每次死锁前都是一个线程先输出FetchWrite后输出Fetch header page txn: xx。问题是这时候没人拥有锁啊！为什么会死锁呢？</p>
</li>
<li>
<p>原来死锁来自同一线程重复请求header page，第二次请求来自<code>next_page_id</code>（从根到叶的途中），理论上这不可能发生。发现是因为buffer pool manager没有将此前<code>NewPageGuard</code>得到的page flush到磁盘（通过打印log验证确实没有），导致可能获取一个空页，进而导致死锁。</p>
</li>
<li>
<p>问题确认了，是BPM的并发控制没有做好。试着给BPM上一把全局锁，所有问题都消失了。</p>
</li>
</ol>
<p><img src="/images/bpm-dirty-page-eviction.png" alt="铁证" /></p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="plain"><span class="giallo-l"><span>上图：page12663没有Write page to disk就再次被请求</span></span></code></pre>
<p>问题来了，BPM出现了什么问题？见下文。</p>
<h3 id="zong-jie-3">总结</h3>
<p>这次实验真的花费了我大量的时间和精力，而其中大部份都花费在debug上。而出现的问题大多又是一些特殊情况的判断和边界细节处理上。总之就是，以后出现问题，优先考虑程序在<strong>特殊情况</strong>下的表现。最难的操作：</p>
<ul>
<li>Borrow 哪个key被借走，借到哪里，哪些key要删除</li>
<li>Merge 搞清楚哪些key被删除，哪个key被上传</li>
<li>Split 主要是leaf可能会溢出，需要特殊照顾</li>
</ul>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="cpp"><span class="giallo-l"><span> (</span><span style="color: #56B6C2;">4</span><span>,</span><span style="color: #56B6C2;">7</span><span>) </span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">2</span><span>,</span><span style="color: #56B6C2;">3</span><span>) (</span><span style="color: #56B6C2;">5</span><span>,</span><span style="color: #56B6C2;">6</span><span>) (</span><span style="color: #56B6C2;">8</span><span>,</span><span style="color: #56B6C2;">9</span><span>,</span><span style="color: #56B6C2;">10</span><span>,</span><span style="color: #56B6C2;">11</span><span>) </span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">1</span><span>) (</span><span style="color: #56B6C2;">2</span><span>) (</span><span style="color: #56B6C2;">3</span><span>) (</span><span style="color: #56B6C2;">4</span><span>) (</span><span style="color: #56B6C2;">5</span><span>) (</span><span style="color: #56B6C2;">6</span><span>) (</span><span style="color: #56B6C2;">7</span><span>) (</span><span style="color: #56B6C2;">8</span><span>) (</span><span style="color: #56B6C2;">9</span><span>) (</span><span style="color: #56B6C2;">10</span><span>) (</span><span style="color: #56B6C2;">11</span><span>,</span><span style="color: #56B6C2;">12</span><span>) </span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">></span><span> d4</span></span>
<span class="giallo-l"><span style="color: #E06C75;">></span><span> p</span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">8</span><span>,</span><span style="color: #56B6C2;">7</span><span>) </span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">2</span><span>,</span><span style="color: #56B6C2;">3</span><span>) (</span><span style="color: #56B6C2;">4</span><span>,</span><span style="color: #56B6C2;">8</span><span>) (</span><span style="color: #56B6C2;">9</span><span>,</span><span style="color: #56B6C2;">10</span><span>,</span><span style="color: #56B6C2;">11</span><span>) </span></span>
<span class="giallo-l"><span> (</span><span style="color: #56B6C2;">1</span><span>) (</span><span style="color: #56B6C2;">2</span><span>) (</span><span style="color: #56B6C2;">3</span><span>) (</span><span style="color: #56B6C2;">5</span><span>) (</span><span style="color: #56B6C2;">6</span><span>) (</span><span style="color: #56B6C2;">7</span><span>) (</span><span style="color: #56B6C2;">8</span><span>) (</span><span style="color: #56B6C2;">9</span><span>) (</span><span style="color: #56B6C2;">10</span><span>) (</span><span style="color: #56B6C2;">11</span><span>,</span><span style="color: #56B6C2;">12</span><span>) </span></span></code></pre><h3 id="leaderboard-tiao-zhan">Leaderboard 挑战</h3>
<p>虽然做完了，但是排名特别低，不是很满意，写个乐观锁吧。</p>
<p>然而实现后才发现问题并不简单。遇到了和之前类似的问题，脏page不写回！但是后来修改了一些代码，复现不了了，甚至加了Flush也不管用。</p>
<ol>
<li>BPM大锁+B+无手动FLUSH，可过Contention ： 4.39和1.46</li>
<li>BPM原样+B+手动FlushNewPage，可过Contention：5.35和1.1</li>
<li>BPM大锁+乐观锁，不可过Contention：0.62和</li>
</ol>
<p>最大疑点：BPM在Evict的时候为什么不Write脏页？</p>
<p>最后终于找到原因：虽然每个页都上锁了，但是这只能保证页在写入时之间是互不干扰的，读没有保证。例如：</p>
<blockquote>
<p><strong>线程A</strong>：evict其他某个页，然后读取page 1</p>
<p><strong>线程B</strong>：出于evict，要刷page 1脏，已经标记page#1不在buffer中</p>
<p>现在，线程B正在执行，但是还没有开始刷脏，同时A开始读取page 1。虽然他们都获取了锁，但是锁是各自的target（buffer_pool对应槽位，而不是page_id），线程A如果比B先完成Evict操作，那么A就会尝试从磁盘读page1，但是此时磁盘的page1不是最新的，因为线程B还没有刷脏完成。这就导致前面的疑点发生，事实上，并非是Evict了但是没有write，而是还没write就Read了。还没来得及打印write结束，程序就终止或死锁了。</p>
</blockquote>
<p>对于这个问题，做完所有Projects后，我现在有两个可以解决的想法：</p>
<ol>
<li>在脏页成功刷盘之前不要把<code>table_map_</code>的锁释放，这样其它线程就会在刷盘完成了以后才会去试图从磁盘FetchPage</li>
<li>参考P4的做法，为写evict这个操作配置一个请求队列，每个page_id都有一个队列，读者来了可以读尾部（最新）的page，每个page写入过程可以另起线程，并行I/O</li>
</ol>
<hr />
<p>至于为什么乐观锁出错，出错场景展示在下面。推测插入顺序是7、6、8</p>
<p><strong>猜想</strong>：<code>8</code>申请了<code>[6, 7]</code>页的写锁，而这个写锁目前正在被一个线程插入<code>9</code>。页发生分裂，变成<code>[6], [7, 9]</code>，这时<code>8</code>才获得<code>[6]</code>的写锁，但是这个页已经不是<code>8</code>应该插入的了。</p>
<p><img src="/images/optimistic-lock-error.png" alt="乐观锁错误原因" /></p>
<p>在获取写锁之前提前判断一下大小，相当于双重判断size是否符合，结果测试通过了。然后试了一下把获取写锁以后的大小判断去掉，也可以。原来是我的条件判断位置错了。</p>
<p>再次印证了那句话：<em>线程只有拿到锁以后数据才是安全的</em>。</p>
<p><strong>还原一下两种判断位置的<em>区别</em>：</strong> 如果是获取写锁才判断，很显然会发生上面提到的错误。然而，如果读锁期间判断，由于本线程还握着读锁，可能有其他线程正在等待写锁，那么这时我提前判断，就避免了因为没有马上拿到写锁而导致页不一样的问题。因为如果页会分裂，我在读锁就保证了信息全面。但是如果有两个或者以上写锁在等待呢？</p>
<p>这样就通过了，但是分数反而低了。</p>
<h2 id="project-3">Project 3</h2>
<h3 id="task-1">Task#1</h3>
<p>实现以下Executor：</p>
<ul>
<li>
<p>SeqScan</p>
<p>要注意判断是否被标记为已删除</p>
</li>
<li>
<p>Insert</p>
<p>注意插入index的key是怎么获取的（通过key_schema锁定当前index涉及的列）</p>
</li>
<li>
<p>Update</p>
<p>同样，要注意处理index，先删除后插入</p>
</li>
<li>
<p>Delete</p>
<p>Update的子集</p>
</li>
<li>
<p>IndexScan</p>
<p>通过获取此前完成的IndexIterator来获取，值得注意的是发现并完善了此前边界情况<code>End()</code>一些不太好的处理，比如初始化是<code>INVALID_PAGE_ID</code>会导致报错等等</p>
</li>
</ul>
<p>坑点：</p>
<ul>
<li>insert, update, delete操作都是一次完成，输出的tuple是发生变动的row数</li>
<li>注意首次Next必须返回True尽管可能得到的tuple是大小为0</li>
<li>注意index需要根据它的key_schema进行插入key，不能直接将整个tuple插入index</li>
</ul>
<h3 id="task-2">Task#2</h3>
<p>最耗时的就是看懂源码和调试，但是一旦理解了源码原理，上手就很快</p>
<p>完成聚合和加入：</p>
<ul>
<li>
<p>Aggregation</p>
<p>在Init的阶段把所有来自child的数据flow截断并使用hash计算和存储要求的结果，在Next迭代返回</p>
<p>头文件的实现学到了把<strong>相同hash值根据不同情况（如count每次+1、sum每次+input[i]、minmax记录最大最小）合并计算</strong>的方法</p>
</li>
<li>
<p>NestedLoopJoin</p>
<p>使用两层循环迭代来自child的tuple，值得注意的是为了保存进度，使用左右成员变量记录当前位置。如果能一次读完所有tuple一次性返回会简单很多</p>
</li>
<li>
<p>HashJoin</p>
<p>参考Aggregation实现了JoinKey和JoinValue，不友好的一点是不能直接用来调试，因为还没完成优化器实现</p>
<p>优化器实现后发现漏了哈希冲突处理，完善策略使用了简单的线性探测</p>
</li>
<li>
<p>NLJasHashJoin</p>
<p>实现将NestedJoin转换成HashJoin的优化器，凡事<code>=</code>条件的Join都能转换成HashJoin</p>
<p>这里只要求实现至多一个<code>and</code>的情况，如果要实现多个and都能处理应该写一个子程序对expression二叉树进行处理，把所有左和右（表）叶子结点（ColumnValueExpression）的列都拼接到一起，然后基于此构建一个新的HashPlanNode继承原有NestJoinPlanNode的children，特别注意ColumnExpression是左表还是右表，PlanNode倒是默认都是分好的</p>
</li>
</ul>
<h3 id="task-3">Task#3</h3>
<p>实现以下功能：</p>
<ul>
<li>
<p>Sort</p>
<p>由于是内存内的sort，直接收集（阻塞操作）所有tuple，然后对它们进行排序即可</p>
<p>排序需要实现一个自定义的排序函数，结合order_by所给的线索进行排序</p>
</li>
<li>
<p>Limit</p>
<p>Limit的实现更加直接维护一个计数器，在计数器达到限定的tuple数就可以返回false了</p>
</li>
<li>
<p>Top-N优化</p>
<p>简单来说，就是如果出现了<code>order by xxx limit n</code>的语句，不要把所有tuple保存完再统一排序，而是一边读取一边排序，同时丢弃超过容量的tuple</p>
<p>因为上面实现了HashJoin，这里的优化规则实现起来就比较熟悉了</p>
</li>
</ul>
<p>总体来说，做了Task#2再来做这里会觉得轻松很多，也有可能是因为对代码理解更深了，知道每个部份是在做什么。</p>
<h3 id="leaderboard">Leaderboard</h3>
<h4 id="q1">Q1</h4>
<p>利用Index的有序性，对where操作进行优化，包括where的列是index主键子集的情况。</p>
<p>初步分析，对于where是index列的子集的情况，可以按以下步骤：</p>
<ol>
<li>根据FilterPlan找出条件<code>condition</code></li>
<li>遍历Index，找出包含<code>condition</code>所有列的index，key列少的优先（利用率大）</li>
<li>因为index是<strong>递增</strong>的，只需要找到左侧起点（要么从头，要么从某个点）开始遍历B+树，==在遇到第一个不符合<code>predicate</code>的表明结束，返回false即可。==</li>
</ol>
<p>目测难点在于找到左侧起点</p>
<p><strong>完成后总结</strong>：</p>
<p>上面的分析基本正确，需要注意：</p>
<ul>
<li>“第一个不符合predicate就表明结束”这一结论，不适用于<code>x>=90 and y=10</code>中<code>y</code>的情况，很显然这种情况下应该返回的分布不是连续的，所以没办法提前锁定结束位置，但是以下这三种情况：<code>x>, x>=, <x, <=x</code>可以帮助我们锁定起点，跳转到最早的可用点。</li>
</ul>
<h4 id="q2">Q2</h4>
<p><strong>谓词下推</strong>，就是把在join后的Filter移动到join前，减少Join的工作量</p>
<p><img src="/images/decomposing_predicate.png" alt="decomposing_predicate" /></p>
<p>初步分析：</p>
<ol>
<li>找到Filter跟随Join的情况</li>
<li>分解Filter的predicate，把它的col分配给Join的左右子结点</li>
<li>如果子结点是Scan，直接把predicate赋值给它</li>
<li>如果子结点是Join，那么赋值给它之后，再次对它调用本函数</li>
<li>两个子结点执行完后，就应该会得到右图。注意Filter并非实际存在，而是以<code>predicate</code>的形式存在<code>PlanNode</code>中</li>
</ol>
<p><strong>完成后总结</strong>：</p>
<p>首先，弄清楚自己在做什么真的很重要。根据图来理解，事半功倍，一些实现时的细节：</p>
<ul>
<li>Join一定会在Join的左子结点，知道这一点可以简化实现时的一些操作</li>
<li>正确地判断Filter中的列是属于Join左边的那个Filter还是右边的（通过获取左表的列数判断）</li>
<li>将Join上方的Filter“肢解”给子结点后，分别继续对子结点递归调用本函数，因为子结点也有可能是Join</li>
<li>Expression树有可能会有Logic, Comparison, ColumnValue和Constant，应对策略分别如下：
<ul>
<li>Logic，子结点可能是Comparison或Constant，直接将它们分别传入本函数</li>
<li>Comparison，子结点可能是ColumnValue或Constant，对于前者，根据它的<code>ColIdx</code>决定是要把当前Comparison放到“左边”还是“右边”。Comparison本身应该被视为叶子结点</li>
<li>有时Logic的子结点会是Constant结点，什么都不用做，不要放到“左边”或者“右边”</li>
</ul>
</li>
<li>最终得到的新的predicate会是3个，分别是要给左Filter、右Filter和自己的Filter的，值得注意的是predicate有可能是空的，那么需要配置一个true Filter</li>
<li>Merge Filter操作放到这个优化规则后(<code>MergeFilterScan</code>, <code>MergeFilterNLJ</code>)</li>
</ul>
<h4 id="q3">Q3</h4>
<p>Q3的测试用例包含很多奇怪的SQL，其中穿插了一个永远为false的条件，这会导致SeqScan无意义地循环，必须提前制止。此外，还有很多无用的聚集函数，也需要优化掉。</p>
<p><strong>实现后总结</strong>：</p>
<p>要实现的有：</p>
<ul>
<li><code>IsPredicateFalse</code>检查函数</li>
<li><code>OptimizeDropFalseScan</code>替换永为false的那层executer为dummy scan（空的ValuesPlan）</li>
<li><code>FindUsefulColumnsforAgg</code>传入Expression，根据这个Expression树寻找实际需要的<code>col_id</code></li>
<li><code>DropUselessAgg</code>通过检查双层Projection，确认没有必要的聚合函数计算，只保留有需要的部份</li>
</ul>
<p>重点总结一下<code>IsPredicateFalse</code>：</p>
<ul>
<li>考虑三种结点：Logic、Comparison和ConstantValue</li>
<li>Logic只能是And或Or，只需要递归判断它子结点是否为永False并统计，结合And和Or就能知道是否为永否</li>
<li>Comparison的子结点有ColumnValue和ConstantValue两种，对应的情况有1==2, 1!=1和v1!=v1这三种永否</li>
<li>ConstantValue被认为直接反应这个Filter的predicate就是永真或永否，直接返回相反值（比如<code>ConstantValue==false</code>，说明永否，返回<code>true</code>）即可</li>
</ul>
<p>再说说<code>DropUselessAgg</code>：</p>
<ul>
<li>规则考虑的是双层Projection，最后一层是前面一层的子集</li>
<li>首先压缩两层projection为一层：把不需要的从子层去掉，把需要的提取出来替换到上层，只保留上层</li>
<li>对于此时的Projection层，<code>expressions_</code>即子集，寻找有效<code>col_id</code>，也就是在Aggregation层要保留的</li>
<li>步骤简述：
<ol>
<li>中间结点为ArithmeticExpression，叶子结点为ColumnValueExpression（当然也可能是ConstantValue，但是没有用处）</li>
<li>中间结点则递归子结点</li>
<li>遇到ColumnValue，记录这个column的<code>col_id</code></li>
</ol>
</li>
<li>得到有用的<code>col_id</code>以后，就可以通过<code>agg_plan</code>的schema中的<code>column_</code>找到有用的列，通过这些列的名称又可以找到聚合函数（agg开头e.g. <code>agg#1</code>）</li>
<li>最终利用上面的信息，就可以构建一个简化的plan了</li>
</ul>
<p>不足之处：</p>
<ul>
<li>只考虑了两层Projection的情况</li>
<li>得到最后的<code>col_id</code>等信息后，没有将它们向前“靠拢”，虽然这似乎不会导致运行错误，因为虽然数值不是连续的，但是它们都是对应的</li>
<li><code>group_by_</code>没有去掉无用的key</li>
</ul>
<p><strong>结果表现</strong>：</p>
<p>优化完成后，Q1表现最佳，排名到了13；Q2和Q3分别排33和36位。如果前面BPM的并行化I/O有实现的话应该还能再高一点。</p>
<h2 id="project-4">Project 4</h2>
<h3 id="task-1-1">Task#1</h3>
<p>实现LockManager，具体包括对Table和Row的Lock和Unlock。</p>
<p>最开始按照笔记自己写了一遍，但代码结构混乱，评测不通过。于是按照官方的函数框架重写了一遍，比较顺利地通过了测试。</p>
<p><em>最初的实现想法</em>：</p>
<p><img src="/images/first_idea_queue.png" alt="first_idea_queue" />{:height="600px" width="auto"}</p>
<p>要注意的点：</p>
<ul>
<li>按照笔记提示完成函数</li>
<li>Lock顺序简单描述如下：
<ol>
<li>事务是否能请求锁（根据隔离级别</li>
<li>请求row锁要额外检查是否已经在表上添加了对应的锁</li>
<li>检查自己是否已经持有锁，如果是低级的锁，标记升级；如果是持有更高级的锁，或者是不匹配的升级，中止事务</li>
<li>现在是否有事务在等待，是则wait；否则跳过沉睡</li>
<li>唤醒后检查是否可以获取锁，具体为先查看自己是否为下一批目标事务，然后检查当前队列已获得锁的请求与自己是否兼容</li>
<li>若都满足，走出wait循环；否则继续wait。每次唤醒后还要在循环体内检查是否已经被abort了，是的话要将自己的请求从队列中删除并<strong>广播一次唤醒其他可能在等待中且符合条件的事务</strong>，最后返回false</li>
<li>赋予事务锁，更新事务的book，修改granted为true，完成</li>
</ol>
</li>
<li>Unlock顺序简单描述如下：
<ol>
<li>检查是否真的有锁</li>
<li>根据隔离级别修改事务状态</li>
<li>table锁要额外检查是否还有row锁没释放</li>
<li>更新事务book</li>
<li>从队列中删除当前事务的请求</li>
<li>唤起下一波事务</li>
</ol>
</li>
<li>多线程编程还是那句话，只有拿到了锁之后的数据才是可信的！</li>
</ul>
<p><strong>可以改进的点</strong>：</p>
<ul>
<li>因为事务每次被唤醒会调用IsAllowedToGrant函数，这个函数的做法是遍历当前队列，检查兼容性（隔离）和优先性（FIFO）是否同时满足。对于其中的兼容性，可以通过<strong>维护一个highest变量</strong>记录当前锁的最高等级，所有不与它兼容的锁在它释放（然后更新highest）前都不能被授予。这也是我最开始的实现。但是实现可能出错了，为了确保正确性还是先用循环实现了。</li>
</ul>
<h3 id="task-2-1">Task#2</h3>
<p>死锁检测实现，还算比较好实现。简单来说就是有一个背景线程持续地根据当前的队列情况建立一个Graph，使用DFS检测环的存在，剔除<strong>最年轻</strong>的事务。</p>
<p>顺序简述：</p>
<ol>
<li>首先，通过当前所有的请求队列记录等待的<strong>边</strong>，形成Graph</li>
<li>waitings --> holdings，而且是互不兼容的锁才会形成等待边</li>
<li>获得的waits_for_ map存放了所有边，将所有事务id排序存放，从lowest开始DFS（为了test的正确性），map的second（vector）中的元素也是顺序存放的</li>
<li>找到环了以后，找出最大id的事务，然后将这个事务的请求从所有队列中删除并标记为aborted，再次找环，直到没有环了为止。</li>
<li>清除所有环后，在所有队列上尝试授予下一波事务</li>
</ol>
<p>这里卡了一段时间，倒不是因为很难，而是它有一个GetEdgeList函数要返回所有边，来检测我的正确性。我是用txn_mgr来获取所有事务id的，而test只是给waits_for_专门赋值了而已，并没有真正事务存在。</p>
<h3 id="task-3-1">Task#3</h3>
<p>逻辑并发控制！简单来说，就是Task#1完成了锁的功能，现在要来用这些锁了。要求的其实很简单，只需要满足SeqScan、Insert和Delete这三个executor就可以。</p>
<ul>
<li>
<p>SeqScan：最主要的。有一个IsDelete()的Context上下文函数，提示是否需要持<strong>排他</strong>的锁。</p>
<ul>
<li>对于Table（<code>Init</code>函数），拿到oid后，先检查是否必要加锁，因为可能已经有了更高级的锁。</li>
<li>对于Row（<code>Next</code>函数），拿到rid后，同样先检查是否必要加锁。此外，因为实现了predicate push down，如果这个row不符合条件或者是已经被删除的，要再将它Force Unlock掉，因为事务不需要这个row。</li>
<li>最后，根据隔离级别选择性的释放S锁</li>
</ul>
</li>
<li>
<p>Insert：</p>
<ul>
<li>在<code>Init</code>函数对表上IX锁（？）</li>
<li>在<code>Next</code>函数记录加入的row到write_set，以便后续对abort事务进行回滚</li>
</ul>
</li>
<li>
<p>Delete：</p>
<ul>
<li>如果在SeqScan是通过IsDelete()判断并提前上了排他锁，这里在<code>Init</code>函数就不需要上锁了</li>
<li>同样在<code>Next</code>函数记录删除的row，以便回滚</li>
</ul>
</li>
<li>
<p>一定是要拿到锁，再Fetch资源！</p>
</li>
</ul>
<p>然后就是两个事务终结函数：Commit和Abort</p>
<ul>
<li>Commit：没什么要做的，只需要把事务所有持有的锁释放即可（根据book）</li>
<li>Abort：根据之前记录wrtie_set_，对其元素倒序执行，就可以实现回滚。</li>
</ul>
<h3 id="leatherboard">Leatherboard</h3>
<h4 id="hint1">Hint1</h4>
<p>Predicate pushdown to SeqScan，这个在P3就实现了，提交后比最低有3到5的提升。好处是减少SeqScan上传的元组数，进而减少上锁的row数。</p>
<h4 id="hint2">Hint2</h4>
<p>Implement In-Place UpdateExecutor，因为之前UPDATE操作是用DELETE+INSERT实现的，如果update一步到位，想必会快很多吧！但是这个要改的话涉及到B+树的原地操作，直接贯穿了3个Project！而且我第一时间也没把握能顺利完成。遂遗憾放弃。</p>
<h4 id="hint3">Hint3</h4>
<p>Use Index，好巧，这个我在P3也实现了。但是这里有一点不同：因为key不会重复，所以如果predicate是<code>CompareType::Equal</code>的话（e.g. 只有一个=），就可以<strong>获取一个tuple（若有）后确信后面再没有匹配的row，可以提前退出</strong>。但是可惜的是，运行bench的时候出现了死锁。而且是latch死锁而不是lock死锁。</p>
<p>现象有二：</p>
<ol>
<li>DeleteExecutor在执行DeleteEntry的时候Fetch B+树的根结点页发生死锁，理论上不应该发生的，怀疑是线程两次获取了这个页面。</li>
<li>注释掉DelteExecutor关于更新Index的操作后，程序在调试时不会发生死锁（快速点击其实还是会），在直接运行下会发生死锁，怀疑在别处还有死锁。</li>
</ol>
<p>原因推测：</p>
<ol>
<li>线程在运行到索引的Remove函数必定发生死锁，很大概率是这个线程获取两次根结点的问题。</li>
<li>一开始以为是在IndexScan，树的Iter在最底部从左往右走；而在Delete和Insert，锁从上到下，获取锁的方向不一致（写到一半，发现这个不是原因。因为基于Hint3本身的实现，iter不会向右移动，但仍然会出现死锁）说明程序在别的地方还有其它重复两次获取同样的锁</li>
</ol>
<h3 id="zong-jie-4">总结</h3>
<p>至此，所有Project都已经完成。感谢Andy将这门课程公开，让我有机会接触到在本校无法学到的知识；感谢TA们打造了bustub并在discord上解答非CMU学生的问题；也感谢自己的坚持。</p>
<p>一些<strong>可改进</strong>的点集合：</p>
<ul>
<li>BPM并行I/O实现</li>
<li>B+树结点可在满size前借给邻居，减少连续地分裂</li>
<li>执行计划还可以有更多优化</li>
<li>Update操作原地进行</li>
</ul>
Notes of Git
2023-12-01T00:00:00+00:00
2023-12-01T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/notes-of-git/
<h1 id="notes-of-git">Notes of Git</h1>
<h2 id="basic-settings">Basic Settings</h2>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #B57EDC;">git</span><span style="color: #98C379;"> config</span><span style="color: #56B6C2;"> --global</span><span style="color: #98C379;"> user.name</span><span> "</span><span style="color: #98C379;">kvn</span><span>"</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">git</span><span style="color: #98C379;"> config</span><span style="color: #56B6C2;"> --global</span><span style="color: #98C379;"> user.email username@email.com</span></span>
<span class="giallo-l"><span style="color: #B57EDC;">git</span><span style="color: #98C379;"> config</span><span style="color: #56B6C2;"> --global</span><span style="color: #98C379;"> core.autocrlf input</span><span style="color: #5F6672;font-style: italic;"> # for MacOS</span></span></code></pre><h2 id="appending-operations">Appending Operations</h2>
<ul>
<li>Showing files in the staging area by <code>git ls-files</code></li>
<li>Storing files that are <code>git add</code> .</li>
<li>Modified files will have to be <code>git add</code> again</li>
</ul>
<p>If <code>git status</code> gives green info, it means it's all the same for <strong>Working Directory</strong> and <strong>Staging Area</strong>. If it says committed, then <strong>Repository</strong> too.</p>
<p><em><strong>Skip Staging</strong></em>:</p>
<p>use <code>git commit -am "info"</code> for adding</p>
<p>use <code>git rm filename1...</code> for deleting</p>
<p>use <code>git mv file file1</code> for renaming or moving</p>
<p><strong>Only do this when you know what you are doing</strong></p>
<h2 id="ignoring-files">Ignoring Files</h2>
<p>Files like logs usually shouldn't be tracked.</p>
<p>use .gitignore file to book files unneeded in the following form:</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="plain"><span class="giallo-l"><span>logs/</span></span>
<span class="giallo-l"><span>main.log</span></span>
<span class="giallo-l"><span>*.log</span></span></code></pre>
<p>If files are already in repo, adding them to .gitignore file is not going to let git ignore them. To solve this, we use <code>git rm --cached dir_dont_want_to_track</code> to only remove it from the staging area(may have to use <code>-r</code> to recursively <code>rm</code>).</p>
<h2 id="viewing-operation">Viewing Operation</h2>
<h3 id="simple-view">Simple View</h3>
<p>use <code>git status -s</code> for simplified info.</p>
<p><code>M</code> for modified; <code>A</code> for added.</p>
<h3 id="see-diff">See Diff</h3>
<p>use <code>git diff --staged</code> to compare files changed between the last committed and currently staged.</p>
<p>use <code>git diff</code> to compare files changed between the currently staged and working directory.</p>
<h3 id="diff-tool">Diff Tool</h3>
<p><strong>Set up</strong>:</p>
<p><code>git config --global diff.tool vscode</code></p>
<p><code>git config --global difftool.vscode.cmd "code --wait --diff $LOCAL $REMOTE"</code></p>
<p>check by:</p>
<p><code>git config --global -e</code></p>
<h3 id="history">History</h3>
<p>use <code>git log</code> to see history.</p>
<p><strong>Options</strong>:</p>
<ul>
<li>
<p><code>--oneline</code>: simplified output, ID & comment</p>
</li>
<li>
<p><code>--reverse</code>: presented in the way of bottom up</p>
</li>
</ul>
<p>use <code>git show ID</code> or <code>git show HEAD~step_before</code> to see the diff of the commits</p>
<p>use <code>git show ID:file</code> to see the file of the commit <code>ID</code></p>
<p>use <code>git ls-tree ID</code> to list all files with their <code>unique_id</code> of version <code>ID</code></p>
<p>use <code>git show unique_id</code> to see <code>blob</code> or <code>tree</code> by their <code>unique_id</code></p>
<h2 id="restoring-files">Restoring Files</h2>
<p>use <code>git restore --staged file</code> to restore files from staging area to the working directory</p>
<ul>
<li>
<p>when restoring modified files, git uses copies from the repo to replace files in the staging area.</p>
</li>
<li>
<p>when restoring added files, git reverts the stage of files back to untracked</p>
</li>
</ul>
<p>use <code>git restore file</code> to restore files from the working directory to the previous version</p>
<ul>
<li>when restoring modified files, git replaces them with the latest copies from the repo</li>
<li>git doesn't restore untracked files, if no <strong>Options</strong> like <code>-fd</code> are given, the move is considered dangerous and will not process.</li>
</ul>
<p>use <code>git restore --source=HEAD~1 file</code> to restore files to the version <code>ID</code></p>
<h2 id="remote">Remote</h2>
<p>use <code>git remote add name url</code> to add new remote repo</p>
<p>use <code>git pull</code> to directly pull the repo to the working directory(space)</p>
<p>use <code>git fetch</code> to fetch the repo to the local repo</p>
<p>use <code>git push</code> to push the local repo to the remote</p>
<h2 id="branches">Branches</h2>
<h3 id="basics">Basics</h3>
<p>use <code>git branch name</code> to create a new branch from the current branch</p>
<p>use <code>git branch switch name</code> to switch HEAD to branch name</p>
<h3 id="merge-or-rebase">Merge or Rebase</h3>
<p>The following graph presents 3 different ways of merging branches.</p>
<p><img src="/images/git-merge-rebase.png" alt="Git merge and rebase comparison" />{:height="400px" width="auto"}</p>
<p>use <code>git merge feature</code> to merge the feature branch to the master when HEAD -> master</p>
<p>use <code>git rebase feature</code> to merge the branch in terms of rebase to the master</p>
<h3 id="dealing-conflicts">Dealing Conflicts</h3>
<p>when two branches change the same line of the same file and attend to merge, one has to decide which version to stay.</p>
Notes of MySQL
2023-07-13T00:00:00+00:00
2023-07-13T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/notes-of-mysql/
<h1 id="mysql">MySQL</h1>
<h2 id="sql">SQL</h2>
<h3 id="ddl">DDL</h3>
<p>数据定义语言 Data Definition Language</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="plain"><span class="giallo-l"><span>show databases;</span></span>
<span class="giallo-l"><span>select database0;</span></span>
<span class="giallo-l"><span>create database [if not exists] base_name [default charset] [collate];</span></span>
<span class="giallo-l"><span>drop database [if exists] base_name;</span></span>
<span class="giallo-l"><span>use base_name;</span></span>
<span class="giallo-l"><span>#************************#</span></span>
<span class="giallo-l"><span>show tables;</span></span>
<span class="giallo-l"><span>desc table_name;</span></span>
<span class="giallo-l"><span>show create table;</span></span>
<span class="giallo-l"><span>create table table_name(</span></span>
<span class="giallo-l"><span> field_1 field_type [comment cmt],</span></span>
<span class="giallo-l"><span> ...</span></span>
<span class="giallo-l"><span>)[comment cmt];</span></span>
<span class="giallo-l"><span>#************************#</span></span>
<span class="giallo-l"><span>alter table table_name modify target_field new_type(len);</span></span>
<span class="giallo-l"><span>alter table table_name change old_field new_field new_type(len) [comment cmt] [restrain];</span></span>
<span class="giallo-l"><span>alter table table_name drop field;</span></span>
<span class="giallo-l"><span>alter table table_name rename to new_table_name;</span></span>
<span class="giallo-l"><span>drop table [if exists] table_name;#删目录</span></span>
<span class="giallo-l"><span>truncate table table_name;#删内容</span></span></code></pre><h3 id="dml">DML</h3>
<p>数据操作语言 Data Manipulation Language</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="plain"><span class="giallo-l"><span>insert into table_name(field1,...) values(val1,...);</span></span>
<span class="giallo-l"><span>insert into table_name values(val...);</span></span>
<span class="giallo-l"><span>insert into table_name(field1,...) values(val1,...),...;</span></span>
<span class="giallo-l"><span>insert into table_name values(val...),...;</span></span>
<span class="giallo-l"><span>#************************#</span></span>
<span class="giallo-l"><span>update table_name set field1=val1,... [where condition];</span></span>
<span class="giallo-l"><span>#************************#</span></span>
<span class="giallo-l"><span>delete from table_name [where condition];</span></span></code></pre><h3 id="dql">DQL</h3>
<p>数据查询语言 Data Query Language</p>
<ul>
<li>基本查询</li>
<li>条件查询(where)</li>
<li>聚合查询(count, max, min, avg, sum)</li>
<li>分组查询(group by)</li>
<li>排序查询(order by)</li>
<li>分页查询(limit)</li>
</ul>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="plain"><span class="giallo-l"><span>select field1,... from table_name;</span></span>
<span class="giallo-l"><span>select * from table_name;</span></span>
<span class="giallo-l"><span>select field [[as] niknme],... from table_name;</span></span>
<span class="giallo-l"><span>select distinct field from table_name;</span></span>
<span class="giallo-l"><span>#************************#</span></span>
<span class="giallo-l"><span>select ... from table_name where condition;</span></span>
<span class="giallo-l"><span>select ... from table_name in (val1,...);</span></span>
<span class="giallo-l"><span>select ... from table_name between ... and ...;#[]</span></span>
<span class="giallo-l"><span>select ... from table_name like '_?' or '%?';</span></span>
<span class="giallo-l"><span>#************************#</span></span>
<span class="giallo-l"><span>select ... from ... [where con] group by evidence_field [hanving con];</span></span>
<span class="giallo-l"><span>select ... from ... order by field1 asc_desc,...;</span></span>
<span class="giallo-l"><span>#************************#</span></span>
<span class="giallo-l"><span>select ... from ... limit start_index, following_amount;</span></span></code></pre><h3 id="dcl">DCL</h3>
<p>数据控制语言 Data Control Language</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="plain"><span class="giallo-l"><span>use mysql;</span></span>
<span class="giallo-l"><span>select * from user;</span></span>
<span class="giallo-l"><span>create user 'username'@'host' identified by 'pwd';</span></span>
<span class="giallo-l"><span>alter user 'username'@'host' identified with old_pwd by 'new_one';</span></span>
<span class="giallo-l"><span>drop user 'username'@'host';</span></span>
<span class="giallo-l"><span>#************************#</span></span>
<span class="giallo-l"><span>show grants for 'username'@'host';</span></span>
<span class="giallo-l"><span>grant grant_list on base.table to 'username'@'host';</span></span>
<span class="giallo-l"><span>revoke grant_list on base.table from 'username'@'host';</span></span></code></pre><h2 id="han-shu">函数</h2>
<p>字符串函数</p>
<ul>
<li>
<p>Concat(s1,...)</p>
</li>
<li>
<p>Lower(str)</p>
</li>
<li>
<p>Upper(str)</p>
</li>
<li>
<p>LPad(str,n,pad)</p>
</li>
<li>
<p>RPad(str,n,pad)</p>
</li>
<li>
<p>Trim(str)</p>
</li>
<li>
<p>Substring(str, start,len)</p>
</li>
</ul>
<p>数值函数</p>
<ul>
<li>Ceil(x)</li>
<li>Floor(x)</li>
<li>Mod(x,y) x/y的模</li>
<li>Rand()</li>
<li>Round(x,y) 四舍五入，保留y位小数</li>
</ul>
<p>日期函数</p>
<ul>
<li>Curdate()</li>
<li>Curtime()</li>
<li>Now()</li>
<li>Year(date)</li>
<li>Month(date)</li>
<li>Day(date)</li>
<li>Date_add(date,INTERVAL expr type)</li>
<li>Datediff(date1,date2)</li>
</ul>
<p>流程函数</p>
<ul>
<li>if(value t, f)</li>
<li>Ifnull(value1, value2)</li>
<li>Case when [val1] then [res1] ... else [default] end</li>
<li>Case [expr] then [res1] ... else [default] end</li>
</ul>
<h2 id="yue-shu">约束</h2>
<ol>
<li>非空 not null</li>
<li>唯一 unique</li>
<li>主键 primary key (AUTO_INCREMENT)</li>
<li>默认 default</li>
<li>检查 check</li>
<li>外键 foreign key</li>
</ol>
<h2 id="duo-biao-cha-xun">多表查询</h2>
<p>关系</p>
<ol>
<li>一对多：多的一方设置外键，关联另一方主键</li>
<li>多对多：建立中间表，中间表包含两个外键，关联两张表主键</li>
<li>一对一：用于表结构拆分，任意表中设置唯一外键，关联另一表主键</li>
</ol>
<p>查询</p>
<ol>
<li>
<p>内连接</p>
<ul>
<li>隐式：select ... from A,B where ...</li>
<li>显式：select ... from A inner join b on ...</li>
</ul>
</li>
<li>
<p>外连接</p>
<ul>
<li>左外：select ... from A left join B on ...</li>
<li>右外：select ... from A right join B on ...</li>
</ul>
</li>
<li>
<p>自连接</p>
<p>Select ... from A a1,A a2 where ...</p>
</li>
<li>
<p>子查询</p>
<p>标量子查询、列子查询、行子查询、表子查询</p>
</li>
</ol>
<h2 id="shi-wu">事务</h2>
<p>@@autocommit=0 手动提交事务</p>
<p>commit 提交事务</p>
<p>rollback 回滚</p>
<p>事务的四大特性</p>
<ul>
<li>原子性 Atomicity</li>
<li>一致性 Consistency</li>
<li>隔离性 Isolation</li>
<li>持久性 Durability</li>
</ul>
<p>事务并发存在的问题</p>
<ol>
<li>脏读：读另一个事务没有提交的数据</li>
<li>不可重复读：先后读一个记录，但是数据不同（期间有其他事务提交）</li>
<li>幻读：无法插入</li>
</ol>
<p>隔离级别</p>
<ol>
<li>Read uncommitted</li>
<li>Read commited</li>
<li>Repeatable Read</li>
<li>Serializable</li>
</ol>
<p>Select @@transaction_isolation</p>
<p>set [session|global] transaction isolation level {level1234}</p>
<h2 id="suo">锁</h2>
<h3 id="quan-ju-suo">全局锁</h3>
<p>加锁后，整个数据库实例就处于只读状态，典型场景：全库逻辑备份</p>
<p><code>flush tables with read lock;</code></p>
<p><code>mysqldump -uroot -p密码 basename > base.sql</code> 终端执行</p>
<p><code>unlock tables</code></p>
<ol>
<li>备份期间，所有业务停摆</li>
<li>如果在库上备份，那么在备份期间不能从库执行主库同步过来的二进制日志会导致主从延迟</li>
</ol>
<p>在InnoDB引擎中，可以在备份时加上参数--single-transaction来完成不加锁的一致性数据备份</p>
<h3 id="biao-ji-suo">表级锁</h3>
<h4 id="biao-suo">表锁</h4>
<ul>
<li>表共享读锁 read lock</li>
<li>表独占写锁 write lock</li>
</ul>
<p><code>lock tables name read/write</code></p>
<p><code>unlock tables </code></p>
<p>读锁阻塞写，写锁阻塞所有操作。</p>
<h4 id="yuan-shu-ju-suo-meta-data-lock-mdl">元数据锁 meta data lock, MDL</h4>
<p>由系统自动控制，无需显式使用</p>
<p>事务A，B开启，操作对象表t</p>
<ol>
<li>A select t</li>
<li>B select lock_info</li>
</ol>
<p><img src="/images/table_lock_1.png" alt="Screenshot 2023-07-12 at 10.19.00" /></p>
<ol start="3">
<li>B insert bob into employee</li>
</ol>
<p><img src="/images/table_lock_2.png" alt="image-20230712102406447" /></p>
<ol start="4">
<li>A select * from employee</li>
</ol>
<p><img src="/images/table_lock_3.png" alt="Screenshot 2023-07-12 at 10.25.28" /></p>
<ol start="5">
<li>A insert id3</li>
<li>B alter id to primary key (blocked) exclusive与其他不兼容</li>
<li>A commit;</li>
<li>B throws error, 'cause A insert id3, primary key cannot be add</li>
</ol>
<h4 id="yi-xiang-suo">意向锁</h4>
<ul>
<li>意向共享锁 IS <code>select ... lock in share mode</code>添加</li>
</ul>
<p> 与表锁共享锁兼容（read），与表锁排他锁（write）互斥</p>
<ul>
<li>意向排他锁 IX <code>insert、update、delete、select ... for update</code>添加</li>
</ul>
<p> 与所有表锁互斥，意向锁之间不排斥</p>
<p>当表已经存在意向锁时试图加互斥表锁，会导致加表锁的事务线程阻塞，直到意向锁解除</p>
<h3 id="xing-ji-suo">行级锁</h3>
<h4 id="xing-suo-record-lock">行锁 Record Lock</h4>
<p>锁定单行的纪录，防止其他事务update和delete</p>
<p>共享锁 S 排他锁 X 与之前类似<img src="/images/Record_lock.png" alt="Screenshot 2023-07-12 at 10.51.57" /></p>
<p>InnoDB使用next-key锁进行搜索和索引扫描，以防止幻读</p>
<ol>
<li>针对唯一索引进行检索时，对已经存在的纪录进行等值匹配时，将自动优化为行锁</li>
<li>InnoDB的行锁是针对于索引加的锁，不通过索引条件检索数据，那么InnoDB将对所有记录加锁，此时升级为<strong>表锁</strong>。（SQL优化update）</li>
</ol>
<h4 id="jian-xi-suo-gaplock">间隙锁 GapLock</h4>
<p>锁定索引记录的间隙，不包含行记录，确保间隙不变，防止其他事务insert，不同事务的间隙锁可以兼容</p>
<p>InnoDB使用next-key锁扫描，</p>
<ol>
<li>索引上的等值查询（唯一索引），给不存在的纪录加锁时优化为间隙锁</li>
</ol>
<table><thead><tr><th style="text-align: center">id</th><th style="text-align: center">name</th><th style="text-align: center">age</th></tr></thead><tbody>
<tr><td style="text-align: center">3</td><td style="text-align: center">PHP</td><td style="text-align: center">3</td></tr>
<tr><td style="text-align: center">8</td><td style="text-align: center">Java</td><td style="text-align: center">4</td></tr>
</tbody></table>
<p>A：<code>update table set age = 10 where id = 5;</code></p>
<p>B：<code>insert into table values(7,Python,7);</code> (blocked)<img src="/images/gap_lock.png" alt="Screenshot 2023-07-14 at 10.59.57" /></p>
<h4 id="lin-jian-suo-next-key-lock">临键锁 Next-Key Lock</h4>
<p>行锁和间隙锁的结合，同时锁住数据和间隙</p>
<ol start="2">
<li>索引上的等值查询（普通索引），遍历到最后一个值不满足查询需求时，next-key lock退化为间隙锁</li>
<li>索引上的范围查询（唯一索引），会访问到不满足的条件的第一个值为止</li>
</ol>
<p>锁的范围表示，间隙锁范围左开右闭。</p>
<h2 id="innodbyin-qing">InnoDB引擎</h2>
<h3 id="luo-ji-cun-chu-jie-gou">逻辑存储结构</h3>
<p><img src="/images/storing_structure.png" alt="点击查看图片来源" /></p>
<h4 id="biao-kong-jian-ibd">表空间 ibd</h4>
<p>一个mysql实例可以对应多个表空间，用于存储记录、索引等数据</p>
<h4 id="duan-segment">段 segment</h4>
<ul>
<li>Leaf node segment 数据段</li>
<li>Non-leaf node segment 索引段</li>
<li>Rollback segment 回滚段</li>
</ul>
<p>InnoDB是索引组织表，数据段就是B+树的叶子结点，索引段为非叶子结点。段用来管理多个Extent</p>
<h4 id="qu-extent">区 extent</h4>
<p>表空间的单元结构，每个区大小为1M。默认下InnoDB存储引擎页大小为16K，即一个区中一共有64个连续的页</p>
<h4 id="ye-page">页 page</h4>
<p>InnoDB存储引擎管理磁盘的最小单元，每页大小默认16KB。为保证页的连续性，InnoDB每次从磁盘申请4-5个区</p>
<h4 id="xing-row">行 row</h4>
<p>InnoDB数据是按行存放的</p>
<ul>
<li>Trx_id 每次对某条记录进行改动时，都会把对应的事务id赋值给trx_id隐藏列。</li>
<li>Roll_pointer 每次对某条记录进行改动时，都会把旧的版本写入到undo日志中，如何隐藏列就相当于一个指针可以用来找到该记录修改前的信息</li>
</ul>
<h3 id="jia-gou">架构</h3>
<p><img src="/images/db_structure.png" alt="Screenshot 2023-07-17 at 08.56.06" /></p>
<h4 id="nei-cun-jie-gou">内存结构</h4>
<ul>
<li>缓冲池 主内存的一个区域
<ul>
<li>单位 Page
<ul>
<li>free 空闲未被使用</li>
<li>clean 被使用但没修改过</li>
<li>dirty 数据被修改过</li>
</ul>
</li>
</ul>
</li>
<li>自适应哈希索引
<ul>
<li>用于优化对缓冲池数据的查询，InnoDB会监控对表上各索引页的查询，如果观察到hash索引可以提升速度，则建立hash索引</li>
</ul>
</li>
<li>更改缓冲区
<ul>
<li>存储变更的数据，之后再同步到缓冲池，最后刷新到磁盘</li>
</ul>
</li>
<li>日志缓冲区
<ul>
<li>保存要写入到磁盘中的log日志数据（redo log、undo log），默认大小16MB，日志缓冲区的日志回定期刷新到磁盘中。如果需要更新、插入或删除许多行的事务，增加日志缓冲区的大小可以节省磁盘I/O</li>
<li>innodb_log_buffer_size 缓冲区大小</li>
<li>innodb_flush_log_at_trx_commit 刷新到磁盘时机
<ul>
<li>1 日志每次事务提交时写入并刷新到磁盘</li>
<li>0 每秒写入并刷新到磁盘一次</li>
<li>2 日志在每次事务提交后写入并每秒刷新到磁盘一次？？？？</li>
</ul>
</li>
</ul>
</li>
</ul>
<h4 id="ci-pan-jie-gou">磁盘结构</h4>
<ul>
<li>System Tablespace
<ul>
<li>系统表空间，更改缓冲区的存储区域</li>
</ul>
</li>
<li>File-Per-Table Tablespaces
<ul>
<li>独立表空间，每个表的文件表空间包含单个InnoDB表的数据和索引，并存储在文件系统上的单个数据文件中</li>
</ul>
</li>
<li>General Tablespace
<ul>
<li>通用表空间，需要通过create tablespace创建，在创建表时，可以指定该表空间 e.g. Create tablespace xxx add datafile xxx engine = engine_name</li>
</ul>
</li>
<li>Undo Tablespaces
<ul>
<li>撤销表空间，MySQL实例在初始化时会自动创建两个默认的undo表空间（初始大小16M），用于存储undo日志</li>
</ul>
</li>
<li>Temporary Tablespaces
<ul>
<li>临时表空间，使用临时表空间和全局临时表空间存储用户创建的临时表</li>
</ul>
</li>
<li>Doublewrite Buffer Files
<ul>
<li>双写缓冲区，innoDB将数据页从缓冲池刷新到磁盘前，先将数据页写入双写缓冲区文件中，便于系统异常时恢复数据</li>
</ul>
</li>
<li>Redo Log
<ul>
<li>重做日志，用来实现事务持久性，该日志文件由两部分组成：重做日志缓冲（redo log buffer）以及重做日志文件（redo log），前者在内存中，后者在磁盘中。当事务提交之后会把所有修改信息都存到该日志中，用于刷新脏页到磁盘中、发生错误时，进行数据恢复使用</li>
</ul>
</li>
</ul>
<h4 id="hou-tai-xian-cheng">后台线程</h4>
<ol>
<li>
<p>Master Thread</p>
<p>核心后台线程，负责调度其他线程，还负责将缓冲池的数据异步刷新到磁盘中，保持数据一致性，包括脏页刷新、合并插入缓存、undo页的回收</p>
</li>
<li>
<p>IO Thread</p>
<p>在InnoDB中大量使用了AIO来处理IO请求，这样可以极大地提高数据库性能，而IO Thread主要负责这些IO请求的回调</p>
<ul>
<li>Read thread 4</li>
<li>Write thread 4</li>
<li>Log thread 1</li>
<li>Insert buffer thread 1</li>
</ul>
</li>
<li>
<p>Purge Thread</p>
<p>主要用于回收事务已经提交了的undo log</p>
</li>
<li>
<p>Page Cleaner Thread</p>
<p>协助Master Thread刷新脏页到磁盘的线程，可以减轻Master Thread的工作压力，减少阻塞</p>
</li>
</ol>
<h4 id="shi-wu-yuan-li">事务原理</h4>
<ul>
<li>事务是一组操作的集合，是不可分割的工作单位，这些操作要么同时成功要么同时失败。</li>
<li>特性ACID</li>
</ul>
<p>特性的实现</p>
<p><strong>redo log和undo log</strong></p>
<ul>
<li>原子性</li>
<li>一致性</li>
<li>持久性</li>
</ul>
<p><strong>锁和MVCC</strong></p>
<ul>
<li>隔离性</li>
</ul>
<h5 id="redo-log">redo log</h5>
<p>重做日志，记录事务提交时数据页的物理修改，用于实现事务的持久性。</p>
<p>文件由两部分组成，重做日志缓冲和重做日志文件。当事务提交后，会把所有修改信息都存到该日志文件中，用于在刷新脏页到磁盘<strong>发生错误</strong>时进行数据恢复</p>
<ul>
<li>WAL write-ahead logging</li>
</ul>
<h5 id="undo-log">undo log</h5>
<p>回滚日志，记录数据被修改前的信息，作用包括提供回滚和MVCC</p>
<p>undo log不同于redo log，属于逻辑日志，其中记录的是实际操作的反向操作，如delete操作会使文件中增加insert记录。</p>
<ul>
<li>undo log销毁：事务执行时产生，事务提交时不会立即删除undo log，因为这些日志可能还用于MVCC</li>
<li>undo log存储：undo log采用段的方式进行管理和记录，存放在rollback segment回滚段中，内部包含1024个undo log segment</li>
</ul>
<h5 id="mvcc">MVCC</h5>
<p>Multi-Version Concurrency Control</p>
<ul>
<li>
<p>当前读</p>
<p>读取的是记录的最新版本，读取时还要保证其他并发事务不能修改当前记录，会对当前读取的记录进行加锁。select...lock in share mode, selcet... for update, update, insert, delete都是当前读</p>
</li>
<li>
<p>快照读</p>
<p>简单的select（不加锁）就是快照读，读到的是数据的历史版本，有可能是历史数据，不加锁是非阻塞读</p>
<ul>
<li>Read Committed 每次select都生成一个快照读</li>
<li>Repeatable Read 开启事务后第一个select语句才是快照读的地方</li>
<li>Serializable 快照读退化为当前读</li>
</ul>
</li>
<li>
<p>MVCC——多版本并发控制，指维护一个数据的多个版本，使得读写操作没有冲突，快照读为MySQL实现MVCC提供了一个非阻塞读功能。MVCC具体实现依赖数据库记录中的<strong>三个隐式字段</strong>、<strong>undo log</strong>、<strong>realView</strong></p>
</li>
</ul>
<h6 id="yin-cang-zi-duan">隐藏字段</h6>
<ol>
<li>
<p>DB_TRX_ID</p>
<p>最近修改事务ID，记录插入这条记录或最后一次修改改记录的事务ID</p>
</li>
<li>
<p>DB_ROLL_PTR</p>
<p>回滚指针，指向这条记录的上一个版本，用于配合undo log指向上一版本</p>
</li>
<li>
<p>DB_ROW_ID</p>
<p>隐式主键，表结构本身没有主键会生成</p>
</li>
</ol>
<h6 id="undo-log-1">undo log</h6>
<p>回滚日志，当insert的时候，产生的undo log只在回滚时需要，事务提交后可被立即删除；而update、delete时，产生的undo log不仅在回滚时需要，在快照读时也需要，不会被立即删除。</p>
<ul>
<li>
<p>undo log版本链</p>
<p><img src="/images/undolog_chain.png" alt="Screenshot 2023-07-13 at 11.08.59" /></p>
</li>
</ul>
<h6 id="readview">readview</h6>
<p>readview（读视图）是快照读执行时MVCC提取数据的依据，记录并维护系统当前活跃的事务（未提交）的id</p>
<p>readview包含四个核心字段：</p>
<ul>
<li>m_ids 当前活跃事务id集合</li>
<li>min_trx_id 最小活跃事务ID</li>
<li>max_trx_id 预分配事务ID，即当前最大事务ID+1</li>
<li>creator_trx_id readview创建者的事务ID</li>
</ul>
<p>版本链数据访问规则：trx_id代表当前版本事务ID</p>
<ol>
<li>trx_id == creator_trx_id 可以访问 （说明数据是当前事务更改的</li>
<li>trx_id < min_trx_id 可以访问 （说明数据已经提交了</li>
<li>trx_id > max_trx_id 不可以访问 （说明事务是在readview生成后才开启</li>
<li>min_trx_id <= trx_id <= max_trx_id 若trx_id不在m_ids中是可以访问的 （说明数据已经提交</li>
</ol>
<p>不同的隔离级别，生成readview的时机不同：</p>
<ul>
<li>Read Committed 在事务中每次执行快照读时生成readview</li>
<li>Repeatable Read 仅在事务中第一次执行快照读时生成readview，后续复用该readview</li>
</ul>
<p>结合<strong>undo log</strong>例子和<strong>readview</strong>规则理解</p>
Notes of Rust
2023-04-24T00:00:00+00:00
2023-04-24T00:00:00+00:00
Unknown
https://kev1n8.github.io/posts/notes-of-rust/
<h2 id="ke-bian-he-bu-ke-bian-bian-liang">可变和不可变变量</h2>
<p><code>let mut, let </code></p>
<p><code>const</code> 与 <code>let</code>无<code>mut</code>区别在于，<code>let</code>有<code>ownership</code>，而<code>const</code>随便使用</p>
<h2 id="shu-ju-lei-xing">数据类型</h2>
<h3 id="scalar"><code>Scalar</code></h3>
<p><code>integer, float, boolean, character</code></p>
<h3 id="compound"><code>Compound</code></h3>
<p><code>tuple., array[]</code></p>
<h2 id="han-shu">函数</h2>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> func_name</span><span>(</span><span style="color: #C6CCD7;">para</span><span style="color: #E06C75;">:</span><span style="color: #61AFEF;">type</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #C6CCD7;"> return_type</span><span>{}</span></span></code></pre><h2 id="zhu-shi">注释</h2>
<h2 id="kong-zhi-liu">控制流</h2>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">if</span><span>,</span><span style="color: #E06C75;">else if</span><span>,</span><span style="color: #E06C75;">else</span><span>,</span><span style="color: #E06C75;">loop</span><span>,</span><span style="color: #E06C75;">break</span><span>,</span><span style="color: #E06C75;">while</span><span>,</span><span style="color: #E06C75;">for</span><span>,</span><span style="color: #E06C75;">..</span><span>(</span><span style="color: #E06C75;">=</span><span>)</span></span></code></pre><h2 id="suo-you-quan">所有权</h2>
<p>会转让或释放所有权的情况（储存堆上的变量）：
=移动、函数传参数、所在域结束
禁止发生的情况：
在当前域有两个只向同一变量的可变引用、
在当前域内同时有可变引用和不可变引用并且在可变引用后再次使用了之前的不可变引用、
代码块返回一个生命已经终结的引用、
可变引用会触发<code>borrowing</code>，引用指向的变量失去改变自己的权利，在引用死亡前禁止变量改变自己、
引用指向变量生命必须比引用本身长</p>
<h2 id="jie-gou">结构</h2>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> Struct_name</span><span>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> data_name</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;">type_name</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Struct_name</span><span>{</span><span style="color: #5F6672;font-style: italic;">//赋值};</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Struct_name</span><span>{</span><span style="color: #E06C75;">..</span><span style="color: #C6CCD7;">struct1</span><span>};</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> build_struct</span><span>(</span><span style="color: #C6CCD7;">name</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;">type_name</span><span>)</span><span style="color: #E06C75;">-></span><span>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//note that all operation above takes the</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//ownership from the parameters</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> Point</span><span>(</span><span style="color: #E5C07B;">i32</span><span>,</span><span style="color: #E5C07B;">i32</span><span>,</span><span style="color: #E5C07B;">i32</span><span>);</span></span></code></pre>
<p>实现结构内的方法：<code>impl,self</code></p>
<h2 id="mei-ju">枚举</h2>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> Kind</span><span>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> k1</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> k2</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>访问方式::</p>
<p><code>varient</code>可以附带有数据类型<code>()</code>互相可以不一样。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> Message</span><span>{</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Quit</span><span>,</span><span style="color: #5F6672;font-style: italic;">//no data associated</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Move</span><span>{</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">i32</span><span>,</span><span style="color: #C6CCD7;">y</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">i32</span><span>},</span><span style="color: #5F6672;font-style: italic;">//has a name fields, like a struct</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> Write</span><span>(</span><span style="color: #E5C07B;">String</span><span>),</span><span style="color: #5F6672;font-style: italic;">// has a String</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> ChangeColor</span><span>(</span><span style="color: #E5C07B;">i32</span><span>,</span><span style="color: #E5C07B;">i32</span><span>,</span><span style="color: #E5C07B;">i32</span><span>),</span><span style="color: #5F6672;font-style: italic;">// includes 3 value, like a tuple</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">match</span><span style="color: #C6CCD7;"> enum_type</span><span>{</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ::</span><span style="color: #C6CCD7;">case</span><span style="color: #E06C75;"> =></span><span style="color: #C6CCD7;"> return_value</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> _</span><span style="color: #E06C75;"> =></span><span style="color: #C6CCD7;"> return_else_value</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">if</span><span style="color: #61AFEF;"> let</span><span style="color: #5F6672;font-style: italic;"> //for sequence only contains 2 branches</span></span>
<span class="giallo-l"><span style="color: #E06C75;">if</span><span style="color: #61AFEF;"> let</span><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #C6CCD7;">num</span><span>)</span><span style="color: #E06C75;"> =</span><span> num_type_is_Option{</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("{</span><span style="color: #98C379;">num</span><span>}");</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #E06C75;">else</span><span>{</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">not a num</span><span>");</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//match 必须包含所有branches</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//Rust 内置Option枚举</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> Option</span><span><</span><span style="color: #E5C07B;">T</span><span>>{</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #E5C07B;">T</span><span>),</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> None</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h2 id="packages-crates-and-modules">Packages, Crates and Modules</h2>
<p>调用<code>cargo new</code>会创建一个新项目，就是一个<code>Package</code></p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E5C07B;">Cargo</span><span style="color: #E06C75;">.</span><span>toml</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">src</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> main</span><span style="color: #E06C75;">.</span><span>rs</span></span></code></pre>
<p><code>Crate</code>有两种类型：<code>bin</code>和<code>lib</code></p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #C6CCD7;">src</span><span style="color: #E06C75;">/</span><span style="color: #C6CCD7;">main</span><span style="color: #E06C75;">.</span><span>rs </span><span style="color: #5F6672;font-style: italic;">//bin类型的Crate</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">src</span><span style="color: #E06C75;">/</span><span style="color: #C6CCD7;">lib</span><span style="color: #E06C75;">.</span><span>rs </span><span style="color: #5F6672;font-style: italic;">//lib类型的Crate</span></span>
<span class="giallo-l"><span>一个package中的Crate可以有多个bin类型，最多只能有一个lib类型，</span></span>
<span class="giallo-l"><span>当有多个bin，需要放到src</span><span style="color: #E06C75;">/</span><span>bin下。</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>bin类型use的path不能使用crate开头，必须是Crate本名。</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;">re</span><span style="color: #E06C75;">-</span><span style="color: #C6CCD7;">exporting</span><span>： 利用pub修饰use，让导入当前scope的第三者可以引用当前scope已经use的item</span></span></code></pre>
<p><code>module:</code>在Crate内组织代码</p>
<h2 id="chang-yong-de-collections">常用的<code>Collections</code></h2>
<h3 id="vectors"><code>vectors</code></h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #56B6C2;">1</span><span>,</span><span style="color: #56B6C2;">2</span><span>,</span><span style="color: #56B6C2;">3</span><span>];</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//let v: Vec<i32> = Vec::new();</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">v</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #56B6C2;">4</span><span>);</span><span style="color: #5F6672;font-style: italic;">//mutable reference</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">v</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">get</span><span>(</span><span style="color: #56B6C2;">2</span><span>);</span><span style="color: #5F6672;font-style: italic;">//return an Option</span></span></code></pre><h3 id="string-and-utf-8"><code>String and UTF-8</code></h3>
<p>use chars() or bytes() to visit item in side string</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #C6CCD7;">s</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">hello</span><span>");</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//for i in "hello".chars()</span></span>
<span class="giallo-l"><span style="color: #E06C75;">for</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;"> in</span><span style="color: #C6CCD7;"> s</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">chars</span><span>(){</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("{</span><span style="color: #98C379;">i</span><span>}");</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="hashmap"><code>HashMap</code></h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">collections</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">HashMap</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">let mut</span><span style="color: #C6CCD7;"> scores</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> HashMap</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;">scores</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">insert</span><span>("</span><span style="color: #98C379;">Blue</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>(),</span><span style="color: #56B6C2;">10</span><span>);</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//insert会夺走ownership</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> team_name</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">Blue</span><span>");</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> score</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> score</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">get</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">team_name</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">copied</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap_or</span><span>(</span><span style="color: #56B6C2;">0</span><span>);</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//get()返回一个Option<&V>，copied返回Option<&V>的一个拷贝，得到</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//ownership，用unwrap_or()“解包”返回一个i32</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">for</span><span> (</span><span style="color: #C6CCD7;">k</span><span>,</span><span style="color: #C6CCD7;">v</span><span>)</span><span style="color: #E06C75;"> in &</span><span style="color: #C6CCD7;">scores</span><span>{</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("{</span><span style="color: #98C379;">k</span><span>}</span><span style="color: #98C379;">:</span><span>{</span><span style="color: #98C379;">v</span><span>}");</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;">overwriting</span><span style="color: #E06C75;">:</span><span style="color: #B57EDC;"> insert</span><span>()</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">adding</span><span style="color: #E06C75;">:</span><span style="color: #B57EDC;"> entry</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">or_insert</span><span>()</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">//entry()return &mut to the value and use or_insert() to insert</span></span></code></pre><h2 id="error-handling">Error Handling</h2>
<h2 id="tong-cheng">统称</h2>
<p>可用于<code>fn,struct,enum</code>
e.g.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> example</span><span><</span><span style="color: #E5C07B;">T</span><span>>(</span><span style="color: #C6CCD7;">pa</span><span style="color: #E06C75;">:&</span><span style="color: #E5C07B;">T</span><span>)</span><span style="color: #E06C75;">->&</span><span style="color: #E5C07B;">T</span><span>{</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> do</span><span style="color: #C6CCD7;"> stuff</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">struct</span><span><</span><span style="color: #E5C07B;">T</span><span>,</span><span style="color: #E5C07B;">Y</span><span>>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">T</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> gender</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">Y</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> E</span><span><</span><span style="color: #E5C07B;">T</span><span>>{</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> name</span><span>(</span><span style="color: #E5C07B;">T</span><span>),</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> example</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #C6CCD7;"> some_trait</span><span>>(</span><span style="color: #C6CCD7;">pa</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">T</span><span>){</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> //pass</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h2 id="te-dian">特点</h2>
<h3 id="te-dian-de-sheng-ming">特点的声明：</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> trait</span><span style="color: #E5C07B;"> Summary</span><span>{</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> summarize</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;">-></span><span style="color: #E5C07B;">String</span><span>;</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="shi-yong-guan-jian-ci-forfu-yu-jie-gou-te-dian-li-zi-gei-liang-ge-bu-tong-de-jie-gou-ti-fu-yu-summaryde-te-dian">使用关键词<code>for</code>赋予结构特点，例子给两个不同的结构体赋予<code>Summary</code>的特点：</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> NewsArticle</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> headline</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> String</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> location</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> String</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> author</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> String</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> content</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> String</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Summary</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> NewsArticle</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> summarize</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> String</span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> format!</span><span>("{}</span><span style="color: #98C379;">, by </span><span>{}</span><span style="color: #98C379;"> (</span><span>{}</span><span style="color: #98C379;">)</span><span>",</span><span style="color: #E06C75;"> self.</span><span>headline,</span><span style="color: #E06C75;">self.</span><span>author,</span><span style="color: #E06C75;">self.</span><span>location)</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> Tweet</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> username</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> String</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub</span><span style="color: #C6CCD7;"> content</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> String</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Summary</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> Tweet</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> summarize</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> String</span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> format!</span><span>("{}</span><span style="color: #98C379;">: </span><span>{}",</span><span style="color: #E06C75;"> self.</span><span>username,</span><span style="color: #E06C75;"> self.</span><span>content)</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="diao-yong-fang-shi-lei-si-han-shu">调用方式，类似函数：</h3>
<p>这里use还要包括<code>Summary</code>，因为<code>trait</code>是储存在块本地的；
另外，已经定义的结构和特点不能重复定义</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> aggregator</span><span style="color: #E06C75;">::</span><span>{</span><span style="color: #E5C07B;">Summary</span><span>,</span><span style="color: #E5C07B;"> Tweet</span><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> tweet</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Tweet</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> username</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">horse_ebooks</span><span>"),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> content</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">of course, as you probably already know, people</span><span>",</span></span>
<span class="giallo-l"><span> ),</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> reply</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> false</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> retweet</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> false</span><span>,</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">1 new tweet: </span><span>{}",</span><span style="color: #C6CCD7;"> tweet</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">summarize</span><span>());</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="ke-yi-she-zhi-mo-ren-de-te-dian">可以设置默认的特点：</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> trait</span><span style="color: #E5C07B;"> Summary</span><span>{</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> summarize</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> String</span><span>{</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">Read more...</span><span>")</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Summary</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> NewsArticle</span><span> {}</span><span style="color: #5F6672;font-style: italic;"> //必须有这句话，而且括号内没有对应的方法</span></span></code></pre><h3 id="ba-te-dian-zuo-wei-han-shu-can-shu">把特点作为函数参数：</h3>
<p>把特点作为参数的意义在于，任何实现了这个特点的数据类型都能作为这个函数的参数</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> notify</span><span>(</span><span style="color: #C6CCD7;">item</span><span style="color: #E06C75;">: &impl</span><span style="color: #E5C07B;"> Summary</span><span>){</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Breaking news! </span><span>{}",</span><span style="color: #C6CCD7;">item</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">summarize</span><span>());</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> notify</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Summary</span><span>>(</span><span style="color: #C6CCD7;">item</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> T</span><span>){</span><span style="color: #5F6672;font-style: italic;">//等价</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Breaking news! </span><span>{}",</span><span style="color: #C6CCD7;">item</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">summarize</span><span>());</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="tong-shi-man-zu-duo-ge-te-dian">同时满足多个特点：</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> notify</span><span>(</span><span style="color: #C6CCD7;">item</span><span style="color: #E06C75;">: &</span><span>(</span><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Summary</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Display</span><span>)) {}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> notify</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Summary</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Display</span><span>>(</span><span style="color: #C6CCD7;">item</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">T</span><span>){}</span></span></code></pre><h3 id="geng-duo-li-zi">更多例子：</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> func</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Display</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Clone</span><span>,</span><span style="color: #E5C07B;"> U</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Clone</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Debug</span><span>>(</span><span style="color: #C6CCD7;">t</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">T</span><span>,</span><span style="color: #C6CCD7;"> u</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">U</span><span>){}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> func</span><span><</span><span style="color: #E5C07B;">T</span><span>,</span><span style="color: #E5C07B;">U</span><span>>(</span><span style="color: #C6CCD7;">t</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">T</span><span>,</span><span style="color: #C6CCD7;"> u</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">U</span><span>)</span></span>
<span class="giallo-l"><span style="color: #E06C75;">where</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> T</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Display</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Clone</span><span>,</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> U</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Clone</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> Denug</span><span>,</span></span>
<span class="giallo-l"><span>{}</span></span></code></pre><h3 id="yao-qiu-fan-hui-zhi-shi-xian-te-dian">要求返回值实现特点：</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> returns_summarizable</span><span>()</span><span style="color: #E06C75;"> -> impl</span><span style="color: #E5C07B;"> Summary</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Article</span><span>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> name</span><span style="color: #E06C75;">:</span><span>"</span><span style="color: #98C379;">Jane</span><span>"</span><span style="color: #E06C75;">.</span><span>to_string,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> published</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;">false</span><span>,</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="shi-xian-bao-han-fan-xing-de-jie-gou-fang-fa-shi-qu-bie-dui-dai">实现包含泛型的结构方法时，区别对待：</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> Pair</span><span><</span><span style="color: #E5C07B;">T</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">T</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">T</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span><</span><span style="color: #E5C07B;">T</span><span>></span><span style="color: #E5C07B;"> Pair</span><span><</span><span style="color: #E5C07B;">T</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> new</span><span>(</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">T</span><span>,</span><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">T</span><span>)</span><span style="color: #E06C75;">->Self</span><span>{</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> Self</span><span> {</span><span style="color: #C6CCD7;"> x</span><span>,</span><span style="color: #C6CCD7;"> y</span><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span><</span><span style="color: #E5C07B;">T</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Display</span><span style="color: #E06C75;"> +</span><span style="color: #E5C07B;"> ParialOrd</span><span>></span><span style="color: #E5C07B;"> Pari</span><span><</span><span style="color: #E5C07B;">T</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> cmp_display</span><span>(</span><span style="color: #E06C75;">&self</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if self.</span><span>x </span><span style="color: #E06C75;">>= self.</span><span>y {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> ...</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h2 id="lifetime"><code>Lifetime</code></h2>
<p>基本的在之前已经提到过</p>
<h3 id="borrow-checker"><code>Borrow Checker</code></h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> r</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // ---------+-- 'a</span></span>
<span class="giallo-l"><span> {</span><span style="color: #5F6672;font-style: italic;"> // |</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 5</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // -+-- 'b |</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> r</span><span style="color: #E06C75;"> = &</span><span style="color: #C6CCD7;">x</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // | |</span></span>
<span class="giallo-l"><span> }</span><span style="color: #5F6672;font-style: italic;"> // -+ |</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">r: </span><span>{}",</span><span style="color: #C6CCD7;"> r</span><span>);</span><span style="color: #5F6672;font-style: italic;"> // |</span></span>
<span class="giallo-l"><span>}</span><span style="color: #5F6672;font-style: italic;"> // ---------+</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 5</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // ----------+-- 'b</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> r</span><span style="color: #E06C75;"> = &</span><span style="color: #C6CCD7;">x</span><span>;</span><span style="color: #5F6672;font-style: italic;"> // --+-- 'a |</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">r: </span><span>{}",</span><span style="color: #C6CCD7;"> r</span><span>);</span><span style="color: #5F6672;font-style: italic;"> // --+ |</span></span>
<span class="giallo-l"><span>}</span><span style="color: #5F6672;font-style: italic;"> // ----------+</span></span></code></pre><h3 id="can-shu-wei-yin-yong-bian-yi-qi-wu-fa-pan-duan-sheng-ming-zhou-qi-shi">参数为引用编译器无法判断生命周期时</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> longest</span><span>(</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>,</span><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>)</span><span style="color: #E06C75;"> -> &</span><span style="color: #E5C07B;">str</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>() ></span><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> x</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> y</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> longest</span><span><'</span><span style="color: #E5C07B;">a</span><span>>(</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>)</span><span style="color: #E06C75;"> -> &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>{}</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// 参数和返回值具有相同的生命周期'a </span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// 返回值的lifetime = min(x,y)</span></span></code></pre>
<p>之所以参数都要需要标记，是因为<code>x</code>，<code>y</code>都有可能被返回</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> longest</span><span><'</span><span style="color: #E5C07B;">a</span><span>>(</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>)</span><span style="color: #E06C75;"> -> &</span><span>'</span><span style="color: #E5C07B;">a str</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> x</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="jie-gou-ti-nei-de-bian-liang-ye-ke-yi-shi-yin-yong-dan-shi-bi-xu-zhi-ming-sheng-ming-zhou-qi">结构体内的变量也可以是引用，但是必须指明生命周期</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> SSS</span><span><'</span><span style="color: #E5C07B;">a</span><span>>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> a_str</span><span style="color: #E06C75;">:&</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span></span>
<span class="giallo-l"><span>}</span><span style="color: #5F6672;font-style: italic;">//意味着，一个SSS实例的lifetime不能大于a_str</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>(){</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> sss</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> SSS</span><span>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> a_str</span><span style="color: #E06C75;">:</span><span>{</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> tmp</span><span style="color: #E06C75;"> =</span><span> "</span><span style="color: #98C379;">hello</span><span>"</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">to_string</span><span>();</span><span style="color: #5F6672;font-style: italic;">//生命周期过短</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> tmp</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">as_str</span><span>()</span></span>
<span class="giallo-l"><span> },</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> //a_str:"hello",//正常运行</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("{}",</span><span style="color: #C6CCD7;">sss</span><span style="color: #E06C75;">.</span><span>a_str);</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="wu-xu-sheng-ming-sheng-ming-zhou-qi-de-qing-kuang-lifetime-elision-rules">无需声明生命周期的情况(<code>lifetime elision rules</code>)</h3>
<p>编译器检查生命周期的三条规则，一旦能确认所有参数和输出值的生命周期，编译器不强制要求添加<code>lifetime annotation</code>：</p>
<ol>
<li>
<p>给每个参数赋予不同的生命周期</p>
</li>
<li>
<p>只有一个参数时，返回值的生命周期和这个参数一样</p>
</li>
<li>
<p>当参数形式为<code>&self</code>时，因为<code>self</code>，所以所有生命周期都是<code>self</code>的生命周期</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> first_word</span><span>(</span><span style="color: #C6CCD7;">s</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>)</span><span style="color: #E06C75;"> -> &</span><span style="color: #E5C07B;">str</span><span>{</span><span style="color: #5F6672;font-style: italic;">//rule 1</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> first_word</span><span><'</span><span style="color: #E5C07B;">a</span><span>>(</span><span style="color: #C6CCD7;">s</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>)</span><span style="color: #E06C75;"> -> &</span><span style="color: #E5C07B;">str</span><span>{</span><span style="color: #5F6672;font-style: italic;">//rule 2</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> first_word</span><span><'</span><span style="color: #E5C07B;">a</span><span>>(</span><span style="color: #C6CCD7;">s</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>)</span><span style="color: #E06C75;"> -> &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>{</span><span style="color: #5F6672;font-style: italic;">//rule 3</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> longest</span><span>(</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>,</span><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>)</span><span style="color: #E06C75;"> -> &</span><span style="color: #E5C07B;">str</span><span> {</span><span style="color: #5F6672;font-style: italic;">//rule1</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> longest</span><span><'</span><span style="color: #E5C07B;">a</span><span>, '</span><span style="color: #E5C07B;">b</span><span>>(</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">b str</span><span>)</span><span style="color: #E06C75;"> -> &</span><span style="color: #E5C07B;">str</span><span> {</span><span style="color: #5F6672;font-style: italic;">//rule2</span></span></code></pre></li>
</ol>
<h2 id="fan-xing-te-dian-sheng-ming-zhou-qi-da-za-hui">泛型、特点、生命周期大杂烩</h2>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">fmt</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Display</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> longest_with_an_announcement</span><span><'</span><span style="color: #E5C07B;">a</span><span>,</span><span style="color: #E5C07B;"> T</span><span>>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a str</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> ann</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> T</span><span>,</span></span>
<span class="giallo-l"><span>)</span><span style="color: #E06C75;"> -> &</span><span>'</span><span style="color: #E5C07B;">a str</span></span>
<span class="giallo-l"><span style="color: #E06C75;">where</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> T</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Display</span><span>,</span></span>
<span class="giallo-l"><span>{</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Announcement! </span><span>{}",</span><span style="color: #C6CCD7;"> ann</span><span>);</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>() ></span><span style="color: #C6CCD7;"> y</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> x</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> y</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h2 id="ce-shi">测试</h2>
<h3 id="san-ge-ce-shi-han-shu">三个测试函数</h3>
<p>在<code>Rust</code>中，可以为自己编写的程序编写测试以测试代码的正确性。
在终端调用<code>cargo new proj_name --lib</code>会自动生成项目，<code>src</code>文件夹的<code>lib.rs</code>会自带<code>test module</code>。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[cfg(test)]</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">mod</span><span> tests {</span></span>
<span class="giallo-l"><span> #[test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> it_works</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> result</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 2</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 2</span><span>;</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">result</span><span>,</span><span style="color: #56B6C2;"> 4</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>其中<code>#[tests]</code>用来指示接下来的函数是用于测试的函数，<code>assert_eq!()</code>判断参数是否相同，当结果不同或者代码发生<code>panic!</code>就会测试失败，另<code>assert_ne!()</code>则判断是否不同。</p>
<p><code>assert!</code>参数类型是<code>Boolean</code>，如果参数为<code>false</code>则测试失败。</p>
<h3 id="partialeq-debug-te-dian">PartialEq Debug 特点</h3>
<p>如果一个结构体或者枚举想要作为参数传入上述函数，必须实现<code>PartialEq</code>和<code>Debug</code>这两个特点，（前者在于<code>==、!=</code>的比较，后者在于代码判断为<code>Failed</code>时要打印错误信息）</p>
<h3 id="zi-ding-yi-shi-bai-xin-xi">自定义失败信息</h3>
<p><code>assert</code>函数第二个参数即为自定义字符串信息：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub fn</span><span style="color: #B57EDC;"> greeting</span><span>(</span><span style="color: #C6CCD7;">name</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> String</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">Hello!</span><span>")</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span>#[cfg(test)]</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">mod</span><span> tests {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> use super::*</span><span>;</span></span>
<span class="giallo-l"><span> #[test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> greeting_contains_name</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> result</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> greeting</span><span>("</span><span style="color: #98C379;">Carol</span><span>");</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert!</span><span>(</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> result</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">contains</span><span>("</span><span style="color: #98C379;">Carol</span><span>"),</span></span>
<span class="giallo-l"><span> "</span><span style="color: #98C379;">Greeting did not contain name, value was `</span><span>{}</span><span style="color: #98C379;">`</span><span>",</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> result</span></span>
<span class="giallo-l"><span> );</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>报错信息则为<code>"Greeting did not contain name, value was 'Carol'"</code>。</p>
<h3 id="should-panic">should_panic</h3>
<p>在<code>#[test]</code>属性下一行添加<code>#[should_panic]</code>，让编译器认为检测函数应该<code>panic</code>，否则测试不通过。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[test]</span></span>
<span class="giallo-l"><span>#[should_panic]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> example</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> panic!</span><span>("</span><span style="color: #98C379;">This is supposed to panic.</span><span>");</span><span style="color: #5F6672;font-style: italic;">//测试ok</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>使用<code>expected</code>可以使<code>should_panic</code>更加精确：
例如<code>#[should_panic(expected = "is supposed")]</code>替换上面的<code>#[should_panic]</code>，
则测试可以通过，因为<code>panic contained expected string</code>。</p>
<h3 id="result-t-e">Result<T, E></h3>
<p>在<code>test module</code>中，可以使用<code>Result</code>枚举来代替<code>assert</code>系列函数：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[cfg(test)]</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">mod</span><span> tests {</span></span>
<span class="giallo-l"><span> #[test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> it_works</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Result</span><span><(),</span><span style="color: #E5C07B;"> String</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> value</span><span style="color: #E06C75;"> =</span><span> </span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #56B6C2;"> 2</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 2</span><span style="color: #E06C75;"> ==</span><span style="color: #56B6C2;"> 4</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Ok</span><span>(())</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Err</span><span>(</span><span style="color: #E5C07B;">String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">two plus two does not equal four</span><span>"))</span></span>
<span class="giallo-l"><span> };</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> value</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>值得注意的是，在这种情况下不能使用<code>should_panic</code>，非要使用的话，可以在函数最后<code>assert!(value.is_err())</code>;</p>
<h3 id="kong-zhi-ce-shi-fang-shi">控制测试方式</h3>
<p><code>cargo test</code></p>
<ol>
<li><code>-- --test-thread=1 </code>线程设置为<code>1</code>，禁止测试函数并行运行</li>
<li><code>-- --show-output </code>就算测试不通过也打印过程中<code>println</code>的信息</li>
<li><code>cargo test prefix*</code> 只测试以<code>prefix</code>打头的函数</li>
<li>在代码中<code>#[test]</code>下一行添加<code>#[ignore]</code>可以使测试忽略当前函数</li>
<li><code>-- --ignored</code> 只运行标记了<code>ignore</code>的函数</li>
</ol>
<h3 id="dan-yuan-ce-shi-he-ji-cheng-ce-shi">单元测试和集成测试</h3>
<p>单元测试：
更小更集中，在隔离的环境中测试一个模块，上面的例子都是单元测试；
使用<code>#[cfg(test)]</code>标注，只在<code>cargo test</code>时编译和运行</p>
<p>集成测试：
在外部的，位于与一般代码相同的文件中。需要新建与<code>src</code>同级的<code>tests</code>目录，在目录下新建<code>test_name.rs</code>，需要使用<code>use</code>导入测试目标<code>crate</code>，代码无需<code>cfg</code>标记，只需在每个测试函数前<code>#[test]</code>即可。</p>
<h2 id="bi-bao">闭包</h2>
<p>任何花括号包括的区域最后一条语句没有加分号都有返回值，{}闭包就是可以保存进变量 或 作为参数传递给其他函数 的 匿名函数。
函数变量e.g.</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #C6CCD7;"> add_one_v1</span><span> (</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> u32</span><span> {</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span> }</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> add_one_v2</span><span style="color: #E06C75;"> = |</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> u32</span><span style="color: #E06C75;">| -></span><span style="color: #E5C07B;"> u32</span><span> {</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span> };</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> add_one_v3</span><span style="color: #E06C75;"> = |</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">|</span><span> {</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span> };</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> add_one_v4</span><span style="color: #E06C75;"> = |</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;"> x</span><span style="color: #E06C75;"> +</span><span style="color: #56B6C2;"> 1</span><span> ;</span></span></code></pre><h2 id="die-dai-qi">迭代器</h2>
<h3 id="iterator-next">Iterator & next</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> trait</span><span style="color: #E5C07B;"> Iterator</span><span>{</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> type</span><span style="color: #E5C07B;"> Item</span><span>;</span></span>
<span class="giallo-l"><span> </span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> next</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Option</span><span><</span><span style="color: #E06C75;">Self::</span><span style="color: #E5C07B;">Item</span><span>>;</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>如果直接把迭代器 <code>iter()</code> 赋值给变量 <code>v1</code>，那么在直接调用 <code>v1.next()</code> 要求 <code>v1 </code>是可变的，如果是 <code>for</code> 来遍历则不需要，因为 <code>rust </code>后台自动获取 <code>v1</code> 的所有权并使它可变。</p>
<p>需要注意，<code>vector </code>迭代器的 <code>next </code> 返回的是不可变引用，如果需要得到所有权，可以调用<code>into_iter</code>；如果需要可变迭代引用，可以调用 <code>iter_mut</code>。</p>
<h3 id="xiao-fei-die-dai-qi-de-fang-fa">消费迭代器的方法</h3>
<p><code>Iterator trait </code>中调用了 <code>next </code>的方法被称为<strong>消费适配器</strong>( <code>consuming adaptors</code> )，这些方法会消耗迭代器，夺走所有权。</p>
<p>以sum为例：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> iterator_sum</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> v1</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #56B6C2;">1</span><span>,</span><span style="color: #56B6C2;"> 2</span><span>,</span><span style="color: #56B6C2;"> 3</span><span>];</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> v1_iter</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> v1</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">iter</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> total</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> i32</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> v1_iter</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">sum</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">total</span><span>,</span><span style="color: #56B6C2;"> 6</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>这里调用 <code>sum</code> 之后不再允许使用 <code>v1_iter</code>，因为所有权用完了。</p>
<h3 id="jian-dan-bian-xie-zi-ding-yi-next">简单编写自定义next</h3>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #C6CCD7;"> cnt</span><span>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> count</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">u32</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #C6CCD7;"> cnt</span><span>{</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> new</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #C6CCD7;"> cnt</span><span>{</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> cnt</span><span>{</span><span style="color: #C6CCD7;">count</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;">0</span><span>}</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span style="color: #E5C07B;"> Iterator</span><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> cnt</span><span>{</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> type</span><span style="color: #E5C07B;"> Item</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> u32</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> next</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> Option</span><span><</span><span style="color: #E06C75;">Self::</span><span style="color: #E5C07B;">Item</span><span>>{</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>count </span><span style="color: #E06C75;">+=</span><span style="color: #56B6C2;"> 1</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> if self.</span><span>count</span><span style="color: #E06C75;"> <</span><span style="color: #56B6C2;"> 6</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Some</span><span>(</span><span style="color: #E06C75;">self.</span><span>count)</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> None</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>这样，就可以获得可以<code>next</code>六次的迭代器了。</p>
<h2 id="xun-huan-vsdie-dai-qi">循环VS迭代器</h2>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> buffer</span><span style="color: #E06C75;">: &</span><span style="color: #61AFEF;">mut</span><span> [</span><span style="color: #E5C07B;">i32</span><span>];</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> coefficients</span><span style="color: #E06C75;">:</span><span> [</span><span style="color: #E5C07B;">i64</span><span>;</span><span style="color: #56B6C2;"> 12</span><span>];</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">let</span><span style="color: #C6CCD7;"> qlp_shift</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> i16</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">for</span><span style="color: #C6CCD7;"> i</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 12</span><span style="color: #E06C75;">..</span><span style="color: #C6CCD7;">buffer</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> prediction</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> coefficients</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">iter</span><span>()</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">zip</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">buffer</span><span>[</span><span style="color: #C6CCD7;">i</span><span style="color: #E06C75;"> -</span><span style="color: #56B6C2;"> 12</span><span style="color: #E06C75;">..</span><span style="color: #C6CCD7;">i</span><span>])</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">map</span><span>(</span><span style="color: #E06C75;">|</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">c</span><span>,</span><span style="color: #E06C75;"> &</span><span style="color: #C6CCD7;">s</span><span>)</span><span style="color: #E06C75;">|</span><span style="color: #C6CCD7;"> c</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;"> s</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> i64</span><span>)</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> .</span><span style="color: #B57EDC;">sum</span><span style="color: #E06C75;">::</span><span><</span><span style="color: #E5C07B;">i64</span><span>>()</span><span style="color: #E06C75;"> >></span><span style="color: #C6CCD7;"> qlp_shift</span><span>;</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> delta</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> buffer</span><span>[</span><span style="color: #C6CCD7;">i</span><span>];</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> buffer</span><span>[</span><span style="color: #C6CCD7;">i</span><span>]</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> prediction</span><span style="color: #E06C75;"> as</span><span style="color: #E5C07B;"> i32</span><span style="color: #E06C75;"> +</span><span style="color: #C6CCD7;"> delta</span><span>;</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>为了计算 <code>prediction</code> 的值，这些代码遍历了 <code>coefficients</code> 中的 12 个值，使用 <code>zip</code> 方法将系数与 <code>buffer</code> 的前 12 个值组合在一起。接着将每一对值相乘，再将所有结果相加，然后将总和右移 <code>qlp_shift</code> 位。</p>
<p>这里，我们创建了一个迭代器，使用了两个适配器，接着消费了其值。Rust 代码将会被编译为什么样的汇编代码呢？遍历 <code>coefficients</code> 的值完全用不到循环：Rust 知道这里会迭代 12 次，所以它“展开”（unroll）了循环。展开是一种移除循环控制代码的开销并替换为每个迭代中的重复代码的优化。</p>
<p>所有的系数都被储存在了寄存器中，这意味着访问他们非常快。这里也没有运行时数组访问边界检查。所有这些 Rust 能够提供的优化使得结果代码极为高效。</p>
<h2 id="zhi-neng-zhi-zhen">智能指针</h2>
<p>其实，应用最多的指针是 <code>&</code> 引用，但是Rust中有一类结构实现了 <code>Deref</code> 和 <code>Drop</code> trait，前者让结构体实例能像引用那样（可以“解包”），后者可以自定义当智能指针离开作用域要运行的代码。</p>
<p>其实，<code>String </code> 和 <code>Vec<T></code> 都属于智能指针。</p>
<h3 id="box-t"><code>Box<T></code></h3>
<p>在堆上储存数据，在栈上储存指针。</p>
<h4 id="di-gui-mei-ju">递归枚举</h4>
<p>如果要自定义一个枚举列表：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> List</span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #E5C07B;">i32</span><span>,</span><span style="color: #E5C07B;"> List</span><span>),</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Nil</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>上面的代码是不能编译的，因为List中永远有下一个List，而每个List都附带一个i32（在栈上），编译器不知道具体要分配多少空间。</p>
<p><img src="/images/trpl15-01.svg" alt="An infinite Cons list" /></p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> List</span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #E5C07B;">i32</span><span>,</span><span style="color: #E5C07B;"> Box</span><span><</span><span style="color: #E5C07B;">List</span><span>>),</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Nil</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">use crate::</span><span style="color: #61AFEF;">List</span><span style="color: #E06C75;">::</span><span>{</span><span style="color: #E5C07B;">Cons</span><span>,</span><span style="color: #E5C07B;"> Nil</span><span>};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> list</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #56B6C2;">1</span><span>,</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Box</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">Cons</span><span>(</span><span style="color: #56B6C2;">2</span><span>,</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Box</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">Cons</span><span>(</span><span style="color: #56B6C2;">3</span><span>,</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> Box</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">Nil</span><span>))))));</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>上面的代码就是可行的<img src="/images/trpl15-02.svg" alt="A finite Cons list" /></p>
<h3 id="zi-ding-yi-yi-ge-zhi-neng-zhi-zhen">自定义一个智能指针</h3>
<h4 id="shi-xian-deref-trait">实现Deref trait</h4>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">ops</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Deref</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;">struct</span><span style="color: #E5C07B;"> MyBox</span><span><</span><span style="color: #E5C07B;">T</span><span>>(</span><span style="color: #E5C07B;">T</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span><</span><span style="color: #E5C07B;">T</span><span>></span><span style="color: #E5C07B;"> MyBox</span><span><</span><span style="color: #E5C07B;">T</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> new</span><span>(</span><span style="color: #C6CCD7;">x</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;">T</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> MyBox</span><span><</span><span style="color: #E5C07B;">T</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> MyBox</span><span>(</span><span style="color: #C6CCD7;">x</span><span>)</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span><</span><span style="color: #E5C07B;">T</span><span>></span><span style="color: #E5C07B;"> Deref</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> MyBox</span><span><</span><span style="color: #E5C07B;">T</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> type</span><span style="color: #E5C07B;"> Target</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> T</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> deref</span><span>(</span><span style="color: #E06C75;">&self</span><span>)</span><span style="color: #E06C75;"> -> &</span><span style="color: #E5C07B;">T</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> &self.</span><span style="color: #56B6C2;">0</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>以上代码能够让 <code>MyBox</code> 能够像 <code>dereference</code> 引用 <code>&</code> 一样解包 <code>MyBox</code> 实例，这里返回的是 <code>&x</code>，因此实际Rust在底层运行的代码是：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">*</span><span>(</span><span style="color: #C6CCD7;">y</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">deref</span><span>())</span><span style="color: #5F6672;font-style: italic;">//y是一个MyBox实例，deref()返回的是引用</span></span></code></pre><h3 id="jie-zhu-deref-traitshi-xian-yin-shi-jie-yin-yong-qiang-zhi-zhuan-huan">借助Deref trait实现隐式解引用强制转换</h3>
<p>有这样一个函数：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> hello</span><span>(</span><span style="color: #C6CCD7;">name</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Hello, </span><span>{}</span><span style="color: #98C379;">!</span><span>",</span><span style="color: #C6CCD7;"> name</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>要求参数是 <code>&str</code>，而：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> m</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> MyBox</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">Rust</span><span>"));</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> hello</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">m</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>这段代码却可以正常运行，这是因为 <code>MyBox</code> 和 <code>String</code> 都有实现 <code>Deref</code> trait，这里使用 <code>&m</code> 调用 <code>hello</code> 函数，其为 <code>MyBox<String></code> 值的引用。 Rust 可以通过 <code>deref</code> 调用将 <code>&MyBox<String></code> 变为 <code>&String</code>。Rust 再次调用 <code>deref</code> 将 <code>&String</code> 变为 <code>&str</code>，这就符合 <code>hello</code> 函数的定义了。</p>
<p>如果没有实现解引用强制转换，代码将必须写成这样：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> m</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> MyBox</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">Rust</span><span>"));</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> hello</span><span>(</span><span style="color: #E06C75;">&</span><span>(</span><span style="color: #E06C75;">*</span><span style="color: #C6CCD7;">m</span><span>)[</span><span style="color: #E06C75;">..</span><span>]);</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>以上解析都是发生在编译时，不会有运行损耗。</p>
<h3 id="drop-trait">Drop Trait</h3>
<p>之所以把 <code>Drop</code> trait 放在智能指针这里，是因为 <code>Drop</code> 几乎都是用于实现智能指针。</p>
<p>可以直接把 <code>Drop</code> 里的 <code>drop()</code> 理解为析构函数，我们不能直接调用 <code>drop</code> 因为这样的话作用域结束后又会调用一次，如果需要提前释放变量可以使用 <code>std::mem::drop</code> 这个函数不同于前者，位于 prelude 中，通过调用它可以提前释放变量。无需担心之后发生“空指针”的问题，因为编译起会帮你检测，确保引用总是有效的。</p>
<h3 id="rc-t-yin-yong-ji-shu-zhi-neng-zhi-zhen"><code>Rc<T></code> 引用计数智能指针</h3>
<p>Rust 内置一个叫做 <code>Rc<T></code> 的类型，是 <code>reference counting</code> 的缩写。</p>
<p><code>Rc<T></code> 用于当我们希望在堆上分配内存给程序多个部分读取，而且无法在编译时确定程序的哪一部分会最后结束使用它的时候。如果确实知道哪一部分是最后结束使用的话，就能令其成为数据的所有者，正常的所有权规则就可以在编译时运行。</p>
<p><code>RC<T></code> 只能用于单线程场景。</p>
<p>类似浅拷贝，不会完整拷贝 <code>T</code> 所有数据，而是让<strong>引用计数</strong>增加。在场景：多个列表需要共享一段列表时，不会有所有权“不够用”或者深拷贝低效率的问题。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> List</span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #E5C07B;">i32</span><span>,</span><span style="color: #E5C07B;"> Rc</span><span><</span><span style="color: #E5C07B;">List</span><span>>),</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Nil</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">use crate::</span><span style="color: #61AFEF;">List</span><span style="color: #E06C75;">::</span><span>{</span><span style="color: #E5C07B;">Cons</span><span>,</span><span style="color: #E5C07B;"> Nil</span><span>};</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">rc</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Rc</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">Cons</span><span>(</span><span style="color: #56B6C2;">5</span><span>,</span><span style="color: #61AFEF;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">Cons</span><span>(</span><span style="color: #56B6C2;">10</span><span>,</span><span style="color: #61AFEF;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">Nil</span><span>)))));</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">count after creating a = </span><span>{}",</span><span style="color: #E5C07B;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">strong_count</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #56B6C2;">3</span><span>,</span><span style="color: #61AFEF;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">count after creating b = </span><span>{}",</span><span style="color: #E5C07B;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">strong_count</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> c</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #56B6C2;">4</span><span>,</span><span style="color: #61AFEF;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">count after creating c = </span><span>{}",</span><span style="color: #E5C07B;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">strong_count</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">count after c goes out of scope = </span><span>{}",</span><span style="color: #E5C07B;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">strong_count</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #56B6C2;">OUTPUT</span><span style="color: #E06C75;">:</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">count after creating a</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 1</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">count after creating b</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 2</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">count after creating c</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 3</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;">count after c goes out of scope</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 2</span></span></code></pre>
<p><code>Rc<T></code> 允许程序多个部分只读共享数据。</p>
<h3 id="refcell-t-he-nei-bu-ke-bian-xing-mo-shi"><code>RefCell<T></code> 和内部可变性模式</h3>
<p><strong>内部可变性</strong>（<em>Interior mutability</em>）是 Rust 中的一个设计模式，允许你在有不可变量引用存在的情况下也能改变数据。使用 <code>unsafe</code> 来模糊 Rust 通常的可变性和借用规则。</p>
<ul>
<li>
<p><code>RefCell<T></code> 代表其数据唯一所有权。</p>
</li>
<li>
<p><code>RefCell<T></code> 用于当你确定代码是正确的，但是编译器不这么认为的时候。</p>
</li>
<li>
<p><code>RefCell<T></code> 同样只能用于单线程场景。</p>
</li>
</ul>
<p>如下为选择 <code>Box<T></code>，<code>Rc<T></code> 或 <code>RefCell<T></code> 的理由：</p>
<ul>
<li><code>Rc<T></code> 允许相同数据有多个所有者；<code>Box<T></code> 和 <code>RefCell<T></code> 有单一所有者。</li>
<li><code>Box<T></code> 允许在编译时执行不可变或可变借用检查；<code>Rc<T></code>仅允许在编译时执行不可变借用检查；<code>RefCell<T></code> 允许在运行时执行不可变或可变借用检查。</li>
<li>因为 <code>RefCell<T></code> 允许在运行时执行可变借用检查，所以我们可以在即便 <code>RefCell<T></code> 自身是不可变的情况下修改其内部的值。</li>
</ul>
<p>假如现在有一个结构 <code>A</code> ，结构 <code>B</code> 内的变量有 <code>&A</code> ，一般来说，我们是不能改变 <code>B</code> 的，因为它不是可变引用，但是如果 <code>B</code> 里面的某个变量是 <code>RefCell<T></code> ，那么就可以调用 <code>B.some_variable.borrow_mut()</code> ，这会返回一个可变引用在<strong>内部</strong>实现数据更改。</p>
<p>实例代码：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> trait</span><span style="color: #E5C07B;"> Messenger</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> send</span><span>(</span><span style="color: #E06C75;">&self</span><span>,</span><span style="color: #C6CCD7;"> msg</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">pub</span><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> LimitTracker</span><span><'</span><span style="color: #E5C07B;">a</span><span>,</span><span style="color: #E5C07B;"> T</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Messenger</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> messenger</span><span style="color: #E06C75;">: &</span><span>'</span><span style="color: #E5C07B;">a T</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> value</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> usize</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> max</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> usize</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">impl</span><span><'</span><span style="color: #E5C07B;">a</span><span>,</span><span style="color: #E5C07B;"> T</span><span>></span><span style="color: #E5C07B;"> LimitTracker</span><span><'</span><span style="color: #E5C07B;">a</span><span>,</span><span style="color: #E5C07B;"> T</span><span>></span></span>
<span class="giallo-l"><span style="color: #E06C75;"> where</span><span style="color: #E5C07B;"> T</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> Messenger</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> new</span><span>(</span><span style="color: #C6CCD7;">messenger</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">T</span><span>,</span><span style="color: #C6CCD7;"> max</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> usize</span><span>)</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> LimitTracker</span><span><</span><span style="color: #E5C07B;">T</span><span>> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> LimitTracker</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> messenger</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> value</span><span style="color: #E06C75;">:</span><span style="color: #56B6C2;"> 0</span><span>,</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> max</span><span>,</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> pub fn</span><span style="color: #B57EDC;"> set_value</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #61AFEF;">mut</span><span style="color: #E06C75;"> self</span><span>,</span><span style="color: #C6CCD7;"> value</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> usize</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>value </span><span style="color: #E06C75;">=</span><span style="color: #C6CCD7;"> value</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> percentage_of_max</span><span style="color: #E06C75;"> = self.</span><span>value </span><span style="color: #E06C75;">as</span><span style="color: #E5C07B;"> f64</span><span style="color: #E06C75;"> / self.</span><span>max </span><span style="color: #E06C75;">as</span><span style="color: #E5C07B;"> f64</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> if</span><span style="color: #C6CCD7;"> percentage_of_max</span><span style="color: #E06C75;"> >=</span><span style="color: #56B6C2;"> 1</span><span>.</span><span style="color: #56B6C2;">0</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>messenger</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">send</span><span>("</span><span style="color: #98C379;">Error: You are over your quota!</span><span>");</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else if</span><span style="color: #C6CCD7;"> percentage_of_max</span><span style="color: #E06C75;"> >=</span><span style="color: #56B6C2;"> 0</span><span>.</span><span style="color: #56B6C2;">9</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>messenger</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">send</span><span>("</span><span style="color: #98C379;">Urgent warning: You've used up over 90% of your quota!</span><span>");</span></span>
<span class="giallo-l"><span> }</span><span style="color: #E06C75;"> else if</span><span style="color: #C6CCD7;"> percentage_of_max</span><span style="color: #E06C75;"> >=</span><span style="color: #56B6C2;"> 0</span><span>.</span><span style="color: #56B6C2;">75</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>messenger</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">send</span><span>("</span><span style="color: #98C379;">Warning: You've used up over 75% of your quota!</span><span>");</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>#[cfg(test)]</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">mod</span><span> tests {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> use super::*</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">cell</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">RefCell</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> struct</span><span style="color: #E5C07B;"> MockMessenger</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> sent_messages</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> RefCell</span><span><</span><span style="color: #E5C07B;">Vec</span><span><</span><span style="color: #E5C07B;">String</span><span>>>,</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> impl</span><span style="color: #E5C07B;"> MockMessenger</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> new</span><span>()</span><span style="color: #E06C75;"> -></span><span style="color: #E5C07B;"> MockMessenger</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> MockMessenger</span><span> {</span><span style="color: #C6CCD7;"> sent_messages</span><span style="color: #E06C75;">:</span><span style="color: #E5C07B;"> RefCell</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">vec!</span><span>[]) }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> impl</span><span style="color: #E5C07B;"> Messenger</span><span style="color: #E06C75;"> for</span><span style="color: #E5C07B;"> MockMessenger</span><span> {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> send</span><span>(</span><span style="color: #E06C75;">&self</span><span>,</span><span style="color: #C6CCD7;"> message</span><span style="color: #E06C75;">: &</span><span style="color: #E5C07B;">str</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> self.</span><span>sent_messages</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">borrow_mut</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #61AFEF;">String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>(</span><span style="color: #C6CCD7;">message</span><span>));</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> #[test]</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> fn</span><span style="color: #B57EDC;"> it_sends_an_over_75_percent_warning_message</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // --snip--</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> mock_messenger</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> MockMessenger</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> limit_tracker</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> LimitTracker</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">mock_messenger</span><span>,</span><span style="color: #56B6C2;"> 100</span><span>);</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> limit_tracker</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">set_value</span><span>(</span><span style="color: #56B6C2;">75</span><span>);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;"> assert_eq!</span><span>(</span><span style="color: #C6CCD7;">mock_messenger</span><span style="color: #E06C75;">.</span><span>sent_messages</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">borrow</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">len</span><span>(),</span><span style="color: #56B6C2;"> 1</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {}</span></span></code></pre><h4 id="yuan-li">原理</h4>
<p><code>borrow</code> 返回 <code>Ref<T></code>，而 <code>borrow_mut</code> 则返回 <code>RefMut<T></code> 二者都是智能指针。类似的，<code>RefCell</code> 只允许有多个不可变借用或一个可变借用。如果违反规则的话，相比编译时出错， <code>RefCell<T></code> 会在运行时 panic。</p>
<h3 id="rc-t-he-refcell-t-da-pei-shi-yong"><code>Rc<T></code> 和 <code>RefCell<T></code>搭配使用</h3>
<ul>
<li><code>Rc<T></code> 允许统一数据有多个所有者但不可变引用</li>
<li><code>RefCell</code> 可以返回可变引用修改不可变引用的值</li>
</ul>
<p>由此，如果 <code>Rc<RefCell<T>></code> 这样存放数据，就可以得到多个所有者<strong>并且</strong>可以修改的值了。</p>
<p>例子：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span>#[derive(</span><span style="color: #E5C07B;">Debug</span><span>)]</span></span>
<span class="giallo-l"><span style="color: #61AFEF;">enum</span><span style="color: #E5C07B;"> List</span><span> {</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #E5C07B;">Rc</span><span><</span><span style="color: #E5C07B;">RefCell</span><span><</span><span style="color: #E5C07B;">i32</span><span>>>,</span><span style="color: #E5C07B;"> Rc</span><span><</span><span style="color: #E5C07B;">List</span><span>>),</span></span>
<span class="giallo-l"><span style="color: #E5C07B;"> Nil</span><span>,</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">use crate::</span><span style="color: #61AFEF;">List</span><span style="color: #E06C75;">::</span><span>{</span><span style="color: #E5C07B;">Cons</span><span>,</span><span style="color: #E5C07B;"> Nil</span><span>};</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">rc</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Rc</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">cell</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">RefCell</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> value</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">RefCell</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">5</span><span>));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #B57EDC;">Cons</span><span>(</span><span style="color: #61AFEF;">Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">value</span><span>),</span><span style="color: #61AFEF;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #E5C07B;">Nil</span><span>)));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> b</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #61AFEF;">Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">RefCell</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">6</span><span>)),</span><span style="color: #61AFEF;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> c</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> Cons</span><span>(</span><span style="color: #61AFEF;">Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">RefCell</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">10</span><span>)),</span><span style="color: #61AFEF;"> Rc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">a</span><span>));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;">value</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">borrow_mut</span><span>()</span><span style="color: #E06C75;"> +=</span><span style="color: #56B6C2;"> 10</span><span>;</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> // 这里*解引用得到 RefCell，然后对其调用.borrow_mut</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">a after = </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">b after = </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> b</span><span>);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">c after = </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> c</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="li-yong-thread-spawn-shi-xian-duo-xian-cheng">利用 <code>thread::spawn</code> 实现多线程</h3>
<p>结合闭包和 <code>spawn</code> 函数产生子线程：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span>thread;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> v</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> vec!</span><span>[</span><span style="color: #56B6C2;">1</span><span>,</span><span style="color: #56B6C2;"> 2</span><span>,</span><span style="color: #56B6C2;"> 3</span><span>];</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> a</span><span style="color: #E06C75;"> =</span><span> '</span><span style="color: #98C379;">a</span><span>';</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> handle</span><span style="color: #E06C75;"> =</span><span style="color: #61AFEF;"> thread</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">spawn</span><span>(</span><span style="color: #E06C75;">move ||</span><span> {</span><span style="color: #5F6672;font-style: italic;">//转移所有权</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Here's a vector: </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> v</span><span>);</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Here's a character: </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>);</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> print!</span><span>("</span><span style="color: #98C379;">Here's a character: </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> a</span><span>);</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> //println!("Here's a vector: {:?}", v);会报错，无所有权</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> handle</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">join</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span><span style="color: #5F6672;font-style: italic;">//等待子线程结束</span></span>
<span class="giallo-l"><span>}</span></span></code></pre><h3 id="li-yong-mpsc-channel-chuang-jian-tong-dao-yong-yu-jin-cheng-jian-tong-xun">利用 <code>mpsc::channel</code> 创建通道用于进程间通讯</h3>
<p><code>mpsc</code> 的含义是 <em>mutiple producer, single consumer</em>。</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span>thread;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">sync</span><span style="color: #E06C75;">::</span><span>mpsc;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span> (</span><span style="color: #C6CCD7;">tx</span><span>,</span><span style="color: #C6CCD7;"> rx</span><span>)</span><span style="color: #E06C75;"> =</span><span style="color: #61AFEF;"> mpsc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">channel</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> thread</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">spawn</span><span>(</span><span style="color: #E06C75;">move ||</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> val</span><span style="color: #E06C75;"> =</span><span style="color: #61AFEF;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">hi</span><span>");</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> tx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">send</span><span>(</span><span style="color: #C6CCD7;">val</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> received</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> rx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">recv</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;"> //rx.try_recv()尝试接收但是不阻塞</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Got: </span><span>{}",</span><span style="color: #C6CCD7;"> received</span><span>);</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>使用 <code>send</code> 发送讯息，同样送走了变量的所有权（堆）。</p>
<p>可以将 <code>rx</code> 作为一个迭代器使用：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span>thread;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">sync</span><span style="color: #E06C75;">::</span><span>mpsc;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">time</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Duration</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span> (</span><span style="color: #C6CCD7;">tx</span><span>,</span><span style="color: #C6CCD7;"> rx</span><span>)</span><span style="color: #E06C75;"> =</span><span style="color: #61AFEF;"> mpsc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">channel</span><span>();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #61AFEF;"> thread</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">spawn</span><span>(</span><span style="color: #E06C75;">move ||</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> vals</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> vec!</span><span>[</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">hi</span><span>"),</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">from</span><span>"),</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">the</span><span>"),</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> String</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from</span><span>("</span><span style="color: #98C379;">thread</span><span>"),</span></span>
<span class="giallo-l"><span> ];</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> val</span><span style="color: #E06C75;"> in</span><span style="color: #C6CCD7;"> vals</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> tx</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">send</span><span>(</span><span style="color: #C6CCD7;">val</span><span>)</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> thread</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">sleep</span><span>(</span><span style="color: #61AFEF;">Duration</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">from_secs</span><span>(</span><span style="color: #56B6C2;">1</span><span>));</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> received</span><span style="color: #E06C75;"> in</span><span style="color: #C6CCD7;"> rx</span><span> {</span><span style="color: #5F6672;font-style: italic;">//这里！阻塞地等待</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Got: </span><span>{}",</span><span style="color: #C6CCD7;"> received</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>如果要实现多个生产者端口，可以使用 <code>clone</code> 得到多个 <code>tx</code>。</p>
<h3 id="mutex-t-hu-chi-qi"><code>Mutex<T></code> 互斥器</h3>
<p>单线程 <code>Mutex</code> 的一个简单实例：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">sync</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Mutex</span><span>;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> m</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Mutex</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">5</span><span>);</span><span style="color: #5F6672;font-style: italic;">//创建Mutex</span></span>
<span class="giallo-l"><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> num</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> m</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">lock</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span><span style="color: #5F6672;font-style: italic;">//返回Mutex内的可变引用</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;">num</span><span style="color: #E06C75;"> =</span><span style="color: #56B6C2;"> 6</span><span>;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">m = </span><span>{</span><span style="color: #98C379;">:?</span><span>}",</span><span style="color: #C6CCD7;"> m</span><span>);</span><span style="color: #5F6672;font-style: italic;">//将发现m内为6</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>如果多个线程都要使用一个 <code>Mutex</code> ，那就不好搞啦，因为 <code>spawn</code> 创建的子线程如若要调用 <code>Mutex</code> ，则 <code>Mutex</code> 所有权进入子线程，后面创建的子线程就不能用了，如下：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">sync</span><span style="color: #E06C75;">::</span><span style="color: #E5C07B;">Mutex</span><span>;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span>thread;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> counter</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Mutex</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">0</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> handles</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> vec!</span><span>[];</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> _</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..</span><span style="color: #56B6C2;">10</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> handle</span><span style="color: #E06C75;"> =</span><span style="color: #61AFEF;"> thread</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">spawn</span><span>(</span><span style="color: #E06C75;">move ||</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> num</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> counter</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">lock</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;">num</span><span style="color: #E06C75;"> +=</span><span style="color: #56B6C2;"> 1</span><span>;</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> handles</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #C6CCD7;">handle</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> handle</span><span style="color: #E06C75;"> in</span><span style="color: #C6CCD7;"> handles</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> handle</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">join</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Result: </span><span>{}",</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;">counter</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">lock</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>());</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"><span style="color: #5F6672;font-style: italic;">// 这段代码无法编译通过</span></span></code></pre>
<p>但是我们也不能用之前提到过的智能指针 <code>Rc<T></code> 实现多引用，我们需要一个和它类似的类型。</p>
<p>从例子也可以看出， <code>Mutex</code> 也是具有内部可变性的。</p>
<h4 id="arc-t-yuan-zi-yin-yong-ji-shu"><code>Arc<T></code> 原子引用计数</h4>
<p>使用例：</p>
<pre class="giallo" style="color: #A9B2C3; background-color: #21252B;"><code data-lang="rust"><span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span style="color: #61AFEF;">sync</span><span style="color: #E06C75;">::</span><span>{</span><span style="color: #E5C07B;">Mutex</span><span>,</span><span style="color: #E5C07B;"> Arc</span><span>};</span></span>
<span class="giallo-l"><span style="color: #E06C75;">use</span><span style="color: #61AFEF;"> std</span><span style="color: #E06C75;">::</span><span>thread;</span></span>
<span class="giallo-l"><span style="color: #E06C75;">fn</span><span style="color: #B57EDC;"> main</span><span>() {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> counter</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #61AFEF;">Mutex</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">new</span><span>(</span><span style="color: #56B6C2;">0</span><span>));</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> handles</span><span style="color: #E06C75;"> =</span><span style="color: #B57EDC;"> vec!</span><span>[];</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> _</span><span style="color: #E06C75;"> in</span><span style="color: #56B6C2;"> 0</span><span style="color: #E06C75;">..</span><span style="color: #56B6C2;">10</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> counter</span><span style="color: #E06C75;"> =</span><span style="color: #E5C07B;"> Arc</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">clone</span><span>(</span><span style="color: #E06C75;">&</span><span style="color: #C6CCD7;">counter</span><span>);</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let</span><span style="color: #C6CCD7;"> handle</span><span style="color: #E06C75;"> =</span><span style="color: #61AFEF;"> thread</span><span style="color: #E06C75;">::</span><span style="color: #B57EDC;">spawn</span><span>(</span><span style="color: #E06C75;">move ||</span><span> {</span></span>
<span class="giallo-l"><span style="color: #61AFEF;"> let mut</span><span style="color: #C6CCD7;"> num</span><span style="color: #E06C75;"> =</span><span style="color: #C6CCD7;"> counter</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">lock</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;">num</span><span style="color: #E06C75;"> +=</span><span style="color: #56B6C2;"> 1</span><span>;</span></span>
<span class="giallo-l"><span> });</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> handles</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">push</span><span>(</span><span style="color: #C6CCD7;">handle</span><span>);</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #E06C75;"> for</span><span style="color: #C6CCD7;"> handle</span><span style="color: #E06C75;"> in</span><span style="color: #C6CCD7;"> handles</span><span> {</span></span>
<span class="giallo-l"><span style="color: #C6CCD7;"> handle</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">join</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>();</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #B57EDC;"> println!</span><span>("</span><span style="color: #98C379;">Result: </span><span>{}",</span><span style="color: #E06C75;"> *</span><span style="color: #C6CCD7;">counter</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">lock</span><span>()</span><span style="color: #E06C75;">.</span><span style="color: #B57EDC;">unwrap</span><span>());</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>