Switch lexer to byte based scanning

[Lurk]

No description provided.

[codecov]

Codecov Report

❌ Patch coverage is 99.61832% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 99.6%. Comparing base (5fbf890) to head (2ad60bf).

Files with missing lines	Patch %	Lines
src/lexer/mod.rs	99.5%	1 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
src/lexer/token.rs	100.0% <100.0%> (ø)
src/op/anchor.rs	100.0% <100.0%> (ø)
src/op/bold.rs	100.0% <100.0%> (ø)
src/op/code.rs	100.0% <100.0%> (ø)
src/op/code_span.rs	100.0% <100.0%> (ø)
src/op/collapsible.rs	100.0% <100.0%> (ø)
src/op/embed.rs	100.0% <100.0%> (ø)
src/op/emphasis.rs	100.0% <100.0%> (ø)
src/op/heading.rs	100.0% <100.0%> (ø)
src/op/highlight.rs	100.0% <100.0%> (ø)
... and 10 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Benchmark for cf15c47

Click to view benchmark

Test	Base	PR	%
ast/small yamd document	1402.1±27.73ns	1377.4±41.51ns	-1.76%
ast/~344kb of YAMD written by humman	631.7±6.01µs	628.9±7.80µs	-0.44%
ast/~344kb with high density of tokens	2.3±0.02ms	2.3±0.02ms	0.00%
ast/~346kb with low density of tokens	54.5±2.35µs	49.3±0.72µs	-9.54%
lexer/small yamd document	2.9±0.03µs	600.7±8.00ns	-79.29%
lexer/~344kb of YAMD written by humman	3.5±0.10ms	484.0±8.46µs	-86.17%
lexer/~344kb with high density of tokens	5.8±0.01ms	2.5±0.03ms	-56.90%
lexer/~346kb with low density of tokens	920.7±47.78µs	305.6±5.05µs	-66.81%
parser/small yamd document	4.1±0.04µs	1813.0±15.63ns	-55.78%
parser/~344kb of YAMD written by humman	3.9±0.12ms	981.2±10.55µs	-74.84%
parser/~344kb with high density of tokens	9.8±0.04ms	6.7±0.06ms	-31.63%
parser/~346kb with low density of tokens	998.9±56.74µs	391.6±4.26µs	-60.80%
roundtrip/small yamd document	11.3±0.21µs	9.1±0.14µs	-19.47%
roundtrip/~344kb of YAMD written by humman	9.0±0.13ms	6.1±0.07ms	-32.22%
roundtrip/~344kb with high density of tokens	23.6±0.25ms	20.5±0.17ms	-13.14%
roundtrip/~346kb with low density of tokens	3.4±0.07ms	2.8±0.03ms	-17.65%

[github-actions]

Benchmark for c38c172

Click to view benchmark

Test	Base	PR	%
ast/small yamd document	888.7±12.25ns	863.8±55.79ns	-2.80%
ast/~344kb of YAMD written by humman	391.0±7.87µs	385.6±7.96µs	-1.38%
ast/~344kb with high density of tokens	1429.9±25.48µs	1440.1±40.90µs	+0.71%
ast/~346kb with low density of tokens	28.3±1.68µs	28.7±1.60µs	+1.41%
lexer/small yamd document	1464.4±117.99ns	565.7±11.62ns	-61.37%
lexer/~344kb of YAMD written by humman	1780.3±96.73µs	340.7±16.34µs	-80.86%
lexer/~344kb with high density of tokens	3.4±0.13ms	2.0±0.03ms	-41.18%
lexer/~346kb with low density of tokens	482.4±55.03µs	149.7±6.16µs	-68.97%
parser/small yamd document	2.3±0.09µs	1277.5±19.42ns	-44.46%
parser/~344kb of YAMD written by humman	2.1±0.08ms	580.5±11.84µs	-72.36%
parser/~344kb with high density of tokens	5.7±0.12ms	4.3±0.10ms	-24.56%
parser/~346kb with low density of tokens	555.6±49.56µs	199.0±3.93µs	-64.18%
roundtrip/small yamd document	6.3±0.35µs	5.4±0.22µs	-14.29%
roundtrip/~344kb of YAMD written by humman	5.1±0.11ms	4.0±0.07ms	-21.57%
roundtrip/~344kb with high density of tokens	14.4±0.18ms	12.9±0.21ms	-10.42%
roundtrip/~346kb with low density of tokens	1870.1±53.11µs	1760.1±34.49µs	-5.88%

[github-actions]

Benchmark for 9249f84

Click to view benchmark

Test	Base	PR	%
ast/small yamd document	1436.7±26.51ns	1418.6±22.13ns	-1.26%
ast/~344kb of YAMD written by humman	617.7±4.69µs	615.9±7.42µs	-0.29%
ast/~344kb of only backslashes	989.1±10.70µs	N/A	N/A
ast/~344kb with high density of tokens	2.2±0.03ms	2.2±0.04ms	0.00%
ast/~346kb with low density of tokens	44.3±0.48µs	52.1±2.81µs	+17.61%
lexer/small yamd document	2.2±0.03µs	908.0±16.10ns	-58.73%
lexer/~344kb of YAMD written by humman	2.8±0.06ms	519.2±7.77µs	-81.46%
lexer/~344kb of only backslashes	252.7±2.46µs	N/A	N/A
lexer/~344kb with high density of tokens	4.3±0.02ms	2.5±0.02ms	-41.86%
lexer/~346kb with low density of tokens	750.8±13.74µs	227.5±2.10µs	-69.70%
parser/small yamd document	3.6±0.05µs	2.2±0.05µs	-38.89%
parser/~344kb of YAMD written by humman	3.3±0.06ms	942.1±18.89µs	-71.45%
parser/~344kb of only backslashes	251.0±3.33µs	N/A	N/A
parser/~344kb with high density of tokens	8.5±0.09ms	6.2±0.04ms	-27.06%
parser/~346kb with low density of tokens	843.8±12.36µs	313.2±4.40µs	-62.88%
roundtrip/small yamd document	10.8±0.15µs	9.0±0.18µs	-16.67%
roundtrip/~344kb of YAMD written by humman	8.3±0.07ms	5.8±0.03ms	-30.12%
roundtrip/~344kb of only backslashes	3.6±0.04ms	N/A	N/A
roundtrip/~344kb with high density of tokens	22.9±0.16ms	20.1±0.12ms	-12.23%
roundtrip/~346kb with low density of tokens	3.0±0.03ms	2.3±0.02ms	-23.33%