Skip to content

Commit 406b74e

Browse files
Copilotdrmingdrmer
Copilot
authored and
drmingdrmer
committed
README: add protobuf docs; fix CI for modern Go toolchain
Add "Slim internal" section to README.md and docs/README.md.j2 covering protobuf data structures, usage, and regeneration instructions. Update CI workflows: - Bump actions/cache v2->v4, actions/checkout v2->v4, actions/setup-go v2->v5 - Upgrade golangci-lint to v1.61.0 with action v6, pin Go 1.23.x - Bump local-lint Go from 1.18.x to 1.23.x for modern lint tools Reformat Go source files with gofmt to match Go 1.23 doc comment style (tab indentation in code blocks, # headings). Fixes #5
1 parent e045937 commit 406b74e

File tree

13 files changed

+175
-140
lines changed

13 files changed

+175
-140
lines changed

‎.github/workflows/golangci-lint.yml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,16 @@ jobs:
1111
name: lint
1212
runs-on: ubuntu-latest
1313
steps:
14-
- uses: actions/checkout@v2
14+
- uses: actions/checkout@v4
15+
16+
- uses: actions/setup-go@v5
17+
with:
18+
go-version: '1.23.x'
1519

1620
- name: golangci-lint
17-
uses: golangci/golangci-lint-action@v3
21+
uses: golangci/golangci-lint-action@v6
1822
with:
19-
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
20-
version: v1.29
23+
version: v1.61.0
2124

2225
# Optional: working directory, useful for monorepos
2326
# working-directory: somedir

‎.github/workflows/local-lint.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,23 +8,23 @@ jobs:
88
strategy:
99
matrix:
1010
go-version:
11-
- 1.18.x
11+
- '1.23.x'
1212
os:
1313
- ubuntu-latest
1414

1515
runs-on: ${{ matrix.os }}
1616

1717
steps:
1818
- name: Install Go
19-
uses: actions/setup-go@v2
19+
uses: actions/setup-go@v5
2020
with:
2121
go-version: ${{ matrix.go-version }}
2222

2323
- name: checkout
24-
uses: actions/checkout@v2
24+
uses: actions/checkout@v4
2525

2626
- name: cache
27-
uses: actions/cache@v2
27+
uses: actions/cache@v4
2828
with:
2929
path: ~/go/pkg/mod
3030
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}

‎.github/workflows/test-short.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,15 +16,15 @@ jobs:
1616

1717
steps:
1818
- name: Install Go
19-
uses: actions/setup-go@v2
19+
uses: actions/setup-go@v5
2020
with:
2121
go-version: ${{ matrix.go-version }}
2222

2323
- name: checkout
24-
uses: actions/checkout@v2
24+
uses: actions/checkout@v4
2525

2626
- name: cache
27-
uses: actions/cache@v2
27+
uses: actions/cache@v4
2828
with:
2929
path: ~/go/pkg/mod
3030
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}

‎.github/workflows/test.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,15 +19,15 @@ jobs:
1919

2020
steps:
2121
- name: Install Go
22-
uses: actions/setup-go@v2
22+
uses: actions/setup-go@v5
2323
with:
2424
go-version: ${{ matrix.go-version }}
2525

2626
- name: checkout
27-
uses: actions/checkout@v2
27+
uses: actions/checkout@v4
2828

2929
- name: cache
30-
uses: actions/cache@v2
30+
uses: actions/cache@v4
3131
with:
3232
path: ~/go/pkg/mod
3333
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}

‎README.md‎

Lines changed: 38 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ corresponding serialization APIs to persisting them on-disk or for transport.
3131
- [Try it](#try-it)
3232
- [Versions](#versions)
3333
- [Who are using slim](#who-are-using-slim)
34+
- [Slim internal](#slim-internal)
35+
- [Protobuf data structures](#protobuf-data-structures)
36+
- [Regenerating protobuf code](#regenerating-protobuf-code)
3437
- [Feedback and contributions](#feedback-and-contributions)
3538
- [Authors](#authors)
3639
- [License](#license)
@@ -469,52 +472,61 @@ a newer version `y`.
469472

470473

471474
<!-- TODO add FAQ -->
472-
<!-- TODO add serialization explanation, on-disk data structure etc. -->
473475

474476
## Who are using slim
475477

476478
<span> <span> ![][baishancloud-favicon] </span> <span> [baishancloud][] </span> </span>
477479

478-
<!-- ## Slim internal -->
480+
## Slim internal
479481

480-
<!-- ### Built With -->
482+
### Protobuf data structures
481483

482-
<!-- - [protobuf][] - Define on-disk data-structure and serialization engine. -->
483-
<!-- - [dep][] - Dependency Management. -->
484-
<!-- - [semver][] - For versioning data-structure. -->
484+
Slim uses [protobuf][] to define its on-disk data structures and as its
485+
serialization engine. All `.proto` files use **proto3** syntax.
485486

486-
<!-- ### Directory Layout -->
487+
The protobuf definitions are in the following files:
487488

488-
<!-- We follow the: [golang-standards-project-layout][]. -->
489+
- `array/bitmap.proto`: `Bits` – a bitmap with rank index, used as the
490+
building block for sparse arrays.
489491

490-
<!-- [> TODO read the doc and add more standards <] -->
492+
- `array/array.proto`: `Array32` – a 32-bit sparse array backed by bitmaps
493+
and offset tables.
491494

492-
<!-- - `vendor/`: dependency packages. -->
493-
<!-- - `prototype/`: on-disk data-structure. -->
494-
<!-- - `docs/`: documents about design, trade-off, etc -->
495-
<!-- - `tools/`: documents about design, trade-off, etc -->
496-
<!-- - `expamples/`: documents about design, trade-off, etc -->
495+
- `trie/slim.proto`: `Bitmap`, `VLenArray`, and `Slim` – the core trie
496+
structures. `Slim` stores node-type bitmaps, inner-node label bitmaps,
497+
short-bitmap tables, inner/leaf prefixes, and serialized leaf values.
497498

498-
<!-- Other directories are sub-package. -->
499+
These structures are serialized with `proto.Marshal()` and deserialized with
500+
`proto.Unmarshal()` from the [`github.com/golang/protobuf`][protobuf-go]
501+
package (v1.3.1).
499502

503+
### Regenerating protobuf code
500504

501-
<!-- ### Versioning -->
505+
The generated Go files (`*.pb.go`) should **not** be edited by hand. To
506+
regenerate them after modifying a `.proto` file:
502507

503-
<!-- We use [SemVer](http://semver.org/) for versioning. -->
508+
1. Install `protoc` (the Protocol Buffers compiler).
509+
See [protoc installation][protoc-install].
504510

505-
<!-- For the versions available, see the [tags on this repository](https://github.com/your/project/tags). -->
511+
2. Install the Go protobuf plugin:
506512

507-
<!-- ### Data structure explained -->
508-
<!-- [> TODO <] -->
513+
```sh
514+
go install github.com/golang/protobuf/protoc-gen-go@latest
515+
```
509516

510-
<!-- ## Limitation -->
511-
<!-- [> TODO <] -->
517+
3. Re-generate the Go source:
512518

519+
```sh
520+
# array package (has a go:generate directive in array/gen.go)
521+
go generate ./array/...
513522
514-
<!-- - [ ] bitrie: 1 byte-per-key implementation. -->
515-
<!-- - [ ] balanced bitrie: which gives better worst-case performance. -->
516-
<!-- - [ ] generalised API as a drop-in replacement for map etc. -->
523+
# trie package
524+
cd trie && protoc --proto_path=. --go_out=. slim.proto
525+
```
517526

527+
> **Note:** `trie/slim.proto` was originally built with `protoc-gen-go`
528+
> **v1.2.0**. When regenerating, make sure the generated code is compatible
529+
> with the module dependency `github.com/golang/protobuf v1.3.1`.
518530

519531
## Feedback and contributions
520532

@@ -657,6 +669,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
657669
658670
[dep]: https://github.com/golang/dep
659671
[protobuf]: https://github.com/protocolbuffers/protobuf
672+
[protobuf-go]: https://github.com/golang/protobuf
660673
[semver]: http://semver.org/
661674
662675
[protoc-install]: http://google.github.io/proto-lens/installing-protoc.html

‎array/array.go‎

Lines changed: 25 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -3,21 +3,21 @@
33
// Unlike a normal array, it does not allocate space for an absent element.
44
// In some way it is like a map[int32]interface{} .
55
//
6-
// Memory overhead
6+
// # Memory overhead
77
//
88
// Internally it use a bitmap to indicate at which index there is an element.
99
//
1010
// This implementation allocate 1 bit for every abscent or present element to
1111
// inidicate if there is an element at this position.
1212
// Thus the memory overhead is about 1 bit / load-factor.
1313
//
14-
// count(elements)
15-
// load-factor = ----------------
16-
// max(indexes)
14+
// count(elements)
15+
// load-factor = ----------------
16+
// max(indexes)
1717
//
1818
// An array with load factor = 0.5 requires about 2 extra bit pre present element.
1919
//
20-
// Implementation note
20+
// # Implementation note
2121
//
2222
// - The bottom level Array32 is a protobuf message that defines in memory
2323
// and on-disk structure.
@@ -29,21 +29,22 @@
2929
// efficient. "U32" accepts only uint32 as its element thus its performance is
3030
// much better.
3131
//
32-
// Array U32 U64 // ready-to-use types
33-
// `----. | .----'
34-
// v v v
35-
// Base // access supporting methods.
36-
// |
37-
// v
38-
// protobuf:Array32 // in-memory and on-disk structure.
32+
// Array U32 U64 // ready-to-use types
33+
// `----. | .----'
34+
// v v v
35+
// Base // access supporting methods.
36+
// |
37+
// v
38+
// protobuf:Array32 // in-memory and on-disk structure.
3939
//
40-
// Performance note
40+
// # Performance note
4141
//
4242
// A Get involves at least 2 memory access to a.Bitmaps and a.Elts.
4343
//
4444
// An "Array" of general type requires one additional alloc for a Get:
45-
// // when Decode convert a concrete type to interface{}
46-
// a.EltEncoder.Decode(bs)
45+
//
46+
// // when Decode convert a concrete type to interface{}
47+
// a.EltEncoder.Decode(bs)
4748
//
4849
// An array of specific type such as "U32" does not requires additional alloc.
4950
package array
@@ -57,16 +58,18 @@ import (
5758
// Array is a space efficient array of fixed-size element.
5859
//
5960
// A fixed size type could be:
60-
// int32
61-
// struct { X int32; Y uint16 }
61+
//
62+
// int32
63+
// struct { X int32; Y uint16 }
6264
//
6365
// A non-fixed size type could be:
64-
// int
65-
// []uint32
66-
// map
67-
// etc.
6866
//
69-
// Performance note
67+
// int
68+
// []uint32
69+
// map
70+
// etc.
71+
//
72+
// # Performance note
7073
//
7174
// A general Array.Get is implemented with reflect.
7275
// Benchmark shows a Get() costs ~ 168 ns/op and involves 4 alloc.

‎array/base.go‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -149,13 +149,14 @@ func (a *Base) Get(idx int32) (interface{}, bool) {
149149

150150
// GetBytes retrieves the raw data of value in []byte at "idx" and return it.
151151
//
152-
// Performance note
152+
// # Performance note
153153
//
154154
// Involves 2 memory access:
155-
// a.Bitmaps
156-
// a.Elts
157155
//
158-
// Involves 0 alloc
156+
// a.Bitmaps
157+
// a.Elts
158+
//
159+
// # Involves 0 alloc
159160
//
160161
// Since 0.2.0
161162
func (a *Base) GetBytes(idx int32, eltsize int) ([]byte, bool) {

‎docs/README.md.j2‎

Lines changed: 35 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -234,52 +234,61 @@ Change-log: [Change-log](docs/change-log.yaml)
234234

235235

236236
<!-- TODO add FAQ -->
237-
<!-- TODO add serialization explanation, on-disk data structure etc. -->
238237

239238
## Who are using slim
240239

241240
<span> <span> ![][baishancloud-favicon] </span> <span> [baishancloud][] </span> </span>
242241

243-
<!-- ## Slim internal -->
242+
## Slim internal
244243

245-
<!-- ### Built With -->
244+
### Protobuf data structures
246245

247-
<!-- - [protobuf][] - Define on-disk data-structure and serialization engine. -->
248-
<!-- - [dep][] - Dependency Management. -->
249-
<!-- - [semver][] - For versioning data-structure. -->
246+
Slim uses [protobuf][] to define its on-disk data structures and as its
247+
serialization engine. All `.proto` files use **proto3** syntax.
250248

251-
<!-- ### Directory Layout -->
249+
The protobuf definitions are in the following files:
252250

253-
<!-- We follow the: [golang-standards-project-layout][]. -->
251+
- `array/bitmap.proto`: `Bits` – a bitmap with rank index, used as the
252+
building block for sparse arrays.
254253

255-
<!-- [> TODO read the doc and add more standards <] -->
254+
- `array/array.proto`: `Array32` – a 32-bit sparse array backed by bitmaps
255+
and offset tables.
256256

257-
<!-- - `vendor/`: dependency packages. -->
258-
<!-- - `prototype/`: on-disk data-structure. -->
259-
<!-- - `docs/`: documents about design, trade-off, etc -->
260-
<!-- - `tools/`: documents about design, trade-off, etc -->
261-
<!-- - `expamples/`: documents about design, trade-off, etc -->
257+
- `trie/slim.proto`: `Bitmap`, `VLenArray`, and `Slim` – the core trie
258+
structures. `Slim` stores node-type bitmaps, inner-node label bitmaps,
259+
short-bitmap tables, inner/leaf prefixes, and serialized leaf values.
262260

263-
<!-- Other directories are sub-package. -->
261+
These structures are serialized with `proto.Marshal()` and deserialized with
262+
`proto.Unmarshal()` from the [`github.com/golang/protobuf`][protobuf-go]
263+
package (v1.3.1).
264264

265+
### Regenerating protobuf code
265266

266-
<!-- ### Versioning -->
267+
The generated Go files (`*.pb.go`) should **not** be edited by hand. To
268+
regenerate them after modifying a `.proto` file:
267269

268-
<!-- We use [SemVer](http://semver.org/) for versioning. -->
270+
1. Install `protoc` (the Protocol Buffers compiler).
271+
See [protoc installation][protoc-install].
269272

270-
<!-- For the versions available, see the [tags on this repository](https://github.com/your/project/tags). -->
273+
2. Install the Go protobuf plugin:
271274

272-
<!-- ### Data structure explained -->
273-
<!-- [> TODO <] -->
275+
```sh
276+
go install github.com/golang/protobuf/protoc-gen-go@latest
277+
```
274278

275-
<!-- ## Limitation -->
276-
<!-- [> TODO <] -->
279+
3. Re-generate the Go source:
277280

281+
```sh
282+
# array package (has a go:generate directive in array/gen.go)
283+
go generate ./array/...
278284

279-
<!-- - [ ] bitrie: 1 byte-per-key implementation. -->
280-
<!-- - [ ] balanced bitrie: which gives better worst-case performance. -->
281-
<!-- - [ ] generalised API as a drop-in replacement for map etc. -->
285+
# trie package
286+
cd trie && protoc --proto_path=. --go_out=. slim.proto
287+
```
282288

289+
> **Note:** `trie/slim.proto` was originally built with `protoc-gen-go`
290+
> **v1.2.0**. When regenerating, make sure the generated code is compatible
291+
> with the module dependency `github.com/golang/protobuf v1.3.1`.
283292

284293
## Feedback and contributions
285294

@@ -422,6 +431,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
422431

423432
[dep]: https://github.com/golang/dep
424433
[protobuf]: https://github.com/protocolbuffers/protobuf
434+
[protobuf-go]: https://github.com/golang/protobuf
425435
[semver]: http://semver.org/
426436

427437
[protoc-install]: http://google.github.io/proto-lens/installing-protoc.html

0 commit comments

Comments
 (0)