Everyone loves //go:embed: two lines and your templates, migrations and web assets ship inside a single binary. But how does a comment end up as bytes in your program? This talk follows one directive through every stage of go build: package discovery, the lexer, pattern resolution and embedcfg, the parser’s syntax tree, the compiler’s static data and finally the linker. It then covers what embedding costs you, from leaked secrets with the all: prefix to rebuilding for every change, and how to test embedded file systems with os.DirFS, fs.FS and testing/fstest.
//go:embed: From Comment to Binary - Linker Internals, Tradeoffs and Production Patterns
This talk follows a single //go:embed directive through go build, the compiler and the linker, then looks at what embedding costs you and how to test it.
Sources & links
Barak Amar. lakeFS blog.
1. Mastering Embed in Go 1.16
Russ Cox, 2 September 2020. Proposal accepted for Go 1.16.
2. golang/go#41191: embed, cmd/go: add support for embedded files
The go command walking a package directory and noting every //go:embed it passes.
3. (*Context).Import
The scan that reads on past the imports and picks out //go:embed comments as tokens.
4. readGoInfo
Patterns turned into real files and trees against the filesystem.
5. resolveEmbed
The struct the go command builds from the Patterns and Files maps, and marshals to JSON.
6. The embedcfg struct
Writing embedcfg to the work directory and passing -embedcfg to the compiler.
7. Writing embedcfg
The switch nearly every //go:embed error message comes from.
8. checkEmbed
The compiler laying the variable out in memory and writing the file's bytes as static data.
9. WriteEmbed
Russ Cox & Brad Fitzpatrick, July 2020.
10. Draft design: Go command support for embedded static assets
The companion proposal that makes embed.FS compose with the rest of the standard library.
11. golang/go#41190: io/fs: add file system interfaces
D's import() and Zig's @embedFile are both expressions; Go shipped a file system.
12. Embedding in other languages
TestFS and MapFS in the standard library.
13. testing/fstest