failpoint

An implementation of failpoints for Golang. Fail points are used to add code points where errors may be injected in a user controlled fashion. Fail point is a code snippet that is only executed when the corresponding failpoint is active.

Quick Start (use failpoint-ctl)

1. Build failpoint-ctl from source

   git clone https://github.com/pingcap/failpoint.git
   cd failpoint
   make
   ls bin/failpoint-ctl
   

2. Inject failpoints to your program, eg:

   package main
   
   import "github.com/pingcap/failpoint"
   
   func main() {
       failpoint.Inject("testPanic", func() {
           panic("failpoint triggerd")
       })
   }
   

3. Transfrom your code with failpoint-ctl enable

4. Build with go build

5. Enable failpoints with GO_FAILPOINTS environment variable

   GO_FAILPOINTS="main/testPanic=return(true)" ./your-program
   

   Note: GO_FAILPOINTS does not work with InjectCall type of marker.

6. If you use go run to run the test, don't forget to add the generated binding__failpoint_binding__.go in your command, like:

   GO_FAILPOINTS="main/testPanic=return(true)" go run your-program.go binding__failpoint_binding__.go
   

Quick Start (use failpoint-toolexec)

1. Build failpoint-toolexec from source

   git clone https://github.com/pingcap/failpoint.git
   cd failpoint
   make
   ls bin/failpoint-toolexec
   

2. Inject failpoints to your program, eg:

   package main
   
   import "github.com/pingcap/failpoint"
   
   func main() {
       failpoint.Inject("testPanic", func() {
           panic("failpoint triggerd")
       })
   }
   

3. Use a separate build cache to avoid mixing caches without failpoint-toolexec, and build

   GOCACHE=/tmp/failpoint-cache go build -toolexec path/to/failpoint-toolexec

4. Enable failpoints with GO_FAILPOINTS environment variable

   GO_FAILPOINTS="main/testPanic=return(true)" ./your-program
   

5. You can also use go run or go test, like:

   GOCACHE=/tmp/failpoint-cache GO_FAILPOINTS="main/testPanic=return(true)" go run -toolexec path/to/failpoint-toolexec your-program.go
   

Design principles

• Define failpoint in valid Golang code, not comments or anything else

• Failpoint does not have any extra cost

  • Will not take effect on regular logic
  • Will not cause regular code performance regression
  • Failpoint code will not appear in the final binary

• Failpoint routine is writable/readable and should be checked by a compiler

• Generated code by failpoint definition is easy to read

• Keep the line numbers same with the injecting codes(easier to debug)

• Support parallel tests with context.Context

Key concepts

• Failpoint

  Faillpoint is a code snippet that is only executed when the corresponding failpoint is active. The closure will never be executed if failpoint.Disable("failpoint-name-for-demo") is executed.

  var outerVar = "declare in outer scope"
  failpoint.Inject("failpoint-name-for-demo", func(val failpoint.Value) {
      fmt.Println("unit-test", val, outerVar)
  })
  

• Marker functions

  • It is just an empty function

    • To hint the rewriter to rewrite with an equality statement
    • To receive some parameters as the rewrite rule
    • It will be inline in the compiling time and emit nothing to binary (zero cost)
    • The variables in external scope can be accessed in closure by capturing, and the converted code is still legal because all the captured-variables location in outer scope of IF statement.

  • It is easy to write/read

  • Introduce a compiler check for failpoints which cannot compile in the regular mode if failpoint code is invalid

• Marker funtion list

  • func Inject(fpname string, fpblock func(val Value)) {}
  • func InjectContext(fpname string, ctx context.Context, fpblock func(val Value)) {}
  • func InjectCall(fpname string, args ...any) {}
  • func Break(label ...string) {}
  • func Goto(label string) {}
  • func Continue(label ...string) {}
  • func Fallthrough() {}
  • func Return(results ...interface{}) {}
  • func Label(label string) {}

• Supported failpoint environment variable

  failpoint can be enabled by export environment variables with the following patten, which is quite similar to freebsd failpoint SYSCTL VARIABLES

  Note: InjectCall cannot be enabled by environment variables.

  [<percent>%][<count>*]<type>[(args...)][-><more terms>]
  

  The argument specifies which action to take; it can be one of:

  • off: Take no action (does not trigger failpoint code)
  • return: Trigger failpoint with specified argument
  • sleep: Sleep the specified number of milliseconds
  • panic: Panic
  • break: Execute gdb and break into debugger
  • print: Print failpoint path for inject variable
  • pause: Pause will pause until the failpoint is disabled

How to inject a failpoint to your program

• You can call failpoint.Inject to inject a failpoint to the call site, where failpoint-name is used to trigger the failpoint and failpoint-closure will be expanded as the body of the IF statement.

  failpoint.Inject("failpoint-name", func(val failpoint.Value) {
      failpoint.Return("unit-test", val)
  })
  

  The converted code looks like:

  if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
      return "unit-test", val
  }
  

• failpoint.Value is the value that passes by failpoint.Enable("failpoint-name", "return(5)") which can be ignored.

  failpoint.Inject("failpoint-name", func(_ failpoint.Value) {
      fmt.Println("unit-test")
  })
  

  OR

  failpoint.Inject("failpoint-name", func() {
      fmt.Println("unit-test")
  })
  

  And the converted code looks like:

  if _, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
      fmt.Println("unit-test")
  }
  

• Also, the failpoint closure can be a function which takes context.Context. You can do some customized things with context.Context like controlling whether a failpoint is active in parallel tests or other cases. For example,

  failpoint.InjectContext(ctx, "failpoint-name", func(val failpoint.Value) {
      fmt.Println("unit-test", val)
  })
  

  The converted code looks like:

  if val, _err_ := failpoint.EvalContext(ctx, _curpkg_("failpoint-name")); _err_ == nil {
      fmt.Println("unit-test", val)
  }
  

• You can ignore context.Context, and this will generate the same code as above non-context version. For example,

  failpoint.InjectContext(nil, "failpoint-name", func(val failpoint.Value) {
      fmt.Println("unit-test", val)
  })
  

  Becomes