failpoint

An implementation of failpoints for Golang. Fail points are used to add code points where errors may be injected in a user controlled fashion. Fail point is a code snippet that is only executed when the corresponding failpoint is active.

Quick Start (use failpoint-ctl)

1. Build failpoint-ctl from source

   root@kitploit:~

   git clone https://github.com/pingcap/failpoint.git
   cd failpoint
   make
   ls bin/failpoint-ctl
   

2. Inject failpoints to your program, eg:

   root@kitploit:~

   package main
   
   import "github.com/pingcap/failpoint"
   
   func main() {
       failpoint.Inject("testPanic", func() {
           panic("failpoint triggerd")
       })
   }
   

3. Transfrom your code with failpoint-ctl enable

4. Build with go build

5. Enable failpoints with GO_FAILPOINTS environment variable

   root@kitploit:~

   GO_FAILPOINTS="main/testPanic=return(true)" ./your-program
   

   Note: GO_FAILPOINTS does not work with InjectCall type of marker.

6. If you use go run to run the test, don't forget to add the generated in your command, like:

Quick Start (use failpoint-toolexec)

1. Build failpoint-toolexec from source

   root@kitploit:~

   git clone https://github.com/pingcap/failpoint.git
   cd failpoint
   make
   ls bin/failpoint-toolexec
   

2. Inject failpoints to your program, eg:

   root@kitploit:~

   package main
   
   import "github.com/pingcap/failpoint"
   
   func main() {
       failpoint.Inject("testPanic", func() {
           panic("failpoint triggerd")
       })
   }
   

3. Use a separate build cache to avoid mixing caches without failpoint-toolexec, and build

   GOCACHE=/tmp/failpoint-cache go build -toolexec path/to/failpoint-toolexec

4. Enable failpoints with GO_FAILPOINTS environment variable

   root@kitploit:~

   GO_FAILPOINTS="main/testPanic=return(true)" ./your-program
   

5. You can also use go run or go test, like:

   root@kitploit:~

   GOCACHE=/tmp/failpoint-cache GO_FAILPOINTS="main/testPanic=return(true)" go run -toolexec path/to/failpoint-toolexec your-program.go
   

Design principles

• Define failpoint in valid Golang code, not comments or anything else

• Failpoint does not have any extra cost

  • Will not take effect on regular logic
  • Will not cause regular code performance regression
  • Failpoint code will not appear in the final binary

• Failpoint routine is writable/readable and should be checked by a compiler

• Generated code by failpoint definition is easy to read

• Keep the line numbers same with the injecting codes(easier to debug)

• Support parallel tests with context.Context

Key concepts

• Failpoint

  Faillpoint is a code snippet that is only executed when the corresponding failpoint is active. The closure will never be executed if failpoint.Disable("failpoint-name-for-demo") is executed.

  root@kitploit:~

  var outerVar = "declare in outer scope"
  failpoint.Inject("failpoint-name-for-demo", func(val failpoint.Value) {
      fmt.Println("unit-test", val, outerVar)
  })
  

• Marker functions

  • It is just an empty function

    • To hint the rewriter to rewrite with an equality statement
    • To receive some parameters as the rewrite rule
    • It will be inline in the compiling time and emit nothing to binary (zero cost)
    • The variables in external scope can be accessed in closure by capturing, and the converted code is still legal because all the captured-variables location in outer scope of IF statement.

  • It is easy to write/read

  • Introduce a compiler check for failpoints which cannot compile in the regular mode if failpoint code is invalid

• Marker funtion list

  • func Inject(fpname string, fpblock func(val Value)) {}
  • func InjectContext(fpname string, ctx context.Context, fpblock func(val Value)) {}
  • func InjectCall(fpname string, args ...any) {}
  • func Break(label ...string) {}

How to inject a failpoint to your program

• You can call failpoint.Inject to inject a failpoint to the call site, where failpoint-name is used to trigger the failpoint and failpoint-closure will be expanded as the body of the IF statement.

  root@kitploit:~

  failpoint.Inject("failpoint-name", func(val failpoint.Value) {
      failpoint.Return("unit-test", val)
  })
  

  The converted code looks like:

  root@kitploit:~

  if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
      return "unit-test", val
  }
  

• failpoint.Value is the value that passes by failpoint.Enable("failpoint-name", "return(5)") which can be ignored.

  root@kitploit:~

  failpoint.Inject("failpoint-name", func(_ failpoint.Value) {
      fmt.Println("unit-test")
  })
  

  OR

  root@kitploit:~

  failpoint.Inject("failpoint-name", func() {
      fmt.Println("unit-test")
  })
  

  And the converted code looks like:

Some complicated failpoints demo

• Inject a failpoint to the IF INITIAL statement or CONDITIONAL expression

  root@kitploit:~

  if a, b := func() {
      failpoint.Inject("failpoint-name", func(val failpoint.Value) {
          fmt.Println("unit-test", val)
      })
  }, func() int { return rand.Intn(200) }(); b > func() int {
      failpoint.Inject("failpoint-name", func(val failpoint.Value) int {
          return val.(int)
      })
      return rand.Intn(3000)
  }() && b < func() int {
      failpoint.Inject("failpoint-name-2", func(val failpoint.Value) {
          return rand.Intn(val.(int))
      })
      return rand.Intn(6000)
  }() {
      a()
      failpoint.Inject("failpoint-name-3", func(val failpoint.Value) {
          fmt.Println("unit-test", val)
      })
  }
  

  The above code block will generate something like this:

  root@kitploit:~

  if a, b := func() {
      if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
          fmt.Println("unit-test", val)
      }
  }, func() int { return rand.Intn(200) }(); b > func() int {
      if val, _err_ := failpoint.Eval(_curpkg_("failpoint-name")); _err_ == nil {
          return val.(int)
      }
      return rand.Intn(3000)
  }() && b < func() int {
      if val, ok := failpoint.Eval(_curpkg_("failpoint-name-2")); ok {
          return rand.Intn(val.(int))
      }
      return rand.Intn(6000)
  }() {
      a()
      if val, ok := failpoint.Eval(_curpkg_("failpoint-name-3")); ok {
          fmt.Println("unit-test", val)
      }
  }
  

• Inject a failpoint to the SELECT statement to make it block one CASE if the failpoint is active

Failpoint name best practice

As you see above, _curpkg_ will automatically wrap the original failpoint name in failpoint.Eval call. You can think of _curpkg_ as a macro that automatically prepends the current package path to the failpoint name. For example,

package ddl // which parent package is `github.com/pingcap/tidb`

func demo() {
	// _curpkg_("the-original-failpoint-name") will be expanded as `github.com/pingcap/tidb/ddl/the-original-failpoint-name`
	if val, ok := failpoint.Eval(_curpkg_("the-original-failpoint-name")); ok {...}
}

You do not need to care about _curpkg_ in your application. It is automatically generated after running failpoint-ctl enable and is deleted with failpoint-ctl disable.

Because all failpoints in a package share the same namespace, we need to be careful to avoid name conflict. There are some recommended naming rules to improve this situation.

• Keep name unique in current subpackage

• Use a self-explanatory name for the failpoint

  You can enable failpoints by environment variables

  root@kitploit:~

  GO_FAILPOINTS="github.com/pingcap/tidb/ddl/renameTableErr=return(100);github.com/pingcap/tidb/planner/core/illegalPushDown=return(true);github.com/pingcap/pd/server/schedulers/balanceLeaderFailed=return(true)"
  

Implementation details

1. Define a group of marker functions
2. Parse imports and prune a source file which does not import a failpoint
3. Traverse AST to find marker function calls
4. Marker function calls will be rewritten with an IF statement, which calls failpoint.Eval to determine whether a failpoint is active and executes failpoint code if the failpoint is enabled

Acknowledgments

• Thanks gofail to provide initial implementation.