Simulate File I/O Faults

IOChaos is a type of fault in Chaos Mesh. By creating an IOChaos experiment, you can simulate a scenario of file system fault. Currently, IOChaos supports the following fault types:

• : delays file system calls
• fault: returns an error for filesystem calls
• attrOverride: modifies file properties
• mistake: makes the file read or write a wrong value

For specific features, refer to .

Notes

1. Before creating an IOChaos experiment, make sure there is no Control Manager of Chaos Mesh running on the target Pod.

2. IOChaos may damage your data. Use IOChaos with caution in the production environment.

1. Open Chaos Dashboard, and click NEW EXPERIMENT on the page to create a new experiment:

2. In the Choose a Target area, choose FILE SYSTEM INJECTION and select a specific fault type, such as LATENCY.

   

3. Fill out the experiment information, and specify the experiment scope and the scheduled experiment duration.

4. Submit the experiment information.

Create experiments using the YAML files

1. In this configuration example, Chaos Mesh injects a delay into the directory /var/run/etcd and causes latency of 100 milliseconds to all file system operations (including read, writing, list contents, and so on) in this directory.

2. After the configuration file is prepared, use kubectl to create an experiment:

   kubectl apply -f ./io-latency.yaml

1. After the configuration file is prepared, use to create an experiment:

   kubectl apply -f ./io-fault.yaml

1. Write the experiment configuration to the io-attr.yaml file:

   In this configuration example, Chaos Mesh injects /var/run/etcd directories attrOverride fault, giving a 10% probability that all file system operations in this directory will change the target file permissions to 72 (110 in octal), which will allow files to be executed only by the owner and their group and not authorized to perform other actions.

2. After the configuration file is prepared, use kubectl to create an experiment:

   kubectl apply -f ./io-attr.yaml

1. Write the experiment configuration to the io-mistake.yaml file:

   In this configuration example, Chaos Mesh injects read and write faults into the directory /var/run/etcd, which gives a 10% probability of failure in the read and write operations under this directory. During this process, one random position with a maximum length of 10 bytes will be replaced with 0 bytes.

2. After the configuration file is prepared, use kubectl to create an experiment:

General fields

Fields related to action

• latency

  Parameter	Type	Description	Default value	Required	Example
  delay	string	Specific delay time		Yes	100 ms

• fault

  For common error numbers, see Appendix B.

• attrOverride

  Parameter	Type	Description	Default value	Required	Example
  attr	AttrOverrideSpec	Specific property override rules		Yes	As follows

  AttrOverrideSpec is defined as follows:

  TimeSpec is defined as follows:

  Parameter	Type	Description	Default value	Required	Example
  sec	int	timestamp in seconds		No
  nsec	int	Timestamp in nanoseconds		No

  For the specific meaning of parameters, you can refer to .

• mistake

  MistakeSpec is defined as follows:

  Parameter	Type	Description	Default value	Required	Example
  filling	string	The wrong data to be filled. Only zero (fill 0) or random (fill random bytes) are supported.		Yes
  maxOccurrences	int	Maximum number of errors in each operation.		Yes	1
  maxLength	int	Maximum length of each error (in bytes).		Yes	1

::warning It is suggested that you only use mistake on READ and WRITE file system calls. Using mistake on other file system calls may lead to unexpected consequences, including but not limited to file system damage and program crashes. :::

If you are not sure about the effect of a certain Chaos, you can use to test the feature locally. Chaos Mesh also uses toda to implement IOChaos.

Appendix A: methods type

• lookup
• forget
• getattr
• setattr
• readlink
• mknod
• mkdir
• unlink
• rmdir
• symlink
• rename
• link
• open
• read
• write
• release
• fsync
• opendir
• readdir
• releasedir
• fsyncdir
• statfs
• setxattr
• getxattr
• listxattr
• removexatr
• access
• create
• getlk
• setlk
• bmap

• 1: Operation not permitted
• 2: No such file or directory
• 5: I/O error
• 6: No such device or address
• 12: Out of memory
• 16: Device or resource busy
• 17: File exists
• 20: Not a directory
• 22: Invalid argument
• 24: Too many open files
• 28: No space left on device

For more information, refer to Linux source code.