Skip to content

security-testing.md

Latest commit

 

History

History
132 lines (102 loc) · 5.63 KB

security-testing.md

File metadata and controls

132 lines (102 loc) · 5.63 KB
Raw
type Test Strategy
title Security Testing
description Fail-point injection and layered security regression testing strategy.
tags
bashkit
testing
security

Security Testing with Fail Points

Status

Implemented

Overview

Bashkit uses fail-rs for fault injection security testing of error handling paths and resource limit enforcement. Fail points are disabled by default, zero runtime overhead when disabled.

cargo test --features failpoints security_ -- --test-threads=1

Available Fail Points

Resource Limits (limits.rs)

Fail Point Actions Security Test Purpose
limits::tick_command skip_increment, force_overflow, corrupt_high Test command limit bypass resistance
limits::tick_loop skip_check, reset_counter Test loop limit bypass resistance
limits::push_function skip_check, corrupt_depth Test recursion limit bypass resistance

Filesystem (fs/memory.rs)

Fail Point Actions Security Test Purpose
fs::read_file io_error, permission_denied, corrupt_data Test read failure handling
fs::write_file io_error, disk_full, permission_denied, partial_write Test write failure handling

Interpreter (interpreter/mod.rs)

Fail Point Actions Security Test Purpose
interp::execute_command panic, error, exit_nonzero Test command execution failure handling

yq atomic replacement (builtins/yq.rs)

Fail Point Actions Security Test Purpose
yq::temp_allocate exhausted Source preservation when no sibling temporary can be allocated
yq::temp_chmod error Temporary cleanup and source preservation on mode-copy failure
yq::temp_rename error Temporary cleanup and source preservation before atomic replacement

The existing fs::write_file actions cover yq temporary-write failures, including the partial-write class. security_yq_inplace_* asserts original bytes and absence of .bashkit-yq-* files after every injected stage.

Usage

In tests: fail::cfg("limits::tick_command", "return(skip_increment)") before, fail::cfg("name", "off") after. Action syntax (return/panic/sleep/pause, probability 10%, count 5*): see fail crate docs.

Security Test Categories

  1. Resource limit bypass: counter corruption, check skipping, overflow/underflow
  2. Filesystem failure: I/O errors, permission denied, disk full, partial writes, data corruption
  3. Interpreter failure: execution errors, panic recovery, unexpected exit codes

Adding New Fail Points

Add fail_point!("module::function", |action| ...) under #[cfg(feature = "failpoints")] at the critical location, document it in this spec, and add tests in tests/security_failpoint_tests.rs.

Best Practices

  1. Always clean up: Call fail::cfg("name", "off") after tests.
  2. Single-threaded tests: Use --test-threads=1 for fail point tests due to global state.
  3. Document actions: List all supported actions in code comments and this spec.
  4. Test both paths: Test that fail points affect behavior AND that normal operation works without them.

Filesystem Certification

The backend-neutral helper in crates/bashkit/tests/support/filesystem_security_conformance.rs certifies binary/null content, unsafe pathname rejection, root-clamped canonical identity, and normalized public error kinds. The integration and RealFs feature suites run the same helper against every production storage adapter. Wrapper cases cover symlink identity, read-only policy, mount boundaries, transactional failures, quota retention, and tar preflight. security_fs_failed_writes_are_atomic runs all fs::write_file failpoint actions and proves prior bytes and usage survive.

JavaScript Security Tests

The JavaScript/TypeScript bindings have a dedicated security test suite at crates/bashkit-js/__test__/security.spec.ts (run: cd crates/bashkit-js && pnpm test). White-box and black-box scenarios across 18 categories:

  1. Resource limit enforcement (TM-DOS)
  2. Output truncation (TM-DOS)
  3. Sandbox escape prevention (TM-ESC)
  4. VFS security, path traversal, file count, nesting, filename limits (TM-DOS, TM-INJ)
  5. Instance isolation (TM-ISO)
  6. Error message safety (TM-INT)
  7. TypeScript wrapper injection prevention (TM-INJ)
  8. Adversarial script inputs, null bytes, deep nesting, expansion bombs
  9. Unicode & encoding attacks (TM-UNI)
  10. Injection via constructor options (TM-INJ)
  11. Concurrency & cancellation (TM-DOS)
  12. Async API security
  13. BashTool metadata safety
  14. Bash feature abuse, traps, special variables, /dev/tcp
  15. Mounted files security
  16. Rapid instance creation/destruction
  17. Edge case inputs
  18. Async factory security

Related Files

  • crates/bashkit/tests/security_failpoint_tests.rs - Fail-point security tests
  • crates/bashkit/tests/threat_model_tests.rs - Threat model tests (51 tests)
  • crates/bashkit/tests/builtin_error_security_tests.rs - Builtin error security tests (39 tests, includes TM-INT-003)
  • crates/bashkit-js/__test__/security.spec.ts - JavaScript security tests (90+ tests, 18 categories)
  • crates/bashkit/src/limits.rs - Resource limit fail points
  • crates/bashkit/src/fs/memory.rs - Filesystem fail points
  • crates/bashkit/src/interpreter/mod.rs - Interpreter fail points, panic catching
  • crates/bashkit/src/builtins/system.rs - Hardcoded system builtins
  • Threat Model - Threat model specification