Refs: offline install via vendored wheels and test verification workflow (see README/playbook updates)

2026-01-08 01:45:25 +08:00
parent 2807982209
commit 369c8595a1
75 changed files with 1094 additions and 388 deletions
--- a/playbook.md
+++ b/playbook.md
@@ -10,13 +10,18 @@ Repository structure (current):
  - `bootstrap_vectors.py` — one-time script to generate test vectors
  - `vectors.json` — frozen expected values (committed)
  - `data/recipient.asc` — test PGP key (committed)
- `.wallet/` — generated wallet artifacts (should be gitignored)
+- `vendor/` — vendored Python wheels for offline installation
+  - `macos-arm64/` — macOS Apple Silicon wheels (Python 3.12)
+  - `linux-x86_64/` — Linux x86_64 wheels (Python 3.12)
+- `.wallet/` — generated wallet artifacts (gitignored)
 - `requirements.in`, `requirements.txt`
+- `install_offline.sh` — automated offline installer
+- `Dockerfile`, `Makefile` — Docker-based build tools
 - `README.md`, `playbook.md`, `LICENSE`

 **Note**: This tool requires a subcommand (e.g., `gen`, `recover`). Running without one displays help. Use `<subcommand> -h` for detailed options.

---
+***

 ## Features

@@ -27,33 +32,101 @@ Repository structure (current):
 - **TTY safety guard + --force**: If stdout is piped/redirected (non-TTY), the tool refuses to print sensitive data unless `--force` is explicitly set (to avoid accidental leaks into logs/files). `isatty()` is the classic way to detect whether stdout is connected to a terminal.
 - **Off-screen mode**: `--off-screen` suppresses printing sensitive data to stdout.
 - **Regression test suite**: Offline tests with frozen vectors ensure derivation logic, PGP fingerprinting, and seed generation remain stable across code changes.
+- **Vendored dependencies**: Pre-built wheels for macOS ARM64 and Linux x86_64 enable true air-gapped installation.

---
+***

 ## Installation

 ### Prerequisites

- Python 3.11+
- Recommended: a virtual environment
+- Python 3.12.x (required for vendored wheels)
+- Git (for cloning repository)

-### Setup
+### Quick Installation (macOS/Linux with Internet)

 ```bash
-python -m venv .venv
-source .venv/bin/activate
-python -m pip install -r requirements.txt
+# Clone repository
+git clone https://github.com/<your-username>/hdwalletpy.git
+cd hdwalletpy
+
+# Run automated installer
+./install_offline.sh
 ```

-### Dependencies (top-level intent)
+The script automatically:
+
+- Detects your platform (macOS ARM64 or Linux x86_64)
+- Creates Python 3.12 virtual environment
+- Installs from vendored wheels (no PyPI access needed)
+- Verifies installation with test suite
+- Leaves you in activated venv
+
+### Air-Gapped Installation (No Internet)
+
+For maximum security on an air-gapped machine (Tails OS, Ubuntu Live USB, etc.):
+
+**On online machine:**
+
+```bash
+# Clone and verify
+git clone https://github.com/<your-username>/hdwalletpy.git
+cd hdwalletpy
+
+# Verify checksums
+cd vendor/linux-x86_64  # or macos-arm64 for Mac
+sha256sum -c SHA256SUMS  # Linux
+shasum -a 256 -c SHA256SUMS  # macOS
+
+# Transfer entire repo to USB
+```
+
+**On air-gapped machine:**
+
+```bash
+# Ensure Python 3.12 is installed
+python3.12 --version
+
+# Navigate to repository
+cd hdwalletpy
+
+# Run installer (creates venv, installs offline)
+./install_offline.sh
+
+# Generate wallet
+python src/pyhdwallet.py gen --off-screen --file
+```
+
+### Manual Installation (Without Script)
+
+```bash
+# Create venv
+python3.12 -m venv .venv
+source .venv/bin/activate
+
+# Install from vendored wheels
+# For macOS ARM64:
+pip install --no-index --find-links=vendor/macos-arm64 -r requirements.txt
+
+# For Linux x86_64:
+pip install --no-index --find-links=vendor/linux-x86_64 -r requirements.txt
+
+# Verify installation
+pytest -v tests/test_vectors.py
+python src/pyhdwallet.py test
+```
+
+### Dependencies (Top-Level Intent)

 - `bip-utils` — BIP39 + BIP derivation logic
 - `PGPy` — encryption to OpenPGP public keys
 - `pynacl` + `base58` — Solana seed/key handling
- `pyzipper` — AES-encrypted ZIP writing (only needed when using `--file`)
+- `pyzipper` — AES-encrypted ZIP writing
 - `pytest` — test framework (development only)

---
+All dependencies are pre-downloaded in `vendor/` for offline use.
+
+***

 ## Quick Start

@@ -100,7 +173,7 @@ python ./src/pyhdwallet.py test
 pytest -v tests/test_vectors.py
 ```

---
+***

 ## Testing

@@ -154,16 +227,75 @@ git commit -m "test: update vectors after intentional derivation change"

 **Warning**: If tests fail after a code change and you didn't intend to change behavior, **do not regenerate vectors**. Fix the code regression instead.

---
+***

-## Outputs and file structure
+## Vendored Dependencies

-### Stdout behavior
+### What is Vendoring?
+
+This repository includes pre-built Python wheels in the `vendor/` directory, enabling installation without internet access or PyPI. This is critical for air-gapped security.
+
+**Supported platforms:**
+
+- macOS ARM64 (M1/M2/M3/M4) - Python 3.12
+- Linux x86_64 (Ubuntu, Tails, Debian) - Python 3.12
+
+### Building Vendor Wheels (Maintainers)
+
+If you need to update dependencies or add new platforms:
+
+```bash
+# Using Makefile (recommended)
+make vendor-all           # Build for both macOS and Linux
+make vendor-macos         # macOS ARM64 only
+make vendor-linux         # Linux x86_64 only (requires Docker)
+make verify-vendor        # Test offline installation
+
+# Commit updated wheels
+git add vendor/
+git commit -m "vendor: update dependencies to latest versions"
+```
+
+**Manual build (macOS):**
+
+```bash
+mkdir -p vendor/macos-arm64
+source .venv312/bin/activate
+pip download --dest vendor/macos-arm64 -r requirements.txt
+cd vendor/macos-arm64 && shasum -a 256 *.whl > SHA256SUMS
+```
+
+**Manual build (Linux via Docker):**
+
+```bash
+docker run --rm -v $(pwd):/work -w /work python:3.12-slim bash -c "
+  apt-get update && apt-get install -y gcc g++ make libffi-dev
+  pip install --upgrade pip
+  mkdir -p vendor/linux-x86_64
+  pip download --dest vendor/linux-x86_64 -r requirements.txt
+  cd vendor/linux-x86_64 && sha256sum *.whl > SHA256SUMS
+"
+```
+
+### Verifying Vendor Integrity
+
+```bash
+# Check checksums before using on air-gapped machine
+cd vendor/linux-x86_64
+sha256sum -c SHA256SUMS     # Linux
+shasum -a 256 -c SHA256SUMS  # macOS
+```
+
+***
+
+## Outputs and File Structure
+
+### Stdout Behavior

 - `gen` prints the mnemonic + derived addresses by default (intended for debug/test comparisons).
 - `--off-screen` suppresses printing sensitive data to stdout.

-### `--file` behavior (deterministic, secured output)
+### `--file` Behavior (Deterministic, Secured Output)

 If `--file` is present, the tool writes **only** an AES-encrypted ZIP file (no raw `.json`/`.asc` file is left on disk). AES ZIP is implemented using `pyzipper`.

@@ -175,12 +307,12 @@ Override folder:

 - `--wallet-location /path/to/folder`

-Naming uses UTC timestamps (e.g. `20260106_161830Z`):
+Naming uses UTC timestamps (e.g. `20260108_011830Z`):

 - No PGP: zip contains `test_wallet_<UTC>.json`, zip name `test_wallet_<UTC>.zip`
 - With PGP: zip contains `encrypted_wallet_<UTC>.asc`, zip name `encrypted_wallet_<UTC>.zip`

-### Password handling for ZIP
+### Password Handling for ZIP

 - Default: `--zip-password-mode prompt` prompts for the ZIP password (attempts hidden entry).
 - If hidden entry is not supported in the environment, it falls back to visible `input()` with loud warnings.
@@ -189,7 +321,7 @@ Naming uses UTC timestamps (e.g. `20260106_161830Z`):

 `pyzipper` supports AES encryption via `AESZipFile` and password-setting APIs.

---
+***

 ## Commands

@@ -209,7 +341,7 @@ Options:
 - `--off-screen`: reduced output / temp-file behavior
 - `--expected-fingerprint`: refuse if downloaded key fingerprint doesn't match (40 hex chars)

---
+***

 ### gen (offline)

@@ -248,7 +380,7 @@ Safety options:
 - `--off-screen` (don't print sensitive output)
 - `--force` (only for non-TTY stdout; see below)

---
+***

 ### recover (offline)

@@ -274,7 +406,7 @@ Notes:
 - `--export-private` requires `--pgp-pubkey-file` (private keys only travel inside encrypted payload).
 - `--include-source` controls whether mnemonic is included inside the encrypted payload (only meaningful when PGP is used).

---
+***

 ### test (offline)

@@ -290,9 +422,9 @@ For comprehensive regression testing, use the pytest suite:
 pytest -v tests/test_vectors.py
 ```

---
+***

-## When to use `--force`
+## When to Use `--force`

 Use `--force` only if you **intentionally** want to print sensitive output while stdout is being piped/redirected.

@@ -314,17 +446,193 @@ python ./src/pyhdwallet.py gen --force > out.txt

 If running normally in your interactive terminal, stdout is a TTY and `--force` does nothing (output looks the same).

---
+***

-## Security notes (practical)
+## Security Notes (Practical)

 - `gen` printing the mnemonic is intentionally "debug/test" behavior. Assume stdout can be recorded (scrollback, logging, screen recording, CI logs).
 - Prefer `--off-screen` for reduced exposure.
 - Prefer `--file` so artifacts go into `.wallet/` and are AES-encrypted via `pyzipper`.
 - For stronger at-rest security: combine `--pgp-pubkey-file` + `--file` so the ZIP contains only an encrypted `.asc` payload. PGPy encryption style follows `PGPMessage.new(...)` then `pubkey.encrypt(...)`.
 - Use `--expected-fingerprint` to enforce PGP key pinning and prevent key substitution attacks.
+- Install from vendored wheels on air-gapped machines to avoid supply chain attacks.

---
+***
+
+## Offline Security Best Practices
+
+This tool is designed for offline use, but true security depends on the **environment** where you run it. Below are operational security recommendations for generating keys you can trust.
+
+### Air-Gapped Setup (Highest Security)
+
+For maximum security when generating production wallets, use an **air-gapped computer**—a device that has never been and will never be connected to any network.
+
+**Recommended procedure:**
+
+1. **Prepare a clean machine**:
+   - Use a dedicated laptop or bootable USB with a fresh Linux installation (e.g., Tails, Ubuntu Live USB)
+   - Never connect this device to WiFi, Ethernet, or Bluetooth
+   - Physically disable network interfaces if possible (remove WiFi card, tape over Ethernet port)
+
+2. **Transfer repository offline**:
+
+   ```bash
+   # On trusted online machine, clone and verify
+   git clone https://github.com/<your-username>/hdwalletpy.git
+   cd hdwalletpy
+   
+   # Verify checksums
+   cd vendor/linux-x86_64
+   sha256sum -c SHA256SUMS
+   
+   # Transfer entire repo to USB
+   ```
+
+3. **Verify code integrity on air-gapped machine**:
+
+   ```bash
+   # Check checksums again
+   cd vendor/linux-x86_64
+   sha256sum -c SHA256SUMS
+   
+   # Run test suite to verify derivation logic
+   ./install_offline.sh
+   pytest -v tests/test_vectors.py
+   ```
+
+4. **Generate wallet**:
+
+   ```bash
+   python ./src/pyhdwallet.py gen --pgp-pubkey-file pubkeys/recipient.asc --file --off-screen
+   ```
+
+5. **Transfer output safely**:
+   - Copy only the `.wallet/*.zip` file to USB (never copy `pyhdwallet.py` or Python environment back to online machine)
+   - The ZIP is AES-encrypted; the inner `.asc` is PGP-encrypted
+
+6. **Destroy or securely wipe** the USB after transfer if it contained unencrypted secrets
+
+### Threats Air-Gapping Mitigates
+
+- **Remote attacks**: Malware cannot exfiltrate keys over the network
+- **Clipboard hijacking**: No clipboard manager or remote access tool can intercept data
+- **Browser/OS telemetry**: No accidental upload of terminal history or crash dumps
+
+### Threats Air-Gapping Does NOT Fully Mitigate
+
+Research shows that sophisticated attackers with physical access can potentially exfiltrate data from air-gapped systems via covert channels (acoustic, electromagnetic, optical). However:
+
+- These attacks require physical proximity and pre-installed malware
+- For individual users (not nation-state targets), air-gapping remains highly effective
+- Countermeasures: Use the machine in a secure location, inspect for unfamiliar USB devices, verify software integrity before installation
+
+### Physical Security
+
+**Mnemonic handling:**
+
+- Write the mnemonic on paper immediately; never store it digitally on the air-gapped machine
+- Use a metal backup (e.g., Cryptosteel) for fire/water resistance
+- Split storage across multiple secure locations if desired (Shamir's Secret Sharing for advanced users)
+
+**Device handling:**
+
+- After generating the wallet, optionally wipe the air-gapped device or destroy the bootable USB
+- If reusing the device, use secure-erase tools (not just `rm`)
+
+### Verification Through Testing
+
+**Why the test suite matters for trust:**
+
+The committed test suite (`tests/test_vectors.py`) allows you to verify that derivation logic hasn't been tampered with before generating production keys:
+
+```bash
+# On the air-gapped machine, before generating your wallet:
+pytest -v tests/test_vectors.py
+```
+
+If all tests pass, you have cryptographic proof that:
+
+- BIP39 seed derivation matches the well-known test vector ("abandon abandon..." → known seed)
+- Derivation paths produce addresses matching public BIP39/BIP44 test vectors
+- PGP fingerprinting logic works correctly
+
+If tests fail, **do not generate keys**—the code may be compromised or buggy.
+
+### Entropy Sources
+
+**Built-in entropy (default):**
+
+- Python's `secrets.token_bytes()` uses OS-provided CSPRNG (`/dev/urandom` on Linux, `CryptGenRandom` on Windows)
+- This is cryptographically secure for typical use
+
+**Additional user entropy (optional):**
+
+```bash
+python ./src/pyhdwallet.py gen --dice-rolls "4 2 6 1 3 5 ..." --file
+```
+
+- Roll a die 50+ times and input the results
+- Your dice rolls are mixed with OS entropy via SHA256
+- Protects against potential CSPRNG backdoors (theoretical concern)
+
+### PGP Key Fingerprint Pinning
+
+Always use `--expected-fingerprint` when encrypting to a PGP key to prevent key substitution attacks:
+
+```bash
+# First, get and verify the fingerprint of your recipient key
+python ./src/pyhdwallet.py fetchkey https://example.com/mykey.asc --out mykey.asc
+# Manually verify fingerprint matches what you expect (check via another channel)
+
+# Then use it with pinning:
+python ./src/pyhdwallet.py gen \
+  --pgp-pubkey-file mykey.asc \
+  --expected-fingerprint A27B96F2B169B5491013D2DA892B822C14A9AA18 \
+  --file
+```
+
+Without `--expected-fingerprint`, an attacker who controls the filesystem could swap `mykey.asc` with their own key.
+
+### Operational Checklist
+
+Before generating production wallets:
+
+- [ ] Running on air-gapped machine or fresh Live USB
+- [ ] Network physically disabled (no WiFi/Ethernet/Bluetooth)
+- [ ] Code integrity verified (checksums + test suite passes)
+- [ ] Using `--off-screen` to minimize terminal scrollback exposure
+- [ ] Using `--file` to avoid leaving unencrypted files on disk
+- [ ] Using `--pgp-pubkey-file` with `--expected-fingerprint` for key pinning
+- [ ] Paper/metal backup prepared for mnemonic
+- [ ] Output ZIP password stored separately from ZIP file
+- [ ] Plan for USB secure wipe after transfer
+
+### Lower-Risk Scenarios
+
+Air-gapping is overkill for:
+
+- **Learning/testing**: Use your regular laptop with `--off-screen` and `--file`
+- **Small amounts**: Generate on a clean, updated machine with minimal software
+- **Testnet wallets**: Standard laptop is fine
+
+Air-gapping is recommended for:
+
+- **Life savings or business funds**
+- **Long-term cold storage** (multi-year hold)
+- **Institutional custody scenarios**
+
+### Trust But Verify
+
+The only way to fully trust this tool (or any wallet software) is to:
+
+1. **Read the source code** (`src/pyhdwallet.py` is ~1400 lines, single file)
+2. **Verify test vectors** match published BIP39 test data
+3. **Run the test suite** on your air-gapped machine
+4. **Generate a test wallet** and verify addresses on a block explorer using a separate tool
+
+Never trust wallet software blindly—especially for significant funds.
+
+***

 ## Troubleshooting

@@ -339,26 +647,51 @@ chmod +x ./src/pyhdwallet.py
 ./src/pyhdwallet.py gen
 ```

-### "Missing dependency: pyzipper" but pip says installed
+### Python 3.12 not found

-You likely installed into a different venv. Use:
+Vendored wheels require Python 3.12:

 ```bash
-python -m pip install pyzipper
-python -c "import pyzipper; print(pyzipper.__version__)"
+# macOS
+brew install python@3.12
+
+# Ubuntu/Debian
+sudo apt-get install python3.12 python3.12-venv
+
+# Verify
+python3.12 --version
 ```

-### "Missing dependency: pytest" when running tests
+### "Missing dependency" errors

-Install pytest in your virtualenv:
+Make sure you're installing from the correct vendor directory:

 ```bash
-python -m pip install pytest
+# Check your platform
+uname -sm
+
+# Install for macOS ARM64
+pip install --no-index --find-links=vendor/macos-arm64 -r requirements.txt
+
+# Install for Linux x86_64
+pip install --no-index --find-links=vendor/linux-x86_64 -r requirements.txt
+```
+
+### Checksum verification failures
+
+If `SHA256SUMS` verification fails, the wheels may be corrupted or tampered with:
+
+```bash
+cd vendor/linux-x86_64
+sha256sum -c SHA256SUMS
+
+# If failures occur, re-download from trusted source
+# DO NOT use corrupted wheels for production wallets
 ```

 ### Unzipping AES ZIP issues

-Some `unzip` tools may not support AES-encrypted ZIPs. Use Python + pyzipper to extract if needed:
+Some `unzip` tools may not support AES-encrypted ZIPs. Use Python + pyzipper to extract:

 ```bash
 python - <<'PY'
@@ -373,8 +706,6 @@ print("Extracted to", out_dir)
 PY
 ```

-AES ZIP support is the reason `pyzipper` is used.
-
 ### Test failures after code changes

 If `pytest -v tests/test_vectors.py` fails after you modified code:
@@ -386,7 +717,45 @@ If `pytest -v tests/test_vectors.py` fails after you modified code:
 2. **Is only one test failing?**
   - Run with `-vv` for detailed diff: `pytest -vv tests/test_vectors.py::test_name`

---
+***
+
+## Development
+
+### Building Vendor Wheels
+
+See [Vendored Dependencies](#vendored-dependencies) section above.
+
+### Running Tests (Dev)
+
+```bash
+# Quick test
+python src/pyhdwallet.py test
+
+# Full regression suite
+pytest -v tests/test_vectors.py
+
+# Specific test
+pytest -v tests/test_vectors.py::test_address_derivation_integrity
+
+# With coverage
+pytest --cov=src tests/
+```
+
+### Docker Development Environment
+
+```bash
+# Build image
+make build-image
+
+# Start warm container
+make up
+make shell
+
+# Inside container
+python src/pyhdwallet.py test
+```
+
+***

 ## Changelog

@@ -398,22 +767,12 @@ If `pytest -v tests/test_vectors.py` fails after you modified code:
  - Added: `--force` to override the non-TTY printing safety guard (use only when redirecting/piping).
  - Added: `--expected-fingerprint` for PGP key pinning.
  - Added: Offline regression test suite with frozen vectors (`tests/test_vectors.py`).
+  - Added: Vendored dependencies for macOS ARM64 and Linux x86_64 (Python 3.12).
+  - Added: `install_offline.sh` automated installer for air-gapped deployment.
  - Changed: `recover` now uses `--mnemonic-stdin` and `--interactive` instead of `--mnemonic` CLI arg.
+  - Updated: Dockerfile and Makefile now support vendoring workflow.
+
 - **v1.0.3** Documentation and CLI behavior updates.
 - **v1.0.2** Added off-screen behaviors and security improvements.
 - **v1.0.1** Renamed to pyhdwallet and added version flag.
 - **v1.0.0** Initial release.
-
-Key additions:
-
-1. **Testing section** with full coverage explanation
-2. **Repository structure** updated to include `tests/` folder
-3. **Test artifacts** documentation
-4. **When to regenerate vectors** guidance
-5. **Troubleshooting** section for test failures
-6. **Changelog** updated with v1.0.5 test suite addition
-7. **Quick Start** updated with correct `recover` command syntax (`--mnemonic-stdin`)
-8. **Dependencies** updated to include pytest
-
-[1](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/50321846/e5caab46-cb40-4f76-8ea9-60ed65bf4927/pyhdwallet.py)
-[2](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/images/50321846/085d5710-8637-45d5-8686-77c98df9dcdf/image.jpg)