first commit

.github/instructions/background-jobs.instructions.md

@@ -0,0 +1,84 @@
+---
+applyTo: "**/jobs.py,**/views/imports/**,**/import_utils.py,**/import_validation_helpers.py"
+description: Background job architecture, import workflow, and task management patterns
+---
+
+# Background Jobs & Import Workflow
+
+## Job Architecture
+- Background jobs use NetBox's `JobRunner` base class (`netbox.jobs.JobRunner`) for long-running operations like device filtering with VC detection.
+- Jobs run via Redis Queue (RQ) in Redis, separate from the database Job model. Real-time status must be checked via RQ, not the database.
+
+## Critical Job Architecture Points
+- Job UUID (`job.job_id`) is used for RQ API endpoints: `/api/core/background-tasks/{uuid}/`
+- Job PK (`job.pk`) is used for database endpoints and result loading
+- RQ status values: `queued`, `started`, `finished`, `stopped`, `failed` (NOT `completed`)
+- Database Job status values: `pending`, `scheduled`, `running`, `completed`, `failed`, `errored` (NO `cancelled` status exists)
+- Check `rq_job.is_stopped` or `rq_job.is_failed` flags in Redis for cancellation detection, not database status
+
+## Job Cancellation Flow
+1. Call `/api/core/background-tasks/{uuid}/stop/` to stop RQ job
+2. Call plugin's sync endpoint `/api/plugins/librenms_plugin/jobs/{pk}/sync-status/` to update database
+3. Frontend polling detects status changes and redirects appropriately
+
+## Polling Implementation
+- Poll `/api/core/background-tasks/{uuid}/` for real-time RQ status
+- Update modal messages based on status: "Job queued...", "Processing...", "Job completed!"
+- Handle all RQ status values explicitly to avoid infinite polling
+- Use `cancelInProgress` flag to prevent polling interference during cancellation
+
+## Superuser Requirement for Background Jobs
+- NetBox's `/api/core/background-tasks/` endpoint requires **superuser** (`IsSuperuser` in `BaseRQViewSet`).
+- Non-superuser users cannot poll job status; they get 403 Forbidden.
+- The plugin automatically falls back to synchronous mode for non-superusers—see `should_use_background_job()` in `list.py` and `actions.py`.
+- This is a NetBox core design decision, not a plugin limitation. No amount of permissions (including `core.view_job`) bypasses it.
+
+## Import Jobs
+- **`FilterDevicesJob`** — background device filtering with VC detection. `job.data` keys: `device_ids`, `total_processed`, `filters`, `server_key`, `vc_detection_enabled`, `cache_timeout`, `cached_at`, `completed`. Devices are cached individually via shared cache keys from `get_validated_device_cache_key()`.
+- **`ImportDevicesJob`** — background device/VM import. Calls `bulk_import_devices_shared()` for devices and `bulk_import_vms()` for VMs. `job.data` keys: `imported_device_pks`, `imported_vm_pks`, `imported_libre_device_ids`, `imported_libre_vm_ids`, `server_key`, `total`, `success_count`, `failed_count`, `skipped_count`, `virtual_chassis_created`, `errors`, `completed`.
+
+## Shared Cache Key Pattern
+- Both synchronous and background modes use `get_validated_device_cache_key()` from `import_utils.py` to generate cache keys. This ensures `_load_job_results()` in the list view can retrieve devices regardless of which mode produced them.
+- `get_active_cached_searches()` manages multi-search cache to let users run and switch between searches.
+- Never hardcode cache key formats; always use the helper functions.
+
+## Permission Checks in Jobs
+- Background jobs run outside view context, so they cannot use view mixins.
+- Use standalone helpers from `import_utils.py` for permission checks inside job code:
+  - `check_user_permissions(user, permissions)` → `(bool, missing_list)`
+  - `require_permissions(user, permissions, action_description)` — raises `PermissionDenied`.
+
+## Custom Sync Endpoint
+`api/views.py::sync_job_status()` syncs database Job status with RQ job status, needed because NetBox worker doesn't always update DB when jobs stop before processing starts.
+
+## Import Page Flow
+The import page (`LibreNMSImportView` in `views/imports/list.py`) supports two modes:
+
+1. **Synchronous** — calls `process_device_filters()` directly, renders results inline.
+2. **Background** — enqueues `FilterDevicesJob`, returns `JsonResponse` with `job_id`/`job_pk`/`poll_url`. Frontend polls and redirects to `?job_id={pk}` on completion.
+
+Result loading: `_load_job_results(job_id)` reads `job.data["device_ids"]`, reconstructs devices from per-device cache using `get_validated_device_cache_key()`.
+
+Filter fields: `librenms_location`, `librenms_type`, `librenms_os`, `librenms_hostname`, `librenms_sysname`, `librenms_hardware`, `enable_vc_detection`, `show_disabled`, `exclude_existing`.
+
+## Import Action Views (`views/imports/actions.py`)
+- **`DeviceImportHelperMixin`** — provides `get_validated_device_with_selections()` and `render_device_row()` for HTMX row rendering. Shared by update views.
+- **`BulkImportConfirmView`** (POST) — renders confirmation modal with selected device list. Returns `htmx/bulk_import_confirm.html`.
+- **`BulkImportDevicesView`** (POST) — executes import. Background mode enqueues `ImportDevicesJob`; sync mode calls `bulk_import_devices()` + `bulk_import_vms()` and returns OOB row swaps with `HX-Trigger: closeModal`.
+- **`DeviceValidationDetailsView`** (GET) — renders expandable validation details via `htmx/device_validation_details.html`.
+- **`DeviceVCDetailsView`** (GET) — renders VC member details via `htmx/device_vc_details.html`.
+- **`DeviceRoleUpdateView`**, **`DeviceClusterUpdateView`**, **`DeviceRackUpdateView`** (POST) — per-device dropdown updates. Apply selection to validation state and return re-rendered row via `render_device_row()`.
+
+## Key Import Utilities (`import_utils.py`)
+- `process_device_filters(filters, ...)` — fetches and validates devices from LibreNMS, returns list.
+- `validate_device_for_import(device, ...)` — core validation function, produces validation state dict.
+- `bulk_import_devices_shared(devices, user, ...)` — shared implementation between sync and background import.
+- `bulk_import_vms(vm_imports, user, ...)` — VM import implementation.
+- `fetch_device_with_cache(device_id, ...)` — retrieves/caches individual device data.
+- Cache key functions: `get_validated_device_cache_key()`, `get_cache_metadata_key()`, `get_active_cached_searches()`, `get_import_device_cache_key()`.
+
+## Validation Helpers (`import_validation_helpers.py`)
+Centralizes validation state mutation used by the role/cluster/rack update views:
+- `apply_role_to_validation()`, `apply_cluster_to_validation()`, `apply_rack_to_validation()` — update validation state when user selects a role/cluster/rack.
+- `remove_validation_issue()`, `recalculate_validation_status()` — maintain issue list and overall status.
+- `fetch_model_by_id()`, `extract_device_selections()` — helpers for reading form data.