first commit

.github/instructions/background-jobs.instructions.md

@@ -0,0 +1,84 @@
+---
+applyTo: "**/jobs.py,**/views/imports/**,**/import_utils.py,**/import_validation_helpers.py"
+description: Background job architecture, import workflow, and task management patterns
+---
+
+# Background Jobs & Import Workflow
+
+## Job Architecture
+- Background jobs use NetBox's `JobRunner` base class (`netbox.jobs.JobRunner`) for long-running operations like device filtering with VC detection.
+- Jobs run via Redis Queue (RQ) in Redis, separate from the database Job model. Real-time status must be checked via RQ, not the database.
+
+## Critical Job Architecture Points
+- Job UUID (`job.job_id`) is used for RQ API endpoints: `/api/core/background-tasks/{uuid}/`
+- Job PK (`job.pk`) is used for database endpoints and result loading
+- RQ status values: `queued`, `started`, `finished`, `stopped`, `failed` (NOT `completed`)
+- Database Job status values: `pending`, `scheduled`, `running`, `completed`, `failed`, `errored` (NO `cancelled` status exists)
+- Check `rq_job.is_stopped` or `rq_job.is_failed` flags in Redis for cancellation detection, not database status
+
+## Job Cancellation Flow
+1. Call `/api/core/background-tasks/{uuid}/stop/` to stop RQ job
+2. Call plugin's sync endpoint `/api/plugins/librenms_plugin/jobs/{pk}/sync-status/` to update database
+3. Frontend polling detects status changes and redirects appropriately
+
+## Polling Implementation
+- Poll `/api/core/background-tasks/{uuid}/` for real-time RQ status
+- Update modal messages based on status: "Job queued...", "Processing...", "Job completed!"
+- Handle all RQ status values explicitly to avoid infinite polling
+- Use `cancelInProgress` flag to prevent polling interference during cancellation
+
+## Superuser Requirement for Background Jobs
+- NetBox's `/api/core/background-tasks/` endpoint requires **superuser** (`IsSuperuser` in `BaseRQViewSet`).
+- Non-superuser users cannot poll job status; they get 403 Forbidden.
+- The plugin automatically falls back to synchronous mode for non-superusers—see `should_use_background_job()` in `list.py` and `actions.py`.
+- This is a NetBox core design decision, not a plugin limitation. No amount of permissions (including `core.view_job`) bypasses it.
+
+## Import Jobs
+- **`FilterDevicesJob`** — background device filtering with VC detection. `job.data` keys: `device_ids`, `total_processed`, `filters`, `server_key`, `vc_detection_enabled`, `cache_timeout`, `cached_at`, `completed`. Devices are cached individually via shared cache keys from `get_validated_device_cache_key()`.
+- **`ImportDevicesJob`** — background device/VM import. Calls `bulk_import_devices_shared()` for devices and `bulk_import_vms()` for VMs. `job.data` keys: `imported_device_pks`, `imported_vm_pks`, `imported_libre_device_ids`, `imported_libre_vm_ids`, `server_key`, `total`, `success_count`, `failed_count`, `skipped_count`, `virtual_chassis_created`, `errors`, `completed`.
+
+## Shared Cache Key Pattern
+- Both synchronous and background modes use `get_validated_device_cache_key()` from `import_utils.py` to generate cache keys. This ensures `_load_job_results()` in the list view can retrieve devices regardless of which mode produced them.
+- `get_active_cached_searches()` manages multi-search cache to let users run and switch between searches.
+- Never hardcode cache key formats; always use the helper functions.
+
+## Permission Checks in Jobs
+- Background jobs run outside view context, so they cannot use view mixins.
+- Use standalone helpers from `import_utils.py` for permission checks inside job code:
+  - `check_user_permissions(user, permissions)` → `(bool, missing_list)`
+  - `require_permissions(user, permissions, action_description)` — raises `PermissionDenied`.
+
+## Custom Sync Endpoint
+`api/views.py::sync_job_status()` syncs database Job status with RQ job status, needed because NetBox worker doesn't always update DB when jobs stop before processing starts.
+
+## Import Page Flow
+The import page (`LibreNMSImportView` in `views/imports/list.py`) supports two modes:
+
+1. **Synchronous** — calls `process_device_filters()` directly, renders results inline.
+2. **Background** — enqueues `FilterDevicesJob`, returns `JsonResponse` with `job_id`/`job_pk`/`poll_url`. Frontend polls and redirects to `?job_id={pk}` on completion.
+
+Result loading: `_load_job_results(job_id)` reads `job.data["device_ids"]`, reconstructs devices from per-device cache using `get_validated_device_cache_key()`.
+
+Filter fields: `librenms_location`, `librenms_type`, `librenms_os`, `librenms_hostname`, `librenms_sysname`, `librenms_hardware`, `enable_vc_detection`, `show_disabled`, `exclude_existing`.
+
+## Import Action Views (`views/imports/actions.py`)
+- **`DeviceImportHelperMixin`** — provides `get_validated_device_with_selections()` and `render_device_row()` for HTMX row rendering. Shared by update views.
+- **`BulkImportConfirmView`** (POST) — renders confirmation modal with selected device list. Returns `htmx/bulk_import_confirm.html`.
+- **`BulkImportDevicesView`** (POST) — executes import. Background mode enqueues `ImportDevicesJob`; sync mode calls `bulk_import_devices()` + `bulk_import_vms()` and returns OOB row swaps with `HX-Trigger: closeModal`.
+- **`DeviceValidationDetailsView`** (GET) — renders expandable validation details via `htmx/device_validation_details.html`.
+- **`DeviceVCDetailsView`** (GET) — renders VC member details via `htmx/device_vc_details.html`.
+- **`DeviceRoleUpdateView`**, **`DeviceClusterUpdateView`**, **`DeviceRackUpdateView`** (POST) — per-device dropdown updates. Apply selection to validation state and return re-rendered row via `render_device_row()`.
+
+## Key Import Utilities (`import_utils.py`)
+- `process_device_filters(filters, ...)` — fetches and validates devices from LibreNMS, returns list.
+- `validate_device_for_import(device, ...)` — core validation function, produces validation state dict.
+- `bulk_import_devices_shared(devices, user, ...)` — shared implementation between sync and background import.
+- `bulk_import_vms(vm_imports, user, ...)` — VM import implementation.
+- `fetch_device_with_cache(device_id, ...)` — retrieves/caches individual device data.
+- Cache key functions: `get_validated_device_cache_key()`, `get_cache_metadata_key()`, `get_active_cached_searches()`, `get_import_device_cache_key()`.
+
+## Validation Helpers (`import_validation_helpers.py`)
+Centralizes validation state mutation used by the role/cluster/rack update views:
+- `apply_role_to_validation()`, `apply_cluster_to_validation()`, `apply_rack_to_validation()` — update validation state when user selects a role/cluster/rack.
+- `remove_validation_issue()`, `recalculate_validation_status()` — maintain issue list and overall status.
+- `fetch_model_by_id()`, `extract_device_selections()` — helpers for reading form data.

.github/instructions/frontend.instructions.md

@@ -0,0 +1,66 @@
+---
+applyTo: "netbox_librenms_plugin/templates/**,netbox_librenms_plugin/static/**"
+description: Frontend patterns for templates, HTMX, and static assets
+---
+
+# Frontend Patterns
+
+## HTMX Conventions
+- HTMX 2.x is the primary async layer. Table row updates return `<tr hx-swap-oob="true">`.
+- Avoid `outerHTML` swaps; use OOB or targeted `innerHTML` swaps to keep table layout intact.
+- All HTMX requests and `fetch()` calls must include a CSRF token. The standard pattern is `document.querySelector('[name=csrfmiddlewaretoken]').value` (from a hidden form input). The import JS also uses `getCookie('csrftoken')` as a fallback — prefer the hidden input approach for consistency.
+
+## Modal Implementation
+- Modals use Tabler (Bootstrap-like) but **without** `bootstrap.Modal` helpers.
+- Buttons target the `htmx-modal-content` element and JavaScript in `librenms_import.html` toggles the wrapper.
+- Do not reintroduce `data-bs-toggle` or duplicate modal IDs.
+- The import page uses `ModalManager` class and `filterModalManager` instance—always use this reference in fetch callbacks, not undefined `modalInstance` variables.
+
+## JavaScript Fetch Patterns
+- Always check `response.ok` before processing fetch responses to catch HTTP errors.
+- In catch blocks, show `error.message` for debugging rather than generic messages.
+- The import filter form uses fetch with `Accept: application/json, text/html`—JSON for background jobs, HTML for synchronous mode.
+
+## Form Controls
+- Device import dropdowns rely on TomSelect decorators set up elsewhere.
+- Keep `<select class="device-role-select">` markup stable to preserve JS hook-up.
+
+## Styling
+- Styling assumes Tabler defaults.
+- Removing `table-responsive` wrappers was deliberate to prevent dropdown clipping—do not re-add them.
+
+## Template Structure
+- Templates live in `templates/netbox_librenms_plugin/`; reuse/includes under `inc/`.
+- Sync pages extend `librenms_sync_base.html`.
+- Tables emit HTMX-enabled columns and buttons (`tables/*.py`), so prefer updating the table renderer in Python rather than templates when changing row actions.
+
+## Sync Tab Template Pattern
+- Each sync resource has two templates following a naming convention:
+  - `_<resource>_sync.html` — the tab wrapper, loaded once when the tab is selected.
+  - `_<resource>_sync_content.html` — the HTMX-swappable inner fragment, refreshed on data changes without a full page reload.
+- Current resources: `_interface_sync`, `_cable_sync`, `_ipaddress_sync`, `_vlan_sync`.
+- When adding a new sync resource, create both the wrapper and content templates following this pattern.
+
+## HTMX Fragments
+- HTMX fragments live in `templates/netbox_librenms_plugin/htmx/` and include:
+  - `device_import_row.html` — individual import row updates.
+  - `device_validation_details.html` — expandable validation details.
+  - `device_vc_details.html` — virtual chassis member details.
+  - `bulk_import_confirm.html` — import confirmation modal content.
+- Keep server responses and HTMX targets in sync when modifying these fragments.
+
+## Settings Page
+- `settings.html` uses a split-form pattern: two separate Django forms (`ServerConfigForm` + `ImportSettingsForm`) sharing one page, differentiated by a hidden `form_type` field (`"server_config"` or `"import_settings"`).
+- The test-connection button is an HTMX POST to `TestLibreNMSConnectionView`, returning an inline alert fragment.
+
+## Paginator
+- `inc/paginator.html` is a custom paginator that preserves tab state and `interface_name_field` in pagination URLs. Used across all sync tables.
+
+## Import Page JavaScript (`librenms_import.js`)
+- Wrapped in an IIFE with `window.LibreNMSImportInitialized` guard to prevent re-initialization during HTMX swaps.
+- **`ModalManager`** class wraps Bootstrap 5 modal show/hide with fallback.
+- **`pollJobStatus()`** — polls `/api/core/background-tasks/{jobId}/` every 2s, updates progress messages, handles cancel button, redirects on completion.
+- **`captureSelectionState()` / `restoreSelectionState()`** — preserves checkbox state across HTMX content swaps.
+- **`createCacheCountdown()`** — generic countdown timer for cache expiration display.
+- **`initializeFilterForm()`** — intercepts form submit, detects JSON response (background job), starts polling.
+- CSRF token extracted via `getCookie('csrftoken')` (cookie-based).

.github/instructions/sync.instructions.md

@@ -0,0 +1,72 @@
+---
+applyTo: "**/views/base/**,**/views/object_sync/**,**/views/sync/**,**/tables/**,**/librenms_sync.js"
+description: Sync page architecture, base views, and sync action patterns
+---
+
+# Sync Pages
+
+## Three-Layer View Architecture
+All four sync resources (interfaces, cables, IP addresses, VLANs) follow the same pattern:
+
+1. **Base views** (`views/base/`) — abstract classes that define the data pipeline:
+   - `BaseLibreNMSSyncView` — tabbed sync page, orchestrates all tabs via abstract `get_*_context()` methods.
+   - `BaseInterfaceTableView` — fetch ports → enrich with VLANs → cache → compare with NetBox interfaces → render table.
+   - `BaseCableTableView` — fetch links → match remote devices → check cable status → render table.
+   - `BaseIPAddressTableView` — fetch IPs → resolve interfaces → detect existing/update/new → render table.
+   - `BaseVLANTableView` — fetch VLANs → compare with NetBox VLANs → auto-select groups → render table.
+
+2. **Object sync views** (`views/object_sync/`) — wire base views to NetBox models:
+   - Use `@register_model_view(Device, name="librenms_sync", path="librenms-sync")` to inject as a tab on Device/VM detail pages.
+   - Each `get_*_context()` method creates an instance of the concrete table view, copies `request`, and calls `get_context_data()`.
+   - VMs skip cables and VLANs (return `None`).
+
+3. **Sync action views** (`views/sync/`) — POST-only views that create/update/delete NetBox objects:
+   - Follow a consistent pattern: check permissions → read selected rows from POST → load cached data → apply changes in `transaction.atomic()` → redirect to sync tab.
+
+## Data Pipeline (Base Views)
+Every base table view follows: **fetch → cache → compare → render**.
+
+- **Fetch:** Call LibreNMS API (e.g., `get_ports()`, `get_device_ips()`, `get_device_vlans()`).
+- **Cache:** Store results via `CacheMixin` keys: `librenms_{data_type}_{model_name}_{pk}`. Also store fetch timestamp at `librenms_{data_type}_last_fetched_{model_name}_{pk}`.
+- **Compare:** Match LibreNMS data against NetBox objects. Each resource implements its own comparison (interface matching by name, IP matching by address/mask, VLAN matching by VID+group).
+- **Render:** Build a django-tables2 table, return a partial template (`_*_sync_content.html`).
+
+## Sync Action View Pattern
+```python
+class SyncSomeResourceView(LibreNMSPermissionMixin, NetBoxObjectPermissionMixin, CacheMixin, View):
+    required_object_permissions = {"POST": [("add", Model), ("change", Model)]}
+
+    def post(self, request, object_type, object_id):
+        if error := self.require_all_permissions("POST"):
+            return error
+        # 1. Resolve object (Device or VM)
+        # 2. Read selected items from request.POST.getlist("select")
+        # 3. Load cached data from cache.get(self.get_cache_key(obj, "..."))
+        # 4. Apply changes inside transaction.atomic()
+        # 5. Redirect to sync tab with ?tab=<resource>
+```
+
+## Table Conventions (`tables/*.py`)
+- Tables define HTMX-enabled columns and checkboxes. Selection uses `ToggleColumn(attrs={"input": {"name": "select"}})`.
+- Constructor takes contextual params (e.g., `device`, `interface_name_field`, `vlan_groups`) to customize rendering.
+- Tables set `self.tab` and `self.prefix` for multi-table pagination via `get_table_paginate_count()`.
+- Row attrs include `data-*` attributes for JavaScript filtering and identification.
+- VLAN columns use `render_vlans()` with hidden inputs for per-row group selection and JSON data for modals.
+
+## Key Mixins Used by Sync Views
+- **`LibreNMSAPIMixin`** — lazy-creates `LibreNMSAPI` instance via `self.librenms_api` property. Also provides `get_server_info()` for template context.
+- **`CacheMixin`** — generates consistent cache keys via `get_cache_key(obj, data_type)` and `get_last_fetched_key(obj, data_type)`. Also provides `get_vlan_overrides_key(obj)` for VLAN group override persistence.
+- **`VlanAssignmentMixin`** — VLAN group scope resolution: Rack → Location → Site → SiteGroup → Region → Global. Used by interface and VLAN sync for auto-selecting the most-specific VLAN group and building lookup maps.
+
+## JavaScript (`librenms_sync.js`)
+- Not wrapped in an IIFE — functions are global. Master initializer `initializeScripts()` runs on `DOMContentLoaded` and `htmx:afterSwap`.
+- **Key function groups:**
+  - Checkbox management: `initializeTableCheckboxes()`, `updateBulkActionButton()`.
+  - TomSelect dropdowns: `initializeVCMemberSelect()`, `initializeVRFSelects()`, `initializeVlanGroupSelects()`, `initializeVlanSyncGroupSelects()`. Uses `TOMSELECT_INIT_DELAY_MS = 100` for delayed initialization after HTMX swaps.
+  - Verification: `handleInterfaceChange()`, `handleCableChange()`, `handleVRFChange()` — POST to single-item verify endpoints.
+  - VLAN modals: `openVlanDetailModal()`, `verifyVlanInGroup()`, `verifyVlanSyncGroup()` — per-interface VLAN detail editing.
+  - Bulk operations: `initializeBulkEditApply()`, `deleteSelectedInterfaces()`.
+  - Table filtering: `initializeTableFilters()`, `filterTable()` — client-side row filtering.
+  - URL/tab state: `initializeTabs()`, `getDeviceIdFromUrl()`, `setInterfaceNameFieldFromURL()`.
+  - Cache countdowns: `initializeCountdown()`, `initializeCountdowns()`.
+- CSRF token extracted via `document.querySelector('[name=csrfmiddlewaretoken]').value`.

.github/instructions/testing.instructions.md

@@ -0,0 +1,50 @@
+---
+applyTo: "tests/**"
+description: Testing patterns and conventions for the NetBox LibreNMS plugin
+---
+
+# Testing Patterns
+
+## General Test Conventions
+- Use plain **pytest classes**, not Django `TestCase`. Avoid `from django.test import TestCase`.
+- **Never use `@pytest.mark.django_db`** for unit tests—mock all database interactions with `MagicMock`.
+- Use **inline imports** inside test methods to avoid Django initialization at module load time.
+- Mock NetBox models (Device, Job, User) with `MagicMock()` instead of creating real instances.
+- Use `assert x == y` syntax, not `self.assertEqual(x, y)` (no TestCase inheritance).
+- See [docs/development/testing.md](../../docs/development/testing.md) for test file structure and running instructions.
+
+## Background Job Tests
+- Instantiate `JobRunner` subclasses using `object.__new__(JobClass)` to bypass `__init__`, then set `job.job = MagicMock()` and `job.logger = MagicMock()`. See `create_mock_job_runner()` helper in `tests/test_background_jobs.py`.
+- Patch deferred/inline imports at their **source** module (e.g., `netbox_librenms_plugin.import_utils.process_device_filters`), not the consuming module.
+- Patch `cache` where imported: `netbox_librenms_plugin.views.imports.list.cache`, not `django.core.cache.cache`.
+- Test view decision logic by setting `view._filter_form_data = {...}` directly, not via HTTP requests.
+- **Never use `RequestFactory`**—mock request objects directly or test method logic in isolation.
+- Cache key tests must patch `get_validated_device_cache_key` from `import_utils.py`; never hardcode key formats like `job_123_device_1`.
+
+## Test File Naming
+- Follow the `test_{module_name}.py` convention for new test files.
+- `test_netbox_librenms_plugin.py` is an empty placeholder — do not add tests there.
+
+## Test Coverage by Module
+- `librenms_api.py` → `test_librenms_api.py`, `test_librenms_api_helpers.py`
+- `import_utils.py`, `import_validation_helpers.py`, `utils.py` → `test_import_utils.py`, `test_import_validation_helpers.py`, `test_utils.py`
+- `jobs.py`, `views/imports/list.py` → `test_background_jobs.py`
+- Permission mixins, API permissions, constants → `test_permissions.py`
+- VLAN API, mode detection, comparison, sync → `test_vlan_sync.py`
+- `VlanAssignmentMixin`, VLAN enrichment → `test_interface_vlan_sync.py`
+- Views (`views/sync/`, `views/object_sync/`, `views/imports/actions.py`) — no dedicated test files yet. Test business logic via the utility modules they call, not via HTTP requests.
+
+## Permission Test Patterns
+When testing permissions (see `test_permissions.py` for reference):
+- Create a mock view instance with `object.__new__(ViewClass)`, set `request = MagicMock()` with `request.user.has_perm.side_effect = lambda p: p in allowed_perms`.
+- For `NetBoxObjectPermissionMixin` tests, set `required_object_permissions` on the instance before calling `check_object_permissions()`.
+- Test both the individual methods (`has_write_permission()`, `check_object_permissions()`) and the combined `require_all_permissions()` flow.
+- For JSON variants (`require_all_permissions_json`), assert `isinstance(response, JsonResponse)` and check `response.status_code == 403`.
+
+## Shared Fixtures (`conftest.py`)
+Reuse fixtures from `tests/conftest.py` instead of creating ad-hoc mocks:
+- **Configuration**: `mock_multi_server_config`, `mock_legacy_config`
+- **API client**: `mock_librenms_api`
+- **NetBox objects**: `mock_netbox_device`, `mock_netbox_vm`, `mock_netbox_site`, `mock_netbox_platform`, `mock_netbox_device_type`, `mock_netbox_device_role`, `mock_netbox_cluster`, `mock_netbox_rack`
+- **HTTP responses**: `mock_response_factory`, `mock_success_response`, `mock_device_response`, `mock_error_response`, `mock_auth_error_response`
+- **Import workflow**: `sample_librenms_device`, `sample_librenms_device_minimal`, `sample_validation_state`, `sample_validation_state_vm`