netbox-librenms-plugin/.github/instructions/background-jobs.instructions.md

applyTo, description

applyTo	description
**/jobs.py,**/views/imports/**,**/import_utils.py,**/import_validation_helpers.py	Background job architecture, import workflow, and task management patterns

Background Jobs & Import Workflow

Job Architecture

• Background jobs use NetBox's JobRunner base class (netbox.jobs.JobRunner) for long-running operations like device filtering with VC detection.
• Jobs run via Redis Queue (RQ) in Redis, separate from the database Job model. Real-time status must be checked via RQ, not the database.

Critical Job Architecture Points

• Job UUID (job.job_id) is used for RQ API endpoints: /api/core/background-tasks/{uuid}/
• Job PK (job.pk) is used for database endpoints and result loading
• RQ status values: queued, started, finished, stopped, failed (NOT completed)
• Database Job status values: pending, scheduled, running, completed, failed, errored (NO cancelled status exists)
• Check rq_job.is_stopped or rq_job.is_failed flags in Redis for cancellation detection, not database status

Job Cancellation Flow

1. Call /api/core/background-tasks/{uuid}/stop/ to stop RQ job
2. Call plugin's sync endpoint /api/plugins/librenms_plugin/jobs/{pk}/sync-status/ to update database
3. Frontend polling detects status changes and redirects appropriately

Polling Implementation

• Poll /api/core/background-tasks/{uuid}/ for real-time RQ status
• Update modal messages based on status: "Job queued...", "Processing...", "Job completed!"
• Handle all RQ status values explicitly to avoid infinite polling
• Use cancelInProgress flag to prevent polling interference during cancellation

Superuser Requirement for Background Jobs

• NetBox's /api/core/background-tasks/ endpoint requires superuser (IsSuperuser in BaseRQViewSet).
• Non-superuser users cannot poll job status; they get 403 Forbidden.
• The plugin automatically falls back to synchronous mode for non-superusers—see should_use_background_job() in list.py and actions.py.
• This is a NetBox core design decision, not a plugin limitation. No amount of permissions (including core.view_job) bypasses it.

Import Jobs

• FilterDevicesJob — background device filtering with VC detection. job.data keys: device_ids, total_processed, filters, server_key, vc_detection_enabled, cache_timeout, cached_at, completed. Devices are cached individually via shared cache keys from get_validated_device_cache_key().
• ImportDevicesJob — background device/VM import. Calls bulk_import_devices_shared() for devices and bulk_import_vms() for VMs. job.data keys: imported_device_pks, imported_vm_pks, imported_libre_device_ids, imported_libre_vm_ids, server_key, total, success_count, failed_count, skipped_count, virtual_chassis_created, errors, completed.

Shared Cache Key Pattern

• Both synchronous and background modes use get_validated_device_cache_key() from import_utils.py to generate cache keys. This ensures _load_job_results() in the list view can retrieve devices regardless of which mode produced them.
• get_active_cached_searches() manages multi-search cache to let users run and switch between searches.
• Never hardcode cache key formats; always use the helper functions.

Permission Checks in Jobs

• Background jobs run outside view context, so they cannot use view mixins.
• Use standalone helpers from import_utils.py for permission checks inside job code:
  • check_user_permissions(user, permissions) → (bool, missing_list)
  • require_permissions(user, permissions, action_description) — raises PermissionDenied.

Custom Sync Endpoint

api/views.py::sync_job_status() syncs database Job status with RQ job status, needed because NetBox worker doesn't always update DB when jobs stop before processing starts.

Import Page Flow

The import page (LibreNMSImportView in views/imports/list.py) supports two modes:

1. Synchronous — calls process_device_filters() directly, renders results inline.
2. Background — enqueues FilterDevicesJob, returns JsonResponse with job_id/job_pk/poll_url. Frontend polls and redirects to ?job_id={pk} on completion.

Result loading: _load_job_results(job_id) reads job.data["device_ids"], reconstructs devices from per-device cache using get_validated_device_cache_key().

Filter fields: librenms_location, librenms_type, librenms_os, librenms_hostname, librenms_sysname, librenms_hardware, enable_vc_detection, show_disabled, exclude_existing.

Import Action Views (views/imports/actions.py)

• DeviceImportHelperMixin — provides get_validated_device_with_selections() and render_device_row() for HTMX row rendering. Shared by update views.
• BulkImportConfirmView (POST) — renders confirmation modal with selected device list. Returns htmx/bulk_import_confirm.html.
• BulkImportDevicesView (POST) — executes import. Background mode enqueues ImportDevicesJob; sync mode calls bulk_import_devices() + bulk_import_vms() and returns OOB row swaps with HX-Trigger: closeModal.
• DeviceValidationDetailsView (GET) — renders expandable validation details via htmx/device_validation_details.html.
• DeviceVCDetailsView (GET) — renders VC member details via htmx/device_vc_details.html.
• DeviceRoleUpdateView, DeviceClusterUpdateView, DeviceRackUpdateView (POST) — per-device dropdown updates. Apply selection to validation state and return re-rendered row via render_device_row().

Key Import Utilities (import_utils.py)

• process_device_filters(filters, ...) — fetches and validates devices from LibreNMS, returns list.
• validate_device_for_import(device, ...) — core validation function, produces validation state dict.
• bulk_import_devices_shared(devices, user, ...) — shared implementation between sync and background import.
• bulk_import_vms(vm_imports, user, ...) — VM import implementation.
• fetch_device_with_cache(device_id, ...) — retrieves/caches individual device data.
• Cache key functions: get_validated_device_cache_key(), get_cache_metadata_key(), get_active_cached_searches(), get_import_device_cache_key().

Validation Helpers (import_validation_helpers.py)

Centralizes validation state mutation used by the role/cluster/rack update views:

• apply_role_to_validation(), apply_cluster_to_validation(), apply_rack_to_validation() — update validation state when user selects a role/cluster/rack.
• remove_validation_issue(), recalculate_validation_status() — maintain issue list and overall status.
• fetch_model_by_id(), extract_device_selections() — helpers for reading form data.