first commit
Some checks failed
ci / deploy (push) Has been cancelled
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled

This commit is contained in:
Vlastislav Svatek
2026-06-05 10:39:05 +02:00
commit 673e67106e
217 changed files with 76612 additions and 0 deletions

View File

@@ -0,0 +1,55 @@
# NetBox Development Environment Variables
# Copy this file to .env and customize as needed
# NetBox Version to install (latest, v4.1-3.3.0, v4.0-3.3.0, snapshot)
NETBOX_VERSION=latest
# Database Configuration
DB_HOST=postgres
DB_NAME=netbox
DB_USER=netbox
DB_PASSWORD=netbox
# Redis Configuration
REDIS_HOST=redis
REDIS_PASSWORD=
# Development Settings
DEBUG=True
DEVELOPER=True
SECRET_KEY=dev-secret-key-not-for-production-use-12345678901234
# Superuser Configuration (auto-created on first run)
SUPERUSER_NAME=admin
SUPERUSER_EMAIL=admin@example.com
SUPERUSER_PASSWORD=admin
# Auto-creation control
SKIP_SUPERUSER=false
# Plugins are configured only in:
# .devcontainer/plugin-config.py.example → .devcontainer/plugin-config.py
# Advanced NetBox configuration (optional):
# .devcontainer/extra-configuration.py.example → .devcontainer/extra-configuration.py
# Proxy Configuration (optional, for corporate networks with MITM proxies)
# Uncomment and set these if you're behind a proxy
# HTTP_PROXY=http://proxy.example.com:8080
# HTTPS_PROXY=http://proxy.example.com:8080
# NO_PROXY=localhost,127.0.0.1,postgres,redis
#
# CA bundle configuration:
# Normally you SHOULD NOT set REQUESTS_CA_BUNDLE, SSL_CERT_FILE, or CURL_CA_BUNDLE here.
# Instead, place a ca-bundle.crt file in the workspace root and setup.sh will install it
# into the system trust store and set these variables automatically to:
# /etc/ssl/certs/ca-certificates.crt
# Only set the following manually for custom CA setups that cannot use the automatic
# configuration provided by setup.sh.
# REQUESTS_CA_BUNDLE=/custom/path/to/ca-bundle.crt
# SSL_CERT_FILE=/custom/path/to/ca-bundle.crt
# CURL_CA_BUNDLE=/custom/path/to/ca-bundle.crt
# Git SSL verification override (default: false)
# Only set to true if behind a MITM proxy and you cannot provide a CA bundle.
# Prefer placing a ca-bundle.crt in the workspace root instead.
# ALLOW_GIT_SSL_DISABLE=false

396
.devcontainer/README.md Normal file
View File

@@ -0,0 +1,396 @@
# NetBox LibreNMS Plugin - Development Container
The Dev container was created to help aid with development without the need for a full NetBox installation locally. It provides a complete development environment using the official NetBox Docker images with PostgreSQL and Redis.
This directory contains the development container configuration for the NetBox LibreNMS Plugin.
## Table of contents
- [Prerequisites](#-prerequisites)
- [Quick Start](#-quick-start)
- [Out-of-the-box defaults](#out-of-the-box-defaults)
- [Configuration](#-configuration)
- [NetBox Version and Environment](#netbox-version-and-environment-use-devcontainerenv)
- [Changing NetBox Versions](#-changing-netbox-versions)
- [Other environment variables](#other-environment-variables)
- [Other configurations](#other-configurations)
- [NetBox Configuration](#netbox-configuration)
- [Additional packages](#additional-packages-including-other-netbox-plugins)
- [Git Setup](#-git-setup)
- [Commands](#-commands-aliases)
- [Troubleshooting](#-troubleshooting)
- [Cleanup](#-cleanup-remove-the-dev-containers)
## 📋 Prerequisites
### For Local Development (VS Code)
To use this dev container locally, you need:
- **[Docker](https://docs.docker.com/get-docker/)** (Docker Engine + Docker Compose)
- **[Visual Studio Code](https://code.visualstudio.com/)**
- **[Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)** for VS Code
### For GitHub Codespaces
If using GitHub Codespaces, all prerequisites are automatically available - just click "Code" → "Create codespace" in the GitHub repository.
**⚠️ Network Limitation with Codespaces:** GitHub Codespaces runs in the cloud and can only access publicly available LibreNMS servers.
If you need to test with a LibreNMS instance on a private network (local lab, corporate network, etc.), you'll need to use the local dev container instead.
## 🚀 Quick Start
1. Fork and Clone: fork the plugin repo in Github and clone locally
2. Open in VS Code and choose "Reopen in Container" (or Ctrl+Shift+P → Dev Containers: Reopen in Container)
2. Wait for setup (~5min on first run or when new NetBox image is used). The container will install the plugin and prep NetBox
3. Set up GitHub access: `gh auth login` (for pushing/pulling code changes)
4. Create your plugin config — see [Plugin configuration](#plugin-configuration):
- `cp .devcontainer/config/plugin-config.py.example .devcontainer/config/plugin-config.py`
- Edit it with your server details (tokens/URLs)
5. Start NetBox with `netbox-run` (or `netbox-run-bg` in background) (see [Commands](#-commands-aliases))
6. Access NetBox at http://localhost:8000
- Username: `admin`
- Password: `admin`
### 🔄 Code changes and Committing
8. Edit code in the repo root. Check out [contributing docs](../docs/contributing.md)
9. Use `netbox-logs` to follow log output on screen
6. Commit changes and contribute as normal by submitting a PR on GitHub.
### Quick Tips
- **Auto-reload**: Works for most code changes when `DEBUG=True`
- **Config changes**: Always restart NetBox after changing plugin settings
- **GitHub CLI**: Automatically configured for easy PR submission
- **Logs**: Use `netbox-logs` to debug issues in real-time
### 📡 LibreNMS Server Configuration
You need a LibreNMS instance to use this plugin. Configure your LibreNMS server(s) in `plugin-config.py`:
1. Copy the example config:
```bash
cp .devcontainer/config/plugin-config.py.example .devcontainer/config/plugin-config.py
```
2. Edit it with your LibreNMS server URL(s) and API token(s)
3. Restart NetBox: `netbox-restart`
## Out-of-the-box defaults
Below are the dev container defaults. The field name to change these defaults is listed below each line.
- NetBox image: `netboxcommunity/netbox:${NETBOX_VERSION:-latest}` (default `latest`)
- .env: `NETBOX_VERSION`
- DB: PostgreSQL 15 (db: `netbox`, user: `netbox`, password: `netbox`)
- .env: `DB_HOST`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`
- Redis: 7-alpine
- .env: `REDIS_HOST`, `REDIS_PASSWORD`
- NetBox DEBUG: `True` (dev only)
- .env: `DEBUG`
- Secret key: dev placeholder (not for production)
- .env: `SECRET_KEY` (optional). If unset, a dev-safe default is used inside the container.
- Superuser: `admin` / `admin`
- .env: `SUPERUSER_NAME`, `SUPERUSER_EMAIL`, `SUPERUSER_PASSWORD`, `SKIP_SUPERUSER`
- Plugin loader: enabled; reads `.devcontainer/config/plugin-config.py` if present
- If `plugin-config.py` is missing: plugin is enabled with empty config (features wont work until configured)
## 🔧 Configuration
### NetBox Version and Environment (use .devcontainer/.env)
The default NetBox docker image version is set to `latest`.
To update the NetBox image version create `.devcontainer/.env` using `NETBOX_VERSION` Example:
If you dont have an env file yet, create it in the `.devcontainer/` folder from the example and customize:
```bash
cp .devcontainer/.env.example .devcontainer/.env
```
Change the NetBox image version in the `.env` file:
```bash
# .devcontainer/.env
NETBOX_VERSION=v4.2-3.3.4
```
After changing `.devcontainer/.env`, rebuild the dev container to apply it (Command Palette → Dev Containers: Rebuild Container).
See NetBox Docker tag docs for available tags:
https://hub.docker.com/r/netboxcommunity/netbox/#container-image-tags
### 🔄 Changing NetBox Versions
You might experience issues with database schemas and migrations when changing NetBox version. Since this is a development container, the simplest way to handle NetBox version changes is to reset the database completely.
**To change NetBox versions:**
1. **Update the version** in `.devcontainer/.env`:
```bash
NETBOX_VERSION=v4.1-3.1.1 # or whatever version you need
```
2. **Reset the development environment:**
```bash
# Stop containers and remove volumes (removes all dev data)
docker compose down -v
# Rebuild container with new NetBox version
# VS Code: Ctrl+Shift+P → "Dev Containers: Rebuild Container"
```
3. **Start fresh:** The container will automatically set up the new NetBox version with a clean database.
**Note:** This removes all development data (test devices, configurations, etc.), but that's typically fine for development and testing scenarios.
### Other environment variables:
- Core: `NETBOX_VERSION`, `DEBUG`, `SECRET_KEY`
- Database: `DB_HOST`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`
- Redis: `REDIS_HOST`, `REDIS_PASSWORD`
- Superuser: `SUPERUSER_NAME`, `SUPERUSER_EMAIL`, `SUPERUSER_PASSWORD`, `SKIP_SUPERUSER`
- Proxy: `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`, `REQUESTS_CA_BUNDLE`, `SSL_CERT_FILE`, `CURL_CA_BUNDLE`
### 🌐 Proxy Configuration (MITM Proxies)
If you're behind a corporate proxy or MITM proxy (like Zscaler, BlueCoat, etc.), you need to configure the proxy at two levels: the Docker client (for building) and the container runtime (for package installation inside the container).
**Step 1: Configure Docker client proxy** (`~/.docker/config.json`)
This is **required** so that `apt-get`, `curl`, etc. work during the container image build (e.g., when installing devcontainer features like `git` and `github-cli`).
Create or edit `~/.docker/config.json`:
```json
{
"proxies": {
"default": {
"httpProxy": "http://proxy.example.com:8080",
"httpsProxy": "http://proxy.example.com:8080",
"noProxy": "localhost,127.0.0.1,postgres,redis"
}
}
}
```
Docker automatically injects these as environment variables into every `RUN` instruction during `docker build`. No VS Code restart is needed — this takes effect immediately.
> **Docker Desktop users:** You can configure the same settings via Docker Desktop Settings → Resources → Proxies, which writes this file for you.
**Step 2: Create `.devcontainer/.env`** (for container runtime)
```bash
cp .devcontainer/.env.example .devcontainer/.env
```
Add your proxy settings to `.devcontainer/.env`:
```bash
# Proxy Configuration
HTTP_PROXY=http://proxy.example.com:8080
HTTPS_PROXY=http://proxy.example.com:8080
NO_PROXY=localhost,127.0.0.1,postgres,redis
```
> **Note:** You do **not** need to set `REQUESTS_CA_BUNDLE`, `SSL_CERT_FILE`, or `CURL_CA_BUNDLE` manually. When a `ca-bundle.crt` file is present in the workspace root, `setup.sh` automatically installs it into the system trust store and sets these variables to `/etc/ssl/certs/ca-certificates.crt`.
**Step 3: Add your CA certificate** (optional, only if your proxy intercepts TLS):
- Export your proxy's CA certificate (usually available from your IT department or browser)
- Save it as `ca-bundle.crt` in the root of your workspace
- `setup.sh` will automatically install it and configure CA bundle environment variables
**Step 4: Rebuild the container**:
- VS Code: Ctrl+Shift+P → "Dev Containers: Rebuild Container"
**What gets configured:**
- `~/.docker/config.json` → proxy for Docker build steps (devcontainer features, apt in Dockerfile)
- `.devcontainer/.env` → proxy for running containers (apt, pip, curl at runtime)
- `setup.sh` auto-configures apt proxy and git SSL settings inside the container
**Important Notes:**
- The `.env` file is ignored by git, so your proxy credentials stay private
- `~/.docker/config.json` is a per-user file outside the repo
- Add internal service names to `NO_PROXY` to avoid routing internal Docker traffic through the proxy
- **Proxy authentication:** Embedding credentials directly in the proxy URL (e.g., `http://username:password@proxy.example.com:8080`) is insecure — credentials can be visible in process listings, environment dumps, `docker inspect` output, and logs. Prefer safer alternatives such as Docker's `config.json` with `credsStore` or a secret manager for storing proxy credentials securely.
**Common Issues:**
*"Could not connect to archive.ubuntu.com" during build*
- → `~/.docker/config.json` is missing or has wrong proxy URL
*"SSL certificate errors" during build*
- → Your proxy uses a MITM certificate. Export it and add it to the system trust store, or set `SSL_CERT_FILE` in `.env`
*Container builds but apt/pip fails inside*
- → .env file is missing or has wrong proxy settings. Check .env matches Docker Desktop settings
After any `.env` change, rebuild the dev container to apply environment updates.
- VS Code: “Dev Containers: Rebuild Container” (from the Command Palette)
## Other configurations
### NetBox Configuration:
- Create `.devcontainer/config/extra-configuration.py` for additional NetBox settings (TIME_ZONE, banners, logging, etc)
- After changes: run `netbox-restart` (see [Commands](#-commands-aliases))
### Additional packages (including other netbox plugins)
- Create `.devcontainer/extra-requirements.txt` for extra Python packages. Example: `.devcontainer/extra-requirements.txt.example`.
- After changes: run `plugins-install` to install packages, then `netbox-restart` (see [Commands](#-commands-aliases))
## 🔧 Git Setup
The dev container includes Git and GitHub CLI pre-installed. You'll need to configure authentication for commits and pushes.
### Important: SSH vs HTTPS Remote URLs
**Common Issue**: If you cloned this repository using SSH (`git@github.com:...`), you may encounter authentication errors when pushing changes. This is because:
- Dev containers don't have SSH keys by default
- GitHub CLI authentication uses HTTPS protocol
**Solution**: The setup script automatically converts SSH remote URLs to HTTPS. If you encounter issues, manually fix with:
```bash
# Check current remote URL
git remote -v
# If it shows git@github.com:..., convert to HTTPS
git remote set-url origin https://github.com/bonzo81/netbox-librenms-plugin.git
```
### Recommended: GitHub CLI (Easiest)
```bash
# Authenticate with GitHub (handles Git credentials automatically)
gh auth login
# Verify authentication
gh auth status
```
The GitHub CLI automatically configures Git to use your GitHub credentials for this repository.
### GitHub Codespaces
In Codespaces, GitHub authentication is often pre-configured, but you can verify with:
```bash
# Check current status
gh auth status
# If needed, authenticate
gh auth login
```
### Manual Git Setup (Alternative)
If you prefer manual setup or need non-GitHub authentication:
#### Local Dev Container
```bash
# Set your Git identity
git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"
# Optional: Set default branch name
git config --global init.defaultBranch main
```
#### SSH Key Setup (for private repositories)
```bash
# Generate SSH key (if you don't have one)
ssh-keygen -t ed25519 -C "your.email@example.com"
# Add to SSH agent
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
# Display public key to add to GitHub
cat ~/.ssh/id_ed25519.pub
```
> **💡 Authentication Persistence:**
> - **GitHub CLI**: Authentication persists across container rebuilds (stored in persistent volume)
> - **Manual Git Config**: Git identity settings are **NOT persistent** across rebuilds
> - **GitHub Codespaces**: Authentication is automatically handled by the Codespaces platform
>
> **Recommendation**: Use `gh auth login` for the best experience - it's persistent and handles everything automatically.
## 📋 Commands (aliases)
- `netbox-run-bg` - start NetBox and RQ worker in background
- `netbox-run` - start NetBox and RQ worker in foreground (with Django logs showing)
- `netbox-stop` - stop both NetBox and RQ worker
- `netbox-restart` - restart NetBox and RQ worker
- `netbox-reload` - reinstall plugin and restart
- `netbox-status` - show server and RQ worker status
- `netbox-logs` - tail NetBox server logs
- `rq-status` - check RQ worker status
- `rq-logs` - tail RQ worker logs
- `netbox-shell` - Django shell
- `netbox-manage` - Django manage.py
- `netbox-test` - run tests
- `plugin-install` - reinstall plugin
- `ruff-check|format|fix` - Ruff helpers
## 🐛 Troubleshooting
### Git Authentication Issues
**Problem**: `Permission denied (publickey)` or authentication errors when pushing
```
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
```
**Solutions**:
1. **Check remote URL** - should use HTTPS, not SSH:
```bash
git remote -v
# Should show: https://github.com/bonzo81/netbox-librenms-plugin.git
# NOT: git@github.com:bonzo81/netbox-librenms-plugin.git
```
2. **Fix SSH remote URL**:
```bash
git remote set-url origin https://github.com/bonzo81/netbox-librenms-plugin.git
```
3. **Authenticate with GitHub CLI**:
```bash
gh auth login
gh auth setup-git # Optional: explicitly setup Git integration
```
### Other Issues
- Rebuild container if setup fails (Ctrl+Shift+P → Rebuild Container)
- Check logs `docker-compose logs postgres redis devcontainer`
- Ensure plugin is importable inside container: `python -c "import netbox_librenms_plugin"`
- Run `diagnose` to see whether `plugin-config.py` was detected and the NetBox config path
## 🧹 Cleanup: remove the dev containers
Data warning: removing volumes deletes all dev data (PostgreSQL DB, Redis AOF, NetBox media/static).
1) Close the VS Code Dev Container session first (Command Palette → Dev Containers: Close Remote)
2) From the repo root:
```bash
# Stop and remove containers
docker compose -f .devcontainer/docker-compose.yml down
# Also remove named volumes (DB/media/static) — irreversible
docker compose -f .devcontainer/docker-compose.yml down -v
# Optional: reclaim image space built/pulled for this project
docker compose -f .devcontainer/docker-compose.yml down --rmi local -v
```
Alternatively, run from inside the .devcontainer folder without -f:
```bash
cd .devcontainer
docker compose down # containers only
docker compose down -v # containers + volumes
```

View File

@@ -0,0 +1,30 @@
# GitHub Codespaces NetBox Configuration
# CSRF/hosts setup for Codespaces URLs
import os
codespace_name = os.environ.get("CODESPACE_NAME")
port_domain = os.environ.get("GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN", "app.github.dev")
if codespace_name:
codespaces_url = f"https://{codespace_name}-8000.{port_domain}"
CSRF_TRUSTED_ORIGINS = [
codespaces_url,
"http://localhost:8000",
"http://127.0.0.1:8000",
]
ALLOWED_HOSTS = [
f"{codespace_name}-8000.{port_domain}",
"localhost",
"127.0.0.1",
"*",
]
print(f"🔗 Codespaces detected: {codespace_name}")
print(f"🔒 CSRF Trusted Origins: {CSRF_TRUSTED_ORIGINS}")
print(f"🌐 Allowed Hosts: {ALLOWED_HOSTS}")
else:
CSRF_TRUSTED_ORIGINS = [
"http://localhost:8000",
"http://127.0.0.1:8000",
]
ALLOWED_HOSTS = ["*"]

View File

@@ -0,0 +1,49 @@
# Example: Extra NetBox Configuration
# Copy to 'extra-configuration.py' and customize
import os
# Example: Custom logging configuration
# LOGGING = {
# 'version': 1,
# 'disable_existing_loggers': False,
# 'handlers': {
# 'file': {
# 'level': 'INFO',
# 'class': 'logging.FileHandler',
# 'filename': '/opt/netbox/logs/netbox.log',
# },
# },
# 'loggers': {
# 'netbox_librenms_plugin': {
# 'handlers': ['file'],
# 'level': 'DEBUG',
# 'propagate': True,
# },
# },
# }
# Example: Time zone
# TIME_ZONE = os.environ.get('TIME_ZONE', 'UTC')
# Example: Auth backends
# AUTHENTICATION_BACKENDS = [
# 'django.contrib.auth.backends.RemoteUserBackend',
# 'django.contrib.auth.backends.ModelBackend',
# ]
# Example: Paths
# MEDIA_ROOT = '/opt/netbox/netbox/media'
# STATIC_ROOT = '/opt/netbox/netbox/static'
# Dev banners
BANNER_TOP = "Development Environment"
BANNER_BOTTOM = "NetBox LibreNMS Plugin Dev Container"
BANNER_LOGIN = "NetBox LibreNMS Plugin Development access only"
# Plugin-specific configuration
# PLUGINS_CONFIG = {
# 'netbox_librenms_plugin': {
# 'debug_logging': True,
# },
# }

View File

@@ -0,0 +1,46 @@
"""
Default plugin configuration for the NetBox LibreNMS Plugin in the dev container.
- This file is an example of Plugin configuration
- Copy this file to .devcontainer/plugin-config.py
- Edit values as needed.
- Add config for all other plugins here if any.
"""
# Ensure our plugin is enabled in dev (the loader sets this as a default too)
PLUGINS = [
"netbox_librenms_plugin",
]
# Sample configuration with example servers
PLUGINS_CONFIG = {
"netbox_librenms_plugin": {
"servers": {
"production": {
"display_name": "Production LibreNMS",
"librenms_url": "https://librenms-prod.example.com",
"api_token": "your-prod-token",
"cache_timeout": 300,
"verify_ssl": True,
"interface_name_field": "ifDescr",
},
"testing": {
"display_name": "Test LibreNMS",
"librenms_url": "https://librenms-test.example.com",
"api_token": "your_test_token",
"cache_timeout": 300,
"verify_ssl": False,
"interface_name_field": "ifName",
},
"development": {
"display_name": "Dev LibreNMS",
"librenms_url": "https://librenms-dev.example.com",
"api_token": "your_dev_token",
"cache_timeout": 180,
"verify_ssl": False,
"interface_name_field": "ifDescr",
},
}
}
}

View File

@@ -0,0 +1,82 @@
{
"name": "NetBox LibreNMS Plugin Dev",
"dockerComposeFile": "docker-compose.yml",
"service": "devcontainer",
"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
"forwardPorts": [
8000
],
"portsAttributes": {
"8000": {
"label": "NetBox Web Interface",
"protocol": "http",
"requireLocalPort": false,
"elevateIfNeeded": false
}
},
"otherPortsAttributes": {
"onAutoForward": "ignore"
},
"hostRequirements": {
"memory": "4gb",
"storage": "32gb"
},
"containerEnv": {
"NETBOX_VERSION": "${localEnv:NETBOX_VERSION:latest}",
"DEBUG": "${localEnv:DEBUG:True}",
"DEVELOPER": "${localEnv:DEVELOPER:True}",
"DB_HOST": "${localEnv:DB_HOST:postgres}",
"DB_NAME": "${localEnv:DB_NAME:netbox}",
"DB_USER": "${localEnv:DB_USER:netbox}",
"DB_PASSWORD": "${localEnv:DB_PASSWORD:netbox}",
"REDIS_HOST": "${localEnv:REDIS_HOST:redis}",
"REDIS_PASSWORD": "${localEnv:REDIS_PASSWORD:}",
"SUPERUSER_NAME": "${localEnv:SUPERUSER_NAME:admin}",
"SUPERUSER_EMAIL": "${localEnv:SUPERUSER_EMAIL:admin@example.com}",
"SUPERUSER_PASSWORD": "${localEnv:SUPERUSER_PASSWORD:admin}",
"SKIP_SUPERUSER": "${localEnv:SKIP_SUPERUSER:false}",
"HTTP_PROXY": "${localEnv:HTTP_PROXY}",
"HTTPS_PROXY": "${localEnv:HTTPS_PROXY}",
"http_proxy": "${localEnv:HTTP_PROXY}",
"https_proxy": "${localEnv:HTTPS_PROXY}",
"NO_PROXY": "${localEnv:NO_PROXY}",
"no_proxy": "${localEnv:NO_PROXY}",
"REQUESTS_CA_BUNDLE": "${localEnv:REQUESTS_CA_BUNDLE}",
"SSL_CERT_FILE": "${localEnv:SSL_CERT_FILE}",
"CURL_CA_BUNDLE": "${localEnv:CURL_CA_BUNDLE}",
"ALLOW_GIT_SSL_DISABLE": "${localEnv:ALLOW_GIT_SSL_DISABLE:false}"
},
"features": {},
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"charliermarsh.ruff",
"ms-vscode.vscode-json",
"ms-vscode-remote.remote-containers"
],
"settings": {
"python.defaultInterpreterPath": "/opt/netbox/venv/bin/python",
"python.linting.enabled": false,
"python.formatting.provider": "none",
"[python]": {
"editor.defaultFormatter": "charliermarsh.ruff",
"editor.codeActionsOnSave": {
"source.organizeImports": "explicit",
"source.fixAll.ruff": "explicit"
}
},
"ruff.organizeImports": true,
"ruff.fixAll": true,
"editor.formatOnSave": true,
"files.trimTrailingWhitespace": true,
"editor.rulers": [
88
]
}
}
},
"postCreateCommand": "bash .devcontainer/scripts/setup.sh",
"postAttachCommand": "bash .devcontainer/scripts/welcome.sh",
"remoteUser": "root"
}

View File

@@ -0,0 +1,74 @@
services:
devcontainer:
image: netboxcommunity/netbox:${NETBOX_VERSION:-latest}
volumes:
- ../..:/workspaces:cached
- netbox_media:/opt/netbox/netbox/media
- netbox_static:/opt/netbox/netbox/static
- gh_config:/root/.config/gh
command: sleep infinity
environment:
NETBOX_VERSION: ${NETBOX_VERSION:-latest}
DEBUG: ${DEBUG:-True}
DEVELOPER: ${DEVELOPER:-True}
DB_HOST: ${DB_HOST:-postgres}
DB_NAME: ${DB_NAME:-netbox}
DB_USER: ${DB_USER:-netbox}
DB_PASSWORD: ${DB_PASSWORD:-netbox}
REDIS_HOST: ${REDIS_HOST:-redis}
REDIS_PASSWORD: ${REDIS_PASSWORD:-}
SECRET_KEY: ${SECRET_KEY:-dummydummydummydummydummydummydummydummydummydummydummydummy}
SUPERUSER_NAME: ${SUPERUSER_NAME:-admin}
SUPERUSER_EMAIL: ${SUPERUSER_EMAIL:-admin@example.com}
SUPERUSER_PASSWORD: ${SUPERUSER_PASSWORD:-admin}
SKIP_SUPERUSER: ${SKIP_SUPERUSER:-false}
# Proxy settings (optional)
HTTP_PROXY: ${HTTP_PROXY:-}
HTTPS_PROXY: ${HTTPS_PROXY:-}
http_proxy: ${HTTP_PROXY:-}
https_proxy: ${HTTPS_PROXY:-}
NO_PROXY: ${NO_PROXY:-}
no_proxy: ${NO_PROXY:-}
REQUESTS_CA_BUNDLE: ${REQUESTS_CA_BUNDLE:-}
SSL_CERT_FILE: ${SSL_CERT_FILE:-}
CURL_CA_BUNDLE: ${CURL_CA_BUNDLE:-}
ALLOW_GIT_SSL_DISABLE: ${ALLOW_GIT_SSL_DISABLE:-false}
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
ports:
- "8000:8000"
postgres:
image: postgres:15
environment:
POSTGRES_DB: ${DB_NAME:-netbox}
POSTGRES_USER: ${DB_USER:-netbox}
POSTGRES_PASSWORD: ${DB_PASSWORD:-netbox}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-netbox}"]
interval: 30s
timeout: 10s
retries: 5
redis:
image: redis:7-alpine
command: redis-server --appendonly yes
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 30s
timeout: 10s
retries: 5
volumes:
postgres_data:
redis_data:
netbox_media:
netbox_static:
gh_config:

View File

@@ -0,0 +1,27 @@
# Example: Extra Python Requirements
# Copy this file to 'extra-requirements.txt' and customize as needed
# Example NetBox plugins (uncomment to install)
# netbox-secrets>=1.9.0
# netbox-topology-views>=3.8.0
# netbox-dns>=1.1.0
# netbox-documents>=0.6.0
# Example development/debugging tools
# ipython>=8.0.0
# django-debug-toolbar>=4.0.0
# django-extensions>=3.2.0
# Example testing tools
# factory-boy>=3.3.0
# faker>=22.0.0
# responses>=0.24.0
# Example monitoring
# django-prometheus>=2.3.0
# sentry-sdk>=1.40.0
# Example additional network libs
# netaddr>=0.10.0
# dnspython>=2.4.0
# pysnmp>=5.0.0

View File

@@ -0,0 +1,65 @@
#!/bin/bash
echo "🔍 DevContainer Startup Diagnostics"
echo "=================================="
PLUGIN_WS_DIR="${PLUGIN_DIR:-$(cd "$(dirname "$0")/../.." && pwd)}"
echo "📍 Current working directory: $(pwd)"
echo "👤 Current user: $(whoami)"
echo "🆔 User ID: $(id)"
echo ""
echo "🐳 Container Environment:"
echo " - NETBOX_VERSION: ${NETBOX_VERSION:-not set}"
echo " - DEBUG: ${DEBUG:-not set}"
echo " - SECRET_KEY: ${SECRET_KEY:0:20}... (truncated)"
echo " - DB_HOST: ${DB_HOST:-not set}"
echo " - DB_NAME: ${DB_NAME:-not set}"
echo " - DB_USER: ${DB_USER:-not set}"
echo " - REDIS_HOST: ${REDIS_HOST:-not set}"
echo " - SUPERUSER_NAME: ${SUPERUSER_NAME:-not set}"
echo ""
echo "🔗 Service Connectivity:"
echo " - PostgreSQL: $(timeout 3 bash -c 'cat < /dev/null > /dev/tcp/postgres/5432' 2>/dev/null && echo 'Connected' || echo 'Not reachable')"
echo " - Redis: $(timeout 3 bash -c 'cat < /dev/null > /dev/tcp/redis/6379' 2>/dev/null && echo 'Connected' || echo 'Not reachable')"
echo ""
echo "🗂️ File System:"
echo " - NetBox venv: $(test -f /opt/netbox/venv/bin/activate && echo 'Exists' || echo 'Missing')"
echo " - Plugin directory: $(test -d "$PLUGIN_WS_DIR" && echo 'Exists' || echo 'Missing')"
echo " - Setup script: $(test -f "$PLUGIN_WS_DIR/.devcontainer/scripts/setup.sh" && echo 'Exists' || echo 'Missing')"
echo " - Start script: $(test -f "$PLUGIN_WS_DIR/.devcontainer/scripts/start-netbox.sh" && echo 'Exists' || echo 'Missing')"
echo " - Start script executable: $(test -x "$PLUGIN_WS_DIR/.devcontainer/scripts/start-netbox.sh" && echo 'Yes' || echo 'No')"
echo " - Plugin config: $(test -f "$PLUGIN_WS_DIR/.devcontainer/config/plugin-config.py" && echo 'Found' || echo 'Missing (using defaults)')"
echo " - NetBox config path: /opt/netbox/netbox/netbox/configuration.py"
echo ""
echo "🚀 Process Status:"
if [ -f /tmp/netbox.pid ]; then
PID=$(cat /tmp/netbox.pid)
if [ -z "$PID" ]; then
echo " - NetBox server: PID file exists but is empty"
elif kill -0 "$PID" 2>/dev/null; then
echo " - NetBox server: Running (PID: $PID)"
else
echo " - NetBox server: PID file exists but process not running"
echo " (PID $PID is dead - NetBox may have crashed)"
fi
else
echo " - NetBox server: Not started"
fi
# Check port listening
echo ""
echo "🌍 Port Check:"
if command -v netstat >/dev/null 2>&1; then
echo " - Port 8000: $(netstat -tuln 2>/dev/null | grep :8000 >/dev/null && echo 'Listening' || echo 'Not listening')"
elif command -v ss >/dev/null 2>&1; then
echo " - Port 8000: $(ss -tuln 2>/dev/null | grep :8000 >/dev/null && echo 'Listening' || echo 'Not listening')"
else
echo " - Port 8000: $(cat /proc/net/tcp 2>/dev/null | awk '$2 ~ /:1F40$/ {print "Listening"; exit}' | grep -q "Listening" && echo 'Listening' || echo 'Not listening')"
fi
echo ""
echo "✅ Diagnostic complete!"

View File

@@ -0,0 +1,224 @@
#!/bin/bash
# Quick alias loader for current session
# Usage: source .devcontainer/scripts/load-aliases.sh
export PATH="/opt/netbox/venv/bin:$PATH"
export DEBUG="${DEBUG:-True}"
PLUGIN_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
# Clean up empty CA bundle vars (Compose/devcontainer inject "" when host var is
# unset, which breaks requests/curl). When setup.sh has installed custom CAs
# into the system trust store, point to it instead.
for _ca_var in REQUESTS_CA_BUNDLE SSL_CERT_FILE CURL_CA_BUNDLE; do
_val="${!_ca_var}"
if [ -z "$_val" ]; then
if [ -f /etc/ssl/certs/ca-certificates.crt ]; then
declare -x "$_ca_var=/etc/ssl/certs/ca-certificates.crt"
else
unset "$_ca_var"
fi
fi
done
unset _ca_var _val
# Load shared process management helpers
if ! source "$PLUGIN_DIR/.devcontainer/scripts/process-helpers.sh"; then
printf '%s\n' "Failed to load process-helpers.sh" >&2
return 1
fi
netbox-run-bg() { "$PLUGIN_DIR/.devcontainer/scripts/start-netbox.sh" --background; }
netbox-run() { "$PLUGIN_DIR/.devcontainer/scripts/start-netbox.sh"; }
# Robust stop command that kills both tracked and orphaned processes
netbox-stop() {
echo "🛑 Stopping NetBox and RQ workers..."
if [ -f /tmp/netbox.pid ]; then
local PID
PID=$(cat /tmp/netbox.pid 2>/dev/null)
if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then
if is_expected_pid "$PID" "python.*runserver.*8000"; then
graceful_kill_pid "$PID"
echo " Stopped NetBox (PID: $PID)"
else
echo " Skipping stale /tmp/netbox.pid (PID $PID is not NetBox runserver)"
fi
fi
rm -f /tmp/netbox.pid
fi
if [ -f /tmp/rqworker.pid ]; then
local PID
PID=$(cat /tmp/rqworker.pid 2>/dev/null)
if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then
if is_expected_pid "$PID" "python.*rqworker"; then
graceful_kill_pid "$PID"
echo " Stopped RQ worker (PID: $PID)"
else
echo " Skipping stale /tmp/rqworker.pid (PID $PID is not rqworker)"
fi
fi
rm -f /tmp/rqworker.pid
fi
if pgrep -f "python.*rqworker" >/dev/null 2>&1; then
local ORPHAN_COUNT
ORPHAN_COUNT=$(pgrep -cf "python.*rqworker" 2>/dev/null || echo 0)
graceful_kill_pattern "python.*rqworker"
echo " Killed $ORPHAN_COUNT orphaned RQ worker(s)"
fi
if pgrep -f "python.*runserver.*8000" >/dev/null 2>&1; then
graceful_kill_pattern "python.*runserver.*8000"
echo " Killed orphaned NetBox server(s)"
fi
echo "✅ All processes stopped"
}
netbox-restart() {
netbox-stop && sleep 1 && netbox-run-bg
}
netbox-reload() {
cd "$PLUGIN_DIR" || return 1
if command -v uv >/dev/null 2>&1; then
uv pip install -e . || return 1
else
pip install -e . || return 1
fi
netbox-restart
}
alias netbox-logs="tail -f /tmp/netbox.log"
alias rq-logs="tail -f /tmp/rqworker.log"
netbox-status() {
local PID
if [ -f /tmp/netbox.pid ]; then
PID=$(cat /tmp/netbox.pid 2>/dev/null)
if [ -n "$PID" ] && is_expected_pid "$PID" "python.*runserver.*8000"; then
echo "NetBox is running (PID: $PID)"
else
echo "NetBox is not running"
fi
else
echo "NetBox is not running"
fi
if [ -f /tmp/rqworker.pid ]; then
PID=$(cat /tmp/rqworker.pid 2>/dev/null)
if [ -n "$PID" ] && is_expected_pid "$PID" "python.*rqworker"; then
echo "RQ worker is running (PID: $PID)"
else
echo "RQ worker is not running"
fi
else
echo "RQ worker is not running"
fi
}
rq-status() {
local PID
if [ -f /tmp/rqworker.pid ]; then
PID=$(cat /tmp/rqworker.pid 2>/dev/null)
if [ -n "$PID" ] && is_expected_pid "$PID" "python.*rqworker"; then
echo "RQ worker is running (PID: $PID)"
else
echo "RQ worker is not running"
fi
else
echo "RQ worker is not running"
fi
}
netbox-shell() {
cd /opt/netbox/netbox && source /opt/netbox/venv/bin/activate && python manage.py shell
}
netbox-test() {
cd "$PLUGIN_DIR" && source /opt/netbox/venv/bin/activate && python -m pytest "$@"
}
netbox-manage() {
cd /opt/netbox/netbox && source /opt/netbox/venv/bin/activate && python manage.py "$@"
}
plugin-install() {
cd "$PLUGIN_DIR" || return 1
if command -v uv >/dev/null 2>&1; then
uv pip install -e .
else
pip install -e .
fi
}
plugins-install() {
if [ -f "$PLUGIN_DIR/.devcontainer/extra-requirements.txt" ]; then
source /opt/netbox/venv/bin/activate && pip install -r "$PLUGIN_DIR/.devcontainer/extra-requirements.txt"
else
echo "No .devcontainer/extra-requirements.txt found"
fi
}
ruff-check() { cd "$PLUGIN_DIR" && command ruff check .; }
ruff-format() { cd "$PLUGIN_DIR" && command ruff format .; }
ruff-fix() { cd "$PLUGIN_DIR" && command ruff check --fix .; }
diagnose() { "$PLUGIN_DIR/.devcontainer/scripts/diagnose.sh"; }
# RQ job inspection commands
rq-stats() {
cd /opt/netbox/netbox && source /opt/netbox/venv/bin/activate && python manage.py rqstats
}
rq-jobs() {
cd /opt/netbox/netbox && source /opt/netbox/venv/bin/activate && python manage.py shell -c \
"from django_rq import get_queue; q = get_queue('default'); print(f'Jobs in queue: {len(q)}'); [print(f' {job.id[:8]}: {job.func_name} - {job.get_status()}') for job in q.jobs[:10]]"
}
rq-failed() {
cd /opt/netbox/netbox && source /opt/netbox/venv/bin/activate && python manage.py shell -c \
"from django_rq import get_failed_queue; q = get_failed_queue(); print(f'Failed jobs: {len(q)}'); [print(f' {job.id[:8]}: {job.func_name}') for job in q.jobs[:10]]"
}
rq-recent() {
cd /opt/netbox/netbox && source /opt/netbox/venv/bin/activate && python manage.py shell -c \
"from core.models import Job; jobs = Job.objects.all().order_by('-created')[:10]; [print(f'{j.id}: {j.name[:50]} - {getattr(j.status, \"value\", j.status)} ({j.user})') for j in jobs]"
}
# Help
dev-help() {
echo "🎯 NetBox LibreNMS Plugin Development Commands:"
echo ""
echo "📊 NetBox Server Management:"
echo " netbox-run-bg : Start NetBox in background"
echo " netbox-run : Start NetBox in foreground (for debugging)"
echo " netbox-stop : Stop NetBox and RQ worker"
echo " netbox-restart : Restart NetBox and RQ worker"
echo " netbox-reload : Reinstall plugin and restart NetBox"
echo " netbox-status : Check if NetBox and RQ worker are running"
echo " netbox-logs : View NetBox server logs"
echo ""
echo "⚙️ Background Jobs (RQ Worker):"
echo " rq-status : Check if RQ worker is running"
echo " rq-logs : View RQ worker logs"
echo " rq-stats : Show RQ queue statistics"
echo " rq-jobs : List jobs in default queue"
echo " rq-failed : List failed jobs"
echo " rq-recent : Show recent NetBox jobs"
echo ""
echo "🛠️ Development Tools:"
echo " netbox-shell : Open NetBox Django shell"
echo " netbox-test : Run plugin tests"
echo " netbox-manage : Run Django management commands"
echo " plugin-install : Reinstall plugin in development mode"
echo ""
echo "🧹 Code Quality:"
echo " ruff-check : Check code with Ruff"
echo " ruff-format : Format code with Ruff"
echo " ruff-fix : Auto-fix code issues with Ruff"
echo ""
echo "🔎 Diagnostics:"
echo " diagnose : Run startup diagnostics"
echo " dev-help : Show this help message"
echo ""
echo "📖 NetBox available at: http://localhost:8000 (admin/admin)"
}
echo "✅ Dev helpers loaded! Try: rq-status, rq-stats, rq-recent, dev-help"

View File

@@ -0,0 +1,24 @@
#!/bin/bash
# Shared process management helpers.
# Sourced by load-aliases.sh and start-netbox.sh.
# Graceful termination: SIGTERM, wait, then SIGKILL if still alive.
graceful_kill_pid() {
local pid="$1"
kill -15 "$pid" 2>/dev/null || true
sleep 2
kill -0 "$pid" 2>/dev/null && kill -9 "$pid" 2>/dev/null || true
}
graceful_kill_pattern() {
local pattern="$1"
pkill -15 -f "$pattern" 2>/dev/null || true
sleep 2
pgrep -f "$pattern" >/dev/null 2>&1 && pkill -9 -f "$pattern" 2>/dev/null || true
}
# Verify a PID matches the expected process before killing it
is_expected_pid() {
local pid="$1" pattern="$2"
ps -p "$pid" -o args= 2>/dev/null | grep -Eq "$pattern"
}

329
.devcontainer/scripts/setup.sh Executable file
View File

@@ -0,0 +1,329 @@
#!/bin/bash
set -e
echo "🚀 Setting up NetBox LibreNMS Plugin development environment..."
echo "📍 Current working directory: $(pwd)"
echo "👤 Current user: $(whoami)"
NETBOX_VERSION=${NETBOX_VERSION:-"latest"}
echo "📦 Using NetBox Docker image: netboxcommunity/netbox:${NETBOX_VERSION}"
# ---------------------------------------------------------------------------
# Detect plugin workspace directory (must contain pyproject.toml).
# Prints the resolved path to stdout on success, or an empty string on
# failure. Always exits 0 — callers must check for an empty result.
# ---------------------------------------------------------------------------
detect_plugin_workspace() {
if [ -f "$PWD/pyproject.toml" ]; then
echo "$PWD"
elif [ -d "/workspaces/netbox-librenms-plugin" ] && [ -f "/workspaces/netbox-librenms-plugin/pyproject.toml" ]; then
echo "/workspaces/netbox-librenms-plugin"
else
local candidate
candidate=$(find /workspaces -maxdepth 2 -type f -name pyproject.toml 2>/dev/null | head -n1 | xargs -r dirname || true)
if [ -n "$candidate" ] && [ -f "$candidate/pyproject.toml" ]; then
echo "$candidate"
else
echo ""
fi
fi
}
# Clean up empty CA bundle vars (Compose injects "" when host var is unset)
for _ca_var in REQUESTS_CA_BUNDLE SSL_CERT_FILE CURL_CA_BUNDLE; do
_val="${!_ca_var}"
[ -z "$_val" ] && unset "$_ca_var"
done
unset _ca_var _val
# Configure proxy for apt and pip if proxy environment variables are set
if [ -n "$HTTP_PROXY" ] || [ -n "$HTTPS_PROXY" ]; then
echo "🌐 Configuring proxy settings..."
# Configure apt proxy
if [ -n "$HTTP_PROXY" ]; then
echo "Acquire::http::Proxy \"$HTTP_PROXY\";" > /etc/apt/apt.conf.d/80proxy
SAFE_HTTP_PROXY=$(echo "$HTTP_PROXY" | sed 's|://[^@]*@|://***:***@|')
echo " ✓ apt HTTP proxy: $SAFE_HTTP_PROXY"
fi
if [ -n "$HTTPS_PROXY" ]; then
echo "Acquire::https::Proxy \"$HTTPS_PROXY\";" >> /etc/apt/apt.conf.d/80proxy
SAFE_HTTPS_PROXY=$(echo "$HTTPS_PROXY" | sed 's|://[^@]*@|://***:***@|')
echo " ✓ apt HTTPS proxy: $SAFE_HTTPS_PROXY"
fi
# Configure pip proxy via environment (already set, but ensure it's exported)
export HTTP_PROXY HTTPS_PROXY http_proxy https_proxy NO_PROXY no_proxy
# Install custom CA certificate into the system trust store (for MITM proxies)
PLUGIN_WS_DIR_EARLY="$(detect_plugin_workspace)"
[ -z "$PLUGIN_WS_DIR_EARLY" ] && PLUGIN_WS_DIR_EARLY="/workspaces/netbox-librenms-plugin"
CA_BUNDLE_SRC="$PLUGIN_WS_DIR_EARLY/ca-bundle.crt"
if [ -f "$CA_BUNDLE_SRC" ]; then
echo "🔐 Installing custom CA certificate into system trust store..."
cert_count=$(grep -c '-----BEGIN CERTIFICATE-----' "$CA_BUNDLE_SRC" 2>/dev/null || true)
if [ "${cert_count:-0}" -eq 0 ]; then
echo " ⚠️ ca-bundle.crt does not contain any PEM certificate blocks; skipping CA install."
else
mkdir -p /usr/local/share/ca-certificates/proxy
# Remove stale split fragments so they don't accumulate across rebuilds
find /usr/local/share/ca-certificates/proxy -maxdepth 1 -name 'cert-*' -delete 2>/dev/null || true
# Split the bundle into individual certs — update-ca-certificates needs one
# cert per file and skips non-CA leaf certs, so extract each PEM block as
# a separate .crt file.
csplit -z -f /usr/local/share/ca-certificates/proxy/cert- \
"$CA_BUNDLE_SRC" '/-----BEGIN CERTIFICATE-----/' '{*}' \
>/dev/null 2>&1
CSPLIT_STATUS=$?
if [ "$CSPLIT_STATUS" -ne 0 ]; then
echo " ⚠️ Failed to split ca-bundle.crt (csplit exit code: $CSPLIT_STATUS). Skipping CA install."
elif compgen -G "/usr/local/share/ca-certificates/proxy/cert-*" > /dev/null; then
# Rename split fragments to .crt
for f in /usr/local/share/ca-certificates/proxy/cert-*; do
mv "$f" "${f}.crt" 2>/dev/null || true
done
update-ca-certificates 2>/dev/null
echo " ✓ CA certificate installed into system trust store ($cert_count cert(s))"
else
echo " ⚠️ No certificate fragments were generated from ca-bundle.crt; skipping CA install."
fi
fi
# Point environment variables to the system bundle
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
export GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
# Configure pip globally so isolated virtualenvs (e.g. pre-commit) also
# use the system CA bundle instead of their bundled certifi.
pip config set global.cert /etc/ssl/certs/ca-certificates.crt 2>/dev/null || true
else
echo " No ca-bundle.crt found at $CA_BUNDLE_SRC, skipping CA install"
# Only disable git SSL verification if explicitly opted-in via ALLOW_GIT_SSL_DISABLE.
# Silently disabling SSL is a security risk; prefer providing a CA bundle instead.
if [ "${ALLOW_GIT_SSL_DISABLE:-false}" = "true" ]; then
git config --global http.sslVerify false
echo " ⚠️ git SSL verification disabled globally (ALLOW_GIT_SSL_DISABLE=true)"
else
echo " ⚠️ No CA bundle found and git SSL verification was NOT disabled."
echo " If you need to disable it, set ALLOW_GIT_SSL_DISABLE=true in .devcontainer/.env"
echo " Preferred: provide a ca-bundle.crt in the workspace root instead."
fi
fi
fi
# Verify NetBox virtual environment exists
if [ ! -f "/opt/netbox/venv/bin/activate" ]; then
echo "❌ NetBox virtual environment not found at /opt/netbox/venv/"
echo "This might indicate an issue with the NetBox Docker image."
exit 1
fi
echo "🐍 Activating NetBox virtual environment..."
source /opt/netbox/venv/bin/activate
# Choose installer (uv if available, else pip)
if command -v uv >/dev/null 2>&1; then
PIP_CMD="uv pip"
else
PIP_CMD="pip"
fi
# Install dev tools
echo "🔧 Installing development dependencies..."
apt-get update -qq
apt-get install -y -qq net-tools git
$PIP_CMD install pytest pytest-django ruff pre-commit
# Install GitHub CLI (gh)
# NOTE: The chained && commands below mean a partial failure (e.g. wget succeeds
# but apt-get install gh fails) may leave artifacts (keyring, sources list, temp
# file). This is acceptable here because it only runs during container build —
# a rebuild will retry from scratch. If this block is ever moved to a runtime
# script, consider adding a trap or explicit cleanup on error.
if ! command -v gh >/dev/null 2>&1; then
echo "🔧 Installing GitHub CLI..."
(type -p wget >/dev/null || apt-get install -y -qq wget) \
&& install -d -m 755 /etc/apt/keyrings \
&& out=$(mktemp) \
&& wget -qO "$out" https://cli.github.com/packages/githubcli-archive-keyring.gpg \
&& cat "$out" | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
&& chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& apt-get update -qq \
&& apt-get install -y -qq gh \
&& rm -f "$out" \
&& echo " ✓ GitHub CLI installed: $(gh --version | head -1)" \
|| echo "⚠️ GitHub CLI installation failed (non-fatal)"
fi
# Detect plugin workspace directory using the shared helper
PLUGIN_WS_DIR="$(detect_plugin_workspace)"
if [ -z "$PLUGIN_WS_DIR" ]; then
echo "❌ Could not locate plugin workspace directory (pyproject.toml not found)."
echo " Checked: $PWD and /workspaces/*"
exit 1
fi
echo "📂 Plugin workspace: $PLUGIN_WS_DIR"
# Install this plugin in development mode
echo "📦 Installing plugin in development mode from: $PLUGIN_WS_DIR"
if [ ! -f "$PLUGIN_WS_DIR/pyproject.toml" ] && [ ! -f "$PLUGIN_WS_DIR/setup.py" ]; then
echo "❌ Neither pyproject.toml nor setup.py found in $PLUGIN_WS_DIR"
ls -la "$PLUGIN_WS_DIR" || true
exit 2
fi
cd "$PLUGIN_WS_DIR"
$PIP_CMD install -e .
CONF_FILE="/opt/netbox/netbox/netbox/configuration.py"
# Optional extras
if [ -f "$PLUGIN_WS_DIR/.devcontainer/extra-requirements.txt" ]; then
echo "📦 Installing extra packages from extra-requirements.txt..."
$PIP_CMD install -r "$PLUGIN_WS_DIR/.devcontainer/extra-requirements.txt"
fi
# Inject plugin loader into standard NetBox configuration if present
if [ -f "$CONF_FILE" ]; then
if ! grep -q "# Devcontainer Plugins Loader" "$CONF_FILE" 2>/dev/null; then
{
echo "";
echo "# Devcontainer Plugins Loader";
echo "# Import PLUGINS/PLUGINS_CONFIG and optional extras dynamically from the workspace";
echo "import importlib.util, os";
echo "PLUGINS = ['netbox_librenms_plugin']";
echo "PLUGINS_CONFIG = {'netbox_librenms_plugin': {}}";
echo "_pc_path = '$PLUGIN_WS_DIR/.devcontainer/config/plugin-config.py'";
echo "if os.path.isfile(_pc_path):";
echo " _spec = importlib.util.spec_from_file_location('workspace_plugin_config', _pc_path)";
echo " _mod = importlib.util.module_from_spec(_spec)";
echo " try:";
echo " _spec.loader.exec_module(_mod) # type: ignore[attr-defined]";
echo " PLUGINS = getattr(_mod, 'PLUGINS', PLUGINS)";
echo " PLUGINS_CONFIG = getattr(_mod, 'PLUGINS_CONFIG', PLUGINS_CONFIG)";
echo " except Exception as e:";
echo " print(f'⚠️ Failed to load plugin-config.py: {e}')";
echo "else:";
echo " print(' plugin-config.py not found; using defaults')";
echo "# Import optional extra NetBox configuration (uppercase settings)";
echo "_xc_path = '$PLUGIN_WS_DIR/.devcontainer/config/extra-configuration.py'";
echo "if os.path.isfile(_xc_path):";
echo " _xc_spec = importlib.util.spec_from_file_location('workspace_extra_configuration', _xc_path)";
echo " _xc_mod = importlib.util.module_from_spec(_xc_spec)";
echo " try:";
echo " _xc_spec.loader.exec_module(_xc_mod) # type: ignore[attr-defined]";
echo " for _name in dir(_xc_mod):";
echo " if _name.isupper():";
echo " globals()[_name] = getattr(_xc_mod, _name)";
echo " except Exception as e:";
echo " print(f'⚠️ Failed to apply extra-configuration.py: {e}')";
echo "# Import Codespaces configuration when applicable (uppercase settings)";
echo "_cs_path = '$PLUGIN_WS_DIR/.devcontainer/config/codespaces-configuration.py'";
echo "if os.environ.get('CODESPACES') == 'true' and os.path.isfile(_cs_path):";
echo " _cs_spec = importlib.util.spec_from_file_location('workspace_codespaces_configuration', _cs_path)";
echo " _cs_mod = importlib.util.module_from_spec(_cs_spec)";
echo " try:";
echo " _cs_spec.loader.exec_module(_cs_mod) # type: ignore[attr-defined]";
echo " for _name in dir(_cs_mod):";
echo " if _name.isupper():";
echo " globals()[_name] = getattr(_cs_mod, _name)";
echo " except Exception as e:";
echo " print(f'⚠️ Failed to apply codespaces-configuration.py: {e}')";
echo "# Ensure SECRET_KEY exists: prefer environment, fallback to a dev placeholder";
echo "if 'SECRET_KEY' not in globals() or not SECRET_KEY:";
echo " SECRET_KEY = os.environ.get('SECRET_KEY', 'dummydummydummydummydummydummydummydummydummydummydummydummy')";
} >> "$CONF_FILE"
fi
if grep -q "netbox_librenms_plugin" "$CONF_FILE" 2>/dev/null; then
echo "✅ Plugin configuration exists in NetBox settings"
fi
else
echo "⚠️ Warning: $CONF_FILE not found"
echo "Plugin configuration may need to be added manually"
fi
# Run migrations and collectstatic
cd /opt/netbox/netbox
# Wait briefly for DB (compose healthchecks should ensure availability)
export DEBUG="${DEBUG:-True}"
echo "🗃️ Applying database migrations..."
python manage.py migrate 2>&1 | grep -E "(Operations to perform|Running migrations|Apply all migrations|No migrations to apply|\s+Applying|\s+OK)" || true
echo "🔐 Creating superuser (if not exists)..."
echo " Credentials are read from environment variables (see .devcontainer/.env)"
python manage.py shell -c "
import os
from django.contrib.auth import get_user_model
User = get_user_model()
username = (os.environ.get('SUPERUSER_NAME') or '').strip() or 'admin'
email = (os.environ.get('SUPERUSER_EMAIL') or '').strip() or 'admin@example.com'
password = (os.environ.get('SUPERUSER_PASSWORD') or '').strip() or 'admin'
if not User.objects.filter(username=username).exists():
User.objects.create_superuser(username, email, password)
print(f'Created superuser: {username}')
else:
print(f'Superuser {username} already exists')
" 2>/dev/null || true
echo "📊 Collecting static files..."
python manage.py collectstatic --noinput >/dev/null 2>&1 || true
# Set up pre-commit hooks
echo "🪝 Installing pre-commit hooks..."
cd "$PLUGIN_WS_DIR"
git config --global --add safe.directory "$PLUGIN_WS_DIR"
pre-commit install --install-hooks 2>/dev/null || echo "⚠️ Pre-commit hook installation failed (may already be installed)"
# Ensure scripts are executable
chmod +x "$PLUGIN_WS_DIR/.devcontainer/scripts/start-netbox.sh" || true
chmod +x "$PLUGIN_WS_DIR/.devcontainer/scripts/diagnose.sh" || true
chmod +x "$PLUGIN_WS_DIR/.devcontainer/scripts/load-aliases.sh" || true
# Load aliases and welcome message from the canonical source (load-aliases.sh).
# Appended to .bashrc so every interactive shell gets them automatically.
# Guard with a sentinel so rerunning setup.sh doesn't create duplicate entries.
BASHRC_SENTINEL="# NetBox LibreNMS Plugin — source aliases from the single canonical file"
if ! grep -qF "$BASHRC_SENTINEL" ~/.bashrc 2>/dev/null; then
cat >> ~/.bashrc << EOF
$BASHRC_SENTINEL
source "$PLUGIN_WS_DIR/.devcontainer/scripts/load-aliases.sh"
# Show welcome message for new terminals
bash "$PLUGIN_WS_DIR/.devcontainer/scripts/welcome.sh"
EOF
fi
# Fix Git remote URLs for dev container compatibility
echo "🔧 Checking Git remote configuration..."
cd "$PLUGIN_WS_DIR"
CURRENT_REMOTE=$(git remote get-url origin 2>/dev/null || echo "")
if [[ "$CURRENT_REMOTE" == git@github.com:* ]]; then
# Convert SSH URL to HTTPS for dev container compatibility
HTTPS_URL=$(echo "$CURRENT_REMOTE" | sed 's|git@github.com:|https://github.com/|')
git remote set-url origin "$HTTPS_URL"
echo "✅ Converted Git remote from SSH to HTTPS: $HTTPS_URL"
echo " This ensures compatibility with GitHub CLI authentication in dev containers"
elif [[ "$CURRENT_REMOTE" == https://github.com/* ]]; then
echo "✅ Git remote already uses HTTPS: $CURRENT_REMOTE"
else
echo " Git remote URL: $CURRENT_REMOTE (no changes needed)"
fi
# Final validation
cd /opt/netbox/netbox
if python -c "import netbox_librenms_plugin; print('✅ Plugin import successful')" 2>/dev/null | grep -q "✅ Plugin import successful"; then
echo "✅ Plugin is properly installed and importable"
else
echo "⚠️ Warning: Plugin may not be properly installed"
fi
echo ""
echo "🚀 NetBox LibreNMS Plugin Dev Environment Ready!"

View File

@@ -0,0 +1,107 @@
#!/bin/bash
# Check if we should run in background or foreground
BACKGROUND=false
if [ "$1" = "--background" ] || [ "$1" = "-b" ]; then
BACKGROUND=true
fi
echo "🌐 Starting NetBox development server..."
# Set required environment variables
export DEBUG="${DEBUG:-True}"
# Detect Codespaces and set access URL
if [ "$CODESPACES" = "true" ] && [ -n "$CODESPACE_NAME" ]; then
GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN="${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN:-app.github.dev}"
ACCESS_URL="https://${CODESPACE_NAME}-8000.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}"
echo "🔗 GitHub Codespaces detected"
else
ACCESS_URL="http://localhost:8000"
echo "🐛 Debug: ACCESS_URL is set to: $ACCESS_URL"
fi
# Load shared process management helpers
if ! source "$(dirname "$0")/process-helpers.sh"; then
echo "ERROR: Failed to load process-helpers.sh" >&2
exit 1
fi
# Kill any orphaned processes (not tracked by PID file)
echo "🧹 Cleaning up orphaned processes..."
if pgrep -f "python.*rqworker" >/dev/null 2>&1; then
echo " Found orphaned RQ workers, killing..."
graceful_kill_pattern "python.*rqworker"
fi
if pgrep -f "python.*runserver.*8000" >/dev/null 2>&1; then
echo " Found orphaned NetBox servers, killing..."
graceful_kill_pattern "python.*runserver.*8000"
fi
# Stop any tracked processes from PID files
if [ -f /tmp/netbox.pid ]; then
OLD_PID=$(cat /tmp/netbox.pid 2>/dev/null)
if [ -n "$OLD_PID" ] && kill -0 "$OLD_PID" 2>/dev/null; then
if is_expected_pid "$OLD_PID" "python.*runserver.*8000"; then
graceful_kill_pid "$OLD_PID"
else
echo "⚠️ Skipping stale /tmp/netbox.pid (PID $OLD_PID is not NetBox runserver)"
fi
fi
rm -f /tmp/netbox.pid
fi
if [ -f /tmp/rqworker.pid ]; then
OLD_PID=$(cat /tmp/rqworker.pid 2>/dev/null)
if [ -n "$OLD_PID" ] && kill -0 "$OLD_PID" 2>/dev/null; then
if is_expected_pid "$OLD_PID" "python.*rqworker"; then
graceful_kill_pid "$OLD_PID"
else
echo "⚠️ Skipping stale /tmp/rqworker.pid (PID $OLD_PID is not rqworker)"
fi
fi
rm -f /tmp/rqworker.pid
fi
# Activate NetBox virtual environment
source /opt/netbox/venv/bin/activate
# Navigate to NetBox directory
cd /opt/netbox/netbox
# Start RQ worker in background
echo "⚙️ Starting RQ worker..."
(
source /opt/netbox/venv/bin/activate
cd /opt/netbox/netbox
python manage.py rqworker --verbosity=1
) > /tmp/rqworker.log 2>&1 &
RQ_PID=$!
echo $RQ_PID > /tmp/rqworker.pid
echo "✅ RQ worker started (PID: $RQ_PID)"
if [ "$BACKGROUND" = true ]; then
echo "🚀 Starting NetBox in background"
(
export DEBUG="${DEBUG:-True}"
source /opt/netbox/venv/bin/activate
cd /opt/netbox/netbox
python manage.py runserver 0.0.0.0:8000 --verbosity=0
) > /tmp/netbox.log 2>&1 &
NETBOX_PID=$!
echo $NETBOX_PID > /tmp/netbox.pid
echo "✅ NetBox started in background (PID: $NETBOX_PID)"
echo "📍 Access NetBox at: $ACCESS_URL"
echo "💡 If clicking the URL opens 0.0.0.0:8000, manually type: localhost:8000"
echo "📄 View logs with: netbox-logs"
echo "🛑 Stop NetBox with: netbox-stop"
else
echo "🌍 Starting NetBox in foreground"
echo "📍 Access NetBox at: $ACCESS_URL"
echo "💡 If clicking the URL opens 0.0.0.0:8000, manually type: localhost:8000"
echo ""
python manage.py runserver 0.0.0.0:8000
fi

View File

@@ -0,0 +1,56 @@
#!/bin/bash
# Ensure aliases are available in the postAttach terminal session
source "$(dirname "$0")/load-aliases.sh" 2>/dev/null
echo ""
echo "🎯 NetBox LibreNMS Plugin Development Environment"
PLUGIN_WS_DIR="${PLUGIN_DIR:-$(cd "$(dirname "$0")/../.." && pwd)}"
if [ ! -f "$PLUGIN_WS_DIR/.devcontainer/config/plugin-config.py" ]; then
echo ""
echo "⚠️ Plugin configuration not found: .devcontainer/config/plugin-config.py"
echo " Create it first: cp .devcontainer/config/plugin-config.py.example .devcontainer/config/plugin-config.py"
echo " Then edit it and set your plugin values (e.g. LibreNMS server URL/token)"
fi
# Check GitHub CLI authentication status
echo ""
if command -v gh >/dev/null 2>&1; then
if gh auth status >/dev/null 2>&1; then
# Get the authenticated user info
GH_USER=$(gh api user --jq '.login' 2>/dev/null || echo "unknown")
echo "✅ GitHub authenticated as: $GH_USER"
echo " Git is configured for GitHub operations"
else
echo "🔑 GitHub CLI available but not authenticated"
echo " Run 'gh auth login' to authenticate with GitHub"
echo " This will automatically configure Git for pushing/pulling"
fi
else
echo "⚠️ GitHub CLI not available"
fi
echo ""
if [ -n "$CODESPACES" ]; then
echo "🌐 GitHub Codespaces Environment:"
echo " NetBox will be available via automatic port forwarding"
echo " Check the 'Ports' panel for the forwarded port labeled 'NetBox Web Interface'"
if [ -n "$CODESPACE_NAME" ]; then
# Try to construct the likely URL (GitHub Codespaces pattern)
CODESPACE_URL="https://${CODESPACE_NAME}-8000.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN:-preview.app.github.dev}"
echo " Expected URL: $CODESPACE_URL"
fi
echo " 💡 Click the link in the Ports panel or look for the 'Open in Browser' button"
else
echo "🖥️ Local Development Environment:"
echo " NetBox will be available at: http://localhost:8000 (paste into you browser)"
fi
echo ""
echo "🚀 Quick start:"
echo " • Type 'netbox-run' to start the development server"
echo " • Type 'netbox-restart' to restart NetBox (after config changes)"
echo " • Type 'dev-help' to see all available commands"
echo " • Edit code in the workspace - auto-reload is enabled"
echo ""

21
.editorconfig Normal file
View File

@@ -0,0 +1,21 @@
# http://editorconfig.org
root = true
[*]
indent_style = space
indent_size = 4
trim_trailing_whitespace = true
insert_final_newline = true
charset = utf-8
end_of_line = lf
[*.bat]
indent_style = tab
end_of_line = crlf
[LICENSE]
insert_final_newline = false
[Makefile]
indent_style = tab

2
.flake8 Normal file
View File

@@ -0,0 +1,2 @@
[flake8]
max-line-length = 120

15
.github/FUNDING.yml vendored Normal file
View File

@@ -0,0 +1,15 @@
# These are supported funding model platforms
github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
patreon: # Replace with a single Patreon username
open_collective: # Replace with a single Open Collective username
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
polar: # Replace with a single Polar username
buy_me_a_coffee: bonzo81 # Replace with a single Buy Me a Coffee username
thanks_dev: # Replace with a single thanks.dev username
custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

61
.github/ISSUE_TEMPLATE/bug_report.yaml vendored Normal file
View File

@@ -0,0 +1,61 @@
---
name: 🐛 Bug Report
description: Report a reproducible bug in the current release of NetBox Librenms Plugin
labels: ["type: bug"]
body:
- type: markdown
attributes:
value: >
**NOTE:** This form is only for reporting _reproducible bugs_ in a current NetBox Librenms Plugin release.
- type: input
attributes:
label: NetBox Librenms Plugin version
description: What version of NetBox Librenms Plugin are you currently running?
placeholder: v0.1.0
validations:
required: true
- type: input
attributes:
label: NetBox version
description: What version of NetBox are you currently running?
placeholder: v3.6.0
validations:
required: true
- type: dropdown
attributes:
label: Python version
description: What version of Python are you currently running?
options:
- "3.8"
- "3.9"
- "3.10"
- "3.11"
- "3.12"
- "3.13"
validations:
required: true
- type: textarea
attributes:
label: Steps to Reproduce
description: >
Please provide a minimal working example to demonstrate the bug. Ensure that your example is as concise as possible
while adequately illustrating the issue.
_Please refrain from including any confidential or sensitive
information in your example._
validations:
required: true
- type: textarea
attributes:
label: Expected Behavior
description: What did you expect to happen?
placeholder: The script should execute without raising any errors or exceptions
validations:
required: true
- type: textarea
attributes:
label: Observed Behavior
description: What happened instead?
placeholder: A TypeError exception was raised
validations:
required: true

12
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@@ -0,0 +1,12 @@
# Reference: https://help.github.com/en/github/building-a-strong-community/configuring-issue-templates-for-your-repository#configuring-the-template-chooser
blank_issues_enabled: false
contact_links:
- name: 📖 Contributing Policy
url: https://github.com/bonzo81/netbox-librenms-plugin/blob/master/docs/contributing.md
about: "Please read through our contributing policy before opening an issue or pull request."
- name: ❓ Discussion
url: https://github.com/bonzo81/netbox-librenms-plugin/discussions
about: "If you're just looking for help, try starting a discussion instead."
- name: 💬 Community Slack
url: https://netdev.chat
about: "Join #netbox on the NetDev Community Slack for assistance with installation issues and other problems."

View File

@@ -0,0 +1,59 @@
---
name: ✨ Feature Request
description: Propose a new NetBox Librenms Plugin feature or enhancement
labels: ["type: feature"]
body:
- type: markdown
attributes:
value: >
**NOTE:** This form is only for submitting well-formed proposals to extend or modify
NetBox Librenms Plugin in some way. If you're trying to solve a problem but can't figure out how, or if
you still need time to work on the details of a proposed new feature, please start a
[discussion](https://github.com/bonzo81/netbox-librenms-plugin/discussions) instead.
- type: input
attributes:
label: NetBox Librenms Plugin version
description: What version of NetBox Librenms Plugin are you currently running?
placeholder: v0.1.0
validations:
required: true
- type: input
attributes:
label: NetBox version
description: What version of NetBox are you currently running?
placeholder: v3.6.0
validations:
required: true
- type: dropdown
attributes:
label: Feature type
options:
- Data model extension
- New functionality
- Change to existing functionality
validations:
required: true
- type: textarea
attributes:
label: Proposed functionality
description: >
Describe in detail the new feature or behavior you are proposing. Include any specific changes
to work flows, data models, and/or the user interface. The more detail you provide here, the
greater chance your proposal has of being discussed. Feature requests which don't include an
actionable implementation plan will be rejected.
validations:
required: true
- type: textarea
attributes:
label: Use case
description: >
Explain how adding this functionality would benefit NetBox Librenms Plugin users. What need does it address?
validations:
required: true
- type: textarea
attributes:
label: External dependencies
description: >
List any new dependencies on external libraries or services that this new feature would
introduce. For example, does the proposal require the installation of a new Python package?
(Not all new features introduce new dependencies.)

View File

@@ -0,0 +1,24 @@
---
name: 🏡 Housekeeping
description: A change pertaining to the codebase itself (developers only)
labels: ["type: housekeeping"]
body:
- type: markdown
attributes:
value: >
**NOTE:** This template is for use by maintainers only. Please do not submit
an issue using this template unless you have been specifically asked to do so.
- type: textarea
attributes:
label: Proposed Changes
description: >
Describe in detail the new feature or behavior you'd like to propose.
Include any specific changes to work flows, data models, or the user interface.
validations:
required: true
- type: textarea
attributes:
label: Justification
description: Please provide justification for the proposed change(s).
validations:
required: true

75
.github/copilot-instructions.md vendored Normal file
View File

@@ -0,0 +1,75 @@
# NetBox LibreNMS Plugin AI Assistant Guide
> **Note:** Additional context-specific instructions are in `.github/instructions/`:
> - [testing.instructions.md](instructions/testing.instructions.md) applies to `tests/**`
> - [frontend.instructions.md](instructions/frontend.instructions.md) applies to templates and static files
> - [background-jobs.instructions.md](instructions/background-jobs.instructions.md) applies to `jobs.py`, import views, and import utilities
> - [sync.instructions.md](instructions/sync.instructions.md) applies to sync views, base views, tables, and sync JS
## Architecture & Key Modules
- Plugin hooks into NetBox (Django 5) under `netbox_librenms_plugin/`; respect NetBox plugin APIs (`navigation.py`, `urls.py`, `api/`).
- LibreNMS communication lives in `librenms_api.py`; reuse this client instead of new `requests` calls. It handles multi-server configs via `LibreNMSSettings` model and the `servers` plugin config, plus caching via Django cache + custom fields.
- Views follow a three-layer structure:
- **Base views** (`views/base/`) — abstract views for each sync resource (`BaseInterfaceTableView`, `BaseCableTableView`, `BaseIPAddressTableView`, `BaseVLANTableView`).
- **Object sync views** (`views/object_sync/`) — concrete per-model views registered as tabs on NetBox's Device/VM detail pages via `@register_model_view(Device, ...)`. These wire base views to models.
- **Sync action views** (`views/sync/`) — POST-only views that apply changes (add/change/delete NetBox objects). Includes `interfaces.py`, `cables.py`, `ip_addresses.py`, `vlans.py`, `devices.py`, `device_fields.py`, `locations.py`.
- **Shared mixins** (`views/mixins.py`) — `LibreNMSPermissionMixin`, `NetBoxObjectPermissionMixin`, `LibreNMSAPIMixin`, `CacheMixin`, `VlanAssignmentMixin`.
- All four sync resources (interfaces, cables, IP addresses, VLANs) follow the same three-layer pattern. VLAN sync additionally uses `VlanAssignmentMixin` for VLAN group scope resolution (Rack → Location → Site → SiteGroup → Region → Global).
- New views should extend the closest base class and compose mixins.
- Tables (`tables/*.py`) and templates (`templates/netbox_librenms_plugin/`) drive the UI. See `frontend.instructions.md` for HTMX, template, and styling conventions.
- Forms (`forms.py`) include dynamic LibreNMS API-populated choices (location dropdowns, poller groups) and a split-form pattern for settings (server config form + import settings form).
- `import_validation_helpers.py` centralizes validation state mutation during import (role/cluster/rack assignment, issue removal, status recalculation).
## Data & Sync Conventions
- Devices/VMs map to LibreNMS via the `librenms_id` custom field, then cached if absent. Always call `LibreNMSAPI.get_librenms_id` instead of touching the field directly.
- Matching is intentionally **exact-only** for site, platform, device type, and role. See `utils.py` (`find_matching_site`, `match_librenms_hardware_to_device_type`, `find_matching_platform`). Do not add fuzzy matching.
- Sync pipelines generally fetch LibreNMS data (`librenms_api.py`), cache it (`CacheMixin`), build comparison tables (`tables/`), and render HTMX fragments (`templates/netbox_librenms_plugin/htmx/`). Follow that flow for new resources.
- Virtual chassis support uses `get_virtual_chassis_member()` for port-to-member mapping and `get_librenms_sync_device()` for VC priority-based device selection.
## Developer Workflow
- Prefer the devcontainer commands (`netbox-run`, `netbox-run-bg`, `netbox-reload`, `netbox-logs`) described in `.devcontainer/README.md`. They manage NetBox + plugin reloading.
- Static assets belong in `static/netbox_librenms_plugin/`; run NetBox's `collectstatic` when bundling, but the devcontainer handles this automatically.
## Integration Touchpoints
- REST endpoints for imports live in `views/imports/actions.py` (with the list view in `views/imports/list.py`) and surface via `urls.py`. They also emit HTMX fragments (`templates/netbox_librenms_plugin/htmx/device_import_row.html`, etc.). Keep server responses and HTMX targets in sync.
- API serializers (`api/serializers.py`) mirror models for external consumption. Update serializers and `api/views.py` together to avoid contract drift.
- Navigation and menu items are registered in `navigation.py`; extend there for new sections so NetBox renders links correctly.
## Permission System
- Uses two-tier permissions via `LibreNMSSettings` model: `view_librenmssettings` (read) and `change_librenmssettings` (write). See `docs/development/permissions.md`.
- Permission constants in `constants.py`: `PERM_VIEW_PLUGIN` and `PERM_CHANGE_PLUGIN`.
### Plugin-Level Permissions
- All views inherit `LibreNMSPermissionMixin` from `views/mixins.py`, which sets `permission_required = PERM_VIEW_PLUGIN` and provides:
- `has_write_permission()` — checks `PERM_CHANGE_PLUGIN`.
- `require_write_permission()` — returns error response (HTMX `HX-Redirect` or standard redirect) if denied.
- `require_write_permission_json()` — returns `JsonResponse(403)` if denied (for AJAX endpoints).
### Object-Level Permissions
- `NetBoxObjectPermissionMixin` adds a **second layer** of permission checking for NetBox model operations (add/change/delete on Device, Interface, VLAN, etc.).
- Views declare `required_object_permissions` dict mapping HTTP methods to `[(action, Model)]` tuples, e.g.:
```python
required_object_permissions = {"POST": [("add", VLAN), ("change", VLAN)]}
```
- Some views set `required_object_permissions` dynamically per-request (e.g., `SyncInterfacesView` switches between `Interface` and `VMInterface` based on object type).
- Provides:
- `check_object_permissions(method)` → `(bool, missing_perms_list)`
- `require_object_permissions(method)` — redirect/HTMX on failure.
- `require_object_permissions_json(method)` — JSON 403 on failure.
- `require_all_permissions(method)` — combined plugin write + object perms check (redirect/HTMX).
- `require_all_permissions_json(method)` — combined check, JSON variant.
- **Sync POST handlers** must call `require_all_permissions("POST")` (not just `require_write_permission()`) and return early if it returns a response. AJAX/JSON endpoints use `require_all_permissions_json("POST")`.
- `_get_safe_redirect_url(request)` validates referrer URLs to prevent open-redirect attacks.
### Permission Helpers for Background Jobs
- Background jobs run outside view context and cannot use view mixins. Use standalone helpers from `import_utils.py` (`check_user_permissions`, `require_permissions`). See `background-jobs.instructions.md` for details.
### API & Navigation Permissions
- API endpoints use `LibreNMSPluginPermission` class in `api/views.py` (GET=view, others=change).
- Navigation menu (`navigation.py`) has 3 groups: **Settings** (Plugin Settings, Interface Mappings), **Import** (LibreNMS Import), **Status Check** (Site & Location Sync, Device Status, VM Status). All items use `permissions=[PERM_VIEW_PLUGIN]`.
- **Background job polling requires superuser** — non-superusers fall back to synchronous mode. See `background-jobs.instructions.md` for details.
## When in Doubt
- Check docs in `docs/development/` for structure, view inheritance, mixins, and template conventions before introducing new patterns.
- Review the existing sync views (e.g., `views/sync/interfaces.py`) as reference implementations for data flow and caching patterns.
- Coordinate any schema changes through Django migrations in `migrations/` and update `models.py` + admin/pydantic representations accordingly.

14
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,14 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
groups:
github-actions:
patterns:
- "*"
- package-ecosystem: "uv"
directory: "/"
schedule:
interval: "weekly"

View File

@@ -0,0 +1,84 @@
---
applyTo: "**/jobs.py,**/views/imports/**,**/import_utils.py,**/import_validation_helpers.py"
description: Background job architecture, import workflow, and task management patterns
---
# Background Jobs & Import Workflow
## Job Architecture
- Background jobs use NetBox's `JobRunner` base class (`netbox.jobs.JobRunner`) for long-running operations like device filtering with VC detection.
- Jobs run via Redis Queue (RQ) in Redis, separate from the database Job model. Real-time status must be checked via RQ, not the database.
## Critical Job Architecture Points
- Job UUID (`job.job_id`) is used for RQ API endpoints: `/api/core/background-tasks/{uuid}/`
- Job PK (`job.pk`) is used for database endpoints and result loading
- RQ status values: `queued`, `started`, `finished`, `stopped`, `failed` (NOT `completed`)
- Database Job status values: `pending`, `scheduled`, `running`, `completed`, `failed`, `errored` (NO `cancelled` status exists)
- Check `rq_job.is_stopped` or `rq_job.is_failed` flags in Redis for cancellation detection, not database status
## Job Cancellation Flow
1. Call `/api/core/background-tasks/{uuid}/stop/` to stop RQ job
2. Call plugin's sync endpoint `/api/plugins/librenms_plugin/jobs/{pk}/sync-status/` to update database
3. Frontend polling detects status changes and redirects appropriately
## Polling Implementation
- Poll `/api/core/background-tasks/{uuid}/` for real-time RQ status
- Update modal messages based on status: "Job queued...", "Processing...", "Job completed!"
- Handle all RQ status values explicitly to avoid infinite polling
- Use `cancelInProgress` flag to prevent polling interference during cancellation
## Superuser Requirement for Background Jobs
- NetBox's `/api/core/background-tasks/` endpoint requires **superuser** (`IsSuperuser` in `BaseRQViewSet`).
- Non-superuser users cannot poll job status; they get 403 Forbidden.
- The plugin automatically falls back to synchronous mode for non-superusers—see `should_use_background_job()` in `list.py` and `actions.py`.
- This is a NetBox core design decision, not a plugin limitation. No amount of permissions (including `core.view_job`) bypasses it.
## Import Jobs
- **`FilterDevicesJob`** — background device filtering with VC detection. `job.data` keys: `device_ids`, `total_processed`, `filters`, `server_key`, `vc_detection_enabled`, `cache_timeout`, `cached_at`, `completed`. Devices are cached individually via shared cache keys from `get_validated_device_cache_key()`.
- **`ImportDevicesJob`** — background device/VM import. Calls `bulk_import_devices_shared()` for devices and `bulk_import_vms()` for VMs. `job.data` keys: `imported_device_pks`, `imported_vm_pks`, `imported_libre_device_ids`, `imported_libre_vm_ids`, `server_key`, `total`, `success_count`, `failed_count`, `skipped_count`, `virtual_chassis_created`, `errors`, `completed`.
## Shared Cache Key Pattern
- Both synchronous and background modes use `get_validated_device_cache_key()` from `import_utils.py` to generate cache keys. This ensures `_load_job_results()` in the list view can retrieve devices regardless of which mode produced them.
- `get_active_cached_searches()` manages multi-search cache to let users run and switch between searches.
- Never hardcode cache key formats; always use the helper functions.
## Permission Checks in Jobs
- Background jobs run outside view context, so they cannot use view mixins.
- Use standalone helpers from `import_utils.py` for permission checks inside job code:
- `check_user_permissions(user, permissions)``(bool, missing_list)`
- `require_permissions(user, permissions, action_description)` — raises `PermissionDenied`.
## Custom Sync Endpoint
`api/views.py::sync_job_status()` syncs database Job status with RQ job status, needed because NetBox worker doesn't always update DB when jobs stop before processing starts.
## Import Page Flow
The import page (`LibreNMSImportView` in `views/imports/list.py`) supports two modes:
1. **Synchronous** — calls `process_device_filters()` directly, renders results inline.
2. **Background** — enqueues `FilterDevicesJob`, returns `JsonResponse` with `job_id`/`job_pk`/`poll_url`. Frontend polls and redirects to `?job_id={pk}` on completion.
Result loading: `_load_job_results(job_id)` reads `job.data["device_ids"]`, reconstructs devices from per-device cache using `get_validated_device_cache_key()`.
Filter fields: `librenms_location`, `librenms_type`, `librenms_os`, `librenms_hostname`, `librenms_sysname`, `librenms_hardware`, `enable_vc_detection`, `show_disabled`, `exclude_existing`.
## Import Action Views (`views/imports/actions.py`)
- **`DeviceImportHelperMixin`** — provides `get_validated_device_with_selections()` and `render_device_row()` for HTMX row rendering. Shared by update views.
- **`BulkImportConfirmView`** (POST) — renders confirmation modal with selected device list. Returns `htmx/bulk_import_confirm.html`.
- **`BulkImportDevicesView`** (POST) — executes import. Background mode enqueues `ImportDevicesJob`; sync mode calls `bulk_import_devices()` + `bulk_import_vms()` and returns OOB row swaps with `HX-Trigger: closeModal`.
- **`DeviceValidationDetailsView`** (GET) — renders expandable validation details via `htmx/device_validation_details.html`.
- **`DeviceVCDetailsView`** (GET) — renders VC member details via `htmx/device_vc_details.html`.
- **`DeviceRoleUpdateView`**, **`DeviceClusterUpdateView`**, **`DeviceRackUpdateView`** (POST) — per-device dropdown updates. Apply selection to validation state and return re-rendered row via `render_device_row()`.
## Key Import Utilities (`import_utils.py`)
- `process_device_filters(filters, ...)` — fetches and validates devices from LibreNMS, returns list.
- `validate_device_for_import(device, ...)` — core validation function, produces validation state dict.
- `bulk_import_devices_shared(devices, user, ...)` — shared implementation between sync and background import.
- `bulk_import_vms(vm_imports, user, ...)` — VM import implementation.
- `fetch_device_with_cache(device_id, ...)` — retrieves/caches individual device data.
- Cache key functions: `get_validated_device_cache_key()`, `get_cache_metadata_key()`, `get_active_cached_searches()`, `get_import_device_cache_key()`.
## Validation Helpers (`import_validation_helpers.py`)
Centralizes validation state mutation used by the role/cluster/rack update views:
- `apply_role_to_validation()`, `apply_cluster_to_validation()`, `apply_rack_to_validation()` — update validation state when user selects a role/cluster/rack.
- `remove_validation_issue()`, `recalculate_validation_status()` — maintain issue list and overall status.
- `fetch_model_by_id()`, `extract_device_selections()` — helpers for reading form data.

View File

@@ -0,0 +1,66 @@
---
applyTo: "netbox_librenms_plugin/templates/**,netbox_librenms_plugin/static/**"
description: Frontend patterns for templates, HTMX, and static assets
---
# Frontend Patterns
## HTMX Conventions
- HTMX 2.x is the primary async layer. Table row updates return `<tr hx-swap-oob="true">`.
- Avoid `outerHTML` swaps; use OOB or targeted `innerHTML` swaps to keep table layout intact.
- All HTMX requests and `fetch()` calls must include a CSRF token. The standard pattern is `document.querySelector('[name=csrfmiddlewaretoken]').value` (from a hidden form input). The import JS also uses `getCookie('csrftoken')` as a fallback — prefer the hidden input approach for consistency.
## Modal Implementation
- Modals use Tabler (Bootstrap-like) but **without** `bootstrap.Modal` helpers.
- Buttons target the `htmx-modal-content` element and JavaScript in `librenms_import.html` toggles the wrapper.
- Do not reintroduce `data-bs-toggle` or duplicate modal IDs.
- The import page uses `ModalManager` class and `filterModalManager` instance—always use this reference in fetch callbacks, not undefined `modalInstance` variables.
## JavaScript Fetch Patterns
- Always check `response.ok` before processing fetch responses to catch HTTP errors.
- In catch blocks, show `error.message` for debugging rather than generic messages.
- The import filter form uses fetch with `Accept: application/json, text/html`—JSON for background jobs, HTML for synchronous mode.
## Form Controls
- Device import dropdowns rely on TomSelect decorators set up elsewhere.
- Keep `<select class="device-role-select">` markup stable to preserve JS hook-up.
## Styling
- Styling assumes Tabler defaults.
- Removing `table-responsive` wrappers was deliberate to prevent dropdown clipping—do not re-add them.
## Template Structure
- Templates live in `templates/netbox_librenms_plugin/`; reuse/includes under `inc/`.
- Sync pages extend `librenms_sync_base.html`.
- Tables emit HTMX-enabled columns and buttons (`tables/*.py`), so prefer updating the table renderer in Python rather than templates when changing row actions.
## Sync Tab Template Pattern
- Each sync resource has two templates following a naming convention:
- `_<resource>_sync.html` — the tab wrapper, loaded once when the tab is selected.
- `_<resource>_sync_content.html` — the HTMX-swappable inner fragment, refreshed on data changes without a full page reload.
- Current resources: `_interface_sync`, `_cable_sync`, `_ipaddress_sync`, `_vlan_sync`.
- When adding a new sync resource, create both the wrapper and content templates following this pattern.
## HTMX Fragments
- HTMX fragments live in `templates/netbox_librenms_plugin/htmx/` and include:
- `device_import_row.html` — individual import row updates.
- `device_validation_details.html` — expandable validation details.
- `device_vc_details.html` — virtual chassis member details.
- `bulk_import_confirm.html` — import confirmation modal content.
- Keep server responses and HTMX targets in sync when modifying these fragments.
## Settings Page
- `settings.html` uses a split-form pattern: two separate Django forms (`ServerConfigForm` + `ImportSettingsForm`) sharing one page, differentiated by a hidden `form_type` field (`"server_config"` or `"import_settings"`).
- The test-connection button is an HTMX POST to `TestLibreNMSConnectionView`, returning an inline alert fragment.
## Paginator
- `inc/paginator.html` is a custom paginator that preserves tab state and `interface_name_field` in pagination URLs. Used across all sync tables.
## Import Page JavaScript (`librenms_import.js`)
- Wrapped in an IIFE with `window.LibreNMSImportInitialized` guard to prevent re-initialization during HTMX swaps.
- **`ModalManager`** class wraps Bootstrap 5 modal show/hide with fallback.
- **`pollJobStatus()`** — polls `/api/core/background-tasks/{jobId}/` every 2s, updates progress messages, handles cancel button, redirects on completion.
- **`captureSelectionState()` / `restoreSelectionState()`** — preserves checkbox state across HTMX content swaps.
- **`createCacheCountdown()`** — generic countdown timer for cache expiration display.
- **`initializeFilterForm()`** — intercepts form submit, detects JSON response (background job), starts polling.
- CSRF token extracted via `getCookie('csrftoken')` (cookie-based).

View File

@@ -0,0 +1,72 @@
---
applyTo: "**/views/base/**,**/views/object_sync/**,**/views/sync/**,**/tables/**,**/librenms_sync.js"
description: Sync page architecture, base views, and sync action patterns
---
# Sync Pages
## Three-Layer View Architecture
All four sync resources (interfaces, cables, IP addresses, VLANs) follow the same pattern:
1. **Base views** (`views/base/`) — abstract classes that define the data pipeline:
- `BaseLibreNMSSyncView` — tabbed sync page, orchestrates all tabs via abstract `get_*_context()` methods.
- `BaseInterfaceTableView` — fetch ports → enrich with VLANs → cache → compare with NetBox interfaces → render table.
- `BaseCableTableView` — fetch links → match remote devices → check cable status → render table.
- `BaseIPAddressTableView` — fetch IPs → resolve interfaces → detect existing/update/new → render table.
- `BaseVLANTableView` — fetch VLANs → compare with NetBox VLANs → auto-select groups → render table.
2. **Object sync views** (`views/object_sync/`) — wire base views to NetBox models:
- Use `@register_model_view(Device, name="librenms_sync", path="librenms-sync")` to inject as a tab on Device/VM detail pages.
- Each `get_*_context()` method creates an instance of the concrete table view, copies `request`, and calls `get_context_data()`.
- VMs skip cables and VLANs (return `None`).
3. **Sync action views** (`views/sync/`) — POST-only views that create/update/delete NetBox objects:
- Follow a consistent pattern: check permissions → read selected rows from POST → load cached data → apply changes in `transaction.atomic()` → redirect to sync tab.
## Data Pipeline (Base Views)
Every base table view follows: **fetch → cache → compare → render**.
- **Fetch:** Call LibreNMS API (e.g., `get_ports()`, `get_device_ips()`, `get_device_vlans()`).
- **Cache:** Store results via `CacheMixin` keys: `librenms_{data_type}_{model_name}_{pk}`. Also store fetch timestamp at `librenms_{data_type}_last_fetched_{model_name}_{pk}`.
- **Compare:** Match LibreNMS data against NetBox objects. Each resource implements its own comparison (interface matching by name, IP matching by address/mask, VLAN matching by VID+group).
- **Render:** Build a django-tables2 table, return a partial template (`_*_sync_content.html`).
## Sync Action View Pattern
```python
class SyncSomeResourceView(LibreNMSPermissionMixin, NetBoxObjectPermissionMixin, CacheMixin, View):
required_object_permissions = {"POST": [("add", Model), ("change", Model)]}
def post(self, request, object_type, object_id):
if error := self.require_all_permissions("POST"):
return error
# 1. Resolve object (Device or VM)
# 2. Read selected items from request.POST.getlist("select")
# 3. Load cached data from cache.get(self.get_cache_key(obj, "..."))
# 4. Apply changes inside transaction.atomic()
# 5. Redirect to sync tab with ?tab=<resource>
```
## Table Conventions (`tables/*.py`)
- Tables define HTMX-enabled columns and checkboxes. Selection uses `ToggleColumn(attrs={"input": {"name": "select"}})`.
- Constructor takes contextual params (e.g., `device`, `interface_name_field`, `vlan_groups`) to customize rendering.
- Tables set `self.tab` and `self.prefix` for multi-table pagination via `get_table_paginate_count()`.
- Row attrs include `data-*` attributes for JavaScript filtering and identification.
- VLAN columns use `render_vlans()` with hidden inputs for per-row group selection and JSON data for modals.
## Key Mixins Used by Sync Views
- **`LibreNMSAPIMixin`** — lazy-creates `LibreNMSAPI` instance via `self.librenms_api` property. Also provides `get_server_info()` for template context.
- **`CacheMixin`** — generates consistent cache keys via `get_cache_key(obj, data_type)` and `get_last_fetched_key(obj, data_type)`. Also provides `get_vlan_overrides_key(obj)` for VLAN group override persistence.
- **`VlanAssignmentMixin`** — VLAN group scope resolution: Rack → Location → Site → SiteGroup → Region → Global. Used by interface and VLAN sync for auto-selecting the most-specific VLAN group and building lookup maps.
## JavaScript (`librenms_sync.js`)
- Not wrapped in an IIFE — functions are global. Master initializer `initializeScripts()` runs on `DOMContentLoaded` and `htmx:afterSwap`.
- **Key function groups:**
- Checkbox management: `initializeTableCheckboxes()`, `updateBulkActionButton()`.
- TomSelect dropdowns: `initializeVCMemberSelect()`, `initializeVRFSelects()`, `initializeVlanGroupSelects()`, `initializeVlanSyncGroupSelects()`. Uses `TOMSELECT_INIT_DELAY_MS = 100` for delayed initialization after HTMX swaps.
- Verification: `handleInterfaceChange()`, `handleCableChange()`, `handleVRFChange()` — POST to single-item verify endpoints.
- VLAN modals: `openVlanDetailModal()`, `verifyVlanInGroup()`, `verifyVlanSyncGroup()` — per-interface VLAN detail editing.
- Bulk operations: `initializeBulkEditApply()`, `deleteSelectedInterfaces()`.
- Table filtering: `initializeTableFilters()`, `filterTable()` — client-side row filtering.
- URL/tab state: `initializeTabs()`, `getDeviceIdFromUrl()`, `setInterfaceNameFieldFromURL()`.
- Cache countdowns: `initializeCountdown()`, `initializeCountdowns()`.
- CSRF token extracted via `document.querySelector('[name=csrfmiddlewaretoken]').value`.

View File

@@ -0,0 +1,50 @@
---
applyTo: "tests/**"
description: Testing patterns and conventions for the NetBox LibreNMS plugin
---
# Testing Patterns
## General Test Conventions
- Use plain **pytest classes**, not Django `TestCase`. Avoid `from django.test import TestCase`.
- **Never use `@pytest.mark.django_db`** for unit tests—mock all database interactions with `MagicMock`.
- Use **inline imports** inside test methods to avoid Django initialization at module load time.
- Mock NetBox models (Device, Job, User) with `MagicMock()` instead of creating real instances.
- Use `assert x == y` syntax, not `self.assertEqual(x, y)` (no TestCase inheritance).
- See [docs/development/testing.md](../../docs/development/testing.md) for test file structure and running instructions.
## Background Job Tests
- Instantiate `JobRunner` subclasses using `object.__new__(JobClass)` to bypass `__init__`, then set `job.job = MagicMock()` and `job.logger = MagicMock()`. See `create_mock_job_runner()` helper in `tests/test_background_jobs.py`.
- Patch deferred/inline imports at their **source** module (e.g., `netbox_librenms_plugin.import_utils.process_device_filters`), not the consuming module.
- Patch `cache` where imported: `netbox_librenms_plugin.views.imports.list.cache`, not `django.core.cache.cache`.
- Test view decision logic by setting `view._filter_form_data = {...}` directly, not via HTTP requests.
- **Never use `RequestFactory`**—mock request objects directly or test method logic in isolation.
- Cache key tests must patch `get_validated_device_cache_key` from `import_utils.py`; never hardcode key formats like `job_123_device_1`.
## Test File Naming
- Follow the `test_{module_name}.py` convention for new test files.
- `test_netbox_librenms_plugin.py` is an empty placeholder — do not add tests there.
## Test Coverage by Module
- `librenms_api.py``test_librenms_api.py`, `test_librenms_api_helpers.py`
- `import_utils.py`, `import_validation_helpers.py`, `utils.py``test_import_utils.py`, `test_import_validation_helpers.py`, `test_utils.py`
- `jobs.py`, `views/imports/list.py``test_background_jobs.py`
- Permission mixins, API permissions, constants → `test_permissions.py`
- VLAN API, mode detection, comparison, sync → `test_vlan_sync.py`
- `VlanAssignmentMixin`, VLAN enrichment → `test_interface_vlan_sync.py`
- Views (`views/sync/`, `views/object_sync/`, `views/imports/actions.py`) — no dedicated test files yet. Test business logic via the utility modules they call, not via HTTP requests.
## Permission Test Patterns
When testing permissions (see `test_permissions.py` for reference):
- Create a mock view instance with `object.__new__(ViewClass)`, set `request = MagicMock()` with `request.user.has_perm.side_effect = lambda p: p in allowed_perms`.
- For `NetBoxObjectPermissionMixin` tests, set `required_object_permissions` on the instance before calling `check_object_permissions()`.
- Test both the individual methods (`has_write_permission()`, `check_object_permissions()`) and the combined `require_all_permissions()` flow.
- For JSON variants (`require_all_permissions_json`), assert `isinstance(response, JsonResponse)` and check `response.status_code == 403`.
## Shared Fixtures (`conftest.py`)
Reuse fixtures from `tests/conftest.py` instead of creating ad-hoc mocks:
- **Configuration**: `mock_multi_server_config`, `mock_legacy_config`
- **API client**: `mock_librenms_api`
- **NetBox objects**: `mock_netbox_device`, `mock_netbox_vm`, `mock_netbox_site`, `mock_netbox_platform`, `mock_netbox_device_type`, `mock_netbox_device_role`, `mock_netbox_cluster`, `mock_netbox_rack`
- **HTTP responses**: `mock_response_factory`, `mock_success_response`, `mock_device_response`, `mock_error_response`, `mock_auth_error_response`
- **Import workflow**: `sample_librenms_device`, `sample_librenms_device_minimal`, `sample_validation_state`, `sample_validation_state_vm`

49
.github/pull_request_template.md vendored Normal file
View File

@@ -0,0 +1,49 @@
## Summary
Briefly describe what this PR does in plain English, and provide as much of the following information as possible.
## Motivation / Problem
What issue does this solve?
- Bug
- Feature
- Refactor
- Maintenance / cleanup
Link any related issues if applicable.
## Scope of Change
Delete items that dont apply:
- Sync/Import logic
- NetBox models / ORM
- LibreNMS API interaction
- Config / settings
- Web UI / templates
- Database migrations
- Tests
- Docs only
- Other: <describe>
## How Was This Tested?
Delete items that dont apply and describe briefly.
- Unit tests: <yes/no + what>
- Manual testing: <yes/no + what>
- Not tested: <explain why>
### Manual Test Steps (if applicable)
1.
2.
3.
## Risk Assessment
- Does this change affect existing users?
- Could this cause unintended imports / updates?
Explain briefly.
## Backwards Compatibility
- No breaking changes
- Breaking change (explain and document)
## Other Notes
Anything the maintainer(s) should pay particular attention to?

103
.github/workflows/codeql.yml vendored Normal file
View File

@@ -0,0 +1,103 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL Advanced"
on:
push:
branches: [ "master", "develop" ]
pull_request:
branches: [ "master", "develop" ]
schedule:
- cron: '35 13 * * 0'
jobs:
analyze:
name: Analyze (${{ matrix.language }})
# Runner size impacts CodeQL analysis time. To learn more, please see:
# - https://gh.io/recommended-hardware-resources-for-running-codeql
# - https://gh.io/supported-runners-and-hardware-resources
# - https://gh.io/using-larger-runners (GitHub.com only)
# Consider using larger runners or machines with greater resources for possible analysis time improvements.
runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
permissions:
# required for all workflows
security-events: write
# required to fetch internal or private CodeQL packs
packages: read
# only required for workflows in private repositories
actions: read
contents: read
strategy:
fail-fast: false
matrix:
include:
- language: actions
build-mode: none
- language: javascript-typescript
build-mode: none
- language: python
build-mode: none
# CodeQL supports the following values keywords for 'language': 'actions', 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'rust', 'swift'
# Use `c-cpp` to analyze code written in C, C++ or both
# Use 'java-kotlin' to analyze code written in Java, Kotlin or both
# Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
# To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
# see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
# If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
steps:
- name: Checkout repository
uses: actions/checkout@v6
# Add any setup steps before running the `github/codeql-action/init` action.
# This includes steps like installing compilers or runtimes (`actions/setup-node`
# or others). This is typically only required for manual builds.
# - name: Setup runtime (example)
# uses: actions/setup-example@v1
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
# queries: security-extended,security-and-quality
# If the analyze step fails for one of the languages you are analyzing with
# "We were unable to automatically build your code", modify the matrix above
# to set the build mode to "manual" for that language. Then modify this step
# to build your code.
# Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
- name: Run manual build steps
if: matrix.build-mode == 'manual'
shell: bash
run: |
echo 'If you are using a "manual" build mode for one or more of the' \
'languages you are analyzing, replace this with the commands to build' \
'your code, for example:'
echo ' make bootstrap'
echo ' make release'
exit 1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
with:
category: "/language:${{matrix.language}}"

50
.github/workflows/lint-format.yaml vendored Normal file
View File

@@ -0,0 +1,50 @@
name: Lint and Format
on:
push:
branches:
- master
- develop
pull_request:
branches:
- master
- develop
jobs:
format-and-lint:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: '3.9'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install ruff
- name: Run Ruff linting
run: |
echo "::group::Ruff Linting"
ruff check . --output-format=github
echo "::endgroup::"
- name: Run Ruff formatting check
run: |
echo "::group::Ruff Formatting"
ruff format --check .
echo "::endgroup::"
- name: Report formatting issues
if: failure()
run: |
echo "::error::Formatting or linting issues detected!"
echo "To fix locally, run:"
echo " ruff check --fix ."
echo " ruff format ."
echo "Then commit and push the changes."

18
.github/workflows/mkdocs.yaml vendored Normal file
View File

@@ -0,0 +1,18 @@
name: ci
on:
push:
branches:
- master
- main
permissions:
contents: write
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: 3.x
- run: pip install mkdocs-material mkdocs-autorefs mkdocs-material-extensions mkdocstrings mkdocstrings-python-legacy mkdocs-include-markdown-plugin
- run: mkdocs gh-deploy --force

58
.github/workflows/publish-pypi.yaml vendored Normal file
View File

@@ -0,0 +1,58 @@
# see: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/
name: Publish Python 🐍 distribution 📦 to PyPI
on:
push:
branches:
- master
tags:
- '*'
jobs:
build:
name: Build distribution 📦
runs-on: ubuntu-latest
steps:
- name: Clean previous builds
run: rm -rf dist/
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: "3.x"
- name: Install pypa/build
run: >-
python3 -m
pip install
build
--user
- name: Build a binary wheel and a source tarball
run: python3 -m build
- name: Store the distribution packages
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: python-package-distributions
path: dist/
publish-to-pypi:
name: >-
Publish Python 🐍 distribution 📦 to PyPI
if: startsWith(github.ref, 'refs/tags/') # only publish to PyPI on tag pushes
needs:
- build
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/netbox-librenms-plugin
permissions:
id-token: write # IMPORTANT: mandatory for trusted publishing
steps:
- name: Download all the dists
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: python-package-distributions
path: dist/
- name: Publish distribution 📦 to PyPI
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1

88
.github/workflows/test.yaml vendored Normal file
View File

@@ -0,0 +1,88 @@
name: Test with all supported NetBox versions
on:
push:
branches:
- master
- develop
pull_request:
branches:
- master
- develop
jobs:
test-netbox:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.12", "3.13", "3.14"]
services:
redis:
image: redis
ports:
- 6379:6379
postgres:
image: postgres
env:
POSTGRES_USER: netbox
POSTGRES_PASSWORD: netbox
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
path: netbox-librenms-plugin
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: ${{ matrix.python-version }}
- name: Checkout NetBox
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: "netbox-community/netbox"
path: netbox
ref: main
- name: Install NetBox LibreNMS Plugin
working-directory: netbox-librenms-plugin
run: |
pip install -e .
pip install pytest pytest-django pytest-cov
- name: Set up configuration
working-directory: netbox
run: |
ln -s "$(pwd)/../netbox-librenms-plugin/media/configuration.testing.py" netbox/netbox/configuration.py
python -m pip install --upgrade pip
python -m pip install tblib
pip install -r requirements.txt -U
- name: Run tests
working-directory: netbox/netbox
env:
NETBOX_CONFIGURATION: netbox.configuration
run: |
python -m pytest ../../netbox-librenms-plugin/netbox_librenms_plugin/tests/ -v \
--cov=netbox_librenms_plugin \
--cov-report=html:../../netbox-librenms-plugin/coverage_html \
--cov-report=term-missing
- name: Upload coverage report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: matrix.python-version == '3.12'
with:
name: coverage-report
path: netbox-librenms-plugin/coverage_html/

292
.gitignore vendored Normal file
View File

@@ -0,0 +1,292 @@
# https://github.com/github/gitignore/blob/main/Global/macOS.gitignore
# General
.DS_Store
.AppleDouble
.LSOverride
# Icon must end with two \r
Icon
# Thumbnails
._*
# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent
# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
# https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
!.vscode/*.code-snippets
# Local History for Visual Studio Code
.history/
# Built Visual Studio Code Extensions
*.vsix
# https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
# AWS User-specific
.idea/**/aws.xml
# Generated files
.idea/**/contentModel.xml
# Sensitive or high-churn files
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
# Gradle
.idea/**/gradle.xml
.idea/**/libraries
# Gradle and Maven with auto-import
# When using Gradle or Maven with auto-import, you should exclude module files,
# since they will be recreated, and may cause churn. Uncomment if using
# auto-import.
# .idea/artifacts
# .idea/compiler.xml
# .idea/jarRepositories.xml
# .idea/modules.xml
# .idea/*.iml
# .idea/modules
# *.iml
# *.ipr
# CMake
cmake-build-*/
# Mongo Explorer plugin
.idea/**/mongoSettings.xml
# File-based project format
*.iws
# IntelliJ
out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Cursive Clojure plugin
.idea/replstate.xml
# SonarLint plugin
.idea/sonarlint/
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
# Editor-based Rest Client
.idea/httpRequests
# Android studio 3.1+ serialized cache file
.idea/caches/build_file_checksums.ser
# https://github.com/github/gitignore/blob/main/Python.gitignore
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
coverage_html/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
.vscode/settings.json
# Dev Container files
.devcontainer/.env
.devcontainer/extra-requirements.txt
.devcontainer/config/plugin-config.py
.devcontainer/config/extra-configuration.py
.devcontainer/config/extra-plugins.py
# Proxy CA certificates (keep local)
ca-bundle.crt
*.pem
.github/hooks/

19
.pre-commit-config.yaml Normal file
View File

@@ -0,0 +1,19 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.4 # Use the latest version from https://github.com/astral-sh/ruff-pre-commit/releases
hooks:
# Run the linter
- id: ruff-check
args: [--fix]
# Run the formatter
- id: ruff-format
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
exclude: ^mkdocs\.yml$
- id: check-added-large-files
- id: check-merge-conflict

0
.pypirc Normal file
View File

201
LICENSE Normal file
View File

@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

12
MANIFEST.in Normal file
View File

@@ -0,0 +1,12 @@
include CONTRIBUTING.md
include LICENSE
include README.md
recursive-include tests *
recursive-exclude * __pycache__
recursive-exclude * *.py[co]
recursive-include docs *.rst conf.py Makefile make.bat *.jpg *.png *.gif
recursive-include netbox_librenms_plugin/static *
graft templates

22
Makefile Normal file
View File

@@ -0,0 +1,22 @@
sources = netbox_librenms_plugin
.PHONY: test format lint unittest pre-commit clean
test: format lint unittest
format:
ruff format $(sources)
ruff check --select I --fix $(sources)
lint:
ruff check $(sources)
unittest:
pytest netbox_librenms_plugin/tests/ -v
pre-commit:
pre-commit run --all-files
clean:
rm -rf *.egg-info
rm -rf .tox dist site

259
README.md Normal file
View File

@@ -0,0 +1,259 @@
# NetBox LibreNMS Plugin
The NetBox LibreNMS Plugin enables integration between NetBox and LibreNMS, allowing you to leverage data from both systems. NetBox remains the Source of Truth (SoT) for you network, but
this plugin allows you to easily onboard device objects from existing data in LibreNMS. The plugin does not automatically create objects in NetBox to ensure only verified data is used to populate NetBox.
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/bonzo81/netbox-librenms-plugin)
## Features
The plugin offers the following key features:
### Device Import
Search and import devices from LibreNMS into NetBox with comprehensive validation and control:
- Filter devices by location, type, OS, hostname, or system name
- Validate import prerequisites (Site, Device Type, Device Role)
- Smart matching for Sites, Device Types, and Platforms
- Import as physical Devices or Virtual Machines
- Bulk import multiple devices
- Automatic Virtual Chassis creation for stackable switches
- Background job processing for large device sets
See the [Device Import Guide](docs/librenms_import/overview.md) for detailed usage instructions.
### Device Field Sync
Synchronize device information from LibreNMS to NetBox. The following device fields can be synchronized:
- Device Name (with naming preference support)
- Serial Number (including virtual chassis members)
- Device Type
- Platform
### Interface Sync
Pull interface data from Devices and Virtual Machines from LibreNMS into NetBox. The following interface attributes are synchronized:
- Name
- Description
- Status (Enabled/Disabled)
- Type (with [custom mapping support](docs/usage_tips/interface_mappings.md))
- Speed
- MTU
- MAC Address
- VLAN (Tagged and untagged)
> Set custom mappings for interface types to ensure that the correct interface type is used when syncing from LibreNMS to NetBox.
### Cable Sync
Create cable connection in NetBox from LibreNMS links data.
### IP Address Sync
Create IP address in NetBox from LibreNMS device IP data.
### VLAN Sync
- Create VLAN objects in NetBox from LibreNMS device VLAN data
- Per-VLAN group assignment with scope-aware auto-selection
### Add device to LibreNMS from Netbox
- Add device to LibreNMS from Netbox device page. SNMP v2c and v3 are supported.
### Site & Location Synchronization
The plugin also supports synchronizing NetBox Sites with LibreNMS locations:
- Compare NetBox sites to LibreNMS location data
- Create LibreNMS locations to match NetBox sites
- Update existing LibreNMS locations latitude and longitude values based on NetBox data ⚠️ *(currently not working due to LibreNMS API issue)*
- Sync device site to LibreNMS location
### Multi LibreNMS Server Configuration
- Configure multiple LibreNMS instances in your NetBox configuration
- Switch between different LibreNMS servers through the web interface
- Maintain backward compatibility with single-server configurations
## Screenshots/GIFs
>Screenshots from older plugin version
#### Site & Location Sync
![Site Location Sync](docs/img/Netbox-librenms-plugin-Sites.gif)
#### Sync devices and Interfaces
![Add device and interfaces](docs/img/Netbox-librenms-plugin-interfaceadd.gif)
#### Virtual Chassis Member Select
![Virtual Chassis Member Selection](docs/img/Netbox-librenms-plugin-virtualchassis.gif)
#### Interface Type Mappings
![Interfaces Type Mappings](docs/img/Netbox-librenms-plugin-mappings.png)
## Contributing
There's more to do! Coding is not my day job. Bugs will exist and improvements will be needed. Contributions are very welcome! I've got more ideas for new features and improvements but please [contribute](docs/contributing.md) if you can!
### Development Environment
An easy way to get started with development is to use the included **development container**:
- **Local Development**: Open the project in VS Code and choose "Reopen in Container"
- **GitHub Codespaces**: Click "Code" → "Create codespace" in the GitHub repository
- **Ready in 5 minutes**: A complete NetBox environment with PostgreSQL, Redis, and the plugin pre-installed
📖 **[See the Dev Container README](.devcontainer/README.md)** for detailed setup instructions, available commands, and troubleshooting.
Alternatively, share your ideas for the plugin over in [discussions](https://github.com/bonzo81/netbox-librenms-plugin/discussions).
## Compatibility
| NetBox Version | Plugin Version |
|----------------|----------------|
| 4.1 | 0.2.x - 0.3.5 |
| 4.2 - 4.5 | 0.3.6+ |
## Installing
### Standard Installation
Activate your virtual environment and install the plugin:
```bash
source /opt/netbox/venv/bin/activate
```
Install with pip:
```bash
(venv) $ pip install netbox-librenms-plugin
```
Add to your `local_requirements.txt` to ensure it is automatically reinstalled during future upgrades.
```bash
echo "netbox-librenms-plugin" >> /opt/netbox/local_requirements.txt #Check your NetBox install location
```
### Docker
For adding to a NetBox Docker setup see how to create a custom Docker image.
[the general instructions for using netbox-docker with plugins](https://github.com/netbox-community/netbox-docker/wiki/Using-Netbox-Plugins).
Add the plugin to `plugin_requirements.txt` (netbox-docker):
```bash
# plugin_requirements.txt
netbox-librenms-plugin
```
## Configuration
### 1. Enable the Plugin
Enable the plugin in `/opt/netbox/netbox/netbox/configuration.py`, or if you use netbox-docker, your `/configuration/plugins.py` file :
```python
PLUGINS = [
'netbox_librenms_plugin'
]
```
### 2. Apply the plugin configuration
Multi server example:
```python
PLUGINS_CONFIG = {
'netbox_librenms_plugin': {
'servers': {
'production': {
'display_name': 'Production LibreNMS',
'librenms_url': 'https://librenms-prod.example.com',
'api_token': 'your_production_token',
'cache_timeout': 300,
'verify_ssl': True,
'interface_name_field': 'ifDescr'
},
'testing': {
'display_name': 'Test LibreNMS',
'librenms_url': 'https://librenms-test.example.com',
'api_token': 'your_test_token',
'cache_timeout': 300,
'verify_ssl': False,
'interface_name_field': 'ifName'
},
'development': {
'display_name': 'Dev LibreNMS',
'librenms_url': 'https://librenms-dev.example.com',
'api_token': 'your_dev_token',
'cache_timeout': 180,
'verify_ssl': False,
'interface_name_field': 'ifDescr'
}
}
}
}
```
Or use the original single server confiig example:
```python
PLUGINS_CONFIG = {
'netbox_librenms_plugin': {
'librenms_url': 'https://your-librenms-instance.com',
'api_token': 'your_librenms_api_token',
'cache_timeout': 300,
'verify_ssl': True, # Optional: Change to False if needed,
'interface_name_field': 'ifDescr', # Optional: LibreNMS field used for interface name. ifName used as default
}
}
```
### 3. Apply Database Migrations
Apply database migrations with Netbox `manage.py`:
```
(venv) $ python manage.py migrate
```
### 4. Collect Static Files
The plugin includes static files that need to be collected by NetBox. Run the following command to collect static files:
```
(venv) $ python manage.py collectstatic --no-input
```
### 5. Restart Netbox
Restart the Netbox service to apply changes:
```
sudo systemctl restart netbox
```
### 6. Custom Field
As of version 0.4.4, the plugin **automatically creates** the `librenms_id` custom field (JSON type) when migrations are run. No manual setup is required.
The field is created for Device, Virtual Machine, Interface, and VM Interface objects and stores a per-server mapping (e.g., `{"production": 42}`).
For more info check out [custom field docs](docs/usage_tips/custom_field.md)
## Update
```
source /opt/netbox/venv/bin/activate
pip install -U netbox-librenms-plugin
python manage.py migrate
python manage.py collectstatic --no-input
systemctl restart netbox
```
## Uninstall
See [the instructions for uninstalling plugins](https://netboxlabs.com/docs/netbox/en/stable/plugins/removal/).
## Credits
Based on the NetBox plugin tutorial and docs:
- [demo repository](https://github.com/netbox-community/netbox-plugin-demo)
- [tutorial](https://github.com/netbox-community/netbox-plugin-tutorial)
- [docs](https://netboxlabs.com/docs/netbox/en/stable/plugins/development/)
This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter). Thanks to the [`netbox-community/cookiecutter-netbox-plugin`](https://github.com/netbox-community/cookiecutter-netbox-plugin) for the project template.

259
docs/README.md Normal file
View File

@@ -0,0 +1,259 @@
# NetBox LibreNMS Plugin
### Intro
The NetBox LibreNMS Plugin enables integration between NetBox and LibreNMS, allowing you to leverage data from both systems. NetBox remains the Source of Truth (SoT) for you network, but this plugin allows you to easily onboard device objects from existing data in LibreNMS. The plugin does not automatically create objects in NetBox to ensure only verified data is used to populate NetBox.
## Features
The plugin offers the following key features:
### Device Import
Search and import devices from LibreNMS into NetBox with comprehensive validation and control:
* Filter devices by location, type, OS, hostname, system name, or hardware model
* Validate import prerequisites (Site, Device Type, Device Role)
* Smart matching for Sites, Device Types, and Platforms
* Import as physical Devices or Virtual Machines
* Bulk import multiple devices
* Automatic Virtual Chassis creation for stackable switches
* Background job processing for large device sets
See the [Device Import Guide](librenms_import/overview.md) for detailed usage instructions.
### Device Field Sync
Synchronize device information from LibreNMS to NetBox. The following device fields can be synchronized:
* Device Name (with naming preference support)
* Serial Number (including virtual chassis members)
* Device Type
* Platform
### Interface Sync
Pull interface data from Devices and Virtual Machines from LibreNMS into NetBox. The following interface attributes are synchronized:
* Name
* Description
* Status (Enabled/Disabled)
* Type (with [custom mapping support](usage_tips/interface_mappings.md))
* Speed
* MTU
* MAC Address
* VLAN (Tagged and untagged)
> Set custom mappings for interface types to ensure that the correct interface type is used when syncing from LibreNMS to NetBox.
### Cable Sync
Create cable connection in NetBox from LibreNMS links data.
### IP Address Sync
Create IP address in NetBox from LibreNMS device IP data.
### VLAN Sync
- Create VLAN objects in NetBox from LibreNMS device VLAN data
- Per-VLAN group assignment with scope-aware auto-selection
### Add device to LibreNMS from Netbox
* Add device to LibreNMS from Netbox device page. SNMP v2c and v3 are supported.
### Site & Location Sync
The plugin also supports synchronizing NetBox Sites with LibreNMS locations:
* Compare NetBox sites to LibreNMS location data
* Create LibreNMS locations to match NetBox sites
* Update existing LibreNMS locations latitude and longitude values based on NetBox data ⚠️ *(currently not working due to LibreNMS API issue)*
* Sync device site to LibreNMS location
### Screenshots/GIFs
> Screenshots from older plugin version
#### Site & Location Sync
![Site Location Sync](img/Netbox-librenms-plugin-Sites.gif)
#### Sync devices and Interfaces
![Add device and interfaces](img/Netbox-librenms-plugin-interfaceadd.gif)
#### Virtual Chassis Member Select
![Virtual Chassis Member Selection](img/Netbox-librenms-plugin-virtualchassis.gif)
#### Interface Type Mappings
![Interfaces Type Mappings](img/Netbox-librenms-plugin-mappings.png)
## Contributing
There's more to do! Coding is not my day job so bugs will exist and improvements will be needed. Contributions are very welcome! I've got more ideas for new features and improvements but please [contribute](contributing.md) if you can!
Or just share your ideas for the plugin over in [discussions](https://github.com/bonzo81/netbox-librenms-plugin/discussions).
## Compatibility
| NetBox Version | Plugin Version |
| -------------- | -------------- |
| 4.1 | 0.2.x - 0.3.5 |
| 4.2 - 4.5 | 0.3.6+ |
## Installing
### Standard Installation
Activate your virtual environment and install the plugin:
```bash
source /opt/netbox/venv/bin/activate
```
Install with pip:
```bash
(venv) $ pip install netbox-librenms-plugin
```
Add to your `local_requirements.txt` to ensure it is automatically reinstalled during future upgrades.
```bash
echo "netbox-librenms-plugin" >> /opt/netbox/local_requirements.txt #Check your NetBox install location
```
### Docker
For adding to a NetBox Docker setup see how to create a custom Docker image. [the general instructions for using netbox-docker with plugins](https://github.com/netbox-community/netbox-docker/wiki/Using-Netbox-Plugins).
Add the plugin to `plugin_requirements.txt` (netbox-docker):
```bash
# plugin_requirements.txt
netbox-librenms-plugin
```
## Configuration
### 1. Enable the Plugin
Enable the plugin in `/opt/netbox/netbox/netbox/configuration.py`, or if you use netbox-docker, your `/configuration/plugins.py` file :
```python
PLUGINS = [
'netbox_librenms_plugin'
]
```
### 2. Apply the plugin configuration
Multi server example:
```python
PLUGINS_CONFIG = {
'netbox_librenms_plugin': {
'servers': {
'production': {
'display_name': 'Production LibreNMS',
'librenms_url': 'https://librenms-prod.example.com',
'api_token': 'your_production_token',
'cache_timeout': 300,
'verify_ssl': True,
'interface_name_field': 'ifDescr'
},
'testing': {
'display_name': 'Test LibreNMS',
'librenms_url': 'https://librenms-test.example.com',
'api_token': 'your_test_token',
'cache_timeout': 300,
'verify_ssl': False,
'interface_name_field': 'ifName'
},
'development': {
'display_name': 'Dev LibreNMS',
'librenms_url': 'https://librenms-dev.example.com',
'api_token': 'your_dev_token',
'cache_timeout': 180,
'verify_ssl': False,
'interface_name_field': 'ifDescr'
}
}
}
}
```
Or use the original single server confiig example:
```python
PLUGINS_CONFIG = {
'netbox_librenms_plugin': {
'librenms_url': 'https://your-librenms-instance.com',
'api_token': 'your_librenms_api_token',
'cache_timeout': 300,
'verify_ssl': True, # Optional: Change to False if needed,
'interface_name_field': 'ifDescr', # Optional: LibreNMS field used for interface name. ifName used as default
}
}
```
### 3. Apply Database Migrations
Apply database migrations with Netbox `manage.py`:
```
(venv) $ python manage.py migrate
```
### 4. Collect Static Files
The plugin includes static files that need to be collected by NetBox. Run the following command to collect static files:
```
(venv) $ python manage.py collectstatic --no-input
```
### 5. Restart Netbox
Restart the Netbox service to apply changes:
```
sudo systemctl restart netbox
```
### 6. Custom Field
As of version 0.4.4, the plugin **automatically creates** the `librenms_id` custom field (JSON type) when migrations are run. No manual setup is required.
The field is created for Device, Virtual Machine, Interface, and VM Interface objects and stores a per-server mapping (e.g., `{"production": 42}`).
For more info check out [custom field docs](usage_tips/custom_field.md)
## Update
```
source /opt/netbox/venv/bin/activate
pip install -U netbox-librenms-plugin
python manage.py migrate
python manage.py collectstatic --no-input
systemctl restart netbox
```
## Uninstall
See [the instructions for uninstalling plugins](https://netboxlabs.com/docs/netbox/en/stable/plugins/removal/).
## Credits
Based on the NetBox plugin tutorial and docs:
* [demo repository](https://github.com/netbox-community/netbox-plugin-demo)
* [tutorial](https://github.com/netbox-community/netbox-plugin-tutorial)
* [docs](https://netboxlabs.com/docs/netbox/en/stable/plugins/development/)
This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter). Thanks to the [`netbox-community/cookiecutter-netbox-plugin`](https://github.com/netbox-community/cookiecutter-netbox-plugin) for the project template.

28
docs/SUMMARY.md Normal file
View File

@@ -0,0 +1,28 @@
# Table of contents
* [NetBox LibreNMS Plugin](README.md)
* [Getting Started](usage_tips/README.md)
* [Feature Overview](feature_list.md)
* [Initial Setup](usage_tips/README.md)
* [Custom Field Setup](usage_tips/custom_field.md)
* [Multi-Server Setup](usage_tips/multi_server_configuration.md)
* [Permissions & Access](usage_tips/permissions.md)
* [Suggested Workflow](usage_tips/suggested_workflow.md)
* [Import Devices](librenms_import/overview.md)
* [Overview](librenms_import/overview.md)
* [Searching for Devices](librenms_import/search.md)
* [Validation & Configuration](librenms_import/validation.md)
* [Import Settings](librenms_import/import_settings.md)
* [Background Jobs & Caching](librenms_import/background_jobs.md)
* [Sync & Configuration](usage_tips/virtual_chassis.md)
* [Virtual Chassis](usage_tips/virtual_chassis.md)
* [Interface Mappings](usage_tips/interface_mappings.md)
* [Development](development/README.md)
* [Overview](development/README.md)
* [Project Structure](development/structure.md)
* [Views & Inheritance](development/views.md)
* [Mixins](development/mixins.md)
* [Templates](development/templates.md)
* [Testing](development/testing.md)
* [Changelog](changelog.md)
* [Contributing](contributing.md)

428
docs/changelog.md Normal file
View File

@@ -0,0 +1,428 @@
# Changelog
## 0.4.6 (2026-04-20)
### Fixes
* Fix VC master device showing 0 module bays after import by preventing stale counter overwrites during virtual chassis creation (#275)
* Defer VC master assignment until after all members are attached
* Add `_sync_module_bay_counter()` safety net to reconcile counter after master assignment
## 0.4.5 (2026-04-16)
### Fixes
* Fix VLAN group modal not closing on dismiss button, cancel button, save button, or backdrop click on the interface sync page
* Align `showModal()`/`hideModal()` with `ModalManager` pattern — try Bootstrap 5 native first, fall back to manual DOM manipulation
* Prevent stacking backdrop click handlers on repeated `showModal()` calls
## 0.4.4 (2026-04-15)
### New Features
* **Multi-Server Support**: JSON-based `librenms_id` custom field with per-server device tracking, server management, and migration from legacy integer format
* **Auto-Create Custom Field**: Automatically create the `librenms_id` custom field via `post_migrate` signal
* **Sync Page Naming Preferences**: Apply `use_sysname` and `strip_domain` naming preferences to device name matching on sync pages
* **VC Import Enhancements**: Show virtual chassis members in import confirm modal, enforce stack VC permissions before import, and fix VC flag propagation across the import flow
### Improvements
* Refactor `import_utils.py` into a package with aligned module boundaries
* Security hardening — URL-encode `server_key` in redirects, reject `librenms_id` <= 0, CSRF safety, XSS label escaping in cable verify
* JS hardening — extract `showModal`/`hideModal` helpers, harden `getDeviceIdFromUrl`, guard `interfaceNameField`, remove dead code
* Server-key propagation, UI polish, and view hardening across multi-server import pipeline
* Cable sync matching by `local_port_id` instead of display name; strip stale fields on cable verify POST
* Simplify migration — always revert to integer, handle non-integer custom field types gracefully
* Skip `device_type` check for VMs; add `sync_platform` to VM supported actions
* Update supported NetBox versions in README
### Fixes
* Fix VC master detection, 0-based position correction, and import propagation
* Fix VC verify view crash by using `get_librenms_sync_device()` in verify views
* Fix cables view caching, `interface_name_field` per-request, and `LibreNMSAPIMixin`
* Fix operator precedence in message fallback expressions
* Fix import validation, template bugs, and pagination
* Correct `vc_detection_enabled` default in `validate_device_for_import` call
* Fix stale docstrings, test names, and duplicate functions
* Guard `dt_match` None on ambiguous hardware match in `validate_device_for_import`
* Write enriched links data back to cache on non-fresh path
* Restore multi-key naming preferences and early-exit on missing name
### Development
* Expanded test coverage with new test files and reorganized existing tests
* Smoke tests, integration tests, and mock LibreNMS server
* Add `pytest-cov` coverage report (no gating)
* Updated pre-commit hooks, ruff config, and uv Dependabot
* Bump GitHub Actions dependencies (upload-artifact, download-artifact, pypi-publish)
* Update pull request template for contributions
### Documentation
* Update documentation for 0.4.4 release
* Testing docs updates with new test files and philosophy
## 0.4.3 (2026-03-03)
### New Features
* **VLAN Sync**: Full VLAN synchronization with per-row verification, per-VLAN group selection, color-coded summary UI, edit modal, and form spinners
* **Two-Tier Permission System**: Plugin-level and object-level access control with `LibreNMSPermissionMixin` and `NetBoxObjectPermissionMixin`
* **Serial Number Matching & Conflict Resolution**: Import detects serial number matches and guides users through device conflicts
* **Per-User Preferences**: Persistent user preferences for import toggles and interface name field
* **Device Identity Mismatch Modal**: Informational mismatch display on sync pages when device name differs between NetBox and LibreNMS
* **Empty State Cards**: Friendly empty state cards on sync tabs when no data is available
* **Naming Preferences in Validation**: Honour `use_sysname` and `strip_domain` settings in the import validation path with badge indicators
* **SNMPv1 Support**: Add SNMPv1 option to the LibreNMS add device form
### Improvements
* Security hardening — XSS escaping, input validation, open-redirect prevention, and VC member validation
* Block import when validation issues are present; fix modal titles and hostname conflict display
* Code deduplication and minor bug fixes across views and utilities
* Add docstrings to models, tables, views, forms, and API modules
* Pin all GitHub Actions to commit SHAs and add Dependabot configuration
* Add pull request template for contributions
### Fixes
* Fix KeyError for missing non-default server keys in `LibreNMSAPI`
* Fix device redirect URL and inaccurate docstrings
* Correct device role refresh to preserve schema keys
* Fix validation readiness, VM form guards, and sync_info accuracy
* Use flash message + redirect for import permission denied
* Add virtual chassis permission check and validate `object_type`
### Development
* Comprehensive test suite with CI workflow (GitHub Actions, pytest, ruff lint/format)
* Pre-commit hooks and updated format/lint dependencies
* DevContainer refactor with script consolidation, proxy/MITM support, and diagnostic tooling
* Modular copilot instructions split into per-concern `.instructions.md` files
### Documentation
* VLAN sync feature and view architecture documentation
* Permission system documentation
* Improved README, workflow guides, and troubleshooting notes
## 0.4.2 (2026-01-16)
### Fixes
* Fix device sync not working after update (Issue #199)
* Fix API endpoints for hostname lookup and port retrieval that were incorrectly changed
* Fix NameError in get_device_id_by_ip method - correct undefined 'mac_address' variable error (Issue #197)
## 0.4.1 (2026-01-12)
### Fixes
* Fix librenms_id custom field to use integer values in import process
## 0.4.0 (2026-01-09)
## Major Features
- **LibreNMS Bulk Device Import**: Complete workflow for importing devices from LibreNMS with HTMX-based UI, validation, and background job processing
- **Background Job Support**: Asynchronous processing for large device imports with real-time status tracking and cancellation
- **Enhanced Caching**: Improved caching system with expiration countdown, metadata storage, and multi-server isolation
- **Virtual Chassis Detection**: Automatic detection and handling of virtual chassis during import
- Add hardware and exclude-existing filters to device import
- Client-side table sorting for import page
- Active LibreNMS server display on import page
- Improved logging patterns and documentation
- Cache management with discard controls
- Refactored settings page with HTMX (removed fetch-based JavaScript)
- Enhanced virtual chassis sync device detection logic
- Better error handling and connection exhaustion prevention
## Development
- Improved dev container with RQ worker management
- Comprehensive test coverage for background jobs
- Enhanced copilot instructions for AI-assisted development
- Better process management and reload scripts
## Documentation
- Extensive import documentation (overview, process, settings, validation, search)
- Background job architecture documentation
- Improved README and workflow guides
- Virtual chassis usage documentation
## 0.3.18 (2025-11-21)
### Improvements
* Add all LibreNMS SNMP fields to add device form
## 0.3.17 (2025-10-24)
### Improvements
* Improve device type matching with prioritized matching strategy
* Centralize device type matching logic in utils.py (DRY principle)
* Add word boundary detection for more precise substring matches
## 0.3.16 (2025-10-24)
### New Features
* Add device field synchronization (serial number, device type, platform)
* Add virtual chassis inventory support with serial assignment to individual members
* Add VM information display in LibreNMS status card
* Add bulk interface mapping import functionality
### Improvements
* Improve virtual chassis device matching with LibreNMS for devices with VC member suffixes
* Enhance virtual chassis logic and member selection
### Development
* Add GitHub DevContainer setup for easier development environment
### Fixes
* Fix FieldError for librenms_status on device/VM status pages
### Documentation
* Update README with device field sync details
* Improve virtual chassis documentation
* Added example Interface Mapping YAML for bulk import
## 0.3.15 (2025-07-12)
### Improvements
* Improve multi-server configuration handling and add connection testing
## 0.3.14 (2025-07-08)
### Fixes
* Filter out invalid IP entries in BaseIPAddressTableView
### New Features
* View/Delete NetBox-only (unmatched) interfaces
* Add multi LibreNMS server configuration support for LibreNMS plugin
### Documentation
* Add page for multi server configuration instructions and example
## 0.3.13 (2025-06=27)
### New Feature
* Add support for IPv6 handling in IP address synchronization
### Documentation
* Add basic development documentation
## 0.3.12 (2025-04-11)
### Improvements
* Add VRF selection support to IP address table and sync
* Implement single IP address verification and VRF assignment
* Extend single IP verification to support Virtual Machines
### Under the hood
* Refactor cable and IP address synchronization methods for improved transaction handling
* Refactor IP address enrichment for improved performance
## 0.3.11 (2025-03-31)
### Improvements
* Enhance remote port enrichment for virtual chassis devices
## 0.3.10 (2025-03-17)
### Fixes
* Fix URL error when no interfaces are selected during sync
* Add hidden SNMP version field to forms and update sync logic
## 0.3.9 (2025-03-14)
### Fixes
* Fix missing add_device_modal.html template and form handling
* Fix missing interfacetypemapping template
## 0.3.8 (2025-03-06)
### Fixes
* Fix cable table error when more than one remote device returned
* Fix cable table checkboxes controls for virtual chassis devices
### Improvements
* Add slug check to Site and Location Sync
## 0.3.7 (2025-01-22)
### Fixes
* Fix issue with empty queryset to stop fielderror
### Improvements
* Enhance filtering options for devices and virtual machines
### Under the hood
* Review and refactor docstrings across all files
## 0.3.6 (2025-01-21)
### NOTE
***Netbox v4.2+ required for this release***
### New Feature
* New dedicated plugin menu item
* Add device and VM status pages
### Fixes
* Add description to interface mapping page
### Under the hood
* Update to use new Mac Address object for Netbox v4.2
## 0.3.5 (2025-01-13)
### Fixes
* Fix IP Address table not displaying for Virutal Machines
## 0.3.4 (2025-01-08)
### Fixes
* Fix VM Interface table not dispalying
## 0.3.3 (2025-01-03)
### New Feature
* Add IP address synchronization
### Fixes
* Refactor librenms_id handling in SyncInterfacesView
### Under the hood
* Refactor table.py into separate modules for better maintainability
* Enhance interface data retrieval efficiency
## 0.3.2 (2024-12-16)
### Fixes
* Refactor tab handling for interface and cable views
* Fix Duplicate ID in SNMP forms
* Refactor cable link processing and fix CSRF token error.
* Generate unique base ID for TomSelect components in VCInterfaceTable
* Add countdown interval variable to initializeCountdown function
## 0.3.1 (2024-12-13)
### Fixes
* Fix issue with tab selection not working after sync task
* Updated interface name field tooltip
## 0.3.0 (2024-12-13)
### New Setting
* Add `interface_name_feild` optional setting to allow choice of interface name field used when syncing interface data.
* Add `interface_name_field` override in GUI for per device control and flexibility.
### Improvements
* Add `librenms_id` to interface sync table and data sync
* Use of `librenms_id` custom field on interface lookup for improved matching in the cables table.
* Add Pagination support to the cables table.
### Fixes
* Fix issue with case sensitive hostname matching
### Under the hood
* Refactor views into seperate modules for better maintainability
## 0.2.9 (2024-11-30)
## Fix pypi release
Add static include in MANIFEST.in for pypi release
## 0.2.8 (2024-11-29)
### Use of Custom Field
This release introduces the option of using a custom field `librenms_id` (integer) to device and virtual machine objects in NetBox. The plugin will work without it but it is recommended for LibreNMS API lookups especially if no primary IP or FQDN available.
**Note: New static javascript file requires running collectstatic after update**
```
(venv) $ python manage.py collectstatic --no-input
```
### New Features
* Add device to LibreNMS using SNMPv3
* Create cable connection from LIbreNMS links data31
* Plugin can now use primary IP, hostname or Primary IP DNS Name to identify device in LibreNMS
* Exclude specific columns when syncing data
* Filter interface and cable tables
* Bulk edit Virtual Chassis members
### Improvements
* Add pagination to SiteLocationSyncTable
* Add site location filtering functionality and update template for search
* Refactor LibreNMSAPI to enhance device ID retrieval logic and include DNS name handling
* Enhance cable sync with device ID handling and user guidance modal
* Add device mismatch check and user feedback
* Add check for empty MAC address in format_mac_address function
* Increase API request timeout to 20 seconds
* Fix dropdown menu size issue on click
### Under the hood
* Refactor interface enabled status logic
* Fix handling of data-enabled attribute in interface table
* Improve interface mapping logic for speed matchingpull/24
* Refactor cable context handling and improve data rendering in cable tables
* Refactor Javascript into single file. Add cable sync filters and countdown timer
* Refactor device addition and enhance SNMP v3 support
## 0.2.7 (2024-11-11)
### What's Changed
* Add new interface table logic to handle virtual chassis member selection
* Update LibreNMS plugin configuration to allow disabling of SSL verification
### Interface name change
*The LibreNMS Sync interface names now use the ifDescr from Librenms. This displays the full interface name to better align with the device type library convention. e.g GigabitEthernet1/0/1 instead of Gig1/0/1.*
## 0.2.6 (2024-10-25)
### New Feature
* Sync Virtual Machine interfaces
### Bug fix
* Pagination bug where page contents would duplicate now fixed.
### Under the hood
* Refactoring of views into separate files for better maintainability.
* Code formatting improvements
* Remove unused elements
## 0.2.5 (2024-10-21)
Bug fix release:
* Missing commas in LibreNMS api module
## 0.2.4 (2024-10-21)
### Enhancements
* Add mac_address, MTU to interface sync
* Enable select all and shift click on interface sync page rows, and other improvements
* Interface mapping now accounts for speed of interface
> Update to Interface mapping modal may require recreation of existing mapping.
* Updated LibreNMS Sync page layout to prepare for new features
### Under the hood
* Refactor all views to be class-based
* Big refactor of device LibreNMS sync views to make way for new features
## 0.2.3 (2024-09-30)
* Fix bug where wrong template is used when editing interface mappings
* Remove unused templates from view
## 0.2.2 (2024-09-27)
* Fix too many arguments to add_device error
## 0.2.1 (2024-09-27)
* Fix LibreNMS hardware variable not found
* Add update_device_field to LibreNMS API
* Add device location Sync button to device plugin tab
* Change SNMP community from 'text' to 'password' for privacy
## 0.2.0 (2024-09-25)
* Update to v0.2.0 of the plugin
## 0.1.1 (2024-09-24)
* First release on PyPI.

112
docs/contributing.md Normal file
View File

@@ -0,0 +1,112 @@
# Contributing
Contributions are welcome, and they are greatly appreciated! Every little bit
helps, and credit will always be given.
We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
- Reporting a bug
- Discussing the current state of the code
- Submitting a fix
- Proposing new features
- Becoming a maintainer
## Types of Contributions
### Report Bugs
Report bugs at https://github.com/bonzo81/netbox-librenms-plugin/issues.
If you are reporting a bug, please include:
* Any details about your local environment that might be helpful in troubleshooting.
* Detailed steps to reproduce the bug.
### Fix Bugs
Look through the GitHub issues for bugs. Anything tagged with "bug" and "help
wanted" is open to whoever wants to implement it.
### Implement Features
Look through the GitHub issues for features. Anything tagged with "enhancement"
and "help wanted" is open to whoever wants to implement it.
### Write Documentation
NetBox LibreNMS Plugin could always use more documentation, whether as part of the
official NetBox LibreNMS Plugin docs, in docstrings, or even on the web in blog posts,
articles, and such.
### Submit Feedback
The best way to send feedback is to file an issue at https://github.com/bonzo81/netbox-librenms-plugin/issues.
If you are proposing a feature:
* Explain in detail how it would work.
* Keep the scope as narrow as possible, to make it easier to implement.
* Remember that this is a volunteer-driven project, and that contributions
are welcome :)
## Get Started!
Ready to contribute? Here's how to set up `netbox-librenms-plugin` for local development.
1. Fork the `netbox-librenms-plugin` repo on GitHub.
2. Clone your fork locally
```
$ git clone git@github.com:<username>/netbox-librenms-plugin.git
```
3. Activate the NetBox virtual environment (see the NetBox documentation under [Setting up a Development Environment](https://docs.netbox.dev/en/stable/development/getting-started/)):
```
$ source /opt/netbox/venv/bin/activate
```
4. Add the plugin to NetBox virtual environment in Develop mode (see [Plugins Development](https://docs.netbox.dev/en/stable/plugins/development/)):
To ease development, it is recommended to go ahead and install the plugin at this point using setuptools' `develop` mode. This will create symbolic links within your Python environment to the plugin development directory. Call `pip` from the plugin's root directory with the `-e` flag:
```
$ pip install -e .
```
5. Create a branch for local development:
```
$ git checkout -b name-of-your-bugfix-or-feature
```
Now you can make your changes locally.
6. Commit your changes and push your branch to GitHub:
```
$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature
```
7. Submit a pull request through the GitHub website.
## Pull Request Guidelines
Before you submit a pull request, check that it meets these guidelines:
1. The pull request should include tests.
2. If the pull request adds functionality, the docs should be updated. Put
your new functionality into a function with a docstring, and add the
feature to the list in README.md.
3. The pull request should work for Python 3.10+. Check
https://github.com/bonzo81/netbox-librenms-plugin/actions
and make sure that the tests pass for all supported Python versions.
## Deploying
A reminder for the maintainers on how to deploy.
Make sure all your changes are committed (including an entry in CHANGELOG.md) and that all tests pass.
Then in the github project go to `Releases` and create a new release with a new tag. This will automatically upload the release to pypi:

View File

@@ -0,0 +1,10 @@
# Development Guide: Navigating the Codebase
This guide is intended for developers and contributors working on the NetBox LibreNMS Plugin. It provides a broad overview of the codebase structure and key elements. For detailed NetBox plugin development documentation, see the [official NetBox documentation](https://docs.netbox.dev/en/stable/plugins/development/).
## Contents
- [Project Structure](./structure.md): Overview of the main folders and files in the plugin.
- [Views & Inheritance](./views.md): How views are organized, inheritance patterns, and extension tips.
- [Mixins](./mixins.md): Reusable logic for views, including API access and caching.
- [Templates](./templates.md): Template structure, conventions, and customization tips.

View File

@@ -0,0 +1,44 @@
# Mixins
Mixins in `views/mixins.py` provide reusable logic to keep views clean and DRY (Don't Repeat Yourself). They are designed to be combined with Django or NetBox views to add specific behaviors or shared functionality. When adding new views, consider using or extending these mixins to maintain consistency and reduce code duplication.
### Key Mixins
**LibreNMSAPIMixin**
- Provides a `librenms_api` property for accessing the LibreNMS API from any view.
- Ensures a single instance of the API client is reused per view instance.
- Example usage: Add to views that need to fetch or sync data with LibreNMS.
**CacheMixin**
- Supplies helper methods for generating cache keys related to objects and data types (e.g., ports, links, vlans).
- Useful for views that cache data fetched from LibreNMS to improve performance.
- Methods:
- `get_cache_key(obj, data_type="ports")`: Returns a unique cache key for the object and data type.
- `get_last_fetched_key(obj, data_type="ports")`: Returns a cache key for tracking when data was last fetched.
- `get_vlan_overrides_key(obj)`: Returns a cache key for storing user VLAN group override selections.
**VlanAssignmentMixin**
- Provides VLAN group resolution and assignment logic used by both the Interfaces tab (per-interface VLAN assignments) and the VLANs tab (VLAN object sync).
- Resolves which VLAN groups are relevant to a device based on a scope hierarchy: Rack → Location → Site → SiteGroup → Region → Global.
- Methods:
- `get_vlan_groups_for_device(device)`: Returns all VLAN groups relevant to the device based on scope hierarchy.
- `_build_vlan_lookup_maps(vlan_groups)`: Builds lookup dictionaries mapping VIDs to groups, VLANs, and names.
- `_select_most_specific_group(groups, device)`: Resolves ambiguity when a VID exists in multiple groups by selecting the most specific scope.
- `_find_vlan_in_group(vid, vlan_group_id, lookup_maps)`: Finds a VLAN by VID, preferring the specified group.
- `_update_interface_vlan_assignment(interface, vlan_data, vlan_group_map, lookup_maps)`: Updates interface mode, untagged VLAN, and tagged VLANs in NetBox.
### How to Use Mixins
To use a mixin, simply add it to the inheritance list of your view class. For example:
```python
from .mixins import LibreNMSAPIMixin, CacheMixin
class MyCustomView(LibreNMSAPIMixin, CacheMixin, SomeBaseView):
# ... your view logic ...
```
Mixins can be combined as needed. Place mixins before the main base view to ensure their methods and properties are available.

View File

@@ -0,0 +1,35 @@
# Project Structure
This document provides an overview of the NetBox LibreNMS Plugin's codebase organization.
## Main Directories
- `netbox_librenms_plugin/` — Main plugin code
- `views/` — Custom views for devices, mappings, VMs, etc.
- `base/` — Abstract base views for shared logic (interfaces, cables, IP addresses, VLANs)
- `object_sync/` — Per-model sync views registered as tabs on Device/VM detail pages
- `sync/` — POST-only views that apply sync changes (interfaces, cables, IP addresses, VLANs, devices)
- `models.py` — Database models
- `forms.py` — Custom forms
- `tables/` — Table definitions for UI
- `templates/` — Custom templates
- `netbox_librenms_plugin/` — Main template directory
- `inc/` — Shared template fragments (e.g., paginator)
- `api/` — API serializers, views, and URLs
- `import_utils/` — Import pipeline logic, split into focused modules
- `device_operations.py` — Device validation, single-device import, filtered fetch
- `vm_operations.py` — VM creation and import logic
- `bulk_import.py` — Multi-device / bulk import orchestration
- `filters.py` — LibreNMS device filtering and retrieval
- `permissions.py` — User permission checking helpers
- `cache.py` — Cache key generation
- `virtual_chassis.py` — Virtual chassis data helpers
- `import_validation_helpers.py` — Validation state mutation during import (role/cluster/rack assignment, issue removal, status recalculation)
- `migrations/` — Django migrations
- `utils.py` — Utility functions
- `navigation.py` — Menu/navigation integration
- `static/` — Static assets (JS, CSS)
- `netbox_librenms_plugin/` — Plugin-specific static files
- `js/` — JavaScript files
- `tests/` — Test suite
- `docs/` — Documentation

View File

@@ -0,0 +1,28 @@
# Templates
Templates are located in `templates/netbox_librenms_plugin/` and follow NetBox's conventions, using Django's template language. The plugin uses a combination of base templates, partials, and includes to keep the UI modular and maintainable.
### Structure and Conventions
- **Base templates** (e.g., `librenms_sync_base.html`, `interfacetypemapping.html`) typically extend NetBox's generic templates (like `generic/object.html` or `generic/object_list.html`).
- **Partials and includes** (e.g., `_interface_sync.html`, `_interface_sync_content.html`, `_cable_sync.html`) are used for reusable UI components and AJAX/HTMX content updates.
- **The `inc/` directory** contains shared fragments, such as pagination controls (`paginator.html`).
### Customization and Inheritance
- Use the Django template tag `extends` to build on top of NetBox or plugin base templates, and the `block` tag to override or inject content.
- Use the Django template tag `include` for reusable sections (e.g., tables, forms, or modal dialogs).
- Static assets (JS/CSS) are loaded with the Django template tag `load static` and referenced using the `static` tag.
- Context variables and template tags (e.g., `helpers`, `plugins`, `render_table`) are used to render dynamic content and integrate with NetBox features.
### Examples
**Sync Views:**
- `librenms_sync_base.html` provides the main layout for device/VM sync pages, extending NetBox's object template and including custom blocks for status, actions, and content.
- `_interface_sync.html` and `_interface_sync_content.html` are used for the interface sync tab, supporting dynamic updates and user actions (like syncing selected interfaces).
- `_vlan_sync.html` and `_vlan_sync_content.html` provide the VLAN sync tab (Devices only), with per-VLAN group selection dropdowns, color-coded status indicators (green/yellow/red), and a cache countdown timer. The VLANs tab is conditionally rendered only for devices in `librenms_sync_base.html`.
**Mapping Views:**
- `interfacetypemapping.html` and `interfacetypemapping_list.html` display and manage interface type mappings, using table layouts and info alerts.
For more on NetBox's template system, see the [NetBox documentation](https://netbox.readthedocs.io/en/stable/plugins/development/#templates).

247
docs/development/testing.md Normal file
View File

@@ -0,0 +1,247 @@
# Testing Guide
This guide explains how to run the test suite, write new tests, and debug failures.
## Quick Start
Run all tests with a single command:
```bash
make unittest
```
Or run pytest directly:
```bash
pytest netbox_librenms_plugin/tests/ -v
```
## Test Structure
The test suite covers all major plugin functionality. Tests are organized by the module they verify:
| Test File | What It Tests |
|-----------|---------------|
| [test_librenms_api.py](../../netbox_librenms_plugin/tests/test_librenms_api.py) | LibreNMS API client—connections, device operations, locations, ports, and error handling |
| [test_import_utils.py](../../netbox_librenms_plugin/tests/test_import_utils.py) | Device import logic—filtering, validation, and data transformation |
| [test_import_validation_helpers.py](../../netbox_librenms_plugin/tests/test_import_validation_helpers.py) | Field validation for sites, roles, platforms, and device types |
| [test_utils.py](../../netbox_librenms_plugin/tests/test_utils.py) | General utilities—name matching, speed conversion, and data formatting |
| [test_background_jobs.py](../../netbox_librenms_plugin/tests/test_background_jobs.py) | Background job execution and view decision logic |
| [test_vlan_sync.py](../../netbox_librenms_plugin/tests/test_vlan_sync.py) | VLAN sync—API fetching, comparison logic, CSS class utilities, and sync actions |
| [test_interface_vlan_sync.py](../../netbox_librenms_plugin/tests/test_interface_vlan_sync.py) | Interface VLAN assignments—group resolution, mode detection, and per-interface VLAN assignment |
| [test_librenms_id.py](../../netbox_librenms_plugin/tests/test_librenms_id.py) | Multi-server librenms_id helpers—get/set/find/migrate and boolean rejection |
| [test_mixins.py](../../netbox_librenms_plugin/tests/test_mixins.py) | View mixins—CacheMixin key generation, LibreNMSAPIMixin lazy init |
| [test_sync_devices.py](../../netbox_librenms_plugin/tests/test_sync_devices.py) | Device sync views—field updates, platform creation |
| [test_sync_interfaces.py](../../netbox_librenms_plugin/tests/test_sync_interfaces.py) | Interface sync—port matching, attribute updates, MAC handling, librenms_id assignment |
| [test_virtual_chassis.py](../../netbox_librenms_plugin/tests/test_virtual_chassis.py) | Virtual chassis detection—VC member naming patterns and name generation |
| [test_sync_view_mismatch.py](../../netbox_librenms_plugin/tests/test_sync_view_mismatch.py) | Sync page context—device type mismatch detection and badge rendering |
| [test_coverage_device_fields.py](../../netbox_librenms_plugin/tests/test_coverage_device_fields.py) | Device field sync view—field update logic and device field mapping |
| [test_coverage_list.py](../../netbox_librenms_plugin/tests/test_coverage_list.py) | Import list view—background job decision, job result loading, and GET handler |
| [test_coverage_api.py](../../netbox_librenms_plugin/tests/test_coverage_api.py) | LibreNMS API client—malformed payload guards, error paths, and edge cases |
| [test_coverage_api2.py](../../netbox_librenms_plugin/tests/test_coverage_api2.py) | API views—device status, background job management, VM status endpoints |
| [test_coverage_base_views.py](../../netbox_librenms_plugin/tests/test_coverage_base_views.py) | Base view coverage tests—sync table views, context data, and data pipeline |
| [test_coverage_base_views2.py](../../netbox_librenms_plugin/tests/test_coverage_base_views2.py) | Additional base view coverage—IP address sync, cable matching, edge cases |
| [test_coverage_cache.py](../../netbox_librenms_plugin/tests/test_coverage_cache.py) | Import cache helpers—cache key generation, active search tracking, metadata |
| [test_coverage_device_operations.py](../../netbox_librenms_plugin/tests/test_coverage_device_operations.py) | Device validation—type matching, serial handling, VC detection, role lookup |
| [test_coverage_forms.py](../../netbox_librenms_plugin/tests/test_coverage_forms.py) | Import forms—filter form choices, background-job option guards, field validation |
| [test_coverage_mixins.py](../../netbox_librenms_plugin/tests/test_coverage_mixins.py) | View mixins—VLAN group scope resolution, VlanAssignmentMixin, scope priority |
| [test_coverage_sync_interfaces.py](../../netbox_librenms_plugin/tests/test_coverage_sync_interfaces.py) | Interface sync view—port caching, attribute updates, MAC handling, VC member routing |
| [test_coverage_sync_view.py](../../netbox_librenms_plugin/tests/test_coverage_sync_view.py) | Sync view base class—context preparation and tab rendering |
| [test_coverage_sync_views.py](../../netbox_librenms_plugin/tests/test_coverage_sync_views.py) | Sync action views—cables, IP addresses, VLAN sync action handlers |
| [test_coverage_sync_views2.py](../../netbox_librenms_plugin/tests/test_coverage_sync_views2.py) | Additional sync action view coverage—device fields, device name/type sync |
| [test_coverage_sync_views3.py](../../netbox_librenms_plugin/tests/test_coverage_sync_views3.py) | Further sync action view coverage—location sync, VLAN assignment edge cases |
| [test_coverage_actions.py](../../netbox_librenms_plugin/tests/test_coverage_actions.py) | Import action views—bulk import, device role/cluster/rack update, validation details |
| [test_coverage_filters.py](../../netbox_librenms_plugin/tests/test_coverage_filters.py) | Import filter logic—filter form processing and device count helpers |
| [test_init.py](../../netbox_librenms_plugin/tests/test_init.py) | Plugin startup—`_ensure_librenms_id_custom_field` creation, type migration, and multi-DB alias handling |
| [test_coverage_tables.py](../../netbox_librenms_plugin/tests/test_coverage_tables.py) | Sync tables—column rendering, row data, interface and cable table helpers |
| [test_coverage_utils.py](../../netbox_librenms_plugin/tests/test_coverage_utils.py) | Utility function coverage—name matching, speed conversion, site/platform lookup |
| [test_coverage_virtual_chassis.py](../../netbox_librenms_plugin/tests/test_coverage_virtual_chassis.py) | Virtual chassis coverage—VC creation, position conflict handling, member naming |
| [test_coverage_vlans_table.py](../../netbox_librenms_plugin/tests/test_coverage_vlans_table.py) | VLAN sync table—column rendering, group assignment, VLAN comparison rows |
| [test_sync_modules.py](../../netbox_librenms_plugin/tests/test_sync_modules.py) | Module sync—inventory matching, module type resolution, and normalization rules |
| [test_modules_view.py](../../netbox_librenms_plugin/tests/test_modules_view.py) | Module sync view—context preparation, table rendering, and module bay mapping |
| [test_tables_modules.py](../../netbox_librenms_plugin/tests/test_tables_modules.py) | Module tables—column rendering, row formatting, and action buttons |
| [test_permissions.py](../../netbox_librenms_plugin/tests/test_permissions.py) | Permission enforcement—mixin contracts, object-level permissions, and write guards |
| [test_vm_operations.py](../../netbox_librenms_plugin/tests/test_vm_operations.py) | VM operations—virtual machine sync, interface handling, and VM-specific views |
| [test_integration_sync.py](../../netbox_librenms_plugin/tests/test_integration_sync.py) | Integration tests—API client against local mock HTTP server |
| [test_integration_virtual_chassis.py](../../netbox_librenms_plugin/tests/test_integration_virtual_chassis.py) | Integration tests—VC detection, negative cache, multi-server cache isolation |
| [test_view_wiring.py](../../netbox_librenms_plugin/tests/test_view_wiring.py) | Smoke tests—view class MRO, mixin wiring, permission contracts, and template syntax |
Supporting files:
| File | Purpose |
|------|---------|
| [conftest.py](../../netbox_librenms_plugin/tests/conftest.py) | Shared pytest fixtures |
| [test_librenms_api_helpers.py](../../netbox_librenms_plugin/tests/test_librenms_api_helpers.py) | Auto-use fixture for API configuration mocking |
| [mock_librenms_server.py](../../netbox_librenms_plugin/tests/mock_librenms_server.py) | Minimal HTTP mock server for integration tests |
## Running Tests
### Running Specific Tests
```bash
# Run a specific test file
pytest netbox_librenms_plugin/tests/test_librenms_api.py -v
# Run a specific test class
pytest netbox_librenms_plugin/tests/test_librenms_api.py::TestLibreNMSAPIConnection -v
# Run a specific test method
pytest netbox_librenms_plugin/tests/test_librenms_api.py::TestLibreNMSAPIConnection::test_connection_success -v
```
### Running Tests by Area
```bash
# API client tests
pytest netbox_librenms_plugin/tests/test_librenms_api.py netbox_librenms_plugin/tests/test_coverage_api.py netbox_librenms_plugin/tests/test_coverage_api2.py -v
# Import and validation tests
pytest netbox_librenms_plugin/tests/test_import_utils.py netbox_librenms_plugin/tests/test_import_validation_helpers.py netbox_librenms_plugin/tests/test_utils.py -v
# Background job tests
pytest netbox_librenms_plugin/tests/test_background_jobs.py -v
# Multi-server librenms_id tests
pytest netbox_librenms_plugin/tests/test_librenms_id.py -v
# Sync view tests (devices, interfaces, modules)
pytest netbox_librenms_plugin/tests/test_sync_devices.py netbox_librenms_plugin/tests/test_sync_interfaces.py netbox_librenms_plugin/tests/test_sync_modules.py -v
# Integration tests (API client against mock HTTP server)
pytest netbox_librenms_plugin/tests/test_integration_*.py -v
# Sync view mismatch detection and permission enforcement
pytest netbox_librenms_plugin/tests/test_sync_view_mismatch.py netbox_librenms_plugin/tests/test_permissions.py -v
# View wiring and template syntax smoke tests
pytest netbox_librenms_plugin/tests/test_view_wiring.py -v
```
### Debugging Failed Tests
```bash
# Show full traceback
pytest netbox_librenms_plugin/tests/ -v --tb=long
# Show print statements during tests
pytest netbox_librenms_plugin/tests/ -v -s
# Stop on first failure
pytest netbox_librenms_plugin/tests/ -v -x
# Re-run only failed tests from last run
pytest netbox_librenms_plugin/tests/ -v --lf
```
## Testing Philosophy
The test suite prioritizes speed and isolation so you can run tests frequently during development:
- **Mock-based**: Unit tests use `MagicMock` instead of real database objects. No Django database setup required.
- **Fast execution**: The full suite runs in approximately 15-20 seconds (varies by environment).
- **Isolated**: Each test is independent with no shared state between tests.
- **No external network access**: Tests never call external services. Integration tests use a local loopback HTTP server (`mock_librenms_server.py`) to exercise the real API client against realistic HTTP responses without requiring a running LibreNMS instance.
- **Coverage exclusions**: Test files themselves are excluded from coverage reports (see `[tool.coverage.run]` omit list in `pyproject.toml`).
This approach means tests work identically in your local development environment, in the devcontainer, and in CI pipelines.
## Writing New Tests
### Basic Test Template
New tests should follow this structure:
```python
from unittest.mock import MagicMock, patch
class TestFeatureName:
"""Tests for [feature description]."""
pytest_plugins = ["tests.test_librenms_api_helpers"]
@patch("netbox_librenms_plugin.module_name.external_dependency")
def test_specific_behavior(self, mock_dependency, mock_librenms_config):
"""Describe what this test verifies."""
# Arrange - set up test data and mocks
mock_dependency.return_value = {"expected": "response"}
# Act - call the code being tested
from netbox_librenms_plugin.module_name import function_to_test
result = function_to_test(input_data)
# Assert - verify the results
assert result == expected_value
mock_dependency.assert_called_once_with(expected_args)
```
### Key Testing Conventions
**Use inline imports** inside test methods to avoid Django initialization at module load time:
```python
def test_something(self):
from netbox_librenms_plugin.librenms_api import LibreNMSAPI
api = LibreNMSAPI(server_key="default")
```
**Mock NetBox models** with `MagicMock()` instead of creating real database objects:
```python
device = MagicMock()
device.name = "test-device"
device.primary_ip.address.ip = "192.168.1.1"
```
**Patch at the source module**, not where the function is imported:
```python
# Correct - patch where the function is defined
@patch("netbox_librenms_plugin.import_utils.process_device_filters")
# Incorrect - patching the import location
@patch("netbox_librenms_plugin.views.imports.list.process_device_filters")
```
### Available Fixtures
These fixtures are defined in [conftest.py](../../netbox_librenms_plugin/tests/conftest.py):
- `mock_librenms_config` — Automatically mocks plugin configuration for all tests
- `mock_response_factory` — Factory for creating mock HTTP responses
- `mock_netbox_device` — Pre-configured mock NetBox Device object
- `mock_netbox_vm` — Pre-configured mock NetBox VM object
### Common Assertion Patterns
```python
# Methods returning (success, data) tuples
success, data = api.get_device_info(123)
assert success is True
assert data["hostname"] == "expected-hostname"
# Methods returning dicts with error flags
result = api.test_connection()
assert "error" not in result
# Verifying exceptions are raised
with pytest.raises(ValueError, match="Invalid configuration"):
api.method_that_should_fail()
# Verifying mock calls
mock_get.assert_called_once()
mock_post.assert_called_with(expected_url, headers=expected_headers, json=expected_data)
mock_delete.assert_not_called()
```
## CI/CD Compatibility
The tests run in any environment without external dependencies:
- No database connection required
- No external network access needed (integration tests use local loopback only)
- Fast execution suitable for pre-commit hooks
- Clear failure messages for debugging
- Works in containerized environments
This makes the test suite suitable for GitHub Actions, pre-commit hooks, or any CI pipeline you choose to implement.

74
docs/development/views.md Normal file
View File

@@ -0,0 +1,74 @@
# Views & Inheritance
Views are organized by resource type (e.g., devices, mappings, VMs) in the `views/` directory. The codebase uses a layered approach to views, leveraging inheritance and mixins to maximize code reuse and maintainability.
### View Organization
**Resource-specific views:**
- Device and VM sync tabs live under `object_sync/` (see `object_sync/devices.py` and `object_sync/vms.py`), while mappings/settings/status views remain as individual modules alongside the package.
- The LibreNMS import workflow is grouped under `views/imports/`: `list.py` renders the main table view and `actions.py` contains the HTMX endpoints (preview, validation, bulk execute). All legacy handlers formerly in `librenms_import_views.py` and `device_import_views.py` were folded into this package.
**Base views:**
- The `base/` subdirectory contains abstract base views (e.g., `BaseLibreNMSSyncView`, `BaseInterfaceTableView`, `BaseCableTableView`, `BaseIPAddressTableView`, `BaseVLANTableView`) that encapsulate shared logic for related resources.
**Mixins:**
- Shared behaviors (e.g., API access, caching) are factored into mixins in `mixins.py` and combined with base or resource-specific views as needed.
### Inheritance Patterns
- Most resource-specific views inherit from a base view in `base/` and one or more mixins.
- Base views themselves often inherit from NetBox or Django generic views (e.g., `generic.ObjectListView`, `django.views.View`).
- This allows resource-specific views to override or extend only the methods they need, while inheriting default behaviors from base classes and mixins.
#### Example: Device Sync View
```python
from .base.librenms_sync_view import BaseLibreNMSSyncView
from .mixins import LibreNMSAPIMixin
class DeviceLibreNMSSyncView(BaseLibreNMSSyncView):
# Inherits API access and sync logic from base/mixins
# Only device-specific logic needs to be implemented here
...
```
#### Example: Interface Table View
```python
from .base.interfaces_view import BaseInterfaceTableView
from .mixins import CacheMixin, LibreNMSAPIMixin
class DeviceInterfaceTableView(BaseInterfaceTableView):
model = Device
# Implements get_interfaces and get_redirect_url for devices
...
```
#### Example: VLAN Table View
```python
from .base.vlan_table_view import BaseVLANTableView
class DeviceVLANTableView(BaseVLANTableView):
model = Device
# Inherits VLAN comparison, group resolution, and caching from base view
# Only the model attribute needs to be set
...
```
`BaseVLANTableView` additionally inherits `VlanAssignmentMixin` for VLAN group scope resolution. It fetches device VLANs from LibreNMS, compares them against NetBox VLAN objects across relevant VLAN groups, and renders a color-coded table with per-VLAN group dropdowns.
### Customizing or Adding Views
- To add a new view for a resource, inherit from the relevant base view and mixins, then override or extend methods as needed.
- Use the base views as templates for structure and required methods.
- Register new views in `urls.py` and add templates if needed.
### Tips
- Check the `base/` directory for reusable logic before writing new view code.
- Use mixins for cross-cutting concerns (API, caching, permissions).
- Keep resource-specific views focused on their unique logic; delegate shared logic to base classes and mixins.

72
docs/feature_list.md Normal file
View File

@@ -0,0 +1,72 @@
### [Device Import](librenms_import/overview.md)
* Search and discover devices from LibreNMS using flexible filters
* Validate device prerequisites before import (Site, Device Type, Device Role)
* Import devices as physical Devices or Virtual Machines
* Smart matching for Sites, Device Types, and Platforms
* Bulk import support
* Automatic Virtual Chassis creation for stackable devices
* Background job processing for large device sets
* Duplicate detection to prevent re-importing existing devices
### Plugin Settings
* Multi-server LibreNMS configuration support
* Configurable device naming defaults (sysName vs hostname)
* Domain stripping options during import for cleaner device names
* Virtual Chassis member naming pattern customization during import
### Device
* LibreNMS device identification via:
* [Custom field `librenms_id`](usage_tips/custom_field.md) _(recommended)_
* Primary IP address
* Primary IP DNS name
* Hostname
* Add device to LibreNMS from netbox via SNMP v2c or v3
### [Virtual Chassis Support](usage_tips/virtual_chassis.md)
* Automatic VC member selection for each interface
* Member-specific interface synchronization
* Bulk member editing capabilities
### Interface Sync {#interface-sync}
* Create or Update interface in NetBox from LibreNMS interface data
* Name
* Description
* Status (Enabled/Disabled)
* Type (with custom mapping support)
* Speed
* MAC Address
* MTU
* VLAN assignments
* Sync all or specific fields
### Cable Sync {#cable-sync}
* Create Cable connection in NetBox from LibreNMS links data
* Best results when the [custom field](usage_tips/custom_field.md) `librenms_id` is populated on interfaces
### IP Address Sync {#ip-address-sync}
* Create IP address objects in Netbox from LibreNMS device IP data
* Best results when the [custom field](usage_tips/custom_field.md) `librenms_id` is populated on interfaces
### VLAN Sync {#vlan-sync}
* Create VLAN objects in NetBox from LibreNMS device VLAN data
* Per-VLAN group assignment with scope-aware auto-selection
### Location
* NetBox Site to LibreNMS location synchronization
* Sync location latitude and longitude values from NetBox to LibreNMS
### [Interface Mapping](usage_tips/interface_mappings.md)
* Customizable LibreNMS to NetBox interface type mappings
* Interface Speed-based mapping rules
* Bulk import support

Binary file not shown.

After

Width:  |  Height:  |  Size: 672 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 77 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.0 KiB

View File

@@ -0,0 +1,76 @@
# Background Jobs & Caching
The Device Import feature uses background job processing and intelligent caching for both searching and importing devices. Background jobs are enabled by default for both operations to handle large device sets efficiently.
## Background Jobs
Background jobs run asynchronously in NetBox's job system for both device searches and import operations.
### Background Job Processing
Both device searches and import operations can run as background jobs (default) or synchronously. Background jobs are recommended for:
- Large device sets (especially searches with more than 50 devices)
- Operations with Virtual Chassis detection enabled
- Import operations of any size
**Benefits of background jobs:**
- Avoid browser timeouts on long-running operations
- Cancel operations in progress if needed
- Continue using NetBox while the job runs
- Review detailed logs and results after completion
### Viewing Job Status
All background jobs appear in NetBox's **Jobs** interface, where you can view status, start time, duration, and results.
## Caching
The import table caches data for 5 minutes to reduce load times and minimize API calls to LibreNMS. Cache keys are unique per filter combination.
### What Gets Cached
The cache includes both LibreNMS device data AND NetBox reference data used in the import table:
**From LibreNMS:**
- Device lists matching your search filters
- Device details (hostname, sysName, location, hardware, etc.)
- Virtual chassis detection results
**From NetBox:**
- Available device roles (for the role dropdown in each row)
- Available VM clusters (for VM imports)
- Available racks for each site (filtered by the device's matched site)
This means if you add a new role, create a new rack, or add a new cluster in NetBox, those changes won't appear in the import table dropdowns until you clear the cache or wait for it to expire (5 minutes).
### Controlling Cache
The search form includes a "Clear cache before search" checkbox:
| Setting | Behavior |
|---------|----------|
| Unchecked (default) | Uses cached data if available. Fastest results. |
| Checked | Forces fresh data retrieval from both LibreNMS and NetBox. |
**When to clear cache:**
- After adding or updating devices in LibreNMS
- After adding new roles, racks, or clusters in NetBox that should appear in import dropdowns
- When troubleshooting import issues
- When you need to verify current state
**When to keep cache enabled:**
- Normal operations and when refining search filters
- When repeatedly working with the same set of devices
- When NetBox reference data hasn't changed
### Active Cached Searches
The import page displays all your recent searches at the top, showing which filter combinations, that are still found in the cache. Each cached search shows the filters used, device count, and time remaining before expiration.
Click any cached search to instantly reload those results without re-running filters or Virtual Chassis detection. This is particularly useful when switching between different filter combinations.
Cached searches expire after 5 minutes of inactivity (or what you set as the cache timeout). The countdown timer shows how long each search remains available.
The "Clear cache before search" option only clears the cache for the specific filter combination you're searching—other cached searches remain available.

View File

@@ -0,0 +1,71 @@
# Import Settings
Configure how devices are named and what data is imported from LibreNMS to NetBox.
## Setting Defaults
To configure global defaults for all imports:
1. Navigate to **Plugins → LibreNMS Plugin → Settings**
2. Click **Plugin Settings**
3. Configure Use sysName and Strip Domain to your preferred defaults
4. Save changes
These defaults apply to all future imports unless overridden during the import process.
## User Preferences and Defaults
The plugin uses a two-tier preference system for the **Use sysName** and **Strip Domain** toggles:
1. **Plugin defaults** (set by admins on the Settings page) apply to all users who have not yet changed their own toggle settings.
2. **Per-user preferences** are saved automatically when a user changes a toggle on the import page. Once saved, the user's preference takes priority over the plugin default.
**Important notes:**
- Changing the plugin defaults does **not** override existing user preferences. Users who have previously changed a toggle keep their personal setting.
- When an admin saves import settings, only the admin's own preferences are updated to match the new defaults. Other users are unaffected.
- There is no "reset to defaults" for individual users. To revert to the plugin default, a user simply needs to toggle the setting to match.
## Device Naming Options
The plugin provides two settings that control how device names are created in NetBox. Both are configured in Plugin Settings under **Plugins → LibreNMS Plugin → Settings → Plugin Settings** and can be overridden on the LibreNMS import page.
### Use sysName
Controls which field from LibreNMS becomes the device name in NetBox.
- **Enabled** (default): Uses the SNMP sysName, falling back to LibreNMS hostname if sysName is not available
- **Disabled**: Uses the LibreNMS hostname field
### Strip Domain
Removes domain suffixes from device names to create shorter, cleaner names.
- **Enabled**: Removes domain suffixes (e.g., "router.example.com" becomes "router"). IP addresses are preserved without modification
- **Disabled**: Keeps the full name as-is
### Naming Examples
```
LibreNMS sysName: router-core-01.example.com
LibreNMS hostname: 10.0.0.1
Use sysName + Strip domain → "router-core-01"
Use sysName + Keep domain → "router-core-01.example.com"
Use hostname + Strip domain → "10.0.0.1" (IP preserved)
Use hostname + Keep domain → "10.0.0.1"
```
If neither sysName nor hostname exists, the plugin generates a name as `device-{librenms_id}`.
## Per-Import Overrides
On the import page, the **Use sysName** and **Strip Domain** toggles are pre-populated from your saved preference (or the plugin default if you haven't set one). Changing a toggle immediately saves your preference for next time and applies to the current import.
This allows you to:
- Import some devices with sysName and others with hostname
- Apply domain stripping selectively based on device type or location
- Test different naming conventions — your last choice is remembered automatically

View File

@@ -0,0 +1,71 @@
# Device Import Overview
The Device Import feature allows you to discover and import devices from LibreNMS into NetBox. This streamlines the process of populating NetBox with devices that are already monitored in LibreNMS, while giving you full control over how devices are imported.
The import page should be clear and intuitive to use, but this overview provides additional context and details.
## How It Works
The import workflow consists of three main steps:
1. **[Search & Filter](search.md)** - Find devices in LibreNMS using flexible filter criteria
2. **[Review & Validate](validation.md)** - Validate import readiness and configure missing NetBox objects
3. **[Import](import_settings.md)** - Configure import settings and create devices in NetBox
The plugin validates all required NetBox objects (Site, Device Type, Device Role) before allowing import.
## Key Features
**Flexible Filtering**
: Search by location, type, operating system, hostname, system name, or hardware model. Combine filters for precise device selection.
**Smart Validation**
: Automatic matching for Sites, Device Types, and Platforms based on LibreNMS data. Clear indicators for what's missing.
**Device or VM**
: Import as physical Devices (requires Site, Device Type, Role) or Virtual Machines (requires Cluster).
**Virtual Chassis Support**
: Automatic detection and creation of Virtual Chassis objects for stackable switches.
**Background Processing**
: Large device sets can be processed using NetBox background jobs with progress tracking and cancellation.
## Accessing the Feature
Navigate to the import interface through the NetBox menu:
**Plugins → LibreNMS Plugin → Import → LibreNMS Import**
This opens the device import page where you can search for and import devices from your LibreNMS instance.
## What Gets Created
When a device is imported, the plugin creates:
**Device or VirtualMachine Object**
: With all validated attributes (name, site, device type, role, platform, serial, rack, etc.). Note that rack assignment places the device in the rack without setting a specific rack unit (U) position—devices appear in the "Non racked" section and require manual U assignment.
**LibreNMS ID Custom Field**
: Automatically set to link the NetBox object to the LibreNMS device. This enables all other plugin features (interface sync, cable sync, etc.)
**Virtual Chassis** (if detected)
: For stackable devices, creates the Virtual Chassis object and assigns member positions based on detected inventory data.
After import, devices appear in NetBox with a comment indicating they were imported by the plugin, including the import timestamp.
## Multi-Server Support
If your NetBox installation is configured with multiple LibreNMS servers, the import feature automatically uses the currently selected server from Plugin Settings.
All imported devices are linked to the server used during import, allowing you to maintain devices from multiple LibreNMS instances in a single NetBox installation.
## Next Steps
Explore each step of the import workflow:
- [Search for Devices](search.md) - Learn about filters, matching rules, and search options
- [Validation & Configuration](validation.md) - Understand validation status and resolve issues
- [Import Settings](import_settings.md) - Configure device naming and import options
- [Background Jobs & Caching](background_jobs.md) - Job processing and performance optimization

View File

@@ -0,0 +1,98 @@
The import feature requires at least one filter to search for devices. This prevents accidentally loading thousands of devices and helps you work with focused device sets.
## Available Filters
LibreNMS Location
: Exact match by LibreNMS location ID. The dropdown shows all locations from your LibreNMS instance with their names and IDs.
LibreNMS Type
: Exact match by device type (network, server, storage, wireless, firewall, power, appliance, printer, loadbalancer, other).
Operating System
: Partial match by OS name. For example, "ios" matches "cisco-ios", "ios-xe", "cisco-ios-xr".
LibreNMS Hostname
: Partial match by the hostname or IP address used to add the device to LibreNMS.
LibreNMS System Name
: When used alone, performs an exact match on the SNMP sysName. When combined with other filters, performs a partial match.
Hardware
: Partial match by LibreNMS hardware model. For example, "C9300" matches devices with hardware like "Cisco C9300-48P".
## Additional Search Options
Include Disabled Devices
: When checked, includes devices marked as disabled in LibreNMS. By default, only active devices are shown.
Include Virtual Chassis Detection
: When checked, analyzes device inventory to detect stackable switches and chassis. This adds processing time but provides helpful information about multi-member devices. See [Virtual Chassis](../usage_tips/virtual_chassis.md) for details.
Clear cache before search
: Forces the plugin to fetch fresh data from LibreNMS instead of using cached results. LibreNMS data is normally cached for 5 minutes to improve performance.
Exclude Existing Devices
: When checked, hides devices that already exist in NetBox. By default, all devices are shown including those already imported.
## Filter Matching Rules
Understanding how filters work helps you get the right results:
Exact Match Filters
: Location, Type, and OS (when used alone) must match exactly as shown in LibreNMS.
Partial Match Filters
: Hostname, OS (with other filters), and System Name (with other filters) find devices containing your search text.
Multiple Filters
: All filters must match for a device to appear in results. Start with Location or Type to narrow results, then refine with additional filters.
### Filter Examples
**Find all network devices in New York**
```
Location: New York
Type: network
```
**Find Cisco devices**
```
Type: network
OS: ios
```
**Find specific hardware model at New York**
```
Location: New York
Hardware: C9300
Type: network
```
**Find a specific device by name**
```
System Name: router-core-01.example.com
```
**Find devices with "F" in hostname at New York with device type firewall**
```
Hostname: F
Location: New York
Type: firewall
```
## Search Options
Run as background job
: Enabled by default. Runs searches asynchronously, allowing you to track progress and cancel operations. Recommended for most use cases, especially with Virtual Chassis detection or large device sets. See [Background Jobs & Caching](background_jobs.md) for details.
Clear cache before search
: Forces fresh data from LibreNMS instead of using cached results. LibreNMS data is normally cached for 5 minutes to improve performance. See [Background Jobs & Caching](background_jobs.md) for caching details.
## Saved Cached Searches
The import page displays all your recent searches at the top, showing which filter combinations, that are still found in the cache. Each cached search shows the filters used, device count, and time remaining before expiration. Click any cached search to instantly reload those results without re-running filters. This is particularly useful when switching between different filter combinations.
## Next Steps
After searching, proceed to:
- [Validation & Configuration](validation.md) - Review and configure devices for import
- [Background Jobs & Caching](background_jobs.md) - Understand job processing and performance optimization

View File

@@ -0,0 +1,51 @@
# Validation & Configuration
After searching, the import table displays devices with action buttons that reflect their validation status.
## Validation States
Import Button (Green)
: Device is ready to import. All required fields are matched or configured.
Disabled Import Button + Details Button (Gray/Red)
: Device has missing required fields. Click Details to configure.
Link to Existing Device
: Device already exists in NetBox. Link navigates to the existing device.
## Required Fields
NetBox requires three fields before importing a device: **Site**, **Device Type**, and **Device Role**. The plugin attempts to match Site and Device Type automatically by comparing LibreNMS data to existing NetBox objects. Device Role must always be selected manually.
Click the validation details button to review what's missing and select values from the dropdowns. The validation status updates immediately.
### Import as Device
- **Site** (required) - Auto-matched from LibreNMS location
- **Device Type** (required) - Auto-matched from LibreNMS hardware string
- **Device Role** (required) - Must be selected manually
- **Platform** (optional) - Auto-matched from LibreNMS OS
- **Rack** (optional) - Available if Site has racks
### Import as Virtual Machine
- **Cluster** (required) - Must be selected manually
- **Platform** (optional) - Auto-matched from LibreNMS OS
## Virtual Chassis Detection
When Virtual Chassis Detection is enabled during search, the validation details show detected stack members with their positions, serials, and suggested names. The plugin automatically creates the Virtual Chassis object during import. See [Virtual Chassis](../usage_tips/virtual_chassis.md) for details.
## Duplicate Detection
The plugin checks for existing devices using:
1. **LibreNMS ID custom field** (most reliable) - If set, device is marked "Already Exists"
2. **Hostname match** - Exact name match against Devices and VMs
3. **Primary IP address** (weak match) - If IP is already assigned to a device
If both a VM and Device with the same hostname exist, the plugin cannot determine which to match and allows import. Set the `librenms_id` custom field on the correct existing object to clarify the match.
## Next Steps
- [Import Settings](import_settings.md) - Configure device naming and import options

94
docs/usage_tips/README.md Normal file
View File

@@ -0,0 +1,94 @@
# Usage Tips
## Initial Setup
1. [Configure Custom Field](custom_field.md)
- Set up the `librenms_id` custom field for optimal device matching
- This ensures reliable device identification between NetBox and LibreNMS
2. [Configure Interface Mappings](interface_mappings.md)
- Review and set up interface type mappings before synchronization
- Create specific mappings for your network equipment types
- Pay attention to speed-based mappings for accurate interface types
3. [Multi Server Configuration](multi_server_configuration.md)
- Configure multiple LibreNMS instances in your NetBox configuration
- Switch between different LibreNMS servers through the web interface
- Maintain backward compatibility with single-server configurations
## Device Import
[Device Import Guide](../librenms_import/overview.md) - Import devices from LibreNMS into NetBox
1. Search for devices using flexible filters (location, type, OS, hostname, sysname)
2. Validate import prerequisites (Site, Device Type, Device Role)
3. Configure missing mappings or select from suggestions
4. Import devices individually or in bulk
5. Automatic Virtual Chassis creation for stackable switches
The Device Import feature automatically sets the `librenms_id` custom field, enabling all other plugin features.
> **Rack Position Assignment:** Imported devices can be assigned to racks without specific rack unit (U) positions. After import, assign U positions through the "Non racked" section of each rack. The [NetBox Reorder Rack plugin](https://github.com/minitriga/netbox-reorder-rack) simplifies this workflow.
## Device Synchronization
### Devices
> **Note:** If you imported devices using the [Device Import feature](../librenms_import/overview.md), the `librenms_id` is already set and will be used automatically. The steps below apply to devices added to NetBox manually.
1. Ensure devices have either:
- Primary IP configured
- Valid DNS name (set on the Primary IP)
- hostname (that matches LibreNMS hostname)
2. The plugin will populate the `librenms_id` custom field if the device is found in LibreNMS
### Virtual Chassis
LibreNMS treats a Virtual Chassis as one logical device. The plugin selects a single "sync device" from your chassis to communicate with LibreNMS using this priority:
1. **Member with `librenms_id` set** (if already configured)
2. **Master device with primary IP** (most common)
3. **Any member with primary IP** (fallback)
4. **Member with lowest position number** (last resort)
Only the selected sync device should have the `librenms_id` custom field populated—leave it empty on all other members.
For best results, align chassis member positions with interface naming patterns. For example, if switch 1 has interfaces like `eth1/0/1` and switch 2 has `eth2/0/1`, the plugin can auto-detect the correct member for each interface. Always verify the member selection before running bulk synchronization.
## Interface Management
1. Verify Before Sync
- Review interface mappings indicated by the icons (🔗 shows a mapping is configured)
- Check speed and type matches
- Confirm member assignments for virtual chassis
2. Exclude columns to exclude from interface sync
- Sync only the values you want to sync
3. Sync VLANs first to ensure that VLANs are created in NetBox before syncing interfaces, allowing for proper VLAN assignments. Use the VLAN tab on the device sync page to create VLANs from LibreNMS data.
## Cable Management
1. Preparation
- Ensure devices are properly identified in both systems
- Open LibreNMS Sync on all devices to populate librenms_id custom field
- Remote Device and Remote interface need to be found in NetBox for cable creation to work
- Check Device and Interface naming
## VLAN Management
1. Preparation
- Configure VLAN Groups in NetBox if you want scoped VLAN assignment (e.g., per-site or per-rack groups)
- The plugin resolves VLAN groups using a scope hierarchy: Rack → Location → Site → SiteGroup → Region → Global
2. Review the VLANs tab on the device sync page
- Select the appropriate VLAN Group for each VLAN, or let the plugin auto-select based on scope
- A warning icon appears when a VID does not exist in the selected VLAN group.
3. Sync selected VLANs to create or update them in NetBox
## Best Practices
1. Regular Maintenance
- Periodically review and update interface mappings
- Keep custom fields current
## Optimization
- DNS lookup time can slow response of the API call to LibreNMS

View File

@@ -0,0 +1,97 @@
# Using the `librenms_id` Custom Field
## Overview
To enhance device identification and synchronization between NetBox and LibreNMS, this plugin supports using a custom field `librenms_id` on Device, Virtual Machine and Interface objects. While the plugin works without it, using this custom field is recommended for LibreNMS API lookups, and to assist with matching the remote device and remote interfaces for cable creation in Netbox. It can also be entered manually if no primary IP or FQDN is available.
!!! info "Automatic Creation"
As of version 0.4.4, the plugin **automatically creates** the `librenms_id` custom field when migrations are run. You no longer need to create it manually. The field is created for Device, Virtual Machine, Interface, and VM Interface objects with JSON type for per-server device tracking.
For the Device and Virtual Machine objects the plugin will automatically populate the LibreNMS ID custom field when opening the LibreNMS Sync page if the device has been found in LibreNMS.
For the Interface object, the plugin will automatically populate the LibreNMS ID custom field when the interface data is synced from LibreNMS.
## Benefits of Using `librenms_id`
- **Improved Device Matching:** Ensures accurate matching between NetBox and LibreNMS devices.
- **Fallback Identification:** Useful when devices lack a primary IP or FQDN.
- **Efficient Synchronization:** Enhances the reliability of API lookups.
- **Cable creation:** Allows better device identification for the creation of cables between NetBox devices.
## Manual Custom Field Setup
!!! note
On 0.4.4+, rerun migrations first (`manage.py migrate`). If you need to recreate the field manually on current releases, use the JSON schema below. Pre-0.4.2 releases used an Integer field — do not use Integer for new entries.
Follow these steps to create the `librenms_id` custom field in NetBox:
1. **Navigate to Custom Fields:**
- Go to **Customization** in the NetBox sidebar.
- Click on **Custom Fields**.
2. **Add a New Custom Field:**
- Click the **Add a custom field** button.
3. **Configure the Custom Field:**
- **Object Types:**
- Check **dcim > device**
- Check **virtualization > virtual machine**
- Check **dcim > interface**
- Check **virtualization > interfaces (optional)**
- **Name:** `librenms_id`
- **Label:** `LibreNMS ID`
- **Description:** (Optional) Add a description like "LibreNMS Device ID for synchronization".
- **Type:** JSON (object) — stores a per-server mapping.
- Multi-server example:
```json
{"production": 42, "staging": 17}
```
- Legacy single-server example (integer) — read-only/deprecated; do not use for new entries:
```
42
```
> Note: to create new entries manually use the JSON format shown above.
- **Required:** Leave unchecked (optional).
- **Default Value:** Leave blank.
4. **Save the Custom Field:**
- Click **Create** to save the custom field.
### Manually assign a value to `librenms_id`
You can manually assign a value to the `librenms_id` custom field for a device using the following steps:
1. **Edit the Device:**
- Navigate to the device in NetBox.
- Click the **Edit** button.
2. **Set the LibreNMS ID:**
- Scroll to the **Custom Fields** section.
- Enter the `librenms_id` value as a JSON object with your server key(s):
```json
{"production": 42}
```
For multiple servers: `{"production": 42, "staging": 17}`
3. **Save Changes:**
- Click **Update** to save the device.
## Notes
- If `librenms_id` is set, the plugin will prioritize it over other identification methods.
- Ensure the `librenms_id` corresponds to the correct device ID in LibreNMS to prevent mismatches.
- The custom field is optional but recommended for optimal plugin performance.
- Using the custom field on interfaces will greatly improve the interface matching required for cable synchronization.

View File

@@ -0,0 +1,137 @@
# interface\_mappings
### Quick Intro
Interface type mappings control how LibreNMS interface types are translated to NetBox interface types during synchronization.
The mappings can be customized in the plugin settings menu.
A mapping of LibreNMS Type an LibreNMS Speed combine to make a unique group that map to a Netbox interface type. This means multiple mapping for the same LibreNMS Type can be created.
> Note: The LibreNMS Speed is entered as Kbps
Example:
```
* ethernetCsmacd + 10000000 = 10GBASE-T (10GE)
* ethernetCsmacd + 1000000 = 1000BASE-T (1GE)
* ethernetCsmacd + 100000 = 100BASE-TX (10/100ME)
```
### How to Use Interface Mappings
#### Accessing the Page:
![Interface Mappings Page](../img/interface_mappings/interfacemappings_menu.png){ width="250" }
* From the main menu, navigate to the Plugins section
* Under Netbox Librenms Plugin, Select "Interface Mappings"
#### Creating a New Mapping:
![](../img/interface_mappings/addmapping.png){ width="50" }
* Click the green `+` or `Add` button either from the menu or on the Interface Mappings page
* Enter LibreNMS interface type. _You can copy this from plugin's device interface sync page_
* Enter Librenms interface speed as Kbps
* Select the Netbox interface type from the dropdown
* Click `Create` to save the mapping
#### Bulk Importing Mappings:
The plugin supports NetBox's standard bulk import feature for interface mappings. Click the **Import** button on the Interface Mappings page to access the import interface.
**YAML Example:**
```yaml
---
- librenms_type: ethernetCsmacd
librenms_speed: 1000000
netbox_type: 1000base-t
description: "Standard Gigabit Ethernet ports"
- librenms_type: propVirtual
librenms_speed: 1000000
netbox_type: virtual
description: "Virtual interfaces with 1G speed"
- librenms_type: softwareLoopback
librenms_speed: 8000000
netbox_type: virtual
description: "Loopback interfaces"
- librenms_type: ethernetCsmacd
librenms_speed: 10000000
netbox_type: 10gbase-t
description: "10 Gigabit Ethernet copper connections"
- librenms_type: ethernetCsmacd
librenms_speed: 100000
netbox_type: 100base-tx
description: "Fast Ethernet 100Mbps ports"
- librenms_type: ethernetCsmacd
librenms_speed: null
netbox_type: 1000base-t
description: "Default mapping for Ethernet without speed detection"
- librenms_type: ethernetCsmacd
librenms_speed: 40000000
netbox_type: 40gbase-x-qsfpp
description: "40 Gigabit QSFP+ interfaces"
- librenms_type: ethernetCsmacd
librenms_speed: 25000000
netbox_type: 25gbase-x-sfp28
description: "25 Gigabit SFP28 interfaces"
- librenms_type: propVirtual
librenms_speed: null
netbox_type: virtual
description: "Generic virtual interfaces"
- librenms_type: ieee8023adLag
librenms_speed: null
netbox_type: lag
description: "Link aggregation groups (port channels)"
- librenms_type: softwareLoopback
librenms_speed: null
netbox_type: virtual
description: "Software loopback interfaces"
```
**Notes:**
* `librenms_speed` is optional - use `null` or omit for type-only mappings
* `description` is optional - provides context for each mapping
* The combination of `librenms_type` and `librenms_speed` must be unique
* Supports CSV, JSON, and YAML formats
#### Editing Existing Mappings:
![](../img/interface_mappings/editmapping.png){ width="50" }
* On the Mappings page, Locate the desired mapping in the list
* Click the `edit` (pencil icon) button
* Modify the field mappings as needed
* Save the changes
#### Deleting Mappings:
![](../img/interface_mappings/deletemapping.png){ width="150" }
* Find the mapping you wish to remove
* Select the `Delete` button from the drop down
* Confirm the deletion when prompted
#### Applying Mappings:
* Mappings are automatically applied when interface data is synced between LibreNMS and Netbox
* If a mapping exist for an interface, it will show on the interface sync page with the icon :material-link-variant:
* If a mapping does not exist, it will show the icon :material-link-variant-off:
### Best Practices
* Check mappings are correct before performing a sync to avoid data errors
* Regularly review and update your mappings to ensure they remain accurate

View File

@@ -0,0 +1,92 @@
# Multi-Server LibreNMS Configuration
## Overview
The NetBox LibreNMS plugin now supports multiple LibreNMS servers. This allows you to:
- Configure multiple LibreNMS instances in your NetBox configuration
- Switch between different LibreNMS servers through the web interface
- Maintain backward compatibility with single-server configurations
## Configuration
### Multi-Server Configuration
Update your NetBox `configuration.py` file:
```python
PLUGINS_CONFIG = {
'netbox_librenms_plugin': {
'servers': {
'production': {
'display_name': 'Production LibreNMS',
'librenms_url': 'https://librenms-prod.example.com',
'api_token': 'your_production_token',
'cache_timeout': 300,
'verify_ssl': True,
'interface_name_field': 'ifDescr'
},
'testing': {
'display_name': 'Test LibreNMS',
'librenms_url': 'https://librenms-test.example.com',
'api_token': 'your_test_token',
'cache_timeout': 300,
'verify_ssl': False,
'interface_name_field': 'ifName'
},
'development': {
'display_name': 'Dev LibreNMS',
'librenms_url': 'https://librenms-dev.example.com',
'api_token': 'your_dev_token',
'cache_timeout': 180,
'verify_ssl': False,
'interface_name_field': 'ifDescr'
}
}
}
}
```
### Legacy Single-Server Configuration (Backward Compatible)
The original configuration format is still supported:
```python
PLUGINS_CONFIG = {
'netbox_librenms_plugin': {
'librenms_url': 'https://your-librenms-instance.com',
'api_token': 'your_librenms_api_token',
'cache_timeout': 300,
'verify_ssl': True,
'interface_name_field': 'ifDescr'
}
}
```
## Usage
1. Navigate to **LibreNMS Plugin** > **Settings** > **Server Settings**
2. Select your desired LibreNMS server from the dropdown
3. Click **Save Settings**
All subsequent LibreNMS operations will use the selected server.
## Configuration Options
Each server configuration supports the following options:
- `display_name`: Human-readable name for the server (optional)
- `librenms_url`: URL of the LibreNMS instance (required)
- `api_token`: API token for authentication (required)
- `cache_timeout`: Cache timeout in seconds (optional, default: 300)
- `verify_ssl`: Whether to verify SSL certificates (optional, default: True)
- `interface_name_field`: LibreNMS field for interface names (optional, default: 'ifDescr')
## Migration from Single to Multi-Server
1. Add the `servers` configuration block to your `configuration.py`
2. Move your existing single-server configuration into a server block (e.g., 'default' or 'production')
3. Restart NetBox
4. Select your server in the plugin settings
The plugin will automatically detect and use the new configuration format.

View File

@@ -0,0 +1,153 @@
# Permissions & Access Control
## Overview
The plugin uses a two-tier permission system that works with NetBox's built-in permissions. Both tiers must be satisfied for users to perform actions:
1. **Plugin permissions** control access to plugin pages and features
2. **NetBox object permissions** control what objects users can create or modify
This design ensures the plugin respects your existing NetBox permission structure. A user might have full plugin access but still be restricted to creating certain objects based on their NetBox permissions.
> Superusers have full access to all plugin features and NetBox objects by default. So the following applies only to regular users who can be granted specific permissions as needed.
## Two-Tier Permission Model
### How Do the Tiers Work Together?
A user needs both tiers of permissions to complete an action. For example, to view the Librenms Import page AND import a device:
1. **Tier 1: Plugin permission**: User needs View AND Change permission on **LibreNMS Settings**
- View: allows access to the plugin pages and pulling data from LibreNMS.
- Change: allows performing actions that modify Netbox or Librenms data
The Plugin also enforces Netbox object permissions so the following permission would also be required:
2. **Tier 2: Object permission**: User needs `dcim.add_device` (to create the device in NetBox)
If either permission is missing, the operation fails with an appropriate error message.
## Creating Permissions
All permissions are created using Netbox's standard Object permissions UI.
For details on how NetBox permissions work, see the [NetBox Permissions documentation](https://netboxlabs.com/docs/netbox/administration/permissions/).
### Plugin Permissions
To grant a user or group access to the plugin:
1. Go to **Admin → Permissions**
2. Click **Add**
3. For **Object types**, select "NetBox Librenms Plugin | LibreNMS Settings"
4. Under **Actions**, check the permissions to grant:
-**Can view** — for read-only access
-**Can change** — for write access (requires View as well)
5. Assign to specific **Users** or **Groups**
6. Click **Save**
### NetBox Object Permissions
NetBox object permissions are created similarly but for different object types (DCIM, IPAM, VIRTUALIZATION, etc.).
### Interface Type Mapping Permissions
The Interface Type Mapping feature uses its own object permissions in addition to the plugin permissions. To manage interface mappings, users need:
- **Plugin permission**: View permission on LibreNMS Settings (to access the page)
- **Object permissions**: `netbox_librenms_plugin.add_interfacetypemapping`, `netbox_librenms_plugin.change_interfacetypemapping`, or `netbox_librenms_plugin.delete_interfacetypemapping` as needed
These permissions are enforced automatically by NetBox's generic views.
## Example Scenarios
### Read-Only Access
- **Plugin permissions**: Librenms Setting View only (Can view LibreNMS Plugin pages)
- **NetBox permissions**: View permissions for devices, interfaces, etc.
Users can access all plugin pages, refresh data from LibreNMS, and review comparison tables, but cannot import devices or sync data.
### Full Plugin Access
- **Plugin permissions**: View + Change (Can view LibreNMS Plugin Pages and Import and sync devices data)
- **NetBox permissions**: Add/change permissions for devices, interfaces, cables, IP addresses, VLANs
Users have full access to all plugin features and can import devices, sync interfaces, and create cables.
## Further Details
### Tier 1: Plugin Permissions
Plugins permissions use the **LibreNMS Settings** model permissions:
| Permission | NetBox UI Selection | Grants |
|------------|---------------------|--------|
| `view_librenmssettings` | LibreNMS Settings → ☑ Can view | Access all plugin pages, view LibreNMS data |
| `change_librenmssettings` | LibreNMS Settings → ☑ Can change | Import devices, sync data, save settings |
Users without View permission won't see the LibreNMS menu or the LibreNMS Sync tab. Users with **View** but not **Change** can browse all plugin pages but cannot perform import or sync actions that modify Netbox data and Librenms data like Locations and Adding devices.
### Tier 2: NetBox Object Permissions
When the plugin creates or modifies NetBox objects (devices, interfaces, cables, IP addresses, VLANs), NetBox enforces its standard object permissions. The plugin checks these permissions and will block operations if the user lacks the required access.
| Plugin Action | Required Object Permissions |
|---------------|----------------------------|
| Import device | `dcim.add_device`, `dcim.add_interface` |
| Import device with VC | Above + `dcim.add_virtualchassis` |
| Import VM | `virtualization.add_virtualmachine` |
| Sync interfaces | `dcim.add_interface`, `dcim.change_interface` |
| Delete interfaces | `dcim.delete_interface` |
| Sync VM interfaces | `virtualization.add_vminterface`, `virtualization.change_vminterface` |
| Delete VM interfaces | `virtualization.delete_vminterface` |
| Sync cables | `dcim.add_cable`, `dcim.change_cable` |
| Sync IP addresses | `ipam.add_ipaddress`, `ipam.change_ipaddress` |
| Sync VLANs | `ipam.add_vlan`, `ipam.change_vlan` |
| Sync device fields | `dcim.change_device` |
| Create platform | `dcim.add_platform` |
### Why LibreNMS Settings Permissions?
NetBox's permission system is object-based—permissions are tied to specific models like Device, Interface, or Cable. However, the plugin's Import and Sync pages are feature pages that don't have their own dedicated models. They work with LibreNMS data and create or modify existing NetBox objects.
To control access to these pages, the plugin uses the **LibreNMS Settings** model permissions as a gate for all plugin features:
- **No dedicated models for pages** — The Import and Sync pages aren't objects, so we need an existing model to attach permissions to
- **No custom migrations required** — Uses Django's built-in model permissions that NetBox already understands
- **Standard NetBox workflow** — Administrators assign permissions the same way they do for any other NetBox object
- **Single permission per access level** — One "View" permission for read access, one "Change" permission for write access
While using a settings model for access control may seem unconventional, it provides a simple and maintainable way to gate plugin access without introducing custom permission infrastructure.
## Special note: Background Jobs and Superuser Access
The device import page can use background jobs to help support large device sets, and virtual chassis detection. However, NetBox restricts access to background job status APIs to superusers. There is no permission in NetBox for this. This is a core design decision, not a plugin limitation.
| User Type | Background Jobs |
|-----------|-----------------|
| Superuser | Full access to background jobs with real-time status updates |
| Non-superuser | Automatic fallback to synchronous processing |
The plugin automatically detects whether the current user is a superuser and adjusts behavior accordingly. Non-superuser users don't need to change any settings—the plugin simply processes requests synchronously instead of as background jobs. All import and filter operations work correctly regardless of superuser status.
## Troubleshooting
**User can't see the LibreNMS menu**
: The user doesn't have View permission for the plugin. Add an Object Permission for "LibreNMS Settings" with "Can view" checked.
**User sees pages but can't import or sync**
: The user has View permission but not Change permission. Edit their Object Permission to also include "Can change".
**User gets "permission denied" when importing devices**
: The user has plugin permissions but may be missing NetBox object permissions. Check that they have `dcim.add_device` and related permissions.
**Background jobs show 403 errors in console**
: In normal usage, the UI only enables background jobs for users allowed to use them and falls back to synchronous processing otherwise, so 403s from background job APIs should not appear. If you see these errors, it usually means a direct API call or custom integration is hitting background-task endpoints without superuser access; update that integration to use synchronous flows or run it with appropriate permissions.

View File

@@ -0,0 +1,88 @@
# Suggested Workflow
This guide provides a recommended workflow for using the plugin after installation. Following this order helps ensure smooth operation and avoids common issues.
## 1. Configure Plugin Settings
Navigate to **Plugins → LibreNMS Plugin → Settings** and configure:
- **LibreNMS Server**: Select which server to use (if multi-server setup)
- **Device Naming**: Set your preferred defaults for "Use sysName" and "Strip Domain" - see [Import Settings](../librenms_import/import_settings.md)
- **Virtual Chassis Naming**: Configure the member naming pattern if you plan to import stackable devices
**Why first**: These defaults apply to all imports and save time by reducing per-import configuration.
## 2. Verify Custom Field
As of version 0.4.4, the plugin **automatically creates** the `librenms_id` custom field when migrations are run. No manual setup is required. See the [Custom Field Setup](custom_field.md) guide for details on how the field works and optional manual configuration.
**Why early**: This field enables the most reliable device matching and is required for interface, cable, and IP address synchronization features. It is created automatically during `manage.py migrate`, so just verify it exists before importing.
## 3. Prepare NetBox Data
Ensure NetBox has the basic objects needed for device imports:
- **Sites**: Create Sites that match your LibreNMS locations (exact name matching works best)
- **Device Types**: Add Device Types for your common hardware models
- **Device Roles**: Create appropriate roles (Switch, Router, Firewall, etc.)
- **Platforms**: Add Platforms matching your LibreNMS OS names (optional but helpful)
**Why before importing**: The plugin auto-matches these objects during import. Pre-creating them reduces manual configuration during the import process.
## 4. Configure Interface Mappings
If you have specific interface type mapping requirements, configure them via **Plugins → LibreNMS Plugin → Interface Type Mappings** - see [Interface Mappings](interface_mappings.md).
**Why**: Ensure specific NetBox interface types are used for your LibreNMS interface data.
## 5. Import Devices
Use the [Device Import](../librenms_import/overview.md) feature to bring devices into NetBox:
1. Navigate to **Plugins → LibreNMS Plugin → Import → LibreNMS Import**
2. Apply filters to find devices (start with Location or Type)
3. Review validation status and configure missing fields
4. Import devices individually or in bulk
**Tips**:
- Start with a small set (single location or device type) to verify your setup
- Enable Virtual Chassis detection only when importing stackable switches
## 6. Sync VLAN
- Create VLAN objects in NetBox from LibreNMS device VLAN data
- Per-VLAN group assignment with scope-aware auto-selection
## 7. Sync Interfaces
After devices are imported, sync their interfaces:
1. Navigate to a device in NetBox
2. Use the LibreNMS sync button to pull interface data
3. Review and adjust [Interface Mappings](interface_mappings.md) if needed
**Why after import**: Interfaces require the device to exist in NetBox first. The `librenms_id` field set during import enables accurate synchronization.
## 8. Sync Cables and IP Addresses
Complete your device data by syncing:
- **Cables**: Pull link data from LibreNMS to create cable connections
- **IP Addresses**: Import IP assignments to populate NetBox's IPAM
**Why last**: Both features require that interfaces already exist in NetBox and ideally with the `librenms_id` field set. The `librenms_id` field on interfaces ensures accurate matching.
## 9. Sync Locations (Optional)
If you want to synchronize location latitude/longitude data between NetBox Sites and LibreNMS locations, use the location sync feature.
**Why optional**: Only needed if you maintain geographic coordinates and want bidirectional sync.
## Next Steps
After completing the initial workflow:
- Regular imports: Use the same import process for new devices as they're added to LibreNMS
- Interface updates: Re-sync interfaces periodically to capture configuration changes
- Virtual Chassis: See [Virtual Chassis](virtual_chassis.md) for managing multi-member devices
- Background Jobs: Understand [Background Jobs & Caching](../librenms_import/background_jobs.md) for performance optimization

View File

@@ -0,0 +1,31 @@
# Virtual Chassis Support
## Overview
The plugin automatically detects Virtual Chassis configurations and displays all VC interfaces on the LibreNMS Sync page of the designated sync device.
**LibreNMS Sync Device Selection Priority:**
1. Member with `librenms_id` custom field (highest priority)
2. Master device with primary IP
3. Any member with primary IP
4. Member with lowest VC position
> **Note:** LibreNMS treats a Virtual Chassis as a single logical device. Only one member (the sync device) should have the `librenms_id` custom field set.
## How It Works
### Member Selection
When viewing a device that is part of a virtual chassis, the plugin will:
1. Detects if the device is part of a virtual chassis and displays 'Virtual Chassis Member' column.
2. Automatically select the VC member by matching the device VC position to the first number in the interface name.
3. Allows selection of specific members if the auto select is not correct.
> Selecting a new member will trigger a new interface details comparison against the newly selected NetBox VC member.
Interfaces data is then synced to the selected VC member in Netbox.
#### Virtual Chassis Member Select
![Virtual Chassis Member Selection](../img/Netbox-librenms-plugin-virtualchassis.gif)

View File

@@ -0,0 +1,52 @@
###################################################################
# This file serves as a base configuration for testing purposes #
# only. It is not intended for production use. #
###################################################################
ALLOWED_HOSTS = ["*"]
DATABASE = {
"NAME": "netbox",
"USER": "netbox",
"PASSWORD": "netbox",
"HOST": "localhost",
"PORT": "",
"CONN_MAX_AGE": 300,
}
PLUGINS = [
"netbox_librenms_plugin",
]
PLUGINS_CONFIG = {
"netbox_librenms_plugin": {
"servers": {
"default": {
"librenms_url": "https://librenms.example.com",
"api_token": "test-token-for-testing",
}
}
}
}
REDIS = {
"tasks": {
"HOST": "localhost",
"PORT": 6379,
"PASSWORD": "",
"DATABASE": 0,
"SSL": False,
},
"caching": {
"HOST": "localhost",
"PORT": 6379,
"PASSWORD": "",
"DATABASE": 1,
"SSL": False,
},
}
SECRET_KEY = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
API_TOKEN_PEPPERS = {
1: "TEST-VALUE-DO-NOT-USE-TEST-VALUE-DO-NOT-USE-TEST-VALUE-DO-NOT-USE",
}

88
mkdocs.yml Normal file
View File

@@ -0,0 +1,88 @@
site_name: NetBox librenms Plugin
site_url: https://bonzo81.github.io/netbox-librenms-plugin
repo_url: https://github.com/bonzo81/netbox-librenms-plugin
repo_name: /netbox-librenms-plugin
#strict: true
nav:
- Home: README.md
- Getting Started:
- Feature Overview: feature_list.md
- Initial Setup: usage_tips/README.md
- Custom Field Setup: usage_tips/custom_field.md
- Multi-Server Setup: usage_tips/multi_server_configuration.md
- Permissions & Access: usage_tips/permissions.md
- Suggested Workflow: usage_tips/suggested_workflow.md
- Import Devices:
- Overview: librenms_import/overview.md
- Searching for Devices: librenms_import/search.md
- Validation & Configuration: librenms_import/validation.md
- Import Settings: librenms_import/import_settings.md
- Background Jobs & Caching: librenms_import/background_jobs.md
- Sync & Configuration:
- Virtual Chassis: usage_tips/virtual_chassis.md
- Interface Mappings: usage_tips/interface_mappings.md
- Development:
- Overview: development/README.md
- Project Structure: development/structure.md
- Views & Inheritance: development/views.md
- Mixins: development/mixins.md
- Templates: development/templates.md
- Testing: development/testing.md
- Changelog: changelog.md
- Contributing: contributing.md
theme:
name: material
language: en
#logo: assets/logo.png
palette:
scheme: preference
primary: indigo
accent: indigo
features:
- navigation.indexes
- navigation.instant
- navigation.tabs.sticky
markdown_extensions:
- attr_list
- pymdownx.emoji:
emoji_index: !!python/name:material.extensions.emoji.twemoji
emoji_generator: !!python/name:material.extensions.emoji.to_svg
- pymdownx.critic
- pymdownx.caret
- pymdownx.mark
- pymdownx.tilde
- pymdownx.tabbed
- attr_list
- pymdownx.arithmatex:
generic: true
- pymdownx.highlight:
linenums: false
- pymdownx.superfences
- pymdownx.inlinehilite
- pymdownx.details
- admonition
- pymdownx.escapeall:
hardbreak: true
nbsp: true
- toc:
baselevel: 2
permalink: true
- meta
plugins:
- include-markdown
- search:
lang: en
- mkdocstrings:
extra:
social:
- icon: fontawesome/brands/github
link: https://github.com//netbox-librenms-plugin
name: Github
# to enable disqus, uncomment the following and put your disqus id below
# disqus: disqus_id
# uncomment the following and put your google tracking id below to enable GA
#google_analytics:
#- UA-xxx
#- auto

View File

@@ -0,0 +1,142 @@
from django.core.exceptions import ImproperlyConfigured
from netbox.plugins import PluginConfig
__author__ = "Andy Norwood"
__version__ = "0.4.6"
class LibreNMSSyncConfig(PluginConfig):
name = "netbox_librenms_plugin"
verbose_name = "NetBox Librenms Plugin"
description = "Netbox plugin to sync data between LibreNMS and Netbox."
author = __author__
version = __version__
base_url = "librenms_plugin"
min_version = "4.2.0"
required_settings = [] # Custom validation in ready() method
default_settings = {
"enable_caching": True,
"verify_ssl": True,
"interface_name_field": "ifName",
}
def ready(self):
"""
Perform custom validation for plugin configuration.
Supports both legacy single-server and new multi-server configurations.
"""
super().ready()
from django.conf import settings
from django.db.models.signals import post_migrate
plugin_config = getattr(settings, "PLUGINS_CONFIG", {}).get(self.name, {})
# Check if using new multi-server configuration
if "servers" in plugin_config:
self._validate_multi_server_config(plugin_config["servers"])
else:
self._validate_legacy_config(plugin_config)
# Auto-create the librenms_id custom field after migrations complete
post_migrate.connect(
_ensure_librenms_id_custom_field,
dispatch_uid="netbox_librenms_plugin_ensure_cf",
)
def _validate_multi_server_config(self, servers_config):
"""Validate multi-server configuration."""
if not servers_config or not isinstance(servers_config, dict):
raise ImproperlyConfigured(
f"Plugin {self.name} requires at least one server configuration in the 'servers' section."
)
for server_key, server_config in servers_config.items():
if not isinstance(server_config, dict):
raise ImproperlyConfigured(f"Plugin {self.name} server '{server_key}' must be a dictionary.")
for setting in ["librenms_url", "api_token"]:
if setting not in server_config:
raise ImproperlyConfigured(f"Plugin {self.name} server '{server_key}' requires '{setting}'.")
def _validate_legacy_config(self, plugin_config):
"""Validate legacy single-server configuration."""
for setting in ["librenms_url", "api_token"]:
if setting not in plugin_config:
raise ImproperlyConfigured(
f"Plugin {self.name} requires either 'servers' configuration or legacy '{setting}' setting."
)
def _ensure_librenms_id_custom_field(sender, **kwargs):
"""
Auto-create (or migrate) the 'librenms_id' custom field.
Runs after migrations via post_migrate signal to ensure tables exist.
Uses dispatch_uid to avoid duplicate connections.
librenms_id stores a per-server JSON mapping {"server_key": device_id}.
Legacy installations may have this field typed as 'integer'; we upgrade it
to 'json' automatically so the UI and API accept the dict format.
"""
# Track per-alias execution so each database alias is bootstrapped exactly once.
db_alias = kwargs.get("using") or "default"
executed_aliases = getattr(_ensure_librenms_id_custom_field, "_executed_aliases", set())
if db_alias in executed_aliases:
return
import logging
try:
from django.contrib.contenttypes.models import ContentType
from extras.models import CustomField
cf, created = CustomField.objects.using(db_alias).get_or_create(
name="librenms_id",
defaults={
"type": "json",
"label": "LibreNMS ID",
"description": "LibreNMS Device ID for synchronization (auto-created by plugin)",
"required": False,
"ui_visible": "if-set",
"ui_editable": "yes",
"is_cloneable": False,
},
)
# Migrate legacy integer-typed field to JSON so the multi-server
# dict format {"server_key": device_id} is accepted by the UI/API.
if not created and cf.type == "integer":
cf.type = "json"
cf.save(using=db_alias, update_fields=["type"])
logging.getLogger("netbox_librenms_plugin").info(
"Migrated 'librenms_id' custom field type from integer to json"
)
# Ensure the field is assigned to the required object types
from dcim.models import Device, Interface
from virtualization.models import VirtualMachine, VMInterface
required_models = [Device, VirtualMachine, Interface, VMInterface]
current_types = set(cf.object_types.values_list("pk", flat=True))
for model in required_models:
ct = ContentType.objects.db_manager(db_alias).get_for_model(model)
if ct.pk not in current_types:
cf.object_types.add(ct)
if created:
logging.getLogger("netbox_librenms_plugin").info(
"Auto-created 'librenms_id' custom field for Device, VirtualMachine, Interface, VMInterface"
)
# Mark this alias as executed after successful completion to allow retry on failure.
executed_aliases.add(db_alias)
_ensure_librenms_id_custom_field._executed_aliases = executed_aliases
except Exception as e:
# Don't break startup if custom field creation fails (e.g., during initial migration),
# but log the error so it's not silently swallowed.
logging.getLogger("netbox_librenms_plugin").exception("Failed to auto-create 'librenms_id' custom field: %s", e)
config = LibreNMSSyncConfig

View File

View File

View File

@@ -0,0 +1,13 @@
from netbox.api.serializers import NetBoxModelSerializer
from netbox_librenms_plugin.models import InterfaceTypeMapping
class InterfaceTypeMappingSerializer(NetBoxModelSerializer):
"""Serialize InterfaceTypeMapping model for REST API."""
class Meta:
"""Meta options for InterfaceTypeMappingSerializer."""
model = InterfaceTypeMapping
fields = ["id", "librenms_type", "librenms_speed", "netbox_type", "description"]

View File

@@ -0,0 +1,13 @@
from django.urls import path
from netbox.api.routers import NetBoxRouter
from . import views
app_name = "netbox_librenms_plugin"
router = NetBoxRouter()
router.register("interface-type-mappings", views.InterfaceTypeMappingViewSet)
urlpatterns = [
path("jobs/<int:job_pk>/sync-status/", views.sync_job_status, name="sync_job_status"),
] + router.urls

View File

@@ -0,0 +1,106 @@
import logging
from core.choices import JobStatusChoices
from core.models import Job
from django.http import JsonResponse
from django.utils import timezone
from django_rq import get_queue
from netbox.api.viewsets import NetBoxModelViewSet
from rest_framework.decorators import api_view, permission_classes
from rest_framework.permissions import BasePermission, SAFE_METHODS
from rq.exceptions import NoSuchJobError
from rq.job import Job as RQJob
from netbox_librenms_plugin.constants import PERM_CHANGE_PLUGIN, PERM_VIEW_PLUGIN
from netbox_librenms_plugin.filters import InterfaceTypeMappingFilterSet
from netbox_librenms_plugin.jobs import FilterDevicesJob, ImportDevicesJob
from netbox_librenms_plugin.models import InterfaceTypeMapping
from .serializers import InterfaceTypeMappingSerializer
logger = logging.getLogger(__name__)
class LibreNMSPluginPermission(BasePermission):
"""
Permission class for LibreNMS plugin API endpoints.
- Safe requests (GET, HEAD, OPTIONS) require netbox_librenms_plugin.view_librenmssettings
- All other requests require netbox_librenms_plugin.change_librenmssettings
"""
def has_permission(self, request, view):
if request.method in SAFE_METHODS:
return request.user.has_perm(PERM_VIEW_PLUGIN)
return request.user.has_perm(PERM_CHANGE_PLUGIN)
class InterfaceTypeMappingViewSet(NetBoxModelViewSet):
"""API viewset for InterfaceTypeMapping CRUD operations."""
permission_classes = [LibreNMSPluginPermission]
filterset_class = InterfaceTypeMappingFilterSet
queryset = InterfaceTypeMapping.objects.all()
serializer_class = InterfaceTypeMappingSerializer
@api_view(["POST"])
@permission_classes([LibreNMSPluginPermission])
def sync_job_status(request, job_pk):
"""
Sync database Job status with RQ job status.
This is needed because NetBox's worker doesn't always update the database
when a job is stopped before it starts processing.
Only allows users to sync their own LibreNMS jobs.
Args:
request: Django request
job_pk: Primary key of the Job to sync
Returns:
JsonResponse with updated status
"""
_LIBRENMS_JOB_NAMES = (FilterDevicesJob.Meta.name, ImportDevicesJob.Meta.name)
try:
job = Job.objects.get(pk=job_pk, user=request.user, name__in=_LIBRENMS_JOB_NAMES)
except Job.DoesNotExist:
return JsonResponse({"error": "Job not found"}, status=404)
# Get RQ job status
queue = get_queue("default")
try:
rq_job = RQJob.fetch(str(job.job_id), connection=queue.connection)
rq_status = rq_job.get_status()
# If RQ job is stopped or failed, update database
if rq_job.is_stopped or rq_job.is_failed:
job.status = JobStatusChoices.STATUS_FAILED
if not job.completed:
job.completed = timezone.now()
job.save(update_fields=["status", "completed"])
logger.info("Synced Job #%s: DB status updated to failed (RQ: %s)", job.pk, rq_status)
return JsonResponse({"status": "updated", "db_status": job.status, "rq_status": rq_status})
else:
# Job still active in RQ
return JsonResponse({"status": "no_change", "db_status": job.status, "rq_status": rq_status})
except NoSuchJobError:
# Job not in RQ queue — mark any non-terminal DB job as failed
logger.warning("Job #%s not found in RQ (NoSuchJobError)", job.pk)
terminal_states = {
JobStatusChoices.STATUS_COMPLETED,
JobStatusChoices.STATUS_FAILED,
JobStatusChoices.STATUS_ERRORED,
}
if job.status not in terminal_states:
job.status = JobStatusChoices.STATUS_FAILED
if not job.completed:
job.completed = timezone.now()
job.save(update_fields=["status", "completed"])
return JsonResponse({"status": "updated", "db_status": job.status, "rq_status": "not_found"})
return JsonResponse({"status": "no_change", "db_status": job.status, "rq_status": "not_found"})
except Exception as e:
logger.exception("Unexpected error fetching RQ job for Job #%s: %s", job.pk, e)
return JsonResponse({"error": "Failed to fetch RQ job status"}, status=500)

View File

@@ -0,0 +1,6 @@
# Plugin permissions (from LibreNMSSettings model)
PERM_VIEW_PLUGIN = "netbox_librenms_plugin.view_librenmssettings"
PERM_CHANGE_PLUGIN = "netbox_librenms_plugin.change_librenmssettings"
# LibreNMS VLAN state values
LIBRENMS_VLAN_STATE_ACTIVE = 1

View File

@@ -0,0 +1,13 @@
import django_filters
from .models import InterfaceTypeMapping
class InterfaceTypeMappingFilterSet(django_filters.FilterSet):
"""Filter set for InterfaceTypeMapping model."""
class Meta:
"""Meta options for InterfaceTypeMappingFilterSet."""
model = InterfaceTypeMapping
fields = ["librenms_type", "librenms_speed", "netbox_type", "description"]

View File

@@ -0,0 +1,140 @@
import django_filters
from dcim.models import Device, DeviceRole, DeviceType, Platform, Site
from django import forms
from django.db.models import Q
from netbox.filtersets import NetBoxModelFilterSet
from virtualization.models import Cluster, VirtualMachine
class SiteLocationFilterSet:
"""
Filter sites and locations by search term.
"""
def __init__(self, data, queryset):
"""Initialize with form data and queryset."""
self.form_data = data
self.queryset = queryset
@property
def qs(self):
"""Return the filtered queryset."""
queryset = self.queryset
if q := self.form_data.get("q"):
return self._filter_queryset(q)
return queryset
def _filter_queryset(self, search_term):
"""Filter queryset by search term."""
search_term = str(search_term).lower()
return [item for item in self.queryset if self._matches_search_criteria(item, search_term)]
def _matches_search_criteria(self, item, search_term):
"""Check if item matches search criteria."""
searchable_fields = [
str(item.netbox_site.name),
str(item.netbox_site.latitude),
str(item.netbox_site.longitude),
str(item.librenms_location) if item.librenms_location else "",
]
return any(search_term in field.lower() for field in searchable_fields)
@property
def form(self):
"""Return a bound filter form instance."""
class FilterForm(forms.Form):
"""
Form to filter sites and locations by search term.
"""
q = forms.CharField(
required=False,
label="Search sites and locations",
widget=forms.TextInput(attrs={"placeholder": "Search by site name, coordinates or location"}),
)
return FilterForm(self.form_data)
class DeviceStatusFilterSet(NetBoxModelFilterSet):
"""
Filter devices by search term.
"""
device = django_filters.ModelMultipleChoiceFilter(
field_name="name",
queryset=Device.objects.all(),
)
site = django_filters.ModelMultipleChoiceFilter(
field_name="site",
queryset=Site.objects.all(),
)
device_type = django_filters.ModelMultipleChoiceFilter(
field_name="device_type",
queryset=DeviceType.objects.all(),
)
role = django_filters.ModelMultipleChoiceFilter(
field_name="role",
queryset=DeviceRole.objects.all(),
)
class Meta:
"""Meta options for DeviceStatusFilterSet."""
model = Device
fields = ["site", "location", "device_type", "rack", "role"]
search_fields = ["device", "site", "device_type", "rack", "role"]
def search(self, queryset, name, value):
"""Search devices by name, site, device type, rack or role."""
if not value.strip():
return queryset
return queryset.filter(
Q(name__icontains=value)
| Q(site__name__icontains=value)
| Q(device_type__model__icontains=value)
| Q(rack__name__icontains=value)
| Q(role__name__icontains=value)
)
class VMStatusFilterSet(NetBoxModelFilterSet):
"""
Filter virtual machines by search term.
"""
virtualmachine = django_filters.ModelMultipleChoiceFilter(
field_name="name",
queryset=VirtualMachine.objects.all(),
)
site = django_filters.ModelMultipleChoiceFilter(
field_name="site",
queryset=Site.objects.all(),
)
cluster = django_filters.ModelMultipleChoiceFilter(
field_name="cluster",
queryset=Cluster.objects.all(),
)
platform = django_filters.ModelMultipleChoiceFilter(
field_name="platform",
queryset=Platform.objects.all(),
)
class Meta:
"""Meta options for VMStatusFilterSet."""
model = VirtualMachine
fields = ["site", "cluster", "platform"]
search_fields = ["virtualmachine", "site", "cluster", "platform"]
def search(self, queryset, name, value):
"""Search VMs by name, site, cluster, or platform."""
if not value.strip():
return queryset
return queryset.filter(
Q(name__icontains=value)
| Q(site__name__icontains=value)
| Q(cluster__name__icontains=value)
| Q(platform__name__icontains=value)
)

View File

@@ -0,0 +1,787 @@
# forms.py
import logging
from dcim.choices import InterfaceTypeChoices
from dcim.models import Device, DeviceRole, DeviceType, Location, Rack, Site
from django import forms
from django.db.models import Case, IntegerField, Value, When
from django.http import QueryDict
from django.utils.translation import gettext_lazy as _
from netbox.forms import (
NetBoxModelFilterSetForm,
NetBoxModelForm,
NetBoxModelImportForm,
)
from netbox.plugins import get_plugin_config
from utilities.forms.fields import CSVChoiceField, DynamicModelMultipleChoiceField
from virtualization.models import Cluster, VirtualMachine
from .models import InterfaceTypeMapping, LibreNMSSettings
logger = logging.getLogger(__name__)
def _get_librenms_server_choices():
"""
Helper function to get server choices from plugin configuration.
Shared between ServerConfigForm and other forms that need server selection.
"""
choices = []
# Try to get multi-server configuration
servers_config = get_plugin_config("netbox_librenms_plugin", "servers")
if servers_config and isinstance(servers_config, dict):
# Multi-server configuration
for key, config in servers_config.items():
display_name = config.get("display_name", key)
url = config.get("librenms_url", "Unknown URL")
choices.append((key, f"{display_name} ({url})"))
else:
# Legacy single-server configuration
legacy_url = get_plugin_config("netbox_librenms_plugin", "librenms_url")
if legacy_url:
choices.append(("default", f"Default Server ({legacy_url})"))
else:
choices.append(("default", "Default Server"))
return choices
def _get_librenms_poller_group_choices():
"""
Helper function to get poller group choices from LibreNMS API.
Shared between AddToLIbreSNMPV1V2 and AddToLIbreSNMPV3 forms.
"""
from .librenms_api import LibreNMSAPI
choices = [("0", "Default (0)")]
try:
api = LibreNMSAPI()
success, poller_groups = api.get_poller_groups()
if success:
for group in poller_groups or []:
group_id = str(group.get("id", ""))
group_name = group.get("group_name", "")
group_descr = group.get("descr", "")
if group_id:
if group_descr and group_descr != group_name:
label = f"{group_name} - {group_descr} ({group_id})"
else:
label = f"{group_name} ({group_id})"
choices.append((group_id, label))
except Exception:
logger.exception("Failed to fetch LibreNMS poller groups; using default choices")
return choices
class ServerConfigForm(NetBoxModelForm):
"""
Form for selecting the active LibreNMS server from configured servers.
Handles server configuration changes only.
"""
selected_server = forms.ChoiceField(
label="LibreNMS Server",
help_text="Select which LibreNMS server to use for synchronization operations",
)
class Meta:
model = LibreNMSSettings
fields = ["selected_server"]
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields["selected_server"].choices = _get_librenms_server_choices()
class ImportSettingsForm(NetBoxModelForm):
"""
Form for configuring device import settings including naming patterns
and virtual chassis member naming.
"""
vc_member_name_pattern = forms.CharField(
label="Virtual Chassis Member Naming Pattern",
max_length=100,
required=False,
strip=False, # Preserve leading/trailing whitespace
widget=forms.TextInput(
attrs={
"placeholder": "-M{position}",
}
),
)
use_sysname_default = forms.BooleanField(
label="Use sysName",
required=False,
widget=forms.CheckboxInput(attrs={"class": "form-check-input"}),
help_text="Use SNMP sysName instead of LibreNMS hostname when importing devices",
)
strip_domain_default = forms.BooleanField(
label="Strip domain",
required=False,
widget=forms.CheckboxInput(attrs={"class": "form-check-input"}),
help_text="Remove domain suffix from device names during import",
)
class Meta:
model = LibreNMSSettings
fields = [
"vc_member_name_pattern",
"use_sysname_default",
"strip_domain_default",
]
def clean_vc_member_name_pattern(self):
"""
Validate VC member name pattern for valid placeholders and formatting.
The pattern is used as a suffix appended to the master device name.
Valid placeholders: {position}, {serial}
At least one is required for uniqueness.
"""
pattern = self.cleaned_data.get("vc_member_name_pattern")
if not pattern:
return pattern
# Check for valid placeholder names using regex
import re
valid_placeholders = {"position", "serial"}
found_placeholders = set(re.findall(r"\{(\w+)\}", pattern))
invalid_placeholders = found_placeholders - valid_placeholders
if invalid_placeholders:
invalid_list = ", ".join(f"{{{p}}}" for p in sorted(invalid_placeholders))
error_msg = f"Invalid placeholder(s): {invalid_list}. Valid options are: {{position}}, {{serial}}"
raise forms.ValidationError(error_msg)
# Check required: must have at least one unique identifier
if "{position}" not in pattern and "{serial}" not in pattern:
raise forms.ValidationError(
"The naming pattern must include either {{position}} or {{serial}} "
"placeholder to ensure unique member names."
)
# Test the pattern can be formatted without errors
test_vars = {
"position": 1,
"serial": "ABC123",
}
try:
test_result = pattern.format(**test_vars)
# Check result isn't empty or just whitespace
if not test_result.strip():
raise forms.ValidationError(
"The pattern results in an empty suffix. Please include some text content in the pattern."
)
except KeyError as e:
# This should be caught by check above, but just in case
raise forms.ValidationError(
f"Invalid placeholder in pattern: {e}. Valid options are: {{position}}, {{serial}}"
)
except (ValueError, IndexError) as e:
raise forms.ValidationError(f"Invalid pattern syntax: {str(e)}")
return pattern
# Keep for backward compatibility if needed elsewhere
class LibreNMSSettingsForm(ServerConfigForm):
"""
Deprecated: Use ServerConfigForm or ImportSettingsForm instead.
Kept for backward compatibility.
"""
pass
class InterfaceTypeMappingForm(NetBoxModelForm):
"""
Form for creating and editing interface type mappings between LibreNMS and NetBox.
Allows mapping of LibreNMS interface types and speeds to NetBox interface types.
"""
class Meta:
model = InterfaceTypeMapping
fields = ["librenms_type", "librenms_speed", "netbox_type", "description"]
class InterfaceTypeMappingImportForm(NetBoxModelImportForm):
"""
Form for bulk importing interface type mappings from CSV/JSON/YAML.
Supports importing LibreNMS interface type and speed mappings to NetBox interface types.
"""
netbox_type = CSVChoiceField(
label=_("NetBox Type"),
choices=InterfaceTypeChoices,
help_text=_("NetBox interface type"),
)
class Meta:
model = InterfaceTypeMapping
fields = ["librenms_type", "librenms_speed", "netbox_type", "description"]
class InterfaceTypeMappingFilterForm(NetBoxModelFilterSetForm):
"""
Form for filtering interface type mappings based on LibreNMS and NetBox attributes.
Provides filtering options for LibreNMS type, speed, and NetBox type.
"""
librenms_type = forms.CharField(required=False, label="LibreNMS Type")
librenms_speed = forms.IntegerField(
required=False,
label="LibreNMS Speed (Kbps)",
help_text="Filter by interface speed in Kbps",
)
netbox_type = forms.ChoiceField(
required=False,
label="NetBox Type",
choices=[("", "---------")] + list(InterfaceTypeChoices),
)
description = forms.CharField(
required=False,
label="Description",
help_text="Filter by description (partial match)",
)
model = InterfaceTypeMapping
class AddToLIbreSNMPV1V2(forms.Form):
"""
Form for adding devices to LibreNMS using SNMPv1 or SNMPv2c authentication.
Collects hostname/IP and SNMP community string information.
The SNMP version (v1 or v2c) is selected via a toggle button in the template.
"""
hostname = forms.CharField(
label="Hostname/IP",
max_length=255,
required=True,
)
community = forms.CharField(label="SNMP Community", max_length=255, required=True)
port = forms.IntegerField(
label="SNMP Port",
required=False,
help_text="Leave blank to use default SNMP port (161)",
widget=forms.NumberInput(attrs={"placeholder": "161"}),
)
transport = forms.ChoiceField(
label="Transport",
choices=[
("udp", "UDP"),
("tcp", "TCP"),
("udp6", "UDP6"),
("tcp6", "TCP6"),
],
required=False,
initial="udp",
)
port_association_mode = forms.ChoiceField(
label="Port Association Mode",
choices=[
("ifIndex", "ifIndex"),
("ifName", "ifName"),
("ifDescr", "ifDescr"),
("ifAlias", "ifAlias"),
],
required=False,
initial="ifIndex",
help_text="Method to identify ports",
)
poller_group = forms.ChoiceField(
label="Poller Group",
required=False,
help_text="Poller group for distributed poller setup",
)
force_add = forms.BooleanField(
label="Force Add",
required=False,
initial=False,
help_text="Skip duplicate device and SNMP reachability checks (hostname must still be unique)",
)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields["poller_group"].choices = _get_librenms_poller_group_choices()
class AddToLIbreSNMPV3(forms.Form):
"""
Form for adding devices to LibreNMS using SNMPv3 authentication.
Provides comprehensive SNMPv3 configuration options including authentication and encryption settings.
"""
hostname = forms.CharField(
label="Hostname/IP",
max_length=255,
required=True,
)
snmp_version = forms.CharField(widget=forms.HiddenInput(), initial="v3")
authlevel = forms.ChoiceField(
label="Auth Level",
choices=[
("noAuthNoPriv", "noAuthNoPriv"),
("authNoPriv", "authNoPriv"),
("authPriv", "authPriv"),
],
required=True,
)
authname = forms.CharField(label="Auth Username", max_length=255, required=True)
authpass = forms.CharField(
label="Auth Password",
max_length=255,
required=True,
widget=forms.PasswordInput(render_value=True),
)
authalgo = forms.ChoiceField(
label="Auth Algorithm",
choices=[
("SHA", "SHA"),
("MD5", "MD5"),
("SHA-224", "SHA-224"),
("SHA-256", "SHA-256"),
("SHA-384", "SHA-384"),
("SHA-512", "SHA-512"),
],
required=True,
)
cryptopass = forms.CharField(
label="Crypto Password",
max_length=255,
required=True,
widget=forms.PasswordInput(render_value=True),
)
cryptoalgo = forms.ChoiceField(
label="Crypto Algorithm",
choices=[("AES", "AES"), ("DES", "DES")],
required=True,
)
port = forms.IntegerField(
label="SNMP Port",
required=False,
help_text="Leave blank to use default SNMP port (161)",
widget=forms.NumberInput(attrs={"placeholder": "161"}),
)
transport = forms.ChoiceField(
label="Transport",
choices=[
("udp", "UDP"),
("tcp", "TCP"),
("udp6", "UDP6"),
("tcp6", "TCP6"),
],
required=False,
initial="udp",
)
port_association_mode = forms.ChoiceField(
label="Port Association Mode",
choices=[
("ifIndex", "ifIndex"),
("ifName", "ifName"),
("ifDescr", "ifDescr"),
("ifAlias", "ifAlias"),
],
required=False,
initial="ifIndex",
help_text="Method to identify ports",
)
poller_group = forms.ChoiceField(
label="Poller Group",
required=False,
help_text="Poller group for distributed poller setup",
)
force_add = forms.BooleanField(
label="Force Add",
required=False,
initial=False,
help_text="Skip duplicate device and SNMP reachability checks (hostname must still be unique)",
)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields["poller_group"].choices = _get_librenms_poller_group_choices()
class DeviceStatusFilterForm(NetBoxModelFilterSetForm):
"""
Filter form for Device Status view - shows NetBox devices and their LibreNMS status.
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# Remove the saved filter field if it exists
if "filter_id" in self.fields:
del self.fields["filter_id"]
site = DynamicModelMultipleChoiceField(queryset=Site.objects.all(), required=False)
location = DynamicModelMultipleChoiceField(queryset=Location.objects.all(), required=False)
rack = DynamicModelMultipleChoiceField(queryset=Rack.objects.all(), required=False)
device_type = DynamicModelMultipleChoiceField(queryset=DeviceType.objects.all(), required=False)
role = DynamicModelMultipleChoiceField(queryset=DeviceRole.objects.all(), required=False)
model = Device
class LibreNMSImportFilterForm(forms.Form):
"""
Filter form for LibreNMS Import view - shows LibreNMS devices for import.
Uses a simple Django form instead of NetBox model forms.
"""
# LibreNMS filters
librenms_location = forms.ChoiceField(
required=False,
label="LibreNMS Location",
choices=[("", "All Locations")], # Default, will be populated in __init__
widget=forms.Select(attrs={"class": "form-select"}),
)
librenms_type = forms.ChoiceField(
required=False,
label="LibreNMS Type",
choices=[
("", "All Types"),
("network", "Network"),
("server", "Server"),
("storage", "Storage"),
("wireless", "Wireless"),
("firewall", "Firewall"),
("power", "Power"),
("appliance", "Appliance"),
("printer", "Printer"),
("loadbalancer", "Load Balancer"),
("other", "Other"),
],
)
librenms_os = forms.CharField(
required=False,
label="Operating System",
widget=forms.TextInput(attrs={"placeholder": "e.g., ios, linux, junos"}),
)
librenms_hostname = forms.CharField(
required=False,
label="LibreNMS Hostname",
widget=forms.TextInput(attrs={"placeholder": "Partial hostname match"}),
help_text="IP address or FQDN used to add device to LibreNMS",
)
librenms_sysname = forms.CharField(
required=False,
label="LibreNMS System Name",
widget=forms.TextInput(attrs={"placeholder": "Exact or partial sysName match"}),
help_text="SNMP sysName. (exact match only; combine with another filter for partial matching)",
)
librenms_hardware = forms.CharField(
required=False,
label="Hardware",
widget=forms.TextInput(attrs={"placeholder": "e.g., C9300-48P, ASR-920"}),
help_text="LibreNMS hardware model (partial match)",
)
show_disabled = forms.BooleanField(
required=False,
initial=False,
label="Include Disabled Devices",
widget=forms.CheckboxInput(attrs={"class": "form-check-input"}),
)
enable_vc_detection = forms.BooleanField(
required=False,
initial=False,
label="Include Virtual Chassis Detection",
help_text="Run additional stack checks during the search. Will increase processing time.",
widget=forms.CheckboxInput(attrs={"class": "form-check-input"}),
)
clear_cache = forms.BooleanField(
required=False,
initial=False,
label="Clear cache before search",
help_text="Discard the cache and pull fresh data from both LibreNMS and NetBox.",
widget=forms.CheckboxInput(attrs={"class": "form-check-input"}),
)
exclude_existing = forms.BooleanField(
required=False,
initial=False,
label="Exclude Existing Devices",
help_text="Hide devices that already exist in NetBox",
widget=forms.CheckboxInput(attrs={"class": "form-check-input"}),
)
use_background_job = forms.BooleanField(
required=False,
initial=True,
label="Run as background job",
help_text="Recommended: Jobs are logged and can be cancelled.",
widget=forms.CheckboxInput(attrs={"class": "form-check-input"}),
)
def __init__(self, *args, **kwargs):
"""Initialize the form and populate dynamic choices."""
# For bound forms, ensure use_background_job defaults to 'on' if not present
# This handles the case where checkbox is checked by default but not in GET params
# Only apply this default when no filters are applied (initial page load)
if args and isinstance(args[0], (dict, QueryDict)):
# Form is being bound with data (GET/POST dict or QueryDict)
data = args[0].copy() if hasattr(args[0], "copy") else dict(args[0])
# If use_background_job is not in the data, add it with default 'on'
# This makes the checkbox checked by default even on first submission
# Only do this if no filter fields are set (initial page load scenario)
filter_fields = [
"librenms_location",
"librenms_type",
"librenms_os",
"librenms_hostname",
"librenms_sysname",
"librenms_hardware",
]
has_filters = any(data.get(field) for field in filter_fields)
non_option_fields = [
f for f in filter_fields if data.get(f) not in (None, "", []) and str(data.get(f, "")).strip()
]
has_option_only = bool(data) and not bool(non_option_fields) and not has_filters
# Apply default only on initial load (no filters, no job_id, no real submission)
if "use_background_job" not in data and not data.get("job_id") and not has_filters and not has_option_only:
data["use_background_job"] = "on"
args = (data,) + args[1:]
super().__init__(*args, **kwargs)
# Populate LibreNMS location choices dynamically
self._populate_librenms_locations()
def clean(self):
cleaned_data = super().clean()
# Only enforce filter requirement when the user explicitly submits the form
if self.data.get("apply_filters"):
filter_fields = (
"librenms_location",
"librenms_type",
"librenms_os",
"librenms_hostname",
"librenms_sysname",
"librenms_hardware",
)
if not any(cleaned_data.get(field) for field in filter_fields):
raise forms.ValidationError("Please select at least one LibreNMS filter before applying the search.")
return cleaned_data
def _populate_librenms_locations(self):
"""Fetch and populate LibreNMS locations in the dropdown."""
from django.core.cache import cache
from netbox_librenms_plugin.import_utils.cache import get_location_choices_cache_key
from netbox_librenms_plugin.librenms_api import LibreNMSAPI
try:
# Determine server_key cheaply from settings to check cache before instantiating the API
try:
from netbox_librenms_plugin.models import LibreNMSSettings
_settings = LibreNMSSettings.objects.first()
_server_key = (_settings.selected_server if _settings else None) or "default"
except Exception:
_server_key = "default"
cache_key = get_location_choices_cache_key(_server_key)
cached_choices = cache.get(cache_key)
if cached_choices is not None:
self.fields["librenms_location"].choices = cached_choices
return
# Cache miss — instantiate the API client and fetch
api = LibreNMSAPI()
# Recompute cache_key with the resolved server_key in case it differs from settings
cache_key = get_location_choices_cache_key(api.server_key)
# Second cache check: the resolved server_key may differ from the settings key
cached_choices = cache.get(cache_key)
if cached_choices is not None:
self.fields["librenms_location"].choices = cached_choices
return
# Fetch locations from LibreNMS
success, locations = api.get_locations()
if success and locations:
# Build choices list: (id, name)
choices = [("", "All Locations")]
for loc in locations:
loc_id = str(loc.get("id", ""))
loc_name = loc.get("location", f"Location {loc_id}")
choices.append((loc_id, loc_name))
# Sort by name
choices[1:] = sorted(choices[1:], key=lambda x: x[1])
self.fields["librenms_location"].choices = choices
# Cache using configured timeout (default 300s)
cache.set(cache_key, choices, timeout=api.cache_timeout)
logger.info(f"Loaded {len(choices) - 1} LibreNMS locations")
else:
logger.warning(f"Failed to load LibreNMS locations: {locations}")
except Exception as e:
logger.exception(f"Error loading LibreNMS locations: {e}")
# Keep default choices on error
class VirtualMachineStatusFilterForm(NetBoxModelFilterSetForm):
"""
Form for filtering virtual machine status information in NetBox.
"""
def __init__(self, *args, **kwargs):
"""Initialize the form and remove the filter_id field if it exists."""
super().__init__(*args, **kwargs)
# Remove the saved filter field if it exists
if "filter_id" in self.fields:
del self.fields["filter_id"]
virtualmachine = DynamicModelMultipleChoiceField(queryset=VirtualMachine.objects.all(), required=False)
site = DynamicModelMultipleChoiceField(queryset=Site.objects.all(), required=False)
cluster = DynamicModelMultipleChoiceField(queryset=Cluster.objects.all(), required=False)
model = VirtualMachine
class DeviceImportConfigForm(forms.Form):
"""
Form for configuring import of LibreNMS devices with missing prerequisites.
Allows user to manually map LibreNMS device data to NetBox objects.
"""
device_id = forms.IntegerField(widget=forms.HiddenInput(), required=True)
hostname = forms.CharField(disabled=True, required=False, label="Device Hostname")
hardware = forms.CharField(disabled=True, required=False, label="Hardware")
librenms_location = forms.CharField(disabled=True, required=False, label="LibreNMS Location")
# Required mappings
site = forms.ModelChoiceField(
queryset=Site.objects.all(),
required=True,
label="NetBox Site",
help_text="Select the NetBox site for this device",
widget=forms.Select(attrs={"class": "form-select"}),
)
device_type = forms.ModelChoiceField(
queryset=DeviceType.objects.all(),
required=True,
label="Device Type",
help_text="Select the NetBox device type",
widget=forms.Select(attrs={"class": "form-select"}),
)
device_role = forms.ModelChoiceField(
queryset=DeviceRole.objects.all(),
required=True,
label="Device Role",
help_text="Select the device role",
widget=forms.Select(attrs={"class": "form-select"}),
)
# Optional mappings
platform = forms.ModelChoiceField(
queryset=None,
required=False,
label="Platform",
help_text="Select platform (optional)",
widget=forms.Select(attrs={"class": "form-select"}),
)
# Sync options
sync_interfaces = forms.BooleanField(
initial=True,
required=False,
label="Sync Interfaces",
help_text="Automatically sync interfaces from LibreNMS after import",
)
sync_cables = forms.BooleanField(
initial=True,
required=False,
label="Sync Cables",
help_text="Automatically sync cable connections from LibreNMS after import",
)
sync_ips = forms.BooleanField(
initial=True,
required=False,
label="Sync IP Addresses",
help_text="Automatically sync IP addresses from LibreNMS after import",
)
def __init__(self, *args, **kwargs):
"""
Initialize form with LibreNMS device data and validation results.
Accepts additional kwargs:
- libre_device: LibreNMS device dictionary
- validation: Validation result dictionary
- suggested_site: Pre-selected site
- suggested_device_type: Pre-selected device type
- suggested_role: Pre-selected device role
"""
# Extract custom kwargs
libre_device = kwargs.pop("libre_device", {})
validation = kwargs.pop("validation", {})
suggested_site = kwargs.pop("suggested_site", None)
suggested_device_type = kwargs.pop("suggested_device_type", None)
suggested_role = kwargs.pop("suggested_role", None)
super().__init__(*args, **kwargs)
# Import Platform here to avoid circular imports
from dcim.models import Platform
self.fields["platform"].queryset = Platform.objects.all()
# Set initial values from LibreNMS device
if libre_device:
self.fields["device_id"].initial = libre_device.get("device_id")
self.fields["hostname"].initial = libre_device.get("hostname", "")
self.fields["hardware"].initial = libre_device.get("hardware", "")
self.fields["librenms_location"].initial = libre_device.get("location", "")
# Set suggested values from validation
if suggested_site:
self.fields["site"].initial = suggested_site
elif validation and validation.get("site", {}).get("site"):
self.fields["site"].initial = validation["site"]["site"]
if suggested_device_type:
self.fields["device_type"].initial = suggested_device_type
elif validation and validation.get("device_type", {}).get("device_type"):
self.fields["device_type"].initial = validation["device_type"]["device_type"]
if suggested_role:
self.fields["device_role"].initial = suggested_role
elif validation and validation.get("device_role", {}).get("role"):
self.fields["device_role"].initial = validation["device_role"]["role"]
if validation and validation.get("platform", {}).get("platform"):
self.fields["platform"].initial = validation["platform"]["platform"]
# Filter device types by suggestions if available
if validation and validation.get("device_type", {}).get("suggestions"):
suggestions = validation["device_type"]["suggestions"]
if suggestions:
# Annotate with suggested_order so suggested types sort first
suggested_ids = [s["device_type"].id for s in suggestions]
priority = Case(
*[When(id=pk, then=Value(i)) for i, pk in enumerate(suggested_ids)],
default=Value(len(suggested_ids)),
output_field=IntegerField(),
)
self.fields["device_type"].queryset = DeviceType.objects.annotate(suggested_order=priority).order_by(
"suggested_order", "manufacturer__name", "model"
)

View File

@@ -0,0 +1,53 @@
"""
Utilities for importing devices from LibreNMS to NetBox.
This package provides functions for:
- Validating LibreNMS devices for import
- Retrieving filtered LibreNMS devices
- Importing single and multiple devices
- Smart matching of NetBox objects
- Permission checking for import operations
- Virtual chassis detection and creation
All imports below are intentional re-exports so that existing callers
can continue using ``from netbox_librenms_plugin.import_utils import X``.
The F401 suppressions prevent linters from flagging them as unused.
"""
from .bulk_import import ( # noqa: F401
bulk_import_devices,
bulk_import_devices_shared,
process_device_filters,
)
from .cache import ( # noqa: F401
get_active_cached_searches,
get_cache_metadata_key,
get_import_device_cache_key,
get_import_search_cache_key,
get_validated_device_cache_key,
)
from .device_operations import ( # noqa: F401
_determine_device_name,
fetch_device_with_cache,
get_librenms_device_by_id,
import_single_device,
validate_device_for_import,
)
from .filters import ( # noqa: F401
_apply_client_filters,
get_device_count_for_filters,
get_librenms_devices_for_import,
)
from .permissions import check_user_permissions, require_permissions # noqa: F401
from .virtual_chassis import ( # noqa: F401
_clone_virtual_chassis_data,
_generate_vc_member_name,
_vc_cache_key,
create_virtual_chassis_with_members,
detect_virtual_chassis_from_inventory,
empty_virtual_chassis_data,
get_virtual_chassis_data,
prefetch_vc_data_for_devices,
update_vc_member_suggested_names,
)
from .vm_operations import bulk_import_vms, create_vm_from_librenms # noqa: F401

View File

@@ -0,0 +1,706 @@
"""Bulk import orchestration for devices and filter processing."""
import hashlib
import logging
from typing import List
from django.core.cache import cache
from ..import_validation_helpers import apply_role_to_validation, recalculate_validation_status, remove_validation_issue
from ..librenms_api import LibreNMSAPI
from ..utils import find_by_librenms_id
from .cache import get_cache_metadata_key, get_import_device_cache_key, get_validated_device_cache_key
from .device_operations import import_single_device, validate_device_for_import
from .filters import _safe_disabled, get_librenms_devices_for_import
from .permissions import check_user_permissions, require_permissions
from .virtual_chassis import (
create_virtual_chassis_with_members,
empty_virtual_chassis_data,
prefetch_vc_data_for_devices,
)
logger = logging.getLogger(__name__)
def _is_job_cancelled(job) -> bool:
"""
Return True if a background job has been stopped or cancelled.
Checks RQ/Redis state only (reflects stop API calls immediately).
On Redis connectivity issues or a missing RQ job, returns False to avoid
false cancellation. Unexpected exceptions are logged and also return False.
"""
from django_rq import get_queue
from redis.exceptions import RedisError
from rq.exceptions import NoSuchJobError
from rq.job import Job as RQJob
try:
queue = get_queue("default")
rq_job = RQJob.fetch(str(job.job.job_id), connection=queue.connection)
return rq_job.is_failed or rq_job.is_stopped
except (RedisError, NoSuchJobError):
return False
except Exception:
logger.warning("Unexpected error checking RQ job cancellation state", exc_info=True)
return False
def bulk_import_devices_shared(
device_ids: List[int],
server_key: str = None,
sync_options: dict = None,
manual_mappings_per_device: dict = None,
libre_devices_cache: dict = None,
job=None,
user=None,
) -> dict:
"""
Shared function for importing multiple LibreNMS devices to NetBox.
Used by both synchronous imports and background jobs. Handles per-device error
collection and optional progress logging when job context is provided.
Args:
device_ids: List of LibreNMS device IDs to import
server_key: LibreNMS server configuration key
sync_options: Sync options to apply to all devices
manual_mappings_per_device: Dict mapping device_id to manual_mappings dict
Example: {1179: {'device_role_id': 5}, 1180: {'device_role_id': 3}}
libre_devices_cache: Optional dict mapping device_id to pre-fetched device data
to avoid redundant API calls. Example: {123: {...device_data...}}
job: Optional JobRunner instance for progress logging and cancellation checks
user: User performing the import (for permission checks). If job is provided,
user is extracted from job.job.user if not explicitly passed.
Returns:
dict: Bulk import result with structure:
{
'total': int,
'success': List[dict], # Successfully imported devices
'failed': List[dict], # Failed imports with errors
'skipped': List[dict], # Skipped devices (already exist, etc.)
'virtual_chassis_created': int # Number of VCs created
}
Raises:
PermissionDenied: If user lacks required permissions
Example:
>>> # Synchronous usage
>>> result = bulk_import_devices_shared([1, 2, 3, 4, 5], user=request.user)
>>> # Background job usage
>>> result = bulk_import_devices_shared([1, 2, 3], job=self)
"""
# Extract user from job if not explicitly provided
if user is None and job is not None:
user = getattr(job.job, "user", None)
# Check permissions at start of bulk operation — device and VM add perms are
# required because any device may be flagged as import_as_vm during validation.
# change_device is needed for VC master/member updates.
required_perms = [
"dcim.add_device",
"dcim.change_device",
"virtualization.add_virtualmachine",
]
require_permissions(user, required_perms, "import devices")
total = len(device_ids)
success_list = []
failed_list = []
skipped_list = []
vc_created_count = 0
processed_vc_domains = set() # Track VCs already created by domain
_cancelled = False
# Initialize API client once for all devices to avoid repeated config parsing
api = LibreNMSAPI(server_key=server_key)
for idx, device_id in enumerate(device_ids, start=1):
# Check for job cancellation on first iteration and every 5th thereafter.
if job and (idx == 1 or idx % 5 == 0) and _is_job_cancelled(job):
if job.logger:
job.logger.warning(f"Import job stopped at device {idx} of {total}")
else:
logger.warning(f"Import cancelled at device {idx} of {total}")
_cancelled = True
break
try:
# Use cached device data if available to avoid redundant API calls
if libre_devices_cache and device_id in libre_devices_cache:
libre_device = libre_devices_cache[device_id]
success = True
else:
success, libre_device = api.get_device_info(device_id)
if not success or not libre_device:
error_msg = f"Failed to retrieve device {device_id} from LibreNMS"
failed_list.append({"device_id": device_id, "error": error_msg})
if job and job.logger:
job.logger.error(error_msg)
else:
logger.error(error_msg)
continue
use_sysname_opt = sync_options.get("use_sysname", True) if sync_options else True
strip_domain_opt = sync_options.get("strip_domain", False) if sync_options else False
validation = validate_device_for_import(
libre_device,
api=api,
use_sysname=use_sysname_opt,
strip_domain=strip_domain_opt,
server_key=api.server_key,
# Import-time behavior: always evaluate VC state from live/cached
# LibreNMS inventory so stack members are created even when preview
# flags are stale or omitted.
include_vc_detection=True,
)
vc_data = validation.get("virtual_chassis", {})
if vc_data.get("is_stack", False):
has_vc_perm, _ = check_user_permissions(user, ["dcim.add_virtualchassis"])
if not has_vc_perm:
error_msg = f"Cannot import stack device {device_id}: missing permission dcim.add_virtualchassis"
failed_list.append({"device_id": device_id, "error": error_msg})
if job and job.logger:
job.logger.error(error_msg)
else:
logger.error(error_msg)
continue
# Build manual mappings from validation + any provided overrides
device_mappings = {}
# Get site and device_type from validation
if validation["site"].get("found") and validation["site"].get("site"):
device_mappings["site_id"] = validation["site"]["site"].id
if validation["device_type"].get("found") and validation["device_type"].get("device_type"):
device_mappings["device_type_id"] = validation["device_type"]["device_type"].id
if validation["platform"].get("found") and validation["platform"].get("platform"):
device_mappings["platform_id"] = validation["platform"]["platform"].id
# Override with any manual mappings provided for this device
if manual_mappings_per_device and device_id in manual_mappings_per_device:
device_mappings.update(manual_mappings_per_device[device_id])
result = import_single_device(
device_id,
server_key=api.server_key, # use resolved key, not raw parameter (may be None)
validation=validation,
sync_options=sync_options,
manual_mappings=device_mappings if device_mappings else None,
libre_device=libre_device,
)
if result["success"]:
success_list.append(
{
"device_id": device_id,
"device": result["device"],
"message": result["message"],
}
)
# Log progress after each successful import
if job and job.logger:
job.logger.info(f"Imported device {idx} of {total}")
# Handle virtual chassis creation for stacks
if vc_data.get("is_stack", False):
# Derive a stack-level dedup key from member serials so that all
# LibreNMS devices belonging to the same physical stack (e.g. each
# switch in a stacked chassis that appears as a separate device in
# LibreNMS) share the same key and VC creation is triggered only once.
# Fall back to device_id when no member serials are available.
member_serials = sorted(
serial
for m in vc_data.get("members", [])
if (serial := str(m.get("serial") or "").strip()) and serial != "-"
)
if member_serials:
vc_domain = f"librenms-stack-{','.join(member_serials)}"
else:
# No serials available — build a stable fingerprint from member name/model/position
# so all LibreNMS devices in the same physical stack share the same dedup key.
member_parts = sorted(
f"{m.get('name', '')}/{m.get('model', '')}:{m.get('position', 0)}"
for m in vc_data.get("members", [])
)
if member_parts:
fingerprint = hashlib.md5(",".join(member_parts).encode()).hexdigest()[:12]
vc_domain = f"librenms-stack-{fingerprint}"
else:
vc_domain = f"librenms-{device_id}"
# Only create VC if we haven't processed this stack yet.
# Permission was already validated before device import.
if vc_domain not in processed_vc_domains:
# Add to set BEFORE attempting creation to prevent race condition
processed_vc_domains.add(vc_domain)
try:
vc = create_virtual_chassis_with_members(
result["device"],
vc_data["members"],
libre_device,
server_key=api.server_key,
)
vc_created_count += 1
log_msg = f"Created VC '{vc.name}' during bulk import for device {device_id}"
if job and job.logger:
job.logger.info(log_msg)
else:
logger.info(log_msg)
except Exception as vc_error:
# Remove from set on failure so retry is possible
processed_vc_domains.discard(vc_domain)
warn_msg = f"Failed to create VC for device {device_id}: {vc_error}"
if job and job.logger:
job.logger.warning(warn_msg)
else:
logger.warning(warn_msg)
# Don't fail the import, just log the warning
elif result.get("device"): # Device exists
skipped_list.append({"device_id": device_id, "reason": result["error"]})
else: # Failed to import
failed_list.append({"device_id": device_id, "error": result["error"]})
if job and job.logger:
job.logger.error(f"Failed to import device {device_id}: {result['error']}")
except Exception as e:
error_msg = f"Unexpected error importing device {device_id}: {str(e)}"
if job and job.logger:
job.logger.error(error_msg, exc_info=True)
else:
logger.exception(f"Unexpected error importing device {device_id}")
failed_list.append({"device_id": device_id, "error": str(e)})
return {
"total": total,
"success": success_list,
"failed": failed_list,
"skipped": skipped_list,
"virtual_chassis_created": vc_created_count,
"cancelled": _cancelled,
}
def bulk_import_devices(
device_ids: List[int],
server_key: str = None,
sync_options: dict = None,
manual_mappings_per_device: dict = None,
libre_devices_cache: dict = None,
user=None,
) -> dict:
"""
Import multiple LibreNMS devices to NetBox (synchronous).
This is the public API for synchronous imports. For background job usage,
use bulk_import_devices_shared() with a job context.
Args:
device_ids: List of LibreNMS device IDs to import
server_key: LibreNMS server configuration key
sync_options: Sync options to apply to all devices
manual_mappings_per_device: Dict mapping device_id to manual_mappings dict
Example: {1179: {'device_role_id': 5}, 1180: {'device_role_id': 3}}
libre_devices_cache: Optional dict mapping device_id to pre-fetched device data
to avoid redundant API calls. Example: {123: {...device_data...}}
user: User performing the import (for permission checks)
Returns:
dict: Bulk import result with structure:
{
'total': int,
'success': List[dict], # Successfully imported devices
'failed': List[dict], # Failed imports with errors
'skipped': List[dict], # Skipped devices (already exist, etc.)
'virtual_chassis_created': int # Number of VCs created
}
Raises:
PermissionDenied: If user lacks required permissions
"""
return bulk_import_devices_shared(
device_ids=device_ids,
server_key=server_key,
sync_options=sync_options,
manual_mappings_per_device=manual_mappings_per_device,
libre_devices_cache=libre_devices_cache,
job=None, # No job context for synchronous imports
user=user,
)
def _refresh_existing_device(validation: dict, libre_device: dict = None, server_key: str = "default") -> None:
"""
Refresh existing_device from DB to pick up changes made in NetBox since caching.
When existing_device is None (wasn't found at cache time), re-check if the device
was imported since caching by looking up librenms_id or hostname.
"""
existing = validation.get("existing_device")
if existing and hasattr(existing, "pk"):
try:
from dcim.models import Device
from virtualization.models import VirtualMachine
if validation.get("import_as_vm"):
refreshed = VirtualMachine.objects.filter(pk=existing.pk).first()
else:
refreshed = Device.objects.filter(pk=existing.pk).first()
if refreshed:
validation["existing_device"] = refreshed
if hasattr(refreshed, "role") and refreshed.role:
apply_role_to_validation(validation, refreshed.role, is_vm=bool(validation.get("import_as_vm")))
elif not validation.get("import_as_vm"):
validation["device_role"] = {"found": False, "role": None}
remove_validation_issue(validation, "role")
recalculate_validation_status(validation, is_vm=bool(validation.get("import_as_vm")))
# Re-assert non-importable state: recalculate bases can_import on
# issues alone, but an existing matched device must never be import-ready.
validation["can_import"] = False
validation["is_ready"] = False
return
else:
# Device was deleted since caching — recompute readiness to match
# validate_device_for_import logic.
validation["existing_device"] = None
validation["existing_match_type"] = None
# Clear stale device_role so is_ready is computed from scratch.
# Guard: VMs don't use device_role for readiness, so preserve any
# user-selected role rather than silently dropping it.
if not validation.get("import_as_vm"):
validation["device_role"] = {"found": False, "role": None}
recalculate_validation_status(validation, is_vm=bool(validation.get("import_as_vm")))
except Exception as e:
existing_id = getattr(existing, "pk", "unknown") if existing else "none"
logger.error(f"Failed to refresh existing device (pk={existing_id}): {e}")
return
# existing_device was None at cache time — check if device was imported since
if not libre_device:
return
try:
from dcim.models import Device
from virtualization.models import VirtualMachine
import_as_vm = validation.get("import_as_vm", False)
Model = VirtualMachine if import_as_vm else Device
# Also check the opposite model — the LibreNMS object may have been
# imported as a VM even though import_as_vm=False (or vice versa).
CrossModel = Device if import_as_vm else VirtualMachine
librenms_id = libre_device.get("device_id")
hostname = libre_device.get("hostname", "")
sys_name = libre_device.get("sysName", "")
new_device = None
match_type = None
found_as_cross_model = False
def _lookup_in_model(m):
"""Return (device, match_type) for model m, or (None, None)."""
if librenms_id is not None and not isinstance(librenms_id, bool):
try:
dev = find_by_librenms_id(m, int(librenms_id), server_key)
if dev:
return dev, "librenms_id"
except (ValueError, TypeError):
pass
resolved_name = validation.get("resolved_name")
if resolved_name:
dev = m.objects.filter(name__iexact=resolved_name).first()
if dev:
return dev, "resolved_name"
if hostname:
dev = m.objects.filter(name__iexact=hostname).first()
if dev:
return dev, "hostname"
if sys_name:
dev = m.objects.filter(name__iexact=sys_name).first()
if dev:
return dev, "sysname"
return None, None
new_device, match_type = _lookup_in_model(Model)
if not new_device:
# Try the opposite model: catches cross-model imports that happened
# after the cache was built (e.g. LibreNMS device imported as VM).
new_device, match_type = _lookup_in_model(CrossModel)
if new_device:
found_as_cross_model = True
if new_device:
validation["existing_device"] = new_device
validation["existing_match_type"] = match_type
validation["can_import"] = False
validation["is_ready"] = False
# Determine actual model from the found object, not from import_as_vm flag
actual_is_vm = found_as_cross_model != import_as_vm # XOR: cross flips the flag
validation["import_as_vm"] = actual_is_vm # Update so future refreshes query correct model
if not actual_is_vm and hasattr(new_device, "role") and new_device.role:
apply_role_to_validation(validation, new_device.role, is_vm=False)
elif not actual_is_vm:
validation["device_role"] = {"found": False, "role": None}
recalculate_validation_status(validation, is_vm=actual_is_vm)
except Exception as e:
logger.error(f"Failed to check for newly imported device: {e}")
def _empty_return(return_cache_status: bool):
"""Centralised empty-result return value for process_device_filters."""
return ([], False) if return_cache_status else []
def process_device_filters(
api: LibreNMSAPI,
filters: dict,
vc_detection_enabled: bool,
clear_cache: bool,
show_disabled: bool,
exclude_existing: bool = False,
job=None,
request=None,
return_cache_status: bool = False,
use_sysname: bool = True,
strip_domain: bool = False,
) -> List[dict] | tuple[List[dict], bool]:
"""
Process LibreNMS device filters and return validated devices.
Shared function used by both synchronous view and background job processing.
Fetches devices, optionally pre-warms VC cache, validates each device, and
caches results for HTMX row updates.
Args:
api: LibreNMS API client instance
filters: Filter dict with location, type, os, hostname, sysname, hardware keys
vc_detection_enabled: Whether to detect virtual chassis
clear_cache: Whether to force cache refresh
show_disabled: Whether to include disabled devices
exclude_existing: Whether to exclude devices that already exist in NetBox
job: Optional JobRunner instance for logging job events
request: Optional Django request for client disconnect detection (synchronous only)
return_cache_status: When True, returns (devices, from_cache) tuple
use_sysname: If True, prefer sysName over hostname for device name resolution
strip_domain: If True, strip domain suffix from device name
Returns:
List[dict]: Validated devices with _validation key, or tuple of (devices, from_cache)
if return_cache_status is True. from_cache=True means data was loaded from existing
cache; from_cache=False means data was just fetched from LibreNMS.
"""
# Fetch devices from LibreNMS
if job:
job.logger.info(f"Fetching devices with filters: {filters}")
if _is_job_cancelled(job):
job.logger.warning("Job was stopped before fetching devices")
return _empty_return(return_cache_status)
else:
logger.info(f"Fetching devices with filters: {filters}")
# Always get cache status internally, even if not returning it
# We need it to determine if metadata should be updated
libre_devices, from_cache = get_librenms_devices_for_import(
api,
filters=filters,
force_refresh=clear_cache,
return_cache_status=True,
)
# Filter out disabled devices if requested. LibreNMS's "disabled" field (1=disabled,
# 0=enabled) reflects manual device disablement; "status" reflects SNMP reachability.
# show_disabled controls the former: hidden when disabled==1, shown regardless of status.
if not show_disabled:
libre_devices = [d for d in libre_devices if _safe_disabled(d) != 1]
if job:
job.logger.info(f"Found {len(libre_devices)} devices to process")
else:
logger.info(f"Found {len(libre_devices)} devices")
# Check for early cancellation before the expensive VC prefetch
if job and _is_job_cancelled(job):
job.logger.warning("Job was stopped before VC pre-fetch")
return _empty_return(return_cache_status)
# Pre-warm VC cache if needed
if vc_detection_enabled and libre_devices:
device_ids = [d["device_id"] for d in libre_devices]
if job:
job.logger.info(
f"Pre-fetching virtual chassis data for {len(device_ids)} devices. This may take some time..."
)
else:
logger.info(f"Pre-fetching VC data for {len(device_ids)} devices")
try:
prefetch_vc_data_for_devices(api, device_ids, force_refresh=clear_cache)
if job:
job.logger.info("Virtual chassis data pre-fetch completed")
except (BrokenPipeError, ConnectionError, IOError) as e:
if request:
logger.info(f"Client disconnected during VC prefetch: {e}")
return _empty_return(return_cache_status)
raise
# Validate each device
validated_devices = []
total = len(libre_devices)
# Always pass api so validate_device_for_import can run hardware/chassis lookups.
# vc_detection_enabled only gates VC-specific paths inside that function.
if job:
job.logger.info(f"Starting validation of {total} devices")
if _is_job_cancelled(job):
job.logger.warning("Job was already stopped before validation started")
return _empty_return(return_cache_status)
else:
logger.info(f"Validating {total} devices")
for idx, device in enumerate(libre_devices, 1):
# Check for job termination periodically
if (idx % 5 == 0 or idx == 1) and job and _is_job_cancelled(job):
job.logger.info(f"Job stopped at device {idx}/{total}. Exiting gracefully.")
return _empty_return(return_cache_status)
# Drop any cached validation/meta keys before recomputing
device.pop("_validation", None)
# Generate shared cache key for this validated device
device_id = device["device_id"]
cache_key = get_validated_device_cache_key(
server_key=api.server_key,
filters=filters,
device_id=device_id,
vc_enabled=vc_detection_enabled,
use_sysname=use_sysname,
strip_domain=strip_domain,
)
# Check if we already have cached validation for this device
# (only if not forcing refresh)
if not clear_cache:
cached_device = cache.get(cache_key)
if cached_device:
# Use cached validation
device["_validation"] = cached_device["_validation"]
# Refresh existing_device from DB to avoid stale data
# (user may have changed role, name, etc. in NetBox)
_refresh_existing_device(device["_validation"], libre_device=device, server_key=api.server_key)
# Apply exclude_existing filter if enabled
if exclude_existing:
validation = device["_validation"]
if validation["existing_device"]:
continue
validated_devices.append(device)
continue
# Not in cache or forcing refresh - validate now
try:
validation = validate_device_for_import(
device,
api=api,
include_vc_detection=vc_detection_enabled,
force_vc_refresh=False,
server_key=api.server_key,
use_sysname=use_sysname,
strip_domain=strip_domain,
)
except (BrokenPipeError, ConnectionError, IOError) as e:
if request:
logger.info(f"Client disconnected during device validation: {e}")
return _empty_return(return_cache_status)
raise
# Set VC detection metadata
if not vc_detection_enabled:
validation["virtual_chassis"] = empty_virtual_chassis_data()
# Apply exclude_existing filter if enabled
if exclude_existing and validation["existing_device"]:
continue
device["_validation"] = validation
validated_devices.append(device)
# Cache with TWO keys for different purposes:
# 1. Complex key (with filter context) - for full validated device with all metadata
cache.set(cache_key, device, timeout=api.cache_timeout)
# 2. Simple key (device ID only) - for quick device data lookup by role/rack updates
# This avoids redundant API calls when user interacts with dropdowns
simple_cache_key = get_import_device_cache_key(device_id, api.server_key)
# Cache just the raw device data (not the full validation result)
# This is what get_validated_device_with_selections() expects
device_data_only = {k: v for k, v in device.items() if k != "_validation"}
cache.set(simple_cache_key, device_data_only, timeout=api.cache_timeout)
# Store cache metadata (timestamp) for all filter operations
# This enables countdown display regardless of background job vs synchronous execution
# Always store metadata when we have validated devices, even if from_cache
# This ensures metadata is available for countdown display
if validated_devices:
from datetime import datetime, timezone
cache_metadata_key = get_cache_metadata_key(
server_key=api.server_key,
filters=filters,
vc_enabled=vc_detection_enabled,
use_sysname=use_sysname,
strip_domain=strip_domain,
)
# Check if metadata already exists to preserve original timestamp
# BUT: if clear_cache was requested or data came fresh from LibreNMS, update it
existing_metadata = cache.get(cache_metadata_key)
should_update = clear_cache or not from_cache
if existing_metadata and not should_update:
# Metadata exists and cache wasn't cleared, keep using it (preserves original cache time)
pass
else:
# No metadata exists, OR cache was cleared, OR fresh data - create/update it now
cache_metadata = {
"cached_at": datetime.now(timezone.utc).isoformat(),
"cache_timeout": api.cache_timeout,
"filters": filters,
"vc_enabled": vc_detection_enabled,
"device_count": len(validated_devices),
}
cache.set(cache_metadata_key, cache_metadata, timeout=api.cache_timeout)
# Maintain cache index for this server to enable listing active searches
cache_index_key = f"librenms_cache_index_{api.server_key}"
cache_index = cache.get(cache_index_key, [])
# Add this cache key if not already in index
if cache_metadata_key not in cache_index:
cache_index.append(cache_metadata_key)
# Always re-write the index so its TTL matches the freshly-written metadata.
# Without this the index can expire before the metadata and the active
# search entry disappears from the UI.
cache.set(cache_index_key, cache_index, timeout=api.cache_timeout)
if job:
if exclude_existing:
filtered_count = total - len(validated_devices)
job.logger.info(
f"Validation complete: {len(validated_devices)} devices passed filter, "
f"{filtered_count} filtered out (existing devices excluded)"
)
else:
job.logger.info(f"Validation complete: {len(validated_devices)} devices ready for import")
else:
logger.info(f"Processed {len(validated_devices)} validated devices")
if return_cache_status:
return validated_devices, from_cache
return validated_devices

View File

@@ -0,0 +1,232 @@
"""Cache key generation and management for device import operations."""
import hashlib
import json
import logging
from django.core.cache import cache
logger = logging.getLogger(__name__)
def _build_filter_hash(filters: dict) -> str:
"""
Build a stable, collision-free hash from a filter dict.
Removes None values (preserves valid falsy values like 0 and False),
sorts by key, and returns the first 16 hex characters of the SHA-256
digest of the JSON-serialized result.
"""
return hashlib.sha256(
json.dumps({k: v for k, v in filters.items() if v is not None}, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()[:16]
def get_location_choices_cache_key(server_key: str) -> str:
"""Return the cache key for LibreNMS location choices for a given server."""
return f"librenms_locations_choices:{server_key}"
def get_cache_metadata_key(
server_key: str, filters: dict, vc_enabled: bool, use_sysname: bool = True, strip_domain: bool = False
) -> str:
"""
Generate a consistent cache metadata key from filter parameters.
Args:
server_key: LibreNMS server identifier
filters: Filter dictionary
vc_enabled: Whether VC detection is enabled
use_sysname: Whether sysName is preferred over hostname for device naming
strip_domain: Whether domain suffix is stripped from device names
Returns:
str: Consistent cache key for metadata
"""
# Sort filter items to ensure consistent key generation; use "is not None" to preserve
# valid falsy values like 0 and False (filtering only None/missing entries).
# Use JSON serialization for a stable, collision-free hash (avoids issues with
# values containing "=" or "_" that could collide with the key separators).
filter_hash = _build_filter_hash(filters)
return f"librenms_filter_cache_metadata_{server_key}_{filter_hash}_{vc_enabled}_sysname={use_sysname}_strip={strip_domain}"
def get_active_cached_searches(server_key: str) -> list[dict]:
"""
Retrieve all active cached searches for a server and enrich with display-friendly values.
Enriches raw filter IDs with human-readable names by looking up location names
from cached choices and converting type codes to display names.
Args:
server_key: LibreNMS server identifier
Returns:
List of dicts containing cache metadata with enriched display_filters
"""
from datetime import datetime, timezone
cache_index_key = f"librenms_cache_index_{server_key}"
cache_index = cache.get(cache_index_key, [])
active_searches = []
valid_cache_keys = []
# Get location and type choices for enriching display
location_choices = {}
type_choices = {
"": "All Types",
"network": "Network",
"server": "Server",
"storage": "Storage",
"wireless": "Wireless",
"firewall": "Firewall",
"power": "Power",
"appliance": "Appliance",
"printer": "Printer",
"loadbalancer": "Load Balancer",
"other": "Other",
}
# Get cached location choices for enrichment; scoped by server_key so labels
# from different LibreNMS servers don't bleed into each other's filter summaries.
location_cache_key = get_location_choices_cache_key(server_key)
cached_locations = cache.get(location_cache_key)
if cached_locations:
location_choices = dict(cached_locations)
for cache_key in cache_index:
metadata = cache.get(cache_key)
if metadata:
# Cache still exists, calculate time remaining
cache_timeout = metadata.get("cache_timeout", 300)
now = datetime.now(timezone.utc)
try:
cached_at_raw = metadata.get("cached_at")
if isinstance(cached_at_raw, datetime):
cached_at = cached_at_raw
elif cached_at_raw:
cached_at = datetime.fromisoformat(cached_at_raw)
else:
cached_at = datetime.fromtimestamp(0, timezone.utc)
# Normalize naive datetimes (e.g., stored without tzinfo) to UTC
if cached_at.tzinfo is None:
cached_at = cached_at.replace(tzinfo=timezone.utc)
except (ValueError, TypeError):
cached_at = datetime.fromtimestamp(0, timezone.utc)
age_seconds = (now - cached_at).total_seconds()
remaining_seconds = max(0, cache_timeout - age_seconds)
if remaining_seconds > 0:
# Add remaining time and cache key
metadata["remaining_seconds"] = int(remaining_seconds)
metadata["cache_key"] = cache_key
# Store numeric sort key so the final sort is unambiguous
metadata["cached_at_ts"] = cached_at.timestamp()
# Enrich filters with human-readable display values
if "filters" in metadata:
display_filters = metadata["filters"].copy()
# Convert location ID to location name
if "location" in display_filters and display_filters["location"] in location_choices:
display_filters["location"] = location_choices[display_filters["location"]]
# Convert type code to display name
if "type" in display_filters and display_filters["type"] in type_choices:
display_filters["type"] = type_choices[display_filters["type"]]
metadata["display_filters"] = display_filters
else:
# Fallback if filters key missing
metadata["display_filters"] = {}
active_searches.append(metadata)
valid_cache_keys.append(cache_key)
# Clean up index if any keys have expired
if len(valid_cache_keys) < len(cache_index):
cache.set(cache_index_key, valid_cache_keys, timeout=3600)
# Sort by most recent first
active_searches.sort(key=lambda x: x.get("cached_at_ts", 0.0), reverse=True)
return active_searches
def get_validated_device_cache_key(
server_key: str,
filters: dict,
device_id: int | str,
vc_enabled: bool,
use_sysname: bool = True,
strip_domain: bool = False,
) -> str:
"""
Generate a consistent cache key for validated device data.
This ensures both synchronous and background job processing use the same
cache keys, avoiding duplicate validation work and cache entries.
Args:
server_key: LibreNMS server key
filters: Filter dict with location, type, os, hostname, sysname, hardware keys
device_id: LibreNMS device ID
vc_enabled: Whether virtual chassis detection was enabled
use_sysname: Whether sysName is preferred over hostname for device naming
strip_domain: Whether domain suffix is stripped from device names
Returns:
str: Cache key for the validated device
Example:
>>> key = get_validated_device_cache_key('default', {'location': 'NYC'}, 123, True)
>>> key
'validated_device_default_e3b0c44298fc1c14_123_vc'
"""
# Sort filters for a deterministic, cross-process stable hash; None values are excluded
# (consistent with get_cache_metadata_key).
filter_hash = _build_filter_hash(filters)
vc_part = "vc" if vc_enabled else "novc"
return (
f"validated_device_{server_key}_{filter_hash}_{device_id}_{vc_part}_sysname={use_sysname}_strip={strip_domain}"
)
def get_import_device_cache_key(device_id: int | str, server_key: str = "default") -> str:
"""
Generate cache key for raw LibreNMS device data.
This key is used to cache raw device data (without validation metadata)
to avoid redundant API calls when users interact with dropdowns during
the import workflow.
Args:
device_id: LibreNMS device ID
server_key: LibreNMS server identifier for multi-server setups. Defaults to "default" for backward compatibility.
Returns:
str: Cache key for the device data
Example:
>>> get_import_device_cache_key(123, "production")
'import_device_data_production_123'
"""
return f"import_device_data_{server_key}_{device_id}"
def get_import_search_cache_key(server_key: str, api_filters: dict, client_filters: dict) -> str:
"""
Generate a deterministic cache key for a LibreNMS device search result.
The key encodes the server, API-side filters, and client-side filters so
that different filter combinations produce distinct cache entries.
Args:
server_key: Resolved LibreNMS server key (use ``api.server_key``).
api_filters: Filters forwarded to the LibreNMS API.
client_filters: Filters applied client-side after the API response.
Returns:
str: Cache key for the import search result.
"""
return (
f"librenms_devices_import_{server_key}_{_build_filter_hash(api_filters)}_{_build_filter_hash(client_filters)}"
)

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,288 @@
"""Device filtering and retrieval from LibreNMS."""
import logging
from typing import List
from django.core.cache import cache
from .cache import get_import_search_cache_key
from ..librenms_api import LibreNMSAPI
logger = logging.getLogger(__name__)
def _safe_disabled(device: dict) -> int:
"""
Return 1 if the device is disabled, 0 otherwise.
Handles None, booleans, numeric strings, and common truthy/falsy tokens
(e.g. "true"/"yes"/"on" → 1, "false"/"no"/"off" → 0) without raising.
"""
val = device.get("disabled", 0)
if isinstance(val, bool):
return int(val)
if isinstance(val, str):
normalized = val.strip().lower()
if normalized in ("1", "true", "yes", "on"):
return 1
if normalized in ("0", "false", "no", "off", ""):
return 0
try:
int_val = int(val)
return 1 if int_val else 0
except (TypeError, ValueError):
return 0
def get_device_count_for_filters(
api: LibreNMSAPI,
filters: dict,
clear_cache: bool = False,
show_disabled: bool = True,
) -> int:
"""
Get count of LibreNMS devices matching filters.
This is a lightweight function to determine device count for background job
decision making. Uses the same caching as get_librenms_devices_for_import().
Args:
api: LibreNMS API client instance
filters: Filter dict with location, type, os, hostname, sysname keys
clear_cache: Whether to force cache refresh
show_disabled: Whether to include disabled devices
Returns:
int: Count of devices matching filters
"""
devices = get_librenms_devices_for_import(api, filters=filters, force_refresh=clear_cache)
# Filter out disabled devices if requested. LibreNMS's "disabled" field (1=disabled,
# 0=enabled) reflects manual device disablement; "status" reflects SNMP reachability.
# show_disabled controls the former: hidden when disabled==1, shown regardless of status.
if not show_disabled:
devices = [d for d in devices if _safe_disabled(d) != 1]
return len(devices)
def get_librenms_devices_for_import(
api: LibreNMSAPI = None,
filters: dict = None,
server_key: str = None,
*,
force_refresh: bool = False,
return_cache_status: bool = False,
) -> List[dict] | tuple[List[dict], bool]:
"""
Retrieve LibreNMS devices based on filters.
Args:
api: LibreNMSAPI instance (if not provided, creates one with server_key)
filters: Dict containing filter parameters:
- location: LibreNMS location/site filter
- type: Device type filter
- os: Operating system filter
- hostname: Hostname filter (partial match)
- sysname: System name filter (partial match)
- status: Device status filter (1=up, 0=down)
- disabled: Include disabled devices (0=active only, 1=all)
server_key: Key for specific server configuration (used if api not provided)
force_refresh: When True, bypass the cache and fetch fresh data
return_cache_status: When True, returns (devices, from_cache) tuple
Returns:
List of device dictionaries from LibreNMS, or tuple of (devices, from_cache)
if return_cache_status is True. from_cache=True means data was loaded from
existing cache; from_cache=False means data was just fetched from LibreNMS.
"""
try:
# Use provided API instance or create a new one
if api is None:
api = LibreNMSAPI(server_key=server_key)
# Build LibreNMS API filters using the type/query format
# LibreNMS API v0 expects ?type=X&query=Y format, not direct parameters
# NOTE: API only supports ONE type/query pair, so we'll use the most
# specific filter for the API and apply others client-side
api_filters = {}
client_filters = {} # Filters to apply after fetching from API
if filters:
# Check for status filter first - it has special handling
if filters.get("status") is not None:
# Normalize to int: form fields send strings ("1"/"0"), API may send ints
try:
status_val = int(filters["status"])
except (ValueError, TypeError):
status_val = None
# Status filter uses special types that don't need query param
if status_val == 1:
api_filters["type"] = "up"
elif status_val == 0:
api_filters["type"] = "down"
# Save ALL other filters for client-side filtering when status is used
if filters.get("location"):
client_filters["location"] = filters["location"]
if filters.get("type"):
client_filters["type"] = filters["type"]
if filters.get("os"):
client_filters["os"] = filters["os"]
if filters.get("hostname"):
client_filters["hostname"] = filters["hostname"]
if filters.get("sysname"):
client_filters["sysname"] = filters["sysname"]
if filters.get("hardware"):
client_filters["hardware"] = filters["hardware"]
else:
# Priority order for type/query filters: location > type > os > hostname > sysname
# Note: When sysname is combined with other filters, it's applied client-side for partial matching
# When sysname is alone, it uses API exact match (type=sysName)
# Note: hardware is always applied client-side for partial matching
# Use first available for API, save others for client-side filtering
if filters.get("location"):
api_filters["type"] = "location_id"
api_filters["query"] = filters["location"]
# Save remaining filters for client-side
if filters.get("type"):
client_filters["type"] = filters["type"]
if filters.get("os"):
client_filters["os"] = filters["os"]
if filters.get("hostname"):
client_filters["hostname"] = filters["hostname"]
if filters.get("sysname"):
client_filters["sysname"] = filters["sysname"]
if filters.get("hardware"):
client_filters["hardware"] = filters["hardware"]
elif filters.get("type"):
api_filters["type"] = "type"
api_filters["query"] = filters["type"]
# Save remaining filters for client-side
if filters.get("os"):
client_filters["os"] = filters["os"]
if filters.get("hostname"):
client_filters["hostname"] = filters["hostname"]
if filters.get("sysname"):
client_filters["sysname"] = filters["sysname"]
if filters.get("hardware"):
client_filters["hardware"] = filters["hardware"]
elif filters.get("os"):
api_filters["type"] = "os"
api_filters["query"] = filters["os"]
# Save remaining filters for client-side
if filters.get("hostname"):
client_filters["hostname"] = filters["hostname"]
if filters.get("sysname"):
client_filters["sysname"] = filters["sysname"]
if filters.get("hardware"):
client_filters["hardware"] = filters["hardware"]
elif filters.get("hostname"):
api_filters["type"] = "hostname"
api_filters["query"] = filters["hostname"]
# Save sysname and hardware for client-side
if filters.get("sysname"):
client_filters["sysname"] = filters["sysname"]
if filters.get("hardware"):
client_filters["hardware"] = filters["hardware"]
elif filters.get("sysname"):
# sysname-only filter: Use API exact match (type=sysName&query=<value>)
# This is safe - returns empty if no exact match found
api_filters["type"] = "sysName"
api_filters["query"] = filters["sysname"]
# Save hardware for client-side
if filters.get("hardware"):
client_filters["hardware"] = filters["hardware"]
elif filters.get("hardware"):
# hardware-only filter: apply client-side for partial matching
client_filters["hardware"] = filters["hardware"]
# Note: disabled filter isn't directly supported by LibreNMS API
# We'll filter client-side if needed
# Use caching to avoid repeated API calls
# Include both API and client filters in cache key (deterministic, cross-process stable).
# Use api.server_key (always resolved) rather than the raw server_key arg (may differ).
cache_key = get_import_search_cache_key(api.server_key, api_filters, client_filters)
from_cache = False
if force_refresh:
cache.delete(cache_key)
else:
cached_result = cache.get(cache_key)
if cached_result is not None:
# No need to deepcopy - cached data isn't mutated
devices = cached_result
from_cache = True
if return_cache_status:
return devices, from_cache
return devices
success, devices = api.list_devices(api_filters if api_filters else None)
if not success:
logger.error(f"Failed to retrieve devices from LibreNMS: {devices}")
# Cache a brief negative result to prevent hammering the API on repeated failures.
cache.set(cache_key, [], timeout=min(60, api.cache_timeout))
if return_cache_status:
return [], False
return []
# Apply client-side filters if any
if client_filters:
devices = _apply_client_filters(devices, client_filters)
# Cache using configured timeout (default 300s)
# No need to deepcopy - Django's cache backend handles serialization
cache.set(cache_key, devices, timeout=api.cache_timeout)
if return_cache_status:
return devices, from_cache
return devices
except Exception:
logger.exception("Error retrieving LibreNMS devices for import")
if return_cache_status:
return [], False
return []
def _apply_client_filters(devices: List[dict], filters: dict) -> List[dict]:
"""
Apply client-side filters to device list.
Args:
devices: List of device dicts from LibreNMS
filters: Dict of filters to apply (location, type, os, hostname, sysname)
Returns:
Filtered list of devices
"""
filtered = devices
if filters.get("location"):
location_id = str(filters["location"])
filtered = [d for d in filtered if str(d.get("location_id", "")) == location_id]
if filters.get("type"):
device_type = filters["type"].lower()
filtered = [d for d in filtered if (d.get("type") or "").lower() == device_type]
if filters.get("os"):
os_filter = filters["os"].lower()
filtered = [d for d in filtered if os_filter in (d.get("os") or "").lower()]
if filters.get("hostname"):
hostname_filter = filters["hostname"].lower()
filtered = [d for d in filtered if hostname_filter in (d.get("hostname") or "").lower()]
if filters.get("sysname"):
sysname_filter = filters["sysname"].lower()
filtered = [d for d in filtered if sysname_filter in (d.get("sysName") or "").lower()]
if filters.get("hardware"):
hardware_filter = filters["hardware"].lower()
filtered = [d for d in filtered if hardware_filter in (d.get("hardware") or "").lower()]
return filtered

View File

@@ -0,0 +1,44 @@
"""Permission check helpers for device import operations."""
from django.core.exceptions import PermissionDenied
def check_user_permissions(user, permissions):
"""
Check if user has all required permissions.
Args:
user: The user object to check permissions for
permissions: List of permission strings (e.g., ['dcim.add_device', 'dcim.add_interface'])
Returns:
tuple: (has_all_permissions: bool, missing_permissions: list[str])
Raises:
PermissionDenied: If user is None (no user context available)
"""
if user is None:
raise PermissionDenied("No user context available for permission check")
missing = [perm for perm in permissions if not user.has_perm(perm)]
return (len(missing) == 0, missing)
def require_permissions(user, permissions, action_description="perform this action"):
"""
Require user has all permissions, raising PermissionDenied if not.
Args:
user: The user object to check permissions for
permissions: List of permission strings
action_description: Human-readable description for error message
Raises:
PermissionDenied: If user lacks any required permission
"""
has_perms, missing = check_user_permissions(user, permissions)
if not has_perms:
missing_str = ", ".join(missing)
raise PermissionDenied(
f"You do not have permission to {action_description}. Missing permissions: {missing_str}"
)

View File

@@ -0,0 +1,640 @@
"""Virtual chassis detection, creation, and management."""
import logging
from typing import List
from dcim.models import Device, VirtualChassis
from django.core.cache import cache
from django.db import transaction
from ..librenms_api import LibreNMSAPI
logger = logging.getLogger(__name__)
def empty_virtual_chassis_data() -> dict:
"""Public helper for callers that need a blank VC payload."""
return {
"is_stack": False,
"member_count": 0,
"members": [],
"detection_error": None,
}
def _clone_virtual_chassis_data(data: dict | None) -> dict:
"""Return a defensive copy of cached VC data to avoid shared references."""
if not data:
return empty_virtual_chassis_data()
members = []
for idx, member in enumerate(data.get("members", [])):
member_copy = member.copy()
raw_position = member_copy.get("position", idx + 1)
try:
pos = int(raw_position)
member_copy["position"] = pos if pos > 0 else idx + 1
except (TypeError, ValueError):
member_copy["position"] = idx + 1 # 1-based fallback; position 0 is invalid
members.append(member_copy)
member_count = data.get("member_count") or len(members)
return {
"is_stack": bool(data.get("is_stack")),
"member_count": member_count,
"members": members,
"detection_error": data.get("detection_error"),
}
_VC_CACHE_VERSION = "v1"
def _vc_cache_key(api: LibreNMSAPI, device_id: int | str) -> str:
server_key = getattr(api, "server_key", "default")
return f"librenms_vc_detection_{_VC_CACHE_VERSION}_{server_key}_{device_id}"
def get_virtual_chassis_data(api: LibreNMSAPI, device_id: int | str, *, force_refresh: bool = False) -> dict:
"""Fetch (and cache) virtual chassis data for a LibreNMS device."""
if not api or device_id is None:
return empty_virtual_chassis_data()
cache_key = _vc_cache_key(api, device_id)
_cache_timeout = getattr(api, "cache_timeout", None)
cache_timeout = 300 if _cache_timeout is None else _cache_timeout
if not force_refresh and cache_timeout != 0:
cached = cache.get(cache_key)
if cached is not None:
return _clone_virtual_chassis_data(cached)
detection_data = detect_virtual_chassis_from_inventory(api, device_id)
if detection_data is None:
# Non-stack device or transient API failure — cache the negative result so
# prefetch_vc_data_for_devices() can skip these on subsequent renders.
# Use force_refresh=True to bypass the cache if needed.
empty = empty_virtual_chassis_data()
if cache_timeout != 0:
cache.set(cache_key, empty, timeout=cache_timeout)
return _clone_virtual_chassis_data(empty)
if "detection_error" not in detection_data:
detection_data["detection_error"] = None
cache_value = _clone_virtual_chassis_data(detection_data)
if cache_timeout != 0:
cache.set(cache_key, cache_value, timeout=cache_timeout)
return _clone_virtual_chassis_data(cache_value)
def prefetch_vc_data_for_devices(api: LibreNMSAPI, device_ids: List[int], *, force_refresh: bool = False) -> None:
"""
Pre-warm the virtual chassis cache for multiple devices.
This eliminates the 0.5-1s delay when rendering the import table
by proactively fetching VC data before validation.
Args:
api: LibreNMSAPI instance
device_ids: List of LibreNMS device IDs to prefetch VC data for
force_refresh: When True, bypass cache and fetch fresh data
Example:
>>> # Before rendering import table
>>> prefetch_vc_data_for_devices(api, [123, 124, 125])
>>> # Now all validate_device_for_import() calls hit cache instantly
"""
if not api or not device_ids:
return
logger.debug(f"Pre-warming VC cache for {len(device_ids)} devices")
for idx, device_id in enumerate(device_ids):
# This populates the cache if empty, or skips if already cached
try:
get_virtual_chassis_data(api, device_id, force_refresh=force_refresh)
except (BrokenPipeError, ConnectionError, IOError, OSError) as e:
logger.warning(f"Connection error during VC prefetch at device {idx}: {e}")
# Stop processing if connection is broken
return
except Exception as e:
# Log but continue for other errors
logger.warning(f"Error prefetching VC data for device {device_id}: {e}")
logger.debug(f"VC cache warming complete for {len(device_ids)} devices")
def detect_virtual_chassis_from_inventory(api: LibreNMSAPI, device_id: int) -> dict | None:
"""
Detect if device is a stack/Virtual Chassis by analyzing ENTITY-MIB inventory.
Vendor-agnostic using standard hierarchical structure.
Args:
api: LibreNMSAPI instance
device_id: LibreNMS device ID
Returns:
dict with structure:
{
'is_stack': bool,
'member_count': int,
'members': [
{
'serial': str,
'position': int,
'model': str,
'name': str,
'index': int,
'description': str,
'suggested_name': str # Generated using master device name
}
]
}
Returns None if not a stack or detection fails.
Detection Logic:
1. Check root level (entPhysicalContainedIn=0) for parent container
2. Find parent index (entPhysicalClass='stack' or 'chassis')
3. Get children chassis at that parent's index
4. If multiple chassis found -> Stack detected
"""
try:
# Get the master device info to use for naming
success, device_info = api.get_device_info(device_id)
master_name = None
if success and device_info:
master_name = device_info.get("sysName") or device_info.get("hostname")
# Step 1: Get root level items
success, root_items = api.get_inventory_filtered(device_id, ent_physical_contained_in=0)
if not success or not root_items:
logger.debug(f"No root inventory items found for device {device_id}")
return None
# Step 2: Find parent container index
# Prefer "stack" over "chassis" for deterministic VC detection
parent_index = None
stack_index = None
chassis_index = None
for item in root_items:
item_class = item.get("entPhysicalClass")
if item_class == "stack" and stack_index is None:
stack_index = item.get("entPhysicalIndex")
elif item_class == "chassis" and chassis_index is None:
chassis_index = item.get("entPhysicalIndex")
parent_index = stack_index if stack_index is not None else chassis_index
if parent_index is not None:
logger.debug(f"VC detection: Found parent container at index {parent_index} for device {device_id}")
if parent_index is None:
return None
# Step 3: Get children chassis at next level
success, child_items = api.get_inventory_filtered(
device_id,
ent_physical_class="chassis",
ent_physical_contained_in=parent_index,
)
if not success:
return None
# Filter for chassis only (in case API filter didn't work)
chassis_items = [item for item in (child_items or []) if item.get("entPhysicalClass") == "chassis"]
# Step 4: Multiple chassis = stack
if len(chassis_items) <= 1:
return None
# Step 5: Extract member info
# First pass: collect raw entPhysicalParentRelPos values to detect 0-based
# indexing. Some vendors use 0-based positions (0,1,2,3,4) instead of the
# RFC 2737 standard 1-based (1,2,3,4,5). If any raw position is 0, shift
# all valid positions up by 1 so the resulting set is always 1-based.
raw_positions = []
for chassis in chassis_items:
raw = chassis.get("entPhysicalParentRelPos")
try:
raw_positions.append(int(raw))
except (TypeError, ValueError):
raw_positions.append(None)
valid_positions = [p for p in raw_positions if p is not None]
zero_based = bool(valid_positions) and min(valid_positions) == 0
# Identify the master member by matching the LibreNMS device serial
# against the ENTITY-MIB serials. The device-level serial reported by
# LibreNMS corresponds to the active/master switch in the stack.
device_serial = ""
if device_info:
device_serial = _norm_serial(device_info.get("serial"))
# Load naming pattern once to avoid a DB query per member.
vc_name_pattern = _load_vc_member_name_pattern() if master_name else None
members = []
for idx, chassis in enumerate(chassis_items):
raw_pos = raw_positions[idx]
if raw_pos is not None:
position = raw_pos + 1 if zero_based else raw_pos
# Guard against negative or zero after shift
if position <= 0:
position = idx + 1
else:
position = idx + 1
serial = chassis.get("entPhysicalSerialNum", "")
is_master = bool(device_serial and _norm_serial(serial) == device_serial)
member_data = {
"serial": serial,
"position": position,
"model": chassis.get("entPhysicalModelName", ""),
"name": chassis.get("entPhysicalName", ""),
"index": chassis.get("entPhysicalIndex"),
"description": chassis.get("entPhysicalDescr", ""),
"is_master": is_master,
}
# Generate suggested name if we have master name.
# position is already 1-based, so pass it directly (no +1).
if master_name:
member_data["suggested_name"] = _generate_vc_member_name(
master_name, position, serial=_norm_serial(serial), pattern=vc_name_pattern
)
else:
member_data["suggested_name"] = f"Member-{position}"
members.append(member_data)
# Sort by position
members.sort(key=lambda m: m["position"])
if zero_based:
logger.debug(
f"VC detection: corrected 0-based entPhysicalParentRelPos for device {device_id} "
f"(raw min={min(valid_positions)})"
)
master_member = next((m for m in members if m["is_master"]), None)
if master_member:
logger.info(
f"Detected stack with {len(members)} members for device {device_id}; "
f"master at position {master_member['position']} (serial {device_serial})"
)
else:
logger.info(
f"Detected stack with {len(members)} members for device {device_id}; "
f"master could not be identified by serial"
)
return {"is_stack": True, "member_count": len(members), "members": members}
except Exception as e:
logger.exception(f"Error detecting virtual chassis for device {device_id}: {e}")
return None
def _load_vc_member_name_pattern() -> str:
"""Load the VC member name pattern from settings, with fallback to default."""
from ..models import LibreNMSSettings
default = "-M{position}"
try:
settings = LibreNMSSettings.objects.order_by("pk").first()
if not settings:
return default
pattern = settings.vc_member_name_pattern
return pattern if isinstance(pattern, str) and pattern.strip() else default
except Exception as e:
logger.warning(f"Could not load VC member name pattern from settings: {e}. Using default.")
return default
def _generate_vc_member_name(master_name: str, position: int, serial: str = None, pattern: str = None) -> str:
"""
Generate name for VC member device using configured pattern from settings.
Args:
master_name: Name of the master/primary device
position: VC position number
serial: Optional serial number of the member device
pattern: Optional pre-loaded name pattern; if None, loaded from settings.
Pass a pre-loaded pattern when calling inside a loop to avoid
repeated DB queries.
Returns:
Generated member device name
Examples:
pattern="-M{position}" -> "switch01-M2"
pattern=" ({position})" -> "switch01 (2)"
pattern="-SW{position}" -> "switch01-SW2"
pattern=" [{serial}]" -> "switch01 [ABC123]"
"""
if pattern is None:
pattern = _load_vc_member_name_pattern()
# Prepare format variables
format_vars = {
"master_name": master_name,
"position": position,
"serial": serial or "",
}
# Apply pattern - pattern should be suffix/prefix, not full name
try:
formatted_suffix = pattern.format(**format_vars)
return f"{master_name}{formatted_suffix}"
except (KeyError, ValueError, IndexError) as e:
logger.error(f"Invalid placeholder in VC naming pattern '{pattern}': {e}. Using default.")
return f"{master_name}-M{position}"
def update_vc_member_suggested_names(vc_data: dict, master_name: str) -> dict:
"""
Regenerate suggested VC member names using the actual master device name.
This ensures preview shows accurate names after use_sysname and strip_domain
are applied to the master device name.
Args:
vc_data: Virtual chassis detection data dict
master_name: The actual name that will be used for master device in NetBox
Returns:
Updated vc_data dict with corrected suggested_name for each member
"""
if not vc_data or not vc_data.get("is_stack"):
return vc_data
# Load naming pattern once to avoid a DB query per member
vc_pattern = _load_vc_member_name_pattern()
for idx, member in enumerate(vc_data.get("members", [])):
# Positions are stored as 1-based (from entPhysicalParentRelPos or idx+1 fallback).
# Use them directly for name generation; only replace 0/negative with 1-based fallback.
raw_position = member.get("position", idx + 1)
try:
position = int(raw_position)
if position <= 0:
position = idx + 1
except (TypeError, ValueError):
position = idx + 1
member["position"] = position
member["suggested_name"] = _generate_vc_member_name(
master_name, position, serial=_norm_serial(member.get("serial")), pattern=vc_pattern
)
return vc_data
def _safe_pos(value) -> int | None:
"""Return int position or None if not parseable."""
try:
return int(value)
except (TypeError, ValueError):
return None
def _norm_serial(s) -> str:
"""Normalize serial: strip whitespace; treat '-' as absent."""
s = str(s or "").strip()
return "" if s == "-" else s
def _sync_module_bay_counter(device: Device) -> None:
"""Reconcile device module_bay_count with actual ModuleBay rows in the DB."""
try:
actual_count = device.modulebays.count()
if getattr(device, "module_bay_count", None) != actual_count:
Device.objects.filter(pk=device.pk).update(module_bay_count=actual_count)
device.module_bay_count = actual_count
except Exception as e:
logger.warning(
"Could not sync module_bay_count for device '%s': %s",
getattr(device, "name", "unknown"),
e,
)
def create_virtual_chassis_with_members(
master_device: Device, members_info: list, libre_device: dict, server_key: str | None = None
) -> VirtualChassis:
"""
Create Virtual Chassis and member devices from detection info.
This function creates a NetBox VirtualChassis with the master device
and all detected member devices, wrapped in a transaction for safety.
Args:
master_device: The imported device (becomes VC master)
members_info: List of member dicts from VC detection
libre_device: Original LibreNMS device data
Returns:
VirtualChassis: The created virtual chassis instance
Raises:
ValidationError: If member count validation fails
IntegrityError: If duplicate serials/names are detected
Exception: For other creation errors
Example members_info:
[
{'serial': 'ABC123', 'position': 0, 'model': 'C9300-48U', 'name': 'Switch 1'},
{'serial': 'ABC124', 'position': 1, 'model': 'C9300-48U', 'name': 'Switch 2'}
]
"""
# Save originals for in-memory rollback — transaction.atomic() rolls back DB but
# not in-memory model fields.
original_master_name = master_device.name
original_vc = master_device.virtual_chassis
original_vc_position = master_device.vc_position
# Find master's actual VC position from members_info.
# Priority: is_master flag (set during detection) → serial match → default 1.
_master_pos = 1
_master_member = next((m for m in members_info if m.get("is_master")), None)
if _master_member:
_found_pos = _safe_pos(_master_member.get("position"))
if _found_pos and _found_pos >= 1:
_master_pos = _found_pos
elif _norm_serial(master_device.serial):
for _m in members_info:
if _norm_serial(_m.get("serial")) == _norm_serial(master_device.serial):
_found_pos = _safe_pos(_m.get("position"))
if _found_pos and _found_pos >= 1:
_master_pos = _found_pos
break
try:
with transaction.atomic():
# Load naming pattern once to avoid a DB query per member
vc_pattern = _load_vc_member_name_pattern()
# Rename master device to include position 1 pattern
master_device_new_name = _generate_vc_member_name(
original_master_name, _master_pos, serial=_norm_serial(master_device.serial), pattern=vc_pattern
)
# Check if renamed master conflicts with existing device
if Device.objects.filter(name=master_device_new_name).exclude(pk=master_device.pk).exists():
logger.warning(
f"Cannot rename master to '{master_device_new_name}' - name already exists. "
f"Keeping original name '{original_master_name}'"
)
master_base_name = original_master_name
rename_master = False
else:
master_device.name = master_device_new_name
master_base_name = original_master_name
rename_master = True
# Create VC using original base name
vc_name = master_base_name
_device_id = libre_device.get("device_id") or master_device.pk
_domain_prefix = f"librenms-{server_key}" if server_key else "librenms"
vc = VirtualChassis.objects.create(
name=vc_name,
domain=f"{_domain_prefix}-{_device_id}",
)
# Update master device
master_device.virtual_chassis = vc
master_device.vc_position = _master_pos
save_fields = ["virtual_chassis", "vc_position"]
if rename_master:
save_fields.append("name")
master_device.save(update_fields=save_fields)
# Create member devices for remaining positions
position = _master_pos + 1 # Start after master position
used_positions = {_master_pos} # Master occupies its actual position
members_created = 0
for member in members_info:
# Normalize serial and position up front so all skip-checks and
# downstream logic use consistent values (strips whitespace and
# treats the sentinel "-" as "no serial").
serial = str(member.get("serial") or "").strip()
if serial == "-":
serial = ""
member_pos = _safe_pos(member.get("position"))
# Skip the master member — identified by is_master flag, serial match,
# or position match.
if member.get("is_master"):
continue
# Skip if this is the master's serial (only when both serials are non-empty)
if serial and serial == _norm_serial(master_device.serial):
continue
# Skip blank-serial entries that represent the master slot by position
if (
not serial
and member_pos is not None
and master_device.vc_position is not None
and member_pos == master_device.vc_position
):
continue
member_rack = master_device.rack
member_location = master_device.location or (
member_rack.location if member_rack and member_rack.location else None
)
# Check for duplicate serial
if serial and Device.objects.filter(serial=serial).exists():
logger.warning(f"Device with serial '{serial}' already exists, skipping VC member creation")
continue
# Prefer the discovered SNMP position; fall back to sequential counter.
# member_pos was normalized via _safe_pos() above; 0 is not a valid vc_position.
discovered_pos = member_pos if (member_pos is not None and member_pos >= 1) else None
# If discovered_pos is already taken by another member, treat as absent.
if discovered_pos is not None and discovered_pos in used_positions:
discovered_pos = None
# Consume next free sequential slot when no valid discovered_pos.
if discovered_pos is None:
while position in used_positions:
position += 1
chosen_pos = position
position += 1
else:
chosen_pos = discovered_pos
# Advance sequential counter past chosen position.
position = max(position, chosen_pos + 1)
used_positions.add(chosen_pos)
member_name = _generate_vc_member_name(master_base_name, chosen_pos, serial=serial, pattern=vc_pattern)
# Check for duplicate name
if Device.objects.filter(name=member_name).exists():
logger.warning(f"Device with name '{member_name}' already exists, skipping VC member creation")
continue
Device.objects.create(
name=member_name,
device_type=master_device.device_type,
role=master_device.role,
site=master_device.site,
location=member_location,
rack=member_rack,
platform=master_device.platform,
serial=serial,
virtual_chassis=vc,
vc_position=chosen_pos,
comments=f"VC member (LibreNMS: {member.get('name', 'Unknown')})\n"
f"Auto-created from stack inventory",
)
members_created += 1
# Validate member count
# Validate member count — exclude master-slot entries with blank serials
expected_members = len(
[
m
for m in members_info
if not (
_norm_serial(m.get("serial"))
and _norm_serial(m.get("serial")) == _norm_serial(master_device.serial)
)
and not (
not _norm_serial(m.get("serial"))
and m.get("position") is not None
and master_device.vc_position is not None
and _safe_pos(m["position"]) == master_device.vc_position
)
]
)
if members_created < expected_members:
logger.warning(
f"Created {members_created} members but expected {expected_members}. "
"Some members may have been skipped due to duplicates."
)
# Assign VC master only after all members are attached to avoid
# NetBox's create-time auto-master signal changing order/state.
vc.master = master_device
vc.save(update_fields=["master"])
_sync_module_bay_counter(master_device)
logger.info(
f"Created Virtual Chassis '{vc.name}' with {vc.members.count()} total members "
f"(1 master + {members_created} additional)"
)
return vc
except Exception as e:
master_device.name = original_master_name
master_device.virtual_chassis = original_vc
master_device.vc_position = original_vc_position
logger.error(
f"Virtual Chassis creation failed for device {original_master_name}: {e}",
exc_info=True,
)
raise

View File

@@ -0,0 +1,257 @@
"""Virtual machine creation and import operations."""
import logging
from dcim.models import DeviceRole
from django.db import transaction
from django.utils import timezone
from virtualization.models import Cluster
from ..librenms_api import LibreNMSAPI
from .bulk_import import _is_job_cancelled
from .device_operations import _determine_device_name, fetch_device_with_cache, validate_device_for_import
from .permissions import require_permissions
logger = logging.getLogger(__name__)
def create_vm_from_librenms(
libre_device: dict,
validation: dict,
server_key: str = "default",
use_sysname: bool = True,
strip_domain: bool = False,
role=None,
):
"""
Create a NetBox VirtualMachine from LibreNMS device data.
Args:
libre_device: Device data from LibreNMS
validation: Validation result from validate_device_for_import with import_as_vm=True
use_sysname: If True, prefer sysName; if False, use hostname
server_key: LibreNMS server key used to store the librenms_id custom field
Returns:
Created VirtualMachine instance
Raises:
Exception if VM cannot be created
"""
from virtualization.models import VirtualMachine
if not validation["can_import"]:
raise ValueError(f"VM cannot be imported: {', '.join(validation['issues'])}")
# Extract matched objects from validation
cluster = validation["cluster"]["cluster"]
platform = validation["platform"].get("platform")
role = role if role is not None else validation.get("device_role", {}).get("role")
# Determine VM name - use pre-computed name if available (handles strip_domain),
# falling back to the validated resolved_name before recomputing from raw fields.
vm_name = libre_device.get("_computed_name") or validation.get("resolved_name")
if not vm_name:
vm_name = _determine_device_name(
libre_device,
use_sysname=use_sysname,
strip_domain=strip_domain,
device_id=libre_device.get("device_id"),
)
# Generate import timestamp comment
import_time = timezone.now().strftime("%Y-%m-%d %H:%M:%S %Z")
# Validate device_id before creating the VM so a missing/invalid value
# never leaves a VM without a librenms_id (partial persistence).
raw_device_id = libre_device["device_id"]
if isinstance(raw_device_id, bool):
raise ValueError(f"device_id is a boolean ({raw_device_id!r}); expected an integer")
librenms_device_id = int(raw_device_id)
from ..utils import set_librenms_device_id
# Create the VM and assign its LibreNMS ID atomically so a failure in
# set_librenms_device_id never leaves a VM without a mapping.
with transaction.atomic():
vm = VirtualMachine.objects.create(
name=vm_name,
cluster=cluster,
role=role,
platform=platform,
comments=f"Imported from LibreNMS (device_id={librenms_device_id}) by netbox-librenms-plugin on {import_time}",
)
set_librenms_device_id(vm, librenms_device_id, server_key)
vm.save()
logger.info(f"Created VM {vm.name} (ID: {vm.pk}) from LibreNMS device {libre_device['device_id']}")
return vm
def bulk_import_vms(
vm_imports: dict[int, dict[str, int]],
api: LibreNMSAPI,
sync_options: dict = None,
libre_devices_cache: dict = None,
job=None,
user=None,
) -> dict:
"""
Import multiple LibreNMS devices as VMs in NetBox.
Handles validation, cluster/role assignment, name determination,
and VM creation. Supports both synchronous and background job execution.
This function consolidates VM import logic that was previously duplicated
in BulkImportDevicesView and ImportDevicesJob, ensuring consistent behavior
across synchronous and background import paths.
Args:
vm_imports: Dict mapping device_id to {"cluster_id": int, "device_role_id": int}
api: LibreNMSAPI instance for device fetching
sync_options: Optional dict with use_sysname, strip_domain settings
libre_devices_cache: Optional pre-fetched device data cache
job: Optional JobRunner instance for background job logging/cancellation
user: User performing the import (for permission checks). If job is provided,
user is extracted from job.job.user if not explicitly passed.
Returns:
Dict with keys:
- success: List of {"device_id": int, "device": VM, "message": str}
- failed: List of {"device_id": int, "error": str}
- skipped: List of {"device_id": int, "reason": str}
Raises:
PermissionDenied: If user lacks required permissions
Example:
>>> # Synchronous import from view
>>> vm_imports = {123: {"cluster_id": 5, "device_role_id": 2}}
>>> result = bulk_import_vms(vm_imports, api, sync_options, user=request.user)
>>> print(f"Created {len(result['success'])} VMs")
>>>
>>> # Background job import
>>> result = bulk_import_vms(vm_imports, api, sync_options, cache, job=self)
"""
from netbox_librenms_plugin.import_validation_helpers import (
apply_cluster_to_validation,
apply_role_to_validation,
)
# Extract user from job if not explicitly provided
if user is None and job is not None:
user = getattr(job.job, "user", None)
# Check permissions at start of bulk operation
require_permissions(user, ["virtualization.add_virtualmachine"], "import VMs")
result = {"success": [], "failed": [], "skipped": []}
vm_ids = list(vm_imports.keys())
# Use job logger if available, otherwise standard logger
log = job.logger if job else logger
for idx, vm_id in enumerate(vm_ids, start=1):
# Check for job cancellation before first VM and every 5 thereafter
if job and (idx == 1 or idx % 5 == 0) and _is_job_cancelled(job):
log.warning(f"Job cancelled at VM {idx} of {len(vm_ids)}")
break
log.info(f"Processing VM {idx} of {len(vm_ids)}")
try:
# Fetch device data (uses cache helper)
libre_device = fetch_device_with_cache(vm_id, api, api.server_key, libre_devices_cache)
if not libre_device:
result["failed"].append(
{
"device_id": vm_id,
"error": f"Device {vm_id} not found in LibreNMS",
}
)
log.error(f"Device {vm_id} not found in LibreNMS")
continue
# Validate as VM
use_sysname_opt = sync_options.get("use_sysname", True) if sync_options else True
strip_domain_opt = sync_options.get("strip_domain", False) if sync_options else False
validation = validate_device_for_import(
libre_device,
import_as_vm=True,
api=api,
use_sysname=use_sysname_opt,
strip_domain=strip_domain_opt,
server_key=api.server_key,
)
# Check if VM already exists
if validation.get("existing_device"):
result["skipped"].append(
{
"device_id": vm_id,
"reason": f"VM already exists: {validation['existing_device'].name}",
}
)
log.info(f"VM already exists: {validation['existing_device'].name}")
continue
# Apply manual cluster and role selections
vm_mappings = vm_imports[vm_id]
cluster_id = vm_mappings.get("cluster_id")
role_id = vm_mappings.get("device_role_id")
if cluster_id:
cluster = Cluster.objects.filter(id=cluster_id).first()
if cluster:
apply_cluster_to_validation(validation, cluster)
else:
result["failed"].append(
{"device_id": vm_id, "error": f"Selected cluster (id={cluster_id}) no longer exists"}
)
continue
role = None
if role_id:
role = DeviceRole.objects.filter(id=role_id).first()
if role:
apply_role_to_validation(validation, role, is_vm=True)
else:
result["failed"].append(
{"device_id": vm_id, "error": f"Selected role (id={role_id}) no longer exists"}
)
continue
# Determine VM name
vm_name = _determine_device_name(
libre_device,
use_sysname=use_sysname_opt,
strip_domain=strip_domain_opt,
device_id=vm_id,
)
# Update validation with computed name
libre_device["_computed_name"] = vm_name
# Create VM
vm = create_vm_from_librenms(
libre_device,
validation,
use_sysname=use_sysname_opt,
strip_domain=strip_domain_opt,
server_key=api.server_key,
)
result["success"].append(
{
"device_id": vm_id,
"device": vm,
"message": f"VM {vm.name} created successfully",
}
)
log.info(f"Successfully imported VM {vm.name} (ID: {vm_id})")
except Exception as vm_error:
log.error(f"Failed to import VM {vm_id}: {vm_error}", exc_info=True)
result["failed"].append({"device_id": vm_id, "error": str(vm_error)})
return result

View File

@@ -0,0 +1,168 @@
"""
Helper functions for validation state mutation during import workflow.
These functions centralize the logic for updating validation dictionaries
when users select roles, clusters, or racks during the device import process.
"""
import logging
logger = logging.getLogger(__name__)
def fetch_model_by_id(model_class, pk):
"""
Generic helper to fetch a model instance by primary key.
Args:
model_class: Django model class (e.g., DeviceRole, Cluster, Rack)
pk: Primary key value (int, str, or None)
Returns:
Model instance if found and valid, None otherwise
Example:
>>> from dcim.models import DeviceRole
>>> role = fetch_model_by_id(DeviceRole, "5")
>>> role.name
'Router'
"""
if pk is None:
return None
try:
return model_class.objects.get(pk=int(pk))
except (model_class.DoesNotExist, ValueError, TypeError):
return None
def extract_device_selections(request, device_id):
"""
Extract cluster, role, and rack selections from request POST/GET data.
Args:
request: Django request object
device_id: LibreNMS device ID
Returns:
dict with keys: cluster_id, role_id, rack_id (all may be None)
Example:
>>> selections = extract_device_selections(request, 1234)
>>> selections
{'cluster_id': None, 'role_id': '5', 'rack_id': '12'}
"""
# Check both POST and GET data (different views use different methods)
data_source = request.POST if request.method == "POST" else request.GET
return {
"cluster_id": data_source.get(f"cluster_{device_id}"),
"role_id": data_source.get(f"role_{device_id}"),
"rack_id": data_source.get(f"rack_{device_id}"),
}
def apply_role_to_validation(validation: dict, role, is_vm: bool = False) -> None:
"""
Update validation state after device/VM role selection.
Args:
validation: Validation dict from validate_device_for_import()
role: DeviceRole instance selected by user
is_vm: True if importing as VM, False for device
Modifies validation dict in-place:
- Sets device_role["found"] = True
- Sets device_role["role"] = role
- Removes "role" related issues
- Recalculates can_import and is_ready flags
"""
validation["device_role"]["found"] = True
validation["device_role"]["role"] = role
remove_validation_issue(validation, "role")
recalculate_validation_status(validation, is_vm)
def apply_cluster_to_validation(validation: dict, cluster) -> None:
"""
Update validation state after cluster selection (VM import only).
Args:
validation: Validation dict from validate_device_for_import()
cluster: Cluster instance selected by user
Modifies validation dict in-place:
- Sets cluster["found"] = True
- Sets cluster["cluster"] = cluster
- Removes "cluster" related issues
- Recalculates can_import and is_ready flags (as VM)
"""
validation["cluster"]["found"] = True
validation["cluster"]["cluster"] = cluster
remove_validation_issue(validation, "cluster")
recalculate_validation_status(validation, is_vm=True)
def apply_rack_to_validation(validation: dict, rack) -> None:
"""
Update validation state after rack selection (device import only).
Args:
validation: Validation dict from validate_device_for_import()
rack: Rack instance selected by user
Modifies validation dict in-place:
- Sets rack["found"] = True
- Sets rack["rack"] = rack
Note: Rack is optional, so this doesn't affect can_import/is_ready.
"""
validation.setdefault("rack", {})
validation["rack"]["found"] = True
validation["rack"]["rack"] = rack
def remove_validation_issue(validation: dict, keyword: str) -> None:
"""
Remove validation issues containing the specified keyword.
Args:
validation: Validation dict
keyword: Keyword to search for in issue messages (case-insensitive)
Example:
>>> remove_validation_issue(validation, "role")
# Removes "Device role must be manually selected before import"
"""
validation["issues"] = [issue for issue in validation["issues"] if keyword.lower() not in issue.lower()]
def recalculate_validation_status(validation: dict, is_vm: bool = False) -> None:
"""
Recalculate can_import and is_ready flags based on current validation state.
Args:
validation: Validation dict
is_vm: True if importing as VM, False for device
Updates:
- can_import: True if no blocking issues remain
- is_ready: True if can_import AND all required fields are found
Required fields for devices:
- site, device_type, device_role
Required fields for VMs:
- cluster
"""
validation["can_import"] = len(validation["issues"]) == 0
if is_vm:
validation["is_ready"] = validation["can_import"] and validation["cluster"]["found"]
else:
validation["is_ready"] = (
validation["can_import"]
and validation["site"]["found"]
and validation["device_type"]["found"]
and validation["device_role"]["found"]
)

View File

@@ -0,0 +1,275 @@
"""
Background jobs for LibreNMS plugin.
This module provides background job implementations for long-running operations
such as device filtering with Virtual Chassis detection.
"""
import logging
from netbox.jobs import JobRunner
logger = logging.getLogger(__name__)
class FilterDevicesJob(JobRunner):
"""
Background job for processing LibreNMS device filters with VC detection.
Background jobs provide several benefits over synchronous processing:
- Active cancellation via NetBox Jobs interface
- Browser remains responsive (no "page loading" state)
- Job progress tracked in NetBox Jobs table
- Results persist in cache for later retrieval
Users control background job execution via the "Run as background job" checkbox
in the filter form. When enabled, the job runs asynchronously; when disabled,
filtering runs synchronously.
Note: Both synchronous and background processing complete once started,
even if the user navigates away. The key difference is cancellation ability
and browser responsiveness.
Results are cached individually per device to avoid exceeding job data size limits.
"""
class Meta:
"""Meta options for FilterDevicesJob."""
name = "LibreNMS Device Filter"
def run(
self,
filters,
vc_detection_enabled,
clear_cache,
show_disabled,
exclude_existing=False,
server_key=None,
use_sysname=True,
strip_domain=False,
**kwargs,
):
"""
Execute filter processing in background.
Logs job start, completion, and any early termination events.
Args:
filters: Dict with location, type, os, hostname, sysname keys
vc_detection_enabled: Whether to detect virtual chassis
clear_cache: Whether to force cache refresh
show_disabled: Whether to include disabled devices
exclude_existing: Whether to exclude devices that already exist in NetBox
server_key: Optional LibreNMS server key for multi-server setups
use_sysname: If True, prefer sysName over hostname for device name resolution
strip_domain: If True, strip domain suffix from device names
**kwargs: Additional job parameters
"""
from netbox_librenms_plugin.import_utils import process_device_filters
from netbox_librenms_plugin.librenms_api import LibreNMSAPI
self.logger.info("Starting LibreNMS device filter job")
self.logger.info(f"Filters: {filters}")
self.logger.info(f"VC detection: {vc_detection_enabled}")
self.logger.info(f"Clear cache: {clear_cache}")
self.logger.info(f"Show disabled: {show_disabled}")
if exclude_existing:
self.logger.info("Excluding existing devices")
if server_key:
self.logger.info(f"Using LibreNMS server: {server_key}")
# Initialize API client
api = LibreNMSAPI(server_key=server_key)
self.logger.info(f"LibreNMS API initialized (cache timeout: {api.cache_timeout}s)")
# Process filters using shared function
validated_devices = process_device_filters(
api=api,
filters=filters,
vc_detection_enabled=vc_detection_enabled,
clear_cache=clear_cache,
show_disabled=show_disabled,
exclude_existing=exclude_existing,
job=self,
use_sysname=use_sysname,
strip_domain=strip_domain,
)
# Store device IDs for result retrieval
# Note: Validated devices are cached with shared keys by process_device_filters
device_ids = [device["device_id"] for device in validated_devices]
# Track cache timestamp for frontend expiration warnings
from datetime import datetime, timezone
cached_at = datetime.now(timezone.utc).isoformat()
# Store only metadata in job data (not the full device list)
# Devices are retrieved via shared cache keys in _load_job_results
self.job.data = {
"device_ids": device_ids,
"total_processed": len(validated_devices),
"filters": filters,
"server_key": api.server_key,
"vc_detection_enabled": vc_detection_enabled,
"use_sysname": use_sysname,
"strip_domain": strip_domain,
"cache_timeout": api.cache_timeout,
"cached_at": cached_at,
"completed": True,
}
self.job.save(update_fields=["data"])
self.logger.info(
f"Job completed successfully. Processed {len(validated_devices)} devices. "
f"Results available via shared cache for {api.cache_timeout} seconds."
)
class ImportDevicesJob(JobRunner):
"""
Background job for importing LibreNMS devices to NetBox.
Handles bulk device/VM imports in the background to keep browser responsive.
Benefits:
- Active cancellation via NetBox Jobs interface
- Browser remains responsive during large imports
- Job progress tracked with device count logging
- Errors collected per device without stopping entire import
Users control background job execution via the "Run as background job" checkbox
in the import confirmation modal. When enabled, the job runs asynchronously;
when disabled, imports run synchronously.
Results stored in job.data with structure:
{
"imported_device_pks": [1, 2, 3], # NetBox Device PKs
"imported_vm_pks": [10, 11], # NetBox VirtualMachine PKs
"total": 5,
"success_count": 4,
"failed_count": 1,
"skipped_count": 0,
"errors": [{"device_id": 123, "error": "..."}]
}
"""
class Meta:
"""Meta options for ImportDevicesJob."""
name = "LibreNMS Device Import"
def run(
self,
device_ids,
vm_imports,
server_key=None,
sync_options=None,
manual_mappings_per_device=None,
libre_devices_cache=None,
**kwargs,
):
"""
Execute device/VM imports in background.
Args:
device_ids: List of LibreNMS device IDs to import as Devices
vm_imports: Dict mapping device_id to cluster/role info for VM imports
server_key: Optional LibreNMS server key for multi-server setups
sync_options: Dict with sync_interfaces, sync_cables, sync_ips,
use_sysname, strip_domain, and vc_detection_enabled
manual_mappings_per_device: Dict mapping device_id to manual_mappings dict
libre_devices_cache: Optional dict mapping device_id to pre-fetched device data
**kwargs: Additional job parameters
"""
from netbox_librenms_plugin.import_utils import (
bulk_import_devices_shared,
)
from netbox_librenms_plugin.librenms_api import LibreNMSAPI
total_count = len(device_ids) + len(vm_imports)
self.logger.info(f"Starting LibreNMS import job for {total_count} devices/VMs")
self.logger.info(f"Device imports: {len(device_ids)}, VM imports: {len(vm_imports)}")
if server_key:
self.logger.info(f"Using LibreNMS server: {server_key}")
# Initialize API client
api = LibreNMSAPI(server_key=server_key)
# Import devices using shared function with job context
device_result = {
"success": [],
"failed": [],
"skipped": [],
"virtual_chassis_created": 0,
}
if device_ids:
self.logger.info(f"Importing {len(device_ids)} devices...")
device_result = bulk_import_devices_shared(
device_ids=device_ids,
server_key=api.server_key,
sync_options=sync_options,
manual_mappings_per_device=manual_mappings_per_device,
libre_devices_cache=libre_devices_cache,
job=self, # Pass job context for logging and cancellation
user=self.job.user, # Pass user for permission checks
)
# Import VMs
vm_result = {"success": [], "failed": [], "skipped": []}
if vm_imports:
self.logger.info(f"Importing {len(vm_imports)} VMs...")
from netbox_librenms_plugin.import_utils import bulk_import_vms
vm_result = bulk_import_vms(
vm_imports, api, sync_options, libre_devices_cache, job=self, user=self.job.user
)
# Combine results — partition device_result successes by model type since
# bulk_import_devices_shared() may return VirtualMachine objects when import_as_vm=True.
device_successes = []
vm_successes = list(vm_result.get("success", []))
for item in device_result.get("success", []):
obj = item.get("device")
if not obj:
continue
if obj._meta.model_name == "virtualmachine":
vm_successes.append(item)
else:
device_successes.append(item)
imported_device_pks = [item["device"].pk for item in device_successes]
imported_vm_pks = [item["device"].pk for item in vm_successes]
# Also store LibreNMS device IDs for re-rendering table rows
imported_libre_device_ids = [item["device_id"] for item in device_successes]
imported_libre_vm_ids = [item["device_id"] for item in vm_successes]
success_count = len(device_result.get("success", [])) + len(vm_result.get("success", []))
failed_count = len(device_result.get("failed", [])) + len(vm_result.get("failed", []))
skipped_count = len(device_result.get("skipped", [])) + len(vm_result.get("skipped", []))
all_errors = device_result.get("failed", []) + vm_result.get("failed", [])
# Store results in job.data
self.job.data = {
"imported_device_pks": imported_device_pks,
"imported_vm_pks": imported_vm_pks,
"imported_libre_device_ids": imported_libre_device_ids,
"imported_libre_vm_ids": imported_libre_vm_ids,
"server_key": api.server_key,
"total": total_count,
"success_count": success_count,
"failed_count": failed_count,
"skipped_count": skipped_count,
"virtual_chassis_created": device_result.get("virtual_chassis_created", 0),
"errors": all_errors,
"completed": True,
}
self.job.save(update_fields=["data"])
self.logger.info(
f"Import job completed. Success: {success_count}, Failed: {failed_count}, Skipped: {skipped_count}"
)

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,23 @@
# Generated by Django 5.0.9 on 2024-09-19 10:17
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = []
operations = [
migrations.CreateModel(
name="InterfaceTypeMapping",
fields=[
(
"id",
models.BigAutoField(auto_created=True, primary_key=True, serialize=False),
),
("librenms_type", models.CharField(max_length=100, unique=True)),
("netbox_type", models.CharField(default="other", max_length=50)),
],
),
]

View File

@@ -0,0 +1,35 @@
# Generated by Django 5.0.9 on 2024-09-19 11:14
import taggit.managers
import utilities.json
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("extras", "0121_customfield_related_object_filter"),
("netbox_librenms_plugin", "0001_initial"),
]
operations = [
migrations.AddField(
model_name="interfacetypemapping",
name="created",
field=models.DateTimeField(auto_now_add=True, null=True),
),
migrations.AddField(
model_name="interfacetypemapping",
name="custom_field_data",
field=models.JSONField(blank=True, default=dict, encoder=utilities.json.CustomFieldJSONEncoder),
),
migrations.AddField(
model_name="interfacetypemapping",
name="last_updated",
field=models.DateTimeField(auto_now=True, null=True),
),
migrations.AddField(
model_name="interfacetypemapping",
name="tags",
field=taggit.managers.TaggableManager(through="extras.TaggedItem", to="extras.Tag"),
),
]

View File

@@ -0,0 +1,26 @@
# Generated by Django 5.0.9 on 2024-10-17 10:19
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("netbox_librenms_plugin", "0002_interfacetypemapping_created_and_more"),
]
operations = [
migrations.AddField(
model_name="interfacetypemapping",
name="librenms_speed",
field=models.BigIntegerField(blank=True, null=True),
),
migrations.AlterField(
model_name="interfacetypemapping",
name="librenms_type",
field=models.CharField(max_length=100),
),
migrations.AlterUniqueTogether(
name="interfacetypemapping",
unique_together={("librenms_type", "librenms_speed")},
),
]

View File

@@ -0,0 +1,46 @@
# Generated by Django 5.1.4 on 2025-08-07 12:50
import taggit.managers
import utilities.json
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("extras", "0122_charfield_null_choices"),
("netbox_librenms_plugin", "0003_interfacetypemapping_librenms_speed_and_more"),
]
operations = [
migrations.CreateModel(
name="LibreNMSSettings",
fields=[
(
"id",
models.BigAutoField(auto_created=True, primary_key=True, serialize=False),
),
("created", models.DateTimeField(auto_now_add=True, null=True)),
("last_updated", models.DateTimeField(auto_now=True, null=True)),
(
"custom_field_data",
models.JSONField(
blank=True,
default=dict,
encoder=utilities.json.CustomFieldJSONEncoder,
),
),
(
"selected_server",
models.CharField(default="default", max_length=100),
),
(
"tags",
taggit.managers.TaggableManager(through="extras.TaggedItem", to="extras.Tag"),
),
],
options={
"verbose_name": "LibreNMS Settings",
"verbose_name_plural": "LibreNMS Settings",
},
),
]

Some files were not shown because too many files have changed in this diff Show More