Grafana Dashboard Provisioning: From YAML Nightmares to GitOps Automation

Let me be blunt: the official Grafana provisioning docs are garbage.

Not trying to be edgy. Last week our team migrated 50+ production dashboards from manual imports to a GitOps pipeline, and the number of landmines we hit could fill a book titled Grafana: A Tragedy in YAML. Someone on r/sre said “configuring provisioning is more painful than writing the dashboards themselves” — I felt that in my soul.

But once it works? It’s beautiful.

Why Bother with Provisioning?

Manually importing dashboards through the UI? That’s a 2020 move.

• Environment consistency: Dev, staging, prod — three manual imports means three chances to screw up. Every. Single. Time.
• Version control: JSON files in Git. Something breaks? git revert. No begging the DBA to restore the Grafana database.
• Multi-tenant isolation: Different orgs need different dashboards? Provisioning handles it natively. No hacky scripts hitting the API.

Directory Structure: Don’t Wing It

Grafana reads from /etc/grafana/provisioning/ by default. Here’s the layout:

provisioning/
├── dashboards/          # Dashboard providers
│   ├── main.yaml
│   └── sample.json
├── datasources/         # Data source configs
│   └── prometheus.yaml
├── notifiers/           # Notification channels (5.0+)
│   └── slack.yaml
├── plugins/             # Plugin configs (7.0+)
└── alerting/            # Alert rules (8.0+)

Critical point: The dashboards/ directory holds YAML provider configs, not raw JSON dashboard files. The YAML tells Grafana where to find the actual JSON definitions.

Your First YAML Provider Config

Simplest setup — loading dashboards from the local filesystem:

# /etc/grafana/provisioning/dashboards/main.yaml
apiVersion: 1

providers:
  - name: 'Production Dashboards'
    orgId: 1
    folder: 'Production'
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

Key parameters explained:

• disableDeletion: true — Prevents someone from deleting a dashboard in the UI. If it’s in Git, it stays. Set this to true unless you enjoy phantom re-appearing dashboards.
• updateIntervalSeconds: 30 — Grafana scans the directory every 30 seconds. Too low wastes CPU, too high means slow change propagation.
• allowUiUpdates: false — Blocks UI edits. If you’re doing GitOps, UI changes get wiped on the next sync anyway. Don’t let people waste their time.

Multi-DataSource Config: The Trap

Data sources go in datasources/:

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 15s
      queryTimeout: 60s

Here’s the trap: Only one data source can have isDefault: true. Set it on multiple sources and Grafana picks one at random. This blew up our staging environment — Prometheus and Thanos both had isDefault: true, so half the dashboards hit Prometheus and the other half hit Thanos. Data didn’t match. Took two hours to debug.

Multi-Org Provisioning: The Docs Skip This

Someone on Reddit asked “how do I provision dashboards for different orgs?” The official answer is basically “figure it out yourself.”

Here’s the actual solution:

# provisioning/dashboards/org1.yaml
apiVersion: 1

providers:
  - name: 'Org1 Dashboards'
    orgId: 1
    folder: 'Team Alpha'
    type: file
    options:
      path: /var/lib/grafana/dashboards/org1

---

# provisioning/dashboards/org2.yaml
apiVersion: 1

providers:
  - name: 'Org2 Dashboards'
    orgId: 2
    folder: 'Team Beta'
    type: file
    options:
      path: /var/lib/grafana/dashboards/org2

Keep directory paths separate. Otherwise Org1’s dashboards show up in Org2’s list with N/A data sources. Not fun.

JSON Dashboard Templating: Variable Injection

Hardcoding JSON is amateur hour. Use Go template syntax:

{
  "title": "{{ .Name }} - Overview",
  "panels": [
    {
      "title": "CPU Usage",
      "targets": [
        {
          "expr": "avg(rate(node_cpu_seconds_total{mode!=\"idle\", instance=\"{{ .Instance }}\"}[5m]))"
        }
      ]
    }
  ]
}

Pass variables in the YAML provider:

providers:
  - name: 'Template Dashboards'
    orgId: 1
    type: file
    options:
      path: /var/lib/grafana/dashboards
    jsonData:
      Name: "Web Server"
      Instance: "web-01:9100"

This requires Grafana 7.0+. Don’t try it on older versions.

Docker Compose Integration

Our production setup uses Docker Compose with volume mounts:

version: '3.8'

services:
  grafana:
    image: grafana/grafana:10.4.2
    volumes:
      - ./provisioning:/etc/grafana/provisioning
      - ./dashboards:/var/lib/grafana/dashboards
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

Start the container and dashboards load automatically. Someone on Reddit asked “how to auto-load dashboards in Docker” — mount the provisioning directory. That’s it.

Common Failure Modes

FAQ

Q: What’s the difference between provisioning and the Grafana API?

Provisioning is declarative — Grafana loads configs on startup, perfect for GitOps. The API is imperative, useful for dynamic scenarios. Production should use provisioning as the primary method, API as a supplement.

Q: Can I use provisioning and the UI simultaneously?

Technically yes, practically no. allowUiUpdates: true lets you edit in the UI, but the next provisioning sync overwrites everything. Best practice: provisioning only, UI for read-only viewing.

Q: Which data sources does provisioning support?

All official data sources: Prometheus, InfluxDB, Elasticsearch, MySQL, PostgreSQL, CloudWatch, Azure Monitor, and more. Third-party plugins need extra configuration.

Q: How do I debug provisioning failures?

Check Grafana logs: docker logs grafana | grep -i provisioning. Common issues: wrong file paths, permission errors, malformed YAML. Run yamllint on your configs.

Q: Does provisioning support alert rules?

Yes, in Grafana 8.0+. Put YAML files in the alerting/ directory. The syntax is more complex than dashboards — start with the official examples.

References & Community Insights

The following authoritative resources were referenced for architectural best practices and specifications:

• Google Site Reliability Engineering Book
• OpenTelemetry Documentation
• Prometheus Best Practices