Working with JSON Data in Ansible

Suresh Vina -

Hello! In today's post, we're going to explore how to work with JSON data in Ansible. JSON, or JavaScript Object Notation, is a common format for data interchange, and it's often used in automation scenarios, especially when dealing with APIs or configuration files. We'll dive into handling and extracting data from a nested JSON structure, demonstrating how Ansible can be used to parse and retrieve the specific information you need.

Sample JSON

Here's a sample of a nested, somewhat complex JSON structure that we will use in our examples. This JSON represents a hypothetical scenario where you have information about a company, including details about its departments, employees, and some project data

{
  "company": "TechCorp",
  "location": "San Francisco",
  "departments": [
    {
      "name": "Development",
      "manager": "Alice Johnson",
      "employees": [
        {
          "name": "John Doe",
          "role": "Senior Developer",
          "skills": ["Python", "Django", "React"]
        },
        {
          "name": "Jane Smith",
          "role": "Junior Developer",
          "skills": ["Python", "Flask"]
        }
      ],
      "projects": [
        {
          "name": "Project Alpha",
          "status": "In Progress"
        },
        {
          "name": "Project Beta",
          "status": "Completed"
        }
      ]
    },
    {
      "name": "Marketing",
      "manager": "Bob Brown",
      "employees": [
        {
          "name": "Rick White",
          "role": "SEO Specialist",
          "skills": ["SEO", "Content Marketing"]
        },
        {
          "name": "Sara Adams",
          "role": "Social Media Manager",
          "skills": ["Social Media", "Branding"]
        }
      ],
      "projects": [
        {
          "name": "Brand Awareness Campaign",
          "status": "In Progress"
        },
        {
          "name": "New Product Launch",
          "status": "Planned"
        }
      ]
    }
  ]
}

Reading JSON Data from a File in Ansible

Instead of embedding the JSON data directly in the vars section of your Ansible playbook, you can read it from an external JSON file. This approach is cleaner and more manageable, especially with complex JSON structures. Here's how you can modify the Ansible playbook to read JSON data from a file.

---
- name: Read JSON Data from a File
  hosts: localhost
  gather_facts: False
  tasks:
    - name: Load company data from JSON file
      set_fact:
        company_data: "{{ lookup('file', 'company_data.json') | from_json }}"

The set_fact module is used to define a new variable named company_data. This variable is assigned the contents of a JSON file named 'company_data.json'. The file is read using Ansible's lookup plugin with the 'file' parameter, which fetches the raw contents of the file. The from_json filter then parses these contents, converting them from a JSON formatted string into a data structure (like a dictionary or list) that Ansible can work with. This process effectively loads the JSON data into the company_data variable for use in subsequent tasks within the playbook.

Working with JSON Data in Ansible - Simple Example

Let's start with a straightforward example of how to work with JSON data in Ansible. In this initial example, we'll extract a simple piece of information from the JSON structure provided.

Suppose you want to extract the name of the company from the JSON data. This is a basic operation, but it's a good starting point for understanding how to navigate JSON structures in Ansible.

---
- name: Read JSON Data from a File
  hosts: localhost
  gather_facts: False
  
  tasks:
    - name: Load company data from JSON file
      set_fact:
        company_data: "{{ lookup('file', 'company_data.json') | from_json }}"
    
    - name: Get the name of the company
      debug:
        msg: "The name of the company is {{ company_data.company }}"
  • The variable company_data holds the JSON structure.
  • We use the debug module to print the name of the company, which is accessed using the dot notation (company_data.company).
  • The output will be "The name of the company is TechCorp"

Extracting All Employee Names

In this second example, we'll extract a list of all employee names (not including names of managers) from the JSON file. Assuming the JSON data is stored in company_data.json, here's how you can achieve this:

This playbook will read the JSON data from the file and then extract the names of all employees across all departments.

---
- name: Read and Process JSON Data from File
  hosts: localhost
  gather_facts: False
  
  tasks:
    - name: Load company data from JSON file
      set_fact:
        company_data: "{{ lookup('file', 'company_data.json') | from_json }}"
    
    - name: Extract all employee names
      set_fact:
        employee_names: "{{ company_data.departments | map(attribute='employees') | flatten | map(attribute='name') }}"
    
    - name: Print all employee names
      debug:
        msg: "Employee names: {{ employee_names }}"
  • The lookup plugin reads company_data.json and from_json parses it into a structured format.
  • The map filter is applied to a sequence (like a list) and allows you to specify an attribute or a method to be applied to each element in the sequence. In essence, it "maps" a function over all the elements of the list, transforming each element based on the attribute or method you specify.
  • company_data.departments | map(attribute='employees') Here, the map filter is used to extract the employees attribute from each element in the departments list. Since departments is a list of dictionaries (each representing a department), this operation results in a new list where each element is the list of employees from each department.
  • This intermediate result is a list of lists (each inner list containing employees of one department). To turn this into a single list of employees, the flatten filter is used.
  • flatten | map(attribute='name') After flattening, we have a single list of employees (where each employee is represented as a dictionary). We apply the map filter again, this time to extract the name attribute from each employee dictionary. This creates a list of employee names.

If you want to get the names of both employees and managers, then you can use the below filter.

- name: Extract names of managers and employees
      set_fact:
        all_names: "{{ company_data.departments | map(attribute='manager') | list + (company_data.departments | map(attribute='employees') | flatten | map(attribute='name') }}"

Extract Specific Items from JSON

Let's look at another example of manipulating JSON data using the standard filters available in Ansible. In this example, we will extract and list the skills of all employees in a specific department. Let's say we want to create a unique list of all skills possessed by employees in the Development department.

---
- name: Extract Skills of Employees in Development Department
  hosts: localhost
  gather_facts: False
  
  tasks:
    - name: Load company data from JSON file
      set_fact:
        company_data: "{{ lookup('file', 'company_data.json') | from_json }}"
        
    - name: Extract skills from Development department employees
      set_fact:
        development_skills: "{{ company_data.departments | selectattr('name', 'equalto', 'Development') | map(attribute='employees') | flatten | map(attribute='skills') | flatten | unique }}"
    
    - name: Print skills of Development department employees
      debug:
        msg: "Skills in Development Department: {{ development_skills }}"
  • We load the JSON data from company_data.json.
  • To extract the skills, we first filter the department by name (selectattr('name', 'equalto', 'Development')).
  • We then use map(attribute='employees') to get the list of employees in the Development department.
  • The flatten filter is used to change the list of lists into a single list.
  • Next, map(attribute='skills') extracts the skills of each employee, resulting in another list of lists.
  • Another flatten is applied to create a single list of skills.
  • Finally, the unique filter removes any duplicate entries, leaving us with a unique set of skills.
  • The development_skills variable now holds this list, and we use debug to print it.

Closing Up

And there we have it, a straightforward look at handling JSON data in Ansible. This skill is incredibly useful for a wide range of automation tasks, especially in today's data-driven environments.