Parsing, Reading & Writing JSON With Python

JSON is the universal standard for exchanging textual data. You can encounter it in virtually any part of modern codebases, be it frontend, backend, APIs or database.

In this article, you’ll learn how to read, modify, and write JSON with Python.

What Is JSON?

JSON (JavaScript Object Notation) is a lightweight, text-based format for storing and transmitting data.

In JSON, data is organized in key-value pairs. Keys must be strings, while values can be in any of the supported types .

These key-value pairs can be further grouped into objects, which are separated by curly braces ({}), and arrays, which are separated by square brackets([]).

Here’s an example JSON with an array of two objects representing people, each holding a “name” value and an “age” value:

JSON is quite simple and easy to use, and it matches the syntax that popular languages like JavaScript and Python already use. For this reason, JSON is one of the most common formats for different kinds of APIs. It can also be used for data storage or configuration files.

How to Work With a JSON String in Python

A common way you’ll encounter JSON in Python is as a JSON string: a piece of text that’s formatted using JSON.

For example, you might receive a JSON response from an API you call on the internet:

import requests as r
 
todo = r.get("https://jsonplaceholder.typicode.com/todos/1").text
 
print(todo)
 
# {
#   "userId": 1,
#   "id": 1,
#   "title": "delectus aut autem",
#   "completed": false
# }

Here, todo is a Python string that follows the JSON format.

It’s usually not convenient to directly work with such strings in your Python program. To access their data, you want to convert them to a Python-specific representation that better reflects the structure of the data.

For example, what if instead of having the JSON object as a string, we had a Python dictionary whose keys and values reflected the keys and values of the JSON object? It would certainly make it much easier to work with it.

To convert the string, you can use Python’s json library. To use it, you need to import it at the top of the code.

import json

Its json.loads() (short for “load string”) function takes a JSON string, parses its contents, and converts objects to dictionaries, arrays to lists, and the data inside key-value pairs to data in equivalent Python data types.

import requests as r
import json

todo = r.get("https://jsonplaceholder.typicode.com/todos/1").text
todo_dict = json.loads(todo)

print(todo_dict["id"])
# 1 
print(todo_dict["title"])
# delectus aut autem
print(todo_dict["completed"])
# False

Once you have parsed the JSON string, you can easily make changes to it since it’s a regular Python structure. For example, here’s a function that capitalizes a todo’s title.

def capitalize_title(todo):
    todo['title'] = todo['title'].capitalize()
    return todo

You can use this function to capitalize the title of the parsed todo item.

todo_dict_capitalized = capitalize_title(todo_dict)

After making changes to data, you can use the json.dumps() (short for dump string) function to convert the structure back to a JSON string.

todo_json_updated = json.dumps(todo_dict_capitalized)
print(todo_json_updated)

Once executed, this should print the following in the terminal:

{"userId": 1, "id": 1, "title": "Delectus aut autem", "completed": false}

Whitespace (spaces, new lines, etc.) doesn’t matter in JSON, so it’s OK if the output string has different spacing than the input. To get the original spacing, you can add an indent=4 argument to the dumps() function.

todo_json_updated = json.dumps(todo_dict_capitalized, indent=4)

Now it should return the following:

{
    "userId": 1,
    "id": 1,
    "title": "Delectus aut autem",
    "completed": false
}

Here’s the full code for this section:

import requests as r
import json


def capitalize_title(todo):
    todo['title'] = todo['title'].capitalize()
    return todo


todo = r.get("https://jsonplaceholder.typicode.com/todos/1").text
todo_dict = json.loads(todo)

todo_dict_capitalized = capitalize_title(todo_dict)

todo_json_updated = json.dumps(todo_dict_capitalized, indent=4)
print(todo_json_updated)

How to Read a JSON File in Python

There are cases where you won’t receive JSON as a string, but will instead have a .json file.

First, you’ll need a file to use as an example. Create a new file called todos.json and paste in the first five todos from the mock API:

[
    {
        "userId": 1,
        "id": 1,
        "title": "delectus aut autem",
        "completed": false
    },
    {
        "userId": 1,
        "id": 2,
        "title": "quis ut nam facilis et officia qui",
        "completed": false
    },
    {
        "userId": 1,
        "id": 3,
        "title": "fugiat veniam minus",
        "completed": false
    },
    {
        "userId": 1,
        "id": 4,
        "title": "et porro tempora",
        "completed": true
    },
    {
        "userId": 1,
        "id": 5,
        "title": "laboriosam mollitia et enim quasi adipisci quia provident illum",
        "completed": false
    }
]

There are two steps to reading JSON with Python: opening the file itself and parsing the contents.

You can open a file with the open() function.

with open("posts.json") as f:

After that, you need to load the data from the file. Since a .json file is not a string but a binary, you need to use the load() function, not loads().

with open("posts.json") as f:
    todos = json.load(f)

Same as loads(), this will parse the JSON and return a Python structure that you can work with.

How to Use Python to Write a JSON File

In the same way as we did with the JSON string, we can capitalize and dump these todos back into a file.

Since there is now an array of objects, we need to iterate over it instead of capitalizing just one todo.

for todo in todos:
    todo = capitalize_title(todo)

Same as when reading from a file, you’ll need to use dump() not dumps() to write to a file. You’ll also need to supply the file you’re writing to as an argument to the function.

The code below will create a new JSON file called capitalized_todos.json and dump the JSON data there.

with open("capitalized_todos.json", "w+") as f:
    json.dump(todos, f, indent=4)

Here’s the full code for last two sections:

import json

def capitalize_title(todo):
    todo['title'] = todo['title'].capitalize()
    return todo

with open("todos.json", "r") as f:
    todos = json.load(f)

for todo in todos:
    todo = capitalize_title(todo)

with open("capitalized_todos.json", "w+") as f:
    json.dump(todos, f, indent=4)

Decoding JSON Into Custom Objects

By default, the json library converts JSON objects to Python dictionaries. This is good enough if you want to make quick, shallow changes to the data. But if you have some data that’s integral to your program, you’ll want to represent the data as a custom Python object instead.

This enables you to group data and functions that use it together, specify how the data will be printed to the console, as well as giving access to the dot notation for invoking functions.

Here’s an example Python class for todos:

class Todo:
    def __init__(self, user_id, id, title, completed):
        self.user_id = user_id
        self.id = id
        self.title = title
        self.completed = completed

    def __repr__(self):
        return f"Todo {self.id} by user {self.user_id}\n{self.title}\nCompleted: {self.completed}\n"

    def capitalize_title(self):
        self.title = self.title.capitalize()
        return self

    def complete(self):
        self.completed = True
        return self

By using class methods, it’s much easier for you to capitalize the title of any given todo and/or mark it as completed.

todo = Todo(1, 1, "delectus aut autem", True)
todo.capitalize_title().complete()

But this also creates a problem: how to convert JSON to this specific representation of todo items?

Decoding JSON to a particular object representation can be done using object hooks .) — functions that are called after decoding the JSON to a dictionary.

Here’s the object hook for the Todo class (add it to the class definition).

def from_dict(dct):
        user_id = dct["userId"]
        id = dct["id"]
        title = dct["title"]
        completed = dct["completed"]

        return Todo(user_id, id, title, completed)

It extracts the necessary values from the dictionary and initializes an object with those values.

All you need to do to use the hook is to add it as an argument to the loading function.

with open("todos.json", "r") as f:
    todos = json.load(f, object_hook=Todo.from_dict)

Now you can capitalize, complete, and print the todos using nice formatting.

for todo in todos:
    todo.capitalize_title().complete()
    print(todo)

Here’s the full code for this section:

import json

class Todo:
    def __init__(self, user_id, id, title, completed):
        self.user_id = user_id
        self.id = id
        self.title = title
        self.completed = completed

    def __repr__(self):
        return f"Todo {self.id} by user {self.user_id}\n{self.title}\nCompleted: {self.completed}\n"

    def capitalize_title(self):
        self.title = self.title.capitalize()
        return self

    def complete(self):
        self.completed = True
        return self

    def from_dict(dct):
        user_id = dct["userId"]
        id = dct["id"]
        title = dct["title"]
        completed = dct["completed"]

        return Todo(user_id, id, title, completed)

with open("todos.json", "r") as f:
    todos = json.load(f, object_hook=Todo.from_dict)

for todo in todos:
    todo.capitalize_title().complete()
    print(todo)

Encoding Custom Python Objects With JSONEncoder

The reverse process — dumping a Python object to JSON — is a bit harder.

Instead of making an object hook, you need to write a custom encoder class that inherits from JSONEncoder, the class responsible for encoding JSON in the json library. But the main idea is similar: you need to provide a method (this time called default) that shows how to convert the particular object to a dictionary.

class TodoEncoder(json.JSONEncoder):
    def default(self, obj):
        return {"userId": obj.user_id, "id": obj.id, "title": obj.title, "completed": obj.completed}

Now you can provide this class to the json.dump() function via the cls argument.

with open("capitalized_completed_todos.json", "w+") as f:
    json.dump(todos, f, indent=4, cls=TodoEncoder)

This should create a new file called capitalized_completed_todos.json that contains the todos properly converted to JSON.

Here’s the full code to decode, modify, and encode a JSON file using a custom Python class:

import json


class Todo:
    def __init__(self, user_id, id, title, completed):
        self.user_id = user_id
        self.id = id
        self.title = title
        self.completed = completed

    def __repr__(self):
        return f"Todo {self.id} by user {self.user_id}\n{self.title}\nCompleted: {self.completed}\n"

    def capitalize_title(self):
        self.title = self.title.capitalize()
        return self

    def complete(self):
        self.completed = True
        return self

    def from_dict(dct):
        user_id = dct["userId"]
        id = dct["id"]
        title = dct["title"]
        completed = dct["completed"]

        return Todo(user_id, id, title, completed)


with open("todos.json", "r") as f:
    todos = json.load(f, object_hook=Todo.from_dict)


class TodoEncoder(json.JSONEncoder):
    def default(self, obj):
        return {"userId": obj.user_id, "id": obj.id, "title": obj.title, "completed": obj.completed}

for todo in todos:
    todo.capitalize_title().complete()
    print(todo)

with open("capitalized_completed_todos.json", "w+") as f:
json.dump(todos, f, indent=4, cls=TodoEncoder)

Conclusion

In this tutorial, you learned three ways of decoding and encoding JSON with Python. These should cover most situations where you need to work with JSON in your Python programs.

To try out your skills with JSON in practice, you can try doing a web scraping project and storing its results in JSON for further use.

Additionally, you can experiment with some of the free JSON APIs that can be found on the internet.