How Radical API Design Changed the Way We Access Databases – The New Stack

John Page

John Page is a document database veteran who, after 18 years of developing comprehensive document database technologies for the intelligence community, joined MongoDB. He now builds robots for fun and tests and writes to databases to pay for robot parts.

API design starts with knowing who your consumers are, how they will use your API, and how they will expect it to work. But too often, API design is more about dutifully adhering to established patterns and long-held conventions.

When we started working on MongoDB in 2007, we wanted to reinvent the way developers interacted with databases. The service API we created, the MongoDB Query API, was a radical approach because it allowed developers to access the database through the use of objects rather than pseudo-English text strings. MongoDB’s pure object API was revolutionary in 2007. Since then, the MongoDB Query API has inspired legions of imitators, even though we believe they correspond to a less feature-rich underlying database.

Back to SQL

For decades, developers had to request operations in a database using SQL, the structured query language designed for relational databases. When client-server applications arrived, SQL was wrapped in thin layers such as Data Access Objects (DAO) and Open Database Connectivity (ODBC) designed to send SQL statements to the database. and analyze the response.

SQL emerged at a time of dramatic changes in user interface design. Teletype printers gave way to green-on-black CRT displays, and a radical new editor was created to allow developers to view and edit multiple lines of code at once. This publisher was called Visuallater abbreviated to viwhich evolved into Vigor.

At the same time, a new data access user interface was created to allow users without formal math or programming training to construct complex queries. This user interface was called Structured English Query Language (SEQUEL) and allowed users to speak to a computer in formalized English to view and edit data. (It was later shortened to SQL for branding reasons.)

SQL’s original designers, Donald Chamberlin and Ray Boyce, thought it would be used by planners and other professionals — not database management experts — to perform ad hoc queries. Chamberlin wrote that he was surprised to see SQL so frequently used by trained database specialists for repetitive transactions. (Boyce died at age 26, shortly after an article featuring SEQUEL.)

Whether you write analytical queries or your application code interacts with the database, if you are a developer who understands what imperative coding is, SQL is not the best way to access a database. data.

At the same time, SQL isn’t just declarative either, not considering that the difference in performance between good and bad SQL queries, each performing the same task, is directly related to how the query is written.

A change of mentality

One of the first design decisions we made at MongoDB was to focus on interacting with the database using a purely object-based API. There would be no query language. Instead, each request to the database would be described as a set of objects intended to be constructed by a computer as much as a human (in many cases, more often by a computer).

This approach allowed programmers to treat a complex query in the same way as creating an imperative piece of code. Want to retrieve all the animals in your database that have exactly two legs? Then create an object, set a limb, “legs”, out of two, and query the database for matching objects. What you get back is an array of objects. This model extends to even the most complex operations.

This approach allowed developers to create database queries as codeit was a leap from a query language mindset to a programmer’s mindset. This would significantly speed up development time and improve query performance. This API approach to database operations helped launch the rapid adoption and growth of MongoDB in our early years.

The challenge of using object-oriented programming with tabular databases has been a source of friction for years. Object-relational mappers (ORMs) have proliferated as band-aids, obscuring the end-developer’s tabular model for basic use cases. But ORMs thrive at the expense of developer understanding and influence on performance, while optimization is still subject to the inherent frictions of the tabular database.

The practice of object-oriented programming

Thinking in objects is not always easy. Many novice MongoDB users use a JavaScript-based shell in an interactive client-side JavaScript REPL (read, evaluate, print, loop). Although you are actually writing programs, the experience is so immediate that you might mistake it for an SQL prompt rather than the code development playground that it really is. This can cause you to think of MongoDB queries as separate from the API you’re calling – you can think of them as JavaScript even when the rest of your code is in another language. To master MongoDB, it is important to consider queries not as a language in itself but as an object model.

I would go so far as to say that the most powerful queries are not written by hand but generated by higher level functions, and powerful aggregate queries should be considered imperative coding, more like Apache Spark than SQL.

For example, here are two ways to write the same query, one with a query language mindset and one with a programmer’s mindset. I use the classic programmer interview question FizzBuzz as an analytical query in both styles.

Although the total size of the code in this example is about the same, breaking it down makes it easier to read, write, debug, and maintain. For a less trivial case, this also creates a much smaller query code.

Conclusion

In the same way that MongoDB reinvented the concept of interacting with data and aligned it with developers, I urge you to examine your own service APIs and ensure that you are not blindly following an established pattern. Focus on who your consumers are, how they will use it, and how they will want to work with it.

And, while your APIs should initially be simple and easy to use, consider how they can be both extensible and backward compatible in later releases. Above all, don’t fall into situations where you’re afraid to break long-held conventions to better help your users. Remember, early cars were steered with levers rather than steering wheels. Redesigning one of the most common service API classes was a key part of MongoDB’s strategy.

Characteristic image via Pixabay.

About Jason Jones

Check Also

Android Auto to spice up your car with split-screen design and more smart elements

Android Auto gets an upgrade that lets you run apps in split-screen and enjoy a …