💡The Big Question in MongoDB Schema Design: Arrays vs. Separate Documents

𝗛𝗮𝘃𝗲 𝘆𝗼𝘂 𝗘𝘃𝗲𝗿 𝗳𝗼𝘂𝗻𝗱 𝘆𝗼𝘂𝗿𝘀𝗲𝗹𝗳 𝗱𝗲𝗯𝗮𝘁𝗶𝗻𝗴 𝗯𝗲𝘁𝘄𝗲𝗲𝗻 𝘂𝘀𝗶𝗻𝗴 𝗮𝗿𝗿𝗮𝘆𝘀 𝗼𝗿 𝘀𝗲𝗽𝗮𝗿𝗮𝘁𝗲 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀 𝗶𝗻 𝗠𝗼𝗻𝗴𝗼𝗗𝗕?

Today, I came across this very scenario when working with appointment data that could exceed 50,000 records. Deciding the best approach is crucial for the performance and scalability of your application, so let’s explore this together!

The Scenario

Imagine this: you’re tasked with storing appointment information in MongoDB, and your dataset is rapidly growing. You have two example documents:

Document 1:

{
  "_id": "09fb5c63-3ff3-47a4-8638-e06cf42e3f3d",
  "orgId": "63c5cb11f5cf9a7d2c143967",
  "centerId": "3005",
  "appointmentDate": { "$date": "2024-10-15T04:00:00.000Z" },
  "appointmentTimeInMins": 540,
  "slotQueueNumber": 1,
  "clientIpFromServer": "106.222.222.176:26760",
  "clientIpFromBrowser": "106.222.222.176",
  "browserName": "Mozilla/5.0...",
  "visitCreationCompleted": false,
  "createdDate": { "$date": "2024-10-15T12:35:28.275Z" },
  "updatedDate": { "$date": "2024-10-15T12:35:28.275Z" }
}

Document 2:

{
  "_id": "ecb8fc9b-f593-4c5e-8700-f6eac9882d3f",
  "orgId": "63c5cb11f5cf9a7d2c143967",
  "centerId": "3005",
  "appointmentDate": { "$date": "2024-10-15T04:00:00.000Z" },
  "appointmentTimeInMins": 600,
  "slotQueueNumber": 1,
  "clientIpFromServer": "::1",
  "clientIpFromBrowser": "106.222.222.176",
  "browserName": "Mozilla/5.0...",
  "visitCreationCompleted": false,
  "createdDate": { "$date": "2024-10-15T12:54:18.653Z" },
  "updatedDate": { "$date": "2024-10-15T12:54:18.653Z" }
}

Here’s a sample code for both approaches: Array Structure and Separate Documents.

1. Array Structure Example

In this approach, we store multiple click events for a single day under the same document. We group these by centerId and appointmentDate.

Schema

{
  "_id": "centerId_appointmentDate",
  "centerId": "3005",
  "appointmentDate": { "$date": "2024-10-15T04:00:00.000Z" },
  "clickEvents": [
    {
      "eventId": "click-001",
      "appointmentTimeInMins": 540,
      "slotQueueNumber": 1,
      "clientIpFromBrowser": "106.222.222.176",
      "browserName": "Mozilla/5.0...",
      "userAgent": "Mozilla/5.0...",
      "createdDate": { "$date": "2024-10-15T12:35:28.275Z" }
    },
    {
      "eventId": "click-002",
      "appointmentTimeInMins": 600,
      "slotQueueNumber": 2,
      "clientIpFromBrowser": "106.222.222.177",
      "browserName": "Mozilla/5.0...",
      "userAgent": "Mozilla/5.0...",
      "createdDate": { "$date": "2024-10-15T13:00:00.000Z" }
    }
  ]
}

Code to Insert a New Click Event

db.clickEvents.update(
  { centerId: "3005", appointmentDate: new Date("2024-10-15T04:00:00.000Z") },
  { $push: { clickEvents: {
      eventId: "click-003",
      appointmentTimeInMins: 720,
      slotQueueNumber: 3,
      clientIpFromBrowser: "106.222.222.178",
      browserName: "Mozilla/5.0...",
      userAgent: "Mozilla/5.0...",
      createdDate: new Date()
    } }
  },
  { upsert: true }
);

• Pros: This structure allows for quick grouping and retrieval of all click events for a single day and center.

• Cons: If the number of events exceeds MongoDB’s document size limit (16 MB), it becomes unmanageable.

2. Separate Documents Example

In this approach, each click event is stored as a separate document. We still associate them by centerId and appointmentDate, but they are independent records.

Schema

{
  "_id": "click-001",
  "centerId": "3005",
  "appointmentDate": { "$date": "2024-10-15T04:00:00.000Z" },
  "appointmentTimeInMins": 540,
  "slotQueueNumber": 1,
  "clientIpFromBrowser": "106.222.222.176",
  "browserName": "Mozilla/5.0...",
  "userAgent": "Mozilla/5.0...",
  "createdDate": { "$date": "2024-10-15T12:35:28.275Z" }
}

Code to Insert a New Click Event

db.clickEvents.insertOne({
  _id: "click-003",
  centerId: "3005",
  appointmentDate: new Date("2024-10-15T04:00:00.000Z"),
  appointmentTimeInMins: 720,
  slotQueueNumber: 3,
  clientIpFromBrowser: "106.222.222.178",
  browserName: "Mozilla/5.0...",
  userAgent: "Mozilla/5.0...",
  createdDate: new Date()
});

• Pros: This structure scales well as each document is independent, supports efficient indexing, and avoids hitting the document size limit.

• Cons: It may result in more documents, but MongoDB handles large collections efficiently with indexing and sharding.

The Decision Point: Arrays vs. Separate Documents

At first, you might think, “Why not group these appointments in an array under centerId and appointmentDate as keys?” However, when the data volume can go beyond 50K entries, several factors come into play:

1. Document Size Limit 🚧

• MongoDB has a 16 MB document size limit. If you store all appointments in an array under a single document, you could quickly hit this limit. For large datasets like 50K+ records, storing everything in a single document isn’t scalable .

2. Query Performance ⚡

• Large arrays can significantly slow down queries. MongoDB will need to scan through the entire array, which isn’t efficient as the array grows. On the other hand, when using separate documents, MongoDB can efficiently index fields like centerId and appointmentDate, resulting in faster and more efficient queries .

3. Ease of Updates and Modifications 🔄

• Modifying or deleting an individual element within an array can be cumbersome and inefficient in MongoDB. When each appointment is stored as a separate document, updates and deletions become much simpler and more efficient. You can precisely target the document you want to update without affecting others .

The Recommended Approach: Separate Documents

After considering these factors, the best approach is to store each appointment as a separate document. Here’s why:

1. Scalability and Sharding 🚀

• By storing each appointment as its own document, you can leverage MongoDB’s sharding capabilities. This is essential when your dataset grows beyond the capacity of a single server. MongoDB distributes these documents across multiple nodes, ensuring the system remains performant .

2. Indexing and Query Optimization 📈

• With separate documents, you can create indexes on fields like centerId and appointmentDate, which makes your queries incredibly fast, even as the number of appointments grows into the millions.

3. Maintaining Data Integrity and Flexibility 🛠️

• Managing data as separate documents helps maintain data integrity. Each document is self-contained, meaning updates, modifications, and validations are straightforward. There’s no risk of hitting MongoDB’s document size limit, ensuring your system remains flexible.

How It Looks: The Ideal Schema

Here’s what a schema for storing separate documents might look like:

{
  "_id": "unique-appointment-id",
  "centerId": "3005",
  "appointmentDate": { "$date": "2024-10-15T04:00:00.000Z" },
  "appointmentTimeInMins": 540,
  "slotQueueNumber": 1,
  "clientIpFromServer": "106.222.222.176:26760",
  "clientIpFromBrowser": "106.222.222.176",
  "browserName": "Mozilla/5.0...",
  "visitCreationCompleted": false,
  "createdDate": { "$date": "2024-10-15T12:35:28.275Z" },
  "updatedDate": { "$date": "2024-10-15T12:35:28.275Z" }
}

Each appointment has its own document, making queries and updates simple, efficient, and fast.

Conclusion: Designing for Scalability

Choosing between arrays and separate documents isn’t just about what feels right; it’s about anticipating future growth and designing for scalability. If you’re working with data that’s expected to grow large, separate documents offer the best solution for flexibility, performance, and efficiency.

Next time you face this decision in MongoDB, consider the size and growth of your data, and remember: design for the future, not just the present.

Have you faced similar scenarios? How did you decide on your approach? Let’s discuss in the comments below!

❤️ Share Your Thoughts!

Feel free to repost ♻️ if you found this helpful. For more great content on microservices, follow 🛠 Apurv Upadhyay. Until next time, happy coding! 🚀

#MongoDB #DatabaseDesign #Scalability #DataModeling #NoSQL #DeveloperTips #PerformanceOptimization #Tech