Monitoring Options

• DBMS monitoring extensions: Extensions or add-ons within the database layer that provide insights on query efficiency, database connections, and more.
• Cloud provider monitoring tools: Most cloud providers include free monitoring for your database infrastructure to show metrics like CPU usage and network transfer. Some cloud providers also offer database monitoring as part of their managed database service
• External monitoring tools: Database-specific monitoring tools are designed to provide more insight beyond metrics like your underlying infrastructure’s CPU usage or network traffic. There is a wide variety of these tools available that fit different workloads. Here are a few examples of free open source database monitoring tools
  • Prometheus & MySQL Exporter: Built on top of the popular open source infrastructure monitoring tool, Prometheus, MySQL Server Exporter allows you to create a series of collector flags.
  • VictoriaMetrics: A time series database and monitoring solution to help process real-time metrics using the PromQL, MetricsQL, and Graphite query languages Best suited for small to medium size database environments Deploy via the Linode Marketplace
  • Percona Monitoring & Management: Optimize database performance and track behavior patterns for MySQL, PostgreSQL, and MongoDB Deploy via the Linode Marketplace.

When should you separate your database monitoring layer?

If you just want to keep an eye on CPU usage, relying on more standard infrastructure monitoring solutions will suffice. If you want to be able to observe performance at the query or table level, or check database logs during very specific points in time, an external monitoring tool will give you additional insight that can benefit your application

The post Designing Database Architecture – Incorporating Monitoring first appeared on Techcity Company Limited.

To address this issue, an architecture that implements database sharding can be used With sharding, a single, large table is split into multiple smaller tables The process of splitting a table is referred to as partitioning. When stored across multiple servers, these smaller tables are referred to as shards The databases in your cluster each store one of the shards Together, the databases in the cluster constitute your full data set For example, if a human resources application stored employee records for 1,000 companies, but found that the dataset size was too big for a single table, then the records could be split into two shards representing 500 companies each

MySQL NDB Cluster is a solution that provides automatic sharding for MySQL The diagram below shows
an example of how a MySQL NDB Cluster would fit into the example web application. This diagram omits the load balancer, cloud firewall, and web server cluster as those remain the same:

Database Sharding for Large Data Sets

The MySQL NDB Cluster in this diagram provides three MySQL servers that accept read and write SQL commands from the web servers The number of these servers can be horizontally scaled to meet demand and provide failover These servers do not store data Instead, they update and retrieve records from a separate set of data nodes.

The data nodes store shards of the dataset and run the ndbd data node daemon process Each shard can have multiple replica nodes, and you can configure how many replicas per shard there should be. The total number of data nodes is equal to the number of shards multiplied by the number of replicas per shard In the diagram above, there are two shards, and two replicas per shard (a primary replica and a secondary replica), for a total of four data nodes Having multiple replicas per shard allows for recovery for failing nodes, and this recovery process is automatic (similar to MySQL Group Replication).

Using sharding can offer very high scaling of dataset size, but it can also make your application logic more complex. In particular, you need to carefully configure how your data is partitioned into multiple shards, because this decision impacts overall database performance Sharding requires a more in-depth understanding of your database and the underlying infrastructure.

The post Designing Database Architecture – Horizontal Scaling with MySQL NDB Cluster and Database Sharding first appeared on Techcity Company Limited.

In the previous section, a two-server solution was presented that separated the web server from the database server While it did not solve vertical scaling’s problems, the separation of web servers and database servers is an important step towards creating a horizontal scaling solution.

In a horizontal scaling solution, the single web server is replaced with a high availability cluster of web servers, and the single database server is replaced with another, separate high availability cluster of database servers By having two separate clusters in your architecture for these services, you can horizontally scale them independently of each other and choose server instances with different hardware that is better suited for each workload.

There are several different ways to architect your database cluster, each with varying degrees of complexity and different trade-offs. This section and the diagram below illustrate how to use MySQL replicas to facilitate high availability.

Multi-Replication for Application High Availability

In the diagram above, a load balancer receives traffic from the Internet. A load balancer is a service that forwards and distributes traffic among another set of servers. Cloud providers generally offer load balancing as a service (Linode’s NodeBalancer solution is pictured). This service can also be implemented with open source software, including HAProxy, NGINX, and the Apache HTTP Server.

Before the forwarded traffic arrives at your web servers, it is intercepted by a cloud firewall. The cloud firewall lifts the firewalling burden from the operating system to the network level. Many cloud providers offer managed cloud firewalls, but this service can also be implemented with open source software on commodity hardware.

After being filtered, the traffic is distributed among a cluster of web servers. In many cases, these web servers can all be identical clones of each other. You can add or remove web servers in this cluster as needed to handle scaling traffic demand, which is an example of horizontal scalability. If a server fails, the load balancer will re-route traffic to other healthy servers, so the service remains available. Server creation can be automated with configuration management and Infrastructure as Code (IaC) tools like Ansible and Terraform.

The web servers in turn request data from a database cluster Unlike the web servers, the database server are not all identical in function In particular, only one database is set to receive write operations from the web application This server is designated as the primary database For example, if you operated a news website, the new articles that your writers published would be added to the database on this server.

The other databases in this cluster act as replicas of the primary database These servers receive all data added to the primary database through an asynchronous process Because this process is asynchronous, write operations on the primary database are still fast to execute These servers then receive all read operations from the web server cluster.

MySQL offers tools to help add new replicas to the cluster when needed, so your applications’ read operations can be horizontally scaled However, there is only one primary database in this architecture, so write operations cannot also be horizontally scaled. Still, this architecture can offer significant benefits to certain kinds of applications. In the news organization website example, the source of high-traffic demand is from readers of the website, not from writers adding articles. This traffic pattern aligns with the trade-offs in this setup.

There are a few other notable issues with this architecture First, when a new write operation is received by the primary database, it immediately persists the updated records The database does not wait for the replication process to complete, because that is asynchronous This means that in the event of a primary database failure, the replicas might not have the most recently updated records Also, when the primary database fails, it acts as a single point of failure for write operations Still, read operations will continue on the replicas in this scenario, so a website can continue to display existing content If the primary database fails, MySQL provides tools to manually promote one of the replicas to the position of the new primary database, after which write operations can resume.

Enhancements and alternative replication solutions exist to address these issues:

• MySQL replication can be configured to be semisynchronous instead of asynchronous This means that the primary database will wait for at least one replica to process new write operations before they are persisted Because at least one replica will have the latest data, you can be sure that you will not lose data in the event of a primary database failure However, semisynchronous replication is slower than asynchronous replication.
• MySQL Group Replication is a replication solution in which the database servers automatically coordinate with each other For example, in the event of a primary database outage, a replica server is automatically promoted to the new primary position. MySQL Group Replication can be configured with a single primary database, or it can be configured to have multiple primary databases that receive write operations. However, maintaining multiple primary databases involves other trade-offs.
• InnoDB Cluster bundles a MySQL Group Replication cluster with MySQL Router, which facilitates routing traffic from the web servers to the database cluster, and MySQL Shell, which is an advanced administration client for the cluster
• Galera is a multi-primary database solution where all databases in the cluster can receive write and read operations using completely synchronous replication between them It is also compatible with forks of MySQL like MariaDB and Percona.

The post Designing Database Architecture – Horizontal Scaling with MySQL Replication first appeared on Techcity Company Limited.

Provisioning a LAMP stack on a single node can be done in two steps

1. Deploy a Linux server, then install the other three software packages via your distribution’s package manager. Once the stack is provisioned, your web application’s code should be installed and configured. This work can often be automated with open source tooling; for example, WP-CLI can be used to install WordPress on a server
2. You need to consider the network security and availability of your server. Many cloud providers offer network filtering to prevent or reduce the impact of denial-of-service and other kinds of network attacks. However, it’s still important to fine-tune your network security configuration with a firewall.

In this section’s example architecture, the firewall is a process that runs on your server. The firewall intercepts network packets when they are first processed by the operating system and only allows traffic on certain network ports to proceed. For a web application, your firewall would allow HTTP (port 80) and HTTPS (port 443) traffic. Linux and other operating systems have built-in firewall options, like iptables, ufw, and FirewallD

Single Node LAMP Stack with Cloud Firewall

The resulting setup has one server running the LAMP stack where network traffic is filtered by a firewall also

running on the server There are limitations for this setup:

• Growth and scaling: With one server, you’re confined to that server’s memory, compute, and disk space resources. If your website starts seeing more traffic, you might need to increase the memory, compute, and/or disk space of the server to handle the load This is referred to as vertical scaling This process can vary in difficulty, depending on your cloud provider’s tooling, or on hardware availability for on-premise deployments. Vertical scaling can only offer a limited solution for capacity, as there are upper limits on the resources for a single server.
• Availability: Operating a single server means that there is a single point of failure for your web application. If the server needs hardware maintenance, or if one of the software components in the stack halts unexpectedly, then your web application will not be available To some extent, the issue of traffic growth can be addressed by moving the database to its own server, as in the following diagram:

Vertical Scaling with Separate Web and Database Nodes

By doing this, the compute, memory, and disk space needs of your web server and database can be adjusted independently This is still an example of vertical scaling, and it does not resolve availability issues.

The post Designing Database Architecture – Single Server Deployment and Vertical Scaling first appeared on Techcity Company Limited.

The typical real-world use case for a database is to store and handle structured information as part
of an application’s software stack. The foundation of a simple web application is the operating system of your server, with a web server, database, and application code layer running on top of it The OS provides basic networking functionality that the web server relies on to serve content In the code layer, a programming language and application software framework are used to direct the client-side UI in your users’ browser. We can look at a server running a WordPress site to illustrate these concepts:

Single Node Web Server Architecture

A WordPress site commonly leverages a software stack called LAMP, which is composed of the Linux operating system, Apache HTTP web server, MySQL database, and PHP programming language LAMP is one of the most popular web application software stacks in use. There are also alternatives to the LAMP stack. For example, you can use Python and an associated web application framework (like Django) instead of PHP Or, you can use the NGINX web server instead of Apache; this is referred to as the LEMP stack (the “E” comes from the pronunciation of NGINX, which is “Engine X”) There are many other variations of this concept, but the following scenario focuses on a traditional LAMP stack.

The post Designing Database Architecture – Single Node Web Server Architecture first appeared on Techcity Company Limited.

Many websites and web applications can be operated with a single-server configuration, where your database exists on the same server as all of the application’s other software components. This can be appropriate when your server (possibly when paired with external storage, like block storage) has enough compute and storage resources to accommodate your application’s data and traffic. This approach is fine for smaller, general use websites that might host some forms and media files. For example, a WordPress blog can run well on a single-server setup.

As your application’s features and data requirements grow, your database infrastructure needs additional flexibility and scalability to meet the performance expectations of your users or customers. Necessary changes could include moving the database to its own compute instance(s), or designing an advanced architecture with replicas, read-only instances, sharding, and/or other components These changes can enhance both the performance and security of your database.

Application performance is a shared responsibility Even if you or your team doesn’t currently prioritize conversion rates, poor application performance will come back to haunt you in other ways, including: decreased user trust, violation of your application’s SLA (the guaranteed availability of your services), and an increase in support tickets.

Research into human response indicates applications have roughly 100 milliseconds (ms)—one third of the time it takes to blink— before users feel like they’re waiting for a response

Lee Atchison
Caching at Scale with Redis

To help get you started, here are five examples of database-specific reference architecture that show how these setups differ, and their benefits. Please read more next post!

The post Designing Database Architecture first appeared on Techcity Company Limited.

Strengths

• “Ring” Architecture: Multiple nodes are organized logically in a “ring ” Each node can accept read and write requests so there is no single point of failure
• Ease of Use: Cassandra’s query language, CQL, has a syntax similar to traditional SQL and has a reduced learning curve for developers switching from a SQL database
• Write Speed: Cassandra is efficient for writing very large amounts of data. When performing a write operation, replicas of a new record are stored across multiple cluster nodes and these replicas are created in parallel Only a subset of those nodes need to complete a replica update for the write operation to be considered successful, which means that the write operation can finish sooner.

Weaknesses

• Read Time: Records in a Cassandra database are assigned a primary key attribute The value of the primary key determines which cluster nodes a record is stored on When querying data by primary key, read performance is fast, because the node that stores the data can be found quickly However, querying data using attributes other than the primary key is slower
• Not ACID-compliant out of the box. Instead, Cassandra offers several levels of trade-off between data consistency and availability for developers Use Cases: Personalization Engines, Content Management, Games, IOT Applications

The post Overview of Popular DBMSeS – Cassandra first appeared on Techcity Company Limited.

Strengths

• Speed: Since Redis only stores data in memory, it’s a highly-performant database that rapidly returns results
• Benchmarking: The built-in benchmark tool, redis-benchmark, provides insights on the average number of requests your Redis server is able to handle per second
• Ease of Use: Supports a variety of extensions, including RediSearch, which provides full text search for a Redis database

Weaknesses

• Memory Restrictions: Since Redis is an in-memory store, all data must fit within your available memory
• Rigidity: Redis doesn’t have a query language, and entering functions via the available commands can limit your ability to customize results
• Not ACID compliant out of the box. Additional configuration is required

Use Cases: Caching, Real-Time Data Updates

The post Overview of Popular DBMSeS – Redis first appeared on Techcity Company Limited.

Strengths

• Scalability: It’s in the name: Mongo, short for “humongous,” is built to store large volumes of data
• Search Flexibility: Supports graph search, geo-search, map-reduce queries, and text search
• Flexible Data Options: More functionality for temporary tables to support complex processes or test new applications/features without needing to switch databases

Weaknesses

• Concurrent Multi-Location Data Write Operations: Updating a consistent field in several locations (e g a commonly used location) can cause lags in performance compared to a relational database
• Query Formatting: MongoDB can have a steeper learning curve for developers who are familiar with other databases because queries are written with a JSON syntax, instead of SQL

Use Cases: Personalization Engines, Content Management, Games, IOT Applications

The post Overview of Popular DBMSeS – MongoDB first appeared on Techcity Company Limited.

Strengths

• True Open Source: Postgres is maintained by a global community, and a single version of the software is available that contains all features and has no fees.
• Extensibility: Postgres supports a wider range of data types and indices than MySQL, and additional feature extensions are provided by the open source community
• Data Encryption Options: Multiple encrypted connection options

Weaknesses

• Server Memory Demands: Postgres forks a new process for each new client connection Connections allocate a non-trivial amount of memory (about 10 MB)
• Increased Complexity over MySQL: Postgres adheres more strongly to SQL standards so it can be a difficult starter database for standard web applications. For example, querying in Postgres is case-sensitive

Use Cases: High-Scaling Applications, Multi-Database Workloads

The post Overview of Popular DBMSeS – PostgreSQL first appeared on Techcity Company Limited.

MySQL is one of the most widely used database engines MySQL is a component of the LAMP stack, which serves as the foundation for many content management systems, including WordPress MySQL is known for its high performance (especially for processing read operations), ease of use, and scalability Originally founded as an open source project, MySQL was acquired by Oracle in 2010 A free community edition of MySQL is still available, but some features and scalability require a paid enterprise license

Strengths

• Speed: Query limits and default read-only permissions make MySQL extremely performant, and there are more options available for memory-optimized tables
• Integrations: Due to its popularity, MySQL has a larger variety of third-party tools and integrations than other database types
• Reliability: MySQL is widely used and vendor-supported

Weaknesses

• Write Performance: Without optimizations, performance will decrease for write-heavy applications
• Tiers: Features are split between the free Community edition and the paid Enterprise edition, which limits access to features and potential scalability

Use Cases: Websites (General), Ecommerce, Catalog/Directory

The post Overview of Popular DBMSeS – MySQL first appeared on Techcity Company Limited.

Databases are anything but “one size fits all.” From privacy and compliance to supporting specific data types, databases only get more advanced as an application scales Even if you aren’t a database administrator (DBA), you need to factor in database capabilities and limitations when adding new features or optimizing existing features to your applications.

Current Database Landscape

The process of scaling applications and improving services can introduce new data types The need for more flexible data types has launched many new databases in the market in the past decade. As a result, database skills have become increasingly more valued by developers.

Most Wanted Database Skills Among Developers Worldwide as of 2021

According to the survey, just under 18% of respondents identified PostgreSQL as one of their most-wanted database skills. MongoDB ranked almost the same with software developers stating they are not developing with it, but want to. Source: Statista

Databases can be split into two types: relational and non-relational

• Relational Databases (also referred to as SQL Databases) Data is organized in tables, where the columns of a table correspond to the data’s attributes, including the type of an attribute For example, an employee table for an HR software suite could contain a column that stores an integer value representing an employee’s salary, and another column could store a text value representing their name The rows of the table represent instances of the data; for example, each row in an employee table would correspond to a different employee. Information between multiple tables is linked via keys. Relational databases use the Structured Query Language (SQL)
• Non-Relational Databases (also referred to as NoSQL (“Not Only SQL”) Databases) Non-relational databases have more flexible query methods. These query methods vary significantly by DBMS type. There are five types of non-relational databases:
  • Columnar Data Stores: Data is organized in a similar row-and-column table structure like
    in a relational database. However, the data is stored by column, and data from specific columns can be fetched In comparison, data in a relational database is stored by row, which allows you to fetch specific rows from the table.
  • Key-Value Stores: Data is stored as a collection of key-value pairs, where data can be retrieved by specifying a key that is associated with the data
  • Document Stores: Data is stored in documents, like those on your computer’s filesystem.
  • Document Data Stores: A document store where the files’ type is JSON, XML, or some other data-encoding format. The structure of the encoded data is flexible and can allow for complicated queries
  • Graph Databases: Data is stored using a graph structure, where entities correspond to nodes in the graph and the edges between nodes represent relationships between entities. An example: of a graph structure would be the connections between users on a social network Graph databases offer efficient querying for highly-interconnected data.

What is ACID Compliance?

ACID compliance for databases is a set of principles that ensures data integrity while processing a transaction These principles ensure that the data will not end up in an inconsistent state or be altered as a result of being added to the database, even if the transaction fails ACID compliance is particularly important for applications that are essential to the financial industry, with millions of transactions processed every second.

ACID Compliance:

• Atomic: Transactions are made up of components, and all components in a transaction must succeed Otherwise, the transaction will fail and the database will remain unchanged A transaction must have defined success or fail actions, or unit of work.
• Consistent: Successful transactions follow the database’s pre-defined parameters and restrictions.
• Isolated: Concurrent transactions are isolated from one another The result of executing several concurrent transactions will be the same as if they were executed in a sequence
• Durable: Once a transaction is written to the database, it will persist, even in the event of a system failure for the server that the database is running on Out of the box, NoSQL databases are not ACID compliant, because they are designed to prioritize flexibility and scalability instead of pure consistency

What is the BASE Model?

With the increasing use of NoSQL databases, a different transaction model was created to better align with their functionality Sticking with the chemistry theme, we have the BASE Model

BASE Model:

• Basically Available: Data is replicated and dispersed across clusters so that failures of part of a cluster should still leave the data available in other locations.
• Soft State: Consistency is not immediately enforced across the database cluster after an item in the database is updated During this time, fetching an updated record from the database may result
  in different values for different read operations.
• Eventually Consistent: Data will eventually reach consistency as updated records are replicated across the nodes in the database cluster.

The clear difference between the ACID and BASE transaction models is seen in the practical use of SQL vs NoSQL databases. NoSQL databases adhering to the BASE model are built for the flexibility and scalability required by massive high-availability deployments SQL databases adhering to the ACID model value consistency and data integrity above all else.

The post Evaluating the Strengths and Limitations of Database Options first appeared on Techcity Company Limited.