Et dybtgående kig på databaseindeksering

Ydeevne er yderst vigtig i mange forbrugerprodukter som e-handel, betalingssystemer, spil, transportapps osv. Selvom databaser er optimeret internt gennem flere mekanismer for at opfylde deres præstationskrav i den moderne verden, afhænger meget også af applikationsudvikleren - når alt kommer til alt, er det kun en udvikler, der ved, hvilke spørgsmål applikationen skal udføre.

Udviklere, der beskæftiger sig med relationsdatabaser, har brugt eller i det mindste hørt om indeksering, og det er et meget almindeligt koncept i databaseverdenen. Den vigtigste del er imidlertid at forstå, hvad man skal indeksere, og hvordan indekseringen vil øge forespørgslens responstid. For at gøre det skal du forstå, hvordan du vil forespørge dine databasetabeller. Et korrekt indeks kan kun oprettes, når du ved nøjagtigt, hvordan dine forespørgsler og dataadgangsmønstre ser ud.

I simpel terminologi kortlægger et indeks søgetaster til tilsvarende data på disken ved hjælp af forskellige in-memory & on-disk datastrukturer. Indeks bruges til at gøre søgningen hurtigere ved at reducere antallet af poster, der skal søges efter.

Der oprettes hovedsageligt et indeks på de kolonner, der er specificeret i WHEREen forespørgsel, da databasen henter og filtrerer data fra tabellerne baseret på disse kolonner. Hvis du ikke opretter et indeks, scanner databasen alle rækkerne, filtrerer de matchende rækker ud og returnerer resultatet. Med millioner af poster kan denne scanningshandling tage mange sekunder, og denne høje responstid gør API'er og applikationer langsommere og ubrugelige. Lad os se et eksempel -

Vi bruger MySQL med en standard InnoDB-databasemotor, selvom begreber, der forklares i denne artikel, er mere eller mindre de samme i andre databaseservere som Oracle, MSSQL osv.

Opret en tabel kaldet index_demomed følgende skema:

CREATE TABLE index_demo ( name VARCHAR(20) NOT NULL, age INT, pan_no VARCHAR(20), phone_no VARCHAR(20) );

Hvordan verificerer vi, at vi bruger InnoDB-motoren?

Kør nedenstående kommando:

SHOW TABLE STATUS WHERE name = 'index_demo' \G;

Den Enginekolonne i ovenstående skærmbillede repræsenterer den motor, der bruges til at oprette tabellen. Her InnoDBbruges.

Indsæt nu nogle tilfældige data i tabellen, min tabel med 5 rækker ligner følgende:

Jeg har ikke oprettet noget indeks indtil nu på denne tabel. Lad os kontrollere dette ved kommandoen: SHOW INDEX. Det returnerer 0 resultater.

I øjeblikket, hvis vi kører en simpel SELECTforespørgsel, da der ikke er noget brugerdefineret indeks, scanner forespørgslen hele tabellen for at finde ud af resultatet:

EXPLAIN SELECT * FROM index_demo WHERE name = 'alex';

EXPLAINviser, hvordan forespørgselsmotoren planlægger at udføre forespørgslen. I ovenstående skærmbillede, kan du se, at rowskolonne afkast 5og possible_keysafkast null. possible_keysrepræsenterer, hvad alle tilgængelige indekser er der, der kan bruges i denne forespørgsel. Den keykolonne repræsenterer som indeks rent faktisk vil blive brugt af alle mulige indeks i denne forespørgsel.

Primærnøgle:

Ovenstående forespørgsel er meget ineffektiv. Lad os optimere denne forespørgsel. Vi vil gøre phone_nokolonnen til en PRIMARY KEYantagelse om, at der ikke kan eksistere to brugere i vores system med det samme telefonnummer. Tag følgende i betragtning, når du opretter en primær nøgle:

  • En primær nøgle skal være en del af mange vitale forespørgsler i din applikation.
  • Primær nøgle er en begrænsning, der entydigt identificerer hver række i en tabel. Hvis flere kolonner er en del af den primære nøgle, skal denne kombination være unik for hver række.
  • Primær nøgle skal være Ikke-nul. Gør aldrig felter, der ikke er i stand til, din primære nøgle. Ved ANSI SQL-standarder skal primære nøgler være sammenlignelige med hinanden, og du bør helt sikkert være i stand til at fortælle, om den primære nøglekolonneværdi for en bestemt række er større, mindre eller lig med den samme fra en anden række. Da NULLbetyder en udefineret værdi i SQL-standarder, kan du ikke deterministisk sammenligne NULLmed nogen anden værdi, så det er logisk NULLikke tilladt.
  • Den ideelle primære nøgletype skal være et tal som INTeller BIGINTfordi heltalssammenligninger er hurtigere, så at krydse gennem indekset vil være meget hurtigt.

Ofte definerer vi et idfelt som AUTO INCREMENTi tabeller og bruger det som en primær nøgle, men valget af en primær nøgle afhænger af udviklere.

Hvad hvis du ikke selv opretter nogen primær nøgle?

Det er ikke obligatorisk at oprette en primær nøgle selv. Hvis du ikke har defineret nogen primær nøgle, opretter InnoDB implicit en til dig, fordi InnoDB efter design skal have en primær nøgle i hver tabel. Så når du opretter en primær nøgle senere til denne tabel, sletter InnoDB den tidligere automatisk definerede primære nøgle.

Da vi ikke har nogen primær nøgle defineret fra nu af, lad os se, hvad InnoDB som standard oprettet for os:

SHOW EXTENDED INDEX FROM index_demo;

EXTENDED viser alle de indekser, der ikke kan bruges af brugeren, men administreres fuldstændigt af MySQL.

Her ser vi, at MySQL har defineret et sammensat indeks (vi vil diskutere sammensatte indeks senere) på DB_ROW_ID, DB_TRX_ID, DB_ROLL_PTR, og alle kolonner defineret i tabellen. I mangel af en brugerdefineret primærnøgle bruges dette indeks til at finde poster entydigt.

Hvad er forskellen mellem nøgle og indeks?

Selvom udtrykkene key& indexbruges om hverandre, keybetyder det en begrænsning, der pålægges kolonnens opførsel. I dette tilfælde er begrænsningen, at den primære nøgle er et ikke-null-felt, der entydigt identificerer hver række. På den anden side indexer der en speciel datastruktur, der letter datasøgning på tværs af tabellen.

Lad os nu oprette det primære indeks på phone_noog undersøge det oprettede indeks:

ALTER TABLE index_demo ADD PRIMARY KEY (phone_no); SHOW INDEXES FROM index_demo;

Bemærk, at det CREATE INDEXikke kan bruges til at oprette et primært indeks, men ALTER TABLEbruges.

I ovenstående skærmbillede ser vi, at der oprettes et primært indeks i kolonnen phone_no. Kolonnerne i de følgende billeder er beskrevet som følger:

Table : Tabellen, som indekset oprettes på.

Non_unique: Hvis værdien er 1, er indekset ikke entydigt, hvis værdien er 0, er indekset entydigt.

Key_name: Navnet på det oprettede indeks. Navnet på det primære indeks er altid PRIMARYi MySQL, uanset om du har angivet et indeksnavn eller ikke, mens du oprettede indekset.

Seq_in_index: Kolonnens sekvensnummer i indekset. Hvis flere kolonner er en del af indekset, tildeles sekvensnummeret baseret på, hvordan kolonnerne blev ordnet i løbet af indeksoprettelsestiden. Sekvensnummer starter fra 1.

Collation: how the column is sorted in the index. A means ascending, D means descending, NULL means not sorted.

Cardinality: The estimated number of unique values in the index. More cardinality means higher chances that the query optimizer will pick the index for queries.

Sub_part : The index prefix. It is NULL if the entire column is indexed. Otherwise, it shows the number of indexed bytes in case the column is partially indexed. We will define partial index later.

Packed: Indicates how the key is packed; NULL if it is not.

Null: YES if the column may contain NULL values and blank if it does not.

Index_type: Indicates which indexing data structure is used for this index. Some possible candidates are — BTREE, HASH, RTREE, or FULLTEXT.

Comment: The information about the index not described in its own column.

Index_comment: The comment for the index specified when you created the index with the COMMENT attribute.

Now let’s see if this index reduces the number of rows which will be searched for a given phone_no in the WHERE clause of a query.

EXPLAIN SELECT * FROM index_demo WHERE phone_no = '9281072002';

In this snapshot, notice that the rows column has returned 1 only, the possible_keys & key both returns PRIMARY . So it essentially means that using the primary index named as PRIMARY (the name is auto assigned when you create the primary key), the query optimizer just goes directly to the record & fetches it. It’s very efficient. This is exactly what an index is for — to minimize the search scope at the cost of extra space.

Clustered Index:

A clustered index is collocated with the data in the same table space or same disk file. You can consider that a clustered index is a B-Tree index whose leaf nodes are the actual data blocks on disk, since the index & data reside together. This kind of index physically organizes the data on disk as per the logical order of the index key.

What does physical data organization mean?

Physically, data is organized on disk across thousands or millions of disk / data blocks. For a clustered index, it’s not mandatory that all the disk blocks are contagiously stored. Physical data blocks are all the time moved around here & there by the OS whenever it’s necessary. A database system does not have any absolute control over how physical data space is managed, but inside a data block, records can be stored or managed in the logical order of the index key. The following simplified diagram explains it:

  • The yellow coloured big rectangle represents a disk block / data block
  • the blue coloured rectangles represent data stored as rows inside that block
  • the footer area represents the index of the block where red coloured small rectangles reside in sorted order of a particular key. These small blocks are nothing but sort of pointers pointing to offsets of the records.

Records are stored on the disk block in any arbitrary order. Whenever new records are added, they get added in the next available space. Whenever an existing record is updated, the OS decides whether that record can still fit into the same position or a new position has to be allocated for that record.

So position of records are completely handled by OS & no definite relation exists between the order of any two records. In order to fetch the records in the logical order of key, disk pages contain an index section in the footer, the index contains a list of offset pointers in the order of the key. Every time a record is altered or created, the index is adjusted.

In this way, you really don’t need to care about actually organizing the physical record in a certain order, rather a small index section is maintained in that order & fetching or maintaining records becomes very easy.

Advantage of Clustered Index:

This ordering or co-location of related data actually makes a clustered index faster. When data is fetched from disk, the complete block containing the data is read by the system since our disk IO system writes & reads data in blocks. So in case of range queries, it’s quite possible that the collocated data is buffered in memory. Say you fire the following query:

SELECT * FROM index_demo WHERE phone_no > '9010000000' AND phone_no < '9020000000'

A data block is fetched in memory when the query is executed. Say the data block contains phone_no in the range from 9010000000 to 9030000000 . So whatever range you requested for in the query is just a subset of the data present in the block. If you now fire the next query to get all the phone numbers in the range, say from 9015000000 to 9019000000 , you don’t need to fetch any more blocks from the disk. The complete data can be found in the current block of data, thus clustered_index reduces the number of disk IO by collocating related data as much as possible in the same data block. This reduced disk IO causes improvement in performance.

So if you have a well thought of primary key & your queries are based on the primary key, the performance will be super fast.

Constraints of Clustered Index:

Da et klynget indeks påvirker den fysiske organisering af dataene, kan der kun være et klynget indeks pr. Tabel.

Forholdet mellem primær nøgle og grupperet indeks:

Du kan ikke oprette et klynget indeks manuelt ved hjælp af InnoDB i MySQL. MySQL vælger det for dig. Men hvordan vælger det? Følgende uddrag er fra MySQL-dokumentation:

Når du definerer a PRIMARY KEYpå din tabel, skal du InnoDBbruge det som det klyngede indeks. Definer en primær nøgle for hver tabel, du opretter. Hvis der ikke er nogen logisk unik og ikke-nul-kolonne eller et sæt kolonner, skal du tilføje en ny kolonne med automatisk forøgelse, hvis værdier udfyldes automatisk.

Hvis du ikke definerer en PRIMARY KEYtil din tabel, finder MySQL det første UNIQUEindeks, hvor alle nøglekolonnerne er, NOT NULLog InnoDBbruger det som det klyngede indeks.

If the table has no PRIMARY KEY or suitable UNIQUE index, InnoDB internally generates a hidden clustered index named GEN_CLUST_INDEX on a synthetic column containing row ID values. The rows are ordered by the ID that InnoDB assigns to the rows in such a table. The row ID is a 6-byte field that increases monotonically as new rows are inserted. Thus, the rows ordered by the row ID are physically in insertion order.

In short, the MySQL InnoDB engine actually manages the primary index as clustered index for improving performance, so the primary key & the actual record on disk are clustered together.

Structure of Primary key (clustered) Index:

An index is usually maintained as a B+ Tree on disk & in-memory, and any index is stored in blocks on disk. These blocks are called index blocks. The entries in the index block are always sorted on the index/search key. The leaf index block of the index contains a row locator. For the primary index, the row locator refers to virtual address of the corresponding physical location of the data blocks on disk where rows reside being sorted as per the index key.

In the following diagram, the left side rectangles represent leaf level index blocks, and the right side rectangles represent the data blocks. Logically the data blocks look to be aligned in a sorted order, but as already described earlier, the actual physical locations may be scattered here & there.

Is it possible to create a primary index on a non-primary key?

In MySQL, a primary index is automatically created, and we have already described above how MySQL chooses the primary index. But in the database world, it’s actually not necessary to create an index on the primary key column — the primary index can be created on any non primary key column as well. But when created on the primary key, all key entries are unique in the index, while in the other case, the primary index may have a duplicated key as well.

Is it possible to delete a primary key?

It’s possible to delete a primary key. When you delete a primary key, the related clustered index as well as the uniqueness property of that column gets lost.

ALTER TABLE `index_demo` DROP PRIMARY KEY; - If the primary key does not exist, you get the following error: "ERROR 1091 (42000): Can't DROP 'PRIMARY'; check that column/key exists"

Advantages of Primary Index:

  • Primary index based range queries are very efficient. There might be a possibility that the disk block that the database has read from the disk contains all the data belonging to the query, since the primary index is clustered & records are ordered physically. So the locality of data can be provided by the primary index.
  • Any query that takes advantage of primary key is very fast.

Disadvantages of Primary Index:

  • Since the primary index contains a direct reference to the data block address through the virtual address space & disk blocks are physically organized in the order of the index key, every time the OS does some disk page split due to DML operations like INSERT / UPDATE / DELETE, the primary index also needs to be updated. So DML operations puts some pressure on the performance of the primary index.

Secondary Index:

Any index other than a clustered index is called a secondary index. Secondary indices does not impact physical storage locations unlike primary indices.

When do you need a Secondary Index?

You might have several use cases in your application where you don’t query the database with a primary key. In our example phone_no is the primary key but we may need to query the database with pan_no, or name . In such cases you need secondary indices on these columns if the frequency of such queries is very high.

How to create a secondary index in MySQL?

The following command creates a secondary index in the name column in the index_demo table.

CREATE INDEX secondary_idx_1 ON index_demo (name);

Structure of Secondary Index:

In the diagram below, the red coloured rectangles represent secondary index blocks. Secondary index is also maintained in the B+ Tree and it’s sorted as per the key on which the index was created. The leaf nodes contain a copy of the key of the corresponding data in the primary index.

So to understand, you can assume that the secondary index has reference to the primary key’s address, although it’s not the case. Retrieving data through the secondary index means you have to traverse two B+ trees — one is the secondary index B+ tree itself, and the other is the primary index B+ tree.

Advantages of a Secondary Index:

Logically you can create as many secondary indices as you want. But in reality how many indices actually required needs a serious thought process since each index has its own penalty.

Disadvantages of a Secondary Index:

With DML operations like DELETE / INSERT , the secondary index also needs to be updated so that the copy of the primary key column can be deleted / inserted. In such cases, the existence of lots of secondary indexes can create issues.

Also, if a primary key is very large like a URL, since secondary indexes contain a copy of the primary key column value, it can be inefficient in terms of storage. More secondary keys means a greater number of duplicate copies of the primary key column value, so more storage in case of a large primary key. Also the primary key itself stores the keys, so the combined effect on storage will be very high.

Consideration before you delete a Primary Index:

In MySQL, you can delete a primary index by dropping the primary key. We have already seen that a secondary index depends on a primary index. So if you delete a primary index, all secondary indices have to be updated to contain a copy of the new primary index key which MySQL auto adjusts.

This process is expensive when several secondary indexes exist. Also other tables may have a foreign key reference to the primary key, so you need to delete those foreign key references before you delete the primary key.

When a primary key is deleted, MySQL automatically creates another primary key internally, and that’s a costly operation.

UNIQUE Key Index:

Like primary keys, unique keys can also identify records uniquely with one difference — the unique key column can contain null values.

Unlike other database servers, in MySQL a unique key column can have as many null values as possible. In SQL standard, null means an undefined value. So if MySQL has to contain only one null value in a unique key column, it has to assume that all null values are the same.

But logically this is not correct since null means undefined — and undefined values can’t be compared with each other, it’s the nature of null. As MySQL can’t assert if all nulls mean the same, it allows multiple null values in the column.

The following command shows how to create a unique key index in MySQL:

CREATE UNIQUE INDEX unique_idx_1 ON index_demo (pan_no);

Composite Index:

MySQL lets you define indices on multiple columns, up to 16 columns. This index is called a Multi-column / Composite / Compound index.

Let’s say we have an index defined on 4 columns — col1, col2, col3, col4. With a composite index, we have search capability on col1, (col1, col2) , (col1, col2, col3) , (col1, col2, col3, col4). So we can use any left side prefix of the indexed columns, but we can’t omit a column from the middle & use that like — (col1, col3) or (col1, col2, col4) or col3 or col4 etc. These are invalid combinations.

The following commands create 2 composite indexes in our table:

CREATE INDEX composite_index_1 ON index_demo (phone_no, name, age); CREATE INDEX composite_index_2 ON index_demo (pan_no, name, age);

If you have queries containing a WHERE clause on multiple columns, write the clause in the order of the columns of the composite index. The index will benefit that query. In fact, while deciding the columns for a composite index, you can analyze different use cases of your system & try to come up with the order of columns that will benefit most of your use cases.

Composite indices can help you in JOIN & SELECT queries as well. Example: in the following SELECT * query, composite_index_2 is used.

When several indexes are defined, the MySQL query optimizer chooses that index which eliminates the greatest number of rows or scans as few rows as possible for better efficiency.

Why do we use composite indices? Why not define multiple secondary indices on the columns we are interested in?

MySQL uses only one index per table per query except for UNION. (In a UNION, each logical query is run separately, and the results are merged.) So defining multiple indices on multiple columns does not guarantee those indices will be used even if they are part of the query.

MySQL maintains something called index statistics which helps MySQL infer what the data looks like in the system. Index statistics is a generilization though, but based on this meta data, MySQL decides which index is appropriate for the current query.

How does composite index work?

The columns used in composite indices are concatenated together, and those concatenated keys are stored in sorted order using a B+ Tree. When you perform a search, concatenation of your search keys is matched against those of the composite index. Then if there is any mismatch between the ordering of your search keys & ordering of the composite index columns, the index can’t be used.

In our example, for the following record, a composite index key is formed by concatenating pan_no, name, ageHJKXS9086Wkousik28.

+--------+------+------------+------------+ name age pan_no phone_no +--------+------+------------+------------+ kousik 28 HJKXS9086W 9090909090

How to identify if you need a composite index:

  • Analyze your queries first according to your use cases. If you see certain fields are appearing together in many queries, you may consider creating a composite index.
  • If you are creating an index in col1 & a composite index in (col1, col2), then only the composite index should be fine. col1 alone can be served by the composite index itself since it’s a left side prefix of the index.
  • Consider cardinality. If columns used in the composite index end up having high cardinality together, they are good candidate for the composite index.

Covering Index:

A covering index is a special kind of composite index where all the columns specified in the query somewhere exist in the index. So the query optimizer does not need to hit the database to get the data — rather it gets the result from the index itself. Example: we have already defined a composite index on (pan_no, name, age) , so now consider the following query:

SELECT age FROM index_demo WHERE pan_no = 'HJKXS9086W' AND name = 'kousik'

The columns mentioned in the SELECT & WHERE clauses are part of the composite index. So in this case, we can actually get the value of the age column from the composite index itself. Let’s see what the EXPLAIN command shows for this query:

EXPLAIN FORMAT=JSON SELECT age FROM index_demo WHERE pan_no = 'HJKXS9086W' AND name = '111kousik1';

In the above response, note that there is a key — using_index which is set to true which signifies that the covering index has been used to answer the query.

I don’t know how much covering indices are appreciated in production environments, but apparently it seems to be a good optimization in case the query fits the bill.

Partial Index:

We already know that Indices speed up our queries at the cost of space. The more indices you have, the more the storage requirement. We have already created an index called secondary_idx_1 on the column name. The column name can contain large values of any length. Also in the index, the row locators’ or row pointers’ metadata have their own size. So overall, an index can have a high storage & memory load.

In MySQL, it’s possible to create an index on the first few bytes of data as well. Example: the following command creates an index on the first 4 bytes of name. Though this method reduces memory overhead by a certain amount, the index can’t eliminate many rows, since in this example the first 4 bytes may be common across many names. Usually this kind of prefix indexing is supported on CHAR ,VARCHAR, BINARY, VARBINARY type of columns.

CREATE INDEX secondary_index_1 ON index_demo (name(4));

What happens under the hood when we define an index?

Let’s run the SHOW EXTENDED command again:

SHOW EXTENDED INDEXES FROM index_demo;

We defined secondary_index_1 on name, but MySQL has created a composite index on (name, phone_no) where phone_no is the primary key column. We created secondary_index_2 on age & MySQL created a composite index on (age, phone_no). We created composite_index_2 on (pan_no, name, age) & MySQL has created a composite index on (pan_no, name, age, phone_no). The composite index composite_index_1 already has phone_no as part of it.

So whatever index we create, MySQL in the background creates a backing composite index which in-turn points to the primary key. This means that the primary key is a first class citizen in the MySQL indexing world. It also proves that all the indexes are backed by a copy of the primary index —but I am not sure whether a single copy of the primary index is shared or different copies are used for different indexes.

There are many other indices as well like Spatial index and Full Text Search index offered by MySQL. I have not yet experimented with those indices, so I’m not discussing them in this post.

General Indexing guidelines:

  • Since indices consume extra memory, carefully decide how many & what type of index will suffice your need.
  • With DML operations, indices are updated, so write operations are quite costly with indexes. The more indices you have, the greater the cost. Indexes are used to make read operations faster. So if you have a system that is write heavy but not read heavy, think hard about whether you need an index or not.
  • Cardinality is important — cardinality means the number of distinct values in a column. If you create an index in a column that has low cardinality, that’s not going to be beneficial since the index should reduce search space. Low cardinality does not significantly reduce search space.

    Example: if you create an index on a boolean ( int1 or 0 only ) type column, the index will be very skewed since cardinality is less (cardinality is 2 here). But if this boolean field can be combined with other columns to produce high cardinality, go for that index when necessary.

  • Indices might need some maintenance as well if old data still remains in the index. They need to be deleted otherwise memory will be hogged, so try to have a monitoring plan for your indices.

In the end, it’s extremely important to understand the different aspects of database indexing. It will help while doing low level system designing. Many real-life optimizations of our applications depend on knowledge of such intricate details. A carefully chosen index will surely help you boost up your application’s performance.

Venligst klapp og del med dine venner og på sociale medier, hvis du kan lide denne artikel. :)

Referencer:

  1. //dev.mysql.com/doc/refman/5.7/da/innodb-index-types.html
  2. //www.quora.com/What-is-difference-mellem-primary-index-and-secondary-index-exactly-And-whats-fordel-of-one-over-another
  3. //dev.mysql.com/doc/refman/8.0/en/create-index.html
  4. //www.oreilly.com/library/view/high-performance-mysql/0596003064/ch04.html
  5. //www.unofficialmysqlguide.com/covering-indexes.html
  6. //dev.mysql.com/doc/refman/8.0/en/multiple-column-indexes.html
  7. //dev.mysql.com/doc/refman/8.0/da/show-index.html
  8. //dev.mysql.com/doc/refman/8.0/en/create-index.html