主页 / dba / 问题 / 114471
Accepted
Nawaz Sohail
Asked: 2015-09-09 06:50:19 +0800 CST

计算表的行大小和最大行大小

  • 772

有什么方法可以计算表格占用的字节数吗？我知道您可以从中获得一些信息，information_schema.tables但这些信息不够准确。

实际需要的是根据InnoDB表定义的字节数，排序规则也可以被认为是utf-8-general-ci。

例如，一个表test如下

create table test   (
   col1 varchar(25),
   col2 int,
   col3 varchar(3),
   col4 char(15),
   col5 datetime    
);

根据表中列的类型，我需要知道一行中可以累积的总行大小。

为 MS SQL Server找到了某种类似的解决方案，但需要它的 MySQL 版本

mysql database-design

6 个回答

  1. Best Answer

    经过大量思考和研究，找到了一个真正有助于实现所需的答案。这是一个 perl 脚本，参考链接是

    http://dev.mysql.com/doc/refman/5.6/en/storage-requirements.html

    #!/usr/bin/perl
use strict;
$| = 1;

my %DataType = (
"TINYINT"=>1, "SMALLINT"=>2, "MEDIUMINT"=>3, "INT"=>4, "INTEGER"=>4, "BIGINT"=>8,
"FLOAT"=>'$M<=24?4:8', "DOUBLE"=>8,
"DECIMAL"=>'int(($M-$D)/9)*4+int(((($M-$D)%9)+1)/2)+int($D/9)*4+int((($D%9)+1)/2)',
"NUMERIC"=>'int(($M-$D)/9)*4+int(((($M-$D)%9)+1)/2)+int($D/9)*4+int((($D%9)+1)/2)',
"BIT"=>'($M+7)>>3',
"DATE"=>3, "TIME"=>3, "DATETIME"=>8, "TIMESTAMP"=>4, "YEAR"=>1,
"BINARY"=>'$M',"CHAR"=>'$M*$CL',
"VARBINARY"=>'$M+($M>255?2:1)', "VARCHAR"=>'$M*$CL+($M>255?2:1)',
"ENUM"=>'$M>255?2:1', "SET"=>'($M+7)>>3',
"TINYBLOB"=>9, "TINYTEXT"=>9,
"BLOB"=>10, "TEXT"=>10,
"MEDIUMBLOB"=>11, "MEDIUMTEXT"=>11,
"LONGBLOB"=>12, "LONGTEXT"=>12
);

my %DataTypeMin = (
"VARBINARY"=>'($M>255?2:1)', "VARCHAR"=>'($M>255?2:1)'
);

my ($D, $M, $S, $C, $L, $dt, $dp ,$bc, $CL);
my $fieldCount = 0;
my $byteCount = 0;
my $byteCountMin = 0;
my @fields = ();
my $fieldName;
my $tableName;
my $defaultDbCL = 1;
my $defaultTableCL = 1;
my %charsetMaxLen;
my %collationMaxLen;

open (CHARSETS, "mysql -B --skip-column-names information_schema -e 'select CHARACTER_SET_NAME,MAXLEN from CHARACTER_SETS;' |");
%charsetMaxLen = map ( ( /^(\w+)/ => /(\d+)$/ ), <CHARSETS>);
close CHARSETS;

open (COLLATIONS, "mysql -B --skip-column-names information_schema -e 'select COLLATION_NAME,MAXLEN from CHARACTER_SETS INNER JOIN COLLATIONS USING(CHARACTER_SET_NAME);' |");
%collationMaxLen = map ( ( /^(\w+)/ => /(\d+)$/ ), <COLLATIONS>);
close COLLATIONS;

open (TABLEINFO, "mysqldump -d --compact ".join(" ",@ARGV)." |");

while (<TABLEINFO>) {
chomp;
if ( ($S,$C) = /create database.*?`([^`]+)`.*default\scharacter\sset\s+(\w+)/i ) {
$defaultDbCL = exists $charsetMaxLen{$C} ? $charsetMaxLen{$C} : 1;
print "Database: $S".($C?" DEFAULT":"").($C?" CHARSET $C":"")." (bytes per char: $defaultDbCL)\n\n";
next;
}
if ( /^create table\s+`([^`]+)`.*/i ) {
$tableName = $1;
@fields = ();
next;
}
if ( $tableName && (($C,$L) = /^\)(?:.*?default\scharset=(\w+))?(?:.*?collate=(\w+))?/i) ) {
$defaultTableCL = exists $charsetMaxLen{$C} ? $charsetMaxLen{$C} : (exists $collationMaxLen{$L} ? $collationMaxLen{$L} : $defaultDbCL);
print "Table: $tableName".($C||$L?" DEFAULT":"").($C?" CHARSET $C":"").($L?" COLLATION $L":"")." (bytes per char: $defaultTableCL)\n";
$tableName = "";
$fieldCount = 0;
$byteCount = 0;
$byteCountMin = 0;
while ($_ = shift @fields) {
if ( ($fieldName,$dt,$dp,$M,$D,$S,$C,$L) = /\s\s`([^`]+)`\s+([a-z]+)(\((\d+)(?:,(\d+))?\)|\((.*)\))?(?:.*?character\sset\s+(\w+))?(?:.*?collate\s+(\w+))?/i ) {
$dt = uc $dt;
if (exists $DataType{$dt}) {
if (length $S) {
$M = ($S =~ s/(\'.*?\'(?!\')(?=,|$))/$1/g);
$dp = "($M : $S)"
}
$D = 0 if !$D;
$CL = exists $charsetMaxLen{$C} ? $charsetMaxLen{$C} : (exists $collationMaxLen{$L} ? $collationMaxLen{$L} : $defaultTableCL);
$bc = eval($DataType{$dt});
$byteCount += $bc;
$byteCountMin += exists $DataTypeMin{$dt} ? $DataTypeMin{$dt} : $bc;
} else {
$bc = "??";
}
$fieldName.="\t" if length($fieldName) < 8;
print "bytes:\t".$bc."\t$fieldName\t$dt$dp".($C?" $C":"").($L?" COLL $L":"")."\n";
++$fieldCount;
}
}
print "total:\t$byteCount".($byteCountMin!=$byteCount?"\tleast: $byteCountMin":"\t\t")."\tcolumns: $fieldCount\n\n";
next;
}
push @fields, $_;
}
close TABLEINFO;

    感谢大家的大力帮助。

    • 4

  2. 步骤1：

    col1 varchar(25),  2 + avg_byte_len
col2 int,          4
col4 char(15),     1*15 or 3*15 or...
col5 datetime      Pre-5.6: 8; then 5

SELECT AVG(LENGTH(col1)) as avg_byte_len,
       AVG(CHAR_LENGTH(col1) as avg_num_chars FROM ...;

    20 个英文字符：2 + 1*20
    20 个中东/斯拉夫字符：2 + 2*20
    20 个亚洲字符：2 + 3*20
    20 个表情符号：2 + 4*20（你需要utf8mb4）

    第2步：把这些加起来。

    第 3 步：乘以 2 到 3 之间的某个值以允许 InnoDB 开销。我发现这个因素通常有效。（但不适用于小型表，也不一定适用于分区表。）

    我认为没有理由采用每列的最大大小。

    您可以获得比SHOW TABLE STATUS或等效数据更接近的information_schema数据：

    第 1 步： SELECT COUNT(*)-- 我们用这个代替Rows

    第 2 步：获取Data_length + Index_length + Data_free

    第三步：划分。

    • 2

  3. 这是我使用 SQL 并根据 MySQL 文档“数据类型存储要求”查询 INFORMATION_SCHEMA.COLUMNS 表的两分钱。

    Note: this is not be entirely accurate, as ENUM and SET column sizes depend on the number of enumeration values, respectively set members. Here I used the "worst-case" values (2 and 8 bytes respectively), this already gives a good hint about the row size. One might parse the INFORMATION_SCHEMA.COLUMNS.COLUMN_TYPE to get the number of enum values/set members for such columns and then compute the amount of bytes based on what the documentation says.

    SELECT col_sizes.TABLE_SCHEMA, col_sizes.TABLE_NAME, SUM(col_sizes.col_size) AS EST_MAX_ROW_SIZE
FROM (
    SELECT 
        cols.TABLE_SCHEMA, 
        cols.TABLE_NAME, 
        cols.COLUMN_NAME,
        CASE cols.DATA_TYPE
            WHEN 'tinyint' THEN 1
            WHEN 'smallint' THEN 2
            WHEN 'mediumint' THEN 3
            WHEN 'int' THEN 4
            WHEN 'bigint' THEN 8
            WHEN 'float' THEN IF(cols.NUMERIC_PRECISION > 24, 8, 4)
            WHEN 'double' THEN 8
            WHEN 'decimal' THEN ((cols.NUMERIC_PRECISION - cols.NUMERIC_SCALE) DIV 9)*4  + (cols.NUMERIC_SCALE DIV 9)*4 + CEIL(MOD(cols.NUMERIC_PRECISION - cols.NUMERIC_SCALE,9)/2) + CEIL(MOD(cols.NUMERIC_SCALE,9)/2)
            WHEN 'bit' THEN (cols.NUMERIC_PRECISION + 7) DIV 8
            WHEN 'year' THEN 1
            WHEN 'date' THEN 3
            WHEN 'time' THEN 3 + CEIL(cols.DATETIME_PRECISION /2)
            WHEN 'datetime' THEN 5 + CEIL(cols.DATETIME_PRECISION /2)
            WHEN 'timestamp' THEN 4 + CEIL(cols.DATETIME_PRECISION /2)
            WHEN 'char' THEN cols.CHARACTER_OCTET_LENGTH
            WHEN 'binary' THEN cols.CHARACTER_OCTET_LENGTH
            WHEN 'varchar' THEN IF(cols.CHARACTER_OCTET_LENGTH > 255, 2, 1) + cols.CHARACTER_OCTET_LENGTH
            WHEN 'varbinary' THEN IF(cols.CHARACTER_OCTET_LENGTH > 255, 2, 1) + cols.CHARACTER_OCTET_LENGTH
            WHEN 'tinyblob' THEN 9
            WHEN 'tinytext' THEN 9
            WHEN 'blob' THEN 10
            WHEN 'text' THEN 10
            WHEN 'mediumblob' THEN 11
            WHEN 'mediumtext' THEN 11
            WHEN 'longblob' THEN 12
            WHEN 'longtext' THEN 12
            WHEN 'enum' THEN 2
            WHEN 'set' THEN 8
            ELSE 0
        END AS col_size
    FROM INFORMATION_SCHEMA.COLUMNS cols
) AS col_sizes
GROUP BY col_sizes.TABLE_SCHEMA, col_sizes.TABLE_NAME
    • 2

  4. 您需要根据数据类型（此处为 MySQL 参考）知道每个字段的字节大小，然后将这些值汇总在一起。

    • 0

  5. 我制作了一个粗略的 bash 脚本来计算行大小并警告它是否超过了基于模式的限制：

    #!/bin/bash
​
#
# usage: mysqldump --no-data | check_row_size.sh
#
​
#
#
# https://dev.mysql.com/doc/refman/8.0/en/column-count-limit.html#row-size-limits
#
# The maximum row size for an InnoDB table, which applies to data stored locally within a database page, is slightly less than half a page for 4KB, 8KB, 16KB, and 32KB innodb_page_size settings.
# For example, the maximum row size is slightly less than 8KB for the default 16KB InnoDB page size.
#
#
# MariaDB [(none)]> show variables like 'innodb_page_size';
#+------------------+-------+
#| Variable_name    | Value |
#+------------------+-------+
#| innodb_page_size | 16384 |
#+------------------+-------+
#1 row in set (0.00 sec)
#
#
# Options:
# 1. Change default innodb_page_size to 32k
# 2. Change storage engine to DYNAMIC for tables
# 3. ?
#
​
#===========================================================================================
# Functions
#===========================================================================================
RETVAL=0
​
calc_row_size() {
    local -n TABLE_FIELDS=$1
    local -n TABLE_CHARSET=$2
    local FIELD_TYPE=""
    local FIELD_SIZE=""
    local FIELD=""
    local ROW_SIZE=0
    local IFS=$'|' # To split the vars using set
    for FIELD in "${TABLE_FIELDS[@]}"  
    do  
        set $FIELD
        FIELD_NAME=$1
        FIELD_TYPE=$2
        FIELD_SIZE=$3        
        calc_field_size_in_bytes $FIELD_TYPE $FIELD_SIZE $TABLE_CHARSET
        ROW_SIZE=$((ROW_SIZE + RETVAL))
        [ $DEBUG -gt 0 ] && echo "DEBUG1: Field name: $FIELD_NAME type: $FIELD_TYPE lenght: $FIELD_SIZE size: $RETVAL bytes Row size: $ROW_SIZE"
    done  
    RETVAL=$ROW_SIZE
}
​
calc_field_size_in_bytes() {
    local TYPE=$1
    local SIZE=$2
    local CHARSET=$3

    case $FIELD_TYPE in
        varchar)
            # https://adayinthelifeof.nl/2010/12/04/about-using-utf-8-fields-in-mysql/
            # Max 3 bytes per utf-8 chat in mysql
            case $CHARSET in
                utf8)
                    RETVAL=$((SIZE * 3))  # 3 bytes per character for utf8 
                ;;
                latin1)
                    RETVAL=$((SIZE))  # 1 byte per character for latin1
                ;;
                *)
                    echo "Unknown charset ($CHARSET), please fix the script"
                    exit 1
                ;;
            esac
        ;;
        smallint|int|bigint|tinyint|varbinary)
            RETVAL=$SIZE
        ;;
        blob)
            # https://dev.mysql.com/doc/refman/8.0/en/column-count-limit.html#row-size-limits
            # BLOB and TEXT columns only contribute 9 to 12 bytes toward the row size limit because their contents are stored separately from the rest of the row.
            RETVAL=9
        ;;
        text)
            RETVAL=12
        ;;
        timestamp)
            RETVAL=4 
        ;; 
        decimal)
            # https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html#data-types-storage-reqs-numeric
            # Each multiple of nine digits requires four bytes, and the “leftover” digits require some fraction of four bytes. 
            if [[ $SIZE =~ ([0-9]+),([0-9]+) ]] 
            then
              INTEGER_PART=${BASH_REMATCH[1]}
              FRACTIONAL_PART=${BASH_REMATCH[2]}
​
              INTEGER_BYTES=$((INTEGER_PART / 9 * 4))
              REMAINDER=$((INTEGER_PART % 9))
              case $REMAINDER in
                  0) INTEGER_BYTES=$((INTEGER_BYTES + 0)); ;;
                  1) INTEGER_BYTES=$((INTEGER_BYTES + 1)); ;;
                  2) INTEGER_BYTES=$((INTEGER_BYTES + 1)); ;;
                  3) INTEGER_BYTES=$((INTEGER_BYTES + 2)); ;;
                  4) INTEGER_BYTES=$((INTEGER_BYTES + 2)); ;;
                  5) INTEGER_BYTES=$((INTEGER_BYTES + 3)); ;;
                  6) INTEGER_BYTES=$((INTEGER_BYTES + 3)); ;;
                  7) INTEGER_BYTES=$((INTEGER_BYTES + 4)); ;;
                  8) INTEGER_BYTES=$((INTEGER_BYTES + 4)); ;;
              esac
​
              FRACTIONAL_BYTES=$((FRACTIONAL_PART / 9 * 4))
              REMAINDER=$((FRACTIONAL_PART % 9))
              case $REMAINDER in
                  0) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 0)); ;;
                  1) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 1)); ;;
                  2) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 1)); ;;
                  3) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 2)); ;;
                  4) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 2)); ;;
                  5) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 3)); ;;
                  6) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 3)); ;;
                  7) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 4)); ;;
                  8) FRACTIONAL_BYTES=$((FRACTIONAL_BYTES + 4)); ;;
              esac
              [ $DEBUG -gt 0 ] && echo "DEBUG1: Calulation of decimal: SIZE: $SIZE INTEGER_PART:$INTEGER_PART FRACTIONAL_PART:$FRACTIONAL_PART TOTAL = INTEGER_BYTES($INTEGER_BYTES) + FRACTIONAL_BYTES($FRACTIONAL_BYTES)"
              RETVAL=$((INTEGER_BYTES + FRACTIONAL_BYTES)) 
            else
                echo "Seems like SIZE ($SIZE) for a decimal field doesn't match pattern ([0-9]+),([0-9]+). Please investigate"
                exit 1
            fi
        ;;
        *)
            echo "Found a field type that is not handled: $TYPE. Please fix before proceeding."
            exit 1
        ;;
    esac
}
​
​
#===========================================================================================
# INIT
#===========================================================================================
INSIDE_CREATE_TABLE_STATEMENT=false # True if we are within a create table statement
TABLE_NAME=''  # Current table name
ROW_SIZE=0 # Current row size being calculated
DEBUG=0
VERBOSE=0
MAX_SIZE=8126 # Default
declare -a FIELDS # List of fields from the current CREATE TABLE statement
​
#===========================================================================================
# Parameters
#===========================================================================================
OPTIND=1         # Reset in case getopts has been used previously in the shell.
​
while getopts "hvdt:" opt; do
    case "$opt" in
    h)
        echo "Usage: mysqldump --no-data | ./check_row_size [-v|-d] [-t threshold]"
        exit 0
        ;;
    v) VERBOSE=1
        ;;
    d) DEBUG=2
        ;;
    t) MAX_SIZE=$OPTARG
        ;;
    esac
done
​
​
#===========================================================================================
# MAIN Loop - parses schema then calc row_size based on charset
#===========================================================================================
while IFS= read -r LINE
do
    [ $DEBUG -gt 1 ] && echo "DEBUG2: Read: $LINE"
    # Are we within a CREATE TABLE statement?
    if [ $INSIDE_CREATE_TABLE_STATEMENT == "false" ]
    then
        # Nope, is the current line a 'CREATE TABLE' statement?
        if [[ $LINE =~ ^"CREATE TABLE \`"([^\`]+) ]] 
        then
            [ $DEBUG -gt 0 ] && echo "CREATE TABLE FOUND!: $TABLE_NAME"
            TABLE_NAME=${BASH_REMATCH[1]} # What has been caught between pattern parenthesis
            INSIDE_CREATE_TABLE_STATEMENT='true'
            FIELDS=()
        fi
        continue # Ok, next line 
    fi
    # Is this a create table field definition line?
    if [[ $LINE =~ ^' '+'`'([^'`']+)'` '([a-z]+)'('([^')']+) ]]
    then
        FIELD_NAME=${BASH_REMATCH[1]}
        FIELD_TYPE=${BASH_REMATCH[2]}
        FIELD_SIZE=${BASH_REMATCH[3]}
        FIELDS+=( "$FIELD_NAME|$FIELD_TYPE|$FIELD_SIZE" )
        continue
    fi
    # Have we reached the end of the CREATE TABLE statement?
    if [[ $LINE =~ ^") ENGINE=InnoDB DEFAULT CHARSET="([^ ]+) ]] 
    then
        CHARSET=${BASH_REMATCH[1]}
        [ $DEBUG -gt 0 ] && echo "End of CREATE TABLE statement"
        calc_row_size FIELDS CHARSET
        ROW_SIZE=$RETVAL
        if [ $ROW_SIZE -gt $MAX_SIZE ]
        then
            echo "Table: $TABLE_NAME has a row size: $ROW_SIZE Bytes > $MAX_SIZE Bytes Charset: $CHARSET"
            # and is going to cause problem if the we upgrade to tables in ROW_FORMAT compact. See https://mariadb.com/kb/en/library/troubleshooting-row-size-too-large-errors-with-innodb/ for more details."
        fi
        INSIDE_CREATE_TABLE_STATEMENT='false'
    fi
done

    ​​​

    • 0

  6. 已经有一些此类问题，例如这个：How to estimate/predict data size and index size of a table in MySQL

    该问题与您的表格之间的一个区别是您的表格中存在可变长度的字符串 - 请记住考虑它们可以达到的最大大小。

    还要记住，从版本 5 开始，varchar(25)最多 25个字符而不是 25个字节，因此如果您可能在字符串中看到非 ASCII 字符，则列大小可能会膨胀到最多 100 个字节，因为某些字符占用 4 个字节表示 - 例如“一堆便便表情符号”（我开玩笑说它确实存在 - 如果你当前的浏览器+字体支持它看起来像：?）是 0xF0 0x9F 0x92 0xA9。在 v5 之前，mySQL 在指定字符串类型长度时计算字节而不是字符。

    关于自动化的编辑

    在自动化过程方面，您应该能够以INFORMATION_SCHEMA与您为 MS SQL Server 找到的脚本类似的方式从表中获取所需的所有信息。请参阅https://dev.mysql.com/doc/refman/5.0/en/information-schema.html了解有关该内容的一些文档。

    • -1

相关问题