Início / dba / Perguntas / 10467
Accepted
Derek Downey
Asked: 2012-01-11 12:10:19 +0800 CST

Como converter caracteres de controle no MySQL de latin1 para UTF-8?

  • 772

Ao converter um banco de dados para UTF-8, notei um comportamento estranho em relação aos caracteres de controle 0x80-0x9F. Por exemplo, 0x92 (apóstrofe à direita) não seria convertido em UTF-8 e truncaria o restante do conteúdo de uma coluna, usando este método:

CREATE TABLE `bar` (
 `content` text
) ENGINE=MyISAM DEFAULT CHARSET=latin1

INSERT INTO bar VALUES (0x8081828384858687898A8B8C8D8E8F909192939495969798999A9B9C9D9E9F);
Query OK, 1 row affected (0.06 sec)

SELECT content FROM bar;
+---------------------------------------------------------------------------------+
| content                                                                         |
+---------------------------------------------------------------------------------+
| €‚ƒ„…†‡‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ                                                 |
+---------------------------------------------------------------------------------+
1 row in set (0.06 sec)

ALTER TABLE bar CHANGE content content TEXT CHARACTER SET UTF8;
Query OK, 1 row affected, 1 warning (0.06 sec)
Records: 1  Duplicates: 0  Warnings: 1

SHOW WARNINGS;
+---------+------+-------------------------------------------------------------------------------------+
| Level   | Code | Message                                                                             |
+---------+------+-------------------------------------------------------------------------------------+
| Warning | 1366 | Incorrect string value: '\x80\x81\x82\x83\x84\x85...' for column 'content' at row 1 |
+---------+------+-------------------------------------------------------------------------------------+
1 row in set (0.06 sec)

SELECT * FROM bar;
+---------+
| content |
+---------+
|         |
+---------+
1 row in set (0.06 sec)

Embora normalmente 0x80-0x9F não seja permitido no Latin1, o MySQL parece lidar com isso de maneira diferente:

O latin1 do MySQL é o mesmo que o conjunto de caracteres cp1252 do Windows. Isso significa que é o mesmo que o oficial ISO 8859-1 ou IANA (Internet Assigned Numbers Authority) latin1, exceto que IANA latin1 trata os pontos de código entre 0x80 e 0x9f como “indefinidos”, enquanto cp1252 e, portanto, o latin1 do MySQL, atribui caracteres para essas posições. [origem]

Mas o MySQL não consegue converter o intervalo de valores acima de seu conjunto de caracteres latin1 para seu conjunto de caracteres UTF-8.

Esses caracteres estão entrando em meu banco de dados copiando/colando de um documento do Word (cp1252) e, embora eu possa ter encontrado uma maneira de fazer com que o aplicativo force os valores UTF-8 corretos para novas entradas, preciso garantir que o antigo obtenha convertido corretamente.

Existe uma maneira dentro do MySQL que estou perdendo para convertê-los para o equivalente UTF-8 sem passar por cada linha de cada coluna de texto e substituí-los por uma versão compatível com ASCII?

mysql character-set

2 respostas

  1. Best Answer

    Não tenho certeza. Tentei começar reproduzindo seu problema, mas o alter funcionou bem para mim. 

    test > CREATE TABLE `bar` (  `content` text ) ENGINE=MyISAM DEFAULT CHARSET=latin1;  INSERT INTO bar VALUES (0x8081828384858687898A8B8C8D8E8F909192939495969798999A9B9C9D9E9F);
Query OK, 0 rows affected (0.02 sec)

Query OK, 1 row affected (0.00 sec)

test > ALTER TABLE bar CHANGE content content TEXT CHARACTER SET UTF8;
Query OK, 1 row affected (0.04 sec)
Records: 1  Duplicates: 0  Warnings: 0

test > select * from bar;
+---------------------------------+
| content                         |
+---------------------------------+
| ����������������������������� |
+---------------------------------+
1 row in set (0.00 sec)

test > set names utf8;
Query OK, 0 rows affected (0.00 sec)

test > select * from bar;
+---------------------------------------------------------------------------------+
| content                                                                         |
+---------------------------------------------------------------------------------+
| €‚ƒ„…†‡‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ |
+---------------------------------------------------------------------------------+
1 row in set (0.00 sec)

    Aqui estão minhas configurações de char relacionadas

    test > show variables like '%char%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | latin1                     |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | latin1                     |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+

    Editar

    Minhas configurações de char antes de rodar set names utf8

    test > show variables like '%char%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | latin1                     |
| character_set_connection | latin1                     |
| character_set_database   | latin1                     |
| character_set_filesystem | binary                     |
| character_set_results    | latin1                     |
| character_set_server     | latin1                     |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

    Versão

    test > select version();
+-------------------------+
| version()               |
+-------------------------+
| 5.1.41-3ubuntu12.10-log |
+-------------------------+
1 row in set (0.00 sec)
    • 4

  2. Você pode ter que converter o conjunto de caracteres para cp1250 antes de carregar os dados.

    Eu corri isso primeiro

    mysql> show character set like 'cp%';
+---------+---------------------------+-------------------+--------+
| Charset | Description               | Default collation | Maxlen |
+---------+---------------------------+-------------------+--------+
| cp850   | DOS West European         | cp850_general_ci  |      1 |
| cp1250  | Windows Central European  | cp1250_general_ci |      1 |
| cp866   | DOS Russian               | cp866_general_ci  |      1 |
| cp852   | DOS Central European      | cp852_general_ci  |      1 |
| cp1251  | Windows Cyrillic          | cp1251_general_ci |      1 |
| cp1256  | Windows Arabic            | cp1256_general_ci |      1 |
| cp1257  | Windows Baltic            | cp1257_general_ci |      1 |
| cp932   | SJIS for Windows Japanese | cp932_japanese_ci |      2 |
+---------+---------------------------+-------------------+--------+
8 rows in set (0.00 sec)

    cp1252 não existe aqui. O mais próximo é cp1250.

    Tente esta sequência:

    drop database if exists dtest;
create database dtest;
use dtest
set names cp1250;
CREATE TABLE `bar` ( 
 `content` text 
) ENGINE=MyISAM DEFAULT CHARSET=latin1 ;
INSERT INTO bar VALUES (0x8081828384858687898A8B8C8D8E8F909192939495969798999A9B9C9D9E9F); 
SELECT content FROM bar; 
SHOW VARIABLES LIKE '%char%';
set names utf8;
SHOW VARIABLES LIKE '%char%';
ALTER TABLE bar CHANGE content content TEXT CHARACTER SET UTF8; 
SELECT content FROM bar;

    e veja o que acontece.

    Eu tenho isso no MySQL 5.5.19 no Linux

    mysql> drop database if exists dtest;
Query OK, 0 rows affected (0.00 sec)

mysql> create database dtest;
Query OK, 1 row affected (0.00 sec)

mysql> use dtest
Database changed
mysql> set names cp1250;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TABLE `bar` (
    ->  `content` text
    -> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 ;
Query OK, 0 rows affected (0.01 sec)

mysql> INSERT INTO bar VALUES (0x8081828384858687898A8B8C8D8E8F909192939495969798999A9B9C9D9E9F);
Query OK, 1 row affected (0.00 sec)

mysql> SELECT content FROM bar;
+---------------------------------+
| content                         |
+---------------------------------+
| ??

??????                      |
+---------------------------------+
1 row in set (0.00 sec)

mysql> SHOW VARIABLES LIKE '%char%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | cp1250                     |
| character_set_connection | cp1250                     |
| character_set_database   | latin1                     |
| character_set_filesystem | binary                     |
| character_set_results    | cp1250                     |
| character_set_server     | latin1                     |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

mysql> set names utf8;
Query OK, 0 rows affected (0.00 sec)

mysql> SHOW VARIABLES LIKE '%char%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | latin1                     |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | latin1                     |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

mysql> ALTER TABLE bar CHANGE content content TEXT CHARACTER SET UTF8;
Query OK, 1 row affected (0.01 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> SELECT content FROM bar;
+---------------------------------------------------------------------------------+
| content                                                                         |
+---------------------ŽÂÂâââââ---------------------------------------------------+
| â¬ÂâÆââ¦â â¡â°Å â¹Å         ¢Å¡âºÅÂ
                                      +---------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql>

    e eu tenho isso no MySQL 5.5.12 para Windows na minha máquina com Windows 7

    mysql> drop database if exists dtest;
Query OK, 1 row affected (0.00 sec)

mysql> create database dtest;
Query OK, 1 row affected (0.02 sec)

mysql> use dtest
Database changed
mysql> set names cp1250;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TABLE `bar` (
    ->  `content` text
    -> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 ;
Query OK, 0 rows affected (0.06 sec)

mysql> INSERT INTO bar VALUES (0x8081828384858687898A8B8C8D8E8F909192939495969798999A9B9C9D9E9F);
Query OK, 1 row affected (0.00 sec)

mysql> SELECT content FROM bar;
+---------------------------------+
| content                         |
+---------------------------------+
| Ç?é?äàåçëèï??Ä??æÆôöòûù?ÖÜ¢??₧? |
+---------------------------------+
1 row in set (0.00 sec)

mysql> SHOW VARIABLES LIKE '%char%';
+--------------------------+---------------------------------+
| Variable_name            | Value                           |
+--------------------------+---------------------------------+
| character_set_client     | cp1250                          |
| character_set_connection | cp1250                          |
| character_set_database   | latin1                          |
| character_set_filesystem | binary                          |
| character_set_results    | cp1250                          |
| character_set_server     | latin1                          |
| character_set_system     | utf8                            |
| character_sets_dir       | C:\MySQL_5.5.12\share\charsets\ |
+--------------------------+---------------------------------+
8 rows in set (0.00 sec)

mysql> set names utf8;
Query OK, 0 rows affected (0.00 sec)

mysql> SHOW VARIABLES LIKE '%char%';
+--------------------------+---------------------------------+
| Variable_name            | Value                           |
+--------------------------+---------------------------------+
| character_set_client     | utf8                            |
| character_set_connection | utf8                            |
| character_set_database   | latin1                          |
| character_set_filesystem | binary                          |
| character_set_results    | utf8                            |
| character_set_server     | latin1                          |
| character_set_system     | utf8                            |
| character_sets_dir       | C:\MySQL_5.5.12\share\charsets\ |
+--------------------------+---------------------------------+
8 rows in set (0.00 sec)

mysql> ALTER TABLE bar CHANGE content content TEXT CHARACTER SET UTF8;
Query OK, 1 row affected (0.06 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> SELECT content FROM bar;
+---------------------------------------------------------------------------------+
| content                                                                         |
+---------------------------------------------------------------------------------+
| €‚ƒ„…†‡‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ |
+---------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql>

    De uma chance !!!

    • 1

relate perguntas