cameronyule.com » Server

Configuring MySQL to use UTF-8

Cameron — Tue, 15 Jul 2008 10:07:13 +0000

A project I’m working on at the moment is going to have multiple language options available, not all of which use the same alphabet (e.g. Russian and Chinese).

To lessen the pain commonly associated with internationalisation on the web, it’s beneficial to use the UTF-8 character set. This short summary from the Unicode Consortium may help explain better;

Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language.
…
Unicode enables a single software product or a single website to be targeted across multiple platforms, languages and countries without re-engineering. It allows data to be transported through many different systems without corruption.

Thankfully MySQL has supported Unicode for quite some time now, even if it’s not configured to use it by default.

First, let’s check what our settings are at the moment;

mysql> SHOW VARIABLES LIKE 'collation%';
+----------------------+-------------------+
| Variable_name        | Value             |
+----------------------+-------------------+
| collation_connection | latin1_swedish_ci |
| collation_database   | latin1_swedish_ci |
| collation_server     | latin1_swedish_ci |
+----------------------+-------------------+
3 rows in set (0.01 sec)

mysql> SHOW VARIABLES LIKE 'character_set%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | latin1                     | 
| character_set_connection | latin1                     | 
| character_set_database   | latin1                     | 
| character_set_filesystem | binary                     | 
| character_set_results    | latin1                     | 
| character_set_server     | latin1                     | 
| character_set_system     | utf8                       | 
| character_sets_dir       | /usr/share/mysql/charsets/ | 
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

That’s to be expected, but it’s not really what we wanted.

Find your MySQL configuration file (on most Linux/BSD systems it’s /etc/my.cnf) and make sure it’s got the following statements under the relevant headers.

[mysqld]
default-character-set=utf8
default-collation=utf8_general_ci
character-set-server=utf8
collation-server=utf8_general_ci
init-connect='SET NAMES utf8'
 
[client]
default-character-set=utf8

Restart MySQL and make sure it’s working;

service mysql restart

mysql> SHOW VARIABLES LIKE 'collation%';
+----------------------+-----------------+
| Variable_name        | Value           |
+----------------------+-----------------+
| collation_connection | utf8_general_ci | 
| collation_database   | utf8_general_ci | 
| collation_server     | utf8_general_ci | 
+----------------------+-----------------+
3 rows in set (0.00 sec)

mysql> SHOW VARIABLES LIKE 'character_set%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       | 
| character_set_connection | utf8                       | 
| character_set_database   | utf8                       | 
| character_set_filesystem | binary                     | 
| character_set_results    | utf8                       | 
| character_set_server     | utf8                       | 
| character_set_system     | utf8                       | 
| character_sets_dir       | /usr/share/mysql/charsets/ | 
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

Update; Mo pointed out that it’s worth demonstrating setting the charset and collation when creating tables too (good point!).

CREATE TABLE `content` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `language` VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `title` VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL DEFAULT '',
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM CHARACTER SET utf8 COLLATE utf8_general_ci;

Multiple users with svn+ssh

Cameron — Fri, 11 Jul 2008 09:07:54 +0000

Following on from my last post on moving from svnserve to svn+ssh, I had to then get this setup working for multiple users. This is how I achieved it on CentOS 5.2, hopefully it’ll prove useful for someone else.

This is based on a tutorial I found on the subversion dev mailing list.

# We need a group for our svn users
groupadd svnusers
 
# Add the users
useradd svn -g svnusers # this is the svn 'admin' user
useradd user1 -g svnusers
useradd user2 -g svnusers
useradd user3 -g svnusers
useradd user4 -g svnusers
# … etc
 
# Edit each svn user's .bashrc file, e.g.
su - user1
nano -w .bashrc
 
# Add the followng line to the bottom
umask 002 # allow user + group to write, no other. 
 
# Create the dir where we're storing the repo (as root)
mkdir -p /usr/share/subversion/repositories
 
# Set permissions on our new dir
chown -R root.svnusers /usr/share/subversion/repositories
chmod -R u+wrx,g+wrx,o-wxr /usr/share/subversion/repositories
chown -R root.svnusers /usr/share/subversion
 
# The following may be overkill
# chown -R root.svnusers /usr/share
 
# Rename the original svnserve command into svnserve.bin 
mv /usr/bin/svnserve /usr/bin/svnserve.bin
 
# Create a new svnserve file
nano -w /usr/bin/svnserve
 
# Paste the following into nano and save as svnserve
#!/bin/sh 
# wrap in order to put root in by default 
# Script implemented by Adrian Robert  
exec /usr/bin/svnserve.bin -r /usr/share/subversion/repositories "$@"
 
# Make our svnserve wrapper executable
chmod u+wrx,g+rx-w,o+xr-w svnserve
chmod u+wrx,g+rx-w,o+xr-w /usr/bin/svnserve
 
# Log in as the svn user
su - svn
 
# Create our repos
svnadmin create /usr/share/subversion/repositories/projects
cd /usr/share/subversion/repositories/projects/conf 
 
# Edit the repo conf
nano -w svnserve.conf
 
# Add the following under the [general] header
anon-access = none
auth-access = write

Moving from svnserve to svn+ssh

Cameron — Wed, 09 Jul 2008 11:22:54 +0000

After mentioning that I was using svnserve, a friend sent me a twitter message asking why I wasn’t using svn+ssh, so I thought I’d have a look. It turned out to be really simple to make the move, and I’m left with a nicer setup than before.

Firstly, let’s kill the running svnserve process as we’re not going to be using that anymore.

[user@server ~]$ ps aux | grep svn
user     2495  0.0  0.3  45856   940 ?          Ss    20:02   0:00 svnserve -d -r repository/
user     2501  0.0  0.2   3936    724 pts/0     S+   20:02   0:00 grep svn
[user@server ~]$ kill 2495

If you’ve specifically opened port 3690 (which is the default svnserve port) in your firewall you should close that again now. You can check your current firewall ruleset using the list flag.

[user@server ~]$ iptables --list

I happen to run ssh on a non-standard port, so to save myself having to specify this each time I’ve overridden the default tunnel subversion has setup for ssh. This stage is run on your local machine (or where you want to check out your repository).

local:~ cameron$ nano -w .subversion/config

Under the [tunnels] section, I added the following line. Note that XXXX is where my non-standard port is specified, and REMOTEUSERNAME is the username I’m using to connect to the server to checkout my repository. The -C flag enables compression, and seems to greatly speed things up.

ssh = /usr/bin/ssh -p XXXX -l REMOTEUSERNAME -C

Okay, so let’s try and check something out! I’m hoping you have ssh keys configured already, but if not here’s an SSH Keys Howto. Note you have to specify absolute paths now.

local:~ cameron$ svn co svn+ssh://yoururl/path/to/repository/trunk
A    trunk/wp-config.php
A    trunk/wp-pass.php
A    trunk/wp-rss.php
…

With any luck, that’ll checkout your repository.

If you already have a repository checked out and don’t want to delete it and checkout again, there’s a solution for you too. Change into your local working copy and run the following;

local:projects cameron$ svn switch --relocate svn://yoururl/repository/trunk svn+ssh://yoururl/path/to/repository/trunk

Installing PHP on 64-bit systems

Cameron — Sat, 19 Apr 2008 19:57:24 +0000

I was trying to install PHP 5.2.5 using Litespeed’s handy admin interface (saves ssh’ing into the server), but kept being met with the following error when it was trying to compile;

configure: error: libpng.(a|so) not found.

Turns out it’s related to my system being 64-bit (I’m using CentOS 5 on Slicehost) and where libpng is installed and so on. Found the answer in this PHP bug report where we specifically tell configure where to look;

./configure --with-libdir=lib64 …

Sorted!