I am changing ISPs and there will be a time gap with no internet connection at home. So to say my pipe will drain and to keep this ship on water I moved it temporarily to a free host – http://byethost.com. The whole transition process went really smooth and I am planning to write a post about it. I hope I will have soon the connection and take the steering in my own hands again. Until then you will be served by 64bit Linux Kernel 2.6.32-279, Apache 2.2.3, PHP 5.3.14 and MySQL 5.5.28. Enjoy the new host.

Good news everyone    

Just got the email through the mailing list debian-announce@lists.debian.org and it is GREAT NEWS!!!

“After many months of constant development, the Debian project is proud
to present its new stable version 7.0 (code name “Wheezy”).
This new version of Debian includes various interesting features such as
multiarch support [1], several specific tools to deploy private
clouds [2], an improved installer, and a complete set of multimedia
codecs and front-ends which remove the need for third-party
repositories.”

Big THANKS to *all* Debian developers. Debian FTW!

Now I have actually a dilemma. I bought my laptop (ASUS X55SV Intel(R) Core(TM)2 Duo CPU T8300 [2 x 2.40GHz] 4GB RAM on 667 MHz) in the beginning of 2008 and this is still my primary “desktop” machine. After I bought it, I immediately dd’ed the hard disc with zeros and put Kubuntu 8.04 LTS (Hardy Heron) GNU/Linux. KDE 3.5 was my favorite X and I refused to change it, so that system configuration runs smooth and robust still today. What a great piece of hardware machinery and what a great piece of software system. On top of that I run many virtual machines where I test everything I like to test or need to run. For some time now, my favorite X has crystallized to be XFCE4 so I think I am ready to move on. It is tough though, 5 years with that great Kubuntu. I am not in a hurry to switch, time will show, but that is definitely the next step.
Sail with the wind Debian

Update 10.05.2013 Lol, just read that article Three Ubuntu releases reach end of life

Three releases of Ubuntu reached their end of life on May 9, 2013, which means they will no longer receive updates of any kind. Users of Ubuntu 8.04 LTS (“Hardy Heron”), Ubuntu 10.04 LTS Desktop (“Lucid Lynx”), and Ubuntu 11.10 (“Oneiric Ocelot”) should upgrade.

Continue Reading »

Debian live USB (or other live distributions like SystemRescueCd or KNOPPIX) are pretty handy in many situations. One can check (fsck) unmounted harddrives, scan system partitions or in general use the machine without touching the installed OS. Quite handy, indeed. What was missing on a Debian 6.0.7 live USB with XFCE4 was one SSH server to control the machine securely and remotely over the LAN (local area network).
I thought that a fast network setting, installing and starting the SSH server will suffice

# ifconfig
# ifconfig -a
# ifconfig eth0 192.168.132.111/24
# route add default gw 192.168.132.1
# cat /etc/resolv.conf
# echo "nameserver 8.8.8.8" > /etc/resolv.conf
# nslookup kernel.org
# apt-get update
# apt-get install ssh
# /etc/init.d/ssh start
or
# service ssh start

, but it didn’t start. There was nothing to be seen in

# ps aux | grep -i ssh
nor in
# netstat -plantu

I thought I was missing something and gave the users “root” and “user” passwords

# passwd
and
# passwd user

also changed /etc/ssh/sshd_config

# vim.tiny /etc/ssh/sshd_config
to
ListenAddress ::
ListenAddress 0.0.0.0

, but nada it just didn’t want to start with the init.d scripts. Before digging in deeper, I asked in the IRC #debian channel and tadaaa – the solution was to just manually start the sshd daemon:

# /usr/sbin/sshd

Simple, … now I am happy . Be happy you too!

P.S. for the ones that don’t read comments, this is the reason why it works this way:

I now checked, and I think I found out why. In the BASH script /etc/init.d/ssh this is the line that starts the daemon
“start-stop-daemon –start –quiet –oknodo –pidfile /var/run/sshd.pid –exec /usr/sbin/sshd”
In a normal installtation /sbin/start-stop-daemon is a binary file, in a live [debian] USB it is just a small bash script consisting of:

#!/bin/bash
 
exit 0

so that is why it does not start with the init.d scirpt. All cool?

P.S. Yep that is the reason, I replaced start-stop-daemon of the live debeian USB with one of debian wheezy installation and it works fine again with “/etc/init.d/ssh start”.

Well in the physical world it is always preferable to use strong locks and keys that are hard to be reproduced. Same thing is in the virtual world. Recently I saw that some friend from Russia bruteforced my wordpress login account. Am I scared? Fuck no! I mean, I don’t use password “123456″ or “password”, or “p@ssw0rd”. That is why I suggest to everyone to use strong passwords and to keep it cool. And under strong understand more than 10 characters that are mixed (alpha, numeric, symbols) – [a-z], [A-Z], [0-9] and [*symbols*].
Life can sometimes be so hard, it is not fair for that bruteforcing friend. That PHP is soooo slow my site is able to return 10-30 request per second. With that rate hat guy will grow a beard while bruteforcing. Hang on guy, I have planned to use some kind of cache – memcached or varnish, but I am still in testing mode, and my site is actually not that busy so the current speed is enough for now. But even with a cache and possibility to return thousands of request per second, bruteforcing is still pathetic. Let us calculate why.

[a-z] = 26 letters
[A-Z] = 26 letters
[0-9] = 10 numbers

That means that a 11 characters password composed of [a-zA-Z0-9] is a keyspace with (26+26+10)^11 possibilities which is ~ 51×10^18, which is crazy big number. If we are generous and give him 10000 tries per second, that means he is able to do 10000 tries per second x 60 sec. x 60 min. x 24 hours x 365 days = 315×10^9 tries per year. So our friend will need roughly a billion (10^9) years to complete the bruteforce. Good luck friend!
Maybe this mate is better off finding a 0-day in wordpress, but yeah, good luck with that too … finding a 0-day in an open source application used by millions x 100+. And I stress on open source, because java and adobe reader are also used by millions, but suck badly. They will never learn, or do it the hard way – http://eugene.kaspersky.com/2012/05/25/the-dangers-of-exploits-and-zero-days-and-their-prevention/.

If you want to see excerpt of the access log of my server follow the link
Continue Reading »

Often the packages in most GNU/Linux distributions are not the latest and greatest stable version. And that is ok, they need to work, not to be the latest. It is a lot of work to keep *all* the packages up to the latest version and often not so necessary for stable and robust OS. Sometimes it is preferable to compile and install software from source so that you can keep with the latest stable version. If you have one installation you can directly compile and install the newest software and if you have multiple you can create a distribution package, put it on local repository and install on multiple machines after it has been tested on one or two. And with servers and databases it is crucial to have a strategy how to stay up to date, because of the 0day vulnerabilities that are constantly found in the wild.
I have written how to compile and install from source Nginx with SSL, PHP, MariaDB and WordPress on Debian Wheezy and how to upgrade from source MariaDB and Nginx. So now I will write how to upgrade PHP from source.

Here we go, we need to download the newest stable source of PHP, *always* check the hash for integrity, extract the archive, configure and compile it like that (also it is good to test your compiled binary):

$ wget -c http://de1.php.net/get/php-5.4.12.tar.bz2/from/this/mirror -O php-5.4.12.tar.bz2
$ md5sum php-5.4.12.tar.bz2
5c7b614242ae12e9cacca21c8ab84818  php-5.4.12.tar.bz2
$ tar xjf php-5.4.12.tar.bz2
$ cd php-5.4.12/
$ ./configure --enable-fpm --with-mcrypt --with-zlib --with-curl --disable-debug --disable-rpath --enable-inline-optimization --with-bz2 --enable-sockets --enable-zip --with-fpm-user=www-data --with-fpm-group=www-data --with-mysql=/usr/local/mysql/
$ make
$ make test

Now stop php-fpm, if you have monitoring scripts that restart the service stop them too maybe with commands like # ps aux | grep -i php and then stop the master proces with # kill pid-master. After that we can install the compiled binary and start it with:

# export LD_LIBRARY_PATH=/usr/local/mysql/lib:LD_LIBRARY_PATH
# make install
# php-fpm

That is, we now have the latest and greatest PHP stable version. Have a good day .

Spam is just annoying. I wonder what are they actually achieving and what stupid people are checking spam links? I mean when I see a spam links in comments or wherever I consider this an anti-advertisement, but whatever.

When I started my blog spam comments started to increase, so I had to find a solution, because deleting 50 spam comments was starting to be frustrating. I analyzed my nginx access.log and saw that the spammers were using the track-back functionality of wordpress. This function is rarely used for good purpose so I found this very useful plugin – Disable Trackbacks and disabled it. It was a relief, not a single spam comment any more. I use also simple captcha just for a bit extra.

From this point in time the track-back spam comments were appearing only in my access.log file. It was all normal until some mad spammer started trashing my blog. He was not doing any damage, but just filling my log with his useless:

93.182.168.42 - - [11/Mar/2013:18:33:50 +0100] "GET /blog/?p=286 HTTP/1.1" 200 5780 "http://linuxtechnocrats.dyndns.org/" "Opera/9.80 (Windows NT 5.1; U; en) Presto/2.10.289 Version/12.01"
93.182.168.42 - - [11/Mar/2013:18:33:50 +0100] "POST /blog/wp-trackback.php?p=286 HTTP/1.1" 200 152 "http://www.*some stupid site*" "PHP/5.2.10"

I got tens to hundreds of these per minute when I decided that even that is too much and just blocked the ip and moved on.

# iptables -I INPUT -s 93.182.168.42 -j DROP

Bye bye Mr. Spammer. You need to get a new ip, but remember – My iptables is ready to engage!!!

Update: well I want to say that I like *open* Internet and do not find blocking or filtering techniques as something good. Although, in a situation of abuse these techniques become *necessary*. And in a situation of attack, it is good to have the ability to mitigate it.
Now on my case with the stupid spammer, I had to block another 3 IPs and my logs became clean and quiet. You can see that iptables has blocked more than 3000 packets from this looser.

# iptables -nvL

 pkts  bytes target     prot opt in     out     source               destination
  224  10752 DROP       all  --  *      *       93.182.189.56        0.0.0.0/0
  535  25680 DROP       all  --  *      *       93.182.170.142      0.0.0.0/0
 1896 95808 DROP       all  --  *      *       93.182.180.14        0.0.0.0/0
  616  29568 DROP       all  --  *      *       93.182.168.42        0.0.0.0/0

Rock on iptables!!!

While writing scripts we often have to save data in temporary files for further processing. As we know the workhorse programs of GNU/Linux (like grep, sed, awk, cut, head, sort, uniq …) work well with files and not with variables. Writing to hard disk is SLOW, yes SLOW. Even on SSD ones. Well things tend to get better on SSD, but still. I need to test them myself.

What is FAST and efficient for temporary files is RAM. Writing and reading temporary files from RAM is awesome. But how do we do it? Easy of course! Mount a directory, which is physically located in RAM with file system type tmpfs or ramfs.

# mkdir /mnt/temp
// if you want user access
# chown -R user:user /mnt/temp/
// fixed size, uses swap, vollatile, all in ram
# mount -t tmpfs -o size=20m,uid=1000,gid=1000 tmpfs /mnt/temp/
or
// dynamic size, no swap, vollatile, all in ram
# mount -t ramfs -o size=20m,uid=1000,gid=1000 tmpfs /mnt/temp/

Now you can read/write as much as you want, as fast as you can! Let us test the theory. We can create a single 1 Giga Byte file with writing 1000 times one Mega Byte chunks each. To create a 1GB file in RAM it takes about one second.

# time dd if=/dev/zero of=/mnt/tmpfs/1GB_file.txt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 1.18641 s, 884 MB/s

real    0m1.189s
user    0m0.000s
sys     0m1.056s

# time dd if=/dev/zero of=/mnt/tmpfs/1GB_file.txt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 1.06656 s, 983 MB/s

real    0m1.198s
user    0m0.000s
sys     0m1.180s

# time dd if=/dev/zero of=/mnt/tmpfs/1GB_file.txt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 1.02866 s, 1.0 GB/s

real    0m1.147s
user    0m0.004s
sys     0m1.128s

So do you still wonder how Google returns millions of links in a split of a second? I guess they keep the whole Internet (or the processed database out of it) in their RAM.

Now writing to the hard drive. I repeat the test several times to show the cache effect and that the tests are consistent. So to create 1GB file on the hard disk it takes about 25 seconds.

# time dd if=/dev/zero of=/mnt/sda2/vboxshare/1GB_file.txt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 30.82 s, 34.0 MB/s

real    0m30.871s
user    0m0.000s
sys     0m1.936s

# time dd if=/dev/zero of=/mnt/sda2/vboxshare/1GB_file.txt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 22.7691 s, 46.1 MB/s

real    0m22.993s
user    0m0.000s
sys     0m2.152s

# time dd if=/dev/zero of=/mnt/sda2/vboxshare/1GB_file.txt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 19.4128 s, 54.0 MB/s

real    0m25.087s
user    0m0.004s
sys     0m2.292s

# time dd if=/dev/zero of=/mnt/sda2/vboxshare/1GB_file.txt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 20.7324 s, 50.6 MB/s

real    0m23.648s
user    0m0.000s
sys     0m2.276s

# time dd if=/dev/zero of=/mnt/sda2/vboxshare/1GB_file.txt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 20.0088 s, 52.4 MB/s

real    0m23.806s
user    0m0.004s
sys     0m2.124s

Well there you have it – about 20 times faster on my mashine. My laptop hard drive is not very fast, but that is the case with my RAM too.

Well, upgrading infrastructure can be a bit of a headache, especially upgrading web servers where no downtime is expected. Upgrading Nginx though, is a pleasant job. And of course we all know that keeping software up-to-date is very good security practice.
Now here is how its done. If you have compiled Nginx yourself, you already have all the dependencies on your system so it just a matter of compile, build, install. If you haven’t you can check the installation from source here. Now because Nginx is so sophisticated (this word is one of my favorite) we do not need to stop/kill/shut it down. The command “make upgrade” does that for us while the web server is running, so no lost requests, no side effects. Here are the commands:

$ cd /home/debian/programms/nginx
$ wget -c http://nginx.org/download/nginx-1.2.6.tar.gz
$ tar xvzf nginx-1.2.6.tar.gz
$ cd nginx-1.2.6/
$ ./configure --with-http_ssl_module
$ make
# make install
# make upgrade

You can test your version with:

$ curl -I host
$ /usr/local/nginx/sbin/nginx -V

That’s it!

P.S.
For completeness I will show what exactly the “make upgrade” command does:

# -t Don't run, just test the configuration file for correctness
/usr/local/nginx/sbin/nginx -t
# USR2 upgrade executable on the fly 
kill -USR2 `cat /usr/local/nginx/logs/nginx.pid`
sleep 1
# check if file exist and is a regular file
test -f /usr/local/nginx/logs/nginx.pid.oldbin
# QUIT graceful shutdown of the old executable 
kill -QUIT `cat /usr/local/nginx/logs/nginx.pid.oldbin`

What can I say more, that is the beauty of open source. Kudos Nginx people!

As we all know, there are no perfect things in our world. As this applies to anything, so is software one of this things. Using quality open source software such as GNU/Linux and BSD we are several steps ahead towards perfection. This software is made *by design* to be robust, flexible, fast and secure. A base server installation of Debian OS is about 800MB and consists of the Linux kernel, several GNU components like powerful shell (bash, sh, zsh, ksh, tcsh, csh ..) and some command line tools. No GUI (grafical user interface) nonsense what so ever, no bloatware, no stupid programs leaking memory. And there are enough facts proving that the theoretical design and its practical implementation works more than good, like *nix machines with years of uptime (w/o restart). My server has 51 days uptime as of 16.12.2012 137 days uptime as of 11.03.2013.
That said, we can never exclude the possibility of failure and we better be prepared for it. Nginx and MariaDB are great open source pieces of software, very robust, very fast, very secure and very reliable. The thing is that the web server is exposed to the wild wild interwebz and is constantly under load and hacking attacks. That is why at some point it may stop or fail. This won’t be big deal if we monitor it and restart it, so here is my bash script monitoring Nginx, MariaDB and PHP:

#!/bin/bash
 
#define services to be monitored without false positive, test with ps
#services="mysql nginx php-fpm", separated with space
services="/usr/local/mysql/bin/mysqld /usr/local/nginx/sbin/nginx php-fpm"
 
#define the commands to restart those services
mysqlCommand='/usr/local/mysql/bin/mysqld_safe --user=mysql &'
nginxCommand='/usr/local/nginx/sbin/nginx'
phpCommand='export LD_LIBRARY_PATH=/usr/local/mysql/lib:LD_LIBRARY_PATH;php-fpm;'
 
#log the actions
LOG="/home/debian/documents/servicesMonitorLog.txt"
 
#start monitoring
while true; do
for s in $services; do
	#echo "$services $s"
	currentService=`ps aux | grep -i $s | grep -i root | grep -iv grep`; #echo "$currentService"
	if [ -z "$currentService" ]; then true; #echo "$s is not working"; 
		if [ "$s" = "/usr/local/mysql/bin/mysqld" ]; then
		#echo "starting $mysqlCommand";
		eval $mysqlCommand; echo "`date +"%Y%m%d %H:%M:%S"` restarted $s" >> $LOG
		elif [ "$s" = "/usr/local/nginx/sbin/nginx" ]; then
		#echo "starting $nginxCommand";
		eval $nginxCommand; echo "`date +"%Y%m%d %H:%M:%S"` restarted $s" >> $LOG
		elif [ "$s" = "php-fpm" ]; then
		#echo "starting $phpCommand";
		eval $phpCommand; echo "`date +"%Y%m%d %H:%M:%S"` restarted $s" >> $LOG
		else true #do nothing
		fi
		#echo "sleeping 7s to give process time to start";
		sleep 7; #give time to the restarted service
	else true #echo "not empty, service is working";
	fi
done #for loop
sleep 5 #check every 5 seconds
done #while loop

And the command to start it (redirecting standard out and standard error to /dev/null (or the so called “bit bucket”)) is:

# sh /home/debian/programms/myscripts/monitorService002.sh > /dev/null 2>&1 &

If you want to put the script in a cron job, just remove the while loop (leave the for loop though) and you are good to go.

Using latest software version is a good security practice. Recently I saw on the mailing list a security warning, so I decided to upgrade my installation. The good thing is that the new installation does not delete the existing users and databases. It just replaces the binaries and adds the “test” database, which we will remove with the “mysql_secure_installation” bash script.
I want to add here, that before I make critical changes I always test it in my test VM (virtual machine). Then I backup things. And then I make the changes. It is the best way to save yourself a headache.
Alright here is how I did it:

First download the latest source *tar.gz file. Then check the integrity of the file, extract, compile and build like this:

$ md5sum mariadb-5.5.28a.tar.gz
$ tar xvzf mariadb-5.5.28a.tar.gz
$ cd mariadb-5.5.28a/
$ cmake -LAH
$ make

Now stop your server, if it is running. I just kill the [two] mysql processes:

# ps aux | grep -i myqsl
# kill -9 "process pid" "proces pid"

After that install the newly compiled binary:

# make install

Start your server:

# /usr/local/mysql/bin/mysqld_safe --user=mysql &

The “test” database is added as user “root” so we need to change its privileges to be able to drop (delete) it:

# chown -R mysql:mysql /usr/local/mysql/data/

Now we can secure the installation with the bash script (My answers were: “n” “Y” “Y” “Y” “Y”.):

$ /usr/local/mysql/bin/mysql_secure_installation

That’s it. Enjoy your shiny new MariaDB database server.