( cs / en )

Gitorious @ Ubuntu server 9.10 (April+May 2010)

In this tutorial we will install Gitorious git hosting to the fresh Ubuntu 9.10 server. Fake location used here is git.githosting (and others for development and so), change to your taste. Message queuing will be done by Apache ActiveMQ and the Ruby server itself will be served by Apache+Passenger.

Everything will be installed to the following locations (of course you can change it but don't miss any place where it is hardwritten):

/opt
  gitorious
    .ssh
    conf
    log
    mainline
  repositories
  tarballs
    cache
    work

Plus we will be using this temporary directory (because server install needs a lot of stuff):

/root/src

Server has the first user with sudo rights but Gitorious is using another one named git with limited rights. (Since I installed Redmine on the same virtual machine also, the group is gitslave and not the default git. I let it be for this text.) Majority of steps is performed as root - they are often marked as sudo but in reality I was running most of them as sudo -i. For various reasons operation with /root/src are not marked as needing sudo, keep that in mind. Remarks in code are marked with []. Text to write in files is nested even more and background color is different.

Program versions are typically the last ones in the time of installing (which means April+May 2010), not writing. Sometimes it's not a problem, sometimes is (see for example remarks for ActiveMQ later). Some gems will install an army of others with them, in the following text all important/main ones should be listed.

Big thanks to the gitorious mailing list and following sources:

Ruby on Rails (with MySQL and Apache+Passenger)

GIT and some other libraries:

sudo apt-get install -y git-core git-svn
sudo apt-get install -y apg build-essential libpcre3 libpcre3-dev sendmail make zlib1g zlib1g-dev ssh

Since Gitorious is Ruby-on-Rails application we need Ruby also. Gotcha for Ubuntu (and Debian) at least - don't mix Ruby gems from packages with Ruby ones, they install to different locations. Gitorious should run fine on Ruby 1.9 (in which case you get RubyGems for free), but since Redmine needed Ruby 1.8, here's the more complicated way:

Ruby 1.8 and some other libraries:

sudo apt-get install -y ruby1.8 libbluecloth-ruby libopenssl-ruby1.8 ruby1.8-dev ri rdoc irb
sudo ln -s /usr/bin/ruby1.8 /usr/bin/ruby

RubyGems, from sources:

[mkdir ~/src]
cd ~/src
wget http://rubyforge.org/frs/download.php/70696/rubygems-1.3.7.tgz
tar xzf rubygems-1.3.7.tgz
cd rubygems-1.3.7
sudo ruby setup.rb
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem
cd ..

If you are not going to miss gem's documentation (on the server), turn it off in configuration install-wise:

sudo vim ~/.gemrc

gem: --no-ri --no-rdoc

Otherwise you can use sudo gem install --no-ri --no-rdoc GemName with every single gem.

Some other common libraries:

sudo apt-get install -y libonig-dev libyaml-dev geoip-bin libgeoip-dev libgeoip1

RMagick:

sudo apt-get install -y imagemagick libmagickwand-dev
sudo gem install rmagick

Now we need database. Ruby gem mysql-ruby officially doesn't support MySQL greater than 5.0.x. On Ubuntu 9.10 you can install this version but newer one will force MySQL 5.1 on to you. AFAICT gem mysql-ruby works with it pretty well in this case.

MySQL 5.0:

sudo apt-get install -y mysql-client-5.0 mysql-server-5.0 libmysqlclient15-dev
sudo gem install mysql

For MySQL 5.1 you would run:

sudo apt-get install -y mysql-client mysql-server libmysqlclient-dev

Next step is search infrastructure. Searching is important functionality but getting it right is not an easy task *_* In fact in the end I've never used my local installation of Gitorious that much so I don't know if everything is ok. But I was brave (or stupid ^_^) enough to run Gitorious tests so throughout the text are scattered remarks about getting Ultrasphinx to run without (most of) errors.

Sphinx and Ultrasphinx:

cd ~/src
wget http://www.sphinxsearch.com/downloads/sphinx-0.9.9.tar.gz
tar xzf sphinx-0.9.9.tar.gz
cd sphinx-0.9.9
./configure
make && sudo make install
cd ..
sudo gem install ultrasphinx

Ultrasphinx needs aspell:

sudo apt-get install -y aspell libaspell-dev aspell-en
sudo gem install raspell

Well, this is not everything needed for search by far -_-" See you later...

ActiveMQ:

sudo apt-get install -y uuid uuid-dev openjdk-6-jre
cd ~/src
wget http://ftp.sh.cvut.cz/MIRRORS/apache/activemq/apache-activemq/5.3.2/apache-activemq-5.3.2-bin.tar.gz    [use mirror near to you]
sudo tar xzf apache-activemq-5.3.2-bin.tar.gz -C /usr/local/
sudo sh -c 'echo "export ACTIVEMQ_HOME=/usr/local/apache-activemq-5.3.2" >> /etc/activemq.conf'
sudo sh -c 'echo "export JAVA_HOME=/usr/" >> /etc/activemq.conf'
sudo adduser --system --no-create-home activemq
sudo chown -R activemq /usr/local/apache-activemq-5.3.2/data

Now we have to configure it. Trouble is the configuration is different for different versions of ActiveMQ. For version 5.3.2 use this:

sudo vim /usr/local/apache-activemq-5.3.2/conf/activemq.xml

<transportConnectors>
  <!-- Add this line -->
  <transportConnector name="stomp" uri="stomp://localhost:61613"/>
</transportConnectors>

And for version 5.2.0 use this:

<networkConnectors>
  <!-- by default just auto discover the other brokers -->
  <!--networkConnector name="default-nc" uri="multicast://default"/-->
  <!-- Example of a static configuration: -->
  <networkConnector name="localhost" uri="static://(tcp://127.0.0.1:61616)"/>
</networkConnectors>

After configuration let's demonize it:

cd ~/src
wget http://launchpadlibrarian.net/15645459/activemq
sudo cp activemq /etc/init.d/activemq
sudo chmod +x /etc/init.d/activemq
sudo update-rc.d activemq defaults

If defaults doesn't satisfy you try for example:

sudo update-rc.d activemq start 70 2 3 4 5 . stop 30 0 1 6 .

PS - if you're lucky (I was not) you can see the current queue at http://<host>:8161/admin.jsp after issuing the next command:

/usr/local/apache-activemq-5.3.2/bin/activemq

Memcache:

sudo apt-get install -y memcached
vim /etc/default/memcached      [and set it from no to yes]

PS - you shouldn't need this in case of successful installation:

sudo update-rc.d memcached defaults

Apache:

sudo apt-get install -y apache2

Later you will need these for running Passenger:

sudo apt-get install -y apache2-prefork-dev libapr1-dev libaprutil1-dev

Another needed gems (not installed so far):

sudo gem install rails mongrel mime-types textpow ruby-hmac daemons oniguruma textpow BlueCloth ruby-yadis ruby-openid geoip rspec rspec-rails RedCloth echoe rdiscount stomp diff-lcs json passenger

If you want to run tests you will also need these gems (mocha is for "rake test", next two for subtests):

sudo gem install [--no-ri --no-rdoc] mocha
sudo gem install [--no-ri --no-rdoc] ruby-prof sqlite3-ruby

Mail configuration:

...is left up to you. Sendmail works but can be tricky to configure if there is something wrong. Probably you will need "sudo vim /etc/mail/local-host-names" or something like that at least. Any help appreciated.

GITORIOUS

User for Gitorious:

sudo groupadd gitslave
sudo adduser --gecos 'git management system' --ingroup gitslave --disabled-password --home /opt/gitorious git

Installation:

su - git        [Mind the -. It changes your path to /opt/gitorious among other things.]
#mkdir log conf repositories tarballs-cache tarballs-work
mkdir log conf
git clone git://gitorious.org/gitorious/mainline.git mainline
cd mainline/
mv public/.htaccess public/.htaccess.ORIGINAL    [in case you want to look at it later]
mkdir -p tmp/pids
chmod ug+x script/*
chmod -R g+w config/ log/ public/ tmp/
exit        [So we are now back to root.]
sudo ln -s /opt/gitorious/mainline/script/gitorious /usr/local/bin/gitorious

Configuration, step 1 - git daemon:

Change GID_DAEMON a PID_FILE (/usr/bin/ruby and path to www, ie. /opt/gitorious/mainline/XXX) in:

sudo vim /opt/gitorious/mainline/doc/templates/ubuntu/git-daemon

And demonize it:

sudo update-rc.d -f git-daemon start 80 2 3 4 5 . stop 20 0 1 6 .    [or "sudo update-rc.d git-daemon defaults"]

Ultrasphinx, story continues:

Change paths from /var/www/gitorious/ to /opt/gitorious/mainline/ (5 times).

sudo vim /opt/gitorious/mainline/doc/templates/ubuntu/git-ultrasphinx

sudo ln -s /opt/gitorious/mainline/doc/templates/ubuntu/git-ultrasphinx /etc/init.d/git-ultrasphinx
sudo ln -s /opt/gitorious/mainline/doc/templates/ubuntu/git-daemon /etc/init.d/git-daemon
sudo chmod +x /etc/init.d/git-ultrasphinx
sudo chmod +x /etc/init.d/git-daemon

Demonize it (or maybe wait for later since it is STILL not everything):

sudo update-rc.d -f git-ultrasphinx start 80 2 3 4 5 . stop 20 0 1 6 .        [or "sudo update-rc.d git-ultrasphinx defaults"]

Other needed gems and directories:

Gitorious needs a lot of gems but we took care of them earlier already. For the record it were (I probably missed some): rails mongrel mime-types textpow chronic ruby-hmac daemons oniguruma textpow BlueCloth ruby-yadis ruby-openid geoip rspec rspec-rails RedCloth echoe .

sudo mkdir /opt/repositories
sudo mkdir -p /opt/tarballs/cache
sudo mkdir /opt/tarballs/work
sudo chown -R git:gitslave /opt/repositories /opt/tarballs /opt/tarballs/cache /opt/tarballs/work

PS - Since you're not installing Redmine you don't need this:

sudo chmod -R g+sw repositories/ tarballs/   [ie. let's keep new files as :gitslave]

Prepare ssh access:

su - git
mkdir ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

If something is not working check the rights and ssh-keys first ^_~

Configuration, step 2 - gitorious:

[we are still logged in as user git!]
cd /opt/gitorious/mainline
cp config/database.sample.yml config/database.yml
cp config/gitorious.sample.yml config/gitorious.yml
cp config/broker.yml.example config/broker.yml

Generate the secret string (and copy it to in gitorious.yml instead of "ssssht" string):

apg -m 64

Theoretically you need production environment only so adjust accordingly (in fact running tests is close to nightmare ^_^ if you want to do that don't change default gitorious.test for gitorious_host setting, many of tests expect this exact name here...):

vim config/gitorious.yml

production:
 cookie_secret: ssssht
 repository_base_path: "/opt/repositories"
 extra_html_head_data:
 system_message:
 gitorious_client_port: 80
 gitorious_client_host: git.githosting
 gitorious_host: git.githosting
 gitorious_user: git
 exception_notification_emails: USER@git.githosting
 mangle_email_addresses: true
 public_mode: true
 locale: en
 archive_cache_dir: "/opt/tarballs/cache"
 archive_work_dir: "/opt/tarballs/work"
 only_site_admins_can_create_projects: true
 hide_http_clone_urls: false
 is_gitorious_dot_org: false
 #default_license: GNU Affero General Public License (AGPLv3)

development:
 cookie_secret: ssssht
 repository_base_path: "/tmp/git-repos"
 extra_html_head_data:
 system_message:
 gitorious_client_port: 80
 gitorious_client_host: git.local
 gitorious_host: git.local
 gitorious_user: git
 exception_notification_emails: USER@git.githosting
 mangle_email_addresses: true
 public_mode: true
 locale: en
 archive_cache_dir: "/tmp/git-tarballs-cache"
 archive_work_dir: "/tmp/git-tarballs-work"
 only_site_admins_can_create_projects: true
 hide_http_clone_urls: false
 is_gitorious_dot_org: false
 #default_license: GNU Affero General Public License (AGPLv3)

test:
 cookie_secret: ssssht
 repository_base_path: "/tmp/git-repos"        # The path where git repositories are stored. The actual (bare) repositories resides in repository_base_path/#{project.slug}/#{repository.name}.git/
 extra_html_head_data:
 system_message:       # System message that will appear on all pages if present
 gitorious_client_port: 3000   # Port the ./script/gitorious script should use:
 gitorious_client_host: localhost      # Host the ./script/gitorious script should use:
 gitorious_host: gitorious.test     # Host which is serving the gitorious app, eg "gitorious.org" [Again - here has to be 'gitorious.test', otherwise hundrets of tests will fail!]
 gitorious_user: git   # User which is running git daemon
 exception_notification_emails: USER@git.githosting
 mangle_email_addresses: true
 public_mode: true     # Enable or Disable Public Mode (true) or Private Mode (false)
 locale: en
 archive_cache_dir: "/tmp/git-tarballs-cache"
 archive_work_dir: "/tmp/git-tarballs-work"
 only_site_admins_can_create_projects: false
 hide_http_clone_urls: false           # Should we hide HTTP clone urls?
 is_gitorious_dot_org: false
 #default_license: GNU Affero General Public License (AGPLv3)

Configuration, step 3 - database:

Modify access to your database (probably you will need to change user and password, see next step):

vim config/database.yml

Create needed tables (replace YOURPASSWORD with your chosen password for this database user, no need for him to be git):

mysql -u root -p
> create database gitorious_production character set utf8;
> create database gitorious_test character set utf8;
> create database gitorious_development character set utf8;
> create user 'git'@'localhost' identified by 'YOURPASSWORD';
> grant all privileges on gitorious_production.* to git@localhost;
> grant all privileges on gitorious_test.* to git@localhost;
> grant all privileges on gitorious_development.* to git@localhost;
> \q

exit    [So we're now back to root again.]

Other configurations:

sudo vim /etc/hosts

127.0.0.1   git.githosting git.local

Let's prepare the rest:

Gitorious won't run with the gems installed above. It needs these two versions explicitly:

sudo gem install stomp -v 1.1
sudo gem install rdiscount -v 1.3.1.1

You wouldn't run into this problem if you installed them through RoR mechanism, but then they would be installed locally for user git. Which is not necessarily bad thing. I've just tried to have everything as simple as possible.

Now we can proceed with database migration:

su - git
cd ~/mainline
rake db:migrate RAILS_ENV=production

And create admin for the gitorious web:

env RAILS_ENV=production ruby script/create_admin

exit [Now we're back to root.]

Ultrasphinx, story continues, again:

We are missing word-list:

cd /opt/gitorious/mainline
sudo cp vendor/plugins/ultrasphinx/examples/ap.multi /usr/lib/aspell

Configuration (the first command is run as user git, the other two not):

rake ultrasphinx:configure RAILS_ENV=production
sudo rake ultrasphinx:spelling:build RAILS_ENV=production
sudo chown -R git:gitslave /opt/gitorious/mainline/db/sphinx

su - git        [So we are now git and in his home /opt/gitorious.]

Initialization:

cd mainline
rake ultrasphinx:bootstrap RAILS_ENV=production

You can replace "address" with "listen" (10th row) and save some errors in logs:

cd mainline/config/ultrasphinx
vim production.conf
Cron-job for Ultrasphinx (runs every hour):

~ git under crontab is missing indexer, so (we are still logged in as git):

vim .bash_profile

export PATH="$PATH:/usr/local/bin"


crontab -e

> 0 * * * * source /opt/gitorious/.bash_profile && cd /opt/gitorious/mainline && rake ultrasphinx:index RAILS_ENV=production >>/opt/gitorious/log/cron.log 2>>/opt/gitorious/log/cron-error.log

PS: Those logs are typically >/dev/null 2>&1 but Ultrasphinx was such a beast to tame that I dedicated logs for it personally...

exit    [So again back to root.]

Back to Gitorious

Do we have the permissions right?

cd /opt/gitorious/mainline
chmod g+w script/poller

Let's test it!

For running by hand:

sudo /etc/init.d/activemq start
sudo env RAILS_ENV=production /etc/init.d/git-daemon start
su git -c "cd /opt/gitorious/mainline && env RAILS_ENV=production script/poller start"

For interactive watching (while running by hand) change the last line (mind run at the end):

su git -c "cd /opt/gitorious/mainline && env RAILS_ENV=production script/poller run"

Test WWW-server:

su git -c "cd /opt/gitorious/mainline && script/server -e production"

Demonizing

When everything above is ok, you want to demonize it all and move from test server to Passenger+Apache.

For demonizing you can use daemontools (which guards process by default) or init.d-scripts (and use cron-jobs for guarding, since some people report activemq and poller dying heavilly). I'm not familiar enough with both options so I provide just some remarks here. Help appreciated.

Demonizing ActiveMQ:

We've already done that during installation.

PS: During development you can use stompserver instead of Apache ActiveMQ. Don't mix them together.

Demonizing git-daemon:

We've already done that earlier.

Demonizing script/poller:

Try this (I haven't tested this one):

sudo vim /etc/init.d/git-poller

#!/bin/sh
# Start/stop the git poller
#

### BEGIN INIT INFO
# Provides: git-poller
# Required-Start: stomp
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 1
# Short-Description: Gitorious poller
# Description: Gitorious poller
### END INIT INFO

/bin/su – git -c "cd /opt/gitorious/mainline;RAILS_ENV=production script/poller $@"


sudo chmod 755 /etc/init.d/git-poller
sudo update-rc.d git-poller defaults

Or this:

sudo vim /etc/init.d/git-poller

#!/bin/sh

start()
{
    /bin/su git -c "cd /opt/gitorious/mainline && env RAILS_ENV=production script/poller start"
}

stop()
{
    /bin/su git -c "cd /opt/gitorious/mainline && env RAILS_ENV=production script/poller stop"
}

case "$1" in
    start)
           start
           ;;
    stop)
           stop
           ;;
    restart)
           stop
           start
           ;;
    *)
           echo $"Usage: $0 {start|stop|restart]"
esac


sudo chmod +x /etc/init.d/git-poller
sudo update-rc.d -f git-poller start 80 2 3 4 5 . stop 20 0 1 6 .        [or "sudo update-rc.d git-poller defaults"]

Preparing the server

Prepare logrotate:

cd /opt/gitorious/mainline

Replace /var/www with /opt/gitorious/mainline (2 times):

sudo vim /opt/gitorious/mainline/doc/templates/ubuntu/gitorious-logrotate

sudo ln -s /opt/gitorious/mainline/doc/templates/ubuntu/gitorious-logrotate /etc/logrotate.d/gitorious

These two should not be needed:

sudo chmod +x /etc/logrotate.d/gitorious
sudo chmod 644 /etc/logrotate.d/gitorious

Finish installation of Passenger:

For the record - we've already done this:

sudo gem install passenger
sudo apt-get install apache2-prefork-dev libapr1-dev libaprutil1-dev

So let's finish it with:

sudo passenger-install-apache2-module

Which tells us to do (for gem passenger version 2.2.11 and Ruby 1.8, mind your versions!):

sudo vim /etc/apache2/mods-available/passenger.load

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.11/ext/apache2/mod_passenger.so


sudo vim /etc/apache2/mods-available/passenger.conf

PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.11
PassengerRuby /usr/bin/ruby1.8

Needed Apache modules:

sudo a2enmod rewrite
sudo a2enmod passenger
sudo a2enmod expires

You also need deflate but it is probably already enabled:

sudo a2enmod deflate

And if you want to use https acces, you'll also need module ssl (and appropriate configuration as follows):

sudo a2enmod ssl

Apache configuration:

su git        [We are git now.]
vim /opt/gitorious/conf/vhost.conf

<VirtualHost *:80>
  ServerName git.githosting
  DocumentRoot /opt/gitorious/mainline/public
  RailsEnv production
  <Directory /opt/gitorious/mainline/public>
    AllowOverride all
    Options -MultiViews
  </Directory>
  ErrorLog /opt/gitorious/log/error.log
  CustomLog /opt/gitorious/log/access.log combined
</VirtualHost>
<IfModule mod_ssl.c>
  <VirtualHost _default_:443>
    DocumentRoot /opt/gitorious/mainline/public
    ErrorLog /opt/gitorious/log/error-ssl.log
    CustomLog /opt/gitorious/log/access-ssl.log combined
    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
    SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
    BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
  </VirtualHost>
</IfModule>


exit    [And now we are back to root.]
sudo ln -s /opt/gitorious/conf/vhost.conf /etc/apache2/sites-available/git.githosting
sudo a2ensite git.githosting
sudo /etc/init.d/apache2 restart

Congratulation! You can now register to your Gitorious installation and start to play with it.