Efficiently View Apache Log Files on Windows

One of the annoying things about using multiple platforms is when one platform has a useful utility (no matter how small) and the other platform doesn't. Have you ever needed to regularly check an Apache log file on your Windows development machine? The shell user inside me says "just tail it"... but this is Windows.

However, I just found a really amazing tool called BareTail.



BareTail "connects" to a log file, and shows you an automatically updated (live) tail of that file. It basically allows this situation to happen:

I have a window up on my developer machine (in the 2nd monitor) which is the tail of the Apache error log. Every time something is written to that log, BareTail pushes it into the window on my screen. I see log entries in real time.

Let me tell you, it makes debugging Apache error log issues much more efficient.

Mercurial hgwebdir under Apache using WSGI

The doc for setting up HgWebDirStepByStep sure make it seem harder than it really is. Maybe it is hard. Maybe it's just that I spent so much time trying to understand the doc, that when I actually just tried it myself it wasn't that bad.

I've been thinking of moving over from SVN to Mercurial (hg) but wasn't sure if it was worth the fuss. I moved. It seems nice. Here's a guide, including tutorials for both the server and client, to switch to hgwebdir (server) /TortoiseHg (client).

Let's get started.

Setting up Mercurial (Part 1)
Before we configure the server, the following must already be installed and running:

• Python 2.x (not ActivePython)
  
• Apache 2
• mod_wsgi under Apache

I'm going to be demonstrating the setup on Windows 2000 Server as the host for hgwebdir/apache, and Windows Xp as the developer machine.

1. Setup the Directory Structure
Create the following structure:

C:\Mercurial
\src
\hgwebdir
\repositories

We'll be putting files into these folders shortly.

2. Download mercurial source
Whatever you do, don't install TortoiseHg on the server. To run hgwebdir you're going to want to have compiled mercurial yourself.

Download Mercuria sourcel (Be sure to not get the binary)

After download, extract the contents to C:\Mercurial\src, or wherever you created your 'src' folder in the directory structure from above.

3. Build Mercurial
(Instructions taken mostly from WindowsInstall wiki page)
Open up the command prompt...

C:\>
C:\> cd mercurial\src
C:\Mercurial\src> python setup.py --pure build_py -c -d . build_ext -i build_mo --force
C:\Mercurial\src> python setup.py --pure install --force--force

Mercurial will now be built and installed into your python site-packages folder.

4. Setup a 3-way Diff tool
Download KDiff3 from SourceForge. Install using the included windows setup binary.

5. Configure Environment Variables

Add "C:\Python26\Scripts" and "C:\Program Files\KDiff3" to your PATH environment variable.

You will need to be logged into an account that has admin right on the machine to setup this next part. Right-click on "My Computer" and choose "Properties". Choose "Advanced", then click the button "Environment Variables." In the list at the bottom, "System variables", scroll down and find PATH. Click "Edit". At the end of the textbox add a semi-colon, then type in the path from above. Your PATH variable should then resemble...

...;C:\Program Files\KDiff3;C:\Program Files\KDiff3;

6. Create a local Mercurial configuration file
Windows Explorer has a limitation that won't let you create a new file that starts with a period. The way around this is to use notepad on the command line to type in the name of a file with a period.

Mercurial uses a local file named ".hgrc" to configure itself about which 3-way merge tool it should use and similar things to that nature.

Follow these commands to create the .hgrc file for the server:

C:\>
C:\> cd Python26\Scripts
C:\Python26\Scripts> notepad .hgrc

This will cause notepad to pop up and ask if you want to create a new file named .hgrc. You do. Here's the basic configuration file for mercurial. Edit these settings to whatever you need to be.

[ui]
editor = Notepad
username = hgadmin <webmaster@yourdomain.com>

[merge-tools]
kdiff3.priority=-1
kdiff3.args=-L1 base --L2 local --L3 other $base $local $other -o $output
kdiff3.regkey=Software\KDiff3
kdiff3.regappend=\kdiff3.exe
kdiff3.fixeol=True
kdiff3.gui=True

You've got Mercurial installed now! If you open a command prompt and type 'hg' it should print out the command's help.

Setting up Apache/hgwebdir
Hmmm... ok maybe that is a lot of work. It didn't seem like it at the time.

Going into this, I'm assuming you've already got Apache with mod_wsgi up and running. I should point out that you can technically run hgwebdir under CGI or mod_python, but as WSGI is the hotness now days probably a lot of people have already started using that.

Good news is, I think this part is a lot easier.

On my setup I've only got 1 server, and so I tend to run things as Virtual Hosts. If you're already running Apache I'm assuming you've got the smarts to change this config if you like a different setup, but for the example httpd.conf below, I'll be doing a virtual host.

1. Create our WSGI handler file
The WSGI file is really simple. We basically need to 1.) import the necessary modules, then 2.) pass a new instance of hgwebdir with the configuration file path back to WSGI handler.

Create this script and save it as "hgwebdir.wsgi" into the C:\Mercurial\hgwebdir folder.

2. WSGI app config file
You probably noticed in the code above that we're referencing a config file that doesn't yet exist. Let's go ahead and create that.

[web]
style = coal

[paths]
/ = C:/Mercurial/repositories/**

Save this file as "hgweb.config" into the C:\Mercurial\hgwebdir folder.

3. Configure Apache
So we've got our WSGI script and config file setup. All that's left now is to configure Apache.

We need to have a few things in our Apache conf in order to get everything working: 1.) A new virtual host to hold all of our Mercurial settings, 2.) a WSGIScriptAlias to make sure our hgwebdir.wsgi file handles web request, 3.) authentication (skip this if you want a public repository), and 4.) SSL.

All of the above goes in httpd.conf.

There's lots of stuff going on in there. Let's look at a bit of it.

Even though we've configured hg.yourdomain.com, unless you have actually setup an A record of "hg" for yourdomain.com, that address won't actually resolve (ie. it won't work).

The WSGIScriptAliasMatch is actually what routes our incoming Mercurial requests to the hgwebdir.wsgi file for handling. In order for this directive to work, you'll need to have mod_wsgi enabled in your httpd.conf file. (this is outside the scope of this post)

Just copy and paste the SSL configuration from elsewhere in your config file if you're already using it. If you don't have a cert or don't want to use SSL then delete all of that section and change the ports at the top back to 80.

Lastly, did you notice the auth part of the configuration? We're using AuthType Basic and we have a user file "C:\Mercurial\accounts". We haven't created that yet, so let's go ahead and do so.

In Part 2 when we look at setting up the client, I'll show you how to configure TortoiseHg to save the user/pass that will be authenticated here by Apache.

4. Setting up Mercurial Users
It's important to note at this point that the authentication is being handled by Apache and not Mercurial. In my setup we're just using HTTP Basic, however if your Apache install is already using something better (LDAP, DBM, whatever), then by all means use that instead.

If you've never made an Apache passwd file, it's an easy process. Apache gives you a tool "htpasswd" which allows you to create a new passwd file or to add new users to an existing file.

* Important: I don't know where your Apache is installed, so you'll need to interpret some of these paths for your specific machine.

htpasswd is located in the Apache bin folder. We're talking something to the effect of C:\Apache2.2\bin\htpasswd.exe. Figure that out, and then use the command line as indicated below to create our accounts file for Mercurial.

C:\>
C:\> cd Apache2.2\bin
C:\Apache2.2\bin> htpasswd "C:\Mercurial\accounts" testuser

This will create the file "accounts" in C:\Mercurial, add a new user "testuser" and then prompt you for a password for the new user. You can repeat this command later on to add new users.

5. Bask in how fashionable you are using Mercurial
If everything went well up to this point, you should be able to start Apache and try out your new hgwebdir install.

But upon arrival you'll probably notice that... it's empty. Let's create a new repository.

6. Adding repositories
Remember the folder we made C:\Mercurial\repositories? That's where all the repos will be living. Create a new repository like you would locally...

C:\>
C:\> cd Mercurial\repositories
C:\Mercurial\repositories\> mkdir testrepo
C:\Mercurial\repositories\> cd testrepo
C:\Mercurial\repositories\testrepo\> hg init

Refreshing your the URL of your hgwebdir install should now show a new read-only repository. By default new repositories are not push-able.

7. Create a repository configuration file
We can enable features and also fill out the details about repository by create a configuration file. Assuming the repository you created was "testrepo", create a new file named "hgrc" in C:\Mecurial\repositories\testrepo\.hg\ folder.

[web]
contact = Your Name
name  = Test Repository
description = A repository that I can show how cool I am using hg
allow_push = *
allow_archive = gz zip

Most of this is self-explanatory. The "allow_push" directive enables files to be synchronized from the client's repository up to the repository on the server. The "allow_archives" will enable a snapshot of a given revision to be downloaded in those formats.

Save this file, refresh and there you go.

Next, we'll look at setting up TortoiseHg on a Windows XP developer box to connect to our new hgwebdir instance.

Django, Apache 2.2, mod_ssl, and mod_wsgi (Part 3)

Part 1 | Part 2 | Part 3

Serving up Django
Following the completion of Part 2, we have an install of Django and Apache, and 2 virtual hosts running on it. What we need to do is connect those virtual hosts up to the content we need to serve.

Before we can do that, we need to add our Django app to the server. Create a folder on your webserver (anywhere really) that isn't in your document root (don't use htdocs). In my case I just made a folder off of the drive "Root_Django". Copy your django app folder into there. You should end up with something like...

C:\Root_Django\{nameofyourwebapp}

To connect Django up to Apache, we need to create a python file that will initialize WSGI and pass that information on to Django. In your {nameofwebapp} folder, create a sub folder named "apache", and then create a new file in that folder named "django.wsgi".

Paste this script into that file:

import os, sys
sys.path.append('C:\\Root_Django')
sys.path.append('C:\\Root_Django\\{nameofwebapp}')
os.environ['DJANGO_SETTINGS_MODULE'] = '{nameofwebapp}.settings'

# Remap stdout to err
sys.stdout = sys.stderr

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Be sure of course to change "{nameofwebapp}" to the actual name of your django application. This short script will take care of a few things...

• the sys.path.append portion needs to point to your actual Django root. Adjust that if it's not C:\Root_Django
• sys.stdout = sys.stderrd will redirect all "print" statements to the Apache error log. If you don't have any print statements in your code you can drop this, but I prefer to keep it "just in case"

This script will take care of initializing Django. All we need to do is connect up the requests to our apache virtual host to this script. Edit httpd.confi, and inside of the virtual host for your "www" host, add these lines:

WSGIScriptAlias / C:\Root_Django\{nameofwebapp}\apache\django.wsgi

<Directory "C:\Root_Django\{nameofwebapp}">
Order deny,allow
Allow from all
</Directory>

This tells Apache that all requests coming in from / and up should be handled through the django.wsgi script. Save your httpd.conf file and restart Apache. You should see your Django app being served up now, minus the media (images, css, etc.).

Serving up Static Media
So you're serving python now, but your web app without images and css is pretty limp right?

Here's what we need to do:

• Configure an Apache virtual host to serve our static media
• Configure Django to know the correct place for our media
• Make sure we're also serving the grappelli media instead of the default admin media

Open up your httpd.conf file again, and find the Virtual host for your media.example.com (or whatever your static host name is). Since this will just be basic serving without WSGI, we simply need to configur the document root to the proper location...

DocumentRoot "C:\Root_Django\{nameofwebapp}\media"
<Directory "C:\Root_Django\{nameofwebapp}\media">
Order deny,allow
Allow from all
</Directory>

Tada! If you were to save and restart apache, then navigation to media.example.com/some/css/file.css it should work. What we still need is to serve up the grappelli media. To do this we'll add another alias so that media.example.com/admin will serve the grappelli content.

Alias /admin "C:\Root_Django\{nameofwebapp}\grappelli\media"

<Directory "C:\Root_Django\{nameofwebapp}\grappelli\media">
Order deny,allow
Allow from all
</Directory>

This will make sure that all requests to /admin on your media host will serve the grappelli media files. If you installed grappelli into your Python site-packages folder, be sure to set the path to that folder instead of the django root like shown above.

Configure Django With Media Prefixes
Lastly for this section, we need to do some configuration in Django.

In Part 1 we installed grappelli but we haven't configured it, so we'll do that now. Also, we're using a static media virtual host, so we need to make sure that Django is prefixing our media files correctly. All of this takes place in the settings.py file.

Configure grappelli
If you'd rather see the officiall grappelli installation instructions, they are available on the grappelli google code wiki.

First, add grappelli to the content processors:

TEMPLATE_CONTEXT_PROCESSORS = (
 "django.core.context_processors.auth",
 "django.core.context_processors.request",
    "grappelli.context_processors.admin_url",
)

Next, add the grappelli template directory:

import os
here = lambda x: os.path.join(os.path.abspath(os.path.dirname(__file__)), x)
TEMPLATE_DIRS = (
 # ...
 here('grappelli/templates/'),
)

Lastly, of course, add grappelli to your list of installed apps:

INSTALLED_APPS = (
 # ...
 'grappelli',
)

Configure Media Prefixes
We want to configure Django to automatically insert "http://media.example.com/" in front of all of our media files. The setting in the settings.py to do this is:

MEDIA_URL = 'http://media.example.com/'

But this could easily become problematic when we're testing django with runserver. A better result would be to conditionally test if we're in DEBUG mode, and to change the value based on that.

if DEBUG:
 MEDIA_URL = '/media/'
else:
 MEDIA_URL = 'http://media.example.com/'

We'll also want to make sure the admin media files from grappelli are being served. The code is almost identical, expect the setting is ADMIN_MEDIA_PREFIX:

if DEBUG:
 ADMIN_MEDIA_PREFIX = '/grappelli/media/'
else:
 ADMIN_MEDIA_PREFIX = 'http://media.example.com/admin/'

At this point Django should be running, and serving media files.

Still to come:

• Configuring HTTPS
• Forcing HTTPS for django admin
  

Django, Apache 2.2, mod_ssl, and mod_wsgi (Part 2)

Part 1 | Part 2 | Part 3

DNS
I'm going to make an assumption. You have 1 domain and can create hosts under that domain. Fr example, say you own example.com. I'm assuming you have the ability to create "media.example.com" with whomever hosts your DNS.

You'll want to setup 2 domains:
www.example.com
media.example.com

Since this process is virtually different for everyone I won't cover it. Contact your DNS provider for assistance here.

Setting up Apache
If you've never used Apache, you can pretty much control the entire webserver from 1 file: httpd.conf. By default it's located in {apache install direcotory}\conf\httpd.conf. I like to edit it in Wordpad becuase I'm a lightweight, but you're welcome to edit it in whatever hacker-rific editor you prefer.

Installing mod_wsgi
mod_wsgi is a module addon for Apache which is responsible for accepting the incoming HTTP connection, turning that connection into a python object, and then passing it to Django. In other words, it's really important.

• Copy the .so file you downloaded into {apache install directory}\modules.
• Open the httpd.conf file and find the section that has all the "LoadModule"s in it
• Add this line:
  LoadModule wsgi_module modules/mod_wsgi.so
  
• This will load the wgi module whenever Apache runs
  

Configuring Virtual Hosts
Let's recall what our objectives for this project are:

• Serve up Django with mod_wsgi by default from the site root
• Serve up static media (images, css, js, etc.) from a separate virtual host
• Offer the Django app in both HTTP and HTTPS

To achieve this, we'll need to setup at least 2 virtual hosts:

1. to hold the Django application
2. to hold all of our static media

We'll also need a way to configure Django to know where our static virtual host is.

In case you're wondering, there are a few reasons why this is an ideal setup:

1. We want our Django application to worry about 1 thing: serving up python
2. It would probably be a performance hit if python had to process all of our media files. It makes more sense to have them directly served by apache
3. We can enable caching for everything on our static media virtual host, also improving performance
4. It's uber leet cool (no, not really)

You can create virtual hosts in apache with the directive. Here's a sample of 2 virtual hosts:

NameVirtualHost *:80

<VirtualHost *:80>
ServerName www.example.com:80
ServerAdmin you@example.com
</virtualhost>

<VirtualHost *:80>
ServerName media.example.com:80
ServerAdmin you@example.com
</VirtualHost>

This will give us 2 domains to serve from.

Still to come....

• Configuring wsgi and django
• Configuring django settings to use our virtual hosts
• Adding https support
• Forcing /admin to use https only

Django, Apache 2.2, mod_ssl, and mod_wsgi (Part 1)

Part 1 | Part 2 | Part 3

It's funny how experience changes your perspective on things. 5 years ago I tried apache and thought it was too hard. "Why can't I just have a GUI to click the settings I want?!?" I would think in frustration. Now that I'm making the transition to Django, I thought I'd give Apache another try. Wow -- What a luxury it is to be able to just write a config file to do exactly what you want to do and not have to monkey around with where some stupid settings is in an extensive maze of windows and tabs. Hah... how times change indeed.

Anyways, here's a bit about setting up Django with Apache. Here's what will be achieved:

• Serve up Django with mod_wsgi by default from the site root
• Serve up static media (images, css, js, etc.) from a separate virtual host
• Offer the Django app in both HTTP and HTTPS
  
• Force the Django admin to always use HTTPS
• Use django-grappelli for a slicker looking Django admin, as well as some other goodies
• And lastly... do all this on a Windows Server (hah)
  

Installing Software
If you're following along at home, here's the list of things you'll need to download for a complete deployment:

• The latest Apache Web Server Win32 binary package (get the installer)
• The lastest mod_wsgi binary package (pre-compiled *.so)
• TortoiseSVN (to get Django and grappelli with)
• Python 2.6 Windows installer
  

Install all of the above, accepting the defaults (unless you otherwise know what you're doing). For mod_wsgi I'll cover installation in the next part, so hang onto it for now.

Now we'll need to do 2 SVN checkouts of code...

Installing the latest Django trunk from SVN
(note: this method will have the Django development trunk as your production copy of Django. If you don't like that, then just use the installer or the tar/zip from the Django site).

• In your python install directory (default: C:\Python26) open Lib and then site-packages
• Create a new folder named "django"
• Right-click on that folder and choose "SVN Checkout" (this will launch TortoiseSVN)
• Paste this URL into that window:
http://code.djangoproject.com/svn/django/trunk/django

You've now got a fresh copy of Django, but we need to configure Windows to be able to find Python and the Django tools.

• Right-click "My Computer" and choose Properties (this might be slightly different for Vista+)
• "Advanced" tab
• "Environment Variables" at the bottom
• In the "System variables" list, find "Path"
• Highlight it, and hit "Edit"
• Add this value to the end of it (change the path to your python path if it's not default):
  C:\Python26;C:\Python26\Lib\site-packages\django\bin;

Now you'll want to test to make sure this works.

Open a command prompt and type "python" and hit enter. Python should run. Ctrl+Z out of that, and type "django-admin.py help" and hit enter. It also should run.

Django and python are good to go.

Installing grappelli
This next part uses my preferences. You can do this different if you want, but keep in mind you'll need to adjust paths later on to account for your differences.

I like to install my django re-usable apps into my project directory. This adds them to the SVN trunk and I don't have to worry about if others in the office have the app installed on their local machine or not.

Anyways, open your Django project (the site you've made), and create a new folder called "grappelli". Right-click that folder and choose "SVN Checkout". Paste this URL and hit ok:

http://django-grappelli.googlecode.com/svn/trunk/grappelli

It should download a copy of grappelli. We'll look at adding it to Django in the next part.