I started working on CSProjects quite a few months ago. Problems started early on. I began by bringing our software up-to-date. This included Apache, Python, Subversion, Trac and mod_python. It took some time, but I didn’t experience any problems… until I tried to run them. Seemingly at random, but quite frequently, the Apache children would get a Bus Error. I …
Thursday 22 February. That’s the day it all went wrong. I was on my way home from a shopping trip at Sainsburys. We’d been on a Thursday instead of a Friday because we had to go to Cornwall on the Friday afternoon for a funeral (that’s bad thing number one). Just after leaving Sainsburys we came across an area that was …
Back in the middle of last year I wrote about our plans to tackle our NFS performance issues by introducing a direct and dedicated network link to carry our NFS traffic between the clients and the servers. We’d done the tests so we just had to implement it. First we waited for the financial year (1 August) to roll over …
Back in May I wrote about the performance problems we were having with our new NFS based user filestore. It’s been a while since then, and the problems have continued. We have noticed that it appears to be load related – not just the network, but also the machine. This suggests that our theories about IPsec causing the slow down …
This week we’ve finally seen the end of some things I’ve been trying to sort out for some time now. The old storage arrays (Sun T3s and A1000s) are finally gone. The T3 arrays in particular have caused us endless grief over the past few years, so I’m more than happy to see them go. It also marks the end …
We’ve recently moved to having our filestore NFS exported from a cluster. This provides almost complete resilience from hardware failures, and moves us away from depending on individual end-user systems with locally attached filestore. Given the inherent insecurities with NFS we opted to use IPsec authentication (but not encryption) between the hosts involved. The NFS server only accepts connections from …
I’d finally finished migrating everything off the old myrtle disk arrays, so I was feeling quite pleased. I’d just unplugged the last array from myrtle and plugged it in to the test machine for wiping. Then I tried to log in to the machine room SunRay, but strangely it didn’t work. I checked the console logs for myrtle and was surprised …
The Computer Science department got it’s new website (more of a re-skin, actually) yesterday, so I decided it was about time to update my staff page. I’ve brought it in-line with the new look of the main website, although it doesn’t completely follow the standard templates. It’s also XHTML 1.1 valid, which makes the pedantic side of me feel much better. There’s …
A few days ago one of our users reported that they couldn’t change their password. The error coming out of the passwd command was confusing in itself – it said ‘bad old password’, or similar, which turns out to be a bug in our wrapper script. After some investigation we discovered that neither kadmin or kpasswd worked: tdb [~] % …
So after copying everyone off the limping T3 arrays I arranged for a Sun engineer to return to site to fix it properly. Sun Dispatch had a bit of a moan because I’d had the parts for too long, but they realised it’d make most sense to keep the parts on site rather than collect them and then send them …