MARC Tools & MARC::Record errors

I know next to nothing about MARC,though being a shambrarian I have to fight it sometimes. My knowledge is somewhat binary, absolutely nothing for most fields/subfields/tags but ‘fairly ok’ for the bits I’ve had to wrestle with.

[If you don’t know that MARC21 is an ageing bibliographic metadata standard, move on. This is not the blog post you’re looking for]

Recent encounters with MARC

  • Importing MARC files in to our Library System (Talis Capita Alto), mainly for our e-journals (so users can search our catalogue and find a link to a journal if we subscribe to it online). Many of the MARC records were of poor quality and often did not even state the item was (a) a journal (b) online. Additionally Alto will only import if there is a 001 field, even though the first thing it does is move the 001 field to the 035 field and create its own. To handle these I used a very simple script to run through the MARC file – using MARC::Record – to add an 001/006/007 where required.
  • Setting up sabre – a web catalogue which searches the records of both the University of Sussex and the University of Brighton – we need to pre-process the MARC records to add extra fields, in particular a field to tell the software (vufind) which organisation the record was from.

Record problems

One of the issues was that not all the records from the University of Brighton were present in sabre. Where were they going missing? Were they being exported from the Brighton system? copied to the sabre server ok? Being output through the perl scritp? lost during the vufind import process?
To answer these questions I needed to see what was in the MARC files, the problem is that MARC is a binary format so you can’t just fire up vi to investigate. The first tool of the trade is a quick script using MARC::Record to convert a MARC file to text file. But this wasn’t getting to the bottom of it. This lead me to a few PC tools that were of use.

PC Tools

MarcEdit : Probably the best known PC application. It allows you to convert a MARC file to text, and contains an editor as well as a host of other tools. A good swiss army knife.
MARCView : Originally from Systems Planning and now provided by OCLC, I had not come across MARCView until recently. It allows you to browse and search through a file containing MARC records. Though the browsing element does not work on larger files.
marcview
marcview

 

USEMARCON is the final utility. It comes with a GUI interface, both of which can be downloaded from The National Library of Finland. The British Library also have some information on it. Its main use is to convert MARC files from one type of MARC to another, something I haven’t looked in to, but the GUI provides a way to delve in to a set of MARC records.

Back to the problem…

So we were pre-processing MARC records from two Universities before importing them in to vufind using a Perl script which had been supplied by another University.

It turns out the script was crashing on certain records, all records after the problematic record were not being processed. It wasn’t just that script, any perl script using MARC::Record (and MARC::batch) would crash when it hit a certain point.

By writing a simple script that just printed out each record we could as least see what the record was immediately before the record causing it to crash (i.e. the last in the list of output). This is where the PC applications were useful. Once we know the record before the problematic record, we could find it using the PC viewers and then move to the next record.

The issue was certain characters (here in the 245 and 700 fields). I haven’t got to the bottom of what the exact issue is. There are two kinds of popular encodings: MARC-8 and records in UTF-8, and this can be designated in the Leader (9th character). I think Alto (via it’s marcgrabber tool) exports in MARC-8 but perhaps the characters in the record did not match the specified encoding.

The title (245) on the orignal catalogue looks like this:

One work around was to use a slightly hidden feature of MarcEdit to convert the file to UTF:

I was then able to run the records through the perl script, and import it in to vufind.

But clearly this was not a sustainable solution. Copying files to my PC and running MarcEdit was not something that would be easy to automate.

Back to MARC::Record

The error message produced looked something like this:

utf8 "xC4" does not map to Unicode at /usr/lib/perl/5.10/Encode.pm line 174

I didn’t find much help via Google, though did find a few mentions of this error related to working with MARC Records.

The issue was that the script loops through each record, the moment it tries to start a loop with a record it does not like it crashes, so there is no way to check for certain characters in the record as it will already be too late.

Unless we use something like exceptions. The closest to this perl has out-of-the-box is eval.

By putting the whole loop in to an eval, if it hits a problem the eval simply passed the flow down to the or do part of the code. But we want to continue processing the records, so this simply calls the eval again, until it reaches the end of the record. You can see a basic working example of this here.

So if you’ve having problems processing a file of MARC records using perl MARC::Record / MARC::batch try wrapping it in a eval. You’ll still loose the records it can not process but it wont stop in it’s tracks (and you can output an error log to record the record number of the records with errors).

Post-script

So, after pulling my hair out, I finally found a way to process a filewhich contains records which cause MARC::Record to crash. It had caused me much stress as I needed to get this working, and quickly, in an automated manner. As I said, the script had been passed to us by another University and it already did quite a few things so I was a little unwilling to rewrite using another language (though a good candidate would be php as the vufind script was written in that language and didn’t seem to have these problems).

But in writing this blog post, I was searching using Google to re-find the various sites and pages I had found when I encountered the problem. And in doing so I had found this: http://keeneworks.posterous.com/marcrecord-and-utf 

Yes. I had actually already resolved the issue, and blogged about it, back in early May. I had somehow – worryingly – completely forgotten any of this. Unbelievable! You can find a copy of a script based on that solution (which is a little similar to the one above) here.

So there you are, a few PC applications and a couple of solutions to perl/MARC issue.

Posted on by chriskeene 5

UK deficit and protest

The large peaceful protests on Saturday were a success. Clearly Cameron and Osborne (and the little work experience boy standing closely behind them) did not come out and announce an immediate reversal of policy. But the need to think about what these cuts are doing to jobs, living standards and essentially services hopefully will be strong in the mind. Few politicians, no matter how ideological, want to ignore the views of a decent proportion of the public. And while U-turns are unlikely, the knowledge the further cuts announcements will lead to even stronger anger will be hopefully reduced the gun-ho spirit that the Tories have gone about them so far.

It made me think quite a lot about where I stood on it all. From everything I have read, the UK is in a dire situation, by a number of metrics we are (and were) in a worse situation than Greece and Ireland. It is only that the worlds banks have not panicked over lending to us, which more of less was the catalyst of the final nail to requiring international intervention and aid, that has avoided it so far. To an extent this is partly because the UK is seen as a stronger and more resilient economy, but it also has to be said that Osborne’s strong (perhaps too strong) actions last year (especially during the budget) reassured the financial markets (no matter how right or wrong those measures were over all).

It’s worth remembering that there are two key issues: our total debt (how much we owe overall) and the annual deficit (how much our spending is more than our income). Economists tend to break up deficit in to two types: Cyclical and Structural. Cyclical: when we have an annual deficit because the economy has taken a turn for the worst, which means we have less tax money (because of less jobs, less spending, etc) and at the same time our running costs are higher due to increases in things like unemployment pay. Structural: where our spending outstrips our income in both the good and bad years, which is a much more serious and needs addressing. It seems to be the general consensus that the UK has a Structural Deficit.

But is there a problem? As I was interested in the goals of the Saturday march and had a look at their website. Clearly they were against cuts, but what were they advocating instead.

While looking I came across a page called ‘How big is the problem?’. This page does them a disservice as it focuses purely on debt (and debt as a % of GDP) not deficit. It’s my understanding that debt is not seen as such a big issues as deficit, you might have a lot of debt to pay off but it’s quite possible to pay this off over time, especially if a country has an annual surplus (i.e. opposite to a deficit) or small deficit. They point out that the UK has a smaller debt as a percentage of GDP compared to many other countries, including Japan and Germany (their point being our situation isn’t too bad). True, but ask a group of financial experts which countries they are concerned about and they will almost certainly mention the UK, but less likely to mention Japan and Germany.

Talking about the deficit may not help the anti-cuts cause, as it clearly shows the UK is in a worse situation to many (and hence action is needed) but by selectively choosing facts to make an argument does no good. Better to be upfront and explain that while action is needed, the current action is not the right way to go about it. (much of the rest of the site is very good and informative I should add, and the previous page on the site does talk about the Deficit).

So how does our deficit compare:

Screen shot 2011-03-27 at 14.41.44.png

[image source: http://www.guardian.co.uk/world/interactive/2011/mar/14/new-europe-statistics-interactive ]

So it’s a problem. More so for us than for others. What can we do? three options:

  1. Cut spending, as the Government currently is.
  2. Cut spending but over a longer period than currently planned
  3. no cuts

Along with these options we have other options that can be implemented in tandem to help the problem:

  • Increase taxes and other Government income
  • Decrease tax avoidance and loop holes
  • Expand and grow the economy

The Government is going for ‘option 1’ above, and to some extent is trying all the options in the second list for increasing revenue.

It would be wrong to say that everyone on the anti-cuts march were all of exactly the same views. But that is to be expected, the point was a broad dissatisfaction about the level, speed and direction of the cuts. It’s articulated quite well here.

I saw some were against any cuts at all. I think this is silly. The deficit needs addressing, and simply hoping that higher tax income and a growing economy will solve it all is wishful thinking. Especially as economies often grow faster due to tax cuts, so one revenue raising option puts the dampers on the other one.

I’m not normally in agreement with Ed Balls, but what he says here I broadly agree with.

My main criticism is the speed of the cuts, and where they have been targeted. For example hearing of community workers in deprived London suburbs being laid off and the funding for such work being cut is not good. They can be the only lifeline stopping people spending a life with no education or skills being unemployed, causing trouble and crime. They give people a chance of a much better life, and, to be blunt to avoid being a burden on the state.

Cuts are hard. For example, national quangos and organisations are easy targets, such as the BECTA, the MLA, and the Film Council. But they can often bring more savings (and growth) than they cost. It might be hard to quantity all the work the Film Council did, but if they managed to persuade large films to be filmed in the UK, perhaps by advocating to the Government to provide incentives and getting councils to allow filming at locations even when it will cause disruption, it can create a whole ecosystem of small companies which support the industry. My own area of work (Libraries) could well do with national provision of back room technical provision and licence negotiation, rather than each Library providing it in house (essentially duplicating the same job all across the UK). At the same time, there are national services/organisations I can think which I would miss if they were gone, but not THAT much.

Cuts are, again, hard. Think of the UK Government departments. Where should the cuts fall? Education, Health, International aid? I think not! Business, skills, transport? Essential for long term growth of the country. Social Services/Welfare? Already over stretched in many parts. Higher Education, the Arts? Already cut to the bone.

So where: Defence: we are the third biggest spender on defence in the world, I think we could be sixth or seventh and be ok, we’re not a super power. EU: While most sections of Government were facing cuts (like most countries), the EU budget contribution actually went up, I would have thought it would be a prime candidate for a decent cut (I’m not against the EU, but it’s not a front line service, and plenty of fat that could be trimmed). The Welfare system provides essential support to many who need it for a variety of reasons. The minority who abuse the system need to be cracked down on without affecting honest claimants.

I digress, this article is not here to provide a full list of where the cuts should be. The point is that while any sort of cut is a hard pill to swallow, the cuts currently sometimes feel like they are aimed at those who most need support. Raising HE fees is one (not good) thing, abolishing the organisation that works with young people who would not normally go to University from poorer backgrounds and would be best place to show that University is still an option (especially with the ‘only pay back once earning a certain amount’ and bursaries) is just cruel and badly thought-out.

In terms of speed, cutting too fast will slow the economy. While there are some who feel that every penny spent on public services is dead money, public organisations do spend that money on staff and many third party services (just look at council spending above £500 on most council websites). This all adds to the economy, and I’ve recently heard small businesses say that the cuts to public services have a direct knock-on affect to their business.

However, we can not forget the situation we are in, and we are already talking about four years, if we are not careful we will be entering the next downturn before we even get out of the current one. Furthermore, if the financial markets think we are not doing enough our credit rating could be downgraded, we could even need a bail out.

Finally, Tax. Of course tax avoidance should be reduced, loop holes cut, if someone lives or works in the UK, or a company does business in the UK, they should be contributing a fair amount to the Government. Having the right rules and enforcing it can be difficult in this globalised world, but we need a system that ensures everyone involved in the UK pays their bit. the 20% VAT hits us all, the downside it includes those who can least afford it, but the upside it’s almost impossible to dodge, if your living in the UK, you’re going to pay it.

Listening to the comments on the Telegraph website you’d think the 50% tax rate for those earning over £150,000 has lead to the end of the world. They claim it means anyone with a brain or a business will leave the country and there’s no reason to try and create wealth. This is crap. They pay a little extra tax on some of their earnings, I don’t believe for a second that people decide not to start a business or take a high paid job as a result. It’s not about penalising the rich, it is about us as a whole needing to pay for the essential services for our society and it is right to ask people who can afford it to pay a little more. Suggesting that a person on a low wage, with kids, a mortgage and very little cash, should pay more tax so that high earners don’t have too is frankly shocking.

So that’s me. I guess I broadly agree with the aims of the march on Saturday. Not all perhaps, but most. And the Tories, for all their many (many) faults are tackling a serious problem. Taking measures to try and kick start the economy with lower corporation tax and other incentives is a right move, but one can not help think that in the short run, much of the money going in to the public sector was finding it’s way to UK businesses before it was cut.

Let’s hope we can get out of this rut. History suggests that will come, but how quickly. And let’s hope whoever is in Government when that comes will ensure we continue to offer good education, health and support to those who need it, while making this a country where small businesses and new ideas can flourish.

Posted on by chriskeene 1

Broadband speed test

Utterly unscientific, but who ever you are, please spend a few minutes filling out this form about broadband speed.

You can choose which broadband speed test site you use, and if you have time perhaps repeat the test with a different site (with enough data we can spot if certain test sites report higher/lower speeds compared to others).

You can see the results of the test here: http://snipr.com/ud0h0

No specific reason for doing this, but interesting to see variations in speed based on location/provider/timing, etc.

Anyway, before you forget, pop along to this brief form and add your speed results. It’s anonymous, and you can be as vague as you want about your location.

Thanks

Posted on by chriskeene

VuFind in 8 minutes using Amazon EC2

I’ve created a screencast showing how easy it can be to install VuFind. Here I go from nothing (no server, no OS) to full VuFind install in under 8 minutes.

It would probably take less than two minutes (under 10mins in total) to add MARC records to the installation, but I didn’t have any to hand at the time.

This demo cheats a bit by using a script that does the heavy work, the script was a mash up I created taking existing scripts and commands that come with VuFind with a few tweaks. It probably would have been only slightly slow to run most commands manually.

The script in question is at http://www.nostuff.org/vufind-install.txt and of course anyone is free to download (and improve, please share changes). There’s lot of potential to improve it’s ability to work on different flavours of Linux.

Multi Instance

One of the key aspects of the script is that is allows you to easily install multiple instances of VuFind on to a server. By default VuFind installs in to /usr/local/vufind and has other things (databases, apache conf) names vufind. The script prompts for an ‘instance name’ and then uses that in place of ‘vufind’.

The rational for this is my feeling that VuFind is an excellent tool for creating niche catalogues that are a subset of the full Library collection (or as someone put it a ‘tranche of the catalogue’). A branch Library, particular collection, rare books, a particular School, Core reading (short loan books), specialist resources (AV / laptop items for loan) etc. The idea of a organisation’s records being in system, rather than many (of varying quality) makes sense, but it’s reasonable for those moving their records to a central system to want to be able to search their records independently of other records (and expecting users to go to an Advanced search of using a refine option of the main catalogue is not really not an option). VuFind seems like an obvious answer to this. Especially if new instances can be set up quickly.

In fact it seems to be a failing of most of the main Library Manage Systems (ILS) and their web interfaces that being able to create lots of interfaces (slicing and dicing the underlying single large pool of records). Most require a large amount of effort to create a second web interface to a catalogue. This seems like such an obvious flaw and a barrier to moving to one system to manage resources such as books and other items for an entire organisation.

Amazon EC2

Amazon AWS is a great tool to use for this. A small instance will cost around $0.10 an hour, the micro instance is even cheaper (just over $0.02). Create an instance for ten hours and you have spent around a dollar. Mess it up, just destroy it and create a new one. No risk and no hassle (for first time users the hardest thing is probably the ssh private keys).

Posted on by chriskeene

Pinboard.in suites me just just fine

About a month a go I wrote a quick rant about Delicious.com and why it could have been a sucessful web2.0 business (relatively low costs and lots of opportunity for advertising and pro/paid for features). At that time I said I was moving to pinboard.in, especially as delicious.com’s future was being questioned at the time.

This is how I use/used the two sites

  • When trying to find a site, page or link – often for a product/app/site with a name I don’t remember – I will use Pinboard’s (and used Delicious’) search feature with a keyword to try and find it, hoping that I either tagged it using the word, or title/description included it. This is my primary use case.
  • My second use case was picking a tag and browsing by it for the same reason.
  • Third use case was having no idea what I can search for, so just browsing through my entire bookmark list, perhaps picking roughly the time period I might have saved it.

I’m not looking back. I like:

  • It’s fast, clean and simple.
  • It has the features I use from Delicious and not those I was not interested in.
  • I never used the Delicious, nor can I think if a good reason to have them. Pinboard does not have them by design.
  • Automatically adding favourite tweets is a brilliant idea, I, like others, use the favourite tweet feature more as a ‘read later’ option for interesting links.
  • I use pinboard (and used delicious) as a cross browser, access anywhere, bookmarking system. No browser plugin or sync feature comes close.
  • When I searched delicious it would show three of my bookmarks and the the rest of the page would show bookmarks with the search term saved by others. This was useless to me, and just created an extra click for me to select ‘show me all bookmarks that match this search. I have one less click with pinboard which does not show me other people’s bookmarks by default.
  • I quite like the pay a little to use it concept. Even with freemium sites (which have a free option, normally the most popular, and a pro version) you are ultimately paying for both you and the cost of those using the free version. There are often anti-social users, who devalue the site and add to the cost of running it. Not so with pinboard, if you can’t be arsed to spend a few quid (or dollars) on the service then, no offense, go elsewhere. I like the fact that anyone using the site has shown a small monetary commitment to it. It makes it not another site where the whole world signs up and then forgets what it is (god knows I get enough emails a month from web2.0 services keeping me updated about their service which I have no idea what they do or why I signed up).
  • However I do find the concept of the signup cost going up with each new user somewhat strange. It might encourage early adoption but it will mean the site eventually becomes too expensive that many will choose not to sign up.
  • Likewise it’s a one off cost for a service which will have ongoing costs. So new signups will have to fund the service. Perhaps move to a $4/yr model (perhaps with multiyear discounts $10 for three years) to provide a consistent and ongoing income.

In summary pinboard keeps things simple, fast and is created by committed developers. Suites me just fine.

Posted on by chriskeene

hifi

I’ve got the other place (here or here) for random ill-thought musings. But today I decided to put one here. Hi-Fi

< vaguely interesting background story with slight element of personal touch>

I left University in 1999. Having been in full time work before I had even finished my last term, and living in cheap shared accommodation, I could splash out. But being somewhat conservative (small ‘c’, you see that? SMALL ‘C’. I mean ‘c’. Just want us to be clear on this ok) I waited a year, before buying the hi-fi I had seen and desperatly wanted for a massive £280.

Now, should you have been looking at purchasing a stereo/hi-fi (what do we call them nowadays?) around this time, which I was, clearly, then you will be aware at just how vile they all were. Bulging like their biceps were about to explode (pendants will argue about their lack of biceps). The one pictured below is quite a modest example, laziness stopped my from finding a more accurate example.

The issue was not was not that these existed, I can understand there is a market for them, just like there is one for JD Sport. But that they dominated the market so. Walk in to a Comet or Currys and aisle after aisle was full of them. If they had 30 models on display then only one would be purchasable by sane persons.

[an aside: it looks like with the demise of (a) vinyl (b) tapes (c) club culture that design has moved on and most hifi’s on sale today have completely different dimensions. A good example is this from Onkyo, which is excellent and recommended]

But I had found one that was above all this, minimalistic in design, always good in my book, and stunning on the glass and wood stand it was displayed on where I first saw it (which was Dixons, yes I know). It was the Pioneer NS-9.

It consisted of one small unit, which was basically the hifi, a separate display, two small speakers and a woofer. It looked and sounded great. The front of the display – with all the buttons – could actually come off and act as a remote but this was more or less unworkable. The UI was awful, trying to set sleep mode or retune a preset radio setting still requires the manual.

The two small speakers and a bass combination was unusual and worked well. My flat mates and near neighbours through the years will attest to the kick arse bass it produced (literally as I type this, Open Up from Leftfield is playing).

It was the first, and still inexcusably very rare, system with FM/RDS I saw that showed not just radio station name but extended information, such as a the show, DJ, and maybe even the song that was playing, depending what the station sent out. This was way before DAB radio, and at the time at most you would normally see was the station name. I must have been one of the few reading the text that was being transmitted. I remember the Radio 1 Top 40 would show the song name and artist (and position in the chart)  which sometimes was displayed before the song was announced on air. It was if I was tapping in to some secret message no one else had access to.

Anyway, apart from the woofer, separate display and extra programme information, it was just a hifi, I still can’t quite believe I’m writing a post about it.

<finally getting around to the point of the post>

But I’m not writing about it.

For years I’ve been glancing an eye at a Hi-Fi separates system. Now isn’t the time for me to be buying one having just bought a property, but I’ve been keeping a look out for something that can replace my stereo. The CD player isn’t what it once was, and a few other niggles remain.

It started when I was a kid. I had a Matsui (Dixon’s home brand, made in Wales I think, but with a reassuring foreign name) cheap hifi. Why would anyone pay anymore than this? I can turn it up and the treble sounds high and the bass is low. It all sounds clear. What more could you want? One day, on a “I need to get out the house on a Saturday but I don’t know what to do so I’ll go to HMV and look at all the albums I could buy, again” I was in, well, HMV and Blue Monday came on. This was a time when HMV was a music shop that sold a few movies in the corner. Their sound system was amazing. I didn’t know what they all did but the black boxes when from floor to ceiling.

The song blew me away. Consumed me. Took me over. And I had it at home! I walked back to my parents house and put it straight on. It didn’t give me the same feeling. That was when I realised what a good sound system could do. And though the sound my system made sounded ‘ok’ it just didn’t have that magic. It sounded flat.

But how often do I use the CD player now? Not enough. Spotify more than anything dominates now. This isn’t always healthy. I can never settle on just playing an album, and spend forever playing favourite tracks. Anything that doesn’t grab my attention is skipped. And I know this is bad.

With audiophiles preaching about the importance of cable, good source, good amp, etc, this all goes to pot when your source is a live streaming free Spotify (subscribers get higher bit rate) going through your Macbook Pro headphone socket. So here’s the question. In this age of Spotify and similar, is the hi-fi stack still valid? I use my Hi-fi more for radio then I do CDs, yet the radio turner is often seen as an afterthought. And what about all in one compact systems, from good names such as Onyko, Cambridge Audio, Denon, Marantz, are separates really that much better than these?

One issue was the tuner, CD players and Amps start at around £100 (though it’s made clear these are bottom of the range). The cheapest tuner is £150, and that has bad reviews, and can’t do DAB+ (DAB+ is a major improvement over DAB, not man realise that the iplayer has a higher quality stream than DAB, with the possible exception of Radio 3). This is an odd situation. You can buy a decent all-in-one unit for say £200, yet a semi-decent turner separate costs about the same, but with probably less features. With the tuner on top of the CD and Amp you’re look at £350 for the most basic of basic systems, extra for the speakers. Is it worth it? How much of that expense goes on the physically boxes and extra overheads of a separates system? The reviews all make somewhat patronising comments ‘excellent sound – for a all-in-one system’, ‘good if you need a basic set in another room’ etc. But how much is snobbishness and how much is fact. Is it worth it?

So I can’t afford a new system. And not even sure I need one based on most of my listening is of Spotify, iplayer and radio 4. But I can’t decide that if I were buying a new system, which I’m not, whether I would buy an all-in-one system (such as this or this) or separates.

I’ll continue to debate what I wont be buying, but might buy if I could afford it, and update you if need be. But I still am not sure such things are needed especially as we move towards the internet/computer being the key source of music, and the main output of which is currently a crappy tiny headphone socket. If we want to hear good quality music via our computer a different route my need to be developed.

Posted on by chriskeene 3

Personal URL shortener

Back in 2009 I wrote a post about creating my own url shortener. I didn’t get very far, partly due to the challenge of how to create the short url codes, and partly due to my development philosophy of having an idea, musing about it, and never getting anywhere with it. Though I did install some code Ben Charlton has written and set it up as u.nostuff.org.

Part of the drive to do this was a bizarre feeling of guilt when using ‘short urls’ from the main shortener services. Each short url I created would mean we – as a userbase – were one step closer to the URLs being one digit longer (i.e. when that service ran out of free letter/number combinations with the current number of digits). Short urls were a scarce good. Did I really need this short URL? Worse, useful services such as twitterfeed use a short url each time they add something to twitter, even if most of those links would never be followed.

However there should be no need for all of us to be using the same small set of url shorteners, if we used our own, or one for a particular group of people, or for a particular service, then we would be free to use them as we pleased. What we (internet users at large, and especially twitter users) needed was lots of URL shorteners, each able to produce short codes.

Back in April this year I came across YOURLS, an open source simple URL shortener you can install on to your site [main website | blog with new releases | Google code]. This fitted the bill exactly. Installing it was easy on my dreamhost account.

Now I just needed the short domain name. I came up with a few catchy/clever domain names, but quickly found that two digit domains were pricey or just not allowed for purchasing. especially those such as .im, ly etc.

It would have to be three digits, and if I couldn’t get or afford ‘catchy’ why not just go for something completely random? After all I just needed to remember it.

I also decided that I would buy from my existing domain name provider, rather than signing up with another organisation, especailly some of the dodgy looking ones providing sole access to unusual country top level domain names.. I also wanted to use a country that was stable and unlikely to be awkward with renewing the domain or changing the rules (sn.im didn’t work for a while while the owners argued with the Isle of Man registrar). Using 123reg, at the time the cheapest three digit domains were to be found  (amongst a couple of others) in Belgium, .be. The only concern being as the two halves of Belgium seem to drift further apart from each other, there is even talk of one day Belgium splitting up (and I bet they wouldn’t even think if my domain name during the split, well I really).

So I ended up with xd5.be. No good reason. I used a combination with an x as I figured fewer organisations will have an acronym with an x in it, the number in the name may help in that respect as well.

The solution works well, with the usual browser bookmarklet (I’ve always shortened urls before pasting them to a twitter client). One thing I did find was that I was reluctant to use it at first as I was aware I was creating the shortest urls available on this domain (i.e. one digit code) and felt the need to preserve them for stuff that was important. Now I am on to two digits I am more care free! YOURLS allows you to choose between just using lowercase or including uppercase digits as well, I went for the former while the latter will obviously provide far more combinations of characters. I could also install additional copies of the software on to subdomains, e.g. a.xd5.be b.xd5.be, these are still seven characters in total, four less than tinyurl.com. drop me a line if you would like one of these.

In all, I think it is worth doing, it keeps you in control of your links and the software you are using.

Posted on by chriskeene 1

Jisc bid writing

Today I submitted a JISC bid on behalf of a team, as part of the recent JISC call Infrastructure for Education and Research (’15/10′ to its friends). The call was actually a set of broadly related different strands, we submitted (with a whole 30 mins to spare) under a strand called Infrastructure for Resource Discovery, and there’s a nice web based version of the call on Jiscpress.

Jiscpress was created in part by Joss Winn and a post of his I saw today inspired me to knock out this this rambling.  Go read it before you read this, thanks.

I admire his openness and I should strive to do the same. Funny that I try – and to an extent automatically do – make much of what I do open, but with this sort of thing there is a tendency to keep it close to your chest. There were very few tweets in the run up to the bid. Why are we not more open? He also talks about his JISC bid writing and tips, here’s mine.

My first experience was attending a ‘town hall meeting’ in Birmingham about the JISC Capital programme, around 2006. For a starts I didn’t even know what a Town hall meeting was (I think it means a briefing day, everyone presumed you should know this). I do remember it felt daunting, there were lots of people in suits. Lots and lots of sentences I didn’t understand (We’re going to innovate to take vertical solutions and silos and break them in to a horizontal landscape to help foster a new wave of disruption to help inject in to the information environment testbed) and no one I knew. I looked at the massive documents that accompanied the programme, many of them, many times. And looked at what I needed to do to write a bid. Budgets, signatures, partners, matched funding. I didn’t submit one.

Since then the community has developed, in no small part thanks to Twitter, but also to things like mashlib and many one day events (which either never used to exist in the field that I work, or I was just more ignorant then than I am now). Beer has been a big part of forging links in the HE Library / tech community. Seriously. It really needs its own cost code.

I looked at a number of potential calls over the last few years – often they required a report or development that I had no really knowledge in, I almost came close to putting something in for the JISC rapid innovation call (and helped mark it). When the JISC LMS call came out about a year a go the time and subject were right to submit a bid. I knew the subject matter, I had a natural interest and passion, and I knew the people who would be involved in these sorts of things.

These are tips for people who are thinking of putting in a bid, especially those who are stupidly disorganised like me:

  • Time between a call being released and the submission deadline is short, normally about a month, which in HE terms is not long. Use the JISC funding timeline to get a heads up of future funding opportunities so that you can prepare for working on a bid (including blocking off time during a month, and arranging initial meetings with others) before it comes out and not taken by surprise. The JISC 15/10 call had a blog post a few weeks before the call came out giving a feel for the call and confirming the date it should be released. It helped me to start thinking about ideas and block out time to read it (even if some of that time was in the evening) on the day it was released.
  • Every organisation is different (that applies to everything I say) but for us, setting up a meeting a couple of days after the call was out was very useful. It included those who it could affect and relevant senior staff. The call had lots of areas which matched our goals (and some, not always the same, that matched my personal interests), it was good to prepare a briefing and then bounce those ideas around to see what had potential and see what other ideas came up. It helps in many ways, to quickly focus and refine potential ideas (and drop those that people show no interest in), keep everyone in the loop and see whose willing to work on it. It stops it being one person or departments little side project.
  • The briefing day was very useful, especially for talking to people, finding potential partners and getting great advice.
  • Now I have an incredible amount of bad points, but two of them are leaving everything to the last minute and working in a very linear fashion. Often things that feel like they are the last things you need to do are actually things you need to set in motion earlier on. This seems so simple typing it now but I’ll probably (be warned colleagues) do the same next time. These include budget, project name and supporting letters.
  • The budget is hard. See if your org offers support in doing this. The problem is certain magic numbers (the wonders of trac and fec) can only be calculated once you know all your other costs. However I tend to find that near the end of the bid writing process you suddenly think of some work a particular group/person/org will need to do so you need to factor in those hours and costs, or you invite someone from the other end of the country to be on a panel and need to cater for their travel and hotel costs. In best ‘do as I say don’t do as I do’ tradition I would try and bash this out well in advance so it can be sent to those who can then check it over and fill in the magic numbers.
  • Asking a friend at another Uni if they don’t mind asking their (P)VC to drop everything so that he/she can write a nice supporting letter for your project is hard. So try and avoid it by getting it done sooner. Again often easier said than done as projects tend to evolve during the bid writing process which can make letters reflect out of date ideas or stress the wrong areas.
  • Letters and other stuff need a project name. I’m guilty of really not thinking a name is that important. The acronym will be meaningless to all. On my first bid I just used a working name (all of 5 seconds thought) and right at the end asked everyone if they are happy to go with it. Mistake. Changing project name at the last minute is a pain.
  • A key point. You need a good idea. And a good idea is one that is a good fit to the call. You may have a perfect methodology but if the idea doesn’t fit with the call then you could be in trouble. I’m guessing ‘It’s not really what you’re after but it’s such a good idea you must want to fund it’ is not a good sign.
  • Speak to people, I mentioned the briefing day above, but also speak to the programme manager, they’re nice people! Talk about it on twitter.
  • You don’t need to be an expert. I was put off for years from writing a bid about things I was interested in but didn’t think I knew enough about. You can ask people to work with you! People who know how to do stuff. I’ve just submitted a bid about Linked Data. Now I’ve followed the rise of Linked Data for years and tried to learn about it, but taking an XML file and ‘converting’ (is that the right term?) in to Linked Data, I had no idea how to even start. But I spoke to some people, who recommended someone, and they do know what to do
  • Approaching others out the blue is difficult, especially if you don’t feel ‘part of it’. All I can say is ask. And if you don’t know who to approach ask people (either at the JISC or via twitter) for advice.
  • If you have a clear(ish) idea of what you are going to do, broken down in to mini packages of work, andwho is going to do each one of them, then writing the actually bid is easy. Treat it like a job application. We all know that when writing a job app use the Person spec as a structure, a paragraph for each entry of the person spec, perhaps even with headings to help those marking your application. A bid is just the same, the clearly laid out structure of a bid is worth sticking to, it’s the same thing the markers will have to use to score each section. If the JISC Call document ‘proposal outline’ refers to a section which talks about Project Management, leadership, copyright, evaluation and sustainability. Then write about those things together as clearly as possible. Long winded paragraphs which ramble on about everything and make subtle implied passing references are a bain to the marker and no help to you.
  • But, I have been involved in marking and assessing bids, what impressed me was the impartial way bids were judged, and the real desire of wanting to fund Good Ideas, even if the actually bid document needs a little clarity in some parts. Especially from first bidders. To stress, there was a real desire to see first time bidders (with a good idea) be successful.
  • So the actually bid write up can in a way be left later than other tasks mentioned above, as it mainly involves just you, (and probably a couple of people who work closely with you to check it).
  • In an ideal world this would all be done weeks before it needed submission. In real life other factors (and work) can mean it can be a last minute dash. That’s fine. But make sure a few days before it has to be submitted you put in to an email to yourself (and others involved in writing it up): The email address to send it to, the submission date time, cut and paste things such as the exact format it needs to be submitted in (how many fiels, how big), number of pages. Add a link below each of these facts to the actually source of information (jiscpress is excellent for this), so when you’re panicing and presume everything you know is wrong you can follow a link and see for yourself that it really is eight pages max for the proposal, direct from the horses mouth.

The whole process of submitting a bid, and running a project, is good experience. It often involves working with people you would not normally, and doing differently to your normal job. Now if I get a chance in the next few days I will follow Joss’ example and blog about our proposal.

Posted on by chriskeene

Top posts

For reasons that escape me, here are nostuff.org/word’s top hits

Top Posts for 90 days ending 2010-10-05 (Summarized)

Summarize: 7 Days 30 Days Quarter Year All Time

2010-07-07 to Today

Title Views
Top 20 UK Universities 905 More stats
Fun with Lisa 797 More stats
Home page 359 More stats
Top UK Universities : Combined Rankings 310 More stats
Summon @ Huddersfield 285 More stats
Library search/discovery apps : intro 181 More stats
University league tables combined data 102 More stats
JISC Library Management System Review 91 More stats
webpad : a web based text editor 80 More stats
SQL update: doing a find/replace on part of a field 80 More stats
Adverts that follow you 71 More stats
Library catalogues, search systems and data 66 More stats
short urls, perl and base64 62 More stats
Library Catalogues need to cater for light-weight discovery clients 55 More stats
Talis Aspire, checking if a course has a list 52 More stats
Look! I can post links straight from B3ta 43 More stats
Google Books API 40 More stats
Zoho and WordPress themes 36 More stats
Katie Price and Peter Andre : A whole new world (reviews) 34 More stats
Twitter clients 32 More stats
BBC Three 27 More stats
IMAP5 : or developments that should happen in email 25 More stats
Amazon AWS EC2 and vufind 24 More stats
Nick Clegg’s Fault. Beware the REAL Nasty. 18 More stats
PubSubHubbub instant RSS and Atom 17 More stats
blip.fm 17 More stats
VAT ‘offset’? No just a tax rise via the backdoor 15 More stats
to do list software 14 More stats
html5 and nostuff 13 More stats
SQL grouping mulitple values in to one SELECT field 13 More stats
Free e-books online via University of Pittsburgh Press 12 More stats
Linked data & RDF : draft notes for comment 11

2010-07-07 to Today

Referrer Views
tweetalondoncab.co.uk/ALittleAboutTwi… 20
library.hud.ac.uk/blogs/summon4hn/?p=… 18
daveyp.com/blog/archives/310 17
google.co.in/ 14
photoshopdisasters.blogspot.com/2010/… 14
google.co.uk/ 13
library.hud.ac.uk/blogs/summon4hn/?p=… 12
twitter.com/ 8
library.hud.ac.uk/blogs/summon4hn/ 7
library20.org/profiles/blogs/librarie… 7
commonplace.net/2009/06/linked-data-f… 6
philbradley.typepad.com/phil_bradleys… 5
nostuff.org/ 5

And search terms

Today

Search Views
example of summon opac 3
integrated search engine “metalib” 2
primo encore summon ebsco 1
zoho is bad for business 1
chmod 400 and amazon 1
british library opac began 1
perl short base64 encode 1
top 20 universities of uk 1
impressions of joomla 1
top 20 uk universitys 1

Yesterday

Search Views
top 10 marketing universities in the uk 2
sql update portion of field 2
summon discovery primo 1
top universities in uk for marketing 1
top 10 u.k universities for masters in r 1
good universities for marketing 1
wikepedia ranking of universities offeri 1
imap5 1
warwick not in thes university rankings 1
base 64 shorten 1
Posted on by chriskeene