Libraries, Library Technologies &Amp; Open Data on nostuff.org archive | Chris Keene

A play with the Talis Aspire API

Tue, 10 Jul 2012 19:35:39 +0000

I wanted to have a go with the Talis Aspire API and see what potential it had and how easy it was to use. Talis Aspire is a Reading List system with the key purposes being, firstly, a method to inform libraries which books and resources they need to buy, secondly, a unified way to access lists and content, especially as more and more stuff is online.

But the array of systems across a University delivering the online ‘student experience’ is staggering and often confusing, hence the need to to investigate ways of embedding data from one system (such as this) in to other systems. My end goal is to display a list on a webpage outside of the Aspire website.

Talis

Mon, 09 Jul 2012 20:02:18 +0000

Today Talis announced they are moving their focus away from Linked Data. I think my initial reaction, which I tweeted, holds true: this is on similar lines to Microsoft announcing plans to move away from Windows.

Earlier this year I was at an event where I bumped in to two people from Talis. The one who knew me introduced me to the other, I think he described me as ‘a long time Talis watcher’, whether or not I quoted him accurately, that statement is probably true, and I therefore want to muse a little. Forgive me (especially if you are involved).

MARC Tools & MARC::Record errors

Fri, 15 Jul 2011 12:17:59 +0000

I know next to nothing about MARC,though being a shambrarian I have to fight it sometimes. My knowledge is somewhat binary, absolutely nothing for most fields/subfields/tags but ‘fairly ok’ for the bits I’ve had to wrestle with.

[If you don’t know that MARC21 is an ageing bibliographic metadata standard, move on. This is not the blog post you’re looking for]

Recent encounters with MARC

Importing MARC files in to our Library System (~~Talis~~ Capita Alto), mainly for our e-journals (so users can search our catalogue and find a link to a journal if we subscribe to it online). Many of the MARC records were of poor quality and often did not even state the item was (a) a journal (b) online. Additionally Alto will only import if there is a 001 field, even though the first thing it does is move the 001 field to the 035 field and create its own. To handle these I used a very simple script to run through the MARC file – using MARC::Record – to add an 001/006/007 where required.
Setting up sabre – a web catalogue which searches the records of both the University of Sussex and the University of Brighton – we need to pre-process the MARC records to add extra fields, in particular a field to tell the software (vufind) which organisation the record was from.

Record problems

One of the issues was that not all the records from the University of Brighton were present in sabre. Where were they going missing? Were they being exported from the Brighton system? copied to the sabre server ok? Being output through the perl scritp? lost during the vufind import process?

To answer these questions I needed to see what was in the MARC files, the problem is that MARC is a binary format so you can’t just fire up vi to investigate. The first tool of the trade is a quick script using MARC::Record to convert a MARC file to text file. But this wasn’t getting to the bottom of it. This lead me to a few PC tools that were of use.

PC Tools

MarcEdit : Probably the best known PC application. It allows you to convert a MARC file to text, and contains an editor as well as a host of other tools. A good swiss army knife.

MARCView : Originally from Systems Planning and now provided by OCLC, I had not come across MARCView until recently. It allows you to browse and search through a file containing MARC records. Though the browsing element does not work on larger files.

VuFind in 8 minutes using Amazon EC2

Wed, 26 Jan 2011 13:02:24 +0000

I’ve created a screencast showing how easy it can be to install VuFind. Here I go from nothing (no server, no OS) to full VuFind install in under 8 minutes.

It would probably take less than two minutes (under 10mins in total) to add MARC records to the installation, but I didn’t have any to hand at the time.

This demo cheats a bit by using a script that does the heavy work, the script was a mash up I created taking existing scripts and commands that come with VuFind with a few tweaks. It probably would have been only slightly slow to run most commands manually.

Jisc bid writing

Mon, 15 Nov 2010 23:18:33 +0000

Today I submitted a JISC bid on behalf of a team, as part of the recent JISC call Infrastructure for Education and Research (’15/10′ to its friends). The call was actually a set of broadly related different strands, we submitted (with a whole 30 mins to spare) under a strand called Infrastructure for Resource Discovery, and there’s a nice web based version of the call on Jiscpress.

Jiscpress was created in part by Joss Winn and a post of his I saw today inspired me to knock out this this rambling. Go read it before you read this, thanks.

Talis Aspire, checking if a course has a list

Tue, 31 Aug 2010 14:26:18 +0000

Talis Aspire is a new-ish Reading List system used at the University of Sussex Library.

On Aspire, a url for a Department looks like this:
http://liblists.sussex.ac.uk/departments/anthropology.html

A page for a course looks like this (for course l6061):
http://liblists.sussex.ac.uk/courses/l6061.html

The nice thing is that you can replace the ‘.html’ with .json or .rdf – while the html only has a link to the related list, the json and rdf expose other links and relationships, such as the department.

Library Catalogues need to cater for light-weight discovery clients

Wed, 25 Aug 2010 13:11:26 +0000

Way back I wrote a piece about the changing model for library catalogues, you can see it here. The main premise was that trying to maintain records in a Library Management System (LMS/ILS) for all the items you want your users to discover is no longer feasible. This is especially true in this here digital age, trying to maintain records for all the e-journals a University has access to is an almost impossible task, and LMS were not designed for thousands of MARC records to be dropped and then re-imported (i.e. sync’d) with another source. And what about all the free stuff, is an e-book not worth being discovered by users because it is free?

ircount : update

Sun, 30 May 2010 17:45:42 +0000

One Sunday morning in January this year I got an email sent automatically from the webhosting company. It contained the output of the script that ran weekly, when all ran fine the script produced no output. When something went wrong the error messages were emailed to me. Judging by the length of the email something big had gone wrong.

The script collected data from http://roar.eprints.org/ – to be used as this weeks ‘number of records’ for each repository.

Summon @ Huddersfield

Tue, 25 May 2010 21:38:05 +0000

I attended an event at Huddersfield looking at their and Northumberland’s experience of Summon. http://library.hud.ac.uk/blogs/summon4hn/?p=22 These are my rough notes. Take all with a pinch of salt.

The day reaffirmed my view of Summon, it is ground breaking in the Library market, and with no major stumbling blocks. They are very aware that coverage is key and seem to be adding items and publishers. It searches items that a organisation has access to (though users can tick a option to search all items in the kb, not just those they can access). They have good metadata, merging records from a number of sources, and making use of subject headings (to refine or exclude from the search).

Dev8D

Sun, 28 Feb 2010 20:07:22 +0000

This isn’t a comprehensive review of dev8d, checkout the twitter stream (that link uses twapperkeeper by the way, the guy who created it was there and it was great talking to him), and wiki for and much else on the web for what went on.

ircount development

Thu, 26 Nov 2009 20:59:18 +0000

I’ve finally got around to spending a bit of time on the ircount code.

This post goes through some of the techy stuff behind it. If you’re just interested in features, I’m afraid there’s none yet, but you can now compare more than 4 repositories, but that’s as far as you’ll want to read.

Amazon AWS EC2 and vufind

Sat, 11 Jul 2009 21:53:36 +0000

Today I saw a tweet from juliancheal, which mentioned he was setting up his virtual server on slicehost. I hadn’t heard of this company but their offering looks interesting. This got me thinking about cloud hosting and I decided it was time to actually try out Amazon’s AWS EC2.This allows you to run a virtual server (or multiple servers) in the Amazon cloud, servers can be created and destroyed by a click of a button.

The Data Imperative: Libraries and Research Data : comment

Wed, 03 Jun 2009 20:20:08 +0000

I put this in to a seperate post. It continues on from my previous post, but didn’t want my notes of the day to be taken over by my ill thought views.

Personal Thoughts

Reluctant to give some thoughts as I know so little about the service. However… (!)

There seems to be two clear areas here: Data formatting and Data storing. There is some linkage (Preserving surely covers both, formats can become obsolete, Servers die), yet the two seem to be somewhat seperate.

Event: The Data Imperative: Libraries and Research Data

Wed, 03 Jun 2009 17:26:06 +0000

Today I’m at the one day event ‘The Data Imperative: Libraries and Research Data’ at the Oxford e-research Centre. As usual, these are my own rough notes. There are mistakes, gaps and my own interpretation of what was said.

Paul Jeffreys : Director of IT, Oxford University.

Started off giving an overview of where this has come from. e-Research is more than just e-Infrastructure. e-Research is not just about outputs, but outputs (articles/data) are a part of this, and an discreet area to work on.

Research in the Open: How Mandates Work in Practice

Fri, 29 May 2009 09:28:35 +0000

Today I’m at the RSP/RIN Research in the Open: How Mandates Work in Practice at the impressive RIBA 66 Portland Place.

Slides can be found here (not available when I made this post, as semi excuse as to why my notes miss so much). These are rough notes, which I’m making available in case others are interested, apologies for mistakes and don’t take it as gospel!

After an introduction by Stéphane Goldstein, kicking off with Robert Kiley from the Wellcome Trust.

Library search/discovery apps : intro

Wed, 27 May 2009 14:41:41 +0000

There’s a lot of talk in the Library world about ‘next generation catalogues’, library search tools and ‘discovery’. There’s good reason for this talk, in this domain the world has been turned on its head.

History in a nutshell:

The card catalogue became the online catalogue, the online catalogue let users search for physical items within the Library.
Journals became online journals. Libraries needed to let users find the online journals they subscribed to through various large and small publishers and hosts. They built simple in-house databases (containing journal titles and links to their homepages), or added them to the catalogue, or used a third party web based tool. As the number of e-journals grew, most ended up using the last option, a third party tool (which could offer other services, such as link resolving, and do much of the heavy lifting with managing a knowledge base).
But users wanted one place to search. Quite understandable. If you are after a journal, why should you look in one place to see if there is a physical copy, and another place if they had access to it online. Same with books/ebooks.
So libraries started to try and find ways to add online content to catalogue systems in bulk (which weren’t really designed for this).

The online catalogues (OPAC) were simple web interfaces supplied with the much larger Library management system (ILS or LMS) which ran the backend the public never saw. These were nearly always slow, ugly, unloved and not very useful.

Free e-books online via University of Pittsburgh Press

Tue, 26 May 2009 09:27:09 +0000

The University of Pittsburgh Press has put nearly 500 out of print books online and Open Access. You can access them via their Digital Editions website. This is excellent news, making work which could be lost openly available to all.

For years there has been a movement towards making Journal articles Open Access, i.e. publicly available. However some subjects (especially in the Humanities) publish much of their research in books, not journals. Letting the world gain from the (normally publicly funded) research contained within books is more complex, and it’s not an area I fully understand. The author normally receives royalties from book sales. However I understand this are normally very small 99% of the time, and normally tail down to tiny amounts after a few years. What if funders and Universities demanded that any book written with their money (or during their employment) must be made publicly available after x number of years (let’s say 10 years)? Academics and Publishers would not welcome the move, but would still allow a window where they can gain revenue, and if this became the norm it would be something they just have to accept. Meanwhile, once open access, the book becomes much easier to archive and preserve, and ensure the knowledge is available to all in the long term. Just a thought.

ircount : Repository Record Statistics

Thu, 14 May 2009 19:34:53 +0000

I’ve updated Repository Record Statistics (which I refer to as ircount).

Some key points:

Repository names with non-standard characters were not displaying properly. They now should, though there are some parts of the site where I have not updated the code yet. Many Russian IRs are also not showing the correct name, even though it is correct in the database, which I will look in to.
If a repository changes its name (in ROAR) then the new name should be shown in ircount
When looking at the details for one Repository, you can now compare it with any other repository, not just those from the same country. You’ll see two drop down boxes, one for those from the same country (for convenience) and one listing all repositories.
I’ve removed the Full Text numbers, which only appeared for some repositories. They were inaccurate and not very useful.
There’s now a link on the homepage to ircount news (which is actually just posts on this blog which have been tagged ‘ircount’, like this one).

The code is now in a subversion (svn) repository, my first time using such a system, which should help me keep track of changes. I can make it available to others if anyone is interested.

Linked data & RDF : draft notes for comment

Fri, 24 Apr 2009 14:12:59 +0000

I started to try and write an email about Linked Data, which I was planning to send to some staff in Library I work in.

I felt that it was a topic that will greatly impact on the Library / Information management world and wanted to sow the seeds until I could do something more. However after defining linked data, I felt I should also mention RDF and the Semantic Web, and also try and define these. And then in a nutshell try and explain what these were all about and why they are good. And then add some links.

Academic discovery and library catalogues

Sun, 29 Mar 2009 14:24:25 +0000

A slightly disjointed post. The useful Librarytecholgy.org website by Marshall Breeding announced the eXtensibleCataloge project has just released a number of webcasts in preparation for their software release later this year.

[][3]

eXtensible Cataloge webcast screenshot

I’ve come across this project before, and put a little simply, is in the same field as the Next Generation Catalogues such as Primo, Aquabrowser and VuFind.

Library catalogues, search systems and data

Thu, 12 Feb 2009 12:34:29 +0000

Below is an email I sent to UK Library e-resources mailing list (lis-e-resources@jiscmail.ac.uk). I’m putting it here for the same reason then I sent the original email: I think there are questions relating to the changing role of the library catalogue and new models are developing in how and where metadata exists for the items libraries provide access to.

My points in a nutshell:

The way we work with Library system (LMS) catalogues is changing, with the need to import and export large batches of records from other systems increasing, especially with online materials such as journals and e-books. This is quite a different need to those before the web, when item = physical thing in the library building, records were added one at a time, with
Library systems have not adapted to this new need, and though technically possible is often fiddly and can feel like a hack.
While it is possible to import batches of records, there are issues regarding keeping everything in sync. For example, Libraries often subscribe to publisher (online) ‘journal bundles’, the titles included in these bundles can change over time, but how to easily update/sync the catalogue to reflect this. One option is to regularly delete the imported records and reimport from the data source completely, though, if I understand correctly, Library Systems often do not delete records, but instead simple ‘suppress’ from the public view. So doing this for twenty thousand e-journal records each month would leave 240,000 hidden records on the system each year!
Why do we want them on the catalogue? Because users search the web interface to the catalogue to look for journals, books, theses etc. So we need to ensure our e-journals, e-books etc can be discovered via the catalogue interface.
A Library System (LMS) will typically have a catalogue, which cataloguers and other library staff maintain, and a public web front end for users to search (and locate) items.
However, ‘next generation’ of web search systems are now on the market, these allow the user to search the LMS catalogue and other data sources simultaneously in one modern interface.
Setting up these systems to search other data sources (in addition to the library system catalogue) so that they include records for online journals and e-books (and more) is a much neater solution, then trying to add/sync complete cataloging records in to the library system catalogue.
This to me (and I’m no scholar on the subject) has changed the model. The Library System catalogue was one and the same as the public web catalogue. What was on one was on the other. Librarians would discuss ‘should we put X on to the catalogue?’. But now these two entities are separate. Do we want something to be discoverable on our web catalogue search system, Do we want a record on our back-end library System for something? These are two separate questions, and it is possible to have an item on one and not the other. It would be easy to say that if you just want users to be able to discover a set of items, just make them available on your next generation search systems if it wasn’t for the fact that…
Third party systems can cross search or import records from multiple library catalogues, getting data from their library system. This was a simple thing to consider: do we want to allow these systems to have access to our records/holdings, and if so they would search our catalogue. These are examples and not the only things to consider, for example Endnote allows you to search a library catalogue within the application, of course this is the Library System catalogue.
This creates questions: which items do we want to make available to our users searching our web catalogue? which items do we want to expose to other systems out there? What items do we want to keep in our Library system back-end catalogue for administrative/inventory purposes? With the old simpler model these questions did not need to be asked.

I’ve drawn some rather embarrassingly bad diagrams to try and illustrate the point:

“Sitting on a gold mine” – improving provision and services for learners by aggregating and using ‘learner behaviour data’

Fri, 12 Dec 2008 18:03:16 +0000

I’m at a workshop today called “Sitting on a gold mine” – improving provision and services for learners by aggregating and using ‘learner behaviour data’ (it rolls off the tongue!), which is part of a wider JISC TILE project looking at, in a nutshell, how we can use data collected from user and user activity to provide useful services, and the issues and challenges involved (and some Library 2.0 concepts as well). As ever, these are just my notes, at some points I took more notes than others, there will be mistakes and I will badly misquote the speakers, please keep this in mind.

Mashed Libraries

Thu, 04 Dec 2008 22:16:09 +0000

Exactly a week a go I was coming home from Mashed Libraries in London (Birkbeck).

I wont bore you with details of the day (or more to the point, I’m lazy and others have already done it better than i could (of course, I should have made each one of those words a link to a different blog but I’m laz… or never-mind)).

Playing with OAI-PMH with Simple DC

Thu, 13 Nov 2008 14:15:14 +0000

Setting up ircount has got me quite interested in OAI-PMH, so I thought I would have a little play. I was particularly interested in seeing if there was a way to count the number of full text items in a repository, as ROAR does not generally provide this information.

Perl script

I decided to use the http::oai perl module by Tim Brody (who not-so-coincidentally is also responsible for ROAR, which ircount gets its data from).

ircount : new location, new functionality

Tue, 21 Oct 2008 20:18:08 +0000

A while a go, I released a simple website which reported on the number of items in UK repositories over time. It collected its data from ROAR but by collecting it on a weekly basis could provide a table showing growth week by week.

First it has a new home: http://www.nostuff.org/ircount/

Secondly, it now collects data for every institutional (and departmental) repository registered in ROAR across the world. Not just the UK. It has been collecting the data since July.

RIOJA – Repository Interface for Overlaid Journal Archives – live blog

Mon, 07 Jul 2008 09:57:52 +0000

Living Blogging http://www.ucl.ac.uk/ls/rioja/ see http://is.gd/N8T or , in fact, see below…

Update (8 July 2008):

This is my first attempt at using coveritlive.com (as used very sucessfully many times by Andy Powell at Eduserv). On the whole thought it was sucessful, and the software was great.

Few points: The software doesn’t seem to allow you to edit entries, I guess this keeps with the spirit of liveblogging but can be a pain when you notice a mistake just after sending a message. In fact trying to take notes and listen to what is being said lead to quite a few typos etc, the small font on the screen – further away on my lap than it would be on a desk – didn’t help. Secondly, difficult to let people know you are doing this (the other extreme would be to bombared mailing lists announcing your intentions), so some who may have had an interest in keeping an eye on the days event would not have been aware of this.

JISC Library Management System Review

Thu, 17 Apr 2008 14:48:09 +0000

The JISC and SCONUL have just released a Review (Horizon Scan) of Library Management Systems (LMS or ILS). Intersting stuff. A lot of stuff to be expected, and some interesting findings:

Dire need to embrace web 2.0 and to get OPACs out of the 90s
The need to intergrate with campus systems such as Financial and registry systems.
Open Source is used within current systems, and as systems in their own right, the latter have no real penetration in the UK market and are unlikely to do so in the short term (lack of advantages), and are not currently more ‘open’ than other systems.
Most major systems are much of a muchness (they put it far more elegantly, though after glancing through 100 odd pages I’m not much inclined to find each quote), little reason to change system at the moment. Especially as…
It is a good time for the role and definition of a LMS to be looked at.
We are already moving away from the ‘one big system approach’ i.e. Aquabrowser, ERMs and Metasearch are all separate products. Almost certainly going to move in this direction, and this is a good thing. This requires standards and cooperation for different systems to talk to each other.
UK market is small compared to global market, though not that different to the norm, except for BL ILL server and our concept of Short Loans.
Libraries have not made good use of looking at how users work and interact with their systems.
There is a need and movement to liberate data (silios and all that)
LMS may become a back of house system (though this should not be seen as a bad thing)
Recommends that libraries increase the value of their investment by implementing additional services around the LMS. Which is fine if these new services are using standard protocols to interact with the LMS, if their are using proprietary api’s or talk direct to the database then it is another thing to lock the library in to one provider (or at least, another reason why changing LMS will have a large impact).
Procurement process is overall expensive, especially when LMSs are more or less the same-ish.
Encourages libraries to review their contracts (ie not changing system, just getting more – or better value – out of the current system with better contract).

One thing that interested me were the Vendor comments (and something the report reccommends to libraries) on the lack of consortia in the UK, and noted that other countries make good use of this (ie this is best practice). I can certainly see scope for this, especially as (like the report notes) the software has already been designed to handle consortiams.But what would the consortiams be? geographic based (London? South East? M25 group? Scotland) or perhaps using other groupings (Russel group, 94 group, CURL). Or perhaps smaller groupings based on counties or similar institutions.

or08: eprints track, session 2

Thu, 03 Apr 2008 11:44:32 +0000

After coffee a little more talk about new features and the future as we ran out of time before. Christopher Gutteridge has now turned up, he may have had a few grown up fizzy drinks last night.

(lost concentration here: salt grain take) Eprints plugin will try and pick when people enter their names wrong (e.g. get first/lastnames mixed up). Report an eprint (or report an issue with an eprint) link on item/record pages?

or08: Eprints 3.1

Thu, 03 Apr 2008 09:35:18 +0000

At the sucks-less-than-dspace Eprints track today. First up Eprints 3.1 and Future. Haven’t seen anything about 3.1 before this, v3 was released over a year a go so looking forward to seeing what is new.

9:10am: Les Carr is talking. reviewing v3 released last year. Talking about the large amount of work surrounding a repository (for all), which he experienced first hand running the soton ecs repository, and the work they have put in to help this. He found that when he contacted academics to point out problems he has fixed with their items/records they seemed pleased glad that someone was doing this. Last year they (eprints team) wanted to focus on ‘things on the ground’ to make things easier and not focus too much on rejiging the internals.

UK repositories : growth of records

Sun, 30 Mar 2008 19:06:36 +0000

For a while now I’ve been running a weekly script which connects to ROAR and grabs the number of records for each UK based Institutional Repository. I’ve finally got around to writing a web front end to this, which you can see here. All quite basic at the moment, and I have lots of ideas of what I could do to improve this (one idea based on the compare average number of deposits per repository). Have a look and let me know what you think, and let me know of any bugs.

Google Books API

Sun, 16 Mar 2008 23:58:15 +0000

Google released an API for their Book Search at the end of last week. You can implement this in two ways: static (just linking to a URL with a ISBN inserted) or dynamic, which basically means using javascript, which will check to see if Google have the book available as the page loads and can show a link accordingly.

The (very rough and basic) ‘catalogue’ I recently created already had a link to Google Books for each item. This was just a very simple URL with an ISBN stuck on the end, e.g. http://books.google.com/books?as_isbn=0596003722

nostuff library catalogue using the Talis Platform

Fri, 08 Feb 2008 11:17:15 +0000

You may have already seen that a while a go I had a little play with the Talis Platform, and specifically a ‘store’ on the platform called ukbib which holds bibliographic records. I’ve written some pages on learning about the various parts to the platform. I had seen on the Talis Developer Network (TDN) various web services, but they all blurred in to one, the pages linked to above first work out the simple ones (e.g. linking to catalogues, holdings etc), this helped clear up the different bits that are available.

Talis Platform

Wed, 19 Sep 2007 22:10:47 +0000

A few days a go I was bored. I had read loads about the ‘Talis Platform’ a open standards ‘you can talk to it with an API’ type thing that holds data, and shares it with you (or more to the point, shares it with your application).

Could a non-programmer like me make use of this? Hello No! but I did wonder if a non-programmer like me could cut ‘n’ paste their examples and play for myself.