I suppose you could also add to these another, where the original document has no existing pagination/foliation and you have to provide it yourself, but this fits within the design of the above categories - you could include a field to mark whether they are paginated in the original or not if you wished.
Each of these page types has their own utility and my database keeps track of three of the four (all except the page range, which I probably should have). The distinction between the Notes.Pages and the Notessub.Page fields is particularly important for two reasons. First, it allows you to automatically include the appropriate page in the Cite query (to be inserted into a Word footnote, for example). Otherwise, you'll only paste the generic page start (or page range if you prefer), or nothing at all, and have to manually enter the correct page number for that specific passage, possibly having to look back to previous NsIDs to find what page they left off with. Second, with this field you don't have to repeat at the beginning of every Notes field the page number, in case you divide a page up into several separate NsIDs.
For passages in a single NsID that continue onto the next page, I haven't come up with any better solution than to simply note it in the Notes field in brackets: "I saw them [40] the other day. Then we went home" with the Notessub.Page = 39. If your citation is completely on the next page, e.g. you quote "Then we went home" in a paper, you will need to remember to change the footnote citation from p. 39 to p. 40. I can't see any other way to automate this, unless you wanted to make the Notessub.Page into a Notessub.PageRange.

Ideally, it might be nice to be able to quickly calculate the number of pages spent on a particular topic, but I'm not at all sure how to do it with this design, perhaps a separate field somehow, or include Notessub.PageStart and Notessub.PageEnd fields? In my Secondary database I have a PageStart and PageEnd field, with a calculated field that automatically does this.

5) The keyword field is quite a poser. Much ink has been spilled (and many pixels darkened) on what is the "best" keywording strategy - librarians apparently get into this kind of thing. I had initially used a strategy common in other databases (e.g. EndNote) where you have one memo field in which you put as many different keywords as you want in no particular order. Unfortunately, this method does not provide adequate precision and consistency on the entry side, two very important features that are required for efficient document-level searching IMHO.

Instead, I have adopted a much more complicated system that attempts to keep track of the many substantive questions I might find of interest about the content of a particular document: who, what, where, when, why and how. Its complexity requires I spell out in detail the logic I used, for my own sanity at least. There are three strategies I have used to provide detail, to ensure that I consistently keyword my records the same way, and to make finding records easy.

First, I have divided the generic Keyword field used by many other programs into many separate fields for each keyword type: separate fields for War, Theater of content, Province, Place of content, (type of) Event, EventID (a specific event), Person mentioned, Army, Side of content, Topic... By dividing up these various sub-types of keywords, I can keep the Topic keyword field constrained to topics and themes rather than having it cluttered with the who, when, and where: with only a single Keyword field, some records might have eight or more keywords covering each of the above mentioned sub-types. You are much more likely to forget to code for one or more of the categories in such a system. I did this all the time in EndNote - forgetting that I not only wanted "France" in the Keywords field but also "Louis XIV" and "17C" and perhaps even "Western Europe." With only "France" and "Louis XIV," to take an example, if I wanted to compare French and English records discussing the 17thC (NB: not just those published in the 17th century) I would have to include every 17th century monarch of both countries in my query in order to find them all. Potentially quite a pain. Much of this keywording overhead can be avoided if when you select the most precise keyword, your code behind the form sets the values of all keywords higher up in the hierarchy. For example: selecting "Louis XIV" sets the Century field to "17C," the Country field to "France," Region to "Western Europe," Continent to "Europe," etc.

Second, for the Topic field itself, which is limited to topics and themes discussed, I use headings and subheadings, with a progressively  narrower scope. For example: Sieges-Outworks, Sieges-Outworks-Assault, Sieges-Outworks-Assault-Repulsed. I did this because I found when combining multiple keyword fields together or multiple keywords in a single field together to form one compound keyword, I was unable to keyword consistently. I have found it much easier to change keywords with update queries (enforcing referential integrity means these changes are propogated throughout the database automatically) than to try and remember the combinations of keywords I used before - we're talking several hundred, quite-detailed keyword strings here. If I had made separate fields breaking up these keywords (e.g. KW1=Sieges, KW2=Opinion of, KW3=Criticism, all as one multi-field, compound keyword), then I could quickly become inconsistent with my keywording. For example, I might remember to type Sieges (KW1) and Opinion of (KW2) in many records, but forget on some occasions that I always want to also record whether the opinion was critical or complimentary (KW3) - this happens a lot as you refine your keywords and by habit use the more familiar outdated keyword groupings instead of the new ones. It's also useful as you refine your keywords to make them more specific and discriminating. And it is difficult to come up with a reasonable number of distinct keyword "categories" - you might want one keyword for the general object or topic, a subset of the general object (Artillery-Mortar vs. Artillery-Howitzer), a judgment on the topic (Sieges-Opinion of-Criticism), the result or status of (Sieges-Open Trenches-Delayed), and so on. But then you need to decide: should KW1 always be a type of event? KW2 an intensifier? Can "Criticism" appear in either fields KW2 or KW3, depending on how long the chain is, or would you do it KW1=Sieges, KW2=blank, KW3=Criticism? The more keywords, the more types of keywords, and the more detailed the keywords you have, the more complicated it gets. My solution is to use the first "unit" root (e.g. Sieges-*) to keep all the related topics together, so if I know the keyword is about Sieges or Intelligence but I'm not exactly sure how I phrased the entire compound keyword, I can find the right compound keyword by simply browsing through the drop down list (since they're sorted alphabetically) and all the Sieges-* will be together in alphabetical order. Without a hierarchical system, if you wanted to search for the more general category, you would either need to separately add 'Artillery' to every 'Mortar' and 'Howitzer' keyword field, or else you will need to select all possible types of artillery (mortar, cannon, howitzer, pierrier, Coehoorn mortar, transport of...) in your query (the same general concept as the earlier France-Louis XIV-17thC example). My solution takes a little bit more typing with this approach, but I think it improves the accuracy and utility of the coding and querying. You can search for all documents dealing with Siege-Outworks by searching with a wildcard, e.g. "Siege-Outworks*". A search of "Siege*" would find all documents with Siege-Outworks, Siege-Outworks-Assault, and Siege-Outworks-Assault-Repulsed, as well as Siege-Capitulation, Siege-Investment...
Each record on the NotesKeyword side is one of these hyphen groups. Example: Marlborough writes in letter A: "a letter of Villars' brought to me complains bitterly of the garrison's weak defense." In the NotesKeyword table, record 1 linked to letter A in Notes table is "Sieges-Opinion of-Criticism"; while NotesKeyword record 2 linked to that same letter A is "Intelligence-Intercepted correspondence" - each of these records is one field. This translates into: this letter contains criticism of one side's conduct of the siege (the side is in the separate Army field) and the criticism is actually from an intercepted letter that the author of this document mentions, i.e. Marlborough himself is not criticizing the defense. These keywords can be searched for separately or together; there is no inherent relationship between the keywords in NotesKeyword records 1 and 2. That means that this design makes it difficult to know when such letters were critcial of one side versus the other, because you don't necessarily know which Topic the Army field refers to - the Intelligence keyword or the Sieges keyword?
As a warning, every time you disaggregate keywords (i.e. add more detailed keyword levels to an already-existing broader one), you should create it as a new keyword entry (unless you are certain you've always interpreted "Sieges-Outworks-Assault" as "Sieges-Outworks-Assault-Repulsed" and never thought about assaults being successful) and then be sure to go back and recheck all your old generic records with that old keyword to see whether they should be Sieges-Outworks-Assault-Repulsed or Sieges-Outworks-Assault-Successful. Otherwise the distinction made with your newly disaggregated keyword will not be reflected in previously coded documents.
I have also separated out Stage and Branch from the Topic field and put them on the same sub-subrecord, so that for a Topic of "Sieges-Opinion of-Criticism" I can note that the criticism is of the engineers specifically (Branch), or of the gunners, etc. Or I might note that the "Sieges-Outworks-Covered way-Assault" was delayed or in preparation (Stage), whereas a blank Stage value (the default) would indicate that the assault itself was being discussed. Note that since these are the same NotesKeyword record, so they are specific to that keyword.

Confused yet?

Third, the search side of the equation becomes more complicated as a result. An important design decision needs to be made - what are the most common types of searches you want to perform? The single Keyword field approach is the best suited for straightforward querying, as you simply put wildcards around each keyword term, so that a query for "*Fortifications*" and "*Intelligence*" would find a record that had a Topic of "Fortifications; Intelligence; Costs." But if you have lots of records on each topic and you want to quickly narrow your search down to a particular one, this is not the way to go.

Separating out the many different sub-types of keywords means that that there is far less need for searching for multiple Topic keywords - and you're reminded to limit your search when seeing the variety of various keyword sub-types. This is important, since such a task is convoluted to do with my setup, as each Topic value, as a subrecord, is its own record.

The results:

6) The Notes field. Here are some general issues to consider when taking notes.

If you are having particular trouble reading the handwriting, it might help to mimic the line breaks of the original with a carriage return rather than let the program automatically wrap the text down to the next line - it's easier to follow the words and pick up where you left off this way.

To avoid the possibility of plagiarism charges, limit use of the Notes field to transcriptions only, using the Summary field for your own recapitulations. If you mix your transcribing and summarizing in a single field, be sure to consistently use quotation marks (of course it's easier if you're quoting text in a different language than what you take notes in).

Note more generally that a number of trade-offs need to be balanced in note-taking mentioned here.

Improvements Made

Six shortcomings mentioned in earlier versions of this page and database have been fixed through a combination of code and redesign:

a) Sorting text fields numerically, e.g. Letter number or Page, now done with the Sort By combo box (code courtesy of Liz).

b) I have made the content fields a separate subform (Notessub), so that I can break up a single document (e.g. a letter, a diary...) into multiple subrecords, in order to keyword each passage according to its content or date of content. For example, paragraph 1 discusses the events in Italy (subrecord 1), paragraphs 2-3 discuss the siege of Mons (subrecord 2), and paragraph 4 consists of a request for a promotion.(subrecord 3). As long as you keep the subrecords in the same order, you can always reconstitute the letter (the Text button). These subrecords can also be used to break up a document by day, so that I can compare events on a daily basis, even comparing what each source says happened each day. See Sample Queries for a list of some of the reports I can run with this feature. More simply, you could put your content fields on the Notes table instead of a subtable, but then, you will have to skim through the entire letter to find where it talks about the promotion request.

As the Order field discussion above indicates, this strategy does have its drawbacks if you are just skimming through a long document in a record and light upon something that you want to keyword without first parsing all the text that came before it. There are several problems to be avoided when you have a long text in a single NsID (e.g. pasted in from OCR, or you didn't want to parse the document as you were transcribing it in the archives). You need to keep track of the order of the subrecords so you can reconstitute the original document, so if you just cut the relevant passage out of the longer text and make it a new record, you won't know what was the order of the subrecords in the original document. You could simply keyword the very long NsID with the codes relevant to that particular passage, but then you would have to skim through the entire document to find where exactly the keyword-relevant passage is. The best choice is to cut the desired passage out of the larger text (e.g. NsID 1), create a new subrecord for it (NsID 2) and keyword it, then enter an obviously false Order# for this NsID (e.g. Order#=1000) and note in the larger text subrecord (back in NsID 1) what that Order# is (e.g. type "{Order# 1000 here}" or "{NsID 2 here, change its Order#}"). Using this method, when you are later finishing your parsing of NsID 1, you will be parsing the document in the correct order (the Order#  for NsID 3 - or whatever autonumber you are on by then - would be 2, NsID 4's Order# = 3, NsID 5's Order# = 4...), and when come across that notation for NsID 2, you will know to find the NsID 2 record and change its Order from 1000 to the next Order# (e.g. NsID 2's Order# = 5 since it physically occurs in the original after the text in NsID 5). Then you can continuing to parse the remaining text (NsID 5's Order# = 6...) in order. This is not as easy as simply being able to write the keywords on page 4 paragraph 3 of a photocopy, but by now you know the other advantages a digital database gives you over hard copy that far outweigh this minor work-around.
Or, if you used XML, you could just tag the relevant passage of the document and the order would be kept intact since you are not breaking apart the document, but the other advantages are worth the tradeoff IMHO. A bit unwieldy perhaps, but it's not that hard to do once you remember the several steps, and it's not needed that often anyway.

c) I have made the primary key for individuals an autonumber. See Personalities for the issues.

d) I added a field to indicate the language the document is written in. To give one example of its utility, you might want to find how French-language sources phrased an abstract concept without knowing the exact text strings used, or perhaps compare French and Dutch use of a French word (e.g. honneur), or, maybe you just remember that that great quote was in French instead of English. Ideally you'd just base the document's language off of the author, but there are a few complications that I've come across. First, the recipient or the author's education might demand writing in other than the mother tongue: the Dutch field deputy Vegelin van Claerbergen wrote in Dutch to the Raad van State - as all its correspondents were ordered to do - but wrote in French to the Dutch raadpensionaris Heinsius. So you can't assign language with 100% certainty by the service the author was in (especially given the cosmopolitan travels of early modern European elites), or even something like their place of birth (in fact, many of the Dutch-born authors - particularly Frisians - I use appear to have preferred writing in French over Dutch even to fellow Dutchmen). You could, however, have code set a default language to save entry time and change it when needed. I mass updated a lot of this by first assigning language by Service of Author, and then refined it (e.g. for the Dutch and English authors who might write in more than one language) by looking for words in the text that are unique to a specific language and then update the language field's value to the appropriate language - assuming the author didn't use a lot of foreign phrases and you didn't paraphrase in English and occasionally quote passages in the original language (oops). Or perhaps better still, you could update it by Recipient, as, for example, the Raad van State wanted all its correspondence in Dutch.
Occasionally there are even letters written in one language and postscripts in another: for this reason you should have a Language field on the Notessub table and have it automatically entered (which could be overwritten as needed) when you choose the Author. Having a language field is complicated, however, if the language varied within a single Notessub record (is it worth creating a Notessub record just to indicate a language change?). This is an example where XML tags would be better.

e) When doing searches on text in the notes field, the actual text string being queried will appear highlighted within the larger block of text if you use the Find command on the search form (search Anywhere in Field, not Whole Field). The "Next" button will jump from one record to another automatically. The KWIC button described above also helps with this.

f) I've added Circa checkbox fields to both the Notes and Notessub tables, so I can identify if the stated day is approximate or not (e.g. I know the letter was sent shortly after the battle of Ramillies, which took place on 4/25). I can also use this to get less-precise dates to sort in their approximately correct order: c. 3/15 can be interpreted to mean around the middle of the month; c. 3/1 for at the beginning of the month, c. 3/30 for at the end of the month. I suppose you could even break it down further into weeks if you wanted.

g) I've added a higher-level "Executive Summary" field that summarizes the gist of a passage (Notessub record) in a single sentence or two. I was inconsistent in the Summary field whether I should spell out each point, or whether a brief summary would suffice. Not only is this field good for skimming, but you can also include these fields in a search results form, creating a sort of "fisheye" report (see Thomas Landauer's The Trouble with Computers for a brief discussion of this).

Room for Improvement

There are still several minor modifications to be made, and I'm sure this is not the most-efficient organization possible. Overall, however, this database is so useful as I continue to "tweak" it that I'm amazed every time I use it. At this point, it's more a matter of adding new minor features than an overhaul. That and getting the data entered in and coded.

• I'm sure there are more elegant ways to hide the irrelevant null fields by displaying subforms based on the value in a combo box (e.g. the Source fields when it's an Archive source), but I kept the design as simple as I could. Going against ideal database design, there are many blank fields in each record, but the size of the database is not an issue (75 MB with 22,000 records in the Notes table thus far in Access 2002, almost half that size in earlier versions of Access) - I've got 30 GB of hard drive space. Nor is speed a serious concern: searching for a text string  (e.g. "*Maubeuge*) within the Notes memo field in 15,000 records, which is the slowest type of query, takes perhaps 15 seconds or so on my 475 MHz 64 MB RAM laptop, hardly a crawl.
• How to note separately dated documents in the same letter (postscripts and enclosures). For example, the original letter is written the 3rd, but a post-script is added on the 5th. I suppose I could enter them in separate records and make the postscript an appendix in Form (or for that matter, I could create a new type of Form called postscript). This is similar to newspapers, which might include a date range (e.g. issue for 3/5 through 3/11). I could add new fields for that, but it doesn't come up often enough in my newspapers to warrant the extra space on the Notes form. Or I could just add the postscript as a separate Notes subrecord and make a cross-reference to the original. Similar issues arise with enclosures.
• Along those lines, I could add a more formalized Date Range when it's not clear exactly when a letter was written. This would allow a letter written between August and October 1706 to show up in queries searching for letters written before September 1706 (this would be something akin to Historical Event Markup Language's dateSpan tag). Right now I use the Circa checkbox when the day is uncertain. If I don't want to leave the Day field blank, for example, I could conceivably use "dummy" values, such as 1, 15 and 30, to indicate what third of the month it is, in order to have it sort somewhat in order in queries and reports.
• I could also make a new view of the Notes records. In addition to the Single record form (as seen in the screen shot), Access also gives you the option of a spreadsheet-type display ("datasheet view") of all the records. I could make some combination of the two - the top pane showing a spreadsheet-style display of the records (most likely resulting from a search) and the bottom pane of the window showing selected fields from the record selected in the top pane.
• A way to indicate if a particular record has already been quoted or cited in a chapter (a check box or a field pinpointing the document and page number?).
• I could add a field that would indicate the catalog location of the document - in the case of microfilm held by the library, the call number, reel number and even the microfilm machine's counter number (i.e. where on the reel) of a particular document that I frequently use.
• You could add a separate field to keep track of the salutation and subscription of each letter if you wanted to compare these. These might give you an indication of the social relationships between correspondents. Similarly, if it was important enough you could also have a field separate from the Notes field to record references to other correspondence (usually the first line of the letter starts something like "I received your last letter of the 9th of this month...") - it might help keep track of the web of correspondence and allow you to track down important documents. Social Network Analysis is particularly interested in these types of information.
• I could work out some notation to keep track of the paragraph structure when I break the documents up into several subrecords (perhaps an XML tag). This seems quite a low priority however, especially since several of the printed editions I use do not do this anyway, and contemporaries in my period were hardly consistent in their use of line breaks.
• There could be some way to identify changes in the handwriting of the document, probably with some sort of marker in the text field itself - for now I use the XML tag <handShift> when I remember. To an extent this feature already exists with the Form of Document field, where you can indicate (if you know) whether the letter was written in the author's hand or perhaps in the hand of a secretary. But you would need something more with letter- or resolution-books, where different secretaries or members recorded different entries, and in some cases, the handwriting even changed in mid-sentence! Very odd, but perhaps worth keeping track of, if you were into that sort of thing. Of course you'd need to be careful using such information, since we can't always assume that whoever wrote the document actually composed the document....

Suggestions or comments on any of this are welcome.

Limitations

There are, of course, a number of limitations with this type of database. The ideal research tool would reproduce images of the documents (unlikely given most archives' close protection of their holdings), transcribe them into readable, digital text alongside the original handwriting (even difficult-to-read handwriting), translate the text if necessary into a language you understand, and perform automated keywording as well as allow unlimited searching and cross-referencing. Different software packages offer only a small subset of these features, but ultimately the combination of price (free with Office Pro), availability (many manuals, Websites and user groups around) and utility made me choose Access for my own use. Perhaps if there were some standard that hundreds of other historians used and encouraged graduate students to use early in their studies, I might have chosen that instead. But I'm not bitter for having to develop this system on my own (he says with sarcasm). It is incredibly liberating to design a form exactly the way you want it, says the control freak with little sense of design.

Note-taking and SGML/XML/TEI/MEP:
A relatively new type of software using SGML/XML might have real potential to solve a lot of the issues I've had to deal with in the creation of this database - search the Web for oodles of sites on these acronyms, but particularly check out the Historical Event Markup Language at www.heml.org (not for neophytes). However, from my cursory reading of the literature thus far, this is still in the future, particularly regarding the (parsing) software. Additionally, SGML/XML seems wonderful for electronic editions, publishers and  literary analysis dealing with parts of speech and structures (structural and presentation markup), but it strikes me as being of little utility for research with the types of documents I deal with. At the very least numerous additional tags would need to be developed to mimic the entities easily created (via fields) in my database. Perhaps I'll become a convert once an easy-to-use SGML/XML editor is widely available and a single standard appears (I was initially skeptical of the utility of electronic databases for note-management! you know what they say about recent converts being the most fervent) - only time will tell. From my limited perspective, it seems that RDBMS also have many other advantages, especially in data entry (see the examples of field values setting other fields' values in the field descriptions above). Access 2002 also lets you export your databases into an XML format, so this will make converting my existing database much easier if a killer XML app ever appears. Looking at the Model Editions Partnership (MEP) guidelines for historical documents based off of the Text Encoding Initiative's (TEI) XML tags, it turns out that I've already duplicated a lot of their meta-data tags with my own database fields and I will try to adopt several more (<gap>, <handShift>, <docketing>, and <endorsement>) to use within the Notes field (see http://adh.sc.edu/MepGuide.html for details). It should be relatively easy to export all the fields with the appropriate tags should I ever need or want to. In any case, converting the data from a RDBMS to SGML/XML format should not be at all difficult, so don't use this as an excuse to get started right away.

Note-taking and Transcribing Documents (a digressive rant)
In the literature on computerized databases and history, much emphasis is placed on transcribing the full text of the document, in order to not lose any of the information. In general I would agree with this admonition (record as much of the information in your note-taking as possible, time permitting), but purists might criticize me for not transcribing all sources exactly as they appeared. Sadly, almost every article or book I've seen on the subject of computers and humanities or scholarly electronic documents is intended either for those aspiring to produce critical scholarly editions of works, or for literary scholars who are much more interested in the semantic structure of documents than I am (and, I'd think, many other historians as well), and all seem to assume an amount of time and resources that I know I will never have. I would make several responses as the "traditional" lone historian:

1) My database design does an admirable job of saving the textual basis of the quantitative data I create from it, so that we can always return to the source if there is any confusion or contention. Oddly enough, articles from several years ago (circa 1996) acted as if relational databases were incapable of searching text; a concern which software advances have largely eliminated. Similarly, the requirements of archivists and librarians are not necessarily the same as those of historians. So rely only on up-to-date information on software reviews, and don't be disappointed if no one mentions capabilities you want - it may require some effort, but many features can be implemented if you know what you're doing!

2) If we followed the idea of transcribing the source exactly-as-it-is too slavishly, we'd use colons instead of periods in some Dutch correspondence, make a line break at the end of every line in the manuscript, paragraphs would disappear (did the writer start a new paragraph or just not want to start the sentence out at the end of the line? I have no idea), we would have either fragmented words which should be one word or compound words that should be separated, and let's not even start on the whole capitalization thing.... Believe it or not, I'm converting "poffeffion" to "possession"! (This 'f' replacing 's' spelling, which I'd heard was a function of early printing press techniques, even occurred in handwritten English and French!). Some people might very well want to explore my data for that purpose, but frankly most people won't, and I'm not going to do extra work for some hypothetical scholar in the hazy future (assuming I start handing out all of my research notes in the first place). The effort is simply not worth the payoff. Not to mention, the whole issue of copyright and archival materials is still fuzzy to me.

3) I'm rather put off by how such discussions occasionally approach near-worship of THE SOURCE (read with a thundering voice). Keats-Rohan (cited below) argues that translating and standardizing orthography (especially proper nouns) corrupt the data (p69) and should be avoided at all costs. This may make sense for ancient and medieval history, prosopography and literary analysis, but I'm sure many historians have no need to study one particular document (or a few documents) as closely as a literature scholar might study a piece of literature, nor do most historians have multiple versions of long documents to compare, so the concern seems misplaced for many historians. The people I study wrote or dictated ten or more occasionally-repetitious letters every day, after fulfilling their tiring military duties which took up most of their time. I wouldn't expect the language or structure of their letters to require detailed analysis. Ultimately, if someone wants that degree of faithfulness to the original source they are of course welcome to it - others should not, however, feel shamed into providing such detail in their own work if it does not require it. Decide what you need and implement it.

4) Unless we actually have the original document, there are always going to be types of information which even the best electronic edition cannot replicate, at least for the foreseeable future. For example, does a person's handwriting style shift when he writes in French versus Dutch? Even if we could scan images of the original document (archives can be sticklers for copyright), we might need to test the ink or paper to see if the document is a forgery, or perhaps something was scribbled on the back, or an impression of something written on top of the document that is discernible with shading or the proper lighting, or perhaps something was erased, or perhaps the handwriting subtly changed. Art historians and scientists perform these types of analysis on artwork and some documents today, so this is not science fiction; in fact, I found a document in a library that purports to be a long letter written by a contemporary whose provenance I would like to verify. The issue of diminishing returns needs to be addressed, however, and I think the bar has been set far too high by some. But then that's what quotidian historians get when they leave the software development to more literary scholars in the humanities, to the more specialized fields of ancient and medieval history where sources are quite scarce and therefore each one receives lavish amounts of attention, and to managers of large-scale historical research projects - I created my database so I could keep track of too many sources, quite a different problem.

5) Since transcription purists acknowledge that the desirability of complete replication is based on the fact that not every scholar needs the same information from the document, it would seem to make sense that we should spend our efforts maximizing the amount of material made available so as to benefit the greatest number of scholars (i.e. its content), rather than continue past a point of diminishing returns for the few hypothetical future users who would slow down our progress by demanding the ability to analyze a particular document in extreme detail, dealing with grammar or typefaces, for example. We must prioritize, and I'd hope we could widen the offerings of digital history currently available on the web far more, particularly for us early modern Europe partisans (kudos to the Bibliothèque nationale's online Gallica project for going this route, even if they're only scans rather than text). As a cardinal rule, your tools and methods should be determined by your sources and your objective, and history is chock-full of different flavors to choose from, so don't expect any single program to have them all, or for any program to meet the needs of all users, or for any transcription to meet all needs. Hence my preference for knowing the software yourself: customize, man!

6) It's worth noting that those who encourage the use of such strict standards enjoy large-scale institutional support - resources hardly available to most historians, while their material may be interesting but of limited value to the research projects of many historians. Enough institutional support to pay enough scholars to critically examine the millions of documents in archives with the same degree of attention that works by particularly renowned individuals have merited simply does not exist. Time and cost are important in a field like history where there is little money to go around - I think we could spend it more wisely "for the greater good." And for individual scholars and projects of one (like mine), don't let anyone tell you it's better to enter printed documents in manually than to use OCR (perhaps another artifact of constantly-improving software) - with decent OCR software like FineReader, you will save yourself an unimaginable amount of time, even with many 18th and 19th century printed documents. Must be nice for some libraries and private companies to have the money to outsource their data entry to Third World computer "sweatshops"...

7) While much of the raw material of history is document-centric (as composed to data-centric information better served by relational databases), much of the metadata surrounding these textual sources are data-centric. As a perusal of the many fields in my database indicate, the text often forms only the basis for my analysis - the people, places, events, things and concepts in the texts are what interest me just as much, and most of those can be easily "data-fied" (if not quantified) without the need for much methodological concern over lost data in the process, especially when you still have the original text in the Notes field. You obviously need to choose the best tool for your task: I would contend that for many historical tasks, using a readily available and customizable RDBMS allows you to take advantage of the highly data-intensive nature of even purely textual documents, allowing some limited document-centric abilities used by XML within an overall data-centric model. Check out www.rpbourret.com/xml/XMLAndDatabases.htm for an interesting discussion of the differences between the two (written by an XML programmer but with a more general business audience in mind). In early December 2002 the Humanist listserv again discussed the issue, pointing to hybrid relational database-XML approaches.

8) Finally, as a philosophical issue, I would argue that the partisans of the Source overemphasize the dangers of interpreting documents for others - everything we do is an interpretation, but we do it all anyway and most of the time most reasonable people can agree upon a single convincing interpretation (in a well-defined, mutually-agreed upon context), or even a few interpretations for a single document or source. Much of our research is necessarily based on secondary sources, and while I distrust other historians as much as the next scholar (maybe more), we will have to continue this act of faith. To me it's a matter of cost-benefit analysis (or opportunity cost): providing a few new, electronic sources worthy of the title "critical edition," although important and worthwhile, is setting our sights far too low, for there are a few problems with this approach. First, the heavy reliance on a few "canonical" sources will continue to blind us to variations and contradictions in our subjects, often giving us a stereotyped impression - one of the best (and easiest) ways to revise the historiography is to show how less-well-known sources differ from the oft-quoted ones (is Machiavelli representative of most political philosophers of the Renaissance?). More generally, the most important challenge should be to facilitate the easy and more-widespread use of many more digital sources. A few scholars over a century might use the entirety of an 18th century French chevalier's memoirs to analyze its internal structure, but many more scholars might use more mundane parts of that work for their particular research interest: the chevalier's discussion of gender, of sieges, of honor, or of the consumption of potatoes, to name but a few examples. Imagine how far Louis XIV historians would get without the thematic and proper noun indices of Saint-Simon's 41-volume memoirs! In the time it takes to put out the critical edition, scholars could have put out several slightly less-exacting works (which might forego, for example, identifying ever proper noun referred to in the text with a footnoted reference - perhaps linking it to a meta-person index?), with the same amount of money. Reliance on other historians' work is a necessity, but ideally we would be relying on them to provide us with the raw materials from which they fashioned their arguments as much as their more focused, finished scholarship. My point: the literature on historical computing is dominated by the interests of large projects strictly circumscribed usually in time and certainly in geography. Quite understandably, their recommendations are aimed at their peers, something that newbies looking for helpful advice on individual projects like mine need to be constantly reminded of. Just be sure to document and follow rigorous standardization procedures and you should be fine.

To take one example that emphasizes the need for this everyman-a-digital-historian approach, Keats-Rohan again seems to limit the utility of historical digital databases by arguing that attempts to categorize her People (drawn from the Domesday Book and its variants) by more abstract categories than what exists in the text "fatally impairs" the reliability of the analysis. Perhaps this is a good idea for sources which are flawed enough without introducing additional errors, but this is a bad idea for historians more generally - unfortunately, it's hard to find any published articles on the kind of quotidian database I've designed for early modern or modern scholars. That's exactly what historians do: we all rely on the conclusions of secondary accounts that are much more filtered than a database assigning modern analytical categories anachronistically to past societies and events, and there's a large literature from sociologists and social historians about how to categorize, for example, class and occupation in order to test modern theories with past data. If you've ever used an index of a monograph to look for a particular topic or place, you know how useful an expert's analysis and use of categories can be (and what additional categories you wish they included). I have yet to find the comprehensive printed index, whereas a customizable database such as mine allows just such a beast - a public electronic edition would allow an additional layer of added categories found useful by others as well as the original text from the document. If historians could develop some relatively standardized set of keywords (à la Library of Congress but with more detail), a document could be progressively coded (perhaps even by several people, according to their research interest) and these codes would then be made available to others. Or you could even compare different coders' codes historiographically, comment on their decisions, and choose whichever you prefer, or even modify the codes for your own purposes. As long as the original document is not altered, I don't see the problem for the vast majority of historians.
In fact, not coding the text for others actually hinders the synthetic/comparative history we all should support. Merely transcribing a source, no matter how painstakingly, does not radically speed up history; it simply accelerates the storage and transmission of the raw data, doing little to speed up the processing or analysis of this data - the most difficult and time-consuming aspect of research. Other historians, who might be interested in only a particular aspect of a document's content, will still have to wade through the entire document to find the few kernels (unless they are fortunate enough to know the exact text strings used to embody their abstract concept being analyzed), along with the hundreds of other documents they might have as well - I have tens of thousands of documents to consult, and hardly the time to analyze the backgrounds of each individual author and the details of their relationship with the recipient beyond a certain basic level. Hopefully the editor of the collection will provide the summary you need without you having to develop your own by going through the author's entire corpus. Richard Bonney has discussed the wasted energy spent by modern scholars re-researching various aspects of European fiscal history (www.le.ac.uk/hi/bon/ESFDB/bapres.html), and Randy Roth has similarly had to redo the archival research of earlier scholars on colonial American violence rates because of their failure to publish their raw, disaggregated data. I would argue that qualitative research is similar to quantitative research in this respect. History would "progress" more quickly if historians could start at where previous historians left off, rather than being forced to follow in the exact same footsteps that were already made by earlier scholars (step 1: look at secondary bibliographies and footnotes, step 2: go to the exact same sources in the archives that they did, step 3: take notes on the exact same sources in the archives that they did, and only then can you start doing something original). And this might even let undergrads in history start doing useful research much sooner in their careers. Making electronic versions of these sources is important, but just as important is to process the information into data for others. New arrivals can of course disagree with some of the coding or conclusions of their predecessors, but I'm sure that much of the processing will remain uncontested (e.g. Who, When, and Where). And if it is contested, History would also benefit from the transparency of the methodology used to interpret the documents - we talk all about historiography but very little about methodology. The ideal electronic source would allow a personalized keywording feature, so that you could add your own coding as well as comments on the coding of others. In fantasyland, every historian would fully examine the context of every document (in its entirety) that they use, but this is simply unachievable in reality, assuming we don't want to give up macro-historical research, and our projects should be moderated to what would work best for most historians. Robert Darnton's discussion of the future of the e-book ("The New Age of the Book" 3/18/1999 at www.nybooks.com) gives an excellent model: a pyramid made of progressively focused layers, so those with only a passing interest can skim the top succinct layer (e.g. you've got to give a lecture on the topic), while others more intimately interested in the project can delve deeper into layers focusing on themes, documentation, methodology, debates over the work, etc. My vision is of large-scale historical theorizing (at the very least, across several centuries or several countries) that could be based on the entire universe of historical knowledge represented in databases, rather than being limited to the language, period and regional horizon of a single historian, as is too often the case today. The questions historians ask about the past may differ from generation to generation, but most of the evidence they rely upon is relatively fixed.

I initiated a brief thread on RDBMS vs. XML on the E-DOCS listserv at the beginning of December 2001, setting out my impressions and asking for feedback. The results confirmed my belief that XML is (currently at least) for research projects with big bucks to spend on data entry and software programmers who are interested in making their research widely-available to many scholars at the same time (e.g. over the Web). For those interested, they can check the discussion logs at H-Net. My database is designed for individual scholars with their own personalized research who cannot rely on such significant funds. A number of examples have been mentioned on the Humanist listserv c. 12/11/2002.

8a) An important related issue that must be addressed is the Web-centric focus of current digital history. While this is not necessarily inherent in markup languages' structure, nor is the Web necessarily anathema to scholarly research (it was created at CERN to facilitate scientific research after all), most of the discussion of webs and digital history, as well as many of the examples available online, only go as far as to put their sources up online and make them searchable. However, I have seen little discussion thus far as to how these resources are to be best incorporated into broader scholarly research - in large part this is a function of who is providing these documents, usually archivists and librarians who must maintain the original and focus organization and provenance of the sources. Usually, however, historians have to combine and compare documents from different provenances. Even with various critical electronic editions online and the many impressive historical website projects available to date, it is unclear how they can be analyzed together as part of a larger whole and still retain the advantages that accrue from their digital nature. Instead, most websites with historical content seem to be standalone, self-contained sites, making multimedia material on their site browsable and searchable, but not customizable, while sometimes their underlying data is even un-exportable. If you want to study the Civil War experience of two Virginia communities, the University of Virginia's excellent "The Valley of the Shadow" site at http://www.iath.virginia.edu/vshadow2/ allows you to search through their information online and explore the wide range of sources presented (or the correspondence of Victorian Women Writers, or Charles Darwin...). But could you write a scholarly research article in a peer-reviewed journal using just their website? Probably not, for even if peer reviewers would accept a heavy reliance on web-based sources, the most professional of websites is unlikely to include all the categories or methods of analysis that should be available with digital sources: can you, for example, perform simple calculations, such as adding up the number of casualties (by rank) these two counties suffered in the Civil War based off their hospital records, or compare the average number of letters written to X in years A through D, not to mention more complicated quantitative and qualitative analyses? Only very sophisticated search engines will allow this (any examples would be greatly appreciated). And what if you want to compare the Virginia experience of war to that of six other communities in three other states, whose sources are available on other websites, in published primary accounts and in various state and local archives? We would have to replicate the entire retrieval and analytical process separately for each documentary collection, including the need to return later to the same sources for new investigations (historical research being an iterative process after all). Can we save our web-based queries and rerun them as we include additional sources from other collections? Can we exclude certain cases from our online queries because we consider them exceptional outliers? Can we recode the underlying web-based source tagging according to our own criteria, or add more keywords? The Historical Event Markup Language project (www.heml.org) hopes to use XML to develop standards whereby a variety of historical content on websites could be combined to create, among other things, chronologies and timelines, and while a good idea, it will not be backwards compatible (i.e. be able to process non-HEML sources), and many of the analytical features readily available in existing applications from the various humanities quadrants are, apparently, far in HEML's future, and are dependent on finding interested and knowledgeable scholars/programmers who are willing to write (most likely for free) applets to perform them.
My point: The individual historian still needs the ability to manage and manipulate all these documents whether they are available electronically or not, thus the need to collect all relevant sources in a single place controlled by the individual scholar. We cannot expect archivists, web designers and XML programmers to anticipate or emulate all the analytical methodologies we may want to use, thus scholars need to have unfettered access to the original data. Hopefully the digitized documents would be available to be downloaded (printing them defeats the purpose of having them digitized in the first place and would require re-entering the data, while even selecting the text off the website and pasting it into a program might lose any underlying mark-up), but then you still need to put them into something and fold them in with the many other types of sources you have collected, which is where a system like the Notes database comes in.
In contrast, most of the historical websites out there are highly-polished products geared towards presentation: great public history and useful educational tools, but not always intended for seriously scholarly historical research, certainly not maximized for use beyond its own subject matter (HEML might offer an opportunity to integrate such a variety of websites, but this would presumably require each site to reformat its material). Am I the only one to copy text from Websites and put them in a Word doc or in my own database? Rather than focus on production values (perhaps inevitable given the ever-increasing multimedia capabilities of the Web), my ideal research-oriented website would dedicate more of its resources towards making digitized raw sources and the basic coding of its data available for downloading (tables summarizing the data with links back to the underlying sources would be good too). Ideally, after manipulating the raw data, scholars could then upload their own additions to the website (including the results of the manipulations, new coding, additional sources, and new information on people and events mentioned in the documents), but the fundamental need is to be able to fully control digital sources just as if we had typed them into our own word processor or put marginal comments or post-it notes in our own books. See the EMWWeb for my Quixotic attempt to start such a beast. This is certainly possible in XML; the issue is whether the many web projects using XML-tagged sources will allow its potential to be realized by making the underlying data available for further scholarship. The Web will not fulfill its real promise for academia if it remains limited to bibliographic searches, archive access, email and the dispersed mass of materials available only in extract via a multitude of WEbsite-specific queries.

</rant> (a little XML joke)
 

But back to my database. There are of course a few limitations that it would be nice (although hardly requisite) to eliminate:

• A thesaurus which allows the orthographic variations of proper nouns (people's names and place names) to be accurately transcribed. Ultimately this is of little interest or use for me, and I use lookup tables to explicitly standardize proper noun spelling (so sue me). It might be possible to create such a thesaurus in Access; I haven't given it much thought. Perhaps someone would be interested in examining all the different ways in which early 18th-century people misspelled their compatriots' names - it certainly strikes me as odd, but then I have a unique name that's been butchered enough in the past, so you think I'd be hyper-sensitive to the issue. In any case, I'm sure some system could be created, but I can't think of anything lower on my "to do" list. I'm not a linguist or literature specialist, I'm a historian.

A number of improvements are needed particularly for the Notes memo-type field, which might depend on finding Microsoft or third-party utilities such as ActiveX controls. This is one area where HTML/XML-type systems have an advantage, but from my research they cannot come close to Access's many other features such as data entry aids (on the RDBMS vs. XML thread in E-DOCS, no one responded to my question whether XML software could replicate the many useful data entry features of an RDBMS for both textual and semi-quantified sources). Issues with memo fields include:

• Hyperlinks within the Notes field. It would be nice to make cross-reference links to other records, pop-up forms with info on the linked term, etc. Nevertheless, with my current design it's only one additional step to finding this information (e.g. by opening a form from the Open combo box, or by finding the letter referenced) and you can have pop-up forms to look up some info, so this is a useful but not critical feature in my humble opinion. Besides, that seems to be more of an issue for display, rather than analysis, and you could design a query or report that would display all of these interlinked documents together.
• Unlimited text lengths allowed. Admittedly, the way I have it organized, rarely will 32 pages of text all be dealing with the same keyword. If it does, you probably need to refine your keywords into more-detailed categories or else you've got really detailed documents or else your keywords aren't specific enough.
• A way to enter marginalia and indicate where in the main text it refers to. This feature could similarly be used for footnote text. I suppose you could rig up two subforms (one for body text, other for marginalia) linked together, but it seems quite clumsy and rather wasteful of space. Right now, when I remember, I use the <docketing> and <endorsement> XML tags in the Notes field. That's when I can even read the marginal scribbles.
• Ability to draw and display images that are incorporated with the text. This is a shortcoming of most text-based software, notwithstanding Microsoft's rudimentary drawing feature in Word. This of course is another reason why in-class note-taking will remain paper-and-pen for awhile yet. Hopefully Palm devices will figure something out. But to be honest, I can count the number of in-text drawings I've come across in the archives on one hand, so it's probably not as high a priority for me. Once again: customize the database to your specific needs.

Bibliography

This is a very limited bibliography of just a few of the works and sites I've found most useful.

• For various higher ed website guides on note-taking, search under "taking notes," "history research note taking" or some such combination, in a Web meta search engine, such as www.google.com. Most of the resulting hits, however, are for college students, rather than more advanced researchers. One useful survey site I have recently happened upon is http://www.princeton.edu/~aiteachs/nttr/capturing.html.  For an incredibly comprehensive site listing oodles of software, see http://ist-socrates.berkeley.edu/~jaytate/software.htm.
• David L. Clark, Database Design: Applications of Library Cataloging Techniques. For a discussion of the Library of Congress classification system.
• C. Harvey and J. Press, Databases in Historical Research: Theory, Methods and Applications (London: Macmillan, 1996).
• Several relevant papers in Terry Coppock, ed. Information Technology and Scholarship (British Academy, 1999).
• Susan Hockey, Electronic Texts in the Humanities: Principles and Practice (Oxford University Press, 2000).
• Also see the numerous journal articles in the journals History and Computing, Computer History Review and the more intimidating Computers and the Humanities. You'll notice that these are usually monopolized by either the quantitative, qualitative or literary quadrants of the humanities quadrangle.
• One example of a medieval prosopographical database is  K.S.B. Keats-Rohan, "Historical Text Archives and Prosopography: the COEL database system." History and Computing 10, no. 1-3 (1998?).
• You might also consult Alison Gaukroger, "Databases and Historians: A Problem of Mutual Incomprehension." History Computer Review 14, no. 1 (1998), a very brief overview which is more intended for project managers of historical databases. For a good overview of some of the issues involved (targeted at History Museums), see Jan Ballard, et al, "Computerizing your Collections Records," AASLH History News, 56(4) Autumn 2001, technical leaflet after p. 16.
• For brief mention of the late medieval and Renaissance parallel tradition of parsing texts into smaller segments (e.g. commonplace books), see Leah Marcus, "The Silence of the Archive and the Noise of Cyberspace," in  N. Rhodes and J. Sawday, eds. The Renaissance Computer: Knowledge Technology in the First Age of Print (Routledge, 2000). For an interesting discussion of how Renaissance scholars compared texts (including the bookwheel!), see the first part of Lisa Jardine and Anthony Grafton, '"Studied for Action": How Gabriel Harvey Read His Livy,' Past and Present, 129 (1990).
• Several of the texts on qualitative data analysis (QDA) and qualitative methods (QM) and research offer a variety of methods used in the social sciences (one of the quadrants of the humanities rectangle). You might find these helpful as well.
• For an interesting comparison of the utility of paper vs. digital media, I'd strongly recommend Abigail J. Sellen and Richard Harper, The myth of the paperless office, (Cambridge, MA: MIT Press, 2002). They conclude, among other things, that paper offers greater flexibility in the reading process (particularly the expanded spatial and simultaneous note-taking aspects) and that computers offer greater flexibility in the writing process. I would personally like better annotation features in this existing database - I've tried to replicate this aspect by allowing multiple pop-up forms so that I can compare side-by-side several letters on the screen at one time. In the future, electronic paper will solve that problem (projected to be on the market in 2010!).
• For a useful discussion of the MEP XML standards used by several historical e-collections, see adh.sc.edu/MepGuide.html. Unfortunately, this long document illustrates exactly the problem I've been having with the creation of this database: most written works are from a librarian or archivist perspective, which is interested almost exclusively with the presentation of the document to the public (bibliographical and typographical aspects). What historians would DO with these documents, such as keywording and analysis, is rarely discussed. A better example of XML aimed towards historians in particular can be found at the Historical Event Markup Language website (www.heml.org), although this appears to be still in very early stages and is again devoted primarily towards displaying data automatically in a variety of formats.
• For a brief discussion of RDBMS vs. XML in early December 2001, check out the E-DOCS discussion logs I initiated at H-Net. In early December 2002 the Humanist listserv also discussed the state of software. See also the helpful discussion comparing XML and relational databases by R.P. Bourret at http://www.rpbourret.com/xml/XMLAndDatabases.htm
• For a very interesting site on Exploring and Collecting History Online, see echo.gmu.edu.
• The History Data Service has some good information on digitizing history at http://hds.essex.ac.uk/index.asp, its parent at http://www.data-archive.ac.uk/home/. Required reading is their paper on "Digitising History: A Guide to Creating Digital Resources from Historical Documents" at http://hds.essex.ac.uk/g2gp/digitising_history/index.asp. There are also several good case studies to read (e.g. the Gloucester Portbooks and the Early Medieval Patronage Project) at ahds.ac.uk
• For an interesting look at the origins of the modern information age (i.e. the 18th and early 19th centuries), see Daniel Headrick, When Information Came of Age: Technologies of Knowledge in the Age of Reason and Revolution, 1700-1850 (Oxford: Oxford University Press, 2000), particularly chapter 5 on Storing Information. I'll bet Enlightened bureaucrats wished they had a database like mine!
• For a discussion of search techniques, see http://www.iath.virginia.edu/~jmu2m/sdl.html.
• For an introductory description of analyzing social and interpersonal networks for historical research, see Katherine Patterson's "The 'Anna Jameson and her Friends Database': Mapping Anna Jameson's Associative Links with the Victorian Intellectual Community," TEXT Technology, at texttechnology.mcmaster.ca/patterson.html. The website implementing this can be found at http://delos.lib.sfu.ca/projects/VWWLP/. There is a large amount of literature available (including software such as InFlow) on Social Network Analysis, with lots of fancy network/node graphing.
• For the early stages of a handwriting search engine, see Amherst's work at http://www.umass.edu/umhome/news/articles/7683.php.

I presented a paper on the panel "Databases and the Future of French History" at the November 2000 Western Society for French History conference (at UCLA). Unfortunately, I didn't learn much new there.

Various websites and usenet groups can also be helpful, although you shouldn't expect an expert to create your database for you (at least not without paying them for it first).

I would warn you to be very careful of older discussions (i.e. several years old) on databases, particularly regarding their recommendations for software (for example, someone posted an incorrect warning about Access not handling text fields over 256 characters on the RDBMS vs. XML thread). Although most of the theory of good database design (and implementation for historians) hasn't changed much, database software has changed immensely in the past five years, allowing me to do things in Access that were possible only with "textbases" several years ago. So don't automatically think you can only use textbases or "hypertext" for what you want to do.

Most computer use in history/the humanities seems to be either hard-core quantitative tools on the one hand or qualitative/literary analysis on the other (the humanities quadrangle). Therefore, none of the works that I've read (including most of the XML literature) spend any time talking about the most basic of databases that every researcher can use, like my note-taking one, which is why I created these pages. But if you want to know about how to use databases to examine wills, port records, census data, the structures of your texts, etc., then these works on straightforward relational databases are for you. For database manuals more generally, you have to apply the business concepts/examples discussed in the manuals to your, often quite different, needs - not always an easy thing to do.

My immense thanks must go to my programmer/analyst wife, whose knowledge of VBA and database theory is far greater than mine, and who has given much of her time working on the coding for some of these advanced features.

Last Updated May 30, 2005 3:43 PM