Terry's TMG Tips

Splitting Lumped Sources

This page created 24 Apr 2011

Version note: Applies to TMG 8 & 9

This article describes how to convert from so-called "lumped" Sources to the "split" style. For example, one might have created a single Source in TMG for all census records in a given county and now wants to convert to a system with one Source for each household. Or one has created a single Source for all e-mails from a correspondent, and now wants to have one Source for each e-mail. My article on Using the Split CD to “Lump” Sources describes one way of creating such "lumped" Sources. This article can help if one has used a "lumped" method and has now decided, as I have, that converting to more discrete Sources is preferable.

There are various methods for making such a conversion. The method I describe here uses John Cardinal's TMG Utility to significantly reduce the effort required. John's program is designed to assist in many tasks in managing your TMG Project, including the one described here. The program is "donorware," meaning it is free. But if you find it useful, John asks that you make a donation in his name to his favorite charity. I would suggest that if you have made a donation in the past and still find TMG Utility is helping you in the use of TMG, especially if you are making use of features such as the one described here which have been added by recent updates, that you consider making a new donation.

The Conversion Process

The basic process in converting a group of Source from "lumped" style to "split" style is straightforward:

  1. Identify the Source to be split.
  2. Create new Sources for each of the portions of the existing Source you want to now stand alone. For example, if you have one Source for the census in each county, you might what to create a new source for each household you cited. Typically the details needed to create the new Sources will have been recorded in the Citation Detail (CD) of the various Citations to the old Source.
  3. Find each citation that is to be moved to the new Source, change it's Source Number, and remove the information in CD that has been transferred to the new "split" Source, while retaining any notes about that source which are specific to a given Tag.

This last step can be daunting. If there are only one or two citations to be transferred to the new Source, doing this manually in TMG is probably the easiest method. But if there are many, as is often the case with census Sources, for example, doing this manually will be a massive task. The method described here substantially reduces that effort. Nevertheless, converting a substantial number of Sources will take some time, and requires concentration and focus to avoid introducing errors into your database.

The process is described below in more detail.

Creating a Checklist

I find it essential to create a checklist of the Sources and Citations to be converted. The best way I've found to do that is to use the List of Citations report, sent to a text file, to use as a working checklist. Having the list in a text file enables you to copy text strings and paste them into fields in TMG Utility during the process. I create my check list with a List of Citations report using this filter:

Source Number   = Equals [?]

The term "[?]" for the value, rather than entering the Source Number directly, causes TMG to prompt you for the Source Number each time the report is run, thus avoiding the need to change the Filter each time the report is run for a different Source. I've found the Indented List Type (under Options) to work best, as it sorts the list by the contents of the CD field. If your CD entries contain the principal information (such as name of head of household in a census source) first, the groups the citations as they will be used for the new "split" sources. Here is an example:


For use as a checklist, note that I made several edits to the report after it was created by TMG, as marked by the red arrows in the screenshot above:

  1. I set off the groups of citations to be converted to each new Source by adding blank lines and strings of equal signs. I identified the groups by examining the name of the subject in the CD text, in this case the subject of a draft card.
  2. As I added the new Sources in TMG, I recorded the new Source Numbers in the list for use during the conversion process.
  3. Some of my CDs contained strings long enough to wrap to a second line in the report. Since I would be copying a portion of the CD which is to be removed into TMG Utility, I removed the return and added back the space that had been removed by the report to the first Citation in each group.

After testing one or two conversions to be sure I understood the process, I found it helpful to work the process in batches. I created a number of new Sources as illustrated in the screenshot above, then converted that group before making more new Sources.

Use TMG Utility to Transfer Citations to their New Sources

The actual transfer of Citations to new Sources is in theory simple: find the Tag with the Citation, open the Citation, change the Source number, and delete any text in the CD that is no longer needed. The problem is that with potentially hundreds of Citations to modify, this would be a very time-consuming and error-prone process if done manually. The use TMG Utility makes the process considerably easier and more dependable.

If you have not done so previously, first download and install the TMG Utility.

Important: The changes described here are "major surgery" to your Project. Errors are possible. Make a backup of your Project before you begin.

Your Project cannot be open in both TMG and the TMG Utility. Before starting the Utility, close your Project in TMG, using the File > Close Project menu command.

Setting Up the Change Citation Parts “Rules”

Open TMG Utility, and use the File > Open menu command to open your Project in the Utility. Then select Citations > Change Citation Parts in the left panel (version 7.1 or later) to see this:

Empty screen

This function of the Utility requires the user to define "Rules" which tell program which Citations to change, and then what changes to make. As we will see below, defining these Rules is not difficult. Click the Set Rules... button to open the Rules Editor:

Rules Editor

Note that there are two parts to the Rules: Filters, which tell the Utility which Citation to change; and Actions, which tell it which changes to make. You create Rules by defining them with the fields below each of the two sections. Select a data element from the first drop-down list, then select an operator from the second drop-down. Finally, enter a value in the third field. In the screenshot above, I have defined the "old" Source Number whose Citations are to be changed. Once the term had been defined, enter it in the list by clicking the Add Filter or Add Action button.

If you need to edit an existing Rule term, select it and make the necessary changes. Then click the Update Filter or Update Action button to modify the term.

I found the following Rules, illustrated in the screenshot above, worked well for converting my lumped Sources:

Close the Rule Editor when you have finished with the creation of Rules.

Applying the Changes

Once the Rules are defined we are ready to proceed with modifying the groups of citations. We now can see the Rules we created in the Rules box on the main screen of the Utility:

Main Screen

To proceed with the first set of changes, make sure the "Log Only" box is checked. This allows you to see a log of what changes will be made so you can verify that they are correct before any actual changes are made. Then click the "Change Citation Parts" button to start the process. Since I used variables in the Rules, I was then prompted to enter text for those terms:


The first prompt is for the CD text to be used for the search Filter and also for the replace Action. I copied that from the checklist, as shown below.


Note that this entry does not include the final double bars and additional note, unlike the following one, which continues "...||shows date, age 27..." Since I remove that final double bars in a separate Action, I am careful to not include it in this text string.

Click the OK button on the Set User Variable 1 dialog box, and a second prompt will appear for the new Source Number. Enter that, and the Log will appear, with the notation "No changes were made because the Log Only option is selected" at the bottom.

Check the log carefully to insure that the desired changes are being made to the CD text, and that the correct new Source is specified. Compare the number of Citations to be changed with the number shown in the checklist. If there are fewer on the log than on the checklist, that generally means that some of the citations have a subtle difference in the CD text. Find them and mark them so they can be addressed in a separate run with a modified search string.

Once you are satisfied that all is well, uncheck the "Log Only" box, and click the "Change Citation Parts" button again. The same prompts appear, populated with your previous entries. Click the OK button on each prompt to accept those same entries. The changes will be made and log will be updated (removing the notation that no changes were made).

Processing Additional Groups of Citations

Initially it is best to open the Project in TMG and verify that the changes were made as expected (no need to close the Utility; just remember to close the project in TMG before making another run in the Utility). Once you have verified that the process is working as expected, you can make a group of changes without checking each one in TMG.

To change additional groups of Citations use these steps:

These steps quickly become rote, enabling a large number of changes to be processed in a reasonable amount of time. It is essential to take care to do each one correctly to avoid errors, as it is easy to miss one and potentially create a substantial number of erroneous entries.

ReigelRidge Home Terry's Tips Home Contact Terry



Copyright 2000- by Terry Reigel