KitzKikz: MigratePVCStoSubversion

KitzKikz: MigratePVCStoSubversion	RecentChanges PopularPages SearchPages
Home \| Trail - SogudiIssues_2006... > SogudiIssues_2007... > SogudiIssues_2007... > SogudiFAQ > MigratePVCStoSubv...

Migrating from PVCS to Subversion

I found myself needing to migrate 17 years worth of a corporation's PVCS repositories to Subversion (svn).

There is an excellent and free tool at http://www.polarion.com/products/svn/svn_importer

The included documentation is good, but not great. Searching the web for more answers did not turn up much. So, I'm sharing my notes below for the next lucky soul. Maybe, when the motivation strikes, I'll write up a "How To".

Svn-Importer Executes in Phases

pcli listversionedfiles
pcli vlog -- Parses the vlog to learn about all revisions, properties, comments, and version labels. The parser is pretty good unless your developers have taken to using a series of dashes in their comments. I suggest running the vlog yourself beforehand, searching for all such comments (see perl below), and modifying them in the PVCS GUI. Look for any lines containing a series of dashes not immediately followed by a "Rev ..." line of text. Simply remove any series of dashes in a comment.
pcli get -- Does a get of all revisions of all files into a temporary directory, see note about disk space requirements.
Creates the dump files while running checksums against the files pulled in step 3 (see config.properties "validate_checkouts").
Cleans up temporary files and exits (see config.properties "disable_cleanup")

Line Endings

Do the conversion on the same platform that your PVCS repository is hosted on. If your PVCS projects are hosted on a unix platform, then execute pcli, svn-importer, and svnadmin on unix. They don't all have to be on the same machine, just the same flavor of OS. While all these tools are cross platform, that does not mean you should do the conversion across platforms. Once you have the final subversion repository, you may move that to wherever you like.

Disk Space

This process needs a lot of disk space, three to six times the size of your PVCS repository. Because:

The original PVCS file, probably in a compressed "delta.d" file. Call this 1 disk unit.
Upon receiving a "get" request, PVCS expands each revision out to full size. These are the files like "1.1.3u" next to the delta.d inside the archive directories. These do not go away, even though PVCS is technically done with them. This is 1.5 disk units.
Svn-importer keeps each revision of each file in it's temporary area, full size, until the dump files are complete. This is 1.5 disk units.
Even though svn-importer places deltas in the dump files, they'll take up about as much disk as your original PVCS repository. Another 1 disk unit.
After svn-importer has created the dump file, you'll still need to load them into a new svn respository using svnadmin. Another 1 disk unit.
Total ~ 6 disk units

If your PVCS repository is 5Gb, you'll need 30Gb for the conversion. Don't skimp on this. It's not worth the headache. In fact, double it just to be safe. Beg, borrow, steal the space. Bribe your unix and storage admins if you have to.

For example, I had one versioned file in PVCS containing over 500 revisions in a 1Gb compressed delta. During import, PVCS created 4Gb of the "?.?.?u" files. Svn-importer needed 4Gb to get the revisions out of PVCS plus 1Gb to generate the dump file. After loading the dump file into subversion, the result of this single file was 1Gb. (1+4+4+1+1=11 ~ eleven times the disk space).

Memory

In phase 2, svn-importer parses the vlog completely in the java heap. A large project with a lot of revisions may cause the heap to overflow. I modified the "JAVA_OPTS=-Xmx100m" in run.sh to 500m. I didn't have any problem with the heap after that. I've heard of some people needing up to 64Gb of memory (see links near the bottom of this page).

Log Files

Svn-Importer does not manage to write to it's log file. I suggest capturing the output of the command using redirection. Also, use the property "disable_cleanup=yes" while you troubleshoot the conversion process. You may have to run the pcli commands (without the -q) yourself to find out what caused svn-importer to stop. The pcli error messages are not captured by svn-importer anywhere.

Promotion Groups

Svn-Importer does not transfer PVCS Promotion Group information. Instead, use pcli, before migrating, to recursively assign version labels to match the promotion groups (see script below). Svn-Importer will create tags for these version labels. Then, in subversion, you can switch these tags into branches (example below). Fixed to automatically create tags for promotion groups, see update below.

PVCS Attributes vs. SVN Properties

The config setting "pvcs.import_attributes=yes" only partially works. The importer creates the properties for svn:keywords and svn:eol-style, only, which is probably enough. However, it doesn't seem to know about the $Header$ keyword. The properties are added to the first revision only. Later revisions, tags, and branches do not get these properties. As such, this feature is virtually worthless. The suggestion in the comments to use auto-props won't help, either, because there are always exceptions to such generic actions. Fixed to place properties on all revisions of a file and include the $Header$ keyword, see update below.

One-Shot vs. Incremental

The config setting "use_file_copy=yes" will only allow for one-shot conversion. You cannot perform incremental updates afterwards. For incremental flexibility:

Set use_file_copy=no
Perform a full action: run.sh full path/config.properties
Perform as many incremental actions as you want: run.sh incremental path/config.properties

An incremental execution takes just as long as a full because in both cases, svn-importer performs a "pcli get" of all revisions of all files, regardless if they are included in the incremental or not. I'm certain this is a bug. I have one repository that takes 32 hours for svn-importer to scan, making it impossible to run a daily incremental update. Fixed to pre-fetch only the files needed for the incremental portion, see update below.

Some Helpful Shell Scripts

Find comments in vlog that contain dashes

perl -e 'while(<>){next if $_ !~ m/----/; $x=<>; print "$.\t$x" }' < vlog.txt | grep -v Rev

Assign Version Label to Promotion Group

for pgroup in Development-1 Development-2 Integration UAT Production
do
   plabel=$(echo $pgroup | sed 's/-//g')
   pcli run -y Label -pr''project_name'' -v"svnbranch_$plabel" -r"$pgroup" -z /
done

Copy a tag to a branch in Subversion

svn copy ''svn-path''/tags/svnbranch_Production ''svn-path''/branches/Production

Send Email When Process Completes

while pgrep -P ''pid'' > /dev/null
do
   sleep 60
done
echo "done" | mail -s"svn-importer" ''sms.address.of.your.phone''

Update: Bugs Fixed

I managed to get hold of the source code for svn-importer and modify it to fix some of the issues I report above. You can download my modifications at http://kitzkikz.com/files/svn-importer-pvcs-mods.zip , however you're on your own when it comes to implementing them. Use at your own risk.