If you read this, you will have a better idea of what I've done and what I know about the behavior of our tool. I think it's somewhat worth while to read this, just to get an idea of a couple things. Just trying to contribute. I'll test more in this manner over Thanksgiving break (but probably not a lot more, Compilers has my priority.) ------------------------------------------------------------------------------------------------------ Testing outside of a local repository. We (I) ran the tool using a public Subversion repository at zope.org. In particular, calls were made specifying repository locations at "svn://svn.zope.org/..." We're not sure of the expected results because we haven't investigated the actual files in this repository to see if the turmoil metrics are correct. We also don't know exactly how far back the revision numbers in certain repository locations (like ".../trunk/utilities") go . From these few runs though, the turmoil metrics look believable, and in one case (the first test below), we manually "svn catted" a file and ran 'file_turmoil' on it, and got the same result. So again, this turmoil output looks believable. The point of these tests on the tool includes some of everything (I think): - Test how it acts on something other than a local repository (on the machine). Not using the "file:///" notation, in other words. - Test its performance. How long does it take with how many files when it has to go out to the web/a remote repository? - Test how it acts on a repository with multiple directories. We seemed to have a problem/defect with a local repository when the specified repository location was multiple directories deep. Note that the file lists for using a remote repository (zope.org) were created using the "svn list..." command, usually multiple times, and modified manually when necessary. Determining which revisions to pass to the tool was done by observing the revision information via the Web interface to the repository at http://svn.zope.org. (This also can be done "svn info..." I believe.) ------------------------------------------------------------------------------------------------------ *** Test #1 - Nominal call to the tool passing in 12 files and a specific remote location. *** Call: "perl run_turmoil -p product -i zope_util_files.txt -o svn://svn.zope.org/repos/main/Zope/trunk/utilities@40000 -r svn://svn.zope.org/repos/main/Zope/trunk/utilities@40218" Expected Output: Not exactly sure. Tool should report the turmoil nominally. File List: 12 file names. They are single files with no path included. (Ex. "README.txt", not "/doc/README.txt") Results: The tool ran nominally. It took about 30 seconds, and the entire time, Internet activity was being shown by the "Local Area Connection" Windows tool. All the intended files were created. Turmoil was reported for three files, including one that was new across the revisions 40000 and 40218. *** Test #2 - Nominal call to the tool passing in files that are both in the specified location, and one directory away from the specified location. (i.e. The file list includes file names of the form "tests/testUndo.py".) Call: "perl run_turmoil -p product -i zope_pythonlib_files.txt -o svn://svn.zope.org/repos/main/Zope/trunk/lib/python/App@30000 -r svn://svn.zope.org/repos/main/Zope/trunk/lib/python/App@40218" Expected Output: Not exactly sure. Tool should report the turmoil, but given the directory structure (some files are one directory deep), it's hard to say what we expect. Some result may not be correct. File List: 16 names, including the following, with the number of each listed. - Files that reside in /App. In the file list, they look like "config.py." 3 total. - Files that are in /App/dhtml. In the file list, they look like "dtml/activity.dtml". 3 total. - Files that are in /App/tests. In the file list, they look like "tests/testUndo.py". 4 total. - Files that are in /App/www. In the file list, they look like "www/zopelogo.jpg". 3 total. - The names of these three directories. In the file list, they look like "www/". 3 total. Results: The tool ran without crashing or hanging. It took ~ 42 seconds, and again, the entire time, Internet activity was being shown by the "Local Area Connection" Windows tool. The intended files were created. However, some files (three image files) were supposed to be "ignored", but were not. Instead, the "ignored" file list and turmoil output files were both empty, and these files were in the "kept" file list and turmoil output files instead. Turmoil measurement other than zero was reported for three of the files in the "tests" directory, as these files were marked as new. Turmoil was "0" for the rest of the files, and the directories (listed in the file list) were reported as "file not found" in the turmoil output file. So the directory strucutre - for right now - appears to have something to do with the results. Some manual diffing of the first file in the file list causes me to believe it didn't change between the given revisions, but does between revisions 20000 and 40218. First I used "svn diff" on these revisions with this first file, and then our tool using file_turmoil on two svn catted files at these revisions. The svn diff output was hard to read, and our tool did report turmoil. This leads to Test #3. *** Test #3 - Same as Test #2, but with an older revision. Call: "perl run_turmoil -p product -i zope_pythonlib_files.txt -o svn://svn.zope.org/repos/main/Zope/trunk/lib/python/App@20000 -r svn://svn.zope.org/repos/main/Zope/trunk/lib/python/App@40218" -- Notice the "20000" instead of "30000", as in Test #2 Expected Output: At least to report the turmoil on the file I manually determined had changed. Not sure otherwise. I imagine the file lists won't get created properly again in the same way as in Test #2. File list: Exact same as in Test #2. Results: The tool ran again without crashing or hanging. It took ~ 45 seconds, with the same Internet activity being shown the whole time. The call to svn cat within the tool caused more "could not find" otuput lines with this call because many of the files were not found in revision 20000. This means that they SHOULD be reported as new in the turmoil output file. The file I manually investigated had the correct turmoil reported with this run. And indeed, all the files that svn cat couldn't find across both revisions were marked new in the turmoil output file. This turmoil looks right to me. More tests should be done, obviously, but I can't see what our defects are in regard to the directory. I'm starting to believe that it has to do with a repository being local (on the machine you're running the tool on), and I'm not sure how much Bruce will be doing this anyway. The results of Test #3 look good - testing can pick up here (i.e. Test #4, Test #5, etc.) and include multiple directories deep, and more complex directory structures, which the Zope repository certainly provides. The one defect I think we do have is the reaction to files that are supposed to be ignored. I don't know that anything I've done has resulted in anything but an empty "prodcut.ignored" file list. Perhaps Travis would be best-suited (no surprise) to formally describe this defect, although it does seem a little straightforward: Tool does not properly create "ignored" file list; all files are in the "kept" file list and turmoil on these files is in the "kept" turmoil output file. I will note this defect in the Release Notes, and will also note the behavior on a LOCAL REPOSITORY with the directory thing - because far as I can tell (although I haven't tested this extensively by any means), it works alright on a remote repository.