All my comments are assuming English units: inches and threads per inch (TPI). Metric would require separate considerations.
1) How do I figure out the geometry of the setup? Specifically, how many teeth does the engagement gear need? And how many divisions on the display dial?
The threading dial is just a measure of the carriage's travel along the lathe axis. So the major divisions are usually marked 1, 2, etc. which mean 1 INCH, 2 INCHES, etc.
The math that it uses must include the pitch of the lead screw, in your case 6 TPI. The first thing this TPI is used for is calculating the number of teeth on the threading dial gear. Any thread that has a whole number of threads in an inch, a whole number of TPIs can be synchronized at those major, numbered divisions. So for whole number TPIs, any whole number multiple of six teeth on the gear will work. For threads with a 1/2 in their TPIs a full two inches will synchronize them. So your threading dial's gear would need multiples of 12 to span that 2" of travel. Likewise, for the odd threads with 1/4 or 3/4 in their TPIs, a full four inches would be required to synchronize them. So multiples of 4 x 6 for a full four inches of travel would be required for the number of teeth on that gear.
Since English/inch threads only VERY rarely have any other fractions in their TPIs, most thread dials on English/inch lathes will have gears with four times the lead screw pitch or, in your case, 4 X 6 = 24 teeth. And they would have major divisions of 1, 2, 3, and 4.
Now those major divisions will work for virtually every English/inch thread. But waiting for a or the correct numbered division to come around takes time and time can be money. So additional, minor divisions can save time for many threads. As an example, on your lathe a 6 TPI thread can be synchronized on any 1/6th of that inch. So an optimum thread dial would have five minor divisions between those four major ones (in all, six per inch). And while cutting a 6 TPI thread the lead screw could be engaged at any of the 24 divisions. This would also work for any multiple of 6 TPI, like 12, 18, 24, etc. TPI. But an 8 TPI thread would synchronize at the half inch points as well as at the major divisions. So there would be 8 points where the half nuts could be engaged: the four major, numbered divisons and the four half divisions. Those 1/2" divisions should/would have longer lines than the other four minor divisions.
Thus a threading indicator for your lathe would have a 24 tooth gear and a dial with 24 divisions, 4 major divisions which are marked and 20 unmarked, minor divisions. Which divisions to use for any particular thread would depend on which of these 1/6" increments would allow synchronization.
... the lead screw is 1 1/8" diameter and 6 threads/inch.
The diameter and TPI of the lead screw is used to calculate the helix angle of the threading dial's gear if the dial is to be mounted vertically. Assuming a standard Acme thread, it goes like this:
PD (Pitch Diameter) = OD - linear thread pitch
PD = OD - 1/TPI
or in your case:
PD = 1.125" - 1/6" = 0.9583"
HAG (Helix Angle of Gear) = arcTan(linear thread pitch / (π x PD))
HAG = arcTan((1/TPI) / (π x PD))
or in your case:
HAG = arcTan((1/6) / (π x 0.9583")
HAG = arcTan(0.0553)
HAG = 3.169°
2) Are there tables that tell how to use the dial? Where I trained, the lathe had a sticker with a table that told things like (making this up off the top of my head): If threading 2,4,6,8,12,16,32 TPI, then start at any mark. But if 3,6,9,15 TPI then start on even marks, and so on. I am guessing the exact details of relationship between the threaded TPI and the thread dial is given by some formula. Does anyone know how to figure this out?
I am sure there are tables. The hitch here is the table depends on both the number of divisions and/or gear teeth on the thread dial AND on the TPI of the lead screw. A problem could be created if the design of the thread dial is not what is a standard one for your lead screw's TPI. I started talking about this above.
But here is how I would derive the table for your 6 TPI lead screw and a 24 division thread dial as I described above. First, as I said above, you want divisions that make sense with your 6 TPI lead screw and that means six minor divisions per each major/numbered one inch division. And all TPI thread dials I have ever seen used a four inch total span to allow for TPIs with 1/4, 1/2, and 3/4 fractions so I am using that as well.
With those considerations in mind, I created an Excel worksheet and first listed the thread pitches starting with 1 TPI. Then the following columns show which pitches can be synchronized with the different whole number and fractional divisions on that dial. That includes divisions of halves, thirds, and sixths, which are all the possible ones.
But that table does not include the TPIs with fractions so I added two additional entries at the top-right for the 1/2 and the 1/4 & 3/4 fractions.
Finally at the bottom-right I created a table of rules from them. The rules are listed in the order in which you should choose the first one that applies. This order is strictly necessary for the fractional TPIs as none of the other rules will work properly for them. In the case of the last four rules you may choose any rule below the first one that applies: it will just take longer to synchronize on some passes so the overall threading process may take longer.
This is just an image of the spreadsheet here. I tried to keep the resolution high enough to be read. I will upload it to my Drop Box and add the link below for anyone who wants the actual Excel workbook.
Shared with Dropbox
I have nothing to add on your third question.