Sounds like the cross-sectional area of the groove might be more important than perfectly parallel sides, etc. If so, might engrave it?
The skiving suggestion could also work. No matter what you do, chip evacuation and keeping the plastic from melting may not be easy. A saw, running slow with water coolant, might be best if you can get one made or re-ground to .004". With a 10:1 depth to width aspect ratio and such thin sections in acrylic, this thing is also going to be fragile. Most any university prototype shop should be able to give engraving, skiving, and saw slitting a try to see if you're happy with the finished channels.
I do wonder about acrylic as a heat exchanger, but will assume there is a reason for this. To see flow???
Could also be you could laminate this out of .004" sheet, every other sheet displaced .04 and glued. The film would likely have better structural integrity than engraved/skived/sawn acrylic. You could probably find both a clear film and a double-stick tape near .004".
FWIW, microscope cover slips can be had near .004" thick and a stack of them is surprisingly strong. Be a total pain, but a stack of those, every other one displaced, could be embedded in epoxy at one end. Much like gage blocks, cover slips tend to stick together on their own. Even easier to laminate out of a metal, perhaps alternating with a film or clear double stick tape if you want some visible access???
You could also make a mold with .004" x .04" strips of aluminum tensioned across the bottom. Cast acrylic over it. Dissolve out the aluminum.