Solid Composite Cores
Side note: Again, this is a product with which I am familiar. There may be alternatives under different brand names, but I am not aware of them.
A solid alternative to Urethane foam is Coosa Board. Now we're talking about something that's appropriate for transoms. Coosa Board is basically micro thin layers of fiberglass laminate built up to be a specific thickness. It can be purchased from the factory in varying densities and weights depending on its intended purpose. It is perfectly compatible with fiberglass, it will never rot, and it has fantastic compressive and shear strength.
Coosa Board is a suitable replacement for any wood core on a boat, from transoms to stringers, to bulkheads to floors. What's more, it's compatibility with resin and its imperviousness to rot make it the only true permanent solution to many of the problems associated with boats, both old and new.
So what are the downsides to Coosa Board? The same as all genuine improvements to the marine repair industry. It's expensive. It's prohibitively expensive. It's unrealistically expensive. But the price has come down since its introduction to the market, so that gives me hope.
Even with its high price, I always recommend it. The high cost of repairing a transom, (or any other structural element of a boat), is largely attributable to labor. Using an expensive core material like Coosa Board might change the total bill from $2,000 to $2,200, but in that $200 difference is the peace of mind that comes from knowing the solution will probably outlive you.
Unfortunately, Coosa Board is not a panacea for all boats. Its rigidity makes it unsuitable for curved transoms. Additionally, it is hell on tools. The manufacturer states that it can be cut and shaped with standard woodworking tools, but that's only true if you plan on throwing the blades away at the end of the project. Blades will dull down to the nub in a hurry when working with this product. Still, it's worth it in my opinion.