Is it possible to define custom symbols that could be used in the lyrics staff or perhaps some kind of a special purpose staff below the standard staff? I guess there are really two questions here: can I add a custom staff, and can I define a set of custom symbols to be used on that staff, perhaps from a palette as notes and rests are inserted.
Yes, it's feasible. We used that mechanism to develop the "Figured Bass" script. It consists in an additional (invisible) staff at the bottom of the score and "Text effects" attached to invisible rests in that staff. These text effects themselves use a special character font we built in such a way the proper symbols appear. It' quite a complex and difficult process, however. If you are interested, please contact me on my private mail address, (see JP)