This is probably pretty tricky.
As a hint: it's going to be similar to combo custom field, but with something like parent_id to get the tree structure. In the front-end, you'll probably need to use something like context menus for building the tree (or maybe an 'add node' button etc). I've done some cool stuff with the Extjs3 tree before, so I'm sure what you want is doable. Also, you'll have to think about how you want to represent the data in for example the grid.