There are two things here:
1) Using your custom classes as to define types of process variables, and using them in your script tasks,
2) Your service node handler implementations
for 1) you need to add your jar files to Guvnor (as a jar model). Your package when built will be built using the class definitions of your model jars. You also will need to make those jars available on your application classpath (since the .pkg file will not include them). If you place them in the $installer/depencencies dir they will be placed on the classpath for you (follow instructions below on the ant calls).
for 2) best thing is to use the jBPM 5 installer. In this case you can simple place your dependency jars and jars that include handler impls in the $installer/dependencies directory and run
ant clean.demo
ant install.demo (or ant install.demo.noeclipse)
and start.demo (or ant start.demo.noeclipse)
Hope this helps.