How do I write a SCons script with hard-to-predict dynamic sources? -

i'm trying set build system involving code generator. exact files generated unknown until after generator run, i'd able run further build steps pattern matching (run program on files extension). possible?

some of answers here involving code generation seem assume output known or listing of generated files created. isn't impossible in case, i'd avoid since makes things more complicated.

https://bitbucket.org/scons/scons/wiki/dynamicsourcegenerator seems indicate it's possible add additional targets during builder actions, while build run , list generated files, build steps introduced don't run.

https://bitbucket.org/scons/scons/wiki/nondeterministicdependencies uses scanners add build steps. put glob(...) in scanner, , succeeds in detecting generated files, files inexplicably deleted before runs dependent step.

is use case possible? , why scons deleting generated files?

a toy example

source (the file referenced in sconscript)

an example generator, constructs 3 files (not known build system) , puts them in argument folder

echo "echo 1" > $1/gen1.txt echo "echo 2" > $1/gen2.txt echo "echo 3" > $1/gen3.txt 

sconstruct

just sets variant_dir

sconscript('sconscript', variant_dir='build') 

sconscript

the goal to:

1. "compile" generator (in toy example, copies file called 'source' , adds execute permissions
2. run "compiled" generator ('source' script generates files)
3. perform operation on each of generated files extension. example runs "compile" copy operation on them (for simplicity).

---

env = environment()  env.append(builders = {'examplecompiler' :                        builder(action=[copy('$target', '$source'),                                        chmod('$target', 0755)])})  generator = env.examplecompiler('generator', 'source')  env.append(builders = {'generatorrun' :                        builder(action=[mkdir('$target'),                                        '$source $target'])})  generated_dir = env.generatorrun(dir('generated'), generator) 

everything's fine here, targets explicitly known build system ahead of time.

---

attempting use block of code glob on generated files causes scons delete (!!) generated files:

for generated in generated_dir[0].glob('*.txt'):   generated_run = env.examplecompiler(generated.abspath + '.sh', generated) 

---

attempting use action update build tree results in additional actions not being run:

def generated_scanner(target, source, env):   generated in source[0].glob('*.txt'):     print "scanned " + generated.abspath     generated_target = env.examplecompiler(generated.abspath + '.sh', generated)     alias('toplevelalias', generated_target)  env.append(builders = {'generatedoperation' :                        builder(action=[generated_scanner])})  dummy = env.generatedoperation(generated_dir[0].file('#dummy'), generated_dir) alias('toplevelalias', dummy) 

the alias operations suggested in above dynamic source generator guide, don't seem anything. prints execute , indicate action gets run.

running build pattern on special file extensions possible scons. c/cpp files preferred scheme, example:

env = environment() env.program('main', glob('*.cpp')) 

the main task of scons, build system, minimum amount of work such targets up-to-date. makes things complicated use case you've described above, because it's not clear how can reach "stable" situation no generated files added , targets built. you're better off using simple python script directly...i don't see how using scons (or other build system matter) mission-critical in case.

edit:

at point have tell scons created files (*.txt in example above), , tracking dependencies properly, list of *.txt files has complete. task of emitter within scons, responsible returning list of resulting target , source files builder call. note, these files don't have exist physically during "parse" phase of scons. please have @ answer scons: create late targets , goes more detail.

once have proper emitter in place (see https://bitbucket.org/scons/scons/wiki/toolsforfools , "using emitters") should able use glob('*.txt') call, detect , track created files automatically.

finally, on our page "talks , slides" ( https://bitbucket.org/scons/scons/wiki/talksandslides ) can find talk pycon fr.2014, "why scons not slow", explains shortly how scons works internally. might helpful in understanding problem better , coming full solution.