Introduction to the sstate.bbclass
Contents:
1) Introduction
2) Usage
3) Tasks to be sstated
4) The implementation of storing the sstate file
5) Re-usage of the sstate file
6) Pitfalls
1) Introduction
Stored the task's result and re-use it when re-building, seems like
ccache and ldat's rpm cache.
2) Usage
- inherit sstate
It has been inheritted by default, in meta/conf/distro/defaultsetup.conf
INHERIT_DISTRO ?= "debian devshell sstate license"
- setup the SSTATE_DIR, in meta/conf/bitbake.conf:
SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
The sstate file would be stored and re-used.
3) Tasks to be sstated
do_populate_sysroot
do_populate_lic
do_package_write_ipk
do_package_write_deb
do_package_write_rpm
do_package
do_deploy
4) The implementation of storing the sstate file
In meta/classes/sstate.bbclass:
21 python () {
...
45 for task in unique_tasks:
46 namemap.append(bb.data.getVarFlag(task, 'sstate-name', d))
47 funcs = bb.data.getVarFlag(task, 'prefuncs', d) or ""
48 funcs = "sstate_task_prefunc " + funcs
49 bb.data.setVarFlag(task, 'prefuncs', funcs, d)
50 funcs = bb.data.getVarFlag(task, 'postfuncs', d) or ""
51 funcs = funcs + " sstate_task_postfunc"
52 bb.data.setVarFlag(task, 'postfuncs', funcs, d)
53 d.setVar('SSTATETASKNAMES', " ".join(namemap))
54 }
Note the line of:
48 funcs = "sstate_task_prefunc " + funcs
51 funcs = funcs + " sstate_task_postfunc"
The two functions sstate_task_prefunc and sstate_task_postfunc are
used to store the sstate file.
The sstate_task_prefunc is used to preparing the environment, for
example, remove the manifest file in tmp/sstate-control.
And the sstate_task_postfunc:
426 python sstate_task_postfunc () {
427 shared_state = sstate_state_fromvars(d)
428 sstate_install(shared_state, d)
429 for intercept in shared_state['interceptfuncs']:
430 bb.build.exec_func(intercept, d)
431 sstate_package(shared_state, d)
432 }
And the sstate_package:
320 def sstate_package(ss, d):
...
373 bb.build.exec_func('sstate_create_package', d)
374
375 bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)
And the sstate_create_package:
439 sstate_create_package () {
440 cd ${SSTATE_BUILDDIR}
441 # Need to handle empty directories
442 if [ "$(ls -A)" ]; then
443 tar -czf ${SSTATE_PKG} *
444 else
445 tar -cz --file=${SSTATE_PKG} --files-from=/dev/null
446 fi
447
448 cd ${WORKDIR}
449 rm -rf ${SSTATE_BUILDDIR}
450 }
We can see that the sstate file has been stored by tar, e.g:
sstate-gzip-armv5te-poky-linux-gnueabi-1.4-r0-armv5te-2-079607fa7ece189eb79c856e9a16baa2_package.tgz
sstate-gzip-armv5te-poky-linux-gnueabi-1.4-r0-armv5te-2-079607fa7ece189eb79c856e9a16baa2_package.tgz.siginfo
5) Re-usage of the sstate file
- In classes/sstate.bbclass:
461 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
- In bitbake/lib/bb/runqueue.py:
self.hashvalidate = bb.data.getVar("BB_HASHCHECK_FUNCTION", cfgData, True) or None
- In the task's bbclass, for example, meta/classes/package.bbclass:
python do_package_setscene () {
sstate_setscene(d)
}
addtask do_package_setscene
The sstate_checkhashes is used to check whether there is a valid sstate
file, if there is one, then save the task name and return it to
bitbake/lib/bb/runqueue.py, and runqueue.py will set the task which
need to be run and skip other tasks, for example, only run do_package_rpm
for the package, and skip do_fetch, do_patch, do_config, do_compile and so on.
The sstate_setscene will used to restore the sstate file, let's see the
keypoint code in each file.
In classes/sstate.bbclass:
461 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
462
463 def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
464
465 ret = []
466 # This needs to go away, FIXME
467 mapping = {
468 "do_populate_sysroot" : "populate-sysroot",
469 "do_populate_lic" : "populate-lic",
470 "do_package_write_ipk" : "deploy-ipk",
471 "do_package_write_deb" : "deploy-deb",
472 "do_package_write_rpm" : "deploy-rpm",
473 "do_package" : "package",
474 "do_deploy" : "deploy",
475 }
376
477 for task in range(len(sq_fn)):
478 sstatefile = bb.data.expand("${SSTATE_DIR}/" + sq_hashfn[task] + "_" + mapping[sq_task[task]] + ".tgz", d)
479 sstatefile = sstatefile.replace("${BB_TASKHASH}", sq_hash[task])
480 if os.path.exists(sstatefile):
481 bb.debug(2, "SState: Found valid sstate file %s" % sstatefile)
482 ret.append(task)
483 continue
484 else:
485 bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile)
Note the array of "ret" which saves the tasks that we need run and
return to runqueue.py
In bitbake/lib/bb/runqueue.py:
761 class RunQueue:
762 def __init__(self, cooker, cfgData, dataCache, taskData, targets):
...
769 self.hashvalidate = bb.data.getVar("BB_HASHCHECK_FUNCTION", cfgData, True) or None
...
1371 class RunQueueExecuteScenequeue(RunQueueExecute):
...
1458 if self.rq.hashvalidate:
1459 sq_hash = []
1460 sq_hashfn = []
1461 sq_fn = []
1462 sq_taskname = []
1463 sq_task = []
1464 noexec = []
1465 stamppresent = []
1466 for task in xrange(len(self.sq_revdeps)):
...
1487 sq_taskname.append(taskname)
1488 sq_task.append(task)
...
At last, the sstate_setscene which is defined in sstate.bbclass and
invoked by the task's bbclass:
415 def sstate_setscene(d):
416 shared_state = sstate_state_fromvars(d)
417 accelerate = sstate_installpkg(shared_state, d)
418 if not accelerate:
419 raise bb.build.FuncFailed("No suitable staging package found")
The sstate_installpkg:
147 def sstate_installpkg(ss, d):
...
171 bb.build.exec_func('sstate_unpack_package', d)
...
The sstate_unpack_package:
455 sstate_unpack_package () {
456 mkdir -p ${SSTATE_INSTDIR}
457 cd ${SSTATE_INSTDIR}
458 tar -xvzf ${SSTATE_PKG}
459 }
The sstate_unpack_package will extract the files to the directory, and
bitbake will go on running the task based on the directory.
6) Pitfalls
- 'bitbake -cclean' doesn't remove the sstate file, If you
need rebuild the package completely, please remove the sstate file
from the SSTATE_DIR, e.g.:
rm -f SSTATE_DIR/sstate--*
- It doesn't calculate the md5sum of the file in the white list:
BB_HASHTASK_WHITELIST ?= "(.*-cross$|.*-native$|.*-cross-initial$|.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
This would make bitbake hit the sstate file incorrectly for some
packages, for example, as far as we know is eglibc.
- Sometimes, it should not hit the sstate file , but it does, this may
be caused by the problem of the depends file, please file a bug if
you met such a problem