slitaz-doc-wiki-data diff pages/en/devnotes/new-tazwok-illustrated.txt @ rev 7
Add pages/en folder.
| author | Christopher Rogers <slaxemulator@gmail.com> |
|---|---|
| date | Sat Feb 26 12:17:18 2011 +0000 (2011-02-26) |
| parents | |
| children | df9950c99d41 |
line diff
1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000 1.2 +++ b/pages/en/devnotes/new-tazwok-illustrated.txt Sat Feb 26 12:17:18 2011 +0000 1.3 @@ -0,0 +1,270 @@ 1.4 +====== The new tazwok illustrated! ====== 1.5 + 1.6 +Some items in revenue no longer needed with the new version of tazwok. During migration, a number of problems appear.The information on these two points are available below, with examples. 1.7 + 1.8 +By simplifying the writing of recipes, we simplify the work and contribution that benefits everyone! 1.9 + 1.10 +==== DEPENDS/BUILD_DEPENDS : ==== 1.11 +Tazwok now uses the dependent variable for finding the necessary compilation dependencies. 1.12 +Here's how it works: 1.13 + 1.14 + * Tazwok tree list of all dependencies from the dependent variable 1.15 + * For each package, if there is a shareholder *-dev package, it adds it to the dependencies 1.16 + * Tazwok done the same for BUILD_DEPENDS. 1.17 + 1.18 +So far, when a packet was both dependency and build dependency, the recipe looked like this: 1.19 + 1.20 +<file> 1.21 +DEPENDS="pkgX" 1.22 +BUILD_DEPENDS="pkgX pkgX-dev" 1.23 +</file> 1.24 +Now that is enough: 1.25 +<file> 1.26 +DEPENDS="pkgX" 1.27 +</file> 1.28 +Where there were trees more complex dependencies, the recipe looked like this: 1.29 +<file> 1.30 +# pkgY depend's on pkgX 1.31 +DEPENDS="pkgY" 1.32 +BUILD_DEPENDS="pkgY pkgY-dev pkgX pkgX-dev" 1.33 +</file> 1.34 +Now that is enough: 1.35 +<file> 1.36 +DEPENDS="pkgY" 1.37 +</file> 1.38 + 1.39 +The recipes also contain many redundancies in the definition of addiction, for example: 1.40 +<file> 1.41 +# pkgY depend's on pkgX 1.42 +DEPENDS="pkgY pkgX" 1.43 +</file> 1.44 +Here, needless to say since pkgX will be installed along pkgY anyway (Tazpkg manages dependencies automatically!). 1.45 + 1.46 +By following these three councils, it appears that about half the packages in DEPENDS / BUILD_DEPENDS can be removed without changing the revenue system behavior, it is not nothing! 1.47 + 1.48 +<note tip>An automated cleaning using some scripts is expected, after all recipes have been compiled at least once successfully using the new version of tazwok; In the meantime, these tips can be applied to writing new recipe for simplicity or manually with existing revenues when updating / corrections.</note> 1.49 + 1.50 +**Examples:** 1.51 + * graveman: http://hg.slitaz.org/wok/rev/7f0604e0bde0 1.52 + * enlightenment & cie: http://hg.slitaz.org/wok/rev/85cd798d6997 1.53 + 1.54 +==== TARBALL/WGET_URL/SOURCE/download from the VCS ==== 1.55 + 1.56 +This is important: always put the necessary tools to download / decompression sources in DEPENDS or BUILD_DEPENDS. This allows to define tazwok right firing order (do not try to cook a package that needs wget wget before himself). 1.57 + 1.58 +//The packets affected by this:// 1.59 + * wget url for https, ftps and some URLs that busybox does not include 1.60 + * mercurial/subversion/git: they are used to obtain the source 1.61 + * tar/unzip: sometimes necessary to unpack the sources. 1.62 + 1.63 +By default, tazwok re-sources in compact format .tar.lzma. He calls PACKAGE-VERSION.tar.lzma or SOURCE-VERSION.tar.lzma if SOURCE is defined. Note: Choose the name of the archive is now the only function of the variable SOURCE! 1.64 + 1.65 +Tazwok now supports files or "weird" URL (download.php?version=foo&blah=Idontknowwhat). The logic is: if WGET_URL does not end with tarball, then names the file downloaded tarball. 1.66 + 1.67 +Tazwok also supports the use of mercurial/subversion/git in WGET_URL. The syntax is: 1.68 +<file>WGET_URL="subversion|svn://svn.mplayerhq.hu/mplayer/trunk</file> 1.69 +An optional variable is BRANCH: it allows to specify the revision /tag/branch to use (see examples below). Where BRANCH is used, it is important that $VERSION is part of its definition. 1.70 + 1.71 +Note that the sources will be obtained through the requested tool, then packaged in .tar.lzma. The archive will be named as explained above. This means that the variable source can be used to ensure that many recipes use the same repository without creating multiple archives. 1.72 + 1.73 +First, it helps to know what revision is installed when using the package manager. Second, it allows to differentiate tazwok compressed sources. Indeed, if the archive keeps the same name, it will not be re-downloaded, which is undesirable when trying to update the package. 1.74 + 1.75 +**Examples:** 1.76 + * Here was necessary wget: http://hg.slitaz.org/wok/rev/012847ddd0cb 1.77 + * Tinyproxy did not report the URL of its source code is corrected: http://hg.slitaz.org/wok/rev/25967da0e1af 1.78 + * WGET_URL now supports xpi: http://hg.slitaz.org/wok/rev/37738b3ee08f 1.79 + * WGET_URL with a "weird" URL: http://hg.slitaz.org/wok/rev/102de15fea8d 1.80 + * WGET_URL using git: http://hg.slitaz.org/wok/rev/e06d60ae03eb 1.81 + * WGET_URL using subversion: http://hg.slitaz.org/wok/rev/c4c54646489a 1.82 + * WGET_URL using mercurial: http://hg.slitaz.org/wok/rev/756ed4b1daac 1.83 + * It was difficult to choose how to define BRANCH and VERSION for aufs: http://hg.slitaz.org/wok/rev/67231cfc5475 1.84 + * Here are two sources of records were in conflict, resolved by SOURCE: http://hg.slitaz.org/wok/rev/b891cba4f48e 1.85 + * slitaz-dev-tools contains the sources for SliTaz tools that contain very little code, using SOURCE="slitaz-dev-tools" in recipes that use this deposit to avoid having duplicate tarballs: http://hg.slitaz.org/wok/rev/808826645cc2 1.86 + 1.87 +==== Exceptions dependencies cooking ==== 1.88 +In some cases, no dependence cooking is installed: 1.89 + * For recipes with WANTED 1.90 + * For recipes without compile_rules() 1.91 + 1.92 +Note that packets may be required to obtain / decompressing the source code will still be installed if they are in DEPENDS / BUILD_DEPENDS. These are wget, mercurial, subversion, git, tar and unzip. 1.93 + 1.94 +If you do not compile_rules() but want to force the installation of all dependencies of cooking, there is a little hack: 1.95 +<file> 1.96 +compiles_rules() 1.97 +{ 1.98 + : 1.99 +} 1.100 +</file> 1.101 + 1.102 +**Examples:** 1.103 + * Removal of compiles_rules() to avoid installing unnecessary dependencies Cooking: http://hg.slitaz.org/wok/rev/f579356b437f 1.104 + * Remove a hack with fake compiles_rules was useless ... http://hg.slitaz.org/wok/rev/5b4581f8e476 1.105 + 1.106 +==== Define src/_pkg & move in the right place (hacks in the recipe) ==== 1.107 + 1.108 +By default, the new sources in place tazwok $WOK/$PACKAGES/$PACKAGE-$VERSION: it renames the parent directory of sources if necessary. So far, $src was not properly defined for recipes using both SOURCE and WANTED. Many recipes implement their own solution in different ways, which is difficult to consider a standardized way and can cause compatibility problems. 1.109 + 1.110 +If tazwok detects src=/_pkg= in a recipe, it continues to use the old behavior to ensure compatibility (this produces errors in some cases). It is no longer necessary and not ideal. 1.111 + 1.112 +The hacks in the recipe source that move in the right place are no longer needed either, and can also cause problems. 1.113 + 1.114 +In conclusion, it is better to consider that $src/$_pkg are defined by default and try to rely as much as possible. 1.115 + 1.116 +** Examples: ** 1.117 + * Removing src= by Godane: http://hg.slitaz.org/wok/rev/a1c1d35d9f92 1.118 + * src=/_pkg= can / should also be removed from WANTED: http://hg.slitaz.org/wok/rev/07adb7cbd0c8 1.119 + * Here, an old hack was the problem: http://hg.slitaz.org/wok/rev/62f6142d9fb3 1.120 + * Sources are now //always// placed in a sub-directory $src http://hg.slitaz.org/wok/rev/e64069568fe7 1.121 + * Another case: call the configure script from a folder separate compilation (*-build): http://hg.slitaz.org/wok/rev/7461a0c31d62 1.122 + * Fixed dmraid: http://hg.slitaz.org/wok/rev/f5b7e0c47763 http://hg.slitaz.org/wok/rev/59ea9409ad8a 1.123 + 1.124 +==== Set the default paths in configure: ==== 1.125 +<note tip>See /etc/slitaz/slitaz.conf, /etc/config.site and the new revenue model in place by New tazwok tree</note> 1.126 + 1.127 +The new version of tazwok attempts to pass the default paths to configure using the environment variable CONFIG_SITE calling /etc/config.site, which works in most cases. Nevertheless configure scripts are specific to each source and sometimes CONFIG_SITE not be supported. For this reason, the best way to remove the definitions of paths is unnecessary to do so on a case by case, when the updated recipe, and make sure everything works. 1.128 + 1.129 +In rare cases, this new product functionality problems. It happens that some recipes that did not use the default paths used by CONFIG_SITE now, and an update function genpkg_rules() is then mandatory. 1.130 + 1.131 +** Examples: ** 1.132 + * A file or did not install properly in acl is corrected by CONFIG_SITE: http://hg.slitaz.org/wok/rev/f831ecb652a6 1.133 + * Another example: http://hg.slitaz.org/wok/rev/259214792e30 1.134 + 1.135 +<note tip>CONFIG_SITE= can be used in recipes to use a different file than the default (can be useful for packages gnome or something like that ...)</note> 1.136 + 1.137 +==== DESTDIR=$PWD/_pkg ==== 1.138 + 1.139 +DESTDIR is passed to make install using the environment variable of the same name. The new path for installation is $ WOK/$PACKAGE/install. This will remove the source folder after packaging, it does not contain any file used by a recipe in his genpkg_rules(). 1.140 + 1.141 +Most recipes still use DESTDIR=$PWD/_pkg. However, if no revenue is redefining the variables src/_pkg, automatically move to the tazwok $WOK/$PACKAGE/install. 1.142 + 1.143 +In some cases, as with other variables, DESTDIR is not taken into account or the package is not installed by make. In these cases, the variable $DESTDIR is available to define the installation directory in the recipe. 1.144 + 1.145 +In rare cases, this behavior causes incompatibilities. This happens when revenues define the path to the installation folder without using src/_pkg. The solution is not to set these paths in revenue (that calling the main recipe with WANTED included), make sure the installation is done well in $WOK/$PACKAGE/install and trust the variables provided by tazwok. 1.146 + 1.147 +** Examples: ** 1.148 + * Remove _pkg= & DESTDIR= same time for this to work: http://hg.slitaz.org/wok/rev/cf088243a4a5 1.149 + * Remove references "useless" to $src to the sources are withdrawn: http://hg.slitaz.org/wok/rev/0731792c3994 http://hg.slitaz.org/wok/rev/5d6340961543 1.150 + * Bash does not take into account DESTDIR environment variable: http://hg.slitaz.org/wok/rev/fa7b7514e1d8 1.151 + * acl attr does not include DESTDIR (in this installation the destination was still $PWD/_pkg): http://hg.slitaz.org/wok/rev/fa7b7514e1d8 1.152 + 1.153 +==== MAKEFLAGS ==== 1.154 + 1.155 +MAKEFLAGS also increased to make using environment variables, and again this does not always work. In most cases, -j4 can be removed. In some cases it is necessary to use MAKEFLAGS directly to make the recipe: 1.156 +make $MAKEFLAGS 1.157 + 1.158 +Tazwok automatically sets the value for $MAKEFLAGS in the number of heart that contains the processor -j4 should be removed from all the recipes can be compiled on computers that have more resources (4 cores can use -j5) 1.159 + 1.160 +**Problems MAKEFLAGS:** 1.161 + 1.162 +So far, only recipes with-j4 compiled using the multi-threaded, whereas now all make and make install use. This behavior can cause errors. Some sources do not support multi-threaded compilation but do not disable. This is the most common problem associated with the changes explained here. 1.163 + 1.164 +//Problem in compiling:// 1.165 + 1.166 +During compilation, it happens that rely on other libraries compiled with the same sources. If they are compiled at the same time, this causes an error about a missing library. In this case, we see in the compilation text that the library in question has begun to be built few lines earlier, but that this process was not finished yet. To resolve this problem, add -j1 to make. It is the most common mistake, there are other more rare that take a similar form. 1.167 + 1.168 +//Problem in installation:// 1.169 + 1.170 +The characteristic of this error is that the installation stops and an error message says that it is impossible to create a folder because it already exists: a parallel process is actually creating it. In this case, add-j1 to make install. 1.171 + 1.172 +** Examples: ** 1.173 + * Several changes explained here in the recipe for gettext: http://hg.slitaz.org/wok/rev/9411655af0e2 1.174 + 1.175 +==== Variables $stuff, $wanted_stuff and $fs ==== 1.176 + 1.177 +Now the variable $stuff is available and returns the record stuff in the recipe, it uses an absolute path. The variable $wanted_stuff refers to the file defined in the package stuff WANTED, if any. The variable $fs refers to the future content of the package in taz/*/fs, as before, the difference is that now $fs uses an absolute path 1.178 + 1.179 +** Examples: ** 1.180 + * A commit with several changes regarding the variable $stuff: http://hg.slitaz.org/wok/rev/be13f25e790b 1.181 + * A correction necessary when we have made an absolute path $fs: http://hg.slitaz.org/wok/rev/8c897d2542ab 1.182 + 1.183 +==== Do not use 'exit' but 'return' ==== 1.184 + 1.185 +Now when cooking several packages with a list tazwok does not call for new tazwok cook. There is only one session tazwok so that the execution is faster. If a recipe uses exit, it leaves tazwok and following list is not cooked. 1.186 + 1.187 +**Example:** 1.188 + * Removing all exit the wok recipes: http://hg.slitaz.org/wok/rev/0b4cf0d9e1b5 1.189 + 1.190 +==== Conclusion - What to do when updating a recipe: ==== 1.191 + 1.192 + * Remove src=/_pkg= recipe and those who declare as WANTED. 1.193 + * Remove DESTDIR=$PWD/_pkg and if it does not work, or if the means to define the installation directory is not + make DESTDIR, rather than using $ DESTDIR $PWD/_pkg. 1.194 + * Remove the definition of default paths and see if it works, otherwise leave. 1.195 + * Remove-j4 and see if it works; If the mutli-threaded does not work, re-activate using $MAKEFLAGS; if multi-threading causes problems, add-j1 to the right place. 1.196 + * Remove BUILD_DEPENDS / DEPENDS redundant. 1.197 + * Check that packages are created correctly, otherwise update the paths in genpkg_rules(). 1.198 + * Try to declare all sources of revenue in that SliTaz can be compiled without internet connection (requires to download any source before). 1.199 + * Check that the package needed to download / extract the source code are defined in BUILD_DEPENDS. 1.200 + * Check as EXIT is not used in the recipe. 1.201 + 1.202 +==== Some more complex cases ... ==== 1.203 + 1.204 +I put them there in the end because there are already too many to be integrated:) 1.205 +The items below correspond to specific cases. 1.206 + 1.207 +== Variable COOK_OPT == 1.208 + 1.209 +This new variable can contain options that alter the behavior of tazwok. They are useful in very special cases. 1.210 + 1.211 +**genpkg=** 1.212 + 1.213 +In the recipe PACKAGE defines a priority order for the revenue package containing WANTED="PACKAGES" (and only them!). If you include multiple packages, separate them with double points ':'. If packets are not defined in this option, they will be packaged later, in alphabetical order (default) 1.214 + 1.215 +Used in glibc: http://hg.slitaz.org/wok/file/tip/glibc/receipt 1.216 + 1.217 +**!repack_src** 1.218 + 1.219 +Disable re-compression format sources .tar.lzma. 1.220 + 1.221 +Ruby-pkgconfig used for the sources remain in gem: http://hg.slitaz.org/wok/file/tip/ruby-pkgconfig/receipt 1.222 + 1.223 +**!unpack** 1.224 + 1.225 +Prevents decompression of the archive-source in the wok. 1.226 + 1.227 +This is used by ruby-pkgconfig as well (see link above) 1.228 + 1.229 +This is the only case yet! 1.230 + 1.231 +==== Cooking the toolchain ==== 1.232 + 1.233 +To cook the toolchain SliTaz, we use a temporary toolchain. Some recipes use specific rules in this step. When cooking this toolchain temporarily affected software packaged but are not directly installed in the chroot purpose built. The affected packages are listed in the variable SLITAZ_TOOLCHAIN configuration file /etc/slitaz/slitaz.conf 1.234 + 1.235 +Additional features are: 1.236 + * Precook_tmp_toolchain() - Used only by gcc & binutils for the moment, because they are cooked twice during the preparation of the temporary toolchain. 1.237 + * Cook_tmp_toolchain() - Used by the most packets SLITAZ_TOOLCHAIN to define how they should be compiled for the temporary toolchain. When cook_tmp_toolchain() is absent, compile_rules() is used instead. This avoids writing two identical functions. Note that in this case. ./configure does not set the default paths in the recipe, because the temporary toolchain must be able to do it via the environment variable CONFIG_SITE. Indeed, packets compiled during this stage are not installed in the usual place but in /tools. 1.238 + 1.239 +** Examples: ** 1.240 + * Binutils: http://hg.slitaz.org/wok/file/tip/binutils/receipt 1.241 + * Gettext: http://hg.slitaz.org/wok/file/tip/gettext/receipt 1.242 + * Bash: http://hg.slitaz.org/wok/file/tip/bash/receipt 1.243 + * Patch does not need cook_tmp_toolchain(): http://hg.slitaz.org/wok/file/tip/patch/receipt 1.244 + * Autoconf either: http://hg.slitaz.org/wok/file/tip/autoconf/receipt 1.245 + 1.246 +== tazwok get-src / report in recipes == 1.247 + 1.248 +Report is a module libtaz for organizing the display commands in the terminal and create logs available including http://bb.slitaz.org interface. It can be used in recipes, as follows (that is abstract, the following examples of actual application): 1.249 + 1.250 +<file> 1.251 +compile_rules() # Per example 1.252 +{ 1.253 + report open-bloc #compiles_rules is a step, declaring that there will substeps 1.254 + report step "Action machin" 1.255 + ... 1.256 + report step "Action truc" 1.257 + .. 1.258 + report close-bloc #Close the open block previously 1.259 +} 1.260 +</file> 1.261 + 1.262 +Specifically, there is one case where we use it: when using get-src tazwok PACKAGE --target=... . This command creates a new step (postponement step). We need to open a block before and after the close, as well as adding several other deferral step "..." to log and display in the terminal is correct. Each farm deferral step the previous step, if we do not open the block, get-src shut tazwok step "Executing compiles rules" 1.263 + 1.264 +Delaying closed block must be completely executed, otherwise the log/display will be broken. That's why we use { report-block closed; return 1; } return rather than alone. 1.265 + 1.266 +The practical use of this tazwok get-src is that you can unpack the sources of designated PACKAGE at target. 1.267 + 1.268 +In the examples below, observe the correlation between the delay step and displayed in the log. Observe also the correlation between tazwok get-src and the message "Checking for source tarball ..." in the log. You learn how to carry report open-bloc/closed block create a subset in genpkg_rules (named "Executing compile_rules" in the log). If it were not for open-bloc/close-bloc, the new steps would be posted to the result of "Executing compile_rules" This is not what we wanted. 1.269 + 1.270 +** Examples (recipe + log): ** 1.271 + * Linux needs patches contained in the sources of aufs, Godane took the opportunity to improve the log. Recipe: http://hg.slitaz.org/wok/file/tip/linux/receipt; log: http://bb.slitaz.org/log.php?version=cooking&package=linux 1.272 + * Gcc uses several other sources of packages during the cooking of the temporary toolchain. Recipe: http://hg.slitaz.org/wok/file/tip/gcc/receipt; log: http://bb.slitaz.org/log.php?version=cooking&package=tmp-toolchain-gcc 1.273 + * mingw32-gcc was corrected using this approach, it also allowed to declare all sources used. Commit: http://hg.slitaz.org/wok/rev/fd43246b4613; log: http://bb.slitaz.org/log.php?version=cooking&package=mingw32-gcc 1.274 \ No newline at end of file